Skip to content

feat: auto-detect VLM checkpoints on the CLI, and fix a false positive it exposed - #152

Merged
solderzzc merged 2 commits into
mainfrom
feat/cli-vlm-auto-detect
Aug 16, 2026
Merged

feat: auto-detect VLM checkpoints on the CLI, and fix a false positive it exposed#152
solderzzc merged 2 commits into
mainfrom
feat/cli-vlm-auto-detect

Conversation

@solderzzc

Copy link
Copy Markdown
Member

Extracted from `fix/pr57-speculative-ci` rather than a rebase (context: #109's resolution). Its `ModelArchitectureProbe.swift` and tests were already shipped verbatim, presumably via #57 itself — only the CLI wiring and its test were still missing.

The gap

`Server.swift` already probed the architecture at load time and then discarded the result for the CLI: `let isVision = self.vision`. Confirmed the user-facing effect directly before touching anything:

```
$ SwiftLM --model LiquidAI/LFM2.5-VL-450M-MLX-4bit # no --vision
[SwiftLM] Loading LLM (large language model)...

subsequent image request fails

```

`InferenceEngine.swift` — SwiftBuddy's own loader — already auto-detects unconditionally. The CLI now mirrors that, keeping `--vision`/`--audio` as explicit overrides:

```swift
let isVision = self.vision || (!self.audio && architecture.supportsVision)
```

Verified all three directions: LFM2.5-VL-450M now loads as VLM and answers an image request with no flag; a plain LLM (Qwen2.5-0.5B) does not false-positive; explicit `--vision` still forces VLM loading.

What wiring this up exposed

The fixtures suite broke. `moe-nested` (#145) started failing with `Key vision_tower.patch_embedder... not found`, because its `model_type: "gemma4"` — chosen for the unrelated reason that `Gemma4Configuration` decodes `text_config` only, giving the MoE-nesting test a real nested shape to exercise — was now read by the newly-active CLI auto-detection and routed to `VLMModelFactory`.

Not a fixture-only bug. `MLXLLM/Gemma4.swift`'s `Gemma4Configuration` (the `LLMModelFactory` `"gemma4"` entry) has no `vision_config` field at all — a real, intentionally-supported text-only checkpoint shape in this codebase's own registry. `ModelArchitectureProbe.knownVisionModelTypes` listed bare `"gemma4"` as vision-triggering regardless of that. This would have broken loading a real text-only Gemma4 checkpoint the moment anything started acting on the probe's result — which is exactly what this PR does. Removed `"gemma4"` from that list; the separate `vision_config != nil` check already distinguishes the two correctly, since only `VLMModelFactory`'s struct requires that field.

Also swapped the fixture's decoy container from `vision_config` to `audio_config` — independently the more honest choice, since a real text-only Gemma4 checkpoint would never carry `vision_config` at all. `audio_config` is excluded from the expert-count walk by `ModelProfiler.nonLanguageContainers` the same way, so the decoy still proves what it proved before.

Regression test

`testVLM_AutoDetectsLFM25WithoutVisionFlag` in `tests/SwiftBuddyTests/VLMTests.swift`, adapted from `fix/pr57-speculative-ci` to match main's already-refactored `captureStartupOutput` helper (main had the refactor; only the second test case was missing).

Red-green verified: fails with `Unsupported model type: lfm2-vl` against the pre-fix `isVision` line, passes once restored.

Verification

check result
fixtures 6 passed, 0 failed
contract 10 passed, 0 failed, 2 skipped
VLMTests both cases pass
LFM2.5-VL-450M auto-detects, answers image request
Qwen2.5-0.5B (plain LLM) no false positive
explicit `--vision` still works

Refs #109

🤖 Generated with Claude Code

solderzzc and others added 2 commits August 15, 2026 22:51
…e it exposed

Extracted from fix/pr57-speculative-ci rather than a rebase (see #109's
resolution and the discussion around it). Its ModelArchitectureProbe.swift and
tests were already shipped verbatim; only the CLI wiring and its test were
still missing.

**The gap.** Server.swift already probed the architecture at load time but
discarded the result for the CLI entry point: `let isVision = self.vision`.
Confirmed the user-facing effect directly: `SwiftLM --model
LiquidAI/LFM2.5-VL-450M-MLX-4bit` (no --vision) printed "Loading LLM" and the
subsequent image request failed. InferenceEngine.swift — SwiftBuddy's own
loader — already auto-detects unconditionally; the CLI now mirrors that,
keeping --vision/--audio as explicit overrides:

    let isVision = self.vision || (!self.audio && architecture.supportsVision)

Verified both directions: LFM2.5-VL-450M-MLX-4bit now loads as a VLM and
answers an image request without --vision; a plain LLM (Qwen2.5-0.5B) still
loads as LLM with no flags; --vision still forces VLM loading explicitly.

**What wiring this up exposed.** The fixtures suite broke: `moe-nested`
started failing with "Key vision_tower.patch_embedder... not found", because
that fixture's `model_type: "gemma4"` — carried since #145, for the earlier,
narrower reason that Gemma4Configuration decodes text_config only, giving the
MoE-nesting test a real nested config to exercise — was now read by the CLI's
newly-active auto-detection and routed to VLMModelFactory.

That is not a fixture bug on its own. `MLXLLM/Gemma4.swift`'s
Gemma4Configuration (the LLMModelFactory "gemma4" entry) has no vision_config
field at all — a real, intentionally-supported text-only checkpoint shape.
ModelArchitectureProbe.knownVisionModelTypes listed bare "gemma4" as
vision-triggering regardless, which is a genuine false positive for any real
text-only Gemma4 checkpoint using that shape, not just this fixture — auto-
detection would have broken loading one in production. Removed "gemma4" from
that list; the separate `vision_config != nil` check already distinguishes
the two correctly, since only VLMModelFactory's Gemma4Configuration requires
that field.

Also swapped the fixture's decoy container from vision_config to audio_config
— unrelated to the bug above, but vision_config as a decoy was already the
wrong choice: a real text-only Gemma4 checkpoint would never carry that key,
so the fixture is more honest this way regardless of the probe fix.
audio_config is excluded from the expert-count walk by
ModelProfiler.nonLanguageContainers the same way vision_config was, so the
decoy still proves what it proved before.

Regression test added: testVLM_AutoDetectsLFM25WithoutVisionFlag in
tests/SwiftBuddyTests/VLMTests.swift, adapted from fix/pr57-speculative-ci to
match main's already-refactored captureStartupOutput helper (main had the
refactor; only this second test case was missing). Red-green verified: fails
with "Unsupported model type: lfm2-vl" against the pre-fix isVision line,
passes once restored.

Verified together: fixtures 6/0, contract 10/0/2, both VLMTests cases,
LFM2.5-VL-450M-MLX-4bit auto-detects, Qwen2.5-0.5B does not false-positive,
explicit --vision still works.

Refs #109

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's speculative-decoding suite started crashing (Trace/BPT trap on the
first real generation request) after auto-detection began routing
mlx-community/Qwen3.5-*-4bit through VLMModelFactory instead of
LLMModelFactory. That checkpoint genuinely ships a vision_config even
when used purely as a text draft/main pair in this test, and qwen3_5 is
registered in both factories — the same ambiguity as gemma4, just
surfaced through a different flag combination.

--draft-model/--dflash/--mtp only wire up BaseLanguageModel from
LLMModelFactory, so auto-detection now backs off whenever speculative
decoding is requested, matching the pre-auto-detect behavior for that
path. --vision remains a valid explicit override.
@solderzzc
solderzzc merged commit cc39e61 into main Aug 16, 2026
14 checks passed
@solderzzc
solderzzc deleted the feat/cli-vlm-auto-detect branch August 16, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant