feat: auto-detect VLM checkpoints on the CLI, and fix a false positive it exposed - #152
Merged
Conversation
…e it exposed Extracted from fix/pr57-speculative-ci rather than a rebase (see #109's resolution and the discussion around it). Its ModelArchitectureProbe.swift and tests were already shipped verbatim; only the CLI wiring and its test were still missing. **The gap.** Server.swift already probed the architecture at load time but discarded the result for the CLI entry point: `let isVision = self.vision`. Confirmed the user-facing effect directly: `SwiftLM --model LiquidAI/LFM2.5-VL-450M-MLX-4bit` (no --vision) printed "Loading LLM" and the subsequent image request failed. InferenceEngine.swift — SwiftBuddy's own loader — already auto-detects unconditionally; the CLI now mirrors that, keeping --vision/--audio as explicit overrides: let isVision = self.vision || (!self.audio && architecture.supportsVision) Verified both directions: LFM2.5-VL-450M-MLX-4bit now loads as a VLM and answers an image request without --vision; a plain LLM (Qwen2.5-0.5B) still loads as LLM with no flags; --vision still forces VLM loading explicitly. **What wiring this up exposed.** The fixtures suite broke: `moe-nested` started failing with "Key vision_tower.patch_embedder... not found", because that fixture's `model_type: "gemma4"` — carried since #145, for the earlier, narrower reason that Gemma4Configuration decodes text_config only, giving the MoE-nesting test a real nested config to exercise — was now read by the CLI's newly-active auto-detection and routed to VLMModelFactory. That is not a fixture bug on its own. `MLXLLM/Gemma4.swift`'s Gemma4Configuration (the LLMModelFactory "gemma4" entry) has no vision_config field at all — a real, intentionally-supported text-only checkpoint shape. ModelArchitectureProbe.knownVisionModelTypes listed bare "gemma4" as vision-triggering regardless, which is a genuine false positive for any real text-only Gemma4 checkpoint using that shape, not just this fixture — auto- detection would have broken loading one in production. Removed "gemma4" from that list; the separate `vision_config != nil` check already distinguishes the two correctly, since only VLMModelFactory's Gemma4Configuration requires that field. Also swapped the fixture's decoy container from vision_config to audio_config — unrelated to the bug above, but vision_config as a decoy was already the wrong choice: a real text-only Gemma4 checkpoint would never carry that key, so the fixture is more honest this way regardless of the probe fix. audio_config is excluded from the expert-count walk by ModelProfiler.nonLanguageContainers the same way vision_config was, so the decoy still proves what it proved before. Regression test added: testVLM_AutoDetectsLFM25WithoutVisionFlag in tests/SwiftBuddyTests/VLMTests.swift, adapted from fix/pr57-speculative-ci to match main's already-refactored captureStartupOutput helper (main had the refactor; only this second test case was missing). Red-green verified: fails with "Unsupported model type: lfm2-vl" against the pre-fix isVision line, passes once restored. Verified together: fixtures 6/0, contract 10/0/2, both VLMTests cases, LFM2.5-VL-450M-MLX-4bit auto-detects, Qwen2.5-0.5B does not false-positive, explicit --vision still works. Refs #109 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's speculative-decoding suite started crashing (Trace/BPT trap on the first real generation request) after auto-detection began routing mlx-community/Qwen3.5-*-4bit through VLMModelFactory instead of LLMModelFactory. That checkpoint genuinely ships a vision_config even when used purely as a text draft/main pair in this test, and qwen3_5 is registered in both factories — the same ambiguity as gemma4, just surfaced through a different flag combination. --draft-model/--dflash/--mtp only wire up BaseLanguageModel from LLMModelFactory, so auto-detection now backs off whenever speculative decoding is requested, matching the pre-auto-detect behavior for that path. --vision remains a valid explicit override.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracted from `fix/pr57-speculative-ci` rather than a rebase (context: #109's resolution). Its `ModelArchitectureProbe.swift` and tests were already shipped verbatim, presumably via #57 itself — only the CLI wiring and its test were still missing.
The gap
`Server.swift` already probed the architecture at load time and then discarded the result for the CLI: `let isVision = self.vision`. Confirmed the user-facing effect directly before touching anything:
```
$ SwiftLM --model LiquidAI/LFM2.5-VL-450M-MLX-4bit # no --vision
[SwiftLM] Loading LLM (large language model)...
subsequent image request fails
```
`InferenceEngine.swift` — SwiftBuddy's own loader — already auto-detects unconditionally. The CLI now mirrors that, keeping `--vision`/`--audio` as explicit overrides:
```swift
let isVision = self.vision || (!self.audio && architecture.supportsVision)
```
Verified all three directions: LFM2.5-VL-450M now loads as VLM and answers an image request with no flag; a plain LLM (Qwen2.5-0.5B) does not false-positive; explicit `--vision` still forces VLM loading.
What wiring this up exposed
The fixtures suite broke. `moe-nested` (#145) started failing with `Key vision_tower.patch_embedder... not found`, because its `model_type: "gemma4"` — chosen for the unrelated reason that `Gemma4Configuration` decodes `text_config` only, giving the MoE-nesting test a real nested shape to exercise — was now read by the newly-active CLI auto-detection and routed to `VLMModelFactory`.
Not a fixture-only bug. `MLXLLM/Gemma4.swift`'s `Gemma4Configuration` (the `LLMModelFactory` `"gemma4"` entry) has no `vision_config` field at all — a real, intentionally-supported text-only checkpoint shape in this codebase's own registry. `ModelArchitectureProbe.knownVisionModelTypes` listed bare `"gemma4"` as vision-triggering regardless of that. This would have broken loading a real text-only Gemma4 checkpoint the moment anything started acting on the probe's result — which is exactly what this PR does. Removed `"gemma4"` from that list; the separate `vision_config != nil` check already distinguishes the two correctly, since only `VLMModelFactory`'s struct requires that field.
Also swapped the fixture's decoy container from `vision_config` to `audio_config` — independently the more honest choice, since a real text-only Gemma4 checkpoint would never carry `vision_config` at all. `audio_config` is excluded from the expert-count walk by `ModelProfiler.nonLanguageContainers` the same way, so the decoy still proves what it proved before.
Regression test
`testVLM_AutoDetectsLFM25WithoutVisionFlag` in `tests/SwiftBuddyTests/VLMTests.swift`, adapted from `fix/pr57-speculative-ci` to match main's already-refactored `captureStartupOutput` helper (main had the refactor; only the second test case was missing).
Red-green verified: fails with `Unsupported model type: lfm2-vl` against the pre-fix `isVision` line, passes once restored.
Verification
Refs #109
🤖 Generated with Claude Code