You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: preset base is not enforced - the preset may use any model #4197
type: presetname: my-presetbase: Qwen/Qwen3.8-27Btrials: 4# Agentic shape: long context, short turns, most of it cache-hittable, small-scale concurrencymin_context_length: 262144max_ttft: 5000concurrency: 4input_tokens: 131072output_tokens: 512shared_prefix_tokens: 130048env:
- HF_TOKEN
Compare base and model in ~/.dstack/presets/<id>/final_report.json.
Actual behaviour
base is not enforced. The agent may serve any repo it chooses, including a model from a different family, and the preset is still verified and reported as successful.
Verification checks only that the service advertises the requested name and that the report echoes the requested base:
raiseCLIError("Claude final report base does not match the requested model")
elifreport.model!=configuration.model.exact_repo:
raiseCLIError("Claude changed an exact model request")
Observed with base: Qwen/Qwen3.8-27B: the report recorded model: Qwen/Qwen3.5-27B-GPTQ-Int4, a different model generation, and the exported service ran vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name Qwen/Qwen3.8-27B. Whether a substitution happens depends on the agent's judgment about what counts as a compatible repo, so it does not reproduce on every run.
I suspect agent will use a different model if it fails to satisfy the requirements. In my case, it failed to meet TTFT on A100 for 2 trials and switched the model generation.
changed the title [-][Bug]: preset `base` is not enforced - the agent may serve a model from a different family[/-][+][Bug]: preset `base` is not enforced - the preset may use any model[/+]on Aug 24, 2026
Activity
[-][Bug]: preset `base` is not enforced - the agent may serve a model from a different family[/-][+][Bug]: preset `base` is not enforced - the preset may use any model[/+]github-actions commentedon Sep 24, 2026
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Oct 8, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.