fix(fmsr): take the FMSR model from FMSR_MODEL_ID, with no built-in default

[DhavalRepo18]

fix(fmsr): take the FMSR model from FMSR_MODEL_ID, with no built-in default

Base: harbor-private-mount-docker-md (at 460543f) · Head: fmsr-model-pin
Ported from: main_fmsr_llm (1b28baa), FMSR changes only, with the watsonx
fallback dropped rather than retained.

The bug

generate_failure_modes calls an LLM inside the FMSR MCP server. The model came
from FMSR_MODEL_ID, and with that unset the server fell back to
watsonx/meta-llama/llama-3-3-70b-instruct. Nothing ever set it, so every run
needed WATSONX_APIKEY and WATSONX_PROJECT_ID regardless of which model was
being benchmarked:

ERROR:fmsr-mcp-server:generate_failure_modes failed: litellm.BadRequestError:
WatsonxException - {"errorCode":"BXNIM0415E",
"errorMessage":"Provided API key could not be found."}

Found on a mini-suite sweep: five trials hit it, fmsr-902 four times and
fmsr-913 once. The tool was dead for the whole trial, and the error named
neither the model nor the missing variables, so the cause was invisible from the
trajectory.

The fix

FMSR_MODEL_ID is the only source, and there is no default. A hardcoded
default sent every run at one provider and demanded that provider's credentials
whatever model was under test. That is the bug, not a convenience, so it is gone
rather than changed to a different vendor.

Agent runners pin FMSR_MODEL_ID for the MCP servers they spawn. An explicit
value from the shell or .env wins; otherwise the agent's own --model-id. The
value is always set, even when it equals the agent model, so the FMSR model is
fixed and visible for every run. A normal run needs no extra configuration and
the FMSR tools use the same model and credentials as the agent under test.

src/agent/runner.py holds three helpers:

resolve_fmsr_model_id(agent_model_id)  # explicit > agent model > None
fmsr_env_overrides(agent_model_id)     # {FMSR_MODEL_ID: ...} or {}
mcp_server_env(agent_model_id)         # {**os.environ, **overrides}

Each runner calls one where it builds its stdio server specs: Stirrup, deep,
openai and plan-execute take the full mcp_server_env (the MCP SDK otherwise
passes only HOME/PATH/SHELL to children); claude and opencode already
forward a parent environment and take fmsr_env_overrides on top.

A server started standalone with FMSR_MODEL_ID unset says so instead of
guessing:

LLM unavailable (FMSR_MODEL_ID is not set; the generate_* tools need a model id,
e.g. tokenrouter/<model> or litellm_proxy/<provider>/<model>)

_build_llm takes the model as an argument and keeps its per-prefix credential
checks, so an explicit watsonx model still reports the right missing variables
for anyone who has it. Empty values count as unset rather than as a model id.

Why the Harbor path needs nothing extra

_build_mcp_config in the Stirrup runner already forwarded the parent
environment so the servers could see COUCHDB_URL and the *_DBNAME overrides.
mcp_server_env keeps that and adds the pin, so the one call site covers both
the CLI and the Harbor adapter. CREDENTIAL_ENV_VARS in
assetops_harbor/stirrup.py is untouched.

Verification

FMSR_MODEL_ID unset (standalone)
  model : None
  usable: False
  error : FMSR_MODEL_ID is not set; the generate_* tools need a model id, ...

a tool call in that state
  error='LLM unavailable (FMSR_MODEL_ID is not set; ...)'

pinned by the runner, no other provider's credentials present
  model : tokenrouter/MiniMax-M3
  usable: True

The Stirrup MCP config carries the pin without losing the environment forwarding:

FMSR_MODEL_ID  -> tokenrouter/MiniMax-M3
COUCHDB_URL    -> http://couchdb:5984
explicit wins  -> litellm_proxy/aws/claude-opus-5

246 tests pass across src/agent, src/servers/fmsr and
src/assetops_harbor (1 skipped, the watsonx-gated live call). ruff reports
one finding fewer than the base branch on the touched files.

Docs

docs/running_benchmark.md loses its "WatsonX — needed more often than it looks"
section, since WatsonX is no longer implicated, and gains one describing how the
model is pinned. The troubleshooting row now points at the model named in the
error rather than at absent WatsonX credentials. INSTRUCTIONS.md and
.env.public state that there is no default.

Two notes for review

FMSR_MODEL_ID is not in CREDENTIAL_ENV_VARS, so a host-side override does not
reach the agent container through Harbor. It is not needed for the fix, since the
in-container runner derives the value from --model-id, but add it there if you
want to pin a single FMSR model across a Harbor sweep.

src/servers/fmsr/tests/test_tools.py still gates its one live-LLM test on
requires_watsonx, so that test now always skips. Generalising it to whatever
FMSR_MODEL_ID names would be a good follow-up; I left it alone to keep this
diff to the fix.

Not ported

main_fmsr_llm sits on an older main, so a plain diff against this branch shows
thousands of unrelated deletions (the whole benchmarks/harbor/ tree,
assetops_harbor/, uv.lock, pyproject.toml). This PR carries only the files
that commit actually changed, plus the doc updates the removed default requires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m