Repository navigation
Doc for code execution - #581
Merged
ShuxinLin merged 17 commits intoSep 30, 2026
Merged
Conversation
generate_tasks.py gains three options for corpora that do not ship in the repo, such as AssetOpsBenchScenarioGeneration/scenarios_data: - --runtime-image: the image each task builds FROM. - --data-dir: where the corpus lives inside that image. The per-task layer copies the scenario there and the healthcheck runs init_data.py with SCENARIOS_DATA_DIR pointed at it; the agent keeps the repo copy, as scenario_suite_runner does. - --skip-missing: warn and skip profile scenarios absent from the corpus (all.yaml lists wosr-62, which the corpus lacks). corpus-image/Dockerfile bakes the corpus (2.1 GB, mostly shared/iot) into one layer over the runtime image, so tasks share it instead of each carrying it in their build context. Also stop copying scenario 1's description and the wosr keyword into every generated task. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Runs a scenario corpus through Harbor with the same profile, the same
stirrup-agent and the same Docker code sandbox as benchmarks/run.sh, but
with scenarios in parallel, each trial on its own CouchDB.
- Builds assetopsbench/runtime:corpus from -s, and the code sandbox tar
for the per-trial Docker-in-Docker daemon.
- Regenerates the dataset from scratch (the generator never removes stale
task folders), skipping profile entries the corpus lacks.
- One Harbor job per model under <leaderboard>/harbor-jobs; re-running
resumes it, the equivalent of --skip-existing.
- Credentials reach the Harbor process through uv run --env-file only.
Uses [[ -z "${arr[*]+set}" ]] and ${arr[@]+...} so empty arrays work
under set -u in macOS's bash 3.2.
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
corpus-image/ -> suite-image/, assetopsbench/runtime:corpus -> assetopsbench/runtime:suite, /opt/corpus/scenarios_data -> /opt/suite/scenarios_data, and the generated dataset assetopsbench-corpus -> assetopsbench-suite, with matching wording in run.sh, the generator's help and the README. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Completes 533bc5a, which only moved corpus-image/ to suite-image/: assetopsbench/runtime:corpus -> assetopsbench/runtime:suite, /opt/corpus/scenarios_data -> /opt/suite/scenarios_data, the dataset assetopsbench-corpus -> assetopsbench-suite, and the wording in run.sh, the generator's help and the README. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
A dropped VPN made the LiteLLM proxy unreachable mid-session. Every trial still built its containers, loaded its data, retried the model for four minutes and exited 1, and the job recorded each as done, so a resume would not have re-run them. - Probe LITELLM_BASE_URL / TOKENROUTER_BASE_URL (per model prefix) before starting or resuming a job, and skip the model with a clear message when the router does not answer. - Resume with --filter-error-type NonZeroAgentExitCodeError, so trials whose agent crashed run again while scored trials are kept. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Resolve the README conflict by keeping both new sections. The base's hand-generation section now uses this branch's --runtime-image, --data-dir and --skip-missing flags, and its known-gap note becomes what happens without the suite image. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Add overlays/private-data.yaml, which bind-mounts the directory in AOB_PRIVATE_DIR read-only into main at /opt/suite/scenarios_data, the path the suite image uses. Tasks generated with --data-dir then load their manifests' shared/ files from the mount, with no suite image rebuild when the data changes. The variable is required with :? so an unset value fails naming it rather than as an invalid volume spec. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Drop --data-dir. Any --scenario-root other than the repo's own src/couchdb/scenarios_data now marks the scenarios as an external suite, and the healthcheck loads them from /opt/suite/scenarios_data, where both the suite image and overlays/private-data.yaml put the suite. With no --scenario-root, or the repo's folder named explicitly, tasks load the repo copy as before, and manifests' shared/ paths resolve against it. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
…efault
generate_failure_modes calls an LLM inside the FMSR MCP server. The model came
from FMSR_MODEL_ID, and with that unset the server fell back to
watsonx/meta-llama/llama-3-3-70b-instruct. Nothing set it, so every run needed
WATSONX_APIKEY and WATSONX_PROJECT_ID regardless of which model the agent was
being benchmarked on, and without them the tool was dead for the whole trial:
ERROR:fmsr-mcp-server:generate_failure_modes failed:
litellm.BadRequestError: WatsonxException - {"errorCode":"BXNIM0415E",
"errorMessage":"Provided API key could not be found."}
Observed on five trials of a mini-suite sweep (fmsr-902 four times, fmsr-913
once), and the error text named neither the model nor the missing variables, so
the cause was not visible from the trajectory.
FMSR_MODEL_ID is now the only source for that model, and there is no default at
all. A hardcoded one sent every run at a single provider and demanded that
provider's credentials whatever model was under test, which is the bug rather
than a convenience.
Agent runners pin FMSR_MODEL_ID for the MCP servers they spawn: an explicit
value from the shell or .env wins, otherwise the agent's own --model-id. The
value is always set, even when it equals the agent model, so the FMSR model is
fixed and visible for every run. src/agent/runner.py holds the three helpers;
each runner calls one where it builds its stdio server specs. A normal run
therefore needs no extra configuration and the FMSR tools use the same model and
credentials as the agent under test.
A server started standalone with FMSR_MODEL_ID unset reports the generate_*
tools as unavailable and says why, rather than picking a provider:
LLM unavailable (FMSR_MODEL_ID is not set; the generate_* tools need a
model id, e.g. tokenrouter/<model> or litellm_proxy/<provider>/<model>)
_build_llm takes the model as an argument and keeps its per-prefix credential
checks, so an explicit watsonx model still reports the right missing variables.
Empty values count as unset rather than as a model id.
The Stirrup runner already forwarded the parent environment so the servers could
see COUCHDB_URL and the *_DBNAME overrides; mcp_server_env keeps that and adds
the pin, so the Harbor path is covered by the same change.
Docs follow: running_benchmark.md loses its "WatsonX - needed more often than it
looks" section, since WatsonX is no longer implicated, and the troubleshooting
row now points at the model named in the error.
Verified: unset resolves to no model and the tools report the cause; pinned to
tokenrouter/MiniMax-M3 the backend builds with no other provider's credentials
present. The Stirrup MCP config carries FMSR_MODEL_ID=tokenrouter/MiniMax-M3
alongside COUCHDB_URL, and an explicit FMSR_MODEL_ID overrides the agent model.
246 tests pass across src/agent, src/servers/fmsr and src/assetops_harbor; ruff
reports one finding fewer than the base branch on these files.
Ported from main_fmsr_llm (1b28baa), FMSR changes only, with the watsonx
fallback dropped rather than retained.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
We do not run WatsonX, so the FMSR server should not carry a half-supported
path for it. _build_llm now takes its accepted prefixes and their credential
variables from llm.routers.PROXY_ROUTERS, which already declares itself the
single source of truth for router prefixes and lists exactly litellm_proxy/ and
tokenrouter/. The hand-written per-prefix branches are gone, so the two lists
cannot drift and a new router is added in one place.
A model id with no known prefix is now rejected rather than pushed at LiteLLM on
a guess, which the old else-branch did:
FMSR_MODEL_ID='watsonx/...' -> no supported router prefix (litellm_proxy/<model>, tokenrouter/<model>)
FMSR_MODEL_ID='gpt-4o' -> no supported router prefix (...)
tokenrouter/ without keys -> Missing env vars for tokenrouter: [...]
tokenrouter/ with keys -> OpenAICompatBackend
litellm_proxy/ with keys -> LiteLLMBackend
The FMSR test gate followed: requires_watsonx skipped on WATSONX_APIKEY, so the
one live-LLM test never ran once WatsonX went away. requires_fmsr_llm now gates
on whether the configured FMSR model is actually usable, so the test runs
whenever FMSR_MODEL_ID and that router's credentials are set.
247 tests pass across src/agent, src/servers/fmsr and src/assetops_harbor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Credentials already reached the FMSR server: _credential_env forwards every non-empty CREDENTIAL_ENV_VARS entry unfiltered by prefix, so both routers' keys land in the container, and mcp_server_env passes the whole parent environment to the spawned servers. The default case was therefore fine, because FMSR runs on the agent's own model and those credentials are present by definition. FMSR_MODEL_ID itself was not forwarded. An operator who exported it on the host had it dropped at the container boundary, and the in-container runner then set it to the agent's --model-id, so the documented "explicit value wins" held for the CLI and silently did not for Harbor. SETTING_ENV_VARS carries it across as a setting rather than a credential. That makes a differently-routed FMSR model reachable, so _require_router_credentials now checks FMSR_MODEL_ID alongside --model-id. An agent on tokenrouter with FMSR pinned to litellm_proxy and no LITELLM_* present used to build an image and a container per trial before failing at the first generate_failure_modes call; it now fails before Harbor builds anything, naming which of the two model settings needs the variables. FMSR_MODEL_ENV is a literal here for the same reason ROUTER_CREDENTIALS is a copy: importing agent.runner would pull the agent SDKs into the host-side Harbor process. A test asserts it tracks agent.runner.FMSR_MODEL_ENV. Verified end to end. Default: FMSR model tokenrouter/MiniMax-M3 with both TOKENROUTER_* present. Operator pins litellm_proxy/aws/claude-opus-5: that model arrives in the server env with both LITELLM_* present. 250 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
requires_fmsr_llm replaced requires_watsonx so the one live generate_failure_modes
test would run again once a model was configured. It then ran on a machine with no
CouchDB and failed:
assert 'generated' in {'error': "HTTPConnectionPool(host='localhost', port=5984)
... Connection refused"}
The test needs the LLM and the failure_mode database. The old watsonx gate hid
that by skipping for an unrelated reason, so the missing database requirement was
never expressed.
requires_failure_mode_db probes fm_db with a real query, mirroring
_couchdb_reachable in the iot server's conftest, so a configured-but-down CouchDB
skips rather than fails. The integration test carries both markers.
Verified with no CouchDB running: 22 passed, 1 skipped, the skip naming the
database. 250 tests pass across src/agent, src/servers/fmsr and
src/assetops_harbor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
generate_failure_modes documents the behaviour it did not have:
If no stored modes exist, the LLM generates a new list from scratch.
_known_failure_modes raises when there is no catalog to read, and that
propagated out as an ErrorResult, so the documented from-scratch path was
unreachable. On the mini suite this bit hardest where it mattered: the scenarios
whose manifests never load failure_mode are exactly the ones where models
reached for generate_* after get_failure_modes returned 'database not
connected', and the generate call failed for the same reason.
The stored catalog is now optional context. _optional_known_failure_modes
returns ([], note) when there is no catalog to read and propagates everything
else, so the four states separate:
CouchDB unreachable -> generates, 'catalog unreachable'
fm_db unset -> generates, 'catalog not initialised'
server up, database never created -> generates, 'catalog not initialised'
catalog exists but the read fails -> ErrorResult
The last one stays an error deliberately. Stored modes may exist and be
temporarily unreadable, and generating a list that quietly ignores them would
hide a real fault behind plausible output. TestGenerateFailureModes::
test_database_read_error_returns_error already asserted this and still passes;
it is the reason the fix distinguishes absent from unreadable rather than
treating every database failure as empty context. The result message names
which state applied, so a run can be read back later.
get_failure_modes and add_failure_modes are untouched: a read tool with no data
has nothing to return, and a write needs somewhere to write.
Also strips markdown emphasis from generated mode names. kimi-k3 answers
'**Cavitation**' and the parser stripped one leading asterisk, producing
'*Cavitation**' as the mode name. Emphasis is removed before bullets, because
'*Cavitation*' is italic rather than a bullet plus a stray asterisk, and again
after, for '- **Bearing wear**'. Nobody had seen this because the tool never ran.
252 tests pass, 2 new.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
fix(fmsr): take the FMSR model from FMSR_MODEL_ID, with no built-in default
ShuxinLin
changed the base branch from
feat/harbor-private-mount
to
feature/harbor-integration
September 30, 2026 15:56
ShuxinLin
approved these changes
Sep 30, 2026
ShuxinLin
force-pushed
the
harbor-private-mount-docker-md
branch
from
September 30, 2026 16:17
cbc26f6 to
b74b621
Compare
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.