feat(harbor): run mini/lite/all by mounting the private suite

[ShuxinLin]

Description

Makes the mini, lite and all profiles run correctly on #558. Today their tasks generate, but every trial loads empty CouchDB collections, because the private suite's shared/ data never reaches the container.

This is the part of the closed #560 that mini needs, rebased onto feature/harbor-integration. It leaves out run.sh, the suite image and --runtime-image.

What was wrong

Each scenario's manifest.json names its data as shared/... paths, and init_data.py resolves them against the parent of the scenario folder. The generator copied only scenario_<id>/ into the task image, so the healthcheck resolved shared/ against the repo's open subset.

All 35 mini scenarios reference files that exist only in the private suite. For example, FMEA needs shared/catalog/assets_fmea.csv, and the repo has only assets.csv.

The loader skips a missing file with a warning, so the healthcheck passed and the agent ran against empty databases. On #558, three FMEA trials with gpt-5.6-sol got Database does not exist from the catalog tools and scored 0.0. The oracle cannot see this, because it never touches CouchDB.

Changes

• overlays/private-data.yaml (new): a read-only bind mount of AOB_PRIVATE_DIR at /opt/suite/scenarios_data.
  • It uses ${AOB_PRIVATE_DIR:?...}, so an unset variable fails with a message naming it, rather than as invalid spec: :/opt/suite/...: empty section between colons.
  • It mounts beside the repo copy, not over it, because the suite's shared/ is not a superset of the repo's. It lacks work_order/workorders.csv and others that scenarios 1–3 need, and its catalog/*.csv differ.
• adapter/generate_tasks.py:

  • Data folder from --scenario-root. For any root other than the repo's own src/couchdb/scenarios_data:

    • it copies the scenario to /opt/suite/scenarios_data/;
    • the healthcheck runs SCENARIOS_DATA_DIR=/opt/suite/scenarios_data init_data.py <id>;
    • the agent's own SCENARIOS_DATA_DIR stays on the repo copy.

    With no root, or the repo's root named explicitly, the output is the same as before apart from the fix below.

  • --skip-missing: warn about and skip profile ids that the suite lacks, such as wosr-62 in all.yaml. Without it, generation stops at the first missing folder.

  • Fix: every task inherited scenario 1's description and the wosr keyword.

• README:
  • "Other profiles" now uses the mount;
  • the "Known gap" section becomes "How the private data gets in" and "Without the mount, data loads silently empty";
  • adds open/mini smoke-test commands;
  • the code-track section replaces the out-of-date docker push assetopsbench/code:dev step with the local-tar flow the sandbox overlay uses (docker save, then AOB_CODE_TAR).

How to test

export AOB_PRIVATE_DIR=/path/to/AssetOpsBenchScenarioGeneration/scenarios_data

uv run python benchmarks/harbor/adapter/generate_tasks.py \
  --scenario-root "$AOB_PRIVATE_DIR" \
  --profile benchmarks/scenario_suite/mini.yaml \
  --output-dir benchmarks/harbor/datasets/assetopsbench-mini \
  --dataset-name assetopsbench/mini \
  --overwrite && \
uv run harbor run -y \
  -p benchmarks/harbor/datasets/assetopsbench-mini \
  -i 'fmsr-*' \
  --extra-docker-compose benchmarks/harbor/overlays/private-data.yaml \
  --agent assetops_harbor.stirrup:StirrupAgent \
  --model litellm_proxy/azure/gpt-5.6-sol \
  --ak code_enabled=true --ak code_backend=local \
  --n-concurrent 3

The data loaded if no trial's agent/*.stdout.txt contains Database does not exist. The README has the Docker-sandbox variant.

Type of Change

• New Benchmark Scenario (Industry/Asset type)
• Evaluation Metric / Scorer
• Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
• Infrastructure / Tooling Improvement

Industry Relevance

None directly. It lets the published mini, lite and all profiles run under Harbor's per-trial isolation.

Related Issues

• Refs: feat(harbor): run AssetOpsBench scenarios as Harbor tasks #558, feat(harbor): run a full scenario suite in parallel #560 (closed; this carries its mini-relevant part)

Testing & Validation

Generator:

Case	Result
no --scenario-root	healthcheck init_data.py 1, copy to /opt/aob/src/couchdb/scenarios_data/
--scenario-root src/couchdb/scenarios_data	identical to the row above
external root, mini	healthcheck SCENARIOS_DATA_DIR=/opt/suite/scenarios_data init_data.py 913, copy to /opt/suite/scenarios_data/
same, compared with the mini tasks the FMSR run below used	identical, all 35 tasks and the dataset files
all.yaml without / with --skip-missing	scenario folder not found / 231 tasks, wosr-62 skipped
--runtime-image, --data-dir	rejected as unrecognized arguments

ruff check and ruff format --check pass on generate_tasks.py. uv run pytest src/assetops_harbor gives 19 passed.

Overlay: docker compose config, with Harbor's docker-compose-build.yaml, the task compose file and the overlay:

• with the variable set, it renders a read-only bind at /opt/suite/scenarios_data;
• with it unset, it fails with required variable AOB_PRIVATE_DIR is missing a value.

Agent runs through the mount:

• fmsr-913 on this branch, local code backend: the healthcheck ran SCENARIOS_DATA_DIR=/opt/suite/scenarios_data init_data.py 913 and passed. The agent made 10 catalog and FMSR tool calls with no errors, and no Database does not exist appeared.
  • It scored 0.0 by answering yes against a ground truth of no.
  • The same task scored 1.0 with no in the run below.
  • gpt-5.6-sol accepts only its default temperature, so its answers vary between runs.
• Mini's five FMSR tasks, the same generator output via feat(harbor): run a full scenario suite in parallel #560's branch, Docker code sandbox: mean reward 0.760, 3/5 passed, 0 exceptions. Data loaded in every trial. The two misses were the model's answers.

Checklist items:

• Scenario Validation: see the agent runs above.
• Data Integrity: the overlay mounts private data at run time only; nothing from the suite is committed, and generated tasks stay under the gitignored datasets/.

Checklist

• My code follows the project's Ruff formatting and linting rules.
• I have performed a self-review of my code.
• I have updated the documentation (README or /docs) accordingly.
• I have signed off my commits (DCO).