Skip to content

fix(harbor): run.sh mounts only the private suite's shared/ - #586

Merged
ShuxinLin merged 2 commits into
feature/harbor-integrationfrom
fix/harbor-run-private-mount
Sep 30, 2026
Merged

ShuxinLin merged 2 commits into
feature/harbor-integrationfrom
fix/harbor-run-private-mount

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Description

benchmarks/harbor/run.sh passed --runtime-image to generate_tasks.py, but the merge of the private-mount branch (b74b621, #581) dropped that option. The script therefore stopped with an argparse error before any Harbor job ran. This was the top finding in the review of #558.

Instead of restoring the flag and the suite image it fed, run.sh now uses the mount. The overlay mounts only the suite's shared/ folder, so the agent's container gets no scenario answers.

Fix Details

  • run.sh
    • Exports AOB_PRIVATE_DIR="$scenario_dir" and adds overlays/private-data.yaml to each harbor run. Resume picks up the same variable.
    • Removes the suite-image build and --runtime-image.
    • Exits early if -s has no shared/. Without that check, Docker would create an empty directory for the mount and every collection would load empty with no error.
  • overlays/private-data.yaml mounts $AOB_PRIVATE_DIR/shared alone, read-only. It used to mount the whole suite, which put all 410 scenario_<id>/ folders, answers included, into main, the container the agent runs in.
  • generate_tasks.py leaves the verifier's inputs (groundtruth.txt, groundtruth_eval.json, scenario_meta.json, rubric.json, reference_answer.json, question.txt) out of each task's copied scenario folder. The full mount used to hide that copy; with the narrower mount it would be visible. The verifier still gets these files through tests/, which Harbor copies in only after the agent phase.
  • Deletes suite-image/Dockerfile and updates the README and the template README.

Impact on Benchmarking

  • No change to baselines: this changes how data reaches the container, not which data loads or how it is scored.
  • Baseline change

Related Issues

Verification Steps

  1. uv run pytest: not run. This change touches no Python under src/, and the generator has no tests.
  2. Manual verification:
    • Generated mini from the private suite (35 tasks). Every environment/scenario_*/ holds only manifest.json, and tests/scenarios/ still has all the answer files.
    • docker compose config with the overlay renders …/scenarios_data/shared → /opt/suite/scenarios_data/shared (read-only). Leaving AOB_PRIVATE_DIR unset still fails with the named-variable message.
    • Brought up the real task environments for fmea-9001 and tsfm-1019: the per-task image plus the CouchDB sidecar and the overlay.
      • /opt/suite/scenarios_data contains only scenario_<id>/ (just manifest.json) and shared/.
      • No answer files exist under /opt/suite.
      • shared/ is read-only.
      • init_data.py 1019 loaded all 5 collections with data (iot 1201, asset 25, failure_code 13, model_catalog 59, feature_catalog 234).
    • bash -n run.sh passes. I did not run run.sh end to end or test its shared/ check.

Checklist

  • I have added tests that prove my fix is effective.
  • My code follows the project's Ruff formatting and linting rules.
  • I have signed off my commits (DCO).

run.sh passed --runtime-image to generate_tasks.py, which the merge of
the private-mount branch (b74b621) dropped, so the script exited on an
argparse error before any job ran. Rather than restore the flag and the
suite image it fed, run.sh now uses overlays/private-data.yaml: it
exports AOB_PRIVATE_DIR as the -s directory, adds the overlay to each
job, and refuses a directory with no shared/, which Docker would
otherwise create empty and load silently.

The overlay now mounts $AOB_PRIVATE_DIR/shared alone. Mounting the whole
suite put every scenario_<id>/ folder, answers included, into `main`,
the agent's container. The generator also leaves the verifier's inputs
(groundtruth.txt, rubric.json, ...) out of each task's build context,
which the full mount used to shadow; the task layer keeps only what
init_data.py reads. The verifier still gets them via tests/.

Delete suite-image/Dockerfile and update the README to match.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Review of #586:
- generate_tasks.py copies into each task only manifest.json and the
  files it names inside the scenario folder. Skipping the verifier's
  input names let any other answer file through (a .bak, a new scorer
  input, GroundTruth.txt on a case-insensitive disk).
- overlays/private-data.yaml uses the long bind syntax with
  create_host_path: false. A mistyped or relative AOB_PRIVATE_DIR used
  to make Docker create an empty shared/ and load empty collections
  with the healthcheck passing; it now fails the trial at start with
  "bind source path does not exist". A path with ':' also works now.
- run.sh reports a failed resume instead of hiding it behind `|| true`.
  Harbor refuses to resume a job whose tasks or overlays no longer
  match its lock.json, which is every job from the old suite-image
  run.sh.
- The overlay no longer claims suite edits need no rebuild: a changed
  manifest.json needs the tasks regenerated. The README asks for an
  absolute AOB_PRIVATE_DIR.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin

Copy link
Copy Markdown
Collaborator Author

Second review pass, addressed in 7f66b87:

  • The answer filter is now an allowlist. Each task gets only manifest.json plus files it names inside the scenario folder. The old approach skipped the verifier's input names, so a .bak, a new scorer input, or GroundTruth.txt on a case-insensitive disk would have shipped. Checked on all 231 all tasks: each holds only manifest.json.
  • The overlay fails on a bad path. It now uses the long bind syntax with create_host_path: false. A mistyped or relative AOB_PRIVATE_DIR now fails at startup with bind source path does not exist instead of loading empty collections. Tested: the absolute path loads all 5 collections for tsfm-1019; the mistyped and relative paths fail and create no host folder.
  • run.sh reports a failed resume. Harbor refuses to resume jobs whose lock no longer matches (harbor/job.py _write_job_lock), and this was hidden by || true.
  • Docs: the overlay comment and README updated (manifest edits need the tasks regenerated; AOB_PRIVATE_DIR must be absolute).

Verification:

  • Harbor oracle, 13 private tasks through the overlay: 0 exceptions, reward 1.000.
  • Unit tests: the same 19 failures as feature/harbor-integration, none new. They are in iot/test_tools, evaluation/test_static_json_scorer and observability/test_file_exporter.

Not changed (follow-ups):

  • loader.py still logs a missing manifest file as an empty collection instead of failing. The strict mount covers the Harbor path.
  • run.sh still requires shared/, which is consistent with the overlay now requiring it.

@ShuxinLin
ShuxinLin merged commit ec62c34 into feature/harbor-integration Sep 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant