Repository navigation
fix(harbor): run.sh mounts only the private suite's shared/ - #586
Merged
ShuxinLin merged 2 commits intoSep 30, 2026
Merged
Conversation
run.sh passed --runtime-image to generate_tasks.py, which the merge of the private-mount branch (b74b621) dropped, so the script exited on an argparse error before any job ran. Rather than restore the flag and the suite image it fed, run.sh now uses overlays/private-data.yaml: it exports AOB_PRIVATE_DIR as the -s directory, adds the overlay to each job, and refuses a directory with no shared/, which Docker would otherwise create empty and load silently. The overlay now mounts $AOB_PRIVATE_DIR/shared alone. Mounting the whole suite put every scenario_<id>/ folder, answers included, into `main`, the agent's container. The generator also leaves the verifier's inputs (groundtruth.txt, rubric.json, ...) out of each task's build context, which the full mount used to shadow; the task layer keeps only what init_data.py reads. The verifier still gets them via tests/. Delete suite-image/Dockerfile and update the README to match. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
3 of 5 tasks
Review of #586: - generate_tasks.py copies into each task only manifest.json and the files it names inside the scenario folder. Skipping the verifier's input names let any other answer file through (a .bak, a new scorer input, GroundTruth.txt on a case-insensitive disk). - overlays/private-data.yaml uses the long bind syntax with create_host_path: false. A mistyped or relative AOB_PRIVATE_DIR used to make Docker create an empty shared/ and load empty collections with the healthcheck passing; it now fails the trial at start with "bind source path does not exist". A path with ':' also works now. - run.sh reports a failed resume instead of hiding it behind `|| true`. Harbor refuses to resume a job whose tasks or overlays no longer match its lock.json, which is every job from the old suite-image run.sh. - The overlay no longer claims suite edits need no rebuild: a changed manifest.json needs the tasks regenerated. The README asks for an absolute AOB_PRIVATE_DIR. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Collaborator
Author
|
Second review pass, addressed in 7f66b87:
Verification:
Not changed (follow-ups):
|
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
benchmarks/harbor/run.shpassed--runtime-imagetogenerate_tasks.py, but the merge of the private-mount branch (b74b621, #581) dropped that option. The script therefore stopped with an argparse error before any Harbor job ran. This was the top finding in the review of #558.Instead of restoring the flag and the suite image it fed,
run.shnow uses the mount. The overlay mounts only the suite'sshared/folder, so the agent's container gets no scenario answers.Fix Details
run.shAOB_PRIVATE_DIR="$scenario_dir"and addsoverlays/private-data.yamlto eachharbor run. Resume picks up the same variable.--runtime-image.-shas noshared/. Without that check, Docker would create an empty directory for the mount and every collection would load empty with no error.overlays/private-data.yamlmounts$AOB_PRIVATE_DIR/sharedalone, read-only. It used to mount the whole suite, which put all 410scenario_<id>/folders, answers included, intomain, the container the agent runs in.generate_tasks.pyleaves the verifier's inputs (groundtruth.txt,groundtruth_eval.json,scenario_meta.json,rubric.json,reference_answer.json,question.txt) out of each task's copied scenario folder. The full mount used to hide that copy; with the narrower mount it would be visible. The verifier still gets these files throughtests/, which Harbor copies in only after the agent phase.suite-image/Dockerfileand updates the README and the template README.Impact on Benchmarking
Related Issues
Verification Steps
uv run pytest: not run. This change touches no Python undersrc/, and the generator has no tests.environment/scenario_*/holds onlymanifest.json, andtests/scenarios/still has all the answer files.docker compose configwith the overlay renders…/scenarios_data/shared → /opt/suite/scenarios_data/shared(read-only). LeavingAOB_PRIVATE_DIRunset still fails with the named-variable message.fmea-9001andtsfm-1019: the per-task image plus the CouchDB sidecar and the overlay./opt/suite/scenarios_datacontains onlyscenario_<id>/(justmanifest.json) andshared/./opt/suite.shared/is read-only.init_data.py 1019loaded all 5 collections with data (iot 1201, asset 25, failure_code 13, model_catalog 59, feature_catalog 234).bash -n run.shpasses. I did not runrun.shend to end or test itsshared/check.Checklist