Repository navigation
study_area_run.sh: record a run's groups, and carry on past a refused consolidate source (#358) - #359
Merged
NewGraphEnvironment merged 9 commits intoOct 19, 2026
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…#358) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…that host (#358) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…-only keeps them out (#358) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…ash fails closed (#358) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…e, not re-modelled (#358) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
NewGraphEnvironment
deleted the
358-study-area-run-sh-a-run-that-stops-befor
branch
October 19, 2026 15:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #358.
What changes
The run records its groups before any spend.
study_area_run.shwrites<TS>_run_wsgs.tsv(
wsg,host,skip,run_uid) as soon as the buckets resolve, and rewrites it after the per-hostaccounting with each host's SKIPs.
--recompute-only --run-uid=<uid>reads<run_uid minus its suffix>_run_wsgs.tsvtogether with every group holding a log row for therun_uid, so a group that failed on a cloud host, or before its first log row, is named. A run from
before this change has no file and falls back to the log alone;
--wsgs=still wins.A refused consolidate source no longer stops the run.
schema_consolidate()already loops pasta source it refuses or fails on. The driver now writes one outcome per source to
<TS>_consolidate.tsvand, when every source has one, carries on to coverage, the recompute andthe compare:
not_consolidated,kept out of the recompute whatever the dispatcher's log says (a failure after the DELETE can leave
log rows copied and
streamsnot);host_releasekeeps the host while its finished groups are left on it, withconsolidate failed at stage <stage>as the reason inkept_hosts.tsv;When an outcome is missing for any source (R died), the old exit after consolidate stays.
A crash mid-copy still leaves a record.
<TS>_consolidate_state.tsvlists every source's groupsin_flightbefore the copy andok/failed:<stage>after. A bare--recompute-onlynames everygroup not
okand keeps it out of the recompute until--wsgs=names it.Acceptance
--recompute-onlywith no--wsgs=names a group that failed on a cloud host: tested on ascratch schema (GGGG, run on
job1, no log row, named under "no rows").refused host: tested at the lib level (
consolidate_outcomes,run_finishwith the excluded set,host_release). The driver's full-run path needs cloud hosts; a structural test pins its wiring,and the driver's R writer is run against the reader.
Verification
test-study_area_finish.Randtest-study_area_hosts.R: all pass, none skipped, local dockerfwapg up. Full suite: 3293 pass, 3 fail, 0 skip; the three failures predate this branch and sit in files it does not touch (two
lnk_wsg_resolvetests need the:63333tunnel, which was down; the golden ADMS dams pin expects 8 wherecabd.damsnow gives 7).red (the build's six in the archive's
findings.md, the review rounds' eight in itsprogress.md)./code-check branch: three rounds, five fragile findings, all fixed, none open. Rounds 2 and 3found defects inside the previous fix; round 3's enumeration of every set combination ended it.
The shared mechanism: "which groups may not be safely on the dispatcher" was derived separately at
each consumer from records covering different populations or times. Fixes from review: a failed
host's finished groups stay expected;
<TS>_consolidate_state.tsvis writtenin_flightbeforethe copy so a crash fails closed; a bare
--recompute-onlykeeps groups not consolidatedokoutof the recompute until
--wsgs=names them; the report's kept-host set uses the same unionkept_hosts.tsvchecks.RUNBOOK §8d ("A missing group costs only itself") is updated.
Relates to NewGraphEnvironment/sred#24
🤖 Generated with Claude Code
https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx