You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
study_area_run.sh counts finished groups from log text and burns hosts holding unconsolidated work #347
study_area_run.sh decides which groups each cloud host finished by reading the host's run log, and the reading is wrong often enough to lose work.
bucket_done() matches ^\[wsg_run_one\] <WSG> … done, anchored at the start of a line. R writes warnings to stderr in the same stream, and a warning flushed without a newline puts the marker mid-line:
8: In st_read.DBIObject(conn, query = query) :[wsg_run_one] FIRE done in 4.6 min (persist=fresh_default)
In run 20261017_004619-3f4c713b (#246, 241 groups, default) that hid 18 finished groups: 2 of 50 on job1, 12 of 55 on job2, 4 of 60 on job3. Each was persisted on its host with one log row for the run_uid (checked on the hosts before burn).
The driver then does what its design says with a short count:
consolidates only the groups it counted;
burns the cloud hosts right after consolidate, destroying the 18 groups' only copy;
fails the coverage check on whatever is missing and exits before the recompute.
They were saved only because the operator paused the driver (kill -STOP) before the burn and the 18 were consolidated by hand.
Proposal
Count completion from the database, not the log. For each host, read <schema>.log for this run_uid (one row per finished group, with date_end set) over the same ssh the consolidate uses. Corrected while building: the column is date_end; date_completed does not exist. As built, the log text still counts, read anywhere on a line, but it only adds to the database set, never removes from it: .lnk_log_run_finish() is a soft UPDATE, so a persisted group can lack date_end. SKIP groups come from the log alone, because they write no row.
Never burn a host whose work is not consolidated. As built, this is wider than "finished set larger than what consolidate took". A host is held from dispatch, on every exit path including abort. It is released only when its session ended 0, consolidate is not running, and its database holds no finished group after consolidate. A kept host is listed in <TS>_kept_hosts.tsv and the run exits non-zero. Pre-flight fails on a droplet left up in this run's workspaces. RUNBOOK §8d, "Which host may be burned", has the rules.
Quieter logs:configs/default/overrides/cabd_blkey_xref.csv has no final newline (as built: fixed in all four bundles, which moves each bundle's config_hash; operator call, provenance not a gate), so every group's log carries an incomplete final line warning. It is noise, not the cause.
Acceptance
A host log with every done marker mid-line still reports the host complete (fixture).
A host with an unconsolidated finished group is not burned (mocked cypher_down.sh).
Activity
Logs of the 241-group default run and the four-group parity run (#246)
Merge pull request #352 from NewGraphEnvironment/347-study-area-run-s…
Fold in the plan review: name re-thrown timeouts, check each group's …