[NewGraphEnvironment]

Summary

• Each host's finished groups now come from its database, not its log text. study_area_run.sh used to count [wsg_run_one] <WSG> done lines anchored at line start. In run 20261017_004619-3f4c713b an R warning flushed without a newline put the marker mid-line on 18 finished groups, so the driver left them out of consolidate and would have burned the only copy (study_area_run.sh counts finished groups from log text and burns hosts holding unconsolidated work #347).
  • Finished = <schema>.log rows for the run_uid with date_end set, read over the route schema_consolidate.R uses (ssh → docker exec -i fresh-db psql). Output is framed by sentinels: no sentinels means unknown, never "nothing", because ssh can exit 0 whatever the remote command did.
  • The log is still read, anywhere on a line, and only adds: consolidate takes database ∪ log done, because .lnk_log_run_finish() is a soft UPDATE after persist and a lost LNK_RUN_UID leaves run_uid NULL. SKIP comes from the log alone; it writes no row.
• A cloud host that may hold unconsolidated work is never burned, on any exit path.
  • Each cypher is held from dispatch. It is released only when it is settled (its dispatch session ended 0, recorded only after wait returns, and consolidate is not running) and, after consolidate, its database holds no group this run finished. consolidate deletes what it copied from the source, so this measures what was not consolidated.
  • What is left on a kept host is split: finished (database row, log done, or streams_mapping_code rows, which within a run only the final persist writes), partial (a group that failed after the pre-persist: never consolidated, does not hold the host), and unknown (rows that were there before dispatch, per a pre-dispatch baseline, or mapping-code rows with no baseline: they hold the host and are never recovered). On an unsettled host nothing is called partial.
  • A host whose database did not answer is left out of consolidate (it would sink the others) and kept.
  • Kept hosts, and hosts whose burn did not complete, are listed in <TS>_kept_hosts.tsv: reason, run_uid, schema, the three group sets, whether they were measured, and the commands that print the address and burn the host. The banner prints from the EXIT trap and says whether the recompute ran, and the run exits non-zero. --keep-cyphers alone is not a failure; a host it keeps for cause still is.
  • Pre-flight fails on a cypher droplet already up in this run's workspaces, which is most likely a host an earlier run kept.
  • CYPHERS_UP is set before the spin, so a failed spin of one workspace no longer exits with the others' droplets unburned.
• The incomplete final line warning is gone from every group's log: cabd_blkey_xref.csv ends with a newline in all four bundles. Each bundle's config_hash moves (provenance, by operator call).
• RUNBOOK §8d "Which host may be burned" carries the rules, including how to recover a kept host; the CLAUDE.md status fact is updated.

Evidence

• tests/testthat/test-study_area_hosts.R: 120 expectations, run in bash ≥ 4 with stubbed ssh / psql / tofu / doctl and a mock cypher_down.sh. They include:

  • a host log with every done marker mid-line reports complete;
  • a host with a finished group left after consolidate is not burned, and a failed group's partial rows neither hold it nor get recovered;
  • a driver killed while waiting on a host, or during consolidate, reports the host unsettled;
  • an abort before dispatch burns everything, as before.

  They skip when data-raw/ is absent (R CMD check), and fail rather than skip without bash ≥ 4.

• 26 mutants of the lib, each caught. The issue's "restore the anchored match", read as counting from the log with main's sed, fails 9 tests, including the mid-line fixture.

• Live, local fwapg:

  • the finished probe returns 241 for run 20261017_004619-3f4c713b;
  • the remote half of the route returns both sentinels and 241 rows;
  • the baseline and mapping-code probes answer, and a missing table reads as unknown;
  • a --preflight-only run sources the lib and passes the doctl and droplet legs.

  The ssh leg to a live cypher was not run (it needs a paid spin).

• devtools::test(): 3058 pass, 5 fail, none from this branch. Three need the :63333 tunnel, which was down; two are the golden dams count, which fails the same on clean main.

• Review: a plan review and five /code-check rounds, 13 findings fixed. Every round from 2 on found a defect inside the previous fix, all one mechanism: a label on observed rows needs provenance and settledness. The loop ended by enumerating both (the two writers to a cypher after dispatch, the two sources of its rows, every printed label), not on a quiet round. Records are in planning/archive/2026-10-issue-347-host-completeness-db/.

Fixes #347.

Relates to NewGraphEnvironment/sred#24

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx