Skip to content

study_area_run.sh: a run that stops before its tail leaves no record of which groups it was running #358

Closed

Description

@NewGraphEnvironment

Problem

#348 lets a run that loses groups still recompute and compare the rest. It also adds --recompute-only --run-uid=<uid> to finish a run once the missing groups have been re-modelled. Two gaps remain.

  • The expected set has no record of its own. Without --wsgs=, --recompute-only expects the groups that have a <schema>.log row for the run_uid on the dispatcher. Some groups have no such row:

    • a group that failed on a cloud host, since consolidate copies log rows only for groups that finished;
    • a group that failed before .lnk_log_run_start(), for example in lnk_load_overrides(), at the downstream guard's quit(1), or on a connect failure (wsg_run_one.R).

    The driver knows the full set once the buckets resolve, but it writes that set nowhere.

  • A consolidate failure still exits before coverage. If one source is refused (dest_log_newer, which a single group can trigger), the driver exits after consolidate, before the coverage step and the recompute. That is the same failure class as One failed group skips the post-consolidate recompute for every group #348. Since One failed group skips the post-consolidate recompute for every group #348 the banner prints the --recompute-only recovery, but the groups that were consolidated are not recomputed in that run.

Proposal

  • Write <TS>_run_wsgs.tsv (group, host, SKIP) as soon as the buckets resolve. --recompute-only reads it by default (<run_uid minus its suffix>_run_wsgs.tsv, since a run_uid's prefix is its TS) and falls back to the log.
  • After a consolidate refusal, carry on to coverage with the sources that consolidated, keep the refused host, and exit non-zero as the kept-host path already does.

Acceptance

  • --recompute-only with no --wsgs= names a group that failed on a cloud host.
  • A run with one refused source recomputes the other hosts' groups and exits non-zero naming the refused host.

As built (PR #359)

  • <TS>_run_wsgs.tsv is written before any spend and rewritten after per-host accounting with each host's SKIPs. --recompute-only reads the file together with the log, not instead of it.
  • A refused or failed source no longer ends the run when every source has a recorded outcome (<TS>_consolidate.tsv). Its groups get the new coverage class not_consolidated and are kept out of the recompute.
  • The refused host is kept only while finished groups remain on it. A host whose database holds nothing to lose is released and burned, as study_area_run.sh counts finished groups from log text and burns hosts holding unconsolidated work #347's rule says. The run still exits non-zero.
  • A crash mid-copy is covered too. <TS>_consolidate_state.tsv is written in_flight before the copy, and a later bare --recompute-only keeps any group not ok out of the recompute until --wsgs= names it.

Activity

  1. added a commit that references this issue on Oct 19, 2026
  2. added a commit that references this issue on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions