Skip to content

Cheap access-only post-consolidate recompute (bulletproof cross-WSG mapping_code parity, efficiently) #205

Closed

Description

@NewGraphEnvironment

Problem

The #175 tunnel-free per-segment mapping_code parity needs a post-consolidate recompute: a WSG's accessibility (hence its mapping_code token1/token2) depends on whether a blocking barrier exists downstream — possibly in a different WSG (the provincial-accumulation property, RUNBOOK.md §5). When WSGs are distributed across machines, each machine only holds its own bucket's barriers while it runs, so a WSG's access is computed against an incomplete barrier set.

Caught 2026-05-25 (study-area run): FINA 75.5% / PARA 68.6% per-host → both 99%+ after re-modelling on the full consolidated barrier set. Drainage-closed + DS-first bucketing reduces but does not eliminate this (downstream barriers can be cross-bucket or arrive late in DS-first order). So the correct methodology is: distribute (any bucketing) → consolidate → recompute → compare.

The orchestrator has no recompute today because the provincial run compares habitat-km rollups, not per-segment mapping_code — this is new to #175.

DELIVERED 2026-05-26 — see the close-out comment

This issue's ask shipped in #206
(commit 0a332cc, link v0.36.0) and the section below is stale.
It is preserved
because it states the problem correctly; only "current state" is out of date.

lnk_access(merge = TRUE) is the cheap path, data-raw/wsg_recompute_one.R is the
driver, and study_area_run.sh has called it on every run since. The recompute has
not used the full pipeline since 2026-05-28.

The issue stayed open for three months and was repeatedly cited as pending
optimisation work that had in fact already been done. The remaining recompute cost
is not the full-vs-cheap distinction described below — it is that the recompute loop
is serial, which is tracked separately.

Current state AS FILED 2026-05-25 (superseded, kept for the record)

The driver recomputes diverged WSGs via the full pipeline (wsg_run_one.R = lnk_pipeline_run(mapping_code=TRUE)). That works but is slow and architecturally wasteful: re-running the full pipeline on the dispatcher re-derives streams + streams_habitat (which are already correct + persisted) just to redo streams_access + streams_mapping_code. Recompute-ALL via full pipeline would defeat distribution (dispatcher redoes everything ≈ running it all single-host twice).

Ask: a cheap access-only recompute

Add a recompute path that redoes only lnk_pipeline_access + lnk_mapping_code for a WSG, reusing the consolidated persisted streams, streams_habitat_<sp>, and barriers — no full working-schema rebuild. Then post-consolidate recompute-ALL becomes cheap (~seconds/WSG) → bulletproof correctness regardless of machine count or WSG bucketing, efficiently. That is the durable methodology #175 is after.

Blocker as filed (RESOLVED — lnk_access() reads its inputs from persist): lnk_pipeline_access currently needs observations + crossings working-schema artifacts (not persisted). Options:

lnk_mapping_code() (portable, schema-aware) already recomputes mapping_code from persist tables — the missing half is the persist-based access recompute.

Refs

Activity

  1. added 6 commits that reference this issue on May 25, 2026
  2. added a commit that references this issue on Jul 4, 2026
  3. NewGraphEnvironment commented on Aug 31, 2026

    @NewGraphEnvironment
    OwnerAuthor

    Closing as delivered — three months late, which is itself the point of this comment.

    What shipped, and when

    The ask was a recompute that redoes only access + mapping_code from the persisted
    streams / streams_habitat_<sp> / barriers, with no working-schema rebuild.
    That landed in #206 (0a332cc, 2026-05-26, link v0.36.0):

    • lnk_access() — exported, with merge = TRUE as the surgical cross-WSG update
    • data-raw/wsg_recompute_one.R — the driver, reading persist only
      (<persist>.streams, .barriers, .streams_habitat_long_vw)
    • study_area_run.sh has called it on every run since 2026-05-28

    The blocker in the body — that lnk_pipeline_access needed working-schema
    observations / crossings — was resolved by lnk_access() taking explicit
    table_* arguments and reading from persist.

    NEWS records ~8× on the full-pipeline path: FINA 11.9 s wall vs ~90 s, identical
    bcfp parity.

    Why it stayed open, and what that cost

    Nothing closed it, so it kept reading as pending optimisation. As recently as
    2026-08-31 it was being cited — in planning conversation, more than once — as "the
    obvious next lever" for a recompute that had already been optimised. The stale
    ## Current state section actively supported that misreading, because it described
    the full-pipeline recompute in the present tense.

    Caught only because someone asked "what is 205?" and the issue got read rather than
    recalled. Same failure the repo already has a convention for — an issue body is a
    spec, and a spec that outlives its own delivery is worse than no spec, because it
    carries authority.

    What is actually left

    Not this. The recompute is cheap per WSG and serial across WSGs — a
    single-threaded loop over $ALL_WSGS on a 10-core host. Estimated ~84 min of a
    ~2.6 h field-scope run, and materially more provincially. Filed separately as #250
    so it is scoped as what it is — parallelism in the orchestrator — rather than
    inherited by a closed issue about a code path that already exists.

    Version pin: delivered link@v0.36.0; verified against link@v0.47.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions