Repository navigation
Cheap access-only post-consolidate recompute (bulletproof cross-WSG mapping_code parity, efficiently) #205
Description
Problem
The #175 tunnel-free per-segment mapping_code parity needs a post-consolidate recompute: a WSG's accessibility (hence its mapping_code token1/token2) depends on whether a blocking barrier exists downstream — possibly in a different WSG (the provincial-accumulation property, RUNBOOK.md §5). When WSGs are distributed across machines, each machine only holds its own bucket's barriers while it runs, so a WSG's access is computed against an incomplete barrier set.
Caught 2026-05-25 (study-area run): FINA 75.5% / PARA 68.6% per-host → both 99%+ after re-modelling on the full consolidated barrier set. Drainage-closed + DS-first bucketing reduces but does not eliminate this (downstream barriers can be cross-bucket or arrive late in DS-first order). So the correct methodology is: distribute (any bucketing) → consolidate → recompute → compare.
The orchestrator has no recompute today because the provincial run compares habitat-km rollups, not per-segment mapping_code — this is new to #175.
DELIVERED 2026-05-26 — see the close-out comment
This issue's ask shipped in #206
(commit 0a332cc, link v0.36.0) and the section below is stale. It is preserved
because it states the problem correctly; only "current state" is out of date.
lnk_access(merge = TRUE) is the cheap path, data-raw/wsg_recompute_one.R is the
driver, and study_area_run.sh has called it on every run since. The recompute has
not used the full pipeline since 2026-05-28.
The issue stayed open for three months and was repeatedly cited as pending
optimisation work that had in fact already been done. The remaining recompute cost
is not the full-vs-cheap distinction described below — it is that the recompute loop
is serial, which is tracked separately.
Current state AS FILED 2026-05-25 (superseded, kept for the record)
The driver recomputes diverged WSGs via the full pipeline (wsg_run_one.R = lnk_pipeline_run(mapping_code=TRUE)). That works but is slow and architecturally wasteful: re-running the full pipeline on the dispatcher re-derives streams + streams_habitat (which are already correct + persisted) just to redo streams_access + streams_mapping_code. Recompute-ALL via full pipeline would defeat distribution (dispatcher redoes everything ≈ running it all single-host twice).
Ask: a cheap access-only recompute
Add a recompute path that redoes only lnk_pipeline_access + lnk_mapping_code for a WSG, reusing the consolidated persisted streams, streams_habitat_<sp>, and barriers — no full working-schema rebuild. Then post-consolidate recompute-ALL becomes cheap (~seconds/WSG) → bulletproof correctness regardless of machine count or WSG bucketing, efficiently. That is the durable methodology #175 is after.
Blocker as filed (RESOLVED — lnk_access() reads its inputs from persist): lnk_pipeline_access currently needs observations + crossings working-schema artifacts (not persisted). Options:
- persist the access inputs (or the per-species
barriers_<sp>_accessviews are already persist-backed — audit what's actually missing), OR - a
lnk_pipeline_accessvariant that reads its inputs from persist, OR - pre-persist barriers only in pass 1 + a single full pass 2 with complete barriers (relates to lnk_persist_init: persist per-source dnstr flag columns in streams_access (mapping_code second-token NONE bug) #196 double-persist optimization).
lnk_mapping_code() (portable, schema-aware) already recomputes mapping_code from persist tables — the missing half is the persist-based access recompute.
Refs
- link#175 (study-area mapping_code parity;
research/study_area_run.md), lnk_persist_init: persist per-source dnstr flag columns in streams_access (mapping_code second-token NONE bug) #196 (double-persist / pre-persist barriers), Persist+consolidate must be host-/species-count-agnostic: persist_init blind to species-column-set drift #204 (persist shape-drift), RUNBOOK.md §5. - SRED: NewGraphEnvironment/sred-2025-2026#24.
Activity
study_area_run: post-consolidate recompute = cross-WSG correctness gu…
CLAUDE.md: refresh Status handoff to #175 (study-area parity, post-co…
PWF: #205 Phase 7 baseline — lnk_access cheap recompute (plan approved)
#205: cheap post-consolidate recompute — lnk_access + 4 supporting fixes
wsg_recompute_one: wrap mc DELETE+INSERT in dbWithTransaction (#205 d…
Study-area mapping_code parity (tunnel-free) + cheap post-consolidate…
#223 Phase 2+3: break streams at every gradient frontier (fix + valid…
NewGraphEnvironment commentedon Aug 31, 2026
Closing as delivered — three months late, which is itself the point of this comment.
What shipped, and when
The ask was a recompute that redoes only access + mapping_code from the persisted
streams/streams_habitat_<sp>/barriers, with no working-schema rebuild.That landed in #206 (
0a332cc, 2026-05-26, link v0.36.0):lnk_access()— exported, withmerge = TRUEas the surgical cross-WSG updatedata-raw/wsg_recompute_one.R— the driver, reading persist only(
<persist>.streams,.barriers,.streams_habitat_long_vw)study_area_run.shhas called it on every run since 2026-05-28The blocker in the body — that
lnk_pipeline_accessneeded working-schemaobservations/crossings— was resolved bylnk_access()taking explicittable_*arguments and reading from persist.NEWS records ~8× on the full-pipeline path: FINA 11.9 s wall vs ~90 s, identical
bcfp parity.
Why it stayed open, and what that cost
Nothing closed it, so it kept reading as pending optimisation. As recently as
2026-08-31 it was being cited — in planning conversation, more than once — as "the
obvious next lever" for a recompute that had already been optimised. The stale
## Current statesection actively supported that misreading, because it describedthe full-pipeline recompute in the present tense.
Caught only because someone asked "what is 205?" and the issue got read rather than
recalled. Same failure the repo already has a convention for — an issue body is a
spec, and a spec that outlives its own delivery is worse than no spec, because it
carries authority.
What is actually left
Not this. The recompute is cheap per WSG and serial across WSGs — a
single-threaded loop over
$ALL_WSGSon a 10-core host. Estimated ~84 min of a~2.6 h field-scope run, and materially more provincially. Filed separately as #250
so it is scoped as what it is — parallelism in the orchestrator — rather than
inherited by a closed issue about a code path that already exists.
Version pin: delivered link@v0.36.0; verified against link@v0.47.0.