Repository navigation
v0.40.4: per-species access reproduces bcfp barriers_<sp> so dam-downstream emits ;DAM (#200) - #202
Merged
NewGraphEnvironment merged 7 commits intoMay 23, 2026
Conversation
Make all per-species access inputs province-wide so the cross-WSG downstream walk in barriers_<sp>_access (Phase 2) is correct in every WSG, not just the run's own WSG. - lnk_barriers_unify: USER_DEFINITE barrier family (mirrors the FALLS branch — ltree/linear_feature_id/geom resolved via the FWA fwa_stream_networks_sp join; blocks_species = all; reads <schema>.barriers_definite, referencing only blue_line_key + downstream_route_measure so it is safe against the empty-fallback shape). - lnk_persist_init: cols_barrier_overrides + <persist>.barrier_overrides DDL (one vector drives DDL + INSERT, avoiding the v0.40.3 matched-pair drift) + indexes + drift-validate. - lnk_pipeline_persist: probe-gated DELETE-WHERE-WSG + INSERT of barrier_overrides, injecting the AOI as watershed_group_code (the working table lacks it). The mapping_code-phase pre-persist already calls this, so persist holds the WSG's overrides before views build. Tests 96 pass; DB-smoke validated DDL + USER_DEFINITE branch (empty + one-row). /code-check round 1 clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…efinite-exempt) Add a per-species access view in lnk_barriers_views that reproduces bcfp barriers_<sp>: natural barriers only (GRADIENT/FALLS/SUBSURFACE_FLOW — gradient-at-species-threshold is already encoded in blocks_species) MINUS the observation/habitat override (anti-join the province-wide barrier_overrides), UNION all USER_DEFINITE (override-exempt). Dams/PSCIS/ modelled are excluded — they annotate token2 via the per-source views, never block access. Same feature shape as _unified (barriers_<sp>_access_id + ltrees + geom) so frs_network_features walks it unchanged. Override table derived from the barriers source schema so persist-barriers pairs with the province-wide overrides (cross-WSG correct). Not yet consumed — Phase 3 repoints barriers_per_sp. Tests 30 pass; DB-smoke: barriers_bt_access = 904,262 natural rows, 0 non-natural (vs _unified 1,045,358). /code-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The mapping_code phase now drives `accessible` from barriers_<sp>_access (natural-only + override, definite-exempt) instead of barriers_<sp>_unified (all barriers incl dams). Dam-downstream-but-otherwise-accessible segments now stay accessible and emit `;DAM` via token2, matching bcfp. Comment block rewritten from KNOWN-DIVERGENT to describe the landed fix. One-line table-name swap; consumption + alias contract validated in Phase 2's code-check. Full suite 1193 pass (lone FAIL is the env-only db_conn test — needs the real db_newgraph tunnel; CI skips it). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DB validation surfaced a real bug: the barrier_overrides persist PK (blue_line_key, downstream_route_measure, species_code) collided when a second adjacent WSG (UPCE after PCEA) persisted — the SAME override position is computed by two WSG runs (boundary streams whose blue_line_key spans WSGs). Fix: add watershed_group_code to the PK (mirrors cols_barriers' (id_barrier, wsg) pattern). Per-WSG DELETE-WHERE-WSG + INSERT is now clean; the WSG-agnostic access anti-join makes the duplicate harmless. Validation (local docker fwapg, bcfp baseline v0.7.15-14-ge12c1a5): - PARS BT 98.95% per-segment match; ;DAM tokens now emit (SPAWN;DAM 5293 vs 5263 bcfp). Required PCEA+UPCE barriers persisted first so the cross-WSG downstream walk sees the Bennett/Peace Canyon dams — confirms the province-wide design. - LFRA (anadromous) bt 97.77%, co 97.90%; coho DAM tokens 4672 vs 4636. - Residual ~1-2% is token1 habitat-presence (ACCESS<->SPAWN/REAR, dimensions/rules) + minor token2 ordering — not the dam-access fix. Full details in planning/active/findings.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docs for the landed #200 fix: RUNBOOK §2a/§3/§5/§7 (accessibility now reads the natural-only barriers_<sp>_access view; override + user_definite persisted province-wide; provincial-accumulation note), NEWS + DESCRIPTION bump to 0.40.4. Validated against bcfishpass@v0.7.15: PARS BT 98.95%, LFRA BT 97.77% / CO 97.90% per-segment mapping_code match; dam-downstream segments now emit ;DAM. Residual ~1-2% is token1 habitat-presence (dimensions/rules), not the dam-access fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NewGraphEnvironment
deleted the
200-mapping-code-accessibility-reproduce-bcf
branch
May 23, 2026 23:42
NewGraphEnvironment
added a commit
that referenced
this pull request
May 26, 2026
… recompute (#175, #205) (#206) * Archive #200 PWF (v0.40.4 shipped via PR #202) * Initialize PWF baseline for #175 (tunnel-free compare + orchestrator) * Phase 1 (#175): tunnel-free lnk_compare_mapping_code + fix id_segment cartesian New export lnk_compare_mapping_code() — segment-level mapping_code parity that reads the bcfp reference from the LOCAL snapshot fresh.streams_vw_bcfp (no :63333 tunnel, no conn_ref by default). Diffs <persist>.streams_mapping_code vs the snapshot on (blue_line_key, downstream_route_measure) per WSG-active species. .lnk_compare_wsg_mapping_code_diff now delegates; shared merge/match in .lnk_mc_diff. Tunnel path kept (pass conn_ref) for back-compat. Caught + fixed a real id_segment-collision bug: id_segment is per-WSG (80,555 distinct / 1.5M persist rows), unique only on the PK (id_segment, watershed_group_code). Joining persist tables on id_segment alone is a ~22x cartesian. Fixed lnk_compare_rollup's 3 habitat joins to the full PK (PARS BT spawning_km 36,820 -> 1,681). Added WSG-active species resolution so absent species (link "" vs bcfp NULL) don't register a spurious 0% match. Root fix (globally-unique position-derived id_segment, bcfp-style) filed as #203. Live PARS BT 98.95% reproduced tunnel-free; 1216 tests pass (lone fail is the env-only db_conn tunnel test). /code-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Phase 2 (#175): route mapping_code compare tunnel-free in the compare family lnk_compare_wsg(mapping_code = TRUE) now diffs against the LOCAL bcfp snapshot (fresh.streams_vw_bcfp) via lnk_compare_mapping_code with no conn_ref — the mapping_code lens is tunnel-free. conn_ref is still required for the rollup (lnk_compare_rollup needs bcfp habitat_linear, not in the snapshot). Species now auto-resolve to the WSG-active set rather than a hardcoded 8. Removed the now-dead .lnk_compare_wsg_mapping_code_diff helper (merge/match lives in .lnk_mc_diff); fixed the lnk_mapping_code doc ref. data-raw/wsg_compare.R: added wsg_compare_mapping_code() — tunnel-free (local conn only, no PG_PASS_SHARE/:63333). This is the per-segment mapping_code compare the orchestrator will run on the dispatcher after consolidate (cyphers just run + persist). Verified live: PARS BT 98.95% with PG_PASS_SHARE unset. Composition test repointed to mock lnk_compare_mapping_code. 93 compare / 1216 total tests pass (lone fail = env-only db_conn). /code-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Shape-tolerant consolidate + cypher_prep species alignment (#204, #175) Phase 3a: make persist/consolidate host- and species-count-agnostic so the 3-WSG smoke's cross-host wide-table drift can't recur. - schema_consolidate.R: COPY by shared column name (runtime intersection, dest ordinal order) instead of positional SELECT-star / FROM STDIN. Handles hosts whose streams_access/streams_mapping_code carry different species column sets. Nothing hardcoded (cols/species/host discovered at runtime). - cypher_prep.sh: seed lnk_persist_init from cfg$species (mirrors lnk_pipeline_run.R:157), not parameters_fresh (11 sp incl CT/DV/RB) - removes the drift at source. Surfaced by the 3-WSG smoke (CRKD/LCHL/ZYMO): cyphers' wide tables had 30 cols vs M1's 24 -> positional COPY failed. Filed #204 for the deeper class (persist_init blind to species-column-set drift). /code-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Lean study-area runner: drainage-closed DS-first, tunnel-free, M1-dispatch (#175) Phase 3 REVISED: instead of refactoring the 1594-line M4-centric orchestrator (per user "are these already dealt with in our start-to-finish scripts?"), productionize the proven smoke flow into 4 small reusable scripts that reuse every existing piece (cypher_up/prep/down, schema_consolidate, lnk_pipeline_run, wsg_compare_mapping_code). Cross-WSG ;DAM solved WITHOUT a post-consolidate recompute: each host gets a drainage-CLOSED bucket (focal + downstream closure via public.wsg_outlet) run DOWNSTREAM-FIRST, so a WSG's downstream dam barriers are persisted before its access/mapping_code is computed. Validated on PARS (depth 3) -> its Bennett-dam WSGs PCEA/UPCE/LPCE (depth 2) come first. One study area per host; areas are drainage-independent (roots 100/200/400). New: - study_area_wsgs.R closure + DS-first list (public.wsg_outlet) - wsg_run_one.R lnk_pipeline_run(mapping_code=TRUE) for one WSG, local, host-agnostic (LNK_LOAD=loadall dispatcher / library cyphers) - study_area_compare.R tunnel-free wsg_compare_mapping_code loop -> CSV - study_area_run.sh driver: tunnel-free pre-flight -> spin -> prep -> run DS-first buckets (dispatcher + cyphers) -> consolidate -> BURN (minimise idle) -> compare -> CSV; trap-EXIT burn No M4, no ssh m1, no :63333/PG_PASS_SHARE. /code-check clean (fixed burn-verify pipefail, added bucket-overlap warning). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PWF: Stage A pass (PARS ;DAM confirmed, 99% match); full 3-area run launched (#175) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * study_area_run.sh: SSH-readiness wait before cypher prep + largest-area-on-dispatcher note (#175) Fresh cyphers race: cypher_up returns when the IP is assigned, before sshd is up, so scp of cypher_prep.sh hit "Connection closed". Poll ssh (up to ~150s, accept-new host key) before scp. Also document: put the largest study area on the dispatcher (fast/free M1); cyphers are slow+paid, give them smaller areas. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * study_area_run.sh: cyphers run dispatcher branch via CYPHER_PREP_BRANCH (#175) Cyphers git checkout main by default, but the driver scripts (wsg_run_one.R etc.) + branch link live on the feature branch. Pass the dispatcher`s current branch to cypher_prep so cyphers carry the same ref. Branch must be pushed first (cypher_prep does git fetch origin + reset --hard origin/$BRANCH). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * study_area runner: species-presence filter + per-WSG soft-fail (#175, #157) Root-causes the 2026-05-25 data loss: the drainage closure pulled in a species-less WSG (LEUT) -> lnk_pipeline_run errored "No species resolved for AOI" -> dispatcher loop `|| exit 1` -> driver FATAL -> trap burned the cyphers WITH their un-consolidated Peace+Skeena data. One bad WSG lost a whole run. Per the records (research/provincial_run_runbook.md, data-raw/wsgs_run_host.R :88 #157) the proven runner already solved both: - study_area_wsgs.R: filter the closure to bundle-species presence (cfg$species in wsg_species_presence) — drops species-less closure WSGs (Fraser: LEUT, LNRS). Matches wsgs_run_host.R exactly. - study_area_run.sh: per-WSG SOFT-FAIL — a WSG error logs WARN and the loop continues; a non-zero host exit is logged, not fatal. Always reaches consolidate so a late failure can't burn cyphers with unconsolidated data. Mirrors wsgs_run_host.R resume-safe behaviour. - wsg_run_one.R: defensive exit-0 skip when lnk_pipeline_species() is empty. /code-check clean (1 round, 0 findings). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PWF: data-loss incident root-caused + records-based fixes (#175) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: study-area run runbook + README entry (#175) research/study_area_run.md — lean tunnel-free M1-dispatch study-area parity procedure + the 2026-05-25 gotchas (trap-burn data loss, species filter #157, soft-fail, sshd race, wide-table drift, cypher $0.06/hr). data-raw/README.md drivers table gains study_area_run.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * study_area_run: post-consolidate recompute = cross-WSG correctness guarantee (#175, #205) The deliverable is a methodology correct regardless of machine count + WSG bucketing. Drainage-closed + DS-first per-host is NOT sufficient: a WSG's downstream barriers can be cross-bucket or arrive late in DS-first order, so its access (token1/token2) is computed against an incomplete barrier set. Caught 2026-05-25: FINA 75.5% / PARA 68.6% per-host -> 99%+ after re-modelling on the full consolidated barrier set. So: distribute (any bucketing) -> consolidate -> POST-CONSOLIDATE RECOMPUTE the diverged WSGs (any species <99%) on the dispatcher with the complete barrier set -> re-compare. Bucketing becomes a speed knob, not a correctness lever. Recompute uses the full pipeline today (slow); filed #205 for the cheap access-only recompute (reuse persisted streams/habitat) that makes recompute-ALL bulletproof + fast. Docs: research/study_area_run.md (procedure corrected). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PWF correction + 2026-05-25 parity doc (#175) Correct the task_plan claim "cross-WSG ;DAM solved without recompute" — full run disproved it (FINA 75.5%/PARA 68.6% per-host -> 99%+ after recompute). Record authoritative post-recompute parity (median 99.66%) + the methodology finding + genuine divergences (UNRS reservoir, SETN salmon) in research/provincial_parity_2026_05_25.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CLAUDE.md: refresh Status handoff to #175 (study-area parity, post-consolidate recompute, #205) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * PWF: #205 Phase 7 baseline — lnk_access cheap recompute (plan approved) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #205: cheap post-consolidate recompute — lnk_access + 4 supporting fixes The post-consolidate recompute (#175) was running the full pipeline on diverged WSGs — ~2× cost on those WSGs because it re-derived streams/habitat (already correct) just to redo the cheap access step. #205 implements the access-only recompute that reuses persisted streams/habitat/barriers; FINA validated: 11.86s wall (vs ~90s full pipeline = ~8× faster), bcfp parity 99.8% / 57 diffs / ACCESS;DAM top — IDENTICAL to the full-pipeline recompute. Five things had to be right; four are real gotchas worth knowing (RUNBOOK §6): - R/lnk_access.R (new export, @family compare): portable access builder, twin of lnk_mapping_code (table_<role> params). Builds per-species _access + source `_unified` views internally via lnk_barriers_views over the persist barriers; merge=FALSE overwrites, merge=TRUE surgically UPDATEs cross-WSG cols (has_barriers_*, dam_dnstr_ind) on the target while PRESERVING remediated_dnstr_ind and observed access_<sp>=2 from the prior compute. - lnk_access materialises the AOI streams as a REAL TABLE (CREATE TABLE + index id_segment, wscode_ltree GiST, localcode_ltree GiST, blue_line_key, ANALYZE), NOT a view. A view didn't carry small-table stats so the planner picked the ~800k-row barriers as the nested-loop outer driver, blowing cost ~1000× (>10min). With the real table, planner picks the 26k AOI streams as outer, walk takes seconds. - R/lnk_persist_init.R: persist streams + barriers now get the same wscode_ltree / localcode_ltree GIST + btree indexes that fresh::utils.R builds on its working network table (frs_network_features needs them). - R/lnk_mapping_code.R: #203 cross-WSG cartesian fix. The access read was `SELECT * FROM <access> WHERE id_segment IN (SELECT id_segment FROM streams WHERE wsg=aoi)`. Against persist (where id_segment is per-WSG, not globally unique) this matched N×WSGs of duplicates → 50× rows in mc_scratch → PK violation on persist write. Filter by watershed_group_code when the table carries it. - data-raw/wsg_recompute_one.R (new): sibling of wsg_run_one.R. Sets statement_timeout (600s) + lock_timeout (60s) so a runaway/locked query cancels server-side instead of orphaning a backend. data-raw/study_area_run.sh wired to call it + switched to recompute-ALL (cheap → bulletproof; bucketing is now a speed knob, not a correctness lever). Docs: research/study_area_run.md procedure updated; RUNBOOK.md §6 gotchas (orphaned backend / statement_timeout + pkill ≠ cancel query; view-vs-table planner gotcha; #203 cartesian-on-persist). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * wsg_recompute_one: wrap mc DELETE+INSERT in dbWithTransaction (#205 data-loss followup) Without the transaction, a failed INSERT (e.g. the #203 PK-violation that caused FINA mc data loss 2026-05-25) leaves the WSG`s rows deleted but not re-inserted. dbWithTransaction makes the pair atomic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reproduce bcfp's per-species accessibility so dam-downstream segments emit the dam descriptor (
SPAWN;DAM/REAR;DAM/ACCESS;DAM) instead of a bare habitat token.Previously the mapping_code phase drove
accessiblefrombarriers_<sp>_unified(all barriers, incl dams), so every segment below a dam read inaccessible and lost its;DAM/;MODELLED/;ASSESSEDsecond token. It now uses a new per-speciesbarriers_<sp>_accessview reproducing bcfp'sbarriers_<sp>: natural barriers only (gradient@species-threshold ∪ falls ∪ subsurface), MINUS the observation/habitat override, ∪ all user-definite (override-exempt). Dams stay inbarrier_sources→ token2 only.All three access inputs are now persisted province-wide so the cross-WSG downstream walk is correct in every WSG: natural barriers (already),
user_barriers_definite(newUSER_DEFINITEfamily inlnk_barriers_unify, ltree-resolved via the FWA join), and the override (new<persist>.barrier_overridestable, PK includeswatershed_group_codefor boundary-stream overrides shared across adjacent WSGs).Phases: 1 persist override+definite · 2
barriers_<sp>_accessview · 3 repointbarriers_per_sp· 4 DB validation + PK fix · 5 docs + v0.40.4. Mechanism inRUNBOOK.md§5.Validation
Against
smnorris/bcfishpass@v0.7.15: PARS BT 98.95%, LFRA BT 97.77% / CO 97.90% per-segment mapping_code match. Dam-downstream-but-accessible segments now emit;DAM(e.g. PARSSPAWN;DAM5293 vs bcfp 5263). Residual ~1–2% is token1 habitat-presence (ACCESS↔SPAWN/REAR, governed by dimensions/rules), a separate pre-existing concern, not the dam-access fix.Follow-up (next session)
Province-wide parity across the 3 study areas — Peace (16 WSGs), Fraser (8), Skeena (5), drainage-closed to ~52 WSGs — via 4-way cypher dispatch (
wsgs_run_pipeline.sh, 3 cyphers + local). Caveat to resolve there: distributed dispatch computes mapping_code per-WSG per-host, so cross-WSG;DAM(e.g. PARS→PCEA/UPCE dams) may need a two-pass (run+consolidate, then recompute mapping_code on the merged schema) — the orchestrator predates #200's cross-WSG accessibility and may need updating.Closes #200.
Relates to NewGraphEnvironment/sred-2025-2026#24.
🤖 Generated with Claude Code