Skip to content

Promote with_mapping_code flag to stand-alone lnk_compare_mapping_code() export #175

Closed

Description

@NewGraphEnvironment

Context

The link R package (NewGraphEnvironment/link, currently v0.38.0) ships a lnk_compare_* family for parity comparisons against bcfishpass. In v0.37.0 (#168) the family split into:

  • lnk_pipeline_run(conn, aoi, cfg, loaded, ...) — modelling-only umbrella; writes <persist_schema>.streams + per-species habitat + barriers.
  • lnk_compare_rollup(conn, aoi, cfg, reference, conn_ref, species) — reference-agnostic comparison reader; reads persisted state + reference DB, returns long-format rollup tibble.
  • lnk_compare_wsg(conn, aoi, cfg, loaded, reference, with_mapping_code, ...) — thin wrapper over both, plus an additional mapping_code lens when with_mapping_code = TRUE.

The mapping_code branch (existing helper .lnk_compare_wsg_mapping_code at R/lnk_compare_wsg.R:600-696) builds streams_mapping_code tokens per segment via lnk_pipeline_access + lnk_pipeline_mapping_code, then diffs against bcfishpass.streams_mapping_code. It's the segment-level QA artifact that complements the rollup-km diff.

Today the mapping_code is a flag (with_mapping_code = TRUE) inside lnk_compare_wsg, not its own family member. The flag's implementation operates on the per-WSG working schema and is fully coupled to lnk_compare_wsg's bundled pipeline-run flow (i.e. it cannot run against persisted state alone).

Problem

  1. Family-shape inconsistency. The lnk_compare_* family currently has lnk_compare_rollup (its own export) and lnk_compare_wsg(with_mapping_code = TRUE) (a flag). A naming family should have stand-alone members per concern, not a bundled-with-flag pattern. Other family members (lnk_pipeline_*, lnk_barriers_*) already follow this rule.
  2. Coupling to working schema. The current mapping_code build runs after lnk_pipeline_* phases left a working schema populated. It cannot be re-run against an already-persisted WSG without re-running the entire pipeline. This blocks an operator workflow like "rebuild streams; later compute mapping_code on the same WSG without re-pipeline."
  3. Resume gate doesn't apply. The v0.37.0 PG-state resume in data-raw/wsgs_run_host.R only handles the rollup. A --with-mapping-code run does its own full pipeline every time.

Goals

  • Expose lnk_compare_mapping_code(conn, aoi, cfg, reference = "bcfishpass", conn_ref, ...) as a stand-alone export, sibling to lnk_compare_rollup.
  • Read inputs from <persist_schema> (streams + per-species habitat + barriers). No working-schema dependency on entry.
  • Rebuild the access/streams_mapping_code intermediate tables in a fresh per-WSG working schema (or accept a schema= arg to control). Drop on exit unless cleanup_working = FALSE.
  • lnk_compare_wsg(with_mapping_code = TRUE) becomes a thin convenience wrapper that calls lnk_pipeline_run() + lnk_compare_rollup() + lnk_compare_mapping_code(). Preserves operator-facing call surface; signals that the flag-mode is no longer the primary path.
  • Resume gate in the orchestrator extends to mapping_code: when PG state is canonical for streams/habitat/barriers AND a prior mapping_code RDS exists, skip; when PG state is canonical but mapping_code RDS is missing, run mapping_code only (~10-20s instead of full pipeline).

Acceptance

  • New export R/lnk_compare_mapping_code.R with the signature above; reads <persist_schema> only (no loaded arg).
  • tests/testthat/test-lnk_compare_mapping_code.R covers arg validation + composition (working-schema rebuild + diff) via with_mocked_bindings. Live-DB bit-identical test against the existing lnk_compare_wsg(with_mapping_code=TRUE) output for one WSG.
  • Refactor .lnk_compare_wsg_mapping_code into lnk_compare_mapping_code proper; old helper becomes a @noRd wrapper for back-compat or is removed.
  • lnk_compare_wsg(with_mapping_code = TRUE) still works (back-compat); now routes through the new exports.
  • data-raw/wsgs_run_host.R resume gate gains a third branch for "mapping_code missing but pipeline + rollup OK" → calls mapping_code only.
  • data-raw/wsg_compare.R companion — accept a with_mapping_code arg or expose a second wsg_compare_mapping_code per the data-raw split convention from Decouple bcfp compare from link pipeline run — separate compare-only step #168.
  • NEWS entry + devtools::test() clean + /code-check clean per phase commit.

References

  • v0.37.0 NEWS entry (NewGraphEnvironment/link NEWS.md) — describes the Decouple bcfp compare from link pipeline run — separate compare-only step #168 decouple and the mapping_code-stays-bundled-for-v1 decision.
  • Archived PWF for Decouple bcfp compare from link pipeline run — separate compare-only step #168: planning/archive/2026-05-issue-168-decouple-pipeline-compare/{task_plan,findings,progress}.md documents why mapping_code was deferred.
  • Current mapping_code implementation: R/lnk_compare_wsg.R:199-256 (orchestration path) + R/lnk_compare_wsg.R:600-696 (.lnk_compare_wsg_mapping_code helper).
  • Sibling export pattern: R/lnk_compare_rollup.R (added in v0.37.0).
  • Related future-issue (filed alongside): family-rename lnk_compare_wsg → lnk_compare_run (pairs naturally with this one once all family members exist).

Out of scope

  • The lnk_compare_wsg → lnk_compare_run rename — separate issue.
  • Persist family rename — separate issue.

Update 2026-05-24 (post-#200): make the reference tunnel-free + add cross-WSG recompute

#200 (v0.40.4, merged) reproduced bcfp's per-species access so dam-downstream segments emit ;DAM, validated by a per-segment mapping_code match vs the local snapshot fresh.streams_vw_bcfp (PARS BT 98.95%, LFRA BT 97.77% / CO 97.90%). That sharpens this issue:

Reference is the local snapshot, not the tunnel. lnk_compare_mapping_code should diff link's <persist>.streams_mapping_code against fresh.streams_vw_bcfp (loaded tunnel-free by snapshot_bcfp.sh --with-bcfp-views), joined on blue_line_key + downstream_route_measure per species. Single local conn, no conn_ref, no :63333, no PG_PASS_SHARE. Supersedes the tunnel-coupled framing above and resolves #167 (no tunnel → no drops). bcfp build identity verified against s3://fresh-bc/bcfishpass/log.json (currently v0.7.15-14-ge12c1a5).

Orchestrator: compare on the dispatcher, post-consolidate (the #200 cross-WSG fix). In distributed runs each host computes mapping_code against only its local bucket's barriers, so cross-WSG ;DAM (e.g. PARS→PCEA/UPCE dams) is wrong across hosts. Move compare off the per-host path: after Step 9 consolidate (all barriers province-wide in one schema), recompute streams_access + streams_mapping_code for all WSGs, then run lnk_compare_mapping_code once. Net simplification: cyphers only run+persist (no tunnel, no compare); the dispatcher does recompute + one local snapshot-join compare.

Also in scope (orchestrator): M1-as-dispatcher wiring in wsgs_run_pipeline.sh / wsgs_dispatch.sh (drop the hardcoded ssh m1 self-ssh + the M4-centric host model).

Validation target: the 3 study areas — Peace (16 WSGs), Fraser (8), Skeena (5), drainage-closed to ~52 WSGs, 4-way (3 cyphers + M1). Relates to SRED NewGraphEnvironment/sred-2025-2026#24.

Activity

  1. added a commit that references this issue on Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions