Repository navigation
lnk_persist_init: wide tables sized to active_species break on heterogeneous WSG runs #194
Description
Context
v0.40.1 (#192) made lnk_pipeline_run(mapping_code = TRUE) use active_species for the working schema's streams_access columns — matching the persist DDL. That fix works ONLY when all WSGs in the run share the same active species set. Live smoke 2026-05-19 surfaced the next failure mode: wide-table species-set evolution.
lnk_persist_init(conn, cfg, species = active_species) at R/lnk_pipeline_run.R:130 uses the per-WSG active_species to size the wide persist tables (streams_access, streams_mapping_code). First WSG locks the table to its species set. Subsequent WSGs with different active sets fail:
- PARS first (default config):
active_species = c("BT","GR","KO","RB")→ persiststreams_accesshas 4 species columns. - BULK next (same default config):
active_species = c("BT","CH","CO","PK","SK","ST","RB")(BULK is in the Skeena, salmon-bearing) → 7-column INSERT projection against 4-column persist table. - Result:
ERROR: column "has_barriers_ch_dnstr" of relation "streams_access" does not exist.
The per-species habitat tables (streams_habitat_<sp>) don't have this problem because they're one-table-per-species (independent schemas). Only the wide tables introduced in #187 have it.
Goals
lnk_persist_initis called with the bundle's full species set (cfg$species), not the per-WSGactive_species. Persist DDL = bundle shape.lnk_pipeline_persistkeeps usingactive_speciesfor the per-WSG INSERT projection. Unused species' columns get NULL for WSGs that don't model them.- Per-species habitat tables (
streams_habitat_<sp>) get created for all bundle species; tables that no WSG populates stay empty. Storage cost is negligible.
Acceptance
- Drop existing
fresh_default.streams_access+fresh_default.streams_mapping_code+fresh_default.streams_habitat_long_vw, then runlnk_pipeline_run(aoi="PARS", cfg=lnk_config("default"), mapping_code=TRUE)→ persist tables created with 11-species columns (all of default bundle'scfg$species). PARS INSERT populates the 4 active columns; rest NULL. - Subsequent
lnk_pipeline_run(aoi="BULK", cfg=lnk_config("default"), mapping_code=TRUE)→ BULK INSERT populates its 7 active columns; rest NULL. No INSERT error. - Bcfishpass-bundle path unchanged:
cfg$species= full bcfp 8 → persist tables sized to 8 → bcfp-bundle WSGs (which have all 8 active) insert all 8 columns. Bit-identical to pre-fix behavior. - v0.40.2 patch release.
References
R/lnk_pipeline_run.R:130— single-line change site (active_species→cfg$species).R/lnk_persist_init.R— no change needed (it already acceptsspeciesarg and drives DDL from it).R/lnk_pipeline_persist.R— no change needed (already usesspecies = active_speciesfor INSERT projection).- Follows from lnk_pipeline_run: mapping_code phase hardcodes bcfp 8 species → fails on non-bcfp bundles #192 (the previous species-set fix). Code-check on Phase 1 of mapping_code build decoupled from tunnel — persist streams_access + lnk_pipeline_run phase + rename with_mapping_code → mapping_code #187 flagged this exact issue as Phase-2-level evolution risk; deferred and bit us.
Activity
v0.40.2: lnk_pipeline_run sizes persist DDL to cfg$species (#194) (#195)
v0.40.3: persist per-source flags in streams_access (#196) (#199)