Skip to content

lnk_persist_init: wide tables sized to active_species break on heterogeneous WSG runs #194

Closed

Description

@NewGraphEnvironment

Context

v0.40.1 (#192) made lnk_pipeline_run(mapping_code = TRUE) use active_species for the working schema's streams_access columns — matching the persist DDL. That fix works ONLY when all WSGs in the run share the same active species set. Live smoke 2026-05-19 surfaced the next failure mode: wide-table species-set evolution.

lnk_persist_init(conn, cfg, species = active_species) at R/lnk_pipeline_run.R:130 uses the per-WSG active_species to size the wide persist tables (streams_access, streams_mapping_code). First WSG locks the table to its species set. Subsequent WSGs with different active sets fail:

  • PARS first (default config): active_species = c("BT","GR","KO","RB") → persist streams_access has 4 species columns.
  • BULK next (same default config): active_species = c("BT","CH","CO","PK","SK","ST","RB") (BULK is in the Skeena, salmon-bearing) → 7-column INSERT projection against 4-column persist table.
  • Result: ERROR: column "has_barriers_ch_dnstr" of relation "streams_access" does not exist.

The per-species habitat tables (streams_habitat_<sp>) don't have this problem because they're one-table-per-species (independent schemas). Only the wide tables introduced in #187 have it.

Goals

  • lnk_persist_init is called with the bundle's full species set (cfg$species), not the per-WSG active_species. Persist DDL = bundle shape.
  • lnk_pipeline_persist keeps using active_species for the per-WSG INSERT projection. Unused species' columns get NULL for WSGs that don't model them.
  • Per-species habitat tables (streams_habitat_<sp>) get created for all bundle species; tables that no WSG populates stay empty. Storage cost is negligible.

Acceptance

  • Drop existing fresh_default.streams_access + fresh_default.streams_mapping_code + fresh_default.streams_habitat_long_vw, then run lnk_pipeline_run(aoi="PARS", cfg=lnk_config("default"), mapping_code=TRUE) → persist tables created with 11-species columns (all of default bundle's cfg$species). PARS INSERT populates the 4 active columns; rest NULL.
  • Subsequent lnk_pipeline_run(aoi="BULK", cfg=lnk_config("default"), mapping_code=TRUE) → BULK INSERT populates its 7 active columns; rest NULL. No INSERT error.
  • Bcfishpass-bundle path unchanged: cfg$species = full bcfp 8 → persist tables sized to 8 → bcfp-bundle WSGs (which have all 8 active) insert all 8 columns. Bit-identical to pre-fix behavior.
  • v0.40.2 patch release.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions