Repository navigation
v0.40.2: lnk_pipeline_run sizes persist DDL to cfg$species (#194) - #195
Merged
Merged
Conversation
…194) v0.40.1 (#192) made the mapping_code phase use active_species (per-WSG subset of bundle species) for working schema's streams_access columns. That matched persist DDL because lnk_persist_init was passed active_species too. But persist is province-wide — when WSG #2 has a different active set than WSG #1, the persist table is locked to #1's column set and #2's INSERT projection fails. Fix: pass cfg$species (full bundle) to lnk_persist_init. Wide persist tables (streams_access, streams_mapping_code) are sized to the bundle; per-WSG INSERTs in lnk_pipeline_persist continue using active_species for projection. Unused species' columns get NULL for WSGs that don't model them. Per-species habitat tables (streams_habitat_<sp>) also get created for the full bundle — empty tables for species no WSG populates are cheap. Live smoke 2026-05-19: PARS (default bundle, active=bt/gr/ko/rb) ran first → 4-column persist DDL. BULK next (active=bt/ch/co/pk/sk/st/rb) → 7-column INSERT projection against 4-column table → fail. Migration: existing fresh_default.streams_access / fresh_default.streams_mapping_code (sized to PARS's 4 active species) do NOT auto-grow. Operator drops + re-inits to pick up the wider DDL. No regression in bcfishpass bundle: cfg$species = full bcfp 8 = active for most bcfp-bundle WSGs → bit-identical INSERT projection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.40.1 hotfix.
lnk_pipeline_runnow passescfg$species(full bundle) tolnk_persist_initinstead ofactive_species(per-WSG subset). Persist DDL is bundle-sized so heterogeneous-active-set WSGs coexist in the same persist tables. Closes #194.Live verified: PARS (active = bt/gr/ko/rb) then BULK (active = bt/ch/co/pk/sk/st/rb) both populate
fresh_default.streams_access+fresh_default.streams_mapping_codeafter drop+re-init. Persist DDL = 24 cols (id_segment + watershed_group_code + 11 species × 2). Each WSG's INSERT covers its active subset; non-active species' columns get NULL. Tunnel was down throughout — confirms tunnel-free pipeline_run path.Fix
One-line change in
R/lnk_pipeline_run.R:lnk_persist_init(conn, cfg, species = active_species)— first WSG locks persist DDL to its species subset.lnk_persist_init(conn, cfg, species = cfg$species)— bundle-wide persist DDL.lnk_pipeline_persistcontinues usingactive_speciesfor per-WSG INSERT projection (unchanged) so unused species' columns get NULL.Migration
Existing persist tables created with narrower DDL do NOT auto-grow. Drop and re-init:
Then re-run
lnk_pipeline_run(mapping_code = TRUE)for any WSG to recreate them at bundle width.Test plan
/code-checkround 1 cleancfg$species= bcfp 8 = active for most bcfp-bundle WSGs → identical projectionRelated