Run provenance log on persist schema: log + full config snapshot + input fingerprints #127

Problem

<persist_schema>.streams, streams_habitat_<sp>, streams_access, streams_mapping_code and barriers accumulate rows across many runs but carry no provenance. Given a network in the database today you cannot answer:

• which config produced it (and what that config actually said at the time)
• which vintage of the DB primitives it was built from
• which link/fresh code wrote it
• when it was written, and whether neighbouring WSGs in the same schema came from a different run (mixed-vintage state)

lnk_pipeline_persist does DELETE-WHERE-WSG + INSERT, so re-runs cleanly overwrite a WSG — that's the rewrite-is-enough property the orchestrator depends on, but it leaves no audit trail.

This blocks scenario work directly. #236 proposes running the DV-as-BT override with and without and adding a watershed-group axis; those runs land in the same persist schema and are indistinguishable a week later.

What can change between two runs (inventory)

Two notes:

• (2) is the bigger hole. config.yaml pins the bcfp CSVs (each carries upstream_sha), but the bcfp-derived data in the database arrives via snapshot_bcfp.sh from objectstore/BCDC/s3 and is recorded nowhere. Results move with observation vintage and nothing explains why.
• (5) is link-specific. Per RUNBOOK §5, a WSG's ;DAM tokens depend on which other WSGs were already in persist — same config, same code, same data, different run order gives different output. bcfp never hits this (province in one shot). If we log nothing else unique to link, log this.

Also: lnk_config_verify() exists (R/lnk_config_verify.R:67) but nothing calls it during a run. Config provenance is declared, not verified — so hash at run time rather than copying the provenance: block, which could record a lie.

Reference: how bcfishpass does it

bcfishpass.log (model_run_id PK, model_type, date_completed, model_version) with FK'd children that snapshot the actual parameter values, not pointers:

• log_parameters_habitat_thresholds — per species: spawn_gradient_max, channel widths, mad, rear_lake_ha_min
• log_parameters_habitat_method, log_wsg_linear_summary, log_wsg_crossing_summary, log_aw_linear_summary, log_wcrp_habitat_connectivity

Run 134 (v0.7.15-45-gf8db4b9, 2026-08-04) records BT at spawn_gradient_max 0.0549 in the database. The network stays interpretable after the config file moves on. That's the property we want.

Proposed solution

Rename from the original persist_log: "persist" is the phase that writes, not the thing described — the unit is a run. Noun-first matches link's existing tables (streams, barriers, streams_access) and mirrors bcfp's log / log_<noun>.

<schema>.log — one row per lnk_pipeline_run() call

INSERT-only. Latest per WSG = SELECT DISTINCT ON (watershed_group_code) … ORDER BY watershed_group_code, date_start DESC.

<schema>.log_parameters_fresh and <schema>.log_dimensions — full rows

Full contents of parameters_fresh.csv and dimensions.csv keyed on (config_hash, species_code), columns mirroring the CSVs — bcfp's log_parameters_habitat_thresholds pattern. Stored once per distinct config_hash, not per run, so a provincial pass adds one set.

Generate DDL + INSERT projection from a single cols_* named vector (per the R conventions in CLAUDE.md) so column drift can't desync the two sites — the failure mode #204 describes.

This is what makes a network self-describing: observation_species = BT;DV at threshold 1 with no date floor is recorded in the database, so a scenario run is identifiable after the config changes.

<schema>.log_input — DB primitive fingerprints

One row per primitive per run: run_id, table_name, row_count, fingerprint, loaded_at (NULL until snapshot_bcfp.sh stamps).

Fingerprint at run time (row count plus a cheap content digest — e.g. max(observation_date) for observations) so this works before snapshot_bcfp.sh learns to stamp. Teaching the snapshot script to write real load provenance is the follow-up that fills loaded_at / source SHA.

Acceptance

• After a run, <schema>.log has one row with date_start/date_end populated and wsg_upstream listing WSGs already persisted.
• log_parameters_fresh / log_dimensions hold the full CSV contents for that run's config_hash; a second run on the same config adds no new rows.
• Changing one value in parameters_fresh.csv yields a new config_hash and a new parameter set — two networks in the same schema are distinguishable.
• log_input records row counts for every primitive in the inventory above.
• A killed run leaves date_end NULL (partial state is visible, not silent).
• Editing a tracked config file without committing sets link_dirty = true.

Related

• Research: DV as BT access evidence — regional scope + multi-rule observation overrides #236 — DV as BT access evidence; scenario runs need this to be tellable apart
• run_provincial_parity.R cache key doesn't capture code/config version (silent stale-mix risk) #110 — parity cache key doesn't capture code/config version (same class of risk)
• lnk_config_audit + crt_schema_audit: verify bundled CSVs against schema, provenance, and peer DB #86 — lnk_config_audit: verify bundled CSVs against schema/provenance (lnk_config_verify() exists, is uncalled)
• Persist+consolidate must be host-/species-count-agnostic: persist_init blind to species-column-set drift #204 — persist_init blind to species-column-set drift (why DDL+INSERT share one cols_* source)
• Single-WSG downstream-state guard: fail loud when downstream dams aren't persisted #227 — reproducible public.wsg_outlet builder (a primitive with no provenance at all)