lnk_scenario_export(): publish habitat per config as Parquet + manifest + fwapg load (wet#7 pattern) #308

If done: habitat flags for every link config (default, default_tuned, bcfishpass) and build are published as keyed tables that anyone can load into fwapg and join in SQL, with a manifest recording which build produced them. If never: every consumer hand-joins persist tables, re-solves the (id_segment, watershed_group_code) fan-out (#203) and the mad_m3s-not-persisted gap, and cannot tell which build a habitat number came from.

Shipped 2026-10-14 (PR #337), first cut; where it departs from the plan below, the body says so inline. Mechanics: RUNBOOK §6d.

Revised 2026-10-12. The first version of this issue proposed one wide GeoPackage stream layer per WSG. It now follows the pattern settled in NewGraphEnvironment/wet#7 for per-segment discharge: Parquet on S3, a manifest, and a load into fwapg. No STAC: these are attribute rows keyed to FWA positions, not spatial assets with a footprint of their own, and the geometry is already in fwapg. The GeoPackage stays as an optional desktop export built from the same tables.

Dataset and layout

Dataset ID habitat/fwa/v1/segment, named the way wet#7 names its datasets (<variable>/<network>/<version>/<unit>):

s3://fresh-bc/habitat/fwa/v1/
  manifest.json
  segment/config=<config>/species=<sp>/part-<code>.parquet

• Hive keys: config (the link config name), species. The build is not a key: data are overwritten in place within a schema version, as in wet#7, and the manifest names the build.
• part-<code>: one FWA top-level drainage, as in wet#7, labelled in the manifest. Rows sorted by watershed_group_code, so a one-group read prunes on row-group statistics.
• Long form, no geometry. One row per segment × config × species; flags are columns.

Key: FWA position, not linear_feature_id and not id_segment

link's segments are FWA segments split at break points, and the break points differ by config, so linear_feature_id (wet's key) is not unique and id_segment is numbered per group (#203). The key is (blue_line_key, downstream_route_measure, upstream_route_measure). A consumer cuts a segment's line from the FWA blue line at those measures, so no link geometry is published.

Columns

blue_line_key, downstream_route_measure, upstream_route_measure, linear_feature_id, watershed_group_code, stream_order, edge_type, length_metre, gradient, channel_width, channel_width_source, mad_m3s, mad_m3s_source, then per stage accessible, spawning, rearing, lake_rearing, wetland_rearing, and per stage the ratios <stage>_ratio_cw, <stage>_ratio_mad (size ÷ minimum) and <stage>_ratio_gradient (gradient ÷ maximum). A species with no habitat in a group yields FALSE, never a missing column.

As shipped:

• has_barriers_dnstr (link's streams_access) sits beside accessible (fresh's flag), because the two disagree on the segment that starts at a barrier: accessible and streams_access are built from different barrier sets (DUNC BT gradient barriers) #336.
• Ratios exist for spawning and rearing (cw, mad, gradient) and accessible_ratio_gradient; lake and wetland rearing are area-only and get none.
• Ratios use the thresholds the build logged, so builds before Per-bundle habitat thresholds CSV in config.yaml #282 have NA ratios (thresholds_logged in the manifest).
• Rows exist only for the species a group's build modelled: FALSE means modelled with no habitat, and an unmodelled species has no rows.
• The contract is inst/extdata/schemas/habitat_fwa_v1_segment.yaml in crate's shape; registering it in crate is Register the habitat export's column contract in crate (nge/habitat/fwa/v1/segment) #333. Column types and meanings registered and conformed before writing, as wet#7 does with crate.

Manifest

Per part: dataset, config, species, code, drainage, href, rows, bytes, md5. Per config: link's log row as read (config_name, config_hash, run_uid, link_sha, fwapg_sha, bundle, model, fill), and the discharge input (below). No s3:ListBucket: readers go through the manifest.

Discharge comes from wet

mad_m3s today comes through .lnk_discharge_sql() per each scenario's applied fill. Once NewGraphEnvironment/wet#7 loads discharge.fwa_v1_segment_monthly into fwapg, link reads it from there (month = 0, pick), and the manifest records the discharge dataset ID and the md5s of the wet parts it used. Habitat → discharge provenance is then one chain of manifests.

As shipped: the export reads the discharge the run classified on (.lnk_discharge_sql() with the logged fill), so *_ratio_mad agrees with the flags. Moving classify and the export onto wet's table changes outputs and is #331 (operator, 2026-10-14).

Pieces

• lnk_scenario_export(): builds the tree and manifest from the persist tables for a set of configs and WSGs. First asserts that the scenario schemas compared within one config share segmentation, using the variants harness's streams digest, and refuses on a mismatch (the Extend PARS vignette to demonstrate accessible_km bcfp-equivalence #226 vignette trap). Across configs the segmentation may differ, which the position key handles.
• lnk_catalog(): reads manifest.json, one row per part, with build provenance.
• lnk_db_load(): downloads parts, checks each md5 against the manifest, writes habitat.fwa_v1_segment in fwapg.
• Desktop export (optional): the same rows joined to FWA geometry, written to a GeoPackage per WSG, with a scenarios layer from the manifest. Deferred to Desktop GeoPackage of the habitat export: segments cut from FWA at their measures #332.
• As shipped, also: data-raw/habitat_publish.R (BUILD, then LNK_PUBLISH=1). An export replaces each config wholesale. One judgement, .lnk_manifest_regressions(), refuses in both BUILD and PUBLISH any tree that would lose a live config, a group:species pair, a newer build or recompute, or a newer export (override replace = TRUE / LNK_PUBLISH_REPLACE=1). PUBLISH has not been run.

Consumers

• Publish and region-run whole-network floodplains: GeoParquet, join geometry, species: all in regions floodplains#123: whole-network floodplains are keyed by blue_line_key and the FWA version and carry no link output. Habitat is joined afterwards by FWA position, for whichever config the consumer picks. floodplains records the run_uid and part md5s it joined against, so a new build never blocks a join and "is this the build we joined?" is a lookup.
• fresh, through the fwapg table.

Tests

Segmentation-mismatch refusal; full-key joins (no fan-out across groups); a species with no habitat yields FALSE; manifest md5 round trip; lnk_db_load() refuses a part whose md5 does not match.

Staged private first (operator, 2026-10-14)

• No public read and no versioning on habitat/ yet. The tunings are still moving the outputs (Research: calibrate CH and BT gradient and channel-width thresholds from observations #284, Score the discharge (mad) habitat model against observations, as #284 did for gradient #300, default MAD ranges for BT, GR, KO and RB, from its own width minima #307, Default lake and wetland rearing: connected to spawning, centrelines kept, km and ha broken out #310, rearing_km means two things: lnk_rollup_wsg() keeps lake connection lines, the compare rollups do not #319), and how habitat ties to the floodplain items (Publish and region-run whole-network floodplains: GeoParquet, join geometry, species: all in regions floodplains#123) is still being explored. The tree goes to s3://fresh-bc/habitat/fwa/v1/ under the bucket's private default.

  • So lnk_catalog() and lnk_db_load() must read through credentials (an s3:// path), not only anonymously over https:// as wet does.

• Opening it later is small: one entry in fresh-bc's public_read_prefixes and a tofu apply, as discharge/ was opened (NewGraphEnvironment/rtj#390). Have NewGraphEnvironment/rtj#168's budget and anomaly alerts in place first. Anonymous downloads are billed to us at about $0.09/GB once the account's 100 GB/month free tier is used, and it was used in May 2026 (rtj#168).

• History: data are overwritten in place, and fresh-bc versioning is Suspended, so a superseded build cannot be fetched again. That is acceptable while exploring. Versioning is bucket-wide (NewGraphEnvironment/rtj#263). When it is on and habitat/ should keep history:

  • the manifest records each part's versionId;
  • habitat/ gets a keep-the-last-N rule.

  Decide this before the first build that a published floodplain item pins.

Provenance published beside the data

The run's evidence is written once and never overwritten:

habitat/fwa/v1/provenance/run_uid=<run_uid>/
  log.parquet
  log_input.parquet                           (with artifact_* from #265)
  log_parameters_habitat_thresholds.parquet
  source_artifact.parquet                     (provenance.source_artifact rows for the hosts the run used)
habitat/fwa/v1/config/<config_hash>/          (the bundle files config_hash was computed over)

As shipped (code review, 2026-10-14): a dispatch's rows grow as its groups close, so a per-dispatch file cannot be written once. Provenance is therefore keyed per finished build:

provenance/run_uid=<uid>/run_id=<id>/{log,log_input}.parquet    (run_id=<id>/ for a build outside a dispatch)
provenance/config_hash=<hex>/log_parameters_{habitat_thresholds,fresh}.parquet
provenance/recompute_id=<id>/log_recompute.parquet
provenance/source_artifact/<md5>.parquet       (this database's record at export time, stored by content; what a run read is log_input.artifact_*)
config/sha256-<hex>/                           (only when today's bundle hashes as the build did; otherwise the manifest says why)

Write-once compares records, not bytes (Parquet's footer names the duckdb version). Gates on run_uid and link_dirty before a floodplain item pins a build are #334.

The manifest's per-config entry names its run_uids and config_hash. A consumer, such as the provenance table in NewGraphEnvironment/floodplains#52, records those plus the part md5s, and the versionIds once versioning exists. Everything else resolves from the bucket.

Webmap (later, not the first cut): #335

PMTiles built from the published rows and FWA lines cut at the measures (tippecanoe), regenerated per build at a fixed key. Publication is NewGraphEnvironment/rtj#182; a browser reader also needs NewGraphEnvironment/rtj#392's CORS fix. Open question: one tileset per config, or one per config × species.

Relates to #305, #307, #203, #226, NewGraphEnvironment/wet#7, NewGraphEnvironment/floodplains#123.