Repository navigation
Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it #331
Description
If done: classify, the validator, the scoring harness and the habitat export (#308) all read discharge from wet's published discharge.fwa_v1_segment_monthly (month = 0, pick), and the export's manifest records which wet build the run read (the md5s of its parts wait on NewGraphEnvironment/wet#88, which records them at load), so habitat → discharge provenance is one chain of manifests. BT's MAD range is re-tuned on wet's values the way #302 tuned it on fwapg's, so whether to move any group to mad is decided on the source it would run on. If never: link keeps reading whse_basemapping.fwa_stream_networks_discharge, whose version is only what provenance.source_artifact happened to record on the exporting host, and default_tuned's BT 0.078 m³/s stays a threshold fitted to a source we mean to leave.
Split out of #308 (operator, 2026-10-14). The export reads the discharge the run classified on, so *_ratio_mad agrees with the flags; moving the source changes outputs, so it is its own change.
- One subquery already serves every consumer:
.lnk_discharge_sql()(R/lnk_discharge.R), table named by.lnk_hv_discharge_tbl(). The fill (Main stems with no discharge drop out of mad habitat #305) is layered on it. - wet's loader is
wet_db_load()(Publish per-segment discharge: parquet + manifest + fwapg load for fresh#114 wet#7); it writesdischarge.fwa_v1_segment_monthlywith a partial index(scenario, linear_feature_id) WHERE month = 0 AND pick. Nothing needs building: the parts are published ats3://fresh-bc/discharge/fwa/v1/, so the first step on any database iswet_db_load(conn, wet_catalog())(43 parts, 103 M rows, 1.1 GB forhistorical). - Every host that runs the pipeline needs the table once classify reads it: the multi-host dispatcher's prep and the vintage gate (
lnk_preflight_vintage(),snapshot_bcfp.sh) should load or check it, andprovenance.source_artifactshould record the manifest it came from. Measured 2026-10-15 on a local fwapg: a fullhistoricalload took 15.5 min, 102,790,688 rows, 20 GB with indexes; 4,532,843 lines carry an annual pick (3,152,838pcic_routed, 1,380,005wet_balance, none twice; 29,491 of them 0 m³/s) against 4,907,441 FWA lines and 2,716,652 rows in fwapg's discharge table (2,003,189 with a value). Decide: whether the Main stems with no discharge drop out of mad habitat #305 fill still applies to wet's values, and what to log per run.Decided: the fill applies unchanged, and on wet it reaches almost nothing (edge 1250 km with no value in Tune MAD (discharge) thresholds for BT, GR, KO and RB rather than leave them missing #302's BT groups: 506 on fwapg, 5 on wet). Each run logsdischarge_sourceper WSG, a wet run'slog_inputcarries wet's table and its comment (the wet build), and the export manifest names the source per WSG plus adischarge.wet_loadblock. Per-part md5s cannot be recorded untilwet_db_load()records them: wet_db_load(): record which parts were loaded (key, md5) in the database wet#88.Decide: switch the source outright, or make it a bundle setting.Decided: a bundle setting,pipeline: discharge_source: fwapg | wet, defaultfwapg. Onlydefault_tunedreads wet (operator); wet's 0 m³/s reads as no value (operator).
Re-tune BT discharge on wet's values (operator, 2026-10-15)
The same exercise #284 / #302 ran for gradient, width and discharge, repeated on the new source before any group moves to mad:
- Unblock the scoring harness at HEAD. A
setcell the base already carries is now dropped with a message; a rung whose own cell equals the base still stops the build.data-raw/habitat_variants_build.R/_score.Rstop at any HEAD after default MAD ranges for BT, GR, KO and RB, from its own width minima #307: the variant bundlesextends: defaultby name anddefault's thresholds moved, so the guard reports "variant … equals default" (habitat_variants_build.R:311): Tune MAD (discharge) thresholds for BT, GR, KO and RB rather than leave them missing #302's rungs set*_mad_max = 9999, whichdefaulthas carried since default MAD ranges for BT, GR, KO and RB, from its own width minima #307. Vignette: default against default_tuned for bull trout in the Parsnip, side by side #342 needs the same fix; do it once. -
RecomputeMeasured, not landed: the conversion barely moves (BT spawning 2 m: 0.043 on wet, 0.042 on fwapg; rearing 1.5 m: 0.023 and 0.021), anddefault's width-converted MAD minimums on wet.defaultstays on fwapg (operator:default_tunedonly).default's 0.041 (spawn, 2 m), 0.20 (GR spawn, 4 m) and 0.021 m³/s (rear, 1.5 m) are the median fwapgmad_m3sat each width minimum (default MAD ranges for BT, GR, KO and RB, from its own width minima #307), so they move with the source. Producer:data-raw/query_width_mad_equivalent.R, with its source swapped. - Re-run Tune MAD (discharge) thresholds for BT, GR, KO and RB rather than leave them missing #302's BT MAD ladder on wet's discharge: same fixed rule (P10, P05 loosening refused unless it scores), same held-out WSGs, same species pooling. Report whether BT's 0.078 m³/s spawn / rear minimum moves. It moves to 0.043 / 0.043. Rearing walks to its wet P10 (P10 → P05 refused). Spawning could not be scored on spawning-staged records (3 in the held-out core) and takes the same any-stage value (operator). The values move because wet covers nine Skeena groups fwapg does not; on locations both value, the sources agree.
- Re-run Score the discharge (mad) habitat model against observations, as #284 did for gradient #300's
cw-against-madscore for BT (with the Main stems with no discharge drop out of mad habitat #305 fill; without it was not run, since the fill reaches 5 km on wet), size-adjusted, on the same held-out WSGs. Compare with Score the discharge (mad) habitat model against observations, as #284 did for gradient #300 / Main stems with no discharge drop out of mad habitat #305: BT rearingcw-only 0.19 after the fill,mad−32 % BT stream rearing againstcw. At 0.043 on wetmadis the closer model:cw-only rearing used at 0.18 of the size-matched core,mad-only at 1.64;madkeeps −19.8 % BT rearing againstcw(fwapg at the same thresholds −21.3 %). - Write the result into
research/habitat_thresholds.mdbeside the fwapg sections, and land any changed value indefault_tunedonly.
Out of scope: a spawning-season discharge rule from wet's monthly values. That is a new kind of rule, not a re-tune; file it separately if this result makes it worth doing.
GR, KO and RB on wet: #343.
Relates to #308, #305, #302, #300, #342, NewGraphEnvironment/wet#7.
Activity
[-]Read discharge from wet's discharge.fwa_v1_segment_monthly in classify and the habitat export[/-][+]Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it[/+]Plain-language summary atop the #331 section
Merge pull request #344 from NewGraphEnvironment/331-read-discharge-f…