Skip to content

Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it #331

Closed

Description

@NewGraphEnvironment

If done: classify, the validator, the scoring harness and the habitat export (#308) all read discharge from wet's published discharge.fwa_v1_segment_monthly (month = 0, pick), and the export's manifest records which wet build the run read (the md5s of its parts wait on NewGraphEnvironment/wet#88, which records them at load), so habitat → discharge provenance is one chain of manifests. BT's MAD range is re-tuned on wet's values the way #302 tuned it on fwapg's, so whether to move any group to mad is decided on the source it would run on. If never: link keeps reading whse_basemapping.fwa_stream_networks_discharge, whose version is only what provenance.source_artifact happened to record on the exporting host, and default_tuned's BT 0.078 m³/s stays a threshold fitted to a source we mean to leave.

Split out of #308 (operator, 2026-10-14). The export reads the discharge the run classified on, so *_ratio_mad agrees with the flags; moving the source changes outputs, so it is its own change.

  • One subquery already serves every consumer: .lnk_discharge_sql() (R/lnk_discharge.R), table named by .lnk_hv_discharge_tbl(). The fill (Main stems with no discharge drop out of mad habitat #305) is layered on it.
  • wet's loader is wet_db_load() (Publish per-segment discharge: parquet + manifest + fwapg load for fresh#114 wet#7); it writes discharge.fwa_v1_segment_monthly with a partial index (scenario, linear_feature_id) WHERE month = 0 AND pick. Nothing needs building: the parts are published at s3://fresh-bc/discharge/fwa/v1/, so the first step on any database is wet_db_load(conn, wet_catalog()) (43 parts, 103 M rows, 1.1 GB for historical).
  • Every host that runs the pipeline needs the table once classify reads it: the multi-host dispatcher's prep and the vintage gate (lnk_preflight_vintage(), snapshot_bcfp.sh) should load or check it, and provenance.source_artifact should record the manifest it came from. Measured 2026-10-15 on a local fwapg: a full historical load took 15.5 min, 102,790,688 rows, 20 GB with indexes; 4,532,843 lines carry an annual pick (3,152,838 pcic_routed, 1,380,005 wet_balance, none twice; 29,491 of them 0 m³/s) against 4,907,441 FWA lines and 2,716,652 rows in fwapg's discharge table (2,003,189 with a value).
  • Decide: whether the Main stems with no discharge drop out of mad habitat #305 fill still applies to wet's values, and what to log per run. Decided: the fill applies unchanged, and on wet it reaches almost nothing (edge 1250 km with no value in Tune MAD (discharge) thresholds for BT, GR, KO and RB rather than leave them missing #302's BT groups: 506 on fwapg, 5 on wet). Each run logs discharge_source per WSG, a wet run's log_input carries wet's table and its comment (the wet build), and the export manifest names the source per WSG plus a discharge.wet_load block. Per-part md5s cannot be recorded until wet_db_load() records them: wet_db_load(): record which parts were loaded (key, md5) in the database wet#88.
  • Decide: switch the source outright, or make it a bundle setting. Decided: a bundle setting, pipeline: discharge_source: fwapg | wet, default fwapg. Only default_tuned reads wet (operator); wet's 0 m³/s reads as no value (operator).

Re-tune BT discharge on wet's values (operator, 2026-10-15)

The same exercise #284 / #302 ran for gradient, width and discharge, repeated on the new source before any group moves to mad:

Out of scope: a spawning-season discharge rule from wet's monthly values. That is a new kind of rule, not a re-tune; file it separately if this result makes it worth doing.

GR, KO and RB on wet: #343.

Relates to #308, #305, #302, #300, #342, NewGraphEnvironment/wet#7.

Activity

  1. changed the title [-]Read discharge from wet's discharge.fwa_v1_segment_monthly in classify and the habitat export[/-] [+]Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it[/+] on Oct 15, 2026
  2. added a commit that references this issue on Oct 16, 2026
  3. added a commit that references this issue on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions