Skip to content

Read discharge from wet; re-set BT's MAD minima on it (#331) - #344

Merged
NewGraphEnvironment merged 23 commits into
mainfrom
331-read-discharge-from-wet-in-classify-and
Oct 16, 2026
Merged

NewGraphEnvironment merged 23 commits into
mainfrom
331-read-discharge-from-wet-in-classify-and

Conversation

@NewGraphEnvironment

Copy link
Copy Markdown
Owner

Summary

  • A bundle chooses its discharge source: pipeline: discharge_source: fwapg | wet (default fwapg). The source is threaded through prepare, the run log (<schema>.log.discharge_source), the validator, the export manifest (discharge.source, discharge.wet_load), the scoring harness and the MAD producers. New export lnk_preflight_discharge(), loader data-raw/discharge_wet_load.R, and host prep in study_area_run.sh / cypher_prep.sh. The fwapg SQL is byte-identical to main's.
  • default_tuned reads wet; BT spawn_mad_min / rear_mad_min 0.078 → 0.043.
    • Rearing was scored on held-out groups (P10 → P05 refused).
    • Spawning takes the same any-stage percentile, because the held-out core holds only 3 spawning-staged locations.
    • default and bcfishpass stay on fwapg.
    • No habitat flag moves while every group is on cw: on NATR, 0 of 189,200 rows changed.
  • Why the values moved: coverage, not the source. On BT locations both sources value, P10 is 0.075 on wet and 0.076 on fwapg. wet adds nine Skeena groups fwapg carries no discharge for, and their water is smaller.
  • At the shipped values mad is the closer model for BT.
    • On held-out groups, the water only cw keeps is used at 0.18 (rearing) and 0.14 (spawning) of the size-matched rate. The water only mad keeps is used at 1.64 and 0.69.
    • mad keeps 19.8 % less BT rearing than cw (Main stems with no discharge drop out of mad habitat #305: 32 % at 0.078 on fwapg).
  • Write-up: research/habitat_thresholds.md, "MAD on wet's discharge (Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it #331)". Release 0.66.0.

Related Issues

Test plan

  • Full devtools::test() at 5bd9b325: 2962 pass, 7 fail.
    • 2 failures were this branch's (deefda33 fixes them).
    • 3 need the bcfishpass tunnel on :63333, which was down: test-lnk_db_conn.R:10, test-lnk_wsg_resolve.R:143,154.
    • 2 are the golden dams pin (test-lnk_pipeline_prepare.R:958-959). Local cabd.dams went from 2,594 to 2,592 rows upstream, and this branch doesn't touch the dams code. Re-pin deliberately.
  • devtools::check(--no-tests): 0 errors. Its 3 warnings and 4 notes are all on main (non-ASCII in untouched files, Rd links, setNames imports).
  • Scoring builds on one segmentation:
  • cw unchanged on NATR: one prepare, mad_m3s re-joined from each source, 0 habitat rows differ (data-raw/logs/discharge_wet_331/cw_unchanged_natr.csv).
  • /code-check on the branch, five rounds. Rounds 1–3 covered the code and ended by enumeration. Rounds 4–5 covered the Phase 5 values and the write-up: no code defects, and published claims were narrowed to their evidence, ending with a grep of each claim class.

Notes

  • A host that runs default_tuned needs wet's table: LNK_LOAD=loadall Rscript data-raw/discharge_wet_load.R.
    • That script loads the annual pick only.
    • On a host holding wet's full load, --force or a differing --expect-load replaces it with the pick.
  • default_tuned's config_hash changes.
  • While Build B ran, the local DB volume filled. 62 stale working_<wsg> / zz311_* schemas were dropped; every score* schema was kept.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx

NewGraphEnvironment and others added 23 commits October 15, 2026 15:22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
A bundle sets pipeline: discharge_source (fwapg by default, as every run
before this read). wet is the annual pick of discharge.fwa_v1_segment_monthly,
with 0 read as no value. One relation, .lnk_discharge_rel(), serves prepare,
the fill, the validator, the export and the variants harness; the source is
logged per WSG (<schema>.log.discharge_source) and named per WSG in the
export manifest. lnk_config() refuses an unknown source at load.

The variants harness drops set cells the base already carries (#302's
*_mad_max = 9999, carried by default since #307), still stops on a rung
whose own cell equals the base, and writes each variant's resolved
discharge source and fill into its bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
query_habitat_thresholds_mad.R and query_width_mad_equivalent.R take
--discharge-source=fwapg|wet and read through .lnk_discharge_rel(); the
width producer's --common keeps to lines both sources value. Stamps name
the source and the table's load comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
A host running a wet config needs wet's table, cw groups included:
prepare now stops with the load command when the source is missing, and
lnk_preflight_discharge() checks a database. data-raw/discharge_wet_load.R
loads the annual pick only; study_area_run.sh gates the dispatcher on the
config's source and cypher_prep.sh loads it on each cypher
(CYPHER_PREP_DISCHARGE=wet), installing wet at its DESCRIPTION pin.

log_input records each primitive's table comment, and a wet run adds
wet's table to its inputs, so which published load a build read is kept
with the run. The export manifest's discharge.wet_load sets the logged
loads beside the exporting database's and says whether they match. wet is
not a vintage-gate primitive: a host running an fwapg config needs none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Width-converted minima barely move (BT spawn 0.042 fwapg, 0.043 wet).
BT at observations: wet values 99.8 % of locations (fwapg 89.8 %) and
adds nine Skeena groups, which bring enough spawn-staged locations for
spawning to be decided on them for the first time. Edge 1250 km with no
value in #302's BT groups falls from 506 to 5 on wet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Rungs are the fixed rule's P10 / P05 / P02 from Phase 3: wet rear from
any-stage locations, wet spawn from spawn-staged (58, above the rule's
floor of 30), and fwapg at HEAD for a same-day comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
- The score counted set cells the build had dropped (a base already at the
  value), so it would refuse every #331 and #302 variant; it now counts
  only the set cells that move, and still checks each one's value.
- Every cypher must hold the dispatcher's wet load: study_area_run.sh
  passes its load id, and discharge_wet_load.R reloads a host holding
  another and fails when the published one differs too.
- discharge_wet_load.R --force: the header says it leaves only the pick.
- The discharge gate's hint names the source's own loader.
- RUNBOOK: a source mismatch in the validator warns; it does not refuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
- The load-consistency guard failed toward pass: an empty dispatcher load
  id disabled the cypher check. study_area_run.sh now stops when a wet
  config's dispatcher holds no load, and cypher_prep.sh refuses wet
  without the dispatcher's load id.
- wsgs_run_pipeline.sh, the second caller of cypher_prep.sh, prepares
  hosts for fwapg only: it refuses a config that reads wet (resolved from
  the checkout, not the installed link).
- discharge_wet_load.R's mismatch message names a full reload; --force
  would keep only the annual pick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
The rule that a wet config needs wet's table, and one load, on every host
was enforced only by study_area_run.sh; the other runners failed open.
wsgs_run_host.R, shared by wsgs_dispatch.sh, trifecta_smoke.sh and
wsgs_run_m4_offline.sh, now checks the config's discharge once before its
loop and stops the host, instead of writing an error stub per WSG and
exiting 0. wsgs_dispatch.sh, which cannot hold hosts to one wet load,
refuses a config that reads wet. The planned Phase 5 check re-joins
discharge between its two classify passes, or both read one source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Build A on one segmentation, held-out rule of #302 (floor on locations
expected). Rearing on wet walks to 0.043; spawning on wet is underpowered
on spawn-staged records (3 in the held-out core) and refuses P05 on
any-stage records, so it holds P10 0.012.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…n it (#331)

BT spawn_mad_min 0.078 -> 0.012 and rear_mad_min 0.078 -> 0.043, scored on
held-out groups (Build A). GR, KO and RB keep their fwapg-fitted values.
Adds Build B's variants (model-only BT mad on wet and on fwapg).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
wet's lower BT percentiles come from nine Skeena groups fwapg never
covered, not from the source: on locations both value they agree. 44 of
64 spawning-staged records are Skeena and no held-out group is, so
spawning is set on any-stage records, as #302 did (spawn = rear).

cw classification is unchanged by the source on NATR (one prepare, both
joins). Build B's first run died on a full disk; its log is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
.lnk_discharge_join() checks its source table before writing (Phase 6),
which a mock connection cannot answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
At BT 0.043 / 0.043 mad is the closer model on both sources: cw-only
water is used at 0.18 (rearing) and 0.14 (spawning) of the size-matched
core, mad-only at 1.64 and 0.69. mad keeps 19.8 % less BT rearing than
cw on wet, 21.3 % on fwapg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
research/habitat_thresholds.md gains the section: verdict, why the
values moved (Skeena coverage), Build A's ladders and Build B's cw
against mad. CLAUDE.md status and the default_tuned README follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Code-check round 4 on the post-review diff: no code defects, six claims
narrowed (valued lines, not rows; BT spawning unscored at 0.043; mad-only
spawning used at 0.69 of the core, not above it; locations, not records).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Code-check round 5 found claims the round 4 fixes had not reached
(RUNBOOK's line counts, NEWS's headline, #302's ladders 'only at
v0.58.0', the default_tuned fill comment). Each claim class was then
grepped across the tracked text and every live copy fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
…om-wet-in-classify-and

* origin/main:
  Remove comms/
  Credit New Graph Environment Ltd. as publisher in the citation
  Hex sticker: keep the small logo's name clear of the icon
  pkgdown: cap the job at 90 minutes
  Exclude a worktree's .git file from the build
  update-citation-cff: drop CRAN namesake fields; name the publisher
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJE2szSCkeV9R3qKhuSwyx
@NewGraphEnvironment
NewGraphEnvironment merged commit 78f16df into main Oct 16, 2026
1 check passed
@NewGraphEnvironment
NewGraphEnvironment deleted the 331-read-discharge-from-wet-in-classify-and branch October 16, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read discharge from wet in classify and the habitat export, and re-tune BT's MAD range on it

1 participant