Skip to content

M12.5 groundwork: record what the classifier said, not only what the router used - #42

Merged
codeitlikemiley merged 1 commit into
mainfrom
m12.5-shadow-wiring
Aug 24, 2026
Merged

M12.5 groundwork: record what the classifier said, not only what the router used#42
codeitlikemiley merged 1 commit into
mainfrom
m12.5-shadow-wiring

Conversation

@codeitlikemiley

Copy link
Copy Markdown
Owner

The brief's task 7 was "wire ShadowClassifier behind a flag — zero production call sites". Two things turned out otherwise.

The seam already exists

GatewayBuilder::classifier() takes an Arc<dyn Classifier>, and ShadowClassifier is a Classifier that returns the incumbent's answer by construction. Wiring it is one line.

The blocker isn't a missing flag — there is no candidate classifier to shadow, and won't be until M19.3. A flag guarding a comparison against nothing would be ceremony.

The stated goal had a real gap, upstream of shadow mode

Gateway::resolve read:

let (task, _confidence, _trusted) = classify_or_default()

Both discarded. So route_decisions recorded the class the router used and nothing about how it got there. A row saying chat means either:

  • the heuristic was certain, or
  • it guessed something at 0.2 and docs/12's confidence gate fell back to chat

Opposite facts about the same field — and that difference is exactly the label quality a learned router (M19.3) trains on. Every request was discarding it, unrecoverably. That is "accumulate routing evidence before a model exists."

Migration 0011 adds confidence real and trusted boolean: nullable, no DEFAULT. Rows written before this genuinely have no value, and 0.0 would read as "the classifier was certain of nothing" rather than "nobody asked". A gap should look like a gap. Both are content-free like the rest of the table (docs/20 T5) — a float and a bool say nothing about what the user typed.

A third finding, recorded not fixed

The comment on that same line claimed "A declared class wins; otherwise classify." It doesn't — CallMeta.task is never read, so a caller declaring Code gets whatever the heuristic guesses. docs/12 says "Callers that know, say", so spec and code disagree and the spec isn't the wrong one.

Latent (no ingress populates the field), and honouring it changes routing — so it's a question in RUN-REPORT.md and its own commit, not a silent behaviour change smuggled in here. The false comment is replaced with an accurate one rather than left to mislead.

This is the third field this session that reads like a control and controls nothing, after plugin.toml's net: and NetPolicy::allow. Three instances is a pattern.

On the test

It uses routes::insert, not PgRouteAudit::record — the trait impl detaches the write onto a background task so inference never waits on the database. Reading straight after record races the spawn; it did, once, with RowNotFound. What's under test is the columns, not the detachment.

Verification

  • Integration lane 149/149, twice, run as CI runs it.
  • Falsified: unbinding the two columns turns the new test red and nothing else.
  • fmt · clippy -D warnings · 1114 workspace tests · schemas · ts-sdk · sbom · deny.

https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf

…router used

Task 7 of the run brief asked me to wire `ShadowClassifier` behind a flag,
"zero production call sites". Two things turned out otherwise, and the second is
the reason this commit exists.

**The seam already exists.** `GatewayBuilder::classifier()` takes an
`Arc<dyn Classifier>`, and `ShadowClassifier` *is* a `Classifier` that returns
the incumbent's answer by construction. Wiring it is one line. The blocker is
not a missing flag — it is that there is no candidate classifier to shadow, and
there will not be until M19.3. A flag guarding a comparison against nothing
would be ceremony.

**But the stated goal — accumulate routing evidence before a model exists — had
a real gap, upstream of shadow mode.** `Gateway::resolve` read:

    let (task, _confidence, _trusted) = classify_or_default(..)

Both discarded. So `route_decisions` recorded the class the router *used* and
nothing about how it was arrived at: a row saying `chat` means either "the
heuristic was certain" or "it guessed something at 0.2 and docs/12's confidence
gate fell back to chat". Those are opposite facts about the same field, and the
difference is exactly the label quality a learned router (M19.3) would train on.
Every request was discarding it, unrecoverably.

Migration 0011 adds `confidence real` and `trusted boolean`, nullable and with
no DEFAULT — rows written before this genuinely have no value, and 0.0 would
read as "the classifier was certain of nothing" rather than "nobody asked". A
gap is visibly a gap. Both columns are content-free like the rest of the table
(docs/20 T5): a float and a bool say nothing about what the user typed.

Also fixed, because I was editing the line it sits on: the comment there claimed
"A declared class wins; otherwise classify". It does not — `CallMeta.task` is
never read, so a caller declaring `Code` gets whatever the heuristic guesses.
docs/12 says "Callers that know, say", so the spec and the code disagree and the
spec is not the one that is wrong. Latent (no ingress populates the field), and
honouring it changes routing, so it is a question in RUN-REPORT.md and its own
commit — not a silent behaviour change smuggled in here. The false comment is
replaced by an accurate one rather than left to mislead.

The round-trip test uses `routes::insert` rather than `PgRouteAudit::record`,
because the trait impl detaches the write onto a background task so inference
never waits on the database. Reading straight after `record` races the spawn —
it did, once, with `RowNotFound`. What is under test is the columns, not the
detachment.

Verified: integration lane 149/149 twice; unbinding the two columns turns the
new test red and nothing else. fmt, clippy -D warnings, 1114 workspace tests,
schemas, ts-sdk, sbom, deny.

Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf
@codeitlikemiley
codeitlikemiley marked this pull request as ready for review August 24, 2026 03:58
@codeitlikemiley
codeitlikemiley merged commit 4e1ce9c into main Aug 24, 2026
6 checks passed
@codeitlikemiley
codeitlikemiley deleted the m12.5-shadow-wiring branch August 24, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant