M12.5 groundwork: record what the classifier said, not only what the router used - #42
Merged
Merged
Conversation
…router used
Task 7 of the run brief asked me to wire `ShadowClassifier` behind a flag,
"zero production call sites". Two things turned out otherwise, and the second is
the reason this commit exists.
**The seam already exists.** `GatewayBuilder::classifier()` takes an
`Arc<dyn Classifier>`, and `ShadowClassifier` *is* a `Classifier` that returns
the incumbent's answer by construction. Wiring it is one line. The blocker is
not a missing flag — it is that there is no candidate classifier to shadow, and
there will not be until M19.3. A flag guarding a comparison against nothing
would be ceremony.
**But the stated goal — accumulate routing evidence before a model exists — had
a real gap, upstream of shadow mode.** `Gateway::resolve` read:
let (task, _confidence, _trusted) = classify_or_default(..)
Both discarded. So `route_decisions` recorded the class the router *used* and
nothing about how it was arrived at: a row saying `chat` means either "the
heuristic was certain" or "it guessed something at 0.2 and docs/12's confidence
gate fell back to chat". Those are opposite facts about the same field, and the
difference is exactly the label quality a learned router (M19.3) would train on.
Every request was discarding it, unrecoverably.
Migration 0011 adds `confidence real` and `trusted boolean`, nullable and with
no DEFAULT — rows written before this genuinely have no value, and 0.0 would
read as "the classifier was certain of nothing" rather than "nobody asked". A
gap is visibly a gap. Both columns are content-free like the rest of the table
(docs/20 T5): a float and a bool say nothing about what the user typed.
Also fixed, because I was editing the line it sits on: the comment there claimed
"A declared class wins; otherwise classify". It does not — `CallMeta.task` is
never read, so a caller declaring `Code` gets whatever the heuristic guesses.
docs/12 says "Callers that know, say", so the spec and the code disagree and the
spec is not the one that is wrong. Latent (no ingress populates the field), and
honouring it changes routing, so it is a question in RUN-REPORT.md and its own
commit — not a silent behaviour change smuggled in here. The false comment is
replaced by an accurate one rather than left to mislead.
The round-trip test uses `routes::insert` rather than `PgRouteAudit::record`,
because the trait impl detaches the write onto a background task so inference
never waits on the database. Reading straight after `record` races the spawn —
it did, once, with `RowNotFound`. What is under test is the columns, not the
detachment.
Verified: integration lane 149/149 twice; unbinding the two columns turns the
new test red and nothing else. fmt, clippy -D warnings, 1114 workspace tests,
schemas, ts-sdk, sbom, deny.
Claude-Session: https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The brief's task 7 was "wire
ShadowClassifierbehind a flag — zero production call sites". Two things turned out otherwise.The seam already exists
GatewayBuilder::classifier()takes anArc<dyn Classifier>, andShadowClassifieris aClassifierthat returns the incumbent's answer by construction. Wiring it is one line.The blocker isn't a missing flag — there is no candidate classifier to shadow, and won't be until M19.3. A flag guarding a comparison against nothing would be ceremony.
The stated goal had a real gap, upstream of shadow mode
Gateway::resolveread:Both discarded. So
route_decisionsrecorded the class the router used and nothing about how it got there. A row sayingchatmeans either:chatOpposite facts about the same field — and that difference is exactly the label quality a learned router (M19.3) trains on. Every request was discarding it, unrecoverably. That is "accumulate routing evidence before a model exists."
Migration
0011addsconfidence realandtrusted boolean: nullable, no DEFAULT. Rows written before this genuinely have no value, and0.0would read as "the classifier was certain of nothing" rather than "nobody asked". A gap should look like a gap. Both are content-free like the rest of the table (docs/20 T5) — a float and a bool say nothing about what the user typed.A third finding, recorded not fixed
The comment on that same line claimed "A declared class wins; otherwise classify." It doesn't —
CallMeta.taskis never read, so a caller declaringCodegets whatever the heuristic guesses. docs/12 says "Callers that know, say", so spec and code disagree and the spec isn't the wrong one.Latent (no ingress populates the field), and honouring it changes routing — so it's a question in
RUN-REPORT.mdand its own commit, not a silent behaviour change smuggled in here. The false comment is replaced with an accurate one rather than left to mislead.This is the third field this session that reads like a control and controls nothing, after
plugin.toml'snet:andNetPolicy::allow. Three instances is a pattern.On the test
It uses
routes::insert, notPgRouteAudit::record— the trait impl detaches the write onto a background task so inference never waits on the database. Reading straight afterrecordraces the spawn; it did, once, withRowNotFound. What's under test is the columns, not the detachment.Verification
-D warnings· 1114 workspace tests · schemas · ts-sdk · sbom · deny.https://claude.ai/code/session_017kFpYDqvz6sKGSkM4YKaRf