Repository navigation
test(daemon): scrub every provider key var, not just the auto rung's - #1310
Merged
Merged
Conversation
auto_unavailable_when_only_a_subscription_credential_exists scrubs the provider key vars before asserting the auto rung is unavailable, so the fixture rather than the developer's environment decides the result. The list was scoped to "vars the auto rung counts", and when OpenRouter joined that ladder in #1303 the list did not follow. The effect was a test that failed deterministically on any machine exporting OPENROUTER_API_KEY while staying green in CI, which has no such key — so it read as a regression from whatever the developer had just changed. Track provider existence instead of ladder membership: a new provider now only has to be a provider to be scrubbed, whether or not it is in the ladder yet, and scrubbing a var the ladder ignores costs nothing. Also picks up GROK_API_KEY / XAI_API_KEY, which were missing for the same reason and would have bitten the day grok joined the ladder.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
availability::tests::auto_unavailable_when_only_a_subscription_credential_existsscrubs the provider key env vars before asserting theautorung is unavailable, so that the fixture — not the developer's environment — decides the result.That list was scoped to "every direct-API-key var the auto rung counts". When OpenRouter joined the ladder in #1303, the list didn't follow:
So on any machine exporting
OPENROUTER_API_KEY, the developer's own key satisfied the rung and the assertion failed — deterministically, not intermittently, and only locally. CI has no such key, so it stayed green there. The practical cost is that it reads as a regression from whatever you just changed; it surfaced on both of today's unrelated TUI PRs (#1308, #1309).The test's own comment predicted this exact failure: "A var missing from this list makes the test pass or fail on the developer's own exported keys rather than on the fixture."
The fix
Track provider existence rather than ladder membership. A new provider now only has to be a provider to get scrubbed, whether or not it's in the auto ladder yet, and scrubbing a var the ladder ignores costs nothing.
That also picks up
GROK_API_KEY/XAI_API_KEY, missing for the same reason. Grok isn't in the ladder today, so they weren't causing failures — they'd have started the day it joined.This is why the narrower rule was the wrong one to encode: it required the list to be updated in lockstep with a different list, and nothing enforced that.
Verification
The test now gives the same answer either way, where before the first case failed:
Full workspace suite green, daemon included (759 daemon tests, 1459 CLI).
Binaries
Test-only change to
crates/daemon— no behavior change, nothing to run.