Skip to content

test(daemon): scrub every provider key var, not just the auto rung's - #1310

Merged
edwin-zvs merged 1 commit into
mainfrom
fix-availability-scrub
Aug 25, 2026
Merged

edwin-zvs merged 1 commit into
mainfrom
fix-availability-scrub

Conversation

@edwin-zvs

Copy link
Copy Markdown
Contributor

The bug

availability::tests::auto_unavailable_when_only_a_subscription_credential_exists scrubs the provider key env vars before asserting the auto rung is unavailable, so that the fixture — not the developer's environment — decides the result.

That list was scoped to "every direct-API-key var the auto rung counts". When OpenRouter joined the ladder in #1303, the list didn't follow:

let auto_available = anthropic.available
    || openai.available
    || gemini.available
    || meta.available
    || deepseek_key.available
    || openrouter_key.available;   // ← added by #1303, never scrubbed

So on any machine exporting OPENROUTER_API_KEY, the developer's own key satisfied the rung and the assertion failed — deterministically, not intermittently, and only locally. CI has no such key, so it stayed green there. The practical cost is that it reads as a regression from whatever you just changed; it surfaced on both of today's unrelated TUI PRs (#1308, #1309).

The test's own comment predicted this exact failure: "A var missing from this list makes the test pass or fail on the developer's own exported keys rather than on the fixture."

The fix

Track provider existence rather than ladder membership. A new provider now only has to be a provider to get scrubbed, whether or not it's in the auto ladder yet, and scrubbing a var the ladder ignores costs nothing.

That also picks up GROK_API_KEY / XAI_API_KEY, missing for the same reason. Grok isn't in the ladder today, so they weren't causing failures — they'd have started the day it joined.

This is why the narrower rule was the wrong one to encode: it required the list to be updated in lockstep with a different list, and nothing enforced that.

Verification

The test now gives the same answer either way, where before the first case failed:

with OPENROUTER_API_KEY set    →  ok. 1 passed
with OPENROUTER_API_KEY unset  →  ok. 1 passed

Full workspace suite green, daemon included (759 daemon tests, 1459 CLI).

Binaries

Test-only change to crates/daemon — no behavior change, nothing to run.

auto_unavailable_when_only_a_subscription_credential_exists scrubs the
provider key vars before asserting the auto rung is unavailable, so the
fixture rather than the developer's environment decides the result. The
list was scoped to "vars the auto rung counts", and when OpenRouter
joined that ladder in #1303 the list did not follow.

The effect was a test that failed deterministically on any machine
exporting OPENROUTER_API_KEY while staying green in CI, which has no
such key — so it read as a regression from whatever the developer had
just changed.

Track provider existence instead of ladder membership: a new provider
now only has to be a provider to be scrubbed, whether or not it is in
the ladder yet, and scrubbing a var the ladder ignores costs nothing.
Also picks up GROK_API_KEY / XAI_API_KEY, which were missing for the
same reason and would have bitten the day grok joined the ladder.
@edwin-zvs
edwin-zvs merged commit e177457 into main Aug 25, 2026
1 check passed
@edwin-zvs
edwin-zvs deleted the fix-availability-scrub branch August 25, 2026 19:02
@edwin-zvs edwin-zvs mentioned this pull request Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant