Skip to content

fix(client): a slow server no longer mints a new relay credential on every reconnect - #13687

Open
t3dotgg wants to merge 13 commits into
mainfrom
t3code/relay-token-timeout
Open

t3dotgg wants to merge 13 commits into
mainfrom
t3code/relay-token-timeout

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

When a relay client reconnects to a slow server, it treats a timed-out websocket-ticket request as a rejected DPoP token and mints a new credential. The mint (relay t3MintCredential, descriptor fetch, /oauth/token) runs on that same slow server, so each reconnect attempt adds a new auth session there, about twice a minute, and the client still does not connect. This came from a long-uptime slowdown report (0.0.40, 9 days, about 5,000 threads) where slow requests had a p50 of 2.9 s against the 3 s cached-ticket budget.

Fix

All in authorizeDpop in packages/client-runtime/src/authorization/service.ts, which web, desktop, and mobile share:

  • A cached ticket request that times out at 3 s is retried once with the same token and a 7 s budget. The ticket step still ends by 10 s, inside the 15 s setup deadline.
  • If the retry also times out, the token is kept and the attempt fails as a transient timeout, without the network hint. The supervisor backs off and tries the same token again.
  • The third such timeout in a row drops the token and asks the relay again, as before. A success resets the count.
  • A newly minted token whose ticket request times out is kept for the next attempt.
  • Auth-invalid, 5xx, and network errors still drop the token and ask the relay in the same attempt.

#4878 removed an older counter that held back every failure. This one counts only timeouts after the retry, so a lost request still recovers in about 3 s.

Tradeoff

A cached endpoint that accepts connections but never answers is now left after about 40 s (the third attempt) instead of about 3 s. I do not expect this today. Relay connect only returns managed Cloudflare tunnel endpoints, and their hostname is derived from the environment (infra/relay/src/environments/ManagedEndpointProvider.ts), so a new credential goes to the same stalled server anyway. A removed hostname or tunnel fails fast and still moves to the relay's endpoint in the first attempt.

Merge order

#5463 also edits service.ts (createDpopSocketUrl and a new authorizeDpopDirect). The behaviors do not conflict. Whichever PR lands second needs a rebase.

Verification

  • vp test run packages/client-runtime/src/authorization/ packages/client-runtime/src/connection/: 141 passed. With main's service.ts, the 5 new tests fail.
  • The supervisor tests run the real supervisor with the real authorization service: healthy server, lost request, stalled server, third timeout, and moved endpoint.
  • vp lint and vp fmt on the changed files, and vp run --filter @t3tools/client-runtime typecheck.

Related: #13695 prunes the server-side DPoP replay files that each mint adds.

Made by Claude Opus 5.5 (1M context) in Claude Code, running in T3 Code.

🤖 Generated with Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Sep 25, 2026
Comment thread packages/client-runtime/src/authorization/service.ts Outdated
@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: pingdotgg/t3code/.coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 45e2fdde-41b6-42d4-b002-198b8e5391e0

📥 Commits

Reviewing files that changed from the base of the PR and between 4f253d6 and ebf9c37.

📒 Files selected for processing (2)
  • packages/client-runtime/src/authorization/layer.test.ts
  • packages/client-runtime/src/authorization/service.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

Authorization tracks consecutive websocket-ticket timeouts for cached tokens. The first two timeouts retain the token; the third triggers renewal. A successful ticket request resets the timeout count. Timeout tests cover cached and refreshed tokens.

Changes

Websocket ticket authorization

Layer / File(s) Summary
Cached-token timeout handling
packages/client-runtime/src/authorization/service.ts
authorizeDpop counts consecutive cached-token ticket timeouts. It retains the token for the first two timeouts and renews after the third. A successful request resets the count. Timeout failures do not remove the selected token.
Timeout test harness and cases
packages/client-runtime/src/authorization/layer.test.ts
The harness supports stalled fetch responses and signals when they occur. Tests cover token retention after timeouts, renewal after three consecutive cached-token timeouts, and count reset after success. The cached-endpoint test description now refers to a server error.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to ebf9c

The inspected timeout path retains the token for a later attempt. No issue requiring a fix before merge was established.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main fix: preventing repeated relay-credential minting when reconnects encounter a slow server.
Description check ✅ Passed The description explains the problem, implementation, tradeoff, merge considerations, and verification results. It omits the template checklist and uses a Fix section instead of explicit What Changed …
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

Comment thread packages/client-runtime/src/authorization/service.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This focused fix changes the shared production DPoP authorization path, including timeout retries, token retention, and relay credential minting. Because it modifies the authorization package, human review is required despite the limited scope and added test coverage.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.5 KiB 13.5 KiB +26 B (+0.2%) 15.1 KiB ✅
Codex Thread snapshot wire 7.1 KiB 7.1 KiB +9 B (+0.1%) 7.3 KiB ✅
Codex Live turn WebSocket wire 6.4 KiB 6.5 KiB +17 B (+0.3%) 7.8 KiB ✅
Codex Live turn WebSocket decoded 56.2 KiB 56.3 KiB +44 B (+0.1%) 66.4 KiB ✅
Codex Live turn messages 9 10 +1 (+11.1%) 21 ✅
Claude Total thread wire 13.5 KiB 13.5 KiB +8 B (+0.1%) 15.1 KiB ✅
Claude Thread snapshot wire 7.1 KiB 7.1 KiB +4 B (+0.1%) 7.3 KiB ✅
Claude Live turn WebSocket wire 6.4 KiB 6.4 KiB +4 B (+0.1%) 7.8 KiB ✅
Claude Live turn WebSocket decoded 57.0 KiB 57.0 KiB 0 B (0.0%) 66.4 KiB ✅
Claude Live turn messages 9 9 0 (0.0%) 21 ✅

Baseline: 295d7cb · PR result: b0d71b1 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 114.0 KiB
  • Claude decoded thread snapshot: 114.7 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@t3dotgg
t3dotgg force-pushed the t3code/relay-token-timeout branch 3 times, most recently from 4f253d6 to ebf9c37 Compare September 26, 2026 02:37
@github-actions github-actions Bot added size:M 30-99 changed lines (additions + deletions). and removed size:S 10-29 changed lines (additions + deletions). labels Sep 26, 2026
@t3dotgg
t3dotgg force-pushed the t3code/relay-token-timeout branch 5 times, most recently from 20651c7 to 8d80247 Compare September 26, 2026 07:23
Comment thread packages/client-runtime/src/authorization/service.ts
t3dotgg and others added 12 commits September 26, 2026 01:42
…every reconnect

When the websocket-ticket request with a cached, still-valid DPoP token
timed out after 3 s, the client treated the token as rejected. It then
minted a new relay credential and auth session. A freshly minted token
was also thrown away when its ticket request timed out. During a server
stall this happened on every reconnect.

Now a cached-token ticket timeout retries the same token once with the
default 10 s budget (13 s total, inside the 15 s establishment deadline).
If that also times out, the attempt fails as a transient timeout and the
token stays cached. A fresh token is also kept after a ticket timeout.
Auth-invalid, 5xx, and network failures still re-bootstrap and evict, so
moved-tunnel recovery is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cached-token ticket retry used the default 10 s budget, so the ticket
step could take 13 s. After it, the supervisor's 15 s setup race still has
to fetch the descriptor, open the websocket, and load the initial config.
On a server slow enough to need the retry, 2 s is not enough, so the
attempt timed out late anyway.

The retry now gets 7 s, so the whole ticket step ends by 10 s, the same as
one default request. At least 5 s of the setup deadline is left. A retry
timeout still fails as transient and keeps the token, so nothing is minted.

Also use Effect.catchTags for the known timeout tag, per the Effect service
conventions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Correct the setup-deadline note, name the hung-endpoint tradeoff, and
rename the cached-endpoint test to match what it now covers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A ticket timeout no longer mints a new credential, so the 3 s cached budget
and the 7 s retry only split one 10 s wait into two requests. The cached
token now gets the default 10 s budget once. A timeout fails as transient
and keeps the token, so the next attempt reuses it. Auth rejections,
server errors, and network errors still ask the relay for a new token.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…n a row

A relay endpoint that accepts the ticket request and never answers kept
the cached token until it expired, up to an hour. Count cached-token
ticket timeouts per environment and token. The third timeout in a row
drops the token and mints a new one through the relay. A success resets
the count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A supervisor test with the real authorization service now shows that a
stalled ticket request keeps the setup waiting for the full 10 s budget,
backs off, and reconnects on the stored token with no relay call. A second
test shows that a cached endpoint that fails at the network level moves to
the relay's endpoint in the first attempt. The shared relay setup moves
into one helper that the existing HTTP refresh test also uses, and the
service-level stalled-token test is removed because the supervisor test
covers it.

The timeout comment now says why a new credential cannot help: the relay
returns the same hostname for an environment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… token

With one 10 s budget, a single lost ticket request on a healthy server
took about 13 s to recover (10 s timeout, 3 s backoff, new attempt).
On main it took about 3 s plus a mint.

The cached ticket request gets 3 s again, and a timeout retries the
same token once with 7 s, so the ticket step still ends by 10 s. A lost
request now recovers in about 3 s with no mint. If the retry also times
out, the token is kept as before, and the third such attempt in a row
asks the relay again.

Adds supervisor tests for the lost request and for the third-timeout
path, where the mint runs past the setup deadline and the next attempt
connects with the minted token after one relay call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The HTTP refresh supervisor test now checks that a healthy server connects
in the first attempt with the stored token, one ticket request, and no
relay call. The timeout count comment notes that it lives as long as the
service, so a foreground resume does not reset it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The supervisor test already covers the three-timeout limit. The service
test now only shows that a retry that succeeds resets the count, with a
script built from named attempts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s network

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@t3dotgg
t3dotgg force-pushed the t3code/relay-token-timeout branch from 8d80247 to b0d71b1 Compare September 26, 2026 08:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant