Skip to content

fix(client-runtime): a slow first config snapshot no longer drops a working connection - #16913

Closed
dione24 wants to merge 1 commit into
pingdotgg:mainfrom
dione24:fix/client-runtime-first-snapshot-timeout
Closed

dione24 wants to merge 1 commit into
pingdotgg:mainfrom
dione24:fix/client-runtime-first-snapshot-timeout

Conversation

@dione24

@dione24 dione24 commented Oct 7, 2026

Copy link
Copy Markdown

Problem

A client connecting to a server whose first config snapshot takes longer than 15 s never finishes connecting. The supervisor tears down the working socket at 15 s, waits 3s/4s/8s/16s, and starts subscribeServerConfig from scratch, so the snapshot never gets to complete until the server warms up. The 15 s budget (CONNECTION_ESTABLISHMENT_TIMEOUT in packages/client-runtime/src/connection/supervisor.ts) covers credentials, session creation, socket open and the first snapshot, because ConnectionDriver.connect only returns at session.ready (packages/client-runtime/src/rpc/session.ts).

Change

The supervisor keeps the 15 s (per route) establishment budget. If the attempt has reached the synchronizing stage when that budget runs out, it now gets an additional FIRST_SNAPSHOT_TIMEOUT (2 minutes) before it is recorded as a setup timeout. The supervisor tracks this with a synchronizing flag set from the driver's progress reports, so the driver, RpcSession and the public types are untouched, as are the backoff policy and the failure shape.

Liveness is not weakened: a socket that never opens still fails at the socket open timeout, and a socket that stops answering pings fails session.ready via the existing ping-timeout disconnect, which reconnects immediately. The 2 minute bound only applies to an open, live connection whose snapshot handler stays silent.

Scope and approval

Fixes #14516. The issue is a triaged bug and the fix is confined to the shared supervisor timeout.

Verification

  • supervisor.test.ts, using TestClock:
    • the existing "never becomes ready" test now expects the session to survive past 15 s and time out at the new bound;
    • a snapshot that takes 40 s connects on attempt 1 without releasing the session;
    • a synchronizing session that fails at 20 s reconnects immediately instead of waiting for the bound;
    • the existing "setup times out" test (prepare never completes) still fails at 15 s.
  • The three new/changed tests fail on main and pass with the fix.
  • vp test run in packages/client-runtime: 128 files, 2091 tests pass. tsc --noEmit, vp lint and vp fmt --check on the changed directory pass.
  • Not checked: a real client against a server with a slow snapshot.

🤖 Generated with Claude Code (Claude Sonnet 5.5)

…orking connection

The supervisor raced every connect attempt against one 15 s budget that also
covered the first config snapshot. Once the session is synchronizing, the
attempt now gets a separate 2 minute bound for the snapshot. Dead sockets
still fail early through the socket open timeout and ping check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Oct 7, 2026
@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 24247f13-4220-4f07-a1e9-163acd45b6bd
📥 Commits

Reviewing files that changed from the base of the PR and between 9f48092 and 991d877.

📒 Files selected for processing (2)
  • packages/client-runtime/src/connection/supervisor.test.ts
  • packages/client-runtime/src/connection/supervisor.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The supervisor now allows a synchronizing connection attempt additional time to receive its first config snapshot after the route-scaled setup timeout. Tests cover delayed readiness, expiration of the extended timeout, and a transport failure during synchronization.

Changes

First config snapshot wait

Layer / File(s) Summary
Synchronization timeout and outcomes
packages/client-runtime/src/connection/supervisor.ts, packages/client-runtime/src/connection/supervisor.test.ts
The supervisor tracks whether an attempt is synchronizing and allows up to two additional minutes after the route-scaled setup timeout. Tests verify delayed readiness, timeout after the extension, and release after a transport failure.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to 991d8

Clients connecting to a server with a slow first config snapshot get up to two extra minutes before the attempt times out, instead of repeatedly dropping a working socket. No merge-blocking risk was found.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 991d8

Identity checks and connection cleanup remain intact. However, a failed route can leave the snapshot-wait state active during fallback preparation, allowing the extra two-minute wait after the original connection has already closed. The demonstrated impact is delayed connection recovery, not increased access or privileges.

Retained concerns

  • Low · reliability · inferred: Snapshot grace is not owned by the active route. If one route reaches synchronization and then fails readiness, its session closes but synchronizing remains true during the next route's checks or preparation. If that work crosses the original establishment budget, it can receive two additional minutes despite having no live session. This weakens setup-timeout containment across fallback transitions; independent request bounds and explicit cancellation limit the impact.
Security review details

Security Blast Radius

  • inferred — The changed behavior applies independently to environment connections using this shared library. A slow or failing configured endpoint can delay readiness or recovery for its connecting clients. The inspected change does not expand connection authority to additional environments.

Security Findings and Attack Paths

  • inferred — The supported adverse path is readiness failure on a synchronizing route, followed by slow fallback preparation while stale synchronization state grants additional time. This is a bounded recovery-delay concern, not a demonstrated authentication bypass, credential disclosure, or indefinite retention of the failed session.

Trust Boundaries and Controls

  • observed — Socket-open timeout remains configured, ping-timeout disconnects fail readiness, and the initial snapshot is checked against the expected environment identity. The driver still performs preparation before creating a session and returning a ready lease.

Resilience and Maintainability Implications

  • observed — Explicit disconnect, retry, offline changes, and relevant wakeups can interrupt establishment during grace. Attempts remain scoped and clear published leases on completion. SSH preparation delegates to an injected capability whose concrete production timeout and cancellation guarantees were not verified, limiting confidence in end-to-end cleanup timing.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix: preserving a working connection when the first config snapshot is slow.
Description check ✅ Passed The description covers the problem, expected behavior, implementation, scope, linked issue, approval basis, focused verification, test results, and unverified real-client testing. It matches the requi…
Linked Issues check ✅ Passed Issue #14516 requires the client to keep an open, responding connection while the first config snapshot completes. The PR extends the setup timeout only after the driver reports the synchronizing st…
Out of Scope Changes check ✅ Passed The reported changes are limited to packages/client-runtime/src/connection/supervisor.ts and its tests. The implementation changes supervisor timeout handling for the slow first snapshot described b…
Approvability ✅ Passed Passes Approvability. The authoritative PR diff changes only packages/client-runtime/src/connection/supervisor.ts and its test. It is a focused connection-timeout bug fix, not a new workflow or subs…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 11, 2026 — with ChatGPT Codex Connector
if ("prepared" in progress) {
yield* SubscriptionRef.set(prepared, Option.some(progress.prepared));
}
yield* Ref.set(synchronizing, progress.stage === "synchronizing");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High connection/supervisor.ts:427

A setup attempt can wait an extra firstSnapshotTimeout and delay retry even though its current route is still preparing credentials. synchronizing remains true if an earlier route reached that stage and its session failed, because driver.ts reports preparing only once before trying routes; reset the flag when each route begins preparation.

🤖 Copy this AI Prompt to have your agent fix this:
In file @packages/client-runtime/src/connection/supervisor.ts around line 427:

A setup attempt can wait an extra `firstSnapshotTimeout` and delay retry even though its current route is still preparing credentials. `synchronizing` remains true if an earlier route reached that stage and its session failed, because `driver.ts` reports `preparing` only once before trying routes; reset the flag when each route begins preparation.

@macroscopeapp

macroscopeapp Bot commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Would Approve

Macroscope's review found this PR approvable — This is a focused client-runtime bug fix with targeted tests that preserves existing setup timeouts while allowing a live connection more time to receive its first configuration snapshot. A shared synchronizing flag can remain set after one route fails, potentially making a later credentials-preparation route wait an unnecessary two minutes.

Not approved because:

  • 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

@juliusmarminge

Copy link
Copy Markdown
Member

Note

Grok responding on behalf of Julius.

Thanks for working on this! #14516 was fixed by #14518, which just merged and takes the same approach (extra budget once an attempt reaches synchronizing), so I'm closing this as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:S 10-29 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Client: a slow first config snapshot makes clients drop working connections and reconnect-loop

2 participants