fix(connect): remove tunnels after hosts go offline

[t3dotgg]

Cloudflare tunnels stayed allocated after their host went offline. Closed and down tunnels cost the same as active tunnels, so this left an increasing number of paid resources with no useful connection.

This change adds managed tunnel cleanup, with RELAY_TUNNEL_CLEANUP_MODE=off as the default. For existing allocation records, the relay only deletes tunnels whose hosts registered recovery with the public key on their current environment link. It skips legacy and incomplete allocations. Expired same-namespace tunnels with no allocation record are orphans; the relay rechecks their Cloudflare status before deletion. Each sweep has page, deletion, and time limits, and it stops the deletion loop when Cloudflare returns a structured rate-limit response.

When cleanup removes a tunnel, the host provisions a replacement and keeps the same public hostname. The endpoint URL does not change, so web, desktop, and mobile clients do not need new bindings. The host also watches repeated Cloudflare authorization failures, which covers a connector that stays alive after laptop sleep and wake.

Healthy startup does not provision a tunnel. A confirmed connector starts from its stored config when the saved loopback origin still matches. The host then registers recovery with the relay. Cloudflare ingress is updated only when the local origin changed. Registration retries transient failures with capped exponential backoff, while local connector startup retries do not repeat relay or Cloudflare work.

A valid legacy connector config without a stored tunnel ID requests recovery directly. The relay uses its allocation record to return a complete replacement config, which the host persists before starting the connector. The host does not parse the opaque connector token.

The rollout is relay first:

1. Deploy the relay migration and recovery endpoints with cleanup set to off.
2. Release the server through CLI and desktop builds so current hosts can register recovery.
3. Run dry-run and inspect cleanup counters across several sweeps.
4. Run the disposable relay stage and test Cloudflare account canary in the operations guide.
5. Enable cleanup only after the host recovers without a server restart.

The live Cloudflare sleep and wake test has not been run. Production deletion must stay disabled with off or dry-run until the disposable stage canary passes. For rollback, disable cleanup before downgrading any host, keep the recovery endpoints deployed while updated hosts remain in use, and retain the additive nullable migration.

The focused server and relay tests cover registration, origin changes, connector recovery, unlink races, generation checks, cleanup eligibility, fairness, deletion caps, and rate limits. Scoped lint, formatting, and typechecks also pass.

Authored with GPT-5.6 Sol in Codex.

---

Note

High Risk
Touches cloud link persistence, tunnel lifecycle, relay allocation concurrency (generation), and optional production tunnel deletion; mis-timed cleanup rollout or race bugs could break remote reachability for linked environments.

Overview
Adds managed tunnel recovery so hosts can register loopback origin with the relay, replace tunnels deleted while offline, and auto-recover when cloudflared reports repeated Cloudflare authorization failures—without changing the public hostname.

The server now persists a confirmed-origin marker (CLOUD_ENDPOINT_CONFIRMED_ORIGIN), starts the connector on boot only when that marker matches the current config and port, and gates relay-config / link flows on successful tunnel recovery registration (with retries, longer provision timeouts, and serialized link-state updates). ManagedEndpointRuntime exposes recovery request streams, classifies retryable spawn failures, and launches cloudflared with explicit logging flags.

On the relay, a migration adds recovery/origin/generation fields to allocations; RELAY_TUNNEL_CLEANUP_MODE (off / dry-run / enabled) controls periodic inactive-tunnel cleanup for hosts that registered recovery. Deploy workflow and docs describe rollout order (relay first, server builds, then dry-run → enabled).

^{Reviewed by Cursor Bugbot for commit 185a29f. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add managed tunnel reaper and recovery flow for offline hosts

• Adds ManagedEndpointReaper service that periodically sweeps and deletes orphaned managed tunnels, controlled by RELAY_TUNNEL_CLEANUP_MODE (off | dry-run | enabled, defaults to off) in worker.ts
• Introduces a managed tunnel recovery flow: the relay exposes registerManagedEndpointRecovery and recoverManagedEndpoint API endpoints backed by signed JWT proofs in Api.ts; the server runtime emits recovery requests when repeated tunnel authorization rejections are observed and consumes them to provision replacement tunnels
• Replaces updatedAt-based concurrency in ManagedEndpointAllocations with a numeric generation column; mutations now require a generation match and return the previous generation or null; adds withClaimedTunnel for transactional row locking
• Server startup in server.ts now registers managed tunnel recovery with retry, starts tunnels only when a confirmed-origin marker matches, and reconciles desired links only when still desired
• DB migration migration.sql adds origin, generation, recovery_enabled_at, and recovery_environment_public_key columns to relay_managed_endpoint_allocations
• Risk: ManagedEndpointAllocations service method signatures and return types change broadly (recordTunnel, recordDns, markReady, claimRelease, claimDeprovision, removeClaimed now return Effect<number | null> or Effect<boolean> and require generation); release on ManagedEndpointProvider now returns Effect<boolean> instead of void; all tunnel client ops in layerCloudflareBindings are capped at 8s timeout

^{Macroscope summarized 185a29f.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a03ad921-65b6-4164-8f3e-7cf926d4f4f3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	13.3 KiB	13.3 KiB	+15 B (+0.1%)	15.1 KiB	✅
Codex	Thread snapshot wire	6.9 KiB	6.9 KiB	+4 B (+0.1%)	7.3 KiB	✅
Codex	Live turn WebSocket wire	6.4 KiB	6.4 KiB	+11 B (+0.2%)	7.8 KiB	✅
Codex	Live turn WebSocket decoded	55.6 KiB	55.6 KiB	0 B (0.0%)	66.4 KiB	✅
Codex	Live turn messages	11	11	0 (0.0%)	21	✅
Claude	Total thread wire	13.4 KiB	13.4 KiB	+10 B (+0.1%)	15.1 KiB	✅
Claude	Thread snapshot wire	6.9 KiB	6.9 KiB	−6 B (−0.1%)	7.3 KiB	✅
Claude	Live turn WebSocket wire	6.5 KiB	6.5 KiB	+16 B (+0.2%)	7.8 KiB	✅
Claude	Live turn WebSocket decoded	56.4 KiB	56.4 KiB	0 B (0.0%)	66.4 KiB	✅
Claude	Live turn messages	11	11	0 (0.0%)	21	✅

Baseline: 8f49132 · PR result: 185a29f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 109.4 KiB
• Claude decoded thread snapshot: 110.1 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This is a substantial cross-component capability adding authenticated tunnel recovery, Cloudflare resource deletion, schema/concurrency changes, and server lifecycle behavior. The default-off cleanup mode limits immediate exposure, but the production networking, irreversible resource-management, and authentication-sensitive changes still require human review.

^{You can add or adjust custom eligibility rules. Learn more.}

[macroscopeapp]

Effect service conventions: four findings, all in newly added error-handling code (three Effect.catchTag usages and one startup Effect.catchCause). Details inline.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

One convention finding on the new retry predicate. Previously reported Effect.catchTag usages in ManagedEndpointReaper.ts, ManagedEndpointProvider.ts and the Effect.catchCause in server.ts are resolved in this head commit.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Reviewed the Effect service changes in this PR against the repository's service conventions. One convention violation found: a newly added Effect.catchTag in apps/server/src/cloud/http.ts. Everything else (the new ManagedEndpointReaper service definition, dependency acquisition via the environment, make/layer exports, structural failure discriminator on the runtime status, and interruption-preserving Cause.hasInterrupts handling in server.ts) matches the conventions.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6a0992d. Configure here.}

[t3dotgg]

Note

🤖 GPT-5.6 Sol responding on behalf of Theo

The verified catchTag convention finding is fixed in e02577fd8. The handler now uses one Effect.catchTags object. The SchemaError and PlatformError mappings and messages are unchanged.

The focused HTTP suite passes with 33 tests. Server typecheck and scoped lint also pass. The current PR head is dcfdd3d78.

[t3dotgg]

Note

🤖 GPT-5.6 Sol responding on behalf of Theo

Fixed the five verified issues from this review round:

• The two reports for live relay-config recovery and its duplicate now queue the returned recovery config. The route still returns the existing unavailable response and keeps the connector stopped until recovery starts it.
• Explicit default ports now pass through one structured origin parser used by registration, confirmed startup, recovery, and CLI reconciliation. HTTP port 80 and HTTPS port 443 are accepted. Credentials, query strings, fragments, non-root paths, and unsupported protocols are rejected.
• Legacy token-only connector configs now request recovery without a registration request or connector start. Recovery uses the relay allocation and does not parse the opaque token.
• Provisioning timeout budgets now allow two minutes for CLI link creation and managed tunnel recovery. Registration keeps the short 10-second timeout so relay failures retry promptly.
• The cleanup ownership documentation now qualifies legacy protection to existing allocation records and documents rechecked deletion of expired same-namespace orphan tunnels. Incomplete and mismatched allocations remain skipped.

The focused HTTP suite passes with 41 tests. Server typecheck, scoped lint, formatting, and diff checks also pass. Route integration coverage was added for the queued recovery response.

[t3dotgg]

Note

🤖 GPT-6 Astra (preview) responding on behalf of Theo

This was closed as part of an automated cleanup pass. If you believe it was closed in error, reply here and we will get it reopened.

Closing in favor of #9386. It carries forward the original commits and adds the recovery-state fix.

The remaining review and rollout checks belong there. We do not need two open PRs for this work.