Skip to content

fix(server): unload Codex threads left idle on the shared app-server - #16917

Merged
juliusmarminge merged 2 commits into
pingdotgg:mainfrom
RhysSullivan:fix/codex-idle-thread-unload
Oct 7, 2026
Merged

juliusmarminge merged 2 commits into
pingdotgg:mainfrom
RhysSullivan:fix/codex-idle-thread-unload

Conversation

@RhysSullivan

Copy link
Copy Markdown
Contributor

Follow-up to #13331 (item 4) and #14187.

Problem

All Codex threads share one codex app-server, and every loaded native thread runs its own copy of each configured stdio MCP server. Since #14187, a thread is unloaded (thread/unsubscribe) when it detaches: settle, archive, delete, selection change, or handoff. A thread that finishes its turn and is left alone never detaches, so its native thread and MCP servers stay up as long as the shared app-server does. The app-server itself is only released after 30 idle minutes across all threads, which a busy environment rarely reaches.

Delegated tasks hit this constantly. Each delegate_task child is a new thread that runs one turn, completes, and is never settled. On one machine over about 22 hours, T3's provider logs showed 89 thread/start against 5 thread/unsubscribe, and the shared app-server had 1,244 descendant processes using about 28 GB (about 19 MCP servers per native thread, 4 of them stdio).

Fix

ProviderSessionManager now applies its existing idle timeout to each thread on a shared runtime. When a thread's turn ends (turn.terminal, or a failed start) and it has no other busy turn, a per-thread timer starts. If no turn starts within idleTimeoutMs, the thread is unloaded through the same unloadThread (thread/unsubscribe) that detach uses, and its loaded mark is cleared, so the next turn's resume calls thread/resume and reloads it.

  • A resume or turn start cancels the pending timer before it checks whether the thread is loaded, so an unload cannot land between a resume that skipped reloading and the turn that relies on it. The unload runs under the per-thread attachment lock, like detach.
  • A thread with pending background work (hasPendingBackgroundWorkForThread) stays loaded, because unloading stops Codex background terminals. The timer re-checks after each further idle window.
  • Detach drops the timer and unloads the thread itself, as before. Runtimes without unloadThread, and exclusive runtimes, are unaffected.

Evidence

Codex contract. The script below drives codex app-server (codex-cli 0.157.0) directly, with an isolated CODEX_HOME, one stdio MCP server, and thread_unload_delay_secs = 2:

$ node repro.mjs
after 3x thread/start: 3 MCP server processes
thread/unsubscribe -> unsubscribed   (x3)
7s later (unsubscribed): 0 MCP server processes

$ UNSUBSCRIBE=0 node repro.mjs
after 3x thread/start: 3 MCP server processes
7s later (still subscribed): 3 MCP server processes

MCP servers exit if and only if the client unsubscribes, so the cleanup has to come from T3.

Through T3 Code (Nightly 0.0.46-nightly.20261005.2676, macOS), counting descendants of T3's shared app-server:

19:16     descendants: 0
19:17     delegate_task to Codex ("Reply with exactly OK"), completed in ~5s
19:19     descendants: 8      # the child's MCP servers; no thread/unsubscribe sent
19:19:56  settle the child -> T3 sends thread/unsubscribe
19:20:56  Codex emits thread/closed
19:21     descendants: 0

Tests. There are two new tests in ProviderSessionManager.test.ts. Both fail on main and pass with this change.

  • unloads a shared-runtime thread left idle, and reloads it on its next turn: a follow-up turn inside the timeout keeps the thread loaded. A full idle timeout after its last turn unloads it, while another thread's running turn keeps the runtime up. The next resume reaches the provider again.
  • keeps an idle shared-runtime thread loaded while its background work runs: no unload while the per-thread probe reports pending work; unloaded one idle window after it clears.

Commands run:

  • vp test run on ProviderSessionManager.test.ts, CodexAdapterV2.test.ts, ProviderTurnStartService.test.ts, ProviderTurnControlService.test.ts, runtimeLayer.test.ts, EffectWorker.test.ts, BackgroundWorkStop.integration.test.ts, RuntimeRequestService.test.ts, CheckpointRollbackService.test.ts, OpenCode2AdapterV2.test.ts: all passed.
  • vp exec tsc --noEmit -p . in apps/server: no errors.
  • vp lint on the touched files: clean.

Not checked: a patched server running end to end against a live Codex app-server for the full 30-minute default timeout.

repro.mjs and fake-mcp.mjs
// fake-mcp.mjs
// Minimal stdio MCP server. argv[2] is a marker so the repro can count instances.
import { createInterface } from "node:readline";
const send = (m) => process.stdout.write(JSON.stringify(m) + "\n");
createInterface({ input: process.stdin }).on("line", (line) => {
  const msg = JSON.parse(line);
  if (msg.id === undefined) return;
  if (msg.method === "initialize")
    send({ jsonrpc: "2.0", id: msg.id, result: { protocolVersion: msg.params.protocolVersion, capabilities: { tools: {} }, serverInfo: { name: "fake", version: "0" } } });
  else if (msg.method === "tools/list") send({ jsonrpc: "2.0", id: msg.id, result: { tools: [] } });
  else send({ jsonrpc: "2.0", id: msg.id, error: { code: -32601, message: "nope" } });
});
// repro.mjs  (CODEX=/path/to/codex node repro.mjs; UNSUBSCRIBE=0 for the control)
// Drives `codex app-server` over stdio: start N threads, unsubscribe them, and
// count the MCP server processes left behind after the unload delay.
import { spawn, execFileSync } from "node:child_process";
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { createInterface } from "node:readline";

const N = Number(process.env.N ?? 3);
const UNSUBSCRIBE = process.env.UNSUBSCRIBE !== "0";
const DELAY = 2;
const marker = `fake-mcp-${process.pid}`;
const home = mkdtempSync(join(tmpdir(), "codex-leak-"));
writeFileSync(join(home, "config.toml"), `thread_unload_delay_secs = ${DELAY}
[mcp_servers.fake]
command = "node"
args = [${JSON.stringify(resolve("fake-mcp.mjs"))}, "${marker}"]
`);

const server = spawn(process.env.CODEX ?? "codex", ["app-server"], {
  env: { ...process.env, CODEX_HOME: home },
  stdio: ["pipe", "pipe", "inherit"],
});
let nextId = 1;
const pending = new Map();
createInterface({ input: server.stdout }).on("line", (line) => {
  const msg = JSON.parse(line);
  if (msg.id !== undefined && pending.has(msg.id)) {
    const { ok, fail } = pending.get(msg.id);
    pending.delete(msg.id);
    msg.error ? fail(new Error(JSON.stringify(msg.error))) : ok(msg.result);
  }
});
const request = (method, params) =>
  new Promise((ok, fail) => {
    const id = nextId++;
    pending.set(id, { ok, fail });
    server.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
  });
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const count = () => {
  try { return execFileSync("pgrep", ["-f", marker]).toString().trim().split("\n").length; }
  catch { return 0; }
};

await request("initialize", { clientInfo: { name: "leak-repro", version: "0" } });
server.stdin.write(JSON.stringify({ jsonrpc: "2.0", method: "initialized" }) + "\n");

const threads = [];
for (let i = 0; i < N; i++) {
  const { thread } = await request("thread/start", { cwd: home });
  threads.push(thread.id);
}
await sleep(2000);
console.log(`after ${N}x thread/start: ${count()} MCP server processes`);

if (UNSUBSCRIBE) {
  for (const threadId of threads)
    console.log("thread/unsubscribe ->", (await request("thread/unsubscribe", { threadId })).status);
}
await sleep((DELAY + 5) * 1000);
console.log(`${DELAY + 5}s later (${UNSUBSCRIBE ? "unsubscribed" : "still subscribed"}): ${count()} MCP server processes`);

server.kill();
await sleep(1000);
console.log(`after app-server exit: ${count()}`);
process.exit(0);

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Oct 7, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@juliusmarminge
juliusmarminge marked this pull request as ready for review October 7, 2026 20:47
@juliusmarminge
juliusmarminge enabled auto-merge (squash) October 7, 2026 20:52
@juliusmarminge
juliusmarminge merged commit eba0521 into pingdotgg:main Oct 7, 2026
32 of 33 checks passed
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Oct 8, 2026
## What's Changed
* feat(models): add Claude Haiku 5.5 and retire Sonnet 5 and Opus 5 to legacy by @juliusmarminge in pingdotgg/t3code#16903
* fix(web): iPhone Duo folds animate, center on the hinge, and keep the phone's orientation by @gabrielelpidio in pingdotgg/t3code#16885
* fix(server): Claude 5-series models always run 1M context by @juliusmarminge in pingdotgg/t3code#16908
* fix(desktop): setup prompts name a t3 that runs on desktop installs by @juliusmarminge in pingdotgg/t3code#16676
* feat(desktop): install the t3 command from Settings by @juliusmarminge in pingdotgg/t3code#16683
* fix(web): show live names in thread-read activity by @Bil0000 in pingdotgg/t3code#13140
* feat(mobile): adopt v5 navigation and native iPad columns by @juliusmarminge in pingdotgg/t3code#16733
* fix(mobile): Android composer picker scrolls past the first four rows by @shivamhwp in pingdotgg/t3code#15856
* fix(desktop): sign-in and captchas work again in desktop browser tabs by @juliusmarminge in pingdotgg/t3code#16939
* fix(server): missing project folders no longer log favicon warnings by @yordis in pingdotgg/t3code#16757
* fix(server): unload Codex threads left idle on the shared app-server by @RhysSullivan in pingdotgg/t3code#16917
* fix(web): fast typing no longer scrambles text when type-to-focus kicks in by @otavio in pingdotgg/t3code#14595
* fix(web): simplify workspace card rows by @Bil0000 in pingdotgg/t3code#16823
* fix(web): Copy MCP URL shows up for environments reached over plain http by @SunkenInTime in pingdotgg/t3code#16909
* fix(web): C#, Java, PHP and 11 other languages get file icons by @juliusmarminge in pingdotgg/t3code#16974
* feat(clients): live row shows the agent's latest thought by @t3dotgg in pingdotgg/t3code#16284
* feat(web): block-level Markdown in the rich text composer by @chrisdeeming in pingdotgg/t3code#14677
* fix(web): center project monograms in settled rows by @Aforno in pingdotgg/t3code#16841
* fix(web): cancelling a new citation no longer leaves a stray space by @Aforno in pingdotgg/t3code#16828
* fix(settings): provider updates show live progress instead of a bare spinner by @shivamhwp in pingdotgg/t3code#16958
* feat(web): find in diffs with Cmd+F by @juliusmarminge in pingdotgg/t3code#14623
* refactor(server): GitHub services are named for the API they call, not gh by @juliusmarminge in pingdotgg/t3code#16967
* refactor(server): GitHub GraphQL batches use variables and share one pager by @juliusmarminge in pingdotgg/t3code#16960
* refactor(server): GitHub source control reads GitHubApi directly by @juliusmarminge in pingdotgg/t3code#16982
* refactor(server): GitHub rate limits read the response headers by @juliusmarminge in pingdotgg/t3code#16986

## New Contributors
* @RhysSullivan made their first contribution in pingdotgg/t3code#16917
* @Aforno made their first contribution in pingdotgg/t3code#16841

**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261007.2787...v0.0.46-nightly.20261008.2801

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261008.2801
github-actions Bot added a commit to davidvanderklay/t3code-flake that referenced this pull request Oct 8, 2026
## What's Changed
* feat(models): add Claude Haiku 5.5 and retire Sonnet 5 and Opus 5 to legacy by @juliusmarminge in pingdotgg/t3code#16903
* fix(web): iPhone Duo folds animate, center on the hinge, and keep the phone's orientation by @gabrielelpidio in pingdotgg/t3code#16885
* fix(server): Claude 5-series models always run 1M context by @juliusmarminge in pingdotgg/t3code#16908
* fix(desktop): setup prompts name a t3 that runs on desktop installs by @juliusmarminge in pingdotgg/t3code#16676
* feat(desktop): install the t3 command from Settings by @juliusmarminge in pingdotgg/t3code#16683
* fix(web): show live names in thread-read activity by @Bil0000 in pingdotgg/t3code#13140
* feat(mobile): adopt v5 navigation and native iPad columns by @juliusmarminge in pingdotgg/t3code#16733
* fix(mobile): Android composer picker scrolls past the first four rows by @shivamhwp in pingdotgg/t3code#15856
* fix(desktop): sign-in and captchas work again in desktop browser tabs by @juliusmarminge in pingdotgg/t3code#16939
* fix(server): missing project folders no longer log favicon warnings by @yordis in pingdotgg/t3code#16757
* fix(server): unload Codex threads left idle on the shared app-server by @RhysSullivan in pingdotgg/t3code#16917
* fix(web): fast typing no longer scrambles text when type-to-focus kicks in by @otavio in pingdotgg/t3code#14595
* fix(web): simplify workspace card rows by @Bil0000 in pingdotgg/t3code#16823
* fix(web): Copy MCP URL shows up for environments reached over plain http by @SunkenInTime in pingdotgg/t3code#16909
* fix(web): C#, Java, PHP and 11 other languages get file icons by @juliusmarminge in pingdotgg/t3code#16974
* feat(clients): live row shows the agent's latest thought by @t3dotgg in pingdotgg/t3code#16284
* feat(web): block-level Markdown in the rich text composer by @chrisdeeming in pingdotgg/t3code#14677
* fix(web): center project monograms in settled rows by @Aforno in pingdotgg/t3code#16841
* fix(web): cancelling a new citation no longer leaves a stray space by @Aforno in pingdotgg/t3code#16828
* fix(settings): provider updates show live progress instead of a bare spinner by @shivamhwp in pingdotgg/t3code#16958
* feat(web): find in diffs with Cmd+F by @juliusmarminge in pingdotgg/t3code#14623
* refactor(server): GitHub services are named for the API they call, not gh by @juliusmarminge in pingdotgg/t3code#16967
* refactor(server): GitHub GraphQL batches use variables and share one pager by @juliusmarminge in pingdotgg/t3code#16960
* refactor(server): GitHub source control reads GitHubApi directly by @juliusmarminge in pingdotgg/t3code#16982
* refactor(server): GitHub rate limits read the response headers by @juliusmarminge in pingdotgg/t3code#16986

## New Contributors
* @RhysSullivan made their first contribution in pingdotgg/t3code#16917
* @Aforno made their first contribution in pingdotgg/t3code#16841

**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261007.2787...v0.0.46-nightly.20261008.2801

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261008.2801
aorwall added a commit to aorwall/t3code that referenced this pull request Oct 12, 2026
Merges `pingdotgg/t3code` `0f09627a11` (44 commits since `cd41c4ada0`)
into the fork. Branched from `main` at `75baf5f94`; no other merge PR is
open.

## Resolution

- **Counts:** 320 files landed against 320 in the upstream range, an
exact match. The fork delta is still 765 files.
- **Conflicts (2, both `converged`):**
- `apps/web/src/components/ChatMarkdown.tsx`: the fork's Moatless
Artifact chip still renders before upstream's new thread-link renderer
(pingdotgg#17017).
- `apps/web/src/components/chat/ThreadDetailsPanel.tsx`: took upstream's
merged environment and branch row (pingdotgg#16823) and re-applied
`FEATURES.openInEditor`.
- `pnpm-lock.yaml` was re-derived with `install.mjs`.
- `duplicate-adds.mjs` and `resolution-check.mjs` found nothing.
- **Unsupported methods:** added `UnsupportedMethodError` to
`mcpApps.callTool`, `toolInfo`, `readResource` and `updateModelContext`.
Nothing to drop.
- Formatted `docs/fork/inventory.json`, which was unformatted on `main`.

## Verification

`verify.mjs` passed duplicate-adds, tripwires, resolution-check,
unsupported-methods, lockfile, fmt, lint, typecheck and build. The
`test` step failed:
- `@t3tools/web` failed the full run and passed in isolation.
- In `t3`, `OpenCodeServerLedger`, `AcpAdapterV2`,
`OrchestratorReplayFixtures` and `ServerBrowserPage` fail when run
alone. This is the same set the 2026-10-07 merge recorded: the sandbox
does not reap process groups, `CLAUDE_CONFIG_DIR` leaks, and
Playwright's headless shell is missing. Machine, not code.

## Usable as-is
- Thread links reference the thread id and open in the app (pingdotgg#17017,
pingdotgg#16782 client half)
- Find in diffs with Cmd+F (pingdotgg#14623)
- Block-level Markdown in the rich-text composer (pingdotgg#14677)
- The live row shows the agent's latest thought (pingdotgg#16284), and running
subagent cards stay visible (pingdotgg#16878)
- Simplified workspace card rows (pingdotgg#16823), centred project monograms
(pingdotgg#16841), file icons for 14 more languages (pingdotgg#16974)
- Composer fixes: citation-cancel spacing (pingdotgg#16828), type-to-focus
scrambling (pingdotgg#14595)
- Lineage: Stop on hover (pingdotgg#16791), live names in thread-read activity
(pingdotgg#13140)
- Model catalogue: Claude Haiku 5.5, with Sonnet 5 and Opus 5 retired to
legacy (pingdotgg#16903)
- Mobile, desktop and onboarding fixes (pingdotgg#16713, pingdotgg#15856, pingdotgg#16733, pingdotgg#16886,
pingdotgg#16885, pingdotgg#16939, pingdotgg#16676, pingdotgg#16887)

## Unsupported in Moatless / needs implementation
- **MCP apps inline in threads** (pingdotgg#16236): `mcpApps.callTool`,
`mcpApps.toolInfo`, `mcpApps.readResource`,
`mcpApps.updateModelContext`, and the `McpAppFrame.tsx` UI. The frame
mounts only for a work-log item that carries an MCP app snapshot.
Upstream's Codex adapter projects that snapshot; Moatless does not, so
nothing renders. Moatless would need to project the snapshots and
dispatch the family. The new union entries are recorded in
`docs/fork/gaps.md`.
- **Provider update live progress** (pingdotgg#16958) and **install the `t3`
command from Settings** (pingdotgg#16683): behind `providerConfiguration` and the
desktop bridge respectively. Nothing new is reachable.
- **Copy MCP URL over plain http** (pingdotgg#16909): still behind
`FEATURES.connections`.

## Backend behavior to consider reproducing in Moatless
- The orchestrator MCP thread list and detail report and filter on
snooze state (`snoozed`, `snoozedUntil`; pingdotgg#16782).
- Idle Codex threads are unloaded from the shared app-server (pingdotgg#16917).
- A refused Claude turn keeps its session (pingdotgg#16287), and a prompt does
not hang on a uuid the session already holds (pingdotgg#16897).
- Claude 5-series models always run with 1M context (pingdotgg#16908).
- A background Claude subagent's work shows while its parent is idle
(pingdotgg#16486).
- Worktree cleanup: freed for terminal thread statuses (pingdotgg#15150), removed
after squash merges (pingdotgg#14847), files hidden by `showUntrackedFiles=no`
are kept (pingdotgg#15834), and Windows long paths work (pingdotgg#14917).
- Replaying a command or opening a long thread no longer blocks the
server (pingdotgg#17041, pingdotgg#17029).
- GitHub: rate limits are read from response headers, and GraphQL
batches use variables (pingdotgg#16986, pingdotgg#16960). `ChangeRequest` gained an
optional `headSha`.

All of these are recorded under "Runtime fixes upstream made to its own
server" in `docs/fork/gaps.md`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
Moatless task:
https://moatless.soaplabstest.com/tasks/af277a83-a9de-4d81-9c77-866e383cdb99
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants