Skip to content

test(server): record Codex multi-agent V2 subagents live on 0.156.1 - #13508

Merged
juliusmarminge merged 3 commits into
t3code/codex-turn-mappingfrom
v2/codex-subagent-v2-live
Sep 24, 2026
Merged

juliusmarminge merged 3 commits into
t3code/codex-turn-mappingfrom
v2/codex-subagent-v2-live

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

The subagent_v2 and subagent_v2_nested Codex transcripts were hand-built, with synthetic thread ids and hand-ordered frames. This records both live against Codex 0.156.1 on gpt-5.6-sol, which runs multi-agent v2.

Stacked on #13505, which adds the recorder changes this uses: the adapter's turn/start defaults, --model, and <workspace>.

What changed

  • Recorder:
    • adds the subagent_v2 and subagent_v2_nested scenarios;
    • adds a per-run thread/start config override. The nested scenario sets agents.max_depth: 3, because by default Codex only lets the root agent spawn. Replay ignores thread/start config, so the transcript still matches the adapter's frames.
  • Transcripts: both are straight recordings, with no hand edits.
  • Prompts: the new prompts ask for the spawns explicitly. On gpt-5.6-sol, "just say hello" answers directly and spawns nothing.
  • Model: each fixture's Codex variant uses its own modelSelection (gpt-5.6-sol). The shared selection is unchanged.
  • Assertions: they now use the agent paths and answers the model actually chose (/root/hello; /root/relay_one/relay_two/hello_child). The CodexReplayFixtures expectations use the recorder's method labels instead of the hand-written role labels.

What the live recordings showed

  • In both recordings, and at every nesting level, Codex emits:

    1. the child's thread/status/changed (idle),
    2. the parent's subAgentActivity(started),
    3. the child's thread/status/changed (active) and turn/started.

    So the adapter registers the child before its first turn. This matches the 0.145 capture in codexMultiAgentWire.json. The "child turn starts before the parent's activity" race did not happen, so this PR contains no adapter change.

  • Children finish first. Each parent then emits subAgentActivity(completed), a wait collabAgentToolCall, and its own answer. The adapter projects all four nested threads with the right lineage and results.

Verification

  • vp test run src/orchestration-v2/testkit/OrchestratorReplayFixtures.integration.test.ts -t "subagent_v2|thread_rollback": 4 passed.
  • vp test run on CodexReplayFixtures, OrchestratorReplayFixtures.contract, CodexAdapterV2.test.ts, ThreadFork, ThreadMergeBack and scripts/codexReplayRecordingRecords.test.ts: 139 passed.
  • vp exec tsc --noEmit -p . in apps/server: no errors or warnings. vp run knip:check passes. vp lint on the touched files reports only an unused assertUserMessagesExclude in fixtures/shared.ts, which is already on the base branch.
  • Recorded with T3_CODEX_BIN=<codex 0.156.1> node scripts/record-codex-app-server-replay-fixture.ts --scenario subagent_v2 --model gpt-5.6-sol, and the same with --scenario subagent_v2_nested.
  • Not addressed here: plan_questions/codex fails on the base branch too. Its 0.120 transcript has no isBlocking on item/tool/requestUserInput, so the 0.156 decode rejects the request. Replay then stalls for 60 seconds instead of reporting the error.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code


Devin Review

juliusmarminge and others added 2 commits September 24, 2026 15:00
The Codex thread_rollback transcript was captured on 0.120 and its rollback
exchange was hand-edited to the 0.156 thread/turns/list + thread/revert shape.
Re-record it against a real 0.156.1 app-server so the revert path is proven
by Codex itself.

The recorder now sends the adapter's initialize opt-out and turn/start
defaults, reads the thread's history mode before reverting (as the adapter
does), accepts --model, and writes the recording cwd as <workspace>, so the
transcript replays without edits. The fixture's Codex variant uses
gpt-5.6-luna because the hub no longer serves gpt-5.4.

The live recording shows a paginated thread from thread/start, a revert
response with empty turns plus backwards cursors, and a post-revert recall
that only contains the first turn. The output assertion now checks that
recall, and the extra reasoning items the current model emits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The subagent_v2 and subagent_v2_nested transcripts were hand-built with
synthetic thread ids and hand-ordered frames. Record both against a real
Codex 0.156.1 app-server on gpt-5.6-sol, which runs multi-agent v2.

The recorder gains the two scenarios and a per-run thread/start config
override: the nested scenario raises agents.max_depth to 3, since Codex only
lets root spawn by default. Replay ignores thread/start config, so the
transcript still matches the adapter's frames. The prompts now ask for the
spawns explicitly; "just say hello" alone no longer spawns anything on this
model.

In both recordings Codex emits the child's idle status, then the parent's
subAgentActivity(started), then the child's turn/started, so the adapter
registers the child before its first turn. Assertions now name the agent
paths and answers the model chose, and the scenario expectations use the
recorder's method labels.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Sep 24, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 24, 2026
@macroscopeapp

macroscopeapp Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

A prior Macroscope approval was dismissed. Re-evaluating e79ede0…

@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 4.9 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.1 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 20.4 KiB — 29.3 KiB ✅
Codex Live turn messages — 1 — 8 ✅
Claude Total thread wire — 4.9 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 20.8 KiB — 29.3 KiB ✅
Claude Live turn messages — 2 — 8 ✅

Baseline: unavailable · PR result: a955a2c · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 106.1 KiB
  • Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Base automatically changed from v2/codex-rollback-live to t3code/codex-turn-mapping September 24, 2026 22:19
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 24, 2026 22:19

Dismissing prior approval to re-evaluate a955a2c

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 24, 2026
…turn

The live multi-agent V2 recordings show the parent's subAgentActivity(started)
arriving before each child's turn/started, and the adapter registers children
from that activity. Check the ordering in both transcripts, so a re-recording
that reverses it fails loudly. Also drop `_LIVE_` from the V2 prompt names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 24, 2026 22:26

Dismissing prior approval to re-evaluate e79ede0

@juliusmarminge
juliusmarminge merged commit 6448cfe into t3code/codex-turn-mapping Sep 24, 2026
23 checks passed
@juliusmarminge
juliusmarminge deleted the v2/codex-subagent-v2-live branch September 24, 2026 22:26
@github-actions github-actions Bot added size:XL 500-999 changed lines (additions + deletions). and removed size:L 100-499 changed lines (additions + deletions). labels Sep 24, 2026
juliusmarminge added a commit that referenced this pull request Sep 24, 2026
…13508)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
juliusmarminge added a commit that referenced this pull request Sep 25, 2026
…13508)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant