Skip to content

test(server): drop Claude background unit tests the live fixtures cover - #13519

Merged
juliusmarminge merged 2 commits into
v2/claude-backgroundfrom
v2/claude-background-tests
Sep 25, 2026
Merged

juliusmarminge merged 2 commits into
v2/claude-backgroundfrom
v2/claude-background-tests

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Stacked on #13517. Now that the live fixtures cover Claude background work, the hand-built-frame unit tests that restated the same flows can go. The tests that stay should use frames the CLI actually sends.

Deleted: 8 tests, about 1,030 lines of ClaudeAdapterV2.test.ts

For each test, the adapter rule it guards was reverted and the named fixture failed:

Test Covered by
projects an authoritative background_tasks_changed roster claude_background_task_after_root
clears the native-thread roster when a turn is interrupted claude_background_task_interrupt
drains buffered wake messages into a continuation turn claude_background_task_wake
terminalizes a continuation turn from a task-notification origin wake result claude_background_task_wake
wakes and hydrates a subagent that completes after the root turn settled claude_background_subagent_after_root
re-opens a resumed subagent and hydrates its second result claude_background_subagent_lifecycle
orders nonempty level, empty level, notification, and continuation drain claude_background_task_wake
admits only local_bash from a mixed background_tasks_changed snapshot claude_background_subagent_after_root, claude_background_subagent_lifecycle

The fixtures pick up the assertions those tests made: the continuation detail, and the subagent's run and node attribution. Two sub-checks go without a fixture, because the CLI has not been seen producing them:

  • a task_progress after a subagent's notification (none among 5,565 logged progress frames)
  • a duplicate local_bash notification (the logged duplicates are all subagents)

Kept, with realistic frames

The remaining background tests cover three kinds of behavior:

  • orderings the CLI emits nondeterministically (held frames before task_started, a resume task_started racing past settle, interrupt races)
  • failures that cannot be produced on demand (process replacement, failed reopen)
  • the idle-release probe

Their frames now match what the CLI sends:

  • terminal_reason on results, except a zero-turn task-notification result, which omits it as seen live
  • tool_use_id and is_backgrounded on task_started
  • tool_use_id on task_notification
  • message.id on assistant frames, one content block per frame
  • SendMessage input {to, summary, message}, the resume ACK text as recorded, and the resumed notification under the SendMessage tool_use_id
  • background Bash ids and texts taken from the claude_background_task_wake recording

The held-frame test still sends a task_started without tool_use_id. The SDK types that field as optional, but no recorded or logged frame has omitted it. The test now says so.

Verification

  • vp test run on ClaudeAdapterV2.test.ts, ClaudeAdapterV2.testkit.test.ts, OrchestratorReplayFixtures.contract.test.ts, and scripts/claudeReplayRecordingConfig.test.ts: 140 passed.
  • vp test run src/orchestration-v2/testkit/OrchestratorReplayFixtures.integration.test.ts -t claude: 24 passed.
  • vp exec tsc --noEmit -p . in apps/server: 0 error TS, 0 warning TS.
  • vp run knip:check: passes.
  • vp lint on the touched files: clean.
  • Not run: repo-wide checks.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code


Devin Review

Eight hand-built-frame tests in ClaudeAdapterV2.test.ts restated flows the
live recordings now replay through the whole orchestrator. For each, the
adapter rule it guards was reverted and a fixture failed:

- authoritative background_tasks_changed roster: claude_background_task_after_root
- interrupted turn clears the roster: claude_background_task_interrupt
- buffered wake drains into a continuation, and a task-notification wake
  result settles it: claude_background_task_wake
- background subagent completing after root settle, attributed to its
  launch run: claude_background_subagent_after_root
- SendMessage resume re-opens the subagent and hydrates its second result:
  claude_background_subagent_lifecycle
- empty roster level before the notification keeps wake eligibility:
  claude_background_task_wake
- mixed snapshot admits only local_bash: claude_background_subagent_after_root
  and claude_background_subagent_lifecycle

The fixtures gain the assertions those tests carried (continuation detail,
subagent run and node attribution). Two sub-checks go without a fixture
because the CLI has not been seen producing them: a task_progress after a
subagent's notification (none in 5,565 logged progress frames) and a
duplicate local_bash notification (logged duplicates are subagents only).

The remaining background tests cover orderings the CLI emits
nondeterministically, failures that cannot be produced on demand, and the
idle-release probe. Their frames now match what the CLI sends:
terminal_reason on results, tool_use_id and is_backgrounded on
task_started, tool_use_id on task_notification, message.id on assistant
frames, one content block per assistant frame, SendMessage input with
`to`, the resumed notification under the SendMessage tool_use_id, and
background Bash ids and texts taken from the recording.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Sep 24, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 24, 2026
@macroscopeapp

macroscopeapp Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 419592a

Macroscope's review found this PR approvable — This PR is confined to test-only unit and replay-fixture assertions: it removes duplicated synthetic-frame coverage and strengthens live-fixture checks without touching production code, defaults, or deployment configuration. Its runtime blast radius is limited to test execution.

Notes:

  • No code objects were reviewed. Approvability was decided on eligibility alone.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 4.9 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.1 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 20.4 KiB — 29.3 KiB ✅
Codex Live turn messages — 1 — 8 ✅
Claude Total thread wire — 4.9 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.7 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 20.8 KiB — 29.3 KiB ✅
Claude Live turn messages — 2 — 8 ✅

Baseline: unavailable · PR result: 419592a · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 106.1 KiB
  • Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

The resume test's ACK text dropped the short agent id and the pin object
the recorded frame carries, and omitted tool_use_result. Build both from
one object shaped like the claude_background_subagent_lifecycle recording.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@macroscopeapp
macroscopeapp Bot dismissed their stale review September 24, 2026 23:47

Dismissing prior approval to re-evaluate 419592a

@juliusmarminge
juliusmarminge merged commit 4b2fe2e into v2/claude-background Sep 25, 2026
23 of 24 checks passed
@juliusmarminge
juliusmarminge deleted the v2/claude-background-tests branch September 25, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant