Skip to content

test(server): remove replay harness self-tests - #13531

Merged
juliusmarminge merged 1 commit into
t3code/codex-turn-mappingfrom
v2/drop-testkit-tests
Sep 25, 2026
Merged

juliusmarminge merged 1 commit into
t3code/codex-turn-mappingfrom
v2/drop-testkit-tests

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

The V2 test tree had tests for the replay harness itself: the Claude, Cursor, and OpenCode replay runners, the replay gate, the NDJSON loader, the Codex app-server replay transport, and the fixture recorders. The replay fixtures already run the real adapters through the whole orchestrator (OrchestratorReplayFixtures.integration.test.ts), so a harness bug fails those. Testing the harness against hand-built transcripts only restates it.

Removed

Named in the request

  • Adapters/ClaudeAdapterV2.testkit.test.ts (17 tests): replay gate hold and release, mismatch wakeups, interrupt bookkeeping, permission-callback abort and per-query scoping, cancelled runtime_exit, recorder interrupt and finalize behavior, denied-write allow substitution, and resolveClaudeRecordingExecutablePath.
  • Adapters/CursorAdapterV2.testkit.test.ts (4 tests): runtime-instruction prompt mismatch, and wakeups when an outbound frame mismatches.
  • Adapters/OpenCodeAdapterV2.testkit.test.ts (5 tests): OpenCodeReplayController message-ID binding and abort handling.

Other harness self-tests in the V2 tree

  • testkit/ProviderReplayGate.testkit.test.ts (2 tests). The gate holds frames in the claude_background_* and queued_cancelled_while_active fixtures.
  • testkit/ReplayTranscriptNdjson.test.ts (7 tests). Every fixture loads through this decoder, with <workspace> placeholders in 37 transcripts.
  • packages/effect-codex-app-server/src/replay.test.ts (2 tests). Every Codex fixture runs through layerReplay.
  • apps/server/scripts/{acp-replay-agent,replayRecorderDeferredRegistry,cursorReplayRecordingWorkspace,codexReplayRecordingRecords,claudeReplayRecordingConfig}.test.ts (11 tests). These covered recorder and replay-agent helpers. A broken recorder shows up when you record or replay a fixture.
  • OrchestratorReplayFixtures.contract.test.ts: dropped 6 cases that fed hand-built step lists to the fixture-input materializer (queue barriers, steer/answer ordering, run-ID derivation). The real fixtures cover each step type: queued_turn, queued_cancelled_while_active, plan_questions, message_steering, and claude_background_subagent_after_root.
  • CodexReplayFixtures.integration.test.ts: dropped the case that fed a made-up conflict to the test file's own dedupe helper.

The testkit helpers that only these tests imported are no longer exported. Nothing else about them changed, and the recorders and harness still call them internally.

Kept, and why

  • OrchestratorReplayFixtures.contract.test.ts keeps its two fixture-registry checks: every fixture has a transcript, a matching scenario and provider, unique command IDs, decodable commands, and gate labels that exist in the transcript, and Codex transcripts start and end at the app-server boundary. These guard the fixture set itself.
  • CodexReplayFixtures.integration.test.ts and ClaudeReplayFixtures.integration.test.ts keep their checks that transcripts match the real protocol and that the fixture set is complete.
  • scripts/acpMockCancellationState.test.ts stays. It tests the ACP mock agent that the adapter and provider unit tests use, not the replay harness.

Moved

Nothing. resolveClaudeRecordingExecutablePath is recorder-only. The production function it wraps, resolveClaudeSdkExecutablePath, keeps its tests in provider/Drivers/ClaudeExecutable.test.ts, including shim-to-package-entry resolution. permissionResultFromDecision keeps its tests in ClaudeAdapterV2.test.ts.

Line delta: +24 / −2218 across 16 files.

Verification

  • vp test run for OrchestratorReplayFixtures.integration, OrchestratorReplayFixtures.contract, CodexReplayFixtures.integration, ClaudeReplayFixtures.integration, OrchestratorReplayRecovery.integration, ClaudeAdapterV2, CursorAdapterV2, OpenCodeAdapterV2, and ClaudeExecutable: 9 files, 264 tests passed.
  • vp exec tsc --noEmit -p . in apps/server and in packages/effect-codex-app-server: no error TS or warning TS.
  • vp run knip:check: clean. The baseline before the change was also clean.
  • vp lint on the touched files: one warning, the unused makeClaudeAgentSdkReplayQueryRunnerLayer in ClaudeAdapterV2.testkit.ts. The same warning appears at the base commit, so this change did not introduce it.
  • Not run: repo-wide checks and live provider recordings.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code


Devin Review

Replay fixtures that drive the real adapters through the whole orchestrator
are the coverage for the replay harness: if a runner, gate, controller, or
transcript loader misbehaves, the fixture suite fails. Tests that assert the
harness and recorders against hand-built transcripts only restate the
harness.

Delete the Claude, Cursor, and OpenCode testkit self-tests, the
ProviderReplayGate and NDJSON loader tests, the Codex app-server replay
transport test, the recorder script helper tests, and the fixture-input
materializer and Codex canonical-dedupe helper cases. Keep the fixture
registry contract checks. Unexport the testkit helpers only those tests
imported.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Sep 25, 2026
@juliusmarminge
juliusmarminge merged commit d41bac5 into t3code/codex-turn-mapping Sep 25, 2026
23 of 24 checks passed
@macroscopeapp

macroscopeapp Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at bacd51c

Macroscope's review found this PR approvable — The PR removes replay-harness self-tests and narrows testkit exports while leaving helper implementations and production paths unchanged. Its impact is limited to test coverage and internal test infrastructure, with no product-default or static-analysis suppression changes.

You can add or adjust custom eligibility rules. Learn more.

@juliusmarminge
juliusmarminge deleted the v2/drop-testkit-tests branch September 25, 2026 00:49
juliusmarminge added a commit that referenced this pull request Sep 25, 2026
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S 10-29 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant