test(server): replace Pi adapter unit tests with live replay fixtures

[juliusmarminge]

Pi had no replay fixtures, no replay harness and no recorder. PiAdapterV2 was tested only against a hand-written fake pi process, and some of the fake's frames don't match what Pi actually sends. This PR adds a replay harness and a live recorder, records seven Pi scenarios through the whole orchestrator, and deletes the unit tests those recordings now cover.

Harness and recorder

• PiAdapterV2.testkit.ts replays a transcript through the ChildProcessSpawner that PiAdapterV2 uses to spawn pi --mode rpc, so the real adapter and PiRpc framing run unchanged.
  • Transcript entries are stdio records: expect_outbound for what the adapter writes to stdin, emit_inbound for what Pi writes to stdout, and a synthetic process_start carrying the spawn argv. Records from a second process carry an @pN label suffix.
  • Only the adapter's own t3-N request ids are rebound. Any other id, such as the Pi request id an extension_ui_response must echo, has to match exactly.
  • Independent adapter fibers can race to stdin (skill discovery next to thread setup), so a write may match ahead of its recorded position, but never past the next prompt, compact or process start. The recorded fixtures exercise this: multi_turn fails when the adapter sends its second prompt early.
  • A trailing runtime_exit is consumed as Pi exiting on its own. Mismatch and incomplete errors report the cursor, entry label and frame type, not whole frames, like the other providers' replay errors.
• scripts/record-pi-rpc-replay-fixture.ts (vp run record:pi-replay -- --scenario <name>) runs a registered fixture's own input through the real orchestrator and real PiAdapterV2 against a live pi.
  • It pins openrouter / deepseek/deepseek-v4-flash.
  • It disables the user's extensions, skills, prompt templates and context files, and writes sessions to a temp dir. The user's ~/.pi session store is never touched.
  • It normalizes session files, UUIDs, timestamps, the home dir and the workspace, replaces system-prompt text with placeholders, and drops reasoning signatures.
  • The throwaway workspace gets a tiny compaction.keepRecentTokens, so /compact has something to summarize, plus any workspaceFiles the fixture declares.
  • Before writing, it replays the normalized transcript and runs the fixture's assertions, so a failing recording never replaces an existing fixture.

Fixtures (all recorded live on Pi 0.87.1)

Fixture	What its Pi output assertion checks
simple	Thinking deltas become one reasoning item with the recorded text. Live usage moves once per new streamed total. The settled turn carries its get_session_stats usage. The turn's native ref is the strong session-tree id of its user message.
multi_turn	Each settled turn carries its own get_session_stats usage.
message_steering	The steer is sent as prompt + streamingBehavior: "steer" with no abort, and the run keeps one provider turn.
turn_interrupt_mid_tool	Stop aborts after the bash tool starts. Pi ends the tool as an error, the item projects as interrupted, and the session ends stopped with no error.
provider_thread_resume (new Pi-only registration)	After the 30-minute idle release, a fresh process switch_sessions to the recorded file and the model still knows turn one.
thread_rollback	Rollback forks the tree at the discarded turn's user entry, and the next turn runs on the forked session file.
pi_compaction (new)	/compact <instructions> goes out as RPC compact with custom instructions. Pi refuses it as too small, which shows as a failed row. A bare /compact really summarizes and carries Pi's summary and token counts. The meter keeps the post-compaction estimate, and the next turn recalls the marker.

Unit tests deleted (PiAdapterV2.test.ts: 56 → 49 tests, +39 / −495 lines)

To prove each fixture covers what its deleted test covered, I broke the adapter path that test exercised and checked the covering fixture fails. All 8 mutations failed their fixture.

Deleted test	Covered by	Mutation that fails the fixture
streams assistant text and settles on agent_settled	simple, multi_turn	drop the settled turn's tokenUsage
captures session-tree refs and rolls back via fork	thread_rollback, simple	drop the turn-boundary nativeTurnRef
sends RPC compact for /compact	pi_compaction	treat /compact as a prompt
compacts a bare /compact through compactThread	pi_compaction	(same)
keeps a too-small compact as a failed item	pi_compaction	project the refused compaction as completed
stops with restart by aborting then terminating	turn_interrupt_mid_tool	skip the abort; don't mark stop requested
steers through an atomic prompt	message_steering	send the steer without streamingBehavior
steers the active turn (steer half)	message_steering	(same)
registers from get_state and resumes via switch_session	provider_thread_resume	resume with new_session

Two narrow tests replace parts of the deleted ones that a live Pi can't produce on demand: rejects a resume while a turn is active and keeps a settled turn's late prompt rejection off the next turn.

Kept: lifecycle, race and failure tests (slow/failed/vetoed lifecycle requests, retry and extension-error paths, settle-probe races, detached compaction, extension UI dialogs, MCP injection, unsolicited activity), the native-fork tests (no Pi fork fixture yet), the snapshot tests (orchestration never calls readThreadSnapshot, so no fixture reaches them), and the PiRpc framing/early-exit tests.

makeFakePi shapes now come from the recordings. get_state returns Pi's recorded idle shape (steeringMode, followUpMode, messageCount, pendingMessageCount, UUID sessionId). fork returns {text, cancelled}. Failed compactions omit result instead of sending null, which matches both the docs and Pi's source. The "nonpersistent session" test now sets sessionFile: undefined, which is what a --no-session Pi reports.

Net: +2362 / −496 across 23 files, of which +822 are recorded transcripts.

Findings

• No adapter bug surfaced: PiAdapterV2 handled every recorded frame correctly, including DeepSeek's thinking deltas, Pi's queue_update steering acks, compaction_end without result, and contextUsage.tokens: null right after a compaction.
• Pi's get_state default thinkingLevel for this model is high. A short conversation is never compactable under the default keepRecentTokens (20k), so the recorder shrinks it in the throwaway workspace only.

Verification

• vp test run src/orchestration-v2/testkit/OrchestratorReplayFixtures.integration.test.ts -t "/pi ": 7 passed. The full file passed 82/82 once, and the Pi subset was run repeatedly with no flakes.
• vp test run PiAdapterV2.test.ts OrchestratorReplayFixtures.contract.test.ts: 57 passed.
• Mutations: the 8 in the table above, plus an early second-turn prompt that fails multi_turn. For the review fixes: a trailing runtime_exit replays green with the fix and fails as unconsumed without it; a tampered frame fails with a bounded mismatch message; a live re-record of simple passes the replay-then-assert path; a forced assertion failure leaves the --out target untouched.
• vp exec tsc --noEmit -p . in apps/server: no error TS or warning TS.
• vp run knip:check: clean.
• vp lint on touched files: clean apart from an existing no-unused-vars in fixtures/shared.ts.
• Secret scan of all 7 transcripts: no OpenRouter key, sk-or-, bearer or auth header, and no home path.

Not run: repo-wide checks.

Model: Claude Opus 5.5 (Claude Code)

🤖 Generated with Claude Code

[macroscopeapp]

🟡 Medium scripts/record-pi-rpc-replay-fixture.ts:214

Pi records fixtures with workspaceFiles against an empty workspace, so the transcript and variant.assertOutput result do not represent the fixture's configured workspace. Build the fixture input before calling makeCheckpointWorkspace and pass fixtureInput.workspaceFiles when seeding it.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/server/scripts/record-pi-rpc-replay-fixture.ts around line 214:

Pi records fixtures with `workspaceFiles` against an empty workspace, so the transcript and `variant.assertOutput` result do not represent the fixture's configured workspace. Build the fixture input before calling `makeCheckpointWorkspace` and pass `fixtureInput.workspaceFiles` when seeding it.

[macroscopeapp]

🟡 Medium Adapters/PiAdapterV2.testkit.ts:309

Pi replays with a runtime_exit entry always fail with PiReplayIncompleteError, even after all process I/O has been consumed. drainInbound only advances over emit_inbound, so the terminal record remains at the cursor; consume runtime_exit as a terminal transcript record as well.

      const entry = this.transcript.entries[this.cursor];
+      if (entry?.type === "runtime_exit") {
+        this.cursor += 1;
+        emitted = true;
+        continue;
+      }
       if (entry?.type !== "emit_inbound") return emitted;

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/server/src/orchestration-v2/Adapters/PiAdapterV2.testkit.ts around lines 309-310:

Pi replays with a `runtime_exit` entry always fail with `PiReplayIncompleteError`, even after all process I/O has been consumed. `drainInbound` only advances over `emit_inbound`, so the terminal record remains at the cursor; consume `runtime_exit` as a terminal transcript record as well.

[macroscopeapp]

🟡 Medium scripts/record-pi-rpc-replay-fixture.ts:290

The recorder writes the transcript before variant.assertOutput, so a failed live assertion replaces the existing fixture or --out target with an unverified recording. Run the assertion before writing the file.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @apps/server/scripts/record-pi-rpc-replay-fixture.ts around line 290:

The recorder writes the transcript before `variant.assertOutput`, so a failed live assertion replaces the existing fixture or `--out` target with an unverified recording. Run the assertion before writing the file.

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The PR is confined to Pi replay/testing infrastructure and does not alter production behavior or product defaults. Human review is still warranted because replay mismatch errors retain complete RPC frames, which can expose arbitrary command arguments or output through test diagnostics.

Not approved because:

• 3 blocking correctness issues found at or above your repo's Minimum Blocking Severity

^{Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.}

[macroscopeapp]

This aliases a service module's layer at a service boundary, hiding the module's public shape. Please import * as IdAllocator and use IdAllocator.layer at the Effect.provide call.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Please keep the service module namespace here: import * as IdAllocator and provide IdAllocator.layer instead of renaming its layer export.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

These error attributes store whole RPC frames, including arbitrary command arguments and output; message also stringifies them. Please retain only bounded, safe diagnostics (for example frame type and cursor), and keep raw failure data out of error attributes and messages. This needs changes to the error definition and construction, so there is no single-hunk fix.

^{Posted via Macroscope — Effect Service Conventions}

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.4 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	4.9 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.7 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	20.8 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	2	—	8	✅

Baseline: unavailable · PR result: 09db321 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 106.1 KiB
• Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[macroscopeapp]

next stores an unconsumed RPC frame (which may contain command arguments or output) and the error message stringifies it. Please replace it with bounded diagnostics such as the entry type and cursor; update both the error definition and its construction in assertComplete().

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Macroscope skipped reviewing this pull request. Per-review cost limit exceeded (workspace setting).

This review would cost an estimated $19.63, which exceeds your per-review limit of $15.00.

The top 3 files driving up this estimate:

File	Diff Size	Estimate
apps/server/src/orchestration-v2/testkit/fixtures/pi_compaction/pi_transcript.ndjson	90.23KB	$3.79
apps/server/src/orchestration-v2/testkit/fixtures/thread_rollback/pi_transcript.ndjson	87.25KB	$3.66
apps/server/src/orchestration-v2/testkit/fixtures/provider_thread_resume/pi_transcript.ndjson	66.83KB	$2.81

Tip

To get this pull request reviewed, you can:

1. Comment @macroscope-app on this PR to request a manual review (monthly spend limits still apply).
2. Exclude the file(s) above from review by adding a pattern to your .macroscope/ignore.md — note that creating this file replaces Macroscope's built-in default ignores rather than extending them.
3. Raise your cost limit in your workspace billing settings.

^{Turn off this reminder going forward}