relayflowd run.get times out under CPU load and kills the run: a flow's own test suite can destroy 30 steps of journaled work

[khaliqgant]

Summary

relayflowd fails run.get under CPU load and the CLI kills the run for it. Two runs of the same 67-step flow died this way, 28 and 30 steps deep, hours of agent work discarded:

FAILED [protocol_error] relayflowd could not complete the run request:
journal client: run.get timed out after 30000ms

The daemon was not resource-starved. At the time of failure it held 11 open file descriptors against a soft limit of 1,048,576, and 25 MB RSS. Journals were small — the largest run journal is 1.8 MB. It was CPU-starved.

By a step the flow itself launched.

What causes it

The flow's unit-tests step runs npx vitest run, which schedules 194 test files across all 8 cores. For the ~30s that takes, the daemon cannot get enough CPU to answer a read within its fixed 30-second budget, and the run dies.

Both failures landed in that window. Run g's evidence file for unit-tests is stamped 18:02:49; the timeout follows immediately.

This is a bad shape: a flow that runs a normal test suite can kill itself, and the failure is reported as a protocol error, which points the author at the daemon rather than at load.

Why the workaround is not the answer

I capped the flow at --maxWorkers=4 and it stopped happening. That works, and it is the wrong place for the fix — it means every flow that runs a build, a test suite, or anything else CPU-hungry has to know to throttle itself to protect the orchestrator. Most authors will discover this the way I did, by losing a long run twice.

What to change

Any of these, roughly in order of value:

• Retry run.get before declaring the run dead. A read that times out under load is not evidence the run is unrecoverable. The CLI already knows how to resume; a transient read failure should not be terminal.
• Make the budget adaptive, or raise it materially. 30 seconds is generous when the machine is idle and far too short when it is not. Back off and retry rather than failing closed on the first miss.
• Distinguish "the daemon is unreachable" from "the daemon did not answer in time." The current message reads as a protocol fault; it is a scheduling one. protocol_error sent me looking at the journal and the socket before I looked at load.
• Consider whether the run should die at all. The work was complete and journaled. A read timeout lost 30 steps of state that was already durable on disk — the run could have been parked for resume instead.

Acceptance

• A flow whose deterministic step saturates the machine completes rather than dying on run.get.
• A read timeout is retried, and the resulting failure (if any) names load or unreachability specifically.
• A run that cannot be read is parked, not failed, when its journal is intact.

Environment

CLI 2.0.22, relayflowd from @relayflows/runtime-darwin-arm64, macOS arm64, 8 cores. Nine daemons running on the box from unrelated projects, all with live data dirs; the affected one had 8h25m uptime.

[khaliqgant]

Third occurrence, and two new facts that make this worse than first reported.

It is not specific to vitest. I capped the flow's test suite at --maxWorkers=4 and restarted the daemon (it had 8h25m uptime). The next run still died the same way — this time during an agent step that was running the parity suites, which spawn real broker processes. So any heavy step kills the run, not just one known-noisy command. An author cannot throttle what an agent decides to run.

There is no lever. requestTimeoutMs is a JournalClient constructor option defaulting to 30_000 (packages/sdk/src/journal-client.ts:30,74), and nothing reads an environment variable for it. Grepping for FLOWS_*TIMEOUT in packages/sdk/src returns nothing. An operator hitting this repeatedly has no supported way to raise the budget, reduce the blast radius, or even observe it coming.

Cost so far: three runs of the same 67-step flow killed at steps 28, 30 and 33. Each relaunch costs a full re-execution of whatever --reuse-from cannot carry, and the run that died at 33 was halfway through, 27 minutes in, with a real regression fix in progress.

Minimum viable mitigations, in order of how cheap they look from outside:

1. FLOWS_REQUEST_TIMEOUT_MS (or similar) so an operator on a loaded machine has any lever at all.
2. Retry a timed-out run.get before declaring the run dead.
3. Park rather than fail. All three runs had intact journals — the state was durable and was discarded anyway.

[khaliqgant]

Fourth occurrence, now on a fully matched and freshly started 2.0.25 toolchain.

Environment:

• @relayflows/sdk 2.0.25
• @relayflows/surface 2.0.25
• @relayflows/runtime-darwin-arm64 2.0.25
• fresh daemon launched explicitly from the local 2.0.25 runtime
• run 01M327P6TEM36VX86R8NB8WNN6, resumed via --reuse-from

The failure happened immediately after the repair-seam-rules agent subprocess exited. This was not a CPU-saturating test step: at observation time the client and daemon were both alive and reporting 0.0% CPU, and the daemon remained alive after the CLI exited. The client then emitted:

FAILED [protocol_error] relayflowd could not complete the run request: journal client: run.get timed out after 30000ms

This adds two useful constraints to the original report:

1. The bug is still present in 2.0.25 with SDK, surface, and runtime versions aligned.
2. It is not limited to host CPU saturation; the same timeout occurs at the agent completion/step-transition boundary while the daemon process remains alive.

Update: the immediately following continuation, run 01M328GR56AJW2F11CKZS5J5S1, reproduced the timeout again. The foreground CLI stopped printing after repair-ts, but direct read-only inspection of the run journal showed the daemon subsequently completed repair-ts, ts-final, ts-assert, and the 10-minute parity step. Only after all of that did the CLI exit with the same 30-second run.get timeout. This directly shows the durable run can remain healthy and advance while the client status/read path is wedged; treating the read timeout as a terminal protocol error is incorrect.

The journaled work remains intact and is being recovered with continuation runs. No duplicate issue filed.

[khaliqgant]

Additional matched-2.0.25 reproduction from Relay native-delivery Phase 1:

• CLI run 01M32PCWPWS2RJSQ6NB6QNBPZF displayed no further progress after the last worker-lease snapshot and its client-side wait became stale.
• Direct journal inspection showed repair-native-evidence actually completed successfully at 2026-09-21 12:18:01 PDT (61s, completionReason=success) and the daemon had already started native-evidence-final.
• An earlier flows resume on 01M32CZK7DWK0BXQGK403SG2JV reported worker lease expired without completion after the CLI stopped receiving fresh snapshots. This can therefore be a false observer verdict: the CLI retains the last 30s lease deadline when run.get stalls, while journal execution continues.

This strengthens the impact: the failure is not only missing status; it can incorrectly present an actively advancing run as a worker lease expiry.