Repository navigation
No per-step inactivity/heartbeat watchdog — a stalled agent step hangs indefinitely #12
Description
Activity
The broker already has idle detection — relayflows can consume it instead of polling. (relay/crates/broker)
worker.rs:67AgentWorkState::Idle;worker.rs:226,274idle_threshold_secsplumbed to the worker (--idle-threshold-secs)pty_worker.rs:291idle_threshold;protocol.rs:638 AgentIdle { idle_secs }
So the broker emits an
AgentIdle { idle_secs }event when an agent sits idle past a threshold. The per-step watchdog here can subscribe toAgentIdlerather than build its own file-mtime/CPU poller: onAgentIdlepast the step budget → ifverificationalready passes, complete; else retry →needs_human(#6). Pairs with #11 (consumeagent_exit) — together they turn "hang forever" into "complete or retry on a broker event."
Summary
A `running` agent step that makes no progress (no agent CPU, no file mtime in cwd, no channel message) never times out. Only the coarse workflow-level timeout (e.g. 7_200_000ms) eventually fires. There is no per-step watchdog to retry/fail a stalled step.
Evidence
Every wedge in the factory build-out presented as a step stuck `running` 15–30+ min with zero activity, requiring manual detection + TaskStop. A per-step inactivity timeout would have auto-retried each.
Proposed fix
Configurable per-step inactivity timeout (default ~5–10 min of no completion-evidence event): on expiry → retry up to `retries`, then fail → `.repairable` / `needs_human` (#6). Heartbeat = any of {agent CPU, file mtime in cwd, channel message, verification poll}. Never hang.
Why this matters
This is the safety net that makes the other completion-reliability bugs non-fatal: a missed completion signal becomes an auto-retry instead of a hung run.