You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
So when the worker hits a lease error on an attempt the kernel has already expired and retried, the whole run ends as protocol_error and can't be resumed. The retry succeeds in the journal anyway. Seen two ways in relayflows 2.0.29:
Renewal racing completion. A step whose child run journaled step.completed success + run.completed success was reported as lease_conflict: attempt has no active worker lease, and the run died. Intermittent: once in about 50 sequential calls. 00-prove-attempt1-lease-conflict-after-success.txt
Scope
A lease refusal for an attempt the kernel no longer owns shouldn't abort the run. Examples: already expired before execute, lease_conflict / no active worker lease on heartbeat or complete. The kernel owns that attempt's fate (retry, or a completion that already landed). Drop it with a diagnostic (WAITING/warning line); don't close the client.
Any other worker error stays fatal, exactly as today. Fail closed.
The agent worker path (AgentWorker) likely has the same shape. Check it and apply the same rule if so.
Acceptance
A unit test injects a dispatch whose lease_deadline_ms is already past. The run continues, and the kernel's retry completes the step.
A unit test makes stepHeartbeat / stepComplete reject with lease_conflict after the step's journal is success. The run reports success.
A non-lease worker error still ends the run as today (regression test).
Mutation-verified per AGENTS.md.
Related: the root cause of case 1 under concurrency is tracked separately (it's why the dispatch was stale).
Problem
packages/sdk/src/cli/direct-run.ts:85treats any error from the localLlmWorkeras fatal:So when the worker hits a lease error on an attempt the kernel has already expired and retried, the whole run ends as
protocol_errorand can't be resumed. The retry succeeds in the journal anyway. Seen two ways in relayflows 2.0.29:withWorkerLeasethrowsAgent lease is already expired for <run>/<step>, and that becomes fatal. The kernel had already journaledlease_expired→ retry, and attempt 2 completedsuccess. Journal: 00-job1-attempt2-parallel-lease-expired.txt; the replay is quoted in feat(examples): prompt-lab — the Prompt Lab product brief as one relayflow #559.step.completed success+run.completed successwas reported aslease_conflict: attempt has no active worker lease, and the run died. Intermittent: once in about 50 sequential calls. 00-prove-attempt1-lease-conflict-after-success.txtScope
already expiredbefore execute,lease_conflict/no active worker leaseon heartbeat or complete. The kernel owns that attempt's fate (retry, or a completion that already landed). Drop it with a diagnostic (WAITING/warning line); don't close the client.AgentWorker) likely has the same shape. Check it and apply the same rule if so.Acceptance
lease_deadline_msis already past. The run continues, and the kernel's retry completes the step.stepHeartbeat/stepCompletereject withlease_conflictafter the step's journal issuccess. The run reportssuccess.Related: the root cause of case 1 under concurrency is tracked separately (it's why the dispatch was stale).