Skip to content

sdk: a stale lease error from the local LLM worker kills the whole run, though the kernel already retried the step #560

Description

@khaliqgant

Problem

packages/sdk/src/cli/direct-run.ts:85 treats any error from the local LlmWorker as fatal:

localLlm.on('error', error => { llmFailure = error; client.close(); });

So when the worker hits a lease error on an attempt the kernel has already expired and retried, the whole run ends as protocol_error and can't be resumed. The retry succeeds in the journal anyway. Seen two ways in relayflows 2.0.29:

  1. Stale dispatch. The worker receives a dispatch after its lease deadline. withWorkerLease throws Agent lease is already expired for <run>/<step>, and that becomes fatal. The kernel had already journaled lease_expired → retry, and attempt 2 completed success. Journal: 00-job1-attempt2-parallel-lease-expired.txt; the replay is quoted in feat(examples): prompt-lab — the Prompt Lab product brief as one relayflow #559.
  2. Renewal racing completion. A step whose child run journaled step.completed success + run.completed success was reported as lease_conflict: attempt has no active worker lease, and the run died. Intermittent: once in about 50 sequential calls. 00-prove-attempt1-lease-conflict-after-success.txt

Scope

  • A lease refusal for an attempt the kernel no longer owns shouldn't abort the run. Examples: already expired before execute, lease_conflict / no active worker lease on heartbeat or complete. The kernel owns that attempt's fate (retry, or a completion that already landed). Drop it with a diagnostic (WAITING/warning line); don't close the client.
  • Any other worker error stays fatal, exactly as today. Fail closed.
  • The agent worker path (AgentWorker) likely has the same shape. Check it and apply the same rule if so.

Acceptance

  • A unit test injects a dispatch whose lease_deadline_ms is already past. The run continues, and the kernel's retry completes the step.
  • A unit test makes stepHeartbeat / stepComplete reject with lease_conflict after the step's journal is success. The run reports success.
  • A non-lease worker error still ends the run as today (regression test).
  • Mutation-verified per AGENTS.md.

Related: the root cause of case 1 under concurrency is tracked separately (it's why the dispatch was stale).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    garden-readyScoped and ready for an agent to pick up

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions