sdk: a stale lease error from the local LLM worker kills the whole run, though the kernel already retried the step

evidence/worker-lease-lost/README.md

@@ -0,0 +1,77 @@
+# Worker lease refusal evidence
+
+All commands run from the repository root unless the transcript starts with
+`cd packages/sdk`. Each raw transcript includes the command and captured output.
+
+The SDK tests inject stale dispatches and refusals; they do not claim to reproduce
+the separate concurrent-dispatch root cause or to verify kernel lease expiry.
+The live tests use the repository-built daemon and assert actual `step.completed`
+and `run.completed` success entries before injecting completion/heartbeat refusals.
+
+## Mutation checks
+
+Run `python3 evidence/worker-lease-lost/mutate.py` from the repository root.
+The script saves each source file as bytes, applies the exact change in
+[mutations.txt](mutations.txt), captures failure, restores those bytes in a
+`finally` block, checks byte equality, and captures the passing rerun.
+The ledger includes SHA256 values for the restored source.
+
+The fatal-policy check deliberately removes the retained fatal callback. It is
+a negative-control mutation, not a claim that fatal behavior was newly added.
+The terminal-only check removes terminal-refusal support separately from the
+whole-filter reversion.
+
+| Check | Command (in packages/sdk) | Mutated output | Restored output |
+|---|---|---|---|
+| filter | `npx vitest run tests/worker-lease-lost.test.ts` | [filter-mutant.txt](filter-mutant.txt) | [filter-restored.txt](filter-restored.txt) |
+| terminal | `npx vitest run tests/worker-lease-lost.test.ts -t run_terminal` | [terminal-mutant.txt](terminal-mutant.txt) | [terminal-restored.txt](terminal-restored.txt) |
+| fatal | `npx vitest run tests/worker-lease-lost.test.ts -t 'non-lease worker error'` | [fatal-mutant.txt](fatal-mutant.txt) | [fatal-restored.txt](fatal-restored.txt) |
+| live | `npx vitest run tests/worker-lease-lost-live.test.ts` | [live-mutant.txt](live-mutant.txt) | [live-restored.txt](live-restored.txt) |
+| sweep | `npx vitest run tests/worker-lease-sweep.test.ts` | [sweep-mutant.txt](sweep-mutant.txt) | [sweep-restored.txt](sweep-restored.txt) |
+| direct | `npx vitest run tests/direct-run-worker-lease.test.ts` | [direct-mutant.txt](direct-mutant.txt) | [direct-restored.txt](direct-restored.txt) |
+| resume | `npx vitest run tests/resume-worker-lease.test.ts` | [resume-mutant.txt](resume-mutant.txt) | [resume-restored.txt](resume-restored.txt) |
+
+## Full checks
+
+- [Final npm test](npm-test-final.txt): final source, Rust on PATH, explicit
+  RELAYFLOWD_BIN pointing at this checkout's build. Full captured output.
+- [New test typecheck](new-test-types.txt): `npm run typecheck:tests` from
+  packages/sdk. That config enumerates its files by name rather than globbing,
+  so every new regression file is listed in it; the gate covers them.
+- [Sandbox probe](sandbox-probe.txt): direct OS isolation probe.
+
+## Development transcripts
+
+- [Initial npm test](npm-test-initial.txt): superseded development run, without
+  RELAYFLOWD_BIN. This was started before the final test/source corrections;
+  it includes interim CLI deadline and malformed-deadline failures.
+  Use the final run for review.
+- [Sweep before the fix](sweep-before.txt): reproduces immediate executor failure
+  on an expired running snapshot.
+- [First live completion check](live-first.txt): the two completion-refusal
+  variants before adding the real-kernel heartbeat case. The restored live
+  transcript above includes all three.
+
+## Baseline comparison for the live-kernel suite
+
+`python3 evidence/worker-lease-lost/baseline.py` replaces only the SDK source
+files this change touches with their bytes at the **rebased parent**
+(`git merge-base HEAD origin/main`; `BASE=<rev>` overrides), rebuilds, and
+runs the unchanged live-kernel suite. It restores the implementation
+byte-for-byte in `finally`, asserts equality, and rebuilds it. Each transcript
+records the base it used.
+
+Rerun after the rebase onto `2e2043f` (macOS, with `packages/sdk` built against
+the workspace `@relayflows/surface`, as CI does):
+
+- [Baseline build](baseline-build.txt): exit 0.
+- [Baseline live-kernel](baseline-live-kernel.txt): `Tests 31 passed (31)` at
+  base `2e2043f`.
+- [Live-kernel at this head](live-kernel-head.txt): `Tests 31 passed (31)`.
+- [Restored implementation build](restored-build.txt): exit 0.
+
+The eight live-kernel failures the factory first reported came from its
+sandbox, not from either revision. `/home/daytona/package.json` sits above the
+checkout and declares `"type": "commonjs"`, which breaks
+`testdata/preflight`'s extensionless ESM fixtures. On a clean machine the
+suite passes before and after this change.

evidence/worker-lease-lost/baseline-build.txt

@@ -0,0 +1,8 @@
+# base 2e2043f4c82f1a5c83b090d8a3a82c2d067a79fe
+$ cd packages/sdk && npm run build
+
+> @relayflows/sdk@2.0.30 build
+> tsc && node scripts/make-cli-executable.mjs
+
+
+Exit code: 0

evidence/worker-lease-lost/baseline-live-kernel.txt

@@ -0,0 +1,44 @@
+# base 2e2043f4c82f1a5c83b090d8a3a82c2d067a79fe
+$ cd packages/sdk && npx vitest run tests/live-kernel.test.ts
+
+ RUN  v2.1.9 /Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk
+
+stdout | tests/live-kernel.test.ts
+LIVE_KERNEL relayflowd=/Users/khaliqgant/.relayflows-toolchain/target/1166253295/debug/relayflowd
+LIVE_KERNEL flows=/Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk/dist/cli.js
+
+stdout | tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI
+LIVE_ANALYZER ready: claude -p --model claude-haiku-4-5-20251001 round-trip OK
+
+stdout | tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI
+LIVE_ANALYZER analysis: {"reasoning":"This story is directly about an AI agent performing sophisticated automation tasks—opening pull requests and conducting code reviews—which is core to agent-based development workflows and directly relevant to the Relayflow project's focus on agent automation and workflow orchestration.","relevance_score":9,"story_title":"Show HN: an agent that opens and reviews its own pull requests [wake-nonce-7f3a91c4]"}
+
+stdout | tests/live-kernel.test.ts > surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once
+LIVE_KERNEL kill -9 pid=91432 run=01M36HY65T82B19WXM8827FP7X while step=two state=Running
+
+ ✓ tests/live-kernel.test.ts (31 tests) 70036ms
+   ✓ built flows CLI against live relayflowd > twenty-six-step reuses 25 durable completions after editing the failed final step 1666ms
+   ✓ built flows CLI against live relayflowd > runs rung (a), parks rung (b), and keeps JSON report-shaped 4563ms
+   ✓ built flows CLI against live relayflowd > allows a deterministic run to exceed the bounded request timeout 32336ms
+   ✓ built flows CLI against live relayflowd > follows a live worker dispatch through flows run 2296ms
+   ✓ built flows CLI against live relayflowd > runs an agent CLI end to end through the SDK worker 1076ms
+   ✓ built flows CLI against live relayflowd > f.agent lowers to a real agent step and dispatches through a live worker 553ms
+   ✓ built flows CLI against live relayflowd > f.agent's default flowPath anchors on cwd, not cwd's parent 367ms
+   ✓ built flows CLI against live relayflowd > can always get a parked run to a late-attaching worker 5566ms
+   ✓ built flows CLI against live relayflowd > reports a real manual-recovery NeedsHuman state as parked 1388ms
+   ✓ built flows CLI against live relayflowd > AgentWorker passes a declared model to an identified wrapper as RELAYFLOW_MODEL 426ms
+   ✓ built flows CLI against live relayflowd > AgentWorker refuses a nonconforming journal-submitted wrapper before exposing RELAYFLOW_MODEL 422ms
+   ✓ built flows CLI against live relayflowd > AgentWorker executes the raw claude adapter with its real model flag 436ms
+   ✓ built flows CLI against live relayflowd > AgentWorker executes the raw codex adapter with its real model flag 415ms
+   ✓ built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI 11511ms
+   ✓ built flows CLI against live relayflowd > preflights before journaling and names an unreachable socket 2199ms
+   ✓ built flows CLI against live relayflowd > starts exactly one daemon when two runs race for one empty data dir 937ms
+   ✓ surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once 1708ms
+
+ Test Files  1 passed (1)
+      Tests  31 passed (31)
+   Start at  00:14:52
+   Duration  71.50s (transform 549ms, setup 0ms, collect 1.25s, tests 70.04s, environment 0ms, prepare 52ms)
+
+
+Exit code: 0

evidence/worker-lease-lost/baseline.py

@@ -0,0 +1,23 @@
+from pathlib import Path
+import subprocess,os
+root=Path.cwd()
+# The rebased parent, not a hard-coded commit: BASE=<rev> overrides.
+base=os.environ.get('BASE') or subprocess.check_output(['git','merge-base','HEAD','origin/main'],text=True).strip()
+files=subprocess.check_output(['git','diff','--name-only',base,'HEAD','--','packages/sdk/src'],text=True).splitlines()
+originals={name:(root/name).read_bytes() for name in files}
+env={**os.environ,'PATH':os.path.expanduser('~/.cargo/bin')+':'+os.environ['PATH']}  # RELAYFLOWD_BIN, if needed, comes from the caller
+def run(cmd,name):
+ with (root/'evidence/worker-lease-lost'/name).open('w') as f:
+  f.write('# base '+base+'\n$ cd packages/sdk && '+cmd+'\n');f.flush()
+  result=subprocess.run(cmd,shell=True,cwd=root/'packages/sdk',env=env,stdout=f,stderr=subprocess.STDOUT)
+  f.write('\nExit code: '+str(result.returncode)+'\n')
+ return result.returncode
+try:
+ for name in files:
+  (root/name).write_bytes(subprocess.check_output(['git','show',base+':'+name]))
+ assert run('npm run build','baseline-build.txt')==0
+ run('npx vitest run tests/live-kernel.test.ts','baseline-live-kernel.txt')
+finally:
+ for name,data in originals.items(): (root/name).write_bytes(data)
+ assert all((root/name).read_bytes()==data for name,data in originals.items())
+ assert run('npm run build','restored-build.txt')==0

evidence/worker-lease-lost/direct-mutant.txt

@@ -0,0 +1,33 @@
+$ cd packages/sdk && npx vitest run tests/direct-run-worker-lease.test.ts
+
+ RUN  v2.1.9 /Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk
+
+ ❯ tests/direct-run-worker-lease.test.ts (2 tests | 1 failed) 7ms
+   × direct run handles LLM errors with leaseLost=true 4ms
+     → expected '{"ok":false,"command":"run","resoluti…' to contain 'connection closed'
+
+⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
+
+ FAIL  tests/direct-run-worker-lease.test.ts > direct run handles LLM errors with leaseLost=true
+AssertionError: expected '{"ok":false,"command":"run","resoluti…' to contain 'connection closed'
+
+Expected: "connection closed"
+Received: "{"ok":false,"command":"run","resolutions":[],"diagnostics":[{"severity":"failure","kind":"protocol_error","message":"relayflowd could not complete the run request: lease_conflict: lost"}],"path":"flow.ts","socketPath":"/var/folders/6d/0x5fkt8d01gfmmjdzkxqzwnh0000gn/T/relayflowd-4d1d0e012c91.sock"}"
+
+ ❯ tests/direct-run-worker-lease.test.ts:33:41
+     31|   const result = await runDirectFlow('flow.ts', '{}', '/unused', { loc…
+     32|   expect(result.exitCode).toBe(1);
+     33|   expect(JSON.stringify(result.report)).toContain(leaseLost ? 'connect…
+       |                                         ^
+     34| });
+     35| 
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
+
+ Test Files  1 failed (1)
+      Tests  1 failed | 1 passed (2)
+   Start at  00:14:32
+   Duration  707ms (transform 284ms, setup 0ms, collect 519ms, tests 7ms, environment 0ms, prepare 42ms)
+
+
+Exit code: 1

evidence/worker-lease-lost/direct-restored.txt

@@ -0,0 +1,13 @@
+$ cd packages/sdk && npx vitest run tests/direct-run-worker-lease.test.ts
+
+ RUN  v2.1.9 /Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk
+
+ ✓ tests/direct-run-worker-lease.test.ts (2 tests) 4ms
+
+ Test Files  1 passed (1)
+      Tests  2 passed (2)
+   Start at  00:14:33
+   Duration  793ms (transform 282ms, setup 0ms, collect 599ms, tests 4ms, environment 0ms, prepare 26ms)
+
+
+Exit code: 0

evidence/worker-lease-lost/fatal-mutant.txt

@@ -0,0 +1,53 @@
+$ cd packages/sdk && npx vitest run tests/worker-lease-lost.test.ts -t 'non-lease worker error'
+
+ RUN  v2.1.9 /Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk
+
+ ❯ tests/worker-lease-lost.test.ts (16 tests | 2 failed | 14 skipped) 6ms
+   × agent stale lease subscriber > keeps a non-lease worker error fatal with its original identity 5ms
+     → expected "spy" to be called with arguments: [ Error: cli exploded ]
+
+Received: 
+
+
+
+Number of calls: 0
+
+   × llm stale lease subscriber > keeps a non-lease worker error fatal with its original identity 1ms
+     → expected "spy" to be called with arguments: [ Error: cli exploded ]
+
+Received: 
+
+
+
+Number of calls: 0
+
+
+⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯
+
+ FAIL  tests/worker-lease-lost.test.ts > agent stale lease subscriber > keeps a non-lease worker error fatal with its original identity
+ FAIL  tests/worker-lease-lost.test.ts > llm stale lease subscriber > keeps a non-lease worker error fatal with its original identity
+AssertionError: expected "spy" to be called with arguments: [ Error: cli exploded ]
+
+Received: 
+
+
+
+Number of calls: 0
+
+ ❯ tests/worker-lease-lost.test.ts:79:19
+     77|     client.emit('step.dispatch', dispatch);
+     78|     await worker.close();
+     79|     expect(fatal).toHaveBeenCalledWith(error);
+       |                   ^
+     80|     expect(client.close).toHaveBeenCalledWith(error);
+     81|     expect(warning).not.toHaveBeenCalled();
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯
+
+ Test Files  1 failed (1)
+      Tests  2 failed | 14 skipped (16)
+   Start at  00:14:22
+   Duration  529ms (transform 150ms, setup 0ms, collect 334ms, tests 6ms, environment 0ms, prepare 43ms)
+
+
+Exit code: 1

evidence/worker-lease-lost/fatal-restored.txt

@@ -0,0 +1,13 @@
+$ cd packages/sdk && npx vitest run tests/worker-lease-lost.test.ts -t 'non-lease worker error'
+
+ RUN  v2.1.9 /Users/khaliqgant/Projects/AgentWorkforce/flows/packages/sdk
+
+ ✓ tests/worker-lease-lost.test.ts (16 tests | 14 skipped) 4ms
+
+ Test Files  1 passed (1)
+      Tests  2 passed | 14 skipped (16)
+   Start at  00:14:23
+   Duration  545ms (transform 159ms, setup 0ms, collect 358ms, tests 4ms, environment 0ms, prepare 58ms)
+
+
+Exit code: 0