Repository navigation
Step cache doesn't survive a killed worker process (disk-backed persistence + resume) #244
Description
When a for_each job (e.g. tool-musical-trailer's 12-entry shot step) is killed partway through, rerun_job restarts from the first entry (shot@s01) rather than resuming after the last completed member.
Suggested fix: commit each for_each member's step-cache entry as soon as that member completes, rather than only at job completion. This is what would make a mid-run failure (OOM kill, manual cancel, crash) cheap to recover from instead of re-paying the full cost of every already-finished member.
Activity
dkackman commentedon Sep 19, 2026
triage: escalate — the premise is off, and the real fix is an engine change.
step_cache.putalready runs per step, immediately after each step saves (dw/workflow.py ~L1319), and afor_eachmember is an ordinary step after expansion — so completed members are committed as they finish. What is lost is the cache itself:dw/step_cache.pyis in-memory in the worker process, so a SIGKILL (or a cancel that terminates the worker) takes every entry with it, and the rerun starts fromshot@s01. Also the cache is disabled entirely when the workflow sets noseed. Resuming a killed job means either a disk-backed step cache or rebuilding hits from the previous run's manifest + files on rerun — a new engine concept (partial-run resume) that needs sign-off on semantics (what counts as a valid hit after a crash, cache invalidation, interaction withnew_seed). (triage by implementer agent, model opus via provider anthropic)dkackman commentedon Sep 19, 2026
Split per discussion with Don:
Result(not just file paths) from disk for any downstream step whose result is still needed after restart — per media type (image/video/audio/text), since a fresh process has nothing in memorynew_seedreruns and the existing file-existence/generation invalidation checks instep_cache.py, which should carry over largely as-isStaying parked at
owner:don/status:needs-approvalpending a written proposal on the above.[-]A killed for_each job doesn't commit completed members' step cache[/-][+]Step cache doesn't survive a killed worker process (disk-backed persistence + resume)[/+]dkackman commentedon Sep 22, 2026
More evidence from Don's dw findings ledger (DW-15,
tool-musical-trailer): the same kill, logged independently. Triage against develop f11ac2a found nothing new beyond this issue's existing analysis. (claude-opus-5-5 via anthropic, filing session)dkackman commentedon Sep 23, 2026
Linking the design: resume.md scopes this issue's fix as a feature: rehydrate the existing step cache from run-directory manifests, rather than building a second cache. A run is resumable only if it's reproducible, so the seed changes are a prerequisite.
todo.mdranks it Tier 3: meaningful for expensive multi-step runs, but a correctness-sensitive engine change.Labeled
featureandpriority:3as part of migratingdocs/proposals/to issues (harnest roadmap R11). It stays parked with Don. Starting it means handing it to the feature lead rather than to the implementer.claude-opus-5-5 via anthropic, interactive session with Don.
docs(proposals): retire the two shipped proposals