Repository navigation
cancel_job doesn't take effect during a model's loading phase, and the job shows no pending cancel #892
Description
Filed by tester (claude-opus-5-5 via anthropic) on mps on mini-ai.lan, develop @ dfc7c49, while verifying #883.
Tool: cancel_job / wait_for_job
Repro
run_workflow(name="templates/minimax/first-and-last-frame", arguments={first/last image = "asset:qa-cast/priya-portrait.jpg", "width": 512, "height": 288, "num_frames": 124, "seed": 42}, acknowledged_cost=true)started joba1264f45fe71. On this server,validate_workflowreportsdownloads_required:MiniMaxAI/MiniMax-H3464.2 GB andlightx2v/Minimax-h3-Turbo54.7 GB.- The job sat in
phase: loading,phase_detail: "pipeline: MiniMaxAI/MiniMax-H3"for over 13 minutes, emitting onlyphase_stallevents. cancel_job(job_id="a1264f45fe71")→{"id": "a1264f45fe71", "status": "running"}.wait_for_jobfor a total of 420 s afterwards → stillstatus: running,phase: loading,seconds_in_phase: 1237. Nothing in the reply shows a cancel was requested.
Expected: a cancel requested during loading, including a model download, takes effect within a bounded time. If loading genuinely can't be interrupted, the job reply should at least show the pending cancel (e.g. cancel_requested: true), and the cancel_job docstring should say loading isn't a boundary.
Actual: the job keeps the single GPU worker indefinitely, and the consumer can't tell its cancel was recorded.
A related question for the implementer: H3 can't be expected to load on a 64 GB unified-memory box. If validate or run could warn about that before starting a ~520 GB download, this job would never have been queued here.
Activity
dkackman commentedon Oct 14, 2026
Moved to lem's loop (Don's call): the repro runs
templates/minimax/first-and-last-frame, and H3 can't load on mini-ai, so the load-phase cancel has to be reproduced and verified on lem, where H3 is cached and loads. Claim swappedtarget:mini-ai→target:lem;backend:sharedkept, since cancel-during-load isn't CUDA code. The validate warning asked for at the end (H3 on a box too small for it) is still worth deciding in triage.dkackman commentedon Oct 14, 2026
triage: work — scope is the cancel path: a cancel during
loading(model load/download, which never reaches the worker's step-boundarycontext.cancel()check) must take effect in bounded time (e.g. terminate and respawn the worker after a grace period), and the job reply must show the pending cancel; update thecancel_jobdocstring. Not in scope: the closing "warn before H3 on a 64 GB box" ask — that is a newvalidate_workflowrule and would need Don's sign-off as its own issue; do not build it here. Verify on lem with an uncached/slow-loading pipeline. No duplicate found. (claude-opus-5-5 / anthropic)dkackman commentedon Oct 14, 2026
Fixed in 07f5612 (merged to develop, deployed to lem 04:38; server healthy). Worked as claude-sonnet-5-5 (anthropic).
What changed:
get_job/list_jobssummaries carrycancel_requested: truewhile a cancelled job is stillrunning.cancel_jobon a running job now starts a 30 s timer. If the job is then still inloadingor ataskphase (the phases with no checkpoint), the worker process is terminated and the job endscancelled. The next job respawns the worker with a cold model cache, so cached models are lost. In an interruptible phase (generating, etc.) nothing is killed; the cooperative cancel applies as before.cancel_jobnow says a load is killed after 30 s andcancel_requestedshows until then.Tests: real-path tests in tests/test_worker_pool.py (escalation in a loading phase, none in an interruptible phase, a dead worker under a requested cancel reads
cancelled). The full suite has 18 failures, including kandinsky6 template/skill tests that also fail on a clean develop. The last full run was before the final docstring trim, so the docstring-budget failure it showed is fixed but not re-run in full.To verify on lem: start a slow or uncached load, call
cancel_job, and check thatcancel_requested: trueshows at once and the job readscancelledabout 30 s later. Not run on a real model load myself.Proposed regression case (
regression-suite-complete.md): that same call sequence, expectingcancelledwithin about 60 s.dkackman commentedon Oct 14, 2026
Architecture review queued by the loop driver: this hand-off changed
dw/ordw_mcp/ondevelop(b25f6b407a..31e4501304: dw/acceleration_warnings.py dw/server/job_record.py dw/server/job_results.py dw/server/jobs.py dw/validation.py dw/worker_manager.py dw_mcp/tools_jobs.py). The reviewer checks it againstdocs/ARCHITECTURE.mdbefore the tester verifies.dkackman commentedon Oct 14, 2026
Architecture review: pass (claude-opus-5-5 via anthropic).
Read:
a6c9a5a3, the only #892 commit inb25f6b40..31e45013(dw/server/job_record.py, dw/server/job_results.py, dw/server/jobs.py, dw/worker_manager.py, dw_mcp/tools_jobs.py), againstdocs/ARCHITECTURE.mdand stage-c-guardrails item 3.NON_INTERRUPTIBLE_PHASESfromdw/events.py, which owns it, and doesn't redefine the list. Cancel stays inJobManagerand the worker process stays inWorkerManager. The crash→CANCELLEDmapping is inconsume_results, which the Job queue row assigns to fold worker outcomes. The Worker pool dispatch row's rule ("cancel or a crash touches only the worker running that job") still holds:_terminate_if_stuckterminates onlyslot.managerfor the job's own slot.threading.Timer, and nothing inpyproject.tomlcovers it.JobManager._escalate_cancel,WorkerManager.terminate_worker). NoCLAUDE.mdchanged.Non-blocking note:
WorkerManager.terminate_workerrepeats the terminate → join → kill sequence thatshutdown_workeralready runs, in the same module. It isn't a second owner, but whoever next edits worker teardown could factor it into one helper.Nothing was waived (the issue has no
arch-approved). Back to the tester for the lem verify.dkackman commentedon Oct 14, 2026
Verified on lem (develop @ fb829e7) by the tester, claude-opus-5-5 via anthropic.
Load-phase cancel (the fix):
validate_workflow(name="templates/interpolate-frames")→downloads_required: genmo/mochi-1-preview 124.3 GB(uncached on lem).run_workflow(...)with a bound ack (downloads: ["genmo/mochi-1-preview"],wait_seconds=10) → job725a5ccd94a4,phase: loading,phase_detail: "downloading genmo/mochi-1-preview: 149.5 MB, 29.9 MB/s".cancel_job("725a5ccd94a4")→{"status": "running"}. Straight afterwards,get_job→status: running,phase: loading,cancel_requested: true.list_jobs(limit=1)also showscancel_requested: true.wait_for_job(timeout_seconds=90)→status: cancelledafter 29.1 s of waiting, about 31 s after the cancel.finished_at − started_at= 42.7 s.error: null, empty manifest.get_healthafterwards →status: ok. cuda:0 showsalive: false(the killed worker; on-demand, so it respawns with the next job), and cuda:1 was untouched.Adjacent case, interruptible phase (no kill expected): I ran
templates/minimax/video-with-audiotwice: warm, then afterclear_memory. On lem the H3 load takes about 5 s (events:loadingat 0 s, LoRA at 4 s,generatingat 5 s), so both cancels landed ingenerating. Both jobs readcancelledwithin 6 s and 12 s, through the cooperative path. Neither was killed: cuda:0 stayed alive between them.The
cancel_jobdocstring now says "a load is killed after 30 s (cancel_requestedtill then)". ✔Notes, not blocking:
wait_for_job/run_workflow(wait_seconds)replies doesn't carrycancel_requested, thoughget_job/list_jobsdo. A consumer that only waits sees justrunninguntil the kill, which is at most about 30 s.genmo/mochi-1-previewin lem's HF cache.No regression case added: the proposed sequence needs an uncached model, and on lem that means starting a 124 GB download on every run, which also leaves partial cache behind. That fits a deliberate one-off verify, not a recurring suite case.