bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed

What happened

2026-10-13 ~02:15-02:24 on lem (125 GB host RAM): two H3 int8 jobs dispatched to the two cards at once.

7cba28c2095b (H3Q1_base_int8_30step, cuda:0, submitted first): "Worker process died: killed by SIGKILL (typically the out-of-memory killer)"
7cbac071da26 (H3Q4_turbo8_int8, cuda:1, submitted ~9 min later): "Worker process died: killed by SIGTERM"

An int8 H3 Ref2VA job holds ~62-63.5 GB RSS steady after load (host_memory_rss_mb 63471 in job 27b9d0bff514's memory event at +44 s; transformer_ref quantized cache alone is 34 GB, text encoder 35.6 GB on disk). Two of them don't fit 125 GB.

Two problems

Dispatch doesn't gate on host RAM. server/admission.py computes vram_need and the pool routes on it (idea: support multiple GPUs for concurrent workload execution #462), and host_memory_projection only produces validate warnings. The second job should have waited (or been refused for that card) when projected RSS of the running job + the new one exceeds available host memory.
The wrong job died. _rank_for_oom in server/jobs.py raises the later-started worker's oom_score_adj so "a host-RAM squeeze kills the later-started worker - one job - rather than whichever is largest". Here the earlier job (Q1) got SIGKILL. Possibilities: the cuda:1 worker was already alive from an earlier job (so it was not "this call just started" and kept a lower score), or the ranking ran after the squeeze. Then Q4 also died (SIGTERM) - unclear by whom; maybe the pool restarting after the first death.

Expected

The second int8 job queues until the first finishes (or the agent is told at submit that it will wait for host RAM). If a squeeze happens anyway, the later job dies and the earlier survives.

Workaround in use

Run int8/bf16 H3 one at a time (tools/ladder.py waits for lem to be idle). int4 jobs (~35 GB) do run two at once.

Questions

Use observed peak RSS per (pipeline, weights_dtype) from observed_cost as the projection, or host_memory_projection's fit?
Should a waiting-for-host-RAM job show a distinct status so the agent can quote the delay?

Source: H3 quality matrix on lem (2x RTX 3090 24 GB, 125 GB host RAM), workspace h3-quality, Known Issues ep 6 shots, seed 42. Full data and per-run assessments: ~/testing/opus/known-issues/quality/matrix.json, report quality/REPORT.md; every run in h3-quality carries a note with its config, minutes and score. Related: #861 (follow-up experiments).

triage: escalate — part 1 adds a new dispatch gate on host RAM in admission/the pool. That needs a choice of projection (observed peak RSS vs host_memory_projection's fit) and maybe a new waiting status consumers would see. Both are open questions in the issue. Part 2 (_rank_for_oom picking the wrong victim) is a self-contained bug and could go to a fix session on its own if you'd rather split it off. (claude-opus-5-5, anthropic)

Review: split. Part 2 is now #881, going to a fix session. Part 1 goes to the lead for a plan.

Part 2 (wrong job died). The cause is in _rank_for_oom. Its +100 bump is only about 12.5 GB of OOM badness on a 125 GB host, so a 63 GB earlier job still outscores a smaller job that is still loading. It also ranks once per worker pid, not per dispatch. The fix: on every dispatch, set the dispatching worker to 1000 and the others to 0. Details, plus the SIGTERM investigation (the kernel's OOM killer never sends SIGTERM), are in #881.

Part 1 (dispatch gate on host RAM): approved for planning. Answers to the open questions:

Projection: use the observed host_memory_job_peak_rss_mb per (pipeline, weights dtype) from history first, then the host_memory_projection fit. If neither has a figure, don't gate, as today.
Status: no new status. The job stays queued, and its warnings and events carry the reason, e.g. "waiting for host RAM: needs ~63 GB, ~60 GB free". A new status would ripple through the UI, the API models and MCP for something a reason string already says.
When this ships, idea: sweep runs + experiment tracking - one shot across N configurations, with structured assessments in notes #872's sweep guide drops its "wait between variants" sentence.

claude-opus-5-5 / anthropic, for Don

Spec for plan v2. All cases are in regression-suite-complete.md (harnest), each marked pending: for its stage.

#909 (A: memory key + projected host-RAM peak, no gating)

C-F486: an observed figure after a seeded run, the same figure for inline and renamed copies, none for a different offload, no waiting_for/queue_wait (dispatch unchanged)
C-F487: H3 weights_dtype int4 and int8 are distinct keys, and a never-run dtype has no figure
C-F488: a list workflow with no same-key run gives basis: "fit", matching the projection warning
C-F489: no figure means no host_memory key and no error (a new workflow, a task-only one, pre-column history)
C-F490: a cancelled (and optionally failed) run never feeds the figure

#910 (B: host-RAM dispatch gate, queue_wait, waiting_for)

C-F491: the second int8 H3 job is held with a card idle; waiting_for appears on get_job and list_jobs; one queue_wait per hold; the job starts after the first finishes; all succeed; nothing lands in warnings
C-F492: a small job backfills past the held one; cancelling a held job clears its waiting_for and leaves the other held job intact
C-F493: two int4 jobs start at once
C-F494: a never-run workflow is never held, and neither is a lone job (the escape)
C-F495: on a single-worker server or MPS, dispatch is unchanged
C-F496: the served guide and the minimax-h3 skill reference drop "One int8 job at a time" / "wait between variants"

Notes (not blocking):

C-F488 depends on acorn-wars/shots-batch keeping only pre-column list history (M-F029). It reports itself skipped, not failed, if that history is gone or has gained a same-key run.
C-F491 to C-F494 are lem-only, need an idle box and cost roughly 2-3 GPU-hours together. C-F492 and C-F494 piggyback on C-F491's and C-F493's hold windows.
SERVER.md's wording isn't served, so it needs docs-review. I left it out of the cases.
The needed-GB tolerance in C-F491 is ±10% of projected_mb. If the message rounds differently, say so at hand-off.

— tester, claude-opus-5-5 via anthropic

Merge feat/ui-ux-improvements into develop

Merge docs/869-closeout (#869)

Close-out (claude-opus-5-5, anthropic): both stages are closed and verified on lem. #909 (A: memory key and projected host-RAM peak) and #910 (B: the dispatch gate, queue_wait and waiting_for).

Design record: docs/proposals/complete/host-ram-dispatch-gate-complete.md, merged to develop as 36e20e7 (commit 215688d). It covers what was built, what was deferred and why, and the bounces per stage. Also in that commit:

the todo.md shipped entry;
the multi-gpu record's deferred host-RAM gate is marked shipped;
series-episodes/references/cinema-path.md loses a stale "One int8 job at a time" that stage B missed.

Docs only, so no deploy. C-F495 (single worker or MPS) was never run on lem and still carries pending: #910.

verify: run every case whose source: names one of this feature's stages, and close the parent if they all pass.

Feature-parent verify, progress so far — claude-opus-5-5 via anthropic, lem on develop (stages #909/#910 deployed).

Free / cheap cases whose source: names #909 or #910, all run over MCP this session in workspace regression-complete:

C-F486 PASS. c-f486-t2i (text-to-image, seed 42): validate before → host_memory {3347, observed, runs 11}. Ran J1 8e8e51e217fe (succeeded, no waiting_for, no queue_wait event). P = max host_memory_job_peak_rss_mb = 3215. Re-validate → {3347, observed, runs 12} (≥ P, not > 2×P). Inline W gives the identical {3347, observed, 12}. W2 (offload model) → valid, no host_memory key.
C-F487 PASS. reference-to-video seed 42: {int4} → {47481, observed, 11}; {int8} → {89391, observed, 4}; {bf16} → no host_memory. Distinct keys. (Note: int8 at ~87 GB is well over the ~63 GB in the issue body; not a C-F487 finding.)
C-F488 PASS. 5- and 12-entry list both {61566, fit, runs 7}, no warnings.
C-F489 PASS (calls 1-2), call 3 skipped. Inline t2i with offload model → valid, no host_memory, no host-memory warning (sequential already had 1 observed run on this box, so I used the other enum value). qr_code task-only → valid, no host_memory. Call 3 not run: I didn't find a named non-list workflow with pre-column history and no run since stage A.
C-F490 PASS. c-f490-t2i (offload model, 200 steps): no host_memory before; J1 48ac33b942f5 cancelled at denoise step 9/201 → status cancelled; re-validate still has no host_memory. Failed arm not run.
C-F496 PASS. Workspaces guide "Running a sweep" and the minimax-h3 skill reference describe the host-RAM hold / waiting_for; neither forbidden phrase present.

Case-wording nit (not a server finding): C-F486/C-F490 say run_workflow(workflow="<name>"), but workflow is the inline-JSON parameter; a saved name goes in name.

Next: the lem-only dispatch cases C-F491–C-F494. C-F495 is single-worker only and is skipped on lem (stays pending: #910).

Feature-parent verify, final: all cases pass. Run by claude-opus-5-5 via anthropic against lem on develop. Every call was made over MCP this session in workspace regression-complete. The earlier progress comment has C-F486–C-F490 and C-F496.

The lem dispatch cases (2× RTX 3090, 125 GB host RAM):

C-F491 PASS.
- J1 05bb7dad9239 (int8, seed 42) ran alone on cuda:0 and succeeded in 431 s. Re-validate gave M = 94950 MB (observed, 5 runs).
- J2 545371b214f7 started on cuda:0 at once.
- J3 9687dbfce77d stayed queued while cuda:1 was idle. Both get_job and list_jobs showed waiting_for: "waiting for host RAM: needs ~93 GB, ~60 GB free" (later "~59 GB free"). It had exactly one queue_wait event (reason: host_ram, same message), and that was still one on two re-checks minutes apart. warnings: [].
- J2 finished at 1792071022.220 and J3 started at 1792071022.228, also on cuda:0. J3 succeeded with warnings: [] and still exactly one queue_wait. Nothing was killed.
C-F492 PASS.
- J4 d3366af99adb (text-to-image) backfilled onto cuda:1 past the held J3. It succeeded with no queue_wait.
- J5 90a092528653 (int8) was held: one queue_wait, "needs ~93 GB, ~59 GB free".
- After cancel_job(J5): status cancelled, started_at/device/run_dir null, no waiting_for. Its events are queued → queue_wait → cancelled, with no load phase. It was gone from list_jobs(status="queued,running"), and J3 was still queued with waiting_for intact.
C-F493 PASS.
- N = {47481, observed, runs 11} for int4, which is below M. K1 was skipped because int4 already had 11 observed runs, and seeding that history is all K1 does.
- K2 8f0a3c3b016a (cuda:0) and K3 3df6483eace5 (cuda:1) were submitted back to back and both started at once. Neither had waiting_for or any queue_wait event, and both succeeded.
- K3's only warning is an audio-clipping notice (+0.72 dBFS), which has nothing to do with host RAM.
C-F494 PASS.
- L1 78a1de7051e7 was an inline t2i with offload: "model". Validate gave no host_memory. It ran while J2 held cuda:0 and J3 was host-RAM-held: it started on cuda:1 at once, had no queue_wait, and succeeded.
- L2 aff3f5842861 was a lone int8 run on the idle box. It started on cuda:0 immediately with no waiting_for. I then cancelled it to free the box for C-F493, since the case only checks the start.
C-F495 not run. It is single-worker only and lem has two workers, so it stays pending: #910.

Deviations from the cases as written:

J2, J3 and J5 used seeds 43, 44 and 45. With seed 42 again the step cache would have served them (cached_steps: 1). The memory key is the same.
K2 and K3 used seeds 47 and 48.
C-F489 call 3 was skipped, as noted in the progress comment.

Metrics:

M is 94950 MB after J1, and 98489 MB (7 runs) after J2/J3.
The hold message said needs ~93 GB, ~59–60 GB free.
N = 47481 MB.

Cleanup: deleted the outputs of J1–J4, L1, L2, K2 and K3. J5 never ran, so it has no run directory.

Navigation Menu

Repository navigation

bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869

What happened

Two problems

Expected

Workaround in use

Questions

21 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869

Description

What happened

Two problems

Expected

Workaround in use

Questions

Activity

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

21 remaining items

dkackman commented on Oct 14, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions