bug: OOM ranking kills the earlier job - +100 adj is too small and ranking is per worker, not per dispatch

Split from #869, part 2. When the host runs out of RAM, the kernel kills the earlier-dispatched job instead of the later one.

Cause

_rank_for_oom (dw/server/jobs.py:863) has two flaws:

The +100 bump is too small. The kernel's badness score is roughly RSS plus oom_score_adj × total_RAM / 1000. On lem's 125 GB, +100 is worth about 12.5 GB. In bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869, Q1 (dispatched first) held about 63 GB while Q4 was still loading and smaller. Q4's 12.5 GB bonus didn't close that gap, so Q1 was the job killed.
It ranks per worker pid, not per job. The pid == slot.ranked_pid early return means a long-lived worker keeps the score it got when it started. The ranking follows worker start order, not job dispatch order.

Fix

On every dispatch with more than one live worker:

set the dispatching worker's oom_score_adj to 1000;
set every other live worker back to 0.

With 1000 the newest job outranks the others whatever their RSS. Resetting to 0 needs no privilege: the kernel refuses an unprivileged write only below oom_score_adj_min, and an unprivileged write never moves that minimum. Leave the single-worker case untouched, as today: write nothing.

Also investigate

Q4 died by SIGTERM, which the kernel's OOM killer never sends. Check lem's journal for 2026-10-13 02:15-02:24 for earlyoom, systemd-oomd or a pool restart, and report what sent it. If a userspace OOM daemon is running, say whether it honors oom_score_adj.

Acceptance

Unit tests:

two slots, two dispatches: the second dispatch's worker gets 1000 and the first gets 0;
a third dispatch back onto the first slot, whose worker pid is unchanged, flips the scores;
with one live worker, nothing is written.

On lem, the tester checks /proc/<pid>/oom_score_adj for both workers after back-to-back dispatches.

Filed by claude-opus-5-5 / anthropic, for Don

triage: work — not on develop (_rank_for_oom still early-returns on slot.ranked_pid, dw/server/jobs.py:863). Self-contained: per-dispatch 1000/0 ranking + the three unit tests, plus the lem journal check for the SIGTERM sender. Split from #869, not a duplicate. (claude-opus-5-5, anthropic)

docs: #881 - architecture map names the renamed OOM ranking test

Fixed on develop (60b1982, plus 39d5969 for the architecture-map test name), deployed to lem at 39d5969, healthy 02:58.

What changed (claude-sonnet-5-5 / anthropic): _rank_for_oom (dw/server/jobs.py) no longer early-returns on ranked_pid (field removed). On every dispatch with more than one live worker it writes oom_score_adj 1000 to the dispatching worker and 0 to every other live worker. With one live worker nothing is written. SERVER.md updated. Unit tests in tests/test_worker_pool.py::TestOomRanking: two dispatches flip 1000/0; a dispatch back onto the first slot (same pid) flips them again; one live worker writes nothing.

Tester check on lem: dispatch two jobs back to back to different cards, then read /proc/<pid>/oom_score_adj for both workers. The later-dispatched one should read 1000 and the other 0.

SIGTERM investigation (2026-10-13 02:26 journal): no earlyoom or systemd-oomd is installed or running, and nothing in the 02:14-02:26 window restarted the pool. At 02:26:11 the kernel OOM killer killed worker pid 4725 (anon-rss 62.6 GB, oom_score_adj 200) with SIGKILL. systemd then logged "dw-serve.service: A process of this unit has been killed by the OOM killer" and "Failed with result 'oom-kill'". The server logged the other job as "Worker process died: killed by SIGTERM". So the SIGTERM most likely came from systemd's default OOMPolicy=stop, which stops the whole unit once any process in it is OOM-killed. No userspace daemon was involved, so oom_score_adj does not apply to that step. Setting OOMPolicy=continue on the dw-serve unit would stop the surviving job being torn down. I did not change the unit; that is Don's call.

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (dfc7c49c4a..39d59692b4: dw/server/jobs.py dw/server/pool.py dw/worker_manager.py). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: pass (claude-opus-5-5 / anthropic)

Read git diff dfc7c49c..39d59692 under dw/ (commits 60b1982 and 39d5969; nothing else is in the range) against the docs/ARCHITECTURE.md map and item 3 of stage-c-guardrails.md. No findings:

Second owner: none. The OOM ranking stays in JobManager._rank_for_oom (dw/server/jobs.py), under the Worker pool dispatch row. It still writes through WorkerManager.set_oom_score_adj (dw/worker_manager.py) and doesn't re-derive the /proc write. WorkerSlot.ranked_pid (dw/server/pool.py) is removed, and nothing else in the tree references it.
Build vs. buy: nothing is hand-rolled. The change is a policy edit to existing code.
Map: no owner added or moved. The Worker pool dispatch row now names the renamed test TestOomRanking::test_each_dispatch_scores_its_worker_highest, which exists at tests/test_worker_pool.py:425.
Context: the rationale (bug: OOM ranking kills the earlier job - +100 adj is too small and ranking is per worker, not per dispatch #881, why lowering to 0 needs no privilege) is in the _rank_for_oom and set_oom_score_adj docstrings. No CLAUDE.md was touched. SERVER.md describes the new 1000/0 behavior.

The issue has no arch-approved label, and none was needed. Back to the tester for the check on lem.

triage: work — verify showed the reset to 0 is refused (baseline 200, a 1000 never comes back down). Likely cause, to confirm on lem: the kernel refuses an unprivileged write below the target's oom_score_adj_min, and a privileged write (e.g. systemd OOMScoreAdjust=200 on the service) sets that min, which workers inherit. So "reset" should go to the floor (the worker's spawn-time value / the server's own adj), not a literal 0, and a refused write should surface (log + job warning) instead of being silent. Check the unit file and /proc/<server pid>/oom_score_adj; the SIGTERM/journal question from the body is still owed. (claude-opus-5-5 / anthropic)

test: #881 - Linux /proc test asserts only the reset to the spawn value

fix(server): #881 - dw-serve.service sets OOMPolicy=continue so one w…

Fixed, third pass (claude-opus-5-5 / anthropic). The loop put this session on a stronger model than the two before it because the issue had bounced twice. The tester's verify comment and the triage after it were my spec.

Shipped on develop @ 23c7a12d (merge of fix/881-oom-floor: a2af00b6 floor reset + refusal warning, 16e73edb test, f4613912 OOMPolicy). Deployed to lem with deploy.sh develop, healthy 04:27:15 UTC.

Cause, confirmed on lem. Your inference was right, with one correction. The kernel refuses an unprivileged write below a process's oom_score_adj_min. Only a privileged write moves that minimum, and children inherit it. On lem, PID 1 starts the systemd user manager at adj 100, which becomes the minimum. The manager then starts dw-serve at 200, and the workers inherit 200. I tested under systemd-run --user on lem: writing 150 or 100 succeeds, writing 99 or 0 gets EACCES. So the reset to 0 was refused every time, and the refusal was only logged at debug. Every worker that was ever dispatched second stayed at 1000.

What changed

WorkerManager reads the worker's oom_score_adj right after spawn and keeps it as oom_score_floor (200 on lem). The new reset_oom_score_adj() writes that floor, not 0. The spawn value is always at or above the kernel minimum, so the write always takes. dw/worker_manager.py
_rank_for_oom still gives the dispatching worker 1000 and now resets every other live worker to its floor. With one live worker it still writes nothing. A write the kernel refuses is now a warning on the job (it shows in warnings and the event log), so a 1000/1000 tie can't happen silently again. dw/server/jobs.py
The get_health oom_score_adj description and SERVER.md now say "the spawn-time value" for the non-latest workers, not 0. The generated OpenAPI/UI files changed text only.

What sent the SIGTERM (owed from the body). It wasn't a userspace OOM daemon. earlyoom isn't installed, and systemd-oomd is not-found/inactive. lem's user journal for 2026-10-13 shows the event at 02:26:11, not 02:15-02:24:

02:26:11 systemd[1240]: dw-serve.service: A process of this unit has been killed by the OOM killer.
02:26:16 ERROR:dw:Job 7cba28c2095b: Worker process died: killed by SIGKILL (typically the out-of-memory killer)
02:26:16 ERROR:dw:Job 7cbac071da26: Worker process died: killed by SIGTERM
02:26:16 systemd[1240]: dw-serve.service: Failed with result 'oom-kill'.
02:26:21 systemd[1240]: dw-serve.service: Scheduled restart job, restart counter is at 1.

The kernel OOM-killed one worker. systemd's default OOMPolicy=stop then stopped the whole unit, and that SIGTERMed the other worker. So even with perfect ranking, any OOM kill took down every job on the server. scripts/dw-serve.service now sets OOMPolicy=continue, so an OOM-killed worker fails only its own job. I installed the updated unit on lem and ran daemon-reload before the deploy. systemctl --user show dw-serve -p OOMPolicy now reports continue. SERVER.md notes the requirement too.

Tests

tests/test_worker_pool.py::TestOomRanking: the three acceptance cases. I also added a case with a fake kernel minimum of 200: after back-to-back dispatches the scores are 1000/200, and a third dispatch flips them back to 1000/200. A refused write is a job warning, and a clean ranking warns nothing.
tests/test_worker_manager.py::TestOomScoreFloor: ensure_worker records the spawn-time value, and the reset writes the floor.
TestOomScoreAdjOnLinux uses the real /proc write against a real child process. It's skipped on macOS. I ran it on lem under systemd-run --user (child at 200, minimum 100): raise to 1000, reset, read back 200. It passed, along with all 90 tests in the two files.
The full suite has the same 17 failures as develop before this work (test_examples.py, kandinsky6 plugin/catalog tests). No new failures.

Tester check on lem, using the same steps as your last verify:

Dispatch A while it's the only live worker: its adj stays 200, because nothing is written.
Dispatch B on the other card while A is still alive: get_health should show B 1000, A 200.
Dispatch C back onto A's card (same pid): A 1000, B 200.

The baseline is 200, not 0, and that's expected: it's the floor systemd gives the unit. What matters is the 800-point gap, about 100 GB of badness on lem's 125 GB, which outweighs any RSS difference. No job's warnings should mention oom_score_adj.

Proposed regression case (regression-suite-complete.md, claude-opus-5-5 / anthropic): on a 2-GPU pool, run two jobs back to back (the second dispatched while the first runs), then run a third back onto the first card. After each, call get_health. Expected: the latest dispatch's worker has oom_score_adj 1000, and the other has the baseline value it reported while idle (200 on lem), never 1000 on both. No job warning mentions oom_score_adj.

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (cd0bdba204..23c7a12d3c: dw/server/api_models.py dw/server/jobs.py dw/worker_manager.py). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: pass (claude-opus-5-5 / anthropic)

I read git diff cd0bdba2..23c7a12d under dw/ and dw_mcp/. The #881 commits there are a2af00b and 16e73ed; f461391 only touches scripts/dw-serve.service, outside this scope, and 23bb31d (#885) isn't this issue's. I checked the change against the docs/ARCHITECTURE.md map and item 3 of stage-c-guardrails.md. No findings:

Second owner: none.
- The spawn-time read calls the existing dw/host_memory.py::read_oom_score_adj and doesn't re-read /proc itself.
- The kernel write is still WorkerManager.set_oom_score_adj's alone, and reset_oom_score_adj delegates to it.
- The ranking policy stays in JobManager._rank_for_oom (Worker pool dispatch row).
- The refused-write warning goes through job.add_event({"event": "warning"}), which job_record.py already folds into the job's warnings. The Run-time warnings row's dw/events.py::emit_warning doesn't apply, because it needs a worker-side RunContext, which the dispatcher thread doesn't have. The server already posts its own events to the job record the same way (job_status in jobs.py).
Build vs. buy: nothing is hand-rolled, and no pyproject.toml dependency covers oom_score_adj.
Map: no owner was added or moved, so no row is owed. The set_oom_score_adj return change (False → None when nothing was attempted) has only one caller, _rank_for_oom, which checks is False.
Context: the rationale for the floor over 0 (oom_score_adj_min, which is inherited from a privileged write, bug: OOM ranking kills the earlier job - +100 adj is too small and ranking is per worker, not per dispatch #881) is in the reset_oom_score_adj and _rank_for_oom docstrings, and the WorkerInfo.oom_score_adj description is updated to match. No CLAUDE.md was touched.

There's no arch-approved label on this issue, and none was needed. It goes back to the tester for the /proc/<pid>/oom_score_adj check on lem.

Verified (claude-opus-5-5 / anthropic)

Run against lem develop @ fb829e7, which includes the 23c7a12d merge. Every run used run_workflow(name="templates/image-processors", acknowledged_cost={fingerprint: sha256:c931bf87…, minutes: 1.1, downloads: []}) in default. I read oom_score_adj from get_health workers[] after each dispatch.

step	dispatch	cuda:0 (pid 35844)	cuda:1 (pid 35859)
0	idle	not alive, no pid/adj	not alive, no pid/adj
1–2	A `beaf93a12976` → cuda:0, then B `0fe8400832ba` → cuda:1 right after, both running	200	1000
3	A done; C `6f92d48200c1` → cuda:0 (same pid), cuda:1 alive and idle	1000	200
4	D `66469d86da07` → cuda:1 while C still running (second flip)	200	1000

The newest dispatch's worker gets 1000. Every other live worker goes back to its spawn-time floor (200 on lem), which is the amended spec from the triage and the hand-off, not the body's literal 0. That is the value the kernel will accept, as the hand-off explains.
The flip happens per dispatch, not per worker pid: pid 35844 kept its process and moved 200 → 1000 → 200. The 1000/1000 tie from the last bounce is gone in both directions.
None of the four jobs has warnings, so no write was refused.
I didn't observe the single-worker case separately this time. Step 1 dispatched A while no worker was alive, and B followed before I could read health. The value A held was the 200 floor in any case.

The SIGTERM finding (systemd OOMPolicy=stop, now continue) is box-side, and I can't observe it over MCP. I'm taking it on the implementer's word, and it isn't part of this verify.

All four run directories are deleted.

Navigation Menu

Repository navigation

bug: OOM ranking kills the earlier job - +100 adj is too small and ranking is per worker, not per dispatch #881

Cause

Fix

Also investigate

Acceptance

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug: OOM ranking kills the earlier job - +100 adj is too small and ranking is per worker, not per dispatch #881

Description

Cause

Fix

Also investigate

Acceptance

Activity

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

17 remaining items

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions