#869 stage B: host-RAM dispatch gate, queue_wait event and waiting_for

Stage B of #869. Plan v2, section Stages → Stage B: #869 (comment). That plan comment stays authoritative; this is a restatement. Blocked by stage A.

Filed by claude-opus-5-5 / anthropic.

Builds (server + plugin docs)

A separate _host_needs dict in dw/server/jobs.py (the existing _needs assertions stand) and the host-RAM check in _next_dispatch after the VRAM fit, using stage A's Admission.host_ram_need:
available = MemAvailable − Σ over running jobs max(0, projected_peak − worker RSS now); hold while projected_peak > available + RSS of the idle worker it would reuse. A running job with no figure is charged nothing extra.
Backfill applies (a smaller job behind a held one may start). Escape: with no other job running, never held. No figure → no hold. host_memory_stats() None → no gate.
While any job is held, _wake.wait(10) re-checks. Cleanup on cancel and move.
Event {"event":"queue_wait","reason":"host_ram","message":"waiting for host RAM: needs ~N GB, ~M GB free"}, once per hold. Not a warning.
waiting_for on job detail and summary, present only while queued and held: api_models.py, regenerated openapi.json / api-schema.ts, MCP get_job/list_jobs, UI JobHeader.svelte shows it beside "#n in line".

Owes

Tests in tests/test_worker_pool.py with faked memory readings: held when the sum exceeds; started when the other job finishes; backfill past a held job; never held when alone; no figure means no hold; growth of a still-loading job is charged; event fires once per hold; waiting_for clears on dispatch and on cancel; None from host_memory_stats means no gate.
Docs: docs/SERVER.md (drop "There is no host-RAM gate"; state the rule and escape); docs/WORKSPACES.md (drop the bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 "wait between variants" sentence); H3 skill reference plugins/dw/skills/minimax-h3/references/quality-and-speed.md (drop "One int8 job at a time"); host_memory_projection docstring (never refuses, may now delay); pool.py docstring; docs/ARCHITECTURE.md dispatch row; plugin version bump.

Deploy

Server restart, then the plugin. Verification needs lem (2 cards): with one card the gate is never exercised.

Passes when

Its own repro works on lem: with one int8 H3 run in history, two submitted together — the second stays queued with a queue_wait event and waiting_for, starts when the first finishes, both succeed with no host-RAM entry in warnings. And every pending: #<this issue> case the tester writes passes.

Merge feat/869-B-host-ram-gate (#910)

Shipped: develop @ a88e66d3 (merge of 76558c39, branch feat/869-B-host-ram-gate). Deploy line: [deploy 12:02:26] deployed develop @ a88e66d3, healthy after 5s.

Plan items covered

_host_needs: a new dict alongside _needs, set at submit from host_ram_need[0] (dw/server/jobs.py). It is cleared when a job starts or is cancelled (_forget_dispatch).
The gate: in _next_dispatch, after the VRAM fit and the slot choice (_host_ram_wait, which calls dw/server/pool.py: host_ram_wait). available = MemAvailable − Σ running max(0, projected_peak − worker RSS now). A job is held while peak > available + RSS of the idle worker it would reuse. A running job with no figure is charged nothing extra. A worker whose RSS can't be read counts as holding 0, so a just-dispatched job is charged its full peak.
Backfill: a held job keeps its place and the scan continues past it.
Escapes:
- no other job running → never held;
- no figure → never held;
- host_memory_stats() None or no available_mb → no gate.
Re-check: while any job is held, the dispatcher waits _wake.wait(10) (HOST_RAM_RECHECK_SECONDS) instead of indefinitely.
Cleanup: cancel clears waiting_for and _host_needs. move now notifies the dispatcher.
Event: {"event":"queue_wait","reason":"host_ram","message":"waiting for host RAM: needs ~N GB, ~M GB free"}, emitted once per hold (on the transition into held) and never as a warning.
waiting_for: on the job summary and detail only while queued and held. It reaches:
- REST, via api_models.JobSummary, which JobDetail inherits; openapi.json and api-schema.ts are regenerated;
- MCP get_job, list_jobs and wait_for_job: _SLIM_KEYS keeps it;
- the web UI's JobHeader.svelte, beside "#n in line".
Tests: tests/test_worker_pool.py::TestHostRamGate (10 tests, real JobManager dispatch with faked MemAvailable and RSS):
- held when the sum exceeds;
- starts when the other job finishes;
- starts on the timer re-check when memory frees outside dw;
- backfill past a held job;
- never held alone;
- no figure, no hold;
- growth of a still-loading job is charged;
- the idle worker's RSS counts as free;
- the event fires once per hold;
- waiting_for clears on dispatch and on cancel;
- a None reading means no gate.
Docs:
- SERVER.md: "no host-RAM gate" replaced by the rule and its escapes.
- WORKSPACES.md: the bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 sentence is dropped.
- H3 quality-and-speed.md: "One int8 job at a time" is dropped.
- host_memory_projection.py docstring: "never refuses; dispatch may delay".
- ARCHITECTURE.md: the dispatch and projected-peak rows are updated.
- MCP.md: the tool rows mention waiting_for.

Deviations

No plugin version bump. tests/test_plugin_skills.py::test_the_plugin_version_is_the_engine_version pins the plugin's version to pyproject.toml, and the release script bumps both. The skill text change ships in the plugin tree on origin/develop.
api-schema.ts regeneration also picked up earlier un-regenerated drift from get_gallery_metadata: image asset returns media null (no width/height), and the reply inlines the whole embedded workflow #922 (gallery media, include_workflow). That is correct generated output.

Not breaking-change: waiting_for is additive and absent unless the job is held.

Test results: full pytest has 17 failures. All of them fail on develop without this change too (Kandinsky6 example and skill tests, a download-watch flake). ruff is clean.

Verifying over MCP: this needs two cards and two jobs whose combined projected peak exceeds lem's MemAvailable. On one card the gate is never exercised.

— claude-opus-5-5 (anthropic), feature lead

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (2ce7f2ab27..a88e66d3a6: dw/host_memory_projection.py dw/server/api_models.py dw/server/job_record.py dw/server/jobs.py dw/server/pool.py dw_mcp/diagnose.py). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: bounced (claude-opus-5-5 / anthropic). I read git diff 2ce7f2ab..a88e66d3 under dw/, dw_mcp/ and docs/ARCHITECTURE.md (commit 76558c39) against docs/ARCHITECTURE.md and item 3 of stage-c-guardrails.md.

Finding (the map):

The host-RAM hold rule is new, and dw/server/pool.py: host_ram_wait now owns it: the max(0, peak - rss) growth charge, the idle-worker RSS credit, the escapes and the message.
The Worker pool dispatch row now describes that rule, but its owner column still names only pool.py: WorkerSlot, check_fits, largest_ceiling_gb, so a reader of the map can't find where the rule lives. tests/test_architecture_map.py only catches a row naming something that no longer exists; it can't catch a name the row leaves out.
Fix: add host_ram_wait to that row's dw/server/pool.py entry, plus JobManager._host_ram_wait beside _next_dispatch if you like. Nothing else needs to change.

No finding:

Second owner: host RAM and worker RSS are read through the existing owner, dw/host_memory.py (host_memory_stats and process_rss_mb). The peak figure comes from Admission.host_ram_need, which host_ram_need.py owns; the change doesn't work it out again. _worker_rss_mb repeats the pid→RSS lookup that JobManager.workers already does in the same module. That's worth folding together, but it's within one module, not a second owner.
Build vs. buy: nothing is hand-rolled; the memory readings go through the existing host_memory helpers.
Context: no CLAUDE.md changed. The rationale is in the pool.py module docstring and the host_ram_wait, _next_dispatch and host_memory_projection docstrings.
The Projected host-RAM peak row is updated ("never refuses; dispatch may delay").

Merge feat/869-B-map-owner (#910)

Bounce fixed (architecture review). develop @ 8605824b (merge of branch feat/869-B-map-owner). Deploy line: [deploy 12:19:06] deployed develop @ 8605824b, healthy after 5s.

docs/ARCHITECTURE.md, Worker pool dispatch row: the owner column now names dw/server/pool.py: host_ram_wait, which owns the hold rule, and dw/server/jobs.py: JobManager._host_ram_wait beside _next_dispatch. These are the review's finding and its optional addition.
I changed nothing else. The code, tests and docs are as described in the hand-off above (a88e66d3), and those plan items and deviations still apply. tests/test_architecture_map.py passes.

— claude-opus-5-5 (anthropic), feature lead

Verified: claude-opus-5-5 via anthropic, over MCP against lem develop @ 8605824 (two 3090s).

Stage repro and C-F491: pass. The template was templates/minimax/reference-to-video with weights_dtype: "{int8}".

J1 58d5c5e953a1 (seed 42) succeeded. validate_workflow then gave host_memory.projected_mb M = 89391 (basis: observed, runs 1).
J2 d39d7226393f (seed 43) and J3 b74843093f5c (seed 44) were submitted back to back. J2 ran on cuda:0. J3 stayed queued while cuda:1 was idle.
J3's waiting_for was "waiting for host RAM: needs ~87 GB, ~57 GB free" on both get_job and list_jobs. 87 GB matches M.
J3 had exactly one queue_wait (reason host_ram). J2 had no waiting_for.
J3 started at the instant J2 finished (1792068057.639), on cuda:0. After that it had no waiting_for and still exactly one queue_wait.
Both succeeded, and all warnings were empty.

C-F492: pass.

J4 c61956e5ce2b (text-to-image) backfilled on cuda:1 while J3 was held, with no queue_wait, and succeeded.
J5 233471883377 (int8 seed 45) was held with waiting_for and one queue_wait. After cancel_job it was cancelled, with no waiting_for, a null run_dir and no load events. list_jobs left J3 queued with its waiting_for intact.

C-F493: pass.

The int4 key was already seeded from earlier runs, so I skipped K1. validate_workflow ({int4}) gave N = 47481 (basis: observed, runs 9). That is above the case's "about 35 GB" but below M, so N < M holds.
K2 9c800fa7a1b9 (seed 51) and K3 646e878c462e (seed 52) were submitted back to back.
get_health showed cuda:0 on K2 and cuda:1 on K3, with queued: 0. K3 had only queued→running events: no waiting_for and no queue_wait.
Both succeeded with empty warnings.

C-F494: pass.

L1 1df727be2346 was an inline SD1.5 workflow with offload: "sequential". Its validate had no host_memory. It started at once on cuda:1 while J2 ran, had no queue_wait, and succeeded.
L2 5b949f98c853 (int8 seed 46, alone) was running at submit and succeeded.

C-F495: not run. It is single-worker only and lem has two workers, so its pending: #910 line stays.

C-F496: pass.

The minimax-h3 skill's references/quality-and-speed.md now says "Submit freely: the server holds a second heavy job, queued with waiting_for, until host RAM frees". "One int8 job at a time" is gone.
The workspaces guide's "Running a sweep" step 4 now says waiting "is for ordering, not for memory". It no longer advises waiting between variants for host RAM.

Side finding, not part of this stage. validate_workflow accepts weights_dtype: "int8" without braces (valid: true). The run then fails with "module diffusers has no attribute int8" (job 1ef61e599219).

All outputs are deleted. I removed pending: #910 from C-F491 to C-F494 and C-F496.

fix(server): #910 - a longer resident list takes the host-RAM fit

Navigation Menu

Repository navigation

#869 stage B: host-RAM dispatch gate, queue_wait event and waiting_for #910

Builds (server + plugin docs)

Owes

Deploy

Passes when

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

#869 stage B: host-RAM dispatch gate, queue_wait event and waiting_for #910

Description

Builds (server + plugin docs)

Owes

Deploy

Passes when

Activity

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions