Repository navigation
No host memory budget modeling: for_each/composed workflows can SIGKILL on RAM, not just VRAM #243
Description
Three SIGKILLs during a single testing session, three different mechanisms, all host-RAM exhaustion rather than a graceful engine error:
- A twelve-entry H3
for_eachstep (tool-musical-trailer'sshotstep, 12 shots up to 345 frames each) where the pipeline stayed resident across iterations at a footprint that, summed with the list length, exceeded host RAM. - A frame-extraction step whose cost scales with clip length, run against a long cut.
- (third case from the same session — details in the original report if more repro detail is needed.)
Nothing today models host memory as a budget the way validate_workflow already models GPU-minutes. The pieces needed already exist separately: the per-run recipe documents a workflow's own footprint (one incident recipe documented 63 GiB on a 64 GiB box), and list_workflows/get_workflow already reports list length (lists.<name>.entries). "12 members x H3's per-run footprint exceeds host RAM, batch to 6" is derivable from data the server already has.
Suggested fix: validate_workflow projects peak host memory for the last generative step of a composed/list-driven workflow (per-entry footprint x list length, or the largest single resident model for a chained/sequential-release workflow) and warns or refuses when it exceeds a configured host RAM ceiling, the same way it already refuses an out-of-bound num_frames before the run starts rather than after weights are loaded.
Note on scope of the same failure shape: templates/minimax/music-video looked structurally similar at a glance (composes a shots list + has-audio + generates video), but on inspection it is not the same risk: it defaults to 4 shots of 124 frames each (not 12 of up to 345), explicitly releases the singer-portrait and song-generation pipelines before the H3 shot loop starts, and has a completed measured run (35 min curated / 25.75 min observed). It's the same workflow family, not a confirmed instance of this bug — worth a regression case at larger shots/audio_duration to check where it starts to scale badly, but not a duplicate report.
This is new validate_workflow surface (a memory-projection rule), not a narrow repro fix — flagging for triage to judge whether it needs a written proposal before implementation.
Activity
dkackman commentedon Sep 19, 2026
triage: escalate — this adds a new
validate_workflowrule (a host-memory projection: per-entry footprint × list length against a RAM ceiling) plus the data to feed it, which nothing in the catalog declares today. #98 closed the cross-job leak (release on workflow switch) but never modeled a budget. Needs sign-off on scope before a fix session: where the per-workflow footprint comes from (curated field besidecost, or observedhost_memory_peak_rss_mbfrom jobs.sqlite), whether it warns or refuses, and what a caller can override. (triage by implementer agent, model opus via provider anthropic)dkackman commentedon Sep 19, 2026
Proposal for scope sign-off (Don):
Data source: observed, not curated.
dw/host_memory.pyalready computespeak_rss_mb(process high-water mark) per run, but today it's surfaced only in the REPL (dw/repl.py:284) — never persisted, never read byplan.py's cost model. Rather than adding a hand-authored footprint field to the catalog (which goes stale as templates/models change, with no way to notice), persisthost_memory_peak_rss_mbper completed job intojobs.sqlite, keyed by(workflow_id, list length). This mirrors the patternregression-perf/already uses for timing baselines: compare a new request's projected list length against the median of recent observed runs at that or a nearby list length, rather than requiring someone to type in a number that immediately starts drifting.Cold-start is expected and fine: a workflow with no observed runs yet at a given list length simply can't be checked — same as a
regression-perfmetric with no baseline yet. The rule only engages once real data exists.Warn, not refuse, by default. GPU-minute cost is a budget the caller explicitly agrees to spend before the run starts — refusing outright makes sense there. Host RAM headroom is a property of whatever machine the server happens to be running on, not something the caller chose; the same workflow that SIGKILLs a 64GB box may be fine on a bigger one.
validate_workflowshould warn with the projected number (observed footprint × requested list length, scaled from the nearest observed list length) and let the caller proceed, the same way an existingnum_frameswarning works — reserve outright refusal for a case with enough history to say the run reliably fails at this size on this machine.The projection needs a shape distinction, not just a multiply. The issue's own "music-video isn't actually the same risk" note is the proof:
templates/minimax/music-videolooks structurally identical (composes a shots list + has-audio + generates video) but explicitly releases the singer-portrait/song pipelines before its shot loop, so its peak isn'tper-entry × Nthe waytool-musical-trailer's resident-across-iterations shape is. The projection rule needs to distinguish "pipelines released between iterations" (peak ≈ largest single resident step) from "pipeline stays resident across the list" (peak ≈ per-entry × N) — probably derivable from whether the workflow's steps between iterations include an explicit release, rather than needing a new declared flag.Scope for v1: persist
host_memory_peak_rss_mbper job; project only for the two shapes above (resident-across-list vs. releases-between-iterations); warn only, no refuse; no cross-machine normalization (a projection is only meaningful against the machine that produced the observed data, which for now is fine since there's one server). Cross-host portability of the projection, and refuse-mode, are follow-ups once there's enough observed data to trust the numbers.Releasing to implementer to build if this scope looks right; flag back here if any of the above needs to be split further.
fix(server): #243 - project host memory for list-driven runs, warn only
dkackman commentedon Sep 20, 2026
Implemented the v1 scope approved on this issue (observed not curated, warn not refuse, resident-vs-released shape via
release_pipeline/release_models, cold start = no check).Changes (commit
997121b, merged--no-ffintodevelopat0a09ad1):dw/server/jobs.py:Job/JobManagernow capture the worker'shost_memory_peak_rss_mbper run (via the existingmemory_infoIPC message) and persist it into a new nullablejobs.sqlitecolumn (idempotentALTER TABLE, same pattern as the other job columns).finished_runs()now returns it per row.record()reads it viagetattr(job, "host_memory_peak_rss_mb", None)so older/test-double job objects that predate the column stay compatible.dw/host_memory_projection.py: pure functionsreleases_between_iterations(definition)(true when everyfor_eachstep carriesrelease_pipeline/release_models, or there's nofor_eachstep at all) andhost_memory_warnings(definition, list_entries, rows, ceiling_mb). Projects the largest single observed peak for the released shape, or median-peak-per-entry × requested-count for the resident shape; returns [] on cold start (no rows / no memory readings) or when nothing exceeds the ceiling.dw/server/app.py:POST /api/validate(and thus MCPvalidate_workflow) now calls this after the existing gate-warnings, usingapp.state.observed_costs.rows_for(name)for history andhost_memory_stats()["total_mb"] * 0.9for the ceiling. Best-effort (wrapped, logs+swallows on failure) so a projection bug can't 500 the free pre-flight. Only runs for a catalog workflow (has a name), matching howobserved_costsis keyed.Tests: new
tests/test_host_memory_projection.py(10 cases covering both shapes, threshold crossing, cold start, no-memory-reading history, no-list-in-request). Full suite: 5183 passed, 16 skipped, 0 failed.Deploy to
lem: pulleddevelop(c2770dd..0a09ad1, fast-forward) into/home/don/diffusers-workflow. The runningdw.serveprocess had no systemd/tmux/screen supervision — it was a plainnohup ... & disownbackground job (confirmed via/proc/<pid>parent chain and shell history) — so Ikill'd it (needed-9; it was holding an open MCP streamable-HTTP connection past the graceful-shutdown grace period) and relaunched with the same flags (--host 0.0.0.0 --mcp --workspace /home/don/diffusers-workspace --examples-dir /home/don/diffusers-workflow/workflows) from the updated checkout. New PID confirmed up,GET /api/health(with the bearer token) returns{"status":"ok","version":"0.4.0-beta.6","device":"cuda","mcp":true}. Timestamp: 2026-09-19 ~00:29 lem-local.Worked as model Sonnet 5, provider Anthropic.
Suggested regression case (for the tester to add once verified, in
regression-suite-complete.md— this needs prior job history to trigger, so it's edge-case-y rather than smoke): run afor_each-shaped catalog workflow (resident, i.e. norelease_pipeline/release_modelson its for_each step) enough times viarun_workflowto build history with a small list, then callvalidate_workflowwith a much larger list for the same workflow's list variable and confirm a warning mentioning "Projected host memory" appears when the projected figure exceeds ~90% of the server's RAM; also confirmvalidate_workflowon a workflow with no prior run history produces no such warning (cold start).dkackman commentedon Sep 20, 2026
Verified over MCP (tester agent, model opus via provider anthropic), server
0.4.0-beta.6onlem, in a throwaway workspaceqa-verify-243.Setup: saved a catalog workflow
qa-243-qr-list— onefor_each: variable:codestask step (qr_code, no pipeline, norelease_pipeline/release_models) so it's cheap and resident-shaped by the v1 rule.What I ran
validate_workflow(name="qa-243-qr-list", arguments={codes: [1 entry]})before any run →valid: true,warnings: []— cold start, no projection, as specified.run_workflowwith that 1-entry list → job59156e208b6asucceeded in <1 s (builds one history row with a memory reading).validate_workflowwith 32 entries (thefor_eachcap) →valid: trueplus:Warn-not-refuse ✔, figure is observed-per-entry × N (~1953 MB × 32) ✔, ceiling is ~90% of host RAM ✔.
Adjacent: same workflow, 16 entries (~31 GB projected, under ceiling) →
warnings: []✔.Adjacent: patched the for_each step with
release_pipeline: true(released shape), same 32-entry list →warnings: []— largest-single-peak projection, well under ceiling ✔.Also noted:
for_eachrefuses more than 32 entries outright (for_each has 200 entries; the limit is 32), so the resident projection can only ever engage on a footprint ≥ ceiling/32 per entry — fine for H3-class steps, which is what the report was about.One observation, not a defect against the approved scope: for a pipeline-free task
for_eachlike this one, the "per-entry" figure is the whole worker's peak RSS (~1.9 GB of process baseline), so the projection overstates for lightweight task loops. Harmless at warn-only; worth remembering if refuse-mode is ever added.Matches
expected:— closing as verified. Regression case added toregression-suite-complete.md(C-level, since it needs a run of history first).