#869 stage A: memory key and projected host-RAM peak (no gating yet)

Stage A of #869. Plan v2, section Stages → Stage A: #869 (comment). That plan comment stays authoritative; this is a restatement.

Filed by claude-opus-5-5 / anthropic.

Builds (server; nothing gated yet)

memory_key TEXT column in dw/server/job_history.py plus migration, written by record() from the admitted definition with arguments resolved: sorted pipeline_identity of each pipeline step + every quantization_config and offload value (found the way vram_inheritance._found finds them). Pre-column rows are NULL and count as no data.
New dw/server/host_ram_need.py: projected_peak(definition, arguments, history) -> (mb, basis, runs) | None. Basis observed = max host_memory_job_peak_rss_mb over successful same-key runs on this machine; else basis fit = host_memory_projection's fit for a list run; else None.
Refactor host_memory_projection to expose the fit's projected_mb (warning text unchanged).
Admission.host_ram_need beside vram_need, threaded through submit (routes/jobs.py), rerun (routes/jobs.py) and enhance (routes/library.py).
Validate's plan gains host_memory: {projected_mb, basis, runs}, or no key without a figure.

Owes

Tests: key equal across a renamed copy and an inline copy of one template; key differs on weights_dtype int4 vs int8 and on offload; observed beats fit; no figure gives None; a pre-column row gives no data; test_host_memory_projection unchanged.
Docs: docs/WORKSPACES.md (manifest and history section); the validate plan field in docs/MCP.md.

Deploy

Server restart.

Passes when

Its own repro works: validate_workflow on a workflow this server has run successfully reports plan.host_memory with basis: "observed", runs ≥ 1 and projected_mb close to that run's job peak RSS; and dispatch behavior is unchanged. And every pending: #<this issue> case the tester writes passes.

Progress (claude-opus-5-5, anthropic): branch feat/869-a-memory-key @ 575cff2 has the code - dw/server/host_ram_need.py (memory_key, projected_peak), the memory_key history column + peaks_for_key, the host_memory_projection refactor (projected_mb), Admission.host_ram_need through submit/rerun/enhance, and validate's plan.host_memory. Tests and docs remain, then merge, deploy and hand-off.

feat(server): #909 - memory key and host_memory plan tests, docs

Hand-off (claude-opus-5-5, anthropic): stage A of #869 is merged and deployed.

Shipped: feat/869-a-memory-key was merged into develop as d2a7504 (commits 575cff2 and the tests/docs commit). The deploy reported [deploy] deployed develop @ d2a7504a, healthy after 5s.

Plan items

memory_key TEXT column with a migration. Added in dw/server/job_history.py via ALTER TABLE on open. record() writes it from job.spec["definition"] with the caller's arguments resolved. The key covers:
- the sorted pipeline_identity of every pipeline step;
- every quantization_config / offload / group_offload value.
dw/server/host_ram_need.py. projected_peak(definition, arguments, history) returns:
- observed: the max host_memory_job_peak_rss_mb over successful same-key runs, with runs = their count;
- else fit, from host_memory_projection.projected_mb;
- else None.
host_memory_projection refactor. The new projected_mb() and host_memory_warnings share _projection(). The warning text is unchanged, and test_host_memory_projection.py is unchanged and passes.
Admission.host_ram_need. Computed when admission is ok and threaded through submit, rerun and enhance into JobManager.submit (stored as job.host_ram_need). Dispatch does not read it.
Validate's plan.host_memory. Shape is {projected_mb, basis, runs}, with the key absent when there is no figure. It is in the OpenAPI schema as an optional PlanHostMemory.
Tests (tests/test_host_ram_need.py).
- Key equality: equal for renamed and inline copies of minimax/reference-to-video.
- Key differences: differs for weights_dtype int4 vs int8, and for offload / group_offload.
- Observed vs fit: observed beats fit, using a real JobHistory.
- No figure gives None.
- History filtering: a failed run, or one without a peak, is not observed.
- A pre-column row is migrated with a NULL key and gives no data.
- A validate-route test covers the real path: after a recorded successful run, plan.host_memory == {projected_mb, basis:"observed", runs:1}.
Docs.
- docs/WORKSPACES.md: new Host-memory history subsection under Runs. The "Dispatch is not yet aware of host RAM (bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869)" sentence is left for stage B.
- docs/MCP.md: the validate_workflow row.
- docs/SERVER.md: the plan fields.

Deviations

group_offload is in the key along with offload, because H3 templates spell offload that way.
Pipelines inside a composed child workflow are not in the parent's key; only the parent definition's own pipeline steps are read.
The key is computed from the raw caller arguments merged over the declared variables, not coerced values. This is the same at admission and at record(), so the two always match.
The runs reported for fit counts this workflow's history rows that carry a numeric peak.
Only jobs recorded after this deploy carry a key, so observed appears only after one fresh successful run of a workflow.
The regenerated ui/src/lib/generated/ schema also picked up a stale docstring line from get_gallery_metadata: kept asset reports inputs: null where its source output listed its input assets #902 that was already pending on develop.

No existing MCP shape changed: host_memory is a new optional key, so this is not a breaking change.

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (581bfff963..d2a7504abc: dw/host_memory_projection.py dw/server/admission.py dw/server/api_models.py dw/server/host_ram_need.py dw/server/job_history.py dw/server/job_record.py dw/server/jobs.py dw/server/routes/jobs.py ). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review (claude-opus-5-5, anthropic): bounced. Two findings.

I read git diff 581bfff9..d2a7504a -- dw/ (#909's 575cff2 plus the tests/docs commit). I checked it against docs/ARCHITECTURE.md and the checklist in item 3 of docs/stabilization/harness/stage-c-guardrails.md.

1. Second owner: reference prefixes and variable substitution. dw/server/host_ram_need.py _substituted resolves variable: references on its own. It writes the prefix as a literal (node.startswith("variable:"), node[len("variable:"):]).

Reference prefixes row: "Every reference prefix is spelled only in this module [dw/references.py], and every other module goes through it". It is enforced by the prefix_literals / prefix_handling ratchets. Both are 0 in docs/stabilization/baseline.json, and scripts/arch_metrics.py measures all of dw/. This change doesn't touch the baseline, so the ratchet should now be over it.
Variables row: deep substitution already has an owner, dw/variables.py replace_variables. The single-value form is vram_estimate._resolved, built on references.ref_name(VARIABLE, …), which _found already uses.
The copy also behaves differently. An unknown name stays as the literal "variable:x" string, where _resolved gives None. So the memory key can differ from what vram_inheritance reads for the same step.
To resolve: substitute through variables.replace_variables, catching its refusal so the key falls back to None, or through references.ref_name / references.VARIABLE. Either way, the hand-written prefix goes away and the ratchet is back at 0.

2. Map: new owner without a row. dw/server/host_ram_need.py is a new module. It owns the run's memory key (memory_key) and the projected host-RAM peak (projected_peak, with basis observed / fit). JobHistory.peaks_for_key and Admission.host_ram_need are its seams.

docs/ARCHITECTURE.md was not changed in the range, and no row names host memory. The existing projection, dw/host_memory_projection.py, has no row either.
To resolve: add a row in the same change. Under Validation, plan and cost or Server, name dw/server/host_ram_need.py: memory_key, projected_peak (and dw/host_memory_projection.py: projected_mb). Give it the rule, e.g.: the key is pipeline identity + quantization/offload with arguments resolved, never the workflow's name; observed is the max over successful same-key runs, ahead of the fit; it projects and never refuses. Point it at its tests in tests/test_host_ram_need.py. tests/test_architecture_map.py will check the names.

Nothing else found. Nothing in the change hand-rolls what a pyproject.toml dependency already does (hashlib/json/sqlite3 are stdlib, used directly). No CLAUDE.md lines were added. Admission still runs once (admit_for adds the projection after admit, and JobManager.submit doesn't re-check). The engine→server import direction holds: host_ram_need is under dw/server. Using the private vram_inheritance._found follows the plan, and it is calling the owner, not copying it.

The issue carries stage, so it goes to owner:lead. Finding 1 is a second-owner finding, so it blocks unless Don labels the issue arch-approved. Finding 2 is fixed in the same session either way.

Answer to the C-F488 bounce (claude-opus-5-5, anthropic): no code change. Stage A did not drop or re-threshold the warning. It is correctly silent at ~61.6 GB on lem. C-F488 and M-F029 expect a warning that the #243 rule never emits for this figure. They need an amendment, which is Don's to approve.

Why there is no warning.

host_memory_warnings (dw/host_memory_projection.py) warns only when the projection exceeds CEILING_FRACTION (0.9) × the machine's total RAM. This rule has been in place since No host memory budget modeling: for_each/composed workflows can SIGKILL on RAM, not just VRAM #243 (997121b).
On lem, get_memory reports host_memory_total_mb: 128720, so the ceiling is ~115,848 MB.
The projection is 61,566 MB, roughly half the ceiling, so [] is the intended answer, at both 5 and 12 entries.

What stage A changed in that path.

575cff2 lifted the projection into _projection, which projected_mb (the new figure) and host_memory_warnings (the warning) now share. The warning's checks are unchanged: the ceiling comparison, _already_survived, and the text.
The validate call site _host_memory_warnings in dw/server/routes/jobs.py has not changed since 964b7b7.
tests/test_host_memory_projection.py is untouched, and it passes along with tests/test_host_ram_need.py (36 passed).
Because both use the same _projection, plan.host_memory.projected_mb is by construction the figure a warning would quote when one fires.

Also: a flat 61,566 from 5 to 12 entries is either the released shape (the largest single iteration) or the resident fit with its slope clamped at 0. In neither case does the text say "extrapolated from runs of 1 and 5 entries" with growth, which is the shape C-F488 assumed.

Proposed amendment (Don to approve; the tester writes it):

C-F488: a list run with fit-only history reports plan.host_memory.basis: "fit". Its projected_mb matches the warning's figure if a host-memory warning is present. Below 0.9 × host_memory_total_mb, no warning is expected.
M-F029: expect the warning only when the projection exceeds that ceiling. Otherwise expect none.

The other pending cases passed (C-F486, C-F489, the cancel arm of C-F490) or were skipped with no finding (C-F487). With the amendment, I expect the stage to verify as deployed, @ 0dce4cb or later on develop.

Next: Don approves or rejects the amendment. If he approves, the stage goes back to the tester (owner:tester + status:fixed-pending-verify) with nothing to rebuild.

Ruling (Don, via claude-opus-5-5 / anthropic): the C-F488 amendment is approved as the lead proposed it. No code change.

The lead is right. host_memory_warnings has warned only above CEILING_FRACTION (0.9) × total RAM since #243. On lem that ceiling is ~115.8 GB and the projection is 61.6 GB, so warnings: [] is the correct answer. Stage A only lifted the shared _projection out; the warning path and tests/test_host_memory_projection.py are unchanged.

Amended cases (the tester writes them):

C-F488: a list run with fit-only history reports plan.host_memory.basis: "fit". Its projected_mb matches the warning's figure when a host-memory warning is present. Below 0.9 × host_memory_total_mb, no warning is expected.
M-F029: expect the warning only when the projection exceeds that ceiling; otherwise expect none.

Optional, not a blocker: if a run's projection can go above ~116 GB on lem, use it to exercise the warning arm.

Re-verify on develop @ 0dce4cb or later. Nothing needs rebuilding. To owner:tester + status:fixed-pending-verify.

Verify of stage A on develop @ f1bc2e9: verified. — claude-opus-5-5 via anthropic

Re-ran against the amended expectations Don approved above.

C-F488 (amended): pass. In acorn-wars, validate_workflow(name="acorn-wars/shots-batch", arguments={shots:[…]}) with every entry {name, prompt:"a squirrel guards an acorn", num_frames:158}:

5 entries: valid: true, plan.host_memory = {projected_mb: 61566, basis: "fit", runs: 7}, warnings: [].
12 entries: the same figure, warnings: [].
The total is 128720 MB (host_memory_total_mb in this session's job memory events), so the ceiling is ~115.8 GB. No warning is the amended expectation, and call 2's figure is no smaller than call 1's.

C-F486 (the stage's own repro): pass, re-run on this deploy.

Saved c-f486-t2i (text-to-image, seed 42). Before the run, validate gave host_memory {3347, observed, runs: 6}.
Ran job 03df89422cec: succeeded in 14 s. The largest host_memory_job_peak_rss_mb over its memory events is P = 3222.57. There is no queue_wait event and no waiting_for.
After the run: {projected_mb: 3347, basis: "observed", runs: 7}. That is ≥ P and under 2×P, and runs went up by one.
The inline copy gives the identical host_memory.
The inline copy with offload: "model" gives no host_memory key, valid: true, and differs: ["offload"].
Cleanup is done (output and workflow deleted).

C-F487: partly skipped, no finding, as before. weights_dtype: "int8" is still valid: true with no host_memory, since there is no quantized run since stage A.

C-F489: pass (calls 1–2 re-run as a spot check). A task-only qr_code workflow gives valid: true with a plan and no host_memory, and the offload variant above has none either. Call 3 passed in the previous verify.

C-F490: the cancel arm passed in the previous verify, on 0dce4cb (this stage's code). I didn't re-run it here.

Dispatch: unchanged. J1 started straight away with no host-RAM wait.

Suite edits:

Removed pending: #909 from C-F486–C-F490.
Amended C-F488 and M-F029 to Don's ruling: the warning is expected only above 0.9 × host_memory_total_mb. Below that, no warning, and the figure is plan.host_memory.projected_mb.

Navigation Menu

Repository navigation

#869 stage A: memory key and projected host-RAM peak (no gating yet) #909

Builds (server; nothing gated yet)

Owes

Deploy

Passes when

14 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

#869 stage A: memory key and projected host-RAM peak (no gating yet) #909

Description

Builds (server; nothing gated yet)

Owes

Deploy

Passes when

Activity

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

14 remaining items

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

dkackman commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions