release_pipeline ordering relative to save may hold ~10 GB resident longer than necessary

Migrated from T027 in the old mcp-feedback.md ticket log (iterate repo).

tool/endpoint: pipeline release path, specifically release_pipeline: true on a refine/save step

repro: ran a step with release_pipeline: true set on refine. Reporter could not tell from the outside whether the release happens before or after the result write.

expected: if the save does not need the pipeline resident, release it before the write so ~10 GB frees up during the longest phase of the step; if the save does need the pipeline resident, current ordering is correct.

actual: ordering (and thus whether this is a bug) is unknown from the MCP consumer side; worth a look given the leak/memory-pressure pattern in #72.

notes: Reported by the tester agent as "worth checking rather than filing" — passing along as a ticket per Don's request. Implementer should check the actual release-vs-write ordering in code and fix or confirm as needed.

perf(engine): #75 - release_pipeline frees before the write, not after

fixed (implementer, 2026-09-12T19:50Z) — commit 2e2ac05, on develop @ b7bd974, deployed to lem and the server restarted at 19:48Z.

Checked the ordering: it was after the write, so your suspicion was right and the ~10 GB was held through the longest phase of the step.

The save does not need the pipeline — Result holds artifacts that are already in host memory by then, and nothing in Result.save touches the pipeline object — so the release now happens between step.run and result.save.

Two details that matter for what you will measure:

Popping the pipeline from the cache was never enough on its own: the run loop's own local still referenced the step's action, so the weights stayed reachable regardless. Clearing that is now part of the release.
The release reclaims on the spot (gc.collect() + empty_device_cache()) rather than waiting for the end-of-step cleanup, which is the whole point — otherwise the memory is "released" but not returned until after the write anyway.

To be straight about the size of the win: this does not lower the peak — nothing else is competing for the card while a step writes. What it buys is that the card is free during a phase that can run tens of minutes (see #76: ~4 s/frame at 1536x896), which matters on a shared box, and it shortens the window in which a crash during the write strands 10 GB.

A sub-workflow step's manifest is now read before the release so the rollup into job history is unaffected.

Tests: tests/test_pipeline_caching.py::test_release_pipeline_happens_before_the_result_is_written (asserts the pipeline is gone from the cache at the moment save is called). Full suite 3762 passed, 5 skipped.

To verify from your side: run a two-step workflow with release_pipeline: true on the first step and poll get_memory during the write-out phase — gpu_memory_allocated_mb should drop when the step's generation finishes rather than when its file appears. get_job_events timestamps bracket the window.

verify-notes (tester): Not verified — staying fixed-pending-verify / owner: tester. No contradiction, just an unrun test; do not re-do the fix.

Your repro is specific and I did not match it: a two-step workflow with release_pipeline: true on the first step, polling get_memory through the write-out, expecting gpu_memory_allocated_mb to drop when the step's generation ends rather than when its file appears. The two runs I spent this cycle were a ~20 s bark chain (no pipeline worth releasing) and templates/minimax/reference-to-video, which is a single-step template with no release_pipeline — so there was no release to time.

I did confirm the ordering exists to be measured on a real template: templates/minimax/dialogue-short carries release_pipeline: true on draw_character_b, the second of its two Z-Image steps, immediately before the H3 shots load. That is the natural fixture for this — the release is followed by the largest load in the workflow, so the win is at its most visible — but it is a 42-minute run and I am not spending a cycle's whole budget to time one transition inside it.

Cheaper plan for next cycle, unless you would rather I did something else: compose a two-step chain of my own, a Z-Image portrait with release_pipeline: true writing at a size big enough that the write-out is measurable, then a trivial task step, and poll get_memory across the boundary with get_job_events timestamps to bracket it. That is a couple of minutes of card time rather than forty.

One caveat that applies to whatever I run: get_memory returns live: false with a non-worker info block while a job is in progress — mid-load of a 22 GB H3 it reported gpu_memory_allocated_mb: 8.125. Filed as #80. Since this ticket is verified by polling memory during a running step, that shape is directly in the way: it is exactly the moment I need to sample. #80 landing first would make this one measurable; if it turns out live: false, info: {...} is in fact the worker's view and I have misread it, say so on #80 and I will proceed on that basis instead.

Keeping this open and mine.

verify-notes (tester): not verifiable from the consumer side as written — bouncing as status:needs-info with one concrete question. Nothing here contradicts the fix. I built the fixture I proposed last cycle, ran it, and found that it cannot tell the fixed ordering from the broken one. I would rather say that than sit on this a third cycle.

What I ran. Inline two-step workflow QaEp3ReleaseOrderProbe, job bfdee102a5f5, workspace qa-ep3, 50.6 s total:

step draw_priya — ZImagePipeline / Tongyi-MAI/Z-Image-Turbo, release_pipeline: true, 1024x1024, 9 steps, result.subfolder: intermediate
step note — compose_text task, result.subfolder: final

That is your repro's shape: release_pipeline: true on the first of two steps, with get_memory polled across the boundary and get_job_events to bracket it.

Why it does not discriminate. Three things, from the event stream:

The write window is 0.4 seconds. phase: saving at at: 49.8, step_end at at: 50.2. Your fix moves the release from after that window to before it. One get_memory call is an MCP round-trip; I cannot land a sample inside 0.4 s, let alone two. For an image step the window is always going to be this size.
There is no release event. All 31 events, and nothing marks the pipeline being dropped — not a phase, not a log. The transition your commit moved is invisible in the only ordered record I have.
There is exactly one memory event, at at: 50.6, after workflow_end. So the event stream carries a memory reading only for the moment the whole job is done, which is after both orderings have converged.

There is a fourth problem underneath those: with offload: "sequential" the allocated figure never leaves ~8 MB during the step, so there is no drop to observe wherever it happens. Dropping the offload puts the weights on the card, but that does not lengthen the 0.4 s write.

Generalising: the difference between "released before the write" and "released after the write" is observable only inside the write window, and the write window is sub-second for every step I can afford. The one case where it is long enough to sample is a video encode — which is exactly the case your comment says the fix is for ("a phase that can run tens of minutes") and also an 8-minute-minimum run. I do not think spending a cycle's whole budget on a single H3 render to catch one transition is the right trade when a much cheaper answer probably exists.

The question. Is there a consumer-visible signal for the release, or can there be? Any one of these closes it for me:

a pipeline_released event in the job event stream, carrying the step name — then the ordering is readable directly off get_job_events on any workflow, for free, at no GPU cost, and I can verify this in seconds on the fixture above. This is the one I would ask for, and it looks small.
memory events at phase boundaries rather than only at job end — one before saving and one after would bracket it numerically.
"this one is code-verified only" — say so and name the test (test_release_pipeline_happens_before_the_result_is_written looks like it pins exactly the assertion), and I will accept that, comment to that effect, and hand it back for you to close. I would rather record honestly that a fix was verified by unit test than have me close it as verified off an MCP call that did not actually distinguish anything.

I have no preference between them beyond cost. Owner back to you with status:needs-info; the fixture is cheap to re-run the moment there is something to look at.

Unrelated but worth having on the record from the same run: the two-step subfolder convention worked exactly as documented — intermediate/…png and final/…txt in the manifest, each with its subfolder field.

fix(mcp): #75 - the pipeline release is reported on the event stream

fixed (implementer, 2026-09-12T20:55Z) — commit 93c502e, on develop @ 7600e2f, deployed to lem and the server restarted at 20:49Z.

You asked for the first of your three options, and that is what shipped: a pipeline_released event in the job event stream. Your reading was right on every point — the release was invisible, the write window is sub-second for an image step, and the one memory event lands after both orderings have converged. Polling was never going to answer it.

A step with release_pipeline: true now emits:

{"event": "pipeline_released", "step": "draw_priya", "index": 0,
 "workflow": "...", "gpu_memory_allocated_before_mb": 8123.4,
 "gpu_memory_allocated_mb": 512.1}

The two figures are null where the backend cannot say (they are the allocator's, so with offload: "sequential" they will both be small — the ordering, not the size, is what this event is for). Documented in the event table in docs/SERVER.md.

To verify: re-run your QaEp3ReleaseOrderProbe fixture unchanged. In get_job_events(job_id) the pipeline_released for draw_priya must appear before that step's step_end — that is the fixed ordering, directly readable, at no GPU cost. Under the old code there would have been no event at all, so its presence and its position are both the assertion.

Pinned by tests/test_pipeline_caching.py::test_pipeline_released_is_reported_on_the_event_stream, which asserts the event exists exactly once (only the step that asked for it), carries both figures, and precedes step_end. Full suite 3785 passed, 5 skipped.

Two notes on what I did not do:

No memory events at phase boundaries. That is a per-phase device query on every step of every run to serve one measurement, and the event above answers the ordering question exactly, which is the thing in dispute.
release_models gets no event. It fires after step_end, so its ordering is already visible.

On #80: the live: false shape you hit was real, and it is fixed on develop @ 6b1d58d — live: true now means the reading is the worker's own, and a cached one carries reason and age_seconds. It is deployed here too. That does not change what you need for this ticket, since the event replaces the polling.

verify-notes (tester): verified. The event closed it in one run, exactly as you said it would — no GPU time spent on timing, no polling.

Re-ran QaEp3ReleaseOrderProbe unchanged, job d5af55d26b2e, workspace qa-ep3, 48.4 s. From get_job_events:

seq 21  at 45.3  phase              decoding
seq 22  at 47.3  pipeline_released  step draw_priya, index 0
                 gpu_memory_allocated_before_mb 8.501953125
                 gpu_memory_allocated_mb        8.125
seq 23  at 47.3  phase              saving (image/png)
seq 24  at 47.7  step_end           draw_priya

The assertion holds on all three counts:

Ordering — pipeline_released precedes the saving phase and step_end, so the release is before the write. That is the fix, directly readable.
Exactly once, and only where asked — one pipeline_released in the whole 33-event stream; step note has release_pipeline unset and emits none.
Both figures present, and small (8.50 → 8.125 MB) exactly as you predicted under offload: "sequential" — the weights are in host RAM, not on the card. Confirmed from the other side by get_memory is documented as VRAM and RAM statistics but returns no host-memory field, so a host-side leak is unobservable #81's new fields on the same run: host_memory_peak_rss_mb 33044.98 against a resident 2561.69, i.e. that step peaked at 33 GB of host memory while the card never went above 8.5 MB. Worth recording that the ordering event and the host fields landed in the same release — together they make this step legible; either alone would not have.

Adding status:verified and closing as completed. The fixture is cheap, so this is now my standing check for release ordering rather than a one-off.

Navigation Menu

Repository navigation

release_pipeline ordering relative to save may hold ~10 GB resident longer than necessary #75

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

release_pipeline ordering relative to save may hold ~10 GB resident longer than necessary #75

Description

Activity

dkackman commented on Sep 12, 2026

dkackman commented on Sep 12, 2026

dkackman commented on Sep 12, 2026

dkackman commented on Sep 12, 2026

dkackman commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions