Repository navigation
get_memory is documented as VRAM and RAM statistics but returns no host-memory field, so a host-side leak is unobservable #81
Description
tool: get_memory
workspace: qa-ep3 (not workspace-specific)
repro:
Call get_memory on an idle server with the worker up. Every field of info:
{
"run_count": 1,
"gpu_available": true,
"gpu_memory_allocated_mb": 536.29,
"gpu_memory_reserved_mb": 690.0,
"gpu_memory_free_mb": 23075.06,
"gpu_device_name": "NVIDIA GeForce RTX 3090",
"gpu_memory_total_mb": 24122.19
}expected: a host-memory figure alongside the gpu_* ones — the tool description opens "Get the worker's VRAM and RAM statistics" — or, if host RAM is deliberately out of scope, a description that does not promise it.
actual: seven keys, all of them VRAM or device identity apart from run_count. There is no host-memory reading of any kind, and no other tool exposes one: get_health and get_server_info carry none either. From the MCP surface, host memory on lem is unobservable.
why this is more than a wording nit. Two things I hit this cycle depend on it:
-
Failed pipeline loads leak GPU/host memory instead of being released, and leaks accumulate across retries #72 is titled "Failed pipeline loads leak GPU/host memory". I ran its repro three times this cycle (jobs
b0a3237495b3,d9b43e353659,09395157badb: a warmTongyi-MAI/Z-Image-Turbobase with a nonexistent LoRA weight file, so the load fails partway through, after the base is built). The GPU figures did not move by a byte across all three. But the event stream shows the base pipeline was built betweenat: 0.6andat: 3.9before the LoRA phase failed atat: 4.8, and the allocated figure never rose during that window either — which reads like the base was built in host memory and never placed on the card. If that is right, then the leak path Failed pipeline loads leak GPU/host memory instead of being released, and leaks accumulate across retries #72 describes is precisely the one I cannot see, and I verified the half that was never in question. -
This server's templates keep weights in host RAM by design.
offload: "sequential"on the Z-Image steps,group_offloadon the H3 transformer and leaf-level on the text encoder. Under those configurations the card stays near-empty by construction — I measured ~8 MB allocated through an entire 1024x1024 Z-Image generation — so VRAM is close to the least informative place to look for a leak here. Host RAM is where the weights actually live.
The practical effect is that "check memory before a near-ceiling render", which both the ltx-2.5 and minimax-music3 skills now tell a consumer to do (#79), can only ever answer half the question on this box.
suggested: add host fields to info — resident set size of the worker process, and system total/available — beside the gpu_* ones. Same additive shape as the stale/reason/age_seconds keys added in #80, so nothing existing changes. If that is more than it is worth, tightening the description to say VRAM only would at least stop the interface promising a number it does not return.
Not urgent and not blocking anything I am doing; filing it because it is the reason I cannot fully close out #72's subject matter from the consumer side.
Activity
feat(mcp): #81 - get_memory reports host RAM beside VRAM
dkackman commentedon Sep 12, 2026
fixed (implementer, 2026-09-12T20:55Z) — commit
9598a95, ondevelop@7600e2f, deployed tolemand the server restarted at 20:49Z.Straightforwardly right: the description promised VRAM and RAM and the payload was all card.
infonow carries four host fields beside thegpu_*ones. Measured live onlemjust now:host_memory_rss_mb— the worker process's resident set, taken inside the worker, so it is the same "is this the worker's own view?" contractlive: truegives the GPU figures.host_memory_peak_rss_mb— its high-water mark. This is the one that answers your Failed pipeline loads leak GPU/host memory instead of being released, and leaks accumulate across retries #72 question: a load that fails partway through and leaks shows as a peak the resident figure never comes back down to, which is invisible in an instantaneous reading of either memory.host_memory_total_mb/host_memory_available_mb— the machine. Available, not free: page cache is reclaimable, so free understates what a load can have.Additive, exactly as you suggested — nothing existing changed shape. One deliberate detail: a reading the platform cannot take is absent rather than null, so a missing key means "not measurable here" and never "measured, and nothing". On
lemall four are present (psutil is installed; there is a/procfallback and aresource(2)one behind it, so this does not depend on that staying true).Your second point is the one that made this worth doing rather than a doc fix: on a box whose templates offload by design, the card is the least informative place to look. The tool description and docs/MCP.md now say that outright, so "check memory before a near-ceiling render" reaches the half that matters here.
memorycommand in the REPL prints them too.Tests:
tests/test_host_memory.py(8 tests — the fields, the absent-not-null rule, the psutil-less fallback, the worker payload, both REPL paths). Full suite 3785 passed, 5 skipped.To verify:
get_memory()on the idle server —infoshould carry the fourhost_memory_*keys beside thegpu_*ones. For the #72 subject matter, take a reading, run the failing-LoRA repro, and comparehost_memory_rss_mbandhost_memory_peak_rss_mbbefore and after; note the worker only reports while it is alive, so take both readings without a restart in between.dkackman commentedon Sep 12, 2026
verify-notes (tester): verified. All four fields are present and live on
lem, and — the part that matters — they answer questions the card could not.1. The fields, idle,
live: true:2. The offload case, which is the one that makes this worth having. Job
d5af55d26b2e(the #75 probe — a Z-Image step withoffload: "sequential"). Idle before, and immediately after:gpu_memory_allocated_mbhost_memory_rss_mbhost_memory_peak_rss_mbThat run peaked at 33 GB of host memory while the card never exceeded 8.5 MB. Under the old payload the honest consumer-side summary of that job would have been "used no memory", which is off by four orders of magnitude. Your point about the card being the least informative place to look on this box is not a caveat, it is the headline.
3. The #72 subject matter — the failed-load question, re-asked on the host side. Two runs of the failing-LoRA fixture (
aebd3af6d098,f262445180b6;ZImagePipeline+ aweight_namethat does not exist, fails at the LoRA phase in ~4 s), no restart between any reading:Two readings from that:
Peak behaves as documented as a high-water mark — it holds 33 GB across runs that never approach it, which is what makes the failed-load comparison possible at all.
Adding
status:verifiedand closing as completed. One small contract nit onpeak_rss < rssat idle is filed separately as #83 rather than held against this — it does not affect anything above.