Repository navigation
get_memory returns three different shapes and 'live: false' is undocumented, so the memory-hygiene check the skills prescribe can silently read the wrong process #80
Description
tool/endpoint: get_memory
repro: workspace qa-ep3, three calls this cycle, in this order. Nothing but the server state differs between them.
(a) Worker not running — get_health reported worker_alive: false, current_job: null, queued: 0:
{"live": false, "info": null}(b) Worker up and idle, immediately after job e19756398dc8 (a ~20 s bark speech job) succeeded:
{"live": true,
"info": {"run_count": 1, "gpu_available": true,
"gpu_memory_allocated_mb": 819.61865234375,
"gpu_memory_reserved_mb": 846.0,
"gpu_memory_free_mb": 22949.0625,
"gpu_device_name": "NVIDIA GeForce RTX 3090",
"gpu_memory_total_mb": 24122.1875}}(c) Worker busy, ~58 s into job b0dbf9c4d0a3 (templates/minimax/reference-to-video), while wait_for_job reported phase: "loading", phase_detail: "pipeline: MiniMaxAI/MiniMax-H3", seconds_in_phase: 57.7 — i.e. mid-load of a ~22 GB quantized H3:
{"live": false,
"info": {"run_count": 1, "gpu_available": true,
"gpu_memory_allocated_mb": 8.125,
"gpu_memory_reserved_mb": 40.0,
"gpu_memory_free_mb": 23755.0625,
"gpu_device_name": "NVIDIA GeForce RTX 3090",
"gpu_memory_total_mb": 24122.1875}}expected: one of two things, either is fine.
Either get_memory always answers for the worker, and says so when it cannot — an explicit "info": null plus a reason, the way (a) does — rather than returning a full, plausible-looking info block that is not the worker's.
Or it keeps returning what it can, and the tool description says what live means, what info: null means, and — the load-bearing part — that a live: false reading with a populated info is not the worker's memory and must not be compared against a live: true one.
actual: three shapes, none of them documented. The tool description is two sentences: "Get the worker's VRAM and RAM statistics. Check this first when a job fails with an out-of-memory error." It does not mention live, does not say info can be null, and does not distinguish (b) from (c) — in which the same gpu_memory_allocated_mb key comes back with a value that dropped from 819 MB to 8 MB while a 22 GB model was loading. run_count is 1 in both, so nothing in the payload flags (c) as a different vantage point either.
The reading in (c) is the one that worries me. 8 MB allocated / 23.7 GB free is what an agent gets if it asks the question at the wrong moment, and it is wrong in the dangerous direction: it says the card is empty when the card is nearly full. Whatever live: false means internally, from the consumer side (c) is indistinguishable from (b) except by a boolean nobody has been told to check.
Why this is more than cosmetic: #79 just shipped a step 4 into both the ltx-2.5 and minimax-music3 skills that reads, in full:
get_memoryon an idle server and readgpu_memory_allocated_mb. A non-trivial idle figure is what an earlier run left behind and comes off what this one has to work with.
That instruction has no defined result in (a) — there is no gpu_memory_allocated_mb to read, and the skill gives the agent no rule for the null, even though (a) is the good state: a worker that is not up is holding nothing. And in (c) it silently reads a different process and reports "plenty of room". An agent following the skill literally, which is what a skill is for, gets a dead end in one case and a wrong answer in the other. #72's "check the figure does not ratchet across failed attempts" has the same exposure — two readings taken at different live values are not comparable, and nothing tells the caller to compare them only within a value.
Suggested fix, cheapest first: state it in the tool description — what live is, that info may be null, and that only live: true readings are the worker's and only those are comparable with each other. Then add the same sentence to the two skills' step 4, with the rule for each state: live: false, info: null means nothing is resident, go ahead; live: false, info: {...} means ask again when the server is idle; live: true is the figure the advice is about. If instead the right answer is that get_memory should never return a non-worker info, that is a smaller consumer surface and I would prefer it, but it is your call.
Not a blocker for me — I can check get_health.worker_alive and current_job alongside and work out which state I am in. That workaround is the report: I should not have to.
repro workspace: qa-ep3. Jobs e19756398dc8 (bark, succeeded) and b0dbf9c4d0a3 (H3 reference-to-video) are both on the server.
Activity
fix(mcp): #80 - get_memory says whether a reading is the worker's
dkackman commentedon Sep 12, 2026
Fixed and deployed — both halves, since the payload was the cheaper half to fix properly.
Code (
ff0a498, merged todevelopas6b1d58d, pushed):/api/memory(and soget_memory) now carries three additive fields besidelive/info, which keep their meanings:stale— true wheninfois a cached reading rather than one measured by this call.reason— null whenlive, elsejob_running,worker_stopped,worker_busy(the poll raced a job start) orworker_unreachable(the worker did not answer or has died).age_seconds— how old the cached reading is; null when there is none.So your three states are now distinguishable without
get_health: (a){live:false, info:null, stale:false, reason:"worker_stopped"}, (b){live:true, stale:false, reason:null, age_seconds:0.0}, (c){live:false, info:{...}, stale:true, reason:"job_running", age_seconds:<n>}. I kept the cachedinforather than nulling it — it is the reading that matters most when the worker has just died OOM (there is a test pinning that) — but it now says it is not this moment's.Not a breaking change: no existing key changed shape or meaning.
Docs: the
get_memorytool description now states whatlivemeans, thatinfocan be null (and that null means nothing is resident, not unknown), and the load-bearing part — that onlylive: truereadings are the worker's and only those are comparable with each other, because a reading cached mid-load understates what is resident. Same in docs/MCP.md and docs/SERVER.md.Skills (plugin change, no server involvement): step 4 of
ltx-2.5andminimax-music3now gives the rule per state — read alive: truereading'sgpu_memory_allocated_mb;info: nullmeans nothing is resident, go ahead; alive: falsereading with aninfois cached, ask again when idle. The checkout is ondevelopwith that in it. (ltx-2.5 sits 1 byte under the 12 KiB skill cap, so a couple of secondary clauses elsewhere were tightened to make room.)Deploy:
lempulled6b1d58d,dw.serverestarted in its screen session at 15:2x CDT 2026-09-12, healthok/worker_alive:false(fresh worker, nothing resident), and/api/memoryanswers{"live":false,"info":null,"stale":false,"reason":"worker_stopped","age_seconds":null}.tools/listover/mcpconfirms the newget_memorydescription is live. Note the restart cleared the worker, so jobse19756398dc8/b0dbf9c4d0a3are still in history but nothing of theirs is resident.3763 tests pass.
dkackman commentedon Sep 12, 2026
verify-notes (tester): verified. All three states reproduced over MCP this cycle, on
0.4.0-beta.3/lem, and the shape that prompted this ticket is now self-describing.(a) worker stopped — first call of the cycle,
get_health→worker_alive: false:(b) live — between jobs,
get_health→current_job: null:(c) job running, cached reading — sampled ~1 min into job
d4cdcd044ca0(templates/minimax/reference-to-video, the 22 GB H3 ref2va load that produced the original report):That is the exact reading that caused this ticket — 8.125 MB allocated, 23.7 GB free, quoted mid-load of a model that peaks around 22 GB. It is still 8.125 MB, and it should be:
age_seconds: 57.0andstale: truenow say it is a 57-second-old measurement from before the load started, not this moment's. Nothing is wrong with the number; what was missing was any way for a caller to know what it was a number of. It is there now.A fourth shape, which I think is correct and worth recording: early in a job, before any live reading exists to cache, I got
Not in your enumeration of three, but it follows from the documented rule —
info: nullwithstale: falsemeans nothing has been measured — and it is unambiguous, so I read it as intended rather than as a gap. Flagging it only so the combination is on the record.Docs: read the
get_memorydescription live offtools/listthrough the session's MCP connection. It states whatlivemeans, thatinfo: nullmeans nothing resident rather than unknown, and the load-bearing rule that onlylive: truereadings are comparable with each other. That is what I needed.Not a breaking change, confirmed from this side:
liveandinfokept their shapes; the three new keys are additive and my existing calls were unaffected.This also unblocks what it was blocking — #72's before/after readings are now comparable with confidence, and I ran that repro this cycle off the back of it.
Closing as verified.
fix(security): confine a refused prompt lookup lexically before probi…