bug: MagCache calibrate ratios never reach an MCP/REST client - docs say they're logged

Carved out of #875 (plan v1, approved: Q1 yes). Item 4 of #875, a MagCache rung for H3, can't finish over MCP until this lands.

Filed by the lead agent, model claude-opus-5-5 via the anthropic provider.

The bug

docs/ACCELERATION.md:78 and the calibrate description in dw/workflow_schema.json (~line 808) say a mag cache with calibrate: true "logs the measured ratios … paste that array into mag_ratios". No MCP or REST client can see them:

At the last step, diffusers' _advance_step (hooks/mag_cache.py:379-394 in the installed venv) print()s the ratios, logs them at INFO on diffusers.hooks.mag_cache, and clears the list in the same call.
The worker is a spawned process whose stdout goes to the dw.serve console (dw/worker_manager.py:73-81).
The file handler is attached only to the dw logger (dw/log_setup.py:23-39), and diffusers sits at WARNING, so the INFO line is dropped.
Job events come only from explicit emit_* calls (dw/events.py:246-249), so get_job, get_job_events and export_job carry no trace of the ratios.

The fix (no new surface)

While a mag cache with calibrate: true runs, capture the ratio list before diffusers clears it. Emit it as:

an existing log event, and
a job warning, so it survives the 200-event history trim.

Message: MagCache calibration (N steps): [ … ] — paste into mag_ratios.

Capture. Reading after the forward is too late. Either snapshot the state's list each forward and take the final value, or capture the diffusers.hooks.mag_cache log record. You pick. A unit test pins the choice against the installed diffusers.
CFG (Don's addition). Under CFG, dw nests cache_context("cond") and cache_context("uncond"), each with its own MagCacheState, so a CFG run yields two ratio lists while mag_ratios takes one array. Emit both, each labeled with its context, or emit cond only and say so in the message. State which you chose in the docs.
Docs. Fix docs/ACCELERATION.md:78 and the schema's calibrate description to say where the ratios appear: job warnings and events. Include the CFG behavior.
No new event type, MCP tool, field or syntax.

Where: this probably lands in stateful_cache_context (dw/pipeline_processors/components.py, ~833-864). #867 stage 2 hooks the same mag_cache_leader_block_hook state there. Per Don (Q2), the two are independent: whichever lands second rebases.

Owes

A unit test for the capture against the installed diffusers, covering CFG (two contexts).
The two doc fixes above.
A restart of dw.serve before verification. The persistent worker caches engine code.

Passes when

On a running server, over MCP:

Run a small cached step with a CFG template (an SD template, not a guidance-distilled one such as Flux), ~10 steps, cache: {type: mag, calibrate: true}. get_job.warnings and get_job_events carry the calibration message. Each emitted list has length equal to the step count, and the CFG labeling matches the docs.
Run the same step without calibrate. No calibration message appears.

triage: work — no new surface (existing log event + job warning), likely in stateful_cache_context (dw/pipeline_processors/components.py); independent of #867 stage 2 per Don. Needs a server restart before verification. (claude-opus-5-5, anthropic)

fix(cache): #891 - mag calibrate ratios reach job warnings and log ev…

style: #891 - ruff format

Fixed in a27aaab (+ format-only 95813b3) on develop, deployed to lem @ 95813b3 (03:02, healthy).

What changed

New captured_mag_calibration (dw/pipeline_processors/components.py), entered around each pipeline call in pipeline.py. Active only when the transformer's cache config has calibrate: true. It attaches a handler to the diffusers.hooks.mag_cache logger and catches the "Calibration Results" record, which diffusers emits before clearing the list.
Each record becomes an emit_log and an emit_warning: MagCache calibration (N steps) [cond]: [...] — paste into mag_ratios.
CFG choice: both lists are emitted, labeled [cond] / [uncond] from the active cache context. The cond message adds a note that cond is the usual choice. A non-CFG run has no label.
Docs: docs/ACCELERATION.md and the schema calibrate description now say the ratios appear as job warnings and log events, and describe the CFG behavior.

Tests: two new cases in tests/test_cache_blocks.py drive the real installed-diffusers mag hooks (cond + uncond) and check both messages, plus no message without calibrate. Not exercised end-to-end on a GPU. The full suite has 9 failures in test_examples.py (Kandinsky6 classes missing from installed diffusers); they also fail on the prior commit.

To verify: per Passes-when: ~10-step SD (CFG) step with cache: {type: mag, calibrate: true}, then without calibrate. Expect two lists of length = step count, labeled cond/uncond.

(claude-sonnet-5-5, anthropic)

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (39d59692b4..95813b3deb: dw/pipeline_processors/components.py dw/pipeline_processors/pipeline.py dw/workflow_schema.json). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Fixed again, on a stronger model (claude-opus-5-5, anthropic) than the sessions behind the earlier fixes. Fix commit 6e506bae merged to develop as 8b9dad7 and deployed to lem at 2026-10-14 05:25 (systemd restart, healthy).

Why Z-Image showed nothing. The capture code was fine. On Z-Image the cache never attached at all: ZImageTransformer2DModel has no diffusers enable_cache() (it isn't a CacheMixin). dw skipped it with a logger.warning that only went to the server log. So neither the calibration message nor its step-count warning could appear.

What changed

dw/pipeline_processors/components.py: skipping a transformer without enable_cache() is now a job warning, <step>: MagCacheConfig was not applied: ZImageTransformer2DModel does not support diffusers' enable_cache(), so this step runs uncached. It goes through emit_warning, so get_job.warnings shows it. It covers every cache type, not only mag.
dw/cache_blocks.json: registers QwenImage21TransformerBlock, so Qwen-Image-2.1 (a CacheMixin running two-pass CFG in cond/uncond contexts) can take mag/first_block caches.
docs/ACCELERATION.md and the schema's calibrate description:
- The wrong "SD-style template" CFG wording is gone.
- They now say which pipelines emit two labeled lists and which emit one, and that SD 1.5 and Z-Image warn and report nothing.
- They add that Qwen-Image-2.1 needs "use_kv_cache": false under mag/first_block: its own KV cache drops text tokens after step 0, and the step-to-step residual comparison then fails.
Test: test_cache_on_transformer_without_enable_cache_warns in tests/test_cache_blocks.py.

Confirmed on lem through the deployed server (MCP run_workflow, worker path), as asked:

job	shape	result
`94aa0f241feb`	Qwen-Image-2.1, 10 steps, 512², `true_cfg_scale: 4.0`, `negative_prompt`, `use_kv_cache: false`, `cache: {type: mag, calibrate: true}`	`get_job.warnings` and `get_job_events` (log) both hold `MagCache calibration (10 steps) [cond]: [10 values] — paste into mag_ratios …` and `… [uncond]: [10 values] …`, plus the "declares 28 but this step runs 10" warning
`b37f0dbb5692`	same, no `calibrate`, the cond list pasted into `mag_ratios`, `num_inference_steps: 10`	succeeded, no calibration message
`ccc8470cde05`	Z-Image Turbo, mag + calibrate	the new "was not applied … runs uncached" warning

(Outputs deleted afterwards.) Mag without calibrate and without mag_ratios fails in diffusers ("mag_ratios must be provided"). That is upstream behavior, so the negative case needs ratios.

To verify: use an inline workflow with component_type: QwenImage21Pipeline, Qwen/Qwen-Image-2.1 in bf16, offload: model, and configuration.cache: {type: mag, calibrate: true}. Its arguments are true_cfg_scale: 4.0, a negative_prompt, use_kv_cache: false, num_inference_steps: 10 and 512x512. It takes about 45 s.

Proposed regression case (regression-suite-model-specific.md): the Qwen-Image-2.1 call above. Expect two warnings starting main: MagCache calibration (10 steps) [cond] / [uncond], each list holding 10 numbers.

— claude-opus-5-5 (anthropic)

Architecture review queued by the loop driver: this hand-off changed dw/ or dw_mcp/ on develop (c96e95318d..8b9dad7f2d: dw/cache_blocks.json dw/media_compare.py dw/pipeline_processors/components.py dw/server/guides.py dw/server/routes/media.py dw/workflow_schema.json dw_mcp/media.py dw_mcp/server.py ). The reviewer checks it against docs/ARCHITECTURE.md before the tester verifies.

Architecture review: pass (6e506ba, merged as 8b9dad7; only the #891 commit in c96e9531..8b9dad7f under dw/, dw_mcp/. The #889 compare_outputs commits in that range are other work and weren't reviewed here.)

I read the dw/pipeline_processors/components.py, dw/cache_blocks.json and dw/workflow_schema.json diffs against docs/ARCHITECTURE.md and item 3 of stage-c-guardrails.md:

Second owner: none.
- The "does not support diffusers' enable_cache(), so this step runs uncached" skip in enable_cache_on_transformer used to be a logger.warning. It now goes out through dw/events.py's emit_warning, the owner the "Run-time warnings" row names. That matches the no-transformer warning above it from the previous round.
- "Acceleration warnings" (dw/acceleration_warnings.py) has no rule about a transformer's CacheMixin support. The new message is a run-time fact about the loaded model, not a copy of a validate-time rule.
Build vs. buy: none. The Qwen-Image-2.1 entry is data for the existing owner dw/cache_blocks.py, which feeds diffusers' own TransformerBlockRegistry and skips any block diffusers already registers. Nothing is hand-rolled.
Map: no row owed. There's no new module and no rule moves. A new JSON entry in an existing registry doesn't change who owns what.
Context: fine. No CLAUDE.md changed. The schema's calibrate description is user-facing documentation, not rationale.

Note, not a finding: components.py is now about 1,009 lines. That is still past the 1,000-line SIZE_WARNING and under the 1,100 ceiling. My earlier note stands: the cache-hook helpers are the natural split if it grows.

Nothing was waived (the issue has no arch-approved label). It goes on to the tester's MCP verification as labelled.

Reviewer: model claude-opus-5-5 via the anthropic provider.

Verified. lem: develop @ 92c3952 (includes 8b9dad7). Every run was an inline run_workflow in workspace regression-complete. All outputs were deleted afterwards.

Positive, CFG (job c2a64b60e158): QwenImage21Pipeline, Qwen/Qwen-Image-2.1 in bf16, offload: model, cache: {type: mag, calibrate: true}, true_cfg_scale: 4.0, negative_prompt, use_kv_cache: false, 10 steps, 512². It succeeded in about 43 s.
- get_job.warnings holds gen: MagCache calibration (10 steps) [cond]: [10 values] — paste into mag_ratios (classifier-free guidance runs two lists; the cond one is the usual choice) and … [uncond]: [10 values] — paste into mag_ratios.
- It also holds the "declares num_inference_steps 28 but this step runs 10; calibrating over 10" warning.
- get_job_events(kinds=["log"]) carries both messages as log events (seq 24 and 26).
- Each list has 10 entries, which equals the step count, and the lists are labeled per context as the hand-off describes.
Negative (job 5f9113245108): the same step, but without calibrate, with num_inference_steps: 10 and the cond list pasted into mag_ratios. It succeeded, and its warnings show no calibration or step-mismatch message.
Adjacent, Z-Image Turbo (job 3d7e310556b9): the shape from my earlier bounce, ZImagePipeline with mag + calibrate at 10 steps. get_job.warnings now carries gen: MagCacheConfig was not applied: ZImageTransformer2DModel does not support diffusers' enable_cache(), so this step runs uncached. The cache is no longer silently ignored.

The served schema/docs wording wasn't re-read in this session. The behavior matches the issue's "Passes when", run on Qwen-Image-2.1 as the hand-off's CFG pipeline in place of an SD template, which can't take a transformer cache.

Tester: model claude-opus-5-5 via the anthropic provider.

Follow-up: #927. On H3, calibration emits nothing because the scheduler runs num_inference_steps − 1 forwards and the capture sets the counter to num_inference_steps. claude-opus-5-5 / anthropic

Navigation Menu

Repository navigation

bug: MagCache calibrate ratios never reach an MCP/REST client - docs say they're logged #891

The bug

The fix (no new surface)

Owes

Passes when

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug: MagCache calibrate ratios never reach an MCP/REST client - docs say they're logged #891

Description

The bug

The fix (no new surface)

Owes

Passes when

Activity

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

17 remaining items

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions