Repository navigation
idea: compare_outputs MCP tool - face-crop grids, contact sheets and diff metrics across runs #871
Description
Problem
Every quality judgement in the matrix needed local tooling: download each mp4, ffmpeg a contact sheet (every 30th frame tiled), crop each face at fixed frames into a grid (tools/facegrid2.py), and ffmpeg psnr against a baseline. An agent on a text-only client without local ffmpeg can't do any of it; get_output_frames returns one clip's frames, not an aligned comparison.
The comparisons that mattered:
- Faces: same 300x260 native-res crop around each character at fixed frames (60, 100), rows = runs. Showed turbo smear vs base clarity at a glance, and that cache/compile rungs were indistinguishable.
- Staging: contact sheet rows = runs, same frame indices. Showed which knobs re-stage (LoRA family, base weights) and which don't.
- Numeric diff: PSNR vs a baseline separated "rerun noise" (44 dB) from "numerics changed" (23-27 dB) from "re-staged" (~19 dB).
Proposal
compare_outputs(outputs=[...], frames=[...], crops=[{label, x, y, w, h}] | "faces", layout="grid"|"sheet", baseline=<output>) returning one image plus per-output metrics (PSNR/SSIM vs baseline, mean abs diff, a same_staging heuristic from low-res frame similarity).
crops: "faces"could use a cheap face detector so the agent doesn't need coordinates (positions moved per run: B3 Howie at 244,42 vs L5 at 264,32).- Server-side so it works from any client; output saved under the run/workspace and returned as an image.
Questions
- Face detection dependency (mediapipe / insightface) vs requiring coordinates?
- Is
same_stagingreliable enough to report, or just give the numbers?
Source: H3 quality matrix on lem (2x RTX 3090 24 GB, 125 GB host RAM), workspace h3-quality, Known Issues ep 6 shots, seed 42. Full data and per-run assessments: ~/testing/opus/known-issues/quality/matrix.json, report quality/REPORT.md; every run in h3-quality carries a note with its config, minutes and score. Related: #861 (follow-up experiments).
Activity
dkackman commentedon Oct 14, 2026
Plan v2:
compare_outputs, aligned grids and diff metrics across runsLead agent, model
claude-opus-5-5viaanthropic. v2 records Don's decisions from his review of v1 (approved); see "Decisions" below.Decisions (from Don's review of v1)
compare_outputs.SURFACE_BUDGETby the measured cost, with a dated ledger entry.plugins/dw/skills/minimax-h3/SKILL.md(it is at 12,282 of 12,288 bytes). The pointer goes inplugins/dw/skills/minimax-h3/references/quality-and-speed.md.def, so the decode and SSIM CPU runs in FastAPI's threadpool.face_trackdownloads them today).Verdict: build smaller
Build a read-only
compare_outputsthat returns one aligned grid (rows = outputs, columns = frames, optionally cropped) plus PSNR, SSIM and mean abs diff against a baseline. Cut from the proposal:crops: "faces": deferred to a possible stage B. It needs no new dependency, because YuNet is already indw/tasks/face_track.py. It would come back if a field report shows per-output coordinates are still the bottleneck after stage A ships.same_staging: cut. dw reports findings, not verdicts (assess_outputsays so in its description). Instead, the bands the matrix measured go in the guide as a reading aid: ~44 dB rerun noise, 23–27 dB numerics changed, ~19 dB re-staged. It would come back if two or more comparison sessions read those bands the same way and it is wanted as a field.layout: cut. A "sheet" and a "grid" are the same output with more or fewer columns.list_gallery, run versions anddelete_outputall assume one can't exist. The grid comes back inline, likeget_output_frames. It would come back if someone asks to keep a grid, as akeep_asthat writes to the asset library the wayupload_assetdoes.Value
h3-qualityworkspace has 63 jobs. Every judgement in the matrix needed local ffmpeg andtools/facegrid2.py.get_output_frames(name, at=[...], crop=[...])(get_output_frames has no crop parameter, unlike get_output_image — checking a small region of a video frame needs full-frame sampling #303, Feature proposal: contact-sheet/frame-grid tool for previewing a clip without authoring a frames workflow #245) shows one output's frames. A grid of N runs therefore takes N calls and N sets of tiles in context, and the agent has to align them by eye. Nothing anywhere computes PSNR or SSIM, so the numeric half can't be done from a text-only client at all.Build cost
Inertia
SURFACE_BUDGEThas 0.5 tokens of headroom (tests/test_mcp_server.py:1594). A tool with this schema costs about 200–300 tokens. Stage A raises the ceiling with a dated ledger entry (Q3).POST /api/gallery/compare.docs/MCP.md,docs/SERVER.md, the OpenAPI snapshot, and the guide.resolve_output_file/asset_filepath asget_output_frames, and the decode goes throughopen_bounded.assess_output.Reversibility
breaking-changefor MCP clients, but nothing stores data that depends on it. Low lock-in.Cheaper alternatives
get_output_framescalls. This gives up the numbers entirely, plus the single aligned view.baseline=toget_output_frames: declined (Q1).compareworkflow task: no MCP surface cost. But it needs a queued job per comparison and writes a run output each time. That is heavier to use than the read-only call, and the GPU queue would block it behind renders.Overlap
dw/media_frames.pyalready has two grid compositors,contact_sheet/compose_gridandingredients_grid's cell fitting (docs/stabilization/phase-3-surveys/media.md:134). Stage A reuses them and adds no third.Problem and non-goals
Problem: judging "did this knob change the picture, and how much" across runs needs local tools. A client with no ffmpeg can't do it.
Non-goals:
latest/vNexpansion in names (gallery routes take literal names, asget_output_framesdoes);Design
MCP
names: 2–8 gallery names orasset:references. Videos and still images can both be given. A still is a single frame, andatdoesn't apply to it.at: the moments to compare, usingget_output_frames' grammar (seconds or"frame:N"), 1–8 of them. The default is 4 evenly spaced frames of the shortest video. With rows × columns capped at 8 × 8 = 64 cells, the grid stays withinMAX_CONTACT_SHEET_FRAMES.atkeeps that. Each moment resolves per output to a frame index, and outputs whose fps differs are compared at the same frame number. The reply flags this asfps_differs, so the agent knows the columns are not the same instant.crop: a single[x, y, w, h]box in source pixels, applied to every output, or one box per name. It is applied before any downscale, as inget_output_image/get_output_frames.metrics region: full frame (per-output crops are display only)).baseline: a name fromnames. It defaults tonames[0].fit_tiles_within_budget), and the reply saysdownscaled_to.psnr_db(∞ is reported asidentical);ssim(greyscale, Gaussian window, numpy +cv2.GaussianBlur; no new dependency, noskimage);mean_abs_diff(0–255).size differs from baseline: 1344x768 vs 960x544), or when a moment is past an output's end. In that second case the cell is blank as well, and the reply sayspast end (N frames). Never resize to force a number: a resize is a difference.REST
POST /api/gallery/compare: the body is{names, at, crop, baseline, max_dimension, max_total_bytes}.POST /api/gallery/archiveis the precedent for a multi-name gallery route. The handler is a syncdef, so the decode and SSIM CPU runs in FastAPI's threadpool, not on the event loop.names, a crop out of bounds for an output, and a list of crops whose length is notlen(names).Engine
dw/media_compare.pyholds the metrics:psnr,ssim,mean_abs_diffon uint8 RGB arrays, plus row and column assembly that reusesdw/media_frames.py(grid_tile,compose_grid).dw/media.pyread_frames.assess_output.Syntax
Docs
README.md,dw/server/guides.pyanddocs/MCP.md.docs/MCP.md: a row in the Media table.docs/SERVER.md: the new route.docs/WORKFLOW_GUIDE.md: a paragraph in Assessing a run's output. It covers what each metric means, which region it covers (per-output crops are display only), and the matrix's dB bands as a reading aid, not thresholds. The bands were measured on full frames.plugins/dw/skills/minimax-h3/references/quality-and-speed.md: one line pointing A/B comparisons at the tool. NotSKILL.md(at its byte cap).Stages
Stage A:
compare_outputs, end to end (server deploy, plus plugin reference line)What it builds: everything in the design above.
Tests and docs it owes:
identical/ SSIM 1.0, a known noise gives the expected PSNR, and a shift lowers SSIM;EXPECTED_TOOLS,TOOL_WIRING,WRAPPER_HANDLER_MAP, the stated tool count, and theSURFACE_BUDGETledger entry.Deploy: server restart on lem.
Estimate: $4–6.
Acceptance intent, as seen over MCP:
at=["frame:60","frame:100"]. The reply has one image with 2 rows × 2 columns, labelled. Metrics for the second row show a high PSNR (tens of dB) and SSIM near 1.identicaland SSIM 1.0.crop=[x,y,w,h]shows only that region in every cell. The metrics change from the uncropped call's, because they are computed on the region.at. Omittingatgives 4 evenly spaced columns.baselinenot innames;downscaled_toreported.list_galleryorlist_assetsafterwards.Stage B (deferred, not filed):
crop="faces"What it builds: YuNet (already pinned in
face_track.py) per output per moment. The k-th face, left to right, would be cropped to a fixed box centred on it, and the reply would give each detected box. It would be filed only if the trigger under the verdict is met.To settle before filing: where the YuNet weight download happens.
face_trackdownloads them in the worker today; stage B would otherwise fetch them from HF in the server process on first call.Order and dependencies
Risks
defroute keeps it off the event loop.Questions
None open. Q1–Q3 are answered under "Decisions".
dkackman commentedon Oct 14, 2026
Review of plan v1: approve with one design fix and one doc move. The code facts were checked against current develop:
dw/tasks/face_track.py.contact_sheet,grid_tile,compose_gridandMAX_CONTACT_SHEET_FRAMES = 64are indw/media_frames.py.fit_tiles_within_budgetandopen_boundedare indw/server/inline_media.py.POST /api/gallery/archiveis the precedent route.dwordw_mcpcomputes PSNR or SSIM.The cuts of
same_staging,layoutand saving the grid are right.Design fix: metrics must compare the same pixels.
With per-output crops (Q2), the plan computes metrics whenever the boxes are the same size. Two boxes at different positions are different pixels, though. PSNR between them mixes "the face moved" with "the picture changed", and the guide's reading bands (44 / 23–27 / ~19 dB) came from full-frame comparisons. Rule:
Doc move: no line in
minimax-h3/SKILL.md.It is at 12,282 of 12,288 bytes, and #862 and #868 already compete for its only trim. Put the pointer in
minimax-h3/references/quality-and-speed.md, or in #862's ramp section, where "how to judge" lives.Minor
face_trackdownloads them in the worker today. Settle where the download happens before filing stage B.def, so the decode and SSIM CPU runs in FastAPI's threadpool.Questions
SURFACE_BUDGETwith a dated ledger entry.claude-opus-5-5 / anthropic, for Don
dkackman commentedon Oct 14, 2026
Plan v2: changed since v1. Records Don's review decisions; scope unchanged.
SURFACE_BUDGETwith a ledger entry: all recorded as decided.minimax-h3/SKILL.mdtominimax-h3/references/quality-and-speed.md.def(threadpool).Lead agent, claude-opus-5-5 / anthropic
4 remaining items
dkackman commentedon Oct 14, 2026
Spec cases written from plan v2. Model: claude-opus-5-5 via anthropic. Each case carries
pending: #889.#889, stage A: the
compare_outputstoolregression-suite-complete.md:qr_code→film_grain→loop_framesworkflow with size, length and fps variants, so no model and no lem-only assets.identical, with SSIM 1.0, and the last frame is valid.baselineselects the reference without reordering rows.past end (N frames)(49/50 off-by-one), andfps_differsis flagged.atis 4 evenly spaced frames of the shortest clip.asset:names and still/video mixes are accepted.downscaled_to, and the metrics don't change with the downscale.regression-suite-security.md:baselinecan't reach a file outside the gallery or asset library (traversal, absolute paths,asset:escapes, percent-encoding). An existing file and a missing one get the same error, and no path is disclosed.Notes (not blocking):
minimax-h3/references/quality-and-speed.mdisn't something a consumer can check. Mark itdocs-reviewat hand-off if it's the only part left unverified.at=[]and negative moments may be refused or reported per cell, but a negative moment must never wrap to the end of the clip.aton a still may give one column or repeat the image.Merge docs/871-close-out into develop (#871)
dkackman commentedon Oct 14, 2026
Close-out: every stage of #871 is closed and the design record is on
develop. Lead agent, modelclaude-opus-5-5viaanthropic.compare_outputsandPOST /api/gallery/compare. It was verified on mps (mini-ai) and all 14 cases pass. It had 2 bounces: one architecture review (map rows and single owners) and one verify (C-F463, negative moments).crop="faces"). Its trigger, and the YuNet download question to settle first, are in the record.docs/proposals/complete/compare-outputs-complete.md, anddocs/proposals/todo.mdhas a shipped entry. Commit 5340e63, merged intodevelopat 9e8f15d. Docs only, so no deploy is needed.verify: run every case whose
source:names one of this feature's stages (#889: C-F454–C-F466, SE-F049), and close the parent if they all pass.dkackman commentedon Oct 14, 2026
Verified, by claude-opus-5-5 via anthropic, on mps on mini-ai.lan (develop @ 9e8f15d).
I ran every case whose
source:names #889, exactly as written, over MCP. All of them pass.crop [0,0,128,128], the region line readscrop [0, 0, 128, 128]and the metrics change (6.84 / 0.6066 / 55.78). The edge box[384,384,128,128]is accepted.full frame (per-output crops are display only), and the metrics equal the no-crop ones.size differs from baseline: …. A2 is identical. The out-of-bounds crop is refused withcrop [0, 0, 600, 300] is out of bounds for …, which is 512x512.skipped: past end (50 frames). Frame 49 is measured and frame 50 is past end. Different frame rates addfps_differs: the videos' frame rates differ.columns: still. Withat, the reply addsnote: \at` ignored: every output is a still image`. An asset compared with its own still is identical, and a png against an mp4 is measured.is negative - moments count from the start of the clip). The 8-name boundary is accepted.max_dimension1344 and 384. Note: the 1344 grid came back at 10752×10976 with nodownscaled_to, so this run never took the downscale path.get_server_infostates a tool count, so the tool-count check had nothing to read.baseline=) were refused, with no image, no metrics and no server path. Each reply echoes only the name as sent:Unknown file: <name> - path contains a disallowed pattern, or forasset:names a validator message (segment '..' starts with '.',segment 1 is empty,'%' … is not a character this kind of name may contain)./etc/hostsand../../../../etc/no-such-fileget identical wording, so the reply doesn't reveal whether a file exists. The[G, G]control returnedidentical.C-F459 fixture note: the case expects S to be 640×384, but the fixture's
qr_coderenders square, so S is really 640×640 (get_gallery_metadataagrees). The reason line correctly readsize differs from baseline: 640x640 vs 512x512. The case's literal is wrong, not the code. I filed the amendment as dkackman/harnest#119 rather than editing the case.I deleted all the fixtures (8 c-f454-cmp runs and the se-f049 run) with
delete_output(job_id=…). The assetc-f462-still.pngwas removed during the run. CUDA hasn't seen this, so the label isverified-on:mps.