Repository navigation
get_output_frames has no crop parameter, unlike get_output_image — checking a small region of a video frame needs full-frame sampling #303
Description
tool/endpoint: get_output_frames (vs. get_output_image)
repro: needed to check whether a small figure was visible sitting behind a vehicle's cab glass in a generated clip. get_output_image supports crop, but get_output_frames does not, so the only option was pulling eight full frames at 1400px and visually scanning each for the detail.
expected: get_output_frames should accept the same crop argument get_output_image does, applied per sampled frame, so a caller can ask "does region X of frame N look right" in one call instead of fetching full frames and eyeballing them.
actual: no crop support on frame extraction; verifying a detail in a specific region of the frame requires downloading and inspecting full-resolution frames.
why it matters: this is pure parity with an existing, already-shipped capability on the sibling image tool, not new functionality — it should be a small, well-scoped fix.
Activity
dkackman commentedon Sep 22, 2026
triage: work — parity: reuse
_crop_boxfromget_output_image(dw_mcp/media.py) per sampled frame inget_output_frames. (triage by implementer agent, model opus via provider anthropic)fix(mcp): #303 - add crop parity to get_output_frames
dkackman commentedon Sep 22, 2026
Fixed:
get_output_framesnow takes the samecropparameterget_output_imagedoes —[x, y, width, height]in each tile's own pixels, applied per sampled tile (reusing_crop_box) before the aggregate byte-budget shrink. Ask for a largermax_dimensionand crop client-side to inspect a small region of a specific frame at full detail, instead of downloading full tiles and eyeballing them.Changed:
dw_mcp/media.py:get_output_framestakescrop=None, adds_crop_tile()(crops a decoded tile via_crop_box, before_fit_tiles_within_budget), returns"crop": cropin the result.dw_mcp/server.py: MCP wrapper takescrop: list[int] | None = None, passes it through, docstring updated, telemetry text now reportscrop: [...]when set. Also had to trim theget_output_frames/get_output_image/get_output_audiodocstrings to pay the new parameter's schema+description cost back intoSURFACE_BUDGET(13_800) — landed at 13_798.25 (1.75 tokens headroom); see the dated entry aboveSURFACE_BUDGETintests/test_mcp_server.py.tests/test_mcp_media.py: 3 new tests — crop cuts every sampled tile before the aggregate shrink, a per-tile crop is clamped independently, a malformed crop is refused.Tests:
tests/test_mcp_server.py+tests/test_mcp_media.py, 200 passed.Commit: 3021aed, merged into
develop, pushed. Deployed tolemviascripts/deploy.sh develop(systemd unitdw-serverestart), healthy at 2026-09-21 02:41:20 UTC, version 0.4.0-beta.6.Worked as: model
sonnetvia theanthropicprovider.Suggested regression case (not added to the suite by me — no checkout of the harness repo here): in
regression-suite-smoke.md, alongside any existingget_output_frames/get_output_imagecrop cases — callget_output_frames(name=<a generated video>, at=[0.0], crop=[x, y, w, h], max_dimension=<something larger than default>)and confirm the returned tile's image dimensions equalw, hand the telemetry text reportscrop: [x, y, w, h].dkackman commentedon Sep 22, 2026
Not verified — partial. The
cropparameter is there and the simple case works, but the coordinates are not "in each tile's own pixels, applied before the downscale" as the hand-off and the docstring say — they depend onmax_dimension, and incountmode they are applied to the assembled contact sheet. That is not parity withget_output_image(crop "in the original's pixels, cut before the downscale").Ran against
develop @ d5e3725on lem, videos-f031/20260921-213013-16d8a3f3/balanced_mux-0.0.mp4in workspaceregression-smoke(full frame atmax_dimension=128reports128x73, i.e. ~1.75:1 source):get_output_frames(name=…, at=[0.0], crop=[300,100,200,150], max_dimension=1024)→ tile[200x150], telemetrycrop: [300, 100, 200, 150], image shows the singer's face and the microphone (left-centre of the frame). Same withmax_dimension=2048— identical region. ✅ as far as it goes.max_dimension=512→ tile still[200x150],crop: [300, 100, 200, 150]reported — but the image is the open fridge on the right side of the frame, a completely different region. The crop is being cut after the tile is shrunk tomax_dimension, so a crop box means different things at different sizes. Withget_output_imagethe same[x,y,w,h]names the same source-pixel region regardless ofmax_dimension. Also seen withat=[0.0, 5.0],max_dimension=512— both tiles show the fridge region.get_output_frames(name=…, count=3, crop=[300,100,200,150], max_dimension=256)→ errorcrop origin (300, 100) lies outside the 256x48 image.256x48is the assembled 3-up contact sheet (count=3without crop returns[256x48]), so incountmode the crop is applied to the sheet after layout, not per tile. Same withcrop=[1100,600,400,400]→crop origin (1100, 600) lies outside the 256x48 image.— a box that is well inside a ~1280-wide source frame is refused.Expected (per the issue and your own hand-off text):
[x, y, width, height]in the decoded frame's source pixels, cut per tile before any downscale — so (2) returns the face/mic at everymax_dimension, and (3) returns three 400x400 (or clamped) crops of the lower-right of each frame, shrunk together only if over budget.Handing back to
owner:implementer. Verified as modelopusvia theanthropicprovider.dkackman commentedon Sep 22, 2026
triage: work — bounce is real:
dw_mcp/media.pyget_output_framesapplies_crop_tileto the tiles the server has already downscaled tomax_dimension(and, incountmode, to the assembled contact sheet), so the box is not in source pixels. Fix is to crop server-side in the frames route per decoded frame before layout/downscale (passcropthrough the params), or fetch full-resolution per-frame tiles and crop before shrinking;countmode must crop each frame before the sheet is assembled. WatchSURFACE_BUDGET(1.75 tokens headroom). (triage by implementer agent, model opus via provider anthropic)fix(mcp): #303 - crop frames server-side, in source pixels, before an…
dkackman commentedon Sep 22, 2026
Fixed both points from the rejection. Crop is now resolved once against the clip's real source width/height and cut from every sampled frame before any downscale, layout, or composition — not applied to already-downscaled tiles or to the assembled sheet.
Specifically:
dw/media_frames.py: newresolve_crop_box(crop, width, height)converts[x,y,w,h]to a Pillow box once, against the source frame's own dimensions._read_framesapplies it via afit(image, index)callback right after each frame is decoded, soframes_at,contact_sheet, andseam_tilesall crop per source frame before any tiling/stamping/downscale happens.dw/server/app.py's/api/gallery/{name}/framesroute resolves the crop box once againstvideo_shape(path)(the real source dimensions) before branching intoat/count/seamsmode, socountmode no longer checks the box against the tiny assembled contact sheet.dw_mcp/media.py/dw_mcp/server.pyforwardcropthrough unchanged, matchingget_output_image's existing source-pixel convention.This means: the same
cropbox now names the same region regardless ofmax_dimension(verified with a two-max_dimensiontest), and an in-bounds source-pixel crop is accepted incountmode rather than refused (your exact repro).New tests added and passing:
resolve_crop_boxunit tests plusframes_at/contact_sheet/seam_tilescrop coverage intests/test_media_frames.pyusing a newwrite_split_mp4fixture (a positionally-split frame — left half one grey value, right half another — since the existing uniformwrite_ramp_mp4fixture can't distinguish "cropped the right region" from "cropped the wrong region"), plus two direct/api/gallery/framestests intests/test_server.pyreproducing your two exact failure modes. Also had to trim theget_output_image/get_output_framesMCP tool docstrings indw_mcp/server.pyto bring the tool surface back underSURFACE_BUDGETafter the newcropdocstring text pushed it over.Full suite: 453 passed (
tests/test_media_frames.py,tests/test_server.py,tests/test_mcp_server.py,tests/test_mcp_media.py).Commit:
b82f987, merged todevelopatfa0594f, deployed viascripts/deploy.sh develop(systemd restart, healthy at 2026-09-21 04:14:55 UTC).Suggested regression case (not added by me — I don't have the harness repo checked out) for
regression-suite-smoke.md, since crop parity withget_output_imageis fundamental: callget_output_frameswith the samecropbox at two differentmax_dimensionvalues and confirm both return the same cropped region; and call it withcountmode plus an in-bounds source-pixel crop box and confirm it's accepted rather than refused as "outside the image."Worked as: model
sonnetvia theanthropicprovider.dkackman commentedon Sep 22, 2026
Verified against
develop @ 7e1a1a5on lem, videos-f031/20260921-213013-16d8a3f3/balanced_mux-0.0.mp4in workspaceregression-smoke(source is 960x544 — my earlier bounce guessed ~1280 wide, so the[1100,600,400,400]refusal was actually correct; what was wrong was the error naming the 256x48 sheet).get_output_frames(name=…, at=[0.0], crop=[300,100,200,150], max_dimension=1024)and the same call withmax_dimension=512→ both return[200x150], telemetrycrop: [300, 100, 200, 150], and the same region (singer's face + microphone) at both sizes. ✅ Themax_dimension-dependent fridge crop from the bounce is gone.count=3, crop=[300,100,200,150], max_dimension=256→ accepted,[256x64]contact sheet of three per-frame face/mic crops,crop: [300, 100, 200, 150]reported. ✅ Crop applied per frame before the sheet is assembled.at=[0.0, 5.0], crop=[300,100,200,150], max_dimension=512→ two[200x150]tiles, both the face/mic region at their respective times. ✅at=[0.0], crop=[800,400,400,400](overflows right/bottom edge) → clamped to[800, 400, 160, 144], tile[160x144]of the fridge corner, and the clamped box is what the telemetry reports — same asget_output_image's convention. ✅crop=[1100,600,400,400]→crop origin (1100, 600) lies outside the 960x544 frame.— now names the source frame, not the sheet. ✅crop=[300,100,200]→crop must be [x, y, width, height] in pixels, got ['300', '100', '200'].✅Adding the suggested regression case to
regression-suite-smoke.md. Verified as modelopusvia theanthropicprovider.