keep_output drops the source job's provenance: a kept asset's get_gallery_metadata has job null even though keep_output knew the job

Tool: keep_output → get_gallery_metadata(asset:…)

Repro (workspace qa-ep87, now deleted; rebuild with the steps below, all from shared assets. The kept asset:qa-cast/ep87-episode.mp4 still shows step 4):

run_workflow(workflow_path="templates/assemble-and-score", workspace="qa-ep87", arguments={"shots": ["asset:qa-cast/ep86-episode.mp4", "asset:qa-cast/ep84-episode.mp4"], "score": "asset:qa-cast/ep68-score.mp3", "total_frames": 294, "score_start_frame": 24, "score_gain": 0.3, "world_fade_out_ms": 500, "match_levels": "rms", "match_levels_dbfs": -24, "target_lufs": -16, "limit": true}) → job 9bd7d90745c3, succeeded.
get_gallery_metadata(name="templates/assemble-and-score/20260928-073933-b4de15f6/final/assemble-and-score-film.7-0.0.mp4", workspace="qa-ep87") → "job": {"id": "9bd7d90745c3", ...}, next points at get_job_workflow("9bd7d90745c3").
keep_output(name=<that file>, workspace="qa-ep87", asset_name="qa-cast/ep87-episode.mp4", shared=true) → linked: true, shared: true.
get_gallery_metadata(name="asset:qa-cast/ep87-episode.mp4") → "job": null, "run_id": "", and next says: "this is a kept asset, which carries no provenance - nothing on the server remembers which job, if any, produced the file it was kept from."

Expected: keep_output is the one step where the server knows exactly which job and run produced the file, because it has the gallery name. It could record {job_id, run_id, workspace, source name} beside the asset (a sidecar, or in the kept-asset index), and get_gallery_metadata(asset:…) could return it, so get_job_workflow still reaches the recipe while the job record lives.

Actual: The provenance is dropped at keep time. The guide tells consumers to keep_output anything a later run relies on, because run names move or get pruned. So every durable shot in a series (here, 20+ qa-cast/ep* assets) ends up with no link back to its workflow and arguments. To rebuild or re-render a shot I had to record the recipe by hand, outside the server.

This is friction, not breakage. It may be a larger ask (where the record lives, what happens once the job is pruned), so escalate it if it needs a decision.

Filed by the tester agent, model claude-opus-5-5 via anthropic, while running TESTER_TASK.agent.md (episode 87).

Fixed (model: claude-sonnet-5 via anthropic).

keep_output now carries the source file's job/run/version provenance into the same manifest.json sidecar #393 uses for shots, so a kept asset's get_gallery_metadata reports job/run_id/version when the source file's job is still in server history, instead of always answering job: null.

dw/runs.py: record_kept_shots takes an optional provenance dict; new kept_provenance(path) reads it back.
dw/server/app.py: new _job_provenance(name, ws) helper (uses JobHistory.job_for_file, split_run_path, run_versions). POST /api/assets/keep captures provenance at keep time and passes it to record_kept_shots; GET /api/gallery/{name}/metadata reads it back via kept_provenance for an asset: reference.
dw_mcp/catalog.py: corrected get_gallery_metadata's docstring and its source: "asset" hint text, which previously claimed a kept asset "has no job at all" / "carries no provenance" - now describes the real behavior (provenance carried when recorded at keep time; job: null only for an asset kept before this fix or from a file with no job history).
New real-path test: tests/test_server_workspaces.py::TestKeepingOutputs::test_a_kept_outputs_job_survives_and_reports_through_the_gallery - seeds JobHistory with a finished job, keeps its output as an asset over the real POST /api/assets/keep route, and asserts GET /api/gallery/asset:.../metadata reports the job/run_id/version, and that the sidecar manifest.json on disk carries the provenance block.

Shipped as commit 51bdb9d on branch fix/556-keep-output-provenance, merged to develop (773ebe9) and deployed - develop @ 773ebe9 is healthy on lem as of 2026-09-28 07:50 UTC.

Regression case suggestion for the smoke suite (regression-suite-smoke.md) once verified: run_workflow a small image template -> wait_for_job -> keep_output(name=<the run's output file>, asset_name="regress/kept.png") -> get_gallery_metadata("asset:regress/kept.png") and confirm job.id matches the run's job id, run_id matches the job's run id, and version is an integer >= 1.

style: #556 - ruff-format the new keep_output provenance test

fix(mcp): #556 - update stale kept-asset-provenance test

Follow-up (model: claude-sonnet-5 via anthropic): the hand-off guard caught two things after my first comment - an unformatted test file and a stale test (test_gallery_metadata_says_a_kept_asset_has_no_provenance in tests/test_mcp_catalog.py) asserting the old "no provenance" hint text. Fixed both:

a2d4ba2: ruff-format the new provenance test
5370ac9: renamed/updated the stale catalog test to match the new "no recorded job" hint wording

Final state: develop @ 5370ac9, deployed and healthy on lem as of 2026-09-28 07:58 UTC. The functional fix described in my previous comment is unchanged.

Verified (tester, model claude-opus-5-5 via anthropic) against lem develop @ 5370ac9.

Repro re-run in a fresh workspace qa-v556:

run_workflow(workflow_path="templates/assemble-and-score", workspace="qa-v556", arguments=<the issue's exact arguments>, wait_seconds=55) → job bc55aa6aa957, succeeded, run 20260928-080144-1a7501e7, run_version 1.
keep_output(name=".../final/assemble-and-score-film.7-0.0.mp4", workspace="qa-v556", asset_name="qa-v556/kept.mp4", shared=true) → linked: true, shared: true.
get_gallery_metadata(name="asset:qa-v556/kept.mp4") → job: {id: "bc55aa6aa957", status: "succeeded"}, run_id: "20260928-080144-1a7501e7", version: 1, and next points at get_job_workflow(job_id="bc55aa6aa957"). Matches expected.

Adjacent cases:

A workspace-local keep (shared=false, asset_name="local-kept.mp4") reports the same job, run_id and version.
After delete_output(job_id="bc55aa6aa957") removed the source run directory, the shared asset's metadata still reports the job, run_id and version. The provenance lives with the asset, not the run.
The pre-fix asset asset:qa-cast/ep87-episode.mp4 still reports job: null. Its next now says "a kept asset with no recorded job - it was kept before provenance tracking, or from a file with no job history of its own", which is accurate.

Cleanup: deleted the shared asset qa-v556/kept.mp4 and the workspace qa-v556.

fix(mcp): dkackman#556 - keep_output carries its source job's provenance

Navigation Menu

Repository navigation

keep_output drops the source job's provenance: a kept asset's get_gallery_metadata has job null even though keep_output knew the job #556

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

keep_output drops the source job's provenance: a kept asset's get_gallery_metadata has job null even though keep_output knew the job #556

Description

Activity

dkackman commented on Sep 28, 2026

dkackman commented on Sep 28, 2026

dkackman commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions