Derive a workflow's cost from this server's job history (proposal: observed vs curated) #93

Activity

1. dkackman

   added

   owner:donParked for human decisionParked for human decision

   status:needs-approvalParked pending a written proposal + human approvalParked pending a written proposal + human approval

   on Sep 13, 2026
2. dkackman commented on Sep 14, 2026

   

   dkackman

   on Sep 14, 2026

   OwnerAuthor

   More actions

   Approved — ship it as observed in the listing, per the proposal's recommendation (bucketed by declared cost_drivers, default-arguments-only fallback, cache-hit runs excluded, cold/warm split), with four conditions.

   Answer to the open question: a derived sibling in the listing is fine. "Never derived" protects what cost means — a person's claim on a named card — not the catalog's right to report its own history under a different name. A separate get_workflow_history call fails exactly the way #91 did: the consumer has to know to ask. Keep the shapes distinct, as the proposal already insists.

   Conditions:

   1. No unqualified median_minutes. Curated cost.minutes is defined in dw/workflow_schema.json as wall clock including model load — cold. Report cold_minutes and warm_minutes, each with its own run count, so the number comparable to cost is the one so named. Confirm the job history actually records a load phase per run before committing to the split; if it doesn't, ship cold-only and say so in the hand-off.

   2. Bucket from the job row, not the run directory. dw/server/jobs.py already stores arguments and manifest per job. One SQL query; no per-run JSON reads; survives run-dir deletion.

   3. cost_drivers ships populated. Declare it in the same change on every template that currently carries a curated cost, plus the MiniMax templates from list_workflows reports cost:null for templates this box has run many times, so a consumer told to quote a price before spending has nothing to quote #91. With no drivers the fallback is default-args-only and real runs pass arguments, so an undeclared catalog keeps answering null.

   4. Compact listing budget (Assess: context cost of the dw MCP tool schemas, plugin, and skills — are they too verbose for the agents? #101). The full block goes in the full listing / get_workflow. The compact listing carries at most observed_minutes (cold) and observed_runs.

   Accepted limits: static drivers can't express a cost that depends on an asset (a reference video's length); since + runs is the whole pruning story.

   Regression cases to propose in the hand-off comment (tester adds them once verified over MCP):

   • Fresh workspace, template with cost_drivers, two runs at the same driver values with different prompts and seeds → get_workflow shows observed with runs: 2, one bucket.
   • Third run with a different num_frames → the original bucket still reports runs: 2, never 3.
   • Seeded rerun that hits the step cache on every step → runs unchanged.
   • Template with no cost_drivers and no default-args runs → observed absent, cost_basis still "curated".

   Released to the implementer.
3. dkackman

   added

   owner:implementerImplementer's turn to actImplementer's turn to act

   and removed

   owner:donParked for human decisionParked for human decision

   status:needs-approvalParked pending a written proposal + human approvalParked pending a written proposal + human approval

   on Sep 14, 2026
4. dkackman

   added 3 commits that reference this issue on Sep 14, 2026

   feat: #93 - what this box's own history says a workflow costs

   ...

   33f0b21

   Merge fixes for #122 #141 #142 #96 #93 - elision, task signatures, so…

   ...

   95e9e6b

   fix: #93 - the observed figure says which accelerator it is about

   ...

   ac55683
5. dkackman commented on Sep 14, 2026

   

   dkackman

   on Sep 14, 2026

   OwnerAuthor

   More actions

   Implemented as approved. Implementer agent, model opus via provider anthropic.

   Commits: 33f0b21 and ac55683 on develop (merge 95e9e6b); deployed on lem at ac55683 (server restarted, health confirmed, 4471 tests pass on the box).

   What shipped

   dw/server/observed_cost.py reports observed beside cost. cost is untouched and still means what it means.

   The report that started this now answers. Live on lem:

   GET /api/workflows/templates/minimax/reference-to-video/variables
   "observed": {"device": "cuda", "name": "NVIDIA GeForce RTX 3090",
                "runs": 10, "comparable": "drivers",
                "cold_minutes": 8.04, "cold_runs": 10, "cold_range_minutes": [7.75, 11.08],
                "drivers": {"height": 544, "num_frames": 124, "num_inference_steps": 20, "width": 960},
                "since": "2026-09-11T20:30:38Z"}
   

   Against a curated 8.0, and against the tester's hand-kept ~7.8. Thirteen entries in the compact listing now carry a figure, including four with no curated cost at all.

   The four conditions

   1. No unqualified figure. cold_minutes/cold_runs and warm_minutes/warm_runs, each with its own count; a side with no runs is absent rather than null. There is no minutes or median_minutes key — a test asserts that.
      The split is real, confirmed before committing to it, as you asked: lem's history has 106 named succeeded runs, 77 carrying a loading phase event. But persisted events are trimmed to the newest 200 (MAX_PERSISTED_EVENTS), and 10 of those 106 hit that cap — so a long run's loading phase can be gone from the record. Those count toward runs and toward neither side, reported as unclassified_runs. Calling a run warm because the evidence was trimmed is the one answer that would have been wrong, so the numbers visibly do not add up rather than quietly misadding.
   2. Bucketed from the job row. One query (JobHistory.finished_runs), no per-run JSON reads, so a figure survives a pruned run directory. The cold/warm decision is made in SQL on the event tail, so 200 events per row are never parsed to answer a yes/no. Cached against JobHistory.watermark() rather than a file mtime — a job landing changes every figure and changes no file.
   3. cost_drivers ships populated — every workflow carrying a curated cost, plus every MiniMax and LTX-2.5 template (37 in all). Two abstain deliberately and are listed with reasons in the test: upscale-spandrel (its cost follows the input image, which is an asset, not a variable — your accepted limit) and consistent-set (nothing numeric moves it).
   4. Compact budget. observed_minutes (cold) and observed_runs only. I raised COMPACT_BUDGET 6,400 → 7,100 and changed how it is measured: the budget test now attaches observed figures to every entry, because measuring against an empty jobs table passed at 6,276 while the real server was at 6,381 and the worst case (all 65 compact entries with history) is 7,056. A budget that only holds on a box with no history is not a budget. I kept two numeric keys rather than one terse phrase (constraints' shape) — it costs 179 tokens and an agent quoting a price should read a number, not parse a sentence.

   One design decision I made that the proposal left open

   A list driver buckets on its length, not its contents. A shots list costs what its number of entries costs; bucketing on contents would give every run its own bucket and so a permanent runs: 1, which is the same as reporting nothing. drivers reports the length ("shots": 4) rather than carrying a default shot list into the listing.

   Also: the reported bucket is the one the defaults give, which is what keeps it comparable to a curated figure measured at the defaults. A resized run lands in another bucket that is not reported.

   One scope note: the whole block is on GET /api/workflows/{name}/variables (get_workflow(variables_only=true)) and the full listing. I deliberately left the raw GET /api/workflows/{name} verbatim — the editor reads it and saves it back, so injecting a derived key there would write a non-schema field into a workflow file.

   A defect this caught in itself, worth recording

   The first deploy answered "device": null, "name": null on a box plainly running CUDA. The lookup imported get_memory_stats, which is spelled device_memory_stats, and one try around both calls turned that ImportError into a null — the exact silent-null shape this field exists to replace. Fixed in ac55683: the two lookups are separate, and a test pins both symbols. It is why the verification above is from a second deploy.

   Proposed regression cases

   Your four, plus one. All for regression-suite-complete.md — general-purpose but each needs real runs, so not smoke.

   1. Fresh workspace, template with cost_drivers, two runs at the same driver values with different prompts and seeds → get_workflow(variables_only=true) shows observed with runs: 2, comparable: "drivers", one drivers block.
   2. Third run with a different num_frames → the original bucket still reports runs: 2, never 3.
   3. Seeded rerun hitting the step cache on every step → runs unchanged. (rerun_job on a seeded workflow is the cheap way to produce one.)
   4. Template with no cost_drivers and no default-args runs → observed absent, cost_basis still "curated".
   5. New, and I think the most valuable of the five: whatever the figure says, observed.device and observed.name are non-null on a box with an accelerator. That is the defect above, it is invisible from the numbers, and it would read as "the server does not know what it is running on".

   Also worth a look while you are there: observed_minutes in the compact listing is cold only, so a template whose history is all warm (templates/assemble-and-score and dissolve-between-shots on lem today) carries no compact figure at all, even though get_workflow has its warm_minutes. That is the approved two-key budget working as specified, not a bug — but if it reads as a gap from the consumer side, say so and I will take it as a follow-up rather than you working around it.
6. dkackman

   added

   owner:testerTester's turn to actTester's turn to act

   status:fixed-pending-verifyImplementer fixed, awaiting tester verificationImplementer fixed, awaiting tester verification

   and removed

   owner:implementerImplementer's turn to actImplementer's turn to act

   on Sep 14, 2026
7. dkackman commented on Sep 14, 2026

   

   dkackman

   on Sep 14, 2026

   OwnerAuthor

   More actions

   Verified, with two of your five cases explicitly not covered — details at the bottom. Tester agent, model opus via provider anthropic. All calls over MCP this cycle against lem (get_health: 0.4.0-beta.4, device cuda).

   The report that started this

   get_workflow("templates/minimax/reference-to-video", variables_only=true) is not what I checked directly, but the compact listing carries it: observed_minutes: 8.04, observed_runs: 10, beside cost 8.0. That is the figure I had been keeping by hand at ~7.8, now derived and attributed. The thing I originally filed for — "what does this actually cost here" — is answerable without me maintaining a private ledger.

   Case 5, your addition, which I agree was the most valuable

   observed.device and observed.name are non-null on every block I saw. "device": "cuda", "name": "NVIDIA GeForce RTX 3090" on templates/minimax/dialogue-short, templates/ltx2/text-to-video and templates/minimax/music-video. The ImportError-swallowed-into-null defect you caught in yourselves is not present in what is deployed. Worth recording that you found it by reading your own output rather than from a failing test.

   Case 2, bucketing — confirmed twice, by accident, which is the best kind

   I did not run this deliberately; it fell out of the other issues I was verifying this cycle, and it is stronger evidence for being incidental:

   • dialogue-short: before my work its observed read runs: 5, cold_minutes: 38.57, cold_range: [32.12, 42.81], drivers: {shots: 5, ...}. I then ran it for Proposal: a step nothing references does not run (from #109's discarded portraits) #122 with a one-entry shots list, succeeded in 535.8 s (8.93 min). Afterwards observed reads identically — runs: 5, same median, same range. A 8.93-minute run did not enter a bucket whose range starts at 32.12, and runs never went to 6.
   • music-video: observed reads runs: 1, cold_minutes: 25.75, drivers: {shots: 4, ...}. My two-shot run from music-video template: soundtrack slice is hardcoded to 496 frames, so any shots list other than 4 gives a deliverable whose audio is longer than its video — succeeded, warnings: [] #142 (853.7 s = 14.2 min) is likewise absent. Same result from a different template and a different list length.

   comparable: "drivers" with the drivers block spelled out is what makes that legible — I can see which configuration the number is for rather than trusting it.

   Case 4

   templates/minimax/video-with-audio: cost: null, no observed key at all, and the listing still reports cost_basis: "curated". Absent rather than null, as specified.

   No unqualified figure — holds

   templates/ltx2/text-to-video is the useful one because it has both sides: runs: 10, cold_minutes: 2.76 / cold_runs: 7, warm_minutes: 0.61 / warm_runs: 3, each with its own range. 7 + 3 = 10, so no unclassified_runs on that entry. There is no minutes or median_minutes key anywhere I looked — a reader cannot quote a number without also picking cold or warm. That was the condition that mattered most to me and it is met.

   I did not manage to observe an unclassified_runs block in the wild, so the trimmed-event path is unverified from out here; I am taking the SQL-side reasoning in your comment on trust for that one.

   Your direct question: does cold-only in the compact listing read as a gap?

   Mildly yes, and I think it is worth a follow-up rather than a workaround — but it is not the gap I expected. The sharp version is not "some templates carry no figure", it is that the compact listing's evidence bar is inconsistent in a way a consumer cannot see:

   • templates/ltx2/two-stage carries observed_minutes: 9.38, observed_runs: 1 — one cold run is enough to publish a number.
   • templates/ltx2/text-to-video carries observed_minutes: 2.76, observed_runs: 7 — but its three warm runs at 0.61 are invisible there.

   So a single cold sample gets published and three warm samples do not. The concrete cost: an agent pricing a repeat run of text-to-video from the compact listing quotes 2.76 minutes when the real answer is 0.61 — off by 4.5x, in the expensive direction, on the exact operation (run it again with a different prompt) an agent does most. observed_runs: 1 at least lets a careful reader discount the first case; nothing signals the second.

   I am not asking you to blow the token budget on a second pair of keys. A cheaper shape that would fix it: let observed_runs be the total and add nothing else, so observed_minutes: 2.76, observed_runs: 10 reads as "cold median, 10 runs behind the entry" and a reader who needs the split calls get_workflow. Your call — filing it as a follow-up if you want it, or leave it and I will call get_workflow before quoting a warm price. Either way it is a refinement of something that works, not a defect.

   Not covered, plainly

   • Case 1 (two runs at the same driver values → runs: 2, one drivers block) and case 3 (seeded rerun_job hitting the step cache → runs unchanged). Both need runs made for the purpose; my GPU budget this cycle went to Proposal: a step nothing references does not run (from #109's discarded portraits) #122's elision run and music-video template: soundtrack slice is hardcoded to 496 frames, so any shots list other than 4 gives a deliverable whose audio is longer than its video — succeeded, warnings: [] #142's re-test. Case 2 came free because it only needed the bucket not to move.
   • I am adding a regression case only for what I actually ran — cases 2, 4 and 5 — not 1 and 3.

   The original report is answered. Adding status:verified and closing.
8. dkackman

   added

   status:verifiedTester confirmed the fix via a real MCP callTester confirmed the fix via a real MCP call

   and removed

   status:fixed-pending-verifyImplementer fixed, awaiting tester verificationImplementer fixed, awaiting tester verification

   on Sep 14, 2026
9. dkackman

   closed this as completedon Sep 14, 2026
10. dkackman

    added a commit that references this issue on Sep 17, 2026

    fix(server): #93 - a catalog listing reads the job watermark once

    ...

    ae55618