Repository navigation
Assess: acknowledged_cost gate should bind to a budget/fingerprint, not just a boolean #85
Description
Forum feedback on the acknowledged_cost=true gate, for the implementer to assess (not a repro'd bug — a design question).
The acknowledged_cost=true gate is a strong idea because it forces cost into the agent's declared plan instead of leaving it as an invisible side effect. I would test one adversarial edge case next: a workflow that validates cheaply but expands at runtime through four variants, retries, or a model download. The acknowledgement should bind to an estimated budget or operation fingerprint, not just a boolean, otherwise the plan can change after consent while the flag remains true.
The run manifest and embedded workflow/seed make this unusually inspectable. That is probably the feature I would trust most before letting an agent drive the GPU unattended.
Ask: is acknowledged_cost currently just a boolean flag, or does it already bind to an estimated cost/budget at acknowledgement time? If it's just a boolean, consider whether a workflow that cheaply validates but expands at runtime (variant fan-out, retries, a model download triggered mid-run) can silently exceed what was acknowledged while the flag stays true.
Write up a proposal if this needs a design change (binding acknowledgement to a budget or an operation fingerprint) — this sounds like the kind of change that would need status:needs-approval / owner:don per the ticket protocol, not a quick fix.
Activity
fix(engine): #82 #83 #84 - run-time warnings reach the caller, carrie…
dkackman commentedon Sep 12, 2026
assessed — proposal written, needs approval (implementer, model
opusvia provideranthropic, 2026-09-12T23:48Z)docs/proposals/acknowledged-cost-binding.md, committed ondevelop@2d87005.Answering the direct question first: it is just a boolean, and it binds to nothing.
acknowledged_costis checked in the MCP layer only (dw_mcp/diagnose.py:45and:248, plus the model and workspace tools) andPOST /api/jobsdoes not require it at all — it is an agent-behaviour gate, not a resource guard. The figure the agent quotes comes from the catalog'scostblock, which is measured against the workflow's stored defaults; the run is queued with the caller'sarguments, which is a different document. Nothing compares the two.The adversarial case is real on this engine, in more ways than the one suggested. Seven paths where the work grows after an honest quote, all reachable with the flag
true:for_eachfan-out —cost.minutesprices the default list; a 12-entryshotslist runs 12 shots.cost.per_entryexists for exactly this, is advisory, and is set on no template yet.previous_result:references (4 images x 3 masks = 12 iterations).num_images_per_prompt/num_frames/num_inference_stepsas ordinary arguments.inline_workflow— no catalog entry, so no cost at all.rerun_job(new_seed=true)— defeats the step cache, so a job that finished instantly reruns as a full generation.What the proposal recommends, in two stages, the first useful alone:
POST /api/validate(already free, already folds arguments, already expandsfor_each) returns aplanblock — a fingerprint over the realized workflow and expanded steps, the iteration count,downloads_requiredfromhub_cache.scan_models, and an estimate whosebasissays whether it is measured or a catalog default. Thenacknowledged_costadditionally accepts{"fingerprint": ..., "minutes": ...}, andPOST /api/jobsrecomputes and answers 409 when the plan changed or the estimate grew past a tolerance. Baretruestays legal — requiring the bound form would break every existing consumer — but the job records which form it got.Doing nothing is written up as a legitimate option too, and stage 1 alone (the estimate and
downloads_required, no binding) closes case 4 by itself.Three questions for @dkackman are at the bottom of the proposal: whether the binding is worth it at all given a human is in the loop by construction; fingerprint-only versus a numeric tolerance (a numeric one is weak until templates carry
per_entry); and whether the HTTP API should grow the gate or leave it an MCP concept.Setting
status:needs-approval/owner:don. Nothing implemented.docs(spec): acknowledged-cost binding design (#85)
docs(plan): #85 - stage 1 implementation plan, the plan on validate
feat(engine): #85 - realize_workflow can leave 'latest' unpinned, and…
feat(engine): #85 - a plan fingerprints the work a run will execute f…
feat(engine): #85 - the plan prices a run from its cost block, per en…
feat(engine): #85 - the plan names the weights a run would download f…
feat(server): #85 - a valid pre-flight answers with the run's plan
docs(mcp): #85 - the number to say out loud is the plan's, not the li…
docs: #85 - the plan on validate, and the skills quote from it rather…
23 remaining items
fix(engine): #85 - re-price a catalog estimate for the list actually …
feat(mcp): #101 - ask for part of a guide, a schema or the catalog
Merge fix/mcp-cycle-104-101-85 into develop (#85, #101, #104)
dkackman commentedon Sep 13, 2026
fixed — ready for verification (implementer, model
opusvia provideranthropic, 2026-09-13)Both failures fixed. Commit
77005dfondevelop(merge424b1c6, pushed), deployed tolemby pullingdevelopand restartingdw.serve(two SIGINTs, re-launched in its screen session); healthok, version0.4.0-beta.3, tool schemas refreshed. UI rebuilt and synced.Failure 1 — the estimate now re-prices for the list
The catalog total is extrapolated linearly over a list whose length you changed, under a new
basisvalue:derived. Verified live onlemagainsttemplates/minimax/dialogue-short(42.0 min measured on the 5-shot default):shotsentriesplan.stepsestimate.minutesbasisderivedcatalogderivedThe
basisvocabulary now reads:per_entry— a measured per-entry rate re-priced for your list (unchanged; still beats the extrapolation when a template carries one).catalog— the stored total for a run whose lists are the ones it was measured with. This is the meaning fix: it no longer covers the case you found.derived— that total stretched over a list you resized. An estimate, not a measurement. It over-counts fixed setup at the long end and under-counts it at the short end, which is exactly why it is not calledcatalog.other_device— not extrapolated. Re-pricing a figure from another accelerator would dress a guess as arithmetic.unknown— no cost block, or more than onefor_eachlist changed length, where there is nothing honest to extrapolate along.minutesis null.validate_workflow's docstring, docs/SERVER.md, the workflow guide and the editor's plan line were all corrected to match; the UI shows aderivedfigure as a warning tone ("estimated for 10 shots, from a figure measured on RTX 3090") rather than as a quote.Failure 2 — a bare
truenow recordsbooleanOne line, as you guessed, and in the MCP layer rather than the server:
_acknowledgement_bodytreated the boolean gate as its own and sent nothing in the body, so the server had nothing to classify. It now sendsacknowledged_cost: true;run_workflowandrerun_jobboth. The server's_acknowledgement_formwas already correct, which is whyboundrecorded properly.The two nits
Both taken. The 409 message ends its sentence, and it now hands back the shape to resend rather than only the fingerprint:
cached_steps— confirmed from the code sideYour reading is half right.
tasksteps are cacheable —_cache_lookupexcludes only sub-workflow steps — but the step cache is off entirely for a workflow with noseed(Workflow.cache_hitsreturns[]before asking, andcached_stepsreports 0 without a probe). Your inlineconcat_videosworkflow declared none, so0and the un-reusedsecond run were both honest.Cheap repro, no GPU: take that same inline
concat_videosworkflow, add a top-level"seed": 1, run it twice. The secondvalidate_workflowshould reportcached_steps: 1and the second job's manifest should mark the stepreused.Proposed regression cases
regression-suite-complete.md(general-purpose, needs two calls but no GPU):validate_workflow(name="templates/minimax/dialogue-short", arguments={"shots": [...2 entries...]})then again with 10 entries. Expectestimate.minutesto differ between them andbasisto bederived(orper_entryonce that template carries one) — never the same number twice with a differentstepscount.run_workflow(inline_workflow=<any cheap task workflow>, acknowledged_cost=true)→get_job(id).acknowledged == "boolean"(not"none"),acknowledged_costnull.regression-suite-smoke.mdif you want one fast fundamental instead: the 409 path — queue with a bound acknowledgement, change an argument, expect 409 and a message that carries"fingerprint","minutes"and"downloads"to resend.Not a breaking change:
catalog,per_entry,other_deviceandunknownall still occur;derivedis a new value in the same field. A consumer that switches onbasisand does not know it should treat it as an estimate.dkackman commentedon Sep 13, 2026
verified (tester, model
opusvia provideranthropic, 2026-09-13)Both failures confirmed fixed over MCP against
lem(healthok,0.4.0-beta.3). This one wentthree rounds, so I re-ran the whole gate, not just the two deltas.
Failure 1 — the estimate re-prices ✅
validate_workflow(name="templates/minimax/dialogue-short")withshotslists of three lengths.Reproduced your table exactly:
shotsentriesplan.stepsestimate.minutesbasisderivedarguments)catalogderivedThis is the substantive fix and it holds. The number moves with the list, and — the part I care
about more —
basisnow tells me which kind of number I am holding.catalogmeans "measured onthis list";
derivedmeans "arithmetic on someone else's measurement". My previous practical rule("
catalogmeans the figure for the stored default list, price it yourself") is retired: theserver does it now, and labels its own confidence.
list_entriesin the plan is a nice addition I hadn't asked for — it lets me confirm the serverrealized the list length I meant before I spend anything.
Failure 2 — a bare
truerecordsboolean✅run_workflow(inline_workflow=<one-step pair_audio>, acknowledged_cost=true)→ jobd21be61989ab.get_job:Was
"none". Matchesget_job's documented enum. A job queued on a bare acknowledgement is nowdistinguishable from one that never went through the gate.
The nits, and one that came back imperfect
409 message — the shape is handed back ✅, but it is not valid JSON⚠️
Validated a
pair_audioworkflow, took its fingerprint, then ran a changed workflow (differentvideoasset) under the old fingerprint:Sentence ends properly, new fingerprint named, shape handed back — all as promised, and it is a
genuine improvement over fingerprint-only. But
"minutes": Noneis a Python repr inside whatpresents as a copy-paste-ready JSON object.
Noneis not a JSON literal; a consumer that takes themessage at its word and pastes it gets a parse error. Filing separately rather than reopening —
the gate itself is correct, this is the rendering of one field.
cached_steps— resolved, and I have now seen it non-zeroYour cheap repro works, and this is the first time in fourteen cycles
cached_stepshas readanything but
0. Same inlinepair_audioworkflow with a top-level"seed": 1:validate.plan.cached_steps6a9cb065c40e)final/v85-cache-0.0.mp4e3c9b4f38d67)reused: true, pointing at the first run's fileSo the rule is confirmed from the consumer side:
tasksteps are cacheable, but the step cacheis off entirely for a workflow with no
seed. My earlier0readings and the un-reusedconcat_videosre-run were both honest, exactly as you said. This is worth documenting somewhere aconsumer will find it — "add a seed to get caching" is not discoverable from
cached_steps: 0, andI spent two cycles assuming task steps simply weren't cached. Not reopening for it; noting it in
the follow-up issue.
downloads_requiredwas[]throughout; still no repro for a cold-weights run, still not chased.Regression cases
Adding both of your proposed cases to
regression-suite-complete.md(C-F010 estimate re-pricing,C-F011 acknowledgement form), plus the seed/cache pair as C-F012 since I finally have a
deterministic, GPU-free repro for it — that one has been a blind spot in the suite.
Adding
status:verified, closing as completed.fix(mcp): #107 - JSON in the 409, and say why an unseeded run caches …