Repository navigation
validate_workflow: overriding a declared cost_driver silently returns basis: catalog with the wrong (unmatched) figure #267
Description
templates/ltx2/text-to-video declares num_frames as a cost_driver. Overriding it from the curated 121 to 345 should invalidate the curated quote — and bucketing correctly finds no matching observed bucket for 345 — but validate_workflow then falls back to basis: "catalog", minutes: 1.8, runs: null: the curated figure measured for 121 frames, presented as if it still applied to 345.
Contrast: the identical override (121 → 345) on the H3 template instead produces basis: "unknown". Not a great answer either, but an honest one.
Why this is worse than unknown: catalog is documented as "a measured total for a run whose lists are the ones it was measured with" — which is exactly the case this isn't. It looks authoritative. Acting on it, I quoted "8 minutes" to a user from prompt-adjacent arithmetic instead of the tool's own (wrong) number, which is the only reason the wrong quote didn't ship further.
Ask: when a declared cost_driver is overridden away from the value the curated figure was measured against, and no observed bucket matches the new value, the plan should fall back to basis: "unknown" (as H3 already does), not silently keep serving the stale catalog figure.
Related: #255 (a different case of estimate.minutes not reflecting what the plan actually is — that one is cached_steps ignored, this one is a cost_driver override ignored).
Activity
dkackman commentedon Sep 21, 2026
triage: batch with #255, #268 — same
dw/plan.pyestimate code; #255's session does the work. Note for that session:_priceonly reprices a list-length change (_repriced); a scalarcost_drivermoved off the value the curated figure was measured at is not checked at all, which is why LTX2 answerscatalogwhile H3 (no curatedcostfor that shape) answersunknown. (triage by implementer agent, model opus via provider anthropic; lem is at develop head c5d4ac5, so nothing here is fixed-but-undeployed)fix(mcp): #255, #267, #268 - plan.estimate pricing fixes
dkackman commentedon Sep 21, 2026
Fixed. A declared `cost_driver` overridden away from its catalog-measured value now falls back to `basis: "unknown"` instead of quoting the stale catalog figure. Added `_scalar_driver_shifted()` in `dw/plan.py`, which extends the existing `_repriced()` check (previously only `for_each` list-length changes) to also catch a scalar driver variable overridden away from the value the catalog `cost` entry was measured at.
Shipped as part of a batch with #255 and #268 (all three are fixes to
dw/plan.py's pricing logic), one branch, one deploy.fix/255-267-268-plan-estimate, merged todevelopat commit859e43c(git logmessage: "fix(mcp): validate_workflow: plan.estimate.minutes ignores plan.cached_steps — 14.2 min quoted for a 285 s run with one shot cached, and still 14.2 with every step cached #255, validate_workflow: overriding a declared cost_driver silently returns basis: catalog with the wrong (unmatched) figure #267, validate_workflow: composing a template through a workflow step loses its measured cost basis #268 - plan.estimate pricing fixes")lem: pulleddevelop(fast-forward to859e43c), restarted thedw.serve --mcpprocess running in thedon-owned screen session there, confirmed healthy viaGET /api/healthbefore handing off.Suggested regression case (for the tester to add once verified): in
regression-suite-smoke.md, validate a workflow that declares a scalarcost_driversentry withargumentsoverriding that variable away from its catalog-measured default, and confirmplan.estimate.basis == "unknown"rather than the catalog minutes.dkackman commentedon Sep 21, 2026
Verified over MCP against lem (develop
859e43c).Ran
validate_workflow(name="templates/ltx2/text-to-video", arguments={"num_frames": 345})→plan.estimateis now{minutes: null, basis: "unknown", runs: null, measured_on: null}— the stale catalog 1.8 min figure is gone. Matchesexpected:.Adjacent cases:
basis: "observed", 2.5 min, 12 runs (pricing itself is intact).{"num_frames": 121, "prompt": "a quiet lake at dawn"}(driver explicitly set to its measured value plus a non-driver override) → stillbasis: "observed", 2.5 min — an override that doesn't move a driver off its measured value isn't punished.{"width": 1280, "height": 704}(the other two declared drivers shifted) →basis: "unknown"— the check generalises beyondnum_frames.Adding the suggested smoke regression case. Verified by tester agent, model opus via provider anthropic.
fix(plan): #341 - price a composed child by the arguments its step pa…