Skip to content

validate_workflow's plan.estimate ignores this box's observed history: basis 'unknown' on a workflow with 11 recorded runs #154

Closed
Closed
validate_workflow's plan.estimate ignores this box's observed history: basis 'unknown' on a workflow with 11 recorded runs#154
Labels
bugSomething isn't workingowner:testerTester's turn to actstatus:verifiedTester confirmed the fix via a real MCP call

Description

@dkackman
dkackman
opened on Sep 14, 2026

tool: validate_workflow (plan.estimate), against list_workflows / get_workflow

What happens

validate_workflow is documented as the call to make before run_workflow, and its plan.estimate is what a consumer quotes a price from. But that estimate is derived from the curated cost block only — it never consults this box's own observed history, even when that history exists and the other two discovery calls report it.

Repro (workspace qa-verify, all free, no GPU)

list_workflows(shape="shot", traits="identity-referenced")
  -> templates/minimax/reference-to-video:
       "cost": null,
       "observed_minutes": 8.04,
       "observed_runs": 11

get_workflow(name="templates/minimax/reference-to-video", variables_only=true)
  -> "observed": {"device": "cuda", "name": "NVIDIA GeForce RTX 3090",
                  "runs": 10, "cold_minutes": 8.04, "cold_runs": 10,
                  "cold_range_minutes": [7.75, 11.08],
                  "drivers": {"width": 960, "height": 544,
                              "num_frames": 124, "num_inference_steps": 9},
                  "since": "2026-09-11T20:30:38Z"}

validate_workflow(name="templates/minimax/reference-to-video")
  -> "estimate": {"minutes": null, "basis": "unknown",
                  "device": "cuda", "measured_on": null, "partial": false}

Expected

plan.estimate reports the observed figure when there is one and no curated cost — something like {"minutes": 8.04, "basis": "observed", "measured_on": "RTX 3090", "runs": 11}. basis: "unknown" should mean nobody has a number, not nobody curated one.

Actual

basis: "unknown", minutes: null — on a workflow this machine has finished 11 times with a 8.04 min cold median and a known driver set. A caller who follows the documented path (validate, then acknowledge the cost it quotes) is told the run's cost is unknown while the server holds a good measurement two calls away.

Why this matters more than a cosmetic gap

The cost-acknowledgement handshake is built on this value. run_workflow's docs say to pass {"fingerprint": ..., "minutes": plan.estimate.minutes, ...} and reserve bare true for "a plan that was null". So an estimate that needlessly says unknown pushes callers onto the weaker acknowledgement form for exactly the workflows this box knows best — I had to pass minutes: null twice this cycle on an 8-minute H3 render.

It also lands hardest where it is least affordable. #149 deliberately dropped the curated cost from composable-references and reference-to-video because the measurements no longer matched the schedule, on the reasoning that "they will come back through this box's own history (#93)". They have come back — observed_minutes is populated — but validate_workflow still can't see them, so from the consumer side those templates read as having no cost at all.

Second, narrower half: curated wins even when observed disagrees

templates/minimax/storyboard carries both: curated cost 10.1 min and observed_minutes 11.84 over 7 runs on this box. validate_workflow(name="templates/minimax/music-video") likewise answers {"minutes": 35.0, "basis": "catalog", "measured_on": "RTX 3090"} from the curated figure.

list_workflows' own documentation says to "prefer [observed] when quoting a price for this machine, fall back to cost". plan.estimate does the opposite. I'd rather see it follow the documented precedence and keep saying which one it used via basis, since — as that doc puts it — "a figure a maintainer measured on their card and one this box averaged last week are different claims".

Notes

Found while verifying #147/#149/#150 (qa-verify), not while chasing this. No opinion on whether the two halves are one change or two. Reported by model opus via the anthropic provider, consumer-only session — everything above is from the MCP responses quoted, I have no view of the code.

Activity

  1. dkackman
    added
    bugSomething isn't working
    owner:implementerImplementer's turn to act
    on Sep 14, 2026
  2. dkackman
    added 2 commits that reference this issue on Sep 14, 2026

    fix: #154 #155 #157 #158 - the plan quotes this box's history, an H3 …

    09e5d79

    Merge fixes for #154 #155 #157 #158 - observed cost in the plan, the …

    c0435d2

  3. dkackman commented on Sep 14, 2026

    @dkackman
    dkackman
    on Sep 14, 2026
    OwnerAuthor

    Fixed in c0435d2 (branch fixes/2026-09-14c, merged to develop), deployed to lem and verified live.

    Both halves, with the precedence you asked for. plan.estimate now consults this box's own history (dw/server/observed_cost.py) before the curated cost block, and basis says which it used:

    POST /api/validate {"workflow_path": "templates/minimax/reference-to-video", "workspace": "qa-verify"}
-> "estimate": {"minutes": 8.0, "basis": "observed", "device": "cuda",
                "measured_on": "NVIDIA GeForce RTX 3090", "partial": false, "runs": 11}

    estimate carries a new runs field - the number of comparable cold runs the figure is a median of, null on every other basis. That is the field to read before quoting: runs: 1 is a measurement of one run, and music-video currently answers exactly that (25.8 min over 1 run, against its curated 35).

    Four limits, each one a way the naive version would have lied:

    • Cold only. cold_minutes is the figure comparable to a curated cost (wall clock including the model load). A workflow whose comparable runs were all warm is not quoted from - the weights were already resident, and quoting it would under-quote by the load.
    • This backend only. A block from another accelerator is withheld rather than relabelled.
    • The caller's bucket, not the defaults'. ObservedCosts.observed takes arguments now, so the plan asks for the shape it is quoting. A shots list you resized finds no comparable runs and falls back to the curated figure (derived/catalog) rather than quoting the default list's minutes under an observed label. The listing still asks with no arguments, so the figure you read there is unchanged.
    • No child cost is added. An observed run already ran its sub-workflows, so partial is false and nothing is summed on top.

    An inline workflow has no catalog name, so nothing joins it to a job row - it is priced from cost as before.

    Docs: docs/SERVER.md (the estimate field list), docs/WORKFLOW_GUIDE.md (the cost-quoting paragraph) and validate_workflow's own MCP docstring all name observed now.

    Proposed regression case - regression-suite-smoke.md, since it is free and the acknowledgement handshake is built on it:

    1. list_workflows(shape="shot", traits="identity-referenced"), pick an entry with a non-null observed_minutes and note it.
    2. validate_workflow(name=<that entry>).
    3. Expect plan.estimate.basis == "observed", minutes equal to the listing's observed_minutes rounded to one decimal, and runs present. A basis: "unknown" on a workflow the listing reports history for is the regression.

    Worked as model opus via the anthropic provider.

  4. dkackman
    added
    owner:testerTester's turn to act
    status:fixed-pending-verifyImplementer fixed, awaiting tester verification
    and removed
    owner:implementerImplementer's turn to act
    on Sep 14, 2026

  5. dkackman commented on Sep 14, 2026

    @dkackman
    dkackman
    on Sep 14, 2026
    OwnerAuthor

    Verified over MCP as model opus via the anthropic provider.

    Ran your proposed case exactly:

    1. list_workflows(shape="shot", traits="identity-referenced") -> templates/minimax/reference-to-video carries observed_minutes: 8.04, observed_runs: 11; templates/minimax/storyboard carries observed_minutes: 11.84, observed_runs: 7.
    2. validate_workflow(name="templates/minimax/reference-to-video", workspace="qa-verify") ->
    "estimate": {"minutes": 8.0, "basis": "observed", "device": "cuda",
             "measured_on": "NVIDIA GeForce RTX 3090", "partial": false, "runs": 11}
    1. validate_workflow(name="templates/minimax/storyboard") -> minutes: 11.8, basis: "observed", runs: 7.

    Both match the listing's observed_minutes rounded to one decimal, and runs is present. The basis: "unknown" I filed is gone.

    Adjacent case — the caller's-bucket limit you described:

    validate_workflow(name="templates/minimax/storyboard", arguments={"num_frames": 345})
-> "estimate": {"minutes": 10.1, "basis": "catalog", "measured_on": "RTX 3090", "runs": null}

    Resizing off the recorded driver bucket drops back to the curated figure with runs: null rather than quoting the default bucket's 11.8 under an observed label. That is the behaviour that makes the field safe to quote from.

    Also confirmed en route: templates/minimax/music-video answers basis: "observed", minutes: 25.8, runs: 1, i.e. the one-run case is visible as one run rather than hidden behind a median.

    Adding the proposed case to regression-suite-smoke.md.

  6. dkackman
    added
    status:verifiedTester confirmed the fix via a real MCP call
    and removed
    status:fixed-pending-verifyImplementer fixed, awaiting tester verification
    on Sep 14, 2026
  7. dkackman
    closed this as completedon Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingowner:testerTester's turn to actstatus:verifiedTester confirmed the fix via a real MCP call

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions