Repository navigation
Deterministic N→1 selection between a fan-out and an expensive stage (select reducer + judge scorer) #119
Description
tool/endpoint: the workflow language — a new select reducer task and a judge scorer task; list_tasks, validate_workflow, run_workflow; the manifest / get_job
repro: the README's headline shape, 4 stills → choose one → video, when the choice is a rule rather than a judgment:
still (for_each over 4 seeds)
judge (for_each over the same list: a number per still against a rubric)
pick (select argmax(scores))
video (LTX-2.5 image-to-video on previous_result:pick)
expected: expressible as one workflow. The rule that made the choice is recorded in the manifest (selected: position, member name, score) so the recipe replays; on a seeded run, changing only the rule reuses the cached stills and scores and regenerates only pick and the video.
actual: two jobs with the agent in between: job A fans out, the agent picks, keep_output, job B on asset:name. That is the right boundary for a subjective choice and stays. For a deterministic one it is a round trip that records the rule nowhere. Two gaps behind it: no task reduces a list to one member by a rule, and no task emits a number per candidate for a rule to read. (A third, found while writing this up: no reference hands a task all of one step's artifacts — every previous_result: iterates — so gather: over a for_each group is the only whole-list reference. The proposal builds on that rather than adding a reference kind.)
Proposal: docs/proposals/score-and-select.md — select with a closed rule set (argmax, argmin, first_above, first_below, index), judge as the first scorer reusing the image_to_text VLM loader, one catalog template, selected in the manifest. Explicit non-goals: when/skipped steps, retry, semantic selection.
Parked with owner:don + status:needs-approval per the protocol (engine/syntax change). Approval is flipping the owner label to owner:implementer. Sibling preflight ticket: #118.
Raised by an external reviewer's forum reply.
Activity
docs: score-and-select proposal (#119) - deterministic N->1 reduction…
dkackman commentedon Sep 17, 2026
Approved as written. The proposal's own reasoning holds up: keeping this to a reducer rather than general `when`/branching avoids the whole class of "what does previous_result: mean for a step that didn't run" questions, the caching story (change only the rule, reuse cached candidates/scores) follows directly from the existing `(workflow id, step name)` cache key, and it stays clear of #118's scope (`select`/`judge` are task commands with their own argument contract, not new step-level keys, so the now-closed `$defs/step` object is unaffected).
Settling the one open question: `judge` is its own command, not an `image_to_text` mode. `image_to_text` returns text; a `"parse": "number"` mode would make its return type depend on an argument rather than the command's identity, which breaks the one-job-per-task convention `task.py`'s introspection-based checking assumes throughout the rest of the surface. A separate `judge` command sharing the VLM loader keeps `list_tasks`'s listing honest about what each command does.
No relation to #193/#200/#207 (the audio evaluate/correct track) - different evaluation mechanism (in-engine deterministic selection vs. agent-driven post-production over MCP), no shared code or ordering dependency. Approving this doesn't block or get blocked by that work.
Moving to owner:implementer.
dkackman commentedon Sep 18, 2026
triage: work — approved proposal (
docs/proposals/score-and-select.md, adade32) with the one open question settled in the approval comment (judgeis its own command sharingimage_to_text's VLM loader, not a mode). Nothing implemented yet ondevelop(noselect/judgeindw/tasks/), no duplicate found (#118, the sibling, is closed and unaffected since these are task commands, not step keys). Shares no file with #203-#206, so it runs alone: new tasks +task_domainsentries, one catalog template,selectedin the manifest, tests pinning the rule set. Largest item in this cycle — expect a full session.— triage by the implementer agent running as model
opusvia theanthropicprovider.feat: #119 - select reducer, judge scorer, selected manifest field
Merge fix/119-score-and-select into develop
dkackman commentedon Sep 18, 2026
Implemented per the approved proposal (docs/proposals/score-and-select.md), on
fix/119-score-and-select, merged todevelop(c29e2f1) and deployed to lem.Shipped:
selectreducer task command (dw/tasks/select.py): rulesargmax/argmin/first_above/first_below/index, threshold/index arguments as the proposal specifies.judgescorer task command (dw/tasks/judge.py), kept as its own command per the owner's approval comment settling the open question (not animage_to_textmode, since a"parse": "number"mode would make the return type depend on an argument, breakingtask.py's one-job-per-task introspection convention).selectedmanifest /step_endfield.select's arguments (dw/select_validation.py, wired intoWorkflow.validation_errors): unknownrule, missing/misplacedthreshold, missing/misplacedindex, andcandidates/scoresgather-consistency, all reported at the JSON path before the queue rather than surfacing asselect's own run-timeValueErrorafter an expensive fan-out has already generated.Full test suite green (5047 passed, 16 skipped), including 12 new
select_validationtests (TDD, RED verified before implementation).Scope deferred to follow-up (explicit judgment call under this session's budget, model
sonnetvia provideranthropic): the catalog template demonstratingselect/judgetogether (templates/flux/best-of-n-to-videoor similarly named), and the template-level tests from the proposal's Testing section (manifestselected-field check, step-cache reuse when onlyrulechanges on a seeded rerun). The engine-level pieces (select, judge, selected field, validation) are complete and independently tested; only the catalog demonstration and its template-level tests remain.Suggested regression case (model
sonnetvia provideranthropic): a workflow with afor_eachfan-out intojudgescores followed by aselectstep using rulefirst_abovewith nothreshold— should failPOST /api/validateatsteps[N].task.arguments.thresholdbefore any job is queued, not at run time after the fan-out has generated.dkackman commentedon Sep 18, 2026
Verified — tester agent, model
opusvia provideranthropic, all over MCP againstdw 0.4.0-beta.6onlem(cuda), workspaceqa-verify-119.What I ran
get_task("select")/get_task("judge")— both listed with the documented signatures (select: candidates/scores/rule/threshold/index/device,accepts_kwargs: false; judge: image/rubric/scale/device/model_name/max_new_tokens).qa119-score-and-select— the issue's one-workflow shape:still(ZImagePipeline,for_each: variable:candidates, 4 entries, seed 7) →judge(for_eachover the same list,image: previous_result:still, rubric "how red…", scale [0,10], device cuda) →pick(select,candidates: gather:still,scores: gather:judge,rule: variable:rule) →edges(cannyonprevious_result:pick).4011ed6e1007(rule argmax) — succeeded. Manifest:pickinfinalwith"selected": {"position": 1, "score": 3.0, "entry": "still@green"};edgesconsumedprevious_result:pickand produced one file. So the rule's choice is recorded in the manifest and the winner feeds a downstream step, asexpected:asks.7d3e270b148a—run_workflow(workflow_path="qa119-score-and-select", arguments={"rule":"argmin"}), no other change — succeeded in 8 s. All fourstill@*and all fourjudge@*carryreused: true(from run20260918-011656-e4d67e64); onlypickandedgesregenerated, with"selected": {"position": 0, "score": 0.0, "entry": "still@red"}. That is the "changing only the rule reuses the cached stills and scores and regenerates onlypickand the video" clause, confirmed (with canny standing in for the video step to keep it cheap).validate_workflowprobes (the implementer's proposed regression case, plus neighbours):first_above, nothreshold→valid: false, pathsteps[2].task.arguments.threshold, "select rule 'first_above' requires a threshold." ✅ (adding this to the smoke suite as S-F046)best→valid: false,steps[2].task.arguments.rule, "select: unknown rule: 'best'." ✅threshold/indexsupplied under ruleargmax→ refused as "only meaningful for rule 'first_above'/'first_below'". ✅ (but see gap 3)Gaps found — not blocking the verify, filed as separate issues referencing this one
dw/select_validation.pychecks thatcandidatesandscoresgather from steps over the same list. It doesn't:candidates: gather:still(over a 2-entry list) withscores: gather:judge(over a different 1-entry list) →valid: truewith a plan, only a warning about an unread field.judgestep aresult: {content_type: "text/plain"}crashes the job after the whole fan-out has been generated: job7f238fbcc108failed withwrite() argument must be str, not float(dw/result.pysave_artifact). The float score needs either a documented "noresulton judge" rule or str-coercion.threshold: "variable:threshold"with the variable defaulting to null is treated as "threshold present" and refused under argmax, so one stored workflow can't be driven tofirst_abovepurely througharguments— you have to edit the step.Scoring-quality note, not a protocol matter: the VLM judged the green pear the "most red" at 3.0/10 and the red apple 0.0. The engine did exactly what the scores told it to.
Closing as completed.
fix: #213 - select_errors catches gather: candidates/scores size mism…
Develop (#241)