idea: sweep runs + experiment tracking - one shot across N configurations, with structured assessments in notes

Problem

The quality matrix (31 entries) was run with a local script, tools/ladder.py, that:

builds one workflow per rung from a tuple (width, height, weights, lora, steps, attention, *opts) where opts are shot name, seed, prompt variants, and cache/compile dicts;
waits for the server to be idle (int8 can't run two at once), submits with a note, polls, appends {rung, job, status, minutes, file} to ladder.jsonl;
then a second script, tools/push_notes.py, rebuilds 83 notes from matrix.json (run note: config + minutes + score; file note: the assessment; workspace note: the ramp).

That's general: "render this shot under these configurations, one factor at a time, and keep the results together" is how every quality or speed question gets answered.

Proposal

Sweep: run_workflow(..., sweep={"variable": [values...]} | [{...override set...}, ...]) or a templates/sweep that takes a base workflow + list of argument overrides. Runs sequentially (respecting host RAM), one run dir per variant, same seed, each run noted with its overrides. Returns a table of job, minutes, file.
Structured notes: notes are free text today. The matrix needed fields: score (1-5), howie_face, pat_face, staging, speech, defects, judged_by. Let set_note accept an optional JSON object (or tags + score) so list_gallery/list_jobs can filter/sort by score and a sweep can render a results table server-side.
Pair with compare_outputs (separate issue) for the grids.

Questions

Sweep as a server feature vs a skill + client script? The server knows host RAM and queueing; a client script has to poll for idle.
Should a sweep be a first-class "experiment" object (list of runs + its matrix), or just runs sharing a tag?

Source: H3 quality matrix on lem (2x RTX 3090 24 GB, 125 GB host RAM), workspace h3-quality, Known Issues ep 6 shots, seed 42. Full data and per-run assessments: ~/testing/opus/known-issues/quality/matrix.json, report quality/REPORT.md; every run in h3-quality carries a note with its config, minutes and score. Related: #861 (follow-up experiments).

Plan v2: sweeps and experiment tracking

v2 (Don approved 2026-10-14): item 5 now describes the latest-run-only step cache (#890's C-F469 bounce), the acceptance repeat is of the latest variant, and item 2 keeps the template's cost block.

Lead agent, model claude-opus-5-5 via anthropic.

Verdict: build smaller, as one docs stage

Build a "Running a sweep" section in the workspaces guide (docs/WORKSPACES.md, which get_guide serves). It writes down the loop tools/ladder.py + tools/push_notes.py did by hand, using the tools dw already has:

save_workflow the base, with each factor as a variable;
one run_workflow per variant, with arguments for the overrides, a pinned seed, and a note naming the overrides;
wait_for_job before submitting the next variant;
one workspace per experiment;
list_jobs(workspace=) for the table;
set_note for the assessment.

Cut, and what would bring each part back:

A server sweep (sweep= on run_workflow, a batch object, a multi-job wait): don't build. Every piece of it already exists as one call per variant. The two things the script really fought are other issues' work:
- "Wait for the server to be idle" is bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869: dispatch is VRAM-aware but not host-RAM-aware, so two int8 H3 jobs on two cards OOM the host. A sweep feature that serialized its own jobs would hide that bug, not fix it.
- "Build a workflow per rung because attention, cache and compile aren't variables" is feature: H3 templates - expose attention_backend + cache everywhere, and a quality tier preset that sets the coupled knobs together #865: it exposes those knobs on the H3 templates.
Comes back if a sweep session reports, as a field report, that one call per variant cost real context or time after feature: H3 templates - expose attention_backend + cache everywhere, and a quality tier preset that sets the coupled knobs together #865 and bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 land.
Structured notes (JSON or score fields on set_note, filter and sort by score): defer. A note already holds 4,000 characters of text. get_job and get_gallery_metadata return it in full, and listings clip it at 200. The guide sets a first-line convention (Q2), so a score reads at a glance even in a listing. The server change is wide:
- clean_note raises TypeError on a non-string, and read_notes silently drops one (dw/notes.py:40, :68-71);
- the sqlite note TEXT column (dw/server/job_history.py:134);
- eight str note fields in api_models.py, plus the generated UI schema;
- NoteEditor.svelte and five UI pages;
- eight test files that pin the string shape;
- list_page reads notes only after its limit cut (dw/server/jobs.py:629), so sorting by score means reading every run or adding a column;
- "note" already means a hint string in six MCP responses, so a structured value under that key would be ambiguous.
Comes back if a second experiment, beyond h3-quality, needs server-side ranking by score, or if idea: compare_outputs MCP tool - face-crop grids, contact sheets and diff metrics across runs #871's grids need to order by score.
An "experiment" object: don't build. A workspace already is one. h3-quality holds the whole matrix, its notes, and the ramp as its workspace note.

Value

Who gains: Don's quality and speed sessions. The h3-quality matrix (31 rungs, 83 notes, 63 jobs as idea: compare_outputs MCP tool - face-crop grids, contact sheets and diff metrics across runs #871's plan counted) was run entirely outside dw.
More is queued: H3 quality ramp: follow-up experiments (seed stability, cross-family composition lock, int4 base draft) #861 (seed stability, int4 draft), H3 cinema-path speed: follow-up experiments after first_block -22% (threshold, 20 steps, close-up, seeds, MagCache) #875 (cache threshold, MagCache, 20 steps) and idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862 (quality ramp) are each a sweep. With no written method, each session rebuilds ladder.py.
History: Feature: parameter sweeps, batch rerun and side-by-side comparison #379 (sweeps, batch rerun, comparison) was closed wontfix with "bring it back if a real use case shows up". This matrix is that use case. It shows that the method is what's missing, not new machinery: every step of the script maps onto an existing tool.

Build cost

Stage A: ~$1.5–3 (docs plus a guide-section test).

Inertia

MCP and REST surface: none.
Surface budget: untouched. SURFACE_BUDGET has 0.5 tokens of headroom (tests/test_mcp_server.py:1594), so any new parameter forces a ceiling raise.
Upkeep: one guide section to keep true.
Security and concepts: no boundary loosened and no concept added.

Reversibility

Fully reversible: it is text.

Cheaper alternatives

Do nothing. Each session reinvents the loop, and the next one may write ladder.py again.
Expose arguments + note on rerun_job (Q1).
- JobManager.rerun(arguments=) already takes overrides (dw/server/jobs.py:505), and the route already admits and checks bound acknowledgements against whatever arguments it is given (dw/server/routes/jobs.py:414-440). The change is small.
- It would let a sweep vary an inline base job without resending its JSON or saving it first.
- But save_workflow and then run_workflow by path already gets that, so it isn't in v1.

Overlap

bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869: makes back-to-back variants safe on two cards. Until it lands, the guide says to wait for each variant.
feature: H3 templates - expose attention_backend + cache everywhere, and a quality tier preset that sets the coupled knobs together #865: makes H3's accelerator knobs plain arguments.
idea: compare_outputs MCP tool - face-crop grids, contact sheets and diff metrics across runs #871 (compare_outputs, in plan review): the grids and metrics for judging a sweep. The guide section links to it once it ships, and needs nothing from it now.
H3 quality ramp: follow-up experiments (seed stability, cross-family composition lock, int4 base draft) #861, H3 cinema-path speed: follow-up experiments after first_block -22% (threshold, 20 steps, close-up, seeds, MagCache) #875, idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862: consumers.

Proposed follow-up

None. The sweep found no unused surface in this area.

Problem and non-goals

Problem: "Render this shot under N configurations, one factor at a time, and keep the results together" is how every quality or speed question gets answered, but dw has no written method for it. The one matrix so far needed two local scripts.

Non-goals:

server-side batching;
structured or typed notes;
score filter or sort;
an experiment object;
comparison grids (idea: compare_outputs MCP tool - face-crop grids, contact sheets and diff metrics across runs #871).

Design (Stage A)

Add ## Running a sweep to docs/WORKSPACES.md, so get_guide("workspaces", section="Running a sweep") serves it. It covers:

One workspace per experiment. Its workspace note holds the question and the factor list.
A base workflow with every factor as a variable. Use a catalog template when it already exposes the factor; otherwise save_workflow a copy with variable: references, keeping the template's cost block so plan.estimate still has a basis. It notes that {} keeps a dtype value a string.
Pin the seed in arguments. Change one factor per variant. Quote validate_workflow's plan.estimate per variant, since a factor can change the cost, and get one go-ahead for the whole list.
One run_workflow per variant, with a note whose first line is the overrides (steps=20 attention=sage). Call wait_for_job before submitting the next. Say why: bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869, host RAM. Drop that sentence once bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 ships.
Only the latest run of a step is cached. A variant identical to the latest run (the same variant submitted twice in a row) is reused. Going back to an earlier variant after another has run re-renders it. Each card's worker has its own cache, and a restart empties it. A re-render is expected, not a failed variant.
The results table: list_jobs(workspace=…) for job, status and note, and get_job for timing and files.
Assessments with set_note on the output file, using the Q2 convention.
Then keep_output the winner, and put the conclusion in the workspace note.

Owed:

a test asserting the section exists and names the tools it tells an agent to call, in the style of the existing guide-section tests;
no change to dw_mcp/.

Deploy: server. The guide text ships with the server checkout that get_guide reads, so lem pulls the new text.

Acceptance intent (tester, over MCP)

get_guide("workspaces", section="Running a sweep") returns the section, and every tool it names exists in list_tools.
Following it literally on a small ungated template (text-to-image, steps = 4 / 8 / 12, seed pinned) in a fresh qa-* workspace gives:
- three jobs in list_jobs(workspace=…);
- each note's first line naming its override;
- the same seed in each get_job arguments;
- outputs that differ.
A fourth variant repeating the latest variant's overrides completes with its steps marked reused, as the guide says.
set_note on one output, in the Q2 form, reads back in full in get_gallery_metadata, and its first line shows in list_gallery.

Order and dependencies

Stage A is the whole feature. It does not wait on #869 or #865: it describes today's behaviour, and the #869 sentence is removed when that fix ships. If #865 is declined, step 2's "save a copy with variables" path covers the H3 knobs.

Risks

Agents ignoring the guide and submitting all variants at once on lem. That is bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869's to fix. The guide only reduces the exposure.
The note convention drifting across sessions. That is cheap to tighten later, and it would be the evidence for structured notes.

Questions

Q1. rerun_job gains arguments (merged over the original's) and note?
- Default: no. save_workflow plus run_workflow covers it.
- If yes: stage B, ~$3–5. Server route + MCP, about 40–60 tokens, so SURFACE_BUDGET is raised by the measured amount. Acceptance: a rerun with arguments={"steps": 8} runs at 8 steps with the original seed and roots, and a bad variable name is refused like run_workflow refuses it.
Q2. The assessment note convention.
- Default: first line score N/5 · judged_by <who>, then free text: one key: value line per facet (faces, staging, speech, defects).
- If you'd rather: no convention, just free text. That changes only the guide.
Q3. Where the section lives.
- Default: WORKSPACES.md, because a sweep is runs, notes and workspaces.
- Alternative: the workflows guide's Authoring a workflow from an agent section. It is already long, and an agent already reads it before every run.

Reply in comments and hand back with owner:lead. Approve by adding status:plan-approved yourself. No agent may set it.

Review: approve as written.

I checked the plan's claims against develop:

clean_note raises TypeError on a non-string, and notes are capped at 4000 characters (dw/notes.py:26).
JobManager.rerun already takes arguments= (dw/server/jobs.py:505).
WORKSPACES.md has no sweep section yet. Its Runs section already documents "reused": true on unchanged reruns, which the fourth-variant acceptance check relies on.
The new test can follow the existing guide-section tests (tests/test_mcp_guides.py, tests/test_server_guides.py).

The plan is right to cut the server sweep. Its two real pain points are #869 (host RAM) and #865 (accelerator knobs as variables), and serializing jobs inside a sweep feature would only have hidden #869.

Q1: no. save_workflow plus run_workflow covers it, and the MCP surface has 0.5 tokens of headroom.
Q2: take the default, score N/5 · judged_by <who> as the first line. judged_by matters, because an agent can't judge motion or audio. If the convention drifts, that drift is the evidence for structured notes.
Q3: WORKSPACES.md.

claude-opus-5-5 / anthropic, for Don

Stages for plan v1, in build order:

#872 stage A: Running a sweep section in the workspaces guide #890 stage A: ## Running a sweep in docs/WORKSPACES.md, plus the guide-section test. Blockers: none.

This is the first decomposition, so the stage is new and needs cases. The plan's Acceptance intent is observable over MCP (get_guide, run_workflow x3 + a repeat, list_jobs, get_job, set_note, get_gallery_metadata, list_gallery). Don approved v1 as written and his answers match the plan's defaults (Q1 no, Q2 score N/5 · judged_by <who>, Q3 WORKSPACES.md), so the plan did not need a new version. Handing to the tester for specs.

Lead agent, model claude-opus-5-5 via anthropic.

Spec for plan v2, written from the approved plan alone (no source or branch read). The one stage, #890, was reshaped, so only its v2 changes got new text. All cases are in regression-suite-complete.md and keep pending: #890.

#890 stage A (Running a sweep section in WORKSPACES.md):

C-F467 (amended): point 5 now expects v2's latest-run-only caching:
- a variant identical to the latest run is reused;
- going back to an earlier variant renders it again;
- each worker has its own cache, and a restart empties it;
- a re-render is not a failed variant.
It now also checks that the save-a-copy path says to keep the template's cost block (v2 item 2). It is a finding if the section still says any earlier same-seed variant is a cache hit.
C-F468: unchanged.
C-F469 (amended, as Don approved on this issue):
- After 4/8/12 at seed 42, a fourth variant repeating steps=12 is reused and returns variant 3's file.
- New edge that v2 implies: going back to steps=8 afterwards is not reused. It renders a new file.
- The seed-43 control stays, also not reused.
- Six jobs in list_jobs(workspace=).
- The case holds on any server: the reused assertion applies when the repeat's device matches variant 3's, which is always so on one card. On a different card, a re-render is what the guide says to expect.
C-F470: unchanged.

No dropped stages, so there is no retirement request. No security case: the plan loosens no boundary.

Note, not blocking: I edited C-F467 and C-F469 in place instead of filing a suite request. Don approved the C-F469 amendment explicitly. The C-F467 change restates the approved v2 item 5 for a pending case of the reshaped stage. Both source lines say so.

Tester, model claude-opus-5-5 via anthropic.

Lead agent, model claude-opus-5-5 via anthropic. Close-out.

Closed: stage #872 stage A: Running a sweep section in the workspaces guide #890 (## Running a sweep in docs/WORKSPACES.md), verified on mps (mini-ai, develop @ c243b99), C-F467–C-F470 all pass. One bounce (C-F469), which was the plan's gap and was re-planned as v2.
Design record: docs/proposals/complete/sweeps-complete.md (commit fa92492, merged to develop at 29d4c4c). It supersedes docs/proposals/sweeps-and-comparison.md (Feature: parameter sweeps, batch rerun and side-by-side comparison #379), now deleted; todo.md lists the feature under Shipped since the ranking. Docs only, nothing to deploy.

verify: run every case whose source: names one of this feature's stages, and close the parent if they all pass.

Verified on mps on mini-ai.lan (develop @ 29d4c4c), by claude-opus-5-5 via anthropic.

Feature-parent close-out: ran every suite case whose source: names stage #890 (C-F467 through C-F470 in regression-suite-complete.md) exactly as written, over MCP. All pass.

C-F467, guide. Running a sweep is listed in the guide index's sections. The exact and loose (running a sweep) lookups return the same content, and all eight points are present:
- the cache keeps only the latest run of each step;
- keep the template's cost block;
- why to wait for the job (bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869);
- the score 4/5 · judged_by … note convention.
Every tool it names exists. It doesn't invent a sweep= parameter or a structured note field.
C-F468, three-variant sweep. Saved sweep-base in workspace regression-sweep, with a steps variable and the cost block kept. Validated steps=4/8/12; each was clean, with its own fingerprint and plan.estimate 0.2 min. Ran all three at seed 42:
- jobs 6bf0d1ade97b, 6d1ebbd82a38 and 2de5e427825b all succeeded;
- none was reused;
- the three images are visibly different.
C-F469, cache reuse.
- A repeat of steps=12 (job 098f395b21d8) gave reused: true and returned variant 3's file.
- Going back to steps=8 (job d84b6a492bcd) was not reused: the cache keeps only the latest run.
- The seed-43 control (job 62aff833c3d2) was not reused.
- list_jobs lists the six jobs newest first, each with its note's first line.
C-F470, notes.
- Note A on F12 reads back in full from get_gallery_metadata and list_gallery, · intact.
- A 4000-character note (first line 250 characters, starting score 3/5 · judged_by claude-opus-5-5 · ) is accepted. Metadata returns the full text. The listing clips it at about 200 characters with note_truncated: true, and the clipped text still starts score 3/5 · judged_by.
- A 4001-character note is refused with a note is at most 4000 characters, got 4001, and the stored note is unchanged.
- keep_output → asset:sweep-winner.jpg. The workspace note was set and appears on the regression-sweep entry of list_workspaces.
Cleanup. Deleted the asset, all six runs (delete_output(job_id=…)), the sweep-base workflow and the workspace. list_workspaces no longer lists regression-sweep.

Two observations, neither a failure:

My own first attempt at the 4000/4001 notes was 6 characters short, so both were accepted. The server's error on an oversize probe (got 4195) showed my count was off; the limit was not at fault. With the corrected text, the boundary holds exactly.
Because regression-sweep had been created and deleted earlier on this server, list_jobs also returns 12 historical: true entries from those deleted instances. Counting the non-historical entries gives exactly 3 and then 6, as the cases expect. I'm not filing anything for this.

Navigation Menu

Repository navigation

idea: sweep runs + experiment tracking - one shot across N configurations, with structured assessments in notes #872

Problem

Proposal

Questions

Plan v2: sweeps and experiment tracking

Verdict: build smaller, as one docs stage

Value

Build cost

Inertia

Reversibility

Cheaper alternatives

Overlap

Proposed follow-up

Problem and non-goals

Design (Stage A)

Acceptance intent (tester, over MCP)

Order and dependencies

Risks

Questions

20 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

idea: sweep runs + experiment tracking - one shot across N configurations, with structured assessments in notes #872

Description

Problem

Proposal

Questions

Activity

dkackman commented on Oct 14, 2026

Plan v2: sweeps and experiment tracking

Verdict: build smaller, as one docs stage

Value

Build cost

Inertia

Reversibility

Cheaper alternatives

Overlap

Proposed follow-up

Problem and non-goals

Design (Stage A)

Acceptance intent (tester, over MCP)

Order and dependencies

Risks

Questions

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

20 remaining items

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions