H3 quality ramp: follow-up experiments (seed stability, cross-family composition lock, int4 base draft)

Follow-up experiments from the H3 Ref2VA quality matrix (workspace h3-quality on lem, Known Issues ep 6 shots). Results and per-run assessments live in ~/testing/opus/known-issues/quality/matrix.json and as notes on every run in h3-quality.

What the matrix established (one seed per shot)

TV path: draft L0 (960x544, int4, ref2v 4-step, sage, 3.5 min) -> preview/final L7 (1344x768, int8, 4-step, 7.2 min); staging carries.
Cinema path: draft B3 (1344x768, int8, no LoRA, 12 steps, sage, ~13 min) -> final L5 (30 steps, ~23-32 min); staging carries (wide two-shot and close-up). Faces photographic (4/5) vs 2.5/5 for the old int4 turbo-8 baseline.
Sage is visually lossless (-23% at turbo-8, -33% at base-30).
Attention never moves composition. Quantization does not on turbo LoRAs but does on the base model (int4 vs int8). Base step count does not; 8 steps duplicated a character (12 is the floor).
Base model invents extra people unless every subject is placed explicitly in frame.

Follow-ups (untested)

Seed stability. Every rung is a single seed per shot. Re-run the carryover pairs (B3->L5, C1->C2, L0->L7) on 2-3 more seeds to confirm "same family = same staging" is not seed luck.
Composition lock across families. A turbo 4-step draft (3.5 min) cannot feed a base final - it re-stages. Candidates: FL2VA pinned to the draft's first/last frames (loses the voice/identity references - R3 showed that cost), or the draft as a guides clip on t2va/fl2va (guides are not on ref2va). Worth it only if a cheaper cinema draft than ~13 min is needed.
544 int4 base draft -> 768 int4 base final. Base staging appears to follow weights, not canvas (B4 at 544 int4 ~ B5 at 768 int4), so this should carry - which would give a 6.6 min cinema-ish draft - but the int4 base final is ~3.5/5, below int8.

Related dw gaps seen during the matrix

validate quotes the catalog's 42 min for every H3 variant regardless of steps/weights/LoRA; measured range was 3.5-44 min.
check-script does not validate lines[].shot against real shot names (job fails late).
find_loop_bed returned no candidate on ep 6 where a usable 0.5 s stretch existed (picked by hand at ~-65/-72 dBFS).

Update: the cinema-path final is now P1 = L5 + first_block 0.05, 23.1 min (-22% vs 29.5), with identical staging and faces. Speed follow-ups: #875.

The dw gaps listed above are split into their own issues: estimate #866, check-script shot names #873, find_loop_bed #874.

Other issues filed from the matrix: quality-ramp skill #862, minimax-h3 docs #863, series-episodes #864, template tier presets #865, timing and cache stats #867, acceleration validate warnings #868, host-RAM dispatch #869, LoRA metadata check #870, compare_outputs #871, sweeps and structured notes #872.

Plan v2: #861 H3 quality-ramp follow-up experiments

Written by the lead agent, model claude-opus-5-5 via the anthropic provider. Design session; no code read beyond what's cited below. v2 (decompose session, same model and provider) records Don's answers to Q1–Q3. Scope is unchanged.

Verdict: don't build (no dw change). Run it as a hands-on session instead. Approved by Don (Q1).

This is an idea issue, so the first question is whether it's code at all. It isn't. All three follow-ups are GPU experiments that the current MCP surface can already run (run_workflow with arguments/note, set_note, check-script, get_output_frames). That's how the 60 jobs in h3-quality were run. No stage would change the engine, the MCP, REST or syntax, so the implementer and tester loop has nothing to build or verify. The "Related dw gaps" section is already split into #866, #873 and #874, which leaves no code in this issue.

Stages: none. Nothing here has MCP-observable behavior for the tester to spec. The verifier is Don's hands-on session, judged by #862's rubric.

The axes:

Value. Real. The one consumer is idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862 (the quality-ramp skill), whose Open questions list item 1 (seed stability) and item 2 (cross-family lock) as things to answer "before the skill hardens". Don and the Claude sessions that drive H3 work gain from a skill whose "same family = same staging" rule isn't seed luck. The evidence is the h3-quality workspace: 60 jobs, all Don's matrix, none qa-*.
Build cost. $0 of agent build. The cost is GPU time on lem, from the measured minutes in idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's table (int8 runs one at a time, per bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869):
- Item 1, per extra seed: B3→P1 (12.9 + 23.1), C1→C2 (13.9 + 22.8), L0→L7 (3.5 + 7.2), about 84 min.
- Item 3 joins item 1 as a fourth pair (Q2): B4 (544 int4 base) → B5 (768 int4 base), 6.6 + 11.8, about 18 min per seed. It carried at seed 42. Its ceiling stays at about 3.5/5, below int8.
- Budget (Don): the four pairs take about 102 min per seed. At seeds 7 and 1234 that's about 3.4 h of lem GPU time, run one at a time. Add a third seed only if those two disagree.
- Item 2 is deferred (Q3).
Inertia. None if it's run as experiments. The results land as notes in h3-quality and as rows in idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's table.
Reversibility. Total.
Cheaper alternatives. Doing nothing leaves idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862 shipping its rules with a "one seed per shot" caveat. That's acceptable, but it's the weaker version of the skill.
Overlap.
- idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862 consumes the results, and its open questions duplicate items 1–2.
- idea: sweep runs + experiment tracking - one shot across N configurations, with structured assessments in notes #872 (sweeps and structured notes) would make item 1 a single call instead of tools/ladder.py. It's not needed, though: the local script already works.
- H3 cinema-path speed: follow-up experiments after first_block -22% (threshold, 20 steps, close-up, seeds, MagCache) #875 (speed follow-ups) may cut the B3 draft below about 13 min, which removes the reason for item 2.

Item 2: deferred (Q3)

The issue gives two routes to a cross-family composition lock:

FL2VA pinned to the draft's first and last frames. This runs with the existing templates. It's an experiment, not code, and R3 already showed it costs the voice and identity references.
The draft as a guides clip on ref2va. In the packaged workflows, guides appears only in dw/workflows/h3_context_ir.json. Ref2VA doesn't take it, so this route needs a template change, maybe an engine one too.

Decision: deferred. Reopen it only if #875 can't get the cinema draft under about 7 min, or if a production episode's draft budget is actually blocked on it. If it comes back, it starts as a fresh idea for "guides input on ref2va" and is designed there, not here.

Runbook for Don's hands-on session (not a stage)

In h3-quality, rerun the four pairs B3→P1, C1→C2, L0→L7 and B4→B5 at seeds 7 and 1234. Add a third seed only if the two disagree. Use the same canvas, prompt and references as at seed 42. Each run's note names its rung and seed.
Judge each pair's staging from contact sheets (every 30th frame) and check the head count. That's idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's rubric.
Write the per-seed verdicts into idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's Open questions. Then Don closes this issue as completed (agents can't).

Decisions (Don, answering v1)

Q1: yes. No dw work. It runs as a hands-on session, and the results go to idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's Open questions. Use seeds 7 and 1234, a third only if they disagree, within a budget of about 3.4 h. Don starts the session separately.
Q2: yes. B4→B5 is the fourth pair in item 1.
Q3: yes. Item 2 is deferred, with the trigger above.

Answers to plan v1:

Q1 — yes. No dw work. Run it as a hands-on session and put the results in idea: quality-ramp skill - draft -> preview -> final tiers with measured cost/quality trade-offs (H3 first) #862's Open questions. Run item 1 at seeds 7 and 1234, and add a third seed only if those two disagree. The budget is about 3.4 h of lem GPU time: the four pairs come to about 102 min per seed, run one at a time.
Q2 — yes. Item 3 (B4→B5) becomes the fourth pair in item 1.
Q3 — yes, defer item 2 with your trigger: reopen it only if H3 cinema-path speed: follow-up experiments after first_block -22% (threshold, 20 steps, close-up, seeds, MagCache) #875 can't get the cinema draft under about 7 min, or if a production episode's draft budget is actually blocked on it. If it comes back, it starts as a fresh idea for guides input on ref2va.

No stage in the loop runs a hands-on session, so Don will start that session separately. When the per-seed verdicts are in #862, close this issue as completed. claude-opus-5-5 / anthropic, for Don

Plan v2: changed since v1. It records Don's answers to Q1–Q3: the disposition is approved, B4→B5 is the fourth pair, and item 2 is deferred with its reopen trigger. The budget is seeds 7 and 1234, about 3.4 h, with a third seed only if they disagree. Questions is replaced by Decisions, and the plan now says Stages: none. Scope is unchanged. claude-opus-5-5 / anthropic

Decomposition of plan v2: no stages. The approved verdict is "don't build": every item is a GPU experiment on the existing MCP surface (item 1, with item 3 folded in), or deferred (item 2). So no sub-issues are filed and nothing is blocked. claude-opus-5-5 / anthropic

No acceptance cases are needed. Plan v2 has no stages and nothing MCP-observable for the tester to spec. The verifier is Don's hands-on session, which runs the plan's runbook and judges by #862's rubric.

Handing this to Don, since the next step is his. He runs the hands-on session (seeds 7 and 1234, about 3.4 h on lem), writes the per-seed verdicts into #862's Open questions, and closes this issue as completed. Agents can't close it. claude-opus-5-5 / anthropic

#882 (stage 1 of #862, the H3 quality-ramp section) is now blocked by this issue. It waits for the seed-stability results so it can state each draft → final carryover claim with its seed count. Fallback if this issue is declined: #882 ships with "measured on one seed" against each claim. Labels here are unchanged. claude-opus-5-5 / anthropic

Done 2026-10-15. All the GPU runs were on lem, and Don judged them.

Item 1, seed stability (with item 3 folded in): seeds 7 and 1234, plus seed 99 for B4→B5 because seeds 7 and 1234 disagreed.

B3→P1 (cinema) holds on 3 of 3 seeds.
C1→C2 (close-up) holds on 3 of 3. Blocking carries; props may not.
L0→L7 (TV) holds on 3 of 3. L0 smears Pat's face, a draft-only defect.
B4→B5 (544 int4 base → 768 int4 base) holds on only 2 of 4 seeds (it breaks at 1234 and 99), so it's unsafe as a cinema draft.

Item 2 stays deferred under its trigger. #875 got the cinema final to 18.1 min with MagCache, but the cinema draft (B3, ~13 min) is still above the ~7 min trigger. Reopen it as a fresh idea if a production episode's draft budget is blocked.

The verdicts are in #862's Open questions, and #882 is unblocked. Closing as completed (Don). claude-opus-5-5 / anthropic, for Don

Navigation Menu

Repository navigation

H3 quality ramp: follow-up experiments (seed stability, cross-family composition lock, int4 base draft) #861

What the matrix established (one seed per shot)

Follow-ups (untested)

Related dw gaps seen during the matrix

Plan v2: #861 H3 quality-ramp follow-up experiments

Verdict: don't build (no dw change). Run it as a hands-on session instead. Approved by Don (Q1).

Item 2: deferred (Q3)

Runbook for Don's hands-on session (not a stage)

Decisions (Don, answering v1)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

H3 quality ramp: follow-up experiments (seed stability, cross-family composition lock, int4 base draft) #861

Description

What the matrix established (one seed per shot)

Follow-ups (untested)

Related dw gaps seen during the matrix

Activity

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

Plan v2: #861 H3 quality-ramp follow-up experiments

Verdict: don't build (no dw change). Run it as a hands-on session instead. Approved by Don (Q1).

Item 2: deferred (Q3)

Runbook for Don's hands-on session (not a stage)

Decisions (Don, answering v1)

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 14, 2026

dkackman commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions