Activity

1. dkackman

   added

   ideaAn idea for the researcher agent to assessAn idea for the researcher agent to assess

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   on Oct 15, 2026
2. dkackman commented on Oct 15, 2026

   

   dkackman

   on Oct 15, 2026

   OwnerAuthor

   More actions

   Plan v1: step cache across workflow identities

   claude-opus-5-5 / anthropic, lead. Checked against develop @ 0dce4cb (#905 and #911 landed).

   Verdict: don't build. Instead, add one paragraph of docs saying a copy starts cold

   The idea assumes a renamed or inline copy would hit if the card kept its cache. The code check found two places where it still wouldn't. Separately, the demand check found no real run that would have hit.

   Value: none found in Don's use.

   • Don's real driving in h3-quality and known-issues runs inline workflows. Each variant gets its own id: H3Ladder_P1_base30_fbc05, H3Ladder_P2_base12_fbc05, KnownIssuesRestorePointHD_tag, and so on.
   • The step cache keys entries on (definition id, step_name) (dw/step_cache.py:44-48, :627), not on the identity tuple. A copy with a new id misses whether or not the card keeps its cache.
   • Those variants also differ in the one expensive step: steps, cache config and attention backend change on every H3 rung. There was nothing upstream for a hit to skip.
   • No field-report issue asks for this, and no same-class bug does either. The only cross-identity runs in the job list are the tester's bug: validate's cached_steps probe ignores the card's warm workflow identity #911 verification, in regression-complete.

   What already hits today.

   • Re-running the same stored workflow with changed arguments hits.
   • Re-running the same inline JSON under the same id hits, because the identity is ("inline", id) (dw/worker_protocol.py:242-249).
   • rerun_job hits.
   • That covers the edit-and-retry loop. What it misses is running one stored workflow by name and its inline copy back and forth on the same card. Nothing in the job history does that.

   Build cost: about $10-16 over 3 stages.

   • (1) Retain the cache across a switch, with eviction.
   • (2) Copy files on a hit to a new destination.
   • (3) Prefer the warm card in routing.
   • Each touches worker, routing and probe code at the reach of recent server fixes, so $3-6 a session.

   Inertia: high for what it buys.

   • Correctness seams.
     • A hit republishes files from the other workflow's output folder: the folder is <output_dir>/<identity>/<run>/ (dw/runs.py:296), and the step cache compares only the output_dir root.
     • Republishing those paths mis-attributes outputs in the gallery, in job_for_file and on delete.
     • Avoiding that needs a copy on every cross-identity hit: disk use, plus one more write path through validate_output_path.
   • Memory.
     • _cleanup_all clears the step cache on a switch because a cached Result can hold GPU tensors tied to components released at the same moment (dw/worker.py:703-705).
     • Keeping those Results across a switch means keeping their components, or demoting them to file-only entries. Either is a new retention policy on a 24 GB card. bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 is working on host-RAM pressure on the same cards right now.
   • Routing. A third preference in choose_slot (dw/server/pool.py:119-135) by output identity, which has to agree with the probe. bug: validate's cached_steps probe ignores the card's warm workflow identity #911 just fixed the case where those two disagreed.
   • Tests that pin today's rule. These all need rewriting:
     • tests/test_worker_execute.py:94-173, :690 (bug: validate's cached_steps probe ignores the card's warm workflow identity #911);
     • tests/test_worker_pool.py:704-883;
     • tests/test_step_cache.py:399, which is what currently stops one workflow republishing another's paths.
   • Docs. docs/WORKER_GUIDE.md:21,31-36 and docs/ARCHITECTURE.md:69.

   Reversibility. Good. There's no MCP, REST or syntax surface, so it could be taken out without a breaking-change. That's the one point in its favour.

   Cheaper alternatives:

   1. Do nothing. validate_workflow already reports plan.cached_steps: 0 for a copy, now that bug: validate's cached_steps probe ignores the card's warm workflow identity #911 has landed. An agent that checks before quoting sees the cold run coming. Gives up nothing measured.

   2. One paragraph of docs (recommended). Add it beside the existing step-cache scope paragraph at docs/WORKFLOW_GUIDE.md:2636-2644, which already explains the workspace scope. It would say:

      • the cache is per card and per workflow identity: a stored name, or an inline id;
      • to reuse cached upstream steps, rerun the same workflow (same name, or the same inline id) with changed arguments;
      • a renamed copy, a new inline id, or switching between a stored workflow and its inline copy starts cold.

      It goes to the implementer, docs only: an estimated $1-2, verified over MCP by get_guide("workflows", section=...) returning the text.

   Overlap.

   • plan.estimate prices every H3 variant at the catalog figure - measured range is 3.5-44 min #866 (via #866 stage 0: key the plan fingerprint and step cache only by what changes the output #905) set the current scope deliberately: "Caching across identities is idea: step cache across workflow identities (renamed and inline copies hit) #912".
   • Step cache doesn't survive a killed worker process (disk-backed persistence + resume) #244 (disk-persistent step cache) was declined, which is a precedent against widening the cache's lifetime without demand.
   • bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869 (host RAM) touches the same retention budget.

   What would reopen it:

   • a field-report showing an agent switching between a stored workflow and its inline copy (same id), where the upstream steps are costly;
   • or a multi-step template whose expensive first stage is commonly shared across differently-named workflows. Then it would be worth keying on output identity and copying files.

   If Don overrides to build: the shape it would take

   So the override is a one-reply decision, this is the design I'd write up as v2.

   • Problem. A cached step is lost when a card switches to a different workflow identity that would produce the same step.
   • Non-goals:
     • persistence across worker restarts (Step cache doesn't survive a killed worker process (disk-backed persistence + resume) #244, declined);
     • cross-card sharing;
     • a hit across different definition ids, unless Q2 says otherwise.

   Stages:

   • A: keep the cache across a switch, as file-only entries.
     • On a switch, demote retained Results to saved_files-only entries instead of clearing them, so no GPU tensors outlive their components.
     • Key stays (id, step), plus the output identity from #866 stage 0: key the plan fingerprint and step cache only by what changes the output #905.
     • A hit on an entry from another identity copies the files into the new run's folder, using validate_output_path. It doesn't republish them.
     • The LRU cap stays at 128 entries / 4 GiB.
     • The probe drops its bug: validate's cached_steps probe ignores the card's warm workflow identity #911 short-circuit only for entries the new identity could actually hit.
     • Server deploy. About $5.
     • Acceptance:
       • run stored A with seed s, then its inline copy (same id) with the same arguments on the same card: cached_steps > 0 in validate, reused: true steps in get_job, and output files under the copy's own run folder, not A's;
       • deleting A's run doesn't break the copy's files;
       • a different-id copy reports cached_steps: 0.
   • B: route to the card that holds the entry.
     • choose_slot prefers it after rerun affinity and before same-identity affinity, and the probe routes the same way.
     • Server deploy. About $4.
     • Acceptance:
       • with two cards, the copy goes to the card that ran A, and get_job.device matches;
       • when that card is busy, the copy goes to an idle card and reports cold.
   • Risks:
     • copying large video files on a hit;
     • the demotion pass adding time to a switch;
     • the interaction with bug: two int8 H3 jobs on two cards OOM-kill the host - dispatch is VRAM-aware only, and the earlier job was the one killed #869's host-RAM gate.

   Questions

   • Q1. Do you want the docs paragraph (alternative 2) filed as a fix for the implementer?
     • Default: yes. I'd file it after your reply, as an idea handed straight to owner:implementer.
     • If you say no, this closes wontfix with nothing filed.
   • Q2. If you override to build, should a copy with a different id hit too?
     • Default: no. That needs dropping id from the step-cache key, which reopens the guard at tests/test_step_cache.py:399.
     • If you say yes, it becomes a key change, a third stage.

   Reply in comments and hand back with owner:lead. Approve by adding status:plan-approved yourself. No agent may set it.
3. dkackman

   added

   owner:donParked for human decisionParked for human decision

   status:plan-reviewA feature plan is posted and waiting for DonA feature plan is posted and waiting for Don

   and removed

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   on Oct 15, 2026
4. dkackman commented on Oct 15, 2026

   

   dkackman

   on Oct 15, 2026

   OwnerAuthor

   More actions

   Review of plan v1 — approved: don't build, file the docs fix (Q1 yes, Q2 no)

   Checked against origin/develop @ 9dfb9e6. The code claims hold:

   • the step cache is keyed on (workflow_id, step_name) (dw/step_cache.py, StepCache.get);
   • an inline job's identity is ("inline", id) (dw/worker_protocol.py, workflow_identity);
   • an identity switch runs _cleanup_all, which calls step_cache.clear() (dw/worker.py);
   • test_entries_are_scoped_to_the_workflow_id is the guard against one workflow republishing another's paths;
   • choose_slot prefers rerun card, then warm identity, then an idle card.

   Q1: yes, file the docs paragraph, with two amendments:

   1. Name the commoner cold path. Running any other workflow on the card in between empties the cache: A → B → A runs the second A cold. That's far likelier in real use than switching between a stored workflow and its inline copy. On a multi-card server, warm-identity routing mostly avoids it unless that card is busy. On a single card it happens every time.
   2. Be exact about "changed arguments". Only arguments that reach later steps leave the earlier steps cached. Changing an argument the expensive first step reads re-runs it. Add a line saying validate_workflow's plan.cached_steps is the way to check before running.

   Q2: no. Moot under "don't build", and dropping id from the key would remove the guard test.

   Reopen criteria: add a third trigger. A field report showing an agent alternating two different workflows on one card (A → B → A), where the upstream steps are costly. That would argue for keeping the cache across a switch even without cross-identity hits.

   Lead: file the docs fix to owner:implementer with the amendments above, then close #912 wontfix.
5. dkackman

   added

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   status:plan-approvedDon approved the feature plan (Don only)Don approved the feature plan (Don only)

   and removed

   owner:donParked for human decisionParked for human decision

   status:plan-reviewA feature plan is posted and waiting for DonA feature plan is posted and waiting for Don

   on Oct 15, 2026
6. dkackman commented on Oct 15, 2026

   

   dkackman

   on Oct 15, 2026

   OwnerAuthor

   More actions

   Disposition: don't build (plan v1, approved)

   claude-opus-5-5 / anthropic, lead

   Per Don's review: no stages are filed. The docs paragraph (Q1 yes) went to the implementer as #914, with both amendments:

   • it names the common cold path: A → B → A on one card;
   • it says exactly which argument changes keep earlier steps cached, and to check with validate_workflow's plan.cached_steps.

   Q2: no, so the (id, step) key and test_entries_are_scoped_to_the_workflow_id stay.

   This reopens if any one of these turns up:

   1. a field-report of an agent switching between a stored workflow and its inline copy (same id), where the upstream steps are costly;
   2. a multi-step template whose expensive first stage is commonly shared across differently named workflows;
   3. a field-report of an agent alternating two different workflows on one card (A → B → A) where the upstream steps are costly. That would argue for keeping the cache across a switch even without cross-identity hits.

   Closing wontfix.
7. dkackman

   added

   wontfixThis will not be worked onThis will not be worked on

   and removed

   owner:leadFeature lead's turn (harnest R11)Feature lead's turn (harnest R11)

   on Oct 15, 2026
8. dkackman

   closed this as not plannedon Oct 15, 2026