Repository navigation
T3 fork: Attach a durable report-driven workflow backend through the plugin host #2
Description
Problem Statement
I want T3 Code to execute an authored development process, including conditional reviews, bounded rework, parallel reviewers and a final human gate. A provider turn ending does not tell the application whether a workflow step achieved its intended result. Inferring that result from final prose, a marker such as CEZ:DONE, or another model would make routing unreliable and hard to audit.
T3 supplies native providers, durable threads, workspaces, orchestration and scheduling, but its execution graph is not an authored workflow graph. The missing workflow domain must be added without embedding workflow decisions in provider adapters, core orchestration or client rendering. It also needs to survive crashes between accepting a report, choosing a route and launching the next attempt.
Solution
Implement a workflow backend as the first real plugin. Agents submit a typed report through an injected native tool. The plugin validates and persists that report against the owning workflow node attempt, then chooses an explicit route once the corresponding execution has settled. Definitions, attempts, accepted reports, route decisions, joins, attention and recovery belong to the plugin.
Use the plugin host to execute ordinary T3 threads, inspect their normalized lifecycle, prepare workspaces and schedule work. Offer one graph format that represents simple sequences, typed decisions, bounded repeat routes, parallel groups, wait-for-all joins, checks and human gates. A report is an agent's claim; checks and human approvals remain separate evidence and decisions.
User Stories
- As a workflow author, I want to save a reusable workflow for a project, so that I can run the same development process consistently.
- As a workflow author, I want a simple sequence to be a special case of the graph, so that I do not choose between incompatible workflow types.
- As a workflow author, I want stable node and branch identities, so that renaming a step does not break its routes or history.
- As a workflow author, I want each agent node to select a provider, runtime mode, instructions and an installed skill, so that the process uses the right native harness for each task.
- As a workflow author, I want declared report fields and types, so that conditions operate on validated data rather than arbitrary prose.
- As an agent, I want a native reporting tool with an actionable schema, so that I can submit my result directly to the owning workflow.
- As an agent, I want an exact reporting retry to return the recorded result, so that a lost response does not create a second report.
- As a user, I want reports bound to the current node attempt, so that stale sessions and unrelated threads cannot steer my workflow.
- As a user, I want accepted reports distinguished from verified checks, so that I can assess the evidence behind a result.
- As a workflow author, I want ordered conditions and an explicit fallback, so that the same report always selects the same route.
- As a workflow author, I want equality, membership, presence and numeric conditions with All/Any groups, so that common decisions require no scripting language.
- As a workflow author, I want bounded repeat routes with a defined limit destination, so that rework cannot continue indefinitely.
- As a workflow author, I want deterministic check nodes, so that command results can inform routing independently of an agent's claims.
- As a workflow author, I want explicit typed input bindings between nodes, so that later steps receive the intended reports and artifacts.
- As a user, I want a missing report to trigger a bounded reminder and then a visible unresolved state, so that the workflow neither guesses success nor silently hangs.
- As a user, I want native approval and input requests to remain visible, so that workflow execution does not bypass the harness's controls.
- As a user, I want my explicit stop or cancellation to take precedence, so that recovery cannot restart work I stopped.
- As a user, I want interrupted work to offer a safe resume or retry, so that I can recover without duplicating a completed step.
- As a workflow author, I want several independent reviewers to run concurrently, so that I can combine code, security and UX review.
- As a workflow author, I want two instances of the same review skill to retain distinct branch identities and focus, so that their results cannot be confused.
- As a user, I want all reviewers to inspect the same frozen commit in isolated worktrees, so that the combined review refers to one reproducible change.
- As a workflow author, I want a wait-for-all join, so that one early verdict cannot skip another required reviewer.
- As a user, I want missing, failed, timed-out or stale reviewer results to prevent a successful join, so that an incomplete review is not presented as approval.
- As a user, I want PR changes during review identified explicitly, so that I can rerun reviews on the new head.
- As a user, I want a development workflow to stop at a human review gate, so that completion remains an explicit decision I can make.
- As a user, I want human decisions recorded against the gate and reviewed revision, so that a delayed click cannot approve newer work accidentally.
- As a user, I want to see which predicates were considered and why a route was chosen, so that I can explain the workflow's behavior.
- As a user, I want definitions snapshotted into runs, so that editing a saved workflow does not change a running process.
- As a user, I want local and remote clients to operate on the same server-owned state, so that closing my browser does not stop workflow execution.
- As a user, I want manual and scheduled starts to create the same kind of workflow run, so that scheduling does not change completion semantics.
- As a user, I want restarts and event redelivery to preserve reports, loop counters and joins, so that recovery cannot choose another route or launch duplicate children.
- As a user, I want actionable attention items for review, interruption and unresolved execution, so that I can find every workflow requiring me.
- As a maintainer, I want workflow code to depend only on the plugin interface, so that pulling upstream changes does not require rewriting the engine.
- As a user, I want an unsupported provider or missing skill rejected before execution, so that a workflow cannot quietly fall back to transcript markers.
Implementation Decisions
- Plugin ownership: the backend registers its services, authenticated API, reporting toolkit, scheduling target and attention read model through the plugin host. Its modules cover definition validation/catalog, report acceptance, graph decisions, execution effects, persistence/recovery and public queries/commands. No workflow-specific conditions are added to the T3 orchestrator, provider adapters or core thread statuses. Workflow runs and node attempts are separate domain objects from T3 turns.
- Definition format: use one versioned, schema-validated graph with a stable workflow ID, revision, entry node, named nodes, typed input/report contracts and declared routes. Node kinds are agent, check, decision, parallel group, join, human gate and end. A sequence uses ordinary unconditional edges. Graph agents require reports. There is no separate simple/advanced execution mode.
- Catalog and revisions: discover packaged examples and project-local, committable YAML through one conventional location without a mandatory configuration file. Invalid files remain visible with their validation reason. Save uses atomic writes and an expected revision to reject overwriting a newer edit, and validates the capabilities required for a runnable definition in the target environment. An imported or existing definition with unavailable capabilities remains visible as not runnable. Each start persists the full resolved definition, version, selected skills, input, workspace choice and digest before any external work. Development runs default to a new primary worktree shared only by their sequential implementation/check nodes; parallel reviewers always receive isolated worktrees. Later edits affect future starts only. Read-only loading never rewrites definitions.
- Schema ownership: the plugin exports one canonical Effect/Schema contract for definitions, reports, commands, state and subscriptions. Clients and tools consume it. Centralize and publish finite limits: initially 64 nodes, 256 edges, 32 declared data fields per report, predicate nesting depth 8 and 128 predicate terms per decision. Return routes allow 1–20 repeats; the whole-run visit limit defaults to 100 and can be authored within 1–1,000. Validate limits at save and start. These are data/protocol and loop bounds, not agent concurrency or cost budgets.
- Typed reports: the reporting envelope contains a protocol version, client retry key, outcome (
completed,blockedorfailed), short summary, bounded structured data and evidence references. The definition declares routable data fields with boolean, string, finite number or enum types and required/optional status. Reserved envelope fields are routable under their documented types. Validate both the envelope and the current node's declared fields. Do not infer fields from a skill's prose, shell output or UI tool display. Initially cap the encoded envelope at 64 KiB, summary at 4,000 characters and evidence at 20 bounded references; reject excess before persistence and document the limits in the public contract. - Tool binding: derive the owning attempt from authenticated environment/thread/session context and an immutable persisted thread-to-attempt association. The caller cannot select an arbitrary workflow run, node, branch or attempt. Bind before launching the thread, and verify attempt state inside the acceptance transaction. Include the current node's exact expected report contract and evidence requirements in its launch instructions; the generic reporting tool validates its data against that persisted contract and returns field-specific errors. A reporting thread belongs to exactly one attempt; a repeat or retry creates a new attempt and thread. The bounded missing-report reminder can reuse the same thread/session and attempt. An unrelated delegated thread has no inherited authority to report for its parent.
- Acceptance and receipts: one immutable report is accepted per attempt. Atomically validate ownership/current state and persist the report, attempt identity, timestamp, protocol version, retry key, payload digest, receipt and workflow event. An exact authenticated retry returns its existing receipt without mutation; a reused key with different data or a conflicting second submission is rejected. New submissions from stale, canceled or settled attempts are rejected. Receipt identity uses persistent attempt identity, not a credential namespace regenerated after restart. Revoked credentials remain unauthorized. Redact known secrets from display/logging, cap rendered payloads and avoid logging credentials; the report store must not be presented as proof that its evidence claims were verified.
- Provider eligibility: require the host's proven reporting-tool capability at start and after resume. Initial mandatory support is managed Codex, Claude and OpenCode, including resumed sessions and headless environment execution. Read-only review modes must deliberately permit this mutating reporting operation while retaining their workspace restrictions. Publish decisions for every other selectable provider/mode; disable unsupported choices and reject API starts. Externally managed OpenCode is unsupported until injection is proven. No fallback to a printed marker, observed tool-call frame or judge model is allowed.
- Skills and inputs: discover installed Open Mercato/native skills from the selected environment/provider through the host. Store the selected skill identity and available revision/source fingerprint plus resolved invocation inputs with the attempt. Use the provider's supported native invocation semantics. If exact skill version identity cannot be established, record that limitation and the resolved input snapshot rather than claiming a version lock. Validate availability at start/resume and expose changes as an unresolved reason. Bind later node inputs to explicit prior report/check/artifact references with schema validation and bounded payloads; never implicitly forward an entire transcript.
- Decision semantics: evaluate ordered rules using the first matching rule, then the required explicit Otherwise route. Support typed equality/inequality, enum/string membership, presence/absence, numeric comparisons and nested All/Any groups. A missing optional field makes ordinary comparisons false, including inequality; only explicit presence/absence predicates inspect absence. Reject unknown fields, incompatible operands, unsafe expression strings, missing targets/fallbacks, invalid join references, excessive predicate complexity, unreachable terminal paths and unbounded cycles. Checks expose their typed execution result, not arbitrary output paths. There is no LLM-selected ordinary edge.
- Report versus settlement: an accepted report is the semantic input. It does not by itself finish a node. Before routing, require the owning execution's normalized settlement condition: its relevant turns, child/background work, queued workflow follow-ups and required checkpoint/artifact work have completed or have an explicit terminal failure. Derive that condition from the host work aggregate and lifecycle, not final prose or one provider-exit event. An interrupted or failed execution cannot become a successful node solely because it submitted a
completedclaim earlier. - Missing report protocol: once an otherwise eligible agent execution settles without an accepted report, persist and deliver at most one protocol reminder to the same attempt/session. Invalid submissions return an actionable tool error; if no report is accepted when that reminder settles, record an unresolved result. Automated attempts have a persisted execution timeout, defaulting to two hours and authorable from one minute to 24 hours. Native human-input waits pause the remaining execution timeout; the pause and remaining duration survive restart. An authored human-response deadline can separately bound that wait. Timeout expiry interrupts the owned work and produces a bounded unresolved result. Human-input waits remain visible and exit only through the corresponding input/approval, explicit cancel or an authored deadline policy; they do not masquerade as completed work.
- Checks: execute commands in the selected environment and intended workspace with persisted start intent, timeout/cancellation and bounded output. Record exit code, timeout/interruption and evidence. A crash after command start is not evidence of success: reconcile a retained result or mark it unresolved. Retry a check only under its declared retry policy or an explicit user action; do not assume arbitrary commands are safe to replay. External process execution remains outside the decision transaction.
- Durable transitions: serialize workflow commands. Atomically persist accepted input references, predicates considered, chosen edge, counters, next-attempt identity, workflow events, projection, command receipt and plugin outbox intent before launching the next work. Keep external I/O outside this transaction. Cross into T3 using stable command/thread/message identifiers and launch receipts. After a lost acknowledgement, reconcile the existing host action before retrying; never generate another identity. A cursor gap or snapshot fallback triggers state reconciliation, not simulated missing events.
- Repeat rules: every return route declares a maximum repeat count and an At limit destination. Initial execution plus one repeat means exactly two visits. Persist counters with the selected transition. Validate cycle coverage and require a whole-run visit bound. Count admitted node visits, including branch visits and explicit new-attempt retries; the reminder within one attempt is not a new visit. Once a bound is exhausted, choose the declared limit/human/unresolved path and record why. A bound exit may enter an end or human gate but may not admit further automated work. Never silently reset counters after restart or user resume. A new run has fresh counters.
- Parallel reviews: persist the fork's required branch set, stable branch IDs, child attempt/thread identities, frozen commit and launch intents before starting children. Reserve required branch visits atomically when committing the fork; insufficient remaining run visits takes the bound exit before any child launches. Skill identity is not branch identity; two code-review branches remain distinct. Each reviewer uses a separate worktree prepared from exactly the frozen commit and the declared read-only/external-action policy. Validate that the commit exists and represents the intended committed PR head; uncommitted edits cannot silently enter the review input. Verify the actual reviewer checkout commit and unchanged tracked review input at launch and settlement; mismatched workspace evidence invalidates that branch even if its report claims the correct head. Normal model-directed delegation, which can share a parent checkout, is not the parallel review execution path.
- Wait-for-all join: persist one join result only after every required branch reaches a terminal attempt state. An early request for changes does not cancel or skip siblings. A
Reportedbranch remains pending until its execution settles.all_completedrequires every required branch to have valid accepted reports and successful settlement on the same frozen commit. A failure, missing report, timeout, unsupported execution, cancellation or changed head yields the documentedfailed,unresolved,canceledorstaleaggregate; none can satisfy the success condition. Every review branch additionally has an absolute persisted deadline, defaulting to two hours from launch and including human-input waits, so an abandoned child cannot hold the join forever. Expiry interrupts the child and records an unresolved result. Explicit workflow cancellation remains immediate and authoritative. - Freshness: compare the PR head with the frozen review input before committing the join and again before approving a human gate or consuming the review for subsequent change-sensitive work. A changed head invalidates the review set; unavailable freshness evidence is unresolved rather than presumed current. Rerunning reviews creates a new frozen set and branch attempts. Never mix reports from different review generations.
- Human gates and cancellation: persist the expected gate/run revision and any frozen review reference. Approval or request-for-changes uses an authenticated, idempotent command against that revision. Stale decisions fail without mutating newer work. Built-in development examples end at human review after their checks/review branches. Approval can complete the run or take a declared edge; changes can take a bounded rework route. No gate authorizes automatic PR merge. Workflow cancel first persists cancellation, invalidates future launch intents and then interrupts every owned active attempt. A manual native thread stop becomes an explicit interruption requiring a configured unresolved route or user recovery; it must not trigger automatic retry.
- Lifecycle and exits: public run states are running, awaiting review, unresolved, completed, failed and canceled. Running exits through a persisted next transition, gate, unresolved result, end or cancel. Awaiting review exits through a recorded gate decision or cancel. Unresolved exits through an explicit eligible resume, new-attempt retry, configured failure route or cancel. Completed/failed/canceled runs do not accept new transitions; running again creates a new run. Persist node phases and reasons separately, including native input waits and join waits. Deadline, host lifecycle and manual commands are the named wake sources. Human waits do not hold an active agent process merely to retain workflow state.
- Recovery: recover plugin state after core storage/recovery is available, reconcile pending effects and owned threads, restore subscriptions/deadlines and continue already committed transitions exactly once. Do not change T3's shipped automatic provider-resume policy. When a lost provider cannot continue under existing policy, expose an interrupted unresolved attempt with eligible Resume or Retry actions. Resume is available only when the same native session and attempt can safely continue; otherwise require a new attempt. Successful reports, loop counts and settled join results survive restart without reevaluation from text.
- Scheduling: register a workflow schedule target with the generic host interface. Manual Run and scheduled dispatch call the same start service. Use the host's durable occurrence ID as a start idempotency key and persist the selected definition snapshot with that occurrence. Two distinct occurrences remain distinct runs; retrying one occurrence cannot create another run. Preserve existing timer/missed-slot behavior. A schedule's dispatch receipt is distinct from the eventual workflow outcome.
- Public surface: expose definition validation/catalog/save, start, get/list runs, subscribe/reconcile, cancel, eligible resume/retry, gate decisions and accepted-report/route/join history. Return allowed actions from authoritative state. Attention queries include awaiting review, unresolved/interrupted execution and native requests requiring the user, with stable IDs and related thread links. Clients do not reexecute predicates or infer completion. Reads/subscriptions are bounded and scoped to the intended environment/project.
- Acceptance example: run implementation, checks and three frozen-head reviewers: separate code and security instances of the code-review skill plus UX review. Join waits for all. Any configured changes/blocker verdict enters a single permitted implementation repeat; another matching verdict follows At limit into human attention. All passing reports/checks lead to human review. The run trace must explain each transition and retain the exact frozen input and report references through restart.
Testing Decisions
- The primary seam is the plugin's public workflow commands, queries, subscriptions and real reporting tool exercised through the registered host. Assert resulting state, reports, host execution, route trace and user-visible allowed actions. Do not mock the graph engine, acceptance transaction, projections, join logic or core persistence.
- Use real plugin persistence and real T3 command/launch/adapter paths with deterministic provider-transport transcripts. Substitute actual external processes/network, time, IDs and temporary filesystem/database locations. Await persisted milestones, receipts or drainable workers rather than sleeps.
- Prove real native tool discovery/invocation for the mandatory provider matrix, including new, resumed and headless sessions, readonly review policy and credentials refreshed after restart. Test unrelated threads, stale attempts, canceled/settled attempts, malformed/oversized payloads, conflicting submissions and exact retry after a lost response. A fabricated UI tool-call event or transcript marker must never produce an accepted report.
- Test a declared boolean/enum/numeric route, rule ordering, nested All/Any, absent optional fields and Otherwise. Invalid definitions must reject unknown paths/targets, type mismatches, missing fallback, unbounded cycles, dangling branch references and unreachable terminal paths without partially saving or starting.
- Test that acceptance alone does not advance while child/background/checkpoint work remains. After settlement, one accepted input selects exactly one durable edge. A completed claim followed by runtime failure or explicit stop must not produce a success transition.
- Test missing/invalid reporting with one persisted reminder and then unresolved state, including restart between reminder intent and delivery. Test attempt deadlines, native input waits, interrupted checks and permitted resume/retry. The test must demonstrate every nonterminal state has its documented wake/exit source.
- Test initial implementation plus one permitted repeat, exhausted repeat, whole-run exhaustion and unchanged counters after restart. Assert selected input/report references and recorded conditions rather than recomputing the route in the test client.
- Inject crashes after report acceptance, before/after edge commit, before/after host launch acknowledgement, during fork child creation and before/after join commit. Redeliver commands/events and force subscription snapshot fallback. Assert one report, one chosen transition and one host thread per committed attempt; no receipt uses a regenerated session namespace.
- Exercise all reviewer settlement orders, two branches using one skill, 2/3 accepted reports, 3/3 reports with one still-running child, an early changes verdict, an unresolved/timed-out/failed child, explicit cancel and a changed/unverifiable PR head. Assert no success join before all terminal requirements and no mixing review generations.
- Test gate approval, request for changes, stale gate revision, PR-head change after review, double clicks and cancellation racing report acceptance/launch. Assert server-owned attention and allowed actions survive restart and cannot be cleared by marking a thread read.
- Test workflow edits during a run, catalog parse errors, expected-revision conflicts, unavailable skills/providers and scheduled occurrence redelivery. Run the same acceptance graph manually and through scheduling with the same semantics. Confirm normal T3 threads and prompt schedules still operate independently of the workflow plugin.
- Prior art includes T3's thread launch tests, MCP toolkit integration, command/control reads, delegated completion delivery, replay recovery suites, scheduler tests and the documented V2 integration strategy. Reuse actual persistence and transport replay; focused pure predicate tests are appropriate only for difficult type/absence/boundary cases.
Out of Scope
- Visual editing, workflow pages and client attention presentation; those belong to the dependent UI spec.
- Inferring outcomes from transcript prose, magic markers, displayed tool events or a third-party judge model.
- LLM judge nodes, arbitrary expression languages, automatic PR merge and automatic merging of parallel edit branches.
- New cost budgets, global concurrency controls or a replacement for T3 provider orchestration.
- A new skills marketplace, installer, updater or a fabricated cross-provider skill version lock.
- Automatic import, migration or execution compatibility for Cezar's legacy YAML/run records and marker-based completion. The earlier Cezar issues provide requirements, not a storage/protocol dependency.
- Exactly-once guarantees for arbitrary third-party side effects. Durable intent, host receipts and reconciliation cover the controlled execution boundary; unresolved ambiguity remains visible.
Further Notes
This is specification 2 of 3. It depends on the plugin-system spec and supplies the canonical domain/API for the workflow UI spec. It includes the backend work needed for parallel review and composite predicates; that work cannot be deferred to UI implementation.
Requirements are adapted from structured reporting #10, deterministic routing #11 and unified authoring/review #12. Those issues remain unchanged and target Cezar. These new specs target a T3 fork.
Recheck the implementation base against the inspected T3 main before coding. Architecture and turn settlement, launch service, MCP tool registration, testing strategy.
Implement in dependency order within this issue: report acceptance and one sequential workflow; durable routing/repeat/check/gate behavior; recovery and scheduling; frozen-input fork/join and its failure paths. Keep each intermediate build truthful about its supported capabilities. The UI must only expose graph controls whose final backend contract is implemented and advertised.
Activity
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries
feat(workflows): add durable typed workflow execution
fix(workflows): reconcile recovery and reporting boundaries