-
Notifications
You must be signed in to change notification settings - Fork 0
docs(examples): nightly-implementer design artifact for flows-driven dogfooding #295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # nightly-implementer | ||
|
|
||
| Design artifact for how flows should orchestrate its own implementation | ||
| work overnight. Filed on 2026-09-10 as a target shape; the primitives it | ||
| uses land in the same launch wave (flows#273 `f.llm`, flows#275 | ||
| declarative value binding, structured `output` schemas on agent steps). | ||
| Until those close, treat this as intent rather than a runnable script. | ||
|
|
||
| ## What the shape proves | ||
|
|
||
| Every decision that lets code advance toward merge is a **deterministic | ||
| step reading typed JSON**. The agents produce blockers, reasons, and diffs; | ||
| they never adjudicate their own work. That's the whole point. | ||
|
|
||
| Stage by stage: | ||
|
|
||
| 1. **`worktree` — deterministic setup**. Checks out a fresh pinned base, | ||
| emits `{path, baseSha}`. `json_schema` verification asserts both fields. | ||
| 2. **`plan` — `f.llm` with typed output**. Compiles to a `json_schema` | ||
| verification so a hallucinated plan that doesn't parse as | ||
| `{files, testFiles, risks}` fails the step, doesn't bypass it. | ||
| 3. **Repair loop, bounded to 3 iterations**. Each pass has: | ||
| - **`impl` — agent step**. Typed `output` schema (`{sha, filesTouched}`). | ||
| Prior-iteration blockers arrive via declarative binding as | ||
| `input.blockers`. | ||
| - **`scope-gate` — deterministic**. Compares actually-touched files | ||
| against the declared allowlist, emits `{scopeOk}`. This is the exact | ||
| shape flows#242 / #271 needed and the class fix #284 tracks. | ||
| - **`verify` — deterministic**. `tsc --noEmit` + `vitest --reporter=json`. | ||
| Verification asserts `numFailed === 0 AND tscOk === true`. An agent | ||
| cannot bypass this by asserting success — the JSON is emitted by the | ||
| tools themselves. | ||
| - **Review swarm — three lenses, parallel**. Correctness / regression / | ||
| maintainability. Different providers so shared-model bias cannot | ||
| quorum. Each is told "default to `blocked=true`" — the | ||
| [review-fix-signoff](../../.claude/skills/review-fix-signoff-loop) | ||
| covenant. No lens sees another's verdict. | ||
| - **`aggregate-review` — deterministic**. Combines three lens verdicts | ||
| with the verify gate result into `{approved, blockers}`. Approve | ||
| requires ALL FOUR green. Any lens without `reasons[]` alongside | ||
| `blocked=true` is treated as blocked-with-no-reasons and still blocks. | ||
| 4. **`report-blocked` — needs_human hand-off**. When the repair loop | ||
| exhausts its budget without unanimous approval, the run terminates in | ||
| `needs_human` (RFC-0001 gate 5, flows#251), not `step_failed`. The | ||
| iteration count and every accumulated blocker survive on the outcome | ||
| for a human tail. The batch reports these separately from delivered PRs. | ||
| 5. **`open-pr` — agent step declaring `surfaces.external`**. Drops a JSON | ||
| file at the relayfile mount's PR-creation path. Gate 6 elects one | ||
| attempt via `effect.record`, journals the write, confirms via | ||
| `effect.confirm`. Filename at the mount is the idempotency key — a | ||
| retried attempt writes the same name and the GitHub adapter refuses | ||
| the duplicate. **This is NOT `gh pr create` via a `deterministic` | ||
| step**: shelling that out carries none of the effect-journal or | ||
| idempotency guarantees. | ||
|
|
||
| ## What the shape avoids | ||
|
|
||
| - **Agent-adjudicated gates**. Every merge-forward decision is a | ||
| deterministic step reading typed JSON. No LLM rationalizes its way past | ||
| a boolean. | ||
| - **Single-reviewer rubber-stamping**. Three lenses, distinct angles, | ||
| distinct providers, distinct prompts. | ||
| - **Silent stalls**. `MAX_REPAIR_ITERATIONS` is finite; exhaustion is a | ||
| distinct outcome (`needs_human`) from either success or `step_failed`. | ||
| - **Shell-escape hatch for PR opening**. Every writeback goes through the | ||
| journaled effect protocol with filename-as-idempotency-key. | ||
| - **Free-form verification**. Every `verification:` is `json_schema` over | ||
| a shape the producing tool emits directly, or `output_contains` against | ||
| a specific literal. | ||
|
|
||
| ## What still needs to land | ||
|
|
||
| - **flows#273** — `f.llm` in the TS surface with typed `output`. | ||
| - **flows#275** — declarative value binding (the `{step, path}` selector | ||
| shape this file uses in every `input:`). | ||
| - **flows#274** — YAML agent local-worker path (if the batch driver is | ||
| ever re-authored in YAML). | ||
| - **flows#284** — first-class immutable-acceptance-inputs. The scope gate | ||
| here approximates it as a deterministic step; a proper primitive would | ||
| make the guarantee unbypassable at the schema level rather than at the | ||
| script level. | ||
| - **cloud#3534** — running-stall sweep. Necessary for the batch driver | ||
| to survive a stranded sub-run without keeping the run row in `running` | ||
| forever. | ||
|
|
||
| Once those five close, `flows run --local-agent nightly-batch.flow.ts` runs | ||
| this shape end-to-end. | ||
|
|
||
| ## How to run once primitives land | ||
|
|
||
| ``` | ||
| export RELAYCAST_WORKSPACE_KEY=rk_live_... # observer URL support | ||
| export RELAYFILE_MOUNT=/relayfile # PR writeback path root | ||
|
|
||
| # One-shot batch of five issues: | ||
| flows run --local-agent examples/nightly-implementer/nightly-batch.flow.ts \ | ||
| --input '{"issues":[ | ||
| {"repo":"AgentWorkforce/flows","issue":XXX,"briefPath":"...","declaredFiles":[...]}, | ||
| ... | ||
| ]}' | ||
| ``` | ||
|
|
||
| The `Observer:` line on stdout is the shareable live view (flows#269 / | ||
| #286). The final `RUN <id> completed` carries the aggregate summary | ||
| including per-issue outcomes. | ||
|
|
||
| ## Verify the deterministic pieces | ||
|
|
||
| The future surface primitives prevent an end-to-end run today, but the shell | ||
| gates and their flow wiring are pinned locally: | ||
|
|
||
| ``` | ||
| bash examples/nightly-implementer/tests/verify.sh | ||
| ``` | ||
|
|
||
| ## Provenance | ||
|
|
||
| Shape informed by the [review-fix-signoff-loop](../../.claude/skills/review-fix-signoff-loop), | ||
| [relay-80-100-workflow](../../.claude/skills/relay-80-100-workflow), and | ||
| [writeback-as-files](../../.claude/skills/writeback-as-files) skills — plus | ||
| the concrete drive-local defect class documented at flows#242 / #271 / | ||
| #284 and the 2026-09-10 shakedown evidence at | ||
| `evidence/shakedown-0910/` on branch `shakedown/v2-launch-0910`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| // Top-level batch driver for the nightly implementer. | ||
| // | ||
| // Fan-out across N per-issue implementer runs. `Promise.all` IS the | ||
| // parallelism primitive per RFC-0001 (flows#251) — no separate `parallel:` | ||
| // declaration, no dependency-string sugar. A worker that dies inside one | ||
| // implementer does not stall the others; the runner reports per-branch | ||
| // completion reasons and the aggregate outcome names the survivors. | ||
|
|
||
| import { flow, f } from '@relayflows/surface'; | ||
| import implementIssue from './nightly-implementer.flow'; | ||
|
|
||
| interface IssueBrief { | ||
| repo: string; | ||
| issue: number; | ||
| briefPath: string; | ||
| declaredFiles: string[]; | ||
| } | ||
|
|
||
| interface BatchInput { | ||
| issues: IssueBrief[]; | ||
| } | ||
|
|
||
| interface NeedsHumanResult { | ||
| outcome: 'needs_human'; | ||
| inspectionUrl?: string; | ||
| } | ||
|
|
||
| function isNeedsHuman(result: unknown): result is NeedsHumanResult { | ||
| return typeof result === 'object' && result !== null && | ||
| (result as { outcome?: unknown }).outcome === 'needs_human'; | ||
| } | ||
|
|
||
| export default flow(async (f, input: BatchInput) => { | ||
|
|
||
| // A single implementer's failure must not sink the batch. The catch | ||
| // folds it into a structured per-issue outcome the aggregator reads. | ||
| // The runner still marks the sub-run as failed; this pattern only | ||
| // controls what the enclosing run reports. | ||
| const outcomes = await Promise.all( | ||
| input.issues.map(async (brief) => { | ||
| try { | ||
| const result = await implementIssue(f, brief); | ||
| if (isNeedsHuman(result)) { | ||
| return { | ||
| issue: brief.issue, | ||
| repo: brief.repo, | ||
| outcome: 'needs_human' as const, | ||
| inspectionUrl: result.inspectionUrl ?? null, | ||
| }; | ||
| } | ||
| return { | ||
| issue: brief.issue, | ||
| repo: brief.repo, | ||
| outcome: 'delivered' as const, | ||
| prUrl: (result as { prUrl?: string }).prUrl ?? null, | ||
| }; | ||
| } catch (error) { | ||
| return { | ||
| issue: brief.issue, | ||
| repo: brief.repo, | ||
| outcome: 'failed' as const, | ||
| error: error instanceof Error ? error.message : String(error), | ||
| }; | ||
| } | ||
| }), | ||
| ); | ||
|
|
||
| // Deterministic summary — a single JSON payload the operator sees on | ||
| // wake, including a bounded diagnostic per outcome. No agent adjudicates | ||
| // this either: the aggregate is mechanical, so a human reading the run | ||
| // report can trust its counts. | ||
| return await f.deterministic({ | ||
| id: 'summary', | ||
| command: | ||
| `./scripts/report-batch.sh --outcomes "$OUTCOMES"`, | ||
| input: { OUTCOMES: JSON.stringify(outcomes) }, | ||
| verification: { | ||
| type: 'json_schema', | ||
| schema: { | ||
| type: 'object', | ||
| required: ['delivered', 'needsHuman', 'failed', 'items'], | ||
| properties: { | ||
| delivered: { type: 'number' }, | ||
| needsHuman: { type: 'number' }, | ||
| failed: { type: 'number' }, | ||
| items: { | ||
| type: 'array', | ||
| items: { | ||
| type: 'object', | ||
| required: ['issue', 'outcome'], | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }); | ||
| }); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.