Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/review-swarm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
name: Review swarm

'on':
pull_request:
types: [opened, synchronize, reopened]

permissions:
contents: read
pull-requests: write

concurrency:
group: review-swarm-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
review:
if: >-
${{ contains(fromJSON('["kjgbot", "miyaontherelay"]'),
github.event.pull_request.user.login) }}
runs-on: ubuntu-latest
timeout-minutes: 50
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
GH_TOKEN: ${{ github.token }}
steps:
- name: Check out pull request merge commit
uses: actions/checkout@v4
with:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
fetch-depth: 0

- name: Install agent-relay
run: npm install --global agent-relay

- name: Set review target
run: echo "$PR_NUMBER" > .review-target

- name: Launch review swarm
id: launch
shell: bash
run: |
set -euo pipefail
response=$(agent-relay cloud run workflows/review-swarm.yaml --json)
run_id=$(jq -er '.runId // .run_id // .id' <<<"$response")
echo "run_id=$run_id" >> "$GITHUB_OUTPUT"

- name: Wait for review swarm
shell: bash
env:
RUN_ID: ${{ steps.launch.outputs.run_id }}
run: |
set -euo pipefail
deadline=$((SECONDS + 2700))
while (( SECONDS < deadline )); do
response=$(agent-relay cloud status "$RUN_ID" --json)
status=$(jq -er '.status // .run.status' <<<"$response")
echo "review-swarm status: $status"
if [[ "$status" == "completed" ]]; then
exit 0
fi
if [[ "$status" == "failed" ]]; then
echo "review-swarm completed with a failed verdict"
exit 0
fi
if [[ "$status" == "cancelled" ]]; then
echo "review-swarm was cancelled" >&2
exit 1
fi
sleep 30
done
echo "review-swarm did not complete within 45 minutes" >&2
exit 1

- name: Sync and post reviews
env:
RUN_ID: ${{ steps.launch.outputs.run_id }}
run: bash .github/workflows/scripts/swarm-post.sh "$RUN_ID" "$PR_NUMBER"
50 changes: 50 additions & 0 deletions .github/workflows/scripts/swarm-post.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
set -euo pipefail

if [[ $# -ne 2 || ! $2 =~ ^[0-9]+$ ]]; then
echo "usage: $0 <run-id> <pr-number>" >&2
exit 2
fi

run_id=$1
pr_number=$2

agent-relay cloud sync "$run_id"

declare -A verdicts
for lens in maintainability history structure; do
mapfile -t transcripts < <(
find ops/reviews -maxdepth 1 -type f -name "*-pr${pr_number}-${lens}.md" -printf '%T@ %p\n' |
sort -nr | cut -d' ' -f2-
)
if [[ ${#transcripts[@]} -eq 0 ]]; then
echo "missing $lens transcript for PR #$pr_number" >&2
exit 1
fi

transcript=${transcripts[0]}
if grep -q 'REVIEW_FAILED' "$transcript"; then
verdicts[$lens]=FAILED
elif grep -q 'REVIEW_PASSED' "$transcript"; then
verdicts[$lens]=PASSED
else
echo "$transcript has no review verdict" >&2
exit 1
fi
gh pr comment "$pr_number" --body-file "$transcript"
done

logs=$(agent-relay cloud logs "$run_id")
if grep -q 'SWARM_PASSED' <<<"$logs"; then
aggregate=PASSED
elif grep -q 'SWARM_FAILED' <<<"$logs"; then
aggregate=FAILED
else
echo "cloud log has no aggregate swarm verdict" >&2
exit 1
fi

marker="🎯 review-swarm: $aggregate (M:${verdicts[maintainability]} H:${verdicts[history]} S:${verdicts[structure]})"
gh pr comment "$pr_number" --body "$marker"

[[ $aggregate == PASSED ]]
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,5 @@ Nine gates, in `docs/RFC-0001` §3. Gate 1 first: a relayflow can run — the he
ladder survives `kill -9` at every boundary.

Private while we build. YC 2026-09-15 runs on this base.

The `RELAY_WORKSPACE_KEY` repository secret authenticates review-swarm Actions to the canonical Agent Relay Cloud workspace.
14 changes: 14 additions & 0 deletions ops/NEEDS_HUMAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Gate 3 prerequisites needing a human

The current run cannot verify whether `RELAY_WORKSPACE_KEY` exists on
`AgentWorkforce/flows`: `gh secret list --repo AgentWorkforce/flows` reports
that GitHub CLI authentication is missing. A repository administrator must
verify or add that Actions secret from the workspace key stored on the laptop
at `~/.agentworkforce/relay/cloud-auth.json`.

The required posting dry run against an existing completed cloud run is also
blocked here: Agent Relay reports `Cloud login required`, and GitHub CLI has no
authentication with which to post a PR comment. Run
`.github/workflows/scripts/swarm-post.sh <completed-run-id> <pr-number>` from an
environment authenticated to both services and capture the resulting comment
URLs.
132 changes: 68 additions & 64 deletions ops/NEXT.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,91 @@
# NEXT — work package for this tick

**Scope:** Build a minimal agent worker in the SDK. CODE task, SDK-side.
**Scope:** Wire the review-swarm to fire on PR open via GitHub Actions + `agent-relay cloud run`. CODE task, `.github/workflows/`-side.

This run is pinned to **gate 3** and must not work on any other gate.

## Objective

Promote the throwaway worker the tests already build into a real SDK component
that can execute agent steps by running their declared CLI as a subprocess.
Create `.github/workflows/review-swarm.yml` that automatically triggers the existing `workflows/review-swarm.yaml` workflow when drive-loop PRs are opened, synchronized, or reopened. The swarm produces three independent reviews (maintainability/history/structure) and posts them as PR comments with an aggregate verdict.

## Context

Nothing in this repo can execute an agent step. Searching for `workerAttach` /
`step.complete` finds only TESTS (`sdk/tests/live-kernel.test.ts`,
`journal-client.test.ts`, `journal-client-loopback.ts`) and the protocol
definitions. `sdk/src/cli/run.ts` only OBSERVES worker leases and waits for one
that never arrives.
`workflows/review-swarm.yaml` already exists in this repo — three model-diverse reviewers (claude/codex/opencode) that read a PR diff and produce three independent transcripts + one aggregate verdict. Today it only fires when a human writes `.review-target` and runs `agent-relay cloud run` by hand. It has run zero times against a drive PR on cloud.

The kernel's dispatch, lease and claim machinery is real and tested. The worker
side of the protocol is simply unimplemented, and that is what blocks gate 2
("a workload RUNS as a relayflow" — today a run can only be shown CREATED) and
gate 3 ("every claim/lease/retry served by the kernel").
Meanwhile the reviewers this repo actually depends on (CodeRabbit, Devin) are external SaaS bots: CodeRabbit rate-limits into silence and Devin's trial expired. RFC-0001 §2 rule 7 says the review team must be OUR own — the swarm is that team, and it is not firing.

`sdk/tests/live-kernel.test.ts` around the `live-manual-agent` case (line 288)
shows the whole shape: connect, `hello`, `workerAttach` with pins, receive
`step.dispatch`, act, complete. The protocol is already proven there.
The fix is one GitHub Actions workflow: on PR open/synchronize, write `.review-target`, invoke `agent-relay cloud run workflows/review-swarm.yaml`, poll for completion, `agent-relay cloud sync` the transcripts back, and post each as a PR comment with an aggregate marker.

## Files in scope

- `sdk/src/worker.ts` — new file, the worker implementation
- `sdk/src/index.ts` — export the worker
- `sdk/tests/live-kernel.test.ts` OR a new test file — add a test that runs a
real flow with an agent step end to end against a live `relayflowd`, with
this worker attached, and asserts the step reaches `done`.
1. `.github/workflows/review-swarm.yml` — NEW, the GitHub Actions workflow (the only file this tick should create)
2. `README.md` OR `docs/` — ADD one sentence documenting the required `RELAY_WORKSPACE_KEY` secret

## Definition of done (all required, with captured output)

1. **`.github/workflows/review-swarm.yml` exists and passes `actionlint` if installed, or `yamllint` otherwise**
- Trigger: `pull_request` events `opened`, `synchronize`, `reopened`
- Only run if `github.event.pull_request.user.login` is a drive-loop author (`kjgbot`, `miyaontherelay`) — do not review human PRs
- Concurrency group per PR so a second push cancels the first review
- Job steps:
1. checkout the PR's head at the merge commit
2. install `agent-relay` (curl the release, or use `mise install` if that's how this repo does it — check `.mise.toml` and `.env.example`)
3. `echo "$PR_NUMBER" > .review-target`
4. `agent-relay cloud run workflows/review-swarm.yaml` with `RELAY_WORKSPACE_KEY` from a repo secret; capture the runId
5. poll `agent-relay cloud status <runId> --json` every 30s until `status == completed` or 45 min elapse
6. `agent-relay cloud sync <runId>` to fetch the run's artifacts
7. read `ops/reviews/*-pr<N>-*.md` produced by the swarm; post each as a PR comment via `gh pr comment`
8. post one aggregate marker comment: `🎯 review-swarm: PASSED|FAILED (M:<v> H:<v> S:<v>)` — the aggregate step of review-swarm.yaml prints `SWARM_PASSED` or `SWARM_FAILED` on stdout; grep for that.
- Run validation and paste output:
```
actionlint .github/workflows/review-swarm.yml
```
OR if actionlint not available:
```
yamllint .github/workflows/review-swarm.yml
```

2. **The workflow's `jobs.review.if` correctly gates on drive-loop author only**
- Test the expression by hand: verify it evaluates true for kjgbot and false for khaliqgant
- Paste the test commands and output showing both cases

3. **A dry-run test that proves the shell logic works**
- A companion shell script `.github/workflows/scripts/swarm-post.sh` (or inline) that TAKES a runId as an argument and posts the comments
- Show it working against an EXISTING completed cloud run by running:
```
bash .github/workflows/scripts/swarm-post.sh <some-real-runId> <some-PR>
```
and quoting the posted comment URL

4. **`README.md` or `docs/` describes the required repo secret**
- `RELAY_WORKSPACE_KEY` and what it does — one sentence is enough

5. **`cd sdk && npm test` green (should be unaffected by this change)**
- Paste the final test summary line showing test counts

6. **EVERY new test confirmed to FAIL against current code**
- If any new tests are added, show the literal failing output quoted

7. **As your LAST action, run `git status --porcelain` and paste it**

## Definition of done

ALL of the following must hold:

1. The worker in `sdk/src/worker.ts`, exported from `sdk/src/index.ts`

2. A test that runs a real flow with an agent step end to end against a live
`relayflowd`, with this worker attached, and asserts the step reaches
`done`. `sdk/tests/live-kernel.test.ts` already starts a daemon — follow
that pattern.

3. **The worker must attach BEFORE the run starts.** A run that finds no worker
parks, and attaching afterwards does not re-drive it — `run.resume` is what
picks a parked run back up. That contract is pinned in the live-kernel
suite; do not fight it.

4. The worker must:
- attach for `agent` steps with the pins it holds
- on `step.dispatch`, run the step's declared `cli` as a subprocess
- report the result back through the existing protocol (`step.complete`, and
the failure path when the CLI exits nonzero)
- nothing speculative: no retries of its own, no scheduling, no LLM calls.
The kernel owns retry and lease policy — do not reimplement it.
## Explicitly OUT of scope

5. `cd sdk && npm test` must be green. Run it and paste the literal command and
output tail showing test counts.
- Actually configuring the `RELAY_WORKSPACE_KEY` secret in GitHub repo settings (requires human access)
- Running the workflow end-to-end against a real PR (requires the secret to be configured)
- Modifying `workflows/review-swarm.yaml` (already exists and works)
- Modifying the SDK agent worker (`sdk/src/worker.ts` is shipped per PR #53 — do NOT rewrite it)
- Changes to `ops/NEXT.md` validation, preflight, or kernel code
- Work on any gate other than gate 3

6. `cd kernel && sh ../ops/cargo.sh test` must be green. Run it and paste the
literal command and output tail showing test counts.
## Prerequisites this brief cannot satisfy — surface, don't try to fix

7. EVERY new test confirmed to FAIL against current code, with the literal
failing output quoted in the summary.
- **`RELAY_WORKSPACE_KEY` must exist as a GitHub Actions secret** on `AgentWorkforce/flows`. `gh secret list --repo AgentWorkforce/flows` will show if it's there. If it is missing, write ops/NEEDS_HUMAN.md saying so and STILL end with ASSESS_DONE — this is a human step (add the secret in repo settings; workspace key is on the laptop at `~/.agentworkforce/relay/cloud-auth.json`).
- **`agent-relay cloud run` invoked from GHA must reach the same cloud workspace as the laptop.** If the run fails with an auth error, DO NOT invent a workaround (a scoped token, a different endpoint); file it as a NEEDS_HUMAN and stop.

8. As your LAST action, run `git status --porcelain` and paste it.
## If you cannot finish

## Explicitly OUT of scope

- LLM steps — not in the gate 3 scope
- Retry logic in the worker — the kernel owns retry policy
- Scheduling or lease management — the kernel owns lease policy
- Optimizations, abstractions, or speculative features
- Changes to the kernel
- Changes to existing tests (except adding new test cases)
- Work on any gate other than gate 3
Say so and file what you learned. A working `.github/workflows/review-swarm.yml` that stalls at the auth step, plus a NEEDS_HUMAN naming the missing secret, is a complete deliverable — the runbook is the artifact.

## If blocked
Several drive runs execute in parallel, each pinned to a different gate. Work outside this target collides with a sibling run, so staying inside it is not a preference — it is what makes parallel execution safe.

If gate 3 is genuinely unreachable from the current state, write
ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not
silently substitute different work: a run that reports progress on the wrong
gate is worse than one that reports it is blocked.
If gate 3 is genuinely unreachable from the current state, write ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not silently substitute different work: a run that reports progress on the wrong gate is worse than one that reports it is blocked.
Loading