Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/review-swarm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
name: Review swarm

'on':
pull_request:
types: [opened, synchronize, reopened]

permissions:
contents: read
pull-requests: write

concurrency:
group: review-swarm-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
review:
if: >-
github.event.pull_request.user.login == 'kjgbot' ||
github.event.pull_request.user.login == 'miyaontherelay'
runs-on: ubuntu-latest
timeout-minutes: 50
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
steps:
- name: Check out PR merge commit
uses: actions/checkout@v4
with:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
fetch-depth: 0

- name: Install agent-relay
run: npm install --global agent-relay@11.8.7

- name: Set review target
run: echo "$PR_NUMBER" > .review-target

- name: Launch review swarm
id: launch
run: |
response=$(agent-relay cloud run workflows/review-swarm.yaml --json)
printf '%s\n' "$response"
run_id=$(printf '%s\n' "$response" | jq -er '.runId')
echo "run_id=$run_id" >> "$GITHUB_OUTPUT"

- name: Wait for review swarm
env:
RUN_ID: ${{ steps.launch.outputs.run_id }}
run: |
deadline=$((SECONDS + 2700))
while (( SECONDS < deadline )); do
response=$(agent-relay cloud status "$RUN_ID" --json)
printf '%s\n' "$response"
status=$(printf '%s\n' "$response" | jq -er '.status')
if [[ "$status" == completed ]]; then
exit 0
fi
sleep 30
done
echo "review-swarm timed out after 45 minutes" >&2
exit 1

- name: Sync and post reviews
env:
RUN_ID: ${{ steps.launch.outputs.run_id }}
run: bash .github/workflows/scripts/swarm-post.sh "$RUN_ID" "$PR_NUMBER"
57 changes: 57 additions & 0 deletions .github/workflows/scripts/swarm-post.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
set -euo pipefail

if [[ $# -ne 2 || ! $2 =~ ^[0-9]+$ ]]; then
echo "usage: $0 <run-id> <pr-number>" >&2
exit 64
fi

run_id=$1
pr_number=$2

agent-relay cloud sync "$run_id"
run_logs=$(agent-relay cloud logs "$run_id")
aggregate_output=$(awk '
/\[aggregate\] Output:/ { capture = 1; next }
capture && /\[workflow [^]]*\] \[[^]]+\]/ { exit }
capture { print }
' <<<"$run_logs")

declare -a labels=(maintainability history structure)
declare -a short_labels=(M H S)
declare -a marker_values=()

for index in "${!labels[@]}"; do
lens=${labels[$index]}
review=$(find ops/reviews -maxdepth 1 -type f \
-name "*-pr${pr_number}-${lens}.md" -print | sort | tail -n 1)
if [[ -z $review ]]; then
echo "no $lens review found for PR #$pr_number" >&2
exit 1
fi

verdict=$(grep -Eo 'REVIEW_(PASSED|FAILED)' "$review" | tail -n 1 || true)
case "$verdict" in
REVIEW_PASSED) marker_values+=("${short_labels[$index]}:PASSED") ;;
REVIEW_FAILED) marker_values+=("${short_labels[$index]}:FAILED") ;;
*)
echo "$review has no review verdict" >&2
exit 1
;;
esac

gh pr comment "$pr_number" --body-file "$review"
done

if grep -q 'SWARM_PASSED' <<<"$aggregate_output"; then
aggregate=PASSED
elif grep -q 'SWARM_FAILED' <<<"$aggregate_output"; then
aggregate=FAILED
else
echo "run $run_id has no aggregate swarm verdict" >&2
exit 1
fi

printf -v marker '🎯 review-swarm: %s (%s %s %s)' \
"$aggregate" "${marker_values[0]}" "${marker_values[1]}" "${marker_values[2]}"
gh pr comment "$pr_number" --body "$marker"
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,5 @@ Nine gates, in `docs/RFC-0001` §3. Gate 1 first: a relayflow can run — the he
ladder survives `kill -9` at every boundary.

Private while we build. YC 2026-09-15 runs on this base.

The repository must define a `RELAY_WORKSPACE_KEY` Actions secret so automated review swarms run in the canonical Agent Relay cloud workspace.
8 changes: 8 additions & 0 deletions ops/NEEDS_HUMAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Gate 3 prerequisites

This environment cannot authenticate to either service needed for the required live dry run:

- `gh auth status` reports that no GitHub host is authenticated, so it cannot verify the `RELAY_WORKSPACE_KEY` secret or post PR comments. Authenticate `gh` with access to `AgentWorkforce/flows`, then ensure the repository secret exists using the workspace key from `~/.agentworkforce/relay/cloud-auth.json` on the laptop.
- `agent-relay cloud status 404a8386-a129-40b0-9e5b-8ec9ad433038 --json` starts device authorization and does not complete. Authenticate this machine to the existing cloud workspace; do not substitute another endpoint or token.

The checkout metadata is also incomplete: `.git` points to `/home/daytona/.project-git`, which does not exist. Restore that Git directory before running the required final `git status --porcelain`.
131 changes: 66 additions & 65 deletions ops/NEXT.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,88 @@
# NEXT — work package for this tick

**Scope:** Build a minimal agent worker in the SDK. CODE task, SDK-side.
**Scope (quoted from gate 3 target):**
> Wire the review-swarm to fire on PR open via GitHub Actions + `agent-relay cloud run`. CODE task, `.github/workflows/`-side.

This run is pinned to **gate 3** and must not work on any other gate.

## Objective

Promote the throwaway worker the tests already build into a real SDK component
that can execute agent steps by running their declared CLI as a subprocess.
Create `.github/workflows/review-swarm.yml` that automatically invokes the existing `workflows/review-swarm.yaml` on PR open/synchronize/reopen events for drive-loop PRs, polls for completion, syncs artifacts back, and posts each review as a PR comment.

## Context

Nothing in this repo can execute an agent step. Searching for `workerAttach` /
`step.complete` finds only TESTS (`sdk/tests/live-kernel.test.ts`,
`journal-client.test.ts`, `journal-client-loopback.ts`) and the protocol
definitions. `sdk/src/cli/run.ts` only OBSERVES worker leases and waits for one
that never arrives.
`workflows/review-swarm.yaml` already exists — three model-diverse reviewers (claude/codex/opencode) that read a PR diff and produce three independent transcripts + one aggregate verdict. Today it only fires when a human writes `.review-target` and runs `agent-relay cloud run` by hand. It has run zero times against a drive PR on cloud.

The kernel's dispatch, lease and claim machinery is real and tested. The worker
side of the protocol is simply unimplemented, and that is what blocks gate 2
("a workload RUNS as a relayflow" — today a run can only be shown CREATED) and
gate 3 ("every claim/lease/retry served by the kernel").

`sdk/tests/live-kernel.test.ts` around the `live-manual-agent` case (line 288)
shows the whole shape: connect, `hello`, `workerAttach` with pins, receive
`step.dispatch`, act, complete. The protocol is already proven there.
RFC-0001 §2 rule 7 says the review team must be OUR own. The swarm is that team, but it is not firing automatically. External bots (CodeRabbit, Devin) are rate-limited or expired.

## Files in scope

- `sdk/src/worker.ts` — new file, the worker implementation
- `sdk/src/index.ts` — export the worker
- `sdk/tests/live-kernel.test.ts` OR a new test file — add a test that runs a
real flow with an agent step end to end against a live `relayflowd`, with
this worker attached, and asserts the step reaches `done`.
- `.github/workflows/review-swarm.yml` — the GitHub Actions workflow (NEW FILE)
- `.github/workflows/scripts/swarm-post.sh` — shell script to post comments (NEW FILE)
- `README.md` or a doc in `docs/` — add one sentence about the required `RELAY_WORKSPACE_KEY` secret

## Definition of done

ALL of the following must hold:

1. The worker in `sdk/src/worker.ts`, exported from `sdk/src/index.ts`

2. A test that runs a real flow with an agent step end to end against a live
`relayflowd`, with this worker attached, and asserts the step reaches
`done`. `sdk/tests/live-kernel.test.ts` already starts a daemon — follow
that pattern.

3. **The worker must attach BEFORE the run starts.** A run that finds no worker
parks, and attaching afterwards does not re-drive it — `run.resume` is what
picks a parked run back up. That contract is pinned in the live-kernel
suite; do not fight it.

4. The worker must:
- attach for `agent` steps with the pins it holds
- on `step.dispatch`, run the step's declared `cli` as a subprocess
- report the result back through the existing protocol (`step.complete`, and
the failure path when the CLI exits nonzero)
- nothing speculative: no retries of its own, no scheduling, no LLM calls.
The kernel owns retry and lease policy — do not reimplement it.

5. `cd sdk && npm test` must be green. Run it and paste the literal command and
output tail showing test counts.

6. `cd kernel && sh ../ops/cargo.sh test` must be green. Run it and paste the
literal command and output tail showing test counts.

7. EVERY new test confirmed to FAIL against current code, with the literal
failing output quoted in the summary.

8. As your LAST action, run `git status --porcelain` and paste it.

## Explicitly OUT of scope

- LLM steps — not in the gate 3 scope
- Retry logic in the worker — the kernel owns retry policy
- Scheduling or lease management — the kernel owns lease policy
- Optimizations, abstractions, or speculative features
- Changes to the kernel
- Changes to existing tests (except adding new test cases)
- Work on any gate other than gate 3
1. **`.github/workflows/review-swarm.yml` exists and is valid**
- Passes `actionlint` if installed, or `yamllint` otherwise
- Triggers on: `pull_request` events `opened`, `synchronize`, `reopened`
- Only runs if `github.event.pull_request.user.login` is `kjgbot` or `miyaontherelay`
- Has concurrency group per PR (cancels earlier reviews on new push)

2. **The workflow job has these steps:**
- Checkout PR head at merge commit
- Install `agent-relay` (check `.env.example` for method — curl the release or use mise if configured)
- Write PR number to `.review-target`: `echo "$PR_NUMBER" > .review-target`
- Run `agent-relay cloud run workflows/review-swarm.yaml` with `RELAY_WORKSPACE_KEY` from repo secret; capture runId
- Poll `agent-relay cloud status <runId> --json` every 30s until `status == completed` or 45 min timeout
- Sync artifacts: `agent-relay cloud sync <runId>`
- Read `ops/reviews/*-pr<N>-*.md` produced by swarm; post each as a PR comment via `gh pr comment`
- Post aggregate marker: `🎯 review-swarm: PASSED|FAILED (M:<v> H:<v> S:<v>)` — grep aggregate step stdout for `SWARM_PASSED` or `SWARM_FAILED`

3. **`.github/workflows/scripts/swarm-post.sh` exists** (or inline equivalent)
- Takes runId and PR number as arguments
- Posts the review comments correctly
- Can be tested standalone against an existing completed cloud run

4. **Documentation updated**
- README.md or docs/ contains one sentence describing `RELAY_WORKSPACE_KEY` secret requirement and what it does

5. **The author-gate expression is correct**
- Test the expression manually: verify it evaluates `true` for `kjgbot` and `false` for `khaliqgant`
- Literal passing command and output MUST be quoted

6. **Dry-run test shown**
- Run `.github/workflows/scripts/swarm-post.sh <real-runId> <PR>` against a real completed cloud run
- Quote the literal command and the posted comment URL

7. **As LAST action before commit, run and paste:**
```
git status --porcelain
```

## What is EXPLICITLY OUT OF SCOPE for this tick

- Do NOT touch `workflows/review-swarm.yaml` — it already exists and works
- Do NOT touch `sdk/src/worker.ts` (91 lines, shipped in PR #53) — it is the delivered worker
- Do NOT touch `kernel/relayflowd/src/server/tests.rs` or `server.rs` (gate-1 race test, PR #48)
- Do NOT touch preflight (`sdk/src/work-package-validator.ts`, PR #50)
- Do NOT create or fix the `RELAY_WORKSPACE_KEY` secret if it's missing — file ops/NEEDS_HUMAN.md
- Do NOT create a workaround for auth errors — file ops/NEEDS_HUMAN.md
- SDK tests currently fail due to missing exec bits (known sandbox fault per ops/STATE.md:186-192) — this is NOT a blocker for gate 3

## Prerequisites that may require NEEDS_HUMAN

1. **`RELAY_WORKSPACE_KEY` must exist as a GitHub Actions secret** at `AgentWorkforce/flows`
- `gh secret list --repo AgentWorkforce/flows` shows if it's there (gh not available in this environment)
- If missing during implementation: write ops/NEEDS_HUMAN.md stating the secret needs to be added at repo settings using the workspace key from `~/.agentworkforce/relay/cloud-auth.json` on the laptop

2. **`agent-relay cloud run` must reach the same workspace from GHA**
- If auth fails: write ops/NEEDS_HUMAN.md, do NOT invent workarounds (no scoped token, no different endpoint)

## If blocked

If gate 3 is genuinely unreachable from the current state, write
ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not
silently substitute different work: a run that reports progress on the wrong
gate is worse than one that reports it is blocked.
A working `.github/workflows/review-swarm.yml` that stalls at the auth step, plus a NEEDS_HUMAN naming the missing secret, is a complete deliverable — the runbook is the artifact.

If gate 3 is genuinely unreachable, write ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not silently substitute different work.
Loading