Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .github/workflows/review-swarm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: Review swarm

on:
# Runs main's workflow definition, so the reviewed head cannot alter its judge.
pull_request_target:
types: [opened, synchronize, reopened, ready_for_review]

permissions:
contents: read
pull-requests: write

# Ordering invariant: job 75m > poll 65m > swarm 60m.
jobs:
review:
runs-on: ubuntu-latest
timeout-minutes: 75
steps:
- name: Check out immutable gate from main
uses: actions/checkout@v4
with:
ref: main
path: gate

- name: Check out pull request head
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
path: target

- name: Validate cloud authentication
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
if [ -z "$RELAY_WORKSPACE_KEY" ]; then
echo "RELAY_WORKSPACE_KEY secret not configured; see README §Review swarm secret" >&2
exit 1
fi

- name: Install Agent Relay
run: npm install --global agent-relay

- name: Prepare pull request evidence
working-directory: gate
env:
GH_TOKEN: ${{ github.token }}
run: .github/workflows/scripts/swarm-prepare.sh '${{ github.event.pull_request.number }}'

- name: Launch immutable review swarm
id: launch
working-directory: gate
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
output=$(agent-relay cloud run workflows/review-swarm.yaml --sync-code --json)
printf '%s\n' "$output"
run_id=$(printf '%s\n' "$output" | python3 -c 'import json,sys; print(json.load(sys.stdin)["runId"])')
echo "run_id=$run_id" >> "$GITHUB_OUTPUT"

# Ordering invariant: poll 3900s (65m) > swarm 3600s (60m).
- name: Wait for terminal status
id: wait
if: steps.launch.outputs.run_id != ''
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
deadline=$((SECONDS + 3900))
status=timed_out
while [ "$SECONDS" -lt "$deadline" ]; do
if ! response=$(agent-relay cloud status '${{ steps.launch.outputs.run_id }}'); then
status=status_error
break
fi
status=$(printf '%s\n' "$response" | sed -n 's/^Status:[[:space:]]*//p' | head -n 1)
case "$status" in
completed|failed|cancelled|interrupted) break ;;
esac
sleep 15
done
echo "swarm_status=${status:-unknown}" >> "$GITHUB_OUTPUT"
exit 0

- name: Sync and post review evidence
if: always() && steps.launch.outputs.run_id != ''
working-directory: gate
env:
GH_TOKEN: ${{ github.token }}
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
sync_started=$(date +%s)
agent-relay cloud sync '${{ steps.launch.outputs.run_id }}' --dir .
.github/workflows/scripts/swarm-post.sh \
'${{ github.event.pull_request.number }}' \
'${{ steps.launch.outputs.run_id }}' . "$sync_started"

- name: Enforce successful terminal status
if: steps.wait.outputs.swarm_status != 'completed'
run: |
echo "review swarm did not complete: ${{ steps.wait.outputs.swarm_status }}" >&2
exit 1
46 changes: 46 additions & 0 deletions .github/workflows/scripts/swarm-post.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
set -euo pipefail

pr=${1:?usage: swarm-post.sh PR_NUMBER RUN_ID SYNC_DIR SYNC_STARTED_EPOCH}
run_id=${2:?usage: swarm-post.sh PR_NUMBER RUN_ID SYNC_DIR SYNC_STARTED_EPOCH}
sync_dir=${3:?usage: swarm-post.sh PR_NUMBER RUN_ID SYNC_DIR SYNC_STARTED_EPOCH}
sync_started=${4:?usage: swarm-post.sh PR_NUMBER RUN_ID SYNC_DIR SYNC_STARTED_EPOCH}
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$script_dir/swarm-verdict.sh"

review_dir="$sync_dir/ops/reviews"
overall=FAILED
if swarm_evaluate "$review_dir" "$pr" "$sync_started"; then overall=PASSED; fi

upsert_comment() {
local anchor=$1 body=$2 id
id=$(gh api --paginate "repos/{owner}/{repo}/issues/$pr/comments" \
--jq ".[] | select(.body | contains(\"$anchor\")) | .id" | head -n 1)
if [ -n "$id" ]; then
gh api --method PATCH "repos/{owner}/{repo}/issues/comments/$id" -f body="$body" >/dev/null
else
gh api --method POST "repos/{owner}/{repo}/issues/$pr/comments" -f body="$body" >/dev/null
fi
}

while IFS='|' read -r lens verdict path; do
[ -n "$lens" ] || continue
anchor="<!-- swarm-lens: $lens -->"
if [ -n "$path" ] && [ -f "$path" ]; then
transcript=$(cat "$path")
else
transcript="No transcript was produced for this lens."
fi
upsert_comment "$anchor" "$anchor
### Review swarm: $lens — $verdict

$transcript"
done <<< "$SWARM_RESULTS"

marker='<!-- review-swarm -->'
upsert_comment "$marker" "$marker
### Review swarm: $overall

Cloud run: \`$run_id\`. All three lenses must end in \`REVIEW_PASSED\`; missing, unclear, failed, or stale transcripts fail closed."

[ "$overall" = PASSED ]
11 changes: 11 additions & 0 deletions .github/workflows/scripts/swarm-prepare.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail

pr=${1:?usage: swarm-prepare.sh PR_NUMBER}
case "$pr" in (*[!0-9]*|'') echo "invalid PR number: $pr" >&2; exit 2;; esac

mkdir -p .review-target
printf '%s\n' "$pr" > .review-target/pr-number
gh pr diff "$pr" > .review-target/pr.diff
gh pr view "$pr" --json headRefName,headRefOid,title,url > .review-target/pr.json
git add -f .review-target/pr-number .review-target/pr.diff .review-target/pr.json
54 changes: 54 additions & 0 deletions .github/workflows/scripts/swarm-verdict.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env bash

swarm_latest_transcript() {
local review_dir=$1 pr=$2 lens=$3
find "$review_dir" -maxdepth 1 -type f -name "*-pr${pr}-${lens}.md" -printf '%f\n' 2>/dev/null |
LC_ALL=C sort |
tail -n 1
}

swarm_transcript_verdict() {
awk 'NF { line=$0 } END {
n=split(line, fields, /[[:space:]]+/)
token=fields[n]
if (token == "REVIEW_PASSED" || token == "REVIEW_FAILED") print token
else print "UNCLEAR"
}' "$1"
}

swarm_evaluate() {
local review_dir=$1 pr=$2 fresh_since=${3:-} lens name path verdict overall=0
SWARM_RESULTS=''
for lens in maintainability history structure; do
name=$(swarm_latest_transcript "$review_dir" "$pr" "$lens")
if [ -z "$name" ]; then
verdict=MISSING
path=''
else
path="$review_dir/$name"
verdict=$(swarm_transcript_verdict "$path")
if [ -n "$fresh_since" ] && [ "$(stat -c %Y "$path")" -lt "$fresh_since" ]; then
verdict=STALE
fi
fi
[ "$verdict" = REVIEW_PASSED ] || overall=1
SWARM_RESULTS="${SWARM_RESULTS}${lens}|${verdict}|${path}"$'\n'
done
export SWARM_RESULTS
return "$overall"
}

if [ "${BASH_SOURCE[0]}" = "$0" ]; then
set -euo pipefail
review_dir=${1:?usage: swarm-verdict.sh REVIEW_DIR PR_NUMBER [FRESH_SINCE_EPOCH]}
pr=${2:?usage: swarm-verdict.sh REVIEW_DIR PR_NUMBER [FRESH_SINCE_EPOCH]}
fresh_since=${3:-}
if swarm_evaluate "$review_dir" "$pr" "$fresh_since"; then
printf '%s' "$SWARM_RESULTS"
echo SWARM_PASSED
else
printf '%s' "$SWARM_RESULTS"
echo SWARM_FAILED
exit 1
fi
fi
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ dist/
.env
.agentworkforce/
.cargo-home/
.review-target

# Toolchains materialize inside the workspace in a cloud sandbox and must never
# be committed or delivered. Run f18ec684's patch carried .rustup-home/ files;
# ops/deliver-run.sh scrubs them too, but ignoring them is the durable fix.
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,11 @@ Nine gates, in `docs/RFC-0001` §3. Gate 1 first: a relayflow can run — the he
ladder survives `kill -9` at every boundary.

Private while we build. YC 2026-09-15 runs on this base.

## Review swarm secret

The cloud review swarm requires a repository Actions secret named
`RELAY_WORKSPACE_KEY`. Set it to a non-empty workspace key in the repository's
Settings → Secrets and variables → Actions page. The workflow validates the
secret before starting a cloud run and fails immediately when it is absent;
it never falls back to interactive device login.
130 changes: 67 additions & 63 deletions ops/NEXT.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,91 @@
# NEXT — work package for this tick

**Scope:** Build a minimal agent worker in the SDK. CODE task, SDK-side.
**Scope (from ops/TARGET.md):** **Track D: Cloud review-swarm redesign** — build `.github/workflows/review-swarm.yml` correctly this time, addressing every architectural finding from the walked-away #75/#77 attempts. Parallel to Track A (hn-monitor); different territory (`.github/` + `workflows/` — no overlap with `sdk/` work).

This run is pinned to **gate 3** and must not work on any other gate.

## Objective

Promote the throwaway worker the tests already build into a real SDK component
that can execute agent steps by running their declared CLI as a subprocess.
Build the cloud-executable GitHub Actions review swarm workflow that enforces RFC-0001 §2 rule 7 ("every PR met by a review swarm — our own, not a vendor's"). The local `~/AgentWorkforce/review-swarm-loop.sh` (chief-owned shell) is currently the only enforcement. It works, but lives on the operator's laptop. When that session ends, so does swarm enforcement. The cloud version must exist for gate 3+ work to be trustworthy.

## Context

Nothing in this repo can execute an agent step. Searching for `workerAttach` /
`step.complete` finds only TESTS (`sdk/tests/live-kernel.test.ts`,
`journal-client.test.ts`, `journal-client-loopback.ts`) and the protocol
definitions. `sdk/src/cli/run.ts` only OBSERVES worker leases and waits for one
that never arrives.

The kernel's dispatch, lease and claim machinery is real and tested. The worker
side of the protocol is simply unimplemented, and that is what blocks gate 2
("a workload RUNS as a relayflow" — today a run can only be shown CREATED) and
gate 3 ("every claim/lease/retry served by the kernel").

`sdk/tests/live-kernel.test.ts` around the `live-manual-agent` case (line 288)
shows the whole shape: connect, `hello`, `workerAttach` with pins, receive
`step.dispatch`, act, complete. The protocol is already proven there.
Prior attempts (#75, #77) each shipped real code but were rejected on progressively deeper findings that were never resolved.

## Files in scope

- `sdk/src/worker.ts` — new file, the worker implementation
- `sdk/src/index.ts` — export the worker
- `sdk/tests/live-kernel.test.ts` OR a new test file — add a test that runs a
real flow with an agent step end to end against a live `relayflowd`, with
this worker attached, and asserts the step reaches `done`.

## Definition of done
- `.github/workflows/review-swarm.yml` — GHA trigger (create new directory and file)
- `.github/workflows/scripts/swarm-prepare.sh` — launcher-side PR fetcher (create new)
- `.github/workflows/scripts/swarm-post.sh` — sync + verdict + post script (create new)
- `.github/workflows/scripts/swarm-verdict.sh` — shared verdict logic (create new)
- `workflows/review-swarm.yaml` — aggregate step refactored to use shared verdict logic
- `.gitignore` — drop the `.review-target` mask (currently line 10)
- `README.md` — document `RELAY_WORKSPACE_KEY` secret setup

ALL of the following must hold:
## Requirements (all 9 non-negotiable, from TARGET.md)

1. The worker in `sdk/src/worker.ts`, exported from `sdk/src/index.ts`
### 1. Immutable gate — the reviewed PR must NOT control its own judge
`.github/workflows/review-swarm.yml` must checkout `main`'s copy of `workflows/review-swarm.yaml` + `.github/workflows/scripts/swarm-post.sh` SEPARATELY from the PR head. Use two `actions/checkout@v4` steps with different `path:` values. Launch the swarm using main's gate files, not the PR's. This is RFC-0001 settled decision #6.

2. A test that runs a real flow with an agent step end to end against a live
`relayflowd`, with this worker attached, and asserts the step reaches
`done`. `sdk/tests/live-kernel.test.ts` already starts a daemon — follow
that pattern.
### 2. Unified verdict-extraction logic (one source of truth)
`workflows/review-swarm.yaml`'s aggregate step AND `.github/workflows/scripts/swarm-post.sh` currently duplicate verdict logic and can disagree. Refactor: aggregate logic lives in ONE place — either a shared bash helper file both source (`scripts/swarm-verdict.sh`) OR the yaml aggregate step becomes trivial and swarm-post.sh does all extraction. Rules that must apply uniformly:
- Transcript selection sorts by FILENAME (`YYYYMMDD-HHMM` prefix), not mtime (`b2535aa` fixed this once)
- Verdict is the LAST non-empty line's token, not a whole-file grep (`f59d9cd` fixed this once)
- `overall = ALL lenses PASSED, else FAILED` — fail-closed on MISSING/UNCLEAR/FAILED

3. **The worker must attach BEFORE the run starts.** A run that finds no worker
parks, and attaching afterwards does not re-drive it — `run.resume` is what
picks a parked run back up. That contract is pinned in the live-kernel
suite; do not fight it.
### 3. Auth secret validation fail-fast
Add a preflight step that validates `RELAY_WORKSPACE_KEY` is set and non-empty BEFORE launching the cloud run. If missing, fail the job with a clear message ("secret not configured; see README §<X>"). Do NOT proceed to a 10-min interactive fallback (per RFC covenant 2 real event: `Device login expired before it was approved` was seen on run 33364011379).

4. The worker must:
- attach for `agent` steps with the pins it holds
- on `step.dispatch`, run the step's declared `cli` as a subprocess
- report the result back through the existing protocol (`step.complete`, and
the failure path when the CLI exits nonzero)
- nothing speculative: no retries of its own, no scheduling, no LLM calls.
The kernel owns retry and lease policy — do not reimplement it.
### 4. Sticky marker + sticky transcripts (edit-in-place across pushes)
The marker comment uses a hidden HTML anchor and edits in place. So MUST the three lens transcript comments. A PR with 5 pushes should end with 1 marker + 3 transcripts (edited to latest), NOT 5 markers + 15 transcripts. Use `<!-- swarm-lens: <lens> -->` anchors, find-by-anchor before posting.

5. `cd sdk && npm test` must be green. Run it and paste the literal command and
output tail showing test counts.
### 5. Every PR gets reviewed (RFC-0001 §2 rule 7)
NO author whitelist. If a rollout-scoped filter is needed later, document it as a temporary exception AND file the RFC amendment. Default: all PRs.

6. `cd kernel && sh ../ops/cargo.sh test` must be green. Run it and paste the
literal command and output tail showing test counts.
### 6. Cloud sandbox has no `gh` auth — fetch on launching host
GHA runner has `gh` auth. Cloud sandbox does not. The workflow must fetch PR diff + metadata on the GHA runner via `gh pr diff/view`, stage them into `.review-target/{pr-number,pr.diff,pr.json}`, `git add -f` (the `.gitignore` mask on `.review-target` must be dropped too — it silently drops the file from `git ls-files` per PR #77's audit). Then `agent-relay cloud run` uploads the working tree.

7. EVERY new test confirmed to FAIL against current code, with the literal
failing output quoted in the summary.
### 7. Job timeout > poll deadline > swarm timeoutMs (documented invariant)
- `workflows/review-swarm.yaml` `timeoutMs: 3600000` (60 min)
- Wait step poll deadline: 3900s (65 min)
- Job `timeout-minutes: 75` (65 + 10 min for install/checkout/post)
Add a comment where each value lives naming the ordering invariant.

8. As your LAST action, run `git status --porcelain` and paste it.
### 8. Wait step must record terminal status as output; post step runs on always()
A rejecting swarm's transcripts + marker MUST reach the PR. Structure:
```
wait step: records $swarm_status output, always exits 0
post step: if: always() && steps.launch.outputs.run_id != ''
fail step: if: steps.wait.outputs.swarm_status != 'completed' # exit 1 gates merge
```

## Explicitly OUT of scope
### 9. Transcript-to-run-id binding
Sub-guard: aggregate rejects a transcript that doesn't belong to this run (no way to prove without instrumenting review-swarm.yaml to write run-id into transcripts). For now: require ALL THREE transcripts newly-produced in THIS sync; if any transcript's file mtime is older than the sync started, reject as stale.

- LLM steps — not in the gate 3 scope
- Retry logic in the worker — the kernel owns retry policy
- Scheduling or lease management — the kernel owns lease policy
- Optimizations, abstractions, or speculative features
- Changes to the kernel
- Changes to existing tests (except adding new test cases)
- Work on any gate other than gate 3
## Definition of done

## If blocked
ALL of the following must hold:

If gate 3 is genuinely unreachable from the current state, write
ops/NEEDS_HUMAN.md saying exactly why and still end with ASSESS_DONE. Do not
silently substitute different work: a run that reports progress on the wrong
gate is worse than one that reports it is blocked.
1. All files parse:
- `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/review-swarm.yml'))"`
- `python3 -c "import yaml; yaml.safe_load(open('workflows/review-swarm.yaml'))"`
- `bash -n .github/workflows/scripts/swarm-prepare.sh`
- `bash -n .github/workflows/scripts/swarm-post.sh`
- `bash -n .github/workflows/scripts/swarm-verdict.sh`
2. Aggregate verdict logic exists in ONE file, both callers use it
3. Author whitelist absent (no `if: github.event.pull_request.user.login == ...`)
4. Immutable gate: two checkout steps with different paths
5. PR body explicitly documents each of the 9 requirements above and shows where each is satisfied
6. `cd kernel && sh ../ops/cargo.sh test --workspace` green (should be unaffected)
7. As your LAST action, `git status --porcelain`

## Out of scope

- `sdk/` (Track A owns that)
- `kernel/` (gate 1 done, no changes)
- `ops/*` (chief owns briefs and state)
- Any GHA workflow other than review-swarm.yml
- Actually TESTING the workflow in CI (requires `RELAY_WORKSPACE_KEY` secret set which is a human step; the DoD is the workflow being correct, not proven live)
- Merged and closed PRs (#42, #45, #48, #50, #53, #69) — do not redo

## Status

Gate 3 is reachable. No blockers identified. No open PRs requiring fixes. No unsatisfied directives.
Loading