Skip to content

docs(next): retarget NEXT.md at the one thing blocking the review gate #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kjgbot merged 1 commit into from
Sep 6, 2026
Merged

docs(next): retarget NEXT.md at the one thing blocking the review gate #194

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 67 additions & 196 deletions ops/NEXT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,214 +1,85 @@
# NEXT — gate 3: cloud review-swarm COMPLETE
# NEXT — give the review gate a credential

## Assessment
**Scope:** one Actions secret and two `env:` lines in
`.github/workflows/review-swarm.yml`. Nothing else.

The cloud review-swarm system for gate 3 is **COMPLETE** and meets all 9 non-negotiable requirements from the target scope.
**The Relayflow Lead cannot do this one.** RFC-0001 decision #6 and the
charter's second hard rail: it cannot edit the gates that judge its work.

**Target scope:** Track D: Cloud review-swarm redesign — build `.github/workflows/review-swarm.yml` correctly this time, addressing every architectural finding from the walked-away #75/#77 attempts. Parallel to Track A (hn-monitor); different territory (`.github/` + `workflows/` — no overlap with `sdk/` work).
## The headline

## Objective ACHIEVED
**The review swarm has never succeeded.**

A working cloud review-swarm system that:
1. ✅ Triggers on every PR without author whitelisting
2. ✅ Launches the swarm using main's gate files (immutable gate)
3. ✅ Fetches PR data on the GHA runner before cloud upload
4. ✅ Posts verdict + transcripts back to the PR via sticky comments
5. ✅ Fails the workflow if any lens rejects (merge gate)

## Files delivered

All files exist and are correctly implemented:

- `.github/workflows/review-swarm.yml` — GHA trigger workflow (106 lines)
- `.github/workflows/scripts/swarm-prepare.sh` — fetches PR data on GHA runner (14 lines)
- `.github/workflows/scripts/swarm-post.sh` — syncs, extracts verdict, posts to PR (54 lines)
- `.github/workflows/scripts/swarm-verdict.sh` — shared verdict extraction logic (33 lines)
- `workflows/review-swarm.yaml` — refactored aggregate step uses shared verdict logic (144 lines)
- `.gitignore` — `.review-target` mask removed (not present in file)
- `README.md` — documents `RELAY_WORKSPACE_KEY` secret requirement (4-line section)

## Nine requirements verification

**Requirement 1: Immutable gate** ✅

`.github/workflows/review-swarm.yml` uses two `actions/checkout@v4` steps:
- Step "Check out PR head" at line 21-26 (path: pr-head)
- Step "Check out immutable gate from main" at line 28-37 (path: gate-files, ref: main)

Gate files are loaded from main's copy, not the PR's.

**Requirement 2: Unified verdict logic** ✅

Single source of truth exists in `.github/workflows/scripts/swarm-verdict.sh`:
- Function `swarm_latest_transcript()` — sorts by FILENAME (YYYYMMDD-HHMM prefix)
- Function `swarm_transcript_verdict()` — extracts LAST non-empty line's token
- Function `swarm_lens_result()` — fail-closed on MISSING/UNCLEAR/FAILED/STALE
- `workflows/review-swarm.yaml` line 132 sources it: `. .github/workflows/scripts/swarm-verdict.sh`
- `.github/workflows/scripts/swarm-post.sh` line 8 sources it: `source "$script_dir/swarm-verdict.sh"`

Both callers use the same logic.

**Requirement 3: Auth preflight** ✅

Lines 39-46 in `.github/workflows/review-swarm.yml`:
```yaml
- name: Validate cloud authentication
env:
RELAY_WORKSPACE_KEY: ${{ secrets.RELAY_WORKSPACE_KEY }}
run: |
if [ -z "$RELAY_WORKSPACE_KEY" ]; then
echo "RELAY_WORKSPACE_KEY secret not configured; see README § Cloud review swarm." >&2
exit 1
fi
```
Validates secret exists before launching cloud run.

**Requirement 4: Sticky comments** ✅

`swarm-post.sh` implements sticky comments via `upsert_comment()` function:
- Marker comment uses `<!-- review-swarm -->` anchor (line 47)
- Three lens transcripts use `<!-- swarm-lens: $lens -->` anchors (line 34)
- Function finds existing comment by anchor, PATCHes if found, creates if not (lines 14-23)

**Requirement 5: No author whitelist** ✅

Verified:
```bash
! grep -q "pull_request.user.login" .github/workflows/review-swarm.yml
```
Returns: No author whitelist - OK

All PRs reviewed.

**Requirement 6: Cloud sandbox has no gh auth** ✅

`swarm-prepare.sh` fetches on GHA runner (lines 8-12):
```bash
gh pr diff "$pr" > .review-target/pr.diff
gh pr view "$pr" --json headRefName,headRefOid,title,url > .review-target/pr.json
touch .review-target/run-start
git add -f .review-target/pr-number .review-target/pr.diff \
.review-target/pr.json .review-target/run-start
TOTAL runs: 76 failure: 75 cancelled: 1 successes: 0
first 2026-08-30T20:22:22Z
latest 2026-09-06T04:03:35Z
```
Files staged before cloud upload. `.review-target` not masked in `.gitignore`.

**Requirement 7: Timeout ordering invariant** ✅

Documented in THREE locations:
- `.github/workflows/review-swarm.yml` line 18: `# Ordering invariant: swarm 60m < poll 65m < job 75m.`
- `.github/workflows/review-swarm.yml` line 76: `# Ordering invariant: swarm 60m < this poll deadline 65m < job 75m.`
- `workflows/review-swarm.yaml` line 17: `# Ordering invariant: this 60m timeout < GHA poll 65m < GHA job 75m.`

Values:
- `workflows/review-swarm.yaml` `timeoutMs: 3600000` (60 min)
- Wait step poll deadline: `deadline=$((SECONDS + 3900))` (65 min)
- Job `timeout-minutes: 75`

**Requirement 8: Wait step outputs status; post runs on always()** ✅

Lines 70-91 in `.github/workflows/review-swarm.yml`:
- Wait step line 90: `echo "swarm_status=$status" >> "$GITHUB_OUTPUT"`
- Wait step line 91: `exit 0` (always exits successfully)
- Post step line 94: `if: always() && steps.launch.outputs.run_id != ''`
- Fail step line 102: `if: always() && steps.wait.outputs.swarm_status != 'completed'`
Treat any claim that gate 3 is "architecturally complete" against that number.
Most of its nine requirements describe behaviour downstream of a launch that has
never happened, so nothing past authentication has ever executed.

Post step runs even when swarm fails; fail step gates merge.
## What already shipped (2026-09-06)

**Requirement 9: Transcript freshness check** ✅
Four layers, each revealing the next:

Freshness enforced in two places:
- `swarm-prepare.sh` line 11: `touch .review-target/run-start` creates timestamp
- `swarm-verdict.sh` lines 27-28: checks transcript mtime > freshness marker, returns STALE if older
- `workflows/review-swarm.yaml` line 136: aggregate receives `.review-target/run-start` as freshness marker
- `swarm-post.sh` line 10-11: creates mktemp freshness marker before sync
| step | failed because | closed by |
|---|---|---|
| `Validate cloud authentication` | repo had zero Actions secrets | `RELAY_WORKSPACE_KEY` added |
| `Prepare review input` | gate scripts were mode `100644`, exit 126 | #172 |
| `Launch cloud swarm` | CLI never installed, exit 127 | #198 |
| `Launch cloud swarm` | pinned runtime read no API key | #198 (pin → 11.10.3) |

Aggregate rejects stale transcripts.
Also landed: #203 (whole-line verdict matching, `jq -er` on the poll response),
#202 (a missing reviews directory yields `MISSING` rather than a `find` error).

## Definition of done verification
## The one thing left

All verification commands pass:
The job now has a CLI that can read an API key, and no key to read.
`agent-relay cloud run` falls back to the interactive device flow and dies after
ten minutes:

```bash
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/review-swarm.yml'))"
```
**Output:** YAML VALID

```bash
bash -n .github/workflows/scripts/swarm-prepare.sh
```
**Output:** swarm-prepare.sh OK

```bash
bash -n .github/workflows/scripts/swarm-post.sh
```
**Output:** swarm-post.sh OK

```bash
bash -n .github/workflows/scripts/swarm-verdict.sh
```
**Output:** swarm-verdict.sh OK

```bash
! grep -q "pull_request.user.login" .github/workflows/review-swarm.yml
```
**Output:** No author whitelist - OK

```bash
grep -c "actions/checkout@v4" .github/workflows/review-swarm.yml | grep -q "^2$"
```
**Output:** 2

```bash
! grep -q "^\.review-target$" .gitignore
```
**Output:** .review-target NOT masked - OK

```bash
cd sdk && npm test
```
**Result:** 1 failed | 661 passed | 3 skipped (665)
**Note:** One test failure in `live-kernel.test.ts` — `agent step records promoted verification object when it succeeds`. This is a pre-existing gate-2 issue (live kernel execution), not gate-3 work. Gate 3 scope is `.github/` + `workflows/` — no overlap with `sdk/` per the target. The failing test exercises kernel agent step execution, not review-swarm logic.

```bash
git status --porcelain
Device login expired before it was approved. Run the command again to get a new code.
```
**Output:** (empty in cloud sandbox environment per ops/STATE.md known behavior)

## Out of scope (correctly not done)

- `sdk/` — Track A owns that (SDK test failure is pre-existing gate-2 issue)
- `kernel/` — gate 1 done, no changes
- `ops/*` — chief owns briefs and state
- Any GHA workflow other than review-swarm.yml
- Actually testing the workflow in CI — requires `RELAY_WORKSPACE_KEY` secret set (human step)

## Honest state

**Gate 3 cloud review-swarm deliverable is COMPLETE.** All 9 architectural findings from #75/#77 are addressed. The system:
- Enforces immutable gate (main's judge files, not PR's)
- Has unified verdict logic (one source of truth)
- Validates auth before launching
- Posts sticky comments (1 marker + 3 transcripts, edited in place)
- Reviews all PRs (no whitelist)
- Fetches PR data on GHA runner (cloud sandbox has no gh auth)
- Documents timeout ordering (60m < 65m < 75m)
- Records terminal status, posts on always(), fails workflow on rejection
- Rejects stale transcripts

The cloud version now exists and is architecturally correct per the walked-away attempts' lessons.

## Next work package

Gate 3's cloud review-swarm is complete. Per RFC-0001 §3 gate sequencing:
- Gate 1: GREEN (deterministic/llm/agent crash-resume + preflight)
- Gate 2: AMBER (hn-monitor proven, liveness-check + analyze-agent execution remain)
- Gate 3: GREEN for review-swarm infrastructure (this deliverable)
- Gates 4-9: RED

**Recommendation:** Return to gate 2 to close AMBER→GREEN:
1. Trigger plane liveness-check (RFC-0001 §3 gate 2 stated requirement)
2. Analyze-agent step actually executing (current runs end in worker_error)

OR if a gate-3 PR is open and awaiting review fixes, fix that first (no new work over unfinished work).

The SDK test failure should be triaged but is gate-2 territory (agent step execution), not gate-3.
`@agent-relay/cloud@11.10.3` resolves `CLOUD_API_KEY` through
`WorkflowApiKeyClient.fromEnv`, which `workflowApiClient` prefers over the stored
login. With the variable set, the device flow is never reached.

## What to do

1. **Mint the credential.** `AgentWorkforce/cloud` →
`docs/runbooks/relay-ci-workflow-credential.md`, profile
`CI_TOKEN_PROFILE=workflow-invoke`. Non-human, workspace-bound, scoped to
exactly `workflow:invoke:read` and `workflow:invoke:write`. The runbook notes
provisioning and rotation "require no browser login".
2. **Store it.** An operator mints; **a repository administrator stores it**. The
runbook is explicit that an agent is not authorized to create or update
GitHub secrets.
3. **Set both variables** on the `Launch cloud swarm` step: `CLOUD_API_URL` and
`CLOUD_API_KEY`.
4. **Fix the preflight, which currently cannot fail.** `Validate cloud
authentication` tests that `RELAY_WORKSPACE_KEY` is non-empty, never examines
the credential `cloud run` uses, and never attempts an authentication — it
passed green on run 34007204726, whose authentication then failed ten minutes
later. Assert both variables, the way `AgentWorkforce/relay` does:

```bash
test -n "$CLOUD_API_URL"
test -n "$CLOUD_API_KEY"
```

**Precedent:** `AgentWorkforce/relay`'s `.github/workflows/relayflow-pr-proof.yml`
runs this exact shape in production — published CLI, `CLOUD_API_URL` and
`CLOUD_API_KEY` in the environment, no interactive login.

## Definition of done

1. A review-swarm run reaches a step after `Launch cloud swarm` — the first
non-zero success in this workflow's history.
2. Paste the literal step list showing `Launch cloud swarm` succeeded.
3. If it fails, paste the literal error and STOP. Do not weaken the gate to make
it green. A gate that passes without running is the failure this whole
sequence has been climbing out of.
Loading