Skip to content

docs(next): point drive runs at #174 instead of human-blocked work #210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kjgbot merged 1 commit into from
Sep 6, 2026
Merged

docs(next): point drive runs at #174 instead of human-blocked work #210

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 63 additions & 64 deletions ops/NEXT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,85 +1,84 @@
# NEXT — give the review gate a credential
# NEXT — fix the crash-resume hang (#174)

**Scope:** one Actions secret and two `env:` lines in
`.github/workflows/review-swarm.yml`. Nothing else.
**Scope:** `kernel/relayflowd/`, the crash-resume test suite, and nothing else.

**The Relayflow Lead cannot do this one.** RFC-0001 decision #6 and the
charter's second hard rail: it cannot edit the gates that judge its work.
## Why this and not gate 3

## The headline
The previous package pointed at the review-swarm credential. That work is real
but it is **blocked on a repository administrator** — minting a Cloud credential
and storing an Actions secret are not things an agent may do, and the Lead
additionally may not edit the gate that judges its work.

**The review swarm has never succeeded.**
Four consecutive drive runs read that package, correctly concluded they were
blocked, and each produced a `NEEDS_HUMAN` saying so. That is four cycles spent
re-deriving the same fact. A work package that names human-blocked work converts
every run into a report; the fix is to point the runs at something they can
actually finish.

```
TOTAL runs: 76 failure: 75 cancelled: 1 successes: 0
first 2026-08-30T20:22:22Z
latest 2026-09-06T04:03:35Z
```

Treat any claim that gate 3 is "architecturally complete" against that number.
Most of its nine requirements describe behaviour downstream of a launch that has
never happened, so nothing past authentication has ever executed.
The credential decision is tracked and waiting elsewhere. Do not work on it here.

## What already shipped (2026-09-06)
## The problem

Four layers, each revealing the next:
`llm::sigkill_sweep_covers_before_and_between_the_rung_b_steps` hangs
intermittently on GitHub runners. Issue **#174**, reopened 2026-09-06 with fresh
evidence after being closed.

| step | failed because | closed by |
|---|---|---|
| `Validate cloud authentication` | repo had zero Actions secrets | `RELAY_WORKSPACE_KEY` added |
| `Prepare review input` | gate scripts were mode `100644`, exit 126 | #172 |
| `Launch cloud swarm` | CLI never installed, exit 127 | #198 |
| `Launch cloud swarm` | pinned runtime read no API key | #198 (pin → 11.10.3) |
```
thread 'llm::sigkill_sweep_covers_before_and_between_the_rung_b_steps'
panicked at relayflowd/tests/crash_resume/llm.rs:121:27
test result: FAILED. 33 passed; 1 failed
```

Also landed: #203 (whole-line verdict matching, `jq -er` on the poll response),
#202 (a missing reviews directory yields `MISSING` rather than a `find` error).
Line 121 is the `no step.dispatch after resume` path — the worker never receives

@cubic-dev-ai cubic-dev-ai Bot Sep 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The prose says "Line 121 is the no step.dispatch after resume path", but at current head the panic! emitting that message is on line 122 (llm.rs:122), and line 121 is the Ok(dispatch) => dispatch, success branch. The quoted stack trace (llm.rs:121:27) is from a past run and is fine as quoted evidence, but the present-tense instruction will point an agent at the wrong branch. Drop the hard-coded line number (or fix it to 122) so the dock-up line and the actual panic site agree.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At ops/NEXT.md, line 32:

<comment>The prose says "Line 121 is the `no step.dispatch after resume` path", but at current head the `panic!` emitting that message is on line 122 (`llm.rs:122`), and line 121 is the `Ok(dispatch) => dispatch,` success branch. The quoted stack trace (`llm.rs:121:27`) is from a past run and is fine as quoted evidence, but the present-tense instruction will point an agent at the wrong branch. Drop the hard-coded line number (or fix it to 122) so the dock-up line and the actual panic site agree.</comment>

<file context>
@@ -1,85 +1,84 @@
 
-Also landed: #203 (whole-line verdict matching, `jq -er` on the poll response),
-#202 (a missing reviews directory yields `MISSING` rather than a `find` error).
+Line 121 is the `no step.dispatch after resume` path — the worker never receives
+a dispatch after the daemon is SIGKILLed and resumed. The comment above it
+already attributes this to #174 and captures a daemon-state dump precisely
</file context>
Suggested change
Line 121 is the `no step.dispatch after resume` path — the worker never receives
The `no step.dispatch after resume` path is the panic that the comment above it attributes to #174 — the worker never receives

a dispatch after the daemon is SIGKILLed and resumed. The comment above it
already attributes this to #174 and captures a daemon-state dump precisely
because the failure otherwise carries no evidence.

## The one thing left
## The evidence, and what makes it tractable now

The job now has a CLI that can read an API key, and no key to read.
`agent-relay cloud run` falls back to the interactive device flow and dies after
ten minutes:
It reproduces at roughly one run in eight on `main`:

```
Device login expired before it was approved. Run the command again to get a new code.
main, cloud-runtime-artifact.yml, last 8 runs: 7 success, 1 failure
```

`@agent-relay/cloud@11.10.3` resolves `CLOUD_API_KEY` through
`WorkflowApiKeyClient.fromEnv`, which `workflowApiClient` prefers over the stored
login. With the variable set, the device flow is never reached.
Earlier this looked like a regression from a specific commit, because `main`
normally runs about once a day and seven commits landed within ten minutes. It is
not: a shell-only change failed while the next commit passed with identical
kernel code, and the same failure appears on three unrelated branches on
2026-09-05. **The rate did not change; the sample size did.**

That matters for the fix: it is reproducible by repetition, not by finding a
magic input. Run the crash-resume suite in a loop and it will show up.

## What to do

1. **Mint the credential.** `AgentWorkforce/cloud` →
`docs/runbooks/relay-ci-workflow-credential.md`, profile
`CI_TOKEN_PROFILE=workflow-invoke`. Non-human, workspace-bound, scoped to
exactly `workflow:invoke:read` and `workflow:invoke:write`. The runbook notes
provisioning and rotation "require no browser login".
2. **Store it.** An operator mints; **a repository administrator stores it**. The
runbook is explicit that an agent is not authorized to create or update
GitHub secrets.
3. **Set both variables** on the `Launch cloud swarm` step: `CLOUD_API_URL` and
`CLOUD_API_KEY`.
4. **Fix the preflight, which currently cannot fail.** `Validate cloud
authentication` tests that `RELAY_WORKSPACE_KEY` is non-empty, never examines
the credential `cloud run` uses, and never attempts an authentication — it
passed green on run 34007204726, whose authentication then failed ten minutes
later. Assert both variables, the way `AgentWorkforce/relay` does:

```bash
test -n "$CLOUD_API_URL"
test -n "$CLOUD_API_KEY"
```

**Precedent:** `AgentWorkforce/relay`'s `.github/workflows/relayflow-pr-proof.yml`
runs this exact shape in production — published CLI, `CLOUD_API_URL` and
`CLOUD_API_KEY` in the environment, no interactive login.
1. Reproduce it locally. `cd kernel && sh ../ops/cargo.sh test -p relayflowd --test crash_resume`
in a loop until it fails. Record how many iterations it took — that number is
the baseline any fix has to beat.
2. Find where the dispatch is lost. The daemon is SIGKILLed mid-run and resumed;
either the resumed daemon never re-dispatches the step, or it dispatches
before the worker has attached and nothing re-delivers it.
3. Fix it in `kernel/relayflowd/`. Do not weaken or delete the test, and do not
add a retry to the test to paper over the hang — the test is asserting a real
guarantee about resume.
4. Prove the fix by repetition, not by one green run. State the iteration count
before and after.

## Definition of done

1. A review-swarm run reaches a step after `Launch cloud swarm` — the first
non-zero success in this workflow's history.
2. Paste the literal step list showing `Launch cloud swarm` succeeded.
3. If it fails, paste the literal error and STOP. Do not weaken the gate to make
it green. A gate that passes without running is the failure this whole
sequence has been climbing out of.
1. `cargo test --workspace` green from `kernel/`.
2. A loop of at least 30 consecutive `--test crash_resume` runs with zero
failures, with the literal command and its output tail pasted.
3. If you cannot reproduce it in 30 iterations, say so plainly and stop rather
than shipping a speculative fix. A hang nobody reproduced is not fixed by a
change nobody can test.

## Constraints

- `kernel/` only. Do not touch `.github/workflows/`, `packages/`, or the
publish pipeline.
- Do not edit `testdata/tick-heartbeat.*` or `hello-ladder.*` — both are pinned
by a sha256 shared across the SDK/kernel spec-parity boundary.
- `ops/reviews/`, `ops/DRIVE-LOG.md` and `ops/BACKLOG.md` are records of what was
true when written. Do not rewrite them.
Loading