Skip to content

End the risk-tier report with one machine-readable line - #670

Merged
Birfy merged 1 commit into
mainfrom
fix/609-risk-tier-summary-line
Oct 22, 2026
Merged

Birfy merged 1 commit into
mainfrom
fix/609-risk-tier-summary-line

Conversation

@Birfy

@Birfy Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Fixes #609

Tip SHA: 4c8d98c

Reproduce

The rule text says risk:T* is set by the controller from the diff, never by hand, and the
mechanism does not hold up: github-actions[bot] labelled #574 at 07:28:43Z, while #606, #610,
#611, #612, #616, #617, #618 and #619 carry no risk:T* at all even though risk tier (report only) succeeded on every one of them. The owner decision on the issue is ②: keep the workflow
read-only, never label — and make the computed tier auditable instead of remembered.

At this tip, node .github/scripts/risk-gate.mjs --pr <n> ends its report with

risk-tier: T3; basis: AGENTS.md=T3, plugins/social/src/social.js=T3

so a review can quote the tier the rules produced instead of restating a number.

What changed

  • .github/scripts/risk-gate.mjs: judge() records every changed path (and a rename's old path)
    with the tier tierOf() gave it, and summaryLine() renders the final line of the report from
    that same result — highest tier first, paths capped at 10 with a +N more tail. render() ends
    the job summary with it, after the prose, so the two can never disagree.
  • .github/scripts/risk-gate.test.mjs: four cases — the line's exact shape for a T3 PR, the
    ordering and the cap, the no-changed-files case, and "the line is the last line of the report".
  • AGENTS.md gains a bullet under the label section: quote the summary line, the workflow never
    writes a label, the controller copies it on its poll, no review transcribes a tier.
  • .github/labels.json: every risk:T* description keeps advisory and the merge gate recomputes it and now also names the controller as the writer (all ≤ 100 characters).
  • docs/controller.md: the tier stays auditable without the label — quote the summary line.
  • .github/workflows/risk-tier.yml: header comment only — no permission added, no step changed.
    It keeps contents: read and pull-requests: read and still never labels, comments or merges.

What I ran

  • TZ=UTC node --test .github/scripts/*.test.mjs at this tip (with git on PATH) →
    144/144.
  • node --test .github/scripts/risk-gate.test.mjs → 18/18, including the four new cases.
  • node .github/scripts/risk-gate.mjs behaviour checked by hand through the harness result:
    judge([AGENTS.md, plugins/social/src/social.js]) → risk-tier: T3; basis: AGENTS.md=T3, plugins/social/src/social.js=T3; no files → risk-tier: T0; basis: none (no changed files).
  • changelog.mjs check not run: a change users cannot see needs no fragment (changelog.d/README).

Red without the source change

  • Removing the basis.push([name, t]) line from judge() (the tests untouched) →
    risk-gate.test.mjs 16/18, 2 red, exactly summaryLine: one machine-readable line… and
    summaryLine: highest tier first….
  • Removing summaryLine(result) from the end of render() instead → 17/18, 1 red, exactly
    render: the machine-readable line is the last line of the report. Restored → 18/18.

Not run

  • The workflow itself (it needs Actions): the change to risk-tier.yml is a comment, and the
    existing the workflow stays report only test still passes with it.
  • npm run verify end to end: no core/node_modules in this sandbox (the same on main).
  • The risk-tier job on this PR runs the base copy of the script, so its summary will only show
    the new line after merge — that is the workflow's normal base-copy behaviour, not a gap here.

Risk

T3 by the repository's own rules (.github/**, AGENTS.md): a reporting script plus text. It
adds no permission, no network call, no label, comment or merge, and cannot block anything — the
job is still continue-on-error: true and not a required check. The only behaviour it changes is
the last line of that job's summary. The one thing a reader must not conclude is that the label
now arrives on time: it does not, and the docs keep saying why (#629).

The owner decision on the issue is ②: the workflow stays read-only and never
labels. What was missing is auditability — the tier lived only in prose, so a
review had to restate a number instead of quoting one.

`judge()` now records every changed path with the tier the rules gave it, and
`summaryLine()` renders the last line of the job summary from that same result:

    risk-tier: T3; basis: AGENTS.md=T3, plugins/social/src/social.js=T3

The line is built from the object the prose report is built from, so the two
cannot drift, and a long basis is capped. AGENTS.md, `.github/labels.json` and
`docs/controller.md` now say the same thing: the workflow reports, the
controller copies the label on its poll, no one writes `risk:T*` by hand and no
review transcribes a tier onto a PR; the tier that decides is the one the merge
gate recomputes from the diff.

No write permission is added and no label, comment or merge is made.

Fixes #609
@Birfy Birfy added the agent-pr Pull request opened by an agent label Oct 22, 2026
@Birfy Birfy self-assigned this Oct 22, 2026
@Birfy Birfy added area:automation Scheduled tasks and automation area:docs Documentation bug Something isn't working labels Oct 22, 2026
@Birfy

Birfy commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 4c8d98c
Written-by: Reviewer

Reviewed this tip only. Nine checks completed/success, mergeable clean, no earlier conclusion here. Risk tier T3 (.github/**, AGENTS.md), so it needs a maintainer's approve as well; I have not queued it.

What I checked

  • The line and the prose come from one judge() call. basis is filled where each path's tier is decided, deduplicated by path and sorted by tier then path, and render appends summaryLine(result) as the last line, so the two cannot disagree. SUMMARY_LIMIT caps the list at 10 with +N more.
  • For this PR's own files the gate gives T3, 131 changed lines, 6 files, none generated, and the line reads .github/labels.json=T3, .github/scripts/risk-gate.mjs=T3, .github/scripts/risk-gate.test.mjs=T3, .github/workflows/risk-tier.yml=T3, AGENTS.md=T3, docs/controller.md=T0.
  • The claim the labels now make is true. controller.mjs:24-27,129 adds/removes exactly risk:T0..T3 from the gate's tier, so "the controller labels it, the merge gate recomputes it" matches the code; risk-tier.yml itself only writes the job summary (its change here is a comment).
  • labels.json: 13 labels before and after, and the only content changes are the four risk:T* descriptions — agent-task, hold and the rest are untouched.
  • Tests: node --test ".github/scripts/*.test.mjs" → 153/153; risk-gate.test.mjs + risk-label-docs.test.mjs → 21/21. Red without the source change: .github/scripts/risk-gate.mjs reverted to 8005fc9d, tests untouched → risk-gate.test.mjs cannot even load (summaryLine missing), 0 pass. Restored → 153/153.
  • A rename is listed under both names with its own tier (docs/x.md <- core/src/host/x.js → core/src/host/x.js=T2, docs/x.md=T0), so the highest tier keeps a matching basis entry.

Non-blocking

  • A tier raised without a path has no basis entry. Size and an incomplete file list raise the tier in why but add nothing to basis, so the line can contradict itself: a 13-file, 325-line core/src/host/** change prints risk-tier: T3; basis: core/src/host/p0.js=T2, … — every listed path is T2 while the tier is T3, and only the prose Why says why. The limits make this the common case (T2: 250 lines / 12 files), not a corner. Adding one synthetic entry (for example (size)=T3 / (file list)=T3) when a non-path rule bumps the tier would make the machine line answer "what set it" on its own; the AGENTS.md sentence "the tier and the paths that set it" reads as if it always does.
  • A path containing , or = breaks the line's parse. basis: docs/a,b=c.md=T0 cannot be split reliably; either escape the two characters or say in the docs that the line is human-readable, not a parse contract.
  • The labels.json reformat (one object per line → pretty-printed) turns a four-value edit into a whole-file diff. Harmless, but it hides the real change from anyone reading the diff.

No other finding; the line is report-only, adds no token scope, and no review is expected to copy a tier onto the PR — the controller does that on its next poll.

@Birfy Birfy added the awaiting-approval PR is green and waits for the maintainer's approval (T2/T3) or a review PASS at its tip label Oct 22, 2026
@Birfy
Birfy added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit 372b064 Oct 22, 2026
9 checks passed
@Birfy
Birfy deleted the fix/609-risk-tier-summary-line branch October 22, 2026 03:52
@Birfy Birfy removed the awaiting-approval PR is green and waits for the maintainer's approval (T2/T3) or a review PASS at its tip label Oct 22, 2026
@Birfy Birfy removed the agent-pr Pull request opened by an agent label Oct 22, 2026
Birfy added a commit to Birfy/FlyBot that referenced this pull request Oct 23, 2026
… green run (ThinkFlowLab#697) (ThinkFlowLab#706)

Written-by: Dev3

Fixes ThinkFlowLab#697

Tip SHA: 3e8ea72

## Reproduce

`audit-merges.mjs` is red on every historical bypass, and nothing runs
it. Live, against `main` at `bafe9bad` with the script this branch
changes, `--limit 40`:

```
$ node .github/scripts/audit-merges.mjs --limit 40        # today's behaviour
bypassed queue: ThinkFlowLab#680 6b9820b (merge_group: failure)
bypassed queue: ThinkFlowLab#636 8e80237 (merge_group: failure)
… 24 more lines …
audit-merges: 40 commits on main, 26 bypassed the queue, 0 direct
exit 1
```

26 of the last 40 commits, so the check is red on every run and
therefore unreadable; and no workflow and no document named the script,
so nobody saw it.

## What changed

- `.github/scripts/audit-merges.mjs`: `--since <sha|ISO date>` (a sha is
resolved to its committer date through `GET /commits/{sha}`) and
`--baseline`. A finding at or before the watermark prints as `earlier
bypass: …` and does not fail the run; a finding after it prints as
`bypassed queue: …` and exits 1; `--baseline` treats every finding as
earlier, which is what a first run uses before there is a previous run
to point at. `audit()` carries each commit's committer date,
`classify()` applies the watermark, and `exitCodeFor()` counts only
findings that are not earlier. With neither flag the behaviour is
exactly today's.
- `.github/workflows/audit-merges.yml` (new): daily at 04:41 UTC plus
`workflow_dispatch`, `permissions: contents/actions/pull-requests:
read`, no write scope at all, pinned actions, script taken from `main`.
Its watermark is the `head_sha` of its own previous **successful** run
(`runs?status=success&per_page=1`), so it commits no watermark file;
when that lookup is empty it runs `--baseline`, so the first green run
*is* the baseline.
- `.github/scripts/audit-merges.test.mjs` +
`fixtures/audit-merges/since.json`: nine new cases — all-older exits 0
with every line as `earlier bypass`, one newer asks for exit 1 and
leaves the other earlier, a sha watermark resolves to its committer date
and is itself older, `--baseline` never fails, no flag keeps today's
behaviour, a date needs no commit lookup, a bad `--since` refuses, a
finding with no date is not assumed old, and the workflow's shape
(read-only, daily, previous-run watermark, pinned actions).
- `docs/autonomous-dev.md`, Merge plumbing: one paragraph naming the
patrol and saying what a red run means.

One deliberate deviation from the issue's wording: the walk is **not**
narrowed with `GET /commits?…&since=`. The acceptance asks for a run
where every finding is older to print each one as `earlier bypass:` and
exit 0 — a walk that starts at the watermark has no older commit left to
print. So the watermark is applied to each commit's committer date and
the walk keeps its `--limit`; the API calls are the same three as
before, plus one for a sha watermark.

## What I ran

```
$ node --test .github/scripts/audit-merges.test.mjs      # this branch
1..18  # pass 18  # fail 0

$ node --test ".github/scripts/*.test.mjs"               # clean extraction of main bafe9ba + this branch
# tests 182  # pass 178  # fail 4
$ node --test ".github/scripts/*.test.mjs"               # the same extraction without this change
# tests 173  # pass 169  # fail 4
```

The 4 failures are the `ds-bot-patches` cases that need `git`, which
this box does not have; they are identical on both sides. Everything
else is green, including `test-workflow` and the label docs checks.

Live against this repository, through the branch's own `main()`
(`--limit 40`; `gh` is not installed on this box, so the API answers
were pre-fetched and served to `main()` through its injectable `api`):

```
$ … --since cbce2b0
earlier bypass: ThinkFlowLab#680 6b9820b (merge_group: failure)
… 25 more …
audit-merges: 40 commits on main, 0 bypassed the queue, 0 direct, 26 at or before the watermark
audit-merges: watermark 2026-10-22T07:29:05.000Z (from cbce2b0…)
exit 0

$ … --since 93e5234
bypassed queue: ThinkFlowLab#680 6b9820b (merge_group: failure)
bypassed queue: ThinkFlowLab#636 8e80237 (merge_group: failure)
bypassed queue: ThinkFlowLab#664 7223a1a (merge_group: failure)
earlier bypass: ThinkFlowLab#670 372b064 (merge_group: failure)
… 22 more earlier …
audit-merges: 40 commits on main, 3 bypassed the queue, 0 direct, 23 at or before the watermark
exit 1
```

So the same history is green when the watermark is the head of the last
run and red on exactly the three merges newer than an older watermark —
the two halves the issue asks for, on real data.

## Red without the source change

```
$ # variant A: main's audit-merges.mjs + this branch's tests
not ok 1 - .github/scripts/audit-merges.test.mjs     # imports classify / sinceDate, which main does not export
# tests 1  # pass 0  # fail 1

$ # variant B: this branch's script with the classify() call removed (the watermark filter gone)
not ok 10 - ThinkFlowLab#697: a watermark after every finding prints them as earlier bypass and exits 0
not ok 11 - ThinkFlowLab#697: one finding after the watermark alone fails the run, the other stays earlier
not ok 12 - ThinkFlowLab#697: a sha watermark is resolved through its own committer date, and is itself older
not ok 13 - ThinkFlowLab#697: --baseline is the first run: every finding is earlier and nothing fails
# tests 18  # pass 14  # fail 4
```

Exactly the four watermark cases go red; the "no flag keeps today's
behaviour" case stays green, so the filter is what those four measure
and nothing else moved.

## Not run

- The workflow itself: `audit-merges.yml` does not exist on `main` yet,
so it cannot be dispatched from this branch (a `workflow_dispatch` run
needs the file on the default branch). Its first real run after merge is
the `--baseline` one, and its shape is covered by the text test above.
- `npm run arch-docs`: `core/scripts/check-arch-docs.mjs` answers
`arch-docs: FAIL — cannot diff against origin/main` because this box has
no `git`. CI runs it.
- A hand run through the real `gh` CLI: no `gh` binary here, hence the
injected `api` in the live run above.
- No changelog fragment: an internal audit, not user-visible product
behavior.

## Risk

- The patrol adds a scheduled job: daily at 04:41 UTC, read-only, ~40
commits × 3 API calls, well inside the 1,000/hour budget, and it can be
disabled from the Actions tab like `risk-tier.yml`.
- The watermark is the previous **successful** run, so a bypass that
lands while the patrol is red keeps it red until it is fixed — that is
intended (a bypass is a defect), and the fix is an empty one if the
bypass was legitimate: a manual `workflow_dispatch` after the fact moves
the watermark forward.
- `--limit 100` in the workflow bounds the walk: a bypass older than the
window is not reported at all. That is the same bound the script always
had.
- Tier T3 (`.github/**`); no write permission, no secret, no label or
comment written by the job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:automation Scheduled tasks and automation area:docs Documentation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rule says the controller sets risk:T*, but risk-tier.yml is report-only and never writes labels

1 participant