Repository navigation
End the risk-tier report with one machine-readable line - #670
Merged
Merged
Conversation
The owner decision on the issue is ②: the workflow stays read-only and never
labels. What was missing is auditability — the tier lived only in prose, so a
review had to restate a number instead of quoting one.
`judge()` now records every changed path with the tier the rules gave it, and
`summaryLine()` renders the last line of the job summary from that same result:
risk-tier: T3; basis: AGENTS.md=T3, plugins/social/src/social.js=T3
The line is built from the object the prose report is built from, so the two
cannot drift, and a long basis is capped. AGENTS.md, `.github/labels.json` and
`docs/controller.md` now say the same thing: the workflow reports, the
controller copies the label on its poll, no one writes `risk:T*` by hand and no
review transcribes a tier onto a PR; the tier that decides is the one the merge
gate recomputes from the diff.
No write permission is added and no label, comment or merge is made.
Fixes #609
Collaborator
Author
|
Verdict: PASS at 4c8d98c Reviewed this tip only. Nine checks What I checked
Non-blocking
No other finding; the line is report-only, adds no token scope, and no review is expected to copy a tier onto the PR — the controller does that on its next poll. |
Birfy
added a commit
to Birfy/FlyBot
that referenced
this pull request
Oct 23, 2026
… green run (ThinkFlowLab#697) (ThinkFlowLab#706) Written-by: Dev3 Fixes ThinkFlowLab#697 Tip SHA: 3e8ea72 ## Reproduce `audit-merges.mjs` is red on every historical bypass, and nothing runs it. Live, against `main` at `bafe9bad` with the script this branch changes, `--limit 40`: ``` $ node .github/scripts/audit-merges.mjs --limit 40 # today's behaviour bypassed queue: ThinkFlowLab#680 6b9820b (merge_group: failure) bypassed queue: ThinkFlowLab#636 8e80237 (merge_group: failure) … 24 more lines … audit-merges: 40 commits on main, 26 bypassed the queue, 0 direct exit 1 ``` 26 of the last 40 commits, so the check is red on every run and therefore unreadable; and no workflow and no document named the script, so nobody saw it. ## What changed - `.github/scripts/audit-merges.mjs`: `--since <sha|ISO date>` (a sha is resolved to its committer date through `GET /commits/{sha}`) and `--baseline`. A finding at or before the watermark prints as `earlier bypass: …` and does not fail the run; a finding after it prints as `bypassed queue: …` and exits 1; `--baseline` treats every finding as earlier, which is what a first run uses before there is a previous run to point at. `audit()` carries each commit's committer date, `classify()` applies the watermark, and `exitCodeFor()` counts only findings that are not earlier. With neither flag the behaviour is exactly today's. - `.github/workflows/audit-merges.yml` (new): daily at 04:41 UTC plus `workflow_dispatch`, `permissions: contents/actions/pull-requests: read`, no write scope at all, pinned actions, script taken from `main`. Its watermark is the `head_sha` of its own previous **successful** run (`runs?status=success&per_page=1`), so it commits no watermark file; when that lookup is empty it runs `--baseline`, so the first green run *is* the baseline. - `.github/scripts/audit-merges.test.mjs` + `fixtures/audit-merges/since.json`: nine new cases — all-older exits 0 with every line as `earlier bypass`, one newer asks for exit 1 and leaves the other earlier, a sha watermark resolves to its committer date and is itself older, `--baseline` never fails, no flag keeps today's behaviour, a date needs no commit lookup, a bad `--since` refuses, a finding with no date is not assumed old, and the workflow's shape (read-only, daily, previous-run watermark, pinned actions). - `docs/autonomous-dev.md`, Merge plumbing: one paragraph naming the patrol and saying what a red run means. One deliberate deviation from the issue's wording: the walk is **not** narrowed with `GET /commits?…&since=`. The acceptance asks for a run where every finding is older to print each one as `earlier bypass:` and exit 0 — a walk that starts at the watermark has no older commit left to print. So the watermark is applied to each commit's committer date and the walk keeps its `--limit`; the API calls are the same three as before, plus one for a sha watermark. ## What I ran ``` $ node --test .github/scripts/audit-merges.test.mjs # this branch 1..18 # pass 18 # fail 0 $ node --test ".github/scripts/*.test.mjs" # clean extraction of main bafe9ba + this branch # tests 182 # pass 178 # fail 4 $ node --test ".github/scripts/*.test.mjs" # the same extraction without this change # tests 173 # pass 169 # fail 4 ``` The 4 failures are the `ds-bot-patches` cases that need `git`, which this box does not have; they are identical on both sides. Everything else is green, including `test-workflow` and the label docs checks. Live against this repository, through the branch's own `main()` (`--limit 40`; `gh` is not installed on this box, so the API answers were pre-fetched and served to `main()` through its injectable `api`): ``` $ … --since cbce2b0 earlier bypass: ThinkFlowLab#680 6b9820b (merge_group: failure) … 25 more … audit-merges: 40 commits on main, 0 bypassed the queue, 0 direct, 26 at or before the watermark audit-merges: watermark 2026-10-22T07:29:05.000Z (from cbce2b0…) exit 0 $ … --since 93e5234 bypassed queue: ThinkFlowLab#680 6b9820b (merge_group: failure) bypassed queue: ThinkFlowLab#636 8e80237 (merge_group: failure) bypassed queue: ThinkFlowLab#664 7223a1a (merge_group: failure) earlier bypass: ThinkFlowLab#670 372b064 (merge_group: failure) … 22 more earlier … audit-merges: 40 commits on main, 3 bypassed the queue, 0 direct, 23 at or before the watermark exit 1 ``` So the same history is green when the watermark is the head of the last run and red on exactly the three merges newer than an older watermark — the two halves the issue asks for, on real data. ## Red without the source change ``` $ # variant A: main's audit-merges.mjs + this branch's tests not ok 1 - .github/scripts/audit-merges.test.mjs # imports classify / sinceDate, which main does not export # tests 1 # pass 0 # fail 1 $ # variant B: this branch's script with the classify() call removed (the watermark filter gone) not ok 10 - ThinkFlowLab#697: a watermark after every finding prints them as earlier bypass and exits 0 not ok 11 - ThinkFlowLab#697: one finding after the watermark alone fails the run, the other stays earlier not ok 12 - ThinkFlowLab#697: a sha watermark is resolved through its own committer date, and is itself older not ok 13 - ThinkFlowLab#697: --baseline is the first run: every finding is earlier and nothing fails # tests 18 # pass 14 # fail 4 ``` Exactly the four watermark cases go red; the "no flag keeps today's behaviour" case stays green, so the filter is what those four measure and nothing else moved. ## Not run - The workflow itself: `audit-merges.yml` does not exist on `main` yet, so it cannot be dispatched from this branch (a `workflow_dispatch` run needs the file on the default branch). Its first real run after merge is the `--baseline` one, and its shape is covered by the text test above. - `npm run arch-docs`: `core/scripts/check-arch-docs.mjs` answers `arch-docs: FAIL — cannot diff against origin/main` because this box has no `git`. CI runs it. - A hand run through the real `gh` CLI: no `gh` binary here, hence the injected `api` in the live run above. - No changelog fragment: an internal audit, not user-visible product behavior. ## Risk - The patrol adds a scheduled job: daily at 04:41 UTC, read-only, ~40 commits × 3 API calls, well inside the 1,000/hour budget, and it can be disabled from the Actions tab like `risk-tier.yml`. - The watermark is the previous **successful** run, so a bypass that lands while the patrol is red keeps it red until it is fixed — that is intended (a bypass is a defect), and the fix is an empty one if the bypass was legitimate: a manual `workflow_dispatch` after the fact moves the watermark forward. - `--limit 100` in the workflow bounds the walk: a bypass older than the window is not reported at all. That is the same bound the script always had. - Tier T3 (`.github/**`); no write permission, no secret, no label or comment written by the job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #609
Tip SHA: 4c8d98c
Reproduce
The rule text says
risk:T*is set by the controller from the diff, never by hand, and themechanism does not hold up:
github-actions[bot]labelled #574 at 07:28:43Z, while #606, #610,#611, #612, #616, #617, #618 and #619 carry no
risk:T*at all even thoughrisk tier (report only)succeeded on every one of them. The owner decision on the issue is ②: keep the workflowread-only, never label — and make the computed tier auditable instead of remembered.
At this tip,
node .github/scripts/risk-gate.mjs --pr <n>ends its report withso a review can quote the tier the rules produced instead of restating a number.
What changed
.github/scripts/risk-gate.mjs:judge()records every changed path (and a rename's old path)with the tier
tierOf()gave it, andsummaryLine()renders the final line of the report fromthat same result — highest tier first, paths capped at 10 with a
+N moretail.render()endsthe job summary with it, after the prose, so the two can never disagree.
.github/scripts/risk-gate.test.mjs: four cases — the line's exact shape for a T3 PR, theordering and the cap, the no-changed-files case, and "the line is the last line of the report".
AGENTS.mdgains a bullet under the label section: quote the summary line, the workflow neverwrites a label, the controller copies it on its poll, no review transcribes a tier.
.github/labels.json: everyrisk:T*description keepsadvisoryandthe merge gate recomputes itand now also names the controller as the writer (all ≤ 100 characters).docs/controller.md: the tier stays auditable without the label — quote the summary line..github/workflows/risk-tier.yml: header comment only — no permission added, no step changed.It keeps
contents: readandpull-requests: readand still never labels, comments or merges.What I ran
TZ=UTC node --test .github/scripts/*.test.mjsat this tip (withgitonPATH) →144/144.
node --test .github/scripts/risk-gate.test.mjs→ 18/18, including the four new cases.node .github/scripts/risk-gate.mjsbehaviour checked by hand through the harness result:judge([AGENTS.md, plugins/social/src/social.js])→risk-tier: T3; basis: AGENTS.md=T3, plugins/social/src/social.js=T3; no files →risk-tier: T0; basis: none (no changed files).changelog.mjs checknot run: a change users cannot see needs no fragment (changelog.d/README).Red without the source change
basis.push([name, t])line fromjudge()(the tests untouched) →risk-gate.test.mjs16/18, 2 red, exactlysummaryLine: one machine-readable line…andsummaryLine: highest tier first….summaryLine(result)from the end ofrender()instead → 17/18, 1 red, exactlyrender: the machine-readable line is the last line of the report. Restored → 18/18.Not run
risk-tier.ymlis a comment, and theexisting
the workflow stays report onlytest still passes with it.npm run verifyend to end: nocore/node_modulesin this sandbox (the same onmain).risk-tierjob on this PR runs the base copy of the script, so its summary will only showthe new line after merge — that is the workflow's normal base-copy behaviour, not a gap here.
Risk
T3 by the repository's own rules (
.github/**,AGENTS.md): a reporting script plus text. Itadds no permission, no network call, no label, comment or merge, and cannot block anything — the
job is still
continue-on-error: trueand not a required check. The only behaviour it changes isthe last line of that job's summary. The one thing a reader must not conclude is that the label
now arrives on time: it does not, and the docs keep saying why (#629).