Repository navigation
fix(ci): brain-ci reported an unmeasured value as a score of zero - #957
Conversation
brain-ci has failed on every run in its history, including on main. Neither failure was a health regression. 1. `tri stress --health` currently prints "stress-test: TODO - not implemented yet" and emits no "Score:" line. The step's `grep ... || echo "0"` turned that absence into the NUMBER 0, which the threshold check then reported as "Brain health 0 below threshold 80" -- a fabricated measurement of a command that measured nothing. Same class as A36/A37, where an unmeasured field rendered as an FPGA measurement on a public route. It now distinguishes "not measured" from "measured zero" and prints what the command actually said. The job still fails, because an unimplemented health check is a real deficiency -- but it now fails for the true reason instead of accusing the brain of a regression it never had. 2. "Critical State Notification" carried only `if: failure()`. On a push there is no PR, so context.issue.number is undefined and the call becomes POST /repos/.../issues//comments -> 404, failing the step and the job. That is what "Brain Health Report" has been failing on for every push to main: the reporter, not the check. Now guarded on pull_request, matching the two sibling steps that already were. Also fixes a latent parse bug: the score pattern required "[0-9]+\.[0-9]+", so a Score: line reading "Score: 92" would have parsed as empty and reached bc as an empty string. It now accepts an integer, and an unparseable Score: line is reported as unmeasured rather than silently becoming zero. Verified by running the step's logic over five inputs: the real current output (not measured), a decimal score (pass), an integer score (pass -- the old pattern broke here), a genuine 41.2 regression (still fails, as it must), and an unparseable Score: line (not measured). The gate is not weakened. A check that is always red carries as little information as one that is always green, and this one was red for a reason nobody could act on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first version of this comment claimed the 404 was why Brain Health Report failed on every push. That is only part of it: Generate Health Report is an aggregator and exits 1 whenever an upstream job failed, which is correct. The 404 fired on top of that and buried the real verdict under a 30-line octokit dump, so the log read as an API problem rather than as a failing health check. Guarding the notification removes the noise, not the signal. Caught by reading the aggregator's own body after the fix ran in CI, rather than inferring it from the job disappearing off a list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Correction to §2 of the description, pushed as I wrote that the 404 was what Brain Health Report had been failing on. That is What the 404 actually did was fire on top of the real verdict and bury it I caught this by reading the aggregator's body after the fix ran in CI, rather Verified on the runner, not only locally — the health step now emits: in place of the old |
brain-cihas failed on every run in its history, including onmain.Neither of its two failures is a health regression.
1. An unmeasured value reported as a score of zero
tri stress --healthcurrently prints:and emits no
Score:line. The step'sgrep "Score:" ... || echo "0"turnedthat absence into the number
0, which the threshold check then reported as:That reads as a measured collapse of brain health. It is a fabricated
measurement of a command that measured nothing — the same class as A36/A37,
where an unmeasured field rendered as an FPGA measurement on a public route.
The step now distinguishes not measured from measured zero, and prints what
the command actually said. The job still fails, because an unimplemented
health check is a real deficiency — but it fails for the true reason instead of
accusing the brain of a regression it never had.
2. The reporter, not the check
Critical State Notificationcarried onlyif: failure(). On a push there is noPR, so
context.issue.numberis undefined and the call becomes:which fails the step and takes the job with it. That is what 📋 Brain Health
Report has been failing on for every push to
main. Now guarded onpull_request, matching the two sibling steps that already were. (Audited allfour
context.issue.numberuses in the file; this was the only unguarded one.)3. A latent parse bug
The score pattern required
[0-9]+\.[0-9]+, so a line readingScore: 92wouldhave parsed as empty and reached
bcas an empty string. It now accepts aninteger, and an unparseable
Score:line is reported as unmeasured rather thansilently becoming zero.
Verified
Ran the step's logic over five inputs rather than reading it through:
Score: 92.5 HEALTHYScore: 92(integer)Score: 41.2Score: n/aThe gate is not weakened — a real low score still fails it. A check that is
always red carries as little information as one that is always green, and this
one was red for a reason nobody could act on.
Still open
tri stress --healthis unimplemented. That is the actual work behind thisgate, and it is not in this PR.