The ratchet is blind to a regression smaller than a file's unbanked improvement #881

The ratchet judges each file against its own recorded count and fails if the file passes fewer. That is the right shape, and it has one blind spot that is invisible from the inside: a file whose baseline sits below its current value carries slack, and the gate cannot see a regression smaller than that slack.

Found on a real case

Re-recording the baseline for the Ruby 4.0 move (#799), the lane reported:

language/variables_spec.rb: 118 passing, baseline 115 (+3)

reported as an improvement. But language/variables_spec.rb has passed 119 since #871 closed the last defect of #799. The version move genuinely costs one example there — Multiple assignment with a single splatted RHS value | converts nil to empty array without calling a method, measured directly with cmd/specdelta between two engines built from one commit minus the version string.

So a real regression of −1 was reported as a gain of +3, because the baseline was 4 below the truth. The same file shows it twice:

language/numbered_parameters_spec.rb: 17 passing, baseline 16 (+1)

Why the slack exists

cmd/ratchet prints N file(s) improved -- rerun with UPDATE_BASELINE=1 to lock it in. Until #877 there was no correct way to do that: the gate runs on a pinned ubuntu-24.04 and UPDATE_BASELINE=1 was a local command, so every re-record was made in the wrong environment. Improvements therefore accumulated unbanked, and each one is a blind spot of its own size.

This is the exact complement of #876. That one stops a re-record from banking a loss; this one is the cost of not banking a win.

What would close it

The slack is computable and currently not reported. Options, roughly in order of how much they change:

1. cmd/ratchet reports total slack — N file(s) are above their baseline by M examples in total; the gate is blind to a regression of up to M in those files — so the blind spot is at least visible in every run.
2. CI warns, or fails, when slack passes a threshold.
3. Re-record routinely now that rubyspec: record the baseline where it is enforced #877 makes it correct, which drives slack to zero and keeps it there.

(3) is the real fix and (1) is what makes it maintainable; (2) is the one worth arguing about.

The 4.0 baseline about to land is a fresh measurement, so it clears the current slack as a side effect — but nothing stops it accumulating again.

Refs #799, #876, #877