test: fail when a fixture pre-installs what install.sh provides - #58
Merged
Merged
Conversation
Answering "why didn't CI catch it": the tests were adequate, the fixture defeated them. Measured - with v1.11.9's code and ONLY the tmux prerequisite removed, the existing suite fails immediately with "tmux not found" and 4 failures. `has tmux` was simply true before install.sh ever ran, so 15 green no-sudo cells said nothing about a missing installer. This checks the pattern rather than the instance: a tool with an _install_<name> function must not appear in a no-sudo fixture's prerequisite list, in the workflow or in Dockerfile.nosudo, under its own name or its Debian rename. Its own first draft was a placebo and its red-proof caught that: the package lists sit on shell CONTINUATION lines, so matching physical lines found the install verb and none of the packages - it passed with the masking line present. Continuations are now joined, and three mutations each redden it: tmux in Dockerfile.nosudo, tmux in the workflow job, and batcat for bat. It states its own limits in its output: it only knows tools named by an _install_* function, so a tool the repo ships config for but never installs - the original tmux gap - remains undetectable by it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why CI did not catch the no-sudo tmux gap
Not because the tests were weak. Measured: take v1.11.9's code (before the
installer existed), remove only the tmux prerequisite from
Dockerfile.nosudo,and the existing suite fails at once:
The suite was adequate. The fixture defeated it: both the workflow and
Dockerfile.nosudoinstalled tmux as a root prerequisite, sohas tmuxwas truebefore
install.shran and the missing installer could never matter. Fifteen greenno-sudo cells said nothing.
A second weakness compounded it: the no-sudo profile asserted provenance
(
~/.local/bin) for 8 tools but only presence for tmux, so even without theprerequisite a tmux from anywhere would have passed. Both are fixed - the
prerequisite is gone (1.12.0) and tmux is strict-checked (1.13.0).
batwas a different failure entirely: no test existed at all. Nothing can catchwhat is never asserted.
What this PR adds
tests/no-fixture-masking.py, run by the CI gate. It checks the pattern, not theinstance: a tool with an
_install_<name>function inmodules/base.shmust notappear in a no-sudo fixture's prerequisite list - in the workflow or in
Dockerfile.nosudo, under its own name or its Debian rename (bat/batcat,fd/fd-find).Its own first draft was a placebo, and the red-proof caught that
Worth recording, because it is the same class of error as the bug it guards.
The first version reported
RESULT: PASSEDwith the masking line present. Thepackage lists sit on shell continuation lines:
apt-get -yq install ... \onone line, the packages on the next. Matching physical lines found the install verb
and none of the packages, so it counted 10 packages (all from the workflow) and
compared against nothing that mattered.
Had I shipped on "it passes", I would have added a guard that proves nothing to a
repo whose problem was a check that proved nothing.
Continuations are now joined. Three mutations, each caught:
tmuxback inDockerfile.nosudo(continuation line)_install_tmux()tmuxback in the workflow no-sudo jobinstall.yml:256batcatin the fixture (Debian rename)_install_bat()Control on the fixed tree:
RESULT: PASSED.Limits, printed in its own output
_install_<tool>function. A tool the repoships config for but never installs is invisible to it - which is exactly what
the original tmux gap was. It prevents recurrence of the masking, not the
original omission.
way - a base image that already carries the tool - is not seen.
Verified
bash test.sh workstation62/63, 0 failures.tests/lint-workflows.shPASSED.tests/ci-gate-covers-all.pypasses (no new job, so the gate'sneedsisunchanged).