Skip to content

test: fail when a fixture pre-installs what install.sh provides - #58

Merged
YASoftwareDev merged 1 commit into
masterfrom
test/fixture-masking-guard
Sep 19, 2026
Merged

YASoftwareDev merged 1 commit into
masterfrom
test/fixture-masking-guard

Conversation

@YASoftwareDev

Copy link
Copy Markdown
Owner

Why CI did not catch the no-sudo tmux gap

Not because the tests were weak. Measured: take v1.11.9's code (before the
installer existed), remove only the tmux prerequisite from Dockerfile.nosudo,
and the existing suite fails at once:

✗  tmux not found
✗  tmux not installed
✗  Failed:  4 / 55

The suite was adequate. The fixture defeated it: both the workflow and
Dockerfile.nosudo installed tmux as a root prerequisite, so has tmux was true
before install.sh ran and the missing installer could never matter. Fifteen green
no-sudo cells said nothing.

A second weakness compounded it: the no-sudo profile asserted provenance
(~/.local/bin) for 8 tools but only presence for tmux, so even without the
prerequisite a tmux from anywhere would have passed. Both are fixed - the
prerequisite is gone (1.12.0) and tmux is strict-checked (1.13.0).

bat was a different failure entirely: no test existed at all. Nothing can catch
what is never asserted.

What this PR adds

tests/no-fixture-masking.py, run by the CI gate. It checks the pattern, not the
instance: a tool with an _install_<name> function in modules/base.sh must not
appear in a no-sudo fixture's prerequisite list - in the workflow or in
Dockerfile.nosudo, under its own name or its Debian rename (bat/batcat,
fd/fd-find).

Its own first draft was a placebo, and the red-proof caught that

Worth recording, because it is the same class of error as the bug it guards.

The first version reported RESULT: PASSED with the masking line present. The
package lists sit on shell continuation lines: apt-get -yq install ... \ on
one line, the packages on the next. Matching physical lines found the install verb
and none of the packages, so it counted 10 packages (all from the workflow) and
compared against nothing that mattered.

Had I shipped on "it passes", I would have added a guard that proves nothing to a
repo whose problem was a check that proved nothing.

Continuations are now joined. Three mutations, each caught:

mutation result
tmux back in Dockerfile.nosudo (continuation line) rc=1, names _install_tmux()
tmux back in the workflow no-sudo job rc=1, names install.yml:256
batcat in the fixture (Debian rename) rc=1, names _install_bat()

Control on the fixed tree: RESULT: PASSED.

Limits, printed in its own output

  • It only knows tools named by an _install_<tool> function. A tool the repo
    ships config for but never installs is invisible to it - which is exactly what
    the original tmux gap was.
    It prevents recurrence of the masking, not the
    original omission.
  • It reads apt/dnf prerequisite lines by pattern. A prerequisite arriving another
    way - a base image that already carries the tool - is not seen.

Verified

bash test.sh workstation 62/63, 0 failures. tests/lint-workflows.sh PASSED.
tests/ci-gate-covers-all.py passes (no new job, so the gate's needs is
unchanged).

Answering "why didn't CI catch it": the tests were adequate, the fixture
defeated them. Measured - with v1.11.9's code and ONLY the tmux prerequisite
removed, the existing suite fails immediately with "tmux not found" and 4
failures. `has tmux` was simply true before install.sh ever ran, so 15 green
no-sudo cells said nothing about a missing installer.

This checks the pattern rather than the instance: a tool with an
_install_<name> function must not appear in a no-sudo fixture's prerequisite
list, in the workflow or in Dockerfile.nosudo, under its own name or its
Debian rename.

Its own first draft was a placebo and its red-proof caught that: the package
lists sit on shell CONTINUATION lines, so matching physical lines found the
install verb and none of the packages - it passed with the masking line
present. Continuations are now joined, and three mutations each redden it:
tmux in Dockerfile.nosudo, tmux in the workflow job, and batcat for bat.

It states its own limits in its output: it only knows tools named by an
_install_* function, so a tool the repo ships config for but never installs -
the original tmux gap - remains undetectable by it.
@YASoftwareDev
YASoftwareDev merged commit 2099d0d into master Sep 19, 2026
30 checks passed
@YASoftwareDev
YASoftwareDev deleted the test/fixture-masking-guard branch September 19, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants