Skip to content

test: no test may skip on a failure to build, and a guard says so - #812

Merged
tannevaled merged 1 commit into
mainfrom
test/no-skip-on-build-failure
Oct 12, 2026
Merged

tannevaled merged 1 commit into
mainfrom
test/no-skip-on-build-failure

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Closes issue 687.

The defect it closes

Two tests turned a build failure into a PASS:

if out, err := build.CombinedOutput(); err != nil {
    t.Skipf("cannot build cmd/rbgo (%v): %s", err, out)
}

If cmd/rbgo stopped compiling, both reported skip, the lane stayed green,
and the binary-level HTTP and socket coverage silently stopped being exercised.
t.Skipf there says "this machine cannot build it"; what it reports is "the
product is broken"
. Only one of those may be green.

Both are already t.Fatalf — measured on main, not assumed. So this PR
adds what closes the class rather than the two lines.

What the guard judges, and what it deliberately does not

It walks every *_test.go, reads each t.Skip/Skipf/SkipNow's reason,
and refuses a vocabulary that can only mean the code did not compile.

A skip is legitimate when what is missing belongs to the machine — no
symlinks, no /dev/zero, not root. It is never legitimate when what is missing
is this repository's own build, because every lane builds it and a developer
who cannot wants to be told. Inferring that from the surrounding code would be a
judgement call dressed as a check, so it reads the reason and nothing else.

Three things so it is not decorative

Word boundaries, not substrings. The first version used strings.Contains
and reported cmd/rbgo/build_test.go's

set RBGO_BUILD_IT=1 to run the rbgo build integration test

as an offence — because go build is a substring of rb|go build. One false
positive is enough to get a guard deleted. That is the third
substring-blindness of this kind in two days
, after -v inside --version
and a README phrase inside its own removal note.

Its own control, run before it judges anything. Seven reasons — three that
must match, four that must not, including that false positive. A vocabulary
nobody tested is a guess, and a guess that matches nothing looks exactly like a
clean tree.

It counts what it read and fails at zero. 59 t.Skip call(s) examined, 0 offence(s). A walk that parsed nothing finds no offence either.

Proved in the other direction

Replanting the exact shape issue 687 described, at the site it described:

--- FAIL: TestNoTestSkipsOnAFailureToBuild
    internal/vm/socket_test.go:433:3: skips on a build failure: "cannot build cmd/rbgo (%v): %s"
    60 t.Skip call(s) examined, 1 offence(s)

Also from that issue's inventory

grape_internal_test.go's if r.handler != nil { t.Skip("handler set") } could
not fire — handler is set to nil on the line above — so the test read as
conditional when it is not. It is an assertion now, so a change that did leave
a handler there goes red rather than quiet.

The two "gap recorded" skips in scripts/conformance/webapp are left alone
on purpose: they record a known missing binding with a long diagnostic, and
turning them red would fail the lane for a gap nobody is hiding. Said here so
the next sweep does not treat them as an oversight.

🤖 Generated with Claude Code

#687 named two sites that turned a build failure into a PASS:

    if out, err := build.CombinedOutput(); err != nil {
        t.Skipf("cannot build cmd/rbgo (%v): %s", err, out)
    }

If cmd/rbgo stopped compiling, both reported SKIP, the lane stayed green, and
the binary-level HTTP and socket coverage silently stopped being exercised.
`t.Skipf` there says "this machine cannot build it"; what it reports is "the
product is broken", and only one of those may be green.

Both are already t.Fatalf -- measured, not assumed. So this commit adds what
closes the CLASS rather than the two lines: a walk over every *_test.go that
reads each t.Skip/Skipf/SkipNow's REASON and refuses a vocabulary which can only
mean the code did not compile.

It judges the reason and not the surrounding code, deliberately. A skip is
legitimate when what is missing belongs to the MACHINE -- no symlinks, no
/dev/zero, not root -- and never when what is missing is this repository's own
build, because every lane builds it and a developer who cannot wants to be told.
Inferring that distinction from the call site would be a judgement call dressed
up as a check.

Three things it does to avoid being decorative:

  WORD BOUNDARIES, not substrings. The first version used strings.Contains and
  reported cmd/rbgo/build_test.go's "set RBGO_BUILD_IT=1 to run the rbgo build
  integration test" as an offence, because "go build" is a substring of
  "rb|go build". That is the third substring-blindness of this kind in two days.

  ITS OWN CONTROL runs before it judges anything: seven reasons, three that must
  match and four that must not, including that false positive. A vocabulary
  nobody tested is a guess, and a guess that matches nothing looks exactly like
  a clean tree.

  IT COUNTS WHAT IT READ and fails at zero. 59 skips examined, 0 offences. A
  walk that parsed nothing finds no offence either.

Proved in the other direction too: replanting the exact shape #687 described, at
the site it described, turns it red and names the line --
`internal/vm/socket_test.go:433:3: skips on a build failure`.

Also from #687's inventory: grape_internal_test.go's `if r.handler != nil {
t.Skip }` could not fire, since handler is set to nil on the line above. It is
an assertion now, so a change that did leave a handler there goes red rather
than quiet.

The two "gap recorded" skips in scripts/conformance/webapp are left alone on
purpose: they record a known missing binding with a long diagnostic, and turning
them red would make the lane fail for a gap nobody is hiding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit a4097c8 into main Oct 12, 2026
9 checks passed
@tannevaled
tannevaled deleted the test/no-skip-on-build-failure branch October 12, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant