Repository navigation
A criterion may ask whether the spec builds - #488
Merged
Merged
Conversation
The allowlist held five subcommands, so every criterion filed against this repository was a grep -- and a grep is satisfied by a file that never builds: 388 of the 837 specs the oracle reaches generate Zig that does not compile while satisfying every criterion ever written against them. `test-report` generates the reviewed spec and has Zig build and run that spec's own tests, which is the oracle's question asked before the oracle. The bee already has a shell in this container, so nothing new is reachable to it; what the allowlist prevents is a criterion rigging its own measurement by running a script from the commit under review. The readers added beside it -- coverage, lint, symbols, outline, validate-vacuity and the rest -- were measured on the corpus and answer in under 30 s. `dupes` (~50 s) and `check-calls` (~140 s) are deliberately refused, with tests pinning the refusal: past the 60 s timeout a criterion is inconclusive, which spends a review and establishes nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
❌ Tests failed — 11/2431 failed
Failed tests
|
Every gate that decides whether a bee's pull request lands is a Python script: tools/dupe_scan.py is the Duplicate Body Ratchet, check_assertionless_spec_tests.py is the corpus ratchet, tools/ci/check_specs_still_parse.py is the required Spec Parse Ratchet, and the twenty-one reporting helpers under scripts/tri_loop/ are Python as well. The image declared git, perl-base and tini and said nothing about python3, while the brief has been telling bees since 2026-09-20 to run `python3 tools/dupe_scan.py --name <fn>`. python3 rather than python3-minimal: those scripts import json, hashlib, difflib and subprocess, and Debian's minimal package carries the subset dpkg's maintainer scripts need. Nothing outside the standard library, so no pip and no venv. `python3 --version` runs in the same layer, so a base image that stops shipping it fails the build instead of the swarm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dmitrii-f-t27
pushed a commit
to dmitrii-f-t27/BrowserOS
that referenced
this pull request
Oct 10, 2026
…n the gates (gHashTag#488) * feat(queen): a criterion may ask whether the spec builds The allowlist held five subcommands, so every criterion filed against this repository was a grep -- and a grep is satisfied by a file that never builds: 388 of the 837 specs the oracle reaches generate Zig that does not compile while satisfying every criterion ever written against them. `test-report` generates the reviewed spec and has Zig build and run that spec's own tests, which is the oracle's question asked before the oracle. The bee already has a shell in this container, so nothing new is reachable to it; what the allowlist prevents is a criterion rigging its own measurement by running a script from the commit under review. The readers added beside it -- coverage, lint, symbols, outline, validate-vacuity and the rest -- were measured on the corpus and answer in under 30 s. `dupes` (~50 s) and `check-calls` (~140 s) are deliberately refused, with tests pinning the refusal: past the 60 s timeout a criterion is inconclusive, which spends a review and establishes nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(queen): pin what the criteria allowlist admits and refuses Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(agent): the container can run the gates that judge its work Every gate that decides whether a bee's pull request lands is a Python script: tools/dupe_scan.py is the Duplicate Body Ratchet, check_assertionless_spec_tests.py is the corpus ratchet, tools/ci/check_specs_still_parse.py is the required Spec Parse Ratchet, and the twenty-one reporting helpers under scripts/tri_loop/ are Python as well. The image declared git, perl-base and tini and said nothing about python3, while the brief has been telling bees since 2026-09-20 to run `python3 tools/dupe_scan.py --name <fn>`. python3 rather than python3-minimal: those scripts import json, hashlib, difflib and subprocess, and Debian's minimal package carries the subset dpkg's maintainer scripts need. Nothing outside the standard library, so no pip and no venv. `python3 --version` runs in the same layer, so a base image that stops shipping it fails the build instead of the swarm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong
CRITERION_T27C_SUBCOMMANDSheld five names, so every acceptance criterionfiled against gHashTag/t27 was a grep. A grep is satisfied by a file that never
builds: 388 of the 837 specs the oracle can reach generate Zig that does not
compile, and each of them satisfies every criterion ever written against it.
What changes
test-reportis admitted. It generates the reviewed spec and has Zig build and run that spec's own tests -- the oracle's question, asked before the oracle. The bee already has a shell in this container, so nothing here is newly reachable to it; what the allowlist prevents is a criterion rigging its own measurement by running a script from the commit under review, andtest-reportruns the compiler on one named spec.coverage,lint,symbols,outline,exports,metrics,deadcode,orphans,spellcheck,validate-vacuity,xref,api-diff,diff,eval, ...) were measured on the corpus on 2026-09-20: everything excepttest-report(35 s) answers in under 30 s, most in under one.dupes(~50 s) andcheck-calls(~140 s) are refused, with tests pinning the refusal. Past the 60 s per-command timeout a criterion is inconclusive, which spends a review and establishes nothing. A bee's own shell has no allowlist and can run either.Why it is safe
Everything admitted reads a spec and prints. Nothing on the list writes to the
tree, opens a socket, drives hardware, or runs a script from the commit under
review -- the shape that kept
battery,gates,silicon,fmt,renameand
sealoff it, and that the tests continue to pin.Measured
The companion change in gHashTag/t27 (#4301) puts the same commands in front of
the bees that write the specs.
🤖 Generated with Claude Code