Skip to content

A criterion may ask whether the spec builds - #488

Merged
gHashTag merged 3 commits into
feat/queen-supervisorfrom
feat/criteria-toolbelt
Sep 20, 2026
Merged

gHashTag merged 3 commits into
feat/queen-supervisorfrom
feat/criteria-toolbelt

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

What is wrong

CRITERION_T27C_SUBCOMMANDS held five names, so every acceptance criterion
filed against gHashTag/t27 was a grep. A grep is satisfied by a file that never
builds: 388 of the 837 specs the oracle can reach generate Zig that does not
compile
, and each of them satisfies every criterion ever written against it.

What changes

  • test-report is admitted. It generates the reviewed spec and has Zig build and run that spec's own tests -- the oracle's question, asked before the oracle. The bee already has a shell in this container, so nothing here is newly reachable to it; what the allowlist prevents is a criterion rigging its own measurement by running a script from the commit under review, and test-report runs the compiler on one named spec.
  • The readers beside it (coverage, lint, symbols, outline, exports, metrics, deadcode, orphans, spellcheck, validate-vacuity, xref, api-diff, diff, eval, ...) were measured on the corpus on 2026-09-20: everything except test-report (35 s) answers in under 30 s, most in under one.
  • dupes (~50 s) and check-calls (~140 s) are refused, with tests pinning the refusal. Past the 60 s per-command timeout a criterion is inconclusive, which spends a review and establishes nothing. A bee's own shell has no allowlist and can run either.

Why it is safe

Everything admitted reads a spec and prints. Nothing on the list writes to the
tree, opens a socket, drives hardware, or runs a script from the commit under
review -- the shape that kept battery, gates, silicon, fmt, rename
and seal off it, and that the tests continue to pin.

Measured

$ t27c lint specs/git/diff.t27 | head -1
WARN: fn 'diff' has no test or invariant
$ t27c test-report specs/git/diff.t27 2>&1 | grep -c BLOCKED
1

The companion change in gHashTag/t27 (#4301) puts the same commands in front of
the bees that write the specs.

🤖 Generated with Claude Code

gHashTag and others added 2 commits September 20, 2026 16:23
The allowlist held five subcommands, so every criterion filed against this
repository was a grep -- and a grep is satisfied by a file that never builds:
388 of the 837 specs the oracle reaches generate Zig that does not compile
while satisfying every criterion ever written against them.

`test-report` generates the reviewed spec and has Zig build and run that
spec's own tests, which is the oracle's question asked before the oracle. The
bee already has a shell in this container, so nothing new is reachable to it;
what the allowlist prevents is a criterion rigging its own measurement by
running a script from the commit under review.

The readers added beside it -- coverage, lint, symbols, outline, validate-vacuity
and the rest -- were measured on the corpus and answer in under 30 s. `dupes`
(~50 s) and `check-calls` (~140 s) are deliberately refused, with tests
pinning the refusal: past the 60 s timeout a criterion is inconclusive, which
spends a review and establishes nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

❌ Tests failed — 11/2431 failed

Suite Passed Failed Skipped
✅ agent 87/87 0 0
✅ build 9/9 0 0
✅ cdp-protocol 5/5 0 0
✅ eval 93/93 0 0
✅ server-agent 272/272 0 0
❌ server-api 1248/1311 6 57
✅ server-browser 6/6 0 0
✅ server-integration 10/11 0 1
✅ server-lib 279/279 0 0
❌ server-pglive 1/3 2 0
✅ server-root 68/68 0 0
✅ server-skills 31/31 0 0
❌ server-tools 239/242 3 0
✅ shared 14/14 0 0
Failed tests
  • server-api — the note says who committed the branch, where it can be read > puts the salvage fact where the 1500-character cap cannot cut it
  • server-api — a turn the provider killed does not spend the issue > is judged and sent back, but charges neither counter
  • server-api — deriving work the repository already measured > carries the command that produced it
  • server-api — deriving work the repository already measured > gives every candidate a path that can be a boundary
  • server-api — deriving work the repository already measured > proposes nothing from code this project does not own
  • server-api — salvaging a turn that ended with its work uncommitted > refuses a worktree holding an unmerged path
  • server-pglive — the migration block, applied to a real PostgreSQL > creates every object it promises, and survives a second boot
  • server-pglive — the live gate > ran, or its absence is on the record
  • server-tools — navigation tools > new_hidden_page opens a hidden tab
  • server-tools — navigation tools > show_page restores a hidden page to visible
  • server-tools — window tools > create_hidden_window creates and closes a hidden window

View workflow run

Every gate that decides whether a bee's pull request lands is a Python script:
tools/dupe_scan.py is the Duplicate Body Ratchet, check_assertionless_spec_tests.py
is the corpus ratchet, tools/ci/check_specs_still_parse.py is the required Spec
Parse Ratchet, and the twenty-one reporting helpers under scripts/tri_loop/ are
Python as well. The image declared git, perl-base and tini and said nothing
about python3, while the brief has been telling bees since 2026-09-20 to run
`python3 tools/dupe_scan.py --name <fn>`.

python3 rather than python3-minimal: those scripts import json, hashlib,
difflib and subprocess, and Debian's minimal package carries the subset dpkg's
maintainer scripts need. Nothing outside the standard library, so no pip and no
venv. `python3 --version` runs in the same layer, so a base image that stops
shipping it fails the build instead of the swarm.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gHashTag
gHashTag merged commit d260c63 into feat/queen-supervisor Sep 20, 2026
12 of 16 checks passed
@github-actions
github-actions Bot deleted the feat/criteria-toolbelt branch September 27, 2026 04:43
dmitrii-f-t27 pushed a commit to dmitrii-f-t27/BrowserOS that referenced this pull request Oct 10, 2026
…n the gates (gHashTag#488)

* feat(queen): a criterion may ask whether the spec builds

The allowlist held five subcommands, so every criterion filed against this
repository was a grep -- and a grep is satisfied by a file that never builds:
388 of the 837 specs the oracle reaches generate Zig that does not compile
while satisfying every criterion ever written against them.

`test-report` generates the reviewed spec and has Zig build and run that
spec's own tests, which is the oracle's question asked before the oracle. The
bee already has a shell in this container, so nothing new is reachable to it;
what the allowlist prevents is a criterion rigging its own measurement by
running a script from the commit under review.

The readers added beside it -- coverage, lint, symbols, outline, validate-vacuity
and the rest -- were measured on the corpus and answer in under 30 s. `dupes`
(~50 s) and `check-calls` (~140 s) are deliberately refused, with tests
pinning the refusal: past the 60 s timeout a criterion is inconclusive, which
spends a review and establishes nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(queen): pin what the criteria allowlist admits and refuses

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(agent): the container can run the gates that judge its work

Every gate that decides whether a bee's pull request lands is a Python script:
tools/dupe_scan.py is the Duplicate Body Ratchet, check_assertionless_spec_tests.py
is the corpus ratchet, tools/ci/check_specs_still_parse.py is the required Spec
Parse Ratchet, and the twenty-one reporting helpers under scripts/tri_loop/ are
Python as well. The image declared git, perl-base and tini and said nothing
about python3, while the brief has been telling bees since 2026-09-20 to run
`python3 tools/dupe_scan.py --name <fn>`.

python3 rather than python3-minimal: those scripts import json, hashlib,
difflib and subprocess, and Debian's minimal package carries the subset dpkg's
maintainer scripts need. Nothing outside the standard library, so no pip and no
venv. `python3 --version` runs in the same layer, so a base image that stops
shipping it fails the build instead of the swarm.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant