Repository navigation
The executable-doc guard reads the epilog but not the README, where the defects actually shipped #192
Description
Question
The executable-doc guard reads bench_launch.py's own epilog and never the README — but the README is where all three documented-command defects actually shipped.
test/test_timing.py's TestTheDocumentedColdReset (line 197) extracts the --before '...' reset from scripts/bench_launch.py's text and drives it through main() — built after the same recipe shipped twice with a delete subcommand dl does not have. That guard works: the epilog's reset cannot regress silently again.
But README.md's own benchmarking section carries its own copies of the commands, and that is where the third defect lived: python scripts/bench_launch.py on a host that only has python3 outside pixi (fixed by hand in 5510b1e, after being published). The guard's docstring names the lesson — "the documented reset is exercised rather than read" — and then reads only one of the two documents.
Worth deciding rather than assuming:
- Does the README duplicate the commands at all, or does it point at
pixi run bench/--helpand let the epilog be the single documented copy? Deleting the duplication beats guarding it, if the README's inline example earns nothing. - If the README keeps inline commands, the existing extraction pattern extends: pull the fenced commands from README.md, assert the interpreter is one the repo can vouch for (
pixi run benchorpython3), and drive any--beforereset through the same absent-state harness.
History: three documented-command-never-run defects in #140's lineage — the nonexistent delete subcommand (twice), the -n 5 vs 3-runs mismatch, and the python interpreter. Each was found by running the documentation. Found during PR #190's benchmark validation; recorded in review notes and never ticketed.
Activity
blooop commentedon Aug 15, 2026
breadcrumb: Claimed, building. Restating the contract as I read it.
The open decision (bullet 1), settled: the README keeps its inline commands, and the guard learns to read them. The duplication is not redundant. The epilog writes the invocations as a bare
bench_launch.py ...shape sketch; the README writes the host invocation a reader actually copies, which is where the interpreter lives — and the interpreter is the whole of the third defect. The trend section's command is not in that epilog at all (it belongs to the points script) and the README's copy carries the flag combination in the context that explains it. Deleting the duplication would take the interpreter and that combination with it, so guarding beats deleting here.Seams I will test at, all extraction-then-execute, the same recipe the epilog guard already uses:
python3on the host, or apixi run <task>whose task really exists in the project manifest. A barepythonis refused by name: that is the defect that shipped, on a host with nopythonoutside pixi.--recordas a prefix of a hypothetical renamed--record-path, so a rename slips past a green guard. Both bench parsers must refuse abbreviations; one already does, one does not.--beforethe README shows goes through the same nothing-to-delete harness the epilog's reset does. The README documents none today — this arm is the latch for the day one is added, and the guard that finds the commands is asserted non-empty so an emptied section fails loudly instead of quietly guarding nothing.Not in scope: rewriting the section's prose, and the hand-copied numbers in it — those are a claim about a host, not a command a guard can run.
blooop commentedon Aug 15, 2026
breadcrumb: PR #214 opened.
The guard now reads both documents. Every fenced invocation of a bench script under the launch-time section of the README is extracted and driven through what would have caught each defect in this lineage: the interpreter against the two routes this repo can vouch for, the flags against the script's own parser, the
pixi runshortcuts beside them against the project manifest, and any--beforereset through the same nothing-to-delete harness the epilog's reset already goes through — which moved to module scope so one starting state serves both documents. The section shows no reset today, so that arm holds it to the pointer that stands in for one instead; publish a reset there and it starts being run instead.Two things worth keeping in mind for whoever reads this next. The extraction is asserted non-empty in a test of its own, because pytest skips a parametrization with no cases — a guard whose document was rewritten out from under it otherwise reads exactly like a guard that passed. And the missing-heading case is deliberately not asserted while the module is being collected: doing so aborts the whole suite rather than failing the four tests that lost their document.
The rider from #202's review is folded in: the launch script's parser now refuses abbreviations, the way the points script always has. An abbreviating parser is what made these guards weaker than they read — rename a flag to something the old spelling is a prefix of and every document keeps the old spelling, the parser keeps accepting it, and the rename ships green. Each documented flag is now cut a letter short and required to be refused. Nothing that runs today used an abbreviation.
Every test was mutation-checked against the defect it claims to catch, including the two that shipped verbatim (
pythonfor the interpreter, adeletereset) and the correct-reset case, which goes green rather than merely not-red. Table of the nine mutations is in the PR body.blooop commentedon Aug 15, 2026
breadcrumb: PR #214 is green — every check passes, including the whole 3.10–3.13 matrix and e2e. Build stage ends here; the PR is ready for review in fresh context.
blooop commentedon Aug 15, 2026
wf-review verdict on #214: Approve — both axes clean; Spec axis reproduced both required mutations, confirmed the #202 rider (allow_abbrev=False) is pinned, and judged the keep-and-guard settlement sound. Riders recorded in the review (
--boundary in flag collection, bench_cold_reset.sh reset shape, shared --before extraction). Merging — this completes the original autonomous lane.