[blooop]

Closes #192.

What shipped

The executable-doc guard read bench_launch.py's epilog and never the README — and the README is where the third defect in this lineage actually shipped (python scripts/bench_launch.py, on a host with no python outside pixi, fixed by hand in 5510b1e after publication). The guard now reads both documents.

The ticket's open decision, settled: the README keeps its inline commands and the guard learns to read them. The epilog writes a bare bench_launch.py ... shape sketch; the README writes the host invocation a reader copies, which is where the interpreter lives — and the interpreter is the whole of the third defect. The trend section's command is not in that epilog at all (it belongs to the points script), and the README's copy carries the flag combination in the context that explains it. Deleting the duplication would take both with it.

test/test_bench_doc.py extracts every fenced invocation of a bench script under "Measuring launch time" and hands it to whatever would have caught each defect:

• the interpreter to the two routes this repo can vouch for (python3, or pixi run, whose task brings its own) — no parser will ever see that word;
• the flags to the script's own parser, the epilog guard's seam pointed at the second document;
• the pixi run shortcuts the prose offers beside them to the project manifest, since a renamed task leaves the sentence confidently wrong;
• any --before reset to the same nothing-to-delete harness the epilog's reset goes through, which moves to module scope in test_timing.py so both documents share one starting state.

The section documents no reset today, so that arm holds it to the pointer that stands in for one instead — publish a reset and it is run, publish neither and the page has stopped telling a reader how to bench a cold launch. One further test asserts the extraction is non-empty, because pytest skips a parametrization with no cases: a guard whose document was rewritten out from under it otherwise reads exactly like a guard that passed. (A missing section is deliberately not asserted at import time — that would abort collection for the whole suite instead of failing four tests.)

The rider from #202's review

bench_launch.py's parser is now built with allow_abbrev=False, the way bench_points.py always was. argparse accepts any unambiguous prefix of a long flag, so renaming --record to something it is a prefix of leaves every document saying --record, the parser still accepting it, and the rename shipping behind a green guard. test_no_documented_flag_would_be_accepted_one_letter_short abbreviates each documented flag by one letter and requires the refusal, covering both scripts. No documented or scripted invocation — README, epilog, bench.yml, pixi tasks — used an abbreviation, so nothing that runs today changes.

Mutation checks

Every test was verified to go red against the defect it claims to catch, and the tree restored between each:

mutation	result
README's python3 → python (the defect that shipped)	red: interpreter guard
README's --record → --record-file	red: parse-against-real-parser guard
README's pixi run bench-points → pixi run bench-pts	red: shortcut guard
README's `bench_launch.py --help` pointer deleted	red: reset/deferral guard
README grows --before 'dl-next owner/repo delete' (the historical defect)	red: reset guard, through dl's main()
README grows --before 'dl-next owner/repo rm --force' (correct)	green — the arm is live, not just a refusal
README heading renamed	4 clean failures, no collection abort
allow_abbrev=False reverted	red: --record, --shape
epilog reset regressed to delete	red: the existing guard, through the extracted harness

Checks

pixi run ci exits 0 locally: 1703 passed, 23 deselected in 155s; pylint 10.00/10; ruff and ty clean. python3 scripts/bench_launch.py -n 2 -- true still runs after the parser change.

🤖 Generated with Claude Code

Summary by Sourcery

Extend documentation-based guardrails for benchmarking commands to cover README content and tighten CLI flag validation for bench scripts.

New Features:

• Add tests that extract and validate fenced benchmarking commands from the README, checking interpreter, script paths, flags, pixi shortcuts, and documented resets.

Bug Fixes:

• Guard against README regressions where documented cold-launch recipes, flags, or shortcuts silently disappear or drift from what the bench scripts actually accept.
• Ensure documented --before reset commands are either executed against an empty devpod state or the README still points readers to script help for a valid reset.

Enhancements:

• Refactor the devpod reset harness into a shared module-level helper so both epilog- and README-based reset tests run against the same starting state.
• Disable argparse long-flag abbreviation for bench_launch.py and add coverage that abbreviated documented flags are rejected, aligning behavior with bench_points.py.

Documentation:

• Update the changelog to describe the expanded executable-doc guard over the README and the stricter flag parsing behavior for bench_launch.py.

Tests:

• Introduce test_bench_doc.py to systematically exercise README-based benchmarking commands, pixi tasks, and reset recipes, including failure-on-empty extraction and abbreviation guards.
• Extend timing tests to reuse a common reset harness when validating cold-reset behavior from the script epilog and README-derived commands.

[sourcery-ai]

Sorry @blooop, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Extend the executable-doc guard to validate README’s bench commands alongside the script epilog, introduce a shared reset harness, and harden bench_launch.py’s CLI by disabling argparse abbreviations and testing them via README-derived cases.

Flow diagram for README executable-doc guard validations

flowchart TD
  A["README 'Measuring launch time' fenced bench commands"] --> B[Extract invocations for each bench script]
  B --> C[Check interpreter route is python3 or pixi run]
  B --> D[Parse flags with bench_launch.build_parser]
  B --> E[Validate pixi run shortcut against project manifest]
  B --> F[Run --before reset through shared reset harness]
  F --> G[Call dl.main via reset harness]
  B --> H[Assert at least one invocation extracted so guard cannot silently skip]
  D --> I[Parser rejects abbreviated long flags]
  I --> J[Tests fail if any documented flag is accepted one letter short]

Loading

File-Level Changes

Change	Details	Files
Share a reusable harness for running documented --before resets against an empty devpod state so both epilog and README resets are validated consistently.	Introduce a_reset_from_the_absent_state helper that shells a documented reset, stubs devpod behavior, patches dl helpers, and runs dl.main() Refactor TestTheDocumentedColdReset to use the shared harness instead of inlined devpod/mocking logic Update test docstring to note README commands are now guarded via test_bench_doc.py	test/test_timing.py
Add changelog entry describing README guard coverage and argparse abbreviation disabling for bench_launch.py.	Document that the executable-doc guard now reads README’s fenced bench commands and validates interpreter, flags, pixi shortcuts, and resets Describe allow_abbrev=False change and the new abbreviation-refusal test over all documented flags Clarify historical documented-command defects and how the new guard prevents them	CHANGELOG.md
Harden bench_launch.py’s CLI by disabling long-option abbreviation and updating its parser docstring.	Set allow_abbrev=False on the bench_launch argparse.ArgumentParser Expand build_parser docstring to explain why abbreviations are disabled and align behavior with bench_points.py	scripts/bench_launch.py
Add a new README-focused executable-doc test suite that extracts bench commands from the launch-time section and validates interpreter, flags, pixi shortcuts, resets, and flag abbreviations.	Implement helpers to locate the README bench section, extract fenced commands invoking bench scripts, and derive script names and argv via shlex Import shared section splitter and reset harness to avoid duplicate parsing and ensure consistent reset behavior Parametrize tests over extracted commands to assert vouched-for interpreters, existing scripts, parser-accepted flags, valid pixi run tasks, and non-empty extraction Introduce abbreviation guard that shortens each documented long flag by one letter and requires parser refusal for both bench scripts Add reset behavior test that either runs documented --before resets through the shared harness or asserts the README still points to bench_launch.py --help for the cold recipe	test/test_bench_doc.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#192	Decide whether README should keep its inline benchmarking commands or remove them in favor of pointing to pixi run / --help, and implement that decision in the code/documentation.	✅
#192	If README keeps inline benchmarking commands, extend the executable-doc guard to read README.md as well: extract fenced bench script commands, validate their interpreters and flags against the real parser and project configuration, and drive any documented --before reset through the same absent-state harness used for the epilog.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.80%. Comparing base (a68aab8) to head (94995b3).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #214   +/-   ##
=======================================
  Coverage   92.80%   92.80%           
=======================================
  Files          24       24           
  Lines        3654     3654           
=======================================
  Hits         3391     3391           
  Misses        263      263           

Files with missing lines	Coverage Δ
scripts/bench_launch.py	32.63% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[blooop]

This was generated by AI during review.

Two-axis review of #214 against ticket #192 at merge-base a68aab8. Preflight: all checks green, non-empty diff (4 files, +392/−21), MERGEABLE.

Standards

Nothing blocking. The file passes (17 tests, 0.08s), the naming matches house style (a_document/a_reset_…), and the cross-test-module import is established convention (test_timing already imports test_devpod_spawn_counts; test/conftest.py puts test/ on sys.path).

• The abbreviation guard ignores the -- boundary (non-blocking, sharpest). DOCUMENTED_FLAGS collects every --word in the whole command line, including flags belonging to the benched command after --. Those reach command (nargs=REMAINDER), which accepts anything — verified: parse_args(['-n','5','--','dl-next','owner/repo','rm','--forc']) returns cleanly, so pytest.raises(SystemExit) would fail. No such flag is documented today, but test_the_cold_recipes_reset_… explicitly invites publishing the cold recipe here, whose reset is ... rm --force. Cut at the first bare -- before collecting flags.
• VOUCHED_INTERPRETERS blesses a form another guard in the same file rejects (non-blocking). pixi run python scripts/bench_launch.py … passes the interpreter test, but DOCUMENTED_SHORTCUTS would then yield python and assert it is a pixi task. Two guards, opposite verdicts, on one documented line.
• One harness, two extractions (non-blocking). The run harness is genuinely shared; extraction is not — test_timing.py still regexes --before '([^']+)' out of the epilog, precisely what documented_resets()'s docstring calls "a second parser to be wrong in a second way". The house pattern for shared non-test code is a helper module.
• has_bench_section() guards absence but not duplication (nit). A duplicated heading raises during collection — the session-abort the docstring was written to prevent.
• Primitive Obsession (nit). A command travels as a raw string and is shlex.split four times; splitting once into a small record collapses it and makes the -- fix one line. script_module() re-execs per parametrized case, uncached.
• Proportionality: fine (observation). 320 lines against siblings at 448/202; the module-scope move's blast radius is small.

Spec

Verified: the two suites → 94 passed. Both required mutations reproduced and restored.

• The ticket's open decision, settled as "keep and guard" — sound. Spec: "Deleting the duplication beats guarding it, if the README's inline example earns nothing." It earns something: the epilog writes bare bench_launch.py … with no interpreter, and the interpreter is the whole third defect; scripts/bench_points.py appears in no epilog. Deletion would remove the defect site itself.
• Rider from #202 landed on the real parser and is pinned. scripts/bench_launch.py:268 allow_abbrev=False; removing it turns the --record/--shape abbreviation tests red. No documented invocation used an abbreviation.
• Interpreter defect covered. python3→python in the warm invocation goes red. Minor: the allowlist vouches for any first word pixi, not pixi run <known task>; the shortcut guard covers task names separately.
• Reset defect covered, arm live both directions. --before 'dl-next owner/repo delete' → red; rm --force → green.
• The third named defect is unguarded (non-blocking). Spec: "the -n 5 vs 3-runs mismatch" — nothing couples README's -n 5 to the five runs quoted beside it. Scoped out as "hand-copied numbers", defensible under the ticket's bullet-2 guard list, but it is the one lineage defect a re-run would not catch.
• Harness assumes a reset's first word is dl (non-blocking). --before "scripts/bench_cold_reset.sh owner/repo" — the exact shape bench.yml:158 uses — fails with "which dl refuses". Loud, not silent; the message is wrong.
• Deviations 1, 2, 4 verified/sound — with one count correction: renaming the heading gives 2 failed, 5 skipped, not "4 clean failures"; collection does not abort and the vacuity test is the loud failure.
• CHANGELOG under [Unreleased] → Changed, claims check out.

Verdict

Approve — no blocking findings on either axis. Riders for the next toucher of test_bench_doc.py: cut flag collection at the first bare --; reconcile VOUCHED_INTERPRETERS with DOCUMENTED_SHORTCUTS on the pixi run python form; share the --before extraction; teach the harness the bench_cold_reset.sh reset shape. GitHub refuses --approve from the PR author's account, so this is posted as a comment; this written verdict governs. Merging with a merge commit per repo rules.