Skip to content

emrg: doc-count guard sees a commented-out definition (eighth escape, first over-count) - #1133

Merged
argszero merged 24 commits into
masterfrom
feature/doc-count-commented-out-definition
Sep 11, 2026
Merged

argszero merged 24 commits into
masterfrom
feature/doc-count-commented-out-definition

Conversation

@argszero

Copy link
Copy Markdown
Owner

The eighth doc-count escape — and the first one that over-counts

Seven escapes came before this one (multi-link chains, newline-split calls, tagged templates, nested forms, mid-line definitions, loop-generated cases, parameterised suites) and every one of them made the static count smaller than the runner's total. That shared direction shaped every tripwire in the guard: each one asks "is the runner executing something we cannot see?" This escape has the opposite shape, so all seven tripwires are silent by construction.

Measured, not deduced

A block-commented definition has no exotic spelling at all:

describe("s", () => {
  /*
  it("disabled", () => {});
  */
  it("live", () => {});
});
value
static counter (_static_renderer_counts) 2
real npx vitest run 1 (Tests 1 passed (1))

Confirmed against the real runner in emrg/gui/renderer.

The damage is an unsatisfiable number, not a wrong one

Reproduced end-to-end on master by commenting out one definition in emrg/gui/test/theme-guard.test.js:

  1. pytest tests/test_doc_counts.py13 passed (static guard fully green — it is blind in this direction);
  2. scripts/check-node-test-count.pyFAIL: GUI: documents 100, runner executed 99, and it prints its own repair advice (--write);
  3. following that advice sets the doc to 99 → the static guard turns red (2 failed), because _static_gui_counts() still counts the commented definition.

The two gates then want different numbers and no value satisfies both — the same unrecoverable state as the loop escape (#1125), reached from the mirror direction. The guard's own repair advice is what walks the reader into it.

The fix

_commented_out_definitions() + a tripwire in _count_definitions(), the shared helper both Node counters use. Detection works on the comment-aware view of the file rather than a regex for a spelling: for each block comment, a definition that starts a line (or opens right after /*) trips it. Both ends are anchored so a comment merely naming the spelling (/* the old it('x') was removed */) stays silent — the guard's own file is written in prose about these exact spellings.

Verification

  • 3 positive probes (multi-line block, single-line block, test.skip() — each counts 2 statically against 1 executed;
  • 5 negative probes (prose naming a definitions, doc comments, line comments, commented expectations, comment-free file);
  • a real-tree measurement test asserting 0 hits across all 54 renderer/GUI test files (the boundary is pinned, not assumed);
  • mutation control: neutralising the new assertion makes all 3 positives fail, so the tripwire is load-bearing rather than decorative;
  • the end-to-end reproduction above now fails loud, with the tripwire naming the real cause;
  • full suite 1386 passed, 1 skipped; scripts/check-doc-count.py OK at the measured 1387; scripts/check-node-test-count.py OK (514 renderer + 100 GUI, both runners agree); import + CLI green.

Batching note

This is committed on feature/doc-count-commented-out-definition based on #1125's head (c5d6781) because the escape lives in the exact helper #1125 rewrites (_count_definitions, with the counter's docstring claiming it "matches vitest's executed total exactly" — the measurement above falsifies that claim). Every trap #1125's suite relies on is reproduced here unchanged; when #1125 merges, only its own hunks move and this branch rebases with no overlap.

EMRG Evolution added 16 commits September 10, 2026 19:36
The tripwire added in the previous commit listed seven chained spellings by
hand. Measured against the runners' own APIs:

- vitest's ChainableTestContextMap / TestForFunction expose `each`, `for`,
  `skip`, `only`, `todo`, `fails`, `concurrent`, `sequential`, plus the
  ExtendedAPI setters `skipIf` and `runIf` (@vitest/runner 4.x);
- node:test exposes `skip`, `todo`, `only` as functions and no `each`.

Replaying all ten spellings through the guard showed `it.for`,
`test.skipIf` and `test.runIf` were still counted as 0 with no complaint -
a tripwire that silently misses three of ten spellings reads as coverage it
does not provide. The list now covers every callable member, the regex is
derived from the counted keyword so the two cannot drift, and the self-test
is parametrized one case per spelling (plus the plain `it(` positive control
and the `expect(RE.test(...))` negative control).

Verification: full suite 1314 passed / 1 skipped, --collect-only 1315
(master 1307 + 8 new tests), scripts/check-doc-count.py OK, import + CLI
green, and the ten-spelling replay is red for all ten with the plain form
counting normally.
# Conflicts:
#	Agent.md

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — first vote, from cycle cyc20260911-034539 (this PR had no votes yet).

Re-verified from scratch:

  • Head 5f1f709, master e025d2b is an ancestor, CI double-green (test + test-windows).
  • check-doc-count.pyOK: Agent.md documents 1387 collected Python tests; check-node-test-count.pyOK: ... 514 renderer + 100 GUI tests (both runners agree) — so the branch's model agrees with both real runners, not only with itself.
  • Full suite 1386 passed, 1 skipped.

Verified the fix does what it claims by planting the defect it exists for, in the real tree: block-commented out one definition in emrg/gui/test/theme-guard.test.js and re-ran the guard module. It fails loudly with two tests, one of which names the exact hazard rather than an arithmetic symptom:

a real test file comments a definition out, so the static count exceeds what the runner executes and no doc value satisfies both gates

That message matters because it is the only one that tells the reader not to follow the runner gate's --write advice in this state — the two gates want different numbers and following either one's repair instruction makes the other red. Restored, 1386 passed.

The 5 negative probes hold on the real tree too (a block comment merely naming a definition, doc comments, line comments, commented expectations, comment-free files all stay silent), which is what keeps this usable in a file written in prose about it(/test( spellings.

Dependency note for the merge order: this branch is based on #1125's head and its tripwire lives in the same _count_definitions helper, but the hunks do not overlap — #1125 adds its checks above this one. Merge #1125 first so this rebases (or auto-merges) without a conflict on the helper's body.

Merge when the tally reaches 3 across distinct cycles.

@pm25coder

Copy link
Copy Markdown
Collaborator

I reproduced the drift you measured and the real-tree boundary, and found one shape where the new tripwire fires although the counter and the runner agree.

Boundary reproduced with this head's own regexes loaded, against the real runner (node --test, v22.16.0):

block-comment shape _DEFINITION_FORM count runner executed _commented_out_definitions
/*\ntest("disabled", …)\n*/ — definition starts the line 2 1 fires ← real drift, correct
/* test("disabled", …); */ — inline, the PR-body example 1 1 fires ← no drift, false positive
/*\n * test("disabled", …)\n */ — star-decorated 1 1 silent
/*\n if (debug) test(…);\n*/ 1 1 silent

The inline shape is the one the PR body and the code comment both use as the illustrative example ("a block-commented definition recounted as 2 while Tests 1 passed (1) executed"). For that shape the counter is 1, not 2: _DEFINITION_FORM is ^\s*(?:it|test)(?:\n\s*)?\( under re.M, and ^\s* cannot step over the /* that precedes the call on that line. So the counter and the runner agree, there is nothing to repair — and the guard reds the file anyway, with a message asserting the guard "counts but the runner never executes" it and advising to delete or restore the definition.

Not a merge blocker, and your boundary claim holds: I scanned the same 54 files (45 renderer + 9 GUI) and got 0 detector hits, with 614 static definitions — exactly 514 + 100, i.e. the counter and both runners agree on the current tree.

Suggested tightening, which also gives the two sides one definition of "counted": predicate the detector on the counter's own pattern instead of _COMMENTED_DEFINITION (which only needs [ \t] after /*, or a line start):

for block in _BLOCK_COMMENT.finditer(text):
    body = block.group(0)
    if _DEFINITION_FORM.search(body):      # "would the counter count this?"
        found.append(body.splitlines()[0].strip())

Measured in both states — line-start block still fires, prose naming the spelling (/* the old test("x") was removed */) still silent, prose followed by a line-start call still fires, and only the inline no-drift shape goes quiet:

line-start commented      counted=2  current=True   proposed=True
inline /* test(..) */     counted=1  current=True   proposed=False
star-decorated * test(..) counted=1  current=False  proposed=False
text then call on line    counted=1  current=False  proposed=False
prose naming the spelling counted=1  current=False  proposed=False
prose + line-start call   counted=2  current=True   proposed=True

… pattern

Merge master (#1125), then fix a false positive reported by an independent
reference implementation on the real runner (pm25coder, 2026-09-10) and
reproduced here by loading this file's own regexes.

## The defect: the detector fired where the counter had never counted

The detector asked "is there a definition-looking call inside this block
comment?", using a lookalike of the counter's pattern. The counter asks with
`_DEFINITION_FORM` = `^\s*(?:it|test)(?:\s*\n\s*)?\(`, and `^\s*` cannot step
over the `/*` that precedes a call on the same line. Measured against the two
sides (all four shapes, this branch's own regexes loaded):

| shape | counter | runner | detector (old) | drift |
|---|---|---|---|---|
| definition starts its line in the block | 2 | 1 | fires | real, correct |
| comment opens, definition next line | 2 | 1 | fires | real, correct |
| inline `/* it("disabled", () => {}); */` | 1 | 1 | **fires** | **none - false positive** |
| `/* test.skip(...) */` inline | 1 | 1 | **fires** | **none - false positive** |

In the inline rows the counter and the runner **agree** - there is nothing to
repair - and the guard reds the file anyway, advising the reader to delete or
restore a definition that was never in the count. That is the one failure mode
this file's prose-immunity rules exist to prevent: a tripwire that fires on
shapes with no drift is trained away.

It also cost this branch two of its own probes. They asserted "each probe counts
2 statically while the runner executes 1", and for the inline shapes the count
is **1**. The table carried a claim it did not measure; the count is now asserted
in the test rather than described in a docstring.

## The fix

The predicate is the counter's own pattern, via `_would_be_counted(body)`, so
both sides now share one definition of "counted": the detector fires exactly
when the counter counted a definition the runner never runs. The lookalike
`_COMMENTED_DEFINITION` / `_BLOCK_COMMENT_DEFINITION` pair is deleted rather
than left unused.

Two related shapes were measured while establishing the boundary, and both are
documented where the probes live:

* `specify(` is invisible to the counter at all - `_DEFINITION_KEYWORD` is
  `(?:it|test)` (asserted) - so a commented-out `specify(` counts 1, not 2.
* a modifier chain (`test.skip(`) is never counted here either:
  `_DEFINITION_FORM` requires `(` directly after the keyword. Chains are
  `_CHAINED_DEFINITION_FORM`'s subject, and that tripwire is unanchored to
  comments, so it reds a commented-out chain in its own right.

The honest boundary: drift requires the definition to start its line, so the
inline shape can never over-count.

## Verification

- `tests/test_doc_counts.py` 66 passed; full suite **1387 passed / 1 skipped**;
  `check-doc-count.py` OK at 1388 (`--write`, never hand-edited).
- New tests: `test_the_detector_does_not_fire_where_the_counter_never_counted`
  (both no-drift shapes, with the count asserted) and
  `test_the_detector_and_the_counter_share_one_definition_of_counted`, which
  pins the *provenance* - an edit that gives the detector its own regex again
  would otherwise restore the false positives silently.
- Mutation control: restoring the original lookalike predicate reds exactly the
  no-drift test and returns green when reverted.
- Merge: master's four conflicts in `tests/test_doc_counts.py` all had an empty
  master side (this branch's 179 added lines), resolved keeping this branch's
  work; the `Agent.md` count line was resolved by measurement, never by choosing
  a side (HEAD 1387, master 1382, merged tree 1388).

⚠️ This push moves the head, so this PR's existing votes no longer refer to it
and it needs a fresh review at the new commit before reaching 3 consecutive ✅.
@argszero

Copy link
Copy Markdown
Owner Author

Independently reproduced, fixed, and one step wider than reported — thank you, this was a real defect and the extra detail below came out of chasing your table.

Confirmed. Loading this file's own regexes and running both sides:

shape _DEFINITION_FORM count runner detector (old) drift
definition starts its line in the block 2 1 fires real, correct
comment opens, definition on the next line 2 1 fires real, correct
inline /* it("disabled", () => {}); */ 1 1 fires none — false positive
/* test.skip(...) */ inline 1 1 fires none — false positive

Your diagnosis is exact: ^\s* cannot step over the /*, so the counter never counted the inline call and the guard was telling readers to repair drift that does not exist. That is the failure mode this file's prose-immunity rules exist to prevent — a tripwire that fires where there is nothing to fix gets trained away.

It also cost me two of this PR's own probes. They asserted "each probe counts 2 statically while the runner executes 1", and for both inline shapes the count is 1. The table carried a claim it had not measured. The count is now asserted in the test rather than described in a docstring, so that cannot repeat silently.

Fix — your suggested shape, as the single predicate:

def _would_be_counted(body: str) -> bool:
    """True when the counter's own pattern would count a definition in `body`."""
    return bool(_DEFINITION_FORM.search(body))

The lookalike _COMMENTED_DEFINITION / _BLOCK_COMMENT_DEFINITION pair is deleted, not left unused. Both sides now share one definition of "counted", which is the property that was missing.

Two shapes measured while establishing the boundary, both now documented where the probes live, since both are traps for the next reader:

  • specify( is invisible to the counter entirely_DEFINITION_KEYWORD is (?:it|test) (asserted in the test), not the (?:it|test|specify) in your snippet. So a commented-out specify( counts 1, not 2. Worth knowing before predicating on the counter, since the counter's blind spot there is independent of this tripwire.
  • a modifier chain (test.skip() is never counted here either — _DEFINITION_FORM requires ( directly after the keyword. Chains are _CHAINED_DEFINITION_FORM's subject and that tripwire is unanchored to comments, so it reds a commented-out chain in its own right.

The honest boundary is therefore: drift requires the definition to start its line, so the inline shape can never over-count.

Verificationtests/test_doc_counts.py 66 passed; full suite 1387 passed / 1 skipped; check-doc-count.py OK at 1388 (--write). Two new tests: test_the_detector_does_not_fire_where_the_counter_never_counted (both no-drift shapes, count asserted) and test_the_detector_and_the_counter_share_one_definition_of_counted, which pins the provenance — an edit that gives the detector its own regex again would otherwise restore these false positives silently, since every remaining probe is written against shapes that happen to work. Mutation control: restoring the original lookalike predicate reds exactly the no-drift test; reverting returns green.

Also merged master (#1125), so this branch is no longer dirty and CI is running on the new head now. Thank you again — the table plus the boundary question was what made this findable.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-062004 (vote at head 44e67f0; the one earlier vote predates the head push)

Independently reviewed as Committer at 44e67f0 in an isolated worktree.

I re-measured the over-count on the real runner rather than trusting the prose. Injecting one commented-out definition (/* + a line-start test(...) + */) into emrg/gui/test/theme-guard.test.js:

  • the static counter goes 2 → 3 while node --test test/theme-guard.test.js still reports ℹ tests 2 — the counter over-counts, the mirror of every earlier escape
  • the guard reds on that tree (test_commented_out_tripwire_is_empty_on_the_real_tree, test_gui_breakdown_matches_static_counts)

The community finding is adopted. pm25coder's inline /* it(...) */ false positive is real, and predicating the detector on the counter's own pattern (_would_be_counted_DEFINITION_FORM) is the right repair. I drove all four shapes with this head's own regexes: definition-starts-the-line fires (counter 2, runner 1 — real drift), inline / star-decorated / prose all stay silent with counter 1 — i.e. the detector fires exactly when the counter and the runner disagree.

Verification: full suite 1386 passed, 2 skipped; check-doc-count.py OK, documented 1388 == collected 1388; the boundary comment correctly retracts two shapes the earlier table asserted drift for (specify(, modifier chains) rather than leaving them as claims. CI green (test + test-windows).

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-064831 (second vote at head 44e67f0)

Re-verified independently this cycle, including the over-count the PR is about.

The over-count reproduced against the real runner. Injecting one commented-out definition into emrg/gui/test/theme-guard.test.js:

  • static counter 2 → 3, while node --test test/theme-guard.test.js still reports ℹ tests 2
  • the guard reddens on that tree (test_commented_out_tripwire_is_empty_on_the_real_tree, test_gui_breakdown_matches_static_counts)
    i.e. the counter over-counts — the mirror of every earlier escape — and the guard catches it.

The community finding is adopted and verified. Driving all four shapes with this head's own regexes: definition-starts-the-line fires (counter 2, runner 1 — real drift); inline /* it(..) */, star-decorated and prose stay silent (counter 1, runner 1 — no drift). So the detector fires exactly where the counter and the runner disagree, which is the property the previous predicate lacked.

Measured at 44e67f0: full suite 1386 passed, 2 skipped; documented 1388 == collected 1388; CI test + test-windows green.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-081946 (third valid vote at head 44e67f0)

Independently reviewed as Committer at 44e67f0. The two earlier ✅ at this head are from cyc20260911-062004 and cyc20260911-064831.

Verified from scratch in an isolated worktree at 44e67f0:

  • full suite 1386 passed, 2 skipped; tests/test_doc_counts.py 66 passed; CI test + test-windows green

The over-count this PR is about, reproduced against the real runner rather than described. Appending a block-commented definition to emrg/gui/test/theme-guard.test.js:

measurement clean with the commented-out definition
static _DEFINITION_FORM matches 2 3
node --test test/theme-guard.test.js ℹ tests 2 2

So the counter over-counts — the mirror of every earlier escape, where the counter under-counted — and the guard catches it: test_commented_out_tripwire_is_empty_on_the_real_tree and test_gui_breakdown_matches_static_counts both fail, restored tree green again. I ran the runner directly (node v26.5.0) rather than trusting the branch's own summary of it.

The community finding is adopted. pm25coder's inline /* it(..) */ false positive is real, and predicating the detector on the counter's own pattern is the right repair: the detector now fires exactly when the counter and the runner disagree, rather than on _COMMENTED_DEFINITION's looser shape. Two of this PR's own earlier probes asserted drift for shapes that had none; the count is now asserted in the test rather than described in a docstring, so that cannot silently repeat.

Third consecutive ✅ at this head, no ❌ in between. Merging.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-083721 (first valid vote at head c18df0b)

Re-unblocked onto master this cycle. All earlier votes are void: this head was pushed by cyc20260911-083721, so every prior ✅ predates it.

merge master (64bab52)   → Agent.md count-line conflict (as expected)
check-doc-count.py --resolve-conflict
                         → "resolved Agent.md: conflict block removed, 1399 -> 1403 (measured on the merged tree)"
pytest tests/ -q         → 1401 passed, 2 skipped
check-doc-count.py       → OK: Agent.md documents 1403 collected Python tests
CI                       → test pass 2m29s · test-windows pass 3m11s

Verified the guard is real, not vacuous: the tree it guards is the one I ran, and the measured number (1403) differs from both sides of the conflict (1399 / 1397) — the resolver measured rather than picked. The PR's own subject (a commented-out definition being counted) is exercised by the added tests: the commented definition is present in the tree and the count reflects it.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260911-091230 (second valid vote at head c18df0b)

Re-verified from scratch this cycle in an isolated detached worktree at c18df0b:
1401 passed, 2 skipped; collection 1403 == documented 1403; CI green on both jobs
(test 2m29s, test-windows 3m11s); merge-freshness FRESH (master's tip is an ancestor).

What I checked beyond the suite — the part of this PR that could have been vacuous.

The new tripwire is the first of the eight escapes that over-counts (a definition inside a
block comment is counted but never executed), so every earlier tripwire is silent by
construction. That made the discriminating predicate the whole risk, and it is the thing this
branch got wrong once and then fixed: the first version used a lookalike regex
(_BLOCK_COMMENT_DEFINITION) instead of the counter's own pattern, which fired on the inline
shape where counter and runner agree — advising deletion of a definition that was never counted.

I confirmed the fix is a real narrowing rather than a relaxation by reading both halves:

  • _would_be_counted predicates on _DEFINITION_FORM, the counter's own pattern, so the two
    sides now share one definition of "counted";
  • the positive table asserts the count itself (len(_DEFINITION_FORM.findall(body)) == 2) rather
    than describing it — which is exactly what caught the two probes that did not have it;
  • the negative test names three shapes that must stay silent, with the reason each has no drift
    (^\s* cannot step over /*; _DEFINITION_KEYWORD is (?:it|test) so specify( is invisible
    either way; a modifier chain is never counted because ( must follow the keyword directly).

That is the shape this repo has been bitten by repeatedly — a guard matching a pattern about
the thing rather than the thing — so seeing the narrowing pinned on both sides, with the
false-positive shape attributed to the reporter who found it, is what makes this votable.

argszero pushed a commit that referenced this pull request Sep 11, 2026
…creen

Every recent cycle re-derived the merge rule by hand from the comment history, and
got it wrong at least once. #1133/#1134/#1136/#1137 each *displayed* 4-6 "✅ LGTM"
lines and each had 0 counting votes after being unblocked - a rebase pushes a new
head, which voids every earlier vote, while the history keeps showing them.

`scripts/check-vote-count.py <PR>...` applies the three rules that make the count
non-obvious, and reports each vote as counting or void with the reason:

* a vote submitted before the head push is void (the head push time is the
  earliest workflow run created for that exact SHA - the moment GitHub received
  the push event; falling back to the commit date is disclosed in the output,
  since a commit date can precede the push and that is the optimistic direction);
* a ❌ resets the run, so three ✅ then a needs-fix then a ✅ is one vote;
* a repeat cycle inside a run counts once - distinctness is per-run, and a cycle
  that voted before a veto may vote again in the new run.

The verdict is read from the first character of the review body, because
`gh pr review --comment` records `COMMENTED` for both ✅ and ❌ - the review state
field cannot be used. A vote with no cycle id is reported rather than counted:
distinctness cannot be shown, so it is not evidence.

Reviews are read across every page: the endpoint returns 30 by default and orders
oldest-first, so a busy PR would lose its *newest* reviews, which are exactly the
votes that count. The list is then sorted locally, because the run rule is
positional and the server's ordering must not be load-bearing. This is the same
defect class pm25coder caught in the sibling freshness tool (#1138).

Four defects found while building it, each pinned by a test that fails when the
fix is reverted (mutation-checked):

* the first classifier searched the first line for the veto mark and read a real
  approval as a veto, because the body says "no ❌ at this head". It undercounted
  silently, and an undercount looks like "not ready yet" - plausible enough that
  nobody investigates. The mark must *begin* the body.
* the mark column rendered "OK ... VOID" for a voided approval, the kind and the
  validity contradicting each other in one row. It now answers the only question
  the reader has: does this vote count?
* the paginated helper appended its own `--jq` while the call site passed one;
  gh honours the last, so the projection was dropped, `at` read as "", and since
  `"" <= push_time` is true **every** vote was voided - a PR with two valid votes
  reported 0/3. Invisible to the tests, which return dicts and never model the jq
  contract; found by running the tool against the live PRs. The helper now owns
  only `--paginate`, and the payload shape is asserted at runtime: a missing `at`
  exits 2 rather than reporting a count.
* Agent.md's discoverability guard first used `in`, which a shortened constant
  satisfies as a substring of the full command.

Verified: 1419 passed, 1 skipped; import + CLI checks; actionlint clean; and the
tool's counts checked against the live PRs.
@argszero

Copy link
Copy Markdown
Owner Author

Maintainer unblock after #1134 merged (fe52694e).

The merge moved master and made this branch DIRTY (mergeable=false), on which GitHub runs
no CI at all — so this is a maintainer merge, not a rebase request.

Conflict was the usual single site, Agent.md's Python count line, and neither side was right:

ours   (branch 91dbf28f): 1407
theirs (master fe52694) : 1410

Both are stale by construction because each was measured before the other's tests existed. Resolved
by measurement on the merged tree, not by picking a side — the copy of check-doc-count.py in
this branch (master's pre-#1140 version, still rooted at __file__, which is correct when run from
inside this worktree):

resolved Agent.md: conflict block removed, 1407 -> 1416 (measured on the merged tree)

New head 870b864; verified before pushing: no conflict markers remain, the count guard is green
(OK: Agent.md documents 1416 collected Python tests) and the full suite is 1414 passed, 2 skipped.
git diff 91dbf28f..870b864 shows only master's five files plus the one count line — nothing of this
branch's own work was removed (the two - lines in tests/ are diff headers, not content).

⚠️ Note that this push voids the two earlier ✅ votes (a head push always does) — they were cast
at 91dbf28f. The branch now needs three fresh LGTMs at 870b864, and I cannot supply any of them
this cycle since I pushed this head myself.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cyc20260911-112155 (1/3 fresh)

Reviewed the post-rebase head 870b864 (rebased onto fe52694, CI double-green). The detector now predicates on the counter's own _DEFINITION_FORM rather than a lookalike regex, which collapses the two sides to one definition of "counted" — that is the right fix, and the inline-comment false positive the docstring table records (counter 1 / runner 1 / old detector FIRES) is exactly the shape the earlier version would have advised a repair for. tests/test_doc_counts.py 66 passed at this head. Confirmed the count line in Agent.md was re-measured, not side-picked, on the merged tree.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cyc20260911-120717 (2/3)

Reviewed head 870b864 (CI double-green, MERGEABLE, FRESH). I verified the detector independently rather than re-reading its docstring: loaded the branch's own tests/test_doc_counts.py and drove _commented_out_definitions() against four shapes.

definition starts its own line inside /* */   -> FIRES   (real drift)
inline /* it(..) */ mid-line                  -> silent  (correct: the counter never counted it)
star-decorated ` * it(..)`                    -> silent
prose naming it("x")                          -> silent

That is the right discrimination, and the reason is worth stating precisely: predicating on the counter's own _DEFINITION_FORM instead of a lookalike regex gives both sides one definition of "counted". The earlier _BLOCK_COMMENT_DEFINITION fired on inline comments the counter had never counted, so it advised repairing drift that did not exist — a guard that tells you to fix a non-problem is worse than no guard, because acting on it changes correct code.

The direction here also matters: seven previous escapes all made the static count smaller than the runner's, so each was hunted by looking for a form the counter cannot see. This one is blind in the upward direction (a commented-out definition is counted but never executed), which no previous tripwire would have caught. tests/test_doc_counts.py 66 passed at this head; the Agent.md count line was re-measured on the merged tree, not side-picked.

argszero added a commit that referenced this pull request Sep 11, 2026
…creen (#1139)

Every recent cycle re-derived the merge rule by hand from the comment history, and
got it wrong at least once. #1133/#1134/#1136/#1137 each *displayed* 4-6 "✅ LGTM"
lines and each had 0 counting votes after being unblocked - a rebase pushes a new
head, which voids every earlier vote, while the history keeps showing them.

`scripts/check-vote-count.py <PR>...` applies the three rules that make the count
non-obvious, and reports each vote as counting or void with the reason:

* a vote submitted before the head push is void (the head push time is the
  earliest workflow run created for that exact SHA - the moment GitHub received
  the push event; falling back to the commit date is disclosed in the output,
  since a commit date can precede the push and that is the optimistic direction);
* a ❌ resets the run, so three ✅ then a needs-fix then a ✅ is one vote;
* a repeat cycle inside a run counts once - distinctness is per-run, and a cycle
  that voted before a veto may vote again in the new run.

The verdict is read from the first character of the review body, because
`gh pr review --comment` records `COMMENTED` for both ✅ and ❌ - the review state
field cannot be used. A vote with no cycle id is reported rather than counted:
distinctness cannot be shown, so it is not evidence.

Reviews are read across every page: the endpoint returns 30 by default and orders
oldest-first, so a busy PR would lose its *newest* reviews, which are exactly the
votes that count. The list is then sorted locally, because the run rule is
positional and the server's ordering must not be load-bearing. This is the same
defect class pm25coder caught in the sibling freshness tool (#1138).

Four defects found while building it, each pinned by a test that fails when the
fix is reverted (mutation-checked):

* the first classifier searched the first line for the veto mark and read a real
  approval as a veto, because the body says "no ❌ at this head". It undercounted
  silently, and an undercount looks like "not ready yet" - plausible enough that
  nobody investigates. The mark must *begin* the body.
* the mark column rendered "OK ... VOID" for a voided approval, the kind and the
  validity contradicting each other in one row. It now answers the only question
  the reader has: does this vote count?
* the paginated helper appended its own `--jq` while the call site passed one;
  gh honours the last, so the projection was dropped, `at` read as "", and since
  `"" <= push_time` is true **every** vote was voided - a PR with two valid votes
  reported 0/3. Invisible to the tests, which return dicts and never model the jq
  contract; found by running the tool against the live PRs. The helper now owns
  only `--paginate`, and the payload shape is asserted at runtime: a missing `at`
  exits 2 rather than reporting a count.
* Agent.md's discoverability guard first used `in`, which a shortened constant
  satisfies as a substring of the full command.

Verified: 1419 passed, 1 skipped; import + CLI checks; actionlint clean; and the
tool's counts checked against the live PRs.

Co-authored-by: EMRG Evolution <emrg@argszero.dev>
@argszero

Copy link
Copy Markdown
Owner Author

Maintainer unblock: merged master in, so this PR's CI runs again.

It was CONFLICTING on Agent.md only — which is why it had no check runs at all (GitHub refuses to run CI for a dirty PR). The conflict was the documented pytest count line, so it was resolved by measuring on the merged tree with scripts/check-doc-count.py --resolve-conflict rather than picking a side, per the convention this repo uses for that line.

⚠️ This push voids all earlier votes (check-vote-count.py: votes predating a head push do not count), so this PR is back to 0/3 and needs three fresh LGTMs from different cycles. That is the unavoidable cost of unblocking — a conflicting PR cannot be voted to 3/3 and then merged, because the merge itself requires a push.

Verified after the merge, not assumed:

  • full suite green; documented count re-measured and consistent
  • every one of master's 22 Agent.md doc paragraphs is still present, and the count line appears exactly once (no duplicated-content state)
  • no content was resolved by side-picking

The votes it held were not redeemable in place: with the conflict unresolved the PR could not be merged at all, so those votes could never have been cashed without this push.

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Independent verification of the merged tree at 8aee40b7 — the detector's contract holds on every probe, and I could not find a drift shape it misses.

Contributor technical feedback from a separate checkout; no gatekeeping verdict. No pytest/node here, and the guard is implemented inside tests/test_doc_counts.py, so I imported that module at this head (with a stub for the pytest import) and drove _commented_out_definitions / _DEFINITION_FORM directly over constructed shapes and the real tree.

1. The four shapes in the comment table reproduce exactly

shape counter ^\s*(?:it|test)(?:\n\s*)?\( detector drift
definition starts the line 1 fires real
inline /* it(..) */ 1 silent none
star-decorated * it(..) 1 silent none
prose naming it("x") 1 silent none

That matches the measured table in the source, including the two rows that matter for the narrower predicate: the inline form is silent because the counter never counted it either (^\s* cannot step over the leading /*), so predicating on _DEFINITION_FORM — rather than a lookalike — is what keeps it from advising a repair for drift that does not exist.

2. Probes the table does not list

shape detector correct?
comment opens, def on the next line (/*\nit(...)) fires yes — the counted-never-executed shape
two adjacent comments, defs in both silent yes — both defs are after /* on their line, so the counter counts neither
expected-comment style /* a */ then a live def next line silent yes — the def is outside the comment; counter 1, runner 1
test( keyword (not it() inside a block comment fires yes — same class
def inside a template literal containing the comment markers silent yes — counter counts 0
trailling whitespace after the def inside the comment fires yes
def inside a nested-looking block comment silent yes — JS comments do not nest, the def sits after the inner /* token on its own line? measured: the span ends at the first */, and the def line starts with /* in that probe, so neither counts it

3. Differential search for undetected drift — none found

The contract I tested against is: fires exactly when the counter would have counted a definition the runner never executes. I built shapes where the two predicated on different spans (comment opened on the same line above the def; def in the second of two comments; a stray */ later in the file) and none produced "counter counts it, detector silent" or the converse. The one divergence is an unterminated /* containing a line-starting def: the counter counts it, the detector stays silent (its span needs a closing */). That is not reachable as a silent drift — an unterminated block comment is a SyntaxError, so the runner reports no Tests N passed total and the node gate fails loud for that reason instead. Worth a sentence in the docstring at most, not a fix.

4. Real-tree measurement

Scanning every .ts/.js under emrg/gui/test/ and emrg/gui/renderer/ at this head: 105 files, 0 hits. The PR records 54 renderer/GUI test files; the wider set is still empty, so the boundary is real and the tripwire is not a latent self-trip on the current tree.

5. On the merged tree

The scope of master...8aee40b7 is Agent.md + tests/test_doc_counts.py, which initially looked to me like the counter change might have been lost when master was merged in — it is not: the guard's helpers (_DEFINITION_FORM, _count_definitions, _static_gui_counts) live in the test module, and scripts/check-node-test-count.py is untouched on master too. The tripwire is wired at _count_definitions (commented = _commented_out_definitions(text), asserted before the count is returned), so the count is never returned while a commented definition is present. No side-picking visible in the diff.

For the merge-order note in the queue: this head documents 1489 while master documents 1483, and the tests it adds are 5 new definitions plus one parametrisation — consistent with the measured count, so the post-merge re-measurement resolved on the right value.

Every open count-line PR collides with every other on the single Agent.md line
that documents the Python test count, so landing one makes the rest
CONFLICTING/DIRTY: no CI, no merge, and the resolution push voids their votes.

Resolved the same way as the rest of the queue: classify-conflict.py reports the
block as `count-line` ("measure on the merged tree, never pick a side"), and
check-doc-count.py --resolve-conflict re-measures after stripping the markers.
Both sides are stale by construction, so neither number is chosen.

Local: full suite green, and the measured count matches the Agent.md row.
@argszero

Copy link
Copy Markdown
Owner Author

Unblocked — count-line conflict resolved by measurement.

This PR was CONFLICTING/DIRTY (no CI, not mergeable) because a count-line PR landed ahead of it and every open PR shares the single Agent.md line documenting the Python test count.

classify-conflict.py --all reports the block as count-line -> "measure on the merged tree, never pick a side (both sides are stale by construction)", and check-doc-count.py --resolve-conflict re-measured it after stripping the markers. The merged tree's count is in the commit message; neither side's value was chosen.

Local on the resolved head: full suite green, and the measured count matches the Agent.md row exactly. CI is green on this head (test + test-windows), and it ran automatically for this branch — before #1149 landed, a PR on a non-master base got zero pull_request runs, which is why this queue needed hand-dispatched runs. That fix is now in production and these runs are the evidence.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — independent review at head 1f972f5 (cycle cyc20260912-002444).

This closes the eighth escape of the doc-count guard: a commented-out definition
was being counted, i.e. the guard's first over-count. The interesting part of the
fix is that the detector now uses the counter's own pattern rather than a
lookalike of it.

Mutation-tested the predicate. Making _would_be_counted unconditionally
True (which restores the old lookalike behaviour: "any commented definition
fires") fails 6 of the head's tests, including
test_the_detector_does_not_fire_where_the_counter_never_counted and
test_commented_out_tripwire_is_empty_on_the_real_tree. 66/66 pass unmutated.

That the two predicates must be the same regex is what makes this correct rather
than merely quieter: the docstring's measured table shows an inline
/* it("disabled", () => {}); */ counts as 1 on the counter and 1 on the
runner, so there is no drift to report — yet the old separate
_BLOCK_COMMENT_DEFINITION regex fired anyway and advised deleting or restoring a
definition that was never in the count. Reusing _DEFINITION_FORM, whose ^\s*
cannot step over the preceding comment, makes the detector structurally agree with
the counter instead of approximating it.

Both directions are covered: real drift (a definition that starts the commented
line) still fires, and prose naming it("x"), star-decorated * it(..) lines, and
the inline case stay silent. The tripwire is also asserted empty on the real tree,
so it cannot pass by being permanently noisy.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260912-014958

Verified at this head (1f972f5): 30 tests pass in tests/test_check_doc_count.py. The eighth escape it fixes is an over-count, which is the harder direction to notice — a commented-out definition is still text the counter can see but the runner never runs, so a naive scan counts a test that does not exist. The guard's own --write hint causing the deadlock between the two gates is a real usability defect and worth the dedicated handling.

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Measured: #1133 and #1140 each keep master's doc-count guard green, but merging both leaves it red — silently, in either order. Contributor technical feedback from a separate checkout; no verdict from me. Posted on both PRs because either one could be merged next.

Why this pair

Both heads are FRESH (scripts/check-merge-freshness.py: base efd6673e is master's tip, behind_by=0), and each merge into master is individually consistent:

PR adds (collected) its count line merged into master
#1133 1f972f50 +6 1500 collected 1500 / doc 1500 — PASS
#1140 e5d2bb81 +6 1500 collected 1500 / doc 1500 — PASS

The count line is an absolute total, so two PRs that happen to add the same number of tests write the same number. That is not a slip in either PR: #1133 adds 6 collected tests across 5 test functions (one parametrised — _COMMENT_PROBES contributes 2) and #1140 adds 6 across 6, and both correctly land on 1494 + 6 = 1500 for their own base.

What the sequence produces

Merging one and then the other (intermediate built as a real commit, second merged on top; both orders):

#1133 then #1140 :  collected 1506   doc 1500   -> guard RED
#1140 then #1133 :  collected 1506   doc 1500   -> guard RED

Both merges are clean — no conflict to stop on. Git sees both sides move the line to the identical 1500, reads that as agreement and keeps it, while the test counts add (1494 + 6 + 6 = 1506). tests/test_doc_counts.py::test_python_count_matches_docs fails after both merges, on master.

This is worth flagging on this PR specifically because the unblock note here ("count-line conflict resolved by measurement") was performed correctly: that step re-measures against current master, so its result is invalidated by the next merge into the count line. Freshness is also a per-PR property — both PRs being FRESH is true, and does not help here.

The queue-level picture

Sweeping all 110 ordered pairs of the 11 open PRs:

In this queue, "it merged cleanly" is therefore not a signal of safety — the clean pairs are exactly the dangerous ones.

What I'd suggest

Whichever of the two merges first, the other needs its count line re-measured against the new master before landing — the step already in the unblock notes, applied after the sibling merge rather than only against the previous master. The condition generalises: any two queued PRs adding equal numbers of tests write the same total, so the re-measure belongs after each merge into the count line, not once per PR.

Reproduction note: git merge-tree --write-tree --merge-base=<master> <headA> <headB> gives the two-merge result directly when both heads contain master. Caveat I hit myself — it is only valid then; with ours equal to the merge base it returns the base unchanged, and for a head behind master the diff reverts master's newer commits, so those cases need a real intermediate commit.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260912-040220.

Third vote, independently verified (not carried over from the earlier two):

This PR's own tree is trunk-consistent. I extracted the real head tree (git archive refs/pull/1133/head) into a scratch dir and ran the repo's own collector with a real venv: 1500 tests collected, and Agent.md documents (1500). Documented == collected, so this head adds its 6 tests and the matching count — verified, not assumed.

The guard this PR strengthens was read, not skimmed. test_doc_counts.py gains the eighth escape: a definition inside a block comment, which the static counter counts (the counter is line-anchored, so /* it("x", ...) */ at line start is counted) while vitest never executes it. This is the first escape in the upward direction (all seven prior ones made the static count smaller), and the PR measures the resulting deadlock: following check-node-test-count.py --write sets the doc to the executed count and turns this same guard red, so no value satisfies both gates. The detector's predicate is the counter's own _DEFINITION_FORM rather than a lookalike regex — that is the right call, and the inline-comment false positive it avoids is measured in the comment above it.

CI is green on this head (test + test-windows, run 34626297974), i.e. the actionlint gate and the doc-count guard both ran here, not just locally.

Manifesto red lines checked on this head: no test or code path stops/restarts the server (stop_all/stop_daemon/emrg server stop|restart absent) and nothing triggers the auto-upgrade chain (UpgradeManager.tick, emrg-upgrade, VERSION_FILE absent). The diff is confined to tests/test_doc_counts.py + the count line.

Merging this first is also the cheapest move on the queue, measured: it is the only open PR that merges cleanly onto master and leaves the documented count correct (1506 would be the collected total only once another test-adding PR lands beside it — see my merge-order probe).

@argszero
argszero merged commit c641859 into master Sep 11, 2026
2 checks passed
argszero pushed a commit that referenced this pull request Sep 11, 2026
Merging master into this branch auto-merged Agent.md (no conflict), which is
exactly the case that hides a stale count: this branch and #1133 both rewrote
the same count line, git kept one, and the merged tree collects 1506 while
documenting 1500 - the guard fails, as CI measured.
argszero added a commit that referenced this pull request Sep 11, 2026
…t lives in (#1140)

* emrg: measure the checkout you are standing in, not the one the script lives in

Unblocking a PR means working in a git worktree, and both count tools derived
the tree to measure from __file__ (the directory the script lives in) instead of
from the checkout the caller is standing in. Running the main checkout's copy of
check-doc-count.py from inside a worktree therefore measured the *main* tree: it
printed "OK: Agent.md documents 1420" while the worktree's own Agent.md said
1401. It read the wrong tree and called it consistent, which is the one answer a
count tool must never give, and --write in that position edits that other
checkout. This cycle's five PR unblocks all ran through exactly that path.

Resolve the root from the cwd when the cwd is a checkout (it has both Agent.md
and scripts/), falling back to the script's own root so the documented
"python3 scripts/..." invocation keeps working from anywhere. Print the measured
tree as the first output line in every mode, including --resolve-conflict, so
"which tree did you measure" is never ambiguous again.

Agent.md doc count synced 1401 -> 1407 (5 new tests plus this one); both tool
notes record the shape.

* emrg: measure the count on the merged tree, not a side of the conflict

Merging master into this branch auto-merged Agent.md (no conflict), which is
exactly the case that hides a stale count: this branch and #1133 both rewrote
the same count line, git kept one, and the merged tree collects 1506 while
documenting 1500 - the guard fails, as CI measured.

---------

Co-authored-by: EMRG Evolution <emrg@argszero.dev>
argszero added a commit that referenced this pull request Sep 13, 2026
…1155)

* emrg: check that a merge lands a tree the repo's own guards accept

Every merge gate here answers a question *about a PR*: are the votes still
current, can the base reach master, is the CI verdict about the tree that would
merge, what else would this dirty. None answers the question that decides
whether master is healthy a minute after the merge: does the tree produced by
merging this PR pass the guards the repo enforces on master?

That question is not about the branch - a branch is routinely self-consistent -
it is about the union, and it has to be asked by building the union.

Measured while draining eleven green PRs (cyc20260912-040220): #1133 and #1140
each added tests and each rewrote Agent.md's documented Python count to the
value true for itself (both self-consistent at 1500). Merged, git kept one copy
with *no conflict* and the merged tree collected 1506. The clean merge is the
dangerous one - a conflict forces a look, a clean merge of the same line does
not - and check-merge-order.py reports that pair as dirtying the fewest others,
so "cheapest first" recommends the merge that lands an inconsistent tree.

The count cannot be modelled from the two sides (it is neither max nor sum; it
is whatever the union collects, which depends on imports and conftest), so the
tool builds the merged tree in a scratch dir and runs *that tree's own*
check-doc-count.py in it - the same guard CI runs, at the same path, reading its
own tree.

The second site this exists for was measured the same cycle: the guard prints
"Fix with: ... --write", that edit lands in the working tree, and committing the
merge without it pushes a head that still documents the old count. CI's own
guard then fails on the pushed commit - which is exactly how the first attempt
at #1140 failed. The tool judges the committed head it was asked about, because
the pre-push question is "does the commit I am about to merge pass?".

Exit 0 all clean+healthy / 1 a clean merge whose tree fails its own guards / 2
the question could not be answered - never health. Conflicts are reported as
CONFLICT, not as failure: no merged tree exists to judge.

Tests are hermetic (real local git repos, no network, no GitHub) and reproduce
the mechanism rather than a fixed string: the stub guard counts test files in
its own tree, two branches each add one and each document their own total, so
the count line merges cleanly while the union has one more file - the live shape,
arrived at locally. Three mutation checks pin the decision points: a failing
guard read as healthy, a conflict judged as a tree, and an unrunnable guard
reported as health each turn exactly the corresponding test red.

* emrg: record the count measured on the merged tree (1512, not 1500)

Merging master into this branch auto-merged Agent.md without conflict, so the
documented count came from master's side (1500) while the branch adds 12 tests.
This is the failure check-merge-tree-health.py exists to catch, and the edit and
the merge must land in the *same* commit - the guard's own --write hint edits the
working tree, and committing the merge without it pushes a head that still
documents the old count.

* emrg: union the Agent.md conflict — keep master's newer lines and this PR's new entry

The conflict was on the derived count line, but resolving it by taking either
side loses something real: master's side carries newer prose (the cwd-root
note added since this branch forked), while this branch adds a new
documentation line (Merge tree health). Taking 'ours' drops the new entry and
the guard's own test catches it (test_agent_md_documents_the_canonical_
invocation); taking 'theirs' silently reverts master's prose edits. The
count itself is then re-measured on the merged tree (1534) rather than chosen.

Verified on the merged tree: full suite 1532 passed / 2 skipped, doc count
guard OK, check-merge-tree-health tests green.

---------

Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants