From 6f6a099a6be172067dad331c8d69671bd4673340 Mon Sep 17 00:00:00 2001 From: James Sesler Date: Sun, 20 Sep 2026 17:45:10 -0400 Subject: [PATCH] docs: report the B01 pilot and what its checks caught B01 ran on 2026-09-20; the fix merged as c4030a4 (#421, closing #412). All six preregistered acceptance checks are satisfied. The report states plainly what this was: a host agent executing the contract's checks, NOT a CodeCartographer engine running a change. E02 storage and E05/E06 evidence collection do not exist, so no record was minted, no digest bound, no acceptance classified. It measures whether the checks are usable and load-bearing. The bug also had a public diagnosis in its issue, which makes it a known-answer protocol test rather than evidence of unaided diagnosis; the rubric requires reporting that exposure. What the checks caught that the implementer did not: - A6's mandated independent review found the fix incomplete. Git has four environment-reachable config sources; the issue named one gap and the fix closed it, leaving GIT_CONFIG_PARAMETERS live. The original failure reproduced byte-for-byte on a branch with green CI whose author believed it done. - A3's fresh-child-process requirement is why the tests could not be faked by an earlier test's environment cleanup. - A1's before-the-fix ordering produced a negative control that later proved the target test was never vacuous. - A6 also caught a test that resisted its own fix: it asserted the helper sets exactly three variables, so covering a fourth source broke it. Two claims made during implementation were wrong and are corrected in the report rather than left standing: the guard's old position was a latent hazard, not a realized defect (the reviewer ran that file under injection: 10/10 passed), and git reads an empty GIT_CONFIG_PARAMETERS as zero entries rather than erroring. The report also names a gap the pilot exposed, filed as #422: the checks ask whether the reported defect is fixed and proven, never what else of the same class exists. Three review rounds hit that pattern. #423 records the GIT_TEMPLATE_DIR follow-up left out of scope. --- docs/engineering/README.md | 2 + docs/engineering/pilot-b01-2026-09-20.md | 56 ++++++++++++++++++++++++ docs/engineering/pilot-selection.md | 2 + 3 files changed, 60 insertions(+) create mode 100644 docs/engineering/pilot-b01-2026-09-20.md diff --git a/docs/engineering/README.md b/docs/engineering/README.md index 33dc340..e8343e2 100644 --- a/docs/engineering/README.md +++ b/docs/engineering/README.md @@ -13,6 +13,8 @@ For E11 evaluation, read the [preregistered pilot tasks](pilot-selection.md) and [evaluation rubric](evaluation-rubric.md). These are planning and measurement documents, not permission to execute pilots or start held tasks. +The first pilot has run: the [B01 report](pilot-b01-2026-09-20.md) records what the preregistered acceptance checks caught that the implementer did not, including a blocking defect found by the mandated independent review in a change whose CI was green. It was executed by a host agent against those checks rather than by a CodeCartographer engine, which does not exist yet — it measures whether the checks are usable and load-bearing, not whether an automated loop can run them. + The [Helix/Traverse discussion](../plans/2026-09-17-helix-patterns.md) is retained as design history. Its P1–P6 proposals are not all current implementation instructions; its precedence note identifies the adjustments made after dogfooding. ## A new agent session's handoff diff --git a/docs/engineering/pilot-b01-2026-09-20.md b/docs/engineering/pilot-b01-2026-09-20.md new file mode 100644 index 0000000..3a5185e --- /dev/null +++ b/docs/engineering/pilot-b01-2026-09-20.md @@ -0,0 +1,56 @@ +# B01 pilot report — the first real change through the engineering contract + +**Status:** complete. Bug fixed and merged upstream as [`c4030a4`](https://github.com/HuginnIndustries/CodeCartographer/commit/c4030a4) ([#421](https://github.com/HuginnIndustries/CodeCartographer/pull/421), closing [#412](https://github.com/HuginnIndustries/CodeCartographer/issues/412)). + +**What this is, and is not.** This ran B01 as an *acceptance-check protocol test*, executed by a host agent under the checks preregistered in [pilot-selection.md](pilot-selection.md). It was **not** run through a CodeCartographer engine — that engine does not exist yet; E02 storage and E05/E06 evidence collection are unbuilt. So this reports whether the contract's checks are *usable and load-bearing*, not whether an automated loop can execute them. The distinction matters: no record was minted, no digest was bound, no acceptance was classified. What was exercised is the human/agent workflow the contract specifies. + +The bug also had a public diagnosis and a suggested fix in its issue. That makes it a known-answer protocol test, not evidence of unaided diagnosis. Per the rubric, that exposure is reported rather than described as discovery. + +## Outcome against the preregistered checks + +| ID | Required observation | Result | +|---|---|---| +| A1 | Injected baseline reproduces the named failure before the implementation exists | `not ok 37 - resolvePublishSourceRepo records origin's fetch URL verbatim`, 71/72 | +| A2 | Regression suite passes while the parent runner still injects | 214/214 across all seven guard sites, both vectors | +| A3 | Boundary exercised in a fresh child process; all sites covered | 8 tests, each spawning a child with the vector in its environment | +| A4 | Control, build, full suite pass; counts recorded | 1007/1007 clean and under injection; build exit 0 | +| A5 | Production URL/provenance behavior and user git config unchanged | diff is tests-only; helper assigns only to its own `process.env` | +| A6 | Fresh review for masked failure, deleted assertions, harness workaround | REQUEST_CHANGES → blocking gap found and closed; see below | + +All six satisfied. Three mutation checks bite. + +## What the checks caught that the implementer did not + +This is the part worth keeping. Each item below was found *because* a check demanded it, not because anyone thought to look. + +**A6 caught an incomplete fix.** The issue named one missing config source (`GIT_CONFIG_COUNT`). Git has two the guard missed: the second, `GIT_CONFIG_PARAMETERS`, is how git hands `-c key=value` to its own subprocesses. It carries its own entries, so `GIT_CONFIG_COUNT=0` does not disarm it, and it arrives without anyone setting it deliberately — `git bisect run npm test` is enough. With the implementer's first fix in place, the original failure reproduced byte-for-byte through that vector. **A mandated independent review found a blocking defect in a change whose author believed it complete and whose CI was green.** + +**A3's fresh-child-process requirement was load-bearing.** Asserting on `process.env` in the test process, or letting an earlier test's cleanup set the stage, would have produced a passing suite that proved nothing. The requirement forced probes that spawn a child with the vector injected into *that child's* environment — which is the only shape that tests the boundary. + +**A1's before-the-fix ordering prevented a vacuous suite.** The implementer added a negative control asserting the rewrite genuinely reaches git when only file lookups are redirected. Without it, every other assertion could pass because the rewrite never worked. That control later proved its worth: it was one of the tests that failed under the `GIT_CONFIG_PARAMETERS` vector, demonstrating the target test was never vacuous. + +**A6 also caught a test that resisted its own fix.** The A5 test asserted the helper sets *exactly three* environment variables. When a fourth source had to be neutralized, that test failed — a test whose shape penalized widening the isolation it existed to protect. Rewritten as an allow-list. + +**Cross-platform CI caught a defect in the new test code itself.** The child-process probes interpolated an absolute path directly into an ESM `import`. On Linux that resolves; on Windows `D:\...` parses as URL scheme `d:` and every probe failed. The repository uses `pathToFileURL` everywhere else for exactly this reason. + +## Corrections the pilot forced to its own claims + +Two statements made during implementation were wrong and were corrected rather than left standing: + +- The implementer claimed the guard's position in `broadside-repo-collection.test.mjs` was a *realized* defect. The reviewer checked that file out and ran it under injection: 10/10 passed, because `core/broadside.ts` makes no git call during module evaluation. It was a latent hazard. The reorder is still correct; the claim was not. +- A code comment asserted git rejects an empty `GIT_CONFIG_PARAMETERS` as malformed. Direct testing showed git reads it as zero entries. Corrected. + +Both matter for the rubric's purposes: a contract that produces confident-sounding but unverified claims is worse than one that produces none. + +## What this says about the contract + +**Supported.** The acceptance-check format is usable by a host agent, and the checks are not ceremonial — A6 alone converted a green-CI, author-confident change into a blocked one with a reproducible counterexample. Preregistering the checks before implementation prevented the common failure of writing acceptance criteria that the finished work happens to satisfy. + +**Not supported, and not claimed.** Nothing here exercised record storage, snapshot binding, evidence collection, approval classification, or interruption/resume. `VERIFIED_ACCEPTANCE_INTEGRATIONS` remains empty and no acceptance was classified; per [E12](https://github.com/HuginnIndustries/CodeCartographer/issues/418), storage-protection continuity is not host-attestable, so any acceptance the engine did mint today would be `cooperative`. + +**A gap this surfaced.** The contract has no check requiring that a fix address the *class* of defect rather than the reported instance. Both review rounds on E03, and this pilot, hit the same pattern: a reviewer finds one coercion, one missing config source, one unvalidated field, and the fix closes that one. A future check should ask what else of the same kind exists. + +## Follow-ups recorded, not silently absorbed + +- `GIT_TEMPLATE_DIR` can still inject hooks into fixtures under full config isolation. Not configuration; out of scope for #412 and left open rather than folded in. +- F01 (the bounded feature) has not been run. Per the trial order it should start from this accepted state as a second change in the same project, which is also the first real test of continuity across changes. diff --git a/docs/engineering/pilot-selection.md b/docs/engineering/pilot-selection.md index 89f06e5..2738bc5 100644 --- a/docs/engineering/pilot-selection.md +++ b/docs/engineering/pilot-selection.md @@ -25,6 +25,8 @@ Use a known pinned starting point even if #412 is fixed before Traverse exists. **Existing issue:** [#412](https://github.com/HuginnIndustries/CodeCartographer/issues/412). This is a real test-harness isolation defect, not a defect in production remote-URL resolution. +> **Run and reported.** B01 was executed on 2026-09-20 and its fix merged as [`c4030a4`](https://github.com/HuginnIndustries/CodeCartographer/commit/c4030a4); all six acceptance checks are satisfied. See the [B01 pilot report](pilot-b01-2026-09-20.md). It was run by a host agent against these checks, **not** through a CodeCartographer engine — E02 storage and E05/E06 evidence collection do not exist yet — so it measures whether the checks are usable and load-bearing, not whether an automated loop can execute them. The acceptance checks below are preserved as written; the fix having landed upstream does not invalidate a later replay from the pinned revision, which must be labeled a replay. + **Requested outcome:** Git-backed tests that already isolate global/system configuration must also prevent injected `GIT_CONFIG_COUNT` entries from rewriting fixture remotes. A contributor's command-scoped Git configuration must not make these fixtures fail or alter the intended fixture provenance. ### Selection evidence, observed at the pinned revision