Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/engineering/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@

For E11 evaluation, read the [preregistered pilot tasks](pilot-selection.md) and [evaluation rubric](evaluation-rubric.md). These are planning and measurement documents, not permission to execute pilots or start held tasks.

The first pilot has run: the [B01 report](pilot-b01-2026-09-20.md) records what the preregistered acceptance checks caught that the implementer did not, including a blocking defect found by the mandated independent review in a change whose CI was green. It was executed by a host agent against those checks rather than by a CodeCartographer engine, which does not exist yet — it measures whether the checks are usable and load-bearing, not whether an automated loop can run them.

The [Helix/Traverse discussion](../plans/2026-09-17-helix-patterns.md) is retained as design history. Its P1–P6 proposals are not all current implementation instructions; its precedence note identifies the adjustments made after dogfooding.

## A new agent session's handoff
Expand Down
56 changes: 56 additions & 0 deletions docs/engineering/pilot-b01-2026-09-20.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# B01 pilot report — the first real change through the engineering contract

**Status:** complete. Bug fixed and merged upstream as [`c4030a4`](https://github.com/HuginnIndustries/CodeCartographer/commit/c4030a4) ([#421](https://github.com/HuginnIndustries/CodeCartographer/pull/421), closing [#412](https://github.com/HuginnIndustries/CodeCartographer/issues/412)).

**What this is, and is not.** This ran B01 as an *acceptance-check protocol test*, executed by a host agent under the checks preregistered in [pilot-selection.md](pilot-selection.md). It was **not** run through a CodeCartographer engine — that engine does not exist yet; E02 storage and E05/E06 evidence collection are unbuilt. So this reports whether the contract's checks are *usable and load-bearing*, not whether an automated loop can execute them. The distinction matters: no record was minted, no digest was bound, no acceptance was classified. What was exercised is the human/agent workflow the contract specifies.

The bug also had a public diagnosis and a suggested fix in its issue. That makes it a known-answer protocol test, not evidence of unaided diagnosis. Per the rubric, that exposure is reported rather than described as discovery.

## Outcome against the preregistered checks

| ID | Required observation | Result |
|---|---|---|
| A1 | Injected baseline reproduces the named failure before the implementation exists | `not ok 37 - resolvePublishSourceRepo records origin's fetch URL verbatim`, 71/72 |
| A2 | Regression suite passes while the parent runner still injects | 214/214 across all seven guard sites, both vectors |
| A3 | Boundary exercised in a fresh child process; all sites covered | 8 tests, each spawning a child with the vector in its environment |
| A4 | Control, build, full suite pass; counts recorded | 1007/1007 clean and under injection; build exit 0 |
| A5 | Production URL/provenance behavior and user git config unchanged | diff is tests-only; helper assigns only to its own `process.env` |
| A6 | Fresh review for masked failure, deleted assertions, harness workaround | REQUEST_CHANGES → blocking gap found and closed; see below |

All six satisfied. Three mutation checks bite.

## What the checks caught that the implementer did not

This is the part worth keeping. Each item below was found *because* a check demanded it, not because anyone thought to look.

**A6 caught an incomplete fix.** The issue named one missing config source (`GIT_CONFIG_COUNT`). Git has two the guard missed: the second, `GIT_CONFIG_PARAMETERS`, is how git hands `-c key=value` to its own subprocesses. It carries its own entries, so `GIT_CONFIG_COUNT=0` does not disarm it, and it arrives without anyone setting it deliberately — `git bisect run npm test` is enough. With the implementer's first fix in place, the original failure reproduced byte-for-byte through that vector. **A mandated independent review found a blocking defect in a change whose author believed it complete and whose CI was green.**

**A3's fresh-child-process requirement was load-bearing.** Asserting on `process.env` in the test process, or letting an earlier test's cleanup set the stage, would have produced a passing suite that proved nothing. The requirement forced probes that spawn a child with the vector injected into *that child's* environment — which is the only shape that tests the boundary.

**A1's before-the-fix ordering prevented a vacuous suite.** The implementer added a negative control asserting the rewrite genuinely reaches git when only file lookups are redirected. Without it, every other assertion could pass because the rewrite never worked. That control later proved its worth: it was one of the tests that failed under the `GIT_CONFIG_PARAMETERS` vector, demonstrating the target test was never vacuous.

**A6 also caught a test that resisted its own fix.** The A5 test asserted the helper sets *exactly three* environment variables. When a fourth source had to be neutralized, that test failed — a test whose shape penalized widening the isolation it existed to protect. Rewritten as an allow-list.

**Cross-platform CI caught a defect in the new test code itself.** The child-process probes interpolated an absolute path directly into an ESM `import`. On Linux that resolves; on Windows `D:\...` parses as URL scheme `d:` and every probe failed. The repository uses `pathToFileURL` everywhere else for exactly this reason.

## Corrections the pilot forced to its own claims

Two statements made during implementation were wrong and were corrected rather than left standing:

- The implementer claimed the guard's position in `broadside-repo-collection.test.mjs` was a *realized* defect. The reviewer checked that file out and ran it under injection: 10/10 passed, because `core/broadside.ts` makes no git call during module evaluation. It was a latent hazard. The reorder is still correct; the claim was not.
- A code comment asserted git rejects an empty `GIT_CONFIG_PARAMETERS` as malformed. Direct testing showed git reads it as zero entries. Corrected.

Both matter for the rubric's purposes: a contract that produces confident-sounding but unverified claims is worse than one that produces none.

## What this says about the contract

**Supported.** The acceptance-check format is usable by a host agent, and the checks are not ceremonial — A6 alone converted a green-CI, author-confident change into a blocked one with a reproducible counterexample. Preregistering the checks before implementation prevented the common failure of writing acceptance criteria that the finished work happens to satisfy.

**Not supported, and not claimed.** Nothing here exercised record storage, snapshot binding, evidence collection, approval classification, or interruption/resume. `VERIFIED_ACCEPTANCE_INTEGRATIONS` remains empty and no acceptance was classified; per [E12](https://github.com/HuginnIndustries/CodeCartographer/issues/418), storage-protection continuity is not host-attestable, so any acceptance the engine did mint today would be `cooperative`.

**A gap this surfaced.** The contract has no check requiring that a fix address the *class* of defect rather than the reported instance. Both review rounds on E03, and this pilot, hit the same pattern: a reviewer finds one coercion, one missing config source, one unvalidated field, and the fix closes that one. A future check should ask what else of the same kind exists.

## Follow-ups recorded, not silently absorbed

- `GIT_TEMPLATE_DIR` can still inject hooks into fixtures under full config isolation. Not configuration; out of scope for #412 and left open rather than folded in.
- F01 (the bounded feature) has not been run. Per the trial order it should start from this accepted state as a second change in the same project, which is also the first real test of continuity across changes.
2 changes: 2 additions & 0 deletions docs/engineering/pilot-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ Use a known pinned starting point even if #412 is fixed before Traverse exists.

**Existing issue:** [#412](https://github.com/HuginnIndustries/CodeCartographer/issues/412). This is a real test-harness isolation defect, not a defect in production remote-URL resolution.

> **Run and reported.** B01 was executed on 2026-09-20 and its fix merged as [`c4030a4`](https://github.com/HuginnIndustries/CodeCartographer/commit/c4030a4); all six acceptance checks are satisfied. See the [B01 pilot report](pilot-b01-2026-09-20.md). It was run by a host agent against these checks, **not** through a CodeCartographer engine — E02 storage and E05/E06 evidence collection do not exist yet — so it measures whether the checks are usable and load-bearing, not whether an automated loop can execute them. The acceptance checks below are preserved as written; the fix having landed upstream does not invalidate a later replay from the pinned revision, which must be labeled a replay.

**Requested outcome:** Git-backed tests that already isolate global/system configuration must also prevent injected `GIT_CONFIG_COUNT` entries from rewriting fixture remotes. A contributor's command-scoped Git configuration must not make these fixtures fail or alter the intended fixture provenance.

### Selection evidence, observed at the pinned revision
Expand Down
Loading