Decide: retire the launch-benchmark apparatus, or keep which tiers of it? #296

Question

Is the launch-benchmark apparatus worth its code? Raised by the owner (2026-08-20): "i was considering deprecating the benchmarks as its a lot of code."

It is a lot of code — ~4,150 lines — but it is three separable things with three completely different track records, and the answer is different for each. Deciding this as one lump is the trap.

The recommendation up front: retire Tier A, keep B, keep C. That deletes ~1,860 lines — about 80% of the bench-specific code — and it deletes the tier with the worst record while keeping both tiers that have actually paid for themselves. The rest of this ticket is the evidence, because one part of it is genuinely not ours to decide alone.

Tier A has never once done its job

Charter decision 1 promised a trend that alerts on a real regression without gating. Six days of history, measured from the Actions API and the commit comments:

It is broken more than it works. 64 runs, 35 green and 29 failed. Green only in two bursts — roughly 15 hours on 08-14/15, then about 90 minutes on 08-20. Everything between 08-15T13:20 and 08-20T17:27 failed (~20 consecutive runs, five days), and everything since 08-20T19:19 has failed again — that is the devpod-on-PATH break in #292, now six runs deep and still firing.

Every alert it has ever raised was noise. 13 alerts fired. One — the first, f5f13332 — is the deliberate 1%-threshold dispatch recorded on #198, which named all nine series at once to prove the path worked. Of the remaining 12:

• host-prep in 10 of 12.
• devpod-up in 3 (devpod's own variance — explicitly out of scope for this map: "we time it; we don't fix it").
• total in zero. tools in zero. attach in zero.

So the load-bearing number the whole design protects has never alerted, and the dominant cold cost (tools, ~4.6–6s) has never alerted. Four of those alerts were warm host-prep, and here are the actual values that paged you:

0.000517 s  ->  0.000798 s        1.54x
0.000445 s  ->  0.000798 s        1.79x
0.000506 s  ->  0.000840 s        1.66x
0.000612 s  ->  0.000937 s        1.53x

Three tenths of a millisecond, four times. And this is not bad luck — it is a mis-scope provable from the repo's own tests. The charter's own words, excluding finer spans from the trend: "a 0.03s gh auth token line swinging 50% must not page anyone." But clients/gh.rs asserts, in a test: assert_eq!(span_labels(&document, "host-prep"), ["gh auth token"]). On the warm path host-prep is that line — the exact example the charter used for what must never alert was itself promoted to a first-class alerting stage. A relative threshold on a sub-millisecond stage cannot mean anything, ever.

Net: 13 alerts against 35 published points — 37% of all published points paged the owner, and 0% of them were a regression. An alert channel with a 100% false-positive rate is worse than none: it trains you to ignore the one that matters.

And the numbers it published were the wrong binary's. All 35 points measure the Python build. Until 1654daf these steps ran pixi run dl, which resolved to the editable install's console script — so after the 0.1.0 cutover the trend was charting a build that no longer shipped, and nothing said so. The commit that fixed that is the commit that broke the job. Not one point in the series describes the binary users run.

What is not an argument against it: CI cost. Median green run 2m30s, 100 minutes total across all 64 runs. It is cheap. The cost is code and attention, not compute. (Caveat: the repaired version adds cargo build --release, so a fixed Tier A would be slower than any run measured here.)

Tier B is where all the value actually came from

Every measurement this map and #139 trade on came from the local harness, not the trend:

• Collapse the three sequential devpod ssh round trips on the cold path #157/One cold-path setup pass: fold the hostname into the probe's round trip, with per-stage outcomes #168's fold: warm attach 4.65s → 2.48s, cold 8.07s → 5.99s.
• Cache the staged tools tar instead of re-staging ~342MB every transfer #158's negative result — measuring the tar staging (0.13s warm / 2.6s cold, not the assumed ~5.1s) dissolved the case for a cache and stopped a bad build. A harness that talks you out of work is paying rent.
• Warm path: defer clone-manager construction past the fast-attach check #145's ~0.29ms finding, and with it the discovery that the spawn-count guard had been passing with the regression reintroduced.

423 lines, no CI job, no gh-pages branch, no alert threshold, runs when asked. It is also exactly what #139's open re-baseline question needs — the Rust binary's numbers are still unknown, and pixi run bench --record is the method already built. Keep it.

Tier C has two obligations that outlive the trend

timing.rs is not bench code, and this is the part to be careful with:

1. It is a live cross-repo contract. wayfinder PR #162 merged 2026-08-14 and wf now stamps DEVLAUNCH_HANDOFF_T0 and DEVLAUNCH_PREWARM_FIRED_AT on every launch. dl's timing is the only thing that reads them. Retire it and wf is writing to nobody.
2. The stage names are frozen public API. devlaunch-core module architecture and data model #251 §7 pinned them into devlaunch-core's surface, and public-api.txt is enforced by the public-api CI job. Removing them is a deliberate API break, not a cleanup.

There is still a fair shrink question here — whether the json document mode earns its share of the 905 non-test lines once nothing consumes it on a schedule, or whether prose plus a leaner document would do. That is a separate decision and should not ride on this one.

The one thing that is not ours to decide alone

blooop/wayfinder#159 "Launch benchmarking: wf's side of the seam" is open, and its destination reads "stamped across the seam so the devlaunch trend can carry the true end-to-end total." That map's two tickets are closed and its stamps have shipped; retiring Tier A pulls the consumer out from under an open map in another repo. Charter decision 2 made these two maps deliberately separate, so this needs a decision there, not a unilateral deletion here. Nothing about Tier A's record changes on that repo's side — but the re-chart is theirs to do.

The one thing Tier A caught, and whether it is covered

Honesty requires this: the bench did catch a real user-facing bug — #240, the shared-pixi-cache uid mismatch that hard-fails a launch in any image lacking the directory. But it caught it by failing to launch at all, not by charting anything: the trend line and the alert had no part in it. The gating e2e job already runs real devpod launches on every PR and merge — except that it uses a fixture image, while the bench launches a real GitHub repo (blooop/mcp-devtasks) whose devcontainer has a different remoteUser uid and runs pixi install in postCreate. That difference is precisely what exposed #240. So if Tier A goes, the question to settle in the same breath is whether one e2e case should inherit "launch one real repo with a foreign uid end-to-end" — which is cheap, and gates, and is the part that ever worked.

What settling this decides downstream

• #292 (restore the trend) and #293 (build Can devpod up's output be decomposed into image / create / postCreate? #195's devpod-up sub-lines) are both pure Tier A and both die with it. This ticket blocks them: repairing and then extending an apparatus that may be deleted is the one clearly wrong order.
• Launch latency: dl <spec> -- <cmd> to a running command #139's re-baseline fog item needs Tier B only and is unaffected either way.
• If Tier A is retired, this map's Destination needs rewriting — it is stated entirely in terms of the trend — and the map then closes on B and C, with Can devpod up's output be decomposed into image / create / postCreate? #195's sub-line decision recorded as decided, deliberately not built.

To grill

1. Tier A: retire, or repair once more with host-prep and devpod-up given no threshold (which would leave total and tools — neither of which has ever alerted — as the only alerting series)?
2. If retired: does an e2e case inherit the real-repo/foreign-uid launch coverage?
3. Tier C: hold as-is on the wf contract and the frozen API, or open the shrink question separately?
4. Who tells wayfinder#159?