Re-baseline launch latency on the Rust build

Question

Do this map's headline wins survive the Rust port — and what is the Rust build's
launch-latency decomposition?

This is the map's own standing fog, recorded as "the sharpest thing left on the
map", and it names its own trigger: it waits on
#292 "or sooner, if a Rust
launch ever feels slower than the numbers above promise". A human asked about
startup time on 2026-08-24 and a measurement pass followed, so the "sooner" arm
has fired. This ticket records that pass and says what it does and does not
settle.

What was measured (full detail and provenance in the comment trail; every figure
carries the nested-DinD caveat below):

Cold launch, warm image and warm bare cache: 13.1s (zellij off) / 15.5s
(zellij on). Decomposed: devpod status 0.43 · git fetch 1.2-1.4 · workspace
git clone 0.03 · devpod up 2.3-2.6 · probe trip 2.32 (4.27 with zellij) ·
tools tar 0.21 · transfer trip 4.29 · attach trip 2.07.
Warm launch: 2.7-3.5s — devpod status 0.6 plus one ssh trip 2.1-2.8.
An ssh trip's floor is 2.07-2.33s with an empty payload, of which raw ssh
through devpod's ProxyCommand is 1.79-1.85s and devpod's own binary init is
0.05s.

Against the pre-port numbers this map published: #168 claimed warm attach 2.48s
and ~2.1s per cold trip. The Rust build measures 2.07-2.33s per trip, so the
per-trip arithmetic #157 and #168 were built on has held, and the fold they
shipped has not regressed. That is the answer to "do the wins survive" as far as
these numbers can carry it.

What this ticket must do:

Record the decomposition somewhere durable and say plainly what it is not:
it is not a host measurement and not the CI trend point The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292 owns. Decide
whether any of it belongs in README (which currently quotes host medians of
2.176s warm / 15.899s cold-recreate, copied in by hand — the same
hand-copying --record exists to end).
Decide whether the standing fog entry is thereby cleared, or narrowed to
"still needs the host/CI number" and left open pointing at The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292.
Leave the per-stage numbers in a shape the sibling tickets can cite instead of
re-measuring, so four grillings do not each re-derive the same floor.

Not a code change unless the answer says README is wrong.

Where these numbers come from

All figures below were measured on 2026-08-24 against the Rust build
(rust/target/release/dl, dl 0.13.0), driven from inside this repo's own
devcontainer (nested docker-in-docker, devpod 0.26.1, docker 29.7.2, 8 CPUs).
DEVPOD_HOME, DEVPOD_SSH_CONFIG and XDG_CACHE_HOME were scoped to a scratch
directory; the target was blooop/dotfiles with a warm image and a warm bare
clone cache.

Caveat, and it is not small: nested DinD is not the host. Absolute seconds
here run higher than a host launch, and this is not the CI trend point
#292 is about. What transfers
is the ratios and the floors — the pieces are A/B'd against each other in one
environment. The host cross-check is that README's own host medians (warm
2.176s, cold-recreate 15.899s) sit close to what was measured here (warm
2.7-3.5s, cold 13.1-15.5s).

breadcrumb: the decomposition reproduces on a second session, so it is not one lucky sample. Same environment class as the ticket body (nested DinD in this repo's devcontainer, devpod 0.26.1, 8 CPUs), rust/target/release/dl dl 0.13.0, scoped DEVPOD_HOME / DEVPOD_SSH_CONFIG / XDG_CACHE_HOME, target blooop/dotfiles, warm image.

Warm, 3 consecutive runs — in-process totals 2.946 / 2.910 / 3.099s; devpod status 0.563–0.670, devpod ssh 2.315–2.426. (Ticket body: warm 2.7–3.5s, trip floor 2.07–2.33s. Holds, trip sitting at the top of that band.)

Cold-recreate, warm bare cache + warm image, 2 runs — 16.576 / 16.589s: devpod status 0.44–0.47 · git fetch 1.30 · workspace git clone 0.02–0.04 · devpod up 2.51–2.78 · probe trip 4.59–4.73 · tools tar 0.19–0.20 · transfer trip 4.67–4.96 · attach trip 2.17–2.63. The probe trip near 4.6s rather than 2.3s is the zellij stage, which matches the ticket's zellij-on arm (15.5s) within ~7%, not its zellij-off arm.

One number the ticket body does not have, and it is a trap worth naming: the very first launch into a fresh DEVPOD_HOME measured 22.401s — git clone --bare 1.746 and devpod context 0.418 appear, and the probe trip is 9.028s instead of ~4.6s, because devpod injects and starts its agent inside the container on that trip. So a scratch-DEVPOD_HOME run is not the cold-recreate shape the trend means; it is a shape of its own, ~5.8s more expensive, and anyone re-running these measurements in a fresh scratch home will otherwise read that agent injection as a regression.

breadcrumb: README is wrong in the attributable sense, so the edit is in scope. Traced with git log -S:

2.176s / 15.899s / 33.204s and the 20.834s cold decomposition entered README in d3b8ffb and 461f740, both 2026-08-14.
The Python tree was retired in 1654daf, 2026-08-20 (0.1.0 shipped 2026-08-19).
README says nothing about which build produced them — grep -n "Python build" README.md finds exactly one hit, and it is the "Going back to the Python build" section about 0.0.29, 200 lines later.
Worse than silence: the captures invoke dl-next, which today names the compiled Rust working-tree build and in August named the Python one. A reader takes those seconds for the shipped binary's.
Nothing could have caught it. test/test_bench_doc.py's own docstring: "It does not check the numbers the section quotes: those are a measurement of one host, not a command anything can run."

The formats, by contrast, are still correct — dl-timing: / dl-timing-json: and the stage set survived the port (rust/devlaunch-core/src/timing.rs), and timing.rs's unit tests assert the README example's own values (dl-timing: devpod status 0.454s). So this is a provenance fix, not a re-measurement: nothing in README gets a new number, because nothing has been measured on a host.

agent-decided (1): long-term maintainability. This ticket's whole product is a written record — the risk it exists to remove is a number that outlives the build it measured, and that risk has already fired once in this repo (README, below) and once in the trend (#292: 35 published points measure a build that no longer ships, and nothing said so). So every figure here is stamped with the build, the environment and the shape it was taken in, and the one document that quoted unstamped figures is fixed.

The recorded decomposition

Build: rust/target/release/dl, dl 0.13.0. Environment: nested inside this repo's own devcontainer — docker-in-docker, devpod 0.26.1, docker 29.7.2, 8 CPUs. Method: DEVLAUNCH_TIMING=1, DEVPOD_HOME / DEVPOD_SSH_CONFIG / XDG_CACHE_HOME scoped to a scratch dir, target blooop/dotfiles, warm docker image. Ticket-body pass plus an independent reproduction pass (breadcrumb); ranges below span both.

shape	total	decomposition
warm	2.7–3.5s	`devpod status` 0.43–0.67 + one ssh trip 2.1–2.8
cold-recreate, warm image + warm bare cache, zellij off	13.1s	status 0.43 · `git fetch` 1.2–1.4 · workspace `git clone` 0.02–0.05 · `devpod up` 2.3–2.8 · probe trip 2.32 · `tools tar` 0.19–0.24 · transfer trip 4.29 · attach trip 2.07
cold-recreate, same, zellij on	15.5–16.6s	as above, probe trip 4.27–4.73 (the zellij stage is ~1.95s of it)
first launch into a fresh `DEVPOD_HOME`	22.4s	adds `git clone --bare` 1.75 and `devpod context` 0.42, and the probe trip becomes 9.03 — devpod injects and starts its agent on that trip

Floors, which are the durable part: an empty-payload devpod ssh --command trip is 2.07–2.33s, of which raw ssh through devpod's ProxyCommand is 1.79–1.85s and devpod's own binary init 0.05s — i.e. still ~99% connection setup, as #157 found on the Python build. The 388MB payload spends 1.76s on the wire (0.43s via docker exec -i); a second trip over ControlMaster/ControlPersist costs 0.010s; docker exec into the same container costs 0.07s against 2.3s.

Read as ratios and floors, never as wall clock. Nested DinD is not a host, and it is not the CI runner. Absolute seconds here run high; what is A/B'd inside one environment is what transfers. Sibling tickets should cite the trip floor and the per-stage split from here rather than re-measuring, and should not cite the totals as anybody's launch time.

(a) The fog entry narrows; it does not clear

"Do the wins survive the port?" asked two separable things, and only one of them is now answered.

Answered: the arithmetic the folds were argued from held across the port. #157 measured ~1.73s per ssh trip and ~99% connection setup; the Rust build measures 2.07–2.33s per trip with the same 99% shape. Every decision on this map that reasoned "a trip costs about a trip, so removing one saves about a trip" — #157's refusal of probe→transfer, #168's fold, #143/#200 — still reasons correctly on the shipped binary. The ratio between a trip and the work inside it did not move.

Not answered: #168's two headline numbers — warm attach 4.65s → 2.48s, cold 8.07s → 5.99s — are host wall clocks on the Python build. Nothing here can be compared against them: different implementation and different machine class, one difference per measurement is the whole discipline. Those need a host or CI number.

And that number is not mine to produce. #292 (open, on the benchmarking map #193) owns exactly it: the trend has failed on every push since 1654daf (devpod not found on PATH, exit 127), so all 35 published points measure the Python build, and its own resolution instruction is to name the commit of the first Rust point when it goes green. #193's charter (#197) also settles why a hand-run number would not substitute: the action compares each point only to the immediately previous one, and the first Rust point lands against a Python baseline, so that one comparison is meaningless by construction and must be labelled rather than read. Producing a host median here would duplicate #292's job and produce a figure with no series to sit in.

So the entry should stay open, rewritten to: "still needs the host/CI number — the per-trip arithmetic survived (#388); the headline wall clocks are #292's to publish." Recommended map gist in the report to the manager; per instruction I have not touched #139's body.

(b) The docs were wrong, and it is fixed: PR #403

(Landed in docs/performance.md, not README: #386 split the README while this was being written and took this whole section with it. See the breadcrumb.)

Not incomplete — misattributed. Every second under Measuring launch time (the two captures, the 20.834s cold decomposition, host medians warm 2.176s / cold-recreate 15.899s / first-ever 33.204s) was hand-copied on 2026-08-14 (d3b8ffb, 461f740), five days before 0.1.0 replaced the build that produced them; README says nothing about that, and the captures invoke dl-next, a name that today means the Rust build and then meant the Python one. test_bench_doc.py says in its own docstring that it does not check these numbers, so no guard was ever going to notice. Detail in the breadcrumb.

PR #403 adds one paragraph naming whose seconds they are and where current ones come from (the trend, which --record exists to feed and which has published nothing since the cutover — #292), plus one clause on README's existing "copied into this prose by hand" sentence, since the staleness is the same fact as the hand-copying. pixi run test: 346 passed. Docs-only; no Rust touched.

What deliberately did not go into the page: these nested-DinD numbers. Its figures are host figures offered "for scale"; a docker-in-docker second printed in the same prose gets read as a host second, whatever the sentence beside it says. The formats also stay untouched, because they are still right — the stage names are frozen in rust/devlaunch-core/src/timing.rs and timing.rs's own tests assert the README example's exact values.

Not settled

No host number for the Rust build, and no CI number. The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292's territory. Until it goes green, this map has no wall clock it can compare against One cold-path setup pass: fold the hostname into the probe's round trip, with per-stage outcomes #168's 2.48s / 5.99s, and nobody should claim a regression or an improvement from these figures.
Whether the wins are still the right wins. Trip-removal is out of road at three trips, and the reusable-connection findings (ControlMaster 0.010s, docker exec 0.07s vs 2.3s) say the remaining cost is connection setup that a different mechanism would delete outright. That is the four graduated tickets' business, not this one's.
The zellij stage (~1.95s of every cold launch) is a property of the target repo's dotfiles, not of dl. It is in these totals because blooop/dotfiles was the target; a different target has a different constant. Do not treat it as a dl cost, and note the map already scopes dotfiles/tool provisioning inside the container out.
devpod up is still one opaque number on both shapes — Can devpod up's output be decomposed into image / create / postCreate? #195 decided the sub-lines and they were never built (parser lives on branch research/devpod-up-decomposition). So the 2.3–2.8s here is undecomposed.
Operational note, nobody's ticket: a workspace container from the earlier measurement pass was still running in this devcontainer's docker daemon when I started, and a scratch DEVPOD_HOME is precisely what makes dl --ls / --purge unable to see it (CLAUDE.md's stated trade). I removed it, plus one my own runs left standing (devpod had already reclaimed the other two). Worth knowing before the next scratch-scoped measurement pass, not worth designing against.

breadcrumb: the README edit landed as PR #403, but not in README. #386 merged while this ticket was being written: it split README into docs/*.md and stripped its em dashes, and the whole Measuring launch time section, unattributed numbers included, is now docs/performance.md. So the branch was merged with main and the paragraph re-landed there in the new voice, with test/test_docs_prose.py (new, from that PR) asserting the no-dash rule the note now has to obey. PR touches exactly one file; pixi run test 411 passed.

Two things this changes for anyone reading this ticket later: the (b) answer is about docs/performance.md, not README, and the guard that could have caught the stale numbers but does not is still test_bench_doc.py, now repointed at that page.

breadcrumb: map #139's fog entry rewritten per this ticket's answer: "Do the wins survive the port?" narrowed rather than cleared, pointing at #292 for the host/CI number. PR #403's review check is the repo's own review gate (nothing had reviewed it yet), so a fresh-context reviewer was fired at it rather than the failure being treated as a code defect.

breadcrumb: correction on the record. This ticket's answer asserted the launch trend "has been dark since 1654daf" and that #292 still owed the first Rust point. That is false. #292's workflow fix merged as PR #373 on 2026-08-22; bench.yml has run green since, including today on 57955a3; and gh-pages dev/bench/data.js holds 56 points of which 36-56 are Rust. The claim was read off #292's open state rather than off the artefact, which is the same defect this ticket exists to fix, made one level out. Found by the fresh-context review of PR #403. Map #139's fog entry has been corrected and now says what is actually missing: the first 35 points measure the Python build and are unlabelled, so the cutover boundary is not legible and no comparison across it is sound.

The trend is not dark, and the chart's first 35 points are the real gap

breadcrumb: this ticket's answer claims an exhaustiveness it does not have, and here is the list it is missing.

The answer says "the one document that quoted unstamped figures is fixed". It is not the only one. Found by the fresh-context review of PR #403, verified again here:

docs/workspaces.md:144, docs/workspace-tools.md:137 and docs/workspace-tools.md:612 all quote ~1.7s of devpod ssh, a Python-era figure from Collapse the three sequential devpod ssh round trips on the cold path #157, with no build and no environment on it. This ticket's own Rust measurement puts the trip at 2.07s to 2.33s.
docs/performance.md:8 quoted 0.45s per devpod call unstamped, introduced in 22afd52 on 2026-08-07 on the Python build. That one sat in ## One round trip per question, just outside the Measuring launch time section These launch seconds measured a build that no longer ships #403's paragraph scopes itself to, so the file's most load-bearing number was the one the fix did not reach.

Fixed in #403: the 0.45s, stamped in place with #388's devpod status reading (0.43s to 0.67s, dl 0.13.0, nested in this repo's devcontainer) beside it and an instruction to read it as a floor rather than a stopwatch reading.

Deferred, with a reason rather than by omission: the three ~1.7s sites. They are not a stamp away from correct. #389 has since shown that the reason a trip costs what it costs was misdiagnosed: the 1.8s is one gap before ssh does any work, devpod's agent injection and container connect, and OpenSSH's own KEX/auth/channel work is 11ms of it. Two of those three sentences assert "almost all of it connection setup". So they need a rewrite that carries #389's mechanism, not a parenthesis, and folding that into a provenance-only docs fix would put a freshly corrected mechanism into a PR whose whole subject is not doing that. Whoever picks it up: #157's number, #388's floor, #389's mechanism, three call sites.

handoff

Parked at in review, changes requested. Two review rounds, each found a false
claim, and the second one was introduced by the fix for the first. That is the
signal to stop and hand this to a person rather than fire a third rewrite.

Where we are: PR #403 (docs/launch-timing-numbers-provenance, head 9edfea4).
CI is green apart from the two findings, which no guard covers. pytest on the
four doc guards passes (102), full suite 411. The review check passes.

The trail, because the pattern matters more than either finding:

Round 1 (blocking, fixed). The PR claimed the launch trend "has published
nothing since the cutover". False: The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292's fix merged as PR Put devpod on the PATH the bench runs dl from #373 on 2026-08-22
and gh-pages dev/bench/data.js holds 56 points with 36-56 Rust. The claim
had been read off The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292's open state rather than off the artefact — the exact
defect this PR exists to fix, one level out.
Round 2 (blocking, open). The rewrite replaced it with "a Rust point on
every push since 2026-08-22". Also false, and more finely: git log --first-parent 03b7de2..origin/main is 22 commits against 21 Rust points, so
20 of 22 pushes have one. 5573f77 (The feature README's Security Notes describe a layout it stopped using #381) and 7be9986 (0.12.0's herdr leftovers are not both inert #379) have none,
because four merges landed inside 130s and bench.yml's
concurrency: launch-benchmark keeps only one run queued. Both runs show
cancelled. The fix had re-derived from gh run list conclusions rather
than from data.js contents, and green runs differ from published points by
exactly those two.
Round 2 (minor, open). The document tells the reader to treat 0.45s as
"a floor" that "no host has re-measured since", while all 21 Rust points
publish warm / devpod-up — the lone devpod status probe, by the document's
own definition — at 0.242s to 0.383s on GitHub runners, i.e. consistently
below the stated floor.

The open thread, and why it is a person's call. Three rounds, three wrong
quantified claims about the same series, each smaller than the last. The
mechanical fixes are one sentence each: drop the frequency claim (or say "nearly
every push", naming the concurrency-cancellation cause), and either restate or
drop the floor. But the recurrence is the finding: this document keeps making
precise claims about a live series, and a claim about a live series goes stale by
itself. Whether docs/performance.md should assert frequencies and floors at
all, or should point at the chart and stop, is an editorial decision about the
document's purpose — which is exactly what this ticket set out to fix and is not
something an unattended run should keep re-deciding by rewriting.

First move on resume: decide that question first, then apply whichever of the
two one-sentence fixes it leaves standing. Both findings are in the review at
9edfea4, with the commands that establish them.

Not merged, not approved. Nothing on map #139 depends on this PR.

Navigation Menu

Repository navigation

Re-baseline launch latency on the Rust build #388

Question

Where these numbers come from

The recorded decomposition

(a) The fog entry narrows; it does not clear

(b) The docs were wrong, and it is fixed: PR #403

Not settled

handoff

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Re-baseline launch latency on the Rust build #388

Description

Question

Where these numbers come from

Activity

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

The recorded decomposition

(a) The fog entry narrows; it does not clear

(b) The docs were wrong, and it is fixed: PR #403

Not settled

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

blooop commented on Aug 24, 2026

handoff

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions