A warm attach reports neither a stale checkout nor a skipped setup pass, and an arm in a workspace id kills the launch with exit 126 #560

Three field observations from a fortnight of driving dl against a private
monorepo whose devcontainer brings up compose sidecars. Each cost real time, and
one of them cost a wrong conclusion, which is why it leads.

I checked all three against main before writing. Two are confirmed in code and
cited below. The third is confirmed as far as devlaunch goes and then runs out of
devlaunch to read, so it is marked as needing reproduction rather than dressed up
as a diagnosis.

Sections 1 and 2 share a root: a warm attach skips both the git work and the
setup pass, and the terminal says nothing about either. Both skips are deliberate
and both are documented, and neither is visible at the moment it matters. Section
3 is unrelated in mechanism and can be picked up on its own.

All line numbers are against main at the time of writing.

1. A warm attach verifies neither the commit nor the container, and reports neither

What happened

dl <owner>/<repo>@<branch> printed

Workspace <id> is already running, attaching...

and handed over a shell. Inside it, the checkout sat on the branch's first
commit; the clone's own origin/<branch> pointed at the default branch's tip,
tens of commits away; and the container was running a stock devcontainers/python
image built before the repo had a .devcontainer/ at all. dl <ws> reset, whose
help line is "Clean slate: remove everything, recreate", rebuilt the container and
changed none of that. Only dl <ws> rm and a relaunch did.

I spent that session concluding that a devcontainer booted correctly. It was never
built from the devcontainer I was looking at.

That is the part worth fixing. A stale checkout is an inconvenience. A launch that
looks like it verified new work when it verified neither the commit nor the image
silently invalidates whatever you concluded inside the container.

Where it lives

The attach-or-build fork happens before the verb is consulted at all, in
Launch::place:

• flows/launch.rs:3664 place dispatches on the plan.
• flows/launch.rs:3720 place_triple calls resolve_triple.
• flows/launch.rs:3178 resolve_triple delegates to
  lifecycle::resolve_known_workspace (flows/lifecycle/state.rs:139), which runs
  devpod status <id> --output json. Recognised gives
  Resolution::Warm { placement } (launch.rs:3196); unrecognised gives
  Resolution::Cold { workspace } (launch.rs:3203).
• flows/launch.rs:3777-3787 is the whole of it. The Warm arm returns the
  placement; only the Cold arm calls prepare, and prepare is the only
  production caller of WorkspaceCloneManager::prepare_cold.

So no git command runs on the warm arm. That is not a bug, it is
#144's decision built by
#149 and
#150, and docs/workspaces.md:167
states it outright under "How fresh a launch is":

Attaching to a workspace devpod already knows: no git at all. The workspace
is exactly as you left it; freshness inside it is your own git pull.

The banner is dl's own line on stderr, not devpod's:
LaunchNotice::AlreadyRunningAttaching (declared flows/launch.rs:484, said at
flows/launch.rs:3843 inside run_attach, rendered at dl/src/render.rs:2590).
It names the workspace and nothing else. The full warm trace is already a golden,
in rust/dl/tests/launch.rs:339
a_warm_attach_is_one_status_probe_and_then_the_session: two devpod calls, a
status and an ssh.

reset does not help, and cannot. LaunchVerb::Reset maps to Rebuild::Reset
(flows/launch.rs:3438), which is the single flag --reset
(flows/launch.rs:922), and run_rebuild (flows/launch.rs:3869) goes straight
to bring_up. The verb is applied to a Placement that place already decided,
so reset rides the warm arm too: no fetch, no re-clone, no ref update. Pinned as
argv at rust/dl/tests/launch.rs:930
recreate_and_reset_each_pass_their_own_flag_and_then_attach, which is devpod up <id> ... --reset then devpod ssh <id> and no git.

On devpod's side --reset implies --recreate and additionally removes the
source, but only for a git-sourced workspace: prepareGitWorkspace in devpod's
cmd/agent/workspace/up.go is the only consumer of the flag beyond that implication,
and dl hands devpod a local folder (its own clone under
<cache>/repos/<owner>/<repo>/<workspace-id>), never a git URL. So devpod removes a
content folder that does not exist here, and dl's clone stands at whatever commit
it was on.

The stale image is a consequence of the stale clone rather than a second fault.
reset rebuilds from the .devcontainer/ in the checkout; the checkout predated
it; there was nothing to build from, so devpod fell back to a default image. One
cause, two symptoms.

And rm works for a narrower reason than it looks. Going cold is not by itself
enough. prepare_workspace only moves the working tree to the fetched ref when the
clone directory is new (checkout_reset to origin/<branch>,
flows/workspace_clone.rs:939); an existing directory gets a plain git checkout <branch>, deliberately, to preserve local work (workspace_clone.rs:958-961,
pinned by a_workspace_already_on_disk_is_checked_out_and_nothing_else,
workspace_clone.rs:1796). A workspace devpod has forgotten but whose clone
survives therefore fetches into the bare and then does not advance. rm refreshes
only because it deletes the clone as well as the workspace
(remove_workspace_by_id, flows/lifecycle/delete.rs:292).

Reproduce

dl owner/repo@somebranch          # cold: creates the workspace, fetches the ref
dl owner/repo@somebranch stop
# push several commits to somebranch, including a new .devcontainer/
dl owner/repo@somebranch          # warm: attaches, no fetch
CLONE=<cache>/repos/owner/repo/<workspace-id>   # the path dl derives
git -C "$CLONE" log --oneline -1
git -C "$CLONE" rev-parse --short origin/somebranch
dl owner/repo@somebranch reset    # still no fetch, still the old commit

The two rev-parse outputs disagree, and nothing dl printed said so.

What I think the fix is

Report it at the attach, computed from the clone dl already has, with no network
call. Not a fetch on attach, and not a fetch on reset.

Fetching loses the argument #144
already settled: the launch path makes no broad network call, and the warm path
makes no git call at all so a hot attach stays hot. Putting one back to fix a
reporting problem trades the wrong thing.

A fetch is also not needed, because the fact was available locally the whole time.
In the observed case HEAD and origin/<branch> inside dl's own clone
disagreed by tens of commits. Two rev-parse calls against a local repository cost
single-digit milliseconds, against the 0.43-0.74s a single devpod status costs on
this path (#393). So:

Workspace <id> is already running, attaching...
  its checkout is 37 commits behind origin/somebranch (last fetched 6 days ago)

The claim is narrow and it is honest. dl says how the clone stands against the ref
it last fetched. It does not claim to know the remote, so it needs no network and
cannot be wrong about one. Being behind a six-day-old ref is already the whole of
what a user needs, and it points at "fetch, or rm" without dl deciding for them.

Four things a builder should plan for rather than discover:

• The seam is place_triple (flows/launch.rs:3720), not run_attach.
  run_attach holds only a Placement, and Placement::title's docs
  (flows/launch.rs:3105-3115) say a triple is not recoverable from a workspace id.
  place_triple still has owner, repo, branch and remote_url in hand and
  already forks Warm from Cold at :3777.
• There is no primitive for the comparison yet. clients/git.rs has
  verify_ref (:1042, existence), symbolic_ref, and ls_remote_heads (names
  only, :1060). No method returns a ref's sha. This fix needs one new client
  method, and that is where its red test starts.
• ColdMachinery::open is off limits on this path.
  #145 is why: open
  (flows/launch.rs:2924) takes the metadata lock and runs the id-scheme
  migration. ColdMachinery::recorded (flows/launch.rs:2943) is the lock-free,
  migration-free read that the id-collision guard already uses, and is the
  precedent to follow. Better still, the clone path is a pure function of the
  triple, clone_dir(repos_dir, &WorkspaceId) =
  <repos>/<owner>/<repo>/<workspace-id> (pinned around
  flows/repo_manager.rs:2257), so a triple launch needs no records read at all.
• Two shapes must stay silent. dl <ws> by bare name cannot derive the clone
  path, because Plan::Existing (flows/launch.rs:2790) carries the raw spec and
  nothing else. And Placement::Listed, a foreign devpod workspace, has no clone dl
  owns. Silence on both. The shape that misleads is the triple, because a triple
  names a branch and so implies a claim about it.

There is already a staleness vocabulary to extend rather than invent: BranchBase
(flows/workspace_clone.rs:78) and CacheNotice::PreparedFromStaleBase, built for
#245. It means "a fetch was
attempted and did not land", which is not this, and it only exists on the cold arm.
Worth reading before adding a fourth vocabulary for the same subject. Note also
that PreparedWorkspace.base is currently discarded by launch::prepare
(flows/launch.rs:3388-3395 keeps only prepared.path), so the cold path's own
verdict already goes nowhere except the notice.

Separately and cheaply: "Clean slate: remove everything, recreate"
(dl/src/cli.rs:658, mirrored at README.md:207) is a promise about the checkout
that reset does not keep. The clean slate is the container and its volumes.
Either that line says so, or reset grows a fetch and stops being a plain devpod
passthrough. My vote is the wording, plus a sentence in docs/cli.md saying rm
is the only verb that refreshes git state. docs/cli.md says nothing about launch
freshness today.

How to test it

Red before green, at seams that already exist.

1. The test a naive fix will break, and should not.
   a_launch_that_matches_its_own_record_attaches_and_reads_no_machinery
   (flows/launch.rs:7952) asserts cold.opens.get() == 0. Read it first: it is the
   guard that makes the bullet about ColdMachinery::open load-bearing rather than
   advisory.
2. Unit red test. flows/launch.rs's inline test module has Scene (:4204,
   with .with_running(id)), a fake runner recording every devpod and git argv
   (devpod_commands, :4377), and RealCold (:4390) driving a real cache tree.
   FakeGit and the Cache fixture live in flows/repo_manager.rs's test module
   (:1832, :1962) and are already imported across modules. Build a clone whose
   HEAD is behind its refs/remotes/origin/<branch>, launch warm, assert the new
   notice arm.
3. Pin the negative in the same place. The warm attach must still issue zero
   git fetch and no extra devpod call. That is what stops this fix from quietly
   becoming the fetch it was chosen over. no_launch_fetches_inside_the_workspace_clone
   (flows/workspace_clone.rs:1837) is the existing shape of that assertion.
4. Binary boundary. The wording is the deliverable and render.rs is not
   covered by a core assertion, so
   a_warm_attach_is_one_status_probe_and_then_the_session
   (rust/dl/tests/launch.rs:339) is the golden that has to gain the line. The world
   is built by World::with (rust/dl/tests/launch.rs:59) over
   launch_scenario.py, and the --warm fixture is the one to extend.
5. docs/workspaces.md "How fresh a launch is" gains the sentence saying the attach
   now reports this. Check test_docs_prose.py before writing it, since that page
   is in its glob.

2. Dotfiles never landed in a compose-created container, and dl never said whether it asked for them

What happened

Containers created through the compose path came up bare: no dotfiles clone
anywhere under the container's home, in either a bootstrapped or an unbootstrapped
container, no shell environment, and a statusline configured through a bind-mounted
config doing nothing. DOTFILES_URL was set in ~/.devpod/config.yaml and in
~/.cache/devlaunch/context-options.json, and it made no difference.

What the code settles

devlaunch does forward dotfiles, and the whole of how it decides is four lines:

• flows/launch.rs:554 ContextOptions::dotfiles_url reads the key DOTFILES_URL.
• flows/launch.rs:559 dotfiles_script reads DOTFILES_SCRIPT.
• flows/launch.rs:576-587 up_args turns those into --dotfiles <url> and
  --dotfiles-script <script>.
• flows/launch.rs:1052 extends the devpod up argv with them, at the one
  production call site (flows/launch.rs:1499).

Those values come from devpod context options --output json
(clients/devpod.rs:1017) and from nowhere else. DOTFILES_URL is not read as a
process environment variable anywhere in the tree, and devpod's config.yaml is
only ever stated, never parsed (clients/devpod_home.rs:90, consumed solely by
the mtime comparison in cached_options).

Two things follow that anyone reproducing this needs first.

context-options.json is dl's cache of that command's output, not an input, and
its on-disk shape is flat. cached_options (flows/launch.rs:642) deserialises
it as BTreeMap<String, String>, and the doc comment above it says a file that is
not {string: string} "reads here as no cache at all". devpod's own shape is
nested, {"DOTFILES_URL":{"value":"..."}}. So hand-writing devpod's shape into that
file sets nothing and reports nothing: the read fails silently and dl asks devpod,
which was the only answer that ever counted. Check devpod context options --output json directly and ignore the cache file.

devlaunch has no compose branch at all. It never opens a devcontainer.json.
domain/spec.rs builds a path to hand to devpod up --devcontainer-path and
nothing reads the file. There is no dockerComposeFile, service or runServices
anywhere in Rust source, no docker exec, and no container-name targeting: every
trip is devpod ssh <id>. The only compose-aware code in the tree is
clients/docker.rs:110 filtering docker ps on
label=com.docker.compose.project=<id> for dl <ws> kill, and volume-name sweeping
in dl <ws> rm. devlaunch cannot be treating the compose path differently for
dotfiles, because devlaunch does not know the path is compose.

Needs reproduction, and here is the open question. With devlaunch ruled out, two
explanations remain and the code cannot choose between them:

1. The step is create-time only and every session that looked was attaching, not
   creating. This is very likely at least a contributor. docs/workspace-tools.md
   says "devpod applies dotfiles when it provisions a workspace, so a workspace
   that has been up for a fortnight still has the dotfiles it was born with", and a
   warm attach in dl runs no pass of any kind: run_attach returns at
   flows/launch.rs:3853 before ever reaching bring_up. This is the same family
   as the already-documented "a mount lands only when a container is created"
   (docs/workspace-tools.md, "Existing containers, and what a recreate is for").
2. devpod's compose create path never runs the step. Nothing in this repo can
   answer that.

One clean run tells them apart. With DOTFILES_URL confirmed present in devpod context options, dl <ws> rm and then a fresh cold launch of a compose
devcontainer, watching for --dotfiles in the devpod up argv and for the clone
appearing in the container. If dotfiles land on the cold create it is (1), and
there is nothing to fix in devpod. If they do not it is (2), and it belongs
upstream the way #423 does.

One thing that is not the bug: the dotfiles repo in question has a guard that
deliberately refuses to install when its config trees are bind-mounted from another
home. That refusal is correct behaviour on the dotfiles side and is not devlaunch's
concern. The complaint here is that no clone was present at all, so nothing
container-local had been set up for the guard to refuse.

What I think the fix is, whichever way the reproduction goes

dl should say whether it passed --dotfiles. Today it reads devpod's context,
silently forwards two flags or silently omits them, and there is no way from the
terminal to tell which happened. That is what made this take a fortnight rather
than an afternoon: three plausible causes and no observation to cut between them.

One notice on the cold path is enough. dotfiles: <url> (from devpod context) when
the flags go, and dotfiles: not configured in devpod context when they do not. It
costs nothing, the options are already in hand at flows/launch.rs:1052, and it
turns a silent policy into a checkable fact. Test it as in section 1: assert the
notice against the recorded argv, so the printed line and the flags cannot drift.

The outcome should also name the two existing escape hatches, because they are the
answer for a long-lived container and neither is discoverable from the failure:
dl <ws> dotfiles, and DEVLAUNCH_DOTFILES_ON_ATTACH=1 (flows/launch.rs:104,
documented in docs/workspace-tools.md under "Refreshing dotfiles on attach").

3. A workspace id containing arm gets an arm64 agent, and the launch dies with exit 126

What happened

A branch whose name contained armature produced a workspace that would not
launch. devpod injected the arm64 agent binary into an x86_64 container and the
launch failed with

devcontainer up: inject agent: [version check] failed to get remote agent version: exit status 126

126 reads as "not executable", which is true and useless. Nothing in the message
names the architecture or the name that chose it. Every branch containing alarm,
warm, charm, swarm, harm or farm is a candidate.

Where it lives

The root cause is upstream and it is a substring match, as suspected. In devpod's
injected shell script, pkg/inject/inject.sh:

is_arm() {
    case "$(uname -a)" in
        *arm* | *arm64* | *aarch* | *aarch64*) true ;;
        *) false ;;
    esac
}

uname -a includes the nodename, so any container whose hostname contains arm
reads as an ARM machine. createBinaryLoader in pkg/agent/inject.go takes that
boolean straight through to arch = "arm64", downloads the wrong binary, and
performVersionCheck then fails executing it. Hence 126.

This is already fixed, on a branch nobody has merged. blooop/devpod carries the
one-line change as
fix/detect-arch-with-uname-m,
which globs uname -m with anchored patterns and adds an is_unsupported_arm arm
so a 32-bit ARM userspace gets named rather than leaving the same 126 to be
diagnosed. It is not in skevetter/devpod and there is no PR open there. devlaunch
pins devpod >=0.26.1,<0.27 (pyproject.toml, conda.recipe/recipe.yaml) from a
channel that repackages release binaries and so cannot patch the script, which
means devlaunch inherits this until an upstream release carries it.

The part that is devlaunch's own. The setup pass runs sudo hostname <workspace-id> inside every container it opens:

• flows/provision.rs:249 HOSTNAME_STAGE
• flows/provision.rs:1507-1513 the stage itself,
  format!("sudo hostname {}", quote(workspace))
• pinned by the_hostname_stage_names_the_container_after_the_whole_id
  (flows/provision.rs:5179)

and the workspace id is derived from the branch by slug
(domain/workspace_id.rs:262), which lowercases and replaces non-word characters.
feature/armature slugs to feature-armature, so devlaunch puts arm into uname -a by hand. In the failure I hit, the hostname came from the project's own compose
file rather than from this stage, but devlaunch writing branch text into the
container's nodename is the same trap with devlaunch's name on it.

Reproduce

Confirm the mechanism in any running container, no devlaunch needed:

docker exec <container> sh -c 'case "$(uname -a)" in *arm*) echo ARM;; *) echo NOT;; esac'

Then launch a branch named feature/armature on an amd64 host against a repo whose
devcontainer sets the container hostname from the workspace name, and watch file /usr/local/bin/devpod inside the container report an aarch64 binary.

To unblock a container already in this state without recreating it, copy the correct
binary in (docker cp $(command -v devpod) <container>:/usr/local/bin/devpod) and
reconnect. Do not use restart or recreate: the recreate wipes it.

What I think the fix is

Three candidates, and only one of them is right.

Sanitising the workspace id is the tempting one and it is wrong. slug is pinned by
golden tests against the retired Python implementation
(golden_slugs_reproduce_the_python_slug_rule, golden_triples_do_not_move, both
in domain/workspace_id.rs), the id's suffix is a hash of the triple, and changing
the rule renames every existing workspace and orphans the lot. That is a large
breaking change to dodge a one-line upstream bug.

Masking arm in the hostname stage alone is also wrong. It breaks the
correspondence the stage exists for, that the prompt, the tab and dl --ls all show
the same string, and it hides the trap from every other consumer of the name instead
of fixing it.

So: get the upstream fix released, and until then make 126 legible.

1. Open the PR on skevetter/devpod from the branch that already exists, the way
   #423 is chartered. The
   deliverable is a link.
2. In devlaunch, recognise this failure and name it. LaunchRefusal::UpRefused
   renders nothing of dl's own today (dl/src/render.rs:2856 returns None), so
   devpod's raw sentence is the whole of what a user gets. Match devpod's stderr for
   inject agent together with exit status 126 and print one line saying the
   container's hostname contains arm, so devpod picked the arm64 agent. dl already
   reads devpod's stderr as it arrives (clients/devpod.rs:331), and
   dl/src/render.rs:1612 is the existing shape for a refusal that branches on
   stderr text.

Point 2 is worth doing even after the upstream fix ships, because the pin is a range
and anyone on an older devpod still hits it.

How to test it

Point 1 has no test. The deliverable is the PR link.

Point 2 is a rendering test. Feed the fake devpod a failed up whose stderr is the
exact sentence above with exit 126, and assert the rendered refusal carries the
diagnosis. ContainerRefusal::Refused { exit, stderr } at dl/src/render.rs:1612
is the prior art for both the match and its test. Pin the negatives too: a 126 that
is not an agent injection, and an inject agent failure that is not 126, must both
keep devpod's own wording rather than gaining a wrong explanation.

What I verified, and what I did not

Confirmed against main at the citations above: the warm/cold fork and that reset
rides the warm arm; that the attach banner is dl's own line and carries only the id;
that --reset cannot touch dl's clone; that an existing clone directory is never
advanced to the fetched ref even on the cold arm; that dl forwards --dotfiles only
from devpod context options; that context-options.json is a flat-map cache and
not an input; that devlaunch has no compose branch and never parses
devcontainer.json; that a warm attach runs no setup pass; is_arm's uname -a
glob and the fix branch's existence; and that devlaunch's own hostname stage writes
the branch-derived id into the container's nodename.

Not confirmed, and marked as such above: whether devpod's compose create path runs
the dotfiles step. Section 2 says how to find out.