Nested devpod writes broken ProxyCommand entries into the host's real ~/.ssh/config #112

Activity

1. blooop

   added

   wayfinder:grilling

   on Aug 8, 2026
2. blooop

   added a parent issue on Aug 8, 2026

   • Develop devlaunch inside its own devcontainer, on an isolated Docker daemon #93
3. blooop

   self-assigned this

   on Aug 8, 2026
4. blooop commented on Aug 8, 2026

   

   blooop

   on Aug 8, 2026

   OwnerAuthor

   More actions

   breadcrumb: starting. Snapshot of the real host file before any work: sha256sum ~/.ssh/config = cd9519fbe1fac1a2279f2414349e366cf63e7f1a0a0a346e7833718e463ad205. Two facts established by reading, before any metal:

   1. --ssh-config is up-only, confirmed against the installed binary (v0.26.1), not just against PR Give the test suite a devpod namespace of its own #106's review: devpod up --help has both --configure-ssh (default true, DEVPOD_CONFIGURE_SSH) and --ssh-config (DEVPOD_SSH_CONFIG, "if empty will use ~/.ssh/config"); devpod ssh|delete|stop --help have neither. So candidate 2 (set DEVPOD_SSH_CONFIG in containerEnv) redirects the write but leaves delete's cleanup pointed at $HOME/.ssh/config — it would split write and unwrite across two different files. That is worse than either endpoint: blocks would be written to the container-local file and never removed from it, while delete rewrites the host's real file looking for a block that was never there.

   2. Nothing in this repo reads ~/.ssh except the mount itself. Only hit outside .devcontainer/ is gh_auth.ssh_args_and_env(), which is about --send-env GH_TOKEN, not ssh keys. And git signing is not configured on this host at all — no gpg.format, no user.signingkey in ~/.gitconfig or its ~/.gitconfig.local include — and ~/.gitconfig is not mounted into the devcontainer anyway. So the "signing key" half of the ticket's tension is thinner than stated; what is actually load-bearing is agent.sock (SSH_AUTH_SOCK=/home/vscode/.ssh/agent.sock, 1 identity in the host agent) and known_hosts.

   Next: confirm on metal whether devpod delete removes its own block, using HOME= a scratch dir so the real file is never a candidate target.
5. blooop commented on Aug 8, 2026

   

   blooop

   on Aug 8, 2026

   OwnerAuthor

   More actions

   breadcrumb: live corroborating evidence from PR #106's retry session, directly relevant to this ticket's core question.

   Two DevPod Host blocks disappeared from the host's real ~/.ssh/config mid-session, for workspaces already absent from ~/.devpod (whose hash never changed). The agent that observed it had run only version, --help, list and a scoped provider add, and grep over devlaunch/ test/ for ssh/config|DevPod Start finds zero hits outside the scoping module — so devlaunch did not do it.

   Most likely cause, and it is this ticket's subject seen from the other side: devpod delete does not accept --ssh-config. A concurrent agent running with a scoped DEVPOD_HOME deleted its own workspaces, and devpod pruned the corresponding blocks from the real ~/.ssh/config because that is the only ssh config delete knows about.

   Two consequences for your decision:

   1. It is not only a write leak — devpod running scoped also mutates the real file on delete. Any fix that only redirects writes on devpod up leaves the delete path pointed at the developer's file.
   2. It bears on your "always vs only-on-bad-exit" question from the opposite direction: the cleanup path reaching the real file is why earlier prototypes found zero leftover blocks after cleanup. The leak may look self-healing precisely because the cleanup is itself unscoped.

   Worth confirming rather than taking on trust — the attribution is inferred from timing and elimination, not observed directly.
6. blooop commented on Aug 8, 2026

   

   blooop

   on Aug 8, 2026

   OwnerAuthor

   More actions

   breadcrumb (correction — I was wrong in the last one, and it matters): I read the actual source of the devpod build we ship (~/.pixi/envs/devpod/bin/devpod, v0.26.1 — note it is the skevetter/devpod fork, not loft-sh, per its symbol table).

   DEVPOD_SSH_CONFIG is not write-only-with-no-cleanup. It is persisted per workspace and honoured at delete:

   • pkg/provider/workspace.go:73 — SSHConfigPath string +""+json:"sshConfigPath,omitempty"+""+, i.e. up's --ssh-configvalue is stored in the workspace'sworkspace.json`.
   • pkg/client/clientimplementation/workspace_client.go:441 — delete calls DeleteWorkspaceFolder{SSHConfigPath: s.workspace.SSHConfigPath}, which resolves that path and calls ssh.RemoveFromConfig.
   • pkg/ssh.ResolveSSHConfigPath("") → $HOME/.ssh/config. It reads no env var; the env var only ever reaches it via up's flag.

   Verified against the real state on this host: the three live workspace.json files have no sshConfigPath key (omitempty, never passed), so their deletes resolve to $HOME/.ssh/config. So "subcommand-scoped" is true of the flag and misleading about the behaviour — candidate 2 is symmetric after all, and is a stronger candidate than the ticket assumed.

   A second thing the source changes. pkg/ssh/config.go:351 — writeSSHConfig is os.WriteFile(path, content, 0o600): every up and every delete rewrites the whole file, serialised only by a process-local mutex. The container is therefore not appending a line to the host's ~/.ssh/config; it is rewriting the entire file, my personal Host homeassistant block included, with no cross-process locking against the host's own dl. That is a lost-update hazard on top of the dead-entry one, and it is an argument for a boundary that does not depend on devpod's good behaviour.

   Next: metal test of what candidate 1 actually costs — socket-only mount, known_hosts, and whether git@github.com still authenticates.
7. blooop commented on Aug 8, 2026

   

   blooop

   on Aug 8, 2026

   OwnerAuthor

   More actions

   breadcrumb (on metal — what candidate 1 actually costs, measured): no devpod, no nested workspace, no container image built. Three docker run --rm invocations against the ubuntu:24.04 that was already on this host, reproducing candidate 1's mount set exactly.

   A. Socket-file bind + read-only known_hosts, no config, no private key — everything load-bearing still works:

   --mount type=bind,src=$HOME/.ssh/agent.sock,dst=/root/.ssh/agent.sock
   --mount type=bind,src=$HOME/.ssh/known_hosts,dst=/root/.ssh/known_hosts,readonly
   -e SSH_AUTH_SOCK=/root/.ssh/agent.sock
   
   $ ls -la /root/.ssh
   srw------- agent.sock          # a socket, through a file bind mount
   -rw------- known_hosts
   $ ssh-add -l          → 256 SHA256:C+FU0…QU4 ags@ags-dell (ED25519)   rc=0
   $ ssh -T git@github.com → Hi blooop! You've successfully authenticated…
   $ git ls-remote git@github.com:blooop/devlaunch.git HEAD → 05ee73e…  rc=0
   

   So id_ed25519 is not load-bearing: with no key file in the container at all, the forwarded agent carries the identity and git-over-ssh authenticates. Docker binds a unix socket file fine — this is the docker.sock pattern.

   B. known_hosts is load-bearing — drop it and the container fails closed, non-interactively:

   $ git ls-remote git@github.com:blooop/devlaunch.git HEAD
   Host key verification failed.   rc=128
   

   (StrictHostKeyChecking=accept-new recovers it, and ssh-keyscan at image-build time would too — but mounting the host's file read-only is the option that keeps today's verified behaviour and still cannot write back.)

   C. A --mount with a missing source aborts the container and creates nothing — invalid mount config for type "bind": bind source path does not exist. Relevant because narrowing the mount narrows what must exist on the host.

   Host files unchanged throughout: ~/.ssh/config cd9519fb…ad205 before and after all three, ~/.ssh/known_hosts 679e58fc…0a030 before and after.

   And the delete question is settled — the leak is on abandonment, not always. cmd delete → clientimplementation.DeleteWorkspaceFolder → ssh.RemoveFromConfig runs unconditionally on every delete (workspace_client.go:441), which matches #96's observation that cleanup left zero wf96 lines. Two riders that stop this being reassuring:

   1. It runs unconditionally and against $HOME/.ssh/config whenever sshConfigPath was never recorded. So a nested delete rewrites the host's whole real file even when it has nothing to remove.
   2. transformHostSection re-reads the file through a line scanner and re-joins it, so a lifecycle is no-net-change modulo normalisation, not a no-op. The happy path races; only the abandoned path leaks.
8. blooop commented on Aug 8, 2026

   

   blooop

   on Aug 8, 2026

   OwnerAuthor

   More actions

   agent-decided (principle 1, long-term maintainability — with principle 2 picking the shape): narrow the mount. Stop bind-mounting the host's ~/.ssh directory into the container; mount only the two files that were ever load-bearing, and let /home/vscode/.ssh be container-local so devpod's blocks land in a file that dies with the container.

   // .devcontainer/devcontainer.json — "mounts", replacing the single ~/.ssh line
   "source=${localEnv:HOME}/.ssh/agent.sock,target=/home/vscode/.ssh/agent.sock,type=bind",
   "source=${localEnv:HOME}/.ssh/known_hosts,target=/home/vscode/.ssh/known_hosts,type=bind,readonly",

   containerEnv.SSH_AUTH_SOCK is unchanged (/home/vscode/.ssh/agent.sock — the same path it already names), and the Dockerfile already creates /home/vscode/.ssh mode 700, so nothing else moves. initializeCommand should touch ~/.ssh/known_hosts on the host, because a --mount with a missing source aborts the container.

   Why this and not the other three

   vs. mounting ~/.ssh read-only. Fails closed in the wrong place: writeSSHConfig is a plain os.WriteFile, so nested devpod up errors out rather than degrading, ssh can no longer append known_hosts, and the host's private key stays exposed anyway. Cheapest to state, most expensive to live with.

   vs. DEVPOD_SSH_CONFIG in containerEnv. This one is better than the ticket credited it — I was wrong in my first breadcrumb and corrected it in the second: the path is persisted (sshConfigPath in workspace.json) and delete does honour it, so it is symmetric, not write-only. It is still the wrong fix, for two reasons. (a) Its correctness rests on three undocumented internals of a fork (skevetter/devpod) — the flag being env-bound on up, the value surviving as omitempty into workspace.json, and delete resolving from there. Any one of them changing silently re-points a whole-file os.WriteFile at the developer's real ~/.ssh/config; that is a bad thing to hang a safety property on, and PR #106's review already read this API wrong once in the other direction. (b) It fixes devpod and only devpod. The directory stays mounted and writable, so ssh appending known_hosts, and anything else the container grows, still write to the host's files. Narrowing the mount is a docker-level fact that holds no matter what runs inside.

   vs. DEVPOD_CONFIGURE_SSH=false (a fourth candidate, worth recording because it looks right and is not). dl never uses the <ws>.devpod aliases — workspace_ssh shells out to devpod ssh <id> (devlaunch/dl.py:989) — so suppressing the write costs nothing. But it does not close the hole: delete calls RemoveFromConfig unconditionally, so it still rewrites the host's whole ~/.ssh/config hunting a block that was never written. Fine as belt-and-braces on top; useless alone.

   What it costs, stated plainly

   • The private key stops being visible in the container. Measured as free: with no key file present at all, ssh-add -l returns the agent's identity and git ls-remote git@github.com:… exits 0. The ticket's "signing key" leg turned out not to exist yet — there is no gpg.format or user.signingkey in ~/.gitconfig or its ~/.gitconfig.local, and ~/.gitconfig is not mounted into the devcontainer regardless. If signing is turned on later the answer is a third read-only file bind, not the directory back.
   • known_hosts becomes read-only, so the container cannot learn new hosts. For github.com that is the behaviour you want; for a repo on an unknown host, git fails closed with Host key verification failed instead of prompting. Escape hatch if it bites: StrictHostKeyChecking=accept-new in a container-local ~/.ssh/config — which is now genuinely container-local and therefore safe to write.
   • The socket bind pins an inode. If the host's agent restarts, forwarding dies until the container is restarted; the directory mount survived that. Bounded, and an agent restart usually coincides with a session that restarts the container anyway.
   • Two host paths must exist rather than one. agent.sock was already assumed by the existing SSH_AUTH_SOCK value, so that is not new; known_hosts is, hence the touch in initializeCommand.

   How it gets tested

   Structural, in CI, no docker: assert no mount in devcontainer.json has target exactly /home/vscode/.ssh, and that whatever SSH_AUTH_SOCK names is itself a mount target.

   e2e, opt-in marker, the real assertion: sha256sum ~/.ssh/config on the host → start the devcontainer → inside it run a nested lifecycle → kill the container with the nested workspace still alive → re-hash and assert byte-identical, plus grep -c "DevPod Start" unchanged. The abandonment step is the whole test: the happy path already cleaned up before this fix, so an up-then-delete test would pass against the broken config. Pair it with an in-container liveness check — ssh-add -l non-empty and git ls-remote git@github.com:… exit 0 — because the fix is only correct if agent forwarding survives it. The test is safe to run against the developer's real file by construction: a passing run writes nothing to it.
9. blooop

   closed this as completedon Aug 8, 2026