What a devpod ssh trip costs per connection, and whether it can be reused #389

Activity

1. blooop

   added

   wayfinder:research

   on Aug 24, 2026
2. blooop

   added a parent issue on Aug 24, 2026

   • Launch latency: dl <spec> -- <cmd> to a running command #139
3. blooop

   marked this as blocking on Aug 24, 2026

   • The trip floor, not the trip count: how dl reaches into a running container #390
4. blooop

   self-assigned this

   on Aug 24, 2026
5. blooop commented on Aug 24, 2026

   

   blooop

   on Aug 24, 2026

   OwnerAuthor

   More actions

   breadcrumb: ssh -vvv puts all of the 1.67s in one gap — between OpenSSH writing its version string into the ProxyCommand and reading the remote banner back. KEX + auth + channel + exec + exit is ~11ms. The cost is entirely devpod ssh --stdio getting a Go ssh-server up in the container; inside it, agent-injection ~531ms and the agent-version handshake ~531ms are the two big blocks, and both re-verify per-container-lifetime facts on every single connection.
6. blooop commented on Aug 24, 2026

   

   blooop

   on Aug 24, 2026

   OwnerAuthor

   More actions

   breadcrumb: Two measurements that change the shape of the sibling. (1) The ticket says a reused master still carries --send-env. It does not unless the master was opened declaring that variable: SendEnv is filtered by the master process against its own send_env list, so a reuse whose master lacks it gets an empty GH_TOKEN silently. (2) devpod ssh --stdio takes a per-workspace lock, so N concurrent non-multiplexed trips serialize: 8 parallel ssh runs finished 2s apart in a staircase (9.9s..23.9s), the same 8 over one master all finished at 8.02s.
7. blooop commented on Aug 24, 2026

   

   blooop

   on Aug 24, 2026

   OwnerAuthor

   More actions

   agent-decided (1):

   Findings on branch research/devpod-ssh-trip, full write-up at docs/research/devpod-ssh-trip.md (commit 9209abe). Principle (1) governed the shape of the answer: the two things worth carrying out of this are a safety fact and a silent-failure correction, both of which are about what a change here would cost to live with, not about seconds.

   Measured 2026-08-24 inside this repo's devcontainer, devpod v0.26.1, OpenSSH_9.6p1, docker 29.7.2, 8 CPUs, throwaway one-line devcontainer workspace, DEVPOD_HOME/DEVPOD_SSH_CONFIG scoped to /tmp/res389. Workspace and container deleted afterwards. Disclosure: three other agents were running docker/devpod experiments in the same container for part of the run, so late readings are 20-60% above early ones on identical commands. Both are given in the write-up; nothing rests on a gap smaller than that spread.

   1. Where the 1.8s goes

   All of it is one gap, and the gap is before ssh does any work. Timestamped ssh -vvv:

   0.000  OpenSSH_9.6p1 ...
   0.001  debug1: Executing proxy command: exec ".../devpod" ssh --stdio ... res389 ...
   0.003  debug1: Local version string SSH-2.0-OpenSSH_9.6p1
   1.662  debug1: Remote protocol version 2.0, remote software version Go   <-- one 1.662s gap
   1.673  debug1: Exit status 0
   

   KEX, host key, userauth, channel open, exec, exit-status: 11ms, all of it after the gap. Also note remote software version Go: there is no sshd in the container, devpod starts its own Go ssh server per connection.

   Inside the ProxyCommand (devpod ssh --stdio --debug, timestamps relative):

   block	cost
   devpod CLI init + acquired workspace lock	~100ms
   workspace-info decode/write, docker resolve	~90ms
   starting agent injection -> line=done	531ms
   done exec -> remote agent version matches expected version	531ms
   connected to container	266ms
   then su -c 'devpod helper ssh-server --stdio' -> the banner OpenSSH was waiting for

   The two 531ms blocks are 1.06s of the 1.66s, and both are the in-container agent binary being asked to print its own version string. Source, skevetter/devpod v0.26.1: pkg/agent/inject.go::versionChecker.buildExistsCheck templates the inject guard as ! { [ -x "<p>" ] && [ "$("<p>" version)" = "<ver>" ]; }, and detectRemoteAgentVersion then runs <p> version again as its own exec. Measured directly: docker exec <cid> /usr/local/bin/devpod version = 479-501ms, against docker exec <cid> true = 65-74ms. 74+490 twice is 1128ms against 1062ms observed. The arithmetic closes.

   And ~0.78s of that is telemetry. cmd/version.go prints one string; the time is telemetry.StartCLI in root's PersistentPreRunE plus CollectorCLI.Flush() in Execute (cmd/root.go). A/B: host devpod version 398-441ms, with DEVPOD_DISABLE_TELEMETRY=true 45-55ms; in-container agent 479-501ms, with docker exec -e DEVPOD_DISABLE_TELEMETRY=true 99-111ms. devpod --help short-circuits both at 46-55ms.

   Per-connection vs per-container-lifetime: none of the 1.66s is per-connection in substance. Every part of it answers a question about the container ("is the agent installed", "which version", "where is the workspace info") whose answer cannot change while the container lives. It is re-derived per connection only because each ProxyCommand is a fresh process holding no state. The genuinely per-connection floor is one docker exec plus an ssh handshake: 75-150ms against 1800ms paid.

   2. Does devpod reuse connections? No, and nobody upstream is asking

   No flag for reuse, multiplexing, a control socket or a persistent tunnel in devpod ssh --help at 0.26.1. The two that look promising do nothing, as expected since services start after the tunnel: baseline 2074-2222ms, --start-services=false 2004-2168ms, --agent-forwarding=false 2024-2319ms.

   Which upstream matters here. The pinned devpod is github.com/skevetter/devpod v0.26.1 (read out of the shipped binary's Go build info), a fork of loft-sh/devpod. loft-sh's own latest release is v0.6.15 (2025-03-10), prereleases to v0.7.0-alpha.34 (2025-06-23), last push 2025-11-14. The fork releases on its own line (v0.24.0 to v0.26.1, 2026-05-20 to 2026-06-29). So this repo's version numbers are the fork's, and any 0.26-era question has to be asked of skevetter, not loft-sh. Worth writing down: it is easy to search the wrong repo. (GitHub code search does not index forks, so its source was read from raw refs at tag v0.26.1.)

   Searched both repos for ControlMaster, multiplex, connection reuse, ssh slow/performance/latency, telemetry, inject: no issue or PR proposes or discusses connection reuse. One is relevant for another reason: loft-sh/devpod#1929, see §5.

   3. ControlMaster over devpod's config: incidental, and the config is rewritten

   Incidental. pkg/ssh/config.go::sshConfigBuilder.addSSHOptions emits exactly ForwardAgent yes, LogLevel error, StrictHostKeyChecking no, UserKnownHostsFile /dev/null, HostKeyAlgorithms …, plus ProxyCommand and User. ControlMaster appears nowhere in the v0.26.1 tree. It works because devpod writes an ordinary Host stanza and OpenSSH multiplexes a ProxyCommand transport like any other. Nothing in devpod knows about it, so nothing promises it. That also means it does not need devpod's cooperation, only devpod continuing to write a normal ssh config.

   Who writes it, when. ConfigureSSHConfig targets, in order: context option SSH_CONFIG_INCLUDE_PATH, else context SSH_CONFIG_PATH / devpod up --ssh-config / DEVPOD_SSH_CONFIG, else ~/.ssh/config. It strips the workspace's marker section, re-adds it, and rewrites the whole file with os.WriteFile(path, content, 0o600). Only guard is a process-local sync.Mutex, so two devpod processes writing one config is unprotected.

   Measured:

   	line added inside the markers	Host block outside them
   devpod ssh --command true	survives	survives
   devpod up (no --recreate)	deleted	survives

   Any devpod up, not just recreate. So editing devpod's block is not a route. Two that survive: -o on the ssh command line (which clients/ssh.rs::command_args already does for SendEnv), and a Host *.devpod block outside the markers, which I verified resolves (ssh -G reports controlmaster auto alongside devpod's forwardagent yes), survives devpod up, and gives 9ms reuse. It wins only because devpod's block sets no ControlMaster; a wildcard could never override ForwardAgent.

   And: devpod ssh --command true stays at 2098/2084ms with a live master and a matching wildcard block, because devpod's ssh client is Go and never reads an OpenSSH config. Multiplexing can only help a path that runs OpenSSH.

   The safety fact: it fails closed, four ways

   Measured four ways; no measurement produced a session that silently ran against a dead or wrong container. The mechanism is why: the master's ProxyCommand holds a live docker exec into one container; destroy it and the exec dies, the ProxyCommand dies, the master exits, OpenSSH unlinks its socket.

   (a) --recreate, master idle. ssh -O check -> Control socket connect(...): No such file or directory, rc=255, socket gone.

   (b) 40 back-to-back trips through the master, recreate running concurrently. Each trip read a generation marker in the old container's /tmp and the container hostname:

    1..28  rc=0    GEN-TWO |host=e1fd8e1c2199      <- old container, still alive
    29     rc=255  GEN-TWO                          <- the moment it died
    30..40 rc=0    cat: /tmp/gen.txt: No such file  |host=a38859766f95   <- new container
   

   One trip errors loudly; the next reopens against the new container and reports it correctly, paying the full 1.8s. Failure mode is "one trip errors, the next is correct and slow", never "trips keep succeeding against a ghost". One wrinkle: rc=255 is OpenSSH's transport-failure code and is indistinguishable from a remote command that genuinely exited 255, and trip 29 emitted partial output first. That is inherent to ssh, not made worse by mux.

   (c) docker stop/start, same container id. Master dies and unlinks; next trip 1862ms, reaches the restarted container, reads back the file it wrote before the stop.

   (d) master SIGKILLed, socket orphaned. Next trip logs Stale control socket ..., unlinking / setting up multiplex master socket, costs 1823ms, succeeds; reuse after that 11-12ms. Self-healing.

   Two more, because they would have been hazards: a live master does not block devpod (devpod status 578-640ms, devpod ssh --command 2145/2578ms, devpod up 2101ms, all normal, so the per-workspace lock is released after tunnel setup rather than held for the connection's life); and a plain devpod up on a running container leaves the master alive and still fast (24ms then 12ms).

   Caveat: ControlPath %C hashes local host, remote host, port, user, not the container's identity, so successive generations of a workspace share one socket path. That is fine only because the master reliably dies. It is not defence in depth.

   4. What a reused connection breaks

   --send-env: this contradicts the ticket, and it fails silently

   The ticket says a reused master carries SendEnv, "env still arriving". It does not, unless the master itself was opened declaring that variable. With GH_TOKEN_TEST=hello-389:

   	result
   fresh ssh, -o SendEnv=GH_TOKEN_TEST	GOT=[hello-389]
   fresh ssh, -t -o SendEnv=...	GOT=[hello-389]
   devpod ssh --send-env GH_TOKEN_TEST --command ...	GOT=[hello-389]
   reuse, master opened without SendEnv, client passes it	GOT=[]
   reuse, master opened with it, client passes it	GOT=[hello-389]
   reuse, master opened with GH_TOKEN_TEST, client passes SendEnv=OTHER_VAR	OTHER=[]

   Both ends must permit the name: the client puts its environment in the mux request, and the master filters it against the master's own SendEnv list before emitting the env channel requests. No warning, no "Sending environment" line on the mux path, exit status 0.

   Manageable for dl, whose one variable is GH_TOKEN: open the master with -o SendEnv=GH_TOKEN. One further measurement says the values stay fresh even when the permit list is stale, a master opened with GH_TOKEN_TEST=hello-389 delivered ROTATED-VALUE to a later reuse. But the failure mode if this is ever got wrong is the worst available: a workspace comes up with an empty GH_TOKEN, silently, and gh inside it is simply unauthenticated. Any multiplexing design needs the master's SendEnv list pinned by a test, not by a comment. (devpod's Go server has no AcceptEnv allowlist to worry about; arbitrary names arrive on a fresh connection.)

   ForwardAgent yes: works, and pins the agent to the master

   Survives reuse (ssh-add -l in the container listed the host key over both fresh and reused). But it pins to whichever agent opened the master, for the master's whole life: reuse from a client pointing at a second agent with a different key still saw the first agent's key, and reuse from a client with no SSH_AUTH_SOCK at all still got a forwarded socket with the first agent's key. A ControlPersist window is a window in which every session into that workspace reaches an agent the current caller may not have, and may not want forwarded.

   The remote exit status: a reused ssh does not break it, it removes the need for the machinery

   Confirmed both sides. Observed:

   $ devpod ssh res389 --command "exit 3"     # devpod exits 1
   fatal  tunnel to container: run in container: ssh session: Process exited with status 3
   

   That sentence is exactly devpod.rs's REMOTE_EXIT_MARKER and the fatal tag is FATAL_TAG, so the shipped recovery is reading this. Why it is needed is now confirmed at source: cmd/root.go::Execute does try to propagate, if sshExitErr, ok := err.(*ssh.ExitError); ok { os.Exit(sshExitErr.ExitStatus()) }, an unwrapped type assertion against an error wrapped twice (tunnel to container: then run in container:), so it fails and devpod exits 1. The Try using the --debug flag line StderrFilter holds back is emitted a few lines below in the same function.

   OpenSSH has none of this. Remote status becomes ssh's own status, identically with and without multiplexing:

   requested	fresh ssh	reused master	devpod ssh
   0	0	0	0
   3	3	3	1 + stderr sentence
   7	-	7	-
   130	130	130	1 + stderr sentence

   So a direct ssh path needs neither StderrFilter nor interpret, which is what clients/ssh.rs's own doc comment already claims; this confirms it and extends it to the multiplexed case. The cost of the simplification is the 255 ambiguity in §3(d): devpod's stderr protocol, for all its awkwardness, distinguishes "transport failed" from "exited 255".

   5. Not asked for, and possibly the biggest thing here: concurrency

   devpod ssh --stdio takes a per-workspace lock (acquired workspace lock, workspace_client.go:284/:289), so non-multiplexed trips to one workspace do not merely cost 1.8s each, they cost 1.8s each in series. Eight concurrent ssh runs each doing sleep 8, no mux:

   9861ms  11852ms  13857ms  16115ms  17930ms  19852ms  21864ms  23853ms
   

   A 2s staircase, and devpod says why on stderr: Trying to lock workspace, seems like another process is running that blocks this workspace (machine_client.go:311). The same eight over one master: all 8019-8026ms. Perfectly parallel.

   Same shape as loft-sh/devpod#1929 ("only 4 can be established, the 5th hangs"), which at 0.26.1 presents as serialization rather than failure. Multiplexing is a mitigation: N logical sessions collapse to one devpod ssh --stdio, so the lock is taken once.

   Reference (quiet machine)

   	ms
   devpod ssh <ws> --command true	2014-2363
   raw ssh -F <devpod config> <ws>.devpod true, fresh	1686-1980
   same, -t (pty)	1950-2061
   same, reuse over ControlMaster/ControlPersist	9-12
   same, reuse with -t	10 (17-22 loaded)
   docker exec <cid> bash -lc true	125-140
   docker exec <cid> true	65-74
   devpod status <ws>	552-841
   devpod --help	46-55
   in-container agent devpod version / telemetry off	479-501 / 99-111

   Reuse is 150-200x cheaper than a fresh trip; docker exec is 13-27x cheaper. Both figures in the ticket's table reproduce.

   What this does NOT settle

   • What dl should do. That is The trip floor, not the trip count: how dl reaches into a running container #390, deliberately. Nothing here argues for multiplexing, for docker exec, or for leaving the trip alone.
   • Host numbers. All nested DinD, part of it on a contended machine. The decomposition and ratios transfer; the absolute seconds are not the map's baseline (that is Re-baseline launch latency on the Rust build #388/The launch trend has published nothing since the Rust cutover: the benched dl cannot see devpod #292).
   • Whether the telemetry cost is reachable. ~0.78s of the trip is telemetry inside two in-container agent invocations, measured. Neither DEVPOD_DISABLE_TELEMETRY on the host process (saves ~350ms, one host-side flush: 2074-2222ms -> 1716-1980ms) nor context TELEMETRY=false propagates into those invocations, and one guess at an in-container config file did not either. It might be reachable by an upstream change to buildExistsCheck, by dropping the redundant second version exec, or by an env var the docker driver already forwards. Untested, and the most valuable open thread here: a ~40% cut to the trip that asks nothing of dl's architecture.
   • Whether fail-closed holds on non-docker providers. Measured only on docker, and the mechanism is the ProxyCommand's docker exec dying with the container. A ssh or kubernetes provider's ProxyCommand dies for different reasons, or possibly does not.
   • Long ControlPersist windows. Everything used 600s and reused within seconds. Nothing measured about a master idle for minutes, about devpod's --ssh-keepalive-interval 55s interacting with an idle master, or about a laptop suspending under one.
   • The 4-session ceiling. Reproduced serialization at 8 concurrent sessions on 0.26.1, not #1929's hard hang. Whether the ceiling exists at some higher count, or was fixed on the fork's line, was not established.
   • ForwardAgent policy. That a reused master pins forwarding to the first caller's agent is measured. Whether that is acceptable is a judgement about dl's users.
   • Noticed, not investigated, not this ticket's question: clients/ssh.rs::config_path() hardcodes ~/.ssh/config, while devpod 0.26.1 writes to DEVPOD_SSH_CONFIG / SSH_CONFIG_PATH / SSH_CONFIG_INCLUDE_PATH when any is set, and writes only there. Under DEVPOD_SSH_CONFIG (what a scratch-scoped run uses) ~/.ssh/config was never created at all, so devpod_host_configured answers false and dl silently keeps every command on the devpod transport, losing the pty path.

   Branch: research/devpod-ssh-trip (docs/research/devpod-ssh-trip.md).
8. blooop

   closed this as completedon Aug 24, 2026
9. blooop

   added 4 commits that reference this issue on Aug 24, 2026

   The trend is not dark, and the chart's first 35 points are the real gap

   ...

   9edfea4

   feat: one reusable OpenSSH connection per workspace

   ...

   d568dad

   feat: one reusable OpenSSH connection per workspace

   ...

   85f79ee

   feat: one reusable OpenSSH connection per workspace

   ...

   18edab0