Repository navigation
One authenticated connection per compute machine per session, kept by the gateway (log in once) #487
Description
Today every part of ECCE that talks to a compute machine opens its own ssh connection and authenticates: the Launcher, the job store (file transfer) and each job's monitoring. On a password-only cluster (e.g. HPC2N Kebnekaise) the user is asked at every submit, and monitoring of long jobs can need new logins.
Proposal:
- The gateway (which owns the session) keeps one authenticated libssh session per compute machine for the ECCE session; the first use asks for credentials once (password, two-factor code or key passphrase).
- Launcher, job store and monitors send commands and file transfers through the gateway's connection instead of logging in themselves.
- Reconnect on network loss without asking unless the cluster demands it; close at session end. Credentials only in memory, never on disk (as ssh-agent).
- Works on Linux, macOS and Windows (OpenSSH ControlMaster is not available on Windows, and ECCE uses libssh).
- Side benefit: no handshake per command.
Care: this is the remote-command layer. Start with a root-cause style investigation of how the processes connect today; all three deployment modes (local, central server, shared broker) must keep working; tests in tests/launch and tests/containers. Related: #245 (key login setup), #204 (transport follow-ups). After 9.4.0.
Metadata
Metadata
Assignees
Labels
No labels
Activity
ohlincha commentedon Oct 23, 2026
Observed 2026-10-23: HPC2N (Kebnekaise) did not accept key login for the maintainer's account, so key setup (#245) does not help there; one login per session (#487) is the fix for such clusters.
ohlincha commentedon Oct 23, 2026
Raised in priority (maintainer, 2026-10-23): submitting a set of jobs to a password-only cluster asks for the password at every submit. Interim: the OpenSSH transport already reuses a connection the user's ~/.ssh/config shares (ControlMaster auto, ControlPersist), see the wiki page 'Connecting to a cluster that requires 2FA'. Stage 1 proposal: on Linux and macOS ECCE owns the shared connection for every remote machine itself (as it already does for proxied hosts, OpenSshTransport::ownsControl), so the first login of a session is the only one, with no ssh configuration by the user; stage 2: the gateway-held libssh session for Windows, where OpenSSH has no ControlMaster. Next after #491.
ohlincha commentedon Oct 23, 2026
Step 0 (maintainer, 2026-10-23: 'the old ECCE used to remember the password'): the session password cache is still in the code and is meant to do exactly this. WxLauncher::refreshPasswords pre-fills the password from AuthCache; RCommand::setPassCache adds a successful login to AuthCache 'so it is available to other apps'; sshConnectLibssh tries the given password, then AuthCache, then the dialog; the gateway distributes the cache (AuthCache::msgIn, sessionSave/sessionLoad, pipes to tools; preference Gateway.SavePasswords). Yet a password-only cluster asks at every submit. Find which link is broken (cache never filled, gateway never relays it over the Mosquitto broker since #213, lookup key differs: URL shell://machine vs full host name, user) and fix it first; that may give one login per session on all platforms before stage 1.
Merge wip/487-authcache: a compute machine's password is relayed agai…
ohlincha commentedon Oct 23, 2026
Step 0 merged to main (a76881e): root cause fb9a3ef (#213 stage 2) removed the password from ecce_auth_changed and made AuthCache::msgIn read it from the session store, which by design holds only data-server (http) entries; an ssh:// password therefore never reached the gateway or the next Launcher, and every submit asked again. Now the message carries a compute machine's password only on the account's own Unix-socket broker (local mode; same trust boundary as the -pipe FIFO); never on a TCP broker (central server, shared broker, Windows), never for data-server passwords, nothing on disk. Test authcache-relay (fails before, passes after). Still open: central-server/shared-broker clients and Windows (stage 1 ECCE-owned ControlMaster, stage 2 gateway-held connection); a live check with two submits to a password-only cluster.
ohlincha commentedon Oct 23, 2026
Stage 1 analysis (code read, nothing run yet). Decisions taken: MaxSessions overflow masters are in stage 1 (Launch Selected submits sets of jobs); ControlPersist 8 h with exit at session end; machines whose front end needs nested ssh stay on libssh. Implementation in two parts: A transport + unit tests, B session wiring (ecce.in, reaper, gateway session end) + sshd/central-server tests.
#487 stage 1: an ECCE-owned OpenSSH master for every compute machine (Linux/macOS)
Answer: yes, and the change sits at one choke point. No perl, python or shell script opens ssh to a compute machine; every connection goes through
RCommand->sshConnect(src/comm/rcommand/RCommand.C:460) ->useOpenssh(:446) ->OpenSshTransport::backendFor(OpenSshTransport.C:218). The only place ssh is spawned outside RCommand is the terminal (EcceShell.C:288), and it already passescontrolArgs(). Confidence: high for the map (I read the code; I did not run it). Medium for the session-end and MaxSessions behaviour (taken from the OpenSSH documentation, not tested).How each process connects today (backend AUTO, no ~/.ssh/config sharing)
putOnLogin)copyOnLogin), started by eccejobmaster; a separate process that outlives the session-admin -remoteOpenSSH is used only when
ssh -Greports that the user's config shares the connection (ControlMaster or ControlPath set) or proxies it (ProxyJump or ProxyCommand), or whenECCE_SSH_BACKEND=opensshis set. ECCE owns the master only for a proxied host whose config has no sharing of its own (ownsControl, :351). That master's socket is~/.ECCE/cm/%Cwith ControlPersist=10m (:333-348). It is not per session and nothing removes it. The first login runs asssh -f -NwithSSH_ASKPASS=scripts/ecce-askpassandSSH_ASKPASS_REQUIRE=force(openMaster, :403). That is already a pty-less path that handles a password, a second factor and an unknown host key. Only one thread of a process opens the master at a time (gMasterMu), but nothing stops two processes from doing it at once. In every deployment mode, ssh to a compute machine starts on the client machine, where the Launcher, eccejobmaster/eccejobstore and the Organizer run (waiting-for-login.md: "of the client (in-remotemode too)"). The data server and the broker take no part.Stage 1 design
_WIN32, withECCE_SSH_CONTROL_DIRset andsshpresent,backendForreturns OPENSSH for every host.ownsControl()becomescontrolDir set && !configSharesConnection(...), and thep_jumpHost.empty()condition goes (the jump host is part of the socket key, below).ECCE_SSH_BACKEND=libsshremains the way out. Exception: a machine whose front end needs nested ssh (SshTransport::nested(), libssh only;OpenSshTransport::nested()always returns false) stays on libssh.-Jneeds TCP forwarding on the front end, and that is exactly what a nested front end does not allow.packaging/ecce.in:211creates the directory next toECCE_SESSION_IDand exportsECCE_SSH_CONTROL_DIR, so every child process (eccejobmaster too) sees the same one. The directory is$XDG_RUNTIME_DIR/ecce-ssh/<sessionkey>on Linux: local tmpfs, 0700, short, and it works when home is on NFS, AFS or CIFS, where a socket cannot be bound. On macOS it is$TMPDIR/ecce-ssh/<key>(a per-user 0700 directory of about 49 bytes). The fallback is$ECCE_REALUSERHOME/.ECCE/cm/<key>. The C++ side refuses the directory unlesslstatshows it owned by us with mode 0700 and not a symlink (today'smkdirdoes not check an existing directory). The file name is our own 16-hex hash, not%C, computed from local host, user, host, port and jump host. The reason is that ssh binds to<path>.<16 random chars>first and renames it afterwards, so the real need is len(path)+17 < 104 on macOS. With%C(40 characters) and a 16-character session level, a home path over about 20 bytes already fails on macOS. With the short hash,$TMPDIRfits with room to spare. ssh creates the socket 0600 (umask 0177).ECCE_SSH_PERSIST). This is an idle bound only: running monitor streams keep the master alive anyway.openMasteralso getsServerAliveInterval(today onlysshArgsadds it, not the master).ecce-gateway-reap§1 (lines 125-147) already sweeps per-session files for sessions with no live ECCE process, and a live eccejobstore counts as one. It is extended so that for each socket of a dead session it runsssh -O exit -S <sock> xand removes the directory. When the gateway ends a session (GatewayApp.C:887) while jobstores of that session still run, it sends-O stop(no new clients; running streams continue) and the reaper finishes the job later. A jobstore that restarts after-O stoplogs in on its own. With no display it gets no askpass and parks the job as Waiting, as it does today.flockon<dir>/<hash>.lockaroundopenMaster. The process that gets the lock second seesmasterAlive()and does not prompt. Without the lock the user gets N password dialogs.<hash>-1,-2, ... That costs one login per roughly 9 jobs, which is still far better than one per job. This must be in stage 1 if Andy submits sets of jobs.sshConnectdrops it), so the user would be asked again in the askpass dialog. Pass it through a FIFO inopenMaster's mkdtemp 0700 state directory. A writer thread writes it once, and ecce-askpass reads it only for apasswordprompt and only on the first try; a second factor still goes to the dialog. It is never put in the environment, argv or a file. ssh closes inherited file descriptors, so a pipe fd cannot be used.configSharesConnectionis true and ECCE uses the user's setting as it does now. One limitation:ssh -Gprints the same thing for an explicitControlMaster noas for the default, so such a user would get ECCE's master. That is acceptable, and documented together with the libssh override.Deployment modes
-remote): the client keeps the master in its own runtime directory. This is the case step 0 deliberately left without relay (TCP broker), and stage 1 is what fixes it there. Server side: no change._WIN32gate (stage 2).Security (flag: SSH and process launch)
A control socket is a logged-in shell on the cluster that needs no password. Anyone who can connect to it (the same uid, or root) can run commands as the user. That is the same trust level as
~/.ssh/id_*or ptrace, but it adds time: an abandoned session leaves an open door for up to ControlPersist. On a lab machine with a shared Unix account the next user inherits it unless the reaper has run. Mitigations: §2 (strict 0700/owner check, XDG tmpfs cleared at logout), §4 (exit at session end), and a finite persist. The password never touches disk, argv or the environment (§7). Host-key acceptance stays an explicit dialog (ecce-askpassconfirm). Data-server and broker authentication (#138, #194) are not touched.Other risks
macOS before OpenSSH 8.4 (Catalina) ignores
SSH_ASKPASS_REQUIRE: set a dummyDISPLAYand run without a controlling tty there. Monterey (8.6) is fine. One dropped master kills every monitor at once; they park and restart, and §5 keeps that to one prompt. A site rule that kills idle connections only costs a new login. Changing the default backend means every OpenSSH-path difference (error texts,hop(), streams) now applies to everyone: the full sshd suite has to pass with the new default.Test plan
tests/transport/sshd/{controlmaster,proxyjump,monitor_login,tail,rcommand,frontend}_test.sh, ctesttestOpenSshTransport,testControlMasterLoss,testRCommandSsh,tests/launch/run_tests.py,tests/apps/session_end.py(all three modes).owned_master_test.sh, built on monitor_login_test.sh (the sshd log counts password logins): an sshd with password login only andMaxSessions 3, isolated HOME and ECCE_REALUSERHOME, askpass stub. Check: (a) 5 jobs launched = exactly 2 logins (one master plus one overflow); (b) killing the master, with 5 parked jobstores restarting, gives 1 new prompt; (c) afterecce-gateway-reap, no socket and no ssh master process is left.tests/containers/central_server_test.pyruns no compute ssh today. Add one job submitted from the client container to an sshd container and count the logins there. Two physical machines stay untested.Size
About 60-70 agent tool calls in two agents: (A) transport changes §1-3 and §5-7 plus unit tests, about 35; (B)
ecce.in, the reaper and gateway session end,owned_master_test.sh, the central-server case and the docs/claude entry, about 30.Merge wip/487-stage1: on Linux and macOS ECCE owns one OpenSSH master…
ohlincha commentedon Oct 24, 2026
Stage 1 merged to main 92df925 (9.4.0): on Linux and macOS each session owns one OpenSSH master per compute machine (socket in $XDG_RUNTIME_DIR/ecce-ssh/ or $TMPDIR on macOS, 0700, ControlPersist 8 h, -O stop at gateway session end, -O exit + removal by ecce-gateway-reap for dead sessions); overflow masters when sshd MaxSessions refuses; front ends that refuse forwarding fall back to libssh; a known password is passed to askpass once through a FIFO. Tests: owned_master_test.sh against a password-only sshd with MaxSessions 3 (5 streams = 2 logins; master killed = 1 new login with 4 concurrent openers; known password = 0 dialogs; nothing left after the reaper), controlmaster/proxyjump/tail/frontend/rcommand/monitor_login sshd tests, apps suite 72/72 incl. session_end. Not yet checked: a real cluster (Kebnekaise: two submits, one password), the central-server container case, Windows (stage 2, unchanged). Opt out: ECCE_SSH_BACKEND=libssh.
Owned ssh master: the password dialog's OK no longer counts as a canc…
Merge wip/487-fix: the installed password dialog exited 255 after OK,…
ohlincha commentedon Oct 24, 2026
Checked by the maintainer 2026-10-24 (Linux, Kebnekaise): after the first submit of the session no further password prompts. Windows (stage 2) still open.
ssh control directories: each records the install that made it, and t…
ohlincha commentedon Oct 25, 2026
Stage 1 done and checked (Linux and macOS, Kebnekaise, 2026-10-24): one login per compute machine per session. Windows (stage 2) continues in a separate issue.