Repository navigation
Re-login after the shared connection drops asks once per job (six prompts for six jobs) #599
Description
9.4.0 (Andy 10-26, Linux, KebnekaiseX, password login): six jobs running, one login at the start (the shared OpenSSH master of #487 works). Some hours later the connection had gone and ECCE asked for the password again, once per job: six dialogs, apparently queued one after another.
Wanted: when the master for a machine is gone, re-establish it once: the first job action that needs it prompts, every other job for that machine waits for that login and reuses it; a cancelled prompt cancels the waiting ones with one message. Also find why the master dropped after a few hours (ControlPersist time, no ServerAliveInterval, the cluster dropping idle connections, or the gateway's reaper) and keep it alive for the session where the server allows.
Remote-shell/launch code: root-cause analysis before any fix. Related: #487 (stage 1), #546 (Windows).
Activity
ohlincha commentedon Oct 27, 2026
Cause (from the 10-26 job logs): the VPN disconnect at 14:59 took both ssh masters down at 15:01 (two ECCE sessions had jobs on the machine, each with its own master); every job store restarted at once. Per session the dialogs multiplied three ways: the first login dialog went unanswered for 120 s, and the timeout made that job fall back to the built-in ssh client with a second dialog of its own; the failure was remembered per process only, so the next waiting job asked again; and processes on an overflow master (-1) locked a different file. After the fallback a false "login was refused" line was logged although the job connected.
Fix in progress (wip/599-one-login): one lock per machine, the outcome of a login (ok / cancelled / not answered) shared by all job stores of the session, no fallback to the built-in client on a timeout or cancel, and the jobs parked as Waiting with one message. Masters stay per session: two sessions with jobs on one machine still need one login each. A VPN drop cannot be ridden through; ServerAlive turns it into a clean restart.
Merge #599: one login per machine after a dropped ssh master; no libs…
ohlincha commentedon Oct 27, 2026
Fixed in 59e8a83 (main, for 9.4.2). After the shared ssh master drops, the job stores of a session share one login: one lock per machine, the outcome (logged in / cancelled / not answered) is shared, so the other jobs neither ask again nor fall back to the built-in ssh client with a dialog of their own. A cancelled or unanswered login parks the jobs as Waiting with one message; Reconnect then asks once. The false "login was refused" line after a fallback is gone. Each ECCE session keeps its own master, so jobs from two sessions on one machine still need one login per session. Tested against a password-only sshd: 4–5 processes losing the master, answered, slow, cancelled, and both masters (main + overflow) killed: one dialog each time.
Reconnect asks for the login again right after a cancel