Local bash connections: two distinct, real bugs (SIGHUP hangup + monitoring never completes) -- csh forced on as a stopgap #69

Tracking issue for two separate, confirmed-real bugs that only affect bash as the local/compute-machine shell (RefMachine::shell(), defaulting to "bash" for niobium/localhost/127 -- csh/tcsh does not hit either). As of 2026-09-03 the Launcher's "Use csh/tcsh" checkbox has been forced on and disabled (can't be unchecked) as a stopgap until these are root-caused -- see src/apps/launcher/WxLauncherGUI.C/WxLauncher.C.

Bug 1: "Error: hangup" -- Gaussian (and presumably other codes) killed by a real SIGHUP mid-computation

Confirmed via multiple real coredumps (g16 SIGSEGV + a link executable SIGABRT, register dump matching Gaussian's own signal handler) that a genuine SIGHUP reaches the compute process while it's actively running, under bash, intermittently (2 of 3 jobs in one batch; not reproducible via any standalone simulation of RCommand's own connection-teardown sequence run so far -- including a real test linked against the actual compiled RCommand class, with realistic timing delays before connection teardown).

An existing nohup fix (commit 0f0d99e) for RCommand::execbg() is real and necessary but evidently not sufficient on its own. A follow-up attempt to also stop bash from loading ~/.bashrc for this connection (--norc --noprofile instead of csh-style -f, commit be43e9c) was reverted (288f7d7) after it coincided with a much worse regression (all job submission getting stuck) -- possibly unrelated coincidental timing, possibly a real second bug that fix introduced; not resolved either way.

Not yet root-caused. Next step if picked back up: a live packet/strace capture of an actual failing submission (a temporary strace-wrapped Shell|submitCommand was tried and reverted after it broke submission outright, commit fb0449e/f20d60b -- the wrapper was never verified against the real compiled eccejobstore/Launch.C pipeline before being deployed, only a hand-written simulation, which is exactly what should happen before trying again).

Bug 2: job monitoring gets stuck at "submitted" forever, though the compute job itself completes fine

100% reproducible under bash (both Gaussian and NWChem, both niobium and localhost machine registrations) -- not intermittent, unlike Bug 1. Root-caused via live gdb -p <pid> attachment to a genuinely stuck eccejobstore process (still running, never crashes, never exits -- just hangs forever):

• Stack: main -> jobstore_main -> initMon() -> RCommand::expect1() -> blocked in a real, literal read() syscall with infinite timeout (JobStore.C's stdio-comms branch: remoteconn->exptimeout(-1)).
• The command actually run (perl eccejobmonitor ...) completes successfully -- confirmed via eccejobmonitor's own real, substantial protocol output already sitting in eccejobstore's own read buffer (jmSTATUS/jmPROP/jmDONE, a complete session).
• initMon() is waiting for an exact, literal byte-for-byte match of the command it sent, re-echoed by the shell, before it will proceed. A raw hex dump of the live buffer shows the front of that echo is not intact: 0x0d 0x3c 0x37 0x39 0x30 0x34 0x34 ... = \r<79044... where the real text should have continued from ...-jobId 10.
• 0x3c is a literal < character, not a stray escape byte -- this is GNU readline's horizontal-scroll indicator: for a command line longer than the (unset/default) terminal width, readline substitutes < for the actual character at that screen position to indicate "more text scrolled off to the left." The exact-match pattern eccejobstore waits for can therefore never appear intact in the byte stream -- not a race, not a buffer-size issue, a structural consequence of readline rendering rather than passing input through.
• csh has no line editor at all -- it echoes raw input verbatim, unconditionally -- which is why this is bash-specific and 100% reproducible there (any sufficiently long command triggers it) while never happening under csh.

Not yet fixed. Most promising direction, not yet attempted or verified: force a wide terminal (stty cols <big number> or COLUMNS env var) immediately after connecting, so readline never believes it needs to horizontal-scroll. Should be verified live-attached to a real stuck process (as this diagnosis was) before being shipped, not just simulated -- see Bug 1's cautionary tale above about deploying an unverified fix to this exact connection-handling code.

Related, narrower existing issues

#59 (eliminate csh/tcsh dependency, local shell flag translation), #60 (execbg() "Exit*" pattern assumes csh), #61 (connection visibility design) are all still open and relevant background, but neither fully covers either bug above -- both were found and root-caused today with much more specific evidence than existed when those were filed.