Timed-out local nodes leave the node's spawned processes running as orphans

[JoeyTrribbiani]

Summary

When a local node exceeds timeout_seconds, the runner reports the node as timed out (exit code 124) and the run finishes — but only the node's direct child process is signaled. Any processes that the node spawned (helper workers, dev servers, subshells — or, for real agent CLIs, the MCP servers and plugins they start) survive the timeout, are re-parented to PID 1, and keep running indefinitely after agentflow itself has exited. Nothing is left that will ever reap them.

The runner already acknowledges that node processes have children — the comment above the monitoring loop in agentflow/runners/local.py:337-339 reads:

Key insight: claude spawns child processes (MCP servers, plugins) that inherit stdout/stderr pipes. When claude exits, those children keep the pipes open — so we CANNOT rely on stream EOF to detect completion.

That reality is handled for stream draining, but the timeout/termination path makes no attempt to reclaim those children.

Steps to reproduce

Self-contained script (POSIX; needs bash, python3, git, pip):

#!/usr/bin/env bash
# Repro: a timed-out local node leaves its spawned child processes alive
# (node reports exit code 124; the child survives, orphaned with PPID 1).
set -euo pipefail

REPRO=/tmp/agentflow-repro/issue1-orphaned-children
AF=/tmp/agentflow-repro/.venv/bin/agentflow

mkdir -p "$REPRO"

# --- one-time setup (skip if the CLI is already installed) ---
if [ ! -x "$AF" ]; then
  git clone https://github.com/agentenv/agentflow /tmp/agentflow-repro/agentflow-src
  git -C /tmp/agentflow-repro/agentflow-src checkout 09df017  # commit tested with
  python3 -m venv /tmp/agentflow-repro/.venv
  /tmp/agentflow-repro/.venv/bin/pip install -e /tmp/agentflow-repro/agentflow-src
fi

# --- mock agent executable: spawns one long-lived helper, then idles ---
# (stands in for any real agent that starts servers/workers as children)
cat > "$REPRO/orphan-spawning-agent.py" <<'EOF'
#!/usr/bin/env python3
import subprocess
import time

subprocess.Popen(["sleep", "617"])  # distinctive duration => unambiguous pgrep match
time.sleep(60)
EOF
chmod +x "$REPRO/orphan-spawning-agent.py"

cat > "$REPRO/pipeline.yaml" <<EOF
name: orphan-repro
working_dir: .
nodes:
  - id: spawns_helper
    agent: codex
    executable: $REPRO/orphan-spawning-agent.py
    prompt: work
    timeout_seconds: 5
EOF

cd "$REPRO"
rm -rf .agentflow

echo "== agentflow run (node times out after 5s) =="
set +e
"$AF" run pipeline.yaml > run.log 2>&1
RUN_EXIT=$?
set -e
tail -20 run.log
echo "agentflow exit code: $RUN_EXIT"

echo
echo "== helper process after agentflow exited =="
HELPER_PID="$(pgrep -f 'sleep 617' | head -1 || true)"
if [ -n "$HELPER_PID" ]; then
  ps -o pid,ppid,comm -p "$HELPER_PID"
  echo "BUG: helper still alive (PPID 1 = orphaned)"
  kill "$HELPER_PID"
else
  echo "OK: no helper left behind"
fi

The mock agent stands in for any agent executable that starts helper processes: it spawns one long-lived child (sleep 617), then idles past the 5-second node timeout.

Observed (macOS, agentflow 0.1.0 @ 09df017; output abridged)

== agentflow run (node times out after 5s) ==
          "number": 1,
          "status": "failed",
          ...
          "exit_code": 124,
          ...
agentflow exit code: 1

== helper process after agentflow exited ==
  PID  PPID COMM
21677     1 sleep
BUG: helper still alive (PPID 1 = orphaned)

The run record says the node timed out (exit_code: 124) and failed, agentflow has exited — and the sleep process the node spawned is still alive, re-parented to PID 1.

Expected

Everything the node spawned should be terminated when the node times out (or, at minimum, there should be a supported way to opt into that behavior).

Root cause (at 09df017)

1. agentflow/runners/local.py:303-310 — node processes are launched with plain asyncio.create_subprocess_exec(...); there is no start_new_session / process-group separation (a search for start_new_session, killpg, setsid across the package finds nothing), so each node runs in the orchestrator's own process group.
2. agentflow/runners/local.py:244-250 — _terminate_with_fallback() escalates process.terminate() → 1 s grace (_TERMINATE_GRACE_SECONDS, line 35) → process.kill(), but both signals target the direct child PID only.
3. agentflow/runners/local.py:400-404 — the timeout path calls only _terminate_with_fallback(), so grandchildren are outside the kill radius.

Possible fix

On POSIX, give each node its own process group at spawn time and signal the whole group on timeout:

1. pass start_new_session=True to create_subprocess_exec in execute();
2. in _terminate_with_fallback(), escalate group-wide: os.killpg(os.getpgid(process.pid), signal.SIGTERM) → grace period → os.killpg(..., signal.SIGKILL), falling back to the current PID-only path when the group is already gone;
3. keep the current direct-child-only behavior on Windows, where os.killpg does not exist.

This would not catch a grandchild that deliberately calls setsid() on itself, but it covers the common case of helpers inheriting the node's process group.

Environment

• agentflow 0.1.0, commit 09df017
• macOS 15, arm64, Python 3.14
• Plain POSIX signal semantics — I expect identical behavior on Linux (not separately verified there)

---