Documented torchrun/accelerate/deepspeed multi-GPU launch collapses ForgeLM exit codes 2/3/4/5/6 into 1 and SIGTERMs surviving ranks (CR-A01-020) #400

Wave 1 · theme D1-5: Training, merging and export correctness · highest severity High · 2 finding(s) · effort medium

Summary

A CI job follows docs/reference/distributed_training.md and runs 'torchrun --nproc_per_node=4 -m forgelm.cli --config job.yaml' with evaluation.require_human_approval: true. Training succeeds and the model is staged in final_model.staging.<run_id>, and rank 0 exits 4. torchrun reports ChildFailedError and exits 1. The pipeline's 'case $? in 4) request approval;; 1) fail: fix config;;' takes the config-error branch, so no approval request is raised and the staged model is never reviewed. In the same way, a safety/benchmark gate failure (3) with auto_revert or a runtime crash (2) is reported as a config typo.

Primary finding

CR-A01-020 — Documented torchrun/accelerate/deepspeed multi-GPU launch collapses ForgeLM exit codes 2/3/4/5/6 into 1 and SIGTERMs surviving ranks

severity High · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end

What happens. cli/_training.py:159-161 (and 173-180) exits each rank with ForgeLM's own code (4 awaiting approval, 3 gate failure, 2 training error). Under the launchers the docs prescribe (torchrun/accelerate launch/deepspeed -m forgelm.cli, distributed_training.md:25-45, enterprise_deployment.md:53-62,117-142, usage.md, config_template.yaml:164), the parent launcher turns any non-zero child exit into ChildFailedError/CalledProcessError and exits 1, then SIGTERMs the other ranks mid audit/webhook sequence. The clamp in cli/_exit_codes.py cannot help because the rewrite happens in the parent; no doc warns.

Where.

• docs/reference/distributed_training.md:25-33
• docs/reference/distributed_training.md:32
• docs/reference/distributed_training.md:45
• docs/guides/enterprise_deployment.md:53-62 (confirmed)
• docs/guides/enterprise_deployment.md:141
• docs/reference/usage.md:388-389
• docs/usermanuals/en/training/distributed.md:78-82
• docs/usermanuals/en/training/distributed.md:82
• config_template.yaml:164 (the 'torchrun ... forgelm --config' form is invalid, a separate doc bug)
• forgelm/cli/_training.py:163-180

Evidence.

• plain.log: 'fpy -m forgelm --config approval.yaml' gives "awaiting_approval": true and EXIT=4. accel2.log: unmodified forgelm.cli under 'accelerate.commands.launch --num_processes=1' prints "awaiting_approval": true, then "subprocess.CalledProcessError: Command '[... '-m', 'forgelm.cli', '--config', '../approval_acc.yaml', '--output-format', 'json']' returned non-zero exit status 4." and EXIT=1. torchrun4.log: 'torch.distributed.run --nproc_per_node=1 -m cpu_forgelm' (an MPS-hiding harness that…
• ddp_approval: both children exit 4, runner EXIT=1. ddp_revert: children exit 3 and -15 (SIGTERM), EXIT=1. ddp_div_r1: children exit 3 and 2, EXIT=1. ddp_crash_mps: rank1 exit 2, rank0 SIGTERM; audit holds 2x pipeline.initialized but only 1x pipeline.failed, and the webhooks are 2x training.start plus 1x training.failure. ddp_pipe2_gate: rank1 exit 4, rank0 -15 SIGTERM.
• Launcher exit and per-rank exit files are in tmp/.../a1-W70/ddp/run0-mps-barrier.log (rank1 2 -> EXIT=1), diverge/ (0,3 -> 1), approve/ (4 -> 1), bothfail/ (3 -> 1) and ddp/run1.log (0,0 -> 0). The torch source is cited from venvs/extras torch 2.12.1.

Recommended fix. State in the distributed docs and README exit-code section that launchers (torchrun/accelerate/deepspeed) replace ForgeLM's exit code with 1 on any non-zero worker exit. Give CI a launcher-independent signal: have rank 0 write the public exit code (and the JSON envelope) to a well-known file in output_dir, for example last_exit.json, and document reading it, or provide a 'forgelm' wrapper that spawns the launcher itself and re-maps the code. Fix config_template.yaml:164 to '-m forgelm.cli'. Separately, catch BrokenPipeError when flushing stdout at exit, so success paths cannot leak exit 120.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

CR-A05-015 — The documented multi-GPU launchers turn ForgeLM's exit codes 3 (eval failure), 4 (awaiting approval) and 6 into 1 (config error)

severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end

What happens. The documented multi-GPU entry is torchrun ... -m forgelm.cli or accelerate launch ... -m forgelm.cli (docs/reference/distributed_training.md:25-33, 45, 173-191, 200). Both launchers report any non-zero child exit as 1: torchrun raises ChildFailedError, and accelerate raises CalledProcessError. The public contract (CLAUDE.md principle 4, error-handling.md) defines 1 as config error, and CI is told to branch on 3/4 and alarm on 6. Under the documented launch, an auto-reverted run or one awaiting approval reaches CI as 'config error'. The docs give no caveat or wrapper.

Failure scenario. A CI pipeline waits for human approval when ForgeLM exits 4. Under the documented 4-GPU launch it receives 1 and fails the job as a config error, so the approval step never runs; equally, an exit-6 alarm wired to the run never fires.

Where.

• docs/reference/distributed_training.md:45
• docs/reference/distributed_training.md:32
• docs/reference/usage.md:389
• forgelm/cli/_exit_codes.py:1
• docs/reference/distributed_training.md:169-200

Evidence.

• ddp/run3.log and run_appr: both ranks 'exitcode: 4', torchrun EXIT=1. run_rev.log: both ranks exit 3, torchrun EXIT=1. ddp/exit4.py (sys.exit(4)): torchrun EXIT=1; accelerate.commands.launch --num_processes 1 --cpu exit4.py EXIT=1 (CalledProcessError returned non-zero exit status 4).

Recommended fix. Document the collapse and give a contract-preserving pattern: rank 0 writes its exit code or a result JSON to a known file and CI reads that, or ship a thin launcher wrapper that propagates rank 0's code.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}