Skip to content

Fleet dispatch hardening: retire raw-nohup hunts (timeout, heartbeat, durable logs, sweep-adoptable records) - #454

Merged
aarontrowbridge merged 3 commits into
mainfrom
amico/issue-426-fleet-dispatch-hardening
Aug 20, 2026
Merged

aarontrowbridge merged 3 commits into
mainfrom
amico/issue-426-fleet-dispatch-hardening

Conversation

@aarontrowbridge

Copy link
Copy Markdown
Member

Closes #426

What

The registry could model crashed/orphaned hunts — nothing created records for them, so dispatch stayed fire-and-forget and status stayed ps-grep. This lands the wrapper path, end to end:

  • amico fleet launch --session <id> --pid <n> (packages/amico-run/src/fleet_verb.ts) — record creation for a self-held holder. A wrapper is its own harness: the record is born running, pid/host stamped (that pair is what makes a dead holder sweep-detectable), no spooling, no handoff, and deliberately no liveness probe — an orphan at birth is sweep's call, not the verb's. CREATES ONCE: an existing record is refused byte-identical, so creation can never race a second writer.
  • amico fleet finish --session <id> --outcome settled|crashed --pid <n> [--step <s>] — the holder's terminal write, pid-identity-guarded (only the pid the record names may write it), through the same pure machine (settle/crash apply only to running). pid 0 is refused at both verbs — an unsweepable record is the bug, not a default. Runtime frozen at the holder's last tick; pid zeroed by the write.
  • ops/hunt.sh — the wrapper: timeout -k bounded (TERM-then-KILL babysitter where GNU timeout is absent, the fleet-status.sh pattern), heartbeat file under the hunt dir (mtime is liveness), durable logs under ~/.amico/ops/hunts/<id>/, one fleet record per run (launch → settled/crashed), re-runs uniquify the id, --bg detaches without losing the lifecycle. Killed -9 / host reboot → dead-pid running record → amico fleet sweep on that host adopts it: the registry's designed orphan path.
  • ops/install.sh deploys hunt.sh; ops/README.md documents the one usage block (retired nohup line beside its replacement, status via fleet list, erlich copy via ssh + scp).

Record-creation path choice: a CLI verb owned by the registry code (not bash-written TOML) — schema-drift armor; the extension has no existing creation path to reuse (its fleet files are topology config), so launch/finish are that path.

Acceptance criteria

  • hunt wrapper: timeout -k bounded, heartbeat file, logs under ~/.amico/ops/hunts/<id>/
  • hunts create fleet records at launch; fleet sweep adopts orphans (pid-liveness guarded)
  • status table reads records, not ps-grep (amico fleet list)
  • qec-autoresearch skill's dispatch-section rewrite — out of scope here by design (staged server-side, not in this repo; tracked by the director)

Tests

  • fleet_verb.test.ts +11: create-once, holder guard (pid mismatch/unknown), machine authority on non-running records, pid-0 refusal, dead-pid launch → sweep adoption end to end, no signals from finish.
  • hunt_wrapper.test.ts (new, 7): E2E of the wrapper through a stubbed CLI — lifecycle ordering + same holder pid on both writes, bounded overrun → crashed, durable log/heartbeat artifacts, id uniquify, --bg detach, abort-before-run when launch refuses.
  • Full package suite green (1075 passed; the one agent_spawn failure is pre-existing on pristine main — dev-machine environment bleed, verified on a baseline worktree of HEAD).

…le hunt records (#426)

Nothing created fleet records for hunts, so dispatch stayed fire-and-forget
and status stayed ps-grep. launch CREATES ONCE (a wrapper is its own harness:
record born running, pid/host stamped, no spooling/handoff, no liveness probe
— an orphan at birth is sweep's to call); finish is the holder's TERMINAL
write, pid-identity-guarded, through the same pure machine (settle/crash
apply only to running). pid 0 is refused at both: an unsweepable record is
the bug, not a default. Discipline comments updated: three entitled writers
now (sweep, launch-once, holder-finish), signals still the only other path.
…hunt wrapper (#426)

Retires the raw-nohup dispatch line clause by clause: timeout -k bound (with
a TERM-then-KILL babysitter where GNU timeout is absent, per fleet-status.sh),
heartbeat file under the hunt dir (mtime is liveness), logs under
~/.amico/ops/hunts/<id>/ (never /tmp), and a fleet record per run via the new
holder verbs — launch at start, settled on exit 0, crashed on nonzero/timeout/
signal; a wrapper killed -9 leaves a dead-pid running record for sweep to
adopt. Re-runs uniquify the id; --bg detaches without losing the lifecycle;
helper subshells drop the output pipes so ssh/spawnSync never block on an
orphaned sleep. E2E'd against a stubbed CLI (7 tests) and smoke-tested
against the real one (settle, timeout, sweep adoption).
…atch (#426)

install.sh carries hunt.sh to ~/.amico/ops/ alongside the other versioned
scripts; ops/README.md gains the one usage block — retired nohup line beside
its replacement, status via fleet list (records, not ps-grep), and the erlich
copy (ssh + scp) with its amico-CLI prerequisite.
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d71c957a-bb4e-40a4-8264-9f11a08921cb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@aarontrowbridge
aarontrowbridge marked this pull request as ready for review August 20, 2026 10:46
@aarontrowbridge
aarontrowbridge merged commit 5740c31 into main Aug 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fleet dispatch hardening: retire raw-nohup hunts (timeout, heartbeat, durable logs, sweep-adoptable records)

1 participant