Fleet dispatch hardening: retire raw-nohup hunts (timeout, heartbeat, durable logs, sweep-adoptable records) - #454
Merged
Conversation
…le hunt records (#426) Nothing created fleet records for hunts, so dispatch stayed fire-and-forget and status stayed ps-grep. launch CREATES ONCE (a wrapper is its own harness: record born running, pid/host stamped, no spooling/handoff, no liveness probe — an orphan at birth is sweep's to call); finish is the holder's TERMINAL write, pid-identity-guarded, through the same pure machine (settle/crash apply only to running). pid 0 is refused at both: an unsweepable record is the bug, not a default. Discipline comments updated: three entitled writers now (sweep, launch-once, holder-finish), signals still the only other path.
…hunt wrapper (#426) Retires the raw-nohup dispatch line clause by clause: timeout -k bound (with a TERM-then-KILL babysitter where GNU timeout is absent, per fleet-status.sh), heartbeat file under the hunt dir (mtime is liveness), logs under ~/.amico/ops/hunts/<id>/ (never /tmp), and a fleet record per run via the new holder verbs — launch at start, settled on exit 0, crashed on nonzero/timeout/ signal; a wrapper killed -9 leaves a dead-pid running record for sweep to adopt. Re-runs uniquify the id; --bg detaches without losing the lifecycle; helper subshells drop the output pipes so ssh/spawnSync never block on an orphaned sleep. E2E'd against a stubbed CLI (7 tests) and smoke-tested against the real one (settle, timeout, sweep adoption).
…atch (#426) install.sh carries hunt.sh to ~/.amico/ops/ alongside the other versioned scripts; ops/README.md gains the one usage block — retired nohup line beside its replacement, status via fleet list (records, not ps-grep), and the erlich copy (ssh + scp) with its amico-CLI prerequisite.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #426
What
The registry could model crashed/orphaned hunts — nothing created records for them, so dispatch stayed fire-and-forget and status stayed ps-grep. This lands the wrapper path, end to end:
amico fleet launch --session <id> --pid <n>(packages/amico-run/src/fleet_verb.ts) — record creation for a self-held holder. A wrapper is its own harness: the record is bornrunning, pid/host stamped (that pair is what makes a dead holder sweep-detectable), no spooling, no handoff, and deliberately no liveness probe — an orphan at birth is sweep's call, not the verb's. CREATES ONCE: an existing record is refused byte-identical, so creation can never race a second writer.amico fleet finish --session <id> --outcome settled|crashed --pid <n> [--step <s>]— the holder's terminal write, pid-identity-guarded (only the pid the record names may write it), through the same pure machine (settle/crashapply only torunning). pid 0 is refused at both verbs — an unsweepable record is the bug, not a default. Runtime frozen at the holder's last tick; pid zeroed by the write.ops/hunt.sh— the wrapper:timeout -kbounded (TERM-then-KILL babysitter where GNU timeout is absent, the fleet-status.sh pattern), heartbeat file under the hunt dir (mtime is liveness), durable logs under~/.amico/ops/hunts/<id>/, one fleet record per run (launch → settled/crashed), re-runs uniquify the id,--bgdetaches without losing the lifecycle. Killed -9 / host reboot → dead-pidrunningrecord →amico fleet sweepon that host adopts it: the registry's designed orphan path.hunt.sh; ops/README.md documents the one usage block (retired nohup line beside its replacement, status viafleet list, erlich copy via ssh + scp).Record-creation path choice: a CLI verb owned by the registry code (not bash-written TOML) — schema-drift armor; the extension has no existing creation path to reuse (its fleet files are topology config), so
launch/finishare that path.Acceptance criteria
timeout -kbounded, heartbeat file, logs under~/.amico/ops/hunts/<id>/fleet sweepadopts orphans (pid-liveness guarded)amico fleet list)Tests
fleet_verb.test.ts+11: create-once, holder guard (pid mismatch/unknown), machine authority on non-running records, pid-0 refusal, dead-pid launch → sweep adoption end to end, no signals from finish.hunt_wrapper.test.ts(new, 7): E2E of the wrapper through a stubbed CLI — lifecycle ordering + same holder pid on both writes, bounded overrun → crashed, durable log/heartbeat artifacts, id uniquify,--bgdetach, abort-before-run when launch refuses.agent_spawnfailure is pre-existing on pristine main — dev-machine environment bleed, verified on a baseline worktree of HEAD).