From bbcc3c0323d18e16b7895490501bc3ddb2af682c Mon Sep 17 00:00:00 2001 From: brettchien Date: Sat, 8 Aug 2026 10:30:00 +0800 Subject: [PATCH 1/5] docs(adr): add Agent Lifecycle State Machine Studio's first ADR: a runtime-independent 5-state machine (Starting / Running / Unhealthy / Stopping / Stopped) for classifying an agent instance, with config (identity + version + state) as the through-line. Co-Authored-By: Claude Opus 4.8 --- docs/adr/agent-lifecycle.md | 118 ++++++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 docs/adr/agent-lifecycle.md diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md new file mode 100644 index 0000000..2668d89 --- /dev/null +++ b/docs/adr/agent-lifecycle.md @@ -0,0 +1,118 @@ +# ADR: Agent Lifecycle State Machine + +- **Status:** Proposed +- **Date:** 2026-08-08 +- **Author:** @brettchien +- **Reviewers:** +- **Tracking issues:** TBD + +--- + +## 1. Context & Motivation + +openab runs agents across different runtimes (ECS today; k8s / GKE / +docker-compose planned). We need one runtime-independent way to say "what state +is this agent in" that: any engineer reads at a glance; is identical regardless +of the runtime underneath; and is what the control plane observes and the +director acts on. + +Humans direct; agents do the control. The control plane must classify every +agent, at any moment, into **exactly one** state. + +## 2. Decision + +Every agent is in exactly one of **5 states** (mutually exclusive, exhaustive). +The through-line is **configuration = identity + version + state**. + +```mermaid +stateDiagram-v2 + [*] --> Starting + Starting --> Running : config ready & verified + Starting --> Stopped : failed to start + Running --> Unhealthy : out of sync + Unhealthy --> Running : recovered + Unhealthy --> Stopping : give up + Running --> Stopping : stop / replace + Stopping --> Stopped : state saved + Running --> Stopped : reclaimed (abrupt) + Stopped --> [*] +``` + +| State | Definition | The one thing that matters | +|---|---|---| +| **Starting** | Control plane provisions an authenticated config and injects it; the agent proves its identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. | +| **Running** | Config in sync: right version, alive and authorized (heartbeat / lease). | Only Running agents do work; sync is verified, not self-reported. | +| **Unhealthy** | Out of sync (lost heartbeat / failed check / version not converged). | Fenced off at once; recover within a window or go to Stopping. | +| **Stopping** | Flush mutable state and hand off cleanly, within a deadline. | Persist before the deadline; reclaim skips this, so checkpoint while Running. | +| **Stopped** | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normal / crash / reclaimed) to decide replace vs investigate. | + +## 3. Prior Art & Industry Research + +| Project | How it handles agent lifecycle | Key decisions | What we take / differ | +|---|---|---|---| +| **Hermes Agent** (gateway) | 6 CLI states (run/start/stop/restart/status/install); Start→Running→Stop(SIGTERM→SIGKILL); systemd option; HMAC-signed lifecycle events to webhooks. | Process lifecycle via PID/systemd; **identity = `HERMES_HOME` path + process-name matching**; signed events for observability. | Confirms a small operational set works. But path/name identity is the weak self-report we reject → we require a **control-plane-issued, verified credential** (default-deny). We adopt **signed lifecycle events** for the observe/heartbeat channel. | +| **Pi** (`pi-agent-core`) | Conversation-level `idle ⇄ turn` phases; Turn Snapshot; `flushPendingWrites` between turns; Session tree + JSONL persistence. | Durable checkpoint **between turns**, not only at shutdown. | Their idle/turn = our Running sub-states (Idle/Busy). Validates **checkpoint while Running**; Stopping is best-effort. But it's turn-scope — we operate one layer up (instance → 5 states). | +| **Pi-Desktop** | Tauri shell over `pi --mode rpc` CLI (JSON-RPC/stdio); Tauri backend does CLI process management; session fork/resume; agent logic outside the shell. | Clean **shell/runtime split**; session lifecycle (fork/resume). | Validates **Studio = thin director front-end**, control lives in core. Process-management layer ≈ a runtime driver; fork/resume ≈ Stopping→next-instance via persisted state. | + +> OpenClaw centers on a plugin gateway for message/session management across +> platforms; it has no formal instance-level lifecycle state machine, so Hermes +> and Pi are the directly relevant prior art. + +## 4. Why This Approach + +- **≤5, MECE** → one-glance comprehension; every agent maps to exactly one. +- **Sub-states are attributes, not states** (Idle/Busy; Provisioning/Hydrating/ + Booting; death cause) → keeps the set small. +- **Only Running does work** → scheduler and approval gate become a single + predicate (`state == Running`). +- **Identity default-deny + continuous trust** (heartbeat/lease) → closes the + gap every prior-art tool left open (path/name or no identity). +- **`reclaim` jump** models spot/preemption reality (skips Stopping). + +## 5. Principles + +1. **Default-deny trust:** identity is proven with a control-plane-issued + credential, never accepted from the agent's own claim. +2. **Trust and sync are continuous, not one-shot** — hence heartbeat / lease. +3. **Only `Stopped` is terminal** (absorbing). Restart = a new lifecycle. +4. **`reclaim`** may jump from any live state straight to `Stopped`. +5. **Runtime-independent:** each driver projects native states onto these 5; the + machine never changes per runtime. + +## 6. Runtime Independence (projection) + +Each runtime driver maps its native states onto the 5. "In sync" (Running) means +desired config equals observed — the reconcile loop has zero diff for this agent. + +| canonical | ECS | k8s / GKE | docker-compose | +|---|---|---|---| +| Starting | PROVISIONING/PENDING | Pending/ContainerCreating | created/starting | +| Running | RUNNING + health OK | readinessProbe OK | healthy | +| Unhealthy | health check fail | probe fail / lease lost | healthcheck fail | +| Stopping | DEACTIVATING (stopTimeout) | Terminating (grace + preStop) | stopping (stop_grace_period) | +| Stopped | STOPPED (+reason) | deleted / preempted | exited | + +## 7. Alternatives Considered + +- **Adopt Hermes' 6 operational states verbatim** — rejected: mixes + install/service concerns with runtime state and lacks a health/identity + distinction. +- **Make the conversation turn loop (pi idle/turn) the primary machine** — + rejected: that's a sub-layer of Running, not instance lifecycle. +- **K8s-style granular phases** (Pending/Running/Succeeded/Failed/Unknown + + container states) — rejected: too many for one-glance; folded into attributes. +- **Drop `Unhealthy` (only Running/Stopped)** — rejected: loses the "alive but + fenced-off" distinction the director relies on. + +## 8. Consequences + +- The read-model and Studio report **only these 5 states**. +- Every runtime driver must provide a **native→5-state projection** + (conformance requirement). +- Detailed sub-states are attributes of the 5, not new states. +- **Follow-up:** a `RuntimeDriver` contract ADR defines the verbs + (apply/observe/scale/…) that drive these transitions. + +## 9. Validation + +Docs-only ADR; mermaid renders on GitHub. No code changes. From 364983aeccc9ef89259da7f471f1e84d37b9e52f Mon Sep 17 00:00:00 2001 From: brettchien Date: Sat, 8 Aug 2026 13:12:27 +0800 Subject: [PATCH 2/5] docs(adr): revise agent-lifecycle to 6 states + add review runbook Fold the 8-axis refute-pass findings (Mira/Jellyfish/Falcon): - Paused promoted to a 6th state, defined by (desiredStatus,accepting_work, health); restores single-field dispatch predicate state==Running. - Unhealthy is fault-only; version-skew becomes a Running 'superseded' attribute (drives drain->replace), not Unhealthy. - reclaim split into planned (compressed Stopping) vs hard-loss (->Stopped); diagram gains Starting->Stopped, Unhealthy->Stopped, Paused edges. - identity: name trust root (runtime injection), per-instance credential at Starting, CP-signed instance-bound lease + monotonic fencing epoch, revocation + re-prove on recovery. - projection: desiredStatus discriminator (ECS STOPPED / k8s deletionTimestamp / compose stop-requested), ACTIVATING->Starting, DEACTIVATING conditional, compose requires healthcheck + restart:no, k8s Unknown->Unhealthy. - config model: Instance = Desired Spec (identity+version) + Observed State (state is observed, not desired config). - prior art: add k8s/Nomad/Temporal/OTP/EC2/systemd/Ray rows; death cause enum. - restructure to MADR + Y-statement; add docs/review-runbook.md (8-axis rubric). Co-Authored-By: Claude Opus 4.8 --- docs/adr/agent-lifecycle.md | 236 +++++++++++++++++++++++------------- docs/review-runbook.md | 59 +++++++++ 2 files changed, 211 insertions(+), 84 deletions(-) create mode 100644 docs/review-runbook.md diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md index 2668d89..5deebc9 100644 --- a/docs/adr/agent-lifecycle.md +++ b/docs/adr/agent-lifecycle.md @@ -3,12 +3,19 @@ - **Status:** Proposed - **Date:** 2026-08-08 - **Author:** @brettchien -- **Reviewers:** +- **Reviewers:** Mira (ECS), Jellyfish (control-plane), Falcon (MCP) - **Tracking issues:** TBD +> **Y-statement.** In the context of running agents across heterogeneous +> runtimes, facing the need for one glanceable, runtime-independent notion of +> "what state is this agent in", we decided a canonical **6-state** lifecycle +> discriminated by `(desiredStatus, accepting_work, health)`, to get a +> **single-field dispatch predicate** and a clean native→canonical projection, +> accepting a sixth state and a per-runtime projection/conformance burden. + --- -## 1. Context & Motivation +## 1. Context & Problem openab runs agents across different runtimes (ECS today; k8s / GKE / docker-compose planned). We need one runtime-independent way to say "what state @@ -19,100 +26,161 @@ director acts on. Humans direct; agents do the control. The control plane must classify every agent, at any moment, into **exactly one** state. -## 2. Decision +## 2. Decision Drivers + +- **One-glance comprehension** — a small, mutually-exclusive, exhaustive set. +- **Single-field dispatch** — "may this agent take new work?" should be one + field, not a conjunction every caller must remember. +- **Runtime-independent, decidable projection** — each driver must map native + signals onto the canonical set *without ambiguity*. +- **Honest about faults vs intent vs teardown** — health, admission policy, and + terminate-intent are different axes and must not be conflated. -Every agent is in exactly one of **5 states** (mutually exclusive, exhaustive). -The through-line is **configuration = identity + version + state**. +## 3. Decision + +Every agent is in exactly one of **6 states**, discriminated by three +observable axes — `desiredStatus` (running / stopped), `accepting_work` +(bool), and `health` (in-sync & authorized / not): ```mermaid stateDiagram-v2 [*] --> Starting - Starting --> Running : config ready & verified - Starting --> Stopped : failed to start - Running --> Unhealthy : out of sync - Unhealthy --> Running : recovered - Unhealthy --> Stopping : give up - Running --> Stopping : stop / replace + Starting --> Running : identity verified & config live + Starting --> Stopped : failed to start / cancelled / reclaimed + Running --> Paused : cordon (director hold) + Paused --> Running : resume + Running --> Unhealthy : liveness / authz lost + Paused --> Unhealthy : liveness / authz lost + Unhealthy --> Running : recovered (re-proves identity) + Unhealthy --> Stopping : give up (graceful) + Unhealthy --> Stopped : hard loss (OOM / crash / node death), no flush + Running --> Stopping : stop / replace (desired=stopped) + Paused --> Stopping : stop / replace Stopping --> Stopped : state saved - Running --> Stopped : reclaimed (abrupt) - Stopped --> [*] + Running --> Stopped : reclaim (hard loss) + Paused --> Stopped : reclaim (hard loss) + Stopped --> [*] ``` -| State | Definition | The one thing that matters | -|---|---|---| -| **Starting** | Control plane provisions an authenticated config and injects it; the agent proves its identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. | -| **Running** | Config in sync: right version, alive and authorized (heartbeat / lease). | Only Running agents do work; sync is verified, not self-reported. | -| **Unhealthy** | Out of sync (lost heartbeat / failed check / version not converged). | Fenced off at once; recover within a window or go to Stopping. | -| **Stopping** | Flush mutable state and hand off cleanly, within a deadline. | Persist before the deadline; reclaim skips this, so checkpoint while Running. | -| **Stopped** | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normal / crash / reclaimed) to decide replace vs investigate. | - -## 3. Prior Art & Industry Research - -| Project | How it handles agent lifecycle | Key decisions | What we take / differ | +| State | Discriminator | Definition | The one thing that matters | |---|---|---|---| -| **Hermes Agent** (gateway) | 6 CLI states (run/start/stop/restart/status/install); Start→Running→Stop(SIGTERM→SIGKILL); systemd option; HMAC-signed lifecycle events to webhooks. | Process lifecycle via PID/systemd; **identity = `HERMES_HOME` path + process-name matching**; signed events for observability. | Confirms a small operational set works. But path/name identity is the weak self-report we reject → we require a **control-plane-issued, verified credential** (default-deny). We adopt **signed lifecycle events** for the observe/heartbeat channel. | -| **Pi** (`pi-agent-core`) | Conversation-level `idle ⇄ turn` phases; Turn Snapshot; `flushPendingWrites` between turns; Session tree + JSONL persistence. | Durable checkpoint **between turns**, not only at shutdown. | Their idle/turn = our Running sub-states (Idle/Busy). Validates **checkpoint while Running**; Stopping is best-effort. But it's turn-scope — we operate one layer up (instance → 5 states). | -| **Pi-Desktop** | Tauri shell over `pi --mode rpc` CLI (JSON-RPC/stdio); Tauri backend does CLI process management; session fork/resume; agent logic outside the shell. | Clean **shell/runtime split**; session lifecycle (fork/resume). | Validates **Studio = thin director front-end**, control lives in core. Process-management layer ≈ a runtime driver; fork/resume ≈ Stopping→next-instance via persisted state. | - -> OpenClaw centers on a plugin gateway for message/session management across -> platforms; it has no formal instance-level lifecycle state machine, so Hermes -> and Pi are the directly relevant prior art. - -## 4. Why This Approach - -- **≤5, MECE** → one-glance comprehension; every agent maps to exactly one. -- **Sub-states are attributes, not states** (Idle/Busy; Provisioning/Hydrating/ - Booting; death cause) → keeps the set small. -- **Only Running does work** → scheduler and approval gate become a single - predicate (`state == Running`). -- **Identity default-deny + continuous trust** (heartbeat/lease) → closes the - gap every prior-art tool left open (path/name or no identity). -- **`reclaim` jump** models spot/preemption reality (skips Stopping). - -## 5. Principles - -1. **Default-deny trust:** identity is proven with a control-plane-issued - credential, never accepted from the agent's own claim. -2. **Trust and sync are continuous, not one-shot** — hence heartbeat / lease. -3. **Only `Stopped` is terminal** (absorbing). Restart = a new lifecycle. -4. **`reclaim`** may jump from any live state straight to `Stopped`. -5. **Runtime-independent:** each driver projects native states onto these 5; the - machine never changes per runtime. +| **Starting** | desired=running; identity not yet verified/live | CP provisions an authenticated config and injects it; the agent proves identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. A **per-instance** credential is minted here. | +| **Running** | desired=running ∧ accepting_work ∧ healthy | Alive, authorized, in-sync, and admitting work. | **Only Running admits new work** → dispatch/gate is the single predicate `state == Running`. | +| **Paused** | desired=running ∧ ¬accepting_work ∧ healthy | Healthy and in-sync but deliberately not admitting (director cordon). | Intent, not fault. Resumable; still subject to health edges. Keeping it a peer state is what keeps the dispatch predicate single-field. | +| **Unhealthy** | desired=running ∧ ¬healthy | Alive but fenced: liveness/authz/probe/lease lost. **Not** version skew. | Fenced at once; recover within a window (re-prove identity) or go to Stopping. Split cause: *observed-bad* vs *unobservable* (node lost). | +| **Stopping** | desired=stopped; graceful window open | Terminate committed: flush state and finish in-flight work within a deadline (may still be health-OK). | `desiredStatus==stopped` is the cross-runtime discriminator. Durability was already secured while Running. | +| **Stopped** | terminal (absorbing) | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normative enum: normal / crash / reclaimed). Granularity is **instance-level**. | + +**Attributes, not states** (read alongside the state): `accepting_work` +(Running vs Paused); `superseded` / version-skew (healthy; drives +drain→replace; stays Running); health `cause` = observed-bad vs unobservable; +death `cause` enum; turn-level busy/idle. + +## 4. Principles + +1. **Default-deny identity.** Identity is proven with a control-plane-issued + credential, never accepted from the agent's own claim. The **trust root is + the runtime's injection primitive** (IRSA / k8s projected SA token) that + delegates a platform identity — state it explicitly. **Role identity ≠ + instance identity**: mint a **per-instance** credential at `Starting`. +2. **Trust & sync are continuous.** Heartbeat carries a CP-signed, short-TTL + **lease token bound to the instance id** (task ARN / pod UID). A **monotonic + fencing epoch** guards generations — the CP accepts only the highest epoch, + defeating zombie/split-brain after a partition. Credentials are revoked on + Stopping/Stopped; `Unhealthy→Running` must re-prove identity. +3. **Only `Stopped` is terminal (absorbing), at instance granularity.** A + container restart within the same pod is the *same* instance, not a + `Stopped→Starting` flap; restart = a new lifecycle only when a new instance + is created. +4. **`reclaim` is two paths, not one.** A *planned* interruption (Spot/preempt + notice — ECS ~120s SIGTERM, GKE ~30s + preStop) **compresses `Stopping`** + into a short deadline. Only a *hard* loss (node death / SIGKILL / OOM) jumps + straight to `Stopped`. Durability never relies on the Stopping window — + **checkpoint while Running.** +5. **Runtime-independent.** Each driver projects native signals onto the 6 via + the discriminators `(desiredStatus, accepting_work, health)`; the machine + never changes per runtime. +6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running` + (single field). *Doing in-flight work* = `Running ∪ Stopping`(within + deadline). Don't collapse them into one sentence. + +## 5. Model: config vs observed + +`Instance = Desired Spec (identity + version) + Observed State`. Desired and +observed are strictly separated; **state is observed, not part of the desired +config**. "In sync" (Running) means the reconcile loop has zero diff on the +desired spec. (This replaces the earlier `config = identity + version + state`, +which folded observed state into desired config and could never reconcile to +zero diff.) ## 6. Runtime Independence (projection) -Each runtime driver maps its native states onto the 5. "In sync" (Running) means -desired config equals observed — the reconcile loop has zero diff for this agent. +Discriminators, not native strings. `desiredStatus==stopped` is one signal +across runtimes: **ECS `desiredStatus STOPPED` ⟺ k8s `deletionTimestamp!=null` +⟺ compose stop-requested** — that is what makes `Stopping` decidable rather than +an ECS-only coincidence. | canonical | ECS | k8s / GKE | docker-compose | |---|---|---|---| -| Starting | PROVISIONING/PENDING | Pending/ContainerCreating | created/starting | -| Running | RUNNING + health OK | readinessProbe OK | healthy | -| Unhealthy | health check fail | probe fail / lease lost | healthcheck fail | -| Stopping | DEACTIVATING (stopTimeout) | Terminating (grace + preStop) | stopping (stop_grace_period) | -| Stopped | STOPPED (+reason) | deleted / preempted | exited | - -## 7. Alternatives Considered - -- **Adopt Hermes' 6 operational states verbatim** — rejected: mixes - install/service concerns with runtime state and lacks a health/identity - distinction. -- **Make the conversation turn loop (pi idle/turn) the primary machine** — - rejected: that's a sub-layer of Running, not instance lifecycle. -- **K8s-style granular phases** (Pending/Running/Succeeded/Failed/Unknown + - container states) — rejected: too many for one-glance; folded into attributes. -- **Drop `Unhealthy` (only Running/Stopped)** — rejected: loses the "alive but - fenced-off" distinction the director relies on. - -## 8. Consequences - -- The read-model and Studio report **only these 5 states**. -- Every runtime driver must provide a **native→5-state projection** - (conformance requirement). -- Detailed sub-states are attributes of the 5, not new states. -- **Follow-up:** a `RuntimeDriver` contract ADR defines the verbs - (apply/observe/scale/…) that drive these transitions. - -## 9. Validation - -Docs-only ADR; mermaid renders on GitHub. No code changes. +| Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting | +| Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* | +| Paused | RUNNING + health OK + app-level cordon (`accepting_work=false`) | Ready but cordoned (app-level) | running + app cordon | +| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail | +| Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) | +| Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited | + +**Driver conformance conditions** +- A driver must expose all three discriminators; if it cannot, it does not + conform. +- **docker-compose requires a `healthcheck`** — without one it only sees + running/exited and can never separate Running from Unhealthy. +- **docker-compose must set `restart: "no"`** and hand restart to the control + plane; `restart: unless-stopped` auto-resurrects a crashed container, which + contradicts "Stopped is terminal" and competes with reclaim/replace. + +## 7. Considered Options + +- **6 states with Paused as a peer state (chosen).** Uses the discriminators to + define Paused rigorously; keeps dispatch single-field. +- **5 states, Paused/Draining as a `Running` attribute** (reviewers' converged + proposal) — *rejected as the surface model* because it forces a two-field + dispatch predicate (`Running && accepting_work`); every caller that forgets + `&& accepting_work` silently mis-schedules a paused agent. **We adopt its + `(desiredStatus, accepting_work)` machinery as Paused's definition.** +- **Hermes' 6 operational states verbatim** — rejected: mixes install/service + concerns with runtime state; path/name identity is the self-report we reject. +- **pi `idle/turn` as the primary machine** — rejected: a sub-layer of Running. +- **K8s granular phases** (Pending/Running/Succeeded/Failed/Unknown + container + states) — rejected for the surface set; folded into attributes. +- **Drop `Unhealthy`** — rejected: loses the "alive but fenced" distinction. + +## 8. Prior Art + +| Project | Model | What we take / differ | +|---|---|---| +| **Kubernetes** Pod lifecycle | Phase + Conditions + Probes (three-layer decoupling); `Unknown` on node loss | Direct ancestor; we take the phase/condition/probe split; `Unknown`→Unhealthy(fenced). | +| **HashiCorp Nomad** | alloc states pending/running/complete/failed/**lost**; driver preemption events | `lost`/`unknown` is exactly our *unobservable* Unhealthy case. | +| **Temporal / Cadence** | workflow/activity states + heartbeat **lease fencing** | Validates the fencing epoch on the heartbeat lease. | +| **Erlang/OTP supervisor** | child spec + crash exit reason + `one_for_one`; restart spawns a new child | Supports "restart = new lifecycle / fresh instance". | +| **AWS EC2 instance lifecycle** | pending/running/stopping/stopped/terminated | Near-identical shape; instance-level granularity. | +| **systemd unit** | active / **failed** / … as first-class | `failed` as a first-class fault state. | +| **Ray actor** | PENDING / ALIVE / RESTARTING / DEAD | Close 1:1; `RESTARTING` = our replace path. | +| **Hermes / Pi / Pi-Desktop** | ops CLI states / in-process turn engine / desktop shell | Adjacent code, not instance-level lifecycle. Pi validates **checkpoint-while-Running**. | + +## 9. Consequences + +- The read-model and Studio report **only these 6 states**. +- Every runtime driver must provide a **native→6 projection** via the + discriminators (conformance requirement), including the compose healthcheck + and `restart:"no"` conditions above. +- Detailed sub-states are **attributes** of the 6 (accepting_work, superseded, + health-cause, death-cause enum, busy/idle), not new states. +- **Follow-ups:** a `RuntimeDriver` contract ADR (verbs apply / observe / scale + / cordon / …); an identity / lease / epoch spec ADR. + +## 10. More Information + +Format follows **MADR** (markdown ADR: context → drivers → options → decision → +consequences) with a **Nygard** status/context/decision/consequences spine and a +**Y-statement** summary. See `docs/review-runbook.md` for the review rubric this +ADR was gated on. diff --git a/docs/review-runbook.md b/docs/review-runbook.md new file mode 100644 index 0000000..67f495c --- /dev/null +++ b/docs/review-runbook.md @@ -0,0 +1,59 @@ +# Review Runbook + +How we review ADRs and design docs in this repo. The goal is a **falsifiable** +review — reviewers try to break each load-bearing claim, not nod at it. Peer +"LGTM" carries weight only after the claim has survived an attempt to refute it. + +## The 8 axes + +Every **load-bearing claim** in a doc is scored against all 8. A claim survives +only if it passes **every** axis. + +1. **Simplicity / concise** — minimal surface area; no state/column/sentence that + could be cut. *Fails on:* bloat. +2. **In scope** — decides only what this doc is for; no sprawl (e.g. don't fold a + RuntimeDriver contract or implementation detail into a state-model ADR). +3. **Factcheck** — runtime behaviour and prior-art claims are true, **with a + source**. A claim with no source does not pass. +4. **Refute** — assume the claim is *wrong* and try to prove it (adversarial + default); it survives only if the refutation fails. +5. **Coverage / MECE** — exhaustive and mutually exclusive. Ask "what state / + edge / runtime situation is missing?" and "can one situation fall into two?" + (Distinct from Refute: Refute attacks "what you said is wrong"; Coverage + attacks "you didn't say X".) +6. **Consistency** — sections don't contradict each other (definition ↔ diagram + ↔ projection ↔ principles) and align with the doc's own first-principles. +7. **Decidable / actionable** — the decision is actually made, and an + implementer/driver can act on it without ambiguity. +8. **Reversibility / lock-in** — what this locks in and how expensive it is to + change later. + +## Verdict rule + +- Score each load-bearing claim across all 8 axes. +- **Refute** defaults to *refuted* — a claim is only "survived" once refutation + attempts fail. +- **Factcheck** with no source does not pass. +- Report only the axes a claim **fails**, with the counter-example or source. + Passing axes need no restatement. + +## How to run a refute pass + +1. Enumerate the doc's load-bearing claims (the ones the decision rests on). +2. Assign refuters; each is told to assume the claim is wrong and produce a + counter-example, a missing case, or a contradicting source. +3. A claim survives only if no refuter lands. Surviving-with-fixes → fold the + fix; failed → back to the author. +4. Consolidate into one review comment on the PR; the author decides how to land. + +## References (ADR writing) + +- **Michael Nygard**, *Documenting Architecture Decisions* — the origin; + Status / Context / Decision / Consequences. +- **MADR** — Markdown ADR: context → drivers → considered options → decision + outcome → consequences. +- **adr.github.io** — templates and `adr-tools`. +- **Joel Parker Henderson**, ADR templates & examples collection. + +- **Y-statement** — one-line decision summary: "In context X, facing Y, we + decided Z, to achieve W, accepting V." From 78ccb91691fbd88342e28fc927ccaebf72a14cdc Mon Sep 17 00:00:00 2001 From: brettchien Date: Sat, 8 Aug 2026 14:42:28 +0800 Subject: [PATCH 3/5] =?UTF-8?q?docs(adr):=20v3=20=E2=80=94=204-axis=20disc?= =?UTF-8?q?riminator=20+=204=20review=20remnants?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final 8-axis re-review fixes (Jellyfish F1/F2 + Falcon MCP + Mira): - discriminator 3->4 axes: add latching identity_verified (separates Starting from Unhealthy, whose 3-axis tuples collided). CP-observable per runtime. - accepting_work authority pinned to CP/director, never agent self-report. - superseded => cordon to Paused (accepting_work=false) -> Stopping/replace, so a superseded agent is never left dispatchable in Running. - Principle 6 in-flight set = Running U Paused U Stopping(within deadline). - compose: docker pause (SIGSTOP) -> healthcheck stall -> Unhealthy. Co-Authored-By: Claude Opus 4.8 --- docs/adr/agent-lifecycle.md | 47 ++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md index 5deebc9..03ad683 100644 --- a/docs/adr/agent-lifecycle.md +++ b/docs/adr/agent-lifecycle.md @@ -9,7 +9,8 @@ > **Y-statement.** In the context of running agents across heterogeneous > runtimes, facing the need for one glanceable, runtime-independent notion of > "what state is this agent in", we decided a canonical **6-state** lifecycle -> discriminated by `(desiredStatus, accepting_work, health)`, to get a +> discriminated by `(desiredStatus, accepting_work, health, identity_verified)`, +> to get a > **single-field dispatch predicate** and a clean native→canonical projection, > accepting a sixth state and a per-runtime projection/conformance burden. @@ -38,9 +39,14 @@ agent, at any moment, into **exactly one** state. ## 3. Decision -Every agent is in exactly one of **6 states**, discriminated by three -observable axes — `desiredStatus` (running / stopped), `accepting_work` -(bool), and `health` (in-sync & authorized / not): +Every agent is in exactly one of **6 states**, discriminated by four observable +axes — `desiredStatus` (running / stopped), `accepting_work` (bool), `health` +(in-sync & authorized / not), and `identity_verified` (a **latching** bit: set +true the first time the agent reaches Running, never cleared). The latch is what +separates `Starting` (never verified) from `Unhealthy` (was verified, now +faulted) — without it their `(desiredStatus, accepting_work, health)` tuples +collide. It is CP-observable per runtime: ECS `lastStatus` ever reached RUNNING / +k8s ever Ready / compose ever healthy. ```mermaid stateDiagram-v2 @@ -64,17 +70,19 @@ stateDiagram-v2 | State | Discriminator | Definition | The one thing that matters | |---|---|---|---| -| **Starting** | desired=running; identity not yet verified/live | CP provisions an authenticated config and injects it; the agent proves identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. A **per-instance** credential is minted here. | -| **Running** | desired=running ∧ accepting_work ∧ healthy | Alive, authorized, in-sync, and admitting work. | **Only Running admits new work** → dispatch/gate is the single predicate `state == Running`. | -| **Paused** | desired=running ∧ ¬accepting_work ∧ healthy | Healthy and in-sync but deliberately not admitting (director cordon). | Intent, not fault. Resumable; still subject to health edges. Keeping it a peer state is what keeps the dispatch predicate single-field. | -| **Unhealthy** | desired=running ∧ ¬healthy | Alive but fenced: liveness/authz/probe/lease lost. **Not** version skew. | Fenced at once; recover within a window (re-prove identity) or go to Stopping. Split cause: *observed-bad* vs *unobservable* (node lost). | +| **Starting** | desired=running ∧ ¬identity_verified | CP provisions an authenticated config and injects it; the agent proves identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. A **per-instance** credential is minted here. | +| **Running** | desired=running ∧ identity_verified ∧ accepting_work ∧ healthy | Alive, authorized, in-sync, and admitting work. | **Only Running admits new work** → dispatch/gate is the single predicate `state == Running`. | +| **Paused** | desired=running ∧ identity_verified ∧ ¬accepting_work ∧ healthy | Healthy and in-sync but deliberately not admitting (director cordon). | Intent, not fault. Resumable; still subject to health edges. Keeping it a peer state is what keeps the dispatch predicate single-field. | +| **Unhealthy** | desired=running ∧ identity_verified ∧ ¬healthy | Alive but fenced: liveness/authz/probe/lease lost. **Not** version skew. | Fenced at once; recover within a window (re-prove identity) or go to Stopping. Split cause: *observed-bad* vs *unobservable* (node lost). | | **Stopping** | desired=stopped; graceful window open | Terminate committed: flush state and finish in-flight work within a deadline (may still be health-OK). | `desiredStatus==stopped` is the cross-runtime discriminator. Durability was already secured while Running. | | **Stopped** | terminal (absorbing) | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normative enum: normal / crash / reclaimed). Granularity is **instance-level**. | **Attributes, not states** (read alongside the state): `accepting_work` -(Running vs Paused); `superseded` / version-skew (healthy; drives -drain→replace; stays Running); health `cause` = observed-bad vs unobservable; -death `cause` enum; turn-level busy/idle. +(Running vs Paused) — its authority is the **CP/director**, never the agent's +self-report; `superseded` / version-skew (healthy) ⇒ the agent is **cordoned to +Paused** (`accepting_work=false`) and then goes to Stopping/replace, so a +superseded agent is never left dispatchable in Running; health `cause` = +observed-bad vs unobservable; death `cause` enum; turn-level busy/idle. ## 4. Principles @@ -98,11 +106,12 @@ death `cause` enum; turn-level busy/idle. straight to `Stopped`. Durability never relies on the Stopping window — **checkpoint while Running.** 5. **Runtime-independent.** Each driver projects native signals onto the 6 via - the discriminators `(desiredStatus, accepting_work, health)`; the machine - never changes per runtime. + the discriminators `(desiredStatus, accepting_work, health, identity_verified)`; + the machine never changes per runtime. 6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running` - (single field). *Doing in-flight work* = `Running ∪ Stopping`(within - deadline). Don't collapse them into one sentence. + (single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping`(within + deadline) — a cordoned (Paused) agent still finishes its current turn / MCP + call. Don't collapse them into one sentence. ## 5. Model: config vs observed @@ -124,14 +133,14 @@ an ECS-only coincidence. |---|---|---|---| | Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting | | Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* | -| Paused | RUNNING + health OK + app-level cordon (`accepting_work=false`) | Ready but cordoned (app-level) | running + app cordon | -| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail | +| Paused | RUNNING + health OK + CP/director cordon (`accepting_work=false`) | Ready but cordoned (CP/director) | running + CP/director cordon | +| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (SIGSTOP) → healthcheck stall → Unhealthy | | Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) | | Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited | **Driver conformance conditions** -- A driver must expose all three discriminators; if it cannot, it does not - conform. +- A driver must expose all four discriminators (including the latching + `identity_verified`); if it cannot, it does not conform. - **docker-compose requires a `healthcheck`** — without one it only sees running/exited and can never separate Running from Unhealthy. - **docker-compose must set `restart: "no"`** and hand restart to the control From d12323e5f4b0ff0674accc4fae2b4fafd199c386 Mon Sep 17 00:00:00 2001 From: brettchien Date: Sat, 8 Aug 2026 16:13:22 +0800 Subject: [PATCH 4/5] =?UTF-8?q?docs(adr):=20v4=20=E2=80=94=20fix=20superse?= =?UTF-8?q?ded=20scope-leak=20(R1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jellyfish R1 + Mira (ECS): the v3 'superseded => Paused -> Stopping/replace' sentence prescribed fleet-level rollout ordering (make-before-break, cordon sequencing = deployment controller's job), which is this instance-level ADR's declared non-goal. Keep the instance-level fact (superseded => accepting_work =false => Paused, never dispatchable) and defer drain/replace ordering to a future rollout / RuntimeDriver ADR. Co-Authored-By: Claude Opus 4.8 --- docs/adr/agent-lifecycle.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md index 03ad683..6ef9c53 100644 --- a/docs/adr/agent-lifecycle.md +++ b/docs/adr/agent-lifecycle.md @@ -79,10 +79,13 @@ stateDiagram-v2 **Attributes, not states** (read alongside the state): `accepting_work` (Running vs Paused) — its authority is the **CP/director**, never the agent's -self-report; `superseded` / version-skew (healthy) ⇒ the agent is **cordoned to -Paused** (`accepting_work=false`) and then goes to Stopping/replace, so a -superseded agent is never left dispatchable in Running; health `cause` = -observed-bad vs unobservable; death `cause` enum; turn-level busy/idle. +self-report; `superseded` / version-skew (a healthy instance whose desired +version has moved on) ⇒ `accepting_work=false`, so it classifies as **Paused** +and is never dispatched new work. *When and in what order* a superseded instance +is drained or replaced is a **fleet-level rollout** concern (e.g. +make-before-break) — out of scope for this instance-level ADR; see the future +rollout / RuntimeDriver ADR. Also: health `cause` = observed-bad vs +unobservable; death `cause` enum; turn-level busy/idle. ## 4. Principles From 08905d87f0bd6f21968fb2d6faffea7899c72e3e Mon Sep 17 00:00:00 2001 From: brettchien Date: Sat, 8 Aug 2026 16:26:56 +0800 Subject: [PATCH 5/5] =?UTF-8?q?docs(adr):=20mark=20Accepted=20=E2=80=94=20?= =?UTF-8?q?three-way=20LGTM=20(Mira/Jellyfish/Falcon)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- docs/adr/agent-lifecycle.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md index 6ef9c53..3c69301 100644 --- a/docs/adr/agent-lifecycle.md +++ b/docs/adr/agent-lifecycle.md @@ -1,10 +1,10 @@ # ADR: Agent Lifecycle State Machine -- **Status:** Proposed +- **Status:** Accepted - **Date:** 2026-08-08 - **Author:** @brettchien -- **Reviewers:** Mira (ECS), Jellyfish (control-plane), Falcon (MCP) -- **Tracking issues:** TBD +- **Reviewers:** Mira (ECS), Jellyfish (control-plane), Falcon (MCP) — all LGTM +- **Tracking issues:** implementation openabdev/studio#2 > **Y-statement.** In the context of running agents across heterogeneous > runtimes, facing the need for one glanceable, runtime-independent notion of