diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md new file mode 100644 index 0000000..3c69301 --- /dev/null +++ b/docs/adr/agent-lifecycle.md @@ -0,0 +1,198 @@ +# ADR: Agent Lifecycle State Machine + +- **Status:** Accepted +- **Date:** 2026-08-08 +- **Author:** @brettchien +- **Reviewers:** Mira (ECS), Jellyfish (control-plane), Falcon (MCP) — all LGTM +- **Tracking issues:** implementation openabdev/studio#2 + +> **Y-statement.** In the context of running agents across heterogeneous +> runtimes, facing the need for one glanceable, runtime-independent notion of +> "what state is this agent in", we decided a canonical **6-state** lifecycle +> discriminated by `(desiredStatus, accepting_work, health, identity_verified)`, +> to get a +> **single-field dispatch predicate** and a clean native→canonical projection, +> accepting a sixth state and a per-runtime projection/conformance burden. + +--- + +## 1. Context & Problem + +openab runs agents across different runtimes (ECS today; k8s / GKE / +docker-compose planned). We need one runtime-independent way to say "what state +is this agent in" that: any engineer reads at a glance; is identical regardless +of the runtime underneath; and is what the control plane observes and the +director acts on. + +Humans direct; agents do the control. The control plane must classify every +agent, at any moment, into **exactly one** state. + +## 2. Decision Drivers + +- **One-glance comprehension** — a small, mutually-exclusive, exhaustive set. +- **Single-field dispatch** — "may this agent take new work?" should be one + field, not a conjunction every caller must remember. +- **Runtime-independent, decidable projection** — each driver must map native + signals onto the canonical set *without ambiguity*. +- **Honest about faults vs intent vs teardown** — health, admission policy, and + terminate-intent are different axes and must not be conflated. + +## 3. Decision + +Every agent is in exactly one of **6 states**, discriminated by four observable +axes — `desiredStatus` (running / stopped), `accepting_work` (bool), `health` +(in-sync & authorized / not), and `identity_verified` (a **latching** bit: set +true the first time the agent reaches Running, never cleared). The latch is what +separates `Starting` (never verified) from `Unhealthy` (was verified, now +faulted) — without it their `(desiredStatus, accepting_work, health)` tuples +collide. It is CP-observable per runtime: ECS `lastStatus` ever reached RUNNING / +k8s ever Ready / compose ever healthy. + +```mermaid +stateDiagram-v2 + [*] --> Starting + Starting --> Running : identity verified & config live + Starting --> Stopped : failed to start / cancelled / reclaimed + Running --> Paused : cordon (director hold) + Paused --> Running : resume + Running --> Unhealthy : liveness / authz lost + Paused --> Unhealthy : liveness / authz lost + Unhealthy --> Running : recovered (re-proves identity) + Unhealthy --> Stopping : give up (graceful) + Unhealthy --> Stopped : hard loss (OOM / crash / node death), no flush + Running --> Stopping : stop / replace (desired=stopped) + Paused --> Stopping : stop / replace + Stopping --> Stopped : state saved + Running --> Stopped : reclaim (hard loss) + Paused --> Stopped : reclaim (hard loss) + Stopped --> [*] +``` + +| State | Discriminator | Definition | The one thing that matters | +|---|---|---|---| +| **Starting** | desired=running ∧ ¬identity_verified | CP provisions an authenticated config and injects it; the agent proves identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. A **per-instance** credential is minted here. | +| **Running** | desired=running ∧ identity_verified ∧ accepting_work ∧ healthy | Alive, authorized, in-sync, and admitting work. | **Only Running admits new work** → dispatch/gate is the single predicate `state == Running`. | +| **Paused** | desired=running ∧ identity_verified ∧ ¬accepting_work ∧ healthy | Healthy and in-sync but deliberately not admitting (director cordon). | Intent, not fault. Resumable; still subject to health edges. Keeping it a peer state is what keeps the dispatch predicate single-field. | +| **Unhealthy** | desired=running ∧ identity_verified ∧ ¬healthy | Alive but fenced: liveness/authz/probe/lease lost. **Not** version skew. | Fenced at once; recover within a window (re-prove identity) or go to Stopping. Split cause: *observed-bad* vs *unobservable* (node lost). | +| **Stopping** | desired=stopped; graceful window open | Terminate committed: flush state and finish in-flight work within a deadline (may still be health-OK). | `desiredStatus==stopped` is the cross-runtime discriminator. Durability was already secured while Running. | +| **Stopped** | terminal (absorbing) | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normative enum: normal / crash / reclaimed). Granularity is **instance-level**. | + +**Attributes, not states** (read alongside the state): `accepting_work` +(Running vs Paused) — its authority is the **CP/director**, never the agent's +self-report; `superseded` / version-skew (a healthy instance whose desired +version has moved on) ⇒ `accepting_work=false`, so it classifies as **Paused** +and is never dispatched new work. *When and in what order* a superseded instance +is drained or replaced is a **fleet-level rollout** concern (e.g. +make-before-break) — out of scope for this instance-level ADR; see the future +rollout / RuntimeDriver ADR. Also: health `cause` = observed-bad vs +unobservable; death `cause` enum; turn-level busy/idle. + +## 4. Principles + +1. **Default-deny identity.** Identity is proven with a control-plane-issued + credential, never accepted from the agent's own claim. The **trust root is + the runtime's injection primitive** (IRSA / k8s projected SA token) that + delegates a platform identity — state it explicitly. **Role identity ≠ + instance identity**: mint a **per-instance** credential at `Starting`. +2. **Trust & sync are continuous.** Heartbeat carries a CP-signed, short-TTL + **lease token bound to the instance id** (task ARN / pod UID). A **monotonic + fencing epoch** guards generations — the CP accepts only the highest epoch, + defeating zombie/split-brain after a partition. Credentials are revoked on + Stopping/Stopped; `Unhealthy→Running` must re-prove identity. +3. **Only `Stopped` is terminal (absorbing), at instance granularity.** A + container restart within the same pod is the *same* instance, not a + `Stopped→Starting` flap; restart = a new lifecycle only when a new instance + is created. +4. **`reclaim` is two paths, not one.** A *planned* interruption (Spot/preempt + notice — ECS ~120s SIGTERM, GKE ~30s + preStop) **compresses `Stopping`** + into a short deadline. Only a *hard* loss (node death / SIGKILL / OOM) jumps + straight to `Stopped`. Durability never relies on the Stopping window — + **checkpoint while Running.** +5. **Runtime-independent.** Each driver projects native signals onto the 6 via + the discriminators `(desiredStatus, accepting_work, health, identity_verified)`; + the machine never changes per runtime. +6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running` + (single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping`(within + deadline) — a cordoned (Paused) agent still finishes its current turn / MCP + call. Don't collapse them into one sentence. + +## 5. Model: config vs observed + +`Instance = Desired Spec (identity + version) + Observed State`. Desired and +observed are strictly separated; **state is observed, not part of the desired +config**. "In sync" (Running) means the reconcile loop has zero diff on the +desired spec. (This replaces the earlier `config = identity + version + state`, +which folded observed state into desired config and could never reconcile to +zero diff.) + +## 6. Runtime Independence (projection) + +Discriminators, not native strings. `desiredStatus==stopped` is one signal +across runtimes: **ECS `desiredStatus STOPPED` ⟺ k8s `deletionTimestamp!=null` +⟺ compose stop-requested** — that is what makes `Stopping` decidable rather than +an ECS-only coincidence. + +| canonical | ECS | k8s / GKE | docker-compose | +|---|---|---|---| +| Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting | +| Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* | +| Paused | RUNNING + health OK + CP/director cordon (`accepting_work=false`) | Ready but cordoned (CP/director) | running + CP/director cordon | +| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (SIGSTOP) → healthcheck stall → Unhealthy | +| Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) | +| Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited | + +**Driver conformance conditions** +- A driver must expose all four discriminators (including the latching + `identity_verified`); if it cannot, it does not conform. +- **docker-compose requires a `healthcheck`** — without one it only sees + running/exited and can never separate Running from Unhealthy. +- **docker-compose must set `restart: "no"`** and hand restart to the control + plane; `restart: unless-stopped` auto-resurrects a crashed container, which + contradicts "Stopped is terminal" and competes with reclaim/replace. + +## 7. Considered Options + +- **6 states with Paused as a peer state (chosen).** Uses the discriminators to + define Paused rigorously; keeps dispatch single-field. +- **5 states, Paused/Draining as a `Running` attribute** (reviewers' converged + proposal) — *rejected as the surface model* because it forces a two-field + dispatch predicate (`Running && accepting_work`); every caller that forgets + `&& accepting_work` silently mis-schedules a paused agent. **We adopt its + `(desiredStatus, accepting_work)` machinery as Paused's definition.** +- **Hermes' 6 operational states verbatim** — rejected: mixes install/service + concerns with runtime state; path/name identity is the self-report we reject. +- **pi `idle/turn` as the primary machine** — rejected: a sub-layer of Running. +- **K8s granular phases** (Pending/Running/Succeeded/Failed/Unknown + container + states) — rejected for the surface set; folded into attributes. +- **Drop `Unhealthy`** — rejected: loses the "alive but fenced" distinction. + +## 8. Prior Art + +| Project | Model | What we take / differ | +|---|---|---| +| **Kubernetes** Pod lifecycle | Phase + Conditions + Probes (three-layer decoupling); `Unknown` on node loss | Direct ancestor; we take the phase/condition/probe split; `Unknown`→Unhealthy(fenced). | +| **HashiCorp Nomad** | alloc states pending/running/complete/failed/**lost**; driver preemption events | `lost`/`unknown` is exactly our *unobservable* Unhealthy case. | +| **Temporal / Cadence** | workflow/activity states + heartbeat **lease fencing** | Validates the fencing epoch on the heartbeat lease. | +| **Erlang/OTP supervisor** | child spec + crash exit reason + `one_for_one`; restart spawns a new child | Supports "restart = new lifecycle / fresh instance". | +| **AWS EC2 instance lifecycle** | pending/running/stopping/stopped/terminated | Near-identical shape; instance-level granularity. | +| **systemd unit** | active / **failed** / … as first-class | `failed` as a first-class fault state. | +| **Ray actor** | PENDING / ALIVE / RESTARTING / DEAD | Close 1:1; `RESTARTING` = our replace path. | +| **Hermes / Pi / Pi-Desktop** | ops CLI states / in-process turn engine / desktop shell | Adjacent code, not instance-level lifecycle. Pi validates **checkpoint-while-Running**. | + +## 9. Consequences + +- The read-model and Studio report **only these 6 states**. +- Every runtime driver must provide a **native→6 projection** via the + discriminators (conformance requirement), including the compose healthcheck + and `restart:"no"` conditions above. +- Detailed sub-states are **attributes** of the 6 (accepting_work, superseded, + health-cause, death-cause enum, busy/idle), not new states. +- **Follow-ups:** a `RuntimeDriver` contract ADR (verbs apply / observe / scale + / cordon / …); an identity / lease / epoch spec ADR. + +## 10. More Information + +Format follows **MADR** (markdown ADR: context → drivers → options → decision → +consequences) with a **Nygard** status/context/decision/consequences spine and a +**Y-statement** summary. See `docs/review-runbook.md` for the review rubric this +ADR was gated on. diff --git a/docs/review-runbook.md b/docs/review-runbook.md new file mode 100644 index 0000000..67f495c --- /dev/null +++ b/docs/review-runbook.md @@ -0,0 +1,59 @@ +# Review Runbook + +How we review ADRs and design docs in this repo. The goal is a **falsifiable** +review — reviewers try to break each load-bearing claim, not nod at it. Peer +"LGTM" carries weight only after the claim has survived an attempt to refute it. + +## The 8 axes + +Every **load-bearing claim** in a doc is scored against all 8. A claim survives +only if it passes **every** axis. + +1. **Simplicity / concise** — minimal surface area; no state/column/sentence that + could be cut. *Fails on:* bloat. +2. **In scope** — decides only what this doc is for; no sprawl (e.g. don't fold a + RuntimeDriver contract or implementation detail into a state-model ADR). +3. **Factcheck** — runtime behaviour and prior-art claims are true, **with a + source**. A claim with no source does not pass. +4. **Refute** — assume the claim is *wrong* and try to prove it (adversarial + default); it survives only if the refutation fails. +5. **Coverage / MECE** — exhaustive and mutually exclusive. Ask "what state / + edge / runtime situation is missing?" and "can one situation fall into two?" + (Distinct from Refute: Refute attacks "what you said is wrong"; Coverage + attacks "you didn't say X".) +6. **Consistency** — sections don't contradict each other (definition ↔ diagram + ↔ projection ↔ principles) and align with the doc's own first-principles. +7. **Decidable / actionable** — the decision is actually made, and an + implementer/driver can act on it without ambiguity. +8. **Reversibility / lock-in** — what this locks in and how expensive it is to + change later. + +## Verdict rule + +- Score each load-bearing claim across all 8 axes. +- **Refute** defaults to *refuted* — a claim is only "survived" once refutation + attempts fail. +- **Factcheck** with no source does not pass. +- Report only the axes a claim **fails**, with the counter-example or source. + Passing axes need no restatement. + +## How to run a refute pass + +1. Enumerate the doc's load-bearing claims (the ones the decision rests on). +2. Assign refuters; each is told to assume the claim is wrong and produce a + counter-example, a missing case, or a contradicting source. +3. A claim survives only if no refuter lands. Surviving-with-fixes → fold the + fix; failed → back to the author. +4. Consolidate into one review comment on the PR; the author decides how to land. + +## References (ADR writing) + +- **Michael Nygard**, *Documenting Architecture Decisions* — the origin; + Status / Context / Decision / Consequences. +- **MADR** — Markdown ADR: context → drivers → considered options → decision + outcome → consequences. +- **adr.github.io** — templates and `adr-tools`. +- **Joel Parker Henderson**, ADR templates & examples collection. + +- **Y-statement** — one-line decision summary: "In context X, facing Y, we + decided Z, to achieve W, accepting V."