[maksymhryzodub-prog]

Ticket: CLEAN-139 · spec, plan and tasks in specs/020-agent-event-notifications/

Draft: the screens still need a walk-through in a browser — see "Not verified".

What it does

The team must find out when an agent goes down. Ranch already detected most failures and told nobody; an outside sender would be a second witness and had nowhere to post.

• POST /agent-events — cluster tooling reports an agent failure. Authorised by an API key with one new scope, events:write, that can do nothing else. Body: agentId, status (failed | recovered), optional datetime, reason, source, eventId. The sender's guide is docs/operations/agent-events.md.
• Ranch's own failures become events too. AgentGateway.updateStatus now announces a real transition exactly once — whatever the replica count — through a new AgentStatusChanges port; transitions to failed / unreachable are recorded with Ranch as the witness.
• One incident per agent, one message when it opens and one when it closes. "At most one open per agent" is a unique column, not a check in code. An incident closes when Ranch has seen the agent running for ten quiet minutes; silently when a person stops or deletes the agent.
• An outside event never writes an agent's status, and one about a stopped or restarting agent is recorded and notifies nobody — an outside sender cannot tell a restart from a crash.
• Slack through an incoming webhook, behind an outbox with retries. The message that an agent went down mentions the whole channel; the one that it is back mentions nobody; a sender's own <!channel> is escaped. The address lives in its own table and is never returned by any route or tool (GET /settings hands secret values back to the console today, so it is not a setting).
• Console: an Events page, an Events section on the agent page, Settings → Notifications (set / replace / remove, send a test, what a sender has to send), events:write in the key dialog.
• Agent tools: list_agent_events, list_agent_incidents, get_ / set_ / remove_notification_destination, send_test_notification.

One additive migration, four tables. No new dependency.

Verified

• cd api && bun run build
• cd api && NODE_OPTIONS=--experimental-vm-modules npx jest src/slices/agent/event src/slices/agent/agent src/slices/user/apiKey src/slices/mcp — 34 suites, 428 tests
• cd admin && bun test slices — 171 tests · cd admin && npx nuxt typecheck · bun run locale:check
• Against a real Postgres, with two API processes on it (fresh database, all 51 migrations applied, prisma migrate diff reports no difference from the schema): every answer of the endpoint (201 / 200 duplicate / 400 / 401 / 403 / 429 with Retry-After), the key refused on every other route tried, suppressed_stopped and suppressed_starting, the same failure posted to both processes at once and ten at once (one incident, one opening message), cursor paging, roles on the console routes, non-Slack and in-cluster addresses refused. Ranch's own watch marking pod-less agents failed produced exactly one event per transition with both processes sweeping; a stopped agent's incident closed silently; an incident aged past the quiet period closed with exactly one closing message.
• Real delivery to Slack (a test channel, same local stack): the destination set through the API, the test message, an outside failure → opening message 1.5 s after the event was accepted, five more failures → no further message, the aged incident → "No further reports", Ranch's own watch → "Reported by: Ranch", the agent back and aged → "Agent back … Down for 15 min", and a failure carrying the channel mention. Sender-supplied <…> and & arrived as plain characters.
• The flood limit under concurrency: 200 parallel posts from one key against a running API → 60 accepted, 140 answered 429.
• Every new and changed .vue compiles in the admin dev server.

Not verified

• The screens were not walked through in a browser. They typecheck and compile; no scenario was clicked through.
• What the notifier does with Slack's failure answers (429, 404, a timeout) is covered by specs with a mocked fetch only.
• The Rancher agent was not started to call the new tools (the API validated all 163 tools at boot).
• The ten-minute close was reached by ageing a row, not by waiting with a running pod.

Twin consoles

admin slices with a twin that were touched: agent (one section in sections.ts, one branch in Canvas.vue), user (the events:write scope), common (formatSpan, one sidebar icon). Checked in app/: app/slices/agent has no section list (its workspace is Provider, Rail, RailItem), app/slices/user has no API-key screen, and app shows no time span and has its own sidebar. Events are an operator's concern — a customer is not told that their agent went down — so app needed nothing. app/ is untouched.

Worth a reviewer's eye

• api/src/slices/agent/agent/data/agent.gateway.ts — updateStatus is the most carefully guarded write in the API. It now does a conditional updateMany first and falls back to the old update when the status did not change; what is written is the same. The existing agent-slice specs pass untouched.
• Retries wait the schedule's gap after each failed attempt, not from when the message was queued — a test caught that a message picked up late would otherwise fire all its retries back to back.

Open points that are not this PR's to close

• How many API replicas production runs. k8s/platform/api says 2 with HPA 2–5, k8s/deploy/30-api.yaml says 1, and the bridle hub's presence map is per process. This feature is correct at any count; but with more than one replica agent statuses can flap today and each flap would become an incident. Check before a destination is set (see "Before switching notifications on" in the guide).
• ADMIN_URL on the API deployment — without it messages carry no link.
• Which tool DevOps installs as the sender; "Ranch is down" → Slack from Argo CD directly.

🤖 Generated with Claude Code