Skip to content

supervisor: retain lifetime restart history across operator resets (closes #70) - #77

Merged
ualtinok merged 1 commit into
cortexkit:masterfrom
iceteaSA:feat/lifetime-restarts
Aug 27, 2026
Merged

ualtinok merged 1 commit into
cortexkit:masterfrom
iceteaSA:feat/lifetime-restarts

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Closes #70.

ck module status reports restarts N/M — the live restart budget. Three operator verbs zero that counter:

supervise.rs   restart_count += 1        crash / respawn paths (3 sites)
supervise.rs   reset_restart_count(...)  operator restart · reload · enable (3 sites)

So restarts 0/3 means either "this module has never crashed" or "this module crashed twice and an operator restarted it". Those are opposite operational situations — one is healthy, one has been failing and was papered over — and the status line renders them identically. An instrument that reports the same value for a healthy module and a repeatedly-failing one is not measuring what its label claims.

Change

A second counter, lifetime_restarts, incremented at every site where restart_count is incremented and never reset. reset_restart_count does not touch it.

Rendering shows it only when it diverges from the budget count, so the healthy case stays unchanged:

restarts 0/3               never crashed
restarts 0/3 (2 lifetime)  budget was reset; crashed twice historically

Budget semantics are deliberately untouched. The reset is correct — an operator restart legitimately grants a fresh budget, and set_enabled(true) revival depends on it. This adds an honest second counter; it does not change when a module gives up.

Wire representation

Option<u32> with serde(default, skip_serializing_if = "Option::is_none").

The Option is load-bearing rather than stylistic. My earlier PR #8 used skip_serializing_if = "is_zero" on a bare u32 in this same struct, which collapsed "zero restarts" and "field absent" into one wire state — the absent-vs-empty collapse that issue #12 was filed about, and it was right to be rejected. Here Some(0) ("known: never restarted") stays distinct from None ("old peer, unknown"), so an old payload keeps its honest unknown instead of asserting zero. Existing golden fixtures are unchanged — they serialize None and skip the field.

Tests

Four arms, each proved load-bearing by breaking the seam it guards:

Arm Red proof
crash twice → operator reset → restart_count 0, lifetime_restarts 2 reset mutated to clear lifetime → predicate never satisfied, last state showed lifetime reset
each of the 3 increment sites bumps lifetime health site → left 0 right 1; reload site → left 0 right 1; crash site → lifetime stuck at 0 while restart_count advanced
serde round-trip + old payload → None skip mutated → encoded null instead of 5
render suppressed vs shown render mutated → 0/3 instead of 0/3 (2 lifetime)

The per-site arm is the one worth having: a test that only exercises the common crash path stays green if an increment site is missed, and the counter then under-reports silently forever — which would reproduce the exact defect this fixes.

Gates

cargo test --workspace       769 passed  0 failed  1 ignored   (master baseline 765 + 4 new)
cargo clippy --workspace --all-targets -- -D warnings   PASS
cargo fmt --all --check                                 PASS
cargo build --workspace --locked                        PASS
check-wire-crate-versions.sh origin/master              PASS

Baseline measured independently on 2a0fbb67 rather than taken on trust: 765 passed there, 769 here, difference exactly the four new tests.

Additive wire change — subc-control 0.7.0 with the dependent cascade.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes ck module status so the restart budget no longer hides repeated crashes behind an operator reset, distinguishing "never crashed" from "crashed twice, then reset" (closes #70).

  • Adds lifetime_restarts, incremented wherever restart_count increments but never reset, so reset history renders as 0/3 (2 lifetime) while a fresh budget keeps the plain 0/3.
  • Leaves budget reset semantics unchanged; an operator restart legitimately grants a fresh budget, and the lifetime counter does not alter when a module gives up.
  • Adds tests covering all three increment sites, the wire round-trip (old peers stay None), and both render branches so a missed site can't silently under-report.

Wire change

  • SupervisorEntry gains optional lifetime_restarts; Some(0) stays distinct from None on old peers, so unknown history stays honest.
  • Bumps subc-control to 0.7.0, cascading to subc-client-rs 0.8.1 and subc-core 0.8.2.

Written for commit f8ce018. Summary will update on new commits.

Review in cubic

@iceteaSA

Copy link
Copy Markdown
Collaborator Author

Version collision with #76 — flagging before you hit it, not after.

Both open PRs bump subc-control from 0.6.0 to the same 0.7.0:

#76 (probe pid identity)   subc-control 0.6.0 -> 0.7.0 · subc-core 0.8.1 -> 0.9.0 · subc-client-rs -> 0.9.0
#77 (this)                 subc-control 0.6.0 -> 0.7.0 · subc-core 0.8.1 -> 0.8.2 · subc-client-rs -> 0.8.1

Each is correct in isolation — both add an additive public item to subc-control, so both earn a minor bump from the same base. Whichever merges second will conflict in Cargo.toml and Cargo.lock, and the conflict is not purely textual: the second one needs its bump re-derived from the new base (0.7.0 → 0.8.0), not just re-applied.

They are otherwise independent — #76 touches provenance.rs plus a control.rs import, #77 touches supervise.rs and the SupervisorEntry struct. No logical dependency in either direction.

I will rebase whichever loses and re-cascade the versions; say the word or just merge in whatever order you prefer and I will follow. I would rather do that than have you resolve a version cascade in a merge editor.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 10 files

Re-trigger cubic

@iceteaSA

Copy link
Copy Markdown
Collaborator Author

Rebased onto 7ce6ef51 and re-gated — green:

cargo test --workspace       770 passed  0 failed
clippy --all-targets -D warnings · fmt · build --locked · check-wire-crate-versions   PASS

Head 3e7af9cb. Master's baseline is 766 on this commit; the +4 is this branch's four new arms.

Collision with #76 unchanged — both still bump subc-control to 0.7.0 from the same base. Merge in whatever order suits; I rebase the loser and re-derive the bump from the new base rather than re-applying it.

@subc-alfonso subc-alfonso Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by execution: the reset asymmetry is the whole feature and it holds — all three restart_count increment sites increment lifetime_restarts beside it, and the single operator reset site (set_enabled) touches restart_count alone, verified by reading every assignment site on the branch. The status split (spent-budget vs lifetime) closes #70's never-crashed vs crashed-then-reset ambiguity exactly as ruled. Ten fleet_lint failures during my local gate were proven ENVIRONMENTAL (identical on master in the same twin — fresh-stub first-exec assessment, the documented macOS class; clean rerun after warm-exec: 452/0). Full matrix green on feat/lifetime-restarts at the rebased head. Merging.

The budget display restarts 0/3 currently describes both a never-crashed module and one that crashed twice before an operator reset. Add a monotonic lifetime count so those states remain distinguishable, while deliberately leaving restart_count reset semantics unchanged because operator restart and re-enable are budget grants.\n\nCONSUMER-IMPACT: subc-control 0.7.0 adds optional lifetime_restarts; subc-client-rs 0.8.1 cascades the wire dependency. The field is serde-defaulted and omitted when unknown, so old payloads decode cleanly and existing absent fixtures remain byte-stable.
@ualtinok
ualtinok force-pushed the feat/lifetime-restarts branch from 3e7af9c to f8ce018 Compare August 27, 2026 22:15
@ualtinok
ualtinok merged commit 614f12f into cortexkit:master Aug 27, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

restart_count is reset by operator verbs, so restarts 0/3 cannot distinguish "never crashed" from "crashed twice, then restarted"

2 participants