[mattmattox]

Follow-up to the cluster-services-831 postmortem (a1ubr720p02's root disk went read-only and kubelet died). This enables the kubelet monitor and an automatic remediation that is safe to leave on: at most one rke2-agent restart per incident, only when the disk write probe (#59) passed recently, and never on a node where rke2-agent is not the enabled unit. IPMI power-cycle and node reboot are not configured anywhere.

Why the existing remediation path could not be used as-is

1. Inverted trigger. evaluateRemediation remediates every condition whose status is False. The kubelet monitor reports its healthy state as KubeletDown=False and KubeletUnhealthy=False on every passing check, so enabling remediation would have restarted on every healthy node. TestRemediationPolicy_LegacyRuleWouldRestartHealthyKubelet pins this.
2. Silent no-op restart. In the DaemonSet, the image's systemctl cannot reach the host systemd. It prints Running in chroot, ignoring command and exits 0, so the restart did nothing and was recorded as a success. I verified this live on a1-ops-prd.
3. Dead alert. NodeDoctorKubeletUnhealthy matched condition_type="KubeletHealthy" == 0, which the monitor never emits.

Changes

• Dispatch policy on a monitor's remediation block. All new fields are opt-in; when triggerConditions is unset the legacy rule is unchanged.
  • triggerConditions: problem-polarity trigger. Only the named conditions count, and only when they are True.
  • maxAttemptsPerIncident: the attempt is recorded before dispatch, so a failed or hung restart still counts and is not retried. The count resets when the condition clears.
  • cooldown: minimum time between dispatches, across incidents.
  • preconditions: checked against the latest observed condition, with an optional monitor scope and a required maxAge. A withheld dispatch does not use up the attempt.
• systemd-restart changes.
  • requireServiceEnabled refuses unless systemctl is-enabled reports enabled. On RKE2 server nodes rke2-agent is installed but disabled, and restarting it would start a second kubelet.
  • When PID 1 is systemd and our mount namespace differs from PID 1's (hostPID container), systemctl runs through nsenter --target 1 --mount --uts --ipc --net --pid.
  • The chroot notice is always treated as an error.
• Chart.
  • New opt-in monitors.kubelet. It checks healthz only: kubelet is not a systemd unit on RKE2, and PLEG needs authenticated access to the metrics port.
  • New monitors.kubelet.remediation with defaults rke2-agent, 1 restart per incident, 30m cooldown, write probe passed within 3m, and requireServiceEnabled.
  • The render fails if the write-probe gate is on but the write probe is disabled.
• NodeDoctorKubeletUnhealthy now matches KubeletUnhealthy == 1.

Tests

• Detector policy unit tests:

  • healthy kubelet → no dispatch;
  • 1 dispatch per incident;
  • a failed restart is not retried;
  • read-only, stale, missing or wrong-monitor write probe → withheld, and the attempt is kept;
  • cooldown applies across incidents;
  • recovery resets the incident.

  I mutation-checked them: disabling each guard or inverting the trigger fails at least one test.

• systemd-restart tests: the is-enabled guard (enabled, disabled, masked, static, not-found, and dry-run), chroot detection, and the nsenter decision against a fake /proc.

• Config validation tests.

• End-to-end (test/integration/remediation/kubelet_restart_policy_test.go). The chain is helm template with the kubelet remediation on → util.LoadConfig → ProblemDetector → RemediatorRegistry → SystemdRemediator → a recording fake systemctl. Results:

  • agent node: exactly is-enabled rke2-agent, then restart rke2-agent, once;
  • read-only disk: no systemctl calls;
  • server node: is-enabled only, refused.

  Two chart checks: the default render has no kubelet monitor, and the render fails when the write probe is off.

Live read-only checks on a1-ops-prd (no restarts):

• 127.0.0.1:10248/healthz answers 200 ok from all 13 node-doctor pods.
• nsenter ... systemctl is-enabled reaches the host systemd: rke2-agent is enabled on the 10 workers and disabled on a1ubmgmtp01-03.

Refs: cluster-services-831