[mattmattox]

On 2026-10-06 (cluster-services-831 postmortem) a1ubr720p02's boot RAID virtual disk dropped off the PERC controller. Writes to / returned EROFS/EIO and kubelet died. node-doctor kept reporting DiskHealthy=True for three reasons:

1. The statfs ST_RDONLY check cannot see an ext4 filesystem that shut down after errors. Since Linux 6.7 that case no longer sets SB_RDONLY, and the nodes run kernel 7.0.
2. The check statted the container's own /, not the host's.
3. The chart's NodeDoctorReadOnlyFilesystem rule matched condition_type="ReadOnlyFilesystem", which no monitor emits. The live label values confirm it: ReadonlyFilesystem never appears, and ReadOnlyFilesystem doesn't exist. The rule could never fire.

Changes

• system-disk write probe. New settings writeProbePaths and writeProbeTimeout (default 10s). On every check the monitor creates, writes, fsyncs and unlinks a file in each path.
  • An error or a timeout sets ReadonlyFilesystem=True with reason WriteProbeFailed and emits an event.
  • While a probe is still blocked, no new probe starts, so a dead device that hangs writes cannot pile up goroutines.
  • All paths produce one combined condition, and a probe never reports False over a True from the statfs check.
• Chart. New monitors.disk.writeProbe setting, enabled by default. It mounts the host's /var/lib/node-doctor/write-probe (DirectoryOrCreate) read-write at /host-write-probe and configures the probe. The rest of the host root stays read-only.
• Chart alert fix. NodeDoctorReadOnlyFilesystem now matches ReadonlyFilesystem.

Tests

disk_write_probe_test.go covers:

• a real temp dir, with no files left behind;
• an EIO on one of two paths fires;
• a hang counts as a failure and no second probe starts;
• no False over a statfs True;
• a cancelled check gives no verdict;
• config parsing and defaults.

Mutation-checked: removing the in-flight guard or the failure branch makes the tests fail. go test -race ./pkg/monitors/system/ and go test ./pkg/... pass. helm lint passes, and the render was checked with the probe enabled and disabled.

This change is independent of #58 (condition ownership). With #58 merged, the node condition is written as NodeDoctorReadonlyFilesystem.

Refs: cluster-services-831