Repository navigation
fix(exporter): never write kubelet-owned node conditions - #58
Merged
Merged
Conversation
The system-disk and system-memory monitors emit DiskPressure and MemoryPressure. convertConditionToNodeCondition exempted the Kubernetes built-in names from the NodeDoctor prefix, so node-doctor wrote kubelet's own DiskPressure/MemoryPressure conditions (reason NormalDisk/NormalMemory, lastHeartbeatTime null), fighting kubelet as a second writer; a True value becomes a NoSchedule taint at node-doctor's thresholds via TaintNodesByCondition. - Always prefix: DiskPressure -> NodeDoctorDiskPressure, MemoryPressure -> NodeDoctorMemoryPressure (idempotent for already-prefixed types). - flushPendingUpdates only patches NodeDoctor* conditions. It used to re-send every cached condition, including built-ins loaded at resync, which can write a stale Ready=True over the node-lifecycle controller's Ready=Unknown. - Keep lastTransitionTime while the status is unchanged; stamp lastHeartbeatTime on write. Refs: cluster-services-874
mattmattox
added a commit
that referenced
this pull request
Oct 14, 2026
…exporter fixture (#61) TestConditionManager_Resync was flaky since #58: the fixture's custom condition "TestCustomCondition" is unprefixed, so the exporter now refuses to write it to the node, but a flush still copies it into the local cache. Whether GetConditions ran before or after that flush decided whether the resync count dropped (3 -> 2) and failed the test. With a NodeDoctor* name the condition is really written, so resync keeps it. Before: 13/40 failures (go test -race -count=40 -run TestConditionManager_Resync). After: 0/40.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
node-doctor wrote the kubelet-owned DiskPressure/MemoryPressure node conditions (reason NormalDisk/NormalMemory, null lastHeartbeatTime, lastTransitionTime reset on every write). The cause is
isStandardNodeCondition, which exempted built-in names from theNodeDoctorprefix. The system-disk and system-memory monitors emit exactly those names. If node-doctor writes True, TaintNodesByCondition turns it into anode.kubernetes.io/*-pressureNoSchedule taint at node-doctor's own thresholds.Changes:
NodeDoctorDiskPressure,NodeDoctorMemoryPressure. Already-prefixed types are left unchanged.flushPendingUpdatesonly patchesNodeDoctor*conditions. Before this, it re-sent every cached condition, including the built-ins loaded at resync. That can write a staleReady=Trueover the node-lifecycle controller'sReady=Unknownon a node whose kubelet died.lastTransitionTimeis kept while the status is unchanged, andlastHeartbeatTimeis stamped on every write.node_doctor_monitor_condition_status{condition_type=...}and the chart's PrometheusRule are unchanged.Tests:
condition_ownership_test.gocovers kubelet MemoryPressure left untouched, no bare DiskPressure created, stale Ready not written back, and transition time preserved. The prefixing table test was updated. Mutation-checked: reverting each guard makes its test fail.go test ./pkg/...passes.After rollout, the bare DiskPressure/MemoryPressure conditions on each node keep node-doctor's last value until kubelet's next status report rewrites them. Kubelet reports at least every 5 minutes, so nothing has to be cleaned up by hand.
Refs: cluster-services-874