[chart/redis-ha][Make per-pod announce Services headless to avoid Sentinel CPU loops after pod replacement] #422

Describe the bug

The chart creates one ClusterIP announce Service per Redis/Valkey pod. Redis and Sentinel peers therefore connect through kube-proxy/conntrack instead of directly to pod IPs.

After replacing a pod, we observed a surviving Sentinel consume an entire CPU core indefinitely.

To Reproduce

1. Install a three-node release with ClusterIP announce Services:

replicas: 3

sentinel:
  resolveHostnames: true
  announceHostnames: true

2. Confirm each announce Service has a ClusterIP.
3. Record baseline Sentinel CPU
4. Repeatedly delete replica pods, waiting for recovery after each deletion:

for i in $(seq 1 50); do
  ordinal=$((i % 2 + 1))
  pod="<release>-server-${ordinal}"

  kubectl delete pod "$pod"
  kubectl wait --for=condition=Ready "pod/$pod" --timeout=10m
done

5. Check Sentinel CPU after each replacement:
6. When a Sentinel pins a core, attach strace to its process:

strace -f -p 1 -e trace=epoll_wait,recvfrom

Expected behavior

Sentinel should recover peer connectivity after pod replacement without entering a high-CPU event loop.

Actual behavior

A surviving Sentinel eventually consumes approximately one CPU core indefinitely.

The trace shows a tight event-loop cycle on a socket connected to a peer’s announce Service:

epoll_wait(... [{events=EPOLLERR, data=...}] ...) = 1
recvfrom(..., ...) = -1 EAGAIN

We observed approximately 193,000 epoll_wait and 194,000 recvfrom calls in 15 seconds.

This reproduced with both Valkey 8.0 and Redis 8.2 in Kubernetes, but not with equivalent local container churn.