Question
How are Kubernetes LoadBalancer health checks performed exactly?
If you create an external load balancer in Kubernetes (i.e. you create a Service of type: LoadBalancer), what requests are sent by the external load balancer to the nodes?
What port is used on the nodes for the health checks?
What process responds to those health check requests?
What checks are performed by that process on the node before returning 200 OK or 503 Service Unavailable to the external load balancer?
Answer
Let’s start from the basics. Without Kubernetes, it is very common to have a reverse proxy (like HAproxy) in front of some VMs where the web server (like nginx or Puma) is running. The application (like a Rails app) then exposes an endpoint (like /up) which returns the status. Usually the endpoint just returns 200 OK and if the external load balancer can see that, it means that the app is up and running. So the health checks are sent from the external load balancer to the normal port of the web server that also serves normal HTTP traffic and user requests.
If we move to Kubernetes, things may be surprising and become much more complex.
In Kubernetes the port used by default for health checks is a separate port, used only for health checks, and different from the port that is serving the HTTP traffic.
kube-proxy + externalTrafficPolicy: Cluster
In a standard Kubernetes configuration with kube-proxy (the component that manages the internal Kubernetes networking) and externalTrafficPolicy: Cluster (default), the port used for health checks is node:10256. The endpoint returns 200 OK if kube-proxy configured the IP tables correctly on the node. This is a very basic check that doesn’t say much about the node health in general.
For example, if you have a mixed configuration that uses both kube-proxy and cilium for different networking aspects (which is possible), then kube-proxy health endpoint may return 200 OK even if cilium and CNI is completely down on the node.
In a scenario where externalTrafficPolicy: Cluster is used, it makes sense to send requests to a port dedicated to health checks (10256), instead of sending the requests to the normal HTTP port, because the response to normal HTTP traffic can come from a pod running on another node, so it wouldn’t tell much about the current node status.
kube-proxy + externalTrafficPolicy: Local
A different situation is when you use externalTrafficPolicy: Local.
Kubernetes allocates a separate healthCheckNodePort for the external load balancer to probe. There is no single fixed health-check port in this case (by default Kubernetes allocates one from the NodePort range, which is 30000-32767).
This kube-proxy endpoint returns 200 OK only when some pods exposed by the Service are Ready on the node.
Note that this is only eventually consistent. In this case, kube-proxy returns the status (200 or 503) based on the pod status reported the control plane API. So it’s not in real time, it reads a cached value. It doesn’t send a request to the pods for each load balancer health check request. The status reported by the control plane for a pod is periodically updated with the information collected by the kubelet on the node.
Cilium
Cilium can replace kube-proxy partially or entirely.
When Cilium handles CNI/networking, while kube-proxy still implements Kubernetes Service routing, remember that the kube-proxy endpoint for health checks (10256) is not aware about Cilium/CNI status on the node, and may return successful status codes even when CNI is down. This can happen for example in some DigitalOcean Kubernetes clusters, created a few years ago, which were only partially migrated to Cilium (and still use kube-proxy for some functions).
Another solution is to use only Ciulium and drop kube-proxy completely. For example, the new DigitalOcean Kubernetes clusters use this (Cilium + eBPF). In this case there is a kubeProxyReplacement mode, that allows Cilium to replace kube-proxy on all the health check endpoints.
In this scenario, when you have externalTrafficPolicy: Cluster, Cilium responds to health-check requests on port 10256. It responds with 200 OK when the internal function GetStatus reports that Cilium is initialized properly and the node is not shutting down. Otherwise it returns 503. If Cilium is completely down, no response is expected (obviusly) and the external load balancer can detect the issue when the health check requests reach the timeout.
When you have externalTrafficPolicy: Local and kubeProxyReplacement mode enabled, Cilium behaves exactly like kube-proxy and returns 200 OK only when some pods (at least 1) of the exposed Service are Ready on the current node.
This article is based on real-world experience and technical investigations that we performed for our service Pushpad (a scalable service for web push notifications), which runs entirely on Cuber (an open-source automation tool for deploying web applications on Kubernetes, that we have built from scratch and used successfully in production for many years).