If the cluster administrator has performed latency tests for platform verification, they can discover the need to adjust the operation of the cluster to ensure stability in cases of high latency. The cluster administrator need change only one parameter, recorded in a file, which controls four parameters affecting how supervisory processes read status and interpret the health of the cluster. Changing only the one parameter provides cluster tuning in an easy, supportable manner.
Kubelet process provides the starting point for monitoring cluster health. The
Kubelet sets status values for all nodes in the OKD cluster. The Kubernetes Controller Manager (
kube controller) reads the status values every 10 seconds, by default.
kube controller cannot read a node status value, it loses contact with that node after a configured period. The default behavior is:
The node controller on the control plane updates the node health to
Unhealthy and marks the node
In response, the scheduler stops scheduling pods to that node.
The Node Lifecycle Controller adds a
node.kubernetes.io/unreachable taint with a
NoExecute effect to the node and schedules any pods on the node for eviction after five minutes, by default.
This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager might not receive an update from a healthy node due to network latency. The
Kubelet evicts pods from the node even though the node is healthy.
To avoid this problem, you can use worker latency profiles to adjust the frequency that the
Kubelet and the Kubernetes Controller Manager wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly if network latency between the control plane and the worker nodes is not optimal.
These worker latency profiles contain three sets of parameters that are pre-defined with carefully tuned values to control the reaction of the cluster to increased latency. No need to experimentally find the best values manually.
You can configure worker latency profiles when installing a cluster or at any time you notice increased latency in your cluster network.