1. Documentation
×

About node maintenance

About node maintenance mode

Nodes can be placed into maintenance mode by using the oc adm utility or NodeMaintenance custom resources (CRs).

The node-maintenance-operator (NMO) is no longer shipped with OKD Virtualization. It is deployed as a standalone Operator from the OperatorHub in the OKD web console or by using the OpenShift CLI (oc).

For more information on remediation, fencing, and maintaining nodes, see the Workload Availability for Red Hat OpenShift documentation.

Placing a node into maintenance marks the node as unschedulable and drains all the virtual machines and pods from it. Virtual machine instances that have a LiveMigrate eviction strategy are live migrated to another node without loss of service. This eviction strategy is configured by default in virtual machine created from common templates but must be configured manually for custom virtual machines.

Virtual machine instances without an eviction strategy are shut down. Virtual machines with a runStrategy of Running or RerunOnFailure are recreated on another node. Virtual machines with a runStrategy of Manual are not automatically restarted.

Virtual machines must have a persistent volume claim (PVC) with a shared ReadWriteMany (RWX) access mode to be live migrated.

The Node Maintenance Operator watches for new or deleted NodeMaintenance CRs. When a new NodeMaintenance CR is detected, no new workloads are scheduled and the node is cordoned off from the rest of the cluster. All pods that can be evicted are evicted from the node. When a NodeMaintenance CR is deleted, the node that is referenced in the CR is made available for new workloads.

Using a NodeMaintenance CR for node maintenance tasks achieves the same results as the oc adm cordon and oc adm drain commands using standard OKD custom resource processing.

About run strategies for virtual machines

Run strategies for virtual machines (VMs) determine how virtual machine instances (VMIs) behave under certain conditions.

You configure a run strategy by assigning a value to the runStrategy key in the VirtualMachine manifest as in the following example:

Example run strategy
apiVersion: kubevirt.io/v1
kind: VirtualMachine
spec:
  runStrategy: Always
  template:
# ...

The runStrategy and the running keys are mutually exclusive. Only one of them can be used.

The runStrategy key gives you more flexibility because it has four values, unlike the running key, which has a Boolean value.

runStrategy key values
Always

The VMI is always present when a virtual machine is created. A new VMI is created if the original stops for any reason. This is the same behavior as running: true.

RerunOnFailure

The VMI is re-created if the previous instance fails. The instance is not re-created if the virtual machine stops successfully, such as when it is shut down.

Manual

You control the VMI state manually with the start, stop, and restart virtctl client commands.

Halted

No VMI is present when a virtual machine is created. This is the same behavior as running: false.

Different combinations of the start, stop and restart virtctl commands affect the run strategy.

The following table describes a VM’s transition from different states. The first column shows the VM’s initial run strategy. The remaining columns show a virtctl command and the new run strategy after that command is run.

Table 1. Run strategy before and after virtctl commands
Initial run strategy Start Stop Restart

Always

-

Halted

Always

RerunOnFailure

-

Halted

RerunOnFailure

Manual

Manual

Manual

Manual

Halted

Always

-

-

If a node in a cluster installed by using installer-provisioned infrastructure fails the machine health check and is unavailable, VMs with runStrategy: Always or runStrategy: RerunOnFailure are rescheduled on a new node.

Maintaining bare metal nodes

When you deploy OKD on bare metal infrastructure, there are additional considerations that must be taken into account compared to deploying on cloud infrastructure. Unlike in cloud environments where the cluster nodes are considered ephemeral, re-provisioning a bare metal node requires significantly more time and effort for maintenance tasks.

When a bare metal node fails, for example, if a fatal kernel error happens or a NIC card hardware failure occurs, workloads on the failed node need to be restarted elsewhere else on the cluster while the problem node is repaired or replaced. Node maintenance mode allows cluster administrators to gracefully power down nodes, moving workloads to other parts of the cluster and ensuring workloads do not get interrupted. Detailed progress and node status details are provided during maintenance.

Additional resources