Configuring higher VM workload density - Postinstallation configuration | Virtualization

Swap provisioning methods
Swap space requirements
Provision swap on an NVMe device
Provision swap by using a file
Remove swap from an NVMe device
Enable memory overcommitment and provision swap
Disable memory overcommitment and remove swap
Additional resources

You can increase the number of virtual machines (VMs) on nodes by overcommitting memory (RAM). You overcommit memory by provisioning swap storage on worker nodes. This is useful when you have many similar or underused workloads.

Memory overcommitment can lower workload performance on a highly utilized system.

Swap provisioning methods

You can provision swap storage on worker nodes by using either an NVMe partition or a file-based approach. The method you choose depends on your hardware configuration and performance requirements.

NVMe partition-based swap requires a dedicated NVMe device on each worker node and provides better performance for production workloads with heavy memory overcommitment. This method is permanent and requires repartitioning to remove.

File-based swap uses existing filesystem space and works on any worker node without dedicated hardware. This method has slower I/O performance than partition-based swap but is easier to provision and remove. Use this method for testing, development, or production clusters where dedicated NVMe devices are not available.

Swap space requirements

The swap space that you provision on a node must be at least equal to the overcommitted RAM. This requirement applies to both NVMe partition-based swap and file-based swap.

Calculate the amount of swap space to provision on a node by using the following formula:

NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

Example:

NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
               = 16 GB * (1.5 - 1)
               = 16 GB * (0.5)
               =  8 GB

Provision swap on an NVMe device

You can provision a dedicated NVMe device for swap. The virt-platform-autopilot controller automatically enables a swap partition labeled OCPSWAP for virtualization workloads. Use this method for production clusters requiring better performance.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged in to the cluster with the cluster-admin role.
You have installed OKD Virtualization. The virt-platform-autopilot controller is enabled by default.
The worker node has a dedicated NVMe device.
The NVMe device has enough capacity for the node’s worst-case memory overcommitment.

Procedure

Select a worker node where you want to enable memory overcommitment and start a debug session by running the following command:
```
$ oc debug node/<node_name>
```
Replace <node_name> with the name of the worker node.
Enter the host context by running the following command:
```
$ chroot /host
```

List the available NVMe disks and their World Wide Names (WWNs) by running the following command:

$ lsblk -p -o +WWN

Example output

NAME         TYPE MODEL                  SERIAL            REV TRAN   RQ-SIZE  MQ WWN
/dev/nvme0n1 disk KXG8AZNV512G LA KIOXIA 63QFXXXXXXEU 5108APLA nvme      1023  12 eui.8ce38e0500e775a4

Find the device path by using its WWN. The path follows the /dev/disk/by-id/nvme-<WWN> format. Run the following command, replacing <WWN> with the value from the previous command:
```
$ ls /dev/disk/by-id/nvme-<WWN>
```
Example output
```
/dev/disk/by-id/nvme-eui.8ce38e0500e775a4
```

Create a GUID Partition Table (GPT) partition table and an OCPSWAP partition by running the following command, replacing <WWN> with the WWN of the selected device:

The following command erases all data on the selected NVMe device. Back up any data that you need to retain before you continue.

$ echo -e 'label: gpt\nname="OCPSWAP", type=swap' | sfdisk --wipe always /dev/disk/by-id/nvme-<WWN>

The partition name must be OCPSWAP so that the virt-platform-autopilot controller can discover and enable the partition.

Only one partition on the node can use the OCPSWAP label. If more than one partition uses this label, the /dev/disk/by-partlabel/OCPSWAP path does not resolve deterministically, and later commands can format or enable the wrong partition.

Format the partition as Linux swap and set its filesystem label to OCPSWAP by running the following command:
```
$ mkswap -L OCPSWAP /dev/disk/by-partlabel/OCPSWAP
```
The /dev/disk/by-partlabel/OCPSWAP path resolves to the partition that you created in the previous step. The -L OCPSWAP option sets the swap filesystem label, which the virt-platform-autopilot controller uses to identify the swap partition. You must format the partition, not the whole NVMe device.

Verify the swap partition before rebooting the node by running the following commands:

$ swapon /dev/disk/by-partlabel/OCPSWAP
$ free -m

Example output

               total        used        free      shared  buff/cache   available
Mem:           31728       20964         906        4586       12269       10764
Swap:           8191          95        8096

Reboot the node by running the following command:
```
$ systemctl reboot
```

Verification

After the node reboots, start a debug session by running the following command:
```
$ oc debug node/<node_name>
```
Enter the host context by running the following command:
```
$ chroot /host
```
Verify that the virt-platform-autopilot controller discovered the swap partition by running the following command:
```
$ ls /dev/disk/by-partlabel/OCPSWAP
```

Verify that swap space is available by running the following command:

$ free -m

Example output

               total        used        free      shared  buff/cache   available
Mem:           31728       20964         906        4586       12269       10764
Swap:           8191          95        8096

Provision swap by using a file

You can provision swap by creating a file on the worker node. The MachineConfig object that you create provisions the swap file and enables it for virtualization workloads. Use this method when dedicated NVMe devices are not available.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged in to the cluster with the cluster-admin role.
The node belongs to a worker pool.

Procedure

Create a MachineConfig file with the parameters shown in the following example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 90-worker-swap
spec:
  config:
    ignition:
      version: 3.5.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Provision and enable swap
            ConditionFirstBoot=no
            ConditionPathExists=!/var/tmp/swapfile

            [Service]
            Type=oneshot
            Environment=SWAP_SIZE_MB=5000
            ExecStart=/bin/sh -c "sudo fallocate -l ${SWAP_SIZE_MB}M /var/tmp/swapfile && \
            sudo chmod 600 /var/tmp/swapfile && \
            sudo mkswap /var/tmp/swapfile && \
            sudo swapon /var/tmp/swapfile && \
            free -h"

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: swap-provision.service
        - contents: |
            [Unit]
            Description=Enable swap
            ConditionFirstBoot=no
            ConditionPathExists=/var/tmp/swapfile

            [Service]
            Type=oneshot
            ExecStart=/bin/sh -c "sudo swapon /var/tmp/swapfile"

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: swap-enable.service

Set the SWAP_SIZE_MB value to the amount of swap space to provision on the node, in MB.

Wait for the worker nodes to sync with the new configuration by running the following command:
```
$ oc wait mcp worker --for condition=Updated=True --timeout=-1s
```

Verification

View a list of worker nodes by running the following command:
```
$ oc get nodes -l node-role.kubernetes.io/worker
```

Select a node from the list and display its memory usage by running the following command:

$ oc debug node/<selected_node> -- free -m

Replace <selected_node> with the node name.

The Swap: row displays an amount greater than zero if the node has swap enabled.

Example output

               total        used        free      shared  buff/cache   available
Mem:           31846       23155        1044        6014       14483        8690
Swap:           8191        2337        5854

Remove swap from an NVMe device

You can remove a swap partition from an NVMe device, freeing the device for other use. The node must be placed in maintenance mode before you remove the swap partition.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged in to the cluster with the cluster-admin role.
The worker node has an NVMe swap partition labeled OCPSWAP.

Procedure

Place the node in maintenance mode by using the oc adm cordon and oc adm drain commands.
Verify that all pod and VM workloads are evicted from the node by running the following commands:
```
$ oc get pods --all-namespaces --field-selector spec.nodeName=<node_name>
```
```
$ oc get vmi -A --field-selector spec.nodeName=<node_name>
```
Only DaemonSet pods should remain on the node.
Start a debug session by running the following command:
```
$ oc debug node/<node_name>
```
Replace <node_name> with the name of the worker node.
Enter the host context by running the following command:
```
$ chroot /host
```
Disable the swap partition by running the following command:
```
$ swapoff /dev/disk/by-partlabel/OCPSWAP
```

Verify that swap is disabled by running the following command:

$ free -m

Example output

               total        used        free      shared  buff/cache   available
Mem:           31728       20964         906        4586       12269       10764
Swap:              0           0           0

The Swap: row displays zero values when swap is disabled.

If the Swap: row shows non-zero values, the swap partition is still active. Verify that all workloads are evacuated and retry the swapoff command. If the issue persists, reboot the node to force swap deactivation.

Remove the swap partition:
1. Set the $DISK variable to the disk device path by running the following command:
  $ DISK="/dev/$(lsblk -no PKNAME /dev/disk/by-partlabel/OCPSWAP)"
2. Delete the partition by running the following command:
  
  This command removes all partitions on the device, not just the OCPSWAP partition. If you manually created additional partitions on this device, they will be deleted.
  $ sfdisk --delete $DISK

Verify that the OCPSWAP partition label is removed by running the following command:

$ ls /dev/disk/by-partlabel/OCPSWAP

Example output

ls: cannot access '/dev/disk/by-partlabel/OCPSWAP': No such file or directory

If the partition label still exists after running sfdisk --delete, verify the correct disk device was used and retry the deletion command. Reboot the node if the partition table was not updated.

Exit the debug session by running the following commands:
```
$ exit
```
```
$ exit
```
The first exit command leaves the chroot environment. The second exit command terminates the debug pod.
Remove the node from maintenance mode by using the oc adm uncordon command.

Enable memory overcommitment and provision swap

You can increase the number of virtual machines (VMs) on a node by overcommitting memory and provisioning swap resources. When you enable memory overcommitment, the virt-platform-autopilot controller automatically deploys the necessary node-level configurations.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged in to the cluster with the cluster-admin role.
You have defined a memory overcommit ratio.
You have provisioned swap storage on worker nodes.

Procedure

Choose one of the following methods to enable memory overcommitment in OKD Virtualization:

After you enable memory overcommitment, only virtual machine workloads (VM pods) of the Burstable Quality of Service (QoS) class can use swap resources. VM pods of the Guaranteed QoS class and pods of any QoS class that do not belong to VMs cannot use swap resources. For descriptions of QoS classes, see Configure Quality of Service for Pods (Kubernetes documentation).

When you configure virtual machines for memory overcommitment, use spec.domain.memory.guest in the VM manifest. Using spec.domain.resources.requests.memory disables the memory overcommit configuration.
- To use the web console:
  1. In the OKD web console, go to Virtualization → Settings.
  2. Click the Cluster tab.
  3. Expand General settings.
  4. Expand Memory request ratio.
  5. In the Request ratio field, enter a value from 25 to 100.
    
    The request ratio is the percentage of each virtual machine’s configured memory that the cluster requests. A lower ratio increases overcommitment. At a ratio of 25%, a VM configured with 4 GiB of memory requests 1 GiB on the cluster. At a ratio of 100%, it requests the full 4 GiB.
    
    The web console indicates the level of risk for the value you enter. It shows Recommended at 75% or greater, Use with caution from 50% to 74.9%, and High overcommit risk from 25% to 49.9%.
    
    The Active ratio line shows the ratio currently calculated across all virtual machines on the cluster, which can differ from the ratio you request. It shows No running VMs if no virtual machines are running.
  6. Click Save.
- To use the CLI, run the following command to enable higher memory density and set the overcommit rate:
  $ oc patch -n kubevirt-hyperconverged hco kubevirt-hyperconverged --type='json' -p='[ \ { \ "op": "replace", \ "path": "/spec/virtualization/higherWorkloadDensity/memoryOvercommitPercentage", \ "value": 150 \ } \ ]'
  Example output
  hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

Verification

Verify the OKD Virtualization memory overcommitment configuration by running the following command:
```
$ oc get -n kubevirt-hyperconverged hco kubevirt-hyperconverged -o jsonpath='{.spec.virtualization.higherWorkloadDensity}{"\n"}'
```
Example output
```
{"memoryOvercommitPercentage":150}
```
The returned value must match the value you configured earlier.

Disable memory overcommitment and remove swap

You can disable higher virtual machine (VM) workload density if you no longer need memory overcommitment, and remove the associated swap resources from your cluster.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged in to the cluster with the cluster-admin role.

Procedure

Revert the memory overcommitment configuration by running the following command:

$ oc patch -n kubevirt-hyperconverged hco kubevirt-hyperconverged \
  --type='json' \
  -p='[{"op": "remove", "path": "/spec/virtualization/higherWorkloadDensity"}]'

Delete the MachineConfig objects that provision and configure swap by running the following commands:
```
$ oc delete machineconfig 90-worker-swap
```
```
$ oc delete machineconfig 90-worker-swap-online
```
Wait for the worker nodes to sync with the new configuration by running the following command:
```
$ oc wait mcp worker --for condition=Updated=True --timeout=-1s
```

Verification

Confirm that swap is no longer enabled on a node by running the following command:
```
$ oc debug node/<selected_node> -- free -m
```
Ensure that the Swap: row shows 0 or that no swap space is provisioned.

Configure higher VM workload density

Swap provisioning methods

Swap space requirements

Provision swap on an NVMe device

Provision swap by using a file

Remove swap from an NVMe device

Enable memory overcommitment and provision swap

Disable memory overcommitment and remove swap

Additional resources