Reserving CPUs for OVS-DPDK

OVS-DPDK CPU isolation
Reserve OVS-DPDK CPUs with a performance profile
Update or remove OVS-DPDK CPU reservation
OVS-DPDK performance profile fields
Additional resources

Reserve CPUs for Open vSwitch Data Plane Development Kit (OVS-DPDK) poll mode driver (PMD) threads by using a PerformanceProfile custom resource (CR). The Node Tuning Operator applies multi-layer CPU isolation so that those host-side threads can run with reduced interference from the operating system and Kubernetes scheduling.

OVS-DPDK CPU reservation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

OVS-DPDK CPU isolation

OVS-DPDK poll mode driver (PMD) threads busy-poll the NIC on the host. As a result, they need exclusive CPUs. Use spec.cpu.ovsDpdk in a PerformanceProfile custom resource (CR) when reserved and isolated alone cannot protect those threads from jitter.

OVS-DPDK PMD threads run as host processes outside the pod scheduling path. Interruption from the kernel, system daemons, or other workloads can introduce jitter and reduce throughput. A performance profile already defines the following CPU sets, which do not fully cover that case:

reserved: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, Burstable and BestEffort pods can still run on reserved CPUs.
isolated: CPUs for Guaranteed application pods that the kubelet schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK.

Because OVS-DPDK runs as a daemon on the host, the profile needs a third CPU set that is excluded from kubelet scheduling and from host interference sources.

When you set spec.cpu.ovsDpdk, the Node Tuning Operator ensures that pods of any QoS class are not placed on these CPUs. The Node Tuning Operator also bans them from irqbalance, removes them from systemd CPU affinity, adds them to kernel isolation parameters, and places them under the OVS cgroup hierarchy.

The ovsDpdk CPU set is specific to OVS-DPDK. Do not use it as a generic pool for other DPDK applications. The Node Tuning Operator places these CPUs under the existing ovs.slice hierarchy for ovs-vswitchd.

For the CPU set listed in spec.cpu.ovsDpdk, the Node Tuning Operator applies configuration across several layers, including the following:

Adds the CPUs to the kubelet reservedSystemCPUs set together with spec.cpu.reserved. When spec.cpu.shared is also set, reservedSystemCPUs is the union of reserved, shared, and ovsDpdk. The ovsDpdk CPUs are not added to the CRI-O shared cpuset. For more information about shared CPUs, see "How ExecCPUAffinity prevents latency spikes from exec operations" in Additional resources.
Adds the CPUs to TuneD isolated_cores together with spec.cpu.isolated.
Includes the CPUs in kernel boot parameters such as isolcpus, nohz_full, and rcu_nocbs.
Excludes the CPUs from systemd.cpu_affinity.
Updates IRQBALANCE_BANNED_CPUS so that hardware IRQs are not balanced onto those CPUs.
Creates and configures ovsdpdk.slice under ovs.slice/ovs-vswitchd.service, including exclusive CPU assignment for OVS-DPDK.

You can add the performance.openshift.io/cpu-load-balancing-ovs-dpdk annotation to the performance profile to control kernel scheduler load balancing on the ovsDpdk CPUs.

Set the annotation to disable to exclude ovsDpdk CPUs from the kernel scheduler load-balancing pool. The Node Tuning Operator configures the OVS-DPDK cgroup partition as isolated.
Omit the annotation or set it to enable to keep the default behavior.

The Node Tuning Operator requires either workload partitioning or the kubelet CPU Manager strict-cpu-reservation policy option to prevent Burstable and BestEffort pods from running on OVS-DPDK CPUs. If neither prerequisite is satisfied, the performance profile enters a Degraded state with reason OvsDpdkCPUsPrerequisiteNotMet.

Choose one of the following options:

Recommended: Use workload partitioning. Enable workload partitioning at cluster install time by setting cpuPartitioningMode: AllNodes. The performance profile populates the workload partitioning CPU sets when applied. Workload partitioning provides comprehensive CPU isolation for platform and infrastructure workloads.
Alternative: Configure strict-cpu-reservation using the kubeletconfig.experimental annotation in your OVS-DPDK performance profile. Use this option only if workload partitioning was not enabled during cluster installation.

You must choose NUMA-aligned CPUs for your OVS-DPDK NICs. The Operator does not automatically assign CPUs based on NIC locality.

Reserve OVS-DPDK CPUs with a performance profile

Create or update a PerformanceProfile custom resource (CR) that reserves CPUs for OVS-DPDK by setting spec.cpu.ovsDpdk. You can optionally add the performance.openshift.io/cpu-load-balancing-ovs-dpdk annotation in the same CR to disable kernel scheduler load balancing on the OVS-DPDK CPUs.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI (oc).
You have identified whether workload partitioning was enabled at cluster installation (cpuPartitioningMode: AllNodes).

Procedure

Generate a base performance profile by using the Performance Profile Creator (PPC) tool with must-gather data from your cluster:
1. Collect must-gather data from the cluster by running the following command:
  $ oc adm must-gather
2. Generate the base profile by running the PPC tool:
  $ podman run --entrypoint performance-profile-creator \ -v <path_to_must_gather>:/must-gather:z \ registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4 \ --must-gather-dir-path /must-gather \ --mcp-name=<mcp_name> \ --profile-name=ovs-dpdk-isolation \ --reserved-cpu-count=<reserved_count> \ --ovs-dpdk-cpu-count=<ovs_dpdk_count> \ --rt-kernel=true \ --split-reserved-cpus-across-numa=false \ --power-consumption-mode=ultra-low-latency \ > my-performance-profile.yaml
  <path_to_must_gather>
  
  Replace with the path to the must-gather output directory created by the oc adm must-gather command. The directory name follows the pattern must-gather.local.<id>.
  
  <mcp_name>
  
  Replace with the name of the machine config pool for the target nodes. For single-node OpenShift, use --mcp-name=master.
  
  <reserved_count>
  
  Replace with the number of CPUs to reserve for platform and cluster management.
  
  <ovs_dpdk_count>
  
  Replace with the number of CPUs to reserve for OVS-DPDK. When you specify this parameter, the PPC tool populates spec.cpu.ovsDpdk with the specified number of CPUs.
  
  For more information about other PPC arguments, see "Running the Performance Profile Creator using Podman".
Review and edit the generated profile to ensure that the CPUs in spec.cpu.ovsDpdk are on the same NUMA node as the OVS-DPDK NICs by running the following command:
```
$ cat my-performance-profile.yaml
```
The PPC tool allocates CPUs by count and ensures they are siblings on the same physical core, but does not consider NUMA locality relative to the OVS-DPDK NICs. Verify that the ovsDpdk CPUs match the NUMA node identified in the prerequisites. If the allocated CPUs are on the wrong NUMA node, edit the ovsDpdk field to specify CPUs from the correct NUMA node. Ensure that the CPUs you specify are not already in the reserved set.

If workload partitioning is not enabled on your cluster, edit the generated PerformanceProfile to add the strict-cpu-reservation annotation. This prevents Burstable and BestEffort pods from running on reserved CPUs:

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: ovs-dpdk-isolation
  annotations:
    kubeletconfig.experimental: |
      {
       "cpuManagerPolicyOptions":{"strict-cpu-reservation":"true"}
      }
spec:
  cpu:
    isolated: "4-7"
    ovsDpdk: "2-3"
    reserved: "0-1"
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/worker: ""
  nodeSelector:
    node-role.kubernetes.io/worker: ""

If workload partitioning is enabled, skip this step.

Optional: To disable kernel scheduler load balancing on the OVS-DPDK CPUs, add the performance.openshift.io/cpu-load-balancing-ovs-dpdk annotation set to disable:

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: ovs-dpdk-isolation
  annotations:
    performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"
    kubeletconfig.experimental: |
      {
       "cpuManagerPolicyOptions":{"strict-cpu-reservation":"true"}
      }
spec:
  cpu:
    isolated: "4-7"
    ovsDpdk: "2-3"
    reserved: "0-1"
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/worker: ""
  nodeSelector:
    node-role.kubernetes.io/worker: ""

Omit the kubeletconfig.experimental annotation if workload partitioning is enabled on your cluster.

Apply the performance profile by running the following command:
```
$ oc apply -f my-performance-profile.yaml
```
Wait for the Machine Config Operator to apply the configuration and for the target nodes to reboot. Confirm that the relevant machine config pool has finished updating by running the following command:
```
$ oc get mcp -w
```
The pool that matches your profile selector must show UPDATED=True and UPDATING=False. Press Ctrl+C to stop watching.

Verification

Verify that the performance profile is available by running the following command:

$ oc get performanceprofile ovs-dpdk-isolation -o yaml

Check the status conditions in the output. The Available condition must have status: "True" and the Degraded condition must have status: "False".

A properly configured system generates an output similar to this healthy profile example:

status:
  conditions:
  - status: "True"
    type: Available
  - status: "False"
    type: Degraded
  runtimeClass: performance-ovs-dpdk-isolation

If the Degraded condition shows status: "True" with reason: OvsDpdkCPUsPrerequisiteNotMet, the OVS-DPDK CPU feature requires strict CPU reservation. Verify that workload partitioning is enabled or that you have added the kubeletconfig.experimental annotation to configure strict-cpu-reservation in the performance profile, then reapply the profile.

Verify that kubelet reserved system CPUs are the union of reserved and ovsDpdk by running the following command:
```
$ oc get --raw "/api/v1/nodes/<node_name>/proxy/configz" | \
  python3 -c 'import sys,json; print(json.load(sys.stdin)["kubeletconfig"]["reservedSystemCPUs"])'
```
The reported set must equal the union of reserved and ovsDpdk. For example, with reserved: "0-1" and ovsDpdk: "2-3", the output is 0-3.
Start a debug session on a node that matches the performance profile by running the following command:
```
$ oc debug node/<node_name>
```
Set /host as the root directory for the debug shell by running the following command:
```
# chroot /host
```
Verify that kernel boot parameters include the OVS-DPDK CPUs by running the following command:
```
# cat /proc/cmdline
```
Confirm the following:
- isolcpus includes the union of isolated and ovsDpdk CPUs.
- nohz_full and rcu_nocbs include the ovsDpdk CPUs.
- systemd.cpu_affinity does not include the ovsDpdk CPUs.
Verify that IRQ balancing excludes the OVS-DPDK CPUs by running the following command:
```
# grep -E "^IRQBALANCE_BANNED_CPUS=" /etc/sysconfig/irqbalance
```
Confirm that IRQBALANCE_BANNED_CPUS includes a hex mask that covers the ovsDpdk CPUs. For example, ovsDpdk: "2-3" (CPUs 2 and 3) produces IRQBALANCE_BANNED_CPUS=c. The mask value is not zero-padded.
Verify that the default SMP affinity mask excludes the OVS-DPDK CPUs by running the following command:
```
# cat /proc/irq/default_smp_affinity
```
The mask must clear the bits for ovsDpdk CPUs. For example, with ovsDpdk: "2-3" on an 8-CPU node, the output is f3.
Verify that the OVS-DPDK cgroup slice has the correct exclusive CPUs assigned by running the following command:
```
# cat /sys/fs/cgroup/ovs.slice/ovs-vswitchd.service/ovsdpdk.slice/cpuset.cpus.exclusive
```
The output must match the ovsDpdk value from the performance profile. For example, with ovsDpdk: "2-3", the output is 2-3.

These checks confirm that the platform configuration for OVS-DPDK CPU isolation is in place. Measuring OVS-DPDK dataplane throughput and packet loss requires your OVS-DPDK application test environment.

Update or remove OVS-DPDK CPU reservation

Change the spec.cpu.ovsDpdk CPU set on a Day 2 retune, or remove the field when you no longer need OVS-DPDK CPU isolation. Keep CPU sets disjoint, wait for the Machine Config Operator (MCO) to apply changes, and re-verify isolation or cleanup on the node.