Install and configure the DPF Operator

DPF Operator installation environment variables
Install the DPF Operator
Create the DPFOperatorConfig custom resource
Create the NodeSRIOVDevicePluginConfig custom resource
Create the DPUFlavor custom resource
Create the BFB resource
Create the DPUDeployment custom resource
Create the HBN DPU service configuration
Create the OVN-Kubernetes DPU service configuration
Create the DOCA Telemetry Service DPU service configuration
Create the OVN-Kubernetes credential request and role bindings
Create the DPUServiceInterface custom resources
Create the DPUServiceNAD resource
Create the DPUServiceIPAM resources

After setting up the environment, install the NVIDIA DPF Operator and create the required DPF resources and DPU services.

You must install the DPF Operator before the DPF HCP Provisioner Operator because that Operator requires DPF CRDs such as DPUCluster, DPUFlavor, DPUDeployment, and DPFOperatorConfig.

DPF Operator installation environment variables

Set the following required environment variables before you install and configure the DPF Operator on OKD. The values of these variables are used in multiple DPF Operator installation and configuration procedures.

Table 1. DPF Operator environment variables
Variable	Description	Example value
`CLUSTER_NAME`	The name of the management cluster.	`<YOUR_CLUSTER_NAME>`
`BASE_DOMAIN`	The base domain for the management cluster.	`example.com`
`HOST_CLUSTER_API`	The API server hostname of the management cluster. Derived from `CLUSTER_NAME` and `BASE_DOMAIN`.	`api.mycluster.example.com`
`REGISTRY`	The Helm chart registry URL for the DPF Operator.	`https://helm.ngc.nvidia.com/nvidia/doca`
`TAG`	The version tag for the DPF Operator Helm chart.	`v26.4.1`
`TARGETCLUSTER_API_SERVER_PORT`	The port number of the hosted cluster API server.	`6443`
`VTEP_CIDR`	A dedicated IP range allocated from the DPU high-speed network for VTEP (tunnel endpoint) IPs. Used for DPU service IPs by HBN and OVN-Kubernetes tunnels. The management network and this VTEP CIDR network must be routable to each other in both directions.	`10.0.120.0/22`
`NUM_VFS`	Number of SR-IOV VFs per physical function. Sets `NUM_OF_VFS` in the DPUFlavor `nvconfig`. The `NodeSRIOVDevicePluginConfig` allocates these VFs as VF0 (DPF communication channel), VF1 (OVN-Kubernetes management), and the remainder (workload and RDMA).	`46`
`NODES_MTU`	The MTU value for the node network interfaces. Use `1500` for standard MTU environments or `9000` for jumbo frame environments. This value must be consistent across all DPF networking configuration.	`1500` (standard) or `9000` (jumbo frames)
`OVN_TEMPLATE_CHART_URL`	The OCI chart URL for the OVN-Kubernetes Helm chart.	`oci://ghcr.io/mellanox/charts`
`OVN_CHART_VERSION`	The version of the OVN-Kubernetes Helm chart.	`v26.4.1-ocp-release-v4.22`
`OVN_MTU`	The MTU value for OVN-Kubernetes overlay networking. Set to `1400` for standard MTU (1500 minus encapsulation overhead) or `8900` for jumbo frames (9000 minus encapsulation overhead). Must be kept consistent with `NODES_MTU`.	`1400` or `8900`
`DPU_HOST_CIDR`	The CIDR range of the subnet where the DPU host nodes reside. Used as the OVN `hostCIDR`. You must set this to your actual host subnet; the example value is illustrative only and is not a usable default.	`10.0.110.0/24`
`FLANNEL_POD_CIDR`	The pod CIDR for the Flannel network in the hosted cluster. Required for OKD 4.22 and later.	`10.132.0.0/14`
`BFB_URL`	The download URL for the BlueField Bootstream File image used to provision DPUs.	`https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-10.2/builds/10.2.20260715-0/aarch64/rhcos-10.2.20260715-0-nvidiabluefield.aarch64.bfb`

You must set the following environment variables in your terminal session before you proceed.

$ export CLUSTER_NAME="<YOUR_CLUSTER_NAME>"
$ export BASE_DOMAIN="example.com"
$ export HOST_CLUSTER_API="api.${CLUSTER_NAME}.${BASE_DOMAIN}"
$ export REGISTRY="https://helm.ngc.nvidia.com/nvidia/doca"
$ export TAG="v26.4.1"
$ export TARGETCLUSTER_API_SERVER_PORT="6443"
$ export VTEP_CIDR="10.0.120.0/22"
$ export NUM_VFS="46"
$ export NODES_MTU="1500"
$ export OVN_TEMPLATE_CHART_URL="oci://ghcr.io/mellanox/charts"
$ export OVN_CHART_VERSION="v26.4.1-ocp-release-v4.22"
$ export OVN_MTU=1400
$ export DPU_HOST_CIDR="10.0.110.0/24"
$ export FLANNEL_POD_CIDR="10.132.0.0/14"
$ export BFB_URL="https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-10.2/builds/10.2.20260715-0/aarch64/rhcos-10.2.20260715-0-nvidiabluefield.aarch64.bfb"

Install the DPF Operator

You can install the DPF Operator by using Helm to deploy the Operator into the dpf-operator-system namespace on your management cluster.

You must install the DPF Operator before the DPF HCP Provisioner Operator because that Operator requires the following DPF custom resource definitions to be available: DPUCluster, DPUFlavor, DPUDeployment, and DPFOperatorConfig.

Prerequisites

You have access to the management cluster as a user with the cluster-admin role.
You have installed the oc CLI.
You have installed the helm CLI.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Add the DPF Helm repository and update the local cache:

$ helm repo add --force-update dpf-repository ${REGISTRY}

$ helm repo update

Install the DPF Operator by using Helm:

$ helm upgrade --install dpf-operator dpf-repository/dpf-operator \
    --namespace dpf-operator-system \
    --version "${TAG}" \
    --set kamajiEtcdDefrag.enabled=false \
    --set isOpenshift=true \
    --set enableNodeFeatureRules=false \
    --wait

Verification

Verify that the Operator controller manager deployment has rolled out successfully:

$ oc rollout status deployment --namespace dpf-operator-system dpf-operator-controller-manager

Example output

deployment "dpf-operator-controller-manager" successfully rolled out

Verify that all pods in the dpf-operator-system namespace are ready:

$ oc wait --for=condition=ready --namespace dpf-operator-system pods --all

Example output

pod/argocd-application-controller-0 condition met
pod/argocd-dex-server-6dd56c8469-bhsq4 condition met
pod/argocd-redis-b4f94bb8d-wr86b condition met
pod/argocd-repo-server-96765f997-79k9q condition met
pod/argocd-server-648c7ff85f-7frtg condition met
pod/dpf-operator-controller-manager-7bf9744c5f-cwrgc condition met
pod/maintenance-operator-585767f779-8k2lx condition met

Create the DPFOperatorConfig custom resource

Create a DPFOperatorConfig custom resource to configure the DPF Operator components, including the provisioning controller, static cluster manager, and SR-IOV device plugin controller.

Prerequisites

You have installed the DPF Operator.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named dpfoperatorconfig.yaml with the following content:

MTU Configuration: Set NODES_MTU to 1500 for standard MTU environments or 9000 for jumbo frame environments. This value must be consistent across:

Assisted Installer host configuration
DPFOperatorConfig networking section (this step)
DPUServiceNAD configuration

Choose based on your network infrastructure capabilities.

apiVersion: operator.dpu.nvidia.com/v1alpha1
kind: DPFOperatorConfig
metadata:
  name: dpfoperatorconfig
  namespace: dpf-operator-system
spec:
  kamajiClusterManager:
    disable: true
  multus:
    disable: true
  cniInstaller:
    disable: true
  networking:
    controlPlaneMTU: $NODES_MTU      # management/OOB MTU (= NODES_MTU: 1500 standard, 9000 jumbo)
    highSpeedMTU: $NODES_MTU         # high-speed fabric MTU (= NODES_MTU; must match controlPlaneMTU)
    dpuNodeOOBBridgeName: br-ex      # OOB bridge for DPU provisioning; br-ex on OpenShift
  overrides:
    dpuCNIBinPath: /var/lib/cni/bin/
    dpuCNIPath: /run/multus/cni/net.d/
    dpuOpenvSwitchSystemSharedLib64Path: /lib64
    flannelSkipCNIConfigInstallation: false
    kubernetesAPIServerPort: $TARGETCLUSTER_API_SERVER_PORT
    kubernetesAPIServerVIP: $HOST_CLUSTER_API
    dpuLinkerCachePath: /etc/ld.so.cache
    dpuOptLibraryPath: /usr/opt
  provisioningController:
    enableDynamicBFCFGTemplates: true
    hostAgentDNSPolicy: Default
    dmsTimeout: 900
  nodeSRIOVDevicePluginController:
    devicePlugin:
      defaultResourcePrefix: openshift.io
    disable: false
    replicas: 1
  staticClusterManager:
    disable: false
  dpuServiceController:
    disableHostNetworkReadyNoExecuteTaints: false
  flannel:
    podCIDR: $FLANNEL_POD_CIDR

Apply the resource file:

$ envsubst < dpfoperatorconfig.yaml | oc apply -f -

Verification

Verify that the provisioning controller manager deployment has rolled out:

$ oc rollout status deployment --namespace dpf-operator-system dpf-provisioning-controller-manager

Example output

deployment "dpf-provisioning-controller-manager" successfully rolled out

Verify that the DPU service controller manager deployment has rolled out:

$ oc rollout status deployment --namespace dpf-operator-system dpuservice-controller-manager

Example output

deployment "dpuservice-controller-manager" successfully rolled out

Verify that all Operator deployments in the dpf-operator-system namespace have rolled out:

You might need to run this command more than once, because the deployments become available at different times as the Operator reconciles its resources.
```
$ oc rollout status deployment --namespace dpf-operator-system
```

Optional: List the pods in the dpf-operator-system namespace to review their status:

$ oc get pods -n dpf-operator-system

Example output

NAME                                                    READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                          1/1     Running   0          64m
argocd-dex-server-9dd99cc6c-pkl6j                       1/1     Running   0          64m
argocd-redis-6749d85f98-mw9rc                           1/1     Running   0          64m
argocd-repo-server-5fd79d655c-9rl26                     1/1     Running   0          64m
argocd-server-5cc4f69c45-hzgm9                          1/1     Running   0          64m
bfb-registry                                            1/1     Running   0          98s
dpf-nodesriovdeviceplugin-controller-7c7bb889f6-xk6nm   1/1     Running   0          107s
dpf-operator-controller-manager-6969c6d8bc-4njxd        1/1     Running   0          4m54s
dpf-provisioning-controller-manager-54b6bdc57f-69lr7    1/1     Running   0          109s
dpf-provisioning-controller-manager-54b6bdc57f-fg9f6    1/1     Running   0          109s
dpuservice-controller-manager-5d9c4f6f67-wsdzt          1/1     Running   0          108s
maintenance-operator-68c794b549-pp2dp                   1/1     Running   0          74m
static-cm-controller-manager-745c8fd7d5-n8d2z           1/1     Running   0          110s

Create the NodeSRIOVDevicePluginConfig custom resource

You can create a NodeSRIOVDevicePluginConfig custom resource to define how SR-IOV virtual functions on the management cluster worker nodes are allocated to DPF components.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.

Procedure

Create a file named nodesriovdevicepluginconfig.yaml with the following content:

apiVersion: noderesources.dpu.nvidia.com/v1alpha1
kind: NodeSRIOVDevicePluginConfig
metadata:
  name: bf3-vfs
  namespace: dpf-operator-system
spec:
  devicePluginResources:
    - name: bf3-p0-vfs-mgmt
      type: vf
      ranges:
        - pfIndex: 0
          start: 1
          end: 1
    - name: bf3_vfs
      type: vf
      options:
        isRdma: true
      ranges:
        - pfIndex: 0
          start: 2
          end: 45
        - pfIndex: 1
          start: 0
          end: 45

where:

bf3-p0-vfs-mgmt: Reserves VF index 1 on PF0 for DPU management connectivity.
bf3_vfs: Allocates VF indices 2-45 on PF0 and VF indices 0-45 on PF1 for workload traffic with RDMA enabled.
pfIndex: The pfIndex values refer to the first (0) and second (1) physical functions of the BlueField-3 DPU.

Apply the resource file:

$ oc apply -f nodesriovdevicepluginconfig.yaml

Verification

Verify that the NodeSRIOVDevicePluginConfig resource is created:

$ oc get nodesriovdevicepluginconfig -n dpf-operator-system

Example output

bf3-vfs    30s

Create the DPUFlavor custom resource

You can create a DPUFlavor custom resource to define the DPU configuration, including NVConfig parameters, kernel arguments, hugepages settings, and the OVS initialization script. The DPUFlavor also supports an optional configFiles field for custom DPU configuration files.

The nvconfig section contains BlueField-3 firmware parameters that the DPU agent applies by using mlxconfig during provisioning. If any parameter differs from the current firmware configuration, the provisioning controller triggers a system-level reset so that the changes take effect. The following parameters are required for DPF operation:

Table 2. Required BlueField-3 NVConfig parameters
Parameter	Value	Description
`INTERNAL_CPU_MODEL`	`1`	Switches the BlueField-3 to DPU mode where the ARM cores are active. A value of `0` keeps the card in NIC-only mode, which does not support DPF.
`SRIOV_EN`	`1`	Enables SR-IOV on both physical functions. The host agent creates Virtual Functions (VFs) that carry tenant traffic between the host and the DPU.
`NUM_OF_VFS`	Variable	Number of VFs per physical function. Set this value by using the `$NUM_VFS` environment variable. The default is `46`.
`LINK_TYPE_P1` / `LINK_TYPE_P2`	`ETH`	Sets both ports to Ethernet mode. DPF requires Ethernet. InfiniBand (`IB`) mode is not supported.

If BlueField-3 already has the correct values, the DPU agent reports that no action is required and does not trigger a reset.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named dpuflavor.yaml for your MTU configuration and apply it.

For Standard MTU (1500):

apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
  name: hbn-ovnk
  namespace: dpf-operator-system
  annotations:
    # Required when flavor bfcfg exceeds the DPF limit of 128K
    provisioning.dpu.nvidia.com/skip-bfcfg-size-check: ""
spec:
  grub:
    kernelParameters:
      - console=hvc0
      - console=ttyAMA0
      - earlycon=pl011,0x13010000
      - iommu.passthrough=1
      - cgroup_no_v1=net_prio,net_cls
      - hugepagesz=2048kB
      - hugepages=250
  nvconfig:
    - device: "*"
      parameters:
        - PF_BAR2_ENABLE=0
        - PER_PF_NUM_SF=1
        - PF_TOTAL_SF=20
        - PF_SF_BAR_SIZE=10
        - NUM_PF_MSIX_VALID=0
        - PF_NUM_PF_MSIX_VALID=1
        - PF_NUM_PF_MSIX=228
        - INTERNAL_CPU_MODEL=1
        - INTERNAL_CPU_OFFLOAD_ENGINE=0
        - SRIOV_EN=1
        - NUM_OF_VFS=$NUM_VFS
        - LAG_RESOURCE_ALLOCATION=1
        - LINK_TYPE_P1=ETH
        - LINK_TYPE_P2=ETH
  ovs:
    rawConfigScript: |
      #!/bin/bash
      set -e

      _ovs-vsctl() {
        ovs-vsctl --timeout 15 "$@"
      }

      restart_ovs=false

      _ovs-get-other-config() {
        _ovs-vsctl --if-exists get Open_vSwitch . "other_config:$1" 2>/dev/null | tr -d '"'
      }

      _ovs-set-other-config() {
        if [ "$(_ovs-get-other-config "$1")" != "$2" ]; then
          _ovs-vsctl set Open_vSwitch . "other_config:$1=$2"
          restart_ovs=true
        fi
      }

      _ovs-remove-other-config() {
        if [ -n "$(_ovs-get-other-config "$1")" ]; then
          _ovs-vsctl remove Open_vSwitch . other_config "$1"
          restart_ovs=true
        fi
      }

      _ovs-set-other-config doca-init true
      _ovs-set-other-config dpdk-max-memzones 50000
      _ovs-set-other-config hw-offload true
      _ovs-set-other-config pmd-quiet-idle true
      _ovs-set-other-config max-idle 20000
      _ovs-set-other-config max-revalidator 5000
      _ovs-set-other-config doca-congestion-threshold 60
      _ovs-set-other-config flow-limit 500000
      _ovs-set-other-config hw-offload-ct-unidir-udp-enabled true
      _ovs-remove-other-config default-datapath-type

      if [ "$restart_ovs" = true ]; then
        if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
          systemctl restart openvswitch-switch
        elif systemctl list-unit-files openvswitch.service &>/dev/null; then
          systemctl restart openvswitch
        fi
      fi

      _ovs-vsctl --may-exist add-br br-sfc
      _ovs-vsctl set bridge br-sfc datapath_type=netdev
      _ovs-vsctl set bridge br-sfc fail_mode=secure
      _ovs-vsctl --if-exists del-br br-hbn
      _ovs-vsctl --may-exist add-br br-hbn
      _ovs-vsctl set bridge br-hbn datapath_type=netdev
      _ovs-vsctl set bridge br-hbn fail_mode=secure
      _ovs-vsctl --may-exist add-port br-sfc p0
      _ovs-vsctl set Interface p0 type=dpdk
      _ovs-vsctl set Interface p0 mtu_request=9216
      _ovs-vsctl set Port p0 external_ids:dpf-type=physical

      # Activate DOCA for OVNK
      _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev
      # setup ovnkube managed bridge, br-dpu (this corresponds to br-ex on ovnk docs)
      _ovs-vsctl --may-exist add-br br-dpu
      _ovs-vsctl br-set-external-id br-dpu bridge-id br-dpu
      _ovs-vsctl br-set-external-id br-dpu bridge-uplink pbrdputobrovn
      _ovs-vsctl set bridge br-dpu datapath_type=netdev
      _ovs-vsctl --may-exist add-port br-dpu pf0hpf
      _ovs-vsctl set Interface pf0hpf type=dpdk

      # Create OVS bridge (br-ovn) in between the SC managed bridge and OVNK
      _ovs-vsctl --may-exist add-br br-ovn
      _ovs-vsctl set bridge br-ovn datapath_type=netdev
      _ovs-vsctl --may-exist add-port br-ovn pbrovntobrdpu
      _ovs-vsctl --may-exist add-port br-dpu pbrdputobrovn

      # Patch br-ovn and br-dpu together
      _ovs-vsctl set Interface pbrovntobrdpu type=patch options:peer=pbrdputobrovn
      _ovs-vsctl set Interface pbrdputobrovn type=patch options:peer=pbrovntobrdpu

For Jumbo frames (MTU 9000):

apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
  name: hbn-ovnk
  namespace: dpf-operator-system
  annotations:
    # Required when flavor bfcfg exceeds the DPF limit of 128K
    provisioning.dpu.nvidia.com/skip-bfcfg-size-check: ""
spec:
  grub:
    kernelParameters:
      - console=hvc0
      - console=ttyAMA0
      - earlycon=pl011,0x13010000
      - iommu.passthrough=1
      - cgroup_no_v1=net_prio,net_cls
      - hugepagesz=2048kB
      - hugepages=250
  nvconfig:
    - device: "*"
      parameters:
        - PF_BAR2_ENABLE=0
        - PER_PF_NUM_SF=1
        - PF_TOTAL_SF=20
        - PF_SF_BAR_SIZE=10
        - NUM_PF_MSIX_VALID=0
        - PF_NUM_PF_MSIX_VALID=1
        - PF_NUM_PF_MSIX=228
        - INTERNAL_CPU_MODEL=1
        - INTERNAL_CPU_OFFLOAD_ENGINE=0
        - SRIOV_EN=1
        - NUM_OF_VFS=$NUM_VFS
        - LAG_RESOURCE_ALLOCATION=1
        - NUM_VF_MSIX=30
        - LINK_TYPE_P1=ETH
        - LINK_TYPE_P2=ETH
  ovs:
    rawConfigScript: |
      #!/bin/bash
      set -e

      _ovs-vsctl() {
        ovs-vsctl --timeout 15 "$@"
      }

      restart_ovs=false

      _ovs-get-other-config() {
        _ovs-vsctl --if-exists get Open_vSwitch . "other_config:$1" 2>/dev/null | tr -d '"'
      }

      _ovs-set-other-config() {
        if [ "$(_ovs-get-other-config "$1")" != "$2" ]; then
          _ovs-vsctl set Open_vSwitch . "other_config:$1=$2"
          restart_ovs=true
        fi
      }

      _ovs-remove-other-config() {
        if [ -n "$(_ovs-get-other-config "$1")" ]; then
          _ovs-vsctl remove Open_vSwitch . other_config "$1"
          restart_ovs=true
        fi
      }

      _ovs-set-other-config doca-init true
      _ovs-set-other-config dpdk-max-memzones 50000
      _ovs-set-other-config hw-offload true
      _ovs-set-other-config pmd-quiet-idle true
      _ovs-set-other-config max-idle 20000
      _ovs-set-other-config max-revalidator 5000
      _ovs-set-other-config doca-congestion-threshold 60
      _ovs-set-other-config flow-limit 500000
      _ovs-set-other-config hw-offload-ct-unidir-udp-enabled true
      _ovs-remove-other-config default-datapath-type

      if [ "$restart_ovs" = true ]; then
        if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
          systemctl restart openvswitch-switch
        elif systemctl list-unit-files openvswitch.service &>/dev/null; then
          systemctl restart openvswitch
        fi
      fi

      _ovs-vsctl --may-exist add-br br-sfc
      _ovs-vsctl set bridge br-sfc datapath_type=netdev
      _ovs-vsctl set bridge br-sfc fail_mode=secure
      _ovs-vsctl --if-exists del-br br-hbn
      _ovs-vsctl --may-exist add-br br-hbn
      _ovs-vsctl set bridge br-hbn datapath_type=netdev
      _ovs-vsctl set bridge br-hbn fail_mode=secure
      _ovs-vsctl --may-exist add-port br-sfc p0
      _ovs-vsctl set Interface p0 type=dpdk
      _ovs-vsctl set Interface p0 mtu_request=9216
      _ovs-vsctl set Port p0 external_ids:dpf-type=physical

      # Activate DOCA for OVNK
      _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev
      # setup ovnkube managed bridge, br-dpu (this corresponds to br-ex on ovnk docs)
      _ovs-vsctl --may-exist add-br br-dpu
      _ovs-vsctl br-set-external-id br-dpu bridge-id br-dpu
      _ovs-vsctl br-set-external-id br-dpu bridge-uplink pbrdputobrovn
      _ovs-vsctl set bridge br-dpu datapath_type=netdev
      _ovs-vsctl set Interface br-dpu mtu_request=9000
      _ovs-vsctl --may-exist add-port br-dpu pf0hpf
      _ovs-vsctl set Interface pf0hpf type=dpdk

      # Create OVS bridge (br-ovn) in between the SC managed bridge and OVNK
      _ovs-vsctl --may-exist add-br br-ovn
      _ovs-vsctl set bridge br-ovn datapath_type=netdev
      _ovs-vsctl set Interface br-ovn mtu_request=9000
      _ovs-vsctl --may-exist add-port br-ovn pbrovntobrdpu
      _ovs-vsctl --may-exist add-port br-dpu pbrdputobrovn

      # Patch br-ovn and br-dpu together
      _ovs-vsctl set Interface pbrovntobrdpu type=patch options:peer=pbrdputobrovn
      _ovs-vsctl set Interface pbrdputobrovn type=patch options:peer=pbrovntobrdpu

Apply the resource file:

$ envsubst < dpuflavor.yaml | oc apply -f -

Verification

Verify that the DPUFlavor resource is created:

$ oc get dpuflavor -n dpf-operator-system

Create the `BFB` resource

You can create a BFB custom resource to define the DPU image, known as a BlueField Bootstream File, that is downloaded and placed on shared storage for DPU provisioning.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named bfb.yaml with the following content:

apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: BFB
metadata:
  name: bf-bundle
  namespace: dpf-operator-system
spec:
  url: $BFB_URL
  versions:
    atf: 4.15.0-4-g419fbf393
    bsp: 4.15.0.13998
    doca: 3.4.1
    uefi: 4.15.0-19-g37c6f5adb2

Apply the resource file:
```
$ envsubst < bfb.yaml | oc apply -f -
```

Verification

Verify that the BFB image phase is Ready:

$ oc get bfb -n dpf-operator-system bf-bundle

Example output

NAME        PHASE   AGE
bf-bundle   Ready   3m

Create the DPUDeployment custom resource

You can create a DPUDeployment custom resource as the main orchestration object that connects DPU services with specific BFB images and DPU flavors. The DPUDeployment defines DPU sets for DPU provisioning and configures service chains to deploy services across DPUs.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have created the NodeSRIOVDevicePluginConfig resource.
You have created the DPUFlavor resource.
You have created the BFB resource and it is in the Ready phase.

Procedure

Create a file named dpudeployment.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUDeployment
metadata:
  name: dpudeployment
  namespace: dpf-operator-system
spec:
  dpus:
    nodeEffect:
      drain: true
    dpuSetStrategy:
      type: RollingUpdate
    bfb: bf-bundle
    flavor: hbn-ovnk
    dpuSets:
      - nameSuffix: "dpuset1"
        dpuNodeSelector:
          matchLabels:
            feature.node.kubernetes.io/dpu-enabled: ""
        dpuAnnotations:
          noderesources.dpu.nvidia.com/nodesriovdevicepluginconfig: bf3-vfs
  services:
    hbn:
      serviceTemplate: hbn
      serviceConfiguration: hbn
    ovn:
      serviceTemplate: ovn
      serviceConfiguration: ovn
    doca-telemetry-service:
      serviceTemplate: doca-telemetry-service
      serviceConfiguration: doca-telemetry-service
  serviceChains:
    switches:
      - ports:
          - serviceInterface:
              matchLabels:
                uplink: p0
          - service:
              name: hbn
              interface: p0_if
      - ports:
          - serviceInterface:
              matchLabels:
                uplink: p1
          - service:
              name: hbn
              interface: p1_if
      - ports:
          - serviceInterface:
              matchLabels:
                port: ovn
          - service:
              name: hbn
              interface: pf2dpu2_if

Apply the resource file:
```
$ oc apply -f dpudeployment.yaml
```

Verification

Verify the DPUDeployment state:
```
$ oc get DPUDeployment -n dpf-operator-system
```
Example output
```
NAME            READY   PHASE     AGE
dpudeployment   False   Pending   2m32s
```
A Pending phase is expected at this stage. The DPUDeployment transitions to Ready after DPU provisioning is complete and all services are deployed.

Create the HBN DPU service configuration

You can create a DPUServiceConfiguration custom resource for the Host-Based Networking (HBN) DPU service. The HBN service provides BGP-based networking on the DPU with ECMP routing support.

HBN and OVN-Kubernetes are currently the only supported DPU network services. The DOCA Telemetry Service (DTS), which you configure in a later step, is deployed for observability and is not a network service.

The DPUServiceTemplate resources are automatically created and managed by the dpf-hcp-provisioner-operator. You only need to create the DPUServiceConfiguration resources.

Prerequisites

The DPF Operator is installed.
The DPFOperatorConfig resource is created.
The DPF Operator environment variables are set. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named hbn.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: hbn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "hbn"
  serviceConfiguration:
    serviceDaemonSet:
      annotations:
        k8s.v1.cni.cncf.io/networks: |-
          [
          {"name": "iprequest", "interface": "ip_lo", "cni-args": {"poolNames": ["loopback"], "poolType": "cidrpool"}},
          {"name": "iprequest", "interface": "ip_pf2dpu2", "cni-args": {"poolNames": ["pool1"], "poolType": "cidrpool", "allocateDefaultGateway": true}}
          ]
    helmChart:
      values:
        configuration:
          perDPUValuesYAML: |
            - hostnamePattern: "*"
              values:
                bgp_peer_group: hbn
          startupYAMLJ2: |
            - header:
                model: BLUEFIELD
                nvue-api-version: nvue_v1
                rev-id: 1.0
                version: HBN 2.4.0
            - set:
                interface:
                  lo:
                    ip:
                      address:
                        {{ ipaddresses.ip_lo.ip }}/32: {}
                    type: loopback
                  p0_if,p1_if:
                    type: swp
                    link:
                      mtu: 9216
                  pf2dpu2_if:
                    ip:
                      address:
                        {{ ipaddresses.ip_pf2dpu2.cidr }}: {}
                    type: swp
                    link:
                      mtu: 9216
                router:
                  bgp:
                    autonomous-system: {{ ( ipaddresses.ip_lo.ip.split(".")[3] | int ) + 65101 }}
                    enable: on
                    graceful-restart:
                      mode: full
                    router-id: {{ ipaddresses.ip_lo.ip }}
                vrf:
                  default:
                    router:
                      bgp:
                        address-family:
                          ipv4-unicast:
                            enable: on
                            redistribute:
                              connected:
                                enable: on
                          ipv6-unicast:
                            enable: on
                            redistribute:
                              connected:
                                enable: on
                        enable: on
                        neighbor:
                          p0_if:
                            peer-group: {{ config.bgp_peer_group }}
                            type: unnumbered
                          p1_if:
                            peer-group: {{ config.bgp_peer_group }}
                            type: unnumbered
                        path-selection:
                          multipath:
                            aspath-ignore: on
                        peer-group:
                          {{ config.bgp_peer_group }}:
                            remote-as: external
  interfaces:
    - name: p0_if
      network: mybrhbn
    - name: p1_if
      network: mybrhbn
    - name: pf2dpu2_if
      network: mybrhbn

Apply the resource file:

$ oc apply -f hbn.yaml

Example output

dpuserviceconfiguration.svc.dpu.nvidia.com/hbn created

Create the OVN-Kubernetes DPU service configuration

You can create a DPUServiceConfiguration custom resource for the OVN-Kubernetes DPU service. The OVN-Kubernetes service provides pod networking on the DPU.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have created the HBN DPUServiceConfiguration resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named ovn-k.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "ovn"
  serviceConfiguration:
    helmChart:
      values:
        global:
          enableOvnKubeIdentity: false
        k8sAPIServer: https://$HOST_CLUSTER_API:6443
        podNetwork: 10.128.0.0/14/23
        serviceNetwork: 172.30.0.0/16
        hostNetworkNamespace: "openshift-host-network"
        mtu: $OVN_MTU
        dpuManifests:
          kubernetesSecretName: "ovn-dpu"
          vtepCIDR: $VTEP_CIDR
          hostCIDR: $DPU_HOST_CIDR
          ipamPool: "pool1"
          ipamPoolType: "cidrpool"
          ipamVTEPIPIndex: 0
          ipamPFIPIndex: 1
          cniBinDir: "/var/lib/cni/bin/"
          cniConfDir: "/run/multus/cni/net.d"

Apply the resource file:
```
$ envsubst < ovn-k.yaml | oc apply -f -
```

Verification

Verify that the HBN and OVN-Kubernetes service configurations are created:
```
$ oc get dpuserviceconfiguration -n dpf-operator-system
```

Create the DOCA Telemetry Service DPU service configuration

You can create a DPUServiceConfiguration custom resource for the DOCA Telemetry Service. The DOCA Telemetry Service provides metrics collection from the DPUs by using Prometheus.

The DPUServiceTemplate for DOCA Telemetry Service is automatically created and managed by the dpf-hcp-provisioner-operator controller. The operator uses the correct chart and image versions for the installed DPF version. You only need to create the DPUServiceConfiguration resource.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named dts.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: doca-telemetry-service
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "doca-telemetry-service"
  serviceConfiguration:
    configPorts:
      ports:
        - name: httpserverport
          port: 9189
          protocol: TCP
      serviceType: None

Apply the resource file:
```
$ oc apply -f dts.yaml
```

Verification

Verify that the DOCA Telemetry Service configuration is created:

$ oc get dpuserviceconfiguration -n dpf-operator-system doca-telemetry-service

Create the OVN-Kubernetes credential request and role bindings

To authenticate with the management cluster API server, create a DPUServiceCredentialRequest custom resource and the associated role bindings to enable the OVN-Kubernetes DPU service on the hosted cluster. The ClusterRoleBinding grants the required permissions for OVN node network operations.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.

Procedure

Create a file named dpucredentialreq.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceCredentialRequest
metadata:
  name: ovn-dpu
  namespace: dpf-operator-system
spec:
  serviceAccount:
    name: ovn-kubernetes-node-dpu-service
    namespace: openshift-ovn-kubernetes
  duration: 24h
  type: tokenFile
  secret:
    name: ovn-dpu
    namespace: dpf-operator-system
  metadata:
    labels:
      dpu.nvidia.com/image-pull-secret: ""
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: openshift-ovn-kubernetes-node-limited-dpu-service
  namespace: openshift-ovn-kubernetes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: openshift-ovn-kubernetes-node-limited
subjects:
- kind: ServiceAccount
  name: ovn-kubernetes-node-dpu-service
  namespace: openshift-ovn-kubernetes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ovn-kubernetes-node-limited-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: openshift-ovn-kubernetes-node-limited
subjects:
- kind: ServiceAccount
  name: ovn-kubernetes-node-dpu-service
  namespace: openshift-ovn-kubernetes

Apply the resource file:
```
$ oc apply -f dpucredentialreq.yaml
```

Verification

Verify that the credential request and role bindings are created:

$ oc get dpuservicecredentialrequest -n dpf-operator-system

$ oc get clusterrolebinding ovn-kubernetes-node-limited-binding

Create the DPUServiceInterface custom resources

You can create DPUServiceInterface custom resources to define interface objects that are specified in service chains. You must create physical interface resources for the DPU ports and an OVN-Kubernetes interface resource for host workloads.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.

Procedure

Create a file named physical-if.yaml with the following content to define the physical DPU port interfaces:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: p0
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            uplink: "p0"
        spec:
          interfaceType: physical
          physical:
            interfaceName: p0
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: p1
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            uplink: "p1"
        spec:
          interfaceType: physical
          physical:
            interfaceName: p1

Apply the physical interface resource file:
```
$ oc apply -f physical-if.yaml
```

Create a file named ovnk-if.yaml with the following content to define the OVN-Kubernetes interface:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            port: ovn
        spec:
          interfaceType: ovn

Apply the OVN-Kubernetes interface resource file:
```
$ oc apply -f ovnk-if.yaml
```

Verification

Verify that all DPUServiceInterface resources are created:
```
$ oc get dpuserviceinterface -n dpf-operator-system
```

Create the DPUServiceNAD resource

Create a DPUServiceNAD custom resource to define the network attachment available to DPU services on the hosted cluster. The DPUServiceNAD resource maps to an Open vSwitch (OVS) bridge on the DPU and specifies the resource type, IP address management (IPAM) mode, and maximum transmission unit (MTU) configuration:

mybrhbn maps to the br-hbn bridge, used by the HBN service. IPAM is disabled because IP allocation is handled by DPUServiceIPAM.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named dpuservice-nad.yaml with the following content:

apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceNAD
metadata:
  name: mybrhbn
  namespace: dpf-operator-system
spec:
  resourceType: sf
  ipam: false
  bridge: "br-hbn"
  serviceMTU: $NODES_MTU

Apply the resource file:

$ envsubst < dpuservice-nad.yaml | oc apply -f -

Verification

Verify that the DPUServiceNAD resource is created:

$ oc get dpuservicenad mybrhbn -n dpf-operator-system

Example output

NAME       READY   AGE
mybrhbn    True    2m

Create the `DPUServiceIPAM` resources

You can create DPUServiceIPAM custom resources to configure IP address management for DPU services. Two IPAM pools are required: one for the VTEP network used by the high-speed data plane, and one for loopback addresses used by the HBN service.

Prerequisites

You have installed the DPF Operator.
You have created the DPFOperatorConfig resource.
You have set the DPF Operator environment variables. For details, see "DPF Operator installation environment variables".

Procedure

Create a file named dpuservice-ipam.yaml with the following content:

---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
  name: pool1
  namespace: dpf-operator-system
spec:
  ipv4Network:
    network: $VTEP_CIDR
    gatewayIndex: 3
    prefixSize: 29
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
  name: loopback
  namespace: dpf-operator-system
spec:
  ipv4Network:
    network: "11.0.0.0/24"
    prefixSize: 32

Apply the resource file:

$ envsubst < dpuservice-ipam.yaml | oc apply -f -

Verification

Verify that the DPUServiceIPAM resources are created:
```
$ oc get dpuserviceipam -n dpf-operator-system
```

DPF Operator installation environment variables

Install the DPF Operator

Create the DPFOperatorConfig custom resource

Create the NodeSRIOVDevicePluginConfig custom resource

Create the DPUFlavor custom resource

Create the BFB resource

Create the DPUDeployment custom resource

Create the HBN DPU service configuration

Create the OVN-Kubernetes DPU service configuration

Create the DOCA Telemetry Service DPU service configuration

Create the OVN-Kubernetes credential request and role bindings

Create the DPUServiceInterface custom resources

Create the DPUServiceNAD resource

Create the DPUServiceIPAM resources

Create the `BFB` resource

Create the `DPUServiceIPAM` resources