×

Approaches to managing an additional network

You can manage the life cycle of an additional network by two approaches. Each approach is mutually exclusive and you can only use one approach for managing an additional network at a time. For either approach, the additional network is managed by a Container Network Interface (CNI) plugin that you configure.

For an additional network, IP addresses are provisioned through an IP Address Management (IPAM) CNI plugin that you configure as part of the additional network. The IPAM plugin supports a variety of IP address assignment approaches including DHCP and static assignment.

  • Modify the Cluster Network Operator (CNO) configuration: The CNO automatically creates and manages the NetworkAttachmentDefinition object. In addition to managing the object lifecycle the CNO ensures a DHCP is available for an additional network that uses a DHCP assigned IP address.

  • Applying a YAML manifest: You can manage the additional network directly by creating an NetworkAttachmentDefinition object. This approach allows for the chaining of CNI plugins.

When deploying OKD nodes with multiple network interfaces on OpenStack with OVN SDN, DNS configuration of the secondary interface might take precedence over the DNS configuration of the primary interface. In this case, remove the DNS nameservers for the subnet id that is attached to the secondary interface:

$ openstack subnet set --dns-nameserver 0.0.0.0 <subnet_id>

Configuration for an additional network attachment

An additional network is configured by using the NetworkAttachmentDefinition API in the k8s.cni.cncf.io API group.

Do not store any sensitive information or a secret in the NetworkAttachmentDefinition object because this information is accessible by the project administration user.

The configuration for the API is described in the following table:

Table 1. NetworkAttachmentDefinition API fields
Field Type Description

metadata.name

string

The name for the additional network.

metadata.namespace

string

The namespace that the object is associated with.

spec.config

string

The CNI plugin configuration in JSON format.

Configuration of an additional network through the Cluster Network Operator

The configuration for an additional network attachment is specified as part of the Cluster Network Operator (CNO) configuration.

The following YAML describes the configuration parameters for managing an additional network with the CNO:

Cluster Network Operator configuration
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  # ...
  additionalNetworks: (1)
  - name: <name> (2)
    namespace: <namespace> (3)
    rawCNIConfig: |- (4)
      {
        ...
      }
    type: Raw
1 An array of one or more additional network configurations.
2 The name for the additional network attachment that you are creating. The name must be unique within the specified namespace.
3 The namespace to create the network attachment in. If you do not specify a value, then the default namespace is used.
4 A CNI plugin configuration in JSON format.

Configuration of an additional network from a YAML manifest

The configuration for an additional network is specified from a YAML configuration file, such as in the following example:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: <name> (1)
spec:
  config: |- (2)
    {
      ...
    }
1 The name for the additional network attachment that you are creating.
2 A CNI plugin configuration in JSON format.

Configurations for additional network types

The specific configuration fields for additional networks is described in the following sections.

Configuration for a bridge additional network

The following object describes the configuration parameters for the bridge CNI plugin:

Table 2. Bridge CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: bridge.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

bridge

string

Optional: Specify the name of the virtual bridge to use. If the bridge interface does not exist on the host, it is created. The default value is cni0.

ipMasq

boolean

Optional: Set to true to enable IP masquerading for traffic that leaves the virtual network. The source IP address for all traffic is rewritten to the bridge’s IP address. If the bridge does not have an IP address, this setting has no effect. The default value is false.

isGateway

boolean

Optional: Set to true to assign an IP address to the bridge. The default value is false.

isDefaultGateway

boolean

Optional: Set to true to configure the bridge as the default gateway for the virtual network. The default value is false. If isDefaultGateway is set to true, then isGateway is also set to true automatically.

forceAddress

boolean

Optional: Set to true to allow assignment of a previously assigned IP address to the virtual bridge. When set to false, if an IPv4 address or an IPv6 address from overlapping subsets is assigned to the virtual bridge, an error occurs. The default value is false.

hairpinMode

boolean

Optional: Set to true to allow the virtual bridge to send an Ethernet frame back through the virtual port it was received on. This mode is also known as reflective relay. The default value is false.

promiscMode

boolean

Optional: Set to true to enable promiscuous mode on the bridge. The default value is false.

vlan

string

Optional: Specify a virtual LAN (VLAN) tag as an integer value. By default, no VLAN tag is assigned.

preserveDefaultVlan

string

Optional: Indicates whether the default vlan must be preserved on the veth end connected to the bridge. Defaults to true.

vlanTrunk

list

Optional: Assign a VLAN trunk tag. The default value is none.

mtu

string

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

enabledad

boolean

Optional: Enables duplicate address detection for the container side veth. The default value is false.

macspoofchk

boolean

Optional: Enables mac spoof check, limiting the traffic originating from the container to the mac address of the interface. The default value is false.

The VLAN parameter configures the VLAN tag on the host end of the veth and also enables the vlan_filtering feature on the bridge interface.

To configure uplink for a L2 network you need to allow the vlan on the uplink interface by using the following command:

$  bridge vlan add vid VLAN_ID dev DEV

bridge configuration example

The following example configures an additional network named bridge-net:

{
  "cniVersion": "0.3.1",
  "name": "bridge-net",
  "type": "bridge",
  "isGateway": true,
  "vlan": 2,
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a host device additional network

Specify your network device by setting only one of the following parameters: device,hwaddr, kernelpath, or pciBusID.

The following object describes the configuration parameters for the host-device CNI plugin:

Table 3. Host device CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: host-device.

device

string

Optional: The name of the device, such as eth0.

hwaddr

string

Optional: The device hardware MAC address.

kernelpath

string

Optional: The Linux kernel device path, such as /sys/devices/pci0000:00/0000:00:1f.6.

pciBusID

string

Optional: The PCI address of the network device, such as 0000:00:1f.6.

host-device configuration example

The following example configures an additional network named hostdev-net:

{
  "cniVersion": "0.3.1",
  "name": "hostdev-net",
  "type": "host-device",
  "device": "eth1"
}

Configuration for an VLAN additional network

The following object describes the configuration parameters for the VLAN CNI plugin:

Table 4. VLAN CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: vlan.

master

string

The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

vlanId

integer

Set the id of the vlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

dns

integer

Optional: DNS information to return, for example, a priority-ordered list of DNS nameservers.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

vlan configuration example

The following example configures an additional network named vlan-net:

{
  "name": "vlan-net",
  "cniVersion": "0.3.1",
  "type": "vlan",
  "master": "eth0",
  "mtu": 1500,
  "vlanId": 5,
  "linkInContainer": false,
  "ipam": {
      "type": "host-local",
      "subnet": "10.1.1.0/24"
  },
  "dns": {
      "nameservers": [ "10.1.1.1", "8.8.8.8" ]
  }
}

Configuration for an IPVLAN additional network

The following object describes the configuration parameters for the IPVLAN CNI plugin:

Table 5. IPVLAN CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: ipvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. This is required unless the plugin is chained.

mode

string

Optional: The operating mode for the virtual network. The value must be l2, l3, or l3s. The default value is l2.

master

string

Optional: The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

  • The ipvlan object does not allow virtual interfaces to communicate with the master interface. Therefore the container will not be able to reach the host by using the ipvlan interface. Be sure that the container joins a network that provides connectivity to the host, such as a network supporting the Precision Time Protocol (PTP).

  • A single master interface cannot simultaneously be configured to use both macvlan and ipvlan.

  • For IP allocation schemes that cannot be interface agnostic, the ipvlan plugin can be chained with an earlier plugin that handles this logic. If the master is omitted, then the previous result must contain a single interface name for the ipvlan plugin to enslave. If ipam is omitted, then the previous result is used to configure the ipvlan interface.

ipvlan configuration example

The following example configures an additional network named ipvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "ipvlan-net",
  "type": "ipvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "l3",
  "ipam": {
    "type": "static",
    "addresses": [
       {
         "address": "192.168.10.10/24"
       }
    ]
  }
}

Configuration for a MACVLAN additional network

The following object describes the configuration parameters for the macvlan CNI plugin:

Table 6. MACVLAN CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: macvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mode

string

Optional: Configures traffic visibility on the virtual network. Must be either bridge, passthru, private, or vepa. If a value is not provided, the default value is bridge.

master

string

Optional: The host network interface to associate with the newly created macvlan interface. If a value is not specified, then the default route interface is used.

mtu

string

Optional: The maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

If you specify the master key for the plugin configuration, use a different physical network interface than the one that is associated with your primary network plugin to avoid possible conflicts.

macvlan configuration example

The following example configures an additional network named macvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "macvlan-net",
  "type": "macvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "bridge",
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a TAP additional network

The following object describes the configuration parameters for the TAP CNI plugin:

Table 7. TAP CNI plugin JSON configuration object
Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The value for the name parameter you provided previously for the CNO configuration.

type

string

The name of the CNI plugin to configure: tap.

mac

string

Optional: Request the specified MAC address for the interface.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

selinuxcontext

string

Optional: The SELinux context to associate with the tap device.

The value system_u:system_r:container_t:s0 is required for OKD.

multiQueue

boolean

Optional: Set to true to enable multi-queue.

owner

integer

Optional: The user owning the tap device.

group

integer

Optional: The group owning the tap device.

bridge

string

Optional: Set the tap device as a port of an already existing bridge.

Tap configuration example

The following example configures an additional network named mynet:

{
 "name": "mynet",
 "cniVersion": "0.3.1",
 "type": "tap",
 "mac": "00:11:22:33:44:55",
 "mtu": 1500,
 "selinuxcontext": "system_u:system_r:container_t:s0",
 "multiQueue": true,
 "owner": 0,
 "group": 0
 "bridge": "br1"
}

To create the tap device with the container_t SELinux context, enable the container_use_devices boolean on the host by using the Machine Config Operator (MCO).

Additional resources

Configuration for an OVN-Kubernetes additional network

The Red Hat OpenShift Networking OVN-Kubernetes network plugin allows the configuration of secondary network interfaces for pods. To configure secondary network interfaces, you must define the configurations in the NetworkAttachmentDefinition custom resource definition (CRD).

Configuration for an OVN-Kubernetes additional network is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The following sections provide example configurations for each of the topologies that OVN-Kubernetes currently allows for secondary networks.

Networks names must be unique. For example, creating multiple NetworkAttachmentDefinition CRDs with different configurations that reference the same network is unsupported.

OVN-Kubernetes network plugin JSON configuration table

The following table describes the configuration parameters for the OVN-Kubernetes CNI network plugin:

Table 8. OVN-Kubernetes network plugin JSON configuration table
Field Type Description

cniVersion

string

The CNI specification version. The required value is 0.3.1.

name

string

The name of the network. These networks are not namespaced. For example, you can have a network named l2-network referenced from two different NetworkAttachmentDefinitions that exist on two different namespaces. This ensures that pods making use of the NetworkAttachmentDefinition on their own different namespaces can communicate over the same secondary network. However, those two different NetworkAttachmentDefinitions must also share the same network specific parameters such as topology, subnets, mtu, and excludeSubnets.

type

string

The name of the CNI plugin to configure. The required value is ovn-k8s-cni-overlay.

topology

string

The topological configuration for the network. The required value is layer2.

subnets

string

The subnet to use for the network across the cluster. When specifying layer2 for the topology, only include the CIDR for the node. For example, 10.100.200.0/24.

For "topology":"layer2" deployments, IPv6 (2001:DBB::/64) and dual-stack (192.168.100.0/24,2001:DBB::/64) subnets are supported.

mtu

string

The maximum transmission unit (MTU) to the specified value. The default value, 1300, is automatically set by the kernel.

netAttachDefName

string

The metadata namespace and name of the network attachment definition object where this configuration is included. For example, if this configuration is defined in a NetworkAttachmentDefinition in namespace ns1 named l2-network, this should be set to ns1/l2-network.

excludeSubnets

string

A comma-separated list of CIDRs and IPs. IPs are removed from the assignable IP pool, and are never passed to the pods. When omitted, the logical switch implementing the network only provides layer 2 communication, and users must configure IPs for the pods. Port security only prevents MAC spoofing.

Configuration for a switched topology

The switched (layer 2) topology networks interconnect the workloads through a cluster-wide logical switch. This configuration can be used for IPv6 and dual-stack deployments.

Layer 2 switched topology networks only allow for the transfer of data packets between pods within a cluster.

The following NetworkAttachmentDefinition custom resource definition (CRD) YAML describes the fields needed to configure a switched secondary network.

    {
            "cniVersion": "0.3.1",
            "name": "l2-network",
            "type": "ovn-k8s-cni-overlay",
            "topology":"layer2",
            "subnets": "10.100.200.0/24",
            "mtu": 1300,
            "netAttachDefName": "ns1/l2-network",
            "excludeSubnets": "10.100.200.0/29"
    }

Configuring pods for additional networks

You must specify the secondary network attachments through the k8s.v1.cni.cncf.io/networks annotation.

The following example provisions a pod with two secondary attachments, one for each of the attachment configurations presented in this guide.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: l2-network
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container

Configuring pods with a static IP address

The following example provisions a pod with a static IP address.

  • You can only specify the IP address for a pod’s secondary network attachment for layer 2 attachments.

  • Specifying a static IP address for the pod is only possible when the attachment configuration does not feature subnets.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "l2-network", (1)
        "mac": "02:03:04:05:06:07", (2)
        "interface": "myiface1", (3)
        "ips": [
          "192.0.2.20/24"
          ] (4)
      }
    ]'
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container
1 The name of the network. This value must be unique across all NetworkAttachmentDefinitions.
2 The MAC address to be assigned for the interface.
3 The name of the network interface to be created for the pod.
4 The IP addresses to be assigned to the network interface.

Configuration of IP address assignment for an additional network

The IP address management (IPAM) Container Network Interface (CNI) plugin provides IP addresses for other CNI plugins.

You can use the following IP address assignment types:

  • Static assignment.

  • Dynamic assignment through a DHCP server. The DHCP server you specify must be reachable from the additional network.

  • Dynamic assignment through the Whereabouts IPAM CNI plugin.

Static IP address assignment configuration

The following table describes the configuration for static IP address assignment:

Table 9. ipam static configuration object
Field Type Description

type

string

The IPAM address type. The value static is required.

addresses

array

An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported.

routes

array

An array of objects specifying routes to configure inside the pod.

dns

array

Optional: An array of objects specifying the DNS configuration.

The addresses array requires objects with the following fields:

Table 10. ipam.addresses[] array
Field Type Description

address

string

An IP address and network prefix that you specify. For example, if you specify 10.10.21.10/24, then the additional network is assigned an IP address of 10.10.21.10 and the netmask is 255.255.255.0.

gateway

string

The default gateway to route egress network traffic to.

Table 11. ipam.routes[] array
Field Type Description

dst

string

The IP address range in CIDR format, such as 192.168.17.0/24 or 0.0.0.0/0 for the default route.

gw

string

The gateway where network traffic is routed.

Table 12. ipam.dns object
Field Type Description

nameservers

array

An array of one or more IP addresses for to send DNS queries to.

domain

array

The default domain to append to a hostname. For example, if the domain is set to example.com, a DNS lookup query for example-host is rewritten as example-host.example.com.

search

array

An array of domain names to append to an unqualified hostname, such as example-host, during a DNS lookup query.

Static IP address assignment configuration example
{
  "ipam": {
    "type": "static",
      "addresses": [
        {
          "address": "191.168.1.7/24"
        }
      ]
  }
}

Dynamic IP address (DHCP) assignment configuration

The following JSON describes the configuration for dynamic IP address address assignment with DHCP.

Renewal of DHCP leases

A pod obtains its original DHCP lease when it is created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.

To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:

Example shim network attachment definition
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: dhcp-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
        "name": "dhcp-shim",
        "cniVersion": "0.3.1",
        "type": "bridge",
        "ipam": {
          "type": "dhcp"
        }
      }
  # ...
Table 13. ipam DHCP configuration object
Field Type Description

type

string

The IPAM address type. The value dhcp is required.

Dynamic IP address (DHCP) assignment configuration example
{
  "ipam": {
    "type": "dhcp"
  }
}

Dynamic IP address assignment configuration with Whereabouts

The Whereabouts CNI plugin allows the dynamic assignment of an IP address to an additional network without the use of a DHCP server.

The following table describes the configuration for dynamic IP address assignment with Whereabouts:

Table 14. ipam whereabouts configuration object
Field Type Description

type

string

The IPAM address type. The value whereabouts is required.

range

string

An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses.

exclude

array

Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned.

Dynamic IP address assignment configuration example that uses Whereabouts
{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/27",
    "exclude": [
       "192.0.2.192/30",
       "192.0.2.196/32"
    ]
  }
}

Creating a Whereabouts reconciler daemon set

The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster using the Whereabouts IP Address Management (IPAM) solution. It ensures that each pods gets a unique IP address from the specified IP address range. It also handles IP address releases when pods are deleted or scaled down.

You can also use a NetworkAttachmentDefinition custom resource for dynamic IP address assignment.

The Whereabouts reconciler daemon set is automatically created when you configure an additional network through the Cluster Network Operator. It is not automatically created when you configure an additional network from a YAML manifest.

To trigger the deployment of the Whereabouts reconciler daemonset, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource file.

Use the following procedure to deploy the Whereabouts reconciler daemonset.

Procedure
  1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

    $ oc edit network.operator.openshift.io cluster
  2. Modify the additionalNetworks parameter in the CR to add the whereabouts-shim network attachment definition. For example:

    apiVersion: operator.openshift.io/v1
    kind: Network
    metadata:
      name: cluster
    spec:
      additionalNetworks:
      - name: whereabouts-shim
        namespace: default
        rawCNIConfig: |-
          {
           "name": "whereabouts-shim",
           "cniVersion": "0.3.1",
           "type": "bridge",
           "ipam": {
             "type": "whereabouts"
           }
          }
        type: Raw
  3. Save the file and exit the text editor.

  4. Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

    $ oc get all -n openshift-multus | grep whereabouts-reconciler
    Example output
    pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
    pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
    pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
    pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
    pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
    pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
    daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Creating an additional network attachment with the Cluster Network Operator

The Cluster Network Operator (CNO) manages additional network definitions. When you specify an additional network to create, the CNO creates the NetworkAttachmentDefinition object automatically.

Do not edit the NetworkAttachmentDefinition objects that the Cluster Network Operator manages. Doing so might disrupt network traffic on your additional network.

Prerequisites
  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure
  1. Optional: Create the namespace for the additional networks:

    $ oc create namespace <namespace_name>
  2. To edit the CNO configuration, enter the following command:

    $ oc edit networks.operator.openshift.io cluster
  3. Modify the CR that you are creating by adding the configuration for the additional network that you are creating, as in the following example CR.

    apiVersion: operator.openshift.io/v1
    kind: Network
    metadata:
      name: cluster
    spec:
      # ...
      additionalNetworks:
      - name: tertiary-net
        namespace: namespace2
        type: Raw
        rawCNIConfig: |-
          {
            "cniVersion": "0.3.1",
            "name": "tertiary-net",
            "type": "ipvlan",
            "master": "eth1",
            "mode": "l2",
            "ipam": {
              "type": "static",
              "addresses": [
                {
                  "address": "192.168.1.23/24"
                }
              ]
            }
          }
  4. Save your changes and quit the text editor to commit your changes.

Verification
  • Confirm that the CNO created the NetworkAttachmentDefinition object by running the following command. There might be a delay before the CNO creates the object.

    $ oc get network-attachment-definitions -n <namespace>

    where:

    <namespace>

    Specifies the namespace for the network attachment that you added to the CNO configuration.

    Example output
    NAME                 AGE
    test-network-1       14m

Creating an additional network attachment by applying a YAML manifest

Prerequisites
  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure
  1. Create a YAML file with your additional network configuration, such as in the following example:

    apiVersion: k8s.cni.cncf.io/v1
    kind: NetworkAttachmentDefinition
    metadata:
      name: next-net
    spec:
      config: |-
        {
          "cniVersion": "0.3.1",
          "name": "work-network",
          "type": "host-device",
          "device": "eth1",
          "ipam": {
            "type": "dhcp"
          }
        }
  2. To create the additional network, enter the following command:

    $ oc apply -f <file>.yaml

    where:

    <file>

    Specifies the name of the file contained the YAML manifest.

About configuring the master interface in the container network namespace

In OKD 4.14 and later, the ability to allow users to create a MAC-VLAN, IP-VLAN, and VLAN subinterface based on a master interface in a container namespace is now generally available.

This feature allows you to create the master interfaces as part of the pod network configuration in a separate network attachment definition. You can then base the VLAN, MACVLAN, or IPVLAN on this interface without requiring the knowledge of the network configuration of the node.

To ensure the use of a container namespace master interface specify the linkInContainer and set the value to true in the VLAN, MACVLAN, or IPVLAN plugin configuration depending on the particular type of additional network.

An example use case for utilizing this feature is to create multiple VLANs based on SR-IOV VFs. To do so, begin by creating an SR-IOV network and then define the network attachments for the VLAN interfaces.

The following example shows how to configure the setup illustrated in this diagram.

Creating VLANs
Figure 1. Creating VLANs
Prerequisites
  • You installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

  • You have installed the SR-IOV Network Operator.

Procedure
  1. Create a dedicated container namespace where you want to deploy your pod by using the following command:

    $ oc new-project test-namespace
  2. Create an SR-IOV node policy:

    1. Create an SriovNetworkNodePolicy object, and then save the YAML in the sriov-node-network-policy.yaml file:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetworkNodePolicy
      metadata:
       name: sriovnic
       namespace: openshift-sriov-network-operator
      spec:
       deviceType: netdevice
       isRdma: false
       needVhostNet: true
       nicSelector:
         vendor: "15b3" (1)
         deviceID: "101b" (2)
         rootDevices: ["00:05.0"]
       numVfs: 10
       priority: 99
       resourceName: sriovnic
       nodeSelector:
          feature.node.kubernetes.io/network-sriov.capable: "true"

      The SR-IOV network node policy configuration example, with the setting deviceType: netdevice, is tailored specifically for Mellanox Network Interface Cards (NICs).

      1 The vendor hexadecimal code of the SR-IOV network device. The value 15b3 is associated with a Mellanox NIC.
      2 The device hexadecimal code of the SR-IOV network device.
    2. Apply the YAML by running the following command:

      $ oc apply -f sriov-node-network-policy.yaml

      Applying this might take some time due to the node requiring a reboot.

  3. Create an SR-IOV network:

    1. Create the SriovNetwork custom resource (CR) for the additional SR-IOV network attachment as in the following example CR. Save the YAML as the file sriov-network-attachment.yaml:

      apiVersion: sriovnetwork.openshift.io/v1
      kind: SriovNetwork
      metadata:
       name: sriov-network
       namespace: openshift-sriov-network-operator
      spec:
       networkNamespace: test-namespace
       resourceName: sriovnic
       spoofChk: "off"
       trust: "on"
    2. Apply the YAML by running the following command:

      $ oc apply -f sriov-network-attachment.yaml
  4. Create a YAML file for the VLAN additional network configuration and then save the YAML in the vlan100-additional-network-configuration.yaml file:

    apiVersion: k8s.cni.cncf.io/v1
    kind: NetworkAttachmentDefinition
    metadata:
      name: vlan-100
      namespace: test-namespace
    spec:
      config: |
        {
          "cniVersion": "0.4.0",
          "name": "vlan-100",
          "plugins": [
            {
              "type": "vlan",
              "master": "ext0", (1)
              "mtu": 1500,
              "vlanId": 100,
              "linkInContainer": true, (2)
              "ipam": {"type": "whereabouts", "ipRanges": [{"range": "1.1.1.0/24"}]}
            }
          ]
        }
    1 The VLAN configuration needs to specify the master name. This can be configured in the pod networks annotation.
    2 The linkInContainer parameter must be specified.
  5. Apply the YAML by running the following command:

    $ oc apply -f vlan100-additional-network-configuration.yaml
  6. Create a pod definition by using the earlier specified networks and then save the YAML in the pod-a.yaml file.

    The manifest below includes 2 resources:

    • Namespace with security labels

    • Pod definition with appropriate network annotation

    apiVersion: v1
    kind: Namespace
    metadata:
      name: test-namespace
      labels:
        pod-security.kubernetes.io/enforce: privileged
        pod-security.kubernetes.io/audit: privileged
        pod-security.kubernetes.io/warn: privileged
        security.openshift.io/scc.podSecurityLabelSync: "false"
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx-pod
      namespace: test-namespace
      annotations:
        k8s.v1.cni.cncf.io/networks: '[
          {
            "name": "sriov-network",
            "namespace": "test-namespace",
            "interface": "ext0" (1)
          },
          {
            "name": "vlan-100",
            "namespace": "test-namespace",
            "interface": "ext0.100"
          }
        ]'
    spec:
      securityContext:
        runAsNonRoot: true
      containers:
        - name: nginx-container
          image: nginxinc/nginx-unprivileged:latest
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          ports:
            - containerPort: 80
          seccompProfile:
            type: "RuntimeDefault"
    1 The name to be used as the master for the VLAN interface.
  7. Apply the YAML by running the following command:

    $ oc apply -f pod-a.yaml
  8. Get detailed information about the nginx-pod within the test-namespace by running the following command:

    $ oc describe pods nginx-pod -n test-namespace
    Expected output
    Name:         nginx-pod
    Namespace:    test-namespace
    Priority:     0
    Node:         worker-1/10.46.186.105
    Start Time:   Mon, 14 Aug 2023 16:23:13 -0400
    Labels:       <none>
    Annotations:  k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.131.0.26/23"],"mac_address":"0a:58:0a:83:00:1a","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.131.0.26"
                        ],
                        "mac": "0a:58:0a:83:00:1a",
                        "default": true,
                        "dns": {}
                    },{
                        "name": "test-namespace/sriov-network",
                        "interface": "ext0",
                        "mac": "6e:a7:5e:3f:49:1b",
                        "dns": {},
                        "device-info": {
                            "type": "pci",
                            "version": "1.0.0",
                            "pci": {
                                "pci-address": "0000:d8:00.2"
                            }
                        }
                    },{
                        "name": "test-namespace/vlan-100",
                        "interface": "ext0.100",
                        "ips": [
                            "1.1.1.1"
                        ],
                        "mac": "6e:a7:5e:3f:49:1b",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks:
                    [ { "name": "sriov-network", "namespace": "test-namespace", "interface": "ext0" }, { "name": "vlan-100", "namespace": "test-namespace", "i...
                  openshift.io/scc: privileged
    Status:       Running
    IP:           10.131.0.26
    IPs:
      IP:  10.131.0.26