Creating secondary networks on OVN-Kubernetes - Multiple networks | Networking

Configuration for an OVN-Kubernetes additional network

As a cluster administrator, you can configure an additional secondary network for your cluster using the NetworkAttachmentDefinition (NAD) resource.

Support for user-defined networks as a secondary network will be added in a future version of OKD.

Configuration for an OVN-Kubernetes additional network

The Red Hat OpenShift Networking OVN-Kubernetes network plugin allows the configuration of secondary network interfaces for pods. To configure secondary network interfaces, you must define the configurations in the NetworkAttachmentDefinition custom resource definition (CRD).

Pod and multi-network policy creation might remain in a pending state until the OVN-Kubernetes control plane agent in the nodes processes the associated network-attachment-definition CRD.

You can configure an OVN-Kubernetes additional network in either layer 2 or localnet topologies.

A layer 2 topology supports east-west cluster traffic, but does not allow access to the underlying physical network.
A localnet topology allows connections to the physical network, but requires additional configuration of the underlying Open vSwitch (OVS) bridge on cluster nodes.

The following sections provide example configurations for each of the topologies that OVN-Kubernetes currently allows for secondary networks.

Networks names must be unique. For example, creating multiple NetworkAttachmentDefinition CRDs with different configurations that reference the same network is unsupported.

Supported platforms for OVN-Kubernetes additional network

You can use an OVN-Kubernetes additional network with the following supported platforms:

Bare metal
IBM Power®
IBM Z®
IBM® LinuxONE
VMware vSphere
OpenStack

OVN-Kubernetes network plugin JSON configuration table

The following table describes the configuration parameters for the OVN-Kubernetes CNI network plugin:

Table 1. OVN-Kubernetes network plugin JSON configuration table
Field	Type	Description
`cniVersion`	`string`	The CNI specification version. The required value is `0.3.1`.
`name`	`string`	The name of the network. These networks are not namespaced. For example, you can have a network named `l2-network` referenced from two different `NetworkAttachmentDefinition` CRDs that exist on two different namespaces. This ensures that pods making use of the `NetworkAttachmentDefinition` CRD on their own different namespaces can communicate over the same secondary network. However, those two different `NetworkAttachmentDefinition` CRDs must also share the same network specific parameters such as `topology`, `subnets`, `mtu`, and `excludeSubnets`.
`type`	`string`	The name of the CNI plugin to configure. This value must be set to `ovn-k8s-cni-overlay`.
`topology`	`string`	The topological configuration for the network. Must be one of `layer2` or `localnet`.
`subnets`	`string`	The subnet to use for the network across the cluster. For `"topology":"layer2"` deployments, IPv6 (`2001:DBB::/64`) and dual-stack (`192.168.100.0/24,2001:DBB::/64`) subnets are supported. When omitted, the logical switch implementing the network only provides layer 2 communication, and users must configure IP addresses for the pods. Port security only prevents MAC spoofing.
`mtu`	`string`	The maximum transmission unit (MTU). The default value, `1300`, is automatically set by the kernel.
`netAttachDefName`	`string`	The metadata `namespace` and `name` of the network attachment definition CRD where this configuration is included. For example, if this configuration is defined in a `NetworkAttachmentDefinition` CRD in namespace `ns1` named `l2-network`, this should be set to `ns1/l2-network`.
`excludeSubnets`	`string`	A comma-separated list of CIDRs and IP addresses. IP addresses are removed from the assignable IP address pool and are never passed to the pods.
`vlanID`	`integer`	If topology is set to `localnet`, the specified VLAN tag is assigned to traffic from this additional network. The default is to not assign a VLAN tag.

Compatibility with multi-network policy

The multi-network policy API, which is provided by the MultiNetworkPolicy custom resource definition (CRD) in the k8s.cni.cncf.io API group, is compatible with an OVN-Kubernetes secondary network. When defining a network policy, the network policy rules that can be used depend on whether the OVN-Kubernetes secondary network defines the subnets field. Refer to the following table for details:

Table 2. Supported multi-network policy selectors based on `subnets` CNI configuration
`subnets` field specified	Allowed multi-network policy selectors
Yes	`podSelector` and `namespaceSelector` `ipBlock`
No	`ipBlock`

For example, the following multi-network policy is valid only if the subnets field is defined in the additional network CNI configuration for the additional network named blue2:

Example multi-network policy that uses a pod selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name: allow-same-namespace
  annotations:
    k8s.v1.cni.cncf.io/policy-for: blue2
spec:
  podSelector:
  ingress:
  - from:
    - podSelector: {}

The following example uses the ipBlock network policy selector, which is always valid for an OVN-Kubernetes additional network:

Example multi-network policy that uses an IP block selector

apiVersion: k8s.cni.cncf.io/v1beta1
kind: MultiNetworkPolicy
metadata:
  name:  ingress-ipblock
  annotations:
    k8s.v1.cni.cncf.io/policy-for: default/flatl2net
spec:
  podSelector:
    matchLabels:
      name: access-control
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 10.200.0.0/30

Configuration for a localnet switched topology

The switched localnet topology interconnects the workloads created as Network Attachment Definitions (NADs) through a cluster-wide logical switch to a physical network.

You must map an additional network to the OVN bridge to use it as an OVN-Kubernetes additional network. Bridge mappings allow network traffic to reach the physical network. A bridge mapping associates a physical network name, also known as an interface label, to a bridge created with Open vSwitch (OVS).

You can create an NodeNetworkConfigurationPolicy object, part of the nmstate.io/v1 API group, to declaratively create the mapping. This API is provided by the NMState Operator. By using this API you can apply the bridge mapping to nodes that match your specified nodeSelector expression, such as node-role.kubernetes.io/worker: ''.

When attaching an additional network, you can either use the existing br-ex bridge or create a new bridge. Which approach to use depends on your specific network infrastructure.

If your nodes include only a single network interface, you must use the existing bridge. This network interface is owned and managed by OVN-Kubernetes and you must not remove it from the br-ex bridge or alter the interface configuration. If you remove or alter the network interface, your cluster network will stop working correctly.
If your nodes include several network interfaces, you can attach a different network interface to a new bridge, and use that for your additional network. This approach provides for traffic isolation from your primary cluster network.

The localnet1 network is mapped to the br-ex bridge in the following example:

Example mapping for sharing a bridge

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: mapping (1)
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' (2)
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: localnet1 (3)
        bridge: br-ex (4)
        state: present (5)

1	The name for the configuration object.
2	A node selector that specifies the nodes to apply the node network configuration policy to.
3	The name for the additional network from which traffic is forwarded to the OVS bridge. This additional network must match the name of the `spec.config.name` field of the `NetworkAttachmentDefinition` CRD that defines the OVN-Kubernetes additional network.
4	The name of the OVS bridge on the node. This value is required only if you specify `state: present`.
5	The state for the mapping. Must be either `present` to add the bridge or `absent` to remove the bridge. The default value is `present`.

In the following example, the localnet2 network interface is attached to the ovs-br1 bridge. Through this attachment, the network interface is available to the OVN-Kubernetes network plugin as an additional network.

Example mapping for nodes with multiple interfaces

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ovs-br1-multiple-networks (1)
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' (2)
  desiredState:
    interfaces:
    - name: ovs-br1 (3)
      description: |-
        A dedicated OVS bridge with eth1 as a port
        allowing all VLANs and untagged traffic
      type: ovs-bridge
      state: up
      bridge:
        allow-extra-patch-ports: true
        options:
          stp: false
        port:
        - name: eth1 (4)
    ovn:
      bridge-mappings:
      - localnet: localnet2 (5)
        bridge: ovs-br1 (6)
        state: present (7)

1	The name for the configuration object.
2	A node selector that specifies the nodes to apply the node network configuration policy to.
3	A new OVS bridge, separate from the default bridge used by OVN-Kubernetes for all cluster traffic.
4	A network device on the host system to associate with this new OVS bridge.
5	The name for the additional network from which traffic is forwarded to the OVS bridge. This additional network must match the name of the `spec.config.name` field of the `NetworkAttachmentDefinition` CRD that defines the OVN-Kubernetes additional network.
6	The name of the OVS bridge on the node. This value is required only if you specify `state: present`.
7	The state for the mapping. Must be either `present` to add the bridge or `absent` to remove the bridge. The default value is `present`.

This declarative approach is recommended because the NMState Operator applies additional network configuration to all nodes specified by the node selector automatically and transparently.

The following JSON example configures a localnet secondary network:

{
  "cniVersion": "0.3.1",
  "name": "ns1-localnet-network",
  "type": "ovn-k8s-cni-overlay",
  "topology":"localnet",
  "subnets": "202.10.130.112/28",
  "vlanID": 33,
  "mtu": 1500,
  "netAttachDefName": "ns1/localnet-network"
  "excludeSubnets": "10.100.200.0/29"
}

Configuration for a layer 2 switched topology

The switched (layer 2) topology networks interconnect the workloads through a cluster-wide logical switch. This configuration can be used for IPv6 and dual-stack deployments.

Layer 2 switched topology networks only allow for the transfer of data packets between pods within a cluster.

The following JSON example configures a switched secondary network:

{
  "cniVersion": "0.3.1",
  "name": "l2-network",
  "type": "ovn-k8s-cni-overlay",
  "topology":"layer2",
  "subnets": "10.100.200.0/24",
  "mtu": 1300,
  "netAttachDefName": "ns1/l2-network",
  "excludeSubnets": "10.100.200.0/29"
}

Configuring pods for additional networks

You must specify the secondary network attachments through the k8s.v1.cni.cncf.io/networks annotation.

The following example provisions a pod with two secondary attachments, one for each of the attachment configurations presented in this guide.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: l2-network
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container

Configuring pods with a static IP address

The following example provisions a pod with a static IP address.

You can only specify the IP address for a pod’s secondary network attachment for layer 2 attachments.
Specifying a static IP address for the pod is only possible when the attachment configuration does not feature subnets.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "l2-network", (1)
        "mac": "02:03:04:05:06:07", (2)
        "interface": "myiface1", (3)
        "ips": [
          "192.0.2.20/24"
          ] (4)
      }
    ]'
  name: tinypod
  namespace: ns1
spec:
  containers:
  - args:
    - pause
    image: k8s.gcr.io/e2e-test-images/agnhost:2.36
    imagePullPolicy: IfNotPresent
    name: agnhost-container

1	The name of the network. This value must be unique across all `NetworkAttachmentDefinition` CRDs.
2	The MAC address to be assigned for the interface.
3	The name of the network interface to be created for the pod.
4	The IP addresses to be assigned to the network interface.