Attaching a pod to an additional network - Multiple networks | Networking

Adding a pod to a secondary network
- Specifying pod-specific addressing and routing options

You can attach a pod to a secondary network. Attaching a pod to a secondary network enables the pod to use additional network interfaces beyond the primary cluster network.

Adding a pod to a secondary network

You can add a pod to a secondary network. The pod continues to send normal cluster-related network traffic over the default network.

When a pod is created, a secondary networks is attached to the pod. However, if a pod already exists, you cannot attach a secondary network to it.

The pod must be in the same namespace as the secondary network.

Prerequisites

Install the OpenShift CLI (oc).
Log in to the cluster.

Procedure

Add an annotation to the Pod object. Only one of the following annotation formats can be used:
1. To attach a secondary network without any customization, add an annotation with the following format:
  metadata: annotations: k8s.v1.cni.cncf.io/networks: <network>[,<network>,...]
  where:
  
  k8s.v1.cni.cncf.io/networks
  
  Specifies the name of the secondary network to associate with the pod. To specify more than one secondary network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same secondary network multiple times, that pod will have multiple network interfaces attached to that network.
2. To attach a secondary network with customizations, add an annotation with the following format:
  metadata: annotations: k8s.v1.cni.cncf.io/networks: |- [ { "name": "<network>", "namespace": "<namespace>", "default-route": ["<default_route>"] } ]
  where:
  
  name
  
  Specifies the name of the secondary network defined by a NetworkAttachmentDefinition object.
  
  namespace
  
  Specifies the namespace where the NetworkAttachmentDefinition object is defined.
  
  default-route
  
  Optional parameter. Specifies an override for the default route, such as 192.168.17.1.
Create the pod by entering the following command.
```
$ oc create -f <name>.yaml
```
Replace <name> with the name of the pod.

Optional: Confirm that the annotation exists in the pod CR by entering the following command. Replace <name> with the name of the pod.

$ oc get pod <name> -o yaml

In the following example, the example-pod pod is attached to the net1 secondary network:

$ oc get pod example-pod -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-bridge
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "ovn-kubernetes",
          "interface": "eth0",
          "ips": [
              "10.128.2.14"
          ],
          "default": true,
          "dns": {}
      },{
          "name": "macvlan-bridge",
          "interface": "net1",
          "ips": [
              "20.2.2.100"
          ],
          "mac": "22:2f:60:a5:f8:00",
          "dns": {}
      }]
  name: example-pod
  namespace: default
spec:
  ...
status:
  ...

where:

k8s.v1.cni.cncf.io/network-status: Specifies a JSON array of objects. Each object describes the status of a secondary network attached to the pod. The annotation value is stored as a plain text value.

Specifying pod-specific addressing and routing options

You can configure pod-specific addressing and routing options by using JSON-formatted annotations. Configuring these options enables you to set static IP addresses, MAC addresses, and default routes for the pod.

Prerequisites

The pod must be in the same namespace as the secondary network.
Install the OpenShift CLI (oc).
You must log in to the cluster.

Procedure

Edit the Pod resource definition. If you are editing an existing Pod resource, run the following command to edit its definition in the default editor. Replace <name> with the name of the Pod resource to edit.
```
$ oc edit pod <name>
```

In the Pod resource definition, add the k8s.v1.cni.cncf.io/networks parameter to the pod metadata mapping. The k8s.v1.cni.cncf.io/networks accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition custom resource (CR) names in addition to specifying additional properties.

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]'
# ...

where:

<network>: Replace with a JSON object as shown in the following examples. The single quotes are required.

In the following example the annotation specifies which network attachment will have the default route, using the default-route parameter.

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
    {
      "name": "net1"
    },
    {
      "name": "net2",
      "default-route": ["192.0.2.1"]
    }]'
spec:
  containers:
  - name: example-pod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools

where:

name: Specifies the name of the secondary network to associate with the pod.
default-route: Specifies a value of a gateway for traffic to be routed over if no other routing entry is present in the routing table. If more than one default-route key is specified, this will cause the pod to fail to become active.

The default route will cause any traffic that is not specified in other routes to be routed to the gateway.

Setting the default route to an interface other than the default network interface for OKD may cause traffic that is anticipated for pod-to-pod traffic to be routed over another interface.

To verify the routing properties of a pod, the oc command may be used to execute the ip command within a pod.

$ oc exec -it <pod_name> -- ip route

You may also reference the pod’s k8s.v1.cni.cncf.io/network-status to see which secondary network has been assigned the default route, by the presence of the default-route key in the JSON-formatted list of objects.

To set a static IP address or MAC address for a pod you can use the JSON formatted annotations. This requires you create networks that specifically allow for this functionality. This can be specified in a rawCNIConfig for the CNO.

Edit the CNO CR by running the following command:
```
$ oc edit networks.operator.openshift.io cluster
```
The following YAML describes the configuration parameters for the CNO:
Cluster Network Operator YAML configuration
```
name: <name>
namespace: <namespace>
rawCNIConfig: '{
  ...
}'
type: Raw
```
where:

name

Specifies a name for the secondary network attachment that you are creating. The name must be unique within the specified namespace.

namespace

Specifies the namespace to create the network attachment in. If you do not specify a value, then the default namespace is used.

rawCNIConfig

Specifies the CNI plugin configuration in JSON format, which is based on the following template.

The following object describes the configuration parameters for utilizing static MAC address and IP address using the macvlan CNI plugin:
macvlan CNI plugin JSON configuration object using static IP and MAC address
```
{
  "cniVersion": "0.3.1",
  "name": "<name>",
  "plugins": [{
      "type": "macvlan",
      "capabilities": { "ips": true },
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "static"
      }
    }, {
      "capabilities": { "mac": true },
      "type": "tuning"
    }]
}
```
where:

name

Specifies the name for the secondary network attachment to create. The name must be unique within the specified namespace.

plugins

Specifies an array of CNI plugin configurations. The first object specifies a macvlan plugin configuration and the second object specifies a tuning plugin configuration.

ips

Specifies that a request is made to enable the static IP address functionality of the CNI plugin runtime configuration capabilities.

master

Specifies the interface that the macvlan plugin uses.

mac

Specifies that a request is made to enable the static MAC address functionality of a CNI plugin.

The above network attachment can be referenced in a JSON formatted annotation, along with keys to specify which static IP and MAC address will be assigned to a given pod.
Edit the pod by entering the following command:
```
$ oc edit pod <name>
```
macvlan CNI plugin JSON configuration object using static IP and MAC address
```
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "<name>",
        "ips": [ "192.0.2.205/24" ],
        "mac": "CA:FE:C0:FF:EE:00"
      }
    ]'
```
where:

name

Specifies the name for the secondary network attachment to create. The name must be unique within the specified namespace.

ips

Specifies an IP address including the subnet mask.

mac

Specifies the MAC address.

Static IP addresses and MAC addresses do not have to be used at the same time. You can use them individually, or together.