Configuring IP address assignment for secondary networks - Multiple networks | Networking

Configuration of IP address assignment for a network attachment

The following sections give instructions and information for how to configure IP address assignments for secondary networks.

Configuration of IP address assignment for a network attachment

For secondary networks, IP addresses can be assigned using an IP Address Management (IPAM) CNI plugin, which supports various assignment methods, including Dynamic Host Configuration Protocol (DHCP) and static assignment.

The DHCP IPAM CNI plugin responsible for dynamic assignment of IP addresses operates with two distinct components:

CNI Plugin: Responsible for integrating with the Kubernetes networking stack to request and release IP addresses.
DHCP IPAM CNI Daemon: A listener for DHCP events that coordinates with existing DHCP servers in the environment to handle IP address assignment requests. This daemon is not a DHCP server itself.

For networks requiring type: dhcp in their IPAM configuration, ensure the following:

A DHCP server is available and running in the environment. The DHCP server is external to the cluster and is expected to be part of the customer’s existing network infrastructure.
The DHCP server is appropriately configured to serve IP addresses to the nodes.

In cases where a DHCP server is unavailable in the environment, it is recommended to use the Whereabouts IPAM CNI plugin instead. The Whereabouts CNI provides similar IP address management capabilities without the need for an external DHCP server.

Use the Whereabouts CNI plugin when there is no external DHCP server or where static IP address management is preferred. The Whereabouts plugin includes a reconciler daemon to manage stale IP address allocations.

A DHCP lease must be periodically renewed throughout the container’s lifetime, so a separate daemon, the DHCP IPAM CNI Daemon, is required. To deploy the DHCP IPAM CNI daemon, modify the Cluster Network Operator (CNO) configuration to trigger the deployment of this daemon as part of the secondary network setup.

Static IP address assignment configuration

The following table describes the configuration for static IP address assignment:

Table 1. `ipam` static configuration object
Field	Type	Description
`type`	`string`	The IPAM address type. The value `static` is required.
`addresses`	`array`	An array of objects specifying IP addresses to assign to the virtual interface. Both IPv4 and IPv6 IP addresses are supported.
`routes`	`array`	An array of objects specifying routes to configure inside the pod.
`dns`	`array`	Optional: An array of objects specifying the DNS configuration.

The addresses array requires objects with the following fields:

Table 2. `ipam.addresses[]` array
Field	Type	Description
`address`	`string`	An IP address and network prefix that you specify. For example, if you specify `10.10.21.10/24`, then the secondary network is assigned an IP address of `10.10.21.10` and the netmask is `255.255.255.0`.
`gateway`	`string`	The default gateway to route egress network traffic to.

Table 3. `ipam.routes[]` array
Field	Type	Description
`dst`	`string`	The IP address range in CIDR format, such as `192.168.17.0/24` or `0.0.0.0/0` for the default route.
`gw`	`string`	The gateway where network traffic is routed.

Table 4. `ipam.dns` object
Field	Type	Description
`nameservers`	`array`	An array of one or more IP addresses for to send DNS queries to.
`domain`	`array`	The default domain to append to a hostname. For example, if the domain is set to `example.com`, a DNS lookup query for `example-host` is rewritten as `example-host.example.com`.
`search`	`array`	An array of domain names to append to an unqualified hostname, such as `example-host`, during a DNS lookup query.

Static IP address assignment configuration example

{
  "ipam": {
    "type": "static",
      "addresses": [
        {
          "address": "191.168.1.7/24"
        }
      ]
  }
}

Dynamic IP address (DHCP) assignment configuration

A pod obtains its original DHCP lease when it is created. The lease must be periodically renewed by a minimal DHCP server deployment running on the cluster.

For an Ethernet network attachment, the SR-IOV Network Operator does not create a DHCP server deployment; the Cluster Network Operator is responsible for creating the minimal DHCP server deployment.

To trigger the deployment of the DHCP server, you must create a shim network attachment by editing the Cluster Network Operator configuration, as in the following example:

Example shim network attachment definition

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: dhcp-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
        "name": "dhcp-shim",
        "cniVersion": "0.3.1",
        "type": "bridge",
        "ipam": {
          "type": "dhcp"
        }
      }
  # ...

The following table describes the configuration parameters for dynamic IP address address assignment with DHCP.

Table 5. `ipam` DHCP configuration object
Field	Type	Description
`type`	`string`	The IPAM address type. The value `dhcp` is required.

The following JSON example describes the configuration p for dynamic IP address address assignment with DHCP.

Dynamic IP address (DHCP) assignment configuration example

{
  "ipam": {
    "type": "dhcp"
  }
}

Dynamic IP address assignment configuration with Whereabouts

The Whereabouts CNI plugin helps the dynamic assignment of an IP address to a secondary network without the use of a DHCP server.

The Whereabouts CNI plugin also supports overlapping IP address ranges and configuration of the same CIDR range multiple times within separate NetworkAttachmentDefinition CRDs. This provides greater flexibility and management capabilities in multi-tenant environments.

Dynamic IP address configuration parameters

The following table describes the configuration objects for dynamic IP address assignment with Whereabouts:

Table 6. `ipam` whereabouts configuration parameters
Field	Type	Description
`type`	`string`	The IPAM address type. The value `whereabouts` is required.
`range`	`string`	An IP address and range in CIDR notation. IP addresses are assigned from within this range of addresses.
`exclude`	`array`	Optional: A list of zero or more IP addresses and ranges in CIDR notation. IP addresses within an excluded address range are not assigned.
`network_name`	`string`	Optional: Helps ensure that each group or domain of pods gets its own set of IP addresses, even if they share the same range of IP addresses. Setting this field is important for keeping networks separate and organized, notably in multi-tenant environments.

Dynamic IP address assignment configuration with Whereabouts that excludes IP address ranges

The following example shows a dynamic address assignment configuration in a NAD file that uses Whereabouts:

Whereabouts dynamic IP address assignment that excludes specific IP address ranges

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/27",
    "exclude": [
       "192.0.2.192/30",
       "192.0.2.196/32"
    ]
  }
}

Dynamic IP address assignment that uses Whereabouts with overlapping IP address ranges

The following example shows a dynamic IP address assignment that uses overlapping IP address ranges for multi-tenant networks.

NetworkAttachmentDefinition 1

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/29",
    "network_name": "example_net_common", (1)
  }
}

1	Optional. If set, must match the `network_name` of `NetworkAttachmentDefinition 2`.

NetworkAttachmentDefinition 2

{
  "ipam": {
    "type": "whereabouts",
    "range": "192.0.2.192/24",
    "network_name": "example_net_common", (1)
  }
}

1	Optional. If set, must match the `network_name` of `NetworkAttachmentDefinition 1`.

Creating a whereabouts-reconciler daemon set

The Whereabouts reconciler is responsible for managing dynamic IP address assignments for the pods within a cluster by using the Whereabouts IP Address Management (IPAM) solution. It ensures that each pod gets a unique IP address from the specified IP address range. It also handles IP address releases when pods are deleted or scaled down.

You can also use a NetworkAttachmentDefinition custom resource definition (CRD) for dynamic IP address assignment.

The whereabouts-reconciler daemon set is automatically created when you configure a secondary network through the Cluster Network Operator. It is not automatically created when you configure a secondary network from a YAML manifest.

To trigger the deployment of the whereabouts-reconciler daemon set, you must manually create a whereabouts-shim network attachment by editing the Cluster Network Operator custom resource (CR) file.

Use the following procedure to deploy the whereabouts-reconciler daemon set.

Procedure

Edit the Network.operator.openshift.io custom resource (CR) by running the following command:
```
$ oc edit network.operator.openshift.io cluster
```

Include the additionalNetworks section shown in this example YAML extract within the spec definition of the custom resource (CR):

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
# ...
spec:
  additionalNetworks:
  - name: whereabouts-shim
    namespace: default
    rawCNIConfig: |-
      {
       "name": "whereabouts-shim",
       "cniVersion": "0.3.1",
       "type": "bridge",
       "ipam": {
         "type": "whereabouts"
       }
      }
    type: Raw
# ...

Save the file and exit the text editor.

Verify that the whereabouts-reconciler daemon set deployed successfully by running the following command:

$ oc get all -n openshift-multus | grep whereabouts-reconciler

Example output

pod/whereabouts-reconciler-jnp6g 1/1 Running 0 6s
pod/whereabouts-reconciler-k76gg 1/1 Running 0 6s
pod/whereabouts-reconciler-k86t9 1/1 Running 0 6s
pod/whereabouts-reconciler-p4sxw 1/1 Running 0 6s
pod/whereabouts-reconciler-rvfdv 1/1 Running 0 6s
pod/whereabouts-reconciler-svzw9 1/1 Running 0 6s
daemonset.apps/whereabouts-reconciler 6 6 6 6 6 kubernetes.io/os=linux 6s

Configuring the Whereabouts IP reconciler schedule

The Whereabouts IPAM CNI plugin runs the IP reconciler daily. This process cleans up any stranded IP allocations that might result in exhausting IPs and therefore prevent new pods from getting an IP allocated to them.

Use this procedure to change the frequency at which the IP reconciler runs.

Prerequisites

You installed the OpenShift CLI (oc).
You have access to the cluster as a user with the cluster-admin role.
You have deployed the whereabouts-reconciler daemon set, and the whereabouts-reconciler pods are up and running.

Procedure

Run the following command to create a ConfigMap object named whereabouts-config in the openshift-multus namespace with a specific cron expression for the IP reconciler:
```
$ oc create configmap whereabouts-config -n openshift-multus --from-literal=reconciler_cron_expression="*/15 * * * *"
```
This cron expression indicates the IP reconciler runs every 15 minutes. Adjust the expression based on your specific requirements.

The whereabouts-reconciler daemon set can only consume a cron expression pattern that includes five asterisks. The sixth, which is used to denote seconds, is currently not supported.

Retrieve information about resources related to the whereabouts-reconciler daemon set and pods within the openshift-multus namespace by running the following command:

$ oc get all -n openshift-multus | grep whereabouts-reconciler

Example output

pod/whereabouts-reconciler-2p7hw                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-76jk7                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-94zw6                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-mfh68                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-pgshz                   1/1     Running   0             4m14s
pod/whereabouts-reconciler-xn5xz                   1/1     Running   0             4m14s
daemonset.apps/whereabouts-reconciler          6         6         6       6            6           kubernetes.io/os=linux   4m16s

Run the following command to verify that the whereabouts-reconciler pod runs the IP reconciler with the configured interval:

$ oc -n openshift-multus logs whereabouts-reconciler-2p7hw

Example output

2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CREATE
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_33_54.1375928161": CHMOD
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..data_tmp": RENAME
2024-02-02T16:33:54Z [verbose] using expression: */15 * * * *
2024-02-02T16:33:54Z [verbose] configuration updated to file "/cron-schedule/..data". New cron expression: */15 * * * *
2024-02-02T16:33:54Z [verbose] successfully updated CRON configuration id "00c2d1c9-631d-403f-bb86-73ad104a6817" - new cron expression: */15 * * * *
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/config": CREATE
2024-02-02T16:33:54Z [debug] event not relevant: "/cron-schedule/..2024_02_02_16_26_17.3874177937": REMOVE
2024-02-02T16:45:00Z [verbose] starting reconciler run
2024-02-02T16:45:00Z [debug] NewReconcileLooper - inferred connection data
2024-02-02T16:45:00Z [debug] listing IP pools
2024-02-02T16:45:00Z [debug] no IP addresses to cleanup
2024-02-02T16:45:00Z [verbose] reconciler success

Fast IPAM configuration for the Whereabouts IPAM CNI plugin

Wherabouts is an IP Address Management (IPAM) Container Network Interface (CNI) plugin that assigns IP addresses at a cluster-wide level. Whereabouts does not require a Dynamic Host Configuration Protocol (DHCP) server.

A typical Wherabouts workflow is described as follows:

Whereabouts takes an address range in classless inter-domain routing (CIDR) notation, such as 192.168.2.0/24, and assigns IP addresses within that range, such as 192.168.2.1 to 192.168.2.254.
Whereabouts assigns an IP address, the lowest value address in a CIDR range, to a pod and tracks the IP address in a data store for the lifetime of that pod.
When the pod is removed, Whereabouts frees the address from the pod so that the address is available for assignment.

To improve the performance of Whereabouts, especially if nodes in your cluster run a high amount of pods, you can enable the Fast IPAM feature.

Fast IPAM is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The Fast IPAM feature uses nodeslicepools, which are managed by the Whereabouts Controller, to optimize IP allocation for nodes.

Prerequisites

You added the whereabouts-shim configuration to the Network.operator.openshift.io custom resource (CR), so that the Cluster Network Operator (CNO) can deploy the Whereabouts Controller. See "Creating a Whereabouts reconciler daemon set".
For the Fast IPAM feature to work, ensure that the NetworkAttachmentDefinition (NAD) and the pod exist in the same openshift-multus namesapace.

Procedure

Confirm that the Whereabouts Controller is running by entering the following command.
```
$ oc get pods -n openshift-multus | grep controller
```
Example output
```
multus-admission-controller-d89bc96f-gbf7s   2/2     Running   0              6h3m
...
```
If the Whereabouts Controller is not running, the Fast IPAM does not work.

Create a NAD file for your cluster and add the Fast IPAM details to the file:

Example NAD file with a Fast IPAM configuration

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: wb-ipam
  namespace: openshift-multus (1)
spec:
  config: {
	"cniVersion": "0.3.0",
	"name": "wb-ipam-cni-name", (2)
	"type": "bridge",
	"bridge": "cni0",
	"ipam": {
  	"type": "whereabouts", (3)
  	"range": "10.5.0.0/20", (4)
  	"node_slice_size": "/24" (5)
    }
  }
# ...

1	The namespace where CNO deploys the NAD.
2	The name of the Whereabouts IPAM CNI plugin.
3	The type of IPAM CNI plugin: `whereabouts`.
4	The IP address range for the IP pool that the Whereabouts IPAM CNI plugin uses for allocating IP addresses to pods.
5	Sets the slice size of IP addresses available to each node.

Add the Whereabouts IPAM CNI plugin annotation details to the YAML file for the pod:

apiVersion: v1
kind: Pod
metadata:
  name: <pod_name> (1)
  annotations:
  k8s.v1.cni.cncf.io/networks: openshift-multus/wb-ipam (2)
spec:
  containers:
  - name: samplepod (3)
  command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"] (4)
  image: alpine
# ...

1	The name of the pod.
2	The annotation details that references the Whereabouts IPAM CNI plugin name that exists in the `openshift-multus` namespace.
3	The name of the container for the pod.
4	Defines the entry point for the container and controls the behavior of the container in the Whereabouts IPAM CNI plugin.

Apply the NAD file configuration to pods that exist on nodes that run in your cluster:
```
$ oc create -f <NAD_file_name>.yaml
```

Verification

Show the IP address details of the pod by entering the following command:

$ oc describe pod <pod_name>

Example output

...
IP:     192.168.2.0
IPs:
  IP:   192.168.2.0
Containers:
  samplepod:
    Container ID:   docker://<image_name>
    Image:          <app_name>:v1
    Image ID:
...

Access the pod and confirm its interfaces by entering the following command:

$ oc exec <pod_name> -- ip a

Example output

...
3: net1@if23: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 82:01:98:e5:0c:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.0/24 brd 10.10.0.255 scope global net1 (1)
       valid_lft forever preferred_lft forever
    inet6 fe80::8001:98ff:fee5:cb7/64 scope link
       valid_lft forever preferred_lft forever
...

1	Pod is attached to the `192.168.2.1` IP address on the `net1` interface as expected.

Check that the node selector pool exists in the openshift-multus namespace by entering the following command:
```
$ oc get nodeslicepool -n openshift-multus
```
Example output
```
NAME            AGE
nodeslicepool   32m
```

Creating a configuration for assignment of dual-stack IP addresses dynamically

Dual-stack IP address assignment can be configured with the ipRanges parameter for:

IPv4 addresses
IPv6 addresses
multiple IP address assignment

Procedure

Set type to whereabouts.

Use ipRanges to allocate IP addresses as shown in the following example:

cniVersion: operator.openshift.io/v1
kind: Network
=metadata:
  name: cluster
spec:
  additionalNetworks:
  - name: whereabouts-shim
    namespace: default
    type: Raw
    rawCNIConfig: |-
      {
       "name": "whereabouts-dual-stack",
       "cniVersion": "0.3.1,
       "type": "bridge",
       "ipam": {
         "type": "whereabouts",
         "ipRanges": [
                  {"range": "192.168.10.0/24"},
                  {"range": "2001:db8::/64"}
              ]
       }
      }

Attach network to a pod. For more information, see "Adding a pod to a secondary network".
Verify that all IP addresses are assigned.
Run the following command to ensure the IP addresses are assigned as metadata.
```
$ oc exec -it mypod -- ip a
```

Configuring IP address assignment on secondary networks

Configuration of IP address assignment for a network attachment

Static IP address assignment configuration

Dynamic IP address (DHCP) assignment configuration

Dynamic IP address assignment configuration with Whereabouts

Dynamic IP address configuration parameters

Dynamic IP address assignment configuration with Whereabouts that excludes IP address ranges

Dynamic IP address assignment that uses Whereabouts with overlapping IP address ranges

Creating a whereabouts-reconciler daemon set

Configuring the Whereabouts IP reconciler schedule

Fast IPAM configuration for the Whereabouts IPAM CNI plugin

Creating a configuration for assignment of dual-stack IP addresses dynamically