Migrating from the OpenShift SDN network plugin

Offline migration to the OVN-Kubernetes network plugin overview
Limited live migration to the OVN-Kubernetes network plugin overview
Additional resources

As a cluster administrator, you can migrate to the OVN-Kubernetes network plugin from the OpenShift SDN network plugin using the offline migration method or the limited live migration method.

It is not possible to upgrade a cluster to OKD 4.17 if it is using the OpenShift SDN network plugin. You must migrate to the OVN-Kubernetes plugin before upgrading to OKD 4.17.

To learn more about OVN-Kubernetes, read About the OVN-Kubernetes network plugin.

Do not automate the migration from OpenShift SDN to OVN-Kubernetes with a script or another tool like Red Hat Ansible Automation Platform. This might cause outages or crash your OKD cluster.

Offline migration to the OVN-Kubernetes network plugin overview

The offline migration method is a manual process that includes some downtime, during which your cluster is unreachable. This method is primarily used for self-managed OKD deployments.

Before you migrate your OKD cluster to use the OVN-Kubernetes network plugin, update your cluster to the latest z-stream release so that all the latest bug fixes apply to your cluster.

Although a rollback procedure is provided, the offline migration is intended to be a one-way process.

OpenShift SDN CNI is deprecated as of OKD 4.14. As of OKD 4.15, the network plugin is not an option for new installations. In a subsequent future release, the OpenShift SDN network plugin is planned to be removed and no longer supported. Red Hat will provide bug fixes and support for this feature until it is removed, but this feature will no longer receive enhancements. As an alternative to OpenShift SDN CNI, you can use OVN Kubernetes CNI instead. For more information, see OpenShift SDN CNI removal.

The following sections provide more information about the offline migration method.

Supported platforms when using the offline migration method

The following table provides information about the supported platforms for the offline migration type.

Table 1. Supported platforms for the offline migration method
Platform	Offline Migration
Bare-metal hardware	✓
Amazon Web Services (AWS)	✓
Google Cloud Platform (GCP)	✓
IBM Cloud®	✓
Microsoft Azure	✓
OpenStack	✓
VMware vSphere	✓
Nutanix	✓

Each listed platform supports installing an OKD cluster on installer-provisioned infrastructure and user-provisioned infrastructure.

Best practices for offline migration to the OVN-Kubernetes network plugin

For a list of best practices when migrating to the OVN-Kubernetes network plugin with the offline migration method, see Best practices for OpenShift SDN to OVN Kubernetes CNI Offline migration.

Considerations for offline migration to the OVN-Kubernetes network plugin

If you have more than 150 nodes in your OKD cluster, then open a support case for consultation on your migration to the OVN-Kubernetes network plugin.

The subnets assigned to nodes and the IP addresses assigned to individual pods are not preserved during the migration.

While the OVN-Kubernetes network plugin implements many of the capabilities present in the OpenShift SDN network plugin, the configuration is not the same.

If your cluster uses any of the following OpenShift SDN network plugin capabilities, you must manually configure the same capability in the OVN-Kubernetes network plugin:
- Namespace isolation
- Egress router pods
Before migrating to OVN-Kubernetes, ensure that the following IP address ranges are not in use: 100.64.0.0/16, 169.254.169.0/29, 100.88.0.0/16, fd98::/64, fd69::/125, and fd97::/64. OVN-Kubernetes uses these ranges internally. Do not include any of these ranges in any other CIDR definitions in your cluster or infrastructure.

The following sections highlight the differences in configuration between the aforementioned capabilities in OVN-Kubernetes and OpenShift SDN network plugins.

Primary network interface

The OpenShift SDN plugin allows application of the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) to the primary interface on a node. The OVN-Kubernetes network plugin does not have this capability.

If you have an NNCP applied to the primary interface, you must delete the NNCP before migrating to the OVN-Kubernetes network plugin. Deleting the NNCP does not remove the configuration from the primary interface, but with OVN-Kubernetes, the Kubernetes NMState cannot manage this configuration. Instead, the configure-ovs.sh shell script manages the primary interface and the configuration attached to this interface.

Namespace isolation

OVN-Kubernetes supports only the network policy isolation mode.

For a cluster using OpenShift SDN that is configured in either the multitenant or subnet isolation mode, you can still migrate to the OVN-Kubernetes network plugin. Note that after the migration operation, multitenant isolation mode is dropped, so you must manually configure network policies to achieve the same level of project-level isolation for pods and services.

Egress IP addresses

OpenShift SDN supports two different Egress IP modes:

In the automatically assigned approach, an egress IP address range is assigned to a node.
In the manually assigned approach, a list of one or more egress IP addresses is assigned to a node.

The migration process supports migrating Egress IP configurations that use the automatically assigned mode.

The differences in configuring an egress IP address between OVN-Kubernetes and OpenShift SDN is described in the following table:

Table 2. Differences in egress IP address configuration
OVN-Kubernetes	OpenShift SDN
Create an `EgressIPs` object Add an annotation on a `Node` object	Patch a `NetNamespace` object Patch a `HostSubnet` object

For more information on using egress IP addresses in OVN-Kubernetes, see "Configuring an egress IP address".

Egress network policies

The difference in configuring an egress network policy, also known as an egress firewall, between OVN-Kubernetes and OpenShift SDN is described in the following table:

Table 3. Differences in egress network policy configuration
OVN-Kubernetes	OpenShift SDN
Create an `EgressFirewall` object in a namespace	Create an `EgressNetworkPolicy` object in a namespace

Because the name of an EgressFirewall object can only be set to default, after the migration all migrated EgressNetworkPolicy objects are named default, regardless of what the name was under OpenShift SDN.

If you subsequently rollback to OpenShift SDN, all EgressNetworkPolicy objects are named default as the prior name is lost.

For more information on using an egress firewall in OVN-Kubernetes, see "Configuring an egress firewall for a project".

Egress router pods

OVN-Kubernetes supports egress router pods in redirect mode. OVN-Kubernetes does not support egress router pods in HTTP proxy mode or DNS proxy mode.

When you deploy an egress router with the Cluster Network Operator, you cannot specify a node selector to control which node is used to host the egress router pod.

Multicast

The difference between enabling multicast traffic on OVN-Kubernetes and OpenShift SDN is described in the following table:

Table 4. Differences in multicast configuration
OVN-Kubernetes	OpenShift SDN
Add an annotation on a `Namespace` object	Add an annotation on a `NetNamespace` object

For more information on using multicast in OVN-Kubernetes, see "Enabling multicast for a project".

Network policies

OVN-Kubernetes fully supports the Kubernetes NetworkPolicy API in the networking.k8s.io/v1 API group. No changes are necessary in your network policies when migrating from OpenShift SDN.

Additional resources

How the offline migration process works

The following table summarizes the migration process by segmenting between the user-initiated steps in the process and the actions that the migration performs in response.

Table 5. Offline migration to OVN-Kubernetes from OpenShift SDN
User-initiated steps	Migration activity
Set the `migration` field of the `Network.operator.openshift.io` custom resource (CR) named `cluster` to `OVNKubernetes`. Make sure the `migration` field is `null` before setting it to a value.	Cluster Network Operator (CNO) Updates the status of the `Network.config.openshift.io` CR named `cluster` accordingly. Machine Config Operator (MCO) Rolls out an update to the systemd configuration necessary for OVN-Kubernetes; the MCO updates a single machine per pool at a time by default, causing the total time the migration takes to increase with the size of the cluster.
Update the `networkType` field of the `Network.config.openshift.io` CR.	CNO Performs the following actions: Destroys the OpenShift SDN control plane pods. Deploys the OVN-Kubernetes control plane pods. Updates the Multus daemon sets and config map objects to reflect the new network plugin.
Reboot each node in the cluster.	Cluster As nodes reboot, the cluster assigns IP addresses to pods on the OVN-Kubernetes cluster network.

Migrating to the OVN-Kubernetes network plugin by using the offline migration method

As a cluster administrator, you can change the network plugin for your cluster to OVN-Kubernetes. During the migration, you must reboot every node in your cluster.

While performing the migration, your cluster is unavailable and workloads might be interrupted. Perform the migration only when an interruption in service is acceptable.

Prerequisites

You have a cluster configured with the OpenShift SDN CNI network plugin in the network policy isolation mode.
You installed the OpenShift CLI (oc).
You have access to the cluster as a user with the cluster-admin role.
You have a recent backup of the etcd database.
You can manually reboot each node.
You checked that your cluster is in a known good state without any errors.
You created a security group rule that allows User Datagram Protocol (UDP) packets on port 6081 for all nodes on all cloud platforms.
You set all timeouts for webhooks to 3 seconds or removed the webhooks.

Procedure

To backup the configuration for the cluster network, enter the following command:

$ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml

Verify that the OVN_SDN_MIGRATION_TIMEOUT environment variable is set and is equal to 0s by running the following command:

#!/bin/bash

if [ -n "$OVN_SDN_MIGRATION_TIMEOUT" ] && [ "$OVN_SDN_MIGRATION_TIMEOUT" = "0s" ]; then
    unset OVN_SDN_MIGRATION_TIMEOUT
fi

#loops the timeout command of the script to repeatedly check the cluster Operators until all are available.

co_timeout=${OVN_SDN_MIGRATION_TIMEOUT:-1200s}
timeout "$co_timeout" bash <<EOT
until
  oc wait co --all --for='condition=AVAILABLE=True' --timeout=10s && \
  oc wait co --all --for='condition=PROGRESSING=False' --timeout=10s && \
  oc wait co --all --for='condition=DEGRADED=False' --timeout=10s;
do
  sleep 10
  echo "Some ClusterOperators Degraded=False,Progressing=True,or Available=False";
done
EOT

Remove the configuration from the Cluster Network Operator (CNO) configuration object by running the following command:
```
$ oc patch Network.operator.openshift.io cluster --type='merge' \
--patch '{"spec":{"migration":null}}'
```
Delete the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) that defines the primary network interface for the OpenShift SDN network plugin by completing the following steps:
1. Check that the existing NNCP CR bonded the primary interface to your cluster by entering the following command:
  $ oc get nncp
  Example output
  NAME STATUS REASON bondmaster0 Available SuccessfullyConfigured
  Network Manager stores the connection profile for the bonded primary interface in the /etc/NetworkManager/system-connections system path.
2. Remove the NNCP from your cluster:
  $ oc delete nncp <nncp_manifest_filename>

To prepare all the nodes for the migration, set the migration field on the CNO configuration object by running the following command:

$ oc patch Network.operator.openshift.io cluster --type='merge' \
  --patch '{ "spec": { "migration": { "networkType": "OVNKubernetes" } } }'

This step does not deploy OVN-Kubernetes immediately. Instead, specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for the OVN-Kubernetes deployment.

Check that the reboot is finished by running the following command:
```
$ oc get mcp
```
Check that all cluster Operators are available by running the following command:
```
$ oc get co
```
Alternatively: You can disable automatic migration of several OpenShift SDN capabilities to the OVN-Kubernetes equivalents:
- Egress IPs
- Egress firewall
- Multicast
To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:
```
$ oc patch Network.operator.openshift.io cluster --type='merge' \
  --patch '{
    "spec": {
      "migration": {
        "networkType": "OVNKubernetes",
        "features": {
          "egressIP": <bool>,
          "egressFirewall": <bool>,
          "multicast": <bool>
        }
      }
    }
  }'
```
where:

bool: Specifies whether to enable migration of the feature. The default is true.

Optional: You can customize the following settings for OVN-Kubernetes to meet your network infrastructure requirements:
- Maximum transmission unit (MTU). Consider the following before customizing the MTU for this optional step:
  
  If you use the default MTU, and you want to keep the default MTU during migration, this step can be ignored.
  
  If you used a custom MTU, and you want to keep the custom MTU during migration, you must declare the custom MTU value in this step.
  
  This step does not work if you want to change the MTU value during migration. Instead, you must first follow the instructions for "Changing the cluster MTU". You can then keep the custom MTU value by performing this procedure and declaring the custom MTU value in this step.
  
  OpenShift-SDN and OVN-Kubernetes have different overlay overhead. MTU values should be selected by following the guidelines found on the "MTU value selection" page.
- Geneve (Generic Network Virtualization Encapsulation) overlay network port
- OVN-Kubernetes IPv4 internal subnet
- OVN-Kubernetes IPv6 internal subnet
To customize either of the previously noted settings, enter and customize the following command. If you do not need to change the default value, omit the key from the patch.
```
$ oc patch Network.operator.openshift.io cluster --type=merge \
  --patch '{
    "spec":{
      "defaultNetwork":{
        "ovnKubernetesConfig":{
          "mtu":<mtu>,
          "genevePort":<port>,
          "v4InternalSubnet":"<ipv4_subnet>",
          "v6InternalSubnet":"<ipv6_subnet>"
    }}}}'
```
where:

mtu

The MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

port

The UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by OpenShift SDN. The default value for the VXLAN port is 4789.

ipv4_subnet

An IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

ipv6_subnet

An IPv6 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OKD installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is fd98::/48.
Example patch command to update mtu field
```
$ oc patch Network.operator.openshift.io cluster --type=merge \
  --patch '{
    "spec":{
      "defaultNetwork":{
        "ovnKubernetesConfig":{
          "mtu":1200
    }}}}'
```
As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:
```
$ oc get mcp
```
A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

Confirm the status of the new machine configuration on the hosts:

To list the machine configuration state and the name of the applied machine configuration, enter the following command:
```
$ oc describe node | egrep "hostname|machineconfig"
```
Example output
```
kubernetes.io/hostname=master-0
machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
machineconfiguration.openshift.io/reason:
machineconfiguration.openshift.io/state: Done
```
Verify that the following statements are true:
- The value of machineconfiguration.openshift.io/state field is Done.
- The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.
To confirm that the machine config is correct, enter the following command:
```
$ oc get machineconfig <config_name> -o yaml | grep ExecStart
```
where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

The machine config can include the following update to the systemd configuration:
```
ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes
```

If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

To list the pods, enter the following command:

$ oc get pod -n openshift-machine-config-operator

Example output

NAME                                         READY   STATUS    RESTARTS   AGE
machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
machine-config-daemon-5cf4b                  2/2     Running   0          43h
machine-config-daemon-7wzcd                  2/2     Running   0          43h
machine-config-daemon-fc946                  2/2     Running   0          43h
machine-config-daemon-g2v28                  2/2     Running   0          43h
machine-config-daemon-gcl4f                  2/2     Running   0          43h
machine-config-daemon-l5tnv                  2/2     Running   0          43h
machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
machine-config-server-bsc8h                  1/1     Running   0          43h
machine-config-server-hklrm                  1/1     Running   0          43h
machine-config-server-k9rtx                  1/1     Running   0          43h

The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:
```
$ oc logs <pod> -n openshift-machine-config-operator
```
where pod is the name of a machine config daemon pod.
Resolve any errors in the logs shown by the output from the previous command.

To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:
- To specify the network provider without changing the cluster network IP address block, enter the following command:
  $ oc patch Network.config.openshift.io cluster \ --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'
- To specify a different cluster network IP address block, enter the following command:
  $ oc patch Network.config.openshift.io cluster \ --type='merge' --patch '{ "spec": { "clusterNetwork": [ { "cidr": "<cidr>", "hostPrefix": <prefix> } ], "networkType": "OVNKubernetes" } }'
  where cidr is a CIDR block and prefix is the slice of the CIDR block apportioned to each node in your cluster. You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.
  
  You cannot change the service network address block during the migration.

Verify that the Multus daemon set rollout is complete before continuing with subsequent steps:

$ oc -n openshift-multus rollout status daemonset/multus

The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

Example output

Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
...
Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
daemon set "multus" successfully rolled out

To complete changing the network plugin, reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

The following scripts reboot all of the nodes in the cluster at the same time. This can cause your cluster to be unstable. Another option is to reboot your nodes manually one at a time. Rebooting nodes one-by-one causes considerable downtime in a cluster with many nodes.

Cluster Operators will not work correctly before you reboot the nodes.

With the oc rsh command, you can use a bash script similar to the following:

#!/bin/bash
readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"

for i in "${POD_NODES[@]}"
do
  read -r POD NODE <<< "$i"
  until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
    do
      echo "cannot reboot node $NODE, retry" && sleep 3
    done
done

With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

#!/bin/bash

for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
do
   echo "reboot node $ip"
   ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
done

Confirm that the migration succeeded:
1. To confirm that the network plugin is OVN-Kubernetes, enter the following command. The value of status.networkType must be OVNKubernetes.
  $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'
2. To confirm that the cluster nodes are in the Ready state, enter the following command:
  $ oc get nodes
3. To confirm that your pods are not in an error state, enter the following command:
  $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'
  If pods on a node are in an error state, reboot that node.
4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:
  $ oc get co
  The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.
Complete the following steps only if the migration succeeds and your cluster is in a good state:
1. To remove the migration configuration from the CNO configuration object, enter the following command:
  $ oc patch Network.operator.openshift.io cluster --type='merge' \ --patch '{ "spec": { "migration": null } }'
2. To remove custom configuration for the OpenShift SDN network provider, enter the following command:
  $ oc patch Network.operator.openshift.io cluster --type='merge' \ --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'
3. To remove the OpenShift SDN network provider namespace, enter the following command:
  $ oc delete namespace openshift-sdn

Limited live migration to the OVN-Kubernetes network plugin overview

The limited live migration method is the process in which the OpenShift SDN network plugin and its network configurations, connections, and associated resources, are migrated to the OVN-Kubernetes network plugin without service interruption. It is available for OKD.

Before you migrate your OKD cluster to use the OVN-Kubernetes network plugin, update your cluster to the latest z-stream release so that all the latest bug fixes apply to your cluster.

It is not available for hosted control plane deployment types. This migration method is valuable for deployment types that require constant service availability and offers the following benefits:

Continuous service availability
Minimized downtime
Automatic node rebooting
Seamless transition from the OpenShift SDN network plugin to the OVN-Kubernetes network plugin

Although a rollback procedure is provided, the limited live migration is intended to be a one-way process.

The following sections provide more information about the limited live migration method.

Supported platforms when using the limited live migration method

The following table provides information about the supported platforms for the limited live migration type.

Table 6. Supported platforms for the limited live migration method
Platform	Limited Live Migration
Bare-metal hardware	✓
Amazon Web Services (AWS)	✓
Google Cloud Platform (GCP)	✓
IBM Cloud®	✓
Microsoft Azure	✓
OpenStack	✓
VMware vSphere	✓
Nutanix	✓

Each listed platform supports installing an OKD cluster on installer-provisioned infrastructure and user-provisioned infrastructure.

Best practices for limited live migration to the OVN-Kubernetes network plugin

For a list of best practices when migrating to the OVN-Kubernetes network plugin with the limited live migration method, see Limited Live Migration from OpenShift SDN to OVN-Kubernetes.

Considerations for limited live migration to the OVN-Kubernetes network plugin

Before using the limited live migration method to the OVN-Kubernetes network plugin, cluster administrators should consider the following information:

The limited live migration procedure is unsupported for clusters with OpenShift SDN multitenant mode enabled.
Egress router pods block the limited live migration process. They must be removed before beginning the limited live migration process.
During the limited live migration, multicast, egress IP addresses, and egress firewalls are temporarily disabled. They can be migrated from OpenShift SDN to OVN-Kubernetes after the limited live migration process has finished.
The migration is intended to be a one-way process. However, for users that want to rollback to OpenShift-SDN, migration from OpenShift-SDN to OVN-Kubernetes must have succeeded. Users can follow the same procedure below to migrate to the OpenShift SDN network plugin from the OVN-Kubernetes network plugin.
The limited live migration is not supported on HyperShift clusters.
OpenShift SDN does not support IPsec. After the migration, cluster administrators can enable IPsec.
OpenShift SDN does not support IPv6. After the migration, cluster administrators can enable dual-stack.
The OpenShift SDN plugin allows application of the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) to the primary interface on a node. The OVN-Kubernetes network plugin does not have this capability.
The cluster MTU is the MTU value for pod interfaces. It is always less than your hardware MTU to account for the cluster network overlay overhead. The overhead is 100 bytes for OVN-Kubernetes and 50 bytes for OpenShift SDN.

During the limited live migration, both OVN-Kubernetes and OpenShift SDN run in parallel. OVN-Kubernetes manages the cluster network of some nodes, while OpenShift SDN manages the cluster network of others. To ensure that cross-CNI traffic remains functional, the Cluster Network Operator updates the routable MTU to ensure that both CNIs share the same overlay MTU. As a result, after the migration has completed, the cluster MTU is 50 bytes less.
Some parameters of OVN-Kubernetes cannot be changed after installation. The following parameters can be set only before starting the limited live migration:
- InternalTransitSwitchSubnet
- internalJoinSubnet
OVN-Kubernetes reserves the 100.64.0.0/16 and 100.88.0.0/16 IP address ranges. These subnets cannot be overlapped with any other internal or external network. If these IP addresses have been used by OpenShift SDN or any external networks that might communicate with this cluster, you must patch them to use a different IP address range before starting the limited live migration. See "Patching OVN-Kubernetes address ranges" for more information.
In most cases, the limited live migration is independent of the secondary interfaces of pods created by the Multus CNI plugin. However, if these secondary interfaces were set up on the default network interface controller (NIC) of the host, for example, using MACVLAN, IPVLAN, SR-IOV, or bridge interfaces with the default NIC as the control node, OVN-Kubernetes might encounter malfunctions. Users should remove such configurations before proceeding with the limited live migration.
When there are multiple NICs inside of the host, and the default route is not on the interface that has the Kubernetes NodeIP, you must use the offline migration instead.
All DaemonSet objects in the openshift-sdn namespace, which are not managed by the Cluster Network Operator (CNO), must be removed before initiating the limited live migration. These unmanaged daemon sets can cause the migration status to remain incomplete if not properly handled.
If you run an Operator or you have configured any application with the pod disruption budget, you might experience an interruption during the update process. If minAvailable is set to 1 in PodDisruptionBudget, the nodes are drained to apply pending machine configs which might block the eviction process. If several nodes are rebooted, all the pods might run on only one node, and the PodDisruptionBudget field can prevent the node drain.

Additional resources

How the limited live migration process works

The following table summarizes the limited live migration process by segmenting between the user-initiated steps in the process and the actions that the migration script performs in response.

Table 7. Limited live migration to OVNKubernetes from OpenShiftSDN
User-initiated steps	Migration activity
Patch the cluster-level networking configuration by changing the `networkType` from `OpenShiftSDN` to `OVNKubernetes`.	Cluster Network Operator (CNO) Sets migration related fields in the `network.operator` custom resource (CR) and waits for routable MTUs to be applied to all nodes. Patches the `network.operator` CR to set the migration mode to `Live` for OVN-Kubernetes and deploys the OpenShift SDN network plugin in migration mode. Deploys OVN-Kubernetes with hybrid overlay enabled, ensuring that no racing conditions occur. Waits for the OVN-Kubernetes deployment and updates the conditions in the status of the `network.config` CR. Triggers the Machine Config Operator (MCO) to apply the new machine config to each machine config pool, which includes node cordoning, draining, and rebooting. OVN-Kubernetes adds nodes to the appropriate zones and recreates pods using OVN-Kubernetes as the default CNI plugin. Removes migration-related fields from the network.operator CR and performs cleanup actions, such as deleting OpenShift SDN resources and redeploying OVN-Kubernetes in normal mode with the necessary configurations. Waits for the OVN-Kubernetes redeployment and updates the status conditions in the `network.config` CR to indicate migration completion. If your migration is blocked, see "Checking limited live migration metrics" for information on troubleshooting the issue.

Migrating to the OVN-Kubernetes network plugin by using the limited live migration method

Migrating to the OVN-Kubernetes network plugin by using the limited live migration method is a multiple step process that requires users to check the behavior of egress IP resources, egress firewall resources, and multicast enabled namespaces. Administrators must also review any network policies in their deployment and remove egress router resources before initiating the limited live migration process. The following procedures should be used in succession.

Checking cluster resources before initiating the limited live migration

Before migrating to OVN-Kubernetes by using the limited live migration, you should check for egress IP resources, egress firewall resources, and multicast-enabled namespaces on your OpenShift SDN deployment. You should also review any network policies in your deployment. If you find that your cluster has these resources before migration, you should check their behavior after migration to ensure that they are working as intended.

The following procedure shows you how to check for egress IP resources, egress firewall resources, multicast-enabled namespaces, network policies, and an NNCP. No action is necessary after checking for these resources.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

As an OKD cluster administrator, check for egress firewall resources. You can do this by using the oc CLI, or by using the OKD web console.
1. To check for egress firewall resource by using the oc CLI tool:
  1. To check for egress firewall resources, enter the following command:
    
    $ oc get egressnetworkpolicies.network.openshift.io -A
    
    Example output
    
    NAMESPACE NAME AGE <namespace> <example_egressfirewall> 5d
  2. You can check the intended behavior of an egress firewall resource by using the -o yaml flag. For example:
    
    $ oc get egressnetworkpolicy <example_egressfirewall> -n <namespace> -o yaml
    
    Example output
    
    apiVersion: network.openshift.io/v1 kind: EgressNetworkPolicy metadata: name: <example_egress_policy> namespace: <namespace> spec: egress: - type: Allow to: cidrSelector: 0.0.0.0/0 - type: Deny to: cidrSelector: 10.0.0.0/8
2. To check for egress firewall resources by using the OKD web console:
  1. On the OKD web console, click Observe → Metrics.
  2. In the Expression box, type sdn_controller_num_egress_firewalls and click Run queries. If you have egress firewall resources, they are returned in the Expression box.
Check your cluster for egress IP resources. You can do this by using the oc CLI, or by using the OKD web console.
1. To check for egress IPs by using the oc CLI tool:
  1. To list namespaces with egress IP resources, enter the following command
    
    $ oc get netnamespace -A | awk '$3 != ""'
    
    Example output
    
    NAME NETID EGRESS IPS namespace1 14173093 ["10.0.158.173"] namespace2 14173020 ["10.0.158.173"]
2. To check for egress IPs by using the OKD web console:
  1. On the OKD web console, click Observe → Metrics.
  2. In the Expression box, type sdn_controller_num_egress_ips and click Run queries. If you have egress firewall resources, they are returned in the Expression box.
Check your cluster for multicast enabled namespaces. You can do this by using the oc CLI, or by using the OKD web console.
1. To check for multicast enabled namespaces by using the oc CLI tool:
  1. To locate namespaces with multicast enabled, enter the following command:
    
    $ oc get netnamespace -o json | jq -r '.items[] | select(.metadata.annotations."netnamespace.network.openshift.io/multicast-enabled" == "true") | .metadata.name'
    
    Example output
    
    namespace1 namespace3
2. To check for multicast enabled namespaces by using the OKD web console:
  1. On the OKD web console, click Observe → Metrics.
  2. In the Expression box, type sdn_controller_num_multicast_enabled_namespaces and click Run queries. If you have multicast enabled namespaces, they are returned in the Expression box.
Check your cluster for any network policies. You can do this by using the oc CLI.
1. To check for network policies by using the oc CLI tool, enter the following command:
  $ oc get networkpolicy -n <namespace>
  Example output
  NAME POD-SELECTOR AGE allow-multicast app=my-app 11m

Removing egress router pods before initiating the limited live migration

Before initiating the limited live migration, you must check for, and remove, any egress router pods. If there is an egress router pod on the cluster when performing a limited live migration, the Network Operator blocks the migration and returns the following error:

The cluster configuration is invalid (network type limited live migration is not supported for pods with `pod.network.openshift.io/assign-macvlan` annotation.

Please remove all egress router pods). Use `oc edit network.config.openshift.io cluster` to fix.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

To locate egress router pods on your cluster, enter the following command:

$ oc get pods --all-namespaces -o json | jq '.items[] | select(.metadata.annotations."pod.network.openshift.io/assign-macvlan" == "true") | {name: .metadata.name, namespace: .metadata.namespace}'

Example output

{
  "name": "egress-multi",
  "namespace": "egress-router-project"
}

Alternatively, you can query metrics on the OKD web console.
1. On the OKD web console, click Observe → Metrics.
2. In the Expression box, enter network_attachment_definition_instances{networks="egress-router"}. Then, click Add.

To remove an egress router pod, enter the following command:

$ oc delete pod <egress_pod_name> -n <egress_router_project>

Removing the `NodeNetworkConfigurationPolicy` (NNCP) custom resource (CR)

The OpenShift SDN plugin allows application of the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) to the primary interface on a node. The OVN-Kubernetes network plugin does not have this capability.

If you have an NNCP applied to the primary interface, you must delete the NNCP before migrating to the OVN-Kubernetes network plugin. Deleting the NNCP does not remove the configuration from the primary interface, but the Kubernetes-NMState cannot manage this configuration. Instead, the configure-ovs.sh shell script manages the primary interface and the configuration attached to it.

Prerequisites

You created an NNCP CR and applied it to the primary interface of your network.

Procedure

Remove the configuration from the Cluster Network Operator (CNO) configuration object by running the following command:
```
$ oc patch Network.operator.openshift.io cluster --type='merge' \ --patch '{"spec":{"migration":null}}'
```
Delete the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) that defines the primary network interface for the OpenShift SDN network plugin by completing the following steps:
1. Check that the existing NNCP CR bonded the primary interface to your cluster by entering the following command:
  $ oc get nncp
  Example output
  NAME STATUS REASON bondmaster0 Available SuccessfullyConfigured
  Network Manager stores the connection profile for the bonded primary interface in the /etc/NetworkManager/system-connections system path.
2. Remove the NNCP from your cluster:
  $ oc delete nncp <nncp_manifest_filename>

Initiating the limited live migration process

After you have checked the behavior of egress IP resources, egress firewall resources, and multicast enabled namespaces, and removed any egress router resources, you can initiate the limited live migration process.

Prerequisites

A cluster has been configured with the OpenShift SDN CNI network plugin in the network policy isolation mode.
You have installed the OpenShift CLI (oc).
You have access to the cluster as a user with the cluster-admin role.
You have created a recent backup of the etcd database.
The cluster is in a known good state without any errors.
Before migration to OVN-Kubernetes, a security group rule must be in place to allow UDP packets on port 6081 for all nodes on all cloud platforms.
If the 100.64.0.0/16 and 100.88.0.0/16 address ranges were previously in use by OpenShift-SDN, you have patched them. The first step of this procedure checks whether these address ranges are in use. If they are in use, see "Patching OVN-Kubernetes address ranges".
You have checked for egress IP resources, egress firewall resources, and multicast enabled namespaces.
You have removed any egress router pods before beginning the limited live migration. For more information about egress router pods, see "Deploying an egress router pod in redirect mode".
You have reviewed the "Considerations for limited live migration to the OVN-Kubernetes network plugin" section of this document.

Procedure

To patch the cluster-level networking configuration and initiate the migration from OpenShift SDN to OVN-Kubernetes, enter the following command:

$ oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OVNKubernetes"}}'

After running this command, the migration process begins. During this process, the Machine Config Operator reboots the nodes in your cluster twice. The migration takes approximately twice as long as a cluster upgrade.

This oc patch command checks for overlapping CIDRs in use by OpenShift SDN. If overlapping CIDRs are detected, you must patch them before the limited live migration process can start. For more information, see "Patching OVN-Kubernetes address ranges".

Optional: To ensure that the migration process has completed, and to check the status of the network.config, you can enter the following commands:
```
$ oc get network.config.openshift.io cluster -o jsonpath='{.status.networkType}'
```
```
$ oc get network.config cluster -o=jsonpath='{.status.conditions}' | jq .
```
You can check limited live migration metrics for troubleshooting issues. For more information, see "Checking limited live migration metrics".

Patching OVN-Kubernetes address ranges

OVN-Kubernetes reserves the following IP address ranges:

100.64.0.0/16. This IP address range is used for the internalJoinSubnet parameter of OVN-Kubernetes by default.
100.88.0.0/16. This IP address range is used for the internalTransSwitchSubnet parameter of OVN-Kubernetes by default.

If these IP addresses have been used by OpenShift SDN or any external networks that might communicate with this cluster, you must patch them to use a different IP address range before initiating the limited live migration.

The following procedure can be used to patch CIDR ranges that are in use by OpenShift SDN if the migration was initially blocked.

This is an optional procedure and must only be used if the migration was blocked after using the oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OVNKubernetes"}}' command "Initiating the limited live migration process".

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

If the 100.64.0.0/16 IP address range is already in use, enter the following command to patch it to a different range. The following example uses 100.63.0.0/16.

$ oc patch network.operator.openshift.io cluster --type='merge' -p='{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"ipv4":{"internalJoinSubnet": "100.63.0.0/16"}}}}}'

If the 100.88.0.0/16 IP address range is already in use, enter the following command to patch it to a different range. The following example uses 100.99.0.0/16.

$ oc patch network.operator.openshift.io cluster --type='merge' -p='{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"ipv4":{"internalTransitSwitchSubnet": "100.99.0.0/16"}}}}}'

After patching the 100.64.0.0/16 and 100.88.0.0/16 IP address ranges, you can initiate the limited live migration.

Checking cluster resources after initiating the limited live migration

The following procedure shows you how to check for egress IP resources, egress firewall resources, multicast enabled namespaces, and network policies when your deploying is using OVN-Kubernetes. If you had these resources on OpenShift SDN, you should check them after migration to ensure that they are working properly.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have successfully migrated from OpenShift SDN to OVN-Kubernetes by using the limited live migration.

Procedure

As an OKD cluster administrator, check for egress firewall resources. You can do this by using the oc CLI, or by using the OKD web console.
1. To check for egress firewall resource by using the oc CLI tool:
  1. To check for egress firewall resources, enter the following command:
    
    $ oc get egressfirewalls.k8s.ovn.org -A
    
    Example output
    
    NAMESPACE NAME AGE <namespace> <example_egressfirewall> 5d
  2. You can check the intended behavior of an egress firewall resource by using the -o yaml flag. For example:
    
    $ oc get egressfirewall <example_egressfirewall> -n <namespace> -o yaml
    
    Example output
    
    apiVersion: k8s.ovn.org/v1 kind: EgressFirewall metadata: name: <example_egress_policy> namespace: <namespace> spec: egress: - type: Allow to: cidrSelector: 192.168.0.0/16 - type: Deny to: cidrSelector: 0.0.0.0/0
    
    Ensure that the behavior of this resource is intended because it could have changed after migration. For more information about egress firewalls, see "Configuring an egress firewall for a project".
2. To check for egress firewall resources by using the OKD web console:
  1. On the OKD web console, click Observe → Metrics.
  2. In the Expression box, type ovnkube_controller_num_egress_firewall_rules and click Run queries. If you have egress firewall resources, they are returned in the Expression box.

Check your cluster for egress IP resources. You can do this by using the oc CLI, or by using the OKD web console.

To check for egress IPs by using the oc CLI tool:

To list the namespace with egress IP resources, enter the following command:

$ oc get egressip

Example output

NAME                EGRESSIPS    ASSIGNED NODE                              ASSIGNED EGRESSIPS
egress-sample       192.0.2.10   ip-10-0-42-79.us-east-2.compute.internal   192.0.2.10
egressip-sample-2   192.0.2.14   ip-10-0-42-79.us-east-2.compute.internal   192.0.2.14

To provide detailed information about an egress IP, enter the following command:

$ oc get egressip <egressip_name> -o yaml

Example output

apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"k8s.ovn.org/v1","kind":"EgressIP","metadata":{"annotations":{},"name":"egressip-sample"},"spec":{"egressIPs":["192.0.2.12","192.0.2.13"],"namespaceSelector":{"matchLabels":{"name":"my-namespace"}}}}
  creationTimestamp: "2024-06-27T15:48:36Z"
  generation: 7
  name: egressip-sample
  resourceVersion: "125511578"
  uid: b65833c8-781f-4cc9-bc96-d970259a7631
spec:
  egressIPs:
  - 192.0.2.12
  - 192.0.2.13
  namespaceSelector:
    matchLabels:
      name: my-namespace

Repeat this for all egress IPs. Ensure that the behavior of each resource is intended because it could have changed after migration. For more information about EgressIPs, see "Configuring an EgressIP address".

To check for egress IPs by using the OKD web console:
1. On the OKD web console, click Observe → Metrics.
2. In the Expression box, type ovnkube_clustermanager_num_egress_ips and click Run queries. If you have egress firewall resources, they are returned in the Expression box.

Check your cluster for multicast enabled namespaces. You can only do this by using the oc CLI.

To locate namespaces with multicast enabled, enter the following command:

$ oc get namespace -o json | jq -r '.items[] | select(.metadata.annotations."k8s.ovn.org/multicast-enabled" == "true") | .metadata.name'

Example output

namespace1
namespace3

To describe each multicast enabled namespace, enter the following command:

$ oc describe namespace <namespace>

Example output

Name:         my-namespace
Labels:       kubernetes.io/metadata.name=my-namespace
              pod-security.kubernetes.io/audit=restricted
              pod-security.kubernetes.io/audit-version=v1.24
              pod-security.kubernetes.io/warn=restricted
              pod-security.kubernetes.io/warn-version=v1.24
Annotations:  k8s.ovn.org/multicast-enabled: true
              openshift.io/sa.scc.mcs: s0:c25,c0
              openshift.io/sa.scc.supplemental-groups: 1000600000/10000
              openshift.io/sa.scc.uid-range: 1000600000/10000
Status:       Active

Ensure that multicast functionality is correctly configured and working as expected in each namespace. For more information, see "Enabling multicast for a project".

Check your cluster’s network policies. You can only do this by using the oc CLI.

To obtain information about network policies within a namespace, enter the following command:

$ oc get networkpolicy -n <namespace>

Example output

NAME              POD-SELECTOR   AGE
allow-multicast   app=my-app     11m

To provide detailed information about the network policy, enter the following command:

$ oc describe networkpolicy allow-multicast -n <namespace>

Example output

Name:         allow-multicast
Namespace:    my-namespace
Created on:   2024-07-24 14:55:03 -0400 EDT
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     app=my-app
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      IPBlock:
        CIDR: 224.0.0.0/4
        Except:
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To:
      IPBlock:
        CIDR: 224.0.0.0/4
        Except:
  Policy Types: Ingress, Egress

Ensure that the behavior of the network policy is as intended. Optimization for network policies differ between SDN and OVN-K, so users might need to adjust their policies to achieve optimal performance for different CNIs. For more information, see "About network policy".

Checking limited live migration metrics

Metrics are available to monitor the progress of the limited live migration. Metrics can be viewed on the OKD web console, or by using the oc CLI.

Prerequisites

You have initiated a limited live migration to OVN-Kubernetes.

Procedure

To view limited live migration metrics on the OKD web console:
1. Click Observe → Metrics.
2. In the Expression box, type openshift_network and click the openshift_network_operator_live_migration_procedure option.

To view metrics by using the oc CLI:

Enter the following command to generate a token for the prometheus-k8s service account in the openshift-monitoring namespace:
```
$ oc create token prometheus-k8s -n openshift-monitoring
```
Example output
```
eyJhbGciOiJSUzI1NiIsImtpZCI6IlZiSUtwclcwbEJ2VW9We...
```

Enter the following command to request information about the openshift_network_operator_live_migration_condition metric:

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: <eyJhbGciOiJSUzI1NiIsImtpZCI6IlZiSUtwclcwbEJ2VW9We...>" "https://<openshift_API_endpoint>" --data-urlencode "query=openshift_network_operator_live_migration_condition" | jq

Example output

 "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "openshift_network_operator_live_migration_condition",
          "container": "network-operator",
          "endpoint": "metrics",
          "instance": "10.0.83.62:9104",
          "job": "metrics",
          "namespace": "openshift-network-operator",
          "pod": "network-operator-6c87754bc6-c8qld",
          "prometheus": "openshift-monitoring/k8s",
          "service": "metrics",
          "type": "NetworkTypeMigrationInProgress"
        },
        "value": [
          1717653579.587,
          "1"
        ]
      },
...

The table in "Information about limited live migration metrics" shows you the available metrics and the label values populated from the openshift_network_operator_live_migration_procedure expression. Use this information to monitor progress or to troubleshoot the migration.

Information about limited live migration metrics

The following table shows you the available metrics and the label values populated from the openshift_network_operator_live_migration_procedure expression. Use this information to monitor progress or to troubleshoot the migration.

Table 8. Limited live migration metrics
Metric	Label values
`openshift_network_operator_live_migration_blocked:` A Prometheus gauge vector metric. A metric that contains a constant `1` value labeled with the reason that the CNI limited live migration might not have started. This metric is available when the CNI limited live migration has started by annotating the `Network` custom resource. This metric is not published unless the limited live migration is blocked.	The list of label values includes the following `UnsupportedCNI`: Unable to migrate to the unsupported target CNI. Valid CNI is `OVNKubernetes` when migrating from OpenShift SDN. `UnsupportedHyperShiftCluster`: Limited live migration is unsupported within an HCP cluster. `UnsupportedSDNNetworkIsolationMode`: OpenShift SDN is configured with an unsupported network isolation mode `Multitenant`. Migrate to a supported network isolation mode before performing limited live migration. `UnsupportedMACVLANInterface`: Remove the egress router or any pods which contain the pod annotation `pod.network.openshift.io/assign-macvlan`. Find the offending pod’s namespace or pod name with the following command: `oc get pods -Ao=jsonpath='{range .items[?(@.metadata.annotations.pod\.network\.openshift\.io/assign-macvlan=="")]}{@.metadata.namespace}{"\t"}{@.metadata.name}{"\n"}'`.
`openshift_network_operator_live_migration_condition:` A metric which represents the status of each condition type for the CNI limited live migration. The set of status condition types is defined for `network.config` to support observability of the CNI limited live migration. A `1` value represents condition status `true`. A `0` value represents `false`. `-1` represents unknown. This metric is available when the CNI limited live migration has started by annotating the `Network` custom resource (CR). This metric is only available when the limited live migration has been triggered by adding the relevant annotation to the `Network` CR cluster, otherwise, it is not published. If the following condition types are not present within the Network CR cluster, the metric and their labels are cleared.	The list of label values includes the following `NetworkTypeMigrationInProgress` `NetworkTypeMigrationTargetCNIAvailable` `NetworkTypeMigrationTargetCNIInUse` `NetworkTypeMigrationOriginalCNIPurged` `NetworkTypeMigrationMTUReady`

Offline migration to the OVN-Kubernetes network plugin overview

Supported platforms when using the offline migration method

Best practices for offline migration to the OVN-Kubernetes network plugin

Considerations for offline migration to the OVN-Kubernetes network plugin

Primary network interface

Namespace isolation

Egress IP addresses

Egress network policies

Egress router pods

Multicast

Network policies

How the offline migration process works

Migrating to the OVN-Kubernetes network plugin by using the offline migration method

Limited live migration to the OVN-Kubernetes network plugin overview

Supported platforms when using the limited live migration method

Best practices for limited live migration to the OVN-Kubernetes network plugin

Considerations for limited live migration to the OVN-Kubernetes network plugin

How the limited live migration process works

Migrating to the OVN-Kubernetes network plugin by using the limited live migration method

Checking cluster resources before initiating the limited live migration

Removing egress router pods before initiating the limited live migration

Removing the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR)

Initiating the limited live migration process

Patching OVN-Kubernetes address ranges

Checking cluster resources after initiating the limited live migration

Checking limited live migration metrics

Information about limited live migration metrics

Additional resources

Removing the `NodeNetworkConfigurationPolicy` (NNCP) custom resource (CR)