Migrating to a cluster with multi-architecture compute machines - Performing a cluster update | Updating clusters

Migrating to a cluster with multi-architecture compute machines using the CLI
Migrating the x86 control plane to arm64 architecture on Amazon Web Services
Migrating control plane or infra machine sets between architectures on Google Cloud Platform

You can migrate your current cluster with single-architecture compute machines to a cluster with multi-architecture compute machines by updating to a multi-architecture, manifest-listed payload. This allows you to add mixed architecture compute nodes to your cluster.

For information about configuring your multi-architecture compute machines, see "Configuring multi-architecture compute machines on an OKD cluster".

Before migrating your single-architecture cluster to a cluster with multi-architecture compute machines, it is recommended to install the Multiarch Tuning Operator, and deploy a ClusterPodPlacementConfig custom resource. For more information, see Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator.

Migration from a multi-architecture payload to a single-architecture payload is not supported. Once a cluster has transitioned to using a multi-architecture payload, it can no longer accept a single-architecture update payload.

Migrating to a cluster with multi-architecture compute machines using the CLI

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
Your OKD version is up to date to at least version 4.13.0.

For more information on how to update your cluster version, see Updating a cluster using the web console or Updating a cluster using the CLI.
You have installed the OpenShift CLI (oc) that matches the version for your current cluster.
Your oc client is updated to at least verion 4.13.0.
Your OKD cluster is installed on AWS, Azure, GCP, bare metal or IBM P/Z platforms.

For more information on selecting a supported platform for your cluster installation, see Selecting a cluster installation type.

Procedure

Verify that the RetrievedUpdates condition is True in the Cluster Version Operator (CVO) by running the following command:
```
$ oc get clusterversion/version -o=jsonpath="{.status.conditions[?(.type=='RetrievedUpdates')].status}"
```
If the RetrievedUpates condition is False, you can find supplemental information regarding the failure by using the following command:
```
$ oc adm upgrade
```
For more information about cluster version condition types, see Understanding cluster version condition types.
Migrate to the multi-architecture payload with following command:
```
$ oc adm upgrade --to-multi-arch
```

Verification

You can monitor the migration by running the following command:

$ oc adm upgrade

Example output

working towards ${VERSION}: 106 of 841 done (12% complete), waiting on machine-config

Machine launches may fail as the cluster settles into the new state. To notice and recover when machines fail to launch, we recommend deploying machine health checks. For more information about machine health checks and how to deploy them, see About machine health checks.

Optional: To retrieve more detailed information about the status of your update, monitor the migration by running the following command:
```
$ oc adm upgrade status
```
For more information about how to use the oc adm upgrade status command, see Gathering cluster update status using oc adm upgrade status (Technology Preview).

The migrations must be complete and all the cluster operators must be stable before you can add compute machine sets with different architectures to your cluster.

Additional resources

Migrating the x86 control plane to arm64 architecture on Amazon Web Services

You can migrate the control plane in your cluster from x86 to arm64 architecture on Amazon Web Services (AWS).

Prerequisites

You have installed the OpenShift CLI (oc).
You logged in to oc as a user with cluster-admin privileges.

Procedure

Check the architecture of the control plane nodes by running the following command:

$ oc get nodes -o wide

Example output

NAME                          STATUS   ROLES                  AGE    VERSION   INTERNAL-IP EXTERNAL-IP   OS-IMAGE                                         KERNEL-VERSION                 CONTAINER-RUNTIME
worker-001.example.com        Ready    worker                 100d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
worker-002.example.com        Ready    worker                 98d    v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
worker-003.example.com        Ready    worker                 98d    v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
master-001.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
master-002.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
master-003.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x

The KERNEL-VERSION field in the output indicates the architecture of the nodes.

Check that your cluster uses the multi payload by running the following command:
```
$ oc adm release info -o jsonpath="{ .metadata.metadata}"
```
If you see the following output, the cluster is multi-architecture compatible.
```
{
 "release.openshift.io/architecture": "multi",
 "url": "https://access.redhat.com/errata/<errata_version>"
}
```
If the cluster is not using the multi payload, migrate the cluster to a multi-architecture cluster. For more information, see "Migrating to a cluster with multi-architecture compute machines".

Update your image stream from single-architecture to multi-architecture by running the following command:

$ oc import-image <multiarch_image_stream_tag>  --from=<registry>/<project_name>/<image_name> \
--import-mode='PreserveOriginal'

Get the arm64 compatible Amazon Machine Image (AMI) for configuring the control plane machine set by running the following command:
```
$ oc get configmap/coreos-bootimages -n openshift-machine-config-operator -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64.images.aws.regions."<aws_region>".image' (1)
```
1 Replace <aws_region> with the AWS region where the current cluster is installed. You can get the AWS region for the installed cluster by running the following command:
```
$ oc get infrastructure cluster -o jsonpath='{.status.platformStatus.aws.region}'
```
Example output
```
ami-xxxxxxx
```
Update the control plane machine set to support the arm64 architecture by running the following command:
```
$ oc edit controlplanemachineset.machine.openshift.io cluster -n openshift-machine-api
```
Update the instanceType field to a type that supports the arm64 architecture, and set the ami.id field to an AMI that is compatible with the arm64 architecture. For information about supported instance types, see "Tested instance types for AWS on 64-bit ARM infrastructures".

For more information about configuring the control plane machine set for AWS, see "Control plane configuration options for Amazon Web Services".

Verification

Verify that the control plane nodes are now running on the arm64 architecture:

$ oc get nodes -o wide

Example output

NAME                          STATUS   ROLES                  AGE    VERSION   INTERNAL-IP EXTERNAL-IP   OS-IMAGE                                         KERNEL-VERSION                 CONTAINER-RUNTIME
worker-001.example.com        Ready    worker                 100d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
worker-002.example.com        Ready    worker                 98d    v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
worker-003.example.com        Ready    worker                 98d    v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
master-001.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.aarch64   cri-o://1.30.x
master-002.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.aarch64   cri-o://1.30.x
master-003.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.aarch64   cri-o://1.30.x

Migrating control plane or infra machine sets between architectures on Google Cloud Platform

You can migrate the control plane or infra machine sets in your GCP cluster between x86 and arm64 architectures.

Prerequisites

You have installed the OpenShift CLI (oc).
You logged in to oc as a user with cluster-admin privileges.

Procedure

Check the architecture of the control plane or infra nodes by running the following command:

$ oc get nodes -o wide

Example output

NAME                          STATUS   ROLES                  AGE    VERSION   INTERNAL-IP EXTERNAL-IP   OS-IMAGE                                         KERNEL-VERSION                 CONTAINER-RUNTIME
worker-001.example.com        Ready    infra                 100d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x
master-001.example.com        Ready    control-plane,master   120d   v1.30.7   10.x.x.x    <none>        Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0   5.x.x-xxx.x.x.el9_xx.x86_64    cri-o://1.30.x

The KERNEL-VERSION field in the output indicates the architecture of the nodes.

Check that your cluster uses the multi payload by running the following command:
```
$ oc adm release info -o jsonpath="{ .metadata.metadata}"
```
If you see the following output, the cluster is multi-architecture compatible.
```
{
 "release.openshift.io/architecture": "multi",
 "url": "https://access.redhat.com/errata/<errata_version>"
}
```
If the cluster is not using the multi payload, migrate the cluster to a multi-architecture cluster. For more information, see "Migrating to a cluster with multi-architecture compute machines".
If you use any custom image streams, update them from single-architecture to multi-architecture by running the following command for each image stream:
$ oc import-image <multiarch_image_stream_tag> --from=<registry>/<project_name>/<image_name> \ --import-mode='PreserveOriginal'
Select an instance type that matches the target architecture from General-purpose machine family for Compute engine (Google documentation). Check the Available regions and zones table (Google documentation) to verify that the instance type is supported in your zone.
Select a supported disk type for the instance type that you selected from the "Supported disk types" section of General-purpose machine family for Compute engine (Google documentation).

Determine the GCP image that the machine set uses after migration by running the following command:

$ oc get configmap/coreos-bootimages \
  -n openshift-machine-config-operator \
  -o jsonpath='{.data.stream}' | jq \
  -r '.architectures.aarch64.images.gcp'

Example output

"gcp": {
    "release": "415.92.202309142014-0",
    "project": "rhcos-cloud",
    "name": "rhcos-415-92-202309142014-0-gcp-aarch64"
  }

Use the project and name parameters from the output to form the image parameter in the following format: projects/<project>/global/images/<name>.

To migrate the control plane to another architecture, run the following command:
```
$ oc edit controlplanemachineset.machine.openshift.io cluster -n openshift-machine-api
```
1. Replace the disks.type parameter with the disk type that you selected.
2. Replace the disks.image parameter with the image parameter that you formed previously.
3. Replace the machineType parameter with the instance type that you selected.
To migrate an infra machine set to another architecture, run the following command using the ID of an infra machine set:
```
$ oc edit machineset <infra-machine-set_id> -n openshift-machine-api
```
1. Replace the disks.type parameter with the disk type that you selected.
2. Replace the disks.image parameter with the image parameter that you formed previously.
3. Replace the machineType parameter with the instance type that you selected.

Additional resources