Heterogeneous cluster support - Advanced VM creation | Virtualization

Enabling heterogeneous cluster support
Modifying a common boot source image in a heterogeneous cluster
Adding a custom boot source image in a heterogeneous cluster
Modifying workloads node placement in a heterogeneous cluster

A heterogeneous cluster is a cluster where nodes have differing architectures. Heterogeneous clusters promote optimal compute resource usage by mixing different types of hardware in one cluster.

Boot source image support for heterogeneous clusters is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

With heterogeneous clusters, you can match workloads to hardware intended for the workload task instead of general purpose compute platforms. For example, GPU and general purpose compute resources could be combined and workloads assigned to the appropriate hardware.

If boot source image support is disabled in a heterogeneous cluster, you can encounter inconsistencies between node and image architectures. This happens when images are used for virtual machine creation that do not match the node architecture. This can lead to the failure of virtual machine boot up or virtual machines that do not run as expected. The warning level alert HCOMultiArchGoldenImagesDisabled is produced when this feature is not enabled in a heterogeneous cluster.

If you have a heterogeneous cluster but do not want to enable multiple architecture support, you can modify the workloads node placement in the HyperConverged custom resource (CR) to include only nodes with a specific architecture.

Boot source image support in heterogeneous clusters enables VM creators to deploy persistent VMs with specific architectures, and to define custom boot images that support heterogeneous clusters.

The same image can be used with nodes of different architectures if the boot image supports the required architectures. For example, a boot image that supports both ARM and AMD architectures can be used with both types of nodes.

Boot source image support for heterogeneous clusters is not enabled by default. You can enable heterogeneous cluster support by setting the feature gate in the HyperConverged CR.

Enabling heterogeneous cluster support

You can enable boot source image support for heterogeneous clusters by setting the enableMultiArchBootImageImport feature gate to true in the HyperConverged custom resource (CR).

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have access to the cluster as a user with cluster-admin permissions.
You have installed the OpenShift CLI (oc).

Procedure

Enable the enableMultiArchBootImageImport feature gate by running the following command:

$ oc patch hco kubevirt-hyperconverged -n kubevirt-hyperconverged \
  --type json -p '[{"op": "add", "path": "/spec/featureGates/-", \
  "value": {"name": "enableMultiArchBootImageImport"}}]'

Modifying a common boot source image in a heterogeneous cluster

You can modify the source of a common boot source image in a heterogeneous cluster by specifying the supported architectures in the ssp.kubevirt.io/dict.architectures annotation in the HyperConverged custom resource (CR).

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hco kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, adding the appropriate values for ssp.kubevirt.io/dict.architectures annotation in the dataImportCronTemplates section. For example:

#...
spec:
  workloadSources:
    dataImportCronTemplates:
    - metadata:
        name: kubevirt-hyperconverged
        annotations:
          ssp.kubevirt.io/dict.architectures: "<architecture_list>"
      spec:
        schedule: "0 */12 * * *"
        template:
          spec:
            source:
              registry:
                  url: docker://my-private-registry/my-own-version-of-centos:8
        managedDataSource: centos-stream8
#...

where:

ssp.kubevirt.io/dict.architectures: Specifies a comma-separated list of supported architectures for this image. For example, if the image supports amd64 and arm64 architectures, the value would be "amd64,arm64".

Save and exit the editor to update the HyperConverged CR.

Adding a custom boot source image in a heterogeneous cluster

Add a custom boot source image in a heterogeneous cluster by editing the HyperConverged custom resource (CR).

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hco kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, to add the custom boot source image. You must add the appropriate values for the ssp.kubevirt.io/dict.architectures annotation in the dataImportCronTemplates section. For example:

apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  workloadSources:
    dataImportCronTemplates:
    - metadata:
        name: custom-image1
        annotations:
          ssp.kubevirt.io/dict.architectures: "<architecture_list>"
      spec:
        schedule: "0 */12 * * *"
        template:
          spec:
            source:
              registry:
                url: docker://myprivateregistry/custom1
        managedDataSource: custom1
        retentionPolicy: "All"
#...

where:

<architecture_list>

Specifies a comma-separated list of supported architectures for this image. For example, if the image supports amd64 and arm64 architectures, the value would be "amd64,arm64".

An image may support more architectures than you want to use in your cluster. You do not have to list all of the architectures an image supports, only those for which you want to create a boot source image.

Save and exit the editor to update the HyperConverged CR.

Modifying workloads node placement in a heterogeneous cluster

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hco kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, to modify the workloads node placement to include only nodes with a specific architecture. For example:

apiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
#...
  deployment:
    nodePlacements:
      workload:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: kubernetes.io/arch
                      operator: In
                      values:
                        - <node_architecture>

where:

<node_architecture>: Specifies the target architecture. For example, to limit placement to AMD nodes, use amd64.

Save and exit the editor to update the HyperConverged CR.