Heterogeneous cluster support - Advanced VM creation | Virtualization

Enabling heterogeneous cluster support
Modifying a common golden image source in a heterogeneous cluster
Adding a custom golden image in a heterogeneous cluster
Modifying workloads node placement in a heterogeneous cluster

Golden image support for heterogeneous clusters is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

A heterogeneous cluster is a cluster where nodes have differing architectures. Heterogeneous clusters promote optimal compute resource usage by mixing different types of hardware in one cluster. This allows workloads to be better matched to hardware intended for the workload task instead of general purpose compute platforms. For example, in a heterogeneous cluster, GPU and general purpose compute resources could be combined and workloads assigned to the appropriate hardware.

If golden image support is disabled in a heterogeneous cluster, you can encounter inconsistencies between node and image architectures. This happens when images are used for virtual machine creation that do not match the node architecture. This can lead to the failure of virtual machine boot up or virtual machines that do not run as expected. The warning level alert HCOMultiArchGoldenImagesDisabled is produced when this feature is not enabled in a heterogeneous cluster.

If you have a heterogeneous cluster but do not want to enable multiple architecture support, see Modifying workloads node placement in a hetergeneous cluster for the procedure to limit node placement to a specific architecture.

Golden image support for heterogeneous clusters extends golden image support in the following areas:

Enables VM creators to deploy persistent virtual machines with specific architectures.
Enables VM creators to define custom golden images that support heterogenous clusters.

The same golden image can be used with nodes of different architectures if the boot image supports the required architectures. For example, a golden image that supports both ARM and AMD architectures can be used with both types of nodes.

Golden image support for heterogeneous clusters is not enabled by default. For the procedure to enable this feature, see Enabling hetergenous cluster support

Enabling heterogeneous cluster support

You can enable golden image support for heterogeneous clusters by setting the enableMultiArchBootImageImport feature gate to true in the HyperConverged custom resource (CR).

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have access to the cluster as a user with cluster-admin permissions.
You have installed the OpenShift CLI (oc).

Procedure

Enable the enableMultiArchBootImageImport feature gate by running the following command:

$ oc patch hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged \
  --type json -p '[{"op":"replace","path":"/spec/featureGates/enableMultiArchBootImageImport", "value": true}]'

Modifying a common golden image source in a heterogeneous cluster

You can modify the image source of a common golden image in a heterogeneous cluster by specifying the supported architectures in the ssp.kubevirt.io/dict.architectures annotation in the HyperConverged custom resource (CR).

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, adding the appropriate values for ssp.kubevirt.io/dict.architectures annotation in the dataImportCronTemplates section. For example:

#...
spec:
  dataImportCronTemplates:
  - metadata:
      name: kubevirt-hyperconverged
      annotations:
        ssp.kubevirt.io/dict.architectures: "<architecture_list>" (1)
    spec:
      schedule: "0 */12 * * *"
      template:
        spec:
          source:
            registry:
                url: docker://my-private-registry/my-own-version-of-centos:8
      managedDataSource: centos-stream8
#...

1	The comma-separated list of supported architectures for this image. For example, if the image supports `amd64` and `arm64` architectures, the value would be `"amd64,arm64"`.

Save and exit the editor to update the HyperConverged CR.

Adding a custom golden image in a heterogeneous cluster

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Add a custom golden image in a heterogeneous cluster by setting the ssp.kubevirt.io/dict.architectures annotation in the spec.dataImportCronTemplates.metadata.annotations stanza of the HyperConverged custom resource (CR). This annotation lists the architectures supported by the image.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, to add the custom golden image. You must add the appropriate values for ssp.kubevirt.io/dict.architectures annotation in the dataImportCronTemplates section. For example:

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
  dataImportCronTemplates:
  - metadata:
      name: custom-image1
      annotations:
        ssp.kubevirt.io/dict.architectures: "<architecture_list>" (1)
    spec:
      schedule: "0 */12 * * *"
      template:
        spec:
          source:
            registry:
              url: docker://myprivateregistry/custom1
      managedDataSource: custom1
      retentionPolicy: "All"
#...

1	The comma-separated list of supported architectures for this image. For example, if the image supports `amd64` and `arm64` architectures, the value would be `"amd64,arm64"`.

An image may support more architectures than you want to use in your cluster. You do not have to list all of the architectures an image supports, only those for which you want to create a boot source.

Save and exit the editor to update the HyperConverged CR.

Modifying workloads node placement in a heterogeneous cluster

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If you have a heterogeneous cluster but not want to enable multiple archiecture support, you can modify the workloads node placement in the HyperConverged custom resource (CR) to only include nodes with a specific architecture.

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Open the HyperConverged CR in your default editor by running the following command:
```
$ oc edit hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged
```

Edit the HyperConverged CR, to modify the workloads node placement to include only nodes with a specific architecture. For example:

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
spec:
#...
  workloads:
    nodePlacement:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - <node_architecture> (1)

1	Replace `<node_architecture>` with the target architecture. For example, to limit placement to AMD nodes, use `amd64`.

Save and exit the editor to update the HyperConverged CR.