1. Documentation
×

Creating infrastructure machine sets

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.

In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

OKD infrastructure components

The following infrastructure workloads do not incur OKD worker subscriptions:

  • Kubernetes and OKD control plane services that run on masters

  • The default router

  • The integrated container image registry

  • The HAProxy-based Ingress Controller

  • The cluster metrics collection, or monitoring service, including components for monitoring user-defined projects

  • Cluster aggregated logging

  • Service brokers

  • Red Hat Quay

  • Red Hat OpenShift Data Foundation

  • Red Hat Advanced Cluster Manager

  • Red Hat Advanced Cluster Security for Kubernetes

  • Red Hat OpenShift GitOps

  • Red Hat OpenShift Pipelines

Any node that runs any other container, pod, or component is a worker node that your subscription must cover.

For information about infrastructure nodes and which components can run on infrastructure nodes, see the "Red Hat OpenShift control plane and infrastructure nodes" section in the OpenShift sizing and subscription guide for enterprise Kubernetes document.

To create an infrastructure node, you can use a machine set, label the node, or use a machine config pool.

Creating infrastructure machine sets for production environments

In a production deployment, it is recommended that you deploy at least three compute machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different compute machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

Creating infrastructure machine sets for different clouds

Use the sample compute machine set for your cloud.

Sample YAML for a compute machine set custom resource on Alibaba Cloud

This sample YAML defines a compute machine set that runs in a specified Alibaba Cloud zone in a region and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-<infra>-<zone> (3)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<zone> (3)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<zone> (3)
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1
          credentialsSecret:
            name: alibabacloud-credentials
          imageId: <image_id> (4)
          instanceType: <instance_type> (5)
          kind: AlibabaCloudMachineProviderConfig
          ramRoleName: <infrastructure_id>-role-worker (6)
          regionId: <region> (7)
          resourceGroup: (8)
            id: <resource_group_id>
            type: ID
          securityGroups:
          - tags: (9)
            - Key: Name
              Value: <infrastructure_id>-sg-<role>
            type: Tags
          systemDisk: (10)
            category: cloud_essd
            size: <disk_size>
          tag: (9)
          - Key: kubernetes.io/cluster/<infrastructure_id>
            Value: owned
          userDataSecret:
            name: <user_data_secret> (11)
          vSwitch:
            tags: (9)
            - Key: Name
              Value: <infrastructure_id>-vswitch-<zone>
            type: Tags
          vpcId: ""
          zoneId: <zone> (12)
      taints: (13)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the <infra> node label.
3 Specify the infrastructure ID, <infra> node label, and zone.
4 Specify the image to use. Use an image from an existing default compute machine set for the cluster.
5 Specify the instance type you want to use for the compute machine set.
6 Specify the name of the RAM role to use for the compute machine set. Use the value that the installer populates in the default compute machine set.
7 Specify the region to place machines on.
8 Specify the resource group and type for the cluster. You can use the value that the installer populates in the default compute machine set, or specify a different one.
9 Specify the tags to use for the compute machine set. Minimally, you must include the tags shown in this example, with appropriate values for your cluster. You can include additional tags, including the tags that the installer populates in the default compute machine set it creates, as needed.
10 Specify the type and size of the root disk. Use the category value that the installer populates in the default compute machine set it creates. If required, specify a different value in gigabytes for size.
11 Specify the name of the secret in the user data YAML file that is in the openshift-machine-api namespace. Use the value that the installer populates in the default compute machine set.
12 Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.
13 Specify a taint to prevent user workloads from being scheduled on infra nodes.
Machine set parameters for Alibaba Cloud usage statistics

The default compute machine sets that the installer creates for Alibaba Cloud clusters include nonessential tag values that Alibaba Cloud uses internally to track usage statistics. These tags are populated in the securityGroups, tag, and vSwitch parameters of the spec.template.spec.providerSpec.value list.

When creating compute machine sets to deploy additional machines, you must include the required Kubernetes tags. The usage statistics tags are applied by default, even if they are not specified in the compute machine sets you create. You can also include additional tags as needed.

The following YAML snippets indicate which tags in the default compute machine sets are optional and which are required.

Tags in spec.template.spec.providerSpec.value.securityGroups
spec:
  template:
    spec:
      providerSpec:
        value:
          securityGroups:
          - tags:
            - Key: kubernetes.io/cluster/<infrastructure_id> (1)
              Value: owned
            - Key: GISV
              Value: ocp
            - Key: sigs.k8s.io/cloud-provider-alibaba/origin (1)
              Value: ocp
            - Key: Name
              Value: <infrastructure_id>-sg-<role> (2)
            type: Tags
1 Optional: This tag is applied even when not specified in the compute machine set.
2 Required.

where:

  • <infrastructure_id> is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster.

  • <role> is the node label to add.
Tags in spec.template.spec.providerSpec.value.tag
spec:
  template:
    spec:
      providerSpec:
        value:
          tag:
          - Key: kubernetes.io/cluster/<infrastructure_id> (2)
            Value: owned
          - Key: GISV (1)
            Value: ocp
          - Key: sigs.k8s.io/cloud-provider-alibaba/origin (1)
            Value: ocp
1 Optional: This tag is applied even when not specified in the compute machine set.
2 Required.

where <infrastructure_id> is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster.
Tags in spec.template.spec.providerSpec.value.vSwitch
spec:
  template:
    spec:
      providerSpec:
        value:
          vSwitch:
            tags:
            - Key: kubernetes.io/cluster/<infrastructure_id> (1)
              Value: owned
            - Key: GISV (1)
              Value: ocp
            - Key: sigs.k8s.io/cloud-provider-alibaba/origin (1)
              Value: ocp
            - Key: Name
              Value: <infrastructure_id>-vswitch-<zone> (2)
            type: Tags
1 Optional: This tag is applied even when not specified in the compute machine set.
2 Required.

where:

  • <infrastructure_id> is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster.

  • <zone> is the zone within your region to place machines on.

Sample YAML for a compute machine set custom resource on AWS

This sample YAML defines a compute machine set that runs in the us-east-1a Amazon Web Services (AWS) zone and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
  name: <infrastructure_id>-infra-<zone> (2)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<zone> (2)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: infra (3)
        machine.openshift.io/cluster-api-machine-type: infra (3)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<zone> (2)
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: "" (3)
      providerSpec:
        value:
          ami:
            id: ami-046fe691f52a953f9 (4)
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
            - ebs:
                iops: 0
                volumeSize: 120
                volumeType: gp2
          credentialsSecret:
            name: aws-cloud-credentials
          deviceIndex: 0
          iamInstanceProfile:
            id: <infrastructure_id>-worker-profile (1)
          instanceType: m6i.large
          kind: AWSMachineProviderConfig
          placement:
            availabilityZone: <zone> (6)
            region: <region> (7)
          securityGroups:
            - filters:
                - name: tag:Name
                  values:
                    - <infrastructure_id>-worker-sg (1)
          subnet:
            filters:
              - name: tag:Name
                values:
                  - <infrastructure_id>-private-<zone> (8)
          tags:
            - name: kubernetes.io/cluster/<infrastructure_id> (1)
              value: owned
            - name: <custom_tag_name> (5)
              value: <custom_tag_value> (5)
          userDataSecret:
            name: worker-user-data
      taints: (9)
        - key: node-role.kubernetes.io/infra
          effect: NoSchedule
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the infrastructure ID, infra role node label, and zone.
3 Specify the infra role node label.
4 Specify a valid Fedora CoreOS (FCOS) Amazon Machine Image (AMI) for your AWS zone for your OKD nodes. If you want to use an AWS Marketplace image, you must complete the OKD subscription from the AWS Marketplace to obtain an AMI ID for your region. 
$ oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.ami.id}{"\n"}' \
    get machineset/<infrastructure_id>-<role>-<zone>
5 Optional: Specify custom tag data for your cluster. For example, you might add an admin contact email address by specifying a name:value pair of Email:admin-email@example.com.

Custom tags can also be specified during installation in the install-config.yml file. If the install-config.yml file and the machine set include a tag with the same name data, the value for the tag from the machine set takes priority over the value for the tag in the install-config.yml file.
6 Specify the zone, for example, us-east-1a.
7 Specify the region, for example, us-east-1.
8 Specify the infrastructure ID and zone.
9 Specify a taint to prevent user workloads from being scheduled on infra nodes.

Machine sets running on AWS support non-guaranteed Spot Instances. You can save on costs by using Spot Instances at a lower price compared to On-Demand Instances on AWS. Configure Spot Instances by adding spotMarketOptions to the MachineSet YAML file.

Sample YAML for a compute machine set custom resource on Azure

This sample YAML defines a compute machine set that runs in the 1 Microsoft Azure zone in a region and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-infra-<region> (3)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region> (3)
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region> (3)
    spec:
      metadata:
        creationTimestamp: null
        labels:
          machine.openshift.io/cluster-api-machineset: <machineset_name> (4)
          node-role.kubernetes.io/infra: "" (2)
      providerSpec:
        value:
          apiVersion: azureproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image: (5)
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/<infrastructure_id>-rg/providers/Microsoft.Compute/images/<infrastructure_id> (6)
            sku: ""
            version: ""
          internalLoadBalancer: ""
          kind: AzureMachineProviderSpec
          location: <region> (7)
          managedIdentity: <infrastructure_id>-identity (1)
          metadata:
            creationTimestamp: null
          natRule: null
          networkResourceGroup: ""
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: ""
          resourceGroup: <infrastructure_id>-rg (1)
          sshPrivateKey: ""
          sshPublicKey: ""
          tags:
            - name: <custom_tag_name> (9)
              value: <custom_tag_value> (9)
          subnet: <infrastructure_id>-<role>-subnet  (1) (2)
          userDataSecret:
            name: worker-user-data (2)
          vmSize: Standard_D4s_v3
          vnet: <infrastructure_id>-vnet (1)
          zone: "1" (8)
      taints: (10)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

You can obtain the subnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.subnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

You can obtain the vnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.vnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1
2 Specify the <infra> node label.
3 Specify the infrastructure ID, <infra> node label, and region.
4 Optional: Specify the compute machine set name to enable the use of availability sets. This setting only applies to new compute machines.
5 Specify the image details for your compute machine set. If you want to use an Azure Marketplace image, see "Selecting an Azure Marketplace image".
6 Specify an image that is compatible with your instance type. The Hyper-V generation V2 images created by the installation program have a -gen2 suffix, while V1 images have the same name without the suffix.
7 Specify the region to place machines on.
8 Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.
9 Optional: Specify custom tags in your machine set. Provide the tag name in <custom_tag_name> field and the corresponding tag value in <custom_tag_value> field.
10 Specify a taint to prevent user workloads from being scheduled on infra nodes.

Machine sets running on Azure support non-guaranteed Spot VMs. You can save on costs by using Spot VMs at a lower price compared to standard VMs on Azure. You can configure Spot VMs by adding spotVMOptions to the MachineSet YAML file.

Additional resources

Sample YAML for a compute machine set custom resource on Azure Stack Hub

This sample YAML defines a compute machine set that runs in the 1 Microsoft Azure zone in a region and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-infra-<region> (3)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region> (3)
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra-<region> (3)
    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: "" (2)
      taints: (4)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1beta1
          availabilitySet: <availability_set> (6)
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/<infrastructure_id>-rg/providers/Microsoft.Compute/images/<infrastructure_id> (1)
            sku: ""
            version: ""
          internalLoadBalancer: ""
          kind: AzureMachineProviderSpec
          location: <region> (5)
          managedIdentity: <infrastructure_id>-identity (1)
          metadata:
            creationTimestamp: null
          natRule: null
          networkResourceGroup: ""
          osDisk:
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          publicLoadBalancer: ""
          resourceGroup: <infrastructure_id>-rg (1)
          sshPrivateKey: ""
          sshPublicKey: ""
          subnet: <infrastructure_id>-<role>-subnet  (1) (2)
          userDataSecret:
            name: worker-user-data (2)
          vmSize: Standard_DS4_v2
          vnet: <infrastructure_id>-vnet (1)
          zone: "1" (7)
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

You can obtain the subnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.subnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1

You can obtain the vnet by running the following command:

$  oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.vnet}{"\n"}' \
    get machineset/<infrastructure_id>-worker-centralus1
2 Specify the <infra> node label.
3 Specify the infrastructure ID, <infra> node label, and region.
4 Specify a taint to prevent user workloads from being scheduled on infra nodes.
5 Specify the region to place machines on.
6 Specify the availability set for the cluster.
7 Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.

Machine sets running on Azure Stack Hub do not support non-guaranteed Spot VMs.

Sample YAML for a compute machine set custom resource on IBM Cloud

This sample YAML defines a compute machine set that runs in a specified IBM Cloud zone in a region and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-<infra>-<region> (3)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<region> (3)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<region> (3)
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: ibmcloud-credentials
          image: <infrastructure_id>-rhcos (4)
          kind: IBMCloudMachineProviderSpec
          primaryNetworkInterface:
              securityGroups:
              - <infrastructure_id>-sg-cluster-wide
              - <infrastructure_id>-sg-openshift-net
              subnet: <infrastructure_id>-subnet-compute-<zone> (5)
          profile: <instance_profile> (6)
          region: <region> (7)
          resourceGroup: <resource_group> (8)
          userDataSecret:
              name: <role>-user-data (2)
          vpc: <vpc_name> (9)
          zone: <zone> (10)
        taints: (11)
        - key: node-role.kubernetes.io/infra
          effect: NoSchedule
1 The infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 The <infra> node label.
3 The infrastructure ID, <infra> node label, and region.
4 The custom Fedora CoreOS (FCOS) image that was used for cluster installation.
5 The infrastructure ID and zone within your region to place machines on. Be sure that your region supports the zone that you specify.
6 Specify the IBM Cloud instance profile.
7 Specify the region to place machines on.
8 The resource group that machine resources are placed in. This is either an existing resource group specified at installation time, or an installer-created resource group named based on the infrastructure ID.
9 The VPC name.
10 Specify the zone within your region to place machines on. Be sure that your region supports the zone that you specify.
11 The taint to prevent user workloads from being scheduled on infra nodes.

Sample YAML for a compute machine set custom resource on GCP

This sample YAML defines a compute machine set that runs in Google Cloud Platform (GCP) and creates nodes that are labeled with node-role.kubernetes.io/infra: "", where infra is the node label to add.

Values obtained by using the OpenShift CLI

In the following example, you can obtain some of the values for your cluster by using the OpenShift CLI.

Infrastructure ID

The <infrastructure_id> string is the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
Image path

The <path_to_image> string is the path to the image that was used to create the disk. If you have the OpenShift CLI installed, you can obtain the path to the image by running the following command:

$ oc -n openshift-machine-api \
  -o jsonpath='{.spec.template.spec.providerSpec.value.disks[0].image}{"\n"}' \
  get machineset/<infrastructure_id>-worker-a
Sample GCP MachineSet values
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
  name: <infrastructure_id>-w-a
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-w-a
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: gcpprovider.openshift.io/v1beta1
          canIPForward: false
          credentialsSecret:
            name: gcp-cloud-credentials
          deletionProtection: false
          disks:
          - autoDelete: true
            boot: true
            image: <path_to_image> (3)
            labels: null
            sizeGb: 128
            type: pd-ssd
          gcpMetadata: (4)
          - key: <custom_metadata_key>
            value: <custom_metadata_value>
          kind: GCPMachineProviderSpec
          machineType: n1-standard-4
          metadata:
            creationTimestamp: null
          networkInterfaces:
          - network: <infrastructure_id>-network
            subnetwork: <infrastructure_id>-worker-subnet
          projectID: <project_name> (5)
          region: us-central1
          serviceAccounts:
          - email: <infrastructure_id>-w@<project_name>.iam.gserviceaccount.com
            scopes:
            - https://www.googleapis.com/auth/cloud-platform
          tags:
            - <infrastructure_id>-worker
          userDataSecret:
            name: worker-user-data
          zone: us-central1-a
      taints: (6)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
1 For <infrastructure_id>, specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster.
2 For <infra>, specify the <infra> node label.
3 Specify the path to the image that is used in current compute machine sets.

To use a GCP Marketplace image, specify the offer to use:

  • OKD: https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145

  • OpenShift Platform Plus: https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-opp-48-x86-64-202206140145

  • OpenShift Kubernetes Engine: https://www.googleapis.com/compute/v1/projects/redhat-marketplace-public/global/images/redhat-coreos-oke-48-x86-64-202206140145
4 Optional: Specify custom metadata in the form of a key:value pair. For example use cases, see the GCP documentation for setting custom metadata.
5 For <project_name>, specify the name of the GCP project that you use for your cluster.
6 Specify a taint to prevent user workloads from being scheduled on infra nodes.

Machine sets running on GCP support non-guaranteed preemptible VM instances. You can save on costs by using preemptible VM instances at a lower price compared to normal instances on GCP. You can configure preemptible VM instances by adding preemptible to the MachineSet YAML file.

Sample YAML for a compute machine set custom resource on Nutanix

This sample YAML defines a Nutanix compute machine set that creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-<infra>-<zone> (3)
  namespace: openshift-machine-api
  annotations: (4)
    machine.openshift.io/memoryMb: "16384"
    machine.openshift.io/vCPU: "4"
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<zone> (3)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<infra>-<zone> (3)
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1
          cluster:
            type: uuid
            uuid: <cluster_uuid>
          credentialsSecret:
            name: nutanix-creds-secret
          image:
            name: <infrastructure_id>-rhcos (5)
            type: name
          kind: NutanixMachineProviderConfig
          memorySize: 16Gi (6)
          subnets:
          - type: uuid
            uuid: <subnet_uuid>
          systemDiskSize: 120Gi (7)
          userDataSecret:
            name: <user_data_secret> (8)
          vcpuSockets: 4 (9)
          vcpusPerSocket: 1 (10)
      taints: (11)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the <infra> node label.
3 Specify the infrastructure ID, <infra> node label, and zone.
4 Annotations for the cluster autoscaler.
5 Specify the image to use. Use an image from an existing default compute machine set for the cluster.
6 Specify the amount of memory for the cluster in Gi.
7 Specify the size of the system disk in Gi.
8 Specify the name of the secret in the user data YAML file that is in the openshift-machine-api namespace. Use the value that the installer populates in the default compute machine set.
9 Specify the number of vCPU sockets.
10 Specify the number of vCPUs per socket.
11 Specify a taint to prevent user workloads from being scheduled on infra nodes.

Sample YAML for a compute machine set custom resource on OpenStack

This sample YAML defines a compute machine set that runs on OpenStack and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <infra> (2)
    machine.openshift.io/cluster-api-machine-type: <infra> (2)
  name: <infrastructure_id>-infra (3)
  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas>
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra (3)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (2)
        machine.openshift.io/cluster-api-machine-type: <infra> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra (3)
    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: ""
      taints: (4)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
      providerSpec:
        value:
          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
          cloudName: openstack
          cloudsSecret:
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: <nova_flavor>
          image: <glance_image_name_or_location>
          serverGroupID: <optional_UUID_of_server_group> (5)
          kind: OpenstackProviderSpec
          networks: (6)
          - filter: {}
            subnets:
            - filter:
                name: <subnet_name>
                tags: openshiftClusterID=<infrastructure_id> (1)
          primarySubnet: <rhosp_subnet_UUID> (7)
          securityGroups:
          - filter: {}
            name: <infrastructure_id>-worker (1)
          serverMetadata:
            Name: <infrastructure_id>-worker (1)
            openshiftClusterID: <infrastructure_id> (1)
          tags:
          - openshiftClusterID=<infrastructure_id> (1)
          trunk: true
          userDataSecret:
            name: worker-user-data (2)
          availabilityZone: <optional_openstack_availability_zone>
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the <infra> node label.
3 Specify the infrastructure ID and <infra> node label.
4 Specify a taint to prevent user workloads from being scheduled on infra nodes.
5 To set a server group policy for the MachineSet, enter the value that is returned from creating a server group. For most deployments, anti-affinity or soft-anti-affinity policies are recommended.
6 Required for deployments to multiple networks. If deploying to multiple networks, this list must include the network that is used as the primarySubnet value.
7 Specify the OpenStack subnet that you want the endpoints of nodes to be published on. Usually, this is the same subnet that is used as the value of machinesSubnet in the install-config.yaml file.

Sample YAML for a compute machine set custom resource on oVirt

This sample YAML defines a compute machine set that runs on oVirt and creates nodes that are labeled with node-role.kubernetes.io/<node_role>: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <role> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
    machine.openshift.io/cluster-api-machine-role: <role> (2)
    machine.openshift.io/cluster-api-machine-type: <role> (2)
  name: <infrastructure_id>-<role> (3)
  namespace: openshift-machine-api
spec:
  replicas: <number_of_replicas> (4)
  Selector: (5)
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role> (3)
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <role> (2)
        machine.openshift.io/cluster-api-machine-type: <role> (2)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role> (3)
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/<role>: "" (2)
      providerSpec:
        value:
          apiVersion: ovirtproviderconfig.machine.openshift.io/v1beta1
          cluster_id: <ovirt_cluster_id> (6)
          template_name: <ovirt_template_name> (7)
          sparse: <boolean_value> (8)
          format: <raw_or_cow> (9)
          cpu: (10)
            sockets: <number_of_sockets> (11)
            cores: <number_of_cores> (12)
            threads: <number_of_threads> (13)
          memory_mb: <memory_size> (14)
          guaranteed_memory_mb:  <memory_size> (15)
          os_disk: (16)
            size_gb: <disk_size> (17)
            storage_domain_id: <storage_domain_UUID> (18)
          network_interfaces: (19)
            vnic_profile_id:  <vnic_profile_id> (20)
          credentialsSecret:
            name: ovirt-credentials (21)
          kind: OvirtMachineProviderSpec
          type: <workload_type> (22)
          auto_pinning_policy: <auto_pinning_policy> (23)
          hugepages: <hugepages> (24)
          affinityGroupsNames:
            - compute (25)
          userDataSecret:
            name: worker-user-data
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the node label to add.
3 Specify the infrastructure ID and node label. These two strings together cannot be longer than 35 characters.
4 Specify the number of machines to create.
5 Selector for the machines.
6 Specify the UUID for the oVirt cluster to which this VM instance belongs.
7 Specify the oVirt VM template to use to create the machine.
8 Setting this option to false enables preallocation of disks. The default is true. Setting sparse to true with format set to raw is not available for block storage domains. The raw format writes the entire virtual disk to the underlying physical disk.
9 Can be set to cow or raw. The default is cow. The cow format is optimized for virtual machines.

Preallocating disks on file storage domains writes zeroes to the file. This might not actually preallocate disks depending on the underlying storage.
10 Optional: The CPU field contains the CPU configuration, including sockets, cores, and threads.
11 Optional: Specify the number of sockets for a VM.
12 Optional: Specify the number of cores per socket.
13 Optional: Specify the number of threads per core.
14 Optional: Specify the size of a VM’s memory in MiB.
15 Optional: Specify the size of a virtual machine’s guaranteed memory in MiB. This is the amount of memory that is guaranteed not to be drained by the ballooning mechanism. For more information, see Memory Ballooning and Optimization Settings Explained.

If you are using a version earlier than oVirt 4.4.8, see Guaranteed memory requirements for OpenShift on Red Hat Virtualization clusters.
16 Optional: Root disk of the node.
17 Optional: Specify the size of the bootable disk in GiB.
18 Optional: Specify the UUID of the storage domain for the compute node’s disks. If none is provided, the compute node is created on the same storage domain as the control nodes. (default)
19 Optional: List of the network interfaces of the VM. If you include this parameter, OKD discards all network interfaces from the template and creates new ones.
20 Optional: Specify the vNIC profile ID.
21 Specify the name of the secret object that holds the oVirt credentials.
22 Optional: Specify the workload type for which the instance is optimized. This value affects the oVirt VM parameter. Supported values: desktop, server (default), high_performance. high_performance improves performance on the VM. Limitations exist, for example, you cannot access the VM with a graphical console. For more information, see Configuring High Performance Virtual Machines, Templates, and Pools in the Virtual Machine Management Guide.
23 Optional: AutoPinningPolicy defines the policy that automatically sets CPU and NUMA settings, including pinning to the host for this instance. Supported values: none, resize_and_pin. For more information, see Setting NUMA Nodes in the Virtual Machine Management Guide.
24 Optional: Hugepages is the size in KiB for defining hugepages in a VM. Supported values: 2048 or 1048576. For more information, see Configuring Huge Pages in the Virtual Machine Management Guide.
25 Optional: A list of affinity group names to be applied to the VMs. The affinity groups must exist in oVirt.

Because oVirt uses a template when creating a VM, if you do not specify a value for an optional parameter, oVirt uses the value for that parameter that is specified in the template.

Sample YAML for a compute machine set custom resource on vSphere

This sample YAML defines a compute machine set that runs on VMware vSphere and creates nodes that are labeled with node-role.kubernetes.io/infra: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <infra> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
  name: <infrastructure_id>-infra (2)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra (2)
  template:
    metadata:
      creationTimestamp: null
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: <infra> (3)
        machine.openshift.io/cluster-api-machine-type: <infra> (3)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra (2)
    spec:
      metadata:
        creationTimestamp: null
        labels:
          node-role.kubernetes.io/infra: "" (3)
      taints: (4)
      - key: node-role.kubernetes.io/infra
        effect: NoSchedule
      providerSpec:
        value:
          apiVersion: vsphereprovider.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 120
          kind: VSphereMachineProviderSpec
          memoryMiB: 8192
          metadata:
            creationTimestamp: null
          network:
            devices:
            - networkName: "<vm_network_name>" (5)
          numCPUs: 4
          numCoresPerSocket: 1
          snapshot: ""
          template: <vm_template_name> (6)
          userDataSecret:
            name: worker-user-data
          workspace:
            datacenter: <vcenter_datacenter_name> (7)
            datastore: <vcenter_datastore_name> (8)
            folder: <vcenter_vm_folder_path> (9)
            resourcepool: <vsphere_resource_pool> (10)
            server: <vcenter_server_ip> (11)
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command: 
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the infrastructure ID and <infra>