1. Documentation
×

Creating a compute machine set on vSphere

You can define and create a OKD compute machine set on VMware vSphere to enable the Machine API to automatically scale and manage compute nodes in vSphere. You can create a different compute machine set to serve a specific purpose in your OKD cluster on vSphere. For example, you might create infrastructure machine sets and related machines so that you can move supporting workloads to the new machines.

You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational. Clusters with user-provisioned infrastructure require additional validation and configuration to use the Machine API.

Clusters with the infrastructure platform type none cannot use the Machine API. This limitation applies even if the compute machines that are attached to the cluster are installed on a platform that supports the feature. This parameter cannot be changed after installation.

To view the platform type for your cluster, run the following command:

$ oc get infrastructure cluster -o jsonpath='{.status.platform}'

Sample YAML for a compute machine set custom resource on vSphere

To enable the Machine API to automate node provisioning on VMware vSphere infrastructure, define a MachineSet resource with parameters that are specific to VMware vSphere, for example data center, resource pool, and template.

The sample YAML file defines a compute machine set that runs on VMware vSphere and creates nodes that are labeled with node-role.kubernetes.io/<role>: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <role> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
 creationTimestamp: null
 labels:
   machine.openshift.io/cluster-api-cluster: <infrastructure_id>
 name: <infrastructure_id>-<role>
 namespace: openshift-machine-api
spec:
 replicas: 1
 selector:
   matchLabels:
     machine.openshift.io/cluster-api-cluster: <infrastructure_id>
     machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
 template:
   metadata:
     creationTimestamp: null
     labels:
       machine.openshift.io/cluster-api-cluster: <infrastructure_id>
       machine.openshift.io/cluster-api-machine-role: <role>
       machine.openshift.io/cluster-api-machine-type: <role>
       machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
   spec:
     metadata:
       creationTimestamp: null
       labels:
         node-role.kubernetes.io/<role>: ""
     providerSpec:
       value:
         apiVersion: machine.openshift.io/v1beta1
         credentialsSecret:
           name: vsphere-cloud-credentials
         dataDisks:
         - name: "<disk_name>"
           provisioningMode: "<mode>"
           sizeGiB: 20
         diskGiB: 120
         kind: VSphereMachineProviderSpec
         memoryMiB: 8192
         metadata:
           creationTimestamp: null
         network:
           devices:
           - networkName: "<vm_network_name>"
         numCPUs: 4
         numCoresPerSocket: 1
         snapshot: ""
         template: <vm_template_name>
         userDataSecret:
           name: worker-user-data
         workspace:
           datacenter: <vcenter_data_center_name>
           datastore: <vcenter_datastore_name>
           folder: <vcenter_vm_folder_path>
           resourcepool: <vsphere_resource_pool>
           server: <vcenter_server_ip>

where

<infrastructure_id>

Specifies the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI (oc) installed, you can obtain the infrastructure ID by running the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
<infrastructure_id>-<role>

Specifies the infrastructure ID and node label.

<role>

Specifies the node label to add.

<disk_name>

Specifies one or more data disk definitions. For more information, see "Configuring data disks by using machine sets".

<vm_network_name>

Specifies the vSphere VM network to deploy the compute machine set to. This VM network must be where other compute machines reside in the cluster.

<vm_template_name>

Specifies the vSphere VM template to use, such as user-5ddjd-rhcos.

<vcenter_data_center_name>

Specifies the vCenter datacenter to deploy the compute machine set on.

<vcenter_datastore_name>

Specifies the vCenter datastore to deploy the compute machine set on.

<vcenter_vm_folder_path>

Specifies the path to the vSphere VM folder in vCenter, such as /dc1/vm/user-inst-5ddjd.

<vsphere_resource_pool>

Specifies the vSphere resource pool for your VMs.

<vcenter_server_ip>

Specifies the vCenter server IP or fully qualified domain name.

Minimum required vCenter privileges for compute machine set management

To manage compute machine sets in an OKD cluster on vCenter, you must use an account with privileges to read, create, and delete the required resources. Using an account that has global administrative privileges is the simplest way to access all of the necessary permissions.

If you cannot use an account with global administrative privileges, you must create roles to grant the minimum required privileges. The following table lists the minimum vCenter roles and privileges that are required to create, scale, and delete compute machine sets and to delete machines in your OKD cluster.

Table 1. Minimum vCenter roles and privileges required for compute machine set management
vSphere object for role When required Required privileges

vSphere vCenter

Always

InventoryService.Tagging.AttachTag InventoryService.Tagging.CreateCategory InventoryService.Tagging.CreateTag InventoryService.Tagging.DeleteCategory InventoryService.Tagging.DeleteTag InventoryService.Tagging.EditCategory InventoryService.Tagging.EditTag Sessions.ValidateSession StorageProfile.Update1 StorageProfile.View1

vSphere vCenter Cluster

Always

Resource.AssignVMToPool

vSphere datastore

Always

Datastore.AllocateSpace Datastore.Browse

vSphere Port Group

Always

Network.Assign

Virtual Machine Folder

Always

VirtualMachine.Config.AddRemoveDevice VirtualMachine.Config.AdvancedConfig VirtualMachine.Config.Annotation VirtualMachine.Config.CPUCount VirtualMachine.Config.DiskExtend VirtualMachine.Config.Memory VirtualMachine.Config.Settings VirtualMachine.Interact.PowerOff VirtualMachine.Interact.PowerOn VirtualMachine.Inventory.CreateFromExisting VirtualMachine.Inventory.Delete VirtualMachine.Provisioning.Clone

vSphere vCenter data center

If the installation program creates the virtual machine folder.

Resource.AssignVMToPool VirtualMachine.Provisioning.DeployTemplate

1 The StorageProfile.Update and StorageProfile.View permissions are required only for storage backends that use the Container Storage Interface (CSI).

The following table details the permissions and propagation settings that are required for compute machine set management.

Table 2. Required permissions and propagation settings
vSphere object Folder type Propagate to children Permissions required

vSphere vCenter

Always

Not required

Listed required privileges

vSphere vCenter data center

Existing folder

Not required

ReadOnly permission

Installation program creates the folder

Required

Listed required privileges

vSphere vCenter Cluster

Always

Required

Listed required privileges

vSphere vCenter datastore

Always

Not required

Listed required privileges

vSphere Switch

Always

Not required

ReadOnly permission

vSphere Port Group

Always

Not required

Listed required privileges

vSphere vCenter Virtual Machine Folder

Existing folder

Required

Listed required privileges

For more information about creating an account with only the required privileges, see vSphere Permissions and User Management Tasks in the vSphere documentation.

Requirements for clusters with user-provisioned infrastructure to use compute machine sets

To enable the Machine API to manage and scale compute nodes on user-provisioned infrastructure, you can configure a MachineSet YAML file with specific vSphere parameters, for example data center and disk image. To use compute machine sets on clusters that have user-provisioned infrastructure, you must ensure that you cluster configuration supports using the Machine API.

Obtaining the infrastructure ID

To ensure the Machine API correctly identifies and manages virtual machines (VMs) that belong to a specific cluster, you must add the unique infrastructure ID to the MachineSet YAML file to label and link resources. To create compute machine sets, you must be able to supply the infrastructure ID for your cluster.

Procedure

  • To obtain the infrastructure ID for your cluster, run the following command:

    $ oc get infrastructure cluster -o jsonpath='{.status.infrastructureName}'

Satisfying vSphere credentials requirements

To use compute machine sets and manage virtual machine (VM) resources, the Machine API must be able to interact with vCenter. Credentials that authorize the Machine API components to interact with vCenter must exist in a secret in the openshift-machine-api namespace.

Procedure

  1. To determine whether the required credentials exist, run the following command:

    $ oc get secret \
  -n openshift-machine-api vsphere-cloud-credentials \
  -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
    Sample output
    <vcenter-server>.password=<openshift-user-password>
<vcenter-server>.username=<openshift-user>

    where

    <vcenter_server>

    Specifies the IP address or fully qualified domain name (FQDN) of the vCenter server and <openshift_user_password> and <openshift_user> are the OKD administrator credentials to use.

  2. If the secret does not exist, create it by running the following command:

    $ oc create secret generic vsphere-cloud-credentials \
  -n openshift-machine-api \
  --from-literal=<vcenter-server>.username=<openshift-user> --from-literal=<vcenter-server>.password=<openshift-user-password>

Satisfying Ignition configuration requirements

For the Machine API to provision virtual machines (VMs) with the correct initial configuration using Ignition, a valid Ignition configuration is required. The Ignition configuration contains the machine-config-server address and a system trust bundle for obtaining further Ignition configurations from the Machine Config Operator.

By default, this configuration is stored in the worker-user-data secret in the machine-api-operator namespace. Compute machine sets reference the secret during the machine creation process.

Procedure

  1. To determine whether the required secret exists, run the following command:

    $ oc get secret \
  -n openshift-machine-api worker-user-data \
  -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'
    Sample output
    disableTemplating: false
userData:
  {
    "ignition": {
      ...
      },
    ...
  }

    The full output is omitted here, but this is the format to use.

  2. If the secret does not exist, create it by running the following command:

    $ oc create secret generic worker-user-data \
  -n openshift-machine-api \
  --from-file=<installation_directory>/worker.ign

    Specifies the directory that was used to store your installation assets during cluster installation.

Additional resources

Creating a compute machine set

In addition to the compute machine sets created by the installation program, you can create your own compute machine sets to dynamically manage the machine compute resources for specific workloads of your choice. Use the OKD CLI to automate node provisioning.

Clusters that are installed with user-provisioned infrastructure have a different networking stack than clusters with infrastructure that is provisioned by the installation program. As a result of this difference, automatic load balancer management is unsupported on clusters that have user-provisioned infrastructure. For these clusters, a compute machine set can only create worker and infra type machines.
Prerequisites

  • Deploy an OKD cluster.

  • Install the OpenShift CLI (oc).

  • Log in to oc as a user with cluster-admin permission.

  • Have the necessary permissions to deploy VMs in your vCenter instance and have the required access to the datastore specified.

  • If your cluster uses user-provisioned infrastructure, you have satisfied the specific Machine API requirements for that configuration.

Procedure

  1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

    Ensure that you set the <clusterID> and <role> parameter values.

  2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

    1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api
      Example output
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

    2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
  -n openshift-machine-api -o yaml
      Example output
      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
  name: <infrastructure_id>-<role>
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: <role>
        machine.openshift.io/cluster-api-machine-type: <role>
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
    spec:
      providerSpec:
        ...

      where:

      metadata.labels.machine.openshift.io/cluster-api-cluster

      Specifies the cluster infrastructure ID.

      metadata.labels.name

      Specifies a default node label.

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.
      spec.template.metadata.spec.providerSpec

      Specifies the values of the compute machine set CR. The values are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.

    3. If you are creating a compute machine set for a cluster that has user-provisioned infrastructure, note the following important values:

      Example vSphere providerSpec values
      apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
...
template:
  ...
  spec:
    providerSpec:
      value:
        apiVersion: machine.openshift.io/v1beta1
        credentialsSecret:
          name: vsphere-cloud-credentials
        dataDisks:
        - name: <disk_name>
          provisioningMode: <mode>
          sizeGiB: 10
        diskGiB: 120
        kind: VSphereMachineProviderSpec
        memoryMiB: 16384
        network:
          devices:
            - networkName: "<vm_network_name>"
        numCPUs: 4
        numCoresPerSocket: 4
        snapshot: ""
        template: <vm_template_name>
        userDataSecret:
          name: worker-user-data
        workspace:
          datacenter: <vcenter_data_center_name>
          datastore: <vcenter_datastore_name>
          folder: <vcenter_vm_folder_path>
          resourcepool: <vsphere_resource_pool>
          server: <vcenter_server_address>

where:

vsphere-cloud-credentials

Specifies the name of the secret in the openshift-machine-api namespace that contains the required vCenter credentials.

<disk_name>

Specifies the collection of data disk definitions. For more information, see "Configuring data disks by using machine sets".

<vm_template_name>

Specifies the name of the FCOS VM template for your cluster that was created during installation.

worker-user-data

Specifies the name of the secret in the openshift-machine-api namespace that contains the required Ignition configuration credentials.

<vcenter_server_address>

Specifies the IP address or fully qualified domain name (FQDN) of the vCenter server.

  1. Create a MachineSet CR by running the following command:

    $ oc create -f <file_name>.yaml
Verification

  • View the list of compute machine sets by running the following command:

    $ oc get machineset -n openshift-machine-api
    Example output
    NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-infra-us-east-1a    1         1         1       1           11m
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

    When the new compute machine set is available, the DESIRED and CURRENT values match. If the compute machine set is not available, wait a few minutes and run the command again.

Labeling GPU machine sets for the cluster autoscaler

Label your machine sets to indicate which machines the cluster autoscaler can use for GPU-enabled nodes. Applying the accelerator label helps ensure that the autoscaler deploys the correct resources for your GPU workloads.

Prerequisites

  • Your cluster uses a cluster autoscaler.

Procedure

  • On the machine set that you want to create machines for the cluster autoscaler to use to deploy GPU-enabled nodes, add a cluster-api/accelerator label:

    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: machine-set-name
spec:
  template:
    spec:
      metadata:
        labels:
          cluster-api/accelerator: <accelerator_name>

    where:

    <accelerator_name>

    Specifies a label of your choice that consists of alphanumeric characters, -, _, or . and starts and ends with an alphanumeric character. For example, you might use nvidia-t4 to represent Nvidia T4 GPUs, or nvidia-a10g for A10G GPUs.

    You must specify the value of this label for the spec.resourceLimits.gpus.type parameter in your ClusterAutoscaler CR. For more information, see "Cluster autoscaler resource definition".
Additional resources

Adding tags to machines by using machine sets

To ensure that your cluster remains scalable and resilient, you can use a MachineSet object and machine health checks to automate the provisioning and repair of nodes. OKD adds a cluster-specific tag to each virtual machine (VM) that it creates. The installation program uses these tags to select the VMs to delete when uninstalling a cluster.

In addition to the cluster-specific tags assigned to VMs, you can configure a machine set to add up to 10 additional vSphere tags to the VMs it provisions.

Prerequisites

  • You have access to an OKD cluster installed on vSphere using an account with cluster-admin permissions.

  • You have access to the VMware vCenter console associated with your cluster.

  • You have created a tag in the vCenter console.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Use the vCenter console to find the tag ID for any tag that you want to add to your machines:

    1. Log in to the vCenter console.

    2. From the Home menu, click Tags & Custom Attributes.

    3. Select a tag that you want to add to your machines.

    4. Use the browser URL for the tag that you select to identify the tag ID.

      Example tag URL
      https://vcenter.example.com/ui/app/tags/tag/urn:vmomi:InventoryServiceTag:208e713c-cae3-4b7f-918e-4051ca7d1f97:GLOBAL/permissions
      Example tag ID
      urn:vmomi:InventoryServiceTag:208e713c-cae3-4b7f-918e-4051ca7d1f97:GLOBAL

  2. In a text editor, open the YAML file for an existing machine set or create a new one.

  3. Edit the following lines under the providerSpec field:

    apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
# ...
spec:
  template:
    spec:
      providerSpec:
        value:
          tagIDs:
          - <tag_id_value>
# ...

    where

    spec.template.spec.providerSpec.value.tagIDs

    Specifies a list of up to 10 tags to add to the machines that this machine set provisions. Replace <tag_id_value> with the tag that you want to add to your machines. For example, urn:vmomi:InventoryServiceTag:208e713c-cae3-4b7f-918e-4051ca7d1f97:GLOBAL.