×

About scaling a user-provisioned cluster with the Bare Metal Operator

You can scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO) and other metal3 components. User-provisioned infrastructure installations do not feature the Machine API Operator. The Machine API Operator typically manages the lifecycle of bare-metal hosts in a cluster. However, it is possible to use the BMO and other metal3 components to scale nodes in user-provisioned clusters without requiring the Machine API Operator.

Prerequisites for scaling a user-provisioned cluster

  • You installed a user-provisioned infrastructure cluster on bare metal.

  • You have baseboard management controller (BMC) access to the hosts.

Limitations for scaling a user-provisioned cluster

  • You cannot use a provisioning network to scale user-provisioned infrastructure clusters by using the Bare Metal Operator (BMO).

    • Consequentially, you can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.

  • You cannot scale MachineSet objects in user-provisioned infrastructure clusters by using the BMO.

Configuring a provisioning resource to scale user-provisioned clusters

Create a Provisioning custom resource (CR) to enable Metal platform components on a user-provisioned infrastructure cluster.

Prerequisites
  • You installed a user-provisioned infrastructure cluster on bare metal.

Procedure
  1. Create a Provisioning CR.

    1. Save the following YAML in the provisioning.yaml file:

      apiVersion: metal3.io/v1alpha1
      kind: Provisioning
      metadata:
        name: provisioning-configuration
      spec:
        provisioningNetwork: "Disabled"
        watchAllNamespaces: false

      OKD 4.17 does not support enabling a provisioning network when you scale a user-provisioned cluster by using the Bare Metal Operator.

  2. Create the Provisioning CR by running the following command:

    $ oc create -f provisioning.yaml
    Example output
    provisioning.metal3.io/provisioning-configuration created
Verification
  • Verify that the provisioning service is running by running the following command:

    $ oc get pods -n openshift-machine-api
    Example output
    NAME                                                  READY   STATUS    RESTARTS        AGE
    cluster-autoscaler-operator-678c476f4c-jjdn5          2/2     Running   0               5d21h
    cluster-baremetal-operator-6866f7b976-gmvgh           2/2     Running   0               5d21h
    control-plane-machine-set-operator-7d8566696c-bh4jz   1/1     Running   0               5d21h
    ironic-proxy-64bdw                                    1/1     Running   0               5d21h
    ironic-proxy-rbggf                                    1/1     Running   0               5d21h
    ironic-proxy-vj54c                                    1/1     Running   0               5d21h
    machine-api-controllers-544d6849d5-tgj9l              7/7     Running   1 (5d21h ago)   5d21h
    machine-api-operator-5c4ff4b86d-6fjmq                 2/2     Running   0               5d21h
    metal3-6d98f84cc8-zn2mx                               5/5     Running   0               5d21h
    metal3-image-customization-59d745768d-bhrp7           1/1     Running   0               5d21h

Provisioning new hosts in a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to provision bare-metal hosts in a user-provisioned cluster by creating a BareMetalHost custom resource (CR).

To provision bare-metal hosts to the cluster by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false.

Prerequisites
  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure
  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the bmh.yaml file:

      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: worker1-bmc
        namespace: openshift-machine-api
      type: Opaque
      data:
        username: <base64_of_uid>
        password: <base64_of_pwd>
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: worker1
        namespace: openshift-machine-api
      spec:
        bmc:
          address: <protocol>://<bmc_url> (1)
          credentialsName: "worker1-bmc"
        bootMACAddress: <nic1_mac_address>
        externallyProvisioned: false (2)
        customDeploy:
          method: install_coreos
        online: true
        userData:
          name: worker-user-data-managed
          namespace: openshift-machine-api
      1 You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2 You must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to false. The default value is false.
  2. Create the bare-metal host object by running the following command:

    $ oc create -f bmh.yaml
    Example output
    secret/worker1-bmc created
    baremetalhost.metal3.io/worker1 created
  3. Approve all certificate signing requests (CSRs).

    1. Verify that the provisioning state of the host is provisioned by running the following command:

      $ oc get bmh -A
      Example output
      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
      openshift-machine-api   controller1   externally provisioned              true             5m25s
      openshift-machine-api   worker1       provisioned                         true             4m45s
    2. Get the list of pending CSRs by running the following command:

      $ oc get csr
      Example output
      NAME        AGE   SIGNERNAME                                    REQUESTOR                                         REQUESTEDDURATION CONDITION
      csr-gfm9f   33s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-o
      perator:node-bootstrapper   <none>              Pending
    3. Approve the CSR by running the following command:

      $ oc adm certificate approve <csr_name>
      Example output
      certificatesigningrequest.certificates.k8s.io/<csr_name> approved
Verification
  • Verify that the node is ready by running the following command:

    $ oc get nodes
    Example output
    NAME        STATUS   ROLES           AGE     VERSION
    app1        Ready    worker          47s     v1.24.0+dc5a2fd
    controller1 Ready    master,worker   2d22h   v1.24.0+dc5a2fd

Optional: Managing existing hosts in a user-provisioned cluster by using the BMO

Optionally, you can use the Bare Metal Operator (BMO) to manage existing bare-metal controller hosts in a user-provisioned cluster by creating a BareMetalHost object for the existing host. It is not a requirement to manage existing user-provisioned hosts; however, you can enroll them as externally-provisioned hosts for inventory purposes.

To manage existing hosts by using the BMO, you must set the spec.externallyProvisioned specification in the BareMetalHost custom resource to true to prevent the BMO from re-provisioning the host.

Prerequisites
  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure
  1. Create the Secret CR and the BareMetalHost CR.

    1. Save the following YAML in the controller.yaml file:

      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: controller1-bmc
        namespace: openshift-machine-api
      type: Opaque
      data:
        username: <base64_of_uid>
        password: <base64_of_pwd>
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: controller1
        namespace: openshift-machine-api
      spec:
        bmc:
          address: <protocol>://<bmc_url> (1)
          credentialsName: "controller1-bmc"
        bootMACAddress: <nic1_mac_address>
        customDeploy:
          method: install_coreos
        externallyProvisioned: true (2)
        online: true
        userData:
          name: controller-user-data-managed
          namespace: openshift-machine-api
      1 You can only use bare-metal host drivers that support virtual media networking booting, for example redfish-virtualmedia and idrac-virtualmedia.
      2 You must set the value to true to prevent the BMO from re-provisioning the bare-metal controller host.
  2. Create the bare-metal host object by running the following command:

    $ oc create -f controller.yaml
    Example output
    secret/controller1-bmc created
    baremetalhost.metal3.io/controller1 created
Verification
  • Verify that the BMO created the bare-metal host object by running the following command:

    $ oc get bmh -A
    Example output
    NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
    openshift-machine-api   controller1   externally provisioned              true             13s

Removing hosts from a user-provisioned cluster by using the BMO

You can use the Bare Metal Operator (BMO) to remove bare-metal hosts from a user-provisioned cluster.

Prerequisites
  • You created a user-provisioned bare-metal cluster.

  • You have baseboard management controller (BMC) access to the hosts.

  • You deployed a provisioning service in the cluster by creating a Provisioning CR.

Procedure
  1. Cordon and drain the host by running the following command:

    $ oc adm drain app1 --force --ignore-daemonsets=true
    Example output
    node/app1 cordoned
    WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-tvthg, openshift-dns/dns-
    default-9q6rz, openshift-dns/node-resolver-zvt42, openshift-image-registry/node-ca-mzxth, openshift-ingress-cana
    ry/ingress-canary-qq5lf, openshift-machine-config-operator/machine-config-daemon-v79dm, openshift-monitoring/nod
    e-exporter-2vn59, openshift-multus/multus-additional-cni-plugins-wssvj, openshift-multus/multus-fn8tg, openshift
    -multus/network-metrics-daemon-5qv55, openshift-network-diagnostics/network-check-target-jqxn2, openshift-ovn-ku
    bernetes/ovnkube-node-rsvqg
    evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766965-258vp
    evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766950-kg5mk
    evicting pod openshift-operator-lifecycle-manager/collect-profiles-27766935-stf4s
    pod/collect-profiles-27766965-258vp evicted
    pod/collect-profiles-27766950-kg5mk evicted
    pod/collect-profiles-27766935-stf4s evicted
    node/app1 drained
  2. Delete the customDeploy specification from the BareMetalHost CR.

    1. Edit the BareMetalHost CR for the host by running the following command:

      $ oc edit bmh -n openshift-machine-api <host_name>
    2. Delete the lines spec.customDeploy and spec.customDeploy.method:

      ...
        customDeploy:
          method: install_coreos
    3. Verify that the provisioning state of the host changes to deprovisioning by running the following command:

      $ oc get bmh -A
      Example output
      NAMESPACE               NAME          STATE                    CONSUMER   ONLINE   ERROR   AGE
      openshift-machine-api   controller1   externally provisioned              true             58m
      openshift-machine-api   worker1       deprovisioning                      true             57m
  3. Delete the node by running the following command:

    $ oc delete node <node_name>
Verification
  • Verify the node is deleted by running the following command:

    $ oc get nodes
    Example output
    NAME          STATUS   ROLES           AGE     VERSION
    controller1   Ready    master,worker   2d23h   v1.24.0+dc5a2fd