×

After deploying an installer-provisioned OKD cluster, you can use the following procedures to expand the number of worker nodes. Ensure that each prospective worker node meets the prerequisites.

Expanding the cluster using RedFish Virtual Media involves meeting minimum firmware requirements. See Firmware requirements for installing with virtual media in the Prerequisites section for additional details when expanding the cluster using RedFish Virtual Media.

Preparing the bare metal node

Expanding the cluster requires a DHCP server. Each node must have a DHCP reservation.

Reserving IP addresses so they become static IP addresses

Some administrators prefer to use static IP addresses so that each node’s IP address remains constant in the absence of a DHCP server. To use static IP addresses in the OKD cluster, reserve the IP addresses in the DHCP server with an infinite lease. After the installer provisions the node successfully, the dispatcher script will check the node’s network configuration. If the dispatcher script finds that the network configuration contains a DHCP infinite lease, it will recreate the connection as a static IP connection using the IP address from the DHCP infinite lease. NICs without DHCP infinite leases will remain unmodified.

Setting IP addresses with an infinite lease is incompatible with network configuration deployed by using the Machine Config Operator.

Preparing the bare metal node requires executing the following procedure from the provisioner node.

Procedure
  1. Get the oc binary, if needed. It should already exist on the provisioner node.

    $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc
    $ sudo cp oc /usr/local/bin
  2. Power off the bare metal node by using the baseboard management controller, and ensure it is off.

  3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the user name and password:

    $ echo -ne "root" | base64
    $ echo -ne "password" | base64
  4. Create a configuration file for the bare metal node.

    $ vim bmh.yaml
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: openshift-worker-<num>-bmc-secret
    type: Opaque
    data:
      username: <base64-of-uid>
      password: <base64-of-pwd>
    ---
    apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      name: openshift-worker-<num>
    spec:
      online: true
      bootMACAddress: <NIC1-mac-address>
      bmc:
        address: <protocol>://<bmc-ip>
        credentialsName: openshift-worker-<num>-bmc-secret

    Replace <num> for the worker number of the bare metal node in the two name fields and the credentialsName field. Replace <base64-of-uid> with the base64 string of the user name. Replace <base64-of-pwd> with the base64 string of the password. Replace <NIC1-mac-address> with the MAC address of the bare metal node’s first NIC.

    See the BMC addressing section for additional BMC configuration options. Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc-ip> with the IP address of the bare metal node’s baseboard management controller.

    If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See Diagnosing a host duplicate MAC address for more information.

  5. Create the bare metal node.

    $ oc -n openshift-machine-api create -f bmh.yaml
    secret/openshift-worker-<num>-bmc-secret created
    baremetalhost.metal3.io/openshift-worker-<num> created

    Where <num> will be the worker number.

  6. Power up and inspect the bare metal node.

    $ oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number.

    NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
    openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

Replacing a bare-metal control plane node

Use the following procedure to replace an installer-provisioned OKD control plane node.

If you reuse the BareMetalHost object definition from an existing control plane host, do not leave the externallyProvisioned field set to true.

Existing control plane BareMetalHost objects may have the externallyProvisioned flag set to true if they were provisioned by the OKD installation program.

Prerequisites
  • You have access to the cluster as a user with the cluster-admin role.

  • You have taken an etcd backup.

    Take an etcd backup before performing this procedure so that you can restore your cluster if you encounter any issues. For more information about taking an etcd backup, see the Additional resources section.

Procedure
  1. Ensure that the Bare Metal Operator is available:

    $ oc get clusteroperator baremetal
    Example output
    NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    baremetal   4.8.0     True        False         False      3d15h
  2. Remove the old BareMetalHost and Machine objects:

    $ oc delete bmh -n openshift-machine-api vmaster-0
    $ oc delete machine -n openshift-machine-api kni1-master-0

    After you remove the BareMetalHost and Machine objects, then the Machine controller automatically deletes the Node object.

  3. Create the new BareMetalHost object and the secret to store the BMC credentials:

    $ cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: kni1-master-0-bmc-secret
      namespace: openshift-machine-api
    data:
      password: <username>
      username: <password>
    type: Opaque
    ---
    apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      name: kni1-master-0
      namespace: openshift-machine-api
    spec:
      automatedCleaningMode: disabled
      bmc:
        address: redfish-virtualmedia+http://192.168.124.113:8000/redfish/v1/Systems/f87eaf82-b32d-4291-ace6-b28677964e78
        credentialsName: kni1-master-0-bmc-secret
      bootMACAddress: aa:aa:aa:aa:ab:03
      bootMode: UEFI
      externallyProvisioned: false
      hardwareProfile: unknown
      online: true
    EOF

    After the inspection is complete, the BareMetalHost object is created and available to be provisioned.

  4. View available BareMetalHost objects:

    $ oc get bmh -n openshift-machine-api
    Example output
    NAME           STATE                    CONSUMER                   ONLINE   ERROR   AGE
    kni1-master-0  available                ocp-hkw9p-master-0         true             1h10m
    kni1-master-1  externally provisioned   ocp-hkw9p-master-1         true             4h53m
    kni1-master-2  externally provisioned   ocp-hkw9p-master-2         true             4h53m
    kni1-worker-0  provisioned              ocp-hkw9p-worker-0-ktmmx   true             4h53m
    kni1-worker-1  provisioned              ocp-hkw9p-worker-0-l2zmb   true             4h53m

    There are no MachineSet objects for control plane nodes, so you must create a Machine object instead. You can copy the providerSpec from another control plane Machine object.

  5. Create a Machine object:

    $ cat <<EOF | oc apply -f -
    apiVersion: machine.openshift.io/v1beta1
    kind: Machine
    metadata:
      annotations:
        metal3.io/BareMetalHost: openshift-machine-api/kni1-master-0
      labels:
        machine.openshift.io/cluster-api-cluster: kni1
        machine.openshift.io/cluster-api-machine-role: master
        machine.openshift.io/cluster-api-machine-type: master
      name: kni1-master-0
      namespace: openshift-machine-api
    spec:
      metadata: {}
      providerSpec:
        value:
          apiVersion: baremetal.cluster.k8s.io/v1alpha1
          customDeploy:
            method: install_coreos
          hostSelector: {}
          image:
            checksum: ""
            url: ""
          kind: BareMetalMachineProviderSpec
          metadata:
            creationTimestamp: null
          userData:
            name: master-user-data-managed
    EOF
  6. To define and create the BareMetalHost, Secret, and Machine objects in a single step, create a YAML file (example.yaml) with their definitions and run the following command:

    $ oc create -f example.yaml

    The provisioning process uses the baremetal-operator to install RHCOS and prepare the host to be added to the cluster.

  7. To view the BareMetalHost objects, run the following command:

    $ oc get bmh -A
    Example output
    NAME           STATE                    CONSUMER                   ONLINE   ERROR   AGE
    kni1-master-0  provisioned              ocp-hkw9p-master-0         true             2h53m
    kni1-master-1  externally provisioned   ocp-hkw9p-master-1         true             5h53m
    kni1-master-2  externally provisioned   ocp-hkw9p-master-2         true             5h53m
    kni1-worker-0  provisioned              ocp-hkw9p-worker-0-ktmmx   true             5h53m
    kni1-worker-1  provisioned              ocp-hkw9p-worker-0-l2zmb   true             5h53m
  8. After the RHCOS installation, verify that the BareMetalHost is added to the cluster:

    $ oc get nodes
    Example output
    NAME             STATUS      ROLES    AGE   VERSION
    kni1-master-0    available	 master   4m2s  v1.18.2
    kni1-master-1    available	 master   141m  v1.18.2
    kni1-master-2    available	 master   141m  v1.18.2
    kni1-worker-0    available	 worker   87m   v1.18.2

    After replacement of the new control plane node, the etcd pod running in the new node is in crashloopback status. See "Replacing an unhealthy etcd member" for more information.

Diagnosing a duplicate MAC address when provisioning a new host in the cluster

If the MAC address of an existing bare-metal node in the cluster matches the MAC address of a bare-metal host you are attempting to add to the cluster, the Bare Metal Operator associates the host with the existing node. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. A registration error is displayed for the failed bare-metal host.

You can diagnose a duplicate MAC address by examining the bare-metal hosts that are running in the openshift-machine-api namespace.

Prerequisites
  • Install an OKD cluster on bare metal.

  • Install the OKD CLI oc.

  • Log in as a user with cluster-admin privileges.

Procedure

To determine whether a bare-metal host that fails provisioning has the same MAC address as an existing node, do the following:

  1. Get the bare-metal hosts running in the openshift-machine-api namespace:

    $ oc get bmh -n openshift-machine-api
    Example output
    NAME                 STATUS   PROVISIONING STATUS      CONSUMER
    openshift-master-0   OK       externally provisioned   openshift-zpwpq-master-0
    openshift-master-1   OK       externally provisioned   openshift-zpwpq-master-1
    openshift-master-2   OK       externally provisioned   openshift-zpwpq-master-2
    openshift-worker-0   OK       provisioned              openshift-zpwpq-worker-0-lv84n
    openshift-worker-1   OK       provisioned              openshift-zpwpq-worker-0-zd8lm
    openshift-worker-2   error    registering
  2. To see more detailed information about the status of the failing host, run the following command replacing <bare_metal_host_name> with the name of the host:

    $ oc get -n openshift-machine-api bmh <bare_metal_host_name> -o yaml
    Example output
    ...
    status:
      errorCount: 12
      errorMessage: MAC address b4:96:91:1d:7c:20 conflicts with existing node openshift-worker-1
      errorType: registration error
    ...

Provisioning the bare metal node

Provisioning the bare metal node requires executing the following procedure from the provisioner node.

Procedure
  1. Ensure the PROVISIONING STATUS is ready before provisioning the bare metal node.

    $  oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number.

    NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
    openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true
  2. Get a count of the number of worker nodes.

    $ oc get nodes
    NAME                                                STATUS   ROLES           AGE     VERSION
    provisioner.openshift.example.com            Ready    master          30h     v1.16.2
    openshift-master-1.openshift.example.com            Ready    master          30h     v1.16.2
    openshift-master-2.openshift.example.com            Ready    master          30h     v1.16.2
    openshift-master-3.openshift.example.com            Ready    master          30h     v1.16.2
    openshift-worker-0.openshift.example.com            Ready    master          30h     v1.16.2
    openshift-worker-1.openshift.example.com            Ready    master          30h     v1.16.2
  3. Get the machine set.

    $ oc get machinesets -n openshift-machine-api
    NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
    ...
    openshift-worker-0.example.com      1         1         1       1           55m
    openshift-worker-1.example.com      1         1         1       1           55m
  4. Increase the number of worker nodes by one.

    $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

    Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the machine set from the previous step.

  5. Check the status of the bare metal node.

    $ oc -n openshift-machine-api get bmh openshift-worker-<num>

    Where <num> is the worker node number. The status changes from ready to provisioning.

    NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
    openshift-worker-<num>   OK       provisioning          openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

    The provisioning status remains until the OKD cluster provisions the node. This can take 30 minutes or more. After the node is provisioned, the status will change to provisioned.

    NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
    openshift-worker-<num>   OK       provisioned           openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true
  6. After provisioning completes, ensure the bare metal node is ready.

    $ oc get nodes
    NAME                                          STATUS   ROLES   AGE     VERSION
    provisioner.openshift.example.com             Ready    master  30h     v1.16.2
    openshift-master-1.openshift.example.com      Ready    master  30h     v1.16.2
    openshift-master-2.openshift.example.com      Ready    master  30h     v1.16.2
    openshift-master-3.openshift.example.com      Ready    master  30h     v1.16.2
    openshift-worker-0.openshift.example.com      Ready    master  30h     v1.16.2
    openshift-worker-1.openshift.example.com      Ready    master  30h     v1.16.2
    openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.16.2

    You can also check the kubelet.

    $ ssh openshift-worker-<num>
    [kni@openshift-worker-<num>]$ journalctl -fu kubelet