1. Documentation
×

Adding compute machines to AWS by using CloudFormation templates

To scale your OKD cluster on Amazon Web Services (AWS) after user-provisioned installation, you can add compute machines by creating CloudFormation stacks from your installation templates. You can then approve certificate signing requests so the new nodes join the cluster.

Prerequisites

  • You installed your cluster on AWS by using the provided AWS CloudFormation templates.

  • You have the JSON file and CloudFormation template that you used to create the compute machines during cluster installation. If you do not have these files, you must recreate them by following the instructions in the installation procedure.

Adding more compute machines to your AWS cluster by using CloudFormation templates

To scale your OKD cluster on Amazon Web Services (AWS), you can add more compute machines by creating additional CloudFormation stacks from the sample templates that you used during installation.

The CloudFormation template creates a stack that represents one compute machine. You must create a stack for each compute machine.

If you do not use the provided CloudFormation template to create your compute nodes, you must review the provided information and manually create the infrastructure. If your cluster does not initialize correctly, you might have to contact Red Hat support with your installation logs.
Prerequisites

  • You installed an OKD cluster by using CloudFormation templates and have access to the JSON file and CloudFormation template that you used to create the compute machines during cluster installation.

  • You installed the AWS CLI.

Procedure

  1. Create another compute stack.

    1. Launch the template:

      $ aws cloudformation create-stack --stack-name <name>
     --template-body file://<template>.yaml
     --parameters file://<parameters>.json

      • <name> is the name for the CloudFormation stack, such as cluster-workers. You must provide the name of this stack if you remove the cluster.

      • <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

      • <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

    2. Confirm that the template components exist:

      $ aws cloudformation describe-stacks --stack-name <name>

  2. Continue to create compute stacks until you have created enough compute machines for your cluster.

Approving the certificate signing requests for your machines

To allow newly added machines to join your OKD cluster, confirm that the cluster approves pending certificate signing requests (CSRs), or approve them yourself. Approve client requests first, then server requests.

Prerequisites

  • You added machines to your cluster.

Procedure

  1. Confirm that the cluster recognizes the machines:

    $ oc get nodes
    Example output
    NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  63m  v1.35.4
master-1  Ready     master  63m  v1.35.4
master-2  Ready     master  64m  v1.35.4

    The output lists all of the machines that you created.

    The preceding output might not include the compute nodes until you approve some CSRs.

  2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

    $ oc get csr
    Example output
    NAME        AGE     REQUESTOR                                                                   CONDITION
csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...

    In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

  3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

    You must approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates rotate, and more than two certificates are present for each node. You must approve all of these certificates. After you approve the client CSR, the kubelet creates a secondary CSR for the serving certificate, which requires manual approval. The machine-approver then automatically approves later serving certificate renewal requests if the kubelet requests a new certificate with the same parameters.

    For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If you do not approve a request, the oc exec, oc rsh, and oc logs commands cannot succeed, because the API server requires a serving certificate when it connects to the kubelet. Any operation that contacts the kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the node-bootstrapper service account in the system:node or system:admin groups submitted the CSR, and confirm the identity of the node.

    • To approve them individually, run the following command for each valid CSR:

      $ oc adm certificate approve <csr_name>

      where:

      <csr_name>

      Specifies the name of a CSR from the list of current CSRs.

    • To approve all pending CSRs, run the following command:

      $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

      Some Operators might not become available until you approve some CSRs. Each node submits two CSRs, so you might need to run the command to approve CSRs many times.

  4. After you approve your client requests, review the server requests for each machine that you added to the cluster:

    $ oc get csr
    Example output
    NAME        AGE     REQUESTOR                                                                   CONDITION
csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
...

  5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

    • To approve them individually, run the following command for each valid CSR:

      $ oc adm certificate approve <csr_name>

      where:

      <csr_name>

      Specifies the name of a CSR from the list of current CSRs.

    • To approve all pending CSRs, run the following command:

      $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

  6. After you approve all client and server CSRs, the machines have the Ready status. Verify this by running the following command:

    $ oc get nodes
    Example output
    NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  73m  v1.35.4
master-1  Ready     master  73m  v1.35.4
master-2  Ready     master  74m  v1.35.4
worker-0  Ready     worker  11m  v1.35.4
worker-1  Ready     worker  11m  v1.35.4

    You might need to wait a few minutes after approval of the server CSRs for the machines to change to the Ready status.