1. Documentation
×

Disaster recovery

The disaster recovery documentation provides information for administrators on how to recover from several disaster situations that might occur with their OKD cluster. As an administrator, you might need to follow one or more of the following procedures to return your cluster to a working state.

Disaster recovery requires you to have at least one healthy control plane host.

Quorum restoration

You can use the quorum-restore.sh script to restore etcd quorum on clusters that are offline due to quorum loss. When quorum is lost, the OKD API becomes read-only. After quorum is restored, the OKD API returns to read/write mode.

Restoring etcd quorum for high availability clusters

You can use the quorum-restore.sh script to restore etcd quorum on clusters that are offline due to quorum loss. When quorum is lost, the OKD API becomes read-only. After quorum is restored, the OKD API returns to read/write mode.

The quorum-restore.sh script instantly brings back a new single-member etcd cluster based on its local data directory and marks all other members as invalid by retiring the previous cluster identifier. No prior backup is required to restore the control plane from.

For high availability (HA) clusters, a three-node HA cluster requires you to shut down etcd on two hosts to avoid a cluster split. On four-node and five-node HA clusters, you must shut down three hosts. Quorum requires a simple majority of nodes. The minimum number of nodes required for quorum on a three-node HA cluster is two. On four-node and five-node HA clusters, the minimum number of nodes required for quorum is three. If you start a new cluster from backup on your recovery host, the other etcd members might still be able to form quorum and continue service.

You might experience data loss if the host that runs the restoration does not have all data replicated to it.

Quorum restoration should not be used to decrease the number of nodes outside of the restoration process. Decreasing the number of nodes results in an unsupported cluster configuration.
Prerequisites

  • You have SSH access to the node used to restore quorum.

Procedure

  1. Select a control plane host to use as the recovery host. You run the restore operation on this host.

    1. List the running etcd pods by running the following command:

      $ oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running"

    2. Choose a pod and run the following command to obtain its IP address:

      $ oc exec -n openshift-etcd <etcd-pod> -c etcdctl -- etcdctl endpoint status -w table

      Note the IP address of a member that is not a learner and has the highest Raft index.

    3. Run the following command and note the node name that corresponds to the IP address of the chosen etcd member:

      $ oc get nodes -o jsonpath='{range .items[*]}[{.metadata.name},{.status.addresses[?(@.type=="InternalIP")].address}]{end}'

  2. Using SSH, connect to the chosen recovery node and run the following command to restore etcd quorum:

    $ sudo -E /usr/local/bin/quorum-restore.sh

    After a few minutes, the nodes that went down are automatically synchronized with the node that the recovery script was run on. Any remaining online nodes automatically rejoin the new etcd cluster created by the quorum-restore.sh script. This process takes a few minutes.

  3. Exit the SSH session.

  4. Return to a three-node configuration if any nodes are offline. Repeat the following steps for each node that is offline to delete and re-create them. After the machines are re-created, a new revision is forced and etcd automatically scales up.

    • If you use a user-provisioned bare-metal installation, you can re-create a control plane machine by using the same method that you used to originally create it. For more information, see "Installing a user-provisioned cluster on bare metal".

      Do not delete and re-create the machine for the recovery host.

    • If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps:

      Do not delete and re-create the machine for the recovery host.

      For bare-metal installations on installer-provisioned infrastructure, control plane machines are not re-created. For more information, see "Replacing a bare-metal control plane node".

      1. Obtain the machine for one of the offline nodes.

        In a terminal that has access to the cluster as a cluster-admin user, run the following command:

        $ oc get machines -n openshift-machine-api -o wide
        Example output
        NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped (1)
clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
        1 This is the control plane machine for the offline node, ip-10-0-131-183.ec2.internal.

      2. Delete the machine of the offline node by running:

        $ oc delete machine -n openshift-machine-api clustername-8qw5l-master-0 (1)
        1 Specify the name of the control plane machine for the offline node.

        A new machine is automatically provisioned after deleting the machine of the offline node.

  5. Verify that a new machine has been created by running:

    $ oc get machines -n openshift-machine-api -o wide
    Example output
    NAME                                        PHASE          TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
clustername-8qw5l-master-1                  Running        m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-143-125.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
clustername-8qw5l-master-2                  Running        m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-154-194.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba  running
clustername-8qw5l-master-3                  Provisioning   m4.xlarge   us-east-1   us-east-1a   85s     ip-10-0-173-171.ec2.internal    aws:///us-east-1a/i-015b0888fe17bc2c8  running (1)
clustername-8qw5l-worker-us-east-1a-wbtgd   Running        m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
clustername-8qw5l-worker-us-east-1b-lrdxb   Running        m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
clustername-8qw5l-worker-us-east-1c-pkg26   Running        m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running
    1 The new machine, clustername-8qw5l-master-3 is being created and is ready after the phase changes from Provisioning to Running.

    It might take a few minutes for the new machine to be created. The etcd cluster Operator will automatically synchronize when the machine or node returns to a healthy state.

    1. Repeat these steps for each node that is offline.

  6. Wait until the control plane recovers by running the following command:

    $ oc adm wait-for-stable-cluster

    It can take up to 15 minutes for the control plane to recover.
Troubleshooting

  • If you see no progress rolling out the etcd static pods, you can force redeployment from the etcd cluster Operator by running the following command:

    $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$(date --rfc-3339=ns )"'"}}' --type=merge
Additional resources

If you have a majority of your control plane nodes still available and have an etcd quorum, replace a single unhealthy etcd member.

Restoring to a previous cluster state

To restore the cluster to a previous state, you must have previously backed up the etcd data by creating a snapshot. You will use this snapshot to restore the cluster state. For more information, see "Backing up etcd data".

If applicable, you might also need to recover from expired control plane certificates.

Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This procedure should only be used as a last resort.

Before performing a restore, see "About restoring to a previous cluster state" for more information on the impact to the cluster.

About restoring to a previous cluster state

To restore the cluster to a previous state, you must have previously backed up the etcd data by creating a snapshot. You will use this snapshot to restore the cluster state. For more information, see "Backing up etcd data".

You can use an etcd backup to restore your cluster to a previous state. This can be used to recover from the following situations:

  • The cluster has lost the majority of control plane hosts (quorum loss).

  • An administrator has deleted something critical and must restore to recover the cluster.

Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This should only be used as a last resort.

If you are able to retrieve data using the Kubernetes API server, then etcd is available and you should not restore using an etcd backup.

Restoring etcd effectively takes a cluster back in time and all clients will experience a conflicting, parallel history. This can impact the behavior of watching components like kubelets, Kubernetes controller managers, persistent volume controllers, and OKD Operators, including the network Operator.

It can cause Operator churn when the content in etcd does not match the actual content on disk, causing Operators for the Kubernetes API server, Kubernetes controller manager, Kubernetes scheduler, and etcd to get stuck when files on disk conflict with content in etcd. This can require manual actions to resolve the issues.

In extreme cases, the cluster can lose track of persistent volumes, delete critical workloads that no longer exist, reimage machines, and rewrite CA bundles with expired certificates.

Restoring to a previous cluster state for a single node

You can use a saved etcd backup to restore a previous cluster state on a single node.

When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OKD 4.21.2 cluster must use an etcd backup that was taken from 4.21.2.
Prerequisites

  • Access to the cluster as a user with the cluster-admin role through a certificate-based kubeconfig file, like the one that was used during installation.

  • You have SSH access to control plane hosts.

  • A backup directory containing both the etcd snapshot and the resources for the static pods, which were from the same backup. The file names in the directory must be in the following formats: snapshot_<datetimestamp>.db and static_kuberesources_<datetimestamp>.tar.gz.

Procedure

  1. Use SSH to connect to the single node and copy the etcd backup to the /home/core directory by running the following command:

    $ cp <etcd_backup_directory> /home/core

  2. Run the following command in the single node to restore the cluster from a previous backup:

    $ sudo -E /usr/local/bin/cluster-restore.sh /home/core/<etcd_backup_directory>

  3. Exit the SSH session.

  4. Monitor the recovery progress of the control plane by running the following command:

    $ oc adm wait-for-stable-cluster

    It can take up to 15 minutes for the control plane to recover.

Restoring to a previous cluster state for more than one node

You can use a saved etcd backup to restore a previous cluster state or restore a cluster that has lost the majority of control plane hosts.

For high availability (HA) clusters, a three-node HA cluster requires you to shut down etcd on two hosts to avoid a cluster split. On four-node and five-node HA clusters, you must shut down three hosts. Quorum requires a simple majority of nodes. The minimum number of nodes required for quorum on a three-node HA cluster is two. On four-node and five-node HA clusters, the minimum number of nodes required for quorum is three. If you start a new cluster from backup on your recovery host, the other etcd members might still be able to form quorum and continue service.

If your cluster uses a control plane machine set, see "Recovering a degraded etcd Operator" in "Troubleshooting the control plane machine set" for an etcd recovery procedure. For OKD on a single node, see "Restoring to a previous cluster state for a single node".

When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OKD 4.21.2 cluster must use an etcd backup that was taken from 4.21.2.
Prerequisites

  • Access to the cluster as a user with the cluster-admin role through a certificate-based kubeconfig file, like the one that was used during installation.

  • A healthy control plane host to use as the recovery host.

  • You have SSH access to control plane hosts.

  • A backup directory containing both the etcd snapshot and the resources for the static pods, which were from the same backup. The file names in the directory must be in the following formats: snapshot_<datetimestamp>.db and static_kuberesources_<datetimestamp>.tar.gz.

  • Nodes must be accessible or bootable.

For non-recovery control plane nodes, it is not required to establish SSH connectivity or to stop the static pods. You can delete and re-create other non-recovery, control plane machines, one by one.
Procedure

  1. Select a control plane host to use as the recovery host. This is the host that you run the restore operation on.

  2. Establish SSH connectivity to each of the control plane nodes, including the recovery host.

    kube-apiserver becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.

    If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state.

  3. Using SSH, connect to each control plane node and run the following command to disable etcd:

    $ sudo -E /usr/local/bin/disable-etcd.sh

  4. Copy the etcd backup directory to the recovery control plane host.

    This procedure assumes that you copied the backup directory containing the etcd snapshot and the resources for the static pods to the /home/core/ directory of your recovery control plane host.

  5. Use SSH to connect to the recovery host and restore the cluster from a previous backup by running the following command:

    $ sudo -E /usr/local/bin/cluster-restore.sh /home/core/<etcd-backup-directory>

  6. Exit the SSH session.

  7. Once the API responds, turn off the etcd Operator quorum guard by running the following command:

    $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'

  8. Monitor the recovery progress of the control plane by running the following command:

    $ oc adm wait-for-stable-cluster

    It can take up to 15 minutes for the control plane to recover.

  9. Once recovered, enable the quorum guard by running the following command:

    $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'
Troubleshooting

If you see no progress rolling out the etcd static pods, you can force redeployment from the cluster-etcd-operator by running the following command:

$ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$(date --rfc-3339=ns )"'"}}' --type=merge
Additional resources

Issues and workarounds for restoring a persistent storage state

If your OKD cluster uses persistent storage of any form, a state of the cluster is typically stored outside etcd. When you restore from an etcd backup, the status of the workloads in OKD is also restored. However, if the etcd snapshot is old, the status might be invalid or outdated.

The contents of persistent volumes (PVs) are never part of the etcd snapshot. When you restore an OKD cluster from an etcd snapshot, non-critical workloads might gain access to critical data, or vice-versa.

The following are some example scenarios that produce an out-of-date status:

  • MySQL database is running in a pod backed up by a PV object. Restoring OKD from an etcd snapshot does not bring back the volume on the storage provider, and does not produce a running MySQL pod, despite the pod repeatedly attempting to start. You must manually restore this pod by restoring the volume on the storage provider, and then editing the PV to point to the new volume.

  • Pod P1 is using volume A, which is attached to node X. If the etcd snapshot is taken while another pod uses the same volume on node Y, then when the etcd restore is performed, pod P1 might not be able to start correctly due to the volume still being attached to node Y. OKD is not aware of the attachment, and does not automatically detach it. When this occurs, the volume must be manually detached from node Y so that the volume can attach on node X, and then pod P1 can start.

  • Cloud provider or storage provider credentials were updated after the etcd snapshot was taken. This causes any CSI drivers or Operators that depend on the those credentials to not work. You might have to manually update the credentials required by those drivers or Operators.

  • A device is removed or renamed from OKD nodes after the etcd snapshot is taken. The Local Storage Operator creates symlinks for each PV that it manages from /dev/disk/by-id or /dev directories. This situation might cause the local PVs to refer to devices that no longer exist.

    To fix this problem, an administrator must:

    1. Manually remove the PVs with invalid devices.

    2. Remove symlinks from respective nodes.

    3. Delete LocalVolume or LocalVolumeSet objects (see Storage → Configuring persistent storage → Persistent storage using local volumes → Deleting the Local Storage Operator Resources).

Recovering from expired control plane certificates

The cluster can automatically recover from expired control plane certificates.

However, you must manually approve the pending node-bootstrapper certificate signing requests (CSRs) to recover kubelet certificates. For user-provisioned installations, you might also need to approve pending kubelet serving CSRs.

Use the following steps to approve the pending CSRs:

Procedure

  1. Get the list of current CSRs:

    $ oc get csr
    Example output
    NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-2s94x   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending (1)
csr-4bd6t   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending
csr-4hl85   13m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending (2)
csr-zhhhp   3m8s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
...
    1 A pending kubelet service CSR (for user-provisioned installations).
    2 A pending node-bootstrapper CSR.

  2. Review the details of a CSR to verify that it is valid:

    $ oc describe csr <csr_name> (1)
    1 <csr_name> is the name of a CSR from the list of current CSRs.

  3. Approve each valid node-bootstrapper CSR:

    $ oc adm certificate approve <csr_name>

  4. For user-provisioned installations, approve each valid kubelet serving CSR:

    $ oc adm certificate approve <csr_name>

Testing restore procedures

Testing the restore procedure is important to ensure that your automation and workload handle the new cluster state gracefully. Due to the complex nature of etcd quorum and the etcd Operator attempting to mend automatically, it is often difficult to correctly bring your cluster into a broken enough state that it can be restored.

You must have SSH access to the cluster. Your cluster might be entirely lost without SSH access.
Prerequisites

  • You have SSH access to control plane hosts.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Use SSH to connect to each of your nonrecovery nodes and run the following commands to disable etcd and the kubelet service:

    1. Disable etcd by running the following command:

      $ sudo /usr/local/bin/disable-etcd.sh

    2. Delete variable data for etcd by running the following command:

      $ sudo rm -rf /var/lib/etcd

    3. Disable the kubelet service by running the following command:

      $ sudo systemctl disable kubelet.service

  2. Exit every SSH session.

  3. Run the following command to ensure that your nonrecovery nodes are in a NOT READY state:

    $ oc get nodes

  4. Follow the steps in "Restoring to a previous cluster state" to restore your cluster.

  5. After you restore the cluster and the API responds, use SSH to connect to each nonrecovery node and enable the kubelet service:

    $ sudo systemctl enable kubelet.service

  6. Exit every SSH session.

  7. Run the following command to observe your nodes coming back into the READY state:

    $ oc get nodes

  8. Run the following command to verify that etcd is available:

    $ oc get pods -n openshift-etcd
Additional resources