1. Documentation
×

About disaster recovery

You can restore your OKD cluster after a disaster by choosing the recovery path for etcd quorum loss, expired certificates, or rollback to an earlier state. Choose the option that matches your failure type before you begin recovery.

Disaster recovery requires you to have at least one healthy control plane host.
Quorum restoration

This solution handles situations where you have lost the majority of your control plane hosts, leading to etcd quorum loss and the cluster going offline. This solution does not require an etcd backup.

If you have a majority of your control plane nodes still available and have an etcd quorum, you can replace a single unhealthy etcd member instead of performing quorum restoration.
Restoring to an earlier cluster state

This solution handles situations where you want to restore your cluster to an earlier state, for example, if an administrator deletes something critical. If you have taken an etcd backup, you can restore your cluster to an earlier state.

Restoring to an earlier cluster state is a destructive and destabilizing action to take on a running cluster. This procedure should only be used as a last resort.

Before you restore, understand the impact a state rollback can have on your cluster.
Recovering from expired control plane certificates

This solution handles situations where your control plane certificates have expired. For example, if you shut down your cluster before the first certificate rotation, which occurs 24 hours after installation, your certificates are not rotated and expire. You can follow this procedure to recover from expired control plane certificates.

Testing restore procedures

You can test your cluster restore workflow by simulating etcd failure on nonrecovery nodes and restoring from backup. Use this test to confirm that your etcd backup and restore process works as expected.

You must have SSH access to the cluster. Without SSH access, you cannot disable etcd or manage the kubelet service on nonrecovery nodes.
Prerequisites

  • You have SSH access to control plane hosts.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Use SSH to connect to each of your nonrecovery nodes to disable etcd and the kubelet service:

    1. Disable etcd by running the following command:

      $ sudo /usr/local/bin/disable-etcd.sh

    2. Delete variable data for etcd by running the following command:

      $ sudo rm -rf /var/lib/etcd

    3. Disable the kubelet service by running the following command:

      $ sudo systemctl disable kubelet.service

  2. Exit every SSH session.

  3. Ensure that your nonrecovery nodes are in a NOT READY state by running the following command:

    $ oc get nodes

  4. Restore your cluster to an earlier cluster state using an etcd backup. For more information, see "Restoring to an earlier cluster state".

  5. After you restore the cluster and the API responds, use SSH to connect to each nonrecovery node and enable the kubelet service by running the following command:

    $ sudo systemctl enable kubelet.service

  6. Exit every SSH session.

  7. Verify that your nodes return to the READY state by running the following command:

    $ oc get nodes

  8. Verify that etcd is available by running the following command:

    $ oc get pods -n openshift-etcd
Additional resources