Backing up applications - Application backup and restore | Backup and restore

Creating a Backup CR
Backing up persistent volumes with CSI snapshots
Backing up applications with Restic
Using Data Mover for CSI snapshots
Cleaning up after a backup using Data Mover with OADP 1.1.
- Deleting snapshots in a bucket
- Deleting cluster resources
Creating backup hooks
Scheduling backups
Deleting backups

You back up applications by creating a Backup custom resource (CR). See Creating a Backup CR.

The Backup CR creates backup files for Kubernetes resources and internal images, on S3 object storage, and snapshots for persistent volumes (PVs), if the cloud provider uses a native snapshot API or the Container Storage Interface (CSI) to create snapshots, such as OpenShift Data Foundation 4.

For more information about CSI volume snapshots, see CSI volume snapshots.

The CloudStorage API for S3 storage is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If your cloud provider has a native snapshot API or supports CSI snapshots, the Backup CR backs up persistent volumes (PVs) by creating snapshots. For more information about working with CSI snapshots, see Backing up persistent volumes with CSI snapshots.
If your cloud provider does not support snapshots or if your applications are on NFS data volumes, you can create backups by using Restic. See Backing up applications with Restic.

The OpenShift API for Data Protection (OADP) does not support backing up volume snapshots that were created by other software.

You can create backup hooks to run commands before or after the backup operation. See Creating backup hooks.

You can schedule backups by creating a Schedule CR instead of a Backup CR. See Scheduling backups.

Creating a Backup CR

You back up Kubernetes images, internal images, and persistent volumes (PVs) by creating a Backup custom resource (CR).

Prerequisites

You must install the OpenShift API for Data Protection (OADP) Operator.
The DataProtectionApplication CR must be in a Ready state.
Backup location prerequisites:
- You must have S3 object storage configured for Velero.
- You must have a backup location configured in the DataProtectionApplication CR.
Snapshot location prerequisites:
- Your cloud provider must have a native snapshot API or support Container Storage Interface (CSI) snapshots.
- For CSI snapshots, you must create a VolumeSnapshotClass CR to register the CSI driver.
- You must have a volume location configured in the DataProtectionApplication CR.

Procedure

Retrieve the backupStorageLocations CRs by entering the following command:

$ oc get backupStorageLocations

Example output

NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
velero-sample-1   Available   11s              31m

Create a Backup CR, as in the following example:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  labels:
    velero.io/storage-location: default
  namespace: openshift-adp
spec:
  hooks: {}
  includedNamespaces:
  - <namespace> (1)
  includedResources: [] (2)
  excludedResources: [] (3)
  storageLocation: <velero-sample-1> (4)
  ttl: 720h0m0s
  labelSelector: (5)
    matchLabels:
      app=<label_1>
      app=<label_2>
      app=<label_3>
  orLabelSelectors: (6)
  - matchLabels:
      app=<label_1>
      app=<label_2>
      app=<label_3>

1	Specify an array of namespaces to back up.
2	Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.
3	Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.
4	Specify the name of the `backupStorageLocations` CR.
5	Map of {key,value} pairs of backup resources that have all of the specified labels.
6	Map of {key,value} pairs of backup resources that have one or more of the specified labels.

Verify that the status of the Backup CR is Completed:

$ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'

Backing up persistent volumes with CSI snapshots

You back up persistent volumes with Container Storage Interface (CSI) snapshots by editing the VolumeSnapshotClass custom resource (CR) of the cloud storage before you create the Backup CR.

Prerequisites

The cloud provider must support CSI snapshots.
You must enable CSI in the DataProtectionApplication CR.

Procedure

Add the metadata.labels.velero.io/csi-volumesnapshot-class: "true" key-value pair to the VolumeSnapshotClass CR:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: <volume_snapshot_class_name>
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: <csi_driver>
deletionPolicy: Retain

You can now create a Backup CR.

Backing up applications with Restic

You back up Kubernetes resources, internal images, and persistent volumes with Restic by editing the Backup custom resource (CR).

You do not need to specify a snapshot location in the DataProtectionApplication CR.

Restic does not support backing up hostPath volumes. For more information, see additional Rustic limitations.

Prerequisites

You must install the OpenShift API for Data Protection (OADP) Operator.
You must not disable the default Restic installation by setting spec.configuration.restic.enable to false in the DataProtectionApplication CR.
The DataProtectionApplication CR must be in a Ready state.

Procedure

Edit the Backup CR, as in the following example:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  labels:
    velero.io/storage-location: default
  namespace: openshift-adp
spec:
  defaultVolumesToRestic: true (1)
...

1	Add `defaultVolumesToRestic: true` to the `spec` block.

Using Data Mover for CSI snapshots

Data Mover for CSI snapshots is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The OADP 1.1.0 Data Mover enables customers to back up container storage interface (CSI) volume snapshots to a remote object store. When Data Mover is enabled, you can restore stateful applications from the store if a failure, accidental deletion, or corruption of the cluster occurs. The OADP 1.1.0 Data Mover solution uses the Restic option of VolSync.

Data Mover supports backup and restore of CSI volume snapshots only.

Currently, Data Mover does not support Google Cloud Storage (GCS) buckets.

Prerequisites

You have verified that the StorageClass and VolumeSnapshotClass custom resources (CRs) support CSI.
You have verified that only one volumeSnapshotClass CR has the annotation snapshot.storage.kubernetes.io/is-default-class: true.
You have verified that only one storageClass CR has the annotation storageclass.kubernetes.io/is-default-class: true.
You have included the label velero.io/csi-volumesnapshot-class: 'true' in your VolumeSnapshotClass CR.
You have installed the VolSync Operator by using the Operator Lifecycle Manager (OLM).

The VolSync Operator is required only for use with the Technology Preview Data Mover. The Operator is not required for using OADP production features.
You have installed the OADP operator by using OLM.

Procedure

Configure a Restic secret by creating a .yaml file as following:

apiVersion: v1
kind: Secret
metadata:
  name: <secret_name>
  namespace: openshift-adp
type: Opaque
stringData:
  RESTIC_PASSWORD: <secure_restic_password>

By default, the Operator looks for a secret named dm-credential. If you are using a different name, you need to specify the name through a Data Protection Application (DPA) CR using dpa.spec.features.dataMover.credentialName.

Create a DPA CR similar to the following example. The default plugins include CSI.

Example Data Protection Application (DPA) CR

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  features:
    dataMover:
      enable: true
      credentialName: <secret_name> (1)
  backupLocations:
    - velero:
        config:
          profile: default
          region: us-east-1
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <bucket_prefix>
        provider: aws
  configuration:
    restic:
      enable: <true_or_false>
    velero:
      defaultPlugins:
        - openshift
        - aws
        - csi

1	Add the Restic secret name from the previous step. If this is not done, the default secret name `dm-credential` is used.

The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

Example VolumeSnapshotBackup CRD

apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotBackup
metadata:
  name: <vsb_name>
  namespace: <namespace_name> (1)
spec:
  volumeSnapshotContent:
    name: <snapcontent_name>
  protectedNamespace: <adp_namespace>
  resticSecretRef:
    name: <restic_secret_name>

1	Specify the namespace where the volume snapshot exists.

Example VolumeSnapshotRestore CRD

apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotRestore
metadata:
  name: <vsr_name>
  namespace: <namespace_name> (1)
spec:
  protectedNamespace: <protected_ns> (2)
  resticSecretRef:
    name: <restic_secret_name>
  volumeSnapshotMoverBackupRef:
    sourcePVCData:
      name: <source_pvc_name>
      size: <source_pvc_size>
    resticrepository: <your_restic_repo>
    volumeSnapshotClassName: <vsclass_name>

1	Specify the namespace where the volume snapshot exists.
2	Specify the namespace where the Operator is installed. The default is `openshift-adp`.

You can back up a volume snapshot by performing the following steps:
1. Create a backup CR:
  apiVersion: velero.io/v1 kind: Backup metadata: name: <backup_name> namespace: <protected_ns> (1) spec: includedNamespaces: - <app_ns> storageLocation: velero-sample-1
  1 Specify the namespace where the Operator is installed. The default namespace is openshift-adp.
2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands:
  $ oc get vsb -n <app_ns>
  $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"
  A snapshot is created in the object store was configured in the DPA.
  
  If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.
You can restore a volume snapshot by performing the following steps:
1. Delete the application namespace and the volumeSnapshotContent that was created by the Velero CSI plugin.
2. Create a Restore CR and set restorePVs to true.
  Example Restore CR
  apiVersion: velero.io/v1 kind: Restore metadata: name: <restore_name> namespace: <protected_ns> spec: backupName: <previous_backup_name> restorePVs: true
3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command:
  $ oc get vsr -n <app_ns>
  $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"
4. Check whether your application data and resources have been restored.
  
  If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

Cleaning up after a backup using Data Mover with OADP 1.1.

For OADP 1.1, you must perform a data cleanup after you perform a backup using any version of Data Mover.

The cleanup consists of deleting the following resources:

Snapshots in a bucket
Cluster resources
Volume snapshot backups (VSBs) after a backup procedure that is either run by a schedule or is run repetitively

Deleting snapshots in a bucket

Data Mover might leave one or more snapshots in a bucket after a backup. You can either delete all the snapshots or delete individual snapshots.

Procedure

To delete all snapshots in your bucket, delete the /<protected_namespace> folder that is specified in the Data Protection Application (DPA) .spec.backupLocation.objectStorage.bucket resource.
To delete an individual snapshot:
1. Browse to the /<protected_namespace> folder that is specified in the DPA .spec.backupLocation.objectStorage.bucket resource.
2. Delete the appropriate folders that are prefixed with /<volumeSnapshotContent name>-pvc where <VolumeSnapshotContent_name> is the VolumeSnapshotContent created by Data Mover per PVC.

Deleting cluster resources

Data Mover might leave cluster resources whether or not it successfully backs up your container storage interface (CSI) volume snapshots to a remote object store.

Deleting cluster resources following a successful backup and restore that used Data Mover

You can delete any VolumeSnapshotBackup or VolumeSnapshotRestore CRs that remain in your application namespace after a successful backup and restore where you used Data Mover.

Procedure

Delete cluster resources that remain on the application namespace, the namespace with the application PVCs to backup and restore, after a backup where you use Data Mover:
```
$ oc delete vsb -n <app_namespace> --all
```
Delete cluster resources that remain after a restore where you use Data Mover:
```
$ oc delete vsr -n <app_namespace> --all
```
If needed, delete any VolumeSnapshotContent resources that remain after a backup and restore where you use Data Mover:
```
$ oc delete volumesnapshotcontent --all
```

Deleting cluster resources following a partially successful or a failed backup and restore that used Data Mover

If your backup and restore operation that uses Data Mover either fails or only partially succeeds, you must clean up any VolumeSnapshotBackup (VSB) or VolumeSnapshotRestore custom resource definitions (CRDs) that exist in the application namespace, and clean up any extra resources created by these controllers.

Procedure

Clean up cluster resources that remain after a backup operation where you used Data Mover by entering the following commands:
1. Delete VSB CRDs on the application namespace, the namespace with the application PVCs to backup and restore:
  $ oc delete vsb -n <app_namespace> --all
2. Delete VolumeSnapshot CRs:
  $ oc delete volumesnapshot -A --all
3. Delete VolumeSnapshotContent CRs:
  $ oc delete volumesnapshotcontent --all
4. Delete any PVCs on the protected namespace, the namespace the Operator is installed on.
  $ oc delete pvc -n <protected_namespace> --all
5. Delete any ReplicationSource resources on the namespace.
  $ oc delete replicationsource -n <protected_namespace> --all
Clean up cluster resources that remain after a restore operation using Data Mover by entering the following commands:
1. Delete VSR CRDs:
  $ oc delete vsr -n <app-ns> --all
2. Delete VolumeSnapshot CRs:
  $ oc delete volumesnapshot -A --all
3. Delete VolumeSnapshotContent CRs:
  $ oc delete volumesnapshotcontent --all
4. Delete any ReplicationDestination resources on the namespace.
  $ oc delete replicationdestination -n <protected_namespace> --all

Additional resources

Creating backup hooks

You create backup hooks to run commands in a container in a pod by editing the Backup custom resource (CR).

Pre hooks run before the pod is backed up. Post hooks run after the backup.

Procedure

Add a hook to the spec.hooks block of the Backup CR, as in the following example:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: <backup>
  namespace: openshift-adp
spec:
  hooks:
    resources:
      - name: <hook_name>
        includedNamespaces:
        - <namespace> (1)
        excludedNamespaces: (2)
        - <namespace>
        includedResources: []
        - pods (3)
        excludedResources: [] (4)
        labelSelector: (5)
          matchLabels:
            app: velero
            component: server
        pre: (6)
          - exec:
              container: <container> (7)
              command:
              - /bin/uname (8)
              - -a
              onError: Fail (9)
              timeout: 30s (10)
        post: (11)
...

1	Optional: You can specify namespaces to which the hook applies. If this value is not specified, the hook applies to all namespaces.
2	Optional: You can specify namespaces to which the hook does not apply.
3	Currently, pods are the only supported resource that hooks can apply to.
4	Optional: You can specify resources to which the hook does not apply.
5	Optional: This hook only applies to objects matching the label. If this value is not specified, the hook applies to all namespaces.
6	Array of hooks to run before the backup.
7	Optional: If the container is not specified, the command runs in the first container in the pod.
8	This is the entrypoint for the init container being added.
9	Allowed values for error handling are `Fail` and `Continue`. The default is `Fail`.
10	Optional: How long to wait for the commands to run. The default is `30s`.
11	This block defines an array of hooks to run after the backup, with the same parameters as the pre-backup hooks.

Scheduling backups

You schedule backups by creating a Schedule custom resource (CR) instead of a Backup CR.

Leave enough time in your backup schedule for a backup to finish before another backup is created.

For example, if a backup of a namespace typically takes 10 minutes, do not schedule backups more frequently than every 15 minutes.

Prerequisites

You must install the OpenShift API for Data Protection (OADP) Operator.
The DataProtectionApplication CR must be in a Ready state.

Procedure

Retrieve the backupStorageLocations CRs:

$ oc get backupStorageLocations

Example output

NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
velero-sample-1   Available   11s              31m

Create a Schedule CR, as in the following example:

$ cat << EOF | oc apply -f -
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: <schedule>
  namespace: openshift-adp
spec:
  schedule: 0 7 * * * (1)
  template:
    hooks: {}
    includedNamespaces:
    - <namespace> (2)
    storageLocation: <velero-sample-1> (3)
    defaultVolumesToRestic: true (4)
    ttl: 720h0m0s
EOF

1	`cron` expression to schedule the backup, for example, `0 7 * * *` to perform a backup every day at 7:00.
2	Array of namespaces to back up.
3	Name of the `backupStorageLocations` CR.
4	Optional: Add the `defaultVolumesToRestic: true` key-value pair if you are backing up volumes with Restic.

Verify that the status of the Schedule CR is Completed after the scheduled backup runs:
```
$ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'
```

Deleting backups

You can remove backup files by deleting the Backup custom resource (CR).

After you delete the Backup CR and the associated object storage data, you cannot recover the deleted data.

Prerequisites

You created a Backup CR.
You know the name of the Backup CR and the namespace that contains it.
You downloaded the Velero CLI tool.
You can access the Velero binary in your cluster.

Procedure

Choose one of the following actions to delete the Backup CR:
- To delete the Backup CR and keep the associated object storage data, issue the following command:
  $ oc delete backup <backup_CR_name> -n <velero_namespace>
- To delete the Backup CR and delete the associated object storage data, issue the following command:
  $ velero backup delete <backup_CR_name> -n <velero_namespace>
  Where:
  
  <backup_CR_name>
  
  Specifies the name of the Backup custom resource.
  
  <velero_namespace>
  
  Specifies the namespace that contains the Backup custom resource.

Additional resources

Downloading the Velero CLI tool