Using Data Mover for CSI snapshots - OADP Application backup and restore | Backup and restore

The OADP Data Mover enables customers to back up Container Storage Interface (CSI) volume snapshots to a remote object store. When Data Mover is enabled, you can restore stateful applications, using CSI volume snapshots pulled from the object store if a failure, accidental deletion, or corruption of the cluster occurs.

The Data Mover solution uses the Restic option of VolSync.

Data Mover supports backup and restore of CSI volume snapshots only.

In OADP 1.2 Data Mover VolumeSnapshotBackups (VSBs) and VolumeSnapshotRestores (VSRs) are queued using the VolumeSnapshotMover (VSM). The VSM’s performance is improved by specifying a concurrent number of VSBs and VSRs simultaneously InProgress. After all async plugin operations are complete, the backup is marked as complete.

The OADP 1.1 Data Mover is a Technology Preview feature.

The OADP 1.2 Data Mover has significantly improved features and performances, but is still a Technology Preview feature.

The OADP Data Mover is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Red Hat recommends that customers who use OADP 1.2 Data Mover in order to back up and restore ODF CephFS volumes, upgrade or install OKD version 4.12 or later for improved performance. OADP Data Mover can leverage CephFS shallow volumes in OKD version 4.12 or later, which based on our testing, can improve the performance of backup times.

CephFS ROX details

Prerequisites

You have verified that the StorageClass and VolumeSnapshotClass custom resources (CRs) support CSI.
You have verified that only one VolumeSnapshotClass CR has the annotation snapshot.storage.kubernetes.io/is-default-class: "true".

In OKD version 4.12 or later, verify that this is the only default VolumeSnapshotClass.
You have verified that deletionPolicy of the VolumeSnapshotClass CR is set to Retain.
You have verified that only one StorageClass CR has the annotation storageclass.kubernetes.io/is-default-class: "true".
You have included the label velero.io/csi-volumesnapshot-class: "true" in your VolumeSnapshotClass CR.

You have verified that the OADP namespace has the annotation oc annotate --overwrite namespace/openshift-adp volsync.backube/privileged-movers="true".

In OADP 1.1 the above setting is mandatory.

In OADP 1.2 the privileged-movers setting is not required in most scenarios. The restoring container permissions should be adequate for the Volsync copy. In some user scenarios, there may be permission errors that the privileged-mover= true setting should resolve.

You have installed the VolSync Operator by using the Operator Lifecycle Manager (OLM).

The VolSync Operator is required for using OADP Data Mover.

You have installed the OADP operator by using OLM.

If you format the volume by using XFS filesystem and the volume is at 100% capacity, the backup fails with a no space left on device error. For example:

Error: relabel failed /var/lib/kubelet/pods/3ac..34/volumes/ \
kubernetes.io~csi/pvc-684..12c/mount: lsetxattr /var/lib/kubelet/ \
pods/3ac..34/volumes/kubernetes.io~csi/pvc-68..2c/mount/data-xfs-103: \
no space left on device

In this scenario, consider resizing the volume or using a different filesystem type, for example, ext4, so that the backup completes successfully.

Procedure

Configure a Restic secret by creating a .yaml file as following:

apiVersion: v1
kind: Secret
metadata:
  name: <secret_name>
  namespace: openshift-adp
type: Opaque
stringData:
  RESTIC_PASSWORD: <secure_restic_password>

By default, the Operator looks for a secret named dm-credential. If you are using a different name, you need to specify the name through a Data Protection Application (DPA) CR using dpa.spec.features.dataMover.credentialName.

Create a DPA CR similar to the following example. The default plugins include CSI.

Example Data Protection Application (DPA) CR

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: us-east-1
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <bucket-prefix>
        provider: aws
  configuration:
    restic:
      enable: <true_or_false>
    velero:
       itemOperationSyncFrequency: "10s"
       defaultPlugins:
        - openshift
        - aws
        - csi
        - vsm (1)
  features:
    dataMover:
      credentialName: restic-secret
      enable: true
      maxConcurrentBackupVolumes: "3" (2)
      maxConcurrentRestoreVolumes: "3" (3)
      pruneInterval: "14" (4)
      volumeOptions: (5)
      sourceVolumeOptions:
          accessMode: ReadOnlyMany
          cacheAccessMode: ReadWriteOnce
          cacheCapacity: 2Gi
      destinationVolumeOptions:
          storageClass: other-storageclass-name
          cacheAccessMode: ReadWriteMany
  snapshotLocations:
    - velero:
        config:
          profile: default
          region: us-west-2
        provider: aws

1	OADP 1.2 only.
2	OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for backup. The default value is 10.
3	OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for restore. The default value is 10.
4	OADP 1.2 only. Optional: Specify the number of days, between running Restic pruning on the repository. The prune operation repacks the data to free space, but it can also generate significant I/O traffic as a part of the process. Setting this option allows a trade-off between storage consumption, from no longer referenced data, and access costs.
5	OADP 1.2 only. Optional: Specify VolumeSync volume options for backup and restore.

The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

Example VolumeSnapshotBackup CRD

apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotBackup
metadata:
  name: <vsb_name>
  namespace: <namespace_name> (1)
spec:
  volumeSnapshotContent:
    name: <snapcontent_name>
  protectedNamespace: <adp_namespace> (2)
  resticSecretRef:
    name: <restic_secret_name>

1	Specify the namespace where the volume snapshot exists.
2	Specify the namespace where the OADP Operator is installed. The default is `openshift-adp`.

Example VolumeSnapshotRestore CRD

apiVersion: datamover.oadp.openshift.io/v1alpha1
kind: VolumeSnapshotRestore
metadata:
  name: <vsr_name>
  namespace: <namespace_name> (1)
spec:
  protectedNamespace: <protected_ns> (2)
  resticSecretRef:
    name: <restic_secret_name>
  volumeSnapshotMoverBackupRef:
    sourcePVCData:
      name: <source_pvc_name>
      size: <source_pvc_size>
    resticrepository: <your_restic_repo>
    volumeSnapshotClassName: <vsclass_name>

1	Specify the namespace where the volume snapshot exists.
2	Specify the namespace where the OADP Operator is installed. The default is `openshift-adp`.

You can back up a volume snapshot by performing the following steps:
1. Create a backup CR:
  apiVersion: velero.io/v1 kind: Backup metadata: name: <backup_name> namespace: <protected_ns> (1) spec: includedNamespaces: - <app_ns> (2) storageLocation: velero-sample-1
  1 Specify the namespace where the Operator is installed. The default namespace is openshift-adp.
  
  2 Specify the application namespace or namespaces to be backed up.
2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands:
  $ oc get vsb -n <app_ns>
  $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"
  A snapshot is created in the object store was configured in the DPA.
  
  If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.
You can restore a volume snapshot by performing the following steps:
1. Delete the application namespace and the VolumeSnapshotContent that was created by the Velero CSI plugin.
2. Create a Restore CR and set restorePVs to true.
  Example Restore CR
  apiVersion: velero.io/v1 kind: Restore metadata: name: <restore_name> namespace: <protected_ns> spec: backupName: <previous_backup_name> restorePVs: true
3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command:
  $ oc get vsr -n <app_ns>
  $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"
4. Check whether your application data and resources have been restored.
  
  If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

1	Specify the namespace where the Operator is installed. The default namespace is `openshift-adp`.
2	Specify the application namespace or namespaces to be backed up.