OADP Data protection test - OADP Application backup and restore | Backup and restore

Table of Contents

OADP DataProtectionTest CR specification fields
OADP DataProtectionTest CR status fields
Using the DataProtectionTest custom resource
Running a data protection test by configuring a backup storage location specification
Running a data protection test on an Azure object storage
Troubleshooting the DataProtectionTest custom resource

The DataProtectionTest (DPT) is a custom resource (CR) that provides a framework to validate your OADP configuration. The DPT CR checks and reports information for the following parameters:

The upload performance of the backups to the object storage.
The CSI snapshot readiness for persistent volume claims.
The storage bucket configuration, such as encryption and versioning.

Using this information in the DPT CR, you can ensure that your data protection environment is properly configured and performing according to the set configuration.

OADP DataProtectionTest CR specification fields

You can configure the following specification fields in the DataProtectionTest (DPT) custom resource (CR).

Table 1. DPT CR spec fields
Field	Type	Description
`backupLocationName`	string	Name of the `BackupStorageLocation` CR configured in the `DataProtectionApplication` (DPA) CR.
`backupLocationSpec`	object	Inline specification of the `BackupStorageLocation` CR.
`uploadSpeedTestConfig`	object	Configuration to run an upload speed test to the object storage.
`csiVolumeSnapshotTestConfigs`	list	List of persistent volume claims to take a snapshot of and to verify the snapshot readiness.
`forceRun`	boolean	Re-run the DPT CR even if status is `Complete` or `Failed`.
`skipTLSVerify`	boolean	Bypasses the TLS certificate validation if set to `true`.

OADP DataProtectionTest CR status fields

You can review the status of the DataProtectionTest (DPT) custom resource (CR) by using the following status fields:

Table 2. DPT CR status fields
Field	Type	Description
`phase`	string	Current phase of the DPT CR. Values are `InProgress`, `Complete`, or `Failed`.
`lastTested`	timestamp	The timestamp when the DPT CR was last run.
`uploadTest`	object	Results of the upload speed test.
`bucketMetadata`	object	Information about the storage bucket encryption and versioning.
`snapshotTests`	list	Snapshot test results for each persistent volume claim.
`snapshotSummary`	string	Aggregated pass/fail summary for snapshots. For example, `2/2 passed`.
`s3Vendor`	string	AWS S3-compatible storage bucket vendors. For example, AWS, MinIO, Ceph.
`errorMessage`	string	Error message if the DPT CR fails.

Using the DataProtectionTest custom resource

You can configure the DataProtectionTest (DPT) custom resource (CR) and then run the DPT CR to verify the Container Storage Initiative (CSI) snapshot readiness and the data upload performance to the storage bucket.

Prerequisites

You have logged in to the OKD cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI (oc).
You have installed the OADP Operator.
You have created the DataProtectionApplication (DPA) CR.
You have configured a backup storage location (BSL) to store the backups.
You have an application with persistent volume claims (PVCs) running in a separate namespace.

Procedure

Create a manifest file for the DPT CR as shown in the example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionTest
metadata:
  name: dpt-sample
  namespace: openshift-adp
spec:
  backupLocationName: <bsl_name> (1)
  csiVolumeSnapshotTestConfigs: (2)
  - snapshotClassName: csi-gce-pd-vsc
    timeout: 90s
    volumeSnapshotSource:
      persistentVolumeClaimName: <pvc1_name> (3)
      persistentVolumeClaimNamespace: <pvc_namespace> (4)
  - snapshotClassName: csi-gce-pd-vsc
    timeout: 120s
    volumeSnapshotSource:
      persistentVolumeClaimName: <pvc2_name> (5)
      persistentVolumeClaimNamespace: <pvc_namespace>
  forceRun: false (6)
  uploadSpeedTestConfig: (7)
    fileSize: 200MB
    timeout: 120s

1	Specify the name of the BSL.
2	Specify a list for `csiVolumeSnapshotTestConfigs`. In this example, two PVCs are being tested.
3	Specify the name of the first PVC.
4	Specify the namespace of the PVC.
5	Specify the name of the second PVC.
6	Set the `forceRun` flag to `false` if you want to make the OADP controller skip re-running tests.
7	Configure the `uploadSpeedTestConfig` object by setting the `fileSize` and `timeout` fields.

Create the DPT CR by running the following command:
```
$ oc create -f <dpt_file_name> (1)
```
1 Specify the file name of the DPT manifest.

Verification

Verify that the phase of the DPT CR is Complete by running the following command:

$ oc get dpt dpt-sample

The example output is as following:

NAME         PHASE      LASTTESTED   UPLOADSPEED(MBPS)   ENCRYPTION   VERSIONING   SNAPSHOTS    AGE
dpt-sample   Complete   17m          546                 AES256       Enabled      2/2 passed   17m

Verify that the CSI snapshots are ready and the data upload tests are successful by running the following command:

$ oc get dpt dpt-sample -o yaml

The example output is as following:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionTest
....
status:
  bucketMetadata: (1)
    encryptionAlgorithm: AES256
    versioningStatus: Enabled
  lastTested: "202...:47:51Z"
  phase: Complete
  s3Vendor: AWS (2)
  snapshotSummary: 2/2 passed (3)
  snapshotTests:
  - persistentVolumeClaimName: mysql-data
    persistentVolumeClaimNamespace: ocp-mysql
    readyDuration: 24s
    status: Ready
  - persistentVolumeClaimName: mysql-data1
    persistentVolumeClaimNamespace: ocp-mysql
    readyDuration: 40s
    status: Ready
  uploadTest: (4)
    duration: 3.071s
    speedMbps: 546
    success: true

1	The bucket metadata information.
2	The S3 bucket vendor.
3	Summary of the CSI snapshot tests.
4	The upload test details.

Running a data protection test by configuring a backup storage location specification

You can configure the DataProtectionTest (DPT) custom resource (CR) by specifying the backup storage location (BSL) specification instead of a BSL name. You then run the DPT CR to verify the Container Storage Initiative (CSI) snapshot readiness and the data upload performance to the storage bucket.

Prerequisites

You have logged in to the OKD cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI (oc).
You have installed the OADP Operator.
You have created the DataProtectionApplication (DPA) CR.
You have configured a bucket to store the backups.
You have created the Secret object to access the bucket storage.
You have an application with persistent volume claims (PVCs) running in a separate namespace.

Procedure

Create a manifest file for the DPT CR as shown in the example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionTest
metadata:
  name: dpt-sample
  namespace: openshift-adp
spec:
  backupLocationSpec: (1)
    provider: aws
    default: true
    objectStorage:
      bucket: sample-bucket (2)
      prefix: velero
    config:
      region: us-east-1 (3)
      profile: "default"
      insecureSkipTLSVerify: "true"
      s3Url: "https://s3.amazonaws.com/sample-bucket"
    credential: (4)
      name: cloud-credentials
      key: cloud
  uploadSpeedTestConfig: (5)
    fileSize: 50MB
    timeout: 120s
  csiVolumeSnapshotTestConfigs: (6)
    - volumeSnapshotSource:
        persistentVolumeClaimName: mongo
        persistentVolumeClaimNamespace: mongo-persistent
      snapshotClassName: csi-snapclass
      timeout: 2m
  forceRun: true
  skipTLSVerify: true (7)

1	Configure the BSL spec by specifying details such as the cloud provider.
2	Specify the bucket name. In this example, the bucket name is `sample-bucket`.
3	Specify the cloud provider region.
4	Specify the cloud credentials for the storage bucket.
5	(Optional) Configure the `uploadSpeedTestConfig` object by setting the `fileSize` and `timeout` fields.
6	Configure the `csiVolumeSnapshotTestConfigs` object.
7	Set to `true` to skip the TLS certificate validation during the DPT CR run.

Create the DPT CR by running the following command:
```
$ oc create -f <dpt_file_name> (1)
```
1 Specify the file name of the DPT manifest.

Verification

Verify that the phase of the DPT CR is Complete by running the following command:

$ oc get dpt dpt-sample

The example output is as following:

NAME         PHASE      LASTTESTED   UPLOADSPEED(MBPS)   ENCRYPTION   VERSIONING   SNAPSHOTS    AGE
dpt-sample   Complete   17m          546                 AES256       Enabled      2/2 passed   17m

Running a data protection test on an Azure object storage

If you are using OADP on an Azure object storage, you need to specify the Azure STORAGE_ACCOUNT_ID as part of the secret object. Use the following procedure to run a DataProtectionTest (DPT) custom resource (CR) on an Azure cluster.

Prerequisites

You have logged in to the Azure cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI (oc).
You have installed the OADP Operator.
You have configured a bucket to store the backups.
You have an application with persistent volume claims (PVCs) running in a separate namespace.

Procedure

Add the Storage Blob Data Contributor role to Azure storageAccount object to avoid DPT run failure. Run the following command:

$ az role assignment create \
--assignee "$AZURE_CLIENT_ID" \
--role "Storage Blob Data Contributor" \
--scope "/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$AZURE_RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/$AZURE_STORAGE_ACCOUNT_ID"

In your terminal, export the Azure parameters and create a secret credentials file with the parameters as shown in the following example.

To run the DPT CR on Azure, you need to specify the STORAGE_ACCOUNT_ID parameter in the secret credentials file.
```
AZURE_SUBSCRIPTION_ID=<subscription-id>
AZURE_TENANT_ID=<tenant-id>
AZURE_CLIENT_ID=<client-id>
AZURE_CLIENT_SECRET=<client-secret>
AZURE_RESOURCE_GROUP=<resource-group>
AZURE_STORAGE_ACCOUNT_ID=<storage-account>
```

Create the Secret CR as shown in the following example:

$ oc create secret generic cloud-credentials-azure -n openshift-adp --from-file cloud=<credentials_file_path>

Create the DataProtectionApplication (DPA) CR by using the configuration shown in the following example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: ts-dpa
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - azure
        - openshift
  backupLocations:
    - velero:
        config:
          resourceGroup: oadp-....-b7q4-rg
          storageAccount: oadp...kb7q4
          subscriptionId: 53b8f5...fd54c8a
        credential:
          key: cloud
          name: cloud-credentials-azure (1)
        provider: azure
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: velero

1	Specify the name of the `Secret` object. In this example, the name is `cloud-credentials-azure`.

Create the DPT CR by specifying the name of backup storage location (BSL), VolumeSnapshotClass object, and the persistent volume claim details as shown in the following example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionTest
metadata:
  name: dpt-sample
  namespace: openshift-adp
spec:
  backupLocationName: <bsl_name> (1)
  uploadSpeedTestConfig:
    fileSize: 40MB
    timeout: 120s
  csiVolumeSnapshotTestConfigs:
    - snapshotClassName: csi-azuredisk-vsc (2)
      timeout: 90s
      volumeSnapshotSource:
        persistentVolumeClaimName: mysql-data (3)
        persistentVolumeClaimNamespace: ocp-mysql (4)
    - snapshotClassName: csi-azuredisk-vsc
      timeout: 120s
      volumeSnapshotSource:
        persistentVolumeClaimName: mysql-data1
        persistentVolumeClaimNamespace: ocp-mysql

1	Specify the name of the BSL.
2	The Azure snapshot class name.
3	The name of the persistent volume claim.
4	The name of the persistent volume claim namespace.

Run the DPT CR to verify the snapshot readiness.

Troubleshooting the DataProtectionTest custom resource

Use the following table to troubleshoot common issues when running the DataProtectionTest (DPT) custom resource (CR).

Table 3. DPT CR troubleshooting
Error	Reason	Solution
DPT stuck in `InProgress` state	Bucket credentials or bucket access failure	Check `Secret` object, bucket permissions, and logs.
Upload test failed	Incorrect `Secret` object or S3 endpoint	Check the `BackupStorageLocation` object config and the access keys.
Snapshot tests fail	Incorrect configuration of CSI snapshot controller	Check the `VolumeSnapshotClass` object availability and the CSI driver logs.
Bucket encryption or versioning not populated	Cloud provider limitations	Not all object storage providers expose these fields consistently.