Configuring OADP with AWS S3 compatible storage - OADP Application backup and restore | Backup and restore

Configuring Amazon Web Services
About backup and snapshot locations and their secrets
Configuring the Data Protection Application
- Setting Velero CPU and memory resource allocations
- Enabling self-signed CA certificates
Installing the Data Protection Application 1.2 and earlier
Installing the Data Protection Application 1.3
- Configuring node agents and node labels
Configuring the backup storage location with a MD5 checksum algorithm
Configuring the DPA with client burst and QPS settings
Configuring the DPA with more than one BSL
- Enabling CSI in the DataProtectionApplication CR
- Disabling the node agent in DataProtectionApplication

You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) S3 compatible storage by installing the OADP Operator. The Operator installs Velero 1.14.

IBM Cloud® S3 is supported as an AWS S3 compatible backup storage provider.

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the MTC Operator and are not available as a standalone Operator.

You configure AWS for Velero, create a default Secret, and then install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.

Configuring Amazon Web Services

You configure Amazon Web Services (AWS) for the OpenShift API for Data Protection (OADP).

Prerequisites

You must have the AWS CLI installed.

Procedure

Set the BUCKET variable:
```
$ BUCKET=<your_bucket>
```
Set the REGION variable:
```
$ REGION=<your_region>
```

Create an AWS S3 bucket:

$ aws s3api create-bucket \
    --bucket $BUCKET \
    --region $REGION \
    --create-bucket-configuration LocationConstraint=$REGION (1)

1	`us-east-1` does not support a `LocationConstraint`. If your region is `us-east-1`, omit `--create-bucket-configuration LocationConstraint=$REGION`.

Create an IAM user:
```
$ aws iam create-user --user-name velero (1)
```
1 If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

Create a velero-policy.json file:

$ cat > velero-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}"
            ]
        }
    ]
}
EOF

Attach the policies to give the velero user the minimum necessary permissions:

$ aws iam put-user-policy \
  --user-name velero \
  --policy-name velero \
  --policy-document file://velero-policy.json

Create an access key for the velero user:

$ aws iam create-access-key --user-name velero

Example output

{
  "AccessKey": {
        "UserName": "velero",
        "Status": "Active",
        "CreateDate": "2017-07-31T22:24:41.576Z",
        "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
        "AccessKeyId": <AWS_ACCESS_KEY_ID>
  }
}

Create a credentials-velero file:
```
$ cat << EOF > ./credentials-velero
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF
```
You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

About backup and snapshot locations and their secrets

You specify backup and snapshot locations and their secrets in the DataProtectionApplication custom resource (CR).

Backup locations

You specify AWS S3-compatible object storage as a backup location, such as Multicloud Object Gateway; Red Hat Container Storage; Ceph RADOS Gateway, also known as Ceph Object Gateway; Red Hat OpenShift Data Foundation; or MinIO.

Velero backs up OKD resources, Kubernetes objects, and internal images as an archive file on object storage.

Snapshot locations

If you use your cloud provider’s native snapshot API to back up persistent volumes, you must specify the cloud provider as the snapshot location.

If you use Container Storage Interface (CSI) snapshots, you do not need to specify a snapshot location because you will create a VolumeSnapshotClass CR to register the CSI driver.

If you use File System Backup (FSB), you do not need to specify a snapshot location because FSB backs up the file system on object storage.

Secrets

If the backup and snapshot locations use the same credentials or if you do not require a snapshot location, you create a default Secret.

If the backup and snapshot locations use different credentials, you create two secret objects:

Custom Secret for the backup location, which you specify in the DataProtectionApplication CR.
Default Secret for the snapshot location, which is not referenced in the DataProtectionApplication CR.

The Data Protection Application requires a default Secret. Otherwise, the installation will fail.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file.

Creating a default Secret

You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.

The default name of the Secret is cloud-credentials.

The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.

If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.

Prerequisites

Your object storage and cloud storage, if any, must use the same credentials.
You must configure object storage for Velero.
You must create a credentials-velero file for the object storage in the appropriate format.

Procedure

Create a Secret with the default name:

$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero

The Secret is referenced in the spec.backupLocations.credential block of the DataProtectionApplication CR when you install the Data Protection Application.

Creating profiles for different credentials

If your backup and snapshot locations use different credentials, you create separate profiles in the credentials-velero file.

Then, you create a Secret object and specify the profiles in the DataProtectionApplication custom resource (CR).

Procedure

Create a credentials-velero file with separate profiles for the backup and snapshot locations, as in the following example:

[backupStorage]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

[volumeSnapshot]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

Create a Secret object with the credentials-velero file:

$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero (1)

Add the profiles to the DataProtectionApplication CR, as in the following example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <prefix>
        config:
          region: us-east-1
          profile: "backupStorage"
        credential:
          key: cloud
          name: cloud-credentials
  snapshotLocations:
    - velero:
        provider: aws
        config:
          region: us-west-2
          profile: "volumeSnapshot"

Configuring the backup storage location using AWS

You can configure the AWS backup storage location (BSL) as shown in the following example procedure.

Prerequisites

You have created an object storage bucket using AWS.
You have installed the OADP Operator.

Procedure

Configure the BSL custom resource (CR) with values as applicable to your use case.

Backup storage location

apiVersion: oadp.openshift.io/v1alpha1
kind: BackupStorageLocation
metadata:
  name: default
  namespace: openshift-adp
spec:
  provider: aws (1)
  objectStorage:
    bucket: <bucket_name> (2)
    prefix: <bucket_prefix> (3)
  credential: (4)
    key: cloud (5)
    name: cloud-credentials (6)
  config:
    region: <bucket_region> (7)
    s3ForcePathStyle: "true" (8)
    s3Url: <s3_url> (9)
    publicUrl: <public_s3_url> (10)
    serverSideEncryption: AES256 (11)
    kmsKeyId: "50..c-4da1-419f-a16e-ei...49f" (12)
    customerKeyEncryptionFile: "/credentials/customer-key" (13)
    signatureVersion: "1" (14)
    profile: "default" (15)
    insecureSkipTLSVerify: "true" (16)
    enableSharedConfig: "true" (17)
    tagging: "" (18)
    checksumAlgorithm: "CRC32" (19)

1	The name of the object store plugin. In this example, the plugin is `aws`. This field is required.
2	The name of the bucket in which to store backups. This field is required.
3	The prefix within the bucket in which to store backups. This field is optional.
4	The credentials for the backup storage location. You can set custom credentials. If custom credentials are not set, the default credentials' secret is used.
5	The `key` within the secret credentials' data.
6	The name of the secret containing the credentials.
7	The AWS region where the bucket is located. Optional if s3ForcePathStyle is false.
8	A boolean flag to decide whether to use path-style addressing instead of virtual hosted bucket addressing. Set to `true` if using a storage service such as MinIO or NooBaa. This is an optional field. The default value is `false`.
9	You can specify the AWS S3 URL here for explicitness. This field is primarily for storage services such as MinIO or NooBaa. This is an optional field.
10	This field is primarily used for storage services such as MinIO or NooBaa. This is an optional field.
11	The name of the server-side encryption algorithm to use for uploading objects, for example, `AES256`. This is an optional field.
12	Specify an AWS KMS key ID. You can format, as shown in the example, as an alias, such as `alias/<KMS-key-alias-name>`, or the full `ARN` to enable encryption of the backups stored in S3. Note that `kmsKeyId` cannot be used in with `customerKeyEncryptionFile`. This is an optional field.
13	Specify the file that has the `SSE-C` customer key to enable customer key encryption of the backups stored in S3. The file must contain a 32-byte string. The `customerKeyEncryptionFile` field points to a mounted secret within the `velero` container. Add the following key-value pair to the `velero` `cloud-credentials` secret: `customer-key: <your_b64_encoded_32byte_string>`. Note that the `customerKeyEncryptionFile` field cannot be used with the `kmsKeyId` field. The default value is an empty string (`""`), which means `SSE-C` is disabled. This is an optional field.
14	The version of the signature algorithm used to create signed URLs. You use signed URLs to download the backups, or fetch the logs. Valid values are `1` and `4`. The default version is `4`. This is an optional field.
15	The name of the AWS profile in the credentials file. The default value is `default`. This is an optional field.
16	Set the `insecureSkipTLSVerify` field to `true` if you do not want to verify the TLS certificate when connecting to the object store, for example, for self-signed certificates with MinIO. Setting to `true` is susceptible to man-in-the-middle attacks and is not recommended for production workloads. The default value is `false`. This is an optional field.
17	Set the `enableSharedConfig` field to `true` if you want to load the credentials file as a shared config file. The default value is `false`. This is an optional field.
18	Specify the tags to annotate the AWS S3 objects. Specify the tags in key-value pairs. The default value is an empty string (`""`). This is an optional field.
19	Specify the checksum algorithm to use for uploading objects to S3. The supported values are: `CRC32`, `CRC32C`, `SHA1`, and `SHA256`. If you set the field as an empty string (`""`), the checksum check will be skipped. The default value is `CRC32`. This is an optional field.

Creating an OADP SSE-C encryption key for additional data security

Amazon Web Services (AWS) S3 applies server-side encryption with Amazon S3 managed keys (SSE-S3) as the base level of encryption for every bucket in Amazon S3.

OpenShift API for Data Protection (OADP) encrypts data by using SSL/TLS, HTTPS, and the velero-repo-credentials secret when transferring the data from a cluster to storage. To protect backup data in case of lost or stolen AWS credentials, apply an additional layer of encryption.

The velero-plugin-for-aws plugin provides several additional encryption methods. You should review its configuration options and consider implementing additional encryption.

You can store your own encryption keys by using server-side encryption with customer-provided keys (SSE-C). This feature provides additional security if your AWS credentials become exposed.

Be sure to store cryptographic keys in a secure and safe manner. Encrypted data and backups cannot be recovered if you do not have the encryption key.

Prerequisites

To make OADP mount a secret that contains your SSE-C key to the Velero pod at /credentials, use the following default secret name for AWS: cloud-credentials, and leave at least one of the following labels empty:
- dpa.spec.backupLocations[].velero.credential
- dpa.spec.snapshotLocations[].velero.credential
  
  This is a workaround for a known issue: https://issues.redhat.com/browse/OADP-3971.

The following procedure contains an example of a spec:backupLocations block that does not specify credentials. This example would trigger an OADP secret mounting.

If you need the backup location to have credentials with a different name than cloud-credentials, you must add a snapshot location, such as the one in the following example, that does not contain a credential name. Because the example does not contain a credential name, the snapshot location will use cloud-credentials as its secret for taking snapshots.

Example snapshot location in a DPA without credentials specified

 snapshotLocations:
  - velero:
      config:
        profile: default
        region: <region>
      provider: aws
# ...

Procedure

Create an SSE-C encryption key:
1. Generate a random number and save it as a file named sse.key by running the following command:
  $ dd if=/dev/urandom bs=1 count=32 > sse.key
2. Encode the sse.key by using Base64 and save the result as a file named sse_encoded.key by running the following command:
  $ cat sse.key | base64 > sse_encoded.key
3. Link the file named sse_encoded.key to a new file named customer-key by running the following command:
  $ ln -s sse_encoded.key customer-key

Create an OKD secret:

If you are initially installing and configuring OADP, create the AWS credential and encryption key secret at the same time by running the following command:

$ oc create secret generic cloud-credentials --namespace openshift-adp --from-file cloud=<path>/openshift_aws_credentials,customer-key=<path>/sse_encoded.key

If you are updating an existing installation, edit the values of the cloud-credential secret block of the DataProtectionApplication CR manifest, as in the following example:

apiVersion: v1
data:
  cloud: W2Rfa2V5X2lkPSJBS0lBVkJRWUIyRkQ0TlFHRFFPQiIKYXdzX3NlY3JldF9hY2Nlc3Nfa2V5P<snip>rUE1mNWVSbTN5K2FpeWhUTUQyQk1WZHBOIgo=
  customer-key: v+<snip>TFIiq6aaXPbj8dhos=
kind: Secret
# ...

Edit the value of the customerKeyEncryptionFile attribute in the backupLocations block of the DataProtectionApplication CR manifest, as in the following example:
```
spec:
  backupLocations:
    - velero:
        config:
          customerKeyEncryptionFile: /credentials/customer-key
          profile: default
# ...
```
You must restart the Velero pod to remount the secret credentials properly on an existing installation.

The installation is complete, and you can back up and restore OKD resources. The data saved in AWS S3 storage is encrypted with the new key, and you cannot download it from the AWS S3 console or API without the additional encryption key.

Verification

To verify that you cannot download the encrypted files without the inclusion of an additional key, create a test file, upload it, and then try to download it.

Create a test file by running the following command:
```
$ echo "encrypt me please" > test.txt
```

Upload the test file by running the following command:

$ aws s3api put-object \
  --bucket <bucket> \
  --key test.txt \
  --body test.txt \
  --sse-customer-key fileb://sse.key \
  --sse-customer-algorithm AES256

Try to download the file. In either the Amazon web console or the terminal, run the following command:
```
$ s3cmd get s3://<bucket>/test.txt test.txt
```
The download fails because the file is encrypted with an additional key.

Download the file with the additional encryption key by running the following command:

$ aws s3api get-object \
    --bucket <bucket> \
    --key test.txt \
    --sse-customer-key fileb://sse.key \
    --sse-customer-algorithm AES256 \
    downloaded.txt

Read the file contents by running the following command:
```
$ cat downloaded.txt
```
Example output
```
encrypt me please
```

Additional resources

You can also download the file with the additional encryption key backed up with Velcro by running a different command. See Downloading a file with an SSE-C encryption key for files backed up by Velero.

Downloading a file with an SSE-C encryption key for files backed up by Velero

When you are verifying an SSE-C encryption key, you can also download the file with the additional encryption key for files that were backed up with Velcro.

Procedure

Download the file with the additional encryption key for files backed up by Velero by running the following command:

$ aws s3api get-object \
  --bucket <bucket> \
  --key velero/backups/mysql-persistent-customerkeyencryptionfile4/mysql-persistent-customerkeyencryptionfile4.tar.gz \
  --sse-customer-key fileb://sse.key \
  --sse-customer-algorithm AES256 \
  --debug \
  velero_download.tar.gz

Configuring the Data Protection Application

You can configure the Data Protection Application by setting Velero resource allocations or enabling self-signed CA certificates.

Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
# ...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node_selector> (1)
        resourceAllocations: (2)
          limits:
            cpu: "1"
            memory: 1024Mi
          requests:
            cpu: 200m
            memory: 256Mi

1	Specify the node selector to be supplied to Velero podSpec.
2	The `resourceAllocations` listed are for average usage.

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Use the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint. Any label specified must match the labels on each node.

For more details, see Configuring node agents and node labels.

Enabling self-signed CA certificates

You must enable a self-signed CA certificate for object storage by editing the DataProtectionApplication custom resource (CR) manifest to prevent a certificate signed by unknown authority error.

Prerequisites

You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

Edit the spec.backupLocations.velero.objectStorage.caCert parameter and spec.backupLocations.velero.config parameters of the DataProtectionApplication CR manifest:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
# ...
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket>
          prefix: <prefix>
          caCert: <base64_encoded_cert_string> (1)
        config:
          insecureSkipTLSVerify: "false" (2)
# ...

1	Specify the Base64-encoded CA certificate string.
2	The `insecureSkipTLSVerify` configuration can be set to either `"true"` or `"false"`. If set to `"true"`, SSL/TLS security is disabled. If set to `"false"`, SSL/TLS security is enabled.

Using CA certificates with the velero command aliased for Velero deployment

You might want to use the Velero CLI without installing it locally on your system by creating an alias for it.

Prerequisites

You must be logged in to the OpenShift Container Platform cluster as a user with the cluster-admin role.

You must have the OpenShift CLI (oc) installed.

To use an aliased Velero command, run the following command:

$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'

Check that the alias is working by running the following command:

Example

$ velero version
Client:
	Version: v1.12.1-OADP
	Git commit: -
Server:
	Version: v1.12.1-OADP

To use a CA certificate with this command, you can add a certificate to the Velero deployment by running the following commands:

$ CA_CERT=$(oc -n openshift-adp get dataprotectionapplications.oadp.openshift.io <dpa-name> -o jsonpath='{.spec.backupLocations[0].velero.objectStorage.caCert}')

$ [[ -n $CA_CERT ]] && echo "$CA_CERT" | base64 -d | oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "cat > /tmp/your-cacert.txt" || echo "DPA BSL has no caCert"

$ velero describe backup <backup_name> --details --cacert /tmp/<your_cacert>.txt

To fetch the backup logs, run the following command:
```
$ velero backup logs  <backup_name>  --cacert /tmp/<your_cacert.txt>
```
You can use these logs to view failures and warnings for the resources that you cannot back up.
If the Velero pod restarts, the /tmp/your-cacert.txt file disappears, and you must re-create the /tmp/your-cacert.txt file by re-running the commands from the previous step.
You can check if the /tmp/your-cacert.txt file still exists, in the file location where you stored it, by running the following command:
```
$ oc exec -n openshift-adp -i deploy/velero -c velero -- bash -c "ls /tmp/your-cacert.txt"
/tmp/your-cacert.txt
```

In a future release of OpenShift API for Data Protection (OADP), we plan to mount the certificate to the Velero pod so that this step is not required.

Installing the Data Protection Application 1.2 and earlier

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

You must install the OADP Operator.
You must configure object storage as a backup location.
If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

If the backup and snapshot locations use different credentials, you must create a Secret with the default name, cloud-credentials, which contains separate profiles for the backup and snapshot location credentials.

If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Velero creates a secret named velero-repo-credentials in the OADP namespace, which contains a default backup repository password. You can update the secret with your own password encoded as base64 before you run your first backup targeted to the backup repository. The value of the key to update is Data[repository-password].

After you create your DPA, the first time that you run a backup targeted to the backup repository, Velero creates a backup repository whose secret is velero-repo-credentials, which contains either the default password or the one you replaced it with. If you update the secret password after the first backup, the new password will not match the password in velero-repo-credentials, and therefore, Velero will not be able to connect with the older backups.

Procedure

Click Operators → Installed Operators and select the OADP Operator.
Under Provided APIs, click Create instance in the DataProtectionApplication box.

Click YAML View and update the parameters of the DataProtectionApplication manifest:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift (1)
        - aws
      resourceTimeout: 10m (2)
    restic:
      enable: true (3)
      podConfig:
        nodeSelector: <node_selector> (4)
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name> (5)
          prefix: <prefix> (6)
        config:
          region: <region>
          profile: "default"
          s3ForcePathStyle: "true" (7)
          s3Url: <s3_url> (8)
        credential:
          key: cloud
          name: cloud-credentials (9)
  snapshotLocations: (10)
    - velero:
        provider: aws
        config:
          region: <region> (11)
          profile: "default"
        credential:
          key: cloud
          name: cloud-credentials (12)

1	The `openshift` plugin is mandatory.
2	Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
3	Set this value to `false` if you want to disable the Restic installation. Restic deploys a daemon set, which means that Restic pods run on each working node. In OADP version 1.2 and later, you can configure Restic for backups by adding `spec.defaultVolumesToFsBackup: true` to the `Backup` CR. In OADP version 1.1, add `spec.defaultVolumesToRestic: true` to the `Backup` CR.
4	Specify on which nodes Restic is available. By default, Restic runs on all nodes.
5	Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
6	Specify a prefix for Velero backups, for example, `velero`, if the bucket is used for multiple purposes.
7	Specify whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.
8	Specify the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.
9	Specify the name of the `Secret` object that you created. If you do not specify this value, the default name, `cloud-credentials`, is used. If you specify a custom name, the custom name is used for the backup location.
10	Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.
11	The snapshot location must be in the same region as the PVs.
12	Specify the name of the `Secret` object that you created. If you do not specify this value, the default name, `cloud-credentials`, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in the `credentials-velero` file.

Click Create.

Verification

Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

$ oc get all -n openshift-adp

Example output

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s

Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

Example output

{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

Verify the type is set to Reconciled.

Verify the backup storage location and confirm that the PHASE is Available by running the following command:

$ oc get backupStorageLocation -n openshift-adp

Example output

NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
dpa-sample-1   Available   1s               3d16h   true

Verify that the PHASE is in Available.

Installing the Data Protection Application 1.3

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

You must install the OADP Operator.
You must configure object storage as a backup location.
If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

Procedure

Click Operators → Installed Operators and select the OADP Operator.
Under Provided APIs, click Create instance in the DataProtectionApplication box.

Click YAML View and update the parameters of the DataProtectionApplication manifest:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
  namespace: openshift-adp (1)
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift (2)
        - aws
      resourceTimeout: 10m (3)
    nodeAgent: (4)
      enable: true (5)
      uploaderType: kopia (6)
      podConfig:
        nodeSelector: <node_selector> (7)
  backupLocations:
    - name: default
      velero:
        provider: aws
        default: true
        objectStorage:
          bucket: <bucket_name> (8)
          prefix: <prefix> (9)
        config:
          region: <region>
          profile: "default"
          s3ForcePathStyle: "true" (10)
          s3Url: <s3_url> (11)
        credential:
          key: cloud
          name: cloud-credentials (12)
  snapshotLocations: (13)
    - name: default
      velero:
        provider: aws
        config:
          region: <region> (14)
          profile: "default"
        credential:
          key: cloud
          name: cloud-credentials (15)

1	The default namespace for OADP is `openshift-adp`. The namespace is a variable and is configurable.
2	The `openshift` plugin is mandatory.
3	Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.
4	The administrative agent that routes the administrative requests to servers.
5	Set this value to `true` if you want to enable `nodeAgent` and perform File System Backup.
6	Enter `kopia` or `restic` as your uploader. You cannot change the selection after the installation. For the Built-in DataMover you must use Kopia. The `nodeAgent` deploys a daemon set, which means that the `nodeAgent` pods run on each working node. You can configure File System Backup by adding `spec.defaultVolumesToFsBackup: true` to the `Backup` CR.
7	Specify the nodes on which Kopia or Restic are available. By default, Kopia or Restic run on all nodes.
8	Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
9	Specify a prefix for Velero backups, for example, `velero`, if the bucket is used for multiple purposes.
10	Specify whether to force path style URLs for S3 objects (Boolean). Not Required for AWS S3. Required only for S3 compatible storage.
11	Specify the URL of the object store that you are using to store backups. Not required for AWS S3. Required only for S3 compatible storage.
12	Specify the name of the `Secret` object that you created. If you do not specify this value, the default name, `cloud-credentials`, is used. If you specify a custom name, the custom name is used for the backup location.
13	Specify a snapshot location, unless you use CSI snapshots or a File System Backup (FSB) to back up PVs.
14	The snapshot location must be in the same region as the PVs.
15	Specify the name of the `Secret` object that you created. If you do not specify this value, the default name, `cloud-credentials`, is used. If you specify a custom name, the custom name is used for the snapshot location. If your backup and snapshot locations use different credentials, create separate profiles in the `credentials-velero` file.

Click Create.

Verification

Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

$ oc get all -n openshift-adp

Example output

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/node-agent-9cq4q                                     1/1     Running   0          94s
pod/node-agent-m4lts                                     1/1     Running   0          94s
pod/node-agent-pv4kr                                     1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/node-agent    3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s

Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

Example output

{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

Verify the type is set to Reconciled.

Verify the backup storage location and confirm that the PHASE is Available by running the following command:

$ oc get backupStorageLocation -n openshift-adp

Example output

NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
dpa-sample-1   Available   1s               3d16h   true

Verify that the PHASE is in Available.

Configuring node agents and node labels

The DPA of OADP uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the simplest recommended form of node selection constraint.

Any label specified must match the labels on each node.

The correct way to run the node agent on any node you choose is for you to label the nodes with a custom label:

$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""

Use the same custom label in the DPA.spec.configuration.nodeAgent.podConfig.nodeSelector, which you used for labeling nodes. For example:

configuration:
  nodeAgent:
    enable: true
    podConfig:
      nodeSelector:
        node-role.kubernetes.io/nodeAgent: ""

The following example is an anti-pattern of nodeSelector and does not work unless both labels, 'node-role.kubernetes.io/infra: ""' and 'node-role.kubernetes.io/worker: ""', are on the node:

    configuration:
      nodeAgent:
        enable: true
        podConfig:
          nodeSelector:
            node-role.kubernetes.io/infra: ""
            node-role.kubernetes.io/worker: ""

Configuring the backup storage location with a MD5 checksum algorithm

You can configure the Backup Storage Location (BSL) in the Data Protection Application (DPA) to use a MD5 checksum algorithm for both Amazon Simple Storage Service (Amazon S3) and S3-compatible storage providers. The checksum algorithm calculates the checksum for uploading and downloading objects to Amazon S3. You can use one of the following options to set the checksumAlgorithm field in the spec.backupLocations.velero.config.checksumAlgorithm section of the DPA.

CRC32
CRC32C
SHA1
SHA256

You can also set the checksumAlgorithm field to an empty value to skip the MD5 checksum check.

If you do not set a value for the checksumAlgorithm field, then the default value is set to CRC32.

Prerequisites

You have installed the OADP Operator.
You have configured Amazon S3, or S3-compatible object storage as a backup location.

Procedure

Configure the BSL in the DPA as shown in the following example:

Example Data Protection Application

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: test-dpa
  namespace: openshift-adp
spec:
  backupLocations:
  - name: default
    velero:
      config:
        checksumAlgorithm: "" (1)
        insecureSkipTLSVerify: "true"
        profile: "default"
        region: <bucket_region>
        s3ForcePathStyle: "true"
        s3Url: <bucket_url>
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: <bucket_name>
        prefix: velero
      provider: aws
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - aws
      - csi

1	Specify the `checksumAlgorithm`. In this example, the `checksumAlgorithm` field is set to an empty value. You can select an option from the following list: `CRC32`, `CRC32C`, `SHA1`, `SHA256`.

If you are using Noobaa as the object storage provider, and you do not set the spec.backupLocations.velero.config.checksumAlgorithm field in the DPA, an empty value of checksumAlgorithm is added to the BSL configuration.

The empty value is only added for BSLs that are created using the DPA. This value is not added if you create the BSL by using any other method.

Configuring the DPA with client burst and QPS settings

The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.

You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.

Prerequisites

You have installed the OADP Operator.

Procedure

Configure the client-burst and the client-qps fields in the DPA as shown in the following example:

Example Data Protection Application

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: test-dpa
  namespace: openshift-adp
spec:
  backupLocations:
    - name: default
      velero:
        config:
          insecureSkipTLSVerify: "true"
          profile: "default"
          region: <bucket_region>
          s3ForcePathStyle: "true"
          s3Url: <bucket_url>
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: velero
        provider: aws
  configuration:
    nodeAgent:
      enable: true
      uploaderType: restic
    velero:
      client-burst: 500 (1)
      client-qps: 300 (2)
      defaultPlugins:
        - openshift
        - aws
        - kubevirt

1	Specify the `client-burst` value. In this example, the `client-burst` field is set to 500.
2	Specify the `client-qps` value. In this example, the `client-qps` field is set to 300.

Configuring the DPA with more than one BSL

You can configure the DPA with more than one BSL and specify the credentials provided by the cloud provider.

Prerequisites

You must install the OADP Operator.
You must create the secrets by using the credentials provided by the cloud provider.

Procedure

Configure the DPA with more than one BSL. See the following example.

Example DPA

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
#...
backupLocations:
  - name: aws (1)
    velero:
      provider: aws
      default: true (2)
      objectStorage:
        bucket: <bucket_name> (3)
        prefix: <prefix> (4)
      config:
        region: <region_name> (5)
        profile: "default"
      credential:
        key: cloud
        name: cloud-credentials (6)
  - name: odf (7)
    velero:
      provider: aws
      default: false
      objectStorage:
        bucket: <bucket_name>
        prefix: <prefix>
      config:
        profile: "default"
        region: <region_name>
        s3Url: <url> (8)
        insecureSkipTLSVerify: "true"
        s3ForcePathStyle: "true"
      credential:
        key: cloud
        name: <custom_secret_name_odf> (9)
#...

1	Specify a name for the first BSL.
2	This parameter indicates that this BSL is the default BSL. If a BSL is not set in the `Backup CR`, the default BSL is used. You can set only one BSL as the default.
3	Specify the bucket name.
4	Specify a prefix for Velero backups; for example, `velero`.
5	Specify the AWS region for the bucket.
6	Specify the name of the default `Secret` object that you created.
7	Specify a name for the second BSL.
8	Specify the URL of the S3 endpoint.
9	Specify the correct name for the `Secret`; for example, `custom_secret_name_odf`. If you do not specify a `Secret` name, the default name is used.

Specify the BSL to be used in the backup CR. See the following example.

Example backup CR

apiVersion: velero.io/v1
kind: Backup
# ...
spec:
  includedNamespaces:
  - <namespace> (1)
  storageLocation: <backup_storage_location> (2)
  defaultVolumesToFsBackup: true

1	Specify the namespace to back up.
2	Specify the storage location.

Enabling CSI in the DataProtectionApplication CR

You enable the Container Storage Interface (CSI) in the DataProtectionApplication custom resource (CR) in order to back up persistent volumes with CSI snapshots.

Prerequisites

The cloud provider must support CSI snapshots.

Procedure

Edit the DataProtectionApplication CR, as in the following example:

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
...
spec:
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - csi (1)

1	Add the `csi` default plugin.

Disabling the node agent in DataProtectionApplication

If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.

Procedure

To disable the nodeAgent, set the enable flag to false. See the following example:
Example DataProtectionApplication CR
```
# ...
configuration:
  nodeAgent:
    enable: false  (1)
    uploaderType: kopia
# ...
```
1 Disables the node agent.
To enable the nodeAgent, set the enable flag to true. See the following example:
Example DataProtectionApplication CR
```
# ...
configuration:
  nodeAgent:
    enable: true  (1)
    uploaderType: kopia
# ...
```
1 Enables the node agent.

You can set up a job to enable and disable the nodeAgent field in the DataProtectionApplication CR. For more information, see "Running tasks in pods using jobs".

Additional resources

Configuring the OpenShift API for Data Protection with AWS S3 compatible storage

Configuring Amazon Web Services

About backup and snapshot locations and their secrets

Backup locations

Snapshot locations

Secrets

Creating a default Secret

Creating profiles for different credentials

Configuring the backup storage location using AWS

Creating an OADP SSE-C encryption key for additional data security

Downloading a file with an SSE-C encryption key for files backed up by Velero

Configuring the Data Protection Application

Setting Velero CPU and memory resource allocations

Enabling self-signed CA certificates

Using CA certificates with the velero command aliased for Velero deployment

Installing the Data Protection Application 1.2 and earlier

Installing the Data Protection Application 1.3

Configuring node agents and node labels

Configuring the backup storage location with a MD5 checksum algorithm

Configuring the DPA with client burst and QPS settings

Configuring the DPA with more than one BSL

Enabling CSI in the DataProtectionApplication CR

Disabling the node agent in DataProtectionApplication