1. Documentation
×

Backing up applications on AWS STS using OADP

You install the OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) by installing the OADP Operator. The Operator installs Velero 1.14.

Starting from OADP 1.0.4, all OADP 1.0.z versions can only be used as a dependency of the Migration Toolkit for Containers Operator and are not available as a standalone Operator.

You configure AWS for Velero, create a default Secret, and then install the Data Protection Application. For more details, see Installing the OADP Operator.

To install the OADP Operator in a restricted network environment, you must first disable the default OperatorHub sources and mirror the Operator catalog. See Using Operator Lifecycle Manager on restricted networks for details.

You can install OADP on an AWS Security Token Service (STS) (AWS STS) cluster manually. Amazon AWS provides AWS STS as a web service that enables you to request temporary, limited-privilege credentials for users. You use STS to provide trusted users with temporary access to resources via API calls, your AWS console, or the AWS command line interface (CLI).

Before installing OpenShift API for Data Protection (OADP), you must set up role and policy credentials for OADP so that it can use the Amazon Web Services API.

This process is performed in the following two stages:

  1. Prepare AWS credentials.

  2. Install the OADP Operator and give it an IAM role.

Preparing AWS STS credentials for OADP

An Amazon Web Services account must be prepared and configured to accept an OpenShift API for Data Protection (OADP) installation. Prepare the AWS credentials by using the following procedure.

Procedure

  1. Define the cluster_name environment variable by running the following command:

    $ export CLUSTER_NAME= <AWS_cluster_name> (1)
    1 The variable can be set to any value.

  2. Retrieve all of the details of the cluster such as the AWS_ACCOUNT_ID, OIDC_ENDPOINT by running the following command:

    $ export CLUSTER_VERSION=$(oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}')

export AWS_CLUSTER_ID=$(oc get clusterversion version -o jsonpath='{.spec.clusterID}{"\n"}')

export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||')

export REGION=$(oc get infrastructures cluster -o jsonpath='{.status.platformStatus.aws.region}' --allow-missing-template-keys=false || echo us-east-2)

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

export ROLE_NAME="${CLUSTER_NAME}-openshift-oadp-aws-cloud-credentials"

  3. Create a temporary directory to store all of the files by running the following command:

    $ export SCRATCH="/tmp/${CLUSTER_NAME}/oadp"
mkdir -p ${SCRATCH}

  4. Display all of the gathered details by running the following command:

    $ echo "Cluster ID: ${AWS_CLUSTER_ID}, Region: ${REGION}, OIDC Endpoint:
${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

  5. On the AWS account, create an IAM policy to allow access to AWS S3:

    1. Check to see if the policy exists by running the following commands:

      $ export POLICY_NAME="OadpVer1" (1)
      1 The variable can be set to any value. 
      $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='$POLICY_NAME'].{ARN:Arn}" --output text)

    2. Enter the following command to create the policy JSON file and then create the policy:

      If the policy ARN is not found, the command creates the policy. If the policy ARN already exists, the if statement intentionally skips the policy creation.

      		 
      $ if [[ -z "${POLICY_ARN}" ]]; then
cat << EOF > ${SCRATCH}/policy.json
{
"Version": "2012-10-17",
"Statement": [
 {
   "Effect": "Allow",
   "Action": [
     "s3:CreateBucket",
     "s3:DeleteBucket",
     "s3:PutBucketTagging",
     "s3:GetBucketTagging",
     "s3:PutEncryptionConfiguration",
     "s3:GetEncryptionConfiguration",
     "s3:PutLifecycleConfiguration",
     "s3:GetLifecycleConfiguration",
     "s3:GetBucketLocation",
     "s3:ListBucket",
     "s3:GetObject",
     "s3:PutObject",
     "s3:DeleteObject",
     "s3:ListBucketMultipartUploads",
     "s3:AbortMultipartUpload",
     "s3:ListMultipartUploadParts",
     "ec2:DescribeSnapshots",
     "ec2:DescribeVolumes",
     "ec2:DescribeVolumeAttribute",
     "ec2:DescribeVolumesModifications",
     "ec2:DescribeVolumeStatus",
     "ec2:CreateTags",
     "ec2:CreateVolume",
     "ec2:CreateSnapshot",
     "ec2:DeleteSnapshot"
   ],
   "Resource": "*"
 }
]}
EOF

POLICY_ARN=$(aws iam create-policy --policy-name $POLICY_NAME \
--policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \
--tags Key=openshift_version,Value=${CLUSTER_VERSION} Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=oadp \
--output text) (1)
fi
      1 SCRATCH is a name for a temporary directory created for storing the files.

    3. View the policy ARN by running the following command:

      $ echo ${POLICY_ARN}

  6. Create an IAM role trust policy for the cluster:

    1. Create the trust policy file by running the following command:

      $ cat <<EOF > ${SCRATCH}/trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_ENDPOINT}:sub": [
            "system:serviceaccount:openshift-adp:openshift-adp-controller-manager",
            "system:serviceaccount:openshift-adp:velero"]
        }
      }
    }]
}
EOF

    2. Create an IAM role trust policy for the cluster by running the following command:

      $ ROLE_ARN=$(aws iam create-role --role-name \
  "${ROLE_NAME}" \
  --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
  --tags Key=cluster_id,Value=${AWS_CLUSTER_ID}  Key=openshift_version,Value=${CLUSTER_VERSION} Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=oadp --query Role.Arn --output text)

    3. View the role ARN by running the following command:

      $ echo ${ROLE_ARN}

  7. Attach the IAM policy to the IAM role by running the following command:

    $ aws iam attach-role-policy --role-name "${ROLE_NAME}" --policy-arn ${POLICY_ARN}

Setting Velero CPU and memory resource allocations

You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.

Prerequisites

  • You must have the OpenShift API for Data Protection (OADP) Operator installed.

Procedure

  • Edit the values in the spec.configuration.velero.podConfig.ResourceAllocations block of the DataProtectionApplication CR manifest, as in the following example:

    apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <dpa_sample>
spec:
# ...
  configuration:
    velero:
      podConfig:
        nodeSelector: <node_selector> (1)
        resourceAllocations: (2)
          limits:
            cpu: "1"
            memory: 1024Mi
          requests:
            cpu: 200m
            memory: 256Mi
    1 Specify the node selector to be supplied to Velero podSpec.
    2 The resourceAllocations listed are for average usage.

Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.

Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.

Installing the OADP Operator and providing the IAM role

AWS Security Token Service (AWS STS) is a global web service that provides short-term credentials for IAM or federated users. This document describes how to install OpenShift API for Data Protection (OADP) on an AWS STS cluster manually.

Restic and Kopia are not supported in the OADP AWS STS environment. Verify that the Restic and Kopia node agent is disabled. For backing up volumes, OADP on AWS STS supports only native snapshots and Container Storage Interface (CSI) snapshots.

In an AWS cluster that uses STS authentication, restoring backed-up data in a different AWS region is not supported.

The Data Mover feature is not currently supported in AWS STS clusters. You can use native AWS S3 tools for moving data.
Prerequisites

  • An OKD AWS STS cluster with the required access and tokens. For instructions, see the previous procedure Preparing AWS credentials for OADP. If you plan to use two different clusters for backing up and restoring, you must prepare AWS credentials, including ROLE_ARN, for each cluster.

Procedure

  1. Create an OKD secret from your AWS token file by entering the following commands:

    1. Create the credentials file:

      $ cat <<EOF > ${SCRATCH}/credentials
  [default]
  role_arn = ${ROLE_ARN}
  web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
EOF

    2. Create a namespace for OADP:

      $ oc create namespace openshift-adp

    3. Create the OKD secret:

      $ oc -n openshift-adp create secret generic cloud-credentials \
  --from-file=${SCRATCH}/credentials

      In OKD versions 4.14 and later, the OADP Operator supports a new standardized STS workflow through the Operator Lifecycle Manager (OLM) and Cloud Credentials Operator (CCO). In this workflow, you do not need to create the above secret, you only need to supply the role ARN during the installation of OLM-managed operators using the OKD web console, for more information see Installing from OperatorHub using the web console.

      The preceding secret is created automatically by CCO.

  2. Install the OADP Operator:

    1. In the OKD web console, browse to Operators → OperatorHub.

    2. Search for the OADP Operator.

    3. In the role_ARN field, paste the role_arn that you created previously and click Install.

  3. Create AWS cloud storage using your AWS credentials by entering the following command:

    $ cat << EOF | oc create -f -
  apiVersion: oadp.openshift.io/v1alpha1
  kind: CloudStorage
  metadata:
    name: ${CLUSTER_NAME}-oadp
    namespace: openshift-adp
  spec:
    creationSecret:
      key: credentials
      name: cloud-credentials
    enableSharedConfig: true
    name: ${CLUSTER_NAME}-oadp
    provider: aws
    region: $REGION
EOF

  4. Check your application’s storage default storage class by entering the following command:

    $ oc get pvc -n <namespace>
    Example output
    NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
applog   Bound    pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8   1Gi        RWO            gp3-csi        4d19h
mysql    Bound    pvc-16b8e009-a20a-4379-accc-bc81fedd0621   1Gi        RWO            gp3-csi        4d19h

  5. Get the storage class by running the following command:

    $ oc get storageclass
    Example output
    NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   4d21h
gp2-csi             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
gp3                 ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
gp3-csi (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h

    The following storage classes will work:

    • gp3-csi

    • gp2-csi

    • gp3

    • gp2

    If the application or applications that are being backed up are all using persistent volumes (PVs) with Container Storage Interface (CSI), it is advisable to include the CSI plugin in the OADP DPA configuration.

  6. Create the DataProtectionApplication resource to configure the connection to the storage where the backups and volume snapshots are stored:

    1. If you are using only CSI volumes, deploy a Data Protection Application by entering the following command:

      $ cat << EOF | oc create -f -
  apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
  spec:
    backupImages: true (1)
    features:
      dataMover:
        enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
        - csi
      restic:
        enable: false
EOF
      1 Set this field to false if you do not want to use image backup.

  1. If you are using CSI or non-CSI volumes, deploy a Data Protection Application by entering the following command:

    $ cat << EOF | oc create -f -
  apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
  spec:
    backupImages: true (1)
    features:
      dataMover:
         enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
      nodeAgent: (2)
        enable: false
        uploaderType: restic
    snapshotLocations:
      - velero:
          config:
            credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials (3)
            enableSharedConfig: "true" (4)
            profile: default (5)
            region: ${REGION} (6)
          provider: aws
EOF
    1 Set this field to false if you do not want to use image backup.
    2 See the important note regarding the nodeAgent attribute.
    3 The credentialsFile field is the mounted location of the bucket credential on the pod.
    4 The enableSharedConfig field allows the snapshotLocations to share or reuse the credential defined for the bucket.
    5 Use the profile name set in the AWS credentials file.
    6 Specify region as your AWS region. This must be the same as the cluster region.

    You are now ready to back up and restore OKD applications, as described in Backing up applications.

If you use OADP 1.2, replace this configuration:

nodeAgent:
  enable: false
  uploaderType: restic

with the following configuration:

restic:
  enable: false

If you want to use two different clusters for backing up and restoring, the two clusters must have the same AWS S3 storage names in both the cloud storage CR and the OADP DataProtectionApplication configuration.

Additional resources

Backing up workload on OADP AWS STS, with an optional cleanup

Performing a backup with OADP and AWS STS

The following example hello-world application has no persistent volumes (PVs) attached. Perform a backup with OpenShift API for Data Protection (OADP) with Amazon Web Services (AWS) (AWS STS).

Either Data Protection Application (DPA) configuration will work.

  1. Create a workload to back up by running the following commands:

    $ oc create namespace hello-world
    $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

  2. Expose the route by running the following command:

    $ oc expose service/hello-openshift -n hello-world

  3. Check that the application is working by running the following command:

    $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`
    Example output
    Hello OpenShift!

  4. Back up the workload by running the following command:

    $ cat << EOF | oc create -f -
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: hello-world
    namespace: openshift-adp
  spec:
    includedNamespaces:
    - hello-world
    storageLocation: ${CLUSTER_NAME}-dpa-1
    ttl: 720h0m0s
EOF

  5. Wait until the backup has completed and then run the following command:

    $ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"
    Example output
    {
  "completionTimestamp": "2022-09-07T22:20:44Z",
  "expiration": "2022-10-07T22:20:22Z",
  "formatVersion": "1.1.0",
  "phase": "Completed",
  "progress": {
    "itemsBackedUp": 58,
    "totalItems": 58
  },
  "startTimestamp": "2022-09-07T22:20:22Z",
  "version": 1
}

  6. Delete the demo workload by running the following command:

    $ oc delete ns hello-world

  7. Restore the workload from the backup by running the following command:

    $ cat << EOF | oc create -f -
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: hello-world
    namespace: openshift-adp
  spec:
    backupName: hello-world
EOF

  8. Wait for the Restore to finish by running the following command:

    $ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"
    Example output
    {
  "completionTimestamp": "2022-09-07T22:25:47Z",
  "phase": "Completed",
  "progress": {
    "itemsRestored": 38,
    "totalItems": 38
  },
  "startTimestamp": "2022-09-07T22:25:28Z",
  "warnings": 9
}

  9. Check that the workload is restored by running the following command:

    $ oc -n hello-world get pods
    Example output
    NAME                              READY   STATUS    RESTARTS   AGE
hello-openshift-9f885f7c6-kdjpj   1/1     Running   0          90s

  10. Check the JSONPath by running the following command:

    $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`
    Example output
    Hello OpenShift!

For troubleshooting tips, see the OADP team’s troubleshooting documentation.

Cleaning up a cluster after a backup with OADP and AWS STS

If you need to uninstall the OpenShift API for Data Protection (OADP) Operator together with the backups and the S3 bucket from this example, follow these instructions.

Procedure

  1. Delete the workload by running the following command:

    $ oc delete ns hello-world

  2. Delete the Data Protection Application (DPA) by running the following command:

    $ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa

  3. Delete the cloud storage by running the following command:

    $ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp

    If this command hangs, you might need to delete the finalizer by running the following command:

    $ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge

  4. If the Operator is no longer required, remove it by running the following command:

    $ oc -n openshift-adp delete subscription oadp-operator

  5. Remove the namespace from the Operator by running the following command:

    $ oc delete ns openshift-adp

  6. If the backup and restore resources are no longer required, remove them from the cluster by running the following command:

    $ oc delete backups.velero.io hello-world

  7. To delete backup, restore and remote objects in AWS S3, run the following command:

    $ velero backup delete hello-world

  8. If you no longer need the Custom Resource Definitions (CRD), remove them from the cluster by running the following command:

    $ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done

  9. Delete the AWS S3 bucket by running the following commands:

    $ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive
    $ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp

  10. Detach the policy from the role by running the following command:

    $ aws iam detach-role-policy --role-name "${ROLE_NAME}"  --policy-arn "${POLICY_ARN}"

  11. Delete the role by running the following command:

    $ aws iam delete-role --role-name "${ROLE_NAME}"