1. Documentation
×

Restoring 3scale API Management by using OADP

Restore Red Hat 3scale API Management components by restoring the backed up 3scale operator resources, MySQL database, and Redis database. This helps you to recover your 3scale deployment and resume API management services.

After the data has been restored, you can scale up the 3scale operator and deployment.

Restoring the 3scale API Management operator, secrets, and APIManager

Restore the Red Hat 3scale API Management operator resources, including the Secret and APIManager custom resources (CRs). This helps you to recover your 3scale operator configuration on the same or a different cluster.

Prerequisites

  • You backed up the 3scale operator.

  • You backed up the MySQL and Redis databases.

  • You are restoring the database on the same cluster, where it was backed up.

    If you are restoring the operator to a different cluster that you backed up from, install and configure OADP with nodeAgent enabled on the destination cluster. Ensure that the OADP configuration is same as it was on the source cluster.

Procedure

  1. Delete the 3scale operator custom resource definitions (CRDs) along with the threescale namespace by running the following command:

    $ oc delete project threescale
    "threescale" project deleted successfully

  2. Create a YAML file with the following configuration to restore the 3scale operator:

    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-installation-restore
  namespace: openshift-adp
spec:
  backupName: operator-install-backup
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s

    where:

    operator-install-backup

    Specifies the name of the backup to restore the 3scale operator.

  3. Restore the 3scale operator by running the following command:

    $ oc create -f restore.yaml
    restore.velerio.io/operator-installation-restore created

  4. Manually create the s3-credentials Secret object by running the following command:

    $ oc apply -f - <<EOF
---
apiVersion: v1
kind: Secret
metadata:
      name: s3-credentials
      namespace: threescale
stringData:
  AWS_ACCESS_KEY_ID: <ID_123456>
  AWS_SECRET_ACCESS_KEY: <ID_98765544>
  AWS_BUCKET: <mybucket.example.com>
  AWS_REGION: <us-east-1>
type: Opaque
EOF

    where:

    <AWS_ACCESS_KEY_ID>

    Specifies your AWS credentials ID.

    <AWS_SECRET_ACCESS_KEY>

    Specifies your AWS credentials KEY.

    <mybucket.example.com>

    Specifies your target bucket name.

    <us-east-1>

    Specifies the AWS region of your bucket.

  5. Scale down the 3scale operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=0 -n threescale
    deployment.apps/threescale-operator-controller-manager-v2 scaled

  6. Create a YAML file with the following configuration to restore the Secret:

    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-resources-secrets
  namespace: openshift-adp
spec:
  backupName: operator-resources-secrets
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s

    where:

    operator-resources-secrets

    Specifies the name of the backup to restore the Secret.

  7. Restore the Secret by running the following command:

    $ oc create -f restore-secrets.yaml
    restore.velerio.io/operator-resources-secrets created

  8. Create a YAML file with the following configuration to restore APIManager:

    Example restore-apimanager.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: operator-resources-apim
  namespace: openshift-adp
spec:
  backupName: operator-resources-apim
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 4h0m0s

    where:

    operator-resources-apim

    Specifies the name of the backup to restore the APIManager.

    excludedResources

    Specifies the resources that you do not want to restore.

  9. Restore the APIManager by running the following command:

    $ oc create -f restore-apimanager.yaml
    restore.velerio.io/operator-resources-apim created

  10. Scale up the 3scale operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=1 -n threescale
    deployment.apps/threescale-operator-controller-manager-v2 scaled

Restoring a MySQL database

Restore a MySQL database by scaling down Red Hat 3scale API Management components and creating a Velero Restore custom resource (CR). This helps you to recover your 3scale MySQL data, persistent volumes, and associated resources.

Do not delete the default PV and PVC associated with the database. If you do, your backups are deleted.
Prerequisites

  • You restored the Secret and APIManager custom resources (CRs).

Procedure

  1. Scale down the Red Hat 3scale API Management operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=0 -n threescale
    Example output
    deployment.apps/threescale-operator-controller-manager-v2 scaled

  2. Create the following script to scale down the 3scale operator:

    $ vi ./scaledowndeployment.sh
    Example script:
    for deployment in apicast-production apicast-staging backend-cron backend-listener backend-redis backend-worker system-app system-memcache system-mysql system-redis system-searchd system-sidekiq zync zync-database zync-que; do
    oc scale deployment/$deployment --replicas=0 -n threescale
done

  3. Scale down all the deployment 3scale components by running the following script:

    $ ./scaledowndeployment.sh
    Example output
    deployment.apps.openshift.io/apicast-production scaled
deployment.apps.openshift.io/apicast-staging scaled
deployment.apps.openshift.io/backend-cron scaled
deployment.apps.openshift.io/backend-listener scaled
deployment.apps.openshift.io/backend-redis scaled
deployment.apps.openshift.io/backend-worker scaled
deployment.apps.openshift.io/system-app scaled
deployment.apps.openshift.io/system-memcache scaled
deployment.apps.openshift.io/system-mysql scaled
deployment.apps.openshift.io/system-redis scaled
deployment.apps.openshift.io/system-searchd scaled
deployment.apps.openshift.io/system-sidekiq scaled
deployment.apps.openshift.io/zync scaled
deployment.apps.openshift.io/zync-database scaled
deployment.apps.openshift.io/zync-que scaled

  4. Delete the system-mysql Deployment object by running the following command:

    $ oc delete deployment system-mysql -n threescale
    Example output
    Warning: apps.openshift.io/v1 deployment is deprecated in v4.14+, unavailable in v4.10000+
deployment.apps.openshift.io "system-mysql" deleted

  5. Create the following YAML file to restore the MySQL database:

    Example restore-mysql.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-mysql
  namespace: openshift-adp
spec:
  backupName: mysql-backup
  excludedResources:
    - nodes
    - events
    - events.events.k8s.io
    - backups.velero.io
    - restores.velero.io
    - csinodes.storage.k8s.io
    - volumeattachments.storage.k8s.io
    - backuprepositories.velero.io
    - resticrepositories.velero.io
  hooks:
    resources:
      - name: restoreDB
        postHooks:
          - exec:
              command:
                - /bin/sh
                - '-c'
                - >
                  sleep 30

                  mysql -h 127.0.0.1 -D system -u root
                  --password=$MYSQL_ROOT_PASSWORD <
                  /var/lib/mysqldump/data/dump.sql
              container: system-mysql
              execTimeout: 80s
              onError: Fail
              waitTimeout: 5m
  itemOperationTimeout: 1h0m0s
  restorePVs: true

    where:

    mysql-backup

    Specifies the name of the MySQL backup to restore.

    /var/lib/mysqldump/data/dump.sql

    Specifies the path where the data is restored from.

  6. Restore the MySQL database by running the following command:

    $ oc create -f restore-mysql.yaml
    restore.velerio.io/restore-mysql created
Verification

  1. Verify that the PodVolumeRestore restore is completed by running the following command:

    $ oc get podvolumerestores.velero.io -n openshift-adp
    Example output
    NAME                    NAMESPACE    POD                     UPLOADER TYPE   VOLUME                  STATUS      TOTALBYTES   BYTESDONE   AGE
restore-mysql-rbzvm     threescale   system-mysql-2-kjkhl    kopia           mysql-storage           Completed   771879108    771879108   40m
restore-mysql-z7x7l     threescale   system-mysql-2-kjkhl    kopia           example-claim           Completed   380415       380415      40m

  2. Verify that the additional PVC has been restored by running the following command:

    $ oc get pvc -n threescale
    Example output
    NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
backend-redis-storage   Bound    pvc-3dca410d-3b9f-49d4-aebf-75f47152e09d   1Gi        RWO            gp3-csi        <unset>                 68m
example-claim           Bound    pvc-cbaa49b0-06cd-4b1a-9e90-0ef755c67a54   1Gi        RWO            gp3-csi        <unset>                 57m
mysql-storage           Bound    pvc-4549649f-b9ad-44f7-8f67-dd6b9dbb3896   1Gi        RWO            gp3-csi        <unset>                 68m
system-redis-storage    Bound    pvc-04dadafd-8a3e-4d00-8381-6041800a24fc   1Gi        RWO            gp3-csi        <unset>                 68m
system-searchd          Bound    pvc-afbf606c-d4a8-4041-8ec6-54c5baf1a3b9   1Gi        RWO            gp3-csi        <unset>                 68m

Restoring the back-end Redis database

Restore the back-end Redis database by creating a Restore custom resource (CR) that excludes non-essential cluster resources. This helps you to recover the Redis data store as part of the Red Hat 3scale API Management restoration process.

Prerequisites

  • You restored the Red Hat 3scale API Management operator resources, Secret, and APIManager custom resources.

  • You restored the MySQL database.

Procedure

  1. Delete the backend-redis deployment by running the following command:

    $ oc delete deployment backend-redis -n threescale
    Warning: apps.openshift.io/v1 deployment is deprecated in v4.14+, unavailable in v4.10000+

deployment.apps.openshift.io "backend-redis" deleted

  2. Create a YAML file with the following configuration to restore the Redis database:

    Example restore-backend.yaml file
    apiVersion: velero.io/v1
kind: Restore
metadata:
  name: restore-backend
  namespace: openshift-adp
spec:
  backupName: redis-backup
  excludedResources:
    - nodes
    - events
    - events.events.k8s.io
    - backups.velero.io
    - restores.velero.io
    - resticrepositories.velero.io
    - csinodes.storage.k8s.io
    - volumeattachments.storage.k8s.io
    - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  restorePVs: true

    where:

    redis-backup

    Specifies the name of the Redis backup to restore.

  3. Restore the Redis database by running the following command:

    $ oc create -f restore-backend.yaml
    restore.velerio.io/restore-backend created
Verification

  • Verify that the PodVolumeRestore restore is completed by running the following command:

    $ oc get podvolumerestores.velero.io -n openshift-adp
    NAME                    NAMESPACE    POD                     UPLOADER TYPE   VOLUME                  STATUS      TOTALBYTES   BYTESDONE   AGE
restore-backend-jmrwx   threescale   backend-redis-1-bsfmv   kopia           backend-redis-storage   Completed   76123        76123       21m

Scaling up the 3scale API Management operator and deployment

Scale up the Red Hat 3scale API Management operator and any deployment that was manually scaled down during the restore process. This helps you to bring your 3scale installation back to a fully functional state matching the backed-up configuration.

Prerequisites

  • You restored the 3scale operator resources, and both the Secret and APIManager custom resources (CRs).

  • You restored the MySQL and back-end Redis databases.

  • Ensure that there are no scaled up deployments or no extra pods running. There might be some system-mysql or backend-redis pods running detached from deployments after restoration, which can be removed after the restoration is successful.

Procedure

  1. Scale up the 3scale operator by running the following command:

    $ oc scale deployment threescale-operator-controller-manager-v2 --replicas=1 -n threescale
    Example output
    deployment.apps/threescale-operator-controller-manager-v2 scaled

  2. Ensure that the 3scale pod is running to verify if the 3scale operator was deployed by running the following command:

    $ oc get pods -n threescale
    Example output
    NAME									                    READY        STATUS	  RESTARTS	 AGE
threescale-operator-controller-manager-v2-79546bd8c-b4qbh	1/1	         Running  0          2m5s

  3. Create the following script to scale up the deployments:

    $ vi ./scaledeployment.sh
    Example script file:
    for deployment in apicast-production apicast-staging backend-cron backend-listener backend-redis backend-worker system-app system-memcache system-mysql system-redis system-searchd system-sidekiq zync zync-database zync-que; do
    oc scale deployment/$deployment --replicas=1 -n threescale
done

  4. Scale up the deployments by running the following script:

    $ ./scaledeployment.sh
    Example output
    deployment.apps.openshift.io/apicast-production scaled
deployment.apps.openshift.io/apicast-staging scaled
deployment.apps.openshift.io/backend-cron scaled
deployment.apps.openshift.io/backend-listener scaled
deployment.apps.openshift.io/backend-redis scaled
deployment.apps.openshift.io/backend-worker scaled
deployment.apps.openshift.io/system-app scaled
deployment.apps.openshift.io/system-memcache scaled
deployment.apps.openshift.io/system-mysql scaled
deployment.apps.openshift.io/system-redis scaled
deployment.apps.openshift.io/system-searchd scaled
deployment.apps.openshift.io/system-sidekiq scaled
deployment.apps.openshift.io/zync scaled
deployment.apps.openshift.io/zync-database scaled
deployment.apps.openshift.io/zync-que scaled

  5. Get the 3scale-admin route to log in to the 3scale UI by running the following command:

    $ oc get routes -n threescale
    Example output
    NAME                         HOST/PORT                                                                   PATH   SERVICES             PORT      TERMINATION     WILDCARD
backend                      backend-3scale.apps.custom-cluster-name.openshift.com                         backend-listener     http      edge/Allow      None
zync-3scale-api-b4l4d        api-3scale-apicast-production.apps.custom-cluster-name.openshift.com          apicast-production   gateway   edge/Redirect   None
zync-3scale-api-b6sns        api-3scale-apicast-staging.apps.custom-cluster-name.openshift.com             apicast-staging      gateway   edge/Redirect   None
zync-3scale-master-7sc4j     master.apps.custom-cluster-name.openshift.com                                 system-master        http      edge/Redirect   None
zync-3scale-provider-7r2nm   3scale-admin.apps.custom-cluster-name.openshift.com                           system-provider      http      edge/Redirect   None
zync-3scale-provider-mjxlb   3scale.apps.custom-cluster-name.openshift.com                                 system-developer     http      edge/Redirect   None

    In this example, 3scale-admin.apps.custom-cluster-name.openshift.com is the 3scale-admin URL.

  6. Use the URL from this output to log in to the 3scale operator as an administrator. You can verify that the data, when you took backup, is available.