1. Documentation
×

Troubleshooting

Downloading the Velero CLI tool

You can download and install the Velero CLI tool by following the instructions on the Velero documentation page.

The page includes instructions for:

  • macOS by using Homebrew

  • GitHub

  • Windows by using Chocolatey

Prerequisites

  • You have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.

  • You have installed kubectl locally.

Procedure

  1. Open a browser and navigate to "Install the CLI" on the Verleo website.

  2. Follow the appropriate procedure for macOS, GitHub, or Windows.

  3. Download the Velero version appropriate for your version of OADP, according to the table that follows:

    Table 1. OADP-Velero version relationship
    OADP version Velero version

    0.2.6

    1.6.0

    0.5.5

    1.7.1

    1.0.0

    1.7.1

    1.0.1

    1.7.1

    1.0.2

    1.7.1

    1.0.3

    1.7.1

Accessing the Velero binary in the Velero deployment in the cluster

You can use a shell command to access the Velero binary in the Velero deployment in the cluster.

Prerequisites

  • Your DataProtectionApplication custom resource has a status of Reconcile complete.

Procedure

  • Enter the following command to set the needed alias:

    $ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'

Debugging Velero resources with the OpenShift CLI tool

You can debug a failed backup or restore by checking Velero custom resources (CRs) and the Velero pod log with the OpenShift CLI tool.

Velero CRs

Use the oc describe command to retrieve a summary of warnings and errors associated with a Backup or Restore CR:

$ oc describe <velero_cr> <cr_name>

Velero pod logs

Use the oc logs command to retrieve the Velero pod logs:

$ oc logs pod/<velero>

Velero pod debug logs

You can specify the Velero log level in the DataProtectionApplication resource as shown in the following example.

This option is available starting from OADP 1.0.3.

 
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
spec:
  configuration:
    velero:
      logLevel: warning

The following logLevel values are available:

  • trace

  • debug

  • info

  • warning

  • error

  • fatal

  • panic

It is recommended to use debug for most logs.

Debugging Velero resources with the Velero CLI tool

You can debug Backup and Restore custom resources (CRs) and retrieve logs with the Velero CLI tool.

The Velero CLI tool provides more detailed information than the OpenShift CLI tool.

Syntax

Use the oc exec command to run a Velero CLI command:

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> <command> <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql

Help option

Use the velero --help option to list all Velero CLI commands:

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  --help

Describe command

Use the velero describe command to retrieve a summary of warnings and errors associated with a Backup or Restore CR:

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> describe <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql

Logs command

Use the velero logs command to retrieve the logs of a Backup or Restore CR:

$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  <backup_restore_cr> logs <cr_name>
Example
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
  restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf

Installation issues

You might encounter issues caused by using invalid directories or incorrect credentials when you install the Data Protection Application.

Backup storage contains invalid directories

The Velero pod log displays the error message, Backup storage contains invalid top-level directories.

Cause

The object storage contains top-level directories that are not Velero directories.

Solution

If the object storage is not dedicated to Velero, you must specify a prefix for the bucket by setting the spec.backupLocations.velero.objectStorage.prefix parameter in the DataProtectionApplication manifest.

Incorrect AWS credentials

The oadp-aws-registry pod log displays the error message, InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

The Velero pod log displays the error message, NoCredentialProviders: no valid providers in chain.

Cause

The credentials-velero file used to create the Secret object is incorrectly formatted.

Solution

Ensure that the credentials-velero file is correctly formatted, as in the following example:

Example credentials-velero file
[default] (1)
aws_access_key_id=AKIAIOSFODNN7EXAMPLE (2)
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
1 AWS default profile.
2 Do not enclose the values with quotation marks (", ').

Backup and Restore CR issues

You might encounter these common issues with Backup and Restore custom resources (CRs).

Backup CR cannot retrieve volume

The Backup CR displays the error message, InvalidVolume.NotFound: The volume ‘vol-xxxx’ does not exist.

Cause

The persistent volume (PV) and the snapshot locations are in different regions.

Solution

  1. Edit the value of the spec.snapshotLocations.velero.config.region key in the DataProtectionApplication manifest so that the snapshot location is in the same region as the PV.

  2. Create a new Backup CR.

Backup CR status remains in progress

The status of a Backup CR remains in the InProgress phase and does not complete.

Cause

If a backup is interrupted, it cannot be resumed.

Solution

  1. Retrieve the details of the Backup CR:

    $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
  backup describe <backup>

  2. Delete the Backup CR:

    $ oc delete backup <backup> -n openshift-adp

    You do not need to clean up the backup location because a Backup CR in progress has not uploaded files to object storage.

  3. Create a new Backup CR.

Restic issues

You might encounter these issues when you back up applications with Restic.

Restic permission error for NFS data volumes with root_squash enabled

The Restic pod log displays the error message, controller=pod-volume-backup error="fork/exec/usr/bin/restic: permission denied".

Cause

If your NFS data volumes have root_squash enabled, Restic maps to nfsnobody and does not have permission to create backups.

Solution

You can resolve this issue by creating a supplemental group for Restic and adding the group ID to the DataProtectionApplication manifest:

  1. Create a supplemental group for Restic on the NFS data volume.

  2. Set the setgid bit on the NFS directories so that group ownership is inherited.

  3. Add the spec.configuration.restic.supplementalGroups parameter and the group ID to the DataProtectionApplication manifest, as in the following example:

    spec:
  configuration:
    restic:
      enable: true
      supplementalGroups:
      - <group_id> (1)
    1 Specify the supplemental group ID.

  4. Wait for the Restic pods to restart so that the changes are applied.

Restore CR of Restic backup is "PartiallyFailed", "Failed", or remains "InProgress"

The Restore CR of a Restic backup completes with a PartiallyFailed or Failed status or it remains InProgress and does not complete.

If the status is PartiallyFailed or Failed, the Velero pod log displays the error message, level=error msg="unable to successfully complete restic restores of pod’s volumes".

If the status is InProgress, the Restore CR logs are unavailable and no errors appear in the Restic pod logs.

Cause

The DeploymentConfig object redeploys the Restore pod, causing the Restore CR to fail.

Solution

  1. Create a Restore CR that excludes the ReplicationController, DeploymentConfig, and TemplateInstances resources:

    $ velero restore create --from-backup=<backup> -n openshift-adp \ (1)
  --include-namespaces <namespace> \ (2)
  --exclude-resources replicationcontroller,deploymentconfig,templateinstances.template.openshift.io \
  --restore-volumes=true
    1 Specify the name of the Backup CR.
    2 Specify the include-namespaces in the Backup CR.

  2. Verify that the status of the Restore CR is Completed:

    $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

  3. Create a Restore CR that includes the ReplicationController and DeploymentConfig resources:

    $ velero restore create --from-backup=<backup> -n openshift-adp \
  --include-namespaces <namespace> \
  --include-resources replicationcontroller,deploymentconfig \
  --restore-volumes=true

  4. Verify that the status of the Restore CR is Completed:

    $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

  5. Verify that the backup resources have been restored:

    $ oc get all -n <namespace>

Restic Backup CR cannot be recreated after bucket is emptied

If you create a Restic Backup CR for a namespace, empty the S3 bucket, and then recreate the Backup CR for the same namespace, the recreated Backup CR fails.

The velero pod log displays the error message, msg="Error checking repository for stale locks".

Cause

Velero does not create the Restic repository from the ResticRepository manifest if the Restic directories are deleted on object storage. See (Velero issue 4421) for details.

Using the must-gather tool

You can collect logs, metrics, and information about OADP custom resources by using the must-gather tool.

The must-gather data must be attached to all customer cases.

You can run the must-gather tool with the following data collection options:

  • Full must-gather data collection collects Prometheus metrics, pod logs, and Velero CR information for all namespaces where the OADP Operator is installed.

  • Essential must-gather data collection collects pod logs and Velero CR information for a specific duration of time, for example, one hour or 24 hours. Prometheus metrics and duplicate logs are not included.

  • must-gather data collection with timeout. Data collection can take a long time if there are many failed Backup CRs. You can improve performance by setting a timeout value.

  • Prometheus metrics data dump downloads an archive file containing the metrics data collected by Prometheus.

Prerequisites

  • You must be logged in to the OKD cluster as a user with the cluster-admin role.

  • You must have the OpenShift CLI installed.

Procedure

  1. Navigate to the directory where you want to store the must-gather data.

  2. Run the oc adm must-gather command for one of the following data collection options:

    • Full must-gather data collection, including Prometheus metrics:

      $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0

      The data is saved as must-gather/must-gather.tar.gz. You can upload this file to a support case on the Red Hat Customer Portal.

    • Essential must-gather data collection, without Prometheus metrics, for a specific time duration:

      $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \
  -- /usr/bin/gather_<time>_essential (1)
      1 Specify the time in hours. Allowed values are 1h, 6h, 24h, 72h, or all, for example, gather_1h_essential or gather_all_essential.

    • must-gather data collection with timeout:

      $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \
  -- /usr/bin/gather_with_timeout <timeout> (1)
      1 Specify a timeout value in seconds.

    • Prometheus metrics data dump:

      $ oc adm must-gather --image=registry.redhat.io/oadp/oadp-mustgather-rhel8:v1.0 \
  -- /usr/bin/gather_metrics_dump

      This operation can take a long time. The data is saved as must-gather/metrics/prom_data.tar.gz.

Viewing metrics data with the Prometheus console

You can view the metrics data with the Prometheus console.

Procedure

  1. Decompress the prom_data.tar.gz file:

    $ tar -xvzf must-gather/metrics/prom_data.tar.gz

  2. Create a local Prometheus instance:

    $ make prometheus-run

    The command outputs the Prometheus URL.

    Output
    Started Prometheus on http://localhost:9090

  3. Launch a web browser and navigate to the URL to view the data by using the Prometheus web console.

  4. After you have viewed the data, delete the Prometheus instance and data:

    $ make prometheus-cleanup