OADP 1.5 release notes

OADP includes a kubectl command-line interface (CLI) plugin that provides a kubectl native interface for managing backups and restores. Before this update, you aliased to the Velero CLI to perform backup operations. Non-cluster administrators had no way to independently manage their own backups, creating a dependency on cluster admins for routine data protection tasks. As a result, admins can run kubectl oadp or oc oadp commands to create, delete, and retrieve logs for backups and restores. Non-cluster admins can run the oc oadp nonadmin command to independently create, delete, and inspect non-admin backups within their permitted namespaces.

Before this update, Velero restored pods with the k8s.ovn.org/pod-networks annotation preserved from the source cluster. As a consequence, OVN-Kubernetes skipped allocating new network configurations, resulting in stale IP assignments, subnet mismatches, and lost network connectivity for restored pods. With this release, the k8s.ovn.org/pod-networks annotation is automatically stripped during restore operations. As a result, OVN-Kubernetes successfully re-allocates fresh logical ports and IP addresses to restored pods, ensuring proper network connectivity.

Before this update, the DataProtectionApplication (DPA) reconciliation required credentials to be explicitly specified in the DPA custom resource (CR) even when the credentials were provided in the CloudStorage CR. With this release, the DPA controller inherits credentials, configuration values, and region settings from the referenced CloudStorage CR when using backupLocations.bucket.cloudStorageRef. If both DPA-level and CloudStorage-level values are provided, DPA-level values take precedence. As a result, DPA reconciliation no longer fails due to missing credentials in DPA CR.

Before this update, Velero evaluated backup hooks by using the backup’s top-level label selector. As a consequence, pods dynamically added to a backup by plugins by using AdditionalItems were skipped during hook execution if they did not directly match labelSelector. This resulted in crash-consistent or potentially corrupt backups. With this release, plugin adjustments ensure that dynamic pod items properly call configured pre- and post-backup hooks regardless of whether the pod itself carries the primary backup label. As a result, dynamically included pods run backup hooks as expected.

Before this update, the oadp-cli-server workload pods in the openshift-adp namespace used the default service account instead of dedicated, workload-specific service accounts. As a consequence, access control validation checks failed during deployment verification. With this release, dedicated and valid service accounts are properly assigned to these workload pods. As a result, all OADP workload pods comply with access control verification and adhere to cluster security standards.

Before this update, the oadp-cli-server pod was deployed with an empty or default serviceAccountName without properly scoped role bindings. As a consequence, role bindings used by a pod did not reside within the same workload namespace. With this release, the OADP Operator creates a dedicated openshift-adp-cli-server service account with automountServiceAccountToken: false and no cross-namespace RoleBindings. As a result, the OADP CLI server pod runs with a correctly scoped, least-privilege service account in the openshift-adp namespace.

Before this update, when backing up a virtual machine (VM) with an attached NetworkAttachmentDefinition (NAD), the NAD was not included in the backup. With this release, the Velero kubevirt plugin includes NADs in the resource graph (KVGraph) for VM backups. It also correctly updates the namespace references inside the NAD configuration during alternate-namespace restores. As a result, backing up a VM successfully backs up all attached NADs.

Before this update, the OADP command-line interface (CLI) container did not have configured Kubernetes health probes (liveness, readiness, and startup probes). As a consequence, workloads could not automatically recover from common failures, such as pod, host, or network issues. With this release, the OADP Operator adds liveness, readiness, and startup probes to the deployed containers. As a result, workloads can use native Kubernetes health-checking mechanisms to self-recover from failures and ensure reliable application traffic routing.

When you configure CloudStorage on OpenShift clusters on Google Cloud by using Workload Identity Federation (WIF) authentication, the CloudStorage resource fails to detect that a Google Cloud bucket does not exist. As a consequence, the resource returns a BucketCheckError condition and prevents automatic bucket creation.

Before this update, the DataProtectionApplication (DPA) custom resource reported an error due to an empty podConfig with specified nodeAgent affinity. This caused DPA to fail to reconcile. With this release, DPA reports an error only when nodeAgent.podConfig.nodeSelector does not match spec.configuration.nodeAgent.LoadAffinityConfig when podConfig is explicitly set.

Before this update, the Velero DeleteBackupRequest queue got stuck when attempting to delete backups that referenced non-existent Amazon Web Services (AWS) EBS snapshots. This occurred after removing schedules in the granular schedules-per-hosted-cluster model. With this release, backup deletion handles missing CSI snapshots correctly and completes without getting stuck.

Before this update, Windows virtual machine (VM) snapshot operations failed because the guest agent became unresponsive during the freeze operation, causing QuiesceFailed errors. With this release, CSI snapshot early frequent polling is enabled for all snapshots, which resolves the timing issue with the guest agent during the freeze operation.

Before this update, the OADP Operator incorrectly appended default resources to excludedClusterScopedResources in the NonAdminBackup CR when a wildcard (*) was present. This caused FailedValidation errors when excluding all cluster-scoped resources. With this release, the Operator correctly handles the wildcard without appending default resources.

With this release, you can disable or configure the Container Storage Interface (CSI) snapshot early frequent polling interval in the OADP DataProtectionApplication (DPA) custom resource (CR). This allows you to tune snapshot polling behavior to better suit your environment.

Before this update, the controller crashed during image-based upgrade (IBU) due to missing OpenShift Container Platform custom resource definitions (CRDs) before they were fully initialized. As a consequence, this failure delayed DataProtectionApplication (DPA) reconciliation during IBU upgrade by 8 minutes. This release resolves this issue by requiring the controller to wait for OpenShift Container Platform CRDs to load before starting in the IBU environment on single-node OpenShift, while also disabling leader election. This change shortens the DPA reconciliation window and improves the overall upgrade duration for single-node OpenShift clusters.

Simultaneous updates by several controllers or processes to the same NonAdminBackupStorageLocationRequest objects cause resource conflicts during backup creation in OADP self-service. As a consequence, reconciliation attempts fail with object has been modified errors. No known workaround exists.

Before this update, OADP would only process the first custom CA certificate found among all backup storage locations (BSLs) and apply it globally. This behavior prevented multiple BSLs with different CA certificates from working correctly. Additionally, system-trusted certificates were not included, causing failures when connecting to standard services.

The CloudStorage API feature, available as a Technology Preview before this update, is fully supported from OADP 1.5.1. The CloudStorage API automates the creation of a bucket for object storage.

The DataProtectionTest (DPT) is a custom resource (CR) that provides a framework to validate your OADP configuration.

With this update, users can control the maximum number of node agent operations that can run simultaneously on each node within their cluster. It also enables better resource management, optimizing backup and restore workflows for improved performance and a more streamlined experience.

Before this update, the DataProtectionApplicationSpec used deprecated PodAnnotations, which led to an annotation limit overflow. This caused potential misconfigurations in deployments. In this release, we have added PodConfig for annotations in pods deployed by the Operator, ensuring consistent annotations and improved manageability for end users. As a result, deployments should now be more reliable and easier to manage.

Before this update, the manager container of the openshift-adp-controller-manager-* pod was configured to run with a writable root file system. As a consequence, this could allow for tampering with the container’s file system or the writing of foreign executables. With this release, the container’s security context has been updated to set the root file system to read-only while ensuring necessary functions that require write access, such as the Kopia cache, continue to operate correctly. As a result, the container is hardened against potential threats.

Before this update, when a user attempted to create a second non-admin DataProtectionApplication (DPA) on a cluster where one already existed, the new DPA failed to reconcile. With this release, the restriction on Non-Admin Controller installation to one per cluster has been removed. As a result, users can install multiple Non-Admin Controllers across the cluster without encountering errors.

Before this update, using a self-signed certificate for backup images with a storage provider such as Minio resulted in an x509: certificate signed by unknown authority error during the backup process. With this release, certificate validation has been updated to support self-signed certificates in OADP, ensuring successful backups.

Before this update, the velero describe output command omitted the defaultVolumesToFsBackup flag. This affected the visibility of backup configuration details for users. With this release, the velero describe output includes the defaultVolumesToFsBackup flag information, improving the visibility of backup settings.

Before this update, DataProtectionTest (DPT) failed to run when s3Url was secured due to an unverified certificate because the DPT CR lacked the ability to skip or add the caCert in the spec field. As a consequence, data upload failure occurred due to an unverified certificate. With this release, DPT CR has been updated to accept and skip CA cert in spec field, resolving SSL verification errors. As a result, DPT no longer fails when using secured s3Url.

Before this update, adding a second backupLocation with the same name in DataProtectionApplication (DPA) caused OADP to enter an invalid state, leading to Backup and Restore failures due to Velero’s inability to read Secret credentials. As a consequence, Backup and Restore operations failed. With this release, the duplicate backupLocation names in DPA are no longer allowed, preventing Backup and Restore failures. As a result, duplicate backupLocation names are rejected, ensuring seamless data protection.

When you start a restore operation for a backup that was created on an OpenStack platform using the Cinder Container Storage Interface (CSI) driver, the initial backup only succeeds after the source application is manually scaled down. The restore job fails, preventing you from successfully recovering your application’s data and state from the backup. No known workaround exists.

Due to an issue in Velero 1.14 and later, the OADP node-agent only processes the first nodeSelector element within the loadAffinity array. As a consequence, if you define multiple nodeSelector objects, all objects except the first are ignored, potentially causing datamover pods to be scheduled on unexpected nodes during a backup.

The CA certificate is not stored as a file on the running Velero container. As a consequence, the user experience degraded due to missing caCert in Velero container, requiring manual setup and downloads. To work around this problem, manually add cert to the Velero deployment. For instructions, see Using cacert with velero command aliased via velero deployment.

When a Data Protection Application (DPA) is created with the nodeSelector field set in the nodeAgent parameter, Data Mover restore partially fails instead of completing the restore operation. No known workaround exists.

An unverified certificate in the S3 connection during backups with caCert in DataProtectionApplication (DPA) causes the ocp-django application’s backup to partially fail and result in data loss. No known workaround exists.

When the ephemeral-storage parameter is configured and running file system restore, the cache is not automatically deleted from the worker node. As a consequence, the /var partition overflows during backup restore, causing increased storage usage and potential resource exhaustion. To work around this problem, restart the node agent pod, which clears the cache. As a result, cache is deleted.

When performing a volumeSnapshotLocation (VSL) backup on Google Cloud Workload Identity, the Velero Google Cloud plugin creates an invalid API request if the Google Cloud project is also specified in the snapshotLocations configuration of DataProtectionApplication (DPA). As a consequence, the Google Cloud API returns a RESOURCE_PROJECT_INVALID error, and the backup job finishes with a PartiallyFailed status. No known workaround exists.

The volumeSnapshotLocation (VSL) backup fails because of missing the AZURE_RESOURCE_GROUP parameter in the credentials file, even if AZURE_RESOURCE_GROUP is already mentioned in the DataProtectionApplication (DPA) config for VSL. No known workaround exists.

When backing up applications that include image stream resources on an Azure cluster using STS, the OADP plugin incorrectly attempts to locate a secret-based credential for the container registry. As a consequence, the required secret is not found in the STS environment, causing the ImageStream custom backup action to fail. This results in the overall backup status marked as PartiallyFailed. No known workaround exists.

When a user creates a bucket and uses the backupLocations.bucket.cloudStorageRef configuration, bucket credentials are not present in the DataProtectionApplication (DPA) custom resource (CR). As a result, the DPA reconciliation fails even if bucket credentials are present in the CloudStorage CR. To work around this problem, add the same credentials to the backupLocations section of the DPA CR.

With OADP 1.5.0, the configuration.restic specification field has been deprecated. Use the nodeAgent section with the uploaderType field for selecting kopia or restic as a uploaderType. Note that Restic is deprecated in OADP 1.5.0.

OADP 1.5.0 introduces a new feature named OADP Self-Service, enabling namespace admin users to back up and restore applications on the OKD. In the earlier versions of OADP, you needed the cluster-admin role to perform OADP operations such as backing up and restoring an application, creating a backup storage location, and so on.

You can collect logs, and information about OpenShift API for Data Protection (OADP) custom resources by using the must-gather tool. The must-gather data must be attached to all customer cases. This tool generates a Markdown output file with the collected information, which is located in the must-gather logs clusters directory.

The nodeAgent field in Data Protection Application (DPA) now includes the following parameters:

Velero no longer uses the node-agent-config config map for configuring the nodeAgent daemon set. With this update, you must use the new spec.configuration.nodeAgent parameter in a Data Protection Application (DPA) for configuring the nodeAgent daemon set.

With Velero 1.15 and later, you can now configure the total size of a cache per repository. This prevents pods from being removed due to running out of ephemeral storage. See the following new parameters added to the NodeAgentConfig field in DPA:

With this update, the following changes are added:

By default, only one thread processes an item block. Velero 1.16 supports a parallel item backup, where multiple items within a backup can be processed in parallel.

With the of release OADP 1.5.0, the logs are now available in the JSON format. It helps to have pre-parsed data in their Elastic logs management system.

With this release, the oc get dpa command now displays RECONCILED status instead of displaying only NAME and AGE to improve user experience. For example:

$ oc get dpa -n openshift-adp
NAME            RECONCILED   AGE
velero-sample   True         2m51s

With this release, the terminationMessagePolicy field can now set the FallbackToLogsOnError value for the OpenShift API for Data Protection (OADP) Operator containers such as operator-manager, velero, node-agent, and non-admin-controller.

Previously, the namespace admin could not execute an application after the restore operation.

Previously, status restoration was only configured at the resource type using the restoreStatus field in the Restore custom resource (CR).

metadata:
  annotations:
    velero.io/restore-status: "true"

Previously, on performing the backup of an application with the excludedClusterScopedResources field set to storageclasses, Namespace parameter, the backup was successful but the restore partially failed. With this update, the restore is successful.

Previously, a backup was marked as failed with the following error message:

failureReason: found a backup with status "InProgress" during the server starting,
mark it as "Failed"

Previously, when the spec.configuration.velero.disableFsBackup field from a Data Protection Application (DPA) was set to true, the backup partially failed with an error, which was not informative.

Previously, AWS credentials using STS authentication were not properly validated.

Previously, the new configurations for repository maintenance job pod affinity was missing from a DPA specification.

Previously, when a schedule CR was created with a wrong spec.schedule value and the same was later patched with a correct value, the ValidationErrors field still existed. Consequently, the ValidationErrors field was displaying incorrect information even though the spec was correct.

Previously, when a restore operation was triggered with the includedNamespace field in a restore specification, restore operation was completed successfully but no volumeSnapshotContents custom resources (CR) were created and the PVCs were in a Pending status.

Previously, the container was configured with the readOnlyRootFilesystem: true setting for security, but the code attempted to create temporary files in the /tmp directory using the os.CreateTemp() function. Consequently, while using the AWS STS authentication with the Cloud Credential Operator (CCO) flow, OADP failed to create temporary files that were required for AWS credential handling with the following error:

ERROR unable to determine if bucket exists. {"error": "open /tmp/aws-shared-credentials1211864681: read-only file system"}

Even after deleting a backup, Kopia does not delete the volume artifacts from the ${bucket_name}/kopia/${namespace} on the S3 location after the backup expired. Information related to the expired and removed data files remains in the metadata.

With OpenShift API for Data Protection (OADP) 1.5.0, the configuration.restic specification field has been deprecated. Use the nodeAgent section with the uploaderType field for selecting kopia or restic as a uploaderType. Note that Restic is deprecated in OpenShift API for Data Protection (OADP) 1.5.0.

OADP can support and facilitate application migrations within HyperShift hosted OpenShift clusters as a Technology Preview. It ensures a seamless backup and restore operation for applications in hosted clusters.