OADP 1.6 release notes

OADP supports parallel backup processing. Before this update, OADP processed backups sequentially where each backup had to complete before the next one could begin. This created bottlenecks for environments with many backup schedules or large-scale data protection needs, increasing the overall time required to complete all backup operations. With this release, multiple backups can run in parallel if they do not share namespaces. As a result, you can now define more granular or frequent backup schedules without worrying about queuing delays.

OADP introduces the virtual machine file restore (VMFR) feature. Before this release, restoring even a single file from a KubeVirt VM backup required restoring the entire VM, which was time-consuming and resource-intensive. With this release, you can selectively recover individual files from KubeVirt VM backups directly within OKD. VMFR also includes RestoreItemAction, a plugin for collision prevention in multi-backup scenarios. This plugin renames PVCs during restore, ensuring unique names and avoiding conflicts, thereby improving the reliability and efficiency of VMFR backups and restores.

OADP introduces the virtual machine data protection (VMDP) feature, a command-line tool that runs inside virtual machines (VMs). Before this update, OADP supported only snapshot-based backup of entire VMs, with no option for granular, user-driven file-level data protection from within a VM. Using VMDP, you can selectively back up and restore individual files and directories by using a Kopia-based client/server architecture without cluster administrator intervention.

OADP includes a kubectl command-line interface (CLI) plugin that provides a kubectl native interface for managing backups and restores. Before this update, you aliased to the Velero CLI to perform backup operations. Non-cluster administrators had no way to independently manage their own backups, creating a dependency on cluster admins for routine data protection tasks. As a result, admins can run kubectl oadp commands to create, delete, and retrieve logs for backups and restores. Non-cluster admins can independently create, delete, and inspect non-admin backups within their permitted namespaces.

With this enhancement, the oadp-must-gather tool collects the VirtualMachineBackupsDiscovery and VirtualMachineFileRestore custom resource definitions (CRDs) introduced in OADP 1.6 for the virtual machine file-level restore feature. As a result, when you run the oc adm must-gather command, both new resource types are automatically collected cluster-wide, including CRDs, per-namespace CR YAMLs, and dedicated sections in the markdown summary. This gives support teams and administrators full visibility into VM file restore operations.

OADP supports backup and restore of custom security context constraints (SCC) with ServiceAccounts through ClusterRole and ClusterRoleBinding resources, not only those listed in the SCC user fields. This enhancement simplifies the backup and restore process of custom SCCs, as it no longer relies on the user field of the SCC. Instead, it uses the location of the SCC used by a pod and the service account associated with the pod. This results in a more reliable and secure backup and restore process for OKD clusters, essential for IBM Cloud Pak for Data.

With this update, you can bypass unbound persistent volume claims (PVCs) during backups. This enhancement improves backup success rates, especially when PVCs are in a Pending state. Additionally, it expands the capabilities of volume policy actions, offering greater control and flexibility in managing backups.

With the migration to Go 1.25, the Data Mover defaults the MaxParallelFileReads value to the cgroup CPU bandwidth limit corresponding to the container CPU limit rather than the total number of CPU cores on the node. Before this update, MaxParallelFileReads used the full CPU count of the node unless datamoverConfig.ParallelFilesUpload was explicitly set in the node-agent-config map, which could lead to overuse of resources in constrained container environments. With this update, Data Mover operations automatically respect container CPU limits, improving resource efficiency and stability without requiring manual configuration changes.

Before this update, maintenance job pods did not support custom labels, which made it difficult to apply namespace-level network policies or firewall rules that rely on pod labels. With this release, you can configure custom podLabels and podAnnotations for repository maintenance jobs through the repositoryMaintenance section of the DataProtectionApplication custom resource.

With this update, you can monitor maintenance job success and failures more efficiently using Prometheus metrics. This enhancement replaces the current method of tracking job success by using the --keep-latest-maintenance-jobs argument. The new solution provides a unified approach for monitoring various aspects of the backup process, using existing tools for alerts and notifications. This improvement reduces the risk of performance degradation due to job failures and helps you set up alerts for maintenance job issues, improving overall backup management and system health.

Before this update, ConfigMap validation failures did not return meaningful errors, making it difficult to diagnose and resolve configuration issues. With this release, the Velero command-line interface (CLI) displays detailed error messages when ConfigMap validation fails for backup repository, node agent, and backup repository maintenance job configurations. As a result, you can quickly identify and fix invalid ConfigMap configurations with actionable error messages.

The Data Mover expose diagnostic includes events from backup pods, restore pods, backup persistent volume claims (PVCs), and restore PVCs for both CSI snapshot data movement and file system backup. Before this update, informative messages from these resources were only available in their individual events and were not captured in the diagnostic output. As a result, you can get a more complete picture of expose failures directly from the diagnostic, without manually inspecting events on each related resource.

The Container Storage Interface (CSI) backup and restore process no longer includes or requires the VolumeSnapshotClass resource. Before this update, the VolumeSnapshotClass was included in CSI backups and needed to be present during restore, adding an unnecessary dependency. With this update, CSI snapshot-based backups and restores complete successfully without VolumeSnapshotClass in the cluster, simplifying the backup resource footprint and improving restore reliability in environments where the VolumeSnapshotClass might not be pre-configured.

This release introduces configurable repository cache storage locations for the node agent Data Mover and pod volume restore operations. Before this update, backup engines such as Kopia stored metadata and snapshot caches implicitly within the ephemeral root file system of the pod. This often caused host node disk exhaustion, DiskPressure taints, and unexpected node evictions during heavy or concurrent operations.

Repository maintenance jobs inherit tolerations defined in the DataProtectionApplication configuration. Before this update, maintenance pods did not inherit tolerations from the Velero deployment, which caused scheduling failures in clusters where nodes have taints configured. As a result, repository maintenance jobs can be scheduled correctly on tainted nodes, ensuring reliable backup repository maintenance in environments that use node taints and tolerations.

You can enable automatic space reclamation on restored persistent volume claims (PVCs) by setting the oadp.openshift.io/reclaim-space-schedule annotation on the Restore custom resource. Before this update, after a Data Mover restore on a Red Hat OpenShift Data Foundation (ODF) cluster, you had to manually run rbd sparsify to reclaim unused block-device space. As a result, the PVC restore plugin propagates the schedule annotation to each restored PVC, and the Container Storage Interface controller automatically creates a ReclaimSpaceCronJob that handles sparsification without manual intervention. This annotation is opt-in and has no effect on non-ODF clusters.

Velero scheduled backups no longer accumulate in a New state queue when backup processing is blocked for extended periods, for example, during long-running backups, Velero server downtime, or infrastructure issues. Before this update, the schedule controller prevented new backup execution if any backup from that schedule was in a New or InProgress state, which led to indefinite queuing and backup accumulation. As a result, unreconciled backups, including those with an empty phase, are properly handled by the scheduler, preventing it from creating excessive new backup runs and ensuring reliable scheduled backup behavior.

With this update, you can add annotations to Velero and node agent pods in OADP. This enhancement provides control over cluster-autoscaler behavior, preventing data operations from failing due to the interaction with the autoscaler. By adding annotations, users can mitigate an issue with OADP 1.5 and earlier, where data uploads and downloads can be canceled on node agent startup. This improvement ensures smoother operation and reduced downtime, and provides flexibility to customize the application according to specific requirements.

You can use wildcard patterns when specifying namespaces in backup operations. Before this update, you had to explicitly list every namespace in the backup object and update it each time a new namespace was created. With this release, you can use patterns such as *-helm in the --include-namespaces flag to dynamically match multiple namespaces, simplifying backup management and reducing manual configuration overhead.

With this update, you can secure your backup data by using server side encryption with customer-provided keys (SSE-C) in OADP. This enhancement improves overall data protection and privacy. The feature is compatible with AWS S3 buckets. Additionally, it provides a workaround for using S3-compatible storage.

With this update, you can configure the priorityClassName field for node agent and Velero pods in a DataProtectionApplication object, prioritizing these pods during resource contention. This is particularly useful after worker node outages, ensuring critical pods are scheduled first.

Before this update, a performance regression in the resource discovery phase caused Velero to issue an excessive number of API list calls when a backup combined includedNamespaces: '*', includeClusterResources: true, labelSelector, and a large excludedResources list. As a consequence, backups that completed before in seconds took about 18 minutes to back up as few as 8 resources. With this release, the resource discovery logic is optimized so that unnecessary API list calls are eliminated during the item collection phase. As a result, backup performance is restored to expected levels, completing in seconds.

Before this update, a Velero change introduced 1-second Container Storage Interface (CSI) snapshot readiness polling for the first 10 seconds of every backup, designed specifically for Microsoft Volume Shadow Copy Service (VSS) workloads on Windows. OADP did not expose this behavior as a configurable option in the DataProtectionApplication (DPA) custom resource (CR). As a consequence, all CSI-enabled backups used the earlier polling behavior regardless of whether VSS compatibility was needed, and you had no way to disable or control this behavior through OADP. This release adds a new enableCSISnapshotEarlyFrequentPolling field under configuration.velero in the DPA CR. By default, this feature is disabled. If your workloads require this frequent polling behavior, you must enable it.

Before this update, the controller-manager entered a CrashLoopBackOff loop due to unconditional interaction with a missing ConsoleCLIDownload custom resource definition (CRD) on clusters without Console capability. As a consequence, you experienced pod crashes on single-node OpenShift clusters due to the missing ConsoleCLIDownload CRD. With this release, the OADP Operator skips CLI download setup when the Console CRD is not available. As a result, the controller-manager no longer enters a CrashLoopBackOff loop on single-node OpenShift clusters, improving its stability.

Before this update, DataProtectionApplication (DPA) custom resource reported an error due to an empty podConfig with specified nodeAgent affinity. This caused DPA to fail to reconcile. With this release, DPA reports error only when nodeAgent.podConfig.nodeSelector does not match spec.configuration.nodeAgent.LoadAffinityConfig when podConfig is explicitly set.

Before this update, the Queued and ReadyToStart backup phases in nonAdminController caused reconcile issues due to their unsupported status. With this release, the Queued and ReadyToStart no longer cause reconcile issues in nonAdminController.

Before this update, restoring a VolumeGroupSnapshot (VGS) backed up from Ceph RBD storage failed with a cannot find the group snapshot for VolumeSnapshot error message. This error occurred because Velero deleted the VolumeGroupSnapshotContent after backup, leaving no matching resource during restore. Also the VolumeSnapshotClassName was deleted from restored VolumeSnapshotContent, preventing the Container Storage Interface (CSI) controller from authenticating to verify snapshots. With this update, Velero preserves the group snapshot handle as an annotation during backup, recreates a stub VolumeGroupSnapshotContent during restores, and restores the correct VolumeSnapshotClass reference. As a result, VGS restores with Ceph RBD complete successfully. Note that VGS is a Technology Preview feature.

Before this update, the ServiceAccount backup plugin failed with a PartiallyFailed error due to unsynchronized access to the security context constraints (SCC) map. As a consequence, backup failures occurred, affecting data integrity in multiple namespaces. With this release, the SCC map is synchronized in the ServiceAccount plugin, preventing its crash during backup execution.

The PrepareQueueLength field is a configuration setting for the node agent that limits the number of intermediate resources created during backup and restore processes. Before this update, PrepareQueueLength was missing in the LoadConcurrency field in the DataProtectionApplication (DPA) configuration. As a consequence, you encountered an error while configuring PrepareQueueLength, causing the configuration to be removed. With this release, the PrepareQueueLength field is added to LoadConcurrency in the DPA configuration.

Before this update, the BackupRepository name was used directly as a Kubernetes label value for repository maintenance jobs. As a consequence, when the BackupRepository name exceeded the 63-character Kubernetes label limit, the maintenance job failed to build, preventing scheduled repository maintenance from running. With this release, the BackupRepository name is hashed before being used as a label value, ensuring it always fits within the 63-character limit. As a result, repository maintenance jobs are created and completed successfully regardless of the BackupRepository name length.

Before this update, restores were marked as completed prematurely before all PodVolumeRestores. With this release, restores complete only after PodVolumeRestores.

Before this update, when any node agent pod restarted, Data Mover canceled all DataUploads in Accepted status across all node agent pods, regardless of which pod was handling them. As a consequence, backup operations failed and the backup queue could be blocked for extended periods even when the remaining node agent pods were healthy. With this release, the controller only cancels DataUploads that were accepted by the specific node agent pod that restarted, leaving DataUploads on other nodes unaffected. As a result, backups handled by healthy node agent pods complete successfully and the backup queue is no longer blocked by an unrelated pod restart.

Before this update, the volume policy check required scanning every pod in the namespace for every individual persistent volume claim (PVC). As a consequence, backup performance degraded significantly in clusters with large numbers of pods and PVCs. With this release, a PVC-to-Pod cache is added to eliminate redundant pod listings and reduce iteration overhead. As a result, all PV snapshot checks complete within the same second, improving performance for backup operations that were taking excessively long.

Before this update, the DataProtectionApplication (DPA) custom resource did not support custom labels or annotations, and it was not possible to propagate them to child resources managed by the OADP Operator. As a consequence, it was not possible to exclude these child resources from external reconciliation tools at the DPA level, causing unnecessary reconciliation loops and high CPU usage. With this release, the DPA includes resourceLabels and resourceAnnotations fields that propagate user-defined labels and annotations to all Operator-managed resources. As a result, you can control reconciliation behavior for all managed resources from a single configuration point, reducing unnecessary processing and CPU usage.

Before this update, Velero did not provide an option to run file system backup in a privileged security context. As a consequence, file system backups failed with permission errors on OpenShift clusters that require privileged security context to access mount paths. With this release, a new --privileged-fs-backup install and node agent server option was added, allowing file system backup pods to run in privileged mode independently of the existing node agent privileged flag. As a result, file system backups complete successfully on clusters that require privileged pod security context.

Before this update, pod network annotations injected by OVN-K and Multus container network interface (CNI) were restored during backup. As a consequence, this caused conflicts with the CNI and required manual intervention. With this release, OADP removes OVN-K and Multus CNI annotations during pod restore.

Before this update, creating a backup with Kopia generated a BackupRepository resource that contained a resticIdentifier field referencing a Restic path instead of the actual Kopia repository path. As a consequence, this caused user confusion during troubleshooting. With this release, the resticIdentifier field is removed from Kopia-type BackupRepository resources, as it was only relevant for Restic repositories.

Before this update, the AWS bucket region was missing in the DataProtectionApplication configuration when using S3 BackupStorageLocation (BSL) with snapshotLocations. As a consequence, backups failed due to missing region in AWS S3 bucket. With this release, AWS S3 bucket regions are automatically detected in BSL.

Before this update, when a DataProtectionApplication was configured with a non-existing Azure storage bucket, the BackupStorageLocation (BSL) validation returned a raw HTTP response in the status.message field. As a consequence, the BSL status was unclear. With this release, the BSL status.message field returns a concise, human-readable error such as ContainerNotFound: The specified container does not exist.

Before this update, the node agent loadAffinity setting was implemented only for backup operations, not for restore operations. As a consequence, when nodeSelector was configured in the DataProtectionApplication specification, the Data Mover pod could be scheduled to a node without a running node agent during a restore. This caused the restore to hang indefinitely. With this release, the loadAffinity configuration is applied to both backup and restore operations. As a result, Data Mover restore pods are correctly scheduled on nodes with running node agents, preventing restore hangs and PartiallyFailed errors.

Before this update, during file system restore, the /var partition on worker nodes increased because the cache was not deleted after backup completion. Only a node agent restart cleared the cache. As a consequence, unintended disk space consumption occurred on worker nodes after file system restore. With this release, the cache for Kopia operation stored in ephemeral-storage is cleared after restore operation.

Before this update, when a DataProtectionApplication (DPA) custom resource was created with the nodeSelector field set in the nodeAgent parameter, nodeSelector was not properly handled for the Data Mover restore action. As a consequence, Data Mover restore actions partially failed because they were scheduled on a wrong node. With this release, nodeSelector is correctly handled for Data Mover restore actions. As a result, restore operations complete successfully when the field is configured in the DPA.

Before this update, running Velero containers did not store the certificate authority (CA) certificate as a file. As a consequence, you had to manually pass the --cacert flag with a path to a CA certificate file on every Velero command-line interface (CLI) command, or download the Velero binary and CA certificate bundle to your workstation. This made the experience inefficient in environments with custom or self-signed certificates.

Before this update, Velero did not check whether the restore-wait init container was necessary during a restore operation. As a consequence, when a namespace was backed up with CSI or native Data Mover and then was restored, Velero incorrectly injected the restore-wait init container into the application pod spec. With this release, Velero skips restoring the restore-wait init container when it is not required by the backup method used.

Before this update, the dpa.spec.configuration.velero.defaultVolumesToFSBackup flag used different casing than the backup.spec.defaultVolumesToFsBackup flag. As a consequence, the inconsistency between the two flags could cause confusion when configuring pod volume file system backup settings. With this release, the two flags are identical.

Before this update, the OADP command-line interface (CLI) did not inherit the certificate authority (CA) certificate from the BackupStorageLocation (BSL) when retrieving backup logs. As a consequence, you were unable to retrieve backup logs by using oc oadp backup logs, even though the backup itself completed successfully and the DataProtectionApplication (DPA) custom resource was correctly configured with a valid CA certificate. With this release, the OADP CLI automatically uses the CA certificate from the BSL when retrieving backup logs. As a result, backup log retrieval works correctly when a valid CA certificate is present in the BSL.

Before this update, during a Container Storage Interface (CSI) backup, Velero added a velero.io/volume-snapshot-name label to the persistent volume claim (PVC), and this label was not removed when next file system backup and restore was performed. With this release, PVCs no longer contain leftover velero.io/volume-snapshot-name labels after a successful file system restore.

Before this update, when the backup or restore CR failed, the associated PodVolumeBackup and PodVolumeRestore custom resources (CRs) continued their data transfer. As a consequence, this led to an inconsistent state where individual volume CRs reported Completed while the parent operation was Failed. With this release, PodVolumeBackup and PodVolumeRestore CRs are properly canceled and marked as Failed when the parent backup or restore CR fails. As a result, volume-level CR statuses are consistent with the parent operation status.

The CloudStorage resource fails to detect that a Google Cloud bucket does not exist and returns a BucketCheckError condition, preventing automatic bucket creation.

When OADP Data Mover backs up an encrypted Red Hat OpenShift Data Foundation (ODF) volume, it creates a temporary persistent volume claim (PVC) in the openshift-adp namespace. The ceph-csi driver then attempts to fetch the ceph-csi-kms-token secret from openshift-adp instead of the original application namespace that has the secret. As a consequence, the NonAdminBackup fails with an error similar to the following:

openshift-adp   2h11m       Warning   ProvisioningFailed     persistentvolumeclaim/NAMESPACE-rh-cf00000c-0000-0000-0000-22abf391t6dr   failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd-encrypted": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed fetching token from openshift-adp/ceph-csi-kms-token: secrets "ceph-csi-kms-token" not found

When the Velero server accesses large Kopia repositories, for example, during backup deletion, the Velero server might be terminated by the Out Of Memory (OOM) process. This is typically due to high memory usage loading many Kopia index blobs and related repository metadata.

OADP introduces storage agnostic QCOW2 incremental backup for KubeVirt virtual machines (VMs) as a Technology Preview feature. This feature provides a lightweight alternative to traditional VolumeSnapshot-based backups by using libvirt tools to perform incremental QCOW2 backups. After an initial full backup, later backups capture only changed blocks, improving backup performance and reducing resource usage compared to Container Storage Interface (CSI) snapshot methods. This enables more frequent backup schedules with minimal impact on running VM performance.

With VolumeGroupSnapshot (VGS) you can group and snapshot multiple persistent volume claims (PVCs) together using label-based selection. This ensures point-in-time data consistency across related volumes, improving backup reliability for stateful applications that use multiple persistent volumes. VGS is a Technology Preview feature.

Creating file system backups with Restic as an uploader type is removed and is no longer supported. Setting uploaderType: restic in spec.configuration.nodeAgent.uploaderType in the DataProtectionApplication (DPA) custom resource (CR) results in a reconcile error. You can create file system backups by using Kopia instead.

The fs-restore-action-config config map, which pointed to a separate velero-restore-helper container image for file system restore operations, is removed and is no longer supported. You can restore workloads by using the built-in restore helper binary included in the Velero image instead.

The Changing PVC selected-node feature, which allowed overriding the volume.kubernetes.io/selected-node annotation to change the node selected for a persistent volume claim (PVC) during restore, is removed and is no longer supported. You can ensure PVCs are scheduled to the correct node by using the native Kubernetes WaitForFirstConsumer volume binding mode instead.