# oc create ns lvms-policy-ns
Logical Volume Manager (LVM) Storage uses the TopoLVM CSI driver to dynamically provision local storage on single-node OpenShift clusters.
LVM Storage creates thin-provisioned volumes using Logical Volume Manager and provides dynamic provisioning of block storage on a limited resources single-node OpenShift cluster.
You can create volume groups, persistent volume claims (PVCs), volume snapshots, and volume clones by using LVM Storage.
You can install Logical Volume Manager (LVM) Storage on a single-node OpenShift cluster and configure it to dynamically provision storage for your workloads.
You can deploy LVM Storage on single-node OpenShift clusters by using the OKD CLI (oc
), OKD web console, or Red Hat Advanced Cluster Management (RHACM).
The prerequisites to install LVM Storage are as follows:
Ensure that you have a minimum of 10 milliCPU and 100 MiB of RAM.
Ensure that every managed cluster has dedicated disks that are used to provision storage. LVM Storage uses only those disks that are empty and do not contain file system signatures. To ensure that the disks are empty and do not contain file system signatures, wipe the disks before using them.
Before installing LVM Storage in a private CI environment where you can reuse the storage devices that you configured in the previous LVM Storage installation, ensure that you have wiped the disks that are not in use. If you do not wipe the disks before installing LVM Storage, you cannot reuse the disks without manual intervention.
You cannot wipe the disks that are in use. |
If you want to install LVM Storage by using Red Hat Advanced Cluster Management (RHACM), ensure that you have installed RHACM on an OKD cluster. See the Installing LVM Storage using RHACM section.
You can install LVM Storage using the Red Hat OKD OperatorHub.
You have access to the single-node OpenShift cluster.
You are using an account with the cluster-admin
and Operator installation permissions.
Log in to the OKD Web Console.
Click Operators → OperatorHub.
Scroll or type LVM Storage
into the Filter by keyword box to find LVM Storage.
Click Install.
Set the following options on the Install Operator page:
Update Channel as stable-4.12.
Installation Mode as A specific namespace on the cluster.
Installed Namespace as Operator recommended namespace openshift-storage.
If the openshift-storage
namespace does not exist, it is created during the operator installation.
Approval Strategy as Automatic or Manual.
If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.
If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.
Click Install.
Verify that LVM Storage shows a green tick, indicating successful installation.
LVM Storage is deployed on single-node OpenShift clusters using Red Hat Advanced Cluster Management (RHACM).
You create a Policy
object on RHACM that deploys and configures the Operator when it is applied to managed clusters which match the selector specified in the PlacementRule
resource.
The policy is also applied to clusters that are imported later and satisfy the placement rule.
Access to the RHACM cluster using an account with cluster-admin
and Operator installation permissions.
Dedicated disks on each single-node OpenShift cluster to be used by LVM Storage.
The single-node OpenShift cluster needs to be managed by RHACM, either imported or created.
Log in to the RHACM CLI using your OKD credentials.
Create a namespace in which you will create policies.
# oc create ns lvms-policy-ns
To create a policy, save the following YAML to a file with a name such as policy-lvms-operator.yaml
:
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
name: placement-install-lvms
spec:
clusterConditions:
- status: "True"
type: ManagedClusterConditionAvailable
clusterSelector: (1)
matchExpressions:
- key: mykey
operator: In
values:
- myvalue
---
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
name: binding-install-lvms
placementRef:
apiGroup: apps.open-cluster-management.io
kind: PlacementRule
name: placement-install-lvms
subjects:
- apiGroup: policy.open-cluster-management.io
kind: Policy
name: install-lvms
---
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
annotations:
policy.open-cluster-management.io/categories: CM Configuration Management
policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
policy.open-cluster-management.io/standards: NIST SP 800-53
name: install-lvms
spec:
disabled: false
remediationAction: enforce
policy-templates:
- objectDefinition:
apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
name: install-lvms
spec:
object-templates:
- complianceType: musthave
objectDefinition:
apiVersion: v1
kind: Namespace
metadata:
labels:
openshift.io/cluster-monitoring: "true"
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
name: openshift-storage
- complianceType: musthave
objectDefinition:
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: openshift-storage-operatorgroup
namespace: openshift-storage
spec:
targetNamespaces:
- openshift-storage
- complianceType: musthave
objectDefinition:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: lvms
namespace: openshift-storage
spec:
installPlanApproval: Automatic
name: lvms-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
remediationAction: enforce
severity: low
- objectDefinition:
apiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
name: lvms
spec:
object-templates:
- complianceType: musthave
objectDefinition:
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
name: my-lvmcluster
namespace: openshift-storage
spec:
storage:
deviceClasses:
- name: vg1
deviceSelector: (2)
paths:
- /dev/disk/by-path/pci-0000:87:00.0-nvme-1
- /dev/disk/by-path/pci-0000:88:00.0-nvme-1
thinPoolConfig:
name: thin-pool-1
sizePercent: 90
overprovisionRatio: 10
nodeSelector: (3)
nodeSelectorTerms:
- matchExpressions:
- key: app
operator: In
values:
- test1
remediationAction: enforce
severity: low
1 | Replace the key and value in PlacementRule.spec.clusterSelector to match the labels set on the single-node OpenShift clusters on which you want to install LVM Storage. |
2 | To control or restrict the volume group to your preferred disks, you can manually specify the local paths of the disks in the deviceSelector section of the LVMCluster YAML. |
3 | To add a node filter, which is a subset of the additional worker nodes, specify the required filter in the nodeSelector section. LVM Storage detects and uses the additional worker nodes when the new nodes show up. |
This |
Create the policy in the namespace by running the following command:
# oc create -f policy-lvms-operator.yaml -n lvms-policy-ns (1)
1 | The policy-lvms-operator.yaml is the name of the file to which the policy is saved. |
This creates a Policy
, a PlacementRule
, and a PlacementBinding
object in the lvms-policy-ns
namespace.
The policy creates a Namespace
, OperatorGroup
, Subscription
, and LVMCluster
resource on the clusters that match the placement rule.
This deploys the Operator on the single-node OpenShift clusters which match the selection criteria and configures it to set up the required resources to provision storage.
The Operator uses all the disks specified in the LVMCluster
CR.
If no disks are specified, the Operator uses all the unused disks on the single-node OpenShift node.
After a device is added to the |
The limitations to configure the size of the devices that you can use to provision storage using LVM Storage are as follows:
The total storage size that you can provision is limited by the size of the underlying Logical Volume Manager (LVM) thin pool and the over-provisioning factor.
The size of the logical volume depends on the size of the Physical Extent (PE) and the Logical Extent (LE).
You can define the size of PE and LE during the physical and logical device creation.
The default PE and LE size is 4 MB.
If the size of the PE is increased, the maximum size of the LVM is determined by the kernel limits and your disk space.
Architecture | RHEL 6 | RHEL 7 | RHEL 8 | RHEL 9 |
---|---|---|---|---|
32-bit |
16 TB |
- |
- |
- |
64-bit |
8 EB [1] 100 TB [2] |
8 EB [1] 500 TB [2] |
8 EB |
8 EB |
Theoretical size.
Tested size.
You can provision persistent volume claims (PVCs) using the storage class that is created during the Operator installation. You can provision block and file PVCs, however, the storage is allocated only when a pod that uses the PVC is created.
LVM Storage provisions PVCs in units of 1 GiB. The requested storage is rounded up to the nearest GiB. |
Identify the StorageClass
that is created when LVM Storage is deployed.
The StorageClass
name is in the format, lvms-<device-class-name>
.
The device-class-name
is the name of the device class that you provided in the LVMCluster
of the Policy
YAML.
For example, if the deviceClass
is called vg1
, then the storageClass
name is lvms-vg1
.
The volumeBindingMode
of the storage class is set to WaitForFirstConsumer
.
To create a PVC where the application requires storage, save the following YAML to a file with a name such as pvc.yaml
.
# block pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lvm-block-1
namespace: default
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: lvms-vg1
---
# file pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lvm-file-1
namespace: default
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 10Gi
storageClassName: lvms-vg1
Create the PVC by running the following command:
# oc create -f pvc.yaml -ns <application_namespace>
The created PVCs remain in pending
state until you deploy the pods that use them.
To leverage the new storage after adding additional capacity, you can expand existing persistent volume claims (PVCs) with LVM Storage.
Dynamic provisioning is used.
The controlling StorageClass
object has allowVolumeExpansion
set to true
.
Modify the .spec.resources.requests.storage
field in the desired PVC resource to the new size by running the following command:
oc patch <pvc_name> -n <application_namespace> -p '{ "spec": { "resources": { "requests": { "storage": "<desired_size>" }}}}'
Watch the status.conditions
field of the PVC to see if the resize has completed. OKD adds the Resizing
condition to the PVC during expansion, which is removed after the expansion completes.
Currently, it is not possible to upgrade from OpenShift Data Foundation Logical Volume Manager Operator 4.11 to LVM Storage 4.12 on single-node OpenShift clusters.
The data will not be preserved during this process. |
Back up any data that you want to preserve on the persistent volume claims (PVCs).
Delete all PVCs provisioned by the OpenShift Data Foundation Logical Volume Manager Operator and their pods.
Reinstall LVM Storage on OKD 4.12.
Recreate the workloads.
Copy the backup data to the PVCs created after upgrading to 4.12.
You can take volume snapshots of persistent volumes (PVs) that are provisioned by LVM Storage. You can also create volume snapshots of the cloned volumes. Volume snapshots help you to do the following:
Back up your application data.
Volume snapshots are located on the same devices as the original data. To use the volume snapshots as backups, you need to move the snapshots to a secure location. You can use OpenShift API for Data Protection backup and restore solutions. |
Revert to a state at which the volume snapshot was taken.
You can create volume snapshots based on the available capacity of the thin pool and the overprovisioning limits.
LVM Storage creates a VolumeSnapshotClass
with the lvms-<deviceclass-name>
name.
You ensured that the persistent volume claim (PVC) is in Bound
state. This is required for a consistent snapshot.
You stopped all the I/O to the PVC before taking the snapshot.
Log in to the single-node OpenShift for which you need to run the oc
command.
Save the following YAML to a file with a name such as lvms-vol-snapshot.yaml
.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: lvm-block-1-snap
spec:
volumeSnapshotClassName: lvms-vg1
source:
persistentVolumeClaimName: lvm-block-1
Create the snapshot by running the following command in the same namespace as the PVC:
# oc create -f lvms-vol-snapshot.yaml
A read-only copy of the PVC is created as a volume snapshot.
When you restore a volume snapshot, a new persistent volume claim (PVC) is created. The restored PVC is independent of the volume snapshot and the source PVC.
The storage class must be the same as that of the source PVC.
The size of the requested PVC must be the same as that of the source volume of the snapshot.
A snapshot must be restored to a PVC of the same size as the source volume of the snapshot. If a larger PVC is required, you can resize the PVC after the snapshot is restored successfully. |
Identify the storage class name of the source PVC and volume snapshot name.
Save the following YAML to a file with a name such as lvms-vol-restore.yaml
to restore the snapshot.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: lvm-block-1-restore
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
Resources:
Requests:
storage: 2Gi
storageClassName: lvms-vg1
dataSource:
name: lvm-block-1-snap
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
Create the policy by running the following command in the same namespace as the snapshot:
# oc create -f lvms-vol-restore.yaml
You can delete volume snapshots resources and persistent volume claims (PVCs).
Delete the volume snapshot resource by running the following command:
# oc delete volumesnapshot <volume_snapshot_name> -n <namespace>
When you delete a persistent volume claim (PVC), the snapshots of the PVC are not deleted. |
To delete the restored volume snapshot, delete the PVC that was created to restore the volume snapshot by running the following command:
# oc delete pvc <pvc_name> -n <namespace>
A clone is a duplicate of an existing storage volume that can be used like any standard volume.
You create a clone of a volume to make a point-in-time copy of the data. A persistent volume claim (PVC) cannot be cloned with a different size.
The cloned PVC has write access. |
You ensured that the PVC is in Bound
state. This is required for a consistent snapshot.
You ensured that the StorageClass
is the same as that of the source PVC.
Identify the storage class of the source PVC.
To create a volume clone, save the following YAML to a file with a name such as lvms-vol-clone.yaml
:
apiVersion: v1
kind: PersistentVolumeClaim
Metadata:
name: lvm-block-1-clone
Spec:
storageClassName: lvms-vg1
dataSource:
name: lvm-block-1
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
volumeMode: Block
Resources:
Requests:
storage: 2Gi
Create the policy in the same namespace as the source PVC by running the following command:
# oc create -f lvms-vol-clone.yaml
To enable cluster monitoring, you must add the following label in the namespace where you have installed LVM Storage:
openshift.io/cluster-monitoring=true
For information about enabling cluster monitoring in RHACM, see Observability and Adding custom metrics. |
You can monitor LVM Storage by viewing the metrics.
The following table describes the topolvm
metrics:
Alert | Description |
---|---|
|
Indicates the percentage of data space used in the LVM thinpool. |
|
Indicates the percentage of metadata space used in the LVM thinpool. |
|
Indicates the size of the LVM thin pool in bytes. |
|
Indicates the available space in the LVM volume group in bytes. |
|
Indicates the size of the LVM volume group in bytes. |
|
Indicates the available over-provisioned size of the LVM thin pool in bytes. |
Metrics are updated every 10 minutes or when there is a change, such as a new logical volume creation, in the thin pool. |
When the thin pool and volume group reach maximum storage capacity, further operations fail. This can lead to data loss.
LVM Storage sends the following alerts when the usage of the thin pool and volume group exceeds a certain value:
Alert | Description |
---|---|
|
This alert is triggered when both the volume group and thin pool usage exceeds 75% on nodes. Data deletion or volume group expansion is required. |
|
This alert is triggered when both the volume group and thin pool usage exceeds 85% on nodes. In this case, the volume group is critically full. Data deletion or volume group expansion is required. |
|
This alert is triggered when the thin pool data uusage in the volume group exceeds 75% on nodes. Data deletion or thin pool expansion is required. |
|
This alert is triggered when the thin pool data usage in the volume group exceeds 85% on nodes. Data deletion or thin pool expansion is required. |
|
This alert is triggered when the thin pool metadata usage in the volume group exceeds 75% on nodes. Data deletion or thin pool expansion is required. |
|
This alert is triggered when the thin pool metadata usage in the volume group exceeds 85% on nodes. Data deletion or thin pool expansion is required. |
When LVM Storage is unable to automatically resolve a problem, use the must-gather tool to collect the log files and diagnostic information so that you or the Red Hat Support can review the problem and determine a solution.
Run the must-gather
command from the client connected to the LVM Storage cluster:
$ oc adm must-gather --image=registry.redhat.io/lvms4/lvms-must-gather-rhel9:v4.12 --dest-dir=<directory_name>
A persistent volume claim (PVC) can get stuck in a Pending
state for a number of reasons. For example:
Insufficient computing resources
Network problems
Mismatched storage class or node selector
No available volumes
The node with the persistent volume (PV) is in a Not Ready
state
Identify the cause by using the oc describe
command to review details about the stuck PVC.
Retrieve the list of PVCs by running the following command:
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
lvms-test Pending lvms-vg1 11s
Inspect the events associated with a PVC stuck in the Pending
state by running the following command:
$ oc describe pvc <pvc_name> (1)
1 | Replace <pvc_name> with the name of the PVC. For example, lvms-vg1 . |
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 4s (x2 over 17s) persistentvolume-controller storageclass.storage.k8s.io "lvms-vg1" not found
If you encounter a storage class "not found" error, check the LVMCluster
resource and ensure that all the logical volume manager storage (LVMS) pods are running. You can create an LVMCluster
resource if it does not exist.
Verify the presence of the LVMCluster resource by running the following command:
$ oc get lvmcluster -n openshift-storage
NAME AGE
my-lvmcluster 65m
If the cluster doesn’t have an LVMCluster
resource, create one by running the following command:
$ oc create -n openshift-storage -f <custom_resource> (1)
1 | Replace <custom_resource> with a custom resource URL or file tailored to your requirements. |
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
name: my-lvmcluster
spec:
storage:
deviceClasses:
- name: vg1
default: true
thinPoolConfig:
name: thin-pool-1
sizePercent: 90
overprovisionRatio: 10
Check that all the pods from LVMS are in the Running
state in the openshift-storage
namespace by running the following command:
$ oc get pods -n openshift-storage
NAME READY STATUS RESTARTS AGE
lvms-operator-7b9fb858cb-6nsml 3/3 Running 0 70m
topolvm-controller-5dd9cf78b5-7wwr2 5/5 Running 0 66m
topolvm-node-dr26h 4/4 Running 0 66m
vg-manager-r6zdv 1/1 Running 0 66m
The expected output is one running instance of lvms-operator
and vg-manager
. One instance of topolvm-controller
and topolvm-node
is expected for each node.
If topolvm-node
is stuck in the Init
state, there is a failure to locate an available disk for LVMS to use. To retrieve the information necessary to troubleshoot, review the logs of the vg-manager
pod by running the following command:
$ oc logs -l app.kubernetes.io/component=vg-manager -n openshift-storage
Sometimes a persistent volume claim (PVC) is stuck in a Pending
state because a particular node in the cluster has failed. To identify the failed node, you can examine the restart count of the topolvm-node
pod. An increased restart count indicates potential problems with the underlying node, which may require further investigation and troubleshooting.
Examine the restart count of the topolvm-node
pod instances by running the following command:
$ oc get pods -n openshift-storage
NAME READY STATUS RESTARTS AGE
lvms-operator-7b9fb858cb-6nsml 3/3 Running 0 70m
topolvm-controller-5dd9cf78b5-7wwr2 5/5 Running 0 66m
topolvm-node-dr26h 4/4 Running 0 66m
topolvm-node-54as8 4/4 Running 0 66m
topolvm-node-78fft 4/4 Running 17 (8s ago) 66m
vg-manager-r6zdv 1/1 Running 0 66m
vg-manager-990ut 1/1 Running 0 66m
vg-manager-an118 1/1 Running 0 66m
After you resolve any issues with the node, you might need to perform the forced cleanup procedure if the PVC is still stuck in a Pending
state.
If you see a failure message while inspecting the events associated with the persistent volume claim (PVC), there might be a problem with the underlying volume or disk. Disk and volume provisioning issues often result with a generic error first, such as Failed to provision volume with StorageClass <storage_class_name>
. A second, more specific error message usually follows.
Inspect the events associated with a PVC by running the following command:
$ oc describe pvc <pvc_name> (1)
1 | Replace <pvc_name> with the name of the PVC. Here are some examples of disk or volume failure error messages and their causes:
|
Establish a direct connection to the host where the problem is occurring.
Resolve the disk issue.
After you have resolved the issue with the disk, you might need to perform the forced cleanup procedure if failure messages persist or reoccur.
If disk- or node-related problems persist after you complete the troubleshooting procedures, it might be necessary to perform a forced cleanup procedure. A forced cleanup is used to comprehensively address persistent issues and ensure the proper functioning of the LVMS.
All of the persistent volume claims (PVCs) created using the logical volume manager storage (LVMS) driver have been removed.
The pods using those PVCs have been stopped.
Switch to the openshift-storage
namespace by running the following command:
$ oc project openshift-storage
Ensure there is no Logical Volume
custom resource (CR) remaining by running the following command:
$ oc get logicalvolume
No resources found
If there are any LogicalVolume
CRs remaining, remove their finalizers by running the following command:
$ oc patch logicalvolume <name> -p '{"metadata":{"finalizers":[]}}' --type=merge (1)
1 | Replace <name> with the name of the CR. |
After removing their finalizers, delete the CRs by running the following command:
$ oc delete logicalvolume <name> (1)
1 | Replace <name> with the name of the CR. |
Make sure there are no LVMVolumeGroup
CRs left by running the following command:
$ oc get lvmvolumegroup
No resources found
If there are any LVMVolumeGroup
CRs left, remove their finalizers by running the following command:
$ oc patch lvmvolumegroup <name> -p '{"metadata":{"finalizers":[]}}' --type=merge (1)
1 | Replace <name> with the name of the CR. |
After removing their finalizers, delete the CRs by running the following command:
$ oc delete lvmvolumegroup <name> (1)
1 | Replace <name> with the name of the CR. |
Remove any LVMVolumeGroupNodeStatus
CRs by running the following command:
$ oc delete lvmvolumegroupnodestatus --all
Remove the LVMCluster
CR by running the following command:
$ oc delete lvmcluster --all