Storage configuration | Post-installation configuration

After installing OKD, you can further expand and customize your cluster to your requirements, including storage configuration.

Dynamic provisioning

About dynamic provisioning

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any detailed knowledge about the underlying storage volume sources.

The OKD persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OKD. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plugin APIs.

Available dynamic provisioning plugins

OKD provides the following provisioner plugins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Storage type Provisioner plugin name Notes

Storage type	Provisioner plugin name	Notes
OpenStack Cinder	`kubernetes.io/cinder`
OpenStack Manila Container Storage Interface (CSI)	`manila.csi.openstack.org`	Once installed, the OpenStack Manila CSI Driver Operator and ManilaDriver automatically create the required storage classes for all available Manila share types needed for dynamic provisioning.
AWS Elastic Block Store (EBS)	`kubernetes.io/aws-ebs`	For dynamic provisioning when using multiple clusters in different zones, tag each node with `Key=kubernetes.io/cluster/<cluster_name>,Value=<cluster_id>` where `<cluster_name>` and `<cluster_id>` are unique per cluster.
Azure Disk	`kubernetes.io/azure-disk`
Azure File	`kubernetes.io/azure-file`	The `persistent-volume-binder` service account requires permissions to create and get secrets to store the Azure storage account and keys.
GCE Persistent Disk (gcePD)	`kubernetes.io/gce-pd`	In multi-zone configurations, it is advisable to run one OKD cluster per GCE project to avoid PVs from being created in zones where no node in the current cluster exists.
VMware vSphere	`kubernetes.io/vsphere-volume`

OpenStack Cinder

kubernetes.io/cinder

OpenStack Manila Container Storage Interface (CSI)

manila.csi.openstack.org

Once installed, the OpenStack Manila CSI Driver Operator and ManilaDriver automatically create the required storage classes for all available Manila share types needed for dynamic provisioning.

AWS Elastic Block Store (EBS)

kubernetes.io/aws-ebs

For dynamic provisioning when using multiple clusters in different zones, tag each node with Key=kubernetes.io/cluster/<cluster_name>,Value=<cluster_id> where <cluster_name> and <cluster_id> are unique per cluster.

Azure Disk

kubernetes.io/azure-disk

Azure File

kubernetes.io/azure-file

The persistent-volume-binder service account requires permissions to create and get secrets to store the Azure storage account and keys.

GCE Persistent Disk (gcePD)

kubernetes.io/gce-pd

In multi-zone configurations, it is advisable to run one OKD cluster per GCE project to avoid PVs from being created in zones where no node in the current cluster exists.

VMware vSphere

kubernetes.io/vsphere-volume

Any chosen provisioner plugin also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

Defining a storage class

StorageClass objects are currently a globally scoped object and must be created by cluster-admin or storage-admin users.

The Cluster Storage Operator might install a default storage class depending on the platform in use. This storage class is owned and controlled by the operator. It cannot be deleted or modified beyond defining annotations and labels. If different behavior is desired, you must define a custom storage class.

The following sections describe the basic definition for a StorageClass object and specific examples for each of the supported plugin types.

Basic StorageClass object definition

The following resource shows the parameters and default values that you use to configure a storage class. This example uses the AWS ElasticBlockStore (EBS) object definition.

Sample StorageClass definition

kind: StorageClass (1)
apiVersion: storage.k8s.io/v1 (2)
metadata:
  name: <storage-class-name> (3)
  annotations: (4)
    storageclass.kubernetes.io/is-default-class: 'true'
    ...
provisioner: kubernetes.io/aws-ebs (5)
parameters: (6)
  type: gp2
...

1	(required) The API object type.
2	(required) The current apiVersion.
3	(required) The name of the storage class.
4	(optional) Annotations for the storage class.
5	(required) The type of provisioner associated with this storage class.
6	(optional) The parameters required for the specific provisioner, this will change from plugin to plugin.

Storage class annotations

To set a storage class as the cluster-wide default, add the following annotation to your storage class metadata:

storageclass.kubernetes.io/is-default-class: "true"

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
...

This enables any persistent volume claim (PVC) that does not specify a specific storage class to automatically be provisioned through the default storage class. However, your cluster can have more than one storage class, but only one of them can be the default storage class.

The beta annotation storageclass.beta.kubernetes.io/is-default-class is still working; however, it will be removed in a future release.

To set a storage class description, add the following annotation to your storage class metadata:

kubernetes.io/description: My Storage Class Description

For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubernetes.io/description: My Storage Class Description
...

OpenStack Cinder object definition

cinder-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/cinder
parameters:
  type: fast  (2)
  availability: nova (3)
  fsType: ext4 (4)

1	Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
2	Volume type created in Cinder. Default is empty.
3	Availability Zone. If not specified, volumes are generally round-robined across all active zones where the OKD cluster has a node.
4	File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

AWS Elastic Block Store (EBS) object definition

aws-ebs-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 (2)
  iopsPerGB: "10" (3)
  encrypted: "true" (4)
  kmsKeyId: keyvalue (5)
  fsType: ext4 (6)

1	(required) Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
2	(required) Select from `io1`, `gp2`, `sc1`, `st1`. The default is `gp2`. See the AWS documentation for valid Amazon Resource Name (ARN) values.
3	Optional: Only for io1 volumes. I/O operations per second per GiB. The AWS volume plugin multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See the AWS documentation for further details.
4	Optional: Denotes whether to encrypt the EBS volume. Valid values are `true` or `false`.
5	Optional: The full ARN of the key to use when encrypting the volume. If none is supplied, but `encypted` is set to `true`, then AWS generates a key. See the AWS documentation for a valid ARN value.
6	Optional: File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

Azure Disk object definition

azure-advanced-disk-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/azure-disk
volumeBindingMode: WaitForFirstConsumer (2)
allowVolumeExpansion: true
parameters:
  kind: Managed (3)
  storageaccounttype: Premium_LRS (4)
reclaimPolicy: Delete

1 Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.

2 Using WaitForFirstConsumer is strongly recommended. This provisions the volume while allowing enough storage to schedule the pod on a free worker node from an available zone.

Possible values are Shared (default), Managed, and Dedicated.

Red Hat only supports the use of kind: Managed in the storage class.

With Shared and Dedicated, Azure creates unmanaged disks, while OKD creates a managed disk for machine OS (root) disks. But because Azure Disk does not allow the use of both managed and unmanaged disks on a node, unmanaged disks created with Shared or Dedicated cannot be attached to OKD nodes.

Azure storage account SKU tier. Default is empty. Note that Premium VMs can attach both Standard_LRS and Premium_LRS disks, Standard VMs can only attach Standard_LRS disks, Managed VMs can only attach managed disks, and unmanaged VMs can only attach unmanaged disks.

If kind is set to Shared, Azure creates all unmanaged disks in a few shared storage accounts in the same resource group as the cluster.
If kind is set to Managed, Azure creates new managed disks.
If kind is set to Dedicated and a storageAccount is specified, Azure uses the specified storage account for the new unmanaged disk in the same resource group as the cluster. For this to work:
- The specified storage account must be in the same region.
- Azure Cloud Provider must have write access to the storage account.
If kind is set to Dedicated and a storageAccount is not specified, Azure creates a new dedicated storage account for the new unmanaged disk in the same resource group as the cluster.

Azure File object definition

The Azure File storage class uses secrets to store the Azure storage account name and the storage account key that are required to create an Azure Files share. These permissions are created as part of the following procedure.

Procedure

Define a ClusterRole object that allows access to create and view secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
#  name: system:azure-cloud-provider
  name: <persistent-volume-binder-role> (1)
rules:
- apiGroups: ['']
  resources: ['secrets']
  verbs:     ['get','create']

1	The name of the cluster role to view and create secrets.

Add the cluster role to the service account:

$ oc adm policy add-cluster-role-to-user <persistent-volume-binder-role> system:serviceaccount:kube-system:persistent-volume-binder

Create the Azure File StorageClass object:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <azure-file> (1)
provisioner: kubernetes.io/azure-file
parameters:
  location: eastus (2)
  skuName: Standard_LRS (3)
  storageAccount: <storage-account> (4)
reclaimPolicy: Delete
volumeBindingMode: Immediate

1	Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
2	Location of the Azure storage account, such as `eastus`. Default is empty, meaning that a new Azure storage account will be created in the OKD cluster’s location.
3	SKU tier of the Azure storage account, such as `Standard_LRS`. Default is empty, meaning that a new Azure storage account will be created with the `Standard_LRS` SKU.
4	Name of the Azure storage account. If a storage account is provided, then `skuName` and `location` are ignored. If no storage account is provided, then the storage class searches for any storage account that is associated with the resource group for any accounts that match the defined `skuName` and `location`.

Considerations when using Azure File

The following file system features are not supported by the default Azure File storage class:

Symlinks
Hard links
Extended attributes
Sparse files
Named pipes

Additionally, the owner user identifier (UID) of the Azure File mounted directory is different from the process UID of the container. The uid mount option can be specified in the StorageClass object to define a specific user identifier to use for the mounted directory.

The following StorageClass object demonstrates modifying the user and group identifier, along with enabling symlinks for the mounted directory.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-file
mountOptions:
  - uid=1500 (1)
  - gid=1500 (2)
  - mfsymlinks (3)
provisioner: kubernetes.io/azure-file
parameters:
  location: eastus
  skuName: Standard_LRS
reclaimPolicy: Delete
volumeBindingMode: Immediate

1	Specifies the user identifier to use for the mounted directory.
2	Specifies the group identifier to use for the mounted directory.
3	Enables symlinks.

GCE PersistentDisk (gcePD) object definition

gce-pd-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard (2)
  replication-type: none
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

1	Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
2	Select either `pd-standard` or `pd-ssd`. The default is `pd-standard`.

VMware vSphere object definition

vsphere-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> (1)
provisioner: kubernetes.io/vsphere-volume (2)
parameters:
  diskformat: thin (3)

1	Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
2	For more information about using VMware vSphere with OKD, see the VMware vSphere documentation.
3	`diskformat`: `thin`, `zeroedthick` and `eagerzeroedthick` are all valid disk formats. See vSphere docs for additional details regarding the disk format types. The default value is `thin`.

oVirt object definition

OKD creates a default object of type StorageClass named ovirt-csi-sc which is used for creating dynamically provisioned persistent volumes.

To create additional storage classes for different configurations, create and save a file with the StorageClass object described by the following sample YAML:

ovirt-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name>  (1)
  annotations:
    storageclass.kubernetes.io/is-default-class: "<boolean>"  (2)
provisioner: csi.ovirt.org
allowVolumeExpansion: <boolean> (3)
reclaimPolicy: Delete (4)
volumeBindingMode: Immediate (5)
parameters:
  storageDomainName: <rhv-storage-domain-name> (6)
  thinProvisioning: "<boolean>"  (7)
  csi.storage.k8s.io/fstype: <file_system_type> (8)

1	Name of the storage class.
2	Set to `false` if the storage class is the default storage class in the cluster. If set to `true`, the existing default storage class must be edited and set to `false`.
3	`true` enables dynamic volume expansion, `false` prevents it. `true` is recommended.
4	Dynamically provisioned persistent volumes of this storage class are created with this reclaim policy. This default policy is `Delete`.
5	Indicates how to provision and bind `PersistentVolumeClaims`. When not set, `VolumeBindingImmediate` is used. This field is only applied by servers that enable the `VolumeScheduling` feature.
6	The oVirt storage domain name to use.
7	If `true`, the disk is thin provisioned. If `false`, the disk is preallocated. Thin provisioning is recommended.
8	Optional: File system type to be created. Possible values: `ext4` (default) or `xfs`.

Changing the default storage class

Use the following process to change the default storage class. For example you have two defined storage classes, gp2 and standard, and you want to change the default storage class from gp2 to standard.

List the storage class:

$ oc get storageclass

Example output

NAME                 TYPE
gp2 (default)        kubernetes.io/aws-ebs (1)
standard             kubernetes.io/aws-ebs

1	`(default)` denotes the default storage class.

Change the value of the storageclass.kubernetes.io/is-default-class annotation to false for the default storage class:

$ oc patch storageclass gp2 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

Make another storage class the default by setting the storageclass.kubernetes.io/is-default-class annotation to true:

$ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

Verify the changes:

$ oc get storageclass

Example output

NAME                 TYPE
gp2                  kubernetes.io/aws-ebs
standard (default)   kubernetes.io/aws-ebs

Optimizing storage

Optimizing storage helps to minimize storage use across all resources. By optimizing storage, administrators help ensure that existing storage resources are working in an efficient manner.

Available persistent storage options

Understand your persistent storage options so that you can optimize your OKD environment.

Table 1. Available storage options
Storage type	Description	Examples
Block	Presented to the operating system (OS) as a block device Suitable for applications that need full control of storage and operate at a low level on files bypassing the file system Also referred to as a Storage Area Network (SAN) Non-shareable, which means that only one client at a time can mount an endpoint of this type	AWS EBS and VMware vSphere support dynamic persistent volume (PV) provisioning natively in OKD.
File	Presented to the OS as a file system export to be mounted Also referred to as Network Attached Storage (NAS) Concurrency, latency, file locking mechanisms, and other capabilities vary widely between protocols, implementations, vendors, and scales.	RHEL NFS, NetApp NFS ^[1], and Vendor NFS
Object	Accessible through a REST API endpoint Configurable for use in the OKD Registry Applications must build their drivers into the application and/or container.	AWS S3

NetApp NFS supports dynamic PV provisioning when using the Trident plugin.

Currently, CNS is not supported in OKD 4.9.

Recommended configurable storage technology

The following table summarizes the recommended and configurable storage technologies for the given OKD cluster application.

Table 2. Recommended and configurable storage technology
Storage type	ROX¹	RWX²	Registry	Scaled registry	Metrics³	Logging	Apps
Block	Yes⁴	No	Configurable	Not configurable	Recommended	Recommended	Recommended
File	Yes⁴	Yes	Configurable	Configurable	Configurable⁵	Configurable⁶	Recommended
Object	Yes	Yes	Recommended	Recommended	Not configurable	Not configurable	Not configurable⁷
¹ `ReadOnlyMany` ² `ReadWriteMany` ³ Prometheus is the underlying technology used for metrics. ⁴ This does not apply to physical disk, VM physical disk, VMDK, loopback over NFS, AWS EBS, and Azure Disk. ⁵ For metrics, using file storage with the `ReadWriteMany` (RWX) access mode is unreliable. If you use file storage, do not configure the RWX access mode on any persistent volume claims (PVCs) that are configured for use with metrics. ⁶ For logging, using any shared storage would be an anti-pattern. One volume per elasticsearch is required. ⁷ Object storage is not consumed through OKD’s PVs or PVCs. Apps must integrate with the object storage REST API.

A scaled registry is an OKD registry where two or more pod replicas are running.

Specific application storage recommendations

Testing shows issues with using the NFS server on Fedora as storage backend for core services. This includes the OpenShift Container Registry and Quay, Prometheus for monitoring storage, and Elasticsearch for logging storage. Therefore, using Fedora NFS to back PVs used by core services is not recommended.

Other NFS implementations on the marketplace might not have these issues. Contact the individual NFS implementation vendor for more information on any testing that was possibly completed against these OKD core components.

Registry

In a non-scaled/high-availability (HA) OKD registry cluster deployment:

The storage technology does not have to support RWX access mode.
The storage technology must ensure read-after-write consistency.
The preferred storage technology is object storage followed by block storage.
File storage is not recommended for OKD registry cluster deployment with production workloads.

Scaled registry

In a scaled/HA OKD registry cluster deployment:

The storage technology must support RWX access mode.
The storage technology must ensure read-after-write consistency.
The preferred storage technology is object storage.
Red Hat OpenShift Data Foundation (ODF), Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS), Microsoft Azure Blob Storage, and OpenStack Swift are supported.
Object storage should be S3 or Swift compliant.
For non-cloud platforms, such as vSphere and bare metal installations, the only configurable technology is file storage.
Block storage is not configurable.

Metrics

In an OKD hosted metrics cluster deployment:

The preferred storage technology is block storage.
Object storage is not configurable.

It is not recommended to use file storage for a hosted metrics cluster deployment with production workloads.

Logging

In an OKD hosted logging cluster deployment:

The preferred storage technology is block storage.
Object storage is not configurable.

Applications

Application use cases vary from application to application, as described in the following examples:

Storage technologies that support dynamic PV provisioning have low mount time latencies, and are not tied to nodes to support a healthy cluster.
Application developers are responsible for knowing and understanding the storage requirements for their application, and how it works with the provided storage to ensure that issues do not occur when an application scales or interacts with the storage layer.

Other specific application storage recommendations

It is not recommended to use RAID configurations on Write intensive workloads, such as etcd. If you are running etcd with a RAID configuration, you might be at risk of encountering performance issues with your workloads.

OpenStack Cinder: OpenStack Cinder tends to be adept in ROX access mode use cases.
Databases: Databases (RDBMSs, NoSQL DBs, etc.) tend to perform best with dedicated block storage.
The etcd database must have enough storage and adequate performance capacity to enable a large cluster. Information about monitoring and benchmarking tools to establish ample storage and a high-performance environment is described in Recommended etcd practices.

Additional resources

Recommended etcd practices

Deploy Red Hat OpenShift Container Storage

Red Hat OpenShift Container Storage is a provider of agnostic persistent storage for OKD supporting file, block, and object storage, either in-house or in hybrid clouds. As a Red Hat storage solution, Red Hat OpenShift Container Storage is completely integrated with OKD for deployment, management, and monitoring.

If you are looking for Red Hat OpenShift Container Storage information about…	See the following Red Hat OpenShift Container Storage documentation:
What’s new, known issues, notable bug fixes, and Technology Previews	OpenShift Container Storage 4.7 Release Notes
Supported workloads, layouts, hardware and software requirements, sizing and scaling recommendations	Planning your OpenShift Container Storage 4.5 deployment
Instructions on preparing to deploy when your environment is not directly connected to the internet	Preparing to deploy OpenShift Container Storage 4.5 in a disconnected environment
Instructions on deploying OpenShift Container Storage to use an external Red Hat Ceph Storage cluster	Deploying OpenShift Container Storage 4.5 in external mode
Instructions on deploying OpenShift Container Storage to local storage on bare metal infrastructure	Deploying OpenShift Container Storage 4.5 using bare metal infrastructure
Instructions on deploying OpenShift Container Storage on Red Hat OKD VMware vSphere clusters	Deploying OpenShift Container Storage 4.5 on VMware vSphere
Instructions on deploying OpenShift Container Storage using Amazon Web Services for local or cloud storage	Deploying OpenShift Container Storage 4.5 using Amazon Web Services
Instructions on deploying and managing OpenShift Container Storage on existing Red Hat OKD Google Cloud clusters	Deploying and managing OpenShift Container Storage 4.5 using Google Cloud
Instructions on deploying and managing OpenShift Container Storage on existing Red Hat OKD Azure clusters	Deploying and managing OpenShift Container Storage 4.5 using Microsoft Azure
Managing a Red Hat OpenShift Container Storage 4.5 cluster	Managing OpenShift Container Storage 4.5
Monitoring a Red Hat OpenShift Container Storage 4.5 cluster	Monitoring Red Hat OpenShift Container Storage 4.5
Resolve issues encountered during operations	Troubleshooting OpenShift Container Storage 4.5
Migrating your OKD cluster from version 3 to version 4	Migration

If you are looking for Red Hat OpenShift Container Storage information about…

See the following Red Hat OpenShift Container Storage documentation:

What’s new, known issues, notable bug fixes, and Technology Previews

OpenShift Container Storage 4.7 Release Notes

Supported workloads, layouts, hardware and software requirements, sizing and scaling recommendations

Planning your OpenShift Container Storage 4.5 deployment

Instructions on preparing to deploy when your environment is not directly connected to the internet

Preparing to deploy OpenShift Container Storage 4.5 in a disconnected environment

Instructions on deploying OpenShift Container Storage to use an external Red Hat Ceph Storage cluster

Deploying OpenShift Container Storage 4.5 in external mode

Instructions on deploying OpenShift Container Storage to local storage on bare metal infrastructure

Deploying OpenShift Container Storage 4.5 using bare metal infrastructure