Dynamic Provisioning and Creating Storage Classes

Overview
Available dynamically provisioned plug-ins
Defining a StorageClass
Changing the default StorageClass
Additional information and examples

Overview

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any intimate knowledge about the underlying storage volume sources.

The OKD persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OKD. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plug-in APIs.

To enable dynamic provisioning, add the openshift_master_dynamic_provisioning_enabled variable to the [OSEv3:vars] section of the Ansible inventory file and set its value to True.

[OSEv3:vars]

openshift_master_dynamic_provisioning_enabled=True

Available dynamically provisioned plug-ins

OKD provides the following provisioner plug-ins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Storage Type Provisioner Plug-in Name Required Configuration Notes

Storage Type	Provisioner Plug-in Name	Required Configuration	Notes
OpenStack Cinder	`kubernetes.io/cinder`	Configuring for OpenStack
AWS Elastic Block Store (EBS)	`kubernetes.io/aws-ebs`	Configuring for AWS	For dynamic provisioning when using multiple clusters in different zones, tag each node with `Key=kubernetes.io/cluster/xxxx,Value=clusterid` where `xxxx` and `clusterid` are unique per cluster.
GCE Persistent Disk (gcePD)	`kubernetes.io/gce-pd`	Configuring for GCE	In multi-zone configurations, it is advisable to run one Openshift cluster per GCE project to avoid PVs from getting created in zones where no node from current cluster exists.
GlusterFS	`kubernetes.io/glusterfs`	Configuring GlusterFS
Ceph RBD	`kubernetes.io/rbd`	Configuring Ceph RBD
Trident from NetApp	`netapp.io/trident`	Configuring for Trident	Storage orchestrator for NetApp ONTAP, SolidFire, and E-Series storage.
VMWare vSphere	`kubernetes.io/vsphere-volume`	Getting Started with vSphere and Kubernetes
Azure Disk	`kubernetes.io/azure-disk`	Configuring for Azure

OpenStack Cinder

kubernetes.io/cinder

Configuring for OpenStack

AWS Elastic Block Store (EBS)

kubernetes.io/aws-ebs

Configuring for AWS

For dynamic provisioning when using multiple clusters in different zones, tag each node with Key=kubernetes.io/cluster/xxxx,Value=clusterid where xxxx and clusterid are unique per cluster.

GCE Persistent Disk (gcePD)

kubernetes.io/gce-pd

Configuring for GCE

In multi-zone configurations, it is advisable to run one Openshift cluster per GCE project to avoid PVs from getting created in zones where no node from current cluster exists.

GlusterFS

kubernetes.io/glusterfs

Configuring GlusterFS

Ceph RBD

kubernetes.io/rbd

Configuring Ceph RBD

Trident from NetApp

netapp.io/trident

Configuring for Trident

Storage orchestrator for NetApp ONTAP, SolidFire, and E-Series storage.

VMWare vSphere

kubernetes.io/vsphere-volume

Getting Started with vSphere and Kubernetes

Azure Disk

kubernetes.io/azure-disk

Configuring for Azure

Any chosen provisioner plug-in also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

Defining a StorageClass

StorageClass objects are currently a globally scoped object and need to be created by cluster-admin or storage-admin users.

For GCE and AWS, a default StorageClass is created during OKD installation. You can change the default StorageClass or delete it.

There are currently six plug-ins that are supported. The following sections describe the basic object definition for a StorageClass and specific examples for each of the supported plug-in types.

Basic StorageClass object definition

StorageClass Basic object definition

kind: StorageClass (1)
apiVersion: storage.k8s.io/v1 (2)
metadata:
  name: foo (3)
  annotations: (4)
     ...
provisioner: kubernetes.io/plug-in-type (5)
parameters: (6)
  param1: value
  ...
  paramN: value

1	(required) The API object type.
2	(required) The current apiVersion.
3	(required) The name of the StorageClass.
4	(optional) Annotations for the StorageClass
5	(required) The type of provisioner associated with this storage class.
6	(optional) The parameters required for the specific provisioner, this will change from plug-in to plug-in.

StorageClass annotations

To set a StorageClass as the cluster-wide default:

   storageclass.kubernetes.io/is-default-class: "true"

This enables any Persistent Volume Claim (PVC) that does not specify a specific volume to automatically be provisioned through the default StorageClass

Beta annotation storageclass.beta.kubernetes.io/is-default-class is still working. However it will be removed in a future release.

To set a StorageClass description:

   kubernetes.io/description: My StorageClass Description

OpenStack Cinder object definition

cinder-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gold
provisioner: kubernetes.io/cinder
parameters:
  type: fast  (1)
  availability: nova (2)
  fsType: ext4 (3)

1	Volume type created in Cinder. Default is empty.
2	Availability Zone. If not specified, volumes are generally round-robined across all active zones where the OKD cluster has a node.
3	File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

AWS ElasticBlockStore (EBS) object definition

aws-ebs-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 (1)
  zone: us-east-1d (2)
  iopsPerGB: "10" (3)
  encrypted: "true" (4)
  kmsKeyId: keyvalue (5)
  fsType: ext4 (6)

1	Select from `io1`, `gp2`, `sc1`, `st1`. The default is `gp2`. See AWS documentation for valid Amazon Resource Name (ARN) values.
2	AWS zone. If no zone is specified, volumes are generally round-robined across all active zones where the OKD cluster has a node. Zone and zones parameters must not be used at the same time.
3	Only for io1 volumes. I/O operations per second per GiB. The AWS volume plug-in multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See AWS documentation for further details.
4	Denotes whether to encrypt the EBS volume. Valid values are `true` or `false`.
5	Optional. The full ARN of the key to use when encrypting the volume. If none is supplied, but `encypted` is set to `true`, then AWS generates a key. See AWS documentation for a valid ARN value.
6	File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

GCE PersistentDisk (gcePD) object definition

gce-pd-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard  (1)
  zone: us-central1-a  (2)
  zones: us-central1-a, us-central1-b, us-east1-b  (3)
  fsType: ext4 (4)

1	Select either `pd-standard` or `pd-ssd`. The default is `pd-ssd`.
2	GCE zone. If no zone is specified, volumes are generally round-robined across all active zones where the OKD cluster has a node. Zone and zones parameters must not be used at the same time.
3	A comma-separated list of GCE zone(s). If no zone is specified, volumes are generally round-robined across all active zones where the OKD cluster has a node. Zone and zones parameters must not be used at the same time.
4	File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

GlusterFS object definition

glusterfs-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/glusterfs
parameters: (1)
  resturl: http://127.0.0.1:8081 (2)
  restuser: admin (3)
  secretName: heketi-secret (4)
  secretNamespace: default (5)
  gidMin: "40000" (6)
  gidMax: "50000" (7)
  volumeoptions: group metadata-cache, nl-cache on (8)
  volumetype: replicate:3 (9)
  volumenameprefix: custom (10)

1	Listed are mandatory and a few optional parameters. Please refer to Registering a Storage Class for additional parameters.
2	heketi (volume management REST service for Gluster) URL that provisions GlusterFS volumes on demand. The general format should be `{http/https}://{IPaddress}:{Port}`. This is a mandatory parameter for the GlusterFS dynamic provisioner. If the heketi service is exposed as a routable service in the OKD, it will have a resolvable fully qualified domain name (FQDN) and heketi service URL.
3	heketi user who has access to create volumes. Usually "admin".
4	Identification of a Secret that contains a user password to use when talking to heketi. Optional; an empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must be of type `"kubernetes.io/glusterfs"`.
5	The namespace of mentioned `secretName`. Optional; an empty password will be used when both `secretNamespace` and `secretName` are omitted. The provided secret must be of type `"kubernetes.io/glusterfs"`.
6	Optional. The minimum value of the GID range for volumes of this StorageClass.
7	Optional. The maximum value of the GID range for volumes of this StorageClass.
8	Optional. Options for newly created volumes. It allows for performance tuning. See Tuning Volume Options for more GlusterFS volume options.
9	Optional. The type of volume to use.
10	Optional. Enables custom volume name support using the following format: `<volumenameprefix>_<namespace>_<claimname>_UUID`. If you create a new PVC called `myclaim` in your project `project1` using this storageClass, the volume name will be `custom-project1-myclaim-UUID`.

When the gidMin and gidMax values are not specified, their defaults are 2000 and 2147483647, respectively. Each dynamically provisioned volume will be given a GID in this range (gidMin-gidMax). This GID is released from the pool when the respective volume is deleted. The GID pool is per StorageClass. If two or more storage classes have GID ranges that overlap there may be duplicate GIDs dispatched by the provisioner.

When heketi authentication is used, a Secret containing the admin key should also exist:

heketi-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  key: bXlwYXNzd29yZA== (1)
type: kubernetes.io/glusterfs

1	base64 encoded password, for example: `echo -n "mypassword" \| base64`

When the PVs are dynamically provisioned, the GlusterFS plug-in automatically creates an Endpoints and a headless Service named gluster-dynamic-<claimname>. When the PVC is deleted, these dynamic resources are deleted automatically.

Ceph RBD object definition

ceph-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.16.153.105:6789  (1)
  adminId: admin  (2)
  adminSecretName: ceph-secret  (3)
  adminSecretNamespace: kube-system  (4)
  pool: kube  (5)
  userId: kube  (6)
  userSecretName: ceph-secret-user  (7)
  fsType: ext4 (8)

1	Ceph monitors, comma-delimited. It is required.
2	Ceph client ID that is capable of creating images in the pool. Default is "admin".
3	Secret Name for `adminId`. It is required. The provided secret must have type "kubernetes.io/rbd".
4	The namespace for `adminSecret`. Default is "default".
5	Ceph RBD pool. Default is "rbd".
6	Ceph client ID that is used to map the Ceph RBD image. Default is the same as `adminId`.
7	The name of Ceph Secret for `userId` to map Ceph RBD image. It must exist in the same namespace as PVCs. It is required.
8	File system that is created on dynamically provisioned volumes. This value is copied to the `fsType` field of dynamically provisioned persistent volumes and the file system is created when the volume is mounted for the first time. The default value is `ext4`.

Trident object definition

trident.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gold
provisioner: netapp.io/trident (1)
parameters: (2)
  media: "ssd"
  provisioningType: "thin"
  snapshots: "true"

Trident uses the parameters as selection criteria for the different pools of storage that are registered with it. Trident itself is configured separately.

1	For more information about installing Trident with OKD, see the Trident documentation.
2	For more information about supported parameters, see the storage attributes section of the Trident documentation.

VMware vSphere object definition

vsphere-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/vsphere-volume (1)
parameters:
  diskformat: thin (2)

1	For more information about using VMWare vSphere with OKD, see the VMWare vSphere documentation.
2	`diskformat`: `thin`, `zeroedthick` and `eagerzeroedthick`. See vSphere docs for details. Default: `thin`

Azure File object definition

To configure Azure file dynamic provisioning:

Create the role in the user’s project:

$ cat azf-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: system:controller:persistent-volume-binder
  namespace: <user's project name>
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]

Create the role binding to the persistent-volume-binder service account in the kube-system project:

$ cat azf-rolebind.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: system:controller:persistent-volume-binder
  namespace: <user's project>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: system:controller:persistent-volume-binder
subjects:
- kind: ServiceAccount
  name: persistent-volume-binder
namespace: kube-system

Add the service account as admin to the user’s project:

$ oc policy add-role-to-user admin system:serviceaccount:kube-system:persistent-volume-binder -n <user's project>

Create a storage class for the Azure file:

$ cat azfsc.yaml | oc create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azfsc
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777

The user can now create a PVC that uses this storage class.

Azure Disk object definition

azure-advanced-disk-storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: azure_storage_account_name  (1)
  storageaccounttype: Standard_LRS  (2)
  kind: Dedicated  (3)

1	Azure storage account name. This must reside in the same resource group as the cluster. If a storage account is specified, the `location` is ignored. If a storage account is not specified, a new storage account gets created in the same resource group as the cluster. If you are specifying a `storageAccount`, the value for `kind` must be `Dedicated`.
2	Azure storage account SKU tier. Default is empty. Note: Premium VM can attach both Standard_LRS and Premium_LRS disks, Standard VM can only attach Standard_LRS disks, Managed VM can only attach managed disks, and unmanaged VM can only attach unmanaged disks.
3	Possible values are `Shared` (default), `Dedicated`, and `Managed`. If `kind` is set to `Shared`, Azure creates all unmanaged disks in a few shared storage accounts in the same resource group as the cluster. If `kind` is set to `Managed`, Azure creates new managed disks. If `kind` is set to `Dedicated` and a `storageAccount` is specified, Azure uses the specified storage account for the new unmanaged disk in the same resource group as the cluster. For this to work: The specified storage account must be in the same region. Azure Cloud Provider must have a write access to the storage account. If `kind` is set to `Dedicated` and a `storageAccount` is not specified, Azure creates a new dedicated storage account for the new unmanaged disk in the same resource group as the cluster.

Azure StorageClass is revised in OKD version 3.7. If you upgraded from a previous version, either:

specify the property kind: dedicated to continue using the Azure StorageClass created before the upgrade. Or,
add the location parameter (for example, "location": "southcentralus",) in the azure.conf file to use the default property kind: shared. Doing this creates new storage accounts for future use.

Changing the default StorageClass

If you are using GCE and AWS, use the following process to change the default StorageClass:

List the StorageClass:

$ oc get storageclass

NAME                 TYPE
gp2 (default)        kubernetes.io/aws-ebs (1)
standard             kubernetes.io/gce-pd

1	`(default)` denotes the default StorageClass.

Change the value of the annotation storageclass.kubernetes.io/is-default-class to false for the default StorageClass:

$ oc patch storageclass gp2 -p '{"metadata": {"annotations": \
    {"storageclass.kubernetes.io/is-default-class": "false"}}}'

Make another StorageClass the default by adding or modifying the annotation as storageclass.kubernetes.io/is-default-class=true.

$ oc patch storageclass standard -p '{"metadata": {"annotations": \
    {"storageclass.kubernetes.io/is-default-class": "true"}}}'

If more than one StorageClass is marked as default, a PVC can only be created if the storageClassName is explicitly specified. Therefore, only one StorageClass should be set as the default.

Verify the changes:

$ oc get storageclass

NAME                 TYPE
gp2                  kubernetes.io/aws-ebs
standard (default)   kubernetes.io/gce-pd