1. Documentation
×

Persistent Storage Using FlexVolume Plug-ins

Overview

OKD has built-in volume plug-ins to use different storage technologies. To consume storage from a back-end that does not have a built-in plug-in, you can extend OKD via FlexVolume drivers and provide persistent storage to applications.

A FlexVolume driver is an executable file (typically a shell script), deployed to all machines in the cluster (both masters and nodes) as part of the installation.

Installing FlexVolume Drivers

To install the FlexVolume driver, place the executable file at the volume plug-in path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/<vendor>~<driver>/<driver>. For example, to install the FlexVolume driver for the storage foo, place the executable file at: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/openshift.com~foo/foo.

Ensure that this file exists on all masters and nodes in the cluster.

Consuming Storage using FlexVolume

Use the PersistentVolume object to reference the installed storage. Each PersistentVolume object in OKD represents one storage asset, typically a volume, in the storage back-end.

Persistent Volume Object Definition Using FlexVolume
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001 (1)
spec:
  capacity:
    storage: 1Gi (2)
  accessModes:
    - ReadWriteOnce
  flexVolume:
    driver: openshift.com/foo (3)
    fsType: "ext4" (4)
    secretRef: foo-secret (5)
    readOnly: true (6)
    options: (7)
      fooServer: 192.168.0.1:1234
      fooVolumeName: bar
1 The name of the volume. This is how it is identified via persistent volume claims or from pods. This name can be different from the name of the volume on back-end storage.
2 The amount of storage allocated to this volume.
3 Name of the driver. This field is mandatory.
4 Optional file system that is present on the volume.
5 Optional reference to a secret. Keys and values from this secret are provided to the FlexVolume driver on invocation.
6 Optional read-only flag.
7 Additional options for the FlexVolume driver. This is a free-form dictionary of parameters provided to the FlexVolume driver. Typically, it includes details like the name of the volume that this PersistentVolume represents and addresses of the storage server(s).

FlexVolume Drivers

A FlexVolume driver is an executable file that resides in a well-defined directory on all machines in the cluster, both masters and nodes. OKD calls it whenever it needs to attach, detach, mount, or unmount a volume represented by a PersistentVolume with flexVolume as source.

The first command-line argument of the driver is always an operation name. Other parameters are specific to each operation. Most of the operations takes a JSON(JavaScript Object Notation) string as a parameter. This parameter is a complete JSON string, and not the name of a file with the JSON data.

It contains:

  • All flexVolume.options.

  • Some options from flexVolume prefixed by kubernetes.io/, such as fsType and readwrite.

  • Content of the referenced secret (if specified) prefixed by kubernetes.io/secret/.

Example FlexVolume Driver JSON input
{
	"fooServer": "192.168.0.1:1234", (1)
        "fooVolumeName": "bar",
	"kubernetes.io/fsType": "ext4", (2)
	"kubernetes.io/readwrite": "ro", (3)
	"kubernetes.io/secret/<key name>": "<key value>", (4)
	"kubernetes.io/secret/<another key name>": "<another key value>",
}
1 All options from flexVolume.options.
2 Value of flexVolume.fsType.
3 ro/rw based on flexVolume.readOnly.
4 All keys and their values from the secret referenced by flexVolume.secretRef.

OKD expects JSON data on standard output of the driver. When not specified, the output describes the result of the operation.

FlexVolume Driver Default Output
{
	"status": "<Success/Failure/Not supported>",
	"message": "<Reason for success/failure>"
}

Exit code of the driver should be 0 for success and 1 for error.

Operations should be idempotent, which means that the attachment of an already attached volume or the mounting of an already mounted volume should result in a successful operation.

The FlexVolume driver can work in two modes:

The attach/detach operation is used by the OKD master to attach a volume to a node and to detach it from a node. This is useful when a node becomes unresponsive from any reason. Then, the master can kill all pods on the node, detach all volumes from it, and attach the volumes to other nodes to resume the applications while the original node is still not reachable.

Not all storage back-end supports master-initiated detachment of a volume from another machine.

FlexVolume Drivers with Master-initiated Attach/Detach

A FlexVolume driver that supports master-controlled attach/detach must implement the following operations:

init

Initializes the driver. It is called during initialization of masters and nodes.

  • Arguments: none

  • Executed on: master, node

  • Expected output: default JSON

getvolumename

Returns the unique name of the volume. This name must be consistent among all masters and nodes, because it is used in subsequent detach call as <volume-name>. Any / characters in the <volume-name> are automatically replaced by ~.

  • Arguments: <json>

  • Executed on: master, node

  • Expected output: default JSON + volumeName:

    {
	"status": "Success",
	"message": "",
	"volumeName": "foo-volume-bar" (1)
}
    1 The unique name of the volume in storage back-end foo.
attach

Attaches a volume represented by the JSON to a given node. This operation should return the name of the device on the node if it is known (i.e. it has been assigned by the storage back-end before it runs). If the device is not known, the device must be found on the node by the subsequent waitforattach operation.

  • Arguments: <json> <node-name>

  • Executed on: master

  • Expected output: default JSON + device (if known):

    {
	"status": "Success",
	"message": "",
	"device": "/dev/xvda" (1)
}
    1 Name of the device on the node (if known).
waitforattach

Waits until a volume is fully attached to a node and its device emerges. If the previous attach operation has returned <device-name>, it is provided as an input parameter. Otherwise, <device-name> is empty and the operation must find the device on the node.

  • Arguments: <device-name> <json>

  • Executed on: node

  • Expected output: default JSON + device

    {
	"status": "Success",
	"message": "",
	"device": "/dev/xvda" (1)
}
    1 Name of the device on the node.
detach

Detaches the given volume from a node. <volume-name> is the name of the device returned by the getvolumename operation. Any / characters in the <volume-name> are automatically replaced by ~.

  • Arguments: <volume-name> <node-name>

  • Executed on: master

  • Expected output: default JSON

isattached

Checks that a volume is attached to a node.

  • Arguments: <json> <node-name>

  • Executed on: master

  • Expected output: default JSON + attached

    {
	"status": "Success",
	"message": "",
	"attached": true (1)
}
    1 Status of attachment of the volume to the node.
mountdevice

Mounts a volume’s device to a directory. <device-name> is name of the device as returned by the previous waitforattach operation.

  • Arguments: <mount-dir> <device-name> <json>

  • Executed on: node

  • Expected output: default JSON

unmountdevice

Unmounts a volume’s device from a directory.

  • Arguments: <mount-dir>

  • Executed on: node

All other operations should return JSON with {"status": "Not supported"} and exit code 1.

Master-initiated attach/detach operations are enabled by default in OKD 3.6. They may work in older versions, but must be explicitly enabled. See Enabling Controller-managed Attachment and Detachment. When not enabled, the attach/detach operations are initiated by a node where the volume should be attached to or detached from. Syntax and all parameters of FlexVolume driver invocations are the same in both cases.

FlexVolume Drivers Without Master-initiated Attach/Detach

FlexVolume drivers that do not support master-controlled attach/detach are executed only on the node and must implement these operations:

init

Initializes the driver. It is called during initialization of all nodes.

  • Arguments: none

  • Executed on: node

  • Expected output: default JSON

mount

Mounts a volume to directory. This can include anything that is necessary to mount the volume, including attaching the volume to the node, finding the its device, and then mounting the device.

  • Arguments: <mount-dir> <json>

  • Executed on: node

  • Expected output: default JSON

unmount

Unmounts a volume from a directory. This can include anything that is necessary to clean up the volume after unmounting, such as detaching the volume from the node.

  • Arguments: <mount-dir>

  • Executed on: node

  • Expected output: default JSON

All other operations should return JSON with {"status": "Not supported"} and exit code 1.