1. Documentation
×

AWS Elastic File Service CSI Driver Operator

Overview

OKD is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for AWS Elastic File Service (EFS).

Familiarity with persistent storage and configuring CSI volumes is recommended when working with a CSI Operator and driver.

After installing the AWS EFS CSI Driver Operator, OKD installs the AWS EFS CSI Operator and the AWS EFS CSI driver by default in the openshift-cluster-csi-drivers namespace. This allows the AWS EFS CSI Driver Operator to create CSI-provisioned PVs that mount to AWS EFS assets.

  • The AWS EFS CSI Driver Operator, after being installed, does not create a storage class by default to use to create persistent volume claims (PVCs). However, you can manually create the AWS EFS StorageClass. The AWS EFS CSI Driver Operator supports dynamic volume provisioning by allowing storage volumes to be created on-demand. This eliminates the need for cluster administrators to pre-provision storage.

  • The AWS EFS CSI driver enables you to create and mount AWS EFS PVs.

AWS EFS only supports regional volumes, not zonal volumes.

About CSI

Storage vendors have traditionally provided storage drivers as part of Kubernetes. With the implementation of the Container Storage Interface (CSI), third-party providers can instead deliver storage plugins using a standard interface without ever having to change the core Kubernetes code.

CSI Operators give OKD users storage options, such as volume snapshots, that are not possible with in-tree volume plugins.

Setting up the AWS EFS CSI Driver Operator

  1. If you are using AWS EFS with AWS Secure Token Service (STS), obtain a role Amazon Resource Name (ARN) for STS. This is required for installing the AWS EFS CSI Driver Operator.

  2. Install the AWS EFS CSI Driver Operator.

  3. Install the AWS EFS CSI Driver.

Obtaining a role Amazon Resource Name for Security Token Service

This procedure explains how to obtain a role Amazon Resource Name (ARN) to configure the AWS EFS CSI Driver Operator with OKD on AWS Security Token Service (STS).

Perform this procedure before you install the AWS EFS CSI Driver Operator (see Installing the AWS EFS CSI Driver Operator procedure).
Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

  • AWS account credentials

Procedure

You can obtain the ARN role in multiple ways. The following procedure shows one method that uses the same concept and CCO utility (ccoctl) binary tool as cluster installation.

To obtain a role ARN for configuring AWS EFS CSI Driver Operator using STS:

  1. Extract the ccoctl from the OKD release image, which you used to install the cluster with STS. For more information, see "Configuring the Cloud Credential Operator utility".

  2. Create and save an EFS CredentialsRequest YAML file, such as shown in the following example, and then place it in the credrequests directory:

    Example
    apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: openshift-aws-efs-csi-driver
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - elasticfilesystem:*
      effect: Allow
      resource: '*'
  secretRef:
    name: aws-efs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
  serviceAccountNames:
  - aws-efs-csi-driver-operator
  - aws-efs-csi-driver-controller-sa

  3. Run the ccoctl tool to generate a new IAM role in AWS, and create a YAML file for it in the local file system (<path_to_ccoctl_output_dir>/manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml).

    $ ccoctl aws create-iam-roles --name=<name> --region=<aws_region> --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

    • name=<name> is the name used to tag any cloud resources that are created for tracking.

    • region=<aws_region> is the AWS region where cloud resources are created.

    • dir=<path_to_directory_with_list_of_credentials_requests>/credrequests is the directory containing the EFS CredentialsRequest file in previous step.

    • <aws_account_id> is the AWS account ID.

      Example
      $ ccoctl aws create-iam-roles --name my-aws-efs --credentials-requests-dir credrequests --identity-provider-arn arn:aws:iam::123456789012:oidc-provider/my-aws-efs-oidc.s3.us-east-2.amazonaws.com
      Example output
      2022/03/21 06:24:44 Role arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud- created
2022/03/21 06:24:44 Saved credentials configuration to: /manifests/openshift-cluster-csi-drivers-aws-efs-cloud-credentials-credentials.yaml
2022/03/21 06:24:45 Updated Role policy for Role my-aws-efs-openshift-cluster-csi-drivers-aws-efs-cloud-

  4. Copy the role ARN from the first line of the Example output in the preceding step. The role ARN is between "Role" and "created". In this example, the role ARN is "arn:aws:iam::123456789012:role/my-aws-efs -openshift-cluster-csi-drivers-aws-efs-cloud".

    You will need the role ARN when you install the AWS EFS CSI Driver Operator.

Next steps

Install the AWS EFS CSI Driver Operator.

Additional resources

Installing the AWS EFS CSI Driver Operator

The AWS EFS CSI Driver Operator (a Red Hat Operator) is not installed in OKD by default. Use the following procedure to install and configure the AWS EFS CSI Driver Operator in your cluster.

Prerequisites

  • Access to the OKD web console.

Procedure

To install the AWS EFS CSI Driver Operator from the web console:

  1. Log in to the web console.

  2. Install the AWS EFS CSI Operator:

    1. Click Operators → OperatorHub.

    2. Locate the AWS EFS CSI Operator by typing AWS EFS CSI in the filter box.

    3. Click the AWS EFS CSI Driver Operator button.

    Be sure to select the AWS EFS CSI Driver Operator and not the AWS EFS Operator. The AWS EFS Operator is a community Operator and is not supported by Red Hat.

    1. On the AWS EFS CSI Driver Operator page, click Install.

    2. On the Install Operator page, ensure that:

      • All namespaces on the cluster (default) is selected.

      • Installed Namespace is set to openshift-cluster-csi-drivers.

    3. Click Install.

      After the installation finishes, the AWS EFS CSI Operator is listed in the Installed Operators section of the web console.

Next steps

Install the AWS EFS CSI Driver.

Installing the AWS EFS CSI Driver

After installing the AWS EFS CSI Driver Operator (a Red Hat operator), you install the AWS EFS CSI driver.

Prerequisites

  • Access to the OKD web console.

Procedure

  1. Click Administration → CustomResourceDefinitions → ClusterCSIDriver.

  2. On the Instances tab, click Create ClusterCSIDriver.

  3. Use the following YAML file:

    apiVersion: operator.openshift.io/v1
kind: ClusterCSIDriver
metadata:
    name: efs.csi.aws.com
spec:
  managementState: Managed

  4. Click Create.

  5. Wait for the following Conditions to change to a "True" status:

    • AWSEFSDriverNodeServiceControllerAvailable

    • AWSEFSDriverControllerServiceControllerAvailable

Creating the AWS EFS storage class

Storage classes are used to differentiate and delineate storage levels and usages. By defining a storage class, users can obtain dynamically provisioned persistent volumes.

The AWS EFS CSI Driver Operator (a Red Hat operator), after being installed, does not create a storage class by default. However, you can manually create the AWS EFS storage class.

Creating the AWS EFS storage class using the console

Procedure

  1. In the OKD console, click Storage → StorageClasses.

  2. On the StorageClasses page, click Create StorageClass.

  3. On the StorageClass page, perform the following steps:

    1. Enter a name to reference the storage class.

    2. Optional: Enter the description.

    3. Select the reclaim policy.

    4. Select efs.csi.aws.com from the Provisioner drop-down list.

    5. Optional: Set the configuration parameters for the selected provisioner.

  4. Click Create.

Creating the AWS EFS storage class using the CLI

Procedure

  • Create a StorageClass object:

    kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap (1)
  fileSystemId: fs-a5324911 (2)
  directoryPerms: "700" (3)
  gidRangeStart: "1000" (4)
  gidRangeEnd: "2000" (4)
  basePath: "/dynamic_provisioning" (5)
    1 provisioningMode must be efs-ap to enable dynamic provisioning.
    2 fileSystemId must be the ID of the EFS volume created manually.
    3 directoryPerms is the default permission of the root directory of the volume. In this example, the volume is accessible only by the owner.
    4 gidRangeStart and gidRangeEnd set the range of POSIX Group IDs (GIDs) that are used to set the GID of the AWS access point. If not specified, the default range is 50000-7000000. Each provisioned volume, and thus AWS access point, is assigned a unique GID from this range.
    5 basePath is the directory on the EFS volume that is used to create dynamically provisioned volumes. In this case, a PV is provisioned as “/dynamic_provisioning/<random uuid>” on the EFS volume. Only the subdirectory is mounted to pods that use the PV.

    A cluster admin can create several StorageClass objects, each using a different EFS volume.

AWS EFS CSI cross account support

Cross account support allows you to have an OKD cluster in one AWS account and mount your file system in another AWS account using the AWS Elastic File System (EFS) Container Storage Interface (CSI) driver.

Both the OKD cluster and EFS file system must be in the same region.
Prerequisites

  • Access to an OKD cluster with administrator rights

  • Two valid AWS accounts

Procedure

The following procedure demonstrates how to set up:

  • OKD cluster in AWS account A

  • Mount an AWS EFS file system in account B

To use AWS EFS across accounts:

  1. Install OKD cluster with AWS account A and install the EFS CSI Driver Operator.

  2. Create an EFS volume in AWS account B:

    1. Create a virtual private cloud (VPC) called, for example, "my-efs-vpc” with CIDR, for example, “172.20.0.0/16” and subnet for the AWS EFS volume.

    2. On the AWS console, go to https://console.aws.amazon.com/efs.

    3. Click Create new filesystem:

      1. Create a filesystem named, for example, "my-filesystem”.

      2. Select the VPC created earlier (“my-efs-vpc”).

      3. Accept the default for the remaining settings.

    4. Ensure that the volume and Mount Targets have been created:

      1. Check https://console.aws.amazon.com/efs#/file-systems.

      2. Click your volume, and on the Network tab wait for all Mount Targets to be available (approximately 1-2 minutes).

    5. On the Network tab, copy the Security Group ID. You will need it for the next step.

  3. Configure networking access to the AWS EFS volume on AWS account B:

    1. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups.

    2. Find the Security Group used by the AWS EFS volume by filtering for the group ID copied earlier.

    3. On the Inbound rules tab, click Edit inbound rules, and then add a new rule to allow OKD nodes to access the AWS EFS volumes (that is, use NFS ports from the cluster):

      • Type: NFS

      • Protocol: TCP

      • Port range: 2049

      • Source: Custom/IP address range of your OKD cluster nodes (for example, “10.0.0.0/16”)

    4. Save the rule.

      If you encounter mounting issues, re-check the port number, IP address range, and verify that the AWS EFS volume uses the expected security group.

  4. Create VPC peering between the OKD cluster VPC in AWS account A and the AWS EFS VPC in AWS account B:

    Ensure the two VPCs are using different network CIDRs, and after creating the VPC peering, add routes in each VPC to connect the two VPC networks.

    1. Create a peering connection called, for example, “my-efs-crossaccount-peering-connection” in account B. For the local VPC ID, use the EFS-located VPC. To peer with the VPC for account A, for the VPC ID use the OKD cluster VPC ID.

    2. Accept the peer connection in AWS account A.

    3. Modify the route table of each subnet (EFS-volume used subnets) in AWS account B:

      1. On the left pane, under Virtual private cloud, click the down arrow to expand the available options.

      2. Under Virtual private cloud, click Route tables".

      3. Click the Routes tab.

      4. Under Destination, enter 10.0.0.0/16.

      5. Under Target, use the peer connection type point from the created peer connection.

    4. Modify the route table of each subnet (OKD cluster nodes used subnets) in AWS account A:

      1. On the left pane, under Virtual private cloud, click the down arrow to expand the available options.

      2. Under Virtual private cloud, click Route tables".

      3. Click the Routes tab.

      4. Under Destination, enter the CIDR for the VPC in account B, which for this example is 172.20.0.0/16.

      5. Under Target, use the peer connection type point from the created peer connection.

  5. Create an IAM role, for example, “my-efs-acrossaccount-role” in AWS account B, which has a trust relationship with AWS account A, and add an inline AWS EFS policy with permissions to call “my-efs-acrossaccount-driver-policy”.

    This role is used by the CSI driver’s controller service running on the OKD cluster in AWS account A to determine the mount targets for your file system in AWS account B.

    # Trust relationships trusted entity trusted account A configuration on my-efs-acrossaccount-role in account B

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::301721915996:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

# my-cross-account-assume-policy policy attached to my-efs-acrossaccount-role in account B

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
    }
}

# my-efs-acrossaccount-driver-policy attached to my-efs-acrossaccount-role in account B

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSubnets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:DescribeMountTargets",
                "elasticfilesystem:DeleteAccessPoint",
                "elasticfilesystem:ClientMount",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:ClientWrite",
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:CreateAccessPoint"
            ],
            "Resource": [
                "arn:aws:elasticfilesystem:*:589722580343:access-point/*",
                "arn:aws:elasticfilesystem:*:589722580343:file-system/*"
            ]
        }
    ]
}

  6. In AWS account A, attach an inline policy to the IAM role of the AWS EFS CSI driver’s controller service account with the necessary permissions to perform Security Token Service (STS) assume role on the IAM role created earlier.

    # my-cross-account-assume-policy policy attached to Openshift cluster efs csi driver user in account A

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
    }
}

  7. In AWS account A, attach the AWS-managed policy “AmazonElasticFileSystemClientFullAccess” to OKD cluster master role. The role name is in the form <clusterID>-master-role (for example, my-0120ef-czjrl-master-role).

  8. Create a Kubernetes secret with awsRoleArn as the key and the role created earlier as the value:

    $ oc -n openshift-cluster-csi-drivers create secret generic my-efs-cross-account --from-literal=awsRoleArn='arn:aws:iam::589722580343:role/my-efs-acrossaccount-role'

    Since the driver controller needs to get the cross account role information from the secret, you need to add the secret role binding to the AWS EFS CSI driver controller ServiceAccount (SA):

    $ oc -n openshift-cluster-csi-drivers create role access-secrets --verb=get,list,watch --resource=secrets

$ oc -n openshift-cluster-csi-drivers create rolebinding --role=access-secrets default-to-secrets --serviceaccount=openshift-cluster-csi-drivers:aws-efs-csi-driver-controller-sa

  9. Create a filesystem policy for the file system (AWS EFS volume) in account B, which allows AWS account A to perform a mount on it.

    This step is not mandatory, but can be safer for AWS EFS volume usage.
    # EFS volume filesystem policy in account B
{
    "Version": "2012-10-17",
    "Id": "efs-policy-wizard-8089bf4a-9787-40f0-958e-bc2363012ace",
    "Statement": [
        {
            "Sid": "efs-statement-bd285549-cfa2-4f8b-861e-c372399fd238",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:ClientWrite",
                "elasticfilesystem:ClientMount"
            ],
            "Resource": "arn:aws:elasticfilesystem:us-east-2:589722580343:file-system/fs-091066a9bf9becbd5",
            "Condition": {
                "Bool": {
                    "elasticfilesystem:AccessedViaMountTarget": "true"
                }
            }
        },
        {
            "Sid": "efs-statement-03646e39-d80f-4daf-b396-281be1e43bab",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::589722580343:role/my-efs-acrossaccount-role"
            },
            "Action": [
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:ClientWrite",
                "elasticfilesystem:ClientMount"
            ],
            "Resource": "arn:aws:elasticfilesystem:us-east-2:589722580343:file-system/fs-091066a9bf9becbd5"
        }
    ]
}

  10. Create an AWS EFS volume storage class using a similar configuration to the following:

    # The cross account efs volume storageClass
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-cross-account-mount-sc
provisioner: efs.csi.aws.com
mountOptions:
  - tls
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-00f6c3ae6f06388bb
  directoryPerms: "700"
  gidRangeStart: "1000"
  gidRangeEnd: "2000"
  basePath: "/account-a-data"
  csi.storage.k8s.io/provisioner-secret-name: my-efs-cross-account
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-cluster-csi-drivers
volumeBindingMode: Immediate

Creating and configuring access to EFS volumes in AWS

This procedure explains how to create and configure EFS volumes in AWS so that you can use them in OKD.

Prerequisites

  • AWS account credentials

Procedure

To create and configure access to an EFS volume in AWS:

  1. On the AWS console, open https://console.aws.amazon.com/efs.

  2. Click Create file system:

    • Enter a name for the file system.

    • For Virtual Private Cloud (VPC), select your OKD’s' virtual private cloud (VPC).

    • Accept default settings for all other selections.

  3. Wait for the volume and mount targets to finish being fully created:

    1. Go to https://console.aws.amazon.com/efs#/file-systems.

    2. Click your volume, and on the Network tab wait for all mount targets to become available (~1-2 minutes).

  4. On the Network tab, copy the Security Group ID (you will need this in the next step).

  5. Go to https://console.aws.amazon.com/ec2/v2/home#SecurityGroups, and find the Security Group used by the EFS volume.

  6. On the Inbound rules tab, click Edit inbound rules, and then add a new rule with the following settings to allow OKD nodes to access EFS volumes :

    • Type: NFS

    • Protocol: TCP

    • Port range: 2049

    • Source: Custom/IP address range of your nodes (for example: “10.0.0.0/16”)

      This step allows OKD to use NFS ports from the cluster.

  7. Save the rule.

Dynamic provisioning for Amazon Elastic File Storage

The AWS EFS CSI driver supports a different form of dynamic provisioning than other CSI drivers. It provisions new PVs as subdirectories of a pre-existing EFS volume. The PVs are independent of each other. However, they all share the same EFS volume. When the volume is deleted, all PVs provisioned out of it are deleted too. The EFS CSI driver creates an AWS Access Point for each such subdirectory. Due to AWS AccessPoint limits, you can only dynamically provision 1000 PVs from a single StorageClass/EFS volume.

Note that PVC.spec.resources is not enforced by EFS.

In the example below, you request 5 GiB of space. However, the created PV is limitless and can store any amount of data (like petabytes). A broken application, or even a rogue application, can cause significant expenses when it stores too much data on the volume.

Using monitoring of EFS volume sizes in AWS is strongly recommended.
Prerequisites

  • You have created Amazon Elastic File Storage (Amazon EFS) volumes.

  • You have created the AWS EFS storage class.

Procedure

To enable dynamic provisioning:

  • Create a PVC (or StatefulSet or Template) as usual, referring to the StorageClass created previously.

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test
spec:
  storageClassName: efs-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

If you have problems setting up dynamic provisioning, see AWS EFS troubleshooting.

Additional resources

Creating static PVs with Amazon Elastic File Storage

It is possible to use an Amazon Elastic File Storage (Amazon EFS) volume as a single PV without any dynamic provisioning. The whole volume is mounted to pods.

Prerequisites

  • You have created Amazon EFS volumes.

Procedure

  • Create the PV using the following YAML file:

    apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity: (1)
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-ae66151a (2)
    volumeAttributes:
      encryptInTransit: "false" (3)
    1 spec.capacity does not have any meaning and is ignored by the CSI driver. It is used only when binding to a PVC. Applications can store any amount of data to the volume.
    2 volumeHandle must be the same ID as the EFS volume you created in AWS. If you are providing your own access point, volumeHandle should be <EFS volume ID>::<access point ID>. For example: fs-6e633ada::fsap-081a1d293f0004630.
    3 If desired, you can disable encryption in transit. Encryption is enabled by default.

If you have problems setting up static PVs, see AWS EFS troubleshooting.

Amazon Elastic File Storage security

The following information is important for Amazon Elastic File Storage (Amazon EFS) security.

When using access points, for example, by using dynamic provisioning as described earlier, Amazon automatically replaces GIDs on files with the GID of the access point. In addition, EFS considers the user ID, group ID, and secondary group IDs of the access point when evaluating file system permissions. EFS ignores the NFS client’s IDs. For more information about access points, see https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html.

As a consequence, EFS volumes silently ignore FSGroup; OKD is not able to replace the GIDs of files on the volume with FSGroup. Any pod that can access a mounted EFS access point can access any file on it.

Unrelated to this, encryption in transit is enabled by default. For more information, see https://docs.aws.amazon.com/efs/latest/ug/encryption-in-transit.html.

Amazon Elastic File Storage troubleshooting

The following information provides guidance on how to troubleshoot issues with Amazon Elastic File Storage (Amazon EFS):

  • The AWS EFS Operator and CSI driver run in namespace openshift-cluster-csi-drivers.

  • To initiate gathering of logs of the AWS EFS Operator and CSI driver, run the following command:

    $ oc adm must-gather
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:125f183d13601537ff15b3239df95d47f0a604da2847b561151fedd699f5e3a5
[must-gather      ] OUT namespace/openshift-must-gather-xm4wq created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-2bd8x created
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:125f183d13601537ff15b3239df95d47f0a604da2847b561151fedd699f5e3a5 created

  • To show AWS EFS Operator errors, view the ClusterCSIDriver status:

    $ oc get clustercsidriver efs.csi.aws.com -o yaml

  • If a volume cannot be mounted to a pod (as shown in the output of the following command):

    $ oc describe pod
...
  Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    2m13s  default-scheduler  Successfully assigned default/efs-app to ip-10-0-135-94.ec2.internal
  Warning  FailedMount  13s    kubelet            MountVolume.SetUp failed for volume "pvc-d7c097e6-67ec-4fae-b968-7e7056796449" : rpc error: code = DeadlineExceeded desc = context deadline exceeded (1)
  Warning  FailedMount  10s    kubelet            Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage kube-api-access-9j477]: timed out waiting for the condition
    1 Warning message indicating volume not mounted.

    This error is frequently caused by AWS dropping packets between an OKD node and Amazon EFS.

    Check that the following are correct:

    • AWS firewall and Security Groups

    • Networking: port number and IP addresses

Uninstalling the AWS EFS CSI Driver Operator

All EFS PVs are inaccessible after uninstalling the AWS EFS CSI Driver Operator (a Red Hat operator).

Prerequisites

  • Access to the OKD web console.

Procedure

To uninstall the AWS EFS CSI Driver Operator from the web console:

  1. Log in to the web console.

  2. Stop all applications that use AWS EFS PVs.

  3. Delete all AWS EFS PVs:

    1. Click Storage → PersistentVolumeClaims.

    2. Select each PVC that is in use by the AWS EFS CSI Driver Operator, click the drop-down menu on the far right of the PVC, and then click Delete PersistentVolumeClaims.

  4. Uninstall the AWS EFS CSI driver:

    Before you can uninstall the Operator, you must remove the CSI driver first.

    1. Click Administration → CustomResourceDefinitions → ClusterCSIDriver.

    2. On the Instances tab, for efs.csi.aws.com, on the far left side, click the drop-down menu, and then click Delete ClusterCSIDriver.

    3. When prompted, click Delete.

  5. Uninstall the AWS EFS CSI Operator:

    1. Click Operators → Installed Operators.

    2. On the Installed Operators page, scroll or type AWS EFS CSI into the Search by name box to find the Operator, and then click it.

    3. On the upper, right of the Installed Operators > Operator details page, click Actions → Uninstall Operator.

    4. When prompted on the Uninstall Operator window, click the Uninstall button to remove the Operator from the namespace. Any applications deployed by the Operator on the cluster need to be cleaned up manually.

      After uninstalling, the AWS EFS CSI Driver Operator is no longer listed in the Installed Operators section of the web console.

Before you can destroy a cluster (openshift-install destroy cluster), you must delete the EFS volume in AWS. An OKD cluster cannot be destroyed when there is an EFS volume that uses the cluster’s VPC. Amazon does not allow deletion of such a VPC.

Additional resources