1. Documentation
×

Using manual mode with Amazon Web Services Security Token Service

About manual mode with AWS Security Token Service

In manual mode with STS, the individual OKD cluster components use AWS Security Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

AWS Security Token Service authentication process

The AWS Security Token Service (STS) and the AssumeRole API action allow pods to retrieve access keys that are defined by an IAM role policy.

The OKD cluster includes a Kubernetes service account signing service. This service uses a private key to sign service account JSON web tokens (JWT). A pod that requires a service account token requests one through the pod specification. When the pod is created and assigned to a node, the node retrieves a signed service account from the service account signing service and mounts it onto the pod.

Clusters that use STS contain an IAM role ID in their Kubernetes configuration secrets. Workloads assume the identity of this IAM role ID. The signed service account token issued to the workload aligns with the configuration in AWS, which allows AWS STS to grant access keys for the specified IAM role to the workload.

AWS STS grants access keys only for requests that include service account tokens that meet the following conditions:

  • The token name and namespace match the service account name and namespace.

  • The token is signed by a key that matches the public key. The public key pair for the service account signing key used by the cluster is stored in an AWS S3 bucket. AWS STS federation validates that the service account token signature aligns with the public key stored in the S3 bucket.

Authentication flow for AWS STS

The following diagram illustrates the authentication flow between AWS and the OKD cluster when using AWS STS.

  • Token signing is the Kubernetes service account signing service on the OKD cluster.

  • The Kubernetes service account in the pod is the signed service account token.

Detailed authentication flow between AWS and the cluster when using AWS STS
Figure 1. AWS Security Token Service authentication flow

Requests for new and refreshed credentials are automated by using an appropriately configured AWS IAM OpenID Connect (OIDC) identity provider combined with AWS IAM roles. Service account tokens that are trusted by AWS IAM are signed by OKD and can be projected into a pod and used for authentication.

Token refreshing for AWS STS

The signed service account token that a pod uses expires after a period of time. For clusters that use AWS STS, this time period is 3600 seconds, or one hour.

The kubelet on the node that the pod is assigned to ensures that the token is refreshed. The kubelet attempts to rotate a token when it is older than 80 percent of its time to live.

OpenID Connect requirements for AWS STS

You can store the public portion of the encryption keys for your OIDC configuration in a public or private S3 bucket.

The OIDC spec requires the use of HTTPS. AWS services require a public endpoint to expose the OIDC documents in the form of JSON web key set (JWKS) public keys. This allows AWS services to validate the bound tokens signed by Kubernetes and determine whether to trust certificates. As a result, both S3 bucket options require a public HTTPS endpoint and private endpoints are not supported.

To use AWS STS, the public AWS backbone for the AWS STS service must be able to communicate with a public S3 bucket or a private S3 bucket with a public CloudFront endpoint. You can choose which type of bucket to use when you process CredentialsRequest objects during installation:

  • By default, the CCO utility (ccoctl) stores the OIDC configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint.

  • As an alternative, you can have the ccoctl utility store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL.

AWS component secret formats

Using manual mode with STS changes the content of the AWS credentials that are provided to individual OKD components. Compare the following secret formats:

AWS secret format using long-lived credentials
apiVersion: v1
kind: Secret
metadata:
  namespace: <target-namespace> (1)
  name: <target-secret-name> (2)
data:
  aws_access_key_id: <base64-encoded-access-key-id>
  aws_secret_access_key: <base64-encoded-secret-access-key>
1 The namespace for the component.
2 The name of the component secret.
AWS secret format with STS
apiVersion: v1
kind: Secret
metadata:
  namespace: <target-namespace> (1)
  name: <target-secret-name> (2)
stringData:
  credentials: |-
    [default]
    sts_regional_endpoints = regional
    role_name: <operator-role-name> (3)
    web_identity_token_file: <path-to-token> (4)
1 The namespace for the component.
2 The name of the component secret.
3 The IAM role for the component.
4 The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

Installing an OKD cluster configured for manual mode with STS

To install a cluster that is configured to use the Cloud Credential Operator (CCO) in manual mode with STS:

  1. Configure the Cloud Credential Operator utility.

  2. Create the required AWS resources individually, or with a single command.

  3. Run the OKD installer.

  4. Verify that the cluster is using short-lived credentials.

Because the cluster is operating in manual mode when using STS, it is not able to create new credentials for components with the permissions that they require. When upgrading to a different minor version of OKD, there are often new AWS permission requirements. Before upgrading a cluster that is using STS, the cluster administrator must manually ensure that the AWS permissions are sufficient for existing components and available to any new components.
Additional resources

Configuring the Cloud Credential Operator utility

To create and manage cloud credentials from outside of the cluster when the Cloud Credential Operator (CCO) is operating in manual mode, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl utility is a Linux binary that must run in a Linux environment.
Prerequisites

  • You have access to an OKD account with cluster administrator access.

  • You have installed the OpenShift CLI (oc).

  • You have created an AWS account for the ccoctl utility to use with the following permissions:

    Table 1. Required AWS permissions
    Permission type Required permissions

    iam permissions

    • iam:CreateOpenIDConnectProvider

    • iam:CreateRole

    • iam:DeleteOpenIDConnectProvider

    • iam:DeleteRole

    • iam:DeleteRolePolicy

    • iam:GetOpenIDConnectProvider

    • iam:GetRole

    • iam:GetUser

    • iam:ListOpenIDConnectProviders

    • iam:ListRolePolicies

    • iam:ListRoles

    • iam:PutRolePolicy

    • iam:TagOpenIDConnectProvider

    • iam:TagRole

    s3 permissions

    • s3:CreateBucket

    • s3:DeleteBucket

    • s3:DeleteObject

    • s3:GetBucketAcl

    • s3:GetBucketTagging

    • s3:GetObject

    • s3:GetObjectAcl

    • s3:GetObjectTagging

    • s3:ListBucket

    • s3:PutBucketAcl

    • s3:PutBucketPolicy

    • s3:PutBucketPublicAccessBlock

    • s3:PutBucketTagging

    • s3:PutObject

    • s3:PutObjectAcl

    • s3:PutObjectTagging

    cloudfront permissions

    • cloudfront:ListCloudFrontOriginAccessIdentities

    • cloudfront:ListDistributions

    • cloudfront:ListTagsForResource

    If you plan to store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL, the AWS account that runs the ccoctl utility requires the following additional permissions:

    • cloudfront:CreateCloudFrontOriginAccessIdentity

    • cloudfront:CreateDistribution

    • cloudfront:DeleteCloudFrontOriginAccessIdentity

    • cloudfront:DeleteDistribution

    • cloudfront:GetCloudFrontOriginAccessIdentity

    • cloudfront:GetCloudFrontOriginAccessIdentityConfig

    • cloudfront:GetDistribution

    • cloudfront:TagResource

    • cloudfront:UpdateDistribution

    These additional permissions support the use of the --create-private-s3-bucket option when processing credentials requests with the ccoctl aws create-all command.
Procedure

  1. Obtain the OKD release image by running the following command:

    $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

  2. Obtain the CCO container image from the OKD release image by running the following command:

    $ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)

    Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.

  3. Extract the ccoctl binary from the CCO container image within the OKD release image by running the following command:

    $ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret

  4. Change the permissions to make ccoctl executable by running the following command:

    $ chmod 775 ccoctl
Verification

  • To verify that ccoctl is ready to use, display the help file. Use a relative file name when you run the command, for example:

    $ ./ccoctl.rhel9
    Example output
    OpenShift credentials provisioning tool

Usage:
  ccoctl [command]

Available Commands:
  alibabacloud Manage credentials objects for alibaba cloud
  aws          Manage credentials objects for AWS cloud
  gcp          Manage credentials objects for Google cloud
  help         Help about any command
  ibmcloud     Manage credentials objects for IBM Cloud
  nutanix      Manage credentials objects for Nutanix

Flags:
  -h, --help   help for ccoctl

Use "ccoctl [command] --help" for more information about a command.

Creating AWS resources with the Cloud Credential Operator utility

You can use the CCO utility (ccoctl) to create the required AWS resources individually, or with a single command.

Creating AWS resources individually

If you need to review the JSON files that the ccoctl tool creates before modifying AWS resources, or if the process the ccoctl tool uses to create AWS resources automatically does not meet the requirements of your organization, you can create the AWS resources individually. For example, this option might be useful for an organization that shares the responsibility for creating these resources among different users or departments.

Otherwise, you can use the ccoctl aws create-all command to create the AWS resources automatically.

By default, ccoctl creates objects in the directory in which the commands are run. To create the objects in a different directory, use the --output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this directory.

Some ccoctl commands make AWS API calls to create or modify AWS resources. You can use the --dry-run flag to avoid making API calls. Using this flag creates JSON files on the local file system instead. You can review and modify the JSON files and then apply them with the AWS CLI tool using the --cli-input-json parameters.
Prerequisites

  • Extract and prepare the ccoctl binary.

Procedure

  1. Generate the public and private RSA key files that are used to set up the OpenID Connect provider for the cluster:

    $ ccoctl aws create-key-pair
    Example output:
    2021/04/13 11:01:02 Generating RSA keypair
2021/04/13 11:01:03 Writing private key to /<path_to_ccoctl_output_dir>/serviceaccount-signer.private
2021/04/13 11:01:03 Writing public key to /<path_to_ccoctl_output_dir>/serviceaccount-signer.public
2021/04/13 11:01:03 Copying signing key for use by installer

    where serviceaccount-signer.private and serviceaccount-signer.public are the generated key files.

    This command also creates a private key that the cluster requires during installation in /<path_to_ccoctl_output_dir>/tls/bound-service-account-signing-key.key.

  2. Create an OpenID Connect identity provider and S3 bucket on AWS:

    $ ccoctl aws create-identity-provider \
  --name=<name> \
  --region=<aws_region> \
  --public-key-file=<path_to_ccoctl_output_dir>/serviceaccount-signer.public

    where:

    • <name> is the name used to tag any cloud resources that are created for tracking.

    • <aws-region> is the AWS region in which cloud resources will be created.

    • <path_to_ccoctl_output_dir> is the path to the public key file that the ccoctl aws create-key-pair command generated.

    Example output:
    2021/04/13 11:16:09 Bucket <name>-oidc created
2021/04/13 11:16:10 OpenID Connect discovery document in the S3 bucket <name>-oidc at .well-known/openid-configuration updated
2021/04/13 11:16:10 Reading public key
2021/04/13 11:16:10 JSON web key set (JWKS) in the S3 bucket <name>-oidc at keys.json updated
2021/04/13 11:16:18 Identity Provider created with ARN: arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

    where openid-configuration is a discovery document and keys.json is a JSON web key set file.

    This command also creates a YAML configuration file in /<path_to_ccoctl_output_dir>/manifests/cluster-authentication-02-config.yaml. This file sets the issuer URL field for the service account tokens that the cluster generates, so that the AWS IAM identity provider trusts the tokens.

  3. Create IAM roles for each component in the cluster.

    1. Extract the list of CredentialsRequest objects from the OKD release image:

      $ oc adm release extract \
  --from=$RELEASE_IMAGE \
  --credentials-requests \
  --cloud=aws \
  --to=<path_to_directory_with_list_of_credentials_requests>/credrequests (1)
      1 credrequests is the directory where the list of CredentialsRequest objects is stored. This command creates the directory if it does not exist.

    2. If your cluster uses cluster capabilities to disable one or more optional components, delete the CredentialsRequest custom resources for any disabled components.

      Example credrequests directory contents for OKD 4.13 on AWS
      0000_30_cluster-api_00_credentials-request.yaml (1)
0000_30_machine-api-operator_00_credentials-request.yaml (2)
0000_50_cloud-credential-operator_05-iam-ro-credentialsrequest.yaml (3)
0000_50_cluster-image-registry-operator_01-registry-credentials-request.yaml (4)
0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml (5)
0000_50_cluster-network-operator_02-cncc-credentials.yaml (6)
0000_50_cluster-storage-operator_03_credentials_request_aws.yaml (7)
      1 For clusters that use the TechPreviewNoUpgrade feature set, the Cluster API Operator CR is required.
      2 The Machine API Operator CR is required.
      3 The Cloud Credential Operator CR is required.
      4 The Image Registry Operator CR is required.
      5 The Ingress Operator CR is required.
      6 The Network Operator CR is required.
      7 The Storage Operator CR is an optional component and might be disabled in your cluster.

    3. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

      $ ccoctl aws create-iam-roles \
  --name=<name> \
  --region=<aws_region> \
  --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
  --identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com

      For AWS environments that use alternative IAM API endpoints, such as GovCloud, you must also specify your region with the --region parameter.

      If your cluster uses Technology Preview features that are enabled by the TechPreviewNoUpgrade feature set, you must include the --enable-tech-preview parameter.

      For each CredentialsRequest object, ccoctl creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy as defined in each CredentialsRequest object from the OKD release image.

Verification

  • To verify that the OKD secrets are created, list the files in the <path_to_ccoctl_output_dir>/manifests directory:

    $ ls <path_to_ccoctl_output_dir>/manifests
    Example output:
    cluster-authentication-02-config.yaml
openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
openshift-cloud-network-config-controller-cloud-credentials-credentials.yaml
openshift-cluster-api-capa-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
openshift-image-registry-installer-cloud-credentials-credentials.yaml
openshift-ingress-operator-cloud-credentials-credentials.yaml
openshift-machine-api-aws-cloud-credentials-credentials.yaml

    You can verify that the IAM roles are created by querying AWS. For more information, refer to AWS documentation on listing IAM roles.

Creating AWS resources with a single command

If you do not need to review the JSON files that the ccoctl tool creates before modifying AWS resources, and if the process the ccoctl tool uses to create AWS resources automatically meets the requirements of your organization, you can use the ccoctl aws create-all command to automate the creation of AWS resources.

Otherwise, you can create the AWS resources individually.

By default, ccoctl creates objects in the directory in which the commands are run. To create the objects in a different directory, use the --output-dir flag. This procedure uses <path_to_ccoctl_output_dir> to refer to this directory.
Prerequisites

You must have:

  • Extracted and prepared the ccoctl binary.

Procedure

  1. Set the $RELEASE_IMAGE variable by running the following command:

    $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

  2. Extract the list of CredentialsRequest objects from the OKD release image by running the following command:

    $ oc adm release extract \
  --from=$RELEASE_IMAGE \
  --credentials-requests \
  --cloud=aws \
  --to=<path_to_directory_with_list_of_credentials_requests>/credrequests (1)
    1 credrequests is the directory where the list of CredentialsRequest objects is stored. This command creates the directory if it does not exist.

    This command can take a few moments to run.

  3. If your cluster uses cluster capabilities to disable one or more optional components, delete the CredentialsRequest custom resources for any disabled components.

    Example credrequests directory contents for OKD 4.13 on AWS
    0000_30_cluster-api_00_credentials-request.yaml (1)
0000_30_machine-api-operator_00_credentials-request.yaml (2)
0000_50_cloud-credential-operator_05-iam-ro-credentialsrequest.yaml (3)
0000_50_cluster-image-registry-operator_01-registry-credentials-request.yaml (4)
0000_50_cluster-ingress-operator_00-ingress-credentials-request.yaml (5)
0000_50_cluster-network-operator_02-cncc-credentials.yaml (6)
0000_50_cluster-storage-operator_03_credentials_request_aws.yaml (7)
    1 For clusters that use the TechPreviewNoUpgrade feature set, the Cluster API Operator CR is required.
    2 The Machine API Operator CR is required.
    3 The Cloud Credential Operator CR is required.
    4 The Image Registry Operator CR is required.
    5 The Ingress Operator CR is required.
    6 The Network Operator CR is required.
    7 The Storage Operator CR is an optional component and might be disabled in your cluster.

  4. Use the ccoctl tool to process all CredentialsRequest objects in the credrequests directory:

    $ ccoctl aws create-all \
  --name=<name> \(1)
  --region=<aws_region> \(2)
  --credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \(3)
  --output-dir=<path_to_ccoctl_output_dir> \(4)
  --create-private-s3-bucket (5)
    1 Specify the name used to tag any cloud resources that are created for tracking.
    2 Specify the AWS region in which cloud resources will be created.
    3 Specify the directory containing the files for the component CredentialsRequest objects.
    4 Optional: Specify the directory in which you want the ccoctl utility to create objects. By default, the utility creates objects in the directory in which the commands are run.
    5 Optional: By default, the ccoctl utility stores the OpenID Connect (OIDC) configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint. To store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL instead, use the --create-private-s3-bucket parameter.

    If your cluster uses Technology Preview features that are enabled by the TechPreviewNoUpgrade feature set, you must include the --enable-tech-preview parameter.
Verification

  • To verify that the OKD secrets are created, list the files in the <path_to_ccoctl_output_dir>/manifests directory:

    $ ls <path_to_ccoctl_output_dir>/manifests
    Example output:
    cluster-authentication-02-config.yaml
openshift-cloud-credential-operator-cloud-credential-operator-iam-ro-creds-credentials.yaml
openshift-cloud-network-config-controller-cloud-credentials-credentials.yaml
openshift-cluster-api-capa-manager-bootstrap-credentials-credentials.yaml
openshift-cluster-csi-drivers-ebs-cloud-credentials-credentials.yaml
openshift-image-registry-installer-cloud-credentials-credentials.yaml
openshift-ingress-operator-cloud-credentials-credentials.yaml
openshift-machine-api-aws-cloud-credentials-credentials.yaml

    You can verify that the IAM roles are created by querying AWS. For more information, refer to AWS documentation on listing IAM roles.

Running the installer

Prerequisites

  • Configure an account with the cloud platform that hosts your cluster.

  • Obtain the OKD release image.

Procedure

  1. Change to the directory that contains the installation program and create the install-config.yaml file:

    $ openshift-install create install-config --dir <installation_directory>

    where <installation_directory> is the directory in which the installation program creates files.

  2. Edit the install-config.yaml configuration file so that it contains the credentialsMode parameter set to Manual.

    Example install-config.yaml configuration file
    apiVersion: v1
baseDomain: cluster1.example.com
credentialsMode: Manual (1)
compute:
- architecture: amd64
  hyperthreading: Enabled
    1 This line is added to set the credentialsMode parameter to Manual.

  3. Create the required OKD installation manifests:

    $ openshift-install create manifests

  4. Copy the manifests that ccoctl generated to the manifests directory that the installation program created:

    $ cp /<path_to_ccoctl_output_dir>/manifests/* ./manifests/

  5. Copy the private key that the ccoctl generated in the tls directory to the installation directory:

    $ cp -a /<path_to_ccoctl_output_dir>/tls .

  6. Run the OKD installer:

    $ ./openshift-install create cluster

Verifying the installation

  1. Connect to the OKD cluster.

  2. Verify that the cluster does not have root credentials:

    $ oc get secrets -n kube-system aws-creds

    The output should look similar to:

    Error from server (NotFound): secrets "aws-creds" not found

  3. Verify that the components are assuming the IAM roles that are specified in the secret manifests, instead of using credentials that are created by the CCO:

    Example command with the Image Registry Operator
    $ oc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r .data.credentials | base64 --decode

    The output should show the role and web identity token that are used by the component and look similar to:

    Example output with the Image Registry Operator
    [default]
role_arn = arn:aws:iam::123456789:role/openshift-image-registry-installer-cloud-credentials
web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token

Additional resources