×

During installation, you can configure the Cloud Credential Operator (CCO) to operate in manual mode and use the CCO utility (ccoctl) to implement short-term security credentials for individual components that are created and managed outside the OKD cluster.

This credentials strategy is supported for Amazon Web Services (AWS), Google Cloud Platform (GCP), and global Microsoft Azure only. The strategy must be configured during installation of a new OKD cluster. You cannot configure an existing cluster that uses a different credentials strategy to use this feature.

Cloud providers use different terms for their implementation of this authentication method.

Table 1. Short-term credentials provider terminology
Cloud provider Provider nomenclature

Amazon Web Services (AWS)

AWS Security Token Service (STS)

Google Cloud Platform (GCP)

GCP Workload Identity

Global Microsoft Azure

Azure AD Workload Identity

AWS Security Token Service

In manual mode with STS, the individual OKD cluster components use the AWS Security Token Service (STS) to assign components IAM roles that provide short-term, limited-privilege security credentials. These credentials are associated with IAM roles that are specific to each component that makes AWS API calls.

AWS Security Token Service authentication process

The following diagram details the authentication flow between AWS and the OKD cluster when using AWS STS.

Detailed authentication flow between AWS and the cluster when using AWS STS
Figure 1. AWS Security Token Service authentication flow

AWS component secret formats

Using manual mode with the AWS Security Token Service (STS) changes the content of the AWS credentials that are provided to individual OKD components. Compare the following secret formats:

AWS secret format using long-term credentials
apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  aws_access_key_id: <base64_encoded_access_key_id>
  aws_secret_access_key: <base64_encoded_secret_access_key>
1 The namespace for the component.
2 The name of the component secret.
AWS secret format using AWS STS
apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
stringData:
  credentials: |-
    [default]
    sts_regional_endpoints = regional
    role_name: <operator_role_name> (3)
    web_identity_token_file: <path_to_token> (4)
1 The namespace for the component.
2 The name of the component secret.
3 The IAM role for the component.
4 The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

AWS component secret permissions requirements

OKD components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

These permissions apply to all resources. Unless specified, there are no request conditions on these permissions.

Component Custom resource Required permissions for services

Cluster CAPI Operator

openshift-cluster-api-aws

EC2

  • ec2:CreateTags

  • ec2:DescribeAvailabilityZones

  • ec2:DescribeDhcpOptions

  • ec2:DescribeImages

  • ec2:DescribeInstances

  • ec2:DescribeInternetGateways

  • ec2:DescribeSecurityGroups

  • ec2:DescribeSubnets

  • ec2:DescribeVpcs

  • ec2:DescribeNetworkInterfaces

  • ec2:DescribeNetworkInterfaceAttribute

  • ec2:ModifyNetworkInterfaceAttribute

  • ec2:RunInstances

  • ec2:TerminateInstances

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers

  • elasticloadbalancing:DescribeTargetGroups

  • elasticloadbalancing:DescribeTargetHealth

  • elasticloadbalancing:RegisterInstancesWithLoadBalancer

  • elasticloadbalancing:RegisterTargets

  • elasticloadbalancing:DeregisterTargets

Identity and Access Management (IAM)

  • iam:PassRole

  • iam:CreateServiceLinkedRole

Key Management Service (KMS)

  • kms:Decrypt

  • kms:Encrypt

  • kms:GenerateDataKey

  • kms:GenerateDataKeyWithoutPlainText

  • kms:DescribeKey

  • kms:RevokeGrant[1]

  • kms:CreateGrant [1]

  • kms:ListGrants [1]

Machine API Operator

openshift-machine-api-aws

EC2

  • ec2:CreateTags

  • ec2:DescribeAvailabilityZones

  • ec2:DescribeDhcpOptions

  • ec2:DescribeImages

  • ec2:DescribeInstances

  • ec2:DescribeInstanceTypes

  • ec2:DescribeInternetGateways

  • ec2:DescribeSecurityGroups

  • ec2:DescribeRegions

  • ec2:DescribeSubnets

  • ec2:DescribeVpcs

  • ec2:RunInstances

  • ec2:TerminateInstances

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers

  • elasticloadbalancing:DescribeTargetGroups

  • elasticloadbalancing:DescribeTargetHealth

  • elasticloadbalancing:RegisterInstancesWithLoadBalancer

  • elasticloadbalancing:RegisterTargets

  • elasticloadbalancing:DeregisterTargets

Identity and Access Management (IAM)

  • iam:PassRole

  • iam:CreateServiceLinkedRole

Key Management Service (KMS)

  • kms:Decrypt

  • kms:Encrypt

  • kms:GenerateDataKey

  • kms:GenerateDataKeyWithoutPlainText

  • kms:DescribeKey

  • kms:RevokeGrant[1]

  • kms:CreateGrant [1]

  • kms:ListGrants [1]

Cloud Credential Operator

cloud-credential-operator-iam-ro

Identity and Access Management (IAM)

  • iam:GetUser

  • iam:GetUserPolicy

  • iam:ListAccessKeys

Cluster Image Registry Operator

openshift-image-registry

S3

  • s3:CreateBucket

  • s3:DeleteBucket

  • s3:PutBucketTagging

  • s3:GetBucketTagging

  • s3:PutBucketPublicAccessBlock

  • s3:GetBucketPublicAccessBlock

  • s3:PutEncryptionConfiguration

  • s3:GetEncryptionConfiguration

  • s3:PutLifecycleConfiguration

  • s3:GetLifecycleConfiguration

  • s3:GetBucketLocation

  • s3:ListBucket

  • s3:GetObject

  • s3:PutObject

  • s3:DeleteObject

  • s3:ListBucketMultipartUploads

  • s3:AbortMultipartUpload

  • s3:ListMultipartUploadParts

Ingress Operator

openshift-ingress

Elastic load balancing

  • elasticloadbalancing:DescribeLoadBalancers

Route 53

  • route53:ListHostedZones

  • route53:ListTagsForResources

  • route53:ChangeResourceRecordSets

Tag

  • tag:GetResources

Security Token Service (STS)

  • sts:AssumeRole

Cluster Network Operator

openshift-cloud-network-config-controller-aws

EC2

  • ec2:DescribeInstances

  • ec2:DescribeInstanceStatus

  • ec2:DescribeInstanceTypes

  • ec2:UnassignPrivateIpAddresses

  • ec2:AssignPrivateIpAddresses

  • ec2:UnassignIpv6Addresses

  • ec2:AssignIpv6Addresses

  • ec2:DescribeSubnets

  • ec2:DescribeNetworkInterfaces

AWS Elastic Block Store CSI Driver Operator

aws-ebs-csi-driver-operator

EC2

  • ec2:AttachVolume

  • ec2:CreateSnapshot

  • ec2:CreateTags

  • ec2:CreateVolume

  • ec2:DeleteSnapshot

  • ec2:DeleteTags

  • ec2:DeleteVolume

  • ec2:DescribeInstances

  • ec2:DescribeSnapshots

  • ec2:DescribeTags

  • ec2:DescribeVolumes

  • ec2:DescribeVolumesModifications

  • ec2:DetachVolume

  • ec2:ModifyVolume

  • ec2:DescribeAvailabilityZones

  • ec2:EnableFastSnapshotRestores

Key Management Service (KMS)

  • kms:ReEncrypt*

  • kms:Decrypt

  • kms:Encrypt

  • kms:GenerateDataKey

  • kms:GenerateDataKeyWithoutPlainText

  • kms:DescribeKey

  • kms:RevokeGrant[1]

  • kms:CreateGrant [1]

  • kms:ListGrants [1]

  1. Request condition: kms:GrantIsForAWSResource: true

GCP Workload Identity

In manual mode with GCP Workload Identity, the individual OKD cluster components use the GCP workload identity provider to allow components to impersonate GCP service accounts using short-term, limited-privilege credentials.

GCP Workload Identity authentication process

The following diagram details the authentication flow between GCP and the OKD cluster when using GCP Workload Identity.

Detailed authentication flow between GCP and the cluster when using GCP Workload Identity
Figure 2. GCP Workload Identity authentication flow

GCP component secret formats

Using manual mode with GCP Workload Identity changes the content of the GCP credentials that are provided to individual OKD components. Compare the following secret content:

GCP secret format
apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  service_account.json: <service_account> (3)
1 The namespace for the component.
2 The name of the component secret.
3 The Base64 encoded service account.
Content of the Base64 encoded service_account.json file using long-term credentials
{
   "type": "service_account", (1)
   "project_id": "<project_id>",
   "private_key_id": "<private_key_id>",
   "private_key": "<private_key>", (2)
   "client_email": "<client_email_address>",
   "client_id": "<client_id>",
   "auth_uri": "https://accounts.google.com/o/oauth2/auth",
   "token_uri": "https://oauth2.googleapis.com/token",
   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<client_email_address>"
}
1 The credential type is service_account.
2 The private RSA key that is used to authenticate to GCP. This key must be kept secure and is not rotated.
Content of the Base64 encoded service_account.json file using GCP Workload Identity
{
   "type": "external_account", (1)
   "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", (2)
   "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
   "token_url": "https://sts.googleapis.com/v1/token",
   "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/<client_email_address>:generateAccessToken", (3)
   "credential_source": {
      "file": "<path_to_token>", (4)
      "format": {
         "type": "text"
      }
   }
}
1 The credential type is external_account.
2 The target audience is the GCP Workload Identity provider.
3 The resource URL of the service account that can be impersonated with these credentials.
4 The path to the service account token inside the pod. By convention, this is /var/run/secrets/openshift/serviceaccount/token for OKD components.

Azure AD Workload Identity

In manual mode with Azure AD Workload Identity, the individual OKD cluster components use the Azure AD workload identity provider to assign components short-term security credentials.

Azure component secret formats

Additional resources

Using manual mode with AD Workload Identity changes the content of the Azure credentials that are provided to individual OKD components. Compare the following secret formats:

Azure secret format using long-term credentials
apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  azure_client_id: <client_id> (3)
  azure_client_secret: <client_secret> (4)
  azure_region: <region>
  azure_resource_prefix: <resource_group_prefix> (5)
  azure_resourcegroup: <resource_group_prefix>-rg (6)
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque
1 The namespace for the component.
2 The name of the component secret.
3 The client ID of the Azure AD identity that the component uses to authenticate.
4 The component secret that is used to authenticate with Azure AD for the <client_id> identity.
5 The resource group prefix.
6 The resource group. This value is formed by the <resource_group_prefix> and the suffix -rg.
Azure secret format using AD Workload Identity
apiVersion: v1
kind: Secret
metadata:
  namespace: <target_namespace> (1)
  name: <target_secret_name> (2)
data:
  azure_client_id: <client_id> (3)
  azure_federated_token_file: <path_to_token_file> (4)
  azure_region: <region>
  azure_subscription_id: <subscription_id>
  azure_tenant_id: <tenant_id>
type: Opaque
1 The namespace for the component.
2 The name of the component secret.
3 The client ID of the user-assigned managed identity that the component uses to authenticate.
4 The path to the mounted service account token file.

Azure component secret permissions requirements

OKD components require the following permissions. These values are in the CredentialsRequest custom resource (CR) for each component.

Component Custom resource Required permissions for services

Cloud Controller Manager Operator

openshift-azure-cloud-controller-manager

  • Microsoft.Compute/virtualMachines/read

  • Microsoft.Network/loadBalancers/read

  • Microsoft.Network/loadBalancers/write

  • Microsoft.Network/networkInterfaces/read

  • Microsoft.Network/networkSecurityGroups/read

  • Microsoft.Network/networkSecurityGroups/write

  • Microsoft.Network/publicIPAddresses/join/action

  • Microsoft.Network/publicIPAddresses/read

  • Microsoft.Network/publicIPAddresses/write

Cluster CAPI Operator

openshift-cluster-api-azure

role: Contributor [1]

Machine API Operator

openshift-machine-api-azure

  • Microsoft.Compute/availabilitySets/delete

  • Microsoft.Compute/availabilitySets/read

  • Microsoft.Compute/availabilitySets/write

  • Microsoft.Compute/diskEncryptionSets/read

  • Microsoft.Compute/disks/delete

  • Microsoft.Compute/galleries/images/versions/read

  • Microsoft.Compute/skus/read

  • Microsoft.Compute/virtualMachines/delete

  • Microsoft.Compute/virtualMachines/extensions/delete

  • Microsoft.Compute/virtualMachines/extensions/read

  • Microsoft.Compute/virtualMachines/extensions/write

  • Microsoft.Compute/virtualMachines/read

  • Microsoft.Compute/virtualMachines/write

  • Microsoft.ManagedIdentity/userAssignedIdentities/assign/action

  • Microsoft.Network/applicationSecurityGroups/read

  • Microsoft.Network/loadBalancers/backendAddressPools/join/action

  • Microsoft.Network/loadBalancers/read

  • Microsoft.Network/loadBalancers/write

  • Microsoft.Network/networkInterfaces/delete

  • Microsoft.Network/networkInterfaces/join/action

  • Microsoft.Network/networkInterfaces/loadBalancers/read

  • Microsoft.Network/networkInterfaces/read

  • Microsoft.Network/networkInterfaces/write

  • Microsoft.Network/networkSecurityGroups/read

  • Microsoft.Network/networkSecurityGroups/write

  • Microsoft.Network/publicIPAddresses/delete

  • Microsoft.Network/publicIPAddresses/join/action

  • Microsoft.Network/publicIPAddresses/read

  • Microsoft.Network/publicIPAddresses/write

  • Microsoft.Network/routeTables/read

  • Microsoft.Network/virtualNetworks/delete

  • Microsoft.Network/virtualNetworks/read

  • Microsoft.Network/virtualNetworks/subnets/join/action

  • Microsoft.Network/virtualNetworks/subnets/read

  • Microsoft.Resources/subscriptions/resourceGroups/read

Cluster Image Registry Operator

openshift-image-registry-azure

Data permissions

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action

  • Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action

General permissions

  • Microsoft.Storage/storageAccounts/blobServices/read

  • Microsoft.Storage/storageAccounts/blobServices/containers/read

  • Microsoft.Storage/storageAccounts/blobServices/containers/write

  • Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action

  • Microsoft.Storage/storageAccounts/read

  • Microsoft.Storage/storageAccounts/write

  • Microsoft.Storage/storageAccounts/delete

  • Microsoft.Storage/storageAccounts/listKeys/action

  • Microsoft.Resources/tags/write

Ingress Operator

openshift-ingress-azure

  • Microsoft.Network/dnsZones/A/delete

  • Microsoft.Network/dnsZones/A/write

  • Microsoft.Network/privateDnsZones/A/delete

  • Microsoft.Network/privateDnsZones/A/write

Cluster Network Operator

openshift-cloud-network-config-controller-azure

  • Microsoft.Network/networkInterfaces/read

  • Microsoft.Network/networkInterfaces/write

  • Microsoft.Compute/virtualMachines/read

  • Microsoft.Network/virtualNetworks/read

  • Microsoft.Network/virtualNetworks/subnets/join/action

  • Microsoft.Network/loadBalancers/backendAddressPools/join/action

Azure File CSI Driver Operator

azure-file-csi-driver-operator

  • Microsoft.Network/networkSecurityGroups/join/action

  • Microsoft.Network/virtualNetworks/subnets/read

  • Microsoft.Network/virtualNetworks/subnets/write

  • Microsoft.Storage/storageAccounts/delete

  • Microsoft.Storage/storageAccounts/fileServices/read

  • Microsoft.Storage/storageAccounts/fileServices/shares/delete

  • Microsoft.Storage/storageAccounts/fileServices/shares/read

  • Microsoft.Storage/storageAccounts/fileServices/shares/write

  • Microsoft.Storage/storageAccounts/listKeys/action

  • Microsoft.Storage/storageAccounts/read

  • Microsoft.Storage/storageAccounts/write

Azure Disk CSI Driver Operator

azure-disk-csi-driver-operator

  • Microsoft.Compute/disks/*

  • Microsoft.Compute/snapshots/*

  • Microsoft.Compute/virtualMachineScaleSets/*/read

  • Microsoft.Compute/virtualMachineScaleSets/read

  • Microsoft.Compute/virtualMachineScaleSets/virtualMachines/write

  • Microsoft.Compute/virtualMachines/*/read

  • Microsoft.Compute/virtualMachines/write

  • Microsoft.Resources/subscriptions/resourceGroups/read

  1. This component requires a role rather than a set of permissions.