Manually create IAM

The Cloud Credential Operator (CCO) can be put into manual mode prior to installation in environments where the cloud identity and access management (IAM) APIs are not reachable, or the administrator prefers not to store an administrator-level credential secret in the cluster kube-system namespace.

Procedure
  1. Run the OKD installer to generate manifests:

    $ openshift-install create manifests --dir=<installation_directory> (1)
    1 For <installation_directory>, specify the directory name to store the files that the installation program creates.
  2. Insert a ConfigMap into the manifests directory so that the Cloud Credential Operator is placed in manual mode:

    $ cat <<EOF > mycluster/manifests/cco-configmap.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cloud-credential-operator-config
      namespace: openshift-cloud-credential-operator
      annotations:
        release.openshift.io/create-only: "true"
    data:
      disabled: "true"
    EOF
  3. Remove the admin credential secret created using your local cloud credentials. This removal prevents your admin credential from being stored in the cluster:

    $ rm mycluster/openshift/99_cloud-creds-secret.yaml
  4. Obtain the OKD release image your openshift-install binary is built to use:

    $ openshift-install version
    Example output
    release image quay.io/openshift-release-dev/ocp-release:4.y.z-x86_64
  5. Locate all CredentialsRequests in this release image that target the cloud you are deploying on:

    $ oc adm release extract quay.io/openshift-release-dev/ocp-release:4.y.z-x86_64 --credentials-requests --cloud=azure

    This displays the details for each request.

    Sample CredentialsRequest
    apiVersion: cloudcredential.openshift.io/v1
    kind: CredentialsRequest
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: openshift-image-registry-azure
      namespace: openshift-cloud-credential-operator
    spec:
      secretRef:
        name: installer-cloud-credentials
        namespace: openshift-image-registry
      providerSpec:
        apiVersion: cloudcredential.openshift.io/v1
        kind: AzureProviderSpec
        roleBindings:
        - role: Contributor
  6. Create YAML files for secrets in the openshift-install manifests directory that you generated previously. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each credentialsRequest. The format for the secret data varies for each cloud provider.

  7. Proceed with cluster creation:

    $ openshift-install create cluster --dir=<installation_directory>

    Before upgrading a cluster that uses manually maintained credentials, you must ensure that the CCO is in an upgradeable state. For details, see the Upgrading clusters with manually maintained credentials section of the installation content for your cloud provider.

Admin credentials root secret format

Each cloud provider uses a credentials root secret in the kube-system namespace by convention, which is then used to satisfy all CredentialsRequests and create their respective secrets. This is done either by minting new credentials, with mint mode, or by copying the credentials root secret, with passthrough mode.

The format for the secret varies by cloud, and is also used for each CredentialsRequest secret.

Microsoft Azure secret format
apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: azure-credentials
stringData:
  azure_subscription_id: <SubscriptionID>
  azure_client_id: <ClientID>
  azure_client_secret: <ClientSecret>
  azure_tenant_id: <TenantID>
  azure_resource_prefix: <ResourcePrefix>
  azure_resourcegroup: <ResourceGroup>
  azure_region: <Region>

On Microsoft Azure, the credentials secret format includes two properties that must contain the cluster’s infrastructure ID, generated randomly for each cluster installation. This value can be found after running create manifests:

$ cat .openshift_install_state.json | jq '."*installconfig.ClusterID".InfraID' -r
Example output
mycluster-2mpcn

This value would be used in the secret data as follows:

azure_resource_prefix: mycluster-2mpcn
azure_resourcegroup: mycluster-2mpcn-rg

Upgrading clusters with manually maintained credentials

If credentials are added in a future release, the Cloud Credential Operator (CCO) upgradable status for a cluster with manually maintained credentials changes to false. For minor release, for example, from 4.5 to 4.6, this status prevents you from upgrading until you have addressed any updated permissions. For z-stream releases, for example, from 4.5.10 to 4.5.11, the upgrade is not blocked, but the credentials must still be updated for the new release.

Use the Administrator perspective of the web console to determine if the CCO is upgradeable.

  1. Navigate to AdministrationCluster Settings.

  2. To view the CCO status details, click cloud-credential in the Cluster Operators list.

  3. If the Upgradeable status in the Conditions section is False, examine the credentialsRequests for the new release and update the manually maintained credentials on your cluster to match before upgrading.

In addition to creating new credentials for the release image that you are upgrading to, you must review the required permissions for existing credentials and accommodate any new permissions requirements for existing components in the new release. The CCO cannot detect these mismatches and will not set upgradable to false in this case.

The Manually creating IAM section of the installation content for your cloud provider explains how to obtain and use the credentials required for your cloud.

Mint Mode

Mint Mode is supported for AWS, GCP, and Azure.

The default and recommended best practice for running OKD is to run the installer with an administrator-level cloud credential. The admin credential is stored in the kube-system namespace, and then used by the Cloud Credential Operator to process the CredentialRequests in the cluster and create new users for each with specific permissions.

The benefits of Mint Mode include:

  • Each cluster component only has the permissions it requires.

  • Automatic, on-going reconciliation for cloud credentials including upgrades, which might require additional credentials or permissions.

One drawback is that Mint Mode requires admin credential storage in a cluster kube-system secret.