Configuring KMS - External encryption key management | etcd

Enable KMS encryption
Rotate the Vault encryption key
Migrate from local encryption to KMS encryption
Monitor KMS encryption status
KMS encryption troubleshooting
Additional resources

You can configure external KMS encryption for etcd to centralize key management and meet regulatory compliance requirements.

Kubernetes KMS v2 is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Enable KMS encryption

You can enable external Key Management Service (KMS) encryption for etcd data to centralize key management and meet compliance requirements.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have enabled the TechPreviewNoUpgrade feature set to enable the KMSEncryption feature gate.
You have a HashiCorp Vault Enterprise instance accessible from your control plane nodes. Vault Community Edition is not available.
Your control plane nodes have network access to the Vault server.
You have configured your Vault instance with:
- Transit secrets engine enabled at the default transit/ mount path
- Encryption key created with type aes256-gcm96 (recommended for FIPS 140-3 compliance)
- Vault policy allowing the Kubernetes KMS v2 plugin to encrypt, decrypt, and read key information
- Authentication method (token, AppRole, or userpass) configured with the policy attached

The Kubernetes KMS v2 plugin requires a Vault policy with the following capabilities:

path "transit/encrypt/kms-key" {
  capabilities = ["update"]
}

path "transit/decrypt/kms-key" {
  capabilities = ["update"]
}

path "transit/keys/kms-key" {
  capabilities = ["read"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

Replace kms-key with your Vault Transit key name if using a different name.

Create an etcd backup before enabling KMS encryption.

Procedure

Deploy the KMS plugin on each control plane host as a static pod:
- Configure the plugin to listen at unix:///var/run/kmsplugin/kms.sock
- For your static pod, mount /var/run/kmsplugin as hostPath
- Configure KMS provider connection details and authentication credentials
The following steps show how to deploy the HashiCorp Vault KMS plugin as a static pod.

Create a static pod manifest file:

$ cat > /tmp/vault-kms-plugin.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: vault-kms-plugin
  namespace: kube-system
  labels:
    app: vault-kms-plugin
    tier: control-plane
spec:
  priorityClassName: system-node-critical
  hostNetwork: true
  containers:
  - name: vault-kms-plugin
    image: quay.io/redhat-isv-containers/698df066f8d1ddf179c15ef9:<version>
    command:
    - /vault-kubernetes-kms
    - -vault-address=<https://vault.example.com:8200>
    - -<vault_namespace>=admin
    - -auth-method=userpass
    - -userpass-username=<vault_username>
    - -userpass-password=<vault_password>
    - -transit-mount=transit
    - -transit-key=kms-key
    - -socket=unix:///var/run/kmsplugin/kms.sock
    volumeMounts:
    - name: kmsplugin
      mountPath: /var/run/kmsplugin
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi
    securityContext:
      privileged: true
  volumes:
  - name: kmsplugin
    hostPath:
      path: /var/run/kmsplugin
      type: DirectoryOrCreate
EOF

Replace the following values to match your environment:

<version>

The Vault KMS plugin image version tag. Use 0.1.0-beta-ubi.

<vault_username>

Your Vault username for authentication.

<vault_password>

Your Vault password for authentication.

https://vault.example.com:8200

The Vault address. Update this field to match your Vault server URL.

<vault_namespace>

Optional field.

The manifest uses the Red Hat certified container image from Quay.io (quay.io/redhat-isv-containers/698df066f8d1ddf179c15ef9), which is the recommended image for OKD.

Alternatively, you can use the HashiCorp image from Docker Hub by replacing the image field with docker.io/hashicorp/vault-kube-kms:0.1.0-beta-ubi.

Deploy the static pod manifest to each control plane node by running the following commands:

$ MANIFEST=$(cat /tmp/vault-kms-plugin.yaml | base64)
$ for node in $(oc get nodes --selector=node-role.kubernetes.io/master -o name | cut -d/ -f2); do
    echo "Deploying to $node..."
    oc debug node/$node -- chroot /host bash -c \
      "echo '$MANIFEST' | base64 -d > /etc/kubernetes/manifests/vault-kms-plugin.yaml"
  done

The kubelet automatically detects and starts static pods from /etc/kubernetes/manifests/.

Verify the static pods are running by entering the following command:

$ oc get pods -n kube-system -o wide | grep vault-kms

Example output

vault-kms-plugin-ip-10-0-16-7.compute.internal    1/1  Running  0  2m  10.0.16.7
vault-kms-plugin-ip-10-0-32-93.compute.internal   1/1  Running  0  2m  10.0.32.93
vault-kms-plugin-ip-10-0-69-106.compute.internal  1/1  Running  0  2m  10.0.69.106

Static pod names include the node name as a suffix. You should see one pod per control plane node.

Verify the socket exists on a control plane node by entering the following command:

$ oc debug node/<node_name> -- chroot /host ls -la /var/run/kmsplugin/kms.sock

Example output

srwxr-xr-x. 1 root root 0 <timestamp> /var/run/kmsplugin/kms.sock

Static pods are managed by kubelet on each node and cannot be deleted with oc delete pod. To remove a static pod, delete the manifest file from /etc/kubernetes/manifests/ on each control plane node.

Edit the APIServer custom resource by entering the following command:
```
$ oc edit apiserver cluster
```

Add the KMS configuration to the spec.encryption section:

apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  name: cluster
spec:
  encryption:
    type: KMS

Save and exit.

Migration begins automatically. The kube-apiserver, openshift-apiserver and oauth-apiserver Operators will restart and roll out new revisions.

The openshift-apiserver and authentication operators typically complete migration in 5-10 minutes. The kube-apiserver operator uses a conservative rollout strategy, updating one control plane node at a time and waiting for health checks before proceeding to the next node. This process can take 30 minutes or longer depending on cluster load.

Verification

Verify the encryption type by entering the following command:
```
$ oc get apiserver cluster -o jsonpath='{.spec.encryption.type}'
```
Output should show KMS.

Monitor the kube-apiserver rollout progress by entering the following command:

$ oc get kubeapiserver cluster -o jsonpath='{.status.nodeStatuses}' | jq -r '.[] | "\(.nodeName | split(".")[0]): current=\(.currentRevision) target=\(.targetRevision)"'

Example output during rollout

ip-10-0-16-166: current=10 target=0
ip-10-0-32-93: current=9 target=10
ip-10-0-69-106: current=9 target=0

The operator rolls out one node at a time. When all nodes show the same current revision and target is 0, the rollout is complete.

Check the encryption migration status by entering the following command:

$ oc get kubeapiserver cluster -o jsonpath='{.status.conditions[?(@.type=="Encrypted")]}' | jq .

Example output when complete

{
  "lastTransitionTime": "2026-05-15T17:39:02Z",
  "message": "All resources encrypted: secrets, configmaps",
  "reason": "EncryptionCompleted",
  "status": "True",
  "type": "Encrypted"
}

Wait for reason to show EncryptionCompleted before proceeding to verify encryption in etcd.

Verify secrets are encrypted in etcd:

Wait for the kube-apiserver rollout to complete on all control plane nodes before verifying encryption. During the rollout, API requests are distributed across nodes, and secrets created while some nodes are still on an earlier revision will not be encrypted with Kubernetes KMS v2.

Create a test secret by entering the following command:

$ oc create secret generic test-secret --from-literal=key=value -n default

Get an etcd pod name by entering the following command:

$ oc get pods -n openshift-etcd -l app=etcd -o name | head -1

Check the secret data in etcd by entering the following command:

$ oc exec -n openshift-etcd <etcd_pod_name> -- etcdctl get /kubernetes.io/secrets/default/test-secret --print-value-only | hexdump -C | head -1

Output should begin with k8s:enc:kms:v2: followed by encrypted binary data.

Delete the test secret by entering the following command:
```
$ oc delete secret test-secret -n default
```

Additional resources

Rotate the Vault encryption key

You can rotate your Vault Transit encryption key to generate a new key version while maintaining access to data encrypted with earlier versions.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have access to your Vault instance with permissions to rotate keys.
Kubernetes KMS v2 encryption is enabled and functioning.

Procedure

Rotate the Vault encryption key by entering the following command:
```
$ vault write -f transit/keys/kms-key/rotate
```
Vault creates a new key version while maintaining earlier versions for decryption. The API server automatically uses the correct key version for each secret.
Verify the new key version by entering the following command:
```
$ vault read transit/keys/kms-key
```
The latest_version field shows the current key version number.

Verification

Verify that existing secrets remain accessible by entering the following command:
```
$ oc get secret -A
```

All secrets should be readable without errors.

Existing encrypted secrets do not need re-encryption. Vault maintains all key versions and automatically uses the appropriate version for decryption.

Additional resources

Migrate from local encryption to KMS encryption

You can migrate from local etcd encryption to external KMS encryption to centralize key management and improve compliance.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have enabled the TechPreviewNoUpgrade feature set to enable the KMSEncryption feature gate.
Your cluster is currently using aescbc or aesgcm encryption.
You have deployed the KMS plugin on all control plane nodes.
Control plane nodes have network access to the KMS provider.

Create an etcd backup before migrating.

Procedure

Back up the current etcd encryption configuration by entering the following command:
```
$ oc get apiserver cluster -o yaml > apiserver-backup.yaml
```
Edit the APIServer custom resource by entering the following command:
```
$ oc edit apiserver cluster
```

Change the encryption type from aescbc or aesgcm to KMS:

apiVersion: config.openshift.io/v1
kind: APIServer
metadata:
  name: cluster
spec:
  encryption:
    type: KMS

Save and exit.

Migration starts automatically and typically takes several minutes.

Verify migration completion for all API servers by running the following commands:

$ oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

$ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{end}'

All outputs should show EncryptionCompleted.

Monitor KMS encryption status

You can monitor KMS encryption status by using Operator and API server logs to verify successful configuration and detect issues.

Procedure

Check the kube-apiserver operator logs for KMS-related events by entering the following command:

$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i kms

View API server logs for KMS-related events by entering the following command:

$ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=100 | grep -i kms

Verify the KMS encryption configuration by entering the following command:
```
$ oc get apiserver cluster -o jsonpath='{.spec.encryption}' | jq
```

Additional resources

Kubernetes KMS provider documentation

KMS encryption troubleshooting

You can diagnose and resolve common KMS encryption issues to maintain secure key management and cluster availability.

Invalid KMS configuration

Symptom: APIServer resource shows validation errors during KMS encryption configuration.

Diagnosis: Check kube-apiserver Operator logs:

$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i "kms\|validation\|error"

Solutions:

Verify plugin configuration follows provider requirements
Ensure all required fields are specified

Verify plugin is running on all control plane nodes:

$ oc debug node/<node_name> -- chroot /host ls -la /var/run/kmsplugin/kms.sock

KMS permissions errors

Symptom: Encryption migration fails with permission errors.

Diagnosis: Check Operator and plugin logs:

$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i "kms\|permission\|access denied"

$ oc debug node/<node_name> -- chroot /host journalctl -u kms-plugin

Solutions:

Verify plugin has valid authentication credentials
Check if credentials have expired
Ensure plugin principal has encrypt and decrypt permissions
Verify KMS provider key policy allows plugin access
Confirm encryption key is enabled and not scheduled for deletion

Expired or deleted KMS key

Symptom: API server cannot decrypt secrets when accessing encrypted resources.

Diagnosis: Check logs and verify key status:

$ oc logs -n openshift-kube-apiserver -l apiserver=true | grep -i "decrypt\|kms.*error"

$ oc debug node/<node_name> -- chroot /host journalctl -u kms-plugin | grep -i "key\|error"

Solutions:

Re-enable the encryption key if disabled
Restore from backup if key was permanently deleted
Cancel key deletion if scheduled
Ensure KMS provider maintains access to previous key versions

Deleted KMS keys prevent data recovery. Align key retention with backup policies.

API server degraded or unavailable

Symptom: API server becomes degraded or unresponsive after enabling KMS encryption.

Diagnosis: Check Operator status and logs:

$ oc get clusteroperator kube-apiserver

$ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=200 | grep -i kms

Solutions:

Check network connectivity between control plane and KMS provider
Verify network policies, firewalls, and routes allow communication
Monitor KMS provider rate limits and request increases if needed
Verify DNS resolution and TLS certificate validation

Confirm plugin is running on all control plane nodes:

$ oc debug node/<node_name> -- chroot /host systemctl status kms-plugin

Encryption migration stuck or slow

Symptom: KMS encryption migration takes unusually long or becomes stuck.

Diagnosis: Check Operator status and migration logs:

$ oc get clusteroperator kube-apiserver

$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator | grep -i migration

Solutions:

Migration time depends on data size; monitor progress
Monitor KMS provider audit logs for rate limiting or throttling events
Check network performance between control plane and KMS provider

Collecting debug information

Collect cluster logs:

$ oc adm must-gather

$ oc get apiserver cluster -o yaml > apiserver.yaml

$ oc logs -n openshift-kube-apiserver-operator deploy/kube-apiserver-operator > kube-apiserver-operator.log

$ oc logs -n openshift-kube-apiserver -l apiserver=true --tail=500 > kube-apiserver.log

Collect KMS provider information:

KMS plugin logs from control plane nodes
KMS provider audit logs
KMS provider key policy and permissions
Authentication credentials status
Network connectivity test results

Redact credentials, tokens, and sensitive data before sharing logs.

Configuring Kubernetes KMS v2

Enable KMS encryption

Rotate the Vault encryption key

Migrate from local encryption to KMS encryption

Monitor KMS encryption status

KMS encryption troubleshooting

Invalid KMS configuration

KMS permissions errors

Expired or deleted KMS key

API server degraded or unavailable

Encryption migration stuck or slow

Collecting debug information

Additional resources