The NMState Operator is supported on Google Cloud Platform (GCP) only for configuring IPsec. |
With IPsec enabled, you can encrypt both internal pod-to-pod cluster traffic between nodes and external traffic between pods and IPsec endpoints external to your cluster. All pod-to-pod network traffic between nodes on the OVN-Kubernetes cluster network is encrypted with IPsec in Transport mode.
IPsec is disabled by default. It can be enabled either during or after installing the cluster. For information about cluster installation, see OKD installation overview.
If your cluster uses hosted control planes for Red Hat OKD, IPsec is not supported for IPsec encryption of either pod-to-pod or traffic to external hosts. |
IPsec on IBM Cloud® supports only NAT-T. Using ESP is not supported. |
Use the procedures in the following documentation to:
Enable and disable IPSec after cluster installation
Configure IPsec encryption for traffic between the cluster and external hosts
Verify that IPsec encrypts traffic between pods on different nodes
When using IPsec on your OKD cluster, you can choose from the following operating modes:
Mode | Description | Default |
---|---|---|
|
No traffic is encrypted. This is the cluster default. |
Yes |
|
Pod-to-pod traffic is encrypted as described in "Types of network traffic flows encrypted by pod-to-pod IPsec". Traffic to external nodes may be encrypted after you complete the required configuration steps for IPsec. |
No |
|
Traffic to external nodes may be encrypted after you complete the required configuration steps for IPsec. |
No |
For IPsec support for encrypting traffic to external hosts, ensure that the following prerequisites are met:
The OVN-Kubernetes network plugin must be configured in local gateway mode, where ovnKubernetesConfig.gatewayConfig.routingViaHost=true
.
The NMState Operator is installed. This Operator is required for specifying the IPsec configuration. For more information, see About the Kubernetes NMState Operator.
The NMState Operator is supported on Google Cloud Platform (GCP) only for configuring IPsec. |
The Butane tool (butane
) is installed. To install Butane, see Installing Butane.
These prerequisites are required to add certificates into the host NSS database and to configure IPsec to communicate with external hosts.
You must configure the network connectivity between machines to allow OKD cluster components to communicate. Each machine must be able to resolve the hostnames of all other machines in the cluster.
Protocol | Port | Description |
---|---|---|
UDP |
|
IPsec IKE packets |
|
IPsec NAT-T packets |
|
ESP |
N/A |
IPsec Encapsulating Security Payload (ESP) |
For IPsec encryption of pod-to-pod traffic, the following sections describe which specific pod-to-pod traffic is encrypted, what kind of encryption protocol is used, and how X.509 certificates are handled. These sections do not apply to IPsec encryption between the cluster and external hosts, which you must configure manually for your specific external network infrastructure.
With IPsec enabled, only the following network traffic flows between pods are encrypted:
Traffic between pods on different nodes on the cluster network
Traffic from a pod on the host network to a pod on the cluster network
The following traffic flows are not encrypted:
Traffic between pods on the same node on the cluster network
Traffic between pods on the host network
Traffic from a pod on the cluster network to a pod on the host network
The encrypted and unencrypted flows are illustrated in the following diagram:
The encrypt cipher used is AES-GCM-16-256
. The integrity check value (ICV) is 16
bytes. The key length is 256
bits.
The IPsec mode used is Transport mode, a mode that encrypts end-to-end communication by adding an Encapsulated Security Payload (ESP) header to the IP header of the original packet and encrypts the packet data. OKD does not currently use or support IPsec Tunnel mode for pod-to-pod communication.
The Cluster Network Operator (CNO) generates a self-signed X.509 certificate authority (CA) that is used by IPsec for encryption. Certificate signing requests (CSRs) from each node are automatically fulfilled by the CNO.
The CA is valid for 10 years. The individual node certificates are valid for 5 years and are automatically rotated after 4 1/2 years elapse.
OKD supports IPsec encryption for traffic to external hosts with TLS certificates that you must supply.
This feature is supported on the following platforms:
Bare metal
Google Cloud Platform (GCP)
OpenStack
VMware vSphere
If you have Fedora worker nodes, these do not support IPsec encryption for external traffic. |
If your cluster uses hosted control planes for Red Hat OKD, configuring IPsec for encrypting traffic to external hosts is not supported.
Ensure that the following prohibitions are observed:
IPv6 configuration is not currently supported by the NMState Operator when configuring IPsec for external traffic.
Certificate common names (CN) in the provided certificate bundle must not begin with the ovs_
prefix, because this naming can conflict with pod-to-pod IPsec CN names in the Network Security Services (NSS) database of each node.
As a cluster administrator, you can enable pod-to-pod IPsec encryption and IPsec encryption between the cluster and external IPsec endpoints.
You can configure IPsec in either of the following modes:
Full
: Encryption for pod-to-pod and external traffic
External
: Encryption for external traffic
If you need to configure encryption for external traffic in addition to pod-to-pod traffic, you must also complete the "Configuring IPsec encryption for external traffic" procedure.
Install the OpenShift CLI (oc
).
You are logged in to the cluster as a user with cluster-admin
privileges.
You have reduced the size of your cluster MTU by 46
bytes to allow for the overhead of the IPsec ESP header.
To enable IPsec encryption, enter the following command:
$ oc patch networks.operator.openshift.io cluster --type=merge \
-p '{
"spec":{
"defaultNetwork":{
"ovnKubernetesConfig":{
"ipsecConfig":{
"mode":<mode>
}}}}}'
where:
mode
Specify External
to encrypt only traffic to external hosts or specify Full
to encrypt pod to pod traffic and optionally traffic to external hosts. By default, IPsec is disabled.
Optional: If you need to encrypt traffic to external hosts, complete the "Configuring IPsec encryption for external traffic" procedure.
To find the names of the OVN-Kubernetes data plane pods, enter the following command:
$ oc get pods -n openshift-ovn-kubernetes -l=app=ovnkube-node
ovnkube-node-5xqbf 8/8 Running 0 28m
ovnkube-node-6mwcx 8/8 Running 0 29m
ovnkube-node-ck5fr 8/8 Running 0 31m
ovnkube-node-fr4ld 8/8 Running 0 26m
ovnkube-node-wgs4l 8/8 Running 0 33m
ovnkube-node-zfvcl 8/8 Running 0 34m
Verify that IPsec is enabled on your cluster by running the following command:
As a cluster administrator, you can verify that IPsec is enabled between pods on your cluster when IPsec is configured in |
$ oc -n openshift-ovn-kubernetes rsh ovnkube-node-<XXXXX> ovn-nbctl --no-leader-only get nb_global . ipsec
where:
<XXXXX>
Specifies the random sequence of letters for a pod from the previous step.
true
As a cluster administrator, to encrypt external traffic with IPsec you must configure IPsec for your network infrastructure, including providing PKCS#12 certificates. Because this procedure uses Butane to create machine configs, you must have the butane
command installed.
After you apply the machine config, the Machine Config Operator reboots affected nodes in your cluster to rollout the new machine config. |
Install the OpenShift CLI (oc
).
You have installed the butane
utility on your local computer.
You have installed the NMState Operator on the cluster.
You are logged in to the cluster as a user with cluster-admin
privileges.
You have an existing PKCS#12 certificate for the IPsec endpoint and a CA cert in PEM format.
You enabled IPsec in either Full
or External
mode on your cluster.
The OVN-Kubernetes network plugin must be configured in local gateway mode, where ovnKubernetesConfig.gatewayConfig.routingViaHost=true
.
Create an IPsec configuration with an NMState Operator node network configuration policy. For more information, see Libreswan as an IPsec VPN implementation.
To identify the IP address of the cluster node that is the IPsec endpoint, enter the following command:
$ oc get nodes
Create a file named ipsec-config.yaml
that contains a node network configuration policy for the NMState Operator, such as in the following examples. For an overview about NodeNetworkConfigurationPolicy
objects, see The Kubernetes NMState project.
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ipsec-config
spec:
nodeSelector:
kubernetes.io/hostname: "<hostname>" (1)
desiredState:
interfaces:
- name: <interface_name> (2)
type: ipsec
libreswan:
left: <cluster_node> (3)
leftid: '%fromcert'
leftrsasigkey: '%cert'
leftcert: left_server
leftmodecfgclient: false
right: <external_host> (4)
rightid: '%fromcert'
rightrsasigkey: '%cert'
rightsubnet: <external_address>/32 (5)
ikev2: insist
type: transport
1 | Specifies the host name to apply the policy to. This host serves as the left side host in the IPsec configuration. |
2 | Specifies the name of the interface to create on the host. |
3 | Specifies the host name of the cluster node that terminates the IPsec tunnel on the cluster side. The name should match SAN [Subject Alternate Name] from your supplied PKCS#12 certificates. |
4 | Specifies the external host name, such as host.example.com . The name should match the SAN [Subject Alternate Name] from your supplied PKCS#12 certificates. |
5 | Specifies the IP address of the external host, such as 10.1.2.3/32 . |
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
name: ipsec-config
spec:
nodeSelector:
kubernetes.io/hostname: "<hostname>" (1)
desiredState:
interfaces:
- name: <interface_name> (2)
type: ipsec
libreswan:
left: <cluster_node> (3)
leftid: '%fromcert'
leftmodecfgclient: false
leftrsasigkey: '%cert'
leftcert: left_server
right: <external_host> (4)
rightid: '%fromcert'
rightrsasigkey: '%cert'
rightsubnet: <external_address>/32 (5)
ikev2: insist
type: tunnel
1 | Specifies the host name to apply the policy to. This host serves as the left side host in the IPsec configuration. |
2 | Specifies the name of the interface to create on the host. |
3 | Specifies the host name of the cluster node that terminates the IPsec tunnel on the cluster side. The name should match SAN [Subject Alternate Name] from your supplied PKCS#12 certificates. |
4 | Specifies the external host name, such as host.example.com . The name should match the SAN [Subject Alternate Name] from your supplied PKCS#12 certificates. |
5 | Specifies the IP address of the external host, such as 10.1.2.3/32 . |
To configure the IPsec interface, enter the following command:
$ oc create -f ipsec-config.yaml
Provide the following certificate files to add to the Network Security Services (NSS) database on each host. These files are imported as part of the Butane configuration in subsequent steps.
left_server.p12
: The certificate bundle for the IPsec endpoints
ca.pem
: The certificate authority that you signed your certificates with
Create a machine config to add your certificates to the cluster:
To create Butane config files for the control plane and worker nodes, enter the following command:
$ for role in master worker; do
cat >> "99-ipsec-${role}-endpoint-config.bu" <<-EOF
variant: openshift
version: 4.17.0
metadata:
name: 99-${role}-import-certs
labels:
machineconfiguration.openshift.io/role: $role
systemd:
units:
- name: ipsec-import.service
enabled: true
contents: |
[Unit]
Description=Import external certs into ipsec NSS
Before=ipsec.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/ipsec-addcert.sh
RemainAfterExit=false
StandardOutput=journal
[Install]
WantedBy=multi-user.target
storage:
files:
- path: /etc/pki/certs/ca.pem
mode: 0400
overwrite: true
contents:
local: ca.pem
- path: /etc/pki/certs/left_server.p12
mode: 0400
overwrite: true
contents:
local: left_server.p12
- path: /usr/local/bin/ipsec-addcert.sh
mode: 0740
overwrite: true
contents:
inline: |
#!/bin/bash -e
echo "importing cert to NSS"
certutil -A -n "CA" -t "CT,C,C" -d /var/lib/ipsec/nss/ -i /etc/pki/certs/ca.pem
pk12util -W "" -i /etc/pki/certs/left_server.p12 -d /var/lib/ipsec/nss/
certutil -M -n "left_server" -t "u,u,u" -d /var/lib/ipsec/nss/
EOF
done
To transform the Butane files that you created in the previous step into machine configs, enter the following command:
$ for role in master worker; do
butane 99-ipsec-${role}-endpoint-config.bu -o ./99-ipsec-$role-endpoint-config.yaml
done
To apply the machine configs to your cluster, enter the following command:
$ for role in master worker; do
oc apply -f 99-ipsec-${role}-endpoint-config.yaml
done
As the Machine Config Operator (MCO) updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated before external IPsec connectivity is available. |
Check the machine config pool status by entering the following command:
$ oc get mcp
A successfully updated node has the following status: UPDATED=true
, UPDATING=false
, DEGRADED=false
.
By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster. |
To confirm that IPsec machine configs rolled out successfully, enter the following commands:
Confirm that the IPsec machine configs were created:
$ oc get mc | grep ipsec
80-ipsec-master-extensions 3.2.0 6d15h
80-ipsec-worker-extensions 3.2.0 6d15h
Confirm that the that the IPsec extension are applied to control plane nodes:
$ oc get mcp master -o yaml | grep 80-ipsec-master-extensions -c
2
Confirm that the that the IPsec extension are applied to worker nodes:
$ oc get mcp worker -o yaml | grep 80-ipsec-worker-extensions -c
2
For more information about the nmstate IPsec API, see IPsec Encryption
As a cluster administrator, you can remove an existing IPsec tunnel to an external host.
Install the OpenShift CLI (oc
).
You are logged in to the cluster as a user with cluster-admin
privileges.
You enabled IPsec in either Full
or External
mode on your cluster.
Create a file named remove-ipsec-tunnel.yaml
with the following YAML:
kind: NodeNetworkConfigurationPolicy
apiVersion: nmstate.io/v1
metadata:
name: <name>
spec:
nodeSelector:
kubernetes.io/hostname: <node_name>
desiredState:
interfaces:
- name: <tunnel_name>
type: ipsec
state: absent
where:
name
Specifies a name for the node network configuration policy.
node_name
Specifies the name of the node where the IPsec tunnel that you want to remove exists.
tunnel_name
Specifies the interface name for the existing IPsec tunnel.
To remove the IPsec tunnel, enter the following command:
$ oc apply -f remove-ipsec-tunnel.yaml
As a cluster administrator, you can disable IPsec encryption.
Install the OpenShift CLI (oc
).
Log in to the cluster with a user with cluster-admin
privileges.
To disable IPsec encryption, enter the following command:
$ oc patch networks.operator.openshift.io cluster --type=merge \
-p '{
"spec":{
"defaultNetwork":{
"ovnKubernetesConfig":{
"ipsecConfig":{
"mode":"Disabled"
}}}}}'
Optional: You can increase the size of your cluster MTU by 46
bytes because there is no longer any overhead from the IPsec ESP header in IP packets.