vCenter requirements
Before you install an OKD cluster on your vCenter that uses infrastructure that you provided, you must prepare your environment.
Required vCenter account privileges
To install an OKD cluster in a vCenter, your vSphere account must include privileges for reading and creating the required resources. Using an account that has global administrative privileges is the simplest way to access all of the necessary permissions.
Roles and privileges required for installation in vSphere API
vSphere object for role |
When required |
Required privileges in vSphere API |
|
|
Cns.Searchable
InventoryService.Tagging.AttachTag
InventoryService.Tagging.CreateCategory
InventoryService.Tagging.CreateTag
InventoryService.Tagging.DeleteCategory
InventoryService.Tagging.DeleteTag
InventoryService.Tagging.EditCategory
InventoryService.Tagging.EditTag
Sessions.ValidateSession
StorageProfile.Update
StorageProfile.View
|
|
If VMs will be created in the cluster root
|
Host.Config.Storage
Resource.AssignVMToPool
VApp.AssignResourcePool
VApp.Import
VirtualMachine.Config.AddNewDisk
|
vSphere vCenter Resource Pool
|
If an existing resource pool is provided
|
Resource.AssignVMToPool
VApp.AssignResourcePool
VApp.Import
VirtualMachine.Config.AddNewDisk
|
|
|
Datastore.AllocateSpace
Datastore.Browse
Datastore.FileManagement
InventoryService.Tagging.ObjectAttachable
|
|
|
|
|
|
InventoryService.Tagging.ObjectAttachable
Resource.AssignVMToPool
VApp.Import
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.AdvancedConfig
VirtualMachine.Config.Annotation
VirtualMachine.Config.CPUCount
VirtualMachine.Config.DiskExtend
VirtualMachine.Config.DiskLease
VirtualMachine.Config.EditDevice
VirtualMachine.Config.Memory
VirtualMachine.Config.RemoveDisk
VirtualMachine.Config.Rename
Host.Config.Storage
VirtualMachine.Config.ResetGuestInfo
VirtualMachine.Config.Resource
VirtualMachine.Config.Settings
VirtualMachine.Config.UpgradeVirtualHardware
VirtualMachine.Interact.GuestControl
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Interact.Reset
VirtualMachine.Inventory.Create
VirtualMachine.Inventory.CreateFromExisting
VirtualMachine.Inventory.Delete
VirtualMachine.Provisioning.Clone
VirtualMachine.Provisioning.MarkAsTemplate
VirtualMachine.Provisioning.DeployTemplate
|
vSphere vCenter data center
|
If the installation program creates the virtual machine folder. For user-provisioned infrastructure, VirtualMachine.Inventory.Create and VirtualMachine.Inventory.Delete privileges are optional if your cluster does not use the Machine API. See the "Minimum permissions for the Machine API" table.
|
InventoryService.Tagging.ObjectAttachable
Resource.AssignVMToPool
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.AdvancedConfig
VirtualMachine.Config.Annotation
VirtualMachine.Config.CPUCount
VirtualMachine.Config.DiskExtend
VirtualMachine.Config.DiskLease
VirtualMachine.Config.EditDevice
VirtualMachine.Config.Memory
VirtualMachine.Config.RemoveDisk
VirtualMachine.Config.Rename
VirtualMachine.Config.ResetGuestInfo
VirtualMachine.Config.Resource
VirtualMachine.Config.Settings
VirtualMachine.Config.UpgradeVirtualHardware
VirtualMachine.Interact.GuestControl
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Interact.Reset
VirtualMachine.Inventory.Create
VirtualMachine.Inventory.CreateFromExisting
VirtualMachine.Inventory.Delete
VirtualMachine.Provisioning.Clone
VirtualMachine.Provisioning.DeployTemplate
VirtualMachine.Provisioning.MarkAsTemplate
Folder.Create
Folder.Delete
|
Roles and privileges required for installation in vCenter graphical user interface (GUI)
vSphere object for role |
When required |
Required privileges in vCenter GUI |
|
|
Cns.Searchable
"vSphere Tagging"."Assign or Unassign vSphere Tag"
"vSphere Tagging"."Create vSphere Tag Category"
"vSphere Tagging"."Create vSphere Tag"
vSphere Tagging"."Delete vSphere Tag Category"
"vSphere Tagging"."Delete vSphere Tag"
"vSphere Tagging"."Edit vSphere Tag Category"
"vSphere Tagging"."Edit vSphere Tag"
Sessions."Validate session"
"Profile-driven storage"."Profile-driven storage update"
"Profile-driven storage"."Profile-driven storage view"
|
|
If VMs will be created in the cluster root
|
Host.Configuration."Storage partition configuration"
Resource."Assign virtual machine to resource pool"
VApp."Assign resource pool"
VApp.Import
"Virtual machine"."Change Configuration"."Add new disk"
|
vSphere vCenter Resource Pool
|
If an existing resource pool is provided
|
Host.Configuration."Storage partition configuration"
Resource."Assign virtual machine to resource pool"
VApp."Assign resource pool"
VApp.Import
"Virtual machine"."Change Configuration"."Add new disk"
|
|
|
Datastore."Allocate space"
Datastore."Browse datastore"
Datastore."Low level file operations"
"vSphere Tagging"."Assign or Unassign vSphere Tag on Object"
|
|
|
|
|
|
"vSphere Tagging"."Assign or Unassign vSphere Tag on Object"
Resource."Assign virtual machine to resource pool"
VApp.Import
"Virtual machine"."Change Configuration"."Add existing disk"
"Virtual machine"."Change Configuration"."Add new disk"
"Virtual machine"."Change Configuration"."Add or remove device"
"Virtual machine"."Change Configuration"."Advanced configuration"
"Virtual machine"."Change Configuration"."Set annotation"
"Virtual machine"."Change Configuration"."Change CPU count"
"Virtual machine"."Change Configuration"."Extend virtual disk"
"Virtual machine"."Change Configuration"."Acquire disk lease"
"Virtual machine"."Change Configuration"."Modify device settings"
"Virtual machine"."Change Configuration"."Change Memory"
"Virtual machine"."Change Configuration"."Remove disk"
"Virtual machine"."Change Configuration".Rename
"Virtual machine"."Change Configuration"."Reset guest information"
"Virtual machine"."Change Configuration"."Change resource"
"Virtual machine"."Change Configuration"."Change Settings"
"Virtual machine"."Change Configuration"."Upgrade virtual machine compatibility"
"Virtual machine".Interaction."Guest operating system management by VIX API"
"Virtual machine".Interaction."Power off"
"Virtual machine".Interaction."Power on"
"Virtual machine".Interaction.Reset
"Virtual machine"."Edit Inventory"."Create new"
"Virtual machine"."Edit Inventory"."Create from existing"
"Virtual machine"."Edit Inventory"."Remove"
"Virtual machine".Provisioning."Clone virtual machine"
"Virtual machine".Provisioning."Mark as template"
"Virtual machine".Provisioning."Deploy template"
|
vSphere vCenter data center
|
If the installation program creates the virtual machine folder. For user-provisioned infrastructure, VirtualMachine.Inventory.Create and VirtualMachine.Inventory.Delete privileges are optional if your cluster does not use the Machine API.
|
"vSphere Tagging"."Assign or Unassign vSphere Tag on Object"
Resource."Assign virtual machine to resource pool"
VApp.Import
"Virtual machine"."Change Configuration"."Add existing disk"
"Virtual machine"."Change Configuration"."Add new disk"
"Virtual machine"."Change Configuration"."Add or remove device"
"Virtual machine"."Change Configuration"."Advanced configuration"
"Virtual machine"."Change Configuration"."Set annotation"
"Virtual machine"."Change Configuration"."Change CPU count"
"Virtual machine"."Change Configuration"."Extend virtual disk"
"Virtual machine"."Change Configuration"."Acquire disk lease"
"Virtual machine"."Change Configuration"."Modify device settings"
"Virtual machine"."Change Configuration"."Change Memory"
"Virtual machine"."Change Configuration"."Remove disk"
"Virtual machine"."Change Configuration".Rename
"Virtual machine"."Change Configuration"."Reset guest information"
"Virtual machine"."Change Configuration"."Change resource"
"Virtual machine"."Change Configuration"."Change Settings"
"Virtual machine"."Change Configuration"."Upgrade virtual machine compatibility"
"Virtual machine".Interaction."Guest operating system management by VIX API"
"Virtual machine".Interaction."Power off"
"Virtual machine".Interaction."Power on"
"Virtual machine".Interaction.Reset
"Virtual machine"."Edit Inventory"."Create new"
"Virtual machine"."Edit Inventory"."Create from existing"
"Virtual machine"."Edit Inventory"."Remove"
"Virtual machine".Provisioning."Clone virtual machine"
"Virtual machine".Provisioning."Deploy template"
"Virtual machine".Provisioning."Mark as template"
Folder."Create folder"
Folder."Delete folder"
|
Additionally, the user requires some ReadOnly
permissions, and some of the roles require permission to propogate the permissions to child objects. These settings vary depending on whether or not you install the cluster into an existing folder.
Required permissions and propagation settings
vSphere object |
When required |
Propagate to children |
Permissions required |
|
|
|
Listed required privileges
|
vSphere vCenter data center
|
|
|
|
Installation program creates the folder
|
|
Listed required privileges
|
|
|
|
|
|
|
Listed required privileges
|
vSphere vCenter datastore
|
|
|
Listed required privileges
|
|
|
|
|
|
|
|
Listed required privileges
|
vSphere vCenter Virtual Machine Folder
|
|
|
Listed required privileges
|
vSphere vCenter Resource Pool
|
|
|
Listed required privileges
|
Minimum required vCenter account privileges
After you create a custom role and assign privileges to it, you can create permissions by selecting specific vSphere objects and then assigning the custom role to a user or group for each object.
Before you create permissions or request for the creation of permissions for a vSphere object, determine what minimum permissions apply to the vSphere object. By doing this task, you can ensure a basic interaction exists between a vSphere object and OKD architecture.
|
If you create a custom role and you do not assign privileges to it, the vSphere Server by default assigns a Read Only role to the custom role. Note that for the cloud provider API, the custom role only needs to inherit the privileges of the Read Only role.
|
Consider creating a custom role when an account with global administrative privileges does not meet your needs.
|
Accounts that are not configured with the required privileges are unsupported. Installing an OKD cluster in a vCenter is tested against a full list of privileges as described in the "Required vCenter account privileges" section. By adhering to the full list of privileges, you can reduce the possibility of unexpected behaviors that might occur when creating a custom role with a restricted set of privileges.
|
The following tables list the minimum permissions for a vSphere object that interacts with specific OKD architecture.
Minimum permissions for post-installation management of components
vSphere object for role |
When required |
Required privileges |
|
|
Cns.Searchable
InventoryService.Tagging.AttachTag
InventoryService.Tagging.CreateCategory
InventoryService.Tagging.CreateTag
InventoryService.Tagging.DeleteCategory
InventoryService.Tagging.DeleteTag
InventoryService.Tagging.EditCategory
InventoryService.Tagging.EditTag
Sessions.ValidateSession
StorageProfile.Update
StorageProfile.View
|
|
If you intend to create VMs in the cluster root
|
Host.Config.Storage
Resource.AssignVMToPool
|
vSphere vCenter Resource Pool
|
If you provide an existing resource pool in the install-config.yaml file
|
|
|
|
Datastore.AllocateSpace
Datastore.Browse
Datastore.FileManagement
InventoryService.Tagging.ObjectAttachable
|
|
|
|
|
|
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.AdvancedConfig
VirtualMachine.Config.Annotation
VirtualMachine.Config.CPUCount
VirtualMachine.Config.DiskExtend
VirtualMachine.Config.Memory
VirtualMachine.Config.Settings
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Inventory.CreateFromExisting
VirtualMachine.Inventory.Delete
VirtualMachine.Provisioning.Clone
VirtualMachine.Provisioning.DeployTemplate
|
vSphere vCenter data center
|
If the installation program creates the virtual machine folder. For user-provisioned infrastructure, VirtualMachine.Inventory.Create and VirtualMachine.Inventory.Delete privileges are optional if your cluster does not use the Machine API. If your cluster does use the Machine API and you want to set the minimum set of permissions for the API, see the "Minimum permissions for the Machine API" table.
|
Resource.AssignVMToPool
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Provisioning.DeployTemplate
|
Minimum permissions for the storage components
vSphere object for role |
When required |
Required privileges |
|
|
Cns.Searchable
InventoryService.Tagging.CreateCategory
InventoryService.Tagging.CreateTag
InventoryService.Tagging.EditCategory
InventoryService.Tagging.EditTag
StorageProfile.Update
StorageProfile.View
|
|
If you intend to create VMs in the cluster root
|
|
vSphere vCenter Resource Pool
|
If you provide an existing resource pool in the install-config.yaml file
|
|
|
|
Datastore.Browse
Datastore.FileManagement
InventoryService.Tagging.ObjectAttachable
|
|
|
|
|
|
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddRemoveDevice
|
vSphere vCenter data center
|
If the installation program creates the virtual machine folder. For user-provisioned infrastructure, VirtualMachine.Inventory.Create and VirtualMachine.Inventory.Delete privileges are optional if your cluster does not use the Machine API. If your cluster does use the Machine API and you want to set the minimum set of permissions for the API, see the "Minimum permissions for the Machine API" table.
|
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddRemoveDevice
|
Minimum permissions for the Machine API
vSphere object for role |
When required |
Required privileges |
|
|
InventoryService.Tagging.AttachTag
InventoryService.Tagging.CreateCategory
InventoryService.Tagging.CreateTag
InventoryService.Tagging.DeleteCategory
InventoryService.Tagging.DeleteTag
InventoryService.Tagging.EditCategory
InventoryService.Tagging.EditTag
Sessions.ValidateSession
StorageProfile.Update
StorageProfile.View
|
|
If you intend to create VMs in the cluster root
|
|
vSphere vCenter Resource Pool
|
If you provide an existing resource pool in the install-config.yaml file
|
|
|
|
Datastore.AllocateSpace
Datastore.Browse
|
|
|
|
|
|
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.AdvancedConfig
VirtualMachine.Config.Annotation
VirtualMachine.Config.CPUCount
VirtualMachine.Config.DiskExtend
VirtualMachine.Config.Memory
VirtualMachine.Config.Settings
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Inventory.CreateFromExisting
VirtualMachine.Inventory.Delete
VirtualMachine.Provisioning.Clone
VirtualMachine.Provisioning.DeployTemplate
|
vSphere vCenter data center
|
If the installation program creates the virtual machine folder. For user-provisioned infrastructure, VirtualMachine.Inventory.Create and VirtualMachine.Inventory.Delete privileges are optional if your cluster does not use the Machine API.
|
Resource.AssignVMToPool
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Provisioning.DeployTemplate
|
Using OKD with vMotion
If you intend on using vMotion in your vSphere environment, consider the following before installing an OKD cluster.
-
Using Storage vMotion can cause issues and is not supported.
-
Using VMware compute vMotion to migrate the workloads for both OKD compute machines and control plane machines is generally supported, where generally implies that you meet all VMware best practices for vMotion.
To help ensure the uptime of your compute and control plane nodes, ensure that you follow the VMware best practices for vMotion, and use VMware anti-affinity rules to improve the availability of OKD during maintenance or hardware issues.
-
If you are using VMware vSphere volumes in your pods, migrating a VM across datastores, either manually or through Storage vMotion, causes invalid references within OKD persistent volume (PV) objects that can result in data loss.
-
OKD does not support selective migration of VMDKs across datastores, using datastore clusters for VM provisioning or for dynamic or static provisioning of PVs, or using a datastore that is part of a datastore cluster for dynamic or static provisioning of PVs.
|
You can specify the path of any datastore that exists in a datastore cluster. By default, Storage Distributed Resource Scheduler (SDRS), which uses Storage vMotion, is automatically enabled for a datastore cluster. Red Hat does not support Storage vMotion, so you must disable Storage DRS to avoid data loss issues for your OKD cluster.
If you must specify VMs across multiple datastores, use a datastore object to specify a failure domain in your cluster’s install-config.yaml configuration file. For more information, see "VMware vSphere region and zone enablement".
|
Cluster resources
When you deploy an OKD cluster that uses infrastructure that you provided, you must create the following resources in your vCenter instance:
-
1 Folder
-
1 Tag category
-
1 Tag
-
Virtual machines:
Although these resources use 856 GB of storage, the bootstrap node is destroyed during the cluster installation process. A minimum of 800 GB of storage is required to use a standard cluster.
If you deploy more compute machines, the OKD cluster will use more storage.
Cluster limits
Available resources vary between clusters. The number of possible clusters within a vCenter is limited primarily by available storage space and any limitations on the number of required resources. Be sure to consider both limitations to the vCenter resources that the cluster creates and the resources that you require to deploy a cluster, such as IP addresses and networks.
Networking requirements
Use Dynamic Host Configuration Protocol (DHCP) for the network and ensure that the DHCP server is configured to provide persistent IP addresses to the cluster machines.
|
You do not need to use the DHCP for the network if you want to provision nodes with static IP addresses.
|
Configure the default gateway to use the DHCP server. All nodes must be in the same VLAN. You cannot scale the cluster using a second VLAN as a Day 2 operation.
You must use the Dynamic Host Configuration Protocol (DHCP) for the network and ensure that the DHCP server is configured to provide persistent IP addresses to the cluster machines. In the DHCP lease, you must configure the DHCP to use the default gateway. All nodes must be in the same VLAN. You cannot scale the cluster using a second VLAN as a Day 2 operation.
If you are installing to a restricted environment, the VM in your restricted network must have access to vCenter so that it can provision and manage nodes, persistent volume claims (PVCs), and other resources.
Additionally, you must create the following networking resources before you install the OKD cluster:
|
It is recommended that each OKD node in the cluster must have access to a Network Time Protocol (NTP) server that is discoverable via DHCP. Installation is possible without an NTP server. However, asynchronous server clocks will cause errors, which NTP server prevents.
|
DNS records
You must create DNS records for two static IP addresses in the appropriate DNS server for the vCenter instance that hosts your OKD cluster. In each record, <cluster_name>
is the cluster name and <base_domain>
is the cluster base domain that you specify when you install the cluster. A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>.
.
Table 3. Required DNS records
Component |
Record |
Description |
|
api.<cluster_name>.<base_domain>.
|
This DNS A/AAAA or CNAME (Canonical Name) record must point to the load balancer
for the control plane machines. This record must be resolvable by both clients
external to the cluster and from all the nodes within the cluster.
|
|
*.apps.<cluster_name>.<base_domain>.
|
A wildcard DNS A/AAAA or CNAME record that points to the load balancer that targets the
machines that run the Ingress router pods, which are the worker nodes by
default. This record must be resolvable by both clients external to the cluster
and from all the nodes within the cluster.
|
Networking requirements for user-provisioned infrastructure
All the Fedora CoreOS (FCOS) machines require networking to be configured in initramfs
during boot
to fetch their Ignition config files.
During the initial boot, the machines require an IP address configuration that is set either through a DHCP server or statically by providing the required boot options. After a network connection is established, the machines download their Ignition config files from an HTTP or HTTPS server. The Ignition config files are then used to set the exact state of each machine. The Machine Config Operator completes more changes to the machines, such as the application of new certificates or keys, after installation.
It is recommended to use a DHCP server for long-term management of the cluster machines. Ensure that the DHCP server is configured to provide persistent IP addresses, DNS server information, and hostnames to the cluster machines.
|
If a DHCP service is not available for your user-provisioned infrastructure, you can instead provide the IP networking configuration and the address of the DNS server to the nodes at FCOS install time. These can be passed as boot arguments if you are installing from an ISO image. See the Installing FCOS and starting the OKD bootstrap process section for more information about static IP provisioning and advanced networking options.
|
The Kubernetes API server must be able to resolve the node names of the cluster
machines. If the API servers and worker nodes are in different zones, you can
configure a default DNS search zone to allow the API server to resolve the
node names. Another supported approach is to always refer to hosts by their
fully-qualified domain names in both the node objects and all DNS requests.
Setting the cluster node hostnames through DHCP
On Fedora CoreOS (FCOS) machines, the hostname is set through NetworkManager. By default, the machines obtain their hostname through DHCP. If the hostname is not provided by DHCP, set statically through kernel arguments, or another method, it is obtained through a reverse DNS lookup. Reverse DNS lookup occurs after the network has been initialized on a node and can take time to resolve. Other system services can start prior to this and detect the hostname as localhost
or similar. You can avoid this by using DHCP to provide the hostname for each cluster node.
Additionally, setting the hostnames through DHCP can bypass any manual DNS record name configuration errors in environments that have a DNS split-horizon implementation.
Network connectivity requirements
You must configure the network connectivity between machines to allow OKD cluster
components to communicate. Each machine must be able to resolve the hostnames
of all other machines in the cluster.
This section provides details about the ports that are required.
|
In connected OKD environments, all nodes are required to have internet access to pull images
for platform containers and provide telemetry data to Red Hat.
|
Table 6. Ports used for all-machine to all-machine communications
Protocol |
Port |
Description |
|
|
Network reachability tests
|
|
|
|
|
Host level services, including the node exporter on ports 9100 -9101 and
the Cluster Version Operator on port 9099 .
|
|
The default ports that Kubernetes reserves
|
|
|
|
|
|
|
Host level services, including the node exporter on ports 9100 -9101 .
|
|
|
|
|
|
Network Time Protocol (NTP) on UDP port 123
If an external NTP time server is configured, you must open UDP port 123 .
|
|
|
|
|
|
IPsec Encapsulating Security Payload (ESP)
|
Table 7. Ports used for all-machine to control plane communications
Protocol |
Port |
Description |
|
|
|
Table 8. Ports used for control plane machine to control plane machine communications
Protocol |
Port |
Description |
|
|
etcd server and peer ports
|
Ethernet adaptor hardware address requirements
When provisioning VMs for the cluster, the ethernet interfaces configured for
each VM must use a MAC address from the VMware Organizationally Unique
Identifier (OUI) allocation ranges:
-
00:05:69:00:00:00
to 00:05:69:FF:FF:FF
-
00:0c:29:00:00:00
to 00:0c:29:FF:FF:FF
-
00:1c:14:00:00:00
to 00:1c:14:FF:FF:FF
-
00:50:56:00:00:00
to 00:50:56:3F:FF:FF
If a MAC address outside the VMware OUI is used, the cluster installation will
not succeed.
NTP configuration for user-provisioned infrastructure
OKD clusters are configured to use a public Network Time Protocol (NTP) server by default. If you want to use a local enterprise NTP server, or if your cluster is being deployed in a disconnected network, you can configure the cluster to use a specific time server. For more information, see the documentation for Configuring chrony time service.
If a DHCP server provides NTP server information, the chrony time service on the Fedora CoreOS (FCOS) machines read the information and can sync the clock with the NTP servers.
User-provisioned DNS requirements
In OKD deployments, DNS name resolution is required for the following components:
-
The Kubernetes API
-
The OKD application wildcard
-
The bootstrap, control plane, and compute machines
Reverse DNS resolution is also required for the Kubernetes API, the bootstrap machine, the control plane machines, and the compute machines.
DNS A/AAAA or CNAME records are used for name resolution and PTR records are used for reverse name resolution. The reverse records are important because Fedora CoreOS (FCOS) uses the reverse records to set the hostnames for all the nodes, unless the hostnames are provided by DHCP. Additionally, the reverse records are used to generate the certificate signing requests (CSR) that OKD needs to operate.
|
It is recommended to use a DHCP server to provide the hostnames to each cluster node. See the DHCP recommendations for user-provisioned infrastructure section for more information.
|
The following DNS records are required for a user-provisioned OKD cluster and they must be in place before installation. In each record, <cluster_name>
is the cluster name and <base_domain>
is the base domain that you specify in the install-config.yaml
file. A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>.
.
Table 9. Required DNS records
Component |
Record |
Description |
|
api.<cluster_name>.<base_domain>.
|
A DNS A/AAAA or CNAME record, and a DNS PTR record, to identify the API load balancer. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
|
api-int.<cluster_name>.<base_domain>.
|
A DNS A/AAAA or CNAME record, and a DNS PTR record, to internally identify the API load balancer. These records must be resolvable from all the nodes within the cluster.
|
The API server must be able to resolve the worker nodes by the hostnames
that are recorded in Kubernetes. If the API server cannot resolve the node
names, then proxied API calls can fail, and you cannot retrieve logs from pods.
|
|
|
*.apps.<cluster_name>.<base_domain>.
|
A wildcard DNS A/AAAA or CNAME record that refers to the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller pods. The Ingress Controller pods run on the compute machines by default. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
For example, console-openshift-console.apps.<cluster_name>.<base_domain> is used as a wildcard route to the OKD console.
|
|
bootstrap.<cluster_name>.<base_domain>.
|
A DNS A/AAAA or CNAME record, and a DNS PTR record, to identify the bootstrap
machine. These records must be resolvable by the nodes within the cluster.
|
|
<control_plane><n>.<cluster_name>.<base_domain>.
|
DNS A/AAAA or CNAME records and DNS PTR records to identify each machine
for the control plane nodes. These records must be resolvable by the nodes within the cluster.
|
|
<compute><n>.<cluster_name>.<base_domain>.
|
DNS A/AAAA or CNAME records and DNS PTR records to identify each machine
for the worker nodes. These records must be resolvable by the nodes within the cluster.
|
|
In OKD 4.4 and later, you do not need to specify etcd host and SRV records in your DNS configuration.
|
|
You can use the dig command to verify name and reverse name resolution. See the section on Validating DNS resolution for user-provisioned infrastructure for detailed validation steps.
|
Example DNS configuration for user-provisioned clusters
This section provides A and PTR record configuration samples that meet the DNS requirements for deploying OKD on user-provisioned infrastructure. The samples are not meant to provide advice for choosing one DNS solution over another.
In the examples, the cluster name is ocp4
and the base domain is example.com
.
Example DNS A record configuration for a user-provisioned cluster
The following example is a BIND zone file that shows sample A records for name resolution in a user-provisioned cluster.
Sample DNS zone database
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
IN MX 10 smtp.example.com.
;
;
ns1.example.com. IN A 192.168.1.5
smtp.example.com. IN A 192.168.1.5
;
helper.example.com. IN A 192.168.1.5
helper.ocp4.example.com. IN A 192.168.1.5
;
api.ocp4.example.com. IN A 192.168.1.5 (1)
api-int.ocp4.example.com. IN A 192.168.1.5 (2)
;
*.apps.ocp4.example.com. IN A 192.168.1.5 (3)
;
bootstrap.ocp4.example.com. IN A 192.168.1.96 (4)
;
control-plane0.ocp4.example.com. IN A 192.168.1.97 (5)
control-plane1.ocp4.example.com. IN A 192.168.1.98 (5)
control-plane2.ocp4.example.com. IN A 192.168.1.99 (5)
;
compute0.ocp4.example.com. IN A 192.168.1.11 (6)
compute1.ocp4.example.com. IN A 192.168.1.7 (6)
;
;EOF
1 |
Provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer. |
2 |
Provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer and is used for internal cluster communications. |
3 |
Provides name resolution for the wildcard routes. The record refers to the IP address of the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller pods. The Ingress Controller pods run on the compute machines by default.
|
In the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.
|
|
4 |
Provides name resolution for the bootstrap machine. |
5 |
Provides name resolution for the control plane machines. |
6 |
Provides name resolution for the compute machines. |
Example DNS PTR record configuration for a user-provisioned cluster
The following example BIND zone file shows sample PTR records for reverse name resolution in a user-provisioned cluster.
Sample DNS zone database for reverse records
$TTL 1W
@ IN SOA ns1.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.example.com.
;
5.1.168.192.in-addr.arpa. IN PTR api.ocp4.example.com. (1)
5.1.168.192.in-addr.arpa. IN PTR api-int.ocp4.example.com. (2)
;
96.1.168.192.in-addr.arpa. IN PTR bootstrap.ocp4.example.com. (3)
;
97.1.168.192.in-addr.arpa. IN PTR control-plane0.ocp4.example.com. (4)
98.1.168.192.in-addr.arpa. IN PTR control-plane1.ocp4.example.com. (4)
99.1.168.192.in-addr.arpa. IN PTR control-plane2.ocp4.example.com. (4)
;
11.1.168.192.in-addr.arpa. IN PTR compute0.ocp4.example.com. (5)
7.1.168.192.in-addr.arpa. IN PTR compute1.ocp4.example.com. (5)
;
;EOF
1 |
Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer. |
2 |
Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer and is used for internal cluster communications. |
3 |
Provides reverse DNS resolution for the bootstrap machine. |
4 |
Provides reverse DNS resolution for the control plane machines. |
5 |
Provides reverse DNS resolution for the compute machines. |
|
A PTR record is not required for the OKD application wildcard.
|