1. Documentation
×

Prerequisites

Installer-provisioned installation of OKD requires:

  1. One provisioner node with Fedora CoreOS (FCOS) installed. The provisioner can be removed after installation.

  2. Three control plane nodes

  3. Baseboard management controller (BMC) access to each node

  4. At least one network:

    1. One required routable network

    2. One optional provisioning network

    3. One optional management network

Before starting an installer-provisioned installation of OKD, ensure the hardware environment meets the following requirements.

Node requirements

Installer-provisioned installation involves a number of hardware node requirements:

  • CPU architecture: All nodes must use x86_64 CPU architecture.

  • Similar nodes: Red Hat recommends nodes have an identical configuration per role. That is, Red Hat recommends nodes be the same brand and model with the same CPU, memory, and storage configuration.

  • Baseboard Management Controller: The provisioner node must be able to access the baseboard management controller (BMC) of each OKD cluster node. You may use IPMI, Redfish, or a proprietary protocol.

  • Latest generation: Nodes must be of the most recent generation. Installer-provisioned installation relies on BMC protocols, which must be compatible across nodes. Additionally, Fedora CoreOS (FCOS) ships with the most recent drivers for RAID controllers. Ensure that the nodes are recent enough to support FCOS for the provisioner node and FCOS for the control plane and worker nodes.

  • Registry node: (Optional) If setting up a disconnected mirrored registry, it is recommended the registry reside in its own node.

  • Provisioner node: Installer-provisioned installation requires one provisioner node.

  • Control plane: Installer-provisioned installation requires three control plane nodes for high availability. You can deploy an OKD cluster with only three control plane nodes, making the control plane nodes schedulable as worker nodes. Smaller clusters are more resource efficient for administrators and developers during development, production, and testing.

  • Worker nodes: While not required, a typical production cluster has two or more worker nodes.

    Do not deploy a cluster with only one worker node, because the cluster will deploy with routers and ingress traffic in a degraded state.

  • Network interfaces: Each node must have at least one network interface for the routable baremetal network. Each node must have one network interface for a provisioning network when using the provisioning network for deployment. Using the provisioning network is the default configuration.

  • Unified Extensible Firmware Interface (UEFI): Installer-provisioned installation requires UEFI boot on all OKD nodes when using IPv6 addressing on the provisioning network. In addition, UEFI Device PXE Settings must be set to use the IPv6 protocol on the provisioning network NIC, but omitting the provisioning network removes this requirement.

    When starting the installation from virtual media such as an ISO image, delete all old UEFI boot table entries. If the boot table includes entries that are not generic entries provided by the firmware, the installation might fail.

  • Secure Boot: Many production scenarios require nodes with Secure Boot enabled to verify the node only boots with trusted software, such as UEFI firmware drivers, EFI applications, and the operating system. You may deploy with Secure Boot manually or managed.

    1. Manually: To deploy an OKD cluster with Secure Boot manually, you must enable UEFI boot mode and Secure Boot on each control plane node and each worker node. Red Hat supports Secure Boot with manually enabled UEFI and Secure Boot only when installer-provisioned installations use Redfish virtual media. See "Configuring nodes for Secure Boot manually" in the "Configuring nodes" section for additional details.

    2. Managed: To deploy an OKD cluster with managed Secure Boot, you must set the bootMode value to UEFISecureBoot in the install-config.yaml file. Red Hat only supports installer-provisioned installation with managed Secure Boot on 10th generation HPE hardware and 13th generation Dell hardware running firmware version 2.75.75.75 or greater. Deploying with managed Secure Boot does not require Redfish virtual media. See "Configuring managed Secure Boot" in the "Setting up the environment for an OpenShift installation" section for details.

      Red Hat does not support Secure Boot with self-generated keys.

Planning a bare metal cluster for OKD Virtualization

If you will use OKD Virtualization, it is important to be aware of several requirements before you install your bare metal cluster.

  • If you want to use live migration features, you must have multiple worker nodes at the time of cluster installation. This is because live migration requires the cluster-level high availability (HA) flag to be set to true. The HA flag is set when a cluster is installed and cannot be changed afterwards. If there are fewer than two worker nodes defined when you install your cluster, the HA flag is set to false for the life of the cluster.

    You can install OKD Virtualization on a single-node cluster, but single-node OpenShift does not support high availability.

  • Live migration requires shared storage. Storage for OKD Virtualization must support and use the ReadWriteMany (RWX) access mode.

  • If you plan to use Single Root I/O Virtualization (SR-IOV), ensure that your network interface controllers (NICs) are supported by OKD.

Additional resources

Firmware requirements for installing with virtual media

The installation program for installer-provisioned OKD clusters validates the hardware and firmware compatibility with Redfish virtual media. The installation program does not begin installation on a node if the node firmware is not compatible. The following tables list the minimum firmware versions tested and verified to work for installer-provisioned OKD clusters deployed by using Redfish virtual media.

Red Hat does not test every combination of firmware, hardware, or other third-party components. For further information about third-party support, see Red Hat third-party support policy. For information about updating the firmware, see the hardware documentation for the nodes or contact the hardware vendor.
Table 1. Firmware compatibility for HP hardware with Redfish virtual media
Model Management Firmware versions

10th Generation

iLO5

2.63 or later
Table 2. Firmware compatibility for Dell hardware with Redfish virtual media
Model Management Firmware versions

15th Generation

iDRAC 9

v6.10.30.00

14th Generation

iDRAC 9

v6.10.30.00

13th Generation

iDRAC 8

v2.75.75.75 or later
Additional resources

Unable to discover new bare metal hosts using the BMC

Network requirements

Installer-provisioned installation of OKD involves multiple network requirements. First, installer-provisioned installation involves an optional non-routable provisioning network for provisioning the operating system on each bare-metal node. Second, installer-provisioned installation involves a routable baremetal network.

Installer-provisioned networking

Ensuring required ports are open

Certain ports must be open between cluster nodes for installer-provisioned installations to complete successfully. In certain situations, such as using separate subnets for far edge worker nodes, you must ensure that the nodes in these subnets can communicate with nodes in the other subnets on the following required ports.

Table 3. Required ports
Port Description

67,68

When using a provisioning network, cluster nodes access the dnsmasq DHCP server over their provisioning network interfaces using ports 67 and 68.

69

When using a provisioning network, cluster nodes communicate with the TFTP server on port 69 using their provisioning network interfaces. The TFTP server runs on the bootstrap VM. The bootstrap VM runs on the provisioner node.

80

When not using the image caching option or when using virtual media, the provisioner node must have port 80 open on the baremetal machine network interface to stream the Fedora CoreOS (FCOS) image from the provisioner node to the cluster nodes.

123

The cluster nodes must access the NTP server on port 123 using the baremetal machine network.

5050

The Ironic Inspector API runs on the control plane nodes and listens on port 5050. The Inspector API is responsible for hardware introspection, which collects information about the hardware characteristics of the bare-metal nodes.

5051

Port 5050 uses port 5051 as a proxy.

6180

When deploying with virtual media and not using TLS, the provisioner node and the control plane nodes must have port 6180 open on the baremetal machine network interface so that the baseboard management controller (BMC) of the worker nodes can access the FCOS image. Starting with OKD 4.13, the default HTTP port is 6180.

6183

When deploying with virtual media and using TLS, the provisioner node and the control plane nodes must have port 6183 open on the baremetal machine network interface so that the BMC of the worker nodes can access the FCOS image.

6385

The Ironic API server runs initially on the bootstrap VM and later on the control plane nodes and listens on port 6385. The Ironic API allows clients to interact with Ironic for bare-metal node provisioning and management, including operations such as enrolling new nodes, managing their power state, deploying images, and cleaning the hardware.

6388

Port 6385 uses port 6388 as a proxy.

8080

When using image caching without TLS, port 8080 must be open on the provisioner node and accessible by the BMC interfaces of the cluster nodes.

8083

When using the image caching option with TLS, port 8083 must be open on the provisioner node and accessible by the BMC interfaces of the cluster nodes.

9999

By default, the Ironic Python Agent (IPA) listens on TCP port 9999 for API calls from the Ironic conductor service. Communication between the bare-metal node where IPA is running and the Ironic conductor service uses this port.

Increase the network MTU

Before deploying OKD, increase the network maximum transmission unit (MTU) to 1500 or more. If the MTU is lower than 1500, the Ironic image that is used to boot the node might fail to communicate with the Ironic inspector pod, and inspection will fail. If this occurs, installation stops because the nodes are not available for installation.

Configuring NICs

OKD deploys with two networks:

  • provisioning: The provisioning network is an optional non-routable network used for provisioning the underlying operating system on each node that is a part of the OKD cluster. The network interface for the provisioning network on each cluster node must have the BIOS or UEFI configured to PXE boot.

    The provisioningNetworkInterface configuration setting specifies the provisioning network NIC name on the control plane nodes, which must be identical on the control plane nodes. The bootMACAddress configuration setting provides a means to specify a particular NIC on each node for the provisioning network.

    The provisioning network is optional, but it is required for PXE booting. If you deploy without a provisioning network, you must use a virtual media BMC addressing option such as redfish-virtualmedia or idrac-virtualmedia.

  • baremetal: The baremetal network is a routable network. You can use any NIC to interface with the baremetal network provided the NIC is not configured to use the provisioning network.

When using a VLAN, each NIC must be on a separate VLAN corresponding to the appropriate network.

DNS requirements

Clients access the OKD cluster nodes over the baremetal network. A network administrator must configure a subdomain or subzone where the canonical name extension is the cluster name.

<cluster_name>.<base_domain>

For example:

test-cluster.example.com

OKD includes functionality that uses cluster membership information to generate A/AAAA records. This resolves the node names to their IP addresses. After the nodes are registered with the API, the cluster can disperse node information without using CoreDNS-mDNS. This eliminates the network traffic associated with multicast DNS.

CoreDNS requires both TCP and UDP connections to the upstream DNS server to function correctly. Ensure the upstream DNS server can receive both TCP and UDP connections from OKD cluster nodes.

In OKD deployments, DNS name resolution is required for the following components:

  • The Kubernetes API

  • The OKD application wildcard ingress API

A/AAAA records are used for name resolution and PTR records are used for reverse name resolution. Fedora CoreOS (FCOS) uses the reverse records or DHCP to set the hostnames for all the nodes.

Installer-provisioned installation includes functionality that uses cluster membership information to generate A/AAAA records. This resolves the node names to their IP addresses. In each record, <cluster_name> is the cluster name and <base_domain> is the base domain that you specify in the install-config.yaml file. A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>..

Table 4. Required DNS records
Component Record Description

Kubernetes API

api.<cluster_name>.<base_domain>.

An A/AAAA record and a PTR record identify the API load balancer. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster.

Routes

*.apps.<cluster_name>.<base_domain>.

The wildcard A/AAAA record refers to the application ingress load balancer. The application ingress load balancer targets the nodes that run the Ingress Controller pods. The Ingress Controller pods run on the worker nodes by default. These records must be resolvable by both clients external to the cluster and from all the nodes within the cluster.

For example, console-openshift-console.apps.<cluster_name>.<base_domain> is used as a wildcard route to the OKD console.

You can use the dig command to verify DNS resolution.

Dynamic Host Configuration Protocol (DHCP) requirements

By default, installer-provisioned installation deploys ironic-dnsmasq with DHCP enabled for the provisioning network. No other DHCP servers should be running on the provisioning network when the provisioningNetwork configuration setting is set to managed, which is the default value. If you have a DHCP server running on the provisioning network, you must set the provisioningNetwork configuration setting to unmanaged in the install-config.yaml file.

Network administrators must reserve IP addresses for each node in the OKD cluster for the baremetal network on an external DHCP server.

Reserving IP addresses for nodes with the DHCP server

For the baremetal network, a network administrator must reserve several IP addresses, including:

  1. Two unique virtual IP addresses.

    • One virtual IP address for the API endpoint.

    • One virtual IP address for the wildcard ingress endpoint.

  2. One IP address for the provisioner node.

  3. One IP address for each control plane node.

  4. One IP address for each worker node, if applicable.

Reserving IP addresses so they become static IP addresses

Some administrators prefer to use static IP addresses so that each node’s IP address remains constant in the absence of a DHCP server. To configure static IP addresses with NMState, see "(Optional) Configuring node network interfaces" in the "Setting up the environment for an OpenShift installation" section.
Networking between external load balancers and control plane nodes

External load balancing services and the control plane nodes must run on the same L2 network, and on the same VLAN when using VLANs to route traffic between the load balancing services and the control plane nodes.

The storage interface requires a DHCP reservation or a static IP.

The following table provides an exemplary embodiment of fully qualified domain names. The API and name server addresses begin with canonical name extensions. The hostnames of the control plane and worker nodes are exemplary, so you can use any host naming convention you prefer.

Usage Host Name IP

API

api.<cluster_name>.<base_domain>

<ip>

Ingress LB (apps)

*.apps.<cluster_name>.<base_domain>

<ip>

Provisioner node

provisioner.<cluster_name>.<base_domain>

<ip>

Control-plane-0

openshift-control-plane-0.<cluster_name>.<base_domain>

<ip>

Control-plane-1

openshift-control-plane-1.<cluster_name>-.<base_domain>

<ip>

Control-plane-2

openshift-control-plane-2.<cluster_name>.<base_domain>

<ip>

Worker-0

openshift-worker-0.<cluster_name>.<base_domain>

<ip>

Worker-1

openshift-worker-1.<cluster_name>.<base_domain>

<ip>

Worker-n

openshift-worker-n.<cluster_name>.<base_domain>

<ip>

If you do not create DHCP reservations, the installation program requires reverse DNS resolution to set the hostnames for the Kubernetes API node, the provisioner node, the control plane nodes, and the worker nodes.

Provisioner node requirements

You must specify the MAC address for the provisioner node in your installation configuration. The bootMacAddress specification is typically associated with PXE network booting. However, the Ironic provisioning service also requires the bootMacAddress specification to identify nodes during the inspection of the cluster, or during node redeployment in the cluster.

The provisioner node requires layer 2 connectivity for network booting, DHCP and DNS resolution, and local network communication. The provisioner node requires layer 3 connectivity for virtual media booting.

Network Time Protocol (NTP)

Each OKD node in the cluster must have access to an NTP server. OKD nodes use NTP to synchronize their clocks. For example, cluster nodes use SSL/TLS certificates that require validation, which might fail if the date and time between the nodes are not in sync.

Define a consistent clock date and time format in each cluster node’s BIOS settings, or installation might fail.

You can reconfigure the control plane nodes to act as NTP servers on disconnected clusters, and reconfigure worker nodes to retrieve time from the control plane nodes.

Port access for the out-of-band management IP address

The out-of-band management IP address is on a separate network from the node. To ensure that the out-of-band management can communicate with the provisioner node during installation, the out-of-band management IP address must be granted access to port 6180 on the provisioner node and on the OKD control plane nodes. TLS port 6183 is required for virtual media installation, for example, by using Redfish.

Configuring nodes

Configuring nodes when using the provisioning network

Each node in the cluster requires the following configuration for proper installation.

A mismatch between nodes will cause an installation failure.

While the cluster nodes can contain more than two NICs, the installation process only focuses on the first two NICs. In the following table, NIC1 is a non-routable network (provisioning) that is only used for the installation of the OKD cluster.

NIC Network VLAN

NIC1

provisioning

<provisioning_vlan>

NIC2

baremetal

<baremetal_vlan>

The Fedora CoreOS (FCOS) installation process on the provisioner node might vary. To install FCOS using a local Satellite server or a PXE server, PXE-enable NIC2.

PXE Boot order

NIC1 PXE-enabled provisioning network

1

NIC2 baremetal network. PXE-enabled is optional.

2

Ensure PXE is disabled on all other NICs.

Configure the control plane and worker nodes as follows:

PXE Boot order

NIC1 PXE-enabled (provisioning network)

1

Configuring nodes without the provisioning network

The installation process requires one NIC:

NIC Network VLAN

NICx

baremetal

<baremetal_vlan>

NICx is a routable network (baremetal) that is used for the installation of the OKD cluster, and routable to the internet.

The provisioning network is optional, but it is required for PXE booting. If you deploy without a provisioning network, you must use a virtual media BMC addressing option such as redfish-virtualmedia or idrac-virtualmedia.

Configuring nodes for Secure Boot manually

Secure Boot prevents a node from booting unless it verifies the node is using only trusted software, such as UEFI firmware drivers, EFI applications, and the operating system.

Red Hat only supports manually configured Secure Boot when deploying with Redfish virtual media.

To enable Secure Boot manually, refer to the hardware guide for the node and execute the following:

Procedure

  1. Boot the node and enter the BIOS menu.

  2. Set the node’s boot mode to UEFI Enabled.

  3. Enable Secure Boot.

Red Hat does not support Secure Boot with self-generated keys.

Out-of-band management

Nodes typically have an additional NIC used by the baseboard management controllers (BMCs). These BMCs must be accessible from the provisioner node.

Each node must be accessible via out-of-band management. When using an out-of-band management network, the provisioner node requires access to the out-of-band management network for a successful OKD installation.

The out-of-band management setup is out of scope for this document. Using a separate management network for out-of-band management can enhance performance and improve security. However, using the provisioning network or the bare metal network are valid options.

The bootstrap VM features a maximum of two network interfaces. If you configure a separate management network for out-of-band management, and you are using a provisioning network, the bootstrap VM requires routing access to the management network through one of the network interfaces. In this scenario, the bootstrap VM can then access three networks:

  • the bare metal network

  • the provisioning network

  • the management network routed through one of the network interfaces

Required data for installation

Prior to the installation of the OKD cluster, gather the following information from all cluster nodes:

  • Out-of-band management IP

    • Examples

      • Dell (iDRAC) IP

      • HP (iLO) IP

      • Fujitsu (iRMC) IP

When using the provisioning network

  • NIC (provisioning) MAC address

  • NIC (baremetal) MAC address

When omitting the provisioning network

  • NIC (baremetal) MAC address

Validation checklist for nodes

When using the provisioning network

  • NIC1 VLAN is configured for the provisioning network.

  • NIC1 for the provisioning network is PXE-enabled on the provisioner, control plane, and worker nodes.

  • NIC2 VLAN is configured for the baremetal network.

  • PXE has been disabled on all other NICs.

  • DNS is configured with API and Ingress endpoints.

  • Control plane and worker nodes are configured.

  • All nodes accessible via out-of-band management.

  • (Optional) A separate management network has been created.

  • Required data for installation.

When omitting the provisioning network

  • NIC1 VLAN is configured for the baremetal network.

  • DNS is configured with API and Ingress endpoints.

  • Control plane and worker nodes are configured.

  • All nodes accessible via out-of-band management.

  • (Optional) A separate management network has been created.

  • Required data for installation.