apiserver.config.openshift.io
After installing OKD, you can further expand and customize your cluster to your requirements.
You complete most of the cluster configuration and customization after you deploy your OKD cluster. A number of configuration resources are available.
If you install your cluster on IBM zSystems, not all features and functions are available. |
You modify the configuration resources to configure the major features of the cluster, such as the image registry, networking configuration, image build behavior, and the identity provider.
For current documentation of the settings that you control by using these resources, use
the oc explain
command, for example oc explain builds --api-version=config.openshift.io/v1
All cluster configuration resources are globally scoped (not namespaced) and named cluster
.
Resource name | Description |
---|---|
|
Provides API server configuration such as certificates and certificate authorities. |
|
Controls the identity provider and authentication configuration for the cluster. |
|
Controls default and enforced configuration for all builds on the cluster. |
|
Configures the behavior of the web console interface, including the logout behavior. |
|
Enables FeatureGates so that you can use Tech Preview features. |
|
Configures how specific image registries should be treated (allowed, disallowed, insecure, CA details). |
|
Configuration details related to routing such as the default domain for routes. |
|
Configures identity providers and other behavior related to internal OAuth server flows. |
|
Configures how projects are created including the project template. |
|
Defines proxies to be used by components needing external network access. Note: not all components currently consume this value. |
|
Configures scheduler behavior such as profiles and default node selectors. |
These configuration resources are cluster-scoped instances, named cluster
, which control the behavior of a specific component as
owned by a particular Operator.
Resource name | Description |
---|---|
|
Controls console appearance such as branding customizations |
|
Configures internal image registry settings such as public routing, log levels, proxy settings, resource constraints, replica counts, and storage type. |
|
Configures the Samples Operator to control which example image streams and templates are installed on the cluster. |
These configuration resources represent a single instance of a particular component. In some cases, you can request multiple instances by creating multiple instances of the resource. In other cases, the Operator can use only a specific resource instance name in a specific namespace. Reference the component-specific documentation for details on how and when you can create additional resource instances.
Resource name | Instance name | Namespace | Description |
---|---|---|---|
|
|
|
Controls the Alertmanager deployment parameters. |
|
|
|
Configures Ingress Operator behavior such as domain, number of replicas, certificates, and controller placement. |
You use these resources to retrieve information about the cluster. Some configurations might require you to edit these resources directly.
Resource name | Instance name | Description |
---|---|---|
|
|
In OKD 4, you must not customize the |
|
|
You cannot modify the DNS settings for your cluster. You can view the DNS Operator status. |
|
|
Configuration details allowing the cluster to interact with its cloud provider. |
|
|
You cannot modify your cluster networking after installation. To customize your network, follow the process to customize networking during installation. |
You can update the global pull secret for your cluster by either replacing the current pull secret or appending a new pull secret.
The procedure is required when users use a separate registry to store images than the registry used during installation.
You have access to the cluster as a user with the cluster-admin
role.
Optional: To append a new pull secret to the existing pull secret, complete the following steps:
Enter the following command to download the pull secret:
$ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> (1)
1 | Provide the path to the pull secret file. |
Enter the following command to add the new pull secret:
$ oc registry login --registry="<registry>" \ (1)
--auth-basic="<username>:<password>" \ (2)
--to=<pull_secret_location> (3)
1 | Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>" . |
2 | Provide the credentials of the new registry. |
3 | Provide the path to the pull secret file. |
Alternatively, you can perform a manual update to the pull secret file.
Enter the following command to update the global pull secret for your cluster:
$ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> (1)
1 | Provide the path to the new pull secret file. |
This update is rolled out to all nodes, which can take some time depending on the size of your cluster.
As of OKD 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot. |
After you deploy your OKD cluster, you can add worker nodes to scale cluster resources. There are different ways you can add worker nodes depending on the installation method and the environment of your cluster.
For installer-provisioned infrastructure clusters, you can manually or automatically scale the MachineSet
object to match the number of available bare-metal hosts.
To add a bare-metal host, you must configure all network prerequisites, configure an associated baremetalhost
object, then provision the worker node to the cluster. You can add a bare-metal host manually or by using the web console.
For user-provisioned infrastructure clusters, you can add worker nodes by using a Fedora or FCOS ISO image and connecting it to your cluster using cluster Ignition config files. For RHEL worker nodes, the following example uses Ansible playbooks to add worker nodes to the cluster. For RHCOS worker nodes, the following example uses an ISO image and network booting to add worker nodes to the cluster.
For clusters managed by the Assisted Installer, you can add worker nodes by using the Red Hat OpenShift Cluster Manager console, the Assisted Installer REST API or you can manually add worker nodes using an ISO image and cluster Ignition config files.
If you incorrectly sized the worker nodes during deployment, adjust them by creating one or more new compute machine sets, scale them up, then scale the original compute machine set down before removing them.
MachineSet
objects describe OKD nodes with respect to the cloud or machine provider.
The MachineConfigPool
object allows MachineConfigController
components to define and provide the status of machines in the context of upgrades.
The MachineConfigPool
object allows users to configure how upgrades are rolled out to the OKD nodes in the machine config pool.
The NodeSelector
object can be replaced with a reference to the MachineSet
object.
To add or remove an instance of a machine in a compute machine set, you can manually scale the compute machine set.
This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have compute machine sets.
Install an OKD cluster and the oc
command line.
Log in to oc
as a user with cluster-admin
permission.
View the compute machine sets that are in the cluster by running the following command:
$ oc get machinesets -n openshift-machine-api
The compute machine sets are listed in the form of <clusterid>-worker-<aws-region-az>
.
View the compute machines that are in the cluster by running the following command:
$ oc get machine -n openshift-machine-api
Set the annotation on the compute machine that you want to delete by running the following command:
$ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/delete-machine="true"
Cordon and drain the node that you want to delete by running the following commands:
$ oc adm cordon <node_name>
$ oc adm drain <node_name>
Scale the compute machine set by running one of the following commands:
$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api
Or:
$ oc edit machineset <machineset> -n openshift-machine-api
You can alternatively apply the following YAML to scale the compute machine set:
|
You can scale the compute machine set up or down. It takes several minutes for the new machines to be available.
Verify the deletion of the intended machine by running the following command:
$ oc get machines
Random
, Newest
, and Oldest
are the three supported deletion options. The default is Random
, meaning that random machines are chosen and deleted when scaling compute machine sets down. The deletion policy can be set according to the use case by modifying the particular compute machine set:
spec:
deletePolicy: <delete_policy>
replicas: <desired_replica_count>
Specific machines can also be prioritized for deletion by adding the annotation machine.openshift.io/delete-machine=true
to the machine of interest, regardless of the deletion policy.
By default, the OKD router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker compute machine set to |
Custom compute machine sets can be used for use cases requiring that services run on specific nodes and that those services are ignored by the controller when the worker compute machine sets are scaling down. This prevents service disruption. |
You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.
With cluster-wide node selectors, when you create a pod in that cluster, OKD adds the default node selectors to the pod and schedules the pod on nodes with matching labels.
You configure cluster-wide node selectors by editing the Scheduler Operator custom resource (CR). You add labels to a node, a compute machine set, or a machine config. Adding the label to the compute machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.
You can add additional key/value pairs to a pod. But you cannot add a different value for a default key. |
To add a default cluster-wide node selector:
Edit the Scheduler Operator CR to add the default cluster-wide node selectors:
$ oc edit scheduler cluster
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: type=user-node,region=east (1)
mastersSchedulable: false
1 | Add a node selector with the appropriate <key>:<value> pairs. |
After making this change, wait for the pods in the openshift-kube-apiserver
project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.
Add labels to a node by using a compute machine set or editing the node directly:
Use a compute machine set to add labels to nodes managed by the compute machine set when a node is created:
Run the following command to add labels to a MachineSet
object:
$ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]' -n openshift-machine-api (1)
1 | Add a <key>/<value> pair for each label. |
For example:
$ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]' -n openshift-machine-api
You can alternatively apply the following YAML to add labels to a compute machine set:
|
Verify that the labels are added to the MachineSet
object by using the oc edit
command:
For example:
$ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api
MachineSet
objectapiVersion: machine.openshift.io/v1beta1
kind: MachineSet
...
spec:
...
template:
metadata:
...
spec:
metadata:
labels:
region: east
type: user-node
...
Redeploy the nodes associated with that compute machine set by scaling down to 0
and scaling up the nodes:
For example:
$ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
$ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
When the nodes are ready and available, verify that the label is added to the nodes by using the oc get
command:
$ oc get nodes -l <key>=<value>
For example:
$ oc get nodes -l type=user-node
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp Ready worker 61s v1.25.0
Add labels directly to a node:
Edit the Node
object for the node:
$ oc label nodes <name> <key>=<value>
For example, to label a node:
$ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east
You can alternatively apply the following YAML to add labels to a node:
|
Verify that the labels are added to the node using the oc get
command:
$ oc get nodes -l <key>=<value>,<key>=<value>
For example:
$ oc get nodes -l type=user-node,region=east
NAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 Ready worker 17m v1.25.0
All nodes send heartbeats to the Kubernetes Controller Manager Operator (kube controller) in the OKD cluster every 10 seconds, by default. If the cluster does not receive heartbeats from a node, OKD responds using several default mechanisms.
For example, if the Kubernetes Controller Manager Operator loses contact with a node after a configured period:
The node controller on the control plane updates the node health to Unhealthy
and marks the node Ready
condition as Unknown
.
In response, the scheduler stops scheduling pods to that node.
The on-premise node controller adds a node.kubernetes.io/unreachable
taint with a NoExecute
effect to the node and schedules any pods on the node for eviction after five minutes, by default.
This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager Operator might not receive an update from a healthy node due to network latency. The Kubernetes Controller Manager Operator would then evict pods from the node even though the node is healthy. To avoid this problem, you can use worker latency profiles to adjust the frequency that the kubelet and the Kubernetes Controller Manager Operator wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly in the event that network latency between the control plane and the worker nodes is not optimal.
These worker latency profiles are three sets of parameters that are pre-defined with carefully tuned values that let you control the reaction of the cluster to latency issues without needing to determine the best values manually.
Worker latency profiles are multiple sets of carefully-tuned values for the node-status-update-frequency
, node-monitor-grace-period
, default-not-ready-toleration-seconds
and default-unreachable-toleration-seconds
parameters. These parameters let you control the reaction of the cluster to latency issues without needing to determine the best values manually.
All worker latency profiles configure the following parameters:
node-status-update-frequency
. Specifies the amount of time in seconds that a kubelet updates its status to the Kubernetes Controller Manager Operator.
node-monitor-grace-period
. Specifies the amount of time in seconds that the Kubernetes Controller Manager Operator waits for an update from a kubelet before marking the node unhealthy and adding the node.kubernetes.io/not-ready
or node.kubernetes.io/unreachable
taint to the node.
default-not-ready-toleration-seconds
. Specifies the amount of time in seconds after marking a node unhealthy that the Kubernetes Controller Manager Operator waits before evicting pods from that node.
default-unreachable-toleration-seconds
. Specifies the amount of time in seconds after marking a node unreachable that the Kubernetes Controller Manager Operator waits before evicting pods from that node.
Manually modifying the |
The following Operators monitor the changes to the worker latency profiles and respond accordingly:
The Machine Config Operator (MCO) updates the node-status-update-frequency
parameter on the worker nodes.
The Kubernetes Controller Manager Operator updates the node-monitor-grace-period
parameter on the control plane nodes.
The Kubernetes API Server Operator updates the default-not-ready-toleration-seconds
and default-unreachable-toleration-seconds
parameters on the control plance nodes.
While the default configuration works in most cases, OKD offers two other worker latency profiles for situations where the network is experiencing higher latency than usual. The three worker latency profiles are described in the following sections:
With the Default
profile, each kubelet reports its node status to the Kubelet Controller Manager Operator (kube controller) every 10 seconds. The Kubelet Controller Manager Operator checks the kubelet for a status every 5 seconds.
The Kubernetes Controller Manager Operator waits 40 seconds for a status update before considering that node unhealthy. It marks the node with the node.kubernetes.io/not-ready
or node.kubernetes.io/unreachable
taint and evicts the pods on that node. If a pod on that node has the NoExecute
toleration, the pod gets evicted in 300 seconds. If the pod has the tolerationSeconds
parameter, the eviction waits for the period specified by that parameter.
Profile | Component | Parameter | Value |
---|---|---|---|
Default |
kubelet |
|
10s |
Kubelet Controller Manager |
|
40s |
|
Kubernetes API Server |
|
300s |
|
Kubernetes API Server |
|
300s |
Use the MediumUpdateAverageReaction
profile if the network latency is slightly higher than usual.
The MediumUpdateAverageReaction
profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds
parameter, the eviction waits for the period specified by that parameter.
The Kubernetes Controller Manager Operator waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts.
Profile | Component | Parameter | Value |
---|---|---|---|
MediumUpdateAverageReaction |
kubelet |
|
20s |
Kubelet Controller Manager |
|
2m |
|
Kubernetes API Server |
|
60s |
|
Kubernetes API Server |
|
60s |
Use the LowUpdateSlowReaction
profile if the network latency is extremely high.
The LowUpdateSlowReaction
profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the tolerationSeconds
parameter, the eviction waits for the period specified by that parameter.
The Kubernetes Controller Manager Operator waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts.
Profile | Component | Parameter | Value |
---|---|---|---|
LowUpdateSlowReaction |
kubelet |
|
1m |
Kubelet Controller Manager |
|
5m |
|
Kubernetes API Server |
|
60s |
|
Kubernetes API Server |
|
60s |
To implement a worker latency profile to deal with network latency, edit the node.config
object to add the name of the profile. You can change the profile at any time as latency increases or decreases.
You must move one worker latency profile at a time. For example, you cannot move directly from the Default
profile to the LowUpdateSlowReaction
worker latency profile. You must move from the default
worker latency profile to the MediumUpdateAverageReaction
profile first, then to LowUpdateSlowReaction
. Similarly, when returning to the default profile, you must move from the low profile to the medium profile first, then to the default.
You can also configure worker latency profiles upon installing an OKD cluster. |
To move from the default worker latency profile:
Move to the medium worker latency profile:
Edit the node.config
object:
$ oc edit nodes.config/cluster
Add spec.workerLatencyProfile: MediumUpdateAverageReaction
:
node.config
objectapiVersion: config.openshift.io/v1
kind: Node
metadata:
annotations:
include.release.openshift.io/ibm-cloud-managed: "true"
include.release.openshift.io/self-managed-high-availability: "true"
include.release.openshift.io/single-node-developer: "true"
release.openshift.io/create-only: "true"
creationTimestamp: "2022-07-08T16:02:51Z"
generation: 1
name: cluster
ownerReferences:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
name: version
uid: 36282574-bf9f-409e-a6cd-3032939293eb
resourceVersion: "1865"
uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
spec:
workerLatencyProfile: MediumUpdateAverageReaction (1)
...
1 | Specifies the medium worker latency policy. |
Scheduling on each worker node is disabled as the change is being applied.
When all nodes return to the Ready
condition, you can use the following command to look in the Kubernetes Controller Manager to ensure it was applied:
$ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5
...
- lastTransitionTime: "2022-07-11T19:47:10Z"
reason: ProfileUpdated
status: "False"
type: WorkerLatencyProfileProgressing
- lastTransitionTime: "2022-07-11T19:47:10Z" (1)
message: all static pod revision(s) have updated latency profile
reason: ProfileUpdated
status: "True"
type: WorkerLatencyProfileComplete
- lastTransitionTime: "2022-07-11T19:20:11Z"
reason: AsExpected
status: "False"
type: WorkerLatencyProfileDegraded
- lastTransitionTime: "2022-07-11T19:20:36Z"
status: "False"
...
1 | Specifies that the profile is applied and active. |
Optional: Move to the low worker latency profile:
Edit the node.config
object:
$ oc edit nodes.config/cluster
Change the spec.workerLatencyProfile
value to LowUpdateSlowReaction
:
node.config
objectapiVersion: config.openshift.io/v1
kind: Node
metadata:
annotations:
include.release.openshift.io/ibm-cloud-managed: "true"
include.release.openshift.io/self-managed-high-availability: "true"
include.release.openshift.io/single-node-developer: "true"
release.openshift.io/create-only: "true"
creationTimestamp: "2022-07-08T16:02:51Z"
generation: 1
name: cluster
ownerReferences:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
name: version
uid: 36282574-bf9f-409e-a6cd-3032939293eb
resourceVersion: "1865"
uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
spec:
workerLatencyProfile: LowUpdateSlowReaction (1)
...
1 | Specifies to use the low worker latency policy. |
Scheduling on each worker node is disabled as the change is being applied.
To change the low profile to medium or change the medium to low, edit the node.config
object and set the spec.workerLatencyProfile
parameter to the appropriate value.
Control plane machine sets provide management capabilities for control plane machines that are similar to what compute machine sets provide for compute machines. The availability and initial status of control plane machine sets on your cluster depend on your cloud provider and the version of OKD that you installed. For more information, see Getting started with the Control Plane Machine Set Operator.
You can create a compute machine set to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.
In a production deployment, it is recommended that you deploy at least three compute machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different compute machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
For information on infrastructure nodes and which components can run on infrastructure nodes, see Creating infrastructure machine sets.
To create an infrastructure node, you can use a machine set, assign a label to the nodes, or use a machine config pool.
For sample machine sets that you can use with these procedures, see Creating machine sets for different clouds.
Applying a specific node selector to all infrastructure components causes OKD to schedule those workloads on nodes with that label.
In addition to the ones created by the installation program, you can create your own compute machine sets to dynamically manage the machine compute resources for specific workloads of your choice.
Deploy an OKD cluster.
Install the OpenShift CLI (oc
).
Log in to oc
as a user with cluster-admin
permission.
Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml
.
Ensure that you set the <clusterID>
and <role>
parameter values.
If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster:
$ oc get machinesets -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1d 0 0 55m
agl030519-vplxk-worker-us-east-1e 0 0 55m
agl030519-vplxk-worker-us-east-1f 0 0 55m
Check values of a specific compute machine set:
$ oc get machineset <machineset_name> -n \
openshift-machine-api -o yaml
...
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: agl030519-vplxk (1)
machine.openshift.io/cluster-api-machine-role: worker (2)
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
1 | The cluster ID. |
2 | A default node label. |
Create the new MachineSet
CR:
$ oc create -f <file_name>.yaml
View the list of compute machine sets:
$ oc get machineset -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
agl030519-vplxk-infra-us-east-1a 1 1 1 1 11m
agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1d 0 0 55m
agl030519-vplxk-worker-us-east-1e 0 0 55m
agl030519-vplxk-worker-us-east-1f 0 0 55m
When the new compute machine set is available, the DESIRED
and CURRENT
values match. If the compute machine set is not available, wait a few minutes and run the command again.
See Creating infrastructure machine sets for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the machine API. |
Requirements of the cluster dictate that infrastructure, also called infra
nodes, be provisioned. The installer only provides provisions for control plane and worker nodes. Worker nodes can be designated as infrastructure nodes or application, also called app
, nodes through labeling.
Add a label to the worker node that you want to act as application node:
$ oc label node <node-name> node-role.kubernetes.io/app=""
Add a label to the worker nodes that you want to act as infrastructure nodes:
$ oc label node <node-name> node-role.kubernetes.io/infra=""
Check to see if applicable nodes now have the infra
role and app
roles:
$ oc get nodes
Create a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.
If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied. However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts. |
Edit the Scheduler
object:
$ oc edit scheduler cluster
Add the defaultNodeSelector
field with the appropriate node selector:
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: topology.kubernetes.io/region=us-east-1 (1)
...
1 | This example node selector deploys pods on nodes in the us-east-1 region by default. |
Save the file to apply the changes.
You can now move infrastructure resources to the newly labeled infra
nodes.
For information on how to configure project node selectors to avoid cluster-wide node selector key conflicts, see Project node selectors.
If you need infrastructure machines to have dedicated configurations, you must create an infra pool.
Add a label to the node you want to assign as the infra node with a specific label:
$ oc label node <node_name> <label>
$ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=
Create a machine config pool that contains both the worker role and your custom role as machine config selector:
$ cat infra.mcp.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: infra
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} (1)
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: "" (2)
1 | Add the worker role and your custom role. |
2 | Add the label you added to the node as a nodeSelector . |
Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool. |
After you have the YAML file, you can create the machine config pool:
$ oc create -f infra.mcp.yaml
Check the machine configs to ensure that the infrastructure configuration rendered successfully:
$ oc get machineconfig
NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED
00-master 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
00-worker 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-master-container-runtime 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-master-kubelet 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-worker-container-runtime 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-worker-kubelet 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-master-ssh 3.2.0 31d
99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-worker-ssh 3.2.0 31d
rendered-infra-4e48906dca84ee702959c71a53ee80e7 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 23m
rendered-master-072d4b2da7f88162636902b074e9e28e 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-master-3e88ec72aed3886dec061df60d16d1af 02c07496ba0417b3e12b78fb32baf6293d314f79 3.2.0 31d
rendered-master-419bee7de96134963a15fdf9dd473b25 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 17d
rendered-master-53f5c91c7661708adce18739cc0f40fb 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 13d
rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 7d3h
rendered-master-dc7f874ec77fc4b969674204332da037 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-2640531be11ba43c61d72e82dc634ce6 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-4e48906dca84ee702959c71a53ee80e7 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 7d3h
rendered-worker-4f110718fe88e5f349987854a1147755 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 17d
rendered-worker-afc758e194d6188677eb837842d3b379 02c07496ba0417b3e12b78fb32baf6293d314f79 3.2.0 31d
rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 13d
You should see a new machine config, with the rendered-infra-*
prefix.
Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra
. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.
After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration. |
Create a machine config:
$ cat infra.mc.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 51-infra
labels:
machineconfiguration.openshift.io/role: infra (1)
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: /etc/infratest
mode: 0644
contents:
source: data:,infra
1 | Add the label you added to the node as a nodeSelector . |
Apply the machine config to the infra-labeled nodes:
$ oc create -f infra.mc.yaml
Confirm that your new machine config pool is available:
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
infra rendered-infra-60e35c2e99f42d976e084fa94da4d0fc True False False 1 1 1 0 4m20s
master rendered-master-9360fdb895d4c131c7c4bebbae099c90 True False False 3 3 3 0 91m
worker rendered-worker-60e35c2e99f42d976e084fa94da4d0fc True False False 2 2 2 0 91m
In this example, a worker node was changed to an infra node.
See Node configuration management with machine config pools for more information on grouping infra machines in a custom pool.
After creating an infrastructure machine set, the worker
and infra
roles are applied to new infra nodes. Nodes with the infra
role are not counted toward the total number of subscriptions that are required to run the environment, even when the worker
role is also applied.
However, when an infra node is assigned the worker role, there is a chance that user workloads can get assigned inadvertently to the infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods that you want to control.
If you have an infra node that has the infra
and worker
roles assigned, you must configure the node so that user workloads are not assigned to it.
It is recommended that you preserve the dual |
Configure additional MachineSet
objects in your OKD cluster.
Add a taint to the infra node to prevent scheduling user workloads on it:
Determine if the node has the taint:
$ oc describe nodes <node_name>
oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
Name: ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
Roles: worker
...
Taints: node-role.kubernetes.io/infra:NoSchedule
...
This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.
If you have not configured a taint to prevent scheduling user workloads on it:
$ oc adm taint nodes <node_name> <key>=<value>:<effect>
For example:
$ oc adm taint nodes node1 node-role.kubernetes.io/infra=reserved:NoExecute
You can alternatively apply the following YAML to add the taint:
|
This example places a taint on node1
that has key node-role.kubernetes.io/infra
and taint effect NoSchedule
. Nodes with the NoSchedule
effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
If a descheduler is used, pods violating node taints could be evicted from the cluster. |
Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod
object specification:
tolerations:
- effect: NoExecute (1)
key: node-role.kubernetes.io/infra (2)
operator: Exists (3)
value: reserved (4)
1 | Specify the effect that you added to the node. |
2 | Specify the key that you added to the node. |
3 | Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node. |
4 | Specify the value of the key-value pair taint that you added to the node. |
This toleration matches the taint created by the oc adm taint
command. A pod with this toleration can be scheduled onto the infra node.
Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator. |
Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.
See Controlling pod placement using the scheduler for general information on scheduling a pod to a node.
Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created.
You can deploy the router pod to a different compute machine set. By default, the pod is deployed to a worker node.
Configure additional compute machine sets in your OKD cluster.
View the IngressController
custom resource for the router Operator:
$ oc get ingresscontroller default -n openshift-ingress-operator -o yaml
The command output resembles the following text:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
creationTimestamp: 2019-04-18T12:35:39Z
finalizers:
- ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
generation: 1
name: default
namespace: openshift-ingress-operator
resourceVersion: "11341"
selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
uid: 79509e05-61d6-11e9-bc55-02ce4781844a
spec: {}
status:
availableReplicas: 2
conditions:
- lastTransitionTime: 2019-04-18T12:36:15Z
status: "True"
type: Available
domain: apps.<cluster>.example.com
endpointPublishingStrategy:
type: LoadBalancerService
selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default
Edit the ingresscontroller
resource and change the nodeSelector
to use the infra
label:
$ oc edit ingresscontroller default -n openshift-ingress-operator
spec:
nodePlacement:
nodeSelector: (1)
matchLabels:
node-role.kubernetes.io/infra: ""
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/infra
value: reserved
- effect: NoExecute
key: node-role.kubernetes.io/infra
value: reserved
1 | Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. If you added a taint to the infrastructure node, also add a matching toleration. |
Confirm that the router pod is running on the infra
node.
View the list of router pods and note the node name of the running pod:
$ oc get pod -n openshift-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-86798b4b5d-bdlvd 1/1 Running 0 28s 10.130.2.4 ip-10-0-217-226.ec2.internal <none> <none>
router-default-955d875f4-255g8 0/1 Terminating 0 19h 10.129.2.4 ip-10-0-148-172.ec2.internal <none> <none>
In this example, the running pod is on the ip-10-0-217-226.ec2.internal
node.
View the node status of the running pod:
$ oc get node <node_name> (1)
1 | Specify the <node_name> that you obtained from the pod list. |
NAME STATUS ROLES AGE VERSION
ip-10-0-217-226.ec2.internal Ready infra,worker 17h v1.25.0
Because the role list includes infra
, the pod is running on the correct node.
You configure the registry Operator to deploy its pods to different nodes.
Configure additional compute machine sets in your OKD cluster.
View the config/instance
object:
$ oc get configs.imageregistry.operator.openshift.io/cluster -o yaml
apiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
creationTimestamp: 2019-02-05T13:52:05Z
fi