Boot image skew enforcement

About boot image skew enforcement
- About boot image skew enforcement modes
Configuring boot image skew enforcement
Updating the boot image skew enforcement version
Additional resources

You can use boot image skew enforcement to help ensure that the boot images in a cluster are up-to-date with the OKD and FCOS version being used in the cluster. Using an older boot image could cause issues when scaling new nodes. If the images are older than a predetermined version, the MCO disables cluster upgrades until it deems the boot images to be compliant.

About boot image skew enforcement

Using boot image skew enforcement, you can ensure that the boot images in a cluster are up-to-date with the OKD and FCOS version being used in the cluster. Making sure that your boot images are current can help you avoid the problems associated with running older images.

When boot image skew enforcement is active in a cluster, the Machine Config Operator (MCO) examines the boot image version reported in the MachineConfiguration object to determine if that boot image is appropriate for the cluster. If the boot image version is too old, the Operator reports that boot image version skew is detected and blocks cluster updates until you manually update the boot image or disable boot image skew enforcement by setting the None mode, as described in this section.

The limit for boot image version skew is set within the MCO and cannot be modified.

For information on manually configuring the boot image in your cluster, see "Manually updating the boot image".

About boot image skew enforcement modes

Review the following information to learn about the boot image skew enforcement modes. Use the information to determine the best method for your cluster.

Boot image skew enforcement operates in one of the following modes:

Automatic

When set to Automatic, with boot image management also enabled, if the cluster is updated from one OKD version to the next, the MCO automatically updates the boot image version in the MachineConfiguration object and tests the boot image version for skew.

In OKD 4.22, the automatic mode is available only for AWS, Google Cloud, Azure, and vSphere clusters and is the default for these platforms.

The MCO automatically configures this mode when the following conditions are met:

Boot image management is available for the platform that your cluster uses. Currently boot image management is available for only AWS, Google Cloud, Azure, and vSphere clusters.
You have enabled boot image management for compute machine sets.
You have not set skew enforcement to the manual or none mode.

For information on boot image management, see "Boot image management".

Example MachineConfiguration object with automatic skew enforcement

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
status:
# ...
  bootImageSkewEnforcementStatus:
    automatic:
      ocpVersion: 4.22.0
    mode: Automatic

The MCO examines the boot image reported in the ocpVersion parameter to determine if the cluster is violating the boot image version skew limits.

Manual

When set to Manual, if the boot image version is updated, a cluster administrator is responsible for manually updating the MachineConfiguration object with the FCOS version of the new boot image or the OKD version associated with the new boot image. The MCO then tests the boot image version for skew.

Example MachineConfiguration object with skew enforcement based on an FCOS version

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
# ...
spec:
  bootImageSkewEnforcement:
    mode: Manual
    manual:
      mode: RHCOSVersion
      rhcosVersion: 9.2.20251023-0
# ...
status:
  bootImageSkewEnforcementStatus:
    manual:
      mode: RHCOSVersion
      rhcosVersion: 9.2.20251023-0
    mode: Manual

Example MachineConfiguration object with skew enforcement based on an OKD version

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
# ...
spec:
  bootImageSkewEnforcement:
    manual:
      mode: OCPVersion
      ocpVersion: 4.22.0
    mode: Manual
# ...
status:
  bootImageSkewEnforcementStatus:
    manual:
      mode: OCPVersion
      ocpVersion: 4.22.0
    mode: Manual

The MCO examines the boot image reported in the rhcosVersion or ocpVersion parameter to determine if the cluster is violating the boot image version skew limits.

None

When set to None, boot image skew enforcement is disabled. When disabled, the MCO does not monitor for boot image skew and does not report if new nodes are provisioned with older boot images, which could introduce issues when scaling new nodes.

Example MachineConfiguration object with skew enforcement disabled

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
# ...
spec:
  bootImageSkewEnforcement:
    mode: None
# ...
status:
  bootImageSkewEnforcementStatus:
    mode: None

When in the none mode, the MCO reports a Prometheus alert that skew enforcement is disabled and that scale-up might run into issues due to old boot images. The alert does not cause any functional issues for the cluster.

Single-node OpenShift clusters default to the none mode regardless of platform, because they do not scale. The skew enforcement Prometheus alert is not reported for single-node OpenShift clusters.

Bare-metal clusters running OKD version 4.10 and later do not use the MCO to keep their boot images up-to-date. Skew enforcement defaults to the none mode and the skew enforcement Prometheus alert mentioned is not reported. For bare-metal clusters running OKD version 4.9 and earlier, you need to perform a one-time action to migrate to the 4.10 system, this is explained further in the bare metal boot image update docs. For information, see "Manually updating the boot image".

Configuring boot image skew enforcement

You can configure the current boot image skew enforcement mode that the Machine Config Operator (MCO) uses. By configuring the boot image skew enforcement mode, you can determine if the boot image version in the MachineConfiguration object is updated automatically or manually.

Alternatively, you can disable boot image skew enforcement by setting the mode to None. When disabled, the MCO does not monitor for boot image skew, and older boot images could be used, possibly introducing issues when scaling new nodes.

In OKD 4.22, the automatic mode is available only for AWS, Google Cloud, Azure, and vSphere clusters and is the default for these platforms. If you modify a cluster from the automatic mode to the manual or none mode, you can revert a cluster back to automatic mode only by removing the bootImageSkewEnforcement stanza from the MachineConfiguration object.

All other platforms default to manual mode with the OKD version set as the boot image version in the MachineConfiguration object. In manual mode, you are expected to manually update the MachineConfiguration object with new boot image version whenever you update the boot image.

Procedure

For manual mode, you can obtain the current boot image on a node by using the following command:
```
$ oc debug node/<new-node> -- chroot /host cat /sysroot/.coreos-aleph-version.json
```
Example output
```
# ...
    "ref": "docker://ostree-image-signed:oci-archive:/rhcos-9.6.20251023-0-ostree.x86_64.ociarchive",
    "version": "9.6.20251023-0"
```
You should use the newest node on the cluster, because the boot image might have been updated after the older nodes were created. Ideally, test the newest node from each machine set and use the oldest boot image among them.
Specify the boot image skew enforcement mode and set the boot image version as needed:
```
apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
spec:
# ...
  bootImageSkewEnforcement:
    mode: Manual
    manual:
      mode: RHCOSVersion
      rhcosVersion: 9.6.20251023-0
# ...
```
where:
spec.bootImageSkewEnforcement.mode
Specifies the boot image enforcement mode, one of the following values:
Manual. Specifies that boot image skew management is in manual mode. You must specify the spec.bootImageSkewEnforcement.manual parameters.

None. Specifies that boot image skew management is disabled. You do not need to specify the spec.bootImageSkewEnforcement.manual parameters.
spec.bootImageSkewEnforcement.manual.mode
Specifies the version you want to represent the current boot image, either OCPVersion or RHCOSVersion. You must include one of the following parameters:
For RHCOSVersion, use spec.bootImageSkewEnforcement.manual.rhcosVersion to specify the FCOS version that is being used as a boot image in the [major].[minor].[datestamp(YYYYMMDD)]-[buildnumber] or [major].[minor].[timestamp(YYYYMMDDHHmm)]-[buildnumber] format. This field must be between 14 and 21 characters.

For OCPVersion, use spec.bootImageSkewEnforcement.manual.ocpVersion to specify the OKD version associated with the boot image that is being used in the x.y.z format. This field must be between 5 and 10 characters.

Updating the boot image skew enforcement version

If you are running boot image skew enforcement in the manual mode, you must manually update the boot image version in the MachineConfiguration object each time you update the boot image in your cluster. With the boot image updated in the MachineConfiguration object, the Machine Config Operator (MCO) can properly perform boot image skew enforcement to ensure that your nodes are up-to-date.

Procedure

If necessary, obtain the FCOS or OKD version of the current boot image on an updated node by using one of the following commands:
- Obtain the FCOS version by running the following command:
  $ oc debug node/<new-node> -- chroot /host cat /sysroot/.coreos-aleph-version.json
  Example output
  # ... "ref": "docker://ostree-image-signed:oci-archive:/rhcos-9.6.20251023-0-ostree.x86_64.ociarchive", "version": "9.6.20251023-0"
- Obtain the OKD version by running the following command:
  $ openshift-install version
  Ensure that you use the same openshift-install binary that you used when updating the boot image.
  Example output
  openshift-install 4.22.0
Specify the boot image version in the MachineConfiguration object with either the FCOS or OKD version:
- Update the MachineConfiguration object with the FCOS version:
  apiVersion: operator.openshift.io/v1 kind: MachineConfiguration metadata: name: cluster # ... spec: bootImageSkewEnforcement: mode: Manual manual: mode: RHCOSVersion rhcosVersion: 9.2.20251023-0 # ...
  If the spec.bootImageSkewEnforcement.manual.mode is RHCOSVersion, specify the FCOS version of the boot image with the rhcosVersion parameter, as shown in the example.
- Update the MachineConfiguration object with the OKD version
  apiVersion: operator.openshift.io/v1 kind: MachineConfiguration metadata: name: cluster # ... spec: bootImageSkewEnforcement: mode: Manual manual: mode: OCPVersion ocpVersion: 4.22.0 # ...
  If the spec.bootImageSkewEnforcement.manual.mode is OCPVersion, specify the OKD version of the boot image with the ocpVersion parameter, as shown in the example.

About boot image skew enforcement

About boot image skew enforcement modes

Configuring boot image skew enforcement

Updating the boot image skew enforcement version

Additional resources