About This Upgrade

If you installed using the standard cluster installation process, and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.

OKD 3.10 involves signficant changes to the architecture of the cluster, which are discussed further in the OKD 3.10 Release Notes. The following sections detail how these notable technical changes affect the upgrade process from OKD 3.9.

Defining Node Groups and Host Mappings

Starting in OKD 3.10, node configurations are now bootstrapped from the master. When node services are started, the node checks if a kubeconfig and other node configuration files exist before joining the cluster. If they do not, the node pulls the configuration from the master, then joins the cluster.

This process replaces administrators having to manually maintain the node configuration uniquely on each node host. Instead, the contents of a node host’s /etc/origin/node/node-config.yaml file are now provided by ConfigMaps sourced from the master.

To map which ConfigMap to use for which node host, node groups must be defined and then set for each host in the inventory file. These procedures are handled during the Preparing for an Automated Upgrade section.

Node ConfigMaps

The Configmaps for defining the node configurations must be available in the openshift-node project. ConfigMaps are also now the authoritative definition for node labels; the old openshift_node_labels value is effectively ignored.

The installer provides a playbook for creating the following default ConfigMaps:

  • node-config-master

  • node-config-infra

  • node-config-compute

The following ConfigMaps are also created, which label nodes into multiple roles:

  • node-config-all-in-one

  • node-config-master-infra

Changes should not be made to a node host’s /etc/origin/node/node-config.yaml file. They will be overwritten by the configuration defined in the ConfigMap used by the node.

Node Group Definitions

After updating the openshift-ansible package, you can view what the default set of node group definitions looks like in YAML format in the ~/openshift-ansible/roles/openshift_facts/defaults/main.yml file:

openshift_node_groups:
  - name: node-config-master (1)
    labels:
      - 'node-role.kubernetes.io/master=true' (2)
    edits: [] (3)
  - name: node-config-infra
    labels:
      - 'node-role.kubernetes.io/infra=true'
    edits: []
  - name: node-config-compute
    labels:
      - 'node-role.kubernetes.io/compute=true'
    edits: []
  - name: node-config-master-infra
    labels:
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true'
    edits: []
  - name: node-config-all-in-one
    labels:
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true'
    edits: []
1 Node group name.
2 List of node labels associated with the node group.
3 Any edits to the node group’s configuration.

If you do not set the openshift_node_groups variable in your inventory file’s [OSEv3:vars] group, the defaults defined above will be used. However, if you want to deviate from these defaults, you must define the entire openshift_node_groups structure (including all desired node groups) in your inventory file.

The openshift_node_groups value is not merged with the defaults, and the YAML definition must first be translated into a Python dictionary. You can then use the edits field to modify any node configuration variables as desired by specifying the key-value pairs.

See Master and Node Configuration Files for reference on configurable node variables.

For example, the following entry in an inventory file defines groups named node-config-master, node-config-infra, and node-config-compute, and edits the ConfigMap for node-config-compute to set kubeletArguments.pods-per-core to 20:

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true',]}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

Whenever the openshift_node_group.yml playbook is run, the changes defined in the edits field will update the related ConfigMap (node-config-compute in this example), which will ultimately affect the node’s configuration file on the host.

Upgrade Workflow

The 3.9 to 3.10 control plane upgrade performs the following steps for you:

  • A backup of all etcd data is taken for recovery purposes.

  • The API and controllers are updated from 3.9 to 3.10.

  • Internal data structures are updated to 3.10.

  • The default router, if one exists, is updated from 3.9 to 3.10.

  • The default registry, if one exists, is updated from 3.9 to 3.10.

  • The default image streams and InstantApp templates are updated.

The 3.9 to 3.10 node upgrade performs a rolling update of nodes, which:

  • Marks a subset of nodes unschedulable and drains them of pods.

  • Updates node components from 3.9 to 3.10.

  • Converts from local configuration to bootstrapped TLS and Config.

  • Converts SDN components from systemd services to DaemonSets.

  • Returns those nodes to service.

  • Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

  • If you are using GlusterFS, see Special Considerations When Using Containerized GlusterFS before proceeding.

  • If you are using GCE Persistent Disk (gcePD), see Special Considerations When Using gcePD before proceeding.

  • The day before the upgrade, validate OKD storage migration to ensure potential issues are resolved prior to the outage window:

    $ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig

Automated upgrade playbooks are run via Ansible directly using the ansible-playbook command with an inventory file, similar to the advanced installation method. The same v3_10 upgrade playbooks can be used for either of the following scenarios:

Running Upgrade Playbooks

Ensure that you have the latest openshift-ansible code checked out:

# cd ~/openshift-ansible
# git pull https://github.com/openshift/openshift-ansible master

Then run one of the following upgrade playbooks utilizing the inventory file you used during the advanced installation. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location.

Upgrading OKD Minor Versions

To upgrade to a new OKD minor version run the following playbook:

# ansible-playbook \
    -i </path/to/inventory/file> \
    playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

Upgrading to OKD z-Stream Releases

To upgrade an existing OKD latest z-stream of a minor release (e.g., 3.10.z), run the following playbook:

# ansible-playbook \
    -i </path/to/inventory/file> \
    playbooks/byo/openshift-cluster/upgrades/<version>/upgrade.yml

After rebooting, continue to Verifying the Upgrade.