Overview

This topic serves as an alternative approach for node host upgrades to the in-place upgrade method.

The blue-green deployment upgrade method follows a similar flow to the in-place method: masters and etcd servers are still upgraded first, however a parallel environment is created for new node hosts instead of upgrading them in-place.

This method allows administrators to switch traffic from the old set of node hosts (e.g., the blue deployment) to the new set (e.g., the green deployment) after the new deployment has been verified. If a problem is detected, it is also then easy to rollback to the old deployment quickly.

While blue-green is a proven and valid strategy for deploying just about any software, there are always trade-offs. Not all environments have the same uptime requirements or the resources to properly perform blue-green deployments.

In an OKD environment, the most suitable candidate for blue-green deployments are the node hosts. All user processes run on these systems and even critical pieces of OKD infrastructure are self-hosted on these resources. Uptime is most important for these workloads and the additional complexity of blue-green deployments can be justified.

The exact implementation of this approach varies based on your requirements. Often the main challenge is having the excess capacity to facilitate such an approach.

Blue-Green Deployment Upgrade
Figure 1. Blue-Green Deployment

Preparing for a Blue-Green Upgrade

After you have upgraded your master and etcd hosts using method described for In-place Upgrades, use the following sections to prepare your environment for a blue-green upgrade of the remaining node hosts.

Labeling Blue Nodes

You must ensure that your current node hosts in production are labeled either blue or green. In this example, the current production environment will be blue and the new environment will be green.

  1. Get the current list of node names known to the cluster:

    $ oc get nodes
  2. Label all non-master node hosts (compute nodes) and dedicated infrastructure nodes in your current production environment with color=blue:

    $ oc label node --selector=node-role.kubernetes.io/compute=true,node-role.kubernetes.io/infra=true color=blue

    In the above command, the --selector flag is used to match a subset of the cluster using the relevant node labels, and all matches are labeled with color=blue.

Creating and Labeling Green Nodes

Create the new green environment for any node hosts that are to be replaced by adding an equal number of new node hosts to the existing cluster:

  1. Add the new node hosts using the procedure as described in Adding Hosts to an Existing Cluster. When updating your inventory file with the [new_nodes] group in that procedure, ensure these variables are set:

    • In order to delay workload scheduling until the nodes are deemed healthy (which you will verify in later steps), set the openshift_schedulable=false variable for each new node host to ensure they are unschedulable initially.

    • Because the new hosts will be on OKD 3.10, they must now use the openshift_node_group_name variable to set a node group per host entry. Refer back to Defining Node Groups and Host Mappings for details.

    For example, the following new node hosts would be added to your existing inventory. Everything that was in your inventory previously should remain:

    Example [new_nodes] Host Group
    [new_nodes]
    node4.example.com openshift_node_group_name='node-config-compute' openshift_schedulable=false
    node5.example.com openshift_node_group_name='node-config-compute' openshift_schedulable=false
    node6.example.com openshift_node_group_name='node-config-compute' openshift_schedulable=false
    infra-node3.example.com openshift_node_group_name='node-config-infra' openshift_schedulable=false
    infra-node4.example.com openshift_node_group_name='node-config-infra' openshift_schedulable=false
  2. After the new nodes have been deployed, apply the color=green label to each new node using the oc label node command:

    $ oc label node <node_name> color=green

Verifying Green Nodes

Verify that your new green nodes are in a healthy state. Perform the following checklist:

  1. Verify that new nodes are detected in the cluster and are in Ready,SchedulingDisabled state:

    $ oc get nodes
    
    NAME                STATUS                       ROLES       AGE
    node4.example.com   Ready,SchedulingDisabled     compute     1d
  2. Verify that the green nodes have proper labels:

    $ oc get nodes --show-labels
    
    NAME                STATUS                       ROLES       AGE   LABELS
    node4.example.com   Ready,SchedulingDisabled     compute     1d    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,color=green,kubernetes.io/hostname=m01.example.com,node-role.kubernetes.io/compute=true
  3. Perform a diagnostic check for the cluster:

    $ oc adm diagnostics
    
    [Note] Determining if client configuration exists for client/cluster diagnostics
    Info:  Successfully read a client config file at '/root/.kube/config'
    Info:  Using context for cluster-admin access: 'default/m01-example-com:8443/system:admin'
    [Note] Performing systemd discovery
    
    [Note] Running diagnostic: ConfigContexts[default/m01-example-com:8443/system:admin]
           Description: Validate client config context is complete and has connectivity
    ...
             [Note] Running diagnostic: CheckExternalNetwork
                  Description: Check that external network is accessible within a pod
    
           [Note] Running diagnostic: CheckNodeNetwork
                  Description: Check that pods in the cluster can access its own node.
    
           [Note] Running diagnostic: CheckPodNetwork
                  Description: Check pod to pod communication in the cluster. In case of ovs-subnet network plugin, all pods
    should be able to communicate with each other and in case of multitenant network plugin, pods in non-global projects
    should be isolated and pods in global projects should be able to access any pod in the cluster and vice versa.
    
           [Note] Running diagnostic: CheckServiceNetwork
                  Description: Check pod to service communication in the cluster. In case of ovs-subnet network plugin, all
    pods should be able to communicate with all services and in case of multitenant network plugin, services in non-global
    projects should be isolated and pods in global projects should be able to access any service in the cluster.
    ...

Warming the Green Nodes

In order for pods to be migrated from the blue environment to the green, the required container images must be pulled. Network latency and load on the registry can cause delays if there is not sufficient capacity built in to the environment.

Often, the best way to minimize impact to the running system is to trigger new pod deployments that will land on the new nodes. Accomplish this by importing new image streams.

Major releases of OKD (and sometimes asynchronous errata updates) introduce new image streams for builder images for users of Source-to-Image (S2I). Upon import, any builds or deployments configured with image change triggers are automatically created.

Another benefit of triggering the builds is that it does a fairly good job of fetching the majority of the ancillary images to all node hosts such as the various builder images, the pod infrastructure image, and deployers. The green nodes are then considered warmed (that is, ready for the expected load increase), and everything else can be migrated over using node evacuation in a later step, proceeding more quickly as a result.

When you are ready to continue with the upgrade process, follow these steps to warm the green nodes:

  1. Set the green nodes to schedulable so that new pods only land on them:

    $ oc adm manage-node --schedulable=true --selector=color=green
  2. Disable the blue nodes so that no new pods are run on them by setting them unschedulable:

    $ oc adm manage-node --schedulable=false --selector=color=blue
  3. Update the node selectors for the registry and router deployment configurations to use the node-role.kubernetes.io/infra=true label. This will trigger new deployments so that their new pods land on your new infrastructure nodes.

    1. Edit the docker-registry deployment configuration:

      $ oc edit -n default dc/docker-registry

      Update the nodeSelector field to the following (using the exact formatting, with "true" in quotes) and save your changes:

            nodeSelector:
              node-role.kubernetes.io/infra: "true"
    2. Edit the router deployment configuration:

      $ oc edit -n default dc/router

      Update the nodeSelector field to the following (using the exact formatting, with "true" in quotes) and save your changes:

            nodeSelector:
              node-role.kubernetes.io/infra: "true"
    3. Verify that the docker-registry and router pods are running and in ready state on the new infrastructure nodes:

      $ oc get pods -n default -o wide
      
      NAME                       READY     STATUS    RESTARTS   AGE       IP                NODE
      docker-registry-2-b7xbn    1/1       Running   0          18m       10.128.0.188      infra-node3.example.com
      router-2-mvq6p             1/1       Running   0          6m        192.168.122.184   infra-node4.example.com
  4. Update the default image streams and templates.

  5. Import the latest images. It is important to realize that this process can trigger a large number of builds. The good news is that the builds are performed on the green nodes and, therefore, do not impact any traffic on the blue deployment.

  6. To monitor build progress across all namespaces (projects) in the cluster:

    $ oc get events -w --all-namespaces

    In large environments, builds rarely completely stop. However, you should see a large increase and decrease caused by the administrative image import.

Evacuating and Decommissioning Blue Nodes

For larger deployments, it is possible to have other labels that help determine how evacuation can be coordinated. The most conservative approach for avoiding downtime is to evacuate one node host at a time.

If services are composed of pods using zone anti-affinity, then an entire zone can be evacuated at once. It is important to ensure that the storage volumes used are available in the new zone as this detail can vary among cloud providers.

In OKD 1.2 and later, a node host evacuation is triggered whenever the node service is stopped. Node labeling is very important and can cause issues if nodes are mislabeled or commands are run on nodes with generalized labels. Exercise caution if master hosts are also labeled with color=blue.

When you are ready to continue with the upgrade process, follow these steps.

  1. Evacuate and delete all blue nodes with the following commands:

    $ oc adm manage-node --selector=color=blue --evacuate
    $ oc delete node --selector=color=blue
  2. After the blue node hosts no longer contain pods and have been removed from OKD they are safe to power off. As a safety precaution, leaving the hosts around for a short period of time can prove beneficial if the upgrade has issues.

  3. Ensure that any desired scripts or files are captured before terminating these hosts. After a determined time period and capacity is not an issue, remove these hosts.