1. Documentation
×

Performing Manual In-place Cluster Upgrades

Overview

As an alternative to performing an automated upgrade, you can manually upgrade your OKD cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

If you are using GlusterFS, see Special Considerations When Using Containerized GlusterFS before proceeding.

The day before the upgrade, validate OKD storage migration to ensure potential issues are resolved prior to the outage window:

$ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig

Preparing for a Manual Upgrade

Before attempting the upgrade, follow the steps in Verifying the Upgrade to verify the cluster’s health. This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings.

To prepare for a manual upgrade, follow these steps:

  1. Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections:

    # yum install atomic-openshift-utils

  2. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OKD version:

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  3. Create an etcd backup on each etcd host.

  4. For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system:

    # yum update kernel

  5. There is a small set of configurations that are possible in authorization policy resources in OKD 3.6 that are not supported by RBAC in OKD 3.7. Such configurations require manual migration based on your use case.

    If you are upgrading from OKD 3.6 to 3.7, to guarantee that all authorization policy objects are in sync with RBAC, run:

    $ oc adm migrate authorization

    This read-only command emulates the migration controller logic and reports if any resource is out of sync.

    During a rolling upgrade, avoid actions that require changes to OKD authorization policy resources such as the creation of new projects. If a project is created against a new master, the RBAC resources it creates will be deleted by the migration controller since they will be seen as out of sync from the authorization policy resources.

    If a project is created against an old master and the migration controller is no longer present due to a OKD 3.7 controller process being the leader, then its policy objects will not be synced and it will have no RBAC resources.

Upgrading Master Components

Before upgrading any stand-alone nodes, upgrade the master components (which provide the control plane for the cluster).

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude

  2. Upgrade etcd on all master hosts and any external etcd hosts.

    1. For RHEL 7 systems using the RPM-based method:

      1. Upgrade the etcd package:

        # yum update etcd

      2. Restart the etcd service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd
# journalctl -r -u etcd

    2. For RHEL Atomic Host 7 systems and RHEL 7 systems using the containerized method:

      1. Pull the latest rhel7/etcd image:

        # docker pull registry.access.redhat.com/rhel7/etcd

      2. Restart the etcd_container service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd_container
# journalctl -r -u etcd_container

  3. On each master host, upgrade the origin-master package:

    # yum upgrade origin-master

  4. If you are upgrading from OpenShift Origin 1.0 to 1.1:

    1. Create the following master proxy client certificates on the first master listed in the Ansible host inventory file by default /etc/ansible/hosts:

      # cd /etc/origin/master/
# oc adm ca create-master-certs --cert-dir=/etc/origin/master/ \
            --master=https://<internal-master-fqdn>:8443 \
            --public-master=https://<external-master-fqdn>:8443 \
            --hostnames=<external-master-fqdn>,<internal-master-fqdn>,localhost,127.0.0.1,<master-ip-address>,kubernetes.default.local \
            --overwrite=false

      This creates files at /etc/origin/master/master.proxy-client.{crt,key}. Then, add the master proxy client certificates to the /etc/origin/master/master-config.yml file on each master:

      kubernetesMasterConfig:
  proxyClientInfo:
    certFile: master.proxy-client.crt
    keyFile: master.proxy-client.key

    2. Enable the following renamed service(s) on master hosts.

      # systemctl enable origin-master-api
# systemctl enable origin-master-controllers

  5. Restart the master service(s) on each master and review logs to ensure they restart successfully.

    # systemctl restart origin-master-api origin-master-controllers
# journalctl -r -u origin-master-controllers
# journalctl -r -u origin-master-api

  6. Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the origin-node and openvswitch services:

    # systemctl restart openvswitch
# systemctl restart origin-node
# journalctl -r -u openvswitch
# journalctl -r -u origin-node

  7. Configure Dnsmasq. You need to perform the following steps to configure dnsmasq as part of the upgrade.

    1. Create a /etc/origin/node/node-dnsmasq.conf node configuration file.

      server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

    2. Edit the /etc/dnsmasq.d/origin-dns.conf file as follows:

      no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=5000
cache-size=5000
bind-dynamic
except-interface=lo

    3. Edit the /etc/dnsmasq.d/origin-upstream-dns.conf file as follows:

      server=<dns_server1_ip_address>
server=<dns_server2_ip_address>

    4. Edit the /etc/origin/node/node-config.yaml as follows:

      dnsBindAddress: 127.0.0.1:53
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
dnsDomain: cluster.local
dnsIP: <node_ip_address> (1)
      1 This is the IP address of the node host.

    5. Update the /etc/systemd/system/atomic-openshift-node.service node systemd unit file:

      [Unit]
Description=OpenShift Node
After=docker.service
Wants=openvswitch.service
After=ovsdb-server.service
After=ovs-vswitchd.service
Wants=docker.service
Documentation=https://github.com/openshift/origin
Requires=dnsmasq.service
After=dnsmasq.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/atomic-openshift-node
Environment=GOTRACEBACK=crash
ExecStartPre=/usr/bin/cp /etc/origin/node/node-dnsmasq.conf /etc/dnsmasq.d/
ExecStartPre=/usr/bin/dbus-send --system --dest=uk.org.thekelleys.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string:/in-addr.arpa/127.0.0.1,/cluster.local/127.0.0.1
ExecStopPost=/usr/bin/rm /etc/dnsmasq.d/node-dnsmasq.conf
ExecStopPost=/usr/bin/dbus-send --system --dest=uk.org.thekelleys.dnsmasq /uk/org/thekelleys/dnsmasq uk.org.thekelleys.SetDomainServers array:string:
ExecStart=/usr/bin/openshift start node --config=${CONFIG_FILE} $OPTIONS
LimitNOFILE=65536
LimitCORE=infinity
WorkingDirectory=/var/lib/origin/
SyslogIdentifier=atomic-openshift-node
Restart=always
RestartSec=5s
TimeoutStartSec=300
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

    6. Reload systemd and restart node service.

      # systemctl daemon-reload
# systemctl restart atomic-openshift-node dnsmaq

    7. Restart dnsmasq:

      # systemctl restart dnsmasq

  8. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    The node component on masters is set by default to unschedulable status during initial installation, so that pods are not deployed to them. However, it is possible to set them schedulable during the initial installation or manually thereafter. If any of your masters are also configured as a schedulable node, skip the following Docker upgrade steps for those masters and instead run all steps described in Upgrading Nodes when you get to that section for those hosts as well.

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
# journalctl -r -u docker

      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        If upgrading to RHEL Atomic Host 7.4.2, this upgrades Docker to version 1.12.

        		 
        # atomic host upgrade

    2. After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

      # systemctl reboot
# journalctl -r -u docker

    3. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf

  9. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

During the cluster upgrade, it can sometimes be useful to take a master out of rotation since some DNS client libraries will not properly to the other masters for cluster DNS. In addition to stopping the master and controller services, you can remove the EndPoint from the Kubernetes service’s subsets.addresses.

$ oc edit ep/kubernetes -n default

When the master is restarted, the Kubernetes service will be automatically updated.

Updating Policy Definitions

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

  1. If you customized default cluster roles and want to ensure a role reconciliation does not modify them, protect each role from reconciliation:

    $ oc annotate clusterrole.rbac <role_name> --overwrite rbac.authorization.kubernetes.io/autoupdate=false

    You must manually update the roles that contain this setting to include any new or required permissions after upgrading.

  2. Generate a default bootstrap policy template file:

    $ oc adm create-bootstrap-policy-file --filename=policy.json

    The contents of the file vary based on the OKD version, but the file contains only the default policies.

  3. Update the policy.json file to include any cluster role customizations.

  4. Use the policy file to automatically reconcile roles and role bindings that are not reconcile protected:

    $ oc process --local -f policy.json | oc auth reconcile -f policy.json

  5. Reconcile security context constraints:

    # oc adm policy reconcile-sccs \
    --additive-only=true \
    --confirm

Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the origin-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

You can alternatively use the blue-green deployment method at this point to create a parallel environment for new nodes instead of upgrading them in place.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude

  2. As a user with cluster-admin privileges, disable scheduling for the node:

    # oc adm manage-node <node> --schedulable=false

  3. Evacuate pods on the node to other nodes:

    The --force option deletes any pods that are not backed by a replication controller.

    		 
    # oc adm drain <node> --force --delete-local-data --ignore-daemonsets

  4. On the node host, upgrade all origin packages:

    # yum upgrade origin\*

  5. If you are upgrading from OpenShift Origin 1.0 to 1.1, enable the following renamed service on the node host:

    # systemctl enable origin-node

  6. Restart the origin-node and openvswitch services and review the logs to ensure they restart successfully:

    # systemctl restart openvswitch
# systemctl restart origin-node
# journalctl -r -u origin-node
# journalctl -r -u openvswitch

  7. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
# journalctl -r -u docker

        After Docker is restarted, restart the atomic-openshift-node service again and review the logs to ensure it restarts successfully:

        # systemctl restart atomic-openshift-node
# journalctl -r -u atomic-openshift-node

      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        If upgrading to RHEL Atomic Host 7.4.2, this upgrades Docker to version 1.12.

        		 
        # atomic host upgrade

        After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

        # systemctl reboot
# journalctl -r -u docker

    2. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf

  8. Re-enable scheduling for the node:

    # oc adm manage-node <node> --schedulable

  9. Run the following command on each node to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

  10. Repeat the previous steps on the next node, and continue repeating these steps until all nodes have been upgraded.

  11. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready:

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly available routing service.

If you are upgrading to OpenShift Origin 1.0.4 or 1.0.5, first see the Additional Manual Instructions per Release section for important steps specific to your upgrade, then continue with the router upgrade as described in this section.

Edit your router’s deployment configuration. For example, if it has the default router name:

# oc edit dc/router

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: openshift/origin-haproxy-router:v1.0.6 (1)
        imagePullPolicy: IfNotPresent
        ...
1 Adjust <tag> to match the version you are upgrading to (use v3.7.119 for the latest version).

You should see one router pod updated and then the next.

Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration:

# oc edit dc/docker-registry

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: openshift/origin-docker-registry:v1.0.4 (1)
        imagePullPolicy: IfNotPresent
        ...
1 Adjust <tag> to match the version you are upgrading to (use v3.7.119 for the latest version).

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

Updating Custom Registry Configuration Files

You may safely skip this part if you do not use a custom registry configuration file.

The internal Docker registry version 1.4.0 and higher requires following entries in the middleware section of the configuration file:

middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
  storage:
    - name: openshift

  1. Edit your custom configuration file, adding the missing entries.

  2. Deploy your updated configuration.

  3. Append the --overwrite flag to oc volume dc/docker-registry --add to replace a volume mount of your previous secret.

  4. You can safely remove the old secret.

Enforcing Quota in the Registry

Quota must be enforced to prevent layer blobs that exceed the size limit from being written to the registry’s storage. This can be achieved via a configuration file:

...
middleware:
  repository:
    - name: openshift
      options:
        enforcequota: true
...

Alternatively, use the REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA environment variable, which is set to true for the new registry deployments by default. Existing deployments need to be modified using:

# oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA=true

Updating the Default Image Streams and Templates

By default, the advanced installation method automatically creates default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under /usr/share/openshift/examples.

To update these objects:

  1. Ensure that you have the latest openshift-ansible code checked out, which provides the example JSON files:

    # cd ~/openshift-ansible
# git pull https://github.com/openshift/openshift-ansible master

  2. After a manual upgrade, get the latest templates from openshift-ansible-roles:

    # rpm -ql openshift-ansible-roles | grep examples | grep v3.7

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  3. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/.:

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/image-streams/image-streams-rhel7.json
$ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/image-streams/dotnet_imagestreams.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/image-streams/image-streams-rhel7.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/image-streams/dotnet_imagestreams.json

  4. Update the templates:

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/db-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/infrastructure-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/xpaas-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/xpaas-streams/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/db-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/infrastructure-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/xpaas-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior:

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/dancer.json": templates "dancer-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.7/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run:

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project:

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time:

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

Updating Master and Node Certificates

The following steps may be required for any OpenShift cluster that was originally installed prior to the OpenShift Origin 1.0.8 release. This may include any and all updates from that version.

Node Certificates

With the 1.0.8 release, certificates for each of the kubelet nodes were updated to include the IP address of the node. Any node certificates generated before the 1.0.8 release may not contain the IP address of the node.

If a node is missing the IP address as part of its certificate, clients may refuse to connect to the kubelet endpoint. Usually this will result in errors regarding the certificate not containing an IP SAN.

In order to remedy this situation, you may need to manually update the certificates for your node.

Checking the Node’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the node’s serving certificate. In this example, the Subject Alternative Names are mynode, mynode.mydomain.com, and 1.2.3.4:

# openssl x509 -in /etc/origin/node/server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mynode, DNS:mynode.mydomain.com, IP: 1.2.3.4

Ensure that the nodeIP value set in the /etc/origin/node/node-config.yaml file is present in the IP values from the Subject Alternative Names listed in the node’s serving certificate. If the nodeIP is not present, then it will need to be added to the node’s certificate.

If the nodeIP value is already contained within the Subject Alternative Names, then no further steps are required.

You will need to know the Subject Alternative Names and nodeIP value for the following steps.

Generating a New Node Certificate

If your current node certificate does not contain the proper IP address, then you must regenerate a new certificate for your node.

Node certificates will be regenerated on the master (or first master) and are then copied into place on node systems.

  1. Create a temporary directory in which to perform the following steps on the first master listed in the Ansible host inventory file by default /etc/ansible/hosts:

    # mkdir /tmp/node_certificate_update
# cd /tmp/node_certificate_update

  2. Export the signing options:

    # export signing_opts="--signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt"

  3. Generate the new certificate:

    # oc adm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=<existing_SANs>,<nodeIP>

    For example, if the Subject Alternative Names from before were mynode, mynode.mydomain.com, and 1.2.3.4, and the nodeIP was 10.10.10.1, then you would need to run the following command:

    # oc adm ca create-server-cert --cert=server.crt \
  --key=server.key $signing_opts \
  --hostnames=mynode,mynode.mydomain.com,1.2.3.4,10.10.10.1

Replace Node Serving Certificates

Back up the existing /etc/origin/node/server.crt and /etc/origin/node/server.key files for your node:

# mv /etc/origin/node/server.crt /etc/origin/node/server.crt.bak
# mv /etc/origin/node/server.key /etc/origin/node/server.key.bak

You must now copy the new server.crt and server.key created in the temporary directory during the previous step:

# mv /tmp/node_certificate_update/server.crt /etc/origin/node/server.crt
# mv /tmp/node_certificate_update/server.key /etc/origin/node/server.key

After you have replaced the node’s certificate, restart the node service:

# systemctl restart origin-node

Master Certificates

With the 1.0.8 release, certificates for each of the masters were updated to include all names that pods may use to communicate with masters. Any master certificates generated before the 1.0.8 release may not contain these additional service names.

Checking the Master’s Certificate

The following command can be used to determine which Subject Alternative Names (SANs) are present in the master’s serving certificate. In this example, the Subject Alternative Names are mymaster, mymaster.mydomain.com, and 1.2.3.4:

# openssl x509 -in /etc/origin/master/master.server.crt -text -noout | grep -A 1 "Subject Alternative Name"
X509v3 Subject Alternative Name:
DNS:mymaster, DNS:mymaster.mydomain.com, IP: 1.2.3.4

Ensure that the following entries are present in the Subject Alternative Names for the master’s serving certificate:

Entry Example

Kubernetes service IP address

172.30.0.1

All master host names

master1.example.com

All master IP addresses

192.168.122.1

Public master host name in clustered environments

public-master.example.com

kubernetes

kubernetes.default

kubernetes.default.svc

kubernetes.default.svc.cluster.local

openshift

openshift.default

openshift.default.svc

openshift.default.svc.cluster.local

If these names are already contained within the Subject Alternative Names, then no further steps are required.

Generating a New Master Certificate

If your current master certificate does not contain all names from the list above, then you generate a new certificate for your mater. Perform the following steps on the first master listed in the Ansible host inventory file by default /etc/ansible/hosts:

  1. Back up the existing /etc/origin/master/master.server.crt and /etc/origin/master/master.server.key files for your master:

    # mv /etc/origin/master/master.server.crt /etc/origin/master/master.server.crt.bak
# mv /etc/origin/master/master.server.key /etc/origin/master/master.server.key.bak

  2. Export the service names. These names will be used when generating the new certificate:

    # export service_names="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,openshift,openshift.default,openshift.default.svc,openshift.default.svc.cluster.local"

  3. You will need the first IP in the services subnet (the kubernetes service IP) as well as the values of masterIP, masterURL and publicMasterURL contained in the /etc/origin/master/master-config.yaml file for the following steps.

    The kubernetes service IP can be obtained with:

    # oc get svc/kubernetes --template='{{.spec.clusterIP}}'

  4. Generate the new certificate:

    # oc adm ca create-master-certs \
      --hostnames=<master_hostnames>,<master_IP_addresses>,<kubernetes_service_IP>,$service_names \ (1) (2) (3)
      --master=<internal_master_address> \ (4)
      --public-master=<public_master_address> \ (5)
      --cert-dir=/etc/origin/master/ \
      --overwrite=false
    1 Adjust <master_hostnames> to match your master host name. In a clustered environment, add all master host names.
    2 Adjust <master_IP_addresses> to match the value of masterIP. In a clustered environment, add all master IP addresses.
    3 Adjust <kubernetes_service_IP> to the first IP in the kubernetes services subnet.
    4 Adjust <internal_master_address> to match the value of masterURL.
    5 Adjust <public_master_address> to match the value of masterPublicURL.

  5. Restart master services. For single master deployments:

    # systemctl restart origin-master-api origin-master-controllers

    After the service restarts, the certificate update is complete.

Upgrading the Service Catalog

Manual upgrade steps for the service catalog and service brokers are not available.

Starting with OKD 3.7, the service catalog, OpenShift Ansible broker, and template service broker are enabled and deployed by default for new cluster installations. However, they are not deployed by default during the upgrade from OKD 3.6 to 3.7, so you must run an individual component playbook separate post-upgrade.

Upgrading from the OKD 3.6 Technology Preview version of the service catalog and service brokers is not supported.

To upgrade to these features:

  1. See the following three sections in the Advanced Installation topic and update your inventory file accordingly:

  2. Run the following playbook:

    # ansible-playbook -i </path/to/inventory/file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.yml

Upgrading the EFK Logging Stack

Manual upgrade steps for logging deployments are no longer available starting in OKD 1.5.

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

  1. If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_logging_install_logging=true (1)
openshift_logging_image_version=<tag> (2)
    1 Enables the ability to upgrade the logging stack.
    2 Replace <tag> with v3.7.119 for the latest version.

  2. Add any other openshift_logging_* variables that you want to specify to override the defaults, as described in Specifying Logging Ansible Variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging.yml playbook and complete the logging deployment upgrade.

If your Fluentd DeploymentConfig and DaemonSet for the EFK components are already set with:

        image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If so, manually change the DeploymentConfig and DaemonSet to imagePullPolicy: Always to make sure it is re-pulled.

Upgrading Cluster Metrics

Manual upgrade steps for metrics deployments are no longer available starting in OKD 1.5.

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

  1. If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_metrics_install_metrics=true (1)
openshift_metrics_image_version=<tag> (2)
openshift_metrics_hawkular_hostname=<fqdn> (3)
openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (4)
    1 Enables the ability to upgrade the metrics deployment.
    2 Replace <tag> with v3.7.119 for the latest version.
    3 Used for the Hawkular Metrics route. Should correspond to a fully qualified domain name.
    4 Choose a type that is consistent with the previous deployment.

  2. Add any other openshift_metrics_* variables that you want to specify to override the defaults, as described in Specifying Metrics Ansible Variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift_metrics.yml playbook and complete the metrics deployment upgrade.

Additional Manual Steps Per Release

Some OKD releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster.