To troubleshoot some issues with nodes, establish the kubelet’s log level verbosity depending on the issue to be tracked.

Modifying the kubelet as a one-time scenario

To modify the kubelet in a one-time scenario without rebooting the node due to the change of machine-config(spec":{"paused":false}}), allowing you to modify the kubelet without affecting the service, follow this procedure.

  1. Connect to the node in debug mode:

    $ oc debug node/<node>
    $ chroot /host
  2. After access is established, check the content:

    $ systemctl cat kubelet
    Example output
    # /etc/systemd/system/kubelet.service
    mode: 0644
    path: "/etc/systemd/system/kubelet.service.d/20-logging.conf"
      inline: |
  3. Define the new verbosity required in the /etc/systemd/system/kubelet.service.d/20-logging.conf file. In this example, the verbosity is changed from v=1 to v=8:

    $ vi -i -e 's/--v=1/--v=8/g' /etc/systemd/system/kubelet.service.d/20-logging.conf

    Editing the config file or installing a new logging.conf file overrides the log level.

  4. Restart the service:

    $ systemctl daemon-reload
    $ systemctl restart kubelet
  5. Gather the logs, then edit the kubelet log level to revert to the former value to prevent issues, such as this error:

    E0514 12:47:17.998892    2281 daemon.go:1350] content mismatch for file /etc/systemd/system/kubelet.service: [Unit]

Persistent kubelet log level configuration

  • Use the following MachineConfig object for persistent kubelet log level configuration:

     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
         machineconfiguration.openshift.io/role: master
       name: 99-master-kubelet-loglevel
           version: 3.2.0
             - name: kubelet.service
               enabled: true
                 - name: 30-logging.conf
                   contents: |

    Generally, it is recommended to apply 0-4 as debug-level logs and 5-8 as trace-level logs.

Log verbosity descriptions

Log verbosity Description


Always visible to an Operator.


A reasonable default log level if you do not want verbosity.


Useful steady state information about the service and important log messages that might correlate to significant changes in the system. This is the recommended default log level.


Extended information about changes.


Debug level verbosity.


Display requested resources.


Display HTTP request headers.


Display HTTP request contents.

Gathering kubelet logs

  • After the kubelet’s log level verbosity is configured properly, you can gather logs by running the following commands:

    $ oc adm node-logs --role master -u kubelet
    $ oc adm node-logs --role worker -u kubelet

    Alternatively, inside the node, run the following command:

    $ journalctl -b -f -u kubelet.service
  • To collect master container logs, run the following command:

    $ sudo tail -f /var/log/containers/*
  • To directly gather the logs of all nodes, run the following command:

    - for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done