As you scale up your project, the default logging environment might need some
For example, if you are missing logs, you might have to increase the rate limits for journald.
You can adjust the number of messages to retain for a specified period of time to ensure that
OpenShift Logging does not use excessive resources without dropping logs.
You can also determine if you want the logs compressed, how long to retain logs, how or if the logs are stored,
and other settings.
Create a Butane config file,
40-worker-custom-journald.bu, that includes an
/etc/systemd/journald.conf file with the required settings.
See "Creating machine configs with Butane" for information about Butane.
- path: /etc/systemd/journald.conf
mode: 0644 (1)
||Set the permissions for the
journal.conf file. It is recommended to set
||Specify whether you want logs compressed before they are written to the file system.
yes to compress the message or
no to not compress. The default is
||Configure whether to forward log messages. Defaults to
no for each. Specify:
ForwardToConsole to forward logs to the system console.
ForwardToKsmg to forward logs to the kernel log buffer.
ForwardToSyslog to forward to a syslog daemon.
ForwardToWall to forward messages as wall messages to all logged-in users.
||Specify the maximum time to store journal entries. Enter a number to specify seconds. Or
include a unit: "year", "month", "week", "day", "h" or "m". Enter
0 to disable. The default is
||Configure rate limiting. If more logs are received than what is specified in
RateLimitBurst during the time interval defined by
RateLimitIntervalSec, all further messages within the interval are dropped until the interval is over. It is recommended to set
RateLimitBurst=10000, which are the defaults.
||Specify how logs are stored. The default is
volatile to store logs in memory in
persistent to store logs to disk in
/var/log/journal/. systemd creates the directory if it does not exist.
auto to store logs in
/var/log/journal/ if the directory exists. If it does not exist, systemd temporarily stores logs in
none to not store logs. systemd drops all logs.
||Specify the timeout before synchronizing journal files to disk for ERR, WARNING, NOTICE, INFO, and DEBUG logs.
systemd immediately syncs after receiving a CRIT, ALERT, or EMERG log. The default is
||Specify the maximum size the journal can use. The default is
||Specify how much disk space systemd must leave free. The default is
||Specify the maximum size for individual journal files stored persistently in
/var/log/journal. The default is
If you are removing the rate limit, you might see increased CPU utilization on the
system logging daemons as it processes any messages that would have previously
Use Butane to generate a
MachineConfig object file,
40-worker-custom-journald.yaml, containing the configuration to be delivered to the nodes:
$ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
Apply the machine config. For example:
$ oc apply -f 40-worker-custom-journald.yaml
The controller detects the new
MachineConfig object and generates a new
Monitor the status of the rollout of the new rendered configuration to each node:
$ oc describe machineconfigpool/worker
API Version: machineconfiguration.openshift.io/v1
Reason: All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e