1. Documentation
×

Configuring the logging collector

OKD uses Fluentd to collect operations and application logs from your cluster and enriches the data with Kubernetes Pod and Namespace metadata.

You can configure the CPU and memory limits for the log collector and move the log collector pods to specific nodes. All supported modifications to the log collector can be performed though the spec.collection.log.fluentd stanza in the Cluster Logging Custom Resource CR.

About unsupported configurations

The supported way of configuring cluster logging is by configuring it using the options described in this documentation. Do not use other configurations, as they are unsupported. Configuration paradigms might change across OKD releases, and such cases can only be handled gracefully if all configuration possibilities are controlled. If you use configurations other than those described in this documentation, your changes will disappear because the Elasticsearch Operator and Cluster Logging Operator reconcile any differences. The Operators reverse everything to the defined state by default and by design.

If you must perform configurations not described in the OKD documentation, you must set your Cluster Logging Operator or Elasticsearch Operator to Unmanaged. An unmanaged cluster logging environment is not supported and does not receive updates until you return cluster logging to Managed.

Viewing logging collector pods

You can use the oc get pods --all-namespaces -o wide command to see the nodes where the Fluentd are deployed.

Procedure

Run the following command in the openshift-logging project:

$oc get pods --selector component=fluentd -o wide -n openshift-logging
Example output
NAME           READY  STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
fluentd-8d69v  1/1    Running   0          134m    10.130.2.30   master1.example.com   <none>           <none>
fluentd-bd225  1/1    Running   0          134m    10.131.1.11   master2.example.com   <none>           <none>
fluentd-cvrzs  1/1    Running   0          134m    10.130.0.21   master3.example.com   <none>           <none>
fluentd-gpqg2  1/1    Running   0          134m    10.128.2.27   worker1.example.com   <none>           <none>
fluentd-l9j7j  1/1    Running   0          134m    10.129.2.31   worker2.example.com   <none>           <none>

Configure log collector CPU and memory limits

The log collector allows for adjustments to both the CPU and memory limits.

Procedure

  1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

    $ oc edit ClusterLogging instance
    apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"

....

spec:
  collection:
    logs:
      fluentd:
        resources:
          limits: (1)
            memory: 736Mi
          requests:
            cpu: 100m
            memory: 736Mi
    1 Specify the CPU and memory limits and requests as needed. The values shown are the default values.

Advanced configuration for the log forwarder

Cluster logging includes multiple Fluentd parameters that you can use for tuning the performance of the Fluentd log forwarder. With these parameters, you can change the following Fluentd behaviors:

  • the size of Fluentd chunks and chunk buffer

  • the Fluentd chunk flushing behavior

  • the Fluentd chunk forwarding retry behavior

Fluentd collects log data in a single blob called a chunk. When Fluentd creates a chunk, the chunk is considered to be in the stage, where the chunk gets filled with data. When the chunk is full, Fluentd moves the chunk to the queue, where chunks are held before being flushed, or written out to their destination. Fluentd can fail to flush a chunk for a number of reasons, such as network issues or capacity issues at the destination. If a chunk cannot be flushed, Fluentd retries flushing as configured.

By default in OKD, Fluentd uses the exponential backoff method to retry flushing, where Fluentd doubles the time it waits between attempts to retry flushing again, which helps reduce connection requests to the destination. You can disable exponential backoff and use the periodic retry method instead, which retries flushing the chunks at a specified interval. By default, Fluentd retries chunk flushing indefinitely. In OKD, you cannot change the indefinite retry behavior.

These parameters can help you determine the trade-offs between latency and throughput.

  • To optimize Fluentd for throughput, you could use these parameters to reduce network packet count by configuring larger buffers and queues, delaying flushes, and setting longer times between retries. Be aware that larger buffers require more space on the node file system.

  • To optimize for low latency, you could use the parameters to send data as soon as possible, avoid the build-up of batches, have shorter queues and buffers, and use more frequent flush and retries.

You can configure the chunking and flushing behavior using the following parameters in the Cluster Logging Custom Resource (CR). The parameters are then automatically added to the Fluentd ConfigMap for use by Fluentd.

These parameters are:

  • Not relevant to most users. The default settings should give good general performance.

  • Only for advanced users with detailed knowledge of Fluentd configuration and performance.

  • Only for performance tuning. They have no effect on functional aspects of logging.
Table 1. Advanced Fluentd Configuration Parameters
Parmeter Description Default

chunkLimitSize

The maximum size of each chunk. Fluentd stops writing data to a chunk when it reaches this size. Then, Fluentd sends the chunk to the queue and opens a new chunk.

8m

totalLimitSize

The maximum size of the buffer, which is the total size of the stage and the queue. If the buffer size exceeds this value, Fluentd stops adding data to chunks and fails with an error. All data not in chunks is lost.

8G

flushInterval

The interval between chunk flushes. You can use s (seconds), m (minutes), h (hours), or d (days).

1s

flushMode

The method to perform flushes:

  • lazy: Flush chunks based on the timekey parameter. You cannot modify the timekey parameter.

  • interval: Flush chunks based on the flushInterval parameter.

  • immediate: Flush chunks immediately after data is added to a chunk.

interval

flushThreadCount

The number of threads that perform chunk flushing. Increasing the number of threads improves the flush throughput, which hides network latency.

2

overflowAction

The chunking behavior when the queue is full:

  • throw_exception: Raise an exception to show in the log.

  • block: Stop data chunking until the full buffer issue is resolved.

  • drop_oldest_chunk: Drop the oldest chunk to accept new incoming chunks. Older chunks have less value than newer chunks.

block

queuedChunkLimitSize

The number of chunks in the queue.

32

retryMaxInterval

The maximum time in seconds for the exponential_backoff retry method.

300s

retryType

The retry method when flushing fails:

  • exponential_backoff: Increase the time between flush retries. Fluentd doubles the time it waits until the next retry until the retry_max_interval parameter is reached.

  • periodic: Retries flushes periodically, based on the retryWait parameter.

exponential_backoff

retryWait

The time in seconds before the next chunk flush.

1s

For more information on the Fluentd chunk lifecycle, see Buffer Plugins in the Fluentd documentation.

Procedure

  1. Edit the Cluster Logging Custom Resource (CR) in the openshift-logging project:

    $ oc edit ClusterLogging instance

  2. Add or modify any of the following parameters:

    apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  forwarder:
    fluentd:
      buffer:
        chunkLimitSize: 8m (1)
        flushInterval: 5s (2)
        flushMode: interval (3)
        flushThreadCount: 3 (4)
        overflowAction: throw_exception (5)
        retryMaxInterval: "300s" (6)
        retryType: periodic (7)
        retryWait: 1s (8)
        totalLimitSize: 32m (9)
...
    1 Specify the maximum size of each chunk before it is queued for flushing.
    2 Specify the interval between chunk flushes.
    3 Specify the method to perform chunk flushes: lazy, interval, or immediate.
    4 Specify the number of threads to use for chunk flushes.
    5 Specify the chunking behavior when the queue is full: throw_exception, block, or drop_oldest_chunk.
    6 Specify the maximum interval in seconds for the exponential_backoff chunk flushing method.
    7 Specify the retry type when chunk flushing fails: exponential_backoff or periodic.
    8 Specify the time in seconds before the next chunk flush.
    9 Specify the maximum size of the chunk buffer.

  3. Verify that the Fluentd pods are redeployed:

    $ oc get pods -n openshift-logging

  4. Check that the new values are in the fluentd ConfigMap:

    $ oc extract configmap/fluentd --confirm
    Example fluentd.conf
           <buffer>
        @type file
        path '/var/lib/fluentd/default'
        flush_mode interval
        flush_interval 5s
        flush_thread_count 3
        flush_at_shutdown true
        retry_type periodic
        retry_wait 1s
        retry_max_interval 300s
        retry_forever true
        queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '32' }"
        total_limit_size "#{ENV['TOTAL_LIMIT_SIZE'] ||  8589934592 }" #32M
        chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '8m'}"
        overflow_action throw_exception
      </buffer>