OKD cluster logging allows you to send logs to destinations outside of your OKD cluster instead of the default Elasticsearch instance.

  • Sending logs using Fluentd. You can use the Fluentd out_forward plug-in to securely send logs to another logging collector using the Fluent forward protocol.

  • Sending logs using syslog. You can use the Fluentd syslog plug-in to send logs to another logging collector using the syslog protocol (RFC 3164). Many logging collectors such as Fluentd, Rsyslog, and others support this protocol.

You can use the log forwarding feature, which can be easier to configure than the plug-ins. The log forwarding feature is currently in Technology Preview.

Changes that are introduced by the new log forwarding feature modified the support for the Fluentd syslog plug-in starting with the OKD 4.3 release. In OKD 4.3, you create a ConfigMap, as described below, to configure a Fluentd plug-in.

Configuring the Fluentd forward plug-ins to send logs to an external log aggregator

You can use the Fluentd forward plug-ins to send a copy of your logs to an external log aggregator, instead of the default Elasticsearch.

You can configure OKD to send logs to both the internal Elasticsearch instance and an outside log aggregator using the '@type copy' plug-in within the Fluentd output-application configuration file. Specific instructions for this configuration are beyond the scope of this documentation.

In this documentation, the OKD cluster is called the sender and the external aggregator is called the receiver.

This legacy out_forward method is deprecated and will be removed in a future release.

The forward plug-ins are provided with the Fluentd image as of v1.4.0. The in_forward plug-in implements the server side (receiver), and out_forward implements the client side (sender).

To configure OKD to send logs using out_forward, create a ConfigMap called secure-forward in the openshift-logging namespace that points to a receiver. On the receiver, configure the in_forward plug-in to receive the logs from OKD. For more information on using the in_forward plug-in, see the Fluentd documentation.

Changes introduced by the new log forward feature modified the support for out_forward starting with the OKD 4.3 release. In OKD 4.3, you create a ConfigMap, as described below, to configure out_forward.

Additionally, you can add any certificates required by your configuration to a secret named secure-forward that will be mounted to the Fluentd Pods.

Sample secure-forward.conf
<store>
  @type forward
  <security>
    self_hostname ${hostname} # ${hostname} is a placeholder.
    shared_key "fluent-receiver"
  </security>
  transport tls
  tls_verify_hostname false           # Set false to ignore server cert hostname.

  tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
  <buffer>
    @type file
    path '/var/lib/fluentd/secureforwardlegacy'
    queued_chunks_limit_size "#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }"
    chunk_limit_size "#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }"
    flush_interval "#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}"
    flush_at_shutdown "#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}"
    flush_thread_count "#{ENV['FLUSH_THREAD_COUNT'] || 2}"
    retry_max_interval "#{ENV['FORWARD_RETRY_WAIT'] || '300'}"
    retry_forever true
    # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
    # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
    # buffer to the remote - default is 'exception' because in_tail handles that case
    overflow_action "#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}"
  </buffer>
  <server>
    host fluent-receiver.openshift-logging.svc  # or IP
    port 24224
  </server>
</store>
Sample secure-forward ConfigMap based on the configuration
apiVersion: v1
data:
 secure-forward.conf: "<store>
     \ @type forward
     \ <security>
     \   self_hostname ${hostname} # ${hostname} is a placeholder.
     \   shared_key \"fluent-receiver\"
     \ </security>
     \ transport tls
     \ tls_verify_hostname false           # Set false to ignore server cert hostname.
     \ tls_cert_path '/etc/ocp-forward/ca-bundle.crt'
     \ <buffer>
     \   @type file
     \   path '/var/lib/fluentd/secureforwardlegacy'
     \   queued_chunks_limit_size \"#{ENV['BUFFER_QUEUE_LIMIT'] || '1024' }\"
     \   chunk_limit_size \"#{ENV['BUFFER_SIZE_LIMIT'] || '1m' }\"
     \   flush_interval \"#{ENV['FORWARD_FLUSH_INTERVAL'] || '5s'}\"
     \   flush_at_shutdown \"#{ENV['FLUSH_AT_SHUTDOWN'] || 'false'}\"
     \   flush_thread_count \"#{ENV['FLUSH_THREAD_COUNT'] || 2}\"
     \   retry_max_interval \"#{ENV['FORWARD_RETRY_WAIT'] || '300'}\"
     \   retry_forever true
     \   # the systemd journald 0.0.8 input plugin will just throw away records if the buffer
     \   # queue limit is hit - 'block' will halt further reads and keep retrying to flush the
     \   # buffer to the remote - default is 'exception' because in_tail handles that case
     \   overflow_action \"#{ENV['BUFFER_QUEUE_FULL_ACTION'] || 'exception'}\"
     \ </buffer>
     \ <server>
     \   host fluent-receiver.openshift-logging.svc  # or IP
     \   port 24224
     \ </server>
     </store>"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-01-15T18:56:04Z"
  name: secure-forward
  namespace: openshift-logging
  resourceVersion: "19148"
  selfLink: /api/v1/namespaces/openshift-logging/configmaps/secure-forward
  uid: 6fd83202-93ab-d851b1d0f3e8
Procedure

To configure the out_forward plug-in:

  1. Create a configuration file named secure-forward.conf for the out_forward parameters:

    1. Configure the secrets and TLS information:

       <store>
        @type forward
      
        self_hostname ${hostname} (1)
        shared_key <SECRET_STRING> (2)
      
        transport tls (3)
      
        tls_verify_hostname true (4)
        tls_cert_path <path_to_file> (5)
      1 Specify the default value of the auto-generated certificate common name (CN).
      2 Enter the Shared key between nodes
      3 Specify tls to enable TLS validation.
      4 Set to true to verify the server cert hostname. Set to false to ignore server cert hostname.
      5 Specify the path to private CA certificate file as /etc/ocp-forward/ca_cert.pem.

      To use mTLS, see the Fluentd documentation for information about client certificate, key parameters, and other settings.

    2. Configure the name, host, and port for your external Fluentd server:

        <server>
          name (1)
          host (2)
          hostlabel (3)
          port (4)
        </server>
        <server> (5)
          name
          host
        </server>
      1 Optionally, enter a name for this receiver.
      2 Specify the host name or IP of the receiver.
      3 Specify the host label of the receiver.
      4 Specify the port of the receiver.
      5 Optionally, add additional receivers. If you specify two or more receivers, out_secure_forward uses these server nodes in a round-robin order.

      For example:

        <server>
          name externalserver1
          host 192.168.1.1
          hostlabel externalserver1.example.com
          port 24224
        </server>
        <server>
          name externalserver2
          host externalserver2.example.com
          port 24224
        </server>
        </store>
  2. Create a ConfigMap named secure-forward in the openshift-logging namespace from the configuration file:

    $ oc create configmap secure-forward --from-file=secure-forward.conf -n openshift-logging
  3. Optionally, import any secrets required for the receiver:

    $ oc create secret generic secure-forward --from-file=<arbitrary-name-of-key1>=cert_file_from_fluentd_receiver --from-literal=shared_key=value_from_fluentd_receiver

    For example:

    $ oc create secret generic secure-forward --from-file=ca-bundle.crt=ca-for-fluentd-receiver/ca.crt --from-literal=shared_key=fluentd-receiver
  4. Refresh the fluentd Pods to apply the secure-forward secret and secure-forward ConfigMap:

    $ oc delete pod --selector logging-infra=fluentd
  5. Configure the secure-forward.conf file on the receiver to accept messages securely from OKD.

    When configuring the receiver, it must be able to accept messages securely from OKD.

You can find further explanation of how to set up the in_forward plug-in and the out_forward plug-in.

Sending logs using the Fluentd syslog plug-in (RFC 3164)

You can use the Fluentd syslog plug-in to send a copy of your logs to an external syslog server, not the default Elasticsearch. Note the following about the syslog plug-in:

  • uses syslog protocol (RFC 3164), not RFC 5424

  • does not support TLS and thus, is not secure

  • does not provide Kubernetes metadata, systemd data, or other metadata

The Fluentd syslog plug-in method is deprecated and will be removed in a future release.

There are two versions of the Fluentd syslog plug-in:

  • out_syslog: The non-buffered implementation, which communicates through UDP, does not buffer data and writes out results immediately.

  • out_syslog_buffered: The buffered implementation, which communicates through TCP, buffers data into chunks.

To configure the Fluentd syslog plug-in, create a configuration file, called sysconfig.conf, with the information needed to forward the logs. Then use that file to create a ConfigMap called syslog in the openshift-logging namespace, which OKD uses when forwarding the logs. On the receiver, configure the in_syslog plug-in to receive the logs from OKD. For more information on using the in_forward plug-in, see the Fluentd documentation.

Changes introduced by the new log forward feature modified the support for Fluentd syslog plug-in starting with the OKD 4.3 release. In OKD 4.3, you create a ConfigMap, as described below, to configure the Fluentd syslog plug-in.

You can use multiple syslog servers by specifying separate <store> stanzas in the configuration file.

Sample sysconfig.conf
<store>
@type syslog_buffered (1)
remote_syslog rsyslogserver.openshift-logging.svc.cluster.local (2)
port 514 (3)
hostname fluentd-4nzfz (4)
remove_tag_prefix tag (5)
tag_key ident,systemd.u.SYSLOG_IDENTIFIER (6)
facility local0 (7)
severity info (8)
use_record true (9)
payload_key message (10)
</store>
1 The Fluentd syslog plug-in.
2 The fully qualified domain name (FQDN) or IP address of the syslog server.
3 The port number to connect on. Defaults to 514.
4 The name of the syslog server.
5 Removes the prefix from the tag. Defaults to '' (empty).
6 The field to set the syslog key.
7 The syslog log facility or source.
8 The syslog log severity.
9 Determines whether to use the severity and facility from the record if available.
10 The key to set the payload of the syslog message. Defaults to message.
Sample syslog ConfigMap based on the sample sysconfig.conf
kind: ConfigMap
apiVersion: v1
metadata:
  name: syslog
  namespace: openshift-logging
data:
  sysconfig.conf: |
    <store>
     @type syslog_buffered
     remote_syslog syslogserver.openshift-logging.svc.cluster.local
     port 514
     hostname fluentd-4nzfz
     remove_tag_prefix tag
     tag_key ident,systemd.u.SYSLOG_IDENTIFIER
     facility local0
     severity info
     use_record true
     payload_key message
    </store>
Procedure

To configure the Fluentd syslog plug-in:

  1. Create a configuration file named sysconfig.conf that contains the following parameters within the <store> stanza:

    1. Specify the Fluentd syslog plug-in type:

      @type syslog_buffered (1)
      1 Specify the plug-in to use, either: syslog or syslog_buffered.
    2. Configure the name, host, and port for your external syslog server:

      remote_syslog <remote> (1)
      port <number> (2)
      hostname <name> (3)
      1 Specify the FQDN or IP address of the syslog server.
      2 Specify the port of the syslog server.
      3 Specify a name for this syslog server.

      For example:

      remote_syslog syslogserver.openshift-logging.svc.cluster.local
      port 514
      hostname fluentd-server

      If you specify two or more receivers, the Fluentd syslog plug-in uses these servers in a round-robin order.

    3. Configure the other syslog variables as needed:

      remove_tag_prefix (1)
      tag_key <key> (2)
      facility <value>  (3)
      severity <value>  (4)
      use_record <value> (5)
      payload_key message (6)
      1 Add this parameter to remove the tag field from the syslog prefix.
      2 Specify the field to set the syslog key.
      3 Specify the syslog log facility or source. For values, see RTF 3164.
      4 Specify the syslog log severity. For values, see link:RTF 3164.
      5 Specify true to use the severity and facility from the record if available. If true, the container_name, namespace_name, and pod_name are included in the output content.
      6 Specify the key to set the payload of the syslog message. Defaults to message.

      For example:

      facility local0
      severity info

      The configuration file appears similar to the following:

      <store>
      @type syslog_buffered
      remote_syslog syslogserver.openshift-logging.svc.cluster.local
      port 514
      hostname fluentd-4nzfz
      tag_key ident,systemd.u.SYSLOG_IDENTIFIER
      facility local0
      severity info
      use_record false
      </store>
  2. Create a ConfigMap named syslog in the openshift-logging namespace from the configuration file:

    $ oc create configmap syslog --from-file=sysconfig.conf -n openshift-logging

    The Cluster Logging Operator redeploys the Fluentd Pods. If the Pods do not redeploy, you can delete the Fluentd Pods to force them to redeploy.

    $ oc delete pod --selector logging-infra=fluentd