As a cluster administrator, you can deploy cluster logging to aggregate all the logs from your OKD cluster, such as node system audit logs, application container logs, and infrastructure logs. Cluster logging aggregates these logs from throughout your cluster and stores them in a default log store. You can view the logs in a console or the OKD web console or use Kibana to visualize log data.
OKD cluster administrators can deploy cluster logging using the OKD web console or CLI to install the Elasticsearch Operator and Cluster Logging Operator. When the operators are installed, you create a Cluster Logging Custom Resource (CR) to schedule cluster logging pods and other resources necessary to support cluster logging. The operators are responsible for deploying, upgrading, and maintaining cluster logging.
The Cluster Logging CR defines a complete cluster logging deployment that includes all the components
of the logging stack to collect, store and visualize logs. The Cluster Logging Operator watches the
Custom Resource and adjusts the logging deployment accordingly.
Administrators and application developers can view the logs of the projects for which they have view access.
The cluster logging components include a collector deployed to each node in the OKD cluster that collects all node and container logs and writes them to a log store. You can use a centralized web UI to create rich visualizations and dashboards with the aggregated data.
The major components of cluster logging are:
collection - This is the component that collects logs from the cluster, formats them, and forwards them to the log store. The current implementation is Fluentd.
log store - This is where the logs are stored. The default implementation is Elasticsearch. You can use the default Elasticsearch log store or forward logs to external log stores. The default log store is optimized and tested for short-term storage.
visualization - This is the UI component you can use to view logs, graphs, charts, and so forth. The current implementation is Kibana.
In this document, this document might refer to log store or Elasticsearch, visualization or Kibana, collection or Fluentd, interchangeably, except where noted.
OKD uses Fluentd to collect container and node logs.
By default, the log collector uses the following sources:
journald for all system logs
/var/log/containers/*.log for all container logs
The logging collector is deployed as a DaemonSet that deploys pods to each OKD node. System and infrastructure logs are generated by journald log messages from the operating system, the container runtime, and OKD. Application logs are generated by the CRI-O container engine. Fluentd collects the logs from these sources and forwards them internally or externally as you configure in OKD.
The container runtimes provide minimal information to identify the source of log messages: project, pod name, and container id. This is not sufficient to uniquely identify the source of the logs. If a pod with a given name and project is deleted before the log collector begins processing its logs, information from the API server, such as labels and annotations, might not be available. There might not be a way to distinguish the log messages from a similarly named pod and project or trace the logs to their source. This limitation means log collection and normalization is considered best effort.
The available container runtimes provide minimal information to identify the source of log messages and do not guarantee unique individual log messages or that these messages can be traced to their source.
By default, OKD uses Elasticsearch (ES) to store log data. Optionally, you can use the log forwarding features to forward logs to external log stores using Fluentd protocols, syslsog protocols, or the OKD Log Forwarding API.
The cluster logging Elasticsearch instance is optimized and tested for short term storage, approximately seven days. If you want to retain your logs over a longer term, it is recommended you move the data to a third-party storage system.
Elasticsearch organizes the log data from Fluentd into datastores, or indices, then subdivides each index into multiple pieces called shards, which it spreads across a set of Elasticsearch nodes in an Elasticsearch cluster. You can configure Elasticsearch to make copies of the shards, called replicas, which Elasticsearch also spreads across the Elasticsearch nodes. The Cluster Logging Custom Resource (CR) allows you to specify how the shards are replicated to provide data redundancy and resilience to failure. You can also specify how long the different types of logs are retained using a retention policy in the Cluster Logging CR.
The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.
The Cluster Logging Operator and companion Elasticsearch Operator ensure that each Elasticsearch node is deployed using a unique Deployment that includes its own storage volume. You can use a Cluster Logging Custom Resource (CR) to increase the number of Elasticsearch nodes, as needed. Refer to the Elasticsearch documentation for considerations involved in configuring storage.
A highly-available Elasticsearch environment requires at least three Elasticsearch nodes, each on a different host.
Role-based access control (RBAC) applied on the Elasticsearch indices enables the controlled access of the logs to the developers. Administrators can acccess all logs and developers can access only the logs in their projects.
OKD uses Kibana to display the log data collected by Fluentd and indexed by Elasticsearch.
Kibana is a browser-based console interface to query, discover, and visualize your Elasticsearch data through histograms, line graphs, pie charts, and other visualizations.
In OKD Latest, log curation is performed by Elasticsearch based on a retention policy that you define. The Elasticsearch Curator tool removes Elasticsearch indices that use the data model prior to OKD Latest.