1. Documentation
×

About Logging

As a cluster administrator, you can deploy logging on an OKD cluster, and use it to collect and aggregate node system audit logs, application container logs, and infrastructure logs. You can forward logs to your chosen log outputs, including on-cluster, Red Hat managed log storage. You can also visualize your log data in the OKD web console, or the Kibana web console, depending on your deployed log storage solution.

The Kibana web console is now deprecated is planned to be removed in a future logging release.

OKD cluster administrators can deploy logging by using Operators. For information, see Installing logging.

The Operators are responsible for deploying, upgrading, and maintaining logging. After the Operators are installed, you can create a ClusterLogging custom resource (CR) to schedule logging pods and other resources necessary to support logging. You can also create a ClusterLogForwarder CR to specify which logs are collected, how they are transformed, and where they are forwarded to.

Because the internal OKD Elasticsearch log store does not provide secure storage for audit logs, audit logs are not stored in the internal Elasticsearch instance by default. If you want to send the audit logs to the default internal Elasticsearch log store, for example to view the audit logs in Kibana, you must use the Log Forwarding API as described in Forward audit logs to the log store.

Logging architecture

The major components of the logging are:

Collector

The collector is a daemonset that deploys pods to each OKD node. It collects log data from each node, transforms the data, and forwards it to configured outputs. You can use the Vector collector or the legacy Fluentd collector.

Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead.
Log store

The log store stores log data for analysis and is the default output for the log forwarder. You can use the default LokiStack log store, the legacy Elasticsearch log store, or forward logs to additional external log stores.

The OpenShift Elasticsearch Operator is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator.
Visualization

You can use a UI component to view a visual representation of your log data. The UI provides a graphical interface to search, query, and view stored logs. The OKD web console UI is provided by enabling the OKD console plugin.

The Kibana web console is now deprecated is planned to be removed in a future logging release.

Logging collects container logs and node logs. These are categorized into types:

Application logs

Container logs generated by user applications running in the cluster, except infrastructure container applications.

Infrastructure logs

Container logs generated by infrastructure namespaces: openshift*, kube*, or default, as well as journald messages from nodes.

Audit logs

Logs generated by auditd, the node audit system, which are stored in the /var/log/audit/audit.log file, and logs from the auditd, kube-apiserver, openshift-apiserver services, as well as the ovn project if enabled.

Additional resources

About deploying logging

Administrators can deploy the logging by using the OKD web console or the OpenShift CLI (oc) to install the logging Operators. The Operators are responsible for deploying, upgrading, and maintaining the logging.

Administrators and application developers can view the logs of the projects for which they have view access.

Logging custom resources

You can configure your logging deployment with custom resource (CR) YAML files implemented by each Operator.

Red Hat OpenShift Logging Operator:

  • ClusterLogging (CL) - After the Operators are installed, you create a ClusterLogging custom resource (CR) to schedule logging pods and other resources necessary to support the logging. The ClusterLogging CR deploys the collector and forwarder, which currently are both implemented by a daemonset running on each node. The Red Hat OpenShift Logging Operator watches the ClusterLogging CR and adjusts the logging deployment accordingly.

  • ClusterLogForwarder (CLF) - Generates collector configuration to forward logs per user configuration.

Loki Operator:

  • LokiStack - Controls the Loki cluster as log store and the web proxy with OKD authentication integration to enforce multi-tenancy.

OpenShift Elasticsearch Operator:

These CRs are generated and managed by the OpenShift Elasticsearch Operator. Manual changes cannot be made without being overwritten by the Operator.

  • ElasticSearch - Configure and deploy an Elasticsearch instance as the default log store.

  • Kibana - Configure and deploy Kibana instance to search, query and view logs.

About JSON OKD Logging

You can use JSON logging to configure the Log Forwarding API to parse JSON strings into a structured object. You can perform the following tasks:

  • Parse JSON logs

  • Configure JSON log data for Elasticsearch

  • Forward JSON logs to the Elasticsearch log store

About collecting and storing Kubernetes events

The OKD Event Router is a pod that watches Kubernetes events and logs them for collection by OKD Logging. You must manually deploy the Event Router.

For information, see About collecting and storing Kubernetes events.

About troubleshooting OKD Logging

You can troubleshoot the logging issues by performing the following tasks:

  • Viewing logging status

  • Viewing the status of the log store

  • Understanding logging alerts

  • Collecting logging data for Red Hat Support

  • Troubleshooting for critical alerts

About exporting fields

The logging system exports fields. Exported fields are present in the log records and are available for searching from Elasticsearch and Kibana.

For information, see About exporting fields.

About event routing

The Event Router is a pod that watches OKD events so they can be collected by logging. The Event Router collects events from all projects and writes them to STDOUT. Fluentd collects those events and forwards them into the OKD Elasticsearch instance. Elasticsearch indexes the events to the infra index.

You must manually deploy the Event Router.

For information, see Collecting and storing Kubernetes events.