Installing Logging - Logging | Observability

Prerequisites
Installing Logging and the Loki Operator using the CLI
Installing Logging and the Loki Operator using the web console

OKD Operators use custom resources (CRs) to manage applications and their components. You provide high-level configuration and settings through the CR. The Operator translates high-level directives into low-level actions, based on best practices embedded within the logic of the Operator. A custom resource definition (CRD) defines a CR and lists all the configurations available to users of the Operator. Installing an Operator creates the CRDs to generate CRs.

You must install the Red Hat OpenShift Logging Operator after the log store Operator.

You deploy logging by installing the Loki Operator to manage your log store, followed by the Red Hat OpenShift Logging Operator to manage the components of logging. You can use either the OKD web console or the OpenShift CLI (oc) to install or configure logging.

You can alternatively apply all example objects.

Prerequisites

You have downloaded the pull secret from Red Hat OpenShift Cluster Manager as shown in "Obtaining the installation program" in the installation documentation for your platform.

If you have the pull secret, add the redhat-operators catalog to the OperatorHub custom resource (CR) as shown in "Configuring OKD to use Red Hat Operators".

If there is no retention period defined on the s3 bucket or in the LokiStack custom resource (CR), then the logs are not pruned and they stay in the s3 bucket forever, which might fill up the s3 storage.

Installing Logging and the Loki Operator using the CLI

To install and configure logging on your OKD cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the OKD CLI.

Prerequisites

You have administrator permissions.
You installed the OpenShift CLI (oc).
You have access to a supported object store. For example: AWS S3, Google Cloud Storage, Azure, Swift, Minio, or OpenShift Data Foundation.

Procedure

The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y, where x.y represents the major and minor version of logging you have installed. For example, stable-5.7.

Create a Namespace object for Loki Operator:

Example Namespace object

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-operators-redhat (1)
  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-monitoring: "true" (2)

1 You must specify the openshift-operators-redhat namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat namespace and not the openshift-operators namespace. The openshift-operators namespace might contain community Operators, which are untrusted and could publish a metric with the same name as an OKD metric, which would cause conflicts.

2 A string value that specifies the label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

Apply the Namespace object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create a Subscription object for Loki Operator:

Example Subscription object

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: loki-operator
  namespace: openshift-operators-redhat (1)
spec:
  channel: stable (2)
  name: loki-operator
  source: redhat-operators (3)
  sourceNamespace: openshift-marketplace

1	You must specify the `openshift-operators-redhat` namespace.
2	Specify `stable`, or `stable-5.<y>` as the channel.
3	Specify `redhat-operators`. If your OKD cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the `CatalogSource` object you created when you configured the Operator Lifecycle Manager (OLM).

Apply the Subscription object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create a namespace object for the Red Hat OpenShift Logging Operator:

Example namespace object

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-logging (1)
  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-logging: "true"
    openshift.io/cluster-monitoring: "true" (2)

1	The Red Hat OpenShift Logging Operator is only deployable to the `openshift-logging` namespace.
2	A string value that specifies the label as shown to ensure that cluster monitoring scrapes the `openshift-operators-redhat` namespace.

Apply the namespace object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create an OperatorGroup object

Example OperatorGroup object

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging (1)
spec:
  targetNamespaces:
  - openshift-logging

1	You must specify the `openshift-logging` namespace.

Apply the OperatorGroup object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create a Subscription object:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging (1)
spec:
  channel: stable (2)
  name: cluster-logging
  source: redhat-operators (3)
  sourceNamespace: openshift-marketplace

1	You must specify the `openshift-logging` namespace.
2	Specify `stable`, or `stable-5.<y>` as the channel.
3	Specify `redhat-operators`. If your OKD cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).

Apply the Subscription object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create a LokiStack CR:

Example LokiStack CR

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki (1)
  namespace: openshift-logging (2)
spec:
  size: 1x.small (3)
  storage:
    schemas:
    - version: v13
      effectiveDate: "<yyyy>-<mm>-<dd>"
    secret:
      name: logging-loki-s3 (4)
      type: s3 (5)
      credentialMode: (6)
  storageClassName: <storage_class_name> (7)
  tenants:
    mode: openshift-logging (8)

1	Use the name `logging-loki`.
2	You must specify the `openshift-logging` namespace.
3	Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are `1x.extra-small`, `1x.small`, or `1x.medium`.
4	Specify the name of your log store secret.
5	Specify the corresponding storage type.
6	Optional field, logging 5.9 and later. Supported user configured values are as follows: `static` is the default authentication mode available for all supported object storage types using credentials stored in a Secret. `token` for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types. `token-cco` is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters.
7	Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the `oc get storageclasses` command.
8	LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.

Apply the LokiStack CR object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Create a ClusterLogging CR object:

Example ClusterLogging CR object

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance (1)
  namespace: openshift-logging (2)
spec:
  collection:
    type: vector
  logStore:
    lokistack:
      name: logging-loki
    retentionPolicy:
      application:
        maxAge: 7d
      audit:
        maxAge: 7d
      infra:
        maxAge: 7d
    type: lokistack
  visualization:
    type: ocp-console
    ocpConsole:
      logsLimit: 15
  managementState: Managed

1	Name must be `instance`.
2	Namespace must be `openshift-logging`.

Apply the ClusterLogging CR object by running the following command:
```
$ oc apply -f <filename>.yaml
```

Verify the installation by running the following command:

$ oc get pods -n openshift-logging

Example output

$ oc get pods -n openshift-logging
NAME                                               READY   STATUS    RESTARTS   AGE
cluster-logging-operator-fb7f7cf69-8jsbq           1/1     Running   0          98m
collector-222js                                    2/2     Running   0          18m
collector-g9ddv                                    2/2     Running   0          18m
collector-hfqq8                                    2/2     Running   0          18m
collector-sphwg                                    2/2     Running   0          18m
collector-vv7zn                                    2/2     Running   0          18m
collector-wk5zz                                    2/2     Running   0          18m
logging-view-plugin-6f76fbb78f-n2n4n               1/1     Running   0          18m
lokistack-sample-compactor-0                       1/1     Running   0          42m
lokistack-sample-distributor-7d7688bcb9-dvcj8      1/1     Running   0          42m
lokistack-sample-gateway-5f6c75f879-bl7k9          2/2     Running   0          42m
lokistack-sample-gateway-5f6c75f879-xhq98          2/2     Running   0          42m
lokistack-sample-index-gateway-0                   1/1     Running   0          42m
lokistack-sample-ingester-0                        1/1     Running   0          42m
lokistack-sample-querier-6b7b56bccc-2v9q4          1/1     Running   0          42m
lokistack-sample-query-frontend-84fb57c578-gq2f7   1/1     Running   0          42m

Installing Logging and the Loki Operator using the web console

To install and configure logging on your OKD cluster, an Operator such as Loki Operator for log storage must be installed first. This can be done from the OperatorHub within the web console.

Prerequisites

You have access to a supported object store (AWS S3, Google Cloud Storage, Azure, Swift, Minio, OpenShift Data Foundation).
You have administrator permissions.
You have access to the OKD web console.

Procedure

In the OKD web console Administrator perspective, go to Operators → OperatorHub.
Type Loki Operator in the Filter by keyword field. Click Loki Operator in the list of available Operators, and then click Install.

The Community Loki Operator is not supported by Red Hat.

Select stable or stable-x.y as the Update channel.

The Loki Operator must be deployed to the global operator group namespace openshift-operators-redhat, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it is created for you.

Select Enable Operator-recommended cluster monitoring on this namespace.

This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.
For Update approval select Automatic, then click Install.

If the approval strategy in the subscription is set to Automatic, the update process initiates as soon as a new Operator version is available in the selected channel. If the approval strategy is set to Manual, you must manually approve pending updates.
Install the Red Hat OpenShift Logging Operator:
1. In the OKD web console, click Operators → OperatorHub.
2. Choose Red Hat OpenShift Logging from the list of available Operators, and click Install.
3. Ensure that the A specific namespace on the cluster is selected under Installation Mode.
4. Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.
5. Select Enable Operator recommended cluster monitoring on this namespace.
  
  This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.
6. Select stable-5.y as the Update Channel.
7. Select an Approval Strategy.
  - The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
  - The Manual strategy requires a user with appropriate credentials to approve the Operator update.
8. Click Install.
Go to the Operators → Installed Operators page. Click the All instances tab.
From the Create new drop-down list, select LokiStack.

Select YAML view, and then use the following template to create a LokiStack CR:

Example LokiStack CR

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki (1)
  namespace: openshift-logging (2)
spec:
  size: 1x.small (3)
  storage:
    schemas:
    - version: v13
      effectiveDate: "<yyyy>-<mm>-<dd>"
    secret:
      name: logging-loki-s3 (4)
      type: s3 (5)
      credentialMode: (6)
  storageClassName: <storage_class_name> (7)
  tenants:
    mode: openshift-logging (8)

1	Use the name `logging-loki`.
2	You must specify the `openshift-logging` namespace.
3	Specify the deployment size. In the logging 5.8 and later versions, the supported size options for production instances of Loki are `1x.extra-small`, `1x.small`, or `1x.medium`.
4	Specify the name of your log store secret.
5	Specify the corresponding storage type.
6	Optional field, logging 5.9 and later. Supported user configured values are as follows: static is the default authentication mode available for all supported object storage types using credentials stored in a Secret. token for short-lived tokens retrieved from a credential source. In this mode the static configuration does not contain credentials needed for the object storage. Instead, they are generated during runtime using a service, which allows for shorter-lived credentials and much more granular control. This authentication mode is not supported for all object storage types. token-cco is the default value when Loki is running on managed STS mode and using CCO on STS/WIF clusters.
7	Specify the name of a storage class for temporary storage. For best performance, specify a storage class that allocates block storage. Available storage classes for your cluster can be listed by using the `oc get storageclasses` command.
8	LokiStack defaults to running in multi-tenant mode, which cannot be modified. One tenant is provided for each log type: audit, infrastructure, and application logs. This enables access control for individual users and user groups to different log streams.

It is not possible to change the number 1x for the deployment size.

Click Create.

Create an OpenShift Logging instance:

Switch to the Administration → Custom Resource Definitions page.
On the Custom Resource Definitions page, click ClusterLogging.
On the Custom Resource Definition details page, select View Instances from the Actions menu.
On the ClusterLoggings page, click Create ClusterLogging.

You might have to refresh the page to load the data.

In the YAML field, replace the code with the following:

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance (1)
  namespace: openshift-logging (2)
spec:
  collection:
    type: vector
  logStore:
    lokistack:
      name: logging-loki
    retentionPolicy:
      application:
        maxAge: 7d
      audit:
        maxAge: 7d
      infra:
        maxAge: 7d
    type: lokistack
  visualization:
    type: ocp-console
    ocpConsole:
      logsLimit: 15
  managementState: Managed

1	Name must be `instance`.
2	Namespace must be `openshift-logging`.

Verification

Go to Operators → Installed Operators.
Make sure the openshift-logging project is selected.
In the Status column, verify that you see green checkmarks with InstallSucceeded and the text Up to date.

An Operator might display a Failed status before the installation finishes. If the Operator install completes with an InstallSucceeded message, refresh the page.

Additional resources

About OVN-Kubernetes network policy