Elasticsearch Operator
4.6.0-202007012112.p0 provided
by Red Hat, Inc
After updating the OKD cluster from 4.4 to 4.5, you can then update the Elasticsearch Operator and Cluster Logging Operator from 4.4 to 4.5.
Cluster logging 4.5 introduces a new Elasticsearch version, Elasticsearch 6.8.1, and an enhanced security plug-in, Open Distro for Elasticsearch. The new Elasticsearch version introduces a new Elasticsearch data model, where the Elasticsearch data is indexed only by type: infrastructure, application, and audit. Previously, data was indexed by type (infrastructure and application) and project.
|
Because of the new data model, the update does not migrate existing custom Kibana index patterns and visualizations into the new version. You must re-create your Kibana index patterns and visualizations to match the new indices after updating. |
Due to the nature of these changes, you are not required to update your cluster logging to 4.5. However, when you update to OKD 4.6, you must update cluster logging to 4.6 at that time.
After updating the OKD cluster, you can update cluster logging from 4.5 to 4.6 by changing the subscription for the Elasticsearch Operator and the Cluster Logging Operator.
When you update:
You must update the Elasticsearch Operator before updating the Cluster Logging Operator.
You must update both the Elasticsearch Operator and the Cluster Logging Operator.
Kibana is unusable when the Elasticsearch Operator has been updated but the Cluster Logging Operator has not been updated.
If you update the Cluster Logging Operator before the Elasticsearch Operator, Kibana does not update and the Kibana Custom Resource (CR) is not created. To work around this problem, delete the Cluster Logging Operator pod. When the Cluster Logging Operator pod redeploys, the Kibana CR is created.
|
If your cluster logging version is prior to 4.5, you must upgrade cluster logging to 4.5 before updating to 4.6. |
Update the OKD cluster from 4.5 to 4.6.
Make sure the cluster logging status is healthy:
All pods are ready.
The Elasticsearch cluster is healthy.
Back up your Elasticsearch and Kibana data.
Update the Elasticsearch Operator:
From the web console, click Operators → Installed Operators.
Select the openshift-operators-redhat project.
Click the Elasticsearch Operator.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select 4.6 and click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Elasticsearch Operator is shown as 4.6. For example:
Elasticsearch Operator
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Update the Cluster Logging Operator:
From the web console, click Operators → Installed Operators.
Select the openshift-logging project.
Click the Cluster Logging Operator.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select 4.6 and click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Cluster Logging Operator is shown as 4.6. For example:
Cluster Logging
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Check the logging components:
Ensure that all Elasticsearch pods are in the Ready status:
$ oc get pod -n openshift-logging --selector component=elasticsearchNAME READY STATUS RESTARTS AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk 2/2 Running 0 31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk 2/2 Running 0 30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc 2/2 Running 0 29mEnsure that the Elasticsearch cluster is healthy:
$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health{
"cluster_name" : "elasticsearch",
"status" : "green",
}
....
Ensure that the Elasticsearch CronJobs are created:
$ oc project openshift-logging$ oc get CronJob elasticsearch-*NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
elasticsearch-delete-app */15 * * * * False 0 <none> 27s
elasticsearch-delete-audit */15 * * * * False 0 <none> 27s
elasticsearch-delete-infra */15 * * * * False 0 <none> 27s
elasticsearch-rollover-app */15 * * * * False 0 <none> 27s
elasticsearch-rollover-audit */15 * * * * False 0 <none> 27s
elasticsearch-rollover-infra */15 * * * * False 0 <none> 27sVerify that the log store is updated to 4.6 and the indices are green:
$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indicesYou should see the app-0000x, infra-0000x, audit-0000x, .security indices.
Tue Jun 30 14:30:54 UTC 2020
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open infra-000008 bnBvUFEXTWi92z3zWAzieQ 3 1 222195 0 289 144
green open infra-000004 rtDSzoqsSl6saisSK7Au1Q 3 1 226717 0 297 148
green open infra-000012 RSf_kUwDSR2xEuKRZMPqZQ 3 1 227623 0 295 147
green open .kibana_7 1SJdCqlZTPWlIAaOUd78yg 1 1 4 0 0 0
green open .operations.2020.06.30 aOHMYOa3S_69NJFh2t3yrQ 3 1 4206118 0 8998 4499
green open project.local-storage.d5c8a3d6-30a3-4512-96df-67c537540072.2020.06.30 O_Uldg2wS5K_L6FyqWxOZg 3 1 91052 0 135 67
green open infra-000010 iXwL3bnqTuGEABbUDa6OVw 3 1 248368 0 317 158
green open .searchguard rQhAbWuLQ9iuTsZeHi_2ew 1 1 5 64 0 0
green open infra-000009 YN9EsULWSNaxWeeNvOs0RA 3 1 258799 0 337 168
green open infra-000014 YP0U6R7FQ_GVQVQZ6Yh9Ig 3 1 223788 0 292 146
green open infra-000015 JRBbAbEmSMqK5X40df9HbQ 3 1 224371 0 291 145
green open .orphaned.2020.06.30 n_xQC2dWQzConkvQqei3YA 3 1 9 0 0 0
green open infra-000007 llkkAVSzSOmosWTSAJM_hg 3 1 228584 0 296 148
green open infra-000005 d9BoGQdiQASsS3BBFm2iRA 3 1 227987 0 297 148
green open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac l911Z8dSI23py6GDtyJrA 1 1 5 4 0 0
green open project.ui.29cb9680-864d-43b2-a6cf-134c837d6f0c.2020.06.30 5A_YdRlAT3m1Z-vbqBuGWA 3 1 24 0 0 0
green open infra-000003 1-goREK1QUKlQPAIVkWVaQ 3 1 226719 0 295 147
green open .security zeT65uOuRTKZMjg_bbUc1g 1 1 5 0 0 0
green open .kibana-377444158_kubeadmin wvMhDwJkR-mRZQO84K0gUQ 3 1 1 0 0 0
green open infra-000006 5H-KBSXGQKiO7hdapDE23g 3 1 226676 0 295 147
green open project.nw.6233ad57-aff0-4d5a-976f-370636f47b11.2020.06.30 dtc6J-nLSCC59EygeV41RQ 3 1 10 0 0 0
green open infra-000001 eH53BQ-bSxSWR5xYZB6lVg 3 1 341800 0 443 220
green open .kibana-6 RVp7TemSSemGJcsSUmuf3A 1 1 4 0 0 0
green open infra-000011 J7XWBauWSTe0jnzX02fU6A 3 1 226100 0 293 146
green open app-000001 axSAFfONQDmKwatkjPXdtw 3 1 103186 0 126 57
green open infra-000016 m9c1iRLtStWSF1GopaRyCg 3 1 13685 0 19 9
green open infra-000002 Hz6WvINtTvKcQzw-ewmbYg 3 1 228994 0 296 148
green open project.qt.2c05acbd-bc12-4275-91ab-84d180b53505.2020.06.30 MUm3eFJjSPKQOJWoHskKqw 3 1 12262 0 14 7
green open infra-000013 KR9mMFUpQl-jraYtanyIGw 3 1 228166 0 298 148
green open audit-000001 eERqLdLmQOiQDFES1LBATQ 3 1 0 0 0 0Verify that the log collector is updated to 4.6:
$ oc get ds fluentd -o json | grep fluentd-initYou should see a fluentd-init container:
"containerName": "fluentd-init"Verify that the log visualizer is updated to 4.6 using the Kibana CRD:
$ oc get kibana kibana -o jsonYou should see a Kibana pod with the ready status:
[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]Verify the Curator is updated to 4.6:
$ oc get cronjob -o namecronjob.batch/curator
cronjob.batch/elasticsearch-delete-app
cronjob.batch/elasticsearch-delete-audit
cronjob.batch/elasticsearch-delete-infra
cronjob.batch/elasticsearch-rollover-app
cronjob.batch/elasticsearch-rollover-audit
cronjob.batch/elasticsearch-rollover-infraYou should see the elasticsearch-delete-* and elasticsearch-rollover-* indices.
If you use the Log Forwarding API to forward logs, after the Elasticsearch Operator and Cluster Logging Operator are fully updated to 4.6, you must replace your LogForwarding custom resource (CR) with a ClusterLogForwarder CR.
The OKD Log Forward API has been promoted from Technology Preview to Generally Available in OKD 4.6. The GA release contains some improvements and enhancements that require you to make a change to your Cluster Logging custom resource (CR) and to replace your LogForwarding custom resource (CR) with a ClusterLogForwarder CR.
ClusterLogForwarder instance in OKD 4.6apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
....
spec:
outputs:
- url: http://remote.elasticsearch.com:9200
name: elasticsearch
type: elasticsearch
- url: tls://fluentdserver.example.com:24224
name: fluentd
type: fluentdForward
secret:
name: fluentdserver
pipelines:
- inputRefs:
- infrastructure
- application
name: mylogs
outputRefs:
- elasticsearch
- inputRefs:
- audit
name: auditlogs
outputRefs:
- fluentd
- default
...LogForwarding CR in OKD 4.5apiVersion: logging.openshift.io/v1alpha1
kind: LogForwarding
metadata:
name: instance
namespace: openshift-logging
spec:
disableDefaultForwarding: true
outputs:
- name: elasticsearch
type: elasticsearch
endpoint: remote.elasticsearch.com:9200
- name: fluentd
type: forward
endpoint: fluentdserver.example.com:24224
secret:
name: fluentdserver
pipelines:
- inputSource: logs.infra
name: infra-logs
outputRefs:
- elasticearch
- inputSource: logs.app
name: app-logs
outputRefs:
- elasticearch
- inputSource: logs.audit
name: audit-logs
outputRefs:
- fluentdThe following procedure shows each parameter you must change.
To update the LogForwarding CR in 4.5 to the ClusterLogForwarding CR for 4.6, make the following modifications:
Edit the Cluster Logging custom resource (CR) to remove the logforwardingtechpreview annotation:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
annotations:
clusterlogging.openshift.io/logforwardingtechpreview: enabled (1)
name: "instance"
namespace: "openshift-logging"
....| 1 | Remove the logforwardingtechpreview annotation. |
Export the LogForwarding CR to create a YAML file for the ClusterLogForwarder instance:
$ oc get LogForwarding instance -n openshift-logging -o yaml| tee ClusterLogForwarder.yamlEdit the YAML file to make the following modifications:
ClusterLogForwarder instance in OKD 4.6apiVersion: logging.openshift.io/v1 (1)
kind: ClusterLogForwarder (2)
metadata:
name: instance
namespace: openshift-logging
....
spec: (3)
outputs:
- url: http://remote.elasticsearch.com:9200 (4)
name: elasticsearch
type: elasticsearch
- url: tls://fluentdserver.example.com:24224
name: fluentd
type: fluentdForward (5)
secret:
name: fluentdserver
pipelines:
- inputRefs: (6)
- infrastructure
- application
name: mylogs
outputRefs:
- elasticsearch
- inputRefs:
- audit
name: auditlogs
outputRefs:
- fluentd
- default (7)
...| 1 | Change the apiVersion from "logging.openshift.io/v1alpha1" to "logging.openshift.io/v1". |
||
| 2 | Change the object kind from kind: "LogForwarding" to kind: "ClusterLogForwarder". |
||
| 3 | Remove the disableDefaultForwarding: true parameter. |
||
| 4 | Change the output parameter from spec.outputs.endpoint to spec.outputs.url. Add a prefix to the URL, such as https://, tcp://, and so forth, if a prefix is not present. |
||
| 5 | For Fluentd outputs, change the type from forward to fluentdForward. |
||
| 6 | Change the pipelines:
|
||
| 7 | Optional: Add a default pipeline to send logs to the internal Elasticsearch instance. You are not required to configure a default output.
|
Create the CR object:
$ oc create -f ClusterLogForwarder.yamlFor information on the new capabilities of the Log Forwarding API, see Forwarding logs to third party systems.