Elasticsearch Operator 4.5.0-202005201915 provided by Red Hat, Inc
After updating the OKD cluster from 4.4 to 4.5, you can then update the Elasticsearch Operator and Cluster Logging Operator from 4.4 to 4.5.
Cluster logging 4.5 introduces a new Elasticsearch version, Elasticsearch 6.8.1 and an enhanced security plug-in, Open Distro for Elasticsearch. The new Elasticsearch version introduces a new Elasticsearch data model that improves overall performance and scalability. With the new data model, data is no longer indexed by type (infrastructure and application) and project. Because of these new features, you need to update your Kibana index patterns and visualizations to match the new indices after updating.
|
Because of the nature of these changes, you are not required to update your cluster logging to 4.5. However, when you update to OKD 4.6, you must update cluster logging to 4.6 at that time. |
After updating the OKD cluster, you can update cluster logging from 4.4 to 4.5 by changing the subscription for the Elasticsearch Operator and the Cluster Logging Operator.
When you update:
You must update the Elasticsearch Operator before updating the Cluster Logging Operator.
You must update both the Elasticsearch Operator and the Cluster Logging Operator.
Kibana is unusable when the Elasticsearch Operator has been updated but the Cluster Logging Operator has not been updated.
If you update the Cluster Logging Operator before the Elasticsearch Operator, Kibana does not update and the Kibana Custom Resource (CR) is not created. To work around this problem, delete the Cluster Logging Operator Pod. When the Cluster Logging Operator Pod redeploys, the Kibana CR is created.
|
If your cluster logging version is prior to 4.4, you must upgrade cluster logging to 4.4 before updating to 4.5. |
Update the OKD cluster from 4.4 to 4.5.
Make sure the cluster logging status is healthy:
All Pods are ready.
The Elasticsearch cluster is healthy.
Back up your Elasticsearch and Kibana data.
Update the Elasticsearch Operator:
From the web console, click Operators → Installed Operators.
Select the openshift-operators-redhat project.
Click the Elasticsearch Operator.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select 4.5 and click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Elasticsearch Operator is shown as 4.5. For example:
Elasticsearch Operator 4.5.0-202005201915 provided by Red Hat, Inc
Wait for the Status field to report Succeeded.
Update the Cluster Logging Operator:
From the web console, click Operators → Installed Operators.
Select the openshift-logging project.
Click the Cluster Logging Operator.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select 4.5 and click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Cluster Logging Operator is shown as 4.5. For example:
Cluster Logging 4.5.0-202005201915 provided by Red Hat, Inc
Wait for the Status field to report Succeeded.
Check the logging components:
Ensure that all Elasticsearch Pods are in the Ready status:
$ oc get pod -n openshift-logging --selector component=elasticsearch NAME READY STATUS RESTARTS AGE elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk 2/2 Running 0 31m elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk 2/2 Running 0 30m elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc 2/2 Running 0 29m
Ensure that the Elasticsearch cluster is healthy:
oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health
{
"cluster_name" : "elasticsearch",
"status" : "green",
}
....
Ensure that the Elasticsearch CronJobs are created:
$ oc get CronJob elasticsearch-*
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE elasticsearch-delete-app */15 * * * * False 0 <none> 27s elasticsearch-delete-audit */15 * * * * False 0 <none> 27s elasticsearch-delete-infra */15 * * * * False 0 <none> 27s elasticsearch-rollover-app */15 * * * * False 0 <none> 27s elasticsearch-rollover-audit */15 * * * * False 0 <none> 27s elasticsearch-rollover-infra */15 * * * * False 0 <none> 27s
Verify that the log store is updated to 4.5 and the indices are green:
oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices
You should see the app-0000x, infra-0000x, audit-0000x, .security indices.
Tue Jun 30 14:30:54 UTC 2020 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open infra-000008 bnBvUFEXTWi92z3zWAzieQ 3 1 222195 0 289 144 green open infra-000004 rtDSzoqsSl6saisSK7Au1Q 3 1 226717 0 297 148 green open infra-000012 RSf_kUwDSR2xEuKRZMPqZQ 3 1 227623 0 295 147 green open .kibana_7 1SJdCqlZTPWlIAaOUd78yg 1 1 4 0 0 0 green open .operations.2020.06.30 aOHMYOa3S_69NJFh2t3yrQ 3 1 4206118 0 8998 4499 green open project.local-storage.d5c8a3d6-30a3-4512-96df-67c537540072.2020.06.30 O_Uldg2wS5K_L6FyqWxOZg 3 1 91052 0 135 67 green open infra-000010 iXwL3bnqTuGEABbUDa6OVw 3 1 248368 0 317 158 green open .searchguard rQhAbWuLQ9iuTsZeHi_2ew 1 1 5 64 0 0 green open infra-000009 YN9EsULWSNaxWeeNvOs0RA 3 1 258799 0 337 168 green open infra-000014 YP0U6R7FQ_GVQVQZ6Yh9Ig 3 1 223788 0 292 146 green open infra-000015 JRBbAbEmSMqK5X40df9HbQ 3 1 224371 0 291 145 green open .orphaned.2020.06.30 n_xQC2dWQzConkvQqei3YA 3 1 9 0 0 0 green open infra-000007 llkkAVSzSOmosWTSAJM_hg 3 1 228584 0 296 148 green open infra-000005 d9BoGQdiQASsS3BBFm2iRA 3 1 227987 0 297 148 green open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac l911Z8dSI23py6GDtyJrA 1 1 5 4 0 0 green open project.ui.29cb9680-864d-43b2-a6cf-134c837d6f0c.2020.06.30 5A_YdRlAT3m1Z-vbqBuGWA 3 1 24 0 0 0 green open infra-000003 1-goREK1QUKlQPAIVkWVaQ 3 1 226719 0 295 147 green open .security zeT65uOuRTKZMjg_bbUc1g 1 1 5 0 0 0 green open .kibana-377444158_kubeadmin wvMhDwJkR-mRZQO84K0gUQ 3 1 1 0 0 0 green open infra-000006 5H-KBSXGQKiO7hdapDE23g 3 1 226676 0 295 147 green open project.nw.6233ad57-aff0-4d5a-976f-370636f47b11.2020.06.30 dtc6J-nLSCC59EygeV41RQ 3 1 10 0 0 0 green open infra-000001 eH53BQ-bSxSWR5xYZB6lVg 3 1 341800 0 443 220 green open .kibana-6 RVp7TemSSemGJcsSUmuf3A 1 1 4 0 0 0 green open infra-000011 J7XWBauWSTe0jnzX02fU6A 3 1 226100 0 293 146 green open app-000001 axSAFfONQDmKwatkjPXdtw 3 1 103186 0 126 57 green open infra-000016 m9c1iRLtStWSF1GopaRyCg 3 1 13685 0 19 9 green open infra-000002 Hz6WvINtTvKcQzw-ewmbYg 3 1 228994 0 296 148 green open project.qt.2c05acbd-bc12-4275-91ab-84d180b53505.2020.06.30 MUm3eFJjSPKQOJWoHskKqw 3 1 12262 0 14 7 green open infra-000013 KR9mMFUpQl-jraYtanyIGw 3 1 228166 0 298 148 green open audit-000001 eERqLdLmQOiQDFES1LBATQ 3 1 0 0 0 0
Verify that the log collector is updated to 4.5:
$ oc get ds fluentd -o json | grep fluentd-init
You should see a fluentd-init container:
"containerName": "fluentd-init"
Verify that the log visualizer is updated to 4.5 using the Kibana CRD:
$ oc get kibana kibana -o json
You should see a Kibana Pod with the ready status:
[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]
Verify the Curator is updated to 4.5:
$ oc get cronjob -o name
cronjob.batch/curator cronjob.batch/elasticsearch-delete-app cronjob.batch/elasticsearch-delete-audit cronjob.batch/elasticsearch-delete-infra cronjob.batch/elasticsearch-rollover-app cronjob.batch/elasticsearch-rollover-audit cronjob.batch/elasticsearch-rollover-infra
You should see the elasticsearch-delete-* and elasticsearch-rollover-* indices.
If you use Kibana, after the Elasticsearch Operator and Cluster Logging Operator are fully updated to 4.5, you must perform the following actions to complete the upgrade:
Recreate your Kibana index patterns. Because of changes in the security plug-in, the cluster logging upgrade does not automatically create index patterns.
Regular users must manually create index patterns to see logs for their projects. Because of the new Elasticsearch data model, all logs that were stored in an index prefixed with project- in OKD 4.4 are now co-located in a single index prefixed with app-. Users should create a new index patterns named app and use the @timestamp time field to view their container logs. To view historical data gathered by cluster logging 4.4, users need to create index patterns using project* field.
Admin users need to create index patterns using the app, infra and audit prefix with the @timestamp time field. The infra- indices were previously .opertions-. The audit- indices are the audit logs. To view historical data gathered by cluster logging 4.4, admin users need to create index patterns using the .operation* field to view system logs and create index patterns using the project* field to view project-level logs.
Recreate your Kibana Visualizations to use the new index patterns.