1. Documentation
×

Curation of Elasticsearch Data

The Elasticsearch Curator tool performs scheduled maintenance operations on a global and/or on a per-project basis. Curator performs actions based on its configuration.

The Cluster Logging Operator installs Curator and its configuration. You can configure the Curator cron schedule using the Cluster Logging Custom Resource and further configuration options can be found in the Curator ConfigMap, curator in the openshift-logging project, which incorporates the Curator configuration file, curator5.yaml and an OKD custom configuration file, config.yaml.

OKD uses the config.yaml internally to generate the Curator action file.

Optionally, you can use the action file, directly. Editing this file allows you to use any action that Curator has available to it to be run periodically. However, this is only recommended for advanced users as modifying the file can be destructive to the cluster and can cause removal of required indices/settings from Elasticsearch. Most users only must modify the Curator configuration map and never edit the action file.

Configuring the Curator schedule

You can specify the schedule for Curator using the cluster logging Custom Resource created by the cluster logging installation.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

To configure the Curator schedule:

  1. Edit the Cluster Logging Custom Resource in the openshift-logging project:

    $ oc edit clusterlogging instance
    apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"

...

  curation:
    curator:
      schedule: 30 3 * * * (1)
    type: curator
    1 Specify the schedule for Curator in cron format.

    The time zone is set based on the host node where the Curator pod runs.

Configuring Curator index deletion

You can configure Curator to delete Elasticsearch data based on retention settings. You can configure per-project and global settings. Global settings apply to any project not specified. Per-project settings override global settings.

Prerequisite

  • Cluster logging must be installed.

Procedure

To delete indices:

  1. Edit the OKD custom Curator configuration file:

    $ oc edit configmap/curator

  2. Set the following parameters as needed:

    config.yaml: |
  project_name:
    action
      unit:value

    The available parameters are:

    Table 1. Project options
    Variable Name Description

    project_name

    The actual name of a project, such as myapp-devel. For OKD operations logs, use the name .operations as the project name.

    action

    The action to take, currently only delete is allowed.

    unit

    The period to use for deletion, days, weeks, or months.

    value

    The number of units.
    Table 2. Filter options
    Variable Name Description

    .defaults

    Use .defaults as the project_name to set the defaults for projects that are not specified.

    .regex

    The list of regular expressions that match project names.

    pattern

    The valid and properly escaped regular expression pattern enclosed by single quotation marks.

For example, to configure Curator to:

  • Delete indices in the myapp-dev project older than 1 day

  • Delete indices in the myapp-qe project older than 1 week

  • Delete operations logs older than 8 weeks

  • Delete all other projects indices after they are 31 days old

  • Delete indices older than 1 day that are matched by the ^project\..+\-dev.*$ regex

  • Delete indices older than 2 days that are matched by the ^project\..+\-test.*$ regex

Use:

  config.yaml: |
    .defaults:
      delete:
        days: 31

    .operations:
      delete:
        weeks: 8

    myapp-dev:
      delete:
        days: 1

    myapp-qe:
      delete:
        weeks: 1

    .regex:
      - pattern: '^project\..+\-dev\..*$'
        delete:
          days: 1
      - pattern: '^project\..+\-test\..*$'
        delete:
          days: 2

When you use months as the $UNIT for an operation, Curator starts counting at the first day of the current month, not the current day of the current month. For example, if today is April 15, and you want to delete indices that are 2 months older than today (delete: months: 2), Curator does not delete indices that are dated older than February 15; it deletes indices older than February 1. That is, it goes back to the first day of the current month, then goes back two whole months from that date. If you want to be exact with Curator, it is best to use days (for example, delete: days: 30).

Troubleshooting Curator

You can use information in this section for debugging Curator. For example, if curator is in failed state, but the log messages do not provide a reason, you could increase the log level and trigger a new job, instead of waiting for another scheduled run of the cron job.

Prerequisites

Cluster logging and Elasticsearch must be installed.

Procedure

Enable the Curator debug log and trigger next Curator iteration manually

  1. Enable debug log of Curator:

    $ oc set env cronjob/curator CURATOR_LOG_LEVEL=DEBUG CURATOR_SCRIPT_LOG_LEVEL=DEBUG

    Specify the log level:

    • CRITICAL. Curator displays only critical messages.

    • ERROR. Curator displays only error and critical messages.

    • WARNING. Curator displays only error, warning, and critical messages.

    • INFO. Curator displays only informational, error, warning, and critical messages.

    • DEBUG. Curator displays only debug messages, in addition to all of the above.

      The default value is INFO.

Cluster logging uses the OKD custom environment variable CURATOR_SCRIPT_LOG_LEVEL in OKD wrapper scripts (run.sh and convert.py). The environment variable takes the same values as CURATOR_LOG_LEVEL for script debugging, as needed.

  1. Trigger next curator iteration:

    $ oc create job --from=cronjob/curator <job_name>

  2. Use the following commands to control the CronJob:

    • Suspend a CronJob:

      $ oc patch cronjob curator -p '{"spec":{"suspend":true}}'

    • Resume a CronJob:

      $ oc patch cronjob curator -p '{"spec":{"suspend":false}}'

    • Change a CronJob schedule:

      $ oc patch cronjob curator -p '{"spec":{"schedule":"0 0 * * *"}}' (1)
      1 The schedule option accepts schedules in cron format.

Configuring Curator in scripted deployments

Use the information in this section if you must configure Curator in scripted deployments.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

  • Set cluster logging to the unmanaged state.

Procedure

Use the following snippets to configure Curator in your scripts:

  • For scripted deployments

    1. Create and modify the configuration:

      1. Copy the Curator configuration file and the OKD custom configuration file from the Curator configuration map and create separate files for each:

        $ oc extract configmap/curator --keys=curator5.yaml,config.yaml --to=/my/config

      2. Edit the /my/config/curator5.yaml and /my/config/config.yaml files.

    2. Delete the existing Curator config map and add the edited YAML files to a new Curator config map.

      $ oc delete configmap curator ; sleep 1
$ oc create configmap curator \
    --from-file=curator5.yaml=/my/config/curator5.yaml \
    --from-file=config.yaml=/my/config/config.yaml \
    ; sleep 1

      The next iteration will use this configuration.

  • If you are using the action file:

    1. Create and modify the configuration:

      1. Copy the Curator configuration file and the action file from the Curator configuration map and create separate files for each:

        $ oc extract configmap/curator --keys=curator5.yaml,actions.yaml --to=/my/config

      2. Edit the /my/config/curator5.yaml and /my/config/actions.yaml files.

    2. Delete the existing Curator config map and add the edited YAML files to a new Curator config map.

      $ oc delete configmap curator ; sleep 1
$ oc create configmap curator \
    --from-file=curator5.yaml=/my/config/curator5.yaml \
    --from-file=actions.yaml=/my/config/actions.yaml \
    ; sleep 1

      The next iteration will use this configuration.

Using the Curator Action file

The Curator ConfigMap in the openshift-logging project includes a Curator action file where you configure any Curator action to be run periodically.

However, when you use the action file, OKD ignores the config.yaml section of the curator ConfigMap, which is configured to ensure important internal indices do not get deleted by mistake. In order to use the action file, you should add an exclude rule to your configuration to retain these indices. You also must manually add all the other patterns following the steps in this topic.

The actions and config.yaml are mutually-exclusive configuration files. Once the actions file exist, OKD ignores the config.yaml file. Using the action file is recommended only for advanced users as using this file can be destructive to the cluster and can cause removal of required indices/settings from Elasticsearch.
Prerequisite

  • Cluster logging and Elasticsearch must be installed.

  • Set cluster logging to the unmanaged state.

Procedure

To configure Curator to delete indices:

  1. Edit the Curator ConfigMap:

    oc edit cm/curator -n openshift-logging

  2. Make the following changes to the action file:

    actions:
1:
      action: delete_indices (1)
      description: >-
        Delete .operations indices older than 30 days.
        Ignore the error if the filter does not
        result in an actionable list of indices (ignore_empty_list).
        See https://www.elastic.co/guide/en/elasticsearch/client/curator/5.2/ex_delete_indices.html
      options:
        # Swallow curator.exception.NoIndices exception
        ignore_empty_list: True
        # In seconds, default is 300
        timeout_override: ${CURATOR_TIMEOUT}
        # Don't swallow any other exceptions
        continue_if_exception: False
        # Optionally disable action, useful for debugging
        disable_action: False
      # All filters are bound by logical AND
      filters:            (2)
      - filtertype: pattern
        kind: regex
        value: '^\.operations\..*$'
        exclude: False    (3)
      - filtertype: age
        # Parse timestamp from index name
        source: name
        direction: older
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30
        exclude: False
    1 Specify delete_indices to delete the specified index.
    2 Use the filers parameters to specify the index to be deleted. See the Elastic Search curator documentation for information on these parameters.
    3 Specify false to allow the index to be deleted.