1. Documentation
×

Managing alerts

In OKD 4, the Alerting UI enables you to manage alerts, silences, and alerting rules.

  • Alerting rules. Alerting rules contain a set of conditions that outline a particular state within a cluster. Alerts are triggered when those conditions are true. An alerting rule can be assigned a severity that defines how the alerts are routed.

  • Alerts. An alert is fired when the conditions defined in an alerting rule are true. Alerts provide a notification that a set of circumstances are apparent within an OKD cluster.

  • Silences. A silence can be applied to an alert to prevent notifications from being sent when the conditions for an alert are true. You can mute an alert after the initial notification, while you work on resolving the issue.

The alerts, silences, and alerting rules that are available in the Alerting UI relate to the projects that you have access to. For example, if you are logged in as a user with the cluster-admin role, you can access all alerts, silences, and alerting rules.

Accessing the Alerting UI in the Administrator and Developer perspectives

The Alerting UI is accessible through the Administrator perspective and the Developer perspective of the OKD web console.

  • In the Administrator perspective, go to Observe → Alerting. The three main pages in the Alerting UI in this perspective are the Alerts, Silences, and Alerting rules pages.

  • In the Developer perspective, go to Observe → <project_name> → Alerts. In this perspective, alerts, silences, and alerting rules are all managed from the Alerts page. The results shown in the Alerts page are specific to the selected project.

In the Developer perspective, you can select from core OKD and user-defined projects that you have access to in the Project: <project_name> list. However, alerts, silences, and alerting rules relating to core OKD projects are not displayed if you are not logged in as a cluster administrator.

Searching and filtering alerts, silences, and alerting rules

You can filter the alerts, silences, and alerting rules that are displayed in the Alerting UI. This section provides a description of each of the available filtering options.

Understanding alert filters

In the Administrator perspective, the Alerts page in the Alerting UI provides details about alerts relating to default OKD and user-defined projects. The page includes a summary of severity, state, and source for each alert. The time at which an alert went into its current state is also shown.

You can filter by alert state, severity, and source. By default, only Platform alerts that are Firing are displayed. The following describes each alert filtering option:

  • State filters:

    • Firing. The alert is firing because the alert condition is true and the optional for duration has passed. The alert continues to fire while the condition remains true.

    • Pending. The alert is active but is waiting for the duration that is specified in the alerting rule before it fires.

    • Silenced. The alert is now silenced for a defined time period. Silences temporarily mute alerts based on a set of label selectors that you define. Notifications are not sent for alerts that match all the listed values or regular expressions.

  • Severity filters:

    • Critical. The condition that triggered the alert could have a critical impact. The alert requires immediate attention when fired and is typically paged to an individual or to a critical response team.

    • Warning. The alert provides a warning notification about something that might require attention to prevent a problem from occurring. Warnings are typically routed to a ticketing system for non-immediate review.

    • Info. The alert is provided for informational purposes only.

    • None. The alert has no defined severity.

    • You can also create custom severity definitions for alerts relating to user-defined projects.

  • Source filters:

    • Platform. Platform-level alerts relate only to default OKD projects. These projects provide core OKD functionality.

    • User. User alerts relate to user-defined projects. These alerts are user-created and are customizable. User-defined workload monitoring can be enabled postinstallation to provide observability into your own workloads.

Understanding silence filters

In the Administrator perspective, the Silences page in the Alerting UI provides details about silences applied to alerts in default OKD and user-defined projects. The page includes a summary of the state of each silence and the time at which a silence ends.

You can filter by silence state. By default, only Active and Pending silences are displayed. The following describes each silence state filter option:

  • State filters:

    • Active. The silence is active and the alert will be muted until the silence is expired.

    • Pending. The silence has been scheduled and it is not yet active.

    • Expired. The silence has expired and notifications will be sent if the conditions for an alert are true.

Understanding alerting rule filters

In the Administrator perspective, the Alerting rules page in the Alerting UI provides details about alerting rules relating to default OKD and user-defined projects. The page includes a summary of the state, severity, and source for each alerting rule.

You can filter alerting rules by alert state, severity, and source. By default, only Platform alerting rules are displayed. The following describes each alerting rule filtering option:

  • Alert state filters:

    • Firing. The alert is firing because the alert condition is true and the optional for duration has passed. The alert continues to fire while the condition remains true.

    • Pending. The alert is active but is waiting for the duration that is specified in the alerting rule before it fires.

    • Silenced. The alert is now silenced for a defined time period. Silences temporarily mute alerts based on a set of label selectors that you define. Notifications are not sent for alerts that match all the listed values or regular expressions.

    • Not Firing. The alert is not firing.

  • Severity filters:

    • Critical. The conditions defined in the alerting rule could have a critical impact. When true, these conditions require immediate attention. Alerts relating to the rule are typically paged to an individual or to a critical response team.

    • Warning. The conditions defined in the alerting rule might require attention to prevent a problem from occurring. Alerts relating to the rule are typically routed to a ticketing system for non-immediate review.

    • Info. The alerting rule provides informational alerts only.

    • None. The alerting rule has no defined severity.

    • You can also create custom severity definitions for alerting rules relating to user-defined projects.

  • Source filters:

    • Platform. Platform-level alerting rules relate only to default OKD projects. These projects provide core OKD functionality.

    • User. User-defined workload alerting rules relate to user-defined projects. These alerting rules are user-created and are customizable. User-defined workload monitoring can be enabled postinstallation to provide observability into your own workloads.

Searching and filtering alerts, silences, and alerting rules in the Developer perspective

In the Developer perspective, the Alerts page in the Alerting UI provides a combined view of alerts and silences relating to the selected project. A link to the governing alerting rule is provided for each displayed alert.

In this view, you can filter by alert state and severity. By default, all alerts in the selected project are displayed if you have permission to access the project. These filters are the same as those described for the Administrator perspective.

Getting information about alerts, silences, and alerting rules

The Alerting UI provides detailed information about alerts and their governing alerting rules and silences.

Prerequisites

  • You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing alerts for.

Procedure

To obtain information about alerts in the Administrator perspective:

  1. Open the OKD web console and go to the Observe → Alerting → Alerts page.

  2. Optional: Search for alerts by name by using the Name field in the search list.

  3. Optional: Filter alerts by state, severity, and source by selecting filters in the Filter list.

  4. Optional: Sort the alerts by clicking one or more of the Name, Severity, State, and Source column headers.

  5. Click the name of an alert to view its Alert details page. The page includes a graph that illustrates alert time series data. It also provides the following information about the alert:

    • A description of the alert

    • Messages associated with the alert

    • Labels attached to the alert

    • A link to its governing alerting rule

    • Silences for the alert, if any exist

To obtain information about silences in the Administrator perspective:

  1. Go to the Observe → Alerting → Silences page.

  2. Optional: Filter the silences by name using the Search by name field.

  3. Optional: Filter silences by state by selecting filters in the Filter list. By default, Active and Pending filters are applied.

  4. Optional: Sort the silences by clicking one or more of the Name, Firing alerts, State, and Creator column headers.

  5. Select the name of a silence to view its Silence details page. The page includes the following details:

    • Alert specification

    • Start time

    • End time

    • Silence state

    • Number and list of firing alerts

To obtain information about alerting rules in the Administrator perspective:

  1. Go to the Observe → Alerting → Alerting rules page.

  2. Optional: Filter alerting rules by state, severity, and source by selecting filters in the Filter list.

  3. Optional: Sort the alerting rules by clicking one or more of the Name, Severity, Alert state, and Source column headers.

  4. Select the name of an alerting rule to view its Alerting rule details page. The page provides the following details about the alerting rule:

    • Alerting rule name, severity, and description.

    • The expression that defines the condition for firing the alert.

    • The time for which the condition should be true for an alert to fire.

    • A graph for each alert governed by the alerting rule, showing the value with which the alert is firing.

    • A table of all alerts governed by the alerting rule.

To obtain information about alerts, silences, and alerting rules in the Developer perspective:

  1. Go to the Observe → <project_name> → Alerts page.

  2. View details for an alert, silence, or an alerting rule:

    • Alert details can be viewed by clicking a greater than symbol (>) next to an alert name and then selecting the alert from the list.

    • Silence details can be viewed by clicking a silence in the Silenced by section of the Alert details page. The Silence details page includes the following information:

      • Alert specification

      • Start time

      • End time

      • Silence state

      • Number and list of firing alerts

    • Alerting rule details can be viewed by clicking the kebab menu next to an alert in the Alerts page and then clicking View Alerting Rule.

Only alerts, silences, and alerting rules relating to the selected project are displayed in the Developer perspective.
Additional resources

Managing silences

You can create a silence for an alert in the OKD web console in both the Administrator and Developer perspectives. After you create a silence, you will not receive notifications about an alert when the alert fires.

Creating silences is useful in scenarios where you have received an initial alert notification, and you do not want to receive further notifications during the time in which you resolve the underlying issue causing the alert to fire.

When creating a silence, you must specify whether it becomes active immediately or at a later time. You must also set a duration period after which the silence expires.

After you create silences, you can view, edit, and expire them.

When you create silences, they are replicated across Alertmanager pods. However, if you do not configure persistent storage for Alertmanager, silences might be lost. This can happen, for example, if all Alertmanager pods restart at the same time.
Additional resources

Silencing alerts

You can silence a specific alert or silence alerts that match a specification that you define.

Prerequisites

  • If you are a cluster administrator, you have access to the cluster as a user with the cluster-admin role.

  • If you are a non-administrator user, you have access to the cluster as a user with the following user roles:

    • The cluster-monitoring-view cluster role, which allows you to access Alertmanager.

    • The monitoring-alertmanager-edit role, which permits you to create and silence alerts in the Administrator perspective in the web console.

    • The monitoring-rules-edit cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.

Procedure

To silence a specific alert in the Administrator perspective:

  1. Go to Observe → Alerting → Alerts in the OKD web console.

  2. For the alert that you want to silence, click kebab and select Silence alert to open the Silence alert page with a default configuration for the chosen alert.

  3. Optional: Change the default configuration details for the silence.

    You must add a comment before saving a silence.

  4. To save the silence, click Silence.

To silence a specific alert in the Developer perspective:

  1. Go to Observe → <project_name> → Alerts in the OKD web console.

  2. If necessary, expand the details for the alert by selecting a greater than symbol (>) next to the alert name.

  3. Click the alert message in the expanded view to open the Alert details page for the alert.

  4. Click Silence alert to open the Silence alert page with a default configuration for the alert.

  5. Optional: Change the default configuration details for the silence.

    You must add a comment before saving a silence.

  6. To save the silence, click Silence.

To silence a set of alerts by creating a silence configuration in the Administrator perspective:

  1. Go to Observe → Alerting → Silences in the OKD web console.

  2. Click Create silence.

  3. On the Create silence page, set the schedule, duration, and label details for an alert.

    You must add a comment before saving a silence.

  4. To create silences for alerts that match the labels that you entered, click Silence.

To silence a set of alerts by creating a silence configuration in the Developer perspective:

  1. Go to Observe → <project_name> → Silences in the OKD web console.

  2. Click Create silence.

  3. On the Create silence page, set the duration and label details for an alert.

    You must add a comment before saving a silence.

  4. To create silences for alerts that match the labels that you entered, click Silence.

Editing silences

You can edit a silence, which expires the existing silence and creates a new one with the changed configuration.

Prerequisites

  • If you are a cluster administrator, you have access to the cluster as a user with the cluster-admin role.

  • If you are a non-administrator user, you have access to the cluster as a user with the following user roles:

    • The cluster-monitoring-view cluster role, which allows you to access Alertmanager.

    • The monitoring-alertmanager-edit role, which permits you to create and silence alerts in the Administrator perspective in the web console.

    • The monitoring-rules-edit cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.

Procedure

To edit a silence in the Administrator perspective:

  1. Go to Observe → Alerting → Silences.

  2. For the silence you want to modify, click kebab and select Edit silence.

    Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

  3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

To edit a silence in the Developer perspective:

  1. Go to Observe → <project_name> → Silences.

  2. For the silence you want to modify, click kebab and select Edit silence.

    Alternatively, you can click Actions and select Edit silence on the Silence details page for a silence.

  3. On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Expiring silences

You can expire a single silence or multiple silences. Expiring a silence deactivates it permanently.

You cannot delete expired, silenced alerts. Expired silences older than 120 hours are garbage collected.
Prerequisites

  • If you are a cluster administrator, you have access to the cluster as a user with the cluster-admin role.

  • If you are a non-administrator user, you have access to the cluster as a user with the following user roles:

    • The cluster-monitoring-view cluster role, which allows you to access Alertmanager.

    • The monitoring-alertmanager-edit role, which permits you to create and silence alerts in the Administrator perspective in the web console.

    • The monitoring-rules-edit cluster role, which permits you to create and silence alerts in the Developer perspective in the web console.

Procedure

To expire a silence or silences in the Administrator perspective:

  1. Go to Observe → Alerting → Silences.

  2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

  3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

    Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

To expire a silence in the Developer perspective:

  1. Go to Observe → <project_name> → Silences.

  2. For the silence or silences you want to expire, select the checkbox in the corresponding row.

  3. Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

    Alternatively, to expire a single silence you can click Actions and select Expire silence on the Silence details page for a silence.

Managing alerting rules for core platform monitoring

OKD 4 monitoring ships with a large set of default alerting rules for platform metrics. As a cluster administrator, you can customize this set of rules in two ways:

  • Modify the settings for existing platform alerting rules by adjusting thresholds or by adding and modifying labels. For example, you can change the severity label for an alert from warning to critical to help you route and triage issues flagged by an alert.

  • Define and add new custom alerting rules by constructing a query expression based on core platform metrics in the openshift-monitoring namespace.

Core platform alerting rule considerations

  • New alerting rules must be based on the default OKD monitoring metrics.

  • You must create the AlertingRule and AlertRelabelConfig objects in the openshift-monitoring namespace.

  • You can only add and modify alerting rules. You cannot create new recording rules or modify existing recording rules.

  • If you modify existing platform alerting rules by using an AlertRelabelConfig object, your modifications are not reflected in the Prometheus alerts API. Therefore, any dropped alerts still appear in the OKD web console even though they are no longer forwarded to Alertmanager. Additionally, any modifications to alerts, such as a changed severity label, do not appear in the web console.

Tips for optimizing alerting rules for core platform monitoring

If you customize core platform alerting rules to meet your organization’s specific needs, follow these guidelines to help ensure that the customized rules are efficient and effective.

  • Minimize the number of new rules. Create only rules that are essential to your specific requirements. By minimizing the number of rules, you create a more manageable and focused alerting system in your monitoring environment.

  • Focus on symptoms rather than causes. Create rules that notify users of symptoms instead of underlying causes. This approach ensures that users are promptly notified of a relevant symptom so that they can investigate the root cause after an alert has triggered. This tactic also significantly reduces the overall number of rules you need to create.

  • Plan and assess your needs before implementing changes. First, decide what symptoms are important and what actions you want users to take if these symptoms occur. Then, assess existing rules and decide if you can modify any of them to meet your needs instead of creating entirely new rules for each symptom. By modifying existing rules and creating new ones judiciously, you help to streamline your alerting system.

  • Provide clear alert messaging. When you create alert messages, describe the symptom, possible causes, and recommended actions. Include unambiguous, concise explanations along with troubleshooting steps or links to more information. Doing so helps users quickly assess the situation and respond appropriately.

  • Include severity levels. Assign severity levels to your rules to indicate how a user needs to react when a symptom occurs and triggers an alert. For example, classifying an alert as Critical signals that an individual or a critical response team needs to respond immediately. By defining severity levels, you help users know how to respond to an alert and help ensure that the most urgent issues receive prompt attention.

Creating new alerting rules

As a cluster administrator, you can create new alerting rules based on platform metrics. These alerting rules trigger alerts based on the values of chosen metrics.

  • If you create a customized AlertingRule resource based on an existing platform alerting rule, silence the original alert to avoid receiving conflicting alerts.

  • To help users understand the impact and cause of the alert, ensure that your alerting rule contains an alert message and severity value.
Prerequisites

  • You have access to the cluster as a user that has the cluster-admin cluster role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a new YAML configuration file named example-alerting-rule.yaml.

  2. Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

    apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: example
  namespace: openshift-monitoring (1)
spec:
  groups:
  - name: example-rules
    rules:
    - alert: ExampleAlert (2)
      for: 1m (3)
      expr: vector(1) (4)
      labels:
        severity: warning (5)
      annotations:
        message: This is an example alert. (6)
    1 Ensure that the namespace is openshift-monitoring.
    2 The name of the alerting rule you want to create.
    3 The duration for which the condition should be true before an alert is fired.
    4 The PromQL query expression that defines the new rule.
    5 The severity that alerting rule assigns to the alert.
    6 The message associated with the alert.

    You must create the AlertingRule object in the openshift-monitoring namespace. Otherwise, the alerting rule is not accepted.

  3. Apply the configuration file to the cluster:

    $ oc apply -f example-alerting-rule.yaml

Modifying core platform alerting rules

As a cluster administrator, you can modify core platform alerts before Alertmanager routes them to a receiver. For example, you can change the severity label of an alert, add a custom label, or exclude an alert from being sent to Alertmanager.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a new YAML configuration file named example-modified-alerting-rule.yaml.

  2. Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

    apiVersion: monitoring.openshift.io/v1
kind: AlertRelabelConfig
metadata:
  name: watchdog
  namespace: openshift-monitoring (1)
spec:
  configs:
  - sourceLabels: [alertname,severity] (2)
    regex: "Watchdog;none" (3)
    targetLabel: severity (4)
    replacement: critical (5)
    action: Replace (6)
    1 Ensure that the namespace is openshift-monitoring.
    2 The source labels for the values you want to modify.
    3 The regular expression against which the value of sourceLabels is matched.
    4 The target label of the value you want to modify.
    5 The new value to replace the target label.
    6 The relabel action that replaces the old value based on regex matching. The default action is Replace. Other possible values are Keep, Drop, HashMod, LabelMap, LabelDrop, and LabelKeep.

    You must create the AlertRelabelConfig object in the openshift-monitoring namespace. Otherwise, the alert label will not change.

  3. Apply the configuration file to the cluster:

    $ oc apply -f example-modified-alerting-rule.yaml
Additional resources

Creating alerting rules for user-defined projects

In OKD, you can create alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics.

If you create alerting rules for a user-defined project, consider the following key behaviors and important limitations when you define the new rules:

  • A user-defined alerting rule can include metrics exposed by its own project in addition to the default metrics from core platform monitoring. You cannot include metrics from another user-defined project.

    For example, an alerting rule for the ns1 user-defined project can use metrics exposed by the ns1 project in addition to core platform metrics, such as CPU and memory metrics. However, the rule cannot include metrics from a different ns2 user-defined project.

  • By default, when you create an alerting rule, the namespace label is enforced on it even if a rule with the same name exists in another project. To create alerting rules that are not bound to their project of origin, see "Creating cross-project alerting rules for user-defined projects".

  • To reduce latency and to minimize the load on core platform monitoring components, you can add the openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus label to a rule. This label forces only the Prometheus instance deployed in the openshift-user-workload-monitoring project to evaluate the alerting rule and prevents the Thanos Ruler instance from doing so.

    If an alerting rule has this label, your alerting rule can use only those metrics exposed by your user-defined project. Alerting rules you create based on default platform metrics might not trigger alerts.

Optimizing alerting for user-defined projects

You can optimize alerting for your own projects by considering the following recommendations when creating alerting rules:

  • Minimize the number of alerting rules that you create for your project. Create alerting rules that notify you of conditions that impact you. It is more difficult to notice relevant alerts if you generate many alerts for conditions that do not impact you.

  • Create alerting rules for symptoms instead of causes. Create alerting rules that notify you of conditions regardless of the underlying cause. The cause can then be investigated. You will need many more alerting rules if each relates only to a specific cause. Some causes are then likely to be missed.

  • Plan before you write your alerting rules. Determine what symptoms are important to you and what actions you want to take if they occur. Then build an alerting rule for each symptom.

  • Provide clear alert messaging. State the symptom and recommended actions in the alert message.

  • Include severity levels in your alerting rules. The severity of an alert depends on how you need to react if the reported symptom occurs. For example, a critical alert should be triggered if a symptom requires immediate attention by an individual or a critical response team.

Creating alerting rules for user-defined projects

You can create alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics.

To help users understand the impact and cause of the alert, ensure that your alerting rule contains an alert message and severity value.
Prerequisites

  • You have enabled monitoring for user-defined projects.

  • You are logged in as a user that has the monitoring-rules-edit cluster role for the project where you want to create an alerting rule.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

  2. Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

    apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-alert
  namespace: ns1
spec:
  groups:
  - name: example
    rules:
    - alert: VersionAlert (1)
      for: 1m (2)
      expr: version{job="prometheus-example-app"} == 0 (3)
      labels:
        severity: warning (4)
      annotations:
        message: This is an example alert. (5)
    1 The name of the alerting rule you want to create.
    2 The duration for which the condition should be true before an alert is fired.
    3 The PromQL query expression that defines the new rule.
    4 The severity that alerting rule assigns to the alert.
    5 The message associated with the alert.

  3. Apply the configuration file to the cluster:

    $ oc apply -f example-app-alerting-rule.yaml

Creating cross-project alerting rules for user-defined projects

You can create alerting rules for user-defined projects that are not bound to their project of origin by configuring a project in the user-workload-monitoring-config config map. This allows you to create generic alerting rules that get applied to multiple user-defined projects instead of having individual PrometheusRule objects in each user project.

Prerequisites

  • If you are a cluster administrator, you have access to the cluster as a user with the cluster-admin cluster role.

  • If you are a non-administrator user, you have access to the cluster as a user with the following user roles:

    • The user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project to edit the user-workload-monitoring-config config map.

    • The monitoring-rules-edit cluster role for the project where you want to create an alerting rule.

  • A cluster administrator has enabled monitoring for user-defined projects.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

    $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

  2. Configure projects in which you want to create alerting rules that are not bound to a specific project:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement: [ <namespace> ] (1)
    # ...
    1 Specify one or more projects in which you want to create cross-project alerting rules. Prometheus and Thanos Ruler for user-defined monitoring do not enforce the namespace label in PrometheusRule objects created in these projects.

  3. Create a YAML file for alerting rules. In this example, it is called example-cross-project-alerting-rule.yaml.

  4. Add an alerting rule configuration to the YAML file. The following example creates a new cross-project alerting rule called example-security. The alerting rule fires when a user project does not enforce the restricted pod security policy:

    Example cross-project alerting rule
    apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-security
namespace: ns1 (1)
spec:
  groups:
    - name: pod-security-policy
      rules:
        - alert: "ProjectNotEnforcingRestrictedPolicy" (2)
          for: 5m (3)
          expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"} (4)
          annotations:
            message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy." (5)
          labels:
            severity: warning (6)
    1 Ensure that you specify the project that you defined in the namespacesWithoutLabelEnforcement field.
    2 The name of the alerting rule you want to create.
    3 The duration for which the condition should be true before an alert is fired.
    4 The PromQL query expression that defines the new rule.
    5 The message associated with the alert.
    6 The severity that alerting rule assigns to the alert.

    Ensure that you create a specific cross-project alerting rule in only one of the projects that you specified in the namespacesWithoutLabelEnforcement field. If you create the same cross-project alerting rule in multiple projects, it results in repeated alerts.

  5. Apply the configuration file to the cluster:

    $ oc apply -f example-cross-project-alerting-rule.yaml
Additional resources

Managing alerting rules for user-defined projects

OKD monitoring ships with a set of default alerting rules. As a cluster administrator, you can view the default alerting rules.

In OKD, you can view, edit, and remove alerting rules in user-defined projects.

Alerting rule considerations

  • The default alerting rules are used specifically for the OKD cluster.

  • Some alerting rules intentionally have identical names. They send alerts about the same event with different thresholds, different severity, or both.

  • Inhibition rules prevent notifications for lower severity alerts that are firing when a higher severity alert is also firing.

Accessing alerting rules for user-defined projects

To list alerting rules for a user-defined project, you must have been assigned the monitoring-rules-view cluster role for the project.

Prerequisites

  • You have enabled monitoring for user-defined projects.

  • You are logged in as a user that has the monitoring-rules-view cluster role for your project.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. To list alerting rules in <project>:

    $ oc -n <project> get prometheusrule

  2. To list the configuration of an alerting rule, run the following:

    $ oc -n <project> get prometheusrule <rule> -o yaml

Listing alerting rules for all projects in a single view

As a cluster administrator, you can list alerting rules for core OKD and user-defined projects together in a single view.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. In the Administrator perspective, navigate to Observe → Alerting → Alerting rules.

  2. Select the Platform and User sources in the Filter drop-down menu.

    The Platform source is selected by default.

Removing alerting rules for user-defined projects

You can remove alerting rules for user-defined projects.

Prerequisites

  • You have enabled monitoring for user-defined projects.

  • You are logged in as a user that has the monitoring-rules-edit cluster role for the project where you want to create an alerting rule.

  • You have installed the OpenShift CLI (oc).

Procedure

  • To remove rule <foo> in <namespace>, run the following:

    $ oc -n <namespace> delete prometheusrule <foo>

Disabling cross-project alerting rules for user-defined projects

Creating cross-project alerting rules for user-defined projects is enabled by default. Cluster administrators can disable the capability in the cluster-monitoring-config config map for the following reasons:

  • To prevent user-defined monitoring from overloading the cluster monitoring stack.

  • To prevent buggy alerting rules from being applied to the cluster without having to identify the rule that causes the issue.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. In the cluster-monitoring-config config map, disable the option to create cross-project alerting rules by setting the rulesWithoutLabelEnforcementAllowed value under data/config.yaml/userWorkload to false:

    kind: ConfigMap
apiVersion: v1
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    userWorkload:
      rulesWithoutLabelEnforcementAllowed: false
    # ...

  3. Save the file to apply the changes.

Additional resources

Sending notifications to external systems

In OKD 4, firing alerts can be viewed in the Alerting UI. Alerts are not configured by default to be sent to any notification systems. You can configure OKD to send alerts to the following receiver types:

  • PagerDuty

  • Webhook

  • Email

  • Slack

  • Microsoft Teams

Routing alerts to receivers enables you to send timely notifications to the appropriate teams when failures occur. For example, critical alerts require immediate attention and are typically paged to an individual or a critical response team. Alerts that provide non-critical warning notifications might instead be routed to a ticketing system for non-immediate review.

Checking that alerting is operational by using the watchdog alert

OKD monitoring includes a watchdog alert that fires continuously. Alertmanager repeatedly sends watchdog alert notifications to configured notification providers. The provider is usually configured to notify an administrator when it stops receiving the watchdog alert. This mechanism helps you quickly identify any communication issues between Alertmanager and the notification provider.

Configuring alert receivers

You can configure alert receivers to ensure that you learn about important issues with your cluster.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

Procedure

  1. In the Administrator perspective, go to Administration → Cluster Settings → Configuration → Alertmanager.

    Alternatively, you can go to the same page through the notification drawer. Select the bell icon at the top right of the OKD web console and choose Configure in the AlertmanagerReceiverNotConfigured alert.

  2. Click Create Receiver in the Receivers section of the page.

  3. In the Create Receiver form, add a Receiver name and choose a Receiver type from the list.

  4. Edit the receiver configuration:

    • For PagerDuty receivers:

      1. Choose an integration type and add a PagerDuty integration key.

      2. Add the URL of your PagerDuty installation.

      3. Click Show advanced configuration if you want to edit the client and incident details or the severity specification.

    • For webhook receivers:

      1. Add the endpoint to send HTTP POST requests to.

      2. Click Show advanced configuration if you want to edit the default option to send resolved alerts to the receiver.

    • For email receivers:

      1. Add the email address to send notifications to.

      2. Add SMTP configuration details, including the address to send notifications from, the smarthost and port number used for sending emails, the hostname of the SMTP server, and authentication details.

      3. Select whether TLS is required.

      4. Click Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the body of email notifications configuration.

    • For Slack receivers:

      1. Add the URL of the Slack webhook.

      2. Add the Slack channel or user name to send notifications to.

      3. Select Show advanced configuration if you want to edit the default option not to send resolved alerts to the receiver or edit the icon and username configuration. You can also choose whether to find and link channel names and usernames.

  5. By default, firing alerts with labels that match all of the selectors are sent to the receiver. If you want label values for firing alerts to be matched exactly before they are sent to the receiver, perform the following steps:

    1. Add routing label names and values in the Routing labels section of the form.

    2. Click Add label to add further routing labels.

  6. Click Create to create the receiver.

Configuring different alert receivers for default platform alerts and user-defined alerts

You can configure different alert receivers for default platform alerts and user-defined alerts to ensure the following results:

  • All default platform alerts are sent to a receiver owned by the team in charge of these alerts.

  • All user-defined alerts are sent to another receiver so that the team can focus only on platform alerts.

You can achieve this by using the openshift_io_alert_source="platform" label that is added by the Cluster Monitoring Operator to all platform alerts:

  • Use the openshift_io_alert_source="platform" matcher to match default platform alerts.

  • Use the openshift_io_alert_source!="platform" or 'openshift_io_alert_source=""' matcher to match user-defined alerts.

This configuration does not apply if you have enabled a separate instance of Alertmanager dedicated to user-defined alerts.

Creating alert routing for user-defined projects

If you are a non-administrator user who has been given the alert-routing-edit cluster role, you can create or edit alert routing for user-defined projects.

Prerequisites

  • A cluster administrator has enabled monitoring for user-defined projects.

  • A cluster administrator has enabled alert routing for user-defined projects.

  • You are logged in as a user that has the alert-routing-edit cluster role for the project for which you want to create alert routing.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a YAML file for alert routing. The example in this procedure uses a file called example-app-alert-routing.yaml.

  2. Add an AlertmanagerConfig YAML definition to the file. For example:

    apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: example-routing
  namespace: ns1
spec:
  route:
    receiver: default
    groupBy: [job]
  receivers:
  - name: default
    webhookConfigs:
    - url: https://example.org/post

    For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration defined in the AlertmanagerConfig object for namespace ns1 only applies to PrometheusRules resources in the same namespace.

  3. Save the file.

  4. Apply the resource to the cluster:

    $ oc apply -f example-app-alert-routing.yaml

    The configuration is automatically applied to the Alertmanager pods.

Configuring Alertmanager to send notifications

You can configure Alertmanager to send notifications by editing the alertmanager-main secret for default platform alerts or alertmanager-user-workload secret for user-defined alerts.

All features of a supported version of upstream Alertmanager are also supported in an OpenShift Alertmanager configuration. To check all the configuration options of a supported version of upstream Alertmanager, see Alertmanager configuration.

Configuring notifications for default platform alerts

You can configure Alertmanager to send notifications. Customize where and how Alertmanager sends notifications about default platform alerts by editing the default configuration in the alertmanager-main secret in the openshift-monitoring namespace.

Alertmanager does not send notifications by default. It is recommended to configure Alertmanager to receive notifications by setting up notifications details in the alertmanager-main secret configuration file.
Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

Procedure

  1. Open the Alertmanager YAML configuration file:

    • To open the Alertmanager configuration from the CLI:

      1. Print the currently active Alertmanager configuration from the alertmanager-main secret into alertmanager.yaml file:

        $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

      2. Open the alertmanager.yaml file.

    • To open the Alertmanager configuration from the OKD web console:

      1. Go to the Administration → Cluster Settings → Configuration → Alertmanager → YAML page of the web console.

  2. Edit the Alertmanager configuration by updating parameters in the YAML:

    global:
  resolve_timeout: 5m
  http_config:
    proxy_from_environment: true (1)
route:
  group_wait: 30s (2)
  group_interval: 5m (3)
  repeat_interval: 12h (4)
  receiver: default
  routes:
  - matchers:
    - "alertname=Watchdog"
    repeat_interval: 2m
    receiver: watchdog
  - matchers:
    - "service=<your_service>" (5)
    routes:
    - matchers:
      - <your_matching_rules> (6)
      receiver: <receiver> (7)
receivers:
- name: default
- name: watchdog
- name: <receiver>
  <receiver_configuration> (8)
    1 If you configured an HTTP cluster-wide proxy, set the proxy_from_environment parameter to true to enable proxying for all alert receivers.
    2 Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.
    3 Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.
    4 Specify the minimum amount of time that must pass before an alert notification is repeated. If you want a notification to repeat at each group interval, set the repeat_interval value to less than the group_interval value. The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.
    5 Specify the name of the service that fires the alerts.
    6 Specify labels to match your alerts.
    7 Specify the name of the receiver to use for the alerts.
    8 Specify the receiver configuration.

    • Use the matchers key name to indicate the matchers that an alert has to fulfill to match the node. Do not use the match or match_re key names, which are both deprecated and planned for removal in a future release.

    • If you define inhibition rules, use the following key names:

      • target_matchers: to indicate the target matchers

      • source_matchers: to indicate the source matchers

      Do not use the target_match, target_match_re, source_match, or source_match_re key names, which are deprecated and planned for removal in a future release.
    Example of Alertmanager configuration with PagerDuty as an alert receiver
    global:
  resolve_timeout: 5m
  http_config:
    proxy_from_environment: true
route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: default
  routes:
  - matchers:
    - "alertname=Watchdog"
    repeat_interval: 2m
    receiver: watchdog
  - matchers: (1)
    - "service=example-app"
    routes:
    - matchers:
      - "severity=critical"
      receiver: team-frontend-page
receivers:
- name: default
- name: watchdog
- name: team-frontend-page
  pagerduty_configs:
  - service_key: "<your_key>"
    http_config: (2)
      proxy_from_environment: true
      authorization:
        credentials: xxxxxxxxxx
    1 Alerts of critical severity that are fired by the example-app service are sent through the team-frontend-page receiver. Typically, these types of alerts would be paged to an individual or a critical response team.
    2 Custom HTTP configuration for a specific receiver. If you configure the custom HTTP configuration for a specific alert receiver, that receiver does not inherit the global HTTP config settings.

  3. Apply the new configuration in the file:

    • To apply the changes from the CLI, run the following command:

      $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

    • To apply the changes from the OKD web console, click Save.

Configuring notifications for user-defined alerts

If you have enabled a separate instance of Alertmanager that is dedicated to user-defined alert routing, you can customize where and how the instance sends notifications by editing the alertmanager-user-workload secret in the openshift-user-workload-monitoring namespace.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Print the currently active Alertmanager configuration into the file alertmanager.yaml:

    $ oc -n openshift-user-workload-monitoring get secret alertmanager-user-workload --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml

  2. Edit the configuration in alertmanager.yaml:

    global:
  http_config:
    proxy_from_environment: true (1)
route:
  receiver: Default
  group_by:
  - name: Default
  routes:
  - matchers:
    - "service = prometheus-example-monitor" (2)
    receiver: <receiver> (3)
receivers:
- name: Default
- name: <receiver>
  <receiver_configuration> (4)
    1 If you configured an HTTP cluster-wide proxy, set the proxy_from_environment parameter to true to enable proxying for all alert receivers.
    2 Specify labels to match your alerts. This example targets all alerts that have the service="prometheus-example-monitor" label.
    3 Specify the name of the receiver to use for the alerts group.
    4 Specify the receiver configuration.

  3. Apply the new configuration in the file:

    $ oc -n openshift-user-workload-monitoring create secret generic alertmanager-user-workload --from-file=alertmanager.yaml --dry-run=client -o=yaml |  oc -n openshift-user-workload-monitoring replace secret --filename=-

Additional resources