1. Documentation
×

Enabling alert routing for user-defined projects

In OKD 4.16, a cluster administrator can enable alert routing for user-defined projects. This process consists of two general steps:

  • Enable alert routing for user-defined projects to use the default platform Alertmanager instance or, optionally, a separate Alertmanager instance only for user-defined projects.

  • Grant users permission to configure alert routing for user-defined projects.

After you complete these steps, developers and other users can configure custom alerts and alert routing for their user-defined projects.

Understanding alert routing for user-defined projects

As a cluster administrator, you can enable alert routing for user-defined projects. With this feature, you can allow users with the alert-routing-edit role to configure alert notification routing and receivers for user-defined projects. These notifications are routed by the default Alertmanager instance or, if enabled, an optional Alertmanager instance dedicated to user-defined monitoring.

Users can then create and configure user-defined alert routing by creating or editing the AlertmanagerConfig objects for their user-defined projects without the help of an administrator.

After a user has defined alert routing for a user-defined project, user-defined alert notifications are routed as follows:

  • To the alertmanager-main pods in the openshift-monitoring namespace if using the default platform Alertmanager instance.

  • To the alertmanager-user-workload pods in the openshift-user-workload-monitoring namespace if you have enabled a separate instance of Alertmanager for user-defined projects.

The following are limitations of alert routing for user-defined projects:

  • For user-defined alerting rules, user-defined routing is scoped to the namespace in which the resource is defined. For example, a routing configuration in namespace ns1 only applies to PrometheusRules resources in the same namespace.

  • When a namespace is excluded from user-defined monitoring, AlertmanagerConfig resources in the namespace cease to be part of the Alertmanager configuration.

Enabling the platform Alertmanager instance for user-defined alert routing

You can allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • A cluster administrator has enabled monitoring for user-defined projects.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config ConfigMap object:

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add enableUserAlertmanagerConfig: true in the alertmanagerMain section under data/config.yaml:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    # ...
    alertmanagerMain:
      enableUserAlertmanagerConfig: true (1)
    # ...
    1 Set the enableUserAlertmanagerConfig value to true to allow users to create user-defined alert routing configurations that use the main platform instance of Alertmanager.

  3. Save the file to apply the changes. The new configuration is applied automatically.

Enabling a separate Alertmanager instance for user-defined alert routing

In some clusters, you might want to deploy a dedicated Alertmanager instance for user-defined projects, which can help reduce the load on the default platform Alertmanager instance and can better separate user-defined alerts from default platform alerts. In these cases, you can optionally enable a separate instance of Alertmanager to send alerts for user-defined projects only.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have enabled monitoring for user-defined projects.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the user-workload-monitoring-config ConfigMap object:

    $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

  2. Add enabled: true and enableAlertmanagerConfig: true in the alertmanager section under data/config.yaml:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    alertmanager:
      enabled: true (1)
      enableAlertmanagerConfig: true (2)
    1 Set the enabled value to true to enable a dedicated instance of the Alertmanager for user-defined projects in a cluster. Set the value to false or omit the key entirely to disable the Alertmanager for user-defined projects. If you set this value to false or if the key is omitted, user-defined alerts are routed to the default platform Alertmanager instance.
    2 Set the enableAlertmanagerConfig value to true to enable users to define their own alert routing configurations with AlertmanagerConfig objects.

  3. Save the file to apply the changes. The dedicated instance of Alertmanager for user-defined projects starts automatically.

Verification

  • Verify that the user-workload Alertmanager instance has started:

    # oc -n openshift-user-workload-monitoring get alertmanager
    Example output
    NAME            VERSION   REPLICAS   AGE
user-workload   0.24.0    2          100s

Granting users permission to configure alert routing for user-defined projects

You can grant users permission to configure alert routing for user-defined projects.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin cluster role.

  • You have enabled monitoring for user-defined projects.

  • The user account that you are assigning the role to already exists.

  • You have installed the OpenShift CLI (oc).

Procedure

  • Assign the alert-routing-edit cluster role to a user in the user-defined project:

    $ oc -n <namespace> adm policy add-role-to-user alert-routing-edit <user> (1)
    1 For <namespace>, substitute the namespace for the user-defined project, such as ns1. For <user>, substitute the username for the account to which you want to assign the role.
Additional resources