Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
×

Accessing third-party monitoring UIs and APIs

In OKD 4.10, you cannot access third-party web browser user interfaces (UIs) for the following monitoring components: Alertmanager, Thanos Ruler, and Thanos Querier. However, you can access web UIs for Grafana and Prometheus, although this access is deprecated and is planned to be removed in a future OKD release. In addition, you can access web service APIs for third-party monitoring components from the command line interface (CLI).

Accessing third-party monitoring UIs

OKD does not provide or support direct access to third-party web user interfaces (UIs) for the following components in the monitoring stack: Alertmanager, Thanos Ruler, and Thanos Querier. As an alternative to these third-party UIs, navigate to the Observe section of the OKD web console to access metrics, alerting, metrics targets, and dashboard UIs for platform components.

Although you can access the third-party Grafana and Prometheus web UIs from the web console or the CLI, this access is deprecated and is planned to be removed in a future OKD release.

Accessing third-party monitoring web service APIs

You can directly access third-party web service APIs from the command line for monitoring stack components such as Prometheus, Alertmanager, Thanos Ruler, and Thanos Querier.

The following example shows how to query the service API receivers for Alertmanager. This example requires that the associated user account be bound against the monitoring-alertmanager-edit role in the openshift-monitoring namespace and that the account has the privilege to view the route. This access only supports using a Bearer Token for authentication.

$ host=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.spec.host})
$ token=$(oc whoami -t)
$ curl -H "Authorization: Bearer $token" -k "https://$host/api/v2/receivers"

To access Thanos Ruler and Thanos Querier service APIs, the requesting account must have get permission on the namespaces resource, which can be done by granting the cluster-monitoring-view cluster role to the account.

Querying metrics by using the federation endpoint for Prometheus

From OKD 4.10.17, you can use the federation endpoint to scrape platform and user-defined metrics from a network location outside the cluster. To do so, access the Prometheus /federate endpoint for the cluster via an OKD route.

A delay in retrieving metrics data occurs when you use federation. This delay can affect the accuracy and timeliness of the scraped metrics.

Using the federation endpoint can also degrade the performance and scalability of your cluster, especially if you use the federation endpoint to retrieve large amounts of metrics data. To avoid these issues, follow these recommendations:

  • Do not try to retrieve all metrics data via the federation endpoint. Query it only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation.

  • Avoid querying the federation endpoint frequently. Limit queries to a maximum of one every 30 seconds.

If you need to forward large amounts of data outside the cluster, use remote write instead. For more information, see the Configuring remote write storage section.
Prerequisites

  • You have installed the OpenShift CLI (oc).

  • You have obtained the host URL for the OKD route.

  • You have access to the cluster as a user with the cluster-monitoring-view cluster role or have obtained a bearer token with get permission on the namespaces resource.

    You can only use bearer token authentication to access the federation endpoint.
Procedure

  1. Retrieve the bearer token:

    $ token=`oc whoami -t`

  2. Query metrics from the /federate route. The following example queries up metrics:

    $ curl -G -s -k -H "Authorization: Bearer $token" \
    'https:/<federation_host>/federate' \ (1)
    --data-urlencode 'match[]=up'
    1 For <federation_host>, substitute the host URL for the federation route.
    Example output
    # TYPE up untyped
up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214
up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597
up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
...

Additional resources