Overview

OKD exposes metrics that can be collected and stored in back-ends by the cluster-monitoring-operator. As an OKD administrator, you can view system resources, containers and components metrics in one dashboard interface, Grafana.

This topic provides information on scaling the cluster monitoring operator.

Recommendations for OKD

Capacity Planning for Cluster Monitoring Operator

Various tests were performed for different scale sizes. The Prometheus database grew, as reflected in the table below.

Table 1. Prometheus Database storage requirements based on number of nodes/pods in the cluster
Number of Nodes Number of Pods Prometheus storage growth per day Prometheus storage growth per 15 days RAM Space (per scale size) Network (per tsdb chunk)

50

1800

6.3 GB

94 GB

6 GIB

16 MIB

100

3600

13 GB

195 GB

10 GIB

26 MIB

150

5400

19 GB

283 GB

12 GIB

36 MIB

200

7200

25 GB

375 GB

14 GIB

46 MIB

In the above calculation, approximately 20 percent of the expected size was added as overhead to ensure that the storage requirements do not exceed the calculated value.

The above calculation was developed for the default OKD cluster-monitoring-operator. For higher scale, edit the prometheus_persistent_volume_size variable, which defaults to 50Gib.

CPU utilization has minor impact. The ratio is approximately 1 core out of 40 per 50 nodes and 1800 pods.

Lab Environment

All experiments were performed in an OKD on OpenStack environment:

  • Infra nodes (VMs) - 40 cores, 157 GB RAM.

  • CNS nodes (VMs) - 16 cores, 62 GB RAM, NVMe drives.

Prerequisites

Based on you scale destination, compute and set the relevant PV size for the Prometheus data store. Since the default Prometheus pods replicas is 2, for 100 nodes with 3600 pods you will need 188 Gib.

For example:

94 GB (space per 15 days ) * 2 (pods) = 188 GB free

Based on this equation, set the prometheus_persistent_volume_size=94Gib.

Scaling the Prometheus Components

To scale up the number of OKD Prometheus pods replicas, run:

# oc scale -n openshift-monitoring --replicas=3 statefulset prometheus-k8s
  • The default replicas are 2 Prometheus pods.

  • If you add a new node to or remove an existing node from a Prometheus cluster, the data stored in the cluster rebalances across the cluster.

To scale down:

# oc scale -n openshift-monitoring --replicas=0 statefulset prometheus-k8s

This will remove the Cassandra pod.

If the scale down process completed and the existing Cassandra nodes are functioning as expected, you can also delete the rc for this Cassandra instance and its corresponding persistent volume claim (PVC). Deleting the PVC can permanently delete any data associated with this Cassandra instance, so if the scale down did not fully and successfully complete, you will not be able to recover the lost data.