Prometheus#

Prometheus is part of the alerting stack that we use in Operate-First.

Firstly the alert itself must be constructed through a PrometheusRule file. Typically these files live alongside the application that they are creating the alerts for. Examples of this can be seen for argocd in MOC-Infra, jupyterhub on Smuag, and the blackbox exporter on Smuag. It is important to note that these PrometheusRule files are defined per overlay. For more information on creating the alerts themselves, refer to the prometheus docs
Secondly, we must tell the alertmanager-github-receiver how to route the alerts. This can be done by modifying the alertmanager-main-secret file. This defines which alerts should be forwarded to which receivers. To remove or silence an alert, the entry can either be deleted, or the receiver can be swapped to the "null" value, which will silence it.