ODH Logo

Enable monitoring for your service

We use ODH Prometheus as part of our monitoring stack to scrape metrics.

To enable our Prometheus to monitor your services, you need to add the appropriate RBAC and service monitors.

You can choose to have one servicemonitor per service, or group all your services into one service monitor. The steps are similar in both cases.

Pre-requisite:

You will need pre-requisite tools to follow along with this doc, please do one of the following:

  • Install our toolbox to have the developer setup ready automatically for you.
  • Install the tools manually. You’ll need kustomize, sops and ksops.

Also ensure that each namespace the services belong to are listed here. If not then please file an issue here.

You will also need the opfcli. install the latest version from here.

Please fork/clone the operate-first/apps repository. During this whole setup, we’ll be working within this repository.

Steps:

  1. Run the following command to enable rbac for a namespace:
opfcli enable-monitoring $NAMESPACE

This command will add a line that contains the ‘monitoring-rbac’ component to the components field in the kustomization.yaml of the given namespace.

  1. Add the service monitor filling out the details below accordingly:
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ${SERVICE_MONITOR_NAME}
  labels:
    monitor-component: ${TEAM_OR_SERVICE_NAME}
spec:
  endpoints:
    - targetPort: ${SERVICE_1_PORT}
    - targetPort: ${SERVICE_2_PORT}
    - targetPort: ...
  namespaceSelector:
    matchNames:
      - ${SERVICE_1_NAMESPACE}
      - ${SERVICE_2_NAMESPACE}
      - ...
  selector: {}

Add this service monitor to kfdefs/overlays/${ENV}/${CLUSTER}/opf-monitoring/servicemonitors

  1. Add the servicemonitor from (2) to kfdefs/overlays/${ENV}/${CLUSTER}/opf-monitoring/servicemonitors/kustomization.yaml