Prometheus Operator

The Prometheus Operator uses Kubernetes custom resources to simplify the deployment and configuration of Prometheus, Alertmanager, and related monitoring components. The default installation is intended to suit monitoring a Kubernetes cluster the chart is deployed onto.

Contents

A Default integration will install the following components:

This integration also includes dashboards and alerts.

Use custom secrets for alertmanager

For Alertmanager to work fine, alerting config should be set while deploying the integration. This config also includes sensitive information like SMTP credentials. If you want to skip entering SMTP credentials in plain text, you can do so by following the procedure below

  • Add Prometheus-Operator integration in the Cluster profile with changes to the following values

    • alertmanager.alertmanagerSpec.useExistingSecret - Set to true. This will skip creating the default Alertmanager secret
    • alertmanager.alertmanagerSpec.configSecret - Provide the name of the custom Kubernetes Secret (should exist in the same namespace as the Alertmanager object) which contains configuration for the Alertmanager instance
  • Deploy the cluster with Prometheus-Operator integration. At this point, Alertmanager pod will crash (as the custom secret defined is not yet available)

  • Create the secret manually in the same namespace as that of Alertmanager object

    Prepare alertmanager.yaml contents

    global:
    resolve_timeout: 5m
    receivers:
    - email_configs:
    - auth_identity: noreply@spectrocloud.com
    auth_password: abcd
    auth_username: noreply@spectrocloud.com
    from: noreply@spectrocloud.com
    send_resolved: true
    smarthost: smtp.gmail.com:587
    to: mark@spectrocloud.com
    name: prom-alert
    route:
    group_by:
    - job
    group_interval: 5m
    group_wait: 30s
    receiver: prom-alert
    repeat_interval: 4h
    routes:
    - match:
    alertname: Watchdog
    receiver: prom-alert

    Create the Secret using the Alertmanager config prepared above

    kubectl create secret generic alertmanager-secret -n monitoring --from-file=./alertmanager.yaml
  • Alertmanager pod will recover from the crash in the next reconciliation

Ingress

Follow below steps to configure Ingress on Grafana

  1. Change serviceType from "LoadBalancer" to "ClusterIP" (line #427)
  2. Ingress (line #438)
    • Enable Ingress; change enabled from false to "true"
    • Set Ingress rules like annotations, path, hosts, etc.

With these changes, you can access Grafana service on the Ingress Controller LoadBalancer hostname / IP

References

https://github.com/coreos/prometheus-operator