Prometheus Operator

The Prometheus Operator uses Kubernetes custom resources to simplify the deployment and configuration of Prometheus, Alertmanager, and related monitoring components. The default installation is intended to suit monitoring a Kubernetes cluster the chart is deployed onto.

Versions Supported

  • 35.5.1

Contents

A default integration will install the following components:

This integration also includes dashboards and alerts.

Use Custom Secrets for Alertmanager

For Alertmanager to work, the alerting configuration file should be set while deploying the integration. This configuration file includes sensitive information like SMTP credentials. If you want to skip entering SMTP credentials in plain text, you can do so by following the procedure below:

  1. Add Prometheus-Operator integration in the Cluster profile with changes to the following values:

    ParameterDescription
    alertmanager.alertmanagerSpec.useExistingSecretSet to true. This will skip creating the default Alertmanager secret.
    alertmanager.alertmanagerSpec.configSecretProvide the name of the custom Kubernetes Secret
    (should exist in the same namespace as the Alertmanager object)
    which contains configuration for the Alertmanager instance.
  2. Deploy the cluster with Prometheus-Operator integration. At this point, the Alertmanager pod will crash (as the custom secret defined is not yet available).

  1. Manually, create the secret in the same namespace as that of the Alertmanager object.
  1. Prepare the alertmanager.yaml contents:

    global:
    resolve_timeout: 5m
    receivers:
    - email_configs:
    - auth_identity: noreply@spectrocloud.com
    auth_password: abcd
    auth_username: noreply@spectrocloud.com
    from: noreply@spectrocloud.com
    send_resolved: true
    smarthost: smtp.gmail.com:587
    to: mark@spectrocloud.com
    name: prom-alert
    route:
    group_by:
    - job
    group_interval: 5m
    group_wait: 30s
    receiver: prom-alert
    repeat_interval: 4h
    routes:
    - match:
    alertname: Watchdog
    receiver: prom-alert
  2. Create the Secret using the Alertmanager config prepared above.

    kubectl create secret generic alertmanager-secret -n monitoring --from-file=./alertmanager.yaml
  3. Alertmanager pod will recover from the crash in the next reconciliation.

Configure scrape metrics for controller-manager, kube-schedule, and etcd

Due to security reasons, controller-manager, kube-schedule, and etcd runs in the localhost leading Prometheus to fail scrape metrics. Therefore, these targets are marked as down on Prometheus. Change the following in the Kubernetes pack layer in the cluster profile to scrape metrics for these services.

kubeadmconfig:
controllerManager:
extraArgs:
...
bind-address: "0.0.0.0"
scheduler:
extraArgs:
...
bind-address: "0.0.0.0"
etcd:
local:
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"

Ingress

Follow the below steps to configure Ingress on Grafana.

  1. Change serviceType from LoadBalancer to ClusterIP (line #427).
  2. Ingress (line #438).
    • Enable Ingress; change enabled from false to true.
    • Set Ingress rules like annotations, path, hosts, etc.

With these changes, you can access the Grafana service on the Ingress Controller LoadBalancer hostname/IP.

References

Prometheus-operator