The Prometheus Operator uses Kubernetes custom resources to simplify the deployment and configuration of Prometheus, Alertmanager, and related monitoring components. The default installation is intended to suit monitoring a Kubernetes cluster the chart is deployed onto.

A default integration will install the following components:

• prometheus-operator

• prometheus

• alertmanager.

• node-exporter

• kube-state-metrics

• grafana

• and the service monitors to scrape internal Kubernetes components:

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager
  • etcd
  • kube-dns/coredns
  • kube-proxy

This integration also includes dashboards and alerts.

For Alertmanager to work, the alerting configuration file should be set while deploying the integration. This configuration file includes sensitive information like SMTP credentials. If you want to skip entering SMTP credentials in plain text, you can do so by following the procedure below:

1. Add Prometheus-Operator integration in the Cluster profile with changes to the following values:

   Parameter	Description
   alertmanager.alertmanagerSpec.useExistingSecret	Set to true. This will skip creating the default Alertmanager secret.
   alertmanager.alertmanagerSpec.configSecret	Provide the name of the custom Kubernetes Secret (should exist in the same namespace as the Alertmanager object) which contains configuration for the Alertmanager instance.

2. Deploy the cluster with Prometheus-Operator integration. At this point, the Alertmanager pod will crash (as the custom secret defined is not yet available).

3. Manually, create the secret in the same namespace as that of the Alertmanager object.

4. Prepare the alertmanager.yaml contents:

        global:
          resolve_timeout: 5m
        receivers:
         - email_configs:
           - auth_identity: noreply@spectrocloud.com
             auth_password: abcd
             auth_username: noreply@spectrocloud.com
             from: noreply@spectrocloud.com
             send_resolved: true
             smarthost: smtp.gmail.com:587
             to: mark@spectrocloud.com
           name: prom-alert
        route:
          group_by:
          - job
          group_interval: 5m
          group_wait: 30s
          receiver: prom-alert
          repeat_interval: 4h
          routes:
          - match:
              alertname: Watchdog
            receiver: prom-alert

5. Create the Secret using the Alertmanager config prepared above.

   kubectl create secret generic alertmanager-secret -n monitoring --from-file=./alertmanager.yaml  

6. Alertmanager pod will recover from the crash in the next reconciliation.

Due to security reasons, controller-manager, kube-schedule, and etcd runs in the localhost leading Prometheus to fail scrape metrics. Therefore, these targets are marked as down on Prometheus. Change the following in the Kubernetes pack layer in the cluster profile to scrape metrics for these services.

    kubeadmconfig:
      controllerManager:
        extraArgs:
            ...
          bind-address: "0.0.0.0"
      scheduler:
        extraArgs:
          ...
          bind-address: "0.0.0.0"  
      etcd:
        local:
          extraArgs:
            listen-metrics-urls: "http://0.0.0.0:2381"    

Follow the below steps to configure Ingress on Grafana.

1. Change serviceType from LoadBalancer to ClusterIP (line #427).
2. Ingress (line #438).
   • Enable Ingress; change enabled from false to true.
   • Set Ingress rules like annotations, path, hosts, etc.

With these changes, you can access the Grafana service on the Ingress Controller LoadBalancer hostname/IP.

Prometheus-operator