Skip to main content
Version: latest

Deploy Monitoring Stack

The monitoring stack you will deploy uses the open-source tool, Prometheus, to support your environment's monitoring requirements. The monitoring stack is a centralized server or aggregation spot to which all other clusters will forward metrics. The monitoring stack is a dedicated Kubernetes cluster for monitoring and metrics aggregation in your Palette environment.

The monitoring stack uses a server-client architecture. The monitoring stack uses the Prometheus Operator pack to deploy all the dependencies the Prometheus server requires. The server exposes an API endpoint for all other clients to forward metrics. The clients are Kubernetes clusters with the Prometheus Agent pack installed and configured.

Use the following steps to deploy a monitoring stack, and learn how to configure a host cluster to forward metrics to the monitoring stack.


warning

We recommend you avoid installing applications in your monitoring stack. The monitoring stack will require all the allocated resources to support Prometheus and incoming metrics from all other clusters.

Deploy a Monitoring Stack​

The steps below will deploy a new host cluster with the Prometheus Operator pack. You can add the Prometheus Operator pack to an existing cluster if you already have a host cluster deployed in your environment.

The Prometheus Operator pack will install an unsecured Prometheus server by default. Use the With Authentication and Encryption tab for guidance on how to enable authentication.

You can choose to deploy the monitoring stack with or without authentication.


Without Authentication​

Prerequisites​

  • An infrastructure provider environment registered in Palette. Refer to the Clusters documentation for guidance on how to register your infrastructure provider environment in Palette.

  • The minimum required size for the Prometheus server is 4 CPU, 8 GB Memory, and 10 GB Storage. We recommend the monitoring stack have 1.5x to 2x the minimum required size:

    Recommended size:

    • 8 CPU
    • 16 GB Memory
    • 20 GB Storage.

    As new clusters with the Prometheus agent are added to your environment, review the resource utilization and consider increasing resources if needed. As the Prometheus documentation recommends, each additional agent requires the following resources from the monitoring stack:

    Each added agent:

    • 0.1 CPU

    • 250 MiB Memory

    • 1 GB Storage.


    Refer to the Prometheus Operational aspects documentation for additional guidance.

Create Cluster Profile and Deploy​

  1. Log in to Palette.

  2. Navigate to the left Main Menu and select Profiles.

  3. Click on Add Cluster Profile to create a new cluster profile.

  4. Provide the cluster profile a name and select the type Full. Click on Next.

  5. Select the infrastructure provider and continue.

  6. Go ahead and select the desired operating system, Kubernetes distribution, container network interface (CNI), and container storage interface (CSI). Click on Next Layer after each selection. When you have completed selecting core infrastructure for the profile, click Confirm.

  7. In the next screen that displays, select Add New Pack.

  8. Use the following information to find the Prometheus Operator pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Prometheus Grafana
    • Pack Version: 44.25.X or newer.

  9. Review the YAML configuration on the right. Scroll down in the file until you find the parameter adminPassword. Input the password value for the admin user. The default admin user name is admin.

  10. Next, click on the Presets button to expand the options drawer.

  11. Scroll down the presets option menu and enable Remote Monitoring. Confirm your changes. You can enable several options to expand the functionality of the monitoring stack. Review the Prometheus Operator pack documentation to learn more about the available options.

  12. Confirm your changes by selecting Confirm & Create.

  13. Click on Add New Pack.

  14. Use the following information to find the Spectro Cluster Metrics pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Spectro Cluster Metrics
    • Pack Version: 3.3.X or newer

  15. Use the default values. Confirm your changes by selecting Confirm & Create.

  16. Click on Add New Pack.

  17. Use the following information to find the Spectrocloud Grafana Dashboards pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Spectrocloud Grafana Dashboards
    • Pack Version: 1.0.X or newer

  18. Use the default values. Confirm your changes by selecting Confirm & Create.

  19. Click on Next to review the cluster profile and save it.

  20. Navigate to the left Main Menu and select Clusters.

  21. Click on Add New Cluster. Select Deploy New Cluster.

  22. Choose the infrastructure provider you selected for the cluster profile you created earlier.

  23. Assign a name to the host cluster and select the registered account you will deploy it to. Click on Next.

  24. Choose the cluster profile you created earlier and complete the remainder of the cluster creation process.

When you deploy the cluster, a host cluster with Prometheus will be installed and ready to receive information from Prometheus agents.

Validate​

To validate the monitoring stack is successfully deployed and ready to receive Prometheus agent requests, use the following steps.

  1. Log in to Palette.

  2. Navigate to the left Main Menu and select Clusters.

  3. Select the monitoring stack cluster to review the details page.

  4. Ensure the cluster is in Running state.

  5. Click on the exposed service URL for the service prometheus-operator-kube-prometheus-stack-grafana.

  6. Log in to the Grafana dashboard using the user admin and the password you specified in the cluster profile.

With Authentication and Encryption​

To enable Hypertext Transfer Protocol Secure (HTTPS), you must make several architectural decisions and decide on various options for setting up the environment. These options range from choosing what will generate the Secure Socket Layer (SSL) certificates to how incoming requests for Grafana or Prometheus are handled.

The approach presented in this guide is one pattern you can use. However, we encourage you to discuss this pattern with your system administrator or platform engineering team before changing your infrastructure and Domain Name System (DNS) resources.

The following diagram represents the infrastructure pattern you will use in this guide to enable HTTPS.

An architecture diagram that displays the network flow and infrastructure components

In this guide, the following domains are used to expose the monitoring stack:

DomainDescription
metrics.example.comThe endpoint that all host clusters will forward Prometheus metrics.
monitoring.example.comThe Grafana dashboard.

Prerequisites​

  • Experience with DNS and setting up custom domains that use SSL certificates are required for this guide. In addition, the following actions are needed.

    • Ability to create and update DNS record names.

    • Two custom domain names. One domain is for the Grafana dashboard, and another is for host clusters to forward metrics to the monitoring stack.

    • Ability to create a public certificate for each domain.

  • An infrastructure provider environment registered in Palette. Refer to the Clusters documentation for guidance on how to register your infrastructure provider environment in Palette.

  • htpasswd or similar basic auth password file generator tool.

  • The minimum required size for the Prometheus server is 4 CPU, 8 GB Memory, and 10 GB Storage. We recommend the monitoring stack have 1.5x to 2x the minimum required size:

    Recommended size:

    • 8 CPU
    • 16 GB Memory
    • 20 GB Storage.

    As new clusters with the Prometheus agent are added to your environment, review the resource utilization and consider increasing resources if needed. As the Prometheus documentation recommends, each additional agent requires the following resources from the monitoring stack:

    Each added agent:

    • 0.1 CPU
    • 250 MiB Memory
    • 1 GB Storage.

    Refer to the Prometheus Operational aspects documentation for additional guidance.

Create Cluster Profile and Deploy​

  1. Log in to Palette.

  2. Navigate to the left Main Menu and select Profiles.

  3. Click on Add Cluster Profile to create a new cluster profile.

  4. Provide the cluster profile a name and select the type Full. Click on Next.

  5. Select the infrastructure provider and continue.

  6. Go ahead and select the desired operating system, Kubernetes distribution, container network interface (CNI), and container storage interface (CSI). Click on Next Layer after each selection. When you have completed selecting core infrastructure for the profile, click Confirm.

  7. In the next screen that displays, select Add New Pack.

  8. Use the following information to add the Nginx ingress controller pack.

    • Pack Type: Ingress
    • Registry: Public Repo
    • Pack Name: Nginx
    • Pack Version: 1.5.X or newer.

  9. Review the YAML configuration on the right and add the following changes:


charts:
  ingress-nginx:
    controller:
      extraArgs:
        enable-ssl-passthrough: true

  1. Click on Confirm & Create.

  2. Select Add New Pack.

  3. Use the following information to find the Prometheus Operator pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Prometheus Grafana
    • Pack Version: 44.3.X or newer.

  4. Next, click on the Presets button to expand the options drawer.

  5. Scroll down the presets option menu and enable Remote Monitoring.

  6. Review the YAML configuration on the right. Scroll down in the file until you find the parameter grafana.adminPassword. Input the password value for the admin user. The default admin user name is admin.


    charts:
      kube-prometheus-stack:
        grafana:
          adminPassword: "YourPassword"

  7. Next, update the prometheus.service.type parameter to ClusterIP.


    charts:
      kube-prometheus-stack:
        prometheus:
          service:
            type: ClusterIP

  8. Confirm your changes by selecting Confirm & Create. You can enable several options to expand the functionality of the monitoring stack. Review the Prometheus Operator pack documentation to learn more about the available options.

  9. Click on Add New Pack.

  10. Use the following information to find the Spectro Cluster Metrics pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Spectro Cluster Metrics
    • Pack Version: 3.3.X or newer

  11. Use the default values. Confirm your changes by selecting Confirm & Create.

  12. Click on Add New Pack.

  13. Use the following information to find the Spectrocloud Grafana Dashboards pack.

    • Pack Type: Monitoring
    • Registry: Public Repo
    • Pack Name: Spectrocloud Grafana Dashboards
    • Pack Version: 1.0.X or newer

  14. Use the default values. Confirm your changes by selecting Confirm & Create.

  15. Click on Next to review the cluster profile and save it.

  16. Navigate to the left Main Menu and select Clusters.

  17. Click on Add New Cluster. Select Deploy New Cluster.

  18. Choose the infrastructure provider you selected for the cluster profile you created earlier.

  19. Assign a name to the host cluster and select the registered account you will deploy it to. Click on Next.

  20. Choose the cluster profile you created earlier and complete the remainder of the cluster creation process.

  21. Once the host cluster is deployed, navigate to the left Main Menu and select Clusters. Click on your cluster to display the details page and ensure its status is Running.

  22. Download the Kubernetes configuration file. Click on the URL that has the name of your cluster followed by a period and the word kubeconfig. Refer to the Access Cluster with CLI for additional guidance.

  23. Open a terminal window and set the environment variable KUBECONFIG to point to kubeconfig file you downloaded.


    export KUBECONFIG=~/Downloads/dev-monitoring-stack.config

  24. Create an htpasswd file for the user agent and assign a password. You can choose a different username if you prefer.


    htpasswd -c auth agent

    Output:

    New password: [agent_password_here]
    New password:
    Re-type new password:
    Adding password for user agent

  25. Convert the htpasswd file into a Kubernetes secret.


      kubectl create secret generic basic-auth --from-file=auth --namespace monitoring

    Output:

      secret "basic-auth" created

  26. Navigate back to Palette, and review the cluster profile you created for the monitoring stack. From the left Main Menu > Profiles > select your cluster profile. Click on the Prometheus operator layer to edit the YAML.

  27. Locate the prometheus.ingress section near the end of the file. Update the ingress configuration with the values provided below. Replace the hosts parameter with your custom domain.


    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        nginx.ingress.kubernetes.io/auth-type: basic
        nginx.ingress.kubernetes.io/auth-secret: basic-auth
        nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
      hosts:
        - metrics.example.com

  28. Confirm your updates on the next screen that displays.

  29. From the left Main Menu, select Clusters and access the monitoring stack host cluster.

  30. Click on the Updates Available button to review the changes.

  31. Accept the changes and select Confirm Updates.

warning

The following steps can be complex, depending on your environment and your access. Discuss the remaining step with your network administrator team if you need additional guidance.

  1. Create a Canonical Name (CNAME) record for each of the following services and add the load balancer hostname to the CNAME's record value. Use the table below to identify which mapping to use between the domain and each load balancer hostname.
ServiceDomainCNAME Value Example
nginx-ingress-controllermetrics.example.coma57b622a0c0a148189ed00df614481c9-1803006767.us-east-1.elb.amazonaws.com
prometheus-operator-kube-prometheus-stack-grafanamonitoring.example.coma702f8a14b9684a30b18b875d2cca997-1676466159.us-east-1.elb.amazonaws.com

A screenshot of the Palette user interface with two boxes highlighting the load balancers that you need to add to your CNAME.

info

You can also use kubectl to retrieve the load balancer hostname.

Grafana:

kubectl get service prometheus-operator-kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Prometheus:

kubectl get service nginx-ingress-controller --namespace nginx -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

  1. Create a public certificate for each domain. If you are using a public cloud provider, use the native certificate manager service to generate a public certificate that you can attach to the load balancers with minimal overhead. On-prem, use the default certificate manager used by your organization.

  2. Update the network rules for each of the load balancers to allow inbound port 443.


  1. Next, update the load balancer listeners to forward requests from port 443 to the respective target port on the monitoring stack. The following table will map the service's load balancer listener with the respective configuration. Refer to the architecture diagram from the introduction to help you visualize the mapping.
Inbound Load Balancer PortDomainMonitoring Stack PortService
443monitoring.example.comUse the same instance port the original entry for port 80 is using.prometheus-operator-kube-prometheus-stack-grafana
443metrics.example.comUse the same instance port the original entry for port 80 is using.nginx-ingress-controller
  1. Wait for the DNS changes to propagate. This can take up to five minutes.

Your monitoring stack is now enabled with authentication and network encryption.

Validate​

To validate the monitoring stack is successfully deployed and ready to receive Prometheus agent requests, use the following steps.

  1. Log in to Palette.

  2. Navigate to the left Main Menu and select Clusters.

  3. Select the monitoring stack cluster to review the details page.

  4. Ensure the cluster is in Running state.

  5. Next, open up your web browser and visit the domain name you specified for the Grafana dashboard. Example: https://monitoring.example.com.


  1. Log in to the Grafana dashboard using the user admin and the password you specified in the cluster profile.

  2. After you have verified you can log in to the Grafana dashboard, open a new tab and visit the Prometheus endpoint. Example: https://metrics.example.com

  3. Log in with the user agent and use the password you specified in the htpasswd file.

info

A common error is not updating the network rules to allow inbound connections for port 443 to the load balancers. Ensure you have updated all the required network rules to allow inbound network requests for port 443.

Next Steps​

Now that you have a monitoring stack deployed and available in your environment, start adding the Prometheus agent to new and existing clusters. Check out the Enable Monitoring on Host Cluster to get started.