Kyma comes bundled with third-party applications like Prometheus, Alertmanager, and Grafana, that offer a monitoring functionality for all Kyma resources. These applications are deployed during the Kyma cluster installation, along with a set of pre-defined alerting rules, Grafana dashboards, and Prometheus configuration.

The whole installation package provides the end-to-end Kubernetes cluster monitoring that allows you to:

• View metrics exposed by the Pods.
• Use the metrics to create descriptive dashboards that monitor any Pod anomalies.
• Manage the default alert rules and create new ones.
• Set up channels for notifications informing of any detected alerts.

NOTE: The monitoring component is available by default in the cluster installation, but disabled in the Kyma Lite local installation on Minikube. Enable the component to install it with the local profile.

You can install Monitoring as part of Kyma predefined profiles. For production purposes, use the production profile which ensures:

• Increased retention time to prevent data loss in case of prolonged troubleshooting.
• Increased memory and CPU values that optimize the performance of your component.

Before you go into component details, find out more about the end-to-end monitoring flow in Kyma.

End-to-end monitoring flow

The complete monitoring flow in Kyma comes down to these components and steps:

1. Upon Kyma installation on a cluster, Prometheus Operator creates a Prometheus instance with the default configuration.
2. The Prometheus server periodically polls all metrics exposed on /metrics endpoints of ports specified in ServiceMonitor CRDs. Prometheus stores these metrics in a time-series database.
3. If Prometheus detects any metric values matching the logic of alerting rules, it triggers the alerts and passes them to Alertmanager.
4. If you manually configure a notification channel, you can instantly receive detailed information on metric alerts detected by Prometheus.
5. You can visualize metrics and track their historical data on Grafana dashboards.

Monitoring components

The diagram presents monitoring components and the way they interact with one another.

1. Prometheus Operator creates a Prometheus instance, manages its deployment, and provides configuration for it. It also deploys Alertmanager and operates ServiceMonitor custom resources that specify monitoring definitions for groups of services.

2. Prometheus collects metrics from Pods. Metrics are the time-stamped data that provide information on the running jobs, workload, CPU consumption, memory usage, and more. To obtain such metrics, Prometheus uses the kube-state-metrics service. It generates the metrics from Kubernetes API objects and exposes them on the /metrics HTTP endpoint.
   Pods can also contain applications with custom metrics, such as the total storage space available in the MinIO server. Prometheus stores this polled data in a time-series database (TSDB) and runs rules over them to generate alerts if it detects any metric anomalies. It also scrapes metrics provided by Node Exporter which exposes existing hardware metrics from external systems as Prometheus metrics.

   NOTE: Besides this main Prometheus instance, there is a second Prometheus instance running in the kyma-system Namespace. This second instance is responsible for collecting and aggregating Istio Service Mesh metrics.

3. ServiceMonitors monitor services and specify the endpoints from which Prometheus should poll the metrics. Even if you expose a handful of metrics in your application, Prometheus polls only those from the /metrics endpoints of ports specified in ServiceMonitor CRDs.

4. Alertmanager receives alerts from Prometheus and forwards this data to configured Slack or Victor Ops channels. You can use PrometheusRules to define alert conditions for metrics. Kyma provides a set of out-of-the-box alerting rules that are passed from Prometheus to Alertmanager. The definitions of such rules specify the alert logic, the value at which alerts are triggered, the alerts' severity, and more.

   NOTE: There are no notification channels configured in the default monitoring installation. The current configuration allows you to add either Slack or Victor Ops channels.

5. Grafana provides a dashboard and a graph editor to visualize metrics collected from the Prometheus API. Grafana uses the query language called PromQL to select and aggregate metrics data from the Prometheus database. To access the Grafana UI, use the https://grafana.{DOMAIN} address, where {DOMAIN} is the domain of your Kyma cluster.

The monitoring chart is pre-configured to collect all metrics relevant for observing the in-cluster Istio Service Mesh, including the proxy-level, service-level, and control-plane metrics.

The concept of collecting the service-level metrics is built around the Istio Proxy implemented by Envoy. Istio Proxy collects all communication details inside the service mesh in a decentralized way. After scraping these high cardinality metrics from the envoys, the metrics need to be additionally aggregated on a service level to get the final service-related details.

Following the Istio's observability best practice, the scraping and aggregation of the service-level metrics is done in a dedicated Prometheus instance. That instance has the smallest possible data retention time configured as the raw metrics scraped from the Istio Proxies have high cardinality and are not further required to be kept. Instead, the main Prometheus instance scrapes the aggregated metrics through the /federate endpoint.

The Istio-related instance is a Deployment named monitoring-prometheus-istio-server. This instance has a short data retention time and hardcoded configuration that should not be changed. It also has no PersistentVolume attached. This instance never discovers additional metric endpoints from such resources as ServiceMonitors.

See the diagram for a broader view of how the Istio-related instance fits into the monitoring setup in Kyma:

By default, minitoring-prometheus-istio-server is not provided as a data source in Grafana. However, this can be enabled by adding the override:

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: monitoring-overrides
 namespace: kyma-installer
 labels:
   installer: overrides
   component: monitoring
   kyma-project.io/installation: ""
data:
   grafana.dataSources.prometheusIstio.enabled: "true"
EOF

Run the cluster update process. After finishing the upgrade process, restart the Grafana deployment by using this command:

kubectl rollout restart -n kyma-system deployment monitoring-grafana

To configure the Grafana sub-chart, override the default values of its values.yaml file. This document describes parameters that you can set.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

• Helm overrides for Kyma installation
• Sub-charts overrides

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

To configure the Prometheus sub-chart, override the default values of its values.yaml file. This document describes parameters that you can set.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

• Helm overrides for Kyma installation
• Sub-charts overrides

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

This tutorial shows how you can observe your application metrics. Learn how to list all metrics exposed by a sample Go service and watch their changing values by redirecting the metrics port and the default Prometheus server port to the localhost.

This tutorial uses the monitoring-custom-metrics example and one of its services named sample-metrics-8081. The service exposes its metrics on the standard /metrics endpoint that is available under port 8081. You deploy the service (deployment.yaml) along with the ServiceMonitor custom resource (service-monitor.yaml) that instructs Prometheus to scrape metrics:

• From the service with the k8s-app: metrics label
• From the /metrics endpoint
• At 10s interval

This tutorial focuses on the cpu_temperature_celsius metric, that is one of the custom metrics exposed by the sample-metrics-8081 service. Using the metric logic implemented in the example, you can observe how the CPU temperature changes in the range between 60 and 90 degrees Celsius when Prometheus calls the /metrics endpoint.

Prerequisites

To complete the tutorial, you must meet one of these prerequisites and have:

• A cluster with Kyma 1.3 or higher
• Kyma 1.3 or higher installed locally with the Monitoring module

NOTE: The monitoring module is not installed by default as a part of the Kyma Lite package.

Steps

Follow this tutorial to:

• Deploy the sample service with its default configuration.
• Redirect the metrics to the localhost.
• Redirect the metrics to the Prometheus server to observe the metrics in the Prometheus UI.
• Clean up the deployed example.

Deploy the example configuration

Follow these steps:

1. Create the testing-monitoring Namespace.

   Click to copy

   kubectl create namespace testing-monitoring

2. Deploy the sample service in the testing-monitoring Namespace.

   Click to copy

   kubectl create -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/deployment.yaml --namespace=testing-monitoring

3. Deploy the ServiceMonitor custom resource definition (CRD) in the kyma-system Namespace that is a default Namespace for all ServiceMonitor CRDs.

   Click to copy

   kubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/service-monitor.yaml

4. Test your deployment.

   Click to copy

   kubectl get pods -n testing-monitoring

   You should get a result similar to this one:

   Click to copy

   NAME                              READY   STATUS    RESTARTS   AGE
   sample-metrics-6f7c8fcf4b-mlgbx   2/2     Running   0          26m

View metrics on a localhost

Follow these steps:

1. Run the port-forward command on the sample-metrics-8081 service for port 8081 to check the metrics.

   Click to copy

   kubectl port-forward svc/sample-metrics-8081 -n testing-monitoring 8081:8081

2. Open a browser and access http://localhost:8081/metrics.

You can see the cpu_temperature_celsius metric and its current value of 62 on the list of all metrics exposed by the sample-metrics-8081 service.

Thanks to the example logic, the custom metric value changes each time you refresh the localhost address.

View metrics on the Prometheus UI

You can also observe the cpu_temperature_celsius metric in the Prometheus UI and see how its value changes in the pre-defined 10s interval in which Prometheus scrapes the metric values from the service endpoint.

Follow these steps to redirect the metrics:

1. Run the port-forward command on the monitoring-prometheus service.

   Click to copy

   kubectl port-forward svc/monitoring-prometheus -n kyma-system 9090:9090

2. Access the Prometheus UI to see the service endpoint and its details on the Targets list.

   

3. Click the Graph tab, search for the cpu_temperature_celsius metric in the Expression search box, and click the Execute button to check the last value scraped by Prometheus.

   

   The Prometheus UI shows a new value every 10 seconds upon refreshing the page.

Clean up the configuration

When you finish the tutorial, remove the deployed example and all its resources from the cluster.

NOTE: Do not clean up the resources if you want to continue with the next tutorial as these resources are used there as well.

Follow these steps:

1. Remove the deployed ServiceMonitor CRD from the kyma-system Namespace.

   Click to copy

   kubectl delete servicemonitor -l example=monitoring-custom-metrics -n kyma-system

2. Remove the example deployment from the testing-monitoring Namespace.

   Click to copy

   kubectl delete all -l example=monitoring-custom-metrics -n testing-monitoring

3. Remove the testing-monitoring Namespace.

   Click to copy

   kubectl delete namespace testing-monitoring

This tutorial shows how to create and configure a basic Grafana dashboard of a Gauge type. The dashboard shows how the values of the cpu_temperature_celsius metric change in time, representing the current processor temperature ranging from 60 to 90 degrees Celsius. The dashboard shows explicitly when the CPU temperature exceeds the pre-defined threshold of 75 degrees Celsius.

Prerequisites

This tutorial is a follow-up of the Observe application metrics tutorial that uses the monitoring-custom-metrics example. This example deploys the sample-metrics-8081 service which exposes the cpu_temperature_celsius metric. That configuration is required to complete this tutorial.

Steps

Follow these sections to create the Gauge dashboard type for the cpu_temperature_celsius metric.

Create the dashboard

1. Navigate to Grafana. It is available under the https://grafana.{DOMAIN} address, where {DOMAIN} is the domain of your Kyma cluster, such as https://grafana.34.63.57.190.xip.io or https://grafana.example.com/. To access it from the Console UI, click Metrics on the left navigation menu.

   

2. Click the + icon on the left sidebar and select Dashboard from the Create menu.

   

3. Select Add new panel.

   

4. Select Prometheus data source from the Queries to drop-down list and pick the cpu_temperature_celsius metric.

   

5. Toggle the Instant query to be able to retrieve the latest metric value on demand.

   

6. Switch on the right menu to the Panel tab and select in the Visualization section the Gauge dashboard type.

   

7. Click the save button in the top right corner of the page to save the changes. Provide a name for the dashboard.

   

Configure the dashboard

1. To edit the dashboard settings, go to the Panel Title options and select Edit.

   

2. Back in the right menu switch to the Field tab and go to the Standard options section, set up the measuring unit to Celsius degrees to reflect the metric data type.

   

3. Set the minimum metric value to 60 and the maximum value to 90 to reflect the cpu_temperature_celsius metric value range.

   

4. Set a red color threshold to 75 for the dashboard to turn red once the CPU temperature reaches and exceeds this value.

   

5. Go to the Panel tab and give a title to the dashboard.

   

6. Enable the Labels option to display this range on the dashboard.

   

7. Click the save button in the top right corner of the page to save the changes. Add an optional note to describe the changes made.

   

Verify the dashboard

Refresh the browser to see how the dashboard changes according to the current value of the cpu_temperature_celsius metric.

• It turns green if the current metric value ranges from 60 to 74 degrees Celsius:

  

• It turns red if the current metric value ranges from 75 to 90 degrees Celsius:

  

NOTE: You can also define the dashboard's ConfigMap and add it to the resources folder under the given component's chart. To make the dashboard visible, simply use the kubectl apply command to deploy it. For details on adding monitoring to components, see the README.md document.

This tutorial shows you how to define alerting rules to monitor the health status of your resources. In this example, you will write an alerting rule based on the cpu_temperature_celsius metric. The alert defined in the rule will fire whenever the CPU temperature is equal to or greater than 75 degrees Celsius.

Prerequisites

This tutorial is a follow-up of the observe application metrics tutorial that uses the monitoring-custom-metrics example. Follow this tutorial to deploy the sample-metrics-8081 service which exposes the cpu_temperature_celsius metric. That configuration is required to complete this tutorial.

Steps

Follow these steps to create an alerting rule:

1. Create the PrometheusRule resource holding the configuration of your alerting rule.

   NOTE: Prometheus requires specific labels to identify PrometheusRule definitions. Make sure you set app and release to monitoring.

   Click to copy

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: cpu.temp.rules
     namespace: kyma-system
     labels:
       app: monitoring
       release: monitoring
   spec:
     groups:
     - name: cpu.temp.rules
       rules:
       - alert: CPUTempHigh
         expr: cpu_temperature_celsius >= 75
         for: 10s
         labels:
           severity: critical
         annotations:
           description: "CPU temperature is equal to or greater than 75 degrees Celsius"
           summary: "CPU temperature is too high"

   Configure your alert rule using the following parameters:

   Parameter	Description	Example value
   groups.name	Specifies the name of the group listing the rules.	cpu.temp.rules
   rules.alert	Specifies the name of the alert.	CPUTempHigh
   rules.expr	A PromQL expression which specifies the conditions that must be met for the alarm to fire. Specify the expression using Kubernetes Functions and metrics.	cpu_temperature_celsius >= 75
   rules.for	Specifies the time period between encountering an active alert for the first time during rule evaluation and firing the alert.	10s
   rules.labels.severity	Specifies the severity of the alert.	critical
   rules.annotations.description	Provides the alert details.	CPU temperature is equal to or greater than 75 degrees Celsius
   rules.annotations.summary	Provides a short alert summary.	CPU temperature is too high

   For more details on defining alerting rules, see the Prometheus documentation.

2. Deploy the alerting rule:

   Click to copy

   kubectl apply -f {FILE_NAME}.yaml

3. Run the port-forward command on the monitoring-prometheus service to access the Prometheus dashboard:

   Click to copy

   kubectl port-forward svc/monitoring-prometheus -n kyma-system 9090:9090

4. Go to http://localhost:9090/rules to view the rule in the dashboard.

   

5. Go to http://localhost:9090/alerts to see if the alert fires appropriately.

   

This tutorial shows you how to configure Alertmanager to send notifications. Alertmanager supports several notification receivers, but this tutorial only focuses on sending notifications to Slack.

Prerequisites

This tutorial is a follow-up of the Observe application metrics and the Define alerting rules tutorials that use the monitoring-custom-metrics example. Follow this tutorial to deploy the sample-metrics-8081 service which exposes the cpu_temperature_celsius metric and creates an alert based on it. That configuration is required to complete this tutorial.

Steps

Follow these steps to configure notifications for Slack every time Alertmanager triggers and resolves the CPUTempHigh alert.

1. Install the Incoming WebHooks application using Slack App Directory.

   NOTE: The approval of your Slack workspace administrator may be necessary to install the application.

2. Configure the application to receive notifications coming from third-party services. Read the instructions to find out how to set up the configuration for Slack.

   The integration settings should look similar to the following:

   

3. Override Alertmanager configuration. The configuration for notification receivers is located in the template. By default, it contains settings for VictorOps and Slack. Define a Secret to override default values used by the chart.

   Click to copy

   apiVersion: v1
   kind: Secret
   metadata:
    name: monitoring-config-overrides
    namespace: kyma-installer
    labels:
       kyma-project.io/installation: ""
       installer: overrides
       component: monitoring
   type: Opaque
   stringData:
       global.alertTools.credentials.slack.channel: "{CHANNEL_NAME}"
       global.alertTools.credentials.slack.apiurl: "{WEBHOOK_URL}"

   Use the following parameters:

   Parameter	Description
   global.alertTools.credentials.slack.channel	Specifies the Slack channel which receives notifications on new alerts, such as test-monitoring-alerts.
   global.alertTools.credentials.slack.apiurl	Specifies the URL endpoint which sends alerts triggered by Prometheus rules. The Incoming WebHooks application provides you with the Webhook URL, such as https://hooks.slack.com/services/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, that you can paste in this configuration.

   For details on Alertmanager chart configuration and parameters, see the configuration document.

4. Deploy the Secret. Use this command:

   Click to copy

   kubectl apply -f {FILE_NAME}.yaml

5. Proceed with Kyma installation.

   NOTE: If you add the overrides in the runtime, trigger the update process using Kyma CLI. Provide the same version of the installed Kyma:

   Click to copy

   kyma upgrade -s {VERSION}

   NOTE: If the rule you created is removed during the update, re-apply it following the Define alerting rules tutorial.

6. Verify if your Slack channel receives alert notifications about firing and resolved alerts. See the example: