Monitoring
Overview
Kyma comes bundled with third-party applications like Prometheus, Alertmanager, and Grafana, that offer a monitoring functionality for all Kyma resources. These applications are deployed during the Kyma cluster installation, along with a set of pre-defined alerting rules, Grafana dashboards, and Prometheus configuration.
The whole installation package provides the end-to-end Kubernetes cluster monitoring that allows you to:
- View metrics exposed by the Pods.
- Use the metrics to create descriptive dashboards that monitor any Pod anomalies.
- Manage the default alert rules and create new ones.
- Set up channels for notifications informing of any detected alerts.
NOTE: The monitoring component is available by default in the cluster installation, but disabled in the Kyma Lite local installation on Minikube. Enable the component to install it with the local profile.
You can install Monitoring as part of Kyma predefined profiles. For production purposes, use the production profile which ensures:
- Increased retention time to prevent data loss in case of prolonged troubleshooting.
- Increased memory and CPU values that optimize the performance of your component.
Architecture
Before you go into component details, find out more about the end-to-end monitoring flow in Kyma.
End-to-end monitoring flow
The complete monitoring flow in Kyma comes down to these components and steps:
- Upon Kyma installation on a cluster, Prometheus Operator creates a Prometheus instance with the default configuration.
- The Prometheus server periodically polls all metrics exposed on
/metrics
endpoints of ports specified in ServiceMonitor CRDs. Prometheus stores these metrics in a time-series database. - If Prometheus detects any metric values matching the logic of alerting rules, it triggers the alerts and passes them to Alertmanager.
- If you manually configure a notification channel, you can instantly receive detailed information on metric alerts detected by Prometheus.
- You can visualize metrics and track their historical data on Grafana dashboards.
Monitoring components
The diagram presents monitoring components and the way they interact with one another.
Prometheus Operator creates a Prometheus instance, manages its deployment, and provides configuration for it. It also deploys Alertmanager and operates ServiceMonitor custom resources that specify monitoring definitions for groups of services.
Prometheus collects metrics from Pods. Metrics are the time-stamped data that provide information on the running jobs, workload, CPU consumption, memory usage, and more. To obtain such metrics, Prometheus uses the kube-state-metrics service. It generates the metrics from Kubernetes API objects and exposes them on the
/metrics
HTTP endpoint.
Pods can also contain applications with custom metrics, such as the total storage space available in the MinIO server. Prometheus stores this polled data in a time-series database (TSDB) and runs rules over them to generate alerts if it detects any metric anomalies. It also scrapes metrics provided by Node Exporter which exposes existing hardware metrics from external systems as Prometheus metrics.NOTE: Besides this main Prometheus instance, there is a second Prometheus instance running in the
kyma-system
Namespace. This second instance is responsible for collecting and aggregating Istio Service Mesh metrics.ServiceMonitors monitor services and specify the endpoints from which Prometheus should poll the metrics. Even if you expose a handful of metrics in your application, Prometheus polls only those from the
/metrics
endpoints of ports specified in ServiceMonitor CRDs.Alertmanager receives alerts from Prometheus and forwards this data to configured Slack or Victor Ops channels. You can use PrometheusRules to define alert conditions for metrics. Kyma provides a set of out-of-the-box alerting rules that are passed from Prometheus to Alertmanager. The definitions of such rules specify the alert logic, the value at which alerts are triggered, the alerts' severity, and more.
NOTE: There are no notification channels configured in the default monitoring installation. The current configuration allows you to add either Slack or Victor Ops channels.
Grafana provides a dashboard and a graph editor to visualize metrics collected from the Prometheus API. Grafana uses the query language called PromQL to select and aggregate metrics data from the Prometheus database. To access the Grafana UI, use the
https://grafana.{DOMAIN}
address, where{DOMAIN}
is the domain of your Kyma cluster.
Details
Alertmanager
Alertmanager receives and manages alerts coming from Prometheus. It can then forward the notifications about fired alerts to specific channels, such as Slack or VictorOps.
Alertmanager configuration
Use the following files to configure and manage Alertmanager:
alertmanager.yaml
which deploys the Alertmanager Pod.values.yaml
which you can use to define core Alertmanager configuration and alerting channels. For details on configuration elements, see the Prometheus documentation.
Alerting rules
Kyma comes with a set of alerting rules provided out of the box. These rules provide alerting configuration for logging, web apps, rest services, and custom Kyma rules. You can also define your own alerting rule. To learn how, see the tutorial.
Istio monitoring
The monitoring chart is pre-configured to collect all metrics relevant for observing the in-cluster Istio Service Mesh, including the proxy-level, service-level, and control-plane metrics.
The concept of collecting the service-level metrics is built around the Istio Proxy implemented by Envoy. Istio Proxy collects all communication details inside the service mesh in a decentralized way. After scraping these high cardinality metrics from the envoys, the metrics need to be additionally aggregated on a service level to get the final service-related details.
Following the Istio's observability best practice, the scraping and aggregation of the service-level metrics is done in a dedicated Prometheus instance. That instance has the smallest possible data retention time configured as the raw metrics scraped from the Istio Proxies have high cardinality and are not further required to be kept. Instead, the main Prometheus instance scrapes the aggregated metrics through the /federate
endpoint.
The Istio-related instance is a Deployment named monitoring-prometheus-istio-server
. This instance has a short data retention time and hardcoded configuration that should not be changed. It also has no PersistentVolume attached. This instance never discovers additional metric endpoints from such resources as ServiceMonitors.
See the diagram for a broader view of how the Istio-related instance fits into the monitoring setup in Kyma:
By default, minitoring-prometheus-istio-server
is not provided as a data source in Grafana. However, this can be enabled by adding the override:
cat <<EOF | kubectl apply -f ----apiVersion: v1kind: ConfigMapmetadata: name: monitoring-overrides namespace: kyma-installer labels: installer: overrides component: monitoring kyma-project.io/installation: ""data: grafana.dataSources.prometheusIstio.enabled: "true"EOF
Run the cluster update process. After finishing the upgrade process, restart the Grafana deployment by using this command:
kubectl rollout restart -n kyma-system deployment monitoring-grafana
Configuration
Alertmanager sub-chart
To configure the Alertmanager sub-chart, override the default values of its values.yaml
file. This document describes parameters that you can set.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
Configurable parameters
This table lists the configurable parameters, their descriptions, and default values:
Parameter | Description | Default value |
---|---|---|
global.alertTools.credentials.slack.apiurl | Specifies the URL endpoint which sends alerts triggered by Prometheus rules. | None |
global.alertTools.credentials.slack.channel | Refers to the Slack channel which receives notifications on new alerts. | None |
global.alertTools.credentials.slack.matchExpression | Notifications will be sent only for those alerts whose labels match the specified expression. | "severity: critical" |
global.alertTools.credentials.slack.sendResolved | Specifies whether or not to notify about resolved alerts. | true |
global.alertTools.credentials.victorOps.routingkey | Defines the team routing key in VictorOps. | None |
global.alertTools.credentials.victorOps.apikey | Defines the team API key in VictorOps. | None |
global.alertTools.credentials.victorOps.matchExpression | Notifications will be sent only for those alerts whose labels match the specified expression. | "severity: critical" |
global.alertTools.credentials.victorOps.sendResolved | Specifies whether or not to notify about resolved alerts. | true |
NOTE: Override all configurable values for the Alertmanager sub-chart using Secrets (
kind: Secret
).
Grafana sub-chart
To configure the Grafana sub-chart, override the default values of its values.yaml
file. This document describes parameters that you can set.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
Configurable parameters
This table lists the configurable parameters, their descriptions, and default values:
Parameter | Description | Default value |
---|---|---|
env.GF_USERS_DEFAULT_THEME | Specifies the background color of the Grafana UI. You can change it to dark . | light |
env.GF_AUTH_GENERIC_OAUTH_ENABLED | Enables the generic OAuth plugin for Grafana that is already pre-configured based on the in-cluster Dex setup. | true |
env.GF_USERS_AUTO_ASSIGN_ORG_ROLE | Specifies the automatically assigned user role for a user already authenticated by Grafana. You can change the value to Viewer or Admin . | Editor |
env.GF_AUTH_ANONYMOUS_ENABLED | Enables anonymous login to Grafana. | false |
env.GF_AUTH_ANONYMOUS_ORG_ROLE | Specifies the automatically assigned user role for an anonymous user. You can change the value to Viewer or Admin . | Editor |
env.GF_LOG_LEVEL | Specifies the log level used by Grafana. Be aware that logs at the info level may print logins, which can potentially be users' email addresses. | warn |
persistence.enabled | Specifies whether the user and dashboard data used by Grafana is durably persisted. If enabled, the Grafana database will be mounted to a PersistentVolume and survive restarts. If you use Grafana in a high-available setup using an external database, ensure that this flag is set to false . | true |
env.GF_ALERTING_ENABLED | Enables the Grafana's alerting feature. | true |
env.GF_DASHBOARDS_MIN_REFRESH_INTERVAL | Specifies the minimum refresh interval for dashboards to prevent using a lower value. Use the official syntax to set the value. | 10s |
Prometheus sub-chart
To configure the Prometheus sub-chart, override the default values of its values.yaml
file. This document describes parameters that you can set.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
Configurable parameters
This table lists the configurable parameters, their descriptions, and default values:
Parameter | Description | Default value |
---|---|---|
prometheusSpec.retention | Specifies a period for which Prometheus stores the metrics. | 1d |
prometheusSpec.retentionSize | Specifies the maximum number of bytes that storage blocks can use. The oldest data will be removed first. | 2GB |
prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage | Specifies the size of a PersistentVolumeClaim (PVC). | 10Gi |
Tutorials
Overview
The set of monitoring tutorials you are about to read describes the complete monitoring flow for your services in Kyma. Going through the tutorials, you get the gist of Kyma built-in monitoring applications, such as Prometheus, Grafana, and Alertmanager. This hands-on experience with monitoring helps you understand how and where you can observe and visualize your service metrics to monitor them for any alerting values.
All the tutorials use the monitoring-custom-metrics
example and one of its services called sample-metrics-8081
. This service exposes the cpu_temperature_celsius
custom metric on the /metrics
endpoint. This custom metric is the central element of the whole tutorial set. The metric value simulates the current processor temperature and changes randomly from 60 to 90 degrees Celsius. The alerting threshold in these tutorials is 75 degrees Celsius. If the temperature exceeds this value, the Grafana dashboard, Prometheus rule, and Alertmanager notifications you create inform you about this.
The tutorial set consists of these documents:
Observe application metrics in which you redirect the
cpu_temperature_celsius
metric to the localhost and the Prometheus UI. You later observe how the metric value changes in the predefined 10 seconds interval in which Prometheus scrapes the metric values from the service's/metrics
endpoint.Create a Grafana dashboard in which you create a Grafana dashboard of a Gauge type for the
cpu_temperature_celsius
metric. This dashboard shows explicitly when the CPU temperature is equal to or higher than the predefined threshold of 75 degrees Celsius, at which point the dashboard turns red.Define alerting rules in which you define the
CPUTempHigh
alerting rule by creating a PrometheusRule resource. Prometheus accesses the/metrics
endpoint every 10 seconds and validates the current value of thecpu_temperature_celsius
metric. If the value is equal to or higher than 75 degrees Celsius, Prometheus waits for 10 seconds to recheck it. If the value still exceeds the threshold, Prometheus triggers the rule. You can observe both the rule and the alert it generates on the Prometheus dashboard.Send notifications to Slack in which you configure Alertmanager to send notifications on Prometheus alerts to a Slack channel. This way, whenever Prometheus triggers or resolves the
CPUTempHigh
alert, Alertmanager sends a notification to thetest-monitoring-alerts
Slack channel defined for the tutorial.
See the diagram for an overview of the purpose of the tutorials and the tools used in them:
Observe application metrics
This tutorial shows how you can observe your application metrics. Learn how to list all metrics exposed by a sample Go service and watch their changing values by redirecting the metrics port and the default Prometheus server port to the localhost.
This tutorial uses the monitoring-custom-metrics
example and one of its services named sample-metrics-8081
. The service exposes its metrics on the standard /metrics
endpoint that is available under port 8081
. You deploy the service (deployment.yaml
) along with the ServiceMonitor custom resource (service-monitor.yaml
) that instructs Prometheus to scrape metrics:
- From the service with the
k8s-app: metrics
label - From the
/metrics
endpoint - At
10s
interval
This tutorial focuses on the cpu_temperature_celsius
metric, that is one of the custom metrics exposed by the sample-metrics-8081
service. Using the metric logic implemented in the example, you can observe how the CPU temperature changes in the range between 60 and 90 degrees Celsius when Prometheus calls the /metrics
endpoint.
Prerequisites
To complete the tutorial, you must meet one of these prerequisites and have:
- A cluster with Kyma 1.3 or higher
- Kyma 1.3 or higher installed locally with the Monitoring module
NOTE: The monitoring module is not installed by default as a part of the Kyma Lite package.
Steps
Follow this tutorial to:
- Deploy the sample service with its default configuration.
- Redirect the metrics to the localhost.
- Redirect the metrics to the Prometheus server to observe the metrics in the Prometheus UI.
- Clean up the deployed example.
Deploy the example configuration
Follow these steps:
Create the
testing-monitoring
Namespace.Click to copykubectl create namespace testing-monitoringDeploy the sample service in the
testing-monitoring
Namespace.Click to copykubectl create -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/deployment.yaml --namespace=testing-monitoringDeploy the ServiceMonitor custom resource definition (CRD) in the
kyma-system
Namespace that is a default Namespace for all ServiceMonitor CRDs.Click to copykubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/service-monitor.yamlTest your deployment.
Click to copykubectl get pods -n testing-monitoringYou should get a result similar to this one:
Click to copyNAME READY STATUS RESTARTS AGEsample-metrics-6f7c8fcf4b-mlgbx 2/2 Running 0 26m
View metrics on a localhost
Follow these steps:
Run the
port-forward
command on thesample-metrics-8081
service for port8081
to check the metrics.Click to copykubectl port-forward svc/sample-metrics-8081 -n testing-monitoring 8081:8081Open a browser and access
http://localhost:8081/metrics
.
You can see the cpu_temperature_celsius
metric and its current value of 62
on the list of all metrics exposed by the sample-metrics-8081
service.
Thanks to the example logic, the custom metric value changes each time you refresh the localhost address.
View metrics on the Prometheus UI
You can also observe the cpu_temperature_celsius
metric in the Prometheus UI and see how its value changes in the pre-defined 10s
interval in which Prometheus scrapes the metric values from the service endpoint.
Follow these steps to redirect the metrics:
Run the
port-forward
command on themonitoring-prometheus
service.Click to copykubectl port-forward svc/monitoring-prometheus -n kyma-system 9090:9090Access the Prometheus UI to see the service endpoint and its details on the Targets list.
Click the Graph tab, search for the
cpu_temperature_celsius
metric in the Expression search box, and click the Execute button to check the last value scraped by Prometheus.The Prometheus UI shows a new value every 10 seconds upon refreshing the page.
Clean up the configuration
When you finish the tutorial, remove the deployed example and all its resources from the cluster.
NOTE: Do not clean up the resources if you want to continue with the next tutorial as these resources are used there as well.
Follow these steps:
Remove the deployed ServiceMonitor CRD from the
kyma-system
Namespace.Click to copykubectl delete servicemonitor -l example=monitoring-custom-metrics -n kyma-systemRemove the example deployment from the
testing-monitoring
Namespace.Click to copykubectl delete all -l example=monitoring-custom-metrics -n testing-monitoringRemove the
testing-monitoring
Namespace.Click to copykubectl delete namespace testing-monitoring
Create a Grafana dashboard
This tutorial shows how to create and configure a basic Grafana dashboard of a Gauge type. The dashboard shows how the values of the cpu_temperature_celsius
metric change in time, representing the current processor temperature ranging from 60 to 90 degrees Celsius. The dashboard shows explicitly when the CPU temperature exceeds the pre-defined threshold of 75 degrees Celsius.
Prerequisites
This tutorial is a follow-up of the Observe application metrics tutorial that uses the monitoring-custom-metrics
example. This example deploys the sample-metrics-8081
service which exposes the cpu_temperature_celsius
metric. That configuration is required to complete this tutorial.
Steps
Follow these sections to create the Gauge dashboard type for the cpu_temperature_celsius
metric.
Create the dashboard
Navigate to Grafana. It is available under the
https://grafana.{DOMAIN}
address, where{DOMAIN}
is the domain of your Kyma cluster, such ashttps://grafana.34.63.57.190.xip.io
orhttps://grafana.example.com/
. To access it from the Console UI, click Metrics on the left navigation menu.Click the + icon on the left sidebar and select Dashboard from the Create menu.
Select Add new panel.
Select Prometheus data source from the Queries to drop-down list and pick the
cpu_temperature_celsius
metric.Toggle the Instant query to be able to retrieve the latest metric value on demand.
Switch on the right menu to the Panel tab and select in the Visualization section the Gauge dashboard type.
Click the save button in the top right corner of the page to save the changes. Provide a name for the dashboard.
Configure the dashboard
To edit the dashboard settings, go to the Panel Title options and select Edit.
Back in the right menu switch to the Field tab and go to the Standard options section, set up the measuring unit to Celsius degrees to reflect the metric data type.
Set the minimum metric value to
60
and the maximum value to90
to reflect thecpu_temperature_celsius
metric value range.Set a red color threshold to
75
for the dashboard to turn red once the CPU temperature reaches and exceeds this value.Go to the Panel tab and give a title to the dashboard.
Enable the Labels option to display this range on the dashboard.
Click the save button in the top right corner of the page to save the changes. Add an optional note to describe the changes made.
Verify the dashboard
Refresh the browser to see how the dashboard changes according to the current value of the cpu_temperature_celsius
metric.
It turns green if the current metric value ranges from 60 to 74 degrees Celsius:
It turns red if the current metric value ranges from 75 to 90 degrees Celsius:
NOTE: You can also define the dashboard's ConfigMap and add it to the
resources
folder under the given component's chart. To make the dashboard visible, simply use thekubectl apply
command to deploy it. For details on adding monitoring to components, see theREADME.md
document.
Define alerting rules
This tutorial shows you how to define alerting rules to monitor the health status of your resources. In this example, you will write an alerting rule based on the cpu_temperature_celsius
metric. The alert defined in the rule will fire whenever the CPU temperature is equal to or greater than 75 degrees Celsius.
Prerequisites
This tutorial is a follow-up of the observe application metrics tutorial that uses the monitoring-custom-metrics
example. Follow this tutorial to deploy the sample-metrics-8081
service which exposes the cpu_temperature_celsius
metric. That configuration is required to complete this tutorial.
Steps
Follow these steps to create an alerting rule:
Create the PrometheusRule resource holding the configuration of your alerting rule.
NOTE: Prometheus requires specific labels to identify PrometheusRule definitions. Make sure you set app and release to
monitoring
.Click to copyapiVersion: monitoring.coreos.com/v1kind: PrometheusRulemetadata:name: cpu.temp.rulesnamespace: kyma-systemlabels:app: monitoringrelease: monitoringspec:groups:- name: cpu.temp.rulesrules:- alert: CPUTempHighexpr: cpu_temperature_celsius >= 75for: 10slabels:severity: criticalannotations:description: "CPU temperature is equal to or greater than 75 degrees Celsius"summary: "CPU temperature is too high"Configure your alert rule using the following parameters:
Parameter Description Example value groups.name Specifies the name of the group listing the rules. cpu.temp.rules
rules.alert Specifies the name of the alert. CPUTempHigh
rules.expr A PromQL expression which specifies the conditions that must be met for the alarm to fire. Specify the expression using Kubernetes Functions and metrics. cpu_temperature_celsius >= 75
rules.for Specifies the time period between encountering an active alert for the first time during rule evaluation and firing the alert. 10s
rules.labels.severity Specifies the severity of the alert. critical
rules.annotations.description Provides the alert details. CPU temperature is equal to or greater than 75 degrees Celsius
rules.annotations.summary Provides a short alert summary. CPU temperature is too high
For more details on defining alerting rules, see the Prometheus documentation.
Deploy the alerting rule:
Click to copykubectl apply -f {FILE_NAME}.yamlRun the
port-forward
command on themonitoring-prometheus
service to access the Prometheus dashboard:Click to copykubectl port-forward svc/monitoring-prometheus -n kyma-system 9090:9090Go to
http://localhost:9090/rules
to view the rule in the dashboard.Go to
http://localhost:9090/alerts
to see if the alert fires appropriately.
Send notifications to Slack
This tutorial shows you how to configure Alertmanager to send notifications. Alertmanager supports several notification receivers, but this tutorial only focuses on sending notifications to Slack.
Prerequisites
This tutorial is a follow-up of the Observe application metrics and the Define alerting rules tutorials that use the monitoring-custom-metrics
example. Follow this tutorial to deploy the sample-metrics-8081
service which exposes the cpu_temperature_celsius
metric and creates an alert based on it. That configuration is required to complete this tutorial.
Steps
Follow these steps to configure notifications for Slack every time Alertmanager triggers and resolves the CPUTempHigh
alert.
Install the Incoming WebHooks application using Slack App Directory.
NOTE: The approval of your Slack workspace administrator may be necessary to install the application.
Configure the application to receive notifications coming from third-party services. Read the instructions to find out how to set up the configuration for Slack.
The integration settings should look similar to the following:
Override Alertmanager configuration. The configuration for notification receivers is located in the template. By default, it contains settings for VictorOps and Slack. Define a Secret to override default values used by the chart.
Click to copyapiVersion: v1kind: Secretmetadata:name: monitoring-config-overridesnamespace: kyma-installerlabels:kyma-project.io/installation: ""installer: overridescomponent: monitoringtype: OpaquestringData:global.alertTools.credentials.slack.channel: "{CHANNEL_NAME}"global.alertTools.credentials.slack.apiurl: "{WEBHOOK_URL}"Use the following parameters:
Parameter Description global.alertTools.credentials.slack.channel Specifies the Slack channel which receives notifications on new alerts, such as test-monitoring-alerts
.global.alertTools.credentials.slack.apiurl Specifies the URL endpoint which sends alerts triggered by Prometheus rules. The Incoming WebHooks application provides you with the Webhook URL, such as https://hooks.slack.com/services/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
, that you can paste in this configuration.For details on Alertmanager chart configuration and parameters, see the configuration document.
Deploy the Secret. Use this command:
Click to copykubectl apply -f {FILE_NAME}.yamlProceed with Kyma installation.
NOTE: If you add the overrides in the runtime, trigger the update process using Kyma CLI. Provide the same version of the installed Kyma:
Click to copykyma upgrade -s {VERSION}NOTE: If the rule you created is removed during the update, re-apply it following the Define alerting rules tutorial.
Verify if your Slack channel receives alert notifications about firing and resolved alerts. See the example:
Troubleshooting
Prometheus Istio Server restarting or in crashback loop
Prometheus Istio Server scrapes metrics from all envoy side cars. It might crash because of OOM when the cardinality of the Istio metrics increases too much. This usually happens when a high amount of workloads perform a lot of communication to other workloads. This issue can be tackled by following these steps:
- Increase the memory for
prometheus-istio-server
:Increase the limits for memory:Click to copykubect edit deployment -n kyma monitoring-prometheus-istio-serverClick to copyresources:limits:cpu: 600mmemory: 2000Mirequests:cpu: 40mmemory: 200Mi - Drop labels for the Istio metrics. Edit the ConfigMap for
prometheus-istio server
:Edit:Click to copykubectl edit configmap -n kyma-system monitoring-prometheus-istio-serverChange regex to:Click to copymetric_relabel_configs:- separator: ;regex: ^(grpc_response_status|source_version|destination_version|source_app|destination_app)$replacement: $1action: labeldropSave the ConfigMap and restartClick to copyregex: ^(grpc_response_status|source_version|source_principal|source_app|response_flags|request_protocol|destination_version|destination_principal|destination_app|destination_canonical_service|destination_canonical_revision|source_canonical_revision|source_canonical_service)$prometheus-istio-server
for the changes to take effect:Click to copykubectl rollout restart deployment -n kyma-system monitoring-prometheus-istio-serverCAUTION: The side effect of this change is graphs in Kiali will not work.
Create an override
Follow these steps to override the existing configuration
- Add and apply a ConfigMap in the
kyma-installer
Namespace in which you set the value for the memory limit attribute to 4Gi and/or drop the labels from Istio metrics.
cat <<EOF | kubectl apply -f -apiVersion: v1kind: ConfigMapmetadata: name: monitoring-overrides namespace: kyma-installer labels: installer: overrides component: monitoring kyma-project.io/installation: ""data: prometheus-istio.envoyStats.labeldropRegex: "^(grpc_response_status|source_version|source_principal|source_app|response_flags|request_protocol|destination_version|destination_principal|destination_app|destination_canonical_service|destination_canonical_revision|source_canonical_revision|source_canonical_service)$" prometheus-istio.server.resources.limits.memory: "4Gi"EOF
CAUTION: The side effect of the change to
prometheus-istio.envoyStats.labeldropRegex
(to drop additional labels ) is graphs in Kiali will not work.