Monitoring

Overview

To enrich Kyma with monitoring functionality, third-party resources come by default as packaged tools. The kube-prometheus package is a Prometheus operator from CoreOS responsible for delivering these tools. Monitoring in Kyma includes three primary elements:

  • Prometheus, an open-source system monitoring toolkit.
  • Grafana, a user interface that allows you to query and visualize statistics and metrics.
  • AlertManager, a Prometheus component that handles alerts that originate from Prometheus. AlertManager performs needed deduplicating, grouping, and routing based on rules defined by the Prometheus server.

Convenience and efficiency are the main advantages to using the kube-prometheus package. kube-prometheus delivers a level of monitoring options that would otherwise involve extensive development effort to acquire. Prometheus, Grafana, and AlertManager installed on their own would require the developer to perform customization to achieve the same results as the operator alone. kube-prometheus is configured to run on Kubernetes and monitor clusters without additional configuration.

Architecture

This document outlines the monitoring architecture of Kyma, highlighting information sources that Prometheus polls for data to process.

Monitoring architecture in Kyma

The Prometheus Operator

The Prometheus Operator is a CoreOS component integrated into Kyma that enables Prometheus deployments to be decoupled from the configuration of the entities they monitor. The task of the Operator is to ensure that Prometheus servers with the specified configuration are always running. If the developer does not specify a configuration for Prometheus instances, the Operator is able to generate and deploy one. The Prometheus instance is responsible for the monitoring of services.

The Service Monitor

The Service Monitor works in orchestration with the Prometheus resource that the Operator watches. It dictates to a Prometheus resource how to retrieve metrics and enables exposure of those metrics in a standard manner. It also specifies services the Prometheus instance should monitor. Using labels, the Prometheus resource includes a Service Monitor.

Monitored Data sources

Prometheus contains the flexibility to poll data from a variety of sources. Virtual machines on which Kubernetes runs make time-stamped data available, reporting on jobs started, workload, CPU performance, capacity, and more. In this case, the Service Monitor watches the Kubernetes API master to detect any job creation. The job produces time-stamped data that Prometheus consumes.

Pods may contain applications with custom metrics that Prometheus can poll through the Prometheus exporter.

Grafana

Kyma employs Grafana as a third-party resource in kube-prometheus to deliver a feature-rich metrics dashboard and graph editor.

To access the Grafana UI, use the following URL: https://grafana.{DOMAIN}. Replace DOMAIN with the domain of your Kyma cluster.

Alertmanager

Alertmanager receives harvested metrics from Prometheus and forwards this data on to the configured channels, such as email or incident management systems.

Expose custom metrics in Kyma

This Getting Started guide shows how to expose custom metrics to Prometheus with a Golang service in Kyma. To do so, follow these steps:

  1. Configure Istio.
  2. Expose a sample application serving metrics on 8081 port.
  3. Access the exposed metrics in Prometheus.

Prerequisites

  • Kyma as the target deployment environment
  • Istio 0.8
    • sidecar injection enabled
    • mutual TLS enabled

Installation

Configure Istio

For the default Namespace, the sidecar injection must be enabled. To enable the sidecar injection for all Pods in the default Namespace, run the following command:

kubectl label namespace default istio-injection=enabled
namespace "default" labeled

For more details on deploying your application with Istio, read this documentation.

You must also add the sidecar.istio.io/inject annotation with the value set to true to the Pod template specification, to enable the injection as shown in this example.

spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"

For more details on installing the Istio sidecar, read this documentation.

The following ports are used in the Pod:

  • 8080 - Envoy captures the traffic only for ports listed in Pod's containerPorts (containerPort: 8080), or in the traffic.sidecar.istio.io/includeInboundPorts annotation. Thus, this port is a part of the Service Mesh and can be used for application's needs.

  • 8081 - This is the excluded port from the Service Mesh, which is used for exposing metrics only. The network traffic bypasses Envoy and goes straight to the container. In Kyma, use the suggested 8081 port to expose metrics.

Expose a sample metrics application

To expose Prometheus metrics in Golang, the Prometheus community provides this library.

This is a basic example where Gauge and Counter metrics are exported using the prometheus package.

  1. Deploy the sample metrics application.

     kubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/deployment.yaml
    
     kubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/service-8080.yaml
    
     kubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/service-8081.yaml
    
     kubectl apply -f https://raw.githubusercontent.com/kyma-project/examples/master/monitoring-custom-metrics/deployment/service-monitor.yaml
    
     kubectl get pods
     NAME                             READY     STATUS    RESTARTS   AGE
     sample-metrics-c9f998959-jd2fz   2/2       Running   0          2m
     sample-metrics-c9f998959-kfbp8   2/2       Running   0          2m
     sample-metrics-c9f998959-nnp2n   2/2       Running   0          2m
     sample-metrics-c9f998959-vdnkn   2/2       Running   0          2m
    
  2. Run the port-forward command on the sample-metrics-8081 service for the8081 port to check the metrics.

     kubectl port-forward svc/sample-metrics-8081 8081:8081
    

    Open a browser and access http://localhost:8081/metrics

    metrics on port 8081

Find the source code for the sample application here. See the package prometheus for the reference documentation. Read this documentation to learn more about the Prometheus metric types.

Access the exposed metrics in Prometheus

Run the port-forward command on the core-prometheus service:

kubectl port-forward svc/core-prometheus -n kyma-system 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

All the sample-metrics endpoints appear as the Targets list.

Prometheus Dashboard

Use either the cpu_temperature_celsius or hd_errors_total in the expression field. Click the Execute button to check the values scrapped by Prometheus.

Prometheus Dashboard

ServiceMonitor

Prometheus can reach the service using ServiceMonitor. ServiceMonitor is a specific CRD used by the Prometheus operator to monitor services.

In Kyma, the Prometheus server discovers all ServiceMonitors through the serviceMonitorSelector matching the prometheus: core label.

  serviceMonitorSelector:
    matchLabels:
      prometheus: {{ .Values.prometheusLabelValue | default .Release.Name | quote }}
{{- end }}

In this example, the ServiceMonitor selects a selector with all services matching the k8s-app: metrics label. Find the complete yaml here.

In Kyma, there is a template which serves to discover a list of ServiceMonitors.

Add a Custom Dashboard in Grafana

Out of the box, Kyma includes a set of dashboards. The users can create their own Grafana Dashboard by using the Grafana UI. The dashboards persist even after the Pod restarts.

For details on how to create dashboards in Grafana, see the following documents:

Cleanup

Run the following commands to completely remove the example and all its resources from the cluster:

  1. Remove the istio-injection label from the default Namespace.
     kubectl label namespace default istio-injection-
    
  2. Remove ServiceMonitor in the kyma-system Namespace.
     kubectl delete servicemonitor -l example=monitoring-custom-metrics -n kyma-system
    
  3. Remove the sample-metrics Deployments in the default Namespace. `bash kubectl delete all -l example=monitoring-custom-metrics