Logging in Kyma uses Loki which is a Prometheus-like log management system. This lightweight solution, integrated with Grafana, is easy to understand and operate. The main elements of the logging stack include:

• The Agent acting as a log router for Docker containers. It runs inside Docker, checks each container, and routes the logs to the log management system. Currently, Kyma supports the Fluent Bit log collector.
• Loki main server which stores logs and processes queries.
• Grafana logging and metrics platform used for quering and displaying logs.

NOTE: At the moment, Kyma provides an alpha version of the Logging component. The default Loki Pod log tailing configuration does not work with Kubernetes version 1.14 (for GKE version 1.12.6-gke.X) and above. For setup and preparation of deployment see the cluster installation guide.

CAUTION: Loki is designed for application logging. Do not log any sensitive information, such as passwords or credit card numbers.

This document provides an overview of the logging architecture in Kyma.

1. Container logs are stored under the var/log directory and its subdirectories.
2. The agent queries the Kubernetes API Server which validates and configures data for objects such as Pods or Services.
3. The agent fetches Pod and container details. Based on that, it tails the logs.
4. The agent enriches log data with Pod labels and sends them to the Loki server. To enable faster data processing, log data is organized in log chunks. A log chunk consists of metadata, such as labels, collected over a certain time period.
5. The Loki server processes the log data and stores it in the log store. The labels are stored in the index store.

6. The user queries the logs using the following tools:

   • Grafana dashboards to analyze and visualize logs fetched and processed by Loki.
   • API clients to query log data using the HTTP API for Loki.
   • Log UI, accessed from the Kyma Console, to display and analyze logs.

Storage

By default, Loki comes with the boltDB storage configuration. It includes label and index storage, and the filesystem for object storage. Additionally, Loki supports other object stores, such as S3 or GCS.

This is an example of Loki configuration using boltDB and filesystem storage:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: loki
  name: loki
data:
  loki.yaml: |
    auth_enabled: false
    server:
      http_listen_port: 3100
    ingester:
      lifecycler:
        ring:
          store: inmemory
          replication_factor: 1
    schema_config:
      configs:
      - from: 0
        store: boltdb
        object_store: filesystem
        schema: v9
        index:
          prefix: index_
          period: 168h
    storage_config:
      - name: boltdb
        directory: /tmp/loki/index
      - name: filesystem
        directory: /tmp/loki/chunks

A sample configuration for GCS looks as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: loki
  name: loki
data:
  loki.yaml: |
    auth_enabled: false
    server:
      http_listen_port: 3100
    ingester:
      lifecycler:
        ring:
          store: inmemory
          replication_factor: 1
    schema_config:
      configs:
      - from: 0
        store: gcs
        object_store: gsc
        schema: v9
        index:
          prefix: index_
          period: 168h
    storage_config:
      gcs:
        bucket_name: <YOUR_GCS_BUCKETNAME>
        project: <BIG_TABLE_PROJECT_ID>
        instance: <BIG_TABLE_INSTANCE_ID>
        grpc_client_config: <YOUR_CLIENT_SETTINGS>

To use Logging in a mid-size production environment, you can install Kyma with the Logging production profile. Higher memory limits set for Loki and Fluent Bit logging solutions ensure stable log processing for 40 active Pods without causing any memory issues. If you want to work with a larger number of active Pods or experience a prolonged query time, configure the Logging chart to increase the memory and CPU values.

NOTE: This profile does not allow for horizontal scaling for which you need additional, dedicated storage systems.

Parameters

The table shows the parameters used in the production profile and their values:

Use the production profile

You can deploy a Kyma cluster with Logging configured to use the production profile, or add the configuration in the runtime. Follow these steps: