The microservice architecture differs from the traditional monoliths in many aspects. From the request observability perspective, there are asynchronous boundaries among various different microservices that compose a request flow. Moreover, these microservices can have heterogeneous semantics when it comes to monitoring. A tracing solution that provides a holistic view of the request flow helps you to understand the system and take informed decisions regarding troubleshooting and performance optimization.
Tracing in Kyma uses Jaeger as a backend which serves as the query mechanism for displaying information about traces.
CAUTION: Jaeger is designed for application tracing. Traces should not carry sensitive information, such as passwords or credit card numbers.
The Jaeger-based tracing component provides the necessary functionality to collect and query traces. Both operations may occur at the same time. This way you inspect specific traces using the Jaeger UI, while Jaeger takes care of proper trace collection and storage in parallel. See the diagram for details:
The process of collecting traces by Jaeger looks as follows:
- The application receives a request, either from an internal or external source.
- If the application has Istio injection enabled, Istio proxy propagates the correct HTTP headers of the requests to the Jaeger Deployment. Istio proxy calls Jaeger using the Zipkin service which exposes a Jaeger port compatible with the Zipkin protocol.
- Jaeger processes the data. Specifically, the Jaeger Agent component receives the spans, batches them, and forwards to the Jaeger Collector service.
- The BadgerDB database stores the data and persists it using a PersistentVolume resource.
The process of querying traces from Jaeger looks as follows:
- A Kyma user accesses the Jaeger UI to look for specific traces.
- Jaeger UI passes the request to the Jaeger Query service. The request goes through the Istio Ingress Gateway which forwards the incoming connections to the service.
- Jaeger Query passes the request to the Keycloak Gatekeeper for authorization. The Gatekeeper calls Dex to authenticate the user and the request, and grants further access if the authentication is successful.
- Finally, the functionality provided by the Jaeger Deployment allows you to retrieve trace information.
Observability tools should clearly show the big picture, no matter if they help you monitor just a couple or multiple components. In a cloud-native microservice architecture, a user request often flows through dozens of different microservices. Tools such as logging or monitoring help to track the way, however, they treat each component or microservice in isolation. This individual treatment results in operational issues.
Distributed tracing charts out the transactions in cloud-native systems, helping you to understand the application behavior and relations between the frontend actions and backend implementation.
The diagram shows how the distributed tracing helps to track the request path.
Jaeger is a monitoring and tracing tool for microservice-based distributed systems. Its features include the following:
- Distributed context propagation
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis
- Performance and latency optimization
The Envoy sidecar uses Jaeger to trace the request flow in the Istio Service Mesh. Jaeger is compatible with the Zipkin protocol, which Istio and Envoy use to communicate with the tracing backend. This allows you to use the Zipkin protocol and clients in Istio, Envoy, and Kyma services.
For details, see Istio's Distributed Tracing.
Read the configuration document to learn how to install Jaeger locally.
Access the Jaeger UI either locally at
https://jaeger.kyma.local or on a cluster at
The Envoy proxy controls the inbound and outbound traffic in the application and automatically sends the trace information to Zipkin. To track the flow of the REST API calls or the service injections in Kyma, it requires the application to cooperate with the microservices code. To enable such cooperation, configure the application to propagate the tracing context in HTTP headers when making outbound calls. See the Istio documentation for details on headers required to ensure the correct tracing in Kyma.
Trace comparison allows you to compare the structure of two traces, rendered as a tree of connected services and operations. The colors help you to distinguish the differences between two traces.
Compare the traces using the Jaeger user interface.
In the search page for traces, select the traces to compare and click Compare Traces.
The page shows the comparison of two traces selected in the previous step. The traces are marked with A and B.
Use the top menus for A and B to select the traces you want to compare.
Trace spans have different colors which indicate their meaning:
- Dark colors indicate that the span is missing from one of the traces:
- Dark red: The span is only present in trace A.
- Dark green: The span is only present in trace B.
- Light colors indicate that the span is present in both traces but occurs more often in one of the traces:
- Light red: The span in A has more spans than B.
- Light green: The span in B has more spans than A.
- Gray: indicates that two traces have a span and the same number of further spans grouped in it.
Additionally, spans are marked with numerical values indicating how often they occur in compared traces. The values can be positive or negative.
NOTE: Missing spans can be interpreted as either the application not calling the downstream service, which might be a bug, or that the downstream service is down.
- Dark colors indicate that the span is missing from one of the traces:
You can search traces using tags. Tags are key-value pairs configured for each service. The full list of tags for a service from the details of that service's span.
For example, use these tags for
To search the traces, you can use either a single tag, such as
event-type="order.created", or multiple tags, such as
To configure the Tracing chart, override the default values of its
values.yaml file. This document describes parameters that you can configure.
This table lists the configurable parameters, their descriptions, and default values:
|global.tracing.enabled||Enables emitting traces for the relevant components.||true|
|global.tracing.zipkinAddress||Specifies the address of the Zipkin instance.||"zipkin.kyma-system:9411"|
|jaeger.spec.resources.limits.memory||Defines the maximum amount of memory that is available for storing traces in Jaeger.|
|jaeger.spec.strategy||Deployment strategy to use. The possible values are either |
|jaeger.spec.storage.type||Defines storage type for span data. The possible values are |
|jaeger.spec.storage.options||Defines additional options for the storage type.||-|
By default, only 1% of the requests are sent to Jaeger for trace recording. To change this system behavior, you can override the existing settings or change the value in the Runtime.
NOTE: You can also manually set the
x-b3-sampled: 1header to force sampling for a particular request.
Follow these steps to override the existing configuration
- Add and apply a ConfigMap in the
kyma-installerNamespace in which you set the value for the trace sampling attribute to
cat <<EOF | kubectl apply -f -apiVersion: v1kind: ConfigMapmetadata:name: istio-overridesnamespace: kyma-installerlabels:installer: overridescomponent: istiokyma-project.io/installation: ""data:kyma_istio_operator: |-apiVersion: install.istio.io/v1alpha1kind: IstioOperatormetadata:namespace: istio-systemspec:meshConfig:defaultConfig:tracing:sampling: 60EOF
Proceed with the installation. Once the installation starts, the Kyma Operator will generate the override based on the ConfigMap and set the value of trace sampling to
NOTE: If you add the override in the Runtime, run the following command to trigger the update:Click to copykubectl -n default label installation/kyma-installation action=install
If you have already installed Kyma and do not want to trigger any updates, edit the
istiod deployment to set the desired value for PILOT_TRACE_SAMPLING. For detailed instructions, see the Istio documentation.
Note: the change to PILOT_TRACE_SAMPLING would only take effect if the meshConfig override is not defined.