Join us for the KubeCon, Seattle, December 10-13! Read more

Tracing

Overview

The micro-services based architecture differs from the traditional monoliths in many aspects. From the request observability perspective, there are asynchronous boundaries among various different micro-services that compose a request flow. Moreover, these micro-services can have heterogeneous semantics when it comes to monitoring and observability. It is required to have a tracing solution that can provide a holistic view of the request flow and help the developer understand the system better to take informed decisions regarding troubleshooting and performance optimization.

Tracing in Kyma uses Jaeger as a backend which serves as the query mechanism for displaying information about traces. Jaeger is used for monitoring and troubleshooting microservice-based distributed systems, including:

  • Distributed context propagation
  • Distributed transaction monitoring
  • Root cause analysis
  • Service dependency analysis
  • Performance and latency optimization

Jaeger provides compatibility with the Zipkin protocol. The compatibility makes it possible to use Zipkin protocol and clients in Istio, Envoy, and Kyma services.

You can access the Jaeger UI either locally at https://jaeger.kyma.local or on a cluster at https://jaeger.{domain-of-kyma-cluster}.

Propagate HTTP headers

The envoy proxy controls the inbound and outbound traffic in the application and automatically sends the trace information to the Zipkin. However, to track the flow of the REST API calls or the service injections in Kyma, it requires the minimal application cooperation from the micro-services code. For this purpose, you need to configure the application to propagate the tracing context in HTTP headers when making outbound calls. See the Istio documentation for details on which headers are required to ensure the correct tracing in Kyma.

Architecture

See the diagram and steps for an overview of the tracing flow in Kyma:

Tracing architecture

The central element of the tracing architecture in Kyma is istio-jaeger. This main component serves both as a target of all query requests made from the Jaeger UI and the space for storing and processing the spans and traces created by Envoy and Kyma services.

Request traces

  1. Kyma user accesses Jaeger UI and requests the trace details for a given service by selecting the service from the Services drop-down menu and confirming the choice by selecting the Find Traces button.
  2. Jaeger passes the request to the the UI facade, jaeger-query.
  3. The jaeger-query forwards the details to the istio-jaeger component which sends the information back.

Request traces

Store traces

  1. Kyma user configures the application to propagate the correct HTTP headers for the outbound calls.
  2. Envoy passes the trace details to the Zipkin Kubernetes service. This service acts as a facade to receive the trace and span details.
  3. Zipkin service forwards the tracing information to the istio-jaeger component which processes the received details.

Store traces

Trace Comparison

Trace comparison allows the structure of two traces to be compared. Each trace is rendered as a tree of connected services and operations. The comparison differences between two traces are color coded.

Compare traces

Compare the traces using the Jaeger user interface.

  1. In the search page for traces, select the traces to compare and click Compare Traces.

    Tracing architecture

  2. The page shows the comparison of two traces selected in the previous step. The traces are marked with A and B.

    Tracing architecture

  3. Use the top menus for A and B to select the traces you want to compare.

    Tracing architecture

    Trace spans have different colors which indicate their meaning:

    • Dark colors indicate that the span is missing from one of the traces:
      • Dark red: The span is only present in trace A.
      • Dark green: The span is only present in trace B.
    • Light colors indicate that the span is present in both traces but occurs more often in one of the traces:
      • Light red: The span in A has more spans than B.
      • Light green: The span in B has more spans than A.
    • Gray: indicates that two traces have a span and the same number of further spans grouped in it.

    Additionally, spans are marked with numerical values indicating how often they occur in compared traces. The values can be positive or negative.

    NOTE: Missing spans can be interpreted as either the application not calling the downstream service, which might be a bug, or that the downstream service is down.

    Tracing architecture