Service Mesh


Kyma Service Mesh is the component responsible for service-to-service communication, proxying, service discovery, traceability, and security. To deliver this functionality, Kyma Service Mesh uses Istio open platform.

The main principle of Kyma Service Mesh is to inject Pods of every service with the Envoy sidecar proxy. Envoy intercepts the communication between the services and regulates it by applying and enforcing the rules you create. Kyma Dex, which is also a part of the Service Mesh, allows you to integrate any OpenID Connect-compliant identity provider or a SAML2-based enterprise authentication server with your solution.

By default, Istio in Kyma has mutual TLS (mTLS) enabled and injects a sidecar container to every Pod. You can manage mTLS traffic in services or at a Namespace level by creating Destination Rules and Authentication Policies. If you disable sidecar injection in a service or in a Namespace, you must manage their traffic configuration by creating appropriate Destination Rules and Authentication Policies.

NOTE: The Istio Control Plane doesn't have mTLS enabled.

NOTE: For security and performance we use the distroless version of Istio images. Those images are not Debian-based and are slimmed down to reduce any potential attack surface and increase startup time.

Kyma uses Kiali to enable validation, observe Istio Service Mesh, and provide details on microservices included in the Service Mesh and connections between them. For Kiali chart configuration, see this document.

NOTE: Kiali is disabled by default. Read this document for instructions on how to enable it.


Istio setup in Kyma

Istio in Kyma is installed with the help of the istioctl tool. The tool is driven by a configuration file containing an instance of IstioControlPlane custom resource. There are two configuration files — one for local installation on Minikube and one for cluster installations. The configurations are customized for Kyma and are stored in the resources/istio directory.

Istio components

This list shows the available Istio components and the components enabled by default:

Sidecar Injector
Node agent⛔️

Kyma-specific configuration

These configuration changes are applied to customize Istio for use with Kyma:

  • Only the Ingress Gateway is enabled.
  • The Secret Discovery Service is enabled in the Ingress Gateway.
  • Automatic sidecar injection is enabled by default, excluding the istio-system and kube-system Namespaces.
  • New resource limits for Istio sidecars are introduced: CPU: 100m, memory: 128Mi.
  • Mutual TLS (mTLS) is enabled cluster-wide with the exception of the Istio Control Plane.
  • Global tracing is set to use the Zipkin installation provided by Kyma.
  • Ingress Gateway is expanded to handle ports 80 and 443 for local Kyma deployments.
  • DestinationRules are created by default, which disables mTLS for the istio-ingressgateway.istio-system.svc.cluster.local service.

Istio patch

As a core component, Istio is installed with every Kyma deployment by default. The installation includes the following steps:

  1. Istio is installed using the official charts from the currently supported release. The charts reside in the resources/istio directory.

  2. A custom Istio Patch job runs and checks if the detected Istio deployment meets the following criteria:

  • A specific version of Istio is installed. The required version is defined in the values file.
  • Mutual TLS (mTLS) policy is enabled and set to strict.
  • Istio policy enforcement is enabled.
  • Automatic sidecar injection is enabled.
  • Istio CustomResourceDefinition (CRD) is present in the system.

If the Istio deployment fails to meet any of these criteria, the patch fails, which results in a failed installation. In such case, get the Istio Patch logs to find out which criteria the Istio deployment didn't meet. Run:

Click to copy
kubectl logs -n istio-system -l app=istio-kyma-patch

NOTE: The Istio patch component is enabled by default, but you can disable it at any time using these instructions.

Use an existing Istio installation with Kyma

You can use an existing installation of Istio with your Kyma installation. To use it:

  • Make sure Istio is already running in the cluster.
  • Don't disable the Istio Patch component.
  • Install Kyma without Istio. Read this document for more details.

Istio RBAC configuration

As a core component, Istio is installed with Kyma by default. The ClusterRbacConfig custom resource (CR), which defines the global behavior of Istio, is created as a part of the installation process.

The default Istio RBAC configuration is defined in this file.

Override the default configuration

To override the default configuration of Istio RBAC, edit the ClusterRbacConfig CR on a running cluster. This CR is created in the kyma-system Namespace and therefore requires admin permissions to edit it.

To show the current Istio RBAC configuration in the yaml format, run:

Click to copy
kubectl get -n kyma-system clusterrbacconfig -o yaml

To edit the Istio RBAC configuration, run:

Click to copy
kubectl edit -n kyma-system clusterrbacconfig

NOTE: The ClusterRbacConfig object is a singleton, which means that only a single object of this kind can exist in a cluster. Additionally, the only valid name for the object is default. As such, the best way to customize Istio RBAC is by editing the existing ClusterRbacConfig object.

Sidecar Proxy Injection

By default, the Istio sidecar injector watches all Pod creation operations on all Namespaces and injects the newly created Pods with a sidecar proxy.

You can disable sidecar proxy injection for either an entire Namespace or a single Deployment.

  • To disable sidecar proxy injection for a Namespace, set the istio-injection label value to disabled for the Namespace in which you want to disable the sidecar proxy injection. Use this command: kubectl label namespace {YOUR_NAMESPACE} istio-injection=disabled
  • To disable sidecar proxy injection for a Deployment, add this annotation to the Deployment configuration file: "false"

Read the this document to learn more about sidecar proxy injection.


Kiali chart

To configure the Kiali chart, override the default values of its values.yaml file. This document describes parameters that you can configure.

TIP: To learn more about how to use overrides in Kyma, see the following documents:

Configurable parameters

This table lists the configurable parameters, their descriptions, and default values:

ParameterDescriptionDefault value
server.metrics.enabledSpecifies if the metrics endpoint is available for Prometheus to scrape.true
server.webRootDefines the context root path for Kiali console, API endpoints, and readiness probes./
deployment.viewOnlyModeWhen set to true, Kiali is available in view-only mode, allowing you to view and retrieve management data for the Service Mesh. You cannot modify the Service Mesh.true
deployment.accessibleNamespacesSpecifies the Namespaces Kiali can access to monitor the Service Mesh components deployed there. You can provide the names using regex expressions. The default value is **(two asterisks) meaning Kiali can access any Namespace.**

For details on Kiali configuration and customization, see this documentation.

Service Mesh production profile

By default, every Kyma deployment is installed with the Service Mesh provider Istio, using what is considered a development profile. In this case, this means that:

  • Horizontal Pod Autoscaler (HPA) is enabled for all components, with the default number for replicas.
  • All components have severely reduced resource quotas, which is a performance factor.

This configuration is not considered production-ready. To use the Kyma Service Mesh in a production environment, configure Istio to use the production profile.

Production profile

The production profile introduces the following changes to the Istio Service Mesh:

  • Resource quotas for all Istio components are increased.
  • All Istio components have HPA enabled.
  • HPA for Istio Ingress Gateway has been customized:
    • Minimum number of replicas: 3
    • Maximum number of replicas: 10

System requirements

As the production profile is configured with increased performance it mind, the recommend system setup is different:

RequirementDevelopment setupProduction setup
Example machine type (GKE)n1-standard-4n1-standard-8 or c2-standard-8
Example machine type (AKS)Standard_D4_v3Standard_F8s_v2 or Standard_D8_v3

Use the production profile

CAUTION: Due to changes in the installation options in Istio, Helm-based configuration is now deprecated in favor of the new IstioControlPlane API. Please keep in mind that Helm overrides will be no longer supported in future Istio releases. Refer to IstioControlPlane documentation for details.

You can deploy a Kyma cluster with Istio configured to use the production profile, or configure Istio in a running cluster to use the production profile. Follow these steps:

  • Istio Control Plane API
  • Install Kyma with production-ready Istio
  • Enable production profile in a running cluster



The troubleshooting section aims to identify the most common recurring problems with the Kyma Service Mesh, as well as the most suitable solutions to these problems.

If you can't find a solution to your problem, don't hesitate to create a GitHub issue or reach out to the #service-mesh Slack channel to get direct support from the community.

Can't access Console UI or other endpoints

The 503 status code received when you try to access the Console UI or any other endpoint in Kyma can be caused by a configuration error in the Istio Ingress Gateway. As a result, the endpoint you call is not exposed. To fix this problem, restart the Pods of the Gateway.

  1. List all available endpoints:

    Click to copy
    kubectl get virtualservice --all-namespaces
  2. Check all ports used by the Istio Ingress Gateway:

    Click to copy
    kubectl exec -t -n istio-system $(kubectl get pod -l app=istio-ingressgateway -n istio-system | grep "istio-ingressgateway" | awk '{print $1}') -- netstat -lptnu
  3. If ports 80 and 443 are not used, restart the Pods of the Istio Ingress Gateway to force them to recreate their configuration:

    Click to copy
    kubectl delete pod -l app=istio-ingressgateway -n istio-system

Connection refused errors after upgrade

Starting with Kyma 1.0, mutual TLS (mTLS) is enabled in the Service Mesh by default. As a result, every element of the Service Mesh must have an Istio sidecar to allow TLS communication. Alternatively, you can whitelist a service and disable TLS traffic for it by creating a DestinationRule.

  • To enable sidecar injection for Pods of existing Services, restart them after upgrading to Kyma 1.0 or higher.
  • To create a DestinationRule for a Service, follow the official Istio documentation.

Kubernetes jobs fail on AKS

A known issue related to Istio sidecar handling on AKS causes Kubernetes jobs with Istio Proxy sidecar to run endlessly as the sidecar doesn't terminate. As a workaround, disable Istio sidecar injection for all jobs on AKS by adding the "false" annotation and create appropriate Destination Rules and Authentication Policies.

To get a better understanding of this problem, read this Istio issue and the related discussion.