Service Mesh
Overview
Kyma Service Mesh is the component responsible for service-to-service communication, proxying, service discovery, traceability, and security. To deliver this functionality, Kyma Service Mesh uses Istio open platform.
The main principle of Kyma Service Mesh is to inject Pods of every service with the Envoy sidecar proxy. Envoy intercepts the communication between the services and regulates it by applying and enforcing the rules you create. Kyma Dex, which is also a part of the Service Mesh, allows you to integrate any OpenID Connect-compliant identity provider or a SAML2-based enterprise authentication server with your solution.
By default, Istio in Kyma has mutual TLS (mTLS) enabled and injects a sidecar container to every Pod. You can manage mTLS traffic in services or at a Namespace level by creating Destination Rules and Peer Authentications. If you disable sidecar injection in a service or in a Namespace, you must manage their traffic configuration by creating appropriate Destination Rules and Peer Authentications.
NOTE: The Istio Control Plane doesn't have mTLS enabled.
NOTE: For security and performance we use the distroless version of Istio images. Those images are not Debian-based and are slimmed down to reduce any potential attack surface and increase startup time.
Details
Istio setup in Kyma
Istio in Kyma is installed with the help of the istioctl
tool.
The tool is driven by a configuration file containing an instance of IstioOperator custom resource.
There are two configuration files — one for local installation on Minikube and one for cluster installations.
The configurations are customized for Kyma and are stored in the resources/istio/files
directory.
Istio components
This list shows the available Istio components and the components enabled in Kyma:
Component | Enabled |
---|---|
Istiod | ✅ |
Ingress Gateway | ✅️ |
Egress Gateway | ⛔️ |
CNI | ⛔️ |
Grafana | ⛔️ |
Prometheus | ⛔️ |
Tracing | ⛔️ |
Kiali | ⛔️ |
Kyma-specific configuration
These configuration changes are applied to customize Istio for use with Kyma:
- Automatic sidecar injection is enabled by default, excluding the
istio-system
andkube-system
Namespaces. - New resource requests for Istio sidecars are introduced: CPU:
20m
, memory:32Mi
. - New resource limits for Istio sidecars are introduced: CPU:
200m
, memory:128Mi
. - Mutual TLS (mTLS) is enabled cluster-wide in a STRICT mode.
- Global tracing is set to use the Zipkin installation provided by Kyma.
- Ingress Gateway is expanded to handle ports
80
and443
for local Kyma deployments. - DestinationRules are created by default, which disables mTLS for the
kubernetes.default.svc.cluster.local
service. In local (Minikube) installation mTLS is also disabled foristio-ingressgateway.istio-system.svc.cluster.local
service. - The
istio-sidecar-injector
Mutating Webhook Configuration is patched to exclude Gardener resources in the kube-system namespace and the timeout is set to 10 seconds. - All Istio components have decreased resource requests and limits.
- The use of HTTP 1.0 is enabled in the outbound HTTP listeners by
PILOT_HTTP10
flag set in Istiod component environment variables.
Istio patch
As a core component, Istio is installed with every Kyma deployment by default. The installation includes the following steps:
Istio is installed using the
istioctl
tool. The installation resources reside in theresources/istio
directory.A custom Istio Patch job runs and checks if the detected Istio deployment meets the following criteria:
- A specific version of Istio is installed. The required version is defined in the
values
file. - Mutual TLS (mTLS) policy is enabled and set to
strict
. - Automatic sidecar injection is enabled.
If the Istio deployment fails to meet any of these criteria, the patch fails, which results in a failed installation. In such case, get the Istio Patch logs to find out which criteria the Istio deployment didn't meet. Run:
kubectl logs -n istio-system -l app=istio-kyma-patch
NOTE: The Istio patch component is enabled by default, but you can disable it at any time using the instructions.
Use an existing Istio installation with Kyma
You can use an existing installation of Istio with your Kyma installation. To use it:
- Make sure Istio is already running in the cluster.
- Don't disable the Istio Patch component.
- Install Kyma without Istio. Read the configuration document for more details.
Sidecar Proxy Injection
By default, istiod
watches all Pod creation operations on all Namespaces and injects the newly created Pods with a sidecar proxy.
You can disable sidecar proxy injection for either an entire Namespace or a single Deployment.
- To disable sidecar proxy injection for a Namespace, set the istio-injection label value to
disabled
for the Namespace in which you want to disable the sidecar proxy injection. Use this command:kubectl label namespace {YOUR_NAMESPACE} istio-injection=disabled
- To disable sidecar proxy injection for a Deployment, add this annotation to the Deployment configuration file:
sidecar.istio.io/inject: "false"
Read the Istio documentation to learn more about sidecar proxy injection.
Configuration
Istio custom configuration
The Istio installation in Kyma uses the IstioOperator API. Kyma provides the default IstioOperator configurations for local (Minikube) and cluster installations, but you can add a custom IstioOperator definition that overrides the default settings.
The definition you provide may be a partial one with not all the options specified. In that case, it will be merged with the defaults.
To provide a custom IstioOperator configuration, define a Kyma Installation override with the kyma_istio_operator key. The value for this override must be a single string containing a valid definition of the IstioOperator custom resource, in the YAML format.
TIP: To learn more about how to use overrides in Kyma, see the following documents:
See the following example that customizes settings for the policy
and pilot
components of Istio:
```yamlapiVersion: v1kind: ConfigMapmetadata: name: istio-operator-overrides namespace: kyma-installer labels: installer: overrides component: istio kyma-project.io/installation: ""data: kyma_istio_operator: |- apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: namespace: istio-system name: example-istiooperator spec: components: policy: k8s: hpaSpec: minReplicas: 2 pilot: k8s: resources: requests: memory: 3072Mi env: - name: PILOT_TRACE_SAMPLING value: "20"```
While installing with Kyma CLI, don't forget to provide this file's path via -o
flag.
Refer to the IstioOperator API documentation for details about available options.
Service Mesh production profile
By default, every Kyma deployment is installed with the Service Mesh provider Istio, using what is considered a development profile. In this case, this means that:
- Horizontal Pod Autoscaler (HPA) is enabled for all components, with the default number for replicas.
- All components have severely reduced resource quotas, which is a performance factor.
This configuration is not considered production-ready. To use the Kyma Service Mesh in a production environment, configure Istio to use the production profile.
Production profile
The production profile introduces the following changes to the Istio Service Mesh:
- Resource quotas for all Istio components are increased.
- All Istio components have HPA enabled.
- HPA for Istio Ingress Gateway has been customized:
- Minimum number of replicas:
3
- Maximum number of replicas:
10
- Minimum number of replicas:
System requirements
As the production profile is configured with increased performance it mind, the recommend system setup is different:
Requirement | Development setup | Production setup |
---|---|---|
vCPU | 4 | 8 |
RAM | 16 | 16/32 |
Example machine type (GKE) | n1-standard-4 | n1-standard-8 or c2-standard-8 |
Example machine type (AKS) | Standard_D4_v3 | Standard_F8s_v2 or Standard_D8_v3 |
Size | 3 | 3-5 |
Use the production profile
You can deploy a Kyma cluster with Istio configured to use the production profile, or configure Istio in a running cluster to use the production profile. Follow these steps:
TIP: Read the Istio custom configuration section to learn how to provide your own overrides.
- Install Kyma with production-ready Istio
- Enable production profile in a running cluster
Troubleshooting
Overview
The troubleshooting section aims to identify the most common recurring problems with the Kyma Service Mesh, as well as the most suitable solutions to these problems.
If you can't find a solution to your problem, don't hesitate to create a GitHub issue or reach out to the #service-mesh Slack channel to get direct support from the community.
Can't access Console UI or other endpoints
The 503
status code received when you try to access the Console UI or any other endpoint in Kyma can be caused by a configuration error in the Istio Ingress Gateway. As a result, the endpoint you call is not exposed.
To fix this problem, restart the Pods of the Gateway.
List all available endpoints:
Click to copykubectl get virtualservice --all-namespacesRestart the Pods of the Istio Ingress Gateway to force them to recreate their configuration:
Click to copykubectl delete pod -l app=istio-ingressgateway -n istio-system
If this solution doesn't work, you need to change the image of the Istio Ingress Gateway to allow further investigation. Kyma uses distroless Istio images which are more secure, but you cannot execute commands inside them. Follow this steps:
Edit the Istio Ingress Gateway Deployment:
Click to copykubectl edit deployment -n istio-system istio-ingressgatewayFind the
istio-proxy
container and delete the-distroless
suffix.Check all ports used by the Istio Ingress Gateway:
Click to copykubectl exec -ti -n istio-system $(kubectl get pod -l app=istio-ingressgateway -n istio-system -o name) -c istio-proxy -- netstat -lptnuIf ports
80
and443
are not used, check the logs of the Istio Ingress Gateway container for errors related to certificates. Run:Click to copykubectl logs -n istio-system -l app=istio-ingressgateway -c ingress-sdsIn case of certificate-related issues, make sure
kyma-gateway-certs
andapp-connector-certs
Secrets are available in theistio-system
Namespace and that they contain proper data. Run:Click to copykubectl get secrets -n istio-system kyma-gateway-certs -oyamlkubectl get secrets -n istio-system app-connector-certs -oyamlTo regenerate a corrupted certificate, follow this tutorial. If you are running Kyma provisioned through Gardener, follow this tutorial instead.
NOTE: Remember to switch back to the
distroless
image after you resolved the issue.
Connection refused errors
Mutual TLS (mTLS) is enabled in the Service Mesh by default. As a result, every element of the Service Mesh must have an Istio sidecar with a valid TLS certificate to allow communication. Attempts to establish connection between a service without a sidecar and a service with a sidecar result in a Connection reset by peer
or a GOAWAY
response.
- To enable sidecar injection for Pods of existing Services, restart them after upgrading Kyma.
- To whitelist a Service without a sidecar and disable mTLS traffic for it, create a DestinationRule.
To allow connections between Services without a sidecar and a Service with a sidecar, create a Peer Authentication in the
PERMISSIVE
mode.NOTE: In Istio 1.5, Peer Authentication replaces the deprecated Authentication Policy.
Issues with Istio sidecar injection
Kyma has sidecar injection enabled by default - a sidecar is injected to every Pod in a cluster without the need to add any labels. For more information, read this document. If a Pod doesn't have a sidecar and you did not disable sidecar injection on purpose, follow these steps to troubleshoot:
Check if sidecar injection is disabled in the Namespace of the Pod. Run this command to check the
istio-injection
label:Click to copykubectl get namespaces {NAMESPACE} -o jsonpath='{ .metadata.labels.istio-injection }'If the command returns
disabled
the sidecar injection is disabled in this Namespace. To add a sidecar to the Pod, move the Pod's deployment to a Namespace that has sidecar injection enabled, or remove the label from the Namespace and restart the Pod.WARNING: Removing the
istio-injection=disabled
label from Namespace results in injecting sidecars to all Pods inside of the Namespace.Check if sidecar injection is disabled in the Pod's Deployment:
Click to copykubectl get deployments {DEPLOYMENT_NAME} -n {NAMESPACE} -o jsonpath='{ .spec.template.metadata.annotations }'Sidecar injection is disabled if the output contains the
sidecar.istio.io/inject:false
line. Delete the label and restart the Pod to enable sidecar injection for the Deployment.
For more information, read this document or follow the Istio documentation.
Incompatible Istio sidecar version after Kyma upgrade
Kyma has sidecar injection enabled by default - a sidecar is injected to every Pod in a cluster without the need to add any labels. For more information, read this document.
The sidecar version in Pods must match the installed Istio version. Otherwise, mesh connectivity may be broken.
This issue may appear during Kyma upgrade. When Kyma is upgraded to a new version along with a new Istio version, existing sidecars injected into Pods remain in an original version.
Kyma contains the istio-proxy-reset
job that performs a rollout for most common workload types, such as Deployments, DaemonSets, etc. The job ensures all Kyma components are properly updated.
However, some user-defined workloads can't be rolled out automatically. This applies, for example, to a standalone Pod without any backing management mechanism, such as a ReplicaSet or a Job.
Such user-defined workloads, that are not part of Kyma, must be manually restarted to work correctly with the updated Istio version.
To check if any Pods or workloads require a manual restart, follow these steps:
Check the installed Istio version using one of these methods:
From the
istiod
deployment in a running cluster:Click to copyexport KYMA_ISTIO_VERSION=$(kubectl get deployment istiod -n istio-system -o json | jq '.spec.template.spec.containers | .[].image' | sed 's/[^:"]*[:]//' | sed 's/["]//g')From Kyma sources - run this command from within the directory that contains Kyma sources:
Click to copyexport KYMA_ISTIO_VERSION=$(cat resources/istio/Chart.yaml | grep version | sed 's/[^:]*[:]//' | sed 's/ //g')
Get the list of objects which require rollout using one of these methods:
- Find all Pods with outdated sidecars. The returned list follows the
name/namespace
format. The empty output means that there is no Pod that requires migration. To find all outdated Pods, run:Click to copyCOMMON_ISTIO_PROXY_IMAGE_PREFIX="eu.gcr.io/kyma-project/external/istio/proxyv2"kubectl get pods -A -o json | jq -rc '.items | .[] | select(.spec.containers[].image | startswith("'"${COMMON_ISTIO_PROXY_IMAGE_PREFIX}"'") and (endswith("'"${KYMA_ISTIO_VERSION}"'") | not)) | "\(.metadata.name)/\(.metadata.namespace)"'
- Find all Pods with outdated sidecars. The returned list follows the
* Run the `istio-proxy-reset` script in the dry-run mode. The output contains information about objects, such as Pods, Deployments, etc., that require rollout. To run the script, run this command from within the directory with checked-out Kyma sources: ```bash EXPECTED_ISTIO_PROXY_IMAGE="${KYMA_ISTIO_VERSION}" COMMON_ISTIO_PROXY_IMAGE_PREFIX="eu.gcr.io/kyma-project/external/istio/proxyv2" DRY_RUN="true" ./resources/istio/files/istio-proxy-reset.sh ```
After you find a set of objects that require the manual update, restart their related workloads so that new Istio sidecars are injected into the Pods.