Skip to content

Diagnosing 503 Errors ​

Follow this troubleshooting guide if you receive the 503 Service Unavailable status code when trying to access a Kyma endpoint. A 503 can come from the service mesh or your application, and the fix differs in each case.

Identify the Origin of the 503 Error ​

A 503 generated by Istio Ingress Gateway and a 503 returned by your application look similar, but you can tell them apart by inspecting the response. The decisive signal is whether the request reached an upstream workload.

Send a request to the failing endpoint and include the response headers. The following example uses the external address of Istio Ingress Gateway and a path exposed through a VirtualService:

bash
curl -sD - http://{YOUR_HOST}/{YOUR_PATH}

Compare your output with the following two cases:

  • If the response body is a generic Envoy message, such as no healthy upstream, and the x-envoy-upstream-service-time header is absent, the mesh generated the 503. Istio Ingress Gateway could not route the request to a healthy workload, and the Envoy proxy generated the response before it reached your application. See The 503 Comes from the Service Mesh.
  • If the response body is your application's own payload and the x-envoy-upstream-service-time header is present, the application generated the 503. The request was routed correctly and reached your workload, but the application itself responded with 503, for example, because a dependency is down or the application is overloaded. See The 503 Comes from Your Application.

The server response header is istio-envoy in both cases, because every response passes back through the proxy.

NOTE: ​

Use the response body and the presence of the x-envoy-upstream-service-time header to decide the origin, not the server header. The proxy adds server: istio-envoy to both mesh-generated and application-generated responses.

The 503 Comes from the Service Mesh ​

Symptom ​

You send a request to a Kyma endpoint and receive a 503 Service Unavailable response. The response body is a generic Envoy message rather than your application's output, and the x-envoy-upstream-service-time header is absent:

HTTP/1.1 503 Service Unavailable
content-length: 19
content-type: text/plain
server: istio-envoy

no healthy upstream

A mesh-generated 503 points to a routing or availability problem, not to your application code.

Cause ​

The mesh returns a 503 for one of the following reasons:

  • No Pod backing the Service is in the Ready state, so the Service has no ready endpoints.
  • The VirtualService routes to a host, port, or subset that does not match the Service.
  • The ingress configuration is broken, for example, because of a corrupted certificate.

Solution ​

  1. Confirm that Istio Ingress Gateway has no healthy upstream for the route. Inspect the endpoints that the Envoy proxy sees for the Service.

    1. Get the name of the Istio Ingress Gateway Pod:

      bash
      kubectl get pods --namespace istio-system --selector istio=ingressgateway --output name
    2. List the endpoints for the Service that the route targets. The --cluster value identifies the upstream in the format outbound|{SERVICE_PORT}||{SERVICE_FQDN}, where {SERVICE_FQDN} is the fully qualified Service name, such as app-503.app-503.svc.cluster.local:

      bash
      istioctl proxy-config endpoints {POD_NAME}.istio-system --cluster "outbound|{SERVICE_PORT}||{SERVICE_FQDN}"
      ENDPOINT           STATUS        OUTLIER CHECK     CLUSTER
      100.64.0.27:80     UNHEALTHY     OK                outbound|8000||app-503.app-503.svc.cluster.local

      To list every upstream at once instead of a single cluster, omit the --cluster flag.

    3. Check the STATUS column. If it shows UNHEALTHY, or the list is empty, Envoy has no healthy upstream, which confirms a mesh-generated 503. When the workload becomes ready, the same endpoint changes to HEALTHY:

      ENDPOINT           STATUS      OUTLIER CHECK     CLUSTER
      100.64.0.28:80     HEALTHY     OK                outbound|8000||app-503.app-503.svc.cluster.local
    4. If no endpoint appears at all, verify that the cluster exists for the Service and that the route targets the correct host and port:

      bash
      istioctl proxy-config cluster {POD_NAME}.istio-system --fqdn {SERVICE_FQDN}
  2. Check the workload and its endpoints:

    bash
    kubectl get pods --namespace {YOUR_NAMESPACE} --selector {YOUR_SELECTOR}
    kubectl get endpointslices --namespace {YOUR_NAMESPACE} --selector kubernetes.io/service-name={YOUR_SERVICE}

    If every endpoint reports ready=false or the list is empty, the Service has no healthy upstream. Resolve the workload's readiness issues, such as a failing readiness probe or a crashing container, until at least one endpoint is ready.

  3. Verify that the VirtualService routes to the correct host, port, and subset, and that these values match the Service.

  4. If the workload and the route are correct, restart the Deployment of Istio Ingress Gateway to trigger the recreation of its configuration. A rolling restart starts new Pods before it terminates the old ones, so the gateway stays available:

    bash
    kubectl rollout restart deployment istio-ingressgateway --namespace istio-system

    Wait for the rollout to finish:

    bash
    kubectl rollout status deployment istio-ingressgateway --namespace istio-system

If the restart doesn't help, the ingress itself might be misconfigured, for example, due to a corrupted certificate. Follow these steps:

Kyma Dashboard ​

  1. Check all ports used by Istio Ingress Gateway:

    1. Go to the istio-system namespace.
    2. In the Workloads section, select Pods.
    3. Search for a Pod labeled with app=istio-ingressgateway and select its name.
    4. Scroll down to find the Containers section and check which ports the istio-proxy container uses.
  2. If the ports 80 and 443 are not used, check the logs of the istio-proxy container for errors related to certificates. To do this, select View Logs.

  3. To make sure that a corrupted certificate is regenerated, verify that the spec.enableKymaGateway field of your APIGateway custom resource is set to true. If you are running Kyma provisioned through Gardener, follow the Issues with Certificates on Gardener troubleshooting guide instead.

kubectl ​

  1. Check all ports used by Istio Ingress Gateway:

    1. List all the Pods of Istio Ingress Gateway:

      bash
      kubectl get pod --selector app=istio-ingressgateway --namespace istio-system --output name
    2. For each of the listed Pods, replace {ISTIO_INGRESS_GATEWAY_POD_NAME} with the Pod's name and check the ports that the istio-proxy container uses:

      bash
      kubectl get --namespace istio-system {ISTIO_INGRESS_GATEWAY_POD_NAME} --output jsonpath='{.spec.containers[*].ports[*].containerPort}'
  2. If the ports 80 and 443 are not used, check the logs of the istio-proxy container for errors related to certificates:

    bash
    kubectl logs --namespace istio-system --selector app=istio-ingressgateway --container istio-proxy
  3. To make sure that a corrupted certificate is regenerated, verify that the spec.enableKymaGateway field of your APIGateway custom resource is set to true. If you are running Kyma provisioned through Gardener, follow the Issues with Certificates on Gardener troubleshooting guide instead.

NOTE: ​

To understand how the Istio Ingress Gateway handles each request, enable access logs. The access log entry includes a response flag, such as UH for no healthy upstream, indicating why Envoy returned the status. For more information, see Increase Verbosity of the Istio Ingress Logs.

The 503 Comes from Your Application ​

Symptom ​

The request is routed correctly and reaches a healthy workload, but the application responds with a 503. The response carries the application's own payload, and the x-envoy-upstream-service-time header is present, because Envoy forwarded the request to the upstream workload and measured how long the workload took to respond:

HTTP/1.1 503 Service Unavailable
content-length: 22
x-envoy-upstream-service-time: 9
server: istio-envoy

{application-generated error response}

Cause ​

The application itself returns the 503, for example, because a dependency is unavailable, a connection pool is exhausted, or the application is overloaded. This is outside the scope of Istio. Istio routes the request correctly and passes the application's response back unchanged.

Solution ​

  1. Confirm that the 503 originates in the application. Bypass the mesh and send the request directly to the Pod:

    bash
    kubectl port-forward --namespace {YOUR_NAMESPACE} deploy/{YOUR_DEPLOYMENT} 18080:{CONTAINER_PORT}
    curl -sD - http://127.0.0.1:18080/{YOUR_PATH}

    A 503 returned by a direct request to the Pod indicates that the application, rather than the service mesh, causes the error. The direct response shows the application server in the server header, such as nginx, instead of istio-envoy:

    HTTP/1.1 503 Service Temporarily Unavailable
    Server: nginx/1.30.6
    Content-Length: 22
    
    {application-generated error response}
  2. Debug the workload, not Istio. Check the application logs and the status of its dependencies:

    bash
    kubectl logs --namespace {YOUR_NAMESPACE} deploy/{YOUR_DEPLOYMENT} --container {YOUR_CONTAINER}
  3. Fix the root cause in the application. For example, restore the unavailable dependency or increase the application's capacity.