Diagnosing 503 Errors ​
Follow this troubleshooting guide if you receive the 503 Service Unavailable status code when trying to access a Kyma endpoint. A 503 can come from the service mesh or your application, and the fix differs in each case.
Identify the Origin of the 503 Error ​
A 503 generated by Istio Ingress Gateway and a 503 returned by your application look similar, but you can tell them apart by inspecting the response. The decisive signal is whether the request reached an upstream workload.
Send a request to the failing endpoint and include the response headers. The following example uses the external address of Istio Ingress Gateway and a path exposed through a VirtualService:
curl -sD - http://{YOUR_HOST}/{YOUR_PATH}Compare your output with the following two cases:
- If the response body is a generic Envoy message, such as
no healthy upstream, and thex-envoy-upstream-service-timeheader is absent, the mesh generated the503. Istio Ingress Gateway could not route the request to a healthy workload, and the Envoy proxy generated the response before it reached your application. See The503Comes from the Service Mesh. - If the response body is your application's own payload and the
x-envoy-upstream-service-timeheader is present, the application generated the503. The request was routed correctly and reached your workload, but the application itself responded with503, for example, because a dependency is down or the application is overloaded. See The503Comes from Your Application.
The server response header is istio-envoy in both cases, because every response passes back through the proxy.
NOTE: ​
Use the response body and the presence of the
x-envoy-upstream-service-timeheader to decide the origin, not theserverheader. The proxy addsserver: istio-envoyto both mesh-generated and application-generated responses.
The 503 Comes from the Service Mesh ​
Symptom ​
You send a request to a Kyma endpoint and receive a 503 Service Unavailable response. The response body is a generic Envoy message rather than your application's output, and the x-envoy-upstream-service-time header is absent:
HTTP/1.1 503 Service Unavailable
content-length: 19
content-type: text/plain
server: istio-envoy
no healthy upstreamA mesh-generated 503 points to a routing or availability problem, not to your application code.
Cause ​
The mesh returns a 503 for one of the following reasons:
- No Pod backing the Service is in the
Readystate, so the Service has no ready endpoints. - The VirtualService routes to a host, port, or subset that does not match the Service.
- The ingress configuration is broken, for example, because of a corrupted certificate.
Solution ​
Confirm that Istio Ingress Gateway has no healthy upstream for the route. Inspect the endpoints that the Envoy proxy sees for the Service.
Get the name of the Istio Ingress Gateway Pod:
bashkubectl get pods --namespace istio-system --selector istio=ingressgateway --output nameList the endpoints for the Service that the route targets. The
--clustervalue identifies the upstream in the formatoutbound|{SERVICE_PORT}||{SERVICE_FQDN}, where{SERVICE_FQDN}is the fully qualified Service name, such asapp-503.app-503.svc.cluster.local:bashistioctl proxy-config endpoints {POD_NAME}.istio-system --cluster "outbound|{SERVICE_PORT}||{SERVICE_FQDN}"ENDPOINT STATUS OUTLIER CHECK CLUSTER 100.64.0.27:80 UNHEALTHY OK outbound|8000||app-503.app-503.svc.cluster.localTo list every upstream at once instead of a single cluster, omit the
--clusterflag.Check the
STATUScolumn. If it showsUNHEALTHY, or the list is empty, Envoy has no healthy upstream, which confirms a mesh-generated503. When the workload becomes ready, the same endpoint changes toHEALTHY:ENDPOINT STATUS OUTLIER CHECK CLUSTER 100.64.0.28:80 HEALTHY OK outbound|8000||app-503.app-503.svc.cluster.localIf no endpoint appears at all, verify that the cluster exists for the Service and that the route targets the correct host and port:
bashistioctl proxy-config cluster {POD_NAME}.istio-system --fqdn {SERVICE_FQDN}
Check the workload and its endpoints:
bashkubectl get pods --namespace {YOUR_NAMESPACE} --selector {YOUR_SELECTOR} kubectl get endpointslices --namespace {YOUR_NAMESPACE} --selector kubernetes.io/service-name={YOUR_SERVICE}If every endpoint reports
ready=falseor the list is empty, the Service has no healthy upstream. Resolve the workload's readiness issues, such as a failing readiness probe or a crashing container, until at least one endpoint is ready.Verify that the VirtualService routes to the correct host, port, and subset, and that these values match the Service.
If the workload and the route are correct, restart the Deployment of Istio Ingress Gateway to trigger the recreation of its configuration. A rolling restart starts new Pods before it terminates the old ones, so the gateway stays available:
bashkubectl rollout restart deployment istio-ingressgateway --namespace istio-systemWait for the rollout to finish:
bashkubectl rollout status deployment istio-ingressgateway --namespace istio-system
If the restart doesn't help, the ingress itself might be misconfigured, for example, due to a corrupted certificate. Follow these steps:
Kyma Dashboard ​
Check all ports used by Istio Ingress Gateway:
- Go to the
istio-systemnamespace. - In the Workloads section, select Pods.
- Search for a Pod labeled with
app=istio-ingressgatewayand select its name. - Scroll down to find the
Containerssection and check which ports theistio-proxycontainer uses.
- Go to the
If the ports
80and443are not used, check the logs of theistio-proxycontainer for errors related to certificates. To do this, select View Logs.To make sure that a corrupted certificate is regenerated, verify that the spec.enableKymaGateway field of your APIGateway custom resource is set to
true. If you are running Kyma provisioned through Gardener, follow the Issues with Certificates on Gardener troubleshooting guide instead.
kubectl ​
Check all ports used by Istio Ingress Gateway:
List all the Pods of Istio Ingress Gateway:
bashkubectl get pod --selector app=istio-ingressgateway --namespace istio-system --output nameFor each of the listed Pods, replace
{ISTIO_INGRESS_GATEWAY_POD_NAME}with the Pod's name and check the ports that theistio-proxycontainer uses:bashkubectl get --namespace istio-system {ISTIO_INGRESS_GATEWAY_POD_NAME} --output jsonpath='{.spec.containers[*].ports[*].containerPort}'
If the ports
80and443are not used, check the logs of theistio-proxycontainer for errors related to certificates:bashkubectl logs --namespace istio-system --selector app=istio-ingressgateway --container istio-proxyTo make sure that a corrupted certificate is regenerated, verify that the spec.enableKymaGateway field of your APIGateway custom resource is set to
true. If you are running Kyma provisioned through Gardener, follow the Issues with Certificates on Gardener troubleshooting guide instead.
NOTE: ​
To understand how the Istio Ingress Gateway handles each request, enable access logs. The access log entry includes a response flag, such as
UHfor no healthy upstream, indicating why Envoy returned the status. For more information, see Increase Verbosity of the Istio Ingress Logs.
The 503 Comes from Your Application ​
Symptom ​
The request is routed correctly and reaches a healthy workload, but the application responds with a 503. The response carries the application's own payload, and the x-envoy-upstream-service-time header is present, because Envoy forwarded the request to the upstream workload and measured how long the workload took to respond:
HTTP/1.1 503 Service Unavailable
content-length: 22
x-envoy-upstream-service-time: 9
server: istio-envoy
{application-generated error response}Cause ​
The application itself returns the 503, for example, because a dependency is unavailable, a connection pool is exhausted, or the application is overloaded. This is outside the scope of Istio. Istio routes the request correctly and passes the application's response back unchanged.
Solution ​
Confirm that the
503originates in the application. Bypass the mesh and send the request directly to the Pod:bashkubectl port-forward --namespace {YOUR_NAMESPACE} deploy/{YOUR_DEPLOYMENT} 18080:{CONTAINER_PORT} curl -sD - http://127.0.0.1:18080/{YOUR_PATH}A
503returned by a direct request to the Pod indicates that the application, rather than the service mesh, causes the error. The direct response shows the application server in theserverheader, such asnginx, instead ofistio-envoy:HTTP/1.1 503 Service Temporarily Unavailable Server: nginx/1.30.6 Content-Length: 22 {application-generated error response}Debug the workload, not Istio. Check the application logs and the status of its dependencies:
bashkubectl logs --namespace {YOUR_NAMESPACE} deploy/{YOUR_DEPLOYMENT} --container {YOUR_CONTAINER}Fix the root cause in the application. For example, restore the unavailable dependency or increase the application's capacity.