Diagnosing 403 Errors ​
Follow this troubleshooting guide if you receive the 403 Forbidden status code when trying to access a Kyma endpoint. A 403 can come from an AuthorizationPolicy inside the mesh, from your application, or from something outside the mesh, and the fix differs in each case.
Identify the Origin of the 403 Error ​
A 403 error from an AuthorizationPolicy, a 403 error from your application, and a 403 error from an external firewall look similar, but you can tell them apart by inspecting the response.
The decisive signal is the response body. An AuthorizationPolicy returns the fixed plain-text body RBAC: access denied, whereas your application returns its own payload, and an external component returns a vendor-specific page or nothing.
The x-envoy-upstream-service-time header is a secondary hint, not a reliable discriminator. It records the time an Envoy proxy waited for an upstream, so its presence depends on where the policy is enforced, not on whether an AuthorizationPolicy was involved:
- A policy enforced at Istio Ingress Gateway rejects the request before the gateway selects an upstream, so the header is absent.
- A policy enforced at the workload's sidecar is already the last Envoy hop, so the header is present and reports the time the gateway waited for the sidecar, even though no application code ran.
Because of this, use the response body to decide the origin, and treat the header only as a tiebreaker.
Send a request to the failing endpoint and include the response headers:
curl -sD - http://{YOUR_HOST}/{YOUR_PATH}Compare your output with the following cases:
| Aspect | AuthorizationPolicy (Istio) | Application | External Firewall, Proxy, or WAF |
|---|---|---|---|
| Where it runs | Inside the mesh, enforced by the Envoy proxy | In your workload, behind the Envoy proxy | Outside the mesh, before the request reaches Envoy |
server response header | istio-envoy | istio-envoy | Absent, or a non-Envoy value such as a proxy or WAF product name |
x-envoy-upstream-service-time header | Absent for a gateway-enforced policy, present for a sidecar-enforced policy | Present | Absent |
| Response body | Plain text RBAC: access denied | Your application's own payload | Vendor-specific HTML or JSON error page, or empty |
| Depends on request identity | Yes. The decision depends on the to, from, and when rules, so it varies by path, source, and token | Depends on the application logic | No. The block is usually the same regardless of path, headers, or credentials |
| Fixed by | Updating the AuthorizationPolicy resource | Debugging the application | Cloud provider or network administrator |
To decide which case applies, use the following signals:
- The response has the
server: istio-envoyheader and anRBAC: access deniedbody. An AuthorizationPolicy rejected the request, regardless of whether thex-envoy-upstream-service-timeheader is present. See The403Comes from an AuthorizationPolicy. - The response has the
server: istio-envoyheader and your application's own body, notRBAC: access denied. Thex-envoy-upstream-service-timeheader is present. Your application generated the403error. See The403Comes from Your Application. - The response has no
server: istio-envoyheader, or it carries a vendor-specific error page instead ofRBAC: access denied. Something other than Istio produced the403error, such as an external proxy or WAF in front of the cluster. This issue is outside the scope of Istio. See The403Comes from Outside the Mesh.
NOTE: ​
Use the response body to decide the origin, not the
serverheader or thex-envoy-upstream-service-timeheader. The proxy addsserver: istio-envoyto both an AuthorizationPolicy response and an application response, and thex-envoy-upstream-service-timeheader can appear on a sidecar-enforced AuthorizationPolicy response as well. Only theRBAC: access deniedbody uniquely identifies an AuthorizationPolicy rejection.
The 403 Comes from an AuthorizationPolicy ​
Symptom ​
The request reaches the mesh and the Envoy proxy returns the 403 error with the server: istio-envoy header and an RBAC: access denied body. When Istio Ingress Gateway enforces the policy, the response has no x-envoy-upstream-service-time header:
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy
RBAC: access deniedWhen a workload's sidecar enforces the policy, the same RBAC: access denied body appears, but the response also carries an x-envoy-upstream-service-time header. The header reports the time the gateway waited for the sidecar, not time spent in your application. The RBAC: access denied body, not this header, identifies the AuthorizationPolicy as the source:
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy
x-envoy-upstream-service-time: 4
RBAC: access deniedCause ​
A misconfigured AuthorizationPolicy is a common cause of 403 errors. A frequent example is an unnecessary port number in the Host header. Istio checks the host as-is. If the Host header contains a port number and the AuthorizationPolicy defines only a hostname, Istio denies the request.
The following AuthorizationPolicy allows only the hostname httpbin.local.kyma.dev, without a port:
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: ingress-allow-headers
namespace: istio-system
spec:
action: ALLOW
rules:
- to:
- operation:
hosts: [ "httpbin.local.kyma.dev" ]
methods: ["GET"]
paths: ["/headers"]
selector:
matchLabels:
app: istio-ingressgatewayA request whose Host header includes the port 443 does not match the policy, so Istio denies it:
curl -k -H "Host: httpbin.local.kyma.dev:443" https://httpbin.local.kyma.dev/headersRBAC: access deniedSolution ​
To fix a 403 error caused by a port number in the Host header, follow these steps:
RFC 9110 and RFC 3986 define that HTTP clients remove the port if it is the default port for a given protocol. As a best practice, fix the client implementation.
If you cannot fix the client, modify the AuthorizationPolicy to also include the port number:
yamlapiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: ingress-allow-headers namespace: istio-system spec: action: ALLOW rules: - to: - operation: hosts: [ "httpbin.local.kyma.dev", "httpbin.local.kyma.dev:443" ] methods: ["GET"] paths: ["/headers"] selector: matchLabels: app: istio-ingressgateway
If the port number is not the cause, inspect the AuthorizationPolicy resources that Istio Ingress Gateway enforces.
Check the mesh for over-broad policies. The analyzer reports an
IST0133warning when aDENYpolicy uses only HTTP attributes and therefore blocks all TCP traffic in its scope:bashistioctl analyze --all-namespacesWarning [IST0133] (AuthorizationPolicy app-403/deny-no-port) Schema validation warning: configured AuthorizationPolicy will deny all traffic to TCP ports under its scope due to the use of only HTTP attributes in a DENY rule; it is recommended to explicitly specify the portPrint the AuthorizationPolicy list that the Envoy proxy applies to the Istio Ingress Gateway Pod. Unlike
kubectl get authorizationpolicies, which lists the resources you created, this command shows the policies that Envoy actually enforces, merged from the mesh, namespace, and workload levels.Get the name of the Istio Ingress Gateway Pod:
bashkubectl get pods --namespace istio-system --selector istio=ingressgateway --output nameCheck the policies applied to that Pod:
bashistioctl x authz check {POD_NAME}.istio-systemACTION AuthorizationPolicy RULES ALLOW ingress-allow-headers.istio-system 1If an
ALLOWpolicy applies, Envoy denies every request that does not match one of its rules. If aDENYpolicy applies, Envoy denies every request that matches it. Verify that the policy'sto,from, andwhenrules cover the request that you send.
For more information about debugging AuthorizationPolicy resources, see Network Connectivity - Diagnostics.
The 403 Comes from Your Application ​
Symptom ​
The request reaches your workload and the application returns the 403 error. The proxy still adds the server: istio-envoy header, but the response also carries the x-envoy-upstream-service-time header and your application's own body:
curl -sD - http://REDACTED/forbiddenHTTP/1.1 403 Forbidden
server: istio-envoy
content-type: application/octet-stream
content-length: 29
x-envoy-upstream-service-time: 0
forbidden by the applicationCompare this with a response from an AuthorizationPolicy. The distinguishing signal is the RBAC: access denied body, not the x-envoy-upstream-service-time header, which can also appear when a sidecar enforces the policy:
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy
RBAC: access deniedCause ​
Istio routes the request correctly, but the application rejects it with its own 403 response, for example because of its own authentication or authorization logic. Because the response originates in the workload, no Istio configuration change fixes it.
Solution ​
To confirm that the application returns the 403 error, bypass the mesh and send the request directly to the Pod. If the Pod still returns the 403 error, the application is the source, not Istio.
Forward a local port to the workload Pod:
bashkubectl port-forward deployment/{DEPLOYMENT_NAME} 18080:{CONTAINER_PORT} --namespace {NAMESPACE}In another terminal, send the request to the forwarded port and include the response headers:
bashcurl -sD - http://127.0.0.1:18080/{YOUR_PATH}HTTP/1.1 403 Forbidden Server: nginx/1.30.6 forbidden by the applicationThe
Serverheader now shows the application server, such asnginx, instead ofistio-envoy, because the request did not pass through the proxy. The403response still appears, which confirms that the application generates it.
Because the application returns the 403 error, debug the workload, not Istio. Check the application's authentication and authorization configuration, its logs, and any identity or token that it expects in the request.
The 403 Comes from Outside the Mesh ​
Symptom ​
The 403 response has no server: istio-envoy header, or carries a vendor-specific error page instead of RBAC: access denied. Something outside the mesh produced it.
Cause ​
An external firewall, proxy, or WAF in front of the cluster rejects the request before it reaches Envoy. This is outside the scope of Istio.
Solution ​
Because the block is outside the mesh, you cannot fix it by changing the Istio configuration. Contact your network administrator and provide the request details and the full response. To confirm that the traffic is blocked at the network layer rather than inside the mesh, see Check for an External Firewall Blocking Access.