Skip to content

Diagnosing 403 Errors ​

Follow this troubleshooting guide if you receive the 403 Forbidden status code when trying to access a Kyma endpoint. A 403 can come from an AuthorizationPolicy inside the mesh, from your application, or from something outside the mesh, and the fix differs in each case.

Identify the Origin of the 403 Error ​

A 403 error from an AuthorizationPolicy, a 403 error from your application, and a 403 error from an external firewall look similar, but you can tell them apart by inspecting the response.

The decisive signal is the response body. An AuthorizationPolicy returns the fixed plain-text body RBAC: access denied, whereas your application returns its own payload, and an external component returns a vendor-specific page or nothing.

The x-envoy-upstream-service-time header is a secondary hint, not a reliable discriminator. It records the time an Envoy proxy waited for an upstream, so its presence depends on where the policy is enforced, not on whether an AuthorizationPolicy was involved:

  • A policy enforced at Istio Ingress Gateway rejects the request before the gateway selects an upstream, so the header is absent.
  • A policy enforced at the workload's sidecar is already the last Envoy hop, so the header is present and reports the time the gateway waited for the sidecar, even though no application code ran.

Because of this, use the response body to decide the origin, and treat the header only as a tiebreaker.

Send a request to the failing endpoint and include the response headers:

bash
curl -sD - http://{YOUR_HOST}/{YOUR_PATH}

Compare your output with the following cases:

AspectAuthorizationPolicy (Istio)ApplicationExternal Firewall, Proxy, or WAF
Where it runsInside the mesh, enforced by the Envoy proxyIn your workload, behind the Envoy proxyOutside the mesh, before the request reaches Envoy
server response headeristio-envoyistio-envoyAbsent, or a non-Envoy value such as a proxy or WAF product name
x-envoy-upstream-service-time headerAbsent for a gateway-enforced policy, present for a sidecar-enforced policyPresentAbsent
Response bodyPlain text RBAC: access deniedYour application's own payloadVendor-specific HTML or JSON error page, or empty
Depends on request identityYes. The decision depends on the to, from, and when rules, so it varies by path, source, and tokenDepends on the application logicNo. The block is usually the same regardless of path, headers, or credentials
Fixed byUpdating the AuthorizationPolicy resourceDebugging the applicationCloud provider or network administrator

To decide which case applies, use the following signals:

  • The response has the server: istio-envoy header and an RBAC: access denied body. An AuthorizationPolicy rejected the request, regardless of whether the x-envoy-upstream-service-time header is present. See The 403 Comes from an AuthorizationPolicy.
  • The response has the server: istio-envoy header and your application's own body, not RBAC: access denied. The x-envoy-upstream-service-time header is present. Your application generated the 403 error. See The 403 Comes from Your Application.
  • The response has no server: istio-envoy header, or it carries a vendor-specific error page instead of RBAC: access denied. Something other than Istio produced the 403 error, such as an external proxy or WAF in front of the cluster. This issue is outside the scope of Istio. See The 403 Comes from Outside the Mesh.

NOTE: ​

Use the response body to decide the origin, not the server header or the x-envoy-upstream-service-time header. The proxy adds server: istio-envoy to both an AuthorizationPolicy response and an application response, and the x-envoy-upstream-service-time header can appear on a sidecar-enforced AuthorizationPolicy response as well. Only the RBAC: access denied body uniquely identifies an AuthorizationPolicy rejection.

The 403 Comes from an AuthorizationPolicy ​

Symptom ​

The request reaches the mesh and the Envoy proxy returns the 403 error with the server: istio-envoy header and an RBAC: access denied body. When Istio Ingress Gateway enforces the policy, the response has no x-envoy-upstream-service-time header:

HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy

RBAC: access denied

When a workload's sidecar enforces the policy, the same RBAC: access denied body appears, but the response also carries an x-envoy-upstream-service-time header. The header reports the time the gateway waited for the sidecar, not time spent in your application. The RBAC: access denied body, not this header, identifies the AuthorizationPolicy as the source:

HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy
x-envoy-upstream-service-time: 4

RBAC: access denied

Cause ​

A misconfigured AuthorizationPolicy is a common cause of 403 errors. A frequent example is an unnecessary port number in the Host header. Istio checks the host as-is. If the Host header contains a port number and the AuthorizationPolicy defines only a hostname, Istio denies the request.

The following AuthorizationPolicy allows only the hostname httpbin.local.kyma.dev, without a port:

yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ingress-allow-headers
  namespace: istio-system
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        hosts: [ "httpbin.local.kyma.dev" ]
        methods: ["GET"]
        paths: ["/headers"]
  selector:
    matchLabels:
      app: istio-ingressgateway

A request whose Host header includes the port 443 does not match the policy, so Istio denies it:

bash
curl -k -H "Host: httpbin.local.kyma.dev:443" https://httpbin.local.kyma.dev/headers
RBAC: access denied

Solution ​

To fix a 403 error caused by a port number in the Host header, follow these steps:

  1. RFC 9110 and RFC 3986 define that HTTP clients remove the port if it is the default port for a given protocol. As a best practice, fix the client implementation.

  2. If you cannot fix the client, modify the AuthorizationPolicy to also include the port number:

    yaml
    apiVersion: security.istio.io/v1
    kind: AuthorizationPolicy
    metadata:
      name: ingress-allow-headers
      namespace: istio-system
    spec:
      action: ALLOW
      rules:
      - to:
        - operation:
            hosts: [ "httpbin.local.kyma.dev", "httpbin.local.kyma.dev:443" ]
            methods: ["GET"]
            paths: ["/headers"]
      selector:
        matchLabels:
          app: istio-ingressgateway

If the port number is not the cause, inspect the AuthorizationPolicy resources that Istio Ingress Gateway enforces.

  1. Check the mesh for over-broad policies. The analyzer reports an IST0133 warning when a DENY policy uses only HTTP attributes and therefore blocks all TCP traffic in its scope:

    bash
    istioctl analyze --all-namespaces
    Warning [IST0133] (AuthorizationPolicy app-403/deny-no-port) Schema validation warning: configured AuthorizationPolicy will deny all traffic to TCP ports under its scope due to the use of only HTTP attributes in a DENY rule; it is recommended to explicitly specify the port
  2. Print the AuthorizationPolicy list that the Envoy proxy applies to the Istio Ingress Gateway Pod. Unlike kubectl get authorizationpolicies, which lists the resources you created, this command shows the policies that Envoy actually enforces, merged from the mesh, namespace, and workload levels.

    Get the name of the Istio Ingress Gateway Pod:

    bash
    kubectl get pods --namespace istio-system --selector istio=ingressgateway --output name

    Check the policies applied to that Pod:

    bash
    istioctl x authz check {POD_NAME}.istio-system
    ACTION   AuthorizationPolicy                  RULES
    ALLOW    ingress-allow-headers.istio-system   1

    If an ALLOW policy applies, Envoy denies every request that does not match one of its rules. If a DENY policy applies, Envoy denies every request that matches it. Verify that the policy's to, from, and when rules cover the request that you send.

For more information about debugging AuthorizationPolicy resources, see Network Connectivity - Diagnostics.

The 403 Comes from Your Application ​

Symptom ​

The request reaches your workload and the application returns the 403 error. The proxy still adds the server: istio-envoy header, but the response also carries the x-envoy-upstream-service-time header and your application's own body:

bash
curl -sD - http://REDACTED/forbidden
HTTP/1.1 403 Forbidden
server: istio-envoy
content-type: application/octet-stream
content-length: 29
x-envoy-upstream-service-time: 0

forbidden by the application

Compare this with a response from an AuthorizationPolicy. The distinguishing signal is the RBAC: access denied body, not the x-envoy-upstream-service-time header, which can also appear when a sidecar enforces the policy:

HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
server: istio-envoy

RBAC: access denied

Cause ​

Istio routes the request correctly, but the application rejects it with its own 403 response, for example because of its own authentication or authorization logic. Because the response originates in the workload, no Istio configuration change fixes it.

Solution ​

To confirm that the application returns the 403 error, bypass the mesh and send the request directly to the Pod. If the Pod still returns the 403 error, the application is the source, not Istio.

  1. Forward a local port to the workload Pod:

    bash
    kubectl port-forward deployment/{DEPLOYMENT_NAME} 18080:{CONTAINER_PORT} --namespace {NAMESPACE}
  2. In another terminal, send the request to the forwarded port and include the response headers:

    bash
    curl -sD - http://127.0.0.1:18080/{YOUR_PATH}
    HTTP/1.1 403 Forbidden
    Server: nginx/1.30.6
    
    forbidden by the application

    The Server header now shows the application server, such as nginx, instead of istio-envoy, because the request did not pass through the proxy. The 403 response still appears, which confirms that the application generates it.

Because the application returns the 403 error, debug the workload, not Istio. Check the application's authentication and authorization configuration, its logs, and any identity or token that it expects in the request.

The 403 Comes from Outside the Mesh ​

Symptom ​

The 403 response has no server: istio-envoy header, or carries a vendor-specific error page instead of RBAC: access denied. Something outside the mesh produced it.

Cause ​

An external firewall, proxy, or WAF in front of the cluster rejects the request before it reaches Envoy. This is outside the scope of Istio.

Solution ​

Because the block is outside the mesh, you cannot fix it by changing the Istio configuration. Contact your network administrator and provide the request details and the full response. To confirm that the traffic is blocked at the network layer rather than inside the mesh, see Check for an External Firewall Blocking Access.