Connection Refused Errors ​
Follow this troubleshooting guide if a connection to a workload in the service mesh fails before you receive an HTTP response. A refused or reset connection can come from mTLS or from a NetworkPolicy, and the fix differs in each case.
Identify the Origin of the Error ​
When mTLS resets a connection, you get a different error than when a NetworkPolicy blocks it. The decisive signal is whether the connection reaches the Envoy proxy at all.
Send a request to the failing workload and inspect the error. Compare your output with the following two cases:
- mTLS resets the connection: The connection fails with
Connection reset by peerorGOAWAYwhen a service without a sidecar connects to a service with a sidecar. The connection reached the Envoy proxy, but the proxy rejected it because the peer presented no valid TLS certificate. See mTLS Resets the Connection. - A NetworkPolicy blocks the connection: The connection fails with
Connection refusedor a connection timeout, with no503response and noserver: istio-envoyheader. A NetworkPolicy dropped the packets at the network layer, below Istio, so the connection never reached the Envoy proxy. See A NetworkPolicy Blocks the Connection.
NOTE: ​
Check the error message to help identify the cause. If mTLS causes the failure, the connection reaches the Envoy proxy, which resets it. If a NetworkPolicy blocks the traffic before it reaches the proxy, the connection is refused or times out. Neither case returns an HTTP response, so there is no
server: istio-envoyheader.
mTLS Resets the Connection ​
Symptom ​
You get either the Connection reset by peer response or the GOAWAY response when you attempt to establish the connection between a service without a sidecar and a service with a sidecar.
Cause ​
By default, mutual TLS (mTLS) is enabled in the service mesh. As a result, every element of the service mesh must have an Istio sidecar with a valid TLS certificate to allow communication.
Solution ​
- To add a service without a sidecar to the allowlist and disable mTLS traffic for it, create a DestinationRule resource. See DestinationRule.
- To allow connections between a service without a sidecar and a service with a sidecar, create a PeerAuthentication resource in the
PERMISSIVEmode. See Peer Authentication.
A NetworkPolicy Blocks the Connection ​
Symptom ​
A connection fails with Connection refused or a connection timeout, and no 503 response and no server: istio-envoy header appear. A NetworkPolicy operates at the network layer, below Istio. If it drops the packets, the request never reaches the Envoy proxy, so the connection times out or is refused instead of returning a mesh response.
Cause ​
By default, a Pod accepts traffic from any source. When you create a NetworkPolicy that selects a Pod, that Pod rejects all traffic that the policy does not explicitly allow. Two cases break a workload in the service mesh:
- A NetworkPolicy blocks traffic to or from the application. The policy selects an Istio-enabled workload and does not allow the traffic that you send. The request to the application times out, although Istio is configured correctly.
- A NetworkPolicy blocks a sidecar from reaching the Istio control plane. The policy restricts egress from an Istio-enabled workload and does not allow traffic to
istiodon the xDS port15012. The sidecar cannot fetch its configuration, so its routing becomes stale, and new connections fail.
Solution ​
Follow these steps to find out whether a NetworkPolicy causes the failure and how to fix it.
Check whether any NetworkPolicy resources are present. If none exist, you can rule out a NetworkPolicy as the cause:
bashkubectl get networkpolicies --all-namespacesNAMESPACE NAME POD-SELECTOR AGE kube-system gardener.cloud--deny-all <none> 4hThe
POD-SELECTORcolumn shows which Pods each policy selects. An empty selector, shown as<none>, selects every Pod in the policy's namespace, including Istio-enabled workloads.Find out whether a policy selects an Istio-enabled workload or an Istio component. First, confirm that the affected Pod runs a sidecar. A Pod with a sidecar reports two containers in the
READYcolumn, such as2/2:bashkubectl get pods --namespace {YOUR_NAMESPACE}Then check the policies in the Pod's namespace and the
istio-systemnamespace, and compare each policy'sspec.podSelectorwith the Pod's labels:bashkubectl get networkpolicies --namespace {YOUR_NAMESPACE} --output yaml kubectl get pods --namespace {YOUR_NAMESPACE} --selector {YOUR_SELECTOR} --show-labelsA policy applies to the Pod when its
spec.podSelectormatches the Pod's labels, or when the selector is empty.Check whether the sidecar still receives its configuration from
istiod. If a policy blocks egress to the control plane, the proxy cannot synchronize:bashistioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES httpbin-545d7f5848-4n2x8.netpol-demo Kubernetes istiod-54db84ffcf-mmspn 1.31.1 4 (CDS,LDS,EDS,RDS)A healthy proxy appears in the list, reports an
ISTIODinstance, and subscribes to theCDS,LDS,EDS, andRDSconfiguration types. If the proxy is missing from the output, reportsSTALE, or does not subscribe to all types, it cannot reachistiod. Verify that a NetworkPolicy allows egress from the workload toistiodon the following ports:15012for the xDS configuration and certificate signing15014for control plane monitoring
Adjust the NetworkPolicy resources. A NetworkPolicy only allows traffic, so you add rules until the required traffic is permitted:
- Allow ingress to the application from its clients on the application port.
- Allow egress from Istio-enabled workloads to
istiodin theistio-systemnamespace on ports15012and15014. - Allow egress to the cluster DNS service on port
53, so the sidecar can resolve service names.
As a best practice, follow these recommendations when you use NetworkPolicy resources with the service mesh:
- Start from a known baseline. Apply a default-deny policy, then add explicit allow rules for the traffic that you expect, instead of denying individual flows.
- Keep the control plane reachable. Always allow egress to
istiodon ports15012and15014from every Istio-enabled workload, and allow DNS egress on port53. - Scope policies with precise selectors. A policy with an empty
spec.podSelectorselects every Pod in the namespace, which can block Istio components unintentionally. - Verify the effect after each change. Run
istioctl proxy-statusto confirm that every sidecar still synchronizes withistiod.
Note: ​
A NetworkPolicy is enforced by the cluster's network plugin (CNI). If the plugin does not enforce NetworkPolicy resources, the policies have no effect on traffic. For more information about NetworkPolicy resources, see Network Policies.