Service Mesh Explained: Sidecars, mTLS, Retries and When You Need One

Calls between services fail in ways a function call never does. They time out, reach a copy of the service that just stopped, travel unencrypted, and leave no trace of where the time went. A service mesh answers all of these in one layer.

Picture a common setup. A food-delivery app runs on Kubernetes, and its orders service calls a payments service for every checkout. The setup is made up, but each failure below is real and appears in our tests later.

• A payments copy restarts during a deploy, and a few checkouts fail with errors the user can see.
• Payments gets slow, orders waits with no time limit, and soon every orders thread is stuck waiting.
• Traffic inside the cluster is plain HTTP, so anything that can read the network can read card tokens.
• A new payments version ships to everyone at once, and a bug hits every user instead of a few.
• Checkout feels slow, and nobody can say whether orders, payments or the network is to blame.

Each team can fix these in its own code. That works for two services in one language. It gets hard with forty services in four languages, because every team must build the same retries, certificates and metrics, and keep them identical.

You need six words. Each one comes back in every later section.

• Proxy: a program that receives a network request and forwards it to the real destination. It can change, retry or record the request on the way.
• Sidecar: a helper container that runs inside the same Kubernetes pod as your app, like a sidecar attached to a motorbike. A pod is the smallest unit Kubernetes runs.
• Data plane: the proxies that actually carry the traffic.
• Control plane: the program that tells every proxy what to do and hands out certificates. In Istio it is called istiod.
• mTLS (mutual TLS): normal TLS, the encryption behind HTTPS, where the client also shows a certificate. Both sides prove their identity, not just the server.
• L4 and L7: Layer 4 means connections and bytes, with no idea of HTTP. Layer 7 means HTTP requests, with paths, headers and status codes.

A service mesh is a set of proxies that every service-to-service call passes through, plus a control plane that configures them. Your app keeps making plain calls, and the proxies add security and reliability on the way.

In the classic sidecar design, each app pod gets an extra proxy container. Network rules inside the pod send all its traffic through that proxy. In Istio, these rules come from istio-init, an init container that runs once before the app starts, or from a per-node CNI agent, as of Istio 1.31.

Follow one checkout call in the diagram. The orders app thinks it is calling payments directly. Its sidecar intercepts the call, opens an mTLS connection to the payments sidecar, and the payments sidecar hands the request to the payments app.

Think of it like an office building where every company has its own reception desk. You hand a parcel to your desk, it checks the other desk's ID, sends the parcel, and logs it. The building manager gives every desk the same rule book.

The mapping: your desk is the orders sidecar, the other desk is the payments sidecar, the ID check is mTLS, and the building manager is the control plane. Where the picture breaks: real reception desks are slow, while a proxy adds well under a millisecond, as our measurements show later.

The most common open-source meshes are Istio and Linkerd. Both are graduated projects of the CNCF, the foundation that also hosts Kubernetes. Istio's sidecars use the Envoy proxy, while Linkerd uses its own small proxy written in Rust.

Kubernetes finds and load balances your pods; a mesh controls what happens on each call. Service discovery tells orders where payments lives. The mesh decides how the call is secured, retried, timed, split and measured.

A service account is the Kubernetes identity a pod runs as. The mesh turns it into a certificate, so payments can say: only the orders service may call me.

A sidecar mesh puts one full proxy in every pod, while Istio's ambient mode shares proxies per machine and adds HTTP-level proxies only where you ask for them. Both give you mTLS; they differ in cost, isolation and features.

Sidecars have real costs. Each pod pays for a proxy sized for its busiest moment. Sidecars are added when a pod is created, so joining the mesh or upgrading the proxy means restarting your pods.

How Istio ambient mode works

Ambient mode splits the proxy into two layers. A ztunnel, short for zero-trust tunnel, runs once per node, which is one machine in the cluster. Zero trust means no caller is trusted just for being inside the network. It handles only L4 work: mTLS, simple connection-level allow or deny rules, and connection metrics.

The ztunnel does not read HTTP at all. For retries, timeouts, traffic splitting or path-based rules, you deploy a waypoint proxy. A waypoint is an Envoy deployment that runs outside your app pods, often one per namespace. A namespace is a named group of resources inside a cluster.

Ztunnels talk to each other over HBONE, an mTLS tunnel built on the HTTP CONNECT method. You add a namespace with the label istio.io/dataplane-mode=ambient, and Istio's docs say no app restart is needed.

Two details are easy to miss. Since Linkerd's open-source project stopped shipping stable builds in February 2024, its stable releases come from vendors, while the project ships edge builds. On ambient waypoints, Istio's feature list marks VirtualService as Alpha, while routes from the Gateway API, a newer Kubernetes routing standard, are Stable.

Each workload gets a short-lived certificate that names its identity, and the proxies check each other's certificates on every new connection. The app never touches a key.

The identity follows the SPIFFE standard, an open spec for workload identity. A SPIFFE ID is a URI such as spiffe://cluster.local/ns/shop/sa/orders. Istio builds it from the trust domain, the namespace and the service account.

The certificate carries that URI in its subject alternative name field, the part of a certificate that lists the names it is valid for. The X.509-SVID spec says a certificate must hold exactly one such URI.

• Issuing: the Istio agent in each pod makes a private key and asks istiod to sign it, as of 1.31.
• Rotating: the agent requests 24-hour certificates by default and renews them before expiry. Linkerd's docs also give 24 hours.
• Enforcing: Istio's PeerAuthentication policy has three modes. PERMISSIVE accepts mTLS and plain text, STRICT accepts only mTLS, and DISABLE turns it off.
• Default: when no mode is set, Istio uses PERMISSIVE, so plain-text callers still get through until you switch to STRICT.

What we saw when we ran it

We could not run a Kubernetes cluster in our test box, so we rebuilt the mechanism with two Envoy 1.39.1 proxies. One acted as the orders sidecar and one as the payments sidecar, with a throwaway certificate authority and SPIFFE IDs.

The payments proxy was told to accept only the orders identity. This is the key part of its config.

# Payments sidecar listener: require a client certificate
# and accept only the orders SPIFFE ID (Envoy 1.39.1).
require_client_certificate: true
common_tls_context:
  validation_context:
    trusted_ca: {filename: certs/ca.crt}
    match_typed_subject_alt_names:
    - san_type: URI
      matcher: {exact: "spiffe://cluster.local/ns/shop/sa/orders"}

Row three is the one to remember. Encryption alone is not access control. A certificate from your own mesh is not enough; the receiving side must also check which identity it names, which is what Istio's AuthorizationPolicy does in a real cluster.

The caller's proxy watches each response, retries failures that match a rule, and gives up when a time budget runs out. Your code sees one final answer.

We put a test payments service behind Envoy 1.39.1. On one path, every odd-numbered request returns 503, meaning service unavailable. On another, it sleeps for 3 seconds before replying. This is the route rule we used.

# Envoy 1.39.1 route: total budget 1s, up to 2 retries,
# each attempt limited to 0.4s, retry on any 5xx or no response.
route:
  cluster: payments
  timeout: 1s
  retry_policy:
    retry_on: "5xx"
    num_retries: 2
    per_try_timeout: 0.4s

The fourth row shows a rule from Envoy's docs: the route timeout includes all retries. Retries never stretch your 1-second budget. Without a per-try timeout, a slow first attempt uses the whole budget and nothing is retried.

The retry that charged twice

Now a POST to the payments path, which also fails on odd-numbered attempts. Pause and predict: the caller gets a 200, but how many payment requests did the payments service receive?

It received two. The first attempt reached the app, which returned 503; Envoy retried, and the second attempt succeeded. Envoy's router docs list no rule that skips POST, and our run confirms it retried one.

What just happened? A 503 does not prove the work was not done. If the payments app had charged the card before failing, the retry would charge again. Linkerd's docs say plainly that retries should only be used on idempotent methods, ones that are safe to repeat.

How retries multiply across layers

Next we chained two Envoy proxies, each set to 2 retries, in front of a path that always fails. One client call produced 9 attempts at the failing service. A single layer produced 3. Each layer multiplies, so by the same arithmetic three such layers would give 27.

Linkerd's docs warn that constantly retried requests can take a struggling service down instead of letting it recover. The guide Retry, Backoff and Jitter: How to Avoid a Retry Storm covers budgets that cap this. In a mesh, retry at one layer only, usually the caller closest to the user.

What Istio and Linkerd do by default

Istio retries HTTP requests twice by default and sets no request timeout, although plain Envoy defaults to a 15-second route timeout. The 1.31 reference lists the default triggers as connection failures, refused streams, and the gRPC statuses unavailable and cancelled. gRPC is a remote-call framework that runs over HTTP/2.

That list has no HTTP status codes. Reading the 1.31 source, the default also enables Envoy's retriable-status-codes trigger but supplies no codes, so we found nothing that retries a plain 503. We read this in source and did not run it in a cluster, so check your own version.

Linkerd configures retries and timeouts with annotations such as retry.linkerd.io/http and timeout.linkerd.io/request, on routes or services. Since Linkerd 2.16, a request that times out can be retried. Both apply on the calling side, so the caller must be in the mesh.

The proxy picks a destination for each request by weight, so you can send a fixed share of real traffic to a new version. A canary release means sending a small share to the new version first and watching it.

We ran Envoy 1.39.1 with two payments versions weighted 90 and 10 and sent 1,000 requests twice. Version 1 got 914 then 899, and version 2 got 86 then 101. The split is applied request by request, so short runs wobble around the target.

In Istio you write the same rule as a VirtualService with weights, plus a DestinationRule that defines the two versions by pod label. We checked this YAML with istioctl 1.31.3 validate, which checks the schema offline; we did not deploy it to a cluster.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payments
  namespace: shop
spec:
  hosts:
  - payments.shop.svc.cluster.local
  http:
  - timeout: 1s
    retries:
      attempts: 2
      perTryTimeout: 400ms
      retryOn: 5xx
    route:
    - destination:
        host: payments.shop.svc.cluster.local
        subset: v1
      weight: 90
    - destination:
        host: payments.shop.svc.cluster.local
        subset: v2
      weight: 10

Linkerd does the same with an HTTPRoute whose backend list carries weights, and its docs show the same 90 and 10 example. Whichever mesh you use, move the weight in steps such as 10, 50 and 100, checking error rate and latency for version 2 at each step.

Every proxy counts requests, errors and latency for every call it carries, so you get the same core metrics for every service. Tracing is the exception: apps must still pass a few headers along.

Istio's standard metrics include istio_requests_total and istio_request_duration_milliseconds. Labels such as source_workload, destination_workload and response_code let you ask how many orders-to-payments calls failed. Linkerd's telemetry gives success rate, request rate and latency percentiles, with no developer work.

A trace is the full path of one request across services, built from timed pieces called spans. Proxies send spans automatically, but Istio's docs say apps must forward headers such as traceparent and x-request-id to join them into one trace.

One counter detail caught us out. After the retry tests, Envoy's upstream_rq_503 counter showed only the 5 errors the client saw. The 11 retried 503s appeared under retry.upstream_rq_503, which matches Envoy's stats docs. Alert on both, or retries will hide a sick service.

Each proxy hop adds a fraction of a millisecond, but the bigger cost is people's time. Someone must upgrade, debug and secure the mesh itself.

We measured 3,000 keep-alive GET requests, three times, on a 2-core Linux box over loopback, the machine's internal network. Direct calls had a median of 0.50 to 0.72 milliseconds. Through one Envoy, it was 0.76 to 0.84.

Through two Envoys with mTLS between them, the sidecar-to-sidecar path, the median was 0.96 to 1.17 milliseconds. Across all three paths, the 99th percentile, the slowest 1 in 100 requests, ranged from 2.6 to 6.7 milliseconds and was too noisy to compare.

Treat these as a rough size, not a benchmark. Real clusters add real network hops, bigger payloads and busier CPUs. If your service answers in 50 milliseconds, half a millisecond rarely matters; if a request crosses ten services, the hops add up.

• Memory and CPU: sidecar mode runs one proxy per pod. Istio's own comparison rates ambient's average resource cost as small and sidecar's as large.
• Upgrades: Istio supports each minor version for a limited time. Istio 1.29 support ended on 12 October 2026, so plan an upgrade every few months.
• Security fixes: Istio published two security bulletins in October 2026 alone, each needing a patch release.
• Debugging: a failed call can now fail in the app, either proxy, the control plane or the policy, and the team must know how to tell which.

A mesh earns its keep when many services, often in several languages, need the same encryption, identity and traffic rules, and a team can own it. For a handful of services, a library or a gateway is usually enough.

Advice, not a rule: if you choose a mesh, start with mTLS in PERMISSIVE mode, watch the metrics, then switch to STRICT one namespace at a time. Add HTTP rules later, one service at a time. The guide Circuit Breaker Pattern: Fail Fast When a Dependency Is Sick covers the in-code option.

Between the two, Istio offers two data-plane modes and Envoy's wide feature set. Linkerd's proxy is built only for the mesh and turns mTLS on by default. Pick the one whose docs answer your top three needs, then test it on one non-critical service first.

Lead with the problem, then the mechanism, then the trade-off. Use these steps as your frame.

1. 1

   Name the problem

   Many services, many languages, and every call needs encryption, retries, timeouts and metrics. Doing it in each codebase drifts.

2. 2

   Describe the data plane

   Proxies carry every call: a sidecar per pod, or per-node ztunnels plus waypoints in Istio ambient mode.

3. 3

   Describe the control plane

   It pushes routing rules to proxies and issues short-lived certificates with SPIFFE identities for mTLS.

4. 4

   Walk one request

   Orders calls payments, its proxy opens mTLS, checks identity, applies a 1-second timeout and 2 retries, and records metrics.

5. 5

   State the risks

   Retries can repeat non-idempotent calls and multiply across layers. Each hop adds a little latency, and someone must run the mesh.

6. 6

   Say when you would skip it

   With a few services in one language, a client library plus an API gateway is simpler. Add a mesh when the need is cluster-wide.