← All posts
Infra

Istio/Envoy 503 (UF, UO, NR)? The response flag is the diagnosis, not the 503

For developers and autonomous AI agents · ~6 min read

Your mesh returns 503 Service Unavailable. You retry. Still 503. The problem is that a 503 in Istio can mean a dozen different things — and the HTTP status doesn't tell you which. The Envoy response flag does. It's the single most useful field in the access log, and reading it turns a mystery 503 into a specific fix.

Read the flag, not the 503

In Istio's default access-log format the response flag appears right after the status code: "GET /api HTTP/1.1" 503 UF. Pull it with kubectl logs deploy/<svc> -c istio-proxy | grep " 503 ". The flag narrows a 503 down to one cause:

UF / UH
Upstream connection failure / no healthy upstream. Often a Service port mismatch or an mTLS conflict — not the app.
UO
Upstream overflow: a DestinationRule circuit breaker tripped. The proxy budget is exhausted, not the backend.
NR / DC
No route / downstream termination: the VirtualService is missing or misconfigured. The request never reaches a backend.
URX
Retries exhausted — often combined (URX,UC) meaning retries ran out because connections kept terminating.

UF: it's usually the config, not the app

503 UF means Envoy couldn't connect to the upstream. Before you blame the backend, check two things: the Service targetPort vs the container's actual listening port (a mismatch here is the classic UF), and an mTLS policy conflict (a PeerAuthentication STRICT policy vs a caller outside the mesh). UH (no healthy upstream) instead means zero ready endpoints — check readiness probes and outlier-detection ejection in the DestinationRule.

UO: debug the proxy budget, not the app

503 UO looks like a dead backend but the backend is fine — the DestinationRule circuit breaker (connectionPool / maxPendingRequests) is FULL and Envoy is shedding load, often amplified by retries. The tell is curl 127.0.0.1:15000/stats | grep overflow on the istio-proxy showing upstream_cx_overflow climbing. Fix: raise the pool limits if the backend can take it, OR reduce retries. Do NOT just retry — retries make UO worse.

gRPC UNAVAILABLE / DEADLINE_EXCEEDED in a mesh

A gRPC UNAVAILABLE in a mesh usually maps from a 503 — the real cause is the Envoy flag, so correlate them. DEADLINE_EXCEEDED on a long-lived stream is frequently the Envoy UMSDR (max stream duration) cutting the stream, not a slow server. And only retry UNAVAILABLE if the RPC is idempotent — blind-retrying a mutating RPC can double-execute.

For agents operating a service mesh

An agent that retries a mesh 503 masks the config bug. diagnose_infra_error returns the service-mesh family and decodes the flag:

diagnose_infra_error({error: "503 UO upstream_cx_overflow circuit breaker destinationrule"})
// → { family: "service-mesh", action_class: "config",
//      fix: "circuit breaker tripped — raise the pool limit or cut retries; don't retry" }
For AI agents

Machine-readable guide: https://snapback.sh/llms.txt · Tool discovery: https://snapback.sh/.well-known/mcp.json · MCP endpoint: https://api.snapback.sh/mcp. Free tools need no token; pay-per-call diagnosis via x402 on Solana or EVM (~$0.0032 USDC).

Try Snapback freeRead the docs