# Troubleshooting: traffic

> Routes, timeouts, streams, upstreams, gRPC, rate limits, cache, bodies, external call and policies: the symptom, its cause and what to do.

Troubleshooting - Since version 0.1.0 - Flexway v1.7.1 - https://flexway.io/en/docs/operations-troubleshooting-traffic/

> Operations guide: configure, deploy and observe the gateway. What the feature does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

## Routing and contract

| Symptom | Cause | Action |
| --- | --- | --- |
| `404` on a route you believe is exposed | It is not in the contract. "Served = declared" | Add it to the contract, not to the configuration |
| `400` "Query parameter ... is not declared" | Closed by default: the contract does not declare that parameter | Declare it in the contract, rather than going around the gateway |
| `422` `invalid_request` | The request breaks the contract's schema | The `errors` extension names the field and the rule |
| A header does not reach the upstream | It is not declared in the contract - it is dropped, not refused | Declare it as an `in: header` parameter |
| Refused, "segment … has two parameters side by side" or "a '{' that is never closed" | A path segment the router cannot read | Put text between the two parameters, close the brace |
| Refused, "segments … rank the same and can match the same text" | Two mixed segments of one rank read the same text (`a{x}` and `{x}a`) | Make one more literal, or rename a path |
| `404` on `/reports/2026.pdf` while `/reports/{year}.pdf` is in the contract | Gateway older than `1.6.0`: the mixed segment was taken for a literal | Upgrade ([Serving your first API](https://flexway.io/en/docs/operations-first-api/)) |

## Timeouts, streams and upstreams

| Symptom | Cause | Action |
| --- | --- | --- |
| A stream arrives all at once at the end | Something in the path is buffering - not the gateway | Check the edge and any proxy between the client and it |
| A stream stops midway, log `read budget exhausted` | The upstream went silent longer than `server.timeouts.read` | Have the upstream emit a heartbeat (`: keepalive`), or raise `read` |
| `504` `upstream_timeout` | The upstream did not answer within the operation's budget | Read `flexway_upstream_duration_seconds`; raise `x-flexway-timeout` on **that** operation, not the global default |
| `504` `upstream_timeout` "budget was spent before its upstream could be called" | A request-phase `external-call` consumed the operation's whole budget | Shorten its `timeout`, or raise the operation's `x-flexway-timeout` |
| `503` `upstream_unavailable` with `Retry-After` | The upstream refuses connections: it is down | Check the upstream; the gateway already retried once if the method allowed it |
| `502` `upstream_error` | The upstream answered something that is not usable HTTP | Look upstream, not at the gateway |
| `503` with `Flexway-Degraded: circuit-open` | The upstream is failing; the breaker answers for it | Fix the upstream; one trial call goes through after `cooldown` |
| `503` with `Flexway-Degraded: upstream-saturated` | `max_in_flight` reached on this upstream | Raise the cap, or find why the upstream slows down |
| `503` `overloaded` with `Flexway-Degraded: shedding`, `flexway_shed_total{reason="in_flight"}` rising | The replica is at its in-flight limit (`server.in_flight`, derived from its memory limit unless written): it refuses the excess instead of being OOM-killed | Add replicas, or give each more CPU and memory; the limit in force is in `/config` (`server.in_flight`) and in the start log |

## gRPC

| Symptom | Cause | Action |
| --- | --- | --- |
| Refused at load "google.api.http ... not supported at this version" | A variable spanning segments, a wildcard or a verb in a REST rule | One variable per segment (`/v1/shelves/{shelf}/books/{book}`), or the method in gRPC only |
| Refused at load "is a gRPC method and relays to upstream ... which does not declare protocol: grpc" | The method falls on an HTTP upstream (the `default`) | Name its upstream in `flexway.v1.operation.upstream`, declared `protocol: grpc` |
| gRPC client: `UNIMPLEMENTED` | The method is not in the served set - a stale set | Rebuild the `.binpb`: it is re-read without a restart |
| REST: `upstream_status` | The gRPC upstream answered an error: its answer, not a gateway failure | Read the problem's `grpc.status` and `grpc.details` |

## Rate limits and quotas

| Symptom | Cause | Action |
| --- | --- | --- |
| `429` `rate_limited` | The caller's bucket is empty | Read `Retry-After`; raise the class's `rps`/`burst`, or change the operation's class |
| All anonymous callers seem to share one bucket | The gateway sees the edge's address - or, behind a cloud load balancer without the proxy protocol, the load balancer's - not the client's | Declare the edge in `server.trusted_proxies`; read `x-forwarded-for` in the edge's access log, and turn the proxy protocol on at the load balancer and the edge (guide [Setting quotas](https://flexway.io/en/docs/operations-quotas/)) |

## Response cache

| Symptom | Cause | Action |
| --- | --- | --- |
| `Flexway-Cache: miss` on every request | The upstream forbids keeping it (`no-store`, `no-cache`, `private`, `Set-Cookie`), answers without `Content-Length`, past `server.cache.max_entry_bytes`, or with a status other than `200`, `203`, `204`, `301`, `404` | Read the upstream's response headers; `flexway_cache_lookups_total` per operation |
| No `Flexway-Cache` at all on a cached operation | A bypass: an identified caller on a public cache, a `when` condition that does not hold, a method other than `GET`/`HEAD` | `result="bypass"` in `flexway_cache_lookups_total`; a per-caller cache is declared `scope: subject` on the operation |
| Start refused: `response cache declaration(s) refused` | A cache on a stream, a WebSocket, gRPC, inference, a write, or a public one on an authenticated operation | The message names the operation and the door (`x-flexway-cache` or the policy) |

## Response body and status

| Symptom | Cause | Action |
| --- | --- | --- |
| `502` `response_too_large` | An operation with a body policy received an answer past `server.body.max_bytes` | Raise `server.body.max_bytes` (reloadable), or page on the upstream's side |
| `502` `upstream_error` on a transformed operation, log `an upstream answer could not be transformed` | The upstream answered something other than readable JSON, compressed despite `Accept-Encoding: identity`, or without a field projected with `on_missing: fail` | The log names the cause and the field; fix the upstream, or switch to `on_missing: omit` |
| Start refused: `body policy attachment(s) refused` | A body policy reaches a stream, a non-JSON response, gRPC or inference; or it drops a required field, keeps an undeclared one, masks with a refused type | The message names the operation and the policy; attach it more narrowly, or `!name` |
| `flexway_policy_nonconformant_total` rises | A served body no longer holds its status's schema (sampled) - the upstream changed, or the status is not declared | The `departs from its schema` log gives the field and the keyword; the response is served all the same |
| Load refused "`body` is read by the conditions of transform-body, redact and map-status only" | Another policy reads `body` in its condition or a template | Read the body in a body policy, or do without it |

## External call

| Symptom | Cause | Action |
| --- | --- | --- |
| `403` (or another `4xx`) with the `code` of an `external-call` policy | The external service refused: `deny.when` holds on its answer | Expected; the reason is the service's, never copied into the problem |
| `504`, `503` or `502` without the upstream being called, log `external call` `failed_closed` | The external service did not answer in time, is unreachable, has its circuit open or gave an answer `deny.when` / `set_headers` cannot read - and the policy is `fail-closed` | The log names the policy and the cause; `flexway_policy_external_calls_total{outcome="failed_closed"}`; fix the service, or `fail-open` if the decision may be missing |
| A header expected from an `external-call` is missing, log `external call` at `WARN` | `fail-open` absorbed a failure of the service | `flexway_policy_external_calls_total{outcome="failed_open"}`; the log gives the cause |
| Start refused: `external-call attachment(s) refused` | A `timeout` past an operation's budget, `send.body` on a stream, the `response` phase on a WebSocket, gRPC or inference | The message names the operation and the policy |

## Policies

| Symptom | Cause | Action |
| --- | --- | --- |
| Start refused, "is protected - the gateway alone writes it" | A policy writes a header only the gateway sets (identity, tracing, tenant, `X-Forwarded-*`) | Pick another header name |
| The upstream receives `X-User-Roles: [seller admin]` | Gateway older than `1.6.0`: a list rendered the way Go prints it | Upgrade: a list arrives as `seller,admin`; `{{ auth.claims.roles.join(' ') }}` for another delimiter ([Enriching a flow](https://flexway.io/en/docs/operations-policies/)) |
| Start refused, `undeclared reference to 'x'` or `found no matching overload` in a value | A function the language does not have, or `join` with a delimiter that is not text | Only CEL's functions and `join` exist; fix the expression |
| Start refused, "on_missing is required" | A value is a `{{ … }}` expression with nothing said about a missing value | Add `on_missing: skip`, `empty` or `fail` |
| Start refused, "two … policies at the same level" | Two `cors` (or `rate-limit`, `rewrite-path`) at one level | Keep one, or move the other to a more precise level |
| `500` `internal_error`, log `a policy value is missing` | `on_missing: fail` and the value is missing for this caller | Add a `when` condition, or switch to `skip` |
| A policy does nothing | The start log says `NOT applied`: its type belongs to a later version | Read `effective_chains` in `/config` |