Troubleshooting: traffic

Operations guide: configure, deploy and observe the gateway. What the feature does, product side: product guide; calling it: developer guide.

Routing and contract

Symptom	Cause	Action
404 on a route you believe is exposed	It is not in the contract. "Served = declared"	Add it to the contract, not to the configuration
400 "Query parameter ... is not declared"	Closed by default: the contract does not declare that parameter	Declare it in the contract, rather than going around the gateway
422 invalid_request	The request breaks the contract's schema	The errors extension names the field and the rule
A header does not reach the upstream	It is not declared in the contract - it is dropped, not refused	Declare it as an in: header parameter
Refused, "segment … has two parameters side by side" or "a '{' that is never closed"	A path segment the router cannot read	Put text between the two parameters, close the brace
Refused, "segments … rank the same and can match the same text"	Two mixed segments of one rank read the same text (a{x} and {x}a)	Make one more literal, or rename a path
404 on /reports/2026.pdf while /reports/{year}.pdf is in the contract	Gateway older than 1.6.0: the mixed segment was taken for a literal	Upgrade (Serving your first API)

Timeouts, streams and upstreams

Symptom	Cause	Action
A stream arrives all at once at the end	Something in the path is buffering - not the gateway	Check the edge and any proxy between the client and it
A stream stops midway, log read budget exhausted	The upstream went silent longer than server.timeouts.read	Have the upstream emit a heartbeat (: keepalive), or raise read
504 upstream_timeout	The upstream did not answer within the operation's budget	Read flexway_upstream_duration_seconds; raise x-flexway-timeout on that operation, not the global default
504 upstream_timeout "budget was spent before its upstream could be called"	A request-phase external-call consumed the operation's whole budget	Shorten its timeout, or raise the operation's x-flexway-timeout
503 upstream_unavailable with Retry-After	The upstream refuses connections: it is down	Check the upstream; the gateway already retried once if the method allowed it
502 upstream_error	The upstream answered something that is not usable HTTP	Look upstream, not at the gateway
503 with Flexway-Degraded: circuit-open	The upstream is failing; the breaker answers for it	Fix the upstream; one trial call goes through after cooldown
503 with Flexway-Degraded: upstream-saturated	max_in_flight reached on this upstream	Raise the cap, or find why the upstream slows down
503 overloaded with Flexway-Degraded: shedding, flexway_shed_total{reason="in_flight"} rising	The replica is at its in-flight limit (server.in_flight, derived from its memory limit unless written): it refuses the excess instead of being OOM-killed	Add replicas, or give each more CPU and memory; the limit in force is in /config (server.in_flight) and in the start log

gRPC

Symptom	Cause	Action
Refused at load "google.api.http ... not supported at this version"	A variable spanning segments, a wildcard or a verb in a REST rule	One variable per segment (/v1/shelves/{shelf}/books/{book}), or the method in gRPC only
Refused at load "is a gRPC method and relays to upstream ... which does not declare protocol: grpc"	The method falls on an HTTP upstream (the default)	Name its upstream in flexway.v1.operation.upstream, declared protocol: grpc
gRPC client: UNIMPLEMENTED	The method is not in the served set - a stale set	Rebuild the .binpb: it is re-read without a restart
REST: upstream_status	The gRPC upstream answered an error: its answer, not a gateway failure	Read the problem's grpc.status and grpc.details

Rate limits and quotas

Symptom	Cause	Action
429 rate_limited	The caller's bucket is empty	Read Retry-After; raise the class's rps/burst, or change the operation's class
All anonymous callers seem to share one bucket	The gateway sees the edge's address - or, behind a cloud load balancer without the proxy protocol, the load balancer's - not the client's	Declare the edge in server.trusted_proxies; read x-forwarded-for in the edge's access log, and turn the proxy protocol on at the load balancer and the edge (guide Setting quotas)

Response cache

Symptom	Cause	Action
Flexway-Cache: miss on every request	The upstream forbids keeping it (no-store, no-cache, private, Set-Cookie), answers without Content-Length, past server.cache.max_entry_bytes, or with a status other than 200, 203, 204, 301, 404	Read the upstream's response headers; flexway_cache_lookups_total per operation
No Flexway-Cache at all on a cached operation	A bypass: an identified caller on a public cache, a when condition that does not hold, a method other than GET/HEAD	result="bypass" in flexway_cache_lookups_total; a per-caller cache is declared scope: subject on the operation
Start refused: response cache declaration(s) refused	A cache on a stream, a WebSocket, gRPC, inference, a write, or a public one on an authenticated operation	The message names the operation and the door (x-flexway-cache or the policy)

Response body and status

Symptom	Cause	Action
502 response_too_large	An operation with a body policy received an answer past server.body.max_bytes	Raise server.body.max_bytes (reloadable), or page on the upstream's side
502 upstream_error on a transformed operation, log an upstream answer could not be transformed	The upstream answered something other than readable JSON, compressed despite Accept-Encoding: identity, or without a field projected with on_missing: fail	The log names the cause and the field; fix the upstream, or switch to on_missing: omit
Start refused: body policy attachment(s) refused	A body policy reaches a stream, a non-JSON response, gRPC or inference; or it drops a required field, keeps an undeclared one, masks with a refused type	The message names the operation and the policy; attach it more narrowly, or !name
flexway_policy_nonconformant_total rises	A served body no longer holds its status's schema (sampled) - the upstream changed, or the status is not declared	The departs from its schema log gives the field and the keyword; the response is served all the same
Load refused "body is read by the conditions of transform-body, redact and map-status only"	Another policy reads body in its condition or a template	Read the body in a body policy, or do without it

External call

Symptom	Cause	Action
403 (or another 4xx) with the code of an external-call policy	The external service refused: deny.when holds on its answer	Expected; the reason is the service's, never copied into the problem
504, 503 or 502 without the upstream being called, log external call failed_closed	The external service did not answer in time, is unreachable, has its circuit open or gave an answer deny.when / set_headers cannot read - and the policy is fail-closed	The log names the policy and the cause; flexway_policy_external_calls_total{outcome="failed_closed"}; fix the service, or fail-open if the decision may be missing
A header expected from an external-call is missing, log external call at WARN	fail-open absorbed a failure of the service	flexway_policy_external_calls_total{outcome="failed_open"}; the log gives the cause
Start refused: external-call attachment(s) refused	A timeout past an operation's budget, send.body on a stream, the response phase on a WebSocket, gRPC or inference	The message names the operation and the policy

Policies

Symptom	Cause	Action
Start refused, "is protected - the gateway alone writes it"	A policy writes a header only the gateway sets (identity, tracing, tenant, X-Forwarded-*)	Pick another header name
The upstream receives X-User-Roles: [seller admin]	Gateway older than 1.6.0: a list rendered the way Go prints it	Upgrade: a list arrives as seller,admin; {{ auth.claims.roles.join(' ') }} for another delimiter (Enriching a flow)
Start refused, undeclared reference to 'x' or found no matching overload in a value	A function the language does not have, or join with a delimiter that is not text	Only CEL's functions and join exist; fix the expression
Start refused, "on_missing is required"	A value is a {{ … }} expression with nothing said about a missing value	Add on_missing: skip, empty or fail
Start refused, "two … policies at the same level"	Two cors (or rate-limit, rewrite-path) at one level	Keep one, or move the other to a more precise level
500 internal_error, log a policy value is missing	on_missing: fail and the value is missing for this caller	Add a when condition, or switch to skip
A policy does nothing	The start log says NOT applied: its type belongs to a later version	Read effective_chains in /config