# Troubleshooting

> The console's footer (sign-in page included) shows both; or flexway version (docker run --rm ghcr.io/flexwiz/flexway:<version> version), build in the admin…

Troubleshooting - Since version 0.1.0 - Flexway v1.6.3 - https://flexway.io/en/docs/operations-troubleshooting/

> Operations guide: configure, deploy and observe the gateway. What the capability does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

| Symptom | Likely cause | What to do |
| --- | --- | --- |
| Not sure which version runs | - | The console's footer (sign-in page included) shows both; or `flexway version` (`docker run --rm ghcr.io/flexwiz/flexway:<version> version`), `build` in the admin `/config`. `dev` means a build that is not a release: its commit says which one |
| `flexway validate`: `x-flexway-sunset ... is set on an operation that is not deprecated` | A withdrawal date on an operation (or inherited from `info`) that is not deprecated | Add `deprecated: true` (in gRPC `option deprecated = true;`), or remove the date |
| Start refused: `kms.key "...": kms key absent or unusable` | The field-encryption key does not exist in the KMS (a typo) or cannot wrap (a Vault key not derived) | Fix `kms.key` or create the key - [Field encryption](https://flexway.io/en/docs/operations-kafka-encryption/) |
| Kafka consumers get `CORRUPT_MESSAGE` after `kms.key` changed | Records written before name the former key | Declare it in `kms.previous_keys` - [Field encryption](https://flexway.io/en/docs/operations-kafka-encryption/) |
| Start refused: `reads the schema registry's wire format and is isolated by the tenant attribute alone` | A consumption of a channel in the registry's wire format filtered on the `tenant` attribute, without a binding: an ordinary Kafka producer's messages have no attribute | Bind the channel to the tenant (`x-flexway-bind: tenant`, address `{tenant}.<topic>`), or declare it public (`x-flexway-public: true`) |
| `POST` on an `x-flexway-publish` door: `504 upstream_timeout` | The broker did not acknowledge within the budget: **unknown outcome**, the message may have been written | Send again with the same `Idempotency-Key`; check the broker's health, widen `timeout` |
| `503 overloaded` on a publication | The broker's `max_in_flight` buffer is full: it acknowledges slower than callers publish | Read `flexway_async_publish_duration_seconds`; raise `max_in_flight` or the broker's capacity |
| `502 upstream_error` on a publication | The broker refused: ACL, missing topic (the gateway never creates one), message rejected; AMQP: no queue takes the message | Create the tenant's topic, grant the gateway's principal write access, bind a queue to the exchange |
| Start refused: `is consumed and isolates no tenant` | A consumption on a channel neither bound to the tenant nor filtered on the `tenant` header | `x-flexway-bind: tenant` on a channel parameter, or the `tenant` header in every message's `headers` |
| A WebSocket subscriber receives a message twice | At least once: no acknowledgement within `ack_timeout`, or a cut before it | Acknowledge each `id` (`{"ack": "<id>"}`); deduplicate on `source` + `id` |
| `flexway validate` fails on "operation without upstream" | An operation was added to the contract without `x-flexway-upstream`, and there is no `default` upstream | Add the extension, or declare a `default` upstream |
| `flexway import krakend` exits with `3` | Elements of the imported configuration have no equivalent; the rest is written | Read the "No equivalent" section of `coverage.md`: each line says why and, when there is one, the equivalent to write by hand |
| `flexway import krakend`: `the converted configuration does not load - nothing written` | The result would not start: a contract operation no endpoint covers, an aggregation, `sd: dns`, a validator without `audience`, a policy the contract names and nothing defines | The loader's message names the operation or the key; set `x-flexway-upstream` in the contract or fix it, then run again |
| `flexway import krakend`: `is not empty: the import never overwrites` | `--out` already holds files | An empty or absent directory |
| An endpoint of the imported configuration is missing after the import | The contract does not declare it: it is listed in `coverage.md`, never invented | Declare it in the contract, run the import again |
| The gateway refuses to start | Partial configuration - this is deliberate | Read the message: it names the offending operation or upstream |
| `404` on a route you believe is exposed | It is not in the contract. "Served = declared" | Add it to the contract, not to the configuration |
| A stream arrives all at once at the end | Something in the path is buffering - not the gateway | Check the edge and any proxy between the client and it |
| A stream stops midway, log `read budget exhausted` | The upstream went silent longer than `server.timeouts.read` | Have the upstream emit a heartbeat (`: keepalive`), or raise `read` |
| `504` `upstream_timeout` | The upstream did not answer within the operation's budget | Read `flexway_upstream_duration_seconds`; raise `x-flexway-timeout` on **that** operation, not the global default |
| `Flexway-Cache: miss` on every request | The upstream forbids keeping it (`no-store`, `no-cache`, `private`, `Set-Cookie`), answers without `Content-Length`, past `server.cache.max_entry_bytes`, or with a status other than `200`, `203`, `204`, `301`, `404` | Read the upstream's response headers; `flexway_cache_lookups_total` per operation |
| No `Flexway-Cache` at all on a cached operation | A bypass: an identified caller on a public cache, a `when` condition that does not hold, a method other than `GET`/`HEAD` | `result="bypass"` in `flexway_cache_lookups_total`; a per-caller cache is declared `scope: subject` on the operation |
| Start refused: `response cache declaration(s) refused` | A cache on a stream, a WebSocket, gRPC, inference, a write, or a public one on an authenticated operation | The message names the operation and the door (`x-flexway-cache` or the policy) |
| `502` `response_too_large` | An operation with a body policy received an answer past `server.body.max_bytes` | Raise `server.body.max_bytes` (reloadable), or page on the upstream's side |
| `502` `upstream_error` on a transformed operation, log `an upstream answer could not be transformed` | The upstream answered something other than readable JSON, compressed despite `Accept-Encoding: identity`, or without a field projected with `on_missing: fail` | The log names the cause and the field; fix the upstream, or switch to `on_missing: omit` |
| Start refused: `body policy attachment(s) refused` | A body policy reaches a stream, a non-JSON response, gRPC or inference; or it drops a required field, keeps an undeclared one, masks with a refused type | The message names the operation and the policy; attach it more narrowly, or `!name` |
| `flexway_policy_nonconformant_total` rises | A served body no longer holds its status's schema (sampled) - the upstream changed, or the status is not declared | The `departs from its schema` log gives the field and the keyword; the response is served all the same |
| Load refused "`body` is read by the conditions of transform-body, redact and map-status only" | Another policy reads `body` in its condition or a template | Read the body in a body policy, or do without it |
| `403` (or another `4xx`) with the `code` of an `external-call` policy | The external service refused: `deny.when` holds on its answer | Expected; the reason is the service's, never copied into the problem |
| `504`, `503` or `502` without the upstream being called, log `external call` `failed_closed` | The external service did not answer in time, is unreachable, has its circuit open or gave an answer `deny.when` / `set_headers` cannot read - and the policy is `fail-closed` | The log names the policy and the cause; `flexway_policy_external_calls_total{outcome="failed_closed"}`; fix the service, or `fail-open` if the decision may be missing |
| A header expected from an `external-call` is missing, log `external call` at `WARN` | `fail-open` absorbed a failure of the service | `flexway_policy_external_calls_total{outcome="failed_open"}`; the log gives the cause |
| `504` `upstream_timeout` "budget was spent before its upstream could be called" | A request-phase `external-call` consumed the operation's whole budget | Shorten its `timeout`, or raise the operation's `x-flexway-timeout` |
| Start refused: `external-call attachment(s) refused` | A `timeout` past an operation's budget, `send.body` on a stream, the `response` phase on a WebSocket, gRPC or inference | The message names the operation and the policy |
| `503` `upstream_unavailable` with `Retry-After` | The upstream refuses connections: it is down | Check the upstream; the gateway already retried once if the method allowed it |
| `502` `upstream_error` | The upstream answered something that is not usable HTTP | Look upstream, not at the gateway |
| `401` on valid tokens right after a start | The issuer's keys are not loaded yet | The log says `issuer not reachable yet`; check the gateway reaches the discovery URL |
| `401` `invalid_token` on a token that works elsewhere | Audience, issuer or algorithm: the reason is in the log (`token refused`), never in the response | Compare the token's `aud` and `iss` with `auth.issuers` |
| Inference: `401` `tenant_required` despite the ingress's headers | The peer is not in `server.trusted_proxies`: its headers are not believed | Declare the ingress's range, and only it |
| Inference: `429` `budget_exceeded` | The tenant's budget (day or month) is spent - a request pre-books `max_tokens`, 512 without it | Raise `ai.budget` or the tenant's override; the counter is in `ai.budget.store` |
| Inference: `429` `rate_limited` for every tenant behind the ingress | A rate-limit policy too tight on `/v1/*` | A policy of its own on `routes: { "/v1/**": [...] }` (example 12.17) |
| Inference: `Flexway-Budget: soft` on every answer of a tenant | It passed `soft.at` of its monthly euro budget | Expected; raise the tenant's `monthly_eur` (override) or wait for next month |
| Inference: `Flexway-Degraded: backend-fallback` | The chosen backend is down (connection, `5xx`, timeout, open circuit); the next of `fallback` served | The `flexway.inference.fallback` metric gives the backend left and the cause |
| Refused at load "price is required: the budget is counted in euros" | A euro budget exists and a backend has no price | Declare `price` on every backend, `0` for an internal one |
| Inference: `403` `sensitive_no_internal_backend` | A sensitive request and no internal `ai.sensitive_backend` | Declare an internal backend; never an external one for that flow |
| Inference: `400` `capability_unavailable` | The backend the policy chooses does not serve what the request needs - an image with no `vision`, embeddings with no `embeddings` | Declare the capability on the backend (`capabilities`) or a `match.capability` override toward a backend that serves it |
| Warning "is a relay metered locally" (`flexway validate`, start) | A `kind: relay` without `metering: upstream`: the upstream Flexway counts the cost too | Declare `metering: upstream`; the local cap still applies |
| Inference: `502` `backend_unavailable` | Backend unreachable, or its key not found (`${VAR}` unset, file missing) | The log (`inference backend failed`) gives the cause; the key is read at each call |
| Start refused: `FLEXWAY_CONFIG is not set` | No `flexway.yaml` named; the old `config/policy.yaml` is no longer read since 0.9.0 | Point `FLEXWAY_CONFIG` at a `flexway.yaml`; convert the old file ([Serving inference](https://flexway.io/en/docs/operations-inference/)) |
| `helm install`: `Flexway serves what a contract declares, and none was given` | Neither `openapi` nor `config` in the values | The command it prints ([Moving to Kubernetes](https://flexway.io/en/docs/operations-kubernetes/)) |
| `403` `insufficient_scope` | The token lacks the scope or role the **contract** requires | Read `scope=` in `WWW-Authenticate`; for a role, check `roles_claim` |
| A2A: `200` with JSON-RPC error `-32009` (`VERSION_NOT_SUPPORTED`) | No `A2A-Version` header - version 0.3 for the specification | Send `A2A-Version: 1.0` (or `?A2A-Version=1.0`); only 1.0 is served |
| A2A: `403` `PUSH_TARGET_NOT_ALLOWED` | A push target outside `agents.a2a.push.allowed_hosts`, over `http`, written as an address, with `user@`, or no list declared | Declare the webhook host's **name**; `flexway_a2a_push_refused_total` gives the reason |
| A2A: `403` `TENANT_MISMATCH` | The request's `tenant` (body, path prefix, parameter) is not the identity's, or the identity holds none | Send the identity's tenant, or none: the gateway forwards its own |
| A2A: `429` `TOO_MANY_STREAMS` | The caller already holds `max_streams` open streams to this agent on this replica | Close a stream, or raise the agent's `max_streams` |
| An A2A stream ends at a fixed time | `max_task_duration` reached (`flexway_a2a_calls_total{outcome="duration_exceeded"}`) | Raise it on the agent, or follow the task with `GetTask` |
| Start refused on an A2A card | gRPC binding, a scheme with no mechanism, an upstream without `token_exchange` or not `protocol: a2a`, declared `signatures`, `signer` (planned for `1.9.0`), no `verify: none`, streaming without `max_streams` or `max_task_duration` | The message names the card, the agent and the rule |
| `401` `WWW-Authenticate: ApiKey` on a key believed good | Unknown key (removed, mistyped), expired, or without the `fwk_` prefix | The log (`API key refused`) gives the cause and, for an expired key, its `id`; create a new one |
| `401` "does not accept API keys" | The operation has no `apiKey` alternative in the contract | Add `{ key: [] }` to its `security`, if intended |
| `403` "not bound to this operation" (key or certificate) | The entry does not name this operation | Add the `operationId` to the entry's `operations`; reload |
| Refused at load "no contract declares" on a key or a certificate | The entry names an operation that does not exist (renamed?) | Fix the `operationId` |
| `401` "requires a client certificate" despite a certificate | Undeclared certificate, or `x-forwarded-client-cert` from a peer outside `server.trusted_proxies` | Check the URI SAN or subject against `auth.mtls.clients`; declare the edge's range |
| TLS handshake refused on `:8443` | A client certificate from another authority than `client_ca_file` | Add the authority to the file: it is re-read without a restart |
| Refused at load "google.api.http ... not supported at this version" | A variable spanning segments, a wildcard or a verb in a REST rule | One variable per segment (`/v1/shelves/{shelf}/books/{book}`), or the method in gRPC only |
| Refused at load "is a gRPC method and relays to upstream ... which does not declare protocol: grpc" | The method falls on an HTTP upstream (the `default`) | Name its upstream in `flexway.v1.operation.upstream`, declared `protocol: grpc` |
| gRPC client: `UNIMPLEMENTED` | The method is not in the served set - a stale set | Rebuild the `.binpb`: it is re-read without a restart |
| REST: `upstream_status` | The gRPC upstream answered an error: its answer, not a gateway failure | Read the problem's `grpc.status` and `grpc.details` |
| `400` "Query parameter ... is not declared" | Closed by default: the contract does not declare that parameter | Declare it in the contract, rather than going around the gateway |
| `429` `rate_limited` | The caller's bucket is empty | Read `Retry-After`; raise the class's `rps`/`burst`, or change the operation's class |
| All anonymous callers seem to share one bucket | The gateway sees the edge's address - or, behind a cloud load balancer without the proxy protocol, the load balancer's - not the client's | Declare the edge in `server.trusted_proxies`; read `x-forwarded-for` in the edge's access log, and turn the proxy protocol on at the load balancer and the edge (guide [Setting quotas](https://flexway.io/en/docs/operations-quotas/)) |
| `422` `invalid_request` | The request breaks the contract's schema | The `errors` extension names the field and the rule |
| `503` with `Flexway-Degraded: circuit-open` | The upstream is failing; the breaker answers for it | Fix the upstream; one trial call goes through after `cooldown` |
| `503` with `Flexway-Degraded: upstream-saturated` | `max_in_flight` reached on this upstream | Raise the cap, or find why the upstream slows down |
| `503` `overloaded` with `Flexway-Degraded: shedding`, `flexway_shed_total{reason="in_flight"}` rising | The replica is at its in-flight limit (`server.in_flight`, derived from its memory limit unless written): it refuses the excess instead of being OOM-killed | Add replicas, or give each more CPU and memory; the limit in force is in `/config` (`server.in_flight`) and in the start log |
| A header does not reach the upstream | It is not declared in the contract - it is dropped, not refused | Declare it as an `in: header` parameter |
| `403` "This origin is not allowed" | The browser's origin is in no `cors` policy | Add it to `allow_origins` |
| Quota counters disagree across replicas | No shared store | Declare a Redis, Dragonfly or Valkey |
| A change is not taken | The new version is refused: the previous one still serves | Read `last_reload.cause` in `/config`; in a cluster, check the `ConfigMap` is not mounted with `subPath` |
| Reload refused, "restart required" | `server.shutdown`, `observability` or `server.reload` changed | Restart: it is the only way to apply them |
| `404` on a host that declares `contracts` | That host serves **exactly** its contracts, nothing of the default one | Add the contract to `hosts.<host>.contracts` |
| Agent: `401` with a valid token | The token is not issued **for the surface**: its audience must be the URL in `agents.mcp.resources` | Request the token with the resource indicator (`resource=https://…/mcp`) |
| An expected tool is missing from `tools/list` | The agent's token lacks the scope or role the operation requires - the list is filtered | Give the agent the scope, rather than widening the operation |
| A tool answers `401` from the upstream | The upstream verifies a JWT itself, and the agent's token is never relayed to it | Declare `upstreams.<name>.token_exchange` and the issuer's `client` ([The token the upstream receives](https://flexway.io/en/docs/operations-agents-mcp/#the-token-the-upstream-receives), [Token exchanged toward the upstream](https://flexway.io/en/docs/operations-examples-agents/#token-exchanged-toward-the-upstream)) |
| A federated server's tool does not appear | Server unreachable or not discovered yet; tool outside the declared list; a missing scope; a name already taken | `/config`, `mcp_federation`: state, tools served, `not_served` and why; the log says `MCP server not reachable` |
| Refused at load, "is offered twice … rename one" | A federated tool has the name of a contract tool or of another server's | `agents.mcp.servers.<name>.rename` |
| A federated tool answers `forbidden` | The caller lacks the declared `scopes` (or `tool_scopes`), or is not authenticated | Give the agent the scope |
| `403` "The chain of agents … is longer than allowed" or "older than allowed" | The token names more agents (`act`) than `max_depth`, or the human authenticated more than `max_age` ago - an operation may tighten | The log (`delegation chain refused`) gives the reason and the agents; have the human authenticate again, or shorten the chain |
| `401` "The delegation was revoked" | The issuer says the token is no longer active (introspection) | Expected after a revocation; obtain a new consent |
| `503` "The delegation cannot be verified now" | The issuer's introspection cannot be reached, or it publishes none | Check `introspection_endpoint` in its discovery and the `client`; a direct token still passes |
| `403` on an exchanging upstream, log `widened` | The provider granted a scope the caller did not have | Fix the provider's exchange policy: a hop must never widen |
| Discovery fails with `401` | The server needs a token even to list | `discovery.issuer`, with a `client` on that issuer |
| `403` "The identity provider refused to issue a token for this upstream" | The provider refuses the exchange: a client not allowed to exchange, an unknown audience, a wrong secret | The log (`token exchange refused`) gives the provider's code; fix the client at the provider |
| `503` "could not issue a token", with `Retry-After` | Provider unreachable, or the client secret not found (`${VAR}` unset, file missing) | The log (`token exchange unavailable`) says which; the secret is read at each exchange, no reload needed |
| Refused, "operationId … is declared by both" | Two contracts give two operations one name | Rename one: the id keys policies, metrics and `/config` |
| Start refused, "is protected - the gateway alone writes it" | A policy writes a header only the gateway sets (identity, tracing, tenant, `X-Forwarded-*`) | Pick another header name |
| The upstream receives `X-User-Roles: [seller admin]` | Gateway older than `1.6.0`: a list rendered the way Go prints it | Upgrade: a list arrives as `seller,admin`; `{{ auth.claims.roles.join(' ') }}` for another delimiter ([Enriching a flow](https://flexway.io/en/docs/operations-policies/)) |
| Start refused, `undeclared reference to 'x'` or `found no matching overload` in a value | A function the language does not have, or `join` with a delimiter that is not text | Only CEL's functions and `join` exist; fix the expression |
| Refused, "segment … has two parameters side by side" or "a '{' that is never closed" | A path segment the router cannot read | Put text between the two parameters, close the brace |
| Refused, "segments … rank the same and can match the same text" | Two mixed segments of one rank read the same text (`a{x}` and `{x}a`) | Make one more literal, or rename a path |
| `404` on `/reports/2026.pdf` while `/reports/{year}.pdf` is in the contract | Gateway older than `1.6.0`: the mixed segment was taken for a literal | Upgrade ([Serving your first API](https://flexway.io/en/docs/operations-first-api/)) |
| First request slow after a start, on a large contract in `validation: params` | Gateway older than `1.6.0`: the parameters validator was built on its first request, after `/readyz` | Upgrade: both validators are compiled before `/readyz`; the first request to an upstream still opens its connection |
| Start refused, "on_missing is required" | A value is a `{{ … }}` expression with nothing said about a missing value | Add `on_missing: skip`, `empty` or `fail` |
| Start refused, "two … policies at the same level" | Two `cors` (or `rate-limit`, `rewrite-path`) at one level | Keep one, or move the other to a more precise level |
| `500` `internal_error`, log `a policy value is missing` | `on_missing: fail` and the value is missing for this caller | Add a `when` condition, or switch to `skip` |
| A policy does nothing | The start log says `NOT applied`: its type belongs to a later version | Read `effective_chains` in `/config` |

---

> **On languages.** This documentation exists in English and in French, page for page; both are kept
> in step. The specifications, ADRs and the roadmap are written in French - they are the engineering
> record. The product itself, its configuration keys, its error codes and its logs are in English.