Troubleshooting: inference and agents
Inference, MCP tools, federation, agent chains, token exchange and A2A: the symptom, its cause and what to do.
TroubleshootingOperationsSince version 0.1.0
On this page
Operations guide: configure, deploy and observe the gateway. What the feature does, product side: product guide; calling it: developer guide.
Inference
| Symptom | Cause | Action |
|---|---|---|
Inference: 401 tenant_required despite the ingress's headers |
The peer is not in server.trusted_proxies: its headers are not believed |
Declare the ingress's range, and only it |
Inference: 429 budget_exceeded |
The tenant's budget (day or month) is spent - a request pre-books max_tokens, 512 without it |
Raise ai.budget or the tenant's override; the counter is in ai.budget.store |
Inference: 429 rate_limited for every tenant behind the ingress |
A rate-limit policy too tight on /v1/* |
A policy of its own on routes: { "/v1/**": [...] } (example) |
Inference: Flexway-Budget: soft on every answer of a tenant |
It passed soft.at of its monthly euro budget |
Expected; raise the tenant's monthly_eur (override) or wait for next month |
Inference: Flexway-Degraded: backend-fallback |
The chosen backend is down (connection, 5xx, timeout, open circuit); the next of fallback served |
The flexway.inference.fallback metric gives the backend left and the cause |
| Refused at load "price is required: the budget is counted in euros" | A euro budget exists and a backend has no price | Declare price on every backend, 0 for an internal one |
Inference: 403 sensitive_no_internal_backend |
A sensitive request and no internal ai.sensitive_backend |
Declare an internal backend; never an external one for that flow |
Inference: 400 capability_unavailable |
The backend the policy chooses does not serve what the request needs - an image with no vision, embeddings with no embeddings |
Declare the capability on the backend (capabilities) or a match.capability override toward a backend that serves it |
Warning "is a relay metered locally" (flexway validate, start) |
A kind: relay without metering: upstream: the upstream Flexway counts the cost too |
Declare metering: upstream; the local cap still applies |
Inference: 502 backend_unavailable |
Backend unreachable, or its key not found (${VAR} unset, file missing) |
The log (inference backend failed) gives the cause; the key is read at each call |
Agents and MCP
| Symptom | Cause | Action |
|---|---|---|
Agent: 401 with a valid token |
The token is not issued for the surface: its audience must be the URL in agents.mcp.resources |
Request the token with the resource indicator (resource=https://…/mcp) |
An expected tool is missing from tools/list |
The agent's token lacks the scope or role the operation requires - the list is filtered | Give the agent the scope, rather than widening the operation |
A tool answers 401 from the upstream |
The upstream verifies a JWT itself, and the agent's token is never relayed to it | Declare upstreams.<name>.token_exchange and the issuer's client (The token the upstream receives, Token exchanged toward the upstream) |
| A federated server's tool does not appear | Server unreachable or not discovered yet; tool outside the declared list; a missing scope; a name already taken | /config, mcp_federation: state, tools served, not_served and why; the log says MCP server not reachable |
| Refused at load, "is offered twice … rename one" | A federated tool has the name of a contract tool or of another server's | agents.mcp.servers.<name>.rename |
A federated tool answers forbidden |
The caller lacks the declared scopes (or tool_scopes), or is not authenticated |
Give the agent the scope |
403 "The chain of agents … is longer than allowed" or "older than allowed" |
The token names more agents (act) than max_depth, or the human authenticated more than max_age ago - an operation may tighten |
The log (delegation chain refused) gives the reason and the agents; have the human authenticate again, or shorten the chain |
401 "The delegation was revoked" |
The issuer says the token is no longer active (introspection) | Expected after a revocation; obtain a new consent |
503 "The delegation cannot be verified now" |
The issuer's introspection cannot be reached, or it publishes none | Check introspection_endpoint in its discovery and the client; a direct token still passes |
403 on an exchanging upstream, log widened |
The provider granted a scope the caller did not have | Fix the provider's exchange policy: a hop must never widen |
Discovery fails with 401 |
The server needs a token even to list | discovery.issuer, with a client on that issuer |
403 "The identity provider refused to issue a token for this upstream" |
The provider refuses the exchange: a client not allowed to exchange, an unknown audience, a wrong secret | The log (token exchange refused) gives the provider's code; fix the client at the provider |
503 "could not issue a token", with Retry-After |
Provider unreachable, or the client secret not found (${VAR} unset, file missing) |
The log (token exchange unavailable) says which; the secret is read at each exchange, no reload needed |
A2A
| Symptom | Cause | Action |
|---|---|---|
A2A: 200 with JSON-RPC error -32009 (VERSION_NOT_SUPPORTED) |
No A2A-Version header - version 0.3 for the specification |
Send A2A-Version: 1.0 (or ?A2A-Version=1.0); only 1.0 is served |
A2A: 403 PUSH_TARGET_NOT_ALLOWED |
A push target outside agents.a2a.push.allowed_hosts, over http, written as an address, with user@, or no list declared |
Declare the webhook host's name; flexway_a2a_push_refused_total gives the reason |
A2A: 403 TENANT_MISMATCH |
The request's tenant (body, path prefix, parameter) is not the identity's, or the identity holds none |
Send the identity's tenant, or none: the gateway forwards its own |
A2A: 429 TOO_MANY_STREAMS |
The caller already holds max_streams open streams to this agent on this replica |
Close a stream, or raise the agent's max_streams |
| An A2A stream ends at a fixed time | max_task_duration reached (flexway_a2a_calls_total{outcome="duration_exceeded"}) |
Raise it on the agent, or follow the task with GetTask |
| Start refused on an A2A card | gRPC binding, a scheme with no mechanism, an upstream without token_exchange or not protocol: a2a, declared signatures, signer (planned for 1.10.0), no verify: none, streaming without max_streams or max_task_duration |
The message names the card, the agent and the rule |