# Troubleshooting: inference and agents

> Inference, MCP tools, federation, agent chains, token exchange and A2A: the symptom, its cause and what to do.

Troubleshooting - Since version 0.1.0 - Flexway v1.7.1 - https://flexway.io/en/docs/operations-troubleshooting-ai/

> Operations guide: configure, deploy and observe the gateway. What the feature does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

## Inference

| Symptom | Cause | Action |
| --- | --- | --- |
| Inference: `401` `tenant_required` despite the ingress's headers | The peer is not in `server.trusted_proxies`: its headers are not believed | Declare the ingress's range, and only it |
| Inference: `429` `budget_exceeded` | The tenant's budget (day or month) is spent - a request pre-books `max_tokens`, 512 without it | Raise `ai.budget` or the tenant's override; the counter is in `ai.budget.store` |
| Inference: `429` `rate_limited` for every tenant behind the ingress | A rate-limit policy too tight on `/v1/*` | A policy of its own on `routes: { "/v1/**": [...] }` ([example](https://flexway.io/en/docs/operations-examples-inference/#inference-behind-a-mesh)) |
| Inference: `Flexway-Budget: soft` on every answer of a tenant | It passed `soft.at` of its monthly euro budget | Expected; raise the tenant's `monthly_eur` (override) or wait for next month |
| Inference: `Flexway-Degraded: backend-fallback` | The chosen backend is down (connection, `5xx`, timeout, open circuit); the next of `fallback` served | The `flexway.inference.fallback` metric gives the backend left and the cause |
| Refused at load "price is required: the budget is counted in euros" | A euro budget exists and a backend has no price | Declare `price` on every backend, `0` for an internal one |
| Inference: `403` `sensitive_no_internal_backend` | A sensitive request and no internal `ai.sensitive_backend` | Declare an internal backend; never an external one for that flow |
| Inference: `400` `capability_unavailable` | The backend the policy chooses does not serve what the request needs - an image with no `vision`, embeddings with no `embeddings` | Declare the capability on the backend (`capabilities`) or a `match.capability` override toward a backend that serves it |
| Warning "is a relay metered locally" (`flexway validate`, start) | A `kind: relay` without `metering: upstream`: the upstream Flexway counts the cost too | Declare `metering: upstream`; the local cap still applies |
| Inference: `502` `backend_unavailable` | Backend unreachable, or its key not found (`${VAR}` unset, file missing) | The log (`inference backend failed`) gives the cause; the key is read at each call |

## Agents and MCP

| Symptom | Cause | Action |
| --- | --- | --- |
| Agent: `401` with a valid token | The token is not issued **for the surface**: its audience must be the URL in `agents.mcp.resources` | Request the token with the resource indicator (`resource=https://…/mcp`) |
| An expected tool is missing from `tools/list` | The agent's token lacks the scope or role the operation requires - the list is filtered | Give the agent the scope, rather than widening the operation |
| A tool answers `401` from the upstream | The upstream verifies a JWT itself, and the agent's token is never relayed to it | Declare `upstreams.<name>.token_exchange` and the issuer's `client` ([The token the upstream receives](https://flexway.io/en/docs/operations-agents-mcp/#the-token-the-upstream-receives), [Token exchanged toward the upstream](https://flexway.io/en/docs/operations-examples-agents/#token-exchanged-toward-the-upstream)) |
| A federated server's tool does not appear | Server unreachable or not discovered yet; tool outside the declared list; a missing scope; a name already taken | `/config`, `mcp_federation`: state, tools served, `not_served` and why; the log says `MCP server not reachable` |
| Refused at load, "is offered twice … rename one" | A federated tool has the name of a contract tool or of another server's | `agents.mcp.servers.<name>.rename` |
| A federated tool answers `forbidden` | The caller lacks the declared `scopes` (or `tool_scopes`), or is not authenticated | Give the agent the scope |
| `403` "The chain of agents … is longer than allowed" or "older than allowed" | The token names more agents (`act`) than `max_depth`, or the human authenticated more than `max_age` ago - an operation may tighten | The log (`delegation chain refused`) gives the reason and the agents; have the human authenticate again, or shorten the chain |
| `401` "The delegation was revoked" | The issuer says the token is no longer active (introspection) | Expected after a revocation; obtain a new consent |
| `503` "The delegation cannot be verified now" | The issuer's introspection cannot be reached, or it publishes none | Check `introspection_endpoint` in its discovery and the `client`; a direct token still passes |
| `403` on an exchanging upstream, log `widened` | The provider granted a scope the caller did not have | Fix the provider's exchange policy: a hop must never widen |
| Discovery fails with `401` | The server needs a token even to list | `discovery.issuer`, with a `client` on that issuer |
| `403` "The identity provider refused to issue a token for this upstream" | The provider refuses the exchange: a client not allowed to exchange, an unknown audience, a wrong secret | The log (`token exchange refused`) gives the provider's code; fix the client at the provider |
| `503` "could not issue a token", with `Retry-After` | Provider unreachable, or the client secret not found (`${VAR}` unset, file missing) | The log (`token exchange unavailable`) says which; the secret is read at each exchange, no reload needed |

## A2A

| Symptom | Cause | Action |
| --- | --- | --- |
| A2A: `200` with JSON-RPC error `-32009` (`VERSION_NOT_SUPPORTED`) | No `A2A-Version` header - version 0.3 for the specification | Send `A2A-Version: 1.0` (or `?A2A-Version=1.0`); only 1.0 is served |
| A2A: `403` `PUSH_TARGET_NOT_ALLOWED` | A push target outside `agents.a2a.push.allowed_hosts`, over `http`, written as an address, with `user@`, or no list declared | Declare the webhook host's **name**; `flexway_a2a_push_refused_total` gives the reason |
| A2A: `403` `TENANT_MISMATCH` | The request's `tenant` (body, path prefix, parameter) is not the identity's, or the identity holds none | Send the identity's tenant, or none: the gateway forwards its own |
| A2A: `429` `TOO_MANY_STREAMS` | The caller already holds `max_streams` open streams to this agent on this replica | Close a stream, or raise the agent's `max_streams` |
| An A2A stream ends at a fixed time | `max_task_duration` reached (`flexway_a2a_calls_total{outcome="duration_exceeded"}`) | Raise it on the agent, or follow the task with `GetTask` |
| Start refused on an A2A card | gRPC binding, a scheme with no mechanism, an upstream without `token_exchange` or not `protocol: a2a`, declared `signatures`, `signer` (planned for `1.10.0`), no `verify: none`, streaming without `max_streams` or `max_task_duration` | The message names the card, the agent and the rule |