Your models behind one surface, with a budget that holds.

The gateway serves /v1/* with no contract to write: the built-in operations ai.chatCompletions, ai.listModels and ai.embeddings answer under the ai section of flexway.yaml, with the same identity, policies and tracing as the rest of the traffic. Errors on /v1/* are returned in the OpenAI error envelope, gateway refusals included, and as RFC 9457 everywhere else. Streaming is relayed line by line, unbuffered, and the budget is reconciled on the final usage.

A backend is declared by its kind: openai for any OpenAI-compatible API (the default), relay for another Flexway, grpc for the upstream of a specialised model. No provider is named in the product: a backend is a URL, an authentication method and a price. Authentication is generic - none, bearer by reference, header with a declared name, token_file re-read on every call as a workload identity projects it - and client TLS can be added for an mTLS relay.

The requested model is served if it appears in the backend's models list, otherwise its default_model is used; the audit keeps the model actually served. Earlier configurations load unchanged.

A requests-per-second quota says nothing about the cost of a model call. Flexway counts tokens per day and per month, and euros per month, per tenant: each backend declares its price (euros per million input and output tokens), and the first budget reached decides. Each call is pre-reserved, then reconciled at the price of the backend that actually served it.

At the soft cap (ai.budget.soft), the response carries the Flexway-Budget: soft header and traffic switches to the declared, cheaper backend; at the hard cap, the request is refused with 429 budget_exceeded. A failed or refused call gives its reservation back, tokens and euros: a failing backend does not drain a tenant's budget. Overrides per tenant and per agent set a different backend and different budgets.

Behind a relay to another Flexway that meters on its own (metering: upstream), the local cap still applies, the upstream's 429 is relayed as is and series carry metered_by="upstream", so that consumption is counted once, where it is paid. flexway validate warns about a relay metered locally, which would count twice.

A request flagged as sensitive by the declared header (ai.sensitive_header) is pinned to the designated internal backend (ai.sensitive_backend). That backend must be internal: declaring an external one is refused at load. If no internal backend can serve it, the request is refused rather than sent outside. The header can only narrow routing, never widen it.

A relay to another Flexway is treated as external unless declared otherwise (external: false): only the operator decides that another gateway may receive sensitive traffic. The call's identity is passed under declared names only (identity_headers: tenant, agent, sensitive); nothing from the caller's request is copied. Neither the soft-cap switch nor fallbacks ever send a sensitive request to an external backend.

Backend choice is declarative: default, then tenant and agent override, then capability override, then sensitivity; neither latency, nor content, nor the requested model decides. Declaring which backend serves which tenants helps you meet your data residency requirements; automatic routing by latency or by jurisdiction is upcoming.

Each backend declares a fallback list, tried in order on connection error, 5xx, timeout or open circuit. Fallback happens before any byte reaches the client, each backend once. It never happens on a 4xx, a provider's 429 included, and never to an external backend for a sensitive request. A response served by a fallback carries Flexway-Degraded: backend-fallback.

A circuit breaker per backend avoids insisting on a model that has stopped answering. Fallback applies to the openai and relay kinds and keeps only backends that serve the requested capability: a request with an image falls back only to a vision backend. Fallback on an exhausted budget is the soft cap described above.

Inference-aware routing - by model, request criticality, actual model-server load or context-cache affinity, interfacing with the Gateway API Inference Extension on Kubernetes - is upcoming. The semantic cache, opt-in per operation, is upcoming as well.

Each inference call produces a metadata audit line: tenant, agent, served model, backend, tokens, cost. Prompts and responses are never collected, and identifiers are hashed in logs. Per-tenant reporting, for internal billing or chargeback, comes out of this audit with no extra tooling.

The same facts feed the flexway.inference.cost and flexway.inference.calls metrics, with metered_by and capability dimensions, which the console reads for its per-tenant and per-backend gauges. Keeping a usable record of who called which model at what cost, without keeping the content, helps you meet the logging and data governance requirements that apply to AI usage.

Message content can be a string or text and image_url parts, in the OpenAI format. The image is measured in the body without being decoded (ai.vision.max_image_bytes, 8 MiB by default; body bounded by ai.vision.max_body_bytes, 24 MiB by default); data: URIs are accepted by default, a remote URL only from a declared https origin (ai.vision.allowed_origins), and any other part type is refused. An image goes only to a backend with the vision capability, otherwise 400 capability_unavailable; it is pre-reserved at 2,048 tokens, reconciled on the reported usage.

POST /v1/embeddings follows the same path as a completion: identity, budget on prompt_tokens, fallback before any byte, metadata audit, and a backend with the embeddings capability. A gRPC method of a specialised model is metered when it carries the ai option of (flexway.v1.operation), the gRPC counterpart of x-flexway-ai: per call (body never read) or per byte, with a price per thousand and a feature.

For these models, the tenant's euro budget is reserved before the call and returned if the upstream did not serve; refusals are RESOURCE_EXHAUSTED budget_exceeded and PERMISSION_DENIED sensitive_no_internal_backend. Their upstream must be named by a kind: grpc backend, otherwise the configuration is refused at load.