# Serving inference

> Goal: serve the OpenAI-compatible surface (/v1/chat/completions, /v1/models, /v1/embeddings since 1.5.0) through the same data plane as the API - same…

Guide - Since version 0.1.0 - Flexway v1.6.3 - https://flexway.io/en/docs/operations-inference/

> Operations guide: configure, deploy and observe the gateway. What the capability does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

**Goal**: serve the OpenAI-compatible surface (`/v1/chat/completions`, `/v1/models`,
`/v1/embeddings` since `1.5.0`) **through the
same data plane** as the API - same identity, same rate limits, same policies, same trace -, routed
by policy to backends, under a per-tenant token budget.

**Command** - an `ai:` section. It needs **no contract**: the gateway declares its operations
itself (`ai.chatCompletions`, `ai.listModels`, `ai.embeddings`), served on every host.

```yaml
ai:
  security:                          # the identities accepted, one suffices - as `security` in a contract
    - oidc: { scopes: [inference] }
    - api_key: {}                    # a key bound to ai.chatCompletions (see API keys)
  sensitive_header: X-Sensitive      # "true" pins the request to sensitive_backend
  default_backend: eu-provider
  sensitive_backend: on-prem         # internal; unreachable, or no internal backend at all: refused, never an external one
  backends:
    eu-provider:
      url: https://api.provider.example/v1
      api_key: "file:/etc/flexway/secrets/provider-key"   # a reference, read at each call
      default_model: mistral-small-3.2-24b                # the gateway chooses the model
      external: true
    on-prem:
      url: http://vllm.inference.svc:8000/v1
      default_model: mistral-7b-instruct
  budget:
    store: "${COUNTERS_URL}"         # redis://..., shared by every replica
    monthly_tokens: 40000000         # per tenant, 0 for none
    daily_tokens: 2000000
  overrides:                         # the first match wins
    - { match: { tenant: acme, agent: batch-summaries }, backend: on-prem }
```

**Four possible identities**, each an alternative of `ai.security`:

| Alternative | The caller presents | Tenant |
| --- | --- | --- |
| `trusted_headers` | The headers a mesh ingress wrote (`auth.trusted_headers`), **believed from a peer of `server.trusted_proxies` only** | The `tenant` header; `subject` names the agent |
| `oidc` | A token of one of the issuers, with the declared scopes or roles | The `tenant_claim` claim |
| `api_key` | `Authorization: ApiKey <key>`, the key bound to `ai.chatCompletions` | The key's |
| `mtls` | A declared client certificate, bound to the operation | The certificate's |

Behind a mesh ingress that already authenticated the caller (the header names are the deployment's,
never the product's):

```yaml
server:
  trusted_proxies: [10.42.0.0/16]    # the ingress, and only it: from anyone else, the header is a forgery
auth:
  trusted_headers: { tenant: X-Mesh-Tenant, subject: X-Mesh-Agent }
ai:
  security: [ { trusted_headers: {} } ]
```

## Cost, soft cap and fallback

Each backend carries its **price** per million tokens; the budget in **euros** adds to the token
budget - the first reached decides (full example: [Cost, soft cap and fallback](https://flexway.io/en/docs/operations-examples-inference/#cost-soft-cap-and-fallback)):

```yaml
ai:
  backends:
    eu-large:
      # ...
      price: { input_per_mtok: 2.00, output_per_mtok: 6.00 }   # euros per million tokens
      fallback: [eu-small, on-prem]   # in order, before any byte reaches the client
  budget:
    monthly_eur: 50                   # per tenant, over the calendar month
    soft: { at: 0.8, backend: eu-small }
  overrides:
    - { match: { tenant: acme }, monthly_eur: 500 }
```

| Mechanism | Behaviour |
| --- | --- |
| **Cost** | Pre-booked (input estimated at body size / 4, output at `max_tokens`), reconciled on the real usage **at the price of the backend that served**; a call that fails or is refused gives its booking back, tokens and euros; carried by the audit and the `flexway.inference.cost` metric (euros, by tenant and backend) |
| **Soft cap** | Past `at` x budget: a `Flexway-Budget: soft` header, a metric, a log line; the call goes to `soft.backend` - except a sensitive request, which stays pinned |
| **Hard cap** | A pre-booking that would pass the euro (or token) budget: `429` `budget_exceeded`, before any call |
| **Fallback** | On connection failure, `5xx`, timeout or open circuit: the next of `fallback`, each backend once; **never** on a `4xx` (the provider's `429` included), nor after a stream's first byte, nor to an external backend for a sensitive request. The answer is marked `Flexway-Degraded: backend-fallback` |

> A price is **required** for every backend once a budget in euros exists: a cost nobody can compute
> cannot be budgeted. `0` is a price - an internal backend's.

**Check**:

| Call | Response |
| --- | --- |
| An accepted identity, budget left | relayed as the backend wrote it; the backend gets the client's body with the policy's model, and **its own** key - never the caller's identity |
| `stream: true` | events arrive **as they are produced**; the budget is reconciled on the final usage |
| No identity, or an identity without a tenant | `401` `tenant_required` |
| The tenant's budget spent (completion **or** stream) | `429` `budget_exceeded`, before any backend call |
| `X-Sensitive: true` with no internal backend | `403` `sensitive_no_internal_backend` - never an external backend |
| Backend down, no fallback left | `502` `backend_unavailable`; during a stream, a last event `{"error":{"type":"backend_unavailable"}}` |
| A gateway refusal on `/v1/*` (scope, rate, unknown route) | **OpenAI envelope**, OpenAI's `type` (`permission_error`, `rate_limit_error`...) and the gateway's `code` |
| A tenant past `soft.at` of its euro budget | `Flexway-Budget: soft`; served by `soft.backend` |
| A backend down, with `fallback` | served by the next one, `Flexway-Degraded: backend-fallback` |
| A policy naming an undeclared backend | **refused at load**, the key named |

> **Rate limits**: `/v1/*` follows the policy chain like any route - a `rate-limit` policy of its
> own, attached to `routes: { "/v1/**": [...] }`, bounds it apart (example 12.17). Behind an ingress,
> the bucket is the **tenant and agent** it vouches for, never the ingress's address.
>
> **Migrating from `config/policy.yaml`** - **no longer read since `0.1.0`**: without
> `FLEXWAY_CONFIG`, the gateway refuses to start. The mapping, to apply once by hand:
>
> | `policy.yaml` | `flexway.yaml` |
> | --- | --- |
> | `default_backend`, `backends.<n>.default_model`, `external` | `ai.default_backend`, `ai.backends.<n>.*` |
> | `backends.<n>.base_url_env` | `ai.backends.<n>.url` - the URL itself |
> | `backends.<n>.api_key_env: VAR` | `ai.backends.<n>.api_key: "${VAR}"` or `file:/...` |
> | `pii_guard.sensitive_backend` | `ai.sensitive_backend` |
> | `budget.*_tokens_default` | `ai.budget.*_tokens`; `REDIS_URL` -> `ai.budget.store` |
> | `overrides` | `ai.overrides`, same shape |
> | Headers injected by the ingress | `auth.trusted_headers` + `server.trusted_proxies` + `ai.security: [ { trusted_headers: {} } ]` |

## Several backends: images, embeddings, relays and specialised models

Since `1.5.0`, a backend declares **where** it is (`kind`), what it
**serves** (`capabilities`), the models a caller may **name** (`models`), how the gateway
**authenticates** (`auth`) and **who meters** the cost (`metering`). A configuration written before
stays valid as it is: `kind: openai`, `capabilities: [chat]`, `api_key` is
`auth: { scheme: bearer }`, `metering: local`.

**The three kinds**:

```yaml
ai:
  default_backend: platform
  sensitive_backend: local-vision
  backends:
    platform:                          # another Flexway: it meters, it caps
      kind: relay
      url: https://llm.platform.example/v1
      auth: { scheme: token_file, path: /var/run/flexway/peer-token }   # read again at each call
      identity_headers: { tenant: X-Tenant, agent: X-Agent, sensitive: X-Sensitive }
      metering: upstream
      external: false                  # the operator declares it: sensitive data may go there
      capabilities: [chat, vision, embeddings]
    provider-direct:                   # a provider called directly: the key lives here
      kind: openai
      url: https://api.provider.example/v1
      auth: { scheme: header, header: X-Api-Key, key: "${PROVIDER_KEY}" }
      external: true
      capabilities: [chat, vision, embeddings]
      models: [model-a, model-vision-b, embed-c]   # served when asked for, else default_model
      default_model: model-a
      price: { input_per_mtok: 0.15, output_per_mtok: 0.35 }
    local-vision:                      # a model of the cluster
      url: http://vision.models.svc:8080/v1
      default_model: vision-small
      auth: { scheme: none }
      capabilities: [vision]           # an image; text too
    faces:                             # a specialised model: the upstream of a gRPC contract
      kind: grpc
      upstream: models                 # declared under upstreams:, protocol: grpc
      capabilities: [vision]
  overrides:
    - { match: { agent: screen-reader }, backend: provider-direct }
    - { match: { capability: embeddings }, backend: provider-direct }   # routes only, never a budget
  vision:
    max_image_bytes: 8MiB              # one image, decoded (the default)
    allowed_origins: ["https://cdn.example"]   # otherwise inline images (data:) only
```

| Key | Values | Effect |
| --- | --- | --- |
| `kind` | `openai` (default), `relay`, `grpc` | An OpenAI-compatible API; another Flexway; the upstream of a gRPC specialised model, never a target of `/v1/*` |
| `capabilities` | `chat` (default), `vision`, `embeddings`; `model` (`grpc`) | An image only goes to a `vision` backend, embeddings only to an `embeddings` one, else `400` `capability_unavailable` |
| `models` | list | The model asked for is served when listed, else `default_model`; the audit keeps the model served. A `relay` without `default_model` forwards the model asked for |
| `auth.scheme` | `none`, `bearer` (`key`), `header` (`header`, `key`), `token_file` (`path`) | The key is a reference (`${VAR}`, `file:`), read again at each call; `token_file` reads a short-lived token again (workload identity) |
| `metering` | `local` (default), `upstream` | See below |
| `external` | boolean | A sensitive request never goes there; a `relay` is external unless `external: false` |
| `identity_headers` | `tenant` (required), `agent`, `sensitive` | The names a `relay` receives the call's identity under - those the other Flexway declares in `auth.trusted_headers` and `ai.sensitive_header` |
| `tls` | as an upstream's | Mutual TLS toward a `relay` |
| `match.capability` (override) | `chat`, `vision`, `embeddings` | Chooses the backend of the calls of that capability, after the tenant and agent override; never a budget |

**Metering** - who counts the cost:

| `metering` | Tokens | Euros | Upstream refusal | Series |
| --- | --- | --- | --- | --- |
| `local` | Booked, reconciled on the usage | At the declared price | A provider's `4xx`: `502` `backend_unavailable` | `metered_by="local"` |
| `upstream` | **Local cap always applied**, on the usage the upstream returns | At the declared price if any; without one, the upstream is authoritative | Relayed **as it came**, `429` `budget_exceeded` included | `metered_by="upstream"` |

> A consumption view sums `metered_by!="upstream"` only - never `="local"`: a series from before
> `1.5.0` has no such label -, a relayed call being counted `local` by the Flexway that pays for it.
> The console shows the upstream's share beside, under "Metered upstream" ([Reading with the console](https://flexway.io/en/docs/operations-console/)). `flexway validate` warns of a `relay` metered locally.

**Platform side** - the Flexway receiving the relay believes the identity headers from this peer
only:

```yaml
server:
  trusted_proxies: [10.42.0.0/16]      # the products' Flexway
auth:
  trusted_headers: { tenant: X-Tenant, subject: X-Agent }
ai:
  security: [ { trusted_headers: {} } ]
  sensitive_header: X-Sensitive
```

**Image and embeddings**:

```bash
# an inline image, to a vision backend
curl -s http://localhost:8080/v1/chat/completions -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"model-vision-b","messages":[{"role":"user","content":[{"type":"text","text":"What does this screen show?"},{"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw0..."}}]}]}'
# embeddings
curl -s http://localhost:8080/v1/embeddings -H "Authorization: Bearer $TOKEN" -d '{"model":"embed-c","input":["one","two"]}'
```

**gRPC specialised model** - a method of the product's contract, metered by the `ai` option:

```protobuf
rpc Classify(Image) returns (Labels) {
  option (flexway.v1.operation) = { ai: { unit: "call", price_per_1k: 0.02, capability: "vision" } };
}
```

Each call books its price in the tenant's **euro** budget (`ai.budget.monthly_eur` and the
overrides), before the upstream; `unit: byte` counts the bytes of the request and the answer as they
cross. An upstream that is down costs nothing.

**Check**:

| Call | Answer |
| --- | --- |
| Image to a `vision` backend | served; the body as written, the model admitted |
| Image with no `vision` backend chosen, or embeddings with no `embeddings` backend | `400` `capability_unavailable`, before any backend |
| Image past `max_image_bytes` | `413` `payload_too_large` |
| Image by URL from an undeclared origin, an `input_audio` part | `400` `invalid_request` |
| Relay: tenant past the platform's budget | the platform's `429` `budget_exceeded`, relayed as it came |
| Relay: tenant past the local cap | `429` `budget_exceeded`, the platform never called |
| Metered gRPC method past the budget | `RESOURCE_EXHAUSTED`, reason `budget_exceeded`; the upstream never called |
| Metered gRPC method with no `grpc` backend naming its upstream | **refused at load** |

*Available since `0.1.0`; several backends since `1.5.0`.*