Examples: inference

Operations guide: configure, deploy and observe the gateway. What the capability does, product side: product guide; calling it: developer guide.

Inference alone, external callers

# The inference surface alone, for callers outside a mesh: /v1/chat/completions and /v1/models served
# by the data plane - no contract to write, no upstream of this file. A token with the inference
# scope opens it; the tenant comes from its claim. Routed by policy: a third-party default, an
# internal backend that sensitive requests are pinned to, per-tenant token budgets.
version: 1
auth:
  issuers:
    - name: customers
      discovery: https://login.example.net/.well-known/openid-configuration
      audience: api
      tenant_claim: org
ai:
  security:
    - oidc: { scopes: [inference] }             # one alternative suffices, as in a contract
  sensitive_header: X-Sensitive                  # "true" pins the request to sensitive_backend
  default_backend: eu-provider
  sensitive_backend: on-prem                     # internal; unreachable, or no internal backend at all: refused, never external
  backends:
    eu-provider:
      url: https://api.provider.example/v1
      api_key: "file:/etc/flexway/secrets/provider-key"   # a reference, read at each call
      default_model: mistral-small-3.2-24b      # the gateway chooses the model
      external: true                             # a third party
    on-prem:
      url: http://vllm.inference.svc:8000/v1
      default_model: mistral-7b-instruct
  budget:
    store: "${COUNTERS_URL}"                     # redis://..., shared by every replica
    monthly_tokens: 40000000                     # per tenant, 0 for none
    daily_tokens: 2000000
  overrides:                                     # the first match wins
    - { match: { tenant: trial }, monthly_tokens: 500000 }
    - { match: { tenant: acme, agent: batch-summaries }, backend: on-prem }

Inference behind a mesh

# Inference behind a mesh ingress that already authenticated the caller and writes its tenant into a
# header: believed from the ingress's range only, from anyone else it is a forgery and ignored. The
# same gateway serves a product's contract; /v1/* gets a rate limit of its own at the route level,
# and its cost is bounded by the token budget.
version: 1
contracts:
  openapi: ./openapi.yaml
server:
  trusted_proxies: [10.42.0.0/16]                # the mesh ingress - and only it
upstreams:
  default:
    url: http://catalog.internal:8080
auth:
  trusted_headers:                               # the names are the deployment's; the product knows none
    tenant: X-Mesh-Tenant
    subject: X-Mesh-Agent                        # the agent acting: overrides and audit, its own bucket
ai:
  security:
    - trusted_headers: {}
  sensitive_header: X-Mesh-Sensitive
  default_backend: eu-provider
  backends:
    eu-provider:
      url: https://api.provider.example/v1
      api_key: "${PROVIDER_KEY}"
      default_model: mistral-small-3.2-24b
      external: true
  budget:
    store: redis://counters.internal:6379
    daily_tokens: 2000000
policies:
  definitions:
    per-caller:
      type: rate-limit
      config: { key: caller, classes: { read: { rps: 100, burst: 200 }, write: { rps: 5, burst: 10 } } }
    inference-rate:
      type: rate-limit
      config: { key: caller, classes: { read: { rps: 20, burst: 40 }, write: { rps: 2, burst: 5 } } }
  chains:
    global: [per-caller]
    routes:
      "/v1/**": [inference-rate]                 # the most precise level wins

# excerpt of examples/ai-mesh/openapi.yaml
openapi: 3.1.0
info: { title: Catalog, version: "1.0.0" }
paths:
  /products:
    get:
      operationId: listProducts
      responses: { "200": { description: The products } }

Cost, soft cap and fallback

# What inference costs, and what happens when a backend fails: prices declared per backend, a
# monthly budget in euros per tenant beside the token budget, a soft cap that warns and moves the
# call to a cheaper backend, and fallbacks tried in order before any byte reaches the client - never
# an external one for a sensitive request.
version: 1
auth:
  issuers:
    - name: customers
      discovery: https://login.example.net/.well-known/openid-configuration
      audience: api
      tenant_claim: org
ai:
  security:
    - oidc: { scopes: [inference] }
  sensitive_header: X-Sensitive
  default_backend: eu-large
  sensitive_backend: on-prem
  backends:
    eu-large:
      url: https://api.provider.example/v1
      api_key: "file:/etc/flexway/secrets/provider-key"
      default_model: mistral-large
      external: true
      price: { input_per_mtok: 2.00, output_per_mtok: 6.00 }   # euros per million tokens
      fallback: [eu-small, on-prem]              # in order, on connection failure, 5xx, timeout
    eu-small:
      url: https://api.provider.example/v1
      api_key: "file:/etc/flexway/secrets/provider-key"
      default_model: mistral-small
      external: true
      price: { input_per_mtok: 0.10, output_per_mtok: 0.30 }
      fallback: [on-prem]
    on-prem:
      url: http://vllm.inference.svc:8000/v1
      default_model: mistral-7b-instruct
      price: { input_per_mtok: 0, output_per_mtok: 0 }      # 0 is a price: internal
  budget:
    store: "${COUNTERS_URL}"
    monthly_tokens: 40000000                     # both budgets hold: the first reached decides
    monthly_eur: 50                              # per tenant, over the calendar month
    soft: { at: 0.8, backend: eu-small }         # past 80 %: Flexway-Budget: soft, moved to eu-small
  overrides:
    - { match: { tenant: acme }, monthly_eur: 500 }