# AI and inference

> Serves an OpenAI-compatible surface (completions, model list, embeddings, images as input) through the same data plane as the APIs. Picks the backend by…

Concept - Since version 0.1.0 - Flexway v1.6.3 - https://flexway.io/en/docs/product-ai-inference/

> Product guide: what the capability brings, for whom, its limits and the version that delivered it - no configuration syntax.

| | |
| --- | --- |
| **What it does** | Serves an **OpenAI-compatible** surface (completions, model list, embeddings, images as input) through the same data plane as the APIs. Picks the backend by policy, pins a sensitive request to an internal backend, falls back to another backend on an outage, and holds a **budget in tokens and euros** per tenant |
| **Why it matters** | A requests-per-second limit says nothing about the cost of a model call; an agent in a loop is an invoice, not a rate incident. An existing client (an OpenAI SDK) changes only its base URL. Prompt content is **never** logged: the audit keeps metadata |
| **Use cases** | Offer models to several teams with a monthly budget each; move to a cheaper model past 80 % of the budget; keep sensitive data on an in-house model; meter at a provider or behind another Flexway; meter a specialised model called over gRPC |
| **Limits** | Not built yet: routing by declared task (a small model for a simple task), choice by hosting, region, cost or latency, model canary and allowed models per caller, data protection (classes, residency, detection and masking, input and output guardrails), inference-aware routing and the semantic cache come in `1.9.0`; budgets per user and per use case, then per application, team, project and cost centre, with thresholds and a chargeback export, in `1.8.0` |
| **Since** | `0.1.0` (inference on the data plane, budgets, fallback), `1.5.0` (several backends, images, embeddings, specialised models) |
| **Configure** | [Serving inference](https://flexway.io/en/docs/operations-inference/) |