# Reading traffic and usage

> Read the replicas' health, the traffic per operation and the tenants' usage in the console, and simulate a request.

Guide - Since version 0.1.0 - Flexway v1.7.1 - https://flexway.io/en/docs/operations-console-traffic/

> Operations guide: configure, deploy and observe the gateway. What the feature does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

The screens themselves are described in [Reading with the console](https://flexway.io/en/docs/operations-console/). This page says
how to read them while operating.

## Steps

### Each replica's health

Admins - every replica of the gateway has its status: the status rule on
its own signals (in flight against its ceiling, shedding, share shed, its **last** reload rejected -
the 24-hour count stays shown, as information), after the freshness of its last report.

1. Open the Overview: the replicas are nodes under the gateway.
2. Read the state: a replica pushes every 15 s; no report for **45 s**, "No report"; for **2 min**
   without a stop announced, "Degraded" (and the rule `replica-silent` notifies); forgotten after **1 h**. A
   clean stop (drain) is said by `flexway_server_draining`: the replica is "Retired", never degraded - after
   a rollout, the old pods stay so for an hour.
3. Click a replica: its summary, with a link to its errors (`/errors?replica=`).

A replica that is not well makes the gateway "Watch", never "Degraded": the others serve.

### Asynchronous plane

A broker (`kafka`, `amqp` in 0-9-1 or 1.0, `mqtt`, `nats`) and a webhook target are
**upstreams**: the same status rule, on their own signals - failed publications or failed deliveries,
`503 overloaded` and dead letters counted as refusals; their latency is the **broker's acknowledgement**
and their gauge the bounded buffer (`max_in_flight`).

1. Overview: the map draws them with their protocol, published, delivered and subscribers.
2. Catalogue: filter doors by "Publication" and "Subscription".
3. Traffic: the "Asynchronous plane" panel by broker and channel; a door's Operation screen shows its
   outcomes, the acknowledgement, the subscribers open and its webhooks; Errors names `buffer_full` and
   `rejected`.

A viewer reads the published, delivered and `503` of **their** tenant, the acknowledgement of every tenant
(said on screen), never the subscribers open (no tenant dimension); the Upstreams screen stays the admins'.

### History, markers and links

1. Usage -> **History** tab: 13 UTC months, the current one partial, compared with the previous month and
   the same month a year before **at equal period** (from the 1st to today), never with a whole month; an
   admin without a tenant reads one row per tenant; the export gives `month,tenant,measure,value,partial`
   as CSV.
2. On **every** curve, a line marks each configuration reload, applied or refused, of every replica: time,
   version and result in the reading line.
3. "**Copy the link**" keeps the screen, the window, the tenant and the filters; the link grants no right -
   a recipient outside that scope gets a refusal.

### Compare, variations, hour by day

1. "**Compare**" (Traffic, Operation, Upstreams) lays a window B of the same length, chosen by its start,
   dashed on the same curves; it stays in the link.
2. The **biggest variations** rank the operations by their move against the previous window - rate in
   percent, `5xx` in points, p95 in milliseconds -, from 50 requests on.
3. The **hour by day** map averages each UTC hour of each day over 4 weeks. Out of retention nothing is
   drawn: `--`, never `0`.

### Traces without Tempo

In the integrated mode, the console is also the destination of the gateway's traces and keeps the
**failed** requests itself (`5xx`, refusals written after admission except `401` and `404`) and the
**slow** ones (over 1 s, or over half the operation's budget); the rest is dropped. The gateway traces the
request, its **stages** - routing, identity, policies -, its `external-call`s and each attempt to the
upstream, with `retry_count` and `circuit_state`; the stages cost nothing to the requests not sampled.

1. Diagnostics -> Traces: filter, open a trace as a waterfall.
2. Download it as OTLP JSON to replay it in Jaeger or Tempo.

### Usage

Two clocks. The **gauges** (requests per `quota` policy, tokens of the day and the month, euros of the
month) follow **the gateway's** windows, calendar day and month in **UTC**: the ones that decide a `429`.
The **breakdowns** (per day, per backend, per agent, operations, export) follow the chosen period - this
month, last month, 7 days, or `from`-`to` (93 days at most) - in days of the console's time zone
(`FLEXWAY_CONSOLE_TIMEZONE`).

1. Open Usage; an admin without a chosen tenant reads every tenant ranked against its budget.
2. Read each gauge's status: **degraded** when the cap is reached or will be before the window ends,
   **warning** when the soft cap is crossed or the trend (the pace of the last 24 hours) reaches 90 %,
   **healthy** otherwise; no status without a declared cap. The band shows the slowest and fastest of the
   last 7 days.
3. Export: `GET /api/usage/export` gives the period as CSV (`date, tenant, backend, direction, tokens,
   euros, requests`), a viewer's own tenant only; an admin reads every tenant (`GET /api/usage`).

**Who meters** - totals, days, breakdowns, the ranking's totals, export and history count
what **this** gateway meters only (`metered_by!="upstream"`), with a metrics backend as with the
embedded source, whose daily aggregates keep `metered_by`. The **gauges** (and their trend, band,
date reached, status) count what the **local cap** counts, which applies to relayed calls too: every
series - tokens from the usage returned, euros at a declared price only -, and the gauge says so
("includes the relayed calls, which the local cap counts too"). A backend with `metering: upstream` relays
to another Flexway that counts the call itself: its tokens and calls are shown **apart**, under "Metered
upstream", never added to the totals - the upstream gateway is authoritative for the cost. The breakdown
**by capability** (`chat`, `vision`, `embeddings`, `model`) counts a specialised gRPC model in calls, not
tokens. The **Declared backends** panel says, from `/config`, each backend's kind (`openai`, `relay`,
`grpc`), the `capabilities`, who meters it and where it is; an admin also reads the authentication **scheme**
(`none`, `bearer`, `header`, `token_file`) - never a key or its reference.

### Try a request

Under the pipeline of the Policies screen:

1. Describe the method, path, host, identity (subject, tenant, scopes, roles) and headers.
2. Run it: the description goes through the gateway's `/simulate`, as a dry run.
3. Read the verdict, the request the upstream would have received (secrets `***`) and each step's result:
   applied, skipped, not simulated (an `external-call` is never called), passed.

A viewer simulates within **their** tenant: naming another is refused.

### Go to

`Ctrl+K` (`Cmd+K` on a Mac), or the top bar's field, from any screen: a screen, an operation (opens its
chain), an upstream (opens its detail, admins), an agent (opens its panel; those of their tenant for a
viewer), a tool or a tenant (filters the console; their own for a viewer). Arrows, Enter, Esc.

## Verify

- The configuration version and reading time at the top of every screen are the ones the gateway's admin
  `/config` publishes.
- An applied reload shows as a marker on the Traffic curves within the minute.
- A viewer naming another tenant (link, filter, simulation) gets a refusal, never an empty page.

*Available since `0.1.0`; simulation and palette in `1.2.0`; history, markers, links, comparison,
variations, hour by day and kept traces in `1.3.0`; per-replica health in `1.4.0`; the asynchronous plane
and metering where it is paid in `1.5.0`.*