Flexway - Setting quotas (Operations)

Operations guide: configure, deploy and observe the gateway. What the capability does, product side: product guide; calling it: developer guide.

Goal: limit per caller, not per address of the pod that relays, and keep a slow or dead upstream from becoming a general outage.

Command - a rate-limit policy, attached globally or per host:

policies:
  definitions:
    per-caller:
      type: rate-limit
      config:
        key: caller                  # the token's subject, else the client's address
        classes:
          read:  { rps: 50, burst: 100 }   # GET, HEAD, OPTIONS
          write: { rps: 10, burst: 20 }    # everything else
  chains:
    global: [per-caller]
server:
  trusted_proxies: [10.0.0.0/8]      # the edge: its X-Forwarded-For is believed, nobody else's

Behind a cloud load balancer, the edge may not see the client either. A layer-4 load balancer connects to the edge from its own private address: every request, then, reaches the edge - and the gateway - from one address, and all anonymous callers share one bucket. Turn the proxy protocol on at both ends: the load balancer sends it (a Service annotation of the provider, for example service.beta.kubernetes.io/scw-loadbalancer-proxy-protocol-v2: "*" on Scaleway or service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" on AWS) and the edge's HTTPS listeners require it, which also stops a client from choosing its own address. With Envoy Gateway:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata: { name: public-client, namespace: envoy-gateway-system }
spec:
  targetRefs:
    - { group: gateway.networking.k8s.io, kind: Gateway, name: public, sectionName: https }
  proxyProtocol: {}                  # the load balancer sends it; a connection without it is refused
The two settings go together: one without the other drops the traffic. Check the result in the edge's access log - x-forwarded-for must show public addresses, not the load balancer's - and keep server.trusted_proxies on the edge's range only.

An operation changes class in the contract - the class must exist in the policy, or the gateway refuses to start:

      x-flexway-rate-class: write

A bucket per operation (1.0.0): by default (scope: class), a caller has one bucket per class, which every operation of the class drains - a journey chaining three writes in the same second spends three tokens of the same bucket. scope: operation gives each operation its own bucket per caller, the class setting its size - what most gateways count. A class may override the policy's scope; shared counters (server.counters) follow with nothing more to set.

      config:
        scope: operation                  # default: class
        classes:
          read:  { rps: 50, burst: 100 }
          write: { rps: 10, burst: 20 }
          sensitive: { rps: 1, burst: 3, scope: class }   # one bucket for all the sensitive operations

Behind a client server (1.0.0): a BFF relays anonymous users, all from its own address. The header key counts each one apart, by a header the BFF sets - believed only when the peer proved one of the trusted identities. Otherwise the header is ignored and the address stays the key: a key a caller chooses is no key. An authenticated subject always stays the key.

policies:
  definitions:
    per-user:
      type: rate-limit
      config:
        key: header
        header: X-Client-Key
        trusted: [web-bff.web.serviceaccount.identity.linkerd.cluster.local]
        classes: { read: { rps: 20, burst: 40 } }
server:
  trusted_proxies: [local]                  # the mesh proxy: the pod's own addresses
  peer_identity_header: l5d-client-id       # the mTLS identity it verified

Not 127.0.0.1/32: the mesh proxy connects from the pod's IP, not from the loopback (measured with Linkerd). local names the pod's own addresses, resolved at load - never a CIDR of pods, which would trust every pod. Behind a mesh, count anonymous callers by peer identity as above, never by address: the mesh does not clean X-Forwarded-For.

The peer's identity comes from the certificate the gateway's own TLS listener verified, else - from a peer of server.trusted_proxies only - from server.peer_identity_header when declared (and then from it alone), else from x-forwarded-client-cert. Precondition of peer_identity_header: the local proxy overwrites that header when a caller sends it; without that guarantee, any pod forges the key.

Validation and errors (1.0.0): the gateway validates parameters and body (full, the default). A service that stays the authority on its body asks for params - the body is relayed unread, neither validated nor bounded by max_body - or off - nothing is validated. No mode opens the whitelist: an undeclared parameter is still refused. The errors' type points at your portal:

hosts:
  api.example.com: { default_upstream: orders, validation: params }   # the whole host
errors:
  type_base: https://docs.example.com/errors/    # type = <base><code>
  field_codes: { minLength: too_short, minimum: out_of_range }   # errors[].code in your own words

Each entry of errors carries field, rule (for a human) and code, what a client switches on: the JSON Schema keyword broken (required, type, enum, minLength, maximum, additionalProperties...), or invalid when no schema could be applied (an unreadable body). errors.field_codes renames a keyword to the token of your contract; a keyword the table does not list keeps its name. A missing or extra property is named as the field (body.age), and every broken rule has its own entry.

      x-flexway-validation: full     # in the contract: the operation wins over the host

Per upstream, the bulkhead and the circuit breaker (on by default, values shown are the defaults):

upstreams:
  orders:
    url: http://orders:8080
    max_in_flight: 200              # no cap when absent
    circuit_breaker: { failure_ratio: 0.5, min_requests: 20, window: 10s, cooldown: 5s }

Check:

Situation	Answer
Empty bucket	`429`, `Retry-After`, `RateLimit-Remaining: 0` - two users behind one server-side client have separate buckets
Any rate-limited response	`RateLimit-Limit`, `-Remaining`, `-Reset`
Body or parameter outside the schema	`422`, the `errors` extension names each field and its `code` - under `params`, the body is no longer examined
Method not declared on a declared path, `QUERY` or an extension token included	`405` with `Allow`
Body past `server.limits.max_body` (1 MiB)	`413`
Upstream down, circuit open	`503` at once, `Retry-After`, `Flexway-Degraded: circuit-open`
Upstream saturated	`503` at once, `Flexway-Degraded: upstream-saturated`
Tenant quota spent	`429` `quota_exceeded`, `Retry-After` until midnight or the end of the month (UTC)
Gateway saturated	`503` `overloaded` on the sheddable first, `Flexway-Degraded: shedding`; `critical` never refused

Share across replicas, sell plans, shed load (1.0.0) - without server.counters, counters are local to each replica: three replicas give up to three times the rate. With it, each replica decides alone and reconciles with the store every sync - no round trip on the path, a limit held within one sync window:

server:
  counters: { store: "${COUNTERS_URL}", sync: 1s }   # Redis, Dragonfly or Valkey; not reloadable
  shedding: { target: 20ms }                         # optional; not reloadable
  in_flight: { max: 2048, queue: 0 }                 # optional - derived from the container's memory limit when absent; not reloadable
policies:
  definitions:
    plans:
      type: quota                                    # per tenant; needs server.counters
      config: { limits: { day: 100000, month: 2000000 } }
  chains:
    global: [per-caller, plans]

      x-flexway-priority: sheddable     # in the contract: critical | normal (default) | sheddable

Store lost: the gateway serves, on local counters, and says so (flexway_counters_store_up at 0, a WARN line). Shedding follows the Go runtime's scheduling latency - the gateway's own saturation, not an upstream's slowness.

Setting quotas

Next steps