Rate limits, quotas and 429

How the gateway tells you to slow down, and how a well-written client adjusts.

Product side: Traffic control and quotas; operations side: Setting quotas.

The headers

Header	When	Meaning
RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset	On every limited response, successes included	Slow down before being refused
Retry-After	429, 503	Seconds to wait

The refusals

Response	Meaning	Wait
429 rate_limited	Your rate (per identity, per class or per operation)	Retry-After, a few seconds
429 quota_exceeded	Your tenant's quota for the day or the month (UTC)	Retry-After until the end of the window
429 budget_exceeded	The tenant's inference budget (inference)	End of the window
503 overloaded	The gateway sheds load under saturation (Flexway-Degraded: shedding)	Retry-After

The rate counts per identity (the token's subject, the API key, the certificate), not per address. A cache hit counts too.

import random, time, requests
def call(url, **kw):
    for attempt in range(5):
        r = requests.get(url, **kw)
        if r.status_code not in (429, 503):
            return r
        wait = int(r.headers.get("Retry-After", 1))
        time.sleep(wait + random.uniform(0, wait / 2))   # jitter: do not all come back together
    return r

For Kafka, the rate is told by throttle_time_ms: see Kafka clients.