# Alerting from the console

> Declare rules, signed webhooks and summaries in the console's values, tune the built-in rules and verify the signature.

Guide - Since version 1.3.0 - Flexway v1.7.1 - https://flexway.io/en/docs/operations-console-alerts/

> Operations guide: configure, deploy and observe the gateway. What the feature does, product side: [product guide](https://flexway.io/en/docs/product-overview/); calling it: [developer guide](https://flexway.io/en/docs/developer-overview/).

The console evaluates its rules every 15 s, **in both modes**, and notifies through a **signed webhook**.
Nothing is typed into the console: rules, webhooks and summaries are declared in the chart's values, read
again without a restart; an invalid version is refused and said on the Alerts screen, the previous one
keeps applying.

## Steps

### Declare rules, webhooks and summaries

1. Write the rules, webhooks and summaries in the console's values:

   ```yaml title="values.yaml"
   # The console's values
   alerts:
     rules:     # closed kinds: error_rate, p95, breaker_open, slo_burn, new_error_code
       - { name: reports-5xx, kind: error_rate, upstream: reports, threshold: 5%, for: 1m, notify: [ops] }
       - { name: orders-p95, kind: p95, operation: createOrder, threshold: 300ms, for: 2m }
     webhooks:  # https required (allowHttp lifts it); secretFile: whsec_<base64>, one secret per line
       - { name: ops, url: "https://hooks.example/flexway", secretFile: /etc/flexway-console-alerts/ops }
       - { name: chat, urlFile: /etc/flexway-console-alerts/chat-url, secretFile: /etc/flexway-console-alerts/chat, allowPrivate: true }
     summaries:
       - { name: weekly, every: weekly, at: "07:00", weekday: monday, webhook: chat }   # UTC
     existingSecret: console-alerts    # secrets and URLs carrying a token, mounted at /etc/flexway-console-alerts
   slos:
     - { operation: createOrder, availability: 99.9%, latency: { p95: 300ms, target: 99% }, window: 30d }
     - { name: orders-api, upstream: orders-svc, availability: 99.5% }   # a group: see SLO objectives on groups of operations
   ```

2. Put the secrets and the URLs carrying a token in the `Secret` named by `existingSecret`.
3. Deploy the values: the console reads the file again without a restart.

### Tune the built-in rules

Tune a built-in rule rather than silence it - a few cases:

```yaml title="values.yaml"
alerts:
  builtins:
    # Tighten: a small publication traffic fires too early; 5 % refused, from 100 publications.
    - { name: broker-backpressure, threshold: 5%, minRequests: 100 }
    # Delay: a broker failing over in less than 3 min is not an incident.
    - { name: broker-unavailable, for: 3m }
    # Sooner: an open breaker already fires at once; a degraded gateway, after 30 s.
    - { name: gateway-degraded, for: 30s }
```

With the native Kafka proxy (since `1.6.0`), the `kafka-proxy-refusals` rule counts closed causes and
accepts `excludeCauses` ([Native Kafka clients: a virtual cluster per tenant](https://flexway.io/en/docs/operations-kafka-clients/#console-virtual-clusters-kafka-clients-rules)):

```yaml title="values.yaml"
alerts:
  builtins:
    - { name: kafka-proxy-refusals, threshold: 10%, excludeCauses: [quota] }   # do not fire on a quota reached
```

With the MQTT device listener (since `1.7.0`), five more rules - `device-connections-refused`,
`sparkplug-node-offline`, `mqtt-listener-saturation`, `device-throttled` and `sparkplug-sequence-gaps`, the last two
disabled by default; the `threshold` of the two Sparkplug rules is a count, not a ratio
([Console: the Devices screen](https://flexway.io/en/docs/operations-console-devices/#tuning-the-five-built-in-rules)).

### Verify the signature on the receiving side

The body: `{"type": "flexway.alert.firing", "timestamp": ..., "data": {"rule", "kind", "state", "since",
"target", "value", "unit", "threshold", "subjects"}}` (`flexway.alert.resolved`, `flexway.summary.daily`,
`flexway.summary.weekly`) - never a tenant. Verify the signature ([Standard
Webhooks](https://www.standardwebhooks.com/)) before anything else:

```python
import base64, hashlib, hmac, time

def verify(secret: str, headers, body: bytes) -> bool:
    key = base64.b64decode(secret.removeprefix("whsec_"))
    msg_id, ts = headers["webhook-id"], headers["webhook-timestamp"]
    if abs(time.time() - int(ts)) > 300:          # replay: 5 minutes of tolerance
        return False
    mac = hmac.new(key, f"{msg_id}.{ts}.".encode() + body, hashlib.sha256).digest()
    want = base64.b64encode(mac).decode()
    # two signatures during a rotation, separated by a space: one must match
    return any(hmac.compare_digest(sig.removeprefix("v1,"), want)
               for sig in headers["webhook-signature"].split())
```

### Rotate a secret

1. Put the new secret on the file's second line on the console's side.
2. Deploy it on the receiving side.
3. Remove the old one - the console signs with both meanwhile.

## Verify

- The Diagnostics -> Alerts screen (admins) lists each rule, its state (`ok`, pending, firing, resolved,
  unknown), its value against the threshold, its curve over 6 h and each webhook's deliveries (result,
  status, cause).
- A refused file is said there, with its cause; the previous version keeps applying.
- `GET /api/alerts` gives the built-in rules' values in force and where they come from (`tuning`).

## Good to know

| Topic | What to know |
| --- | --- |
| Built-in rules | Always on: `breaker-open` (an open breaker), `upstream-degraded` and `gateway-degraded` (1 min), `new-error-code` (a code never seen on the operation in 7 days - it reads `flexway_errors_total`), `slo-burn` (as soon as an objective is declared, an operation's or a group's), `replica-silent` (a gateway replica without a report for 2 min and no stop announced), `broker-unavailable` (a broker's breaker open or more than half its publications failing, 1 min), `broker-backpressure` (more than 1 % of `503 overloaded`, 5 min), `webhook-dead-letter` (a delivery in the dead letter over 15 min) |
| Judgement | `error_rate` = share of `5xx` over 1 min, from 20 answers; `p95` over 5 min. Silent metrics: no state changes, nothing is sent nor withdrawn, the screen says `unknown`. A target missing from the configuration makes the rule invalid, never notified |
| Who sends | Every replica evaluates; one notifies, the holder of the chart's `Lease`. The `webhook-id` is the same from one replica to another: deduplicate on it (at least once) |
| Webhook | `https`, the address checked when the connection is made: loopback, link-local (`169.254.169.254`) and multicast always refused, a **private network** only with `allowPrivate: true` (an internal target); no redirection followed; 5 s timeout; retries 5 s, 30 s, 2 min, 10 min, 30 min, then an abandon shown on the screen; a `4xx` (but `408`, `429`) is a final refusal |
| Silence | Alerts screen, 24 h at most, a reason required: it suspends a rule's **notifications**, never its evaluation; when it ends, what still fires is notified |
| Summaries | `daily` or `weekly` at a UTC hour: requests against the previous period, `5xx` and `4xx` shares, new codes, each SLO's budget, the five largest changes, alerts - never a tenant's name |
| SLOs | Availability (non-`5xx` answers) and latency (share of the minutes whose p95 holds the target) over the window; remaining budget, burn rate over 1 h, 6 h, 3 days; `slo-burn` fires above 14.4 over 1 h and 5 min, or 6 over 6 h and 30 min. On an operation or, since `1.6.0`, a **group of operations** - an upstream, a tag, a list, every one ([SLO objectives on groups of operations](https://flexway.io/en/docs/operations-slo-groups/)). Traffic's **Objectives (SLO)** tab, the Operation panel, the global objective on the Overview; "Create an alert" on the charts and the objectives writes the YAML to paste in these values |
| Tuning the built-in rules | `alerts.builtins`: the `name` of a built-in rule, `for` (0 to 1 h) on every one; `threshold` (a ratio) and `minRequests` only on `broker-unavailable` (half, 20) and `broker-backpressure` (1 %, 20); `excludeCauses` only on a rule counting closed causes (the native Kafka proxy's rules, since `1.6.0`). An unknown name, a rule tuned twice or a knob the rule does not carry refuses the file, as the rest does; without the key, the defaults. The Alerts screen shows the tuned threshold |

*Available since `1.3.0`; tuning of the built-in rules in `1.6.0`.*