Rules

A rule is a threshold condition over a metric your application emits, evaluated on a schedule against your own telemetry in ClickHouse. There is one rule engine for the whole product: no separate alerting system for uptime versus custom metrics.

Rules are declared, not created in the console

A rule is authored as a manifest and applied through your connected GitOps repository, the same as a monitor tag or a status page component. The console's /kinds/rules page is a read view: it shows the rule as declared, with its manifest path and the commit that applied it, and there is no edit or create affordance.

There is also no rule CRUD API: GET /api/v1/organizations/{org_id}/rules and GET .../rules/{id} are read-only, and every write verb is refused.

The condition shape

# up0/rules/payments-api-high-latency.yaml
apiVersion: up0.io/v1
kind: Rule
metadata:
  name: payments-api-high-latency
spec:
  metric: http_request_duration_ms
  filters:
    service.name: payments-api
  aggregation: avg              # avg | max | min | sum | count
  windowSeconds: 300
  condition: gt                 # gt | gte | lt | lte | eq | neq
  threshold: 800
  for: 5m
  evaluationIntervalSeconds: 60
  severity: warning
  isEnabled: true

This is a structured aggregation-and-threshold condition, not a query language: pick a metric, an aggregation, a comparison, and how long the condition has to hold continuously (for) before the rule fires. Rules are evaluated against metrics only today; logs and traces are a later addition, with different condition shapes.

An invalid manifest, an unknown metric, an unparseable for, or two rules declaring the same name are all refused at apply time, naming the field and the reason, rather than being silently accepted as a rule that can never fire.

Rule count is capped by plan tier, the same limit shape Workflow already uses: free plans get none, and the limit-crossing apply is refused by name rather than partially applying the batch.

Evaluation and alert state

A firing evaluation moves through a small state machine: ok, pending (the condition is breaching but hasn't held for the full for duration yet), firing (it has), and resolved. A rule whose query returns no data at all enters a separate no_data state rather than being read as a silent recovery, since a collector dying and a genuine recovery would otherwise look identical.

A continuing breach does not repeat notifications every evaluation cycle: one alert, one firing event, no matter how many cycles the condition keeps holding. Several series breaching the same rule at once collapse into one alert group by default, or one group per label combination if the rule declares groupBy, so one upstream outage produces one page, not one per affected series.

What a firing rule produces

Firing reuses the workflow engine rather than a second notification system: crossing into firing emits a rule.firing event, and recovering emits rule.resolved, both of which any workflow can trigger on (a push notification, for example). A firing rule also originates an incident, with its severity derived from the rule's own severity rather than a hardcoded value, and the incident links back to the alert that produced it. See Incidents for what happens after that.

Evidence for the behaviour above

Because no rule exists on this environment to observe firing, the claims in this page are backed by a live walk against a running stack rather than this environment: docs/v2/testing/m07-rules-alerting.md (org safoor, 2026-09-02, 26 test cases). Driving a real metric past a declared threshold produced the ok to pending to firing sequence described above, with both transitions timestamped in the alert's own history; a brief spike that recovered before for elapsed reached pending and returned to ok without ever firing or notifying; ten breaching evaluations in a row produced exactly one firing transition and one notification, not ten; and three series breaching the same ungrouped rule collapsed into one incident, while the same rule declared with groupBy: [host] produced one incident per host.

That walk also found and fixed the two defects that would otherwise have made this milestone non-functional: the evaluator was reading an empty, still-filling data bucket instead of the declared window, so no metric rule could fire at all, and the GitOps sync worker never opened a ClickHouse connection, so no rule manifest could be applied at all. Both are fixed and re-verified live.