clover's git

the-knife

1	---
2	title: "Alerting: monitors & notifiers"
3	description: Turn APL queries into alerts — Threshold, MatchEvent, and AnomalyDetection monitors wired to Slack, email, PagerDuty, and webhook notifiers, all as resources in your Stack.
4	---
5
6	An Axiom `Monitor` is a saved APL query that runs on a fixed cadence
7	and fires when its condition is met. A `Notifier` is the destination
8	it fires to — Slack, email, PagerDuty, a webhook. In alchemy both are
9	resources in your Stack, so a threshold change or a new on-call
10	channel is a reviewable diff, not a console edit.
11
12	Everything below runs inside your Stack's `Effect.gen` — see
13	[Setup](/axiom/setup) for credentials and the
14	[Axiom overview](/axiom) for datasets like `app-traces` and
15	`app-logs` used in the queries.
16
17	## Create a notifier
18
19	```typescript
20	import * as Axiom from "alchemy/Axiom";
21
22	const slack = yield* Axiom.Notifier("ops-slack", {
23	name: "ops-channel",
24	properties: {
25	slack: { slackUrl: process.env.SLACK_WEBHOOK_URL! },
26	},
27	});
28	```
29
30	Exactly one channel goes under `properties` — here `slack` with an
31	incoming webhook URL. The resource's `id` output is what monitors
32	target.
33
34	## Threshold: alert when a value crosses a line
35
36	```typescript
37	yield* Axiom.Monitor("error-rate", {
38	name: "High error rate",
39	description: "Fires when error count exceeds 100/5m",
40	type: "Threshold",
41	aplQuery: `
42	['app-traces']
43	| where status >= 500
44	| summarize count() by bin_auto(_time)
45	`,
46	operator: "Above",
47	threshold: 100,
48	intervalMinutes: 5,
49	rangeMinutes: 5,
50	alertOnNoData: false,
51	resolvable: true,
52	notifierIds: [slack.id],
53	});
54	```
55
56	Every `intervalMinutes`, the monitor runs `aplQuery` over the last
57	`rangeMinutes` and compares the aggregate against `threshold` using
58	`operator` (`Above`, `AboveOrEqual`, `Below`, `BelowOrEqual`, or
59	`AboveOrBelow`). `resolvable: true` sends a resolution notice when
60	the value drops back under the line; `alertOnNoData` controls whether
61	an empty result counts as a problem.
62
63	`notifierIds: [slack.id]` is the wiring: `slack.id` is an output
64	attribute of the notifier flowing into the monitor's inputs, so
65	alchemy deploys the notifier first and fills in the real id.
66
67	## MatchEvent: alert on every matching event
68
69	```typescript
70	yield* Axiom.Monitor("panics", {
71	name: "Service panic",
72	type: "MatchEvent",
73	aplQuery: `['app-logs'] | where message contains "panic:"`,
74	intervalMinutes: 1,
75	rangeMinutes: 1,
76	notifierIds: [slack.id],
77	});
78	```
79
80	A `MatchEvent` monitor has no threshold — it fires for every event
81	the query matches. Use it for rare, individually-actionable events
82	like panics or failed deploys, where one occurrence is already worth
83	a page.
84
85	## AnomalyDetection: alert on deviation from baseline
86
87	```typescript
88	yield* Axiom.Monitor("traffic-anomaly", {
89	name: "Traffic anomaly",
90	type: "AnomalyDetection",
91	aplQuery: `['app-traces'] | summarize count() by bin_auto(_time)`,
92	compareDays: 7,
93	tolerance: 25, // %
94	intervalMinutes: 15,
95	rangeMinutes: 15,
96	notifierIds: [slack.id],
97	});
98	```
99
100	Instead of a static threshold, the monitor learns a baseline from the
101	same window over the previous `compareDays` days and fires when the
102	current result deviates by more than `tolerance` percent. Good for
103	"traffic looks wrong" signals where the right absolute number changes
104	by hour and weekday.
105
106	Changing a monitor's `type` triggers a replacement; every other prop
107	updates in place.
108
109	## Fan out to multiple notifiers
110
111	```typescript
112	const pagerduty = yield* Axiom.Notifier("pagerduty", {
113	name: "primary-oncall",
114	properties: {
115	pagerduty: { routingKey: process.env.PAGERDUTY_ROUTING_KEY!, token: "" },
116	},
117	});
118
119	yield* Axiom.Monitor("error-rate", {
120	// ...same props as above
121	notifierIds: [slack.id, pagerduty.id],
122	});
123	```
124
125	`notifierIds` takes any number of ids, so one monitor can post to the
126	team channel and page on-call at the same time. Notifier channels
127	cover Slack, email (`email: { emails: [...] }`), PagerDuty, Opsgenie,
128	Discord, Microsoft Teams, and generic webhooks.
129
130	## Custom webhook with a templated body
131
132	```typescript
133	yield* Axiom.Notifier("incident-webhook", {
134	name: "incident.io",
135	properties: {
136	customWebhook: {
137	url: "https://api.incident.io/v2/alert_events",
138	headers: { "Content-Type": "application/json" },
139	secretHeaders: { Authorization: `Bearer ${process.env.INCIDENT_TOKEN}` },
140	body: '{"title": "{{.Monitor.Name}}", "status": "firing"}',
141	},
142	},
143	});
144	```
145
146	For anything without a built-in channel, `customWebhook` gives you
147	full control over the request: static `headers`, `secretHeaders` for
148	credentials, and a `body` template with access to monitor fields like
149	`{{.Monitor.Name}}`.
150
151	## Derived alert fields with VirtualField
152
153	```typescript
154	yield* Axiom.VirtualField("status-class", {
155	dataset: "app-traces",
156	name: "status_class",
157	description: "HTTP response class bucket",
158	expression: 'strcat(tostring(toint(status / 100)), "xx")',
159	type: "string",
160	});
161	```
162
163	A `VirtualField` is a saved APL expression that appears as a derived
164	column on its `dataset` at query time. Define common computations —
165	status classes, latency buckets, parsed JSON paths — once, instead of
166	repeating the expression in every monitor and dashboard. A virtual
167	field is bound to one `dataset`; changing the dataset triggers a
168	replacement.
169
170	Monitors on that dataset can then query the derived column directly:
171
172	```typescript
173	yield* Axiom.Monitor("5xx-rate", {
174	name: "5xx rate",
175	type: "Threshold",
176	aplQuery: `
177	['app-traces']
178	| where status_class == "5xx"
179	| summarize count() by bin_auto(_time)
180	`,
181	operator: "Above",
182	threshold: 50,
183	intervalMinutes: 5,
184	rangeMinutes: 5,
185	notifierIds: [slack.id],
186	});
187	```
188
189	If the definition of "5xx" ever changes, you update the virtual
190	field's `expression` — every monitor and dashboard built on
191	`status_class` follows.
192
193	## Where next
194
195	- [Ingest](/axiom/data/ingest) — datasets, ingest tokens, and wiring OTLP
196	endpoints into your Workers and Functions.
197	- [Dashboards](/axiom/guides/dashboards) — chart the same queries
198	your monitors run.
199	- [Axiom overview](/axiom) — all Axiom resources at a glance.
200
201	Reference:
202
203	- [Monitor API reference](/providers/axiom/monitor)
204	- [Notifier API reference](/providers/axiom/notifier)
205	- [VirtualField API reference](/providers/axiom/virtualfield)