clover's git

the-knife

1	---
2	title: Datasets & ingest
3	description: Axiom datasets per OTel signal with OTLP endpoints as outputs, least-privilege ingest tokens, and per-stage naming with Stack.useSync.
4	---
5
6	A `Dataset` is Axiom's top-level container — it stores your events,
7	logs, traces, or metrics. An `ApiToken` is the credential your
8	runtime ships with. Both are one-line resources, declared in the
9	same Stack as the Workers and Functions that emit the data.
10
11	## Create a dataset
12
13	```typescript
14	import * as Axiom from "alchemy/Axiom";
15
16	const logs = yield* Axiom.Dataset("logs", {
17	name: "my-app-logs",
18	kind: "otel:logs:v1",
19	description: "Application logs from prod workers",
20	retentionDays: 30,
21	useRetentionPeriod: true,
22	});
23	```
24
25	`description`, `retentionDays`, and `useRetentionPeriod` update in
26	place, so a retention change is a reviewable diff.
27
28	:::caution[Name and kind changes delete data]
29	`name` is the dataset's stable identifier in Axiom and `kind` is
30	fixed at creation — changing either triggers a **replacement**: the
31	old dataset is deleted along with everything ingested into it. Pick
32	both up front, and never rename a dataset that holds data you care
33	about.
34	:::
35
36	## One dataset per signal
37
38	`kind` determines the schema and how the data renders in the Axiom
39	UI. For OTel pipelines, create one dataset per signal:
40
41	```typescript
42	const traces = yield* Axiom.Dataset("traces", { name: "app-traces", kind: "otel:traces:v1" });
43	const logs = yield* Axiom.Dataset("logs", { name: "app-logs", kind: "otel:logs:v1" });
44	const metrics = yield* Axiom.Dataset("metrics", { name: "app-metrics", kind: "otel:metrics:v1" });
45	```
46
47	The default `kind` is `axiom:events:v1` — Axiom's general-purpose
48	event store — for anything that isn't OTLP.
49
50	## OTLP endpoints as outputs
51
52	Each dataset exposes Axiom's OTLP/HTTP endpoints as output
53	attributes, and alchemy runtimes have
54	[telemetry built in](/infrastructure-as-effects/telemetry) — wiring
55	an exporter is providing the `Axiom.Telemetry` binding layer on the
56	Function/Worker:
57
58	```typescript
59	// src/observability.ts — declared at module scope so the Worker can import them
60	export const Traces = Axiom.Dataset("Traces", { name: "app-traces", kind: "otel:traces:v1" });
61	export const Logs = Axiom.Dataset("Logs", { name: "app-logs", kind: "otel:logs:v1" });
62	export const Ingest = Axiom.ApiToken("Ingest", {
63	name: "app-ingest",
64	datasetCapabilities: {
65	"app-traces": { ingest: ["create"] },
66	"app-logs": { ingest: ["create"] },
67	},
68	});
69
70	// src/worker.ts — on the Worker's constructor Effect, merged with its other binding layers:
71	Effect.provide(
72	Layer.mergeAll(
73	Cloudflare.R2.ReadWriteBucketBinding,
74	Axiom.Telemetry({ token: Ingest, traces: Traces, logs: Logs }),
75	),
76	)
77	```
78
79	Building the layer binds each dataset's endpoint and the ingest
80	token's `Authorization` header (as a secret) onto the host; at
81	runtime the built-in exporter ships each signal to its dataset,
82	flushed per request. See the
83	[end-to-end guide](/cloudflare/observability/axiom-observability)
84	for the full pattern.
85
86	- `otelTracesEndpoint` / `otelLogsEndpoint` / `otelMetricsEndpoint` —
87	the per-signal OTLP/HTTP URLs.
88	- `otelEndpoint` — the root; most exporters auto-append the signal
89	path.
90	- `otelHeaders` — the non-secret headers Axiom expects, i.e.
91	`X-Axiom-Dataset: <name>`. The `Authorization: Bearer` header is
92	deliberately excluded so no secret lives in dataset state — that's
93	what the ingest token below is for.
94
95	## Mint an ingest token
96
97	An `ApiToken` scoped with `datasetCapabilities` can ingest into
98	exactly the datasets you list and nothing else:
99
100	```typescript
101	const ingest = yield* Axiom.ApiToken("ingest", {
102	name: "prod-ingest",
103	description: "OTEL collector ingest",
104	datasetCapabilities: {
105	"my-app-traces": { ingest: ["create"] },
106	},
107	});
108	```
109
110	The same shape covers read-side credentials — for example a
111	query-only token for Grafana:
112
113	```typescript
114	yield* Axiom.ApiToken("query", {
115	name: "grafana-reader",
116	datasetCapabilities: {
117	"my-app-traces": { query: ["read"] },
118	"my-app-logs": { query: ["read"] },
119	},
120	});
121	```
122
123	## Token rotation
124
125	Axiom has no update endpoint for tokens, so capabilities are pinned
126	at creation: changing **any** prop replaces the token and mints a
127	new bearer value; identical props never rotate it. That makes
128	rotation a deploy — tweak the `description` (or any other field)
129	and the next deploy creates a fresh token, rewires every consumer,
130	and deletes the old one.
131
132	:::caution[The bearer persists in alchemy state]
133	Axiom returns the raw token exactly once, at create time. Alchemy
134	captures it into `ingest.token` as a `Redacted<string>` and persists
135	it in resource state — otherwise later deploys couldn't rewire it
136	into consumers. Treat your state store as sensitive: anyone with
137	read access can recover the bearer. Pass `ingest.token` to secret
138	sinks (a Worker env var becomes a `secret_text` binding; or a
139	[secrets store](/cloudflare/security/secrets-env)) rather than ever unwrapping it
140	yourself.
141	:::
142
143	## Per-stage datasets
144
145	Dataset names are global to your Axiom org, so give each stage its
146	own. `Stack.useSync` computes props from the resolved stage — handy
147	at module scope where there's no `yield*` available:
148
149	```typescript
150	import * as Axiom from "alchemy/Axiom";
151	import { Stack } from "alchemy/Stack";
152
153	export const Traces = Axiom.Dataset(
154	"Traces",
155	Stack.useSync(({ stage }) => ({
156	name: `${stage}-traces`,
157	kind: "otel:traces:v1" as const,
158	description: `OTEL traces for stage '${stage}'`,
159	retentionDays: 30,
160	})),
161	);
162	```
163
164	Now `dev-sam` telemetry never lands in the `prod` dataset, and
165	tearing down a preview stage deletes its data with it. See
166	[Stack](/infrastructure-as-code/stack) for the full `useSync` semantics.
167
168	## Where next
169
170	- [Ship Worker telemetry to Axiom](/cloudflare/observability/axiom-observability) —
171	the end-to-end guide: datasets, token, env wiring, and the
172	zero-exporter Cloudflare push path.
173	- [Alerting](/axiom/guides/alerting) — monitors and notifiers over
174	the data you just ingested.
175	- [Telemetry](/infrastructure-as-effects/telemetry) — the built-in
176	OTel exporters and how they flush per request.
177	- Reference: [Dataset](/providers/axiom/dataset) ·
178	[ApiToken](/providers/axiom/apitoken)