clover's git

the-knife

1	---
2	title: Looping the Generation of IaC and SDKs
3	date: 2026-07-02T16:00:00Z
4	excerpt: AI agents write resources and live tests, run them against the real API, and every unmatched error becomes a typed patch to our generated SDK. One run of this loop grew Alchemy's Cloudflare provider from 22 resources to 230 and patched 720 SDK operations, most of it in 22 hours.
5	---
6
7	import LoopDiagram from "../../../components/LoopDiagram.astro";
8
9	<LoopDiagram />
10
11	[Alchemy](/) is a new Infrastructure-as-Code
12	framework built from the ground up in TypeScript and Effect.
13	Everything is typed code, from a resource's inputs and outputs
14	down to the SDK calls and errors inside its lifecycle logic.
15
16	Built from the ground up also means starting from zero
17	coverage. Alchemy can't reuse the providers of existing IaC
18	ecosystems like Terraform or Pulumi, and we're a team of
19	three, so hand-writing lifecycle code for thousands of cloud
20	resources isn't an option. We generate it with AI, using the
21	loop in the diagram above.
22
23	Fleets of AI agents write the resources and their live tests,
24	then run them against a real Cloudflare account. Every API
25	behavior the SDK's types don't capture becomes a patch to the
26	SDK, which regenerates and the tests rerun. One run of this
27	loop grew Alchemy's Cloudflare provider from **22 resources to
28	230** between June 11 and June 17, producing 288 test files
29	with roughly 700 tests and **720 SDK patches**. Most of it
30	landed in a single PR
31	([#601](https://github.com/alchemy-run/alchemy/pull/601))
32	that added 101,178 lines across 542 files and was open for 22
33	hours. The PR description, in its entirety:
34
35	> Using Fable to generate all missing cloudflare resources
36
37	Deterministic codegen has been
38	producing SDKs and IaC providers from API specs for years, and
39	AI makes it even cheaper. The problem is that code generated
40	from a spec is only as truthful as the spec, and real APIs
41	drift from their specs. They return undocumented error codes,
42	misuse HTTP statuses, and null out fields the schema says are
43	required. That's why the interesting number isn't the 230
44	resources. It's the 720 patches, each one a place where a live
45	test caught the API doing something its spec doesn't admit to.
46
47	## A software development flywheel
48
49	Alchemy's resources are built on
50	[distilled](https://github.com/alchemy-run/distilled), our
51	Effect-native SDK generated from Cloudflare's official
52	TypeScript SDK. It covers 114 services and 2,092 operations,
53	and each operation can carry a patch, a small JSON file that
54	corrects the spec where reality disagrees with it.
55
56	The run operated on one rule: **agents never handle an unknown
57	error in their own code.** An unmatched error is never caught
58	in the resource. It becomes an SDK patch and a regeneration.
59	Untyped failure handling is where AI-generated code usually
60	goes wrong, matching message substrings or guessing status
61	codes. The rule takes that option away because the code
62	doesn't typecheck until the SDK knows about the error. It also
63	makes the work compound. Every red test either fixes a
64	resource or improves the SDK for the next one.
65
66	## One turn of the flywheel
67
68	Infrastructure-as-code is mostly disciplined error handling.
69	Deletes tolerate already-gone. Creates absorb losing a race.
70	Reads ride out eventual consistency. In Alchemy that
71	discipline is typed:
72
73	```typescript
74	// idempotent delete: reading back an already-gone widget is success
75	turnstile.getWidget({ accountId, sitekey }).pipe(
76	Effect.catchTag("WidgetNotFound", () => Effect.succeed(undefined)),
77	);
78
79	// R2's endpoint lags a fresh bucket create, so retry the not-found until it settles
80	r2.listBucketDomainCustoms({ accountId, bucketName }).pipe(
81	Effect.retry({
82	while: (e) => e._tag === "NoSuchBucket",
83	schedule: Schedule.exponential("100 millis").pipe(
84	Schedule.both(Schedule.recurs(5)),
85	),
86	}),
87	);
88	```
89
90	The agents write this code first, then run it against the real
91	API. A Turnstile test deletes a widget and reads it back to
92	prove it's gone. The read fails with code `10404`, which the
93	spec doesn't declare. The `catchTag` doesn't typecheck because
94	`WidgetNotFound` isn't in the error union yet, and the only
95	permitted fix is a patch in distilled:
96
97	```json
98	// submodules/distilled/packages/cloudflare/patches/turnstile/getWidget.json
99	{
100	"errors": {
101	"WidgetNotFound": [{ "code": 10404 }, { "code": 10407 }],
102	"Forbidden": [{ "status": 403 }]
103	}
104	}
105	```
106
107	Regenerate the service (`bun scripts/generate.ts --service
108	turnstile`) and the union names what the API actually does:
109
110	```typescript
111	export type GetWidgetError = DefaultErrors | WidgetNotFound | Forbidden;
112	```
113
114	The `catchTag` compiles and the test goes green. The loop
115	continues through reconcile, read, diff, delete, and list
116	until the whole resource passes against the live API. The
117	resource is the product and the truthful SDK is the byproduct.
118	The patch lives in the SDK, not Alchemy, so anyone who uses
119	distilled's Turnstile module from now on gets `WidgetNotFound`
120	as a typed, catchable error.
121
122	## What 1,087 patches look like
123
124	By beta.56 the corpus stood at 1,087 patched operations,
125	covering 52% of the SDK across 94 of 114 services:
126
127	| | count |
128	| --- | --- |
129	| patches declaring typed errors | 1,070 |
130	| patches fixing a response schema | 173 |
131	| patches fixing a request schema | 26 |
132	| distinct error tags introduced | 425 |
133	| matchers on Cloudflare error `code` | 1,367 |
134	| matchers on HTTP `status` | 994 |
135	| matchers needing `message.includes` | 377 |
136	| matchers needing a regex | 0 |
137
138	Most tags are the vocabulary lifecycle code needs, like
139	`Forbidden`, `WorkerNotFound`, and `NoSuchBucket`. The rest
140	catalog how APIs drift from their docs. Snippets returns 400
141	instead of 404 for a missing snippet, so the matcher reads the
142	message. Queues consumers support `http_pull` but the spec's
143	enum doesn't include it. Account settings come back null where
144	the schema says they can't. None of this is discoverable by
145	reading documentation.
146
147	## The factory
148
149	Around twelve agents ran concurrently, each owning one
150	distilled service. Only the owner of `turnstile` writes to
151	`patches/turnstile/`, so agents never race the generator.
152	Every test follows one shape: deploy, verify out-of-band by
153	querying the API directly through distilled, mutate, verify
154	again, destroy, and prove the destroy by watching the resource
155	disappear. A test never trusts the deploy's own report of
156	success. It confirms every step against what the API actually
157	returns.
158
159	Two choices kept the loop fast. distilled is a git submodule
160	inside alchemy's bun workspace, so an agent patches the SDK
161	and the resource that consumes it in one working tree, and a
162	wave lands as two PRs, one to alchemy and one to
163	distilled. And vitest resolves distilled from its TypeScript
164	source, so a regenerated service is visible to the very next
165	test run. The loop's cycle time is the test's runtime.
166
167	## What's next
168
169	We're scaling this loop to every cloud provider, with two
170	goals: 100% IaC coverage in Alchemy, and refined SDKs whose
171	types align with each API's real behavior rather than just its
172	spec. AWS is next, then Railway, GCP, and Azure. The
173	Cloudflare run was the proof, and it shipped in
174	[2.0.0-beta.56](/blog/2026-06-17-beta-56).
175
176	- [2.0.0-beta.56 — the release these resources shipped in](/blog/2026-06-17-beta-56)
177	- [PR #601 — the generation wave](https://github.com/alchemy-run/alchemy/pull/601)
178	- [distilled — the generated, patchable SDK](https://github.com/alchemy-run/distilled)
179	- [Cloudflare provider reference](/providers/cloudflare)