1---
2title: Add an AI Gateway
3description: Wire an AI Gateway into your Worker, turn it into a typed Effect LanguageModel, and run generations and streams through Workers AI with caching, rate limiting, and logs.
4sidebar:
5 order: 6
6---
7
8You've now wired Durable Objects, hibernatable WebSockets, a
9container, and a Workflow into your Worker. The last piece in the
10Cloudflare track is an **AI Gateway** — a stable account-scoped
11endpoint that fronts every model provider (Workers AI, OpenAI,
12Anthropic, Bedrock, …) and gives you caching, rate limiting, retries,
13DLP, and a single dashboard of every request, token, and cost.
14
15In Alchemy the gateway is a single resource. Once you've declared and
16bound it, `.model({...})` returns an Effect `LanguageModel.LanguageModel`
17Layer — and from there you're using the same `generateText` /
18`streamText` APIs you'd use against any other provider.
19
20Don't need caching, rate limiting, or the request log? The plain
21[Workers AI binding](/cloudflare/ai/workers-ai)
22(`Cloudflare.Workers.AI`) calls the same models directly, with no
23gateway resource — and its `ai.model({...})` produces the same
24`LanguageModel` Layer, so you can start there and add the gateway
25later without touching handler code.
26
27## Declare the gateway
28
29Create `src/AiGateway.ts` with a single resource definition. The
30two flags below enable response caching (60-second TTL) and request
31logging — every prompt, completion, latency, and token count will
32show up in the AI Gateway dashboard.
33
34```typescript
35// src/AiGateway.ts
36import * as Cloudflare from "alchemy/Cloudflare";
37
38export const Gateway = Cloudflare.AI.Gateway("Gateway", {
39 cacheTtl: 60,
40 collectLogs: true,
41});
42```
43
44Every prop is optional, but explicit defaults make the intent
45visible. We'll tune more knobs at the end of the tutorial.
46
47## Add it to the stack
48
49```diff lang="typescript"
50// alchemy.run.ts
51+import { Gateway } from "./src/AiGateway.ts";
52import Api from "./src/Api.ts";
53
54export default Alchemy.Stack(
55 "CloudflareWorkerExample",
56 { providers: Cloudflare.providers(), state: Cloudflare.state() },
57 Effect.gen(function* () {
58 const api = yield* Api;
59+ const gateway = yield* Gateway;
60
61 return {
62 url: api.url.as<string>(),
63+ gatewayId: gateway.gatewayId,
64 };
65 }),
66);
67```
68
69`yield* Gateway` registers the resource so it gets created/updated on
70the next deploy. `gateway.gatewayId` is exposed as a stack output so
71you can find it in the dashboard.
72
73## Bind the gateway into the Worker
74
75`Cloudflare.AI.QueryGateway(Gateway)` returns a typed client whose
76methods are wrapped in Effect — `run` for raw inference, `getUrl` for
77the gateway endpoint, `getLog`/`patchLog` for the request log, and
78`model({...})` for building a `LanguageModel` layer.
79
80```diff lang="typescript"
81// src/Api.ts
82import * as Cloudflare from "alchemy/Cloudflare";
83import * as Effect from "effect/Effect";
84+import { Gateway } from "./AiGateway.ts";
85
86export default class Api extends Cloudflare.Worker<Api>()(
87 "Api",
88 { main: import.meta.url },
89 Effect.gen(function* () {
90+ const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
91
92 return {
93 fetch: Effect.gen(function* () {
94 // …existing routes
95 }),
96 };
97- }),
98+ }).pipe(Effect.provide(Cloudflare.AI.QueryGatewayBinding)),
99) {}
100```
101
102`Cloudflare.AI.QueryGatewayBinding` is the runtime side of the
103binding. Provide it once at the bottom of the constructor's layer chain and
104every `QueryGateway(...)` further up will resolve.
105
106## Build a `LanguageModel` layer
107
108Call `aiGateway.model({...})` with a Workers AI model id and
109parameters. The result is a `Layer<LanguageModel.LanguageModel, …>`
110that satisfies Effect's standard AI service.
111
112```diff lang="typescript"
113 Effect.gen(function* () {
114 const aiGateway = yield* Cloudflare.AI.QueryGateway(Gateway);
115+
116+ const languageModel = aiGateway.model({
117+ model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
118+ parameters: { temperature: 0.7, maxTokens: 1024 },
119+ });
120
121 return {
122 fetch: Effect.gen(function* () {
123 // …
124 }),
125 };
126 })
127```
128
129`parameters` is the per-call default; any individual call can still
130override it. The Construction phase is the right place to build this layer —
131construction is pure and the binding factory only exists here.
132
133## Generate text on `/generate`
134
135`LanguageModel.generateText` returns a typed response with `text`,
136`finishReason`, structured token `usage`, and any `toolCalls`. Provide
137the `languageModel` layer to the handler and call it like any other
138Effect.
139
140```diff lang="typescript"
141+import { LanguageModel } from "effect/ai";
142 import { HttpServerRequest } from "effect/http/HttpServerRequest";
143 import * as HttpServerResponse from "effect/http/HttpServerResponse";
144
145 return {
146 fetch: Effect.gen(function* () {
147 const request = yield* HttpServerRequest;
148+ const url = new URL(request.url, "http://api");
149
150+ if (url.pathname === "/generate" && request.method === "POST") {
151+ const body = (yield* request.json) as { prompt?: string };
152+ const prompt = body.prompt?.trim() ?? "Say hello in one sentence.";
153+
154+ const response = yield* LanguageModel.generateText({ prompt }).pipe(
155+ Effect.orDie,
156+ );
157+
158+ return yield* HttpServerResponse.json({
159+ text: response.text,
160+ usage: {
161+ inputTokens: response.usage.inputTokens.total,
162+ outputTokens: response.usage.outputTokens.total,
163+ },
164+ });
165+ }
166
167 return HttpServerResponse.text("Not Found", { status: 404 });
168- }),
169+ }).pipe(Effect.provide(languageModel)),
170 };
171```
172
173`Effect.provide(languageModel)` makes `LanguageModel.LanguageModel`
174available to every handler in `fetch`. `Effect.orDie` collapses
175`AiError` to a defect so a model failure surfaces as a 500 — if you
176need typed handling instead, `Effect.catchTag("AiError", …)` works.
177
178## Try it
179
180```sh
181bun alchemy deploy
182curl -X POST "$(bun alchemy stack output url)/generate" \
183 -H "content-type: application/json" \
184 -d '{"prompt":"Write a haiku about Effect"}'
185```
186
187The first call takes ~1–2 seconds. Send the **exact same** prompt
188again and it returns in milliseconds — that's the `cacheTtl: 60`
189config doing its job. Open the Cloudflare dashboard → **AI** →
190**AI Gateway** → your gateway and you'll see both requests, with the
191second flagged as a cache hit, plus latency and token usage on every
192entry.
193
194## Stream tokens on `/stream`
195
196For chat-style UIs you want tokens to arrive as the model produces
197them, not in one big response. `LanguageModel.streamText` returns an
198`Effect.Stream` of typed response parts — `text-delta`, `tool-call`,
199`finish`, and so on. Pipe it through `Sse.encode` and
200`HttpServerResponse.stream` to get a server-sent-event stream that
201flushes through the Worker → edge → client without buffering.
202
203```diff lang="typescript"
204+import * as Stream from "effect/Stream";
205+import * as Sse from "effect/encoding/Sse";
206
207+ if (url.pathname === "/stream" && request.method === "POST") {
208+ const body = (yield* request.json) as { prompt?: string };
209+ const prompt = body.prompt?.trim() ?? "Tell me a haiku about Effect.";
210+
211+ const stream = LanguageModel.streamText({ prompt }).pipe(
212+ Stream.provide(languageModel),
213+ Sse.encode,
214+ );
215+
216+ return HttpServerResponse.stream(stream, {
217+ headers: {
218+ "content-type": "text/event-stream",
219+ "cache-control": "no-cache",
220+ "x-accel-buffering": "no",
221+ },
222+ });
223+ }
224```
225
226`Stream.provide(languageModel)` is the stream-aware equivalent of
227`Effect.provide` — the `LanguageModel` needs to be available for the
228entire lifetime of the stream, not just the initial setup.
229
230```sh
231curl -N -X POST "$(bun alchemy stack output url)/stream" \
232 -H "content-type: application/json" \
233 -d '{"prompt":"Write a haiku about Effect, slowly."}'
234```
235
236`-N` disables curl's response buffering so you see each SSE event as
237soon as the Worker flushes it.
238
239## Tune caching, rate limits, and DLP
240
241Every prop on `Cloudflare.AI.Gateway` maps to an update API call — no
242replacement, no downtime. A production-grade config might look like:
243
244```typescript
245export const Gateway = Cloudflare.AI.Gateway("Gateway", {
246 id: "prod-gateway",
247 cacheTtl: 300,
248 cacheInvalidateOnUpdate: true,
249 rateLimitingInterval: 60,
250 rateLimitingLimit: 100,
251 rateLimitingTechnique: "sliding",
252 collectLogs: true,
253 logManagement: 100_000,
254 logManagementStrategy: "DELETE_OLDEST",
255 authentication: true,
256});
257```
258
259Bumping `cacheTtl`, `rateLimitingLimit`, or toggling `authentication`
260is a single `bun alchemy deploy` away — the diff updates the gateway
261in place.
262
263## What you have now
264
265- A `Cloudflare.AI.Gateway` resource with caching, logging, and (when
266 you want them) rate limiting and auth.
267- An Effect `LanguageModel.LanguageModel` Layer that proxies Workers
268 AI through that gateway.
269- `/generate` and `/stream` routes that use the standard
270 `LanguageModel.generateText` and `streamText` APIs — provider-
271 agnostic, so swapping Workers AI for OpenAI or Anthropic later is
272 a layer-level change, not a code-level one.
273
274The natural next step is to give the model **memory** — wrapping the
275same `LanguageModel` with `Chat.Service` and storing the conversation
276in a Durable Object's `state.storage`, so every session is resumable
277across requests, restarts, and hibernation.
278
279For the wider API surface — `generateObject`, `Toolkit`, structured
280outputs, and how the same `LanguageModel` composes inside an HTTP
281API or RPC handler — see the [Effect AI guide](/cloudflare/ai/effect-ai).