| 1 | --- |
| 2 | title: React to S3 Events |
| 3 | description: Subscribe a Lambda Function to S3 bucket notifications, process them as an Effect Stream, and let Alchemy wire up the IAM and event-source plumbing. |
| 4 | sidebar: |
| 5 | order: 3 |
| 6 | --- |
| 7 | |
| 8 | Reading and writing S3 objects on demand is one half of the story. |
| 9 | The other half is **reacting** to objects as they arrive — for |
| 10 | indexing, virus scanning, thumbnail generation, you name it. In |
| 11 | this part you'll subscribe the same Lambda from the previous part |
| 12 | to bucket notifications and process them as an Effect `Stream`. |
| 13 | |
| 14 | ## Subscribe to bucket notifications |
| 15 | |
| 16 | Bucket notifications live behind an Alchemy helper: |
| 17 | `S3.consumeBucketEvents(bucket, process)`. Add the smallest |
| 18 | possible subscription — log every new object's key — alongside |
| 19 | the existing bindings: |
| 20 | |
| 21 | ```diff lang="typescript" |
| 22 | // src/api.ts |
| 23 | import * as AWS from "alchemy/AWS"; |
| 24 | import * as S3 from "alchemy/AWS/S3"; |
| 25 | +import * as Console from "effect/Console"; |
| 26 | import * as Effect from "effect/Effect"; |
| 27 | import * as Layer from "effect/Layer"; |
| 28 | +import * as Stream from "effect/Stream"; |
| 29 | // ... |
| 30 | |
| 31 | Effect.gen(function* () { |
| 32 | const bucket = yield* S3.Bucket("Blobs"); |
| 33 | const putObject = yield* S3.PutObject(bucket); |
| 34 | const getObject = yield* S3.GetObject(bucket); |
| 35 | |
| 36 | + yield* S3.consumeBucketEvents(bucket, (stream) => |
| 37 | + stream.pipe( |
| 38 | + Stream.runForEach((event) => |
| 39 | + Console.log(`new object: ${event.key}`), |
| 40 | + ), |
| 41 | + ), |
| 42 | + ); |
| 43 | |
| 44 | return { |
| 45 | fetch: /* ... unchanged ... */, |
| 46 | }; |
| 47 | }); |
| 48 | ``` |
| 49 | |
| 50 | That single `yield*` does three things in one shot: it |
| 51 | configures the bucket's notification settings to push events |
| 52 | into the function, adds the `lambda:InvokeFunction` permission |
| 53 | S3 needs to call you, and registers the stream handler that |
| 54 | runs every time an event batch lands. |
| 55 | |
| 56 | ## Filter the event types |
| 57 | |
| 58 | Without a filter you get every notification S3 emits — object |
| 59 | created, deleted, restored, replicated, the lot. Constrain the |
| 60 | subscription to a specific subset with `events`: |
| 61 | |
| 62 | ```diff lang="typescript" |
| 63 | -yield* S3.consumeBucketEvents(bucket, (stream) => |
| 64 | +yield* S3.consumeBucketEvents( |
| 65 | + bucket, |
| 66 | + { events: ["s3:ObjectCreated:*"] }, |
| 67 | + (stream) => |
| 68 | stream.pipe( |
| 69 | Stream.runForEach((event) => |
| 70 | Console.log(`new object: ${event.key}`), |
| 71 | ), |
| 72 | ), |
| 73 | ); |
| 74 | ``` |
| 75 | |
| 76 | The `events` filter is type-checked against the AWS S3 event |
| 77 | catalog, so typos surface at the call site rather than at deploy |
| 78 | time. |
| 79 | |
| 80 | ## What the stream actually carries |
| 81 | |
| 82 | `stream` is a `Stream.Stream<BucketNotification>` — a normalized |
| 83 | shape with `type`, `bucket`, `key`, `size`, and `eTag`. Each |
| 84 | Lambda invocation delivers one batch as one stream emission |
| 85 | sequence, then completes; the next invocation starts a fresh |
| 86 | stream. Treating events as a `Stream` means the usual operators |
| 87 | (`mapEffect`, `groupedWithin`, `mapAccum`) work without thinking |
| 88 | about the Lambda payload envelope. |
| 89 | |
| 90 | ## Provide the runtime layer |
| 91 | |
| 92 | The runtime side of the subscription lives in |
| 93 | `Lambda.BucketEventSource`. It's the layer that turns the abstract |
| 94 | `BucketEventSource` capability into a concrete Lambda event-source |
| 95 | mapping plus IAM grant. Add it alongside the S3 bindings: |
| 96 | |
| 97 | ```diff lang="typescript" |
| 98 | }).pipe( |
| 99 | Effect.provide( |
| 100 | - Layer.mergeAll(S3.PutObjectHttp, S3.GetObjectHttp), |
| 101 | + Layer.mergeAll( |
| 102 | + AWS.Lambda.BucketEventSource, |
| 103 | + S3.PutObjectHttp, |
| 104 | + S3.GetObjectHttp, |
| 105 | + ), |
| 106 | ), |
| 107 | ), |
| 108 | ``` |
| 109 | |
| 110 | `BucketEventSource` and the `PutObject`/`GetObject` lives are all |
| 111 | peers here — none of them require each other — so a flat |
| 112 | `Layer.mergeAll(...)` is enough. |
| 113 | |
| 114 | ## Run an Effect per event |
| 115 | |
| 116 | Logging is a good smoke test, but a real handler usually wants |
| 117 | to do something async per event. Swap `runForEach` for |
| 118 | `mapEffect` so each event runs through a generator block: |
| 119 | |
| 120 | ```diff lang="typescript" |
| 121 | yield* S3.consumeBucketEvents( |
| 122 | bucket, |
| 123 | { events: ["s3:ObjectCreated:*"] }, |
| 124 | (stream) => |
| 125 | stream.pipe( |
| 126 | - Stream.runForEach((event) => |
| 127 | - Console.log(`new object: ${event.key}`), |
| 128 | - ), |
| 129 | + Stream.mapEffect((event) => |
| 130 | + Console.log(`new object: ${event.key}`), |
| 131 | + ), |
| 132 | + Stream.runDrain, |
| 133 | + ), |
| 134 | ); |
| 135 | ``` |
| 136 | |
| 137 | `Stream.runDrain` consumes the stream without collecting — we |
| 138 | only care about the side effects, not the return values. |
| 139 | |
| 140 | ## Read the object back |
| 141 | |
| 142 | `getObject` is already bound to the bucket, so the per-event |
| 143 | Effect can fetch the new object's contents inline: |
| 144 | |
| 145 | ```diff lang="typescript" |
| 146 | stream.pipe( |
| 147 | Stream.mapEffect((event) => |
| 148 | - Console.log(`new object: ${event.key}`), |
| 149 | + Effect.gen(function* () { |
| 150 | + const result = yield* getObject({ Key: event.key }); |
| 151 | + const body = yield* result.Body!.pipe( |
| 152 | + Stream.decodeText, |
| 153 | + Stream.mkString, |
| 154 | + ); |
| 155 | + yield* Console.log(`${event.key} -> ${body}`); |
| 156 | + }).pipe(Effect.orDie), |
| 157 | ), |
| 158 | Stream.runDrain, |
| 159 | ), |
| 160 | ``` |
| 161 | |
| 162 | `Stream.mapEffect` is back-pressured by default — events flow |
| 163 | through one at a time. Pass a `concurrency` option (we'll see it |
| 164 | in later parts) when you want fan-out. |
| 165 | |
| 166 | ## Deploy and verify |
| 167 | |
| 168 | ```sh |
| 169 | bun alchemy deploy |
| 170 | ``` |
| 171 | |
| 172 | The CLI applies the new bucket notification configuration and |
| 173 | redeploys the function with the broader IAM policy. Then trigger |
| 174 | the subscription end-to-end by `PUT`ing an object via the HTTP |
| 175 | route from the previous part: |
| 176 | |
| 177 | ```typescript |
| 178 | // test/integ.test.ts |
| 179 | test( |
| 180 | "PUT triggers notification handler", |
| 181 | Effect.gen(function* () { |
| 182 | const { url } = yield* stack; |
| 183 | yield* HttpClient.put(`${url}/notify-me.txt`, { |
| 184 | body: HttpBody.text("payload"), |
| 185 | }); |
| 186 | // ...optionally tail CloudWatch logs to assert the message landed |
| 187 | }), |
| 188 | ); |
| 189 | ``` |
| 190 | |
| 191 | You'll see the `notify-me.txt -> payload` line in the function's |
| 192 | log group within a couple of seconds. |
| 193 | |
| 194 | S3 events typically arrive in batches of up to 10 records — the |
| 195 | same `Stream` is fed each batch and completes between invocations, |
| 196 | so you get the same composition story whether the batch is one |
| 197 | event or a hundred. |
| 198 | |
| 199 | Next we'll trade blob storage for structured records and |
| 200 | [add a DynamoDB table](/aws/data/dynamodb). |