| 1 | --- |
| 2 | title: A Test Runner for Agents |
| 3 | date: 2026-07-22T07:00:00Z |
| 4 | excerpt: Alchemy's test suite is 4,000+ live tests against real clouds, and some take minutes. Vitest forked processes and reloaded our generated SDK on every fork; bun test assumes tests are fast and runs them one file at a time. So we wrote our own runner — one bun process, everything concurrent, plain output tuned for agents and a live TUI for humans. |
| 5 | --- |
| 6 | |
| 7 | Alchemy's test suite is not a normal test suite. It's **4,000+ |
| 8 | live tests** that deploy real resources to real clouds and tear |
| 9 | them down again. Some finish in milliseconds. Some take minutes — |
| 10 | provisioning an EKS cluster, waiting for an eventually consistent |
| 11 | API to converge, pushing a message through a queue and polling |
| 12 | until it comes out the other side. |
| 13 | |
| 14 | That shape broke every runner we tried. So we built our own: |
| 15 | [`alchemy-test`](https://github.com/alchemy-run/alchemy/tree/main/packages/alchemy-test) |
| 16 | ([#848](https://github.com/alchemy-run/alchemy/pull/848)), a |
| 17 | vitest-compatible runner built for our scale: the entire suite |
| 18 | runs in a single bun process, plain output is designed for the |
| 19 | agent reading it, and an interactive TUI gives humans a live view |
| 20 | of a run with thousands of tests in flight. |
| 21 | |
| 22 | ## Why not vitest |
| 23 | |
| 24 | Vitest parallelizes across workers, and every worker builds its |
| 25 | own module graph — threads don't share modules any more than |
| 26 | forks do. Everything alchemy depends on includes |
| 27 | [distilled](https://github.com/alchemy-run/distilled), our |
| 28 | generated AWS/Cloudflare SDK, which spans hundreds of service |
| 29 | modules. Paying that import cost once is fine. Paying it per |
| 30 | worker, hundreds of times per run, is not. |
| 31 | |
| 32 | There's no way out of that trade-off inside vitest. It can run |
| 33 | everything in one worker (`singleFork` with `isolate: false`) — |
| 34 | but [by vitest's own |
| 35 | docs](https://vitest.dev/config/#pooloptions-forks-singlefork) |
| 36 | that "will force tests to run one after another", and source code |
| 37 | processed by vite "will still be reevaluated for each test" even |
| 38 | then. Parallel files mean many workers; one worker means a |
| 39 | sequential suite. The quadrant we needed — many files in flight, |
| 40 | one module graph — doesn't exist. |
| 41 | |
| 42 | For us it was worse than that. Our toolchain runs on bun, and |
| 43 | bun's `worker_threads` segfaults under vitest's tinypool at |
| 44 | worker spawn — so we were pinned to `pool: "forks"`, full |
| 45 | `child_process` forks, the most expensive flavor of an already |
| 46 | expensive design. And no, "just run tests on node" doesn't fix |
| 47 | it: the per-worker import cost is the same, and node can't |
| 48 | execute distilled's TypeScript natively, so a transform layer |
| 49 | comes back. |
| 50 | |
| 51 | The module cost also made iteration slow in a way that hurt |
| 52 | agents most. To load distilled's TypeScript source in vitest we |
| 53 | needed a rolldown plugin; the alternative was compiling |
| 54 | distilled's `.ts` to `.js` before every run. Either way there was |
| 55 | a build step between "agent edits code" and "agent sees the test |
| 56 | result". Agents iterate on failing tests in tight loops — every |
| 57 | second of compile tax is paid dozens of times per task. |
| 58 | |
| 59 | ## Why not bun test |
| 60 | |
| 61 | `bun test` solves both problems: it's bun-native, so `.ts` loads |
| 62 | directly with no plugin and no compile step. But it runs test |
| 63 | files one at a time, an assumption that only works when tests are |
| 64 | fast. |
| 65 | |
| 66 | Alchemy tests are slow on purpose — they wait for real clouds. Run |
| 67 | them sequentially and a directory that should take two minutes |
| 68 | takes an hour. We needed dozens of files in flight at once, in one |
| 69 | process, on bun. |
| 70 | |
| 71 | ## One process, everything concurrent |
| 72 | |
| 73 | `alchemy-test` imports every test file up front (imports are cheap |
| 74 | when they happen once) and then runs files and tests concurrently |
| 75 | on the Effect runtime — fibers, not forks. A `--concurrency` flag |
| 76 | bounds how many files run at once, and tests that mutate global |
| 77 | state can take a whole-process lock with `{ exclusive: true }`. |
| 78 | |
| 79 | The CLI is vitest-compatible — positional paths, `-t` filters, |
| 80 | timeouts, retries — so the migration was a codemod. |
| 81 | |
| 82 | ```sh |
| 83 | bun run test test/Cloudflare/Workers -t "cron" |
| 84 | ``` |
| 85 | |
| 86 | ## Output an agent can act on |
| 87 | |
| 88 | Performance forced our hand, but once we owned the runner we |
| 89 | could fix something that had bothered us just as long: the |
| 90 | output. |
| 91 | |
| 92 | Vitest's default reporter prints stdout noisily: interleaved |
| 93 | across parallel tests, indented and reflowed on top of your raw |
| 94 | output. When a test fails, the error is somewhere in the scroll. |
| 95 | We watched agents run a suite, `tail` the output to save context, |
| 96 | and lose the one error they needed — even when prompted not to. |
| 97 | A custom reporter can quiet some of this, but it works by |
| 98 | intercepting `console` and guessing which test a log belongs to — |
| 99 | best-effort attribution you're consuming, not controlling. |
| 100 | |
| 101 | Our tests are Effects, which means we control the runtime they |
| 102 | execute in. Each test runs under a buffering `Logger` and |
| 103 | `Console`, so every log line is captured and attributed to its |
| 104 | test by construction — nothing interleaves, nothing prints as it |
| 105 | happens. The console gets exactly one line per test: |
| 106 | |
| 107 | ``` |
| 108 | ✓ test/Cloudflare/Workers/Subdomain.test.ts > enable and disable the workers.dev subdomain (3.2s) |
| 109 | ✓ test/Cloudflare/Workers/Rpc.test.ts > detects valid envelope (1ms) |
| 110 | ✗ test/Cloudflare/Workers/Route.test.ts > replaces the route when the zone changes (12.4s) |
| 111 | AssertionError: expected "alchemy-test-2.us/api/*" to be "alchemy-test-2.us/v2/*" |
| 112 | at test/Cloudflare/Workers/Route.test.ts:87:22 |
| 113 | --- captured output --- |
| 114 | Plan: 1 to replace |
| 115 | [route] deleting |
| 116 | [route] created |
| 117 | --- end output --- |
| 118 | ✓ test/Cloudflare/Workers/Workflow.test.ts > runs a workflow to completion (51.9s) |
| 119 | ``` |
| 120 | |
| 121 | A failing test prints its error and its captured output inline, |
| 122 | immediately, while the run continues. Passing tests stay silent — |
| 123 | their output would be noise. |
| 124 | |
| 125 | If nothing finishes for ten seconds, the runner prints what's |
| 126 | still running, so a hang is diagnosable from the console instead |
| 127 | of a mystery: |
| 128 | |
| 129 | ``` |
| 130 | ⧗ [16:38:12] no tests finished in the last 10s — 2 still running: |
| 131 | test/Cloudflare/Workers/Workflow.test.ts > runs a workflow to completion (48.1s) |
| 132 | test/Cloudflare/Workers/CronEventSource.test.ts > cron handler fires (32.7s) |
| 133 | ``` |
| 134 | |
| 135 | And the run ends by telling the agent exactly how to go deeper: |
| 136 | the summary, the failures repeated in one place, and a per-run |
| 137 | log file — with its size, so the agent knows whether to read it |
| 138 | or search it: |
| 139 | |
| 140 | ``` |
| 141 | Tests: 1 failed | 171 passed | 1 skipped (31 files, 89.3s) |
| 142 | |
| 143 | Full log: .alchemy/log/test/2026-07-16T23-42-38-pid92186.log (1055 lines, 118 KB) |
| 144 | ``` |
| 145 | |
| 146 | The log file has everything the console suppressed: every test's |
| 147 | captured output, passes included, written color-free as the run |
| 148 | progresses. Each run gets its own timestamped file, so concurrent |
| 149 | runs in different terminals never trample each other. The console |
| 150 | is the summary; the log is the record. The agent reads the one |
| 151 | line it needs and greps the file when it needs more. |
| 152 | |
| 153 | ## Humans get a TUI |
| 154 | |
| 155 | For interactive use, `--tui` opens a k9s-style live view: a |
| 156 | collapsible file/test tree, type-to-filter, and retry/kill on |
| 157 | running tests: |
| 158 | |
| 159 | ``` |
| 160 | alchemy-test ✓ 159 ✗ 0 ◐ 7 · 5 │ 53.4s |
| 161 | ▸ ✓ test/Cloudflare/Workers/AccountSetting.test.ts 3/3 passed |
| 162 | ▾ ◐ test/Cloudflare/Workers/DurableObjectNamespace.test.ts 2 running · 11/13 passed |
| 163 | ✓ getByName round-trips state (1.9s) |
| 164 | ◐ websocket hibernation survives eviction (12.3s…) |
| 165 | ▸ ◐ test/Cloudflare/Workers/HttpApi.test.ts 5/5 passed — afterAll running (18.6s)… |
| 166 | ▸ · test/Cloudflare/Workers/WaitUntil.test.ts 0/2 passed — waiting for a worker slot… |
| 167 | ▸ ✓ test/Cloudflare/Workers/Worker.test.ts 17/17 passed |
| 168 | ``` |
| 169 | |
| 170 | Pressing `enter` on a running test live-tails its captured |
| 171 | output — useful when a deploy is three minutes into `beforeAll` |
| 172 | and you want to know what it's doing. |
| 173 | |
| 174 | The hotkeys are vim-flavored: `j`/`k` and `ctrl-d`/`ctrl-u` to |
| 175 | move, `/` for live type-to-filter, `p`/`f`/`n`/`s` to toggle |
| 176 | passing, failing, pending, and skipped tests in and out of view. |
| 177 | `r` retries the selected test or file, `R` retries everything |
| 178 | that failed, `x` kills a running test, and `y` copies a failure's |
| 179 | error and captured output to the clipboard — handy for pasting |
| 180 | straight into an agent chat. |
| 181 | |
| 182 | ## Owning the runner |
| 183 | |
| 184 | The whole thing was easy to build — a test harness, a scheduler, |
| 185 | and two reporters over an event stream, in a codebase that |
| 186 | already runs everything on Effect. That's the part worth |
| 187 | underlining: it was cheap to make *because* we only target |
| 188 | Effect, and owning it means the feedback loop is ours to tune. |
| 189 | When we notice agents misreading output, we change the output. |
| 190 | No plugin API in between. |
| 191 | |
| 192 | ## Where to go next |
| 193 | |
| 194 | - [alchemy-test on GitHub](https://github.com/alchemy-run/alchemy/tree/main/packages/alchemy-test) |
| 195 | - [The PR](https://github.com/alchemy-run/alchemy/pull/848) |
| 196 | - [Looping the Generation of IaC and SDKs](/blog/2026-07-02-cloudflare-resource-factory) — the agent fleet this runner feeds |