clover's git

the-knife

1	# git-service — Final Design Document
2
3	**A Git hosting service on Cloudflare Workers + Durable Objects + R2, built as `packages/git` with Alchemy Effect-native Workers.**
4
5	Stance: **protocol-first minimalist core (Proposal A's skeleton) with Proposal B's product ergonomics and scale seams grafted on.** v1 is the smallest system that real `git` clients (git ≥ 1.6.6 through current) can clone, fetch, and push against, with transactional refs and a typed REST management plane. Every cut has a named upgrade seam. The architecture is the ripgit/chr33s-validated shape (Worker router → one DO per repo → SQLite refs/objects + R2 overflow), re-expressed in Effect with real transactions (`transactionSync`), typed errors, streaming responses, and auth from day one.
6
7	---
8
9	## 0. Resolved contradictions (A vs B)
10
11	Every place the two proposals disagreed, with the decision and the one-line reason:
12
13	| Topic | A said | B said | **Decision** | Why (one line) |
14	|---|---|---|---|---|
15	| Object bytes at rest | zlib rows in DO SQLite, >1 MiB → R2 | Pack-first: pushed packs verbatim to R2, DO holds index only | **A** (rows + R2 overflow; packs are v1.1) | Pack emission from pre-deflated rows is pure concatenation (zero CPU), ripgit-proven in 128 MB; B's two-pass R2-fixup ingest is the single riskiest subsystem and buys nothing until repos outgrow the 50 MiB push cap — the `location`/`pack_id` columns keep B's endgame reachable without a migration. |
16	| Where the protocol runs | Inside the Repo DO | Worker data plane, DO control plane | **A** (in the DO) | The bytes live in the DO's SQLite so they must transit it anyway; duration billing is ~$0.0001 per clone-minute (0.125 GB × 60 s × $12.50/M GB-s), and B's Worker-side streaming becomes the natural v1.1 fast path only once compacted packs live in R2. |
17	| Fetch protocol | v0 only | v2 primary + v0 fallback | **A** (v0 only) | Every client ≥1.6.6 falls back to v0 silently; v2 is a pure-additive choreography file over the same closure/pack code, so shipping it in v1 doubles conformance surface for zero client reach. |
18	| Push ingestion | Buffer ≤ 50 MiB, `RandomAccess` seam | Stream to R2 multipart + 2-pass fixups | **A** (buffer + cap) | Bounded, simple, ripgit-proven; the parser is written against a `RandomAccess` source so the R2-tee upgrade is one implementation swap, not a protocol change. |
19	| Served deltas / thin packs | Never (fat packs) | Thin REF_DELTA against have-closure | **A** (never) | Thin-pack/ofs-delta serving are protocol MAYs; fat fetches are a bandwidth tax, not a correctness issue, and delta emission is the classic source of corruption bugs. |
20	| Name authority | Singleton Registry DO | Per-namespace NamespaceDO | **A** (singleton) with B's shard seam | Control-plane traffic is tiny in v1 and a singleton gives global listing for free; `getByName("registry:" + owner)` sharding changes no RPC interface. |
21	| Concurrent pushes | Unaddressed (staging ids make it safe-ish) | SQL receive lease + 503 | **Graft B, simplified**: in-memory `Effect.Semaphore` in the DO | Two 50 MiB buffers ≈ the whole isolate heap; one DO instance means an in-memory gate suffices (eviction kills the connection anyway) — no lease table needed. Waiters block ≤ 30 s, then `503 Retry-After: 10`. |
22	| Connectivity check on push | new-oids exist only | Full referenced-object membership | **Graft B** (full check) | With everything in SQLite the membership check is cheap batched SQL, and it's the difference between "corrupt client" and "corrupt repo". |
23	| Commit graph shape | `commits` + `commit_parents` tables | Single row, concatenated parents, `gen`, `commit_time` | **A's normalized tables + B's `gen`/`commit_time` columns** | Normalized parents make the BFS a JOIN; generation numbers bound negotiation walks and cost nothing at ingest. |
24	| Oid encoding | TEXT 40-hex | BLOB 20-byte | **A** (TEXT hex) | zdata BLOBs dominate storage, so index bytes aren't the bottleneck; hex rows are debuggable and match the wire. |
25	| Fork | Vague (test-plan only) | Catalog-row sharing of parent's immutable R2 keys | **Graft B's sharing, adapted**: row snapshot copy + shared R2 keys via `objects.r2_key` | SQLite rows must be copied (no cross-DO sharing), but R2 objects are immutable and shared by full key; parent's R2 prefix is retained while `fork_count > 0`. |
26	| Import / fork / delete execution | Unspecified | Cloudflare Queue consumer jobs | **DO alarm jobs, no Queue** | Alarms have the same 15-min budget as queue consumers, reuse the DO-resident ingest code directly, and add zero infrastructure. |
27	| Async-op status surface | — | `jobs` table + jobs endpoints | **Neither**: `Repo.status` field + `RepoNotReady` error | Polling `GET /repos/:o/:r` for `status: importing\|forking\|deleting → ready\|404` needs no jobs API and no cross-DO status plumbing. |
28	| REST ref writes | Deferred to v2 | PUT/DELETE with CAS in v1 | **Graft B** (in v1) | They reuse the exact `transactionSync` CAS path receive-pack already needs — ~50 lines for a genuinely useful surface (CI tagging, release automation). |
29	| Repo create response | `Repo` | `RepoCreated` with bootstrap write token + remote URL | **Graft B** | Kills the create-then-mint round trip; the token machinery exists anyway. |
30	| `readOnly` repos | — | Repo flag, `ReadOnlyRepo` 403 | **Graft B** | One config row + one check in receive-pack and ref-write; Artifacts parity. |
31	| Token scopes | read/write/admin, names, optional TTL | read/write, mandatory TTL | **A** (three scopes, names, optional TTL) | Per-repo `admin` scope enables delegated token management without handing out the deployer key; mandatory TTL is policy, not mechanism — leave it to the caller. |
32	| Blob REST reads | base64 JSON ≤ 1 MiB + raw streaming route | Octet-stream HttpApi endpoint + `file` path endpoint | **A's split + B's `file` route** (as a raw route) | Raw routes keep binary streaming out of schema land (repo convention); the path-walk `file` read is cheap over existing tree primitives and highly useful. |
33	| Error tag namespacing | Plain tags (`RepoNotFound`) | Prefixed (`git-service/RepoNotFound`) | **A** (plain) | Shorter wire payloads, matches `HttpApiError` conventions; the API name already scopes the client. |
34	| Repo delete semantics | Sync NoContent | Async job id | **NoContent immediately, async purge via alarm** (`status: "deleting"` until gone) | R2 prefix deletion is a list+delete loop that can't run inline; the status field already carries the async story. |
35
36	---
37
38	## 1. Decisions up front (the opinionated core)
39
40	| Decision | Choice | Why |
41	|---|---|---|
42	| Protocol version | **v0 only** (upload-pack and receive-pack) | git ≥ 2.26 sends `Git-Protocol: version=2` but falls back cleanly when the server answers v0. ripgit ships v0-only and hosts 40k-commit repos. v2 is a pure-additive upgrade (§10). |
43	| Negotiation | multi_ack_detailed + no-done, "lazy-ready" | Converges in ≤ 2 POSTs, stateless, answered per-POST from SQLite. |
44	| Served packs | **No deltas, no thin packs.** Every entry = varint header + stored zlib bytes, concatenated | Legal (thin-pack/ofs-delta are MAYs); zero-CPU pack emission because objects are stored as `zlib(content)` — pack generation is literally concatenation + SHA-1. |
45	| Push ingestion | Full thin-pack + ofs/ref-delta support, **pack body buffered, hard cap 50 MiB**, one push at a time per repo | Thin packs on push are non-optional (git always sends them over HTTP). Buffering with a cap is the ripgit-proven memory budget; the streaming-to-R2 upgrade is a contained change (§3.6). |
46	| Where the protocol runs | **Inside the Repo DO** (Worker authenticates + proxies) | The DO owns refs and all loose object bytes; single-threading + input/output gates make ref CAS trivially serializable. Duration billing is a non-issue (§0). |
47	| Object storage | zlib-compressed rows in DO SQLite; objects > 1 MiB compressed → R2; pack compaction to R2 is **v1.1**, schema-ready | Simplest storage that is actually transactional. The 2 MB SQLite row cap forces the R2 split; the `location` column makes compaction additive. |
48	| Refs | DO SQLite, CAS inside `transactionSync` | The single strongest reason this architecture works. R2 is disqualified (1 write/s/key, last-writer-wins). |
49	| Hashing | SHA-1 only (`object-format=sha1`) | sha256 repos are rare; oid schemas are branded so widening later is mechanical. |
50	| Repos | All private in v1. Anonymous = 401. `readOnly` flag rejects writes with a typed 403 | Cuts the visibility model; auth is one code path. |
51	| Shallow | `shallow` + `deepen <n>` (depth only) in v1 | Without it `actions/checkout` (depth=1 default) fails hard. Bounded BFS + boundary lines — cheap. `deepen-since`/`deepen-not` cut. |
52	| Fork / import / delete | Async DO-alarm jobs; status via `Repo.status`; forks share parent R2 keys | Reuses DO-resident ingest and purge code; zero new infrastructure; R2 immutability makes sharing safe. |
53	| Free Workers plan | Unsupported | 10 ms CPU cannot inflate a pack. `GitWorker` sets `limits.cpu_ms = 300_000`. |
54
55	---
56
57	## 2. Architecture
58
59	### 2.1 Components
60
61	```
62	┌────────────────────────────────────────────────┐
63	git CLI / REST │ GitWorker (stateless, Effect-native Worker) │
64	client │ • routes /api/v1/** → Effect HttpApi │
65	──────────────► │ • routes /:owner/:repo.git/** → git protocol │
66	│ • parses Basic/Bearer creds (never verifies │
67	│ repo tokens itself — the Repo DO does) │
68	│ • verifies the admin key (timingSafeEqual) │
69	│ • Registry lookups (owner/name → repoId), │
70	│ 60 s in-isolate LRU cache │
71	└───────┬──────────────────────────┬─────────────┘
72	│ typed RPC / stub.fetch │ typed RPC
73	▼ ▼
74	┌──────────────────────────────┐ ┌────────────────────────┐
75	│ Repo DO (one per repo, │ │ Registry DO │
76	│ getByName(repoId)) │ │ (singleton, │
77	│ owns: refs, objects index, │ │ getByName("registry"))│
78	│ loose object bytes, tokens, │ │ owns: owner/name → │
79	│ commit graph, push staging, │ │ repoId map, listing, │
80	│ alarm jobs (import/fork/ │ │ uniqueness, fork │
81	│ delete-purge/GC) │ │ lineage + counts │
82	│ runs: advertise, upload- │ └────────────────────────┘
83	│ pack, receive-pack │
84	└──────────┬───────────────────┘
85	│ R2 binding (inside the DO for oversize reads/writes)
86	▼
87	┌──────────────────────────────┐
88	│ R2 bucket "git-objects" │
89	│ {repoId}/objects/{oid} │ oversize loose objects (immutable)
90	│ {repoId}/packs/pack-*.pack │ v1.1 compaction output (immutable)
91	└──────────────────────────────┘
92	```
93
94	**Why one DO per repo (not a namespace DO holding many):** per-repo write serialization for free (ref CAS needs no locks), per-repo isolation of the 10 GB SQLite ceiling, horizontal scale by construction. The DO id is `idFromName(repoId)` where `repoId` is a ULID minted at creation — **not** `owner/name` — so renames never move data and a deleted-then-recreated `owner/name` gets a fresh DO (no zombie state).
95
96	**Why a singleton Registry DO:** repo CRUD needs a uniqueness authority, a listing surface, and a fork-lineage ledger. It's low-rate control-plane traffic. Upgrade seam: shard by owner via `getByName("registry:" + owner)` — the RPC interface doesn't change; global listing then requires fan-out or is cut. The Worker caches `owner/name → repoId` per-isolate for 60 s; the Repo DO stores its own `(owner, name)` and rejects mismatched requests, so a stale cache entry fails safe and triggers re-resolve.
97
98	**Why the git protocol runs in the DO, not the Worker:** every byte the protocol needs (refs, object rows, commit graph, tokens) lives in the DO's SQLite. Running upload/receive-pack in the Worker would mean chatty RPC choreography for thin-base lookups, staging batches, and manifest paging. Inside the DO it's direct synchronous SQL. Long pushes don't starve reads: workerd interleaves other events at non-storage `await` points (client body reads, R2 calls), and the ref CAS at the end is a single `transactionSync`. The one inherited rule: **re-verify nothing across an `await` — all ref reads/writes happen inside the final `transactionSync`.**
99
100	**Push serialization:** the Repo DO's runtime closure holds an `Effect.Semaphore(1)` guarding receive-pack ingest (two concurrent 50 MiB buffers would exhaust the 128 MB isolate). A second push waits up to 30 s for the permit, then gets `503` + `Retry-After: 10` (git retries cleanly). In-memory is sufficient: there is exactly one live instance per DO, and an eviction mid-push kills the holder's connection anyway.
101
102	### 2.2 URL space
103
104	```
105	/api/v1/** REST (Effect HttpApi)
106	/api/v1/repos/:o/:r/blobs/:oid/raw raw blob bytes (raw route, streaming)
107	/api/v1/repos/:o/:r/file?ref&path file-at-path bytes (raw route, streaming)
108	/:owner/:repo[.git]/info/refs smart-HTTP ref advertisement
109	/:owner/:repo[.git]/git-upload-pack fetch/clone
110	/:owner/:repo[.git]/git-receive-pack push
111	```
112
113	`api` is a reserved owner name (Registry rejects it at creation). The Worker's `fetch` is the canonical `StateStore/Api.ts` composition: `HttpApiBuilder.layer(GitApi)` + handler layers + `Layer.merge(rawGitRoutes)` (raw routes registered on the same `HttpRouter`) + `[Etag.layer, HttpPlatformStub, Path.layer]` + `HttpRouter.toHttpEffect` (§5, §6).
114
115	### 2.3 Request flows
116
117	**Clone (fresh):**
118	1. `GET /:o/:r.git/info/refs?service=git-upload-pack` → Worker parses Basic creds, Registry lookup (cached) → `repos.getByName(repoId).fetch(request)` → DO verifies token (read scope), rejects with `RepoNotReady` semantics (503) while `status != ready`, then emits the v0 advertisement: `# service=` prelude, flush, `HEAD` first with capability line (`multi_ack_detailed no-done side-band-64k shallow ofs-delta agent=git-service/1 symref=HEAD:refs/heads/<default> object-format=sha1`), refs sorted, annotated tags peeled (`^{}` lines), flush. `Cache-Control: no-cache`. Empty repo: zero-id `capabilities^{}` line.
119	2. `POST git-upload-pack` — body is `want`s + flush + `done` (no haves). DO gunzips if `Content-Encoding: gzip` (`DecompressionStream("gzip")`; sniff `1f 8b` defensively), parses pkt-lines, computes the full closure (§3.5), replies `NAK` then the pack on sideband band 1 (65515-byte frames), progress on band 2, flush. If the client did **not** negotiate `side-band-64k`, the pack is sent raw after the ACK/NAK section. The response body is a `Stream<Uint8Array>` — `HttpServerResponse.stream(...)`, chunked, flushed as written; the Worker proxies the `HttpServerResponse` from `stub.fetch` untouched.
120
121	**Incremental fetch:** same, but the POST carries `have`s. Round 1 (ends in flush): DO `ACK <oid> common` for each have that exists in `objects`, `ACK <oid> ready` once the want-closure is coverable, else `NAK`. With `no-done` the DO may continue straight into the pack after `ready`. Round 2 (ends in `done`): final `ACK`/`NAK` + pack. Closure = commits BFS from wants stopping at ACKed haves (generation numbers bound the walk), plus those commits' full tree/blob closure with an in-walk visited set (§3.5 — the accepted v1 fat: ~one snapshot's worth of redundancy at the boundary; harmless to clients).
122
123	**Shallow clone (`--depth N`):** wants + `deepen N`. DO does a depth-bounded BFS, emits `shallow <oid>` boundary lines (and `unshallow` for previously-shallow client tips now complete), flush, then ACK/NAK + depth-truncated pack. Client `shallow` lines on later fetches are treated as additional walk boundaries.
124
125	**Push:**
126	1. `GET info/refs?service=git-receive-pack` → v0 advertisement with `report-status report-status-v2 delete-refs side-band-64k atomic ofs-delta object-format=sha1 agent=git-service/1`.
127	2. `POST git-receive-pack` → DO: acquire the push semaphore; parse command pkt-lines (`old new refname\0caps` on the first); handle the **empty-flush probe** (git sends a bare `0000` POST when the payload exceeds `http.postBuffer`; reply empty 200 so it retries with the real body). Reject writes to `readOnly` repos with `ng` per ref. Buffer the raw pack (reject > 50 MiB with `unpack pack exceeds 50 MiB limit` + per-ref `ng` — a clean in-band report, not an HTTP error). Ingest (§3.6): index → resolve deltas (thin bases from own store) → hash → stage → **connectivity check** (every oid referenced by staged commits/trees is in staged ∪ live objects; batched SQL). Then one `transactionSync`: CAS every command's old-oid against `refs`, honor `atomic` (all-or-nothing) vs per-ref, flip staged objects live, insert commit-graph rows, update `default_branch` on first branch push. Reply `report-status` (`unpack ok` / `ok|ng <ref>`) **wrapped in sideband band 1** when the client negotiated side-band (unmuxed report while sideband is active breaks the client). `report-status-v2` is advertised and implemented identically (no `option` lines — valid).
128	3. Post-commit, `state.waitUntil`: GC bookkeeping, compaction-threshold check (v1.1 alarm).
129
130	**Fork:** `POST /repos/:o/:r/fork` → Registry CAS-inserts the target row (`status: "forking"`, `fork_of`, bump parent `fork_count`) → new Repo DO initializes and arms an immediate alarm → alarm job streams a row snapshot from the parent DO over an RPC `Stream` (refs, objects metadata **including `zdata` for `location='row'` rows**, commits, config) and copies it in batches; `location='r2'` rows are copied **by reference** — the `r2_key` column carries the parent's full key (immutable, shared) → `status: "ready"`. Documented v1 limit: fork cost is O(parent row bytes); v1.1 compaction (bytes in shared R2 packs) makes forks near-O(index).
131
132	**Import:** `POST /repos/import` → Registry inserts (`status: "importing"`) → Repo DO alarm job acts as a git v0/v2 HTTP client against the source URL (depth-limited fetch), ingests through the same `PackParser` path with the same 50 MiB cap (documented v1 limit: small/depth-limited imports only), sets refs, `status: "ready"`; failures set `status` back plus an error detail readable on the Repo resource.
133
134	**Delete:** `DELETE /repos/:o/:r` → Registry marks `deleted_at` (name freed for reuse only after purge), Repo `status: "deleting"` → NoContent immediately → alarm job purges: R2 prefix list+delete loop (bounded per alarm run, re-armed until done) — **but the R2 prefix is retained while Registry reports `fork_count > 0`** (forks reference those keys); SQLite dropped via `deleteAll`; Registry row removed last. Purge is idempotent and alarm-retried.
135
136	**REST call:** `POST /api/v1/repos/:o/:r/tokens` → HttpApi middleware parses credentials into a `Credentials` service → handler calls `repos.getByName(repoId).createToken(credentials, {...})` → DO enforces scope and returns; tagged errors surface as typed HTTP statuses.
137
138	---
139
140	## 3. Storage design
141
142	### 3.1 Repo DO SQLite schema (exact DDL)
143
144	```sql
145	-- config: repoId, owner, name, default_branch, description, created_at,
146	-- schema_version, status ('ready'|'importing'|'forking'|'deleting'),
147	-- read_only ('0'|'1'), fork_of (parent repoId or absent)
148	CREATE TABLE IF NOT EXISTS config (key TEXT PRIMARY KEY, value TEXT NOT NULL);
149
150	-- all refs, full names. HEAD is virtual: symref to config.default_branch.
151	CREATE TABLE IF NOT EXISTS refs (
152	name TEXT PRIMARY KEY, -- 'refs/heads/main', 'refs/tags/v1'
153	oid TEXT NOT NULL -- 40-hex sha1
154	) WITHOUT ROWID;
155
156	-- object index + loose bytes. zdata is zlib(content) WITHOUT the loose header,
157	-- so pack emission is: varint(type,size) + zdata, verbatim.
158	CREATE TABLE IF NOT EXISTS objects (
159	oid TEXT PRIMARY KEY, -- 40-hex
160	type INTEGER NOT NULL, -- 1=commit 2=tree 3=blob 4=tag
161	size INTEGER NOT NULL, -- uncompressed bytes
162	zsize INTEGER NOT NULL, -- stored/compressed bytes
163	location TEXT NOT NULL DEFAULT 'row', -- 'row' | 'r2' | 'pack' (pack = v1.1)
164	zdata BLOB, -- NULL when location != 'row'
165	r2_key TEXT, -- full R2 key when location='r2'; may point at a
166	-- FORK PARENT's prefix (immutable, shared)
167	pack_id TEXT, -- v1.1: packs/pack-<sha>.pack
168	pack_offset INTEGER, -- v1.1: byte offset of zdata within pack
169	staged_push TEXT -- NULL = live; else the push_id that staged it
170	) WITHOUT ROWID;
171	CREATE INDEX IF NOT EXISTS idx_objects_staged ON objects(staged_push) WHERE staged_push IS NOT NULL;
172
173	-- commit graph for closure/negotiation walks (populated at ingest)
174	CREATE TABLE IF NOT EXISTS commits (
175	oid TEXT PRIMARY KEY,
176	tree TEXT NOT NULL,
177	gen INTEGER NOT NULL, -- generation number: max(parent gen) + 1; roots = 1
178	commit_time INTEGER NOT NULL -- committer timestamp (seam: deepen-since)
179	) WITHOUT ROWID;
180	CREATE TABLE IF NOT EXISTS commit_parents (
181	oid TEXT NOT NULL,
182	parent TEXT NOT NULL,
183	ord INTEGER NOT NULL,
184	PRIMARY KEY (oid, parent)
185	) WITHOUT ROWID;
186
187	-- per-repo access tokens (§8)
188	CREATE TABLE IF NOT EXISTS tokens (
189	id TEXT PRIMARY KEY, -- ULID
190	token_hash TEXT NOT NULL UNIQUE, -- hex(sha256(token))
191	name TEXT NOT NULL,
192	scope TEXT NOT NULL, -- 'read' | 'write' | 'admin'
193	created_at INTEGER NOT NULL,
194	expires_at INTEGER, -- NULL = no expiry
195	last_used_at INTEGER
196	) WITHOUT ROWID;
197
198	-- crash-safety bookkeeping for pushes (GC'd by alarm)
199	CREATE TABLE IF NOT EXISTS pushes (
200	push_id TEXT PRIMARY KEY,
201	started_at INTEGER NOT NULL,
202	state TEXT NOT NULL -- 'staging' | 'committed'
203	) WITHOUT ROWID;
204
205	-- async job bookkeeping (import/fork/delete-purge progress; single row per kind)
206	CREATE TABLE IF NOT EXISTS jobs (
207	kind TEXT PRIMARY KEY, -- 'import' | 'fork' | 'purge' | 'gc'
208	state TEXT NOT NULL, -- 'running' | 'failed'
209	detail TEXT,
210	updated_at INTEGER NOT NULL
211	) WITHOUT ROWID;
212	```
213
214	Registry DO:
215
216	```sql
217	CREATE TABLE IF NOT EXISTS repos (
218	owner TEXT NOT NULL,
219	name TEXT NOT NULL,
220	repo_id TEXT NOT NULL UNIQUE, -- ULID
221	description TEXT,
222	fork_of TEXT, -- parent repo_id, NULL if not a fork
223	fork_count INTEGER NOT NULL DEFAULT 0,
224	created_at INTEGER NOT NULL,
225	deleted_at INTEGER, -- soft-delete marker during purge
226	PRIMARY KEY (owner, name)
227	) WITHOUT ROWID;
228	```
229
230	Schema init runs in the DO's outer init effect under `blockConcurrencyWhile` semantics (idempotent `CREATE TABLE IF NOT EXISTS`), with a `schema_version` config row for future migrations. Note the platform gotcha: DO RPC methods must not be named `delete` (Cloudflare stub proxy reserves it) — use `remove`/`purge`.
231
232	### 3.2 R2 key scheme
233
234	```
235	{repoId}/objects/{oid} zlib(content), immutable, written once
236	{repoId}/packs/pack-{sha1}.pack v1.1 compaction output, immutable
237	```
238
239	All R2 keys are **immutable and content-addressed** — this sidesteps the 1-write/s/key limit, makes conditional-put re-runs (idempotent alarm jobs) safe, and makes **cross-repo fork references safe** (a fork's `objects.r2_key` may point into the parent's prefix). R2 `list()` is never used on a hot path (Class A, $4.50/M); the `objects` table is the sole inventory. List is used only in delete-purge and GC reconciliation.
240
241	### 3.3 Loose vs. R2 thresholds and the promotion story
242
243	- **`location='row'`**: compressed size ≤ **1 MiB** (safety margin under the 2 MB SQLite row cap). This is where every normal object lives in v1. Read cost: DO rows at $0.001/M, synchronous, no 6-connection cap — 360× cheaper and far faster than per-object R2 GETs.
244	- **`location='r2'`**: compressed size > 1 MiB → written to `{repoId}/objects/{oid}` at ingest (single streaming `put`), row records metadata + `r2_key`.
245	- **Per-object cap (v1): 64 MiB uncompressed.** Delta application needs base + result in memory; this cap keeps worst-case ingest memory bounded. Rejected with `unpack object too large` + `ng`.
246	- **Promotion/compaction (v1.1, schema-ready, not in v1):** an alarm fires when `SUM(zsize) WHERE location='row'` > **1 GB** or live loose count > **50 000**. The handler streams a no-delta pack of the oldest N objects to R2 via multipart (5 MiB uniform parts), then in one `transactionSync` flips those rows to `location='pack'` with `pack_id`/`pack_offset` and NULLs `zdata`. Because pack entries are `varint header + zlib(content)` and we store `zlib(content)`, the "pack" is a concatenation — and `pack_offset` points at the zdata span, so fetch does **ranged R2 reads** per object, or (B's fast path, adopted as the v1.1 clone path) streams whole entry-regions verbatim for full clones. Idempotent (content-addressed pack key + conditional put), alarm-retried, 15-min budget respected by capping objects per run. Nothing in the fetch path changes shape: the emitter already dispatches on `location`.
247
248	This ordering (row → r2/pack) is forced by the platform numbers: DO storage $0.20/GB-mo vs R2 $0.015 (13×) makes R2 the long-term home; DO rows' read economics make them the right hot tier; the 2 MB row cap draws the line.
249
250	### 3.4 Transactionality of ref updates
251
252	All ref mutations — push commands, REST ref writes, fork/import finalization — go through one code path in the Repo DO:
253
254	```ts
255	// inside receive-pack finalize, single event, no awaits inside:
256	storage.transactionSync(() => {
257	for (const cmd of commands) {
258	const current = sql.exec("SELECT oid FROM refs WHERE name = ?", cmd.ref).one()?.oid ?? ZERO;
259	if (current !== cmd.oldOid) { results.push(ng(cmd.ref, "fetch first")); if (atomic) throw new RefCasFailed(results); continue; }
260	// delete / insert / update refs row
261	}
262	sql.exec("UPDATE objects SET staged_push = NULL WHERE staged_push = ?", pushId);
263	// insert commits / commit_parents rows for staged commits
264	sql.exec("UPDATE pushes SET state = 'committed' WHERE push_id = ?", pushId);
265	});
266	```
267
268	The DO's input/output gates guarantee no interleaving between statements; `transactionSync` guarantees atomicity against mid-event failure. The client's `old-oid` is the compare-and-swap token (this also makes force-push semantics correct without locking). REST `PUT/DELETE` ref endpoints run the same block with a single synthetic command (`expectedOid` → old-oid; absent `expectedOid` = unconditional). Crash before the transaction ⇒ staged rows with a dangling `push_id`; a daily alarm deletes `pushes` rows in `staging` older than 24 h and their staged objects (R2 orphans under `objects/` are harmless — content-addressed — and re-pushed identical objects hit the same key).
269
270	### 3.5 Fetch closure computation and pack assembly within 128 MB
271
272	**Closure:** BFS from wants over `commit_parents` (in-DO SQL, chunked `IN` lists ≤ 100 params), stopping at haves that exist in `objects`, bounded below by generation numbers (never walk past `min(gen(haves))` unnecessarily), and at the depth bound when `deepen` is present. Then walk each frontier commit's tree: read tree rows, parse entries, recurse, with an in-memory visited `Set<string>` shared across the whole walk. Memory: 500 k oids ≈ 25 MB of Set — cap manifest at 1 M objects for v1 (larger repos are a documented v1 limit; the v1.1 fix is whole-pack streaming from R2 for clones).
273
274	**Emission (all streaming, constant memory):**
275
276	```
277	PACK + version(2) + count(manifest.length) — count known before streaming: manifest is materialized first
278	for each oid in manifest (batched row reads, ≤100/query, ~8 MiB adaptive zsize budget):
279	emit varint(type, size) + zdata — 'row': the BLOB; 'r2': R2 get body; 'pack' (v1.1): R2 ranged get
280	sha1.update(everything) — node:crypto createHash, incremental
281	emit sha1 trailer
282	```
283
284	Wrapped in a sideband-band-1 framing transform (≤ 65515-byte payloads) when negotiated, returned as `HttpServerResponse.stream`. No compression CPU at all — bytes are stored pre-deflated. Progress lines (`Counting objects...`) on band 2 every N objects.
285
286	### 3.6 Push ingest within 128 MB
287
288	Single pass over the buffered pack (≤ 50 MiB), under the push semaphore:
289
290	1. Validate `PACK`, version 2, count.
291	2. Per entry: parse type/size varint. Non-delta: `zlib.createInflate()`, feed the tail; on `end`, `bytesWritten` = exact compressed span (workerd-verified). Hash `sha1("<type> <size>\0" + content)`. **Store the compressed span verbatim as `zdata`** — no recompression. Deltas (OFS with the +1-bias offset decoding, REF with 20-byte base id): resolve base from (a) already-ingested entries via a bounded LRU of resolved contents (20 MiB, shared buffers), (b) re-inflation from the pack buffer on cache miss, or (c) **the object store for thin bases** (REF_DELTA to a prior push — the normal case). Apply the copy/insert instruction stream (`size==0 ⇒ 0x10000` special case), verify `result_size`, hash, `deflateSync` (level 6), store.
292	3. Verify the trailing pack SHA-1; mismatch ⇒ `unpack pack checksum mismatch` + all-`ng`.
293	4. Rows inserted with `staged_push = pushId` as resolved (objects > 1 MiB compressed go to R2 first, then the row). Existing oids skipped (idempotent re-push).
294	5. **Connectivity check (full):** collect every oid referenced by staged commits (tree, parents) and staged trees (entries), and every command's `new-oid`; batched `SELECT` membership against staged ∪ live objects (chunked `IN` ≤ 100). Any miss ⇒ `unpack missing objects` + all-`ng`. Cheap SQL; prevents a buggy/malicious client from corrupting the repo.
295	6. Compute `gen`/`commit_time` for staged commits (parents are all present per step 5).
296
297	Budget: 50 MiB pack + 20 MiB cache + ≤ 64 MiB single-object worst case is theoretical-max-tight; in practice large objects aren't delta'd against large bases, and ripgit ships this exact profile in 128 MB. The semaphore guarantees one ingest at a time. The pack size cap is enforced *before* buffering via `Content-Length` when present and by counting during body read otherwise.
298
299	**Upgrade seam (v1.x, when the 50 MiB cap hurts):** tee the incoming body to R2 multipart (`{repoId}/incoming/{pushId}.pack`) while indexing in the same pass; delta re-resolution then uses R2 ranged reads instead of the memory buffer. The parser is written against a `RandomAccess` abstraction (`PackParser.ingest(source: RandomAccess)`) from day one so this swap touches one implementation, not the protocol code.
300
301	---
302
303	## 4. Git protocol scope for v1
304
305	### Supported (exact)
306
307	| Area | v1 support |
308	|---|---|
309	| Transport | Smart HTTP only. `info/refs` advertisement with `# service=` prelude for both services; `Cache-Control: no-cache`; 401 + `WWW-Authenticate: Basic realm="git-service"`; gzip request bodies decompressed (`DecompressionStream("gzip")` + `1f 8b` sniff); streamed chunked responses; `ERR` pkt / band-3 for in-band fatals; 503 + `Retry-After` on push contention. |
310	| upload-pack (fetch/clone) | **Protocol v0.** Capabilities: `multi_ack_detailed`, `no-done`, `side-band-64k`, `shallow` (+ `deepen <n>` only), `agent`, `symref=HEAD:...`, `object-format=sha1`. Peeled `^{}` lines for annotated tags. Negotiation: ack `common` for known haves, `ready` when coverable, else `NAK`; ≤ 2 rounds. Empty-repo advertisement (`capabilities^{}`). Non-sideband clients get the raw pack. |
311	| receive-pack (push) | Protocol v0 (push has no v2). Capabilities: `report-status`, `report-status-v2`, `delete-refs`, `side-band-64k`, `atomic`, `ofs-delta`, `object-format=sha1`, `agent`. Create/update/delete refs, old-oid CAS, force push (unpoliced in v1), thin-pack + ofs/ref-delta ingestion, full connectivity check, empty-flush probe handling, report inside band 1, one-push-at-a-time serialization. |
312	| Objects | commit, tree, blob, **tag (annotated tags stored and served — not dropped)**. Byte-verbatim round-tripping (we never re-serialize; hashes, gpgsig, timezones survive — automatic, since stored bytes are the received bytes). |
313
314	The cut list with upgrade paths is consolidated in §10.
315
316	---
317
318	## 5. The Effect HttpApi REST surface
319
320	Contract lives in `src/Api.ts`, following `packages/alchemy/src/Cloudflare/StateStore/Api.ts` / `State/HttpStateApi.ts` conventions (effect `4.0.0-beta.105`, `effect/httpapi`). Abbreviated only where repetitive:
321
322	```ts
323	import * as Context from "effect/Context";
324	import * as Redacted from "effect/Redacted";
325	import * as Schema from "effect/Schema";
326	import * as HttpApi from "effect/httpapi/HttpApi";
327	import * as HttpApiGroup from "effect/httpapi/HttpApiGroup";
328	import * as HttpApiEndpoint from "effect/httpapi/HttpApiEndpoint";
329	import * as HttpApiMiddleware from "effect/httpapi/HttpApiMiddleware";
330	import * as HttpApiSecurity from "effect/httpapi/HttpApiSecurity";
331	import * as HttpApiSchema from "effect/httpapi/HttpApiSchema";
332
333	// ── primitives ────────────────────────────────────────────────────────────
334	export const Oid = Schema.String.pipe(Schema.pattern(/^[0-9a-f]{40}$/), Schema.brand("Oid"));
335	export const RepoName = Schema.String.pipe(Schema.pattern(/^[a-z0-9](?:[a-z0-9._-]{0,98}[a-z0-9])?$/i));
336	export const OwnerName = RepoName;
337	export const RefName = Schema.String.pipe(Schema.pattern(/^refs\/[^\s~^:?*\[\\]+$/));
338	export const TokenScope = Schema.Literals(["read", "write", "admin"]);
339	export const RepoStatus = Schema.Literals(["ready", "importing", "forking", "deleting"]);
340
341	// ── error taxonomy (tagged, status-annotated) ─────────────────────────────
342	export class Unauthorized extends Schema.TaggedErrorClass<Unauthorized>()("Unauthorized",
343	{}, HttpApiSchema.annotations({ status: 401 })) {}
344	export class Forbidden extends Schema.TaggedErrorClass<Forbidden>()("Forbidden",
345	{ required: TokenScope }, HttpApiSchema.annotations({ status: 403 })) {}
346	export class ReadOnlyRepo extends Schema.TaggedErrorClass<ReadOnlyRepo>()("ReadOnlyRepo",
347	{}, HttpApiSchema.annotations({ status: 403 })) {}
348	export class RepoNotFound extends Schema.TaggedErrorClass<RepoNotFound>()("RepoNotFound",
349	{ owner: Schema.String, repo: Schema.String }, HttpApiSchema.annotations({ status: 404 })) {}
350	export class RepoAlreadyExists extends Schema.TaggedErrorClass<RepoAlreadyExists>()("RepoAlreadyExists",
351	{ owner: Schema.String, repo: Schema.String }, HttpApiSchema.annotations({ status: 409 })) {}
352	export class RepoNotReady extends Schema.TaggedErrorClass<RepoNotReady>()("RepoNotReady",
353	{ status: RepoStatus }, HttpApiSchema.annotations({ status: 409 })) {}
354	export class RefNotFound extends Schema.TaggedErrorClass<RefNotFound>()("RefNotFound",
355	{ ref: Schema.String }, HttpApiSchema.annotations({ status: 404 })) {}
356	export class RefConflict extends Schema.TaggedErrorClass<RefConflict>()("RefConflict",
357	{ ref: Schema.String, currentOid: Schema.NullOr(Oid) }, HttpApiSchema.annotations({ status: 409 })) {}
358	export class ObjectNotFound extends Schema.TaggedErrorClass<ObjectNotFound>()("ObjectNotFound",
359	{ oid: Schema.String }, HttpApiSchema.annotations({ status: 404 })) {}
360	export class WrongObjectType extends Schema.TaggedErrorClass<WrongObjectType>()("WrongObjectType",
361	{ oid: Schema.String, expected: Schema.String, actual: Schema.String },
362	HttpApiSchema.annotations({ status: 422 })) {}
363	export class ObjectTooLarge extends Schema.TaggedErrorClass<ObjectTooLarge>()("ObjectTooLarge",
364	{ oid: Schema.String, size: Schema.Number }, HttpApiSchema.annotations({ status: 422 })) {}
365	export class TokenNotFound extends Schema.TaggedErrorClass<TokenNotFound>()("TokenNotFound",
366	{ id: Schema.String }, HttpApiSchema.annotations({ status: 404 })) {}
367	export class ImportFailed extends Schema.TaggedErrorClass<ImportFailed>()("ImportFailed",
368	{ reason: Schema.String }, HttpApiSchema.annotations({ status: 502 })) {}
369	export class ValidationError extends Schema.TaggedErrorClass<ValidationError>()("ValidationError",
370	{ message: Schema.String }, HttpApiSchema.annotations({ status: 400 })) {}
371
372	// ── auth middleware ───────────────────────────────────────────────────────
373	// Parses Basic (git CLI: password = token) or Bearer into a Credentials service.
374	// Enforcement happens in the Repo DO, which owns the tokens table (§8).
375	export class Credentials extends Context.Service<Credentials, {
376	readonly token: Redacted.Redacted<string>; // repo token or admin key
377	}>()("git-service/Credentials") {}
378
379	export class GitAuth extends HttpApiMiddleware.Service<GitAuth, { provides: Credentials }>()(
380	"git-service/GitAuth",
381	{ security: { bearer: HttpApiSecurity.bearer, basic: HttpApiSecurity.basic },
382	error: Unauthorized },
383	) {}
384
385	// ── domain shapes ─────────────────────────────────────────────────────────
386	export class Repo extends Schema.Class<Repo>("Repo")({
387	owner: OwnerName, name: RepoName, repoId: Schema.String,
388	defaultBranch: Schema.String, description: Schema.NullOr(Schema.String),
389	readOnly: Schema.Boolean,
390	forkOf: Schema.NullOr(Schema.String), // parent repoId
391	status: RepoStatus, // poll this for async fork/import/delete
392	createdAt: Schema.Number,
393	}) {}
394	export class Ref extends Schema.Class<Ref>("Ref")({
395	name: RefName, oid: Oid, peeled: Schema.optional(Oid),
396	}) {}
397	const Signature = Schema.Struct({
398	name: Schema.String, email: Schema.String, date: Schema.Number, tz: Schema.String });
399	export class CommitInfo extends Schema.Class<CommitInfo>("CommitInfo")({
400	oid: Oid, tree: Oid, parents: Schema.Array(Oid),
401	author: Signature, committer: Signature, message: Schema.String,
402	}) {}
403	export class TreeEntry extends Schema.Class<TreeEntry>("TreeEntry")({
404	mode: Schema.String, name: Schema.String, oid: Oid,
405	type: Schema.Literals(["blob", "tree", "commit"]), // commit = gitlink
406	}) {}
407	export class TokenInfo extends Schema.Class<TokenInfo>("TokenInfo")({
408	id: Schema.String, name: Schema.String, scope: TokenScope,
409	createdAt: Schema.Number, expiresAt: Schema.NullOr(Schema.Number),
410	lastUsedAt: Schema.NullOr(Schema.Number),
411	}) {}
412	export class CreatedToken extends TokenInfo.extend<CreatedToken>("CreatedToken")({
413	token: Schema.String, // shown exactly once
414	}) {}
415	export class RepoCreated extends Schema.Class<RepoCreated>("RepoCreated")({
416	repo: Repo,
417	remote: Schema.String, // https clone URL
418	token: CreatedToken, // bootstrap 'write' token, one round trip
419	}) {}
420	const Paginated = <A extends Schema.Top>(items: A) => Schema.Struct({
421	items: Schema.Array(items), nextCursor: Schema.NullOr(Schema.String), hasMore: Schema.Boolean,
422	});
423
424	// ── groups ────────────────────────────────────────────────────────────────
425	const RepoPath = Schema.Struct({ owner: OwnerName, repo: RepoName });
426
427	const repos = HttpApiGroup.make("repos")
428	.add(HttpApiEndpoint.post("create", "/repos", {
429	payload: Schema.Struct({
430	owner: OwnerName, name: RepoName,
431	defaultBranch: Schema.optional(Schema.String), // default "main"
432	description: Schema.optional(Schema.String),
433	readOnly: Schema.optional(Schema.Boolean),
434	}),
435	success: RepoCreated,
436	error: [RepoAlreadyExists, ValidationError, Forbidden],
437	}))
438	.add(HttpApiEndpoint.get("get", "/repos/:owner/:repo", {
439	params: RepoPath, success: Repo, error: RepoNotFound,
440	}))
441	.add(HttpApiEndpoint.patch("update", "/repos/:owner/:repo", {
442	params: RepoPath,
443	payload: Schema.Struct({
444	description: Schema.optional(Schema.NullOr(Schema.String)),
445	defaultBranch: Schema.optional(Schema.String), // must resolve to an existing branch
446	readOnly: Schema.optional(Schema.Boolean),
447	}),
448	success: Repo,
449	error: [RepoNotFound, RefNotFound, Forbidden],
450	}))
451	.add(HttpApiEndpoint.get("list", "/repos", {
452	query: Schema.Struct({
453	owner: Schema.optional(OwnerName),
454	cursor: Schema.optional(Schema.String),
455	limit: Schema.optional(Schema.Int.pipe(Schema.between(1, 100))),
456	}),
457	success: Paginated(Repo),
458	}))
459	.add(HttpApiEndpoint.del("delete", "/repos/:owner/:repo", {
460	params: RepoPath, success: HttpApiSchema.NoContent, // async purge; status → 'deleting' → 404
461	error: [RepoNotFound, Forbidden],
462	}))
463	.add(HttpApiEndpoint.post("fork", "/repos/:owner/:repo/fork", {
464	params: RepoPath,
465	payload: Schema.Struct({ targetOwner: OwnerName, targetName: RepoName }),
466	success: RepoCreated, // status: 'forking'; poll GET repo
467	error: [RepoNotFound, RepoAlreadyExists, RepoNotReady, Forbidden],
468	}))
469	.add(HttpApiEndpoint.post("import", "/repos/import", {
470	payload: Schema.Struct({
471	owner: OwnerName, name: RepoName,
472	source: Schema.Struct({
473	url: Schema.String,
474	ref: Schema.optional(Schema.String),
475	depth: Schema.optional(Schema.Int.pipe(Schema.greaterThan(0))),
476	}),
477	}),
478	success: RepoCreated, // status: 'importing'; poll GET repo
479	error: [RepoAlreadyExists, ImportFailed, Forbidden],
480	}))
481	.middleware(GitAuth);
482
483	const refs = HttpApiGroup.make("refs")
484	.add(HttpApiEndpoint.get("list", "/repos/:owner/:repo/refs", {
485	params: RepoPath,
486	query: Schema.Struct({ prefix: Schema.optional(Schema.String) }), // e.g. "refs/heads/"
487	success: Schema.Struct({ head: Schema.NullOr(Schema.String), refs: Schema.Array(Ref) }),
488	error: RepoNotFound,
489	}))
490	.add(HttpApiEndpoint.get("get", "/repos/:owner/:repo/ref", {
491	params: RepoPath,
492	query: Schema.Struct({ name: RefName }), // query, not path — refnames contain '/'
493	success: Ref,
494	error: [RepoNotFound, RefNotFound],
495	}))
496	.add(HttpApiEndpoint.put("update", "/repos/:owner/:repo/ref", {
497	params: RepoPath,
498	query: Schema.Struct({ name: RefName }),
499	payload: Schema.Struct({
500	newOid: Oid,
501	expectedOid: Schema.optional(Schema.NullOr(Oid)), // null = must-not-exist; absent = unconditional
502	}),
503	success: Ref,
504	error: [RepoNotFound, RefConflict, ObjectNotFound, ReadOnlyRepo, Forbidden],
505	}))
506	.add(HttpApiEndpoint.del("remove", "/repos/:owner/:repo/ref", {
507	params: RepoPath,
508	query: Schema.Struct({ name: RefName }),
509	payload: Schema.Struct({ expectedOid: Schema.optional(Oid) }),
510	success: HttpApiSchema.NoContent,
511	error: [RepoNotFound, RefNotFound, RefConflict, ReadOnlyRepo, Forbidden],
512	}))
513	.middleware(GitAuth);
514
515	const objects = HttpApiGroup.make("objects")
516	.add(HttpApiEndpoint.get("commit", "/repos/:owner/:repo/commits/:oid", {
517	params: RepoPath.pipe(Schema.extend(Schema.Struct({ oid: Oid }))),
518	success: CommitInfo,
519	error: [RepoNotFound, ObjectNotFound, WrongObjectType],
520	}))
521	.add(HttpApiEndpoint.get("log", "/repos/:owner/:repo/log", {
522	params: RepoPath,
523	query: Schema.Struct({
524	ref: Schema.optional(Schema.String), // refname or oid; default HEAD
525	cursor: Schema.optional(Schema.String),
526	limit: Schema.optional(Schema.Int.pipe(Schema.between(1, 100))),
527	}),
528	success: Paginated(CommitInfo),
529	error: [RepoNotFound, RefNotFound],
530	}))
531	.add(HttpApiEndpoint.get("tree", "/repos/:owner/:repo/trees/:oid", {
532	params: RepoPath.pipe(Schema.extend(Schema.Struct({ oid: Oid }))),
533	success: Schema.Struct({ oid: Oid, entries: Schema.Array(TreeEntry) }),
534	error: [RepoNotFound, ObjectNotFound, WrongObjectType],
535	}))
536	.add(HttpApiEndpoint.get("blob", "/repos/:owner/:repo/blobs/:oid", {
537	params: RepoPath.pipe(Schema.extend(Schema.Struct({ oid: Oid }))),
538	success: Schema.Struct({
539	oid: Oid, size: Schema.Number,
540	encoding: Schema.Literals(["base64"]), content: Schema.String, // ≤ 1 MiB only
541	}),
542	error: [RepoNotFound, ObjectNotFound, WrongObjectType, ObjectTooLarge], // 422 → use /raw
543	}))
544	// Raw streaming reads are registered as RAW HttpRouter routes (octet-stream),
545	// outside HttpApi schema-land by design (same policy as the git wire endpoints):
546	// GET /api/v1/repos/:owner/:repo/blobs/:oid/raw
547	// GET /api/v1/repos/:owner/:repo/file?ref=<refname|oid>&path=<path> (tree-walk per segment)
548	.middleware(GitAuth);
549
550	const tokens = HttpApiGroup.make("tokens")
551	.add(HttpApiEndpoint.post("create", "/repos/:owner/:repo/tokens", {
552	params: RepoPath,
553	payload: Schema.Struct({
554	name: Schema.String, scope: TokenScope,
555	ttlSeconds: Schema.optional(Schema.Int.pipe(Schema.greaterThan(0))),
556	}),
557	success: CreatedToken,
558	error: [RepoNotFound, Forbidden],
559	}))
560	.add(HttpApiEndpoint.get("list", "/repos/:owner/:repo/tokens", {
561	params: RepoPath, success: Schema.Array(TokenInfo), error: [RepoNotFound, Forbidden],
562	}))
563	.add(HttpApiEndpoint.del("revoke", "/repos/:owner/:repo/tokens/:id", {
564	params: RepoPath.pipe(Schema.extend(Schema.Struct({ id: Schema.String }))),
565	success: HttpApiSchema.NoContent,
566	error: [RepoNotFound, TokenNotFound, Forbidden],
567	}))
568	.middleware(GitAuth);
569
570	export const GitApi = HttpApi.make("git-service")
571	.add(repos).add(refs).add(objects).add(tokens)
572	.prefix("/api/v1");
573	```
574
575	**Authorization matrix** (enforced in the Repo DO, §8): admin key → everything; repo `admin` token → its repo's write + token create/list/revoke + repo update/delete; `write` → its repo's reads + receive-pack + ref writes; `read` → its repo's reads + upload-pack only. Repo create/list-all/fork/import are admin-key-only.
576
577	`GitAuthLive` follows `StateAuthLive` exactly: `Layer.effect(GitAuth, ...)` returning `{ bearer, basic }` handlers that wrap the endpoint effect with a `Credentials` provision (Basic: password field is the token, username ignored). Handlers pass `Credentials` to Repo-DO RPCs, which are the enforcement point. Mounting uses the canonical composition — `HttpApiBuilder.layer(GitApi)` + group handler layers + `GitAuthLive` + `[Etag.layer, HttpPlatformStub, Path.layer]` (the `HttpPlatform` stub copied verbatim from `StateStore/Api.ts:333`) + `Layer.merge(rawApi)` + `HttpRouter.toHttpEffect` assigned to `fetch`.
578
579	---
580
581	## 6. Package layout — `packages/git`
582
583	Shape: **library-with-deployable-stack**, modeled on `packages/better-auth` — exports Worker/DO classes + the HttpApi contract for users to compose into their own Stacks, plus a canonical Stack factory and an example app.
584
585	```
586	packages/git/
587	package.json # @alchemy.run/git; exports ".", "./*" (types/bun/worker→src, import→lib);
588	# peerDeps: alchemy workspace:*, effect catalog:, @cloudflare/workers-types catalog:;
589	# devDeps: alchemy-test workspace:*, isomorphic-git (test-only wire client)
590	tsconfig.json # composite, extends ../../tsconfig.base.json, refs ../alchemy, paths @/* → src/*
591	tsconfig.test.json
592	src/
593	index.ts # barrel: Api contract, error classes, GitWorker, Repo, Registry, GitService
594	Api.ts # §5 verbatim — HttpApi contract + schemas + tagged errors + GitAuth tag
595	Auth.ts # Credentials service, GitAuthLive, credential parsing (Basic/Bearer),
596	# admin-key timing-safe compare (crypto.subtle.timingSafeEqual)
597	GitWorker.ts # class GitWorker extends Cloudflare.Worker<GitWorker>()(...):
598	# HttpApi mount, raw routes (git wire, /blobs/:oid/raw, /file),
599	# Registry LRU cache, stub.fetch proxying, nodejs_compat + limits.cpu_ms
600	RepoObject.ts # class Repo extends Cloudflare.DurableObject<Repo>()("GitRepo") {} + Repo.make(impl):
601	# RPC surface (verifyToken, createToken, listTokens, revokeToken,
602	# getRepoMeta, updateRepoMeta, listRefs, getRef, updateRef, removeRef,
603	# readObject, readCommitLog, readFileAtPath, snapshotRows (Stream),
604	# startImport, startFork, startPurge)
605	# + fetch (git wire choreography) + alarm (jobs: import/fork/purge/gc)
606	# + in-memory push Semaphore
607	RegistryObject.ts # class Registry DO: createRepo, resolve, list, forkLineage,
608	# bumpForkCount, markDeleted, remove
609	Service.ts # GitService(options?): a function returning an Effect the user
610	# yields inside their OWN Alchemy.Stack; deploys GitWorker (+ DOs + R2
611	# transitively) and returns { worker, url }. No Stack is shipped.
612	git/ # pure protocol/codec modules — no DO/R2 imports, unit-testable
613	Pkt.ts # pkt-line read/write, flush/delim, ERR pkt; Stream transforms
614	Sideband.ts # band-1/2/3 framing transform (65515 cap)
615	ObjectCodec.ts # oid hashing (node:crypto sha1, incremental), loose header,
616	# commit/tree/tag parse + tree entry sort rules, varints
617	Delta.ts # copy/insert delta application, size verification
618	PackParser.ts # §3.6 ingest over a RandomAccess source; node:zlib Inflate.bytesWritten
619	PackWriter.ts # §3.5 no-delta emitter: header/count, varint+zdata concat, sha1 trailer
620	Advertise.ts # v0 advertisement builders (upload/receive capability sets, peeled tags)
621	UploadPack.ts # v0 fetch choreography: request parse, negotiation, shallow, closure driver
622	ReceivePack.ts # v0 push choreography: command parse, probe, ingest driver,
623	# connectivity check driver, report-status
624	GzipBody.ts # Content-Encoding: gzip detection + DecompressionStream + 1f8b sniff
625	store/
626	Sql.ts # DDL (§3.1), typed query helpers, chunked-IN utilities, transactionSync wrappers
627	ObjectStore.ts # row/r2/pack placement policy, batched reads, thresholds, staging, r2_key resolution
628	Keys.ts # R2 key scheme (§3.2)
629	Closure.ts # commit BFS (gen-bounded), tree walk, visited set, depth bounds
630	jobs/
631	Import.ts # alarm job: git HTTP client against source, PackParser reuse, status flips
632	Fork.ts # alarm job: parent snapshotRows Stream → batched inserts, r2_key sharing
633	Purge.ts # alarm job: bounded R2 prefix delete loop (fork-count-gated), deleteAll, Registry removal
634	test/
635	fixtures/stack.ts # user-authored Alchemy.Stack over GitService() — one per suite
636	fixtures/packs/*.pack # generated once by real git, checked in (never at test time)
637	codec.test.ts # pure: pkt-line, varints, delta apply, object hashing, tree sort
638	pack.test.ts # pure: parse/write round-trips against checked-in fixture packs
639	GitService.test.ts # REST lifecycle via HttpApiClient against deployed URL
640	GitProtocol.e2e.test.ts # real git CLI against deployed URL (§9)
641	GitProtocol.wire.test.ts # isomorphic-git in-process client: malformed packs, forced thin REF_DELTA,
642	# ERR-pkt / band-3 assertions
643	examples/git-service/
644	alchemy.run.ts # user-authored Alchemy.Stack yielding GitService()
645	package.json # alchemy deploy | dev | destroy | logs | tail
646	```
647
648	Conventions honored: no `Input<T>` in props; Effect 4 APIs only (`Effect.result` + `Result.*`, `Schema.TaggedErrorClass`, `Schema.Literals`, `Effect.fn`); no `async/await`/raw Node IO in handlers (Effect platform services; sync CPU crypto/zlib wrapped in `Effect.sync` at module boundaries, not per-byte); file-based `main: import.meta.url` (never inline `script` — unsupported in dev); `@/` path alias; test fixtures own their directory; DO RPC methods avoid the reserved name `delete`; JSDoc `@section`/`@example` on exported classes for `bun generate:api-reference` (synthetic provider dir like better-auth).
649
650	---
651
652	## 7. Core codec plan: write it in Effect; the zlib boundary is `node:zlib`
653
654	**Reuse verdict: write our own; port algorithms, import nothing (at runtime).** isomorphic-git is Promise-based, filesystem-shaped, and not streaming-safe under 128 MB (it *is* used as a **devDependency test client** for wire-level negative tests); wasm-git drags libgit2 through WASM for a problem that is ~1500 lines of well-specified codec; ripgit's Rust is the right recipe in the wrong language. The wire formats are small, frozen, and exhaustively documented — the risk is in choreography, not codecs, and choreography must be ours anyway. Everything is pure functions / Stream transforms in `src/Git/Protocol/` with zero platform imports, so the entire codec layer unit-tests in plain bun without a deploy. Port with attribution: `applyDelta` (~80 lines) and tree sort-order from isomorphic-git; OFS-delta offset codec cross-checked against ripgit and gitformat-pack.
655
656	**The zlib boundary (the one platform-sensitive decision), per empirical workerd verification (workerd 1.20260801.1):**
657
658	| Need | Tool | Why |
659	|---|---|---|
660	| Pack entry parsing (unknown compressed span) | `node:zlib` `createInflate()` per entry | `end` fires at `Z_STREAM_END` mid-`write` without `.end()`, and `inflate.bytesWritten` reports the **exact compressed bytes consumed** — the only way to find the next entry. Native, one instance per object, cheap. `DecompressionStream` is disqualified: errors on trailing bytes, doesn't report consumption. |
661	| One-shot inflate of an exactly-known span (blob reads via index) | `zlib.inflateSync` or `DecompressionStream("deflate")` | Spans are exact so strictness is fine. |
662	| Request-body gunzip | `DecompressionStream("gzip")` | Single well-formed stream; pipe-through. Branch on `Content-Encoding: gzip`, sniff `1f 8b` defensively. |
663	| Compression at rest | `zlib.deflateSync(content)` (level 6) | Only runs for delta-resolved objects — the ingest fast path stores the pack's compressed span **verbatim** for non-delta entries. |
664	| Hashing | `node:crypto` `createHash("sha1")` incremental | Pack trailers and large objects; loose ids hash `"<type> <size>\0" + content`. |
665
666	`Protocol/PackParser.ts` (via a thin `Zlib` helper) is the **only** module allowed to touch `node:zlib`. No pako, no fflate.
667
668	Codec inventory: pkt-line reader/writer (65516 payload cap, flush/delim/ERR), sideband mux, type/size varint (little-endian 7-bit), OFS_DELTA offset (big-endian 7-bit **with the +1 bias per continuation** — the classic bug), delta instruction stream (copy/insert, `size==0 ⇒ 0x10000`), tree entry encode/parse with the `foo` < `foo/` directory sort quirk, commit/tag header parsing (continuation lines, gpgsig), pack header/trailer. Each has table-driven unit tests plus round-trip tests against real-git-generated fixture packs checked into `test/fixtures/packs/` (normal, thin, ofs-heavy, ref-delta, big-blob, empty), with `git index-pack --strict` / `git fsck` as the oracle for writer output.
669
670	---
671
672	## 8. Auth model: nothing inside the engine
673
674	The engine holds no credentials, no users, and no policy. Who may call
675	which route is decided by the `HttpRouter` middleware the user puts in front
676	of the routes, before the engine sees a request; the engine's routes
677	declare no middleware and no auth errors.
678
679	Every endpoint in `Api/*.ts` is an Effect `HttpApiEndpoint`.
680	The user defines their own API and composes its routes beside Git. `Git.ApiLive` is a native HttpRouter route layer,
681	merged beside `HttpApiBuilder.layer(AppApi)` by the application. `Git.ApiHandlersLive`
682	supplies the shared implementation. `Git.GroupsLive` exposes default native
683	group layers for applications that override a Git handler with
684	`HttpApiBuilder.group(Git.Api, ...)` and ordinary `Layer.mergeAll`.
685
686	The application owns authentication and the HTTP server. Its
687	`HttpRouter.middleware` layer applies to whichever public route layers it
688	chooses. Schema composition (`addHttpApi`) is independent of route registration.
689	There is no Git server wrapper and no application API argument.
690
691	The application can instead build its own API from the exported groups and
692	native `HttpApiMiddleware`. Every group must be built against that actual API.
693	Raw handlers still receive middleware-provided services.
694
695	`Git.Engine` provides transport-independent repository operations. The HTTP
696	adapter `ReceivePack.decode` reads bounded commands and retains the streaming
697	body. The handler authorizes the proposed refs, calls `preparePush`, optionally
698	reads staged objects to validate content, then calls `commitPush`. These are
699	scoped, single-use operations. Scope exit aborts uncommitted staging; cleanup
700	checks durable commit state before deleting spilled bytes. Git report-status
701	encoding lives in the HTTP adapter. There is no policy callback service.
702
703	Application policy functions use ordinary typed effects, including the request
704	user and database services. REST ref updates/removals use `prepareRefUpdate` /
705	`prepareRefRemoval`; merges use `prepareMerge`, pinning both inspected tips.
706	Applications call the same policies on every exposed write path, including
707	GitHub merges. `PushDenied` is encoded as a Git rejection or a typed JSON 403.
708
709	The push pipeline's internal hash route is not part of `Git.Api`: it is
710	`Git.InternalApi`, registered by `Git.InternalApiLive` beside public routes and
711	outside the user's middleware; it authenticates with `InternalSecret`, a
712	deploy-time `Alchemy.Random` value no user holds.
713
714	Tokens, users, and sessions are an implementation outside the engine:
715	the example uses Better Auth API keys (`@better-auth/api-key`) as the
716	git password and Better Auth sessions for browsers, both resolved in its
717	one middleware; anonymous callers read public repositories (the
718	middleware asks `Git.RegistryStore`). The test fixtures do the same with
719	two shared secrets. There is no tokens table and no admin key.
720
721	---
722
723	## 9. v1 test plan sketch
724
725	**Tier 1 — pure codec (no cloud, milliseconds):** `codec.test.ts` / `pack.test.ts` in plain alchemy-test. Table-driven pkt-line/varint/delta/tree-sort cases; parse→write→parse round-trips on checked-in fixture packs (generated once by real `git pack-objects`, committed — never generated at test time); thin-pack resolution against a fixture object store; the OFS +1-bias multi-byte offset case; the `0x10000` copy-size case; tree-sort fsck cases; zlib-boundary accounting (concatenated streams, trailing junk, mid-chunk `Z_STREAM_END`); PackWriter output accepted by `git index-pack --strict`.
726
727	**Tier 2 — deployed REST (`GitService.test.ts`):** standard harness —
728
729	```ts
730	const { test, beforeAll, afterAll, deploy, destroy } = Test.make({
731	providers: Cloudflare.providers(), state: Cloudflare.state() });
732	const stack = beforeAll(deploy(Stack)); // fixtures/stack.ts
733	afterAll.skipIf(!!process.env.NO_DESTROY)(destroy(Stack));
734	```
735
736	`HttpApiClient.make(GitApi, { baseUrl: url })` driving: create repo (admin key) → `RepoCreated` carries a working bootstrap token → 409 on duplicate → PATCH description/readOnly → mint read/write tokens → token list is masked → refs empty on fresh repo → REST ref update with CAS (`RefConflict` on stale `expectedOid`, `ObjectNotFound` on bogus oid, `ReadOnlyRepo` when flagged) → typed 404s decode as the tagged classes → token TTL expiry (60 s + bounded `Effect.repeat` poll) and revocation → 401/403 after → delete → `status: "deleting"` → eventual 404. First requests retried through edge propagation (`Schedule.spaced("1500 millis"), times: 40`). Deterministic repo names; pre-test purge of leftovers (delete-if-exists before create).
737
738	**Tier 3 — real git CLI e2e (`GitProtocol.e2e.test.ts`, the money suite):** deploys the same stack; drives the actual `git` binary via the Effect platform `Command` service against `https://x:${token}@${host}/${owner}/${repo}.git`, work-trees under `FileSystem.makeTempDirectory` (Effect platform services only — no `node:fs`). Every step wrapped in bounded retries/timeouts per the speed doctrine (per-test ≤ 120 s):
739
740	1. **Empty-repo clone** — clone succeeds, warns empty, correct default branch (symref/unborn behavior).
741	2. **First push** — init, 3 commits (incl. an executable file, a symlink, a subdirectory), `git push` → `ok` report; REST `refs` + `commits/:oid` + `trees/:oid` + `/file` agree with `git rev-parse`/`git ls-tree`/`git show` byte-for-byte.
742	3. **Clone-back + fsck** — fresh clone, `git fsck --strict` clean, `git log` matches. (Transitively proves advertisement, negotiation, pack emission, object round-tripping.)
743	4. **Incremental fetch** — push 2 more commits from clone A; `git fetch` in clone B; fast-forward merge; content identical (asserts the haves/ACK path, not just re-clone).
744	5. **CAS / force push** — non-FF push rejected (`ng fetch first`), then `push --force` succeeds; fetch-after-force-push works in the other clone. **CAS race**: two clones push conflicting updates concurrently → exactly one `ok`, one `ng`; `--atomic` batch is all-or-nothing.
745	6. **Concurrent push serialization** — two parallel pushes to one repo → both eventually succeed (semaphore wait) or the second sees 503 + retry; never a corrupted repo.
746	7. **Branch + tag lifecycle** — push new branch, push **annotated** tag (round-trips as a tag object, peeled `^{}` in advertisement — ripgit's silent-drop bug is a named regression test), `git push --delete` both.
747	8. **Shallow** — `git clone --depth 1` succeeds; `git fetch --deepen 1` extends it.
748	9. **Large blob** — commit a 3 MiB random binary (deterministic seed), push, clone back, byte-identical (exercises `location='r2'` both directions).
749	10. **Big-ish push** (`skipIf(process.env.FAST)`) — ~2 000 deterministic commits (fixture script, cached tarball) push and clone within timeout (exercises thin-pack ingest with deep delta chains).
750	11. **readOnly** — flag repo readOnly, push → per-ref `ng`; unflag, push succeeds.
751	12. **Auth matrix** — read token clones but push → 403/`ng`; garbage token → 401 + git auth-retry stderr; token in remote URL works.
752	13. **Fork + import** — fork, poll `status` via `Effect.repeat` until `ready`, clone fork, histories identical, push divergent commit to fork → parent unaffected; **delete parent → fork still clones** (R2 fork-retention pin). Import from a sibling repo in the same deployment (avoids external flake), verify refs; optionally a small public repo depth-limited.
753	14. **Cleanup pin** — `stack.destroy()` at suite end; out-of-band distilled verification that the R2 prefix and Registry rows are gone (leak = provider bug by definition).
754
755	**Tier 3b — wire-level negative tests (`GitProtocol.wire.test.ts`):** isomorphic-git as an in-process HTTP client for cases needing byte-level control without shelling out: truncated pack → checksum `ng`; forced thin REF_DELTA against known base; pack referencing a missing tree → connectivity-check `ng`; oversize pack → clean in-band 50 MiB rejection; `ERR` pkt / band-3 assertions.
756
757	Per house rules: `bun run test test/... --profile testing` wrapped in `timeout 240`; `Effect.repeat` with bounded schedules for all polling (never `while(Date.now())`); deterministic names (never `Date.now()`); `NO_DESTROY=1` for local iteration.
758
759	---
760
761	## 10. Cut list & v2 seams
762
763	Every cut, its consequence, and the seam that makes it additive (seams marked **▲** are already load-bearing columns/interfaces in v1's shape):
764
765	| Cut | Consequence | Upgrade path |
766	|---|---|---|
767	| Protocol v2 | None visible — clients fall back to v0 silently | Additive: branch on `Git-Protocol: version=2` in `info/refs`, add `ls-refs` + `fetch` command handlers over the same `Closure` + `PackWriter` code. Advertisement/choreography are per-version files — a new file, not a rewrite. |
768	| ▲ Pack compaction / R2 packs (v1.1) | DO storage cost 13× R2 until it lands; 1 M-object manifest cap | `objects.location='pack'` + `pack_id`/`pack_offset` columns exist; emitter dispatches on location; alarm scaffolding present. Clone fast-path then streams whole entry-regions from R2 (B's verbatim-region trick — OFS offsets are entry-relative, so contiguous regions copy safely). |
769	| ▲ Streaming push > 50 MiB | Large initial pushes rejected (clean in-band report) | `PackParser` is written against `RandomAccess`; swap the memory buffer for an R2-multipart tee + ranged-read re-resolution. Plan body caps (100 MB Pro) remain the outer bound. |
770	| `thin-pack` / `ofs-delta` in **served** packs | Fatter fetches (~1.3–2×, no cross-object compression) | Delta-aware writer, or serve compaction packs (which may carry deltas) verbatim for clones. |
771	| `filter` (partial clone) | v0 clients warn and ignore; `--filter` unavailable | Advertise `filter`; `blob:none`/`blob:limit` are manifest predicates — the walk already knows types/sizes. |
772	| `include-tag` | Clients cope (fetch tags explicitly) | Tag-closure step in `Closure`. |
773	| `deepen-since` / `deepen-not` / `deepen-relative` (`git fetch --deepen`) | Rare flags fail (absolute `--depth <n>` deepening works) | ▲ `commits.commit_time` already stored; add walk predicates. |
774	| Multi-round minimal negotiation | Slightly fat incremental fetches (boundary snapshot redundancy) | ▲ Real common-ancestor negotiation over `commits`/`commit_parents` + `gen` — all data present. |
775	| `packfile-uris` | — | Natural once compacted packs exist: presigned R2 URLs for whole packs. |
776	| push-cert, push-options | Not advertised ⇒ clients don't send them | Add protocol parsing and expose decoded data to application handlers. |
777	| Force-push / branch protection | Any write token can force-push | Policy table consulted inside the existing CAS `transactionSync` — pure addition. |
778	| Object-level GC | Unreachable objects persist (storage is the cheap resource) | Reachability sweep in the GC alarm; fork-aware deletion needs the refcount ledger below. |
779	| Shared-pack refcount ledger | Fork-retention rule is coarse (`fork_count > 0` retains the whole prefix forever) | Registry-maintained per-key refcounts on fork/delete → precise shared-object GC. |
780	| Dumb HTTP | None (modern git never falls back when smart content-types are correct) | Never. |
781	| SHA-256 repos | `object-format=sha1` advertised; mismatch rejected with `ERR` | ▲ `Oid` is branded; widen schema + dual hashing. |
782	| LFS | Large files ride normal objects up to the 64 MiB cap | LFS Batch API as an independent REST group + R2 streaming (chr33s reference); keys `{repoId}/lfs/<sha256>` reserved. |
783	| Registry sharding / global listing at scale | Singleton Registry serializes control-plane writes | ▲ `getByName("registry:" + owner)` — RPC interface unchanged. |
784	| Users / orgs / OAuth / public repos | Single admin key + capability tokens only | ▲ `Credentials` Context service boundary; edge-auth worker via service binding. |
785	| Cheap forks / big-repo import | Fork copies parent row bytes; import capped at 50 MiB pack | Falls out of compaction (bytes in shared R2 packs) + streaming ingest — both seams above. |
786	| PRs / issues / review | — | New tables in the Repo DO (it already holds the commit graph; merge-bases are `commit_parents` BFS), new HttpApi groups; the REST layer grows by group addition, not restructuring. |
787
788	---
789
790	*Implementable from this document plus the repo: start with `src/Git/Protocol/` (Tier-1 tests green offline), then `Store/`, then `RepoObject.ts` choreography, then the Worker/API mount, then the e2e suite. `packages/alchemy/src/Cloudflare/StateStore/Api.ts` is the mounting reference; `test/Cloudflare/Workers/fixtures/http-api/` is the Worker+DO+R2 reference; `packages/better-auth` is the packaging reference.*
791	---
792
793	# Part II — v2: scaling to enormous repositories
794
795	v1 is deliberately one Durable Object per repo holding **everything**: refs,
796	the object index, and the object bytes. That buys transactional ref CAS for
797	free and is provably correct (Part I), but it caps out on two axes:
798
799	- **Storage** — DO SQLite is 10 GB, so bytes-in-rows caps a repo at ~10 GB.
800	- **Bandwidth** — every clone byte transits one single-threaded DO in one
801	datacenter. No horizontal read scaling, no geographic spread.
802
803	The v2 insight: **a git repo is 99.9% immutable content-addressed bytes and
804	0.1% mutable transactional state.** Immutable content-addressed data needs
805	*zero* coordination, so it should never live behind a coordination point.
806	Sharding the DO would be solving a problem we should not have. Instead,
807	split the planes.
808
809	## 11. The three planes
810
811	| Plane | Home | Holds | Scales by |
812	|---|---|---|---|
813	| **Refs** (mutable) | Repo DO | refs, tokens, config, push staging, commit graph | Nothing needed — kilobytes; the only thing that requires serialized CAS |
814	| **Objects** (immutable) | R2 | content-addressed packs + pack indexes + clone bundles | R2 (unbounded, immutable keys, no per-key write contention) |
815	| **Serving** (stateless) | Workers + Cache API | advertisement, negotiation, pack streaming | Per-PoP: every datacenter serves independently |
816
817	The DO stays the authority for "what is the current tip", which is the only
818	question that needs an authority. Everything else is content-addressed and
819	therefore cacheable, shareable, and replicable without coordination.
820
821	### 11.1 Why this is the right shape
822
823	This is the same refs-in-database + objects-in-blob-store architecture that
824	GitHub (Spokes), GitLab (Gitaly + object storage), and AWS CodeCommit
825	(DynamoDB + S3) converged on, expressed in Cloudflare primitives. The
826	difference from v1 is not "more machines" — it is *removing the byte path
827	from the coordination point*.
828
829	### 11.2 Request flows after v2
830
831	```
832	clone (warm) client → Worker → Cache API hit (DO: 0 calls, R2: 0 gets)
833	clone (cold) client → Worker → DO.planClone (refs only, ~1 KB)
834	→ R2 GET bundle → stream to client (DO: 1 tiny RPC)
835	fetch (incr.) client → Worker → DO.planFetch (negotiate on the
836	commit graph) → R2 ranged reads (DO: 1 tiny RPC)
837	push client → Worker → DO (ingest + CAS, unchanged) (serialized, by design)
838	```
839
840	## 12. Object plane: packs, indexes, bundles
841
842	### 12.1 Compaction (loose rows → R2 packs)
843
844	An alarm fires when `SUM(zsize) WHERE location='row'` exceeds **1 GB** or the
845	live loose count exceeds **50 000**. It streams a no-delta pack of the
846	oldest N objects to R2 via multipart (5 MiB parts), then flips those rows to
847	`location='pack'` with `pack_id`/`pack_offset` in one `transactionSync` and
848	NULLs their `zdata`.
849
850	Because objects are stored **pre-deflated** (v1's decision, made for CPU
851	reasons), a pack is literally `varint header + zdata` concatenated — so
852	compaction is a copy, not a recompression, and `pack_offset` points at a
853	byte span that can be served by an R2 ranged read verbatim.
854
855	Keys are content-addressed and immutable:
856
857	```
858	{repoId}/packs/pack-{sha1}.pack compacted objects (immutable)
859	{repoId}/bundles/bundle-{refsHash}.pack full-clone bundle (immutable)
860	{repoId}/objects/{oid} oversize loose objects (v1, unchanged)
861	```
862
863	**Only blobs are packed (DESIGN §22).** `Compact.ts` selects `type = 3`
864	rows exclusively; commits, trees and tags stay `location='row'` in SQLite
865	for the life of the repo, and the REST `ObjectStats` reports them as
866	`resident` (distinct from `loose`, which is now *blobs awaiting
867	compaction*). The reason is the fetch path: the want→have closure is a
868	tree walk, and a tree walk over pack storage is the worst access pattern
869	a window cache can see — trees are tiny and, packed in oid order, land in
870	every 4 MiB window of every pack. On a 15.6k-object repo that walk alone
871	cost 12 s of TTFB after batching (27 s before). Over rows it is a handful
872	of batched `SELECT ... IN` calls. Commits and trees are a few percent of a
873	repo's bytes, so keeping them local buys sub-second negotiation for every
874	incremental fetch at negligible SQLite cost.
875
876	### 12.2 Clone bundles
877
878	A clone is, on the wire, exactly one pack containing the closure of the
879	advertised refs. v1 recomputes that pack per clone. A **bundle** computes it
880	once, after pushes settle (debounced alarm), and stores it under a key
881	derived from the ref snapshot it covers (`refsHash`).
882
883	Serving a clone then becomes: match the request's wants against the current
884	advertisement, and if they agree, stream the bundle bytes. No closure walk,
885	no per-object reads, no compression — and the bytes are immutable, so the
886	Cache API can serve them from every PoP.
887
888	Staleness is handled in three escalating tiers:
889
890	1. **Exact match** (v2.0) — bundle covers exactly the current refs → stream
891	it; otherwise fall back to dynamic assembly. Hot repos with bursty clone
892	traffic (CI fleets cloning the same tip) hit this nearly always.
893	2. **`packfile-uris`** (v2.1, needs protocol v2) — the server hands the
894	client a CDN URL for the big pack and sends only the remainder in-band.
895	The bundle no longer has to be current, only *mostly* current.
896	3. **`bundle-uri`** (v2.2, git ≥ 2.38) — advertise a base bundle plus
897	incrementals with `creationToken`s; the client fetches them all from
898	cache and then does an ordinary incremental fetch. Server work per clone
899	collapses to a tiny negotiation.
900
901	Bundles are derived data: GC keeps the newest base plus K incrementals and
902	deletes older ones lazily. Because keys are content-addressed, a new bundle
903	never overwrites an in-flight one.
904
905	### 12.3 The object index at scale
906
907	If the index itself outgrows the DO (~10M+ objects), the answer is again not
908	a shard: the **pack index files in R2 are the index** (binary-searchable by
909	oid), and the oid's own hash prefix is a perfectly uniform partition key if
910	they ever need splitting. Content addressing *is* the trie — there is no
911	tree structure to invent.
912
913	## 13. Serving plane: Workers + Cache API
914
915	- **Advertisement caching.** `GET /info/refs` output changes only on push,
916	so it is cached under a key derived from the repo id + refs hash and
917	purged on push. A clone storm then costs the DO ~one call *per push*
918	rather than one per clone.
919	- **Worker-side clone fast path.** The Worker asks the DO for a *plan*
920	(`planClone`: refs snapshot + bundle key, ~1 KB) and, when the plan says
921	"bundle", streams R2 → client itself, framing sideband as it goes. The DO
922	never touches a pack byte.
923	- **Dynamic fetches** stay proxied to the DO in v2.0 (they need the commit
924	graph); v2.1 moves negotiation to the Worker over a cached commit-graph
925	file, leaving the DO purely as the ref authority.
926
927	## 14. Scaling model (napkin math)
928
929	Everything is per-repo; repos are independent DOs, so fleet capacity is the
930	sum. These are estimates derived from platform limits and architecture, not
931	measurements — §15 is the plan to replace them with numbers.
932
933	| Dimension | v1 (as built) | v2 (this design) | Bound by |
934	|---|---|---|---|
935	| Clone bandwidth (per repo) | ~50–100 MB/s aggregate | edge line rate; Tbps-class aggregate | v1: one DO streams every byte. v2: immutable bytes in cache/R2 |
936	| Clone TPS (per repo) | ~50–200/s | 10k+/s | v2: advertisement cached; DO called ~once per push |
937	| Incremental fetch | ~1–5 s CPU per 1M-object closure | ms for recent tips | commit graph + (v2.1) bitmaps |
938	| Push TPS (per repo) | ~3–10/s | ~5–20/s | **Inherent**: a ref is a serialization point in git's own semantics |
939	| Repo size | ~10 GB | unbounded (R2) | DO SQLite cap lifted off the byte path |
940	| Push size | 50 MiB | unbounded | `RandomAccess` seam → R2 multipart tee |
941
942	**v2 makes reads scale like a CDN and leaves writes scaling like git** —
943	which is the correct place to land, because git's semantics cap per-ref
944	write concurrency regardless of who hosts it.
945
946	## 15. Remaining bottlenecks (ranked) and how each is measured
947
948	1. **Cold closure computation on enormous repos.** A 10M-object visited set
949	as a JS `Set` is ~400 MB–1 GB, over the Worker's 128 MB. Fix is git's own:
950	binary commit-graph + **reachability bitmaps** (EWAH), turning a 10M-node
951	walk into bitmap ANDs over a few MB. This is the hardest real engineering
952	on the path to enormous repos.
953	2. ~~**Subrequest limit (1000) vs. fragmented packs.**~~ **Addressed**
954	(§16.2): pack reads are served from window-aligned slabs, so a fetch
955	costs a handful of R2 GETs rather than one per object (68.2 s → 1.16 s
956	on a 1202-object pack). Remaining work is pack *ordering* — laying out
957	compaction output so a typical fetch touches few windows — which is an
958	optimization now rather than a cliff.
959	3. **Per-repo push serialization.** Inherent (see above); measured, not
960	fixed.
961	4. **Registry singleton** — ~500 control-plane ops/s globally. Resolve
962	caching shields reads; `getByName("registry:" + owner)` shards writes.
963	5. **First-byte latency for distant clients** — the refs DO lives in one
964	datacenter. Cached advertisements hide most of it; location hints place
965	hot repos near their users.
966	6. **R2 write throughput during compaction/bundling** — a large repack is
967	minutes of alarm work; must use the bounded-per-run + re-arm pattern the
968	purge job already uses (15-minute alarm budget).
969
970	Each gets a benchmark in `test/GitBench.e2e.test.ts` (§16) so the table
971	above becomes measured numbers rather than estimates.
972
973	## 16. Measured (test/GitBench.e2e.test.ts, deployed, SCALE=1)
974
975	Run: `bun run test test/GitBench.e2e.test.ts --profile testing`
976	(`BENCH_SCALE=4` for 4× workloads). Each case asserts only correctness and
977	prints a line, so a slow datacenter minute can never fail the suite.
978
979	### 16.1 End-to-end (via the `git` CLI, so client cost is included)
980
981	| Measurement | Result | Reading |
982	|---|---|---|
983	| push 300 commits | 2.0 s (147 commits/s) | one push, ingest-dominated |
984	| clone — dynamic closure walk | 1.17 s | v1 path: walk + per-object reads |
985	| **clone — R2 bundle** | **0.44 s** | **2.7× faster** |
986	| clone — fully packed repo (ranged R2 reads) | 0.60 s | compaction costs ~35% vs bundle, still beats dynamic |
987	| incremental fetch (1 commit) | 477 ms | 2 round trips + negotiation |
988	| 8 concurrent clones, 256 KiB each | 9.9–10.7 clones/s | bound by `git` process spawn + TLS, **not** by the server |
989	| 10 serial pushes (same branch) | 4.6 pushes/s | the predicted per-ref serialization floor |
990	| 4 concurrent pushes (distinct branches) | 9.2 pushes/s | DO semaphore serializes ingest, by design |
991	| 10 concurrent repo creates | 7.9–12 creates/s | Registry singleton |
992	| repo GET (resolve-cached) | 25 ms | in-isolate cache hit |
993	| repo LIST, 13 rows → 14 rows | **416 ms → 27 ms** | **15×** after fixing the N+1 (below) |
994	| compact 202 loose objects → R2 | 0.9 s | ~220 objects/s |
995
996	### 16.2 Server capacity — the v2 claim, measured
997
998	Driving `git-upload-pack` over raw HTTP removes the client from the
999	measurement. Comparing the bundle path against the dynamic closure walk
1000	(forced by sending a `have` the server does not know, which disqualifies
1001	the bundle while producing the identical pack) is like-for-like:
1002
1003	| Concurrency | dynamic closure walk (v1 path) | R2 bundle via Worker (v2 path) |
1004	|---|---|---|
1005	| 1 | 4.7 clones/s, 2.4 MiB/s | 5.4 clones/s, 2.7 MiB/s |
1006	| 8 | 17.2 clones/s, 8.7 MiB/s | 17.9 clones/s, 9.1 MiB/s |
1007	| **32** | **20.2 clones/s, 10.2 MiB/s** | **62.1 clones/s, 31.5 MiB/s** |
1008
1009	Dynamic fetch over a **compacted** repo (1202 objects in one R2 pack), the
1010	case bottleneck #2 warned about:
1011
1012	| | |
1013	|---|---|
1014	| one ranged GET per object (naive) | **68.2 s** for a 53 KiB fetch |
1015	| coalesced window reads | **1.16 s** (59×) |
1016
1017	Reading a pack object-by-object costs one serialized R2 round trip each.
1018	Window-aligned slabs (4 MiB, ≤6 retained) mean touching any object fetches
1019	its whole neighbourhood once, so a small pack collapses to a single GET —
1020	and the Workers subrequest budget stops being a correctness cliff.
1021
1022	This is the design's central claim, measured. The dynamic path **saturates
1023	at ~20 clones/s** — one Durable Object's single-threaded ceiling, and more
1024	load does not improve it — while the bundle path keeps scaling with
1025	concurrency (3.1× at 32-way, still climbing). Serialized planes flatten;
1026	immutable content-addressed bytes served by Workers do not.
1027
1028	### 16.3 Real world: this repository (test/GitRealWorld.e2e.test.ts)
1029
1030	The strongest test we have is pushing the alchemy monorepo itself — a
1031	depth-1 tree re-committed as one root commit, which keeps the real object
1032	count and byte volume while staying fully connected (pushing from a shallow
1033	clone would reference parents the server does not have, which receive-pack
1034	correctly refuses):
1035
1036	| Step | Result |
1037	|---|---|
1038	| push 12,315 files / 13,699 objects / 38.1 MiB | 20.6 s |
1039	| clone back, loose objects | 19.5 s |
1040	| **clone back after compaction (R2 packs)** | **3.4 s — 5.7×** |
1041	| `git fsck --strict`, HEAD and tree oid equality | identical |
1042	| incremental push on top | works |
1043
1044	Compaction making clones **5.7× faster** on a real repo is the v2 storage
1045	plane paying off end to end: the same objects, read as a few large ranged
1046	GETs out of an immutable pack instead of thousands of row reads.
1047
1048	Verification note: the suite deliberately does **not** `diff -r` the two
1049	working trees. Tree-oid equality is the stronger and correct assertion —
1050	identical tree oids mean every path, mode and blob matches — whereas a
1051	directory diff also compares things git does not track (empty directories,
1052	submodule mount points), which legitimately differ between a working tree
1053	and a fresh clone of it.
1054
1055	### 16.4 Push size: the cap is gone, the large path needs a cursor
1056
1057	A push larger than the in-memory threshold is **no longer rejected**. The
1058	body is parked in R2 and parsed back through `Store/PackSource.ts`, an
1059	R2-backed `RandomAccess` with window-coalesced reads — the seam
1060	`PackParser` was written against from the start. The DO's ingest now runs
1061	on `PackParser` (which the pack fixtures cover directly) rather than its
1062	own buffer-only copy.
1063
1064	Getting there took one real fix. `PackParser` read **the entire remainder
1065	of the pack for every entry** (`source.read(offset, dataEnd - offset)`).
1066	On a buffer that is free — a subarray view — which is why nobody noticed;
1067	against R2 it copies ~19 MiB per entry, so a 13.7k-object pack moved
1068	hundreds of gigabytes and blew a 600 s timeout. Entries are a few KiB, so
1069	the parser now reads a bounded 512 KiB window and grows it only for the
1070	rare object whose compressed stream runs past it.
1071
1072	| 38 MiB alchemy push | before | after |
1073	|---|---|---|
1074	| via R2 (streamed ingest) | >600 s (timeout) | **44.4 s** |
1075	| in memory | 20.6 s | 20.6 s |
1076
1077	So the R2 path costs about **2×** memory rather than 30×, and the
1078	threshold ships at 32 MiB — comfortably inside a 128 MB isolate, covering
1079	the overwhelming majority of pushes at full speed, with anything larger
1080	merely slower instead of refused.
1081
1082	### 16.5 Public repos — anonymous read access
1083
1084	Repos carry a `public` flag (GitHub's model). Tokenless callers are
1085	`{ kind: "anonymous" }` in `CallerAuth` and get exactly one grant: `read`
1086	on repos whose flag is set — REST reads, raw file/blob routes, the
1087	advertisement, and `upload-pack` (tokenless `git clone`). Writes, token
1088	management, and everything on private repos still require a token;
1089	anonymous wire requests to private repos answer `401 + WWW-Authenticate`
1090	so git prompts. Listing is visibility-filtered: the admin key sees all
1091	rows, everyone else sees `is_public = 1` only (denormalized into the
1092	Registry, like the other list columns). Enforcement stays in the Repo DO
1093	(`authorize`/`wireAuth`) — the Worker middleware only parses credentials,
1094	and a missing `Authorization` header now parses to anonymous instead of
1095	401.
1096
1097	### 16.5 The push cap was still there — request buffering
1098
1099	Pushing the **full alchemy history** (~100 MiB pack) found the residual
1100	cap: the DO read the request with `request.arrayBuffer`, materializing the
1101	whole body inside the 128 MB isolate before any parsing — OOM, 500, after
1102	~7 s. Everything downstream (R2 parking, windowed parsing) was already
1103	size-independent; the *receive* wasn't.
1104
1105	The fix is `Store/IncomingBody.ts`: the body is consumed as a stream.
1106	Bodies that finish within `MAX_PACK_BYTES` return as one buffer (the fast
1107	path); the moment the threshold is crossed, everything received and
1108	everything still arriving spills to R2 via **multipart upload** in uniform
1109	8 MiB parts, so peak memory is ~threshold + one part no matter how large
1110	the push. A 1 MiB head is retained for the pkt-line command section, and
1111	the pack is parsed back off R2 through `sliceRandomAccess` + windowed
1112	reads. gzip bodies (git only gzips requests it fully buffered, i.e. small
1113	ones) keep the buffered path; a spilled gzip body is rejected in-band.
1114
1115	Measured, full alchemy `main` (44,051 objects, 67 MiB thin pack):
1116
1117	push: 92 s wall — server ingest 76.8 s (SQL 5.7 s)
1118	clone back: 50.6 s, 152.8 MiB; `git fsck --strict` clean, HEAD identical
1119
1120	Also worth recording: Cloudflare's edge caps request bodies at 100 MB on
1121	most plans, so beyond that git needs its transfer split (`git push` per
1122	ref / partial history first) regardless of what this service does.
1123
1124	### 16.6 Ingest batching, and what it revealed
1125
1126	Staging cost **two statements per object** — an existence probe and an
1127	insert — so a 13.7k-object push ran ~27k statements and ~13.7k transactions
1128	inside one single-threaded Durable Object. Ingest now stages in batches of
1129	256 (or 8 MiB) via a single `transactionSync` with `INSERT OR IGNORE` (no
1130	probe), adopting crashed-push rows with one chunked `UPDATE`, and the
1131	parser hands the sink its already-inflated bytes so commits/trees are not
1132	inflated twice.
1133
1134	38 MiB alchemy push via R2: 44.4s -> 37.4s
1135	clone back: 18.5s -> 16.6s
1136
1137	A 16% win, not the 5x the statement count suggested — which is the useful
1138	result: **transaction commits were not the dominant cost**. What remains is
1139	per-object work that batching cannot remove (inflate, sha1, one statement
1140	each, Effect frames) plus, importantly, *client-side* cost the measurement
1141	does not separate: `git push` also packs 13.7k objects locally and uploads
1142	38 MiB over the tester's uplink. Isolating server ingest needs either
1143	server-side timing or a loopback baseline — until then these numbers are an
1144	upper bound on server cost, not a measurement of it.
1145
1146	The deeper point is that this work is **serial by construction**: one repo
1147	is one Durable Object, and a push is one request to it. No amount of
1148	horizontal Worker/R2 scale touches it. Making a single large push
1149	dramatically faster requires parallelism *within* the repo — see §17.
1150
1151	### 16.7 What the measurements corrected
1152
1153	1. **`repos.list` was O(N) Durable Object wakes** — the handler fanned out
1154	`getRepoMeta` per row, so listing 100 repos woke 100 DOs. **Fixed**:
1155	`default_branch` / `read_only` / `status` are denormalised onto the
1156	registry row (the Repo DO pushes them on change and remains the source
1157	of truth). 416 ms → 27 ms. Live `objects` stats still need the DO, so a
1158	listing reports zeros there and callers who want them read the repo
1159	directly. Note for future schema work: the Registry needed an explicit
1160	`ALTER TABLE` migration, because `CREATE TABLE IF NOT EXISTS` silently
1161	leaves an existing table's columns alone.
1162	2. **Small-clone throughput is round-trip-bound, not bandwidth-bound.** At
1163	256 KiB per clone the git client's process spawn plus two HTTP round
1164	trips dominate, so the bundle win shows up as latency; the bandwidth
1165	ceiling only appears under raw concurrency (§16.2) or on much larger
1166	repos.
1167	3. **Advertisement caching is deliberately NOT implemented.** A TTL cache
1168	on `info/refs` would make "push, then fetch from another machine"
1169	unreliable for the length of the TTL — a real correctness regression for
1170	CI. Doing it properly means an epoch the Worker can read without asking
1171	the DO (Cache API entry purged on push), which is a v2.1 item; the
1172	remaining per-clone DO cost is two small RPCs, and §16.2 shows that is
1173	not the ceiling.
1174
1175	---
1176
1177	# Part III — v3: the maximum-performance DO architecture
1178
1179	## 17. Why v2 still has a floor
1180
1181	v2 made *reads* scale (immutable bytes served by Workers from R2) and left
1182	*writes* where git semantics put them. But the measurements in §16 expose a
1183	floor that no amount of horizontal scale removes:
1184
1185	- one repo = one Durable Object = **one thread**;
1186	- a push is **one request** to it;
1187	- ingest is ~13.7k objects × (inflate + sha1 + one statement + Effect
1188	frames), strictly serial.
1189
1190	Batching transactions bought 16% (§16.5) — proof that the remaining cost is
1191	per-object work, not commit overhead. Making a large push an order of
1192	magnitude faster requires parallelism **inside the repo**.
1193
1194	The enabling observation: a repo holds two kinds of state with completely
1195	different constraints.
1196
1197	| | mutability | ordering | consequence |
1198	|---|---|---|---|
1199	| refs, tokens | mutable | CAS, strictly serialized | must live in one DO |
1200	| objects | **immutable, content-addressed** | **none whatsoever** | can be written anywhere, in any order, in parallel |
1201
1202	Object writes are serialized today only because they share a DO with the
1203	refs. That serialization is *accidental*, not essential.
1204
1205	## 18. Target architecture
1206
1207	| Tier | Instances | Holds | Why here |
1208	|---|---|---|---|
1209	| **Worker** | horizontal | pack parsing (inflate + sha1), fan-out, response assembly | stateless CPU; parsing has no reason to sit in a coordination point |
1210	| **Ref DO** | 1 / repo | refs, tokens, commit graph, push staging state, CAS | the only thing git forces to serialize |
1211	| **Object shard DO** | N / repo, by oid prefix | content-addressed objects (hot tier) | independent writes ⇒ N-way parallel; N × 10 GB ceiling |
1212	| **R2** | — | compacted packs, clone bundles | immutable bulk, ranged reads, no egress |
1213
1214	**Sharding key: the oid itself.** An oid is a uniform hash, so a fixed
1215	2-hex-char prefix distributes perfectly with zero rebalancing. A trie or
1216	dynamic split solves skewed key distributions — a problem content
1217	addressing does not have. Fix `N = 256` at repo creation and never reshard:
1218	256 × 10 GB = 2.5 TB per repo.
1219
1220	**Why fan-out is safe.** Staging across shards is not atomic and does not
1221	need to be: objects are immutable and content-addressed, so a partial write
1222	is inert (worst case some staged rows are GC'd). The only atomic step is
1223	the ref CAS in the ref DO, which runs after the connectivity check passes.
1224	Sharding never touches the consistency-critical path.
1225
1226	**Cost shape.** DO *instances* are not billed; requests, active duration and
1227	storage are. Fan-out means one RPC per shard **touched** (objects batched
1228	per shard), so ~256 requests for a large push (~$0.00004) and far *less*
1229	total active duration, since 256 DOs busy ~50 ms beats one DO busy 20 s.
1230	The real costs are cold-start latency across shards and a wider failure
1231	surface — both mitigated by immutability making every retry free.
1232
1233	## 19. Phases
1234
1235	**Phase 0 — measure honestly (prerequisite). DONE — and it settled the
1236	question.** Repos now report `lastPush` timing measured inside the Durable
1237	Object. For the 38 MiB / 13,701-object alchemy push:
1238
1239	| | |
1240	|---|---|
1241	| **server ingest** | **27.0 s** |
1242	| connectivity check | ~0 ms |
1243	| ref CAS finalize | ~0 ms |
1244	| **server total** | **28.7 s of a 32.3 s wall clock (89%)** |
1245	| client + network | 3.5 s |
1246
1247	So the earlier suspicion that client packing and upload dominated was
1248	**wrong**: the server owns 89% of the time, and effectively all of it is
1249	the per-object ingest loop at **2.10 ms/object**. Connectivity and CAS —
1250	the parts that genuinely must be serialized — are free.
1251
1252	**Phase 0b — where the ingest time actually goes.** Splitting `ingestMs`
1253	settled the next question too:
1254
1255	| | | |
1256	|---|---|---|
1257	| SQL staging | 2.0 s | **9%** |
1258	| CPU (inflate, sha1, parse, Effect frames) | 20.4 s | **91%** |
1259
1260	So **sharding storage addresses only 9%** of push cost. It remains valuable
1261	for the 10 GB → 2.5 TB capacity ceiling, but it is not the throughput fix.
1262	The throughput fix is per-object CPU — and 1.49 ms to inflate ~2.7 KB and
1263	hash it is ~50x the cost of the actual work, i.e. it is overhead, not
1264	computation.
1265
1266	First cut at that overhead: `zlib.createInflate()` built a Node Transform
1267	stream (EventEmitter, buffers) and took an async event-loop round trip **per
1268	object**. Driving the engine synchronously the way Node's own `inflateSync`
1269	does removes both:
1270
1271	CPU 20.4s -> 15.9s per object 1.74ms -> 1.42ms
1272	wall clock 28.1s -> 23.2s
1273
1274	with a hard-won caveat baked into the code: workerd's `node:zlib` shim
1275	returns only the *first chunk* (observed: 525,107 bytes of a 1,337,267-byte
1276	object) and reports a plausible `bytesWritten` alongside it, where Node
1277	loops until done. The fast path therefore validates its output against the
1278	size the pack header already declares and falls back to the stream path on
1279	any mismatch — silent truncation would otherwise corrupt objects. Pack
1280	fixtures alone would not have caught this; only the deployed push did.
1281
1282	Remaining per-object overhead to attack next, in order: `crypto.createHash`
1283	allocated per object (same shape of problem as `createInflate`), Effect
1284	frames in the sink, and window slicing.
1285
1286	That is still the argument for phases 1–2, but with corrected weights: the expensive work
1287	(inflate, sha1, one insert) is *per object* and has no ordering
1288	requirement, yet runs single-threaded because it shares a DO with the refs.
1289	At 2.10 ms/object, 16-way fan-out puts a 13.7k-object push at ~1.7 s.
1290
1291	**Phase 1 — hoist pack parsing to the Worker.** Inflate + sha1 for every
1292	object currently runs on the ref DO's single thread; it is stateless CPU.
1293	Moving it to the Worker frees the DO and is the prerequisite for fan-out
1294	(the Worker must hold resolved objects to route them).
1295
1296	**Phase 2 — shard the object store.** `ObjectShard` DO addressed
1297	`{repoId}:{oidPrefix}`; `ObjectStore` becomes a router that batches by
1298	prefix and writes shards in parallel. Connectivity checks become N parallel
1299	batched queries. Compaction runs per shard, producing per-shard R2 packs.
1300
1301	**Phase 3 — parallel fetch assembly.** A dynamic fetch reads its manifest
1302	from N shards concurrently instead of walking one DO.
1303
1304	**Phase 4 — reachability bitmaps.** With throughput fixed, the remaining
1305	ceiling is closure computation on multi-million-object repos: EWAH bitmaps
1306	over the commit graph replace the visited-set walk.
1307
1308	Expected shape of the win: 13.7k objects at ~1.5 ms serial ≈ 20 s; fanned
1309	across even 16 shards with parsing overlapped in the Worker ≈ 1–2 s.
1310
1311
1312	## 20. GitHub REST v3 compatibility (Tier 1)
1313
1314	`src/GitHubCompat.ts` mounts a `/api/v3/**` facade so GitHub-flavored
1315	tooling talks to the service unmodified:
1316
1317	```sh
1318	export GH_HOST=git.example.com # your deployed host (https)
1319	export GH_ENTERPRISE_TOKEN=gs_... # admin key or repo token
1320	gh api repos/alchemy/alchemy # → GitHub-shaped JSON
1321	gh api repos/alchemy/alchemy/pulls -f state=open
1322	gh api -X POST repos/o/r/pulls -f title=T -f head=topic -f base=main
1323	gh api -X PUT repos/o/r/pulls/1/merge
1324	```
1325
1326	The facade is a pure translation layer over the same Repo-DO RPCs as
1327	`/api/v1` (auth enforcement unchanged, anonymous reads on public repos
1328	included). Covered: `/user`, repo get, branches, commits (list/single with
1329	files), contents, and the full pulls lifecycle. GitHub-isms handled:
1330	`Authorization: token` scheme, merged-PR = `closed`+`merged: true`,
1331	`{ message }` errors, `Link: rel="next"` pagination carrying opaque keyset
1332	cursors (`gh api --paginate` and Octokit follow them verbatim).
1333
1334	Out of scope, deliberately: GraphQL (so `gh pr ...` porcelain does not
1335	work — use `gh api` paths), issues/reviews/comments/checks/search, rename
1336	detection, and additions/deletions counts (always 0; content diffs are
1337	client-side by design). `gh` requires https for enterprise hosts, so the
1338	facade is exercised locally by shape tests replaying gh's exact wire
1339	requests (verified against `GH_DEBUG=api` output) and by `gh` itself only
1340	against https deployments.
1341
1342	## 21. Learnings from Cursor's Continuity (git-at-any-scale)
1343
1344	Cursor's Continuity ([blog](https://cursor.com/blog/git-at-any-scale))
1345	solves the same problem shape — git on object storage with a single
1346	linearization point — with one structural inversion worth studying: for
1347	them **object storage is the source of truth** (a WAL of pushed packs +
1348	a tiny WAL-index object updated by compare-and-swap), and every server
1349	disk is a disposable warm cache rebuilt from the WAL. For us the Durable
1350	Object's SQLite *is* the truth and R2 holds derived artifacts. Their
1351	compute is fungible and their storage is authoritative; ours is the
1352	opposite. What transfers:
1353
1354	**Adopt — the push WAL (durability + provenance).** Today the DO's
1355	SQLite is the only representation of refs/index/graph; the received pack
1356	is exploded into rows and discarded. Appending a small WAL record per
1357	push to the BlobStore — `{ pushId, refTxn (old→new per ref), packKey,
1358	subject, timestamp }` plus the self-contained (fattened) pack — makes
1359	the entire repository reconstructable from blob storage alone: disaster
1360	recovery independent of DO state, byte-level push provenance ("rewind
1361	and fast-forward", audit paired with the auth `subject`), backend
1362	migration, and future read replicas. This composes with the planned
1363	push-epoch redesign (§19 / RFC): the epoch record *is* the WAL entry —
1364	make it durable in the BlobStore instead of a SQLite row and both
1365	designs land in one change. Storage cost is bounded by folding WAL packs
1366	into compaction epochs (Continuity compacts the WAL too).
1367
1368	**Adopt — geometric pack compaction.** `Compact` currently writes one
1369	new pack per run and never merges packs, so pack count grows with
1370	repository age and read fan-out grows with it. Continuity's incremental
1371	geometric compaction (merge packs whenever sizes violate a geometric
1372	progression) bounds pack count logarithmically. Directly applicable to
1373	our Compact job; also the precondition for WAL-pack folding above.
1374
1375	**Adapt — conditional-GET freshness for DO-less reads.** Continuity
1376	replicas serve reads after a ~10 ms conditional GET on the WAL index
1377	(304 = current). Mapped onto us: a tiny `head` object in R2 (ref
1378	snapshot hash + current bundle key), updated at push finalize, would let
1379	**Workers serve advertisements and clone bundles without waking the DO**
1380	— the clone-storm path goes fully DO-less while the DO remains the write
1381	linearization point. The subtlety is commit ordering: SQLite commit is
1382	our truth, so the head object is written after commit and a crash
1383	between the two leaves it stale. Either accept bounded staleness with
1384	repair-on-next-push, or move the commit point into the head-object CAS
1385	itself (the full Continuity inversion — a much bigger change). Start
1386	with the former.
1387
1388	**Adapt — repos as cattle (hibernation).** Continuity garbage-collects
1389	idle replicas and re-materializes from the WAL on access. Our analog:
1390	once a WAL exists, an idle repo's DO can evict object rows (or all
1391	state) and re-materialize on next access — DO storage is the expensive
1392	tier, R2 the cheap one. Matters for the agent-created-repos fleet shape
1393	(their explicit target, and ours).
1394
1395	**Skip — running upstream git.** Their biggest simplification (normal
1396	git repos on NVMe, upstream git does the work) is unavailable on
1397	workerd: no processes, no disk. Our Effect reimplementation is the cost
1398	of running where we run.
1399
1400	**Note — S3 CAS revives the AWS-parity store story.** Continuity's
1401	linearization is a compare-and-swap on an S3 object (`If-Match` /
1402	`If-None-Match` conditional writes) — no DynamoDB, no consensus. That
1403	replaces the RFC §6 assumption that AWS parity needs DynamoDB
1404	transactions for ref CAS: a WAL-index CAS on S3 is sufficient, exactly
1405	as Continuity demonstrates at 120–300 pushes/s (Standard vs Express One
1406	Zone). The wire-plane hosting problem (Lambda's ~6 MB request cap)
1407	remains the real blocker; the store problem is now solved on paper.
1408
1409	Benchmarks worth stealing: sustained small-push throughput per repo
1410	(their 120–300/s number; ours is bounded by one DO — measure it), and
1411	read-scaling under clone storm with the DO-less head-object path.
1412
1413	### §21.2 Push admission, and a bottleneck that wasn't
1414
1415	**Push admission is bounded by memory, not by count.** The Repo DO's
1416	ingest semaphore existed for one reason (§2.1): two concurrent 50 MiB
1417	buffers would exhaust a 128 MB isolate. It is *not* a correctness
1418	device — a DO's input gate only closes across storage awaits, and
1419	receive-pack awaits the network and blob storage, so pushes interleave
1420	regardless; what makes that safe is `staged_push = pushId` scoping plus
1421	a synchronous `transactionSync` finalize. Bounding the count therefore
1422	made a kilobyte push wait out a 50 MiB upload for no reason. It now
1423	reserves permits by buffered ceiling (`min(content-length,
1424	MAX_PACK_BYTES)`, one permit per MiB, budget 64).
1425
1426	| small push (same repo) | latency |
1427	| --- | --- |
1428	| uncontended | 737 ms |
1429	| while a 12 MiB upload is in flight | **802 ms** (+9%) |
1430
1431	Under the old single permit the second push waited for the first to
1432	release, i.e. for the whole upload — by construction, not by measurement.
1433
1434	**What this does NOT fix:** concurrent pushes to *different branches*
1435	still measure ~2.7/s, unchanged. Admission was never the ceiling there;
1436	the DO is one JavaScript thread, so each push's inflate/SHA-1/SQL work
1437	serializes whatever admission allows. Per-repo push throughput remains
1438	the sharding problem of Part III.
1439
1440	**Stale clone bundles are not a bottleneck (measured, hypothesis
1441	withdrawn).** The concern was that any ref moving after a bundle was cut
1442	drops every clone onto a dynamic closure walk until the next re-cut.
1443	Measured in one run against one repo, with the serving plane recorded on
1444	`x-git-served-by` so neither arm can be mistaken for the other:
1445
1446	| clones, concurrency 32 | served by | rate |
1447	| --- | --- | --- |
1448	| fresh bundle | `do-bundle:bundle` | 40.4/s |
1449	| stale bundle | `dynamic` | **35.9/s** |
1450
1451	An 11% cost, not the ~2x the earlier cross-run comparison suggested —
1452	those "dynamic" figures came from requests carrying an unknown `have`,
1453	which forces a *negotiation* walk, a strictly more expensive path than a
1454	plain full clone with no usable bundle. Comparing across runs compared
1455	two different things.
1456
1457	A bundle+delta splice was built and reverted. It is defeated by a
1458	deliberate v1 tradeoff in `Store/Closure.ts`: boundary (common) commits'
1459	tree/blob closures are **not** subtracted, so
1460	`computeClosure(wants, haves = bundle refs)` returns a whole snapshot
1461	rather than a delta. The spliced pack measured exactly 2x the bundle
1462	(1.013 MiB vs 0.507 MiB) at 15.8 clones/s — worse than the dynamic path
1463	it replaced. Making it work needs one of: the bundle recording its own
1464	object set at cut time (`runBundleJob` already holds the manifest) so
1465	the delta is a set subtraction rather than a walk; or protocol v2
1466	`packfile-uris`, which is git's own answer — the client takes the bundle
1467	by URI and runs an ordinary incremental fetch on top. Neither is worth
1468	doing for 11%.
1469
1470	### §21.1 Implemented and measured
1471
1472	Both performance adoptions above are now implemented; measured on a
1473	deployed stack (`test/GitBench.e2e.test.ts --profile testing`, SCALE=1,
1474	same machine, before/after runs ~1 h apart).
1475
1476	**The head snapshot (DO-less anonymous reads).** The Repo DO rewrites a
1477	tiny JSON object (`{repoId}/head`: refs, visibility, current bundle)
1478	after every commit that changes what an anonymous reader can see; the
1479	Worker serves the `info/refs` advertisement and bundle-covered full
1480	clones straight from it. Engagement is proven, not inferred: responses
1481	carry `x-git-served-by: head-snapshot` and the bench asserts it. The
1482	Auth block still decides anonymous access (the Worker asks it with the
1483	snapshot's repo context), a presented credential always goes to the DO,
1484	and `startPurge` deletes the snapshot synchronously so deleted repos
1485	stop serving immediately.
1486
1487	| anonymous reads on a public repo | before | after |
1488	| --- | --- | --- |
1489	| raw full clones, concurrency 32 | 69.4/s · 17.4 MiB/s | **101.0/s · 25.3 MiB/s** |
1490	| same run, authed clones (DO path) | — | 48.1/s · 24.4 MiB/s |
1491	| advertisements, concurrency 32 | 201.2/s | 206.5/s |
1492	| advertisement, serial | — | 87 ms each |
1493
1494	The controlled comparison is the second row: in the same run, on the
1495	same bytes, anonymous DO-less clones sustain **2.1× the throughput of
1496	authed clones that plan through the DO** — cloud noise hits both
1497	equally. The advertisement storm is flat because the *client* is the
1498	bottleneck there (87 ms serial ≈ the bench machine's RTT; 32-way ≈
1499	200/s is its ceiling): the assertion proves the DO left the path, but a
1500	single-machine instrument cannot show the server-side headroom that
1501	buys. The clone rows, which move real bytes, do.
1502
1503	**Geometric pack merging.** `Compact` now restores the geometric
1504	invariant (factor 2) after each run: the smallest violating packs merge
1505	by pure concatenation — our packs store non-delta entries, so a merge
1506	is `header(Σcounts) + bodies + sha1` and one additive offset UPDATE per
1507	source pack, no recompression. Unreferenced source packs grace-delete a
1508	minute later (in-flight ranged reads finish first).
1509
1510	| dynamic fetch, repo compacted in 6 increments | before | after |
1511	| --- | --- | --- |
1512	| ms/fetch (serial ×8, 2.3 MiB pack) | 730 | **300–334** |
1513
1514	**2.2–2.4× faster** reads over the aged repo, reproduced across two
1515	post-change runs. The push path is unchanged by both features (serial
1516	push 1.3–1.5/s across all runs — the per-ref serialization floor §15
1517	already names; that ceiling is v3's problem, not this change's).
1518
1519	## 22. The clone-throughput sweep
1520
1521	The complaint: clones pull 10–20 MB/s. The first question was whose
1522	number that is, and the answer settles what to fix.
1523
1524	### §22.1 Diagnosis (production, `alchemy/alchemy`, 44k objects)
1525
1526	| measurement | value | meaning |
1527	| --- | --- | --- |
1528	| client ← Cloudflare edge, 50 MB static download | **67 MB/s** | the pipe is not the limit |
1529	| bundle path, raw passthrough, 152 MiB | 10–21 MB/s (one sample stalled at 227 KB/s for 264 s) | **our streaming path is the limit** |
1530	| bundle path, sideband | 10.5–17 MB/s | framing is not the difference; the variance is |
1531	| dynamic path | 3.1 MB/s, TTFB 9.2 s | 44k sequential SQLite point reads |
1532	| `info/refs` advertisement (2 refs) | **1.5 s** | per-request planning cost, paid twice per clone |
1533	| repo object stats | loose=44,041 packed=0 r2=10, 152.7 MiB | **never compacted** — `COMPACT_COUNT_THRESHOLD` was 50,000 |
1534	| bytes served vs bytes pushed | 152 MiB served, 70 MiB thin pack pushed | non-delta packs are 2.2x on the wire |
1535
1536	Three independent problems, in order of user impact: the Worker's bundle
1537	streaming pipeline caps single-stream throughput far below the client's
1538	pipe; every wire request pays ~1.5 s before its first byte; and the repo
1539	was serving its fallback path from 44k un-packed SQLite rows.
1540
1541	### §22.2 Root causes
1542
1543	**Streaming (the throughput ceiling).** `serveBundle` piped R2's body
1544	through `Stream.fromReadableStream` → `wrapSideband` (a `Stream.flatMap`
1545	per chunk) → `HttpServerResponse.stream` → `toReadableStream`. Every
1546	R2 chunk crossed an Effect fiber hand-off twice and was copied twice by
1547	the sideband framer (`sidebandFrame` allocated a payload buffer, then
1548	`pktLine` allocated again). At concurrency 4 the aggregate was only
1549	1.5x a single stream — the isolate was CPU-bound on stream machinery,
1550	not waiting on the network.
1551
1552	**TTFB.** `readMeta` ran `SELECT location, COUNT(*), SUM(zsize) FROM
1553	objects GROUP BY location` on **every** request (via `requireMeta` at
1554	the wire dispatch, inside `authorize`, and inside `writeHeadSnapshot`),
1555	a full scan of a `WITHOUT ROWID` table whose leaf pages carry inline
1556	zdata. Only the REST repo-detail response ever reads `meta.objects`.
1557
1558	**Dynamic path.** `packEntryStream` did one `sql.first` and ~6 Effect
1559	stream nodes per object; the closure's tree walk did one SQL round trip
1560	plus one inflate per tree, serially, before the first byte; and the
1561	manifest order (commits → tags → trees → blobs) was uncorrelated with
1562	compacted-pack layout (`ORDER BY oid`), so a 6-window LRU over a large
1563	pack thrashed.
1564
1565	### §22.3 What changed
1566
1567	- **Native passthrough.** `ObjectBody`/`BlobBody` expose the store's
1568	own `ReadableStream` (`readable`); `respondWithPack` pipes it into the
1569	`Response` via `HttpServerResponse.raw`. Raw clones go through an
1570	`IdentityTransformStream` (workerd pipes it without JS touching the
1571	bytes); sideband clones go through `sidebandTransform`, a web
1572	`TransformStream` emitting a 5-byte header + a `subarray` per frame —
1573	no copy, no fiber per chunk. The Effect path remains as the fallback
1574	for stores without a native stream (S3).
1575	- **`requireMeta` is config-only.** The objects aggregate is opt-in
1576	(`requireMetaStats`), paid by `getRepoMeta`/`updateRepoMeta` only.
1577	- **Compaction thresholds** 50k objects / 1 GiB → **4k / 64 MiB**.
1578	- **Batched emission and closure.** `ObjectStore.packEntries` reads
1579	objects 256 per statement, concatenates each batch into one chunk,
1580	and orders pack-resident objects by (pack, offset) so each 4 MiB
1581	window is fetched exactly once; `readContentBatch` + a level-order BFS
1582	read whole tree levels per statement. `packStream` uses the batched
1583	emitter whenever the source offers it.
1584	- `sidebandFrame` builds its frame in one allocation (was two copies).
1585
1586	### §22.4 Measured (36 MiB / 15.6k-object repo, one client, same edge)
1587
1588	All numbers from `GitBench.e2e.test.ts` "single large clone", the same
1589	client machine and edge for every row. "Before" is the row-storage,
1590	Effect-stream code at the start of the sweep; the rest are cumulative.
1591
1592	| path (what real clients use) | before | native pump | +blob-only compaction | +pre-framed twin, per-window emitter |
1593	| --- | --- | --- | --- | --- |
1594	| raw bundle, 1 stream | 20.3 MiB/s | 41.0 | 37.9 | 40.9 (client line rate ≈ 40–70) |
1595	| raw bundle, 4 streams (aggregate) | 29.7 MiB/s | 106.1 | 71.8 | 82.3 |
1596	| **sideband bundle (every `git clone`)** | 15.5 MiB/s | 12.7–15.7 | 15.7 | **32.7** |
1597	| dynamic fetch — TTFB | 1.4 s (rows) / 27.5 s (packed) | 12.0 s | **0.43 s** | 0.56 s |
1598	| dynamic fetch — transfer | 6.2 MiB/s (rows) / 0.2 (packed) | 7.9 | 7.0 | 7.9 |
1599
1600	Run-to-run variance on the client link is large; the best verified run of
1601	the final code (every pack de-framed and trailer-checked): raw 98.4 MiB/s
1602	(0.56 s end to end), sideband **80.2 MiB/s** (0.69 s — a real `git clone`
1603	of the 40 MiB repo), dynamic 9.8 MiB/s with 0.72 s TTFB, four concurrent
1604	raw streams at 93 MiB/s aggregate. Before the sweep the same clone took
1605	about 3 s.
1606
1607	What each step bought, and what it did not:
1608
1609	- **Native pump** (§22.3): the raw path went from an Effect stream to a
1610	platform `pipeTo` and doubled; four streams saturate the client link.
1611	It did NOT move sideband — three different framers (Effect stream,
1612	`TransformStream`, manual 1 MiB batched writes) all landed at ~15 MiB/s.
1613	The constant was never the framing: it is the cost of bundle bytes
1614	crossing into JS at all. Hence the **pre-framed twin**
1615	(`bundleSidebandKey`): the bundle job writes the pack a second time,
1616	already cut into 65515-byte band-1 frames, and sideband serving becomes
1617	a pure pipe plus a 4-byte flush. 2× on the path every real client uses,
1618	for 2× bundle storage.
1619	- **Lowering the compaction thresholds** (50k → 4k objects) exposed a
1620	latent disaster: a fetch over pack storage took 27 s to start and 243 s
1621	to finish — the per-object emitter, in manifest order, missed the
1622	window cache on nearly every object (15.6k × 4 MiB window reads).
1623	Batching + (pack, offset) ordering fixed the transfer (0.2 → 7.9 MiB/s)
1624	but the tree walk still cost 12 s: trees are tiny and, packed in oid
1625	order, touch every window. **Blob-only compaction** removed the walk
1626	from R2 entirely — TTFB 12 s → 0.43 s — at the cost of keeping
1627	commits/trees/tags (a few percent of bytes) as SQLite rows.
1628	- The dynamic transfer rate is now bounded by generation inside the DO
1629	(per-batch SQL + concat for resident objects, one window fetch per
1630	4 MiB of blobs), not by the response path; the per-window emitter and
1631	one-window lookahead did not move it. Clones do not take this path (the
1632	bundle covers them); incremental fetches are small. Left as the next
1633	target.
1634
1635	**Durability finding, same sweep.** A real `git push` of the 36 MiB
1636	clone (a delta-heavy pack, unlike the snapshot the bench pushes) was
1637	rejected: `Durable Object's isolate exceeded its memory limit and was
1638	reset`, mid-`transactionSync`. The ingest's own footprint is bounded
1639	(8 MiB staging batches, 20 MiB resolved-content LRU) — but the whole
1640	pack sat in memory under the 50 MiB spill threshold, and instances of one
1641	DO class share an isolate: the other repo's per-instance window cache and
1642	its push gate were invisible to this push's accounting. Fixed by making
1643	the window cache and the push-admission semaphore **isolate-wide**
1644	(one 32 MiB LRU, one 64-permit gate) and lowering the in-memory spill
1645	threshold to 24 MiB. The same push then succeeded with the first repo's
1646	caches warm; the clone back through the DO was fsck-clean.
1647
1648	**Truncation finding, same sweep.** One bench run reported the dynamic
1649	arm at 0.5 MiB of a 40.6 MiB pack — and passed. The compaction alarm
1650	had moved rows to a pack between the closure (which recorded
1651	`location='row'`) and emission (whose batched `SELECT oid, zdata` then
1652	saw NULL); the emitter failed the stream, the bridge errored the
1653	`ReadableStream`, the pump aborted the writable — and the client's HTTP
1654	library read the aborted body as a clean end. Two fixes: the row emitter
1655	now re-dispatches a moved object on its *current* location instead of
1656	failing (compaction and fetches run concurrently by design), and every
1657	timed clone in the bench de-frames the response and verifies the pack's
1658	SHA-1 trailer, so bytes-received can never again stand in for a pack.
1659	Real git was never at risk of a silent bad clone — it verifies the same
1660	trailer — but it would have seen "unexpected EOF" on a fetch that raced a
1661	compaction.
1662
1663	### §22.5 The push-ingest sweep
1664
1665	Measured with the platform's own per-invocation `cpuTime`/`wallTime`
1666	(a `wrangler tail`), because workerd freezes `performance.now()` during
1667	synchronous work — server-side timers only advance across awaits, which
1668	is itself diagnostic but cannot attribute CPU. The 15.6k-object, 40 MiB
1669	push of this repository, real `git push`, same edge:
1670
1671	| | push wall | DO cpu / wall | ingest | staging | finalize |
1672	| --- | --- | --- | --- | --- | --- |
1673	| start of sweep (in memory, per-row staging) | 63.6 s | — | 46 s | 3.1 s | 2.0 s |
1674	| spilled reads as views, 64 KiB probe | 34 s | 10.0 / 34.9 s | 20 s | 6.6 s | 2.7 s |
1675	| **promoted wire pack** | **24 s** | 8.3 / 16.0 s | 11–15 s | **0.4 s** | **0.7 s** |
1676	| + synchronous per-entry fast path | 16–18 s | 7.2 / 16.9 s | 8.7–10.8 s | 0.3–0.6 s | 0.6–0.7 s |
1677
1678	What the numbers taught, in order:
1679
1680	1. **Production CPU is ~10x this laptop, uniformly.** A probe worker
1681	timing every per-object primitive (sync/streaming inflate, SHA-1,
1682	copies, a JS loop, Effect overhead, deflate) found nothing
1683	pathological — each is 8–15x slower there. Local ingest at
1684	0.14 ms/object *is* production ingest at ~1.5 ms. Every microsecond
1685	per object is worth ten. (`ZlibProbe.e2e.test.ts`; the local variant
1686	pins that workerd keeps the synchronous exact-span inflate.)
1687	2. **The spilled reader copied 512 KiB per entry.** `blobRandomAccess`
1688	returned `slice()` for in-window reads and the parser probes each
1689	entry with a window-sized read: ~8 GB of memcpy per push, 31.9 s of
1690	CPU. Views fixed it (the in-memory reader always handed out views);
1691	the probe dropped to 64 KiB so crossing a window edge is rare and
1692	small, and the window grows to the header's declared size in one step.
1693	3. **Durable Object storage cost is bytes WRITTEN, not statements.**
1694	Multi-row inserts and 8x larger batches did nothing on production
1695	(locally they halved staging time — commits are cheap there); finalize
1696	was 55 ms locally vs 4.9 s in production for the same 7.3k-object push.
1697	Staging writes every blob into SQLite; the staged→live flip rewrites
1698	every one of those records. Both are proportional to blob bytes.
1699	4. **So don't write blobs to SQLite: promote the wire pack.** A push whose
1700	body spilled already holds every non-delta blob in exactly the
1701	compaction layout (typeSize header + zdata). Ingest now stages those
1702	rows as `location='pack'` references into the incoming object (pack id
1703	`wire-<receiveId>`, `packKeyOf` maps it), which then outlives the
1704	request as repo data. 9,633 of this push's 15.6k objects never touch a
1705	row; staging 6.6 s → 0.4 s, finalize 2.7 s → 0.7 s, and the later
1706	compaction rewrite of those blobs never happens. Delta-resolved blobs
1707	(fresh deflate, in no pack), trees, commits and tags still take rows.
1708	5. **Spill early; merge rewrites wire packs.** With promotion the spilled
1709	path is the fast path, so the in-memory threshold is 4 MiB (a 5.7 MiB
1710	in-memory push spent 6 s in finalize; spilled, 2.2 s), and a spilled
1711	push's admission permit is its window working set, not its body. Wire
1712	packs also carry trees, commits and delta entries, so geometric merge
1713	*rewrites* them — copies only the referenced spans with regenerated
1714	headers and repoints each row — into a clean pack, then grace-deletes
1715	the wire object like any other source. Read fan-out stays geometric.
1716
1717	6. **A synchronous inner loop.** With sources exposing `readSync` (a
1718	view into the buffer or a cached window), a non-delta entry is decoded,
1719	inflated, hashed and copied with no fiber hop, and entries reach the
1720	staging sink 256 at a time. 13% of production ingest CPU; workerd's
1721	`_processChunk` closes its binding after one chunk, so an inflate
1722	engine cannot be reused there (probed and recorded).
1723
1724	Where this leaves a 40 MiB push: about 6.7 s is the client (git's own
1725	pack build plus the upload; GitHub pays the same), ~7 s is the server's
1726	per-object verification CPU on a core roughly 10x slower than a laptop's,
1727	and ~1 s is staging plus finalize. The verification is what git requires
1728	to trust a pack — every entry inflated and hashed before any ref moves —
1729	and it runs single-threaded in the repo's Durable Object. Going meaningfully
1730	below this means parallelizing that verification across isolates (a pack
1731	has no index, so boundaries are only known by inflating; hashing could be
1732	fanned out but is under half the work) or changing what the client sends.
1733
1734	Left: the remaining ingest CPU (inflate + SHA-1 + parse, ~7 s here) is
1735	the work git itself requires to trust a pack; finalize's residual is the
1736	flip of the delta-resolved rows. Both scale with object count, not bytes.
1737
1738	### §22.6 Streaming ingest
1739
1740	The push body was received whole (in memory below the threshold, spilled
1741	to blob storage above it) and only then parsed. Two serial phases, each
1742	as long as the other; and a 128 MB isolate that can hold exactly one
1743	large body. Ingest now parses the body **while it arrives**:
1744
1745	- `StreamingSource` (`Store/StreamingSource.ts`) is a `RandomAccess` over
1746	4 MiB slabs the receiver fills. A read past what has arrived waits;
1747	a read behind the retention window (16 MiB) is served from the spilled
1748	object once the body has ended — and fails if it never spills, so a
1749	parser that needs an evicted base defers that entry to a pass after
1750	`awaitEnd` (`BaseEvictedError`). The feeder parks when it is 24 MiB
1751	ahead of the parser, and a waiting reader always releases a parked
1752	feeder: the two never deadlock on each other.
1753	- `receiveWireBodyStreaming` feeds the source and, past the threshold,
1754	uploads uniform 8 MiB parts concurrently (three in flight) so no
1755	round trip to blob storage stalls the request body. The part list is
1756	completed in part-number order regardless of settle order
1757	(`orderedMultipart`) — R2 validates the uniform-part rule against the
1758	list as given, and the tail settles before earlier parts do.
1759	- The parser tolerates an unknown pack length: the entry loop is driven
1760	by the header's count, the trailer SHA-1 is fed incrementally with a
1761	20-byte lag, and the trailer is compared once the source ends.
1762
1763	The head of the body (the pkt-line commands) is read as a 1 MiB range
1764	before anything else; a delete-only push is a 12-byte probe that fails.
1765
1766	### §22.7 The hasher fan-out
1767
1768	Verification is what git requires to trust a pack: every entry inflated
1769	and hashed before any ref moves. It scales with object count, and on a
1770	core ~10x slower than a laptop it was the bulk of a 40 MiB push. It is
1771	now a service — `Hasher` (`Hasher/Hasher.ts`) — with two layers:
1772	`HasherInline` (the same isolate) and `HasherSelf`, which POSTs pack
1773	parts to `/_alchemy/git/hash` on the Worker's own `Self` binding, so a
1774	push is verified by as many isolates as it has parts. The route is
1775	admin-authenticated (the Worker's own key) and answers a compact binary
1776	result: entry coordinates, oids, and for delta-resolved entries the fresh
1777	zlib bytes; non-delta entries ship no bytes (their rows are promoted
1778	coordinates or are read back from the source).
1779
1780	### §22.8 Raw chunks, resync, stitching
1781
1782	A pack has no index: entry boundaries are only known by inflating. The
1783	first fan-out shipped each part with the previous part's incomplete tail,
1784	which made the dispatch a serial chain gated on each part's result — the
1785	fan-out degenerated to one hasher at a time. Two passes (a cheap
1786	boundary pre-scan in the DO, then hashing by known spans) moved the CPU
1787	back into the single-threaded object.
1788
1789	The pump now dispatches **raw 4 MiB chunks as they arrive**, four in
1790	flight, with no dependency between them. Every chunk but the first is
1791	scanned in *resync* mode (`PartialScan.ts`): `findBoundary` walks the
1792	chunk for a zlib header (`0x78` + one of the four FLG bytes) at which two
1793	consecutive entries parse, and the scan proceeds from there — lenient,
1794	because `remaining` is only an upper bound and the last chunk runs into
1795	the trailer. The DO then consumes results in chunk order: the bytes
1796	between the last settled boundary and a chunk's first found boundary —
1797	one straddling entry, normally — are scanned as a region from the
1798	retained payloads; a region that ends exactly at the found boundary
1799	confirms the resync, otherwise the chunk is rescanned sequentially from
1800	the settled boundary. Chunks too small to hold two entries report no
1801	boundary and are settled by a later region. Deltas whose base lies in
1802	another chunk come back unresolved and are applied after the trailer,
1803	against bases from any chunk or from the live store.
1804
1805	The platform's own timeline (a `wrangler tail` of the hash calls) showed
1806	the fan-out working and the push still slow: chunks 0–3 arrived within
1807	0.7 s and the rest dribbled in at ~4 MiB/s, while the receive-pack
1808	invocation's wall time was five times its CPU. Everything the Durable
1809	Object received it also sent out twice — to the spill and to the hashers.
1810
1811	### §22.9 The pack never enters the Durable Object
1812
1813	The whole pipeline above now runs in the **stateless Worker**
1814	(`Server.ts` `receivePackRoute`): it receives the body, verifies it
1815	through the fan-out, stitches, resolves deltas, and ships the Repo DO only
1816	*rows* over RPC — `beginPush` (authorization of the parsed commands, the
1817	staging push row), `stagePush` (batches of staged rows, three in flight
1818	and joined before the commit; promoted rows are coordinates into the wire
1819	pack, inline rows carry their zlib bytes — `PushWire.ts`), `readPushBase`
1820	(thin-delta bases), and `commitPush` (connectivity over the staged rows,
1821	the commit graph, the transactional ref CAS, post-commit bookkeeping).
1822	The DO's memory, CPU and egress are untouched by push size; its part of
1823	a push is SQL.
1824
1825	Moving the pipeline exposed the next wall: the receiver was sending
1826	every byte out twice — to the spill and to the hashers — and the last
1827	part was dispatched only when the multipart upload completed, because
1828	that was when the streaming source ended. Three changes:
1829
1830	- **The hasher writes the spill.** Parts are body-aligned 8 MiB (R2's
1831	uniform-part rule), and each `hashPart` call carries the multipart
1832	upload's key and id: the hasher isolate writes the part it verifies
1833	(`BlobStoreShape.uploadPart` — parts may be uploaded from any isolate;
1834	`complete` takes the collected parts and sorts them). The receiver
1835	sends each byte out once; the first part carries the command section
1836	and the pack header before its first entry (`skip`).
1837	- **The source ends at body end.** `expectFallback` makes an evicted read
1838	wait for the completed spill instead of failing, so the last part is
1839	dispatched the moment the body ends, and results settle while the body
1840	is still arriving: the consumer is a forked fiber, the producer runs
1841	on the request's own (hash fibers are its children).
1842	- **The straddler is speculative.** The entry crossing a part edge is
1843	dispatched the moment its part settles, sized from its own header out
1844	of the retained bytes, so a 10 MB blob spanning three parts is hashed
1845	alongside the parts rather than after the last one — and `findBoundary`
1846	searches the whole part (it searched the first MiB: a part whose head
1847	lay inside such a blob reported no boundary and was hashed serially).
1848
1849	What is left is the Worker's own I/O: a single invocation receives the
1850	body and forwards it, and the two share the isolate's throughput.
1851
1852	### §22.10 Measured, and the wall
1853
1854	The same two pushes as §22.5 — this repository's 15.6k-object, 40 MiB
1855	delta-heavy pack (`loose`) and a 15.3k-object, 44 MiB pack of mostly
1856	whole blobs (`synthetic`) — real `git push` from the same client, same
1857	edge, and GitHub on a fresh repository from the same client for scale.
1858	Cloudflare timings vary run to run by ±20%; the ranges are what was seen.
1859
1860	| | loose push | synthetic push | incremental push | bare clone |
1861	| --- | --- | --- | --- | --- |
1862	| GitHub (fresh repo) | 3.3 s | 4.6 s | 1.2 s | 1.3 / 2.1 s |
1863	| start of this sweep (in-DO, sequential fan-out) | 16 s | 10.8 s | 0.9–1.3 s | 1.3 s |
1864	| pipeline in the Worker (§22.9) | 19 s | 12.9 s | | |
1865	| + hashers write the spill, body-aligned parts | 14–19 s | 9.3–12.9 s | | |
1866	| + speculative straddler, whole-part boundary search | 11.3–14.8 s | 7.7 s | 0.3–0.4 s | |
1867	| + scan-first hash responses, native receive loop | 13–14.6 s | 9.5–11 s | 0.3–0.7 s | 1.3 s |
1868	| **`HasherInline`** (no fan-out; the reference assembly) | 9.9–13.5 s | 8.8 s | 0.24–0.27 s | 1.3 s |
1869
1870	The receive-side phases of a loose push at the end of the sweep (from
1871	the pump's own log, `Date.now` advancing only across I/O): parts 1–3
1872	arrive within 1.3 s (18 MiB/s, the client's rate to Cloudflare's own
1873	speed test), then nothing arrives for 4.5 s while three hash calls are in
1874	flight, then the rest; the body ends at 8.7 s and the last result settles
1875	at 8.7 s. The hash calls' wall times stack instead of overlapping.
1876
1877	**Why: service-binding subrequests run on the caller's thread.** A probe
1878	route fanning one 8 MiB part out N times, timed from the client:
1879
1880	| transport | n=1 | n=3 | n=6 |
1881	| --- | --- | --- | --- |
1882	| self service binding (`Cloudflare.Workers.Self`) | 1.1 s | 1.9–2.6 s | 3.9–4.5 s |
1883	| service binding to a second script | 0.7–1.0 s | 1.4–2.5 s | 3.6–3.9 s |
1884	| `fetch` to the Worker's own workers.dev URL | refused (connection lost) |
1885	| `fetch` to a second script's workers.dev URL | refused (connection lost) |
1886
1887	Six calls cost six times one call over both bindings: the callee executes
1888	in the caller's isolate (four concurrent self calls with 8 MiB bodies hit
1889	the isolate's memory limit, error 1102), and the in-Worker clock does not
1890	even advance across the call. The fan-out therefore never parallelized
1891	CPU; it moved bytes around on one thread. What it did buy — and what the
1892	Worker-side pipeline keeps — is a Durable Object that never sees a pack
1893	byte, a receive that is not blocked on the spill, and hashing that
1894	overlaps the client's upload as well as one thread allows.
1895
1896	Requests arriving from OUTSIDE are spread across isolates: four
1897	concurrent external calls of the same part finished as two pairs (2.0 s
1898	and 4.1 s), i.e. two-way parallelism. Cross-zone `fetch` to a hasher on
1899	another zone would inherit that, at the price of real network egress
1900	competing with the client's upload.
1901
1902	On one thread the floor for a push is roughly `max(ingress, hash CPU)`
1903	plus staging and commit: for the loose pack about 2.5 s of upload and 5 s
1904	of hashing on a core ~10x slower than a laptop's, so 6–7 s before the
1905	commit — the measured 8.7–11 s is that floor plus imperfect overlap. The
1906	synthetic pack hashes in ~2 s and lands nearer 6 s. Reaching GitHub's
1907	3.3 s on the delta-heavy pack needs compute Workers do not offer per
1908	request: native `index-pack` in a Container (verifying 40 MiB in well
1909	under a second, writing R2 from its own network), or a multi-core host
1910	behind a cross-zone HTTP hasher. Both are layers this design admits
1911	(`Hasher` is a service); neither is built here.
1912
1913	With the fan-out removed (`HasherInline`, the receiver hashing as parts
1914	arrive and spilling them itself), the loose push measures 9.9–13.5 s and
1915	the receiver's own CPU shows the hashing — 5.0–7.9 s of it — that the
1916	fan-out had spread over invocations on the same thread. That layer is now
1917	the reference assembly; `HasherSelf` stays as the shape a remote hasher
1918	takes.
1919
1920	Where the pipeline is already at or past parity: incremental pushes
1921	(0.24–0.7 s vs 1.2 s), bare clones (§22.4), and the read path.
1922
1923	### §22.11 Hasher layers: the same push on other compute
1924
1925	`Hasher` is a service precisely so the compute behind verification can be
1926	chosen per deployment. Three layers ship:
1927
1928	| layer | where the scan runs | spill parts written by | parallelism | cost model |
1929	| --- | --- | --- | --- | --- |
1930	| `HasherInline` (default) | the receiving Worker | the receiver | none (one thread) | included in the Worker's CPU |
1931	| `HasherSelf` | another invocation of the same script | the hasher | none on Cloudflare (§22.10) | same |
1932	| `HasherLambda(fn)` (`alchemy/Git/Hasher`) | an AWS Lambda per chunk | the receiver | one Lambda per 4 MiB chunk, all at once | Lambda-seconds + cross-cloud egress |
1933
1934	**`HasherLambda`.** The Git Worker binds `AWS.Lambda.InvokeFunction` on a
1935	Lambda declared in the same stack (`HasherFunction`, an Effect-native
1936	Function whose only handler answers hash events); the stack carries both
1937	provider sets. Binding a Lambda operation to a *Worker* host provisions —
1938	once per Worker — an IAM user, key and least-privilege assume-role Role,
1939	binds the key and role ARN onto the Worker, and signs each invoke with
1940	credentials assumed through a single-flight, expiry-aware resolver: the
1941	cross-cloud scaffolding the MicroVM bindings introduced, now shared by
1942	every function-scoped Lambda `*Http` binding (`BindingHttp.ts`).
1943
1944	The invoke payload is JSON, so a chunk travels base64: chunks are 4 MiB
1945	(≈5.6 MB encoded, under the 6 MB limit) — the pump reads chunks of the
1946	hasher's declared size and, because a Lambda cannot reach the blob store,
1947	uploads the 8 MiB spill parts itself from the retained bytes
1948	(`writesSpill: false`). The response is bounded to the same limit: the
1949	Lambda omits non-blob content (the receiver inflates trees and commits
1950	from its retained bytes) and, when the fresh zdata of delta-resolved
1951	entries would still overflow, demotes the largest of them back to
1952	`unresolved` with their base references, so the receiver resolves those
1953	as it would any cross-chunk delta. Any failure on the Lambda side — a
1954	throttle, a cold-start timeout, a rejected payload — hashes that chunk
1955	inline: a push never fails because its hasher did.
1956
1957	Measured, same pushes and client as §22.10 (`HasherLambda`, `us-east-1`,
1958	3 GB functions, the Worker on Cloudflare's edge):
1959
1960	| | loose push | synthetic push | incremental push |
1961	| --- | --- | --- | --- |
1962	| GitHub (fresh repo) | 3.3 s | 4.6 s | 1.2 s |
1963	| `HasherInline` | 9.9–13.5 s | 8.8 s | 0.24–0.4 s |
1964	| **`HasherLambda`** | **6.8–7.9 s** | **6.3–6.7 s** | 0.27–0.8 s |
1965
1966	The body now arrives in 0.8–2.7 s (the receiver's thread is free), the
1967	last chunk settles within a second of it, and the receiver's own CPU
1968	drops to 1.4–2.7 s: the trailer SHA-1, the staging batches, and — on the
1969	delta-heavy pack — 515 cross-chunk deltas resolved locally plus the
1970	10 MB straddler that exceeds an invoke payload and is hashed inline.
1971	Those, the staging round trips (0.7–1.2 s) and the commit (0.6–0.7 s)
1972	are what remains between this and GitHub.
1973
1974	What this buys: every chunk verifies at once, on a core several times
1975	faster than a Worker's, so hashing leaves the receive's critical path;
1976	what it costs: the chunk crosses to AWS and the scan comes back, so a
1977	Lambda hasher is worth it for pushes of tens of MiB and irrelevant for the
1978	kilobyte pushes that dominate a repository's life — which is why it is a
1979	layer and not the default.
1980
1981	### §22.12 Hasher on dynamically loaded Workers
1982
1983	`WorkerLoader` instantiates a Worker from module source at runtime, under
1984	a name the runtime caches. The question that decides whether it is a
1985	hasher was answered by a probe (four loaded workers each running a fixed
1986	CPU burn, timed from the client):
1987
1988	| | n=1 | n=4 same name | n=4 distinct names | n=8 |
1989	| --- | --- | --- | --- | --- |
1990	| CPU burn, client time | 0.44–0.79 s | 1.94 s (serial) | **0.65 s (parallel)** | refused |
1991	| 100 MB allocation each | ok | ok | ok, four isolates at once | refused |
1992
1993	Distinct names are distinct isolates, and distinct isolates run **in
1994	parallel with the caller** — unlike a service binding (§22.10). Each has
1995	its own 128 MB. The runtime allows four concurrent dynamic-worker
1996	invocations per request; an eighth call fails with "Dynamic worker
1997	concurrency limit exceeded".
1998
1999	`HasherWorkerLoader` (`alchemy/Git/Hasher`) keeps four fixed names
2000	(`git-hasher-0..3`, warm across pushes), dispatches 4 MiB chunks over the
2001	hash route's own protocol, and lets the pump write the spill
2002	(`writesSpill: false`); the loaded hasher has no bindings and
2003	`globalOutbound: null`. Its module is the real scanner: the Worker
2004	bundler gained a `?worker` import (`WorkerModulePlugin.ts`) that bundles
2005	the target into one self-contained module and imports it as a string —
2006	nested `?worker` imports included — so `WorkerLoaderModule.ts` is ordinary
2007	TypeScript that shares `PartialScan` with everything else. (The loader's
2008	`get` also wrapped the native stub as an entrypoint and had no `fetch`;
2009	fixed with a regression test.)
2010
2011	Measured, same pushes and client as §22.10:
2012
2013	| | loose push | synthetic push | incremental push |
2014	| --- | --- | --- | --- |
2015	| GitHub (fresh repo) | 3.3 s | 4.6 s | 1.2 s |
2016	| `HasherInline` | 9.9–13.5 s | 8.8 s | 0.24–0.4 s |
2017	| `HasherLambda` | 6.8–7.9 s | 6.3–6.7 s | 0.27–0.8 s |
2018	| **`HasherWorkerLoader`** | **7.1–7.8 s** | **6.8 s** | 0.24 s |
2019
2020	The body is in by 0.8–1.5 s, chunks settle in waves of four (four by
2021	0.9 s, the next four by 2.1 s, the last by 2.5 s), and the receiver's own
2022	CPU is the lowest of the three (0.9–1.4 s: no base64, no signing). What
2023	remains is the same tail as §22.11 — cross-chunk deltas, the oversize
2024	straddler, staging and the commit — and the four-way cap: a push wider
2025	than four chunks hashes in waves.
2026
2027	### §22.13 Deltas as jobs, and the resolved pack
2028
2029	Two receiver-side costs survived every hasher: the cross-chunk and thin
2030	deltas the scan leaves unresolved (515 on the delta-heavy pack) were
2031	applied, hashed and deflated on the receiver's thread, and every
2032	delta-resolved blob was staged inline — its fresh zlib written into
2033	SQLite and rewritten by the staged→live flip.
2034
2035	**Delta rounds.** `Hasher.resolveDeltas(bases, jobs)` applies a batch of
2036	deltas whose bases the caller supplies (a base's zlib or, for a live
2037	object, its content), returning oid, fresh zlib and non-blob content.
2038	The pump resolves in rounds: every delta whose base is known joins a
2039	byte-bounded batch — bases shipped once per batch — and the batches run
2040	across the hasher's slots at once; a delta on a still-unresolved delta
2041	waits for the next round. The receiver only assembles bytes it already
2042	holds. Every layer implements it (the loaded worker and the self route
2043	answer `mode=deltas`; the Lambda keeps it inline until its payload budget
2044	is worth spending). 515 deltas: 0.26–0.37 s of round trips instead of
2045	receiver CPU.
2046
2047	**The resolved pack.** While a body spills, delta-resolved blobs are held
2048	back from staging; after the rounds their zlib is written as one object,
2049	`wire-<receiveId>-r` (`typeSize header + zdata` per entry, the promoted
2050	layout `packKeyOf` already maps), and their rows become coordinates. The
2051	DO writes no blob bytes for a large push at all: on the delta-heavy pack
2052	every one of its 13,646 blobs is a coordinate and 1,968 trees, commits
2053	and tags stay resident (the resolved pack is 4.1 MB). Compaction's
2054	geometric merge treats it like any wire pack.
2055
2056	### §22.14 The lazy flip
2057
2058	With every blob a coordinate, `finalize` was still 0.6–0.7 s of a
2059	~4.3 s push: `UPDATE objects SET staged_push = NULL WHERE staged_push = ?`
2060	rewrites every staged row of a WITHOUT ROWID table keyed by random oids —
2061	one page per row, on the response path. A staging table would only move
2062	the same random-page insert to commit time.
2063
2064	A push's objects are now live the moment its `pushes` row is committed:
2065	every read predicate is `LIVE_OBJECTS` —
2066	`staged_push IS NULL OR staged_push IN (committed pushes)` — a
2067	materialized subquery over a table that holds at most the pushes whose
2068	flip is pending. The transaction commits the refs, the graph and the
2069	push row; the per-row rewrite runs after the response (`flipPush`, in
2070	the DO's `waitUntil`), and GC flips any committed push whose deferred
2071	flip never ran. A new push may adopt rows only from pushes that are not
2072	committed, so a committed-unflipped row is never re-staged (and then
2073	GC'd) out from under a live object. Finalize on the delta-heavy push:
2074	0.7 s → the refs and graph alone; locally 50 ms → 2 ms.
2075
2076	### §22.15 Where the sweep stands
2077
2078	Same pushes and client; three runs each on the loader hasher after
2079	§22.13–14, best and median (the spread is the client's uplink — the
2080	route's own total is 3.4–3.9 s for the loose push and 2.2–3.6 s for the
2081	synthetic one):
2082
2083	| | loose push | synthetic push |
2084	| --- | --- | --- |
2085	| GitHub (fresh repo) | 3.3 s | 4.6 s |
2086	| start of the sweep | 63.6 s | — |
2087	| `HasherWorkerLoader`, §22.12 | 7.1–7.8 s | 6.8 s |
2088	| + delta jobs, resolved pack, lazy flip, batched graph, edge-only connectivity | **5.9 s best, 6.5 s median** | **4.8 s best, 6.4 s median** |
2089
2090	The whole-blob push meets GitHub on a good run. The delta-heavy push is
2091	2.6 s behind on its best run, and the route's remaining time is the
2092	hashing waves — ten 4 MiB chunks over four isolates, the third wave
2093	landing a second after the body — plus the delta rounds (0.2–0.4 s),
2094	the resolved pack's late write and the commit round trip (0.5–0.7 s).
2095	The next lever for it is more than four hashers per push: a loaded
2096	hasher can hold a loader binding of its own and fan a chunk out a
2097	second time, or a push can split its chunks across the loader and a
2098	Lambda. Neither is built here.