clover's git

the-knife

1	---
2	title: Bring your own store
3	description: Bytes have to live somewhere that is neither R2 nor S3. Implement the five operations of Git.BlobStore and provide it in place of the shipped Layers.
4	---
5
6	The examples use `Authentication` from
7	[Getting Started](/git/getting-started): the application's request middleware.
8
9	Your packs have to live in a store the package does not ship for: a
10	different object store, an on-premises S3 clone with its own signing,
11	a store behind a private gateway. `Git.BlobStore` is a contract of five
12	operations, and a Layer that implements them drops into the graph:
13
14	```diff lang="typescript"
15	const GitLive = Git.ApiLive.pipe(
16	Layer.provide(Git.ApiHandlersLive),
17	Layer.provide(Authentication.layer),
18	Layer.provide(Git.ReposDurableObject),
19	Layer.provide(Git.RegistryDurableObject),
20	Layer.provide(Git.HasherInline),
21	- Layer.provide(Git.BlobStoreR2(GitObjects)),
22	+ Layer.provide(BlobStoreMine),
23	);
24	```
25
26	## The contract
27
28	```typescript
29	interface BlobStoreShape {
30	get(key, range?): Effect<BlobBody | null, BlobStoreError>;
31	put(key, body, { contentLength }): Effect<void, BlobStoreError>;
32	multipart(key): Effect<BlobMultipart, BlobStoreError>;
33	delete(keys): Effect<void, BlobStoreError>;
34	list(prefix): Stream<BlobMeta, BlobStoreError>;
35	}
36	```
37
38	`get` with a range is how packs are read: a fetch asks for a few large
39	windows of an immutable pack. `put` writes a whole object of known
40	length. `multipart` is how a push body larger than memory is parked as
41	it arrives. `delete` and `list` serve compaction and purge, never a
42	serving path.
43
44	## The Layer
45
46	```typescript
47	import * as Git from "alchemy/Git";
48	import * as Effect from "effect/Effect";
49	import * as Layer from "effect/Layer";
50
51	export const BlobStoreMine: Layer.Layer<Git.BlobStore> = Layer.effect(
52	Git.BlobStore,
53	Effect.gen(function* () {
54	const client = yield* MyStoreClient; // yours
55
56	return {
57	get: (key, range) =>
58	client.read(key, range).pipe(
59	Effect.map((r) => (r ? { size: r.size, bytes: r.bytes, stream: r.stream } : null)),
60	Effect.mapError((e) => new Git.BlobStoreError({ reason: String(e) })),
61	),
62	put: (key, body, { contentLength }) => /* ... */,
63	multipart: (key) => /* returns { uploadId, uploadPart, complete, abort } */,
64	delete: (keys) => /* ... */,
65	list: (prefix) => /* ... */,
66	} satisfies Git.BlobStoreShape;
67	}),
68	);
69	```
70
71	The Layer is built once in the Worker and once in each Durable Object,
72	so construction must be cheap and hold nothing with a finalizer. Open
73	connections per call, not per Layer.
74
75	## Rules the push path relies on
76
77	Multipart parts are uniform in size except the last, 8 MiB as written
78	today. `uploadPart` may be called from any isolate, and `complete`
79	receives the collected parts and must accept them in any order. A
80	store that enforces those two rules works with no change above it.
81
82	Errors are `BlobStoreError` with a `reason`. The service retries
83	nothing on your behalf, so a transient failure you want retried is
84	retried inside your Layer.
85
86	## Verifying it
87
88	Deploy the host with your Layer, push a repository, clone it back
89	under `git fsck --strict`, then push enough to trigger compaction and
90	clone again. If both clones are clean, every operation has been
91	exercised: `multipart` and `get` by the push, `put` by compaction and
92	bundling, `delete` and `list` by deleting the repository.