clover's git

the-knife

1	---
2	title: Hasher
3	description: Every object a push sends is inflated and hashed before a ref moves. That is CPU, and where it runs is a Layer, in the Worker, in dynamically loaded Workers, or on Lambda.
4	---
5
6	A push arrives as a pack. Before any ref moves, every object in it is
7	inflated, hashed, and checked against the objects the repository
8	already has. `Git.Hasher` is where that work runs:
9
10	```typescript
11	Layer.provide(Git.HasherInline)
12	```
13
14	The Worker receiving the push never holds it whole:
15
16	1. The first 1 MiB is read for the ref commands, and
17	[Auth](/git/blocks/auth) rules on them before any pack byte is
18	inspected.
19	2. The body streams through in 8 MiB parts. Each part is cut into
20	4 MiB chunks for the hasher as it lands and is parked in the
21	[blob store](/git/blocks/blob-store) as a multipart upload.
22	3. Hashed objects reach the repository's Durable Object as batches of
23	rows over RPC: coordinates into the parked pack for large objects,
24	inline bytes for small ones.
25	4. The Durable Object checks connectivity over the staged rows,
26	updates the commit graph, moves the refs in one transaction, and
27	schedules compaction.
28
29	The pack never enters the Durable Object. Its memory, CPU, and egress
30	are untouched by push size, and its share of a push is SQL.
31
32	The Worker retains a 16 MiB window of the body for objects that span
33	parts, so peak memory is that window plus the parts in flight,
34	whatever the push size. Pushes share a 64 MiB budget per isolate, and
35	one that does not fit waits for an earlier one to drain.
36
37	An object that straddles two chunks is hashed from its own header the
38	moment its first chunk settles, so a 10 MB blob spanning three parts
39	does not wait for the last one. Deltas whose base is in another chunk
40	are resolved as a second round of hasher jobs, and the resolved blobs
41	are written out as a pack of their own so a fetch never re-resolves
42	them.
43
44	Where a chunk is hashed is the only thing the Layers below disagree
45	on.
46
47	## In the Worker
48
49	`HasherInline` hashes chunks on the thread that received them. It costs nothing
50	extra, needs nothing else deployed, and is the right answer for the
51	kilobyte pushes that make up almost all of a repository's life.
52
53	## In dynamically loaded Workers
54
55	```typescript
56	import * as GitHasher from "alchemy/Git/Hasher";
57
58	Layer.provide(GitHasher.HasherWorkerLoader())
59	```
60
61	Four Workers, loaded on demand from a module bundled into your Worker,
62	each in its own isolate with its own 128 MB, hash four chunks at once.
63	Nothing leaves Cloudflare and nothing needs credentials. The runtime
64	allows four loaded Workers per request, so a push wider than that is
65	hashed in waves.
66
67	```typescript
68	GitHasher.HasherWorkerLoader({ concurrency: 2 })
69	```
70
71	## On Lambda
72
73	```typescript
74	import * as AWS from "alchemy/AWS";
75	import * as GitHasher from "alchemy/Git/Hasher";
76
77	Layer.provide(GitHasher.HasherLambda(GitHasher.HasherFunction)),
78	Layer.provide(AWS.Lambda.InvokeFunctionHttp),
79	```
80
81	One Lambda invocation per chunk, all of them at once, each on a 3 GB
82	function. `HasherFunction` is an Effect-native Lambda the package
83	declares. Your Worker binds it cross-cloud: Alchemy creates an IAM user
84	and a least-privilege role for the Worker, binds the key onto it, and
85	signs each invoke with credentials assumed at runtime. The stack that
86	deploys this carries both provider sets.
87
88	If a chunk cannot be hashed there, because of a throttle or a cold
89	start that runs out of time, the Worker hashes it inline. A push never
90	fails because its hasher did.
91
92	## Choosing
93
94	One client, the same edge. A 40 MiB delta-heavy pack of 15.6k
95	objects, a 44 MiB pack of whole blobs, and GitHub on a fresh
96	repository for scale:
97
98	| Layer | Parallelism | Memory per hasher | Leaves Cloudflare | Extra cost | 40 MiB deltas | 44 MiB blobs |
99	| --- | --- | --- | --- | --- | --- | --- |
100	| `HasherInline` | none | the Worker's | no | none | 9.9–13.5 s | 8.8 s |
101	| `HasherWorkerLoader` | 4 isolates | 128 MB | no | none | 5.9–7.2 s | 4.8–6.4 s |
102	| `HasherLambda` | one Lambda per chunk | 3 GB | yes | Lambda time, egress | 6.8–7.9 s | 6.3–6.7 s |
103	| GitHub | | | | | 3.3 s | 4.6 s |
104
105	Incremental pushes take 0.2–0.4 s on every Layer, against 1.2 s on
106	GitHub. The choice only matters for pushes of tens of megabytes.
107
108	Start with `HasherInline`. Move to `HasherWorkerLoader` when large
109	pushes matter and you want to stay on one cloud. Reach for
110	`HasherLambda` when a push is wider than four chunks and you already
111	run AWS. [Scaling](/git/recipes/scaling) walks a large push through
112	the swap.
113
114	## Your own
115
116	The contract is three operations over bytes: scan a chunk and return
117	its entries, resolve a batch of deltas against their bases, and hash
118	the bounds of an entry that straddles two chunks. `Git.HasherSelf` is
119	the smallest implementation to read. It fans out through the Worker's
120	own service binding, which on Cloudflare runs on the caller's thread
121	and buys nothing, but it is the shape a hasher on another host takes.