clover's git

the-knife

1	# Effect Runtime Performance
2
3	This harness measures focused synchronous runtime paths in fresh Node
4	processes. It supports:
5
6	- focused Effect Schema diagnostics;
7	- native Arbitrary comparisons against equivalent fast-check v4 arbitraries;
8	- the upstream Effect, Valibot and Zod benchmark matrix;
9	- paired comparisons between Git revisions or the current working tree.
10
11	Cross-library results are diagnostic. Effect base/head comparisons are the
12	authoritative measurement for source changes.
13
14	## Commands
15
16	Run the complete registry:
17
18	```sh
19	pnpm runtimeperf
20	```
21
22	Run the cases extracted from the `effect@beta`, Valibot and Zod adapters in
23	[`open-circle/schema-benchmarks`](https://github.com/open-circle/schema-benchmarks):
24
25	```sh
26	pnpm runtimeperf schema-benchmarks
27	pnpm runtimeperf-compare schema-benchmarks --base main --head HEAD
28	```
29
30	This suite covers every upstream timing case supported by those adapters:
31	schema and decoder initialization, validation, parsing and Standard Schema
32	with valid/invalid inputs and first/all error modes, plus BigInt codec
33	operations. The upstream bundle and stack reports are not throughput
34	benchmarks, and the adapters do not define the optional string-format cases.
35
36	Select a suite, fixture, shared scenario, tier, family or implementation:
37
38	```sh
39	pnpm runtimeperf schema
40	pnpm runtimeperf arbitrary
41	pnpm runtimeperf object-32-valid
42	pnpm runtimeperf schema/object-32-valid-effect
43	pnpm runtimeperf --family arrays
44	pnpm runtimeperf --implementation zod4
45	pnpm runtimeperf --implementation zod4-compiled
46	```
47
48	Override measurement settings:
49
50	```sh
51	pnpm runtimeperf object-32-valid --rounds 9 --time 500 --warmup-time 150
52	```
53
54	Compare Effect `HEAD` with the working tree:
55
56	```sh
57	pnpm runtimeperf-compare schema/object-32-valid-effect
58	```
59
60	Measure excess-property checking and symbol enumeration in interpreted and JIT parsers:
61
62	```sh
63	pnpm runtimeperf-compare schema-excess-properties
64	pnpm runtimeperf-compare schema-excess-properties --family jit
65	```
66
67	Compare explicit refs:
68
69	```sh
70	pnpm runtimeperf-compare schema --base main --head HEAD
71	```
72
73	Only `--fail-on-regression` turns a statistically classified regression into a
74	non-zero comparison exit code. Worker, fixture, configuration and Git errors
75	always fail.
76
77	Reports are written under `tmp/runtimeperf/results/`. Temporary Git worktrees
78	are created under `tmp/runtimeperf/worktrees/` and removed in `finally`.
79
80	## Registry
81
82	`config.json` groups fixture cases by source file. Every case records:
83
84	- tier and family;
85	- covered Effect AST tags;
86	- operation and path;
87	- input size;
88	- implementation;
89	- a shared scenario name for cross-library comparisons.
90
91	The `schema` suite is Effect-only and retains targeted diagnostics for scaling,
92	template literals, unions, records, transformations, optional properties,
93	adapters, recursion and cold paths. The `schema-benchmarks` suite contains the
94	complete timing matrices exposed by the upstream Effect, Valibot and Zod
95	adapters.
96
97	The `moltar-parse-safe` and `moltar-assert-loose` suites preserve Moltar's
98	object shape. Their valid cases reproduce the upstream timed operation; the
99	extra-property and invalid cases are Effect extensions. They compare
100	interpreted Effect, Effect JIT and AOT, Valibot, ordinary and jitless Zod where
101	applicable, and Zod `compile`, using the dependency versions recorded in each
102	report.
103
104	The `arbitrary` suite compares the native public API with direct fast-check v4 arbitraries in separate processes. It
105	measures derivation through the first recursive sample, steady-state recursive
106	sampling, optional-Struct sampling, fixed-length string generation to exercise constraint pushdown,
107	bounded Number generation, direct Uint8Array generation, a rare residual filter, a fixed-length unique array, and a
108	mixed regular expression through cold derivation, warm generation, and shrinking. It also covers literal sampling as
109	a runner baseline and the public `map`, `filter`, `filterMap`, and `all` combinators for tuples and records. The failure
110	paths include a filtered failure with rejected shrink candidates, dependent `flatMap` sampling, shrinking, replay, a
111	passing property, a failure that shrinks from `1000` to `1`, and replay of that failure. The
112	recursive distributions are implementation-defined, so the fixtures use
113	implementation-specific size settings and validate a comparable total node
114	count for the fixed seed. The bounded Number case is a throughput comparison,
115	not distribution parity: native selects among 64-bit IEEE-754 representations,
116	while the fast-check fixture uses its 64-bit `double`. Replay is an
117	end-to-end public API comparison, but the work is not identical: native replay
118	verifies the original failure and its full shrink path, while fast-check can
119	start directly from its recorded path.
120
121	Array cases cover passing checks and shrinking a four-command sequence while preserving an ordered pair. The latter
122	starts from the same explicit sequence in both engines, using a fixed seed for native generation and an example for
123	fast-check. Base/head validation preserves the failure without requiring identical shrunk outputs, so its timing also
124	reflects changes in how much of the shrink tree is explored. The runtime regression tests assert the smaller result.
125
126	The regular-expression fixtures validate the same language, and both fixed-seed warm fixtures must reach multiple
127	lengths and every alternative family. Their exact distributions remain implementation-defined, so cross-library
128	timings are diagnostic while base/head comparisons protect the cost of each implementation's established behavior.
129
130	An impossible Schema filter is deliberately not a cross-engine timing case.
131	The native runner reports bounded exhaustion, while a direct fast-check arbitrary
132	built with `Arbitrary.filter` does not return from generation
133	when no value can satisfy the predicate. Native exhaustion remains covered by
134	the Arbitrary tests instead of placing a permanently blocking fixture in the
135	performance harness.
136
137	Zod parsing cases import `zod/v4` and call `safeParse` with `{ jitless: true }`;
138	its Standard Schema and codec cases use their native APIs. Valibot uses the
139	corresponding `is`, `safeParse` and Standard Schema APIs. The focused Effect
140	adapter family measures the overhead of public APIs that wrap parser issues.
141	The compiler comparison calls `z.compile(schema, { strict: true })` and uses
142	Zod's `validate` API for boolean checks. Ten representative scenarios also run
143	against equivalent Valibot schemas using `parse` and `is`.
144
145	## Measurement model
146
147	Each worker validates the fixture before and after measuring. The Effect
148	fixture calibrates one batch size that every implementation in the same
149	scenario uses. This keeps the enclosing loop identical across implementations;
150	V8 can otherwise optimize sub-10 ns callbacks differently at different batch
151	sizes. Each implementation executes in a separate process, with rotating order
152	within the scenario.
153
154	Tinybench measures one synchronous batched task. The primary process result is:
155
156	```text
157	nsPerOp = totalTimeMs * 1_000_000 / (task.runs * batchSize)
158	```
159
160	Tinybench latency statistics are retained as diagnostics and normalized by the
161	batch size. Independent Node processes, not Tinybench samples, are the
162	statistical observations.
163
164	Base/head comparisons alternate execution order by round and analyze paired
165	log ratios with a deterministic bootstrap. The report keeps all raw worker
166	results so aggregates can be recalculated.
167
168	## Fixture contract
169
170	Each configured export is a factory:
171
172	```ts
173	type RuntimePerfCase = {
174	readonly run: () => unknown
175	readonly validate: (result: unknown) => void
176	}
177
178	type RuntimePerfCaseFactory = () => RuntimePerfCase
179	```
180
181	Construct schemas, steady-state adapters and deterministic inputs in the
182	factory. Cold fixtures deliberately construct them inside `run`. A fixture
183	must measure one named operation, perform no I/O and return no Promise.
184
185	## Validation
186
187	```sh
188	node --test packages/effect/runtimeperf/test/*.test.mts
189	pnpm exec dprint check package.json packages/effect/runtimeperf
190	```