clover's git

the-knife

1	# Testing Strategy
2
3	V2 should be validated with a small number of high-value integration tests rather than a large suite of unit tests that mock away the behavior being tested.
4
5	The goal is not "no test doubles ever." The goal is that test doubles exist only at true process, network, clock, id, and filesystem boundaries. Core orchestration behavior must run for real.
6
7	## Testing Principle
8
9	The default V2 test shape is:
10
11	```text
12	command dispatch
13	-> real Orchestrator
14	-> real ProviderAdapter
15	-> replayed ProviderRuntime transport
16	-> real adapter normalizer
17	-> real V2 event store/sink using production persistence semantics
18	-> real V2 projection/projector
19	-> real Checkpoint policy
20	-> assertions
21	```
22
23	The replay framework replaces the external provider process or network stream. It does not replace the adapter, normalizer, command/event infrastructure, projection reducers/projectors, checkpoint policy, or business logic.
24
25	Raw provider frames in replay transcripts are realistic transport evidence. In production, equivalent raw frames are diagnostic log data with bounded retention. Integration tests should still use real normalized orchestration persistence/projections so replay input exercises the same adapter and orchestration paths as live provider output.
26
27	## Allowed Test Substitutes
28
29	Allowed substitutes:
30
31	- provider runtime transport, backed by deterministic replay transcripts.
32	- Effect runtime time, controlled in tests with `TestClock` from `effect/testing`.
33	- Effect `Random`, provided with a deterministic test implementation for stable UUIDs/numbers.
34	- temporary filesystem/worktree.
35	- temporary database or in-memory database with the same repository interfaces.
36	- fake process supervisor only when it is testing process failure behavior directly.
37
38	Not allowed in integration tests:
39
40	- mocked orchestrator.
41	- mocked provider adapter.
42	- mocked provider event normalizer.
43	- mocked command/event infrastructure or V2 event sink.
44	- mocked projection reducer.
45	- mocked checkpoint service behavior.
46	- mocked provider capability policy.
47	- pre-normalized domain events used as the input for adapter tests.
48	- custom clock/id services that duplicate Effect's `Clock`, `DateTime`, or `Random` services.
49
50	Pure reducer tests are still valid, but they should be few and targeted. They should test projection invariants directly, not replace integration coverage.
51
52	Production code should read time through Effect runtime APIs, such as `DateTime.now` and `Clock.currentTimeMillis`, not through `Date.now` or ad hoc wrappers. Production code should allocate random values through `effect/Random`, not through direct `crypto.randomUUID`, `Math.random`, or a custom global id generator.
53
54	Tests should provide Effect test services:
55
56	```ts
57	import { TestClock } from "effect/testing";
58	```
59
60	The id allocator can still expose domain-specific helpers such as `newRunId` or `newNodeId`, but those helpers should be implemented on top of `Random` so test layers can produce deterministic values without mocking orchestration logic.
61
62	## Generic Replay Runtime
63
64	Replay must be provider-neutral. Codex NDJSON fixtures are one provider's transcript format, not the framework itself.
65
66	```ts
67	type ProviderReplayTranscript = {
68	provider: ProviderKind;
69	protocol: string;
70	version: string;
71	scenario: string;
72	entries: ProviderReplayEntry[];
73	};
74
75	type ProviderReplayEntry =
76	| {
77	type: "expect_outbound";
78	label?: string;
79	frame: unknown;
80	}
81	| {
82	type: "emit_inbound";
83	label?: string;
84	frame: unknown;
85	afterMs?: number;
86	}
87	| {
88	type: "runtime_exit";
89	status: "success" | "error" | "cancelled";
90	error?: unknown;
91	};
92	```
93
94	The replay runtime owns deterministic transport semantics:
95
96	- ordered inbound event emission.
97	- outbound command assertion.
98	- pause/resume and timing control.
99	- runtime exit/error simulation.
100	- resume cursor/session restoration.
101	- transcript metadata validation.
102
103	The replay runtime must not know what a turn, plan, approval, subagent, or checkpoint means. Provider adapters interpret provider-specific frames.
104
105	Recovery tests use replay only at the provider transport boundary. App restart is tested by tearing down and recreating the outermost orchestrator/server layer against durable persistence. Idle cleanup and crash recovery are tested through production lifecycle services such as the session reaper or runtime recovery policy. Tests must not add adapter or orchestrator methods whose only purpose is to restart sessions for assertions.
106
107	## Provider Transcript Formats
108
109	Each provider can have its own raw frame format inside the generic replay envelope.
110
111	Examples:
112
113	```text
114	Codex replay transcript
115	-> JSON-RPC app-server requests/responses/notifications
116	-> consumed by CodexAdapter
117
118	Claude replay transcript
119	-> Claude Agent SDK query() outbound options and yielded SDKMessage chunks
120	-> consumed by ClaudeAdapter
121
122	Cursor replay transcript
123	-> Cursor Agent SDK open/send calls and ordered onDelta/run results
124	-> consumed by CursorAdapter
125
126	OpenCode replay transcript
127	-> OpenCode SDK requests/responses plus ordered SSE events
128	-> consumed by OpenCodeAdapter
129
130	ACP replay transcript
131	-> logical JSON-RPC requests/responses/notifications over a strict NDJSON peer
132	-> consumed by AcpAdapter and a provider flavor (Grok or ACP Registry)
133	```
134
135	Fixtures should preserve raw provider evidence as closely as possible. Expected V2 events or projections are assertions, not fixture input.
136
137	For Claude, the initial replay boundary is the async iterable returned by the Agent SDK `query()`
138	call. A transcript should include the `query` prompt/options we sent and then replay the raw
139	`SDKMessage` chunks in provider order. This intentionally tests the V2 adapter against real Claude
140	SDK output; it does not test the SDK's own subprocess or transport parser.
141
142	ACP fixtures run against a child-process replay peer that validates every outbound frame and
143	serves recorded inbound frames. This replaces only the external ACP driver transport; the shared
144	runtime, adapter normalization, orchestration, persistence, and projections remain production code.
145	The generic ACP Registry harness retargets protocol-standard ACP transcripts;
146	provider-extension transcripts remain scoped to the flavor that owns the extension.
147
148	OpenCode fixtures replace the SDK client at its HTTP/SSE boundary. They must preserve races between
149	request responses and SSE events, because user messages and terminal `session.status` events can
150	arrive before the corresponding `promptAsync` or `abort` response. Subagent fixtures must retain
151	both parent and child session ids so root-only terminal behavior remains testable.
152
153	Provider transcript recorders live with the server orchestration testkit, not with provider client
154	packages. Use `bun run record:codex-replay -- --scenario <name>` for Codex app-server transcripts
155	and `bun run record:claude-replay -- --scenario <name>` for Claude Agent SDK transcripts. Use
156	`pnpm --filter t3 record:cursor-replay --scenario <name>` for Cursor Agent SDK transcripts.
157
158	## Contract Test Levels
159
160	Recommended levels:
161
162	1. Schema tests for V2 contracts.
163	2. Pure projection tests for hard invariants.
164	3. Provider adapter replay tests from raw transcript to V2 domain events.
165	4. Full orchestration integration tests from commands through replay runtime to final projection.
166
167	The fourth level is the most important one. It is the test that catches lifecycle mismatches such as child turns closing parent runs or checkpoints being captured too early.
168
169	## First Ten Integration Tests
170
171	V2 should start with roughly ten strong tests:
172
173	1. `simple`: sending one message creates one run, one root node, one provider turn, one assistant response, and one root checkpoint.
174	2. `multi_turn`: follow-up messages create monotonically ordered app runs on the same app thread.
175	3. `message_steering`: steering attaches to the active run intent instead of becoming an unrelated run.
176	4. `turn_interrupt`: interrupt acknowledgement does not complete the run until the provider terminal event arrives.
177	5. `steering_restart_fallback`: a provider without native steering interrupts the active attempt and creates a replacement attempt under the same run.
178	6. `subagent`: child provider turns create nested execution nodes and never complete the parent run.
179	7. `subagent_checkpoint`: child/subagent nodes create nested checkpoint scopes without advancing the app run count.
180	8. `thread_rollback`: rollback targets checkpoint scopes and reconciles provider rollback snapshots.
181	9. `approval_request`: provider approval callbacks become durable runtime requests and are resolved through the real adapter path.
182	10. `provider_switch_return`: switching away from a provider creates a context handoff, and switching back resumes the prior provider thread with a delta handoff.
183
184	Additional tests should be added only when they protect a new invariant or reproduce a real failure mode.
185
186	## Fixture Rules
187
188	- Fixtures are raw provider transcripts, not mocked domain events.
189	- Fixtures should include enough outbound expectations to prove the app sent the correct provider commands.
190	- Fixtures should include protocol metadata, provider version, model, cwd policy, and capture timestamp.
191	- Fixture playback must be deterministic under Effect `TestClock` and deterministic `Random` layers.
192	- When a provider transcript is generated from a real run, keep the original provider frame ordering.
193	- Redaction should preserve ids, method names, lifecycle ordering, and correlation structure.
194	- Fixture transcripts are test input and diagnostic evidence. They do not imply production stores all raw provider frames in SQLite.
195
196	## Assertions
197
198	Assertions should prefer final projections and durable normalized orchestration events over incidental implementation calls.
199
200	Good assertions:
201
202	- duplicate command dispatch returns the original receipt sequence without replaying provider transport.
203	- stored-event sequence monotonicity.
204	- snapshot sequence plus stream-after-sequence behavior.
205	- run status and ordinal.
206	- active/final run attempt.
207	- execution node parent/child structure.
208	- provider thread and provider turn correlation.
209	- checkpoint scope hierarchy.
210	- pending/resolved runtime requests.
211	- handoff coverage and strategy.
212	- replay transcript frame count and normalized event count when relevant.
213
214	Weak assertions:
215
216	- exact internal function call counts.
217	- private helper invocation order.
218	- mocked callback arguments below the adapter/runtime boundary.
219
220	## Implementation Order
221
222	Testing infrastructure should be built before production rewrites:
223
224	1. V2 contract schemas.
225	2. Effect service definitions.
226	3. provider runtime transport abstraction.
227	4. generic replay runtime.
228	5. Codex, Claude, Cursor, OpenCode, and ACP transcript loaders for app-owned provider replay fixtures.
229	6. projection reducer tests for core invariants.
230	7. full command-to-projection integration tests.
231	8. production layers.
232
233	This keeps the architecture executable while it is being built and prevents tests from validating only simplified mocks.