clover's git

the-knife

1	# Server updates
2
3	A [stable launcher](../../apps/server/src/serviceLauncher.ts) owns the runtime
4	selected by systemd or launchd. It is the only runtime writer of durable service
5	state. Server children request updates over inherited IPC; they never rewrite
6	their service definition or select their own replacement. Local service commands
7	may replace the launcher and state while the service is stopped. Foreground CLI
8	processes do not self-update.
9
10	Exact-version installs keep restarts independent of npm cache eviction or a moving
11	release tag. Installation and preflight happen in staging before publishing an
12	immutable runtime. Preflight checks the launcher protocol because a target that
13	needs new rollback guarantees cannot safely run under an older launcher. Upgrading
14	that launcher requires a local service update.
15
16	## Commit boundary
17
18	The launcher durably records the pending update before acknowledging it, then
19	stops the old child and starts the target as a trial. Service-state writes use
20	same-directory replacement with file and directory fsync. Invalid state stops
21	startup rather than guessing which runtime to boot.
22
23	The trial must finish migrations, acquire dependencies, bind HTTP, and park every
24	long-running root at the activation gate before reporting `prepared`. The launcher
25	then commits the target version durably and replies `committed`. Only then may the
26	child release its gates, accept commands, and publish ready. Keep fallible startup
27	acquisitions before this boundary. A listener alone does not prove the runtime is
28	ready to commit.
29
30	A failed or timed-out trial returns to the old version. After commit, the target
31	is authoritative and the service manager's ordinary restart policy applies.
32
33	The prepared deadline is long because migrations and SQLite's open-time optimize
34	scale with the database and block the trial's main thread, so it cannot report
35	progress. A missed deadline restores the snapshot, and the retry pays the same
36	cost again. A crashed trial is caught when it exits; the deadline bounds only a
37	stuck one. Client update waits must outlast the deadline, or a slow update shows
38	as failed.
39
40	## Database rollback
41
42	Before stopping the old child, the launcher refuses an update whose snapshot will
43	not fit on disk, so a full disk does not cause an outage. After the old child
44	exits, the launcher snapshots SQLite's main file, WAL, and shared-memory file. This makes trial migrations reversible without down
45	migrations. The snapshot is made once per update and survives launcher restarts;
46	replacing it during a retry could capture changes from the failed trial.
47
48	Rollback stops the trial before restoring. A durable restore marker makes an
49	interrupted restore finish before either version boots. Keep the snapshot until
50	commit, or until both restoration and the terminal rollback state are durable.
51	Attachments and other files outside SQLite are outside this rollback boundary.
52
53	## Client acknowledgement
54
55	An accepted update is still pending. Clients correlate the launcher's update ID
56	with the ready event after reconnecting, then check the outcome and target version.
57	A reconnect alone cannot distinguish successful replacement from rollback. Older
58	servers without an update ID retain version-only correlation.
59
60	Desktop updates have a separate two-phase handoff because installing the app stops
61	its bundled backend. Preparation returns a token while the connection is alive;
62	the client commits that token only after receiving it. Otherwise backend shutdown
63	could lose the only successful RPC result. The client must then observe the
64	prepared version after reconnecting. If installation fails, desktop restarts the
65	stopped backends and replays the failure for the same token.
66
67	## Recovering interrupted threads
68
69	Restart continuation is an environment-owned preference, on by default. A
70	`settings.json` that fails to decode turns it off, since the file may hold an
71	opt-out. The
72	[v2 recovery service](../../apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts)
73	requires matching durable run, provider thread, session, and native resume identity.
74	Queued runs never started, so recovery holds them and continues the run they wait
75	behind. A settled run qualifies only when the restart cancels work the agent waits on
76	(a monitor, subagent, or background task) and its session was live, so stale records
77	from an older crash never wake it. Its continuation prompt is the note of what will not
78	report back. A shell alone does not wake a settled run: most are dev servers nobody
79	waits on, so that note waits for the next user message.
80
81	Recovery retires effects tied to the lost process and records continuation intent
82	in the durable outbox. That intent survives another restart before provider startup.
83	Continuation effects wait for activation; a slow provider must not delay the server's
84	readiness or the launcher's commit boundary. Graceful shutdown captures intent before
85	closing providers, then reconciles after ingestion has stopped so a late completion
86	cannot be overwritten by a stale cancellation.
87
88	The [continuation handler](../../apps/server/src/orchestration-v2/RestartContinuation.ts)
89	rechecks the preference, provider selection, newer user work, a stop the user requested,
90	and maintenance turns such as `/compact` before dispatching. Intent and delivery both
91	refuse a thread that is archived, settled, or still snoozed, a cut turn whose provider
92	turn already ended, and a run waiting on the user: an approval, question, or secret that was pending,
93	or that recovery closed on the cut turn. Message-capable questions stay answerable, so
94	they do not block. Delivery also stops after three consecutive continuations in one
95	`restartContinuationOfRunId` chain, so a turn that crashes the server cannot loop. Stable
96	command and message IDs prevent duplicate submissions after an outbox retry. Codex
97	resumes without adding provider prompt text unless the turn lost background work;
98	other adapters receive the continuation message through their normal turn path.
99
100	Delegated tasks (`delegate_task` child threads) are reconciled as their own threads,
101	never as the parent's background work. The orchestrator settles child results and
102	completion deliveries in a startup pass after reconciliation, because the terminal-run
103	listener ignores reconciliation's cancellations. A cancelled child whose restart
104	continuation is still pending in the outbox is not a result yet; the continuation's run
105	settles it, or the handler settles it when it declines to continue. Schedulers wait for
106	activation so they cannot start runs that reconciliation would then cancel.