authorclover caruso <git@paperclover.net> 2026-10-11 13:56:55-07:00
committerclover caruso <git@paperclover.net> 2026-10-11 14:20:01-07:00
log42a56c47989bd9409c1497b7afb0247620631117
tree9da113f2e9668a0d9114066d2e6b20d6bac61597
parenteda8a790491b2ff2ca460b8550a86b41dc5ec84b
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU (~clover)

docs: select resilient cloud takeover with retained branches

Specify branch-scoped ownership, retained recovery authority, causal coverage and reproducible integration gates. Record actual merge counterexamples and finite rollback repair evidence without claiming provider fencing or convergence. Refs #120 Assisted-by: gpt-6.1-sol

1 files changed, 260 insertions(+), 62 deletions(-)

arc/multi-file-sync.md+260-62
......@@ -12,10 +12,13 @@ between Snowbound clients, with one owner process publishing the real `.one`
1212files. Other builds exchange separate files instead of racing on those files.
1313Generic cloud mounts cannot be assumed to implement reliable distributed locks.
1414This should cover Android/Linux provider-backed notebooks and improve iCloud
15coordination. Practical tests must determine the tradeoffs before selecting an
16ownership/recovery policy.
15coordination. The selected direction is automatic takeover with retained branches
16and canonical repair. On 2026-10-11 Clover chose “whichever is the more resilient
17even if it's more complicated.” Resilience means keeping recoverable history and allowing an
18isolated capable device to progress; it does not mean promising exclusive cloud
19writes. The selection below is a design contract, not an implemented protocol.
1720
18## The contract under investigation
21## Selected contract
1922
2023```text
2124Android / Linux / iOS editor
......@@ -24,7 +27,7 @@ Android / Linux / iOS editor
2427.snowbound/ unique immutable request segments + blobs
2528 │ validated complete bytes and causal dependencies
2629 ▼
27owner existing merge and revision append
30branch owner existing merge and revision append
2831 ├── immutable decision/checkpoint evidence ──► other Snowbound replicas
2932 └── ordinary .one / .onetoc2 files ─────────► eventual provider projection
3033```
......@@ -36,15 +39,15 @@ logical owner are three different facts. The design must retain enough immutable
3639evidence to repair the mutable file after a rollback even if every client has
3740already received a historical publication receipt.
3841
39Proposed safety requirements:
42Required safety properties:
4043
4144- An acknowledged local edit remains in retained local storage until its content
4245 and outcome are recoverable elsewhere under an explicit retention contract.
4346- Seeing a request, seeing a receipt, or seeing a matching file stamp does not
4447 authorize deletion of the only recoverable request/base/blob.
4548- Missing, stale, incomplete, repeated, or reordered provider observations never
46 silently delete notes, select an owner, retire a branch, or establish a
47 complete list of devices.
49 silently delete notes, grant exclusive ownership, retire a branch, or
50 establish a complete list of devices.
4851- Every accepted request has a stable identity, complete validated payload,
4952 known causal base, and a recorded disposition. Retried transport does not
5053 repeat the user edit or create duplicate conflict pages.
......@@ -128,16 +131,44 @@ upload; the provider does not interpret that epoch. Chubby's sequencers address
128131precisely delayed operations by requiring validation at the recipient.
129132[The Chubby lock service](https://research.google.com/archive/chubby-osdi06.pdf).
130133
131**Investigation recommendation, not yet selected:** start with a fixed preferred
132publisher, explicit handoff, and immutable recoverable inputs/outcomes. Other
133clients keep editing from retained local replicas while that publisher is absent;
134publication waits. Preserve branches as the recovery fallback, and treat automatic
135takeover as separate policy work. Timeout takeover is not part of the strict
136owner contract. Majority agreement remains an alternative with its stated
137availability cost. This makes the sidecar necessary recovery authority for
138accepted history when the mutable projection regresses; that authority/backup
139contract needs owner review before production implementation. The recorded
140experiments below support the recovery requirement, not a finished owner protocol.
134**Selected policy: branch-preserving automatic takeover.** Prefer one discovered
135publisher during ordinary connected operation. An isolated capable client may
136start a fresh recovery branch from its retained validated base, without waiting
137for the preferred device or pretending to revoke it. A delay/heartbeat can decide
138when to attempt recovery, but cannot authorize deletion, invalidate another
139branch, or establish exclusive ownership. Explicit handoff remains the efficient
140connected path; a fixed publisher is not a prerequisite for progress. Do not add
141majority membership to the first version: its loss of offline availability buys
142logical serialization, while leaving the demonstrated physical rollback problem.
143
144| Resilience dimension | Selected guarantee / cost |
145| --- | --- |
146| Local edit custody | Existing durable queue plus retained exact dependencies; takeover never discards an absent device's work |
147| Isolated-device progress | A capable unlocked device can retain decisions and attempt publication from its complete local closure; missing ancestry/keys still block interpretation |
148| Accepted-history recovery | All valid accepted branches and pending requests survive reconciliation, including losing projection versions |
149| Canonical availability | Ordinary `.one` remains readable when valid; stale uploads can regress it, and repair can be delayed |
150| Eventual convergence | Requires complete eventual delivery, stale writes to cease, compatible keys/versions and a running reconciler; reproducible selection rules must pass the gates below |
151| Physical exclusion | Not available on this storage model; no timeout, handoff receipt or sidecar epoch supplies it |
152| Storage/backup cost | Sidecar recovery history is essential for registered notebooks; initial version has no automatic GC |
153
154“One owner writes the real file” becomes **one logical publisher per branch**,
155with one locally serialized process owning each live branch incarnation. Peers
156send requests to it while it is reachable. A takeover opens a separate branch;
157it does not seize the old branch's namespace or credentials. During partitions
158there can be several physical publishers, plus provider-delayed uploads from
159retired ones. On rejoin, discovered publishers stop adding independent work,
160retain their unfinished attempts and coordinate through a reconciler. Its
161preference/order is a contention reduction rule, not a global lock. No generic
162file-only protocol can guarantee both isolated takeover and exclusive physical
163writes under these assumptions.
164
165Accepted branch outcomes are never revoked because another branch was preferred.
166They may become conflict/recovery alternatives in the integrated result. A
167publication receipt names its branch and covered input identities; it cannot
168claim a globally final order or that every offline branch has been seen. The
169registered notebook's recovery authority is the validated retained history,
170including pending closures, rather than whichever `.one` snapshot arrived last.
171Bare-file imports and intentional restores remain explicit operations below.
141172
142173## Identities and duplicate processes
143174
......@@ -156,7 +187,7 @@ Use distinct roles:
156187| Process/writer incarnation | Fresh random identity for each process run; disjoint message namespace |
157188| Local edit identity | Existing durable replica edit ID plus its originating replica/writer identity |
158189| Request segment identity | Stable once sealed, retries resend identical bytes |
159| Owner epoch identity | Chained ownership decision, not a timestamp or largest visible number |
190| Owner epoch / branch identity | Unique branch incarnation plus exact parent frontier; no globally increasing epoch or revocation implied |
160191| Decision identity | Immutable owner result naming its predecessor and accepted request identities |
161192| Blob/checkpoint identity | Validated immutable bytes, content digest and exact byte length |
162193
......@@ -166,18 +197,23 @@ new records. Retrying an old outbox may write its original namespace/bytes; it
166197does not rekey old edits into the new process namespace. Otherwise deduplication
167198would turn an ordinary restart into a second user action.
168199
169Two processes on one installation must not both publish because they share a
170device ID. A locally owned publisher database/socket may serialize processes
171sharing reliable private storage. A second process with a copied private profile
172on another device is a different failure: distinguish incarnations, preserve
173both claims, and stop automatic publication when contradictory ownership becomes
174visible. Absence of a contradiction in a stale view is not a uniqueness proof.
175A backup of owner credentials is not permission to run both copies.
176
177For fixed ownership, restart uses the retained durable owner state and a fresh
178process incarnation. For explicit handoff, the old state records retirement
179before a successor becomes eligible. Losing the old durable state requires
180recovery; creating a fresh claim is not a transparent restart.
200Two processes on one installation must not publish the same live branch because
201they share a device ID. A locally owned publisher database/socket serializes
202processes sharing reliable private storage. Every process run uses its own new
203branch incarnation for newly admitted decisions, referencing retained predecessor
204history and unresolved attempts. Recovery can confirm an old prepared attempt
205with its original identities; it cannot generate a second outcome for it merely
206because the process restarted.
207
208Copied private profiles on separate devices therefore create distinct new
209branches, not competing continuations of one immutable namespace. Already sealed
210outbox objects retain their original identity and bytes. Two different decisions
211claiming the same immutable identity or contradictory dispositions for one exact
212prepared transaction remain a protocol contradiction: quarantine/preserve the
213evidence and stop automatic interpretation of the affected history. Valid
214independent branch claims are expected recovery inputs. A handoff records the
215old branch's retirement before its successor is eligible; stale cached retirement
216cannot be assumed visible elsewhere.
181217
182218## Proposed object namespace
183219
......@@ -332,12 +368,13 @@ publication. It cannot appear in an accepted edit list as though the note had
332368merged. A recoverable conflict outcome names the actual stored conflict page;
333369a retained foreign/unreadable branch names recovery evidence instead.
334370
335Owner decisions impose a durable selected order under the chosen ownership
336policy. Concurrent branches remain explicit inputs to the existing merge; do
337not choose the newest wall clock or discard the branch with the smaller owner
338ID. An automatic recovery policy needs a stable, documented reconciliation of
339multiple accepted decision histories. Deterministic sorting for processing is
340not a consensus or permission rule.
371Owner decisions impose an order within their branch. Concurrent branches remain
372explicit inputs to integration; never choose the newest wall clock or discard
373content because its owner ID sorts later. Stable reconciliation ordering chooses
374presentation and replay order among already validated inputs. It is neither
375consensus nor permission to edit. The same request accepted independently by two
376branches still denotes one user action; section revision identity alone is not a
377complete cross-branch request deduplication scheme.
341378
342379## Owner state and write/ack ordering
343380
......@@ -381,9 +418,10 @@ observation must not downgrade its known accepted history. Keep that history
381418and recovery dependencies locally while repairing the projection. An explicit
382419intentional restore is a new recorded operation, not a stale-file inference.
383420
384An owner-unavailable client still edits from its private replica. Requests wait
385for owner processing without blocking local read/navigation/Undo. Publishing
386status cannot report owner custody merely because local creation of a provider
421An owner-unavailable client still reads, navigates, edits and uses Undo from its
422private replica. A capable client may start a recovery branch; otherwise requests
423wait for a capable publisher without blocking local use. Publishing status cannot
424report another device's custody merely because local creation of a provider
387425object succeeded.
388426
389427## Explicit handoff, recovery takeover, and late uploads
......@@ -407,27 +445,136 @@ not prove every cloud cache drained. Waiting an arbitrary duration is not a
407445fence. A fixed owner with no ownership transitions avoids one source of late
408446writes but still needs rollback repair for reordered uploads from its own run.
409447
410Automatic recovery experimentation must keep each candidate's decisions in its
411own immutable branch. A candidate cannot overwrite another owner's mailbox or
412checkpoint, and claims no exclusive owner authority based on silence. The
413mutable `.one` projection may temporarily regress or produce provider conflict
414versions. A later reconciler imports all valid branches, preserves incompatible
415results as conflict pages/recovery data, and republishes normal revisions.
416This is a weaker, explicit safety/availability trade than a strict global
417single publisher, and remains unselected pending the experiments.
448Automatic takeover keeps each candidate's decisions in its own immutable branch.
449Before admitting work it durably records the exact validated parent frontier,
450base/checkpoint closure, its fresh incarnation and any uncertain attempts. Missing
451dependencies remain pending; a candidate cannot claim complete recovery from an
452incomplete listing. It cannot overwrite another owner's mailbox/checkpoint or
453claim exclusive authority based on silence. A branch may progress with a complete
454known subset while clearly retaining that subset's identity; it must not label
455it the complete notebook history.
456
457An old process resuming after takeover must inspect discovered branch/handoff
458state before new publication, preserve its old outbox, and join reconciliation
459when it can. A stale view can still hide that state, so this check is best effort
460contention control. The safety mechanism is retained branches and attempts,
461not an assertion that the old process can always be stopped. Ordinary canonical
462`.one` files may regress or produce provider conflict versions until reconciliation
463and delivery settle.
418464
419465A quorum alternative can choose an ownership/decision chain if acceptor
420466membership and durable promises are implemented. It still needs immutable
421467recovery and projection repair unless the actual provider enforces epochs on
422468writes. Do not add a quorum merely to hide this physical fencing limitation.
423469
470## Reconciliation and canonical repair
471
472The selected policy needs a decision DAG, not a fabricated globally monotonic
473epoch. Each branch decision names its predecessor, exact causal input frontier,
474prepared transaction/outcome and complete recovery closure. An integration
475decision names every covered source request/branch outcome and retained unresolved
476alternative. Its references can arrive in any order. Validation waits for the
477referenced closure; it never invents missing content from visible filenames.
478
479```text
480retained seed ──► A decisions ──► A checkpoint ──┐
481 └► B decisions ──► B checkpoint ──┼► integration plan / checkpoint
482 └► offline pending queue ─────────┘ │
483 ordinary append / repair
484 ▼
485 mutable .one path
486```
487
488A candidate implements this sequence durably:
489
4901. Gather validated known branch frontiers, pending requests, external versions
491 and uncertain attempts. Follow references, preserving exact source bytes;
492 listing completeness is never asserted. Keep the last valid local view if a
493 newly observed frontier is incomplete, malformed, foreign or unsupported.
4942. Resolve exact prepared-attempt identities before admitting a retry. Build a
495 coverage map from `(originating replica, edit ID)` to source request digest,
496 branch dispositions, named revisions and recovery closure. Identical transport
497 retries are one action. Independently rebased outcomes of that action are not
498 automatically identical: retain both, select one semantic application and
499 record the other as an alternate or retained-unresolved. Selecting one outcome
500 cannot discard its alternate's dependent descendants or reinterpret their ops
501 against the chosen base. Require validated causal translation/rebase; otherwise
502 retain the complete alternate-and-descendant closure unresolved. Contradictory
503 bytes for one immutable identity halt automatic interpretation of that history.
5043. Form a reproducible integration plan for the **complete known input set**.
505 Its identity covers notebook lineage, named seed/base, source frontier and
506 coverage, merge/codec version, stable causal processing order and all outcome
507 dependencies. Dependency order comes first, then a specified byte order of
508 immutable source identities for concurrent inputs. Clocks, directory order,
509 local arrival order and largest epoch are excluded. A newly discovered input
510 makes a new plan; it cannot make the old plan evidence disappear.
5114. For incomparable histories, choose the same retained primary checkpoint under
512 that ordering, then rebase the remaining original inputs through existing
513 ops/retained-version merge. Preserve conflict pages and every opaque/unmergeable
514 version. Do not integrate “whatever this device's mutable file currently is”
515 as an unspecified primary. If normal incremental integration depends on
516 arrival order, reproduce the plan from its named retained raw base in a
517 disposable section and append normal transactions. No page-model reconstruction
518 or whole-page diff/save shortcut is allowed. This may cost more I/O.
5195. Persist exact prepared revisions, input coverage, result raw section/catalog
520 checkpoint and external blobs before issuing a recoverability promise. An
521 integration checkpoint is a result over source inputs, not another independent
522 edit. Subsequent branches carry inherited coverage; replaying integrations as
523 new source edits must not multiply conflicts. Same plan/outcome retries reuse
524 their identities. Existing named-revision dedup alone does not solve every
525 independently rebased or differently ordered plan.
5266. Confirm/publish using the actual adapter's append and ambiguity rules. If a
527 valid projection has rolled back, retain those bytes, import unknown external
528 revisions as separate inputs, then restore the chosen raw checkpoint or append
529 repair revisions as the adapter requires. Copying a validated raw checkpoint
530 repairs a materialized store; it is not recreating notes from a page model.
531 Retain the checkpoint even after local confirmation. Receipt coverage names
532 the integrated inputs and retained alternatives, not a globally complete view.
5337. Recheck after publication and on later observations. A delayed stale upload
534 starts another repair with the same retained history; it does not reapply user
535 edits. Once a fresh replica sees the chosen checkpoint and the same inputs,
536 another reconciliation must produce no extra edit/conflict revisions. A
537 publisher that cannot satisfy these steps retains work without claiming a
538 recoverable integrated outcome.
539
540Stable ordering is a proposed integration mechanism to validate, not a claim that
541today's `resolve::merge` is commutative. The existing merge keeps the current
542branch as the main page when edits overlap. Its deterministic revision names help
543retry an original branch, but neither equal names nor presence of both texts
544proves equal primary/conflict placement, causal coverage or byte-identical files.
545Plan equivalence must compare source/outcome coverage, main and conflict page
546identities/content, tombstones, catalog and blob closure. Physical allocation,
547header bookkeeping and encoding can differ without a semantic disagreement;
548such differences must not trigger endless republishing.
549
550The implementation gate is stronger than static sorted input testing: two fresh
551replicas receiving the same branches in opposite orders, including intermediate
552integration checkpoints and new edits based on them, must converge to equivalent
553covered outcomes without duplicate conflict pages or silently repeated requests.
554Test at least three concurrent branches, ancestor/descendant arrivals, one request
555accepted on two branches, namespace operations and protected/opaque payloads.
556If the current merge cannot satisfy this, retain unresolved alternatives and fix
557the integration algorithm before claiming automatic convergence. Recovery custody
558can remain safe while automatic integration is unavailable.
559
560Eventual repair is conditional. For a finite settled input set, its complete
561closures must eventually reach a compatible unlocked reconciler; stale uploads
562must eventually stop replacing the path; and a successful repair must eventually
563be visible. A deterministic plan alone cannot guarantee these properties.
564Repeated infinite old uploads are an explicit liveness counterexample. No finite
565number of quiet reads proves that the last stale upload has arrived. Lack of
566quiescence must consume retry/backoff budget, not user history or retained bases.
567
424568## Authority and disaster recovery
425569
426The `.one` files remain the interoperable materialized section format. The
427sidecar must preserve accepted history beyond those mutable files if it is to
428repair demonstrated rollbacks. This is a proposed recovery requirement, not a
429silent declaration that an arbitrary `.snowbound` folder always overrides every
430`.one` file found beside it.
570The `.one` files remain the interoperable materialized section format. For an
571explicitly registered coordinated notebook, validated retained sidecar history
572and its complete closures are authoritative for accepted/pending work when
573mutable files regress. Registration binds notebook lineage, section identities,
574protocol version and initial recovery closure; an arbitrary adjacent `.snowbound`
575folder has no authority over a bare import or a different section at the same
576path. This changes the backup contract for the selected mode, not the `.one`
577format or native/SMB modes.
431578
432579Distinguish three recovery operations:
433580
......@@ -542,9 +689,9 @@ checkpoint does not automatically include external payload files.
542689
543690## Retention, compaction, and device rejoin
544691
545The first safe implementation can retain immutable logs/checkpoints without
546automatic deletion. Storage use is real, so measure it and design compaction;
547finite retention must not be disguised as a correctness proof.
692The first implementation retains immutable logs/checkpoints without automatic
693deletion. Storage use is real, so measure it and design compaction; finite retention
694must not be disguised as a correctness proof.
548695
549696No “all devices have seen it” rule can be inferred from current listings or
550697heartbeats. An offline device may hold unpublished edits indefinitely. Deleting
......@@ -676,6 +823,56 @@ without copying or changing them. The fixture is a disposable read of
676823`corpus/conflict-page/native/initial/notebook/synthetic.one`; its input hash is
677824recorded. No commercial provider, emulator, or OneNote process ran in this lab.
678825
826### Resilience policy follow-up
827
828The 2026-10-11 follow-up under
829`target/multi-file-sync-resilience-20261011/` tests the selected policy's recovery
830primitives and exposes limits of the current merge. Run 2 completed with exit 0
831in 2.4 seconds after the initial compilation populated the shared cache. The
832first attempt failed to compile a lab-only immutable binding used by
833`Section::conflicts()`; its exact source/log/result remain preserved. Neither
834attempt changes production code.
835
836| Experiment | Observed outcome |
837| --- | --- |
838| Same overlapping ALICE/BOB branches, opposite primary bases | Both alternatives survive, but main/conflict placement differs; three replays of each original branch add zero publications and preserve exact bytes |
839| Cross-deliver those opposite integration artifacts directly to current merge | Both directions publish again, flip the primary text and retain two conflict pages, including an alternative repeating current main text; naïve integration-result exchange is not convergence |
840| Sort the same complete two-branch set before choosing a retained primary | Both arrival orders produce the same main/conflict identities and text; raw bytes and some active revision identities still differ |
841| Five successive stale canonical replacements, alternating seed/A/B images | Every finite round repairs to exact ALPHA BETA using retained originals/checkpoint; repair costs 2/1/1/2/1 publications, and settled same-database/fresh-replica retries add zero |
842| Fresh local recovery with a truncated canonical image | Cold parser rejects the image; retained raw source sections recover exact ALPHA BETA; observing only B lacks ALPHA and cannot establish complete history |
843
844Ten SQLite files pass integrity checks. Each saved valid `.one` is cold parsed
845through the existing store/index/document reader with checksum/current-revision
846validation; actual conflict pages are inspected separately from the ordinary
847page. The immutable archive stays unchanged across repair. These are disposable
848copies of the prior native-corpus fixture and actual production
849`Replica`/`resolve::merge`/transaction calls; provider replacements, visibility,
850source ordering and logical database reopen are simulated by the lab.
851
852The sorted-set result proves only a bounded presentation property for this
853complete two-branch fixture. It does not prove equivalent opaque/protected
854content, downstream causal translation, arbitrary incremental DAG convergence,
855namespace operations, request coverage deduplication, cloud custody or OS-crash
856durability. The opposite-integration result is why the selected algorithm must
857track source coverage and normalize from retained inputs instead of feeding
858integration artifacts back as new independent edits. No automatic convergence
859claim is justified by this investigation.
860
861Rerun from a clean new output directory with one granted build slot:
862
863```sh
864nice -n 10 env CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target" cargo run --offline \
865 --manifest-path target/multi-file-sync-resilience-20261011/lab/Cargo.toml -- \
866 "$PWD/target/multi-file-sync-resilience-20261011/run-new"
867```
868
869The ignored standalone lab links the current source in
870`workspaces/sync-resilience` and reads three bound `.one` inputs from the original
871run 7 without modifying them. Its verification manifest binds lab/dependency
872sources, inputs, raw outputs, SQLite integrity checks, draft architecture review
873and document checks. No commercial provider, emulator or OneNote process runs
874in this follow-up. Native OneNote cold-open remains an implementation gate.
875
679876| Fault | Required result / investigation oracle |
680877| --- | --- |
681878| Prefix/partial body, invalid length/digest | Reject/wait; last valid state untouched |
......@@ -713,14 +910,15 @@ recovery and normal append operations. Platform hosts adapt document/file APIs,
713910retained local directories and actual I/O/error callbacks. Android rendering and
714911native IME/accessibility can ship independently of this protocol.
715912
716Before production protocol code, review practical evidence and select:
913The owner has selected resilient branch-preserving takeover and retained recovery
914authority for registered coordinated notebooks. Before production protocol code,
915validate the reconciliation/receipt algorithm against the gates above and define:
717916
718- Fixed publisher/explicit handoff versus branch-preserving automatic recovery,
719 and whether majority membership offers enough benefit for its availability cost.
720- The authority/recovery contract for registered notebooks, bare imports,
721 intentional restores, duplicate notebook copies and copied owner credentials.
917- Explicit registration, bare import, intentional restore and duplicate-copy flows
918 that preserve the stated lineage/custody boundaries.
722919- Protected-key custody and unsupported-client migration/escape behavior.
723- Retention/compaction guarantees and a measured provider capability floor.
920- Measured provider capability floor and storage cost. Initial automatic GC is
921 excluded; later compaction requires its own recovery proof.
724922
725923Unknown protocol/required-op versions must leave remote data untouched and keep
726924local edits recoverable. Enabling coordination requires an explicit migration
......@@ -730,6 +928,6 @@ needs an honest compatibility boundary and an opt-in/import path, not a claim
730928that an unsupported build will obey a new marker.
731929
732930The next concrete deliverable after the draft is a narrow transport/owner lab
733against real provider fixtures, then one selected policy implemented behind an
931against real provider fixtures, then the selected policy implemented behind an
734932explicit storage mode and tested with actual notes. No timeout election or
735933full-image save shortcut should land as an Android dependency.