From 42a56c47989bd9409c1497b7afb0247620631117 Mon Sep 17 00:00:00 2001 From: clover caruso Date: Sun, 11 Oct 2026 13:56:55 -0700 Subject: [PATCH] docs: select resilient cloud takeover with retained branches Specify branch-scoped ownership, retained recovery authority, causal coverage and reproducible integration gates. Record actual merge counterexamples and finite rollback repair evidence without claiming provider fencing or convergence. Refs #120 Assisted-by: gpt-6.1-sol --- arc/multi-file-sync.md | 320 +++++++++++++++++++++++++++++++++-------- 1 file changed, 259 insertions(+), 61 deletions(-) diff --git a/arc/multi-file-sync.md b/arc/multi-file-sync.md index 8bc1ed44ae775f94e677348e63e214445855c65d..7a363f58f89f6f43353d9ab1bf9ce845e811cb49 100644 --- a/arc/multi-file-sync.md +++ b/arc/multi-file-sync.md @@ -12,10 +12,13 @@ between Snowbound clients, with one owner process publishing the real `.one` files. Other builds exchange separate files instead of racing on those files. Generic cloud mounts cannot be assumed to implement reliable distributed locks. This should cover Android/Linux provider-backed notebooks and improve iCloud -coordination. Practical tests must determine the tradeoffs before selecting an -ownership/recovery policy. +coordination. The selected direction is automatic takeover with retained branches +and canonical repair. On 2026-10-11 Clover chose “whichever is the more resilient +even if it's more complicated.” Resilience means keeping recoverable history and allowing an +isolated capable device to progress; it does not mean promising exclusive cloud +writes. The selection below is a design contract, not an implemented protocol. -## The contract under investigation +## Selected contract ```text Android / Linux / iOS editor @@ -24,7 +27,7 @@ Android / Linux / iOS editor .snowbound/ unique immutable request segments + blobs │ validated complete bytes and causal dependencies ▼ -owner existing merge and revision append +branch owner existing merge and revision append ├── immutable decision/checkpoint evidence ──► other Snowbound replicas └── ordinary .one / .onetoc2 files ─────────► eventual provider projection ``` @@ -36,15 +39,15 @@ logical owner are three different facts. The design must retain enough immutable evidence to repair the mutable file after a rollback even if every client has already received a historical publication receipt. -Proposed safety requirements: +Required safety properties: - An acknowledged local edit remains in retained local storage until its content and outcome are recoverable elsewhere under an explicit retention contract. - Seeing a request, seeing a receipt, or seeing a matching file stamp does not authorize deletion of the only recoverable request/base/blob. - Missing, stale, incomplete, repeated, or reordered provider observations never - silently delete notes, select an owner, retire a branch, or establish a - complete list of devices. + silently delete notes, grant exclusive ownership, retire a branch, or + establish a complete list of devices. - Every accepted request has a stable identity, complete validated payload, known causal base, and a recorded disposition. Retried transport does not repeat the user edit or create duplicate conflict pages. @@ -128,16 +131,44 @@ upload; the provider does not interpret that epoch. Chubby's sequencers address precisely delayed operations by requiring validation at the recipient. [The Chubby lock service](https://research.google.com/archive/chubby-osdi06.pdf). -**Investigation recommendation, not yet selected:** start with a fixed preferred -publisher, explicit handoff, and immutable recoverable inputs/outcomes. Other -clients keep editing from retained local replicas while that publisher is absent; -publication waits. Preserve branches as the recovery fallback, and treat automatic -takeover as separate policy work. Timeout takeover is not part of the strict -owner contract. Majority agreement remains an alternative with its stated -availability cost. This makes the sidecar necessary recovery authority for -accepted history when the mutable projection regresses; that authority/backup -contract needs owner review before production implementation. The recorded -experiments below support the recovery requirement, not a finished owner protocol. +**Selected policy: branch-preserving automatic takeover.** Prefer one discovered +publisher during ordinary connected operation. An isolated capable client may +start a fresh recovery branch from its retained validated base, without waiting +for the preferred device or pretending to revoke it. A delay/heartbeat can decide +when to attempt recovery, but cannot authorize deletion, invalidate another +branch, or establish exclusive ownership. Explicit handoff remains the efficient +connected path; a fixed publisher is not a prerequisite for progress. Do not add +majority membership to the first version: its loss of offline availability buys +logical serialization, while leaving the demonstrated physical rollback problem. + +| Resilience dimension | Selected guarantee / cost | +| --- | --- | +| Local edit custody | Existing durable queue plus retained exact dependencies; takeover never discards an absent device's work | +| Isolated-device progress | A capable unlocked device can retain decisions and attempt publication from its complete local closure; missing ancestry/keys still block interpretation | +| Accepted-history recovery | All valid accepted branches and pending requests survive reconciliation, including losing projection versions | +| Canonical availability | Ordinary `.one` remains readable when valid; stale uploads can regress it, and repair can be delayed | +| Eventual convergence | Requires complete eventual delivery, stale writes to cease, compatible keys/versions and a running reconciler; reproducible selection rules must pass the gates below | +| Physical exclusion | Not available on this storage model; no timeout, handoff receipt or sidecar epoch supplies it | +| Storage/backup cost | Sidecar recovery history is essential for registered notebooks; initial version has no automatic GC | + +“One owner writes the real file” becomes **one logical publisher per branch**, +with one locally serialized process owning each live branch incarnation. Peers +send requests to it while it is reachable. A takeover opens a separate branch; +it does not seize the old branch's namespace or credentials. During partitions +there can be several physical publishers, plus provider-delayed uploads from +retired ones. On rejoin, discovered publishers stop adding independent work, +retain their unfinished attempts and coordinate through a reconciler. Its +preference/order is a contention reduction rule, not a global lock. No generic +file-only protocol can guarantee both isolated takeover and exclusive physical +writes under these assumptions. + +Accepted branch outcomes are never revoked because another branch was preferred. +They may become conflict/recovery alternatives in the integrated result. A +publication receipt names its branch and covered input identities; it cannot +claim a globally final order or that every offline branch has been seen. The +registered notebook's recovery authority is the validated retained history, +including pending closures, rather than whichever `.one` snapshot arrived last. +Bare-file imports and intentional restores remain explicit operations below. ## Identities and duplicate processes @@ -156,7 +187,7 @@ Use distinct roles: | Process/writer incarnation | Fresh random identity for each process run; disjoint message namespace | | Local edit identity | Existing durable replica edit ID plus its originating replica/writer identity | | Request segment identity | Stable once sealed, retries resend identical bytes | -| Owner epoch identity | Chained ownership decision, not a timestamp or largest visible number | +| Owner epoch / branch identity | Unique branch incarnation plus exact parent frontier; no globally increasing epoch or revocation implied | | Decision identity | Immutable owner result naming its predecessor and accepted request identities | | Blob/checkpoint identity | Validated immutable bytes, content digest and exact byte length | @@ -166,18 +197,23 @@ new records. Retrying an old outbox may write its original namespace/bytes; it does not rekey old edits into the new process namespace. Otherwise deduplication would turn an ordinary restart into a second user action. -Two processes on one installation must not both publish because they share a -device ID. A locally owned publisher database/socket may serialize processes -sharing reliable private storage. A second process with a copied private profile -on another device is a different failure: distinguish incarnations, preserve -both claims, and stop automatic publication when contradictory ownership becomes -visible. Absence of a contradiction in a stale view is not a uniqueness proof. -A backup of owner credentials is not permission to run both copies. +Two processes on one installation must not publish the same live branch because +they share a device ID. A locally owned publisher database/socket serializes +processes sharing reliable private storage. Every process run uses its own new +branch incarnation for newly admitted decisions, referencing retained predecessor +history and unresolved attempts. Recovery can confirm an old prepared attempt +with its original identities; it cannot generate a second outcome for it merely +because the process restarted. -For fixed ownership, restart uses the retained durable owner state and a fresh -process incarnation. For explicit handoff, the old state records retirement -before a successor becomes eligible. Losing the old durable state requires -recovery; creating a fresh claim is not a transparent restart. +Copied private profiles on separate devices therefore create distinct new +branches, not competing continuations of one immutable namespace. Already sealed +outbox objects retain their original identity and bytes. Two different decisions +claiming the same immutable identity or contradictory dispositions for one exact +prepared transaction remain a protocol contradiction: quarantine/preserve the +evidence and stop automatic interpretation of the affected history. Valid +independent branch claims are expected recovery inputs. A handoff records the +old branch's retirement before its successor is eligible; stale cached retirement +cannot be assumed visible elsewhere. ## Proposed object namespace @@ -332,12 +368,13 @@ publication. It cannot appear in an accepted edit list as though the note had merged. A recoverable conflict outcome names the actual stored conflict page; a retained foreign/unreadable branch names recovery evidence instead. -Owner decisions impose a durable selected order under the chosen ownership -policy. Concurrent branches remain explicit inputs to the existing merge; do -not choose the newest wall clock or discard the branch with the smaller owner -ID. An automatic recovery policy needs a stable, documented reconciliation of -multiple accepted decision histories. Deterministic sorting for processing is -not a consensus or permission rule. +Owner decisions impose an order within their branch. Concurrent branches remain +explicit inputs to integration; never choose the newest wall clock or discard +content because its owner ID sorts later. Stable reconciliation ordering chooses +presentation and replay order among already validated inputs. It is neither +consensus nor permission to edit. The same request accepted independently by two +branches still denotes one user action; section revision identity alone is not a +complete cross-branch request deduplication scheme. ## Owner state and write/ack ordering @@ -381,9 +418,10 @@ observation must not downgrade its known accepted history. Keep that history and recovery dependencies locally while repairing the projection. An explicit intentional restore is a new recorded operation, not a stale-file inference. -An owner-unavailable client still edits from its private replica. Requests wait -for owner processing without blocking local read/navigation/Undo. Publishing -status cannot report owner custody merely because local creation of a provider +An owner-unavailable client still reads, navigates, edits and uses Undo from its +private replica. A capable client may start a recovery branch; otherwise requests +wait for a capable publisher without blocking local use. Publishing status cannot +report another device's custody merely because local creation of a provider object succeeded. ## Explicit handoff, recovery takeover, and late uploads @@ -407,27 +445,136 @@ not prove every cloud cache drained. Waiting an arbitrary duration is not a fence. A fixed owner with no ownership transitions avoids one source of late writes but still needs rollback repair for reordered uploads from its own run. -Automatic recovery experimentation must keep each candidate's decisions in its -own immutable branch. A candidate cannot overwrite another owner's mailbox or -checkpoint, and claims no exclusive owner authority based on silence. The -mutable `.one` projection may temporarily regress or produce provider conflict -versions. A later reconciler imports all valid branches, preserves incompatible -results as conflict pages/recovery data, and republishes normal revisions. -This is a weaker, explicit safety/availability trade than a strict global -single publisher, and remains unselected pending the experiments. +Automatic takeover keeps each candidate's decisions in its own immutable branch. +Before admitting work it durably records the exact validated parent frontier, +base/checkpoint closure, its fresh incarnation and any uncertain attempts. Missing +dependencies remain pending; a candidate cannot claim complete recovery from an +incomplete listing. It cannot overwrite another owner's mailbox/checkpoint or +claim exclusive authority based on silence. A branch may progress with a complete +known subset while clearly retaining that subset's identity; it must not label +it the complete notebook history. + +An old process resuming after takeover must inspect discovered branch/handoff +state before new publication, preserve its old outbox, and join reconciliation +when it can. A stale view can still hide that state, so this check is best effort +contention control. The safety mechanism is retained branches and attempts, +not an assertion that the old process can always be stopped. Ordinary canonical +`.one` files may regress or produce provider conflict versions until reconciliation +and delivery settle. A quorum alternative can choose an ownership/decision chain if acceptor membership and durable promises are implemented. It still needs immutable recovery and projection repair unless the actual provider enforces epochs on writes. Do not add a quorum merely to hide this physical fencing limitation. +## Reconciliation and canonical repair + +The selected policy needs a decision DAG, not a fabricated globally monotonic +epoch. Each branch decision names its predecessor, exact causal input frontier, +prepared transaction/outcome and complete recovery closure. An integration +decision names every covered source request/branch outcome and retained unresolved +alternative. Its references can arrive in any order. Validation waits for the +referenced closure; it never invents missing content from visible filenames. + +```text +retained seed ──► A decisions ──► A checkpoint ──┐ + └► B decisions ──► B checkpoint ──┼► integration plan / checkpoint + └► offline pending queue ─────────┘ │ + ordinary append / repair + ▼ + mutable .one path +``` + +A candidate implements this sequence durably: + +1. Gather validated known branch frontiers, pending requests, external versions + and uncertain attempts. Follow references, preserving exact source bytes; + listing completeness is never asserted. Keep the last valid local view if a + newly observed frontier is incomplete, malformed, foreign or unsupported. +2. Resolve exact prepared-attempt identities before admitting a retry. Build a + coverage map from `(originating replica, edit ID)` to source request digest, + branch dispositions, named revisions and recovery closure. Identical transport + retries are one action. Independently rebased outcomes of that action are not + automatically identical: retain both, select one semantic application and + record the other as an alternate or retained-unresolved. Selecting one outcome + cannot discard its alternate's dependent descendants or reinterpret their ops + against the chosen base. Require validated causal translation/rebase; otherwise + retain the complete alternate-and-descendant closure unresolved. Contradictory + bytes for one immutable identity halt automatic interpretation of that history. +3. Form a reproducible integration plan for the **complete known input set**. + Its identity covers notebook lineage, named seed/base, source frontier and + coverage, merge/codec version, stable causal processing order and all outcome + dependencies. Dependency order comes first, then a specified byte order of + immutable source identities for concurrent inputs. Clocks, directory order, + local arrival order and largest epoch are excluded. A newly discovered input + makes a new plan; it cannot make the old plan evidence disappear. +4. For incomparable histories, choose the same retained primary checkpoint under + that ordering, then rebase the remaining original inputs through existing + ops/retained-version merge. Preserve conflict pages and every opaque/unmergeable + version. Do not integrate “whatever this device's mutable file currently is” + as an unspecified primary. If normal incremental integration depends on + arrival order, reproduce the plan from its named retained raw base in a + disposable section and append normal transactions. No page-model reconstruction + or whole-page diff/save shortcut is allowed. This may cost more I/O. +5. Persist exact prepared revisions, input coverage, result raw section/catalog + checkpoint and external blobs before issuing a recoverability promise. An + integration checkpoint is a result over source inputs, not another independent + edit. Subsequent branches carry inherited coverage; replaying integrations as + new source edits must not multiply conflicts. Same plan/outcome retries reuse + their identities. Existing named-revision dedup alone does not solve every + independently rebased or differently ordered plan. +6. Confirm/publish using the actual adapter's append and ambiguity rules. If a + valid projection has rolled back, retain those bytes, import unknown external + revisions as separate inputs, then restore the chosen raw checkpoint or append + repair revisions as the adapter requires. Copying a validated raw checkpoint + repairs a materialized store; it is not recreating notes from a page model. + Retain the checkpoint even after local confirmation. Receipt coverage names + the integrated inputs and retained alternatives, not a globally complete view. +7. Recheck after publication and on later observations. A delayed stale upload + starts another repair with the same retained history; it does not reapply user + edits. Once a fresh replica sees the chosen checkpoint and the same inputs, + another reconciliation must produce no extra edit/conflict revisions. A + publisher that cannot satisfy these steps retains work without claiming a + recoverable integrated outcome. + +Stable ordering is a proposed integration mechanism to validate, not a claim that +today's `resolve::merge` is commutative. The existing merge keeps the current +branch as the main page when edits overlap. Its deterministic revision names help +retry an original branch, but neither equal names nor presence of both texts +proves equal primary/conflict placement, causal coverage or byte-identical files. +Plan equivalence must compare source/outcome coverage, main and conflict page +identities/content, tombstones, catalog and blob closure. Physical allocation, +header bookkeeping and encoding can differ without a semantic disagreement; +such differences must not trigger endless republishing. + +The implementation gate is stronger than static sorted input testing: two fresh +replicas receiving the same branches in opposite orders, including intermediate +integration checkpoints and new edits based on them, must converge to equivalent +covered outcomes without duplicate conflict pages or silently repeated requests. +Test at least three concurrent branches, ancestor/descendant arrivals, one request +accepted on two branches, namespace operations and protected/opaque payloads. +If the current merge cannot satisfy this, retain unresolved alternatives and fix +the integration algorithm before claiming automatic convergence. Recovery custody +can remain safe while automatic integration is unavailable. + +Eventual repair is conditional. For a finite settled input set, its complete +closures must eventually reach a compatible unlocked reconciler; stale uploads +must eventually stop replacing the path; and a successful repair must eventually +be visible. A deterministic plan alone cannot guarantee these properties. +Repeated infinite old uploads are an explicit liveness counterexample. No finite +number of quiet reads proves that the last stale upload has arrived. Lack of +quiescence must consume retry/backoff budget, not user history or retained bases. + ## Authority and disaster recovery -The `.one` files remain the interoperable materialized section format. The -sidecar must preserve accepted history beyond those mutable files if it is to -repair demonstrated rollbacks. This is a proposed recovery requirement, not a -silent declaration that an arbitrary `.snowbound` folder always overrides every -`.one` file found beside it. +The `.one` files remain the interoperable materialized section format. For an +explicitly registered coordinated notebook, validated retained sidecar history +and its complete closures are authoritative for accepted/pending work when +mutable files regress. Registration binds notebook lineage, section identities, +protocol version and initial recovery closure; an arbitrary adjacent `.snowbound` +folder has no authority over a bare import or a different section at the same +path. This changes the backup contract for the selected mode, not the `.one` +format or native/SMB modes. Distinguish three recovery operations: @@ -542,9 +689,9 @@ checkpoint does not automatically include external payload files. ## Retention, compaction, and device rejoin -The first safe implementation can retain immutable logs/checkpoints without -automatic deletion. Storage use is real, so measure it and design compaction; -finite retention must not be disguised as a correctness proof. +The first implementation retains immutable logs/checkpoints without automatic +deletion. Storage use is real, so measure it and design compaction; finite retention +must not be disguised as a correctness proof. No “all devices have seen it” rule can be inferred from current listings or heartbeats. An offline device may hold unpublished edits indefinitely. Deleting @@ -676,6 +823,56 @@ without copying or changing them. The fixture is a disposable read of `corpus/conflict-page/native/initial/notebook/synthetic.one`; its input hash is recorded. No commercial provider, emulator, or OneNote process ran in this lab. +### Resilience policy follow-up + +The 2026-10-11 follow-up under +`target/multi-file-sync-resilience-20261011/` tests the selected policy's recovery +primitives and exposes limits of the current merge. Run 2 completed with exit 0 +in 2.4 seconds after the initial compilation populated the shared cache. The +first attempt failed to compile a lab-only immutable binding used by +`Section::conflicts()`; its exact source/log/result remain preserved. Neither +attempt changes production code. + +| Experiment | Observed outcome | +| --- | --- | +| Same overlapping ALICE/BOB branches, opposite primary bases | Both alternatives survive, but main/conflict placement differs; three replays of each original branch add zero publications and preserve exact bytes | +| Cross-deliver those opposite integration artifacts directly to current merge | Both directions publish again, flip the primary text and retain two conflict pages, including an alternative repeating current main text; naïve integration-result exchange is not convergence | +| Sort the same complete two-branch set before choosing a retained primary | Both arrival orders produce the same main/conflict identities and text; raw bytes and some active revision identities still differ | +| Five successive stale canonical replacements, alternating seed/A/B images | Every finite round repairs to exact ALPHA BETA using retained originals/checkpoint; repair costs 2/1/1/2/1 publications, and settled same-database/fresh-replica retries add zero | +| Fresh local recovery with a truncated canonical image | Cold parser rejects the image; retained raw source sections recover exact ALPHA BETA; observing only B lacks ALPHA and cannot establish complete history | + +Ten SQLite files pass integrity checks. Each saved valid `.one` is cold parsed +through the existing store/index/document reader with checksum/current-revision +validation; actual conflict pages are inspected separately from the ordinary +page. The immutable archive stays unchanged across repair. These are disposable +copies of the prior native-corpus fixture and actual production +`Replica`/`resolve::merge`/transaction calls; provider replacements, visibility, +source ordering and logical database reopen are simulated by the lab. + +The sorted-set result proves only a bounded presentation property for this +complete two-branch fixture. It does not prove equivalent opaque/protected +content, downstream causal translation, arbitrary incremental DAG convergence, +namespace operations, request coverage deduplication, cloud custody or OS-crash +durability. The opposite-integration result is why the selected algorithm must +track source coverage and normalize from retained inputs instead of feeding +integration artifacts back as new independent edits. No automatic convergence +claim is justified by this investigation. + +Rerun from a clean new output directory with one granted build slot: + +```sh +nice -n 10 env CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target" cargo run --offline \ + --manifest-path target/multi-file-sync-resilience-20261011/lab/Cargo.toml -- \ + "$PWD/target/multi-file-sync-resilience-20261011/run-new" +``` + +The ignored standalone lab links the current source in +`workspaces/sync-resilience` and reads three bound `.one` inputs from the original +run 7 without modifying them. Its verification manifest binds lab/dependency +sources, inputs, raw outputs, SQLite integrity checks, draft architecture review +and document checks. No commercial provider, emulator or OneNote process runs +in this follow-up. Native OneNote cold-open remains an implementation gate. + | Fault | Required result / investigation oracle | | --- | --- | | Prefix/partial body, invalid length/digest | Reject/wait; last valid state untouched | @@ -713,14 +910,15 @@ recovery and normal append operations. Platform hosts adapt document/file APIs, retained local directories and actual I/O/error callbacks. Android rendering and native IME/accessibility can ship independently of this protocol. -Before production protocol code, review practical evidence and select: +The owner has selected resilient branch-preserving takeover and retained recovery +authority for registered coordinated notebooks. Before production protocol code, +validate the reconciliation/receipt algorithm against the gates above and define: -- Fixed publisher/explicit handoff versus branch-preserving automatic recovery, - and whether majority membership offers enough benefit for its availability cost. -- The authority/recovery contract for registered notebooks, bare imports, - intentional restores, duplicate notebook copies and copied owner credentials. +- Explicit registration, bare import, intentional restore and duplicate-copy flows + that preserve the stated lineage/custody boundaries. - Protected-key custody and unsupported-client migration/escape behavior. -- Retention/compaction guarantees and a measured provider capability floor. +- Measured provider capability floor and storage cost. Initial automatic GC is + excluded; later compaction requires its own recovery proof. Unknown protocol/required-op versions must leave remote data untouched and keep local edits recoverable. Enabling coordination requires an explicit migration @@ -730,6 +928,6 @@ needs an honest compatibility boundary and an opt-in/import path, not a claim that an unsupported build will obey a new marker. The next concrete deliverable after the draft is a narrow transport/owner lab -against real provider fixtures, then one selected policy implemented behind an +against real provider fixtures, then the selected policy implemented behind an explicit storage mode and tested with actual notes. No timeout election or full-image save shortcut should land as an Android dependency. -- 2.54.0