diff --git a/arc/multi-file-sync.md b/arc/multi-file-sync.md new file mode 100644 index 0000000000000000000000000000000000000000..8bc1ed44ae775f94e677348e63e214445855c65d --- /dev/null +++ b/arc/multi-file-sync.md @@ -0,0 +1,735 @@ +# Coordinating cloud edits through separate files + +**Draft specification and investigation, 2026-10-11.** +[#120](https://shale.paperclover.net/snowbound/issues/120). This is separate from +shipping Android. Nothing in this document implements or enables a new storage +mode. Existing local/native and direct SMB publication retain their existing +OneNote-compatible exclusion. The wire examples and ownership policies below +are proposals for review, not frozen public APIs. + +Clover's direction is a notebook's `.snowbound/` directory carrying coordination +between Snowbound clients, with one owner process publishing the real `.one` +files. Other builds exchange separate files instead of racing on those files. +Generic cloud mounts cannot be assumed to implement reliable distributed locks. +This should cover Android/Linux provider-backed notebooks and improve iCloud +coordination. Practical tests must determine the tradeoffs before selecting an +ownership/recovery policy. + +## The contract under investigation + +```text +Android / Linux / iOS editor + │ existing edits, locally durable SQLite queue + ▼ +.snowbound/ unique immutable request segments + blobs + │ validated complete bytes and causal dependencies + ▼ +owner existing merge and revision append + ├── immutable decision/checkpoint evidence ──► other Snowbound replicas + └── ordinary .one / .onetoc2 files ─────────► eventual provider projection +``` + +A single application publisher reduces ordinary write/write contention. It +cannot prevent a provider from delivering an old queued upload after ownership +has moved. A successful local flush, a successful handoff, and a unique current +logical owner are three different facts. The design must retain enough immutable +evidence to repair the mutable file after a rollback even if every client has +already received a historical publication receipt. + +Proposed safety requirements: + +- An acknowledged local edit remains in retained local storage until its content + and outcome are recoverable elsewhere under an explicit retention contract. +- Seeing a request, seeing a receipt, or seeing a matching file stamp does not + authorize deletion of the only recoverable request/base/blob. +- Missing, stale, incomplete, repeated, or reordered provider observations never + silently delete notes, select an owner, retire a branch, or establish a + complete list of devices. +- Every accepted request has a stable identity, complete validated payload, + known causal base, and a recorded disposition. Retried transport does not + repeat the user edit or create duplicate conflict pages. +- Normal edits use existing ops and append normal revisions. A projection is + never rebuilt by diffing or recreating a page model. +- A dependency that has not arrived means wait. Concurrent incompatible content + means the existing merge produces conflict pages or a preserved recovery + branch. A missing dependency is not a content conflict. +- Locked protected content, unavailable providers, and absent owners retain + encrypted pending edits. No network condition turns a protected edit into a + cleartext file. +- Guarantees identify their observation point: local durability, owner custody, + file publication, and provider/cloud visibility are distinct. + +These requirements depend on at least one retained complete copy of each +necessary object eventually being delivered. No file protocol preserves data +when every durable copy is destroyed, or proves delivery when the provider +permanently withholds it. Eventual projection repair additionally requires that +old uploads eventually stop, required objects are eventually delivered, a +capable reconciler keeps running, and necessary keys/versions remain available. +An unbounded stream of stale overwrites can keep the canonical path regressing +forever. Retained history preserves recoverability; it does not make that path +monotonic under an adversarial delivery schedule. + +## Storage model and observations + +Assume independent client views of one provider-backed directory tree. Reads and +listings may be stale; files may become visible before their bytes are complete; +notifications and uploads may reorder; a complete old `.one` image may replace a +newer one. A provider may omit known children, return unknown metadata, rename a +file while changing its opaque handle, or reject a nominally supported operation. +Clients can remain offline indefinitely. Process clocks are unrelated. A process +can stop at any instruction and later resume with old state. + +There is no assumed atomic directory rename, distributed exclusive create, +compare-and-swap, reliable distributed advisory lock, complete listing, or +multi-file transaction. Independent objects with unguessably unique identities +reduce collisions; they do not turn a provider's filename API into exclusive +creation. Object integrity is checked after reading complete bytes. Local +app-private SQLite ownership is a different boundary and can still serialize +processes sharing that same local replica. + +Android is one transport, not the definition of the protocol. SAF `rw` opens +provide seeking when supported; that says nothing about remote exclusion or +upload ordering. Provider handles/IDs remain opaque, and returned rename handles +must be retained. iCloud coordination/versions remain useful host capabilities, +but they do not establish distributed fencing for the generic mode. +[Android ContentResolver](https://developer.android.com/reference/android/content/ContentResolver), +[DocumentsContract](https://developer.android.com/reference/android/provider/DocumentsContract). + +A provider that cannot retain separately named byte objects, enumerate them +again, or represent the selected notebook subtree cannot carry this transport +without another adapter. Unsupported capabilities must be observed in disposable +provider fixtures; a provider name is not evidence of support. + +## Ownership choices and their actual guarantees + +| Candidate | Progress while original owner is absent | Logical ownership | Physical `.one` protection | +| --- | --- | --- | --- | +| Fixed designated publisher | Other clients edit/queue; publication waits | One configured owner under non-duplicated local state | No cross-provider upload fencing; late old uploads still require repair | +| Explicit handoff | New owner proceeds after validated handoff dependencies | Old process durably stops admitting/publishing; successor receives state | Provider may still deliver old process's earlier uploads | +| Timeout/heartbeat takeover | Available to isolated clients | Stale views can authorize two writers | Unsafe as a strict single-writer rule | +| Fixed-member majority consensus | Majority can proceed; minority queues | Possible with durable acceptor promises and valid membership changes | Consensus does not make a generic provider reject stale uploads | +| Branch-preserving recovery takeover | Isolated clients can preserve new branches | Cannot promise one global writer/order without additional agreement | Mutable file remains an eventually repairable projection | +| Fenced server/provider | Depends on service availability | Service validates owner epoch for each accepted write | Can reject stale owner writes when checks cover the actual storage destination | + +No recommendation here may describe a wall-clock lease as sufficient fencing. +Heartbeat objects are discovery/liveness hints only. A process that sees no other +owner cannot distinguish an absent owner from a partition or stale listing. + +Majority agreement is a real alternative, not impossible consensus: fixed +acceptors can persist promises, choose one logical history, and recover after +restart. Its availability cost matters for a phone and one laptop. With two +members, a majority requires both. Membership changes and acceptor-state loss +need a complete protocol; a directory full of claims or votes is not one. +[Paxos Made Simple](https://www.microsoft.com/en-us/research/uploads/prod/2016/12/paxos-simple-Copy.pdf). + +Fencing must be checked by the resource receiving the write. A local owner epoch +inside `.snowbound` cannot force a generic mount to reject a queued older `.one` +upload; the provider does not interpret that epoch. Chubby's sequencers address +precisely delayed operations by requiring validation at the recipient. +[The Chubby lock service](https://research.google.com/archive/chubby-osdi06.pdf). + +**Investigation recommendation, not yet selected:** start with a fixed preferred +publisher, explicit handoff, and immutable recoverable inputs/outcomes. Other +clients keep editing from retained local replicas while that publisher is absent; +publication waits. Preserve branches as the recovery fallback, and treat automatic +takeover as separate policy work. Timeout takeover is not part of the strict +owner contract. Majority agreement remains an alternative with its stated +availability cost. This makes the sidecar necessary recovery authority for +accepted history when the mutable projection regresses; that authority/backup +contract needs owner review before production implementation. The recorded +experiments below support the recovery requirement, not a finished owner protocol. + +## Identities and duplicate processes + +Notebook identity must be independent of mount path, provider document ID, +display name, host registration, and filename case normalization. Existing +OneNote file/root identities remain the content identities. A new coordination +notebook identity distinguishes separate imported copies and ownership histories; +how deliberate notebook copies inherit or reset this identity remains a +migration decision, not an inference from their path. + +Use distinct roles: + +| Identity | Lifetime / purpose | +| --- | --- | +| Local installation/device identity | Retained private registration and diagnosis; not sufficient owner authorization | +| Process/writer incarnation | Fresh random identity for each process run; disjoint message namespace | +| Local edit identity | Existing durable replica edit ID plus its originating replica/writer identity | +| Request segment identity | Stable once sealed, retries resend identical bytes | +| Owner epoch identity | Chained ownership decision, not a timestamp or largest visible number | +| Decision identity | Immutable owner result naming its predecessor and accepted request identities | +| Blob/checkpoint identity | Validated immutable bytes, content digest and exact byte length | + +Previously sealed/outstanding records keep their original request, edit and +segment identities after every restart. A fresh process incarnation names only +new records. Retrying an old outbox may write its original namespace/bytes; it +does not rekey old edits into the new process namespace. Otherwise deduplication +would turn an ordinary restart into a second user action. + +Two processes on one installation must not both publish because they share a +device ID. A locally owned publisher database/socket may serialize processes +sharing reliable private storage. A second process with a copied private profile +on another device is a different failure: distinguish incarnations, preserve +both claims, and stop automatic publication when contradictory ownership becomes +visible. Absence of a contradiction in a stale view is not a uniqueness proof. +A backup of owner credentials is not permission to run both copies. + +For fixed ownership, restart uses the retained durable owner state and a fresh +process incarnation. For explicit handoff, the old state records retirement +before a successor becomes eligible. Losing the old durable state requires +recovery; creating a fresh claim is not a transparent restart. + +## Proposed object namespace + +Names illustrate roles; this is not a frozen layout or extension: + +```text +Notebook/ + Notes.one + Notebook.onetoc2 + .snowbound/ + protocol-- + writers//segments// + owners//decisions// + handoffs/ + blobs// + checkpoints/ + recovery/ +``` + +Every published object is immutable. A transport retry keeps its identity and +bytes. Correcting an object creates a distinct object referencing what it +supersedes. Mutable convenience indexes or heartbeats may accelerate discovery +but cannot authorize publication, retirement, or data loss. No implementation +may overwrite one shared `owner.json` or `inbox.json` as the coordination truth. + +Per-process directories avoid ordinary client/client name collisions. Unique +logical object IDs also appear inside the envelope. Two provider documents with +one display name are not automatically one object; the same object under two +names/handles is one logical object only after matching its validated identity +and bytes. Different bodies claiming one immutable identity are a contradiction +and are preserved/quarantined. This cannot retroactively undo side effects that +already occurred before the contradiction became visible. + +No one file per keystroke: transport sealed batches from the existing queue into +segments bounded by encoded bytes and editing bursts. The durable local receipt +remains immediate; the provider upload can coalesce multiple edits. Individual +edit IDs remain present for receipts/dedup. Attachments and large bodies are +separate immutable blobs; transport does not repeatedly embed them in every +segment. Exact segmentation thresholds require measurements, not a public +extension promise. + +## Framing, validation, and wire examples + +Before accepting an object, check magic/version, bounded envelope/body lengths, +notebook identity, declared kind, complete record digest, sender/object identity, +and all required dependencies. Integrity covers the entire encoded envelope and +body, including identities, owner epoch, predecessor/base/dependency references, +kind and version. A body-only digest permits metadata mixups and is insufficient. +A trailing digest or excluded self-digest field must have an unambiguous encoding +rule. Digests prove byte consistency, not permission to edit. If access/authentication is added, it covers the envelope and dependencies +as well as payload bytes. Provider ACLs and section passwords remain separate. + +Do not trust size/mtime metadata as completeness. A parseable JSON prefix or an +apparently complete `.one` header is not enough; an object references its exact +complete body length/digest. A readiness marker may arrive before the body and +merely adds a dependency to wait for. Atomic rename is unnecessary for acceptance +only if incomplete objects are rejected and complete objects stay immutable. + +Illustrative request envelope (symbolic IDs/digests, not executable wire data): + +```json +{ + "protocol": 1, + "kind": "edit-segment", + "notebook": "N", + "id": "writer-R/segment-S", + "writer": "R", + "predecessor": "writer-R/segment-before-S", + "base": {"section": "OneNote-file-identity", "checkpoint": "C", "decisions": ["D"]}, + "edits": [{"replica": "Q", "id": 42, "author": "Alice", "opsObject": "O"}], + "dependencies": ["C", "D", "O", "blob-B"], + "body": {"length": 1234, "sha256": "complete-body-digest"}, + "recordDigest": "digest-of-envelope-and-body-excluding-this-field" +} +``` + +Illustrative decision and publication receipt: + +```json +{ + "protocol": 1, + "kind": "owner-decision", + "id": "D-next", + "ownerEpoch": "E", + "predecessor": "D", + "accepted": ["writer-R/segment-S"], + "dispositions": [{"replica": "Q", "id": 42, "result": "merged-or-conflict-retained"}], + "section": "OneNote-file-identity", + "baseCheckpoint": "C", + "resultCheckpoint": "C-next", + "revisions": {"page-space": "sealed-revision-identity"}, + "dependencies": ["D", "C", "C-next", "writer-R/segment-S"], + "recordDigest": "digest-of-envelope-and-body-excluding-this-field" +} +``` + +```json +{ + "protocol": 1, + "kind": "projection-receipt", + "id": "P", + "decision": "D-next", + "section": "OneNote-file-identity", + "observed": {"header": "OneNote-stamp-header", "length": 5678}, + "checkpoint": "C-next", + "guarantee": "owner-local-file-confirmed", + "recordDigest": "digest-of-envelope-and-body-excluding-this-field" +} +``` + +The examples omit separate payload bytes. `recordDigest` is symbolic: the codec +must define exact bytes and self-field exclusion before implementation. Every +object kind requires the same metadata integrity, not only edit segments. + +Version 1 is illustrative. The actual serialization must reuse existing +`onestore::op::Edit`/`Op` semantics, with an explicit compatibility boundary +around them. Existing serde representation is not independently promised stable +forever. Do not invent a second page-diff language or CRDT. Live Share's bounded +op batch plus named revisions demonstrates an existing reusable approach; its +network membership/timeout assumptions do not transfer automatically to cloud +files. + +## Causal bases, merges, and replay identity + +A segment names its exact input base and prior writer segment. A client editing +without the owner continues its own local causal chain. The owner waits for a +complete base/checkpoint, referenced blobs, and required predecessor decisions +before interpreting positional text ops. An old base is not permission to apply +UTF-16 offsets directly to a changed page. + +Reuse the current queue/merge machinery: `PendingEdit` carries author and ops; +`Replica` keeps its base, seals batches, rebases against changed remote state, +and makes conflict pages for incompatible operations. Existing +`resolve::merge` discovers common revision ancestry and deterministic merged +revision identities for retained competing section versions. Where no shared +valid ancestry exists, preserve the foreign/unreadable branch for recovery. + +Deduplication is two related checks: + +1. Request identity and encoded content digest identify a transport retry. +2. Stored named revision identities/outcomes identify work already incorporated + into a section/checkpoint, including after owner restart or request replay. + +A numeric edit ID alone is local to one replica. A file stamp alone does not +prove every request's effect is present. A page's visible text matching expected +text also does not prove correct history, attachments, or membership outcome. + +An unmergeable request or section branch remains retained with its original +request/base/blobs and exact error. Custody acknowledgement of +`retained-unresolved` is distinct from semantic acceptance and from file +publication. It cannot appear in an accepted edit list as though the note had +merged. A recoverable conflict outcome names the actual stored conflict page; +a retained foreign/unreadable branch names recovery evidence instead. + +Owner decisions impose a durable selected order under the chosen ownership +policy. Concurrent branches remain explicit inputs to the existing merge; do +not choose the newest wall clock or discard the branch with the smaller owner +ID. An automatic recovery policy needs a stable, documented reconciliation of +multiple accepted decision histories. Deterministic sorting for processing is +not a consensus or permission rule. + +## Owner state and write/ack ordering + +Proposed durable states, persisted before their corresponding externally +observable promise: + +| State | Required evidence | Permitted action | +| --- | --- | --- | +| Discovered | Candidate object handles | Read/validate; no ack | +| Complete | Exact body validated | Wait for dependencies or stage locally | +| Ready | Causal base/blobs/ownership validated | Prepare existing ops/merge | +| Retained unresolved | Original request/base/blob closure and exact failure retained | Report custody only; preserve for later merge/review | +| Prepared | Stable request disposition, named revisions, reconstructible input/base retained | Attempt normal append | +| Attempt uncertain | Append/header/flush result ambiguous | Inspect/confirm exact evidence; never blindly reapply | +| File confirmed locally | Named revisions/stamp confirmed at owner's actual adapter boundary | Record historical local publication evidence | +| Recoverable result | Immutable input/decision/checkpoint closure available under custody contract | Publish decision/result receipt | +| Receipted | Request identity maps to durable outcome and recovery closure | Client may advance display/status; retention rules still apply | + +The exact linearization between durable prepared decision, file append, and +immutable result publication needs implementation tests. The invariant is +stronger than a particular file ordering: no externally acknowledged request +may depend solely on mutable `.one` bytes or a volatile owner map. A crash at any +point must leave either an unacknowledged retained request or a recoverable +recorded outcome. Checkpoints are raw section bytes/named append artifacts, +not projected page models. + +An owner can prepare and retain a reproducible named transaction before an +append; after an ambiguous append it uses revision evidence and existing +`CommitState`/confirmation logic. It does not regenerate new revision IDs and +retry the user action. Appending successfully and crashing before writing the +receipt must converge to the same outcome when the original request returns. + +If the provider exposes the receipt before its referenced checkpoint/decision, +the reader waits. If the `.one` file is behind the receipt, the reader obtains +or reconstructs the referenced section from its recovery closure; it does not +claim the mutable provider path already holds it. If the file is ahead, it can +read validated revisions while separately resolving incomplete decision history. + +Once a client has validated a selected decision/checkpoint, a stale bare `.one` +observation must not downgrade its known accepted history. Keep that history +and recovery dependencies locally while repairing the projection. An explicit +intentional restore is a new recorded operation, not a stale-file inference. + +An owner-unavailable client still edits from its private replica. Requests wait +for owner processing without blocking local read/navigation/Undo. Publishing +status cannot report owner custody merely because local creation of a provider +object succeeded. + +## Explicit handoff, recovery takeover, and late uploads + +Proposed explicit handoff sequence: + +1. Old owner stops admitting new work for the epoch and persists that stop in + its retained local owner state. It retains requests arriving later. +2. Seal/record admitted decisions and their complete recovery closure; publish + an immutable handoff naming successor, last decision, checkpoints and pending + uncertainty. No time-based claim of a complete device/request census. +3. Successor validates the chain, retrieves every required dependency, resolves + uncertain attempts, and records acceptance in its own durable state. +4. Successor publishes only under the selected handoff policy. Old owner never + resumes that epoch after a restart; a failed handoff remains recoverable. +5. Both retain historical recovery evidence, because old provider uploads can + arrive after all application steps have completed. + +A receipt from the successor is useful evidence for operator recovery; it does +not prove every cloud cache drained. Waiting an arbitrary duration is not a +fence. A fixed owner with no ownership transitions avoids one source of late +writes but still needs rollback repair for reordered uploads from its own run. + +Automatic recovery experimentation must keep each candidate's decisions in its +own immutable branch. A candidate cannot overwrite another owner's mailbox or +checkpoint, and claims no exclusive owner authority based on silence. The +mutable `.one` projection may temporarily regress or produce provider conflict +versions. A later reconciler imports all valid branches, preserves incompatible +results as conflict pages/recovery data, and republishes normal revisions. +This is a weaker, explicit safety/availability trade than a strict global +single publisher, and remains unselected pending the experiments. + +A quorum alternative can choose an ownership/decision chain if acceptor +membership and durable promises are implemented. It still needs immutable +recovery and projection repair unless the actual provider enforces epochs on +writes. Do not add a quorum merely to hide this physical fencing limitation. + +## Authority and disaster recovery + +The `.one` files remain the interoperable materialized section format. The +sidecar must preserve accepted history beyond those mutable files if it is to +repair demonstrated rollbacks. This is a proposed recovery requirement, not a +silent declaration that an arbitrary `.snowbound` folder always overrides every +`.one` file found beside it. + +Distinguish three recovery operations: + +| Operation | Meaning | +| --- | --- | +| Open a registered coordinated notebook | Reconcile validated selected history/retained branches with its section identities and observed projection | +| Import a bare `.one`/notebook copy | Treat validated bytes as an import base; no historical receipts or missing sidecar authority inferred | +| Restore a coordinated notebook backup | Restore a consistent closure or explicitly import/reconcile incomplete histories; never reset edit IDs/epochs silently | + +Durable custody is always qualified by its adapter: a private SQLite transaction +is locally retained; a provider close is not a cloud durability receipt. A client +must not evict its only outbox because the owner merely closed a provider file. +Cross-device verified copies or an explicit retained owner-private closure have +their own failure domains, which the recovery contract must name. + +A portable backup of a coordinated notebook needs the section bytes plus every +retained immutable object required by accepted decisions and pending requests. +Copying only `.one` is a valid interoperable snapshot, but not proof that +already receipted cloud edits are recoverable. Copying only `.snowbound` may lack +the seed images or external payloads needed for reconstruction. + +The mode must define when a deliberately restored older `.one` is an intentional +user restore versus provider rollback. Timestamps cannot distinguish them. +A deliberate restore needs a new explicit operation/epoch/disposition preserving +the previous accepted branch, rather than deleting its evidence. Exact copy and +restore UI semantics remain owner-facing design work after evidence; no current +build is changed by this draft. + +## External writers, replacement, and repair + +On each projection change validate store/header/checksums, section root/file +identity, and revision ancestry. The owner compares the actual base stamp at the +moment of normal publication using available adapter guards; coordination mode +never falls back by silently dropping an existing lock. + +A matching identity with extra external revisions is an import/merge candidate. +The existing tolerant reader and retained-version merge can preserve content. +Record the imported exact base/version so later requests have stable ancestry. +A different section at the same filename is foreign content, not a new base for +old queued ops. Preserve it and stop automatic destructive reconciliation. +Unreadable/partially repaired images do not replace the last validated base. +A repair must record the chosen input and retained rejected alternatives. + +Concurrent OneNote edits on a generic cloud mount bypass the coordination +protocol. This mode cannot guarantee safe live interchange with such writers. +Interoperable native filesystem/direct SMB mode remains the route for OneNote +2010 side-by-side editing. A user opening a resulting `.one` offline still gets +a normal OneNote file; that does not make its cloud upload fenced. + +On iCloud, continue reading unresolved `NSFileVersion` conflicts as recovery +inputs. Multi-file coordination can reduce ordinary competing application +writes and give them retained request history; it cannot promise the operating +system will create zero conflict versions, nor retire those versions before +complete recoverability is established. + +## Namespace, TOC, moves, deletes, and Undo + +A notebook has multiple files: section groups, `.onetoc2`, recycle-bin sections, +external payload folders, and section files. A single-section protocol is not +a notebook protocol. Namespace operations are owner-processed intentions named +by stable section/page identities and expected causal state; paths are results. +Only the publisher changes shared canonical namespace/TOC in this mode. + +Create/rename/move/delete outcomes reference all files and object identities +they changed. Provider creation and rename return actual handles; do not infer +success from a requested filename or overwrite an unrelated same-name item. +A provider may expose TOC before a new section or source deletion before a +moved section arrives. Consumers wait for the decision's full closure and keep +last valid local catalog state. + +Multi-file changes use resumable intent records, not fictitious atomicity. +For page/section moves, establish destination durability/recoverability before +retiring the source, validate expected source stamp and queued generation under +actual writer coordination, and make retries identity-stable. Without physical +transaction/fencing, prefer a preserved duplicate to loss. Do not mix this +unsettled transport implementation with the separately investigated +cross-notebook move work. + +Delete is a durable tombstone/outcome, never merely disappearance in a listing. +Retain the historical deleted branch or recycle-bin closure required by Undo +and offline rejoin. An old already-accepted create must not recreate a deleted +object when replayed. A genuinely unseen edit based before deletion receives an +explicit merge/conflict/recovery disposition; its content is never dropped by a +membership timeout. Undo is a new conditional intention against actual latest +content, not blindly replaying an old whole-image inverse over later edits. + +## Protected content and blobs + +The existing protected replica queue seals edits/payloads under keys derived +from the section key; payload names use keyed digests. Reuse those semantics +when transporting protected data, with fresh nonces and authenticated envelope +context as appropriate. A plain SHA-256 of private attachment bytes can leak +cross-notebook equality; choose protected blob identifiers consistently with +the existing keyed queue. No cleartext ops, attachment previews, page titles, or +raw protected payloads may be written into `.snowbound` as a transport shortcut. + +An owner cannot semantically merge protected requests without the required +section key. Proposed behavior is retain encrypted requests and wait for an +unlocked capable publisher; an encrypted raw-transaction path must preserve +existing validation and cannot pretend it can rebase unknown plaintext. Whether +an always-on publisher may retain unlocked keys and for how long is a policy +question for later design, not an implicit background service requirement. +Password changes/rekeying need explicit key-epoch dependencies, retained old +keys/requests policy, and safe receipts before old material is retired. + +Unprotected pictures/attachments/video remain byte payloads associated with +existing ops. Blobs arrive before or after requests; missing blob means wait. +Validate exact length/digest, reject oversized declarations before allocation, +bound concurrent reads, and stream where the existing storage APIs permit. +Preserve external `_onefiles` identities and original bytes. A raw section +checkpoint does not automatically include external payload files. + +## Retention, compaction, and device rejoin + +The first safe implementation can retain immutable logs/checkpoints without +automatic deletion. Storage use is real, so measure it and design compaction; +finite retention must not be disguised as a correctness proof. + +No “all devices have seen it” rule can be inferred from current listings or +heartbeats. An offline device may hold unpublished edits indefinitely. Deleting +an accepted request after observing `.one` alone is unsafe because that file +can later roll back. Receipt existence alone is also insufficient if the +receipt's reconstructible dependencies are removed. + +A compacted checkpoint needs complete section/catalog/blob state, accepted +request identity/outcome coverage, selected history and retained branch/tombstone +coverage, plus its exact predecessor closure or an explicit permanent cutoff. +Only a verified replacement recovery closure can justify retiring old objects. +Keep dedup/tombstone evidence needed to recognize old retries after compaction. + +For an offline client older than retained ancestry, retain its private queue and +import a fresh base without discarding it. If exact rebase is unavailable, preserve +its old base/ops as a recovery branch and require an explicit content outcome. +A destructive administrative reset establishes a new notebook lineage and keeps +an export of displaced data; it does not make stale requests silently valid. +An app uninstall destroying the sole private outbox remains an actual loss +boundary unless that outbox reached retained custody elsewhere. + +## Scale and discovery cost + +A long notebook cannot relist a flat object directory on every keypress. Segment +existing sealed batches, partition by writer/owner and bounded buckets, cache +validated handles/objects locally, follow known dependency/predecessor references, +and use notifications as hints plus periodic reconciliation. Mutable indexes +may reduce scans but cannot establish absence or completeness. + +SAF opaque document IDs make filename-derived constant-time lookup unavailable +in general. Resolve actual handles from queried parents and cache them by +validated object identity. Bucket size and deep directory cost need real provider +measurements; a huge hierarchy can cost more than a flat list. Unknown new writer +discovery still needs enumeration, so “bounded lookup” cannot mean a complete +new-client discovery guarantee without scanning any provider state. + +Measure at least: edits per sealed segment; object and request count; bytes +including retained bases/blobs; cold-open queries; warm sync queries; maximum +listing size; incomplete dependency backlog; latency until local durability, +owner custody and visible remote result; and time to rebuild after projection +rollback. Record provider/API/build identity. A host memory microbenchmark is +not a Google Drive/iCloud listing benchmark. + +## Status and receipt meanings + +These are engineering states, not proposed public UI strings: + +| Evidence | What is known | What is not known | +| --- | --- | --- | +| Local queue durable | This installation can recover the edit after process death | Provider/owner received it | +| Provider create/close succeeded | This adapter completed local object I/O | Another device or cloud service has it durably | +| Owner intake durable | Owner has validated recoverable request/dependencies under stated custody | Canonical `.one` contains it | +| Owner result recoverable | Accepted outcome can be reconstructed from retained closure | Every client has downloaded it | +| Owner file locally confirmed | Normal append/confirmation completed at owner adapter | Cloud globally ordered/durable, no later rollback | +| Other client observed result | That client's validated state includes decision/checkpoint | All devices saw it or GC is safe | +| Historical published receipt | A publication was confirmed at an earlier point | Its effect remains in today's mutable projection | + +Current `EditStatus::Published` is historical by design. Adding transport states +must preserve that meaning and the existing local receipt boundary. Current +mobile `sb_section_poll` coarse saved/saving/offline values cannot silently +encode these extra guarantees; UI flow/copy changes require a separate approved +mapping after protocol behavior exists. + +## Fault matrix and practical investigation + +Lab evidence lives outside source under +`target/multi-file-sync-spec-20261011/`. It is a bounded investigation, not a +production backend or commercial-provider certification. The Rust lab uses the +existing `onestore::Transaction::commit`, `notebook::Replica`, receipt logic, +retained-version merge, and cold parser. Provider delivery is a deterministic +simulation; logical drop/reopen is not an OS-kill durability test. The separate +Python model explores object framing/dependencies and ownership rules only. + +### Recorded results + +The following are observed experiment outcomes. The requirements and future +implementation gates elsewhere in this draft are not proven by these tests. +Native-corpus Rust run 7 completed with exit 0, and transport-model run 3 +completed with exit 0. Reports and raw `.one` outputs remain in the ignored lab +folder; the verification manifest binds their hashes and the exact lab sources. + +| Experiment | Observed outcome | Evidence | +| --- | --- | --- | +| Heartbeat/lease takeover with stale view | Counterexample found: two independently valid append results; winning snapshot omits ALPHA | `run-7/report.json` → `lease_takeover_split_writer` | +| Explicit handoff, then old provider upload | Counterexample found: exact ALPHA BETA becomes ALPHA RIGHT after delayed snapshot delivery, despite no concurrent application writers | `clean_handoff_late_old_upload` | +| Retained independent section branches | Existing merge yields ordinary text exactly ALPHA BETA | `independent_immutable_branches_recover` | +| Late-upload rollback with retained checkpoint | Existing retained-version merge recovers BETA | `late_upload_recovered_from_immutable_checkpoint` | +| Concurrent overlapping changes | Main page retains BOB RIGHT; actual conflict page retains ALICE RIGHT | `overlap_retains_actual_conflict_page` | +| Append succeeds, reply lost, logical close/reopen | One publication total; existing receipt settles without another append | `owner_append_then_crash_before_ack` | +| Historical Published receipt, then projection rollback | Counterexample found: queue empty and historical receipt remains Published while ALPHA is absent | `historical_receipt_does_not_repair_projection_rollback` | +| Resumed old-owner branch and new-owner branch | Existing merge retains OLD and NEW; strict owner uniqueness not established | `old_owner_resumes_branch_preservation` | +| Old offline branch retained through rejoin | Existing merge can import its ALPHA content | `offline_rejoin_request_retention` | +| Original branch redelivered after merge | For each recovery case: three same-database reopen/retries plus one genuinely fresh replica seeded from merged bytes; zero extra publications, identical output | Recovery case `proof` fields | +| Generated minimal section seed | Current cold parse succeeds for both same-file/root branches; exact existing resolve returns “Object space has no active default revision” | `run-7/generated-seed-diagnostic.json` | +| Abstract transport model | 737 truncated prefixes and 737 single-byte mutations rejected; 120 arrival permutations converge once with omission/duplicates; schema/cycle/contradiction checks pass | `visibility-run-3/visibility-report.json` | +| Abstract deep input | 1,101-object dependency chain traversed iteratively; nested 1,100-level JSON rejected without recursion crash | Same transport report | + +The generated-seed diagnostic is specifically the current `create_section` +helper's output and those exact edits. Current-image parsing does not certify +all historical ancestry candidates or OneNote conformance. The existing sync +fallback keeps an unmergeable version whole; this result is not evidence that +all supported production inputs fail to merge. It does mean that a protocol +must represent retained-unresolved custody separately from accepted content. +The native-corpus success demonstrates existing section-version merge/dedup, +not a new request/owner-decision/tombstone implementation. + +Failed investigations remain preserved: run 1 (missing lab dependency), run 3 +(lab type-alias error), runs 2/4/5 (generated-seed refusal), and run 6 (text oracle +looked only at ordinary page-list content, excluding the conflict page). Final +run 7 asserts actual conflict-page text through the existing section conflict +reader and preserves the earlier evidence. No failed run is a product gate. + +Rerun from the repository root on a clean **new output directory**, using the +shared target and one build slot; never reuse a run folder or point the lab at an original notebook: + +```sh +nice -n 10 env CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target" cargo run --offline \ + --manifest-path target/multi-file-sync-spec-20261011/lab/Cargo.toml -- \ + "$PWD/target/multi-file-sync-spec-20261011/run-new" +python3 target/multi-file-sync-spec-20261011/lab/visibility.py \ + target/multi-file-sync-spec-20261011/run-new \ + target/multi-file-sync-spec-20261011/visibility-run-new +``` + +The ignored standalone manifest currently has absolute path dependencies into +`workspaces/android-host-plan`; the verification manifest records them. The lab +links the repository's existing test remote and private merge sources directly, +without copying or changing them. The fixture is a disposable read of +`corpus/conflict-page/native/initial/notebook/synthetic.one`; its input hash is +recorded. No commercial provider, emulator, or OneNote process ran in this lab. + +| Fault | Required result / investigation oracle | +| --- | --- | +| Prefix/partial body, invalid length/digest | Reject/wait; last valid state untouched | +| Reordered/duplicate/omitted listings | Same logical requests applied once; omission never deletion | +| Missing base/blob/predecessor | Wait for dependency; do not apply positional ops to wrong base | +| Same immutable ID, different bytes | Preserve contradiction; stop automatic authority selection | +| Owner unavailable | Local editing/Undo/reopen works; requests retained | +| Owner crash after append before ack | Original named outcome recognized after reopen; no repeated edit | +| Lease takeover while old view is stale | Expected counterexample: both writers eligible; never call this fenced | +| Graceful handoff followed by old queued upload | Expected counterexample: canonical `.one` can regress | +| Old owner resumes after takeover | Preserve both decision/section branches; merge/recover without silent loss | +| Published receipt then `.one` rollback | Historical receipt cannot alone restore removed effect | +| Independent retained branches | Actual merge retains both changes, retry idempotent | +| Overlapping retained branches | Actual conflict page preserves both alternatives | +| Retried old branch after owner restart | No duplicate revisions/conflict pages from original immutable input | +| Offline rejoin after delete/compaction | Dedup/tombstones honored; unseen edits retained for explicit outcome | +| Invalid/foreign/repaired `.one` | Preserve bytes; never rebase old requests onto a different section | +| Close/flush/upload ambiguity | Keep attempt evidence and request; do not promise remote durability | +| Protected request without owner key | Encrypted queue stays recoverable; no plaintext transport fallback | +| Attachment missing/partial | Wait; no receipt claiming complete recoverability | +| TOC arrives before section / move halfway | Keep catalog/intent recoverable; prefer duplicate over loss | + +The Android host/provider plan and its two DocumentsProvider fixtures are at +`target/android-wave-20261011/host/`. They are specified, not yet implemented or +run. They test the real platform boundary separately from these simulations. +Commercial provider observations, native OneNote cold-open of new protocol +outputs, large-notebook scale, protected transport, namespace recovery, and actual +process-kill/crash tests are implementation gates, not claims made by this draft. + +## Implementation boundary and review decisions + +`onestore` remains unaware of providers, coordination folders and ownership. +The data-layer protocol belongs toward `notebook`, reusing queue, merge, +recovery and normal append operations. Platform hosts adapt document/file APIs, +retained local directories and actual I/O/error callbacks. Android rendering and +native IME/accessibility can ship independently of this protocol. + +Before production protocol code, review practical evidence and select: + +- Fixed publisher/explicit handoff versus branch-preserving automatic recovery, + and whether majority membership offers enough benefit for its availability cost. +- The authority/recovery contract for registered notebooks, bare imports, + intentional restores, duplicate notebook copies and copied owner credentials. +- Protected-key custody and unsupported-client migration/escape behavior. +- Retention/compaction guarantees and a measured provider capability floor. + +Unknown protocol/required-op versions must leave remote data untouched and keep +local edits recoverable. Enabling coordination requires an explicit migration +record and closure; old builds that ignore `.snowbound` can still edit `.one` +uncoordinated. A sidecar marker cannot stop those binaries. Rollout therefore +needs an honest compatibility boundary and an opt-in/import path, not a claim +that an unsupported build will obey a new marker. + +The next concrete deliverable after the draft is a narrow transport/owner lab +against real provider fixtures, then one selected policy implemented behind an +explicit storage mode and tested with actual notes. No timeout election or +full-image save shortcut should land as an Android dependency. diff --git a/arc/sync.md b/arc/sync.md index 40eb9668e51ffd7a90cf81382d428a6fd2b8a8d8..d1cc0173d78461f62b9cd32afb0cf0f7b0e91c02 100644 --- a/arc/sync.md +++ b/arc/sync.md @@ -363,3 +363,6 @@ output: every converted page must equal the page the old cache held, or the conversion rolls back and names the archive it made first. The [notebook README](../crates/notebook/README.md) documents the API in detail. + +A separate [draft multi-file coordination protocol](multi-file-sync.md) explores +generic cloud mounts; it is independent of the Android port and is not enabled.