1# Snow Globe infrastructure
2
3This repo runs Clover's home server: her creative file store, shared apps, and
4virtual machines. NixOS provides the host; Nomad schedules Podman containers;
5Pkl describes services; the Rust/Solid dashboard manages the system. Treat
6availability and existing data as part of correctness.
7
8## Access and authorization
9
10- Local edits and rehearsal VM changes are always implied by a task. Carry
11 them through staging and verification without asking for permission again.
12- Production changes require a request to take the work to prod. A request to
13 fix, stage, test, or commit something does not authorize production deployment.
14 Once prod is requested, complete the normal deployment workflow.
15- Read-only infrastructure inspection over SSH is allowed for the task. Confirm
16 the target host before a remote mutation; deployment defaults point at the
17 rehearsal VM, but inherited environment variables can select production.
18- Private file contents require authorization per source. Infrastructure access,
19 a mounted share, or permission to list a tree does not authorize opening its
20 contents. Keep private material and derived model output inside its store.
21- Never print credentials or encryption keys into chat, command arguments,
22 ordinary logs, or tracked files. Use existing private files or stdin for
23 imports; keep secret files owner-readable only. Nomad variables own service
24 secrets; configuration declares names and references, not values.
25- Before destructive storage or OS work, require verified recovery: tested key
26 recovery and app/database restores, with at least two restricted copies stored
27 separately off the server. Local snapshots and a key archive alone do not
28 establish data recovery. Follow the linked cutover runbooks.
29- Agent/model trials use disposable data and credentials, separated from the
30 personal archive and host credentials. A service preview may clone production
31 data and secrets; staging does not itself grant permission to inspect them.
32
33## Machines and paths
34
35| Surface | Connection / role | Configuration |
36| --- | --- | --- |
37| Mac checkout | `/Users/clo/dev/infra-2`; edit and upload here | This working copy |
38| Rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool |
39| Zenith production | `root@10.0.0.1`, SSH port `22` | `nixos/zenith.nix`, `paperclover.net`, physical pool |
40
41`tools/deploy.py` reads `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT`; defaults
42are the rehearsal connection above. Set both explicitly for production.
43Host settings and dataset names have their canonical home in `nixos/zenith.nix`
44and `config/site.pkl`; inspect them instead of copying values into new tooling.
45
46| Local / logical location | Remote location | Meaning |
47| --- | --- | --- |
48| Checkout release sources | `/opt/studio/releases/<digest>` | Uploaded, immutable release trees |
49| Published main revision | `/opt/studio/main` | Release exported from described `main` |
50| Active deployment | `/opt/studio/current` | Active release link; not an editable checkout |
51| Zenith host installation | `/etc/nixos` | Installed host configuration; release activation also manages NixOS |
52| Mac SMB `/Volumes/clover` | Zenith `/srv/clover` | Live personal files; not a disposable test tree |
53| `/Volumes/clover/Media` | Zenith `/srv/clover/Media` | Media child dataset |
54| Mac `/Volumes/media` (launcher default) | VM `/mnt/media-source` → `/srv/clover/Media` | Read-only 9P export and bindfs view for rehearsal |
55| Managed service storage | `/srv/prod/<service>` | Persistent, per-service ZFS datasets |
56| Preview storage | `/srv/staging/<stage>` | Disposable service clones, with separate external clones where needed |
57| VM storage | `/srv/vm` | Guest disks, access files, and installation library |
58| Deployment/control state | `/var/lib/studio` | UID registry, stages, history, backups, dashboard state, run logs |
59| Nomad state | `services.nomad.settings.data_dir` in `nixos/configuration.nix` | Separate from service data; includes ACLs and secrets |
60
61The rehearsal VM has its own Clover dataset. Its media view references the Mac's
62mounted share read-only; those bytes are not synthetic fixtures. See
63`tools/local-vm.sh` for launch/cache overrides and `tools/mac-domains.py` for
64Mac rehearsal routing. Do not hand-edit DNS, trust, or host files to bypass them.
65
66## Where changes belong
67
68- `config/Service.pkl` owns the service schema; `config/site.pkl` owns site
69 defaults and overrides. `config/excluded-services.json` filters deployment
70 sources. Services are discovered from `service/*/*.pkl`.
71- `service/<name>/service.pkl` owns a service's containers, routes, mounts,
72 requirements, secret declarations, health checks, and resource limits.
73 Sibling Pkl files define additional services, e.g. Postgres's pgAdmin.
74- `tools/studio.py` owns rendering, provisioning, staging, and service setup.
75 `tools/deploy.py` uploads local sources or `main`; `tools/release.py` owns
76 release activation and code rollback; `tools/data.py` owns backup/restore.
77- `tools/router.py` generates Caddy routes from service discovery. Change the
78 declaration or generator, not generated routes or Nomad jobs.
79- `nixos/` owns host units, packages, isolation, mounts, and hardware settings.
80 `dashboard/src/` owns the HTTP server and auth; `dashboard/web/` owns the UI;
81 `tools/dashboard-host.py` and `tools/vms.py` own privileged host operations.
82- `guest/` owns guest capture/provisioning and its shared protocol. Inspect
83 `guest/DESIGN.md` before changing the guest/host boundary.
84- Runtime identities live in `/var/lib/studio/identities.json`, initially seeded
85 from `config/identities.json`. Preserve assigned numeric owners; do not
86 renumber services or recursively change ownership as a shortcut.
87
88## Adding or changing a service
89
901. Read the closest existing service and the schema. Declare managed storage,
91 explicit external mounts, health checks, auth, and requirements. Use
92 `prepare` for repeatable work before startup and `setup` for repeatable
93 configuration after health. Declare providers instead of hardcoding their
94 current addresses or credentials.
952. Check side effects before staging: outbound mail, federation, downloads,
96 automation, fixed host ports, host networking, and exclusive GPU use. Use
97 preview-aware configuration or `stageIsolation = "fresh"` when a cloned app
98 would act as production. Fresh stages reject writable external mounts.
993. Run the relevant local checks, then stage the working files:
100
101 ```sh
102 STUDIO_DEPLOY_HOST=root@127.0.0.1 STUDIO_DEPLOY_PORT=2222 python3 tools/deploy.py stage <service>
103 ```
104
105 The command prints `stage`, `release`, `host`, and `dataset`. Use that host;
106 do not invent a staging hostname. Repeating `stage <service>` updates its
107 existing preview and keeps preview data. Storage/isolation or incompatible
108 requirement changes can require destroying and recreating that preview.
1094. Verify the actual app path, authentication, persistence, provider connections,
110 and healthy allocations. Check logs/restarts and sampled unaffected services.
111 An HTTP health response alone does not establish that the app works.
112 Writable external ZFS mounts get clones; read-only external mounts can still
113 reference the source. Review the actual bindings for host-networked services.
1145. Clean up previews when finished using `python3 tools/deploy.py destroy
115 <stage-id>` against the same target. Destruction deletes preview state; retain
116 anything the task needs first. The physical RTX 3090 fits one model instance;
117 stop its AI preview before a production AI deployment.
1186. Commit reviewed changes. Production exports a described, conflict-free
119 **main**, not the working copy and not the preview's edited database. Only
120 after production is requested and the intended changes are on `main`:
121
122 ```sh
123 STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py prod
124 ```
125
126 Check deployment history and real production paths. The release workflow
127 takes a pre-switch backup and performs health checks; use it instead of an
128 ad hoc container replacement or edits inside `/opt/studio/current`.
129
130Code rollback uses `deploy.py rollback [release-id]`. It does **not** restore
131data. Data restore explicitly discards later writes and is a separate operation;
132read `tools/data-restore.md` before proposing it.
133
134## Verification and change hygiene
135
136- Run checks that cover the change. Inspect a test's entry point first: some
137 `tools/*-test.py` scripts mutate host services or require Linux/real guests.
138 Run those in rehearsal, not on production by default.
139- Dashboard: `cd dashboard && npm run check && npm run build`; run Rust tests
140 with `cargo test --manifest-path dashboard/Cargo.toml`. Host tests include
141 `python3 tools/dashboard-host-test.py` and `python3 tools/dashboard-vms-test.py`.
142- Deployment: `python3 tools/deploy-test.py`. On the target, the uploaded
143 `tools/studio.py validate` checks generated Nomad jobs; `check` checks routes.
144 Even render loads/allocates identities, so it is not a pure local formatter.
145- Local AI: use the unit checks and live client probes in `tools/local-ai.md`.
146 Rehearsal has no production GPU/model instance; do not download model weights
147 or enable GPU inference there.
148- UI changes need real browser verification, keyboard checks, and screenshots
149 or recordings for Clover to judge. Test VM consoles in the normal browser
150 panel. Metrics/traces must represent observed app telemetry.
151- Prefer fewer names, fields, branches, casts, and public promises. Derive state
152 instead of storing it. Delete dead code and comments describing removed
153 behavior. One artifact has one home; link additional consumers to it.
154- When a trade depends on how likely something is to change, report the trade
155 and ask Clover. Do not decide it by asserting that the thing is stable.
156- Use Jujutsu, never standalone `git`. Inspect `jj st`, the current description,
157 and `jj diff --git` before editing or committing; preserve unrelated work.
158 Split independent changes when committing. Do not push or move `main` merely
159 because a task requests a local commit. Use `Assisted-by: <model>` if adding
160 model attribution, never `Co-authored-by`.
161
162## Operational references
163
164- Storage layout: [readme.md](readme.md).
165- Physical host and recovery context: [tools/zenith-hardware.md](tools/zenith-hardware.md).
166- Storage and permissions: [tools/media-cutover.md](tools/media-cutover.md),
167 [tools/storage-acl-cutover.md](tools/storage-acl-cutover.md).
168- Legacy recovery and app imports: [tools/legacy-handoff.md](tools/legacy-handoff.md),
169 [tools/personal-cutover.md](tools/personal-cutover.md), and the service's
170 `tools/*-migration.md` runbook.
171- Provider behavior: [tools/service-dependencies.md](tools/service-dependencies.md).
172- Local inference: [tools/local-ai.md](tools/local-ai.md).
173
174Runbooks contain dated rehearsal evidence and migration history. Check current
175configuration and read-only host state before treating an old observation as
176today's topology. Keep deployment history and long investigations in their
177runbooks; AGENTS.md carries the rules and links.