| ... | @@ -0,0 +1,177 @@ |
| 1 | # Snow Globe infrastructure |
| 2 | |
| 3 | This repo runs Clover's home server: her creative file store, shared apps, and |
| 4 | virtual machines. NixOS provides the host; Nomad schedules Podman containers; |
| 5 | Pkl describes services; the Rust/Solid dashboard manages the system. Treat |
| 6 | availability and existing data as part of correctness. |
| 7 | |
| 8 | ## Access and authorization |
| 9 | |
| 10 | - Local edits and rehearsal VM changes are always implied by a task. Carry |
| 11 | them through staging and verification without asking for permission again. |
| 12 | - Production changes require a request to take the work to prod. A request to |
| 13 | fix, stage, test, or commit something does not authorize production deployment. |
| 14 | Once prod is requested, complete the normal deployment workflow. |
| 15 | - Read-only infrastructure inspection over SSH is allowed for the task. Confirm |
| 16 | the target host before a remote mutation; deployment defaults point at the |
| 17 | rehearsal VM, but inherited environment variables can select production. |
| 18 | - Private file contents require authorization per source. Infrastructure access, |
| 19 | a mounted share, or permission to list a tree does not authorize opening its |
| 20 | contents. Keep private material and derived model output inside its store. |
| 21 | - Never print credentials or encryption keys into chat, command arguments, |
| 22 | ordinary logs, or tracked files. Use existing private files or stdin for |
| 23 | imports; keep secret files owner-readable only. Nomad variables own service |
| 24 | secrets; configuration declares names and references, not values. |
| 25 | - Before destructive storage or OS work, require verified recovery: tested key |
| 26 | recovery and app/database restores, with at least two restricted copies stored |
| 27 | separately off the server. Local snapshots and a key archive alone do not |
| 28 | establish data recovery. Follow the linked cutover runbooks. |
| 29 | - Agent/model trials use disposable data and credentials, separated from the |
| 30 | personal archive and host credentials. A service preview may clone production |
| 31 | data and secrets; staging does not itself grant permission to inspect them. |
| 32 | |
| 33 | ## Machines and paths |
| 34 | |
| 35 | | Surface | Connection / role | Configuration | |
| 36 | | --- | --- | --- | |
| 37 | | Mac checkout | `/Users/clo/dev/infra-2`; edit and upload here | This working copy | |
| 38 | | Rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool | |
| 39 | | Zenith production | `root@10.0.0.1`, SSH port `22` | `nixos/zenith.nix`, `paperclover.net`, physical pool | |
| 40 | |
| 41 | `tools/deploy.py` reads `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT`; defaults |
| 42 | are the rehearsal connection above. Set both explicitly for production. |
| 43 | Host settings and dataset names have their canonical home in `nixos/zenith.nix` |
| 44 | and `config/site.pkl`; inspect them instead of copying values into new tooling. |
| 45 | |
| 46 | | Local / logical location | Remote location | Meaning | |
| 47 | | --- | --- | --- | |
| 48 | | Checkout release sources | `/opt/studio/releases/<digest>` | Uploaded, immutable release trees | |
| 49 | | Published main revision | `/opt/studio/main` | Release exported from described `main` | |
| 50 | | Active deployment | `/opt/studio/current` | Active release link; not an editable checkout | |
| 51 | | Zenith host installation | `/etc/nixos` | Installed host configuration; release activation also manages NixOS | |
| 52 | | Mac SMB `/Volumes/clover` | Zenith `/srv/clover` | Live personal files; not a disposable test tree | |
| 53 | | `/Volumes/clover/Media` | Zenith `/srv/clover/Media` | Media child dataset | |
| 54 | | Mac `/Volumes/media` (launcher default) | VM `/mnt/media-source` → `/srv/clover/Media` | Read-only 9P export and bindfs view for rehearsal | |
| 55 | | Managed service storage | `/srv/prod/<service>` | Persistent, per-service ZFS datasets | |
| 56 | | Preview storage | `/srv/staging/<stage>` | Disposable service clones, with separate external clones where needed | |
| 57 | | VM storage | `/srv/vm` | Guest disks, access files, and installation library | |
| 58 | | Deployment/control state | `/var/lib/studio` | UID registry, stages, history, backups, dashboard state, run logs | |
| 59 | | Nomad state | `services.nomad.settings.data_dir` in `nixos/configuration.nix` | Separate from service data; includes ACLs and secrets | |
| 60 | |
| 61 | The rehearsal VM has its own Clover dataset. Its media view references the Mac's |
| 62 | mounted share read-only; those bytes are not synthetic fixtures. See |
| 63 | `tools/local-vm.sh` for launch/cache overrides and `tools/mac-domains.py` for |
| 64 | Mac rehearsal routing. Do not hand-edit DNS, trust, or host files to bypass them. |
| 65 | |
| 66 | ## Where changes belong |
| 67 | |
| 68 | - `config/Service.pkl` owns the service schema; `config/site.pkl` owns site |
| 69 | defaults and overrides. `config/excluded-services.json` filters deployment |
| 70 | sources. Services are discovered from `service/*/*.pkl`. |
| 71 | - `service/<name>/service.pkl` owns a service's containers, routes, mounts, |
| 72 | requirements, secret declarations, health checks, and resource limits. |
| 73 | Sibling Pkl files define additional services, e.g. Postgres's pgAdmin. |
| 74 | - `tools/studio.py` owns rendering, provisioning, staging, and service setup. |
| 75 | `tools/deploy.py` uploads local sources or `main`; `tools/release.py` owns |
| 76 | release activation and code rollback; `tools/data.py` owns backup/restore. |
| 77 | - `tools/router.py` generates Caddy routes from service discovery. Change the |
| 78 | declaration or generator, not generated routes or Nomad jobs. |
| 79 | - `nixos/` owns host units, packages, isolation, mounts, and hardware settings. |
| 80 | `dashboard/src/` owns the HTTP server and auth; `dashboard/web/` owns the UI; |
| 81 | `tools/dashboard-host.py` and `tools/vms.py` own privileged host operations. |
| 82 | - `guest/` owns guest capture/provisioning and its shared protocol. Inspect |
| 83 | `guest/DESIGN.md` before changing the guest/host boundary. |
| 84 | - Runtime identities live in `/var/lib/studio/identities.json`, initially seeded |
| 85 | from `config/identities.json`. Preserve assigned numeric owners; do not |
| 86 | renumber services or recursively change ownership as a shortcut. |
| 87 | |
| 88 | ## Adding or changing a service |
| 89 | |
| 90 | 1. Read the closest existing service and the schema. Declare managed storage, |
| 91 | explicit external mounts, health checks, auth, and requirements. Use |
| 92 | `prepare` for repeatable work before startup and `setup` for repeatable |
| 93 | configuration after health. Declare providers instead of hardcoding their |
| 94 | current addresses or credentials. |
| 95 | 2. Check side effects before staging: outbound mail, federation, downloads, |
| 96 | automation, fixed host ports, host networking, and exclusive GPU use. Use |
| 97 | preview-aware configuration or `stageIsolation = "fresh"` when a cloned app |
| 98 | would act as production. Fresh stages reject writable external mounts. |
| 99 | 3. Run the relevant local checks, then stage the working files: |
| 100 | |
| 101 | ```sh |
| 102 | STUDIO_DEPLOY_HOST=root@127.0.0.1 STUDIO_DEPLOY_PORT=2222 python3 tools/deploy.py stage <service> |
| 103 | ``` |
| 104 | |
| 105 | The command prints `stage`, `release`, `host`, and `dataset`. Use that host; |
| 106 | do not invent a staging hostname. Repeating `stage <service>` updates its |
| 107 | existing preview and keeps preview data. Storage/isolation or incompatible |
| 108 | requirement changes can require destroying and recreating that preview. |
| 109 | 4. Verify the actual app path, authentication, persistence, provider connections, |
| 110 | and healthy allocations. Check logs/restarts and sampled unaffected services. |
| 111 | An HTTP health response alone does not establish that the app works. |
| 112 | Writable external ZFS mounts get clones; read-only external mounts can still |
| 113 | reference the source. Review the actual bindings for host-networked services. |
| 114 | 5. Clean up previews when finished using `python3 tools/deploy.py destroy |
| 115 | <stage-id>` against the same target. Destruction deletes preview state; retain |
| 116 | anything the task needs first. The physical RTX 3090 fits one model instance; |
| 117 | stop its AI preview before a production AI deployment. |
| 118 | 6. Commit reviewed changes. Production exports a described, conflict-free |
| 119 | **main**, not the working copy and not the preview's edited database. Only |
| 120 | after production is requested and the intended changes are on `main`: |
| 121 | |
| 122 | ```sh |
| 123 | STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py prod |
| 124 | ``` |
| 125 | |
| 126 | Check deployment history and real production paths. The release workflow |
| 127 | takes a pre-switch backup and performs health checks; use it instead of an |
| 128 | ad hoc container replacement or edits inside `/opt/studio/current`. |
| 129 | |
| 130 | Code rollback uses `deploy.py rollback [release-id]`. It does **not** restore |
| 131 | data. Data restore explicitly discards later writes and is a separate operation; |
| 132 | read `tools/data-restore.md` before proposing it. |
| 133 | |
| 134 | ## Verification and change hygiene |
| 135 | |
| 136 | - Run checks that cover the change. Inspect a test's entry point first: some |
| 137 | `tools/*-test.py` scripts mutate host services or require Linux/real guests. |
| 138 | Run those in rehearsal, not on production by default. |
| 139 | - Dashboard: `cd dashboard && npm run check && npm run build`; run Rust tests |
| 140 | with `cargo test --manifest-path dashboard/Cargo.toml`. Host tests include |
| 141 | `python3 tools/dashboard-host-test.py` and `python3 tools/dashboard-vms-test.py`. |
| 142 | - Deployment: `python3 tools/deploy-test.py`. On the target, the uploaded |
| 143 | `tools/studio.py validate` checks generated Nomad jobs; `check` checks routes. |
| 144 | Even render loads/allocates identities, so it is not a pure local formatter. |
| 145 | - Local AI: use the unit checks and live client probes in `tools/local-ai.md`. |
| 146 | Rehearsal has no production GPU/model instance; do not download model weights |
| 147 | or enable GPU inference there. |
| 148 | - UI changes need real browser verification, keyboard checks, and screenshots |
| 149 | or recordings for Clover to judge. Test VM consoles in the normal browser |
| 150 | panel. Metrics/traces must represent observed app telemetry. |
| 151 | - Prefer fewer names, fields, branches, casts, and public promises. Derive state |
| 152 | instead of storing it. Delete dead code and comments describing removed |
| 153 | behavior. One artifact has one home; link additional consumers to it. |
| 154 | - When a trade depends on how likely something is to change, report the trade |
| 155 | and ask Clover. Do not decide it by asserting that the thing is stable. |
| 156 | - Use Jujutsu, never standalone `git`. Inspect `jj st`, the current description, |
| 157 | and `jj diff --git` before editing or committing; preserve unrelated work. |
| 158 | Split independent changes when committing. Do not push or move `main` merely |
| 159 | because a task requests a local commit. Use `Assisted-by: <model>` if adding |
| 160 | model attribution, never `Co-authored-by`. |
| 161 | |
| 162 | ## Operational references |
| 163 | |
| 164 | - Storage layout: [readme.md](readme.md). |
| 165 | - Physical host and recovery context: [tools/zenith-hardware.md](tools/zenith-hardware.md). |
| 166 | - Storage and permissions: [tools/media-cutover.md](tools/media-cutover.md), |
| 167 | [tools/storage-acl-cutover.md](tools/storage-acl-cutover.md). |
| 168 | - Legacy recovery and app imports: [tools/legacy-handoff.md](tools/legacy-handoff.md), |
| 169 | [tools/personal-cutover.md](tools/personal-cutover.md), and the service's |
| 170 | `tools/*-migration.md` runbook. |
| 171 | - Provider behavior: [tools/service-dependencies.md](tools/service-dependencies.md). |
| 172 | - Local inference: [tools/local-ai.md](tools/local-ai.md). |
| 173 | |
| 174 | Runbooks contain dated rehearsal evidence and migration history. Check current |
| 175 | configuration and read-only host state before treating an old observation as |
| 176 | today's topology. Keep deployment history and long investigations in their |
| 177 | runbooks; AGENTS.md carries the rules and links. |
clover caruso <git@paperclover.net> 2026-10-06 23:11:12-07:00