From f6b5e343eae39fdf5f4152ae188612773c47c8d4 Mon Sep 17 00:00:00 2001 From: clover caruso Date: Tue, 6 Oct 2026 23:11:12 -0700 Subject: [PATCH] Document infrastructure access, paths, and service workflow --- AGENTS.md | 177 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000000000000000000000000000000000..1d27d77b502d3c5c45c7b663f7310179a22d8cec --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,177 @@ +# Snow Globe infrastructure + +This repo runs Clover's home server: her creative file store, shared apps, and +virtual machines. NixOS provides the host; Nomad schedules Podman containers; +Pkl describes services; the Rust/Solid dashboard manages the system. Treat +availability and existing data as part of correctness. + +## Access and authorization + +- Local edits and rehearsal VM changes are always implied by a task. Carry + them through staging and verification without asking for permission again. +- Production changes require a request to take the work to prod. A request to + fix, stage, test, or commit something does not authorize production deployment. + Once prod is requested, complete the normal deployment workflow. +- Read-only infrastructure inspection over SSH is allowed for the task. Confirm + the target host before a remote mutation; deployment defaults point at the + rehearsal VM, but inherited environment variables can select production. +- Private file contents require authorization per source. Infrastructure access, + a mounted share, or permission to list a tree does not authorize opening its + contents. Keep private material and derived model output inside its store. +- Never print credentials or encryption keys into chat, command arguments, + ordinary logs, or tracked files. Use existing private files or stdin for + imports; keep secret files owner-readable only. Nomad variables own service + secrets; configuration declares names and references, not values. +- Before destructive storage or OS work, require verified recovery: tested key + recovery and app/database restores, with at least two restricted copies stored + separately off the server. Local snapshots and a key archive alone do not + establish data recovery. Follow the linked cutover runbooks. +- Agent/model trials use disposable data and credentials, separated from the + personal archive and host credentials. A service preview may clone production + data and secrets; staging does not itself grant permission to inspect them. + +## Machines and paths + +| Surface | Connection / role | Configuration | +| --- | --- | --- | +| Mac checkout | `/Users/clo/dev/infra-2`; edit and upload here | This working copy | +| Rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool | +| Zenith production | `root@10.0.0.1`, SSH port `22` | `nixos/zenith.nix`, `paperclover.net`, physical pool | + +`tools/deploy.py` reads `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT`; defaults +are the rehearsal connection above. Set both explicitly for production. +Host settings and dataset names have their canonical home in `nixos/zenith.nix` +and `config/site.pkl`; inspect them instead of copying values into new tooling. + +| Local / logical location | Remote location | Meaning | +| --- | --- | --- | +| Checkout release sources | `/opt/studio/releases/` | Uploaded, immutable release trees | +| Published main revision | `/opt/studio/main` | Release exported from described `main` | +| Active deployment | `/opt/studio/current` | Active release link; not an editable checkout | +| Zenith host installation | `/etc/nixos` | Installed host configuration; release activation also manages NixOS | +| Mac SMB `/Volumes/clover` | Zenith `/srv/clover` | Live personal files; not a disposable test tree | +| `/Volumes/clover/Media` | Zenith `/srv/clover/Media` | Media child dataset | +| Mac `/Volumes/media` (launcher default) | VM `/mnt/media-source` → `/srv/clover/Media` | Read-only 9P export and bindfs view for rehearsal | +| Managed service storage | `/srv/prod/` | Persistent, per-service ZFS datasets | +| Preview storage | `/srv/staging/` | Disposable service clones, with separate external clones where needed | +| VM storage | `/srv/vm` | Guest disks, access files, and installation library | +| Deployment/control state | `/var/lib/studio` | UID registry, stages, history, backups, dashboard state, run logs | +| Nomad state | `services.nomad.settings.data_dir` in `nixos/configuration.nix` | Separate from service data; includes ACLs and secrets | + +The rehearsal VM has its own Clover dataset. Its media view references the Mac's +mounted share read-only; those bytes are not synthetic fixtures. See +`tools/local-vm.sh` for launch/cache overrides and `tools/mac-domains.py` for +Mac rehearsal routing. Do not hand-edit DNS, trust, or host files to bypass them. + +## Where changes belong + +- `config/Service.pkl` owns the service schema; `config/site.pkl` owns site + defaults and overrides. `config/excluded-services.json` filters deployment + sources. Services are discovered from `service/*/*.pkl`. +- `service//service.pkl` owns a service's containers, routes, mounts, + requirements, secret declarations, health checks, and resource limits. + Sibling Pkl files define additional services, e.g. Postgres's pgAdmin. +- `tools/studio.py` owns rendering, provisioning, staging, and service setup. + `tools/deploy.py` uploads local sources or `main`; `tools/release.py` owns + release activation and code rollback; `tools/data.py` owns backup/restore. +- `tools/router.py` generates Caddy routes from service discovery. Change the + declaration or generator, not generated routes or Nomad jobs. +- `nixos/` owns host units, packages, isolation, mounts, and hardware settings. + `dashboard/src/` owns the HTTP server and auth; `dashboard/web/` owns the UI; + `tools/dashboard-host.py` and `tools/vms.py` own privileged host operations. +- `guest/` owns guest capture/provisioning and its shared protocol. Inspect + `guest/DESIGN.md` before changing the guest/host boundary. +- Runtime identities live in `/var/lib/studio/identities.json`, initially seeded + from `config/identities.json`. Preserve assigned numeric owners; do not + renumber services or recursively change ownership as a shortcut. + +## Adding or changing a service + +1. Read the closest existing service and the schema. Declare managed storage, + explicit external mounts, health checks, auth, and requirements. Use + `prepare` for repeatable work before startup and `setup` for repeatable + configuration after health. Declare providers instead of hardcoding their + current addresses or credentials. +2. Check side effects before staging: outbound mail, federation, downloads, + automation, fixed host ports, host networking, and exclusive GPU use. Use + preview-aware configuration or `stageIsolation = "fresh"` when a cloned app + would act as production. Fresh stages reject writable external mounts. +3. Run the relevant local checks, then stage the working files: + + ```sh + STUDIO_DEPLOY_HOST=root@127.0.0.1 STUDIO_DEPLOY_PORT=2222 python3 tools/deploy.py stage + ``` + + The command prints `stage`, `release`, `host`, and `dataset`. Use that host; + do not invent a staging hostname. Repeating `stage ` updates its + existing preview and keeps preview data. Storage/isolation or incompatible + requirement changes can require destroying and recreating that preview. +4. Verify the actual app path, authentication, persistence, provider connections, + and healthy allocations. Check logs/restarts and sampled unaffected services. + An HTTP health response alone does not establish that the app works. + Writable external ZFS mounts get clones; read-only external mounts can still + reference the source. Review the actual bindings for host-networked services. +5. Clean up previews when finished using `python3 tools/deploy.py destroy + ` against the same target. Destruction deletes preview state; retain + anything the task needs first. The physical RTX 3090 fits one model instance; + stop its AI preview before a production AI deployment. +6. Commit reviewed changes. Production exports a described, conflict-free + **main**, not the working copy and not the preview's edited database. Only + after production is requested and the intended changes are on `main`: + + ```sh + STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py prod + ``` + + Check deployment history and real production paths. The release workflow + takes a pre-switch backup and performs health checks; use it instead of an + ad hoc container replacement or edits inside `/opt/studio/current`. + +Code rollback uses `deploy.py rollback [release-id]`. It does **not** restore +data. Data restore explicitly discards later writes and is a separate operation; +read `tools/data-restore.md` before proposing it. + +## Verification and change hygiene + +- Run checks that cover the change. Inspect a test's entry point first: some + `tools/*-test.py` scripts mutate host services or require Linux/real guests. + Run those in rehearsal, not on production by default. +- Dashboard: `cd dashboard && npm run check && npm run build`; run Rust tests + with `cargo test --manifest-path dashboard/Cargo.toml`. Host tests include + `python3 tools/dashboard-host-test.py` and `python3 tools/dashboard-vms-test.py`. +- Deployment: `python3 tools/deploy-test.py`. On the target, the uploaded + `tools/studio.py validate` checks generated Nomad jobs; `check` checks routes. + Even render loads/allocates identities, so it is not a pure local formatter. +- Local AI: use the unit checks and live client probes in `tools/local-ai.md`. + Rehearsal has no production GPU/model instance; do not download model weights + or enable GPU inference there. +- UI changes need real browser verification, keyboard checks, and screenshots + or recordings for Clover to judge. Test VM consoles in the normal browser + panel. Metrics/traces must represent observed app telemetry. +- Prefer fewer names, fields, branches, casts, and public promises. Derive state + instead of storing it. Delete dead code and comments describing removed + behavior. One artifact has one home; link additional consumers to it. +- When a trade depends on how likely something is to change, report the trade + and ask Clover. Do not decide it by asserting that the thing is stable. +- Use Jujutsu, never standalone `git`. Inspect `jj st`, the current description, + and `jj diff --git` before editing or committing; preserve unrelated work. + Split independent changes when committing. Do not push or move `main` merely + because a task requests a local commit. Use `Assisted-by: ` if adding + model attribution, never `Co-authored-by`. + +## Operational references + +- Storage layout: [readme.md](readme.md). +- Physical host and recovery context: [tools/zenith-hardware.md](tools/zenith-hardware.md). +- Storage and permissions: [tools/media-cutover.md](tools/media-cutover.md), + [tools/storage-acl-cutover.md](tools/storage-acl-cutover.md). +- Legacy recovery and app imports: [tools/legacy-handoff.md](tools/legacy-handoff.md), + [tools/personal-cutover.md](tools/personal-cutover.md), and the service's + `tools/*-migration.md` runbook. +- Provider behavior: [tools/service-dependencies.md](tools/service-dependencies.md). +- Local inference: [tools/local-ai.md](tools/local-ai.md). + +Runbooks contain dated rehearsal evidence and migration history. Check current +configuration and read-only host state before treating an old observation as +today's topology. Keep deployment history and long investigations in their +runbooks; AGENTS.md carries the rules and links. -- 2.54.0