1# Local AI context cache
2
3## Prepared configuration
4
5The current change configures GPU prefix caching, 8 GiB of vLLM shared-memory
6offload, a 16 GiB NVMe hot copy, and a 256 GiB HDD pool. The container has a
726 GiB memory limit and a 10 GiB `/dev/shm`.
8
9`service/local-ai/cache` installs a vLLM secondary-tier plugin. Blocks are
10written through to HDD and copied to SSD; HDD reads warm SSD. Each disk tier
11evicts its least recently used unpinned blocks within its byte cap. Active
12requests pin blocks; a saturated pinned tier skips writes. Direct I/O avoids
13an additional unbounded filesystem data cache. Disk writes and promotions run
14in background workers, outside scheduler lookups.
15
16Disk blocks use authenticated AES-GCM encryption with the service's Nomad
17`cache_key`, owner-only files/directories, and atomic replacement. Restart
18rebuilds the bounded index. Namespaces include the model manifest, vLLM and
19plugin versions, key, KV layout, and parallel configuration. Corrupt blocks
20fall back to another copy or become misses. Key rotation invalidates reuse;
21old entries remain encrypted and subject to eviction.
22
23`config/site.pkl` owns the NVMe path (`/var/cache/studio/local-ai` by default).
24The prepare hook creates it with the assigned service owner. HDD blocks use
25the managed service dataset. Fresh previews use separate managed directories
26for both disk tiers.
27
28Validation on 2026-10-09: the full image built and its installed plugin loaded;
29nine disk-cache tests and two CPU-only tests against the installed vLLM
30scheduler passed. These check encrypted persistence, quotas, pinning,
31concurrent access, SSD promotion, attention/recurrent-group restoration into
32shared RAM, and corrupt-promotion misses. Production/preview Pkl rendering,
33isolated preparation, deployment checks, and six gateway tests passed. These
34tests use synthetic data and do not establish GPU/hybrid-MTP restore timing.
35
36## Production activation — 2026-10-09
37
38Release `0edf5af161627873` was deployed from local `main` without pushing.
39Pre-switch backup: `20261009T214237Z-734b3d`. Nomad deployment and all route
40checks passed. vLLM initialized the tiering connector and an 8.57 GB shared
41region (rounded below the 8 GiB budget to whole blocks).
42
43An authenticated gateway probe completed an 8,000-token synthetic prompt.
44Repeating it reused 6,656 tokens and completed in 3.439 seconds. The first
45request took 37.596 seconds including first-use kernel compilation, so it is
46not a steady-state prefill comparison. Both disk tiers held 66 encrypted
47blocks totaling 1,912,086,528 bytes. Forced RAM/disk restore probes stopped
48when the endpoint reported busy; end-to-end disk restore timing remains
49unverified.
50
51## Zenith measurements — 2026-10-09
52
53HyperQwen, vLLM 0.27.1, MTP with three draft tokens, 150,000-token
54context, GPU prefix caching enabled. Production stayed running; no cache
55connector was enabled or production configuration changed.
56
57`benchmark-local-ai-cache.py` sends synthetic token-ID prompts with one output
58token. Each measured request had zero prefix-cache hits, one completed request,
59and under 0.1 ms queue time. Server prefill time excludes HTTP overhead.
60
61| Prompt tokens | Server prefill seconds |
62| --- | --- |
63| 16,000 | 12.715, 12.557, 12.601 |
64| 64,000 | 63.174, 63.704, 63.896 |
65| 148,000 | 203.552, 204.495, 204.963 |
66
67Storage probes create disposable files filled with incompressible synthetic
68bytes, then read shuffled 16 MiB files with 16 workers. Both writes and reads use
69`O_DIRECT`; writes are individually fsynced. The HDD path is encrypted ZFS on
70the four-disk RAIDZ1 pool, OpenZFS 2.4.4, `direct=standard`, 128 KiB records,
71LZ4 compression. The SSD path is ext4 on the WD Blue SN5000 500 GB NVMe.
72
73| Storage | 1 GiB read seconds | 4 GiB read seconds | 4 GiB write seconds |
74| --- | --- | --- | --- |
75| HDD (`/srv/staging`) | 3.901, 3.686, 3.660, 3.931 | 15.736, 15.399, 16.848, 17.502 | 31.161, 30.165 |
76| NVMe (`/var/tmp`) | 0.262, 0.258, 0.251, 0.258 | 0.937, 0.960, 0.933, 0.929 | 1.282, 1.342 |
77
78The final storage samples additionally checked physical-device read counters:
79the 4 GiB probe read 4,427,632,640 bytes across the HDDs and exactly
804,294,967,296 bytes from NVMe. This verifies reads reached the drives instead
81of being served from the host's data cache.
82
83Disk probes overlapped part of the prefill benchmark. These are storage-only
84timings, not end-to-end KV restores. Actual restores additionally depend on
85attention KV and recurrent-checkpoint bytes, cache-file layout, GPU transfers,
86and any uncached suffix. A larger, fragmented cache and competing pool I/O can
87increase HDD latency. The disposable files were removed after measurement.
88
89HDD reads are promising against the 64k and 148k uncached prefills. End-to-end
90validation with HyperQwen's patched hybrid-state/MTP implementation remains
91necessary; the storage timings above predate the bounded plugin.
92
93## Repeating the probes
94
95Confirm the target and ensure the AI endpoint is idle before inference probes.
96The prefill script reads the existing backend key internally and prints only
97timings and counters; it never prints prompt text, generated text, or keys.
98
99```sh
100ssh -o BatchMode=yes root@10.0.0.1 \
101 'pid=$(pgrep -x vllm); nsenter -t "$pid" -n python3 - prefill "$pid" --hostname zenith' \
102 < tools/benchmark-local-ai-cache.py
103ssh -o BatchMode=yes root@10.0.0.1 \
104 'python3 - filesystem /srv/staging --hostname zenith' \
105 < tools/benchmark-local-ai-cache.py
106ssh -o BatchMode=yes root@10.0.0.1 \
107 'python3 - filesystem /var/tmp --hostname zenith' \
108 < tools/benchmark-local-ai-cache.py
109```
110