| ... | @@ -202,11 +202,19 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the | ... | @@ -202,11 +202,19 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the |
| 202 | - On an older build linked from web or mobile, the host stays offline until T3 Code on that computer | 202 | - On an older build linked from web or mobile, the host stays offline until T3 Code on that computer |
| 203 | is updated. | 203 | is updated. |
| 204 | | 204 | |
| | 205 | Ship the web and mobile builds that show the offline reason before enabling legacy cleanup, so a |
| | 206 | user whose host is affected sees what to do. The relay adds the `tunnel_released_at` allocation |
| | 207 | column in its first deploy with this change; the legacy switch stays `off` until you set it. |
| | 208 | |
| 205 | 1. Run `vp run --filter t3code-relay tunnels:census` with a read-only Cloudflare token. It counts | 209 | 1. Run `vp run --filter t3code-relay tunnels:census` with a read-only Cloudflare token. It counts |
| 206 | tunnels in every relay stage. The reaper only sees its own stage's tunnels, so clean up the rest | 210 | tunnels in every relay stage. The reaper only sees its own stage's tunnels, so clean up the rest |
| 207 | by hand. | 211 | by hand. |
| 208 | 2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`, | 212 | 2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`, |
| 209 | `totalDown`, and `totalInactive` on the sweep spans for a day. | 213 | `totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the |
| | 214 | tunnels a sweep inspected, at most 500 per status. `totalDown` and `totalInactive` are Cloudflare's |
| | 215 | counts of this stage's tunnels down for over five minutes and never connected for over an hour. |
| | 216 | They include ones the reaper skips, so they are an upper bound on the backlog. The share of `wouldDeleteLegacy` in each sweep's `scanned` estimates how |
| | 217 | much of that total is eligible. |
| 210 | 3. Run the legacy steps of the disposable-host canary below. | 218 | 3. Run the legacy steps of the disposable-host canary below. |
| 211 | 4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer" | 219 | 4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer" |
| 212 | message are live. Without them, a user whose older host lost its tunnel only sees it as offline. | 220 | message are live. Without them, a user whose older host lost its tunnel only sees it as offline. |
| ... | @@ -217,6 +225,21 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the | ... | @@ -217,6 +225,21 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the |
| 217 | budget ran out or Cloudflare rate-limited a deletion. The counters don't say which; the relay | 225 | budget ran out or Cloudflare rate-limited a deletion. The counters don't say which; the relay |
| 218 | logs a warning with the Cloudflare error for each failed deletion. | 226 | logs a warning with the Cloudflare error for each failed deletion. |
| 219 | | 227 | |
| | 228 | In Axiom, filter the relay traces dataset on `name == "relay.managed_endpoint_reaper.sweep"` and |
| | 229 | chart the `attributes.custom.relay.managed_endpoint_reaper.*` fields over time. |
| | 230 | |
| | 231 | Set the legacy mode back to `off` and deploy if any of these happen: |
| | 232 | |
| | 233 | - `failed` stays above a few per sweep. Read the warning log for the Cloudflare error. |
| | 234 | - Users report an environment that is offline with the update message after they have updated T3 |
| | 235 | Code on that computer and restarted it. |
| | 236 | - Relay request errors rise while sweeps run. Deletions share the Postgres connection pool with |
| | 237 | request handlers. |
| | 238 | |
| | 239 | Turning the legacy mode off stops new legacy deletions; `RELAY_TUNNEL_CLEANUP_MODE` keeps deleting |
| | 240 | tunnels of hosts with recovery while it is `enabled`. Deleted tunnels stay deleted; their hosts |
| | 241 | recover as described above. |
| | 242 | |
| 220 | ### Disposable-host canary | 243 | ### Disposable-host canary |
| 221 | | 244 | |
| 222 | This test has not been run against a real Cloudflare account. Run it against a disposable relay | 245 | This test has not been run against a real Cloudflare account. Run it against a disposable relay |
Julius Marminge <julius0216@outlook.com> 2026-10-05 19:53:52-07:00
GitHub <noreply@github.com> 2026-10-05 19:53:52-07:00