authorgravatar for julius0216@outlook.comJulius Marminge <julius0216@outlook.com> 2026-09-24 20:38:31-07:00
committergravatar for noreply@github.comGitHub <noreply@github.com> 2026-09-24 20:38:31-07:00
logfc46b8c3dd691ed2e39da28a2407e1b1597f2832
tree7fcda780ca701a5da13c8cd1d360d640d65437c8
parent568c9bc4d0ef0dddf74d813619c9877bd35624d7
signature Signed by PGP key B5690EEEBB952194

ci(relay): add a forced manual relay deploy (#13550)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2 files changed, 26 insertions(+), 8 deletions(-)

.github/workflows/deploy-relay.yml+12-1
...@@ -4,6 +4,15 @@ on:...@@ -4,6 +4,15 @@ on:
4 push:4 push:
5 branches:5 branches:
6 - main6 - main
7 # Alchemy does not redeploy the Worker when only a Config value read in its
8 # Init changes (alchemy-run/alchemy#1831), so a changed repository variable
9 # needs a forced deploy to reach production.
10 workflow_dispatch:
11 inputs:
12 force:
13 description: Redeploy every resource, including ones with no detected changes
14 type: boolean
15 default: true
716
8permissions:17permissions:
9 contents: read18 contents: read
...@@ -17,6 +26,8 @@ concurrency:...@@ -17,6 +26,8 @@ concurrency:
17jobs:26jobs:
18 deploy_relay:27 deploy_relay:
19 name: Deploy production relay28 name: Deploy production relay
29 # A manual run from another branch would deploy that branch to production.
30 if: github.ref == 'refs/heads/main'
20 runs-on: blacksmith-8vcpu-ubuntu-240431 runs-on: blacksmith-8vcpu-ubuntu-2404
21 timeout-minutes: 1532 timeout-minutes: 15
22 environment:33 environment:
...@@ -55,7 +66,7 @@ jobs:...@@ -55,7 +66,7 @@ jobs:
55 - --filter=t3code-relay...66 - --filter=t3code-relay...
5667
57 - name: Deploy production relay stage68 - name: Deploy production relay stage
58 run: vp run --filter t3code-relay deploy --stage prod --yes --no-input69 run: vp run --filter t3code-relay deploy --stage prod --yes --no-input ${{ inputs.force && '--force' || '' }}
59 env:70 env:
60 # The PublishClientConfig action writes the client env here instead71 # The PublishClientConfig action writes the client env here instead
61 # of the repo-root .env; nothing on the runner reads it.72 # of the repo-root .env; nothing on the runner reads it.
docs/operations/release.md+14-7
...@@ -140,6 +140,11 @@ Required `production` environment secrets:...@@ -140,6 +140,11 @@ Required `production` environment secrets:
140- `CLERK_SECRET_KEY`140- `CLERK_SECRET_KEY`
141- `APNS_PRIVATE_KEY`141- `APNS_PRIVATE_KEY`
142142
143The relay Worker reads these variables and secrets when it is deployed. Alchemy does not redeploy the
144Worker when only one of these values changes ([alchemy-run/alchemy#1831](https://github.com/alchemy-run/alchemy/issues/1831)),
145so a push to `main` without relay code changes leaves the old value in place. After changing one, run
146the **Deploy T3 Connect relay** workflow manually from `main` with **force** checked.
147
143The account-scoped repository credentials are consumed by Alchemy while provisioning relay stages; they148The account-scoped repository credentials are consumed by Alchemy while provisioning relay stages; they
144are not bound into the relay Worker. The production deployment uses an Axiom personal access token,149are not bound into the relay Worker. The production deployment uses an Axiom personal access token,
145so `AXIOM_ORG_ID` must accompany `AXIOM_TOKEN`. The `prod` stage owns the retained PlanetScale150so `AXIOM_ORG_ID` must accompany `AXIOM_TOKEN`. The `prod` stage owns the retained PlanetScale
...@@ -163,18 +168,20 @@ because those builds register recovery and replace a deleted tunnel after wake....@@ -163,18 +168,20 @@ because those builds register recovery and replace a deleted tunnel after wake.
1631. Deploy the relay and migration with cleanup `off`.1681. Deploy the relay and migration with cleanup `off`.
1642. Release the server build and confirm current hosts register recovery. Older hosts stay marked1692. Release the server build and confirm current hosts register recovery. Older hosts stay marked
165 legacy and are never candidates.170 legacy and are never candidates.
1663. Set `dry-run`, deploy, and read the sweep counters (`scanned`, `wouldDelete`, `skippedLegacy`,1713. Set `dry-run`, run a forced relay deploy, and read the sweep counters (`scanned`, `wouldDelete`,
167 `skippedOrphan`, `failed`, `truncated`) across several sweeps.172 `skippedLegacy`, `skippedOrphan`, `failed`, `truncated`) across several sweeps. Each sweep records
173 them, and the active `mode`, as `relay.managed_endpoint_reaper.*` attributes on its
174 `relay.managed_endpoint_reaper.sweep` span in Axiom.
1684. Run the disposable-host canary below.1754. Run the disposable-host canary below.
1695. Set `enabled` only after the canary recovers without a server restart.1765. Set `enabled` only after the canary recovers without a server restart.
170177
171The job runs every five minutes with a five-minute grace period for tunnels that lost their178The job runs every five minutes with a five-minute grace period for tunnels that lost their
172connector, so a candidate is usually removed five to ten minutes after it goes down. Tunnels that179connector, so a candidate is usually removed five to ten minutes after it goes down. Tunnels that
173never connected wait an hour. One sweep attempts at most 100 deletions, so a backlog takes longer.180never connected wait an hour. One sweep attempts at most 100 deletions, so a backlog takes longer.
174`RELAY_TUNNEL_CLEANUP_MODE` is read at deploy time. Changing it, including turning cleanup off during181Changing `RELAY_TUNNEL_CLEANUP_MODE`, including turning cleanup off during an incident, needs a forced
175an incident, needs a relay deploy.182relay deploy. Confirm the new `mode` on the next sweep span.
176183
177To roll back, set cleanup to `off` and deploy the relay before downgrading any host. Keep the184To roll back, set cleanup to `off` and run a forced relay deploy before downgrading any host. Keep the
178recovery endpoints deployed while current server builds are in use. The nullable columns can stay.185recovery endpoints deployed while current server builds are in use. The nullable columns can stay.
179186
180### Disposable-host canary187### Disposable-host canary
...@@ -194,8 +201,8 @@ stage, test Cloudflare account, disposable host, and disposable T3 home. Keep pr...@@ -194,8 +201,8 @@ stage, test Cloudflare account, disposable host, and disposable T3 home. Keep pr
194 and pause it with `kill -STOP <first-pid>`. Wait until Cloudflare reports it down for over five201 and pause it with `kill -STOP <first-pid>`. Wait until Cloudflare reports it down for over five
195 minutes.202 minutes.
1965. Confirm dry-run counts the first tunnel in `wouldDelete` and the second in `skippedLegacy`.2035. Confirm dry-run counts the first tunnel in `wouldDelete` and the second in `skippedLegacy`.
1976. Set cleanup `enabled` on the disposable stage and deploy. Confirm in the test Cloudflare account2046. Set cleanup `enabled` on the disposable stage and deploy it with `--force`. Confirm in the test
198 that the first tunnel is deleted and the legacy tunnel still exists.205 Cloudflare account that the first tunnel is deleted and the legacy tunnel still exists.
1997. Resume the first child with `kill -CONT <first-pid>`. Confirm the running server detects the2067. Resume the first child with `kill -CONT <first-pid>`. Confirm the running server detects the
200 repeated rejection, requests recovery, and becomes reachable at the same hostname without a207 repeated rejection, requests recovery, and becomes reachable at the same hostname without a
201 restart.208 restart.