| 1 | # Relay observability |
| 2 | |
| 3 | > For maintainers. Using T3 Code? See [docs/user](../user/). |
| 4 | |
| 5 | The relay Alchemy stack owns a shared Axiom trace setup. The Knife clients and environment |
| 6 | servers do not export to it; local diagnostics use the configured local or user-owned receiver. |
| 7 | The resources below belong to the upstream relay deployment: |
| 8 | |
| 9 | - `t3-code-relay-traces-prod`, the OpenTelemetry trace dataset shared by the Worker, mobile app, and |
| 10 | first-party relay clients |
| 11 | - `t3-code-relay-otel-ingest-prod`, the dataset-scoped Worker ingest token |
| 12 | - `t3-code-mobile-otel-ingest-prod`, the dataset-scoped mobile ingest token |
| 13 | - `t3-code-relay-client-otel-ingest-prod`, the dataset-scoped first-party relay-client ingest token |
| 14 | - `t3-code-relay-recent-spans-prod`, a view of recent request and endpoint spans |
| 15 | |
| 16 | Alchemy stages append their sanitized stage name to isolate resources, for example |
| 17 | `t3-code-relay-traces-dev-julius` for a personal stage. |
| 18 | |
| 19 | Deploy from `infra/relay` with the normal Alchemy workflow: |
| 20 | |
| 21 | ```sh |
| 22 | vp run deploy |
| 23 | ``` |
| 24 | |
| 25 | Alchemy resolves account-level Axiom deployment credentials through its provider. At runtime, the |
| 26 | Worker receives only its scoped ingest token. The Knife does not consume the provisioned mobile or relay-client ingest tokens. |
| 27 | |
| 28 | The Worker emits Effect's built-in HTTP server spans plus endpoint and database child spans. |
| 29 | Effect's OpenTelemetry exporter stores semantic HTTP attributes below the `attributes.` prefix. |
| 30 | For example: |
| 31 | |
| 32 | ```apl |
| 33 | ['t3-code-relay-traces-prod'] |
| 34 | | where kind == 'server' |
| 35 | | extend endpoint = column_ifexists('attributes.http.route', ''), |
| 36 | customAttributes = column_ifexists('attributes.custom', dynamic({})) |
| 37 | | project _time, name, trace_id, duration, |
| 38 | ['attributes.http.request.method'], |
| 39 | ['attributes.url.path'], |
| 40 | ['attributes.http.response.status_code'], |
| 41 | endpoint, |
| 42 | relayOperation = customAttributes['relay']['operation'] |
| 43 | | order by _time desc |
| 44 | | limit 200 |
| 45 | ``` |
| 46 | |
| 47 | The provisioned view also reads the endpoint from `attributes.http.route`. Relay-specific span |
| 48 | annotations are stored under `attributes.custom`; `relay.operation` is one of the emitted custom |
| 49 | attributes. |
| 50 | |
| 51 | Agents should prefer the provisioned view or APL queries for completed incidents instead of |
| 52 | tailing the Cloudflare Worker. The stack does not provision a separate query token. Responders who |
| 53 | need scripted query access use the authorized account-level `AXIOM_TOKEN` together with |
| 54 | `AXIOM_ORG_ID`; scoped ingest tokens remain write-only credentials for their producers. |
| 55 | |
| 56 | DPoP proof failures include the stable `relay.dpop.failure_code` span attribute. A `time_window` |
| 57 | failure means that a signed proof was too old or too far in the future for the relay's allowed |
| 58 | window. It can point to a date or time problem on either device, but it can also result from a |
| 59 | delayed request. The client uses this category, and the absence of a category from an older relay, |
| 60 | to decide whether clock skew is confirmed or only one possible cause. |
| 61 | |
| 62 | ## Webhooks |
| 63 | |
| 64 | A public webhook request is one `relay.hooks.forward` span. Its `relay.hook.outcome` says what |
| 65 | happened: `forwarded`, `held`, `rate_limited`, `inbox_full`, `not_found`, `payload_too_large`, |
| 66 | `environment_unavailable`, or `environment_timeout`. `relay.hook.endpoint_key` identifies the |
| 67 | managed endpoint, and with it the environment. On a forward, `relay.hook.upstream_status` or |
| 68 | `relay.hook.upstream_error` records the environment's answer, and |
| 69 | `relay.hook.upstream_outcome` what it did with the request (`accepted`, `duplicate`, |
| 70 | `prompt_too_long`, `queue_full`, `rejected_signature`, `expired`, `disabled`, ...), from its |
| 71 | `x-t3-hook-outcome` response header. A held request's delivery records the same on its |
| 72 | `relay.inbox.deliver` span. The relay sends its own `traceparent` with each forward and drops any a |
| 73 | sender supplied, and signs each forward (`x-t3-relay-delivery`) so the environment trusts the |
| 74 | relay's delivery id, receive time and trace context only from the relay. The environment records |
| 75 | its `ScheduledTaskService.triggerWebhook` span locally. `relay.hook.rate_limit` says which |
| 76 | budget ran out: `endpoint` or `hook`. `relay.hook.rate_limiter_failed_open` is set when the |
| 77 | Cloudflare rate limiter was unavailable and the request went through unlimited. |
| 78 | |
| 79 | Held requests are handled in the endpoint's `HookInboxObject`, and each call into it is its own |
| 80 | trace, not a child of the forward span: `relay.inbox.hold`, `relay.inbox.wake`, and |
| 81 | `relay.inbox.deliver` for each alarm run. Each carries `relay.hook.endpoint_key`, so they join to |
| 82 | forwards. The details are on the child spans: `HookInboxStore.hold` carries |
| 83 | `relay.inbox.refused` when the inbox refused a request (`max_per_hook`, `max_requests`, |
| 84 | `max_bytes`, or `already_held`). `HookInboxStore.deliverDue` carries `relay.inbox.run_result` |
| 85 | (`drained`, `more_pending`, `busy`, or `unreachable`), how many requests it sent and delivered, |
| 86 | the consecutive failures behind the retry delay, the longest time a delivered request waited, and |
| 87 | the backlog left. A run that errored outright has `relay.inbox.run_result = failed` on the root |
| 88 | `relay.inbox.deliver` span instead. |
| 89 | |
| 90 | Questions these answer: |
| 91 | |
| 92 | ```apl |
| 93 | // Webhook outcomes per hour |
| 94 | ['t3-code-relay-traces-prod'] |
| 95 | | where name == 'relay.hooks.forward' |
| 96 | | summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome']) |
| 97 | |
| 98 | // Inboxes stuck behind an unreachable environment, by endpoint |
| 99 | ['t3-code-relay-traces-prod'] |
| 100 | | where name == 'HookInboxStore.deliverDue' |
| 101 | | extend c = ['attributes.custom'] |
| 102 | | where tostring(c['relay.inbox.run_result']) == 'unreachable' |
| 103 | | project trace_id, failures = toint(c['relay.inbox.consecutive_failures']), |
| 104 | backlog = toint(c['relay.inbox.held_count']) |
| 105 | | join kind=inner ( |
| 106 | ['t3-code-relay-traces-prod'] |
| 107 | | where name == 'relay.inbox.deliver' |
| 108 | | project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key']) |
| 109 | ) on trace_id |
| 110 | | summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint |
| 111 | ``` |