Status Page — status.gevell.com

Public status page + prober. Code: workers/status/ (README has the full reference). This runbook covers operations.

Architecture

• Where: Cloudflare Worker gevell-status (anglin.com account c6cce84d…), custom domain status.gevell.com, D1 database gevell-status (id 36525b8a-9f7a-46e5-ae26-45684dba2f90, ENAM).
• Why Cloudflare + D1: it must stay up when Supabase, the Node server (.4), or the AWS Lambda are down. Nothing in the worker touches those except as probe targets.
• Cadence: cron every minute, probing in waves of 6 (Workers allow only six connections in the header-wait phase; firing all at once lets six slow targets time out the rest and fabricate a total outage). 3 consecutive failed probes = down, so an alert lands 2–3 minutes after an outage begins: 2 min after the first failed probe, plus up to 1 min of detection lag, longer if a cron trigger slips.
• Uptime figures: one-minute probe resolution bucketed by UTC day in daily_uptime. ⚠️ The page shows rolling 30/90-day windows, not the calendar-month figure the SLA commits to (apps/home/src/content/legal/sla) — compute that from daily_uptime directly. Also note ok/total counts only minutes that were actually probed, so minutes when the prober itself was down are absent from the denominator rather than counted as downtime. Treat these numbers as an operational signal, and reconcile before quoting them in an SLA credit determination.
• Freshness: every freshness claim derives from MAX(last_checked). /health returns 503 when the last cycle is older than STALE_AFTER_SECONDS (300), the banner flips to “Monitoring is not reporting”, and the JSON exposes stale/ageSeconds. A dead prober is visible instead of showing a stale green page.
• Relationship to other monitors: the Python health-monitor daemon (docs/admin/operations/health-monitor.md) still covers the two API hosts and logs to Supabase healthcheck_events; Uptime Kuma is unchanged. This page is the customer-facing surface and covers ~22 targets.

What is monitored

Driven by workers/status/src/targets.ts: 14 applications — every product in the canonical registry (including Gevell Workshop, which lives on a gevell.com path and so needs its own target) plus Forms and Slides, which ship ahead of the registry — 3 API hosts (api.gevell.com, api-pdf.gevell.com, worker.gevell.com — checked for data.status == "healthy"), Supabase Auth / Database API / Storage, gevell.com and help.gevell.com. Critical (major-outage banner): Platform API, PDF Conversion API, Authentication, Database API. That set is pinned by a test — targets.test.ts asserts the complete list, so it cannot drift from this runbook silently.

Common tasks

Task	How
Add / remove / rename a target	Edit TARGETS in src/targets.ts, run npm test (drift guard vs products registry), merge → auto-deploys. ⚠️ A change to packages/accessible-org-ui/src/products.ts alone does not redeploy this worker (the path filter is workers/status/**), so a product rename breaks the name test without refreshing the page — deploy manually via deploy_status.
Post an incident	STATUS_ADMIN_TOKEN=… npm run incident -- open "Title" "Text" --impact major --targets api-node (see README).
Schedule maintenance	npm run incident -- maint "Title" "Text" --start <ISO> --end <ISO> [--targets …]. Inside the window affected targets show “Maintenance” instead of “Down” and the banner reads maintenance.
Silence a flapping target	Raise timeoutMs on that target — the only per-target alerting lever. degradedAboveMs only affects degraded styling; degraded never alerts. FAILURES_BEFORE_DOWN is global, so raising it slows detection everywhere.
Stop a known-broken target paging you	Set mutedReason on it (must cite an issue). It stays probed and visible on the page but never alerts.
Redeploy without a code change	Run the deploy.yml workflow manually with the deploy_status input — the third deploy path, and the one you want mid-outage.
Rotate the admin token	npx wrangler secret put STATUS_ADMIN_TOKEN in workers/status.
Inspect raw data	npx wrangler d1 execute gevell-status --remote --command "select * from target_state".
Check the worker itself	https://status.gevell.com/health; Cloudflare dashboard → Workers → gevell-status → Logs (observability on).

Alerts

Email (Resend, noreply@theaccessible.org → larry@anglin.com) and Telegram on down / still-down (hourly) / recovered. Alert failures are logged, never thrown — probing continues even if Resend is down. One batched message per cycle, so a platform-wide outage is one email, not 23.

If every channel fails (or none is configured), the cycle logs ALERT LOST with the affected targets and clears last_alert_at, so the next cycle retries instead of swallowing the transition. Without that, a lost down would go quiet for a full re-alert window and a lost recovered would never be re-sent at all. Telegram secrets are not currently set — email is the only live channel until TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID are added.

First-deploy checklist

1. cd workers/status && npx wrangler secret put SUPABASE_ANON_KEY (+ STATUS_ADMIN_TOKEN, RESEND_API_KEY, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID).
2. Merge → deploy-status job applies migrations and deploys; or npm run deploy.
3. Confirm https://status.gevell.com/health and that the banner leaves “Collecting first results” after a minute.
4. Add https://status.gevell.com/health to Uptime Kuma (who watches the watcher).

Known caveats

• A target whose DNS does not resolve shows Down with a DNS or HTTP 530 error in target_state.last_error. Check last_error before assuming the app itself is broken. Gevell Photos hit exactly this (the app serves images.gevell.com, not the canonical photos.gevell.com); the product was withdrawn from the registry rather than fixed — see issue #3398.
• A missing or stale SUPABASE_ANON_KEY no longer forges an outage. The probe is marked a self-fault, kept out of target state and uptime history, and logged as probe self-fault — it does not turn the public banner red. If both Supabase Auth and Database API look down while Storage (which is probed without the key) looks fine, suspect the key regardless.
• The Supabase Database API probe queries a small anon-readable table (PostgREST returns 200 even for zero rows under RLS), because /rest/v1/ root now requires the service-role key.
• Redirects are not followed, so a target that starts 302-ing (to a login page, a parked domain) reads as Down. That is deliberate.
• Page HTML is ~135 KB uncompressed (90 day-cells × 22 targets with tooltips); Cloudflare compresses it to ~12 KB on the wire.
• The auto-refresh meta tag was removed: it is WCAG technique F41, a Level A failure of SC 2.2.1. The page has a plain Refresh link instead.