Backup status reporting

Backups run as cron/launchd jobs on scattered machines (the Linux server, the Mac minis, the laptop). Their status used to live only in local logs and email — one job failed silently for seven months. Every backup script now POSTs a heartbeat/result to the API; the admin panel surfaces it at /admin/backups and flags any job whose newest run is older than 36 hours as STALE — even if that run succeeded. A job that stops reporting looks exactly like a job that fails.

Endpoint

POST /api/backup-report
Host: api.gevell.com (Node .4) or api-pdf.gevell.com (Lambda)
Header: X-Backup-Token: <BACKUP_REPORT_TOKEN>
Content-Type: application/json

No user auth — machine-to-machine. The X-Backup-Token header must match the BACKUP_REPORT_TOKEN env var (constant-time compare).
If BACKUP_REPORT_TOKEN is unset on the server, the endpoint fails closed with 503. Bad/missing token → 401. Invalid body → 400. Success → 201 with { "success": true, "data": { "id": "<uuid>" } }.
Rows are appended to the backup_runs table (migration 286); nothing is ever updated or deleted (by convention — the service-role policy does not technically forbid it).

The admin read side is GET /api/admin/backup-runs (admin JWT auth via isUserAdmin, same as the other /api/admin/* routes): latest run per jobName plus the last 20 runs overall, each with a computed stale flag. The latest run is queried per job, and every job in the route’s EXPECTED_JOBS list always appears — one that has never reported at all is synthesized as a red never-reported entry instead of being invisible. When wiring a new backup job, add its jobName to EXPECTED_JOBS in workers/api/src/routes/backup-status.ts — otherwise a job that never starts reporting is only discoverable, not guaranteed-visible.

There is also GET /api/backup-report/status (same X-Backup-Token auth as the POST): a machine-readable summary { ok, jobs: [...] } where ok is false if any job is stale, failed, or never reported. The nightly-audit script on the Mac curls this and emails when ok is false, so staleness pages a human instead of waiting for someone to open the dashboard.

Body (camelCase keys)

Field	Required	Notes
`jobName`	yes	e.g. `github-r2`, `server-3-b2`, `mac-secrets-b2`
`machine`	yes	e.g. `m1-mini`, `server-10.1.1.3`, `mac-laptop`
`status`	yes	`success` \| `failure` \| `partial`
`startedAt`	no	ISO 8601 with offset
`finishedAt`	no	ISO 8601 with offset; defaults to now
`bytes`	no	total bytes backed up
`itemsOk`	no	items backed up successfully
`itemsFailed`	no	items that failed
`detail`	no	error message / summary, max 2000 chars

Shell snippet (copy-paste into backup scripts)

API="https://api.gevell.com"
TOKEN="<BACKUP_REPORT_TOKEN value>"   # source from a root-only file, not inline

report_backup() {  # usage: report_backup <status> <detail>
  curl -fsS -X POST "$API/api/backup-report" \
    -H "X-Backup-Token: $TOKEN" \
    -H 'Content-Type: application/json' \
    -d "{\"jobName\":\"github-r2\",\"machine\":\"m1-mini\",\"status\":\"$1\",\"detail\":\"$2\"}" \
    || echo "backup-report POST failed (non-fatal)" >&2
}

# at the end of the backup:
report_backup success ""
# on failure:
report_backup failure "rclone exited 3 on bucket github-backups"

Keep the POST non-fatal (|| ...) so a reporting outage never breaks the backup itself.

Setting BACKUP_REPORT_TOKEN

Generate once (openssl rand -hex 32) and set it in both API runtimes plus each reporting machine:

Lambda (api-pdf.gevell.com) — secrets are NOT declared per-var in the CDK api-stack; the Lambda loads everything under the SSM prefixes /accessible-pdf/<env>/ and /accessible-pdf/shared/ into process.env on cold start (workers/api/src/utils/load-ssm-secrets.ts, and the inlined copy in index-aws.ts). So wire it the same way as SES_WEBHOOK_SECRET and the Supabase keys:
Terminal window
```
aws ssm put-parameter \
  --name /accessible-pdf/shared/BACKUP_REPORT_TOKEN \
  --type SecureString --value "<token>"
```
SSM only reaches process.env. Routes read c.env, which on Lambda is populated by the explicit per-key copy block in index-aws.ts — the BACKUP_REPORT_TOKEN copy line is there; if you ever add another secret for this route family, it needs a copy line too or the Lambda host fails closed with 503 (the env-parity test in workers/api/src/__tests__/env-parity.test.ts enforces this). No CDK change is needed; the next cold start picks up the SSM value.
Node host (.4, api.gevell.com) — add BACKUP_REPORT_TOKEN=<token> to .env.node-server on 10.1.1.4, then npm run rebuild (the docker-compose services read that env file).
Backup machines — store the token in a root-readable file the cron scripts source (never commit it).

Wiring the existing backup jobs

Add a report_backup call to each of:

~/.claude/scripts/github-backup.sh on the M1 mini (10.1.1.10) — jobName: github-r2, machine: m1-mini
~/.claude/scripts/server-backup.sh on 10.1.1.3 — jobName: server-3-b2, machine: server-10.1.1.3
the laptop secrets backup — jobName: mac-secrets-b2, machine: mac-laptop

Pass bytes / itemsOk / itemsFailed where the script already knows them (e.g. repo counts, pg_dump sizes).