Secrets Inventory and Rotation Guide

1. Secrets Inventory

1.1 AI / ML Provider Keys

Variable	Provider	Storage Location	Used By	Rotation URL
`ANTHROPIC_API_KEY`	Anthropic (Claude)	Cloudflare Worker secrets, AWS SSM	`workers/api` (image enhancement, PDF conversion)	https://console.anthropic.com/settings/keys
`GEMINI_API_KEY`	Google AI Studio	Cloudflare Worker secrets	`workers/api` (PDF conversion, vision)	https://aistudio.google.com/apikey
`OPENAI_API_KEY`	OpenAI	Cloudflare Worker secrets	`workers/api` (GPT-4o mini vision)	https://platform.openai.com/api-keys
`MISTRAL_API_KEY`	Mistral AI	Cloudflare Worker secrets	`workers/api` (OCR extraction, optional)	https://console.mistral.ai/api-keys
`MARKER_API_KEY`	Datalab	Cloudflare Worker secrets	`workers/api` (PDF-to-Markdown)	Datalab dashboard

1.2 Document Processing Keys

Variable	Provider	Storage Location	Used By	Rotation URL
`MATHPIX_APP_ID`	Mathpix	Cloudflare Worker secrets	`workers/api` (math OCR)	https://accounts.mathpix.com
`MATHPIX_APP_KEY`	Mathpix	Cloudflare Worker secrets	`workers/api` (math OCR)	https://accounts.mathpix.com

1.3 Payment Processing

Variable	Provider	Storage Location	Used By	Rotation URL
`STRIPE_SECRET_KEY`	Stripe	Cloudflare Worker secrets	`workers/api`, `workers/org-chart-api` (billing)	https://dashboard.stripe.com/apikeys
`STRIPE_WEBHOOK_SECRET`	Stripe	Cloudflare Worker secrets	`workers/api` (webhook verification)	https://dashboard.stripe.com/webhooks
`STRIPE_PUBLISHABLE_KEY`	Stripe	Cloudflare Worker secrets	Frontend apps (client-side Stripe.js)	https://dashboard.stripe.com/apikeys
`STRIPE_LINKS_PRO_MONTHLY_PRICE_ID`	Stripe	Cloudflare Worker secrets	`workers/api` (pricing)	Stripe Products dashboard
`STRIPE_LINKS_PRO_YEARLY_PRICE_ID`	Stripe	Cloudflare Worker secrets	`workers/api` (pricing)	Stripe Products dashboard

1.4 Database & Auth

Variable	Provider	Storage Location	Used By	Rotation URL
`SUPABASE_URL`	Supabase	`.env` files, Cloudflare secrets	All apps	https://supabase.com/dashboard/project/vuvwmfxssjosfphzpzim/settings/api
`SUPABASE_ANON_KEY`	Supabase	`.env` files, Cloudflare secrets	All apps (public, safe to expose)	Same as above
`SUPABASE_SERVICE_ROLE_KEY`	Supabase	Cloudflare secrets, AWS SSM	`workers/api`, server-side operations	Same as above
`JWT_SECRET`	Self-managed	Cloudflare Worker secrets	`workers/api` (token signing)	Generate new: `openssl rand -base64 48`

1.5 Email & Notifications

Variable	Provider	Storage Location	Used By	Rotation URL
`RESEND_API_KEY`	Resend	Cloudflare Worker secrets, AWS SSM	`workers/api` (transactional email)	https://resend.com/api-keys
`AWS_ACCESS_KEY_ID`	AWS IAM	Cloudflare Worker secrets, GitHub Actions secrets	`workers/api` (SES), deploy pipeline	https://console.aws.amazon.com/iam
`AWS_SECRET_ACCESS_KEY`	AWS IAM	Cloudflare Worker secrets, GitHub Actions secrets	`workers/api` (SES), deploy pipeline	Same as above
`SES_WEBHOOK_SECRET`	Self-managed	Cloudflare Worker secrets	`workers/api` (EventBridge auth)	Generate new: `openssl rand -hex 32`
`TELEGRAM_BOT_TOKEN`	Telegram	Cloudflare Worker secrets	`workers/api` (admin alerts, optional)	https://t.me/BotFather

1.6 Infrastructure & Deployment

Variable	Provider	Storage Location	Used By	Rotation URL
`CLOUDFLARE_API_TOKEN`	Cloudflare	GitHub Actions secrets	CI/CD deploy pipeline	https://dash.cloudflare.com/profile/api-tokens
`PACKAGES_TOKEN`	GitHub	GitHub Actions secrets	CI/CD (private npm registry)	https://github.com/settings/tokens
`GITHUB_TOKEN`	GitHub	Cloudflare Worker secrets	`workers/api` (issue creation)	https://github.com/settings/tokens

1.7 Web Push (Optional)

Variable	Provider	Storage Location	Used By
`VAPID_PUBLIC_KEY`	Self-generated	Cloudflare Worker secrets	Push notifications
`VAPID_PRIVATE_KEY`	Self-generated	Cloudflare Worker secrets	Push notifications

1.8 Search / Enrichment (Optional)

Variable	Provider	Storage Location	Used By
`SERPER_API_KEY`	Serper.dev	Cloudflare Worker secrets	Prospect enrichment

1.9 Non-Secret Configuration (reference only)

These are not secrets but are deployed alongside them:

CLOUDFLARE_ACCOUNT_ID: c6cce84d1636ec85ec946a19edef0103
SUPABASE_PROJECT_REF (prod): vuvwmfxssjosfphzpzim
SUPABASE_PROJECT_REF (dev): rmwscaavdmzoerqyppbf
KV namespace IDs (in wrangler.toml)
R2 bucket names (in wrangler.toml)
SES_FROM_EMAIL, SES_CONFIGURATION_SET, PUBLIC_BASE_URL
Stripe price IDs

2. Where Secrets Are Stored

Store	What Lives There	Access
Cloudflare Worker Secrets	All runtime secrets for Workers	`wrangler secret put <KEY>` or Cloudflare dashboard
AWS SSM Parameter Store	Supabase, S3, Resend keys for Lambda	AWS Console or `aws ssm put-parameter`
GitHub Actions Secrets	`CLOUDFLARE_API_TOKEN`, `PACKAGES_TOKEN`, `AWS_*`	GitHub repo Settings → Secrets
Local `.env` files	Dev-only values, never committed	`.gitignore` covers `.env`, `.env.*`
`tools/benchmark-cli/.env`	Local dev keys for benchmarking	`.gitignore` covers it — not in git

3. Rotation Procedures

General Rotation Protocol

For every key rotation, follow this sequence:

Generate a new key in the provider’s dashboard (do NOT revoke the old key yet)
Deploy the new key to all locations that consume it (see table below)
Verify the service works with the new key (hit a test endpoint, run a smoke test)
Revoke the old key in the provider’s dashboard
Log the rotation date (update the table in Section 5)

3.1 Cloudflare Worker Secrets

Most runtime secrets live here. To rotate:

# Set the new secret on each worker that uses it
wrangler secret put ANTHROPIC_API_KEY --name accessible-pdf-api
wrangler secret put ANTHROPIC_API_KEY --name accessible-org-chart-api
# ... repeat for each worker listed in workers/ and apps/

# Verify by hitting a test endpoint or checking logs
curl -s https://api.theaccessible.org/health | jq .

Workers that consume secrets (check each wrangler.toml for the authoritative list):

Worker	Key Secrets
`accessible-pdf-api`	ANTHROPIC, GEMINI, OPENAI, MISTRAL, MATHPIX, MARKER, STRIPE, RESEND, JWT_SECRET, SUPABASE, AWS, SES_WEBHOOK_SECRET
`accessible-org-chart-api`	STRIPE, SUPABASE
`accessible-convert-email`	SUPABASE, RESEND
`accessible-gateway`	JWT_SECRET
`accessible-phone`	SUPABASE
Other app workers	Varies — check individual `wrangler.toml`

3.2 AWS SSM Parameter Store

# Update a parameter (SecureString)
aws ssm put-parameter \
  --name "/accessible-pdf/production/RESEND_API_KEY" \
  --value "re_NEW_KEY_HERE" \
  --type SecureString \
  --overwrite

# Lambda functions pick up new SSM values on next cold start.
# Force a refresh by redeploying or updating the function config:
aws lambda update-function-configuration \
  --function-name accessible-pdf-api \
  --description "rotated RESEND_API_KEY $(date +%Y-%m-%d)"

3.3 GitHub Actions Secrets

Go to GitHub repo → Settings → Secrets and variables → Actions
Click the secret name → Update
Paste the new value → Save
The next CI/CD run will use the new secret

3.4 Provider-Specific Notes

Supabase Keys

Supabase keys cannot be independently rotated — they are derived from the JWT secret
To rotate: Dashboard → Settings → API → “Generate new keys”
This invalidates the anon key, service_role key, and JWT secret simultaneously
Impact: Every app and worker that uses Supabase will need the new keys deployed
Downtime risk: HIGH — plan for a brief maintenance window

Stripe Keys

Stripe supports rolling keys — create a new key, both work during the overlap
Rotate the webhook secret by creating a new webhook endpoint, testing it, then deleting the old one
Test mode keys (sk_test_*) and live keys (sk_live_*) are rotated independently

AWS IAM Credentials

Create a second access key on the IAM user (AWS allows 2 active keys)
Deploy the new key to all consumers
Verify, then deactivate and delete the old key
Better long-term: migrate to IAM roles (no long-lived keys)

Self-Managed Secrets (JWT_SECRET, SES_WEBHOOK_SECRET)

# Generate a new secret
NEW_SECRET=$(openssl rand -base64 48)

# Deploy to Cloudflare
wrangler secret put JWT_SECRET --name accessible-pdf-api <<< "$NEW_SECRET"
wrangler secret put JWT_SECRET --name accessible-gateway <<< "$NEW_SECRET"

# Deploy to AWS SSM
aws ssm put-parameter \
  --name "/accessible-pdf/production/JWT_SECRET" \
  --value "$NEW_SECRET" \
  --type SecureString \
  --overwrite

Warning: Rotating JWT_SECRET invalidates all active user sessions. Do this during low-traffic hours.

4. Automation

4.1 GitHub Action: Full End-to-End Rotation

Workflow: .github/workflows/rotate-secrets.yml

Automates the complete rotation lifecycle: cloud secrets → test server → rolling production deploy.

How it works:

Updates Cloudflare Worker secrets and AWS SSM Parameter Store
SSHes to 10.1.1.17 (test server), updates .env.node-server, restarts pptx-remediate, runs health check
If test passes, SSHes to 10.1.1.4 (production), updates .env.node-server, rolling-restarts api-node-1 then api-node-2, then sidecars
If any step fails, .env.node-server is rolled back from .bak and containers are restarted with old values

Usage:

# Rotate a single key
gh workflow run rotate-secrets.yml -f anthropic_api_key=sk-ant-api03-newkeyhere

# Rotate multiple keys at once
gh workflow run rotate-secrets.yml \
  -f stripe_secret_key=sk_live_newkey \
  -f resend_api_key=re_newkey

# Rotate JWT (caution: invalidates all sessions)
gh workflow run rotate-secrets.yml -f jwt_secret=$(openssl rand -base64 48)

Prerequisites — add SERVER_SSH_KEY to GitHub Actions secrets (the ~/.ssh/nightly-audit private key). CLOUDFLARE_API_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY already exist.

4.2 Helper Scripts

scripts/update-env-secret.sh — Safely updates a single key in .env.node-server with backup
scripts/smoke-test.sh — Lightweight health check (HTTP 200 + status field)

4.3 Quarterly Reminders

Workflow: .github/workflows/rotation-reminder.yml

Runs quarterly (Jan, Apr, Jul, Oct) and opens a GitHub issue with a rotation checklist.

4.4 Manual Rotation (Cloudflare Workers only)

For quick cloud-only rotation without touching on-prem servers:

# scripts/rotate-secrets.sh — Deploy a rotated secret to Cloudflare Workers
# Usage: ./scripts/rotate-secrets.sh <SECRET_NAME> <NEW_VALUE>

# Map secret names to the workers that consume them
declare -A SECRET_WORKERS
SECRET_WORKERS=(
  ["ANTHROPIC_API_KEY"]="accessible-pdf-api"
  ["GEMINI_API_KEY"]="accessible-pdf-api"
  ["OPENAI_API_KEY"]="accessible-pdf-api"
  ["MISTRAL_API_KEY"]="accessible-pdf-api"
  ["MARKER_API_KEY"]="accessible-pdf-api"
  ["MATHPIX_APP_KEY"]="accessible-pdf-api"
  ["MATHPIX_APP_ID"]="accessible-pdf-api"
  ["RESEND_API_KEY"]="accessible-pdf-api accessible-convert-email"
  ["STRIPE_SECRET_KEY"]="accessible-pdf-api accessible-org-chart-api"
  ["STRIPE_WEBHOOK_SECRET"]="accessible-pdf-api"
  ["JWT_SECRET"]="accessible-pdf-api accessible-gateway"
  ["SES_WEBHOOK_SECRET"]="accessible-pdf-api"
)

# Also update AWS SSM if applicable
SSM_SECRETS=("SUPABASE_SERVICE_ROLE_KEY" "SUPABASE_JWT_SECRET" "RESEND_API_KEY" "JWT_SECRET" "S3_ACCESS_KEY_ID" "S3_SECRET_ACCESS_KEY")
for SSM_KEY in "${SSM_SECRETS[@]}"; do
  if [[ "$SECRET_NAME" == "$SSM_KEY" ]]; then
    echo "  → Updating AWS SSM /accessible-pdf/production/$SECRET_NAME"
    aws ssm put-parameter \
      --name "/accessible-pdf/production/$SECRET_NAME" \
      --value "$NEW_VALUE" \
      --type SecureString \
      --overwrite
  fi
done

echo ""
echo "Verifying health..."
sleep 3
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.theaccessible.org/health)
if [[ "$HTTP_STATUS" == "200" ]]; then
  echo "Health check passed (HTTP $HTTP_STATUS)."
else
  echo "WARNING: Health check returned HTTP $HTTP_STATUS — investigate before revoking old key."
  exit 1
fi

echo ""
echo "Done. Now revoke the old key in the provider dashboard."
echo "Log the rotation date in docs/admin/secrets-inventory-and-rotation.md Section 5."

4.2 Scheduled Rotation Reminders

Add a GitHub Actions workflow that opens an issue every 90 days:

name: Secret Rotation Reminder
on:
  schedule:
    - cron: '0 9 1 */3 *'   # 9 AM on the 1st of Jan, Apr, Jul, Oct

jobs:
  remind:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Create rotation reminder issue
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh issue create \
            --title "Quarterly secret rotation due ($(date +%B\ %Y))" \
            --body "$(cat <<'BODY'
          ## Quarterly Secret Rotation Checklist

          Review and rotate the following keys per
          `docs/admin/secrets-inventory-and-rotation.md`:

          ### High-Priority (rotate every 90 days)
          - [ ] `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
          - [ ] `CLOUDFLARE_API_TOKEN`
          - [ ] `PACKAGES_TOKEN` / `GITHUB_TOKEN`
          - [ ] `JWT_SECRET`

          ### Standard (rotate every 6 months, check quarterly)
          - [ ] `ANTHROPIC_API_KEY`
          - [ ] `GEMINI_API_KEY`
          - [ ] `OPENAI_API_KEY`
          - [ ] `STRIPE_SECRET_KEY` + `STRIPE_WEBHOOK_SECRET`
          - [ ] `RESEND_API_KEY`

          ### Low-Priority (rotate annually or on suspicion)
          - [ ] `MISTRAL_API_KEY`
          - [ ] `MARKER_API_KEY`
          - [ ] `MATHPIX_APP_KEY`
          - [ ] `VAPID_PRIVATE_KEY`
          - [ ] `TELEGRAM_BOT_TOKEN`
          - [ ] `SERPER_API_KEY`

          ### Post-Rotation
          - [ ] Update rotation log in `docs/admin/secrets-inventory-and-rotation.md` Section 5
          - [ ] Run `scripts/rotate-secrets.sh` for each rotated key
          - [ ] Verify all health checks pass
          - [ ] Revoke old keys in provider dashboards
          BODY
          )" \
            --label "security,chore"

4.3 Fully Automated Rotation (Future State)

For keys that support programmatic rotation, the flow can be fully automated:

Provider	Automation Path	Feasibility
AWS IAM	AWS Secrets Manager auto-rotation with a Lambda rotator	Ready — AWS provides built-in rotation for IAM keys
Stripe	Stripe API `POST /v1/api_keys/roll` (rolling key)	Ready — Stripe supports API-driven key rolling
GitHub PAT	GitHub API to create fine-grained tokens, revoke old	Ready — use fine-grained PATs with expiry dates
Cloudflare	Cloudflare API to create/revoke API tokens	Ready — API supports token lifecycle management
Anthropic	No rotation API — manual dashboard only	Manual
OpenAI	No rotation API — manual dashboard only	Manual
Google AI	Google Cloud API to manage API keys	Ready (if using Google Cloud API keys)
Resend	No rotation API — manual dashboard only	Manual
Supabase	No independent key rotation — regenerates all keys	Manual, high-impact

Recommended next step: Start with AWS IAM key rotation via Secrets Manager (highest risk, best automation support), then Stripe and GitHub tokens.

5. Rotation Log

Secret	Last Rotated	Rotated By	Next Due	Notes
Fill in after first rotation cycle

6. Emergency Rotation (Key Compromise)

If a key is suspected compromised:

Immediately revoke the old key in the provider dashboard (do not wait for overlap)
Generate a new key
Run scripts/rotate-secrets.sh <SECRET_NAME> <NEW_VALUE>
Check application logs for unauthorized usage during the exposure window
If AWS keys: check CloudTrail for unauthorized API calls
If Stripe keys: check Stripe dashboard for unauthorized charges
File a security incident in GitHub Issues with label security
Notify larry@anglin.com via ~/.claude/scripts/send-email.sh