Skip to content
AnglinAI Docs

Security Remediation Runbook

Companion to incident-risk-register.md. The register says what could go wrong; this says what to do about it, in order, with a definition of done for each item.

How to use it: work top to bottom. Tracks are ordered by risk reduction per hour of effort, not by risk score — a few 15-minute items beat several multi-day ones. Nothing here depends on a later track, so you can stop after any one of them and be meaningfully better off.

Legend. [you] = only you can do it (a purchase, a login, a phone call). [agent] = an agent can do it and hand you a diff to review. [both] = agent drafts, you execute.

Progress is tracked in docs/Policies/security-remediation-progress.md — one line per task with status, date, and a note. Create it on first run.

Track A — Identity (2 hours, do first)

Everything else assumes your accounts are yours. A phished GitHub or Workspace login makes every other control in this document irrelevant. Note the scope limit: this track defends logins, not tokens — most of your credential exposure is API keys, which Track C handles.

A1. Phishing-resistant MFA on the identity root [you] — 60–90 min

Risk #5, and indirectly #21 and #17.

What this protects, precisely. WebAuthn (a hardware key or a platform passkey) signs a challenge bound to the origin your browser is actually talking to. A lookalike domain or a real-time proxy gets a signature for the wrong origin and the login fails, with no judgement required from you. TOTP has no origin binding — six digits go into whatever page asked for them, and adversary-in-the-middle phishing kits relay them within the 30-second window, then steal the session cookie. This closes exactly one door: being tricked into authenticating to an attacker. It does nothing about leaked API tokens (see Track C), stolen session cookies, or a malicious OAuth grant you approve yourself.

Order matters more than coverage. Almost every account has “email me a reset link” as its recovery path, so the Workspace mailbox is the root of the identity tree and the registrar is the root of the domain — which, with invoice billing, is a payment instrument. Do these in order:

  1. Google Workspace — the reset path for everything else, and where BEC (#21) starts.
  2. Domain registrar — controls DNS, DMARC, and mail delivery.
  3. GitHub — all source code, plus the @anglinai/ui publish path (#18).
  4. Cloudflare — DNS, Workers, Pages, R2.
  5. AWS — root account and your IAM user, separately.
  6. Supabase, Stripe, Backblaze — enroll if they support WebAuthn; several of these are TOTP-only. Don’t treat this line as a checklist, check each one.

Passkeys first, one hardware key as backup. Platform passkeys (Touch ID on the Mac, Face ID on the phone, synced via iCloud Keychain) are the same protocol with the same origin binding, cost nothing, and can be enrolled today. For a solo operator the sensible posture is passkeys as the daily factor plus one physical key stored off-site for the case where the Mac and phone are both gone, or the Apple account itself is the problem. Buying four YubiKeys is not the prerequisite — don’t let procurement delay the highest-priority track.

The step people skip: once a phishing-resistant factor is enrolled, remove SMS and TOTP as fallbacks wherever the provider allows it. Leaving the weak factor enabled means an attacker phishes that instead and the strong factor protected nothing.

  • Store recovery codes on paper, off the laptop, somewhere you’d still have after a house fire. These are now the weakest link — treat them like the backup key.
  • GitHub org settings: require 2FA for all members, restrict repo-visibility changes.
  • Done when: Workspace, the registrar, GitHub, and Cloudflare each accept a passkey; SMS/TOTP fallback is removed where possible and noted where it isn’t; recovery codes are in a physical location you can name; and the backup key is enrolled and stored off-site.

A2. Confirm DMARC is at p=reject [both] — 20 min

Risk #17, #21. At p=none you are collecting evidence of spoofing while allowing it.

Terminal window
dig +short TXT _dmarc.theaccessible.org
dig +short TXT gevell.com | grep spf

If the policy is none or quarantine: first confirm every legitimate sender (Resend, Google Workspace, anything else) passes SPF and DKIM alignment in your existing DMARC aggregate reports, then move to reject. Moving to reject with an unaligned sender silently breaks your auth emails — check docs/admin/security/dmarc-reports.md for what’s currently sending.

  • Done when: p=reject, and one test send from each sending source lands in an external inbox.

A3. Register lookalike domains [you] — 20 min

Most invoice fraud uses a cousin domain rather than a true spoof, precisely because DMARC works. Register and park at minimum: theaccessib1e.org, theaccesible.org, theaccessible.net, theaccessible.com if available. Point them nowhere. Enable auto-renew.

  • Done when: registered, auto-renew on, and no MX records (so they can’t receive mail).

A4. Registrar hardening [you] — 15 min

Registrar lock on, registry lock if the TLD offers it, hardware MFA (from A1), auto-renew with a payment method that won’t expire, and a calendar reminder 60 days before expiry. Add CAA records restricting issuance to your CA.

Track B — Money (3 hours, do second)

Risk #21 and #22. These are the highest-likelihood incidents that cost you money directly, and the controls are mostly words on a page rather than code.

B1. Invoice anti-fraud language [both] — 30 min

Add to every invoice template and to the contract/SOW:

Our banking and remittance details never change. Any notice of changed remittance information — whether by email, letter, or phone — is fraudulent. Before acting on any such notice, verify by calling [number] using this number and not one supplied in the message.

Say the same thing verbally at contract signing so the customer’s AP office has it on file before an attacker gets there.

  • Done when: the line is in the invoice template and in the standard agreement.

B2. Banking-detail handling rule [you] — 15 min

Write it down and follow it: bank details are delivered once, through the customer’s supplier portal or a signed PDF, never as email body text, and are never updated by email. Any inbound request to change details — yours or a customer’s — is verified by phone on a number you already had.

B3. Workspace mailbox-rule alerting [both] — 45 min

Auto-forwarding rules and new mailbox delegates are the standard BEC persistence mechanism and therefore the standard detection signal.

  • Google Workspace Admin → disable automatic forwarding to external addresses org-wide unless you actively need it.
  • Set an alert on rule/delegate creation, routed to Telegram or larry@anglin.com.
  • Review existing rules and delegates on all mailboxes right now.
  • Done when: external auto-forwarding is off or alerted, and you’ve reviewed today’s state.

B4. Manual entitlement grant log [agent] — 2 hours

Risk #22. Stripe was your implicit entitlement ledger; invoicing removed it and nothing replaced it.

  • Append-only table (entitlement_grants): who granted, to which org, what was granted, PO number, authorizing contact, timestamp. No updates, no deletes.
  • Every manual grant in the admin UI writes a row; make the PO field required.
  • Monthly: reconcile grants against signed POs, and AR invoices issued against payments received.
  • Done when: the migration is applied and the admin UI cannot grant without writing a row.

B5. Org-membership request rule [you], then [agent] — 30 min now, ~1 day later

Risk #22 — an attacker asking to be added to an existing customer’s org is a cross-tenant breach achieved by email alone.

  • Now (policy): never add a member to an existing org from an inbound email. Route every membership change through the admin contact named in the contract, confirmed out-of-band.
  • Later (product): give customers their own admin console for membership so the request never reaches your inbox, restrict self-join to verified institutional domains with approval by an existing org admin, and notify existing admins whenever a member is added.

Track C — Secrets (1 day)

Risk #1, the highest-likelihood item on the register.

C1. Secret scanning, both gates [agent] — 2 hours

  • gitleaks as a pre-commit hook and as a CI job — a hook alone is bypassed by --no-verify.
  • Enable GitHub secret scanning with push protection at the org level.
  • Done when: a test commit containing a fake AWS key is blocked locally and in CI.

C2. Historical scan [agent] — 2 hours

Run gitleaks detect over full history on every repo. Anything found is already leaked — rotate it, don’t just delete the commit. Record findings in the progress file by location and type only; the progress file is committed to git, so never write the credential value into it.

C3. .dockerignore audit [agent] — 1 hour

Every service with a Dockerfile must exclude .env*. Add a CI assertion so a new service can’t regress it. You already exclude the large corpora; extend the same file.

C4. Rotation runbook [both] — 3 hours

Create docs/Policies/rotation-runbook.md: every credential, where it lives, who issues it, and the exact commands to rotate and redeploy it. Leaks become disasters because rotation is slow and frightening; the runbook is what makes it a 20-minute chore.

Cover at least: Supabase service_role + anon (both projects), AWS access keys, Cloudflare API token, Anthropic and Bedrock/Gemini keys, Stripe secret + webhook signing secret, Resend, GitHub PATs + NODE_AUTH_TOKEN, Telegram bot token, B2 application keys, SSH keys on .3/.4/.17.

  • Rehearse one rotation per quarter and log it.
  • Done when: the document exists and one credential has actually been rotated using it.

C5. Logger redaction test [agent] — 1 hour

Assert the structured logger redacts by key name (*key*, *token*, *secret*, authorization) before any transport. A unit test, so it can’t silently regress.

Track D — Tenancy (2–3 days)

Risk #4, the highest-impact technical item.

D1. Cross-tenant authz test suite [agent] — 2 days

For every route that accepts a resource ID, a test asserting user A cannot read, update, or delete user B’s resource. Wire it into the existing check-route-test-coverage.mjs gate so a new route without a cross-tenant test fails the build.

D2. Supabase advisors in CI [agent] — 3 hours

Run the security advisor against both projects on a schedule; fail or alert on new findings. Confirm RLS is enabled on every table and that service_role appears only in reviewed server paths.

D3. Signed-URL audit [agent] — 3 hours

Every R2/S3 object served through a short-TTL signed URL; no public r2.dev on production buckets. Add a nightly check that an unauthenticated GET of a known key fails, alerting if it succeeds.

Track E — Recovery (4 hours)

Risk #14, #15. You called this the easy one — it is, and it’s the cheapest severity reduction on the list.

E1. B2 immutability and write-only keys [you] — 1 hour

  • Object Lock on the bucket with 30-day retention.
  • Replace the rclone credential on 10.1.1.3 with an application key scoped to listFiles + writeFiles only, no deleteFiles, per source host.
  • Done when: a delete attempt from the server fails, and the nightly backup still succeeds.

E2. Git mirror to B2 [agent] — 1 hour

“It all lives on GitHub” holds until an account compromise deletes or rewrites it. Add a nightly git clone --mirror of every repo into the existing B2 backup script.

E3. Confirm Supabase PITR [you] — 30 min

Verify PITR is enabled on the production project and note the retention window in the register. Add a nightly logical pg_dump too — it covers a case PITR doesn’t, a bad migration faithfully replicated everywhere.

E4. Test a restore [both] — 90 min

Restore the most recent backup into a scratch project and confirm it’s complete. An untested backup is a hypothesis. Log the result and set a quarterly reminder.

Track F — Spend and abuse (1 day)

Risk #3.

F1. Pre-dispatch spend caps [agent] — 4 hours

Hard per-account and per-org daily caps enforced before dispatching inference, using existing cost_tracking figures. Fail closed.

F2. Edge rate limits [both] — 2 hours

Cloudflare Rate Limiting per IP and per account on upload endpoints, plus Turnstile on the free form. Cheaper at the edge than in the app.

F3. Budget alarms on the rate [both] — 2 hours

AWS Budgets and Anthropic/Bedrock spend alerts at 50/80/100% of plan, wired to Telegram — and alert on the rate of spend, not just the cumulative total, so a scripted abuser trips it in an hour rather than at month end.

F4. Rehearse the kill switch [you] — 30 min

docs/admin/security/anthropic-spend-killswitch.md should be runnable in under a minute. Time yourself.

Track G — Do during the EC2 migration

Risk #7, #13. These are nearly free while you’re already moving things, and expensive to retrofit.

G1. GitHub OIDC instead of static AWS keys [agent] — 4 hours

Deploy via OIDC role assumption. Otherwise you trade “CI runs on prod” for “CI holds the keys to prod,” which is not the win it looks like.

G2. Ephemeral runners on .4 [both] — 2 hours

--ephemeral, workspace wiped per job, runner user unprivileged with no sudo and no SSH keys to .3, .17, or EC2. Never enable fork-PR builds on a self-hosted runner.

G3. LAN firewall and exposure check [both] — 3 hours

  • Default-deny inbound on .3, .4, .17; no port forwards — reach internal services over Tailscale or Cloudflare Access.
  • Every internal service bound to 127.0.0.1 in compose (audit Grafana especially).
  • Add a monthly external nmap of the WAN IP to the nightly audit script, emailing the diff.

Track H — Output safety (1 day)

Risk #2, #20.

H1. Sanitize model-generated HTML [agent] — 4 hours

Allowlist sanitization (DOMPurify or equivalent) before any model-generated fragment is stored, previewed, or delivered. Strip <script>, event handlers, javascript:/data: hrefs, <iframe>. This is the one control that matters for #2.

H2. Sandboxed preview origin [agent] — 3 hours

Serve HTML previews from an origin with no authenticated session and a strict CSP.

H3. SSRF guard on the URL scanner [agent] — 4 hours

Reject RFC1918, loopback, link-local, and IPv6 ULA targets after DNS resolution; block redirects into private ranges; ensure the fetcher has no route to the LAN or to cloud metadata endpoints.

Track I — Paperwork (1 day, and it sells)

Universities will ask for most of this during procurement. Having it ready is a sales asset, not just a control.

  • docs/Policies/incident-response.md — the stolen-laptop, leaked-key, and customer-data-exposure playbooks; notification obligations and the 72-hour clock; who you call.
  • docs/Policies/vendor-inventory.md — each processor, data shared, DPA status, breach-notice SLA and contact.
  • Reviewer confidentiality agreements and FERPA training for anyone doing human review, plus a signed determination from each customer on whether their uploads contain student records (#9).
  • A one-page security summary for procurement, drawn from the above.

Suggested order

If you want a single sequence rather than tracks: A1 → A2 → A3 → B1 → B2 → B3 → E1 → C1 → C3 → E2 → E3 → F3 → then the multi-day work (D1, C4, B4) as it fits, with Track G folded into the EC2 migration.

The first eight items are roughly one working day and remove most of the likelihood-weighted risk.