AI Kill Switch — Runbook

Use this runbook to halt all AI-gated processing during a runaway-cost event, security incident, or any situation requiring immediate shutdown of AI inference.

What this affects: Bedrock PDF conversions. Jobs already queued are parked as paused_budget and held until manually re-enabled. Jobs in flight complete normally. All other conversion paths (non-Bedrock) are unaffected.

How to identify a runaway event

Check these in order — stop when you find the cause.

Email alert — Bedrock budget alerts fire at 80% and 100% of daily/monthly caps to larry@anglin.com. A 🛑 Bedrock PDF conversions PAUSED email means the guardrail already tripped automatically.

Supabase cost ledger — Run in the Supabase SQL editor (project vuvwmfxssjosfphzpzim):

SELECT
  date_trunc('day', created_at) AS day,
  sum(estimated_cost_usd)       AS daily_usd
FROM cost_ledger
WHERE backend = 'bedrock'
  AND created_at > now() - interval '7 days'
GROUP BY 1
ORDER BY 1 DESC;

A vertical spike vs. the prior days is a runaway. (The guardrail reads the same ledger via get_bedrock_pdf_spend() — see bedrock-budget.ts.)

AWS CloudWatch — Bedrock → Usage → Model invocations. A sudden increase in input/output tokens confirms runaway AI calls.
Kill-switch state — Check whether the guardrail has already tripped:
```
GET https://api.gevell.com/api/admin/bedrock/status
Authorization: Bearer <admin token>
```
killSwitch.enabled: false means it is already off. pausedJobs shows how many jobs are held.

Option A — Admin dashboard (primary; requires the API to be running)

Go to admin.gevell.com and sign in with an admin account.
Navigate to AI Controls (sidebar) — this shows the live kill-switch state, the current reason and who last changed it, Bedrock spend vs. caps, and how many jobs are held.
Click Disable AI, enter a required reason in the confirmation dialog, and confirm. The blast-radius warning is shown before you commit: all new AI-gated PDF conversion jobs are held immediately. To reverse, click Re-enable AI and enter a reason.

The page calls the admin API directly. If you prefer curl / need to script it:

Disable (trip the kill-switch):

curl -X POST https://api.gevell.com/api/admin/bedrock/disable \
  -H "Authorization: Bearer <admin token>" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Manual kill — describe the incident here"}'

A reason is required (blank/whitespace returns 400). The action is admin-only and is written to the admin audit trail (admin_audit_log, action bedrock_pdf_disable).

Re-enable (after investigation):

curl -X POST https://api.gevell.com/api/admin/bedrock/resume \
  -H "Authorization: Bearer <admin token>" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Incident resolved — describe what was fixed"}'

This flips the flag on, re-drives all paused_budget jobs to pending, and logs bedrock_pdf_resume to the audit trail.

Option B — Supabase SQL (offline fallback; works when the API is down)

Use the Supabase dashboard at https://supabase.com/dashboard/project/vuvwmfxssjosfphzpzim or the Claude MCP tool.

Disable AI (kill-switch off):

INSERT INTO admin_flags (key, enabled, reason, updated_by, updated_at)
VALUES (
  'bedrock_pdf_conversions_enabled',
  false,
  'Manual kill — <describe incident>',
  'larry@anglin.com',
  now()
)
ON CONFLICT (key) DO UPDATE
  SET enabled    = false,
      reason     = EXCLUDED.reason,
      updated_by = EXCLUDED.updated_by,
      updated_at = EXCLUDED.updated_at;

Verify it took effect:

SELECT key, enabled, reason, updated_by, updated_at
FROM admin_flags
WHERE key = 'bedrock_pdf_conversions_enabled';

enabled = false confirms the kill-switch is active. New Bedrock PDF jobs will park as paused_budget within one dispatch cycle.

Check how many jobs are held:

SELECT status, count(*) FROM large_conversion_jobs WHERE status = 'paused_budget' GROUP BY 1;
SELECT status, count(*) FROM chunk_jobs WHERE status = 'paused_budget' GROUP BY 1;

Re-enable (after investigation):

UPDATE admin_flags
SET enabled    = true,
    reason     = 'break-glass: <describe what was fixed>',
    updated_by = 'larry@anglin.com',
    updated_at = now()
WHERE key = 'bedrock_pdf_conversions_enabled';

-- Re-drive parked jobs
UPDATE chunk_jobs SET status = 'pending' WHERE status = 'paused_budget';
UPDATE large_conversion_jobs SET status = 'pending' WHERE status = 'paused_budget';

Verifying the kill-switch is active

After disabling (either method), confirm new jobs are being held:

curl https://api.gevell.com/api/admin/bedrock/status \
  -H "Authorization: Bearer <admin token>"

Expected response when active:

{
  "killSwitch": { "enabled": false, "reason": "...", "updatedBy": "...", "updatedAt": "..." },
  "pausedJobs": <N>
}

If the API is down, verify via Supabase SQL (see Option B — “Check how many jobs are held”).

Before re-enabling — checklist

Do not re-enable until all of these are resolved:

Root cause identified (runaway script, abuse pattern, misconfiguration, etc.)
Cause is fixed or mitigated (not just paused)
Current MTD spend reviewed — confirm resuming will not immediately re-trip the monthly cap
If abuse-related: offending user/IP blocked or rate-limited
Incident documented (add a note to docs/admin/audits/ or file a GitHub issue)

AWS hard stop (nuclear option)

If both the API and Supabase are unreachable and the runaway is ongoing:

AWS Console → IAM → Users — locate the bedrock-api-user or the role used by the Lambda. Remove the AmazonBedrockFullAccess policy or attach an explicit Deny.
Lambda → Environment variables — set PDF_BEDROCK_ENABLED=false on the ApiHandler Lambda. This is the env-var gate in bedrock-budget.ts:isPdfBedrockEnabled(). New invocations will skip Bedrock immediately without a deploy.

Reverse both changes once the incident is resolved.

Note: PDF_BEDROCK_ENABLED is the master feature gate (Bedrock on/off at all), a separate switch from the admin_flags kill-switch. Setting it false here disables the entire Bedrock path regardless of the kill-switch state; it is the belt-and-suspenders option when the DB path is also unavailable.

AWS Systems Manager Parameter Store — evaluated, NOT wired up

The issue (#3023) asked whether the kill-switch should also live in AWS SSM Parameter Store so it can be flipped from the AWS console when Supabase itself is unreachable. We evaluated this and did not implement it. This section documents how it would work if we adopt it later — it is not currently a live control.

How it would work:

Create a String parameter, e.g. /gevell/bedrock/kill-switch with value enabled / disabled.
Grant the ApiHandler Lambda execution role ssm:GetParameter on that ARN (add to infra/cdk/lib/api-stack.ts).
Have readKillSwitch() in bedrock-budget.ts consult SSM (cached ~30–60s to avoid a GetParameter on every dispatch) and treat disabled as fail-closed, OR run a small scheduled reconciler that mirrors the SSM value into admin_flags.

Why we deferred it:

Redundant fail-safe already exists. The PDF_BEDROCK_ENABLED=false Lambda env var (see the “AWS hard stop” section above) is a console-only, deploy-free master shutoff that does not depend on Supabase. That already covers the “Supabase is down” scenario the SSM idea targets.
Two sources of truth for one flag invites drift/confusion during an incident; the env-var gate keeps the AWS-side control unambiguous.
Adding SSM to the per-dispatch read path adds latency and a new IAM permission for marginal benefit.

If operational experience shows the env-var toggle is too coarse (it kills all Bedrock, not just new jobs), revisit the SSM approach above.

Kill-switch implementation: workers/api/src/services/bedrock-budget.ts
API endpoints: GET /api/admin/bedrock/status, POST /api/admin/bedrock/disable, POST /api/admin/bedrock/resume
Admin dashboard UI: apps/admin/src/app/admin/ai-controls/page.tsx (AI Controls, #3023)
Cost ledger: docs/admin/security/anthropic-spend-killswitch.md
NIST AI RMF MANAGE controls: docs/admin/ai/nist-ai-rmf-alignment.md