GCP Budget Killswitch

A Cloud Function that automatically detaches billing from the gemini-theaccessible-org project when monthly Gemini spend reaches 100% of the budget. This is a true hard cap: once billing is detached, all Gemini API calls fail.

The project is dedicated to Gemini (only generativelanguage.googleapis.com is enabled), so detaching billing has no collateral blast radius — it stops Gemini and nothing else.

Components

Resource	Where
Project	`gemini-theaccessible-org`
Billing account	`0158C6-12C170-5AB1F4`
Budget	`gemini-theaccessible-org monthly $500` — billingAccounts/0158C6-12C170-5AB1F4/budgets/cf57b88e-0918-419a-b6c0-b843b0e257d1
Pub/Sub topic	`projects/gemini-theaccessible-org/topics/budget-alerts`
Cloud Function	`budget-killswitch` (us-central1, gen2, Node 22)
Service account	`budget-killswitch-fn@gemini-theaccessible-org.iam.gserviceaccount.com`
Function source	`infra/gcp/budget-killswitch/` in this repo

Service-account IAM (all required)

Role	Scope	Why
`roles/billing.projectManager`	project `gemini-theaccessible-org`	Detach billing via `updateBillingInfo`
`roles/secretmanager.secretAccessor`	secrets `RESEND_API_KEY`, `TELEGRAM_BOT_TOKEN`, `TELEGRAM_CHAT_ID`	Read alert credentials at runtime
`roles/run.invoker`	Cloud Run service `budget-killswitch`	Eventarc trigger invokes the underlying Cloud Run service
`roles/eventarc.eventReceiver`	project	Receive Pub/Sub events via Eventarc

The last two bindings are NOT auto-created by gcloud functions deploy --gen2 --trigger-topic=... when you bring your own runtime service account. Without them, Pub/Sub messages arrive but the trigger logs IAM principal lacks {run.routes.invoke} permission and the function never runs.

Secrets (Secret Manager)

Stored in project gemini-theaccessible-org, sourced from AWS SSM /accessible-pdf/shared/*:

RESEND_API_KEY — sender domain must be a Resend-verified domain (currently theaccessible.org; the function uses noreply@theaccessible.org)
TELEGRAM_BOT_TOKEN
TELEGRAM_CHAT_ID

Trip thresholds

Threshold	Action
50% of budget (actual spend)	Email to `larry@anglin.com` — informational
90% of budget (actual spend)	Email + Telegram — warning, kill at 100%
100% of budget (actual spend)	Detach billing + email + Telegram page
Forecast ≥ 200% of budget AND ≥ 5 days into period	Detach billing + email + Telegram (catches fast-burn mid-month)
Forecast ≥ 200% of budget but < 5 days into period	Email + Telegram — trip suppressed; forecasts are noisy on a young period
Forecast trigger below 200%	No action (the GCP-configured threshold of 100% forecast just generates a notification, not a kill)

The forecast-trip behavior is governed by two env vars on the Cloud Function:

FORECAST_TRIP_RATIO (default 2.0) — forecast multiple required to detach
FORECAST_TRIP_MIN_DAYS_INTO_PERIOD (default 5) — earliest day in the budget period when a forecast trip can fire

The budget itself must have a “Forecasted spend” threshold rule configured for any forecast event to reach the function. The current monthly budget has one at 100% forecast — the function sees those events but only detaches when the forecast multiple ≥ 2.0 and the period is mature.

When the kill-switch fires

You will receive:

🚨 Telegram message: “Gemini KILL-SWITCH FIRED — billing DETACHED”
Email subject: ”🚨 [gemini] KILL-SWITCH FIRED — billing detached from gemini-theaccessible-org”

All Gemini-dependent services will start failing with 403 BILLING_DISABLED. Affected services (as of writing):

accessible-pdf-api-production Cloudflare worker (PDF conversion)
org-chart-api Cloudflare worker (vision extraction)
vpat-parse Cloudflare worker (VPAT parsing)
accessible-remediate-production Cloudflare worker (alt-text / simplify)
accessible-pdf-production-api Lambda (Gemini batch routes)
Node sidecar containers on 10.1.1.4 and 10.1.1.17

Diagnosis: was this a real breach or a false positive?

Look at the spend graph: https://console.cloud.google.com/billing/0158C6-12C170-5AB1F4/reports;projects=gemini-theaccessible-org
Does the trend match expected traffic, or is there a step-function jump in the last 24h?
Check the cost-spike-monitor table in Supabase: SELECT * FROM cost_monitor_observations WHERE provider='gemini' ORDER BY observed_at DESC LIMIT 100
Check whether the API key was leaked (search GitHub, recent commits, public logs).

If a leak is suspected: rotate the Gemini key BEFORE re-attaching billing (otherwise the leak resumes burning credit the moment you re-enable).

Re-attach procedure (manual — required by policy)

Re-attaching is a two-step authorization gate: you must (a) confirm the trip was legitimate, and (b) explicitly re-link billing.

# 1. Confirm current state
gcloud billing projects describe gemini-theaccessible-org
# Expect: billingEnabled: false

# 2. Re-link billing
gcloud billing projects link gemini-theaccessible-org \
  --billing-account=0158C6-12C170-5AB1F4

# 3. Verify
gcloud billing projects describe gemini-theaccessible-org
# Expect: billingEnabled: true

# 4. Smoke test
curl -sS -X POST \
  "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"ok"}]}]}'
# Expect: HTTP 200 + a candidates response

Cloudflare Workers and Lambdas pick up the restored billing within seconds — no redeploy needed. Node sidecar containers also recover automatically (the key didn’t change; only the billing link did).

Post-incident checklist

Spend graph reviewed and trip root cause identified
If a leak: API key rotated to all 8 surfaces (see docs/admin/secrets-inventory-and-rotation.md)
Billing re-attached
Smoke test against generativelanguage.googleapis.com returns 200
One end-to-end smoke against each consumer worker confirmed healthy
Incident report appended to docs/admin/incidents/
Budget threshold reviewed — bump if the trip was a legitimate growth event

Testing the kill-switch (controlled trip)

Don’t do this in production unless you mean it. The procedure:

# 1. Snapshot the current budget config
gcloud alpha billing budgets list --billing-account=0158C6-12C170-5AB1F4 \
  --filter="displayName:gemini" --format=json > /tmp/budget-snapshot.json

# 2. Edit the budget down to $0.01 (so any spend trips 100%)
#    Use the Cloud Console for safety, OR the patch API.

# 3. Make a single Gemini call to incur a few cents of spend.
#    Wait up to 24h for billing to register (Google's billing data is delayed).

# 4. Confirm the function fired:
gcloud logging read \
  'resource.type="cloud_function" resource.labels.function_name="budget-killswitch"' \
  --project=gemini-theaccessible-org --limit=10

# 5. Confirm billing detached:
gcloud billing projects describe gemini-theaccessible-org
# Expect: billingEnabled: false

# 6. Re-attach (see "Re-attach procedure" above)

# 7. Restore the real budget from /tmp/budget-snapshot.json

Practical alternative for a fast test: synthesize a fake Pub/Sub message to the function with costAmount ≥ budgetAmount. This exercises the alert-only paths (50/90%) without touching real billing. For the 100% path you need a real-money trip, or a temporary IAM binding swap that lets the function dry-run without calling the billing API.

Function logs

gcloud logging read \
  'resource.type="cloud_function" resource.labels.function_name="budget-killswitch"' \
  --project=gemini-theaccessible-org \
  --format="value(timestamp,severity,textPayload)" \
  --limit=50

scripts/cost-spike-monitor.mjs — hourly observation (Phase 0) and faster-than-monthly alerts (Phase 1+)
docs/admin/secrets-inventory-and-rotation.md — where Gemini keys live + rotation procedure
.github/workflows/rotate-secrets.yml — automated key rotation (covers GEMINI_API_KEY across all consumers)