cost_ledger spend monitor

A product-wide backstop on AI spend (#4094). It reads cost_ledger directly, so it covers every product that books spend there. All LLM spend is Claude via Bedrock now, including code paths whose model names still say gemini-*.

Why it exists

On 2026-10-01/02, two flyer-tagger host-lane jobs re-billed Bedrock every ~15 minutes for 15-19 hours. Job 03657749 (8 pages) spent $23.15 across 121 ledger rows, and job d95ae110 (4 pages) spent $7.67. Nothing alerted:

The GCP monitors (scripts/cost-spike-monitor.mjs, scripts/cost-trickle-monitor.mjs) count Google request metrics. They cannot see Bedrock.
Their trickle floor (200 requests per 5 minutes) would miss 2 calls every 15 minutes anyway.
The flyer lane now has its own per-job guards (#4073: attempt cap, cost ceiling, flyer_runaway_guard alert). This monitor catches the same failure shape in every other product, and in the flyer lane too if those guards miss it.

Where it runs

Code: workers/api/src/services/cost-ledger-monitor.ts handles fetching, dedup and alerting. The rules are pure functions in cost-ledger-monitor-rules.ts, and the thresholds are in cost-ledger-monitor-config.ts.
Schedule: every 10 minutes on the EC2 API host. It runs on the scheduler leader only (SCHEDULER_ROLE), with the first run 2 minutes after boot. The timer is in server.ts, next to the CRM reminder sweeps. It is not on Lambda (the Lambda has no resident process to tick).
Data: the RPCs cost_ledger_monitor_rows and cost_ledger_hourly_spend (migration 344). Both read only paid rows (estimated_cost_usd > 0). Both skip backdated reconciliation rows (metadata key backdated_reason, backdated_from, backdate_reason or backfill), because those are bookkeeping, not live spend.
Entity: an entity is operation_type plus the first id found in this order: metadata.jobId, file_id, metadata.conversionJobId, metadata.runId, metadata.chartId, metadata.itemId. Rows with none of these count toward the spend-rate rule only.

The rules

Rule	Fires when	Default
runaway	One entity writes at least N rows, or spends at least $X, within the window	300 rows or $50 in 6 h
trickle	One entity spends in at least K separate bursts spread over at least H hours, within the lookback. A burst is a group of rows less than 5 minutes apart	5 bursts over at least 1 h in 24 h, and at least $0.25
spend-rate	Spend in the last 1 h or 24 h, product-wide (`*`) and per `operation_type`, is above max(3 x the 14-day p95 of the same window, a floor)	Floors: product-wide $40 for 1 h and $100 for 24 h; per operation $35 for 1 h and $80 for 24 h

The trickle rule is what catches redelivery loops. A large legitimate job spends in one continuous burst, however long and expensive it is. A loop spends a little, stops, and comes back. The replayed 03657749 loop reaches its fifth burst about 93 minutes after its first row, so the trickle rule fires on the next run. The unit tests check this, and they check that a 600-page, $30 job fires nothing.

Where the defaults come from

The defaults come from prod cost_ledger, 2026-08-04 to 2026-10-03 (60 days), paid rows only, with backdated rows excluded:

Measure	Legitimate max	The flyer loops
Rows per entity	63	121, 61
Spend per entity	$4.39	$23.15, $7.67
Separate bursts per entity	3 (a conversion re-run the next day)	61, 61
Product-wide spend in 1 h	p95 $1.18, max $15.47
Product-wide spend in a rolling 24 h	p95 $15.17, max $38.63
Busiest `operation_type` hour	conversion, $15.47

Run against those 60 days, the defaults would have fired only on the two flyer loops (trickle). The spend-rate floors sit above every hour and day in the sample, and above one legitimate $30 job, so a single big job never trips them on its own.

Alerts

Channel: dispatchAdminAlert, alert type cost_ledger_anomaly (email and Telegram by default; migration 344 seeds the row on /admin/notifications).
Content: the rule and the threshold it crossed, the operation, the entity id, the user id(s), the row count, the spend, the first and last row timestamps and the span between them, and the burst count (trickle) or the 14-day p95 (spend-rate). Entity alerts also include a SQL lookup for the entity’s ledger rows. All alerts link to /admin/costs.
Dedup: cost_ledger_monitor_alerts holds one row per alert episode. The insert is the claim, and a unique partial index allows one open episode per rule and entity. While a condition holds, its episode stays open and nothing is re-sent. Once the condition has been absent for 60 minutes, the episode is cleared, and a later recurrence alerts again. If an alert reaches nobody (failed or unconfigured), its claim is deleted and the next run retries.
Self-check: every run logs cost monitor run at info, with row, finding, alerted, still-open and cleared counts. Each new finding is logged at warn. A failed run (RPC missing, DB error) logs cost monitor run failed at error and sends nothing. The run never throws. If the info line stops appearing in /admin/logs, the monitor is not running: check the leader role and the Node process.

How to respond

runaway or trickle: run the SQL lookup from the alert. Then check the product’s job table for that id (for example flyer_tagging_jobs, files, or bulk_remediation_items).
- Rows arriving every ~15 minutes, or on a regular beat, look like SQS redelivery. Find why the job never writes a terminal row (check the product’s logs for that job id), then fail or cancel the job so the messages stop. Purge or redrive the message if needed.
- If the entity is a legitimate customer job that is still progressing (for example a very long document resumed after deploys), no action is needed. Raise the threshold if it recurs.
spend-rate: open /admin/costs and break the window down by operation and user. Look for one user or API key driving it, or a new code path calling the model per request. The daily AI cost cap (cost_cap_reached) is the hard stop. This alert is the early warning.
Write down what it was in the alert thread or an issue, so thresholds can be tuned from real cases.

Knobs (env on the EC2 API host)

Every knob is optional. An unset, non-numeric or negative value keeps the default. The values are read on every run, but process.env changes need a container recreate.

Env var	Default
`COST_MONITOR_ENABLED`	`true` (set to `false` to stop the monitor)
`COST_MONITOR_RUNAWAY_WINDOW_HOURS`	6
`COST_MONITOR_RUNAWAY_MAX_ROWS`	300
`COST_MONITOR_RUNAWAY_MAX_USD`	50
`COST_MONITOR_TRICKLE_LOOKBACK_HOURS`	24
`COST_MONITOR_TRICKLE_MIN_BURSTS`	5
`COST_MONITOR_TRICKLE_BURST_GAP_MINUTES`	5
`COST_MONITOR_TRICKLE_MIN_SPAN_HOURS`	1
`COST_MONITOR_TRICKLE_MIN_USD`	0.25
`COST_MONITOR_RATE_MULTIPLIER`	3
`COST_MONITOR_RATE_BASELINE_DAYS`	14
`COST_MONITOR_RATE_FLOOR_1H_USD`	40
`COST_MONITOR_RATE_FLOOR_24H_USD`	100
`COST_MONITOR_RATE_OP_FLOOR_1H_USD`	35
`COST_MONITOR_RATE_OP_FLOOR_24H_USD`	80
`COST_MONITOR_CLEAR_AFTER_MINUTES`	60
`COST_MONITOR_IGNORE_OPS`	empty (comma-separated `operation_type`s to skip)

How to mute

All alerts, keep monitoring: turn off cost_ledger_anomaly on /admin/notifications. Findings are still logged and recorded, with delivery = 'disabled'. Turning the alert type back on does not replay findings that are already open.
One known entity: do nothing. Its episode is already open, so it will not re-alert until the condition has cleared for an hour.
One noisy operation type: add it to COST_MONITOR_IGNORE_OPS.
Everything: COST_MONITOR_ENABLED=false. Each run still logs that it is disabled.

To see what fired:

select rule, entity_key, opened_at, last_seen_at, cleared_at, delivery, details
  from cost_ledger_monitor_alerts order by opened_at desc limit 50;

workers/api/src/services/flyer-stale-sweep.ts: the flyer lane’s own runaway guards (#4073).
workers/api/src/services/bedrock-divergence.ts: the hourly check that the ledger matches CloudWatch’s Bedrock token counts. It catches unmetered calls, which this monitor cannot see.
workers/cost-monitors/README.md: the GCP request-count monitors, which do not cover Bedrock.