flyer-tagger scaling: the assemble cliff

Status (2026-09-25, #3446). The ceiling described below was resolved by routing long documents OFF Lambda entirely rather than by moving assemble to Fargate. See The two-lane resolution at the end. The measurement sections remain accurate and still govern the fast lane; the “Fargate trigger criteria” section is kept for the record but is no longer the plan — Fargate was evaluated and rejected.

The flyer-tagger Lambda tags a PDF for accessibility. A single Lambda invocation is capped at 900 s wall time and a fixed memory size (--memory-size 3008 in deploy/deploy.sh). For most jobs one invocation is enough. Large documents are not — this doc explains how the pipeline scales past that cap, where it can’t, and the concrete criteria for moving the un-scalable step to Fargate.

The chunked pipeline (#2349)

A queue job of >= FLYER_CHUNK_MIN_PAGES pages does not run in one invocation. Instead the SQS tag job becomes a split and fans out:

1. Split (_split_job) — the tag job computes the page count and enqueues one chunkTask message per FLYER_CHUNK_PAGES-page range back onto the same queue, writes the split to the job row, and returns.
2. Chunk analysis (_run_chunk, the map step) — each chunkTask runs on its own Lambda invocation, analyzes only its page range (all the Anthropic / Bedrock vision wall-time), ledgers its own spend, and stores the per-page analyses to chunks/<jobId>/<index>.json. This is the part that scales: analysis time is spread across as many parallel invocations as there are chunks, so adding pages adds invocations, not wall time per invocation. Each chunk also has a band-level checkpoint, so an SQS redelivery resumes mid-chunk instead of re-spending.
3. Assemble (_run_assemble, the reduce step) — the last chunk to land claims assembly exactly once and enqueues a single assembleTask. That one invocation loads every chunk’s analyses, runs the whole deterministic pipeline once over the whole document (process(cached_analyses=...) — one global structure tree, doc-level font passes), runs veraPDF, writes the tagged PDF, and delivers + settles the job row.

Why assemble is the cliff

Analysis scales via fan-out. Assembly does not. It is un-chunked and bounded by the same single-invocation limits:

• 900 s timeout — the whole-document rebuild + veraPDF + PDF write happen in one invocation.
• Lambda memory — the entire document is held and rewritten in memory.
• No assemble-level checkpoint — unlike chunk analysis, assemble has no partial-progress store. On SQS redelivery it re-runs the whole thing.

So for a large enough document, assembly cannot be saved by retries: if one attempt can’t finish in 900 s (or OOMs), neither can the 2nd or 3rd — they redo identical work. After maxReceiveCount (3) the message lands in flyer-tagger-jobs-dlq and the job wedges (the workers/api stale sweep eventually terminal-fails + refunds it). That’s the scaling ceiling.

What the instrumentation measures (#3145)

Every phase emits a CloudWatch Embedded Metric Format (EMF) record to stdout (namespace FlyerTagger/Ceiling, dimensioned on phase). CloudWatch auto-extracts the metrics; the raw JSON line is also greppable. We chose EMF (not the Supabase cost_ledger) because the ledger holds token spend, not wall time or memory, and isn’t a queryable time series — and EMF needs no extra dependency or IAM grant.

Per phase (chunk and, critically, assemble):

Metric	Meaning
PhaseSeconds	wall time for the phase (chunk analysis, or the full assemble)
PhaseCeilingFraction	PhaseSeconds / 900 — how close to the timeout
PhasePeakMemoryMB	peak RSS (/proc/self/status VmHWM) since container start — an upper bound on this invocation, not a per-phase measure (see caveat below)
PhaseMemoryFraction	PhasePeakMemoryMB / memoryLimitMB — how close to the memory ceiling

Log fields on the same record: jobId, pageCount, chunkCount, chunkIndex (chunks only), memoryLimitMB.

When any phase crosses FLYER_CEILING_WARN_FRACTION of the 900 s budget (default 0.70) a near-ceiling WARNING is logged naming the phase, elapsed seconds, and jobId.

DLQ alarm — deploy/deploy.sh provisions the DLQ imperatively (not CDK), so it also upserts a CloudWatch alarm flyer-tagger-dlq-not-empty on the DLQ’s ApproximateNumberOfMessagesVisible > 0 (any message = a job exhausted its 3 attempts). Set FLYER_ALARM_SNS_ARN before deploy to wire notifications.

Fargate trigger criteria (superseded — kept for the record)

Superseded by #3446. Fargate would have meant net-new ECS/IAM/VPC footprint and an unresolved reconciliation with the SQS fan-out model, to provide compute the always-on host already has. The criteria below are still a reasonable read of the DATA; the conclusion they pointed at is not the one taken. Revisit only if volume ever demands per-job isolation and independent autoscaling.

Move the assemble step (only) to Fargate — no timeout, larger configurable memory — while keeping chunk analysis on Lambda (the hybrid), when the collected data shows assemble routinely approaching either ceiling. Concretely, act if either holds over a representative window (say a rolling 30 days of real jobs):

• assemble PhaseSeconds p95 > 70% of 900 s (i.e. p95 > ~630 s), or
• assemble PhasePeakMemoryMB p95 > 80% of the Lambda’s configured memory (i.e. p95 PhaseMemoryFraction > 0.80).

Read the memory metric as an upper bound. PhasePeakMemoryMB comes from VmHWM, the process-lifetime high-water mark, and Lambda reuses execution environments — so on a warm sandbox a small assemble inherits the peak of an earlier, larger job. The metric therefore over-reports on warm containers and is tight only on cold starts. Treat the memory criterion as conservative (it will trip early, never late); when it fires, corroborate with cold-start samples or the platform REPORT Max Memory Used before committing to a Fargate migration. The PhaseSeconds criterion is unaffected and is the more reliable trigger.

Either condition means a meaningful fraction of jobs are one bad document away from a ceiling that retries can’t rescue. A rising flyer-tagger-dlq-not-empty alarm rate is the lagging confirmation.

Chunk analysis is not a Fargate candidate on these criteria: it scales by fan-out and resumes from its checkpoint, so a hot chunk costs at most one re-analyzed range, not a wedged job.

Knobs

Env	Default	Purpose
FLYER_CEILING_WARN_FRACTION	0.70	fraction of 900 s that trips the near-ceiling WARNING
FLYER_LAMBDA_TIMEOUT_S	900	the timeout budget the fractions are computed against (keep in sync with --timeout)
FLYER_ALARM_SNS_ARN	(unset)	SNS topic for the DLQ alarm’s notify + ok actions
FLYER_CHUNK_MIN_PAGES	9	pages at/above which a job splits into chunks (<= 0 disables)
FLYER_CHUNK_PAGES	4	pages per chunk
FLYER_MAX_PAGES	32	workers/api: pages ABOVE which a job routes to the long lane instead of Lambda
FLYER_ABSOLUTE_MAX_PAGES	800	workers/api: rejection ceiling on both lanes, sized for assemble memory

The two-lane resolution (#3446)

The cliff above is real and unfixable on Lambda: the 900 s timeout is a hard AWS limit, so a document whose assemble can’t fit has no retry count that saves it. Raising FLYER_MAX_PAGES only converted a clean DOCUMENT_TOO_LARGE 422 into a silent dead-letter ~45 minutes later.

The resolution was to stop trying to fit long documents into Lambda:

• Fast lane — Lambda, pages <= 32. Unchanged, including the chunked pipeline and every metric above. Covers ~95% of jobs (corpus p95).
• Long lane — the always-on host, pages > 32. The flyer-tagger sidecar (services/flyer-tagger/deploy/server.py, compose service flyer-tagger) runs handler.run_host_job: the same pipeline, with no execution ceiling. Because there is no ceiling, the chunked fan-out is switched OFF (FLYER_CHUNK_MIN_PAGES=0) and map + assemble run straight through in one process — a thread pool over page bands, then one assemble.

The cliff moved, it did not vanish. Assemble is still the phase whose peak memory scales with total page count, and the host is shared with convert and the other sidecars. That is why the long lane is bounded on four axes rather than one:

Bound	Where	Why
One job at a time (FLYER_HOST_MAX_RUNS=1)	sidecar	two concurrent assembles is how the box falls over
deploy.resources.limits memory/cpu	docker-compose.yml	a runaway job can’t consume the whole host
FLYER_HOST_MAP_CONCURRENCY	sidecar	in-flight page rasters are bounded by concurrency, NOT document length — lower this for huge documents instead of growing the box
FLYER_ABSOLUTE_MAX_PAGES	workers/api	assemble memory grows with page count; reject rather than OOM

Sizing the host is therefore an assemble-peak question, not a throughput one: size for the largest expected document (~600–800 pp with headroom) and bound the map phase with concurrency. Those numbers feed the EC2 sizing in #3454.

Long-lane knobs

Env	Default	Where	Purpose
FLYER_TAGGER_LONG_QUEUE_URL	(unset)	API hosts + host	The feature flag. Unset = reject over the cutoff, exactly as before #3446
FLYER_TAGGER_URL	none (set to http://flyer-tagger:5012 in docker-compose.yml)	host	where the long-lane consumer sends work; unset means long jobs are requeued, not run
FLYER_LONG_CONCURRENCY	1	host	in-flight long-lane SQS messages
FLYER_HOST_MAX_RUNS	1	sidecar	simultaneous tagging runs
FLYER_HOST_BAND_PAGES	4	sidecar	pages per map band
FLYER_HOST_MAP_CONCURRENCY	4	sidecar	parallel map bands — the memory knob
FLYER_HOST_HOURLY_USD	0	host	amortized host rate for cost attribution; 0 still records duration
FLYER_HOST_RUN_TTL_S	3600	sidecar	how long a terminal run stays readable via GET /tag/:id
FLYER_HOST_MAX_ATTEMPTS	3	sidecar and API	consecutive attempts without progress that may run a job; the next delivery fails it (#4070). The sweep reads it too, so set it on both containers
FLYER_HOST_COST_CEILING_PER_PAGE_USD	1.00	sidecar	per-job Bedrock ceiling = max(floor, pages × this), across all attempts (#4070)
FLYER_HOST_COST_CEILING_FLOOR_USD	5.00	sidecar	minimum ceiling, for small documents

Operating the long lane

• Is it up? curl http://flyer-tagger:5012/health → activeRuns, maxRuns.
• What is it doing? GET /runs lists every run this process knows about.
• Is the consumer alive? the API’s /health reports the flyer-long lane; a dead receive loop makes the whole payload degraded.
• A job is stuck. The sweep re-drives a stalled Lambda job onto the host lane once (flyer_tagging_jobs.rerouted_at). A job that then stalls on the host is swept and refunded like any other — check the sidecar log and the flyer-tagger-jobs-long-dlq.

Runaway guards (#4070)

A host job that keeps failing to land its terminal row is redelivered by SQS about every 15 minutes, for up to 60 receives. Each redelivery’s processing write refreshes updated_at, so the 240-minute staleness window never sees it. Three guards bound that:

• Attempt cap. flyer_begin_host_attempt (migration 342) claims the row and counts consecutive attempts that made no progress. It increments attempt_count once per run that actually starts, but resets it to 1 when the previous attempt got further through the map than any earlier attempt (the high-water attempt_progress_mark). A long job that survives a few deploys is therefore never mistaken for a loop. A job that keeps stopping at varying points short of its best (584, 600, 588, 600 …) is. A 503 back-off or a 409 reattach is not counted. The delivery after FLYER_HOST_MAX_ATTEMPTS fails the row (guard_tripped = 'attempt-cap') without running, quoting last_attempt_error.
• Cost ceiling. Checked before each map band against the job’s flyer-tagging rows in cost_ledger, across every attempt. Tripping it fails the row with guard_tripped = 'cost-ceiling'.
• Sweep loop arm. Fails a processing host row whose attempt_count is over the cap, or whose time on the host lane (since rerouted_at, else created_at) exceeds 480 + 3 × pages minutes, whatever its updated_at says (guard_tripped = 'redelivery-loop'). The refund path settles the job.

Every tripped guard is logged at error level and sent once as an admin alert (flyer_runaway_guard in /admin/notifications; email and Telegram by default). guard_alerted_at is the exactly-once stamp.

A retry no longer re-buys the analysis. When a delivery is discarded because its row write could not be confirmed, the S3 analysis checkpoint is kept. A retry therefore costs assemble CPU, not Bedrock. The host checkpoint also records which pages have been booked (__ledgered_pages__). A reused page is booked once: never again if an earlier attempt booked it, and on reuse if nobody did, for example a process killed mid-band or a timed-out Lambda attempt before a reroute.

The GCP cost monitors (scripts/cost-spike-monitor.mjs, cost-trickle-monitor.mjs) count Gemini and Vertex requests only. They cannot see Bedrock spend, so they are not a backstop for this lane. The product-wide backstop is the cost_ledger spend monitor (docs/admin/operations/cost-ledger-monitor.md). Its trickle rule fires on this lane’s redelivery-loop shape about 1.5 hours in.