Bedrock batch lane for large and bulk conversions (#3807)

Large conversions (600+ pages by default) and bulk-program jobs do not need an answer in minutes. The batch lane precomputes their per-page hi-fi conversion calls with Bedrock batch inference, which bills at 50% of on-demand, and then runs the normal chunk pipeline, which reuses those results instead of calling the model.

The lane is off by default (BATCH_CONVERSION_ENABLED unset). With the flag off, nothing changes: jobs are created pending, and the driver does not start.

What gets batched, and what does not

Only one call is batched: the smart cascade’s hi-fi single pass (convertPageWithVision, tier sonnet-hifi, Claude Sonnet 4.6 on Bedrock via the shim). It is one request per page, and its input exists before the job starts, so it fits batch inference.

Everything else stays on-demand at full price:

• the coverage-retry re-ask (its prompt depends on the first answer);
• the quality escalation and availability failover to Sonnet agentic vision (an iterative render-and-compare loop, which cannot be batched, see #3367);
• non-hi-fi tiers (Marker, Mathpix, TOC) and every job that bypasses the cascade (parser=opus-*, whole-chunk Mathpix).

This matches the #3367 rule that no batch job may depend on another batch job’s output: there is exactly one batch stage.

How it works

job created (convert-executor / convert.ts)
  └─ eligible? → status waiting_batch (job + chunks)       else pending
BatchConversionDriver (EC2 anchor, leader only, 60 s tick)
  1. claim: oldest parked job with no batch row (next_unclaimed_batch_job),
     insert conversion_batch_jobs row (unique per job); stretch the job's
     funds hold (refresh_hold) over timeout + grace + 90 min
  2. prepare: for each chunk, split pages like the chunk processor, and run
     the REAL convertPageWithVision with a capture transport in the Bedrock
     shim → one Converse record per non-blank page (cachePoints removed)
  3. upload JSONL shards + manifest to S3, CreateModelInvocationJob
  4. poll GetModelInvocationJob every 5 min
  5. Completed / PartiallyCompleted → parse output, write one results file
     per chunk, write the batch cost_ledger row, resume the job with
     options._batchPrimed
     Failed / Stopped / Expired / overdue / prepare error / too few records /
     collect failing 5 times or past the deadline
     → bill any partial output, resume the job on-demand (no _batchPrimed)
  6. resume = resume_batch_conversion_job: job + chunks to pending in one
     transaction; funds hold back to the 90-min on-demand window
  7. a prepare stuck > 2 h: look the Bedrock job up by name and adopt it if
     it exists (it is billing), else resume on-demand
chunk-processor (per chunk)
  └─ _batchPrimed? load results/chunk-N.json, swap in a convertPageWithVision
     whose shim transport answers from the batch result when the request's
     model + prompt hash match; anything else calls Bedrock as usual

Code: workers/api/src/services/batch-conversion/. Hooks in existing files are one call each: initialConversionJobStatus at job creation, loadBatchPrimedDeps + subtractBatchHits in chunk-processor.ts, and startBatchConversionDriver in server.ts.

Why the records cannot drift from the on-demand prompt

The lane never builds a prompt. It runs the on-demand code and captures the translated Converse request at the shim (BedrockShimConfig.transport). At run time the same code runs again, and a batch result is only used when the new request has the same model and prompt hash (system, text, token limit, temperature). A page whose prompt changed between prepare and run (a prompt edit, a different content type, a different model such as a future Haiku route) just runs on-demand. A mismatch costs money, never correctness.

The prompt hash deliberately excludes the page PDF bytes: pdf-lib stamps new creation and modification dates each time a page is split out. The page is pinned by the record id (c<chunkIndex>-p<page>) instead.

Which results are accepted

A batch record is used only when the on-demand path would have accepted the same response: non-empty text, and no guardrail_intervened, content_filtered or max_tokens stop. Rejected, errored and missing records fall back to on-demand for that page. The driver logs a warn when the fallback share is over 20%.

Cost accounting

• One cost_ledger row per batch job, written when the batch row closes: the tokens AWS billed for every returned record (used or not), priced at estimateLlmCost × 0.5, with metadata.batch = true, metadata.batchMultiplier = 0.5, the batch job ARN and the accepted/fallback counts. backend stays chunked-vision. Jobs that end Stopped / Expired / Failed (including ones the driver stops when overdue) and collects that give up after 5 attempts are billed from whatever partial output exists (metadata.endState). The row is written only by the driver that wins the guarded submitted -> closed transition, so it is written at most once even with two leaders.
• large_conversion_jobs.total_cost_usd = chunk costs (batch-served pages excluded) + the batch job’s cost (options._batchCostUsd, stamped on resume), so it matches the ledger.
• Chunk rows (source: chunk-scheduler) have batch-served pages subtracted, so nothing is counted twice. On-demand pages, retries and escalations stay in the chunk rows at full price.
• The saving on a job is roughly the batch row’s cost (on-demand would have cost twice that), less the batch cost of records that were billed but not used. Pages that fell back cost full price in the chunk rows.

Flags (anchor environment)

Variable	Default	Meaning
BATCH_CONVERSION_ENABLED	unset (off)	Master switch. Read at job creation and at driver start.
BATCH_CONVERSION_S3_BUCKET	none	CDK output BucketName. Lane stays off without it.
BATCH_CONVERSION_ROLE_ARN	none	CDK output ServiceRoleArn. Lane stays off without it.
BATCH_CONVERSION_MIN_PAGES	600	Page threshold for non-bulk jobs.
BATCH_CONVERSION_MIN_RECORDS	100	Skip (run on-demand) below this many records. Set to the account quota.
BATCH_CONVERSION_MAX_RECORDS	50000	Skip above this many records. Set to the account quota.
BATCH_CONVERSION_TIMEOUT_HOURS	24	timeoutDurationInHours, clamped to 24–168. The driver gives up 2 h after it.
BATCH_CONVERSION_MAX_FILE_MB	200	Input JSONL shard size (AWS cap is 1 GB per file). Jobs over ~4.5 GB are skipped (cap 5 GB).
BATCH_CONVERSION_POLL_SECONDS	300	Poll interval for submitted jobs.

Eligibility: options.highFidelity === true, parser auto or unset, no Mathpix chunk routing, and either totalPages >= BATCH_CONVERSION_MIN_PAGES or options._batchLane === true (the bulk opt-in). Urgent bulk items must never set _batchLane (#3366).

Jobs created by the Lambda API do not see these variables unless they are also added to the Lambda environment; large conversions are created on the anchor (SQS convert → convert-executor), which is where the flag belongs.

Quotas and facts to check before enabling

These are account-level and not stated inline in the AWS docs. Check them in Service Quotas for account 696081601609, region us-east-1, service Amazon Bedrock:

1. Minimum and maximum records per batch inference job for the Claude Sonnet 4.6 inference profile. Set BATCH_CONVERSION_MIN_RECORDS / _MAX_RECORDS to match.
2. Batch input file size (1 GB) and job size (5 GB).
3. Concurrent batch inference jobs per model (in-progress + submitted).
4. That CreateModelInvocationJob with modelInvocationType: 'Converse' accepts Converse-format modelInput including document blocks. The field itself exists (@aws-sdk/client-bedrock 3.1141.0 types it as 'InvokeModel' | 'Converse'); document block support in batch records is what still needs a real test job. If it does not work, the lane needs an InvokeModel-format record builder; submit fails and every job runs on-demand, which is safe but saves nothing.
5. Whether a single record has a size limit below our largest page PDFs (image-heavy pages can be several MB).

Infrastructure (CDK, not deployed)

infra/cdk/lib/batch-conversion.ts, instantiated in AnchorStack:

• S3 bucket accessible-pdf-production-conversion-batch-<account>: SSE-S3, public access blocked, TLS only, objects expire after 14 days (they contain customer page PDFs; 14 days outlasts the longest wait, 168 h timeout + grace + collect retries, so a manifest can’t expire before collection).
• Service role accessible-pdf-production-conversion-batch-bedrock, assumable only by bedrock.amazonaws.com for this account’s model-invocation-job/*; S3 read/write on conversion-batch/* and bedrock:InvokeModel on the Claude models and us.anthropic.* profiles.
• Managed policy accessible-pdf-production-conversion-batch-caller (create/get/stop/list jobs, tag, iam:PassRole on the service role, S3 on the prefix), attached to the anchor instance role.
• EventBridge rule on aws.bedrock / Batch Inference Job State Change into queue accessible-pdf-production-conversion-batch-events. The driver polls today; the queue is for a future push consumer.

Credentials caveat: until the API reads the instance role (#3648), it signs AWS calls with the .4 IAM user’s static keys. Attach the caller policy to that user as well, or the driver’s calls are denied.

Rollout

1. Check the quotas and facts above; record the numbers here.
2. Apply migration 20260927040000_335_conversion_batch_lane.sql (waiting_batch status + conversion_batch_jobs).
3. cdk deploy AccessiblePdfProd-Anchor; attach the caller policy to the .4 IAM user while static keys are in use.
4. Rebuild the anchor so the new conversion code is live.
5. Set BATCH_CONVERSION_S3_BUCKET and BATCH_CONVERSION_ROLE_ARN in the anchor env, plus BATCH_CONVERSION_MIN_PAGES low enough to catch one test document, then BATCH_CONVERSION_ENABLED=true. Restart.
6. Convert one test document. Watch the batch-conversion logs: batch job submitted → batch job collected (accepted vs fallback) → chunk logs batch lane served N/M primed page(s). Check the conversion_batch_jobs row and the batch: true ledger row.
7. Compare the output against an on-demand conversion of the same file.
8. Restore BATCH_CONVERSION_MIN_PAGES=600. Watch the fallback share and the realised saving for a week.

Rollback: unset BATCH_CONVERSION_ENABLED and restart. New jobs are created pending. Jobs already in waiting_batch stay parked until the driver runs again; to release them without the driver:

UPDATE large_conversion_jobs SET status = 'pending' WHERE status = 'waiting_batch';
UPDATE chunk_jobs SET status = 'pending' WHERE status = 'waiting_batch';

Cancel and Process now (#3823)

Users can stop a conversion (keeping the file) or move a batch-waiting one to on-demand from the dashboard. Routes (owner-only, on every API entry point): POST /api/v1/files/:fileId/cancel and POST /api/v1/files/:fileId/process-now. Code: workers/api/src/services/conversion-control.ts; SQL: migration 20260927060000_337_conversion_cancel.sql (adds the cancelled status; must be applied before the routes can cancel anything).

Cancel, Convert pipeline. cancel_conversion_job moves the job and every chunk that is not completed to cancelled in one transaction, only from pending, processing, paused_budget or waiting_batch. A job already assembling is left alone (409 CONVERSION_FINISHING). Then the API releases the funds hold (release_hold, outcome cancelled), stops a submitted Bedrock batch job if it can, and resets the file to uploaded so Convert is offered again. Nothing is charged: settlement only runs after assembly, and the scheduler never assembles a cancelled job.

• Unstarted chunks are never claimed (the scheduler only claims pending).
• A chunk already running stops within about one heartbeat (CHUNK_HEARTBEAT_INTERVAL_MS) plus the pages in flight: the heartbeat UPDATE is guarded on status = 'processing' and this worker, and when it matches no row it aborts the chunk’s AbortController. The smart cascade checks the signal before each page (like the cost cap), so it starts no new page and returns a truncated result. That partial spend is booked on the cost ledger, and the result is discarded: never written, counted toward done_chunks, assembled, or charged. The opus-harness and whole-chunk Mathpix paths are a single call with no page seam and run to the end. (The same abort now also stops a chunk that a reclaim handed to another worker, instead of both finishing it.)
• Chunk progress writes are a guarded UPDATE on files still converting, so a running chunk cannot put a cancelled file back into chunked_processing.
• The SSE stream (/api/convert/:fileId/stream) sends a cancelled event and closes.

Cancel, Preserve Formatting. cancelFlyerJobsForFile (the delete path) marks the flyer job failed with “Cancelled — you stopped this conversion.” and refunds it. A flyer Lambda still running loses its guarded claim, so the cancel stands.

Process now. For a waiting_batch job, make_conversion_job_interactive closes out the batch row and calls resume_batch_conversion_job (job + chunks to pending, one transaction). The funds hold goes back to the 90-minute on-demand window and the file’s phase to “Queued: 0/N chunks”. Any other state answers 409 NOT_BATCH_WAITING.

If the batch row is already closed (collected, skipped, failed, expired) while the job still waits, the driver is between its close and its resume, or its resume failed (the case stranded_batch_conversion_jobs exists for). Process now then does not resume the job: an empty patch would drop _batchPrimed (paid batch output) and _batchCostUsd. The route still answers 200 and the driver’s own resume or resumeStranded (next tick) resumes it with both. If the driver is not running, release it with the rollback SQL below.

Comped reflows. Cancelling a conversion whose options carry _compedFlyerJobId (the free reflow of a failed Preserve job) clears comp_conversion_id on that flyer job, with the same guard as the reflow route’s own claim rollback, so the user can choose again instead of getting 409 ALREADY_RESOLVED or paying for a manual re-Convert.

The batch row (close_conversion_batch_row, shared by both):

Row state when the user acts	What happens
none (driver has not claimed the job)	a closed skipped row is inserted with the reason, so the driver never claims it
preparing	reason set; the driver finishes preparing and submits, then stops the job on its first poll
submitted	reason set; the API stops the Bedrock job now (best-effort) and the driver stops it on its next poll if that failed
closed	nothing

A stopped batch job stays submitted until Bedrock reports Stopped; the driver then closes it as usual, billing whatever partial output it produced (cost_ledger batch row). Reasons: Cancelled by user, user chose interactive, or Cancelled: the source file was deleted.

Delete of a file with a live Convert job runs the same cancel first (the file row is not reset, it is about to go). Preserve already did this.

Dashboard. GET /api/files sets batchWaiting: true on files whose job is waiting_batch (read from the job, never stored on the file, because the driver resumes jobs without touching file rows). Rows show “Waiting for batch” with a Process now button, and every running conversion has Cancel (with a confirmation dialog). After a cancel, focus moves to the row’s Convert button. Both endpoints are in the OpenAPI spec (workers/api/src/openapi/spec.ts, snapshotted to apps/web/public/openapi.yaml at build).

Admin. cancelled jobs show in the jobs list with their own filter. They are terminal everywhere: orphan recovery, the stuck-file sweep and stranded_batch_conversion_jobs ignore them.

Not interruptible: the inline DOCX/image path (file status processing, no chunk job) and a job already assembling. Both answer 409 with a message that says what to do.

Known limits

• Turnaround: a batch-lane job waits for the batch job, usually hours, at most timeout + 2 h, before its chunks run. Only jobs that can wait belong here.
• One model per batch job. Pages that a future router sends to another model (e.g. Haiku for simple pages, #3806) run on-demand until the lane builds one job per model.
• The driver prepares one job per tick on the leader, in-process; a 1,000-page job’s preparation (split, classify, extract) takes minutes of CPU on the anchor.
• The driver’s S3 and Bedrock clients take the AWS credentials present at startup. That is fine for static keys; with rotating instance-role credentials it should resolve them per tick.