Skip to content
AnglinAI Docs

#1170 — Document-like image OCR routing

Goal

When a JPG/PNG upload is a document (scanned page, worksheet, math problem), produce real OCR’d text + MathML (and, where possible, cropped figures) instead of the current image-passthrough output (one <img> + a large vision alt-text blob). Non-document images (photos, logos, charts-as-art) keep the existing describe-the-whole-image passthrough.

Verified facts (don’t relitigate)

  • Image conversions run on the EC2 batch worker via executePipeline (workers/batch/src/pipeline-executor.ts) → processConversion (workers/api/src/routes/convert.ts). They cannot run on Lambda — the test conversion’s own pipeline timer was 376s vs Lambda’s 300s ceiling.
  • The image-passthrough branch is convert.ts:2808-2843 (the else after the pdf/docx/audio branches).
  • Mathpix creds are available in this runtime. The batch worker path-loads /accessible-pdf/shared/* from SSM into process.env (workers/batch/src/index.ts:45,85-86) and processConversion receives them as apiKeys.MATHPIX_APP_ID / apiKeys.MATHPIX_APP_KEY. No infra work needed.
  • A direct image OCR endpoint already exists: processImage() (workers/api/src/services/mathpix-pdf.ts:499) → Mathpix /v3/text, returns { html, mathml?, latex? }. No PDF-wrapping required for the text/math win.
  • /v3/text does not return cropped figure images; only /v3/pdf → html.zip does (mathpix-chunk-converter.ts). Figure crops are a phase 2.

Design

Branch inside the image else block (convert.ts:2808). Decision order:

if (fileType === image):
    if (hasMathpix && isDocumentLikeImage(...)):
        result = await processImage(imageData, mime, mathpixConfig, {includeMathml:true})
        if (result && isMeaningfulOcr(result)):
            html = wrapMathpixImageHtml(result, h1Text)   # OCR text + MathML
            resolvedBackend = 'image-mathpix-text'
            -> downstream pipeline still runs (MathML validation, alt-text for any
               residual <img>, WCAG fix loop) unchanged
        else:
            fall through to image-passthrough   # photo / empty OCR
    else:
        image-passthrough (current behavior)

“Document-like” gate

Cheapest reliable gate, in order of preference:

  1. Try-and-validate (recommended for phase 1): always call processImage when hasMathpix, then keep the result only if isMeaningfulOcr — e.g. stripped text length ≥ N chars OR ≥1 MathML node. A photo/logo returns little or no text and falls back automatically. One Mathpix call (~$0.004– 0.01) is cheaper and faster than the current ~$0.01 vision + 120s polish, so the speculative call is net-positive even when it falls back.
  2. Optional pre-gate to avoid the Mathpix call on obvious non-docs: a quick heuristic (aspect ratio near a page, or a tiny Gemini-flash classify call). Defer unless Mathpix cost on fallbacks proves material at volume (~20 img/day today → negligible).

Output shaping

wrapMathpixImageHtml(result, h1Text):

  • <h1>{filename-derived}</h1> (keep current heading-one behavior).
  • Insert result.html (Mathpix already returns structured HTML; sanitize via the existing sanitizeText/post-processing sanitizers).
  • MathML from result.mathml flows into the existing LaTeX→MathML validators in runPostProcessing (the image already produced 26 valid MathML tags via the vision path; Mathpix MathML should validate the same way).
  • Any residual <img> Mathpix emits still gets alt text from enhanceImagesInHtml downstream — compose, don’t replace.

Implementation steps

  1. Extract a Mathpix config helper in convert.ts (or reuse the existing one the pdf branch builds) so the image branch can construct { appId, appKey } from apiKeys. Confirm whether the pdf branch already has mathpixConfig in scope to lift.
  2. Add isMeaningfulOcr(result) small pure helper + wrapMathpixImageHtml(result, h1Text) — colocate near the image branch or in a new services/image-ocr.ts. Unit-testable in isolation.
  3. Branch in convert.ts:2808 per the design above. Keep TIFF→PNG sharp conversion (2820-2829) ahead of the Mathpix call so unsupported formats are normalized first (Mathpix /v3/text wants png/jpeg/webp/gif data URI).
  4. Set metadata.resolvedBackend to image-mathpix-text vs image-passthrough and push a pipeline_log step (step: 'image-ocr', detail, estimatedCostUsd) so cost-tracking + the file-detail view reflect the path taken (matches the cost-tracking standard).
  5. Cost tracking: record the Mathpix image call cost (per-image price) the same way chunk-processor records mathpix OK ... cost=$.
  6. Progress feedback (covers the issue’s secondary item): emit updateProgress(..., 'Reading text from image') before the Mathpix call so the UI isn’t stuck on a generic label.

Tests

  • workers/api/src/__tests__/services/image-ocr.test.ts:
    • isMeaningfulOcr — empty/whitespace/short → false; real text or MathML → true.
    • wrapMathpixImageHtml — emits one <h1>, embeds OCR html, preserves MathML.
  • convert.ts image-branch tests (mock processImage):
    • Mathpix returns meaningful OCR → backend image-mathpix-text, html contains MathML.
    • Mathpix returns null/empty → falls back to image-passthrough, no throw.
    • hasMathpix=false → straight passthrough (unchanged behavior).
  • Keep coverage ≥ 80%; add the new file to the route/service test mirror.

Phase 2 (separate PR, only if figure crops are needed)

  • For document images with embedded diagrams, wrap the (PNG/JPEG) image into a 1-page PDF (pdf-lib embedJpg/embedPng + drawImage, both available on the batch worker) and run it through the existing /v3/pdf → html.zip path (mathpix-chunk-converter.ts) to get tightly-cropped figures.
  • Gate on “image contains a figure region” to avoid the heavier /v3/pdf call for pure text/math images.

Out of scope / follow-ups

  • Adding MATHPIX_APP_ID/KEY to the rotation script’s SSM_SECRETS list (hygiene — rotation currently won’t update the SSM copy). File separately.
  • Marker chart_understanding,infographic extras for the passthrough path (issue option 3) — independent lever.

Code review resolution (high-effort multi-agent review)

  • #1 WCAG regression (fixed): OCR output is rejected (→ passthrough) when it contains an <img> without meaningful alt text (htmlHasImageWithoutAlt), so the OCR path can never ship an un-described image. Figures stay on passthrough until phase 2.
  • #2 cost double-count (fixed): the image-ocr pipeline_log step no longer carries estimatedCostUsd; the conversion step owns the Mathpix cost.
  • #3 passthrough XSS (fixed): the passthrough <h1> now escapes the filename via the canonical escapeHtml.
  • #4 classify resilience (fixed): the Gemini classify call is wrapped in callWithRetry.
  • #5 altitude — two Mathpix integrations (partial): shared a mathpixConfigFromApiKeys() helper so the image path builds Mathpix config identically to the PDF paths (deduped 5 sites). Full convergence — routing document images through the batch pipeline’s scanned-document mechanism — is phase 2 (wrap → /v3/pdf), where the figure-crop machinery is shared too. The phase-1 divergence is deliberate: images have no PDF operator stream for the heuristic detector and no PDF for /v3/pdf.
  • #6 efficiency — sequential classify + OCR (deliberately deferred): the classify call gates the Mathpix spend and, unlike isMeaningfulOcr, distinguishes a document from a photo that happens to contain text (which must keep the describe-the-whole-image passthrough). Reordering or parallelizing would either lose that distinction or spend the Mathpix call on photos. A future micro-opt is to downscale the image before the classify call; not worth the sharp coupling + test burden at current volume (~20 img/day).
  • #7 reuse (fixed): escapeHtml and stripHtmlToText now come from utils/html (the latter promoted there and shared with reading-order-verifier). withTimeout left local — no canonical shared util exists and extracting one would expand scope into image-enhancer.
  • #8 simplification (fixed): the kill-switch is now (... ?? process.env.X) !== 'false' with a default-ON comment.

Risk / rollback

  • Single, well-contained branch; hasMathpix=false and the isMeaningfulOcr fallback both preserve current behavior, so the blast radius is document-like images only.
  • Add an env/option kill-switch (e.g. IMAGE_OCR_VIA_MATHPIX=false) to revert routing without a redeploy, mirroring the MARKER_USE_CONVERT_V2 pattern.