Tier-1 Deterministic PDF Remediation API

PDF in → same PDF out, with every deterministic accessibility defect we can fix repaired in place, plus a before/after delta report.

Product framing

• Honest contract: “We fix what can be fixed without judgment calls, and we tell you exactly what we fixed, what we found but couldn’t fix, and what your score was before and after.” No PDF/UA conformance guarantee — a PDF missing alt text or with untagged content cannot be made conformant deterministically. Tier 1 never invents content.
• Same PDF, not a rebuild. Unlike the main conversion pipeline (re-authors PDF → HTML → new PDF, discards layout), this API follows the services/flyer-tagger philosophy: the input bytes are the product. Painting operators are never altered; only structure, metadata, and marked content are touched. Output written as an incremental update (pikepdf) where feasible so the original byte ranges are preserved and diffs are auditable.
• Funnel: apps/tag-editor (Gevell Tags, just merged — free, client-side tag tree editor) is the top of the funnel. Its UI surfaces the same defect classes; a “Fix all automatically” CTA deep-links to this paid API. Manual fixing free, one-click bulk fixing paid.

Engine: what already exists (verified in-repo)

Two deterministic fix engines exist today, in two languages:

1. services/flyer-tagger/ (Python / pikepdf, Lambda) — in-place editor, pixel-identical, veraPDF UA-1 validated on the reference fixture. Its deploy/handler.py already dispatches an edit-op job protocol over R2 keys: list_figures, list_images, set_alt, set_decorative, walk_elements, walk_table, restructure_table, reorder_elements, set_role, recolor. Relevant deterministic modules:

• flyer_tagger/structure.py — StructTreeRoot/ParentTree build, MarkInfo, /Lang, /Tabs /S, DisplayDocTitle, XMP pdfuaid, strips broken /CIDSet streams (UA rule 7.21.4.2).
• flyer_tagger/preflight.py — deterministic, no-model routing classifier + report of unfixable-in-place problems (sub-12pt text, contrast).
• flyer_tagger/verify.py — pixel diff, structure walk, qpdf —check, veraPDF.

2. workers/api (TypeScript / pdf-lib) — flag-gated repair passes that run on pipeline output today, but the logic is input-agnostic structure surgery:

• src/services/pdf-struct-cleaner.ts — contentless-element removal, empty/ redundant Span collapse, THead/TBody/LI role normalization, ParentTree orphan repair (repairParentTreeOrphans), and (behind STRUCT_TAG_NORMALIZE_ENABLED) NonStruct collapse, header-row TD→TH, bare Div→P.
• src/services/pdf-accessibility-postprocessor.ts — /Lang, /Title + DisplayDocTitle, /Tabs /S, MarkInfo; invokes ParentTree repair behind PARENTTREE_REPAIR_ENABLED.
• src/services/list-marker-normalizer.ts (LIST_MARKER_NORMALIZE_ENABLED) and src/services/font-safe-text.ts (FONT_SAFE_TEXT_ENABLED), wired in src/scheduler/chunk-scheduler.ts.

3. Before/after report — workers/api/src/services/pdf-input-audit.ts (PR #2094) already points the output-side structural scorer + veraPDF sidecar at incoming bytes, producing an AccessibilityScore directly comparable to the post-fix score, plus manualRisks (#2281). This is the delta-report mechanism, already built.

Decision: host the fix engine in flyer-tagger (Python/pikepdf) — it is the only engine designed for arbitrary third-party PDFs and incremental, pixel-safe writes. The TS passes assume pdf-lib documents produced by our own WeasyPrint pipeline; their logic (defect taxonomy, normalization rules) gets ported as new deterministic modules in flyer_tagger/, not called cross-runtime.

Tier-1 fix inventory (deterministic only)

#	Fix	Ground truth in repo
1	ParentTree repair (orphaned MCRs, missing /StructParents entries, NumTree gaps)	pdf-struct-cleaner.ts repairParentTreeOrphans; structure.py builds one from scratch
2	Struct tag normalization (empty Span/contentless element removal, NonStruct collapse, TD→TH in header rows, bare Div→P, THead-in-THead splice)	pdf-struct-cleaner.ts steps 0–6
3	Role-map sanity (/RoleMap cycles, mappings to non-standard roles, custom roles resolved to standard set)	extension of #2 (new module, same walk)
4	List structure (L→LI→Lbl/LBody shape, stray P retagged LBody, marker normalization)	pdf-struct-cleaner.ts step 5 + list-marker-normalizer.ts
5	Missing /Lang (heuristic from dominant script or caller-supplied; report “verify” — wrong Lang passes checkers but mis-cues screen readers)	pdf-accessibility-postprocessor.ts, structure.py
6	Missing /Title + ViewerPreferences /DisplayDocTitle + MarkInfo /Marked + XMP sync	pdf-accessibility-postprocessor.ts, structure.py
7	Tab order: /Tabs /S on every page	both engines
8	Obvious artifact cleanup: unmarked paths/shadings/inline images and content outside any struct elem in an otherwise-tagged doc → /Artifact (only when unambiguous)	inject.py artifact logic, artifacts.py
9	Broken /CIDSet stream strip (veraPDF 7.21.4.2)	structure.py

Every fix is idempotent, content-preserving, and reversible via the report. Fixes that would require judgment (is this Figure decorative?) are reported, never applied.

API shape

Async job flow, mirroring existing conventions (SQS + R2 + jobs table).

POST /api/v1/remediate/pdf          multipart upload OR {"sourceUrl": ...}
  → 202 {"data": {"jobId": "…", "status": "queued"}, "error": null, "meta": {}}

GET  /api/v1/remediate/pdf/:jobId
  → {"data": {"jobId": "…", "status": "processing" | "complete" | "failed",
              "pages": 12,
              "report": { …delta report, when complete… },
              "downloadUrl": "…signed R2 URL, when complete…"},
     "error": null, "meta": {}}

GET  /api/v1/remediate/pdf/:jobId/report    report JSON alone

Flow: upload → R2 (remediate/{jobId}/input.pdf) → jobs row → SQS → flyer-tagger Lambda (new {"op": "remediate_tier1", "key": …} edit-op — slots straight into the existing deploy/handler.py dispatch) → input audit → fixes → verify → output audit → write output.pdf + report.json to R2 → job complete. Route registration on both compositions serving it: Lambda entry workers/api/src/index-aws.ts and Node prod server.ts (index.ts is legacy CF worker — skip).

Delta report schema (sketch)

{
  "before": {"score": 41, "verapdf": {"failedRules": 17}, "issues": [...]},
  "after":  {"score": 78, "verapdf": {"failedRules": 4}},
  "fixes":  [{"type": "parenttree_repair", "count": 132, "pages": [1,2,5]},
             {"type": "lang_set", "value": "en-US", "confidence": "verify"}],
  "unfixable": [{"type": "missing_alt", "count": 6, "tier": 2},
                {"type": "untagged_content", "pages": [3], "tier": 2}],
  "manualRisks": {...},
  "pixelIdentical": true
}

before/after come from the pdf-input-audit.ts scorer run on input and output bytes — same yardstick, so the delta is trustworthy by construction.

Output contract & verification gate

• Incremental-update write (pikepdf save(..., incremental) where the input permits; full rewrite fallback flagged in report as "rewrite": true).
• Painting operators never altered — flyer-tagger invariant carries over.
• Gate before delivery (all in verify.py today): qpdf —check clean; veraPDF run (pass not required — deltas reported); render pixel-diff on a sample of pages; after-score ≥ before-score, else fail the job and return the original untouched (never ship a regression).
• veraPDF sidecar is Node-.4-only today (pdf-input-audit.ts header) — flyer-tagger Lambda bundles its own verify chain, so run verification there.

Cost & billing

• New cost_ledger operation_type (e.g. pdf-remediate): requires a migration (DB CHECK is source of truth) + the per-worker enum in workers/api (recordCost) — guarded per-worker by design, see #1286.
• Compute-only cost (no AI in Tier 1): Lambda GB-seconds + R2 ops, cheap and predictable — log per job anyway.
• Billing via entitlement system (account_packages), priced per page (TBD); record usage against the package like conversion does. Free preview: run the input audit + report only (cheap), charge to apply fixes.

Non-goals (= Tier 2, AI-assisted, separate design)

• Alt text generation, reading-order inference, table structure inference, tagging fully untagged content (that’s flyer-tagger’s vision path).
• Scanned/image-only PDFs (OCR), AcroForm field labeling (formfields.py exists but form UX stays on forms.gevell.com).
• Contrast/recolor and font-size fixes (pixel-changing; recolor.py exists but is opt-in visual change, not “same PDF”).
• Encrypted or signed PDFs (signature would be invalidated — reject with a clear error).

Rollout plan

1. Phase 0 — engine port (~1–1.5 wk): new flyer_tagger/tier1.py implementing fixes 1–4 (port pdf-struct-cleaner logic to pikepdf); fixes 5–9 mostly exist in structure.py/inject.py — extract into standalone passes. Corpus test against fixtures/ + a set of real broken PDFs; idempotence + pixel-diff CI.
2. Phase 1 — job plumbing (~3–4 d): remediate_tier1 op in deploy/handler.py; routes in workers/api (index-aws.ts + server.ts); jobs table reuse; report persisted to R2.
3. Phase 2 — billing (~2–3 d): migration for operation_type, entitlement check, per-page usage recording, free audit-only mode.
4. Phase 3 — funnel (~2–3 d): tag-editor “Fix automatically” CTA + API docs in docs/api/openapi.yaml; marketing entry in products.ts if productized.

Open questions

1. Sync fast-path? Small PDFs (< ~20 pages) finish in seconds — offer a synchronous ?wait=true mode, or keep everything async for one code path?
2. Untagged input: Tier 1 on a PDF with no struct tree can only fix metadata (5–7). Reject, or run and report honestly (“3 fixes applied, 94 issues need Tier 2”)? Leaning: run + report — it feeds the upsell.
3. Incremental save vs. object-stream inputs: pikepdf incremental saves on heavily cross-reference-streamed PDFs can balloon; when do we accept a full rewrite silently vs. flag it?
4. /Lang heuristic: ship script-detection default or require the caller to pass lang (report "confidence": "verify" either way)?
5. Public API keys vs. session-auth-only at launch (existing API is session-scoped; a standalone API implies key-based auth — new surface).
6. Where does the TS logic live long-term — port-and-freeze in Python, or keep both engines and accept drift? (Proposal: Python becomes canonical for in-place; TS passes stay pipeline-internal.)