Automating the human-review queue — PDF-only path
Companion to human-review-automation-plan.md (a working draft, not yet committed). That plan assumes a PDF → HTML → PDF round-trip and leans on the fact that you generate the HTML. This one covers the path where there is no HTML: an input PDF is tagged/remediated in place (flyer-tagger / pdfpolish), or a third-party PDF is audited and never regenerated. Time-based media, flashing, and moving content are still excluded — they don’t exist in a static PDF.
Status & dependencies. This is a planning doc, not a spec.
- The census scanner referenced throughout (Tier0 pikepdf + Tier1 veraPDF + clause-class map) lives on the unmerged branch
feature/document-accessibility-census. Rows marked “done (census)” describe that branch, notmain; treat the Auto layer as available once census merges, not shipped.- The veraPDF clause ids in the tables below are drawn from the census
clause-class-map.md(veraPDF 1.30 PDF/UA-1 profile). A—means veraPDF has no machine-checkable clause for that row — it is a human-verification item by design, not an oversight.- The companion HTML plan it cross-references is itself an uncommitted draft; where this doc says “the HTML plan already marked X,” verify against that draft before quoting it.
The classification vocabulary is unchanged:
| Auto | Machine decides. Never enters the queue. |
| Flag | Machine decides the majority, escalates a bounded exception set. |
| Assist | Machine drafts, human always confirms. |
| Human | Stays. |
Confidence on prevalence: M measured, E estimate with reasoning, U unknown. Run costs are marginal per page at August 2026 inference prices.
What changes from the HTML plan, in one paragraph
The single best move in the HTML plan — treat ~8–10 criteria as template invariants, test once per release, assert downstream — does not exist here. There is no generator emitting uniform output, so nothing is a per-release property; every document is bespoke again. That’s the loss. The gain that offsets it: PDF conformance is governed by a formal standard with a reference checker — PDF/UA-1 (ISO 14289-1) and the Matterhorn Protocol, machine-verified by veraPDF — which gives a larger, cleaner deterministic surface than axe-on-a-driven-browser ever does, and which you already run today in the census scanner. The judgment work (reading order, alt text, tables, artifacts, OCR) is format-independent and transfers unchanged. So the plan inverts: Phase 1 gets rewritten, Phases 2–3 survive, and the whole thing maps onto a published standard instead of prevalence data you’d have to generate yourself.
The structural point, before the tables
In the HTML plan the invariant lived in your template. Here the invariant lives in the standard. PDF/UA-1 defines exactly what a conformant tagged PDF must contain, and its veraPDF 1.30 profile machine-checks 106 clauses (per the census clause map: 101 structural + 5 content) with zero inference. Everything veraPDF can decide is Auto by construction — not because you asserted it once, but because a reference implementation of an ISO spec decided it.
The Matterhorn Protocol (31 checkpoints / 136 failure conditions) partitions the same PDF/UA-1 requirements into machine- vs. human-checkable, but this plan maps rows to the veraPDF clause ids the census actually emits, not to Matterhorn condition numbers — the two numbering systems are not interchangeable, and the codebase uses veraPDF. Any Matterhorn-vs-veraPDF coverage fraction (“roughly half”) should be confirmed against Protocol 1.1 before it’s quoted externally.
What veraPDF cannot decide is not a gap in your tooling; it’s by design. The judgment items — logical reading order, meaningful sequence, alt-text appropriateness, heading-level correctness, table semantics — have no machine-checkable veraPDF clause (they are Matterhorn human-verification checkpoints). Those are precisely your Flag/Assist/Human rows. This is a stronger position than the HTML side: your automation boundary is drawn by a standards body, not by you, so “we automate the machine-checkable clauses and assist every human-verification checkpoint” is a claim you can back against a published authority — the substantiation problem the HTML plan had to solve by generating its own prevalence data is already solved here.
Concretely: the census scanner (Tier0 pikepdf + Tier1 veraPDF + clause-class map, zero AI cost) is the Auto layer for this path. Most of Phase 1 below is productizing work that already runs.
PDF stage — the deterministic Auto layer
Most of this is inspectable directly from PDF structure (tag tree, ParentTree, AcroForm dictionary, outline tree, annotation array) with no model and no browser. Much of it is already built in flyer-tagger / census.
veraPDF clause ids are from the census clause-class-map.md (veraPDF 1.30 PDF/UA-1 profile); — = no machine-checkable clause (structural/rendered check, not in the veraPDF profile).
| Review item | veraPDF clause / WCAG | What automates it | Ceiling | How common | Run cost | Build |
|---|---|---|---|---|---|---|
| Tags present / well-formed | 7.1-3, 7.1-11 / 1.3.1 | veraPDF structural pass | Auto | Very high M | negligible | done (census) |
| ParentTree / MCID integrity | 7.1-12 / 1.3.1 | pikepdf structural walk | Auto | Very high E | negligible | mostly done |
| Artifact vs. real content | 7.1-1, 7.1-3 / 1.3.1 | Positional recurrence across pages + content classification | Flag | Very high E | negligible | 2 wk |
| List structure (L/LI/Lbl/LBody) | 7.2-17…7.2-20 / 1.3.1 | Bullet/numbering pattern vs. tag tree | Auto | Very high E | $0.002 | 2–3 wk |
| Tag order vs. content stream | — / 1.3.2 | Same reading-order engine (below) | Flag | Very high E | shared | shared |
| Field tooltips / types / tab order | 7.18.1-3, 7.2-25 / 1.3.1, 4.1.2 | Deterministic from AcroForm dict + field geometry | Auto | High E | negligible | 2 wk |
| Radio-group association | — / 1.3.1 | Shared field name + geometric clustering | Auto | High E | negligible | in above |
| Link–OBJR pairing | 7.18.5-1 / 1.3.1 | Structural check against annotation array | Auto | Moderate E | negligible | 1 wk |
| Bookmarks match outline | — / 2.4.1 | Compare outline tree to heading tags | Auto | High E | negligible | 3–5 d |
| Language of document / parts | 7.2-29, 7.2-34 / 3.1.1, 3.1.2 | Per-run language ID vs. /Lang | Auto | Low–moderate E | negligible | 1 wk |
| Text contrast | — / 1.4.3 | Sample rendered pixels behind glyph bboxes, worst-case ratio | Auto | High E | negligible | 1–2 wk |
| Non-text contrast | — / 1.4.11 | Same engine on borders/UI boundaries | Auto | Moderate E | negligible | 3–5 d |
PDF stage — the judgment layer (transfers from the HTML plan)
These are the human-verification items — no machine-checkable veraPDF clause (—). Where a clause is listed, veraPDF checks presence only; adequacy is the judgment the row adds. Format-independent; the engines are the ones the HTML plan already marked “shared.”
| Review item | veraPDF clause / WCAG | What automates it | Ceiling | How common | Run cost | Build |
|---|---|---|---|---|---|---|
| Logical reading order | — / 1.3.2 | A11y/tag-tree order vs. rendered geometry; linearize; LLM coherence score | Flag | Very high in multi-column E | $0.005 | 3–4 wk (critical path) |
| Alt-text adequacy | 7.3-1 (presence) / 1.1.1 | VLM on crop + page context; decorative/informative split first, then equivalence | Flag | Near-universal in PDF E | $0.01–0.04 | 3–4 wk |
| Complex images needing long desc. | 7.3-1 (presence) / 1.1.1 | Chart/diagram/map classifier gates a long-desc requirement | Assist | High in gov/edu E | $0.005 | 2 wk |
| Table: data vs. layout | 7.5-1 / 1.3.1 | Content-type uniformity + header styling + VLM on render | Flag | Very high E | $0.01 | part of below |
| Table: header assignment | 7.5-1, 7.5-2 / 1.3.1 | Geometry-driven scope; Headers/IDs for spanned/nested | Flag | Very high E | $0.02 | 4–6 wk |
| Heading level correctness | 7.4.2-1, 7.4.4-1 / 1.3.1 | Font size/weight vs. tag level; LLM outline check | Flag | High E | $0.003 | 1–2 wk |
| OCR quality on scanned pages | — / 1.4.5 | Engine confidence + VLM re-read of low-confidence regions, diff | Flag | High for scanned E | $0.01 | 2–3 wk |
| Math and formulas | 7.7-1 (presence) / 1.1.1 | Formula region detection → MathML or generated alt | Assist | Low overall, high in STEM U | $0.01 | 4+ wk |
What does NOT apply (leaves the queue by not existing)
A static PDF has no interaction model and does not reflow. These HTML-plan rows are N/A here — they exit the queue by not applying, which is worth stating explicitly so no one budgets time for them:
- Reflow, text spacing, orientation (1.4.10, 1.4.12, 1.3.4)
- Target size, dragging alternatives (2.5.8, 2.5.7)
- Focus visible / not obscured (2.4.7, 2.4.11)
- Keyboard traps, keyboard operability, tab-order-as-operability (2.1.1, 2.1.2, 2.4.3)
- Consistent navigation / identification (3.2.3, 3.2.4) — no cross-page chrome in a static PDF
- Status messages, live regions (4.1.3)
Bypass blocks / multiple ways (2.4.1, 2.4.5) are not N/A here — their PDF analog is the bookmark/outline tree, which the Auto layer already checks (see “Bookmarks match outline” above). They’re covered, not absent.
Name/role/value (4.1.2) survives only for form PDFs, and there it collapses to static AcroForm dictionary inspection — no browser, no interaction harness. For non-form PDFs it too is N/A.
What stays human
Same residual as the HTML plan, minus the interaction items that no longer exist:
- Authorial intent on ambiguous alt text and long descriptions.
- Scoping — what’s in scope, sample selection, whether an exception applies. Principal-level.
- Novel content patterns the models haven’t seen. Bounded and shrinking.
- The conformance determination and remarks. At 100% detection you still need someone to assert PDF/UA conformance and be accountable. Matterhorn’s human-verification checkpoints are defined as requiring human sign-off — the standard itself keeps this human.
Build order
Ranked by review minutes eliminated per engineer-week. The ranking is different from the HTML plan because Phase 1 there was the template battery — which is gone.
Phase 1 — productize the deterministic layer (largely already built)
The census scanner’s veraPDF + pikepdf structural checks are the Auto layer: tags, ParentTree/MCID, outline-vs-bookmarks, AcroForm fields/radio-groups, Link–OBJR, language runs, contrast sampling. This is the volume of PDF review time and it is deterministic. Wrap it as a per-document report with retained evidence. Add artifact-vs-content and list-structure detection, each ~2–3 weeks.
Phase 2 — the judgment volume drivers (shared engines)
Reading order first — one engine serves reading-order coherence and tag-order-vs-content-stream verification, and it is the critical path; if it slips, the phase slips. Then the decorative/informative image split (the cheaper half of alt adequacy, removes most of the volume). Then OCR verification, which matters more here than in the HTML plan because scanned PDFs are over-represented in the PDF-only inbox.
Phase 3 — expensive, high value
Table semantics — hardest and most valuable; budget the header-assignment work on spanned/nested tables conservatively (the pathological gov/edu cases). Full alt-text adequacy after the classifier. Heading-level correctness.
Deferred
Math (STEM-only, expensive). Anything requiring an interaction model — it doesn’t exist for static PDF.
Cost
Same order of magnitude as the HTML plan, weighted toward the deterministic layer being nearly free. A fully-instrumented text-only page is dominated by veraPDF/pikepdf compute (negligible marginal cost); pages with images and tables reach $0.05–0.08 on inference. Against a $3–8/page manual remediation market, still well under one percent. Compute is not the constraint; engineering weeks and expert minutes are.
Instrument as you go. Every item routed to a reviewer is a labeled example against a Matterhorn human-verification checkpoint. Log per checkpoint: escalation rate, reviewer override rate, time-to-decide, inter-reviewer agreement. Promote Flag → Auto only when the confirmation rate clears a threshold across a meaningful sample, and retain the evidence — mapped to the checkpoint it satisfies. Because the boundary is drawn by a published standard rather than by you, the substantiation file writes itself against Matterhorn, which is the strongest position available after the accessiBe FTC order.