Alt-text auto-approval — generator / judge / escalation

Companion to human-review-automation-plan.md. That plan lists Alt text adequacy (1.1.1) as a Flag item and keeps authorial intent human. This spec is the detail for the one line that can’t be closed by a deterministic check: how a second AI safely takes alt text from Flag toward Auto, per image class, on measured evidence — without a person, and without losing the conformance guarantee.

Alt text is the only review item that is generative, not structural. Reading order, tag order, contrast, and table headers have a checkable ground truth. Alt text has a range of acceptable answers and no single correct string. That is why it needs a confidence-gated ensemble instead of an assertion — and why it’s the highest-value place to get this right.

---

The rule that makes or breaks it

A judge agreeing bounds correctness; it does not prove it. If the judge shares the generator’s blind spots, it confidently rubber-stamps the generator’s mistakes. Three non-negotiables follow:

1. Decorrelate the errors. The judge must be a different model family than the generator. Agreement then means “correct or both models share the error” — cross-vendor diversity shrinks that second term; same-family diversity does not.
2. Calibrate against ground truth — never trust raw self-reported confidence. LLM “95% sure” is typically overconfident. Empirically map the judge’s score to actual acceptance rate on a labeled set, then gate on measured auto-approve precision, not the model’s stated confidence.
3. Judge against a WCAG rubric, with page context — not vibes. Score concrete criteria and feed the judge the surrounding page text, so it can distinguish decorative / informative / functional. Structured scores are auditable; that matters when you have to back the claim.

---

ZDR constraint and the model shortlist

ZDR is a property of Bedrock, not of Amazon. Every Bedrock model runs inside our AWS account under Bedrock’s data terms — no training on our data, no sharing with the provider — the same boundary already vetted when Claude vision moved to Bedrock. So the judge can be any non-Claude vision model on Bedrock and still preserve ZDR/DPA. We are not limited to Nova.

The judge must actually see the image. Vision-capable (image-input) Bedrock candidates:

Role	Model(s)	Why
Generator	Claude (on Bedrock)	Already in the pipeline
Judge — primary	Amazon Nova Pro / Nova Premier	AWS-native, cheap enough per-image, different lineage, cleanest ops
Judge — max decorrelation	Llama 4 Maverick, Qwen3 VL 235B, Pixtral Large	Greatest architectural distance from Claude
Cross-modal sanity check	Amazon Titan Multimodal Embeddings (or Nova Multimodal / Cohere Embed v4)	Embeds image + alt text; low similarity = likely hallucination. A different mechanism from an LLM judge — pennies, catches gross mismatch the judge might rationalize
OCR (text-in-image / scanned)	Amazon Textract or Mathpix (already in pipeline)	Purpose-built OCR beats a general VLM. Textract is an AWS AI service (not a Bedrock FM) but runs under the same data protection

Do not pick the judge on brand. On the gold set (below), run the generator plus two candidate judges and keep whichever’s errors correlate least with Claude’s. That correlation number is the entire decision.

Confirm exact image-input support for any candidate in the Bedrock console before relying on it — several listed models are text-only (Jamba, DeepSeek, Kimi, MiniMax, Nova Micro) and can’t judge alt text.

---

Pipeline

image + page context
      │
  [0] classify image ── decorative → mark artifact / empty alt (rules; auto)
      │                  text-in-image → OCR (Textract/Mathpix) → verify
      ▼
  [1] GENERATE   Claude writes alt text  (already in pipeline)
      ▼
  [2] JUDGE      non-Claude VLM scores it against the WCAG rubric (+page context)
                 Titan embedding similarity(image, alt) as a second signal
      ▼
  [3] GATE       high-confidence agreement AND similarity ≥ floor
                    → AUTO-SHIP
                 else → HUMAN queue
      ▼
  [4] AUDIT      standing random sample of auto-shipped items → human spot-check
                 drift monitor on override rate

Route by image class — the auto-approve rate is high for decorative/simple photos and low for data viz/STEM, so the gate threshold is per-class, not global.

Image class	Expected disposition
Decorative / logos	Auto (rules + judge), earliest
Simple photos	Auto once precision clears the bar — likely first generative class to flip
Functional (buttons/links)	Auto with page context
Charts / graphs	Flag — AI gets trends, botches numbers; human until data-grounded
Complex diagrams / maps / STEM	Human (authorial intent) — see human-review-automation-plan.md

Disagreement is the product, not the failure. Generator-vs-judge conflicts are simultaneously the highest-risk items and the highest-information training examples. Routing them to humans concentrates expert minutes exactly where they’re worth most and generates the hardest labeled data.

---

”When is the AI more accurate than the human?” — measured, per class

Not a single crossover point, and not a guess. Build a gold set: images with alt text written/adjudicated by multiple expert reviewers. This yields two things at once:

• the reference answer, and
• the human baseline — because reviewers disagree with each other. Alt-text inter-rater agreement is only moderate, and a fatigued reviewer at hour six is a weaker bar than a fresh model.

Then, per image class, at each judge threshold, compute auto-approve precision (of items the gate would auto-ship, the fraction actually acceptable) and compare it to the human edit/override rate. The crossover falls out of the data. For decorative and simple photos it is plausibly already past; for STEM it is not.

---

Instrument now — turn the human tier into a labeled-data factory

We are already paying for human review. Start logging today; the calibration set builds itself. One row per image:

Field	Purpose
image_id, doc_id, image_class	Keys + routing
generator_model, generator_alt	The candidate
judge_model, judge_score, judge_verdict, judge_rationale	The gate input
embedding_model, embedding_similarity	Second signal
human_decision (accept / edit / reject), human_alt, edit_distance	Ground-truth label
reviewer_id, time_to_decide_s, timestamp	Human baseline, throughput, drift

Derived, per class: auto-approve precision at each threshold, human override rate, inter-reviewer agreement, judge–generator error correlation.

Promotion (Flag → Auto) criteria for a class: measured auto-approve precision ≥ the risk bar (set higher than the WCAG minimum for the guarantee — e.g. ≥ 99% acceptable) across a meaningful sample and ≥ human agreement rate and calibration validated on a held-out slice. Start conservative (decorative, simple photos); expand as the evidence accrues. Retain the evidence — it’s the substantiation file behind any claim about what the automation achieves.

---

Guardrails

• Auto-approval is per-image detection, never the conformance assertion. Someone still signs the conformance determination and is accountable for it (see human-review-automation-plan.md → “What stays human”). The gate removes review minutes on individual images; it does not remove the human who warrants the document.
• Anchor to a refreshed gold set, not to judge agreement alone. Re-audit periodically; models and corpora drift.
• Standing audit sample forever. Spot-check a small % of auto-shipped items to catch drift and keep the guarantee defensible.
• Watch correlated overconfidence. If judge and generator agree and are both wrong at a rate above the bar for a class, that class stays Flag regardless of stated confidence.

---

Economics

A judge inference is pennies per image; a human minute is real money. Spending ~$0.005 on a decorrelated judge to safely skip a human on the majority of simple images is the highest-ROI lever in the pipeline — it raises the automatable share, drops human cost per document, and compounds into the ~90%-margin automated tier. This is the mechanism behind the pricing model’s flywheel (docs/admin/operating-costs-model.*): the human tier stays a break-even-or-better QA gate whose value is the pipeline improvement, not a profit center — which is exactly the posture we want it to hold.

---

Build order

Deepens Phase 2–3 of the companion plan.

1. Instrument (now, ~1 wk). Ship the logging schema above into the existing human queue. Zero-risk, and every reviewed image from here on is a labeled example.
2. Classifier + gold set (2–3 wk). Image-class classifier (decorative/informative/functional/complex/STEM/text) + a multi-reviewer gold set per class.
3. Judge + embedding gate (2–3 wk). Wire the non-Claude judge and Titan similarity; run in shadow (log the gate decision, still send everything to humans) to measure precision without risk.
4. Flip per class (ongoing). Enable auto-ship for the classes/thresholds that clear the promotion bar, decorative and simple photos first.
5. Audit + drift (ongoing). Standing sample + override-rate monitor.