Polish / flyer-tagger — Haiku 4.5 vs Sonnet 4.6 benchmark (issue #3131)
TL;DR — recommendation: TIERED (hand to #3132), NOT a flat switch
Haiku 4.5 is ~78% cheaper and 4–5× faster than Sonnet 4.6 on the flyer-tagger analysis path, and every output it produced is still veraPDF PDF/UA-1 compliant. But PDF/UA-1 compliance measures structural conformance, not semantic correctness — and on the semantic dimensions that matter for a compliance product (heading structure, figure detection, alt-text accuracy) Haiku regresses materially on rich/visual documents:
- beacom-flyer (figure-heavy): Haiku flattened all 6
H3sub-headings toP, dropped/merged 2 of 7 figures, and produced wrong alt text — conflating people and losing two named headshots. Wrong alt text is worse than none. - math-review (prose + equations): Haiku tagged 26 equations as
Figurewith spoken-math alt (Sonnet left them asP). This is a large behavioral divergence that needs human accessibility review, and it tripped an alt-text-length truncation. Role agreement with Sonnet was only 78.5%. - contact-form (sparse): near-parity — the one profile where Haiku is safe.
Net: Haiku is safe on simple/text-only docs and unsafe unsupervised on figure- and equation-rich docs. That is the exact shape a tiered router (#3132) should exploit: route simple docs to Haiku, keep visual/complex docs on Sonnet. A flat switch is a NO-GO for a compliance product.
What was measured
Real flyer-tagger CLI process() pipeline (inventory → vision analysis →
injection → structure → verify) run per-doc per-model on Bedrock via
anthropic.AnthropicBedrock — the exact ZDR path prod uses. ANTHROPIC_API_KEY
was unset so no first-party/Google path was reachable; every request in the run
log hit bedrock-runtime.us-east-1.amazonaws.com. There is no Google/Gemini code
path in services/flyer-tagger/ (grep-confirmed), so the ZDR constraint holds
structurally.
- Sonnet id:
us.anthropic.claude-sonnet-4-6 - Haiku id:
us.anthropic.claude-haiku-4-5-20251001-v1:0 - Harness:
services/flyer-tagger/tools/haiku_benchmark.py(+haiku_quality_diff.py), committed with this report. - Pricing per 1M tokens (in/out): Sonnet $3/$15, Haiku $1/$5.
Doc set (representative)
| Doc | Profile | Pages |
|---|---|---|
| beacom-flyer | figure-heavy flyer (14 images, 6 headshots + logo) | 1 |
| math-review | prose + inline equations | 6 |
| contact-form | sparse form / text-only | 1 |
| bsee-plan | table/form-dense degree plan (238 widgets, banded) | 3 |
| ut-resources | mixed prose + figures | 5 |
The three complete doc-pairs below (figure-heavy, prose+math, form/sparse) span the decisive quality dimensions. bsee-plan and ut-resources runs were still completing at report time (Sonnet’s adaptive-thinking streams on the widget-dense, banded bsee-plan are very slow — a data point in Haiku’s favor on latency); their partial observations are noted and do not change the recommendation.
Cost + latency (measured, complete pairs)
| Doc | Sonnet $ | Haiku $ | Save | Sonnet out-tok | Haiku out-tok | Sonnet wall | Haiku wall |
|---|---|---|---|---|---|---|---|
| beacom-flyer | 0.1245 | 0.0207 | −83% | 7,472 | 1,463 | 78 s | 15 s |
| math-review | 0.5070 | 0.1070 | −79% | 29,655 | 8,070 | 345 s | 65 s |
| contact-form | 0.0240 | 0.0133 | −45% | 1,113 | 723 | 16 s | 12 s |
| Total | 0.6555 | 0.1410 | −78% | 38,240 | 10,256 | 439 s | 92 s |
Output tokens dominate cost, and Haiku emits ~3.7× fewer of them — which is why it is both cheaper and lower-quality on rich docs (fewer output tokens = sparser/less-detailed structure). The 32-page prod job projection (173,941 in / 148,138 out) at these rates: Sonnet ≈ $2.74, Haiku ≈ $0.92 (−67%). The per-doc measured savings (−78%) run higher than that projection because Haiku’s output-token count collapses on our docs.
Quality scores (Haiku vs Sonnet)
| Dimension | beacom-flyer | math-review | contact-form |
|---|---|---|---|
| veraPDF PDF/UA-1 (both) | PASS / PASS | PASS / PASS | PASS / PASS |
| Line coverage (all lines claimed) | full | full (242/242) | full |
| Role/tag agreement vs Sonnet | 0.889 | 0.785 | 0.917 |
| # role disagreements | 6 | 52 | 1 |
| Reading-order similarity | 0.778 | 1.000 | 1.000 |
| Figures found (Sonnet → Haiku) | 7 → 6 | 1 → 26 | 0 → 0 |
| Truncation / max_tokens events | 0 | 1 (alt too long) | 0 |
| Adaptive-thinking rejected (retry) | yes | yes | yes |
Concrete quality findings
Heading structure (beacom-flyer). Sonnet tagged 6 section titles as H3;
Haiku demoted all 6 to P. That erases heading-navigation for screen-reader
users (WCAG 2.4.6 / 1.3.1). veraPDF still passes because a P is structurally
valid — the defect is invisible to the validator.
Alt-text accuracy (beacom-flyer). Side-by-side:
- Sonnet: 6 distinct, correct headshot descriptions (Jaden Plooster, Alexander Green, George Itah, Anjila Deshemaru, Makayla Van Voorst, Bennett Brusseau) + the USD Beacom logo.
- Haiku: only 4 headshots, dropped Alexander Green and Bennett Brusseau,
and mislabeled attributes (e.g. “wearing a plaid shirt” for a person Sonnet
described with dark hair). It also promoted a decorative location-pin icon to
a
Figure. Producing wrong/lost alt text on named people is a serious accessibility defect — arguably worse than the raw image.
Equation handling (math-review). Haiku wrapped 26 equation lines as
Figure with spoken-math alt (“f subscript 1 of x subscript 1 …”); Sonnet
left them as reading-flow P. Spoken-math can aid comprehension, but wrapping
inline text as figures removes it from normal reading order and is jarring;
Haiku also truncated one 373-char alt string at the 250-char cap. This is a
behavioral divergence requiring human a11y sign-off, not a clean win for
either model. It is the source of the 52 role disagreements / 0.785 agreement.
Adaptive thinking. The flyer-tagger analysis call requests extended
(“adaptive”) thinking. Haiku 4.5 rejects that config on Bedrock (400 Bad Request) and the tagger transparently retries without it. Every Haiku page
therefore pays one extra billed prompt and runs without the thinking budget
Sonnet gets — a plausible contributor to the quality gap. A serious Haiku
adoption should first tune the analysis call for Haiku (drop/adjust thinking),
then re-benchmark; the numbers here are Haiku “as-is” behind Sonnet’s prompt.
ZDR verification
- Client is
anthropic.AnthropicBedrock(noANTHROPIC_API_KEY); all traffic tobedrock-runtime.us-east-1.amazonaws.com(run-log confirmed). - No
google/gemini/genai/verteximport or call site anywhere inservices/flyer-tagger/(grep-confirmed) — the ZDR posture is structural, not configuration-dependent.
Recommendation
TIERED — hand off to #3132. Do NOT flat-switch. Do NOT change the prod default in this PR.
Rationale:
- Cost/latency upside is real and large (−78% measured, ~−67% on the prod 32-page job; 4–5× faster).
- But Haiku regresses on the exact dimensions this product exists to guarantee — heading structure, figure detection, and alt-text accuracy — on figure- and equation-rich docs, and veraPDF does not catch those regressions.
- contact-form shows near-parity on simple/text-only docs — a safe Haiku lane exists.
Concrete path for #3132 (tiered routing):
- Route simple/text-only docs (low image count, no equations, low widget density — e.g. the existing preflight classifier + inventory image/widget counts) to Haiku.
- Keep figure-heavy, equation-heavy, and table/form-dense docs on Sonnet.
- Gate any Haiku lane behind veraPDF plus a semantic check (heading presence, figure count sanity, alt-text non-empty/length) — veraPDF alone is insufficient.
- Before widening the Haiku lane, tune the analysis call for Haiku (extended thinking is rejected today) and re-run this harness.
Reproduce
See services/flyer-tagger/tools/README-haiku-benchmark.md. Raw artifacts
(tagged PDFs, analysis JSON, results.json, quality_diff.json) were generated
under /tmp/haiku-bench* during this run and are not committed (they contain
full tagged renditions); the harness regenerates them deterministically per doc.