NIST AI RMF 1.0 — Alignment Map
Framework version: NIST AI RMF 1.0 (January 2023)
Scope: AI features in Gevell PDF, Gevell Audit, and Gevell Report Card
Owner: Engineering lead
Review cadence: Quarterly and after any material change to an AI feature or provider
Overview
The NIST AI Risk Management Framework organizes AI risk management into four functions: GOVERN, MAP, MEASURE, and MANAGE. This document maps existing controls and open gaps to each function.
Open gaps reference GitHub issues for tracking. Where a control is implemented but not formally documented, this document serves as the record.
Companion documents: the quantified, per-risk scoring lives in the
AI Risk Register (likelihood × disaster, current mitigations, residual, and
owner for every known risk). The adversarial red-team of each AI-facing feature is in
prompt-injection-redteam-2026-09-14.md. Read this
alignment map for framework coverage and those two for the measurement evidence.
GOVERN — Policies, accountability, and culture
Establishes organizational practices, assigns accountability, and creates a culture where AI risk is treated as a first-class concern.
What is in place
| Control | Evidence |
|---|---|
| Designated policy owner for production AI features | docs/admin/ai/ai-development-policy.md — each feature requires a named owner before release |
| Approved-tool register | docs/admin/ai/ai-development-policy.md §“Operating procedure” |
| Human review required before AI-assisted code merges to production | Standard PR workflow; policy requires reviewer approval |
| Human approval gate on agentic features | Triage agent drafts GitHub issues; Larry approves before any issue is created — no autonomous action |
| Provider contractual prohibition on training use | AWS Bedrock ZDR agreement; Mathpix improve_mathpix: false |
| Incident reporting procedure | docs/admin/ai/ai-development-policy.md §“Incidents and exceptions” |
| Quarterly policy review commitment | docs/admin/ai/ai-development-policy.md §“Operating procedure” |
Gaps
| Gap | Issue |
|---|---|
| #3021 | |
| NIST AI RMF alignment not previously documented | This document (closes #3020) |
MAP — Identifying and categorizing AI risk in context
Identifies the AI system’s context, purpose, and risk categories; maps data flows and affected stakeholders.
AI features in scope
| Feature | Product | Purpose | AI provider |
|---|---|---|---|
| Math equation OCR | Gevell PDF | Extracts equations from page images; converts to MathML | Mathpix |
| Diagram alt text generation | Gevell PDF | Generates short accessibility descriptions for figures and diagrams | Claude via AWS Bedrock (ZDR) |
| “Fix with AI” | Gevell PDF | Applies a user-specified accessibility correction; output constrained to find-and-replace pairs | Claude via AWS Bedrock (ZDR) |
| WCAG criterion pre-grading | Gevell Audit | Pre-screens criteria to surface likely failures for human review | Claude Haiku via AWS Bedrock (ZDR) |
Gevell Report Card does not currently use AI inference.
Data flows
- Input: User uploads a PDF to Gevell PDF. The file is stored in Cloudflare R2 (customer-scoped, encrypted at rest).
- AI processing: Relevant pages or images are passed to the AI provider. No file is persisted at the provider; all providers are configured for zero data retention (ZDR).
- Output: Accessible HTML is returned to the user and stored in R2 alongside the source file. AI-generated text (alt text, fix suggestions) is embedded in the output.
- Audit: No customer document content is used to train or fine-tune any model.
Identified risk categories
| Risk | Category | Likelihood | Impact |
|---|---|---|---|
| Customer document content retained by AI provider | Data retention / privacy | Low (ZDR contractually enforced on all paths) | High |
| Prompt injection via uploaded document content | Security | Medium (documents can embed instructions) | Medium |
| Biased or inaccurate alt text for images of people | Fairness / accuracy | Measured & remediated — skin-tone labeling skew found and reduced to zero (bias-evaluation.md, #3016) | Medium |
| ”Fix with AI” instruction used to exfiltrate content | Security | Low (output constrained to find-and-replace; rate limited) | Medium |
| PII/PHI in documents passed to AI without filtering | Privacy | Unmitigated — text filter withdrawn (#3463). It covered three call paths and was ineffective on the largest, which prompts the model with the page image | High |
| Mathpix data access via QA program | Privacy | Mitigated (improve_mathpix: false on all call sites — PR #3018) | Medium |
Gaps
| Gap | Issue |
|---|---|
| Formal data-flow diagram not yet published | #3021 |
| PII/PHI pre-filter not implemented; text-only approach withdrawn — effective on two text-only paths, ineffective on the conversion cascade, never wired to the rest | #3463 |
MEASURE — Analyzing and assessing AI risks
Quantifies risk through testing, monitoring, and evaluation. Tracks whether controls are working.
What is in place
| Measure | Method | Cadence |
|---|---|---|
| PDF/UA-1 conformance | veraPDF automated tests run in CI on every PR | Per-commit |
| Alt text quality (human verification) | human_verification table records human vs. AI agreement on WCAG judgments; docs/admin/ai/ai-development-policy.md requires evaluation before feature release | Per feature release |
| Output format enforcement | ”Fix with AI” output parsed and validated as find-and-replace pairs; rejected if malformed | Per request |
| ZDR enforcement | gemini-client.ts fails closed (throws) if AWS credentials absent; Bedrock is the default with no fallback to Google | Per request |
| Rate limiting | AI features rate-limited at request layer (10/min, 100/day per user); fails closed | Per request |
| Cost tracking | All AI calls record provider, model, tokens, and estimated cost to the cost ledger | Per request |
Gaps
| Gap | Issue |
|---|---|
| #3016 | |
| #3021 | |
| #3021 | |
| #3021 |
MANAGE — Prioritizing and treating AI risks
Responds to identified risks through controls, mitigations, and ongoing monitoring.
Controls currently implemented
Data retention
- All AI inference routes through AWS Bedrock under the ZDR agreement — no customer content is retained or used for training.
- Mathpix configured with
improve_mathpix: falseon all four call sites, opting out of the QA team data access and image persistence program (PR #3018). gemini-client.tsfails closed without AWS credentials — there is no silent fallback to Google’s non-ZDR API.
Prompt injection
- “Fix with AI” strips
<user_instruction>delimiter tags from user input before prompt assembly. - User instruction field is capped at 2,000 characters.
- Model output is constrained to find-and-replace pairs only (max 50 edits, 4,000 bytes each); free-form HTML emission is not possible.
Privilege limitation
- No agentic tools in the main conversion pipeline.
- Triage agent (internal, employee-facing) operates read-only against the codebase; all GitHub issue creation requires Larry’s explicit approval.
- Models receive no persistent memory of prior sessions.
Rate limiting
- “Fix with AI”: 10 requests/minute, 100 requests/day per user; limit failures are closed (request rejected, not silently degraded).
Malware
- File type validated against MIME allowlist on upload.
- Planned: S3 staging bucket + AWS GuardDuty Malware Protection to scan files before they enter R2 (#3019).
Rollback
- All AI features are controlled by feature flags; any feature can be disabled without a code deploy.
- Provider can be switched or disabled via environment variable without a code deploy (GEMINI_BEDROCK, etc.).
- Kill-switch runbook covers online (admin dashboard) and offline (Supabase SQL, AWS Lambda env var) paths:
docs/admin/runbooks/ai-kill-switch.md.
Re-measurement cadence
AI risk is not measured once. The following cadence governs re-measurement; it is the authoritative statement referenced by the risk register and closes the “measurement cadence” item of #3021.
| Trigger | What is re-run |
|---|---|
| Annually (and at minimum quarterly for a documentation review) | Full risk-register review; re-score every entry against current controls; confirm owners and next-review dates |
| Any new AI feature | Add a risk-register entry; extend the prompt-injection red-team to the new feature; confirm ZDR routing for its call sites before launch |
| Any model-family or provider change (e.g. a new vision model, a new Bedrock profile, a non-Bedrock provider) | Re-run the bias evaluation (bias-evaluation.md) — its results do not transfer across models; re-run the prompt-injection red-team; re-verify ZDR routing |
| Any change to a prompt that handles untrusted content | Re-run the relevant red-team payloads against the changed prompt |
| Periodically (and after any Bedrock routing change) | Re-run the log-based ZDR verification (CloudTrail + cost-ledger reconciliation) and attach fresh evidence — method documented in ai-risk-register.md §“ZDR routing verification”. Initial verification completed 2026-09-14 (live smoke test); #3013/#3014 closed |
The register’s own header records a quarterly review with a dated next-review; this table adds the event-driven triggers. Owner: Larry Anglin / AnglinAI.
Open mitigations
| Risk | Planned control | Issue |
|---|---|---|
| Malware in uploaded files | S3 staging + GuardDuty scan before R2 admission | #3019 |
| PII/PHI passed to AI without filtering | No planned control. The Comprehend text layer was built and withdrawn (#3463). It was wired into three call paths: effective on the two text-only ones, ineffective on the PDF→HTML conversion cascade, which prompts the model with the page image rather than the scrubbed text. A control that could be described product-wide must cover the image channel. | #3463 |
| Direct Anthropic SDK calls bypassing Bedrock ZDR | createAnthropicClient(env) factory routing all SDK calls through Bedrock | #3013, #3014 |
| Bias in alt text generation | Structured evaluation across demographic dimensions; remediation if differential treatment found | #3016 |
| Unmeasured AI risks | Formal risk register, red-team testing, independent ZDR verification | #3021 |
Summary status
| Function | Status |
|---|---|
| GOVERN | Substantially implemented; risk register published (ai-risk-register.md); AI development policy is a draft pending adoption (#3066) |
| MAP | Implemented; PII/PHI filtering withdrawn (#3463) as too narrow to describe as a product control, and formal data-flow diagram pending (#3021) |
| MEASURE | Bias evaluation complete (#3016); risk register + prompt-injection red-team complete (#3021); ZDR routing verified in production 2026-09-14 (#3013/#3014 closed), periodic CloudTrail re-check per cadence |
| MANAGE | Core controls implemented; re-measurement cadence defined (see MANAGE §“Re-measurement cadence”); malware scan flag-gated and Bedrock-only Anthropic path pending (#3013/#3014, #3019). PII filtering is not a pending control — it was withdrawn (#3463), and the residual privacy risk is carried openly in the register rather than shown as mitigated. |