Skip to content
AnglinAI Docs

EC2 migration plan: retire 10.1.1.4 (#2246)

Why now: the UTHealth questionnaire (Q22, Q67–78, Q75 “no customer data processed or stored on company premises”) and the approved BCP/DRP describe processing on AWS EC2 only. They are not true until .4 (and anything else on the LAN that touches customer documents) is dark. Submission waits on this.

How to use this doc: the step-by-step mechanics are in node-to-ec2-migration-runbook.md (anchor ASG, images, anchor tunnel, drain-and-flip cutover, rollback). This plan adds what that runbook misses, the decisions still open, and the order of work. Inventory taken 2026-09-23 (live AWS + read-only look at .4).

1. Current state (2026-09-23)

PieceState
Spot worker fleet (Compute stack, batch worker only)Deployed, min 0 / max 4, desired 0. Spot launch failed UnfulfillableCapacity on 2026-09-23
Anchor (always-on Node API + sidecars)Not built — runbook only
cloudflared on EC2Not built
EBS encryptionOff — account default false, launch template sets nothing
SubnetsPublic only, no NAT; SG has no inbound rules
SecretsSSM Parameter Store; Secrets Manager (#2243) unused
Scheduler single-leader gate (SCHEDULER_ROLE)Not in code — today both api-node-1 and -2 run every timer, relying on DB claim guards
DeploysFleet via deploy-aws.yml → ECR :latest (pulled only at boot, no instance refresh); .4 via rebuild-server.sh + self-hosted runners on .4

2. Gaps the runbook does not cover

  1. More services than the runbook lists. Also move: flyer-tagger (needs ~10 GB / 4 CPU on its own), docx2acroform, verapdf, markitdown, audio-transcribe, flowchart-extract, course-map-api, and Traefik routing. Do not move: opus-harness (first-party Anthropic key, @gevell.com-only experiment), audiveris (Gevell Music only; not a released product, out of scope), audio-transcribe (pdf.gevell.com does not accept audio uploads; audio-transcribe also removed from SQS_ALLOWED_TYPES).
  2. Three SQS consumers live only on .4 (pipeline, bulk remediation, flyer-long). The fleet is at 0, so the anchor must consume all three from cutover minute one.
  3. The pptx-remediate service runs on 10.1.1.17 and api-node points PPTX_REMEDIATE_URL at it. Three routes use it: remediate-pptx.ts, remediate-docx.ts and transcribe-audio.ts. None is tied to an active product (confirmed by Larry 2026-09-23), so it is out of scope: leave PPTX_REMEDIATE_URL unset on the anchor. With it unset the routes fall back to http://pptx-remediate:5003, which does not exist on the anchor, so they fail and production has no path to any LAN host.
  4. Self-hosted CI runners on .4 (runner-4, runner-4b) stay: .4 becomes a dedicated CI host after cutover (decided 2026-09-24). CI never handles customer documents, and the other runners (.15/.20/miniLinux) are on-prem too. Production deploys to the anchor go through ECR + SSM Run Command (deploy-anchor.yml).
  5. turbo-cache stays on .4 with the runners.
  6. Non-product job on .4: Shelly EM-50 billing cron stays on .4, but gets its own minimal env file with an R2 token scoped to its bucket and its own Resend key, so no production credential remains on .4.
  7. Logs: Loki/Promtail/Grafana on .4 (30-day retention, no alert rules) back the admin /admin/logs page.
  8. Secret hygiene: 30+ .env.node-server.bak* copies on .4; static AWS IAM keys; direct-provider AI keys (Anthropic, Gemini, OpenAI, Mistral) in the production env despite the zero-data-retention posture.
  9. LAN exposure drift: Traefik :8800 and dashboard :8801 bound to 0.0.0.0 on .4 (compose asks for 127.0.0.1).
  10. Other callers hard-code the .4 path: rotate-secrets.yml (SSH rolling deploy), logs/reset-cost-cap/rebuild-course-map scripts, and the CDK’s reference to the .4 IAM user (env-config.ts).

3. Decisions (recommendations in bold)

#DecisionOptionsRecommendation
D1Anchor count1 instance (ASG min=max=1 across 2 AZs, auto-replaced) vs 2 always-on1 instance. ASG replaces it in ~10 min in the other AZ; queues hold work meanwhile. Reword BCP/DRP to “one instance, automatically replaced” (they currently read as two).
D2Anchor sizem7i.xlarge (4 vCPU / 16 GB) vs m7i.2xlarge (8 vCPU / 32 GB)m7i.xlarge, always on — DECIDED 2026-09-23. ≈ $147/mo on-demand, ≈ $97 with a 1-yr commitment. Measured on .4 (9 days sar): memory in use avg 2.8 GB / peak 5.7 GB; CPU ~7% idle floor, 10–20% hourly averages. A weekday-only schedule was rejected: ~47% of external jobs (90 days) arrive outside weekday 8–5 CT, and a stopped-instance wake takes ~3–5 min (cold ~9–13 min).
D3NetworkKeep public subnets (no inbound SG rules, IMDSv2) vs private subnets + NAT (≈ $33/mo + data)Keep public subnets, zero inbound rules, IMDSv2 required. Egress-only via Cloudflare Tunnel; add S3/SQS/ECR VPC endpoints later if we want private. Matches Q77/Q78 as written.
D4SecretsSSM SecureString (already used) vs Secrets Manager (#2243)SSM SecureString now, loaded at container start; close or re-scope #2243. No .env files on the anchor.
D4aAWS credentials on the anchorInstance role vs the .4 IAM user’s static keysStatic keys for the cutover (2026-09-23). The API signs Bedrock/Textract/S3/SES calls only with AWS_ACCESS_KEY_ID/SECRET from its environment (~18 files), so it cannot use the instance role yet. The keys live in SSM /accessible-pdf/anchor (the Lambda and batch worker load all of /production, where they would override role credentials). The instance role already has the same grants. #3648 (2026-09-26): the Node hosts now load the instance role’s credentials into their env at startup (workers/api/src/server-entry.ts, workers/course-map-api/src/main.ts, refreshed before expiry), and the static keys were removed from SSM. The IAM user’s access key was deactivated the same day; delete it (and the user) once nothing has used it for a few days.
D5LogsLoki/Grafana on the anchor vs CloudWatch LogsCloudWatch Logs (awslogs driver, 30-day retention). Loki on a replaceable instance loses history on every replacement. Rework /admin/logs to CloudWatch Logs Insights (or drop it).
D6Direct AI provider keysKeep vs remove from productionRemove, but after cutover (revised 2026-09-23). ANTHROPIC_API_KEY is still read by active paths (routes/acr.ts, routes/convert.ts, mcp/tools/url-remediate.ts, mcp/tools/acr.ts); removing it before each is proven to use Bedrock risks breaking them. The anchor keeps key parity with .4 for the cutover; a follow-up audits each reference, then drops the keys.

3a. Later: shrink the always-on footprint (after cutover, not a submission blocker)

The preserve pipeline already runs short jobs on the Lambda API and sends only long-running jobs to the server (flyer-long queue). The same split can apply to the reflow pipeline. Org chart already has all its routes in the Lambda (index-aws.ts:179–198) and uses the same bucket (accessible-pdf-files), but the app points at api.theaccessible.org (the Node server). Its AI routes (extract, generate-previews, finalize) do background work after responding and can exceed API Gateway’s 29 s limit, and export uses Puppeteer; those need a queue (or the streaming Function URL) before org chart can move to the Lambda. Once no customer HTTP traffic needs the Node server and the timers run as Lambda crons, the anchor can shrink further or become queue-driven only.

4. Order of work

Phase 0 — quick fixes (today, low risk)

  • Turn on EBS encryption by default for the account (aws ec2 enable-ebs-encryption-by-default) — account setting, Larry runs or approves.
  • Launch template: encrypted gp3 root, IMDSv2 required. (Also fixes the spot fleet.)
  • Widen spot instance types (add c7a/c7i/m7a/m7i .large) to reduce UnfulfillableCapacity.

Phase 1 — code prep (status 2026-09-23: done on branch feature/ec2-phase1-2246, see infra/anchor/README.md)

  • SCHEDULER_ROLE=leader|follower gate in server.ts: timers + SQS consumers run only on the leader (and fixes today’s double-scheduling).
  • Build and push every anchor image to ECR in CI (api-node, flyer-tagger, and the in-scope sidecars), tagged by commit SHA.
  • docker-compose.anchor.yml: images from ECR, awslogs driver, secrets from SSM at start, Traefik on 127.0.0.1, no opus-harness.
  • SSM parity script (infra/anchor/anchor-secrets.py): 85 names on .4 → 52 committed config, 24 already in SSM, 3 dropped (static AWS keys, Shelly invoice key), 7 secrets to copy (5 main + 2 course-map). The triage agent’s /repo mount is not carried over (the employee triage tool needs a repo checkout; decide separately).

Phase 2 — infrastructure (CDK)

  • Anchor stack: ASG min=max=desired=1, m7i.xlarge, 2 AZs, encrypted 100 GB gp3, SSM core, no key pair, no inbound SG rules, IMDSv2.
  • User data: install Docker + cloudflared (token from SSM), compose pull && up -d, health-gate the tunnel.
  • Deploy path: GitHub workflow → ECR → SSM Run Command (compose pull && up -d), rolling api-node-1 then -2. Replaces rebuild-server.sh for production.

Phase 3 — parallel run (no customer traffic)

  • Anchor tunnel on a test hostname; anchor runs as SCHEDULER_ROLE=follower with SQS consumers off.
  • Smoke-test every route and sidecar (PDF convert, flyer, forms/docx2acroform, audit, course-map); run the E2E smoke suite against it.
  • Large-document memory test: api-node’s heap is 6 GB on the anchor (8 GB on .4, #1865). Convert the largest known document (hundreds of pages) through the anchor; if it runs out of memory, move to r7i.xlarge (32 GB, +≈ $46/mo) before cutover.
  • Give the Shelly billing cron its own scoped env file (R2 bucket token + Resend key).

Phase 4 — cutover (runbook §7)

  • Drain .4 (in-flight large_conversion_jobs, form_conversions, flyer_tagging_jobs), stop its consumers and schedulers.
  • Flip api.theaccessible.org to the anchor tunnel; set anchor to leader with consumers on.
  • Watch 48 h. Rollback = flip the route back and restart .4 (kept cold, not wiped, for 14 days).

Phase 5 — sanitize and repurpose .4 as the CI host

  • Rotate every secret that lived on .4 (all of .env.node-server, the .bak copies, CF tunnel token, GitHub runner tokens) — runbook in docs/admin/security/secrets-inventory-and-rotation.md.
  • After the 14-day rollback window, wipe .4 per the Media Handling policy (it held customer documents and production secrets): reinstall the OS, record it in the disposal log, then set it up as CI only — runners, buildx, turbo-cache and the Shelly cron. No production env file, tunnel or API containers.
  • Remove .4 references: CDK env-config.ts IAM user, rotate-secrets.yml, scripts, docs. Delete the 16 legacy EC2 key pairs.
  • Update BCP/DRP wording (D1), the node-to-EC2 runbook, and the HECVAT draft; close #2246.

5. Questionnaire dependencies

AnswerNeeds
Q22, Q72, Q75, Q76Phase 4 done; production has no path to any LAN host (incl. PPTX_REMEDIATE_URL unset)
Q67 (encrypted EC2 disks)Phase 0
Q77/Q78 (no inbound, SSM not SSH, security groups)Phase 2 as specified
Q47 “production servers have no SSH”Phase 5 (.4 gone; anchor has no key pair)

6. Rough effort

Phase 0: an hour. Phase 1: 2–3 days. Phase 2: 2 days. Phase 3: 1–2 days. Phase 4: half a day + 48 h watch. Phase 5: 1 day. About 1.5–2 weeks elapsed.