Runbook: Self-hosted Node server → permanent EC2 + on-demand fleet

Migrate the always-on PDF/heavy-processing workload off the self-hosted Docker box (10.1.1.4) onto one permanent EC2 instance (warm API + sidecars + scheduler leader) backed by the existing ASG spot fleet (0–N, scales on SQS depth) for burst load. End state has no dependency on LAN hardware.

Read first: docs/NODE-SERVER.md (current Node box role), docs/aws-setup.md (existing CDK stacks + ASG), and the memory notes on Node rebuild ordering and orphan recovery. This runbook assumes those stacks are already deployed to the production env in the accessible AWS profile.

0. What actually moves

Role today (`10.1.1.4`)	Target	Notes
2× `api-node` Hono `server.ts` behind Traefik, at `api.theaccessible.org` via CF Tunnel	Permanent EC2 instance, same containers, same tunnel origin	Warm, never scales to 0
Python sidecars: weasyprint, verapdf, audiveris, opus-harness, logo-extractor, flowchart-extract, markitdown, audio-transcribe	Same sidecars, co-located on the permanent instance	Fleet instances reach them over the private network (see §6)
`ChunkScheduler` leader (orphan/wedge recovery of `chunk_jobs`)	Permanent instance ONLY	Single-leader invariant — see §5. This is the migration’s sharpest edge.
`course-map-api` (profile)	Permanent instance (same compose profile)	Serves `api.gevell.com/course-map-api/*`
Burst chunk processing	Existing ASG spot fleet (`workers/batch`)	Already deployed, driven by SQS depth; today it sits at 0

Unchanged / already on AWS: Lambda API (api-pdf.gevell.com), SQS, DynamoDB, S3, SES email intake, CloudWatch monitoring, Supabase (external), R2 (external).

Decision points flagged inline as ⟐ DECIDE — resolve these before Phase 2.

1. Pre-flight (do days ahead, no downtime)

1.1 Inventory the box you’re leaving

ssh -i ~/.ssh/nightly-audit larry@10.1.1.4
cd ~/accessible
git rev-parse HEAD                      # code the box is running
docker compose ps                       # exact running services + profiles
docker compose config --services        # everything defined
cat .env.node-server | cut -d= -f1      # env var NAMES only (do NOT print values)
crontab -l                              # any host cron the box owns?

Record which compose profiles are active (pptx, course-map, monitoring). Anything running that is NOT in docker-compose.yml (host crons, one-off scripts, the nightly backup, DMARC, phone worker) must be re-homed separately — grep the memory index; several workers live only on LAN today.

1.2 Confirm the AWS baseline exists

export AWS_PROFILE=accessible
aws sts get-caller-identity
# The 7 prod stacks should already exist:
aws cloudformation list-stacks --query \
  "StackSummaries[?starts_with(StackName,'AccessiblePdfProd-')].StackName" \
  --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --output text
# ASG max is 4 in prod today (env-config.ts maxWorkerInstances):
aws autoscaling describe-auto-scaling-groups --query \
  "AutoScalingGroups[?contains(AutoScalingGroupName,'AccessiblePdfProd')].[AutoScalingGroupName,MinSize,MaxSize,DesiredCapacity]" \
  --output table

1.3 Parity-audit the SSM parameter set against `.env.node-server`

The permanent instance and the fleet both pull secrets from SSM (/accessible-pdf/production/* + /accessible-pdf/shared/*), NOT from a hand-copied .env file. Every var the Node box needs must exist in SSM or the container will crash-loop on a missing var.

# List what SSM has today:
aws ssm get-parameters-by-path --path /accessible-pdf/shared --recursive \
  --query 'Parameters[].Name' --output text
aws ssm get-parameters-by-path --path /accessible-pdf/production --recursive \
  --query 'Parameters[].Name' --output text

Diff that against the key names from §1.1. Fill gaps with aws ssm put-parameter --type SecureString (see docs/aws-setup.md Step 3 for the exact commands). Pay attention to the sidecar-only keys that only exist in .env.node-server today: WEASYPRINT_API_KEY, VERAPDF_API_KEY, FLOWCHART_EXTRACT_API_KEY, HF_TOKEN, DEEPGRAM_API_KEY, the Vertex service-account JSON (vertex-ai.json), and the Bedrock/Anthropic keys.

vertex-ai.json is mounted as a file today (./vertex-ai.json:/app/vertex-ai.json:ro). On EC2, either bake it into the image at build (bad — secret in layer) or drop it via SSM SecureString + write-to-disk in user-data. Prefer the latter.

1.4 ⟐ DECIDE: how the permanent instance is defined

Two clean options — pick one:

A. Fixed-size ASG (recommended). A second ASG (or reuse the compute-stack launch template) with min=max=desired=1, no scaling policy, running the full compose stack (API + sidecars + scheduler leader). This is the “permanent” tier. The existing burst ASG stays batch-worker-only. Self-heals on instance death; no pets.
B. Standalone EC2 instance. Simplest to reason about, but it’s a pet — manual replacement on failure. Only choose this if you want the box to behave exactly like .4 with minimal CDK change.

The rest of this runbook assumes A and calls the new tier the anchor ASG.

1.5 ⟐ DECIDE: instance size

.4 runs 2× api-node at --max-old-space-size=8192 plus ~7 sidecars on a 32 GB host (see the heap comment in docker-compose.yml). Match that: r7i.xlarge (4 vCPU / 32 GB) is the floor for co-locating API + sidecars + one warm scheduler. If you split sidecars onto the fleet (§6, option 2), the anchor can drop to m7i.large (8 GB) but then a single large assembly can OOM — do not go below the 8 GB heap the code already assumes.

2. Build the images the fleet and anchor will run

Both tiers run from ECR, not from a local docker compose build.

2.1 Batch worker image (fleet) — already wired

cd ~/accessible          # repo root
npm ci
npm run build --workspace=packages/shared
npm run build --workspace=workers/batch

ECR_URI=$(aws cloudformation describe-stacks \
  --stack-name AccessiblePdfProd-Compute \
  --query 'Stacks[0].Outputs[?OutputKey==`EcrRepoUri`].OutputValue' \
  --output text)

aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin "$ECR_URI"
docker build -f infra/cdk/docker/worker/Dockerfile -t "$ECR_URI:latest" .
docker push "$ECR_URI:latest"

2.2 Anchor images (API + sidecars)

The anchor runs the same docker-compose.yml services the .4 box does. Two ways to get them onto the instance:

Build on the instance from git (mirrors scripts/rebuild-server.sh exactly — lowest surprise). User-data clones the repo, writes .env/SSM secrets, and runs docker compose up -d --build.
Pre-build API + each sidecar to ECR and have compose pull tags. Cleaner and faster cold-starts, but you must push 6+ images. Recommended once stable; use git-build for the first cutover so behavior is byte-identical to .4.

workers/api/Dockerfile workspace deps: adding any new @scope/shared dependency needs 3 edits in that Dockerfile or the api-node crash-loops on the instance (Lambda bundles, so it’s unaffected). Verify the current Dockerfile builds cleanly before cutover.

3. Provision the anchor ASG (no traffic yet)

Add to the CDK compute stack (or a new anchor-stack.ts):

Launch template: instance type from §1.5, an instance IAM role granting ssm:GetParametersByPath on /accessible-pdf/*, s3:* on the prod bucket, dynamodb:* on the prod table, sqs:* on the queue + DLQ, bedrock:InvokeModel*, and CloudWatch Logs.
User-data that: installs Docker + compose, clones the repo at a pinned SHA, pulls secrets from SSM into .env.node-server (+ writes vertex-ai.json), sets ENVIRONMENT=production and SCHEDULER_ROLE=leader (see §5), then docker compose up -d --build with the same profiles .4 uses.
ASG: min=max=desired=1, spread across your prod subnets, health check = EC2 + the container /ready endpoint via a target group if you front it with an ALB (see §7 ⟐ DECIDE on origin).
No public ingress except from the Cloudflare Tunnel origin / ALB SG.

Deploy:

cd infra/cdk
DEPLOY_ENV=production CDK_DEFAULT_ACCOUNT=$ACCT CDK_DEFAULT_REGION=us-east-1 \
  cdk diff  AccessiblePdfProd-Compute       # or the new anchor stack
DEPLOY_ENV=production CDK_DEFAULT_ACCOUNT=$ACCT CDK_DEFAULT_REGION=us-east-1 \
  cdk deploy AccessiblePdfProd-Compute

Wait for the instance, then verify on the box (via SSM Session Manager, no SSH keys needed):

aws ssm start-session --target <instance-id>
docker compose ps
curl -s localhost:8790/health   | jq .    # api-node
curl -s localhost:8790/ready               # traefik health target
# sidecars reachable on pdf-net:
docker compose exec api-node-1 wget -qO- http://weasyprint:5001/health
docker compose exec api-node-1 wget -qO- http://verapdf:5002 || true

At this point the anchor is live but receiving no production traffic — the tunnel still points at .4.

4. Point the fleet’s scaling at the queue (still no cutover)

The burst ASG already has a scaling policy on SQS depth (see docs/aws-setup.md “Auto Scaling Behavior”). Confirm it, and confirm the anchor’s scheduler is the only one seeding work while .4 still serves traffic.

# Force a burst to prove the fleet scales from 0 → N and back:
aws sqs send-message-batch --queue-url $PROD_QUEUE_URL --entries file://smoke-batch.json
watch -n15 'aws autoscaling describe-auto-scaling-groups \
  --query "AutoScalingGroups[?contains(AutoScalingGroupName,\`Prod\`)].DesiredCapacity"'

You want to see: desired rises within one cooldown (~5 min, ~2–3 min cold start), messages drain, DLQ stays empty, desired falls back to 0. Fix scaling/cooldown before cutover — you do not want to debug autoscaling and a live cutover together.

5. Scheduler ownership — the single-leader invariant

ChunkScheduler (orphan recovery, reclaimStuckAssemblingJobs, wedge recovery) runs today on .4 and is embedded in workers/batch. Running the recovery leader in two places races: two schedulers can both reset an assembling job to processing and re-drive assembly, or double-claim chunk_jobs. Today .4 is the de-facto singleton because the fleet sits at 0.

Invariant after migration: exactly one scheduler performs orphan/wedge recovery.

Make it the anchor instance only. Gate the recovery passes behind an env flag — SCHEDULER_ROLE=leader on the anchor, SCHEDULER_ROLE=worker on every fleet instance. Worker-role schedulers still claim and process their own SQS-assigned work but skip the global recovery scans (recoverOrphanedJobs / reclaimStuckAssemblingJobs). If that flag doesn’t exist yet, add it in chunk-scheduler.ts before cutover — this is the one code change this migration requires.
The existing prod-guard in chunk-scheduler.ts (refuses to start if ENVIRONMENT != production but SUPABASE_URL points at prod) stays — it’s your backstop against a mis-tagged instance stealing prod work.

Do not skip this and “just let both run.” The wedge-recovery logic assumes a single measurer of assembly silence (STALE_ASSEMBLY_TIMEOUT_MS); two leaders will fight over the 30-min stale window.

6. ⟐ DECIDE: where fleet instances get their sidecars

Fleet (workers/batch) instances need weasyprint/verapdf/etc. to process chunks. Two options:

Self-contained fleet images — bake the sidecars into (or compose alongside) the batch worker on each spot instance. Fully parallel, no shared bottleneck, but fatter images and slower cold-starts. Recommended — matches the scale-to-zero goal and keeps the anchor small.
Shared sidecars on the anchor — fleet calls http://<anchor-private-ip>:5001 etc. Smaller fleet images, but the anchor becomes a throughput bottleneck and a SPOF for burst load, and you must open SG ports between the fleet and anchor on the private network. Only pick this if sidecar cold-start cost dominates.

Whatever you choose, set the sidecar *_URL envs on the fleet launch template to match (they default to the compose DNS names today).

7. Cutover (the only user-visible step)

7.1 ⟐ DECIDE: origin path

Keep the Cloudflare Tunnel, just move its origin from .4 to the anchor (run cloudflared on the anchor in user-data, or point an existing tunnel route at the anchor’s private IP). Lowest DNS churn; api.theaccessible.org hostname unchanged. Recommended for a like-for-like cutover.
Replace tunnel with ALB + ACM cert and a Cloudflare CNAME (proxied). More “AWS-native,” but new cert + DNS + SG work during the cutover window. Defer to a follow-up.

7.1a Run `cloudflared` on the anchor (recommended origin)

cloudflared dials outbound to Cloudflare’s edge (443), so the anchor needs no public ingress — the SG stays egress-only. Use a token-based named tunnel (not a cert file) so a replaced ASG instance re-establishes the tunnel automatically from user-data.

One-time setup (from your workstation, AWS_PROFILE=accessible):

# 1. Create a dedicated tunnel for the anchor (separate from .4's tunnel so you
#    can flip routes at cutover and roll back instantly — §8).
cloudflared tunnel create anchor-api
#    → prints a Tunnel ID and writes credentials JSON. Grab the CONNECTOR TOKEN:
TUNNEL_TOKEN=$(cloudflared tunnel token anchor-api)

# 2. Stash the token in SSM (SecureString) — the anchor pulls it at boot, same
#    pattern as every other secret in §1.3.
aws ssm put-parameter --type SecureString \
  --name /accessible-pdf/production/CLOUDFLARED_TUNNEL_TOKEN \
  --value "$TUNNEL_TOKEN"

# 3. Grant the anchor instance role read on just this param (add to the role in
#    the CDK launch template, §3):
#    Action: ssm:GetParameter
#    Resource: arn:aws:ssm:us-east-1:<acct>:parameter/accessible-pdf/production/CLOUDFLARED_TUNNEL_TOKEN

Do not attach api.theaccessible.org to this tunnel’s public-hostname routing yet — leave it dark until the cutover flip in §7.2 (that keeps .4 serving prod right up to the moment you switch).

Anchor user-data — install and run cloudflared as a systemd service:

#!/usr/bin/env bash
set -euo pipefail

# --- cloudflared install (amd64; use -arm64 on Graviton) ---
curl -fsSL -o /usr/local/bin/cloudflared \
  https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x /usr/local/bin/cloudflared

# --- pull the connector token from SSM (instance role must allow GetParameter) ---
REGION=us-east-1
TOKEN=$(aws ssm get-parameter --region "$REGION" --with-decryption \
  --name /accessible-pdf/production/CLOUDFLARED_TUNNEL_TOKEN \
  --query 'Parameter.Value' --output text)

# --- systemd unit: token-based run, auto-restart, no creds on disk beyond env ---
cat >/etc/systemd/system/cloudflared.service <<UNIT
[Unit]
Description=Cloudflare Tunnel (anchor-api)
After=network-online.target docker.service
Wants=network-online.target
Requires=docker.service

[Service]
# Token carries tunnel ID + secret; --no-autoupdate so the binary is pinned.
ExecStart=/usr/local/bin/cloudflared tunnel --no-autoupdate --protocol quic run --token ${TOKEN}
Restart=always
RestartSec=5
# Wait for the API to answer before the edge sends traffic (health-gated origin).
ExecStartPre=/bin/sh -c 'until curl -sf http://localhost:8790/ready; do sleep 3; done'
User=root

[Install]
WantedBy=multi-user.target
UNIT

systemctl daemon-reload
systemctl enable --now cloudflared

The token embeds the tunnel secret, so it appears in the unit file and systemctl cat. That’s acceptable on a single-tenant box, but keep the SG locked down and the instance role scoped to just this one SSM param. If you’d rather not have it in the unit text, write it to /etc/cloudflared/token (mode 0600) and ExecStart=... run --token-file /etc/cloudflared/token.

Ingress mapping is set on the Cloudflare side (Zero Trust → the anchor-api tunnel → Public Hostname): api.theaccessible.org → http://localhost:8790 (Traefik). Because the tunnel and the API share the instance, the origin is just localhost — no SG ports, no private-IP routing. Leave this hostname unmapped until §7.2, then map it (that is the cutover flip).

Verify the connector is up before flipping the hostname:

aws ssm start-session --target <anchor-id>
systemctl status cloudflared --no-pager
cloudflared tunnel info anchor-api      # should show 1+ healthy connector(s)

7.2 Drain and cut

Pick a low-traffic window (check the pipeline for in-flight chunked jobs first — they run in-process and a hard cut strands them at assembling):

# 1. On .4: stop taking NEW work but let in-flight finish.
ssh -i ~/.ssh/nightly-audit larry@10.1.1.4
cd ~/accessible
# Watch for active large jobs draining (Supabase): large_conversion_jobs in
# ('processing','assembling') AND form_conversions in ('refining') — both have
# NO orphan recovery once their host dies. Wait for zero, or accept the anchor's
# scheduler will recover the chunked ones (form_conversions will NOT — see memory).
docker compose logs -f api-node-1 | grep -i assembl

# 2. THE FLIP: move api.theaccessible.org from .4's tunnel to the anchor tunnel.
#    In CF Zero Trust: remove the api.theaccessible.org public-hostname from .4's
#    tunnel, then add it to the `anchor-api` tunnel → http://localhost:8790
#    (§7.1a). This is a route change at Cloudflare's edge — no DNS TTL wait, no
#    restart needed. Confirm from off-box that traffic now lands on the anchor:
curl -s https://api.gevell.com/health | jq .
# Expect platform=node-server, healthy — then confirm the ANCHOR's logs light up
# (not .4's): aws ssm start-session --target <anchor-id>; docker compose logs -f api-node-1

# 3. Enable the anchor's scheduler leader (if you staged it disabled):
aws ssm start-session --target <anchor-instance-id>
# SCHEDULER_ROLE=leader already set in user-data; confirm it's the ONLY leader:
docker compose logs api-node-1 | grep 'ChunkScheduler started'

# 4. On .4: stop the scheduler so there is exactly one leader. Leave containers
#    up but idle for fast rollback (§8), OR bring them down:
docker compose stop api-node-1 api-node-2   # stops serving + stops its scheduler

7.3 Verify end-to-end

Convert a real multi-page PDF via pdf.gevell.com → confirm the anchor’s api-node logs the job, sidecars fire, output lands in R2/S3.
Trigger a large/chunked conversion → confirm chunks fan out, the fleet ASG scales up under load, and the anchor’s scheduler (not .4) logs any recovery.
Email intake: send to convert@pdf.gevell.com → SES→Lambda→SQS→fleet path still works (unchanged, but smoke it).
api.gevell.com/course-map-api/health returns 200 from the anchor.
DLQ stays at 0; CloudWatch dashboard shows instance count sane.

8. Rollback (keep viable for ~1 week)

Because §7.2 step 4 leaves .4 containers stopped, not removed, rollback is fast:

# 1. Re-flip the route: remove api.theaccessible.org from the anchor-api tunnel,
#    re-add it to .4's tunnel (CF Zero Trust). Instant, no DNS wait. Because both
#    tunnels still exist and .4's containers are only stopped (not removed), this
#    is a seconds-long rollback.
# 2. On .4: restart, which also restarts its scheduler leader:
ssh -i ~/.ssh/nightly-audit larry@10.1.1.4 'cd ~/accessible && docker compose up -d'
# 3. On the anchor: demote/stop its scheduler so there is again ONE leader:
aws ssm start-session --target <anchor-id>   # docker compose stop api-node-1 api-node-2

Never have both .4 and the anchor running scheduler leaders against prod Supabase simultaneously (double-recovery race, §5). Rollback = flip leadership, not “run both.”

Keep .4 recoverable until: (a) 5–7 days of clean anchor+fleet operation, (b) DLQ clean through at least one real burst, (c) a full chunked job recovered correctly by the anchor scheduler after a forced fleet-instance kill.

9. Decommission `10.1.1.4`

Only after the rollback window closes clean:

Re-home anything non-compose still on the box (host crons, DMARC/phone workers, nightly backup steps, services/flowchart-extract rebuild hooks) — audit crontab -l and the memory index for LAN-only services.
Update ops docs: docs/NODE-SERVER.md, scripts/rebuild-server.sh (retire or repoint at the anchor), the memory notes that say “rebuild on 10.1.1.4”, and Uptime Kuma monitors.
docker compose down -v on .4, then power it down (keep the disk image for a month before wiping).
Update the deploy story: the anchor now rebuilds via cdk deploy + instance-refresh (or an SSM RunCommand that does git pull && docker compose up -d --build), replacing the SSH scripts/rebuild-server.sh flow.

10. Post-migration ops cheatsheet

Task	Old (`.4`)	New
Deploy new code (API/sidecars)	`scripts/rebuild-server.sh` over SSH	ASG instance-refresh on the anchor, or SSM RunCommand `git pull && docker compose up -d --build`
Scale for burst	manual / N/A (fleet at 0)	automatic on SQS depth (existing policy)
Shell into the box	`ssh -i ~/.ssh/nightly-audit`	`aws ssm start-session --target <id>`
Read logs	`docker compose logs -f`	same, over SSM session; plus CloudWatch/Loki
Who runs the scheduler leader	`.4` (implicit)	the anchor instance ONLY (`SCHEDULER_ROLE=leader`)
Secrets	`.env.node-server` file	SSM `/accessible-pdf/{production,shared}/*`

Appendix: risk register

Risk	Mitigation
Two scheduler leaders race on `chunk_jobs` recovery	`SCHEDULER_ROLE` gate; only the anchor recovers (§5)
In-flight jobs stranded at cutover	Drain `.4` first; anchor scheduler recovers chunked jobs; `form_conversions` have NO orphan recovery — drain those to zero before cutting
Missing SSM param → container crash-loop	Parity-audit §1.3 before provisioning
`vertex-ai.json` / sidecar API keys absent on EC2	Write from SSM SecureString in user-data (§1.3)
New `@scope/shared` dep breaks anchor build	3-edit Dockerfile rule (§2.2); Lambda unaffected so it hides
Fleet can’t reach sidecars	Decide §6 before Phase 2; set `*_URL` on fleet launch template
Anchor is a SPOF	Fixed-size ASG (option A) self-heals; ALB health check drains a sick instance
Cost surprise from anchor running 24/7	`r7i.xlarge` on-demand ≈ $180–250/mo; consider a 1-yr Savings Plan once sized