Skip to content
AnglinAI Docs

EC2 anchor right-size plan

Phase 0 of the elastic-compute epic (#3712), tracked in #3810. This is a plan and has not been applied. anchorInstanceType is still m7i.xlarge. A resize, a deploy of it, and any Savings Plan purchase each need Larry’s explicit approval.

Where things stand

  • EC2 anchor: one m7i.xlarge (4 vCPU, 16 GB, x86_64, about $147/mo list) in a fixed-size Auto Scaling group. It’s defined in infra/cdk/lib/stacks/anchor-stack.ts, and its size comes from anchorInstanceType in infra/cdk/lib/env-config.ts.
  • Load seen so far: hourly average CPU ran 1.5% to 8.2%, peaking at 38%, over the first 23 hours. The last memory reading came from the old office server: 2.8 GB average and 5.7 GB peak. Until #3810 there was no memory metric on the anchor.
  • New memory metric (#3810): the CloudWatch agent publishes to namespace Gevell/Anchor with dimension AutoScalingGroupName only, so the series continues across instance replacements:
    • mem_used_percent
    • mem_available (bytes)
    • disk_used_percent (root volume; extra dimensions path=/, fstype=xfs)
  • Dashboard and alarm: the dashboard is accessible-pdf-production-anchor. Alarm accessible-pdf-production-anchor-memory-high fires when the 5-minute average of mem_used_percent stays above 85% for 15 minutes, and notifies the existing accessible-pdf-production-alarms topic. It treats missing data as missing (INSUFFICIENT_DATA), never as OK. A second alarm, accessible-pdf-production-anchor-memory-metric-missing, fires on the same topic when mem_used_percent has no samples for 15 minutes, meaning the CloudWatch agent is not reporting and the sizing window has a gap.
  • How the agent gets installed: an SSM State Manager association (accessible-pdf-production-anchor-cloudwatch-agent) installs it, not user data. The ASG has no rolling-update policy, so a user-data change would only reach the next instance. The association runs on the current instance right away, on any replacement the ASG launches, and again daily. Deploying it does not replace the instance.

The compose memory limits are a second ceiling

infra/anchor/docker-compose.anchor.yml sets per-container limits that add up to 25.5 GB on a 16 GB host:

ContainersLimit eachTotal
api-node-1, api-node-27 GB (Node --max-old-space-size=6144)14 GB
flyer-tagger5 GB5 GB
docx2acroform, verapdf, weasyprint, markitdown, flowchart-extract, course-map-api1 GB6 GB
logo-extractor512 MB0.5 GB

These limits are caps, not reservations, so they only matter once real use approaches them. But any move to 8 GB or smaller has to lower them in the same change (the two Node heaps first), or one busy container can push the host into OOM.

Options

Prices are us-east-1 list, on-demand, Linux.

InstancevCPURAMArch$/moNotes
m7i.xlarge (today)416 GBx86_64147No change needed
r7g.large216 GBarm6478Same RAM for about half the cost, but only 2 vCPU. Needs arm64 (see below)
t4g.xlarge416 GBarm6498Burstable. Needs arm64
m7i.large28 GBx86_6474The only true one-line change that halves RAM
m7g.large28 GBarm6460Needs arm64
t4g.large28 GBarm6449Burstable. Needs arm64
t4g.medium24 GBarm6425Burstable. Needs arm64
t4g.small22 GBarm6412Burstable. Needs arm64

arm64 (every g type, including all t4g)

Every step on the #3712 ladder below 16 GB is a t4g, so the arm64 work is a prerequisite for most of the ladder, not just for r7g. Switching to Graviton is more than one line:

  1. Images: .github/workflows/build-anchor-images.yml builds platforms: linux/amd64 only. All nine images in infra/cdk/anchor-images.json need linux/arm64 builds (multi-arch manifests are simplest), and each needs a smoke test on arm64. Watch these:
    • flyer-tagger: Python wheels (PyMuPDF, pikepdf, and pdfpolish’s native dependencies)
    • weasyprint: Pango and Cairo
    • verapdf: JVM, should be fine
    • api-node: Chromium or Puppeteer, sharp, and any other native Node modules
    • course-map-api
  2. The AMI: anchor-stack.ts uses latestAmazonLinux2023({ cpuType: X86_64 }), which must become ARM_64.
  3. User data: it downloads docker-compose-linux-x86_64 and its checksum, which must become docker-compose-linux-aarch64.
  4. Third-party images: traefik:v3 is already multi-arch.

The AMI and user data both live in the launch template, so a replacement instance picks them up, which the resize needs anyway.

Burstable (t4g, and t3 or t3a if staying x86)

  • CPU credits: a t4g earns credits toward a baseline CPU level: 40% per vCPU on t4g.xlarge, 30% on t4g.large, and 20% on t4g.medium and t4g.small. The anchor’s measured hourly average of 1.5–8.2% fits well under every baseline. Heavy jobs spend credits, and the credits refill while the box is idle.
  • Unlimited mode: t4g launches in unlimited mode by default. If the box runs past its baseline for long enough to use up its credits, AWS bills surplus credits at about $0.04 per vCPU-hour, so it slows nothing but costs more. Watch CPUSurplusCreditsCharged for the first month after a move to t4g.
  • Staying on x86: the burstable x86 types need no image work, at a higher price: t3.large (8 GB) is about $61/mo and t3a.large about $55/mo.

Downsizing ladder (from #3712)

Size is set by the largest single job, so the heavy jobs have to move off first.

Anchor memoryCheapest fitMonthlyPossible once
16 GBm7i.xlarge / r7g.large$147 / $78now (r7g needs arm64)
8 GBt4g.large (m7i.large on x86)$49 ($74)large-document assembly and the flyer long lane are off the box
4 GBt4g.medium$25conversion chunks and audits are off the box
2 GBt4g.small$12helper services have moved, leaving the API and timed sweeps
nonenone$0the API is on Lambda and sweeps are on EventBridge

Choosing a size from a week of the metric

Collect at least 7 days after the association first runs. Before trusting the week, make sure it includes the rare heavy work, because a week without a heavy job will undersize the box. Check in Supabase that at least one flyer long-lane job (over 32 pages) and at least one conversion over 30 pages ran in that window. If neither ran, extend the window or run one on purpose (the #3712 “measure assembly at scale” step does both).

Pull the hourly peaks. aws sso login is needed first, and these calls are read-only:

Terminal window
ASG=$(aws cloudformation describe-stacks --stack-name AccessiblePdfProd-Anchor \
  --query 'Stacks[0].Outputs[?OutputKey==`AnchorAsgName`].OutputValue' --output text)


# Highest memory-used % in each hour, for 7 days
aws cloudwatch get-metric-statistics --namespace Gevell/Anchor --metric-name mem_used_percent \
  --dimensions Name=AutoScalingGroupName,Value="$ASG" \
  --start-time "$(date -u -v-7d +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 3600 --statistics Maximum Average --output table


# Lowest memory available in each hour (bytes)
aws cloudwatch get-metric-statistics --namespace Gevell/Anchor --metric-name mem_available \
  --dimensions Name=AutoScalingGroupName,Value="$ASG" \
  --start-time "$(date -u -v-7d +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 3600 --statistics Minimum --output table

(date -v-7d is the macOS form. On Linux, use date -u -d '7 days ago' +%FT%TZ.)

Work out the size from the peak:

  1. Peak used: 16 GB × (highest mem_used_percent Maximum) ÷ 100. Or use 16 GB minus the lowest mem_available, which is the more direct figure.
  2. Needed: the peak used × 1.3 for headroom. The largest known single consumer is the flyer long lane (flyer-tagger capped at 5 GB), so if that lane still runs on the box, the peak must include a long-lane job.
  3. Choose: the smallest rung whose RAM is at least the needed figure, and whose “possible once” condition in the ladder is true. The metric can’t show a job that didn’t run that week.

Worked example: the office server’s 5.7 GB peak × 1.3 is about 7.4 GB, which just fits 8 GB. That is too close to act on without this week of data, and the compose limits would have to come down first.

After any resize, keep the 85% alarm. If it fires more than once in the first two weeks, go back up a rung.

The CDK change

x86 targets (m7i.large, t3.large, t3a.large): one line in infra/cdk/lib/env-config.ts, production block:

anchorInstanceType: 'm7i.large',

Graviton targets (r7g, m7g, t4g): the same line, plus the arm64 work above (multi-arch images, cpuType: ARM_64, the aarch64 compose binary). Ship it as one PR only after the arm64 images are built and smoke-tested.

Either way: lower the compose memory limits in infra/anchor/docker-compose.anchor.yml in the same PR when going below 16 GB.

Deploying it, and the expected downtime

Merging anything under infra/cdk/** or infra/anchor/** to main triggers Deploy AWS (.github/workflows/deploy-aws.yml, cdk deploy --all). Merging a resize PR is the production deploy, so merge only with approval.

That deploy does not resize the running instance. The ASG has no rolling-update policy (its only UpdatePolicy is AutoScalingScheduledAction), so CloudFormation just points the group at the new launch-template version. The swap is a separate step:

  1. Pick a quiet window and check that no jobs are active. A rebuild interrupts conversions in flight, and orphan re-drive can lose the PDF (#3787). Also check the flyer long queue and in-progress chunk_jobs.

  2. Start an instance refresh with at most one instance, terminating before launching:

    Terminal window
    aws autoscaling start-instance-refresh --auto-scaling-group-name "$ASG" \
      --preferences '{"MinHealthyPercentage":0,"MaxHealthyPercentage":100,"InstanceWarmup":900}'

    Do not use a launch-before-terminate refresh (MaxHealthyPercentage 200). Both instances would run the compose stack with API_NODE_1_ROLE=leader, which means two chunk schedulers and two sets of queue consumers at once.

  3. Expect about 5–15 minutes of downtime for api.gevell.com and the SQS consumers. That covers the boot, user data (docker, compose, cloudflared), the SSM association installing the agent, and anchor-up.sh pulling nine images from ECR. SQS holds messages during the gap, so nothing is lost as long as no job is mid-flight. The Lambda API (api-pdf.gevell.com) stays up the whole time.

  4. Verify:

    • The ASG shows 1 InService instance of the new type.
    • /ready returns 200 through the tunnel.
    • The Gevell/Anchor metrics resume on the dashboard.
    • A small conversion completes.

Rollback

  • Normal path: revert the anchorInstanceType line (and any compose-limit change), merge with approval so Deploy AWS runs, then run the same instance refresh. The downtime is the same, about 5–15 minutes.

  • Emergency, faster than CI: point the ASG back at the previous launch-template version, then refresh:

    Terminal window
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name "$ASG" \
      --launch-template LaunchTemplateId=<id>,Version=<previous>

    This leaves CloudFormation drifted, so follow up with the revert PR.

  • arm64 moves: keep the amd64 image tags. ECR keeps the last 100 tags, so rolling back to x86 needs no rebuild.

Savings Plan sizing (read-only)

Status 2026-09-27: AWS has no recommendation yet. Don’t buy now. The read-only recommendation returned an empty summary for Compute plans with 1-year and 3-year terms and 7-, 30- and 60-day lookbacks. The anchor has only been running since 2026-09-24, so Cost Explorer has about three days of steady compute to work from.

What the account showed on 2026-09-27:

ItemValue
Running EC2one m7i.xlarge (the anchor), on-demand, launched 2026-09-24
EC2 compute costabout $4.84 a day, roughly $0.20 an hour on-demand
EC2 other (EBS and similar)about $0.27 a day
Lambda$0 (inside the free tier)
Active Savings Plansnone

Before 2026-09-24, EC2 compute was a few cents a day, so any lookback longer than a week is mostly empty.

Recommendation: wait. Walk the resize ladder first, then let about 30 days of post-resize usage build up and re-run the command below. Committing now would size the plan to the full m7i.xlarge, which the resize is meant to retire. With round-down, the post-resize commitment will be a fraction of the current $0.20 an hour.

Terminal window
aws ce get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT --lookback-period-in-days THIRTY_DAYS

When it returns a summary, record SavingsPlansPurchaseRecommendationSummary.HourlyCommitmentToPurchase, EstimatedMonthlySavingsAmount and EstimatedSavingsPercentage here.

Guidance for reading it:

  • Buy Compute, not EC2 Instance, Savings Plans. A Compute plan covers EC2 of any family, size or region, plus Fargate and Lambda. The discount follows work as #3712 moves it off the anchor. An EC2 Instance plan is locked to the m7i family and would be stranded by a move to t4g or r7g.
  • The 30-day lookback includes the m7i.xlarge at full size, so the recommendation will overshoot the post-resize floor. Commit no more than the spend that will remain after the next planned rung. For example, if the anchor goes to t4g.large (about $0.067/h on-demand), the anchor’s share of a commitment should be well under that. Lambda and Fargate baselines add to it only where they are steady.
  • The trade-off: under-committing only leaves some savings unclaimed, while over-committing is paid every hour for a year. So round down.
  • Buying needs Larry’s explicit approval. Nothing in #3810 purchases anything.