On-Premises Deployment Guide

This document covers how to package and deliver TheAccessible.Org as a self-contained on-premises product for university and government customers. It builds on the existing Docker Compose local development environment (local-dev.md) and extends it into three deployment tiers:

Docker Compose + Management CLI — the primary delivery model
VM Appliance (OVA) — for customers who want zero-Docker-knowledge turnkey deployment
Helm Charts — for customers already running Kubernetes (future)

The management CLI (accessible-server) is the core of the on-prem experience. It wraps Docker Compose with lifecycle management: setup, upgrades, rollbacks, backups, health monitoring, and secrets management.

Architecture Overview
Cloud-to-Local Service Mapping
Container Registry Strategy
Management CLI (accessible-server)
CLI Command Reference
Configuration System
Secrets Management
Upgrade and Rollback Strategy
Backup and Restore
Health Monitoring
VM Appliance (OVA)
Networking and TLS
FERPA Compliance Notes
Sizing Guidelines
File and Directory Layout
Implementation Roadmap

Architecture Overview

The on-prem stack replaces all cloud-managed services with local equivalents running in Docker. The customer’s server runs the entire application — API, workers, database, storage, queueing, and monitoring — behind a reverse proxy with TLS termination.

┌──────────────────────────────────────────────────────────────────────┐
│  Customer Server (Ubuntu 22.04+)                                     │
│                                                                      │
│  accessible-server CLI                                               │
│  ├── manages Docker Compose lifecycle                                │
│  ├── handles config, secrets, backups                                │
│  └── exposes health dashboard                                        │
│                                                                      │
│  ┌────────────────────────────────────────────────────────────────┐  │
│  │  Docker Compose Stack                                          │  │
│  │                                                                │  │
│  │  Caddy (reverse proxy + auto TLS)                              │  │
│  │    ├── :443 → Traefik :8800 (API)                              │  │
│  │    └── :443 → Next.js :3000 (Frontend)                         │  │
│  │                                                                │  │
│  │  Traefik (internal LB)                                         │  │
│  │    ├→ API Node 1 :8790                                         │  │
│  │    └→ API Node 2 :8790                                         │  │
│  │                                                                │  │
│  │  PostgreSQL 15          (replaces Supabase Postgres)            │  │
│  │  GoTrue                 (replaces Supabase Auth)                │  │
│  │  PostgREST              (replaces Supabase REST)                │  │
│  │  MinIO                  (replaces Cloudflare R2 / AWS S3)       │  │
│  │  Redis                  (replaces Cloudflare KV + SQS)          │  │
│  │  Batch Worker           (replaces EC2 ASG / Lambda)             │  │
│  │  WeasyPrint             (HTML-to-PDF engine)                    │  │
│  │  Audiveris              (music notation OCR)                    │  │
│  │  Marker                 (local PDF parsing — no Datalab API)    │  │
│  │  Loki + Grafana         (monitoring)                            │  │
│  └────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

Key Differences from Cloud Architecture

Concern	Cloud (Production)	On-Prem
Frontend hosting	Cloudflare Pages	Caddy serving static export
API compute	CF Workers + Lambda	Node.js in Docker (same Hono app)
Database	Supabase managed Postgres	Self-hosted PostgreSQL 15
Auth	Supabase Auth (GoTrue)	Self-hosted GoTrue
Object storage	Cloudflare R2	MinIO (S3-compatible)
KV / sessions	Cloudflare KV	Redis
Queue	SQS	Redis (BullMQ) or Redis Streams
Batch workers	EC2 ASG spot instances	Docker container(s), scaled by CLI
PDF parsing	Datalab Marker API	Local Marker container (GPU optional)
TLS	Cloudflare edge	Caddy auto-TLS (Let’s Encrypt or customer cert)
DNS	Cloudflare DNS	Customer’s DNS
Monitoring	Grafana Cloud + Uptime Kuma	Bundled Grafana + Loki
Backups	Supabase daily + R2	CLI-managed pg_dump + MinIO mirror

Cloud-to-Local Service Mapping

Already Portable (No Code Changes)

These services use the same code paths that exist today in the Node server mode:

Object Storage: S3ObjectStorage class already supports configurable S3_ENDPOINT. Point at MinIO.
API: Same Hono app running in Node.js via @hono/node-server (identical to current Docker deployment).
PDF Processing: WeasyPrint and Audiveris containers are unchanged.
Browser Rendering: Native Puppeteer in the API Node container (same as current Node server mode).
Monitoring: Loki + Grafana + Promtail Docker Compose stack is already built.

Requires New Abstraction

Service	Current Implementation	On-Prem Replacement	Work Required
KV Store	`CloudflareKvRestStorage` (REST API) or `InMemoryKvStorage` (local)	Redis via `ioredis`	New `RedisKvStorage` class implementing the existing `KvStorage` interface
Queue	SQS via AWS SDK	Redis via BullMQ	New `RedisQueue` class; replace SQS polling in batch worker
Auth	Supabase Auth (GoTrue hosted)	Self-hosted GoTrue container	GoTrue config + JWT secret alignment; no API code changes
Database	`@supabase/supabase-js` via Supabase Kong	Same client pointed at self-hosted Kong/PostgREST	Config-only change (SUPABASE_URL env var)
PDF Parsing	Datalab hosted Marker API	Local Marker Docker container	New provider in the conversion cascade; env var to select local vs hosted

New Components (On-Prem Only)

Component	Purpose	Implementation
Caddy	Reverse proxy + automatic TLS	Official Caddy Docker image with Caddyfile generated by CLI
Redis	KV store + job queue	Official Redis 7 Docker image, AOF persistence
Marker (local)	PDF-to-markdown parsing	Datalab’s open-source Docker image, GPU optional
Management CLI	Lifecycle management	Node.js CLI (`accessible-server`)

Container Registry Strategy

All on-prem Docker images are published to GitHub Container Registry (ghcr.io) under the anglinai org. Customers authenticate with a read-only Personal Access Token (PAT) or a deploy token provided during onboarding.

Image Naming Convention

ghcr.io/anglinai/accessible-api:1.5.0
ghcr.io/anglinai/accessible-api:1.5.0-20260402
ghcr.io/anglinai/accessible-api:latest
ghcr.io/anglinai/accessible-worker:1.5.0
ghcr.io/anglinai/accessible-frontend:1.5.0
ghcr.io/anglinai/accessible-weasyprint:1.5.0
ghcr.io/anglinai/accessible-audiveris:1.5.0
ghcr.io/anglinai/accessible-marker:1.5.0

Versioning

Semantic versioning for release tags: MAJOR.MINOR.PATCH
Date-stamped tags for traceability: 1.5.0-20260402
latest tag always points to the most recent stable release
edge tag for pre-release builds (opt-in via CLI config)

CI/CD Publishing

A GitHub Actions workflow builds and pushes images on every tagged release:

name: Publish On-Prem Images
on:
  push:
    tags: ['v*']
jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          context: .
          file: workers/api/Dockerfile
          push: true
          tags: |
            ghcr.io/anglinai/accessible-api:${{ github.ref_name }}
            ghcr.io/anglinai/accessible-api:latest
      # Repeat for worker, frontend, weasyprint, audiveris, marker

Customer Authentication

During accessible-server init, the CLI prompts for a registry token and writes it to Docker’s credential store:

docker login ghcr.io -u customer-org -p <token>

The token is scoped to read:packages only.

Management CLI (`accessible-server`)

The management CLI is the primary interface for on-prem customers. It wraps Docker Compose with opinionated lifecycle management, eliminating the need for customers to understand Docker internals.

Distribution

The CLI is distributed as an npm global package:

npm install -g @anglinai/accessible-server

For environments without Node.js, a standalone binary is built with pkg or bun build --compile:

# Download standalone binary (Linux x64)
curl -fsSL https://releases.theaccessible.org/cli/latest/accessible-server-linux-x64 \
  -o /usr/local/bin/accessible-server
chmod +x /usr/local/bin/accessible-server

CLI Package Structure

packages/accessible-server/
├── package.json
├── tsconfig.json
├── src/
│   ├── index.ts                # Entry point, command router
│   ├── commands/
│   │   ├── init.ts             # Interactive setup wizard
│   │   ├── start.ts            # Start all services
│   │   ├── stop.ts             # Graceful shutdown
│   │   ├── restart.ts          # Stop + start
│   │   ├── status.ts           # Health dashboard
│   │   ├── upgrade.ts          # Pull new images, migrate, restart
│   │   ├── rollback.ts         # Revert to previous version
│   │   ├── backup.ts           # Database + storage backup
│   │   ├── restore.ts          # Restore from backup
│   │   ├── logs.ts             # Tail service logs
│   │   ├── config.ts           # View/edit configuration
│   │   ├── secrets.ts          # Manage API keys and credentials
│   │   ├── doctor.ts           # Diagnose common issues
│   │   └── uninstall.ts        # Clean removal
│   ├── lib/
│   │   ├── compose.ts          # Docker Compose subprocess wrapper
│   │   ├── config.ts           # Config file read/write
│   │   ├── health.ts           # Health check logic
│   │   ├── migrate.ts          # Database migration runner
│   │   ├── backup.ts           # Backup/restore logic
│   │   ├── registry.ts         # Image pull and version check
│   │   ├── tls.ts              # TLS certificate management
│   │   └── ui.ts               # Terminal UI helpers (spinners, tables)
│   └── templates/
│       ├── docker-compose.yml  # Production Compose template
│       ├── Caddyfile            # Reverse proxy template
│       ├── .env.template        # Environment variable template
│       └── grafana/             # Pre-configured dashboards
├── bin/
│   └── accessible-server       # Shebang entry point
└── __tests__/
    ├── commands/
    └── lib/

Dependencies

commander — CLI framework
inquirer — interactive prompts (for init)
chalk — colored output
ora — spinners
yaml — Compose file manipulation
execa — subprocess execution (docker, docker compose)
semver — version comparison
cli-table3 — formatted tables

CLI Command Reference

`accessible-server init`

Interactive setup wizard. Run once on a fresh server.

$ accessible-server init

  TheAccessible.Org — On-Premises Setup

  ? Enter your license key: XXXX-XXXX-XXXX-XXXX
  ✔ License validated (Organization: State University, Tier: Enterprise)

  ? Server hostname (FQDN): accessible.university.edu
  ? TLS certificate source:
    ❯ Automatic (Let's Encrypt)
      Custom certificate (provide cert + key paths)
      Self-signed (development only)

  ? Container registry token: ghcr_xxxxxxxxxxxxx
  ✔ Authenticated with ghcr.io/anglinai

  ? PostgreSQL password: ••••••••••••
  ? MinIO root password: ••••••••••••
  ? JWT secret (auto-generate recommended): [auto]
  ✔ Generated 256-bit JWT secret

  AI Service Configuration (optional — skip for air-gapped deployments):
  ? Anthropic API key (Claude): sk-ant-••••••••
  ? Google Cloud credentials file: /path/to/vertex-ai.json
  ? Gemini API key: ••••••••
  ? Mathpix App ID: (skipped)
  ? Mathpix App Key: (skipped)

  ? Data directory (for volumes): /opt/accessible/data
  ? Backup directory: /opt/accessible/backups

  ✔ Configuration written to /opt/accessible/config.yml
  ✔ Secrets encrypted and stored in /opt/accessible/.secrets
  ✔ Docker Compose file generated
  ✔ Caddyfile generated for accessible.university.edu
  ✔ Pulling images from ghcr.io/anglinai... (this may take a few minutes)
  ✔ All 12 images pulled (v1.5.0)

  Run 'accessible-server start' to launch the stack.

What init does:

Validates the license key against the AnglinAI license API (or offline with a signed JWT license file for air-gapped installs)
Prompts for hostname, TLS preference, credentials, and AI API keys
Generates a config.yml with all non-secret settings
Encrypts secrets at rest in .secrets (AES-256, key derived from a machine-specific fingerprint)
Generates a docker-compose.yml from the template, injecting the customer’s configuration
Generates a Caddyfile for TLS termination
Authenticates with the container registry and pulls all images
Creates the data directory structure for persistent volumes
Runs a preflight check: Docker version, available disk, available RAM, port conflicts

`accessible-server start`

Starts the entire stack in the correct dependency order.

$ accessible-server start

  Starting TheAccessible.Org v1.5.0...

  ✔ PostgreSQL         healthy (0.8s)
  ✔ GoTrue (Auth)      healthy (1.2s)
  ✔ PostgREST          healthy (0.5s)
  ✔ Redis              healthy (0.3s)
  ✔ MinIO              healthy (0.6s)
  ✔ WeasyPrint         healthy (0.4s)
  ✔ Audiveris          healthy (0.7s)
  ✔ Marker             healthy (1.5s)
  ✔ API Node 1         healthy (2.1s)
  ✔ API Node 2         healthy (2.3s)
  ✔ Batch Worker       healthy (1.8s)
  ✔ Caddy (TLS)        healthy (0.9s)
  ✔ Grafana            healthy (1.1s)

  All 13 services running.
  Application:  https://accessible.university.edu
  Grafana:      https://accessible.university.edu/grafana
  MinIO Console: http://localhost:9001 (internal only)

What start does:

Reads config.yml and .secrets
Runs docker compose up -d
Waits for each service health check in dependency order
Prints the service status table
If any service fails health check after 60s, prints diagnostic info and suggests accessible-server doctor

`accessible-server stop`

Graceful shutdown with configurable drain period.

$ accessible-server stop
  Draining active requests (30s grace period)...
  ✔ All services stopped. Data volumes preserved.

`accessible-server status`

Dashboard view of all services, resource usage, and version info.

$ accessible-server status

  TheAccessible.Org v1.5.0 (build 247, 2026-04-01)
  Hostname: accessible.university.edu
  Uptime:   14d 6h 32m

  Service          Status    CPU    RAM     Restarts
  ─────────────────────────────────────────────────
  PostgreSQL       healthy   2.1%   512MB   0
  GoTrue           healthy   0.3%   64MB    0
  PostgREST        healthy   0.2%   48MB    0
  Redis            healthy   0.5%   128MB   0
  MinIO            healthy   1.0%   256MB   0
  API Node 1       healthy   8.2%   1.1GB   0
  API Node 2       healthy   7.8%   1.0GB   0
  Batch Worker     healthy   3.4%   780MB   0
  WeasyPrint       healthy   0.1%   92MB    0
  Audiveris        healthy   0.1%   180MB   0
  Marker           healthy   0.5%   640MB   0
  Caddy            healthy   0.1%   32MB    0
  Grafana          healthy   0.8%   128MB   0

  Storage: 42.1 GB / 500 GB (8.4%)
  Database: 2.3 GB (1,247 documents processed)
  Last backup: 2026-04-02 02:00 UTC (successful)
  TLS certificate: valid until 2026-06-30 (89 days)

`accessible-server upgrade`

Pulls new images, runs migrations, and restarts services with zero-downtime rolling update.

$ accessible-server upgrade

  Current version: 1.5.0
  Checking for updates...
  ✔ New version available: 1.6.0

  Changelog:
    - Added batch remediation for Word documents
    - Improved math OCR accuracy for scanned PDFs
    - Fixed: empty alt text on decorative images
    - Security: updated Puppeteer to 23.x

  ? Proceed with upgrade to 1.6.0? (Y/n) Y

  ✔ Pre-upgrade backup completed → /opt/accessible/backups/pre-upgrade-1.5.0-20260402.tar.gz
  ✔ Pulling new images... (1.6.0)
  ✔ Running database migrations (3 new)...
    - 20260315_058_add_batch_word_support.sql ✔
    - 20260322_059_alt_text_decorative_flag.sql ✔
    - 20260401_060_usage_metrics_index.sql ✔
  ✔ Rolling restart (API Node 1 → API Node 2 → Worker)...
  ✔ Health checks passing.

  Upgrade complete: 1.5.0 → 1.6.0
  Rollback available: 'accessible-server rollback'

Upgrade flow:

Check the registry for newer image tags (compare semver)
Fetch and display the changelog (from a CHANGELOG.md baked into the image or fetched from a release API)
Create a pre-upgrade backup (database dump + config snapshot)
Pull all new images (tagged with the target version)
Run any new database migrations (from supabase/migrations/ directory inside the API image, diffed against the schema_migrations table)
Rolling restart: update one API node at a time, keeping the other serving traffic via Traefik
Run post-upgrade health checks
If any health check fails, automatically trigger rollback

`accessible-server rollback`

Reverts to the previous version. Uses the pre-upgrade backup and previously-pulled images.

$ accessible-server rollback

  Current version: 1.6.0
  Rollback target: 1.5.0 (backup from 2026-04-02 14:30 UTC)

  ? Confirm rollback to 1.5.0? This will restore the database to pre-upgrade state. (Y/n) Y

  ✔ Stopping services...
  ✔ Restoring database from backup...
  ✔ Switching images to 1.5.0 tags...
  ✔ Starting services...
  ✔ Health checks passing.

  Rollback complete: 1.6.0 → 1.5.0

Rollback mechanics:

Stop all services
Restore the database from the pre-upgrade pg_dump
Update docker-compose.yml image tags to the previous version
Start all services
Verify health

`accessible-server backup`

Creates a point-in-time backup of database, object storage metadata, and configuration.

$ accessible-server backup

  Creating backup...
  ✔ PostgreSQL dump (2.3 GB compressed)
  ✔ Configuration snapshot
  ✔ MinIO metadata export (object list, not file contents)
  ✔ Backup saved: /opt/accessible/backups/backup-20260402-143000.tar.gz (2.4 GB)

  Retention: 14 backups kept (oldest: 2026-03-19)

What’s backed up:

PostgreSQL: full pg_dump (custom format, compressed)
Config: config.yml, secrets (encrypted), Caddyfile, docker-compose.yml
MinIO metadata: bucket listing + object manifest (not the actual files — those are too large for routine backup; customers use MinIO’s built-in replication or mount the data volume on backed-up storage)
Version manifest: current image tags, migration state

Automated backups:

accessible-server init installs a cron job:

0 2 * * * /usr/local/bin/accessible-server backup --quiet

Retention is configurable (default: 14 backups).

`accessible-server restore <backup-file>`

Restores from a backup archive.

$ accessible-server restore /opt/accessible/backups/backup-20260402-143000.tar.gz

  ⚠ This will replace the current database and configuration.
  ? Confirm restore? (Y/n) Y

  ✔ Stopping services...
  ✔ Restoring PostgreSQL...
  ✔ Restoring configuration...
  ✔ Starting services...
  ✔ Health checks passing.

  Restore complete from backup-20260402-143000.

`accessible-server logs [service]`

Tail logs from one or all services.

$ accessible-server logs                  # All services
$ accessible-server logs api              # API nodes only
$ accessible-server logs worker           # Batch worker
$ accessible-server logs --since 1h       # Last hour
$ accessible-server logs --grep "error"   # Filter

Wraps docker compose logs with friendlier service aliases.

`accessible-server config`

View or update configuration.

$ accessible-server config show           # Print current config (secrets redacted)
$ accessible-server config set hostname accessible2.university.edu
$ accessible-server config set workers 4  # Scale batch workers
$ accessible-server config set tls.cert /path/to/cert.pem
$ accessible-server config set tls.key /path/to/key.pem

Changes that require a restart prompt the user:

  ✔ Configuration updated.
  ⚠ This change requires a restart. Run 'accessible-server restart' to apply.

`accessible-server secrets`

Manage API keys and credentials without editing files directly.

$ accessible-server secrets set ANTHROPIC_API_KEY
  ? Enter value: sk-ant-••••••••
  ✔ Secret stored (encrypted at rest).
  ⚠ Restart required to apply.

$ accessible-server secrets list
  ANTHROPIC_API_KEY     ••••••nt-a3Xk   set 2026-03-15
  GEMINI_API_KEY        ••••••Bx9m       set 2026-03-15
  MATHPIX_APP_KEY       (not set)
  POSTGRES_PASSWORD     ••••••••         set 2026-03-01
  MINIO_ROOT_PASSWORD   ••••••••         set 2026-03-01
  JWT_SECRET            ••••••••         set 2026-03-01
  REGISTRY_TOKEN        ••••••gh_x       set 2026-03-01

$ accessible-server secrets rotate JWT_SECRET
  ⚠ Rotating the JWT secret will invalidate all active sessions.
  ? Confirm? (Y/n) Y
  ✔ New JWT secret generated and stored. Restart required.

`accessible-server doctor`

Diagnoses common issues and suggests fixes.

$ accessible-server doctor

  Running diagnostics...

  ✔ Docker version 27.1.0 (minimum: 24.0)
  ✔ Docker Compose v2.29.1
  ✔ Available disk: 423 GB (minimum: 50 GB)
  ✔ Available RAM: 14.2 GB (minimum: 8 GB)
  ✔ All ports available (443, 8800, 5432, 9000, 6379)
  ✔ Container registry reachable (ghcr.io)
  ✔ DNS resolves accessible.university.edu → 10.0.1.50
  ✔ TLS certificate valid (89 days remaining)
  ⚠ Anthropic API key: connection test failed (401 Unauthorized)
    → Run 'accessible-server secrets set ANTHROPIC_API_KEY' to update
  ✔ Gemini API key: connection test passed
  ✔ PostgreSQL accepting connections
  ✔ MinIO accepting connections
  ✔ Redis accepting connections
  ✔ All API health checks passing

  12/13 checks passed. 1 warning.

Configuration System

Config File: `/opt/accessible/config.yml`

# TheAccessible.Org On-Premises Configuration
version: "1"
license_key: "XXXX-XXXX-XXXX-XXXX"

server:
  hostname: accessible.university.edu
  data_dir: /opt/accessible/data
  backup_dir: /opt/accessible/backups
  backup_retention: 14  # number of backups to keep
  log_level: info        # debug, info, warn, error

tls:
  mode: letsencrypt      # letsencrypt, custom, selfsigned
  email: admin@university.edu  # for Let's Encrypt notifications
  # cert: /path/to/cert.pem   # for custom mode
  # key: /path/to/key.pem     # for custom mode

services:
  api_replicas: 2        # number of API node containers
  worker_replicas: 1     # number of batch worker containers
  marker_enabled: true   # local Marker for PDF parsing
  marker_gpu: false      # enable GPU acceleration for Marker

ai:
  # Which AI providers are enabled (at least one vision LLM required)
  anthropic_enabled: true
  gemini_enabled: true
  openai_enabled: false
  mathpix_enabled: false    # disabled by default for FERPA

registry:
  url: ghcr.io/anglinai
  channel: stable          # stable, edge

database:
  max_connections: 100
  shared_buffers: 256MB    # auto-tuned by CLI based on available RAM

storage:
  minio_data_dir: /opt/accessible/data/minio

monitoring:
  grafana_enabled: true
  grafana_anonymous: false  # require login to view dashboards
  alert_email: admin@university.edu

Secrets File: `/opt/accessible/.secrets`

Encrypted at rest using AES-256-GCM. The encryption key is derived from a machine-specific fingerprint (machine-id + install timestamp) using PBKDF2.

The CLI decrypts secrets in memory when generating the Docker Compose environment, then passes them via Docker secrets or env_file (never written to disk in plaintext except in the ephemeral .env consumed by Compose).

Secrets Management

Encryption at Rest

All secrets (API keys, database passwords, JWT secret) are stored in an encrypted file. The decryption key is derived from:

/etc/machine-id (unique per Linux install)
Install timestamp (written during init)
PBKDF2 with 100,000 iterations

This means the secrets file is useless if copied to another machine. For backup/restore across machines, the CLI includes accessible-server secrets export (password-protected archive) and accessible-server secrets import.

Secret Rotation

accessible-server secrets rotate POSTGRES_PASSWORD
accessible-server secrets rotate JWT_SECRET
accessible-server secrets rotate MINIO_ROOT_PASSWORD

Each rotation command:

Generates a new cryptographically random value
Updates the encrypted secrets file
Prompts for a restart to apply
For JWT_SECRET, warns that all active sessions will be invalidated

Upgrade and Rollback Strategy

Version Channels

Channel	Tag Pattern	Use Case
`stable`	`1.5.0`	Production deployments (default)
`edge`	`1.6.0-rc.1`	Pre-release testing

Upgrade Flow (Detail)

accessible-server upgrade
   │
   ├─ 1. Query registry for latest stable tag
   ├─ 2. Compare with current version (semver)
   ├─ 3. If newer: fetch changelog, display to user
   ├─ 4. User confirms
   ├─ 5. Create pre-upgrade backup
   │     ├─ pg_dump → backup archive
   │     ├─ config.yml + .secrets snapshot
   │     └─ Record current image tags in manifest
   ├─ 6. docker compose pull (new images)
   ├─ 7. Extract migrations from new API image
   │     └─ docker run --rm api cat /app/supabase/migrations/ > /tmp/migrations/
   ├─ 8. Diff against schema_migrations table
   │     └─ Run only new migrations in order
   ├─ 9. Rolling restart
   │     ├─ Update api-node-1 → wait for healthy
   │     ├─ Update api-node-2 → wait for healthy
   │     └─ Update worker, marker, etc.
   ├─ 10. Post-upgrade health check (all services)
   │      ├─ If healthy: ✔ upgrade complete
   │      └─ If unhealthy: automatic rollback
   └─ 11. Record upgrade in /opt/accessible/data/upgrade-history.json

Rollback Flow

The CLI keeps the previous version’s images cached locally (Docker image layers). Rollback:

Stops all services
Restores the pre-upgrade database backup
Reverts docker-compose.yml image tags to previous version
Starts all services
Verifies health

Limitation: Rollback restores the database to pre-upgrade state. Any data written between upgrade and rollback is lost. The CLI warns about this explicitly.

Backup and Restore

What’s Backed Up

Component	Method	Included in Routine Backup
PostgreSQL	`pg_dump --format=custom`	Yes
Config	File copy	Yes
Secrets	Encrypted copy	Yes
MinIO metadata	Object manifest (bucket, key, size, etag)	Yes
MinIO objects	Not included (too large)	No — use MinIO replication or volume-level backup
Docker images	Not included (re-pullable)	No
Grafana dashboards	Pre-provisioned (baked into image)	No

MinIO Object Backup

For full object backup, customers have three options:

Volume-level backup: Mount /opt/accessible/data/minio on a filesystem with snapshots (ZFS, LVM, or a SAN)
MinIO site replication: Mirror to a second MinIO instance on another server
Rclone: Schedule rclone sync to an external S3 bucket, NAS, or cloud storage

The CLI does not manage MinIO object backup directly because document volumes vary enormously (100 MB to 10+ TB). The admin guide includes setup instructions for all three options.

Backup Storage

Default: local filesystem at /opt/accessible/backups/. Customers can configure an external target:

backup:
  target: s3           # local, s3, nfs
  s3_endpoint: https://s3.university.edu
  s3_bucket: accessible-backups
  s3_access_key: (via secrets)
  s3_secret_key: (via secrets)

Health Monitoring

Built-in Health Checks

Every container has a Docker HEALTHCHECK. The CLI polls these and exposes a unified status via accessible-server status.

Service	Health Endpoint	Check Interval	Unhealthy After
PostgreSQL	`pg_isready`	10s	3 failures
GoTrue	`GET /health`	10s	3 failures
PostgREST	`GET /` (200 OK)	10s	3 failures
Redis	`redis-cli ping`	10s	3 failures
MinIO	`GET /minio/health/live`	10s	3 failures
API Node	`GET /health`	5s	3 failures
Batch Worker	`GET /health`	10s	3 failures
WeasyPrint	`GET /health`	30s	3 failures
Audiveris	`GET /health`	30s	3 failures
Marker	`GET /health`	30s	3 failures
Caddy	TCP :443	10s	3 failures

Grafana Dashboards (Pre-configured)

The on-prem Grafana instance ships with pre-provisioned dashboards:

System Overview: service health, CPU, RAM, disk usage
API Performance: request rate, latency p50/p95/p99, error rate
Conversion Pipeline: queue depth, processing time, success/failure rate
Storage: MinIO usage, PostgreSQL size, Redis memory

Alert Rules

Alert	Condition	Action
Service down	Any container unhealthy for >2 min	Email to `alert_email`
High error rate	>10 API errors/min for 5 min	Email
Disk usage >85%	Checked every 5 min	Email
TLS cert expiring	<14 days remaining	Email (daily until renewed)
Backup failed	Cron backup exit code != 0	Email

Alerts are sent via the bundled Grafana alerting engine. SMTP is configured during init:

monitoring:
  smtp_host: smtp.university.edu
  smtp_port: 587
  smtp_from: accessible-alerts@university.edu
  alert_email: admin@university.edu

VM Appliance (OVA)

For customers who want a true zero-touch appliance — no Docker knowledge, no terminal comfort required.

What It Is

A pre-baked Ubuntu 22.04 LTS virtual machine image containing:

Docker Engine + Docker Compose pre-installed
accessible-server CLI pre-installed
All container images pre-pulled (for the current release)
A first-boot setup wizard (web-based)
Unattended security updates enabled (Ubuntu unattended-upgrades)

The customer downloads the OVA (or QCOW2/VMDK), imports it into their hypervisor, boots it, and completes setup through a browser-based wizard.

Supported Hypervisors

Format	Hypervisor	Notes
OVA	VMware ESXi, vSphere, Workstation	Primary target for enterprise
QCOW2	Proxmox, KVM, OpenStack	Common in university IT
VMDK	VMware, VirtualBox	Compatibility format
VHD/VHDX	Hyper-V	Windows Server environments

Build Process

The VM image is built with Packer (HashiCorp) using an Ubuntu 22.04 base:

source "qemu" "accessible" {
  iso_url          = "https://releases.ubuntu.com/22.04/ubuntu-22.04.5-live-server-amd64.iso"
  iso_checksum     = "sha256:..."
  vm_name          = "accessible-server-1.5.0"
  disk_size        = "100G"
  memory           = 8192
  cpus             = 4
  ssh_username     = "accessible"
  ssh_password     = "changeme"  # forced reset on first boot
  shutdown_command  = "sudo shutdown -P now"
}

build {
  sources = ["source.qemu.accessible"]

  # Base system
  provisioner "shell" {
    scripts = [
      "packer/scripts/01-base-packages.sh",   # apt update, essential tools
      "packer/scripts/02-docker.sh",           # Docker Engine + Compose
      "packer/scripts/03-node.sh",             # Node.js 22 LTS
      "packer/scripts/04-cli.sh",              # npm install -g @anglinai/accessible-server
      "packer/scripts/05-pull-images.sh",      # docker pull all images for current version
      "packer/scripts/06-first-boot.sh",       # systemd service for setup wizard
      "packer/scripts/07-security.sh",         # UFW, fail2ban, unattended-upgrades
      "packer/scripts/08-cleanup.sh",          # apt clean, zero free space for compression
    ]
  }

  # Export formats
  post-processor "vagrant" {}  # optional

  post-processor "shell-local" {
    inline = [
      # Convert QCOW2 to OVA, VMDK, VHD
      "qemu-img convert -O vmdk output/accessible-server-1.5.0 accessible-server-1.5.0.vmdk",
      "# OVA packaging with ovftool or manual tar",
    ]
  }
}

Packer Provisioning Scripts

`01-base-packages.sh`

#!/bin/bash
set -euo pipefail
apt-get update && apt-get upgrade -y
apt-get install -y \
  curl wget git jq htop net-tools \
  ca-certificates gnupg lsb-release \
  ufw fail2ban unattended-upgrades

`02-docker.sh`

#!/bin/bash
set -euo pipefail
# Docker official install
curl -fsSL https://get.docker.com | sh
usermod -aG docker accessible
systemctl enable docker

`04-cli.sh`

#!/bin/bash
set -euo pipefail
npm install -g @anglinai/accessible-server

`05-pull-images.sh`

#!/bin/bash
set -euo pipefail
VERSION="1.5.0"
IMAGES=(
  "ghcr.io/anglinai/accessible-api:${VERSION}"
  "ghcr.io/anglinai/accessible-worker:${VERSION}"
  "ghcr.io/anglinai/accessible-frontend:${VERSION}"
  "ghcr.io/anglinai/accessible-weasyprint:${VERSION}"
  "ghcr.io/anglinai/accessible-audiveris:${VERSION}"
  "ghcr.io/anglinai/accessible-marker:${VERSION}"
  "traefik:v3"
  "caddy:2"
  "postgres:15"
  "supabase/gotrue:v2.158.1"
  "postgrest/postgrest:v12.2.3"
  "minio/minio:latest"
  "redis:7-alpine"
  "grafana/grafana:11.5.2"
  "grafana/loki:3.4.2"
  "grafana/promtail:3.4.2"
)
for img in "${IMAGES[@]}"; do
  docker pull "$img"
done

`06-first-boot.sh`

#!/bin/bash
set -euo pipefail
# Create a systemd service that runs a web-based setup wizard on port 8080
# on the first boot. After setup completes, the service disables itself.
cat > /etc/systemd/system/accessible-setup.service <<EOF
[Unit]
Description=TheAccessible.Org First-Boot Setup
After=network-online.target docker.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/accessible-server setup-wizard --port 8080
ExecStartPost=/bin/systemctl disable accessible-setup.service
Restart=on-failure
User=accessible

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable accessible-setup.service

First-Boot Web Wizard

When the VM boots for the first time, a lightweight web server on port 8080 serves a setup wizard. The customer navigates to http://<vm-ip>:8080 and completes the same configuration as accessible-server init, but through a browser UI.

The wizard:

Prompts for license key, hostname, TLS config, credentials, AI API keys
Writes config.yml and .secrets
Generates the Docker Compose file and Caddyfile
Starts the stack
Redirects the browser to the live application at https://hostname
Disables the setup wizard service (one-time only)

VM Sizing Recommendations

Tier	vCPUs	RAM	Disk	Concurrent Users
Small	4	8 GB	100 GB	Up to 25
Medium	8	16 GB	250 GB	Up to 100
Large	16	32 GB	500 GB	Up to 500

The CLI auto-tunes PostgreSQL (shared_buffers, work_mem, max_connections) and API node count based on detected RAM.

VM Update Path

Updates on the VM use the same accessible-server upgrade flow. The VM connects to the container registry (if internet-connected) and pulls new images. For air-gapped environments, the customer can:

Download an update bundle (tar of Docker images) from a customer portal
Load images: accessible-server upgrade --from-file update-1.6.0.tar.gz

The update bundle is built by CI:

# Export all images for a release
docker save ghcr.io/anglinai/accessible-api:1.6.0 \
  ghcr.io/anglinai/accessible-worker:1.6.0 \
  ... | gzip > accessible-update-1.6.0.tar.gz

Networking and TLS

Default: Caddy with Automatic TLS

Caddy handles TLS termination with automatic Let’s Encrypt certificates. The customer only needs:

A public DNS record pointing their hostname to the server’s IP
Port 443 open inbound
Port 80 open inbound (for ACME challenge)

Custom Certificates

For customers behind a corporate firewall or using internal CAs:

accessible-server config set tls.mode custom
accessible-server config set tls.cert /path/to/cert.pem
accessible-server config set tls.key /path/to/key.pem
accessible-server restart

Firewall Rules

The CLI configures UFW during init:

Port	Protocol	Direction	Purpose
443	TCP	Inbound	HTTPS (application + API)
80	TCP	Inbound	HTTP → HTTPS redirect + ACME
22	TCP	Inbound	SSH (management)
9001	TCP	Localhost only	MinIO console
3000	TCP	Localhost only	Grafana

All other ports are internal to the Docker network and not exposed to the host.

FERPA Compliance Notes

The on-prem deployment is the strongest FERPA compliance posture because all data stays within the customer’s infrastructure. Key points:

No sub-processor risk for data-at-rest: Documents, database, and storage never leave the server.
Marker runs locally: No documents sent to Datalab’s hosted API. See ferpa-subprocessor-compliance.md for details.
AI API keys are optional: For fully air-gapped deployments, all AI providers can be disabled. The conversion pipeline degrades gracefully — OCR and image descriptions are skipped, but structural HTML conversion still works.
AI providers in the on-prem data path: If AI API keys are configured, document content is sent to those providers for processing. The customer controls which providers are enabled. Recommended for FERPA: Gemini via Vertex AI in the customer’s own GCP project (see ferpa-subprocessor-compliance.md Section: Google Gemini API).
No telemetry, no phone-home: The on-prem stack does not send usage data, analytics, or crash reports to AnglinAI. The only outbound connections are: container registry (for updates), AI API endpoints (if configured), SMTP (if configured), and Let’s Encrypt (if auto-TLS is used).
Data residency: All data stays on the customer’s server. No CDN, no edge caching, no Cloudflare proxy.

Air-Gapped Deployment

For maximum isolation (FISMA, classified environments):

Use the VM appliance (pre-baked images, no registry pull needed)
Disable all AI API keys (conversion runs without OCR/descriptions)
Use custom TLS certificates (no Let’s Encrypt)
Disable Grafana SMTP alerts (or use an internal SMTP relay)
Updates via offline bundle: accessible-server upgrade --from-file

Sizing Guidelines

Minimum Requirements

Resource	Minimum	Recommended
CPU	4 cores	8+ cores
RAM	8 GB	16+ GB
Disk	50 GB (OS + application)	100+ GB (plus document storage)
Docker	24.0+	Latest stable
OS	Ubuntu 22.04 LTS	Ubuntu 22.04 or 24.04 LTS

Scaling

Dimension	How to Scale	CLI Command
API throughput	Add API node replicas	`accessible-server config set services.api_replicas 4`
Processing throughput	Add worker replicas	`accessible-server config set services.worker_replicas 3`
Storage	Expand MinIO data volume or add drives	Mount additional storage at `storage.minio_data_dir`
Database	Tune PostgreSQL or migrate to external RDS	Edit `config.yml` database section
Marker (PDF parsing)	Enable GPU acceleration	`accessible-server config set services.marker_gpu true`

File and Directory Layout

Installation Directory: `/opt/accessible/`

/opt/accessible/
├── config.yml                    # Main configuration
├── .secrets                      # Encrypted secrets
├── docker-compose.yml            # Generated Compose file (do not edit directly)
├── Caddyfile                     # Generated reverse proxy config
├── upgrade-history.json          # Record of all upgrades/rollbacks
├── data/                         # Persistent data volumes
│   ├── postgres/                 # PostgreSQL data directory
│   ├── minio/                    # MinIO object storage
│   ├── redis/                    # Redis AOF persistence
│   ├── grafana/                  # Grafana database
│   └── loki/                     # Log storage
├── backups/                      # Backup archives
│   ├── backup-20260402-020000.tar.gz
│   ├── backup-20260401-020000.tar.gz
│   └── pre-upgrade-1.5.0-20260402.tar.gz
└── logs/                         # CLI operation logs
    └── accessible-server.log

Implementation Roadmap

Phase 1: Docker Compose On-Prem Stack (2-3 weeks)

Goal: A working docker-compose.onprem.yml that replaces all cloud dependencies with local services.

Task	Effort	Status
Add Redis container to Compose	1 day	Not started
Implement `RedisKvStorage` (same interface as `CloudflareKvRestStorage`)	2 days	Not started
Implement `RedisQueue` to replace SQS polling in batch worker	2 days	Not started
Add local Marker container to Compose	1 day	Not started
Add Marker local provider to conversion cascade	1 day	Not started
Add Caddy container for TLS termination	0.5 days	Not started
Add Next.js static export container	1 day	Not started
`ON_PREM=true` env flag to select local providers	1 day	Not started
End-to-end test: full conversion pipeline on local-only stack	2 days	Not started

Prerequisite: The existing local Docker Compose profile already runs Supabase, MinIO, LocalStack, and monitoring. Phase 1 replaces LocalStack (AWS emulation) with native Redis, adds Marker, and adds Caddy.

Phase 2: Management CLI (2-3 weeks)

Goal: Ship @anglinai/accessible-server as an npm package with core commands.

Task	Effort	Status
CLI scaffolding (commander, project structure)	1 day	Not started
`init` command (interactive wizard, config generation)	3 days	Not started
`start` / `stop` / `restart` commands	1 day	Not started
`status` command (health dashboard)	1 day	Not started
`upgrade` command (registry check, pull, migrate, rolling restart)	3 days	Not started
`rollback` command	1 day	Not started
`backup` / `restore` commands	2 days	Not started
`logs` command	0.5 days	Not started
`config` / `secrets` commands	1 day	Not started
`doctor` command (diagnostics)	1 day	Not started
Secrets encryption at rest	1 day	Not started
Test suite for CLI commands	2 days	Not started

Phase 3: Container Registry and CI/CD (1 week)

Goal: Automated image publishing on tagged releases.

Task	Effort	Status
GitHub Actions workflow for building + pushing images	1 day	Not started
Image versioning (semver tags + date stamps)	0.5 days	Not started
Changelog generation (from commits/PRs)	0.5 days	Not started
Air-gapped update bundle generation	1 day	Not started
Customer registry token provisioning	0.5 days	Not started

Phase 4: VM Appliance (1-2 weeks)

Goal: A downloadable OVA/QCOW2 with the full stack pre-baked.

Task	Effort	Status
Packer template for Ubuntu 22.04	2 days	Not started
Provisioning scripts (Docker, Node, CLI, images)	1 day	Not started
First-boot web wizard	3 days	Not started
OVA/VMDK/QCOW2/VHD export pipeline	1 day	Not started
Security hardening (UFW, fail2ban, unattended-upgrades)	0.5 days	Not started
VM build automation in CI	1 day	Not started
Testing on VMware, Proxmox, Hyper-V	2 days	Not started

Phase 5: Documentation and Onboarding (1 week)

Goal: Customer-facing documentation and onboarding materials.

Task	Effort	Status
Customer deployment guide (Docusaurus)	2 days	Not started
Air-gapped deployment addendum	1 day	Not started
CLI reference documentation	1 day	Not started
Video walkthrough (setup to first conversion)	1 day	Not started

Total Estimated Effort

Phase	Effort
Phase 1: Docker Compose stack	2-3 weeks
Phase 2: Management CLI	2-3 weeks
Phase 3: Registry + CI/CD	1 week
Phase 4: VM Appliance	1-2 weeks
Phase 5: Documentation	1 week
Total	~7-10 weeks (one developer)

Phases 1 and 2 can partially overlap — the CLI can be developed against the existing local Compose profile while the on-prem-specific Compose changes are in progress.