PacketStation/docs/DEPLOY.md

Deploying Packet Station

How Packet Station runs on docker-1. Sections not built yet are marked "planned, phase N". The design is in PLAN.md under "Deployment and CI".

Host facts

• Host: docker-1, 192.168.101.12.
• Ubuntu 24.04 LTS, Docker 29, Compose v5. Clock in UTC.
• Nginx Proxy Manager already runs there as container npm_app_1 on network npm_default (172.22.0.0/16). Its compose project is in /home/spencerm/npm.
• The host also runs Gitea (gt.pescoinc.biz, the container registry and the Python index) and pesco-ncr.
• Proxmox Backup Server backs up the whole VM at 03:00 daily.
• Host patching and hardening are IT's (requirement H5).
• Inside containers only full names resolve. Docker's resolver has no search domain, so every host setting is an FQDN (for example vsql.pesco.com\DEV, app-03.pesco.com).

The two stacks

Both stacks run from the same image as separate compose projects.

Stack	Compose project	DNS name	Network alias on npm_default	VISUAL
Test	packetstation-test	packetstation-test.pescoinc.biz	packetstation-test-app	DEV: vsql.pesco.com\DEV / PESCOINCDEV
Production	packetstation	packetstation.pescoinc.biz	packetstation-app	vsql / PESCOINC, set explicitly (planned, phase 10)

• The app publishes no port. It joins npm_default as an external network with an explicit alias per stack (APP_ALIAS). Plain service names would collide, since both stacks share that network.
• PostgreSQL stays on each stack's own internal backend network, never on npm_default.
• The test stack never points at VISUAL production. The production stack sets VISUAL_SERVER and VISUAL_DATABASE explicitly. Spencer starting it with those values is the one-time approval for its scheduled polling.
• There is no M-Files test vault. Both stacks read the production vault, read-only, as mfileskitRO.
• The internal A records for both names point at 192.168.101.12 in the AD-integrated pescoinc.biz zone.

Services

• postgres: PostgreSQL 16, named volume, backend network only. Its init creates the database roles from secrets.
• migrate: one-shot, runs packetstation migrate as the migrate role.
• app: packetstation app (uvicorn). backend network plus egress plus npm_default through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
• worker: packetstation worker. Same image. Assets volume, backup folder read-only (planned, phase 9).
• All services restart unless stopped, log to stdout (json-file), and have healthchecks.

Secrets

Docker secrets, one file each, under /etc/packetstation/<stack>/secrets/. Owner root:10001 (10001 is the image's non-root user), mode 0440.

Secret	Used by
postgres_superuser_password	PostgreSQL bootstrap
postgres_migrate_password	migrate role (ps_migrate)
postgres_app_password	app role (ps_app)
visualkit_password	VISUAL_PASSWORD_FILE for visualkit_ro
mfileskit_password	mfileskitRO; the entrypoint exports it as MFILES_PASSWORD
ldap_bind_password	LDAP_PASSWORD_FILE for svc-packetstation
session_key	session signing
seal_key	HMAC filing seals and controlled-copy stamps

Never put a secret in the repo, the env file, a log or chat. Settings read secrets through *_PASSWORD_FILE paths.

The PESCO CA chain is mounted into the containers once and referenced by LDAP_CA_BUNDLE, MFILES_CA_BUNDLE and SMTP_CA_BUNDLE.

Env files

One folder per stack under /etc/packetstation/: /etc/packetstation/test/ and /etc/packetstation/prod/. Each holds stack.env (settings only, no secrets) and secrets/. The env file sets PROJECT_NAME, SECRETS_DIR and APP_ALIAS, which the compose files require. .env.example in the repo lists every setting.

Key settings:

• VISUAL_SERVER, VISUAL_DATABASE, VISUAL_USER=visualkit_ro, VISUAL_PASSWORD_FILE.
• MFILES_URL=https://mfiles.pescoinc.biz/REST, MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}, MFILES_USERNAME=mfileskitRO, MFILES_CA_BUNDLE.
• LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636, LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com, LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com, LDAP_PASSWORD_FILE, LDAP_CA_BUNDLE.
• SMTP_HOST=app-03.pesco.com, SMTP_PORT=25, SMTP_STARTTLS=true, [email protected], SMTP_CA_BUNDLE.
• APP_ALIAS per stack.
• Identity timing and limits, all optional with these defaults: SESSION_IDLE_MINUTES=15, SESSION_LOCK_MINUTES=3, SESSION_HIDDEN_LOCK_SECONDS=60, SESSION_PASSWORD_HOURS=12, SESSION_PIN_HOURS=10, ROLES_REFRESH_MINUTES=15, SIGNIN_ACCOUNT_LIMIT=5, SIGNIN_ACCOUNT_WINDOW_MINUTES=5, SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10, SIGNIN_IP_LIMIT=30, SIGNIN_IP_WINDOW_MINUTES=5, PIN_MAX_FAILURES=5, STEPUP_MAX_FAILURES=3, DEVICE_COOKIE_DAYS=365, PIN_MIN_DIGITS=6, PIN_MAX_DIGITS=12.
• The session cookie is marked Secure when the request arrived over https (as reported by NPM through X-Forwarded-Proto). Behind the proxy that is always the case.

Deploy command

Test stack:

docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml up -d

Production (planned, phase 10): the same command with -p packetstation and the production env file. The first production start is Spencer's approval.

Images come from gt.pescoinc.biz by tag. CI tags each tested image :sha-<sha>; a release re-tags it vX.Y.Z. Never latest. A deploy is a new image with a short downtime window.

Python packages and the Gitea index

The Gitea PyPI index https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/ serves only visualkit, mfileskit and their dependencies.

• Install visualkit and mfileskit from it with --index-url. Never --extra-index-url, which would let a public package with the same name win.
• Install everything else from public PyPI in a separate pip step.
• The image needs Microsoft ODBC Driver 18 (msodbcsql18) from Microsoft's Debian 12 repository for VisualKit.

NPM proxy host

One proxy host per stack in Nginx Proxy Manager. To be recorded under deploy/ as well.

• Domain: packetstation-test.pescoinc.biz (test) or packetstation.pescoinc.biz (production).
• Scheme http, forward host the stack's alias (packetstation-test-app or packetstation-app), port 8000.
• SSL: the stack's certificate, Force SSL on, HTTP/2 on.
• NPM HSTS off. The app sends HSTS itself.
• Advanced config: client_max_body_size 30m; plus a location block for the event stream:

client_max_body_size 30m;

location /api/events {
    proxy_pass $forward_scheme://$server:$port;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

The block uses NPM's $server variables so the alias is re-resolved. The app also sends X-Accel-Buffering: no on /api/events.

Open check (plan question 16): confirm "Block common exploits" does not interfere with uploads when the test proxy host is made.

The app trusts X-Forwarded-* only from npm_app_1's own address, re-resolved every 60 seconds. Other stacks share npm_default, and the per-address rate limit depends on the real client address.

Doctor

packetstation doctor proves a host and a stack are ready. Run it inside the image:

docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm app doctor

It checks:

• Settings and the stack guard (test refuses a production VISUAL target).
• visualkit doctor and mfileskit doctor. This is the first real Linux test of both libraries.
• Database connection, roles, migrations at head, and an audit UPDATE that must fail.
• LISTEN/NOTIFY round trip.
• Assets volume and free space. Backup folder (planned, phase 9).
• LDAP TLS bind on both controllers, reading memberOf, employeeID and userAccountControl as the service account, the 50-group count and one nested-group query.
• SMTP STARTTLS with certificate verification.
• tzdata.

Use --skip visual,mfiles,ldap,smtp to run only the local checks. The network checks need the secrets and the CA bundle in place.

Verified on docker-1 (2026-10-15, phase 0)

The image built on docker-1 from this tree (python:3.12-slim-bookworm, ODBC Driver 18, visualkit 0.3.0, mfileskit 0.1.2). The CI stack (deploy/compose.ci.yaml with throwaway secrets) came up with postgres, migrate, app and worker all healthy. Inside it, doctor --skip visual,mfiles,ldap,smtp passed: migrations at head, the audit UPDATE denied for ps_app, LISTEN/NOTIFY round trip, assets volume, tzdata. visualkit doctor in the image found the ODBC driver and pyodbc on Linux; the connection itself was not tried because the CI stack has no VISUAL credentials. mfileskit doctor imported and ran on Linux up to its settings check for the same reason. The full doctor with the real secrets, the CA bundle and both domain controllers is the first step of the test stack deployment.

Inside containers the short name VSQL\DEV does not resolve. The env files use vsql.pesco.com\DEV.

CI stack and the Samba domain controller (phase 1)

deploy/compose.ci.yaml adds a throwaway Samba Active Directory domain controller (samba-dc, realm PSTEST.LOCAL, host dc1.pstest.local on the internal backend network) built from deploy/samba/. On first start it provisions the domain, generates a CA and a server certificate, seeds the OU tree, the service account, the groups (49 of the 50, plus one extra, to test the report) and 17 test users with random passwords, and exports ca.pem and users.json to the ci-dc-export volume. The app, migrate and worker mount that volume at /etc/ssl/pstest and deploy/ci/ci.env points LDAP_CA_BUNDLE at it, so CI exercises the real LDAPS path. The playwright service (profile e2e) drives the app at http://ps-app:8000 with the exported users. deploy/samba/README.md has the details and the commands.

The DC runs with cap_add: [SYS_ADMIN], which Samba needs to set the sysvol ACLs. The app carries the alias ps-app for Playwright because app is a top-level domain on Chromium's HSTS preload list, so http://app:8000 is forced to TLS and fails. Running the server test suite against the stack's own PostgreSQL is not safe: the pg_settings fixture resets the role passwords (see deploy/samba/README.md).

The real PESCO domain controllers are never used by CI. The test and production stacks do not run the DC.

Backup (planned, phase 9)

pg_dump of both stacks from a systemd timer on the host at 02:00 America/Denver, before the 03:00 PBS run. Seven dumps kept in /var/backups/packetstation/<stack>/ with last-success and last-failure markers. The worker's backup-check job alerts if the newest dump is missing or older than 26 hours. Scripts and units go in scripts/.

Restore (planned, phase 9)

Restore the dump into a fresh database and copy the assets back, from the live folder or a file-level PBS restore. Rehearsed once on the test stack and written up here. A whole-VM restore from PBS is the disaster path.