Files

T

spencermandClaude Opus 5.5 3fdf063b70 Bump pypdf 6.21.4, certifi 2026.10.23, pyasn1 0.6.5 for pip-audit findings

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-10-30 17:31:58 -06:00

42 KiB

Raw Blame History

Deploying Packet Station

How Packet Station runs on docker-1. Sections not built yet are marked "planned, phase N". The design is in PLAN.md under "Deployment and CI".

Host facts

Host: docker-1, 192.168.101.12.
Ubuntu 24.04 LTS, Docker 29, Compose v5. Clock in UTC.
Nginx Proxy Manager already runs there as container npm_app_1 on network npm_default (172.22.0.0/16). Its compose project is in /home/spencerm/npm.
The host also runs Gitea (gt.pescoinc.biz, the container registry and the Python index) and pesco-ncr.
Proxmox Backup Server backs up the whole VM at 03:00 daily.
Host patching and hardening are IT's (requirement H5).
Inside containers only full names resolve. Docker's resolver has no search domain, so every host setting is an FQDN (for example vsql.pesco.com\DEV, app-03.pesco.com).

The two stacks

Both stacks run from the same image as separate compose projects.

Stack	Compose project	DNS name	Network alias on npm_default	VISUAL
Test	`packetstation-test`	`packetstation-test.pescoinc.biz`	`packetstation-test-app`	DEV: `vsql.pesco.com\DEV` / `PESCOINCDEV`
Production	`packetstation`	`packetstation.pescoinc.biz`	`packetstation-app`	`vsql` / `PESCOINC`, set explicitly (planned, phase 10; moves to its own VM, see below)

The app publishes no port. It joins npm_default as an external network with an explicit alias per stack (APP_ALIAS). Plain service names would collide, since both stacks share that network.
PostgreSQL stays on each stack's own internal backend network, never on npm_default.
The test stack never points at VISUAL production. The production stack sets VISUAL_SERVER and VISUAL_DATABASE explicitly. Spencer starting it with those values is the one-time approval for its scheduled polling.
There is no M-Files test vault. Both stacks read the production vault, read-only, as mfileskitRO.
The internal A records for both names point at 192.168.101.12 in the AD-integrated pescoinc.biz zone.

Services

postgres: PostgreSQL 16, named volume, backend network only. Its init creates the database roles from secrets.
migrate: one-shot, runs packetstation migrate as the migrate role.
app: packetstation app (uvicorn). backend network plus egress plus npm_default through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
worker: packetstation worker. Same image. Runs the VISUAL feed, the drawing lookup, housekeeping and the phase 9 jobs: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications (see FEED.md). backend network plus egress (VISUAL, M-Files), no npm_default, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only at /backups (the backup-check job reads it). Its healthcheck reads the heartbeat file.
clamav (optional, profile clamav, phase 4): clamav/clamav:1.4, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias packetstation-clamav (as postgres has its own alias). backend network plus egress (freshclam downloads the signatures), no npm_default, no published port. Volume clamav-db at /var/lib/clamav, mem_limit: 2g, healthcheck clamdcheck.sh. Started only with --profile clamav. See "Virus scan (ClamAV)".
All services restart unless stopped, log to stdout (json-file), and have healthchecks.

Secrets

Docker secrets, one file each, in the stack's secrets/ folder ("Env files" says where each stack keeps it). Production: owner root:10001 (10001 is the image's non-root user), mode 0440. Test: owner spencerm, mode 0444, reachable only through /home/spencerm and packetstation/test, both mode 0750, so no other host user can read them.

Secret	Used by
`postgres_superuser_password`	PostgreSQL bootstrap
`postgres_migrate_password`	migrate role (`ps_migrate`)
`postgres_app_password`	app role (`ps_app`)
`visualkit_password`	`VISUAL_PASSWORD_FILE` for `visualkit_ro`
`mfileskit_password`	`mfileskitRO`; the entrypoint exports it as `MFILES_PASSWORD`
`ldap_bind_password`	`LDAP_PASSWORD_FILE` for `svc-packetstation`
`session_key`	session signing
`seal_key`	HMAC filing seals and controlled-copy stamps

Never put a secret in the repo, the env file, a log or chat. Settings read secrets through *_PASSWORD_FILE paths.

The worker gets postgres_app_password (it logs in as the app role, with application name ps-worker), visualkit_password, mfileskit_password, ldap_bind_password and seal_key. The worker never logs a password: job errors are redacted before they are stored, and the redaction also strips the VISUAL and M-Files passwords themselves.

The PESCO CA chain is mounted into the containers once and referenced by LDAP_CA_BUNDLE, MFILES_CA_BUNDLE and SMTP_CA_BUNDLE.

Env files

One folder per stack. The test stack: /home/spencerm/packetstation/test/, like the other compose stacks on docker-1 (decided 2026-10-18, Gitea issue 38). It holds stack.env, secrets/, ca/pesco-ca.pem and backups/, and its stack.env sets ENV_FILE, SECRETS_DIR, CA_BUNDLE_FILE and BACKUP_DIR to those paths. The compose files run from the checkout in /home/spencerm/packetstation/repo/deploy. On 2026-10-27 Spencer decided that production and test each run natively on their own Linux VM, without Docker, and that docker-1 keeps CI only (Gitea issue 37). The test stack stays here until the test VM is ready. The production lines in this file describe the old compose plan and are rewritten for the VMs in phase 10. Each folder holds stack.env (settings only, no secrets) and secrets/. The env file sets PROJECT_NAME, SECRETS_DIR and APP_ALIAS, which the compose files require. .env.example in the repo lists every setting.

Key settings:

VISUAL_SERVER, VISUAL_DATABASE, VISUAL_USER=visualkit_ro, VISUAL_PASSWORD_FILE.
MFILES_URL=https://mfiles.pescoinc.biz/REST, MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}, MFILES_USERNAME=mfileskitRO, MFILES_CA_BUNDLE.
LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636, LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com, LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com, LDAP_PASSWORD_FILE, LDAP_CA_BUNDLE.
SMTP_HOST=app-03.pesco.com, SMTP_PORT=25, SMTP_STARTTLS=true, [email protected], SMTP_CA_BUNDLE, NOTIFY_DRY_RUN (true on the test stack, false on production). STACK (test or prod) picks the subject prefix. See "Notifications".
BACKUP_DIR: the stack's backup folder on the host, mounted read-only in the worker. See "Backup".
APP_ALIAS per stack.
Identity timing and limits, all optional with these defaults (an Admin can override most of them within bounds on the Security tab; see SECURITY.md, "Security settings"): SESSION_IDLE_MINUTES=15, SESSION_LOCK_MINUTES=3, SESSION_DESK_MINUTES=60 (Admin and Clerical wait at least this long for both), SESSION_HIDDEN_LOCK_SECONDS=60, SESSION_PASSWORD_HOURS=12, SESSION_PIN_HOURS=10, ROLES_REFRESH_MINUTES=15, SIGNIN_ACCOUNT_LIMIT=5, SIGNIN_ACCOUNT_WINDOW_MINUTES=5, SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10, SIGNIN_IP_LIMIT=30, SIGNIN_IP_WINDOW_MINUTES=5, PIN_MAX_FAILURES=5, STEPUP_MAX_FAILURES=3, DEVICE_COOKIE_DAYS=365, PIN_MIN_DIGITS=6, PIN_MAX_DIGITS=12.
Worker sources, all optional: FEED_SOURCE (visualkit, the default, or fixture), DRAWING_SOURCE (mfiles, the default, or fixture) and FIXTURE_DIR (default /fixtures). A fixture source reads files instead of VISUAL or M-Files (see FEED.md, "Fixture sources"). STACK=prod refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
Uploads and storage, all optional: UPLOAD_MAX_BYTES (default 25 MB, drawings, MTRs and documents), UPLOAD_MAX_BYTES_BOM (default 5 MB, hand BOMs), ASSETS_QUOTA_WARN_PCT (default 85, 1 to 99) and ASSETS_QUOTA_BYTES (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
Virus scan, all optional: CLAMAV_HOST (empty by default, which means no scan; packetstation-clamav for the profile service), CLAMAV_PORT (default 3310) and CLAMAV_TIMEOUT_SECONDS (default 30). The env examples carry #CLAMAV_HOST=packetstation-clamav commented out. See "Virus scan (ClamAV)".
SITE_ID (default PESCO) and SITE_TZ (default America/Denver): the VISUAL site in every call, and the time zone for shop hours and the schedule.
The session cookie is marked Secure when the request arrived over https (as reported by NPM through X-Forwarded-Proto). Behind the proxy that is always the case.

Feed settings live in the database

The feed's schedule and rules are not env settings. Intervals, shop hours, the window, the go-live date, departments, the drawing recheck and budget, labor tickets, auto-release and the pause switch are one settings row (key feed), edited by an Admin on the Admin page, Feed tab. The resource map is the visual_resource_map table, edited on the "VISUAL resources" tab. A save notifies the worker, which applies it without a restart. A new stack starts with the defaults in FEED.md, "Settings". Set the go-live date before the production feed's first run.

Deploy command

Test stack:

docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml up -d

Production (planned, phase 10): the same command with -p packetstation and the production env file. The first production start is Spencer's approval.

Images come from gt.pescoinc.biz by tag. CI tags each tested image :sha-<sha>; a release re-tags it vX.Y.Z. Never latest. A deploy is a new image with a short downtime window.

Virus scan (ClamAV, phase 4)

ClamAV is optional and off by default. It closes requirement D5. IT decided on 2026-10-18 (Gitea issue 19) to leave it off in both stacks: every workstation that uploads runs Microsoft Defender. The steps below stay for the day that changes. Turn it on in the test stack first, check it, then the production stack.

Set CLAMAV_HOST=packetstation-clamav in the stack's env file (~/packetstation/test/stack.env). Leave CLAMAV_PORT and CLAMAV_TIMEOUT_SECONDS at their defaults.
Start the stack with the profile:

docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml --profile clamav up -d

Wait for clamav to turn healthy. On the first start the volume packetstation-test_clamav-db is empty, so freshclam downloads the full signature set (a few hundred MB) before clamd answers. That takes several minutes. Later starts reuse the volume and only fetch updates. freshclam needs egress to the ClamAV mirrors, which is why the service is on the egress network.
Check it: Admin, Storage tab, the ClamAV line reads "packetstation-clamav: reachable". packetstation doctor also has a clamav check (a PING to clamd; SKIP when CLAMAV_HOST is empty). Upload a small PDF to a draft and confirm it is stored.

Every later up for that stack must carry --profile clamav. Without it the service is not started.

Fail closed. While CLAMAV_HOST is set, an upload the scanner cannot check is refused with "The virus scanner is not reachable. Try again in a minute." (503 scan_unavailable). That covers clamd still loading signatures, stopped, or out of memory. The worker's drawing fetches are refused the same way and tried again later. A file with a signature hit is refused (422 infected), never stored, and audited as asset.rejected with the signature name.

To turn it off: remove CLAMAV_HOST from the env file, recreate app and worker (up -d without the profile), then stop the service with docker compose ... --profile clamav stop clamav. The volume can stay for the next time. The CI overlay never runs ClamAV.

clamd holds the signatures in memory, about 1.2 GB with the full set and more while it reloads them, so the service has a 2 GB limit. Check docker-1 has the headroom for both stacks before turning it on in production.

Storage quota (phase 4)

Every stored file and the housekeeping job check the assets volume. With ASSETS_QUOTA_BYTES=0 (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At ASSETS_QUOTA_WARN_PCT percent (85 by default) the app opens one notification_events alert of kind assets_quota and audits storage.quota_warning. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.

The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. The alert is mailed through the assets_quota notification rule, to PS-Admins by default (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold), with a "Cleared" message when usage drops back.

Python packages and the Gitea index

The Gitea PyPI index https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/ serves only visualkit, mfileskit and their dependencies.

Install visualkit and mfileskit from it with --index-url. Never --extra-index-url, which would let a public package with the same name win.
Install everything else from public PyPI in a separate pip step.
The image needs Microsoft ODBC Driver 18 (msodbcsql18) from Microsoft's Debian 12 repository for VisualKit.

NPM proxy host

One proxy host per stack in Nginx Proxy Manager. To be recorded under deploy/ as well.

Domain: packetstation-test.pescoinc.biz (test) or packetstation.pescoinc.biz (production).
Scheme http, forward host the stack's alias (packetstation-test-app or packetstation-app), port 8000.
SSL: the stack's certificate, Force SSL on, HTTP/2 on.
NPM HSTS off. The app sends HSTS itself.
Advanced config: client_max_body_size 30m; plus a location block for the event stream:

client_max_body_size 30m;

location /api/events {
    proxy_pass $forward_scheme://$server:$port;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

The block uses NPM's $server variables so the alias is re-resolved. The app also sends X-Accel-Buffering: no on /api/events.

Open check (plan question 16): confirm "Block common exploits" does not interfere with uploads when the test proxy host is made.

The app trusts X-Forwarded-* only from npm_app_1's own address, re-resolved every 60 seconds. Other stacks share npm_default, and the per-address rate limit depends on the real client address.

Known issue: the internal CA signs with SHA-1 (found 2026-10-17)

PESCO-PSYLOCKE-CA (valid to 2027-09-27) signs with sha1WithRSAEncryption, and so do the certificates it issued to DC-02.PESCO.COM and PDC-1.PESCO.COM on 636 and to APP-03.PESCO.COM on 25 (all valid to April 2027). OpenSSL 3 on Debian 12 refuses SHA-1 signatures at its default security level 2, so LDAPS and SMTP STARTTLS fail from inside the image with "CA signature digest algorithm too weak", while Windows and browsers still accept them. Verification passes at security level 0.

mfiles.pescoinc.biz uses a public Let's Encrypt certificate, so MFILES_CA_BUNDLE must stay unset; only LDAP_CA_BUNDLE and SMTP_CA_BUNDLE point at the PESCO chain.

Resolved 2026-10-17: the CA was switched to SHA-256 (certutil -setreg ca\csp\CNGHashAlgorithm SHA256, restart certsvc; the CA already used the Key Storage Provider) and the Kerberos Authentication, Domain Controller Authentication and Directory Email Replication certificates on DC-02 and PDC-1 and the Machine certificate on APP-03 were renewed with certreq -enroll -machine -q -cert <thumbprint> renew reusekeys. From docker-1 all three endpoints now verify against the PESCO bundle with sha256WithRSAEncryption. The CA certificate itself is still SHA-1 self-signed, which OpenSSL accepts for a trust anchor. If the problem comes back, the fix was: set the CA to issue with SHA-256 and reissue the domain controller and relay certificates. The CA certificate itself may stay SHA-1, since OpenSSL does not check the trust anchor's own signature. Until then the test stack cannot sign in or verify the relay without a test-only OpenSSL security-level override, which is Spencer's decision to apply, never the production stack's. The doctor should report a SHA-1 chain with this hint (follow-up).

The test stack on docker-1 lives under /home/spencerm/packetstation/test (stack.env, secrets/, ca/pesco-ca.pem, backups/). It stays there: that matches the other compose stacks on the host (Gitea issue 38, 2026-10-18).

Known issue: Docker subnet pool on docker-1 (found 2026-10-17)

Docker's default address pool on docker-1 holds fifteen /16 bridge networks (172.17.0.0 to 172.31.0.0). Each compose stack takes one or two (Packet Station takes two: backend and egress), the Gitea runner takes one per job, and the CI e2e stack needs two more. With the test stack up the pool ran out and every e2e job failed within seconds with "all predefined address pools have been fully subnetted". Idle networks of stopped stacks were removed to make room. The lasting fix is a larger pool in /etc/docker/daemon.json, for example {"default-address-pools":[{"base":"10.200.0.0/16","size":24}]}, which needs sudo and a Docker restart (a short outage for every stack on the host), or trimming unused compose projects. Production and test move to their own VMs (Gitea issue 37), so the pool will only have to fit the runner and the CI e2e stack. Count the networks before adding any other stack: docker network ls --filter driver=bridge | wc -l.

Known issue: vsql serves a weak self-signed certificate (found 2026-10-18)

vsql.pesco.com\DEV serves TLS (VISUAL_ENCRYPT=trust connects), but with VISUAL_ENCRYPT=yes OpenSSL in the image refuses it with "EE certificate key too weak": the server certificate's RSA key is under 2048 bits, which is what SQL Server's auto-generated self-signed certificate looks like. IT needs to issue vsql a 2048-bit, SHA-256 certificate from PESCO-PSYLOCKE-CA with vsql.pesco.com in the subject alternative names, install it on the DEV and production instances in SQL Server Configuration Manager and restart them. The image then needs the PESCO CA in its OpenSSL trust store for the ODBC driver (the entrypoint can concatenate the system bundle and /etc/ssl/pesco/ca.pem into /tmp and export SSL_CERT_FILE), and the stacks switch to VISUAL_ENCRYPT=yes. Tracked as Gitea issue 5.

Installing the vsql certificate

scripts/Request-VsqlServerCertificate.ps1 does the request, the checks and the binding. Spencer runs it on the SQL Server host, in an elevated PowerShell, as an account that may enroll from the WebServer template. Every step there is a real change, so preview first. Copy only the script to C:\ProgramData\PacketStation\ and work from there.

Preflight. Nothing changes. Check the instance names, the service accounts, the INF and the list of steps.

cd C:\ProgramData\PacketStation
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Instances DEV -WhatIf

DEV first. This enrolls, validates, grants the DEV service account read on the key, saves a snapshot and binds DEV. Production is untouched.
```
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Instances DEV
```
If the CA holds the request as pending, the script stops with the request id. Issue it on the CA (certsrv.msc, Pending Requests), then run the certreq -retrieve line it printed and continue with -CerFile <that .cer> -Instances DEV. If validation fails, nothing is bound. The usual cause is a template that forces a Key Storage Provider.
Restart DEV and check the log. Restart-Service -Name 'MSSQL$DEV' -Force, then start SQLAgent$DEV again if it ran. Or rerun with -Thumbprint <thumbprint> -Instances DEV -RestartInstances, which leaves the binding alone and does both. The instance's ERRORLOG should read The certificate [Cn = vsql.pesco.com] was successfully loaded for encryption. If the instance does not start, roll back (step 6).
Check from docker-1. Export the certificate (Export-Certificate -Cert Cert:\LocalMachine\My\<thumbprint> -FilePath vsql.cer) and copy it to docker-1. OpenSSL cannot speak TDS, so openssl s_client does not work against SQL Server. Verify the file at the image's security level instead:
```
openssl x509 -inform DER -in vsql.cer -out vsql.pem
openssl x509 -in vsql.pem -noout -subject -dates -ext subjectAltName,keyUsage,extendedKeyUsage
openssl verify -auth_level 2 -purpose sslserver -CAfile ~/packetstation/test/ca/pesco-ca.pem vsql.pem
```
Then prove the live connection with the doctor and VISUAL_ENCRYPT=yes. The image does not yet add the PESCO CA to OpenSSL's trust store for the ODBC driver (the issue 5 follow-up), so this one-off check builds a bundle in /tmp. It skips the entrypoint, so the M-Files password line fails. Read the VISUAL line.
```
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm --no-deps \
  -e VISUAL_ENCRYPT=yes --entrypoint sh app -c \
  'cat /etc/ssl/certs/ca-certificates.crt /etc/ssl/pesco/ca.pem > /tmp/ca.pem && SSL_CERT_FILE=/tmp/ca.pem exec packetstation doctor'
```
That the ODBC driver honours SSL_CERT_FILE is untested. If the VISUAL line fails with an issuer error while openssl verify passed, the trust step is the gap, not the server.

Production, off-hours. Production (vsql, PESCOINC) is the default instance, MSSQLSERVER. Bind the same certificate. VISUAL users lose their connection during the restart.

powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Thumbprint <thumbprint> -Instances MSSQLSERVER -WhatIf
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Thumbprint <thumbprint> -Instances MSSQLSERVER -RestartInstances

Check the ERRORLOG line again. The production stack's own doctor run is Spencer's, with its own approval.

Rollback. Each deployment saved deploy-<time>.json in C:\ProgramData\PacketStation\vsql-cert. -Rollback restores the newest one not yet rolled back. It restores each instance on its own, reads each value back and lists any failure. If a restart fails, fix the cause and run the same command again: it retries only the instances not yet restored or restarted, and starts SQL Server Agent if it ran before the failed attempt. Preview it, then run it:
```
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Rollback -Instances DEV -WhatIf
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Rollback -Instances DEV -RestartInstances
```
-Snapshot deploy-<time>.json restores a specific one. The rollback leaves the certificate and the key grant in place. Removing them is a manual step once nothing is bound to them.

Doctor

packetstation doctor proves a host and a stack are ready. Run it with compose run, not compose exec: the entrypoint exports MFILES_PASSWORD from its secret file, and exec skips the entrypoint, so the M-Files password check fails under exec even though the running worker has it.

docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm app doctor

It checks:

Settings and the stack guard (test refuses a production VISUAL target).
visualkit doctor and mfileskit doctor. This is the first real Linux test of both libraries.
Database connection, roles, migrations at head, and an audit UPDATE that must fail.
LISTEN/NOTIFY round trip.
Assets volume and free space. The backup folder is readable (SKIP when it is not mounted).
LDAP TLS bind on both controllers, reading memberOf, employeeID and userAccountControl as the service account, the 50-group count and one nested-group query.
SMTP STARTTLS with certificate verification.
ClamAV (phase 4): clamd answers PING at CLAMAV_HOST. SKIP when no host is set.
tzdata.

Use --skip visual,mfiles,ldap,smtp to run only the local checks. The network checks need the secrets and the CA bundle in place.

Test stack first start (2026-10-17, phase 5 image)

The test stack came up healthy on docker-1 through NPM (https://packetstation-test.pescoinc.biz). Doctor: VISUAL DEV connect as visualkit_ro through the named instance, M-Files all four drawing classes, SMTP STARTTLS verified, database at 0006, assets volume. Only the LDAP bind failed (AD data 52e, wrong password for svc-packetstation) until the service account password was reset and re-entered.

Verified on docker-1 (2026-10-15, phase 0)

The image built on docker-1 from this tree (python:3.12-slim-bookworm, ODBC Driver 18, visualkit 0.3.0, mfileskit 0.1.2). The CI stack (deploy/compose.ci.yaml with throwaway secrets) came up with postgres, migrate, app and worker all healthy. Inside it, doctor --skip visual,mfiles,ldap,smtp passed: migrations at head, the audit UPDATE denied for ps_app, LISTEN/NOTIFY round trip, assets volume, tzdata. visualkit doctor in the image found the ODBC driver and pyodbc on Linux; the connection itself was not tried because the CI stack has no VISUAL credentials. mfileskit doctor imported and ran on Linux up to its settings check for the same reason. The full doctor with the real secrets, the CA bundle and both domain controllers is the first step of the test stack deployment.

Inside containers the short name VSQL\DEV does not resolve. The env files use vsql.pesco.com\DEV.

CI stack and the Samba domain controller (phase 1)

deploy/compose.ci.yaml adds a throwaway Samba Active Directory domain controller (samba-dc, realm PSTEST.LOCAL, host dc1.pstest.local on the internal backend network) built from deploy/samba/. On first start it provisions the domain, generates a CA and a server certificate, seeds the OU tree, the service account, the groups (49 of the 50, plus one extra, to test the report) and 17 test users with random passwords, and exports ca.pem and users.json to the ci-dc-export volume. The app, migrate and worker mount that volume at /etc/ssl/pstest and deploy/ci/ci.env points LDAP_CA_BUNDLE at it, so CI exercises the real LDAPS path. The playwright service (profile e2e) drives the app at http://ps-app:8000 with the exported users. deploy/samba/README.md has the details and the commands.

The DC runs with cap_add: [SYS_ADMIN], which Samba needs to set the sysvol ACLs. The app carries the alias ps-app for Playwright because app is a top-level domain on Chromium's HSTS preload list, so http://app:8000 is forced to TLS and fails. Running the server test suite against the stack's own PostgreSQL is not safe: the pg_settings fixture resets the role passwords (see deploy/samba/README.md).

The real PESCO domain controllers are never used by CI. The test and production stacks do not run the DC.

CI feed fixtures (phase 3)

CI never reads VISUAL or M-Files. deploy/ci/ci.env sets FEED_SOURCE=fixture, DRAWING_SOURCE=fixture and FIXTURE_DIR=/fixtures. The image does not contain tests/, so the e2e job generates the fixtures on the runner before the stack starts:

python -m pip install pypdf==6.21.4
python -m tests.fixtures.generator --out tests/fixtures/generated

The generator is deterministic and writes feed.1.json, feed.2.json, drawings.1.json, drawings.2.json, the PDFs they name and state (1). tests/fixtures/generated/ is ignored by git. deploy/compose.ci.yaml binds it at /fixtures into the worker (read-only) and the Playwright container (read-write, so the scenario can write state = 2 and press Run now again). packetstation fixtures use N --dir DIR switches the state by hand. The CI feed runs at the default intervals; the scenario does not wait for them, it presses Run now.

Gitea Actions runner (phase 2)

.gitea/workflows/ci.yml runs on a Gitea Actions runner named docker-1, installed on docker-1 at ~/gitea/runner from deploy/runner/: one act_runner container (pinned by digest) with the host Docker socket, label ubuntu-latest mapped to docker.gitea.com/runner-images:ubuntu-latest, capacity 2. The server, frontend and image jobs run in job containers from that image. The e2e job starts the CI compose stack through the runner's Docker proxy on docker-1's daemon, which maps the job workspace so the stack's bind mounts resolve and removes the stack when the job ends. The runner was registered on 2026-10-16 with a token generated by gitea actions generate-runner-token inside the gitea container; the token was never written down. deploy/runner/README.md has the install, update and re-register steps.

The repository is spencerm/PacketStation on Gitea with Actions enabled. The workflow ran there on 2026-10-16: server, frontend and e2e pass (the e2e job builds the image, starts the CI stack with the Samba DC and runs the Playwright scenarios on docker-1's daemon). The image job needs the repository secrets REGISTRY_USER and REGISTRY_TOKEN (a Gitea token with package write) and fails until they are set. Two runner settings were needed and are in deploy/runner: the actions cache server published on docker0 port 8088 (job containers cannot reach the runner's own network, and every cache call timed out), and the job workspace on a host directory allowed in valid_volumes (so the stack's bind mounts resolve).

Image rebuild for phase 9

Phase 9 bumps two pins for security fixes found by the CI dependency scan: pypdf 6.21.0 to 6.21.2 and certifi 2026.7.22 to 2026.10.19 (server/requirements.txt and pyproject.toml). They reach a stack only with a new image. Build and tag the phase 9 commit as usual (:sha-<sha> from CI), set it in the stack's env file, and up -d. Migration 0010 runs on the way up. Then run the doctor.

Notifications (phase 9)

The app sends test messages and the worker sends everything else, through PESCO's relay.

Setting	Test stack	Production	CI
`SMTP_HOST`	`app-03.pesco.com`	`app-03.pesco.com`	empty (mail off)
`SMTP_PORT`	25	25	25
`SMTP_STARTTLS`	`true`	`true`	`true`
`SMTP_FROM`	`[email protected]`	`[email protected]`	`[email protected]`
`SMTP_CA_BUNDLE`	`/etc/ssl/pesco/ca.pem`	`/etc/ssl/pesco/ca.pem`	`/etc/ssl/pesco/ca.pem`
`NOTIFY_DRY_RUN`	`true`	`false`	`true`
`STACK`	`test`	`prod`	`ci`

STARTTLS always, verified against the PESCO CA, with the host name so the certificate name matches. No login. A relay that does not offer STARTTLS, or a certificate that does not verify, fails the message. There is no plain-text fallback.
Subjects start [Packet Station] when STACK=prod, otherwise [Packet Station TEST].
Each message is addressed To SMTP_FROM. The recipients are in the envelope only.
NOTIFY_DRY_RUN=true records every message as dry_run in the sent log and sends nothing. The test stack keeps it on, so the test stack never mails people about test data. Production sets it false.
With SMTP_HOST empty, every message is recorded failed with "Mail is off: SMTP_HOST is empty."
The test send. On the Admin page, Notifications tab, open a rule, put an address in "Send the test to" and press "Send a test". It sends at once from the app, even in dry run, and shows the result. The sent log keeps the row (marked Test). A test with no address goes to the rule's recipients. On a new stack, send one test to your own address before relying on the rules.
Every rule starts on, sent at once, to the AD group PS-Admins. Change the recipients on the Notifications tab. Each person sees on Account settings what reaches them and why.
The worker needs the LDAP bind to resolve group recipients. While AD is down, an immediate message waits and is tried twice more, then recorded failed.

The doctor's SMTP line checks STARTTLS with certificate verification, without sending.

Backup (phase 9)

Host scripts in scripts/backup/, run by root from a systemd timer. They run outside the app, so a broken app still backs up. scripts/backup/README.md has the same steps with more detail.

File	Installed as
`ps-backup.sh`	`/usr/local/sbin/ps-backup.sh` (root, 0755)
`ps-restore.sh`	`/usr/local/sbin/ps-restore.sh` (root, 0755)
`backup.conf.example`	`/etc/packetstation/backup.conf` (root, 0644)
`packetstation-backup.service`	`/etc/systemd/system/` (root, 0644)
`packetstation-backup.timer`	`/etc/systemd/system/` (root, 0644)

backup.conf lists the stacks, one per line: name, container (or project:<compose project>), database, folder.

test     project:packetstation-test   packetstation  /home/spencerm/packetstation/test/backups
#prod    project:packetstation        packetstation  /var/backups/packetstation/prod

The production line stays commented out until the production stack runs (phase 10). Until then it would fail every night.

Where the dumps go:

Test stack: /home/spencerm/packetstation/test/backups, owned by spencerm (the script gives new files the folder's owner and group). The test stack.env sets BACKUP_DIR to this path.
Production: /var/backups/packetstation/prod, root:10001 0750. The production stack.env sets BACKUP_DIR=/var/backups/packetstation/prod.

The worker mounts the stack's folder read-only at /backups. Its backup-check job (07:15) only lists the folder, reads the dumps' sizes and times, and reads the markers.

What a run does, per stack:

Finds the stack's running postgres container (by compose labels for project:). None or two is a failure.
docker exec <pg> pg_dump -U postgres -d packetstation -Fc over the local socket, which the postgres image trusts. No password is read or passed. Written under a temporary name, .ps-<stamp>.partial.
pg_dumpall --globals-only --no-role-passwords: the roles without their password hashes. The role passwords are in the stack secrets.
pg_restore --list with the container's own pg_restore must list at least one entry. Then the files are renamed to ps-<UTC yyyymmddThhmmssZ>.dump and .globals.sql, mode 0600.
Keeps the newest 7 (KEEP=7 in the unit, --keep N), with their globals files.
Writes last-success or last-failure (<ISO UTC time> <dump name>, plus the reason on failure), mode 0644. A failed run leaves no partial file.

Each stack runs on its own, so one failing does not stop the next. Exit 1 if any stack failed, 2 for a usage or config error. flock keeps two runs from overlapping.

The unit is Type=oneshot as root, Nice=10, IOSchedulingClass=idle, TimeoutStartSec=1h, with NoNewPrivileges, PrivateTmp and ProtectSystem=full (no ProtectHome, so it can write the test folder under /home). The timer is OnCalendar=*-*-* 02:00:00 America/Denver with Persistent=true: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot.

Install (Spencer, with sudo, on docker-1)

After the phase 9 tree is synced to ~/packetstation/repo:

cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh  /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer
sudo systemctl start packetstation-backup.service
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups

list-timers shows the next run at 08:00 or 09:00 UTC (02:00 in Denver). The one run by hand should leave a new ps-*.dump and a fresh last-success. After changing a script in the repo, run its install line again: the installed copies are root's, so a sync of the checkout does not change what root runs.

When the production stack exists (phase 10):

sudo install -d -o root -g root  -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf      # uncomment the prod line
sudo systemctl start packetstation-backup.service

A run by hand needs no sudo for the test stack (spencerm is in the docker group):

scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups

The dumps hold the database only. The assets volume (the files) is not in them. It is covered by PBS, since the volume is part of the VM.

Restore (phase 9)

ps-restore.sh never stops, restarts or reconfigures a stack. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.

B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)

# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored

# 2. Assets, from a folder or a tar (for example a PBS file-level restore of the assets volume folder).
#    Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets

# 3. Every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets

# Row counts of any database, to compare with the live one.
ps-restore.sh --project packetstation-test --counts-only --db packetstation

The database is created as 01-roles.sh creates it (owner ps_migrate, UTF8, template0, CONNECT and TEMPORARY for ps_app), then pg_restore --exit-on-error --single-transaction as postgres. A failed restore leaves an empty database, never a half one.
The globals file is applied without stopping on "already exists". Any other error fails. The script refuses a globals file that holds a password hash. On a new cluster the roles come back without passwords: set each from the stack's secret file with \password in psql, or start the stack on an empty volume so initdb creates them.
The assets copy runs in a throwaway container from the image the postgres container already runs, with --network none, so nothing is downloaded.
To put a restored database into service: stop that stack's app and worker, rename as postgres (ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;), start the stack, check the app, then drop the old database. This step is deliberate and manual.

Restore rehearsal 2026-10-21

Done on docker-1 against the test stack (packetstation-test, image sha-ce28d2f, container packetstation-test-postgres-1, postgres:16-bookworm by digest), as spencerm without sudo, from a scratch copy of the scripts. The stack was not stopped, restarted or changed. Times UTC.

Step	Result
Backup by hand into `/home/spencerm/packetstation/test/backups`	21:24:05 to 21:24:06, 0.76 s. `ps-20261021T212405Z.dump`, 2,234,906 bytes, 743 `pg_restore --list` entries. Globals 774 bytes with no password line (roles postgres, ps_app, ps_migrate, attributes only). `last-success` written. Dump and globals 0600, marker 0644, all spencerm. Exit 0. This dump stays in the folder.
Failure path (a database that does not exist, into a scratch folder)	Exit 1, pg_dump's error on stderr, `last-failure` = `2026-10-21T21:24:53Z ps-20261021T212453Z.dump pg_dump failed`, no partial file.
Live row counts	`--counts-only --db packetstation`: 64 tables.
Restore into `ps_restore_rehearsal` with the globals	Globals: 3 statements skipped (the roles exist), no other error, no password touched. Created and restored in 1.65 s. Every one of the 64 tables has the same row count as the live database, checked just before and just after (for example assets 189, audit_events 2137, bom_lines 5742).
Assets out of the live volume	Tarred through a throwaway container (`--network none`, read-only mount): 129,546,240 bytes, 189 files, 3.8 s. sha256 of every file against the 189 `assets` rows of the restored database: 189 match, 0 differ, 0 missing, 0 files without a row.
Asset restore path	`--assets <the tar> --assets-volume ps_restore_rehearsal_assets` into a throwaway volume, 0.47 s. `--verify-assets`: 189 rows, 189 match, 0 differ, 0 missing, 0 orphans. A second copy changed nothing (no overwrite). Files owned 10001:10001.
Guard	Restoring again into `ps_restore_rehearsal` without `--replace`: refused, exit 1.
Cleanup	`DROP DATABASE ps_restore_rehearsal`, the throwaway volume and the scratch folder removed. The cluster lists only packetstation, postgres, template0 and template1. App, worker and postgres stayed healthy throughout.
Timer	`systemd-analyze calendar "--* 02:00:00 America/Denver"`: next run 2026-10-22 08:00 UTC. `systemd-analyze verify` on both units found only the script not installed yet. systemd 255.

Nothing went wrong. Notes from it:

The dump is small (2.2 MB). The assets (125 MB) are the bulk and are not in the nightly dump. PBS covers them; the restore path is a PBS file-level restore of the volume folder, or a tar, through --assets.
The globals restore re-applies role attributes. On the same cluster that changes nothing. On a new cluster it creates the roles without passwords: set them from the secrets.
The next rehearsal is IT's, quarterly (requirement O2).

42 KiB Raw Blame History