PacketStation/docs/DEPLOY.md

# Deploying Packet Station

How Packet Station runs. Production and test move to their own native Linux VMs in phase 10 ("Native VM install"). Until the test VM is ready the test stack runs in Docker on `docker-1`, which keeps CI. Sections not built yet are marked "planned, phase N". The design is in [PLAN.md](PLAN.md) under "Deployment and CI".

## Host facts

- Host: `docker-1`, 192.168.101.12.
- Ubuntu 24.04 LTS, Docker 29, Compose v5. Clock in UTC.
- Nginx Proxy Manager already runs there as container `npm_app_1` on network `npm_default` (172.22.0.0/16). Its compose project is in `/home/spencerm/npm`.
- The host also runs Gitea (`gt.pescoinc.biz`, the container registry and the Python index) and `pesco-ncr`.
- Proxmox Backup Server backs up the whole VM at 03:00 daily.
- Host patching and hardening are IT's (requirement H5).
- Inside containers only full names resolve. Docker's resolver has no search domain, so every host setting is an FQDN (for example `vsql.pesco.com\DEV`, `app-03.pesco.com`).

## The two stacks

Both stacks run from the same image as separate compose projects.

| Stack | Compose project | DNS name | Network alias on npm_default | VISUAL |
|---|---|---|---|---|
| Test | `packetstation-test` | `packetstation-test.pescoinc.biz` | `packetstation-test-app` | DEV: `vsql.pesco.com\DEV` / `PESCOINCDEV` |
| Production | `packetstation` | `packetstation.pescoinc.biz` | `packetstation-app` | `vsql` / `PESCOINC`, set explicitly (planned, phase 10; moves to its own VM, see below) |

- The app publishes no port. It joins `npm_default` as an external network with an explicit alias per stack (`APP_ALIAS`). Plain service names would collide, since both stacks share that network.
- PostgreSQL stays on each stack's own internal `backend` network, never on `npm_default`.
- The test stack never points at VISUAL production. The production stack sets `VISUAL_SERVER` and `VISUAL_DATABASE` explicitly. Spencer starting it with those values is the one-time approval for its scheduled polling.
- There is no M-Files test vault. Both stacks read the production vault, read-only, as `mfileskitRO`.
- The internal A records for both names point at 192.168.101.12 in the AD-integrated `pescoinc.biz` zone.

## Services

- `postgres`: PostgreSQL 16, named volume, `backend` network only. Its init creates the database roles from secrets.
- `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
- `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup, housekeeping and the phase 9 jobs: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only at `/backups` (the backup-check job reads it). Its healthcheck reads the heartbeat file.
- `clamav` (optional, profile `clamav`, phase 4): `clamav/clamav:1.4`, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias `packetstation-clamav` (as `postgres` has its own alias). `backend` network plus egress (freshclam downloads the signatures), no `npm_default`, no published port. Volume `clamav-db` at `/var/lib/clamav`, `mem_limit: 2g`, healthcheck `clamdcheck.sh`. Started only with `--profile clamav`. See "Virus scan (ClamAV)".
- All services restart unless stopped, log to stdout (json-file), and have healthchecks.

## Secrets

Docker secrets, one file each, in the stack's `secrets/` folder ("Env files" says where each stack keeps it). Production: owner `root:10001` (10001 is the image's non-root user), mode `0440`. Test: owner `spencerm`, mode `0444`, reachable only through `/home/spencerm` and `packetstation/test`, both mode `0750`, so no other host user can read them.

| Secret | Used by |
|---|---|
| `postgres_superuser_password` | PostgreSQL bootstrap |
| `postgres_migrate_password` | migrate role (`ps_migrate`) |
| `postgres_app_password` | app role (`ps_app`) |
| `visualkit_password` | `VISUAL_PASSWORD_FILE` for `visualkit_ro` |
| `mfileskit_password` | `mfileskitRO`; the entrypoint exports it as `MFILES_PASSWORD` |
| `ldap_bind_password` | `LDAP_PASSWORD_FILE` for `svc-packetstation` |
| `session_key` | session signing |
| `seal_key` | HMAC filing seals and controlled-copy stamps |

Never put a secret in the repo, the env file, a log or chat. Settings read secrets through `*_PASSWORD_FILE` paths.

The worker gets `postgres_app_password` (it logs in as the app role, with application name `ps-worker`), `visualkit_password`, `mfileskit_password`, `ldap_bind_password` and `seal_key`. The worker never logs a password: job errors are redacted before they are stored, and the redaction also strips the VISUAL and M-Files passwords themselves.

The PESCO CA chain is mounted into the containers once and referenced by `LDAP_CA_BUNDLE`, `MFILES_CA_BUNDLE` and `SMTP_CA_BUNDLE`.

## Env files

One folder per stack. The test stack: `/home/spencerm/packetstation/test/`, like the other compose stacks on docker-1 (decided 2026-10-18, Gitea issue 38). It holds `stack.env`, `secrets/`, `ca/pesco-ca.pem` and `backups/`, and its `stack.env` sets `ENV_FILE`, `SECRETS_DIR`, `CA_BUNDLE_FILE` and `BACKUP_DIR` to those paths. The compose files run from the checkout in `/home/spencerm/packetstation/repo/deploy`. On 2026-10-27 Spencer decided that production and test each run natively on their own Linux VM, without Docker, and that docker-1 keeps CI only (Gitea issue 37). The test stack stays here until the test VM is ready. The VMs keep their settings in `/etc/packetstation/packetstation.env` and their secrets in `/etc/packetstation/secrets/`: see "Native VM install". The production compose lines in this file are the old plan, kept for reference. Each folder holds `stack.env` (settings only, no secrets) and `secrets/`. The env file sets `PROJECT_NAME`, `SECRETS_DIR` and `APP_ALIAS`, which the compose files require. `.env.example` in the repo lists every setting.

Key settings:

- `VISUAL_SERVER`, `VISUAL_DATABASE`, `VISUAL_USER=visualkit_ro`, `VISUAL_PASSWORD_FILE`.
- `MFILES_URL=https://mfiles.pescoinc.biz/REST`, `MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}`, `MFILES_USERNAME=mfileskitRO`, `MFILES_CA_BUNDLE`.
- `LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636`, `LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com`, `LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com`, `LDAP_PASSWORD_FILE`, `LDAP_CA_BUNDLE`.
- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`, `NOTIFY_DRY_RUN` (`true` on the test stack, `false` on production). `STACK` (`test` or `prod`) picks the subject prefix. See "Notifications".
- `BACKUP_DIR`: the stack's backup folder on the host, mounted read-only in the worker. See "Backup".
- `APP_ALIAS` per stack.
- Identity timing and limits, all optional with these defaults (an Admin can override most of them within bounds on the Security tab; see [SECURITY.md](SECURITY.md), "Security settings"): `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_DESK_MINUTES=60` (Admin and Clerical wait at least this long for both), `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
- Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
- Uploads and storage, all optional: `UPLOAD_MAX_BYTES` (default 25 MB, drawings, MTRs and documents), `UPLOAD_MAX_BYTES_BOM` (default 5 MB, hand BOMs), `ASSETS_QUOTA_WARN_PCT` (default 85, 1 to 99) and `ASSETS_QUOTA_BYTES` (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
- Virus scan, all optional: `CLAMAV_HOST` (empty by default, which means no scan; `packetstation-clamav` for the profile service), `CLAMAV_PORT` (default 3310) and `CLAMAV_TIMEOUT_SECONDS` (default 30). The env examples carry `#CLAMAV_HOST=packetstation-clamav` commented out. See "Virus scan (ClamAV)".
- `SITE_ID` (default `PESCO`) and `SITE_TZ` (default `America/Denver`): the VISUAL site in every call, and the time zone for shop hours and the schedule.
- The session cookie is marked `Secure` when the request arrived over https (as reported by NPM through `X-Forwarded-Proto`). Behind the proxy that is always the case.

## Feed settings live in the database

The feed's schedule and rules are not env settings. Intervals, shop hours, the window, the go-live date, departments, the drawing recheck and budget, labor tickets, auto-release and the pause switch are one `settings` row (key `feed`), edited by an Admin on the Admin page, Feed tab. The resource map is the `visual_resource_map` table, edited on the "VISUAL resources" tab. A save notifies the worker, which applies it without a restart. A new stack starts with the defaults in [FEED.md](FEED.md), "Settings". Set the go-live date before the production feed's first run.

## Deploy command

Test stack:

```bash
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml up -d
```

Production (planned, phase 10): the same command with `-p packetstation` and the production env file. The first production start is Spencer's approval.

Images come from `gt.pescoinc.biz` by tag. CI tags each tested image `:sha-<sha>`; a release re-tags it `vX.Y.Z`. Never `latest`. A deploy is a new image with a short downtime window.

## Virus scan (ClamAV, phase 4)

ClamAV is optional and off by default. It closes requirement D5. IT decided on 2026-10-18 (Gitea issue 19) to leave it off in both stacks: every workstation that uploads runs Microsoft Defender. The steps below stay for the day that changes. Turn it on in the test stack first, check it, then the production stack.

1. Set `CLAMAV_HOST=packetstation-clamav` in the stack's env file (`~/packetstation/test/stack.env`). Leave `CLAMAV_PORT` and `CLAMAV_TIMEOUT_SECONDS` at their defaults.
2. Start the stack with the profile:

```bash
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml --profile clamav up -d
```

3. Wait for `clamav` to turn healthy. On the first start the volume `packetstation-test_clamav-db` is empty, so freshclam downloads the full signature set (a few hundred MB) before clamd answers. That takes several minutes. Later starts reuse the volume and only fetch updates. freshclam needs egress to the ClamAV mirrors, which is why the service is on the `egress` network.
4. Check it: Admin, Storage tab, the ClamAV line reads "packetstation-clamav: reachable". `packetstation doctor` also has a `clamav` check (a PING to clamd; SKIP when `CLAMAV_HOST` is empty). Upload a small PDF to a draft and confirm it is stored.

Every later `up` for that stack must carry `--profile clamav`. Without it the service is not started.

Fail closed. While `CLAMAV_HOST` is set, an upload the scanner cannot check is refused with "The virus scanner is not reachable. Try again in a minute." (503 `scan_unavailable`). That covers clamd still loading signatures, stopped, or out of memory. The worker's drawing fetches are refused the same way and tried again later. A file with a signature hit is refused (422 `infected`), never stored, and audited as `asset.rejected` with the signature name.

To turn it off: remove `CLAMAV_HOST` from the env file, recreate `app` and `worker` (`up -d` without the profile), then stop the service with `docker compose ... --profile clamav stop clamav`. The volume can stay for the next time. The CI overlay never runs ClamAV.

clamd holds the signatures in memory, about 1.2 GB with the full set and more while it reloads them, so the service has a 2 GB limit. Check docker-1 has the headroom for both stacks before turning it on in production.

## Storage quota (phase 4)

Every stored file and the housekeeping job check the assets volume. With `ASSETS_QUOTA_BYTES=0` (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At `ASSETS_QUOTA_WARN_PCT` percent (85 by default) the app opens one `notification_events` alert of kind `assets_quota` and audits `storage.quota_warning`. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.

The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. The alert is mailed through the `assets_quota` notification rule, to PS-Admins by default (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold), with a "Cleared" message when usage drops back.

## Python packages and the Gitea index

The Gitea PyPI index `https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/` serves only `visualkit`, `mfileskit` and their dependencies.

- Install `visualkit` and `mfileskit` from it with `--index-url`. Never `--extra-index-url`, which would let a public package with the same name win.
- Install everything else from public PyPI in a separate pip step.
- The image needs Microsoft ODBC Driver 18 (`msodbcsql18`) from Microsoft's Debian 12 repository for VisualKit.

## NPM proxy host

One proxy host per stack in Nginx Proxy Manager, recorded in `deploy/npm-proxy-host.md`. For the VMs see "NPM proxy host for the VMs": only the forward target and `TRUSTED_PROXY_HOST` change.

- Domain: `packetstation-test.pescoinc.biz` (test) or `packetstation.pescoinc.biz` (production).
- Scheme `http`, forward host the stack's alias (`packetstation-test-app` or `packetstation-app`), port `8000`.
- SSL: the stack's certificate, Force SSL on, HTTP/2 on.
- NPM HSTS off. The app sends HSTS itself.
- Advanced config: `client_max_body_size 30m;` plus a location block for the event stream:

```nginx
client_max_body_size 30m;

location /api/events {
    proxy_pass $forward_scheme://$server:$port;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}
```

The block uses NPM's `$server` variables so the alias is re-resolved. The app also sends `X-Accel-Buffering: no` on `/api/events`.

Open check (plan question 16): confirm "Block common exploits" does not interfere with uploads when the test proxy host is made.

The app trusts `X-Forwarded-*` only from `npm_app_1`'s own address, re-resolved every 60 seconds. Other stacks share `npm_default`, and the per-address rate limit depends on the real client address.

## Known issue: the internal CA signs with SHA-1 (found 2026-10-17)

PESCO-PSYLOCKE-CA (valid to 2027-09-27) signs with sha1WithRSAEncryption, and so do the certificates it issued to DC-02.PESCO.COM and PDC-1.PESCO.COM on 636 and to APP-03.PESCO.COM on 25 (all valid to April 2027). OpenSSL 3 on Debian 12 refuses SHA-1 signatures at its default security level 2, so LDAPS and SMTP STARTTLS fail from inside the image with "CA signature digest algorithm too weak", while Windows and browsers still accept them. Verification passes at security level 0.

mfiles.pescoinc.biz uses a public Let's Encrypt certificate, so MFILES_CA_BUNDLE must stay unset; only LDAP_CA_BUNDLE and SMTP_CA_BUNDLE point at the PESCO chain.

Resolved 2026-10-17: the CA was switched to SHA-256 (`certutil -setreg ca\csp\CNGHashAlgorithm SHA256`, restart certsvc; the CA already used the Key Storage Provider) and the Kerberos Authentication, Domain Controller Authentication and Directory Email Replication certificates on DC-02 and PDC-1 and the Machine certificate on APP-03 were renewed with `certreq -enroll -machine -q -cert <thumbprint> renew reusekeys`. From docker-1 all three endpoints now verify against the PESCO bundle with sha256WithRSAEncryption. The CA certificate itself is still SHA-1 self-signed, which OpenSSL accepts for a trust anchor. If the problem comes back, the fix was: set the CA to issue with SHA-256 and reissue the domain controller and relay certificates. The CA certificate itself may stay SHA-1, since OpenSSL does not check the trust anchor's own signature. Until then the test stack cannot sign in or verify the relay without a test-only OpenSSL security-level override, which is Spencer's decision to apply, never the production stack's. The doctor should report a SHA-1 chain with this hint (follow-up).

The test stack on docker-1 lives under /home/spencerm/packetstation/test (stack.env, secrets/, ca/pesco-ca.pem, backups/). It stays there: that matches the other compose stacks on the host (Gitea issue 38, 2026-10-18).

## Known issue: Docker subnet pool on docker-1 (found 2026-10-17)

Docker's default address pool on docker-1 holds fifteen /16 bridge networks (172.17.0.0 to 172.31.0.0). Each compose stack takes one or two (Packet Station takes two: backend and egress), the Gitea runner takes one per job, and the CI e2e stack needs two more. With the test stack up the pool ran out and every e2e job failed within seconds with "all predefined address pools have been fully subnetted". Idle networks of stopped stacks were removed to make room. The lasting fix is a larger pool in /etc/docker/daemon.json, for example `{"default-address-pools":[{"base":"10.200.0.0/16","size":24}]}`, which needs sudo and a Docker restart (a short outage for every stack on the host), or trimming unused compose projects. Production and test move to their own VMs (Gitea issue 37), so the pool will only have to fit the runner and the CI e2e stack. Count the networks before adding any other stack: `docker network ls --filter driver=bridge | wc -l`.

## Known issue: vsql serves a weak self-signed certificate (found 2026-10-18)

`vsql.pesco.com\DEV` serves TLS (`VISUAL_ENCRYPT=trust` connects), but with `VISUAL_ENCRYPT=yes` OpenSSL in the image refuses it with "EE certificate key too weak": the server certificate's RSA key is under 2048 bits, which is what SQL Server's auto-generated self-signed certificate looks like. IT needs to issue vsql a 2048-bit, SHA-256 certificate from PESCO-PSYLOCKE-CA with `vsql.pesco.com` in the subject alternative names, install it on the DEV and production instances in SQL Server Configuration Manager and restart them. The image then needs the PESCO CA in its OpenSSL trust store for the ODBC driver (the entrypoint can concatenate the system bundle and /etc/ssl/pesco/ca.pem into /tmp and export `SSL_CERT_FILE`), and the stacks switch to `VISUAL_ENCRYPT=yes`. Tracked as Gitea issue 5. IT issues and binds the certificate by hand; the repo has no script for it.

## Doctor

`packetstation doctor` proves a host and a stack are ready. On a VM run `sudo ps-doctor` ("Commands on a VM"). On docker-1 run it with `compose run`, not `compose exec`: the entrypoint exports MFILES_PASSWORD from its secret file, and `exec` skips the entrypoint, so the M-Files password check fails under `exec` even though the running worker has it.

```bash
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm app doctor
```

It checks:

- Settings and the stack guard (test refuses a production VISUAL target).
- `visualkit doctor` and `mfileskit doctor`. This is the first real Linux test of both libraries.
- Database connection, roles, migrations at head, and an audit UPDATE that must fail.
- LISTEN/NOTIFY round trip.
- Assets volume and free space. The backup folder is readable (SKIP when it is not mounted).
- LDAP TLS bind on both controllers, reading `memberOf`, `employeeID` and `userAccountControl` as the service account, the 50-group count and one nested-group query.
- SMTP STARTTLS with certificate verification.
- ClamAV (phase 4): clamd answers PING at `CLAMAV_HOST`. SKIP when no host is set.
- tzdata.

Use `--skip visual,mfiles,ldap,smtp` to run only the local checks. The network checks need the secrets and the CA bundle in place.

### Test stack first start (2026-10-17, phase 5 image)

The test stack came up healthy on docker-1 through NPM (https://packetstation-test.pescoinc.biz). Doctor: VISUAL DEV connect as visualkit_ro through the named instance, M-Files all four drawing classes, SMTP STARTTLS verified, database at 0006, assets volume. Only the LDAP bind failed (AD data 52e, wrong password for svc-packetstation) until the service account password was reset and re-entered.

### Verified on docker-1 (2026-10-15, phase 0)

The image built on docker-1 from this tree (python:3.12-slim-bookworm, ODBC Driver 18, visualkit 0.3.0, mfileskit 0.1.2). The CI stack (`deploy/compose.ci.yaml` with throwaway secrets) came up with postgres, migrate, app and worker all healthy. Inside it, `doctor --skip visual,mfiles,ldap,smtp` passed: migrations at head, the audit UPDATE denied for `ps_app`, LISTEN/NOTIFY round trip, assets volume, tzdata. `visualkit doctor` in the image found the ODBC driver and pyodbc on Linux; the connection itself was not tried because the CI stack has no VISUAL credentials. `mfileskit doctor` imported and ran on Linux up to its settings check for the same reason. The full doctor with the real secrets, the CA bundle and both domain controllers is the first step of the test stack deployment.

Inside containers the short name `VSQL\DEV` does not resolve. The env files use `vsql.pesco.com\DEV`.

## CI stack and the Samba domain controller (phase 1)

`deploy/compose.ci.yaml` adds a throwaway Samba Active Directory domain controller (`samba-dc`, realm `PSTEST.LOCAL`, host `dc1.pstest.local` on the internal `backend` network) built from `deploy/samba/`. On first start it provisions the domain, generates a CA and a server certificate, seeds the OU tree, the service account, the groups (49 of the 50, plus one extra, to test the report) and 17 test users with random passwords, and exports `ca.pem` and `users.json` to the `ci-dc-export` volume. The app, migrate and worker mount that volume at `/etc/ssl/pstest` and `deploy/ci/ci.env` points `LDAP_CA_BUNDLE` at it, so CI exercises the real LDAPS path. The `playwright` service (profile `e2e`) drives the app at `http://ps-app:8000` with the exported users. [deploy/samba/README.md](../deploy/samba/README.md) has the details and the commands.

The DC runs with `cap_add: [SYS_ADMIN]`, which Samba needs to set the sysvol ACLs. The app carries the alias `ps-app` for Playwright because `app` is a top-level domain on Chromium's HSTS preload list, so `http://app:8000` is forced to TLS and fails. Running the server test suite against the stack's own PostgreSQL is not safe: the `pg_settings` fixture resets the role passwords (see deploy/samba/README.md).

The real PESCO domain controllers are never used by CI. The test and production stacks do not run the DC.

### CI feed fixtures (phase 3)

CI never reads VISUAL or M-Files. `deploy/ci/ci.env` sets `FEED_SOURCE=fixture`, `DRAWING_SOURCE=fixture` and `FIXTURE_DIR=/fixtures`. The image does not contain `tests/`, so the e2e job generates the fixtures on the runner before the stack starts:

```bash
python -m pip install pypdf==6.21.4
python -m tests.fixtures.generator --out tests/fixtures/generated
```

The generator is deterministic and writes `feed.1.json`, `feed.2.json`, `drawings.1.json`, `drawings.2.json`, the PDFs they name and `state` (`1`). `tests/fixtures/generated/` is ignored by git. `deploy/compose.ci.yaml` binds it at `/fixtures` into the worker (read-only) and the Playwright container (read-write, so the scenario can write `state` = `2` and press Run now again). `packetstation fixtures use N --dir DIR` switches the state by hand. The CI feed runs at the default intervals; the scenario does not wait for them, it presses Run now.

## Gitea Actions runner (phase 2)

`.gitea/workflows/ci.yml` runs on a Gitea Actions runner named `docker-1`, installed on docker-1 at
`~/gitea/runner` from [deploy/runner/](../deploy/runner/): one `act_runner` container (pinned by digest) with
the host Docker socket, label `ubuntu-latest` mapped to `docker.gitea.com/runner-images:ubuntu-latest`,
capacity 2. The server, frontend and image jobs run in job containers from that image. The e2e job starts
the CI compose stack through the runner's Docker proxy on docker-1's daemon, which maps the job workspace so
the stack's bind mounts resolve and removes the stack when the job ends. The runner was registered on
2026-10-16 with a token generated by `gitea actions generate-runner-token` inside the `gitea` container; the
token was never written down. [deploy/runner/README.md](../deploy/runner/README.md) has the install, update
and re-register steps.

The repository is `spencerm/PacketStation` on Gitea with Actions enabled. The workflow ran there on
2026-10-16: server, frontend and e2e pass (the e2e job builds the image, starts the CI stack with the Samba
DC and runs the Playwright scenarios on docker-1's daemon). The image job needs the repository secrets
`REGISTRY_USER` and `REGISTRY_TOKEN` (a Gitea token with package write) and fails until they are set.
Two runner settings were needed and are in deploy/runner: the actions cache server published on docker0
port 8088 (job containers cannot reach the runner's own network, and every cache call timed out), and the
job workspace on a host directory allowed in `valid_volumes` (so the stack's bind mounts resolve).

## Image rebuild for phase 9

Phase 9 bumps two pins for security fixes found by the CI dependency scan: `pypdf` 6.21.0 to 6.21.2 and `certifi` 2026.7.22 to 2026.10.19 (`server/requirements.txt` and `pyproject.toml`). They reach a stack only with a new image. Build and tag the phase 9 commit as usual (`:sha-<sha>` from CI), set it in the stack's env file, and `up -d`. Migration 0010 runs on the way up. Then run the doctor.

## Notifications (phase 9)

The app sends test messages and the worker sends everything else, through PESCO's relay.

| Setting | Test stack | Production | CI |
|---|---|---|---|
| `SMTP_HOST` | `app-03.pesco.com` | `app-03.pesco.com` | empty (mail off) |
| `SMTP_PORT` | 25 | 25 | 25 |
| `SMTP_STARTTLS` | `true` | `true` | `true` |
| `SMTP_FROM` | `[email protected]` | `[email protected]` | `[email protected]` |
| `SMTP_CA_BUNDLE` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` |
| `NOTIFY_DRY_RUN` | `true` | `false` | `true` |
| `STACK` | `test` | `prod` | `ci` |

- STARTTLS always, verified against the PESCO CA, with the host name so the certificate name matches. No login. A relay that does not offer STARTTLS, or a certificate that does not verify, fails the message. There is no plain-text fallback.
- Subjects start `[Packet Station]` when `STACK=prod`, otherwise `[Packet Station TEST]`.
- Each message is addressed To `SMTP_FROM`. The recipients are in the envelope only.
- `NOTIFY_DRY_RUN=true` records every message as `dry_run` in the sent log and sends nothing. The test stack keeps it on, so the test stack never mails people about test data. Production sets it `false`.
- With `SMTP_HOST` empty, every message is recorded `failed` with "Mail is off: SMTP_HOST is empty."
- The test send. On the Admin page, Notifications tab, open a rule, put an address in "Send the test to" and press "Send a test". It sends at once from the app, even in dry run, and shows the result. The sent log keeps the row (marked Test). A test with no address goes to the rule's recipients. On a new stack, send one test to your own address before relying on the rules.
- Every rule starts on, sent at once, to the AD group PS-Admins. Change the recipients on the Notifications tab. Each person sees on Account settings what reaches them and why.
- The worker needs the LDAP bind to resolve group recipients. While AD is down, an immediate message waits and is tried twice more, then recorded `failed`.

The doctor's SMTP line checks STARTTLS with certificate verification, without sending.

## Native VM install (phase 10)

Production and test each run natively on their own Linux VM, without Docker (Spencer, 2026-10-27, Gitea issue 37). docker-1 keeps CI only. The test VM is built first from the same steps, so the install, the backups, the restore and the doctor are rehearsed before production. The docker-1 test stack keeps running until the test VM is ready (see "Retiring the docker-1 test stack").

The kit is in `deploy/vm/`. Spencer runs everything that needs root. Nothing in it starts the worker by itself: on production, Spencer's first start of the worker is the approval of the scheduled VISUAL polling.

### What IT provides (open on issue 37)

- One VM per stack. Ubuntu 24.04 LTS is recommended: it ships Python 3.12, which the server needs (`requires-python >= 3.12`, pins resolved for 3.12). Debian 12 ships Python 3.11, so on Debian 12 a Python 3.12 has to be provided separately and passed to `ps-upgrade` as `PYTHON=/path/to/python3.12`. `install.sh` supports both and refuses anything else. Size: 2 vCPU, 4 GB RAM and 40 GB disk is a starting point (the app and the worker are capped at 1 GB each; the test stack's assets were 125 MB after a month of testing).
- A fixed LAN address per VM. NPM forwards to it.
- Outbound from the VM: the OS mirrors, `apt.postgresql.org`, `packages.microsoft.com`, `www.postgresql.org` (the PGDG key), `pypi.org` and `files.pythonhosted.org`, `gt.pescoinc.biz` (the Python index), and at run time vsql (1433, plus UDP 1434 for the named DEV instance), `mfiles.pescoinc.biz` 443, DC-02 and PDC-1 636, `app-03.pesco.com` 25, the QNAP (NFS 2049).
- Inbound: SSH for Spencer, and port 8000 from docker-1 (192.168.101.12) only. The VM enforces the second itself.
- PBS backs the VM up like any other, at 03:00.

### Layout on a VM

| Path | Owner, mode | What |
|---|---|---|
| `/opt/packetstation/<build>/` | root, 0755 | One release: the source, `BUILD`, `static/` (the built frontend) and `venv/`. Read-only to the services. |
| `/opt/packetstation/current`, `previous` | root | Symlinks to the running and the last release. The units start `current/deploy/vm/ps-launch.sh`. |
| `/etc/packetstation/packetstation.env` | root:packetstation 0640 | Settings, no secrets. From `deploy/vm/prod.env.example` or `test.env.example`. |
| `/etc/packetstation/secrets/` | root:packetstation 0750, files 0440 | One file per secret (table below). The services get them as systemd credentials and cannot open the folder. |
| `/etc/packetstation/ca/pesco-ca.pem` | root, 0644 | The PESCO CA chain for `LDAP_CA_BUNDLE` and `SMTP_CA_BUNDLE`. Also in the system trust store, which the ODBC driver uses. |
| `/etc/packetstation/backup.conf`, `packetstation.nft` | root, 0644 | The backup stack line; the firewall table (written by `install.sh`). |
| `/var/lib/packetstation/assets/` | packetstation, 0750 | `ASSETS_DIR`, the stored files. |
| `/var/backups/packetstation/` | root:packetstation 0750 | `BACKUP_DIR`, the nightly dumps and markers. |
| `/mnt/packetstation-archive/` | root, 0755 until mounted | `ARCHIVE_ROOT`, the QNAP NFS share. |
| `/usr/local/sbin/ps-run`, `ps-doctor`, `ps-upgrade`, `ps-backup`, `ps-restore` | root, 0755 | The commands below. |

The service user is `packetstation`, uid and gid 10001: the image's uid, so the NFS share's rights and the old volume's file owners carry over. PostgreSQL 16 from the PGDG repository listens on the local socket only (`listen_addresses = ''`, server time zone UTC, `conf.d/packetstation.conf`). `pg_hba.conf` gets one block, before Debian's peer line, that lets `ps_app` and `ps_migrate` log in to `packetstation` on the socket with their passwords (scram-sha-256). `install.sh` creates the two roles and the database as `deploy/postgres/initdb` does (owner `ps_migrate`, UTF8, locale `en_US.UTF-8`, `template0`, CONNECT and TEMPORARY for `ps_app`, schema `public` owned by `ps_migrate`). The env file sets `PG_HOST=/var/run/postgresql`.

| Secret file | Made by | Given to |
|---|---|---|
| `postgres_migrate_password` | `install.sh --generate-secrets` | migrate |
| `postgres_app_password` | `install.sh --generate-secrets` | migrate, app, worker |
| `session_key` | `install.sh --generate-secrets` | app |
| `seal_key` | `install.sh --generate-secrets` | app, worker |
| `visualkit_password` | Spencer | worker |
| `mfileskit_password` | Spencer | worker (`ps-launch.sh` exports `MFILES_PASSWORD`) |
| `ldap_bind_password` | Spencer | app, worker |

There is no PostgreSQL superuser password on a VM: root reaches the `postgres` OS user, which PostgreSQL trusts on the socket. `ps-run` hands every secret file present to the command it runs.

### The units

| Unit | What |
|---|---|
| `packetstation-migrate.service` | Oneshot, `RemainAfterExit`: `packetstation migrate` as `ps_migrate`. Pulled in by the app and the worker (`Requires=`), so neither starts on a failed migration. `ps-upgrade` restarts it. |
| `packetstation-app.service` | `packetstation app`, uvicorn on port 8000. Refuses to start without the firewall table. `Restart=on-failure`, `MemoryMax=1G`. |
| `packetstation-worker.service` | `packetstation worker`. Heartbeat in `/run/packetstation/worker.beat`. `Restart=on-failure`, `MemoryMax=1G`. |
| `packetstation-firewall.service` | Loads `/etc/packetstation/packetstation.nft`: its own table `inet packetstation`, which lets only `TRUSTED_PROXY_HOST` (NPM) and the VM itself reach port 8000. Nothing else on the VM is filtered by it. |
| `packetstation-backup.service`, `.timer` | `ps-backup` at 02:00 America/Denver. |

The app, the worker and migrate run as `packetstation` with `EnvironmentFile=/etc/packetstation/packetstation.env` and a sandbox: `ProtectSystem=strict` (writable only `/var/lib/packetstation/assets` and `/mnt/packetstation-archive`, the latter optional so the app starts without the share), `ProtectHome`, `PrivateTmp`, `PrivateDevices`, `NoNewPrivileges`, no capabilities, `InaccessiblePaths=/etc/packetstation/secrets`, `SystemCallFilter=@system-service`, the address families `AF_UNIX AF_INET AF_INET6 AF_NETLINK`, and the kernel protections. `systemd-analyze security` rates the app 1.5 ("OK"). `MemoryDenyWriteExecute` is left off: pypdfium2 uses ctypes, whose callbacks may need writable executable memory. Try it on the test VM after a filing and a print pass before turning it on.

Each unit loads only its own secrets with `LoadCredential=`, as compose did with secrets: the app never gets the VISUAL or M-Files password, and migrate gets only the two database passwords. `deploy/vm/ps-launch.sh` (the native counterpart of `docker/entrypoint.sh`) points each `*_FILE` setting at the credential, exports `MFILES_PASSWORD` from its file, and sets `STATIC_DIR` and `PS_BUILD` from the release, `WORKER_BEAT_FILE`, `TZ=UTC` and `LANG=C.UTF-8`. Those five are ignored in the env file.

Logs go to the journal: `journalctl -u packetstation-app -u packetstation-worker -f`.

### Commands on a VM

| Command (as root) | What |
|---|---|
| `ps-doctor [--skip visual,mfiles,ldap,smtp,archive]` | `packetstation doctor`, run the way the services run. |
| `ps-run <packetstation command>` | Any `packetstation` command as the service user, with the env file, the secrets as credentials and the sandbox, through `systemd-run`. For example `ps-run job run visual-feed`, `ps-run ad-groups --list`, `ps-run healthcheck worker`, `ps-run version`. |
| `ps-upgrade RELEASE.tar.gz [--start] [--no-backup] [--no-doctor]` | Install a release and switch to it (below). |
| `ps-upgrade --rollback` | Back to the previous release. |
| `ps-backup`, `ps-restore` | "Backups on a VM", "Restore on a VM". |

On production the doctor's VISUAL check connects to VISUAL production, read-only, as `visualkit_ro`. Use `--skip visual` before the go-live approval.

### Release archives

A VM never builds the frontend and never runs Docker. `deploy/vm/make-release.sh` runs on a build host as a normal user, in a git checkout, and writes `packetstation-<build>.tar.gz` with a `.sha256`: the source of one commit (`pyproject.toml`, `README.md`, `server/`, `deploy/`, `scripts/`, `docs/`, with LF line endings), `BUILD`, and `static/`. Python packages are not in it.

- `--image gt.pescoinc.biz/spencerm/packetstation:sha-<sha>` (on docker-1): copies `/app/static` out of the CI-tested image with `docker create` and `docker cp` (nothing runs) and takes the commit from the image's build label. The release is then the tested commit with the tested frontend. This is the normal way.
- `--npm [--ref REF]` (on a PC with Node): builds the frontend with `npm ci` and `npm run build` from the same commit.

`ps-upgrade` builds the venv on the VM, as the image does: `server/requirements.txt` from PyPI with `--index-url https://pypi.org/simple/`, `server/requirements-pesco.txt` from the Gitea index with `--index-url` (never `--extra-index-url`), then the package with `--no-deps`, then `pip check`. Host pip settings are ignored (`PIP_CONFIG_FILE=/dev/null`).

### First install (Spencer, on a new VM)

The test VM first. `<build>` is the release, for example `sha-<commit>`.

1. On docker-1, in a git checkout of the repository that has the commit (for example a clone of `gt.pescoinc.biz/spencerm/PacketStation`):

   ```bash
   deploy/vm/make-release.sh --image gt.pescoinc.biz/spencerm/packetstation:sha-<sha> --out ~/releases
   ```

2. Copy `~/releases/packetstation-<build>.tar.gz`, its `.sha256` and the PESCO CA bundle to the VM. The bundle comes from `scripts/Export-PescoCaBundle.ps1 -NoCopy` (or `-Remote` with the VM's path), or from `~/packetstation/test/ca/pesco-ca.pem` on docker-1.

3. On the VM, unpack and run the install. It installs the packages, makes the user, folders, env file, CA, the four internal secrets, PostgreSQL's settings, roles and database, the units, the commands, the backup timer and the firewall:

   ```bash
   cd ~ && sha256sum -c packetstation-<build>.tar.gz.sha256
   tar -xzf packetstation-<build>.tar.gz
   sudo bash ./packetstation-<build>/deploy/vm/install.sh --stack test --ca-bundle ~/pesco-ca.pem --generate-secrets
   ```

   For production, `--stack prod`. If the VM is to carry on the docker-1 test stack's data, copy that stack's `seal_key` and `session_key` into `/etc/packetstation/secrets/` before this step instead of generating them: filed packets are sealed with the seal key.

4. Put the three outside secrets in place, without echo, then run the install again so it checks them and sets their owner and mode:

   ```bash
   for s in visualkit_password mfileskit_password ldap_bind_password; do
     sudo bash -c 'umask 077; read -rsp "$1: " v; echo; printf %s "$v" > "/etc/packetstation/secrets/$1"' _ "$s"
   done
   sudo bash ./packetstation-<build>/deploy/vm/install.sh --skip-packages
   ```

5. Read `/etc/packetstation/packetstation.env` through (`sudoedit`). Check `STACK`, `PUBLIC_URL`, `VISUAL_SERVER` and `VISUAL_DATABASE` (test: `'vsql.pesco.com\DEV'` and `PESCOINCDEV`; production: `vsql.pesco.com` and `PESCOINC`), `NOTIFY_DRY_RUN` (test `true`), and `TRUSTED_PROXY_HOST=192.168.101.12`. After changing `TRUSTED_PROXY_HOST`, run `install.sh --files-only` to rewrite the firewall table.

6. Install the release. It builds the venv, runs the migrations and the doctor, and starts nothing on a first install:

   ```bash
   sudo ps-upgrade ~/packetstation-<build>.tar.gz
   ```

7. Run the doctor until it is clean (on production with `--skip visual` until the approval):

   ```bash
   sudo ps-doctor
   ```

8. The archive share, when the QNAP export exists ("Archive NFS mount").

9. Start. On production this is the approval of the scheduled VISUAL polling:

   ```bash
   sudo systemctl enable --now packetstation-app packetstation-worker
   systemctl status packetstation-app packetstation-worker --no-pager
   curl -fsS http://127.0.0.1:8000/healthz
   ```

10. Point the NPM proxy host at the VM ("NPM proxy host for the VMs") and run its checks.

11. Backups: `sudo systemctl start packetstation-backup.service`, then `sudo ls -la /var/backups/packetstation`. Then rehearse a restore ("Restore on a VM").

On Ubuntu with `ufw` turned on, ufw also has to let docker-1 in: `sudo ufw allow from 192.168.101.12 to any port 8000 proto tcp`. The Packet Station table only narrows port 8000 further; it never opens anything ufw blocks.

### Upgrades

Build the archive for the new commit (step 1), copy it over (step 2), then:

```bash
sudo ps-upgrade ~/packetstation-<build>.tar.gz
```

In order: check the `.sha256` when it lies next to the archive, unpack into `/opt/packetstation/<build>` and build its venv (an already complete release folder is reused), take a backup with `ps-backup`, stop the app and the worker, point `current` at the new release (`previous` keeps the old), install its units and commands (`install.sh --files-only`), run the migrations, start again whichever of the two ran before, wait for `/healthz`, run the doctor, and keep the newest three releases. A short downtime window, as with a new image.

If the migrations fail, `current` goes back to the old release, the app and the worker stay stopped, and the restore command for the backup taken a minute earlier is printed. Read `journalctl -u packetstation-migrate`.

`sudo ps-upgrade --rollback` points `current` back at `previous` and restarts what ran. It does not undo migrations: roll back only to a release whose code works with the schema in the database, or restore the backup.

Open screens see the build change and offer a reload (the version banner).

### NPM proxy host for the VMs

NPM stays on docker-1 and keeps the certificates. The DNS names keep pointing at docker-1 (192.168.101.12). Only the forward target of each proxy host changes, to its VM over the LAN:

| | Production | Test |
|---|---|---|
| Domain name | `packetstation.pescoinc.biz` | `packetstation-test.pescoinc.biz` |
| Scheme | `http` | `http` |
| Forward hostname / IP | the production VM's address | the test VM's address |
| Forward port | `8000` | `8000` |
| `TRUSTED_PROXY_HOST` on the VM | `192.168.101.12` | `192.168.101.12` |

Everything else is as before (`deploy/npm-proxy-host.md`): Cache assets off, Block common exploits on, Websockets off, no access list; the PESCO certificate, Force SSL on, HTTP/2 on, NPM HSTS off (the app sends `Strict-Transport-Security` itself; two headers would conflict); and the same Advanced block, unchanged:

```nginx
client_max_body_size 30m;

location /api/events {
    proxy_pass $forward_scheme://$server:$port;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}
```

With a fixed VM address the forward target can be the address itself; the "never an IP" rule was for container addresses, which change. `$server` keeps the events location on the same target as the rest of the host.

Trust: NPM's container reaches the VM through docker-1's own address, so the VM sees every proxied request from 192.168.101.12. `TRUSTED_PROXY_HOST=192.168.101.12` makes the app trust `X-Forwarded-For` and `X-Forwarded-Proto` from that address only (`proxy.py` resolves an address literal to itself; no server change was needed), and the firewall table lets only that address reach port 8000. Other hosts on the LAN cannot reach the app directly, and so cannot fake a client address for the sign-in rate limit. Traffic from NPM to the VM is plain HTTP on the LAN; TLS to the VM is still open on issue 37.

Check after switching: the four checks in `deploy/npm-proxy-host.md`, plus from another LAN PC `curl -m 5 http://<vm>:8000/healthz` must time out (the firewall), and the Admin page's access log shows the real client addresses, not 192.168.101.12.

### Archive NFS mount

The archive share (Gitea issue 7) lives on the QNAP. Spencer creates the NFS export with read and write for the VMs' addresses and makes the share's folder owned by uid 10001 and gid 10001 (or maps the VM's uid 10001 to an owner with write rights). Root squash on is fine: only `packetstation` writes there.

1. `nfs-common` is installed by `install.sh`. Add one line to `/etc/fstab` (the QNAP's name and export path from IT):

   ```
   <qnap>:/<export>  /mnt/packetstation-archive  nfs4  rw,soft,timeo=150,retrans=3,noatime,nosuid,nodev,noexec,_netdev,nofail  0  0
   ```

   `soft`: an unreachable QNAP makes a copy fail after about a minute instead of hanging the worker for good. The archive job writes each file to a temporary name, fsyncs and renames it, so a failed write shows up as a failed copy and is tried again with backoff. Filing never waits on it. `nofail` lets the VM boot without the QNAP.

2. Mount it and make the archive folder as the service user:

   ```bash
   sudo mount /mnt/packetstation-archive
   sudo setpriv --reuid=packetstation --regid=packetstation --init-groups mkdir /mnt/packetstation-archive/packets
   ```

3. Set `ARCHIVE_ROOT=/mnt/packetstation-archive` in the env file (uncomment the line) and restart: `sudo systemctl restart packetstation-app packetstation-worker`.

4. On the Admin page, File locations, enter `packets`, press Check, then Save. `sudo ps-doctor` reports `archive` PASS from then on.

While the share is not mounted, the mount point is root's and read-only to `packetstation`, and `packets` does not exist under it, so nothing is written to the VM's own disk and the check fails with a plain message. The units list the mount point as an optional writable path, so the app and the worker start without it.

### Retiring the docker-1 test stack

The docker-1 test stack keeps running until the test VM passes its doctor, the NPM switch, a backup and a restore rehearsal. To carry its data over instead of starting empty: take a dump with the backup already installed on docker-1, tar the assets volume as in the 2026-10-21 rehearsal, copy both and the `seal_key` and `session_key` to the VM, and restore with `ps-restore` ("Restore on a VM"). Then on docker-1: stop the stack (`docker compose -p packetstation-test ... down`, keeping its volumes until the VM has run for a while), and remove its line from docker-1's `/etc/packetstation/backup.conf`. docker-1 keeps the old `ps-backup.sh` and `ps-restore.sh` it has installed; the versions in the repo are the VM ones and do not run there.

## Backups on a VM

`scripts/backup/ps-backup.sh` (installed as `/usr/local/sbin/ps-backup`) runs from `packetstation-backup.timer`, as root, outside the app, so a broken app still backs up. [scripts/backup/README.md](../scripts/backup/README.md) has the file list.

- Timer `OnCalendar=*-*-* 02:00:00 America/Denver`, `Persistent=true`: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot. `install.sh` enables it.
- `/etc/packetstation/backup.conf`: one line, `name database folder`, for example `prod packetstation /var/backups/packetstation`. `install.sh` writes it.
- `pg_dump -d packetstation -Fc` and `pg_dumpall --globals-only --no-role-passwords` run as the `postgres` OS user (`setpriv`, no PAM) on the local socket. No password is read or passed. Root writes the output under a temporary name, `pg_restore --list` must list at least one entry, then the files are renamed to `ps-<UTC yyyymmddThhmmssZ>.dump` and `.globals.sql`, mode 0600.
- The newest 7 are kept (`KEEP=7` in the unit, `--keep N`), with their globals files. `last-success` or `last-failure` (`<ISO UTC time> <dump name>`, plus the reason on failure), mode 0644, owner and group taken from the folder (root:packetstation 0750). The worker's `backup-check` job (07:15) reads them through `BACKUP_DIR=/var/backups/packetstation`. A failed run leaves no partial file. `flock` keeps two runs apart. Exit 1 on a failure, 2 on a usage or config error.
- The unit runs with `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=1h`, `ProtectSystem=strict` (writable only `/var/backups/packetstation`), `ProtectHome`, `PrivateTmp`, `NoNewPrivileges`, and only the capabilities to switch user and set owners.
- `ps-upgrade` takes a backup before every upgrade.

What the dump does not hold, and what covers it:

| What | Where | Covered by |
|---|---|---|
| Stored files | `/var/lib/packetstation/assets` | PBS (the VM). Restore with a PBS file-level restore and `ps-restore --assets`. |
| Settings, secrets, CA | `/etc/packetstation` | PBS. The secrets are also needed to rebuild a VM: without `seal_key` the seals of filed packets cannot be checked. Keep a copy of `/etc/packetstation/secrets` where IT keeps the other service secrets. |
| Archive copies | the QNAP share | The QNAP's own backup. |

Check by hand:

```bash
sudo systemctl start packetstation-backup.service
journalctl -u packetstation-backup.service -n 20 --no-pager
sudo ls -la /var/backups/packetstation
systemctl list-timers packetstation-backup.timer      # next run 08:00 or 09:00 UTC, 02:00 in Denver
```

## Restore on a VM

`ps-restore` (`scripts/backup/ps-restore.sh`) never stops, starts or reconfigures a service. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.

```bash
B=/var/backups/packetstation
D=$(sudo ls "$B" | grep '\.dump$' | tail -1)

# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is
#    connected). --replace is refused for packetstation itself.
sudo ps-restore --dump "$B/$D" --globals "$B/${D%.dump}.globals.sql" --db ps_restored

# 2. Stored files, from a folder or a tar (a PBS file-level restore). Existing files are never overwritten.
#    Files end up owned by packetstation. Use a scratch folder for a rehearsal.
sudo ps-restore --assets /path/to/assets.tar --assets-dir /var/lib/packetstation/assets

# 3. Every asset row's sha256 against the files.
sudo ps-restore --verify-assets --db ps_restored --assets-dir /var/lib/packetstation/assets

# Row counts of any database, to compare with the live one.
sudo ps-restore --counts-only --db packetstation
```

- The database is created as `install.sh` creates the live one (owner `ps_migrate`, UTF8, `en_US.UTF-8`, `template0`, CONNECT and TEMPORARY for `ps_app`), then `pg_restore --exit-on-error --single-transaction` as postgres. A failed restore leaves an empty database, never a half one.
- The globals file is applied without stopping on "already exists". Any other error fails. A globals file with a password hash is refused. Roles created from it have no password: run `install.sh --skip-packages`, which sets both role passwords from the secrets.
- To put a restored database into service: `sudo systemctl stop packetstation-app packetstation-worker`; as postgres (`sudo setpriv --reuid=postgres --regid=postgres --init-groups psql -X -d postgres`) run `ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`; then `sudo systemctl restart packetstation-migrate` and `sudo systemctl start packetstation-app packetstation-worker` (migrate brings an older dump up to the release), check the app, and drop the old database. This step is deliberate and manual.
- A rebuilt VM after a disaster: copy the saved `/etc/packetstation/secrets` back first, run `install.sh`, `ps-upgrade` with the same release, then the restore and the rename above (the install made an empty `packetstation`), and the assets.
- The next rehearsal is IT's, quarterly (requirement O2). The first one on the test VM is part of its acceptance.

## Backup and restore on docker-1 (phase 9, until the test VM)

Until the test VM takes over, the docker-1 test stack is backed up by the phase 9 scripts installed on docker-1 as `/usr/local/sbin/ps-backup.sh` and `ps-restore.sh`. They reach the stack's postgres container with `docker exec` and read `/etc/packetstation/backup.conf` in the four-column form `name container database folder` (`test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups`). Those versions are in git history (up to commit 3fdf063). The scripts in the repo are now the VM ones and refuse that config, so do not reinstall from the repo on docker-1. Markers, rotation and timer are the same as on a VM. The rehearsal below was made with them.

### Restore rehearsal 2026-10-21

Done on docker-1 against the test stack (`packetstation-test`, image `sha-ce28d2f`, container `packetstation-test-postgres-1`, postgres:16-bookworm by digest), as spencerm without sudo, from a scratch copy of the scripts. The stack was not stopped, restarted or changed. Times UTC.

| Step | Result |
|---|---|
| Backup by hand into `/home/spencerm/packetstation/test/backups` | 21:24:05 to 21:24:06, 0.76 s. `ps-20261021T212405Z.dump`, 2,234,906 bytes, 743 `pg_restore --list` entries. Globals 774 bytes with no password line (roles postgres, ps_app, ps_migrate, attributes only). `last-success` written. Dump and globals 0600, marker 0644, all spencerm. Exit 0. This dump stays in the folder. |
| Failure path (a database that does not exist, into a scratch folder) | Exit 1, pg_dump's error on stderr, `last-failure` = `2026-10-21T21:24:53Z ps-20261021T212453Z.dump pg_dump failed`, no partial file. |
| Live row counts | `--counts-only --db packetstation`: 64 tables. |
| Restore into `ps_restore_rehearsal` with the globals | Globals: 3 statements skipped (the roles exist), no other error, no password touched. Created and restored in 1.65 s. Every one of the 64 tables has the same row count as the live database, checked just before and just after (for example assets 189, audit_events 2137, bom_lines 5742). |
| Assets out of the live volume | Tarred through a throwaway container (`--network none`, read-only mount): 129,546,240 bytes, 189 files, 3.8 s. sha256 of every file against the 189 `assets` rows of the restored database: 189 match, 0 differ, 0 missing, 0 files without a row. |
| Asset restore path | `--assets <the tar> --assets-volume ps_restore_rehearsal_assets` into a throwaway volume, 0.47 s. `--verify-assets`: 189 rows, 189 match, 0 differ, 0 missing, 0 orphans. A second copy changed nothing (no overwrite). Files owned 10001:10001. |
| Guard | Restoring again into `ps_restore_rehearsal` without `--replace`: refused, exit 1. |
| Cleanup | `DROP DATABASE ps_restore_rehearsal`, the throwaway volume and the scratch folder removed. The cluster lists only packetstation, postgres, template0 and template1. App, worker and postgres stayed healthy throughout. |
| Timer | `systemd-analyze calendar "*-*-* 02:00:00 America/Denver"`: next run 2026-10-22 08:00 UTC. `systemd-analyze verify` on both units found only the script not installed yet. systemd 255. |

Nothing went wrong. Notes from it:

- The dump is small (2.2 MB). The assets (125 MB) are the bulk and are not in the nightly dump. PBS covers them; the restore path is a PBS file-level restore of the volume folder, or a tar, through `--assets`.
- The globals restore re-applies role attributes. On the same cluster that changes nothing. On a new cluster it creates the roles without passwords: set them from the secrets.
- The next rehearsal is IT's, quarterly (requirement O2).