spencerm/PacketStation
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Native VM install kit: systemd units, launcher, install and upgrade scripts; backups, restore and doctor without docker exec; DEPLOY.md VM, NPM and NFS sections

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
spencerm
2026-10-30 18:08:48 -06:00
co-authored by Claude Opus 5.5
parent f025479358
commit 45a1d1a42a
22 changed files with 2357 additions and 413 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOY.md
+256 -80
View File
@@ -1,6 +1,6 @@
# Deploying Packet Station
How Packet Station runs on `docker-1`. Sections not built yet are marked "planned, phase N". The design is in [PLAN.md](PLAN.md) under "Deployment and CI".
How Packet Station runs. Production and test move to their own native Linux VMs in phase 10 ("Native VM install"). Until the test VM is ready the test stack runs in Docker on `docker-1`, which keeps CI. Sections not built yet are marked "planned, phase N". The design is in [PLAN.md](PLAN.md) under "Deployment and CI".
## Host facts
@@ -59,7 +59,7 @@ The PESCO CA chain is mounted into the containers once and referenced by `LDAP_C
## Env files
One folder per stack. The test stack: `/home/spencerm/packetstation/test/`, like the other compose stacks on docker-1 (decided 2026-10-18, Gitea issue 38). It holds `stack.env`, `secrets/`, `ca/pesco-ca.pem` and `backups/`, and its `stack.env` sets `ENV_FILE`, `SECRETS_DIR`, `CA_BUNDLE_FILE` and `BACKUP_DIR` to those paths. The compose files run from the checkout in `/home/spencerm/packetstation/repo/deploy`. On 2026-10-27 Spencer decided that production and test each run natively on their own Linux VM, without Docker, and that docker-1 keeps CI only (Gitea issue 37). The test stack stays here until the test VM is ready. The production lines in this file describe the old compose plan and are rewritten for the VMs in phase 10. Each folder holds `stack.env` (settings only, no secrets) and `secrets/`. The env file sets `PROJECT_NAME`, `SECRETS_DIR` and `APP_ALIAS`, which the compose files require. `.env.example` in the repo lists every setting.
One folder per stack. The test stack: `/home/spencerm/packetstation/test/`, like the other compose stacks on docker-1 (decided 2026-10-18, Gitea issue 38). It holds `stack.env`, `secrets/`, `ca/pesco-ca.pem` and `backups/`, and its `stack.env` sets `ENV_FILE`, `SECRETS_DIR`, `CA_BUNDLE_FILE` and `BACKUP_DIR` to those paths. The compose files run from the checkout in `/home/spencerm/packetstation/repo/deploy`. On 2026-10-27 Spencer decided that production and test each run natively on their own Linux VM, without Docker, and that docker-1 keeps CI only (Gitea issue 37). The test stack stays here until the test VM is ready. The VMs keep their settings in `/etc/packetstation/packetstation.env` and their secrets in `/etc/packetstation/secrets/`: see "Native VM install". The production compose lines in this file are the old plan, kept for reference. Each folder holds `stack.env` (settings only, no secrets) and `secrets/`. The env file sets `PROJECT_NAME`, `SECRETS_DIR` and `APP_ALIAS`, which the compose files require. `.env.example` in the repo lists every setting.
Key settings:
@@ -132,7 +132,7 @@ The Gitea PyPI index `https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/
## NPM proxy host
One proxy host per stack in Nginx Proxy Manager. To be recorded under `deploy/` as well.
One proxy host per stack in Nginx Proxy Manager, recorded in `deploy/npm-proxy-host.md`. For the VMs see "NPM proxy host for the VMs": only the forward target and `TRUSTED_PROXY_HOST` change.
- Domain: `packetstation-test.pescoinc.biz` (test) or `packetstation.pescoinc.biz` (production).
- Scheme `http`, forward host the stack's alias (`packetstation-test-app` or `packetstation-app`), port `8000`.
@@ -241,7 +241,7 @@ Docker's default address pool on docker-1 holds fifteen /16 bridge networks (172
## Doctor
`packetstation doctor` proves a host and a stack are ready. Run it with `compose run`, not `compose exec`: the entrypoint exports MFILES_PASSWORD from its secret file, and `exec` skips the entrypoint, so the M-Files password check fails under `exec` even though the running worker has it.
`packetstation doctor` proves a host and a stack are ready. On a VM run `sudo ps-doctor` ("Commands on a VM"). On docker-1 run it with `compose run`, not `compose exec`: the entrypoint exports MFILES_PASSWORD from its secret file, and `exec` skips the entrypoint, so the M-Files password check fails under `exec` even though the running worker has it.
```bash
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
@@ -340,113 +340,289 @@ The app sends test messages and the worker sends everything else, through PESCO'
The doctor's SMTP line checks STARTTLS with certificate verification, without sending.
## Backup (phase 9)
## Native VM install (phase 10)
Host scripts in `scripts/backup/`, run by root from a systemd timer. They run outside the app, so a broken app still backs up. [scripts/backup/README.md](../scripts/backup/README.md) has the same steps with more detail.
Production and test each run natively on their own Linux VM, without Docker (Spencer, 2026-10-27, Gitea issue 37). docker-1 keeps CI only. The test VM is built first from the same steps, so the install, the backups, the restore and the doctor are rehearsed before production. The docker-1 test stack keeps running until the test VM is ready (see "Retiring the docker-1 test stack").
| File | Installed as |
The kit is in `deploy/vm/`. Spencer runs everything that needs root. Nothing in it starts the worker by itself: on production, Spencer's first start of the worker is the approval of the scheduled VISUAL polling.
### What IT provides (open on issue 37)
- One VM per stack. Ubuntu 24.04 LTS is recommended: it ships Python 3.12, which the server needs (`requires-python >= 3.12`, pins resolved for 3.12). Debian 12 ships Python 3.11, so on Debian 12 a Python 3.12 has to be provided separately and passed to `ps-upgrade` as `PYTHON=/path/to/python3.12`. `install.sh` supports both and refuses anything else. Size: 2 vCPU, 4 GB RAM and 40 GB disk is a starting point (the app and the worker are capped at 1 GB each; the test stack's assets were 125 MB after a month of testing).
- A fixed LAN address per VM. NPM forwards to it.
- Outbound from the VM: the OS mirrors, `apt.postgresql.org`, `packages.microsoft.com`, `www.postgresql.org` (the PGDG key), `pypi.org` and `files.pythonhosted.org`, `gt.pescoinc.biz` (the Python index), and at run time vsql (1433, plus UDP 1434 for the named DEV instance), `mfiles.pescoinc.biz` 443, DC-02 and PDC-1 636, `app-03.pesco.com` 25, the QNAP (NFS 2049).
- Inbound: SSH for Spencer, and port 8000 from docker-1 (192.168.101.12) only. The VM enforces the second itself.
- PBS backs the VM up like any other, at 03:00.
### Layout on a VM
| Path | Owner, mode | What |
|---|---|---|
| `/opt/packetstation/<build>/` | root, 0755 | One release: the source, `BUILD`, `static/` (the built frontend) and `venv/`. Read-only to the services. |
| `/opt/packetstation/current`, `previous` | root | Symlinks to the running and the last release. The units start `current/deploy/vm/ps-launch.sh`. |
| `/etc/packetstation/packetstation.env` | root:packetstation 0640 | Settings, no secrets. From `deploy/vm/prod.env.example` or `test.env.example`. |
| `/etc/packetstation/secrets/` | root:packetstation 0750, files 0440 | One file per secret (table below). The services get them as systemd credentials and cannot open the folder. |
| `/etc/packetstation/ca/pesco-ca.pem` | root, 0644 | The PESCO CA chain for `LDAP_CA_BUNDLE` and `SMTP_CA_BUNDLE`. Also in the system trust store, which the ODBC driver uses. |
| `/etc/packetstation/backup.conf`, `packetstation.nft` | root, 0644 | The backup stack line; the firewall table (written by `install.sh`). |
| `/var/lib/packetstation/assets/` | packetstation, 0750 | `ASSETS_DIR`, the stored files. |
| `/var/backups/packetstation/` | root:packetstation 0750 | `BACKUP_DIR`, the nightly dumps and markers. |
| `/mnt/packetstation-archive/` | root, 0755 until mounted | `ARCHIVE_ROOT`, the QNAP NFS share. |
| `/usr/local/sbin/ps-run`, `ps-doctor`, `ps-upgrade`, `ps-backup`, `ps-restore` | root, 0755 | The commands below. |
The service user is `packetstation`, uid and gid 10001: the image's uid, so the NFS share's rights and the old volume's file owners carry over. PostgreSQL 16 from the PGDG repository listens on the local socket only (`listen_addresses = ''`, server time zone UTC, `conf.d/packetstation.conf`). `pg_hba.conf` gets one block, before Debian's peer line, that lets `ps_app` and `ps_migrate` log in to `packetstation` on the socket with their passwords (scram-sha-256). `install.sh` creates the two roles and the database as `deploy/postgres/initdb` does (owner `ps_migrate`, UTF8, locale `en_US.UTF-8`, `template0`, CONNECT and TEMPORARY for `ps_app`, schema `public` owned by `ps_migrate`). The env file sets `PG_HOST=/var/run/postgresql`.
| Secret file | Made by | Given to |
|---|---|---|
| `postgres_migrate_password` | `install.sh --generate-secrets` | migrate |
| `postgres_app_password` | `install.sh --generate-secrets` | migrate, app, worker |
| `session_key` | `install.sh --generate-secrets` | app |
| `seal_key` | `install.sh --generate-secrets` | app, worker |
| `visualkit_password` | Spencer | worker |
| `mfileskit_password` | Spencer | worker (`ps-launch.sh` exports `MFILES_PASSWORD`) |
| `ldap_bind_password` | Spencer | app, worker |
There is no PostgreSQL superuser password on a VM: root reaches the `postgres` OS user, which PostgreSQL trusts on the socket. `ps-run` hands every secret file present to the command it runs.
### The units
| Unit | What |
|---|---|
| `ps-backup.sh` | `/usr/local/sbin/ps-backup.sh` (root, 0755) |
| `ps-restore.sh` | `/usr/local/sbin/ps-restore.sh` (root, 0755) |
| `backup.conf.example` | `/etc/packetstation/backup.conf` (root, 0644) |
| `packetstation-backup.service` | `/etc/systemd/system/` (root, 0644) |
| `packetstation-backup.timer` | `/etc/systemd/system/` (root, 0644) |
| `packetstation-migrate.service` | Oneshot, `RemainAfterExit`: `packetstation migrate` as `ps_migrate`. Pulled in by the app and the worker (`Requires=`), so neither starts on a failed migration. `ps-upgrade` restarts it. |
| `packetstation-app.service` | `packetstation app`, uvicorn on port 8000. Refuses to start without the firewall table. `Restart=on-failure`, `MemoryMax=1G`. |
| `packetstation-worker.service` | `packetstation worker`. Heartbeat in `/run/packetstation/worker.beat`. `Restart=on-failure`, `MemoryMax=1G`. |
| `packetstation-firewall.service` | Loads `/etc/packetstation/packetstation.nft`: its own table `inet packetstation`, which lets only `TRUSTED_PROXY_HOST` (NPM) and the VM itself reach port 8000. Nothing else on the VM is filtered by it. |
| `packetstation-backup.service`, `.timer` | `ps-backup` at 02:00 America/Denver. |
`backup.conf` lists the stacks, one per line: name, container (or `project:<compose project>`), database, folder.
The app, the worker and migrate run as `packetstation` with `EnvironmentFile=/etc/packetstation/packetstation.env` and a sandbox: `ProtectSystem=strict` (writable only `/var/lib/packetstation/assets` and `/mnt/packetstation-archive`, the latter optional so the app starts without the share), `ProtectHome`, `PrivateTmp`, `PrivateDevices`, `NoNewPrivileges`, no capabilities, `InaccessiblePaths=/etc/packetstation/secrets`, `SystemCallFilter=@system-service`, the address families `AF_UNIX AF_INET AF_INET6 AF_NETLINK`, and the kernel protections. `systemd-analyze security` rates the app 1.5 ("OK"). `MemoryDenyWriteExecute` is left off: pypdfium2 uses ctypes, whose callbacks may need writable executable memory. Try it on the test VM after a filing and a print pass before turning it on.
```
test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
#prod project:packetstation packetstation /var/backups/packetstation/prod
```
Each unit loads only its own secrets with `LoadCredential=`, as compose did with secrets: the app never gets the VISUAL or M-Files password, and migrate gets only the two database passwords. `deploy/vm/ps-launch.sh` (the native counterpart of `docker/entrypoint.sh`) points each `*_FILE` setting at the credential, exports `MFILES_PASSWORD` from its file, and sets `STATIC_DIR` and `PS_BUILD` from the release, `WORKER_BEAT_FILE`, `TZ=UTC` and `LANG=C.UTF-8`. Those five are ignored in the env file.
The production line stays commented out until the production stack runs (phase 10). Until then it would fail every night.
Logs go to the journal: `journalctl -u packetstation-app -u packetstation-worker -f`.
Where the dumps go:
### Commands on a VM
- Test stack: `/home/spencerm/packetstation/test/backups`, owned by spencerm (the script gives new files the folder's owner and group). The test `stack.env` sets `BACKUP_DIR` to this path.
- Production: `/var/backups/packetstation/prod`, `root:10001` 0750. The production `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
| Command (as root) | What |
|---|---|
| `ps-doctor [--skip visual,mfiles,ldap,smtp,archive]` | `packetstation doctor`, run the way the services run. |
| `ps-run <packetstation command>` | Any `packetstation` command as the service user, with the env file, the secrets as credentials and the sandbox, through `systemd-run`. For example `ps-run job run visual-feed`, `ps-run ad-groups --list`, `ps-run healthcheck worker`, `ps-run version`. |
| `ps-upgrade RELEASE.tar.gz [--start] [--no-backup] [--no-doctor]` | Install a release and switch to it (below). |
| `ps-upgrade --rollback` | Back to the previous release. |
| `ps-backup`, `ps-restore` | "Backups on a VM", "Restore on a VM". |
The worker mounts the stack's folder read-only at `/backups`. Its `backup-check` job (07:15) only lists the folder, reads the dumps' sizes and times, and reads the markers.
On production the doctor's VISUAL check connects to VISUAL production, read-only, as `visualkit_ro`. Use `--skip visual` before the go-live approval.
What a run does, per stack:
### Release archives
1. Finds the stack's running postgres container (by compose labels for `project:`). None or two is a failure.
2. `docker exec <pg> pg_dump -U postgres -d packetstation -Fc` over the local socket, which the postgres image trusts. No password is read or passed. Written under a temporary name, `.ps-<stamp>.partial`.
3. `pg_dumpall --globals-only --no-role-passwords`: the roles without their password hashes. The role passwords are in the stack secrets.
4. `pg_restore --list` with the container's own pg_restore must list at least one entry. Then the files are renamed to `ps-<UTC yyyymmddThhmmssZ>.dump` and `.globals.sql`, mode 0600.
5. Keeps the newest 7 (`KEEP=7` in the unit, `--keep N`), with their globals files.
6. Writes `last-success` or `last-failure` (`<ISO UTC time> <dump name>`, plus the reason on failure), mode 0644. A failed run leaves no partial file.
A VM never builds the frontend and never runs Docker. `deploy/vm/make-release.sh` runs on a build host as a normal user, in a git checkout, and writes `packetstation-<build>.tar.gz` with a `.sha256`: the source of one commit (`pyproject.toml`, `README.md`, `server/`, `deploy/`, `scripts/`, `docs/`, with LF line endings), `BUILD`, and `static/`. Python packages are not in it.
Each stack runs on its own, so one failing does not stop the next. Exit 1 if any stack failed, 2 for a usage or config error. `flock` keeps two runs from overlapping.
- `--image gt.pescoinc.biz/spencerm/packetstation:sha-<sha>` (on docker-1): copies `/app/static` out of the CI-tested image with `docker create` and `docker cp` (nothing runs) and takes the commit from the image's build label. The release is then the tested commit with the tested frontend. This is the normal way.
- `--npm [--ref REF]` (on a PC with Node): builds the frontend with `npm ci` and `npm run build` from the same commit.
The unit is `Type=oneshot` as root, `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=1h`, with `NoNewPrivileges`, `PrivateTmp` and `ProtectSystem=full` (no `ProtectHome`, so it can write the test folder under `/home`). The timer is `OnCalendar=*-*-* 02:00:00 America/Denver` with `Persistent=true`: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot.
`ps-upgrade` builds the venv on the VM, as the image does: `server/requirements.txt` from PyPI with `--index-url https://pypi.org/simple/`, `server/requirements-pesco.txt` from the Gitea index with `--index-url` (never `--extra-index-url`), then the package with `--no-deps`, then `pip check`. Host pip settings are ignored (`PIP_CONFIG_FILE=/dev/null`).
### Install (Spencer, with sudo, on docker-1)
### First install (Spencer, on a new VM)
After the phase 9 tree is synced to `~/packetstation/repo`:
The test VM first. `<build>` is the release, for example `sha-<commit>`.
1. On docker-1, in a git checkout of the repository that has the commit (for example a clone of `gt.pescoinc.biz/spencerm/PacketStation`):
```bash
deploy/vm/make-release.sh --image gt.pescoinc.biz/spencerm/packetstation:sha-<sha> --out ~/releases
```
2. Copy `~/releases/packetstation-<build>.tar.gz`, its `.sha256` and the PESCO CA bundle to the VM. The bundle comes from `scripts/Export-PescoCaBundle.ps1 -NoCopy` (or `-Remote` with the VM's path), or from `~/packetstation/test/ca/pesco-ca.pem` on docker-1.
3. On the VM, unpack and run the install. It installs the packages, makes the user, folders, env file, CA, the four internal secrets, PostgreSQL's settings, roles and database, the units, the commands, the backup timer and the firewall:
```bash
cd ~ && sha256sum -c packetstation-<build>.tar.gz.sha256
tar -xzf packetstation-<build>.tar.gz
sudo bash ./packetstation-<build>/deploy/vm/install.sh --stack test --ca-bundle ~/pesco-ca.pem --generate-secrets
```
For production, `--stack prod`. If the VM is to carry on the docker-1 test stack's data, copy that stack's `seal_key` and `session_key` into `/etc/packetstation/secrets/` before this step instead of generating them: filed packets are sealed with the seal key.
4. Put the three outside secrets in place, without echo, then run the install again so it checks them and sets their owner and mode:
```bash
for s in visualkit_password mfileskit_password ldap_bind_password; do
sudo bash -c 'umask 077; read -rsp "$1: " v; echo; printf %s "$v" > "/etc/packetstation/secrets/$1"' _ "$s"
done
sudo bash ./packetstation-<build>/deploy/vm/install.sh --skip-packages
```
5. Read `/etc/packetstation/packetstation.env` through (`sudoedit`). Check `STACK`, `PUBLIC_URL`, `VISUAL_SERVER` and `VISUAL_DATABASE` (test: `'vsql.pesco.com\DEV'` and `PESCOINCDEV`; production: `vsql.pesco.com` and `PESCOINC`), `NOTIFY_DRY_RUN` (test `true`), and `TRUSTED_PROXY_HOST=192.168.101.12`. After changing `TRUSTED_PROXY_HOST`, run `install.sh --files-only` to rewrite the firewall table.
6. Install the release. It builds the venv, runs the migrations and the doctor, and starts nothing on a first install:
```bash
sudo ps-upgrade ~/packetstation-<build>.tar.gz
```
7. Run the doctor until it is clean (on production with `--skip visual` until the approval):
```bash
sudo ps-doctor
```
8. The archive share, when the QNAP export exists ("Archive NFS mount").
9. Start. On production this is the approval of the scheduled VISUAL polling:
```bash
sudo systemctl enable --now packetstation-app packetstation-worker
systemctl status packetstation-app packetstation-worker --no-pager
curl -fsS http://127.0.0.1:8000/healthz
```
10. Point the NPM proxy host at the VM ("NPM proxy host for the VMs") and run its checks.
11. Backups: `sudo systemctl start packetstation-backup.service`, then `sudo ls -la /var/backups/packetstation`. Then rehearse a restore ("Restore on a VM").
On Ubuntu with `ufw` turned on, ufw also has to let docker-1 in: `sudo ufw allow from 192.168.101.12 to any port 8000 proto tcp`. The Packet Station table only narrows port 8000 further; it never opens anything ufw blocks.
### Upgrades
Build the archive for the new commit (step 1), copy it over (step 2), then:
```bash
sudo ps-upgrade ~/packetstation-<build>.tar.gz
```
In order: check the `.sha256` when it lies next to the archive, unpack into `/opt/packetstation/<build>` and build its venv (an already complete release folder is reused), take a backup with `ps-backup`, stop the app and the worker, point `current` at the new release (`previous` keeps the old), install its units and commands (`install.sh --files-only`), run the migrations, start again whichever of the two ran before, wait for `/healthz`, run the doctor, and keep the newest three releases. A short downtime window, as with a new image.
If the migrations fail, `current` goes back to the old release, the app and the worker stay stopped, and the restore command for the backup taken a minute earlier is printed. Read `journalctl -u packetstation-migrate`.
`sudo ps-upgrade --rollback` points `current` back at `previous` and restarts what ran. It does not undo migrations: roll back only to a release whose code works with the schema in the database, or restore the backup.
Open screens see the build change and offer a reload (the version banner).
### NPM proxy host for the VMs
NPM stays on docker-1 and keeps the certificates. The DNS names keep pointing at docker-1 (192.168.101.12). Only the forward target of each proxy host changes, to its VM over the LAN:
| | Production | Test |
|---|---|---|
| Domain name | `packetstation.pescoinc.biz` | `packetstation-test.pescoinc.biz` |
| Scheme | `http` | `http` |
| Forward hostname / IP | the production VM's address | the test VM's address |
| Forward port | `8000` | `8000` |
| `TRUSTED_PROXY_HOST` on the VM | `192.168.101.12` | `192.168.101.12` |
Everything else is as before (`deploy/npm-proxy-host.md`): Cache assets off, Block common exploits on, Websockets off, no access list; the PESCO certificate, Force SSL on, HTTP/2 on, NPM HSTS off (the app sends `Strict-Transport-Security` itself; two headers would conflict); and the same Advanced block, unchanged:
```nginx
client_max_body_size 30m;
location /api/events {
proxy_pass $forward_scheme://$server:$port;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
```
With a fixed VM address the forward target can be the address itself; the "never an IP" rule was for container addresses, which change. `$server` keeps the events location on the same target as the rest of the host.
Trust: NPM's container reaches the VM through docker-1's own address, so the VM sees every proxied request from 192.168.101.12. `TRUSTED_PROXY_HOST=192.168.101.12` makes the app trust `X-Forwarded-For` and `X-Forwarded-Proto` from that address only (`proxy.py` resolves an address literal to itself; no server change was needed), and the firewall table lets only that address reach port 8000. Other hosts on the LAN cannot reach the app directly, and so cannot fake a client address for the sign-in rate limit. Traffic from NPM to the VM is plain HTTP on the LAN; TLS to the VM is still open on issue 37.
Check after switching: the four checks in `deploy/npm-proxy-host.md`, plus from another LAN PC `curl -m 5 http://<vm>:8000/healthz` must time out (the firewall), and the Admin page's access log shows the real client addresses, not 192.168.101.12.
### Archive NFS mount
The archive share (Gitea issue 7) lives on the QNAP. Spencer creates the NFS export with read and write for the VMs' addresses and makes the share's folder owned by uid 10001 and gid 10001 (or maps the VM's uid 10001 to an owner with write rights). Root squash on is fine: only `packetstation` writes there.
1. `nfs-common` is installed by `install.sh`. Add one line to `/etc/fstab` (the QNAP's name and export path from IT):
```
<qnap>:/<export> /mnt/packetstation-archive nfs4 rw,soft,timeo=150,retrans=3,noatime,nosuid,nodev,noexec,_netdev,nofail 0 0
```
`soft`: an unreachable QNAP makes a copy fail after about a minute instead of hanging the worker for good. The archive job writes each file to a temporary name, fsyncs and renames it, so a failed write shows up as a failed copy and is tried again with backoff. Filing never waits on it. `nofail` lets the VM boot without the QNAP.
2. Mount it and make the archive folder as the service user:
```bash
sudo mount /mnt/packetstation-archive
sudo setpriv --reuid=packetstation --regid=packetstation --init-groups mkdir /mnt/packetstation-archive/packets
```
3. Set `ARCHIVE_ROOT=/mnt/packetstation-archive` in the env file (uncomment the line) and restart: `sudo systemctl restart packetstation-app packetstation-worker`.
4. On the Admin page, File locations, enter `packets`, press Check, then Save. `sudo ps-doctor` reports `archive` PASS from then on.
While the share is not mounted, the mount point is root's and read-only to `packetstation`, and `packets` does not exist under it, so nothing is written to the VM's own disk and the check fails with a plain message. The units list the mount point as an optional writable path, so the app and the worker start without it.
### Retiring the docker-1 test stack
The docker-1 test stack keeps running until the test VM passes its doctor, the NPM switch, a backup and a restore rehearsal. To carry its data over instead of starting empty: take a dump with the backup already installed on docker-1, tar the assets volume as in the 2026-10-21 rehearsal, copy both and the `seal_key` and `session_key` to the VM, and restore with `ps-restore` ("Restore on a VM"). Then on docker-1: stop the stack (`docker compose -p packetstation-test ... down`, keeping its volumes until the VM has run for a while), and remove its line from docker-1's `/etc/packetstation/backup.conf`. docker-1 keeps the old `ps-backup.sh` and `ps-restore.sh` it has installed; the versions in the repo are the VM ones and do not run there.
## Backups on a VM
`scripts/backup/ps-backup.sh` (installed as `/usr/local/sbin/ps-backup`) runs from `packetstation-backup.timer`, as root, outside the app, so a broken app still backs up. [scripts/backup/README.md](../scripts/backup/README.md) has the file list.
- Timer `OnCalendar=*-*-* 02:00:00 America/Denver`, `Persistent=true`: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot. `install.sh` enables it.
- `/etc/packetstation/backup.conf`: one line, `name database folder`, for example `prod packetstation /var/backups/packetstation`. `install.sh` writes it.
- `pg_dump -d packetstation -Fc` and `pg_dumpall --globals-only --no-role-passwords` run as the `postgres` OS user (`setpriv`, no PAM) on the local socket. No password is read or passed. Root writes the output under a temporary name, `pg_restore --list` must list at least one entry, then the files are renamed to `ps-<UTC yyyymmddThhmmssZ>.dump` and `.globals.sql`, mode 0600.
- The newest 7 are kept (`KEEP=7` in the unit, `--keep N`), with their globals files. `last-success` or `last-failure` (`<ISO UTC time> <dump name>`, plus the reason on failure), mode 0644, owner and group taken from the folder (root:packetstation 0750). The worker's `backup-check` job (07:15) reads them through `BACKUP_DIR=/var/backups/packetstation`. A failed run leaves no partial file. `flock` keeps two runs apart. Exit 1 on a failure, 2 on a usage or config error.
- The unit runs with `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=1h`, `ProtectSystem=strict` (writable only `/var/backups/packetstation`), `ProtectHome`, `PrivateTmp`, `NoNewPrivileges`, and only the capabilities to switch user and set owners.
- `ps-upgrade` takes a backup before every upgrade.
What the dump does not hold, and what covers it:
| What | Where | Covered by |
|---|---|---|
| Stored files | `/var/lib/packetstation/assets` | PBS (the VM). Restore with a PBS file-level restore and `ps-restore --assets`. |
| Settings, secrets, CA | `/etc/packetstation` | PBS. The secrets are also needed to rebuild a VM: without `seal_key` the seals of filed packets cannot be checked. Keep a copy of `/etc/packetstation/secrets` where IT keeps the other service secrets. |
| Archive copies | the QNAP share | The QNAP's own backup. |
Check by hand:
```bash
cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer
sudo systemctl start packetstation-backup.service
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups
sudo ls -la /var/backups/packetstation
systemctl list-timers packetstation-backup.timer # next run 08:00 or 09:00 UTC, 02:00 in Denver
```
`list-timers` shows the next run at 08:00 or 09:00 UTC (02:00 in Denver). The one run by hand should leave a new `ps-*.dump` and a fresh `last-success`. After changing a script in the repo, run its `install` line again: the installed copies are root's, so a sync of the checkout does not change what root runs.
## Restore on a VM
When the production stack exists (phase 10):
`ps-restore` (`scripts/backup/ps-restore.sh`) never stops, starts or reconfigures a service. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.
```bash
sudo install -d -o root -g root -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf # uncomment the prod line
sudo systemctl start packetstation-backup.service
```
B=/var/backups/packetstation
D=$(sudo ls "$B" | grep '\.dump$' | tail -1)
A run by hand needs no sudo for the test stack (spencerm is in the docker group):
# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is
# connected). --replace is refused for packetstation itself.
sudo ps-restore --dump "$B/$D" --globals "$B/${D%.dump}.globals.sql" --db ps_restored
```bash
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
```
# 2. Stored files, from a folder or a tar (a PBS file-level restore). Existing files are never overwritten.
# Files end up owned by packetstation. Use a scratch folder for a rehearsal.
sudo ps-restore --assets /path/to/assets.tar --assets-dir /var/lib/packetstation/assets
The dumps hold the database only. The assets volume (the files) is not in them. It is covered by PBS, since the volume is part of the VM.
## Restore (phase 9)
`ps-restore.sh` never stops, restarts or reconfigures a stack. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.
```bash
B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)
# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
# 2. Assets, from a folder or a tar (for example a PBS file-level restore of the assets volume folder).
# Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
# 3. Every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
# 3. Every asset row's sha256 against the files.
sudo ps-restore --verify-assets --db ps_restored --assets-dir /var/lib/packetstation/assets
# Row counts of any database, to compare with the live one.
ps-restore.sh --project packetstation-test --counts-only --db packetstation
sudo ps-restore --counts-only --db packetstation
```
- The database is created as `01-roles.sh` creates it (owner `ps_migrate`, UTF8, `template0`, CONNECT and TEMPORARY for `ps_app`), then `pg_restore --exit-on-error --single-transaction` as postgres. A failed restore leaves an empty database, never a half one.
- The globals file is applied without stopping on "already exists". Any other error fails. The script refuses a globals file that holds a password hash. On a new cluster the roles come back without passwords: set each from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates them.
- The assets copy runs in a throwaway container from the image the postgres container already runs, with `--network none`, so nothing is downloaded.
- To put a restored database into service: stop that stack's app and worker, rename as postgres (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack, check the app, then drop the old database. This step is deliberate and manual.
- The database is created as `install.sh` creates the live one (owner `ps_migrate`, UTF8, `en_US.UTF-8`, `template0`, CONNECT and TEMPORARY for `ps_app`), then `pg_restore --exit-on-error --single-transaction` as postgres. A failed restore leaves an empty database, never a half one.
- The globals file is applied without stopping on "already exists". Any other error fails. A globals file with a password hash is refused. Roles created from it have no password: run `install.sh --skip-packages`, which sets both role passwords from the secrets.
- To put a restored database into service: `sudo systemctl stop packetstation-app packetstation-worker`; as postgres (`sudo setpriv --reuid=postgres --regid=postgres --init-groups psql -X -d postgres`) run `ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`; then `sudo systemctl restart packetstation-migrate` and `sudo systemctl start packetstation-app packetstation-worker` (migrate brings an older dump up to the release), check the app, and drop the old database. This step is deliberate and manual.
- A rebuilt VM after a disaster: copy the saved `/etc/packetstation/secrets` back first, run `install.sh`, `ps-upgrade` with the same release, then the restore and the rename above (the install made an empty `packetstation`), and the assets.
- The next rehearsal is IT's, quarterly (requirement O2). The first one on the test VM is part of its acceptance.
## Backup and restore on docker-1 (phase 9, until the test VM)
Until the test VM takes over, the docker-1 test stack is backed up by the phase 9 scripts installed on docker-1 as `/usr/local/sbin/ps-backup.sh` and `ps-restore.sh`. They reach the stack's postgres container with `docker exec` and read `/etc/packetstation/backup.conf` in the four-column form `name container database folder` (`test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups`). Those versions are in git history (up to commit 3fdf063). The scripts in the repo are now the VM ones and refuse that config, so do not reinstall from the repo on docker-1. Markers, rotation and timer are the same as on a VM. The rehearsal below was made with them.
### Restore rehearsal 2026-10-21