spencerm/PacketStation
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Native VM install kit: systemd units, launcher, install and upgrade scripts; backups, restore and doctor without docker exec; DEPLOY.md VM, NPM and NFS sections

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
spencerm
2026-10-30 18:08:48 -06:00
co-authored by Claude Opus 5.5
parent f025479358
commit 45a1d1a42a
22 changed files with 2357 additions and 413 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/backup/README.md
+15 -85
View File
@@ -1,99 +1,29 @@
# Backup and restore
Host scripts for docker-1. They run outside the app, so a broken app still backs up.
Scripts for the native VMs (production and test). They run outside the app, so a broken app still backs up. `deploy/vm/install.sh` installs them; the steps and the restore procedure are in [docs/DEPLOY.md](../../docs/DEPLOY.md), "Backups on a VM" and "Restore on a VM".
| File | What it is |
|---|---|
| `ps-backup.sh` | Nightly `pg_dump -Fc` of each stack's database plus `pg_dumpall --globals-only --no-role-passwords`, checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
| `ps-restore.sh` | Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts. |
| `packetstation-backup.service` | systemd oneshot that runs `ps-backup.sh` as root. |
| `packetstation-backup.timer` | 02:00 America/Denver every day, `Persistent=true`. |
| `backup.conf.example` | The stack list for `/etc/packetstation/backup.conf`. |
| File | Installed as | What it is |
|---|---|---|
| `ps-backup.sh` | `/usr/local/sbin/ps-backup` | Nightly `pg_dump -Fc` plus `pg_dumpall --globals-only --no-role-passwords`, as the postgres OS user on the local socket. Checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
| `ps-restore.sh` | `/usr/local/sbin/ps-restore` | Restores a dump into a fresh database, applies the globals, copies stored files into an assets folder without overwriting, checks every file's sha256 against the database, prints row counts. |
| `packetstation-backup.service` | `/etc/systemd/system/` | Oneshot as root, sandboxed, writes only `/var/backups/packetstation`. |
| `packetstation-backup.timer` | `/etc/systemd/system/` | 02:00 America/Denver every day, `Persistent=true`. |
| `backup.conf.example` | `/etc/packetstation/backup.conf` | `name database folder`, one line per stack. |
Test: `tests/server/test_backup_scripts.py` runs `ps-backup.sh` with a fake `docker` on PATH.
Tests: `tests/server/test_backup_scripts.py` runs both scripts with fake PostgreSQL tools (`PG_BINDIR`) and a fake user switch (`PG_AS`).
## What a backup is
In each stack's folder (`/var/backups/packetstation/prod`, `/home/spencerm/packetstation/test/backups`):
In `/var/backups/packetstation` (root:packetstation 0750):
- `ps-<UTC stamp>.dump`: the app database, custom format. Mode 0600.
- `ps-<UTC stamp>.globals.sql`: the roles, without passwords. Mode 0600.
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job reads them through the folder's group.
Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.
Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. Exit 1 if any stack failed, 2 for a usage or config error.
The script connects inside the postgres container as `postgres` over the local socket, which the image trusts. It reads no password.
Not in the dump: the stored files (`/var/lib/packetstation/assets`), `/etc/packetstation` (env file, secrets, CA) and the archive share. The PBS backup of the VM covers the first two; the archive is on the QNAP.
Why `--no-role-passwords`: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume `deploy/postgres/initdb/01-roles.sh` creates the roles with them. On an existing cluster the roles are already there.
## The docker-1 test stack
Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.
## Install (Spencer, with sudo, on docker-1)
Copy the files from the synced checkout in `~/packetstation/repo`. The installed copies are root's, so a later sync of the checkout does not change what root runs.
```bash
cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer # next run 08:00 or 09:00 UTC, 02:00 in Denver
sudo systemctl start packetstation-backup.service # one run now
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups
```
The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:
```bash
sudo install -d -o root -g root -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf # uncomment the prod line
sudo systemctl start packetstation-backup.service
```
Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
After changing a script in the repo, install it again with the same `install` line.
## Run by hand
As spencerm (in the docker group), against the test stack:
```bash
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
```
`--stack NAME CONTAINER DATABASE FOLDER` can repeat. CONTAINER is a container name or `project:<compose project>`. `--keep N` (or `KEEP=N`) changes the 7.
## Restore
The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.
```bash
B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)
# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
# Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
# 3. Check every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
# Row counts of any database, for comparing.
ps-restore.sh --project packetstation-test --counts-only --db packetstation
```
To put the restored database into service, stop the app and worker of that stack, rename the databases as `postgres` (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.
The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates the roles.
These scripts no longer talk to Docker. The docker-1 test stack keeps the copies installed in `/usr/local/sbin/ps-backup.sh` and `ps-restore.sh` (the phase 9 versions, in git history up to commit 3fdf063) until it is stopped. Do not install the new scripts on docker-1: its `backup.conf` has the old four-column format, which the new script refuses.