PacketStation/scripts/backup/README.md

Backup and restore

Host scripts for docker-1. They run outside the app, so a broken app still backs up.

File	What it is
ps-backup.sh	Nightly pg_dump -Fc of each stack's database plus pg_dumpall --globals-only --no-role-passwords, checked with pg_restore --list, 7 kept, last-success and last-failure markers.
ps-restore.sh	Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts.
packetstation-backup.service	systemd oneshot that runs ps-backup.sh as root.
packetstation-backup.timer	02:00 America/Denver every day, Persistent=true.
backup.conf.example	The stack list for /etc/packetstation/backup.conf.

Test: tests/server/test_backup_scripts.py runs ps-backup.sh with a fake docker on PATH.

What a backup is

In each stack's folder (/var/backups/packetstation/prod, /home/spencerm/packetstation/test/backups):

• ps-<UTC stamp>.dump: the app database, custom format. Mode 0600.
• ps-<UTC stamp>.globals.sql: the roles, without passwords. Mode 0600.
• last-success, last-failure: one line, <ISO UTC time> <dump name> (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.

Each dump is written under a temporary name (.ps-….partial), checked, then renamed. A failed run leaves no partial file and writes last-failure. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.

The script connects inside the postgres container as postgres over the local socket, which the image trusts. It reads no password.

Why --no-role-passwords: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume deploy/postgres/initdb/01-roles.sh creates the roles with them. On an existing cluster the roles are already there.

Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.

Install (Spencer, with sudo, on docker-1)

Copy the files from the synced checkout in ~/packetstation/repo. The installed copies are root's, so a later sync of the checkout does not change what root runs.

cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh  /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer       # next run 08:00 or 09:00 UTC, 02:00 in Denver
sudo systemctl start packetstation-backup.service      # one run now
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups

The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:

sudo install -d -o root -g root  -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf                 # uncomment the prod line
sudo systemctl start packetstation-backup.service

Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its stack.env sets BACKUP_DIR=/var/backups/packetstation/prod.

After changing a script in the repo, install it again with the same install line.

Run by hand

As spencerm (in the docker group), against the test stack:

scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups

--stack NAME CONTAINER DATABASE FOLDER can repeat. CONTAINER is a container name or project:<compose project>. --keep N (or KEEP=N) changes the 7.

Restore

The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.

B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)

# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored

# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
#    Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets

# 3. Check every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets

# Row counts of any database, for comparing.
ps-restore.sh --project packetstation-test --counts-only --db packetstation

To put the restored database into service, stop the app and worker of that stack, rename the databases as postgres (ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.

The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with \password in psql, or start the stack on an empty volume so initdb creates the roles.