PacketStation/scripts/backup/README.md

# Backup and restore

Host scripts for docker-1. They run outside the app, so a broken app still backs up.

| File | What it is |
|---|---|
| `ps-backup.sh` | Nightly `pg_dump -Fc` of each stack's database plus `pg_dumpall --globals-only --no-role-passwords`, checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
| `ps-restore.sh` | Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts. |
| `packetstation-backup.service` | systemd oneshot that runs `ps-backup.sh` as root. |
| `packetstation-backup.timer` | 02:00 America/Denver every day, `Persistent=true`. |
| `backup.conf.example` | The stack list for `/etc/packetstation/backup.conf`. |

Test: `tests/server/test_backup_scripts.py` runs `ps-backup.sh` with a fake `docker` on PATH.

## What a backup is

In each stack's folder (`/var/backups/packetstation/prod`, `/home/spencerm/packetstation/test/backups`):

- `ps-<UTC stamp>.dump`: the app database, custom format. Mode 0600.
- `ps-<UTC stamp>.globals.sql`: the roles, without passwords. Mode 0600.
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.

Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.

The script connects inside the postgres container as `postgres` over the local socket, which the image trusts. It reads no password.

Why `--no-role-passwords`: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume `deploy/postgres/initdb/01-roles.sh` creates the roles with them. On an existing cluster the roles are already there.

Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.

## Install (Spencer, with sudo, on docker-1)

Copy the files from the synced checkout in `~/packetstation/repo`. The installed copies are root's, so a later sync of the checkout does not change what root runs.

```bash
cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh  /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer   /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer       # next run 08:00 or 09:00 UTC, 02:00 in Denver
sudo systemctl start packetstation-backup.service      # one run now
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups
```

The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:

```bash
sudo install -d -o root -g root  -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf                 # uncomment the prod line
sudo systemctl start packetstation-backup.service
```

Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.

After changing a script in the repo, install it again with the same `install` line.

## Run by hand

As spencerm (in the docker group), against the test stack:

```bash
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
```

`--stack NAME CONTAINER DATABASE FOLDER` can repeat. CONTAINER is a container name or `project:<compose project>`. `--keep N` (or `KEEP=N`) changes the 7.

## Restore

The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.

```bash
B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)

# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored

# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
#    Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets

# 3. Check every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets

# Row counts of any database, for comparing.
ps-restore.sh --project packetstation-test --counts-only --db packetstation
```

To put the restored database into service, stop the app and worker of that stack, rename the databases as `postgres` (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.

The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates the roles.