Phase 9: Admin completion, reconcile, notifications, integrity, backups
Admin: Departments (name and stations; add shows the three groups IT must create), Profiles (welder stamp, station limits, active) with a CSV import that previews and applies only when clean, Checklists (weld and plain), Security (session, lockout and rate-limit settings with bounds, alerts for the last 24 hours, the integrity result), Access log with filters and a guarded CSV, Who is in, Stamp check, Notifications, and the employeeID flags. Every Admin change is audited. Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh (15 minutes), integrity (seals, asset hashes, missing and orphan files, the audit chain), backup-check, notify-send and notify-digest. Notifications route per event type to AD groups, users, mailboxes or the department Supervisor group, immediately or in a daily digest, over STARTTLS verified against the PESCO CA, with dry run on the test stack. Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21. CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19. Migration 0010. Closes the app part of I6, M3, O1, O2 and S3. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,99 @@
|
||||
# Backup and restore
|
||||
|
||||
Host scripts for docker-1. They run outside the app, so a broken app still backs up.
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `ps-backup.sh` | Nightly `pg_dump -Fc` of each stack's database plus `pg_dumpall --globals-only --no-role-passwords`, checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
|
||||
| `ps-restore.sh` | Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts. |
|
||||
| `packetstation-backup.service` | systemd oneshot that runs `ps-backup.sh` as root. |
|
||||
| `packetstation-backup.timer` | 02:00 America/Denver every day, `Persistent=true`. |
|
||||
| `backup.conf.example` | The stack list for `/etc/packetstation/backup.conf`. |
|
||||
|
||||
Test: `tests/server/test_backup_scripts.py` runs `ps-backup.sh` with a fake `docker` on PATH.
|
||||
|
||||
## What a backup is
|
||||
|
||||
In each stack's folder (`/var/backups/packetstation/prod`, `/home/spencerm/packetstation/test/backups`):
|
||||
|
||||
- `ps-<UTC stamp>.dump`: the app database, custom format. Mode 0600.
|
||||
- `ps-<UTC stamp>.globals.sql`: the roles, without passwords. Mode 0600.
|
||||
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.
|
||||
|
||||
Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.
|
||||
|
||||
The script connects inside the postgres container as `postgres` over the local socket, which the image trusts. It reads no password.
|
||||
|
||||
Why `--no-role-passwords`: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume `deploy/postgres/initdb/01-roles.sh` creates the roles with them. On an existing cluster the roles are already there.
|
||||
|
||||
Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.
|
||||
|
||||
## Install (Spencer, with sudo, on docker-1)
|
||||
|
||||
Copy the files from the synced checkout in `~/packetstation/repo`. The installed copies are root's, so a later sync of the checkout does not change what root runs.
|
||||
|
||||
```bash
|
||||
cd ~/packetstation/repo
|
||||
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh /usr/local/sbin/ps-backup.sh
|
||||
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
|
||||
sudo install -d -o root -g root -m 0755 /etc/packetstation
|
||||
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
|
||||
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
|
||||
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
|
||||
sudo systemctl enable --now packetstation-backup.timer
|
||||
systemctl list-timers packetstation-backup.timer # next run 08:00 or 09:00 UTC, 02:00 in Denver
|
||||
sudo systemctl start packetstation-backup.service # one run now
|
||||
journalctl -u packetstation-backup.service -n 20 --no-pager
|
||||
ls -la ~/packetstation/test/backups
|
||||
```
|
||||
|
||||
The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:
|
||||
|
||||
```bash
|
||||
sudo install -d -o root -g root -m 0755 /var/backups/packetstation
|
||||
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
|
||||
sudoedit /etc/packetstation/backup.conf # uncomment the prod line
|
||||
sudo systemctl start packetstation-backup.service
|
||||
```
|
||||
|
||||
Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
|
||||
|
||||
After changing a script in the repo, install it again with the same `install` line.
|
||||
|
||||
## Run by hand
|
||||
|
||||
As spencerm (in the docker group), against the test stack:
|
||||
|
||||
```bash
|
||||
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
|
||||
```
|
||||
|
||||
`--stack NAME CONTAINER DATABASE FOLDER` can repeat. CONTAINER is a container name or `project:<compose project>`. `--keep N` (or `KEEP=N`) changes the 7.
|
||||
|
||||
## Restore
|
||||
|
||||
The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.
|
||||
|
||||
```bash
|
||||
B=/home/spencerm/packetstation/test/backups
|
||||
D=$(ls $B/ps-*.dump | tail -1)
|
||||
|
||||
# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
|
||||
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
|
||||
|
||||
# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
|
||||
# Existing files are never overwritten. Files end up owned by 10001:10001.
|
||||
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
|
||||
|
||||
# 3. Check every asset row's sha256 against the files in the volume.
|
||||
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
|
||||
|
||||
# Row counts of any database, for comparing.
|
||||
ps-restore.sh --project packetstation-test --counts-only --db packetstation
|
||||
```
|
||||
|
||||
To put the restored database into service, stop the app and worker of that stack, rename the databases as `postgres` (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.
|
||||
|
||||
The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates the roles.
|
||||
Reference in New Issue
Block a user