Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

scripts/README.md

@@ -1,7 +1,10 @@
 # scripts
 
-Backup and restore scripts arrive in phase 9: a per-stack `pg_dump` backup with a
-systemd timer on docker-1, and a restore script with a rehearsal recorded in docs/DEPLOY.md.
+## backup/
+
+The nightly per-stack `pg_dump` backup with its systemd timer, and the restore script. See
+[backup/README.md](backup/README.md) for what they write and Spencer's install steps. The restore
+rehearsal is recorded in docs/DEPLOY.md.
 
 ## Export-PescoCaBundle.ps1
 

scripts/backup/README.md

@@ -0,0 +1,99 @@
+# Backup and restore
+
+Host scripts for docker-1. They run outside the app, so a broken app still backs up.
+
+| File | What it is |
+|---|---|
+| `ps-backup.sh` | Nightly `pg_dump -Fc` of each stack's database plus `pg_dumpall --globals-only --no-role-passwords`, checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
+| `ps-restore.sh` | Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts. |
+| `packetstation-backup.service` | systemd oneshot that runs `ps-backup.sh` as root. |
+| `packetstation-backup.timer` | 02:00 America/Denver every day, `Persistent=true`. |
+| `backup.conf.example` | The stack list for `/etc/packetstation/backup.conf`. |
+
+Test: `tests/server/test_backup_scripts.py` runs `ps-backup.sh` with a fake `docker` on PATH.
+
+## What a backup is
+
+In each stack's folder (`/var/backups/packetstation/prod`, `/home/spencerm/packetstation/test/backups`):
+
+- `ps-<UTC stamp>.dump`: the app database, custom format. Mode 0600.
+- `ps-<UTC stamp>.globals.sql`: the roles, without passwords. Mode 0600.
+- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.
+
+Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.
+
+The script connects inside the postgres container as `postgres` over the local socket, which the image trusts. It reads no password.
+
+Why `--no-role-passwords`: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume `deploy/postgres/initdb/01-roles.sh` creates the roles with them. On an existing cluster the roles are already there.
+
+Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.
+
+## Install (Spencer, with sudo, on docker-1)
+
+Copy the files from the synced checkout in `~/packetstation/repo`. The installed copies are root's, so a later sync of the checkout does not change what root runs.
+
+```bash
+cd ~/packetstation/repo
+sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh  /usr/local/sbin/ps-backup.sh
+sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
+sudo install -d -o root -g root -m 0755 /etc/packetstation
+sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
+sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
+sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer   /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
+sudo systemctl enable --now packetstation-backup.timer
+systemctl list-timers packetstation-backup.timer       # next run 08:00 or 09:00 UTC, 02:00 in Denver
+sudo systemctl start packetstation-backup.service      # one run now
+journalctl -u packetstation-backup.service -n 20 --no-pager
+ls -la ~/packetstation/test/backups
+```
+
+The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:
+
+```bash
+sudo install -d -o root -g root  -m 0755 /var/backups/packetstation
+sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
+sudoedit /etc/packetstation/backup.conf                 # uncomment the prod line
+sudo systemctl start packetstation-backup.service
+```
+
+Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
+
+After changing a script in the repo, install it again with the same `install` line.
+
+## Run by hand
+
+As spencerm (in the docker group), against the test stack:
+
+```bash
+scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
+```
+
+`--stack NAME CONTAINER DATABASE FOLDER` can repeat. CONTAINER is a container name or `project:<compose project>`. `--keep N` (or `KEEP=N`) changes the 7.
+
+## Restore
+
+The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.
+
+```bash
+B=/home/spencerm/packetstation/test/backups
+D=$(ls $B/ps-*.dump | tail -1)
+
+# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
+ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
+
+# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
+#    Existing files are never overwritten. Files end up owned by 10001:10001.
+ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
+
+# 3. Check every asset row's sha256 against the files in the volume.
+ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
+
+# Row counts of any database, for comparing.
+ps-restore.sh --project packetstation-test --counts-only --db packetstation
+```
+
+To put the restored database into service, stop the app and worker of that stack, rename the databases as `postgres` (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.
+
+The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates the roles.

scripts/backup/backup.conf.example

@@ -0,0 +1,5 @@
+# /etc/packetstation/backup.conf: the stacks ps-backup.sh backs up, one per line.
+# name   container (or project:<compose project>)   database       folder
+test     project:packetstation-test                  packetstation  /home/spencerm/packetstation/test/backups
+# Uncomment once the production stack runs (phase 10). Until then it would fail every night.
+#prod    project:packetstation                       packetstation  /var/backups/packetstation/prod

scripts/backup/packetstation-backup.service

@@ -0,0 +1,25 @@
+# Packet Station nightly database backup (scripts/backup/ps-backup.sh).
+# Installed by root on docker-1: see scripts/backup/README.md. Started by packetstation-backup.timer.
+# Runs as root because it talks to the Docker daemon and writes /var/backups/packetstation.
+[Unit]
+Description=Packet Station database backup (pg_dump of each stack)
+Documentation=file:///home/spencerm/packetstation/repo/scripts/backup/README.md
+Wants=docker.service
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/ps-backup.sh --config /etc/packetstation/backup.conf
+Environment=KEEP=7
+# Gentle on the host: the dump runs while the shop is closed, but the stacks stay up.
+Nice=10
+IOSchedulingClass=idle
+TimeoutStartSec=1h
+# Hardening. /usr, /boot and /etc are read-only to the script; it writes only the backup folders.
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=full
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectControlGroups=yes
+RestrictSUIDSGID=yes

scripts/backup/packetstation-backup.timer

@@ -0,0 +1,13 @@
+# Packet Station nightly backup at 02:00 Denver time, an hour before the 03:00 Proxmox Backup
+# Server run, so PBS captures the fresh dumps. The host clock is UTC; the zone makes the intent
+# explicit and follows daylight saving. Persistent=true runs a missed backup at the next boot.
+[Unit]
+Description=Packet Station nightly database backup
+
+[Timer]
+OnCalendar=*-*-* 02:00:00 America/Denver
+Persistent=true
+Unit=packetstation-backup.service
+
+[Install]
+WantedBy=timers.target

scripts/backup/ps-backup.sh

@@ -0,0 +1,299 @@
+#!/usr/bin/env bash
+# Packet Station nightly backup. Runs on the docker host, never inside the app.
+#
+# For each configured stack it writes, into that stack's backup folder:
+#   ps-<UTC stamp>.dump          pg_dump -Fc of the app database (custom format)
+#   ps-<UTC stamp>.globals.sql   pg_dumpall --globals-only --no-role-passwords (roles, no password hashes)
+#   last-success / last-failure  one line: "<ISO UTC time> <dump name>" (last-failure adds the reason)
+#
+# Each dump is written under a temporary name, checked with `pg_restore --list` run inside the
+# postgres container, and only then renamed into place. The newest KEEP dumps (default 7) are kept,
+# with their globals files. Any failure writes last-failure and the script exits non-zero after
+# trying the remaining stacks.
+#
+# Usage:
+#   ps-backup.sh [--config FILE] [STACK ...]
+#   ps-backup.sh --stack NAME CONTAINER DATABASE FOLDER [--stack ...]
+#
+# Config file (default /etc/packetstation/backup.conf, or PS_BACKUP_CONFIG), one stack per line:
+#   # name   container                    database       folder
+#   prod     project:packetstation        packetstation  /var/backups/packetstation/prod
+#   test     project:packetstation-test   packetstation  /home/spencerm/packetstation/test/backups
+# CONTAINER is a container name, or project:<compose project> to find that project's postgres
+# service by its compose labels. With STACK names given, only those stacks run.
+#
+# pg_dump connects inside the container as the postgres superuser over the local socket, which
+# the official image trusts. No password is read, passed or printed.
+#
+# Globals: --no-role-passwords, so no password hash ever lands in a file (or in the PBS copy of
+# it). The roles' passwords come from the stack's secrets: initdb (deploy/postgres/initdb)
+# recreates them on a fresh volume. The globals file is also 0600, like the dumps.
+#
+# Files: dumps and globals are 0600. The markers are 0644 (a time and a file name, nothing else)
+# so the worker, which mounts the folder read-only as uid 10001, can read them. When run as root
+# the new files take the folder's owner and group, so a stack folder owned by spencerm stays
+# readable by spencerm.
+
+set -euo pipefail
+umask 077
+
+KEEP="${KEEP:-7}"
+PG_USER="${PG_USER:-postgres}"
+DOCKER="${DOCKER:-docker}"
+
+log() { printf '%s ps-backup: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2; }
+
+usage() {
+  sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//' >&2
+  exit 2
+}
+
+# ------------------------------------------------------------------ arguments
+
+config="${PS_BACKUP_CONFIG:-/etc/packetstation/backup.conf}"
+config_given=0
+declare -a only=()
+declare -a stacks=()   # each entry: "name|container|database|folder"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --config)
+      [[ $# -ge 2 ]] || usage
+      config="$2"; config_given=1; shift 2 ;;
+    --stack)
+      [[ $# -ge 5 ]] || usage
+      stacks+=("$2|$3|$4|$5"); shift 5 ;;
+    --keep)
+      [[ $# -ge 2 ]] || usage
+      KEEP="$2"; shift 2 ;;
+    -h|--help) usage ;;
+    -*) log "unknown option $1"; usage ;;
+    *) only+=("$1"); shift ;;
+  esac
+done
+
+if ! [[ "$KEEP" =~ ^[1-9][0-9]*$ ]]; then
+  log "KEEP must be a positive whole number"
+  exit 2
+fi
+
+if [[ ${#stacks[@]} -eq 0 ]]; then
+  if [[ ! -r "$config" ]]; then
+    log "no --stack given and the config $config is missing or unreadable"
+    exit 2
+  fi
+  while read -r name container database folder extra; do
+    [[ -z "${name:-}" || "${name:0:1}" == "#" ]] && continue
+    if [[ -z "${folder:-}" || -n "${extra:-}" ]]; then
+      log "config line for '$name' needs exactly: name container database folder"
+      exit 2
+    fi
+    stacks+=("$name|$container|$database|$folder")
+  done < "$config"
+elif [[ $config_given -eq 1 ]]; then
+  log "use --config or --stack, not both"
+  exit 2
+fi
+
+if [[ ${#stacks[@]} -eq 0 ]]; then
+  log "no stacks configured"
+  exit 2
+fi
+
+# ------------------------------------------------------------------ helpers
+
+# Resolve "project:<name>" to the running postgres container of that compose project.
+resolve_container() {
+  local spec="$1"
+  if [[ "$spec" == project:* ]]; then
+    local project="${spec#project:}" ids
+    ids="$("$DOCKER" ps -q \
+      --filter "label=com.docker.compose.project=$project" \
+      --filter "label=com.docker.compose.service=postgres")"
+    if [[ -z "$ids" ]]; then
+      echo "no running postgres container for compose project $project" >&2
+      return 1
+    fi
+    if [[ "$(printf '%s\n' "$ids" | wc -l)" -ne 1 ]]; then
+      echo "more than one postgres container for compose project $project" >&2
+      return 1
+    fi
+    printf '%s\n' "$ids"
+  else
+    printf '%s\n' "$spec"
+  fi
+}
+
+# Write a marker atomically: temp file in the same folder, then rename.
+write_marker() {
+  local folder="$1" marker="$2" text="$3" tmp
+  tmp="$(mktemp "$folder/.${marker}.XXXXXX")"
+  printf '%s\n' "$text" > "$tmp"
+  chmod 0644 "$tmp"
+  take_folder_owner "$folder" "$tmp"
+  mv -f "$tmp" "$folder/$marker"
+}
+
+take_folder_owner() {
+  local folder="$1"; shift
+  if [[ "$(id -u)" == "0" ]]; then
+    chown --reference="$folder" "$@" 2>/dev/null || true
+  fi
+}
+
+# Keep the newest $KEEP dumps (names sort by their UTC stamp), and drop the globals of any removed dump.
+rotate() {
+  local folder="$1" f base
+  local -a dumps=()
+  while IFS= read -r f; do dumps+=("$f"); done < <(
+    find "$folder" -maxdepth 1 -type f -name 'ps-*.dump' -printf '%f\n' | LC_ALL=C sort
+  )
+  local excess=$(( ${#dumps[@]} - KEEP ))
+  local i
+  for (( i = 0; i < excess; i++ )); do
+    base="${dumps[$i]%.dump}"
+    rm -f -- "$folder/${dumps[$i]}" "$folder/$base.globals.sql"
+    log "removed old dump ${dumps[$i]}"
+  done
+  # Globals with no dump next to them (a run that failed after the globals step) go too.
+  while IFS= read -r f; do
+    base="${f%.globals.sql}"
+    [[ -e "$folder/$base.dump" ]] || rm -f -- "$folder/$f"
+  done < <(find "$folder" -maxdepth 1 -type f -name 'ps-*.globals.sql' -printf '%f\n')
+}
+
+# ------------------------------------------------------------------ one stack
+
+# Set by backup_stack for the cleanup trap. Each stack runs in its own subshell (run_stack), so
+# these never leak from one stack to the next.
+cur_tmp_dump=""
+cur_tmp_globals=""
+cur_folder=""
+cur_dump_name=""
+failure_written=0
+
+cleanup_tmp() {
+  if [[ -n "$cur_tmp_dump" ]]; then rm -f -- "$cur_tmp_dump"; fi
+  if [[ -n "$cur_tmp_globals" ]]; then rm -f -- "$cur_tmp_globals"; fi
+  cur_tmp_dump=""
+  cur_tmp_globals=""
+}
+
+# Record the failure and leave the stack's subshell.
+fail_stack() {
+  local reason="$1"
+  cleanup_tmp
+  log "FAILED: $reason"
+  if [[ -n "$cur_folder" && -d "$cur_folder" ]]; then
+    write_marker "$cur_folder" last-failure "$(date -u +%Y-%m-%dT%H:%M:%SZ) ${cur_dump_name:--} $reason" || true
+  fi
+  failure_written=1
+  exit 1
+}
+
+# Any command that fails without an explicit check still ends up as last-failure.
+on_stack_exit() {
+  local rc=$?
+  cleanup_tmp
+  if [[ $rc -ne 0 && $failure_written -eq 0 && -n "$cur_folder" && -d "$cur_folder" ]]; then
+    log "FAILED: unexpected error (exit $rc)"
+    write_marker "$cur_folder" last-failure \
+      "$(date -u +%Y-%m-%dT%H:%M:%SZ) ${cur_dump_name:--} unexpected error (exit $rc)" || true
+  fi
+}
+
+backup_stack() {
+  local name="$1" spec="$2" database="$3" folder="$4"
+  local stamp globals_name container started entries size
+
+  stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+  cur_dump_name="ps-$stamp.dump"
+  globals_name="ps-$stamp.globals.sql"
+  started=$SECONDS
+  log "stack $name: database $database into $folder"
+
+  if [[ ! -d "$folder" ]]; then
+    # 0750: the worker reaches the folder through its group (see scripts/backup/README.md).
+    mkdir -p -- "$folder"
+    chmod 0750 -- "$folder" 2>/dev/null || log "stack $name: could not set mode 0750 on $folder"
+  fi
+  cur_folder="$folder"
+  [[ -w "$folder" ]] || fail_stack "$folder is not writable"
+
+  if command -v flock >/dev/null 2>&1; then
+    exec 9>"$folder/.ps-backup.lock"
+    flock -n 9 || fail_stack "another backup of this stack is running"
+  fi
+
+  container="$(resolve_container "$spec" 2>&1)" || fail_stack "$container"
+
+  cur_tmp_dump="$(mktemp "$folder/.$cur_dump_name.XXXXXX.partial")"
+  cur_tmp_globals="$(mktemp "$folder/.$globals_name.XXXXXX.partial")"
+
+  "$DOCKER" exec "$container" pg_dump -U "$PG_USER" -d "$database" -Fc > "$cur_tmp_dump" \
+    || fail_stack "pg_dump failed"
+  [[ -s "$cur_tmp_dump" ]] || fail_stack "pg_dump wrote nothing"
+
+  "$DOCKER" exec "$container" pg_dumpall -U "$PG_USER" --globals-only --no-role-passwords \
+    > "$cur_tmp_globals" || fail_stack "pg_dumpall --globals-only failed"
+  [[ -s "$cur_tmp_globals" ]] || fail_stack "pg_dumpall wrote nothing"
+
+  # The check reads the dump from stdin with the container's own pg_restore. Entry lines start
+  # with the entry's id; comment lines start with a semicolon.
+  entries="$("$DOCKER" exec -i "$container" pg_restore --list < "$cur_tmp_dump" | grep -c '^[0-9]')" \
+    || fail_stack "pg_restore --list could not read the dump"
+  [[ "$entries" -ge 1 ]] || fail_stack "pg_restore --list found no entries"
+
+  chmod 0600 "$cur_tmp_dump" "$cur_tmp_globals"
+  take_folder_owner "$folder" "$cur_tmp_dump" "$cur_tmp_globals"
+  mv -f -- "$cur_tmp_globals" "$folder/$globals_name"
+  cur_tmp_globals=""
+  mv -f -- "$cur_tmp_dump" "$folder/$cur_dump_name"
+  cur_tmp_dump=""
+
+  size="$(stat -c %s "$folder/$cur_dump_name")"
+  write_marker "$folder" last-success "$(date -u +%Y-%m-%dT%H:%M:%SZ) $cur_dump_name"
+  rotate "$folder"
+  log "stack $name: ok, $cur_dump_name, $size bytes, $entries entries, $(( SECONDS - started )) s"
+}
+
+# One stack in its own subshell with errexit on, so a failure ends that stack only.
+run_stack() (
+  set -euo pipefail
+  trap on_stack_exit EXIT
+  backup_stack "$@"
+)
+
+# ------------------------------------------------------------------ main
+
+failures=0
+ran=0
+for entry in "${stacks[@]}"; do
+  IFS='|' read -r s_name s_container s_database s_folder <<< "$entry"
+  if [[ ${#only[@]} -gt 0 ]]; then
+    wanted=0
+    for o in "${only[@]}"; do
+      if [[ "$o" == "$s_name" ]]; then wanted=1; fi
+    done
+    [[ $wanted -eq 1 ]] || continue
+  fi
+  ran=$(( ran + 1 ))
+  # Not in an if or ||: that would switch errexit off inside the subshell.
+  set +e
+  run_stack "$s_name" "$s_container" "$s_database" "$s_folder"
+  rc=$?
+  set -e
+  if [[ $rc -ne 0 ]]; then
+    failures=$(( failures + 1 ))
+  fi
+done
+
+if [[ $ran -eq 0 ]]; then
+  log "no configured stack matched: ${only[*]}"
+  exit 2
+fi
+if [[ $failures -gt 0 ]]; then
+  log "$failures of $ran stacks failed"
+  exit 1
+fi
+log "all $ran stacks backed up"

scripts/backup/ps-restore.sh

@@ -0,0 +1,276 @@
+#!/usr/bin/env bash
+# Packet Station restore. Runs on the docker host. Restores a ps-backup.sh dump into a fresh
+# database on a stack's postgres container, optionally applies the globals file, copies assets
+# back into an assets volume, and prints the row count of every table at the end.
+#
+# It never stops, restarts or reconfigures a stack. Pointing the app at the restored database
+# (or renaming it into place) is a separate, deliberate step: see scripts/backup/README.md.
+#
+# Usage:
+#   ps-restore.sh --container NAME | --project PROJECT
+#                 [--dump FILE --db NAME [--replace]]
+#                 [--globals FILE]
+#                 [--assets DIR_OR_TAR --assets-volume VOLUME]
+#                 [--verify-assets --db NAME --assets-volume VOLUME]
+#                 [--counts-only --db NAME]
+#
+#   --container / --project  the postgres container, by name or by compose project.
+#   --dump FILE              a ps-*.dump file (pg_dump -Fc).
+#   --db NAME                the database to restore into. It must not exist, unless --replace,
+#                            which drops it first. DROP DATABASE refuses while anyone is connected.
+#   --owner ROLE             owner of the new database (default ps_migrate, as initdb makes it).
+#   --app-role ROLE          role given CONNECT and TEMPORARY on it (default ps_app).
+#   --globals FILE           a ps-*.globals.sql file. Roles that already exist are reported and
+#                            left alone. The file holds no passwords: set them from the secrets.
+#   --assets SRC             a folder, or a .tar / .tar.gz / .tgz, holding the files that belong
+#                            under ASSETS_DIR. Copied into --assets-volume through a throwaway
+#                            container (no network) that runs the postgres image already on the host.
+#                            Existing files are never overwritten.
+#   --assets-volume VOLUME   the docker volume to copy into (for example packetstation-test_assets).
+#   --assets-owner UID:GID   owner for the copied files (default 10001:10001, the image's ps user).
+#   --verify-assets          compare sha256 of every file in --assets-volume with assets.sha256 in --db.
+#   --counts-only            only print the row counts of --db.
+#
+# Connects inside the container as the postgres superuser over the local socket. No password
+# is read, passed or printed.
+
+set -euo pipefail
+umask 077
+
+PG_USER="${PG_USER:-postgres}"
+DOCKER="${DOCKER:-docker}"
+
+log() { printf '%s ps-restore: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2; }
+die() { log "$*"; exit 1; }
+usage() { sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; }
+
+container=""
+project=""
+dump=""
+db=""
+replace=0
+owner="ps_migrate"
+app_role="ps_app"
+globals=""
+assets=""
+assets_volume=""
+assets_owner="10001:10001"
+verify_assets=0
+counts_only=0
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --container) container="${2:?}"; shift 2 ;;
+    --project) project="${2:?}"; shift 2 ;;
+    --dump) dump="${2:?}"; shift 2 ;;
+    --db) db="${2:?}"; shift 2 ;;
+    --replace) replace=1; shift ;;
+    --owner) owner="${2:?}"; shift 2 ;;
+    --app-role) app_role="${2:?}"; shift 2 ;;
+    --globals) globals="${2:?}"; shift 2 ;;
+    --assets) assets="${2:?}"; shift 2 ;;
+    --assets-volume) assets_volume="${2:?}"; shift 2 ;;
+    --assets-owner) assets_owner="${2:?}"; shift 2 ;;
+    --verify-assets) verify_assets=1; shift ;;
+    --counts-only) counts_only=1; shift ;;
+    -h|--help) usage ;;
+    *) log "unknown argument $1"; usage ;;
+  esac
+done
+
+ident_re='^[a-z_][a-z0-9_]{0,62}$'
+[[ -z "$db" || "$db" =~ $ident_re ]] || die "--db must be a plain lower-case name (letters, digits, _)"
+[[ "$owner" =~ $ident_re ]] || die "--owner must be a plain lower-case role name"
+[[ "$app_role" =~ $ident_re ]] || die "--app-role must be a plain lower-case role name"
+[[ "$assets_owner" =~ ^[0-9]+:[0-9]+$ ]] || die "--assets-owner must be UID:GID"
+
+if [[ -n "$project" ]]; then
+  [[ -z "$container" ]] || die "use --container or --project, not both"
+  container="$("$DOCKER" ps -q \
+    --filter "label=com.docker.compose.project=$project" \
+    --filter "label=com.docker.compose.service=postgres")"
+  [[ -n "$container" ]] || die "no running postgres container for compose project $project"
+  [[ "$(printf '%s\n' "$container" | wc -l)" -eq 1 ]] || die "more than one postgres container for $project"
+fi
+[[ -n "$container" ]] || die "--container or --project is required"
+
+if [[ -n "$dump" || $counts_only -eq 1 || $verify_assets -eq 1 ]]; then
+  [[ -n "$db" ]] || die "--db is required"
+fi
+if [[ -n "$assets" || $verify_assets -eq 1 ]]; then
+  [[ -n "$assets_volume" ]] || die "--assets-volume is required"
+fi
+if [[ -z "$dump" && -z "$globals" && -z "$assets" && $verify_assets -eq 0 && $counts_only -eq 0 ]]; then
+  die "nothing to do: give --dump, --globals, --assets, --verify-assets or --counts-only"
+fi
+
+# psql inside the container, reading SQL from stdin. Variables go through -v and :'name'.
+psql_c() {
+  local dbname="$1"; shift
+  "$DOCKER" exec -i "$container" psql -X -q -v ON_ERROR_STOP=1 -U "$PG_USER" -d "$dbname" "$@"
+}
+
+db_exists() {
+  local n
+  n="$(psql_c postgres -At -v name="$1" <<'SQL'
+SELECT count(*) FROM pg_database WHERE datname = :'name';
+SQL
+)"
+  [[ "$n" == "1" ]]
+}
+
+role_exists() {
+  local n
+  n="$(psql_c postgres -At -v name="$1" <<'SQL'
+SELECT count(*) FROM pg_roles WHERE rolname = :'name';
+SQL
+)"
+  [[ "$n" == "1" ]]
+}
+
+print_counts() {
+  local dbname="$1"
+  log "row counts in $dbname"
+  psql_c "$dbname" -A -F $'\t' -P footer=off <<'SQL'
+SELECT c.relname AS table_name,
+       (xpath('/row/n/text()',
+              query_to_xml(format('SELECT count(*) AS n FROM %I.%I', n.nspname, c.relname), false, true, '')
+       ))[1]::text::bigint AS row_count
+FROM pg_class c
+JOIN pg_namespace n ON n.oid = c.relnamespace
+WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p')
+ORDER BY c.relname;
+SQL
+}
+
+# A throwaway container on no network, running the image the postgres container already uses,
+# so nothing is downloaded.
+helper_image() {
+  "$DOCKER" inspect -f '{{.Config.Image}}' "$container"
+}
+
+# ------------------------------------------------------------------ globals
+
+if [[ -n "$globals" ]]; then
+  [[ -r "$globals" ]] || die "cannot read $globals"
+  if grep -qiE "PASSWORD[[:space:]]+'(SCRAM|md5)" "$globals"; then
+    die "$globals holds password hashes; ps-backup.sh writes globals without them. Refusing."
+  fi
+  log "applying globals from $(basename -- "$globals") (existing roles are left as they are)"
+  # Without ON_ERROR_STOP: CREATE ROLE for a role that exists fails and the rest goes on.
+  out="$("$DOCKER" exec -i "$container" psql -X -q -U "$PG_USER" -d postgres < "$globals" 2>&1 || true)"
+  exists="$(printf '%s\n' "$out" | grep -c 'already exists' || true)"
+  other="$(printf '%s\n' "$out" | grep 'ERROR' | grep -vc 'already exists' || true)"
+  log "globals: $exists statements skipped (already exists), $other other errors"
+  if [[ "$other" -gt 0 ]]; then
+    printf '%s\n' "$out" | grep 'ERROR' | grep -v 'already exists' >&2
+    die "globals had errors"
+  fi
+fi
+
+# ------------------------------------------------------------------ database
+
+if [[ -n "$dump" ]]; then
+  [[ -r "$dump" ]] || die "cannot read $dump"
+  started=$SECONDS
+  if db_exists "$db"; then
+    [[ $replace -eq 1 ]] || die "database $db exists. Give --replace to drop it first."
+    log "dropping database $db (--replace)"
+    psql_c postgres -v name="$db" <<'SQL'
+SELECT format('DROP DATABASE %I', :'name') \gexec
+SQL
+  fi
+  role_exists "$owner" || die "role $owner does not exist (apply --globals or start the stack once)"
+
+  log "creating database $db owned by $owner"
+  # The same shape as deploy/postgres/initdb/01-roles.sh.
+  psql_c postgres -v name="$db" -v owner="$owner" <<'SQL'
+SELECT format('CREATE DATABASE %I OWNER %I ENCODING %L TEMPLATE template0', :'name', :'owner', 'UTF8') \gexec
+SELECT format('REVOKE ALL ON DATABASE %I FROM PUBLIC', :'name') \gexec
+SQL
+  if role_exists "$app_role"; then
+    psql_c postgres -v name="$db" -v app="$app_role" <<'SQL'
+SELECT format('GRANT CONNECT, TEMPORARY ON DATABASE %I TO %I', :'name', :'app') \gexec
+SQL
+  else
+    log "role $app_role does not exist; no CONNECT grant made"
+  fi
+
+  log "restoring $(basename -- "$dump") ($(stat -c %s "$dump") bytes) into $db"
+  # One transaction: a restore that fails leaves an empty database, never a half-restored one.
+  "$DOCKER" exec -i "$container" pg_restore -U "$PG_USER" -d "$db" --exit-on-error --single-transaction \
+    < "$dump" || die "pg_restore failed; database $db is left empty. Drop it with --replace on the next try."
+  log "database restored in $(( SECONDS - started )) s"
+fi
+
+# ------------------------------------------------------------------ assets
+
+if [[ -n "$assets" ]]; then
+  [[ -e "$assets" ]] || die "$assets does not exist"
+  "$DOCKER" volume inspect "$assets_volume" >/dev/null 2>&1 || die "volume $assets_volume does not exist"
+  image="$(helper_image)"
+  started=$SECONDS
+  src="$(cd -- "$(dirname -- "$assets")" && pwd)/$(basename -- "$assets")"
+  if [[ -d "$assets" ]]; then
+    log "copying assets from folder $src into volume $assets_volume"
+    "$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
+      -v "$assets_volume:/dst" -v "$src:/src:ro" "$image" -c \
+      'set -eu; cp -a -n /src/. /dst/; chown -R "$1" /dst' sh "$assets_owner"
+  else
+    case "$assets" in
+      *.tar.gz|*.tgz) tar_flags="-xzf" ;;
+      *.tar) tar_flags="-xf" ;;
+      *) die "--assets must be a folder, .tar, .tar.gz or .tgz" ;;
+    esac
+    log "copying assets from $(basename -- "$src") into volume $assets_volume"
+    "$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
+      -v "$assets_volume:/dst" -v "$src:/src.tar:ro" "$image" -c \
+      'set -eu; tar "$1" /src.tar -C /dst --skip-old-files --no-same-owner; chown -R "$2" /dst' \
+      sh "$tar_flags" "$assets_owner"
+  fi
+  log "assets copied in $(( SECONDS - started )) s"
+fi
+
+# ------------------------------------------------------------------ verify assets
+
+if [[ $verify_assets -eq 1 ]]; then
+  "$DOCKER" volume inspect "$assets_volume" >/dev/null 2>&1 || die "volume $assets_volume does not exist"
+  image="$(helper_image)"
+  work="$(mktemp -d)"
+  trap 'rm -rf -- "$work"' EXIT
+  # sha256sum check format: "<hex>  <path>".
+  psql_c "$db" -At <<'SQL' > "$work/assets.sha256"
+SELECT encode(sha256, 'hex') || '  ' || storage_key FROM assets ORDER BY storage_key;
+SQL
+  rows="$(wc -l < "$work/assets.sha256")"
+  chmod 0755 "$work"
+  chmod 0644 "$work/assets.sha256"
+  log "checking $rows asset rows of $db against volume $assets_volume"
+  set +e
+  "$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
+    -v "$assets_volume:/dst:ro" -v "$work:/work:ro" "$image" -c '
+      cd /dst || exit 3
+      sha256sum -c /work/assets.sha256 2>/dev/null
+      find . -type f ! -path "./tmp/*" | sed "s|^\./||" | sort > /tmp/files
+      cut -c67- /work/assets.sha256 | sort > /tmp/rows
+      comm -23 /tmp/files /tmp/rows | sed "s|^|ORPHAN: |"
+      exit 0' > "$work/result"
+  rc=$?
+  set -e
+  [[ $rc -eq 0 ]] || die "the check container failed (exit $rc)"
+  ok="$(grep -c ': OK$' "$work/result" || true)"
+  bad="$(grep -c ': FAILED$' "$work/result" || true)"
+  missing="$(grep -c ': FAILED open or read$' "$work/result" || true)"
+  orphans="$(grep -c '^ORPHAN: ' "$work/result" || true)"
+  log "assets: $rows rows, $ok match, $bad differ, $missing missing, $orphans files with no row"
+  grep -v ': OK$' "$work/result" >&2 || true
+  if [[ "$ok" -ne "$rows" ]]; then
+    die "asset check failed"
+  fi
+fi
+
+# ------------------------------------------------------------------ counts
+
+if [[ -n "$db" && ( -n "$dump" || $counts_only -eq 1 ) ]]; then
+  print_counts "$db"
+fi