Native VM install kit: systemd units, launcher, install and upgrade scripts; backups, restore and doctor without docker exec; DEPLOY.md VM, NPM and NFS sections
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
+1
-3
@@ -2,9 +2,7 @@
|
||||
|
||||
## backup/
|
||||
|
||||
The nightly per-stack `pg_dump` backup with its systemd timer, and the restore script. See
|
||||
[backup/README.md](backup/README.md) for what they write and Spencer's install steps. The restore
|
||||
rehearsal is recorded in docs/DEPLOY.md.
|
||||
The nightly `pg_dump` backup with its systemd timer, and the restore script, for the native VMs (no Docker). See [backup/README.md](backup/README.md) and docs/DEPLOY.md, "Backups on a VM" and "Restore on a VM". The VM install kit itself (units, launcher, install and upgrade) is in `deploy/vm/`.
|
||||
|
||||
## Export-PescoCaBundle.ps1
|
||||
|
||||
|
||||
+15
-85
@@ -1,99 +1,29 @@
|
||||
# Backup and restore
|
||||
|
||||
Host scripts for docker-1. They run outside the app, so a broken app still backs up.
|
||||
Scripts for the native VMs (production and test). They run outside the app, so a broken app still backs up. `deploy/vm/install.sh` installs them; the steps and the restore procedure are in [docs/DEPLOY.md](../../docs/DEPLOY.md), "Backups on a VM" and "Restore on a VM".
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `ps-backup.sh` | Nightly `pg_dump -Fc` of each stack's database plus `pg_dumpall --globals-only --no-role-passwords`, checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
|
||||
| `ps-restore.sh` | Restores a dump into a fresh database, applies the globals, copies assets into a volume, checks asset hashes, prints row counts. |
|
||||
| `packetstation-backup.service` | systemd oneshot that runs `ps-backup.sh` as root. |
|
||||
| `packetstation-backup.timer` | 02:00 America/Denver every day, `Persistent=true`. |
|
||||
| `backup.conf.example` | The stack list for `/etc/packetstation/backup.conf`. |
|
||||
| File | Installed as | What it is |
|
||||
|---|---|---|
|
||||
| `ps-backup.sh` | `/usr/local/sbin/ps-backup` | Nightly `pg_dump -Fc` plus `pg_dumpall --globals-only --no-role-passwords`, as the postgres OS user on the local socket. Checked with `pg_restore --list`, 7 kept, `last-success` and `last-failure` markers. |
|
||||
| `ps-restore.sh` | `/usr/local/sbin/ps-restore` | Restores a dump into a fresh database, applies the globals, copies stored files into an assets folder without overwriting, checks every file's sha256 against the database, prints row counts. |
|
||||
| `packetstation-backup.service` | `/etc/systemd/system/` | Oneshot as root, sandboxed, writes only `/var/backups/packetstation`. |
|
||||
| `packetstation-backup.timer` | `/etc/systemd/system/` | 02:00 America/Denver every day, `Persistent=true`. |
|
||||
| `backup.conf.example` | `/etc/packetstation/backup.conf` | `name database folder`, one line per stack. |
|
||||
|
||||
Test: `tests/server/test_backup_scripts.py` runs `ps-backup.sh` with a fake `docker` on PATH.
|
||||
Tests: `tests/server/test_backup_scripts.py` runs both scripts with fake PostgreSQL tools (`PG_BINDIR`) and a fake user switch (`PG_AS`).
|
||||
|
||||
## What a backup is
|
||||
|
||||
In each stack's folder (`/var/backups/packetstation/prod`, `/home/spencerm/packetstation/test/backups`):
|
||||
In `/var/backups/packetstation` (root:packetstation 0750):
|
||||
|
||||
- `ps-<UTC stamp>.dump`: the app database, custom format. Mode 0600.
|
||||
- `ps-<UTC stamp>.globals.sql`: the roles, without passwords. Mode 0600.
|
||||
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job compares their modification times.
|
||||
- `last-success`, `last-failure`: one line, `<ISO UTC time> <dump name>` (a failure adds the reason). Mode 0644. The worker's backup-check job reads them through the folder's group.
|
||||
|
||||
Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. The script tries every stack and exits 1 if any failed, 2 for a usage or config error.
|
||||
Each dump is written under a temporary name (`.ps-….partial`), checked, then renamed. A failed run leaves no partial file and writes `last-failure`. Exit 1 if any stack failed, 2 for a usage or config error.
|
||||
|
||||
The script connects inside the postgres container as `postgres` over the local socket, which the image trusts. It reads no password.
|
||||
Not in the dump: the stored files (`/var/lib/packetstation/assets`), `/etc/packetstation` (env file, secrets, CA) and the archive share. The PBS backup of the VM covers the first two; the archive is on the QNAP.
|
||||
|
||||
Why `--no-role-passwords`: a password hash in a backup file is something to crack offline, and the file also lands in the PBS copy of the VM. The role passwords are in the stack's secrets already. On a fresh volume `deploy/postgres/initdb/01-roles.sh` creates the roles with them. On an existing cluster the roles are already there.
|
||||
## The docker-1 test stack
|
||||
|
||||
Ownership: run as root, the new files take the owner and group of the stack's folder, so the test folder stays spencerm's. The worker mounts the folder read-only as uid 10001. It only needs to list the folder and stat the dumps, plus read the markers, so dumps can stay 0600.
|
||||
|
||||
## Install (Spencer, with sudo, on docker-1)
|
||||
|
||||
Copy the files from the synced checkout in `~/packetstation/repo`. The installed copies are root's, so a later sync of the checkout does not change what root runs.
|
||||
|
||||
```bash
|
||||
cd ~/packetstation/repo
|
||||
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh /usr/local/sbin/ps-backup.sh
|
||||
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
|
||||
sudo install -d -o root -g root -m 0755 /etc/packetstation
|
||||
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
|
||||
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
|
||||
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
|
||||
sudo systemctl enable --now packetstation-backup.timer
|
||||
systemctl list-timers packetstation-backup.timer # next run 08:00 or 09:00 UTC, 02:00 in Denver
|
||||
sudo systemctl start packetstation-backup.service # one run now
|
||||
journalctl -u packetstation-backup.service -n 20 --no-pager
|
||||
ls -la ~/packetstation/test/backups
|
||||
```
|
||||
|
||||
The config lists the test stack only. When the production stack exists (phase 10), make its folder and uncomment its line:
|
||||
|
||||
```bash
|
||||
sudo install -d -o root -g root -m 0755 /var/backups/packetstation
|
||||
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
|
||||
sudoedit /etc/packetstation/backup.conf # uncomment the prod line
|
||||
sudo systemctl start packetstation-backup.service
|
||||
```
|
||||
|
||||
Group 10001 lets the production worker (uid 10001, group 10001) list the folder and read the markers. Its `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
|
||||
|
||||
After changing a script in the repo, install it again with the same `install` line.
|
||||
|
||||
## Run by hand
|
||||
|
||||
As spencerm (in the docker group), against the test stack:
|
||||
|
||||
```bash
|
||||
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
|
||||
```
|
||||
|
||||
`--stack NAME CONTAINER DATABASE FOLDER` can repeat. CONTAINER is a container name or `project:<compose project>`. `--keep N` (or `KEEP=N`) changes the 7.
|
||||
|
||||
## Restore
|
||||
|
||||
The script never stops, restarts or reconfigures a stack. Restore into a new database, check it, then decide how to switch.
|
||||
|
||||
```bash
|
||||
B=/home/spencerm/packetstation/test/backups
|
||||
D=$(ls $B/ps-*.dump | tail -1)
|
||||
|
||||
# 1. Into a fresh database (refuses an existing one; --replace drops it first, and DROP refuses while anyone is connected).
|
||||
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
|
||||
|
||||
# 2. Assets: from a folder or a tar (for example a file-level restore out of PBS of the assets volume).
|
||||
# Existing files are never overwritten. Files end up owned by 10001:10001.
|
||||
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
|
||||
|
||||
# 3. Check every asset row's sha256 against the files in the volume.
|
||||
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
|
||||
|
||||
# Row counts of any database, for comparing.
|
||||
ps-restore.sh --project packetstation-test --counts-only --db packetstation
|
||||
```
|
||||
|
||||
To put the restored database into service, stop the app and worker of that stack, rename the databases as `postgres` (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack again, and drop the old one once the app is checked. That step is deliberate and manual.
|
||||
|
||||
The globals file holds no passwords. If a role had to be created from it on a new cluster, set its password from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates the roles.
|
||||
These scripts no longer talk to Docker. The docker-1 test stack keeps the copies installed in `/usr/local/sbin/ps-backup.sh` and `ps-restore.sh` (the phase 9 versions, in git history up to commit 3fdf063) until it is stopped. Do not install the new scripts on docker-1: its `backup.conf` has the old four-column format, which the new script refuses.
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
# /etc/packetstation/backup.conf: the stacks ps-backup.sh backs up, one per line.
|
||||
# name container (or project:<compose project>) database folder
|
||||
test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
|
||||
# Uncomment once the production stack runs (phase 10). Until then it would fail every night.
|
||||
#prod project:packetstation packetstation /var/backups/packetstation/prod
|
||||
# /etc/packetstation/backup.conf: the stacks ps-backup backs up, one per line.
|
||||
# deploy/vm/install.sh writes the VM's own line when the file is missing. A VM has one stack.
|
||||
# name database folder
|
||||
prod packetstation /var/backups/packetstation
|
||||
|
||||
@@ -1,25 +1,29 @@
|
||||
# Packet Station nightly database backup (scripts/backup/ps-backup.sh).
|
||||
# Installed by root on docker-1: see scripts/backup/README.md. Started by packetstation-backup.timer.
|
||||
# Runs as root because it talks to the Docker daemon and writes /var/backups/packetstation.
|
||||
# Packet Station nightly database backup on a native VM (scripts/backup/ps-backup.sh, installed as
|
||||
# /usr/local/sbin/ps-backup by deploy/vm/install.sh). Started by packetstation-backup.timer.
|
||||
# Runs as root: it reads the backup config, writes /var/backups/packetstation and switches to the
|
||||
# postgres OS user (setpriv) for pg_dump on the local socket.
|
||||
[Unit]
|
||||
Description=Packet Station database backup (pg_dump of each stack)
|
||||
Documentation=file:///home/spencerm/packetstation/repo/scripts/backup/README.md
|
||||
Wants=docker.service
|
||||
After=docker.service
|
||||
Description=Packet Station database backup (pg_dump)
|
||||
Documentation=file:///opt/packetstation/current/docs/DEPLOY.md
|
||||
Wants=[email protected]
|
||||
After=[email protected]
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/ps-backup.sh --config /etc/packetstation/backup.conf
|
||||
ExecStart=/usr/local/sbin/ps-backup --config /etc/packetstation/backup.conf
|
||||
Environment=KEEP=7
|
||||
# Gentle on the host: the dump runs while the shop is closed, but the stacks stay up.
|
||||
# Gentle on the VM: the dump runs while the shop is closed, but the app stays up.
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=1h
|
||||
# Hardening. /usr, /boot and /etc are read-only to the script; it writes only the backup folders.
|
||||
# Sandbox. Read-only file system except the backup folder. setpriv needs CAP_SETUID and CAP_SETGID.
|
||||
NoNewPrivileges=yes
|
||||
PrivateTmp=yes
|
||||
ProtectSystem=full
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/var/backups/packetstation
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectControlGroups=yes
|
||||
RestrictSUIDSGID=yes
|
||||
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Packet Station nightly backup at 02:00 Denver time, an hour before the 03:00 Proxmox Backup
|
||||
# Server run, so PBS captures the fresh dumps. The host clock is UTC; the zone makes the intent
|
||||
# Server run, so PBS captures the fresh dumps. The VM clock may be UTC; the zone makes the intent
|
||||
# explicit and follows daylight saving. Persistent=true runs a missed backup at the next boot.
|
||||
[Unit]
|
||||
Description=Packet Station nightly database backup
|
||||
|
||||
+44
-60
@@ -1,50 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Packet Station nightly backup. Runs on the docker host, never inside the app.
|
||||
# Packet Station nightly backup on a native VM. Runs as root from packetstation-backup.timer,
|
||||
# outside the app, so a broken app still backs up.
|
||||
#
|
||||
# For each configured stack it writes, into that stack's backup folder:
|
||||
# ps-<UTC stamp>.dump pg_dump -Fc of the app database (custom format)
|
||||
# ps-<UTC stamp>.globals.sql pg_dumpall --globals-only --no-role-passwords (roles, no password hashes)
|
||||
# last-success / last-failure one line: "<ISO UTC time> <dump name>" (last-failure adds the reason)
|
||||
#
|
||||
# Each dump is written under a temporary name, checked with `pg_restore --list` run inside the
|
||||
# postgres container, and only then renamed into place. The newest KEEP dumps (default 7) are kept,
|
||||
# with their globals files. Any failure writes last-failure and the script exits non-zero after
|
||||
# trying the remaining stacks.
|
||||
# Each dump is written under a temporary name, checked with `pg_restore --list`, and only then
|
||||
# renamed into place. The newest KEEP dumps (default 7) are kept, with their globals files. Any
|
||||
# failure writes last-failure and the script exits non-zero after trying the remaining stacks.
|
||||
#
|
||||
# Usage:
|
||||
# ps-backup.sh [--config FILE] [STACK ...]
|
||||
# ps-backup.sh --stack NAME CONTAINER DATABASE FOLDER [--stack ...]
|
||||
# ps-backup [--config FILE] [--keep N] [STACK ...]
|
||||
# ps-backup --stack NAME DATABASE FOLDER [--stack ...] [--keep N]
|
||||
#
|
||||
# Config file (default /etc/packetstation/backup.conf, or PS_BACKUP_CONFIG), one stack per line:
|
||||
# # name container database folder
|
||||
# prod project:packetstation packetstation /var/backups/packetstation/prod
|
||||
# test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
|
||||
# CONTAINER is a container name, or project:<compose project> to find that project's postgres
|
||||
# service by its compose labels. With STACK names given, only those stacks run.
|
||||
# # name database folder
|
||||
# prod packetstation /var/backups/packetstation
|
||||
# With STACK names given, only those stacks run. A VM has one stack; install.sh writes its line.
|
||||
#
|
||||
# pg_dump connects inside the container as the postgres superuser over the local socket, which
|
||||
# the official image trusts. No password is read, passed or printed.
|
||||
# pg_dump and pg_dumpall run as the postgres OS user (setpriv, no PAM) on the local socket, where
|
||||
# PostgreSQL knows that user by peer authentication. No password is read, passed or printed. Their
|
||||
# output goes to files root opened, so the postgres user needs no access to the folder.
|
||||
# PG_BINDIR (default /usr/lib/postgresql/16/bin) picks the client tools, PG_AS the user switch.
|
||||
#
|
||||
# Globals: --no-role-passwords, so no password hash ever lands in a file (or in the PBS copy of
|
||||
# it). The roles' passwords come from the stack's secrets: initdb (deploy/postgres/initdb)
|
||||
# recreates them on a fresh volume. The globals file is also 0600, like the dumps.
|
||||
# it). The role passwords are in /etc/packetstation/secrets; deploy/vm/install.sh sets them.
|
||||
#
|
||||
# Files: dumps and globals are 0600. The markers are 0644 (a time and a file name, nothing else)
|
||||
# so the worker, which mounts the folder read-only as uid 10001, can read them. When run as root
|
||||
# the new files take the folder's owner and group, so a stack folder owned by spencerm stays
|
||||
# readable by spencerm.
|
||||
# so the worker, which reaches the folder through its group, can read them. When run as root the
|
||||
# new files take the folder's owner and group.
|
||||
#
|
||||
# Not in the dump: the stored files (/var/lib/packetstation/assets), the env file and the secrets
|
||||
# (/etc/packetstation). The PBS backup of the VM covers them. See docs/DEPLOY.md.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
KEEP="${KEEP:-7}"
|
||||
PG_USER="${PG_USER:-postgres}"
|
||||
DOCKER="${DOCKER:-docker}"
|
||||
PG_BINDIR="${PG_BINDIR-/usr/lib/postgresql/16/bin}"
|
||||
# The command prefix that runs a client tool as the postgres OS user, split on spaces.
|
||||
read -r -a PG_AS <<< "${PG_AS:-setpriv --reuid=postgres --regid=postgres --init-groups --reset-env --}"
|
||||
|
||||
log() { printf '%s ps-backup: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//' >&2
|
||||
sed -n '2,35p' "$0" | sed 's/^# \{0,1\}//' >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
@@ -53,7 +55,7 @@ usage() {
|
||||
config="${PS_BACKUP_CONFIG:-/etc/packetstation/backup.conf}"
|
||||
config_given=0
|
||||
declare -a only=()
|
||||
declare -a stacks=() # each entry: "name|container|database|folder"
|
||||
declare -a stacks=() # each entry: "name|database|folder"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
@@ -61,8 +63,8 @@ while [[ $# -gt 0 ]]; do
|
||||
[[ $# -ge 2 ]] || usage
|
||||
config="$2"; config_given=1; shift 2 ;;
|
||||
--stack)
|
||||
[[ $# -ge 5 ]] || usage
|
||||
stacks+=("$2|$3|$4|$5"); shift 5 ;;
|
||||
[[ $# -ge 4 ]] || usage
|
||||
stacks+=("$2|$3|$4"); shift 4 ;;
|
||||
--keep)
|
||||
[[ $# -ge 2 ]] || usage
|
||||
KEEP="$2"; shift 2 ;;
|
||||
@@ -82,13 +84,13 @@ if [[ ${#stacks[@]} -eq 0 ]]; then
|
||||
log "no --stack given and the config $config is missing or unreadable"
|
||||
exit 2
|
||||
fi
|
||||
while read -r name container database folder extra; do
|
||||
while read -r name database folder extra; do
|
||||
[[ -z "${name:-}" || "${name:0:1}" == "#" ]] && continue
|
||||
if [[ -z "${folder:-}" || -n "${extra:-}" ]]; then
|
||||
log "config line for '$name' needs exactly: name container database folder"
|
||||
log "config line for '$name' needs exactly: name database folder"
|
||||
exit 2
|
||||
fi
|
||||
stacks+=("$name|$container|$database|$folder")
|
||||
stacks+=("$name|$database|$folder")
|
||||
done < "$config"
|
||||
elif [[ $config_given -eq 1 ]]; then
|
||||
log "use --config or --stack, not both"
|
||||
@@ -102,26 +104,9 @@ fi
|
||||
|
||||
# ------------------------------------------------------------------ helpers
|
||||
|
||||
# Resolve "project:<name>" to the running postgres container of that compose project.
|
||||
resolve_container() {
|
||||
local spec="$1"
|
||||
if [[ "$spec" == project:* ]]; then
|
||||
local project="${spec#project:}" ids
|
||||
ids="$("$DOCKER" ps -q \
|
||||
--filter "label=com.docker.compose.project=$project" \
|
||||
--filter "label=com.docker.compose.service=postgres")"
|
||||
if [[ -z "$ids" ]]; then
|
||||
echo "no running postgres container for compose project $project" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(printf '%s\n' "$ids" | wc -l)" -ne 1 ]]; then
|
||||
echo "more than one postgres container for compose project $project" >&2
|
||||
return 1
|
||||
fi
|
||||
printf '%s\n' "$ids"
|
||||
else
|
||||
printf '%s\n' "$spec"
|
||||
fi
|
||||
# A PostgreSQL client tool: from PG_BINDIR when it is set, else from PATH.
|
||||
pgbin() {
|
||||
if [[ -n "$PG_BINDIR" ]]; then printf '%s/%s\n' "$PG_BINDIR" "$1"; else printf '%s\n' "$1"; fi
|
||||
}
|
||||
|
||||
# Write a marker atomically: temp file in the same folder, then rename.
|
||||
@@ -203,8 +188,8 @@ on_stack_exit() {
|
||||
}
|
||||
|
||||
backup_stack() {
|
||||
local name="$1" spec="$2" database="$3" folder="$4"
|
||||
local stamp globals_name container started entries size
|
||||
local name="$1" database="$2" folder="$3"
|
||||
local stamp globals_name started entries size
|
||||
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
cur_dump_name="ps-$stamp.dump"
|
||||
@@ -213,34 +198,33 @@ backup_stack() {
|
||||
log "stack $name: database $database into $folder"
|
||||
|
||||
if [[ ! -d "$folder" ]]; then
|
||||
# 0750: the worker reaches the folder through its group (see scripts/backup/README.md).
|
||||
# 0750: the worker reaches the folder through its group (docs/DEPLOY.md, "Backups on a VM").
|
||||
mkdir -p -- "$folder"
|
||||
chmod 0750 -- "$folder" 2>/dev/null || log "stack $name: could not set mode 0750 on $folder"
|
||||
fi
|
||||
cur_folder="$folder"
|
||||
[[ -w "$folder" ]] || fail_stack "$folder is not writable"
|
||||
[[ "$database" =~ ^[A-Za-z_][A-Za-z0-9_]{0,62}$ ]] || fail_stack "database name '$database' is not a plain name"
|
||||
|
||||
if command -v flock >/dev/null 2>&1; then
|
||||
exec 9>"$folder/.ps-backup.lock"
|
||||
flock -n 9 || fail_stack "another backup of this stack is running"
|
||||
fi
|
||||
|
||||
container="$(resolve_container "$spec" 2>&1)" || fail_stack "$container"
|
||||
|
||||
cur_tmp_dump="$(mktemp "$folder/.$cur_dump_name.XXXXXX.partial")"
|
||||
cur_tmp_globals="$(mktemp "$folder/.$globals_name.XXXXXX.partial")"
|
||||
|
||||
"$DOCKER" exec "$container" pg_dump -U "$PG_USER" -d "$database" -Fc > "$cur_tmp_dump" \
|
||||
"${PG_AS[@]}" "$(pgbin pg_dump)" -d "$database" -Fc > "$cur_tmp_dump" \
|
||||
|| fail_stack "pg_dump failed"
|
||||
[[ -s "$cur_tmp_dump" ]] || fail_stack "pg_dump wrote nothing"
|
||||
|
||||
"$DOCKER" exec "$container" pg_dumpall -U "$PG_USER" --globals-only --no-role-passwords \
|
||||
"${PG_AS[@]}" "$(pgbin pg_dumpall)" --globals-only --no-role-passwords \
|
||||
> "$cur_tmp_globals" || fail_stack "pg_dumpall --globals-only failed"
|
||||
[[ -s "$cur_tmp_globals" ]] || fail_stack "pg_dumpall wrote nothing"
|
||||
|
||||
# The check reads the dump from stdin with the container's own pg_restore. Entry lines start
|
||||
# with the entry's id; comment lines start with a semicolon.
|
||||
entries="$("$DOCKER" exec -i "$container" pg_restore --list < "$cur_tmp_dump" | grep -c '^[0-9]')" \
|
||||
# The check reads the dump from stdin and needs no database. Entry lines start with the
|
||||
# entry's id; comment lines start with a semicolon.
|
||||
entries="$("$(pgbin pg_restore)" --list < "$cur_tmp_dump" | grep -c '^[0-9]')" \
|
||||
|| fail_stack "pg_restore --list could not read the dump"
|
||||
[[ "$entries" -ge 1 ]] || fail_stack "pg_restore --list found no entries"
|
||||
|
||||
@@ -269,7 +253,7 @@ run_stack() (
|
||||
failures=0
|
||||
ran=0
|
||||
for entry in "${stacks[@]}"; do
|
||||
IFS='|' read -r s_name s_container s_database s_folder <<< "$entry"
|
||||
IFS='|' read -r s_name s_database s_folder <<< "$entry"
|
||||
if [[ ${#only[@]} -gt 0 ]]; then
|
||||
wanted=0
|
||||
for o in "${only[@]}"; do
|
||||
@@ -280,7 +264,7 @@ for entry in "${stacks[@]}"; do
|
||||
ran=$(( ran + 1 ))
|
||||
# Not in an if or ||: that would switch errexit off inside the subshell.
|
||||
set +e
|
||||
run_stack "$s_name" "$s_container" "$s_database" "$s_folder"
|
||||
run_stack "$s_name" "$s_database" "$s_folder"
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -ne 0 ]]; then
|
||||
|
||||
@@ -1,75 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# Packet Station restore. Runs on the docker host. Restores a ps-backup.sh dump into a fresh
|
||||
# database on a stack's postgres container, optionally applies the globals file, copies assets
|
||||
# back into an assets volume, and prints the row count of every table at the end.
|
||||
# Packet Station restore on a native VM. Run by root. Restores a ps-backup dump into a fresh
|
||||
# database, optionally applies the globals file, copies stored files back into an assets folder,
|
||||
# checks them against the database, and prints the row count of every table at the end.
|
||||
#
|
||||
# It never stops, restarts or reconfigures a stack. Pointing the app at the restored database
|
||||
# (or renaming it into place) is a separate, deliberate step: see scripts/backup/README.md.
|
||||
# It never stops, starts or reconfigures a service. Putting the restored database into service
|
||||
# (renaming it into place) is a separate, deliberate step: see docs/DEPLOY.md, "Restore on a VM".
|
||||
#
|
||||
# Usage:
|
||||
# ps-restore.sh --container NAME | --project PROJECT
|
||||
# [--dump FILE --db NAME [--replace]]
|
||||
# [--globals FILE]
|
||||
# [--assets DIR_OR_TAR --assets-volume VOLUME]
|
||||
# [--verify-assets --db NAME --assets-volume VOLUME]
|
||||
# [--counts-only --db NAME]
|
||||
# ps-restore [--dump FILE --db NAME [--replace]]
|
||||
# [--globals FILE]
|
||||
# [--assets DIR_OR_TAR --assets-dir DIR]
|
||||
# [--verify-assets --db NAME --assets-dir DIR]
|
||||
# [--counts-only --db NAME]
|
||||
#
|
||||
# --container / --project the postgres container, by name or by compose project.
|
||||
# --dump FILE a ps-*.dump file (pg_dump -Fc).
|
||||
# --db NAME the database to restore into. It must not exist, unless --replace,
|
||||
# which drops it first. DROP DATABASE refuses while anyone is connected.
|
||||
# --owner ROLE owner of the new database (default ps_migrate, as initdb makes it).
|
||||
# --replace is refused for packetstation, the live database.
|
||||
# --owner ROLE owner of the new database (default ps_migrate, as install.sh makes it).
|
||||
# --app-role ROLE role given CONNECT and TEMPORARY on it (default ps_app).
|
||||
# --locale NAME the new database's locale (default en_US.UTF-8, as install.sh makes it).
|
||||
# --globals FILE a ps-*.globals.sql file. Roles that already exist are reported and
|
||||
# left alone. The file holds no passwords: set them from the secrets.
|
||||
# left alone. The file holds no passwords: install.sh sets them again.
|
||||
# --assets SRC a folder, or a .tar / .tar.gz / .tgz, holding the files that belong
|
||||
# under ASSETS_DIR. Copied into --assets-volume through a throwaway
|
||||
# container (no network) that runs the postgres image already on the host.
|
||||
# Existing files are never overwritten.
|
||||
# --assets-volume VOLUME the docker volume to copy into (for example packetstation-test_assets).
|
||||
# --assets-owner UID:GID owner for the copied files (default 10001:10001, the image's ps user).
|
||||
# --verify-assets compare sha256 of every file in --assets-volume with assets.sha256 in --db.
|
||||
# under ASSETS_DIR. Existing files are never overwritten.
|
||||
# --assets-dir DIR the folder to copy into or check (/var/lib/packetstation/assets is the
|
||||
# live one; a scratch folder for a rehearsal).
|
||||
# --assets-owner U:G owner for the copied files (default packetstation:packetstation).
|
||||
# --verify-assets compare sha256 of every file in --assets-dir with assets.sha256 in --db.
|
||||
# --counts-only only print the row counts of --db.
|
||||
#
|
||||
# Connects inside the container as the postgres superuser over the local socket. No password
|
||||
# is read, passed or printed.
|
||||
# psql and pg_restore run as the postgres OS user on the local socket (peer authentication). The
|
||||
# files are opened by root and handed over on stdin. No password is read, passed or printed.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
PG_USER="${PG_USER:-postgres}"
|
||||
DOCKER="${DOCKER:-docker}"
|
||||
PG_BINDIR="${PG_BINDIR-/usr/lib/postgresql/16/bin}"
|
||||
read -r -a PG_AS <<< "${PG_AS:-setpriv --reuid=postgres --regid=postgres --init-groups --reset-env --}"
|
||||
LIVE_DB=packetstation
|
||||
|
||||
log() { printf '%s ps-restore: %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2; }
|
||||
die() { log "$*"; exit 1; }
|
||||
usage() { sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; }
|
||||
usage() { sed -n '2,33p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; }
|
||||
|
||||
container=""
|
||||
project=""
|
||||
dump=""
|
||||
db=""
|
||||
replace=0
|
||||
owner="ps_migrate"
|
||||
app_role="ps_app"
|
||||
locale_name="en_US.UTF-8"
|
||||
globals=""
|
||||
assets=""
|
||||
assets_volume=""
|
||||
assets_owner="10001:10001"
|
||||
assets_dir=""
|
||||
assets_owner="packetstation:packetstation"
|
||||
verify_assets=0
|
||||
counts_only=0
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--container) container="${2:?}"; shift 2 ;;
|
||||
--project) project="${2:?}"; shift 2 ;;
|
||||
--dump) dump="${2:?}"; shift 2 ;;
|
||||
--db) db="${2:?}"; shift 2 ;;
|
||||
--replace) replace=1; shift ;;
|
||||
--owner) owner="${2:?}"; shift 2 ;;
|
||||
--app-role) app_role="${2:?}"; shift 2 ;;
|
||||
--locale) locale_name="${2:?}"; shift 2 ;;
|
||||
--globals) globals="${2:?}"; shift 2 ;;
|
||||
--assets) assets="${2:?}"; shift 2 ;;
|
||||
--assets-volume) assets_volume="${2:?}"; shift 2 ;;
|
||||
--assets-dir) assets_dir="${2:?}"; shift 2 ;;
|
||||
--assets-owner) assets_owner="${2:?}"; shift 2 ;;
|
||||
--verify-assets) verify_assets=1; shift ;;
|
||||
--counts-only) counts_only=1; shift ;;
|
||||
@@ -82,32 +80,31 @@ ident_re='^[a-z_][a-z0-9_]{0,62}$'
|
||||
[[ -z "$db" || "$db" =~ $ident_re ]] || die "--db must be a plain lower-case name (letters, digits, _)"
|
||||
[[ "$owner" =~ $ident_re ]] || die "--owner must be a plain lower-case role name"
|
||||
[[ "$app_role" =~ $ident_re ]] || die "--app-role must be a plain lower-case role name"
|
||||
[[ "$assets_owner" =~ ^[0-9]+:[0-9]+$ ]] || die "--assets-owner must be UID:GID"
|
||||
|
||||
if [[ -n "$project" ]]; then
|
||||
[[ -z "$container" ]] || die "use --container or --project, not both"
|
||||
container="$("$DOCKER" ps -q \
|
||||
--filter "label=com.docker.compose.project=$project" \
|
||||
--filter "label=com.docker.compose.service=postgres")"
|
||||
[[ -n "$container" ]] || die "no running postgres container for compose project $project"
|
||||
[[ "$(printf '%s\n' "$container" | wc -l)" -eq 1 ]] || die "more than one postgres container for $project"
|
||||
fi
|
||||
[[ -n "$container" ]] || die "--container or --project is required"
|
||||
[[ "$locale_name" =~ ^[A-Za-z0-9_.@-]+$ ]] || die "--locale must be a locale name like en_US.UTF-8"
|
||||
[[ "$assets_owner" =~ ^[a-z_0-9][a-z0-9_-]*:[a-z_0-9][a-z0-9_-]*$ ]] || die "--assets-owner must be USER:GROUP"
|
||||
|
||||
if [[ -n "$dump" || $counts_only -eq 1 || $verify_assets -eq 1 ]]; then
|
||||
[[ -n "$db" ]] || die "--db is required"
|
||||
fi
|
||||
if [[ -n "$assets" || $verify_assets -eq 1 ]]; then
|
||||
[[ -n "$assets_volume" ]] || die "--assets-volume is required"
|
||||
[[ -n "$assets_dir" ]] || die "--assets-dir is required"
|
||||
[[ -d "$assets_dir" ]] || die "$assets_dir is not a folder"
|
||||
fi
|
||||
if [[ -z "$dump" && -z "$globals" && -z "$assets" && $verify_assets -eq 0 && $counts_only -eq 0 ]]; then
|
||||
die "nothing to do: give --dump, --globals, --assets, --verify-assets or --counts-only"
|
||||
fi
|
||||
if [[ -n "$dump" && $replace -eq 1 && "$db" == "$LIVE_DB" ]]; then
|
||||
die "--replace is refused for $LIVE_DB. Restore into another name and rename it into place (docs/DEPLOY.md)."
|
||||
fi
|
||||
|
||||
# psql inside the container, reading SQL from stdin. Variables go through -v and :'name'.
|
||||
pgbin() {
|
||||
if [[ -n "$PG_BINDIR" ]]; then printf '%s/%s\n' "$PG_BINDIR" "$1"; else printf '%s\n' "$1"; fi
|
||||
}
|
||||
|
||||
# psql as postgres, reading SQL from stdin. Variables go through -v and :'name'.
|
||||
psql_c() {
|
||||
local dbname="$1"; shift
|
||||
"$DOCKER" exec -i "$container" psql -X -q -v ON_ERROR_STOP=1 -U "$PG_USER" -d "$dbname" "$@"
|
||||
"${PG_AS[@]}" "$(pgbin psql)" -X -q -v ON_ERROR_STOP=1 -d "$dbname" "$@"
|
||||
}
|
||||
|
||||
db_exists() {
|
||||
@@ -143,22 +140,16 @@ ORDER BY c.relname;
|
||||
SQL
|
||||
}
|
||||
|
||||
# A throwaway container on no network, running the image the postgres container already uses,
|
||||
# so nothing is downloaded.
|
||||
helper_image() {
|
||||
"$DOCKER" inspect -f '{{.Config.Image}}' "$container"
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------ globals
|
||||
|
||||
if [[ -n "$globals" ]]; then
|
||||
[[ -r "$globals" ]] || die "cannot read $globals"
|
||||
if grep -qiE "PASSWORD[[:space:]]+'(SCRAM|md5)" "$globals"; then
|
||||
die "$globals holds password hashes; ps-backup.sh writes globals without them. Refusing."
|
||||
die "$globals holds password hashes; ps-backup writes globals without them. Refusing."
|
||||
fi
|
||||
log "applying globals from $(basename -- "$globals") (existing roles are left as they are)"
|
||||
# Without ON_ERROR_STOP: CREATE ROLE for a role that exists fails and the rest goes on.
|
||||
out="$("$DOCKER" exec -i "$container" psql -X -q -U "$PG_USER" -d postgres < "$globals" 2>&1 || true)"
|
||||
out="$("${PG_AS[@]}" "$(pgbin psql)" -X -q -d postgres < "$globals" 2>&1 || true)"
|
||||
exists="$(printf '%s\n' "$out" | grep -c 'already exists' || true)"
|
||||
other="$(printf '%s\n' "$out" | grep 'ERROR' | grep -vc 'already exists' || true)"
|
||||
log "globals: $exists statements skipped (already exists), $other other errors"
|
||||
@@ -180,12 +171,13 @@ if [[ -n "$dump" ]]; then
|
||||
SELECT format('DROP DATABASE %I', :'name') \gexec
|
||||
SQL
|
||||
fi
|
||||
role_exists "$owner" || die "role $owner does not exist (apply --globals or start the stack once)"
|
||||
role_exists "$owner" || die "role $owner does not exist (apply --globals, or run deploy/vm/install.sh)"
|
||||
|
||||
log "creating database $db owned by $owner"
|
||||
# The same shape as deploy/postgres/initdb/01-roles.sh.
|
||||
psql_c postgres -v name="$db" -v owner="$owner" <<'SQL'
|
||||
SELECT format('CREATE DATABASE %I OWNER %I ENCODING %L TEMPLATE template0', :'name', :'owner', 'UTF8') \gexec
|
||||
# The same shape as deploy/vm/install.sh makes the live one.
|
||||
psql_c postgres -v name="$db" -v owner="$owner" -v loc="$locale_name" <<'SQL'
|
||||
SELECT format('CREATE DATABASE %I OWNER %I ENCODING %L LOCALE %L TEMPLATE template0',
|
||||
:'name', :'owner', 'UTF8', :'loc') \gexec
|
||||
SELECT format('REVOKE ALL ON DATABASE %I FROM PUBLIC', :'name') \gexec
|
||||
SQL
|
||||
if role_exists "$app_role"; then
|
||||
@@ -198,8 +190,8 @@ SQL
|
||||
|
||||
log "restoring $(basename -- "$dump") ($(stat -c %s "$dump") bytes) into $db"
|
||||
# One transaction: a restore that fails leaves an empty database, never a half-restored one.
|
||||
"$DOCKER" exec -i "$container" pg_restore -U "$PG_USER" -d "$db" --exit-on-error --single-transaction \
|
||||
< "$dump" || die "pg_restore failed; database $db is left empty. Drop it with --replace on the next try."
|
||||
"${PG_AS[@]}" "$(pgbin pg_restore)" -d "$db" --exit-on-error --single-transaction < "$dump" \
|
||||
|| die "pg_restore failed; database $db is left empty. Drop it with --replace on the next try."
|
||||
log "database restored in $(( SECONDS - started )) s"
|
||||
fi
|
||||
|
||||
@@ -207,26 +199,21 @@ fi
|
||||
|
||||
if [[ -n "$assets" ]]; then
|
||||
[[ -e "$assets" ]] || die "$assets does not exist"
|
||||
"$DOCKER" volume inspect "$assets_volume" >/dev/null 2>&1 || die "volume $assets_volume does not exist"
|
||||
image="$(helper_image)"
|
||||
started=$SECONDS
|
||||
src="$(cd -- "$(dirname -- "$assets")" && pwd)/$(basename -- "$assets")"
|
||||
if [[ -d "$assets" ]]; then
|
||||
log "copying assets from folder $src into volume $assets_volume"
|
||||
"$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
|
||||
-v "$assets_volume:/dst" -v "$src:/src:ro" "$image" -c \
|
||||
'set -eu; cp -a -n /src/. /dst/; chown -R "$1" /dst' sh "$assets_owner"
|
||||
log "copying assets from folder $assets into $assets_dir"
|
||||
cp -a -n -- "$assets/." "$assets_dir/"
|
||||
else
|
||||
case "$assets" in
|
||||
*.tar.gz|*.tgz) tar_flags="-xzf" ;;
|
||||
*.tar) tar_flags="-xf" ;;
|
||||
*) die "--assets must be a folder, .tar, .tar.gz or .tgz" ;;
|
||||
esac
|
||||
log "copying assets from $(basename -- "$src") into volume $assets_volume"
|
||||
"$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
|
||||
-v "$assets_volume:/dst" -v "$src:/src.tar:ro" "$image" -c \
|
||||
'set -eu; tar "$1" /src.tar -C /dst --skip-old-files --no-same-owner; chown -R "$2" /dst' \
|
||||
sh "$tar_flags" "$assets_owner"
|
||||
log "copying assets from $(basename -- "$assets") into $assets_dir"
|
||||
tar "$tar_flags" "$assets" -C "$assets_dir" --skip-old-files --no-same-owner
|
||||
fi
|
||||
if [[ "$(id -u)" == "0" ]]; then
|
||||
chown -R -- "$assets_owner" "$assets_dir"
|
||||
fi
|
||||
log "assets copied in $(( SECONDS - started )) s"
|
||||
fi
|
||||
@@ -234,8 +221,6 @@ fi
|
||||
# ------------------------------------------------------------------ verify assets
|
||||
|
||||
if [[ $verify_assets -eq 1 ]]; then
|
||||
"$DOCKER" volume inspect "$assets_volume" >/dev/null 2>&1 || die "volume $assets_volume does not exist"
|
||||
image="$(helper_image)"
|
||||
work="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
# sha256sum check format: "<hex> <path>".
|
||||
@@ -243,21 +228,11 @@ if [[ $verify_assets -eq 1 ]]; then
|
||||
SELECT encode(sha256, 'hex') || ' ' || storage_key FROM assets ORDER BY storage_key;
|
||||
SQL
|
||||
rows="$(wc -l < "$work/assets.sha256")"
|
||||
chmod 0755 "$work"
|
||||
chmod 0644 "$work/assets.sha256"
|
||||
log "checking $rows asset rows of $db against volume $assets_volume"
|
||||
set +e
|
||||
"$DOCKER" run --rm --network none --user 0:0 --entrypoint sh \
|
||||
-v "$assets_volume:/dst:ro" -v "$work:/work:ro" "$image" -c '
|
||||
cd /dst || exit 3
|
||||
sha256sum -c /work/assets.sha256 2>/dev/null
|
||||
find . -type f ! -path "./tmp/*" | sed "s|^\./||" | sort > /tmp/files
|
||||
cut -c67- /work/assets.sha256 | sort > /tmp/rows
|
||||
comm -23 /tmp/files /tmp/rows | sed "s|^|ORPHAN: |"
|
||||
exit 0' > "$work/result"
|
||||
rc=$?
|
||||
set -e
|
||||
[[ $rc -eq 0 ]] || die "the check container failed (exit $rc)"
|
||||
log "checking $rows asset rows of $db against $assets_dir"
|
||||
( cd -- "$assets_dir" && sha256sum -c "$work/assets.sha256" 2>/dev/null ) > "$work/result" || true
|
||||
( cd -- "$assets_dir" && find . -type f ! -path './tmp/*' | sed 's|^\./||' | LC_ALL=C sort ) > "$work/files"
|
||||
cut -c67- "$work/assets.sha256" | LC_ALL=C sort > "$work/rows"
|
||||
LC_ALL=C comm -23 "$work/files" "$work/rows" | sed 's|^|ORPHAN: |' >> "$work/result"
|
||||
ok="$(grep -c ': OK$' "$work/result" || true)"
|
||||
bad="$(grep -c ': FAILED$' "$work/result" || true)"
|
||||
missing="$(grep -c ': FAILED open or read$' "$work/result" || true)"
|
||||
|
||||
Reference in New Issue
Block a user