spencerm/PacketStation
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 9: Admin completion, reconcile, notifications, integrity, backups
ci / frontend (pull_request) Successful in 1m11s
Details
ci / server (pull_request) Successful in 12m2s
Details
ci / image (pull_request) Skipped
Details
ci / e2e (pull_request) Successful in 6m1s
Details

Browse Source 
Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
spencerm
2026-10-21 18:27:09 -06:00
co-authored by Claude Opus 5.5
parent ce28d2f3c6
commit a82e056b6a
116 changed files with 15125 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DEPLOY.md
+162 -10
View File
@@ -32,7 +32,7 @@ Both stacks run from the same image as separate compose projects.
- `postgres`: PostgreSQL 16, named volume, `backend` network only. Its init creates the database roles from secrets.
- `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
- `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup and housekeeping (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only (used from phase 9). Its healthcheck reads the heartbeat file.
- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup, housekeeping and the phase 9 jobs: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only at `/backups` (the backup-check job reads it). Its healthcheck reads the heartbeat file.
- `clamav` (optional, profile `clamav`, phase 4): `clamav/clamav:1.4`, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias `packetstation-clamav` (as `postgres` has its own alias). `backend` network plus egress (freshclam downloads the signatures), no `npm_default`, no published port. Volume `clamav-db` at `/var/lib/clamav`, `mem_limit: 2g`, healthcheck `clamdcheck.sh`. Started only with `--profile clamav`. See "Virus scan (ClamAV)".
- All services restart unless stopped, log to stdout (json-file), and have healthchecks.
@@ -66,9 +66,10 @@ Key settings:
- `VISUAL_SERVER`, `VISUAL_DATABASE`, `VISUAL_USER=visualkit_ro`, `VISUAL_PASSWORD_FILE`.
- `MFILES_URL=https://mfiles.pescoinc.biz/REST`, `MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}`, `MFILES_USERNAME=mfileskitRO`, `MFILES_CA_BUNDLE`.
- `LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636`, `LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com`, `LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com`, `LDAP_PASSWORD_FILE`, `LDAP_CA_BUNDLE`.
- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`.
- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`, `NOTIFY_DRY_RUN` (`true` on the test stack, `false` on production). `STACK` (`test` or `prod`) picks the subject prefix. See "Notifications".
- `BACKUP_DIR`: the stack's backup folder on the host, mounted read-only in the worker. See "Backup".
- `APP_ALIAS` per stack.
- Identity timing and limits, all optional with these defaults: `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_DESK_MINUTES=60` (Admin and Clerical wait at least this long for both), `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
- Identity timing and limits, all optional with these defaults (an Admin can override most of them within bounds on the Security tab; see [SECURITY.md](SECURITY.md), "Security settings"): `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_DESK_MINUTES=60` (Admin and Clerical wait at least this long for both), `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
- Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
- Uploads and storage, all optional: `UPLOAD_MAX_BYTES` (default 25 MB, drawings, MTRs and documents), `UPLOAD_MAX_BYTES_BOM` (default 5 MB, hand BOMs), `ASSETS_QUOTA_WARN_PCT` (default 85, 1 to 99) and `ASSETS_QUOTA_BYTES` (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
- Virus scan, all optional: `CLAMAV_HOST` (empty by default, which means no scan; `packetstation-clamav` for the profile service), `CLAMAV_PORT` (default 3310) and `CLAMAV_TIMEOUT_SECONDS` (default 30). The env examples carry `#CLAMAV_HOST=packetstation-clamav` commented out. See "Virus scan (ClamAV)".
@@ -119,7 +120,7 @@ clamd holds the signatures in memory, about 1.2 GB with the full set and more wh
Every stored file and the housekeeping job check the assets volume. With `ASSETS_QUOTA_BYTES=0` (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At `ASSETS_QUOTA_WARN_PCT` percent (85 by default) the app opens one `notification_events` alert of kind `assets_quota` and audits `storage.quota_warning`. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.
The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. Phase 9 sends the alert by email to PS-Admins (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold). Until then, check the Storage tab.
The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. The alert is mailed through the `assets_quota` notification rule, to PS-Admins by default (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold), with a "Cleared" message when usage drops back.
## Python packages and the Gitea index
@@ -253,7 +254,7 @@ It checks:
- `visualkit doctor` and `mfileskit doctor`. This is the first real Linux test of both libraries.
- Database connection, roles, migrations at head, and an audit UPDATE that must fail.
- LISTEN/NOTIFY round trip.
- Assets volume and free space. Backup folder (planned, phase 9).
- Assets volume and free space. The backup folder is readable (SKIP when it is not mounted).
- LDAP TLS bind on both controllers, reading `memberOf`, `employeeID` and `userAccountControl` as the service account, the 50-group count and one nested-group query.
- SMTP STARTTLS with certificate verification.
- ClamAV (phase 4): clamd answers PING at `CLAMAV_HOST`. SKIP when no host is set.
@@ -284,7 +285,7 @@ The real PESCO domain controllers are never used by CI. The test and production
CI never reads VISUAL or M-Files. `deploy/ci/ci.env` sets `FEED_SOURCE=fixture`, `DRAWING_SOURCE=fixture` and `FIXTURE_DIR=/fixtures`. The image does not contain `tests/`, so the e2e job generates the fixtures on the runner before the stack starts:
```bash
python -m pip install pypdf==6.21.0
python -m pip install pypdf==6.21.1
python -m tests.fixtures.generator --out tests/fixtures/generated
```
@@ -310,10 +311,161 @@ Two runner settings were needed and are in deploy/runner: the actions cache serv
port 8088 (job containers cannot reach the runner's own network, and every cache call timed out), and the
job workspace on a host directory allowed in `valid_volumes` (so the stack's bind mounts resolve).
## Backup (planned, phase 9)
## Image rebuild for phase 9
`pg_dump` of both stacks from a systemd timer on the host at 02:00 America/Denver, before the 03:00 PBS run. Seven dumps kept in `/var/backups/packetstation/<stack>/` with last-success and last-failure markers. The worker's backup-check job alerts if the newest dump is missing or older than 26 hours. Scripts and units go in `scripts/`.
Phase 9 bumps two pins for security fixes found by the CI dependency scan: `pypdf` 6.21.0 to 6.21.1 and `certifi` 2026.7.22 to 2026.10.19 (`server/requirements.txt` and `pyproject.toml`). They reach a stack only with a new image. Build and tag the phase 9 commit as usual (`:sha-<sha>` from CI), set it in the stack's env file, and `up -d`. Migration 0010 runs on the way up. Then run the doctor.
## Restore (planned, phase 9)
## Notifications (phase 9)
Restore the dump into a fresh database and copy the assets back, from the live folder or a file-level PBS restore. Rehearsed once on the test stack and written up here. A whole-VM restore from PBS is the disaster path.
The app sends test messages and the worker sends everything else, through PESCO's relay.
| Setting | Test stack | Production | CI |
|---|---|---|---|
| `SMTP_HOST` | `app-03.pesco.com` | `app-03.pesco.com` | empty (mail off) |
| `SMTP_PORT` | 25 | 25 | 25 |
| `SMTP_STARTTLS` | `true` | `true` | `true` |
| `SMTP_FROM` | `noreply@pescoinc.biz` | `noreply@pescoinc.biz` | `noreply@pescoinc.biz` |
| `SMTP_CA_BUNDLE` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` |
| `NOTIFY_DRY_RUN` | `true` | `false` | `true` |
| `STACK` | `test` | `prod` | `ci` |
- STARTTLS always, verified against the PESCO CA, with the host name so the certificate name matches. No login. A relay that does not offer STARTTLS, or a certificate that does not verify, fails the message. There is no plain-text fallback.
- Subjects start `[Packet Station]` when `STACK=prod`, otherwise `[Packet Station TEST]`.
- Each message is addressed To `SMTP_FROM`. The recipients are in the envelope only.
- `NOTIFY_DRY_RUN=true` records every message as `dry_run` in the sent log and sends nothing. The test stack keeps it on, so the test stack never mails people about test data. Production sets it `false`.
- With `SMTP_HOST` empty, every message is recorded `failed` with "Mail is off: SMTP_HOST is empty."
- The test send. On the Admin page, Notifications tab, open a rule, put an address in "Send the test to" and press "Send a test". It sends at once from the app, even in dry run, and shows the result. The sent log keeps the row (marked Test). A test with no address goes to the rule's recipients. On a new stack, send one test to your own address before relying on the rules.
- Every rule starts on, sent at once, to the AD group PS-Admins. Change the recipients on the Notifications tab. Each person sees on Account settings what reaches them and why.
- The worker needs the LDAP bind to resolve group recipients. While AD is down, an immediate message waits and is tried twice more, then recorded `failed`.
The doctor's SMTP line checks STARTTLS with certificate verification, without sending.
## Backup (phase 9)
Host scripts in `scripts/backup/`, run by root from a systemd timer. They run outside the app, so a broken app still backs up. [scripts/backup/README.md](../scripts/backup/README.md) has the same steps with more detail.
| File | Installed as |
|---|---|
| `ps-backup.sh` | `/usr/local/sbin/ps-backup.sh` (root, 0755) |
| `ps-restore.sh` | `/usr/local/sbin/ps-restore.sh` (root, 0755) |
| `backup.conf.example` | `/etc/packetstation/backup.conf` (root, 0644) |
| `packetstation-backup.service` | `/etc/systemd/system/` (root, 0644) |
| `packetstation-backup.timer` | `/etc/systemd/system/` (root, 0644) |
`backup.conf` lists the stacks, one per line: name, container (or `project:<compose project>`), database, folder.
```
test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
#prod project:packetstation packetstation /var/backups/packetstation/prod
```
The production line stays commented out until the production stack runs (phase 10). Until then it would fail every night.
Where the dumps go:
- Test stack: `/home/spencerm/packetstation/test/backups`, owned by spencerm (the script gives new files the folder's owner and group). The test `stack.env` sets `BACKUP_DIR` to this path.
- Production: `/var/backups/packetstation/prod`, `root:10001` 0750. The production `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
The worker mounts the stack's folder read-only at `/backups`. Its `backup-check` job (07:15) only lists the folder, reads the dumps' sizes and times, and reads the markers.
What a run does, per stack:
1. Finds the stack's running postgres container (by compose labels for `project:`). None or two is a failure.
2. `docker exec <pg> pg_dump -U postgres -d packetstation -Fc` over the local socket, which the postgres image trusts. No password is read or passed. Written under a temporary name, `.ps-<stamp>.partial`.
3. `pg_dumpall --globals-only --no-role-passwords`: the roles without their password hashes. The role passwords are in the stack secrets.
4. `pg_restore --list` with the container's own pg_restore must list at least one entry. Then the files are renamed to `ps-<UTC yyyymmddThhmmssZ>.dump` and `.globals.sql`, mode 0600.
5. Keeps the newest 7 (`KEEP=7` in the unit, `--keep N`), with their globals files.
6. Writes `last-success` or `last-failure` (`<ISO UTC time> <dump name>`, plus the reason on failure), mode 0644. A failed run leaves no partial file.
Each stack runs on its own, so one failing does not stop the next. Exit 1 if any stack failed, 2 for a usage or config error. `flock` keeps two runs from overlapping.
The unit is `Type=oneshot` as root, `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=1h`, with `NoNewPrivileges`, `PrivateTmp` and `ProtectSystem=full` (no `ProtectHome`, so it can write the test folder under `/home`). The timer is `OnCalendar=*-*-* 02:00:00 America/Denver` with `Persistent=true`: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot.
### Install (Spencer, with sudo, on docker-1)
After the phase 9 tree is synced to `~/packetstation/repo`:
```bash
cd ~/packetstation/repo
sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh /usr/local/sbin/ps-backup.sh
sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
sudo install -d -o root -g root -m 0755 /etc/packetstation
sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
sudo systemctl enable --now packetstation-backup.timer
systemctl list-timers packetstation-backup.timer
sudo systemctl start packetstation-backup.service
journalctl -u packetstation-backup.service -n 20 --no-pager
ls -la ~/packetstation/test/backups
```
`list-timers` shows the next run at 08:00 or 09:00 UTC (02:00 in Denver). The one run by hand should leave a new `ps-*.dump` and a fresh `last-success`. After changing a script in the repo, run its `install` line again: the installed copies are root's, so a sync of the checkout does not change what root runs.
When the production stack exists (phase 10):
```bash
sudo install -d -o root -g root -m 0755 /var/backups/packetstation
sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
sudoedit /etc/packetstation/backup.conf # uncomment the prod line
sudo systemctl start packetstation-backup.service
```
A run by hand needs no sudo for the test stack (spencerm is in the docker group):
```bash
scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
```
The dumps hold the database only. The assets volume (the files) is not in them. It is covered by PBS, since the volume is part of the VM.
## Restore (phase 9)
`ps-restore.sh` never stops, restarts or reconfigures a stack. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.
```bash
B=/home/spencerm/packetstation/test/backups
D=$(ls $B/ps-*.dump | tail -1)
# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is connected).
ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
# 2. Assets, from a folder or a tar (for example a PBS file-level restore of the assets volume folder).
# Existing files are never overwritten. Files end up owned by 10001:10001.
ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
# 3. Every asset row's sha256 against the files in the volume.
ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
# Row counts of any database, to compare with the live one.
ps-restore.sh --project packetstation-test --counts-only --db packetstation
```
- The database is created as `01-roles.sh` creates it (owner `ps_migrate`, UTF8, `template0`, CONNECT and TEMPORARY for `ps_app`), then `pg_restore --exit-on-error --single-transaction` as postgres. A failed restore leaves an empty database, never a half one.
- The globals file is applied without stopping on "already exists". Any other error fails. The script refuses a globals file that holds a password hash. On a new cluster the roles come back without passwords: set each from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates them.
- The assets copy runs in a throwaway container from the image the postgres container already runs, with `--network none`, so nothing is downloaded.
- To put a restored database into service: stop that stack's app and worker, rename as postgres (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack, check the app, then drop the old database. This step is deliberate and manual.
### Restore rehearsal 2026-10-21
Done on docker-1 against the test stack (`packetstation-test`, image `sha-ce28d2f`, container `packetstation-test-postgres-1`, postgres:16-bookworm by digest), as spencerm without sudo, from a scratch copy of the scripts. The stack was not stopped, restarted or changed. Times UTC.
| Step | Result |
|---|---|
| Backup by hand into `/home/spencerm/packetstation/test/backups` | 21:24:05 to 21:24:06, 0.76 s. `ps-20261021T212405Z.dump`, 2,234,906 bytes, 743 `pg_restore --list` entries. Globals 774 bytes with no password line (roles postgres, ps_app, ps_migrate, attributes only). `last-success` written. Dump and globals 0600, marker 0644, all spencerm. Exit 0. This dump stays in the folder. |
| Failure path (a database that does not exist, into a scratch folder) | Exit 1, pg_dump's error on stderr, `last-failure` = `2026-10-21T21:24:53Z ps-20261021T212453Z.dump pg_dump failed`, no partial file. |
| Live row counts | `--counts-only --db packetstation`: 64 tables. |
| Restore into `ps_restore_rehearsal` with the globals | Globals: 3 statements skipped (the roles exist), no other error, no password touched. Created and restored in 1.65 s. Every one of the 64 tables has the same row count as the live database, checked just before and just after (for example assets 189, audit_events 2137, bom_lines 5742). |
| Assets out of the live volume | Tarred through a throwaway container (`--network none`, read-only mount): 129,546,240 bytes, 189 files, 3.8 s. sha256 of every file against the 189 `assets` rows of the restored database: 189 match, 0 differ, 0 missing, 0 files without a row. |
| Asset restore path | `--assets <the tar> --assets-volume ps_restore_rehearsal_assets` into a throwaway volume, 0.47 s. `--verify-assets`: 189 rows, 189 match, 0 differ, 0 missing, 0 orphans. A second copy changed nothing (no overwrite). Files owned 10001:10001. |
| Guard | Restoring again into `ps_restore_rehearsal` without `--replace`: refused, exit 1. |
| Cleanup | `DROP DATABASE ps_restore_rehearsal`, the throwaway volume and the scratch folder removed. The cluster lists only packetstation, postgres, template0 and template1. App, worker and postgres stayed healthy throughout. |
| Timer | `systemd-analyze calendar "*-*-* 02:00:00 America/Denver"`: next run 2026-10-22 08:00 UTC. `systemd-analyze verify` on both units found only the script not installed yet. systemd 255. |
Nothing went wrong. Notes from it:
- The dump is small (2.2 MB). The assets (125 MB) are the bulk and are not in the nightly dump. PBS covers them; the restore path is a PBS file-level restore of the volume folder, or a tar, through `--assets`.
- The globals restore re-applies role attributes. On the same cluster that changes nothing. On a new cluster it creates the roles without passwords: set them from the secrets.
- The next rehearsal is IT's, quarterly (requirement O2).