Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/DEPLOY.md

@@ -32,7 +32,7 @@ Both stacks run from the same image as separate compose projects.
 - `postgres`: PostgreSQL 16, named volume, `backend` network only. Its init creates the database roles from secrets.
 - `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
 - `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
-- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup and housekeeping (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only (used from phase 9). Its healthcheck reads the heartbeat file.
+- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup, housekeeping and the phase 9 jobs: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only at `/backups` (the backup-check job reads it). Its healthcheck reads the heartbeat file.
 - `clamav` (optional, profile `clamav`, phase 4): `clamav/clamav:1.4`, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias `packetstation-clamav` (as `postgres` has its own alias). `backend` network plus egress (freshclam downloads the signatures), no `npm_default`, no published port. Volume `clamav-db` at `/var/lib/clamav`, `mem_limit: 2g`, healthcheck `clamdcheck.sh`. Started only with `--profile clamav`. See "Virus scan (ClamAV)".
 - All services restart unless stopped, log to stdout (json-file), and have healthchecks.
 
@@ -66,9 +66,10 @@ Key settings:
 - `VISUAL_SERVER`, `VISUAL_DATABASE`, `VISUAL_USER=visualkit_ro`, `VISUAL_PASSWORD_FILE`.
 - `MFILES_URL=https://mfiles.pescoinc.biz/REST`, `MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}`, `MFILES_USERNAME=mfileskitRO`, `MFILES_CA_BUNDLE`.
 - `LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636`, `LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com`, `LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com`, `LDAP_PASSWORD_FILE`, `LDAP_CA_BUNDLE`.
-- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`.
+- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`, `NOTIFY_DRY_RUN` (`true` on the test stack, `false` on production). `STACK` (`test` or `prod`) picks the subject prefix. See "Notifications".
+- `BACKUP_DIR`: the stack's backup folder on the host, mounted read-only in the worker. See "Backup".
 - `APP_ALIAS` per stack.
-- Identity timing and limits, all optional with these defaults: `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_DESK_MINUTES=60` (Admin and Clerical wait at least this long for both), `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
+- Identity timing and limits, all optional with these defaults (an Admin can override most of them within bounds on the Security tab; see [SECURITY.md](SECURITY.md), "Security settings"): `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_DESK_MINUTES=60` (Admin and Clerical wait at least this long for both), `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
 - Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
 - Uploads and storage, all optional: `UPLOAD_MAX_BYTES` (default 25 MB, drawings, MTRs and documents), `UPLOAD_MAX_BYTES_BOM` (default 5 MB, hand BOMs), `ASSETS_QUOTA_WARN_PCT` (default 85, 1 to 99) and `ASSETS_QUOTA_BYTES` (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
 - Virus scan, all optional: `CLAMAV_HOST` (empty by default, which means no scan; `packetstation-clamav` for the profile service), `CLAMAV_PORT` (default 3310) and `CLAMAV_TIMEOUT_SECONDS` (default 30). The env examples carry `#CLAMAV_HOST=packetstation-clamav` commented out. See "Virus scan (ClamAV)".
@@ -119,7 +120,7 @@ clamd holds the signatures in memory, about 1.2 GB with the full set and more wh
 
 Every stored file and the housekeeping job check the assets volume. With `ASSETS_QUOTA_BYTES=0` (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At `ASSETS_QUOTA_WARN_PCT` percent (85 by default) the app opens one `notification_events` alert of kind `assets_quota` and audits `storage.quota_warning`. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.
 
-The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. Phase 9 sends the alert by email to PS-Admins (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold). Until then, check the Storage tab.
+The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. The alert is mailed through the `assets_quota` notification rule, to PS-Admins by default (decided 2026-10-18, Gitea issue 20, which also kept the 85 percent threshold), with a "Cleared" message when usage drops back.
 
 ## Python packages and the Gitea index
 
@@ -253,7 +254,7 @@ It checks:
 - `visualkit doctor` and `mfileskit doctor`. This is the first real Linux test of both libraries.
 - Database connection, roles, migrations at head, and an audit UPDATE that must fail.
 - LISTEN/NOTIFY round trip.
-- Assets volume and free space. Backup folder (planned, phase 9).
+- Assets volume and free space. The backup folder is readable (SKIP when it is not mounted).
 - LDAP TLS bind on both controllers, reading `memberOf`, `employeeID` and `userAccountControl` as the service account, the 50-group count and one nested-group query.
 - SMTP STARTTLS with certificate verification.
 - ClamAV (phase 4): clamd answers PING at `CLAMAV_HOST`. SKIP when no host is set.
@@ -284,7 +285,7 @@ The real PESCO domain controllers are never used by CI. The test and production
 CI never reads VISUAL or M-Files. `deploy/ci/ci.env` sets `FEED_SOURCE=fixture`, `DRAWING_SOURCE=fixture` and `FIXTURE_DIR=/fixtures`. The image does not contain `tests/`, so the e2e job generates the fixtures on the runner before the stack starts:
 
 ```bash
-python -m pip install pypdf==6.21.0
+python -m pip install pypdf==6.21.1
 python -m tests.fixtures.generator --out tests/fixtures/generated
 ```
 
@@ -310,10 +311,161 @@ Two runner settings were needed and are in deploy/runner: the actions cache serv
 port 8088 (job containers cannot reach the runner's own network, and every cache call timed out), and the
 job workspace on a host directory allowed in `valid_volumes` (so the stack's bind mounts resolve).
 
-## Backup (planned, phase 9)
+## Image rebuild for phase 9
 
-`pg_dump` of both stacks from a systemd timer on the host at 02:00 America/Denver, before the 03:00 PBS run. Seven dumps kept in `/var/backups/packetstation/<stack>/` with last-success and last-failure markers. The worker's backup-check job alerts if the newest dump is missing or older than 26 hours. Scripts and units go in `scripts/`.
+Phase 9 bumps two pins for security fixes found by the CI dependency scan: `pypdf` 6.21.0 to 6.21.1 and `certifi` 2026.7.22 to 2026.10.19 (`server/requirements.txt` and `pyproject.toml`). They reach a stack only with a new image. Build and tag the phase 9 commit as usual (`:sha-<sha>` from CI), set it in the stack's env file, and `up -d`. Migration 0010 runs on the way up. Then run the doctor.
 
-## Restore (planned, phase 9)
+## Notifications (phase 9)
 
-Restore the dump into a fresh database and copy the assets back, from the live folder or a file-level PBS restore. Rehearsed once on the test stack and written up here. A whole-VM restore from PBS is the disaster path.
+The app sends test messages and the worker sends everything else, through PESCO's relay.
+
+| Setting | Test stack | Production | CI |
+|---|---|---|---|
+| `SMTP_HOST` | `app-03.pesco.com` | `app-03.pesco.com` | empty (mail off) |
+| `SMTP_PORT` | 25 | 25 | 25 |
+| `SMTP_STARTTLS` | `true` | `true` | `true` |
+| `SMTP_FROM` | `noreply@pescoinc.biz` | `noreply@pescoinc.biz` | `noreply@pescoinc.biz` |
+| `SMTP_CA_BUNDLE` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` | `/etc/ssl/pesco/ca.pem` |
+| `NOTIFY_DRY_RUN` | `true` | `false` | `true` |
+| `STACK` | `test` | `prod` | `ci` |
+
+- STARTTLS always, verified against the PESCO CA, with the host name so the certificate name matches. No login. A relay that does not offer STARTTLS, or a certificate that does not verify, fails the message. There is no plain-text fallback.
+- Subjects start `[Packet Station]` when `STACK=prod`, otherwise `[Packet Station TEST]`.
+- Each message is addressed To `SMTP_FROM`. The recipients are in the envelope only.
+- `NOTIFY_DRY_RUN=true` records every message as `dry_run` in the sent log and sends nothing. The test stack keeps it on, so the test stack never mails people about test data. Production sets it `false`.
+- With `SMTP_HOST` empty, every message is recorded `failed` with "Mail is off: SMTP_HOST is empty."
+- The test send. On the Admin page, Notifications tab, open a rule, put an address in "Send the test to" and press "Send a test". It sends at once from the app, even in dry run, and shows the result. The sent log keeps the row (marked Test). A test with no address goes to the rule's recipients. On a new stack, send one test to your own address before relying on the rules.
+- Every rule starts on, sent at once, to the AD group PS-Admins. Change the recipients on the Notifications tab. Each person sees on Account settings what reaches them and why.
+- The worker needs the LDAP bind to resolve group recipients. While AD is down, an immediate message waits and is tried twice more, then recorded `failed`.
+
+The doctor's SMTP line checks STARTTLS with certificate verification, without sending.
+
+## Backup (phase 9)
+
+Host scripts in `scripts/backup/`, run by root from a systemd timer. They run outside the app, so a broken app still backs up. [scripts/backup/README.md](../scripts/backup/README.md) has the same steps with more detail.
+
+| File | Installed as |
+|---|---|
+| `ps-backup.sh` | `/usr/local/sbin/ps-backup.sh` (root, 0755) |
+| `ps-restore.sh` | `/usr/local/sbin/ps-restore.sh` (root, 0755) |
+| `backup.conf.example` | `/etc/packetstation/backup.conf` (root, 0644) |
+| `packetstation-backup.service` | `/etc/systemd/system/` (root, 0644) |
+| `packetstation-backup.timer` | `/etc/systemd/system/` (root, 0644) |
+
+`backup.conf` lists the stacks, one per line: name, container (or `project:<compose project>`), database, folder.
+
+```
+test     project:packetstation-test   packetstation  /home/spencerm/packetstation/test/backups
+#prod    project:packetstation        packetstation  /var/backups/packetstation/prod
+```
+
+The production line stays commented out until the production stack runs (phase 10). Until then it would fail every night.
+
+Where the dumps go:
+
+- Test stack: `/home/spencerm/packetstation/test/backups`, owned by spencerm (the script gives new files the folder's owner and group). The test `stack.env` sets `BACKUP_DIR` to this path.
+- Production: `/var/backups/packetstation/prod`, `root:10001` 0750. The production `stack.env` sets `BACKUP_DIR=/var/backups/packetstation/prod`.
+
+The worker mounts the stack's folder read-only at `/backups`. Its `backup-check` job (07:15) only lists the folder, reads the dumps' sizes and times, and reads the markers.
+
+What a run does, per stack:
+
+1. Finds the stack's running postgres container (by compose labels for `project:`). None or two is a failure.
+2. `docker exec <pg> pg_dump -U postgres -d packetstation -Fc` over the local socket, which the postgres image trusts. No password is read or passed. Written under a temporary name, `.ps-<stamp>.partial`.
+3. `pg_dumpall --globals-only --no-role-passwords`: the roles without their password hashes. The role passwords are in the stack secrets.
+4. `pg_restore --list` with the container's own pg_restore must list at least one entry. Then the files are renamed to `ps-<UTC yyyymmddThhmmssZ>.dump` and `.globals.sql`, mode 0600.
+5. Keeps the newest 7 (`KEEP=7` in the unit, `--keep N`), with their globals files.
+6. Writes `last-success` or `last-failure` (`<ISO UTC time> <dump name>`, plus the reason on failure), mode 0644. A failed run leaves no partial file.
+
+Each stack runs on its own, so one failing does not stop the next. Exit 1 if any stack failed, 2 for a usage or config error. `flock` keeps two runs from overlapping.
+
+The unit is `Type=oneshot` as root, `Nice=10`, `IOSchedulingClass=idle`, `TimeoutStartSec=1h`, with `NoNewPrivileges`, `PrivateTmp` and `ProtectSystem=full` (no `ProtectHome`, so it can write the test folder under `/home`). The timer is `OnCalendar=*-*-* 02:00:00 America/Denver` with `Persistent=true`: 02:00 Denver time all year, an hour before the 03:00 PBS run, so PBS captures the fresh dumps. A missed run runs at the next boot.
+
+### Install (Spencer, with sudo, on docker-1)
+
+After the phase 9 tree is synced to `~/packetstation/repo`:
+
+```bash
+cd ~/packetstation/repo
+sudo install -o root -g root -m 0755 scripts/backup/ps-backup.sh  /usr/local/sbin/ps-backup.sh
+sudo install -o root -g root -m 0755 scripts/backup/ps-restore.sh /usr/local/sbin/ps-restore.sh
+sudo install -d -o root -g root -m 0755 /etc/packetstation
+sudo install -o root -g root -m 0644 scripts/backup/backup.conf.example /etc/packetstation/backup.conf
+sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.service /etc/systemd/system/
+sudo install -o root -g root -m 0644 scripts/backup/packetstation-backup.timer   /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemd-analyze verify /etc/systemd/system/packetstation-backup.service
+sudo systemctl enable --now packetstation-backup.timer
+systemctl list-timers packetstation-backup.timer
+sudo systemctl start packetstation-backup.service
+journalctl -u packetstation-backup.service -n 20 --no-pager
+ls -la ~/packetstation/test/backups
+```
+
+`list-timers` shows the next run at 08:00 or 09:00 UTC (02:00 in Denver). The one run by hand should leave a new `ps-*.dump` and a fresh `last-success`. After changing a script in the repo, run its `install` line again: the installed copies are root's, so a sync of the checkout does not change what root runs.
+
+When the production stack exists (phase 10):
+
+```bash
+sudo install -d -o root -g root  -m 0755 /var/backups/packetstation
+sudo install -d -o root -g 10001 -m 0750 /var/backups/packetstation/prod
+sudoedit /etc/packetstation/backup.conf      # uncomment the prod line
+sudo systemctl start packetstation-backup.service
+```
+
+A run by hand needs no sudo for the test stack (spencerm is in the docker group):
+
+```bash
+scripts/backup/ps-backup.sh --stack test project:packetstation-test packetstation /home/spencerm/packetstation/test/backups
+```
+
+The dumps hold the database only. The assets volume (the files) is not in them. It is covered by PBS, since the volume is part of the VM.
+
+## Restore (phase 9)
+
+`ps-restore.sh` never stops, restarts or reconfigures a stack. It restores into a new database next to the live one, and a person decides how to switch. A whole-VM restore from PBS stays the disaster path.
+
+```bash
+B=/home/spencerm/packetstation/test/backups
+D=$(ls $B/ps-*.dump | tail -1)
+
+# 1. Into a fresh database. Refuses an existing one; --replace drops it first (DROP refuses while anyone is connected).
+ps-restore.sh --project packetstation-test --dump "$D" --globals "${D%.dump}.globals.sql" --db ps_restored
+
+# 2. Assets, from a folder or a tar (for example a PBS file-level restore of the assets volume folder).
+#    Existing files are never overwritten. Files end up owned by 10001:10001.
+ps-restore.sh --project packetstation-test --assets /path/to/assets.tar --assets-volume packetstation-test_assets
+
+# 3. Every asset row's sha256 against the files in the volume.
+ps-restore.sh --project packetstation-test --verify-assets --db ps_restored --assets-volume packetstation-test_assets
+
+# Row counts of any database, to compare with the live one.
+ps-restore.sh --project packetstation-test --counts-only --db packetstation
+```
+
+- The database is created as `01-roles.sh` creates it (owner `ps_migrate`, UTF8, `template0`, CONNECT and TEMPORARY for `ps_app`), then `pg_restore --exit-on-error --single-transaction` as postgres. A failed restore leaves an empty database, never a half one.
+- The globals file is applied without stopping on "already exists". Any other error fails. The script refuses a globals file that holds a password hash. On a new cluster the roles come back without passwords: set each from the stack's secret file with `\password` in psql, or start the stack on an empty volume so initdb creates them.
+- The assets copy runs in a throwaway container from the image the postgres container already runs, with `--network none`, so nothing is downloaded.
+- To put a restored database into service: stop that stack's app and worker, rename as postgres (`ALTER DATABASE packetstation RENAME TO packetstation_old; ALTER DATABASE ps_restored RENAME TO packetstation;`), start the stack, check the app, then drop the old database. This step is deliberate and manual.
+
+### Restore rehearsal 2026-10-21
+
+Done on docker-1 against the test stack (`packetstation-test`, image `sha-ce28d2f`, container `packetstation-test-postgres-1`, postgres:16-bookworm by digest), as spencerm without sudo, from a scratch copy of the scripts. The stack was not stopped, restarted or changed. Times UTC.
+
+| Step | Result |
+|---|---|
+| Backup by hand into `/home/spencerm/packetstation/test/backups` | 21:24:05 to 21:24:06, 0.76 s. `ps-20261021T212405Z.dump`, 2,234,906 bytes, 743 `pg_restore --list` entries. Globals 774 bytes with no password line (roles postgres, ps_app, ps_migrate, attributes only). `last-success` written. Dump and globals 0600, marker 0644, all spencerm. Exit 0. This dump stays in the folder. |
+| Failure path (a database that does not exist, into a scratch folder) | Exit 1, pg_dump's error on stderr, `last-failure` = `2026-10-21T21:24:53Z ps-20261021T212453Z.dump pg_dump failed`, no partial file. |
+| Live row counts | `--counts-only --db packetstation`: 64 tables. |
+| Restore into `ps_restore_rehearsal` with the globals | Globals: 3 statements skipped (the roles exist), no other error, no password touched. Created and restored in 1.65 s. Every one of the 64 tables has the same row count as the live database, checked just before and just after (for example assets 189, audit_events 2137, bom_lines 5742). |
+| Assets out of the live volume | Tarred through a throwaway container (`--network none`, read-only mount): 129,546,240 bytes, 189 files, 3.8 s. sha256 of every file against the 189 `assets` rows of the restored database: 189 match, 0 differ, 0 missing, 0 files without a row. |
+| Asset restore path | `--assets <the tar> --assets-volume ps_restore_rehearsal_assets` into a throwaway volume, 0.47 s. `--verify-assets`: 189 rows, 189 match, 0 differ, 0 missing, 0 orphans. A second copy changed nothing (no overwrite). Files owned 10001:10001. |
+| Guard | Restoring again into `ps_restore_rehearsal` without `--replace`: refused, exit 1. |
+| Cleanup | `DROP DATABASE ps_restore_rehearsal`, the throwaway volume and the scratch folder removed. The cluster lists only packetstation, postgres, template0 and template1. App, worker and postgres stayed healthy throughout. |
+| Timer | `systemd-analyze calendar "*-*-* 02:00:00 America/Denver"`: next run 2026-10-22 08:00 UTC. `systemd-analyze verify` on both units found only the script not installed yet. systemd 255. |
+
+Nothing went wrong. Notes from it:
+
+- The dump is small (2.2 MB). The assets (125 MB) are the bulk and are not in the nightly dump. PBS covers them; the restore path is a PBS file-level restore of the volume folder, or a tar, through `--assets`.
+- The globals restore re-applies role attributes. On the same cluster that changes nothing. On a new cluster it creates the roles without passwords: set them from the secrets.
+- The next rehearsal is IT's, quarterly (requirement O2).

docs/FEED.md

@@ -1,6 +1,6 @@
 # Packet Station feed and drawings
 
-Status: phase 3. This file says how the worker reads VISUAL and M-Files and what it does with what it reads. The design is in [PLAN.md](PLAN.md), "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.
+Status: phase 9. This file says how the worker reads VISUAL and M-Files, what it does with what it reads, and the other jobs it runs (phase 9: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications). The design is in [PLAN.md](PLAN.md), "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.
 
 ## Purpose
 
@@ -36,7 +36,7 @@ The listening connection reconnects with backoff when it drops.
 
 ### Pause
 
-When `paused` is on, scheduled fires of the pausable jobs (`visual-feed`, `drawing-lookup`) are skipped. Each skip is recorded as a `job_runs` row with status `skipped` and counts `{"reason": "paused"}`. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping is not paused.
+When `paused` is on, scheduled fires of the pausable jobs (`visual-feed`, `drawing-lookup`) are skipped. Each skip is recorded as a `job_runs` row with status `skipped` and counts `{"reason": "paused"}`. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping and the phase 9 jobs are not paused.
 
 ### Run now and job_requests
 
@@ -82,16 +82,97 @@ At start the worker runs the same stack guard as the app: the test stack refuses
 | `visual-feed` | `interval_shop_minutes` inside shop hours (default 5), `interval_off_minutes` outside (default 30) | yes | The five reads, the cache, the packet rules, auto-release. Chains `drawing-lookup`. |
 | `drawing-lookup` | After every feed run, plus every `drawing_interval_minutes` (default 30) | yes | The drawing queue (see "Drawings"). |
 | `housekeeping` | Daily at 03:30 site time | no | Prunes old rows. The plan keeps the caches 90 days and `job_runs` 180 days. |
+| `ad-reconcile` | Daily at 01:30 site time, plus "Refresh from AD now" | no | Users and roles from the AD groups, inactive users signed out, employee ID flags (phase 9). |
+| `session-refresh` | Every `roles_refresh_minutes` (default 15) | no | Every live session's groups and account state read again (phase 9). |
+| `integrity` | Daily at 04:00 site time | no | Seals, asset hashes, missing and orphan files, the audit hash chain (phase 9). |
+| `backup-check` | Daily at 07:15 site time | no | The newest dump is there, not empty, under 26 hours old, and not followed by a failure (phase 9). |
+| `notify-send` | Every 60 seconds | no | Rule seed, the version check, the watch checks, immediate delivery and the digest queue (phase 9). |
+| `notify-digest` | Every 5 minutes | no | Sends each digest that is due, one message per address (phase 9). |
 
-Planned, not built in phase 3:
+`job_schedule` holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for `visual-feed`, `drawing-lookup` and `housekeeping`; 0010 adds the six phase 9 jobs.
 
-- `session-refresh` (every 15 minutes): re-resolve groups and the account state for every live session.
-- `ad-reconcile` (nightly 01:30 site time, plus "refresh from AD now"), phase 9.
-- `integrity` (04:00): seals, asset hashes, missing and orphan files, the audit hash chain. Phase 9.
-- `backup-check` (07:15): the newest dump is under 26 hours old. Phase 9.
-- `notify-send` (every 60 seconds) and `notify-digest` (daily at the digest time). Phase 9.
+## Phase 9 jobs
 
-`job_schedule` holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for `visual-feed`, `drawing-lookup` and `housekeeping`.
+None of the six phase 9 jobs is pausable. Each is a module in `worker/jobs/` with its own `SPEC`. `load_jobs()` imports them lazily, so a module that fails to import is logged and left out, and the rest run. They use the same run now, advisory lock and `job_runs` rules as the feed. `packetstation job run <name>` runs any of them once.
+
+### ad-reconcile (01:30 site time, plus "Refresh from AD now")
+
+1. Reloads the departments (and with them the group map).
+2. Reads AD, outside any transaction: every group in `LDAP_GROUP_BASE_DN` (name and DN), the members of every mapped group through `LDAP_MATCHING_RULE_IN_CHAIN` in pages of 500 (disabled accounts included, so they can be marked), and, by objectGUID in batches of 50, every active user who was not in any group.
+3. **Refusal guard.** When no mapped group is in the OU (usually a wrong `LDAP_GROUP_BASE_DN`), or when every mapped group is empty while active users exist, the run raises `ReconcileRefused`, fails with that reason in its errors, and changes nothing (PLAN.md open question 44).
+4. One transaction: users upserted on objectGUID and their roles rebuilt; a disabled user marked inactive with their sessions ended `ad_disabled`; a user AD no longer has marked inactive with `ad_gone`; a user in AD with no mapped group left without roles and their sessions ended `roles_lost`. `directory_flags` gets `emp_missing`, `emp_duplicate`, `gone` and `disabled`. Every ended session writes `auth.session_ended` with `by: ad-reconcile`, and the run writes an `ad.refresh` audit row with its counts.
+5. Events, in their own transaction: `group_missing` (a condition per missing group, subject the group name) raised, and resolved for every mapped group present; `ad_reconcile_report` (one-shot, info) when anyone was added or deactivated, an extra group is in the OU, or an employee ID is missing or shared. A notify error is recorded on the run and does not fail it.
+
+AD unreachable fails the run and changes nothing. A failed or refused run raises the `ad_reconcile_report` condition "AD reconcile did not run" (subject `ad-reconcile:failed`), so the rule's recipients get one mail until a run succeeds and clears it. The counts are the report the Admin page shows: `groups_checked`, `groups_missing` and `groups_extra` (names), `users_seen`, `created`, `updated`, `deactivated`, `sessions_ended`, `emp_missing` (accounts) and `emp_duplicate` (employee IDs), lists capped at 200.
+
+"Refresh from AD now" inserts a `job_requests` row for `ad-reconcile` and answers 202 (open question 48). `packetstation ad-refresh` runs the same reconcile in the foreground.
+
+In CI the Samba DC has an extra group and a duplicate employee ID on purpose, so the report is raised every night there.
+
+### session-refresh (every `roles_refresh_minutes`, default 15)
+
+For every user with a live session (not revoked, not expired): AD first, by objectGUID in batches, then each enabled user's groups. Then one transaction: a disabled or gone user is marked inactive and their sessions ended (`ad_disabled`, `ad_gone`); a user left with no role has their roles cleared and sessions ended (`roles_lost`); everyone else gets their user row, `user_roles`, each live session's `roles_snapshot` and `roles_checked_at` updated. Any AD failure makes the run `degraded` with `counts.ad = "unavailable"`, and nothing is written or ended. The interval comes from the Security tab setting, read at each run, so a changed value applies after the next run. Counts: `users`, `refreshed`, `roles_changed`, `deactivated`, `roles_lost`, `sessions_ended`.
+
+### integrity (04:00 site time)
+
+Each step in its own transaction:
+
+- Seals: every unbroken seal of a filed packet is rebuilt and compared with the seal key. A seal made under another `key_id` is a finding. No seal key makes the run degraded and the seals unchecked.
+- Assets: every asset row checked for its file (`asset_missing`). The sha256 is recomputed for assets uploaded in the last 7 days every night, and for every asset on Sundays (site time); `hash_checked_at` and `hash_ok` are set (`asset_hash`).
+- Orphans: files under `ASSETS_DIR` that no asset names, older than one hour (uploads in flight are skipped), `lost+found` skipped (`blob_orphan`). Reported, never deleted.
+- Audit: `audit.verify_chain` over the hash chain (`audit_chain`).
+
+One `integrity_findings` row per finding, at most 500 per kind (the count in the run's counts). Any finding raises `integrity_failure` (condition, critical, subject `integrity`). A clean run that was not degraded resolves it. Counts: `full`, `seals`, `assets`, `hashed`, `files`, `audit_rows`, `findings` per kind, `total`.
+
+### backup-check (07:15 site time)
+
+Reads the stack's backup folder, mounted read-only at `/backups` (`BACKUP_DIR` on the host). Problems: the folder is not there, no `*.dump` at the top of it, the newest dump is empty or older than 26 hours (by modification time), or `last-failure` is newer than `last-success` (or there is a failure marker and no success marker). Any problem raises `backup_stale` (condition, subject the stack name); none resolves it. The run itself stays `ok`; the alert carries the news. Counts: `folder`, `dumps`, `newest`, `newest_at`, `age_hours`, `size`, `last_success`, `last_failure`, `problems`. The CI stack's backup folder is empty, so CI raises `backup_stale` in dry run.
+
+### notify-send (every 60 seconds)
+
+1. Seeds any missing rule (on, at once, digest 07:00, to PS-Admins). Never changes a rule that exists.
+2. On its first run in each worker process: when `PS_BUILD` differs from the settings row `notify.last_build`, updates the row and raises `version_deployed` (subject `build:<build>`). Not at worker start, since the scheduler may start before PostgreSQL answers (open question 47). A restart on the same build raises nothing.
+3. At most every 15 minutes per process, the watch checks: `drawing_failing` when drawing lookups have been failing for more than the rule's hours (default 24; subject `drawings`, the list of up to 50 in the body), resolved when none is; `co_waiting` per change order pending review more than the rule's days (default 3) on a live packet (subject `co:<id>`, the packet's department), resolved when it is no longer waiting.
+4. Delivery: up to 200 undispatched events, oldest first, each in its own transaction with `FOR UPDATE SKIP LOCKED`. No rule: `no_rule`. Rule off: `rule_off`. Digest rule: an outbox row, `queued`. Immediate: recipients resolved now, one message, one `notifications_sent` row, `sent`, `dry_run`, `failed` or `no_recipients`. A failed SMTP send is recorded and not retried. AD unavailable while resolving a group: a `failed` row "directory unavailable", the event stays pending (`directory_retry:N`), and the third failure dispatches it `failed`.
+
+A failed or retried send makes the run degraded.
+
+### notify-digest (every 5 minutes)
+
+For each rule whose digest time (site time) has passed today and that has outbox rows queued before it: recipients resolved per event, then grouped per address across every rule. One message per address lists every due event (`Daily digest: N notifications`), with one `notifications_sent` row per message (event type `digest`). An outbox row is marked sent only when every address got it. A failed address leaves the rows for the next run, which skips addresses that already got an event, and the job backs off 30 minutes (open question 43). Nothing queued: the run ends without touching AD.
+
+### Events from the feed and the drawing job
+
+After each `visual-feed` and `drawing-lookup` run, `worker/runs.py` raises or resolves the feed conditions:
+
+| Run status | `feed_failed` (subject the job name, critical, the first ten errors in the body) | `feed_degraded` (visual-feed only) |
+|---|---|---|
+| failed | raised | unchanged |
+| degraded | resolved | raised |
+| ok | resolved | resolved |
+| skipped | unchanged | unchanged |
+
+`drawing_failing` comes from the notify-send watch check above, not from the run. Each event goes to its rule's recipients, PS-Admins by default. A broken run connection falls back to a fresh one and never fails the run.
+
+### What each event means
+
+| Event type | Raised by | Kind | Cleared by |
+|---|---|---|---|
+| `security_alert` | Account cooldown, PIN disabled, session ended after wrong PINs | one-shot, 15 min per subject | none |
+| `signin_burst` | An address over its limit, or the burst threshold across accounts | one-shot, 30 min per subject | none |
+| `integrity_failure` | integrity | condition | a clean integrity run |
+| `feed_failed` | visual-feed or drawing-lookup failed | condition | an ok or degraded run |
+| `feed_degraded` | visual-feed degraded | condition | an ok run |
+| `drawing_failing` | notify-send watch | condition | no lookup failing that long |
+| `ad_reconcile_report` | ad-reconcile | one-shot | none |
+| `group_missing` | ad-reconcile, per group | condition | the group found |
+| `backup_stale` | backup-check | condition | a good backup |
+| `co_waiting` | notify-send watch, per change order | condition, department | decided or no longer waiting |
+| `qc_reject` | A Q.C. reject on a packet, an ITP line reject | one-shot, department | none |
+| `version_deployed` | notify-send, first run on a new build | one-shot | none |
+| `assets_quota` | The storage check after every upload and in housekeeping | condition | usage back under the threshold |
+
+A condition sends once when it opens and a "Cleared: <title>" message to the same rule when it closes. Raised again while open, it only counts (`raised_count`). The department of an ITP reject is the one department among the unit's live packets that works to the line's page; none or several gives none.
 
 ## Feed reads
 

docs/PLAN.md

@@ -42,7 +42,7 @@ frontend/ (Vite + React + TS; src/theme, src/screens/{clerical,warehouse,supervi
 tests/server/ (pytest: policy matrix, merge rules, packet rules, feed fixtures, auth)
 tests/e2e/ (Playwright scenarios, synthetic fixtures)  tests/fixtures/generator.py
 deploy/ compose.prod.yml compose.test.yml compose.ci.yml  npm/ (proxy host notes)  samba-ad/ (CI DC seed)
-scripts/ backup.sh restore.sh packetstation-backup.service/.timer
+scripts/backup/ ps-backup.sh ps-restore.sh packetstation-backup.service packetstation-backup.timer backup.conf.example
 .gitea/workflows/ci.yml
 ```
 
@@ -115,6 +115,7 @@ Each phase ends with passing tests, docs updated (README, CHANGELOG, SECURITY.md
 - Notifications tab (rules per event type, recipients: group, user, mailbox, dynamic department Supervisor group; immediate or digest; exclusions; test send; sent log; dry run on test), SMTP via app-03 with STARTTLS and CA verify, subject prefixes, seed to PS-Admins.
 - scripts/backup.sh + systemd timer (America/Denver 02:00), 7 dumps, marker; worker backup-check job; restore script; restore rehearsed on the test stack and written up.
 - Closes: I6 (app part), M3, O1, O2 (first rehearsal), S3 (pip-audit and npm audit in CI).
+- Done 2026-10-21. The eight Admin tabs, the Account notifications section, the six worker jobs, the notify package with SMTP over STARTTLS, migration 0010, the backup and restore scripts with the systemd units, and the CI dependency scan are built and tested. The backup and the restore were rehearsed by hand on the test stack on docker-1 (DEPLOY.md, "Restore rehearsal 2026-10-21"). The first scan found two pins to bump (pypdf 6.21.1, certifi 2026.10.19). The separate `notification_state` table was not needed: the open `notification_events` row is the state. The timer is installed by Spencer with sudo (DEPLOY.md, "Backup"). Decisions taken are open questions 43 to 50.
 
 ### Phase 10: Regression bench, hardening, go-live (medium)
 - Rewrite the 38 scenarios against the new UI with synthetic fixtures; run in Gitea Actions against compose.ci.yml. Policy matrix, merge rules and packet-rule tests already run from phase 2 and 3.
@@ -169,7 +170,7 @@ Conventions: identity bigints except users (uuid = AD objectGUID); Postgres enum
 - **Change orders:** change_orders (co_no, origin visual|drawing|manual, review pending|approved|rejected, minor, hold, held_by, issued_*, approved_*, rejected_*, vis jsonb, waiting jsonb, staged jsonb, target_fp; one pending visual CO per packet), co_items, co_required_depts, co_verifications (insert-only, signature required). View v_open_cos ports openCOs.
 - **Documents:** assets (uuid, sha256, size, mime, original_name, storage_key, source, source_ref jsonb, uploaded_by, packet_id); packet_docs; qc_docs (project, unit, type, rev, auto, superseded_by); itps (unit unique, template jsonb frozen, status, signoff_*, reopen_*), itp_lines, itp_rejects (append-only, ncr), itp_blocks.
 - **Audit, settings, jobs:** audit_events (append-only, prev_hash and row_hash chain, user, session, role_snapshot, action, subject, packet_id, detail jsonb, it_reason, ip); settings (key, value jsonb, version); job_requests, job_runs (trigger, requested_by, status ok|degraded|failed|skipped|abandoned, counts jsonb, errors jsonb, build_version, visual_target), job_schedule, worker_heartbeat.
-- **Notifications (per the brief):** notification_rules (event_type, enabled, delivery immediate|digest, digest_time); notification_recipients (rule_id, kind group|user|mailbox|dept_supervisor, group_dn, user_id, address); notification_exclusions (rule_id, user_id); notification_outbox; notifications_sent (event, recipients as resolved, subject, result); notification_state (dedupe and recovery).
+- **Notifications (per the brief):** notification_rules (event_type, enabled, delivery immediate|digest, digest_time); notification_recipients (rule_id, kind group|user|mailbox|dept_supervisor, group_dn, user_id, address); notification_exclusions (rule_id, user_id); notification_outbox; notifications_sent (event, recipients as resolved, subject, result). The planned notification_state (dedupe and recovery) was not needed: the open notification_events row under its unique index is the dedupe, and a recovery row points back at it (phase 9).
 - **VISUAL and drawing caches:** visual_batches; visual_rows (typed key wo, lot, split, sub, seq int; all contract fields; row_hash; first, last and gone batch); visual_materials (plus piece int, mat_hash); visual_orders and visual_order_links (base or project, rank, lines, top_jobs, order_date); visual_labor; visual_row_changes and visual_material_changes (per batch, added|changed|gone); visual_ignored_wos (go-live list with pulled_in_*); visual_resource_map; drawing_lookups (dwg PK, checked_at, ok, rev, name, object and file ids and versions, base, draft_rev, draft_where, in_drafting, note, failing_since, asset_id, visual_rev_seen).
 - **JSONB only here:** foreign-shaped feed payloads (source, visual, partspecs, vis, waiting, staged, raw), template-driven answers (checklist, checks, itp template, settings), and frozen snapshots (roles_snapshot, qc_history, audit detail, job counts). Everything the policy or the boards filter on is a real column.
 - **Versioning:** packets.version goes up in every transaction that touches the packet or a child; it is the ETag and the NOTIFY payload. Child rows carry row_version. Every write takes `SELECT ... FOR UPDATE` on the packet first, which replaces the old 8-second lease.
@@ -204,7 +205,7 @@ Conventions: identity bigints except users (uuid = AD objectGUID); Postgres enum
 ## Deployment and CI (phase 0, 9, 10; docs/DEPLOY.md)
 
 - **Dockerfile:** frontend stage on node:22-bookworm-slim; builder on python:3.12-slim-bookworm with a venv, `pip install --require-hashes --index-url <gitea>/simple/ -r server/requirements.lock` then the package; runtime with the Microsoft Debian 12 repo (msodbcsql18, unixodbc, tini, tzdata), non-root uid 10001, `PS_BUILD` label. entrypoint exports MFILES_PASSWORD from the secret file and dispatches app (uvicorn with no proxy headers; the app's TrustedProxyMiddleware re-resolves npm_app_1 every 60 s), worker, migrate, doctor, job run, fixtures.
-- **Compose:** base file with postgres (16-bookworm by digest, `backend` internal network only, initdb creates ps_migrate, ps_app and ps_worker from secrets), one-shot migrate, app and worker (backend plus egress, assets volume, CA bundle bind, secrets, healthchecks, restart unless-stopped, json-file logs, 1 GB limits; worker also mounts the backup folder read-only). Proxy overlay joins app to npm_default with `APP_ALIAS`. CI overlay adds the Samba DC, fixture sources and dry-run mail. Env files per stack in /etc/packetstation; secrets root:10001 0440. Deploy: `docker compose -p packetstation --env-file ... -f compose.yaml -f compose.proxy.yaml up -d`.
+- **Compose:** base file with postgres (16-bookworm by digest, `backend` internal network only, initdb creates ps_migrate and ps_app from secrets; the app and the worker both log in as ps_app, the worker with application name `ps-worker`), one-shot migrate, app and worker (backend plus egress, assets volume, CA bundle bind, secrets, healthchecks, restart unless-stopped, json-file logs, 1 GB limits; worker also mounts the backup folder read-only). Proxy overlay joins app to npm_default with `APP_ALIAS`. CI overlay adds the Samba DC, fixture sources and dry-run mail. Env files per stack in /etc/packetstation; secrets root:10001 0440. Deploy: `docker compose -p packetstation --env-file ... -f compose.yaml -f compose.proxy.yaml up -d`.
 - **NPM proxy host** (recorded in deploy/npm-proxy-host.md): http to the alias on 8000, Force SSL, HTTP/2, NPM HSTS off (the app sends it), `client_max_body_size 30m`, a `location /api/events` block with buffering off and 3600 s timeouts using the `$server` variables so the alias is re-resolved.
 - **Doctor:** config and stack guard, `visualkit doctor`, `mfileskit doctor`, DB connect and migrations at head and an audit UPDATE that must fail, LISTEN/NOTIFY round trip, assets volume and free space, backup folder, LDAP TLS bind reading memberOf, employeeID and userAccountControl with the 50-group count and one IN_CHAIN query on both controllers, SMTP STARTTLS verify, tzdata.
 - **Backups:** scripts/backup/ps-backup.sh per stack via `docker exec pg_dump -Fc` plus globals, verified with `pg_restore --list`, 7 kept, last-success and last-failure markers in /var/backups/packetstation/<stack>/; systemd timer `OnCalendar=*-*-* 02:00:00 America/Denver`; restore script and rehearsal recorded in DEPLOY.md.
@@ -276,13 +277,26 @@ From phase 8 (2026-10-18), for Seth to veto. The phase 8 design numbered these 3
 41. The find bar filters are applied by the server on every board and list, so a board never fetches more than it shows. Default: the KPI numbers follow the project pick only, not the other filters. Phase 8 (2026-10-18) applied the default.
 42. Mine is decided from the person's AD roles and the screen they are on. Default: the stall's Mine needs the picked station and shows nothing without one. Phase 8 (2026-10-18) applied the default.
 
+From phase 9 (2026-10-21), for Seth or Spencer to veto:
+
+43. The daily digest: default one message per address per digest time, listing every due event of every digest rule that reaches that address, not one message per rule. An address that already got an event is not sent it again after a partial failure. Phase 9 (2026-10-21) applied the default.
+44. The AD reconcile guard: default the reconcile refuses and changes nothing when no mapped group is in the OU (a wrong `LDAP_GROUP_BASE_DN`) or when every mapped group is empty while active users exist. The run is recorded as failed with the reason in its errors, and an `ad_reconcile_report` condition "AD reconcile did not run" goes to the rule once, cleared by the next good run. A wholesale sign-out needs a person to look first. Phase 9 (2026-10-21) applied the default.
+45. `profile_active`: default an inactive profile only hides the person from the welder picker and the worker filter. It does not change their roles or rights, and an assign made through the API still checks only the role. AD decides who may do what. Phase 9 (2026-10-21) applied the default.
+46. Account opt-out: default any person may opt out of an event type on Account settings, in a password session, except the security event types (security alert, sign-in burst, integrity failure). An Admin cannot leave anyone out of those either. Phase 9 (2026-10-21) applied the default.
+47. Version deployed: default the event is raised by the first `notify-send` run in each worker process when `PS_BUILD` differs from the build last announced (settings row `notify.last_build`), not at worker start, since the scheduler may start before PostgreSQL answers. A restart on the same build sends nothing. Phase 9 (2026-10-21) applied the default.
+48. "Refresh from AD now": default the button asks the worker for an `ad-reconcile` run and answers 202 at once, instead of running the reconcile inside the web request. The Admin sees "Refresh requested" and the report when the run finishes. Phase 9 (2026-10-21) applied the default.
+49. New departments: default only shop departments can be added in the app (422 `support_fixed` for a support department). The support roles are fixed in the `role_kind` enum and the group map, so a new support department would map to no role. Phase 9 (2026-10-21) applied the default.
+50. The profile CSV `stations` cell: default the cell is the person's whole list of station limits across departments and replaces every limit they had; `-` clears them all; an empty cell leaves them. Phase 9 (2026-10-21) applied the default.
+51. A direct welder assign: default it does not check `profile_active`, so an Admin's inactive switch hides a person from the picker but a typed assign still works. Phase 9 (2026-10-21) applied the default; say if an inactive profile should be refused there too.
+52. A checklist item added while a stage is open: default it applies to that open stage too, because completeness reads the live template, so the item must be ticked before submit. Phase 9 (2026-10-21) applied the default.
+
 ## Verification
 
 - Every phase: `pytest` server suite, `npm test` and `npm run build` for the frontend, `docker compose -f deploy/compose.ci.yml up` with the Samba DC and the Playwright scenarios for that phase, `packetstation doctor` inside the image.
 - Phase 3: a recorded DEV feed run compared with the packets the old build made for the same window (counts of drafts, units, projects; same keys), run from the developer machine against VISUAL DEV only.
 - Phase 7: server tests for close and file against the policy oracle, the open work order confirm, the seal with its data book, reopen and refile, a failed data book build, the data book sections and index, the traveler, the ITP PDF, every print kind with its stamp and `print_log` row, the CSV formula guard and the stamp signature; frontend tests for the close and file panel, the print menu, the prints list and the ITP buttons; the Playwright close scenario (close, PDFs and CSV, prints, file with the seal, reopen and refile, the signed ITP PDF) in the CI stack on docker-1.
 - Phase 8: the data book of a synthetic filed packet compared page by page with a data book exported from the old build on the same fixture (moved from phase 7). Done in phase 8 (2026-10-18); the page table, the fixes and what is left are in the phase 8 comparison note.
-- Phase 9: restore rehearsal on the test stack from a pg_dump and an assets copy.
+- Phase 9: restore rehearsal on the test stack from a pg_dump and an assets copy. Done 2026-10-21: every table's row count and all 189 asset hashes matched (DEPLOY.md, "Restore rehearsal 2026-10-21").
 - Phase 10: the full 38-scenario bench green in Gitea Actions.
 
 ## Host layout on docker-1

docs/POLICY.md

@@ -823,7 +823,7 @@ The packet page shows who else has the packet open: `Also in this packet now: <N
 - A locked session drops out at once. Lock and unlock publish a `session` event (`locked`, `unlocked`) from `POST /api/auth/lock`, `POST /api/auth/unlock` and the server's idle lock. Unlock brings the person back.
 - A revoked session drops out at once, and its stream ends.
 - Closing the tab or losing the network drops the person out when the stream ends, within the stream's ping cycle (25 seconds). No other idle rule is needed.
-- The server also keeps who is on each screen for the Admin view of phase 9. No endpoint reads it yet.
+- The server also keeps who is on each screen. Phase 9's Admin "Who is in" tab reads it (`GET /api/admin/presence`).
 
 ### Tightenings against the old build
 
@@ -836,6 +836,116 @@ The packet page shows who else has the packet open: `Also in this packet now: <N
 | Each browser published its own presence, with the name, employee ID, role and stall, refreshed every 20 seconds. | The server builds presence from the session: display name and role, no employee ID. A locked or revoked session drops out at once. | Presence shows only what it needs, and a locked screen is not someone in the packet. | Yes |
 | Over target was worked out in the browser from the hours it had. | Over target is timer hours over the work-order target, on the server. Ticket hours are shown beside it and never decide it. | The station timer is the clock of record (open question 1). | Yes |
 
+## Phase 9: Admin, notifications, reconcile
+
+Nothing in the packet rule table changes in phase 9. The new endpoints are Admin endpoints and one Account endpoint. None of them writes a packet.
+
+### Who may use the Admin endpoints
+
+Every endpoint under `/api/admin` needs an Admin (PS-Admins) in a password session. An Admin in a PIN session is 403 `password_session_required`. Every other role is 403. An unsigned request is 401. Every write also passes the Origin check and the body limit. Tests try every endpoint as every role.
+
+| Endpoint | What | Audit |
+|---|---|---|
+| `GET /departments` | Every department, its AD groups (and whether each is in the OU; unknown when AD is down) and its stations | none |
+| `PATCH /departments/{code}` | Name, active, stations (rename, reorder, add, hide) | `admin.department_updated` |
+| `POST /departments` | A new shop department with its stations; answers the three group names IT must create | `admin.department_added` |
+| `GET /users`, `GET /users/{id}` | Gain the welder stamp, `profile_active`, station limits and the directory flags | none |
+| `PATCH /users/{id}/profile` | Welder stamp, `profile_active`, station limits per department | `admin.profile_updated` |
+| `POST /users/import` | The profile CSV, as a preview (`apply: false`) or applied | `admin.profile_updated` per changed person, `admin.profile_import` |
+| `GET /checklists`, `PUT /checklists` | The weld and plain station checklists | `admin.checklists_updated` |
+| `GET /security/settings`, `PUT /security/settings` | The session, lock and sign-in limit settings | `admin.security_settings_updated` |
+| `GET /security/alerts?hours=` | Security alerts for 1 to 168 hours | none |
+| `GET /security/integrity` | The newest finished integrity run and its findings | none |
+| `GET /access-log` | Sign-in attempts and identity audit rows for everyone | none |
+| `GET /access-log.csv` | The same, as CSV, at most 50,000 rows | `admin.access_log_exported`, a `print_log` row of kind `access-log-csv` |
+| `GET /presence` | Live sessions, locked or not, with the screen and packet | none |
+| `POST /stamp-verify`, `POST /stamp-verify/pdf` | Checks a stamp line, a 16-character code or a PDF (25 MB) against the print log | `admin.stamp_verified` |
+| `GET /directory-flags` | Active users with no employee ID, and employee IDs on more than one account | none |
+| `POST /ad/refresh` | Asks the worker for an `ad-reconcile` run; 202 (open question 48) | `ad.refresh_requested` |
+| `GET /ad/reconcile` | The newest finished reconcile run and its report | none |
+| `GET /notifications` | Mail settings, the subject prefix, every rule with its recipients, exclusions and last message | none |
+| `PUT /notifications/{event_type}` | One rule: on or off, delivery, digest time, its hours or days, recipients, exclusions | `admin.notification_rule_updated` |
+| `GET /notifications/groups?q=` | AD group search (2 to 100 characters; 503 when AD is down) | none |
+| `POST /notifications/{event_type}/test` | A test message, to the rule's recipients or to one address | `admin.notification_test` |
+| `GET /notifications/sent` | The sent log | none |
+
+Every Admin change is audited, with the state before and after where there is one.
+
+### Compare and set
+
+The checklists, the security settings and each notification rule carry a version. A save sends the version it read. When someone else saved first the answer is 409 `version_conflict` with the current value, and nothing is written. The screen reloads and says so.
+
+### Departments
+
+- The code and the short name (the slug) never change. They are in the AD group names. A trigger refuses a change, and a PATCH body that names `code` or `slug` is 422.
+- Stations are never deleted. Station limits, packets and history point at them. A station is hidden by setting it inactive. A station listed without an id is added. A station missing from the list is left as it is. Two stations of one department with the same name are refused (422 `duplicate_station`).
+- A new department needs a code of 2 to 4 capital letters or digits and a short name of 2 to 31 letters or digits starting with a letter, both unused (409 `code_taken`, `slug_taken`). Only shop departments can be added; a support department is 422 `support_fixed`, because the support roles are fixed (open question 49). The answer names the three groups IT must create. Until they exist the department has no people.
+- After a change the app rebuilds its group map from the departments table. The worker reloads it at the start of every reconcile and role refresh run.
+
+### Profiles
+
+- The welder stamp is 1 to 12 letters, digits or dashes, unique among users ignoring case (409 `stamp_taken`). Empty clears it. The packet's own welder stamp field still allows 10; the profile stamp is copied onto the packet as it is.
+- `profile_active` off hides the person from the welder picker and the worker filter. It does not change their roles or what they may do, and an assign sent to the API still checks only the role (open question 45). AD decides who may do what.
+- Station limits are set per department. A station must belong to that department. An empty list clears the department's limits.
+- The CSV import matches employee IDs as text among active users. `0042` is not `42`. Columns: `employee_id` (required), `welder_stamp`, `active`, `stations` (`<dept code>:<station name>` joined by `;`, the person's whole list; open question 50). An empty cell leaves the value. `-` clears it. Up to 2,000 rows and 1 MB. Each row lists its changes and its errors: unknown employee ID, an ID on more than one user, an ID twice in the file, a bad or taken stamp, a bad active value, an unknown department or station. Apply with any error is 422 with the same rows and changes nothing. Otherwise every row is applied in one transaction.
+
+### Checklists
+
+Both lists need at least one item and at most 40. A key is lower case letters, digits and underscores, starting with a letter, at most 30 characters, unique in its list. Group and text are not empty, at most 200 characters. The only placeholders are `{drawing}`, `{rev}` and `{wps}`. Answers on open stages are kept by key. A removed item drops out of new checklists only. An item added while a stage is open must also be ticked on that stage before submit, since the completeness check reads the live list.
+
+### Security settings
+
+Stored as the `security` settings row over the environment defaults. A value out of its bounds is refused (422 `bad_settings`), and a stored value out of bounds is ignored. The keys and bounds are in [SECURITY.md](SECURITY.md), "Security settings (phase 9)". A save applies in the app within 30 seconds. The PIN and step-up failure limits (5 and 3) stay environment settings.
+
+### Access log, presence and the stamp check
+
+- The access log shows every person's sign-in attempts (successful and failed) and their `auth.*`, `pin.*`, `device.*` and `sig.*` audit rows, as the Account access log shows a person their own. The address filter takes an address or a range. Request bodies and secrets are never in it.
+- Who is in shows live sessions (not revoked, not expired) seen within the idle minutes that apply to each. The screen and packet come from the event stream, matched by person.
+- The stamp check never trusts the text. It finds the 16-character signature, looks it up in `print_log`, recomputes it with the seal key from the logged row and compares the text with the logged stamp line. It answers genuine, does not verify (the signature or the text differs from the log), or not found.
+
+### Notifications
+
+- An Admin edits every rule. Recipients are any AD group under the base DN, a user, a mailbox, or (for the department events, `co_waiting` and `qc_reject`) the Supervisor group of the event's department. At most 50 recipients and 200 exclusions per rule.
+- An Admin may leave people out of a rule (an exclusion), except on the security event types: `security_alert`, `signin_burst` and `integrity_failure`. An exclusion on those is 422. Nobody can be left out of them (open question 46).
+- A test send goes at once, from the app, even on a stack in dry run. With an address it goes only there.
+- Recipients are resolved when a message goes, not when the rule is saved: group members (nested, enabled accounts with mail), a user's address while the user is active, a mailbox as typed. An exclusion drops the person by user and by their address.
+
+### Account notifications
+
+| Endpoint | Who | What |
+|---|---|---|
+| `GET /api/account/notifications` | Any signed-in, unlocked session, PIN included | Each enabled event type: whether it reaches the person and why (`user`, `mailbox`, `group <name>`, `Supervisor group of <code>`, `left out by an Admin`, `directory unavailable`), and whether they opted out or may |
+| `PUT /api/account/notifications/{event_type}` | The person, in a password session | `{opt_out}`. Writes or removes their own exclusion. 403 for a security event type. Audit `account.notification_opt_out` when something changed |
+
+A person's own opt-out is kept when an Admin saves the rule. An Admin's exclusion is not removed by the person.
+
+### Security events
+
+The three security event types (`security_alert`, `signin_burst`, `integrity_failure`) have no opt-out and no exclusion. An Admin can still turn the rule off or change its recipients, on the record.
+
+- `security_alert`: an account cooldown started, a PIN disabled after five wrong attempts, or a session ended after three wrong PINs or passwords at unlock or signature. One per account or person per 15 minutes.
+- `signin_burst`: one address over its failure limit, or failed sign-ins across accounts in 5 minutes at or over `signin_burst_threshold` (default 20). One per address, or one for the burst, per 30 minutes.
+- `integrity_failure`: the integrity job found a problem. Cleared by a clean run.
+
+### Audit rows added in phase 9
+
+- `policy.denied`: a 403 after a person was known. Method, path and code only, never a body or query string. One per person, method, path and code per minute.
+- `auth.lockout`: an account went into its cooldown. `auth.signin_burst`: an address over its limit, or a burst across accounts.
+- `auth.session_ended` with `{reason, by}` for every session the reconcile or the role refresh ends (`ad_disabled`, `ad_gone`, `roles_lost`).
+- The `admin.*` rows in the table above, `ad.refresh_requested` and `account.notification_opt_out`.
+
+### Tightenings against the old build
+
+| What the old build allowed | What the rebuild does | Why | Can be vetoed |
+|---|---|---|---|
+| The Admin page added a department with any typed code, shop or support, and changed a department's stations as a comma list, so a station could disappear. | Code and short name are checked and never change. Only shop departments are added. Stations are never deleted, only hidden. The three AD group names are shown before the save. | The code and short name are in the AD group names, and limits and packets point at stations. | Yes |
+| A welder stamp was set in the Admin user sheet, checked in the browser, 1 to 10 characters, not unique. | The stamp is checked on the server, 1 to 12 characters, unique among users. | Two welders with one stamp break the weld record. | Yes |
+| Users, levels and departments were edited in the Admin page. | Roles come only from AD groups. The Admin edits only the profile (stamp, profile active, station limits). Turning a profile off hides the person from pickers and changes no right. | AD is the one place access is granted. | No |
+| The security alerts panel and the integrity check ran in IT's browser when Admin was opened. | Both run on the server: alerts from the audit and the sign-in attempts, the integrity check every night in the worker, with an email alert. | A check that runs only when someone looks finds problems late. | No |
+| Nothing was emailed. | Events go by email through app-03 to rules an Admin sets. Security events cannot be turned off by the person or excluded by an Admin. | A security alert that can be muted is not one. | Yes |
+| The access log export ran in the browser. | The CSV is built on the server with the formula guard and logged in the print log and the audit. | The browser alone could skip the log. | No |
+| A stamped copy could be checked only by eye. | The stamp check recomputes the signature with the seal key and compares the text with the print log. | A copy is genuine only if the server says so. | No |
+
 ## Worker writes (phase 3)
 
 The worker has no session. It writes through the same policy as a person, as the system principal `Principal(user_id=None, system="visual-feed")`. Each write locks the packet row (`FOR UPDATE SKIP LOCKED`; a row locked by someone else is left for the next run), builds the packet state, runs `policy.check()`, makes its change, bumps `version`, and writes a `packet_log` row (by nobody, by name "VISUAL feed", role `system`), an audit row (`packet.<action>`, role `system`) and a change event. See [FEED.md](FEED.md) for when each write happens.

docs/SCHEMA.md

@@ -1,6 +1,6 @@
 # Packet Station schema
 
-Status: phases 1 to 4 built (identity, organisation, projects, packets, change orders, documents, jobs, VISUAL and drawing caches, the warehouse columns in use, the `notification_events` placeholder). Phase 5 adds the shop columns, the checklist templates and the clock rules (migration 0006). Phase 6 adds the Q.C. columns, the Q.C. check items, the Q.C. folder metadata, the ITP header, line and block details and the ITP log (migration 0007). Phase 7 adds the print log and the data book on the seal (migration 0008). Phase 8 adds no table and no migration: the PM report exports use new `print_log` kinds and presence uses two new session events. Phase 9 adds the notification rules and delivery. The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
+Status: phases 1 to 4 built (identity, organisation, projects, packets, change orders, documents, jobs, VISUAL and drawing caches, the warehouse columns in use, the `notification_events` placeholder). Phase 5 adds the shop columns, the checklist templates and the clock rules (migration 0006). Phase 6 adds the Q.C. columns, the Q.C. check items, the Q.C. folder metadata, the ITP header, line and block details and the ITP log (migration 0007). Phase 7 adds the print log and the data book on the seal (migration 0008). Phase 8 adds no table and no migration: the PM report exports use new `print_log` kinds and presence uses two new session events. Phase 9 adds the notification rules and delivery, the integrity findings and the Admin profile switch (migration 0010). The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
 
 ## Conventions
 
@@ -339,7 +339,7 @@ A unit folder shows its own current rows and inherits the project's current rows
 
 These tables have the `ps_append_only` row and TRUNCATE triggers, and `ps_app` holds only SELECT and INSERT on them:
 
-`audit_events`, `signin_attempts`, `signatures`, `project_renumbers`, `short_receipts`, `packet_transitions`, `qc_history`, `packet_log`, `itp_rejects`, `itp_log` (0007), `print_log` (0008), `co_verifications`.
+`audit_events`, `signin_attempts`, `signatures`, `project_renumbers`, `short_receipts`, `packet_transitions`, `qc_history`, `packet_log`, `itp_rejects`, `itp_log` (0007), `print_log` (0008), `notifications_sent` and `integrity_findings` (0010), `co_verifications`.
 
 The app role is refused by privilege first. The migrate role owns the tables, so the trigger refuses it. The views `v_units` and `v_open_cos` are SELECT only for `ps_app`.
 
@@ -352,7 +352,7 @@ Phase 3, migration 0004:
 
 - `job_requests`: one row per "Run now". `job`, `requested_at`, `requested_by`, `note`, `claimed_run_id` and `claimed_at` (set by the run that took it). Index on `job` where `claimed_at` is null. The app inserts the row and NOTIFYs `ps_job_request` in the same transaction.
 - `job_runs`: one row per run. `job`, `trigger` (schedule, request, chain, cli), `requested_by` uuid[] (the requests claimed), `started_at`, `finished_at`, `status` (running, ok, degraded, failed, skipped, abandoned), `counts` jsonb, `errors` jsonb (a list of redacted strings), `build_version`, `visual_target` (`"<server> / <database>"`), `host`. Index on (`job`, `id DESC`).
-- `job_schedule`: one row per job (`job` PK). `next_run_at`, `interval_seconds`, `paused`, `updated_at`. Written by the worker after each run for the Admin tab. Seeded for `visual-feed`, `drawing-lookup` and `housekeeping`.
+- `job_schedule`: one row per job (`job` PK). `next_run_at`, `interval_seconds`, `paused`, `updated_at`. Written by the worker after each run for the Admin tab. Seeded for `visual-feed`, `drawing-lookup` and `housekeeping`; 0010 adds `ad-reconcile`, `session-refresh`, `integrity`, `backup-check`, `notify-send` and `notify-digest`.
 - `worker_heartbeat`: a single row (`id` = 1 by CHECK). `at`, `build`, `host`, `pid`, `running` text[] (the jobs running now). Upserted every 30 seconds.
 
 ## Notifications (phases 4 and 9)
@@ -361,7 +361,22 @@ Phase 4, migration 0005:
 
 - `notification_events`: one row per alert. `id` bigserial, `kind` (text, for example `assets_quota`), `severity` (`info`, `warning` or `critical` by CHECK, default `warning`), `subject` (what the alert is about, for example `assets`), `detail` jsonb (for the quota alert, the volume usage: `total`, `used`, `free`, `pct`, `warn`), `created_at`, `resolved_at` (set when the condition clears), `acknowledged_at` and `acknowledged_by` (an Admin saw it). The unique index `notification_events_open` on (`kind`, `coalesce(subject, '')`) where `resolved_at` is null keeps one open alert per kind and subject, so a check that runs again does not stack rows. The table is not append-only: the app sets `resolved_at` and the acknowledgement. 0005 also grants `ps_app` USAGE and SELECT on every sequence in the schema.
 
-Phase 9 builds the delivery on this table: rules, recipients, exclusions, outbox, sent log, state.
+Phase 9, migration 0010, builds the delivery on this table:
+
+- `notification_events` gains `event_type` (the catalog key; `kind` holds the same text), `title` (the mail subject after the prefix), `body`, `dept_code` (for the department events, so the department's Supervisor group can be the recipient), `raised_count` and `last_raised_at` (a condition raised again while open only counts), `recovery_of` (the row a "Cleared" notice follows), `dispatched_at` and `dispatch_result` (set by `notify-send`: sent, dry_run, failed, no_recipients, queued for the digest, rule_off, or no_rule; `directory_retry:N` marks an event left pending while AD was down, and the third failure dispatches it failed). The phase 4 quota rows were marked dispatched with `before_phase_9`, so they are never mailed late. A standing condition (feed failed, backup stale, quota, integrity failure, group missing, drawing failing, change order waiting) stays open until it clears, and the open index is the dedupe. A one-shot event (security alert, sign-in burst, reject, reconcile report, version deployed) is inserted already resolved. The plan's separate `notification_state` table is not needed: the open row is the state.
+- `notification_rules`: one row per event type (`event_type` PK). `enabled`, `delivery` (enum `notify_delivery`: immediate or digest), `digest_time` (site time, default 07:00), `param` (N hours for drawing_failing, N days for co_waiting), `version` (compare and set for the Admin tab), `updated_at`, `updated_by`. The code seeds a missing rule enabled, immediate, with PS-Admins as its recipient, and never changes a rule that exists.
+- `notification_recipients`: `event_type`, `kind` (enum `notify_recipient_kind`: group, user, mailbox, dept_supervisor), `group_dn` and `group_name` (any AD group under LDAP_BASE_DN), `user_id`, `address`. CHECKs make each kind carry its own column. Groups are resolved to their members' mail when a message goes, not stored.
+- `notification_exclusions`: (`event_type`, `user_id`) PK, `self` (set by the person on Account settings; never allowed on the security event types), `created_at`, `created_by`.
+- `notification_outbox`: events waiting for their rule's digest. `event_id`, `event_type`, `queued_at`, `sent_at`, `sent_id` (the `notifications_sent` row).
+- `notifications_sent`: append-only. One row per message: `at`, `event_type` (`digest` for a digest), `event_ids`, `digest`, `test`, `subject`, `recipients` jsonb (`[{address, user_id, why}]`, as resolved), `result` (sent, dry_run, failed, no_recipients), `error`, `requested_by` (for a test send).
+
+## Admin and integrity (phase 9, migration 0010)
+
+- `users.profile_active`: the Admin's switch on a person's profile, default true. `users.active` stays the reconcile's (disabled or gone in AD). An inactive profile drops out of the welder picker and the worker filter. It does not change what the person may do.
+- `departments.created_at` and `created_by`: set when an Admin adds a department in the app. Seeded departments have neither. `code` and `slug` stay immutable by trigger.
+- `assets.hash_checked_at` and `hash_ok`: when the integrity job last rehashed the file and whether it matched `sha256`.
+- `integrity_findings`: append-only. One row per thing the integrity job found wrong: `run_id` (no foreign key, since housekeeping prunes `job_runs`), `at`, `kind` (seal, asset_hash, asset_missing, blob_orphan, audit_chain), `subject` (the packet, asset, file path or audit id), `detail` jsonb.
+- Settings rows used by phase 9: `security` (the Admin's overrides of the session, lockout and rate-limit settings; missing keys fall back to the environment), `checklist_templates` (now edited on the Admin Checklists tab), `notify.last_build` (the build the worker last announced).
 
 ## VISUAL and drawing caches (phase 3)
 

docs/SECURITY.md

@@ -2,7 +2,7 @@
 
 This document is rewritten for the server build. The old browser build's security document is in the old repo (`C:\Users\spencerm\Coding\PacketStation\docs\SECURITY.md`). The design comes from [BRIEF.md](BRIEF.md) and [PLAN.md](PLAN.md).
 
-Status: phase 8. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Phase 4 completes the upload pipeline (a kind per upload with its own types and limit, XLSX and CSV by content, the optional ClamAV scan, the assets volume quota alert) and adds the warehouse rules (the heat gate on release, closed hold and short reasons, receive by custody). Phase 4 closes D5. Phase 5 adds the shop rules (the server as the clock of record, server-assigned mark numbers under the packet row lock, insert-only change-order verifications with a signature, VOID sheets never edited). Phase 6 adds the Q.C. rules (server gates on accept, block sign and sign-off, NCR numbers checked by pattern, Q.C. documents never deleted, the append-only ITP log, server-verified signatures on every Q.C. and ITP sign-off). No requirement ID closes in phase 6. Phase 7 adds the close and file controls (the HMAC seal with the data book filed on it, the controlled-copy and print stamps signed with the seal key, `print_log` and audit rows for every print and export, the CSV formula guard, PDF metadata, escaping in the PDFs). Phase 7 closes M2, S6, S7 and S8. Phase 8 adds the Project Management reads and exports, the server-side find bar filters and the presence rules (display name and role only, locked and revoked sessions dropped). No requirement ID closes in phase 8. Every other control names the phase that builds it.
+Status: phase 9. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Phase 4 completes the upload pipeline (a kind per upload with its own types and limit, XLSX and CSV by content, the optional ClamAV scan, the assets volume quota alert) and adds the warehouse rules (the heat gate on release, closed hold and short reasons, receive by custody). Phase 4 closes D5. Phase 5 adds the shop rules (the server as the clock of record, server-assigned mark numbers under the packet row lock, insert-only change-order verifications with a signature, VOID sheets never edited). Phase 6 adds the Q.C. rules (server gates on accept, block sign and sign-off, NCR numbers checked by pattern, Q.C. documents never deleted, the append-only ITP log, server-verified signatures on every Q.C. and ITP sign-off). No requirement ID closes in phase 6. Phase 7 adds the close and file controls (the HMAC seal with the data book filed on it, the controlled-copy and print stamps signed with the seal key, `print_log` and audit rows for every print and export, the CSV formula guard, PDF metadata, escaping in the PDFs). Phase 7 closes M2, S6, S7 and S8. Phase 8 adds the Project Management reads and exports, the server-side find bar filters and the presence rules (display name and role only, locked and revoked sessions dropped). No requirement ID closes in phase 8. Phase 9 adds the nightly AD reconcile and the 15-minute role refresh, the integrity job with its alert, email notifications over verified STARTTLS with security events nobody can mute, the security alerts and the Security tab settings, the Admin tabs (every change audited), nightly backups with a rehearsed restore, and dependency scanning in CI. Phase 9 closes the app part of I6, M3, O1, O2 and S3. Every other control names the phase that builds it.
 
 Contents:
 
@@ -22,7 +22,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 | 4 | Department and level screens | A static role-to-screen map in the browser, plus server policy on every request. The browser only decides what to show. | 1, 2 |
 | 5 | Per-PC access log | Replaced by the server audit. Each row carries the AD account, session, client IP and user agent. No per-PC browser ids. | 1 |
 | 6 | Filing seals and filed-packet lock | HMAC seals computed on the server with a server key. The filed lock is a server policy rule. Built. | 2, 7 |
-| 7 | Record integrity check | A worker job (seals, asset hashes, audit hash chain) with an alert. | 9 |
+| 7 | Record integrity check | A worker job every night (seals, asset hashes, missing and orphan files, the audit hash chain) with an email alert. Built. | 9 |
 | 8 | Self-hosted libraries | Kept: no CDN. The CSP allows nothing external. | 0 |
 | 9 | Output escaping | React escapes output by default. The PDF builders clean and escape every text on the server. Built. | 1, 7 |
 | 10 | Central permission policy | Server policy, checked on every write, including writes that do not come from the screens. Built. | 2 |
@@ -31,7 +31,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 | 13 | Screen lock | Server-enforced sessions. The browser reports idle and background state; the server records it and refuses an idle or locked session. Unlock is a step-up check on the server. | 1 |
 | 14 | Input and upload validation | On the server: field limits on every write, closed reason sets, uploads typed by content per kind (drawing PDF; MTR and document PDF, PNG or JPEG; BOM XLSX or CSV), a size limit per kind (25 MB, BOM 5 MB), sanitised names, an optional ClamAV scan before anything is stored, and a quota alert on the assets volume. Built. | 2, 4 |
 | 15 | Second-admin approval | Superseded. Roles come only from AD groups and the group-to-role map is derived, so there is nothing to grant in the app. | 1 |
-| 16 | Security alerts and automatic integrity check | Worker jobs and the Notifications tab (email through app-03). | 9 |
+| 16 | Security alerts and automatic integrity check | The Admin Security tab (alerts from the audit and the sign-in attempts, the last integrity run), the worker jobs and the Notifications tab (email through app-03). Built. | 9 |
 | 17 | Controlled-copy stamp on data books | Applied to server-built PDFs and signed with the seal key. Built, phase 7. | 7 |
 | 18 | CSV formula guard | Applied to every server export. Built, phase 7. | 7 |
 | 19 | Update lock and stale-version block | Replaced by a build version endpoint. A screen on an older build shows a banner and reloads. | 0 (endpoint), 10 (banner) |
@@ -42,11 +42,11 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 ### Sessions (phase 1)
 
 - HttpOnly, SameSite=Strict session cookie. The session row on the server decides.
-- Password sessions: idle sign-out after 15 minutes, absolute 12 hours. PIN shop sessions: absolute 10 hours, idle 15 minutes. Admin and Clerical sessions wait 60 idle minutes (issue 45). All are settings.
+- Password sessions: idle sign-out after 15 minutes, absolute 12 hours. PIN shop sessions: absolute 10 hours, idle 15 minutes. Admin and Clerical sessions wait 60 idle minutes (issue 45). All are settings, and from phase 9 an Admin can change them within bounds on the Security tab (see "Security settings").
 - Idle and background lock are reported by the browser and enforced by the server. Unlock is a step-up check with the PIN or the domain password.
 - Origin check on every state-changing request: the `Origin` header (or the `Referer` origin) must match the request host, and `X-Requested-With: PacketStation` must be present. Otherwise 403.
 - Admin is a password session only. A PIN session cannot open Admin or change Account settings.
-- Roles are resolved from AD at sign-in (nested groups through `LDAP_MATCHING_RULE_IN_CHAIN`), cached on the session, and re-resolved every 15 minutes on the session and by the AD refresh. A disabled AD account is always refused. A session whose user has lost every role is ended at the next refresh.
+- Roles are resolved from AD at sign-in (nested groups through `LDAP_MATCHING_RULE_IN_CHAIN`), cached on the session, and re-resolved every 15 minutes on the session, by the `session-refresh` job and by the nightly AD reconcile (see "AD reconcile and role refresh"). A disabled AD account is always refused. A session whose user has lost every role is ended at the next refresh.
 - The session cookie token is stored only as an HMAC with the session key. The AD password is used for the bind and discarded.
 - The group-to-role map is derived from the department table and the `PS-` naming convention at startup. No screen edits it, so nobody can promote a group or themselves inside the app.
 - The browser reports hidden tabs and idle time; the server records `locked_since` and refuses a locked session with 423 until a step-up unlock. The server also locks a session itself when no request arrived for the lock interval (3 minutes, as the old build's screen lock; 60 for Admin and Clerical, issue 45), and when a tab stayed hidden for over a minute (the old build's background lock). A reload or an in-app navigation never locks.
@@ -64,7 +64,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 
 - Checked before any bind to AD, so the app can never lock a domain account out.
 - Defaults: 5 failed password sign-ins per account per 5 minutes, then a 10-minute cooldown. 30 failed attempts per client address per 5 minutes. Successful sign-ins never count.
-- The domain's own lockout (20 attempts in 5 minutes) stays the last line.
+- The domain's own lockout (20 attempts in 5 minutes) stays the last line. From phase 9 the limits can be changed on the Security tab, and the per-account limit is capped at 15, always under AD's 20.
 - The client address comes from `X-Forwarded-For` only when the request comes from `npm_app_1` itself, re-resolved on change. Other containers on `npm_default` are not trusted.
 - A refused attempt answers 429 with `Retry-After`, is recorded as `rate_limited` and never counts toward the limits itself.
 - Failed sign-ins answer one message for an unknown account, a wrong password and a disabled account, so accounts cannot be enumerated. The audit and `signin_attempts` keep the real reason.
@@ -90,7 +90,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - Limits per kind: 25 MB (`upload_max_bytes`) for drawing, mtr and document, 5 MB (`upload_max_bytes_bom`) for a BOM. The Content-Length is checked first, then the bytes as they are read. Over the limit is 413 `too_large`.
 - XLSX and CSV are always served as an attachment, never inline. Every blob keeps `Content-Disposition` and `nosniff`.
 - Optional ClamAV (requirement D5). When `CLAMAV_HOST` is set, every upload and every file the worker stores (drawings from M-Files) is streamed to clamd with `zINSTREAM` after it is read in full and before it is deduped or placed. A hit is 422 `infected`, nothing is stored, and an `asset.rejected` audit row keeps the signature, size and type. A scanner that cannot be reached refuses the upload with 503 `scan_unavailable` (fail closed). With `CLAMAV_HOST` empty there is no scan. IT decided on 2026-10-18 (Gitea issue 19) to leave the scan off in both stacks: every workstation that uploads runs Microsoft Defender. Drawings the worker fetches from M-Files are not scanned on the way in. They come from PESCO's own vault.
-- Quota alert. After every stored file and in the housekeeping job, the assets volume usage is checked against `ASSETS_QUOTA_WARN_PCT` (85 by default). Over it, one open `notification_events` row of kind `assets_quota` is written and audited as `storage.quota_warning`. Back under, the row is resolved. A failed check never fails an upload. Admin sees usage, the alert and the scanner on the Storage tab and can acknowledge an alert (`alert.acknowledged`). Phase 9 sends the email to PS-Admins (Gitea issue 20).
+- Quota alert. After every stored file and in the housekeeping job, the assets volume usage is checked against `ASSETS_QUOTA_WARN_PCT` (85 by default). Over it, one open `notification_events` row of kind `assets_quota` is written and audited as `storage.quota_warning`. Back under, the row is resolved. A failed check never fails an upload. Admin sees usage, the alert and the scanner on the Storage tab and can acknowledge an alert (`alert.acknowledged`). From phase 9 the alert and its "Cleared" notice are mailed through the `assets_quota` rule, to PS-Admins by default (Gitea issue 20).
 
 ### Warehouse rules (phase 4)
 
@@ -136,7 +136,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - Append-only `audit_events`, with a hash chain (`prev_hash`, `row_hash`).
 - The app role may insert and select, never update or delete. A trigger also refuses updates and deletes. Migrations run as a separate role.
 - `packetstation doctor` checks that an audit UPDATE fails.
-- Every identity event is a row: sign-in, failed sign-in with a reason, sign-out, lock, unlock, session ended, PIN set, disabled or cleared, device registered or revoked, signature, AD refresh. Each carries the user, session, roles, client address and a small detail object, never a secret.
+- Every identity event is a row: sign-in, failed sign-in with a reason, sign-out, lock, unlock, session ended, PIN set, disabled or cleared, device registered or revoked, signature, AD refresh. Phase 9 adds `auth.lockout`, `auth.signin_burst`, `policy.denied`, `auth.session_ended` with `by` for sessions the reconcile and role refresh end, and the `admin.*` rows for every Admin change. Each carries the user, session, roles, client address and a small detail object, never a secret.
 - `signin_attempts` and `signatures` are append-only in the same way.
 - Phase 4 audit actions: `asset.rejected` (an upload the virus scan refused, with `reason`, `signature`, `size` and `mime`), `storage.quota_warning` (the assets volume went over the warning threshold, with the usage) and `alert.acknowledged` (an Admin acknowledged an open alert). Every warehouse write is a `packet.<action>` row with `detail.item`, the warehouse stage before and after, and `detail.qty` on short writes.
 
@@ -150,14 +150,14 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - The data book is built on the server inside the filing transaction and stored as an asset on the seal. A data book that cannot be built fails the filing; nothing is filed half way.
 - Filing while the feed shows a work order open in VISUAL needs a second confirmation. The work orders are kept on the seal and in the packet log.
 - A reopen by Admin needs a reason and a signature. It marks the seal broken with who, when and why, and keeps the row and its data book. Filing again writes a new seal beside it.
-- The integrity job checks seals and asset hashes (phase 9).
+- The integrity job re-verifies every seal of a filed packet every night (see "Integrity job").
 
 ### PDFs, prints and exports (phase 7)
 
 - Every PDF and the CSV are built on the server. The browser only opens the URL. Reads need a signed-in session; every role that can open the packet may read them, PM included.
 - Controlled-copy stamp (S7): every exported page carries `Controlled copy - exported by Name (emp) <UTC time>`, the seal's first 12 hex or `not yet filed`, and ` - stamp <sig>`. The filed data book is the stored copy, served as it was built.
 - Printed-copy stamp (S8): every printed page carries `PRINTED COPY - <what> - printed by Name (emp) <UTC time> - <project> Lot <unit> dwg <drawing> R<rev> - verify revision before use` and the stamp signature, in a white box at the bottom of the page.
-- Stamp signature: 16 hex of HMAC-SHA256 with the seal key over the packet, the kind (a printed copy signs `print:<kind>`, so a print and an export of the same thing in the same minute never share a signature), the seal hash, the user and the time. A copy can be checked against the server later with `verify_stamp`; the Admin screen for it comes in phase 9.
+- Stamp signature: 16 hex of HMAC-SHA256 with the seal key over the packet, the kind (a printed copy signs `print:<kind>`, so a print and an export of the same thing in the same minute never share a signature), the seal hash, the user and the time. A copy can be checked against the server with `verify_stamp`; from phase 9 the Admin Stamp check tab does it from a pasted line or a PDF.
 - Logging (S7, S8): every print and export writes one `print_log` row (who, when, session, packet, unit or ITP, kind, what, pages, the stamp line, the stamp signature, the client address) and one audit row (`print` or `export`). `print_log` is append-only: the app role has SELECT and INSERT only, and a trigger refuses updates, deletes and truncate.
 - CSV formula guard (S6): a cell that starts with `=`, `+`, `-`, `@`, a tab or a CR gets a leading `'`. A cell with a quote, a comma or a line break is quoted with the quotes doubled.
 - PDF metadata: title, subject and keywords name the packet and the stamp signature; Author is PESCO and Producer is Packet Station.
@@ -177,6 +177,93 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - A locked session drops out of presence at once, and comes back on unlock. A revoked session drops out at once and its stream ends. A closed tab drops out when the stream ends, within the 25-second ping cycle.
 - Saved filters live in the browser's localStorage (`ps.flt`, `ps.proj`, `ps.pmtab`, `ps.pmper`, `ps.motion`). They hold filter choices only. The search text is never saved. Nothing in them is trusted by the server.
 
+### AD reconcile and role refresh (phase 9)
+
+- `ad-reconcile` runs every night at 01:30 site time, and when an Admin presses "Refresh from AD now" (a job request; the button answers 202 at once). It lists the PacketStation OU, reports mapped groups missing from it and extra groups in it, reads every mapped group's members through `LDAP_MATCHING_RULE_IN_CHAIN` (nested) in pages of 500, and upserts users on their objectGUID. AD is read first, then everything is written in one transaction.
+- A user disabled in AD, or gone from AD, is marked inactive and every session they have is ended (`ad_disabled`, `ad_gone`). A user still in AD but in no mapped group loses their roles and their sessions (`roles_lost`). Each ended session writes an `auth.session_ended` audit row with the reason and `by: ad-reconcile`.
+- Missing and duplicate employee IDs are flagged on the user (`directory_flags`) and listed on the Admin Profiles tab. A PIN sign-in by a duplicate employee ID is refused, as in phase 1.
+- Safety guard: the reconcile refuses and changes nothing when no mapped group is found in the OU (a wrong `LDAP_GROUP_BASE_DN`) or when every mapped group is empty while active users exist (PLAN.md open question 44). AD unreachable fails the run and changes nothing.
+- `session-refresh` runs every 15 minutes (the `roles_refresh_minutes` setting, 5 to 60). For every user with a live session it reads the account by objectGUID and its groups again, rebuilds the roles and each session's role snapshot, and ends the sessions of anyone disabled, gone or left with no role. AD unreachable makes the run degraded and ends nothing.
+- A request still re-reads the roles when its session's snapshot is older than the refresh interval, as in phase 1. Between the two, a person removed from a group loses the role within 15 minutes, and a person disabled in AD is signed out within 15 minutes.
+- Together these close the app's part of I6 (joiner, mover, leaver). The quarterly access review stays IT's.
+
+### Integrity job (phase 9)
+
+- `integrity` runs every night at 04:00 site time. It re-verifies every unbroken seal of a filed packet with the seal key (a seal under another key id is a finding), rehashes asset files (uploads of the last 7 days every night, every asset on Sundays) and records `hash_checked_at` and `hash_ok`, finds asset rows with no file and files under the assets volume with no asset row (older than an hour; reported, never deleted), and walks the audit hash chain.
+- Each finding is a row in the append-only `integrity_findings` (at most 500 per kind per run, the full count in the run). Any finding raises the `integrity_failure` alert, a security event that nobody can opt out of. A clean run clears it with a "Cleared" message. The Admin Security tab shows the last run and its findings.
+- A missing seal key makes the run degraded and leaves the seals unchecked, never "clean".
+- This closes M3 (tamper detection on records, files and the audit).
+
+### Notifications (phase 9)
+
+- Mail goes through `SMTP_HOST` (app-03.pesco.com, port 25) with STARTTLS, verified against `SMTP_CA_BUNDLE` (the PESCO CA). There is no plain-text fallback: a relay without STARTTLS, a TLS error or a wrong certificate fails the message, and the failure is recorded. There is no SMTP login.
+- Each message is addressed To `SMTP_FROM` (`[email protected]`). The recipients are in the envelope only, so nobody sees the list.
+- Recipients are resolved when the message goes. Group members must be enabled accounts with a mail address. An inactive user gets nothing. Every message writes one append-only `notifications_sent` row with each address, the user and why it was chosen.
+- Bodies are plain text from the server. No link carries a token or a session. Raising an event never fails the action that raised it.
+- The test stack runs with `NOTIFY_DRY_RUN=true`: messages are recorded as `dry_run` and not sent. A test send from the Admin tab always sends. Subjects start `[Packet Station]` on production and `[Packet Station TEST]` on any other stack.
+- The security event types (`security_alert`, `signin_burst`, `integrity_failure`) cannot be opted out of on Account settings, and an Admin cannot exclude anyone from them.
+
+### Security alerts (phase 9)
+
+The Admin Security tab lists, for the last 24 hours by default (up to 168):
+
+| Kind | From |
+|---|---|
+| Account cooldowns (`lockout`) | `auth.lockout` audit rows, written when an account goes over its failure limit |
+| Wrong PINs (`pin_failures`) | Three or more wrong PINs for one person in the window, at any step |
+| Wrong unlock or signature (`stepup_wrong_pin`) | A wrong PIN or password at unlock or at a signature |
+| Denied actions (`denied`) | `policy.denied` audit rows: any 403 after the person was known (method, path and code; never a body) |
+| Sign-in bursts (`signin_burst`) | `auth.signin_burst` audit rows: an address over its limit, or failed sign-ins across accounts in 5 minutes at or over the burst threshold |
+| Admin changes (`admin_change`) | `admin.*` audit rows |
+| IT edits (`it_edit`) | Audit rows with an IT reason |
+| Integrity (`integrity`) | `integrity_findings` rows |
+
+Account cooldowns, a PIN disabled, a session ended after wrong PINs and sign-in bursts are also mailed (`security_alert`, `signin_burst`).
+
+### Security settings (phase 9)
+
+The Admin Security tab overrides these environment defaults, stored in the `security` settings row with compare and set, audited with before and after. A value out of bounds is refused; a stored value out of bounds is ignored and the default applies. The app reads them per request, cached for at most 30 seconds.
+
+| Setting | Default | Bounds |
+|---|---|---|
+| `session_lock_minutes` (lock after idle) | 3 | 1 to 30 |
+| `session_idle_minutes` (sign out after idle) | 15 | 1 to 60 |
+| `session_desk_minutes` (Admin and Clerical idle) | 60 | 5 to 240 |
+| `session_password_hours` | 12 | 1 to 24 |
+| `session_pin_hours` | 10 | 1 to 24 |
+| `roles_refresh_minutes` | 15 | 5 to 60 |
+| `signin_account_limit` (failed sign-ins per account before a cooldown) | 5 | 1 to 15 |
+| `signin_account_window_minutes` | 5 | 1 to 60 |
+| `signin_account_cooldown_minutes` | 10 | 0 to 120 |
+| `signin_ip_limit` (failures per address) | 30 | 5 to 200 |
+| `signin_ip_window_minutes` | 5 | 1 to 60 |
+| `signin_burst_threshold` (failures across accounts in 5 minutes that raise an alert) | 20 | 5 to 200 |
+
+The account limit is capped at 15 so the app always stops before AD's own lockout at 20 attempts in 5 minutes. The PIN limit (5) and the step-up limit (3) stay environment settings.
+
+### Admin (phase 9)
+
+- Every Admin endpoint needs PS-Admins in a password session. Every Admin change is audited with before and after. Tests try every endpoint as every role.
+- Roles still come only from AD. The Admin edits departments (name, stations), profiles (stamp, profile active, station limits), checklists, the security settings and the notification rules. None of these grants a role. Turning a profile off hides the person from pickers and changes no right.
+- Department codes and short names never change (a trigger, and 422 on the API). Stations are never deleted.
+- The access log CSV has the formula guard, at most 50,000 rows, and is logged in `print_log` and the audit.
+- The stamp check recomputes the stamp signature with the seal key and compares the text with the print log (S7, S8). It reads a pasted line or a PDF of up to 25 MB.
+
+### Backups (phase 9)
+
+- `scripts/backup/ps-backup.sh` runs as root from a systemd timer at 02:00 Denver time, an hour before the Proxmox Backup Server run of the whole VM. It dumps each stack with `pg_dump -Fc` inside the postgres container over the local socket (no password read or passed), checks each dump with `pg_restore --list`, keeps 7, and writes `last-success` or `last-failure`.
+- The roles are dumped with `--no-role-passwords`. A password hash in a file is an offline cracking target, and the file lands in PBS. The restore script refuses a globals file that holds a password hash. The role passwords stay in the stack secrets.
+- Dumps and globals are mode 0600. The production folder is `root:10001` 0750, so only root and the worker (read-only mount) see it.
+- The worker's `backup-check` job raises `backup_stale` when the newest dump is missing, empty, older than 26 hours, or followed by a failure.
+- The restore was rehearsed on the test stack on 2026-10-21 into a separate database: every table's row count matched and all 189 asset files matched their recorded sha256 ([DEPLOY.md](DEPLOY.md), "Restore rehearsal 2026-10-21"). This closes O1 and O2 for the first rehearsal. The quarterly restore test is IT's.
+
+### Dependency scanning (phase 9)
+
+- Every CI run scans the dependencies. `pip-audit` (pinned, in its own virtual environment) checks every pin in `server/requirements.txt` without resolving and fails on any known vulnerability, which is stricter than high or critical. `npm audit --audit-level=high` fails on a high or critical finding in the frontend, dev dependencies included.
+- `visualkit` and `mfileskit` come from PESCO's own index and are not audited. Their dependencies that we install are pinned in `requirements.txt` and are.
+- No finding is ignored. If one ever has no fix, the way out is `--ignore-vuln <ID>` with a comment and a date.
+- The first scan (2026-10-21) found `pypdf` 6.21.0 (GHSA-5hx5-vpf9-5wcv) and `certifi` 2026.7.22 (PYSEC-2026-4320). Both were bumped. This closes S3.
+
 ### Headers (phase 0)
 
 - The app sets HSTS, CSP and the other security headers itself. NPM's own HSTS is off.
@@ -203,7 +290,7 @@ The IDs come from Part 3 of the old SECURITY.md. The status follows the approved
 | I3 | Closed for "PIN on the server", phase 1; device certificate deferred |
 | I4 | Closed, phase 1 |
 | I5 | Superseded (no in-app admin grant); separate IT admin accounts remain IT's |
-| I6 | Largely closed, phase 9 (groups plus reconcile); quarterly review IT's |
+| I6 | App part closed, phase 9 (roles only from AD groups; the nightly `ad-reconcile` marks disabled and gone users inactive, ends their sessions and flags employee ID problems; `session-refresh` re-reads every live session's groups and account state every 15 minutes and ends the sessions of anyone disabled, gone or left with no role); quarterly access review IT's |
 | I7 | Closed, phase 1 |
 | I8 | App part closed, phase 1; kiosk lock policy IT's (Intune or GPO) |
 | A1, A2, A3 | Closed, phase 2 (server policy on every call, workflow steps as compare-and-set, validation and limits on the server) |
@@ -214,17 +301,17 @@ The IDs come from Part 3 of the old SECURITY.md. The status follows the approved
 | D6, D7 | Deferred (retention period and customer copy need a decision from Seth and Q.C.) |
 | M1 | Closed, phase 2 (append-only audit and packet log written only by the server; every packet write carries both) |
 | M2 | Closed, phase 7 (filing seal is HMAC-SHA256 with the server seal key from a Docker secret; a reopen keeps the broken seal and its data book) |
-| M3 | Closed, phase 9 |
+| M3 | Closed, phase 9 (the nightly `integrity` job re-verifies seals, rehashes asset files, finds missing and orphan files and walks the audit hash chain; findings are kept in the append-only `integrity_findings` and raise `integrity_failure`, which nobody can opt out of) |
 | M4, M5 | Deferred (logs to stdout now; SIEM shipping is IT's) |
 | S1 | Closed, phase 0, tuned phase 10 |
 | S2 | Closed, phase 1 |
-| S3 | Closed, phase 9 (CI scanning) |
+| S3 | Closed, phase 9 (CI scanning on every run: `pip-audit` over every pin in `server/requirements.txt` fails on any known vulnerability; `npm audit --audit-level=high` fails on a high or critical one; the first scan bumped pypdf and certifi) |
 | S4 | Deferred to IT (pentest before go-live) |
 | S5 | Closed, phase 2 (React escapes output; server messages are plain text); phase 7 adds escaping in the PDF builders |
 | S6 | Closed, phase 7 (the CSV formula guard on every export) |
 | S7 | Closed, phase 7 (the controlled-copy stamp on every exported PDF, signed with the seal key; PDF metadata; every export logged in `print_log` and the audit) |
 | S8 | Closed, phase 7 (the PRINTED COPY stamp with "verify revision before use" on every printed page, signed with the seal key; every print logged in `print_log` and the audit). PLAN.md lists S8 under phase 5, but the print stamps come with the PDFs in phase 7, so S8 closes there. |
-| O1, O2 | Closed, phase 9 (pg_dump timer plus PBS; one rehearsed restore); quarterly restore IT's |
+| O1, O2 | Closed, phase 9 (nightly `pg_dump` of each stack at 02:00 Denver time from a systemd timer, checked with `pg_restore --list`, 7 kept, before the 03:00 PBS run of the VM; the `backup-check` job alerts on a missing, empty, stale or failed backup; restore rehearsed on the test stack on 2026-10-21 with every row count and asset hash matching); quarterly restore test IT's |
 | O3, O4 | Closed, phase 10 |
 | O5 | Superseded (no migration; M: retired read-only) |
 
@@ -232,7 +319,7 @@ The brief's wording for the identity IDs, which the table above follows:
 
 - **I4 and I7** (PIN lockout, PIN signatures verified on the server) are closed by this build.
 - **I8** is closed for the app's part. The kiosk device lock policy in I8 is IT's, via Intune or GPO.
-- **I6** (joiner, mover, leaver) is largely carried by the AD groups plus the nightly reconcile. The quarterly access review is still IT's.
+- **I6** (joiner, mover, leaver) is closed for the app's part by the AD groups, the nightly reconcile and the 15-minute role refresh. The quarterly access review is still IT's.
 - **I3** is closed for "PIN checked on the server" and open for "Intune-managed, device certificate". Registered devices use a device cookie for now.
 - **I1** is partly closed by AD sign-in. There is no SSO yet.
 - **I5** is superseded. No in-app admin grant exists, so second-admin approval has nothing to approve. Separate IT admin accounts with MFA remain IT's.