Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/SECURITY.md

@@ -2,7 +2,7 @@
 
 This document is rewritten for the server build. The old browser build's security document is in the old repo (`C:\Users\spencerm\Coding\PacketStation\docs\SECURITY.md`). The design comes from [BRIEF.md](BRIEF.md) and [PLAN.md](PLAN.md).
 
-Status: phase 8. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Phase 4 completes the upload pipeline (a kind per upload with its own types and limit, XLSX and CSV by content, the optional ClamAV scan, the assets volume quota alert) and adds the warehouse rules (the heat gate on release, closed hold and short reasons, receive by custody). Phase 4 closes D5. Phase 5 adds the shop rules (the server as the clock of record, server-assigned mark numbers under the packet row lock, insert-only change-order verifications with a signature, VOID sheets never edited). Phase 6 adds the Q.C. rules (server gates on accept, block sign and sign-off, NCR numbers checked by pattern, Q.C. documents never deleted, the append-only ITP log, server-verified signatures on every Q.C. and ITP sign-off). No requirement ID closes in phase 6. Phase 7 adds the close and file controls (the HMAC seal with the data book filed on it, the controlled-copy and print stamps signed with the seal key, `print_log` and audit rows for every print and export, the CSV formula guard, PDF metadata, escaping in the PDFs). Phase 7 closes M2, S6, S7 and S8. Phase 8 adds the Project Management reads and exports, the server-side find bar filters and the presence rules (display name and role only, locked and revoked sessions dropped). No requirement ID closes in phase 8. Every other control names the phase that builds it.
+Status: phase 9. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Phase 4 completes the upload pipeline (a kind per upload with its own types and limit, XLSX and CSV by content, the optional ClamAV scan, the assets volume quota alert) and adds the warehouse rules (the heat gate on release, closed hold and short reasons, receive by custody). Phase 4 closes D5. Phase 5 adds the shop rules (the server as the clock of record, server-assigned mark numbers under the packet row lock, insert-only change-order verifications with a signature, VOID sheets never edited). Phase 6 adds the Q.C. rules (server gates on accept, block sign and sign-off, NCR numbers checked by pattern, Q.C. documents never deleted, the append-only ITP log, server-verified signatures on every Q.C. and ITP sign-off). No requirement ID closes in phase 6. Phase 7 adds the close and file controls (the HMAC seal with the data book filed on it, the controlled-copy and print stamps signed with the seal key, `print_log` and audit rows for every print and export, the CSV formula guard, PDF metadata, escaping in the PDFs). Phase 7 closes M2, S6, S7 and S8. Phase 8 adds the Project Management reads and exports, the server-side find bar filters and the presence rules (display name and role only, locked and revoked sessions dropped). No requirement ID closes in phase 8. Phase 9 adds the nightly AD reconcile and the 15-minute role refresh, the integrity job with its alert, email notifications over verified STARTTLS with security events nobody can mute, the security alerts and the Security tab settings, the Admin tabs (every change audited), nightly backups with a rehearsed restore, and dependency scanning in CI. Phase 9 closes the app part of I6, M3, O1, O2 and S3. Every other control names the phase that builds it.
 
 Contents:
 
@@ -22,7 +22,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 | 4 | Department and level screens | A static role-to-screen map in the browser, plus server policy on every request. The browser only decides what to show. | 1, 2 |
 | 5 | Per-PC access log | Replaced by the server audit. Each row carries the AD account, session, client IP and user agent. No per-PC browser ids. | 1 |
 | 6 | Filing seals and filed-packet lock | HMAC seals computed on the server with a server key. The filed lock is a server policy rule. Built. | 2, 7 |
-| 7 | Record integrity check | A worker job (seals, asset hashes, audit hash chain) with an alert. | 9 |
+| 7 | Record integrity check | A worker job every night (seals, asset hashes, missing and orphan files, the audit hash chain) with an email alert. Built. | 9 |
 | 8 | Self-hosted libraries | Kept: no CDN. The CSP allows nothing external. | 0 |
 | 9 | Output escaping | React escapes output by default. The PDF builders clean and escape every text on the server. Built. | 1, 7 |
 | 10 | Central permission policy | Server policy, checked on every write, including writes that do not come from the screens. Built. | 2 |
@@ -31,7 +31,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 | 13 | Screen lock | Server-enforced sessions. The browser reports idle and background state; the server records it and refuses an idle or locked session. Unlock is a step-up check on the server. | 1 |
 | 14 | Input and upload validation | On the server: field limits on every write, closed reason sets, uploads typed by content per kind (drawing PDF; MTR and document PDF, PNG or JPEG; BOM XLSX or CSV), a size limit per kind (25 MB, BOM 5 MB), sanitised names, an optional ClamAV scan before anything is stored, and a quota alert on the assets volume. Built. | 2, 4 |
 | 15 | Second-admin approval | Superseded. Roles come only from AD groups and the group-to-role map is derived, so there is nothing to grant in the app. | 1 |
-| 16 | Security alerts and automatic integrity check | Worker jobs and the Notifications tab (email through app-03). | 9 |
+| 16 | Security alerts and automatic integrity check | The Admin Security tab (alerts from the audit and the sign-in attempts, the last integrity run), the worker jobs and the Notifications tab (email through app-03). Built. | 9 |
 | 17 | Controlled-copy stamp on data books | Applied to server-built PDFs and signed with the seal key. Built, phase 7. | 7 |
 | 18 | CSV formula guard | Applied to every server export. Built, phase 7. | 7 |
 | 19 | Update lock and stale-version block | Replaced by a build version endpoint. A screen on an older build shows a banner and reloads. | 0 (endpoint), 10 (banner) |
@@ -42,11 +42,11 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 ### Sessions (phase 1)
 
 - HttpOnly, SameSite=Strict session cookie. The session row on the server decides.
-- Password sessions: idle sign-out after 15 minutes, absolute 12 hours. PIN shop sessions: absolute 10 hours, idle 15 minutes. Admin and Clerical sessions wait 60 idle minutes (issue 45). All are settings.
+- Password sessions: idle sign-out after 15 minutes, absolute 12 hours. PIN shop sessions: absolute 10 hours, idle 15 minutes. Admin and Clerical sessions wait 60 idle minutes (issue 45). All are settings, and from phase 9 an Admin can change them within bounds on the Security tab (see "Security settings").
 - Idle and background lock are reported by the browser and enforced by the server. Unlock is a step-up check with the PIN or the domain password.
 - Origin check on every state-changing request: the `Origin` header (or the `Referer` origin) must match the request host, and `X-Requested-With: PacketStation` must be present. Otherwise 403.
 - Admin is a password session only. A PIN session cannot open Admin or change Account settings.
-- Roles are resolved from AD at sign-in (nested groups through `LDAP_MATCHING_RULE_IN_CHAIN`), cached on the session, and re-resolved every 15 minutes on the session and by the AD refresh. A disabled AD account is always refused. A session whose user has lost every role is ended at the next refresh.
+- Roles are resolved from AD at sign-in (nested groups through `LDAP_MATCHING_RULE_IN_CHAIN`), cached on the session, and re-resolved every 15 minutes on the session, by the `session-refresh` job and by the nightly AD reconcile (see "AD reconcile and role refresh"). A disabled AD account is always refused. A session whose user has lost every role is ended at the next refresh.
 - The session cookie token is stored only as an HMAC with the session key. The AD password is used for the bind and discarded.
 - The group-to-role map is derived from the department table and the `PS-` naming convention at startup. No screen edits it, so nobody can promote a group or themselves inside the app.
 - The browser reports hidden tabs and idle time; the server records `locked_since` and refuses a locked session with 423 until a step-up unlock. The server also locks a session itself when no request arrived for the lock interval (3 minutes, as the old build's screen lock; 60 for Admin and Clerical, issue 45), and when a tab stayed hidden for over a minute (the old build's background lock). A reload or an in-app navigation never locks.
@@ -64,7 +64,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 
 - Checked before any bind to AD, so the app can never lock a domain account out.
 - Defaults: 5 failed password sign-ins per account per 5 minutes, then a 10-minute cooldown. 30 failed attempts per client address per 5 minutes. Successful sign-ins never count.
-- The domain's own lockout (20 attempts in 5 minutes) stays the last line.
+- The domain's own lockout (20 attempts in 5 minutes) stays the last line. From phase 9 the limits can be changed on the Security tab, and the per-account limit is capped at 15, always under AD's 20.
 - The client address comes from `X-Forwarded-For` only when the request comes from `npm_app_1` itself, re-resolved on change. Other containers on `npm_default` are not trusted.
 - A refused attempt answers 429 with `Retry-After`, is recorded as `rate_limited` and never counts toward the limits itself.
 - Failed sign-ins answer one message for an unknown account, a wrong password and a disabled account, so accounts cannot be enumerated. The audit and `signin_attempts` keep the real reason.
@@ -90,7 +90,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - Limits per kind: 25 MB (`upload_max_bytes`) for drawing, mtr and document, 5 MB (`upload_max_bytes_bom`) for a BOM. The Content-Length is checked first, then the bytes as they are read. Over the limit is 413 `too_large`.
 - XLSX and CSV are always served as an attachment, never inline. Every blob keeps `Content-Disposition` and `nosniff`.
 - Optional ClamAV (requirement D5). When `CLAMAV_HOST` is set, every upload and every file the worker stores (drawings from M-Files) is streamed to clamd with `zINSTREAM` after it is read in full and before it is deduped or placed. A hit is 422 `infected`, nothing is stored, and an `asset.rejected` audit row keeps the signature, size and type. A scanner that cannot be reached refuses the upload with 503 `scan_unavailable` (fail closed). With `CLAMAV_HOST` empty there is no scan. IT decided on 2026-10-18 (Gitea issue 19) to leave the scan off in both stacks: every workstation that uploads runs Microsoft Defender. Drawings the worker fetches from M-Files are not scanned on the way in. They come from PESCO's own vault.
-- Quota alert. After every stored file and in the housekeeping job, the assets volume usage is checked against `ASSETS_QUOTA_WARN_PCT` (85 by default). Over it, one open `notification_events` row of kind `assets_quota` is written and audited as `storage.quota_warning`. Back under, the row is resolved. A failed check never fails an upload. Admin sees usage, the alert and the scanner on the Storage tab and can acknowledge an alert (`alert.acknowledged`). Phase 9 sends the email to PS-Admins (Gitea issue 20).
+- Quota alert. After every stored file and in the housekeeping job, the assets volume usage is checked against `ASSETS_QUOTA_WARN_PCT` (85 by default). Over it, one open `notification_events` row of kind `assets_quota` is written and audited as `storage.quota_warning`. Back under, the row is resolved. A failed check never fails an upload. Admin sees usage, the alert and the scanner on the Storage tab and can acknowledge an alert (`alert.acknowledged`). From phase 9 the alert and its "Cleared" notice are mailed through the `assets_quota` rule, to PS-Admins by default (Gitea issue 20).
 
 ### Warehouse rules (phase 4)
 
@@ -136,7 +136,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - Append-only `audit_events`, with a hash chain (`prev_hash`, `row_hash`).
 - The app role may insert and select, never update or delete. A trigger also refuses updates and deletes. Migrations run as a separate role.
 - `packetstation doctor` checks that an audit UPDATE fails.
-- Every identity event is a row: sign-in, failed sign-in with a reason, sign-out, lock, unlock, session ended, PIN set, disabled or cleared, device registered or revoked, signature, AD refresh. Each carries the user, session, roles, client address and a small detail object, never a secret.
+- Every identity event is a row: sign-in, failed sign-in with a reason, sign-out, lock, unlock, session ended, PIN set, disabled or cleared, device registered or revoked, signature, AD refresh. Phase 9 adds `auth.lockout`, `auth.signin_burst`, `policy.denied`, `auth.session_ended` with `by` for sessions the reconcile and role refresh end, and the `admin.*` rows for every Admin change. Each carries the user, session, roles, client address and a small detail object, never a secret.
 - `signin_attempts` and `signatures` are append-only in the same way.
 - Phase 4 audit actions: `asset.rejected` (an upload the virus scan refused, with `reason`, `signature`, `size` and `mime`), `storage.quota_warning` (the assets volume went over the warning threshold, with the usage) and `alert.acknowledged` (an Admin acknowledged an open alert). Every warehouse write is a `packet.<action>` row with `detail.item`, the warehouse stage before and after, and `detail.qty` on short writes.
 
@@ -150,14 +150,14 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - The data book is built on the server inside the filing transaction and stored as an asset on the seal. A data book that cannot be built fails the filing; nothing is filed half way.
 - Filing while the feed shows a work order open in VISUAL needs a second confirmation. The work orders are kept on the seal and in the packet log.
 - A reopen by Admin needs a reason and a signature. It marks the seal broken with who, when and why, and keeps the row and its data book. Filing again writes a new seal beside it.
-- The integrity job checks seals and asset hashes (phase 9).
+- The integrity job re-verifies every seal of a filed packet every night (see "Integrity job").
 
 ### PDFs, prints and exports (phase 7)
 
 - Every PDF and the CSV are built on the server. The browser only opens the URL. Reads need a signed-in session; every role that can open the packet may read them, PM included.
 - Controlled-copy stamp (S7): every exported page carries `Controlled copy - exported by Name (emp) <UTC time>`, the seal's first 12 hex or `not yet filed`, and ` - stamp <sig>`. The filed data book is the stored copy, served as it was built.
 - Printed-copy stamp (S8): every printed page carries `PRINTED COPY - <what> - printed by Name (emp) <UTC time> - <project> Lot <unit> dwg <drawing> R<rev> - verify revision before use` and the stamp signature, in a white box at the bottom of the page.
-- Stamp signature: 16 hex of HMAC-SHA256 with the seal key over the packet, the kind (a printed copy signs `print:<kind>`, so a print and an export of the same thing in the same minute never share a signature), the seal hash, the user and the time. A copy can be checked against the server later with `verify_stamp`; the Admin screen for it comes in phase 9.
+- Stamp signature: 16 hex of HMAC-SHA256 with the seal key over the packet, the kind (a printed copy signs `print:<kind>`, so a print and an export of the same thing in the same minute never share a signature), the seal hash, the user and the time. A copy can be checked against the server with `verify_stamp`; from phase 9 the Admin Stamp check tab does it from a pasted line or a PDF.
 - Logging (S7, S8): every print and export writes one `print_log` row (who, when, session, packet, unit or ITP, kind, what, pages, the stamp line, the stamp signature, the client address) and one audit row (`print` or `export`). `print_log` is append-only: the app role has SELECT and INSERT only, and a trigger refuses updates, deletes and truncate.
 - CSV formula guard (S6): a cell that starts with `=`, `+`, `-`, `@`, a tab or a CR gets a leading `'`. A cell with a quote, a comma or a line break is quoted with the quotes doubled.
 - PDF metadata: title, subject and keywords name the packet and the stamp signature; Author is PESCO and Producer is Packet Station.
@@ -177,6 +177,93 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - A locked session drops out of presence at once, and comes back on unlock. A revoked session drops out at once and its stream ends. A closed tab drops out when the stream ends, within the 25-second ping cycle.
 - Saved filters live in the browser's localStorage (`ps.flt`, `ps.proj`, `ps.pmtab`, `ps.pmper`, `ps.motion`). They hold filter choices only. The search text is never saved. Nothing in them is trusted by the server.
 
+### AD reconcile and role refresh (phase 9)
+
+- `ad-reconcile` runs every night at 01:30 site time, and when an Admin presses "Refresh from AD now" (a job request; the button answers 202 at once). It lists the PacketStation OU, reports mapped groups missing from it and extra groups in it, reads every mapped group's members through `LDAP_MATCHING_RULE_IN_CHAIN` (nested) in pages of 500, and upserts users on their objectGUID. AD is read first, then everything is written in one transaction.
+- A user disabled in AD, or gone from AD, is marked inactive and every session they have is ended (`ad_disabled`, `ad_gone`). A user still in AD but in no mapped group loses their roles and their sessions (`roles_lost`). Each ended session writes an `auth.session_ended` audit row with the reason and `by: ad-reconcile`.
+- Missing and duplicate employee IDs are flagged on the user (`directory_flags`) and listed on the Admin Profiles tab. A PIN sign-in by a duplicate employee ID is refused, as in phase 1.
+- Safety guard: the reconcile refuses and changes nothing when no mapped group is found in the OU (a wrong `LDAP_GROUP_BASE_DN`) or when every mapped group is empty while active users exist (PLAN.md open question 44). AD unreachable fails the run and changes nothing.
+- `session-refresh` runs every 15 minutes (the `roles_refresh_minutes` setting, 5 to 60). For every user with a live session it reads the account by objectGUID and its groups again, rebuilds the roles and each session's role snapshot, and ends the sessions of anyone disabled, gone or left with no role. AD unreachable makes the run degraded and ends nothing.
+- A request still re-reads the roles when its session's snapshot is older than the refresh interval, as in phase 1. Between the two, a person removed from a group loses the role within 15 minutes, and a person disabled in AD is signed out within 15 minutes.
+- Together these close the app's part of I6 (joiner, mover, leaver). The quarterly access review stays IT's.
+
+### Integrity job (phase 9)
+
+- `integrity` runs every night at 04:00 site time. It re-verifies every unbroken seal of a filed packet with the seal key (a seal under another key id is a finding), rehashes asset files (uploads of the last 7 days every night, every asset on Sundays) and records `hash_checked_at` and `hash_ok`, finds asset rows with no file and files under the assets volume with no asset row (older than an hour; reported, never deleted), and walks the audit hash chain.
+- Each finding is a row in the append-only `integrity_findings` (at most 500 per kind per run, the full count in the run). Any finding raises the `integrity_failure` alert, a security event that nobody can opt out of. A clean run clears it with a "Cleared" message. The Admin Security tab shows the last run and its findings.
+- A missing seal key makes the run degraded and leaves the seals unchecked, never "clean".
+- This closes M3 (tamper detection on records, files and the audit).
+
+### Notifications (phase 9)
+
+- Mail goes through `SMTP_HOST` (app-03.pesco.com, port 25) with STARTTLS, verified against `SMTP_CA_BUNDLE` (the PESCO CA). There is no plain-text fallback: a relay without STARTTLS, a TLS error or a wrong certificate fails the message, and the failure is recorded. There is no SMTP login.
+- Each message is addressed To `SMTP_FROM` (`[email protected]`). The recipients are in the envelope only, so nobody sees the list.
+- Recipients are resolved when the message goes. Group members must be enabled accounts with a mail address. An inactive user gets nothing. Every message writes one append-only `notifications_sent` row with each address, the user and why it was chosen.
+- Bodies are plain text from the server. No link carries a token or a session. Raising an event never fails the action that raised it.
+- The test stack runs with `NOTIFY_DRY_RUN=true`: messages are recorded as `dry_run` and not sent. A test send from the Admin tab always sends. Subjects start `[Packet Station]` on production and `[Packet Station TEST]` on any other stack.
+- The security event types (`security_alert`, `signin_burst`, `integrity_failure`) cannot be opted out of on Account settings, and an Admin cannot exclude anyone from them.
+
+### Security alerts (phase 9)
+
+The Admin Security tab lists, for the last 24 hours by default (up to 168):
+
+| Kind | From |
+|---|---|
+| Account cooldowns (`lockout`) | `auth.lockout` audit rows, written when an account goes over its failure limit |
+| Wrong PINs (`pin_failures`) | Three or more wrong PINs for one person in the window, at any step |
+| Wrong unlock or signature (`stepup_wrong_pin`) | A wrong PIN or password at unlock or at a signature |
+| Denied actions (`denied`) | `policy.denied` audit rows: any 403 after the person was known (method, path and code; never a body) |
+| Sign-in bursts (`signin_burst`) | `auth.signin_burst` audit rows: an address over its limit, or failed sign-ins across accounts in 5 minutes at or over the burst threshold |
+| Admin changes (`admin_change`) | `admin.*` audit rows |
+| IT edits (`it_edit`) | Audit rows with an IT reason |
+| Integrity (`integrity`) | `integrity_findings` rows |
+
+Account cooldowns, a PIN disabled, a session ended after wrong PINs and sign-in bursts are also mailed (`security_alert`, `signin_burst`).
+
+### Security settings (phase 9)
+
+The Admin Security tab overrides these environment defaults, stored in the `security` settings row with compare and set, audited with before and after. A value out of bounds is refused; a stored value out of bounds is ignored and the default applies. The app reads them per request, cached for at most 30 seconds.
+
+| Setting | Default | Bounds |
+|---|---|---|
+| `session_lock_minutes` (lock after idle) | 3 | 1 to 30 |
+| `session_idle_minutes` (sign out after idle) | 15 | 1 to 60 |
+| `session_desk_minutes` (Admin and Clerical idle) | 60 | 5 to 240 |
+| `session_password_hours` | 12 | 1 to 24 |
+| `session_pin_hours` | 10 | 1 to 24 |
+| `roles_refresh_minutes` | 15 | 5 to 60 |
+| `signin_account_limit` (failed sign-ins per account before a cooldown) | 5 | 1 to 15 |
+| `signin_account_window_minutes` | 5 | 1 to 60 |
+| `signin_account_cooldown_minutes` | 10 | 0 to 120 |
+| `signin_ip_limit` (failures per address) | 30 | 5 to 200 |
+| `signin_ip_window_minutes` | 5 | 1 to 60 |
+| `signin_burst_threshold` (failures across accounts in 5 minutes that raise an alert) | 20 | 5 to 200 |
+
+The account limit is capped at 15 so the app always stops before AD's own lockout at 20 attempts in 5 minutes. The PIN limit (5) and the step-up limit (3) stay environment settings.
+
+### Admin (phase 9)
+
+- Every Admin endpoint needs PS-Admins in a password session. Every Admin change is audited with before and after. Tests try every endpoint as every role.
+- Roles still come only from AD. The Admin edits departments (name, stations), profiles (stamp, profile active, station limits), checklists, the security settings and the notification rules. None of these grants a role. Turning a profile off hides the person from pickers and changes no right.
+- Department codes and short names never change (a trigger, and 422 on the API). Stations are never deleted.
+- The access log CSV has the formula guard, at most 50,000 rows, and is logged in `print_log` and the audit.
+- The stamp check recomputes the stamp signature with the seal key and compares the text with the print log (S7, S8). It reads a pasted line or a PDF of up to 25 MB.
+
+### Backups (phase 9)
+
+- `scripts/backup/ps-backup.sh` runs as root from a systemd timer at 02:00 Denver time, an hour before the Proxmox Backup Server run of the whole VM. It dumps each stack with `pg_dump -Fc` inside the postgres container over the local socket (no password read or passed), checks each dump with `pg_restore --list`, keeps 7, and writes `last-success` or `last-failure`.
+- The roles are dumped with `--no-role-passwords`. A password hash in a file is an offline cracking target, and the file lands in PBS. The restore script refuses a globals file that holds a password hash. The role passwords stay in the stack secrets.
+- Dumps and globals are mode 0600. The production folder is `root:10001` 0750, so only root and the worker (read-only mount) see it.
+- The worker's `backup-check` job raises `backup_stale` when the newest dump is missing, empty, older than 26 hours, or followed by a failure.
+- The restore was rehearsed on the test stack on 2026-10-21 into a separate database: every table's row count matched and all 189 asset files matched their recorded sha256 ([DEPLOY.md](DEPLOY.md), "Restore rehearsal 2026-10-21"). This closes O1 and O2 for the first rehearsal. The quarterly restore test is IT's.
+
+### Dependency scanning (phase 9)
+
+- Every CI run scans the dependencies. `pip-audit` (pinned, in its own virtual environment) checks every pin in `server/requirements.txt` without resolving and fails on any known vulnerability, which is stricter than high or critical. `npm audit --audit-level=high` fails on a high or critical finding in the frontend, dev dependencies included.
+- `visualkit` and `mfileskit` come from PESCO's own index and are not audited. Their dependencies that we install are pinned in `requirements.txt` and are.
+- No finding is ignored. If one ever has no fix, the way out is `--ignore-vuln <ID>` with a comment and a date.
+- The first scan (2026-10-21) found `pypdf` 6.21.0 (GHSA-5hx5-vpf9-5wcv) and `certifi` 2026.7.22 (PYSEC-2026-4320). Both were bumped. This closes S3.
+
 ### Headers (phase 0)
 
 - The app sets HSTS, CSP and the other security headers itself. NPM's own HSTS is off.
@@ -203,7 +290,7 @@ The IDs come from Part 3 of the old SECURITY.md. The status follows the approved
 | I3 | Closed for "PIN on the server", phase 1; device certificate deferred |
 | I4 | Closed, phase 1 |
 | I5 | Superseded (no in-app admin grant); separate IT admin accounts remain IT's |
-| I6 | Largely closed, phase 9 (groups plus reconcile); quarterly review IT's |
+| I6 | App part closed, phase 9 (roles only from AD groups; the nightly `ad-reconcile` marks disabled and gone users inactive, ends their sessions and flags employee ID problems; `session-refresh` re-reads every live session's groups and account state every 15 minutes and ends the sessions of anyone disabled, gone or left with no role); quarterly access review IT's |
 | I7 | Closed, phase 1 |
 | I8 | App part closed, phase 1; kiosk lock policy IT's (Intune or GPO) |
 | A1, A2, A3 | Closed, phase 2 (server policy on every call, workflow steps as compare-and-set, validation and limits on the server) |
@@ -214,17 +301,17 @@ The IDs come from Part 3 of the old SECURITY.md. The status follows the approved
 | D6, D7 | Deferred (retention period and customer copy need a decision from Seth and Q.C.) |
 | M1 | Closed, phase 2 (append-only audit and packet log written only by the server; every packet write carries both) |
 | M2 | Closed, phase 7 (filing seal is HMAC-SHA256 with the server seal key from a Docker secret; a reopen keeps the broken seal and its data book) |
-| M3 | Closed, phase 9 |
+| M3 | Closed, phase 9 (the nightly `integrity` job re-verifies seals, rehashes asset files, finds missing and orphan files and walks the audit hash chain; findings are kept in the append-only `integrity_findings` and raise `integrity_failure`, which nobody can opt out of) |
 | M4, M5 | Deferred (logs to stdout now; SIEM shipping is IT's) |
 | S1 | Closed, phase 0, tuned phase 10 |
 | S2 | Closed, phase 1 |
-| S3 | Closed, phase 9 (CI scanning) |
+| S3 | Closed, phase 9 (CI scanning on every run: `pip-audit` over every pin in `server/requirements.txt` fails on any known vulnerability; `npm audit --audit-level=high` fails on a high or critical one; the first scan bumped pypdf and certifi) |
 | S4 | Deferred to IT (pentest before go-live) |
 | S5 | Closed, phase 2 (React escapes output; server messages are plain text); phase 7 adds escaping in the PDF builders |
 | S6 | Closed, phase 7 (the CSV formula guard on every export) |
 | S7 | Closed, phase 7 (the controlled-copy stamp on every exported PDF, signed with the seal key; PDF metadata; every export logged in `print_log` and the audit) |
 | S8 | Closed, phase 7 (the PRINTED COPY stamp with "verify revision before use" on every printed page, signed with the seal key; every print logged in `print_log` and the audit). PLAN.md lists S8 under phase 5, but the print stamps come with the PDFs in phase 7, so S8 closes there. |
-| O1, O2 | Closed, phase 9 (pg_dump timer plus PBS; one rehearsed restore); quarterly restore IT's |
+| O1, O2 | Closed, phase 9 (nightly `pg_dump` of each stack at 02:00 Denver time from a systemd timer, checked with `pg_restore --list`, 7 kept, before the 03:00 PBS run of the VM; the `backup-check` job alerts on a missing, empty, stale or failed backup; restore rehearsed on the test stack on 2026-10-21 with every row count and asset hash matching); quarterly restore test IT's |
 | O3, O4 | Closed, phase 10 |
 | O5 | Superseded (no migration; M: retired read-only) |
 
@@ -232,7 +319,7 @@ The brief's wording for the identity IDs, which the table above follows:
 
 - **I4 and I7** (PIN lockout, PIN signatures verified on the server) are closed by this build.
 - **I8** is closed for the app's part. The kiosk device lock policy in I8 is IT's, via Intune or GPO.
-- **I6** (joiner, mover, leaver) is largely carried by the AD groups plus the nightly reconcile. The quarterly access review is still IT's.
+- **I6** (joiner, mover, leaver) is closed for the app's part by the AD groups, the nightly reconcile and the 15-minute role refresh. The quarterly access review is still IT's.
 - **I3** is closed for "PIN checked on the server" and open for "Intune-managed, device certificate". Registered devices use a device cookie for now.
 - **I1** is partly closed by AD sign-in. There is no SSO yet.
 - **I5** is superseded. No in-app admin grant exists, so second-admin approval has nothing to approve. Separate IT admin accounts with MFA remain IT's.