Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

README.md

@@ -6,9 +6,9 @@ This repo is the rebuild of Packet Station as a server application: Docker Compo
 
 ## Status
 
-**Rebuild in progress. Phase 8 done: the Project Management screen with its reports and exports, the find bar on every home screen with server-side filters and the list view, presence in packets, the motion switch, and the data book checked page by page against the old build.**
+**Rebuild in progress. Phase 9 done: the Admin page completed (departments, profiles with the CSV import, checklists, security settings and alerts, the access log, who is in, the stamp check), the nightly AD reconcile and the 15-minute role refresh, the integrity job, notifications by email with rules per event, nightly database backups with a rehearsed restore, and dependency scanning in CI.**
 
-People can sign in, set up projects and units, make hand drafts, attach a drawing, and send a packet to the warehouse, a department or Q.C. The worker reads released work orders from VISUAL on a schedule, makes and updates drafts, opens change orders on packets already in the shop, looks drawings up in M-Files and attaches them at the rev VISUAL calls. Clerical sees it all on the From VISUAL panel and in the Flagged and Changes to approve columns; Admin sets the feed up on the Feed tab. Warehouse works its task board (waiting, being pulled, on hold, pulled, released), pulls parts line by line with heat numbers, shorts, receipts and MTRs, and releases to the supervisor once every line is pulled or short and every pulled line has a heat. A supervisor assigns the packet to a station and a welder, the welder starts the clock, marks welds on the drawing, places the heat table, finishes the checklist and submits with a PIN signature, and the supervisor sends it to Q.C. or routes it to another department with the marks carried. Department supervisors verify change orders with a signature. Q.C. inspects the packet on its own tab (the check items, a visual result per weld, NDE, hardness, notes), accepts it to the data book, to Shipping or on to another department with a PIN signature, or rejects it back to the supervisor with a reason or an NCR number. Q.C. keeps project folders of documents per project and unit (ITP, WPS / PQR, procedures, certificates) where a new revision supersedes the old one and nothing is deleted, adds reports to a packet for the data book and takes them out with a reason, and works each unit's digital ITP from the Q.C. template through line initials, N/A, rejects with NCR numbers, section block signatures and a sign-off. Open hold and witness points show as a warning on the unit's packets. Clerical confirms the job closed in VISUAL (the feed's closed state and any open work orders shown beside it) and files the packet with a signature. Filing seals the packet with an HMAC and builds the manufacturing data book on the server, stored on the seal; filing past open work orders needs a second confirmation. IT can reopen a filed packet with a reason and a signature, which keeps the broken seal and its data book. Anyone who can open a packet can view or download the data book, the job traveler with the work order QR codes, the ITP PDF and the traceability CSV, and print a stamped copy of the traveler, the drawings with the heat or weld map, the drawings without markup, the VOID sheets, the data book or any document. Every copy carries a stamp signed with the seal key, and every print and export is logged. The ITP sign-off files the signed ITP PDF. Every packet write goes through the server policy. The old build on `M:\Production\Packet Station` stays the live system until go-live. The new system starts empty; nothing is migrated from M:.
+People can sign in, set up projects and units, make hand drafts, attach a drawing, and send a packet to the warehouse, a department or Q.C. The worker reads released work orders from VISUAL on a schedule, makes and updates drafts, opens change orders on packets already in the shop, looks drawings up in M-Files and attaches them at the rev VISUAL calls. Clerical sees it all on the From VISUAL panel and in the Flagged and Changes to approve columns; Admin sets the feed up on the Feed tab. Warehouse works its task board (waiting, being pulled, on hold, pulled, released), pulls parts line by line with heat numbers, shorts, receipts and MTRs, and releases to the supervisor once every line is pulled or short and every pulled line has a heat. A supervisor assigns the packet to a station and a welder, the welder starts the clock, marks welds on the drawing, places the heat table, finishes the checklist and submits with a PIN signature, and the supervisor sends it to Q.C. or routes it to another department with the marks carried. Department supervisors verify change orders with a signature. Q.C. inspects the packet on its own tab (the check items, a visual result per weld, NDE, hardness, notes), accepts it to the data book, to Shipping or on to another department with a PIN signature, or rejects it back to the supervisor with a reason or an NCR number. Q.C. keeps project folders of documents per project and unit (ITP, WPS / PQR, procedures, certificates) where a new revision supersedes the old one and nothing is deleted, adds reports to a packet for the data book and takes them out with a reason, and works each unit's digital ITP from the Q.C. template through line initials, N/A, rejects with NCR numbers, section block signatures and a sign-off. Open hold and witness points show as a warning on the unit's packets. Clerical confirms the job closed in VISUAL (the feed's closed state and any open work orders shown beside it) and files the packet with a signature. Filing seals the packet with an HMAC and builds the manufacturing data book on the server, stored on the seal; filing past open work orders needs a second confirmation. IT can reopen a filed packet with a reason and a signature, which keeps the broken seal and its data book. Anyone who can open a packet can view or download the data book, the job traveler with the work order QR codes, the ITP PDF and the traceability CSV, and print a stamped copy of the traveler, the drawings with the heat or weld map, the drawings without markup, the VOID sheets, the data book or any document. Every copy carries a stamp signed with the seal key, and every print and export is logged. The ITP sign-off files the signed ITP PDF. An Admin manages departments and stations, welder stamps and station limits (one at a time or from a CSV), the checklists, the session and lockout settings, and who gets which email. The worker reconciles users with AD every night, re-reads each signed-in person's groups every 15 minutes, checks seals, files and the audit chain every night, and checks that the nightly backup ran. Problems and events reach people by email through app-03. Every packet write goes through the server policy. The old build on `M:\Production\Packet Station` stays the live system until go-live. The new system starts empty; nothing is migrated from M:.
 
 | Area | Exists yet? |
 |---|---|
@@ -22,7 +22,7 @@ People can sign in, set up projects and units, make hand drafts, attach a drawin
 | Q.C. screen, inspection, Q.C. folders, packet documents, digital ITP, hold-point warnings | Yes (phase 6, verified in the CI stack on docker-1) |
 | Close and file, HMAC seal with the data book, reopen, data book, traveler and ITP PDFs, traceability CSV, prints and the print log | Yes (phase 7) |
 | Project Management screen (six tabs, periods, health, CSV and PDF reports), the find bar completed (search, filters, quick flags, Mine, Board and List, saved per browser), presence, motion, iPad layout | Yes (phase 8, verified in the CI stack on docker-1) |
-| Admin completion, notifications, backups | No (phase 9) |
+| Admin completion (departments, profiles, checklists, Security, access log, who is in, stamp check, Notifications), AD reconcile and role refresh, integrity job, email notifications, backups and restore, CI dependency scan | Yes (phase 9) |
 | Regression bench and go-live | No (phase 10) |
 
 ## Phases
@@ -82,7 +82,7 @@ docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
 
 Phase 2 adds `/api/projects`, `/api/units`, `/api/packets` (header, transitions, work orders, pages, documents, BOM, shorts, welds, heat tags, heat table, stage, checklist, clock, warehouse stage, Q.C., change orders, log, seal), `/api/assets` and `/_blob/{id}`. Every packet read carries `can`, `blocked` and `holder` from the policy. See [docs/POLICY.md](docs/POLICY.md) and [docs/SCHEMA.md](docs/SCHEMA.md).
 
-The `packetstation` command has `doctor`, `migrate`, `app`, `worker`, `job run <name>` (one job in the foreground), `fixtures use N` (switch the fixture state, never on prod), `ad-groups --list` (the 50 AD group names the app expects) and `ad-refresh` (create or refresh the users from the groups' members).
+The `packetstation` command has `doctor`, `migrate`, `app`, `worker`, `job run <name>` (one job in the foreground), `fixtures use N` (switch the fixture state, never on prod), `ad-groups --list` (the 50 AD group names the app expects) and `ad-refresh` (the AD reconcile once, in the foreground, with the same refusal guard as the nightly job).
 
 Phase 3 adds the worker (`packetstation worker`: APScheduler, the `visual-feed`, `drawing-lookup` and `housekeeping` jobs, settings re-read on every fire, run now through `job_requests`), the VISUAL and drawing cache tables (migration 0004), `/api/feed/*`, `/api/jobs/*`, `/api/admin/feed/*`, rev approval and hand uploads under `/api/packets/{id}`, and the synthetic fixture generator (`python -m tests.fixtures.generator --out DIR`) that CI feeds the worker with. See [docs/FEED.md](docs/FEED.md).
 
@@ -92,6 +92,10 @@ Phase 7 adds the close and file panel, `confirm_open_wos` on the file transition
 
 Phase 8 adds the Project Management screen (`/pm`) and its API under `/api/pm` (`overview`, `projects` with `expand=`, `aging`, `hours`, `quality`, `find`, `export.csv?table=<key>|jobs|all`, `report.pdf`), read by any signed-in role and writing nothing; every export is logged as `report exported` in the audit and the print log. The find bar params (`q`, `status`, `dept`, `station_id`, `welder_id`, `flags`, `flag`, `sort`, `mine_screen`, `mine_station_id`, `facets`) are read by `GET /api/packets` and by the shop, stall, warehouse and Q.C. boards, so the server applies every filter; the KPI numbers follow the project pick only. Presence over `/api/events` now carries the role and drops locked sessions (session events `locked` and `unlocked`). No new table or migration. See [docs/POLICY.md](docs/POLICY.md) "Phase 8: Project Management, find bar, presence".
 
+Phase 9 adds, under `/api/admin`: `departments` (list, `PATCH /departments/{code}`, `POST /departments`), `PATCH /users/{id}/profile`, `POST /users/import` (CSV preview and apply), `checklists`, `security/settings`, `security/alerts`, `security/integrity`, `access-log` and `access-log.csv`, `presence`, `stamp-verify` and `stamp-verify/pdf`, `directory-flags`, `ad/reconcile`, and `notifications` (rules, group search, test send, sent log). `POST /api/admin/ad/refresh` now asks the worker for an `ad-reconcile` run and answers 202. Any signed-in person reads `GET /api/account/notifications`, and opts out of an event type with `PUT /api/account/notifications/{event_type}` (password session; security events cannot be turned off). The Admin page gains the tabs Departments, Profiles, Checklists, Security, Access log, Who is in, Stamp check and Notifications, and Account settings gains a Notifications section. The worker gains six jobs: `ad-reconcile` (01:30), `session-refresh` (every 15 minutes), `integrity` (04:00), `backup-check` (07:15), `notify-send` (every minute) and `notify-digest` (every 5 minutes). Migration 0010 adds the notification tables, the integrity findings and the profile switch. See [docs/POLICY.md](docs/POLICY.md) "Phase 9: Admin, notifications, reconcile" and [docs/FEED.md](docs/FEED.md) "Phase 9 jobs".
+
+Backups run on the host, outside the app: `scripts/backup/ps-backup.sh` (nightly `pg_dump` of each stack from a systemd timer at 02:00 Denver time, 7 kept, with markers the worker checks) and `scripts/backup/ps-restore.sh` (restore into a fresh database, copy assets back, check hashes, compare row counts). [scripts/backup/README.md](scripts/backup/README.md) and [docs/DEPLOY.md](docs/DEPLOY.md) "Backup" have the install steps and the restore rehearsal of 2026-10-21.
+
 CI stack with the Samba domain controller and the Playwright scenario, on a Docker host:
 
 ```bash
@@ -101,6 +105,8 @@ docker compose -p packetstation-ci --env-file deploy/ci/ci.env -f deploy/compose
 docker compose -p packetstation-ci --env-file deploy/ci/ci.env -f deploy/compose.yaml -f deploy/compose.ci.yaml down -v
 ```
 
+CI scans the dependencies on every run: `pip-audit` over `server/requirements.txt` fails on any known vulnerability, and `npm audit --audit-level=high` fails on a high or critical one.
+
 Server tests that need PostgreSQL run when `TEST_PG_SUPERUSER_URL` is set; the ones that need the domain controller run when the `TEST_LDAP_*` variables and `TEST_DC_USERS_FILE` are set. See [deploy/samba/README.md](deploy/samba/README.md).
 
 ## Docs
@@ -108,10 +114,10 @@ Server tests that need PostgreSQL run when `TEST_PG_SUPERUSER_URL` is set; the o
 - [docs/BRIEF.md](docs/BRIEF.md): the brief this rebuild follows.
 - [docs/PLAN.md](docs/PLAN.md): the approved phased plan and open questions.
 - [docs/SECURITY.md](docs/SECURITY.md): old layers and where they live now, the new controls, the requirement ID table.
-- [docs/DEPLOY.md](docs/DEPLOY.md): host, stacks, secrets, NPM proxy host, doctor, backups.
-- [docs/FEED.md](docs/FEED.md): the worker, the VISUAL feed, the packet rules, the fingerprint, the drawing decision table, the flags and the feed settings.
+- [docs/DEPLOY.md](docs/DEPLOY.md): host, stacks, secrets, NPM proxy host, doctor, notifications, backups and the restore rehearsal.
+- [docs/FEED.md](docs/FEED.md): the worker and its jobs, the VISUAL feed, the packet rules, the fingerprint, the drawing decision table, the flags and the feed settings.
 - [docs/POLICY.md](docs/POLICY.md): who may do what, the preconditions, the order of checks, concurrency, and the tightenings against the old build.
-- [docs/SCHEMA.md](docs/SCHEMA.md): the database schema (phases 1 to 7).
+- [docs/SCHEMA.md](docs/SCHEMA.md): the database schema (migrations 0001 to 0010).
 - [docs/GO-LIVE.md](docs/GO-LIVE.md): go-live procedure (phase 10).
 - [docs/PAPER-FALLBACK.md](docs/PAPER-FALLBACK.md): what the shop does when the server is down (phase 10).
 - [CHANGELOG.md](CHANGELOG.md): what changed.