Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/PLAN.md

@@ -42,7 +42,7 @@ frontend/ (Vite + React + TS; src/theme, src/screens/{clerical,warehouse,supervi
 tests/server/ (pytest: policy matrix, merge rules, packet rules, feed fixtures, auth)
 tests/e2e/ (Playwright scenarios, synthetic fixtures)  tests/fixtures/generator.py
 deploy/ compose.prod.yml compose.test.yml compose.ci.yml  npm/ (proxy host notes)  samba-ad/ (CI DC seed)
-scripts/ backup.sh restore.sh packetstation-backup.service/.timer
+scripts/backup/ ps-backup.sh ps-restore.sh packetstation-backup.service packetstation-backup.timer backup.conf.example
 .gitea/workflows/ci.yml
 ```
 
@@ -115,6 +115,7 @@ Each phase ends with passing tests, docs updated (README, CHANGELOG, SECURITY.md
 - Notifications tab (rules per event type, recipients: group, user, mailbox, dynamic department Supervisor group; immediate or digest; exclusions; test send; sent log; dry run on test), SMTP via app-03 with STARTTLS and CA verify, subject prefixes, seed to PS-Admins.
 - scripts/backup.sh + systemd timer (America/Denver 02:00), 7 dumps, marker; worker backup-check job; restore script; restore rehearsed on the test stack and written up.
 - Closes: I6 (app part), M3, O1, O2 (first rehearsal), S3 (pip-audit and npm audit in CI).
+- Done 2026-10-21. The eight Admin tabs, the Account notifications section, the six worker jobs, the notify package with SMTP over STARTTLS, migration 0010, the backup and restore scripts with the systemd units, and the CI dependency scan are built and tested. The backup and the restore were rehearsed by hand on the test stack on docker-1 (DEPLOY.md, "Restore rehearsal 2026-10-21"). The first scan found two pins to bump (pypdf 6.21.1, certifi 2026.10.19). The separate `notification_state` table was not needed: the open `notification_events` row is the state. The timer is installed by Spencer with sudo (DEPLOY.md, "Backup"). Decisions taken are open questions 43 to 50.
 
 ### Phase 10: Regression bench, hardening, go-live (medium)
 - Rewrite the 38 scenarios against the new UI with synthetic fixtures; run in Gitea Actions against compose.ci.yml. Policy matrix, merge rules and packet-rule tests already run from phase 2 and 3.
@@ -169,7 +170,7 @@ Conventions: identity bigints except users (uuid = AD objectGUID); Postgres enum
 - **Change orders:** change_orders (co_no, origin visual|drawing|manual, review pending|approved|rejected, minor, hold, held_by, issued_*, approved_*, rejected_*, vis jsonb, waiting jsonb, staged jsonb, target_fp; one pending visual CO per packet), co_items, co_required_depts, co_verifications (insert-only, signature required). View v_open_cos ports openCOs.
 - **Documents:** assets (uuid, sha256, size, mime, original_name, storage_key, source, source_ref jsonb, uploaded_by, packet_id); packet_docs; qc_docs (project, unit, type, rev, auto, superseded_by); itps (unit unique, template jsonb frozen, status, signoff_*, reopen_*), itp_lines, itp_rejects (append-only, ncr), itp_blocks.
 - **Audit, settings, jobs:** audit_events (append-only, prev_hash and row_hash chain, user, session, role_snapshot, action, subject, packet_id, detail jsonb, it_reason, ip); settings (key, value jsonb, version); job_requests, job_runs (trigger, requested_by, status ok|degraded|failed|skipped|abandoned, counts jsonb, errors jsonb, build_version, visual_target), job_schedule, worker_heartbeat.
-- **Notifications (per the brief):** notification_rules (event_type, enabled, delivery immediate|digest, digest_time); notification_recipients (rule_id, kind group|user|mailbox|dept_supervisor, group_dn, user_id, address); notification_exclusions (rule_id, user_id); notification_outbox; notifications_sent (event, recipients as resolved, subject, result); notification_state (dedupe and recovery).
+- **Notifications (per the brief):** notification_rules (event_type, enabled, delivery immediate|digest, digest_time); notification_recipients (rule_id, kind group|user|mailbox|dept_supervisor, group_dn, user_id, address); notification_exclusions (rule_id, user_id); notification_outbox; notifications_sent (event, recipients as resolved, subject, result). The planned notification_state (dedupe and recovery) was not needed: the open notification_events row under its unique index is the dedupe, and a recovery row points back at it (phase 9).
 - **VISUAL and drawing caches:** visual_batches; visual_rows (typed key wo, lot, split, sub, seq int; all contract fields; row_hash; first, last and gone batch); visual_materials (plus piece int, mat_hash); visual_orders and visual_order_links (base or project, rank, lines, top_jobs, order_date); visual_labor; visual_row_changes and visual_material_changes (per batch, added|changed|gone); visual_ignored_wos (go-live list with pulled_in_*); visual_resource_map; drawing_lookups (dwg PK, checked_at, ok, rev, name, object and file ids and versions, base, draft_rev, draft_where, in_drafting, note, failing_since, asset_id, visual_rev_seen).
 - **JSONB only here:** foreign-shaped feed payloads (source, visual, partspecs, vis, waiting, staged, raw), template-driven answers (checklist, checks, itp template, settings), and frozen snapshots (roles_snapshot, qc_history, audit detail, job counts). Everything the policy or the boards filter on is a real column.
 - **Versioning:** packets.version goes up in every transaction that touches the packet or a child; it is the ETag and the NOTIFY payload. Child rows carry row_version. Every write takes `SELECT ... FOR UPDATE` on the packet first, which replaces the old 8-second lease.
@@ -204,7 +205,7 @@ Conventions: identity bigints except users (uuid = AD objectGUID); Postgres enum
 ## Deployment and CI (phase 0, 9, 10; docs/DEPLOY.md)
 
 - **Dockerfile:** frontend stage on node:22-bookworm-slim; builder on python:3.12-slim-bookworm with a venv, `pip install --require-hashes --index-url <gitea>/simple/ -r server/requirements.lock` then the package; runtime with the Microsoft Debian 12 repo (msodbcsql18, unixodbc, tini, tzdata), non-root uid 10001, `PS_BUILD` label. entrypoint exports MFILES_PASSWORD from the secret file and dispatches app (uvicorn with no proxy headers; the app's TrustedProxyMiddleware re-resolves npm_app_1 every 60 s), worker, migrate, doctor, job run, fixtures.
-- **Compose:** base file with postgres (16-bookworm by digest, `backend` internal network only, initdb creates ps_migrate, ps_app and ps_worker from secrets), one-shot migrate, app and worker (backend plus egress, assets volume, CA bundle bind, secrets, healthchecks, restart unless-stopped, json-file logs, 1 GB limits; worker also mounts the backup folder read-only). Proxy overlay joins app to npm_default with `APP_ALIAS`. CI overlay adds the Samba DC, fixture sources and dry-run mail. Env files per stack in /etc/packetstation; secrets root:10001 0440. Deploy: `docker compose -p packetstation --env-file ... -f compose.yaml -f compose.proxy.yaml up -d`.
+- **Compose:** base file with postgres (16-bookworm by digest, `backend` internal network only, initdb creates ps_migrate and ps_app from secrets; the app and the worker both log in as ps_app, the worker with application name `ps-worker`), one-shot migrate, app and worker (backend plus egress, assets volume, CA bundle bind, secrets, healthchecks, restart unless-stopped, json-file logs, 1 GB limits; worker also mounts the backup folder read-only). Proxy overlay joins app to npm_default with `APP_ALIAS`. CI overlay adds the Samba DC, fixture sources and dry-run mail. Env files per stack in /etc/packetstation; secrets root:10001 0440. Deploy: `docker compose -p packetstation --env-file ... -f compose.yaml -f compose.proxy.yaml up -d`.
 - **NPM proxy host** (recorded in deploy/npm-proxy-host.md): http to the alias on 8000, Force SSL, HTTP/2, NPM HSTS off (the app sends it), `client_max_body_size 30m`, a `location /api/events` block with buffering off and 3600 s timeouts using the `$server` variables so the alias is re-resolved.
 - **Doctor:** config and stack guard, `visualkit doctor`, `mfileskit doctor`, DB connect and migrations at head and an audit UPDATE that must fail, LISTEN/NOTIFY round trip, assets volume and free space, backup folder, LDAP TLS bind reading memberOf, employeeID and userAccountControl with the 50-group count and one IN_CHAIN query on both controllers, SMTP STARTTLS verify, tzdata.
 - **Backups:** scripts/backup/ps-backup.sh per stack via `docker exec pg_dump -Fc` plus globals, verified with `pg_restore --list`, 7 kept, last-success and last-failure markers in /var/backups/packetstation/<stack>/; systemd timer `OnCalendar=*-*-* 02:00:00 America/Denver`; restore script and rehearsal recorded in DEPLOY.md.
@@ -276,13 +277,26 @@ From phase 8 (2026-10-18), for Seth to veto. The phase 8 design numbered these 3
 41. The find bar filters are applied by the server on every board and list, so a board never fetches more than it shows. Default: the KPI numbers follow the project pick only, not the other filters. Phase 8 (2026-10-18) applied the default.
 42. Mine is decided from the person's AD roles and the screen they are on. Default: the stall's Mine needs the picked station and shows nothing without one. Phase 8 (2026-10-18) applied the default.
 
+From phase 9 (2026-10-21), for Seth or Spencer to veto:
+
+43. The daily digest: default one message per address per digest time, listing every due event of every digest rule that reaches that address, not one message per rule. An address that already got an event is not sent it again after a partial failure. Phase 9 (2026-10-21) applied the default.
+44. The AD reconcile guard: default the reconcile refuses and changes nothing when no mapped group is in the OU (a wrong `LDAP_GROUP_BASE_DN`) or when every mapped group is empty while active users exist. The run is recorded as failed with the reason in its errors, and an `ad_reconcile_report` condition "AD reconcile did not run" goes to the rule once, cleared by the next good run. A wholesale sign-out needs a person to look first. Phase 9 (2026-10-21) applied the default.
+45. `profile_active`: default an inactive profile only hides the person from the welder picker and the worker filter. It does not change their roles or rights, and an assign made through the API still checks only the role. AD decides who may do what. Phase 9 (2026-10-21) applied the default.
+46. Account opt-out: default any person may opt out of an event type on Account settings, in a password session, except the security event types (security alert, sign-in burst, integrity failure). An Admin cannot leave anyone out of those either. Phase 9 (2026-10-21) applied the default.
+47. Version deployed: default the event is raised by the first `notify-send` run in each worker process when `PS_BUILD` differs from the build last announced (settings row `notify.last_build`), not at worker start, since the scheduler may start before PostgreSQL answers. A restart on the same build sends nothing. Phase 9 (2026-10-21) applied the default.
+48. "Refresh from AD now": default the button asks the worker for an `ad-reconcile` run and answers 202 at once, instead of running the reconcile inside the web request. The Admin sees "Refresh requested" and the report when the run finishes. Phase 9 (2026-10-21) applied the default.
+49. New departments: default only shop departments can be added in the app (422 `support_fixed` for a support department). The support roles are fixed in the `role_kind` enum and the group map, so a new support department would map to no role. Phase 9 (2026-10-21) applied the default.
+50. The profile CSV `stations` cell: default the cell is the person's whole list of station limits across departments and replaces every limit they had; `-` clears them all; an empty cell leaves them. Phase 9 (2026-10-21) applied the default.
+51. A direct welder assign: default it does not check `profile_active`, so an Admin's inactive switch hides a person from the picker but a typed assign still works. Phase 9 (2026-10-21) applied the default; say if an inactive profile should be refused there too.
+52. A checklist item added while a stage is open: default it applies to that open stage too, because completeness reads the live template, so the item must be ticked before submit. Phase 9 (2026-10-21) applied the default.
+
 ## Verification
 
 - Every phase: `pytest` server suite, `npm test` and `npm run build` for the frontend, `docker compose -f deploy/compose.ci.yml up` with the Samba DC and the Playwright scenarios for that phase, `packetstation doctor` inside the image.
 - Phase 3: a recorded DEV feed run compared with the packets the old build made for the same window (counts of drafts, units, projects; same keys), run from the developer machine against VISUAL DEV only.
 - Phase 7: server tests for close and file against the policy oracle, the open work order confirm, the seal with its data book, reopen and refile, a failed data book build, the data book sections and index, the traveler, the ITP PDF, every print kind with its stamp and `print_log` row, the CSV formula guard and the stamp signature; frontend tests for the close and file panel, the print menu, the prints list and the ITP buttons; the Playwright close scenario (close, PDFs and CSV, prints, file with the seal, reopen and refile, the signed ITP PDF) in the CI stack on docker-1.
 - Phase 8: the data book of a synthetic filed packet compared page by page with a data book exported from the old build on the same fixture (moved from phase 7). Done in phase 8 (2026-10-18); the page table, the fixes and what is left are in the phase 8 comparison note.
-- Phase 9: restore rehearsal on the test stack from a pg_dump and an assets copy.
+- Phase 9: restore rehearsal on the test stack from a pg_dump and an assets copy. Done 2026-10-21: every table's row count and all 189 asset hashes matched (DEPLOY.md, "Restore rehearsal 2026-10-21").
 - Phase 10: the full 38-scenario bench green in Gitea Actions.
 
 ## Host layout on docker-1