Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/POLICY.md

@@ -823,7 +823,7 @@ The packet page shows who else has the packet open: `Also in this packet now: <N
 - A locked session drops out at once. Lock and unlock publish a `session` event (`locked`, `unlocked`) from `POST /api/auth/lock`, `POST /api/auth/unlock` and the server's idle lock. Unlock brings the person back.
 - A revoked session drops out at once, and its stream ends.
 - Closing the tab or losing the network drops the person out when the stream ends, within the stream's ping cycle (25 seconds). No other idle rule is needed.
-- The server also keeps who is on each screen for the Admin view of phase 9. No endpoint reads it yet.
+- The server also keeps who is on each screen. Phase 9's Admin "Who is in" tab reads it (`GET /api/admin/presence`).
 
 ### Tightenings against the old build
 
@@ -836,6 +836,116 @@ The packet page shows who else has the packet open: `Also in this packet now: <N
 | Each browser published its own presence, with the name, employee ID, role and stall, refreshed every 20 seconds. | The server builds presence from the session: display name and role, no employee ID. A locked or revoked session drops out at once. | Presence shows only what it needs, and a locked screen is not someone in the packet. | Yes |
 | Over target was worked out in the browser from the hours it had. | Over target is timer hours over the work-order target, on the server. Ticket hours are shown beside it and never decide it. | The station timer is the clock of record (open question 1). | Yes |
 
+## Phase 9: Admin, notifications, reconcile
+
+Nothing in the packet rule table changes in phase 9. The new endpoints are Admin endpoints and one Account endpoint. None of them writes a packet.
+
+### Who may use the Admin endpoints
+
+Every endpoint under `/api/admin` needs an Admin (PS-Admins) in a password session. An Admin in a PIN session is 403 `password_session_required`. Every other role is 403. An unsigned request is 401. Every write also passes the Origin check and the body limit. Tests try every endpoint as every role.
+
+| Endpoint | What | Audit |
+|---|---|---|
+| `GET /departments` | Every department, its AD groups (and whether each is in the OU; unknown when AD is down) and its stations | none |
+| `PATCH /departments/{code}` | Name, active, stations (rename, reorder, add, hide) | `admin.department_updated` |
+| `POST /departments` | A new shop department with its stations; answers the three group names IT must create | `admin.department_added` |
+| `GET /users`, `GET /users/{id}` | Gain the welder stamp, `profile_active`, station limits and the directory flags | none |
+| `PATCH /users/{id}/profile` | Welder stamp, `profile_active`, station limits per department | `admin.profile_updated` |
+| `POST /users/import` | The profile CSV, as a preview (`apply: false`) or applied | `admin.profile_updated` per changed person, `admin.profile_import` |
+| `GET /checklists`, `PUT /checklists` | The weld and plain station checklists | `admin.checklists_updated` |
+| `GET /security/settings`, `PUT /security/settings` | The session, lock and sign-in limit settings | `admin.security_settings_updated` |
+| `GET /security/alerts?hours=` | Security alerts for 1 to 168 hours | none |
+| `GET /security/integrity` | The newest finished integrity run and its findings | none |
+| `GET /access-log` | Sign-in attempts and identity audit rows for everyone | none |
+| `GET /access-log.csv` | The same, as CSV, at most 50,000 rows | `admin.access_log_exported`, a `print_log` row of kind `access-log-csv` |
+| `GET /presence` | Live sessions, locked or not, with the screen and packet | none |
+| `POST /stamp-verify`, `POST /stamp-verify/pdf` | Checks a stamp line, a 16-character code or a PDF (25 MB) against the print log | `admin.stamp_verified` |
+| `GET /directory-flags` | Active users with no employee ID, and employee IDs on more than one account | none |
+| `POST /ad/refresh` | Asks the worker for an `ad-reconcile` run; 202 (open question 48) | `ad.refresh_requested` |
+| `GET /ad/reconcile` | The newest finished reconcile run and its report | none |
+| `GET /notifications` | Mail settings, the subject prefix, every rule with its recipients, exclusions and last message | none |
+| `PUT /notifications/{event_type}` | One rule: on or off, delivery, digest time, its hours or days, recipients, exclusions | `admin.notification_rule_updated` |
+| `GET /notifications/groups?q=` | AD group search (2 to 100 characters; 503 when AD is down) | none |
+| `POST /notifications/{event_type}/test` | A test message, to the rule's recipients or to one address | `admin.notification_test` |
+| `GET /notifications/sent` | The sent log | none |
+
+Every Admin change is audited, with the state before and after where there is one.
+
+### Compare and set
+
+The checklists, the security settings and each notification rule carry a version. A save sends the version it read. When someone else saved first the answer is 409 `version_conflict` with the current value, and nothing is written. The screen reloads and says so.
+
+### Departments
+
+- The code and the short name (the slug) never change. They are in the AD group names. A trigger refuses a change, and a PATCH body that names `code` or `slug` is 422.
+- Stations are never deleted. Station limits, packets and history point at them. A station is hidden by setting it inactive. A station listed without an id is added. A station missing from the list is left as it is. Two stations of one department with the same name are refused (422 `duplicate_station`).
+- A new department needs a code of 2 to 4 capital letters or digits and a short name of 2 to 31 letters or digits starting with a letter, both unused (409 `code_taken`, `slug_taken`). Only shop departments can be added; a support department is 422 `support_fixed`, because the support roles are fixed (open question 49). The answer names the three groups IT must create. Until they exist the department has no people.
+- After a change the app rebuilds its group map from the departments table. The worker reloads it at the start of every reconcile and role refresh run.
+
+### Profiles
+
+- The welder stamp is 1 to 12 letters, digits or dashes, unique among users ignoring case (409 `stamp_taken`). Empty clears it. The packet's own welder stamp field still allows 10; the profile stamp is copied onto the packet as it is.
+- `profile_active` off hides the person from the welder picker and the worker filter. It does not change their roles or what they may do, and an assign sent to the API still checks only the role (open question 45). AD decides who may do what.
+- Station limits are set per department. A station must belong to that department. An empty list clears the department's limits.
+- The CSV import matches employee IDs as text among active users. `0042` is not `42`. Columns: `employee_id` (required), `welder_stamp`, `active`, `stations` (`<dept code>:<station name>` joined by `;`, the person's whole list; open question 50). An empty cell leaves the value. `-` clears it. Up to 2,000 rows and 1 MB. Each row lists its changes and its errors: unknown employee ID, an ID on more than one user, an ID twice in the file, a bad or taken stamp, a bad active value, an unknown department or station. Apply with any error is 422 with the same rows and changes nothing. Otherwise every row is applied in one transaction.
+
+### Checklists
+
+Both lists need at least one item and at most 40. A key is lower case letters, digits and underscores, starting with a letter, at most 30 characters, unique in its list. Group and text are not empty, at most 200 characters. The only placeholders are `{drawing}`, `{rev}` and `{wps}`. Answers on open stages are kept by key. A removed item drops out of new checklists only. An item added while a stage is open must also be ticked on that stage before submit, since the completeness check reads the live list.
+
+### Security settings
+
+Stored as the `security` settings row over the environment defaults. A value out of its bounds is refused (422 `bad_settings`), and a stored value out of bounds is ignored. The keys and bounds are in [SECURITY.md](SECURITY.md), "Security settings (phase 9)". A save applies in the app within 30 seconds. The PIN and step-up failure limits (5 and 3) stay environment settings.
+
+### Access log, presence and the stamp check
+
+- The access log shows every person's sign-in attempts (successful and failed) and their `auth.*`, `pin.*`, `device.*` and `sig.*` audit rows, as the Account access log shows a person their own. The address filter takes an address or a range. Request bodies and secrets are never in it.
+- Who is in shows live sessions (not revoked, not expired) seen within the idle minutes that apply to each. The screen and packet come from the event stream, matched by person.
+- The stamp check never trusts the text. It finds the 16-character signature, looks it up in `print_log`, recomputes it with the seal key from the logged row and compares the text with the logged stamp line. It answers genuine, does not verify (the signature or the text differs from the log), or not found.
+
+### Notifications
+
+- An Admin edits every rule. Recipients are any AD group under the base DN, a user, a mailbox, or (for the department events, `co_waiting` and `qc_reject`) the Supervisor group of the event's department. At most 50 recipients and 200 exclusions per rule.
+- An Admin may leave people out of a rule (an exclusion), except on the security event types: `security_alert`, `signin_burst` and `integrity_failure`. An exclusion on those is 422. Nobody can be left out of them (open question 46).
+- A test send goes at once, from the app, even on a stack in dry run. With an address it goes only there.
+- Recipients are resolved when a message goes, not when the rule is saved: group members (nested, enabled accounts with mail), a user's address while the user is active, a mailbox as typed. An exclusion drops the person by user and by their address.
+
+### Account notifications
+
+| Endpoint | Who | What |
+|---|---|---|
+| `GET /api/account/notifications` | Any signed-in, unlocked session, PIN included | Each enabled event type: whether it reaches the person and why (`user`, `mailbox`, `group <name>`, `Supervisor group of <code>`, `left out by an Admin`, `directory unavailable`), and whether they opted out or may |
+| `PUT /api/account/notifications/{event_type}` | The person, in a password session | `{opt_out}`. Writes or removes their own exclusion. 403 for a security event type. Audit `account.notification_opt_out` when something changed |
+
+A person's own opt-out is kept when an Admin saves the rule. An Admin's exclusion is not removed by the person.
+
+### Security events
+
+The three security event types (`security_alert`, `signin_burst`, `integrity_failure`) have no opt-out and no exclusion. An Admin can still turn the rule off or change its recipients, on the record.
+
+- `security_alert`: an account cooldown started, a PIN disabled after five wrong attempts, or a session ended after three wrong PINs or passwords at unlock or signature. One per account or person per 15 minutes.
+- `signin_burst`: one address over its failure limit, or failed sign-ins across accounts in 5 minutes at or over `signin_burst_threshold` (default 20). One per address, or one for the burst, per 30 minutes.
+- `integrity_failure`: the integrity job found a problem. Cleared by a clean run.
+
+### Audit rows added in phase 9
+
+- `policy.denied`: a 403 after a person was known. Method, path and code only, never a body or query string. One per person, method, path and code per minute.
+- `auth.lockout`: an account went into its cooldown. `auth.signin_burst`: an address over its limit, or a burst across accounts.
+- `auth.session_ended` with `{reason, by}` for every session the reconcile or the role refresh ends (`ad_disabled`, `ad_gone`, `roles_lost`).
+- The `admin.*` rows in the table above, `ad.refresh_requested` and `account.notification_opt_out`.
+
+### Tightenings against the old build
+
+| What the old build allowed | What the rebuild does | Why | Can be vetoed |
+|---|---|---|---|
+| The Admin page added a department with any typed code, shop or support, and changed a department's stations as a comma list, so a station could disappear. | Code and short name are checked and never change. Only shop departments are added. Stations are never deleted, only hidden. The three AD group names are shown before the save. | The code and short name are in the AD group names, and limits and packets point at stations. | Yes |
+| A welder stamp was set in the Admin user sheet, checked in the browser, 1 to 10 characters, not unique. | The stamp is checked on the server, 1 to 12 characters, unique among users. | Two welders with one stamp break the weld record. | Yes |
+| Users, levels and departments were edited in the Admin page. | Roles come only from AD groups. The Admin edits only the profile (stamp, profile active, station limits). Turning a profile off hides the person from pickers and changes no right. | AD is the one place access is granted. | No |
+| The security alerts panel and the integrity check ran in IT's browser when Admin was opened. | Both run on the server: alerts from the audit and the sign-in attempts, the integrity check every night in the worker, with an email alert. | A check that runs only when someone looks finds problems late. | No |
+| Nothing was emailed. | Events go by email through app-03 to rules an Admin sets. Security events cannot be turned off by the person or excluded by an Admin. | A security alert that can be muted is not one. | Yes |
+| The access log export ran in the browser. | The CSV is built on the server with the formula guard and logged in the print log and the audit. | The browser alone could skip the log. | No |
+| A stamped copy could be checked only by eye. | The stamp check recomputes the signature with the seal key and compares the text with the print log. | A copy is genuine only if the server says so. | No |
+
 ## Worker writes (phase 3)
 
 The worker has no session. It writes through the same policy as a person, as the system principal `Principal(user_id=None, system="visual-feed")`. Each write locks the packet row (`FOR UPDATE SKIP LOCKED`; a row locked by someone else is left for the next run), builds the packet state, runs `policy.check()`, makes its change, bumps `version`, and writes a `packet_log` row (by nobody, by name "VISUAL feed", role `system`), an audit row (`packet.<action>`, role `system`) and a change event. See [FEED.md](FEED.md) for when each write happens.