Phase 9: Admin completion, reconcile, notifications, integrity, backups

Admin: Departments (name and stations; add shows the three groups IT must
create), Profiles (welder stamp, station limits, active) with a CSV import
that previews and applies only when clean, Checklists (weld and plain),
Security (session, lockout and rate-limit settings with bounds, alerts for
the last 24 hours, the integrity result), Access log with filters and a
guarded CSV, Who is in, Stamp check, Notifications, and the employeeID
flags. Every Admin change is audited.

Worker: ad-reconcile (nightly and on "refresh from AD now"), session-refresh
(15 minutes), integrity (seals, asset hashes, missing and orphan files, the
audit chain), backup-check, notify-send and notify-digest. Notifications
route per event type to AD groups, users, mailboxes or the department
Supervisor group, immediately or in a daily digest, over STARTTLS verified
against the PESCO CA, with dry run on the test stack.

Backups: scripts/backup/ps-backup.sh and ps-restore.sh with a systemd timer
at 02:00 America/Denver. Restore rehearsed on the test stack 2026-10-21.
CI: pip-audit and npm audit. pypdf 6.21.1 and certifi 2026.10.19.

Migration 0010. Closes the app part of I6, M3, O1, O2 and S3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/FEED.md

@@ -1,6 +1,6 @@
 # Packet Station feed and drawings
 
-Status: phase 3. This file says how the worker reads VISUAL and M-Files and what it does with what it reads. The design is in [PLAN.md](PLAN.md), "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.
+Status: phase 9. This file says how the worker reads VISUAL and M-Files, what it does with what it reads, and the other jobs it runs (phase 9: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications). The design is in [PLAN.md](PLAN.md), "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.
 
 ## Purpose
 
@@ -36,7 +36,7 @@ The listening connection reconnects with backoff when it drops.
 
 ### Pause
 
-When `paused` is on, scheduled fires of the pausable jobs (`visual-feed`, `drawing-lookup`) are skipped. Each skip is recorded as a `job_runs` row with status `skipped` and counts `{"reason": "paused"}`. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping is not paused.
+When `paused` is on, scheduled fires of the pausable jobs (`visual-feed`, `drawing-lookup`) are skipped. Each skip is recorded as a `job_runs` row with status `skipped` and counts `{"reason": "paused"}`. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping and the phase 9 jobs are not paused.
 
 ### Run now and job_requests
 
@@ -82,16 +82,97 @@ At start the worker runs the same stack guard as the app: the test stack refuses
 | `visual-feed` | `interval_shop_minutes` inside shop hours (default 5), `interval_off_minutes` outside (default 30) | yes | The five reads, the cache, the packet rules, auto-release. Chains `drawing-lookup`. |
 | `drawing-lookup` | After every feed run, plus every `drawing_interval_minutes` (default 30) | yes | The drawing queue (see "Drawings"). |
 | `housekeeping` | Daily at 03:30 site time | no | Prunes old rows. The plan keeps the caches 90 days and `job_runs` 180 days. |
+| `ad-reconcile` | Daily at 01:30 site time, plus "Refresh from AD now" | no | Users and roles from the AD groups, inactive users signed out, employee ID flags (phase 9). |
+| `session-refresh` | Every `roles_refresh_minutes` (default 15) | no | Every live session's groups and account state read again (phase 9). |
+| `integrity` | Daily at 04:00 site time | no | Seals, asset hashes, missing and orphan files, the audit hash chain (phase 9). |
+| `backup-check` | Daily at 07:15 site time | no | The newest dump is there, not empty, under 26 hours old, and not followed by a failure (phase 9). |
+| `notify-send` | Every 60 seconds | no | Rule seed, the version check, the watch checks, immediate delivery and the digest queue (phase 9). |
+| `notify-digest` | Every 5 minutes | no | Sends each digest that is due, one message per address (phase 9). |
 
-Planned, not built in phase 3:
+`job_schedule` holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for `visual-feed`, `drawing-lookup` and `housekeeping`; 0010 adds the six phase 9 jobs.
 
-- `session-refresh` (every 15 minutes): re-resolve groups and the account state for every live session.
-- `ad-reconcile` (nightly 01:30 site time, plus "refresh from AD now"), phase 9.
-- `integrity` (04:00): seals, asset hashes, missing and orphan files, the audit hash chain. Phase 9.
-- `backup-check` (07:15): the newest dump is under 26 hours old. Phase 9.
-- `notify-send` (every 60 seconds) and `notify-digest` (daily at the digest time). Phase 9.
+## Phase 9 jobs
 
-`job_schedule` holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for `visual-feed`, `drawing-lookup` and `housekeeping`.
+None of the six phase 9 jobs is pausable. Each is a module in `worker/jobs/` with its own `SPEC`. `load_jobs()` imports them lazily, so a module that fails to import is logged and left out, and the rest run. They use the same run now, advisory lock and `job_runs` rules as the feed. `packetstation job run <name>` runs any of them once.
+
+### ad-reconcile (01:30 site time, plus "Refresh from AD now")
+
+1. Reloads the departments (and with them the group map).
+2. Reads AD, outside any transaction: every group in `LDAP_GROUP_BASE_DN` (name and DN), the members of every mapped group through `LDAP_MATCHING_RULE_IN_CHAIN` in pages of 500 (disabled accounts included, so they can be marked), and, by objectGUID in batches of 50, every active user who was not in any group.
+3. **Refusal guard.** When no mapped group is in the OU (usually a wrong `LDAP_GROUP_BASE_DN`), or when every mapped group is empty while active users exist, the run raises `ReconcileRefused`, fails with that reason in its errors, and changes nothing (PLAN.md open question 44).
+4. One transaction: users upserted on objectGUID and their roles rebuilt; a disabled user marked inactive with their sessions ended `ad_disabled`; a user AD no longer has marked inactive with `ad_gone`; a user in AD with no mapped group left without roles and their sessions ended `roles_lost`. `directory_flags` gets `emp_missing`, `emp_duplicate`, `gone` and `disabled`. Every ended session writes `auth.session_ended` with `by: ad-reconcile`, and the run writes an `ad.refresh` audit row with its counts.
+5. Events, in their own transaction: `group_missing` (a condition per missing group, subject the group name) raised, and resolved for every mapped group present; `ad_reconcile_report` (one-shot, info) when anyone was added or deactivated, an extra group is in the OU, or an employee ID is missing or shared. A notify error is recorded on the run and does not fail it.
+
+AD unreachable fails the run and changes nothing. A failed or refused run raises the `ad_reconcile_report` condition "AD reconcile did not run" (subject `ad-reconcile:failed`), so the rule's recipients get one mail until a run succeeds and clears it. The counts are the report the Admin page shows: `groups_checked`, `groups_missing` and `groups_extra` (names), `users_seen`, `created`, `updated`, `deactivated`, `sessions_ended`, `emp_missing` (accounts) and `emp_duplicate` (employee IDs), lists capped at 200.
+
+"Refresh from AD now" inserts a `job_requests` row for `ad-reconcile` and answers 202 (open question 48). `packetstation ad-refresh` runs the same reconcile in the foreground.
+
+In CI the Samba DC has an extra group and a duplicate employee ID on purpose, so the report is raised every night there.
+
+### session-refresh (every `roles_refresh_minutes`, default 15)
+
+For every user with a live session (not revoked, not expired): AD first, by objectGUID in batches, then each enabled user's groups. Then one transaction: a disabled or gone user is marked inactive and their sessions ended (`ad_disabled`, `ad_gone`); a user left with no role has their roles cleared and sessions ended (`roles_lost`); everyone else gets their user row, `user_roles`, each live session's `roles_snapshot` and `roles_checked_at` updated. Any AD failure makes the run `degraded` with `counts.ad = "unavailable"`, and nothing is written or ended. The interval comes from the Security tab setting, read at each run, so a changed value applies after the next run. Counts: `users`, `refreshed`, `roles_changed`, `deactivated`, `roles_lost`, `sessions_ended`.
+
+### integrity (04:00 site time)
+
+Each step in its own transaction:
+
+- Seals: every unbroken seal of a filed packet is rebuilt and compared with the seal key. A seal made under another `key_id` is a finding. No seal key makes the run degraded and the seals unchecked.
+- Assets: every asset row checked for its file (`asset_missing`). The sha256 is recomputed for assets uploaded in the last 7 days every night, and for every asset on Sundays (site time); `hash_checked_at` and `hash_ok` are set (`asset_hash`).
+- Orphans: files under `ASSETS_DIR` that no asset names, older than one hour (uploads in flight are skipped), `lost+found` skipped (`blob_orphan`). Reported, never deleted.
+- Audit: `audit.verify_chain` over the hash chain (`audit_chain`).
+
+One `integrity_findings` row per finding, at most 500 per kind (the count in the run's counts). Any finding raises `integrity_failure` (condition, critical, subject `integrity`). A clean run that was not degraded resolves it. Counts: `full`, `seals`, `assets`, `hashed`, `files`, `audit_rows`, `findings` per kind, `total`.
+
+### backup-check (07:15 site time)
+
+Reads the stack's backup folder, mounted read-only at `/backups` (`BACKUP_DIR` on the host). Problems: the folder is not there, no `*.dump` at the top of it, the newest dump is empty or older than 26 hours (by modification time), or `last-failure` is newer than `last-success` (or there is a failure marker and no success marker). Any problem raises `backup_stale` (condition, subject the stack name); none resolves it. The run itself stays `ok`; the alert carries the news. Counts: `folder`, `dumps`, `newest`, `newest_at`, `age_hours`, `size`, `last_success`, `last_failure`, `problems`. The CI stack's backup folder is empty, so CI raises `backup_stale` in dry run.
+
+### notify-send (every 60 seconds)
+
+1. Seeds any missing rule (on, at once, digest 07:00, to PS-Admins). Never changes a rule that exists.
+2. On its first run in each worker process: when `PS_BUILD` differs from the settings row `notify.last_build`, updates the row and raises `version_deployed` (subject `build:<build>`). Not at worker start, since the scheduler may start before PostgreSQL answers (open question 47). A restart on the same build raises nothing.
+3. At most every 15 minutes per process, the watch checks: `drawing_failing` when drawing lookups have been failing for more than the rule's hours (default 24; subject `drawings`, the list of up to 50 in the body), resolved when none is; `co_waiting` per change order pending review more than the rule's days (default 3) on a live packet (subject `co:<id>`, the packet's department), resolved when it is no longer waiting.
+4. Delivery: up to 200 undispatched events, oldest first, each in its own transaction with `FOR UPDATE SKIP LOCKED`. No rule: `no_rule`. Rule off: `rule_off`. Digest rule: an outbox row, `queued`. Immediate: recipients resolved now, one message, one `notifications_sent` row, `sent`, `dry_run`, `failed` or `no_recipients`. A failed SMTP send is recorded and not retried. AD unavailable while resolving a group: a `failed` row "directory unavailable", the event stays pending (`directory_retry:N`), and the third failure dispatches it `failed`.
+
+A failed or retried send makes the run degraded.
+
+### notify-digest (every 5 minutes)
+
+For each rule whose digest time (site time) has passed today and that has outbox rows queued before it: recipients resolved per event, then grouped per address across every rule. One message per address lists every due event (`Daily digest: N notifications`), with one `notifications_sent` row per message (event type `digest`). An outbox row is marked sent only when every address got it. A failed address leaves the rows for the next run, which skips addresses that already got an event, and the job backs off 30 minutes (open question 43). Nothing queued: the run ends without touching AD.
+
+### Events from the feed and the drawing job
+
+After each `visual-feed` and `drawing-lookup` run, `worker/runs.py` raises or resolves the feed conditions:
+
+| Run status | `feed_failed` (subject the job name, critical, the first ten errors in the body) | `feed_degraded` (visual-feed only) |
+|---|---|---|
+| failed | raised | unchanged |
+| degraded | resolved | raised |
+| ok | resolved | resolved |
+| skipped | unchanged | unchanged |
+
+`drawing_failing` comes from the notify-send watch check above, not from the run. Each event goes to its rule's recipients, PS-Admins by default. A broken run connection falls back to a fresh one and never fails the run.
+
+### What each event means
+
+| Event type | Raised by | Kind | Cleared by |
+|---|---|---|---|
+| `security_alert` | Account cooldown, PIN disabled, session ended after wrong PINs | one-shot, 15 min per subject | none |
+| `signin_burst` | An address over its limit, or the burst threshold across accounts | one-shot, 30 min per subject | none |
+| `integrity_failure` | integrity | condition | a clean integrity run |
+| `feed_failed` | visual-feed or drawing-lookup failed | condition | an ok or degraded run |
+| `feed_degraded` | visual-feed degraded | condition | an ok run |
+| `drawing_failing` | notify-send watch | condition | no lookup failing that long |
+| `ad_reconcile_report` | ad-reconcile | one-shot | none |
+| `group_missing` | ad-reconcile, per group | condition | the group found |
+| `backup_stale` | backup-check | condition | a good backup |
+| `co_waiting` | notify-send watch, per change order | condition, department | decided or no longer waiting |
+| `qc_reject` | A Q.C. reject on a packet, an ITP line reject | one-shot, department | none |
+| `version_deployed` | notify-send, first run on a new build | one-shot | none |
+| `assets_quota` | The storage check after every upload and in housekeeping | condition | usage back under the threshold |
+
+A condition sends once when it opens and a "Cleared: <title>" message to the same rule when it closes. Raised again while open, it only counts (`raised_count`). The department of an ITP reject is the one department among the unit's live packets that works to the line's page; none or several gives none.
 
 ## Feed reads