Phase 3: VISUAL feed and M-Files drawings in the worker

Migration 0004 (VISUAL and drawing caches, go-live ignored list, resource
map, job tables, order-derived columns with provenance; work order key
includes the operation sequence). The real worker on APScheduler with
settings re-read each fire, pause, run now, advisory locks, job_runs and a
heartbeat. The visual-feed job (five VisualKit reads, COPY and MERGE cache,
suspect-batch guard, the apply in one transaction: drafts, change orders,
retire-moved, work order gone, closed and cancelled observed, projects and
units from order links, auto-release) and the fingerprint. The drawing-lookup
job (queue, budget, download on version change, sha256 into assets, decision
table, staging for rev approval, drawing-origin change orders). Clerical From
VISUAL panel, flags, rev approval, hand uploads, change-order review; Admin
Feed settings and resource map. Synthetic fixture generator and a
fixture-driven worker in CI with a Playwright scenario. Docs updated.

Verified: 3249 server tests and 13 Playwright scenarios on the docker-1 CI
stack, 76 frontend tests, doctor at 0004; one feed and one drawing run
against VISUAL DEV and the vault from the developer machine (counts in the
changelog).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

docs/SCHEMA.md

@@ -1,6 +1,6 @@
 # Packet Station schema
 
-Status: phases 1 and 2 built (identity, organisation, projects, packets, change orders, documents). Phase 3 adds the job and VISUAL tables, phase 9 the notification tables. The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
+Status: phases 1 to 3 built (identity, organisation, projects, packets, change orders, documents, jobs, VISUAL and drawing caches). Phase 9 adds the notification tables. The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
 
 ## Conventions
 
@@ -8,7 +8,7 @@ Identity bigints, except users, keyed on the AD objectGUID. Postgres enums for w
 
 ## Database roles
 
-A superuser for bootstrap, the migrate role `ps_migrate` (owns the schema, runs migrations) and the app role `ps_app`. The plan's deployment section also names a worker role `ps_worker`. Phase 0 creates only the first three; whether the worker gets its own role is still to settle.
+A superuser for bootstrap, the migrate role `ps_migrate` (owns the database and the schema, runs migrations) and the app role `ps_app`. The worker shares `ps_app` and sets `application_name` to `ps-worker`, so the two are told apart in `pg_stat_activity`; a separate `ps_worker` role was not needed (decided in phase 3). `ps_app` holds CONNECT and TEMPORARY on the database (the feed COPYs each batch into temp tables) and no CREATE on the schema.
 
 ## Identity (phase 1, migration 0002)
 
@@ -77,7 +77,7 @@ The trigger `ps_packet_touch` sets `updated_at` on every UPDATE. It does not tou
 
 ### Children
 
-- `packet_work_orders`: `wo_base`, `lot`, `split`, `sub` (all text; the last three default to `''` so the key has no nulls), `seq`, `qty`, `target_hrs`, `op`, `description`, `visual_status`, `missing_runs` and `gone_at` (the feed's "work order gone" check), `sort`, `row_version`. Unique on (`packet_id`, `wo_base`, `lot`, `split`, `sub`).
+- `packet_work_orders`: `wo_base`, `lot`, `split`, `sub` (all text; the last three default to `''` so the key has no nulls), `seq`, `qty`, `target_hrs`, `op`, `description`, `visual_status`, `missing_runs` and `gone_at` (the feed's "work order gone" check), `sort`, `row_version`. Phase 3 adds `partspecs`, `rev` and `dwg`. Unique on (`packet_id`, `wo_base`, `lot`, `split`, `sub`, `seq`) with nulls not distinct, since migration 0004 (see "VISUAL and drawing caches").
 - `packet_pages`: `n` (the packet page number, from `next_page_no`), `type`, `asset_id`, `pn` (page within the asset), `dwg`, `rev`, `co_id` (the change order that added it), `void_co_id`, `void_at`, `void_by`, `void_new_n` (a replaced page is voided, never deleted), `row_version`. Unique on (`packet_id`, `n`).
 - `bom_lines`: `item`, `qty`, `part_id` (text), `description`, `dwg_ref`, `cl_item`, `length`, `material`, `heat`, `na`, `issued`, `loc`, `pulled`, `pulled_at`, `pulled_by`, `mtr_asset_id`, `src`, `vkey`, `whs`, `comm`, `cons`, `sub`, `grp`, `sort`, `row_version`. Unique on (`packet_id`, `item`).
 - `bom_shorts`: `bom_line_id` (cascades with the line), `qty`, `qty_open`, `reason`, `ref`, `eta`, `note`, `opened_at`, `opened_by`, `closed_at`, `row_version`. At most one open short per line (unique on `bom_line_id` where `closed_at` is null).
@@ -124,7 +124,15 @@ The app role is refused by privilege first. The migrate role owns the tables, so
 
 ## Audit, settings, jobs (phases 1 and 3)
 
-audit_events with a hash chain, settings, job_requests, job_runs, job_schedule, worker_heartbeat.
+- `audit_events` (migration 0001): append-only. `at`, `user_id`, `session_id`, `role_snapshot` (`system` for the worker), `action`, `subject_type`, `subject_id`, `packet_id`, `detail` jsonb, `it_reason`, `ip`. A hash chain: `row_hash` is sha256 over `prev_hash` and the canonical JSON of the row, and `prev_hash` is the row before it. BRIN index on `at`.
+- `settings` (migration 0001): `key` (text PK), `value` jsonb, `version`, `updated_at`, `updated_by`. Phase 3 adds the row `feed`, the feed settings as one JSON object (see [FEED.md](FEED.md), "Settings"). A missing key takes its default. A save bumps `version` and sends `ps_notify('ps_settings', ...)` so the worker reschedules.
+
+Phase 3, migration 0004:
+
+- `job_requests`: one row per "Run now". `job`, `requested_at`, `requested_by`, `note`, `claimed_run_id` and `claimed_at` (set by the run that took it). Index on `job` where `claimed_at` is null. The app inserts the row and NOTIFYs `ps_job_request` in the same transaction.
+- `job_runs`: one row per run. `job`, `trigger` (schedule, request, chain, cli), `requested_by` uuid[] (the requests claimed), `started_at`, `finished_at`, `status` (running, ok, degraded, failed, skipped, abandoned), `counts` jsonb, `errors` jsonb (a list of redacted strings), `build_version`, `visual_target` (`"<server> / <database>"`), `host`. Index on (`job`, `id DESC`).
+- `job_schedule`: one row per job (`job` PK). `next_run_at`, `interval_seconds`, `paused`, `updated_at`. Written by the worker after each run for the Admin tab. Seeded for `visual-feed`, `drawing-lookup` and `housekeeping`.
+- `worker_heartbeat`: a single row (`id` = 1 by CHECK). `at`, `build`, `host`, `pid`, `running` text[] (the jobs running now). Upserted every 30 seconds.
 
 ## Notifications (phase 9)
 
@@ -132,7 +140,26 @@ Rules, recipients, exclusions, outbox, sent log, state.
 
 ## VISUAL and drawing caches (phase 3)
 
-visual_batches, visual_rows, visual_materials, order links, labor, change rows, ignored work orders, resource map, drawing_lookups.
+Migration 0004. Every ID part is text (`wo_base`, `lot`, `split`, `sub`, part IDs, order IDs), with `''` in place of null in a key. `seq` and `piece` are int. Closed sets are text with a CHECK. The worker fills these tables. Screens read them, never VISUAL. [FEED.md](FEED.md) says how each is written.
+
+- `visual_batches`: one row per feed run. `run_id`, `started_at`, `finished_at`, `server`, `database`, `site`, `since` and `until` (the window), `window_rows`, `tracked_keys`, `missing_keys`, `suspect`, `counts` jsonb.
+- `visual_rows`: one row per work order operation (TYPE W), unique on (`wo_base`, `lot`, `split`, `sub`, `seq`). The contract fields: `res`, `dwg`, `rev`, `ps` (partspecs), `descr`, `qty`, `tgt`, `rel`, `want`, `st` (the status code F, R, U, C, X or `''`), `part`, `barcode`. `row_hash` (sha256 of the canonical row), `first_batch`, `last_batch` and `gone_batch` (set when a row in play was not read; cleared when it returns). Indexes on (`wo_base`, `lot`), on `dwg`, and on `rel` for rows not gone.
+- `visual_materials`: one row per material line, unique on (`wo_base`, `lot`, `split`, `sub`, `seq`, `piece`). `part` (null for a line with no part), `qty`, `issued`, `whs`, `loc`, `st`, `istat`, `dwg`, `descr`, `comm`, `cons`, `mat_hash`, `first_batch`, `last_batch`, `gone_batch`. Index on (`wo_base`, `lot`).
+- `visual_orders`: one row per customer order (`cust_order_id` PK). `status`, `customer_id`, `customer_name`, `customer_po_ref`, `order_date`, `desired_ship_date`, `promise_date`, `last_batch`.
+- `visual_order_links`: `kind` (base or project), `key` (the base or the project number), `cust_order_id`, `rank` (open first, newest first), `lines` int[], `top_jobs` text[], `last_batch`. Unique on (`kind`, `key`, `cust_order_id`). Replaced for every key read.
+- `visual_labor`: ticket hours per leg, unique on (`wo_base`, `lot`, `split`, `sub`, `seq`). `hours`, `tickets`, `last_ticket`, `observed_at`. No cost or pay column.
+- `visual_row_changes`: what VISUAL changed, per batch. `batch_id`, the key and `seq`, `kind` (added, changed, gone, returned), `before` and `after` jsonb (the row as text values), `at`. Index on `batch_id`. `visual_material_changes` is the same plus `piece`.
+- `visual_ignored_wos`: work orders kept out by the go-live date, unique on (`wo_base`, `lot`, `split`, `sub`). `rel`, `project`, `unit_number`, `res`, `dwg`, `first_seen_batch`, `last_seen_at`, and `pulled_in_at`, `pulled_in_by`, `pulled_in_reason` when Clerical pulls one in.
+- `visual_resource_map`: `resource` (PK), `target` (`fold`, `ignore` or a two-digit department code, by CHECK), `confirmed`, `updated_at`, `updated_by`. Seeded from the old build's `visual_map.json`, every entry unconfirmed. "Electical" is spelled that way in VISUAL.
+- `drawing_lookups`: one row per drawing number as VISUAL writes it (`dwg` PK, trimmed). The last answer: `checked_at`, `ok`, `rev` (released), `name`, `obj_id`, `obj_version`, `file_id`, `file_version`, `file_size`, `base` (H sheet), `draft_rev`, `draft_where`, `in_drafting`, `note`. Retry state: `fail_count`, `failing_since`, `next_check_at`. The fetched file: `asset_id`, `sha256`. `visual_rev_seen` is the VISUAL rev at the last lookup, so a rev change moves the drawing to the front of the queue. A rev mismatch keeps the staged M-Files copy here for approval.
+
+Column additions in 0004:
+
+- `packets`: `visual_cancelled_at` and `wo_gone_at`, so boards can filter on cancelled and gone without reading `flags`.
+- `packet_work_orders`: `partspecs`, `rev` and `dwg`, the leg's own planner instructions, rev and drawing as VISUAL has them. The header keeps the highest rev. The unique key now includes `seq`: 0004 drops the 0003 unique on (`packet_id`, `wo_base`, `lot`, `split`, `sub`) and adds `UNIQUE NULLS NOT DISTINCT (packet_id, wo_base, lot, split, sub, seq)`, because the feed puts two operations of one work order on one packet. Hand drafts keep `seq` null, so they still have one row per key.
+- `projects`: `order_date_visual`, `ship_date_visual`, `order_ref` jsonb (the chosen order and how it was chosen), and the Clerical pin `order_pin` (a customer order ID), `order_pin_by`, `order_pin_at`, `order_pin_reason`.
+- `units`: `customer`, `po`, `customer_visual`, `po_visual`, `order_date_visual`, `ship_date_visual`, `order_ref`, and the same pin columns.
+- `change_orders`: nothing new. `vis`, `waiting`, `staged`, `target_fp` and `minor` came in 0003.
 
 ## Versioning