Phase 3: VISUAL feed and M-Files drawings in the worker

Migration 0004 (VISUAL and drawing caches, go-live ignored list, resource
map, job tables, order-derived columns with provenance; work order key
includes the operation sequence). The real worker on APScheduler with
settings re-read each fire, pause, run now, advisory locks, job_runs and a
heartbeat. The visual-feed job (five VisualKit reads, COPY and MERGE cache,
suspect-batch guard, the apply in one transaction: drafts, change orders,
retire-moved, work order gone, closed and cancelled observed, projects and
units from order links, auto-release) and the fingerprint. The drawing-lookup
job (queue, budget, download on version change, sha256 into assets, decision
table, staging for rev approval, drawing-origin change orders). Clerical From
VISUAL panel, flags, rev approval, hand uploads, change-order review; Admin
Feed settings and resource map. Synthetic fixture generator and a
fixture-driven worker in CI with a Playwright scenario. Docs updated.

Verified: 3249 server tests and 13 Playwright scenarios on the docker-1 CI
stack, 76 frontend tests, doctor at 0004; one feed and one drawing run
against VISUAL DEV and the vault from the developer machine (counts in the
changelog).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

CHANGELOG.md

@@ -6,6 +6,19 @@ All notable changes to this project are recorded here. The format follows [Keep
 
 ### Added
 
+- Phase 3: the VISUAL feed and M-Files drawings run in the worker. Migration 0004 with the VISUAL cache (batches, rows, materials, orders and order links, labor, per-batch change rows), the go-live ignored list, the resource map seeded from the old build's map (fold and ignore lines included), drawing lookups, job requests, job runs, job schedule and the worker heartbeat, plus the order-derived columns with provenance on projects and units. The work order key on a packet now includes the operation sequence, so one packet can hold two operations of the same work order.
+- The real worker: APScheduler with a settings-driven trigger that re-reads the feed settings on every fire, shop-hours and off-hours intervals, pause, LISTEN on `ps_settings` and `ps_job_request`, one advisory lock per job, `job_runs` rows with counts and redacted errors, a heartbeat row and file, the stack guard before every VISUAL read, and a housekeeping job that prunes the caches at 90 days and the runs at 180.
+- The `visual-feed` job: five reads through VisualKit (released work orders in the window with `until` always passed, every tracked work order by key with no status or date filter, operations and part specs and parts and materials with `mat_known` per leg, order links by base and by project, labor tickets at most every 30 minutes), COPY and MERGE into the caches with change rows, the suspect-batch guard, and the apply in one transaction: candidates by the old rules (one packet is drawing plus lot plus operation; fold and ignore; project from the base prefix; unit is project plus the three-digit lot), the go-live boundary with the ignored list and pull-in, matching on the VISUAL key across live and removed packets (a removed packet is never recreated), new drafts, drafts updated in place, change orders opened, re-aimed, withdrawn or followed on for packets past draft (minor when only target hours or the want date changed), left-behind drafts retired, work order gone after two misses, closed and cancelled observed per leg, projects and units created with customer, PO and dates from the order links and the several-open-orders rule in one function with the provenance stored, auto-release of clean drafts up to 25 per run, audit rows and NOTIFY.
+- `feed/fingerprint.py`: a canonical hash over legs, quantity, rev, drawing and BOM (major) and target hours and want date (minor), with per-field tests and invariance tests.
+- The `drawing-lookup` job: a prioritised queue, a per-run budget, a new MFilesKit lookup per run, download only when the M-Files object or file version changed, sha256 into assets, pages through pypdfium2, the decision table (attach, rev mismatch staged for Clerical Supervisor approval, not released with the drafting status, not found, lookup failed with backoff, non-numeric rev), the H-sheet note, drawing-origin change orders for a re-released file or a newer rev on an in-shop packet, and the hand drawing taken off on a new VISUAL rev.
+- Clerical: the From VISUAL panel (run now, status, last ten runs, ignored work orders with pull in, unmapped resources, projects without names), the Changes to approve column and KPI, flag kinds in the Flagged column and the find bar, rev-mismatch approval with reason and signature, hand upload of a drawing or a BOM (XLSX or CSV parsed on the server) on flagged drafts with reason and signature, change-order approve, reject and hold with signature, and the project's VISUAL orders with a pin.
+- Admin: the Feed settings tab (intervals, shop hours, window, go-live date, departments, drawing recheck and budget, pause, next run, last ten runs) and the VISUAL resources editor. Every change is audited.
+- `/api/feed/*`, `/api/jobs/*`, `/api/admin/feed/*`, and under `/api/packets/{id}`: `rev-approve`, `hand/drawing`, `hand/bom`. `GET /api/packets` gains `co=pending` and `flag=<kind>`.
+- `packetstation job run <name>` and `packetstation fixtures use N`. Settings `FEED_SOURCE`, `DRAWING_SOURCE` and `FIXTURE_DIR`; a production stack refuses a fixture source.
+- `tests/fixtures/generator.py` writes a synthetic feed in the exact VisualKit contract shape with the edge cases from the plan; CI feeds the worker from it and the Playwright scenario drives a feed run, a drawing attach and a change order on an in-shop packet.
+- docs/FEED.md written in full; docs/SCHEMA.md, DEPLOY.md, POLICY.md and SECURITY.md updated.
+- Integration run from a developer machine against VISUAL DEV and the vault (read-only, 2026-10-16): `visual-feed` ok against `VSQL\DEV / PESCOINCDEV`: 994 firmed work orders in the window, 4,090 operation rows, 13,465 material lines, 100 customer orders with 102 links, 474 new drafts, 47 units and 18 projects created, 232 lettered-lot rows skipped (lots such as 6A make no unit, as in the old build), no unmapped resources, no errors. `drawing-lookup` ok against the vault: 60 drawings looked up and fetched (the per-run budget), 213 drafts got their drawing attached, 68 flagged rev mismatch, none not found or failed. Nothing was written to VISUAL or M-Files.
+
 - Phase 2: core domain. Migration 0003 with projects, project aliases and renumber history, units, packets and every child table (work orders, pages, BOM lines, shorts and receipts, welds, heat tags, heat table placements, stages, warehouse stage rows, transitions, Q.C. inspections and history, seals, packet log, VISUAL state), change orders with items, required departments and verifications, assets, packet documents, Q.C. documents, ITPs, user station limits and the department weld flag. Append-only triggers, soft-delete columns with partial unique indexes, server-assigned counters on packets.
 - The policy engine in Python (`server/packetstation/policy/`): Principal, PacketState, the Action enum, one rule per action, custody, named preconditions, the change-order hold, and the order of checks from the plan. Every packet read returns `can`, `blocked` with reasons, and `holder`. docs/POLICY.md lists the rules and the tightenings against the old build for Seth.
 - The write pipeline: one transaction per write with the packet row locked, the policy decision, compare-and-set on the expected status and department (409 "someone else already took this step" with the current state), If-Match row versions on child rows, server-assigned weld, heat tag, page and change order numbers, field-level compare-and-set on header scalars, packet version bump, packet log and audit rows, NOTIFY. Signatures through the phase 1 helper on every action the plan marks. Admin edits need an IT reason and are marked in the packet log and the audit. Filed packets are locked; filing writes an HMAC seal and Admin reopen breaks it with a reason.