Phase 3: VISUAL feed and M-Files drawings in the worker

Migration 0004 (VISUAL and drawing caches, go-live ignored list, resource
map, job tables, order-derived columns with provenance; work order key
includes the operation sequence). The real worker on APScheduler with
settings re-read each fire, pause, run now, advisory locks, job_runs and a
heartbeat. The visual-feed job (five VisualKit reads, COPY and MERGE cache,
suspect-batch guard, the apply in one transaction: drafts, change orders,
retire-moved, work order gone, closed and cancelled observed, projects and
units from order links, auto-release) and the fingerprint. The drawing-lookup
job (queue, budget, download on version change, sha256 into assets, decision
table, staging for rev approval, drawing-origin change orders). Clerical From
VISUAL panel, flags, rev approval, hand uploads, change-order review; Admin
Feed settings and resource map. Synthetic fixture generator and a
fixture-driven worker in CI with a Playwright scenario. Docs updated.

Verified: 3249 server tests and 13 Playwright scenarios on the docker-1 CI
stack, 76 frontend tests, doctor at 0004; one feed and one drawing run
against VISUAL DEV and the vault from the developer machine (counts in the
changelog).

Co-Authored-By: Claude Fable 5.1 <[email protected]>

server/packetstation/drawings/decide.py

@@ -0,0 +1,190 @@
+"""The drawing decision table. Pure: no database, no M-Files.
+
+`decide(target, found, same_file=...)` says what the drawing job does to one packet for one lookup. Note texts are
+the app's own (from the old revMismatch and draftLine in visual.js, "-" in place of the em-dash), never M-Files'
+note passed through, except inside the lookup_failed text.
+
+Rows, in the order they are tried:
+ 1. lookup failed -> flag lookup_failed, keep what is attached (state lookup_failed only when it was needed).
+ 2. past draft, a pending feed change order waits for this drawing at the released rev -> stage_for_co.
+    Past draft, the pending change order already stages this drawing at the released rev -> unchanged, or
+    stage_for_co again when M-Files has a different file for that rev.
+ 3. no released rev, a draft rev or a drafting assignment -> flag not_released with the drafting status.
+ 4. no released rev otherwise -> flag not_found.
+ 5. VISUAL rev not a number (and no approved rev equal to M-Files') -> flag bad_rev, not attached.
+ 6. revs differ (and no approved rev equal to M-Files') -> flag rev_mismatch. The job stages the M-Files copy
+    for Clerical Supervisor approval on drafts only.
+ 7. revs match, draft with a hand drawing -> unchanged (the feed already took the hand drawing off on a rev move).
+ 8. revs match, draft with this very file attached at this rev -> unchanged.
+ 9. revs match, draft -> attach ("sheet of drawing <base>" for an H sheet).
+10. revs match, past draft, nothing attached -> unchanged.
+11. revs match, past draft, same file as attached -> unchanged.
+12. revs match, past draft, a pending change order already stages this file, or Clerical rejected a change order
+    for this file -> unchanged.
+13. revs match, past draft, attached rev lower -> open_co "M-Files released rev N of <dwg>; this packet has rev M".
+14. revs match, past draft, attached rev equal, new file -> open_co "M-Files re-released rev N of <dwg> (new file)".
+15. revs match, past draft, attached rev higher or not a number -> unchanged.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from .lookup import Found
+
+LOOKUP_FAILED = "lookup_failed"
+NOT_RELEASED = "not_released"
+NOT_FOUND = "not_found"
+BAD_REV = "bad_rev"
+REV_MISMATCH = "rev_mismatch"
+ATTACH = "attach"
+UNCHANGED = "unchanged"
+OPEN_CO = "open_co"
+STAGE_FOR_CO = "stage_for_co"
+
+KINDS = (ATTACH, UNCHANGED, REV_MISMATCH, NOT_RELEASED, NOT_FOUND, LOOKUP_FAILED, BAD_REV, OPEN_CO, STAGE_FOR_CO)
+FLAG_KINDS = (LOOKUP_FAILED, NOT_RELEASED, NOT_FOUND, BAD_REV, REV_MISMATCH)
+
+# drawing_state enum values (0003). bad_rev has no state of its own: it is a rev mismatch Clerical can approve.
+STATE_OF = {NOT_RELEASED: "not_released", NOT_FOUND: "not_found", BAD_REV: "rev_mismatch", REV_MISMATCH: "rev_mismatch"}
+HELD_STATES = ("attached", "hand")
+
+NOT_FOUND_NOTE = "Drawing not in M-Files - check the drawing number in VISUAL"
+_FAILED_PREFIX = "M-Files lookup failed: "
+
+
+@dataclass(frozen=True)
+class Target:
+    dwg: str
+    visual_rev: str
+    attached_rev: str | None  # rev of the live drawing pages; None when no drawing page is attached
+    attached_sha: bytes | None
+    status: str  # packet status
+    drawing_state: str
+    has_hand_drawing: bool
+    rev_ok: str | None  # the rev Clerical approved for this VISUAL rev, if any
+    # Additions to the contract, all defaulted: the pending feed change order (origin visual or drawing), if any.
+    co_waiting: tuple[str, str] | None = None  # (dwg, rev) the pending CO waits for
+    co_staged: tuple[str, str] | None = None  # (dwg, rev) the pending CO already stages
+    co_pending: bool = False
+    co_staged_sha: bytes | None = None  # sha256 of the file the pending CO already stages
+    rejected_sha: bytes | None = None  # sha256 staged on the last drawing-origin CO Clerical rejected
+
+
+@dataclass(frozen=True)
+class Decision:
+    kind: str
+    flag: str | None
+    drawing_state: str
+    note: str
+
+
+def _int(v: object) -> int | None:
+    s = "" if v is None else str(v).strip()
+    return int(s) if s.isdigit() else None
+
+
+def draft_line(f: Found) -> str:
+    """The old draftLine: "rev N <where>", or "in drafting, no released PDF yet", or ""."""
+    if f.draft_rev is not None and f.draft_where:
+        return f"rev {f.draft_rev} {f.draft_where}"
+    if f.in_drafting:
+        return "in drafting, no released PDF yet"
+    if f.draft_rev is not None:
+        return f"rev {f.draft_rev}"
+    return ""
+
+
+def failed_note(f: Found) -> str:
+    detail = (f.note or "").strip()
+    if detail.startswith(_FAILED_PREFIX):  # MFilesKit's own note already carries the prefix
+        detail = detail[len(_FAILED_PREFIX) :]
+    return f"{_FAILED_PREFIX}{detail or 'no detail'}"
+
+
+def rev_confirmed(t: Target, f: Found) -> bool:
+    """The old revConfirmed: M-Files' released rev is the rev VISUAL calls, or Clerical approved that rev."""
+    if not f.ok or f.rev is None:
+        return False
+    v = _int(t.visual_rev)
+    if v is not None and v == f.rev:
+        return True
+    return t.rev_ok is not None and str(t.rev_ok).strip() == str(f.rev)
+
+
+def decide(t: Target, f: Found, *, same_file: bool, fetched_sha: bytes | None = None) -> Decision:
+    """What to do to one packet.
+
+    `same_file`: the M-Files file has the sha256 of the attached drawing asset. `fetched_sha`: the sha256 of the
+    M-Files file, when it was fetched (row 12 compares it with the pending and the rejected change orders).
+    """
+    held = t.drawing_state in HELD_STATES
+    draft = t.status == "draft"
+
+    def flag(kind: str, note: str) -> Decision:
+        return Decision(kind, kind, t.drawing_state if held else STATE_OF[kind], note)
+
+    def keep(kind: str, note: str = "") -> Decision:
+        return Decision(kind, None, t.drawing_state, note)
+
+    # 1
+    if not f.ok:
+        return Decision(
+            LOOKUP_FAILED,
+            LOOKUP_FAILED,
+            LOOKUP_FAILED if t.drawing_state == "needed" else t.drawing_state,
+            failed_note(f),
+        )
+    # 2
+    if not draft and f.rev is not None and t.co_waiting is not None:
+        w_dwg, w_rev = t.co_waiting
+        if w_dwg.strip() == t.dwg.strip() and str(w_rev).strip() == str(f.rev):
+            return keep(STAGE_FOR_CO, f"Drawing for the change order is in M-Files (rev {f.rev})")
+    if not draft and f.rev is not None and t.co_staged is not None:
+        s_dwg, s_rev = t.co_staged
+        if s_dwg.strip() == t.dwg.strip() and str(s_rev).strip() == str(f.rev):
+            if fetched_sha is not None and t.co_staged_sha is not None and fetched_sha != t.co_staged_sha:
+                return keep(STAGE_FOR_CO, f"New file for rev {f.rev} in M-Files; staged again for the change order")
+            return keep(UNCHANGED, f"Rev {f.rev} is staged on the pending change order")
+    # 3, 4
+    if f.rev is None:
+        d = draft_line(f)
+        if d:
+            on = f" (sheet of {f.base})" if f.base else ""
+            return flag(NOT_RELEASED, f"Not released in M-Files yet{on} - drafting status: {d}")
+        return flag(NOT_FOUND, NOT_FOUND_NOTE)
+    approved = t.rev_ok is not None and str(t.rev_ok).strip() == str(f.rev)
+    vrev = (t.visual_rev or "").strip()
+    v = _int(vrev)
+    if not approved:
+        # 5
+        if v is None:
+            return flag(BAD_REV, f"VISUAL rev {vrev or '-'} is not a number; M-Files has rev {f.rev} - not attached")
+        # 6
+        if v != f.rev:
+            d = f"; drafting status: {draft_line(f)}" if f.draft_rev is not None and f.draft_rev > f.rev else ""
+            return flag(REV_MISMATCH, f"Rev mismatch: VISUAL calls rev {vrev}, M-Files has rev {f.rev}{d}")
+    sheet = f"sheet of drawing {f.base}" if f.base else ""
+    if draft:
+        # 7
+        if t.has_hand_drawing:
+            return keep(UNCHANGED, "A drawing uploaded by hand stays at the same rev")
+        # 8
+        if t.drawing_state == "attached" and same_file and t.attached_rev == str(f.rev):
+            return keep(UNCHANGED, sheet)
+        # 9
+        return Decision(ATTACH, None, "attached", sheet)
+    # 10, 11
+    if t.attached_rev is None or same_file:
+        return keep(UNCHANGED, sheet)
+    # 12
+    if fetched_sha is not None and fetched_sha in (t.co_staged_sha, t.rejected_sha):
+        return keep(UNCHANGED, sheet)
+    att = _int(t.attached_rev)
+    if att is None or att > f.rev:
+        return keep(UNCHANGED, sheet)  # 15
+    if att < f.rev:
+        reason = f"M-Files released rev {f.rev} of {t.dwg}; this packet has rev {att}"
+    else:
+        reason = f"M-Files re-released rev {f.rev} of {t.dwg} (new file)"
+    return keep(OPEN_CO, reason)