Phase 3: VISUAL feed and M-Files drawings in the worker

Migration 0004 (VISUAL and drawing caches, go-live ignored list, resource map, job tables, order-derived columns with provenance; work order key includes the operation sequence). The real worker on APScheduler with settings re-read each fire, pause, run now, advisory locks, job_runs and a heartbeat. The visual-feed job (five VisualKit reads, COPY and MERGE cache, suspect-batch guard, the apply in one transaction: drafts, change orders, retire-moved, work order gone, closed and cancelled observed, projects and units from order links, auto-release) and the fingerprint. The drawing-lookup job (queue, budget, download on version change, sha256 into assets, decision table, staging for rev approval, drawing-origin change orders). Clerical From VISUAL panel, flags, rev approval, hand uploads, change-order review; Admin Feed settings and resource map. Synthetic fixture generator and a fixture-driven worker in CI with a Playwright scenario. Docs updated. Verified: 3249 server tests and 13 Playwright scenarios on the docker-1 CI stack, 76 frontend tests, doctor at 0004; one feed and one drawing run against VISUAL DEV and the vault from the developer machine (counts in the changelog). Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-16 14:45:38 -06:00
parent 16950055e3
commit 591fe90076
103 changed files with 16063 additions and 151 deletions
@@ -0,0 +1,60 @@
+"""Housekeeping, nightly at 03:30 site time: prune the VISUAL change rows and gone cache rows, old job runs and
+old claimed run requests. It never touches packets, the audit log or anything append-only."""
+
+from __future__ import annotations
+
+from datetime import datetime, time
+
+from ...feed.config import FeedSettings
+from . import Counts, JobContext, JobSpec, seconds_until
+
+AT = time(3, 30)
+CACHE_DAYS = 90
+RUNS_DAYS = 180
+REQUESTS_DAYS = 30
+
+STEPS: tuple[tuple[str, str, int], ...] = (
+    ("row_changes", "DELETE FROM visual_row_changes WHERE at < now() - make_interval(days => %s)", CACHE_DAYS),
+    (
+        "material_changes",
+        "DELETE FROM visual_material_changes WHERE at < now() - make_interval(days => %s)",
+        CACHE_DAYS,
+    ),
+    (
+        "gone_rows",
+        "DELETE FROM visual_rows WHERE gone_batch IN"
+        " (SELECT id FROM visual_batches WHERE started_at < now() - make_interval(days => %s))",
+        CACHE_DAYS,
+    ),
+    (
+        "gone_materials",
+        "DELETE FROM visual_materials WHERE gone_batch IN"
+        " (SELECT id FROM visual_batches WHERE started_at < now() - make_interval(days => %s))",
+        CACHE_DAYS,
+    ),
+    (
+        "job_runs",
+        "DELETE FROM job_runs WHERE started_at < now() - make_interval(days => %s) AND status <> 'running'",
+        RUNS_DAYS,
+    ),
+    (
+        "job_requests",
+        "DELETE FROM job_requests WHERE claimed_at IS NOT NULL AND claimed_at < now() - make_interval(days => %s)",
+        REQUESTS_DAYS,
+    ),
+)
+
+
+def run(ctx: JobContext) -> Counts:
+    counts: Counts = {}
+    for name, sql, days in STEPS:
+        with ctx.conn.transaction():
+            counts[name] = ctx.conn.execute(sql, (days,)).rowcount
+    return counts
+
+
+def cadence(_: FeedSettings, now_site: datetime) -> int:
+    return seconds_until(now_site, AT)
+
+
+SPEC = JobSpec(name="housekeeping", run=run, cadence=cadence, pausable=False)