PacketStation/docs/FEED.md

Packet Station feed and drawings

Status: phase 9. This file says how the worker reads VISUAL and M-Files, what it does with what it reads, and the other jobs it runs (phase 9: the AD reconcile, the role refresh, the integrity check, the backup check and the notifications). The design is in PLAN.md, "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.

Purpose

The worker is the only part of Packet Station that talks to VISUAL and M-Files. It reads VISUAL through VisualKit and M-Files through MFilesKit, both read-only, on a schedule. It stores what it read in cache tables, turns the rows into projects, units, draft packets and change orders, and works the drawing queue. Screens read PostgreSQL only.

There are three reasons for a scheduled job instead of reads per screen:

• VISUAL production must not be queried on page loads. One modest query every few minutes from one place is fine. One per user per screen is not.
• Turning VISUAL rows into packets is a state change. It runs in one place, in one transaction, with an audit row.
• Drawing lookups and downloads are slow. They belong in the background.

The worker never writes to VISUAL or M-Files.

Worker process

packetstation worker starts the worker. It runs the stack guard, builds the scheduler and blocks. The main thread writes the heartbeat and waits for a stop signal.

Scheduler

• APScheduler 3 with a pool of 4 threads. Each job has coalesce on, max_instances 1 and a misfire grace of 300 seconds.
• Times are scheduled in site time (SITE_TZ, default America/Denver) and stored in UTC.
• Jobs are registered in worker/jobs/__init__.py. Each job has a name, a run function, a cadence function and a pausable flag. A job can name others to chain after it: visual-feed chains drawing-lookup, so every feed run is followed by a drawing run.
• The cadence function returns the seconds to the next run, from the feed settings and the current site time.

Settings re-read

The feed settings live in the database (see "Settings"), not in the env file. The worker keeps one connection that listens on two channels:

• ps_settings: the app sends it when an Admin saves the Feed tab. The worker reloads the settings and recomputes the next run of every job. A new interval or the pause switch applies at once, without a restart.
• ps_job_request: see "Run now" below.

The listening connection reconnects with backoff when it drops.

Pause

When paused is on, scheduled fires of the pausable jobs (visual-feed, drawing-lookup) are skipped. Each skip is recorded as a job_runs row with status skipped and counts {"reason": "paused"}. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping and the phase 9 jobs are not paused.

Run now and job_requests

1. Clerical presses Run now on the From VISUAL panel (or an Admin on the Feed tab). The app inserts a job_requests row and sends NOTIFY ps_job_request with {"job": "<name>"} in the same transaction.
2. The worker moves that job's next run to now.
3. The run claims every pending request for the job at its start. Ten clicks make one run. The run's trigger is request when it claimed any.
4. A request that arrives while the job is running is left pending, and the job runs again straight after.

Advisory lock

Each run opens its own connection and takes pg_try_advisory_lock(7310, hashtext(<job name>)). When the lock is held, the run records status skipped with counts {"reason": "already running"} and stops. One run per job at a time, whether the trigger was the schedule, a request, a chain or the command line.

job_runs

Every run writes one job_runs row:

• trigger: schedule, request, chain or cli.
• requested_by: the users whose requests the run claimed.
• started_at, finished_at.
• status: running, then ok, degraded (finished but part of it failed, for example the order read), failed (an exception), skipped or abandoned. On start the worker marks its own host's rows still running as abandoned.
• counts: a JSON object (below).
• errors: at most 20 strings of at most 500 characters each, redacted. Redaction strips anything that looks like password=..., pwd=... or an Authorization: header, and the VISUAL and M-Files passwords themselves when they are in the environment.
• build_version, host, and visual_target ("<server> / <database>") on feed runs.

Feed run counts, all integers: wos_window, wos_tracked, rows, materials, orders, labor_legs, new_drafts, drafts_updated, cos_opened, cos_reaimed, cos_withdrawn, retired, wo_gone_flagged, closed_observed, ignored_golive, projects_created, units_created, auto_released, unmapped_resources, skipped_removed, skipped_closed and suspect (0 or 1). counts["unmapped"] holds {resource: rows} for the top 30 unmapped resources.

Drawing run counts: checked, fetched, unchanged, attached, flagged ({kind: n}), lookup_failed, cos_opened, staged and budget_left.

packetstation job run <name> runs one job once in the foreground with trigger cli, prints its job_runs row as JSON, and exits 1 when the run failed.

Heartbeat

The worker writes the worker_heartbeat row every 30 seconds (time, build, host, pid and the jobs running now), and touches the beat file the container healthcheck reads. The From VISUAL panel shows "Worker not running" when the heartbeat is stale.

Stack guard

At start the worker runs the same stack guard as the app: the test stack refuses a production VISUAL target, and the production stack refuses a VISUAL target that was not set explicitly. Before every feed run the guard runs again with fresh VisualKit settings. A failed guard fails the run. The run records the target it read.

Jobs and cadence

Job	Cadence	Pausable	What it does
visual-feed	interval_shop_minutes inside shop hours (default 5), interval_off_minutes outside (default 30)	yes	The five reads, the cache, the packet rules, auto-release. Chains drawing-lookup.
drawing-lookup	After every feed run, plus every drawing_interval_minutes (default 30)	yes	The drawing queue (see "Drawings").
housekeeping	Daily at 03:30 site time	no	Prunes old rows. The plan keeps the caches 90 days and job_runs 180 days.
ad-reconcile	Daily at 01:30 site time, plus "Refresh from AD now"	no	Users and roles from the AD groups, inactive users signed out, employee ID flags (phase 9).
session-refresh	Every roles_refresh_minutes (default 15)	no	Every live session's groups and account state read again (phase 9).
integrity	Daily at 04:00 site time	no	Seals, asset hashes, missing and orphan files, the audit hash chain (phase 9).
backup-check	Daily at 07:15 site time	no	The newest dump is there, not empty, under 26 hours old, and not followed by a failure (phase 9).
notify-send	Every 60 seconds	no	Rule seed, the version check, the watch checks, immediate delivery and the digest queue (phase 9).
notify-digest	Every 5 minutes	no	Sends each digest that is due, one message per address (phase 9).

job_schedule holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for visual-feed, drawing-lookup and housekeeping; 0010 adds the six phase 9 jobs.

Phase 9 jobs

None of the six phase 9 jobs is pausable. Each is a module in worker/jobs/ with its own SPEC. load_jobs() imports them lazily, so a module that fails to import is logged and left out, and the rest run. They use the same run now, advisory lock and job_runs rules as the feed. packetstation job run <name> runs any of them once.

ad-reconcile (01:30 site time, plus "Refresh from AD now")

1. Reloads the departments (and with them the group map).
2. Reads AD, outside any transaction: every group in LDAP_GROUP_BASE_DN (name and DN), the members of every mapped group through LDAP_MATCHING_RULE_IN_CHAIN in pages of 500 (disabled accounts included, so they can be marked), and, by objectGUID in batches of 50, every active user who was not in any group.
3. Refusal guard. When no mapped group is in the OU (usually a wrong LDAP_GROUP_BASE_DN), or when every mapped group is empty while active users exist, the run raises ReconcileRefused, fails with that reason in its errors, and changes nothing (PLAN.md open question 44).
4. One transaction: users upserted on objectGUID and their roles rebuilt; a disabled user marked inactive with their sessions ended ad_disabled; a user AD no longer has marked inactive with ad_gone; a user in AD with no mapped group left without roles and their sessions ended roles_lost. directory_flags gets emp_missing, emp_duplicate, gone and disabled. Every ended session writes auth.session_ended with by: ad-reconcile, and the run writes an ad.refresh audit row with its counts.
5. Events, in their own transaction: group_missing (a condition per missing group, subject the group name) raised, and resolved for every mapped group present; ad_reconcile_report (one-shot, info) when anyone was added or deactivated, an extra group is in the OU, or an employee ID is missing or shared. A notify error is recorded on the run and does not fail it.

AD unreachable fails the run and changes nothing. A failed or refused run raises the ad_reconcile_report condition "AD reconcile did not run" (subject ad-reconcile:failed), so the rule's recipients get one mail until a run succeeds and clears it. The counts are the report the Admin page shows: groups_checked, groups_missing and groups_extra (names), users_seen, created, updated, deactivated, sessions_ended, emp_missing (accounts) and emp_duplicate (employee IDs), lists capped at 200.

"Refresh from AD now" inserts a job_requests row for ad-reconcile and answers 202 (open question 48). packetstation ad-refresh runs the same reconcile in the foreground.

In CI the Samba DC has an extra group and a duplicate employee ID on purpose, so the report is raised every night there.

session-refresh (every roles_refresh_minutes, default 15)

For every user with a live session (not revoked, not expired): AD first, by objectGUID in batches, then each enabled user's groups. Then one transaction: a disabled or gone user is marked inactive and their sessions ended (ad_disabled, ad_gone); a user left with no role has their roles cleared and sessions ended (roles_lost); everyone else gets their user row, user_roles, each live session's roles_snapshot and roles_checked_at updated. Any AD failure makes the run degraded with counts.ad = "unavailable", and nothing is written or ended. The interval comes from the Security tab setting, read at each run, so a changed value applies after the next run. Counts: users, refreshed, roles_changed, deactivated, roles_lost, sessions_ended.

integrity (04:00 site time)

Each step in its own transaction:

• Seals: every unbroken seal of a filed packet is rebuilt and compared with the seal key. A seal made under another key_id is a finding. No seal key makes the run degraded and the seals unchecked.
• Assets: every asset row checked for its file (asset_missing). The sha256 is recomputed for assets uploaded in the last 7 days every night, and for every asset on Sundays (site time); hash_checked_at and hash_ok are set (asset_hash).
• Orphans: files under ASSETS_DIR that no asset names, older than one hour (uploads in flight are skipped), lost+found skipped (blob_orphan). Reported, never deleted.
• Audit: audit.verify_chain over the hash chain (audit_chain).

One integrity_findings row per finding, at most 500 per kind (the count in the run's counts). Any finding raises integrity_failure (condition, critical, subject integrity). A clean run that was not degraded resolves it. Counts: full, seals, assets, hashed, files, audit_rows, findings per kind, total.

backup-check (07:15 site time)

Reads the stack's backup folder, mounted read-only at /backups (BACKUP_DIR on the host). Problems: the folder is not there, no *.dump at the top of it, the newest dump is empty or older than 26 hours (by modification time), or last-failure is newer than last-success (or there is a failure marker and no success marker). Any problem raises backup_stale (condition, subject the stack name); none resolves it. The run itself stays ok; the alert carries the news. Counts: folder, dumps, newest, newest_at, age_hours, size, last_success, last_failure, problems. The CI stack's backup folder is empty, so CI raises backup_stale in dry run.

notify-send (every 60 seconds)

1. Seeds any missing rule (on, at once, digest 07:00, to PS-Admins). Never changes a rule that exists.
2. On its first run in each worker process: when PS_BUILD differs from the settings row notify.last_build, updates the row and raises version_deployed (subject build:<build>). Not at worker start, since the scheduler may start before PostgreSQL answers (open question 47). A restart on the same build raises nothing.
3. At most every 15 minutes per process, the watch checks: drawing_failing when drawing lookups have been failing for more than the rule's hours (default 24; subject drawings, the list of up to 50 in the body), resolved when none is; co_waiting per change order pending review more than the rule's days (default 3) on a live packet (subject co:<id>, the packet's department), resolved when it is no longer waiting.
4. Delivery: up to 200 undispatched events, oldest first, each in its own transaction with FOR UPDATE SKIP LOCKED. No rule: no_rule. Rule off: rule_off. Digest rule: an outbox row, queued. Immediate: recipients resolved now, one message, one notifications_sent row, sent, dry_run, failed or no_recipients. A failed SMTP send is recorded and not retried. AD unavailable while resolving a group: a failed row "directory unavailable", the event stays pending (directory_retry:N), and the third failure dispatches it failed.

A failed or retried send makes the run degraded.

notify-digest (every 5 minutes)

For each rule whose digest time (site time) has passed today and that has outbox rows queued before it: recipients resolved per event, then grouped per address across every rule. One message per address lists every due event (Daily digest: N notifications), with one notifications_sent row per message (event type digest). An outbox row is marked sent only when every address got it. A failed address leaves the rows for the next run, which skips addresses that already got an event, and the job backs off 30 minutes (open question 43). Nothing queued: the run ends without touching AD.

Events from the feed and the drawing job

After each visual-feed and drawing-lookup run, worker/runs.py raises or resolves the feed conditions:

Run status	feed_failed (subject the job name, critical, the first ten errors in the body)	feed_degraded (visual-feed only)
failed	raised	unchanged
degraded	resolved	raised
ok	resolved	resolved
skipped	unchanged	unchanged

drawing_failing comes from the notify-send watch check above, not from the run. Each event goes to its rule's recipients, PS-Admins by default. A broken run connection falls back to a fresh one and never fails the run.

What each event means

Event type	Raised by	Kind	Cleared by
security_alert	Account cooldown, PIN disabled, session ended after wrong PINs	one-shot, 15 min per subject	none
signin_burst	An address over its limit, or the burst threshold across accounts	one-shot, 30 min per subject	none
integrity_failure	integrity	condition	a clean integrity run
feed_failed	visual-feed or drawing-lookup failed	condition	an ok or degraded run
feed_degraded	visual-feed degraded	condition	an ok run
drawing_failing	notify-send watch	condition	no lookup failing that long
ad_reconcile_report	ad-reconcile	one-shot	none
group_missing	ad-reconcile, per group	condition	the group found
backup_stale	backup-check	condition	a good backup
co_waiting	notify-send watch, per change order	condition, department	decided or no longer waiting
qc_reject	A Q.C. reject on a packet, an ITP line reject	one-shot, department	none
version_deployed	notify-send, first run on a new build	one-shot	none
assets_quota	The storage check after every upload and in housekeeping	condition	usage back under the threshold

A condition sends once when it opens and a "Cleared: