PacketStation/docs/FEED.md

Packet Station feed and drawings

Status: phase 3. This file says how the worker reads VISUAL and M-Files and what it does with what it reads. The design is in PLAN.md, "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.

Purpose

The worker is the only part of Packet Station that talks to VISUAL and M-Files. It reads VISUAL through VisualKit and M-Files through MFilesKit, both read-only, on a schedule. It stores what it read in cache tables, turns the rows into projects, units, draft packets and change orders, and works the drawing queue. Screens read PostgreSQL only.

There are three reasons for a scheduled job instead of reads per screen:

• VISUAL production must not be queried on page loads. One modest query every few minutes from one place is fine. One per user per screen is not.
• Turning VISUAL rows into packets is a state change. It runs in one place, in one transaction, with an audit row.
• Drawing lookups and downloads are slow. They belong in the background.

The worker never writes to VISUAL or M-Files.

Worker process

packetstation worker starts the worker. It runs the stack guard, builds the scheduler and blocks. The main thread writes the heartbeat and waits for a stop signal.

Scheduler

• APScheduler 3 with a pool of 4 threads. Each job has coalesce on, max_instances 1 and a misfire grace of 300 seconds.
• Times are scheduled in site time (SITE_TZ, default America/Denver) and stored in UTC.
• Jobs are registered in worker/jobs/__init__.py. Each job has a name, a run function, a cadence function and a pausable flag. A job can name others to chain after it: visual-feed chains drawing-lookup, so every feed run is followed by a drawing run.
• The cadence function returns the seconds to the next run, from the feed settings and the current site time.

Settings re-read

The feed settings live in the database (see "Settings"), not in the env file. The worker keeps one connection that listens on two channels:

• ps_settings: the app sends it when an Admin saves the Feed tab. The worker reloads the settings and recomputes the next run of every job. A new interval or the pause switch applies at once, without a restart.
• ps_job_request: see "Run now" below.

The listening connection reconnects with backoff when it drops.

Pause

When paused is on, scheduled fires of the pausable jobs (visual-feed, drawing-lookup) are skipped. Each skip is recorded as a job_runs row with status skipped and counts {"reason": "paused"}. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping is not paused.

Run now and job_requests

1. Clerical presses Run now on the From VISUAL panel (or an Admin on the Feed tab). The app inserts a job_requests row and sends NOTIFY ps_job_request with {"job": "<name>"} in the same transaction.
2. The worker moves that job's next run to now.
3. The run claims every pending request for the job at its start. Ten clicks make one run. The run's trigger is request when it claimed any.
4. A request that arrives while the job is running is left pending, and the job runs again straight after.

Advisory lock

Each run opens its own connection and takes pg_try_advisory_lock(7310, hashtext(<job name>)). When the lock is held, the run records status skipped with counts {"reason": "already running"} and stops. One run per job at a time, whether the trigger was the schedule, a request, a chain or the command line.

job_runs

Every run writes one job_runs row:

• trigger: schedule, request, chain or cli.
• requested_by: the users whose requests the run claimed.
• started_at, finished_at.
• status: running, then ok, degraded (finished but part of it failed, for example the order read), failed (an exception), skipped or abandoned. On start the worker marks its own host's rows still running as abandoned.
• counts: a JSON object (below).
• errors: at most 20 strings of at most 500 characters each, redacted. Redaction strips anything that looks like password=..., pwd=... or an Authorization: header, and the VISUAL and M-Files passwords themselves when they are in the environment.
• build_version, host, and visual_target ("<server> / <database>") on feed runs.

Feed run counts, all integers: wos_window, wos_tracked, rows, materials, orders, labor_legs, new_drafts, drafts_updated, cos_opened, cos_reaimed, cos_withdrawn, retired, wo_gone_flagged, closed_observed, ignored_golive, projects_created, units_created, auto_released, unmapped_resources, skipped_removed, skipped_closed and suspect (0 or 1). counts["unmapped"] holds {resource: rows} for the top 30 unmapped resources.

Drawing run counts: checked, fetched, unchanged, attached, flagged ({kind: n}), lookup_failed, cos_opened, staged and budget_left.

packetstation job run <name> runs one job once in the foreground with trigger cli, prints its job_runs row as JSON, and exits 1 when the run failed.

Heartbeat

The worker writes the worker_heartbeat row every 30 seconds (time, build, host, pid and the jobs running now), and touches the beat file the container healthcheck reads. The From VISUAL panel shows "Worker not running" when the heartbeat is stale.

Stack guard

At start the worker runs the same stack guard as the app: the test stack refuses a production VISUAL target, and the production stack refuses a VISUAL target that was not set explicitly. Before every feed run the guard runs again with fresh VisualKit settings. A failed guard fails the run. The run records the target it read.

Jobs and cadence

Job	Cadence	Pausable	What it does
visual-feed	interval_shop_minutes inside shop hours (default 5), interval_off_minutes outside (default 30)	yes	The five reads, the cache, the packet rules, auto-release. Chains drawing-lookup.
drawing-lookup	After every feed run, plus every drawing_interval_minutes (default 30)	yes	The drawing queue (see "Drawings").
housekeeping	Daily at 03:30 site time	no	Prunes old rows. The plan keeps the caches 90 days and job_runs 180 days.

Planned, not built in phase 3:

• session-refresh (every 15 minutes): re-resolve groups and the account state for every live session.
• ad-reconcile (nightly 01:30 site time, plus "refresh from AD now"), phase 9.
• integrity (04:00): seals, asset hashes, missing and orphan files, the audit hash chain. Phase 9.
• backup-check (07:15): the newest dump is under 26 hours old. Phase 9.
• notify-send (every 60 seconds) and notify-digest (daily at the digest time). Phase 9.

job_schedule holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for visual-feed, drawing-lookup and housekeeping.

Feed reads

Each feed run opens one VisualKit connection (Visual.from_env()), does five reads outside any database transaction, and closes it. Every value goes through one parser, so VisualKit and the fixture source produce the same data. The site is SITE_ID (default PESCO) in every call.

The window comes from today in site time: since is today minus window_back_days (default 14), until is today plus window_ahead_days (default 30).

1. Released work orders in the window. v.work_orders.released(site, since, until, status=("F",)), paged. Firmed only. until is always passed. This read creates and updates drafts.
2. Tracked work orders by key. v.work_orders.get_many(keys) for every work order key on a live packet that is not filed, plus every ignored work order Clerical pulled in. No status filter and no date filter, so a packet already in the shop keeps getting changes and its closed state. Keys VISUAL no longer returns are collected as missing.
3. Operations, text, parts and materials for the union of reads 1 and 2: operations_for, operation_text_for (the planner instructions, called partspecs), parts.get_many for descriptions. Materials: work_orders.for_bases(bases, type="W", status=("F", "R")), requirements_for, requirement_text_for, parts.get_many and parts.sites_for. Materials are known per leg: the legs for_bases returned are the legs whose BOM the run knows (mat_known). A closed leg's materials are not read, and that does not look like a removed BOM.
4. Order links by work order base and by project, through v.links.orders_for_bases, v.links.orders_for_projects, v.orders.get_many and v.orders.customers. When this read fails, the run is degraded and the apply keeps using the last cached orders.
5. Labor tickets, when labor_tickets is on and the last labor read is more than 30 minutes old: v.labor.tickets_for_work_orders(keys, since=...), summed to hours and ticket count per leg and operation. Employee 9999 is left out (VisualKit's default). No cost column is ever read.

A work order that leaves the window is not a removal. Once a packet tracks it, read 2 keeps reading it whatever its date or status.

Suspect batch

A batch is suspect when:

• the window read returned zero rows and the previous batch had some, or
• more than 5 tracked keys are missing and they are more than 10% of the tracked keys.

A suspect batch is still cached and applied, but without the "gone" marking in the cache and without the "work order gone" step. The run is degraded and the reason is in its counts.

Cache

feed/cache.py writes one batch per run, in one transaction:

• Batch. A visual_batches row: run, source (server and database, or the fixture state), site, window, window rows, tracked keys, missing keys, suspect, counts.
• Rows. The rows are copied into a temporary table and merged into visual_rows, keyed on the typed key (base, lot, split, sub, seq). A new row gets first_batch and last_batch. A row whose hash changed is updated. A row seen again moves last_batch. row_hash is a sha256 of the canonical row (the same canonical form as the fingerprint).
• Materials. The same for visual_materials, keyed on the typed key plus piece, with mat_hash.
• Change rows. Each insert, change, return and disappearance writes a visual_row_changes or visual_material_changes row for the batch, with the row before and after. Kinds: added, changed, gone, returned. Admin reads these as "what VISUAL changed".
• Gone marking. A cached row that is in play (its key is tracked, or it is firmed with its release date inside the window) and was not in this batch gets gone_batch and a gone change row. Materials are in play when their base is. A row that comes back clears gone_batch and gets a returned change row. A suspect batch marks nothing gone.
• Orders. visual_orders is upserted and visual_order_links is replaced for the bases and projects read.
• Labor. visual_labor is upserted per leg and operation.

The apply then reads the rows back from the cache, so it works on one consistent snapshot.

Packet rules

One packet

One packet is one drawing, one lot and one operation (resource). Its match key, packets.visual_key, is <unit>|<resource>|<drawing>, with (no drawing) when the drawing is blank. Several work order operations (legs) can share one packet.

Resource map

Each VISUAL resource maps to a shop department code, fold or ignore (visual_resource_map, seeded from the old build's visual_map.json, edited on the Admin "VISUAL resources" tab).

• A department code makes packets in that department.
• fold makes no packet. The row's operation is listed on the packets of the same work order and lot as a folded operation. A resource starting MAT_ with no map entry is fold.
• ignore makes nothing.
• A resource with no entry makes nothing and is counted under unmapped, so IT can assign it.
• When departments is set in the feed settings, rows routed to other departments are skipped silently.

Project and unit

• The project number is the work order base's prefix as the old build took it.
• The unit is the project number plus the lot zero-padded to 3 digits (L900 and lot 1 make L900001).
• A lot that is not all digits (lot 7A) makes no candidate. The old build dropped them the same way. Hand units may still use such a lot.
• A project or unit that does not exist is created (see the apply, step 1).

Which rows take part

A row takes part when it routes to a department and either its key is tracked (any status, any date), or it is firmed with a release date inside the window. A row that is not tracked and was released before the go-live date is set aside (see "Go-live boundary").

The candidate

Rows are grouped by match key. For each group:

• Legs are deduplicated on (key, seq).
• rel is the earliest release date, want the latest want date.
• rev is the highest numeric rev among the legs. A non-numeric rev only wins when there is no rev yet.
• title is the first leg's description.
• folded lists the fold rows of the same work order and lot. partspecs lists the distinct planner instructions of every row of the same work order and lot, in first-seen order.
• The BOM is built from the materials (the old matBom):
  • A line belongs to a leg when its drawing is blank or matches the leg's drawing, ignoring spaces and case.
  • Per piece the best status wins: R, then F, then U, then anything else.
  • Lines without a part are dropped, except as group headers: a piece that is a multiple of 10 heads the next 9 pieces.
  • The item is the piece number, with -<last part of the work order> added on a repeat.
  • A line is a sub-assembly and n/a when its commodity is WIP or its warehouse is ASY.
  • The location is the warehouse (unless MAIN) and the location, joined by a space.
  • When no leg of the candidate has known materials, the BOM is unknown and the packet's BOM is left alone.

The apply

The apply runs in one transaction with a 5-second lock timeout. A packet row locked by someone else is skipped and picked up next run. Steps, in order:

1. Projects and units. Find each candidate's project through the alias table, so renames resolve. Create a missing project (source visual, no name, needs_details on, due = the latest want date) and its alias. Create a missing unit the same way. Fill the order-derived fields (see "Several open orders"). Candidates under a closed project or unit are skipped (skipped_closed).
2. Match. Look up every candidate key among packets, removed ones included. A key whose only packets are removed makes nothing, ever (skipped_removed). A key with a live packet is existing. Anything else is new.
3. New drafts. Apply the go-live boundary first. Then insert the draft: header, work orders per leg (qty, target hours, op, status, partspecs, drawing, rev), BOM lines (source visual), the visual state row, drawing_state needed and flag drawing_needed (or none and flag no_drawing when the drawing is blank), the fingerprint. Log "Packet created from VISUAL" by "VISUAL feed". Audit packet.created with role system.
4. Drafts updated in place when the fingerprint or the status facts changed (action vis_update_draft):
   • Work orders are replaced, keeping rows whose key and seq match.
   • The BOM is replaced unless Clerical uploaded one by hand.
   • When the drawing or the rev moved: drawing_state goes back to needed, every drawing page is taken off (the file stays in assets), a hand drawing is taken off, any approved rev is cleared, the drawing flags are cleared and drawing_needed is added.
   • The fingerprint, want and release dates, folded operations and partspecs are stored.
   • Log "Updated from VISUAL" with the lines that changed ("work order X added", "X target 1.5 h -> 2 h", "drawing rev 1 -> 2", want date, material lines).
5. Past draft: change orders (action vis_open_co). See "Change orders from the feed". Filed packets get nothing.
6. Retire moved drafts (action vis_retire_draft). A live draft made by the feed, with no drawing pages and no hand uploads, that is not among this run's candidates, and whose every leg now sits on another live candidate, is soft-removed with reason moved. visual.moved records where the legs went. Log "Retired: work orders moved to another packet". It is hidden from the boards and, being removed, is never recreated.
7. Work order gone. Skipped on a suspect batch. Each tracked leg VISUAL did not return gets missing_runs + 1. At 2 the leg gets gone_at, the packet gets flag wo_gone and wo_gone_at, and the log says "Work order no longer in VISUAL". A leg that comes back resets to 0. When no leg is gone any more the flag goes and the log says "Work order back in VISUAL". The feed never removes a packet for this.
8. Closed and cancelled. Each leg's status code is stored, with ticket hours and the time observed. When every leg is C, visual_closed_at is set and the log says "Closed in VISUAL (observed)". Any leg X sets flag cancelled and visual_cancelled_at. The flag goes when no leg is X. A cancelled draft is not removed.
9. Auto-release when auto_release is on and the feed is not paused: up to auto_release_max (default 25) live drafts made by the feed with the drawing attached, no flags at all, and the rev confirmed by the drawing job. A draft whose heat map is unanswered gets "yes" when it has a real material line (not a sub-assembly, not n/a, not consumable) and "no" otherwise. Then action vis_auto_release moves it to the warehouse with an automatic warehouse stage row and a transition row. The policy's preconditions (heat map answered, drawing attached) still apply.
10. Summary. One audit row feed.run with the counts, and one feed event to the screens after commit.

Every packet write in the apply writes a packet_log row (by "VISUAL feed", role system), an audit row (packet.<action>, role system) and a change event, through the same rules as a person's write (see POLICY.md, "Worker writes").

Fingerprint

The fingerprint is a canonical hash over the fields a packet is built from. feed/fingerprint.py computes it per candidate. packets.visual_fp_applied stores the full hash, and packets.visual.fp_parts stores the hash of each part, so the next run can say which parts changed.

Canonical form: text in NFC, line ends as LF, trimmed; null and empty text alike; IDs (work order base, part, drawing, lot, split, sub) upper-cased; lot, split and sub kept as text; seq and piece as integers; numbers normalised (2 and 2.0 are the same); dates in ISO form; lists sorted (legs by base, lot, split, sub, seq; BOM by item). JSON with sorted keys, then sha256.

Field	Part	What a change does past draft
Legs (work order keys and operations on the packet)	major	Opens or re-aims a change order
Quantity per leg	major	Opens or re-aims a change order
Rev	major	Opens or re-aims a change order, waiting for the drawing at the new rev
Drawing number (upper-cased, spaces removed)	major	Opens or re-aims a change order, waiting for the new drawing
BOM (item, part, qty per line)	major	Opens or re-aims a change order
Target hours per leg	minor	Opens a minor change order: Clerical approval only, no department verifies
Want date	minor	Opens a minor change order
Release date	never	Nothing
Folded operations	never	Nothing
Partspecs (planner instructions)	never	Nothing
Title	never	Nothing
Work order status	never	Nothing (status is observed separately: closed, cancelled)
Issued quantity, warehouse, location	never	Nothing
Orders, labor, department, operation	never	Nothing

On a draft any fingerprint change updates the draft in place, major or minor. A change order is only for packets past draft. Tests in tests/server/test_fingerprint.py show each row.

Go-live boundary

With no migration from the M: install, the first run would make drafts for every firmed work order in the window, including jobs already running on paper or on M:. The golive_date setting keeps them out:

• A row that is not tracked and was released before the go-live date makes no candidate.
• A candidate none of whose legs is tracked and whose earliest release is before the go-live date is not created. Its legs go into visual_ignored_wos (ignored_golive).
• The From VISUAL panel lists the ignored work orders, newest release first. Pull in asks a reason, records who and when, and requests a feed run. A pulled-in work order is tracked from then on, so the next run makes its draft whatever its release date.

Jobs already open on M: finish there.

Change orders from the feed

A packet past draft (warehouse to complete) is never changed silently. A fingerprint change opens a change order instead.

• Open. A changed fingerprint opens a change order with origin visual, review pending, ECN VISUAL, issued by nobody. It carries vis (from and to drawing and rev, the list of changes in words, the legs, the BOM, want and release dates, folded operations, partspecs and the fingerprint parts), target_fp, minor, and waiting ({dwg, rev, since}) when the drawing or rev changed and the new drawing is still needed.
• One at a time. At most one pending feed change order per packet (origin visual or drawing). A further change re-aims it: reason, vis, waiting, target_fp and minor are updated, the move is appended to vis.moved, and a staged drawing is kept when the target drawing and rev did not change.
• Unchanged. A pending change order with the same target does nothing. A rejected one with the same target does nothing either: Clerical rejected exactly this.
• Withdraw. When VISUAL goes back to what the packet has, the pending change order is withdrawn: review rejected, rejected by nobody, with the reason in the log.
• Follow on. An approved change order with the same target does nothing. A change after an approval opens a fresh change order.
• Minor. When only target hours or the want date changed, the change order is minor. It needs Clerical approval only. On approval no department is asked to verify.
• Hold. A pending change order holds nothing unless Clerical puts it on hold. While held it blocks the packet's held steps (see v_open_cos in SCHEMA.md).
• Approval. Clerical approves with a signature. When the drawing changed, approval needs the new drawing staged first ("The new drawing has not been fetched from M-Files yet."). Approval applies vis: work orders from vis.legs, the BOM unless it was uploaded by hand, the header drawing and rev, the fingerprint, want and release dates. A staged drawing replaces the old drawing pages: the old pages are voided (never deleted) and the new pages added, with the page changes in co_items. A major change order then asks the departments it touches to verify, and holds the packet until they have. A minor one asks nobody.
• Reject. Clerical rejects with a reason and a signature.

A packet at complete still gets a change order. A filed packet gets nothing.

Drawings

The drawing-lookup job finds each packet's drawing in M-Files through MFilesKit, downloads it, and attaches it to drafts.

How MFilesKit finds a drawing

This is what mfileskit/drawings.py does. MFilesKit's own docs/packet-station.md is stale in two places, noted below.

• Key. Hyphens and spaces removed, upper-cased: L073-76-2 becomes L073762.
• Search by Name property, not folders. The released PDF is searched in class "Drawing Production (PDF)" by the Name property starting with the key, and a hit counts only when its name is the key or the key plus X. There is no project-folder rule any more (the P-521 to H777 mapping in the MFilesKit docs is gone).
• Released rev. Only a document whose Controlled Status (or workflow state) is Effective counts. The rev is its Controlled Version Label, else the number after R in its title. The highest released rev wins.
• Drafting rev. Working copies are searched by Controlled Version Title. Copies in a done status (released, effective, retired, obsolete) or a done state (drawing released to shop, drawing already exists, cancel drawing) are skipped. The drafting rev is the highest label among the rest. Its workflow state maps to "still being drawn (unapproved)" or "in the drafting check cycle". (The MFilesKit docs still say "the higher of the Unapproved rev and the Check Cycle rev"; that is stale.)
• In drafting. A Drafting Assignment named for the key means the drawing is in drafting.
• H sheets. A drawing like K978-01-6H13 has no object of its own. When nothing is found for it (no released PDF, no working copy, no assignment), the lookup tries the base drawing K978-01-6 and reports base. The packet note then says "sheet of drawing K978-01-6".
• A search that hits the 500-result cap is a failed lookup, so a wrong rev is never reported.

Queue and priorities

The queue covers live packets made by the feed, not filed, with a drawing number, in the departments in scope. It is distinct by drawing number, and the first item's priority wins:

1. The VISUAL rev changed since the last lookup of that drawing, or a pending change order waits for a drawing at a target rev.
2. Drafts with drawing_state needed.
3. Failed lookups whose backoff is over (15 minutes doubled per failure, at most 24 hours).
4. Rechecks due: last checked more than drawing_recheck_hours ago (default 4).

Budget

At most drawing_budget lookups per run (default 60). The rest wait for the next run. The worker opens one M-Files vault per process and a new DrawingLookup per run, so its cache never outlives a run. When the vault cannot be opened, the run is skipped and degraded with one error.

Download

The file is downloaded only when the object id and version or the file id and version differ from the last lookup, or nothing is stored yet. The download goes to a temporary file, is checked for the PDF magic bytes and the upload size limit, hashed with sha256 while streaming, then fsynced and renamed into the assets volume. A file with the same sha256 and size reuses the existing asset. Pages are counted with pypdfium2 (pypdf as a fallback). Each fetch writes an audit row drawing.fetched.

Decision table

drawings/decide.py decides per packet sharing the drawing. Rows are checked top to bottom.

Case	Decision	Flag	What happens
The lookup failed	lookup_failed	lookup_failed	What is attached stays. drawing_state becomes lookup_failed only if it was needed. Retried with backoff.
No released rev, but a drafting rev or a drafting assignment	not_released	not_released	Note "Not released in M-Files yet (sheet of ) - drafting status: rev N ".
No released rev and nothing in drafting	not_found	not_found	Note "Drawing not in M-Files - check the drawing number in VISUAL".
VISUAL's rev is not a whole number	bad_rev	bad_rev	Not attached.
Revs differ, and no approved rev matches M-Files	rev_mismatch	rev_mismatch	Note "Rev mismatch: VISUAL calls rev V, M-Files has rev N", plus the drafting status. The M-Files copy is fetched and staged for Clerical Supervisor approval. No change order: VISUAL did not change.
Revs match (or the approved rev matches), draft	attach	none	Drawing pages attached, drawing_state attached, drawing flags cleared, rev confirmed. Log "Drawing attached from M-Files".
Revs match, draft with a hand drawing at the same rev	unchanged	none	The hand drawing stays.
Revs match, past draft, same file (same sha256)	unchanged	none	Nothing.
Revs match, past draft, same rev but a new file	open_co	none	Drawing-origin change order: "M-Files re-released rev N of (new file)".
Past draft, M-Files released a higher rev than the one attached	open_co	none	Drawing-origin change order: "M-Files released rev N of ; this packet has rev M". Never a silent swap.
A pending feed change order waits for this drawing at the rev M-Files now has	stage_for_co	none	The fetched file is staged on the change order and waiting is cleared. Log "Drawing for change order N ready for Clerical".

A flag decision sets drawing_state, replaces the drawing flags with the decision's flag, stores the M-Files answer in visual.appr, and clears the rev confirmation. On a draft it is a vis_update_draft write. Past draft it is a vis_open_co write logged "Drawing status from M-Files", since that is the only system action allowed past draft.

"Drawing needed" blocks the warehouse step through the drawing_attached precondition.

Rev mismatch staging and approval

When M-Files has a different released rev from VISUAL, the M-Files copy is fetched and its asset is kept on the drawing_lookups row. A Clerical Supervisor can approve it for one draft with a reason and a signature (action rev_approve). The pages are attached, the drawing flags are cleared, and visual.rev_ok records the rev, the VISUAL rev, who, when and why. The log says "Different rev approved by hand" with "rev N in place of VISUAL rev M". Without a staged copy the answer is "nothing staged".

Hand drawing taken off on a new rev

A draft whose VISUAL drawing or rev moves loses every drawing page, a hand drawing included (step 4 of the apply). The drawing job then looks up the new rev. A hand drawing at the same rev is left alone.

Drawing-origin change orders

A drawing change on a packet past draft opens a change order with origin drawing, ECN M-FILES, the new file already staged, and no target fingerprint. It counts as the packet's one pending feed change order. Approval voids the old drawing pages and adds the new ones, as for a feed change order.

Flags

Flags live in packets.flags. A draft with any flag shows in Clerical's Flagged column. Auto-release never takes a packet with a flag. The find bar filters by flag kind.

Flag	Text shown	What Clerical does
drawing_needed	Made from VISUAL - drawing not attached yet	Usually nothing: the drawing job attaches it. Run now to hurry it. Upload by hand only if it stays stuck.
rev_mismatch	The M-Files note, for example "Rev mismatch: VISUAL calls rev 2, M-Files has rev 3"	Get the rev corrected in VISUAL, or a Clerical Supervisor approves the staged M-Files rev with a reason and a signature.
not_released	The note: "Not released in M-Files yet ... drafting status: rev N "	Wait for drafting to release it, or upload the drawing by hand with a reason and a signature.
not_found	Drawing not in M-Files - check the drawing number in VISUAL	Check the drawing number in VISUAL and have it fixed there. Or upload by hand.
lookup_failed	M-Files lookup failed: (the reason)	Nothing at first: the job retries with backoff. If it persists, tell IT.
bad_rev	VISUAL rev is not a number - drawing not attached	Have the rev fixed in VISUAL, or upload by hand.
no_drawing	(the draft has no drawing number)	Have the drawing number added in VISUAL, or upload by hand.
wo_gone	Work order no longer in VISUAL	Check VISUAL. If the job was deleted on purpose, remove the draft by hand (or handle the in-shop packet). The flag clears itself if the work order comes back.
cancelled	Work order cancelled in VISUAL	Decide: remove the draft by hand, or leave it if the cancel was a mistake. The feed never removes it.
hand_drawing	Drawing uploaded by hand - check it and send it on	Check the drawing and send the draft on by hand.
hand_bom	BOM uploaded by hand - check it and send it on	Check the BOM and send it on. "Use VISUAL's BOM again" puts the feed's BOM back.

With both hand flags the text is "Drawing and BOM uploaded by hand - check it and send it on". Hand drafts keep their phase 2 text.

"Closed in VISUAL" is not a flag. It is an observed state (visual_closed_at), shown on the packet. Filing a Data Book packet needs it, as the vclosed_if_databook precondition says.

Hand uploads

On a flagged draft made by the feed (or one already uploaded by hand), Clerical can:

• Upload a drawing (PDF) with a reason and a signature. It replaces the drawing pages, sets drawing_state hand, clears the drawing flags and adds hand_drawing.
• Upload a BOM (XLSX or CSV) with a reason and a signature. The server parses it: the header row is found in the first 20 rows by the old build's column patterns (item, qty, part, description, length, material, drawing reference); qty and a description or part are needed; empty rows are skipped; items default to the row number. It replaces the BOM lines (source hand) and adds hand_bom. The feed then leaves the BOM alone.
• Use VISUAL's BOM again: rebuilds the BOM from the cache and removes hand_bom.

An unflagged draft refuses a hand upload ("not flagged").

Several open orders

A project can have several customer orders in VISUAL. One function, feed/orders.choose, picks the order for a project or a unit. The first rule that finds one wins:

1. Pin. Clerical pinned an order (a customer order ID among the project's orders) on the project or unit, with a reason. Via pin.
2. Base. The order linked to the work order base itself. Via base.
3. Unit top job. The project order whose line's top job is the unit number (open orders first). Via unit_top_job.
4. Project open. The first open order of the project (status F, H or R), newest first. Via project_open.
5. Project newest. The first order of the project, open or not. Via project_newest.

When nothing matches there is no order.

Provenance

The chosen order is stored as order_ref on the project and the unit: order ID, via, status, customer name, customer PO, order, desired ship and promise dates, and when it was chosen. The fields derived from it:

Field	From the order
customer	customer name
po	customer PO reference
order date	order date
ship date	promise date, else desired ship date

For each field the feed always writes <field>_visual. It writes the field itself, with <field>_src = 'visual', only when <field>_src is empty or already visual. A value Clerical typed (_src = 'hand') is never overwritten. Each change is audited as project.updated with src: visual.

The pin

Clerical can pin an order on a project or a unit (order_pin, with a reason, who and when) from the project page, which lists every cached order of the project. The feed re-derives on its next run. Clearing the pin goes back to the rules.

Settings

The feed settings are one settings row, key feed, a JSON object. An Admin edits them on the Admin page, Feed tab (password session only). Unknown keys are ignored and missing keys take the default. Every save is validated, bumps the row version, is audited as feed.settings_changed with each changed key's before and after, and notifies the worker.

Setting	Default	Range	Meaning
interval_shop_minutes	5	1 to 1440	Feed interval inside shop hours
interval_off_minutes	30	1 to 1440	Feed interval outside shop hours
shop_days	Monday to Friday ([0,1,2,3,4])	0 (Monday) to 6	Shop days, site time
shop_start	06:00	HH:MM	Start of shop hours, site time
shop_end	18:00	HH:MM	End of shop hours, site time
window_back_days	14	0 to 120	Window start, days before today
window_ahead_days	30	0 to 365	Window end, days after today
golive_date	none	a date	Work orders released before it are ignored
departments	all ([])	shop codes	Departments to make packets for. Empty means every department the resource map routes to.
drawing_recheck_hours	4	1 to 168	Recheck each drawing after this long
drawing_budget	60	1 to 500	Drawing lookups per run
drawing_interval_minutes	30	not limited	The drawing job's own cadence
labor_tickets	on	on or off	Read labor tickets, at most every 30 minutes
auto_release	on	on or off	Release ready drafts to the warehouse
auto_release_max	25	not limited	Most drafts auto-released per run
paused	off	on or off	Stops scheduled feed and drawing runs and auto-release. Run now still works.

The Feed tab also shows the next scheduled run and the last ten runs. The "VISUAL resources" tab edits the resource map: each resource with its rows in the last batch, target (a shop department, Fold, Ignore or not assigned) and a confirmed tick. Every map change is audited.

Fixture sources

For CI and local runs the worker can read files instead of VISUAL and M-Files:

• FEED_SOURCE=fixture reads FIXTURE_DIR/feed.<state>.json, a file in the exact shape of VisualKit's examples/packet_station_feed.py output (meta, rows, materials, orders_by_base, orders_by_project) plus an optional labor list. The state is read from FIXTURE_DIR/state (default 1). It returns the firmed rows released inside the window, plus every row whose key is tracked; tracked keys with no rows are missing.
• DRAWING_SOURCE=fixture reads FIXTURE_DIR/drawings.<state>.json, a map from drawing number to its rev, file name, drafting state and ids. A drawing not in the file is not found. A download copies FIXTURE_DIR/<file>.
• FIXTURE_DIR defaults to /fixtures.
• The production stack refuses a fixture source at start.

tests/fixtures/generator.py writes the files: python -m tests.fixtures.generator --out DIR makes feed.1.json, feed.2.json, drawings.1.json, drawings.2.json, the one-page PDFs they name, and state = 1. It is deterministic. State 1 covers the packet rules (fold, ignore, an unmapped resource, a lot 7A, two operations on one work order, no drawing, a non-numeric rev, the BOM rules, a work order before go-live, closed and cancelled legs, several open orders). State 2 changes a rev, a quantity and a target, removes one leg and moves another, so the change order, minor, gone and retire rules run. packetstation fixtures use N --dir DIR switches the state. The Playwright scenario writes the state file itself. See DEPLOY.md for the CI step and tests/fixtures/README.md.

Real job data never enters git or CI. Recordings from VISUAL DEV stay local, in tests/fixtures/local/ and recordings/.

What the screens read

Screens read PostgreSQL only, never VISUAL or M-Files per request:

• Packets, flags, work order status, closed and cancelled state, ticket hours and the chosen order come from the packet tables and packet_visual_state.
• The From VISUAL panel reads /api/feed/overview: pause state, next run, the heartbeat, the last ten feed and drawing runs, the ignored work orders, unmapped resources and projects without names.
• Admin reads the cache tables (/api/feed/batches, /api/feed/changes) for "what VISUAL changed".
• The project page reads the cached orders from visual_order_links and visual_orders.

Scheduled runs come from the worker's own clock, not from an open browser.