Status: phase 3. This file says how the worker reads VISUAL and M-Files and what it does with what it reads. The design is in [PLAN.md](PLAN.md), "Worker, feed and drawings". When this file and the code disagree, the code is the rule and this file is out of date.
The worker is the only part of Packet Station that talks to VISUAL and M-Files. It reads VISUAL through VisualKit and M-Files through MFilesKit, both read-only, on a schedule. It stores what it read in cache tables, turns the rows into projects, units, draft packets and change orders, and works the drawing queue. Screens read PostgreSQL only.
There are three reasons for a scheduled job instead of reads per screen:
- VISUAL production must not be queried on page loads. One modest query every few minutes from one place is fine. One per user per screen is not.
- Turning VISUAL rows into packets is a state change. It runs in one place, in one transaction, with an audit row.
- Drawing lookups and downloads are slow. They belong in the background.
`packetstation worker` starts the worker. It runs the stack guard, builds the scheduler and blocks. The main thread writes the heartbeat and waits for a stop signal.
### Scheduler
- APScheduler 3 with a pool of 4 threads. Each job has `coalesce` on, `max_instances` 1 and a misfire grace of 300 seconds.
- Times are scheduled in site time (`SITE_TZ`, default `America/Denver`) and stored in UTC.
- Jobs are registered in `worker/jobs/__init__.py`. Each job has a name, a run function, a cadence function and a `pausable` flag. A job can name others to chain after it: `visual-feed` chains `drawing-lookup`, so every feed run is followed by a drawing run.
- The cadence function returns the seconds to the next run, from the feed settings and the current site time.
### Settings re-read
The feed settings live in the database (see "Settings"), not in the env file. The worker keeps one connection that listens on two channels:
-`ps_settings`: the app sends it when an Admin saves the Feed tab. The worker reloads the settings and recomputes the next run of every job. A new interval or the pause switch applies at once, without a restart.
-`ps_job_request`: see "Run now" below.
The listening connection reconnects with backoff when it drops.
### Pause
When `paused` is on, scheduled fires of the pausable jobs (`visual-feed`, `drawing-lookup`) are skipped. Each skip is recorded as a `job_runs` row with status `skipped` and counts `{"reason": "paused"}`. Auto-release does not run while paused. "Run now" still works while paused. Housekeeping is not paused.
### Run now and job_requests
1. Clerical presses **Run now** on the From VISUAL panel (or an Admin on the Feed tab). The app inserts a `job_requests` row and sends `NOTIFY ps_job_request` with `{"job": "<name>"}` in the same transaction.
2. The worker moves that job's next run to now.
3. The run claims every pending request for the job at its start. Ten clicks make one run. The run's trigger is `request` when it claimed any.
4. A request that arrives while the job is running is left pending, and the job runs again straight after.
### Advisory lock
Each run opens its own connection and takes `pg_try_advisory_lock(7310, hashtext(<job name>))`. When the lock is held, the run records status `skipped` with counts `{"reason": "already running"}` and stops. One run per job at a time, whether the trigger was the schedule, a request, a chain or the command line.
### job_runs
Every run writes one `job_runs` row:
-`trigger`: `schedule`, `request`, `chain` or `cli`.
-`requested_by`: the users whose requests the run claimed.
-`started_at`, `finished_at`.
-`status`: `running`, then `ok`, `degraded` (finished but part of it failed, for example the order read), `failed` (an exception), `skipped` or `abandoned`. On start the worker marks its own host's rows still `running` as `abandoned`.
-`counts`: a JSON object (below).
-`errors`: at most 20 strings of at most 500 characters each, redacted. Redaction strips anything that looks like `password=...`, `pwd=...` or an `Authorization:` header, and the VISUAL and M-Files passwords themselves when they are in the environment.
-`build_version`, `host`, and `visual_target` (`"<server> / <database>"`) on feed runs.
Feed run counts, all integers: `wos_window`, `wos_tracked`, `rows`, `materials`, `orders`, `labor_legs`, `new_drafts`, `drafts_updated`, `cos_opened`, `cos_reaimed`, `cos_withdrawn`, `retired`, `wo_gone_flagged`, `closed_observed`, `ignored_golive`, `projects_created`, `units_created`, `auto_released`, `unmapped_resources`, `skipped_removed`, `skipped_closed` and `suspect` (0 or 1). `counts["unmapped"]` holds `{resource: rows}` for the top 30 unmapped resources.
Drawing run counts: `checked`, `fetched`, `unchanged`, `attached`, `flagged` (`{kind: n}`), `lookup_failed`, `cos_opened`, `staged` and `budget_left`.
`packetstation job run <name>` runs one job once in the foreground with trigger `cli`, prints its `job_runs` row as JSON, and exits 1 when the run failed.
### Heartbeat
The worker writes the `worker_heartbeat` row every 30 seconds (time, build, host, pid and the jobs running now), and touches the beat file the container healthcheck reads. The From VISUAL panel shows "Worker not running" when the heartbeat is stale.
### Stack guard
At start the worker runs the same stack guard as the app: the test stack refuses a production VISUAL target, and the production stack refuses a VISUAL target that was not set explicitly. Before every feed run the guard runs again with fresh VisualKit settings. A failed guard fails the run. The run records the target it read.
| `visual-feed` | `interval_shop_minutes` inside shop hours (default 5), `interval_off_minutes` outside (default 30) | yes | The five reads, the cache, the packet rules, auto-release. Chains `drawing-lookup`. |
| `drawing-lookup` | After every feed run, plus every `drawing_interval_minutes` (default 30) | yes | The drawing queue (see "Drawings"). |
| `housekeeping` | Daily at 03:30 site time | no | Prunes old rows. The plan keeps the caches 90 days and `job_runs` 180 days. |
Planned, not built in phase 3:
-`session-refresh` (every 15 minutes): re-resolve groups and the account state for every live session.
-`ad-reconcile` (nightly 01:30 site time, plus "refresh from AD now"), phase 9.
-`integrity` (04:00): seals, asset hashes, missing and orphan files, the audit hash chain. Phase 9.
-`backup-check` (07:15): the newest dump is under 26 hours old. Phase 9.
-`notify-send` (every 60 seconds) and `notify-digest` (daily at the digest time). Phase 9.
`job_schedule` holds each job's next run, interval and pause state for the Admin tab. Migration 0004 seeds rows for `visual-feed`, `drawing-lookup` and `housekeeping`.
Each feed run opens one VisualKit connection (`Visual.from_env()`), does five reads outside any database transaction, and closes it. Every value goes through one parser, so VisualKit and the fixture source produce the same data. The site is `SITE_ID` (default `PESCO`) in every call.
The window comes from today in site time: `since` is today minus `window_back_days` (default 14), `until` is today plus `window_ahead_days` (default 30).
1.**Released work orders in the window.**`v.work_orders.released(site, since, until, status=("F",))`, paged. Firmed only. `until` is always passed. This read creates and updates drafts.
2.**Tracked work orders by key.**`v.work_orders.get_many(keys)` for every work order key on a live packet that is not filed, plus every ignored work order Clerical pulled in. No status filter and no date filter, so a packet already in the shop keeps getting changes and its closed state. Keys VISUAL no longer returns are collected as missing.
3.**Operations, text, parts and materials** for the union of reads 1 and 2: `operations_for`, `operation_text_for` (the planner instructions, called partspecs), `parts.get_many` for descriptions. Materials: `work_orders.for_bases(bases, type="W", status=("F", "R"))`, `requirements_for`, `requirement_text_for`, `parts.get_many` and `parts.sites_for`. Materials are known per leg: the legs `for_bases` returned are the legs whose BOM the run knows (`mat_known`). A closed leg's materials are not read, and that does not look like a removed BOM.
4.**Order links** by work order base and by project, through `v.links.orders_for_bases`, `v.links.orders_for_projects`, `v.orders.get_many` and `v.orders.customers`. When this read fails, the run is degraded and the apply keeps using the last cached orders.
5.**Labor tickets**, when `labor_tickets` is on and the last labor read is more than 30 minutes old: `v.labor.tickets_for_work_orders(keys, since=...)`, summed to hours and ticket count per leg and operation. Employee 9999 is left out (VisualKit's default). No cost column is ever read.
A work order that leaves the window is not a removal. Once a packet tracks it, read 2 keeps reading it whatever its date or status.
### Suspect batch
A batch is suspect when:
- the window read returned zero rows and the previous batch had some, or
- more than 5 tracked keys are missing and they are more than 10% of the tracked keys.
A suspect batch is still cached and applied, but without the "gone" marking in the cache and without the "work order gone" step. The run is degraded and the reason is in its counts.
## Cache
`feed/cache.py` writes one batch per run, in one transaction:
- **Batch.** A `visual_batches` row: run, source (server and database, or the fixture state), site, window, window rows, tracked keys, missing keys, suspect, counts.
- **Rows.** The rows are copied into a temporary table and merged into `visual_rows`, keyed on the typed key (base, lot, split, sub, seq). A new row gets `first_batch` and `last_batch`. A row whose hash changed is updated. A row seen again moves `last_batch`. `row_hash` is a sha256 of the canonical row (the same canonical form as the fingerprint).
- **Materials.** The same for `visual_materials`, keyed on the typed key plus piece, with `mat_hash`.
- **Change rows.** Each insert, change, return and disappearance writes a `visual_row_changes` or `visual_material_changes` row for the batch, with the row before and after. Kinds: `added`, `changed`, `gone`, `returned`. Admin reads these as "what VISUAL changed".
- **Gone marking.** A cached row that is in play (its key is tracked, or it is firmed with its release date inside the window) and was not in this batch gets `gone_batch` and a `gone` change row. Materials are in play when their base is. A row that comes back clears `gone_batch` and gets a `returned` change row. A suspect batch marks nothing gone.
- **Orders.** `visual_orders` is upserted and `visual_order_links` is replaced for the bases and projects read.
- **Labor.** `visual_labor` is upserted per leg and operation.
The apply then reads the rows back from the cache, so it works on one consistent snapshot.
One packet is one drawing, one lot and one operation (resource). Its match key, `packets.visual_key`, is `<unit>|<resource>|<drawing>`, with `(no drawing)` when the drawing is blank. Several work order operations (legs) can share one packet.
### Resource map
Each VISUAL resource maps to a shop department code, `fold` or `ignore` (`visual_resource_map`, seeded from the old build's `visual_map.json`, edited on the Admin "VISUAL resources" tab).
- A department code makes packets in that department.
-`fold` makes no packet. The row's operation is listed on the packets of the same work order and lot as a folded operation. A resource starting `MAT_` with no map entry is fold.
-`ignore` makes nothing.
- A resource with no entry makes nothing and is counted under `unmapped`, so IT can assign it.
- When `departments` is set in the feed settings, rows routed to other departments are skipped silently.
### Project and unit
- The project number is the work order base's prefix as the old build took it.
- The unit is the project number plus the lot zero-padded to 3 digits (`L900` and lot `1` make `L900001`).
- A lot that is not all digits (lot `7A`) makes no candidate. The old build dropped them the same way. Hand units may still use such a lot.
- A project or unit that does not exist is created (see the apply, step 1).
### Which rows take part
A row takes part when it routes to a department and either its key is tracked (any status, any date), or it is firmed with a release date inside the window. A row that is not tracked and was released before the go-live date is set aside (see "Go-live boundary").
### The candidate
Rows are grouped by match key. For each group:
- Legs are deduplicated on (key, seq).
-`rel` is the earliest release date, `want` the latest want date.
-`rev` is the highest numeric rev among the legs. A non-numeric rev only wins when there is no rev yet.
-`title` is the first leg's description.
-`folded` lists the fold rows of the same work order and lot. `partspecs` lists the distinct planner instructions of every row of the same work order and lot, in first-seen order.
- The BOM is built from the materials (the old `matBom`):
- A line belongs to a leg when its drawing is blank or matches the leg's drawing, ignoring spaces and case.
- Per piece the best status wins: R, then F, then U, then anything else.
- Lines without a part are dropped, except as group headers: a piece that is a multiple of 10 heads the next 9 pieces.
- The item is the piece number, with `-<last part of the work order>` added on a repeat.
- A line is a sub-assembly and n/a when its commodity is WIP or its warehouse is ASY.
- The location is the warehouse (unless MAIN) and the location, joined by a space.
- When no leg of the candidate has known materials, the BOM is unknown and the packet's BOM is left alone.
### The apply
The apply runs in one transaction with a 5-second lock timeout. A packet row locked by someone else is skipped and picked up next run. Steps, in order:
1.**Projects and units.** Find each candidate's project through the alias table, so renames resolve. Create a missing project (source `visual`, no name, `needs_details` on, due = the latest want date) and its alias. Create a missing unit the same way. Fill the order-derived fields (see "Several open orders"). Candidates under a closed project or unit are skipped (`skipped_closed`).
2.**Match.** Look up every candidate key among packets, removed ones included. A key whose only packets are removed makes nothing, ever (`skipped_removed`). A key with a live packet is existing. Anything else is new.
3.**New drafts.** Apply the go-live boundary first. Then insert the draft: header, work orders per leg (qty, target hours, op, status, partspecs, drawing, rev), BOM lines (source `visual`), the visual state row, `drawing_state``needed` and flag `drawing_needed` (or `none` and flag `no_drawing` when the drawing is blank), the fingerprint. Log "Packet created from VISUAL" by "VISUAL feed". Audit `packet.created` with role `system`.
4.**Drafts updated in place** when the fingerprint or the status facts changed (action `vis_update_draft`):
- Work orders are replaced, keeping rows whose key and seq match.
- The BOM is replaced unless Clerical uploaded one by hand.
- When the drawing or the rev moved: `drawing_state` goes back to `needed`, every drawing page is taken off (the file stays in assets), a hand drawing is taken off, any approved rev is cleared, the drawing flags are cleared and `drawing_needed` is added.
- The fingerprint, want and release dates, folded operations and partspecs are stored.
- Log "Updated from VISUAL" with the lines that changed ("work order X added", "X target 1.5 h -> 2 h", "drawing rev 1 -> 2", want date, material lines).
5.**Past draft: change orders** (action `vis_open_co`). See "Change orders from the feed". Filed packets get nothing.
6.**Retire moved drafts** (action `vis_retire_draft`). A live draft made by the feed, with no drawing pages and no hand uploads, that is not among this run's candidates, and whose every leg now sits on another live candidate, is soft-removed with reason `moved`. `visual.moved` records where the legs went. Log "Retired: work orders moved to another packet". It is hidden from the boards and, being removed, is never recreated.
7.**Work order gone.** Skipped on a suspect batch. Each tracked leg VISUAL did not return gets `missing_runs + 1`. At 2 the leg gets `gone_at`, the packet gets flag `wo_gone` and `wo_gone_at`, and the log says "Work order <key> no longer in VISUAL". A leg that comes back resets to 0. When no leg is gone any more the flag goes and the log says "Work order back in VISUAL". The feed never removes a packet for this.
8.**Closed and cancelled.** Each leg's status code is stored, with ticket hours and the time observed. When every leg is C, `visual_closed_at` is set and the log says "Closed in VISUAL (observed)". Any leg X sets flag `cancelled` and `visual_cancelled_at`. The flag goes when no leg is X. A cancelled draft is not removed.
9.**Auto-release** when `auto_release` is on and the feed is not paused: up to `auto_release_max` (default 25) live drafts made by the feed with the drawing attached, no flags at all, and the rev confirmed by the drawing job. A draft whose heat map is unanswered gets "yes" when it has a real material line (not a sub-assembly, not n/a, not consumable) and "no" otherwise. Then action `vis_auto_release` moves it to the warehouse with an automatic warehouse stage row and a transition row. The policy's preconditions (heat map answered, drawing attached) still apply.
10.**Summary.** One audit row `feed.run` with the counts, and one `feed` event to the screens after commit.
Every packet write in the apply writes a `packet_log` row (by "VISUAL feed", role `system`), an audit row (`packet.<action>`, role `system`) and a change event, through the same rules as a person's write (see [POLICY.md](POLICY.md), "Worker writes").
The fingerprint is a canonical hash over the fields a packet is built from. `feed/fingerprint.py` computes it per candidate. `packets.visual_fp_applied` stores the full hash, and `packets.visual.fp_parts` stores the hash of each part, so the next run can say which parts changed.
Canonical form: text in NFC, line ends as LF, trimmed; null and empty text alike; IDs (work order base, part, drawing, lot, split, sub) upper-cased; lot, split and sub kept as text; seq and piece as integers; numbers normalised (`2` and `2.0` are the same); dates in ISO form; lists sorted (legs by base, lot, split, sub, seq; BOM by item). JSON with sorted keys, then sha256.
| Field | Part | What a change does past draft |
|---|---|---|
| Legs (work order keys and operations on the packet) | major | Opens or re-aims a change order |
| Quantity per leg | major | Opens or re-aims a change order |
| Rev | major | Opens or re-aims a change order, waiting for the drawing at the new rev |
| Drawing number (upper-cased, spaces removed) | major | Opens or re-aims a change order, waiting for the new drawing |
| BOM (item, part, qty per line) | major | Opens or re-aims a change order |
| Target hours per leg | minor | Opens a minor change order: Clerical approval only, no department verifies |
| Want date | minor | Opens a minor change order |
| Release date | never | Nothing |
| Folded operations | never | Nothing |
| Partspecs (planner instructions) | never | Nothing |
| Title | never | Nothing |
| Work order status | never | Nothing (status is observed separately: closed, cancelled) |
| Issued quantity, warehouse, location | never | Nothing |
| Orders, labor, department, operation | never | Nothing |
On a draft any fingerprint change updates the draft in place, major or minor. A change order is only for packets past draft. Tests in `tests/server/test_fingerprint.py` show each row.
With no migration from the M: install, the first run would make drafts for every firmed work order in the window, including jobs already running on paper or on M:. The `golive_date` setting keeps them out:
- A row that is not tracked and was released before the go-live date makes no candidate.
- A candidate none of whose legs is tracked and whose earliest release is before the go-live date is not created. Its legs go into `visual_ignored_wos` (`ignored_golive`).
- The From VISUAL panel lists the ignored work orders, newest release first. **Pull in** asks a reason, records who and when, and requests a feed run. A pulled-in work order is tracked from then on, so the next run makes its draft whatever its release date.
Jobs already open on M: finish there.
## Change orders from the feed
A packet past draft (warehouse to complete) is never changed silently. A fingerprint change opens a change order instead.
- **Open.** A changed fingerprint opens a change order with origin `visual`, review `pending`, ECN `VISUAL`, issued by nobody. It carries `vis` (from and to drawing and rev, the list of changes in words, the legs, the BOM, want and release dates, folded operations, partspecs and the fingerprint parts), `target_fp`, `minor`, and `waiting` (`{dwg, rev, since}`) when the drawing or rev changed and the new drawing is still needed.
- **One at a time.** At most one pending feed change order per packet (origin `visual` or `drawing`). A further change re-aims it: reason, `vis`, `waiting`, `target_fp` and `minor` are updated, the move is appended to `vis.moved`, and a staged drawing is kept when the target drawing and rev did not change.
- **Unchanged.** A pending change order with the same target does nothing. A rejected one with the same target does nothing either: Clerical rejected exactly this.
- **Withdraw.** When VISUAL goes back to what the packet has, the pending change order is withdrawn: review `rejected`, rejected by nobody, with the reason in the log.
- **Follow on.** An approved change order with the same target does nothing. A change after an approval opens a fresh change order.
- **Minor.** When only target hours or the want date changed, the change order is minor. It needs Clerical approval only. On approval no department is asked to verify.
- **Hold.** A pending change order holds nothing unless Clerical puts it on hold. While held it blocks the packet's held steps (see `v_open_cos` in [SCHEMA.md](SCHEMA.md)).
- **Approval.** Clerical approves with a signature. When the drawing changed, approval needs the new drawing staged first ("The new drawing has not been fetched from M-Files yet."). Approval applies `vis`: work orders from `vis.legs`, the BOM unless it was uploaded by hand, the header drawing and rev, the fingerprint, want and release dates. A staged drawing replaces the old drawing pages: the old pages are voided (never deleted) and the new pages added, with the page changes in `co_items`. A major change order then asks the departments it touches to verify, and holds the packet until they have. A minor one asks nobody.
- **Reject.** Clerical rejects with a reason and a signature.
A packet at `complete` still gets a change order. A filed packet gets nothing.
The `drawing-lookup` job finds each packet's drawing in M-Files through MFilesKit, downloads it, and attaches it to drafts.
### How MFilesKit finds a drawing
This is what `mfileskit/drawings.py` does. MFilesKit's own `docs/packet-station.md` is stale in two places, noted below.
- **Key.** Hyphens and spaces removed, upper-cased: `L073-76-2` becomes `L073762`.
- **Search by Name property, not folders.** The released PDF is searched in class "Drawing Production (PDF)" by the Name property starting with the key, and a hit counts only when its name is the key or the key plus `X`. There is no project-folder rule any more (the `P-521` to `H777` mapping in the MFilesKit docs is gone).
- **Released rev.** Only a document whose Controlled Status (or workflow state) is Effective counts. The rev is its Controlled Version Label, else the number after `R` in its title. The highest released rev wins.
- **Drafting rev.** Working copies are searched by Controlled Version Title. Copies in a done status (released, effective, retired, obsolete) or a done state (drawing released to shop, drawing already exists, cancel drawing) are skipped. The drafting rev is the highest label among the rest. Its workflow state maps to "still being drawn (unapproved)" or "in the drafting check cycle". (The MFilesKit docs still say "the higher of the Unapproved rev and the Check Cycle rev"; that is stale.)
- **In drafting.** A Drafting Assignment named for the key means the drawing is in drafting.
- **H sheets.** A drawing like `K978-01-6H13` has no object of its own. When nothing is found for it (no released PDF, no working copy, no assignment), the lookup tries the base drawing `K978-01-6` and reports `base`. The packet note then says "sheet of drawing K978-01-6".
- A search that hits the 500-result cap is a failed lookup, so a wrong rev is never reported.
### Queue and priorities
The queue covers live packets made by the feed, not filed, with a drawing number, in the departments in scope. It is distinct by drawing number, and the first item's priority wins:
1. The VISUAL rev changed since the last lookup of that drawing, or a pending change order waits for a drawing at a target rev.
2. Drafts with `drawing_state``needed`.
3. Failed lookups whose backoff is over (15 minutes doubled per failure, at most 24 hours).
4. Rechecks due: last checked more than `drawing_recheck_hours` ago (default 4).
### Budget
At most `drawing_budget` lookups per run (default 60). The rest wait for the next run. The worker opens one M-Files vault per process and a new `DrawingLookup` per run, so its cache never outlives a run. When the vault cannot be opened, the run is skipped and degraded with one error.
### Download
The file is downloaded only when the object id and version or the file id and version differ from the last lookup, or nothing is stored yet. The download goes to a temporary file, is checked for the PDF magic bytes and the upload size limit, hashed with sha256 while streaming, then fsynced and renamed into the assets volume. A file with the same sha256 and size reuses the existing asset. Pages are counted with pypdfium2 (pypdf as a fallback). Each fetch writes an audit row `drawing.fetched`.
### Decision table
`drawings/decide.py` decides per packet sharing the drawing. Rows are checked top to bottom.
| Case | Decision | Flag | What happens |
|---|---|---|---|
| The lookup failed | `lookup_failed` | `lookup_failed` | What is attached stays. `drawing_state` becomes `lookup_failed` only if it was `needed`. Retried with backoff. |
| No released rev, but a drafting rev or a drafting assignment | `not_released` | `not_released` | Note "Not released in M-Files yet (sheet of <base>) - drafting status: rev N <where>". |
| No released rev and nothing in drafting | `not_found` | `not_found` | Note "Drawing not in M-Files - check the drawing number in VISUAL". |
| VISUAL's rev is not a whole number | `bad_rev` | `bad_rev` | Not attached. |
| Revs differ, and no approved rev matches M-Files | `rev_mismatch` | `rev_mismatch` | Note "Rev mismatch: VISUAL calls rev V, M-Files has rev N", plus the drafting status. The M-Files copy is fetched and staged for Clerical Supervisor approval. On a draft the rev VISUAL calls is looked up too: when M-Files still holds it (Retired, say), that copy is fetched into `drawing_revisions` and kept on the packet as `visual.called`, and the log note says "rev V is Retired in M-Files". When M-Files has no document at that rev, the note says "rev V is not in M-Files". No change order: VISUAL did not change. |
| Revs differ, but Clerical kept the rev VISUAL calls while M-Files' Effective rev is still the one they chose against | `unchanged` | none | The kept rev stays attached and confirmed. No flag and no change order, also past draft. A newer Effective rev in M-Files flags the packet again. |
| Revs match (or the approved rev matches), draft | `attach` | none | Drawing pages attached, `drawing_state``attached`, drawing flags cleared, rev confirmed. Log "Drawing attached from M-Files". |
| Revs match, draft with a hand drawing at the same rev | `unchanged` | none | The hand drawing stays. |
| Revs match, past draft, same file (same sha256) | `unchanged` | none | Nothing. |
| Revs match, past draft, same rev but a new file | `open_co` | none | Drawing-origin change order: "M-Files re-released rev N of <dwg> (new file)". |
| Past draft, M-Files released a higher rev than the one attached | `open_co` | none | Drawing-origin change order: "M-Files released rev N of <dwg>; this packet has rev M". Never a silent swap. |
| A pending feed change order waits for this drawing at the rev M-Files now has | `stage_for_co` | none | The fetched file is staged on the change order and `waiting` is cleared. Log "Drawing for change order N ready for Clerical". |
A flag decision sets `drawing_state`, replaces the drawing flags with the decision's flag, stores the M-Files answer in `visual.appr`, and clears the rev confirmation. On a draft it is a `vis_update_draft` write. Past draft it is a `vis_open_co` write logged "Drawing status from M-Files", since that is the only system action allowed past draft.
"Drawing needed" blocks the warehouse step through the `drawing_attached` precondition.
### Rev mismatch staging and approval
When M-Files has a different released rev from VISUAL, the M-Files copy is fetched and its asset is kept on the `drawing_lookups` row. A Clerical Supervisor can approve it for one draft with a reason and a signature (action `rev_approve`). The pages are attached, the drawing flags are cleared, and `visual.rev_ok` records the rev, the VISUAL rev, who, when and why. The log says "Different rev approved by hand" with "rev N in place of VISUAL rev M". Without a staged copy the answer is "nothing staged".
### Hand drawing taken off on a new rev
A draft whose VISUAL drawing or rev moves loses every drawing page, a hand drawing included (step 4 of the apply). The drawing job then looks up the new rev. A hand drawing at the same rev is left alone.
### Drawing-origin change orders
A drawing change on a packet past draft opens a change order with origin `drawing`, ECN `M-FILES`, the new file already staged, and no target fingerprint. It counts as the packet's one pending feed change order. Approval voids the old drawing pages and adds the new ones, as for a feed change order.
Flags live in `packets.flags`. A draft with any flag shows in Clerical's Flagged column. Auto-release never takes a packet with a flag. The find bar filters by flag kind.
| Flag | Text shown | What Clerical does |
|---|---|---|
| `drawing_needed` | Made from VISUAL - drawing not attached yet | Usually nothing: the drawing job attaches it. Run now to hurry it. Upload by hand only if it stays stuck. |
| `rev_mismatch` | The M-Files note, for example "Rev mismatch: VISUAL calls rev 2, M-Files has rev 3" | Get the rev corrected in VISUAL. Or a Clerical Supervisor chooses, with a reason and a signature: the latest (Effective) rev in M-Files, or the rev VISUAL calls when M-Files still holds it (Retired). Both PDFs can be opened from the packet first. |
| `not_released` | The note: "Not released in M-Files yet ... drafting status: rev N <where>" | Wait for drafting to release it, or upload the drawing by hand with a reason and a signature. |
| `not_found` | Drawing not in M-Files - check the drawing number in VISUAL | Check the drawing number in VISUAL and have it fixed there. Or upload by hand. |
| `lookup_failed` | M-Files lookup failed: (the reason) | Nothing at first: the job retries with backoff. If it persists, tell IT. |
| `bad_rev` | VISUAL rev is not a number - drawing not attached | Have the rev fixed in VISUAL, or upload by hand. |
| `no_drawing` | (the draft has no drawing number) | Have the drawing number added in VISUAL, or upload by hand. |
| `wo_gone` | Work order no longer in VISUAL | Check VISUAL. If the job was deleted on purpose, remove the draft by hand (or handle the in-shop packet). The flag clears itself if the work order comes back. |
| `cancelled` | Work order cancelled in VISUAL | Decide: remove the draft by hand, or leave it if the cancel was a mistake. The feed never removes it. |
| `hand_drawing` | Drawing uploaded by hand - check it and send it on | Check the drawing and send the draft on by hand. |
| `hand_bom` | BOM uploaded by hand - check it and send it on | Check the BOM and send it on. "Use VISUAL's BOM again" puts the feed's BOM back. |
With both hand flags the text is "Drawing and BOM uploaded by hand - check it and send it on". Hand drafts keep their phase 2 text.
"Closed in VISUAL" is not a flag. It is an observed state (`visual_closed_at`), shown on the packet. Filing a Data Book packet needs it, as the `vclosed_if_databook` precondition says.
### Hand uploads
On a flagged draft made by the feed (or one already uploaded by hand), Clerical can:
- **Upload a drawing** (PDF) with a reason and a signature. It replaces the drawing pages, sets `drawing_state``hand`, clears the drawing flags and adds `hand_drawing`.
- **Upload a BOM** (XLSX or CSV) with a reason and a signature. The server parses it: the header row is found in the first 20 rows by the old build's column patterns (item, qty, part, description, length, material, drawing reference); qty and a description or part are needed; empty rows are skipped; items default to the row number. It replaces the BOM lines (source `hand`) and adds `hand_bom`. The feed then leaves the BOM alone.
- **Use VISUAL's BOM again**: rebuilds the BOM from the cache and removes `hand_bom`.
An unflagged draft refuses a hand upload ("not flagged").
A project can have several customer orders in VISUAL. One function, `feed/orders.choose`, picks the order for a project or a unit. The first rule that finds one wins:
1.**Pin.** Clerical pinned an order (a customer order ID among the project's orders) on the project or unit, with a reason. Via `pin`.
2.**Base.** The order linked to the work order base itself. Via `base`.
3.**Unit top job.** The project order whose line's top job is the unit number (open orders first). Via `unit_top_job`.
4.**Project open.** The first open order of the project (status F, H or R), newest first. Via `project_open`.
5.**Project newest.** The first order of the project, open or not. Via `project_newest`.
When nothing matches there is no order.
### Provenance
The chosen order is stored as `order_ref` on the project and the unit: order ID, `via`, status, customer name, customer PO, order, desired ship and promise dates, and when it was chosen. The fields derived from it:
| Field | From the order |
|---|---|
| customer | customer name |
| po | customer PO reference |
| order date | order date |
| ship date | promise date, else desired ship date |
For each field the feed always writes `<field>_visual`. It writes the field itself, with `<field>_src = 'visual'`, only when `<field>_src` is empty or already `visual`. A value Clerical typed (`_src = 'hand'`) is never overwritten. Each change is audited as `project.updated` with `src: visual`.
### The pin
Clerical can pin an order on a project or a unit (`order_pin`, with a reason, who and when) from the project page, which lists every cached order of the project. The feed re-derives on its next run. Clearing the pin goes back to the rules.
## Settings
The feed settings are one `settings` row, key `feed`, a JSON object. An Admin edits them on the Admin page, Feed tab (password session only). Unknown keys are ignored and missing keys take the default. Every save is validated, bumps the row version, is audited as `feed.settings_changed` with each changed key's before and after, and notifies the worker.
| `shop_days` | Monday to Friday (`[0,1,2,3,4]`) | 0 (Monday) to 6 | Shop days, site time |
| `shop_start` | `06:00` | HH:MM | Start of shop hours, site time |
| `shop_end` | `18:00` | HH:MM | End of shop hours, site time |
| `window_back_days` | 14 | 0 to 120 | Window start, days before today |
| `window_ahead_days` | 30 | 0 to 365 | Window end, days after today |
| `golive_date` | none | a date | Work orders released before it are ignored |
| `departments` | all (`[]`) | shop codes | Departments to make packets for. Empty means every department the resource map routes to. |
| `drawing_recheck_hours` | 4 | 1 to 168 | Recheck each drawing after this long |
| `drawing_budget` | 60 | 1 to 500 | Drawing lookups per run |
| `drawing_interval_minutes` | 30 | not limited | The drawing job's own cadence |
| `labor_tickets` | on | on or off | Read labor tickets, at most every 30 minutes |
| `auto_release` | on | on or off | Release ready drafts to the warehouse |
| `auto_release_max` | 25 | not limited | Most drafts auto-released per run |
| `paused` | off | on or off | Stops scheduled feed and drawing runs and auto-release. Run now still works. |
The Feed tab also shows the next scheduled run and the last ten runs. The "VISUAL resources" tab edits the resource map: each resource with its rows in the last batch, target (a shop department, Fold, Ignore or not assigned) and a confirmed tick. Every map change is audited.
## Fixture sources
For CI and local runs the worker can read files instead of VISUAL and M-Files:
-`FEED_SOURCE=fixture` reads `FIXTURE_DIR/feed.<state>.json`, a file in the exact shape of VisualKit's `examples/packet_station_feed.py` output (`meta`, `rows`, `materials`, `orders_by_base`, `orders_by_project`) plus an optional `labor` list. The state is read from `FIXTURE_DIR/state` (default `1`). It returns the firmed rows released inside the window, plus every row whose key is tracked; tracked keys with no rows are missing.
-`DRAWING_SOURCE=fixture` reads `FIXTURE_DIR/drawings.<state>.json`, a map from drawing number to its rev, file name, drafting state and ids. A drawing not in the file is not found. A download copies `FIXTURE_DIR/<file>`.
-`FIXTURE_DIR` defaults to `/fixtures`.
- The production stack refuses a fixture source at start.
`tests/fixtures/generator.py` writes the files: `python -m tests.fixtures.generator --out DIR` makes `feed.1.json`, `feed.2.json`, `drawings.1.json`, `drawings.2.json`, the one-page PDFs they name, and `state` = `1`. It is deterministic. State 1 covers the packet rules (fold, ignore, an unmapped resource, a lot `7A`, two operations on one work order, no drawing, a non-numeric rev, the BOM rules, a work order before go-live, closed and cancelled legs, several open orders). State 2 changes a rev, a quantity and a target, removes one leg and moves another, so the change order, minor, gone and retire rules run. `packetstation fixtures use N --dir DIR` switches the state. The Playwright scenario writes the state file itself. See [DEPLOY.md](DEPLOY.md) for the CI step and [tests/fixtures/README.md](../tests/fixtures/README.md).
Real job data never enters git or CI. Recordings from VISUAL DEV stay local, in `tests/fixtures/local/` and `recordings/`.
The packet read model has a `labor` field from `visual_labor`. It joins the packet's work orders on (`wo_base`, `lot`, `split`, `sub`, `seq`) and sums per work order:
-`hours` and `tickets`: the totals over the packet's work orders.
-`per_wo`: one entry per work order leg, with `wo_base`, `lot`, `split`, `sub`, `seq`, `hours`, `tickets` and `last_ticket`.
-`observed_at`: when the worker last read the tickets.
`labor` is null for a hand packet, or when no `visual_labor` row matches. The packet page shows the sum beside the station clock, as `VISUAL tickets 3.2 h`.
The station timer stays the clock of record (open question 1, default kept). The ticket hours are shown for comparison only. Nothing is computed from them, and they never change the clock. The screen reads them from PostgreSQL, never from VISUAL.
- Packets, flags, work order status, closed and cancelled state, ticket hours and the chosen order come from the packet tables and `packet_visual_state`. The ticket hours beside the station clock come from `visual_labor` (see "Ticket hours beside the station clock").
- The From VISUAL panel reads `/api/feed/overview`: pause state, next run, the heartbeat, the last ten feed and drawing runs, the ignored work orders, unmapped resources and projects without names.
- Admin reads the cache tables (`/api/feed/batches`, `/api/feed/changes`) for "what VISUAL changed".
- The project page reads the cached orders from `visual_order_links` and `visual_orders`.
Scheduled runs come from the worker's own clock, not from an open browser.
