Phase 4: Warehouse screen, pull table, shorts, MTRs and the upload pipeline

The Warehouse home with the numbers, the Open shorts table, the find bar and the five-column task board
(waiting, being pulled, on hold, pulled, released in the last 3 days) with buttons, hold reasons and drag.
Every role sees the warehouse stage on a packet as the WH pill and the line under it. The pull table on the
Materials tab: Pulled with auto-start, issued, location, heat with Enter to the next line, shorts with
receipts, N/A, MTR per line with a viewer. Send back to Clerical with a reason. Release to supervisor stays
blocked until every line is pulled or short and every pulled line has a heat.

Server: the heats_recorded precondition on WH_RELEASE, receive by custody, server-validated hold and short
reasons, the old packet history texts, GET /api/warehouse/board, cards with the warehouse stage and pull
counts. Uploads typed by content per kind (drawing, mtr, document, bom with XLSX and CSV), limits per kind,
optional ClamAV behind the clamav compose profile, a storage quota alert on the new notification_events
table (migration 0005) shown on the Admin Storage tab.

Tests: every warehouse action as every role, the release gate, the short-receive exception, the heat
pattern, upload sniffing and limits, the ClamAV path with a fake scanner, the quota alert; frontend board
and pull table tests; Playwright scenarios for the board and the pull flow. Verified on docker-1 in the CI
stack: 3557 server tests, 106 frontend tests, 17 Playwright scenarios. Docs updated; D5 closed.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

docs/DEPLOY.md

@@ -33,6 +33,7 @@ Both stacks run from the same image as separate compose projects.
 - `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
 - `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
 - `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup and housekeeping (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only (used from phase 9). Its healthcheck reads the heartbeat file.
+- `clamav` (optional, profile `clamav`, phase 4): `clamav/clamav:1.4`, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias `packetstation-clamav` (as `postgres` has its own alias). `backend` network plus egress (freshclam downloads the signatures), no `npm_default`, no published port. Volume `clamav-db` at `/var/lib/clamav`, `mem_limit: 2g`, healthcheck `clamdcheck.sh`. Started only with `--profile clamav`. See "Virus scan (ClamAV)".
 - All services restart unless stopped, log to stdout (json-file), and have healthchecks.
 
 ## Secrets
@@ -69,6 +70,8 @@ Key settings:
 - `APP_ALIAS` per stack.
 - Identity timing and limits, all optional with these defaults: `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
 - Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
+- Uploads and storage, all optional: `UPLOAD_MAX_BYTES` (default 25 MB, drawings, MTRs and documents), `UPLOAD_MAX_BYTES_BOM` (default 5 MB, hand BOMs), `ASSETS_QUOTA_WARN_PCT` (default 85, 1 to 99) and `ASSETS_QUOTA_BYTES` (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
+- Virus scan, all optional: `CLAMAV_HOST` (empty by default, which means no scan; `packetstation-clamav` for the profile service), `CLAMAV_PORT` (default 3310) and `CLAMAV_TIMEOUT_SECONDS` (default 30). The env examples carry `#CLAMAV_HOST=packetstation-clamav` commented out. See "Virus scan (ClamAV)".
 - `SITE_ID` (default `PESCO`) and `SITE_TZ` (default `America/Denver`): the VISUAL site in every call, and the time zone for shop hours and the schedule.
 - The session cookie is marked `Secure` when the request arrived over https (as reported by NPM through `X-Forwarded-Proto`). Behind the proxy that is always the case.
 
@@ -89,6 +92,35 @@ Production (planned, phase 10): the same command with `-p packetstation` and the
 
 Images come from `gt.pescoinc.biz` by tag. CI tags each tested image `:sha-<sha>`; a release re-tags it `vX.Y.Z`. Never `latest`. A deploy is a new image with a short downtime window.
 
+## Virus scan (ClamAV, phase 4)
+
+ClamAV is optional and off by default. It closes requirement D5. Turn it on in the test stack first, check it, then the production stack.
+
+1. Set `CLAMAV_HOST=packetstation-clamav` in the stack's env file (`/etc/packetstation/test/stack.env`). Leave `CLAMAV_PORT` and `CLAMAV_TIMEOUT_SECONDS` at their defaults.
+2. Start the stack with the profile:
+
+```bash
+docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
+  -f deploy/compose.yaml -f deploy/compose.proxy.yaml --profile clamav up -d
+```
+
+3. Wait for `clamav` to turn healthy. On the first start the volume `packetstation-test_clamav-db` is empty, so freshclam downloads the full signature set (a few hundred MB) before clamd answers. That takes several minutes. Later starts reuse the volume and only fetch updates. freshclam needs egress to the ClamAV mirrors, which is why the service is on the `egress` network.
+4. Check it: Admin, Storage tab, the ClamAV line reads "packetstation-clamav: reachable". `packetstation doctor` also has a `clamav` check (a PING to clamd; SKIP when `CLAMAV_HOST` is empty). Upload a small PDF to a draft and confirm it is stored.
+
+Every later `up` for that stack must carry `--profile clamav`. Without it the service is not started.
+
+Fail closed. While `CLAMAV_HOST` is set, an upload the scanner cannot check is refused with "The virus scanner is not reachable. Try again in a minute." (503 `scan_unavailable`). That covers clamd still loading signatures, stopped, or out of memory. The worker's drawing fetches are refused the same way and tried again later. A file with a signature hit is refused (422 `infected`), never stored, and audited as `asset.rejected` with the signature name.
+
+To turn it off: remove `CLAMAV_HOST` from the env file, recreate `app` and `worker` (`up -d` without the profile), then stop the service with `docker compose ... --profile clamav stop clamav`. The volume can stay for the next time. The CI overlay never runs ClamAV.
+
+clamd holds the signatures in memory, about 1.2 GB with the full set and more while it reloads them, so the service has a 2 GB limit. Check docker-1 has the headroom for both stacks before turning it on in production.
+
+## Storage quota (phase 4)
+
+Every stored file and the housekeeping job check the assets volume. With `ASSETS_QUOTA_BYTES=0` (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At `ASSETS_QUOTA_WARN_PCT` percent (85 by default) the app opens one `notification_events` alert of kind `assets_quota` and audits `storage.quota_warning`. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.
+
+The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. Phase 9 sends the alert by email. Until then, check the Storage tab.
+
 ## Python packages and the Gitea index
 
 The Gitea PyPI index `https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/` serves only `visualkit`, `mfileskit` and their dependencies.
@@ -148,6 +180,7 @@ It checks:
 - Assets volume and free space. Backup folder (planned, phase 9).
 - LDAP TLS bind on both controllers, reading `memberOf`, `employeeID` and `userAccountControl` as the service account, the 50-group count and one nested-group query.
 - SMTP STARTTLS with certificate verification.
+- ClamAV (phase 4): clamd answers PING at `CLAMAV_HOST`. SKIP when no host is set.
 - tzdata.
 
 Use `--skip visual,mfiles,ldap,smtp` to run only the local checks. The network checks need the secrets and the CA bundle in place.

docs/PLAN.md

@@ -237,6 +237,14 @@ Smaller questions from the design, with the default the build takes unless told
 17. SET_DATABOOK_AT is writable by anyone in the old build, even when filed: default restrict to Clerical and Q.C.
 18. CO verifications cannot be withdrawn; ITP line signatures can be undone by the same inspector or an Admin before the block is signed.
 
+From phase 4 (2026-10-16), for Seth to veto:
+
+19. The release gate has two server rules: every line not N/A is pulled or on an open short, and every pulled line that issued pieces has a heat unless it is N/A or a sub-assembly. The old build enforced both on the screen only. Default: both on the server.
+20. Hold and short reasons: the old build accepted any text. Default: closed sets on the server; "Other" on a hold needs a note.
+21. Receiving a short on a packet the warehouse no longer holds: default it stays a warehouse right (WH_RECEIVE_SHORT, any status but filed), and the holder of the packet (department lead or supervisor, Q.C., Clerical) can receive too, through edit_bom.
+22. Upload kinds and limits: default an MTR is PDF, PNG or JPG up to 25 MB, a hand BOM is XLSX or CSV up to 5 MB. ClamAV is optional and off unless CLAMAV_HOST is set; when set, an unreachable scanner refuses uploads (fail closed).
+23. The quota alert: default a `notification_events` row (phase 9 builds the delivery on this table) at 85 percent of the assets volume.
+
 ## Verification
 
 - Every phase: `pytest` server suite, `npm test` and `npm run build` for the frontend, `docker compose -f deploy/compose.ci.yml up` with the Samba DC and the Playwright scenarios for that phase, `packetstation doctor` inside the image.

docs/POLICY.md

@@ -79,7 +79,7 @@ Statuses in workflow order: draft, warehouse, ready, assigned, in_progress, revi
 | send_draft_to_qc | CL | draft | qc | no | | heat_map_answered, drawing_attached | | |
 | wh_send_back | WH | warehouse | draft | no | | reason_given | | yes |
 | return_to_clerical | DEPT_LEAD | ready, only when wh_skipped | draft | no | | reason_given | | yes |
-| wh_release | WH | warehouse | ready | no | yes | bom_all_pulled_or_short | | |
+| wh_release | WH | warehouse | ready | no | yes | bom_all_pulled_or_short, heats_recorded | | |
 | send_back_to_warehouse | DEPT_LEAD | ready | warehouse | no | | reason_given | | yes |
 | assign | DEPT_LEAD | ready | assigned | no | | welder_set | | |
 | unassign | DEPT_LEAD | assigned | ready | no | | | | |
@@ -101,6 +101,8 @@ Statuses in workflow order: draft, warehouse, ready, assigned, in_progress, revi
 | edit_wos | CL | draft | | | | | | |
 | edit_bom | HOLDER | draft, warehouse, SHOP | | | | | | |
 | wh_receive_short | WH | any | | | | short_open | | |
+
+A holder of the packet who is not in the warehouse receives a short through edit_bom (see "Receive by custody").
 | edit_heats | HOLDER | warehouse, SHOP | | | | | | |
 | edit_heat_table | HOLDER | warehouse, SHOP | | | | | | |
 | edit_welds | HOLDER | SHOP | | | | | | |
@@ -141,6 +143,7 @@ Each precondition is a named predicate over the packet state and the request inp
 | heat_map_answered | heat_map is yes or no | Say whether this packet needs a heat map. |
 | drawing_attached | a drawing is attached | Attach the drawing first. |
 | bom_all_pulled_or_short | every line not marked n/a is pulled or has an open short | Pull every material line or mark it short first. |
+| heats_recorded | no line needs a heat: every line that is not N/A, not a sub-assembly, is pulled and issued pieces has a heat | Scan or type the heat number of every pulled line first. Use N/A for a part with no traceability. |
 | short_open | at least one BOM line has an open short (the old whCanReceive) | No part is short on this packet. |
 | welder_set (input) | the request gives station_id and welder_id | Pick a station and a welder. |
 | checklist_complete | every key of the department checklist is done | Finish the checklist first. |
@@ -199,6 +202,101 @@ Every packet read returns:
 
 `can(principal, state, assume_it_reason=True) -> (list[Action], dict[Action, list[str]])`. For an admin the IT reason is assumed, so the list shows what the admin may do once they give one. `evaluate(principal, state, action)` returns the read-mode decision with `admin_bypass` set, so the screen knows to ask for the reason. System-only actions are never listed. CO_VERIFY needs a department in the request, so it is not listed; the change order list says which departments the person can verify.
 
+## Warehouse (phase 4)
+
+### Stages
+
+The warehouse stage is `packet_wh`, separate from the packet status. SET_WH_STAGE (WH, status warehouse) moves it, compare-and-set on `expect_stage` (409 `stale_step` when it moved on).
+
+| From | To |
+|---|---|
+| incoming | pulling, hold |
+| pulling | pulled, hold |
+| hold | pulling |
+| pulled | pulling |
+
+A move to the same stage, or one not in the table, is 409 `wh_transition`. A packet with no `packet_wh` row is at incoming. The first Pulled tick on a line while the stage is incoming or on hold starts pulling by itself (`auto` true).
+
+### Hold reasons
+
+A closed set, checked on the server:
+
+- Waiting on material (not in stock)
+- Waiting on purchasing / PO
+- Question for engineering or clerical
+- Drawing / BOM problem
+- Other
+
+Any other text is 400 `invalid_reason` "Pick one of the hold reasons." "Other" needs a note, else 400 `reason_required` "Say what it's waiting on." The note is cleaned and cut to 200 characters.
+
+### Short reasons
+
+A closed set, checked on the server:
+
+- Not in stock
+- On order
+- Back-ordered
+- Damaged / rejected
+- Wrong part received
+- Needs cutting / prep
+- Other
+
+Any other text is 400 `invalid_reason` "Pick one of the short reasons." A short has a quantity from 1 to the line quantity, a reason, an optional PO or requisition, an optional expected date, an optional note, and the optional heat of the pieces that were pulled (never on an N/A line). Opening, editing and clearing a short is edit_bom, so the holder decides. In practice that is the warehouse.
+
+### The release gate
+
+WH_RELEASE checks two preconditions, in this order:
+
+1. `bom_all_pulled_or_short`: every line not N/A is pulled or has an open short.
+2. `heats_recorded`: no line needs a heat. A line needs a heat when it is not N/A, not a sub-assembly (`sub` is `Y`; its heat is on its own packet), is pulled, has issued more than 0 and has no heat.
+
+Both are state preconditions, so `blocked.wh_release` lists the failing message and the screen shows Release disabled with it. Heat numbers are checked against the old pattern: letters, numbers, space . / # _ - only, up to 40 characters, starting with a letter or a number. A bad heat is 422 `invalid_heat`.
+
+### Receive by custody
+
+Receiving short parts (`POST .../bom/{item}/short/receipts`) picks its action from the person:
+
+- A member of the warehouse (WH in `support`) receives through WH_RECEIVE_SHORT, in any status but filed, while the line has an open short. The warehouse does not need to hold the packet.
+- Anyone else receives through EDIT_BOM, so only the holder may: the lead or supervisor of the department that has the packet, Q.C. at Q.C., Clerical in draft.
+- Everyone else is refused by the policy (403 `forbidden`).
+
+The warehouse still cannot change any other field of a packet it no longer holds. EDIT_BOM stays the holder's. A receipt needs a heat unless the line is N/A, and its quantity is 1 to what is still open.
+
+### MTR files
+
+The MTR on a line must be an asset whose type is PDF, PNG or JPEG. Anything else, or an asset that is not there, is 422 `invalid_asset`. Clearing the MTR is allowed.
+
+### Packet log texts
+
+The `action` column of `packet_log` for each warehouse write. `{item}` is the BOM item number.
+
+| Write | action | note |
+|---|---|---|
+| Heat typed or scanned | `Heat recorded · item {item}` | the heat |
+| Heat cleared | `Heat cleared · item {item}` | |
+| Pulled ticked | `Pulled · item {item}` | |
+| Pulled unticked | `Pulled undone · item {item}` | |
+| N/A ticked | `Marked N/A · item {item}` | |
+| N/A unticked | `Traceable again · item {item}` | |
+| Issued only | `Issued qty changed · item {item}` | `{issued} of {qty}` |
+| Location only | `Location set · item {item}` | the location |
+| Several fields | `Line changed · item {item}` | |
+| MTR attached | `MTR attached · item {item}` | |
+| MTR removed | `MTR removed · item {item}` | |
+| Short opened | `Short recorded · item {item}` | `{qty} of {need} short · {reason} · {ref} · ETA {eta}`, parts present only |
+| Short edited | `Short updated · item {item}` | the same as recorded |
+| Not short after all | `Short cleared · item {item}` | |
+| Short received | `Short received · item {item}` | `{qty} pc · heat {heat} · {still} still short` or `... · short closed`, then ` · packet {status}` (lower case) and ` in {station}` when at a station |
+| Stage incoming to pulling | `Warehouse: Started pulling` | |
+| Stage to hold | `Warehouse: Put on hold` | `{reason} · {note}` |
+| Stage pulling to pulled | `Warehouse: Marked pulled, ready for production` | |
+| Stage hold to pulling | `Warehouse: Pulling resumed` | |
+| Stage pulled to pulling | `Warehouse: Back to pulling` | |
+| wh_release | `Released to supervisor`, or `Released to supervisor with {n} short` when shorts are open | `{heats} heats recorded`, plus ` · SHORT: item {item} ×{qty_open} ({reason}), ...` when shorts are open |
+| wh_send_back | `Sent back to Clerical` | the reason |
+
+When a Pulled tick starts pulling by itself, there is no separate stage row in the log: the line's own row gets ` · started pulling` or ` · pulling resumed` on its note. Every write above also writes its audit row through the write pipeline, with `detail.item`, `wh_from` and `wh_to` as before, and `detail.qty` on short writes.
+
 ## Worker writes (phase 3)
 
 The worker has no session. It writes through the same policy as a person, as the system principal `Principal(user_id=None, system="visual-feed")`. Each write locks the packet row (`FOR UPDATE SKIP LOCKED`; a row locked by someone else is left for the next run), builds the packet state, runs `policy.check()`, makes its change, bumps `version`, and writes a `packet_log` row (by nobody, by name "VISUAL feed", role `system`), an audit row (`packet.<action>`, role `system`) and a change event. See [FEED.md](FEED.md) for when each write happens.
@@ -256,6 +354,8 @@ The old policy was looser than the old screens in places. The rebuild follows th
 | Anyone could set the data book date, even on a filed packet. | SET_DATABOOK_AT is Clerical or Q.C. only (open question 17). It stays allowed on a filed packet. | The data book date is a Clerical and Q.C. record. | Yes |
 | Change-order verifications were stored in the packet document, with no rule against changing them later. | Verifications are insert-only and cannot be withdrawn (open question 18). | A signed verification is a record. | Yes |
 | The old policy did not look at the BOM when the warehouse released a packet. | WH_RELEASE needs every BOM line not marked n/a to be pulled or to have an open short. | A packet should reach the shop with its material, or with the shorts on record. | Yes |
+| The old screen asked for a heat number on every pulled line before release, but the policy did not check it. | WH_RELEASE also needs a heat on every pulled line that issued pieces, unless the line is N/A or a sub-assembly (`heats_recorded`), checked on the server. | Traceability is the point of the pull. The screen alone could be skipped. | Yes |
+| Hold reasons and short reasons were any text the screen sent. | Both are closed sets checked on the server. "Other" on a hold needs a note. | The board and the shorts table group and report on the reason, so it must be one of the list. | Yes |
 | A hand draft could leave Clerical with no drawing (the packet PDF upload stood in for it). | A hand draft needs a drawing attached before any send from Clerical. | There is no packet PDF upload in the rebuild. | Yes |
 | Attaching drawing pages by hand needed no signature. | Attaching a drawing by hand needs a signature ("Attach a drawing by hand"). | It replaces what M-Files would supply, so a person stands behind it. | Yes |
 | Admin could write anything on a filed packet with an IT reason. | Admin cannot write a filed packet except through ADMIN_REOPEN (with a reason and a signature), and SET_DATABOOK_AT. | A filed packet is sealed. A change breaks the seal on purpose, on record. | No |

docs/SCHEMA.md

@@ -1,6 +1,6 @@
 # Packet Station schema
 
-Status: phases 1 to 3 built (identity, organisation, projects, packets, change orders, documents, jobs, VISUAL and drawing caches). Phase 9 adds the notification tables. The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
+Status: phases 1 to 4 built (identity, organisation, projects, packets, change orders, documents, jobs, VISUAL and drawing caches, the warehouse columns in use, the `notification_events` placeholder). Phase 9 adds the notification rules and delivery. The DDL is in `server/packetstation/migrations/`; this file explains it. See [PLAN.md](PLAN.md), "Schema".
 
 ## Conventions
 
@@ -82,6 +82,19 @@ The trigger `ps_packet_touch` sets `updated_at` on every UPDATE. It does not tou
 - `bom_lines`: `item`, `qty`, `part_id` (text), `description`, `dwg_ref`, `cl_item`, `length`, `material`, `heat`, `na`, `issued`, `loc`, `pulled`, `pulled_at`, `pulled_by`, `mtr_asset_id`, `src`, `vkey`, `whs`, `comm`, `cons`, `sub`, `grp`, `sort`, `row_version`. Unique on (`packet_id`, `item`).
 - `bom_shorts`: `bom_line_id` (cascades with the line), `qty`, `qty_open`, `reason`, `ref`, `eta`, `note`, `opened_at`, `opened_by`, `closed_at`, `row_version`. At most one open short per line (unique on `bom_line_id` where `closed_at` is null).
 - `short_receipts`: append-only. `short_id`, `packet_id`, `qty`, `heat`, `note`, `at`, `by`.
+
+What the warehouse columns mean in practice (phase 4):
+
+- `bom_lines.pulled`: the line is off the shelf. The first tick sets `pulled_at` and `pulled_by`, and sets `issued` to `qty` when it was empty. Unticking clears all three of `pulled`, `pulled_at` and `pulled_by`. The first tick while the warehouse stage is incoming or on hold also starts pulling (a `packet_wh` row with `auto` true).
+- `bom_lines.issued`: how many pieces went out. Less than `qty` means the rest should be on a short.
+- `bom_lines.loc`: where the parts were taken from, as the warehouse types it.
+- `bom_lines.heat`: the heat number of the pieces issued, checked against the heat pattern. A receipt adds its heat to the line, so a line with parts from two heats reads `H1 / H2` (heats joined with " / ", no repeats).
+- `bom_lines.na`: the part has no traceability. An N/A line needs no heat, takes no heat, and does not count toward the release gate.
+- `bom_lines.mtr_asset_id`: the mill test report for the line, an `assets` row that must be a PDF, PNG or JPEG. Null clears it.
+- A line "needs a heat" when it is not N/A, not a sub-assembly (`sub` is `Y`), is pulled, has `issued` above 0 and has no heat. The release gate counts these lines (see [POLICY.md](POLICY.md)).
+- `bom_shorts`: one open short per line at most. Opening a short sets the line's `issued` to `qty` minus the short quantity and ticks it pulled. `qty` is what was short in total, `qty_open` what is still missing. `reason` is one of the closed short reasons, `ref` the PO or requisition, `eta` the expected date. Editing an open short keeps what was already received in `qty`. "Not short after all" closes it (`closed_at`) and sets `issued` to the line's `qty`. While a short is open, the line's heat, issued and location change only through the short.
+- `short_receipts`: one row per delivery, never changed. Each receipt lowers `qty_open`, adds its pieces to the line's `issued` and its heat to the line's `heat`. The receipt that brings `qty_open` to 0 closes the short.
+- `packet_wh`: `reason` is one of the closed hold reasons when the stage is `hold`, and `note` is up to 200 characters.
 - `packet_stages`: one row per visit to a shop department. `dept_code`, `station_id`, `welder_id`, `welder_stamp`, `checklist` jsonb (key to `{done, at, by, stamp}`, written per key with `jsonb_set`), `filler`, `filler_lot`, `preheat`, `notes`, the clock (`clock_elapsed_ms`, `clock_running_since`, `clock_started_at`), `qc_snapshot`, `opened_at`, `closed_at`, `row_version`. At most one open stage per packet.
 - `welds`: `weld_no` (from `next_weld_no`), `page`, `fx` and `fy` (fractions of the page), `wo_base`, `piece`, `items` int[], `type`, `size`, `process`, `stamp`, `welder_id`, `dept_code`, `stage_id`, `carried_from`, `at`, `by`, `row_version`, soft delete. Unique on (`packet_id`, `weld_no`) among live rows.
 - `heat_tags`: `tag_no` (from `next_heat_no`), `page`, `item`, `fx`, `fy`, `carried_from`, `at`, `by`, `row_version`, soft delete. Unique on (`packet_id`, `tag_no`) among live rows.
@@ -107,6 +120,17 @@ Every who column references `users (id)`. Every `signature_id` references `signa
 ## Documents (phases 2, 4 and 6)
 
 - `assets`: `id` uuid, `sha256` (indexed, for dedupe), `size`, `mime`, `original_name`, `storage_key` (the path under ASSETS_DIR, unique), `source` (`upload` by default), `source_ref` jsonb, `pages`, `uploaded_by`, `uploaded_at`, `packet_id` (the packet it was uploaded for, without a foreign key; `packet_pages` and `packet_docs` are the real links).
+
+Asset kinds (phase 4). An upload names its kind in the `purpose` field, and `source_ref` keeps `{"purpose": kind}`. No purpose means `document`. The type is read from the content, never from the name or the browser.
+
+| Kind | Types | Limit (setting) |
+|---|---|---|
+| drawing | PDF | 25 MB (`upload_max_bytes`) |
+| mtr | PDF, PNG, JPEG | 25 MB (`upload_max_bytes`) |
+| document | PDF, PNG, JPEG | 25 MB (`upload_max_bytes`) |
+| bom | XLSX, CSV | 5 MB (`upload_max_bytes_bom`) |
+
+`mime` is one of `application/pdf`, `image/png`, `image/jpeg`, `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` and `text/csv`. An XLSX is a zip that holds `[Content_Types].xml` and `xl/workbook.xml`; a plain zip is refused. A CSV is UTF-8 text with no NUL and no control character other than tab, CR and LF. An upload the virus scanner refuses is never stored, so it has no row.
 - `packet_docs`: `asset_id`, `kind`, `title`, `added_at`, `added_by`, soft delete.
 - `qc_docs`: `project_id`, `unit_id`, `type`, `rev`, `asset_id`, `auto`, `superseded_by` (the newer row), `added_at`, `added_by`.
 - `itps`: one per unit (`unit_id` unique). `template` jsonb, frozen when the ITP is made. `status` (itp_status), `signoff_*` (with the signature and the signed PDF asset), `reopen_*`, `version`.
@@ -134,9 +158,13 @@ Phase 3, migration 0004:
 - `job_schedule`: one row per job (`job` PK). `next_run_at`, `interval_seconds`, `paused`, `updated_at`. Written by the worker after each run for the Admin tab. Seeded for `visual-feed`, `drawing-lookup` and `housekeeping`.
 - `worker_heartbeat`: a single row (`id` = 1 by CHECK). `at`, `build`, `host`, `pid`, `running` text[] (the jobs running now). Upserted every 30 seconds.
 
-## Notifications (phase 9)
+## Notifications (phases 4 and 9)
 
-Rules, recipients, exclusions, outbox, sent log, state.
+Phase 4, migration 0005:
+
+- `notification_events`: one row per alert. `id` bigserial, `kind` (text, for example `assets_quota`), `severity` (`info`, `warning` or `critical` by CHECK, default `warning`), `subject` (what the alert is about, for example `assets`), `detail` jsonb (for the quota alert, the volume usage: `total`, `used`, `free`, `pct`, `warn`), `created_at`, `resolved_at` (set when the condition clears), `acknowledged_at` and `acknowledged_by` (an Admin saw it). The unique index `notification_events_open` on (`kind`, `coalesce(subject, '')`) where `resolved_at` is null keeps one open alert per kind and subject, so a check that runs again does not stack rows. The table is not append-only: the app sets `resolved_at` and the acknowledgement. 0005 also grants `ps_app` USAGE and SELECT on every sequence in the schema.
+
+Phase 9 builds the delivery on this table: rules, recipients, exclusions, outbox, sent log, state.
 
 ## VISUAL and drawing caches (phase 3)
 

docs/SECURITY.md

@@ -2,7 +2,7 @@
 
 This document is rewritten for the server build. The old browser build's security document is in the old repo (`C:\Users\spencerm\Coding\PacketStation\docs\SECURITY.md`). The design comes from [BRIEF.md](BRIEF.md) and [PLAN.md](PLAN.md).
 
-Status: phase 3. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Every other control names the phase that builds it.
+Status: phase 4. The identity controls (sessions, PIN, rate limits, audit, Origin check, derived group map) and the policy controls (server policy on every write, workflow steps as compare-and-set, server-side validation and limits, IT edit reasons, the filed lock, the upload pipeline for drawings and documents) are built and tested. Phase 3 adds the worker controls (read-only VISUAL and M-Files reads, stack guard, advisory locks, redacted errors, audited system writes). No requirement ID closes in phase 3. Phase 4 completes the upload pipeline (a kind per upload with its own types and limit, XLSX and CSV by content, the optional ClamAV scan, the assets volume quota alert) and adds the warehouse rules (the heat gate on release, closed hold and short reasons, receive by custody). Phase 4 closes D5. Every other control names the phase that builds it.
 
 Contents:
 
@@ -29,7 +29,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 | 11 | PIN sign-off (electronic signature) | Server-verified signature with the PIN or the domain password, stored with who, when, what, factor and a content hash. | 1 |
 | 12 | IT edit reasons | Kept. Admin writes need a reason and are marked as IT edits in the packet history and the audit. | 2 |
 | 13 | Screen lock | Server-enforced sessions. The browser reports idle and background state; the server records it and refuses an idle or locked session. Unlock is a step-up check on the server. | 1 |
-| 14 | Input and upload validation | On the server: field limits on every write, uploads typed by content, size limits, sanitised names. | 2, 4 |
+| 14 | Input and upload validation | On the server: field limits on every write, closed reason sets, uploads typed by content per kind (drawing PDF; MTR and document PDF, PNG or JPEG; BOM XLSX or CSV), a size limit per kind (25 MB, BOM 5 MB), sanitised names, an optional ClamAV scan before anything is stored, and a quota alert on the assets volume. Built. | 2, 4 |
 | 15 | Second-admin approval | Superseded. Roles come only from AD groups and the group-to-role map is derived, so there is nothing to grant in the app. | 1 |
 | 16 | Security alerts and automatic integrity check | Worker jobs and the Notifications tab (email through app-03). | 9 |
 | 17 | Controlled-copy stamp on data books | Applied to server-built PDFs and signed with the seal key. | 7 |
@@ -74,14 +74,31 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - One policy in Python on the server (`server/packetstation/policy/`), pure functions with no I/O. Every write goes: session, policy, transition, validation, version check, packet log row, audit row, NOTIFY, in one transaction with the packet row locked. Nothing in the browser decides.
 - Every packet read returns `can: [...]`, `blocked: {...}` and `holder`, so the browser enables controls from the server's answer. The server re-checks on the write.
 - A workflow step taken twice is refused with 409 and the current state (compare-and-set on the expected status and department). Child rows carry a row version checked with If-Match. Weld, heat tag, page and change order numbers are assigned by the server under the lock.
-- Preconditions (heat map answered, drawing attached, BOM pulled or short, checklist complete, welds recorded, Q.C. items complete, closed in VISUAL before filing) run on the server and are never bypassed, Admin included.
+- Preconditions (heat map answered, drawing attached, BOM pulled or short, heats recorded on pulled lines, checklist complete, welds recorded, Q.C. items complete, closed in VISUAL before filing) run on the server and are never bypassed, Admin included.
 - Admin edits outside custody need an IT reason of at least 8 characters, recorded in the packet log and the audit row (`it_reason`). An Admin cannot write a filed packet except through the reopen action, which needs a reason and a signature and breaks the seal.
 - Signatures (PIN or domain password, verified in the same transaction) on submit, send to Q.C., Q.C. accept and reject, route, file, reopen, change order issue, approve, reject and verify, the revision approval and hand drawing uploads.
 - Input: every string is cleaned (control and invisible characters removed, NFC, trimmed) and cut to its field limit; heat numbers, stamps, project numbers and lots are checked against the old patterns; JSON bodies over 200 KB are refused.
-- Uploads: typed by content (PDF, PNG, JPEG), 25 MB limit read in chunks, sha256, stored by id under the assets volume, served only with a session and with `Content-Disposition` and `X-Content-Type-Options: nosniff`.
+- Uploads: typed by content (PDF, PNG, JPEG), 25 MB limit read in chunks, sha256, stored by id under the assets volume, served only with a session and with `Content-Disposition` and `X-Content-Type-Options: nosniff`. Phase 4 completes it (see "Uploads and storage").
 - The filed lock: a filed packet accepts only the data book date and the Admin reopen. Filing writes an HMAC-SHA256 seal with the server seal key; a reopen keeps the old seal with who broke it and why.
 - Details in [POLICY.md](POLICY.md).
 
+### Uploads and storage (phase 4)
+
+- Every upload names a kind (`purpose`): drawing, mtr, document or bom. No purpose means document. An unknown kind is 422.
+- The type is read from the content, never from the file name or the browser's type. A drawing is a PDF. An MTR or a document is a PDF, PNG or JPEG. A BOM is an XLSX or a CSV. A wrong type for the kind is 415 `unsupported_type` with a message that names the allowed types.
+- An XLSX must be a zip that lists `[Content_Types].xml` and `xl/workbook.xml`; a plain zip is refused. A CSV has no magic, so it is accepted only as a BOM, and only when its first 64 KB decode as UTF-8 with no NUL and no control character other than tab, CR and LF.
+- Limits per kind: 25 MB (`upload_max_bytes`) for drawing, mtr and document, 5 MB (`upload_max_bytes_bom`) for a BOM. The Content-Length is checked first, then the bytes as they are read. Over the limit is 413 `too_large`.
+- XLSX and CSV are always served as an attachment, never inline. Every blob keeps `Content-Disposition` and `nosniff`.
+- Optional ClamAV (requirement D5). When `CLAMAV_HOST` is set, every upload and every file the worker stores (drawings from M-Files) is streamed to clamd with `zINSTREAM` after it is read in full and before it is deduped or placed. A hit is 422 `infected`, nothing is stored, and an `asset.rejected` audit row keeps the signature, size and type. A scanner that cannot be reached refuses the upload with 503 `scan_unavailable` (fail closed). With `CLAMAV_HOST` empty there is no scan.
+- Quota alert. After every stored file and in the housekeeping job, the assets volume usage is checked against `ASSETS_QUOTA_WARN_PCT` (85 by default). Over it, one open `notification_events` row of kind `assets_quota` is written and audited as `storage.quota_warning`. Back under, the row is resolved. A failed check never fails an upload. Admin sees usage, the alert and the scanner on the Storage tab and can acknowledge an alert (`alert.acknowledged`). Phase 9 sends the email.
+
+### Warehouse rules (phase 4)
+
+- Release to the supervisor is refused on the server until every line is pulled or short and every pulled line that issued pieces has a heat (N/A and sub-assembly lines excepted). Heat numbers are checked against the old pattern.
+- Hold reasons and short reasons are closed sets on the server.
+- Short receipts go to the warehouse in any status but filed, or to the holder of the packet through the BOM edit rule. Nobody else.
+- An MTR on a line must be a PDF, PNG or JPEG asset.
+
 ### Worker (phase 3)
 
 - The worker publishes no port and joins no proxy network. It reaches out to VISUAL and M-Files only; nothing reaches in. Admin and Clerical reach it only through rows in PostgreSQL (`job_requests`, the `feed` settings row) and NOTIFY.
@@ -99,6 +116,7 @@ The M: drive build had 20 layers of protection, most of them enforced in the bro
 - `packetstation doctor` checks that an audit UPDATE fails.
 - Every identity event is a row: sign-in, failed sign-in with a reason, sign-out, lock, unlock, session ended, PIN set, disabled or cleared, device registered or revoked, signature, AD refresh. Each carries the user, session, roles, client address and a small detail object, never a secret.
 - `signin_attempts` and `signatures` are append-only in the same way.
+- Phase 4 audit actions: `asset.rejected` (an upload the virus scan refused, with `reason`, `signature`, `size` and `mime`), `storage.quota_warning` (the assets volume went over the warning threshold, with the usage) and `alert.acknowledged` (an Admin acknowledged an open alert). Every warehouse write is a `packet.<action>` row with `detail.item`, the warehouse stage before and after, and `detail.qty` on short writes.
 
 ### CI identity fixtures (phase 1)
 
@@ -142,7 +160,7 @@ The IDs come from Part 3 of the old SECURITY.md. The status follows the approved
 | A4 | Closed, phase 0 |
 | D1, D2, D4 | Deferred (envelope encryption, Key Vault) |
 | D3 | IT's (host disk encryption on docker-1) |
-| D5 | Closed, phase 4 (optional ClamAV) |
+| D5 | Closed, phase 4 (optional ClamAV: a `clamav` compose profile; when `CLAMAV_HOST` is set every upload and every stored drawing is scanned before it is kept, a hit is refused and audited, and an unreachable scanner refuses uploads; off when the host is not set) |
 | D6, D7 | Deferred (retention period and customer copy need a decision from Seth and Q.C.) |
 | M1 | Closed, phase 2 (append-only audit and packet log written only by the server; every packet write carries both) |
 | M2 | Closed, phase 7 |