Archive copy must succeed before a packet counts as filed (Gitea issue 57)

FILE checks the archive folder and copies the packet inside the filing
transaction. A failed check or copy rolls the filing back, so the packet
stays in its previous stage; the person filing gets 503 archive_failed,
and in its own transaction the server writes an archive.filing_blocked
audit row and raises the critical archive_failed condition to Admin for
that packet (who tried, folder, error). The next successful filing clears
it. The archive-copy job now only catches up filings made while the
archive was off. PLAN question 53, CHANGELOG, FEED and SECURITY updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/FEED.md

@@ -144,9 +144,9 @@ For each rule whose digest time (site time) has passed today and that has outbox
 
 ## archive-copy (every 15 minutes, phase 10)
 
-Gitea issue 7. Not pausable. Off when `ARCHIVE_ROOT` is unset or blank: the run does nothing (counts `enabled: false`) and FILE asks for nothing. With the root set but no folder saved on Admin, File locations, the run idles (`skipped: no archive folder saved`).
+Gitea issue 7. Not pausable. Off when `ARCHIVE_ROOT` is unset or blank: the run does nothing (counts `enabled: false`). With the root set but no folder saved on Admin, File locations, the run idles (`skipped: no archive folder saved`).
 
-FILE inserts a `job_requests` row for `archive-copy` and NOTIFYs `ps_job_request` inside its own transaction when the archive is on and a folder is saved, so the copy starts right after the filing commits. The request is made in a savepoint: if it fails, the filing still commits and the next scheduled run copies the packet. A failed copy never blocks filing and never touches the packet. Saving a folder and "Retry now" also request a run.
+FILE makes its own copy inside the filing transaction and refuses to file when it fails (Gitea issue 57, open question 53), so this job does not copy new filings. It catches up the filings made while the archive was off. Those packets are already filed, so a failed copy here never touches the packet. Saving a folder and "Retry now" request a run.
 
 Each run:
 

docs/PLAN.md

@@ -121,7 +121,7 @@ Each phase ends with passing tests, docs updated (README, CHANGELOG, SECURITY.md
 - Rewrite the 38 scenarios against the new UI with synthetic fixtures; run in Gitea Actions against compose.ci.yml. Policy matrix, merge rules and packet-rule tests already run from phase 2 and 3.
 - CSP final pass, dependency scan, build-version banner and reload, DEPLOY.md with the NPM proxy host settings, SECURITY.md rewritten with the ID table, GO-LIVE.md (date, boundary, groups, tablets, URL, M: retirement as read-only archive), PAPER-FALLBACK.md, incident notes.
 - Production and test each on their own Linux VM, natively, with no Docker (Spencer, 2026-10-27, Gitea issue 37; see "Host layout"). The test VM is built first from the same steps, so the production install, backups, restore and doctor are rehearsed before go-live. The app and the worker run as systemd services from a venv built from the same `server/requirements.txt` and `server/requirements-pesco.txt` pins as the image, with PostgreSQL 16, ODBC Driver 18 and the PESCO CA installed on the VM. Settings from an env file, secrets as root-owned files read through the same `*_PASSWORD_FILE` settings, VISUAL_SERVER and VISUAL_DATABASE set explicitly; first start is Spencer's approval. NPM on docker-1 proxies `packetstation.pescoinc.biz` and `packetstation-test.pescoinc.biz` to their VMs over the LAN. Backups, the doctor and the restore are rewritten for the VMs (no `docker exec`). DEPLOY.md gets a VM section; the compose files stay for CI. The docker-1 test stack keeps running until the test VM is ready, then it is stopped.
-- Archive location (Gitea issue 7, Spencer 2026-10-27). Spencer creates an NFS share on the QNAP and mounts it on docker-1 (root, steps in DEPLOY.md), with write access for uid 10001. Compose bind-mounts it into the app and the worker at a fixed root, `ARCHIVE_ROOT` (unset turns the feature off). The Admin page gets a File locations section with one setting for now, the archive folder, as a path under `ARCHIVE_ROOT`. The server refuses a path that leaves the root (`..`, absolute paths elsewhere, symlinks out). "Check" writes a probe file, reads it back, removes it and shows the result, and saving runs the same check first. The setting change and every check are audited. `doctor` runs the check too. At filing, the sealed data book, each of the packet's documents and a manifest are written to the archive folder (SethD, 2026-10-28, issue 7). Whether a failed copy blocks the filing or is retried and raised to Admin is settled in the phase 10 design.
+- Archive location (Gitea issue 7, Spencer 2026-10-27). Spencer creates an NFS share on the QNAP and mounts it on docker-1 (root, steps in DEPLOY.md), with write access for uid 10001. Compose bind-mounts it into the app and the worker at a fixed root, `ARCHIVE_ROOT` (unset turns the feature off). The Admin page gets a File locations section with one setting for now, the archive folder, as a path under `ARCHIVE_ROOT`. The server refuses a path that leaves the root (`..`, absolute paths elsewhere, symlinks out). "Check" writes a probe file, reads it back, removes it and shows the result, and saving runs the same check first. The setting change and every check are audited. `doctor` runs the check too. At filing, the sealed data book, each of the packet's documents and a manifest are written to the archive folder (SethD, 2026-10-28, issue 7). A failed copy blocks the filing and is raised to Admin (Spencer, 2026-11-03, Gitea issue 57; open question 53).
 - Closes: O3, O4; S4 is IT's pentest, deferred.
 - Done 2026-10-30. The bench maps all 38 old scenarios (tests/e2e/BENCH.md; trel is open question 54) and runs 111 Playwright tests, each failing on a CSP violation. The CSP dropped `'unsafe-inline'`. pip-audit found pypdf, pyasn1 and certifi to bump. The archive copy, the version banner, the VM kit in `deploy/vm/` with VM backups and restore, and the go-live, paper-fallback and security docs are built. The VMs, the NFS share, the timers and the go-live date are Spencer's and Seth's (open question 53 records the archive copy policy).
 
@@ -292,7 +292,7 @@ From phase 9 (2026-10-21), for Seth or Spencer to veto:
 50. The profile CSV `stations` cell: default the cell is the person's whole list of station limits across departments and replaces every limit they had; `-` clears them all; an empty cell leaves them. Phase 9 (2026-10-21) applied the default.
 51. A direct welder assign: default it does not check `profile_active`, so an Admin's inactive switch hides a person from the picker but a typed assign still works. Phase 9 (2026-10-21) applied the default; say if an inactive profile should be refused there too.
 52. A checklist item added while a stage is open: default it applies to that open stage too, because completeness reads the live template, so the item must be ticked before submit. Phase 9 (2026-10-21) applied the default. Replaced by change C8: a template change applies to new packets only. Built on 2026-10-30: a stage freezes its list at START, so a change reaches only stages not yet started.
-53. A failed archive copy at filing (phase 10, Gitea issue 7): default it never blocks the filing. The FILE transaction asks the worker for an `archive-copy` run; the job retries a failed copy after 5, 15 and 60 minutes, then hourly, and raises the `archive_failed` condition to Admin after 3 failed tries or a failed folder check. Admin, Storage, File locations shows the failures and has "Retry now". Turning the archive on later also copies the earlier filings. Seth or Spencer may ask for the stricter rule (the copy must succeed before the packet counts as filed) instead.
+53. A failed archive copy at filing (phase 10, Gitea issues 7 and 57). Answered by Spencer on 2026-11-03 (issue 57): the copy must succeed before the packet counts as filed. FILE checks the folder and copies the packet inside the filing transaction. If the check or the copy fails, the whole filing rolls back, so the packet stays in its previous stage and can be filed again. The person filing gets 503 `archive_failed` with the reason. In a transaction of its own the server writes an `archive.filing_blocked` audit row and raises the `archive_failed` condition to Admin (subject `filing:<packet id>`, critical) with the packet, who tried, the archive folder and the error. A retry that fails again only counts on the open condition, and the next successful filing of that packet clears it. With `ARCHIVE_ROOT` unset or no folder saved, the archive is not in use and filing goes ahead. The worker job still copies the filings made while the archive was off, with its retries and its own `archive_failed` condition (subject `archive`). The earlier default (never block, retry in the worker) is replaced.
 54. The old Clerical "Released" section (every released packet by place, with overdue and 7+ day counts; old bench scenario trel) is not in the new build and not on the Drop list (phase 10, tests/e2e/BENCH.md). Default: not built; the boards and the find bar's Mine, Board and List views show where each packet is. Seth decides whether it is needed for parity.
 
 Changes decided by SethD on 2026-10-26. Each replaces a default above. All five were built on 2026-10-30 (branch `seth-changes`):

docs/SECURITY.md

@@ -195,10 +195,10 @@ The archive closes D6 (Gitea issue 7). It is a copy of every filed packet on the
 - An Admin picks the archive folder in the File locations section of the Admin page (setting `files.archive_dir`, versioned with compare and set like the security settings). The folder is a path under `ARCHIVE_ROOT`. The server refuses an empty path, `..`, an absolute path outside the root, a NUL, a path whose real location is outside the real root (a symlink out), and a folder that does not exist. It never creates folders.
 - "Check" writes a probe file in the folder, syncs it to disk, reads it back, compares it and deletes it. It shows the result, the resolved path and the free space. A save runs the same check first and is refused when it fails. `doctor` runs the check on the saved folder.
 - Every setting change (old and new value), every check (passed or not) and every copy (done or failed) writes an audit row. Admin writes need a password session and pass the Origin check, as every Admin write does.
-- At filing, the worker copies the sealed data book, every document of the packet and a `manifest.json` into `<project>/<unit>/<data book name>-seal<seal id>/`. The manifest names the packet, project, unit, seal id, seal hash, key id, who sealed and filed it and when, the build, and for every file its name, kind, sha256, size and source asset. Anyone can check the copy against the manifest without Packet Station.
+- At filing, FILE copies the sealed data book, every document of the packet and a `manifest.json` into `<project>/<unit>/<data book name>-seal<seal id>/`. The manifest names the packet, project, unit, seal id, seal hash, key id, who sealed and filed it and when, the build, and for every file its name, kind, sha256, size and source asset. Anyone can check the copy against the manifest without Packet Station.
 - Each file is written under a temporary name, synced and renamed. A different file is never overwritten; an identical file counts as done. Nothing in the archive is ever deleted by Packet Station. A refile after a reopen gets a new seal and a new folder, so both copies stay.
-- A failed copy never blocks filing (decided in the phase 10 design; recorded as an open question in [PLAN.md](PLAN.md)). The worker job `archive-copy` runs every 15 minutes and right after each filing. It retries a failed copy after 5, 15 and 60 minutes, then hourly. When a copy has failed 3 times, or the folder check fails, it raises the Admin alert `archive_failed`, cleared when nothing is failed. "Retry now" on the File locations section asks for a run at once.
-- When the archive is turned on later, the job also copies every earlier filing that has no copy yet.
+- The copy must succeed before the packet counts as filed (Spencer, 2026-11-03, Gitea issue 57; open question 53 in [PLAN.md](PLAN.md)). FILE checks the folder and copies inside the filing transaction. A failed check or copy rolls the filing back, so the packet stays in its previous stage. The server then writes an `archive.filing_blocked` audit row and raises the critical Admin alert `archive_failed` for that packet, with who tried, the folder and the error. Filing it again once the archive works clears the alert.
+- When the archive is turned on later, the worker job `archive-copy` copies every earlier filing that has no copy yet. It runs every 15 minutes, retries a failed copy after 5, 15 and 60 minutes, then hourly, and raises `archive_failed` when a copy has failed 3 times or the folder check fails. "Retry now" on the File locations section asks for a run at once.
 - The archive is a plain copy outside the app's access control. The files are not encrypted by the app (D1 is deferred). Who can read or change the share, the NFS export rule (the production VM only), snapshots and the share's own backup are IT's.
 
 ### Warehouse rules (phase 4)