Phase 10: regression bench, hardening, archive location, native VM kit, go-live docs #55

Merged

spencerm merged 10 commits from phase-10 into main 2026-11-03 21:16:19 +00:00

Conversation 0 Commits 10 Files Changed 103

+6324 -648

spencerm commented 2026-10-31 00:09:48 +00:00

Owner

Copy Link

Copy Source

Phase 10, the last phase of the plan. Branch phase-10, 10 commits on main 3147fed.

What is in it

• Regression bench. tests/e2e/BENCH.md maps all 38 old scenarios (tests/ps). 22 were already covered, 10 have new specs, 2 are covered by server tests and 2 are on the Drop list (demo mode, Excel Job Release). trel (the old Clerical "Released" section) is not built: PLAN open question 54, for Seth. That makes 111 Playwright tests on synthetic fixtures. Every test fails on a CSP violation.
• Hardening.
  • The final CSP: no 'unsafe-inline' styles, frame-src and object-src are 'none', and a PDF opened in its own tab gets object-src 'self'.
  • pip-audit bumps: pypdf 6.21.4, pyasn1 0.6.5, certifi 2026.10.23. npm audit shows no high findings.
  • A build-version banner offers a reload when the server build changes. It never reloads by itself.
• Archive location (Gitea issue 7).
  • ARCHIVE_ROOT: unset turns the feature off.
  • Admin, Storage, File locations has Check and Save. A path that leaves the root is refused (.., absolute paths elsewhere, symlinks out).
  • Each filing is copied by the new archive-copy job: the sealed data book, the documents and manifest.json.
  • Decision (open question 53): a failed copy never blocks filing. It is retried after 5, 15 and 60 minutes, then hourly. After 3 failures it raises the critical archive_failed alert to Admin, and "Retry now" asks for a run.
  • Turning the archive on later also copies the earlier filings.
  • Setting changes, checks and copies are audited. doctor runs the check. Migration 0012 adds archive_copies.
• Native VM install kit (deploy/vm/):
  • systemd units for the app, worker, migrate and firewall (port 8000 open to NPM only).
  • A launcher that hands each unit only its own secrets (LoadCredential).
  • install.sh (PostgreSQL 16, ODBC 18, the PESCO CA), plus upgrade.sh with rollback, make-release.sh, ps-run and ps-doctor.
  • Env examples with explicit VISUAL_SERVER and VISUAL_DATABASE.
  • Backups and restore rewritten without docker exec.
  • The compose files stay for CI.
• Docs.
  • SECURITY.md is rewritten around the ID table: O3, O4 and D6 closed, S1 tuned, S4 stays with IT.
  • GO-LIVE.md has the incident notes. PAPER-FALLBACK.md is written.
  • DEPLOY.md covers the VM install, the NPM proxy hosts for the VMs, the NFS mount, and VM backup and restore.
  • Also updated: README, CHANGELOG, PLAN (questions 53 and 54, the phase 10 note), POLICY (the four admin routes), SCHEMA and FEED.
• Bug fixes found by the bench.
  • The MTR viewer's iframe was refused by our own headers, so it showed nothing. It now uses pdf.js.
  • The hand upload and rev approval lines showed a user id instead of the person's name.

Verified

• Gitea Actions on 1cf37d9: server, frontend and e2e are all green.
• docker-1 (own compose project; the test stack was not touched):
  • 6161 server tests passed, including the Samba DC tests.
  • 385 frontend tests passed.
  • Playwright passed 111 of 111 in 4.4 minutes, on a clean stack and in two earlier runs.
  • doctor is at 0012, and the archive check passes.
• The VM kit ran the whole chain in a throwaway Ubuntu 24.04 container: install, PGDG and Microsoft repos, venv, migrate 0001 to 0012, doctor, backup and restore. shellcheck and systemd-analyze verify are clean.
• Not tested until a VM exists:
  • real systemd start, the firewall table and NFS;
  • whether Chrome shows a PDF in its own tab under object-src 'self' (only the header is tested).

What Spencer does by hand

1. Issue 37: pick the VM OS, names, addresses and size. Ubuntu 24.04 is recommended, because Debian 12 has Python 3.11 and the server needs 3.12. Also decide HTTP or TLS between NPM and the VM.
2. Build the test VM first. Follow DEPLOY.md, "Native VM install". The short version:
   • Make a release from the CI image with deploy/vm/make-release.sh and copy it to the VM.
   • Run sudo install.sh --stack test --ca-bundle ... --generate-secrets.
   • Write the three passwords without echo, then edit /etc/packetstation/packetstation.env.
   • Run sudo ps-upgrade <tarball>, then sudo ps-doctor.
   • Run systemctl enable --now packetstation-app packetstation-worker.
3. NFS: make the QNAP export, add the fstab line, mount it and create the packets folder as the service user. Set ARCHIVE_ROOT, then save packets under Admin, Storage, File locations.
4. Timers: enable packetstation-backup.timer on each VM. Run one backup, then rehearse a restore into ps_restored and record it in DEPLOY.md.
5. NPM: point packetstation-test.pescoinc.biz, and later packetstation.pescoinc.biz, at the VM on port 8000. Check that port 8000 is closed from other PCs.
6. Retire the docker-1 test stack once the test VM is accepted. Keep its volumes for a while, and remove its line from docker-1's backup.conf. The repo's ps-backup.sh is now VM-only and refuses the old docker config, so leave docker-1's installed copy in place until then.
7. Production: repeat with --stack prod and the explicit VISUAL target. Follow GO-LIVE.md, "Production start": migrate and start the app with the worker stopped, set the go-live date, then start the worker. That start is the VISUAL approval.
8. The go-live date is set with Seth. The other open facts are listed in GO-LIVE.md: the pentest (S4, IT), phone numbers, the tablet list, and Seth's paper-fallback answers.

Open for Seth: questions 53 (archive copy policy) and 54 (trel).

🤖 Generated with Claude Code

Phase 10, the last phase of the plan. Branch `phase-10`, 10 commits on main 3147fed. ## What is in it - **Regression bench.** `tests/e2e/BENCH.md` maps all 38 old scenarios (`tests/ps`). 22 were already covered, 10 have new specs, 2 are covered by server tests and 2 are on the Drop list (demo mode, Excel Job Release). `trel` (the old Clerical "Released" section) is not built: PLAN open question 54, for Seth. That makes 111 Playwright tests on synthetic fixtures. Every test fails on a CSP violation. - **Hardening.** - The final CSP: no `'unsafe-inline'` styles, `frame-src` and `object-src` are `'none'`, and a PDF opened in its own tab gets `object-src 'self'`. - pip-audit bumps: pypdf 6.21.4, pyasn1 0.6.5, certifi 2026.10.23. npm audit shows no high findings. - A build-version banner offers a reload when the server build changes. It never reloads by itself. - **Archive location (Gitea issue 7).** - `ARCHIVE_ROOT`: unset turns the feature off. - Admin, Storage, File locations has Check and Save. A path that leaves the root is refused (`..`, absolute paths elsewhere, symlinks out). - Each filing is copied by the new `archive-copy` job: the sealed data book, the documents and `manifest.json`. - **Decision (open question 53):** a failed copy never blocks filing. It is retried after 5, 15 and 60 minutes, then hourly. After 3 failures it raises the critical `archive_failed` alert to Admin, and "Retry now" asks for a run. - Turning the archive on later also copies the earlier filings. - Setting changes, checks and copies are audited. `doctor` runs the check. Migration 0012 adds `archive_copies`. - **Native VM install kit** (`deploy/vm/`): - systemd units for the app, worker, migrate and firewall (port 8000 open to NPM only). - A launcher that hands each unit only its own secrets (`LoadCredential`). - `install.sh` (PostgreSQL 16, ODBC 18, the PESCO CA), plus `upgrade.sh` with rollback, `make-release.sh`, `ps-run` and `ps-doctor`. - Env examples with explicit `VISUAL_SERVER` and `VISUAL_DATABASE`. - Backups and restore rewritten without `docker exec`. - The compose files stay for CI. - **Docs.** - SECURITY.md is rewritten around the ID table: O3, O4 and D6 closed, S1 tuned, S4 stays with IT. - GO-LIVE.md has the incident notes. PAPER-FALLBACK.md is written. - DEPLOY.md covers the VM install, the NPM proxy hosts for the VMs, the NFS mount, and VM backup and restore. - Also updated: README, CHANGELOG, PLAN (questions 53 and 54, the phase 10 note), POLICY (the four admin routes), SCHEMA and FEED. - **Bug fixes found by the bench.** - The MTR viewer's iframe was refused by our own headers, so it showed nothing. It now uses pdf.js. - The hand upload and rev approval lines showed a user id instead of the person's name. ## Verified - **Gitea Actions on 1cf37d9:** server, frontend and e2e are all green. - **docker-1** (own compose project; the test stack was not touched): - 6161 server tests passed, including the Samba DC tests. - 385 frontend tests passed. - Playwright passed 111 of 111 in 4.4 minutes, on a clean stack and in two earlier runs. - `doctor` is at 0012, and the archive check passes. - **The VM kit** ran the whole chain in a throwaway Ubuntu 24.04 container: install, PGDG and Microsoft repos, venv, migrate 0001 to 0012, doctor, backup and restore. shellcheck and `systemd-analyze verify` are clean. - **Not tested until a VM exists:** - real systemd start, the firewall table and NFS; - whether Chrome shows a PDF in its own tab under `object-src 'self'` (only the header is tested). ## What Spencer does by hand 1. **Issue 37:** pick the VM OS, names, addresses and size. Ubuntu 24.04 is recommended, because Debian 12 has Python 3.11 and the server needs 3.12. Also decide HTTP or TLS between NPM and the VM. 2. **Build the test VM first.** Follow DEPLOY.md, "Native VM install". The short version: - Make a release from the CI image with `deploy/vm/make-release.sh` and copy it to the VM. - Run `sudo install.sh --stack test --ca-bundle ... --generate-secrets`. - Write the three passwords without echo, then edit `/etc/packetstation/packetstation.env`. - Run `sudo ps-upgrade <tarball>`, then `sudo ps-doctor`. - Run `systemctl enable --now packetstation-app packetstation-worker`. 3. **NFS:** make the QNAP export, add the fstab line, mount it and create the `packets` folder as the service user. Set `ARCHIVE_ROOT`, then save `packets` under Admin, Storage, File locations. 4. **Timers:** enable `packetstation-backup.timer` on each VM. Run one backup, then rehearse a restore into `ps_restored` and record it in DEPLOY.md. 5. **NPM:** point `packetstation-test.pescoinc.biz`, and later `packetstation.pescoinc.biz`, at the VM on port 8000. Check that port 8000 is closed from other PCs. 6. **Retire the docker-1 test stack** once the test VM is accepted. Keep its volumes for a while, and remove its line from docker-1's `backup.conf`. The repo's `ps-backup.sh` is now VM-only and refuses the old docker config, so leave docker-1's installed copy in place until then. 7. **Production:** repeat with `--stack prod` and the explicit VISUAL target. Follow GO-LIVE.md, "Production start": migrate and start the app with the worker stopped, set the go-live date, then start the worker. That start is the VISUAL approval. 8. **The go-live date** is set with Seth. The other open facts are listed in GO-LIVE.md: the pentest (S4, IT), phone numbers, the tablet list, and Seth's paper-fallback answers. Open for Seth: questions 53 (archive copy policy) and 54 (`trel`). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

spencerm added 8 commits 2026-10-31 00:09:48 +00:00

Bump pypdf 6.21.4, certifi 2026.10.23, pyasn1 0.6.5 for pip-audit findings ... 3fdf063b70

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Archive location (Gitea issue 7): ARCHIVE_ROOT, Admin file-locations API, archive-copy job, doctor check ... da31b0a87a

A filed packet's sealed data book, documents and a manifest are copied to the archive folder by the worker.
A failed copy never blocks filing: it is retried with backoff and raises the archive_failed condition.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Version banner, File locations section, final CSP (no unsafe-inline styles, frame-src none, object-src none) ... 6e820ba0a5

The MTR viewer draws PDFs with pdf.js instead of an iframe the server's frame-ancestors refused.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Docs: SECURITY.md rewritten around the ID table, GO-LIVE.md with incident notes, PAPER-FALLBACK.md ... cb00ea7c5c

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Docs: Admin file-locations routes in POLICY.md, open question 53 (archive copy never blocks filing), ARCHIVE_ROOT in the env examples ... f025479358

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Native VM install kit: systemd units, launcher, install and upgrade scripts; backups, restore and doctor without docker exec; DEPLOY.md VM, NPM and NFS sections ... 45a1d1a42a

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Env examples: SITE_ID=PESCO, MFILES_CA_BUNDLE left unset; CLAUDE.md and the NPM notes point at the VM kit ... fce4528cdd

Co-Authored-By: Claude Opus 5.5 <[email protected]>

README and CHANGELOG for phase 10 ... c52edfa9b1

Co-Authored-By: Claude Opus 5.5 <[email protected]>

spencerm added 2 commits 2026-10-31 00:18:44 +00:00

Regression bench: the old build's 38 scenarios mapped in tests/e2e/BENCH.md, 20 new Playwright tests, CSP-violation guard on every test ... a5246e2aa5

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Fix: hand upload and rev approval lines name the person, not the user id; PLAN questions 53, 54 and the phase 10 note ... 1cf37d94e7

Co-Authored-By: Claude Opus 5.5 <[email protected]>

spencerm merged commit f5390ac7cf into main 2026-11-03 21:16:19 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

decision: Clerical

decision: IT

decision: Q.C.

decision: Seth

phase 10

phase 3

phase 4

phase 5

phase 6

phase 7

phase 8

phase 9

production

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

SethD (Seth Dostaler) spencerm (Spencer McGuire)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: spencerm/PacketStation#55