How Packet Station goes live, and what to do when something goes wrong after. Written for Spencer (IT), Seth (engineering), Clerical and the shop supervisors. The install itself is in [DEPLOY.md](DEPLOY.md). What the shop does while the server is down is in [PAPER-FALLBACK.md](PAPER-FALLBACK.md).
- **Go-live date: TBD.** Seth and Spencer decide it together. Pick a day at the start of a week, with Spencer on site for the first days.
- The date is also the feed's go-live boundary. Packet Station starts empty; nothing is migrated from M:. Without the boundary, the first feed run would make drafts for every firmed work order in the release window, including jobs already running on M: or on paper.
- How to set it: Admin page, Feed tab, the go-live date field (setting `golive_date`). Save. The save is audited (`feed.settings_changed`) and the worker picks it up without a restart.
- What it does: a work order released before the date, and not already tracked, makes no packet. It is listed under ignored work orders on Clerical's From VISUAL panel ([FEED.md](FEED.md), "Go-live boundary").
- A job released before the date that should be in Packet Station after all: Clerical presses **Pull in** on that work order in the From VISUAL panel and gives a reason. The next feed run makes its draft.
- Set the date before the production worker's first start (see "Production start"). A run before the date is set makes drafts for the whole window.
Every role comes from an AD group in `OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com`. The 50 groups exist (made 2026-10-15). Nobody can be given a role inside Packet Station.
Who adds people: Spencer, as IT, in AD.
The guide is the M: install's user list (its Admin page, Users: employee ID, name, departments and levels). Map each person like this:
| On M: | AD group |
|---|---|
| Admin | `PS-Admins` |
| Clerical, any level but lead or supervisor | `PS-Clerical` |
| Clerical lead or supervisor | `PS-Clerical-Supervisor` (there is no Clerical lead group) |
| A shop department, worker, lead or supervisor | `PS-<code>-<name>-Worker`, `-Lead` or `-Supervisor`, for example `PS-03-WeldParts-Worker` |
`packetstation ad-groups --list` prints all 50 names. A person can be in several groups; the roles add up.
Before go-live, for each person:
- The AD account has `employeeID` set. The PIN sign-in uses it. A missing or shared employee ID is flagged on the Admin Profiles tab, and a shared one cannot sign in by PIN.
- The account has a mail address if the person should get email.
Then, on the Admin page:
1. Group map tab: press **Refresh from AD now**. The worker reads the groups and makes the users. The tab shows the report (users seen, groups missing or extra, employee ID problems).
2. Profiles tab: enter welder stamps and station limits. A CSV keyed by employee ID can be imported for many people at once.
3. Users tab: check a few people. Each shows the roles AD gives them.
After go-live, joiners, movers and leavers are handled in AD only. The nightly reconcile and the 15-minute role refresh follow AD ([SECURITY.md](SECURITY.md), "AD reconcile and role refresh"). The quarterly access review is IT's.
1. Open `https://packetstation.pescoinc.biz` in the browser.
2. Sign in with a domain password, as an Admin or as a Supervisor of a shop department.
3. Open Account settings. Under **Register this device**, type a name that says where it is, for example `Bldg 2 tablet 3`, and press **Register this device**. The tablet now shows the PIN sign-in.
4. Sign out.
5. Check that it is listed on the Admin page, Devices tab.
Each person who will use a tablet sets their own PIN once: sign in with the domain password (on any PC or on the tablet), open Account settings and set a PIN of 6 to 12 digits. The domain password is asked again to set it. Nobody else can set it. An Admin can only clear it.
**TBD (Spencer):** the list of tablets, their names and where each one goes.
Both names resolve inside PESCO only, to `docker-1`, where Nginx Proxy Manager forwards them to their VMs. Put the production address on the shop PCs and tablets as a shortcut or the home page. Training and trials use the test address.
- The production VM's env file sets `VISUAL_SERVER` and `VISUAL_DATABASE` (`PESCOINC`) explicitly, with `STACK=prod` and `NOTIFY_DRY_RUN=false`. The exact values are in DEPLOY.md.
- Spencer starting the production services with those values is the one-time approval for the worker's scheduled reads of VISUAL production. Nobody else starts it, and Claude Code never does. The test VM never points at VISUAL production.
- Order of the first start:
1. Run the migrate service, then start the app only. Leave the worker stopped.
2. Run `sudo ps-run doctor` on the VM. Every check must pass or show a known SKIP.
3. Sign in as an Admin. On the Feed tab set the go-live date. Check the departments and the auto-release switch (on by default; Seth and Clerical decide whether it stays on at go-live, **TBD**). Check the VISUAL resources tab.
4. On the Notifications tab, send a test message to make sure mail reaches PS-Admins.
5. If the archive share is ready, save the archive folder under File locations and press **Check**.
6. Start the worker. Watch the first feed run on the Feed tab and on Clerical's From VISUAL panel.
## Cut-over
- From the go-live date, new work is in Packet Station. VISUAL releases it, the feed makes the drafts, and Clerical works them from the Clerical screen.
- Jobs already open on M: finish on M:. Nobody copies a job from M: into Packet Station. If a job must move, Clerical pulls its work orders in on the From VISUAL panel and the old M: packet is stopped by hand. Seth decides each such case.
- For a while both systems are in use. Each supervisor knows which jobs are on which system: a job released before the go-live date is on M:, unless it was pulled in.
- The M: install stays as it is until its last packet is filed.
## Retiring M:
- When the last M: packet is filed, M: is retired. **TBD:** Seth confirms that the last packet is filed. The 720 packets on M: at the start of the rebuild were all at draft or warehouse, so some may never be filed; Seth decides what happens to those.
- Spencer, as IT, then makes `M:\Production\Packet Station` read-only for everyone (Read only on the folder and `data\`, no Modify). Nothing is deleted. The folder is kept as the archive of the old jobs.
- Nothing from M: is moved into Packet Station (requirement O5 is superseded).
## Go-live checklist
Each line gets a name and a date when it is done. Keep a printed copy with the incident notes.
Before the date:
- [ ] Test VM built from DEPLOY.md, doctor clean, the bench green in CI.
- [ ] Restore rehearsed on the test VM from a real nightly dump and the assets, row counts and asset hashes matching (O2).
- [ ] Production VM built from the same steps. Doctor clean.
- [ ]`TRUSTED_PROXY_HOST` is the IP address of `docker-1`, `packetstation-firewall.service` is active, and port 8000 refuses a connection from another machine. SSH limited to IT's machines (IT).
- [ ] NPM proxy host for `packetstation.pescoinc.biz` points at the production VM, with the SSE block and the body size from DEPLOY.md. HSTS off in NPM.
- [ ] The NPM to VM hop decided: plain HTTP on the LAN or TLS (Gitea issue 37, Spencer).
- [ ] Secret files in place, root-owned, with a copy kept by IT apart from the VM.
- [ ] Backup timer installed on the production VM. A first dump written and checked. PBS backs the VM up (IT).
- [ ] Archive share mounted at `ARCHIVE_ROOT`, the folder saved and checked under File locations (or the archive left off on purpose, with the reason written here).
- [ ] Penetration test of the production VM and the proxy done, or Spencer's written decision to go live without it (S4, IT).
- [ ] Go-live date picked by Seth and Spencer.
- [ ] People in the `PS-*` groups, Refresh from AD now, employee ID flags cleared on the Profiles tab.
- [ ] Welder stamps and station limits entered.
- [ ] Tablets registered and listed on the Devices tab.
- [ ] Each shop worker has set a PIN.
- [ ] Paper fallback forms printed and in place ([PAPER-FALLBACK.md](PAPER-FALLBACK.md)).
- [ ] The incident notes below printed, with the phone numbers filled in.
- [ ] Training on the test address for Clerical, Warehouse, the supervisors, Q.C. and PM. **TBD (Seth):** who trains whom, and when.
On the day:
- [ ] Production started in the order above, the go-live date set before the worker started.
- [ ] First feed run checked: drafts made only for work released on or after the date.
- [ ] One packet walked through from draft to filed, or as far as the day allows.
- [ ] Shortcuts on the shop PCs and tablets point at the production address.
After:
- [ ] The test stack on `docker-1` stopped once the test VM is in use.
- [ ] Last M: packet filed (Seth), M: made read-only (Spencer).
Keep the phone numbers on the printed copy only. Do not put them in git.
First steps for any incident:
1. Write down the time and what was seen.
2. Call Spencer. If the shop cannot work, also call Seth, who decides whether to go to paper.
3. Do not restart or reinstall anything before Spencer has looked, unless the shop is stopped and Spencer cannot be reached.
### Where to look
- Admin page, Security tab: the alerts of the last 24 hours (account cooldowns, wrong PINs, denied actions, sign-in bursts, Admin changes, IT edits, integrity findings), and the last integrity run.
- Admin page, Sign-in attempts tab: recent failed sign-ins with the account, address and reason.
- Admin page, Access log tab: every audit row, with a CSV export.
- Admin page, Who is in tab: who has Packet Station open now.
- Admin page, Users tab: one person's roles, PIN state and live sessions.
- On the VM: the journal of the app and the worker (`journalctl -u packetstation-app`, `-u packetstation-worker`).
### Lock an account
Packet Station has no lock button of its own. Accounts are locked in AD.
1. In AD, disable the account (or take it out of every `PS-*` group).
2. On the Admin page, Group map tab, press **Refresh from AD now**. The reconcile marks the person inactive and ends every session they have. Each ended session is audited (`auth.session_ended`, `by: ad-reconcile`).
3. Without step 2, the role refresh does the same within 15 minutes.
4. If the person has a PIN: Admin page, Users tab, open the person, give a reason of at least 8 characters and press **Clear PIN**. This is audited (`pin.cleared`). It does not end sessions by itself; step 2 does.
The app's own sign-in limits already stop guessing: 5 failed password sign-ins put an account in a 10-minute cooldown, and 5 wrong PINs disable the PIN. These are on the Security tab.
### End sessions
- One person's sessions: lock the account as above. There is no Admin button that ends another person's sessions without AD.
- Your own other sessions: Account settings, **Sign out other sessions** (password session).
- A lost or stolen tablet: Admin page, Devices tab, **Revoke** on that device. Every PIN session on it ends at once and the tablet can no longer show the PIN sign-in. A password session open on it is not ended by the revoke: lock that account in AD as above, or wait for the idle sign-out (15 minutes).
- Everyone at once: rotate the session key (below).
### Rotate the session key
The session key signs the session cookies and the device cookies. Rotate it when it may have leaked.
1. Write a new random key of at least 32 characters to the session key file in `/etc/packetstation/secrets/` (root-owned, same owner and mode as before). Never type it into chat or a ticket.
2. Restart the app and the worker.
3. Effect: every session ends. Every device cookie stops working, so **every tablet must be registered again** (see "Tablets"). PINs are not affected.
### Rotate the seal key
The seal key signs the filing seals and the stamps on every printed and exported copy. Rotate it only when it may have leaked, since the cost is high.
1. Copy the old key to a safe place offline, kept by IT. It is the only way to check the old seals and stamps later.
2. Write a new random key of at least 32 characters to the seal key file in `/etc/packetstation/secrets/`.
3. Restart the app and the worker.
4. Effect: packets filed from now on are sealed with the new key. Packet Station keeps one key, so it can no longer check the seals made before. Every nightly integrity run reports each of them as "sealed with another key", and the stamp check fails on copies printed before the rotation. Filed records are not changed.
5. Write the rotation date down with the incident, so the findings that follow can be read against it.
### Other secrets
Each of these is a file in the same folder. Change the password at its source first, then the file, then restart the app and the worker.
-`svc-packetstation` (AD bind): reset in AD by IT.
-`visualkit_ro` (VISUAL): reset by the VISUAL database admin.
-`mfileskitRO` (M-Files): reset by the M-Files admin.
- The PostgreSQL role passwords: changed with `ALTER ROLE` on the VM by Spencer.
### Run the integrity check
It runs every night at 04:00. To run it now:
- On the VM: `sudo ps-run job run integrity`. `ps-run` runs a `packetstation` command as the service user with the production settings (DEPLOY.md). It prints the run and exits 1 if the run failed.
- Or, as an Admin in a password session, `POST /api/jobs/integrity/run`. The Admin page has no button for it.
The result shows on the Admin page, Security tab, Integrity. Any finding (a seal that does not match, a file whose hash changed, a missing or extra file, a break in the audit chain) raises the `integrity_failure` mail, which nobody can opt out of. Findings are never fixed by the job. A finding is evidence: do not delete or replace files before Spencer has looked.
### Stop the feed
If the VISUAL feed makes wrong drafts or change orders: Admin page, Feed tab, tick the **Paused** box and save. Scheduled feed, drawing and auto-release runs stop at once. Run now still works for a test. Untick to resume.
### Restore
Use this when the database or the assets are lost or damaged beyond repair. The steps and scripts are in DEPLOY.md, "Restore".
1. Seth puts the shop on paper ([PAPER-FALLBACK.md](PAPER-FALLBACK.md)).
2. Stop the app and the worker.
3. Restore the newest good nightly dump into a fresh database, and copy the assets back (from the VM's PBS backup if the assets are lost).
4. Check every asset's hash and compare the row counts, with the restore script.
5. Run `sudo ps-run doctor`, start the app and the worker, and run the integrity check.
6. Anything done after the dump (up to a day) is lost. It is entered again from paper and from what people remember, as in the paper fallback, and checked the same way.
### Afterwards
Write the incident up: what happened, when, what was done, what was rotated or restored, and what should change. **TBD (Spencer):** where IT keeps incident records.
