# Deploying Packet Station

How Packet Station runs on `docker-1`. Sections not built yet are marked "planned, phase N". The design is in [PLAN.md](PLAN.md) under "Deployment and CI".

## Host facts

- Host: `docker-1`, 192.168.101.12.
- Ubuntu 24.04 LTS, Docker 29, Compose v5. Clock in UTC.
- Nginx Proxy Manager already runs there as container `npm_app_1` on network `npm_default` (172.22.0.0/16). Its compose project is in `/home/spencerm/npm`.
- The host also runs Gitea (`gt.pescoinc.biz`, the container registry and the Python index) and `pesco-ncr`.
- Proxmox Backup Server backs up the whole VM at 03:00 daily.
- Host patching and hardening are IT's (requirement H5).
- Inside containers only full names resolve. Docker's resolver has no search domain, so every host setting is an FQDN (for example `vsql.pesco.com\DEV`, `app-03.pesco.com`).

## The two stacks

Both stacks run from the same image as separate compose projects.

| Stack | Compose project | DNS name | Network alias on npm_default | VISUAL |
|---|---|---|---|---|
| Test | `packetstation-test` | `packetstation-test.pescoinc.biz` | `packetstation-test-app` | DEV: `vsql.pesco.com\DEV` / `PESCOINCDEV` |
| Production | `packetstation` | `packetstation.pescoinc.biz` | `packetstation-app` | `vsql` / `PESCOINC`, set explicitly (planned, phase 10) |

- The app publishes no port. It joins `npm_default` as an external network with an explicit alias per stack (`APP_ALIAS`). Plain service names would collide, since both stacks share that network.
- PostgreSQL stays on each stack's own internal `backend` network, never on `npm_default`.
- The test stack never points at VISUAL production. The production stack sets `VISUAL_SERVER` and `VISUAL_DATABASE` explicitly. Spencer starting it with those values is the one-time approval for its scheduled polling.
- There is no M-Files test vault. Both stacks read the production vault, read-only, as `mfileskitRO`.
- The internal A records for both names point at 192.168.101.12 in the AD-integrated `pescoinc.biz` zone.

## Services

- `postgres`: PostgreSQL 16, named volume, `backend` network only. Its init creates the database roles from secrets.
- `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
- `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup and housekeeping (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only (used from phase 9). Its healthcheck reads the heartbeat file.
- `clamav` (optional, profile `clamav`, phase 4): `clamav/clamav:1.4`, pinned by tag (with a comment in the compose file), not by digest. clamd on port 3310 for the app and the worker, reached by the network alias `packetstation-clamav` (as `postgres` has its own alias). `backend` network plus egress (freshclam downloads the signatures), no `npm_default`, no published port. Volume `clamav-db` at `/var/lib/clamav`, `mem_limit: 2g`, healthcheck `clamdcheck.sh`. Started only with `--profile clamav`. See "Virus scan (ClamAV)".
- All services restart unless stopped, log to stdout (json-file), and have healthchecks.

## Secrets

Docker secrets, one file each, under `/etc/packetstation/<stack>/secrets/`. Owner `root:10001` (10001 is the image's non-root user), mode `0440`.

| Secret | Used by |
|---|---|
| `postgres_superuser_password` | PostgreSQL bootstrap |
| `postgres_migrate_password` | migrate role (`ps_migrate`) |
| `postgres_app_password` | app role (`ps_app`) |
| `visualkit_password` | `VISUAL_PASSWORD_FILE` for `visualkit_ro` |
| `mfileskit_password` | `mfileskitRO`; the entrypoint exports it as `MFILES_PASSWORD` |
| `ldap_bind_password` | `LDAP_PASSWORD_FILE` for `svc-packetstation` |
| `session_key` | session signing |
| `seal_key` | HMAC filing seals and controlled-copy stamps |

Never put a secret in the repo, the env file, a log or chat. Settings read secrets through `*_PASSWORD_FILE` paths.

The worker gets `postgres_app_password` (it logs in as the app role, with application name `ps-worker`), `visualkit_password`, `mfileskit_password`, `ldap_bind_password` and `seal_key`. The worker never logs a password: job errors are redacted before they are stored, and the redaction also strips the VISUAL and M-Files passwords themselves.

The PESCO CA chain is mounted into the containers once and referenced by `LDAP_CA_BUNDLE`, `MFILES_CA_BUNDLE` and `SMTP_CA_BUNDLE`.

## Env files

One folder per stack under `/etc/packetstation/`: `/etc/packetstation/test/` and `/etc/packetstation/prod/`. Each holds `stack.env` (settings only, no secrets) and `secrets/`. The env file sets `PROJECT_NAME`, `SECRETS_DIR` and `APP_ALIAS`, which the compose files require. `.env.example` in the repo lists every setting.

Key settings:

- `VISUAL_SERVER`, `VISUAL_DATABASE`, `VISUAL_USER=visualkit_ro`, `VISUAL_PASSWORD_FILE`.
- `MFILES_URL=https://mfiles.pescoinc.biz/REST`, `MFILES_VAULT_GUID={BB69270D-023E-4EA8-B899-AD20B1349702}`, `MFILES_USERNAME=mfileskitRO`, `MFILES_CA_BUNDLE`.
- `LDAP_URL=ldaps://DC-02.PESCO.COM:636 ldaps://PDC-1.PESCO.COM:636`, `LDAP_BIND_DN=CN=svc-packetstation,OU=System_Accounts,OU=Admin,DC=PESCO,DC=com`, `LDAP_GROUP_BASE_DN=OU=PacketStation,OU=Groups,OU=PESCO_FARMINGTON,DC=PESCO,DC=com`, `LDAP_PASSWORD_FILE`, `LDAP_CA_BUNDLE`.
- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `SMTP_FROM=noreply@pescoinc.biz`, `SMTP_CA_BUNDLE`.
- `APP_ALIAS` per stack.
- Identity timing and limits, all optional with these defaults: `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
- Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
- Uploads and storage, all optional: `UPLOAD_MAX_BYTES` (default 25 MB, drawings, MTRs and documents), `UPLOAD_MAX_BYTES_BOM` (default 5 MB, hand BOMs), `ASSETS_QUOTA_WARN_PCT` (default 85, 1 to 99) and `ASSETS_QUOTA_BYTES` (default 0, which means the size of the assets volume; any other value is a byte budget for the stored files). See "Storage quota".
- Virus scan, all optional: `CLAMAV_HOST` (empty by default, which means no scan; `packetstation-clamav` for the profile service), `CLAMAV_PORT` (default 3310) and `CLAMAV_TIMEOUT_SECONDS` (default 30). The env examples carry `#CLAMAV_HOST=packetstation-clamav` commented out. See "Virus scan (ClamAV)".
- `SITE_ID` (default `PESCO`) and `SITE_TZ` (default `America/Denver`): the VISUAL site in every call, and the time zone for shop hours and the schedule.
- The session cookie is marked `Secure` when the request arrived over https (as reported by NPM through `X-Forwarded-Proto`). Behind the proxy that is always the case.

## Feed settings live in the database

The feed's schedule and rules are not env settings. Intervals, shop hours, the window, the go-live date, departments, the drawing recheck and budget, labor tickets, auto-release and the pause switch are one `settings` row (key `feed`), edited by an Admin on the Admin page, Feed tab. The resource map is the `visual_resource_map` table, edited on the "VISUAL resources" tab. A save notifies the worker, which applies it without a restart. A new stack starts with the defaults in [FEED.md](FEED.md), "Settings". Set the go-live date before the production feed's first run.

## Deploy command

Test stack:

```bash
docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml up -d
```

Production (planned, phase 10): the same command with `-p packetstation` and the production env file. The first production start is Spencer's approval.

Images come from `gt.pescoinc.biz` by tag. CI tags each tested image `:sha-<sha>`; a release re-tags it `vX.Y.Z`. Never `latest`. A deploy is a new image with a short downtime window.

## Virus scan (ClamAV, phase 4)

ClamAV is optional and off by default. It closes requirement D5. Turn it on in the test stack first, check it, then the production stack.

1. Set `CLAMAV_HOST=packetstation-clamav` in the stack's env file (`/etc/packetstation/test/stack.env`). Leave `CLAMAV_PORT` and `CLAMAV_TIMEOUT_SECONDS` at their defaults.
2. Start the stack with the profile:

```bash
docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml --profile clamav up -d
```

3. Wait for `clamav` to turn healthy. On the first start the volume `packetstation-test_clamav-db` is empty, so freshclam downloads the full signature set (a few hundred MB) before clamd answers. That takes several minutes. Later starts reuse the volume and only fetch updates. freshclam needs egress to the ClamAV mirrors, which is why the service is on the `egress` network.
4. Check it: Admin, Storage tab, the ClamAV line reads "packetstation-clamav: reachable". `packetstation doctor` also has a `clamav` check (a PING to clamd; SKIP when `CLAMAV_HOST` is empty). Upload a small PDF to a draft and confirm it is stored.

Every later `up` for that stack must carry `--profile clamav`. Without it the service is not started.

Fail closed. While `CLAMAV_HOST` is set, an upload the scanner cannot check is refused with "The virus scanner is not reachable. Try again in a minute." (503 `scan_unavailable`). That covers clamd still loading signatures, stopped, or out of memory. The worker's drawing fetches are refused the same way and tried again later. A file with a signature hit is refused (422 `infected`), never stored, and audited as `asset.rejected` with the signature name.

To turn it off: remove `CLAMAV_HOST` from the env file, recreate `app` and `worker` (`up -d` without the profile), then stop the service with `docker compose ... --profile clamav stop clamav`. The volume can stay for the next time. The CI overlay never runs ClamAV.

clamd holds the signatures in memory, about 1.2 GB with the full set and more while it reloads them, so the service has a 2 GB limit. Check docker-1 has the headroom for both stacks before turning it on in production.

## Storage quota (phase 4)

Every stored file and the housekeeping job check the assets volume. With `ASSETS_QUOTA_BYTES=0` (the default) the check reads the volume's own used and total bytes; with a byte budget it adds up the stored files' sizes against it. At `ASSETS_QUOTA_WARN_PCT` percent (85 by default) the app opens one `notification_events` alert of kind `assets_quota` and audits `storage.quota_warning`. When usage drops back under, the alert is resolved. A failed check is logged and never fails an upload.

The Admin page, Storage tab, shows the usage bar (used of total, percent, the threshold), the open quota alert with an Acknowledge button, the ClamAV line ("Not configured", or the host with reachable or not reachable) and every open alert. Phase 9 sends the alert by email. Until then, check the Storage tab.

## Python packages and the Gitea index

The Gitea PyPI index `https://gt.pescoinc.biz/api/packages/spencerm/pypi/simple/` serves only `visualkit`, `mfileskit` and their dependencies.

- Install `visualkit` and `mfileskit` from it with `--index-url`. Never `--extra-index-url`, which would let a public package with the same name win.
- Install everything else from public PyPI in a separate pip step.
- The image needs Microsoft ODBC Driver 18 (`msodbcsql18`) from Microsoft's Debian 12 repository for VisualKit.

## NPM proxy host

One proxy host per stack in Nginx Proxy Manager. To be recorded under `deploy/` as well.

- Domain: `packetstation-test.pescoinc.biz` (test) or `packetstation.pescoinc.biz` (production).
- Scheme `http`, forward host the stack's alias (`packetstation-test-app` or `packetstation-app`), port `8000`.
- SSL: the stack's certificate, Force SSL on, HTTP/2 on.
- NPM HSTS off. The app sends HSTS itself.
- Advanced config: `client_max_body_size 30m;` plus a location block for the event stream:

```nginx
client_max_body_size 30m;

location /api/events {
    proxy_pass $forward_scheme://$server:$port;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}
```

The block uses NPM's `$server` variables so the alias is re-resolved. The app also sends `X-Accel-Buffering: no` on `/api/events`.

Open check (plan question 16): confirm "Block common exploits" does not interfere with uploads when the test proxy host is made.

The app trusts `X-Forwarded-*` only from `npm_app_1`'s own address, re-resolved every 60 seconds. Other stacks share `npm_default`, and the per-address rate limit depends on the real client address.

## Known issue: the internal CA signs with SHA-1 (found 2026-10-17)

PESCO-PSYLOCKE-CA (valid to 2027-09-27) signs with sha1WithRSAEncryption, and so do the certificates it issued to DC-02.PESCO.COM and PDC-1.PESCO.COM on 636 and to APP-03.PESCO.COM on 25 (all valid to April 2027). OpenSSL 3 on Debian 12 refuses SHA-1 signatures at its default security level 2, so LDAPS and SMTP STARTTLS fail from inside the image with "CA signature digest algorithm too weak", while Windows and browsers still accept them. Verification passes at security level 0.

mfiles.pescoinc.biz uses a public Let's Encrypt certificate, so MFILES_CA_BUNDLE must stay unset; only LDAP_CA_BUNDLE and SMTP_CA_BUNDLE point at the PESCO chain.

The fix is IT's: set the CA to issue with SHA-256 and reissue the domain controller and relay certificates. The CA certificate itself may stay SHA-1, since OpenSSL does not check the trust anchor's own signature. Until then the test stack cannot sign in or verify the relay without a test-only OpenSSL security-level override, which is Spencer's decision to apply, never the production stack's. The doctor should report a SHA-1 chain with this hint (follow-up).

The test stack on docker-1 lives under /home/spencerm/packetstation/test (stack.env, secrets/, ca/pesco-ca.pem, backups/) because the session that set it up had no sudo; move it to /etc/packetstation/test when convenient.

## Doctor

`packetstation doctor` proves a host and a stack are ready. Run it inside the image:

```bash
docker compose -p packetstation-test --env-file /etc/packetstation/test/stack.env \
  -f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm app doctor
```

It checks:

- Settings and the stack guard (test refuses a production VISUAL target).
- `visualkit doctor` and `mfileskit doctor`. This is the first real Linux test of both libraries.
- Database connection, roles, migrations at head, and an audit UPDATE that must fail.
- LISTEN/NOTIFY round trip.
- Assets volume and free space. Backup folder (planned, phase 9).
- LDAP TLS bind on both controllers, reading `memberOf`, `employeeID` and `userAccountControl` as the service account, the 50-group count and one nested-group query.
- SMTP STARTTLS with certificate verification.
- ClamAV (phase 4): clamd answers PING at `CLAMAV_HOST`. SKIP when no host is set.
- tzdata.

Use `--skip visual,mfiles,ldap,smtp` to run only the local checks. The network checks need the secrets and the CA bundle in place.

### Verified on docker-1 (2026-10-15, phase 0)

The image built on docker-1 from this tree (python:3.12-slim-bookworm, ODBC Driver 18, visualkit 0.3.0, mfileskit 0.1.2). The CI stack (`deploy/compose.ci.yaml` with throwaway secrets) came up with postgres, migrate, app and worker all healthy. Inside it, `doctor --skip visual,mfiles,ldap,smtp` passed: migrations at head, the audit UPDATE denied for `ps_app`, LISTEN/NOTIFY round trip, assets volume, tzdata. `visualkit doctor` in the image found the ODBC driver and pyodbc on Linux; the connection itself was not tried because the CI stack has no VISUAL credentials. `mfileskit doctor` imported and ran on Linux up to its settings check for the same reason. The full doctor with the real secrets, the CA bundle and both domain controllers is the first step of the test stack deployment.

Inside containers the short name `VSQL\DEV` does not resolve. The env files use `vsql.pesco.com\DEV`.

## CI stack and the Samba domain controller (phase 1)

`deploy/compose.ci.yaml` adds a throwaway Samba Active Directory domain controller (`samba-dc`, realm `PSTEST.LOCAL`, host `dc1.pstest.local` on the internal `backend` network) built from `deploy/samba/`. On first start it provisions the domain, generates a CA and a server certificate, seeds the OU tree, the service account, the groups (49 of the 50, plus one extra, to test the report) and 17 test users with random passwords, and exports `ca.pem` and `users.json` to the `ci-dc-export` volume. The app, migrate and worker mount that volume at `/etc/ssl/pstest` and `deploy/ci/ci.env` points `LDAP_CA_BUNDLE` at it, so CI exercises the real LDAPS path. The `playwright` service (profile `e2e`) drives the app at `http://ps-app:8000` with the exported users. [deploy/samba/README.md](../deploy/samba/README.md) has the details and the commands.

The DC runs with `cap_add: [SYS_ADMIN]`, which Samba needs to set the sysvol ACLs. The app carries the alias `ps-app` for Playwright because `app` is a top-level domain on Chromium's HSTS preload list, so `http://app:8000` is forced to TLS and fails. Running the server test suite against the stack's own PostgreSQL is not safe: the `pg_settings` fixture resets the role passwords (see deploy/samba/README.md).

The real PESCO domain controllers are never used by CI. The test and production stacks do not run the DC.

### CI feed fixtures (phase 3)

CI never reads VISUAL or M-Files. `deploy/ci/ci.env` sets `FEED_SOURCE=fixture`, `DRAWING_SOURCE=fixture` and `FIXTURE_DIR=/fixtures`. The image does not contain `tests/`, so the e2e job generates the fixtures on the runner before the stack starts:

```bash
python -m pip install pypdf==6.21.0
python -m tests.fixtures.generator --out tests/fixtures/generated
```

The generator is deterministic and writes `feed.1.json`, `feed.2.json`, `drawings.1.json`, `drawings.2.json`, the PDFs they name and `state` (`1`). `tests/fixtures/generated/` is ignored by git. `deploy/compose.ci.yaml` binds it at `/fixtures` into the worker (read-only) and the Playwright container (read-write, so the scenario can write `state` = `2` and press Run now again). `packetstation fixtures use N --dir DIR` switches the state by hand. The CI feed runs at the default intervals; the scenario does not wait for them, it presses Run now.

## Gitea Actions runner (phase 2)

`.gitea/workflows/ci.yml` runs on a Gitea Actions runner named `docker-1`, installed on docker-1 at
`~/gitea/runner` from [deploy/runner/](../deploy/runner/): one `act_runner` container (pinned by digest) with
the host Docker socket, label `ubuntu-latest` mapped to `docker.gitea.com/runner-images:ubuntu-latest`,
capacity 2. The server, frontend and image jobs run in job containers from that image. The e2e job starts
the CI compose stack through the runner's Docker proxy on docker-1's daemon, which maps the job workspace so
the stack's bind mounts resolve and removes the stack when the job ends. The runner was registered on
2026-10-16 with a token generated by `gitea actions generate-runner-token` inside the `gitea` container; the
token was never written down. [deploy/runner/README.md](../deploy/runner/README.md) has the install, update
and re-register steps.

The repository is `spencerm/PacketStation` on Gitea with Actions enabled. The workflow ran there on
2026-10-16: server, frontend and e2e pass (the e2e job builds the image, starts the CI stack with the Samba
DC and runs the Playwright scenarios on docker-1's daemon). The image job needs the repository secrets
`REGISTRY_USER` and `REGISTRY_TOKEN` (a Gitea token with package write) and fails until they are set.
Two runner settings were needed and are in deploy/runner: the actions cache server published on docker0
port 8088 (job containers cannot reach the runner's own network, and every cache call timed out), and the
job workspace on a host directory allowed in `valid_volumes` (so the stack's bind mounts resolve).

## Backup (planned, phase 9)

`pg_dump` of both stacks from a systemd timer on the host at 02:00 America/Denver, before the 03:00 PBS run. Seven dumps kept in `/var/backups/packetstation/<stack>/` with last-success and last-failure markers. The worker's backup-check job alerts if the newest dump is missing or older than 26 hours. Scripts and units go in `scripts/`.

## Restore (planned, phase 9)

Restore the dump into a fresh database and copy the assets back, from the live folder or a file-level PBS restore. Rehearsed once on the test stack and written up here. A whole-VM restore from PBS is the disaster path.