Phase 3: VISUAL feed and M-Files drawings in the worker
Migration 0004 (VISUAL and drawing caches, go-live ignored list, resource map, job tables, order-derived columns with provenance; work order key includes the operation sequence). The real worker on APScheduler with settings re-read each fire, pause, run now, advisory locks, job_runs and a heartbeat. The visual-feed job (five VisualKit reads, COPY and MERGE cache, suspect-batch guard, the apply in one transaction: drafts, change orders, retire-moved, work order gone, closed and cancelled observed, projects and units from order links, auto-release) and the fingerprint. The drawing-lookup job (queue, budget, download on version change, sha256 into assets, decision table, staging for rev approval, drawing-origin change orders). Clerical From VISUAL panel, flags, rev approval, hand uploads, change-order review; Admin Feed settings and resource map. Synthetic fixture generator and a fixture-driven worker in CI with a Playwright scenario. Docs updated. Verified: 3249 server tests and 13 Playwright scenarios on the docker-1 CI stack, 76 frontend tests, doctor at 0004; one feed and one drawing run against VISUAL DEV and the vault from the developer machine (counts in the changelog). Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Claude Fable 5.1
+20
-1
@@ -32,7 +32,7 @@ Both stacks run from the same image as separate compose projects.
|
||||
- `postgres`: PostgreSQL 16, named volume, `backend` network only. Its init creates the database roles from secrets.
|
||||
- `migrate`: one-shot, runs `packetstation migrate` as the migrate role.
|
||||
- `app`: `packetstation app` (uvicorn). `backend` network plus egress plus `npm_default` through the proxy overlay. Assets volume, CA bundle bind, secrets, healthcheck.
|
||||
- `worker`: `packetstation worker`. Same image. Assets volume, backup folder read-only (planned, phase 9).
|
||||
- `worker`: `packetstation worker`. Same image. Runs the VISUAL feed, the drawing lookup and housekeeping (see [FEED.md](FEED.md)). `backend` network plus egress (VISUAL, M-Files), no `npm_default`, no published port. Assets volume (fetched drawings land there), CA bundle bind, backup folder read-only (used from phase 9). Its healthcheck reads the heartbeat file.
|
||||
- All services restart unless stopped, log to stdout (json-file), and have healthchecks.
|
||||
|
||||
## Secrets
|
||||
@@ -52,6 +52,8 @@ Docker secrets, one file each, under `/etc/packetstation/<stack>/secrets/`. Owne
|
||||
|
||||
Never put a secret in the repo, the env file, a log or chat. Settings read secrets through `*_PASSWORD_FILE` paths.
|
||||
|
||||
The worker gets `postgres_app_password` (it logs in as the app role, with application name `ps-worker`), `visualkit_password`, `mfileskit_password`, `ldap_bind_password` and `seal_key`. The worker never logs a password: job errors are redacted before they are stored, and the redaction also strips the VISUAL and M-Files passwords themselves.
|
||||
|
||||
The PESCO CA chain is mounted into the containers once and referenced by `LDAP_CA_BUNDLE`, `MFILES_CA_BUNDLE` and `SMTP_CA_BUNDLE`.
|
||||
|
||||
## Env files
|
||||
@@ -66,8 +68,14 @@ Key settings:
|
||||
- `SMTP_HOST=app-03.pesco.com`, `SMTP_PORT=25`, `SMTP_STARTTLS=true`, `[email protected]`, `SMTP_CA_BUNDLE`.
|
||||
- `APP_ALIAS` per stack.
|
||||
- Identity timing and limits, all optional with these defaults: `SESSION_IDLE_MINUTES=15`, `SESSION_LOCK_MINUTES=3`, `SESSION_HIDDEN_LOCK_SECONDS=60`, `SESSION_PASSWORD_HOURS=12`, `SESSION_PIN_HOURS=10`, `ROLES_REFRESH_MINUTES=15`, `SIGNIN_ACCOUNT_LIMIT=5`, `SIGNIN_ACCOUNT_WINDOW_MINUTES=5`, `SIGNIN_ACCOUNT_COOLDOWN_MINUTES=10`, `SIGNIN_IP_LIMIT=30`, `SIGNIN_IP_WINDOW_MINUTES=5`, `PIN_MAX_FAILURES=5`, `STEPUP_MAX_FAILURES=3`, `DEVICE_COOKIE_DAYS=365`, `PIN_MIN_DIGITS=6`, `PIN_MAX_DIGITS=12`.
|
||||
- Worker sources, all optional: `FEED_SOURCE` (`visualkit`, the default, or `fixture`), `DRAWING_SOURCE` (`mfiles`, the default, or `fixture`) and `FIXTURE_DIR` (default `/fixtures`). A fixture source reads files instead of VISUAL or M-Files (see [FEED.md](FEED.md), "Fixture sources"). `STACK=prod` refuses a fixture source at start. The test and production stacks leave all three unset. Only the CI env file sets them.
|
||||
- `SITE_ID` (default `PESCO`) and `SITE_TZ` (default `America/Denver`): the VISUAL site in every call, and the time zone for shop hours and the schedule.
|
||||
- The session cookie is marked `Secure` when the request arrived over https (as reported by NPM through `X-Forwarded-Proto`). Behind the proxy that is always the case.
|
||||
|
||||
## Feed settings live in the database
|
||||
|
||||
The feed's schedule and rules are not env settings. Intervals, shop hours, the window, the go-live date, departments, the drawing recheck and budget, labor tickets, auto-release and the pause switch are one `settings` row (key `feed`), edited by an Admin on the Admin page, Feed tab. The resource map is the `visual_resource_map` table, edited on the "VISUAL resources" tab. A save notifies the worker, which applies it without a restart. A new stack starts with the defaults in [FEED.md](FEED.md), "Settings". Set the go-live date before the production feed's first run.
|
||||
|
||||
## Deploy command
|
||||
|
||||
Test stack:
|
||||
@@ -158,6 +166,17 @@ The DC runs with `cap_add: [SYS_ADMIN]`, which Samba needs to set the sysvol ACL
|
||||
|
||||
The real PESCO domain controllers are never used by CI. The test and production stacks do not run the DC.
|
||||
|
||||
### CI feed fixtures (phase 3)
|
||||
|
||||
CI never reads VISUAL or M-Files. `deploy/ci/ci.env` sets `FEED_SOURCE=fixture`, `DRAWING_SOURCE=fixture` and `FIXTURE_DIR=/fixtures`. The image does not contain `tests/`, so the e2e job generates the fixtures on the runner before the stack starts:
|
||||
|
||||
```bash
|
||||
python -m pip install pypdf==6.21.0
|
||||
python -m tests.fixtures.generator --out tests/fixtures/generated
|
||||
```
|
||||
|
||||
The generator is deterministic and writes `feed.1.json`, `feed.2.json`, `drawings.1.json`, `drawings.2.json`, the PDFs they name and `state` (`1`). `tests/fixtures/generated/` is ignored by git. `deploy/compose.ci.yaml` binds it at `/fixtures` into the worker (read-only) and the Playwright container (read-write, so the scenario can write `state` = `2` and press Run now again). `packetstation fixtures use N --dir DIR` switches the state by hand. The CI feed runs at the default intervals; the scenario does not wait for them, it presses Run now.
|
||||
|
||||
## Gitea Actions runner (phase 2)
|
||||
|
||||
`.gitea/workflows/ci.yml` runs on a Gitea Actions runner named `docker-1`, installed on docker-1 at
|
||||
|
||||
Reference in New Issue
Block a user