Compare commits
6
Commits
1cf37d94e7
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 spencerm
|
18333600b5 | ||
 spencermandClaude Opus 5.5
|
6adaf7bf25 | ||
|
spencerm
|
5f8165c833 | ||
|
spencermandClaude Opus 5.5
|
dad21db65f | ||
|
spencermandClaude Opus 5.5
|
8fb0818092 | ||
|
spencerm
|
f5390ac7cf |
@@ -6,6 +6,7 @@ All notable changes to this project are recorded here. The format follows [Keep
|
||||
|
||||
### Decided
|
||||
|
||||
- Gitea issue 57: Spencer chose the stricter archive rule (2026-11-03, open question 53). The archive copy must succeed before a packet counts as filed. A failed copy keeps the packet where it was and tells Admin. See Changed.
|
||||
- Gitea issue 9: SethD kept the drawing-origin change order for a same-rev re-release on an in-shop packet (2026-10-29), so the shop confirms the new pages. A byte-identical file is still ignored. Nothing changes in the build.
|
||||
- Gitea issue 4: SethD wants the order a packet shows to follow the shop schedule, which is Presley's Production Planner (2026-10-29). Asked on the issue on 2026-10-30: what it is, where it lives, what a row holds and how often it changes. Packet Station would only read it. The order rule stays as built until then.
|
||||
- Gitea issue 31: two follow-up questions asked on 2026-10-30. A work order deleted from VISUAL while open blocks filing for good: count a leg gone for 2 runs as closed, or let an Admin close it with a reason? And should Shipping packets also wait for the VISUAL close? Both stay as built until answered.
|
||||
@@ -23,6 +24,10 @@ All notable changes to this project are recorded here. The format follows [Keep
|
||||
- Gitea issue 20: the storage alert stays at 85 percent. Phase 9 mails it to PS-Admins.
|
||||
- Gitea issue 38: the test stack's files stay under `~/packetstation/test` on docker-1. The test env example, the compose header and the docs now give that path.
|
||||
|
||||
### Removed
|
||||
|
||||
- `scripts/Request-VsqlServerCertificate.ps1`, its Pester tests and its DEPLOY.md procedure. IT issues and binds the vsql certificate by hand. Gitea issue 5 still tracks the switch to `VISUAL_ENCRYPT=yes`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- The hand drawing, hand BOM and rev approval lines on a VISUAL draft showed the person's user id instead of their name.
|
||||
@@ -34,6 +39,7 @@ All notable changes to this project are recorded here. The format follows [Keep
|
||||
|
||||
### Changed
|
||||
|
||||
- Filing copies the packet to the archive inside the filing transaction (Gitea issue 57). If the folder check or the copy fails, nothing is filed: the packet stays in its previous stage, the person filing sees why (503 `archive_failed`), and Admin gets the critical `archive_failed` alert for that packet with who tried, the folder and the error, plus an `archive.filing_blocked` audit row. Filing it again once the archive works files it and clears the alert. The `archive-copy` worker job now only catches up filings made while the archive was off. With the archive off or no folder saved, filing is unchanged.
|
||||
- Phase 10: the content security policy is final. Styles no longer need `'unsafe-inline'`, `data:` images and `blob:` workers are gone, and `frame-src` and `object-src` are `'none'` (a PDF opened in its own tab gets `object-src 'self'` so the browser can show it). The print area's rules moved into the theme stylesheet.
|
||||
- Phase 10: backups and the restore run on the VM with `pg_dump` and `pg_restore` as the postgres user, with no `docker exec`. Same markers, 7 dumps and the 02:00 timer.
|
||||
- Phase 10: security fixes in the pinned dependencies found by the CI scan: `pypdf` 6.21.2 to 6.21.4, `pyasn1` 0.6.4 to 0.6.5, and `certifi` to 2026.10.23.
|
||||
|
||||
+1
-59
@@ -179,65 +179,7 @@ Docker's default address pool on docker-1 holds fifteen /16 bridge networks (172
|
||||
|
||||
## Known issue: vsql serves a weak self-signed certificate (found 2026-10-18)
|
||||
|
||||
`vsql.pesco.com\DEV` serves TLS (`VISUAL_ENCRYPT=trust` connects), but with `VISUAL_ENCRYPT=yes` OpenSSL in the image refuses it with "EE certificate key too weak": the server certificate's RSA key is under 2048 bits, which is what SQL Server's auto-generated self-signed certificate looks like. IT needs to issue vsql a 2048-bit, SHA-256 certificate from PESCO-PSYLOCKE-CA with `vsql.pesco.com` in the subject alternative names, install it on the DEV and production instances in SQL Server Configuration Manager and restart them. The image then needs the PESCO CA in its OpenSSL trust store for the ODBC driver (the entrypoint can concatenate the system bundle and /etc/ssl/pesco/ca.pem into /tmp and export `SSL_CERT_FILE`), and the stacks switch to `VISUAL_ENCRYPT=yes`. Tracked as Gitea issue 5.
|
||||
|
||||
### Installing the vsql certificate
|
||||
|
||||
`scripts/Request-VsqlServerCertificate.ps1` does the request, the checks and the binding. Spencer runs it on the SQL Server host, in an elevated PowerShell, as an account that may enroll from the WebServer template. Every step there is a real change, so preview first. Copy only the script to `C:\ProgramData\PacketStation\` and work from there.
|
||||
|
||||
1. Preflight. Nothing changes. Check the instance names, the service accounts, the INF and the list of steps.
|
||||
|
||||
```powershell
|
||||
cd C:\ProgramData\PacketStation
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Instances DEV -WhatIf
|
||||
```
|
||||
|
||||
2. DEV first. This enrolls, validates, grants the DEV service account read on the key, saves a snapshot and binds DEV. Production is untouched.
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Instances DEV
|
||||
```
|
||||
|
||||
If the CA holds the request as pending, the script stops with the request id. Issue it on the CA (certsrv.msc, Pending Requests), then run the `certreq -retrieve` line it printed and continue with `-CerFile <that .cer> -Instances DEV`. If validation fails, nothing is bound. The usual cause is a template that forces a Key Storage Provider.
|
||||
|
||||
3. Restart DEV and check the log. `Restart-Service -Name 'MSSQL$DEV' -Force`, then start `SQLAgent$DEV` again if it ran. Or rerun with `-Thumbprint <thumbprint> -Instances DEV -RestartInstances`, which leaves the binding alone and does both. The instance's ERRORLOG should read `The certificate [Cn = vsql.pesco.com] was successfully loaded for encryption`. If the instance does not start, roll back (step 6).
|
||||
|
||||
4. Check from docker-1. Export the certificate (`Export-Certificate -Cert Cert:\LocalMachine\My\<thumbprint> -FilePath vsql.cer`) and copy it to docker-1. OpenSSL cannot speak TDS, so `openssl s_client` does not work against SQL Server. Verify the file at the image's security level instead:
|
||||
|
||||
```bash
|
||||
openssl x509 -inform DER -in vsql.cer -out vsql.pem
|
||||
openssl x509 -in vsql.pem -noout -subject -dates -ext subjectAltName,keyUsage,extendedKeyUsage
|
||||
openssl verify -auth_level 2 -purpose sslserver -CAfile ~/packetstation/test/ca/pesco-ca.pem vsql.pem
|
||||
```
|
||||
|
||||
Then prove the live connection with the doctor and `VISUAL_ENCRYPT=yes`. The image does not yet add the PESCO CA to OpenSSL's trust store for the ODBC driver (the issue 5 follow-up), so this one-off check builds a bundle in `/tmp`. It skips the entrypoint, so the M-Files password line fails. Read the VISUAL line.
|
||||
|
||||
```bash
|
||||
docker compose -p packetstation-test --env-file ~/packetstation/test/stack.env \
|
||||
-f deploy/compose.yaml -f deploy/compose.proxy.yaml run --rm --no-deps \
|
||||
-e VISUAL_ENCRYPT=yes --entrypoint sh app -c \
|
||||
'cat /etc/ssl/certs/ca-certificates.crt /etc/ssl/pesco/ca.pem > /tmp/ca.pem && SSL_CERT_FILE=/tmp/ca.pem exec packetstation doctor'
|
||||
```
|
||||
|
||||
That the ODBC driver honours `SSL_CERT_FILE` is untested. If the VISUAL line fails with an issuer error while `openssl verify` passed, the trust step is the gap, not the server.
|
||||
|
||||
5. Production, off-hours. Production (`vsql`, `PESCOINC`) is the default instance, `MSSQLSERVER`. Bind the same certificate. VISUAL users lose their connection during the restart.
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Thumbprint <thumbprint> -Instances MSSQLSERVER -WhatIf
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Thumbprint <thumbprint> -Instances MSSQLSERVER -RestartInstances
|
||||
```
|
||||
|
||||
Check the ERRORLOG line again. The production stack's own doctor run is Spencer's, with its own approval.
|
||||
|
||||
6. Rollback. Each deployment saved `deploy-<time>.json` in `C:\ProgramData\PacketStation\vsql-cert`. `-Rollback` restores the newest one not yet rolled back. It restores each instance on its own, reads each value back and lists any failure. If a restart fails, fix the cause and run the same command again: it retries only the instances not yet restored or restarted, and starts SQL Server Agent if it ran before the failed attempt. Preview it, then run it:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Rollback -Instances DEV -WhatIf
|
||||
powershell -ExecutionPolicy Bypass -File .\Request-VsqlServerCertificate.ps1 -Rollback -Instances DEV -RestartInstances
|
||||
```
|
||||
|
||||
`-Snapshot deploy-<time>.json` restores a specific one. The rollback leaves the certificate and the key grant in place. Removing them is a manual step once nothing is bound to them.
|
||||
`vsql.pesco.com\DEV` serves TLS (`VISUAL_ENCRYPT=trust` connects), but with `VISUAL_ENCRYPT=yes` OpenSSL in the image refuses it with "EE certificate key too weak": the server certificate's RSA key is under 2048 bits, which is what SQL Server's auto-generated self-signed certificate looks like. IT needs to issue vsql a 2048-bit, SHA-256 certificate from PESCO-PSYLOCKE-CA with `vsql.pesco.com` in the subject alternative names, install it on the DEV and production instances in SQL Server Configuration Manager and restart them. The image then needs the PESCO CA in its OpenSSL trust store for the ODBC driver (the entrypoint can concatenate the system bundle and /etc/ssl/pesco/ca.pem into /tmp and export `SSL_CERT_FILE`), and the stacks switch to `VISUAL_ENCRYPT=yes`. Tracked as Gitea issue 5. IT issues and binds the certificate by hand; the repo has no script for it.
|
||||
|
||||
## Doctor
|
||||
|
||||
|
||||
+2
-2
@@ -144,9 +144,9 @@ For each rule whose digest time (site time) has passed today and that has outbox
|
||||
|
||||
## archive-copy (every 15 minutes, phase 10)
|
||||
|
||||
Gitea issue 7. Not pausable. Off when `ARCHIVE_ROOT` is unset or blank: the run does nothing (counts `enabled: false`) and FILE asks for nothing. With the root set but no folder saved on Admin, File locations, the run idles (`skipped: no archive folder saved`).
|
||||
Gitea issue 7. Not pausable. Off when `ARCHIVE_ROOT` is unset or blank: the run does nothing (counts `enabled: false`). With the root set but no folder saved on Admin, File locations, the run idles (`skipped: no archive folder saved`).
|
||||
|
||||
FILE inserts a `job_requests` row for `archive-copy` and NOTIFYs `ps_job_request` inside its own transaction when the archive is on and a folder is saved, so the copy starts right after the filing commits. The request is made in a savepoint: if it fails, the filing still commits and the next scheduled run copies the packet. A failed copy never blocks filing and never touches the packet. Saving a folder and "Retry now" also request a run.
|
||||
FILE makes its own copy inside the filing transaction and refuses to file when it fails (Gitea issue 57, open question 53), so this job does not copy new filings. It catches up the filings made while the archive was off. Those packets are already filed, so a failed copy here never touches the packet. Saving a folder and "Retry now" request a run.
|
||||
|
||||
Each run:
|
||||
|
||||
|
||||
+2
-2
@@ -121,7 +121,7 @@ Each phase ends with passing tests, docs updated (README, CHANGELOG, SECURITY.md
|
||||
- Rewrite the 38 scenarios against the new UI with synthetic fixtures; run in Gitea Actions against compose.ci.yml. Policy matrix, merge rules and packet-rule tests already run from phase 2 and 3.
|
||||
- CSP final pass, dependency scan, build-version banner and reload, DEPLOY.md with the NPM proxy host settings, SECURITY.md rewritten with the ID table, GO-LIVE.md (date, boundary, groups, tablets, URL, M: retirement as read-only archive), PAPER-FALLBACK.md, incident notes.
|
||||
- Production and test each on their own Linux VM, natively, with no Docker (Spencer, 2026-10-27, Gitea issue 37; see "Host layout"). The test VM is built first from the same steps, so the production install, backups, restore and doctor are rehearsed before go-live. The app and the worker run as systemd services from a venv built from the same `server/requirements.txt` and `server/requirements-pesco.txt` pins as the image, with PostgreSQL 16, ODBC Driver 18 and the PESCO CA installed on the VM. Settings from an env file, secrets as root-owned files read through the same `*_PASSWORD_FILE` settings, VISUAL_SERVER and VISUAL_DATABASE set explicitly; first start is Spencer's approval. NPM on docker-1 proxies `packetstation.pescoinc.biz` and `packetstation-test.pescoinc.biz` to their VMs over the LAN. Backups, the doctor and the restore are rewritten for the VMs (no `docker exec`). DEPLOY.md gets a VM section; the compose files stay for CI. The docker-1 test stack keeps running until the test VM is ready, then it is stopped.
|
||||
- Archive location (Gitea issue 7, Spencer 2026-10-27). Spencer creates an NFS share on the QNAP and mounts it on docker-1 (root, steps in DEPLOY.md), with write access for uid 10001. Compose bind-mounts it into the app and the worker at a fixed root, `ARCHIVE_ROOT` (unset turns the feature off). The Admin page gets a File locations section with one setting for now, the archive folder, as a path under `ARCHIVE_ROOT`. The server refuses a path that leaves the root (`..`, absolute paths elsewhere, symlinks out). "Check" writes a probe file, reads it back, removes it and shows the result, and saving runs the same check first. The setting change and every check are audited. `doctor` runs the check too. At filing, the sealed data book, each of the packet's documents and a manifest are written to the archive folder (SethD, 2026-10-28, issue 7). Whether a failed copy blocks the filing or is retried and raised to Admin is settled in the phase 10 design.
|
||||
- Archive location (Gitea issue 7, Spencer 2026-10-27). Spencer creates an NFS share on the QNAP and mounts it on docker-1 (root, steps in DEPLOY.md), with write access for uid 10001. Compose bind-mounts it into the app and the worker at a fixed root, `ARCHIVE_ROOT` (unset turns the feature off). The Admin page gets a File locations section with one setting for now, the archive folder, as a path under `ARCHIVE_ROOT`. The server refuses a path that leaves the root (`..`, absolute paths elsewhere, symlinks out). "Check" writes a probe file, reads it back, removes it and shows the result, and saving runs the same check first. The setting change and every check are audited. `doctor` runs the check too. At filing, the sealed data book, each of the packet's documents and a manifest are written to the archive folder (SethD, 2026-10-28, issue 7). A failed copy blocks the filing and is raised to Admin (Spencer, 2026-11-03, Gitea issue 57; open question 53).
|
||||
- Closes: O3, O4; S4 is IT's pentest, deferred.
|
||||
- Done 2026-10-30. The bench maps all 38 old scenarios (tests/e2e/BENCH.md; trel is open question 54) and runs 111 Playwright tests, each failing on a CSP violation. The CSP dropped `'unsafe-inline'`. pip-audit found pypdf, pyasn1 and certifi to bump. The archive copy, the version banner, the VM kit in `deploy/vm/` with VM backups and restore, and the go-live, paper-fallback and security docs are built. The VMs, the NFS share, the timers and the go-live date are Spencer's and Seth's (open question 53 records the archive copy policy).
|
||||
|
||||
@@ -292,7 +292,7 @@ From phase 9 (2026-10-21), for Seth or Spencer to veto:
|
||||
50. The profile CSV `stations` cell: default the cell is the person's whole list of station limits across departments and replaces every limit they had; `-` clears them all; an empty cell leaves them. Phase 9 (2026-10-21) applied the default.
|
||||
51. A direct welder assign: default it does not check `profile_active`, so an Admin's inactive switch hides a person from the picker but a typed assign still works. Phase 9 (2026-10-21) applied the default; say if an inactive profile should be refused there too.
|
||||
52. A checklist item added while a stage is open: default it applies to that open stage too, because completeness reads the live template, so the item must be ticked before submit. Phase 9 (2026-10-21) applied the default. Replaced by change C8: a template change applies to new packets only. Built on 2026-10-30: a stage freezes its list at START, so a change reaches only stages not yet started.
|
||||
53. A failed archive copy at filing (phase 10, Gitea issue 7): default it never blocks the filing. The FILE transaction asks the worker for an `archive-copy` run; the job retries a failed copy after 5, 15 and 60 minutes, then hourly, and raises the `archive_failed` condition to Admin after 3 failed tries or a failed folder check. Admin, Storage, File locations shows the failures and has "Retry now". Turning the archive on later also copies the earlier filings. Seth or Spencer may ask for the stricter rule (the copy must succeed before the packet counts as filed) instead.
|
||||
53. A failed archive copy at filing (phase 10, Gitea issues 7 and 57). Answered by Spencer on 2026-11-03 (issue 57): the copy must succeed before the packet counts as filed. FILE checks the folder and copies the packet inside the filing transaction. If the check or the copy fails, the whole filing rolls back, so the packet stays in its previous stage and can be filed again. The person filing gets 503 `archive_failed` with the reason. In a transaction of its own the server writes an `archive.filing_blocked` audit row and raises the `archive_failed` condition to Admin (subject `filing:<packet id>`, critical) with the packet, who tried, the archive folder and the error. A retry that fails again only counts on the open condition, and the next successful filing of that packet clears it. With `ARCHIVE_ROOT` unset or no folder saved, the archive is not in use and filing goes ahead. The worker job still copies the filings made while the archive was off, with its retries and its own `archive_failed` condition (subject `archive`). The earlier default (never block, retry in the worker) is replaced.
|
||||
54. The old Clerical "Released" section (every released packet by place, with overdue and 7+ day counts; old bench scenario trel) is not in the new build and not on the Drop list (phase 10, tests/e2e/BENCH.md). Default: not built; the boards and the find bar's Mine, Board and List views show where each packet is. Seth decides whether it is needed for parity.
|
||||
|
||||
Changes decided by SethD on 2026-10-26. Each replaces a default above. All five were built on 2026-10-30 (branch `seth-changes`):
|
||||
|
||||
+3
-3
@@ -195,10 +195,10 @@ The archive closes D6 (Gitea issue 7). It is a copy of every filed packet on the
|
||||
- An Admin picks the archive folder in the File locations section of the Admin page (setting `files.archive_dir`, versioned with compare and set like the security settings). The folder is a path under `ARCHIVE_ROOT`. The server refuses an empty path, `..`, an absolute path outside the root, a NUL, a path whose real location is outside the real root (a symlink out), and a folder that does not exist. It never creates folders.
|
||||
- "Check" writes a probe file in the folder, syncs it to disk, reads it back, compares it and deletes it. It shows the result, the resolved path and the free space. A save runs the same check first and is refused when it fails. `doctor` runs the check on the saved folder.
|
||||
- Every setting change (old and new value), every check (passed or not) and every copy (done or failed) writes an audit row. Admin writes need a password session and pass the Origin check, as every Admin write does.
|
||||
- At filing, the worker copies the sealed data book, every document of the packet and a `manifest.json` into `<project>/<unit>/<data book name>-seal<seal id>/`. The manifest names the packet, project, unit, seal id, seal hash, key id, who sealed and filed it and when, the build, and for every file its name, kind, sha256, size and source asset. Anyone can check the copy against the manifest without Packet Station.
|
||||
- At filing, FILE copies the sealed data book, every document of the packet and a `manifest.json` into `<project>/<unit>/<data book name>-seal<seal id>/`. The manifest names the packet, project, unit, seal id, seal hash, key id, who sealed and filed it and when, the build, and for every file its name, kind, sha256, size and source asset. Anyone can check the copy against the manifest without Packet Station.
|
||||
- Each file is written under a temporary name, synced and renamed. A different file is never overwritten; an identical file counts as done. Nothing in the archive is ever deleted by Packet Station. A refile after a reopen gets a new seal and a new folder, so both copies stay.
|
||||
- A failed copy never blocks filing (decided in the phase 10 design; recorded as an open question in [PLAN.md](PLAN.md)). The worker job `archive-copy` runs every 15 minutes and right after each filing. It retries a failed copy after 5, 15 and 60 minutes, then hourly. When a copy has failed 3 times, or the folder check fails, it raises the Admin alert `archive_failed`, cleared when nothing is failed. "Retry now" on the File locations section asks for a run at once.
|
||||
- When the archive is turned on later, the job also copies every earlier filing that has no copy yet.
|
||||
- The copy must succeed before the packet counts as filed (Spencer, 2026-11-03, Gitea issue 57; open question 53 in [PLAN.md](PLAN.md)). FILE checks the folder and copies inside the filing transaction. A failed check or copy rolls the filing back, so the packet stays in its previous stage. The server then writes an `archive.filing_blocked` audit row and raises the critical Admin alert `archive_failed` for that packet, with who tried, the folder and the error. Filing it again once the archive works clears the alert.
|
||||
- When the archive is turned on later, the worker job `archive-copy` copies every earlier filing that has no copy yet. It runs every 15 minutes, retries a failed copy after 5, 15 and 60 minutes, then hourly, and raises `archive_failed` when a copy has failed 3 times or the folder check fails. "Retry now" on the File locations section asks for a run at once.
|
||||
- The archive is a plain copy outside the app's access control. The files are not encrypted by the app (D1 is deferred). Who can read or change the share, the NFS export rule (the production VM only), snapshots and the share's own backup are IT's.
|
||||
|
||||
### Warehouse rules (phase 4)
|
||||
|
||||
@@ -7,22 +7,3 @@ The nightly `pg_dump` backup with its systemd timer, and the restore script, for
|
||||
## Export-PescoCaBundle.ps1
|
||||
|
||||
Run on a domain-joined PC. Exports every unexpired PESCO CA certificate from the Windows stores to one PEM bundle and copies it to docker-1 with scp (default target `~/packetstation/test/ca/pesco-ca.pem`). Use `-Remote` for another path and `-NoCopy` to only write the file. It warns on SHA-1 signed certificates; see docs/DEPLOY.md.
|
||||
|
||||
## Request-VsqlServerCertificate.ps1
|
||||
|
||||
Run elevated on the SQL Server host behind vsql, in Windows PowerShell 5.1 or PowerShell 7. Requests a TLS certificate from PESCO-PSYLOCKE-CA with certreq the way SQL Server requires it (legacy Microsoft RSA SChannel Cryptographic Provider, KeySpec AT_KEYEXCHANGE, RSA 2048, SHA-256, Server Authentication, key usage 0xA0, SANs vsql.pesco.com, vsql and the host FQDN and short name). It installs it and validates it before anything is bound. The validation checks the key provider and KeySpec from the certificate's own key property, the key size, the signature, the dates, the EKU, and the SAN DNS names as whole names. It grants only the selected instances' service accounts read on the private key, then binds the thumbprint and reads it back.
|
||||
|
||||
- `-WhatIf` previews every step and changes nothing. `-Confirm` asks before each change. Declining enrollment or a key grant cancels the run with nothing bound (exit code 2).
|
||||
- `-CerFile` continues a request that was pending. `-Thumbprint` binds a certificate already installed. `-Instances` limits the run (`DEV`, `MSSQLSERVER`). `-RestartInstances` restarts them and starts SQL Server Agent again if it ran. The running dependents are written to `dependents-<service>.json` before the stop and the file is removed once they run again, so after a failed restart the next restart of that service starts them too.
|
||||
- Each deployment writes a new `deploy-<time>.json` in `C:\ProgramData\PacketStation\vsql-cert` with the values it replaced. Instances already bound to the thumbprint are left alone and get no snapshot, so a rerun never hides the useful one. A pending request writes `request-<time>.json` with its request id. A registry value that cannot be read stops the run; only a value confirmed absent is saved as absent.
|
||||
- `-Rollback` restores the newest snapshot not yet rolled back, each instance on its own, verified by reading back, and records it in `rollback-<time>.json` with which restarts succeeded. A snapshot stays open while an instance is unrestored or its restart failed, so `-Rollback -RestartInstances` again retries exactly those. `-Snapshot` picks another. Failures are listed and the run exits with an error.
|
||||
- Instances are read from the value names of `Instance Names\SQL`, so a name starting with `PS` (for example `PSREPORTING`) is found.
|
||||
|
||||
Tests: `scripts/tests/Request-VsqlServerCertificate.Tests.ps1` (Pester 5) fakes the host (instances, services, registry, machine store, certreq, icacls) and uses real throwaway certificates in `Cert:\CurrentUser\My`. Run it on both PowerShell versions:
|
||||
|
||||
```powershell
|
||||
Install-Module Pester -RequiredVersion 5.7.1 -Scope CurrentUser -SkipPublisherCheck
|
||||
Invoke-Pester -Path scripts ests -Output Detailed
|
||||
```
|
||||
|
||||
The run procedure is in docs/DEPLOY.md, under the vsql known issue. Gitea issue 5 tracks the switch to `VISUAL_ENCRYPT=yes`.
|
||||
|
||||
@@ -1,795 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Requests a SQL Server TLS certificate for vsql from PESCO-PSYLOCKE-CA and binds it to the instances.
|
||||
|
||||
.DESCRIPTION
|
||||
Run on the SQL Server host (the machine behind vsql.pesco.com) in an elevated PowerShell, as a
|
||||
domain account that may enroll from the chosen template. Works on Windows PowerShell 5.1 and
|
||||
PowerShell 7. Steps:
|
||||
|
||||
1. Resolve the SQL Server instances to configure and their service accounts. Unknown names
|
||||
in -Instances are an error, nothing is touched.
|
||||
2. Write a certreq INF with what SQL Server requires (Microsoft certificate requirements):
|
||||
a legacy CryptoAPI provider, Microsoft RSA SChannel Cryptographic Provider, with KeySpec
|
||||
AT_KEYEXCHANGE; RSA 2048; SHA-256; Server Authentication EKU; key usage digital signature
|
||||
and key encipherment; subject CN=vsql.pesco.com; subject alternative names vsql.pesco.com,
|
||||
vsql, the host FQDN and short name; machine store; exportable.
|
||||
3. certreq -new, -submit and -accept, each confirmed through ShouldProcess. A pending request
|
||||
is saved in its own request-<time>.json; issue it on the CA, retrieve it, then rerun with
|
||||
-CerFile (or -Thumbprint once installed).
|
||||
4. Validate the installed certificate before anything is bound: a private key from a CSP
|
||||
(provider type 12) with KeySpec AT_KEYEXCHANGE, read from the certificate's own key
|
||||
provider property; 2048 bits or more; SHA-256 family signature; valid for 30 more days;
|
||||
Server Authentication EKU; digital signature and key encipherment if key usage is set;
|
||||
every required DNS name, decoded from the SAN extension and compared as whole names.
|
||||
5. Grant the selected instances' service accounts read on the private key. A declined or
|
||||
failed grant cancels the run before any instance is changed.
|
||||
6. Instances already bound to this thumbprint are left alone. For the others, save their
|
||||
current Certificate values in a new deploy-<time>.json (never overwritten), then bind the
|
||||
thumbprint (the registry value SQL Server Configuration Manager sets) and read it back.
|
||||
ForceEncryption is left as it is. A failure restores every instance changed in this run,
|
||||
each on its own, verified.
|
||||
7. Print the restart commands, or restart with -RestartInstances (a short VISUAL outage).
|
||||
Services that depend on the instance (SQL Server Agent) are started again if they ran.
|
||||
|
||||
-WhatIf previews every step without changing anything. -Confirm asks before each change.
|
||||
-Rollback restores a saved deployment snapshot.
|
||||
|
||||
.PARAMETER CertificateAuthority
|
||||
CA config string host\name. Default DC-02.PESCO.COM\PESCO-PSYLOCKE-CA.
|
||||
|
||||
.PARAMETER Template
|
||||
Certificate template name. Default WebServer. The template must allow Server Authentication
|
||||
and must not force a Key Storage Provider.
|
||||
|
||||
.PARAMETER DnsNames
|
||||
Names the certificate must carry. The first is the subject CN. The host FQDN and short name
|
||||
are always added.
|
||||
|
||||
.PARAMETER Instances
|
||||
Instance names to configure (MSSQLSERVER for the default instance, DEV for vsql\DEV).
|
||||
Default: every instance on this host.
|
||||
|
||||
.PARAMETER CerFile
|
||||
Continue from an issued certificate file (after a pending request was issued and retrieved).
|
||||
|
||||
.PARAMETER Thumbprint
|
||||
Continue from a certificate already in the machine store; skips the request.
|
||||
|
||||
.PARAMETER RestartInstances
|
||||
Restart each configured instance so the certificate (or the rollback) takes effect.
|
||||
|
||||
.PARAMETER Rollback
|
||||
Restore the Certificate values saved by a deployment snapshot and stop. Each instance is
|
||||
restored on its own and read back; only verified instances count as restored. The default
|
||||
snapshot is the newest one with instances not yet restored (a rollback-<time>.json records
|
||||
what was restored), so each -Rollback steps back one deployment. Run it with -WhatIf first
|
||||
to see which one it picks.
|
||||
|
||||
.PARAMETER Snapshot
|
||||
With -Rollback: the snapshot file (name or path) to restore instead of the default.
|
||||
|
||||
.EXAMPLE
|
||||
.\Request-VsqlServerCertificate.ps1 -Instances DEV -WhatIf
|
||||
.EXAMPLE
|
||||
.\Request-VsqlServerCertificate.ps1 -Instances DEV
|
||||
.EXAMPLE
|
||||
.\Request-VsqlServerCertificate.ps1 -CerFile C:\ProgramData\PacketStation\vsql-cert\vsql-20261018-090000-000.cer -Instances DEV -RestartInstances
|
||||
.EXAMPLE
|
||||
.\Request-VsqlServerCertificate.ps1 -Rollback -WhatIf
|
||||
#>
|
||||
[CmdletBinding(SupportsShouldProcess, DefaultParameterSetName = 'Request')]
|
||||
param(
|
||||
[string]$CertificateAuthority = 'DC-02.PESCO.COM\PESCO-PSYLOCKE-CA',
|
||||
[string]$Template = 'WebServer',
|
||||
[string[]]$DnsNames = @('vsql.pesco.com', 'vsql'),
|
||||
[string[]]$Instances,
|
||||
[Parameter(ParameterSetName = 'Cer', Mandatory)][string]$CerFile,
|
||||
[Parameter(ParameterSetName = 'Thumb', Mandatory)][string]$Thumbprint,
|
||||
[switch]$RestartInstances,
|
||||
[Parameter(ParameterSetName = 'Rollback', Mandatory)][switch]$Rollback,
|
||||
[Parameter(ParameterSetName = 'Rollback')][string]$Snapshot
|
||||
)
|
||||
|
||||
$script:AcceptedSignatureOids = @{
|
||||
'1.2.840.113549.1.1.11' = 'sha256RSA'
|
||||
'1.2.840.113549.1.1.12' = 'sha384RSA'
|
||||
'1.2.840.113549.1.1.13' = 'sha512RSA'
|
||||
}
|
||||
$script:SchannelProvider = 'Microsoft RSA SChannel Cryptographic Provider'
|
||||
|
||||
# --- helpers that touch nothing --------------------------------------------------------------
|
||||
|
||||
function Test-SameValue($A, $B) {
|
||||
# Registry values: $null means absent, '' means present and empty. Thumbprints ignore case.
|
||||
# Untyped on purpose: a [string] parameter turns $null into ''.
|
||||
if ($null -eq $A -or $null -eq $B) { return ($null -eq $A -and $null -eq $B) }
|
||||
return [string]::Equals([string]$A, [string]$B, [StringComparison]::OrdinalIgnoreCase)
|
||||
}
|
||||
|
||||
function Get-UniqueNames([string[]]$Names) {
|
||||
$seen = New-Object 'System.Collections.Generic.HashSet[string]' ([StringComparer]::OrdinalIgnoreCase)
|
||||
$out = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($n in $Names) { if ($n -and $seen.Add($n)) { $out.Add($n) } }
|
||||
return $out.ToArray()
|
||||
}
|
||||
|
||||
function Get-SqlServiceName([string]$Instance) {
|
||||
if ($Instance -eq 'MSSQLSERVER') { return 'MSSQLSERVER' }
|
||||
return 'MSSQL$' + $Instance
|
||||
}
|
||||
|
||||
function Get-ServiceFilter([string]$ServiceName) {
|
||||
# WQL string literal. `$` needs no escape; a quote or backslash never occurs in a service name.
|
||||
if ($ServiceName -match "['\\]") { throw "Unexpected character in service name '$ServiceName'." }
|
||||
return "Name='$ServiceName'"
|
||||
}
|
||||
|
||||
function Read-DerLength([byte[]]$Bytes, [ref]$Position) {
|
||||
if ($Position.Value -ge $Bytes.Length) { throw 'SAN extension is truncated.' }
|
||||
$first = [int]$Bytes[$Position.Value]; $Position.Value++
|
||||
if ($first -lt 0x80) { return $first }
|
||||
$count = $first -band 0x7F
|
||||
if ($count -lt 1 -or $count -gt 3) { throw 'SAN extension has an unsupported length encoding.' }
|
||||
$length = 0
|
||||
for ($i = 0; $i -lt $count; $i++) {
|
||||
if ($Position.Value -ge $Bytes.Length) { throw 'SAN extension is truncated.' }
|
||||
$length = ($length * 256) + [int]$Bytes[$Position.Value]; $Position.Value++
|
||||
}
|
||||
return $length
|
||||
}
|
||||
|
||||
function Get-SanDnsNames([byte[]]$RawData) {
|
||||
# SubjectAltName ::= SEQUENCE OF GeneralName; dNSName is [2] IMPLICIT IA5String (tag 0x82).
|
||||
# A plain DER walk, the same on Windows PowerShell 5.1 and PowerShell 7, independent of
|
||||
# display formatting and language.
|
||||
if (-not $RawData -or $RawData.Length -lt 2 -or $RawData[0] -ne 0x30) { throw 'SAN extension is not a DER SEQUENCE.' }
|
||||
$pos = 1
|
||||
$length = Read-DerLength $RawData ([ref]$pos)
|
||||
$end = $pos + $length
|
||||
if ($end -ne $RawData.Length) { throw 'SAN extension length does not match its content.' }
|
||||
$names = New-Object System.Collections.Generic.List[string]
|
||||
while ($pos -lt $end) {
|
||||
$tag = [int]$RawData[$pos]; $pos++
|
||||
if (($tag -band 0x1F) -eq 0x1F) { throw 'SAN extension uses a multi-byte tag.' }
|
||||
$len = Read-DerLength $RawData ([ref]$pos)
|
||||
if ($pos + $len -gt $end) { throw 'SAN extension is truncated.' }
|
||||
if ($tag -eq 0x82) { $names.Add([System.Text.Encoding]::ASCII.GetString($RawData, $pos, $len)) }
|
||||
$pos += $len
|
||||
}
|
||||
return $names.ToArray()
|
||||
}
|
||||
|
||||
function Get-CertificateKeyInfo([System.Security.Cryptography.X509Certificates.X509Certificate2]$Certificate) {
|
||||
# The certificate's CERT_KEY_PROV_INFO property, which is what SQL Server uses to find the key.
|
||||
# $cert.PrivateKey is not usable for this: PowerShell 7 (.NET 5 and later) returns RSACng
|
||||
# even for a CSP key, Windows PowerShell 5.1 returns RSACryptoServiceProvider.
|
||||
if (-not ('PacketStation.KeyProvInfo' -as [type])) {
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
namespace PacketStation {
|
||||
public static class KeyProvInfo {
|
||||
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
|
||||
struct CRYPT_KEY_PROV_INFO {
|
||||
public string pwszContainerName;
|
||||
public string pwszProvName;
|
||||
public uint dwProvType;
|
||||
public uint dwFlags;
|
||||
public uint cProvParam;
|
||||
public IntPtr rgProvParam;
|
||||
public uint dwKeySpec;
|
||||
}
|
||||
[DllImport("crypt32.dll", SetLastError = true)]
|
||||
static extern bool CertGetCertificateContextProperty(IntPtr pCertContext, uint dwPropId, IntPtr pvData, ref uint pcbData);
|
||||
public static object[] Get(IntPtr certContext) {
|
||||
uint size = 0;
|
||||
if (!CertGetCertificateContextProperty(certContext, 2, IntPtr.Zero, ref size)) return null;
|
||||
IntPtr buffer = Marshal.AllocHGlobal((int)size);
|
||||
try {
|
||||
if (!CertGetCertificateContextProperty(certContext, 2, buffer, ref size)) return null;
|
||||
CRYPT_KEY_PROV_INFO info = (CRYPT_KEY_PROV_INFO)Marshal.PtrToStructure(buffer, typeof(CRYPT_KEY_PROV_INFO));
|
||||
return new object[] { info.pwszContainerName, info.pwszProvName, (int)info.dwProvType, (int)info.dwFlags, (int)info.dwKeySpec };
|
||||
} finally { Marshal.FreeHGlobal(buffer); }
|
||||
}
|
||||
}
|
||||
}
|
||||
'@
|
||||
}
|
||||
$raw = [PacketStation.KeyProvInfo]::Get($Certificate.Handle)
|
||||
if (-not $raw) { return $null }
|
||||
$info = [pscustomobject]@{
|
||||
Container = [string]$raw[0]
|
||||
Provider = [string]$raw[1]
|
||||
ProviderType = [int]$raw[2]
|
||||
Machine = (([int]$raw[3]) -band 0x20) -ne 0 # CRYPT_MACHINE_KEYSET
|
||||
KeySpec = [int]$raw[4]
|
||||
UniqueName = $null
|
||||
Error = $null
|
||||
}
|
||||
if ($info.ProviderType -ne 0) {
|
||||
try {
|
||||
$cp = New-Object System.Security.Cryptography.CspParameters $info.ProviderType, $info.Provider, $info.Container
|
||||
$cp.KeyNumber = $info.KeySpec
|
||||
$cp.Flags = [System.Security.Cryptography.CspProviderFlags]::UseExistingKey
|
||||
if ($info.Machine) { $cp.Flags = $cp.Flags -bor [System.Security.Cryptography.CspProviderFlags]::UseMachineKeyStore }
|
||||
$rsa = New-Object System.Security.Cryptography.RSACryptoServiceProvider $cp
|
||||
try { $info.UniqueName = $rsa.CspKeyContainerInfo.UniqueKeyContainerName } finally { $rsa.Dispose() }
|
||||
}
|
||||
catch { $info.Error = $_.Exception.Message }
|
||||
}
|
||||
return $info
|
||||
}
|
||||
|
||||
function Test-ServerCertificate {
|
||||
param(
|
||||
[System.Security.Cryptography.X509Certificates.X509Certificate2]$Certificate,
|
||||
[string[]]$RequiredNames,
|
||||
[datetime]$Now = (Get-Date),
|
||||
[object]$KeyInfo = $(if ($Certificate.HasPrivateKey) { Get-CertificateKeyInfo $Certificate })
|
||||
)
|
||||
$problems = New-Object System.Collections.Generic.List[string]
|
||||
if (-not $Certificate.HasPrivateKey) { $problems.Add('no private key') }
|
||||
elseif (-not $KeyInfo) { $problems.Add('no key provider information on the certificate') }
|
||||
elseif ($KeyInfo.ProviderType -eq 0) { $problems.Add("the private key is in a Key Storage Provider ($($KeyInfo.Provider)); SQL Server needs a CryptoAPI (CSP) key") }
|
||||
else {
|
||||
if ($KeyInfo.ProviderType -ne 12) { $problems.Add("provider type $($KeyInfo.ProviderType) ($($KeyInfo.Provider)); expected 12, $script:SchannelProvider") }
|
||||
if ($KeyInfo.KeySpec -ne 1) { $problems.Add("KeySpec is $($KeyInfo.KeySpec); SQL Server needs AT_KEYEXCHANGE (1)") }
|
||||
if ($KeyInfo.Error) { $problems.Add("the private key container cannot be opened: $($KeyInfo.Error)") }
|
||||
}
|
||||
$rsaPublic = [System.Security.Cryptography.X509Certificates.RSACertificateExtensions]::GetRSAPublicKey($Certificate)
|
||||
$keySize = if ($rsaPublic) { $rsaPublic.KeySize } else { 0 }
|
||||
if (-not $rsaPublic) { $problems.Add('the key is not RSA') }
|
||||
elseif ($keySize -lt 2048) { $problems.Add("key size $keySize is below 2048") }
|
||||
$sigOid = $Certificate.SignatureAlgorithm.Value
|
||||
$sig = if ($script:AcceptedSignatureOids.ContainsKey($sigOid)) { $script:AcceptedSignatureOids[$sigOid] } else { "$($Certificate.SignatureAlgorithm.FriendlyName) ($sigOid)" }
|
||||
if (-not $script:AcceptedSignatureOids.ContainsKey($sigOid)) { $problems.Add("signature algorithm $sig is not SHA-256, SHA-384 or SHA-512 with RSA") }
|
||||
if ($Certificate.NotBefore -gt $Now -or $Certificate.NotAfter -lt $Now) { $problems.Add("not valid now (valid $($Certificate.NotBefore.ToString('yyyy-MM-dd')) to $($Certificate.NotAfter.ToString('yyyy-MM-dd')))") }
|
||||
elseif ($Certificate.NotAfter -lt $Now.AddDays(30)) { $problems.Add("expires $($Certificate.NotAfter.ToString('yyyy-MM-dd')), within 30 days") }
|
||||
$ekuExt = $Certificate.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.37' } | Select-Object -First 1
|
||||
$ekus = @()
|
||||
if ($ekuExt) { $ekus = @((New-Object System.Security.Cryptography.X509Certificates.X509EnhancedKeyUsageExtension $ekuExt, $ekuExt.Critical).EnhancedKeyUsages | ForEach-Object { $_.Value }) }
|
||||
if ($ekus -notcontains '1.3.6.1.5.5.7.3.1') { $problems.Add('Server Authentication EKU missing') }
|
||||
$kuExt = $Certificate.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.15' } | Select-Object -First 1
|
||||
if ($kuExt) {
|
||||
$ku = (New-Object System.Security.Cryptography.X509Certificates.X509KeyUsageExtension $kuExt, $kuExt.Critical).KeyUsages
|
||||
$flags = [System.Security.Cryptography.X509Certificates.X509KeyUsageFlags]
|
||||
if (-not ($ku -band $flags::KeyEncipherment)) { $problems.Add('key usage lacks key encipherment') }
|
||||
if (-not ($ku -band $flags::DigitalSignature)) { $problems.Add('key usage lacks digital signature') }
|
||||
}
|
||||
$dns = @()
|
||||
$sanExt = $Certificate.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.17' } | Select-Object -First 1
|
||||
if (-not $sanExt) { $problems.Add('no subject alternative name extension') }
|
||||
else {
|
||||
try { $dns = @(Get-SanDnsNames $sanExt.RawData) } catch { $problems.Add("subject alternative names unreadable: $($_.Exception.Message)") }
|
||||
}
|
||||
foreach ($n in $RequiredNames) {
|
||||
$hit = @($dns | Where-Object { [string]::Equals($_, $n, [StringComparison]::OrdinalIgnoreCase) })
|
||||
if (-not $hit) { $problems.Add("subject alternative name $n missing") }
|
||||
}
|
||||
return [pscustomobject]@{ Problems = $problems; KeyInfo = $KeyInfo; KeySize = $keySize; Signature = $sig; DnsNames = $dns }
|
||||
}
|
||||
|
||||
function New-CertRequestInf([string]$Primary, [string[]]$Names, [string]$Template, [string]$Stamp) {
|
||||
$sanLines = ($Names | ForEach-Object { "_continue_ = `"dns=$_&`"" }) -join "`r`n"
|
||||
return @"
|
||||
[Version]
|
||||
Signature = "`$Windows NT`$"
|
||||
|
||||
[NewRequest]
|
||||
Subject = "CN=$Primary"
|
||||
KeyLength = 2048
|
||||
KeyAlgorithm = RSA
|
||||
HashAlgorithm = SHA256
|
||||
; SQL Server needs a legacy CryptoAPI provider and an exchange key (KeySpec = AT_KEYEXCHANGE).
|
||||
ProviderName = "$script:SchannelProvider"
|
||||
ProviderType = 12
|
||||
KeySpec = 1
|
||||
; digital signature + key encipherment
|
||||
KeyUsage = 0xA0
|
||||
Exportable = TRUE
|
||||
MachineKeySet = TRUE
|
||||
RequestType = PKCS10
|
||||
FriendlyName = "SQL Server TLS $Primary ($Stamp)"
|
||||
|
||||
[EnhancedKeyUsageExtension]
|
||||
OID = 1.3.6.1.5.5.7.3.1
|
||||
|
||||
[Extensions]
|
||||
2.5.29.17 = "{text}"
|
||||
$sanLines
|
||||
|
||||
[RequestAttributes]
|
||||
CertificateTemplate = $Template
|
||||
"@
|
||||
}
|
||||
|
||||
function Invoke-NativeCommand([string]$FilePath, [string[]]$Arguments) {
|
||||
# Windows PowerShell 5.1 turns redirected stderr into a terminating error under 'Stop'.
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$out = @(& $FilePath @Arguments 2>&1 | ForEach-Object { "$_" })
|
||||
return [pscustomobject]@{ ExitCode = $LASTEXITCODE; Output = $out }
|
||||
}
|
||||
|
||||
function Resolve-AccountSid([string]$Account) {
|
||||
switch -Regex ($Account) {
|
||||
'^(LocalSystem|NT AUTHORITY\\SYSTEM)$' { return 'S-1-5-18' }
|
||||
'^NT AUTHORITY\\(LocalService|LOCAL SERVICE)$' { return 'S-1-5-19' }
|
||||
'^NT AUTHORITY\\(NetworkService|NETWORK SERVICE)$' { return 'S-1-5-20' }
|
||||
}
|
||||
$name = if ($Account -like '.\*') { "$env:COMPUTERNAME\$($Account.Substring(2))" } else { $Account }
|
||||
return (New-Object System.Security.Principal.NTAccount $name).Translate([System.Security.Principal.SecurityIdentifier]).Value
|
||||
}
|
||||
|
||||
# --- the host, as replaceable operations ------------------------------------------------------
|
||||
|
||||
function Get-VsqlDefaultProvider {
|
||||
@{
|
||||
WorkDir = Join-Path $env:ProgramData 'PacketStation\vsql-cert'
|
||||
InstanceRoot = 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server'
|
||||
IsAdmin = { ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) }
|
||||
GetInstanceMap = {
|
||||
# The key's own value names. Get-ItemProperty mixes in PSPath and friends, and
|
||||
# filtering those by 'PS*' would drop a real instance such as PSREPORTING.
|
||||
param($Root)
|
||||
$path = "$Root\Instance Names\SQL"
|
||||
$map = [ordered]@{}
|
||||
if (Test-Path -LiteralPath $path) {
|
||||
$key = Get-Item -LiteralPath $path -ErrorAction Stop
|
||||
foreach ($n in $key.GetValueNames()) { if ($n) { $map[$n] = [string]$key.GetValue($n) } }
|
||||
}
|
||||
$map
|
||||
}
|
||||
GetService = { param($Name) Get-CimInstance Win32_Service -Filter (Get-ServiceFilter $Name) -ErrorAction Stop }
|
||||
TestRegKey = { param($Key) Test-Path -LiteralPath $Key }
|
||||
GetRegValue = {
|
||||
# $null only for a value confirmed absent. A missing key or a read error throws, so it
|
||||
# is never saved in a snapshot, or verified after a restore, as "absent".
|
||||
param($Key, $Name)
|
||||
$item = Get-Item -LiteralPath $Key -ErrorAction Stop
|
||||
if (@($item.GetValueNames()) -notcontains $Name) { return $null }
|
||||
[string]$item.GetValue($Name, $null, [Microsoft.Win32.RegistryValueOptions]::DoNotExpandEnvironmentNames)
|
||||
}
|
||||
SetRegValue = { param($Key, $Name, $Value) Set-ItemProperty -LiteralPath $Key -Name $Name -Value $Value -ErrorAction Stop -Confirm:$false }
|
||||
RemoveRegValue = { param($Key, $Name) Remove-ItemProperty -LiteralPath $Key -Name $Name -ErrorAction Stop -Confirm:$false }
|
||||
GetHostNames = { @([System.Net.Dns]::GetHostEntry($env:COMPUTERNAME).HostName, $env:COMPUTERNAME) }
|
||||
GetMachineCert = { param($Thumb) Get-ChildItem Cert:\LocalMachine\My | Where-Object { $_.Thumbprint -eq $Thumb } | Select-Object -First 1 }
|
||||
Certreq = { param([string[]]$Arguments) Invoke-NativeCommand 'certreq.exe' $Arguments }
|
||||
Icacls = { param([string[]]$Arguments) Invoke-NativeCommand 'icacls.exe' $Arguments }
|
||||
ResolveSid = { param($Account) Resolve-AccountSid $Account }
|
||||
GetKeyInfo = { param($Cert) Get-CertificateKeyInfo $Cert }
|
||||
KeyFilePath = { param($KeyInfo) Join-Path $env:ProgramData "Microsoft\Crypto\RSA\MachineKeys\$($KeyInfo.UniqueName)" }
|
||||
TestFile = { param($Path) Test-Path -LiteralPath $Path -PathType Leaf }
|
||||
GetRunningDependents = { param($Name) @((Get-Service -Name $Name -ErrorAction Stop).DependentServices | Where-Object { $_.Status -eq 'Running' } | ForEach-Object { $_.Name }) }
|
||||
RestartService = { param($Name) Restart-Service -Name $Name -Force -ErrorAction Stop -Confirm:$false }
|
||||
StartService = { param($Name) Start-Service -Name $Name -ErrorAction Stop -Confirm:$false }
|
||||
Now = { Get-Date }
|
||||
Log = { param($Message) Write-Host $Message }
|
||||
ShouldProcess = $null # $null: the cmdlet's own ShouldProcess (-WhatIf, -Confirm)
|
||||
}
|
||||
}
|
||||
|
||||
# --- steps ------------------------------------------------------------------------------------
|
||||
|
||||
function Get-SqlInstances($P, [string[]]$Wanted) {
|
||||
$map = & $P.GetInstanceMap $P.InstanceRoot
|
||||
if (-not $map -or $map.Count -eq 0) { throw 'No SQL Server Database Engine instance is registered on this host. Is this the vsql machine?' }
|
||||
$found = @($map.Keys)
|
||||
if ($Wanted) {
|
||||
$unknown = @($Wanted | Where-Object { $found -notcontains $_ })
|
||||
if ($unknown) { throw "Unknown instance(s): $($unknown -join ', '). Found on this host: $($found -join ', '). Nothing was changed." }
|
||||
$found = @($found | Where-Object { $Wanted -contains $_ })
|
||||
}
|
||||
$targets = foreach ($name in $found) {
|
||||
$svc = Get-SqlServiceName $name
|
||||
$service = & $P.GetService $svc
|
||||
if (-not $service) { throw "Service $svc for instance $name was not found." }
|
||||
if (-not $service.StartName) { throw "Service $svc for instance $name has no service account." }
|
||||
$regKey = "$($P.InstanceRoot)\$($map[$name])\MSSQLServer\SuperSocketNetLib"
|
||||
if (-not (& $P.TestRegKey $regKey)) { throw "Registry key for instance $name not found: $regKey" }
|
||||
[pscustomobject]@{ Name = $name; Id = $map[$name]; Service = $svc; Account = [string]$service.StartName; RegKey = $regKey }
|
||||
}
|
||||
return @($targets)
|
||||
}
|
||||
|
||||
function Write-NewJsonFile([string]$Path, [object]$Value) {
|
||||
# Snapshots and request records are immutable: a file is written once and never replaced.
|
||||
$dir = Split-Path -Parent $Path
|
||||
if (-not (Test-Path -LiteralPath $dir)) { New-Item -ItemType Directory -Force -Path $dir -Confirm:$false | Out-Null }
|
||||
$json = ConvertTo-Json -InputObject $Value -Depth 6
|
||||
$stream = [System.IO.File]::Open($Path, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write)
|
||||
try {
|
||||
$writer = New-Object System.IO.StreamWriter $stream, (New-Object System.Text.UTF8Encoding $false)
|
||||
$writer.Write($json); $writer.Flush()
|
||||
}
|
||||
finally { $stream.Dispose() }
|
||||
}
|
||||
|
||||
function Restore-InstanceBinding($P, [string]$RegKey, $Previous) {
|
||||
# One instance, on its own. Returns $null when the value reads back as saved, else the problem.
|
||||
try {
|
||||
if ($null -eq $Previous) {
|
||||
if ($null -ne (& $P.GetRegValue $RegKey 'Certificate')) { & $P.RemoveRegValue $RegKey 'Certificate' }
|
||||
}
|
||||
else { & $P.SetRegValue $RegKey 'Certificate' $Previous }
|
||||
$now = & $P.GetRegValue $RegKey 'Certificate'
|
||||
if (Test-SameValue $now $Previous) { return $null }
|
||||
return "reads '$now' after restoring '$Previous'"
|
||||
}
|
||||
catch { return $_.Exception.Message }
|
||||
}
|
||||
|
||||
function Format-Value($Value) {
|
||||
if ($null -eq $Value) { return '(absent)' }
|
||||
if ($Value -eq '') { return "''" }
|
||||
return $Value
|
||||
}
|
||||
|
||||
function Get-DependentsFile($P, [string]$Service) {
|
||||
return Join-Path $P.WorkDir ('dependents-' + ($Service -replace '[^A-Za-z0-9]', '_') + '.json')
|
||||
}
|
||||
|
||||
function Invoke-InstanceRestart($P, $Confirm, [object[]]$Targets) {
|
||||
# Restart-Service -Force stops dependents (SQL Server Agent) and does not start them again.
|
||||
# The running ones are written to dependents-<service>.json before the stop, and the file is
|
||||
# removed only once they run again. A failed restart therefore does not lose them: the next
|
||||
# restart of that service, by a rerun or a rollback, starts them too.
|
||||
$failed = New-Object System.Collections.Generic.List[string]
|
||||
$restarted = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($t in $Targets) {
|
||||
if (-not (& $Confirm $t.Service 'restart service (and start its running dependents again)')) { continue }
|
||||
try {
|
||||
$file = Get-DependentsFile $P $t.Service
|
||||
$saved = @()
|
||||
if (Test-Path -LiteralPath $file) { $saved = @((Get-Content -LiteralPath $file -Raw | ConvertFrom-Json).services) }
|
||||
$deps = @(Get-UniqueNames (@($saved) + @(& $P.GetRunningDependents $t.Service)))
|
||||
if ($deps.Count) {
|
||||
if (-not (Test-Path -LiteralPath $P.WorkDir)) { New-Item -ItemType Directory -Force -Path $P.WorkDir -Confirm:$false | Out-Null }
|
||||
$json = ConvertTo-Json -InputObject ([ordered]@{ service = $t.Service; services = $deps; at = (& $P.Now).ToString('s') })
|
||||
[System.IO.File]::WriteAllText($file, $json, (New-Object System.Text.UTF8Encoding $false))
|
||||
}
|
||||
& $P.RestartService $t.Service
|
||||
foreach ($d in $deps) { & $P.StartService $d }
|
||||
if (Test-Path -LiteralPath $file) { Remove-Item -LiteralPath $file -Confirm:$false }
|
||||
$restarted.Add($t.Name)
|
||||
& $P.Log "Restarted $($t.Service)$(if ($deps.Count) { "; started again: $($deps -join ', ')" })"
|
||||
}
|
||||
catch { $failed.Add("$($t.Service): $($_.Exception.Message)") }
|
||||
}
|
||||
return [pscustomobject]@{ Restarted = $restarted.ToArray(); Failed = $failed.ToArray() }
|
||||
}
|
||||
|
||||
function New-RecordPath($P, [string]$Prefix) {
|
||||
# deploy-, request- and rollback- records sort by name in time order and are never reused.
|
||||
for ($i = 0; $i -lt 50; $i++) {
|
||||
$path = Join-Path $P.WorkDir ("$Prefix-" + (& $P.Now).ToString('yyyyMMdd-HHmmss-fff') + '.json')
|
||||
if (-not (Test-Path -LiteralPath $path)) { return $path }
|
||||
Start-Sleep -Milliseconds 5
|
||||
}
|
||||
throw "No free $Prefix record name in $($P.WorkDir)."
|
||||
}
|
||||
|
||||
function Get-RollbackState($P) {
|
||||
# Snapshot file name -> what the rollback records say: instances whose registry value was
|
||||
# restored and verified, and instances whose restart was asked for but has not succeeded.
|
||||
# Records are read in time order, so a later successful restart clears an earlier failure.
|
||||
$state = @{}
|
||||
foreach ($f in @(Get-ChildItem -LiteralPath $P.WorkDir -Filter 'rollback-*.json' -File -ErrorAction SilentlyContinue | Sort-Object Name)) {
|
||||
$r = Get-Content -LiteralPath $f.FullName -Raw | ConvertFrom-Json
|
||||
if (-not $state.ContainsKey($r.snapshot)) {
|
||||
$state[$r.snapshot] = [pscustomobject]@{
|
||||
Restored = New-Object 'System.Collections.Generic.HashSet[string]' ([StringComparer]::OrdinalIgnoreCase)
|
||||
RestartPending = New-Object 'System.Collections.Generic.HashSet[string]' ([StringComparer]::OrdinalIgnoreCase)
|
||||
}
|
||||
}
|
||||
$s = $state[$r.snapshot]
|
||||
foreach ($n in @($r.instances)) { if ($n) { [void]$s.Restored.Add($n) } }
|
||||
foreach ($n in @($r.restarted)) { if ($n) { [void]$s.RestartPending.Remove($n) } }
|
||||
foreach ($n in @($r.restartPending)) { if ($n) { [void]$s.RestartPending.Add($n) } }
|
||||
}
|
||||
return $state
|
||||
}
|
||||
|
||||
function Invoke-VsqlRollback($P, $Confirm, [bool]$WhatIf, [string[]]$Instances, [string]$Snapshot, [bool]$RestartInstances) {
|
||||
# Default: the newest snapshot with an instance not yet restored, or restored with a restart
|
||||
# that did not succeed. Each -Rollback steps back one deployment, a retry finishes the one
|
||||
# that failed, and a later run never re-applies a newer snapshot. -Snapshot picks one.
|
||||
$files = @(Get-ChildItem -LiteralPath $P.WorkDir -Filter 'deploy-*.json' -File -ErrorAction SilentlyContinue | Sort-Object Name -Descending)
|
||||
if (-not $files) { throw "No deployment snapshots in $($P.WorkDir)." }
|
||||
$load = { param($f) Get-Content -LiteralPath $f.FullName -Raw | ConvertFrom-Json }
|
||||
$pick = { param($s) @($s.instances | Where-Object { -not $Instances -or $Instances -contains $_.name }) }
|
||||
if ($Instances -and -not $Snapshot) {
|
||||
$known = @($files | ForEach-Object { (& $load $_).instances | ForEach-Object { $_.name } })
|
||||
$unknown = @($Instances | Where-Object { $known -notcontains $_ })
|
||||
if ($unknown) { throw "Instance(s) $($unknown -join ', ') are not in any snapshot. Nothing was changed." }
|
||||
}
|
||||
$chosen = $null
|
||||
$entries = @()
|
||||
if ($Snapshot) {
|
||||
$file = $files | Where-Object { $_.Name -eq $Snapshot -or $_.FullName -eq $Snapshot -or $_.BaseName -eq $Snapshot } | Select-Object -First 1
|
||||
if (-not $file) { throw "Snapshot $Snapshot not found. Snapshots: $(($files | ForEach-Object { $_.Name }) -join ', ')" }
|
||||
$chosen = [pscustomobject]@{ File = $file; Data = (& $load $file) }
|
||||
$entries = @(& $pick $chosen.Data)
|
||||
}
|
||||
else {
|
||||
$state = Get-RollbackState $P
|
||||
foreach ($f in $files) {
|
||||
$s = & $load $f
|
||||
$st = $state[$f.Name]
|
||||
$open = @(& $pick $s | Where-Object { -not $st -or -not $st.Restored.Contains($_.name) -or $st.RestartPending.Contains($_.name) })
|
||||
if ($open) { $chosen = [pscustomobject]@{ File = $f; Data = $s }; $entries = $open; break }
|
||||
}
|
||||
if (-not $chosen) {
|
||||
& $P.Log 'Every snapshot has been rolled back already. Nothing to do. Use -Snapshot to restore a specific one.'
|
||||
return [pscustomobject]@{ Snapshot = $null; Restored = @(); Failed = @(); Restarted = @() }
|
||||
}
|
||||
}
|
||||
if ($Instances) {
|
||||
$unknown = @($Instances | Where-Object { @($chosen.Data.instances | ForEach-Object { $_.name }) -notcontains $_ })
|
||||
if ($unknown) { throw "Instance(s) $($unknown -join ', ') are not in snapshot $($chosen.File.Name). Nothing was changed." }
|
||||
}
|
||||
& $P.Log "Snapshot $($chosen.File.Name): saved $($chosen.Data.at) before binding $($chosen.Data.thumbprint)."
|
||||
$restored = New-Object System.Collections.Generic.List[object]
|
||||
$failed = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($e in $entries) {
|
||||
$prev = if ($null -eq $e.previous) { $null } else { [string]$e.previous }
|
||||
if (-not (& $Confirm "instance $($e.name)" "restore Certificate to $(Format-Value $prev)")) { continue }
|
||||
$problem = Restore-InstanceBinding $P $e.regKey $prev
|
||||
if ($problem) { $failed.Add("$($e.name): $problem"); & $P.Log "FAILED to restore $($e.name): $problem" }
|
||||
else {
|
||||
$restored.Add([pscustomobject]@{ Name = $e.name; Service = $e.service; RegKey = $e.regKey })
|
||||
& $P.Log "Restored $($e.name) to $(Format-Value $prev) (verified)"
|
||||
}
|
||||
}
|
||||
$restart = [pscustomobject]@{ Restarted = @(); Failed = @() }
|
||||
if ($restored.Count) {
|
||||
$names = @($restored | ForEach-Object { $_.Name })
|
||||
if ($RestartInstances) {
|
||||
try { $restart = Invoke-InstanceRestart $P $Confirm $restored.ToArray() }
|
||||
finally {
|
||||
# Written after the restarts and even if they throw. A restored instance that was not
|
||||
# restarted stays open, so the next -Rollback -RestartInstances retries it.
|
||||
$done = @($restart.Restarted)
|
||||
Write-NewJsonFile (New-RecordPath $P 'rollback') ([ordered]@{
|
||||
at = (& $P.Now).ToString('s')
|
||||
snapshot = $chosen.File.Name
|
||||
instances = $names
|
||||
restarted = $done
|
||||
restartPending = @($names | Where-Object { $done -notcontains $_ })
|
||||
})
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-NewJsonFile (New-RecordPath $P 'rollback') ([ordered]@{ at = (& $P.Now).ToString('s'); snapshot = $chosen.File.Name; instances = $names; restarted = @(); restartPending = @() })
|
||||
& $P.Log 'Restart the restored instances for the rollback to take effect:'
|
||||
foreach ($r in $restored) { & $P.Log " Restart-Service -Name '$($r.Service)' -Force" }
|
||||
}
|
||||
}
|
||||
$all = @($failed) + @($restart.Failed)
|
||||
if ($all.Count) { throw "Rollback incomplete:`n - $($all -join "`n - ")" }
|
||||
return [pscustomobject]@{ Snapshot = $chosen.File.FullName; Restored = @($restored | ForEach-Object { $_.Name }); Failed = @(); Restarted = $restart.Restarted }
|
||||
}
|
||||
|
||||
function Invoke-VsqlCertificateDeployment {
|
||||
[CmdletBinding(SupportsShouldProcess)]
|
||||
param(
|
||||
[Parameter(Mandatory)][hashtable]$Provider,
|
||||
[string]$CertificateAuthority = 'DC-02.PESCO.COM\PESCO-PSYLOCKE-CA',
|
||||
[string]$Template = 'WebServer',
|
||||
[string[]]$DnsNames = @('vsql.pesco.com', 'vsql'),
|
||||
[string[]]$Instances,
|
||||
[string]$CerFile,
|
||||
[string]$Thumbprint,
|
||||
[switch]$RestartInstances,
|
||||
[switch]$Rollback,
|
||||
[string]$Snapshot
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$P = $Provider
|
||||
$whatIf = [bool]$WhatIfPreference
|
||||
$cmdlet = $PSCmdlet
|
||||
$confirm = if ($P.ShouldProcess) { $P.ShouldProcess } else { { param($Target, $Action) $cmdlet.ShouldProcess($Target, $Action) }.GetNewClosure() }
|
||||
# A prerequisite (enrollment, key access) that is declined cancels the run; under -WhatIf the preview goes on.
|
||||
$gate = {
|
||||
param($Target, $Action)
|
||||
if (& $confirm $Target $Action) { return $true }
|
||||
if (-not $whatIf) { throw (New-Object System.OperationCanceledException "Cancelled: $Action on $Target was declined. Nothing was bound.") }
|
||||
return $false
|
||||
}.GetNewClosure()
|
||||
|
||||
if (-not (& $P.IsAdmin)) { throw 'Run this in an elevated PowerShell on the SQL Server host.' }
|
||||
if ($CerFile -and $Thumbprint) { throw 'Use -CerFile or -Thumbprint, not both.' }
|
||||
if ($Rollback) { return Invoke-VsqlRollback $P $confirm $whatIf $Instances $Snapshot $RestartInstances.IsPresent }
|
||||
|
||||
try {
|
||||
# --- 1. instances ----------------------------------------------------------------------
|
||||
$targets = @(Get-SqlInstances $P $Instances)
|
||||
$names = @(Get-UniqueNames (@($DnsNames) + @(& $P.GetHostNames)))
|
||||
$primary = $DnsNames[0]
|
||||
& $P.Log "Instances: $(($targets | ForEach-Object { "$($_.Name) [$($_.Account)]" }) -join ', ')"
|
||||
& $P.Log "Certificate CN=$primary, SANs: $($names -join ', ')"
|
||||
$stamp = (& $P.Now).ToString('yyyyMMdd-HHmmss-fff')
|
||||
$cert = $null
|
||||
|
||||
# --- 2 and 3. the certificate ----------------------------------------------------------
|
||||
if ($Thumbprint) {
|
||||
$t = ($Thumbprint -replace '[^0-9A-Fa-f]', '').ToUpperInvariant()
|
||||
if ($t.Length -ne 40) { throw "Thumbprint '$Thumbprint' is not 40 hex digits." }
|
||||
$cert = & $P.GetMachineCert $t
|
||||
if (-not $cert) { throw "Certificate $t is not in the machine store." }
|
||||
}
|
||||
elseif ($CerFile) {
|
||||
if (-not (& $P.TestFile $CerFile)) { throw "Certificate file not found: $CerFile" }
|
||||
$issued = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 (, [System.IO.File]::ReadAllBytes($CerFile))
|
||||
if (& $gate $CerFile 'certreq -accept -machine (install the issued certificate with its pending key)') {
|
||||
$r = & $P.Certreq @('-accept', '-machine', '-q', $CerFile)
|
||||
$r.Output | ForEach-Object { & $P.Log $_ }
|
||||
if ($r.ExitCode -ne 0) { throw "certreq -accept failed ($($r.ExitCode)). The request that made this certificate must be pending on this host; if it is already installed, rerun with -Thumbprint $($issued.Thumbprint)." }
|
||||
}
|
||||
$cert = & $P.GetMachineCert $issued.Thumbprint
|
||||
if (-not $cert -and -not $whatIf) { throw "Issued certificate $($issued.Thumbprint) not found in the machine store after accept." }
|
||||
}
|
||||
else {
|
||||
$work = $P.WorkDir
|
||||
$inf = Join-Path $work "vsql-$stamp.inf"
|
||||
$csr = Join-Path $work "vsql-$stamp.req"
|
||||
$cer = Join-Path $work "vsql-$stamp.cer"
|
||||
$infText = New-CertRequestInf $primary $names $Template $stamp
|
||||
& $P.Log "INF: $inf"
|
||||
if ($whatIf) { & $P.Log $infText }
|
||||
if (& $gate $inf 'write the request INF and run certreq -new (new RSA 2048 key in the machine store)') {
|
||||
if (-not (Test-Path -LiteralPath $work)) { New-Item -ItemType Directory -Force -Path $work -Confirm:$false | Out-Null }
|
||||
[System.IO.File]::WriteAllText($inf, $infText, [System.Text.Encoding]::ASCII)
|
||||
$r = & $P.Certreq @('-new', '-q', $inf, $csr)
|
||||
$r.Output | ForEach-Object { & $P.Log $_ }
|
||||
if ($r.ExitCode -ne 0) { throw "certreq -new failed ($($r.ExitCode))." }
|
||||
}
|
||||
if (& $gate $CertificateAuthority "certreq -submit (template $Template)") {
|
||||
$r = & $P.Certreq @('-submit', '-q', '-config', $CertificateAuthority, $csr, $cer)
|
||||
$r.Output | ForEach-Object { & $P.Log $_ }
|
||||
$reqId = $r.Output | Select-String -Pattern 'RequestId:\s*"?(\d+)' | ForEach-Object { $_.Matches[0].Groups[1].Value } | Select-Object -First 1
|
||||
$requestFile = New-RecordPath $P 'request'
|
||||
Write-NewJsonFile $requestFile ([ordered]@{ id = $reqId; ca = $CertificateAuthority; template = $Template; inf = $inf; req = $csr; cer = $cer; instances = @($targets | ForEach-Object { $_.Name }); at = (& $P.Now).ToString('s'); exitCode = $r.ExitCode })
|
||||
$resume = "-CerFile `"$cer`"$(if ($Instances) { " -Instances $($Instances -join ',')" })"
|
||||
if (-not (& $P.TestFile $cer)) {
|
||||
if ($r.ExitCode -eq 0 -or ($r.Output -join ' ') -match 'pending|Taken Under Submission') {
|
||||
throw "The request is pending on the CA (request id $reqId, saved in $requestFile). Have the CA admin issue it (certsrv.msc, Pending Requests, Issue). Then on this host run: certreq -retrieve -config `"$CertificateAuthority`" $reqId `"$cer`" and rerun this script with $resume. Nothing was bound."
|
||||
}
|
||||
throw "The CA did not issue the certificate (certreq exit $($r.ExitCode), request id $reqId, saved in $requestFile). Nothing was bound."
|
||||
}
|
||||
}
|
||||
if (& $gate $cer 'certreq -accept -machine (install the issued certificate)') {
|
||||
$r = & $P.Certreq @('-accept', '-machine', '-q', $cer)
|
||||
$r.Output | ForEach-Object { & $P.Log $_ }
|
||||
if ($r.ExitCode -ne 0) { throw "certreq -accept failed ($($r.ExitCode))." }
|
||||
$issued = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 (, [System.IO.File]::ReadAllBytes($cer))
|
||||
$cert = & $P.GetMachineCert $issued.Thumbprint
|
||||
if (-not $cert) { throw "Issued certificate $($issued.Thumbprint) not found in the machine store." }
|
||||
}
|
||||
}
|
||||
|
||||
# --- 4. validate -----------------------------------------------------------------------
|
||||
$thumb = $null
|
||||
$keyFile = 'the new private key'
|
||||
if ($cert) {
|
||||
$thumb = $cert.Thumbprint.ToLowerInvariant()
|
||||
$keyInfo = & $P.GetKeyInfo $cert
|
||||
$check = Test-ServerCertificate -Certificate $cert -RequiredNames $names -Now (& $P.Now) -KeyInfo $keyInfo
|
||||
& $P.Log "Certificate $($cert.Subject) thumbprint $thumb, $($check.KeySize)-bit, $($check.Signature), expires $($cert.NotAfter.ToString('yyyy-MM-dd')), SANs $($check.DnsNames -join ', ')"
|
||||
if ($keyInfo -and $keyInfo.ProviderType -ne 0 -and -not $keyInfo.Machine) { $check.Problems.Add('the private key is not in the machine key store') }
|
||||
if ($check.Problems.Count) {
|
||||
throw "The certificate is not usable by SQL Server:`n - $($check.Problems -join "`n - ")`nNothing was bound. Check the template ($Template) on the CA; it must issue through a legacy CSP with an exchange key."
|
||||
}
|
||||
$keyFile = & $P.KeyFilePath $keyInfo
|
||||
if (-not (& $P.TestFile $keyFile)) { throw "Private key container file not found: $keyFile. Nothing was bound." }
|
||||
}
|
||||
else { & $P.Log 'WhatIf: the issued certificate would be validated here before anything is bound.' }
|
||||
|
||||
# --- 5. private key access, per selected instance account -----------------------------
|
||||
foreach ($acct in (Get-UniqueNames @($targets | ForEach-Object { $_.Account }))) {
|
||||
$sid = & $P.ResolveSid $acct
|
||||
if (& $gate $keyFile "grant read to $acct ($sid)") {
|
||||
$r = & $P.Icacls @($keyFile, '/grant', "*${sid}:(R)")
|
||||
if ($r.ExitCode -ne 0) { throw "Could not grant $acct read on the private key ($($r.Output -join ' ')). Nothing was bound." }
|
||||
& $P.Log "Private key read granted to $acct"
|
||||
}
|
||||
}
|
||||
|
||||
# --- 6. bind, with an immutable snapshot of what it replaces --------------------------
|
||||
$already = New-Object System.Collections.Generic.List[object]
|
||||
$plan = New-Object System.Collections.Generic.List[object]
|
||||
foreach ($i in $targets) {
|
||||
$cur = & $P.GetRegValue $i.RegKey 'Certificate'
|
||||
if ($thumb -and (Test-SameValue $cur $thumb)) { $already.Add($i); & $P.Log "Instance $($i.Name): already bound to $thumb; left alone, no snapshot."; continue }
|
||||
if (& $confirm "instance $($i.Name)" "set Certificate=$(if ($thumb) { $thumb } else { '<new thumbprint>' }) (now $(Format-Value $cur))") {
|
||||
$plan.Add([pscustomobject]@{ Target = $i; Previous = $cur })
|
||||
}
|
||||
}
|
||||
$snapshotFile = $null
|
||||
$bound = New-Object System.Collections.Generic.List[object]
|
||||
if ($plan.Count) {
|
||||
$snapshotFile = New-RecordPath $P 'deploy'
|
||||
Write-NewJsonFile $snapshotFile ([ordered]@{
|
||||
at = (& $P.Now).ToString('s')
|
||||
thumbprint = $thumb
|
||||
instances = @($plan | ForEach-Object { [ordered]@{ name = $_.Target.Name; service = $_.Target.Service; regKey = $_.Target.RegKey; previous = $_.Previous } })
|
||||
})
|
||||
& $P.Log "Previous bindings saved to $snapshotFile (rollback: .\Request-VsqlServerCertificate.ps1 -Rollback)"
|
||||
$attempted = New-Object System.Collections.Generic.List[object]
|
||||
try {
|
||||
foreach ($item in $plan) {
|
||||
$attempted.Add($item)
|
||||
& $P.SetRegValue $item.Target.RegKey 'Certificate' $thumb
|
||||
$readBack = & $P.GetRegValue $item.Target.RegKey 'Certificate'
|
||||
if (-not (Test-SameValue $readBack $thumb)) { throw "instance $($item.Target.Name) reads '$readBack' after binding $thumb" }
|
||||
$bound.Add($item.Target)
|
||||
$force = & $P.GetRegValue $item.Target.RegKey 'ForceEncryption'
|
||||
& $P.Log "Instance $($item.Target.Name): Certificate bound (ForceEncryption=$force, unchanged)"
|
||||
}
|
||||
}
|
||||
catch {
|
||||
$original = $_.Exception.Message
|
||||
$restoreFailures = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($item in $attempted) {
|
||||
$problem = Restore-InstanceBinding $P $item.Target.RegKey $item.Previous
|
||||
if ($problem) { $restoreFailures.Add("$($item.Target.Name): $problem") }
|
||||
else { & $P.Log "Restored $($item.Target.Name) to $(Format-Value $item.Previous) (verified)" }
|
||||
}
|
||||
$msg = "Binding failed: $original."
|
||||
if ($restoreFailures.Count) { $msg += " Automatic restore FAILED for:`n - $($restoreFailures -join "`n - ")`nRun -Rollback -Snapshot $(Split-Path -Leaf $snapshotFile)." }
|
||||
else { $msg += ' Every instance changed in this run was restored and verified.' }
|
||||
throw $msg
|
||||
}
|
||||
}
|
||||
|
||||
# --- 7. restart -----------------------------------------------------------------------
|
||||
$live = New-Object System.Collections.Generic.List[object]
|
||||
foreach ($i in $bound) { $live.Add($i) }
|
||||
foreach ($i in $already) { $live.Add($i) }
|
||||
$restart = [pscustomobject]@{ Restarted = @(); Failed = @() }
|
||||
if ($RestartInstances) { $restart = Invoke-InstanceRestart $P $confirm $(if ($whatIf) { $targets } else { $live.ToArray() }) }
|
||||
elseif ($live.Count) {
|
||||
& $P.Log ''
|
||||
& $P.Log 'Restart the instances for the certificate to take effect (off-hours; VISUAL users lose their connection):'
|
||||
foreach ($i in $live) { & $P.Log " Restart-Service -Name '$($i.Service)' -Force" }
|
||||
}
|
||||
if ($restart.Failed.Count) { throw "Restart failed:`n - $($restart.Failed -join "`n - ")`nIf an instance does not start, run this script with -Rollback -RestartInstances." }
|
||||
if ($thumb) {
|
||||
& $P.Log ''
|
||||
& $P.Log "Done. Thumbprint $thumb. After the restart the SQL Server error log should read 'The certificate [Cn = $primary] was successfully loaded for encryption'."
|
||||
}
|
||||
return [pscustomobject]@{
|
||||
Cancelled = $false
|
||||
Thumbprint = $thumb
|
||||
Bound = @($bound | ForEach-Object { $_.Name })
|
||||
AlreadyBound = @($already | ForEach-Object { $_.Name })
|
||||
Snapshot = $snapshotFile
|
||||
Restarted = $restart.Restarted
|
||||
}
|
||||
}
|
||||
catch [System.OperationCanceledException] {
|
||||
Write-Warning $_.Exception.Message
|
||||
return [pscustomobject]@{ Cancelled = $true; Thumbprint = $null; Bound = @(); AlreadyBound = @(); Snapshot = $null; Restarted = @() }
|
||||
}
|
||||
}
|
||||
|
||||
# --- entry -------------------------------------------------------------------------------------
|
||||
# Dot-sourced (by the Pester tests): define the functions only.
|
||||
if ($MyInvocation.InvocationName -eq '.') { return }
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$run = @{
|
||||
Provider = Get-VsqlDefaultProvider
|
||||
CertificateAuthority = $CertificateAuthority
|
||||
Template = $Template
|
||||
DnsNames = $DnsNames
|
||||
Instances = $Instances
|
||||
CerFile = $CerFile
|
||||
Thumbprint = $Thumbprint
|
||||
RestartInstances = $RestartInstances
|
||||
Rollback = $Rollback
|
||||
Snapshot = $Snapshot
|
||||
}
|
||||
$result = Invoke-VsqlCertificateDeployment @run
|
||||
if ($result.Cancelled) { exit 2 }
|
||||
if (-not $Rollback -and $result.Thumbprint) { Write-Host 'Then tell Packet Station: the stacks switch to VISUAL_ENCRYPT=yes (Gitea issue 5).' }
|
||||
@@ -1,716 +0,0 @@
|
||||
#Requires -Modules @{ ModuleName = 'Pester'; ModuleVersion = '5.0' }
|
||||
# Pester 5 tests for Request-VsqlServerCertificate.ps1. Run on Windows PowerShell 5.1 and PowerShell 7:
|
||||
# Invoke-Pester -Path scripts\tests -Output Detailed
|
||||
# The host (registry, services, certreq, icacls, machine store) is a fake. The certificates are
|
||||
# real, made with New-SelfSignedCertificate in Cert:\CurrentUser\My under the friendly name
|
||||
# below, and removed in AfterAll. Nothing touches the machine store, the registry or a CA.
|
||||
|
||||
BeforeAll {
|
||||
. (Join-Path $PSScriptRoot '..\Request-VsqlServerCertificate.ps1')
|
||||
|
||||
$script:TestFriendlyName = 'PS2-VSQL-CERT-TEST'
|
||||
$script:Root = 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server'
|
||||
$script:KeyDefault = "$Root\MSSQL16.MSSQLSERVER\MSSQLServer\SuperSocketNetLib"
|
||||
$script:KeyDev = "$Root\MSSQL16.DEV\MSSQLServer\SuperSocketNetLib"
|
||||
$script:HostSans = 'DNS=vsql.pesco.com&DNS=vsql&DNS=sqlhost.pesco.com&DNS=SQLHOST'
|
||||
$script:Required = @('vsql.pesco.com', 'vsql', 'sqlhost.pesco.com', 'SQLHOST')
|
||||
|
||||
Get-ChildItem Cert:\CurrentUser\My | Where-Object { $_.FriendlyName -eq $TestFriendlyName } | ForEach-Object { Remove-Item -LiteralPath "Cert:\CurrentUser\My\$($_.Thumbprint)" -DeleteKey }
|
||||
|
||||
function New-TestCert {
|
||||
param(
|
||||
[string]$Sans = $HostSans,
|
||||
[string]$Provider = 'Microsoft RSA SChannel Cryptographic Provider',
|
||||
[string]$KeySpec = 'KeyExchange',
|
||||
[int]$KeyLength = 2048,
|
||||
[string]$Hash = 'SHA256',
|
||||
[string]$Eku = '1.3.6.1.5.5.7.3.1',
|
||||
[string[]]$KeyUsage = @('DigitalSignature', 'KeyEncipherment'),
|
||||
[datetime]$NotBefore = (Get-Date).AddDays(-1),
|
||||
[datetime]$NotAfter = (Get-Date).AddYears(1)
|
||||
)
|
||||
$a = @{
|
||||
Subject = 'CN=vsql.pesco.com'
|
||||
CertStoreLocation = 'Cert:\CurrentUser\My'
|
||||
FriendlyName = $TestFriendlyName
|
||||
Provider = $Provider
|
||||
KeyLength = $KeyLength
|
||||
HashAlgorithm = $Hash
|
||||
KeyUsage = $KeyUsage
|
||||
NotBefore = $NotBefore
|
||||
NotAfter = $NotAfter
|
||||
TextExtension = @("2.5.29.17={text}$Sans", "2.5.29.37={text}$Eku")
|
||||
}
|
||||
if ($KeySpec) { $a.KeySpec = $KeySpec }
|
||||
$c = New-SelfSignedCertificate @a
|
||||
Get-Item "Cert:\CurrentUser\My\$($c.Thumbprint)"
|
||||
}
|
||||
|
||||
$script:Good = New-TestCert
|
||||
$script:Good2 = New-TestCert
|
||||
$script:Evil = New-TestCert -Sans 'DNS=vsql.pesco.com.evil.example&DNS=vsql.evil.example&DNS=sqlhost.pesco.com&DNS=SQLHOST'
|
||||
$script:Cased = New-TestCert -Sans 'DNS=VSQL.PESCO.COM&DNS=Vsql&DNS=SqlHost.Pesco.Com&DNS=sqlhost'
|
||||
$script:Ksp = New-TestCert -Provider 'Microsoft Software Key Storage Provider' -KeySpec ''
|
||||
# The SChannel CSP cannot hold a signature key, so these use the Enhanced RSA and AES CSP (type 24).
|
||||
$script:SigSpec = New-TestCert -Provider 'Microsoft Enhanced RSA and AES Cryptographic Provider' -KeySpec 'Signature' -KeyUsage 'DigitalSignature'
|
||||
$script:AesCsp = New-TestCert -Provider 'Microsoft Enhanced RSA and AES Cryptographic Provider'
|
||||
$script:Weak = New-TestCert -KeyLength 1024
|
||||
$script:Sha1 = New-TestCert -Hash 'SHA1'
|
||||
$script:NoServerAuth = New-TestCert -Eku '1.3.6.1.5.5.7.3.2'
|
||||
$script:Expired = New-TestCert -NotBefore (Get-Date).AddDays(-60) -NotAfter (Get-Date).AddDays(-1)
|
||||
|
||||
$script:Mutating = '^(certreq -new|certreq -submit|certreq -accept|icacls|SetReg|RemoveReg|Restart)'
|
||||
|
||||
# A fake SQL host: default instance and DEV, service accounts, registry, machine store,
|
||||
# certreq and icacls. $Fake.Calls records every operation in order.
|
||||
function New-FakeHost {
|
||||
param([string]$Work)
|
||||
$fake = @{
|
||||
Admin = $true
|
||||
Calls = New-Object System.Collections.ArrayList
|
||||
Log = New-Object System.Collections.ArrayList
|
||||
Instances = [ordered]@{ MSSQLSERVER = 'MSSQL16.MSSQLSERVER'; DEV = 'MSSQL16.DEV' }
|
||||
Services = @{
|
||||
'MSSQLSERVER' = [pscustomobject]@{ Name = 'MSSQLSERVER'; StartName = 'NT Service\MSSQLSERVER' }
|
||||
'MSSQL$DEV' = [pscustomobject]@{ Name = 'MSSQL$DEV'; StartName = 'PESCO\svc-sqldev' }
|
||||
}
|
||||
Registry = @{
|
||||
$KeyDefault = @{ Certificate = ''; ForceEncryption = '0' }
|
||||
$KeyDev = @{ Certificate = ''; ForceEncryption = '0' }
|
||||
}
|
||||
Store = @{}
|
||||
Submit = 'Issued' # Issued, Pending, Failure
|
||||
IssueCert = $Good
|
||||
IcaclsFail = $false
|
||||
SetFails = @{} # regKey -> $true: Set-ItemProperty throws
|
||||
SetIgnored = @{} # regKey -> $true: the write does not stick
|
||||
ReadFails = @{} # regKey -> $true: reading throws (access denied)
|
||||
Dependents = @{ 'MSSQLSERVER' = @('SQLSERVERAGENT'); 'MSSQL$DEV' = @('SQLAgent$DEV') }
|
||||
Running = New-Object 'System.Collections.Generic.HashSet[string]' ([StringComparer]::OrdinalIgnoreCase)
|
||||
RestartFails = @{} # service -> $true: stops it and its dependents, then fails to start
|
||||
KeyFile = Join-Path $Work 'machinekey.bin'
|
||||
ShouldProcess = $null
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path $Work | Out-Null
|
||||
Set-Content -Path $fake.KeyFile -Value 'key'
|
||||
$realKeyInfo = ${function:Get-CertificateKeyInfo}
|
||||
$p = @{
|
||||
WorkDir = $Work
|
||||
InstanceRoot = $Root
|
||||
IsAdmin = { [void]$fake.Calls.Add('IsAdmin'); $fake.Admin }.GetNewClosure()
|
||||
GetInstanceMap = { param($r) $fake.Instances }.GetNewClosure()
|
||||
GetService = { param($n) [void]$fake.Calls.Add("GetService $n"); $fake.Services[$n] }.GetNewClosure()
|
||||
TestRegKey = { param($k) $fake.Registry.ContainsKey($k) }.GetNewClosure()
|
||||
GetRegValue = {
|
||||
param($k, $n)
|
||||
if ($fake.ReadFails[$k]) { throw "Requested registry access is not allowed: $k" }
|
||||
$v = $fake.Registry[$k]; if ($v.ContainsKey($n)) { [string]$v[$n] } else { $null }
|
||||
}.GetNewClosure()
|
||||
SetRegValue = {
|
||||
param($k, $n, $v)
|
||||
[void]$fake.Calls.Add("SetReg $k $n=$v")
|
||||
if ($fake.SetFails[$k]) { throw "Access to $k is denied" }
|
||||
if (-not $fake.SetIgnored[$k]) { $fake.Registry[$k][$n] = $v }
|
||||
}.GetNewClosure()
|
||||
RemoveRegValue = { param($k, $n) [void]$fake.Calls.Add("RemoveReg $k $n"); if ($fake.SetFails[$k]) { throw "Access to $k is denied" }; $fake.Registry[$k].Remove($n) }.GetNewClosure()
|
||||
GetHostNames = { @('sqlhost.pesco.com', 'SQLHOST') }
|
||||
GetMachineCert = { param($t) $fake.Store[$t] }.GetNewClosure()
|
||||
Certreq = {
|
||||
param([string[]]$a)
|
||||
[void]$fake.Calls.Add("certreq $($a[0]) $($a[1..($a.Count - 1)] -join ' ')")
|
||||
switch ($a[0]) {
|
||||
'-new' { return [pscustomobject]@{ ExitCode = 0; Output = @('CertReq: Request Created') } }
|
||||
'-submit' {
|
||||
$cer = $a[-1]
|
||||
switch ($fake.Submit) {
|
||||
'Issued' { [IO.File]::WriteAllBytes($cer, $fake.IssueCert.Export('Cert')); return [pscustomobject]@{ ExitCode = 0; Output = @('RequestId: 42', 'RequestId: "42"', 'Certificate retrieved(Issued) Issued') } }
|
||||
'Pending' { return [pscustomobject]@{ ExitCode = 0; Output = @('RequestId: 43', 'RequestId: "43"', 'Certificate request is pending: Taken Under Submission (0)') } }
|
||||
default { return [pscustomobject]@{ ExitCode = -2146877436; Output = @('RequestId: 44', 'Certificate not issued (Denied) Denied by Policy Module') } }
|
||||
}
|
||||
}
|
||||
'-accept' {
|
||||
$c = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 (, [IO.File]::ReadAllBytes($a[-1]))
|
||||
$fake.Store[$c.Thumbprint] = Get-Item "Cert:\CurrentUser\My\$($c.Thumbprint)"
|
||||
return [pscustomobject]@{ ExitCode = 0; Output = @('Installed Certificate') }
|
||||
}
|
||||
}
|
||||
}.GetNewClosure()
|
||||
Icacls = {
|
||||
param([string[]]$a)
|
||||
[void]$fake.Calls.Add("icacls $($a -join ' ')")
|
||||
if ($fake.IcaclsFail) { return [pscustomobject]@{ ExitCode = 5; Output = @('Access is denied.') } }
|
||||
[pscustomobject]@{ ExitCode = 0; Output = @('processed file') }
|
||||
}.GetNewClosure()
|
||||
ResolveSid = { param($acct) @{ 'NT Service\MSSQLSERVER' = 'S-1-5-80-1'; 'PESCO\svc-sqldev' = 'S-1-5-21-9-1105' }[$acct] }
|
||||
# The real key reader on the real test certificate; only the store location is faked,
|
||||
# since the test keys live in the user store and vsql's in the machine store.
|
||||
GetKeyInfo = { param($c) $k = & $realKeyInfo $c; if ($k) { $k.Machine = $true }; $k }.GetNewClosure()
|
||||
KeyFilePath = { param($k) $fake.KeyFile }.GetNewClosure()
|
||||
TestFile = { param($f) Test-Path -LiteralPath $f -PathType Leaf }
|
||||
GetRunningDependents = { param($n) @($fake.Dependents[$n] | Where-Object { $fake.Running.Contains($_) }) }.GetNewClosure()
|
||||
RestartService = {
|
||||
param($n)
|
||||
[void]$fake.Calls.Add("Restart $n")
|
||||
foreach ($d in $fake.Dependents[$n]) { [void]$fake.Running.Remove($d) }
|
||||
[void]$fake.Running.Remove($n)
|
||||
if ($fake.RestartFails[$n]) { throw "Service '$n' cannot be started" }
|
||||
[void]$fake.Running.Add($n)
|
||||
}.GetNewClosure()
|
||||
StartService = { param($n) [void]$fake.Calls.Add("Start $n"); [void]$fake.Running.Add($n) }.GetNewClosure()
|
||||
Now = { Get-Date }
|
||||
Log = { param($m) [void]$fake.Log.Add("$m") }.GetNewClosure()
|
||||
ShouldProcess = $null
|
||||
}
|
||||
foreach ($svc in 'MSSQLSERVER', 'MSSQL$DEV', 'SQLSERVERAGENT', 'SQLAgent$DEV') { [void]$fake.Running.Add($svc) }
|
||||
$fake.Provider = $p
|
||||
return $fake
|
||||
}
|
||||
|
||||
function Get-MutatingCalls($Fake) { @($Fake.Calls | Where-Object { $_ -match $Mutating }) }
|
||||
function Get-Snapshots($Fake) { @(Get-ChildItem -LiteralPath $Fake.Provider.WorkDir -Filter 'deploy-*.json' -File -ErrorAction SilentlyContinue | Sort-Object Name) }
|
||||
function Add-ToStore($Fake, $Cert) { $Fake.Store[$Cert.Thumbprint] = $Cert }
|
||||
}
|
||||
|
||||
AfterAll {
|
||||
Get-ChildItem Cert:\CurrentUser\My | Where-Object { $_.FriendlyName -eq $TestFriendlyName } | ForEach-Object { Remove-Item -LiteralPath "Cert:\CurrentUser\My\$($_.Thumbprint)" -DeleteKey }
|
||||
}
|
||||
|
||||
Describe 'Finding 1: administrator check' {
|
||||
BeforeEach { $f = New-FakeHost (Join-Path $TestDrive ([guid]::NewGuid())); Add-ToStore $f $Good }
|
||||
|
||||
It 'refuses an unelevated session before any lookup or change' {
|
||||
$f.Admin = $false
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint } | Should -Throw '*elevated*'
|
||||
$f.Calls | Should -Be @('IsAdmin')
|
||||
}
|
||||
It 'refuses an unelevated rollback too' {
|
||||
$f.Admin = $false
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback } | Should -Throw '*elevated*'
|
||||
}
|
||||
It 'lets an elevated session through' {
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint
|
||||
$r.Bound | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
}
|
||||
It 'no longer calls Test-Admin with stray arguments' {
|
||||
$text = Get-Content (Join-Path $PSScriptRoot '..\Request-VsqlServerCertificate.ps1') -Raw
|
||||
$text | Should -Not -Match 'Test-Admin\s+-eq'
|
||||
$text | Should -Match 'if \(-not \(& \$P\.IsAdmin\)\)'
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Finding 2: named instance service lookup' {
|
||||
It 'builds a WQL filter that keeps $ unescaped' {
|
||||
Get-ServiceFilter 'MSSQL$DEV' | Should -BeExactly "Name='MSSQL`$DEV'"
|
||||
Get-SqlServiceName 'DEV' | Should -BeExactly 'MSSQL$DEV'
|
||||
Get-SqlServiceName 'MSSQLSERVER' | Should -BeExactly 'MSSQLSERVER'
|
||||
}
|
||||
It 'runs that filter against real WMI without error (the old escaped form is an invalid query)' {
|
||||
{ Get-CimInstance Win32_Service -Filter (Get-ServiceFilter 'MSSQL$DEV') -ErrorAction Stop } | Should -Not -Throw
|
||||
{ Get-CimInstance Win32_Service -Filter "Name='MSSQL\`$DEV'" -ErrorAction Stop } | Should -Throw
|
||||
}
|
||||
It 'discovers the default and DEV instances with their accounts' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'disc')
|
||||
$t = Get-SqlInstances $f.Provider $null
|
||||
$t.Name | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
($t | Where-Object Name -eq 'DEV').Service | Should -BeExactly 'MSSQL$DEV'
|
||||
($t | Where-Object Name -eq 'DEV').Account | Should -Be 'PESCO\svc-sqldev'
|
||||
($t | Where-Object Name -eq 'DEV').RegKey | Should -Be $KeyDev
|
||||
$f.Calls | Should -Contain 'GetService MSSQL$DEV'
|
||||
}
|
||||
It 'fails loudly when a service is missing' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'nosvc')
|
||||
$f.Services.Remove('MSSQL$DEV')
|
||||
{ Get-SqlInstances $f.Provider @('DEV') } | Should -Throw '*MSSQL$DEV*not found*'
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Unknown instance names' {
|
||||
It 'rejects them before any change' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'unknown')
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Instances 'DEV', 'PROD' } | Should -Throw '*Unknown instance(s): PROD*'
|
||||
Get-MutatingCalls $f | Should -BeNullOrEmpty
|
||||
@($f.Calls | Where-Object { $_ -like 'certreq*' }) | Should -BeNullOrEmpty
|
||||
Test-Path (Join-Path $f.Provider.WorkDir '*.inf') | Should -BeFalse
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Finding 3: SAN decoding and matching' {
|
||||
It 'decodes the DNS names from the raw extension' {
|
||||
$ext = $Good.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.17' }
|
||||
Get-SanDnsNames $ext.RawData | Should -Be @('vsql.pesco.com', 'vsql', 'sqlhost.pesco.com', 'SQLHOST')
|
||||
}
|
||||
It 'agrees with .NET where .NET can decode SANs (PowerShell 7)' {
|
||||
$ext = $Good.Extensions | Where-Object { $_.Oid.Value -eq '2.5.29.17' }
|
||||
if (-not ($ext.PSObject.Methods.Name -contains 'EnumerateDnsNames')) { Set-ItResult -Skipped -Because 'this .NET has no X509SubjectAlternativeNameExtension'; return }
|
||||
Get-SanDnsNames $ext.RawData | Should -Be @($ext.EnumerateDnsNames())
|
||||
}
|
||||
It 'skips non-DNS entries and handles long-form lengths' {
|
||||
# SEQUENCE { [7] 4 bytes IP, [2] 130 bytes name }
|
||||
$name = 'a' * 126 + '.com'
|
||||
$dns = [byte[]](0x82, 0x81, 130) + [Text.Encoding]::ASCII.GetBytes($name)
|
||||
$ip = [byte[]](0x87, 4, 192, 168, 1, 1)
|
||||
$body = $ip + $dns
|
||||
$raw = [byte[]](0x30, 0x81, $body.Length) + $body
|
||||
Get-SanDnsNames $raw | Should -Be @($name)
|
||||
}
|
||||
It 'rejects malformed data' {
|
||||
{ Get-SanDnsNames ([byte[]](0x30, 0x05, 0x82, 0x09, 0x61)) } | Should -Throw
|
||||
{ Get-SanDnsNames ([byte[]](0x31, 0x00)) } | Should -Throw
|
||||
}
|
||||
It 'rejects vsql.pesco.com.evil.example for vsql.pesco.com and vsql' {
|
||||
$c = Test-ServerCertificate -Certificate $Evil -RequiredNames $Required
|
||||
$c.Problems | Should -Contain 'subject alternative name vsql.pesco.com missing'
|
||||
$c.Problems | Should -Contain 'subject alternative name vsql missing'
|
||||
@($c.Problems | Where-Object { $_ -like '*sqlhost*' -or $_ -like '*SQLHOST*' }) | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'accepts case variants' {
|
||||
(Test-ServerCertificate -Certificate $Cased -RequiredNames $Required).Problems | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Certificate validation on real keys' {
|
||||
It 'accepts a CSP exchange key, RSA 2048, SHA-256, Server Authentication, all SANs' {
|
||||
$c = Test-ServerCertificate -Certificate $Good -RequiredNames $Required
|
||||
$c.Problems | Should -BeNullOrEmpty
|
||||
$c.KeyInfo.ProviderType | Should -Be 12
|
||||
$c.KeyInfo.KeySpec | Should -Be 1
|
||||
$c.KeyInfo.UniqueName | Should -Not -BeNullOrEmpty
|
||||
}
|
||||
It 'rejects a KSP (CNG) key' {
|
||||
(Test-ServerCertificate -Certificate $Ksp -RequiredNames $Required).Problems -join " | " | Should -Match 'Key Storage Provider'
|
||||
}
|
||||
It 'rejects a CSP signature key' {
|
||||
$p = (Test-ServerCertificate -Certificate $SigSpec -RequiredNames $Required).Problems
|
||||
$p -join " | " | Should -Match 'AT_KEYEXCHANGE'
|
||||
$p | Should -Contain 'key usage lacks key encipherment'
|
||||
}
|
||||
It 'rejects a CSP other than the SChannel provider' {
|
||||
$p = (Test-ServerCertificate -Certificate $AesCsp -RequiredNames $Required).Problems
|
||||
@($p) | Should -HaveCount 1
|
||||
$p -join " | " | Should -Match 'provider type 24'
|
||||
}
|
||||
It 'rejects a key under 2048 bits' {
|
||||
(Test-ServerCertificate -Certificate $Weak -RequiredNames $Required).Problems -join " | " | Should -Match 'key size 1024'
|
||||
}
|
||||
It 'rejects SHA-1' {
|
||||
(Test-ServerCertificate -Certificate $Sha1 -RequiredNames $Required).Problems -join " | " | Should -Match 'not SHA-256'
|
||||
}
|
||||
It 'rejects a certificate without Server Authentication' {
|
||||
(Test-ServerCertificate -Certificate $NoServerAuth -RequiredNames $Required).Problems | Should -Contain 'Server Authentication EKU missing'
|
||||
}
|
||||
It 'rejects an expired certificate' {
|
||||
(Test-ServerCertificate -Certificate $Expired -RequiredNames $Required).Problems -join " | " | Should -Match 'not valid now'
|
||||
}
|
||||
It 'rejects one that expires within 30 days' {
|
||||
(Test-ServerCertificate -Certificate $Good -RequiredNames $Required -Now $Good.NotAfter.AddDays(-10)).Problems -join " | " | Should -Match 'within 30 days'
|
||||
}
|
||||
It 'binds nothing when the issued certificate fails validation' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'invalid')
|
||||
Add-ToStore $f $Ksp
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Ksp.Thumbprint } | Should -Throw '*not usable*'
|
||||
Get-MutatingCalls $f | Should -BeNullOrEmpty
|
||||
Get-Snapshots $f | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Certificate request INF' {
|
||||
It 'asks for exactly what SQL Server needs' {
|
||||
$inf = New-CertRequestInf 'vsql.pesco.com' $Required 'WebServer' '20261018-090000-000'
|
||||
$inf | Should -Match 'Subject = "CN=vsql.pesco.com"'
|
||||
$inf | Should -Match 'ProviderName = "Microsoft RSA SChannel Cryptographic Provider"'
|
||||
$inf | Should -Match 'ProviderType = 12'
|
||||
$inf | Should -Match 'KeySpec = 1'
|
||||
$inf | Should -Match 'KeyLength = 2048'
|
||||
$inf | Should -Match 'KeyAlgorithm = RSA'
|
||||
$inf | Should -Match 'HashAlgorithm = SHA256'
|
||||
$inf | Should -Match 'KeyUsage = 0xA0'
|
||||
$inf | Should -Match 'MachineKeySet = TRUE'
|
||||
$inf | Should -Match 'OID = 1\.3\.6\.1\.5\.5\.7\.3\.1'
|
||||
$inf | Should -Match 'CertificateTemplate = WebServer'
|
||||
$inf | Should -Match 'Signature = "\$Windows NT\$"'
|
||||
foreach ($n in $Required) { $inf | Should -Match ([regex]::Escape("_continue_ = `"dns=$n&`"")) }
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'A full request, issued' {
|
||||
It 'requests, accepts with -machine, validates, grants each account, snapshots and binds' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'issued')
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider
|
||||
$r.Thumbprint | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
$r.Bound | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
$order = @($f.Calls | Where-Object { $_ -match $Mutating } | ForEach-Object { ($_ -split ' ')[0..1] -join ' ' })
|
||||
$order[0..2] | Should -Be @('certreq -new', 'certreq -submit', 'certreq -accept')
|
||||
@($f.Calls | Where-Object { $_ -like 'certreq -accept*' })[0] | Should -Match '-machine -q'
|
||||
@($f.Calls | Where-Object { $_ -like 'icacls*' }) | Should -Be @("icacls $($f.KeyFile) /grant *S-1-5-80-1:(R)", "icacls $($f.KeyFile) /grant *S-1-5-21-9-1105:(R)")
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $r.Thumbprint
|
||||
$f.Registry[$KeyDev].ForceEncryption | Should -Be '0'
|
||||
@(Get-ChildItem $f.Provider.WorkDir -Filter 'request-*.json').Count | Should -Be 1
|
||||
(Get-Content (Get-Snapshots $f)[0].FullName -Raw | ConvertFrom-Json).instances.previous | Should -Be @('', '')
|
||||
@($f.Calls | Where-Object { $_ -like 'Restart*' }) | Should -BeNullOrEmpty
|
||||
$f.Log | Should -Contain " Restart-Service -Name 'MSSQL`$DEV' -Force"
|
||||
}
|
||||
It 'restarts only with -RestartInstances' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'restart')
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Instances DEV -RestartInstances
|
||||
$r.Bound | Should -Be @('DEV')
|
||||
$f.Calls | Should -Contain 'Restart MSSQL$DEV'
|
||||
$f.Calls | Should -Not -Contain 'Restart MSSQLSERVER'
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be ''
|
||||
}
|
||||
It 'fails cleanly when the CA denies the request' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'denied')
|
||||
$f.Submit = 'Failure'
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider } | Should -Throw '*did not issue*request id 44*'
|
||||
@($f.Calls | Where-Object { $_ -match '^(icacls|SetReg|certreq -accept)' }) | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Pending request' {
|
||||
It 'saves the request id, binds nothing, and resumes with -CerFile' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'pending')
|
||||
$f.Submit = 'Pending'
|
||||
$err = $null
|
||||
try { Invoke-VsqlCertificateDeployment -Provider $f.Provider -Instances DEV } catch { $err = $_.Exception.Message }
|
||||
$err | Should -Match 'pending on the CA \(request id 43'
|
||||
$err | Should -Match 'certreq -retrieve -config "DC-02.PESCO.COM\\PESCO-PSYLOCKE-CA" 43'
|
||||
$err | Should -Match '-Instances DEV'
|
||||
@($f.Calls | Where-Object { $_ -match '^(icacls|SetReg|certreq -accept)' }) | Should -BeNullOrEmpty
|
||||
$req = Get-Content (Get-ChildItem $f.Provider.WorkDir -Filter 'request-*.json')[0].FullName -Raw | ConvertFrom-Json
|
||||
$req.id | Should -Be '43'
|
||||
$req.instances | Should -Be @('DEV')
|
||||
|
||||
# The CA admin issues it; certreq -retrieve writes the .cer the record names.
|
||||
[IO.File]::WriteAllBytes($req.cer, $Good.Export('Cert'))
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -CerFile $req.cer -Instances DEV
|
||||
@($f.Calls | Where-Object { $_ -like 'certreq -accept*' })[0] | Should -Be "certreq -accept -machine -q $($req.cer)"
|
||||
$r.Bound | Should -Be @('DEV')
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Finding 5: declined or failed prerequisites' {
|
||||
BeforeEach { $f = New-FakeHost (Join-Path $TestDrive ([guid]::NewGuid())); Add-ToStore $f $Good }
|
||||
|
||||
It 'a declined ACL grant cancels the run and binds nothing' {
|
||||
$f.Provider.ShouldProcess = { param($t, $a) $a -notlike 'grant read*' }
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint 3>$null
|
||||
$r.Cancelled | Should -BeTrue
|
||||
@($f.Calls | Where-Object { $_ -match '^(SetReg|RemoveReg|Restart|icacls)' }) | Should -BeNullOrEmpty
|
||||
Get-Snapshots $f | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'a declined second grant cancels after the first, still binding nothing' {
|
||||
$count = @{ n = 0 }
|
||||
$f.Provider.ShouldProcess = { param($t, $a) if ($a -like 'grant read*') { $count.n++; return ($count.n -lt 2) }; $true }.GetNewClosure()
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint 3>$null
|
||||
$r.Cancelled | Should -BeTrue
|
||||
@($f.Calls | Where-Object { $_ -match '^(SetReg|RemoveReg|Restart)' }) | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'a failed icacls stops the run and binds nothing' {
|
||||
$f.IcaclsFail = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint } | Should -Throw '*Could not grant*Nothing was bound*'
|
||||
@($f.Calls | Where-Object { $_ -match '^(SetReg|RemoveReg|Restart)' }) | Should -BeNullOrEmpty
|
||||
Get-Snapshots $f | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'declining certreq -new, -submit or -accept cancels before the next step' {
|
||||
foreach ($step in 'certreq -new', 'certreq -submit', 'certreq -accept') {
|
||||
$g = New-FakeHost (Join-Path $TestDrive ([guid]::NewGuid()))
|
||||
$g.Provider.ShouldProcess = { param($t, $a) $a -notlike "*$step*" }.GetNewClosure()
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $g.Provider 3>$null
|
||||
$r.Cancelled | Should -BeTrue
|
||||
@($g.Calls | Where-Object { $_ -like "$step*" }) | Should -BeNullOrEmpty
|
||||
@($g.Calls | Where-Object { $_ -match '^(icacls|SetReg)' }) | Should -BeNullOrEmpty
|
||||
}
|
||||
}
|
||||
It 'declining one instance binding leaves that instance out of the snapshot' {
|
||||
$f.Provider.ShouldProcess = { param($t, $a) $t -ne 'instance MSSQLSERVER' }
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint
|
||||
$r.Bound | Should -Be @('DEV')
|
||||
(Get-Content (Get-Snapshots $f)[0].FullName -Raw | ConvertFrom-Json).instances.name | Should -Be @('DEV')
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Finding 4: snapshots per deployment' {
|
||||
It 'keeps the useful snapshot when the bound certificate is run again, and rolls back to it' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'rerun')
|
||||
Add-ToStore $f $Good; Add-ToStore $f $Good2
|
||||
$a = $Good.Thumbprint.ToLowerInvariant(); $b = $Good2.Thumbprint.ToLowerInvariant()
|
||||
$f.Registry[$KeyDefault].Certificate = $a; $f.Registry[$KeyDev].Certificate = $a
|
||||
|
||||
$r1 = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint
|
||||
$r1.Bound | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
$snap = Get-Snapshots $f
|
||||
$snap.Count | Should -Be 1
|
||||
$before = Get-FileHash $snap[0].FullName
|
||||
|
||||
Start-Sleep -Milliseconds 20
|
||||
$f.Calls.Clear()
|
||||
$r2 = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint -RestartInstances
|
||||
$r2.Bound | Should -BeNullOrEmpty
|
||||
$r2.AlreadyBound | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
$r2.Snapshot | Should -BeNullOrEmpty
|
||||
@($f.Calls | Where-Object { $_ -like 'SetReg*' }) | Should -BeNullOrEmpty
|
||||
$f.Calls | Should -Contain 'Restart MSSQL$DEV'
|
||||
(Get-Snapshots $f).Count | Should -Be 1
|
||||
(Get-FileHash $snap[0].FullName).Hash | Should -Be $before.Hash
|
||||
|
||||
$rb = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback
|
||||
$rb.Restored | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be $a
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $a
|
||||
$b | Should -Not -Be $a
|
||||
}
|
||||
It 'never replaces an existing snapshot file' {
|
||||
$path = Join-Path $TestDrive 'immutable.json'
|
||||
Write-NewJsonFile $path @{ a = 1 }
|
||||
{ Write-NewJsonFile $path @{ a = 2 } } | Should -Throw
|
||||
(Get-Content $path -Raw | ConvertFrom-Json).a | Should -Be 1
|
||||
}
|
||||
It 'records an absent value as null and an empty value as empty' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'nullempty')
|
||||
Add-ToStore $f $Good
|
||||
$f.Registry[$KeyDefault].Remove('Certificate')
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint | Out-Null
|
||||
$s = Get-Content (Get-Snapshots $f)[0].FullName -Raw | ConvertFrom-Json
|
||||
$null -eq ($s.instances | Where-Object name -eq 'MSSQLSERVER').previous | Should -BeTrue
|
||||
($s.instances | Where-Object name -eq 'DEV').previous | Should -BeExactly ''
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback | Out-Null
|
||||
$f.Registry[$KeyDefault].ContainsKey('Certificate') | Should -BeFalse
|
||||
$f.Registry[$KeyDev].Certificate | Should -BeExactly ''
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Finding 6: rollback restores each instance on its own and verifies' {
|
||||
BeforeEach {
|
||||
$f = New-FakeHost (Join-Path $TestDrive ([guid]::NewGuid()))
|
||||
Add-ToStore $f $Good
|
||||
$f.Registry[$KeyDefault].Certificate = 'aaaa'; $f.Registry[$KeyDev].Certificate = 'bbbb'
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint | Out-Null
|
||||
$f.Calls.Clear()
|
||||
}
|
||||
|
||||
It 'restores and verifies every instance' {
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback
|
||||
$r.Restored | Should -Be @('MSSQLSERVER', 'DEV')
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be 'aaaa'
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
$f.Log | Should -Contain 'Restored DEV to bbbb (verified)'
|
||||
}
|
||||
It 'keeps going past a failed instance and reports it' {
|
||||
$f.SetFails[$KeyDefault] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback } | Should -Throw '*Rollback incomplete*MSSQLSERVER*denied*'
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
$f.Log | Should -Not -Contain 'Restored MSSQLSERVER to aaaa (verified)'
|
||||
}
|
||||
It 'reports a write that does not stick instead of claiming success' {
|
||||
$f.SetIgnored[$KeyDev] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback } | Should -Throw "*DEV: reads*after restoring 'bbbb'*"
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be 'aaaa'
|
||||
}
|
||||
It 'restores only the named instances and rejects names not in the snapshot' {
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -Instances PROD } | Should -Throw '*not in any snapshot*'
|
||||
@($f.Calls | Where-Object { $_ -like 'SetReg*' }) | Should -BeNullOrEmpty
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -Instances DEV -RestartInstances
|
||||
$r.Restored | Should -Be @('DEV')
|
||||
$f.Calls | Should -Contain 'Restart MSSQL$DEV'
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
# The next default rollback finishes the same snapshot with the instance left.
|
||||
$f.Calls.Clear()
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback).Restored | Should -Be @('MSSQLSERVER')
|
||||
@($f.Calls | Where-Object { $_ -like "SetReg $KeyDev*" }) | Should -BeNullOrEmpty
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be 'aaaa'
|
||||
}
|
||||
It 'retries an instance whose restore failed' {
|
||||
$f.SetFails[$KeyDefault] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback } | Should -Throw '*Rollback incomplete*'
|
||||
$f.SetFails.Clear()
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback).Restored | Should -Be @('MSSQLSERVER')
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be 'aaaa'
|
||||
}
|
||||
It 'defaults to the newest snapshot not yet rolled back; -Snapshot picks one' {
|
||||
Add-ToStore $f $Good2
|
||||
Start-Sleep -Milliseconds 20
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint | Out-Null
|
||||
$snaps = Get-Snapshots $f
|
||||
$snaps.Count | Should -Be 2
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback).Snapshot | Should -Be $snaps[1].FullName
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback).Snapshot | Should -Be $snaps[0].FullName
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
# Both rolled back: a third run does nothing, it does not re-apply the newer snapshot.
|
||||
$f.Calls.Clear()
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback).Snapshot | Should -BeNullOrEmpty
|
||||
@($f.Calls | Where-Object { $_ -like 'SetReg*' }) | Should -BeNullOrEmpty
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -Snapshot $snaps[1].Name).Snapshot | Should -Be $snaps[1].FullName
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
}
|
||||
It 'automatic restore during binding restores each changed instance on its own' {
|
||||
Add-ToStore $f $Good2
|
||||
$f.SetFails[$KeyDev] = $true
|
||||
$err = $null
|
||||
try { Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint } catch { $err = $_.Exception.Message }
|
||||
$err | Should -Match 'Binding failed'
|
||||
$err | Should -Match 'restore FAILED for:\s+- DEV'
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
$f.Log | Should -Contain "Restored MSSQLSERVER to $($Good.Thumbprint.ToLowerInvariant()) (verified)"
|
||||
}
|
||||
It 'automatic restore after a write that does not stick reports success only when verified' {
|
||||
Add-ToStore $f $Good2
|
||||
$f.SetIgnored[$KeyDev] = $true
|
||||
$err = $null
|
||||
try { Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint } catch { $err = $_.Exception.Message }
|
||||
$err | Should -Match 'DEV reads'
|
||||
$err | Should -Match 'Every instance changed in this run was restored and verified'
|
||||
$f.Registry[$KeyDefault].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Second review: recovery paths' {
|
||||
BeforeEach {
|
||||
$f = New-FakeHost (Join-Path $TestDrive ([guid]::NewGuid()))
|
||||
Add-ToStore $f $Good
|
||||
$f.Registry[$KeyDefault].Certificate = 'aaaa'; $f.Registry[$KeyDev].Certificate = 'bbbb'
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint | Out-Null
|
||||
$f.Calls.Clear()
|
||||
}
|
||||
|
||||
It 'a rollback whose restart failed is retried, not skipped and not stepped past' {
|
||||
$f.RestartFails['MSSQL$DEV'] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances } | Should -Throw '*Rollback incomplete*MSSQL$DEV*'
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
$f.RestartFails.Clear()
|
||||
$f.Calls.Clear()
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances
|
||||
$r.Snapshot | Should -Not -BeNullOrEmpty
|
||||
$r.Restarted | Should -Be @('DEV')
|
||||
$f.Calls | Should -Contain 'Restart MSSQL$DEV'
|
||||
$f.Calls | Should -Not -Contain 'Restart MSSQLSERVER'
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be 'bbbb'
|
||||
# Now complete: nothing left to do.
|
||||
(Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances).Snapshot | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'a declined restart under -Confirm also stays open' {
|
||||
$f.Provider.ShouldProcess = { param($t, $a) $a -notlike 'restart service*' -or $t -ne 'MSSQL$DEV' }
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances | Out-Null
|
||||
$f.Provider.ShouldProcess = $null
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances
|
||||
$r.Restarted | Should -Be @('DEV')
|
||||
}
|
||||
It 'SQL Server Agent is started again after a failed restart is retried' {
|
||||
$f.RestartFails['MSSQL$DEV'] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances } | Should -Throw
|
||||
$f.Running.Contains('SQLAgent$DEV') | Should -BeFalse
|
||||
$depFile = Join-Path $f.Provider.WorkDir 'dependents-MSSQL_DEV.json'
|
||||
(Get-Content $depFile -Raw | ConvertFrom-Json).services | Should -Be @('SQLAgent$DEV')
|
||||
$f.RestartFails.Clear()
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances | Out-Null
|
||||
$f.Calls | Should -Contain 'Start SQLAgent$DEV'
|
||||
$f.Running.Contains('SQLAgent$DEV') | Should -BeTrue
|
||||
Test-Path $depFile | Should -BeFalse
|
||||
}
|
||||
It 'a successful restart starts the agent that ran and leaves no dependents file' {
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint -Instances MSSQLSERVER -RestartInstances | Out-Null
|
||||
$f.Calls | Should -Contain 'Start SQLSERVERAGENT'
|
||||
$f.Running.Contains('SQLSERVERAGENT') | Should -BeTrue
|
||||
@(Get-ChildItem $f.Provider.WorkDir -Filter 'dependents-*.json') | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'a registry read failure stops a deployment before any snapshot or change' {
|
||||
Add-ToStore $f $Good2
|
||||
$before = @(Get-Snapshots $f).Count
|
||||
$f.ReadFails[$KeyDev] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good2.Thumbprint } | Should -Throw '*not allowed*'
|
||||
@($f.Calls | Where-Object { $_ -match '^(SetReg|RemoveReg)' }) | Should -BeNullOrEmpty
|
||||
@(Get-Snapshots $f).Count | Should -Be $before
|
||||
}
|
||||
It 'a registry read failure during rollback is a failure, never verified as absent' {
|
||||
$f.ReadFails[$KeyDev] = $true
|
||||
{ Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback } | Should -Throw '*Rollback incomplete*DEV*not allowed*'
|
||||
$f.Log | Should -Not -Contain 'Restored DEV to bbbb (verified)'
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Real registry operations (HKCU test key)' {
|
||||
BeforeAll {
|
||||
$script:TestRoot = "HKCU:\Software\PS2-VSQL-CERT-TEST-$([guid]::NewGuid())"
|
||||
New-Item -Path "$TestRoot\Instance Names\SQL" -Force | Out-Null
|
||||
New-ItemProperty -Path "$TestRoot\Instance Names\SQL" -Name 'PSREPORTING' -Value 'MSSQL16.PSREPORTING' | Out-Null
|
||||
New-ItemProperty -Path "$TestRoot\Instance Names\SQL" -Name 'DEV' -Value 'MSSQL16.DEV' | Out-Null
|
||||
New-Item -Path "$TestRoot\Empty" -Force | Out-Null
|
||||
New-ItemProperty -Path "$TestRoot\Empty" -Name 'Certificate' -Value '' | Out-Null
|
||||
New-Item -Path "$TestRoot\Absent" -Force | Out-Null
|
||||
$script:Real = Get-VsqlDefaultProvider
|
||||
}
|
||||
AfterAll { Remove-Item -LiteralPath $TestRoot -Recurse -Force }
|
||||
|
||||
It 'lists every instance, including one whose name starts with PS' {
|
||||
$map = & $Real.GetInstanceMap $TestRoot
|
||||
@($map.Keys) | Should -Be @('PSREPORTING', 'DEV')
|
||||
$map['PSREPORTING'] | Should -Be 'MSSQL16.PSREPORTING'
|
||||
}
|
||||
It 'returns an empty map when no instance is registered' {
|
||||
(& $Real.GetInstanceMap "$TestRoot\Absent").Count | Should -Be 0
|
||||
}
|
||||
It 'tells an empty value from an absent one' {
|
||||
& $Real.GetRegValue "$TestRoot\Empty" 'Certificate' | Should -BeExactly ''
|
||||
$null -eq (& $Real.GetRegValue "$TestRoot\Absent" 'Certificate') | Should -BeTrue
|
||||
}
|
||||
It 'throws, rather than reporting absent, when the key cannot be read' {
|
||||
{ & $Real.GetRegValue "$TestRoot\Missing" 'Certificate' } | Should -Throw
|
||||
}
|
||||
It 'writes, reads back and removes through the real operations' {
|
||||
& $Real.SetRegValue "$TestRoot\Absent" 'Certificate' 'abcd'
|
||||
& $Real.GetRegValue "$TestRoot\Absent" 'Certificate' | Should -Be 'abcd'
|
||||
& $Real.RemoveRegValue "$TestRoot\Absent" 'Certificate'
|
||||
$null -eq (& $Real.GetRegValue "$TestRoot\Absent" 'Certificate') | Should -BeTrue
|
||||
}
|
||||
}
|
||||
|
||||
Describe '-WhatIf' {
|
||||
It 'changes nothing on the real ShouldProcess path' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'whatif-real')
|
||||
$r = Invoke-VsqlCertificateDeployment -Provider $f.Provider -RestartInstances -WhatIf 6>$null
|
||||
Get-MutatingCalls $f | Should -BeNullOrEmpty
|
||||
@(Get-ChildItem $f.Provider.WorkDir -File | Where-Object Name -ne 'machinekey.bin') | Should -BeNullOrEmpty
|
||||
$r.Bound | Should -BeNullOrEmpty
|
||||
}
|
||||
It 'previews every step of a request' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'whatif-steps')
|
||||
$seen = New-Object System.Collections.ArrayList
|
||||
$f.Provider.ShouldProcess = { param($t, $a) [void]$seen.Add("$a | $t"); $false }.GetNewClosure()
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -RestartInstances -WhatIf | Out-Null
|
||||
Get-MutatingCalls $f | Should -BeNullOrEmpty
|
||||
$expect = 'certreq -new', 'certreq -submit', 'certreq -accept', 'grant read to NT Service\MSSQLSERVER', 'grant read to PESCO\svc-sqldev',
|
||||
'set Certificate=<new thumbprint>* | instance MSSQLSERVER', 'set Certificate=<new thumbprint>* | instance DEV', 'restart service* | MSSQLSERVER', 'restart service* | MSSQL$DEV'
|
||||
foreach ($e in $expect) { @($seen | Where-Object { $_ -like "*$e*" }).Count | Should -Be 1 -Because "the preview should include '$e'" }
|
||||
$f.Log -join "`n" | Should -Match 'ProviderType = 12'
|
||||
}
|
||||
It 'previews a rollback without changing it' {
|
||||
$f = New-FakeHost (Join-Path $TestDrive 'whatif-rollback')
|
||||
Add-ToStore $f $Good
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Thumbprint $Good.Thumbprint | Out-Null
|
||||
$f.Calls.Clear()
|
||||
Invoke-VsqlCertificateDeployment -Provider $f.Provider -Rollback -RestartInstances -WhatIf 6>$null | Out-Null
|
||||
Get-MutatingCalls $f | Should -BeNullOrEmpty
|
||||
$f.Registry[$KeyDev].Certificate | Should -Be $Good.Thumbprint.ToLowerInvariant()
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Account to SID' {
|
||||
It 'maps the built-in service accounts that icacls cannot name' {
|
||||
Resolve-AccountSid 'LocalSystem' | Should -Be 'S-1-5-18'
|
||||
Resolve-AccountSid 'NT AUTHORITY\NetworkService' | Should -Be 'S-1-5-20'
|
||||
Resolve-AccountSid 'NT AUTHORITY\LocalService' | Should -Be 'S-1-5-19'
|
||||
}
|
||||
It 'translates a real account' {
|
||||
Resolve-AccountSid "$env:USERDOMAIN\$env:USERNAME" | Should -Match '^S-1-5-21-'
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Native commands' {
|
||||
It 'captures stderr and the exit code without throwing under Stop' {
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$r = Invoke-NativeCommand 'cmd.exe' @('/c', 'echo out & echo err 1>&2 & exit 3')
|
||||
$r.ExitCode | Should -Be 3
|
||||
$r.Output -join ' ' | Should -Match 'err'
|
||||
}
|
||||
}
|
||||
@@ -18,9 +18,9 @@ from psycopg.types.json import Jsonb
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .. import audit, events, policy
|
||||
from ..domain import archive, packets, search, shop
|
||||
from ..domain import assets as asset_store
|
||||
from ..domain import ncr as ncrs
|
||||
from ..domain import packets, search, shop
|
||||
from ..domain import qc as qc_dom
|
||||
from ..domain.pipeline import (
|
||||
Expect,
|
||||
@@ -543,6 +543,10 @@ TRANSITION_FIELDS: dict[policy.Action, tuple[str, ...]] = {
|
||||
A.FILE: ("note",),
|
||||
A.ADMIN_REOPEN: ("reason",),
|
||||
}
|
||||
MSG_ARCHIVE = (
|
||||
"The packet could not be copied to the archive: {reason}. Nothing was filed, and Admin has been told."
|
||||
" File it again once the archive works."
|
||||
)
|
||||
DEPT_ACTIONS = (A.RELEASE_TO_SHOP, A.ROUTE_FROM_REVIEW, A.QC_ACCEPT_AND_ROUTE)
|
||||
|
||||
|
||||
@@ -591,29 +595,34 @@ def transition(
|
||||
else:
|
||||
del extra["stamp"]
|
||||
content = {"action": str(action), "packet_id": packet_id, "expect": body.expect.model_dump(), **extra}
|
||||
with packet_write(
|
||||
d,
|
||||
p,
|
||||
packet_id,
|
||||
action,
|
||||
request=request,
|
||||
extra=extra,
|
||||
it_reason=body.it_reason,
|
||||
sign=body.sign,
|
||||
expect=body.expect,
|
||||
sign_content=content,
|
||||
policy_principal=pp,
|
||||
) as w:
|
||||
if action is A.ASSIGN:
|
||||
ok = conn.execute(
|
||||
"SELECT 1 FROM stations WHERE id = %s AND dept_code = %s AND active",
|
||||
(extra["station_id"], w.state.dept),
|
||||
).fetchone()
|
||||
if ok is None:
|
||||
raise ApiError(422, "invalid_station", "Choose a station in this department.")
|
||||
active_user(conn, extra["welder_id"], "welder")
|
||||
shop.welder_check(conn, w.state.dept, str(extra["welder_id"]), int(extra["station_id"]))
|
||||
apply_transition(w)
|
||||
try:
|
||||
with packet_write(
|
||||
d,
|
||||
p,
|
||||
packet_id,
|
||||
action,
|
||||
request=request,
|
||||
extra=extra,
|
||||
it_reason=body.it_reason,
|
||||
sign=body.sign,
|
||||
expect=body.expect,
|
||||
sign_content=content,
|
||||
policy_principal=pp,
|
||||
) as w:
|
||||
if action is A.ASSIGN:
|
||||
ok = conn.execute(
|
||||
"SELECT 1 FROM stations WHERE id = %s AND dept_code = %s AND active",
|
||||
(extra["station_id"], w.state.dept),
|
||||
).fetchone()
|
||||
if ok is None:
|
||||
raise ApiError(422, "invalid_station", "Choose a station in this department.")
|
||||
active_user(conn, extra["welder_id"], "welder")
|
||||
shop.welder_check(conn, w.state.dept, str(extra["welder_id"]), int(extra["station_id"]))
|
||||
apply_transition(w)
|
||||
except archive.FilingCopyFailed as e:
|
||||
# The filing rolled back (Gitea issue 57); record it and tell Admin in a transaction of its own.
|
||||
archive.record_filing_failure(conn, p, packet_id, e)
|
||||
raise ApiError(503, "archive_failed", MSG_ARCHIVE.format(reason=e.message.rstrip("."))) from None
|
||||
return _read(d, pp, packet_id)
|
||||
|
||||
|
||||
|
||||
@@ -6,9 +6,14 @@ A path that leaves the root is refused: empty, `..`, an absolute path elsewhere,
|
||||
the root, or a folder that does not exist. The check writes a probe file, reads it back and removes it. Nothing here
|
||||
creates folders or deletes anything in the archive.
|
||||
|
||||
FILE inserts a job request for `archive-copy` in its own transaction. The worker job (worker/jobs/archive_copy.py)
|
||||
queues every unbroken seal with a data book that has no `archive_copies` row, then copies the pending and due failed
|
||||
rows. A failed copy never blocks filing: it is retried with a backoff and raised to Admin as `archive_failed`.
|
||||
FILE copies the packet into the archive inside its own transaction (Gitea issue 57, Spencer 2026-11-03): the copy
|
||||
must succeed before the packet counts as filed. A failed folder check or copy rolls the whole filing back, so the
|
||||
packet stays where it was and can be filed again. The failure is then recorded in its own transaction and raised to
|
||||
Admin as the `archive_failed` condition with subject `filing:<packet id>`, cleared when that packet files.
|
||||
|
||||
The worker job (worker/jobs/archive_copy.py) copies the seals filed while the archive was off: it queues every
|
||||
unbroken seal with a data book that has no `archive_copies` row, then copies the pending and due failed rows with a
|
||||
backoff, and raises `archive_failed` (subject `archive`) to Admin.
|
||||
|
||||
Folder layout under the archive folder: `<project>/<unit>/<data book name>-seal<seal id>/` with the data book PDF,
|
||||
the packet's files (drawing page files, packet documents, the Q.C. folder documents of its unit and project, MTRs)
|
||||
@@ -45,6 +50,8 @@ JOB = "archive-copy"
|
||||
JOB_CHANNEL = "ps_job_request"
|
||||
EVENT = "archive_failed"
|
||||
EVENT_SUBJECT = "archive"
|
||||
FILING_SUBJECT = "filing:{packet_id}"
|
||||
FILING_TITLE = "Filing blocked: archive copy failed"
|
||||
PROBE_PREFIX = ".packetstation-probe-"
|
||||
MANIFEST = "manifest.json"
|
||||
DIR_MAX = 400
|
||||
@@ -72,6 +79,16 @@ class CopyError(Exception):
|
||||
"""One copy failed. The message goes into archive_copies.last_error and the audit row."""
|
||||
|
||||
|
||||
class FilingCopyFailed(Exception):
|
||||
"""FILE could not copy the packet into the archive. The filing rolls back; the message is for the person filing
|
||||
and for Admin."""
|
||||
|
||||
def __init__(self, message: str, archive_dir: str | None):
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
self.archive_dir = archive_dir
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CheckResult:
|
||||
ok: bool
|
||||
@@ -320,20 +337,104 @@ def request_copy(conn: Any, user_id: Any = None, note: str | None = None) -> Non
|
||||
conn.execute("SELECT ps_notify(%s, %s)", (JOB_CHANNEL, json.dumps({"job": JOB}, separators=(",", ":"))))
|
||||
|
||||
|
||||
def on_filed(conn: Any, settings: Settings, user_id: Any, packet_id: int) -> bool:
|
||||
"""FILE's hook: request a prompt copy when the archive is on and a folder is saved. Never raises; a failure
|
||||
here only means the copy waits for the next scheduled run. Returns whether a request went in."""
|
||||
if root_of(settings) is None:
|
||||
return False
|
||||
def copy_at_filing(conn: Any, settings: Settings, packet_id: int, seal_id: int) -> dict[str, Any] | None:
|
||||
"""FILE's copy, in the filing transaction: check the folder, copy the seal, record the copy as done.
|
||||
Returns {folder, files}, or None when the archive is off (ARCHIVE_ROOT unset or no folder saved).
|
||||
Raises FilingCopyFailed, which rolls the filing back."""
|
||||
root = root_of(settings)
|
||||
if root is None:
|
||||
return None
|
||||
archive_dir = load(conn)[0]
|
||||
if archive_dir is None:
|
||||
return None
|
||||
rel, result = check(root, archive_dir)
|
||||
if not result.ok or rel is None:
|
||||
raise FilingCopyFailed(f"The archive folder check failed: {result.message}", archive_dir)
|
||||
base = root.joinpath(*rel.split("/"))
|
||||
try:
|
||||
folder, sha, files = copy_seal(conn, settings, base, rel, seal_id)
|
||||
except CopyError as e:
|
||||
raise FilingCopyFailed(str(e), archive_dir) from None
|
||||
except Exception as e: # anything unexpected blocks the filing the same way
|
||||
log.exception("archive copy of packet %s at filing failed", packet_id)
|
||||
raise FilingCopyFailed(f"{type(e).__name__}: {e}", archive_dir) from None
|
||||
with conn.transaction():
|
||||
conn.execute(
|
||||
"INSERT INTO archive_copies (seal_id, packet_id, status, attempts, folder, manifest_sha256, done_at)"
|
||||
" VALUES (%s, %s, 'done', 1, %s, %s, now())",
|
||||
(seal_id, packet_id, folder, sha),
|
||||
)
|
||||
audit.append(
|
||||
conn,
|
||||
action="archive.copied",
|
||||
subject_type="seal",
|
||||
subject_id=str(seal_id),
|
||||
packet_id=int(packet_id),
|
||||
detail={"seal_id": seal_id, "folder": folder, "files": files, "manifest_sha256": sha, "source": "filing"},
|
||||
)
|
||||
from ..notify import resolve_event
|
||||
|
||||
resolve_event(
|
||||
conn, EVENT, subject=FILING_SUBJECT.format(packet_id=packet_id), body="The packet was filed and copied."
|
||||
)
|
||||
return {"folder": folder, "files": files}
|
||||
|
||||
|
||||
def record_filing_failure(conn: Any, who: Any, packet_id: int, failure: FilingCopyFailed) -> None:
|
||||
"""After the filing rolled back: one `archive.filing_blocked` audit row and the Admin alert, in their own
|
||||
transaction. Never raises; the person filing gets the refusal either way."""
|
||||
from ..notify import raise_event
|
||||
|
||||
try:
|
||||
with conn.transaction():
|
||||
if load(conn)[0] is None:
|
||||
return False
|
||||
request_copy(conn, user_id, f"filed packet {packet_id}")
|
||||
return True
|
||||
row = conn.execute(
|
||||
"SELECT pr.number, u.lot, p.drawing, p.status, p.dept_code FROM packets p"
|
||||
" JOIN units u ON u.id = p.unit_id JOIN projects pr ON pr.id = u.project_id WHERE p.id = %s",
|
||||
(packet_id,),
|
||||
).fetchone()
|
||||
label = packet_label(*row[:3]) if row else f"packet {packet_id}"
|
||||
user_id = getattr(who, "user_id", None)
|
||||
name = conn.execute("SELECT display_name FROM users WHERE id = %s", (user_id,)).fetchone()
|
||||
by = name[0] if name else None
|
||||
status = str(row[3]) if row else None
|
||||
dept = row[4] if row else None
|
||||
detail = {
|
||||
"packet_id": packet_id,
|
||||
"packet_label": label,
|
||||
"status": status,
|
||||
"dept": dept,
|
||||
"archive_dir": failure.archive_dir,
|
||||
"error": failure.message,
|
||||
"by": by,
|
||||
}
|
||||
audit.append(
|
||||
conn,
|
||||
action="archive.filing_blocked",
|
||||
user_id=user_id,
|
||||
session_id=getattr(who, "session_id", None),
|
||||
subject_type="packet",
|
||||
subject_id=str(packet_id),
|
||||
packet_id=int(packet_id),
|
||||
detail=detail,
|
||||
ip=getattr(who, "ip", None),
|
||||
)
|
||||
where = f"{status or 'where it was'}{' in ' + dept if dept else ''}"
|
||||
raise_event(
|
||||
conn,
|
||||
EVENT,
|
||||
subject=FILING_SUBJECT.format(packet_id=packet_id),
|
||||
title=f"{FILING_TITLE}: {label}",
|
||||
body=(
|
||||
f"{by or 'Someone'} tried to file {label}, but the packet could not be copied to the archive,"
|
||||
f" so it was not filed. It stays {where} and can be filed again once the archive works.\n\n"
|
||||
f"Archive folder: {failure.archive_dir or '(none)'}\nError: {failure.message}"
|
||||
),
|
||||
detail=detail,
|
||||
severity="critical",
|
||||
condition=True,
|
||||
)
|
||||
except Exception:
|
||||
log.exception("archive copy request for packet %s not made; the scheduled run picks it up", packet_id)
|
||||
return False
|
||||
log.exception("the blocked filing of packet %s was not recorded", packet_id)
|
||||
|
||||
|
||||
def reset_backoff(conn: Any) -> int:
|
||||
|
||||
@@ -833,7 +833,8 @@ def apply_transition(w: Write) -> None:
|
||||
def _file(w: Write) -> None:
|
||||
"""FILE: the data book date, the seal, and the data book built and stored on the seal, all in this transaction.
|
||||
|
||||
A data book that cannot be built answers 500 databook_failed and the whole filing rolls back."""
|
||||
A data book that cannot be built answers 500 databook_failed and the whole filing rolls back. So does an archive
|
||||
copy that fails (archive.FilingCopyFailed, answered 503 archive_failed by the transition endpoint)."""
|
||||
key = _seal_key(w)
|
||||
_set_packet(w, "databook_at = now()")
|
||||
row = packets.load_row(w.conn, w.id)
|
||||
@@ -850,11 +851,13 @@ def _file(w: Write) -> None:
|
||||
w.log_also("Data book filed", f"{pages} pages")
|
||||
w.detail["seal_id"] = int(seal_id)
|
||||
w.detail["databook_pages"] = pages
|
||||
# The archive copy runs in the worker after this commits. A failed copy never blocks filing.
|
||||
# The archive copy must succeed before the packet counts as filed (Gitea issue 57). A failure raises
|
||||
# archive.FilingCopyFailed and the whole filing rolls back; the transition endpoint records it for Admin.
|
||||
from . import archive
|
||||
|
||||
if archive.on_filed(w.conn, _settings(w), w.user_id, w.id):
|
||||
w.detail["archive_requested"] = True
|
||||
copied = archive.copy_at_filing(w.conn, _settings(w), w.id, int(seal_id))
|
||||
if copied is not None:
|
||||
w.detail["archive_folder"] = copied["folder"]
|
||||
|
||||
|
||||
def _file_databook(w: Write, row: Mapping[str, Any], seal: dict[str, Any], seal_id: int) -> int:
|
||||
|
||||
@@ -75,8 +75,8 @@ _TYPES = (
|
||||
EventType(
|
||||
"archive_failed",
|
||||
"Archive copy failing",
|
||||
"A filed packet could not be copied to the archive folder after three tries, or the archive folder check"
|
||||
" failed.",
|
||||
"A packet could not be filed because its archive copy failed, an earlier filing could not be copied after"
|
||||
" three tries, or the archive folder check failed.",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
"""The archive copy (Gitea issue 7), every 15 minutes and on request (FILE asks for a run when it commits).
|
||||
"""The archive copy (Gitea issue 7) for seals filed while the archive was off, every 15 minutes and on request.
|
||||
|
||||
Off when ARCHIVE_ROOT is unset, and idle until the Admin saves a folder. Each run checks the folder (the probe), queues
|
||||
every unbroken seal with a data book that has no `archive_copies` row (so turning the archive on later copies earlier
|
||||
filings too), then copies the pending rows and the failed rows that are due. A failed copy waits 5, 15, 60 minutes,
|
||||
then an hour between tries. The `archive_failed` condition (subject `archive`) is raised when a copy has failed three
|
||||
times or the folder check fails, and cleared when no copy is failed. A failed copy never touches the packet.
|
||||
FILE copies its own packet in the filing transaction and refuses to file when the copy fails (Gitea issue 57; see
|
||||
domain/archive.py). This job catches up the rest: off when ARCHIVE_ROOT is unset, and idle until the Admin saves a
|
||||
folder. Each run checks the folder (the probe), queues every unbroken seal with a data book that has no
|
||||
`archive_copies` row (so turning the archive on later copies earlier filings too), then copies the pending rows and
|
||||
the failed rows that are due. A failed copy waits 5, 15, 60 minutes, then an hour between tries. The `archive_failed`
|
||||
condition (subject `archive`) is raised when a copy has failed three times or the folder check fails, and cleared when
|
||||
no copy is failed. These packets are already filed, so a failed copy here never touches the packet.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -72,7 +72,7 @@ test.describe.serial("archive", () => {
|
||||
expect((await packet(clara, p.id)).seal).toBeTruthy();
|
||||
await closeAll(clara, wade, wes, andy, quinn);
|
||||
|
||||
// FILE asks the worker for a copy at once. The Admin section shows it done.
|
||||
// FILE made the copy in the filing transaction (Gitea issue 57). The Admin section shows it done.
|
||||
const ann = await admin(browser);
|
||||
await expect
|
||||
.poll(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""The archive location (Gitea issue 7): the path rules, the check, the File locations API, the archive-copy job
|
||||
(copy, idempotence, failure and backoff, the Admin condition, old seals queued) and filing with the archive off or
|
||||
broken. Synthetic data only; the archive root is a temp folder."""
|
||||
"""The archive location (Gitea issue 7): the path rules, the check, the File locations API, the copy FILE makes in
|
||||
its own transaction, filing refused when that copy fails (Gitea issue 57), and the archive-copy job (idempotence,
|
||||
failure and backoff, the Admin condition, old seals queued). Synthetic data only; the archive root is a temp
|
||||
folder."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -13,7 +14,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from closehelpers import file_packet, filed, q1, qall, rich_complete, vclose
|
||||
from closehelpers import at_complete, file_packet, filed, q1, qall, rich_complete, vclose
|
||||
from fakedir import FakeDirectory
|
||||
from pshelpers import make_settings, needs_pg
|
||||
from shophelpers import Shop, ok, open_shop, refused, run
|
||||
@@ -283,12 +284,9 @@ def test_copy_job(shop: Shop, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
run(s, "UPDATE bom_shorts SET closed_at = now() WHERE packet_id = %s", (pid,))
|
||||
out = ok(file_packet(shop, vclose(shop, pkt)))
|
||||
assert out["status"] == "filed"
|
||||
assert q1(s, "SELECT count(*) FROM job_requests WHERE job = 'archive-copy' AND claimed_at IS NULL")[0] == 1
|
||||
# FILE made the copy itself, in the filing transaction: nothing waits for the worker
|
||||
assert q1(s, "SELECT count(*) FROM job_requests WHERE job = 'archive-copy'")[0] == 0
|
||||
seal_id, book_id = q1(s, "SELECT id, databook_asset_id FROM packet_seals WHERE packet_id = %s", (pid,))
|
||||
|
||||
row = run_copy(s)
|
||||
assert row["status"] == "ok", row
|
||||
assert row["trigger"] == "request" and row["counts"]["queued"] == 1 and row["counts"]["copied"] == 1
|
||||
((sid, status, attempts, err, nxt, folder),) = copies(s)
|
||||
assert (sid, status, attempts, err, nxt) == (seal_id, "done", 1, None, None)
|
||||
project, unit = q1(
|
||||
@@ -315,11 +313,13 @@ def test_copy_job(shop: Shop, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
assert q1(s, "SELECT manifest_sha256 FROM archive_copies")[0] == sha(where / "manifest.json")
|
||||
(done,) = audits(s, "archive.copied")
|
||||
assert done["seal_id"] == seal_id and done["folder"] == folder and done["files"] == len(names)
|
||||
assert events == [("resolve", "archive_failed", {"subject": "archive", "title": "Archive copy failing"})]
|
||||
assert done["source"] == "filing"
|
||||
|
||||
# idempotent: a second run copies nothing; a row set back to pending finds the identical files and is done
|
||||
# idempotent: a run copies nothing; a row set back to pending finds the identical files and is done
|
||||
row = run_copy(s)
|
||||
assert row["status"] == "ok" and row["counts"]["queued"] == 0 and row["counts"]["copied"] == 0, row
|
||||
assert events == [("resolve", "archive_failed", {"subject": "archive", "title": "Archive copy failing"})]
|
||||
events.clear()
|
||||
assert run_copy(s)["counts"]["copied"] == 0
|
||||
run(s, "UPDATE archive_copies SET status = 'pending'")
|
||||
before = {p.name: p.stat().st_mtime_ns for p in where.iterdir()}
|
||||
assert run_copy(s)["counts"]["copied"] == 1
|
||||
@@ -390,50 +390,89 @@ def test_old_seals_are_queued_when_the_archive_is_turned_on(off_shop: Shop, tmp_
|
||||
assert copies(s)[0][:2] == (seal, "done")
|
||||
|
||||
|
||||
def open_filing_events(settings: Settings, pid: int) -> list[Any]:
|
||||
return qall(
|
||||
settings,
|
||||
"SELECT title, body, detail, severity, raised_count FROM notification_events"
|
||||
" WHERE event_type = 'archive_failed' AND subject = %s AND resolved_at IS NULL",
|
||||
(f"filing:{pid}",),
|
||||
)
|
||||
|
||||
|
||||
@needs_pg
|
||||
def test_filing_succeeds_with_the_archive_broken(shop: Shop, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def test_filing_is_refused_when_the_archive_folder_is_gone(shop: Shop) -> None:
|
||||
s = shop.settings
|
||||
root = Path(str(s.archive_root))
|
||||
events: list[tuple[str, str, dict[str, Any]]] = []
|
||||
monkeypatch.setattr(jobs, "raise_event", lambda conn, t, **kw: events.append(("raise", t, kw)))
|
||||
monkeypatch.setattr(jobs, "resolve_event", lambda conn, t, **kw: events.append(("resolve", t, kw)))
|
||||
save_dir(s, "gone") # saved, then the folder went away
|
||||
pkt = filed(shop)
|
||||
assert pkt["status"] == "filed"
|
||||
assert q1(s, "SELECT count(*) FROM job_requests WHERE job = 'archive-copy'")[0] == 1
|
||||
pkt = vclose(shop, at_complete(shop))
|
||||
pid = pkt["id"]
|
||||
|
||||
row = run_copy(s)
|
||||
assert row["status"] == "degraded" and "does not exist" in row["counts"]["check"]
|
||||
body = refused(file_packet(shop, pkt), 503, "archive_failed")
|
||||
assert body["detail"].startswith("The packet could not be copied to the archive: The archive folder check failed:")
|
||||
assert "does not exist" in body["detail"] and "Nothing was filed, and Admin has been told." in body["detail"]
|
||||
# the whole filing rolled back: still complete, no seal, no data book, no copy row
|
||||
view = ok(shop("clerk.clara").get(f"/api/packets/{pid}"))
|
||||
assert view["status"] == "complete" and view["seal"] is None
|
||||
assert q1(s, "SELECT count(*) FROM packet_seals WHERE packet_id = %s", (pid,))[0] == 0
|
||||
assert q1(s, "SELECT databook_at FROM packets WHERE id = %s", (pid,))[0] is None
|
||||
assert copies(s) == []
|
||||
((kind, etype, kw),) = events
|
||||
assert (kind, etype, kw["subject"], kw["condition"]) == ("raise", "archive_failed", "archive", True)
|
||||
checks = audits(s, "archive.checked")
|
||||
assert [(c["source"], c["ok"]) for c in checks] == [("job", False)]
|
||||
run_copy(s)
|
||||
assert len(audits(s, "archive.checked")) == 1 # the same result again is not audited again
|
||||
# recorded on its own: the audit row and one open Admin condition with the details
|
||||
(blocked,) = audits(s, "archive.filing_blocked")
|
||||
assert blocked["packet_id"] == pid and blocked["archive_dir"] == "gone" and blocked["by"] == "Clara Clerk"
|
||||
assert blocked["status"] == "complete" and "does not exist" in blocked["error"]
|
||||
((title, text, detail, severity, raised),) = open_filing_events(s, pid)
|
||||
assert title.startswith("Filing blocked: archive copy failed: ") and blocked["packet_label"] in title
|
||||
assert "Clara Clerk tried to file" in text and "Archive folder: gone" in text and "does not exist" in text
|
||||
assert severity == "critical" and raised == 1 and detail["packet_id"] == pid
|
||||
|
||||
# the folder comes back: the copy is made and the condition cleared
|
||||
# a second try fails the same way: the open condition only counts it
|
||||
refused(file_packet(shop, pkt), 503, "archive_failed")
|
||||
assert open_filing_events(s, pid)[0][4] == 2 and len(audits(s, "archive.filing_blocked")) == 2
|
||||
|
||||
# the folder comes back: the filing goes through, the copy is done and the condition is cleared
|
||||
(root / "gone").mkdir()
|
||||
events.clear()
|
||||
row = run_copy(s)
|
||||
assert row["status"] == "ok" and row["counts"]["copied"] == 1
|
||||
assert events == [("resolve", "archive_failed", {"subject": "archive", "title": "Archive copy failing"})]
|
||||
assert [(c["source"], c["ok"]) for c in audits(s, "archive.checked")] == [("job", False), ("job", True)]
|
||||
out = ok(file_packet(shop, pkt))
|
||||
assert out["status"] == "filed"
|
||||
((_, status, attempts, err, _, folder),) = copies(s)
|
||||
assert (status, attempts, err) == ("done", 1, None) and folder.startswith("gone/")
|
||||
assert open_filing_events(s, pid) == []
|
||||
cleared = q1(
|
||||
s,
|
||||
"SELECT title, body FROM notification_events WHERE event_type = 'archive_failed' AND subject = %s"
|
||||
" AND recovery_of IS NOT NULL",
|
||||
(f"filing:{pid}",),
|
||||
)
|
||||
assert cleared[0].startswith("Cleared: Filing blocked") and cleared[1] == "The packet was filed and copied."
|
||||
|
||||
|
||||
@needs_pg
|
||||
def test_filing_succeeds_when_the_request_fails(shop: Shop, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def test_filing_is_refused_when_the_copy_fails(shop: Shop, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
s = shop.settings
|
||||
save_dir(s)
|
||||
|
||||
def boom(*a: Any, **k: Any) -> None:
|
||||
raise RuntimeError("no requests today")
|
||||
def boom(*a: Any, **k: Any) -> Any:
|
||||
raise archive.CopyError("A different file is already at Data_Book.pdf in the archive; it was not overwritten.")
|
||||
|
||||
monkeypatch.setattr(archive, "request_copy", boom)
|
||||
monkeypatch.setattr(archive, "copy_seal", boom)
|
||||
pkt = vclose(shop, at_complete(shop))
|
||||
body = refused(file_packet(shop, pkt), 503, "archive_failed")
|
||||
assert "it was not overwritten. Nothing was filed" in body["detail"]
|
||||
assert q1(s, "SELECT status FROM packets WHERE id = %s", (pkt["id"],))[0] == "complete"
|
||||
assert q1(s, "SELECT count(*) FROM packet_seals WHERE packet_id = %s", (pkt["id"],))[0] == 0
|
||||
((_, text, detail, _, _),) = open_filing_events(s, pkt["id"])
|
||||
assert "not overwritten" in text and detail["archive_dir"] == "packets"
|
||||
|
||||
# anything unexpected blocks the filing the same way
|
||||
monkeypatch.setattr(archive, "copy_seal", lambda *a, **k: 1 / 0)
|
||||
body = refused(file_packet(shop, pkt), 503, "archive_failed")
|
||||
assert "ZeroDivisionError" in body["detail"]
|
||||
|
||||
|
||||
@needs_pg
|
||||
def test_filing_with_no_folder_saved_copies_nothing(shop: Shop) -> None:
|
||||
# ARCHIVE_ROOT is set but the Admin has not saved a folder: the archive is not in use yet, so filing goes ahead
|
||||
pkt = filed(shop)
|
||||
assert pkt["status"] == "filed"
|
||||
assert q1(s, "SELECT count(*) FROM job_requests WHERE job = 'archive-copy'")[0] == 0
|
||||
assert run_copy(s)["counts"]["copied"] == 1 # the scheduled run picks it up
|
||||
assert pkt["status"] == "filed" and copies(shop.settings) == []
|
||||
|
||||
|
||||
@needs_pg
|
||||
|
||||
Reference in New Issue
Block a user