Make rollback, first upgrades and pin changes work as tested on Windows

Found in an end-to-end test on a Windows 11 VM against a self-signed server.

- Rollback: a 1.20 agent re-enrolls within a second of an un-enroll, so an
  older agent pushed afterwards arrived on an enrolled PC and was refused.
  Revoke now takes {"blockreenroll": true, "reason": ...}; a blocked PC is
  never issued a credential and keeps working on the shared key until
  POST .../assets/<assetid>/allow-reenroll. The Enrollment page offers the
  block when un-enrolling and shows blocked PCs with an allow button.
- First upgrade of old agents: agents before 1.20 check the server
  certificate when downloading the upgrade, so a store wave failed on PCs that
  do not trust a self-signed server while the Migration tab said ready. A
  failed download now marks the PC blocked with "server certificate not
  trusted by this PC". New Install-ShopDBServerCertificate.ps1 (GE-Enforce,
  SYSTEM) adds a non-CA server certificate to the trusted roots after checking
  its thumbprint; the server certificate page offers a .cer download and the
  SHA-1 thumbprint.
- The generated installer writes ServerPins (with an optional next pin for a
  planned rotation), can leave the shared key out so the PC waits for
  approval, never touches an existing credential, and skips the trusted-root
  import for a CA certificate.
- The agent re-reads server pins about every 60 seconds and recycles pooled
  connections within 2 minutes, so a pin change needs no restart.
- An alert can only be acknowledged by a PC it targets.
- A wave no longer counts a PC as failed after a later wave upgraded it.
- The scripts report a pin mismatch only when the pin really is missing, and
  otherwise log the real error.
- Runbook: trust step before the first wave, rollback order, pin re-read,
  verifying the pin against the live URL; agent contract updated.
This commit is contained in:
cproudlock
2026-09-15 13:37:37 -04:00
parent 8eee6f19d8
commit 70d5531b4d
20 changed files with 1713 additions and 151 deletions

View File

@@ -76,6 +76,8 @@ Then widen by hostname list (or machine number list) in batches:
```
The script compares before it writes, so `Always` costs nothing per cycle.
A running agent picks up a changed `ServerPins` within about 60 seconds; no
service restart is needed.
**Gate:** on the pilot, `HKLM\SOFTWARE\GE\ShopDB` and
`HKLM\SOFTWARE\WOW6432Node\GE\ShopDB` both hold `ServerPins` (REG_MULTI_SZ)
@@ -83,9 +85,56 @@ equal to the pin on the settings page, and the next `report-asset-*.log` under
`C:\Logs\Shopfloor` says `server pin check on (1 pin(s))` followed by a
RESPONSE line.
**Rotating the server key later:** put the old AND the new pin in `-Pins`
(`"<oldpin>,<newpin>"`), wait until every PC has both, switch the certificate,
then drop the old pin.
**Rotating the server key later:** any server-side change that gives the
certificate a new key (a regenerated or re-issued certificate) changes the pin.
Add the NEXT pin everywhere first: put the old AND the new pin in `-Pins`
(`"<oldpin>,<newpin>"`), wait until every PC has both, only then switch the
certificate on the server, then drop the old pin. Switching the server first
cuts off every pinned PC, including its ability to receive the fixed pin list
through anything that talks to ShopDB. A renewal that keeps the same key keeps
the same pin. PCs trusted through the Root store (below) also need the new
certificate imported before the switch.
### Trust the server certificate on PCs with agents older than 1.20
Required before ANY store wave. Agents older than 1.20 know nothing about
`ServerPins`: when they download the upgrade package they check the normal
Windows certificate chain. With a self-signed server certificate that check
fails and the upgrade never lands.
- PCs installed by the server's generated installer (Build site agent) already
trust the certificate.
- PCs installed from the share do NOT, and need this entry.
Get the certificate and its thumbprint from the server:
- Settings -> Endpoints -> Server Certificate -> download, or
`GET /api/endpoints/settings/server-cert/download` (a `.cer`).
- The thumbprint is `sha1thumbprint` on `GET /api/endpoints/settings/server-cert`.
Stage the `.cer` and `plugins/endpoints/client/Install-ShopDBServerCertificate.ps1`
on the share and add a PS1 entry with the SAME targeting as the pin entry:
```json
{
"_comment": "Trust the ShopDB server certificate for pre-1.20 agent upgrades.",
"Name": "ShopDB server certificate",
"Type": "PS1",
"Script": "scripts/Install-ShopDBServerCertificate.ps1",
"Args": "-CertPath <share>\\scripts\\shopdb-server.cer -Thumbprint <sha1thumbprint>",
"DetectionMethod": "Always",
"TargetMachineNumbers": ["<pilotbay>"]
}
```
The script refuses a file whose thumbprint differs, refuses a CA certificate
(one in Root would trust everything it signs), imports into
`LocalMachine\Root` only when the certificate is not already there, and removes
nothing. A non-CA leaf in Root trusts only this one server.
If a wave reaches a PC that does not trust the certificate, the download fails
and the Migration tab marks the PC **Blocked** with `server certificate not
trusted by this PC`. Fix trust with the entry above, then queue the PC again.
## Phase 2: re-stamp the share's agent entries
@@ -158,7 +207,10 @@ Repeat until **Ready** is 0.
`no serial on record` / `duplicate hostname`: fix the computer record.
`commands locked`: release the endpoint lock or policy hold. `no agent package
in store`: upload 1.20. `agent too old for remote upgrade`: move it via the
share (targeted agent entry as in phase 2).
share (targeted agent entry as in phase 2). `server certificate not trusted
by this PC`: add the PC to the certificate entry (phase 1), wait a cycle,
queue it again. `reenroll blocked`: the PC was rolled back on purpose; clear
it with allow-reenroll (see Rollback) only when moving it forward again.
- **Not reporting:** the PC has no working agent. It can only move via the
share channel: add its hostname to a targeted 1.20 agent entry. It shows as
Pending or Enrolled once the agent runs.
@@ -208,22 +260,49 @@ read it until this table is done.
## Rollback
The order matters. A 1.20 agent that is plainly un-enrolled enrolls again
within about a second, before any downgrade can land, and the old agent it is
then replaced with only knows the shared key, which the server refuses for an
enrolled PC. Block re-enrollment FIRST.
**One PC:**
1. Make sure **Shared key allowed** is on.
2. Credentials tab -> **Un-enroll** the PC. Its credential stops working
immediately; the server admits the shared key for it again.
3. Push the previous agent: Upload New/Legacy Agents -> make the legacy build
current and upgrade that PC, or re-stamp its share entry to the old hash.
4. Remove the PC from the pin entry's targets only if pinning itself is the
2. Credentials tab -> **Un-enroll** the PC with **Keep this PC from enrolling
again** ticked. API equivalent:
`POST /api/endpoints/enrollment/assets/<assetid>/revoke`
`{"blockreenroll": true, "reason": "rollback"}`.
Its credential stops working immediately, the server admits the shared key
for it again, and its enroll attempts stay pending with `reenroll blocked`.
3. Make the previous agent current in the store (Upload New/Legacy Agents) and
queue the upgrade for that PC, or re-stamp its share entry to the old hash.
4. Verify: the Fleet row reports the old agent version, and the reporter logs
show `RESPONSE (shared key)`.
5. Remove the PC from the pin entry's targets only if pinning itself is the
problem (`Set-ShopDBServerPins.ps1 -Clear` removes the value).
A 1.20 agent that is un-enrolled but NOT downgraded simply enrolls again on its
next cycle; that is expected.
6. Later, when moving the PC forward again, lift the block:
`POST /api/endpoints/enrollment/assets/<assetid>/allow-reenroll` (or the
same action in the UI). The PC enrolls on its next attempt once it runs
1.20. Until then the Migration tab shows it blocked with `reenroll blocked`.
**Whole fleet:** turn **Shared key allowed** back on first. Every PC that still
holds the shared key reports again immediately. Then un-enroll and downgrade
PCs as above, in waves.
holds the shared key reports again immediately. Then revoke with the block and
downgrade PCs as above, in waves.
## Verified on Windows
A full pass on a Windows test PC against a test server proved:
- the agent enrolls on its own;
- the shared key is refused for the enrolled PC (`SHARED_KEY_REFUSED`);
- a request naming another asset is refused (`ASSET_MISMATCH`);
- server pins are enforced;
- the tray pipe works for a standard (non-admin) user;
- alert acknowledge works;
- both SYSTEM reporters send with the per-PC credential;
- revoke is followed by re-enrollment;
- rollback to the previous agent works;
- with the shared key switched off, enrolled PCs keep working.
## Quick reference