Make rollback, first upgrades and pin changes work as tested on Windows
Found in an end-to-end test on a Windows 11 VM against a self-signed server.
- Rollback: a 1.20 agent re-enrolls within a second of an un-enroll, so an
older agent pushed afterwards arrived on an enrolled PC and was refused.
Revoke now takes {"blockreenroll": true, "reason": ...}; a blocked PC is
never issued a credential and keeps working on the shared key until
POST .../assets/<assetid>/allow-reenroll. The Enrollment page offers the
block when un-enrolling and shows blocked PCs with an allow button.
- First upgrade of old agents: agents before 1.20 check the server
certificate when downloading the upgrade, so a store wave failed on PCs that
do not trust a self-signed server while the Migration tab said ready. A
failed download now marks the PC blocked with "server certificate not
trusted by this PC". New Install-ShopDBServerCertificate.ps1 (GE-Enforce,
SYSTEM) adds a non-CA server certificate to the trusted roots after checking
its thumbprint; the server certificate page offers a .cer download and the
SHA-1 thumbprint.
- The generated installer writes ServerPins (with an optional next pin for a
planned rotation), can leave the shared key out so the PC waits for
approval, never touches an existing credential, and skips the trusted-root
import for a CA certificate.
- The agent re-reads server pins about every 60 seconds and recycles pooled
connections within 2 minutes, so a pin change needs no restart.
- An alert can only be acknowledged by a PC it targets.
- A wave no longer counts a PC as failed after a later wave upgraded it.
- The scripts report a pin mismatch only when the pin really is missing, and
otherwise log the real error.
- Runbook: trust step before the first wave, rollback order, pin re-read,
verifying the pin against the live URL; agent contract updated.
This commit is contained in:
@@ -76,6 +76,8 @@ Then widen by hostname list (or machine number list) in batches:
|
||||
```
|
||||
|
||||
The script compares before it writes, so `Always` costs nothing per cycle.
|
||||
A running agent picks up a changed `ServerPins` within about 60 seconds; no
|
||||
service restart is needed.
|
||||
|
||||
**Gate:** on the pilot, `HKLM\SOFTWARE\GE\ShopDB` and
|
||||
`HKLM\SOFTWARE\WOW6432Node\GE\ShopDB` both hold `ServerPins` (REG_MULTI_SZ)
|
||||
@@ -83,9 +85,56 @@ equal to the pin on the settings page, and the next `report-asset-*.log` under
|
||||
`C:\Logs\Shopfloor` says `server pin check on (1 pin(s))` followed by a
|
||||
RESPONSE line.
|
||||
|
||||
**Rotating the server key later:** put the old AND the new pin in `-Pins`
|
||||
(`"<oldpin>,<newpin>"`), wait until every PC has both, switch the certificate,
|
||||
then drop the old pin.
|
||||
**Rotating the server key later:** any server-side change that gives the
|
||||
certificate a new key (a regenerated or re-issued certificate) changes the pin.
|
||||
Add the NEXT pin everywhere first: put the old AND the new pin in `-Pins`
|
||||
(`"<oldpin>,<newpin>"`), wait until every PC has both, only then switch the
|
||||
certificate on the server, then drop the old pin. Switching the server first
|
||||
cuts off every pinned PC, including its ability to receive the fixed pin list
|
||||
through anything that talks to ShopDB. A renewal that keeps the same key keeps
|
||||
the same pin. PCs trusted through the Root store (below) also need the new
|
||||
certificate imported before the switch.
|
||||
|
||||
### Trust the server certificate on PCs with agents older than 1.20
|
||||
|
||||
Required before ANY store wave. Agents older than 1.20 know nothing about
|
||||
`ServerPins`: when they download the upgrade package they check the normal
|
||||
Windows certificate chain. With a self-signed server certificate that check
|
||||
fails and the upgrade never lands.
|
||||
|
||||
- PCs installed by the server's generated installer (Build site agent) already
|
||||
trust the certificate.
|
||||
- PCs installed from the share do NOT, and need this entry.
|
||||
|
||||
Get the certificate and its thumbprint from the server:
|
||||
|
||||
- Settings -> Endpoints -> Server Certificate -> download, or
|
||||
`GET /api/endpoints/settings/server-cert/download` (a `.cer`).
|
||||
- The thumbprint is `sha1thumbprint` on `GET /api/endpoints/settings/server-cert`.
|
||||
|
||||
Stage the `.cer` and `plugins/endpoints/client/Install-ShopDBServerCertificate.ps1`
|
||||
on the share and add a PS1 entry with the SAME targeting as the pin entry:
|
||||
|
||||
```json
|
||||
{
|
||||
"_comment": "Trust the ShopDB server certificate for pre-1.20 agent upgrades.",
|
||||
"Name": "ShopDB server certificate",
|
||||
"Type": "PS1",
|
||||
"Script": "scripts/Install-ShopDBServerCertificate.ps1",
|
||||
"Args": "-CertPath <share>\\scripts\\shopdb-server.cer -Thumbprint <sha1thumbprint>",
|
||||
"DetectionMethod": "Always",
|
||||
"TargetMachineNumbers": ["<pilotbay>"]
|
||||
}
|
||||
```
|
||||
|
||||
The script refuses a file whose thumbprint differs, refuses a CA certificate
|
||||
(one in Root would trust everything it signs), imports into
|
||||
`LocalMachine\Root` only when the certificate is not already there, and removes
|
||||
nothing. A non-CA leaf in Root trusts only this one server.
|
||||
|
||||
If a wave reaches a PC that does not trust the certificate, the download fails
|
||||
and the Migration tab marks the PC **Blocked** with `server certificate not
|
||||
trusted by this PC`. Fix trust with the entry above, then queue the PC again.
|
||||
|
||||
## Phase 2: re-stamp the share's agent entries
|
||||
|
||||
@@ -158,7 +207,10 @@ Repeat until **Ready** is 0.
|
||||
`no serial on record` / `duplicate hostname`: fix the computer record.
|
||||
`commands locked`: release the endpoint lock or policy hold. `no agent package
|
||||
in store`: upload 1.20. `agent too old for remote upgrade`: move it via the
|
||||
share (targeted agent entry as in phase 2).
|
||||
share (targeted agent entry as in phase 2). `server certificate not trusted
|
||||
by this PC`: add the PC to the certificate entry (phase 1), wait a cycle,
|
||||
queue it again. `reenroll blocked`: the PC was rolled back on purpose; clear
|
||||
it with allow-reenroll (see Rollback) only when moving it forward again.
|
||||
- **Not reporting:** the PC has no working agent. It can only move via the
|
||||
share channel: add its hostname to a targeted 1.20 agent entry. It shows as
|
||||
Pending or Enrolled once the agent runs.
|
||||
@@ -208,22 +260,49 @@ read it until this table is done.
|
||||
|
||||
## Rollback
|
||||
|
||||
The order matters. A 1.20 agent that is plainly un-enrolled enrolls again
|
||||
within about a second, before any downgrade can land, and the old agent it is
|
||||
then replaced with only knows the shared key, which the server refuses for an
|
||||
enrolled PC. Block re-enrollment FIRST.
|
||||
|
||||
**One PC:**
|
||||
|
||||
1. Make sure **Shared key allowed** is on.
|
||||
2. Credentials tab -> **Un-enroll** the PC. Its credential stops working
|
||||
immediately; the server admits the shared key for it again.
|
||||
3. Push the previous agent: Upload New/Legacy Agents -> make the legacy build
|
||||
current and upgrade that PC, or re-stamp its share entry to the old hash.
|
||||
4. Remove the PC from the pin entry's targets only if pinning itself is the
|
||||
2. Credentials tab -> **Un-enroll** the PC with **Keep this PC from enrolling
|
||||
again** ticked. API equivalent:
|
||||
`POST /api/endpoints/enrollment/assets/<assetid>/revoke`
|
||||
`{"blockreenroll": true, "reason": "rollback"}`.
|
||||
Its credential stops working immediately, the server admits the shared key
|
||||
for it again, and its enroll attempts stay pending with `reenroll blocked`.
|
||||
3. Make the previous agent current in the store (Upload New/Legacy Agents) and
|
||||
queue the upgrade for that PC, or re-stamp its share entry to the old hash.
|
||||
4. Verify: the Fleet row reports the old agent version, and the reporter logs
|
||||
show `RESPONSE (shared key)`.
|
||||
5. Remove the PC from the pin entry's targets only if pinning itself is the
|
||||
problem (`Set-ShopDBServerPins.ps1 -Clear` removes the value).
|
||||
|
||||
A 1.20 agent that is un-enrolled but NOT downgraded simply enrolls again on its
|
||||
next cycle; that is expected.
|
||||
6. Later, when moving the PC forward again, lift the block:
|
||||
`POST /api/endpoints/enrollment/assets/<assetid>/allow-reenroll` (or the
|
||||
same action in the UI). The PC enrolls on its next attempt once it runs
|
||||
1.20. Until then the Migration tab shows it blocked with `reenroll blocked`.
|
||||
|
||||
**Whole fleet:** turn **Shared key allowed** back on first. Every PC that still
|
||||
holds the shared key reports again immediately. Then un-enroll and downgrade
|
||||
PCs as above, in waves.
|
||||
holds the shared key reports again immediately. Then revoke with the block and
|
||||
downgrade PCs as above, in waves.
|
||||
|
||||
## Verified on Windows
|
||||
|
||||
A full pass on a Windows test PC against a test server proved:
|
||||
|
||||
- the agent enrolls on its own;
|
||||
- the shared key is refused for the enrolled PC (`SHARED_KEY_REFUSED`);
|
||||
- a request naming another asset is refused (`ASSET_MISMATCH`);
|
||||
- server pins are enforced;
|
||||
- the tray pipe works for a standard (non-admin) user;
|
||||
- alert acknowledge works;
|
||||
- both SYSTEM reporters send with the per-PC credential;
|
||||
- revoke is followed by re-enrollment;
|
||||
- rollback to the previous agent works;
|
||||
- with the shared key switched off, enrolled PCs keep working.
|
||||
|
||||
## Quick reference
|
||||
|
||||
|
||||
Reference in New Issue
Block a user