shopdb-flask/plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md

# Per-PC credentials rollout

Runbook for moving a site's endpoint fleet from the shared collector key to
per-PC credentials. The wire contract is `AGENT-CONTRACT.md` section 9.

Placeholders: `<server>` is the ShopDB base URL, `<share>` the GE-Enforce share
root, `<pin>` a server key pin, `<pilotbay>` / `<pilothost>` the pilot PC.

## What changes

- Every endpoint gets its own secret (`sdbe_...`), stored in
  `%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
  LocalMachine, readable by SYSTEM and Administrators only.
- The server knows which asset is calling from the secret. A PC can no longer
  report as another PC.
- The tray holds no secret. It talks to the SYSTEM agent over a local pipe.
- The agent, the upgrade script and the SYSTEM reporters
  (`Report-AssetToShopDB.ps1`, `Report-PrintersToShopDB.ps1`) pin the server's
  public key when `ServerPins` is set.
- The shared key keeps working for PCs that have NOT enrolled, until an admin
  turns it off. An enrolled PC is always refused the shared key
  (`401 SHARED_KEY_REFUSED`).

## Where to look in the app

- **Settings -> Endpoints -> Enrollment**: Migration tab (readiness, waves) and
  Credentials tab (enrolled PCs, pending requests, un-enroll, allow
  re-enrollment, the shared key and auto-enroll switches).
- **Settings -> Endpoints** (the agent key page): the shared key PCs use until
  they enroll, with an enrolled-of-total summary and the pending request count.
  Regenerate and revoke there say how many PCs are not enrolled and would be
  affected.
- **Settings -> Endpoints -> Server Certificate**: the certificate file's pin
  and thumbprint, and **What PCs receive**, a live check of the certificate the
  site's base URL (`site_base_url`) actually presents. When the two differ, pin
  the presented one.
- **Fleet** (sidebar, under Assets): the **Credential** column (Enrolled,
  Pending, Re-enroll blocked, Shared key (not enrolled), Not reporting) and a
  filter on it.
- **PC detail**: the Endpoint status card shows the PC's credential state (with
  prefix, issued, last used, enrolled via and block reason for
  `endpoints.fleet` holders). Admins get **Revoke credential** (with **Keep this
  PC from enrolling again**) and **Allow re-enrollment** beside the status chip.
- **Dashboard**: "N PCs waiting for enrollment approval" (warning, links to
  Enrollment) and "PCs blocked from re-enrolling over 7 days" (info, counted
  from when the credential was revoked).

## Before you start

1. Server deployed with the endpoints migration `endpoints0023credentials`
   applied (`flask plugin upgrade-all`).
2. Settings -> Endpoints -> Server Certificate shows a **Public key pin (SPKI
   SHA-256)**. Cross-check it from a PC:
   `Get-ShopDBServerPin.ps1 -Url https://<server>`. The two must match.
3. Agent 1.20 (and tray 1.7) uploaded as the current release under Settings ->
   Endpoints -> Upload New/Legacy Agents. Keep the previous release staged as a
   legacy build: it is the rollback.
4. Settings -> Endpoints -> Enrollment -> Credentials: **Shared key allowed**
   on, **Auto-enroll** on (both are the defaults).
5. The GE-Enforce manifest scope that will carry the entries below is at
   `Version` 2.2 or later, and the fleet's GE-Enforce client library
   understands `TargetHostnames` / `TargetMachineNumbers`. An older manifest or
   library ignores those lists and applies the entry to EVERY PC in scope. The
   same happens with an empty list: never publish a targeted entry with a blank
   target.

## Phase 1: pins on the PCs

Pins must be on a PC BEFORE agent 1.20 arrives there. A 1.20 agent with no pin
falls back to plain Windows validation, which a self-signed or internal-CA
server may fail; a pin set after the fact is not a rollout order you want to
debug per bay.

Stage `plugins/endpoints/client/Set-ShopDBServerPins.ps1` on the share and add
a PS1 entry. Pilot first:

```json
{
  "_comment": "Server key pin for per-PC credentials. Pilot only.",
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetMachineNumbers": ["<pilotbay>"]
}
```

Then widen by hostname list (or machine number list) in batches:

```json
{
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetHostnames": ["<host1>", "<host2>", "<host3>"]
}
```

The script compares before it writes, so `Always` costs nothing per cycle.
A running agent picks up a changed `ServerPins` within about 60 seconds; no
service restart is needed.

**The registry wins.** The installer and the store upgrade also write
`serverpins` into appsettings.json. When `ServerPins` exists in the registry
(either view), even empty, the agent uses only the registry and ignores the
appsettings.json pins; only a PC with no `ServerPins` value falls back to them.
The agent log says which source is in effect (`Server certificate trust from
registry` / `from appsettings.json`, and `Server pins changed, source ...`).

- `-Pins <pin>[,<pin>]`: write exactly this set.
- `-None`: write an EMPTY value. No pins, normal Windows certificate checks,
  appsettings.json pins ignored.
- `-Clear`: DELETE the value. The appsettings.json pins apply again (if any).

**Gate:** on the pilot, `HKLM\SOFTWARE\GE\ShopDB` and
`HKLM\SOFTWARE\WOW6432Node\GE\ShopDB` both hold `ServerPins` (REG_MULTI_SZ)
equal to the pin on the settings page, and the next `report-asset-*.log` under
`C:\Logs\Shopfloor` says `server pin check on (1 pin(s))` followed by a
RESPONSE line.

**Rotating the server key later:** any server-side change that gives the
certificate a new key (a regenerated or re-issued certificate) changes the pin.
Add the NEXT pin everywhere first: put the old AND the new pin in `-Pins`
(`"<oldpin>,<newpin>"`), wait until every PC has both, only then switch the
certificate on the server, then retire the old pin by writing ONLY the new pin
to the registry (`-Pins <newpin>`). Do not use `-Clear` for this: deleting the
value brings back whatever pins appsettings.json carries, which may be the old
one. Switching the server first
cuts off every pinned PC, including its ability to receive the fixed pin list
through anything that talks to ShopDB. A renewal that keeps the same key keeps
the same pin. PCs trusted through the Root store (below) also need the new
certificate imported before the switch.

### Trust the server certificate on PCs with agents older than 1.20

Required before ANY store wave. Agents older than 1.20 know nothing about
`ServerPins`: when they download the upgrade package they check the normal
Windows certificate chain. With a self-signed server certificate that check
fails and the upgrade never lands.

- PCs installed by the server's generated installer (Build site agent) already
  trust the certificate.
- PCs installed from the share do NOT, and need this entry.

Get the certificate and its thumbprint from the server:

- Settings -> Endpoints -> Server Certificate -> download, or
  `GET /api/endpoints/settings/server-cert/download` (a `.cer`).
- The thumbprint is `sha1thumbprint` on `GET /api/endpoints/settings/server-cert`.

Stage the `.cer` and `plugins/endpoints/client/Install-ShopDBServerCertificate.ps1`
on the share and add a PS1 entry with the SAME targeting as the pin entry:

```json
{
  "_comment": "Trust the ShopDB server certificate for pre-1.20 agent upgrades.",
  "Name": "ShopDB server certificate",
  "Type": "PS1",
  "Script": "scripts/Install-ShopDBServerCertificate.ps1",
  "Args": "-CertPath <share>\\scripts\\shopdb-server.cer -Thumbprint <sha1thumbprint>",
  "DetectionMethod": "Always",
  "TargetMachineNumbers": ["<pilotbay>"]
}
```

The script refuses a file whose thumbprint differs, refuses a CA certificate
(one in Root would trust everything it signs), imports into
`LocalMachine\Root` only when the certificate is not already there, and removes
nothing. A non-CA leaf in Root trusts only this one server.

If a wave reaches a PC that does not trust the certificate, the download fails
and the Migration tab marks the PC **Blocked** with `server certificate not
trusted by this PC`. The server only knows about trust from that failure, so
the blocker stays until the PC is queued again. Fix trust with the entry above,
then in the Migration tab tick the PC (a PC whose only blocker is this one keeps
a checkbox and shows "Retry after fixing trust") and click **Upgrade selected**.
**Upgrade next 25** never picks it, and a PC with any other blocker as well
stays unselectable until that blocker is fixed.

## Phase 2: re-stamp the share's agent entries

PCs that got the agent from the share (collections manifest entries with a
`Hash` detection on `shopdbagent.exe`) are held to that hash by GE-Enforce. If
a store upgrade wave puts 1.20 on such a PC while the entry still carries the
1.19 hash, GE-Enforce reinstalls 1.19 on its next cycle and the PC flips back.

Before any wave reaches a share-installed PC, either:

- re-stamp those entries to the 1.20 payload and its `DetectionValue` SHA-256
  (and keep the `ProductCode`/version guards the entries already have), or
- remove the entries for the PCs the wave will cover.

Agent entries stay targeted. Illustrative shape for the pilot (keep your
existing entry's type, installer and guards; what matters is the hash and a
non-empty target):

```json
{
  "Name": "ShopDB endpoint agent 1.20",
  "Type": "PS1",
  "Script": "apps/shopdbagent/1.20.0/install-shopdbagent.ps1",
  "DetectionMethod": "Hash",
  "DetectionPath": "C:\\Program Files\\ShopDB\\EndpointAgent\\shopdbagent.exe",
  "DetectionValue": "<sha256 of the 1.20 shopdbagent.exe>",
  "TargetHostnames": ["<pilothost>"]
}
```

**Gate:** the manifest preview for the pilot shows the 1.20 hash, and no entry
for the same file with a blank target list exists in any scope that reaches it.

## Phase 3: pilot one PC

1. Enrollment -> Migration: find `<pilothost>`. It should be **Ready**. If it is
   **Blocked**, the Blockers column lists every reason; fix those first.
2. Tick it, **Upgrade selected**.
3. Watch the row: **Upgrading**, then **Enrolled** within a few minutes of the
   new agent starting. The agent enrolls on its own with the shared key it
   already holds.

**Gate before any wave:**

- Credentials tab shows the PC enrolled, `enrolledvia` = `auto`, and a
  `lastusedat` that keeps moving.
- The PC's Fleet row keeps reporting; commands (Collect now) still work.
- The tray shows alerts and Send Status works (it now goes through the pipe).
- `report-asset-*.log` and `report-printers-*.log` show `RESPONSE (credential)`.
- Nothing reinstalled the old agent after two GE-Enforce cycles.

## Phase 4: waves

Migration tab -> **Upgrade next 25**. The confirm dialog states how many PCs get
agent 1.20. The server re-checks readiness per PC and packages the agent once
per wave; PCs that stopped being ready are returned as skipped.

Between waves, in the Waves list:

- wait until the wave shows **Done** (every PC enrolled or failed),
- investigate every **failed** PC before the next wave,
- check nothing moved from Enrolled back to Upgrading or Ready (a share entry
  reinstalling the old agent, see phase 2).

Repeat until **Ready** is 0.

## Phase 5: stragglers and new PCs

- **Blocked:** each blocker names its fix. `offline`: bring the PC back or wait.
  `no serial on record` / `duplicate hostname`: fix the computer record.
  `commands locked`: release the endpoint lock or policy hold. `no agent package
  in store`: upload 1.20. `agent too old for remote upgrade`: move it via the
  share (targeted agent entry as in phase 2). `server certificate not trusted
  by this PC`: add the PC to the certificate entry (phase 1), wait a cycle,
  then tick it and **Upgrade selected** (phase 1 explains the retry). `reenroll blocked`: the PC was rolled back on purpose; clear
  it with allow-reenroll (see Rollback) only when moving it forward again.
- **Not reporting:** the PC has no working agent. It can only move via the
  share channel: add its hostname to a targeted 1.20 agent entry. It shows as
  Pending or Enrolled once the agent runs.
- **Pending:** the agent asked but auto-enroll could not decide (serial
  mismatch, duplicate or unknown hostname, already enrolled, or auto-enroll
  off). Credentials tab -> Requests -> **Approve**; the picker defaults to the
  matched computer, search to choose another. The PC collects its credential on
  its next retry (within 15 minutes). **Reject** anything you do not recognise.
- **Newly imaged PCs:** imaging installs 1.20 with the shared key, and the agent
  auto-enrolls on first start. A new PC can also be installed WITHOUT the
  shared key at any time, before or after cutover (the generated installer's
  "Include shared key" option off): the enroll call needs no key, so its request
  lands in Pending and it waits there until an admin approves it in the UI. An
  invalid key behaves the same (the server logs it). After the shared key is
  turned off (phase 6) that is the only path, and the steady-state path for new
  PCs.

## Installing a one-off PC

A PC nobody is going to author a GE-Enforce manifest entry for - a loaner, a
one-off office machine, a bay being rebuilt out of band - can be installed with
a single command instead of a downloaded package. **Managed bays keep using
GE-Enforce**; this does not replace the share channel, and the script itself
says so when it finds an agent already installed.

1. Settings -> Endpoints -> Build site agent -> **One-line install**. Set how
   long the token lives (hours, 1 to 168) and how many installs it may serve
   (1 to 500), then **Generate install command**.
2. Copy the command. The token is shown ONCE and cannot be recovered; generate
   a new one if it is lost.
3. On the target PC, open PowerShell **as administrator** and paste it:

```
powershell -ExecutionPolicy Bypass -c "irm https://<server>/api/endpoints/siteagent/bootstrap.ps1?t=<token> | iex"
```

The token is short-lived and only fetches the installer. It carries no shared
key and provisions nothing by itself: the package it downloads is built with
"Include shared key" off, whatever the shared-key setting is, so the PC asks to
enroll and **waits for approval**. Approve it in Enrollment -> Credentials ->
Requests, the same as any other pending PC (phase 5).

The script checks the downloaded binaries against the SHA-256 values the server
stamped into it, refuses to run unelevated, and refuses to run on a PC that
already has the agent service (set `SHOPDB_BOOTSTRAP_FORCE=1` in the same
window to override). After a successful install it waits up to two minutes for
the credential to land and prints `approved and enrolled` if an admin approves
it in that window; still pending is a normal outcome, not a failure.

Revoke a token from the same block when the install round is done. Tokens list
with their prefix, expiry and uses; they are also spent automatically
(`usecount` reaching `maxuses`) and expire on their own.

## Phase 6: cutover

**Gate:** Credentials tab **Not enrolled** is 0, or every remaining PC is one
you have decided may stop reporting. Remember that turning the shared key off
also refuses it on `/api/collector/computers` and `/api/collector/printers`, so
a PC with reporter scripts but no enrolled agent goes silent.

1. Turn **Shared key allowed** off. The confirm shows how many PCs are not
   enrolled; type `OFF`.
2. Watch Fleet for an hour. A PC that went quiet is either unenrolled (turn the
   shared key back on, fix it, try again) or failed pinning.
3. Remove the shared key from every place it lives (below).
4. Rotate the old key so any copy you missed is dead: Settings -> Endpoints ->
   **Regenerate key** (and rotate the collector token the reporters used, if it
   was a separate token). With the shared key off nothing on the floor needs
   the new key: approved enroll requests are collected without it. Store it
   the way the key page describes and keep it out of every file below. A
   fleet rollback after this point (see Rollback) needs the new key pushed to
   the PCs again.

### Every place the shared key lives

| Where | Put there by | How to clear at cutover |
|---|---|---|
| GE-Enforce manifest entry `Args` (`-ApiKey ...`) | whoever authored the entry | Remove the argument from the entry and publish. Search every scope for the key prefix. |
| `HKLM\SOFTWARE\GE\ShopDB` `CollectorKey`, both registry views | agent `install.ps1`; `Set-ShopdbCollectorKey.ps1` from `<share>\common\configs\shopdb-collector.txt` | First retire the `Set-ShopdbCollectorKey.ps1` entry (or it rewrites the value every cycle), then a targeted PS1 entry that deletes `CollectorKey` in both views. |
| `<share>\common\configs\shopdb-collector.txt` | share staging | Delete the file after the entry above is retired. |
| `appsettings.json` `apikey` in the agent install folder | agent package / hand edits | Targeted PS1 entry that sets `apikey` to an empty string. The agent never needs it once enrolled. |
| Generated `install.ps1` default `-CollectorKey` (Build site agent downloads) | Settings -> Endpoints -> Build site agent | Delete every kept copy (share, USB, work folders). Rebuild after rotation only if you still need the package. |
| Display kiosk `shopdb-key.txt` | display imaging | Confirm what the kiosk uses it for before clearing; a kiosk has no endpoint agent, so anything it reports with the shared key stops at cutover. Delete the file once that is settled. |

The agent never deletes the shared key itself: other SYSTEM scripts may still
read it until this table is done.

## Rollback

The order matters. A 1.20 agent that is plainly un-enrolled enrolls again
within about a second, before any downgrade can land, and the old agent it is
then replaced with only knows the shared key, which the server refuses for an
enrolled PC. Block re-enrollment FIRST.

**One PC:**

1. Make sure **Shared key allowed** is on.
2. Credentials tab -> **Un-enroll** the PC with **Keep this PC from enrolling
   again** ticked. API equivalent:
   `POST /api/endpoints/enrollment/assets/<assetid>/revoke`
   `{"blockreenroll": true, "reason": "rollback"}`.
   Its credential stops working immediately, the server admits the shared key
   for it again, and its enroll attempts stay pending with `reenroll blocked`.
3. Make the previous agent current in the store (Upload New/Legacy Agents) and
   queue the upgrade for that PC, or re-stamp its share entry to the old hash.
4. Verify: the Fleet row reports the old agent version, and the reporter logs
   show `RESPONSE (shared key)`.
5. Remove the PC from the pin entry's targets only if pinning itself is the
   problem. `Set-ShopDBServerPins.ps1 -None` turns pinning off outright;
   `-Clear` deletes the value, which falls back to any appsettings.json pins.
6. Later, when moving the PC forward again, lift the block:
   `POST /api/endpoints/enrollment/assets/<assetid>/allow-reenroll` (or the
   same action in the UI). The PC enrolls on its next attempt once it runs
   1.20. Until then the Migration tab shows it blocked with `reenroll blocked`.

**Whole fleet:** turn **Shared key allowed** back on first. Every PC that still
holds the shared key reports again immediately. Then revoke with the block and
downgrade PCs as above, in waves.

## Verified on Windows

A full pass on a Windows test PC against a test server proved:

- the agent enrolls on its own;
- the shared key is refused for the enrolled PC (`SHARED_KEY_REFUSED`);
- a request naming another asset is refused (`ASSET_MISMATCH`);
- server pins are enforced;
- the tray pipe works for a standard (non-admin) user;
- alert acknowledge works;
- both SYSTEM reporters send with the per-PC credential;
- revoke is followed by re-enrollment;
- rollback to the previous agent works;
- with the shared key switched off, enrolled PCs keep working.

## Quick reference

- Pin of the live server: Server Certificate settings page, or
  `Get-ShopDBServerPin.ps1 -Url https://<server>`.
- Pin of a certificate file: `Get-ShopDBServerPin.ps1 -Path <file>`.
- Agent log of enrollment: `%ProgramData%\ShopDB\EndpointAgent\logs`.
- Reporter logs: `C:\Logs\Shopfloor\report-asset-*.log`,
  `report-printers-*.log`. `SHARED_KEY_REFUSED` there means the PC is enrolled
  or the shared key is off and the reporter could not read the credential;
  `CREDENTIAL_INVALID` means the PC was un-enrolled and the agent is
  re-enrolling.