shopdb-flask/plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md

Per-PC credentials rollout

Runbook for moving a site's endpoint fleet from the shared collector key to per-PC credentials. The wire contract is AGENT-CONTRACT.md section 9.

Placeholders: <server> is the ShopDB base URL, <share> the GE-Enforce share root, <pin> a server key pin, <pilotbay> / <pilothost> the pilot PC.

What changes

• Every endpoint gets its own secret (sdbe_...), stored in %ProgramData%\ShopDB\EndpointAgent\credential\credential.bin, DPAPI LocalMachine, readable by SYSTEM and Administrators only.
• The server knows which asset is calling from the secret. A PC can no longer report as another PC.
• The tray holds no secret. It talks to the SYSTEM agent over a local pipe.
• The agent, the upgrade script and the SYSTEM reporters (Report-AssetToShopDB.ps1, Report-PrintersToShopDB.ps1) pin the server's public key when ServerPins is set.
• The shared key keeps working for PCs that have NOT enrolled, until an admin turns it off. An enrolled PC is always refused the shared key (401 SHARED_KEY_REFUSED).

Before you start

1. Server deployed with the endpoints migration endpoints0023credentials applied (flask plugin upgrade-all).
2. Settings -> Endpoints -> Server Certificate shows a Public key pin (SPKI SHA-256). Cross-check it from a PC: Get-ShopDBServerPin.ps1 -Url https://<server>. The two must match.
3. Agent 1.20 (and tray 1.7) uploaded as the current release under Settings -> Endpoints -> Upload New/Legacy Agents. Keep the previous release staged as a legacy build: it is the rollback.
4. Settings -> Endpoints -> Enrollment -> Credentials: Shared key allowed on, Auto-enroll on (both are the defaults).
5. The GE-Enforce manifest scope that will carry the entries below is at Version 2.2 or later, and the fleet's GE-Enforce client library understands TargetHostnames / TargetMachineNumbers. An older manifest or library ignores those lists and applies the entry to EVERY PC in scope. The same happens with an empty list: never publish a targeted entry with a blank target.

Phase 1: pins on the PCs

Pins must be on a PC BEFORE agent 1.20 arrives there. A 1.20 agent with no pin falls back to plain Windows validation, which a self-signed or internal-CA server may fail; a pin set after the fact is not a rollout order you want to debug per bay.

Stage plugins/endpoints/client/Set-ShopDBServerPins.ps1 on the share and add a PS1 entry. Pilot first:

{
  "_comment": "Server key pin for per-PC credentials. Pilot only.",
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetMachineNumbers": ["<pilotbay>"]
}

Then widen by hostname list (or machine number list) in batches:

{
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetHostnames": ["<host1>", "<host2>", "<host3>"]
}

The script compares before it writes, so Always costs nothing per cycle.

Gate: on the pilot, HKLM\SOFTWARE\GE\ShopDB and HKLM\SOFTWARE\WOW6432Node\GE\ShopDB both hold ServerPins (REG_MULTI_SZ) equal to the pin on the settings page, and the next report-asset-*.log under C:\Logs\Shopfloor says server pin check on (1 pin(s)) followed by a RESPONSE line.

Rotating the server key later: put the old AND the new pin in -Pins ("<oldpin>,<newpin>"), wait until every PC has both, switch the certificate, then drop the old pin.

Phase 2: re-stamp the share's agent entries

PCs that got the agent from the share (collections manifest entries with a Hash detection on shopdbagent.exe) are held to that hash by GE-Enforce. If a store upgrade wave puts 1.20 on such a PC while the entry still carries the 1.19 hash, GE-Enforce reinstalls 1.19 on its next cycle and the PC flips back.

Before any wave reaches a share-installed PC, either:

• re-stamp those entries to the 1.20 payload and its DetectionValue SHA-256 (and keep the ProductCode/version guards the entries already have), or
• remove the entries for the PCs the wave will cover.

Agent entries stay targeted. Illustrative shape for the pilot (keep your existing entry's type, installer and guards; what matters is the hash and a non-empty target):

{
  "Name": "ShopDB endpoint agent 1.20",
  "Type": "PS1",
  "Script": "apps/shopdbagent/1.20.0/install-shopdbagent.ps1",
  "DetectionMethod": "Hash",
  "DetectionPath": "C:\\Program Files\\ShopDB\\EndpointAgent\\shopdbagent.exe",
  "DetectionValue": "<sha256 of the 1.20 shopdbagent.exe>",
  "TargetHostnames": ["<pilothost>"]
}

Gate: the manifest preview for the pilot shows the 1.20 hash, and no entry for the same file with a blank target list exists in any scope that reaches it.

Phase 3: pilot one PC

1. Enrollment -> Migration: find <pilothost>. It should be Ready. If it is Blocked, the Blockers column lists every reason; fix those first.
2. Tick it, Upgrade selected.
3. Watch the row: Upgrading, then Enrolled within a few minutes of the new agent starting. The agent enrolls on its own with the shared key it already holds.

Gate before any wave:

• Credentials tab shows the PC enrolled, enrolledvia = auto, and a lastusedat that keeps moving.
• The PC's Fleet row keeps reporting; commands (Collect now) still work.
• The tray shows alerts and Send Status works (it now goes through the pipe).
• report-asset-*.log and report-printers-*.log show RESPONSE (credential).
• Nothing reinstalled the old agent after two GE-Enforce cycles.

Phase 4: waves

Migration tab -> Upgrade next 25. The confirm dialog states how many PCs get agent 1.20. The server re-checks readiness per PC and packages the agent once per wave; PCs that stopped being ready are returned as skipped.

Between waves, in the Waves list:

• wait until the wave shows Done (every PC enrolled or failed),
• investigate every failed PC before the next wave,
• check nothing moved from Enrolled back to Upgrading or Ready (a share entry reinstalling the old agent, see phase 2).

Repeat until Ready is 0.

Phase 5: stragglers and new PCs

• Blocked: each blocker names its fix. offline: bring the PC back or wait. no serial on record / duplicate hostname: fix the computer record. commands locked: release the endpoint lock or policy hold. no agent package in store: upload 1.20. agent too old for remote upgrade: move it via the share (targeted agent entry as in phase 2).
• Not reporting: the PC has no working agent. It can only move via the share channel: add its hostname to a targeted 1.20 agent entry. It shows as Pending or Enrolled once the agent runs.
• Pending: the agent asked but auto-enroll could not decide (serial mismatch, duplicate or unknown hostname, already enrolled, or auto-enroll off). Credentials tab -> Requests -> Approve; the picker defaults to the matched computer, search to choose another. The PC collects its credential on its next retry (within 15 minutes). Reject anything you do not recognise.
• Newly imaged PCs: imaging installs 1.20 with the shared key, and the agent auto-enrolls on first start. After the shared key is turned off (phase 6) a new PC cannot auto-enroll: its request lands in Pending and an admin approves it in the UI. That is the steady-state path for new PCs.

Phase 6: cutover

Gate: Credentials tab Not enrolled is 0, or every remaining PC is one you have decided may stop reporting. Remember that turning the shared key off also refuses it on /api/collector/computers and /api/collector/printers, so a PC with reporter scripts but no enrolled agent goes silent.

1. Turn Shared key allowed off. The confirm shows how many PCs are not enrolled; type OFF.
2. Watch Fleet for an hour. A PC that went quiet is either unenrolled (turn the shared key back on, fix it, try again) or failed pinning.
3. Remove the shared key from every place it lives (below).
4. Rotate the old key so any copy you missed is dead: Settings -> Endpoints -> Regenerate key (and rotate the collector token the reporters used, if it was a separate token). With the shared key off nothing on the floor needs the new key: approved enroll requests are collected without it. Store it the way the key page describes and keep it out of every file below. A fleet rollback after this point (see Rollback) needs the new key pushed to the PCs again.

Every place the shared key lives

Where	Put there by	How to clear at cutover
GE-Enforce manifest entry Args (-ApiKey ...)	whoever authored the entry	Remove the argument from the entry and publish. Search every scope for the key prefix.
HKLM\SOFTWARE\GE\ShopDB CollectorKey, both registry views	agent install.ps1; Set-ShopdbCollectorKey.ps1 from <share>\common\configs\shopdb-collector.txt	First retire the Set-ShopdbCollectorKey.ps1 entry (or it rewrites the value every cycle), then a targeted PS1 entry that deletes CollectorKey in both views.
<share>\common\configs\shopdb-collector.txt	share staging	Delete the file after the entry above is retired.
appsettings.json apikey in the agent install folder	agent package / hand edits	Targeted PS1 entry that sets apikey to an empty string. The agent never needs it once enrolled.
Generated install.ps1 default -CollectorKey (Build site agent downloads)	Settings -> Endpoints -> Build site agent	Delete every kept copy (share, USB, work folders). Rebuild after rotation only if you still need the package.
Display kiosk shopdb-key.txt	display imaging	Confirm what the kiosk uses it for before clearing; a kiosk has no endpoint agent, so anything it reports with the shared key stops at cutover. Delete the file once that is settled.

The agent never deletes the shared key itself: other SYSTEM scripts may still read it until this table is done.

Rollback

One PC:

1. Make sure Shared key allowed is on.
2. Credentials tab -> Un-enroll the PC. Its credential stops working immediately; the server admits the shared key for it again.
3. Push the previous agent: Upload New/Legacy Agents -> make the legacy build current and upgrade that PC, or re-stamp its share entry to the old hash.
4. Remove the PC from the pin entry's targets only if pinning itself is the problem (Set-ShopDBServerPins.ps1 -Clear removes the value).

A 1.20 agent that is un-enrolled but NOT downgraded simply enrolls again on its next cycle; that is expected.

Whole fleet: turn Shared key allowed back on first. Every PC that still holds the shared key reports again immediately. Then un-enroll and downgrade PCs as above, in waves.

Quick reference

• Pin of the live server: Server Certificate settings page, or Get-ShopDBServerPin.ps1 -Url https://<server>.
• Pin of a certificate file: Get-ShopDBServerPin.ps1 -Path <file>.
• Agent log of enrollment: %ProgramData%\ShopDB\EndpointAgent\logs.
• Reporter logs: C:\Logs\Shopfloor\report-asset-*.log, report-printers-*.log. SHARED_KEY_REFUSED there means the PC is enrolled or the shared key is off and the reporter could not read the credential; CREDENTIAL_INVALID means the PC was un-enrolled and the agent is re-enrolling.