shopdb-flask/plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md

Per-PC credentials rollout

Runbook for moving a site's endpoint fleet from the shared collector key to per-PC credentials. The wire contract is AGENT-CONTRACT.md section 9.

Placeholders: <server> is the ShopDB base URL, <share> the GE-Enforce share root, <pin> a server key pin, <pilotbay> / <pilothost> the pilot PC.

What changes

• Every endpoint gets its own secret (sdbe_...), stored in %ProgramData%\ShopDB\EndpointAgent\credential\credential.bin, DPAPI LocalMachine, readable by SYSTEM and Administrators only.
• The server knows which asset is calling from the secret. A PC can no longer report as another PC.
• The tray holds no secret. It talks to the SYSTEM agent over a local pipe.
• The agent, the upgrade script and the SYSTEM reporters (Report-AssetToShopDB.ps1, Report-PrintersToShopDB.ps1) pin the server's public key when ServerPins is set.
• The shared key keeps working for PCs that have NOT enrolled, until an admin turns it off. An enrolled PC is always refused the shared key (401 SHARED_KEY_REFUSED).

Where to look in the app

• Settings -> Endpoints -> Enrollment: Migration tab (readiness, waves) and Credentials tab (enrolled PCs, pending requests, un-enroll, allow re-enrollment, the shared key and auto-enroll switches).
• Settings -> Endpoints (the agent key page): the shared key PCs use until they enroll, with an enrolled-of-total summary and the pending request count. Regenerate and revoke there say how many PCs are not enrolled and would be affected.
• Settings -> Endpoints -> Server Certificate: the certificate file's pin and thumbprint, and What PCs receive, a live check of the certificate the site's base URL (site_base_url) actually presents. When the two differ, pin the presented one.
• Fleet (sidebar, under Assets): the Credential column (Enrolled, Pending, Re-enroll blocked, Shared key (not enrolled), Not reporting) and a filter on it.
• PC detail: the Endpoint status card shows the PC's credential state (with prefix, issued, last used, enrolled via and block reason for endpoints.fleet holders). Admins get Revoke credential (with Keep this PC from enrolling again) and Allow re-enrollment beside the status chip.
• Dashboard: "N PCs waiting for enrollment approval" (warning, links to Enrollment) and "PCs blocked from re-enrolling over 7 days" (info, counted from when the credential was revoked).

Before you start

1. Server deployed with the endpoints migration endpoints0023credentials applied (flask plugin upgrade-all).
2. Settings -> Endpoints -> Server Certificate shows a Public key pin (SPKI SHA-256). Cross-check it from a PC: Get-ShopDBServerPin.ps1 -Url https://<server>. The two must match.
3. Agent 1.20 (and tray 1.7) uploaded as the current release under Settings -> Endpoints -> Upload New/Legacy Agents. Keep the previous release staged as a legacy build: it is the rollback.
4. Settings -> Endpoints -> Enrollment -> Credentials: Shared key allowed on, Auto-enroll on (both are the defaults).
5. The GE-Enforce manifest scope that will carry the entries below is at Version 2.2 or later, and the fleet's GE-Enforce client library understands TargetHostnames / TargetMachineNumbers. An older manifest or library ignores those lists and applies the entry to EVERY PC in scope. The same happens with an empty list: never publish a targeted entry with a blank target.

Phase 1: pins on the PCs

Pins must be on a PC BEFORE agent 1.20 arrives there. A 1.20 agent with no pin falls back to plain Windows validation, which a self-signed or internal-CA server may fail; a pin set after the fact is not a rollout order you want to debug per bay.

Stage plugins/endpoints/client/Set-ShopDBServerPins.ps1 on the share and add a PS1 entry. Pilot first:

{
  "_comment": "Server key pin for per-PC credentials. Pilot only.",
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetMachineNumbers": ["<pilotbay>"]
}

Then widen by hostname list (or machine number list) in batches:

{
  "Name": "ShopDB server pins",
  "Type": "PS1",
  "Script": "scripts/Set-ShopDBServerPins.ps1",
  "Args": "-Pins <pin>",
  "DetectionMethod": "Always",
  "TargetHostnames": ["<host1>", "<host2>", "<host3>"]
}

The script compares before it writes, so Always costs nothing per cycle. A running agent picks up a changed ServerPins within about 60 seconds; no service restart is needed.

The registry wins. The installer and the store upgrade also write serverpins into appsettings.json. When ServerPins exists in the registry (either view), even empty, the agent uses only the registry and ignores the appsettings.json pins; only a PC with no ServerPins value falls back to them. The agent log says which source is in effect (Server certificate trust from registry / from appsettings.json, and Server pins changed, source ...).

• -Pins <pin>[,<pin>]: write exactly this set.
• -None: write an EMPTY value. No pins, normal Windows certificate checks, appsettings.json pins ignored.
• -Clear: DELETE the value. The appsettings.json pins apply again (if any).

Gate: on the pilot, HKLM\SOFTWARE\GE\ShopDB and HKLM\SOFTWARE\WOW6432Node\GE\ShopDB both hold ServerPins (REG_MULTI_SZ) equal to the pin on the settings page, and the next report-asset-*.log under C:\Logs\Shopfloor says server pin check on (1 pin(s)) followed by a RESPONSE line.

Rotating the server key later: any server-side change that gives the certificate a new key (a regenerated or re-issued certificate) changes the pin. Add the NEXT pin everywhere first: put the old AND the new pin in -Pins ("<oldpin>,<newpin>"), wait until every PC has both, only then switch the certificate on the server, then retire the old pin by writing ONLY the new pin to the registry (-Pins <newpin>). Do not use -Clear for this: deleting the value brings back whatever pins appsettings.json carries, which may be the old one. Switching the server first cuts off every pinned PC, including its ability to receive the fixed pin list through anything that talks to ShopDB. A renewal that keeps the same key keeps the same pin. PCs trusted through the Root store (below) also need the new certificate imported before the switch.

Trust the server certificate on PCs with agents older than 1.20

Required before ANY store wave. Agents older than 1.20 know nothing about ServerPins: when they download the upgrade package they check the normal Windows certificate chain. With a self-signed server certificate that check fails and the upgrade never lands.

• PCs installed by the server's generated installer (Build site agent) already trust the certificate.
• PCs installed from the share do NOT, and need this entry.

Get the certificate and its thumbprint from the server:

• Settings -> Endpoints -> Server Certificate -> download, or GET /api/endpoints/settings/server-cert/download (a .cer).
• The thumbprint is sha1thumbprint on GET /api/endpoints/settings/server-cert.

Stage the .cer and plugins/endpoints/client/Install-ShopDBServerCertificate.ps1 on the share and add a PS1 entry with the SAME targeting as the pin entry:

{
  "_comment": "Trust the ShopDB server certificate for pre-1.20 agent upgrades.",
  "Name": "ShopDB server certificate",
  "Type": "PS1",
  "Script": "scripts/Install-ShopDBServerCertificate.ps1",
  "Args": "-CertPath <share>\\scripts\\shopdb-server.cer -Thumbprint <sha1thumbprint>",
  "DetectionMethod": "Always",
  "TargetMachineNumbers": ["<pilotbay>"]
}

The script refuses a file whose thumbprint differs, refuses a CA certificate (one in Root would trust everything it signs), imports into LocalMachine\Root only when the certificate is not already there, and removes nothing. A non-CA leaf in Root trusts only this one server.

If a wave reaches a PC that does not trust the certificate, the download fails and the Migration tab marks the PC Blocked with server certificate not trusted by this PC. The server only knows about trust from that failure, so the blocker stays until the PC is queued again. Fix trust with the entry above, then in the Migration tab tick the PC (a PC whose only blocker is this one keeps a checkbox and shows "Retry after fixing trust") and click Upgrade selected. Upgrade next 25 never picks it, and a PC with any other blocker as well stays unselectable until that blocker is fixed.

Phase 2: re-stamp the share's agent entries

PCs that got the agent from the share (collections manifest entries with a Hash detection on shopdbagent.exe) are held to that hash by GE-Enforce. If a store upgrade wave puts 1.20 on such a PC while the entry still carries the 1.19 hash, GE-Enforce reinstalls 1.19 on its next cycle and the PC flips back.

Before any wave reaches a share-installed PC, either:

• re-stamp those entries to the 1.20 payload and its DetectionValue SHA-256 (and keep the ProductCode/version guards the entries already have), or
• remove the entries for the PCs the wave will cover.

Agent entries stay targeted. Illustrative shape for the pilot (keep your existing entry's type, installer and guards; what matters is the hash and a non-empty target):

{
  "Name": "ShopDB endpoint agent 1.20",
  "Type": "PS1",
  "Script": "apps/shopdbagent/1.20.0/install-shopdbagent.ps1",
  "DetectionMethod": "Hash",
  "DetectionPath": "C:\\Program Files\\ShopDB\\EndpointAgent\\shopdbagent.exe",
  "DetectionValue": "<sha256 of the 1.20 shopdbagent.exe>",
  "TargetHostnames": ["<pilothost>"]
}

Gate: the manifest preview for the pilot shows the 1.20 hash, and no entry for the same file with a blank target list exists in any scope that reaches it.

Phase 3: pilot one PC

1. Enrollment -> Migration: find <pilothost>. It should be Ready. If it is Blocked, the Blockers column lists every reason; fix those first.
2. Tick it, Upgrade selected.
3. Watch the row: Upgrading, then Enrolled within a few minutes of the new agent starting. The agent enrolls on its own with the shared key it already holds.

Gate before any wave:

• Credentials tab shows the PC enrolled, enrolledvia = auto, and a lastusedat that keeps moving.
• The PC's Fleet row keeps reporting; commands (Collect now) still work.
• The tray shows alerts and Send Status works (it now goes through the pipe).
• report-asset-*.log and report-printers-*.log show RESPONSE (credential).
• Nothing reinstalled the old agent after two GE-Enforce cycles.

Phase 4: waves

Migration tab -> Upgrade next 25. The confirm dialog states how many PCs get agent 1.20. The server re-checks readiness per PC and packages the agent once per wave; PCs that stopped being ready are returned as skipped.

Between waves, in the Waves list:

• wait until the wave shows Done (every PC enrolled or failed),
• investigate every failed PC before the next wave,
• check nothing moved from Enrolled back to Upgrading or Ready (a share entry reinstalling the old agent, see phase 2).

Repeat until Ready is 0.

Phase 5: stragglers and new PCs

• Blocked: each blocker names its fix. offline: bring the PC back or wait. no serial on record / duplicate hostname: fix the computer record. commands locked: release the endpoint lock or policy hold. no agent package in store: upload 1.20. agent too old for remote upgrade: move it via the share (targeted agent entry as in phase 2). server certificate not trusted by this PC: add the PC to the certificate entry (phase 1), wait a cycle, then tick it and Upgrade selected (phase 1 explains the retry). reenroll blocked: the PC was rolled back on purpose; clear it with allow-reenroll (see Rollback) only when moving it forward again.
• Not reporting: the PC has no working agent. It can only move via the share channel: add its hostname to a targeted 1.20 agent entry. It shows as Pending or Enrolled once the agent runs.
• Pending: the agent asked but auto-enroll could not decide (serial mismatch, duplicate or unknown hostname, already enrolled, or auto-enroll off). Credentials tab -> Requests -> Approve; the picker defaults to the matched computer, search to choose another. The PC collects its credential on its next retry - about a minute for a PC that started asking recently, slowing to 15 minutes once it has been pending a while. Reject anything you do not recognise. Approving also records the hostname the PC reports on the computer you picked when that record has none; if the record carries a DIFFERENT hostname the approval stops and asks, because the PC cannot authenticate against a computer recorded under another name.
• Newly imaged PCs: imaging installs 1.20 with the shared key, and the agent auto-enrolls on first start. A new PC can also be installed WITHOUT the shared key at any time, before or after cutover (the generated installer's "Include shared key" option off): the enroll call needs no key, so its request lands in Pending and it waits there until an admin approves it in the UI. An invalid key behaves the same (the server logs it). After the shared key is turned off (phase 6) that is the only path, and the steady-state path for new PCs.

Installing a one-off PC

A PC nobody is going to author a GE-Enforce manifest entry for - a loaner, a one-off office machine, a bay being rebuilt out of band - can be installed with a single command instead of a downloaded package. Managed bays keep using GE-Enforce; this does not replace the share channel, and the script itself says so when it finds an agent already installed.

1. Settings -> Endpoints -> Build site agent -> One-line install. Set how long the token lives (hours, 1 to 168) and how many installs it may serve (1 to 500), then Generate install command.
2. Copy the command. The token is shown ONCE and cannot be recovered; generate a new one if it is lost.
3. On the target PC, open PowerShell as administrator and paste it:

Copy the command the UI gives you. Do not retype it from here and do not reuse one you saved from an older round - see the warning under this block. It looks like this:

powershell -ExecutionPolicy Bypass -c "try { New-Item -ItemType Directory -Force -Path 'C:\Program Files\ShopDB\bootstrap' | Out-Null; icacls 'C:\Program Files\ShopDB\bootstrap' /inheritance:r /grant '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-18:(OI)(CI)F' | Out-Null; if ((Get-Variable LASTEXITCODE -ValueOnly) -ne 0) { Write-Host 'Refusing to run: could not restrict C:\Program Files\ShopDB\bootstrap to administrators (icacls failed). Fix the directory or delete it, then run the command again.' -ForegroundColor Red; exit 6 }; Remove-Item 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' -Force -ErrorAction SilentlyContinue; Invoke-WebRequest -UseBasicParsing 'https://<server>/api/endpoints/siteagent/bootstrap.ps1' -Headers @{'X-Endpoint-Bootstrap-Token'='<token>'} -OutFile 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1'; if ((Get-Acl 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1').GetOwner([Security.Principal.SecurityIdentifier]).Value -notin @('S-1-5-32-544','S-1-5-18',[Security.Principal.WindowsIdentity]::GetCurrent().User.Value)) { Write-Host 'Refusing to run: the staged bootstrap script is not owned by an administrator.' -ForegroundColor Red; exit 5 }; & 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' } finally { Remove-Item 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' -Force -ErrorAction SilentlyContinue }"

It creates an admin-only staging directory, downloads the script into it, runs it and deletes it again. Do not shorten it back to irm ... | iex: Defender scores that shape as Trojan:Win32/Commando.A!ml and kills the process before anything runs, on a default install with no ASR rules. Paste it as one line into an elevated cmd or PowerShell window; it means the same in both. Download .ps1 in the same block saves the script instead, for a PC where pasting a long line is awkward - it carries the same token, so treat the file the way you treat the command.

Do not use the old ?t=<token> one-liner that staged into C:\Windows\Temp. Any standard user on the bay can create that exact path first and wait for a tech to paste the command, which then runs the planted file elevated; the token also lands in the web server's access log. The command above stages under C:\Program Files, locks the directory to Administrators and SYSTEM and stops if that fails, refuses to run a staged file owned by anyone else, and sends the token in a header.

The ?t= form is still ACCEPTED, but only so a bay provisioned before this change keeps working until its command is re-issued. Every such fetch logs a server warning and writes an audit row against the token:

bootstrap token presented in the query string by <caller> with no admin
download marker, so this is an un-migrated bay running a command pasted
before the header form; ... re-issue the one-line install command

<caller> is 127.0.0.1 on a stock Windows install. IIS runs the app through HttpPlatformHandler over loopback and sets no X-Forwarded-For of its own; deploy/windows/web.config ships the rewrite rule that would add one commented out, so waitress has nothing to rewrite REMOTE_ADDR from. Check before you rely on the address: if these lines all say 127.0.0.1, the warning still tells you un-migrated bays EXIST and how many fetches they made, but not which bays. Get real addresses by enabling that rule (the Windows installer's -ClientIpSource direct, which installs URL Rewrite and allows HTTP_X_FORWARDED_FOR at server level); behind a further reverse proxy leave it off and set TRUSTED_PROXY_COUNT instead. Without that, the usable identifier is the token: each warning has a matching audit row on EndpointBootstrapToken carrying the 12-character tokenprefix, so you can at least tell which install round the stale command came from.

Work that list down: mint a new token and hand out the new command. Clicking Download .ps1 yourself does NOT add to it - that link is marked &via=download and logs at info, precisely so the warnings stay a list of bays. Once the list is empty, clear Accept the token in the URL in Settings -> Endpoints -> Build site agent -> One-line install (plugin.endpoints.bootstrapquerytoken) to refuse the query form outright. That also disables the Download .ps1 link and kills every command still in a technician's hands, so do it deliberately and re-issue first.

The token is short-lived and only fetches the installer. It carries no shared key and provisions nothing by itself: the package it downloads is built with "Include shared key" off, whatever the shared-key setting is, so the PC asks to enroll and waits for approval. Approve it in Enrollment -> Credentials -> Requests, the same as any other pending PC (phase 5).

A one-off PC is pending for unknown hostname by definition, so pick (or create) the computer record it belongs to. The approval records the hostname the PC reports on that record, which is what the PC authenticates against afterwards; picking a record already named something else stops the approval and asks before renaming it. Skipping that step is what used to leave a bootstrapped PC enrolled and then refused (403) on every call.

The script checks the downloaded binaries against the SHA-256 values the server stamped into it, refuses to run unelevated, and refuses to run on a PC that already has the agent service (set SHOPDB_BOOTSTRAP_FORCE=1 in the same window to override). After a successful install it waits up to two minutes for the credential to land and prints approved and enrolled if an admin approves it in that window; still pending is a normal outcome, not a failure.

If it stops before any of that with a red Refusing to run, nothing was installed and nothing was downloaded. Two cases, both meaning look at the bay rather than retry:

Exit	Message	What to do
6	could not restrict C:\Program Files\ShopDB\bootstrap to administrators	the ACL could not be set - usually the directory already exists with an owner or DACL you did not put there. Inspect it (icacls), delete it, run the command again.
5	the staged bootstrap script is not owned by an administrator	something else wrote the staged file. Treat the bay as suspect, do not force past it.

Revoke a token from the same block when the install round is done. Tokens list with their prefix, expiry and uses; they are also spent automatically (usecount reaching maxuses) and expire on their own.

Phase 6: cutover

Gate: Credentials tab Not enrolled is 0, or every remaining PC is one you have decided may stop reporting. Remember that turning the shared key off also refuses it on /api/collector/computers and /api/collector/printers, so a PC with reporter scripts but no enrolled agent goes silent.

1. Turn Shared key allowed off. The confirm shows how many PCs are not enrolled; type OFF.
2. Watch Fleet for an hour. A PC that went quiet is either unenrolled (turn the shared key back on, fix it, try again) or failed pinning.
3. Remove the shared key from every place it lives (below).
4. Rotate the old key so any copy you missed is dead: Settings -> Endpoints -> Regenerate key (and rotate the collector token the reporters used, if it was a separate token). With the shared key off nothing on the floor needs the new key: approved enroll requests are collected without it. Store it the way the key page describes and keep it out of every file below. A fleet rollback after this point (see Rollback) needs the new key pushed to the PCs again.

Every place the shared key lives

Where	Put there by	How to clear at cutover
GE-Enforce manifest entry Args (-ApiKey ...)	whoever authored the entry	Remove the argument from the entry and publish. Search every scope for the key prefix.
HKLM\SOFTWARE\GE\ShopDB CollectorKey, both registry views	agent install.ps1; Set-ShopdbCollectorKey.ps1 from <share>\common\configs\shopdb-collector.txt	First retire the Set-ShopdbCollectorKey.ps1 entry (or it rewrites the value every cycle), then a targeted PS1 entry that deletes CollectorKey in both views.
<share>\common\configs\shopdb-collector.txt	share staging	Delete the file after the entry above is retired.
appsettings.json apikey in the agent install folder	agent package / hand edits	Targeted PS1 entry that sets apikey to an empty string. The agent never needs it once enrolled.
Generated install.ps1 default -CollectorKey (Build site agent downloads)	Settings -> Endpoints -> Build site agent	Delete every kept copy (share, USB, work folders). Rebuild after rotation only if you still need the package.
Display kiosk shopdb-key.txt	display imaging	Confirm what the kiosk uses it for before clearing; a kiosk has no endpoint agent, so anything it reports with the shared key stops at cutover. Delete the file once that is settled.

The agent never deletes the shared key itself: other SYSTEM scripts may still read it until this table is done.

Rollback

The order matters. A 1.20 agent that is plainly un-enrolled enrolls again within about a second, before any downgrade can land, and the old agent it is then replaced with only knows the shared key, which the server refuses for an enrolled PC. Block re-enrollment FIRST.

One PC:

1. Make sure Shared key allowed is on.
2. Credentials tab -> Un-enroll the PC with Keep this PC from enrolling again ticked. API equivalent: POST /api/endpoints/enrollment/assets/<assetid>/revoke {"blockreenroll": true, "reason": "rollback"}. Its credential stops working immediately, the server admits the shared key for it again, and its enroll attempts stay pending with reenroll blocked.
3. Make the previous agent current in the store (Upload New/Legacy Agents) and queue the upgrade for that PC, or re-stamp its share entry to the old hash.
4. Verify: the Fleet row reports the old agent version, and the reporter logs show RESPONSE (shared key).
5. Remove the PC from the pin entry's targets only if pinning itself is the problem. Set-ShopDBServerPins.ps1 -None turns pinning off outright; -Clear deletes the value, which falls back to any appsettings.json pins.
6. Later, when moving the PC forward again, lift the block: POST /api/endpoints/enrollment/assets/<assetid>/allow-reenroll (or the same action in the UI). The PC enrolls on its next attempt once it runs 1.20. Until then the Migration tab shows it blocked with reenroll blocked.

Whole fleet: turn Shared key allowed back on first. Every PC that still holds the shared key reports again immediately. Then revoke with the block and downgrade PCs as above, in waves.

Verified on Windows

A full pass on a Windows test PC against a test server proved:

• the agent enrolls on its own;
• the shared key is refused for the enrolled PC (SHARED_KEY_REFUSED);
• a request naming another asset is refused (ASSET_MISMATCH);
• server pins are enforced;
• the tray pipe works for a standard (non-admin) user;
• alert acknowledge works;
• both SYSTEM reporters send with the per-PC credential;
• revoke is followed by re-enrollment;
• rollback to the previous agent works;
• with the shared key switched off, enrolled PCs keep working.

Quick reference

• Pin of the live server: Server Certificate settings page, or Get-ShopDBServerPin.ps1 -Url https://<server>.
• Pin of a certificate file: Get-ShopDBServerPin.ps1 -Path <file>.
• Agent log of enrollment: %ProgramData%\ShopDB\EndpointAgent\logs.
• Reporter logs: C:\Logs\Shopfloor\report-asset-*.log, report-printers-*.log. SHARED_KEY_REFUSED there means the PC is enrolled or the shared key is off and the reporter could not read the credential; CREDENTIAL_INVALID means the PC was un-enrolled and the agent is re-enrolling.