The bootstrap command staged its script into C:\Windows\Temp and then ran it elevated. That directory grants Users create-file with CREATOR OWNER full control, so somebody without admin rights can pre-plant the exact path the elevated run is about to execute. It now stages under Program Files, creates the directory with an explicit DACL when it is missing, and refuses to run a staged file that is not owned by an admin principal. A failed hardening is loud rather than silent: the run stops instead of carrying on with a file it could not protect. The token moved out of the query string into a header. In a query string it lands verbatim in the reverse proxy and IIS logs, which is a credential written to a file nobody thinks of as a secret store. The query form is still accepted, because bays provisioned before this change are holding install commands that use it, and it logs that it was used - without logging the token - so a site can see who still needs a re-issued command before turning it off with plugin.endpoints.bootstrapquerytoken. Two smaller things this also closes: A request with no token at all answered 400 naming the header, which told an anonymous caller the state of that switch and contradicted the module's own promise that anything unusable answers 404. No token is now 404. The 400 is kept for a token genuinely presented in the query form while the query form is off, which is a caller who needs to know their command is stale. The rollout documentation was republishing the vulnerable one-liner verbatim, so a tech following the runbook reinstated exactly what this fixes. Both documents now carry the current command. Also corrects the operator-facing copy for the token, which named a query argument and a header that do not exist: it is ?t= and X-Endpoint-Bootstrap-Token, per bootstrap.py's own constants. Anybody following the old text got a 404 with no reason given. Both switches added in this range are now declared with the other plugin settings and reachable from their settings page. Read through Setting.get but never declared, they were rows that did not exist until someone wrote one by hand, which on an air-gapped box means the feature was unreachable.
478 lines
26 KiB
Markdown
478 lines
26 KiB
Markdown
# Per-PC credentials rollout
|
|
|
|
Runbook for moving a site's endpoint fleet from the shared collector key to
|
|
per-PC credentials. The wire contract is `AGENT-CONTRACT.md` section 9.
|
|
|
|
Placeholders: `<server>` is the ShopDB base URL, `<share>` the GE-Enforce share
|
|
root, `<pin>` a server key pin, `<pilotbay>` / `<pilothost>` the pilot PC.
|
|
|
|
## What changes
|
|
|
|
- Every endpoint gets its own secret (`sdbe_...`), stored in
|
|
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
|
|
LocalMachine, readable by SYSTEM and Administrators only.
|
|
- The server knows which asset is calling from the secret. A PC can no longer
|
|
report as another PC.
|
|
- The tray holds no secret. It talks to the SYSTEM agent over a local pipe.
|
|
- The agent, the upgrade script and the SYSTEM reporters
|
|
(`Report-AssetToShopDB.ps1`, `Report-PrintersToShopDB.ps1`) pin the server's
|
|
public key when `ServerPins` is set.
|
|
- The shared key keeps working for PCs that have NOT enrolled, until an admin
|
|
turns it off. An enrolled PC is always refused the shared key
|
|
(`401 SHARED_KEY_REFUSED`).
|
|
|
|
## Where to look in the app
|
|
|
|
- **Settings -> Endpoints -> Enrollment**: Migration tab (readiness, waves) and
|
|
Credentials tab (enrolled PCs, pending requests, un-enroll, allow
|
|
re-enrollment, the shared key and auto-enroll switches).
|
|
- **Settings -> Endpoints** (the agent key page): the shared key PCs use until
|
|
they enroll, with an enrolled-of-total summary and the pending request count.
|
|
Regenerate and revoke there say how many PCs are not enrolled and would be
|
|
affected.
|
|
- **Settings -> Endpoints -> Server Certificate**: the certificate file's pin
|
|
and thumbprint, and **What PCs receive**, a live check of the certificate the
|
|
site's base URL (`site_base_url`) actually presents. When the two differ, pin
|
|
the presented one.
|
|
- **Fleet** (sidebar, under Assets): the **Credential** column (Enrolled,
|
|
Pending, Re-enroll blocked, Shared key (not enrolled), Not reporting) and a
|
|
filter on it.
|
|
- **PC detail**: the Endpoint status card shows the PC's credential state (with
|
|
prefix, issued, last used, enrolled via and block reason for
|
|
`endpoints.fleet` holders). Admins get **Revoke credential** (with **Keep this
|
|
PC from enrolling again**) and **Allow re-enrollment** beside the status chip.
|
|
- **Dashboard**: "N PCs waiting for enrollment approval" (warning, links to
|
|
Enrollment) and "PCs blocked from re-enrolling over 7 days" (info, counted
|
|
from when the credential was revoked).
|
|
|
|
## Before you start
|
|
|
|
1. Server deployed with the endpoints migration `endpoints0023credentials`
|
|
applied (`flask plugin upgrade-all`).
|
|
2. Settings -> Endpoints -> Server Certificate shows a **Public key pin (SPKI
|
|
SHA-256)**. Cross-check it from a PC:
|
|
`Get-ShopDBServerPin.ps1 -Url https://<server>`. The two must match.
|
|
3. Agent 1.20 (and tray 1.7) uploaded as the current release under Settings ->
|
|
Endpoints -> Upload New/Legacy Agents. Keep the previous release staged as a
|
|
legacy build: it is the rollback.
|
|
4. Settings -> Endpoints -> Enrollment -> Credentials: **Shared key allowed**
|
|
on, **Auto-enroll** on (both are the defaults).
|
|
5. The GE-Enforce manifest scope that will carry the entries below is at
|
|
`Version` 2.2 or later, and the fleet's GE-Enforce client library
|
|
understands `TargetHostnames` / `TargetMachineNumbers`. An older manifest or
|
|
library ignores those lists and applies the entry to EVERY PC in scope. The
|
|
same happens with an empty list: never publish a targeted entry with a blank
|
|
target.
|
|
|
|
## Phase 1: pins on the PCs
|
|
|
|
Pins must be on a PC BEFORE agent 1.20 arrives there. A 1.20 agent with no pin
|
|
falls back to plain Windows validation, which a self-signed or internal-CA
|
|
server may fail; a pin set after the fact is not a rollout order you want to
|
|
debug per bay.
|
|
|
|
Stage `plugins/endpoints/client/Set-ShopDBServerPins.ps1` on the share and add
|
|
a PS1 entry. Pilot first:
|
|
|
|
```json
|
|
{
|
|
"_comment": "Server key pin for per-PC credentials. Pilot only.",
|
|
"Name": "ShopDB server pins",
|
|
"Type": "PS1",
|
|
"Script": "scripts/Set-ShopDBServerPins.ps1",
|
|
"Args": "-Pins <pin>",
|
|
"DetectionMethod": "Always",
|
|
"TargetMachineNumbers": ["<pilotbay>"]
|
|
}
|
|
```
|
|
|
|
Then widen by hostname list (or machine number list) in batches:
|
|
|
|
```json
|
|
{
|
|
"Name": "ShopDB server pins",
|
|
"Type": "PS1",
|
|
"Script": "scripts/Set-ShopDBServerPins.ps1",
|
|
"Args": "-Pins <pin>",
|
|
"DetectionMethod": "Always",
|
|
"TargetHostnames": ["<host1>", "<host2>", "<host3>"]
|
|
}
|
|
```
|
|
|
|
The script compares before it writes, so `Always` costs nothing per cycle.
|
|
A running agent picks up a changed `ServerPins` within about 60 seconds; no
|
|
service restart is needed.
|
|
|
|
**The registry wins.** The installer and the store upgrade also write
|
|
`serverpins` into appsettings.json. When `ServerPins` exists in the registry
|
|
(either view), even empty, the agent uses only the registry and ignores the
|
|
appsettings.json pins; only a PC with no `ServerPins` value falls back to them.
|
|
The agent log says which source is in effect (`Server certificate trust from
|
|
registry` / `from appsettings.json`, and `Server pins changed, source ...`).
|
|
|
|
- `-Pins <pin>[,<pin>]`: write exactly this set.
|
|
- `-None`: write an EMPTY value. No pins, normal Windows certificate checks,
|
|
appsettings.json pins ignored.
|
|
- `-Clear`: DELETE the value. The appsettings.json pins apply again (if any).
|
|
|
|
**Gate:** on the pilot, `HKLM\SOFTWARE\GE\ShopDB` and
|
|
`HKLM\SOFTWARE\WOW6432Node\GE\ShopDB` both hold `ServerPins` (REG_MULTI_SZ)
|
|
equal to the pin on the settings page, and the next `report-asset-*.log` under
|
|
`C:\Logs\Shopfloor` says `server pin check on (1 pin(s))` followed by a
|
|
RESPONSE line.
|
|
|
|
**Rotating the server key later:** any server-side change that gives the
|
|
certificate a new key (a regenerated or re-issued certificate) changes the pin.
|
|
Add the NEXT pin everywhere first: put the old AND the new pin in `-Pins`
|
|
(`"<oldpin>,<newpin>"`), wait until every PC has both, only then switch the
|
|
certificate on the server, then retire the old pin by writing ONLY the new pin
|
|
to the registry (`-Pins <newpin>`). Do not use `-Clear` for this: deleting the
|
|
value brings back whatever pins appsettings.json carries, which may be the old
|
|
one. Switching the server first
|
|
cuts off every pinned PC, including its ability to receive the fixed pin list
|
|
through anything that talks to ShopDB. A renewal that keeps the same key keeps
|
|
the same pin. PCs trusted through the Root store (below) also need the new
|
|
certificate imported before the switch.
|
|
|
|
### Trust the server certificate on PCs with agents older than 1.20
|
|
|
|
Required before ANY store wave. Agents older than 1.20 know nothing about
|
|
`ServerPins`: when they download the upgrade package they check the normal
|
|
Windows certificate chain. With a self-signed server certificate that check
|
|
fails and the upgrade never lands.
|
|
|
|
- PCs installed by the server's generated installer (Build site agent) already
|
|
trust the certificate.
|
|
- PCs installed from the share do NOT, and need this entry.
|
|
|
|
Get the certificate and its thumbprint from the server:
|
|
|
|
- Settings -> Endpoints -> Server Certificate -> download, or
|
|
`GET /api/endpoints/settings/server-cert/download` (a `.cer`).
|
|
- The thumbprint is `sha1thumbprint` on `GET /api/endpoints/settings/server-cert`.
|
|
|
|
Stage the `.cer` and `plugins/endpoints/client/Install-ShopDBServerCertificate.ps1`
|
|
on the share and add a PS1 entry with the SAME targeting as the pin entry:
|
|
|
|
```json
|
|
{
|
|
"_comment": "Trust the ShopDB server certificate for pre-1.20 agent upgrades.",
|
|
"Name": "ShopDB server certificate",
|
|
"Type": "PS1",
|
|
"Script": "scripts/Install-ShopDBServerCertificate.ps1",
|
|
"Args": "-CertPath <share>\\scripts\\shopdb-server.cer -Thumbprint <sha1thumbprint>",
|
|
"DetectionMethod": "Always",
|
|
"TargetMachineNumbers": ["<pilotbay>"]
|
|
}
|
|
```
|
|
|
|
The script refuses a file whose thumbprint differs, refuses a CA certificate
|
|
(one in Root would trust everything it signs), imports into
|
|
`LocalMachine\Root` only when the certificate is not already there, and removes
|
|
nothing. A non-CA leaf in Root trusts only this one server.
|
|
|
|
If a wave reaches a PC that does not trust the certificate, the download fails
|
|
and the Migration tab marks the PC **Blocked** with `server certificate not
|
|
trusted by this PC`. The server only knows about trust from that failure, so
|
|
the blocker stays until the PC is queued again. Fix trust with the entry above,
|
|
then in the Migration tab tick the PC (a PC whose only blocker is this one keeps
|
|
a checkbox and shows "Retry after fixing trust") and click **Upgrade selected**.
|
|
**Upgrade next 25** never picks it, and a PC with any other blocker as well
|
|
stays unselectable until that blocker is fixed.
|
|
|
|
## Phase 2: re-stamp the share's agent entries
|
|
|
|
PCs that got the agent from the share (collections manifest entries with a
|
|
`Hash` detection on `shopdbagent.exe`) are held to that hash by GE-Enforce. If
|
|
a store upgrade wave puts 1.20 on such a PC while the entry still carries the
|
|
1.19 hash, GE-Enforce reinstalls 1.19 on its next cycle and the PC flips back.
|
|
|
|
Before any wave reaches a share-installed PC, either:
|
|
|
|
- re-stamp those entries to the 1.20 payload and its `DetectionValue` SHA-256
|
|
(and keep the `ProductCode`/version guards the entries already have), or
|
|
- remove the entries for the PCs the wave will cover.
|
|
|
|
Agent entries stay targeted. Illustrative shape for the pilot (keep your
|
|
existing entry's type, installer and guards; what matters is the hash and a
|
|
non-empty target):
|
|
|
|
```json
|
|
{
|
|
"Name": "ShopDB endpoint agent 1.20",
|
|
"Type": "PS1",
|
|
"Script": "apps/shopdbagent/1.20.0/install-shopdbagent.ps1",
|
|
"DetectionMethod": "Hash",
|
|
"DetectionPath": "C:\\Program Files\\ShopDB\\EndpointAgent\\shopdbagent.exe",
|
|
"DetectionValue": "<sha256 of the 1.20 shopdbagent.exe>",
|
|
"TargetHostnames": ["<pilothost>"]
|
|
}
|
|
```
|
|
|
|
**Gate:** the manifest preview for the pilot shows the 1.20 hash, and no entry
|
|
for the same file with a blank target list exists in any scope that reaches it.
|
|
|
|
## Phase 3: pilot one PC
|
|
|
|
1. Enrollment -> Migration: find `<pilothost>`. It should be **Ready**. If it is
|
|
**Blocked**, the Blockers column lists every reason; fix those first.
|
|
2. Tick it, **Upgrade selected**.
|
|
3. Watch the row: **Upgrading**, then **Enrolled** within a few minutes of the
|
|
new agent starting. The agent enrolls on its own with the shared key it
|
|
already holds.
|
|
|
|
**Gate before any wave:**
|
|
|
|
- Credentials tab shows the PC enrolled, `enrolledvia` = `auto`, and a
|
|
`lastusedat` that keeps moving.
|
|
- The PC's Fleet row keeps reporting; commands (Collect now) still work.
|
|
- The tray shows alerts and Send Status works (it now goes through the pipe).
|
|
- `report-asset-*.log` and `report-printers-*.log` show `RESPONSE (credential)`.
|
|
- Nothing reinstalled the old agent after two GE-Enforce cycles.
|
|
|
|
## Phase 4: waves
|
|
|
|
Migration tab -> **Upgrade next 25**. The confirm dialog states how many PCs get
|
|
agent 1.20. The server re-checks readiness per PC and packages the agent once
|
|
per wave; PCs that stopped being ready are returned as skipped.
|
|
|
|
Between waves, in the Waves list:
|
|
|
|
- wait until the wave shows **Done** (every PC enrolled or failed),
|
|
- investigate every **failed** PC before the next wave,
|
|
- check nothing moved from Enrolled back to Upgrading or Ready (a share entry
|
|
reinstalling the old agent, see phase 2).
|
|
|
|
Repeat until **Ready** is 0.
|
|
|
|
## Phase 5: stragglers and new PCs
|
|
|
|
- **Blocked:** each blocker names its fix. `offline`: bring the PC back or wait.
|
|
`no serial on record` / `duplicate hostname`: fix the computer record.
|
|
`commands locked`: release the endpoint lock or policy hold. `no agent package
|
|
in store`: upload 1.20. `agent too old for remote upgrade`: move it via the
|
|
share (targeted agent entry as in phase 2). `server certificate not trusted
|
|
by this PC`: add the PC to the certificate entry (phase 1), wait a cycle,
|
|
then tick it and **Upgrade selected** (phase 1 explains the retry). `reenroll blocked`: the PC was rolled back on purpose; clear
|
|
it with allow-reenroll (see Rollback) only when moving it forward again.
|
|
- **Not reporting:** the PC has no working agent. It can only move via the
|
|
share channel: add its hostname to a targeted 1.20 agent entry. It shows as
|
|
Pending or Enrolled once the agent runs.
|
|
- **Pending:** the agent asked but auto-enroll could not decide (serial
|
|
mismatch, duplicate or unknown hostname, already enrolled, or auto-enroll
|
|
off). Credentials tab -> Requests -> **Approve**; the picker defaults to the
|
|
matched computer, search to choose another. The PC collects its credential on
|
|
its next retry - about a minute for a PC that started asking recently,
|
|
slowing to 15 minutes once it has been pending a while. **Reject** anything
|
|
you do not recognise. Approving also records the hostname the PC reports on
|
|
the computer you picked when that record has none; if the record carries a
|
|
DIFFERENT hostname the approval stops and asks, because the PC cannot
|
|
authenticate against a computer recorded under another name.
|
|
- **Newly imaged PCs:** imaging installs 1.20 with the shared key, and the agent
|
|
auto-enrolls on first start. A new PC can also be installed WITHOUT the
|
|
shared key at any time, before or after cutover (the generated installer's
|
|
"Include shared key" option off): the enroll call needs no key, so its request
|
|
lands in Pending and it waits there until an admin approves it in the UI. An
|
|
invalid key behaves the same (the server logs it). After the shared key is
|
|
turned off (phase 6) that is the only path, and the steady-state path for new
|
|
PCs.
|
|
|
|
## Installing a one-off PC
|
|
|
|
A PC nobody is going to author a GE-Enforce manifest entry for - a loaner, a
|
|
one-off office machine, a bay being rebuilt out of band - can be installed with
|
|
a single command instead of a downloaded package. **Managed bays keep using
|
|
GE-Enforce**; this does not replace the share channel, and the script itself
|
|
says so when it finds an agent already installed.
|
|
|
|
1. Settings -> Endpoints -> Build site agent -> **One-line install**. Set how
|
|
long the token lives (hours, 1 to 168) and how many installs it may serve
|
|
(1 to 500), then **Generate install command**.
|
|
2. Copy the command. The token is shown ONCE and cannot be recovered; generate
|
|
a new one if it is lost.
|
|
3. On the target PC, open PowerShell **as administrator** and paste it:
|
|
|
|
**Copy the command the UI gives you.** Do not retype it from here and do not
|
|
reuse one you saved from an older round - see the warning under this block. It
|
|
looks like this:
|
|
|
|
```
|
|
powershell -ExecutionPolicy Bypass -c "try { New-Item -ItemType Directory -Force -Path 'C:\Program Files\ShopDB\bootstrap' | Out-Null; icacls 'C:\Program Files\ShopDB\bootstrap' /inheritance:r /grant '*S-1-5-32-544:(OI)(CI)F' '*S-1-5-18:(OI)(CI)F' | Out-Null; if ((Get-Variable LASTEXITCODE -ValueOnly) -ne 0) { Write-Host 'Refusing to run: could not restrict C:\Program Files\ShopDB\bootstrap to administrators (icacls failed). Fix the directory or delete it, then run the command again.' -ForegroundColor Red; exit 6 }; Remove-Item 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' -Force -ErrorAction SilentlyContinue; Invoke-WebRequest -UseBasicParsing 'https://<server>/api/endpoints/siteagent/bootstrap.ps1' -Headers @{'X-Endpoint-Bootstrap-Token'='<token>'} -OutFile 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1'; if ((Get-Acl 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1').GetOwner([Security.Principal.SecurityIdentifier]).Value -notin @('S-1-5-32-544','S-1-5-18',[Security.Principal.WindowsIdentity]::GetCurrent().User.Value)) { Write-Host 'Refusing to run: the staged bootstrap script is not owned by an administrator.' -ForegroundColor Red; exit 5 }; & 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' } finally { Remove-Item 'C:\Program Files\ShopDB\bootstrap\shopdb-bootstrap.ps1' -Force -ErrorAction SilentlyContinue }"
|
|
```
|
|
|
|
It creates an admin-only staging directory, downloads the script into it, runs
|
|
it and deletes it again. Do not shorten it back to `irm ... | iex`: Defender
|
|
scores that shape as `Trojan:Win32/Commando.A!ml` and kills the process before
|
|
anything runs, on a default install with no ASR rules. Paste it as one line
|
|
into an elevated **cmd** or **PowerShell** window; it means the same in both.
|
|
**Download .ps1** in the same block saves the script instead, for a PC where
|
|
pasting a long line is awkward - it carries the same token, so treat the file
|
|
the way you treat the command.
|
|
|
|
> **Do not use the old `?t=<token>` one-liner that staged into
|
|
> `C:\Windows\Temp`.** Any standard user on the bay can create that exact path
|
|
> first and wait for a tech to paste the command, which then runs the planted
|
|
> file elevated; the token also lands in the web server's access log. The
|
|
> command above stages under `C:\Program Files`, locks the directory to
|
|
> Administrators and SYSTEM and stops if that fails, refuses to run a staged
|
|
> file owned by anyone else, and sends the token in a header.
|
|
>
|
|
> The `?t=` form is still ACCEPTED, but only so a bay provisioned before this
|
|
> change keeps working until its command is re-issued. Every such fetch logs a
|
|
> server warning and writes an audit row against the token:
|
|
>
|
|
> ```
|
|
> bootstrap token presented in the query string by <caller> with no admin
|
|
> download marker, so this is an un-migrated bay running a command pasted
|
|
> before the header form; ... re-issue the one-line install command
|
|
> ```
|
|
>
|
|
> **`<caller>` is 127.0.0.1 on a stock Windows install.** IIS runs the app
|
|
> through HttpPlatformHandler over loopback and sets no `X-Forwarded-For` of
|
|
> its own; `deploy/windows/web.config` ships the rewrite rule that would add
|
|
> one commented out, so waitress has nothing to rewrite `REMOTE_ADDR` from.
|
|
> Check before you rely on the address: if these lines all say 127.0.0.1, the
|
|
> warning still tells you un-migrated bays EXIST and how many fetches they
|
|
> made, but not which bays. Get real addresses by enabling that rule (the
|
|
> Windows installer's `-ClientIpSource direct`, which installs URL Rewrite and
|
|
> allows `HTTP_X_FORWARDED_FOR` at server level); behind a further reverse
|
|
> proxy leave it off and set `TRUSTED_PROXY_COUNT` instead. Without that, the
|
|
> usable identifier is the token: each warning has a matching audit row on
|
|
> `EndpointBootstrapToken` carrying the 12-character `tokenprefix`, so you can
|
|
> at least tell which install round the stale command came from.
|
|
>
|
|
> Work that list down: mint a new token and hand out the new command. Clicking
|
|
> **Download .ps1** yourself does NOT add to it - that link is marked
|
|
> `&via=download` and logs at info, precisely so the warnings stay a list of
|
|
> bays. Once the list is empty, clear **Accept the token in the URL** in
|
|
> Settings -> Endpoints -> Build site agent -> One-line install
|
|
> (`plugin.endpoints.bootstrapquerytoken`) to refuse the query form outright.
|
|
> That also disables the Download .ps1 link and kills every command still in a
|
|
> technician's hands, so do it deliberately and re-issue first.
|
|
|
|
The token is short-lived and only fetches the installer. It carries no shared
|
|
key and provisions nothing by itself: the package it downloads is built with
|
|
"Include shared key" off, whatever the shared-key setting is, so the PC asks to
|
|
enroll and **waits for approval**. Approve it in Enrollment -> Credentials ->
|
|
Requests, the same as any other pending PC (phase 5).
|
|
|
|
A one-off PC is pending for `unknown hostname` by definition, so pick (or
|
|
create) the computer record it belongs to. The approval records the hostname
|
|
the PC reports on that record, which is what the PC authenticates against
|
|
afterwards; picking a record already named something else stops the approval
|
|
and asks before renaming it. Skipping that step is what used to leave a
|
|
bootstrapped PC enrolled and then refused (403) on every call.
|
|
|
|
The script checks the downloaded binaries against the SHA-256 values the server
|
|
stamped into it, refuses to run unelevated, and refuses to run on a PC that
|
|
already has the agent service (set `SHOPDB_BOOTSTRAP_FORCE=1` in the same
|
|
window to override). After a successful install it waits up to two minutes for
|
|
the credential to land and prints `approved and enrolled` if an admin approves
|
|
it in that window; still pending is a normal outcome, not a failure.
|
|
|
|
If it stops before any of that with a red **Refusing to run**, nothing was
|
|
installed and nothing was downloaded. Two cases, both meaning look at the bay
|
|
rather than retry:
|
|
|
|
| Exit | Message | What to do |
|
|
|---|---|---|
|
|
| 6 | could not restrict `C:\Program Files\ShopDB\bootstrap` to administrators | the ACL could not be set - usually the directory already exists with an owner or DACL you did not put there. Inspect it (`icacls`), delete it, run the command again. |
|
|
| 5 | the staged bootstrap script is not owned by an administrator | something else wrote the staged file. Treat the bay as suspect, do not force past it. |
|
|
|
|
Revoke a token from the same block when the install round is done. Tokens list
|
|
with their prefix, expiry and uses; they are also spent automatically
|
|
(`usecount` reaching `maxuses`) and expire on their own.
|
|
|
|
## Phase 6: cutover
|
|
|
|
**Gate:** Credentials tab **Not enrolled** is 0, or every remaining PC is one
|
|
you have decided may stop reporting. Remember that turning the shared key off
|
|
also refuses it on `/api/collector/computers` and `/api/collector/printers`, so
|
|
a PC with reporter scripts but no enrolled agent goes silent.
|
|
|
|
1. Turn **Shared key allowed** off. The confirm shows how many PCs are not
|
|
enrolled; type `OFF`.
|
|
2. Watch Fleet for an hour. A PC that went quiet is either unenrolled (turn the
|
|
shared key back on, fix it, try again) or failed pinning.
|
|
3. Remove the shared key from every place it lives (below).
|
|
4. Rotate the old key so any copy you missed is dead: Settings -> Endpoints ->
|
|
**Regenerate key** (and rotate the collector token the reporters used, if it
|
|
was a separate token). With the shared key off nothing on the floor needs
|
|
the new key: approved enroll requests are collected without it. Store it
|
|
the way the key page describes and keep it out of every file below. A
|
|
fleet rollback after this point (see Rollback) needs the new key pushed to
|
|
the PCs again.
|
|
|
|
### Every place the shared key lives
|
|
|
|
| Where | Put there by | How to clear at cutover |
|
|
|---|---|---|
|
|
| GE-Enforce manifest entry `Args` (`-ApiKey ...`) | whoever authored the entry | Remove the argument from the entry and publish. Search every scope for the key prefix. |
|
|
| `HKLM\SOFTWARE\GE\ShopDB` `CollectorKey`, both registry views | agent `install.ps1`; `Set-ShopdbCollectorKey.ps1` from `<share>\common\configs\shopdb-collector.txt` | First retire the `Set-ShopdbCollectorKey.ps1` entry (or it rewrites the value every cycle), then a targeted PS1 entry that deletes `CollectorKey` in both views. |
|
|
| `<share>\common\configs\shopdb-collector.txt` | share staging | Delete the file after the entry above is retired. |
|
|
| `appsettings.json` `apikey` in the agent install folder | agent package / hand edits | Targeted PS1 entry that sets `apikey` to an empty string. The agent never needs it once enrolled. |
|
|
| Generated `install.ps1` default `-CollectorKey` (Build site agent downloads) | Settings -> Endpoints -> Build site agent | Delete every kept copy (share, USB, work folders). Rebuild after rotation only if you still need the package. |
|
|
| Display kiosk `shopdb-key.txt` | display imaging | Confirm what the kiosk uses it for before clearing; a kiosk has no endpoint agent, so anything it reports with the shared key stops at cutover. Delete the file once that is settled. |
|
|
|
|
The agent never deletes the shared key itself: other SYSTEM scripts may still
|
|
read it until this table is done.
|
|
|
|
## Rollback
|
|
|
|
The order matters. A 1.20 agent that is plainly un-enrolled enrolls again
|
|
within about a second, before any downgrade can land, and the old agent it is
|
|
then replaced with only knows the shared key, which the server refuses for an
|
|
enrolled PC. Block re-enrollment FIRST.
|
|
|
|
**One PC:**
|
|
|
|
1. Make sure **Shared key allowed** is on.
|
|
2. Credentials tab -> **Un-enroll** the PC with **Keep this PC from enrolling
|
|
again** ticked. API equivalent:
|
|
`POST /api/endpoints/enrollment/assets/<assetid>/revoke`
|
|
`{"blockreenroll": true, "reason": "rollback"}`.
|
|
Its credential stops working immediately, the server admits the shared key
|
|
for it again, and its enroll attempts stay pending with `reenroll blocked`.
|
|
3. Make the previous agent current in the store (Upload New/Legacy Agents) and
|
|
queue the upgrade for that PC, or re-stamp its share entry to the old hash.
|
|
4. Verify: the Fleet row reports the old agent version, and the reporter logs
|
|
show `RESPONSE (shared key)`.
|
|
5. Remove the PC from the pin entry's targets only if pinning itself is the
|
|
problem. `Set-ShopDBServerPins.ps1 -None` turns pinning off outright;
|
|
`-Clear` deletes the value, which falls back to any appsettings.json pins.
|
|
6. Later, when moving the PC forward again, lift the block:
|
|
`POST /api/endpoints/enrollment/assets/<assetid>/allow-reenroll` (or the
|
|
same action in the UI). The PC enrolls on its next attempt once it runs
|
|
1.20. Until then the Migration tab shows it blocked with `reenroll blocked`.
|
|
|
|
**Whole fleet:** turn **Shared key allowed** back on first. Every PC that still
|
|
holds the shared key reports again immediately. Then revoke with the block and
|
|
downgrade PCs as above, in waves.
|
|
|
|
## Verified on Windows
|
|
|
|
A full pass on a Windows test PC against a test server proved:
|
|
|
|
- the agent enrolls on its own;
|
|
- the shared key is refused for the enrolled PC (`SHARED_KEY_REFUSED`);
|
|
- a request naming another asset is refused (`ASSET_MISMATCH`);
|
|
- server pins are enforced;
|
|
- the tray pipe works for a standard (non-admin) user;
|
|
- alert acknowledge works;
|
|
- both SYSTEM reporters send with the per-PC credential;
|
|
- revoke is followed by re-enrollment;
|
|
- rollback to the previous agent works;
|
|
- with the shared key switched off, enrolled PCs keep working.
|
|
|
|
## Quick reference
|
|
|
|
- Pin of the live server: Server Certificate settings page, or
|
|
`Get-ShopDBServerPin.ps1 -Url https://<server>`.
|
|
- Pin of a certificate file: `Get-ShopDBServerPin.ps1 -Path <file>`.
|
|
- Agent log of enrollment: `%ProgramData%\ShopDB\EndpointAgent\logs`.
|
|
- Reporter logs: `C:\Logs\Shopfloor\report-asset-*.log`,
|
|
`report-printers-*.log`. `SHARED_KEY_REFUSED` there means the PC is enrolled
|
|
or the shared key is off and the reporter could not read the credential;
|
|
`CREDENTIAL_INVALID` means the PC was un-enrolled and the agent is
|
|
re-enrolling.
|