shopdb-flask/docs/geenforce-api-cutover.md

GE-Enforce over HTTPS API: cutover reference

This is the operator/developer reference for moving a shopfloor PC type off the SMB-share GE-Enforce delivery and onto the shopdb HTTPS API. The first cohort cut over was the displays/kiosks (gea-shopfloor-display): share-less, Intune/Entra-joined PCs with no SFLD share credentials. This doc captures everything learned doing that, so extending the cutover to the other pc-types (gea-shopfloor-cmm, -collections, -keyence, -genspect, -heattreat, -partmarker, -nocollections, common) does not require re-learning it.

It pairs with the existing docs (which describe the pieces; this one describes the CUTOVER):

• docs/GE-ENFORCE.md - concepts and the plugin
• docs/GE-ENFORCE-CLIENT.md - client fetch/report contract
• docs/GE-ENFORCE-DEPLOY.md - what must land on a PC
• docs/GE-ENFORCE-DISPLAY.md - the display scope specifics
• the PXE repo (docs/ge-enforce-v2-architecture.md) - the SMB world being cut away from

Contents:

1. Overview and why
2. Server architecture
3. Delivery models: smb vs http/inline payloads
4. Authoring a scope
5. The on-PC client and engine
6. Bootstrap for share-less PCs
7. Asset reporting via the collector
8. HARD-WON GOTCHAS
9. How this was verified
10. Deploy
11. PLAYBOOK: extending to a new pc-type
12. Open items / TODO

---

1. Overview and why

GE-Enforce v2 delivers desired-state manifests and installer payloads from the SFLD SMB share (\\tsgwp00525.wjs.geaerospace.net\shared\dt\shopfloor\). Every PC mounts the share with Azure-DSC-provisioned SFLD credentials, reads <scope>\manifest.json, and runs the engine (Install-FromManifest.ps1). That works for the domain fleet but is a hard dead end for share-less PCs.

The HTTPS API path replaces the share as the transport, keeping the engine and its detection/self-heal behavior untouched:

Concern	SMB share	HTTPS API
Manifest source	<scope>\manifest.json on the share	GET /api/geenforce/manifest?pctype=<scope> (published snapshot)
Payload source	share paths (apps\...)	GET /api/geenforce/payload/<sha256> (content-addressed)
Auth	SFLD share credential (DSC)	geenforce.fetch service token OR source-IP allowlist
Result visibility	log files on the PC	POST /api/geenforce/report -> Enforcement Reports UI
Versioning	file overwrite + _meta/history	immutable published versions, rollback, ETag

Which PCs MUST use the API: the share-less ones. Displays/kiosks are Intune/Entra-joined with no SFLD credentials and no domain trust, so SMB is not an option at all. The rest of the fleet CAN stay on the share (and currently does); for them the API is an opt-in migration, not a forced one.

West Jefferson facts used throughout this doc:

Fact	Value
Prod host	tsgwp00525.wjs.geaerospace.net
App mount	/shopdb (IIS, app dir C:\inetpub\wwwroot\shopdb, pool shopdbflask-prod)
BaseUrl clients use	https://tsgwp00525.wjs.geaerospace.net/shopdb
Prod DB	shopdb_flask (MySQL)
Client allowlist CIDRs	10.134.48.0/23,10.48.249.0/26 (the WJ corp/AESFMA shopfloor subnets)
Dev/staging instance	/ops mount, DB shopdb_flask_dev, pool shopdbflask

---

2. Server architecture

All server code is in plugins/geenforce/ (routes: plugins/geenforce/api/routes.py).

Client-facing endpoints (three)

Endpoint	Method	Auth	What it does
/api/geenforce/manifest?pctype=<scope>&phase=runtime	GET	geenforce.fetch token OR IP allowlist	Serves the CURRENT PUBLISHED manifest snapshot for a scope (never the draft). ETag: "<scopeid>-v<version>", X-Manifest-Version header, 304 on If-None-Match.
/api/geenforce/payload/<sha256>	GET	geenforce.fetch token OR IP allowlist	Streams a payload blob by content hash: blob store first (service.blob_path), then inline ManifestPayload. ETag = the hash. Per-IP rate limited (120/min default) and size-capped (512 MB default, 413 above).
/api/geenforce/report	POST	geenforce.report token OR IP allowlist	Records one enforcement cycle via service.record_enforcement_report: hostname, scopename, appliedversion, enforcerversion, counts, per-entry results. Upserts the current report per (hostname, scopename, phase); older reports kept as history.

Plus the collector for asset reporting (section 7): POST /api/collector/computers in shopdb/core/api/collector.py - a DIFFERENT auth domain (collector.ingest), NOT covered by the geenforce allowlist.

Auth model

_require_service_token(scope) in routes.py is the decorator factory. Two paths, fail-closed (neither -> 401):

1. A managed service token with the scope (geenforce.fetch for manifest/payload, geenforce.report for report), sent as X-API-Key or a Bearer PAT (authorized_service_token). A token may carry resourcescopelist bindings: a bound token can only fetch its own scope's manifest (403 otherwise) and only blobs those scopes' published manifests reference (service.blob_referenced_by_scopes, 404 so hashes cannot be probed). Displays get a token bound to gea-shopfloor-display (see GE-ENFORCE-DISPLAY.md).
2. The IP allowlist: _ip_allowlisted() checks the caller against the setting geenforce_allowed_cidrs (comma-separated CIDRs/IPs, empty = disabled). Network trust replaces the shared secret for a vaulted fleet. The allowlisted path has no resource-scope binding (unrestricted).

The token paths use RBAC permissions the plugin registers in GeEnforcePlugin.get_permissions(): geenforce.manage (edit), geenforce.publish (ship), geenforce.fetch and geenforce.report (client service tokens). Admin CRUD/publish/simulate/compliance routes are JWT plus geenforce.manage/geenforce.publish.

Why the allowlist uses remote_addr (the IIS XFF dependency)

_trusted_client_ip() returns request.remote_addr, NOT the raw X-Forwarded-For header. Proxies APPEND to X-Forwarded-For, so its first hop is attacker-controlled: parsing it (as _client_ip() does, acceptably, for rate limiting only) would let any caller send X-Forwarded-For: <allowlisted-ip> and bypass the token entirely.

remote_addr is trustworthy only because of a two-piece chain that MUST stay in place:

1. The IIS URL-Rewrite rule in the /shopdb web.config OVERWRITES (not appends) the inbound X-Forwarded-For with REMOTE_ADDR, the real TCP peer.
2. waitress runs with --trusted-proxy=127.0.0.1 --trusted-proxy-headers=x-forwarded-for, so it derives remote_addr from that overwritten header only when the request comes from IIS on localhost. A client that somehow hits waitress directly is not a trusted proxy, so its remote_addr is its own real peer address. Either way, real client IP.

If the IIS rule is ever removed, the allowlist becomes SPOOFABLE. This document previously claimed the opposite; it was wrong, and the reasoning matters.

IIS does not set X-Forwarded-For on its own - the rewrite rule is the only thing that does. Remove the rule and IIS still forwards whatever X-Forwarded-For the caller sent. waitress trusts that header because it arrives from 127.0.0.1, which is IIS, and sets remote_addr from it. So a caller who sends X-Forwarded-For: 10.134.48.5 gets remote_addr = 10.134.48.5, matches the allowlist and fetches manifests token-less from anywhere on the network.

The rule is not a nicety that improves logging. It is the control that makes remote_addr trustworthy, and everything downstream - the allowlist, the dashboard visitor-location lookup, per-host login rate limiting - depends on it.

Three consequences worth stating plainly:

• The Windows installer enables the rule when told IIS faces clients directly (-ClientIpSource direct), installs URL Rewrite from the bundle to make that possible offline, and its stage-5 check fails if the rule is not live.
• On a hand-built server, verify it: deploy/windows/web.config must have the <rewrite> block ACTIVE, not inside the SHOPDB-CLIENTIP comment markers.
• Behind a real reverse proxy the rule is the wrong answer, because REMOTE_ADDR is then the proxy. There, the proxy must set X-Forwarded-For itself and be the only thing that can reach IIS. -ClientIpSource proxy covers that case.

---

3. Delivery models: smb vs http/inline payloads

Every ManifestEntry carries a payloadsource (plugins/geenforce/serializer.py and importer.py):

PayloadSource	Meaning	Manifest emission
smb (default)	Entry installs from the share exactly as v2 does; the entry's Installer/Script/Source is a share-relative path.	Nothing emitted - share manifests round-trip byte-identical, parity preserved.
http	Payload lives in the server's content-addressed blob store (instance/geenforce/payloads/<sha256>, registry row ManifestBlob). For big files (MSIs, EXEs). Upload via flask geenforce add-payload <file> or service.store_blob.	PayloadSource, PayloadSha256, PayloadRef keys on the entry.
inline	Payload bytes live IN the DB (ManifestPayload, <= 1 MB) - small scripts and configs. Attach via service.store_inline_payload(entry, filename, contenttype, rawbytes) or POST /api/geenforce/entries/<id>/payload.	Same three keys.

Both http and inline are served from the same client URL: GET /api/geenforce/payload/<sha256> (blob store checked first, then inline). The sha256 IS the integrity contract - the client re-hashes after download.

How the client stages payloads (Resolve-ShopdbPayloads)

Resolve-ShopdbPayloads in plugins/geenforce/client/ShopdbEnforceClient.psm1 is the bridge that lets the UNCHANGED engine install share-less:

1. For each entry with PayloadSha256 and PayloadSource http/inline, call Get-ShopdbPayload: download to C:\ProgramData\ShopDB\geenforce\payloads\<sha><ext> (ext from PayloadRef), verify the sha256, keep it as a content-addressed last-known-good cache (a cache hit only counts if the bytes still hash right).
2. Rewrite the entry's path field to the LEAF filename of the staged file (Split-Path -Leaf) - NOT the absolute path. Field by Type: Installer for MSI/EXE/CMD/BAT/INF, Script for PS1, Source for File.
3. Write a sibling <scope>.resolved.json manifest and return its path (or the original path if nothing needed resolving). A payload that cannot be fetched/verified THROWS - the runner's fail-safe catch decides what happens.

The runner (Invoke-ShopdbEnforce.ps1) then sets the engine's -InstallerRoot to that same payloads directory, so the engine's Join-Path $InstallerRoot <leaf> resolves to the staged file. smb entries in a mixed manifest are left untouched and still resolve against the share (a PC that has it).

---

4. Authoring a scope

Two authoring paths, both ending in service.replace_scope_draft(scopename, phase, manifest_dict) (idempotent draft rebuild - published versions are never touched by a re-import):

A. import-share: adopt an existing SMB manifest

flask geenforce import-share --shareroot <path> [--scope <name>] [--preinstall <path>]
flask geenforce publish <scopename> [--phase runtime] [--notes "..."]

importer.discover_share walks the share root and ingests common/manifest.json, display/manifest.json, and every gea-shopfloor-*/manifest.json (skipping .bak variants). Entries come in as smb payloads. Run flask geenforce parity --shareroot <path> first (Gate A): proves import+re-export is behaviorally lossless before anything ships.

B. authoring in code: seed_display_scope as the template

plugins/geenforce/seed_display_scope.py is the reference for a scope that never existed on the share. Pattern:

• Build the manifest dict in Python (build_display_manifest()): four Type=Registry drift-heal entries re-asserting the Edge kiosk relaunch policies from imaging (09-Setup-Display.ps1), one inline PS1 dispatcher, one inline PS1 always-on script. Registry heals use DetectionMethod=ValueMatches against the same path/name they write, so drift self-heals; the PS1s use DetectionMethod=Always and are idempotent.
• The dispatcher (Invoke-DisplayKioskDispatch.ps1, generated by build_dispatcher_script()) reads C:\Enrollment\display-type.txt, maps the subtype through the data-driven DISPLAY_TYPE_TARGETS table (Dashboard -> /shopfloor, Lobby -> /tv, 3DPrintRoom -> /parts-kiosk), and writes an all-users Startup shortcut (ShopDB Kiosk.lnk) launching Edge --kiosk fullscreen at {BaseUrl}{route}. It does NOT Start-Process Edge (see gotchas). Base URL comes from HKLM BaseUrl, falling back to the WJ host.
• seed_display_scope(publish=False): replace_scope_draft, flush (entries need entryids), then service.store_inline_payload(...) for each script entry (sets payloadsource='inline', payloadsha256, payloadref), optionally service.publish_scope(...), commit. Draft rebuild is idempotent; publish always creates a NEW version.

Run it on the server:

cd C:\inetpub\wwwroot\shopdb
$env:FLASK_APP = 'shopdb'
'from plugins.geenforce.seed_display_scope import seed_display_scope; print(seed_display_scope(publish=True))' | venv\Scripts\python -m flask shell

Expected: {scopeid, entrycount: 6, entrytypes: [Registry x4, PS1, PS1], dispatchersha256, alwaysonsha256, publishedversion: N}.

Publishing

service.publish_scope freezes the draft (rendered by serializer.scope_to_json) into an immutable ManifestPublishedVersion and flips iscurrent. Clients only ever see published versions. rollback_scope re-currents an older version. Also available over the API (POST /scopes/<id>/publish, permission geenforce.publish) and the GE-Enforce UI.

Attaching payloads

• Inline (<= 1 MB): service.store_inline_payload in code, or POST /api/geenforce/entries/<entryid>/payload (multipart file).
• Blob (http): flask geenforce add-payload <filepath> prints the sha256; set PayloadSource=http + PayloadSha256 (+ PayloadRef for the filename/extension) on the entry.

Publish AFTER attaching - the published JSON is what carries the PayloadSha256 the client fetches, and blob access for resource-bound tokens is checked against the CURRENT published manifest.

---

5. The on-PC client and engine

Config: HKLM:\SOFTWARE\GE\ShopDB

Value	Used by	Notes
BaseUrl	enforce client + kiosk dispatcher	e.g. https://tsgwp00525.wjs.geaerospace.net/shopdb. Required.
ApiToken	enforce client	geenforce.fetch (+ report) PAT. OPTIONAL - a token-less client relies on the IP allowlist (Get-ShopdbConfig treats BaseUrl-only as valid).
CollectorKey	Report-AssetToShopDB.ps1	collector.ingest PAT. REQUIRED for asset reporting (allowlist does not cover the collector).

The key's ACL is restricted to SYSTEM + Administrators (the bootstrap does this) so the kiosk auto-login user cannot read the PATs.

The pieces on disk (kiosk layout, C:\ProgramData\GE-Enforce)

• Invoke-ShopdbEnforce.ps1 - the runner
• ShopdbEnforceClient.psm1 - the client module
• lib\Install-FromManifest.ps1 - the engine (>= 2.6)
• Report-AssetToShopDB.ps1 - the asset collector
• Cache: C:\ProgramData\ShopDB\geenforce\ (<scope>.json, .etag, .version, payloads\), logs C:\Logs\Shopfloor\

Scheduled tasks (SYSTEM, RunLevel Highest)

Task	Runs	Interval
ShopDB GE-Enforce	Invoke-ShopdbEnforce.ps1 -Scope <scope> -EnginePath <engine> -BaseUrl <url>	AtStartup + every 15 min
ShopDB Asset Report	Report-AssetToShopDB.ps1	AtStartup + every 60 min

Tokens are NOT in the task arguments (visible in task XML) - the scripts read them from HKLM.

The runner flow (Invoke-ShopdbEnforce.ps1)

1. Get-ShopdbConfig (params override registry). No BaseUrl -> exit 0, retry next cycle.
2. Sync-ShopdbManifest -Scope <scope>: ETag-conditional GET; 200 validates the JSON before overwriting the cache (a proxy error page served as 200 must not clobber last-known-good); 304 or any network failure -> cached copy. Nothing at all -> Windows event log entry (source ShopdbEnforce, id 1001) plus a best-effort failure report so it is visible server-side, then exit 0.
3. -ShadowMode (with -ShareManifestPath): Compare-ShopdbShadow logs name/order diffs, engine runs against the SHARE (zero behavior change). This is the first step of every cutover.
4. Cutover mode: optional -IncludeCommon merges the fleet common scope (Merge-ShopdbManifests: common's unique entries first, pctype wins on Name conflict). OFF by default - a scope is enforced ALONE and displays are self-sufficient. Then Resolve-ShopdbPayloads stages http/inline payloads (section 3).
5. Engine call (the integration point): & $EnginePath -ManifestPath <resolved> -PCType $Scope -InstallerRoot <payloads dir> -LogFile <log>.
6. ConvertTo-ShopdbSummary normalizes whatever came back (summary object, array of emitted objects, bare int, $null) into @{Installed;Skipped;Failed;Filtered;Results;EnforcerVersion}, then New-ShopdbReport maps to the lowercase wire contract and Send-ShopdbReport POSTs it. All best-effort; the whole script exits 0 no matter what (fail-safe: a broken web app never breaks a PC).

The engine contract (Install-FromManifest.ps1, lib 2.6)

Mandatory params: -ManifestPath, -InstallerRoot, -LogFile; optional -PCType, -PCSubType. Entry Types: MSI, EXE, CMD/BAT, PS1, INF, File, Registry. Detection: Registry, File, FileVersion, Hash, MarkerFile, ValueMatches, pnputil, Always. Filters: PCTypes (with old/new-name alias groups), TargetHostnames, TargetMachineNumbers, _CmmVersion. Exit 0/1/2 unchanged for the SMB path; NEW in the API cutover: the engine ends with Write-Output of a summary pscustomobject (Installed/Skipped/Failed/Filtered/EnforcerVersion/Results), which is the only thing on the success stream (logs go via Write-Host), so & $EnginePath ... captures it cleanly.

SelfHealed on a result means a REAL drift correction (a detected-missing entry that got reinstalled). Always/no-detection entries install every cycle by design and are not flagged, so the server-side status derivation (service.record_enforcement_report: failed > selfhealed > ok) stays honest.

The display dispatcher: server-resolved role, file fallback

For kiosks, per-subtype behavior does not fork the scope: ONE scope (gea-shopfloor-display), one inline dispatcher entry (built by plugins/geenforce/seed_display_scope.py) that resolves what the display should show at enforce time, in two steps:

1. Server (authoritative): GET $KioskBaseUrl/api/dashboarddefaults/display-role?fqdn=<own-fqdn>. This is a PUBLIC endpoint (no token). The server matches the FQDN against the dashboarddefaults table (IP fallback) and returns {role, path, businessunitid, businessunit}. Roles: dashboard, lobby, partskiosk. Changing a display's job is now a server-side edit; no touch on the PC.
2. Fallback (offline, or unmapped): the local C:\Enrollment\display-type.txt value against the DISPLAY_TYPE_TARGETS map baked into the script. If neither resolves, the dispatcher logs and configures nothing.

The FQDN is built as F<BIOS serial>.<domain> (GE device naming); the domain comes from HKLM DisplayFqdnDomain or the built-in default (device.geaerospace.net). DetectionMethod = Always, but the script is idempotent: it rewrites the all-users Startup shortcut (never Start-Process - see gotchas) only when the resolved target changed.

Legacy autostart self-heal

The old GE Aerospace Dashboard / Lobby Display Inno installers planted three autostarts each: a Public-Desktop .lnk, an all-users Startup .lnk, and an HKLM ...\CurrentVersion\Run value, all launching Edge at now-dead URLs (/shopfloor-dashboard/, /tv-dashboard/) which 404 to a white screen. Because those installers were 32-bit, the Run value was WOW64-redirected into HKLM\SOFTWARE\Wow6432Node\...\Run, invisible to 64-bit tooling - the reason it survived earlier cleanup. The dispatcher (build_dispatcher_script in seed_display_scope.py) now sweeps, every enforce cycle: both the native and Wow6432Node registry views, every loaded user hive (HKU), Run + RunOnce + Policies\Explorer\Run, matching by the legacy value names AND by any value pointing at the old URLs; plus every per-user and common Startup folder; then kills any running old-URL Edge. A read-only locator, pxe-images/github/find-legacy-kiosk-autostart.ps1, hunts all these locations (and Edge startup-URL policy, scheduled tasks, Assigned Access) when a straggler persists.

The kiosk shortcut is a direct Edge shortcut (no launcher/VBS); the fix ships by re-publishing the code-authored gea-shopfloor-display scope (seed_display_scope(publish=True)), not an import-share.

---

6. Bootstrap for share-less PCs

A share-less PC cannot pull its first files from the share, so the bootstrap itself is downloadable from the web app.

Install-ShopdbKiosk.ps1 (source of truth: the imaging share (shopdb-migration/kiosk-installer/)) is hosted at C:\inetpub\wwwroot\shopdb\installers\kiosk\ and downloadable at {BaseUrl}/installers/kiosk/Install-ShopdbKiosk.ps1. Run elevated on the PC:

Set-ExecutionPolicy Bypass -Scope Process -Force
$u = 'https://tsgwp00525.wjs.geaerospace.net/shopdb/installers/kiosk/Install-ShopdbKiosk.ps1'
Invoke-RestMethod $u -OutFile "$env:TEMP\Install-ShopdbKiosk.ps1"
& "$env:TEMP\Install-ShopdbKiosk.ps1" -DisplayType Lobby -CollectorKey 'shopdb_pat_...'
# add -ShopdbToken 'shopdb_pat_...' only if the subnet is NOT allowlisted

What it does (idempotent; re-running is also the manual update path):

1. Writes C:\Enrollment\display-type.txt (the subtype) and C:\Enrollment\pc-type.txt (the scope, default gea-shopfloor-display).
2. Writes HKLM BaseUrl [+ ApiToken] + CollectorKey, then locks the key ACL to SYSTEM + Administrators.
3. Downloads runner + module + engine + collector from {BaseUrl}/installers/kiosk/ over HTTPS (TLS 1.2 forced).
4. Registers the two SYSTEM tasks (section 5).
5. Starts both once so the PC is live immediately.

IIS prerequisite: the installers\kiosk web.config MUST carry <staticContent> MIME maps for .ps1/.psm1 (text/plain) or IIS 404.3s the downloads (see gotchas).

Delivery options for the bootstrap itself:

• Imaging-baked: the display image runs it (or lays down the same state) at imaging time - see project-display-self-contained.
• Installer push: Intune/hand-run the one-liner above on an already-deployed PC. This is how the pilot kiosks were done (shopdb-migration/run-on-kiosk-F.txt).

---

7. Asset reporting via the collector

Report-AssetToShopDB.ps1 (in the kiosk bundle; also deployed in the SMB common\ scope for the share fleet) POSTs the PC's identity to:

POST {BaseUrl}/api/collector/computers
X-API-Key: <collector.ingest PAT or COLLECTOR_API_KEY env key>

Auth (shopdb/core/api/collector.py, _check_collector_auth): a managed token scoped collector.ingest (Bearer or X-API-Key) OR the COLLECTOR_API_KEY/COLLECTOR_API_KEY_COMPUTERS env key. The GE-Enforce IP allowlist does NOT apply here - the collector always needs a key, read from HKLM CollectorKey (or the manifest entry's Args -ApiKey).

Schema (plugins/computers/plugin.py, get_collector_schema): identity field hostname (required); optional machinenumber, pctype, pcsubtype, serialnumber, loggedinuser, lastboottime, lastcheckin, ipaddress, vendorname, modelnumber, osname, installedsoftware, defaultprinter, printers. All lowercase concatenated (the project naming convention).

apply_collector_payload upserts idempotently by hostname (falls back to Asset.assetnumber), creates the Asset+Computer when missing, maps machinenumber -> Asset.assetnumber (imaging placeholder 9999 skipped both client- and server-side), pctype -> ComputerType via the settings mapping, and creates Vendor/Model/OS rows as needed. Fields not posted are not touched - a partial read never blanks a good value, so the script only includes fields it actually resolved.

Client details worth keeping: machine number resolution order is eDNC registry MachineNo (WOW6432Node then native) -> C:\Enrollment\cmm\cmmid.txt -> C:\Enrollment\machine-number.txt; the reported ipaddress is filtered to the corp ranges (same two CIDRs as the allowlist) so a machine-LAN controller NIC never lands in shopdb.

---

8. HARD-WON GOTCHAS

Read this section before touching ANY of the moving parts. Every bullet cost real debugging time. Format: symptom -> cause -> fix.

• PS crash "property 'X' cannot be found" under Set-StrictMode -> the module runs Set-StrictMode -Version Latest, and engine results/summaries arrive as EITHER hashtables or PSCustomObjects with varying key casing; direct $obj.Key access on an absent key throws -> route every dynamic property read through Get-ShopdbProperty (handles both shapes, case-insensitive, returns $null when absent). Never dot into parsed JSON or engine output directly.
• Register-ScheduledTask rejects the repeating trigger -> passing -RepetitionDuration [TimeSpan]::MaxValue serializes to an out-of-range Duration the Task Scheduler XML schema rejects -> use -RepetitionInterval ALONE; it defaults to indefinite repetition (verified Win11 / PS 5.1). See Register-SystemTask in Install-ShopdbKiosk.ps1.
• Engine exits 2 / "InstallerRoot not found" -> -InstallerRoot and -LogFile are MANDATORY engine params and InstallerRoot must EXIST -> the runner always passes both and pre-creates the payloads dir before the engine call. Any new caller must do the same.
• http payload "not found: C:...\C:..." (path doubling) -> the engine resolves entry paths as Join-Path $InstallerRoot <field>; writing the staged payload's ABSOLUTE path into the entry made the engine double it -> Resolve-ShopdbPayloads writes the LEAF filename only, and the runner sets -InstallerRoot to the payloads cache dir. Keep those two in lockstep.
• Manifest fetch "works" but parsing fails / cache garbage -> if the manifest is served with a non-JSON content type, PowerShell 5.1 Invoke-WebRequest .Content comes back as a byte[] instead of a string -> the server route returns mimetype='application/json' (see get_manifest); any mock server or proxy in the chain must do the same. The client also validates JSON before overwriting last-known-good.
• Bootstrap download 404 (HTTP 404.3) -> IIS refuses to serve unknown static extensions; .ps1/.psm1 have no default MIME map -> add <staticContent><mimeMap fileExtension=".ps1" mimeType="text/plain" /> (and .psm1) in the installers\kiosk web.config, with <remove> first if inherited.
• IP allowlist spoofable / mysteriously not matching -> raw X-Forwarded-For is attacker-controlled (proxies append; first hop is the caller's to write) -> _ip_allowlisted uses request.remote_addr via _trusted_client_ip, which is only correct because the IIS URL-Rewrite rule OVERWRITES X-Forwarded-For with REMOTE_ADDR and waitress trusts only 127.0.0.1 as proxy. The rule is a hard dependency: never remove it, and verify the spoof is closed after server changes (curl -H "X-Forwarded-For: 10.134.48.10" from a non-allowlisted host must get 401).
• Kiosk browser never appears though the dispatcher "ran fine" -> the enforce task runs as SYSTEM in session 0, which has no interactive desktop; Start-Process msedge.exe opens INVISIBLY there -> write an all-users Startup shortcut (C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup\ShopDB Kiosk.lnk) and let the auto-login user launch it in a visible session. This is why the dispatcher is shortcut-based.
• TLS/transport errors on older images ("could not create SSL/TLS secure channel") -> Windows PowerShell 5.1 does not reliably negotiate TLS 1.2 by default -> every network helper calls Set-ShopdbTls ([Net.ServicePointManager]::SecurityProtocol = Tls12) first; the bootstrap and collector force it too. Any new script that touches the API must do the same.
• Secrets readable by the kiosk user / visible in Task Scheduler -> default HKLM\SOFTWARE ACL grants BUILTIN\Users read, and task arguments are world-readable in the task XML -> restrict the HKLM:\SOFTWARE\GE\ShopDB key ACL to SYSTEM + Administrators (the bootstrap does), and NEVER put a token in a task's -Argument string - scripts read ApiToken/CollectorKey from HKLM at run time.
• Enforcement Reports show 0/0/0 with no per-entry detail -> the engine historically returned nothing on the success stream, so the runner had no counts to report -> the 2.6 engine emits the summary object as its ONLY Write-Output (all logging is Write-Host), and ConvertTo-ShopdbSummary tolerates non-compliant engines by zero-filling. If reports go 0/0/0 again, the engine on that PC is pre-summary - update it.
• Two kiosk browsers fighting / stale kiosk launch after retarget -> old installs left their own Startup launchers behind, in several flavors -> the dispatcher's sweep must match ALL of: single- AND double-dash -kiosk arguments (the regex -kiosk matches both), shortcuts whose args carry shopdb URLs (tsgwp00525, /shopdb/, the dead shopfloor-dashboard route), the imaging installers' GE Aerospace Dashboard* / GE Aerospace Lobby* shortcut names, and .url files pointing at the kiosk routes. Extend the sweep whenever a new launcher naming appears.

---

9. How this was verified

Two complementary verification passes; keep BOTH for future cohorts because they catch different bug classes:

• Code review finds logic bugs: the XFF spoof hole, the StrictMode absent-key crashes, the status-derivation trap (installed>0 is not self-heal), the report casing mismatch.
• VM smoke test finds integration/OS-version bugs: the RepetitionDuration serialization rejection, the .ps1 MIME 404, session-0 invisibility, TLS negotiation, byte[] vs string response content - none of which a read of the code surfaces.

The VM rig:

• The win11 virt-manager VM (see project-win11-vm / reference-vm-qga-as-system memory), driven by the imaging share (ednc-bins/qga-run.py) - qemu guest agent guest-exec, which runs PowerShell AS SYSTEM. That matters: the scheduled tasks run as SYSTEM, so testing as SYSTEM reproduced the session-0 and profile-less behaviors an interactive test would have masked.
• A mock HTTP server on the host exposing /api/geenforce/manifest (served application/json), /api/geenforce/payload/<sha>, and capturing the /api/geenforce/report POST body. This exercised the full client loop - ETag/304, cache fallback (kill the server mid-test), payload hash verification, resolved-manifest rewrite, engine run, summary -> report mapping - without touching prod.
• Full pass = bootstrap installer run end to end in the VM, then assert: both tasks registered, HKLM written + ACLed, manifest cached, payloads staged by sha, Startup shortcut written, report captured with real counts.

---

10. Deploy

TWO independent channels. Confusing them is the classic mistake: the client/ engine/collector/bootstrap are pxe-images SHARE artifacts, NOT deployed by the git pipeline.

Channel 1: backend + frontend (the git .cmd pipeline)

Prod is air-gapped from dev; code moves via a git bundle on the share (\\172.16.9.9\pxe-images\github\shopdb-flask-pub.bundle) and the .cmd scripts in the imaging share (github/), run on the work PC:

1. pull-shopdb-bundle.cmd - fetch the bundle into the local clone (ff-only).
2. update-shopdb-github.cmd - push the clone to GitHub.
3. update-dev-server.cmd - robocopy the clone to X: (dev /ops tree) + the /ops-base frontend dist. Validate on /ops FIRST.
4. update-prod-server.cmd - robocopy to Y: (C:\inetpub\wwwroot\shopdb)
   • the /shopdb-base dist (frontend-dist-subpath-shopdb). Both scripts carry instance guards (web.config MOUNT_PATH check) - do not bypass them.

Then RDP: Restart-WebAppPool shopdbflask-prod, and if migrations/deps changed: flask db upgrade, flask plugin upgrade-all, flask seed permissions, flask seed settings.

(The fast path used during the pilot - robocopy just the changed plugin files from shopdb-migration\prod-patch-geenforce\ + pool restart, per deploy-server-patch.txt - works, but the same commits must ALSO go through the bundle pipeline or prod drifts from git.)

Channel 2: client + engine + collector + bootstrap (the kiosk bundle)

These live at the imaging share (shopdb-migration/kiosk-installer/) and deploy by robocopy from the work PC (Z: = share, Y: = prod app dir):

robocopy Z:\shopdb-migration\kiosk-installer Y:\installers\kiosk /E

That directory (bundle contents: Install-ShopdbKiosk.ps1, Invoke-ShopdbEnforce.ps1, ShopdbEnforceClient.psm1, lib\Install-FromManifest.ps1, Report-AssetToShopDB.ps1, web.config with the MIME maps) IS the distribution point - PCs download from {BaseUrl}/installers/kiosk/. Reference copies of the client kit also live in the repo at plugins/geenforce/client/ and the engine's source of truth is the imaging share, common/lib/Install-FromManifest.ps1; when the engine or client changes, update the kiosk bundle copy too (nothing syncs it automatically). PCs pick up new bytes by re-running the bootstrap one-liner.

Server prerequisites (once per site, all three or token-less clients 401)

1. Publish the scope(s) - seed_display_scope(publish=True) or flask geenforce publish <scope>.
2. Seed geenforce_allowed_cidrs = 10.134.48.0/23,10.48.249.0/26 (Settings rail > GE-Enforce Settings, or SQL upsert into settings).
3. Mint tokens (Settings > API Tokens, Restrict permissions ON): collector.ingest (required, the kiosk -CollectorKey) and geenforce.fetch (fallback for non-allowlisted subnets; resource-bind it to the scope).
4. Keep the IIS URL-Rewrite XFF-overwrite rule enabled (section 2).

---

11. PLAYBOOK: extending to a new pc-type

Checklist for cutting any of the remaining scopes (gea-shopfloor-cmm, -collections, -keyence, -genspect, -heattreat, -partmarker, -nocollections, common) over to the API.

1. Decide the delivery model. Does this pc-type keep SMB access? If yes, the cheap cutover is manifest-over-API + payloads-still-smb (entries stay smb, nothing to upload, the engine resolves share paths as today). Only a genuinely share-less PC needs http/inline payload conversion. Note the payload endpoint's 512 MB default ceiling (GEENFORCE_PAYLOAD_MAX_BYTES) before promising huge installers over HTTPS.
2. Get the scope into shopdb. Existing share manifest: flask geenforce parity then flask geenforce import-share --scope <name>. New/reworked scope: author in code following seed_display_scope.py (registry heals with ValueMatches detection, idempotent Always PS1s, data-driven tables for anything per-subtype).
3. Convert payloads (share-less only). Small scripts/configs -> store_inline_payload / the entry payload upload endpoint. Installers -> flask geenforce add-payload <file>, set PayloadSource=http + PayloadSha256 + PayloadRef on the entry. Remember: PayloadRef's extension decides the staged filename's extension.
4. Publish. New version every publish; clients converge within one enforce cycle. Verify with curl "{BaseUrl}/api/geenforce/manifest?pctype=<scope>" from an allowlisted host.
5. Auth for the PCs. Subnet already inside 10.134.48.0/23,10.48.249.0/26 -> token-less, nothing to do. New subnet -> add its CIDR to geenforce_allowed_cidrs (Settings rail validates). Not network-trustable -> mint a geenforce.fetch token resource-bound to the scope and deliver it to HKLM ApiToken.
6. Bootstrap the client. Share-attached fleet: adapt the dispatcher / Install-GEEnforce.ps1 path (the pilot flow in shopdb-migration/kiosk-api-pilot.txt: shadow first, then flip, then DISABLE the old share enforce task so the two do not fight). Share-less: the Install-ShopdbKiosk.ps1 pattern - generalize -Scope and skip the display-only pieces. Decide -IncludeCommon: displays run without it; a non-display share-less PC that needs the fleet-wide common entries over HTTPS turns it on AND requires common's entries to be payload-converted first (an SMB-payload common entry will fail on a share-less PC).
7. Run SHADOW mode first on one pilot PC (-ShadowMode -ShareManifestPath <share manifest>): fetch + compare + report with zero behavior change. Watch the shadow diff log lines and the Enforcement Reports row before flipping.
8. Re-read section 8 (gotchas). Especially: LEAF filenames, StrictMode property access, SYSTEM/session-0, task trigger serialization, MIME maps if you host new downloadables.
9. Verify on the VM (section 9) before the pilot PC: bootstrap + enforce cycle against a mock or the dev /ops instance, as SYSTEM via qga-run.py.
10. Pilot one PC, then the cohort. Keep the rollback in your pocket: disable/remove the new task, re-enable the share task, remove HKLM BaseUrl - the share path is untouched by all of this.
11. pc-type mapping. Make sure the collector's pctype mapping (computers plugin settings, pctypemap) covers the scope name so asset reports do not warn no ComputerType mapping.

---

12. Open items / TODO

• Name resolution for reported users. loggedinuser lands as a bare username; resolving it to a display name depends on either the wjf_employees First_Name/Last_Name data or shopdb User accounts existing for shopfloor users. Not wired up; reports show raw usernames until it is.
• Old imaging-installer registry cleanup. Displays imaged before the API cutover carry leftover state from the old imaging-time kiosk installers (superseded shortcuts are already swept by the dispatcher; stale registry values are not yet cleaned). A cleanup entry in the display scope is the natural vehicle.
• The SMB fleet is still on the share. Only displays/kiosks are on the API. cmm/collections/keyence/genspect/heattreat/partmarker/nocollections/ common still enforce from SFLD; section 11 is the path. Shadow mode makes each migration observable before it changes anything.
• 3DPrintRoom kiosk target is a placeholder. DISPLAY_TYPE_TARGETS points it at /parts-kiosk; confirm the real route with the floor team before publishing to production 3D-print-room displays (flagged in seed_display_scope.py).
• Fast-path prod patches vs git. The prod-patch-geenforce robocopy path can leave prod ahead of the repo; reconcile by pushing the same changes through the bundle pipeline (section 10, channel 1).