From f442ff2f65d4b4b40594c111317b4b671179c651 Mon Sep 17 00:00:00 2001 From: cproudlock Date: Thu, 6 Aug 2026 15:02:08 -0400 Subject: [PATCH] Propose zero-touch, ShopDB-driven imaging Design proposal, nothing built. Two ideas in one architecture: the image becomes thin (partition, apply, join Entra, install the enforce client - everything else is data pulled from the API and enforced by GE-Enforce), and imaging becomes a job rather than a menu (open a machine in ShopDB, click Create this PC, pick the zeroed box waiting in WinPE, and it images start to finish reporting progress onto that machine's record). Every claim about current behaviour is sourced from 2026-08-06 evidence - decompiled binaries, live logs off bay 579C144, and the live share - and where something is inferred it says so. Covers: why the current early-decision chain fails silently (five worked examples from one day); what already exists to build on; per-machine configuration records replacing 147 .reg files and two CSVs, referencing payloads rather than embedding them; the dual-NIC design with the firewall work that gates it; whether PESetup needs replacing at all; the BPRT token shelf-life constraint on pre-imaged stock; an ownership table; six delivery phases each useful alone; risks; and seven open questions. Two findings in here are worth acting on before any of it: disableauto.json is ABSENT from the media and AutoStartCountDown is 31, while the operator on 579C144 clicked Next at 11 seconds. PESetup may already auto-start and nobody has waited long enough to find out. One bay, hands off the keyboard, count to 40. dnsmasq is already safely bound (listen-address + bind-dynamic) so DHCP will not leak onto a GE-side NIC - but Samba binds to all interfaces and every ufw rule is ALLOW IN Anywhere, including SSH on a pxe/pxe account. That work gates dual-homing. --- docs/ZERO-TOUCH-IMAGING-PROPOSAL.md | 468 ++++++++++++++++++++++++++++ 1 file changed, 468 insertions(+) create mode 100644 docs/ZERO-TOUCH-IMAGING-PROPOSAL.md diff --git a/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md b/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md new file mode 100644 index 0000000..c4a8f05 --- /dev/null +++ b/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md @@ -0,0 +1,468 @@ +# Zero-touch, ShopDB-driven imaging - proposal + +**Status:** proposal, nothing built +**Written:** 2026-08-06 +**Author's note:** every claim about current behaviour below is from evidence +gathered on 2026-08-06 - decompiled binaries, live logs off bay 579C144, and the +live share. Where something is inferred rather than observed it says so. + +--- + +## 1. What we want + +Two ideas, one architecture. + +**A. The image becomes thin.** It carries only what must happen at image time - +partition, apply the WIM, join Entra, install the enforce client. Everything else +(PC type, subtype, tool association, apps, settings) is data, pulled at first +boot from the ShopDB-Flask API and enforced continuously by GE-Enforce. + +**B. Imaging becomes a job, not a menu.** Open a machine in ShopDB, click +*Create this PC*, pick the zeroed box that is sitting in WinPE waiting, and it +images start to finish with nobody touching the keyboard - reporting progress +onto that machine's record in real time. + +Between them: no menus in WinPE, no per-type media, no decisions baked into text +files by whoever happened to be standing at the bay. + +--- + +## 2. Why - what today actually costs + +Today every decision is made in WinPE, written to files, and carried through a +chain where each link can fail silently: + +``` +startnet.cmd -> unattend (4 passes) -> FirstLogonCommands (18 orders) + -> ppkg -> DSC -> GE-Enforce +``` + +On 2026-08-06 alone, that chain produced: + +| Failure | Root cause | How long it hid | +|---|---|---| +| Every build stopped at a dialog | one unattend `Path` at 676 chars against a 259 limit | ~1 day, and it masked the next two | +| Bays staged nothing, silently | a volume scan raced PESetup's `diskpart` and staged into a partition being erased | weeks (4 Display bays sat "green") | +| Deployment destroyed 4s after autologon | an at-logon task ran `-ManualFallback`, which is `sysprep /oobe /reboot` | until traced through Panther | +| Machines named `E` not `F` | our script re-applied a rename over the package's own | unknown | +| Six copies reported success having moved 0 bytes | unconditional `echo` after `robocopy` | weeks | + +The pattern is identical in every case: **a decision made early, carried far, and +verified nowhere.** Moving decisions to a running, networked machine with an API +and a UI makes them recoverable - re-runnable, correctable, and observable. + +--- + +## 3. What we already have to build on + +This is not a greenfield project. Most of the mechanism exists. + +### 3.1 ShopDB-Flask +- Assets keyed by serial; CMM bays, measuring tools, machine numbers already modelled +- Collector API: `POST /shopdb/api/collector/computers` (X-API-Key), ingests inventory by serial +- GE-Enforce plugin: scopes, manifests, **content-addressed payload store** + (`instance/geenforce/payloads/`), `GET /api/geenforce/payload/` +- Display scope (`plugins/geenforce/seed_display_scope.py`) already proves the pattern: + read a local subtype, prefer a **server-side role resolved by device IP**, pull + payloads over HTTPS, write the kiosk shortcut, sweep stale ones, self-heal each cycle +- Per-device role API: `GET /api/dashboarddefaults/display-role` + +### 3.2 PXE server +- Per-bay state: `/var/log/pxe-imaging/.json` with stage history +- `winpe-status-push.ps1` already POSTs stage updates from WinPE +- Data-driven menus already: `menu.json` on the share, rendered by + `select-shopfloor-type.ps1`, edited by the webapp +- Imaging dashboard with per-bay tiles +- As of today: staging verification, imaging-log harvest to + `enrollment/imaging-logs//`, `preflight.ps1` at the bay, + `preflight.py` + `lint-unattend.py` + `lint-driver-catalogue.py` + + `share-drift.py` on the server + +### 3.3 Hardware +- Second NIC already present and unused: `enp0s31f6` (onboard, `e1000e`, currently down) +- PXE LAN currently on a USB adapter, `enx34c8d6b11010` +- That adapter was silently capped at USB 2.0 (327 Mb/s measured); moved to a + USB 3 port on 2026-08-06 and now runs at line-rate gigabit (**937 Mb/s measured**) + +--- + +## 4. Target architecture + +### 4.1 Three (or one) pre-imaged types + +Proposed: **Displays, Inspection, Shopfloor** as pre-imaged stock, with the +subtype chosen later by the user at first boot. + +**Open question worth settling early:** if the base image is identical and all +configuration comes from the API, what actually differs between the three? +Drivers are selected by PESetup per *model*, not per type. Unless disk layout or +a must-be-offline application differs, **one image** is better - three media trees +mean three things to keep in sync, and drift is what caused the day-long outage. + +### 4.2 First-boot selection + +A friendly PS1 runs when no PC type is recorded yet: + +``` +Inspection -> CMM | Genspect | Wax Trace | Keyence + └─ CMM / Keyence / WaxTrace -> which measuring tool? + (list served from ShopDB assets, not a CSV) +Displays -> Lobby | Dashboard | 3D Print Kiosk +Shopfloor -> Collections | No Collections | Common | Heattreat | Part Marker | ... +``` + +The tool selection is the important half: it drives PC-DMIS version, FormTracePak +version, DODA flag, NTLARS/eDNC registry, UDC settings, machine number. + +**Design points:** + +- **Server is the source of truth.** The selection POSTs to ShopDB keyed on + serial; the local file is a cache. A mis-selection is then fixable from a + browser, not by walking to the bay. +- **Re-runnable.** Available from the Start menu, shows what was chosen and what + was installed. People pick wrong; that must not mean re-imaging. +- **Audited.** ShopDB records who chose what and when - that is what separates + "misconfigured" from "chose wrong" six months later. +- **Do not require the LAPS password.** Handing a shop-floor user the local admin + password to run a selection dialog is a real control change, and the app runs + elevated. Prefer a SYSTEM-run UI on the logon desktop, or a kiosk shell. + +### 4.3 Per-machine configuration records + +Today a machine's identity is scattered: + +| What | Where today | Size | +|---|---|---| +| eDNC / NTLARS registry | `_ntlars-backups/` - 147 per-bay `.reg` files | ~1.5 MB total | +| WaxTrace bay config | `bay-config.csv` - 15 bays, 7 FTPak versions, per-bay user ID | small | +| CMM bay config | `cmm-bay-config.csv` -> version.txt, doda.txt | small | +| UDC settings | `pre-install/udc-backups/`, per-bay | ~240 MB per bay | +| Machine number | `machine-number.txt` written by startnet | trivial | + +Every bay gets **all** of it staged, then picks its own. + +Proposed: one record per machine in ShopDB. + +``` +ShopDB asset (key: serial and/or machine number) + ├─ pctype + subtype -> selects the GE-Enforce scope + ├─ tool association -> CMM bay, Keyence model, FTPak version, DODA + ├─ machine number -> UDC / eDNC naming + └─ payload refs (sha256) -> NTLARS .reg, UDC backup, per-bay settings +``` + +**Reference payloads, do not embed them.** NTLARS `.reg` files carry binary +values and UDC backups are hundreds of MB. GE-Enforce already has a +content-addressed blob store with an HTTP fetch endpoint - the record names the +payload, the store holds it. + +This also removes a live bug class. `Restore-UDCData.ps1` and +`Update-MachineNumber.ps1` each mount SMB backup shares on their own drive +letters, which collided with GE-Enforce's `W:` and killed the manifest entry +running next (fixed in `66c24b5` by moving them to `R:` and `N:`). Payloads over +HTTPS remove drive-letter juggling entirely. + +Migration is a one-time import of two CSVs and 147 `.reg` files. A script, not a +project. + +--- + +## 5. Zero-touch: ShopDB-triggered imaging + +### 5.1 Flow + +``` +1. Zeroed PC, boot order = IPv4 only. PXE boots into WinPE. +2. WinPE reports serial + MAC + model, then WAITS. +3. ShopDB shows it under "awaiting assignment". +4. Operator opens the target machine -> "Create this PC" -> picks the waiting box. +5. WinPE polls, receives a job: + { pctype, subtype, tool/bay, tenant, purpose, target asset id } + and images with NO menus. +6. Progress streams to that machine's record in real time. +7. On completion the PC binds to the asset record. +``` + +Every menu in `startnet.cmd` becomes a field in the job. If no job exists, it +falls back to today's interactive menu and displays its own serial so one can be +created. + +### 5.2 Safety - this is a remote-triggered disk wipe + +Clicking a web button destroys a machine, and the target is chosen from a list. +The nightmare is picking the wrong row. + +**Mitigation:** the waiting PC displays a short code on screen; the job cannot be +claimed unless that code matches what the operator confirms in ShopDB. Cheap, and +it makes wrong-machine wipes structurally hard rather than merely unlikely. + +Additionally: a job may only target a machine currently in the waiting state, and +the wipe must be logged against both the operator and the asset. + +### 5.3 Real-time record updates + +Each phase already produces data; it simply is not joined up. + +| Phase | Existing source | Lands on the record as | +|---|---|---| +| PXE boot | dnsmasq lease + WinPE check-in | serial, MAC, model, "imaging started" | +| WIM apply | `winpe-status-push.ps1` | live stage / progress | +| Staging | `winpe-staging.log` + harvested bundle | what payload actually landed | +| First boot | collector API | CPU, RAM, disk, OS build | +| Enrollment | `criticalChecks.json`, `TokenMatch.json` | Entra join state, tenant, purpose | +| Steady state | GE-Enforce report | installed apps, drift, self-heal events | + +End state is one timeline per machine: *imaged 14:32 -> staged +gea-shopfloor-cmm, bay CMM4 -> enrolled GCCH_SH -> PC-DMIS 2019 R2 installed -> +enforcing clean since*. + +**Two rules so it does not rot:** + +1. **Push, best-effort.** If ShopDB is unreachable the bay must still image. + Queue events on the PXE server and drain them; the harvested logs are the + durable local copy. +2. **One writer per field.** Imaging status from the PXE server, inventory from + the collector, app state from GE-Enforce. Two writers on one field is the + exact bug that ate 2026-08-06. + +--- + +## 6. Network design + +### 6.1 Dual-homed PXE server + +- **PXE side** - isolated `172.16.9.0/24`, dnsmasq DHCP bound to that adapter +- **GE side** - corporate, reachable from anywhere; carries ShopDB API traffic, + the job queue, status pushes, and (if wanted) the BPRT token fetch + +The onboard `enp0s31f6` is free for this today. + +### 6.2 Security prerequisites - these gate the whole thing + +Checked 2026-08-06: + +**Already safe.** dnsmasq has `listen-address=172.16.9.1` + `bind-dynamic`, so it +binds only to the interface holding that address. DHCP will **not** leak onto a +GE-side NIC. This was the failure I most expected and it is already handled. + +**Not safe yet:** + +``` +smb.conf no `interfaces =` / `bind interfaces only` -> Samba listens on ALL interfaces +ufw every rule is ALLOW IN Anywhere, not interface-scoped: + 22/tcp SSH (account is pxe/pxe) + 445/tcp shares (pxe-upload, password in scripts) + 69/udp TFTP + 9009,9011 webapp +``` + +The moment a GE-side interface comes up, all of that is corporate-reachable. + +**Required before dual-homing:** +1. Scope every UFW rule to the PXE interface (`ufw allow in on ...`) +2. `interfaces = ` + `bind interfaces only = yes` in `smb.conf` +3. Expose deliberately on the GE side only what must be - realistically the + webapp, behind real authentication +4. Reconsider SSH credentials if the host becomes corporate-reachable + +*(Correction to an earlier note: the shares are **not** guest-accessible - +`guest ok = no` on every share, with `map to guest = bad user`. They require the +`pxe-upload` account. That account's password is in scripts, so it is not a +control once 445 is corporate-reachable.)* + +### 6.3 What dual-homing unlocks + +- Job queue sync and live status without the share as intermediary +- BPRT token fetch (`mcl.dwcdn.geaerospace.com`) reachable at imaging time +- Possibly the Entra join during imaging rather than after re-cabling - see §8.2 + +--- + +## 7. The PESetup question + +### 7.1 It may already be automatic + +`AutoStartCountDown = 31` (decompiled), and **`disableauto.json` is absent** from +the media - its *presence* is what disables autostart. Now the observed log: + +``` +11:30:07.74 'Selection View' started (LTSC warning displayed) +11:30:18.63 Btn 'Next' pressed. +``` + +The operator clicked at **11 seconds**. Nobody has waited past 31. + +**Test first:** one bay, hands off the keyboard, count to 40. If it proceeds, +zero-touch needs no new imaging code at all. + +### 7.2 If it does hang, re-implementation is tractable + +Nothing in the mechanism is proprietary - only the data, which we already own and +already parse. + +| PESetup step | Equivalent | +|---|---| +| `PrepareDisk` | `diskpart /s diskpartEFI.txt` | +| `PrepareUnattend` | token substitution (`%serialnumber%`, `*arch*`) + `Packages.xml` merge | +| `CopyTools` / `CopyPackages` / `CopyHWPackages` | robocopy | +| `CopyDrivers` | select by model from `HardwareDriver.json`, unzip | +| `ApplyImage` | `DISM /Apply-Image` per `OperatingSystem.json` | +| `ApplyPackages` / `ApplyLanguagePacks` | `DISM /Add-Package` | +| `ApplyUnattend` | `DISM /Apply-Unattend` | +| `CreateBCD` / `CopyWinRE` | `bcdboot`, `reagentc` | + +A few hundred lines of PowerShell. We would also get to fix the traps documented +in `docs/PESETUP-INTERNALS.md`: substring first-match-wins driver matching, a +family filter that knows only Latitude/OptiPlex/Precision, and a driver miss +being a **warning** rather than a failure. + +**The cost is ownership, not effort.** `Release.json` points at MCL auto-update +URLs, so GE expects this media to self-update. Diverging means tracking their +format changes indefinitely, plus a likely compliance conversation about the +mandated imaging tool. Hence: run the 40-second test first. + +### 7.3 What we would keep either way + +Image-time preflight stays valuable regardless: secure boot (PESetup hard-fails +without it), >=120 GB disk (`MinRequiredSpaceWithoutCompression`), driver match +for the model, media age (30-day expiry). Already built as +`playbook/scripts/preflight.ps1`. + +--- + +## 8. Enrollment + +### 8.1 How it works today + +- Six MCL packages on the media: `PERS` / `SH` / `LOAN` x `GCCH` / `RoW`, plus + the SFLD package for shopfloor +- `Start-BulkEnrollOrchestrator.ps1` asks tenant + purpose via + `Select-PCConfig.ps1` (a GUI), then applies the matching package +- `Insert-BPRTToPPKG.ps1` fetches an **encrypted token table** from + `mcl.dwcdn.geaerospace.com`, decrypts with a static passphrase + (PBKDF2 310k / SHA-256 / AES-CBC), matches on **Tenant AND Purpose**, injects + the token and stamps the real expiry into the filename + (`Exp_XXXXXXXX` -> `Exp_yyyyMMdd`) +- The package itself sets `F%SERIAL%` and + performs the Entra join +- A human then assigns the device category in Intune + +As of 2026-08-06 the tenant/purpose choice can be pre-seeded at the PXE menu +(`C:\Enrollment\pc-config.txt`, consumed by a shim at the path the orchestrator +already calls). Injection deliberately stays at first logon - it needs internet. + +### 8.2 The constraint that shapes pre-imaged stock + +The BPRT token is stamped into the package with an expiry. A PC imaged, powered +off, and left in a cupboard for three months has a **dead token** and will never +join - and the failure looks identical to "not on the production network yet", +because `Entra ID Joined: false` right after imaging is normal. + +Three options, pick deliberately: + +1. **Join at imaging time** - requires the production network at the bench + (which dual-homing could provide), and pre-imaged stock is then already joined +2. **Join at first power-on** - cleaner logistically, but stock has a shelf life + bounded by token expiry +3. **Re-inject on demand** - first boot detects a dead token and fetches a fresh + one; needs the machine to reach the token blob + +Whichever is chosen, the *silent* failure must be closed: a check that +distinguishes "token expired" from "no network yet". + +--- + +## 9. Ownership model + +Four times on 2026-08-06 two systems owned one thing and the last writer won +silently. A clean slate is the moment to fix that. See `docs/OWNERSHIP.md`. + +| Concern | Owner | Not | +|---|---|---| +| Computer name | the provisioning package (`F%SERIAL%`) | any script in this repo | +| Drive letters during imaging | PESetup (`W:`, hardcoded 9 places) | anything running `diskpart` concurrently | +| Entra enrollment | the package + orchestrator | `run-enrollment.ps1` | +| Kiosk / display targets | GE-Enforce scope | `site-config.json` (backstop only) | +| Imaging status | PXE server | the collector | +| Inventory | collector API | the PXE server | +| Installed app state | GE-Enforce reports | anything else | +| Job queue | **ShopDB** (owns assets) | the PXE webapp (executor only) | + +--- + +## 10. Phased delivery + +Each phase is useful standing alone. No phase requires the next. + +**Phase 0 - settle the unknowns (hours)** +- The 40-second PESetup autostart test +- Decide: one image or three +- Decide: join at imaging time, at first power-on, or re-inject + +**Phase 1 - visibility (small)** +- WinPE reports serial / MAC / model on boot; "awaiting assignment" list in the webapp +- Relay imaging status to ShopDB over the GE side +- *Useful immediately: you can see what is booting and what it did.* + +**Phase 2 - dual-home safely (small, gating)** +- Interface-scope UFW, bind Samba, bring up `enp0s31f6` +- Nothing else proceeds safely until this is done + +**Phase 3 - job queue** +- Job model + API in ShopDB (create / claim / complete), with the on-screen + confirmation code +- WinPE polls once where the menu is today; falls back to the menu if no job + +**Phase 4 - first-boot selection** +- Selection UI, server-backed lists, POST the choice, re-runnable +- Retire the WinPE sub-menus for subtype and tool association + +**Phase 5 - per-machine records** +- Import the CSVs and `.reg` files into ShopDB with payload refs +- Retire `bay-config.csv`, `cmm-bay-config.csv`, and staging all 147 `.reg` files + to every bay + +**Phase 6 - thin the image** +- Remove per-type media (`_media/`), the staging block, the Office ppkg + variants, the CMM bay picker, the WaxTrace ISO cherry-pick +- WinPE's job becomes: partition, apply, join, install enforce client, reboot + +--- + +## 11. Risks + +| Risk | Why it matters | Mitigation | +|---|---|---| +| Remote-triggered wipe hits the wrong machine | destructive, irreversible | on-screen confirmation code; job may only target a waiting machine; log against operator and asset | +| ShopDB becomes a single point of failure | today a bay configures from the share with ShopDB down | GE-Enforce ETag / last-known-good cache; explicit failure UI, never a silent stall | +| Dual-homing exposes the PXE server | SSH `pxe/pxe`, SMB, TFTP currently `ALLOW IN Anywhere` | Phase 2 gates everything | +| BPRT expiry on shelf stock | silent, indistinguishable from "no network yet" | pick a join strategy (§8.2) and add an explicit expiry check | +| Selection UI becomes the new single point of correctness | wrong subtype = wrong PC-DMIS, wrong bay config | re-runnable, server-corrected, audited | +| Re-implementing PESetup | ongoing format tracking + compliance | run the autostart test first | +| Big payloads over the corporate network | PC-DMIS, FormTracePak ISOs (~2 GB each), Keyence | GE-Enforce supports `smb` / `http` / `inline` per entry - decide per payload | +| Three images drift apart | drift caused the 2026-08-06 outage | prefer one image; `share-drift.py` guards what remains | + +--- + +## 12. Open questions + +1. One image or three? What genuinely differs between them? +2. Join at imaging time, at first power-on, or re-inject on demand? +3. Does PESetup auto-start after 31 seconds? +4. Is replacing the GE-mandated imaging tool acceptable, if it comes to that? +5. Does the selection UI run as SYSTEM (no LAPS password), or as an admin user? +6. Which payloads stay on SMB and which move to HTTPS? +7. Who operates the job queue day to day - is "Create this PC" a technician + action, or does it need approval? + +--- + +## 13. References + +- `docs/PESETUP-INTERNALS.md` - decompiled imaging tool behaviour +- `docs/OWNERSHIP.md` - who owns what, with the 2026-08-06 collisions +- `scripts/preflight.py`, `scripts/lint-unattend.py`, + `scripts/lint-driver-catalogue.py`, `scripts/share-drift.py` +- `playbook/scripts/preflight.ps1` - bay-side checks +- shopdb-flask: `plugins/geenforce/seed_display_scope.py` - the pattern this + proposal generalises