diff --git a/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md b/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md new file mode 100644 index 0000000..c4a8f05 --- /dev/null +++ b/docs/ZERO-TOUCH-IMAGING-PROPOSAL.md @@ -0,0 +1,468 @@ +# Zero-touch, ShopDB-driven imaging - proposal + +**Status:** proposal, nothing built +**Written:** 2026-08-06 +**Author's note:** every claim about current behaviour below is from evidence +gathered on 2026-08-06 - decompiled binaries, live logs off bay 579C144, and the +live share. Where something is inferred rather than observed it says so. + +--- + +## 1. What we want + +Two ideas, one architecture. + +**A. The image becomes thin.** It carries only what must happen at image time - +partition, apply the WIM, join Entra, install the enforce client. Everything else +(PC type, subtype, tool association, apps, settings) is data, pulled at first +boot from the ShopDB-Flask API and enforced continuously by GE-Enforce. + +**B. Imaging becomes a job, not a menu.** Open a machine in ShopDB, click +*Create this PC*, pick the zeroed box that is sitting in WinPE waiting, and it +images start to finish with nobody touching the keyboard - reporting progress +onto that machine's record in real time. + +Between them: no menus in WinPE, no per-type media, no decisions baked into text +files by whoever happened to be standing at the bay. + +--- + +## 2. Why - what today actually costs + +Today every decision is made in WinPE, written to files, and carried through a +chain where each link can fail silently: + +``` +startnet.cmd -> unattend (4 passes) -> FirstLogonCommands (18 orders) + -> ppkg -> DSC -> GE-Enforce +``` + +On 2026-08-06 alone, that chain produced: + +| Failure | Root cause | How long it hid | +|---|---|---| +| Every build stopped at a dialog | one unattend `Path` at 676 chars against a 259 limit | ~1 day, and it masked the next two | +| Bays staged nothing, silently | a volume scan raced PESetup's `diskpart` and staged into a partition being erased | weeks (4 Display bays sat "green") | +| Deployment destroyed 4s after autologon | an at-logon task ran `-ManualFallback`, which is `sysprep /oobe /reboot` | until traced through Panther | +| Machines named `E` not `F` | our script re-applied a rename over the package's own | unknown | +| Six copies reported success having moved 0 bytes | unconditional `echo` after `robocopy` | weeks | + +The pattern is identical in every case: **a decision made early, carried far, and +verified nowhere.** Moving decisions to a running, networked machine with an API +and a UI makes them recoverable - re-runnable, correctable, and observable. + +--- + +## 3. What we already have to build on + +This is not a greenfield project. Most of the mechanism exists. + +### 3.1 ShopDB-Flask +- Assets keyed by serial; CMM bays, measuring tools, machine numbers already modelled +- Collector API: `POST /shopdb/api/collector/computers` (X-API-Key), ingests inventory by serial +- GE-Enforce plugin: scopes, manifests, **content-addressed payload store** + (`instance/geenforce/payloads/`), `GET /api/geenforce/payload/` +- Display scope (`plugins/geenforce/seed_display_scope.py`) already proves the pattern: + read a local subtype, prefer a **server-side role resolved by device IP**, pull + payloads over HTTPS, write the kiosk shortcut, sweep stale ones, self-heal each cycle +- Per-device role API: `GET /api/dashboarddefaults/display-role` + +### 3.2 PXE server +- Per-bay state: `/var/log/pxe-imaging/.json` with stage history +- `winpe-status-push.ps1` already POSTs stage updates from WinPE +- Data-driven menus already: `menu.json` on the share, rendered by + `select-shopfloor-type.ps1`, edited by the webapp +- Imaging dashboard with per-bay tiles +- As of today: staging verification, imaging-log harvest to + `enrollment/imaging-logs//`, `preflight.ps1` at the bay, + `preflight.py` + `lint-unattend.py` + `lint-driver-catalogue.py` + + `share-drift.py` on the server + +### 3.3 Hardware +- Second NIC already present and unused: `enp0s31f6` (onboard, `e1000e`, currently down) +- PXE LAN currently on a USB adapter, `enx34c8d6b11010` +- That adapter was silently capped at USB 2.0 (327 Mb/s measured); moved to a + USB 3 port on 2026-08-06 and now runs at line-rate gigabit (**937 Mb/s measured**) + +--- + +## 4. Target architecture + +### 4.1 Three (or one) pre-imaged types + +Proposed: **Displays, Inspection, Shopfloor** as pre-imaged stock, with the +subtype chosen later by the user at first boot. + +**Open question worth settling early:** if the base image is identical and all +configuration comes from the API, what actually differs between the three? +Drivers are selected by PESetup per *model*, not per type. Unless disk layout or +a must-be-offline application differs, **one image** is better - three media trees +mean three things to keep in sync, and drift is what caused the day-long outage. + +### 4.2 First-boot selection + +A friendly PS1 runs when no PC type is recorded yet: + +``` +Inspection -> CMM | Genspect | Wax Trace | Keyence + └─ CMM / Keyence / WaxTrace -> which measuring tool? + (list served from ShopDB assets, not a CSV) +Displays -> Lobby | Dashboard | 3D Print Kiosk +Shopfloor -> Collections | No Collections | Common | Heattreat | Part Marker | ... +``` + +The tool selection is the important half: it drives PC-DMIS version, FormTracePak +version, DODA flag, NTLARS/eDNC registry, UDC settings, machine number. + +**Design points:** + +- **Server is the source of truth.** The selection POSTs to ShopDB keyed on + serial; the local file is a cache. A mis-selection is then fixable from a + browser, not by walking to the bay. +- **Re-runnable.** Available from the Start menu, shows what was chosen and what + was installed. People pick wrong; that must not mean re-imaging. +- **Audited.** ShopDB records who chose what and when - that is what separates + "misconfigured" from "chose wrong" six months later. +- **Do not require the LAPS password.** Handing a shop-floor user the local admin + password to run a selection dialog is a real control change, and the app runs + elevated. Prefer a SYSTEM-run UI on the logon desktop, or a kiosk shell. + +### 4.3 Per-machine configuration records + +Today a machine's identity is scattered: + +| What | Where today | Size | +|---|---|---| +| eDNC / NTLARS registry | `_ntlars-backups/` - 147 per-bay `.reg` files | ~1.5 MB total | +| WaxTrace bay config | `bay-config.csv` - 15 bays, 7 FTPak versions, per-bay user ID | small | +| CMM bay config | `cmm-bay-config.csv` -> version.txt, doda.txt | small | +| UDC settings | `pre-install/udc-backups/`, per-bay | ~240 MB per bay | +| Machine number | `machine-number.txt` written by startnet | trivial | + +Every bay gets **all** of it staged, then picks its own. + +Proposed: one record per machine in ShopDB. + +``` +ShopDB asset (key: serial and/or machine number) + ├─ pctype + subtype -> selects the GE-Enforce scope + ├─ tool association -> CMM bay, Keyence model, FTPak version, DODA + ├─ machine number -> UDC / eDNC naming + └─ payload refs (sha256) -> NTLARS .reg, UDC backup, per-bay settings +``` + +**Reference payloads, do not embed them.** NTLARS `.reg` files carry binary +values and UDC backups are hundreds of MB. GE-Enforce already has a +content-addressed blob store with an HTTP fetch endpoint - the record names the +payload, the store holds it. + +This also removes a live bug class. `Restore-UDCData.ps1` and +`Update-MachineNumber.ps1` each mount SMB backup shares on their own drive +letters, which collided with GE-Enforce's `W:` and killed the manifest entry +running next (fixed in `66c24b5` by moving them to `R:` and `N:`). Payloads over +HTTPS remove drive-letter juggling entirely. + +Migration is a one-time import of two CSVs and 147 `.reg` files. A script, not a +project. + +--- + +## 5. Zero-touch: ShopDB-triggered imaging + +### 5.1 Flow + +``` +1. Zeroed PC, boot order = IPv4 only. PXE boots into WinPE. +2. WinPE reports serial + MAC + model, then WAITS. +3. ShopDB shows it under "awaiting assignment". +4. Operator opens the target machine -> "Create this PC" -> picks the waiting box. +5. WinPE polls, receives a job: + { pctype, subtype, tool/bay, tenant, purpose, target asset id } + and images with NO menus. +6. Progress streams to that machine's record in real time. +7. On completion the PC binds to the asset record. +``` + +Every menu in `startnet.cmd` becomes a field in the job. If no job exists, it +falls back to today's interactive menu and displays its own serial so one can be +created. + +### 5.2 Safety - this is a remote-triggered disk wipe + +Clicking a web button destroys a machine, and the target is chosen from a list. +The nightmare is picking the wrong row. + +**Mitigation:** the waiting PC displays a short code on screen; the job cannot be +claimed unless that code matches what the operator confirms in ShopDB. Cheap, and +it makes wrong-machine wipes structurally hard rather than merely unlikely. + +Additionally: a job may only target a machine currently in the waiting state, and +the wipe must be logged against both the operator and the asset. + +### 5.3 Real-time record updates + +Each phase already produces data; it simply is not joined up. + +| Phase | Existing source | Lands on the record as | +|---|---|---| +| PXE boot | dnsmasq lease + WinPE check-in | serial, MAC, model, "imaging started" | +| WIM apply | `winpe-status-push.ps1` | live stage / progress | +| Staging | `winpe-staging.log` + harvested bundle | what payload actually landed | +| First boot | collector API | CPU, RAM, disk, OS build | +| Enrollment | `criticalChecks.json`, `TokenMatch.json` | Entra join state, tenant, purpose | +| Steady state | GE-Enforce report | installed apps, drift, self-heal events | + +End state is one timeline per machine: *imaged 14:32 -> staged +gea-shopfloor-cmm, bay CMM4 -> enrolled GCCH_SH -> PC-DMIS 2019 R2 installed -> +enforcing clean since*. + +**Two rules so it does not rot:** + +1. **Push, best-effort.** If ShopDB is unreachable the bay must still image. + Queue events on the PXE server and drain them; the harvested logs are the + durable local copy. +2. **One writer per field.** Imaging status from the PXE server, inventory from + the collector, app state from GE-Enforce. Two writers on one field is the + exact bug that ate 2026-08-06. + +--- + +## 6. Network design + +### 6.1 Dual-homed PXE server + +- **PXE side** - isolated `172.16.9.0/24`, dnsmasq DHCP bound to that adapter +- **GE side** - corporate, reachable from anywhere; carries ShopDB API traffic, + the job queue, status pushes, and (if wanted) the BPRT token fetch + +The onboard `enp0s31f6` is free for this today. + +### 6.2 Security prerequisites - these gate the whole thing + +Checked 2026-08-06: + +**Already safe.** dnsmasq has `listen-address=172.16.9.1` + `bind-dynamic`, so it +binds only to the interface holding that address. DHCP will **not** leak onto a +GE-side NIC. This was the failure I most expected and it is already handled. + +**Not safe yet:** + +``` +smb.conf no `interfaces =` / `bind interfaces only` -> Samba listens on ALL interfaces +ufw every rule is ALLOW IN Anywhere, not interface-scoped: + 22/tcp SSH (account is pxe/pxe) + 445/tcp shares (pxe-upload, password in scripts) + 69/udp TFTP + 9009,9011 webapp +``` + +The moment a GE-side interface comes up, all of that is corporate-reachable. + +**Required before dual-homing:** +1. Scope every UFW rule to the PXE interface (`ufw allow in on ...`) +2. `interfaces = ` + `bind interfaces only = yes` in `smb.conf` +3. Expose deliberately on the GE side only what must be - realistically the + webapp, behind real authentication +4. Reconsider SSH credentials if the host becomes corporate-reachable + +*(Correction to an earlier note: the shares are **not** guest-accessible - +`guest ok = no` on every share, with `map to guest = bad user`. They require the +`pxe-upload` account. That account's password is in scripts, so it is not a +control once 445 is corporate-reachable.)* + +### 6.3 What dual-homing unlocks + +- Job queue sync and live status without the share as intermediary +- BPRT token fetch (`mcl.dwcdn.geaerospace.com`) reachable at imaging time +- Possibly the Entra join during imaging rather than after re-cabling - see §8.2 + +--- + +## 7. The PESetup question + +### 7.1 It may already be automatic + +`AutoStartCountDown = 31` (decompiled), and **`disableauto.json` is absent** from +the media - its *presence* is what disables autostart. Now the observed log: + +``` +11:30:07.74 'Selection View' started (LTSC warning displayed) +11:30:18.63 Btn 'Next' pressed. +``` + +The operator clicked at **11 seconds**. Nobody has waited past 31. + +**Test first:** one bay, hands off the keyboard, count to 40. If it proceeds, +zero-touch needs no new imaging code at all. + +### 7.2 If it does hang, re-implementation is tractable + +Nothing in the mechanism is proprietary - only the data, which we already own and +already parse. + +| PESetup step | Equivalent | +|---|---| +| `PrepareDisk` | `diskpart /s diskpartEFI.txt` | +| `PrepareUnattend` | token substitution (`%serialnumber%`, `*arch*`) + `Packages.xml` merge | +| `CopyTools` / `CopyPackages` / `CopyHWPackages` | robocopy | +| `CopyDrivers` | select by model from `HardwareDriver.json`, unzip | +| `ApplyImage` | `DISM /Apply-Image` per `OperatingSystem.json` | +| `ApplyPackages` / `ApplyLanguagePacks` | `DISM /Add-Package` | +| `ApplyUnattend` | `DISM /Apply-Unattend` | +| `CreateBCD` / `CopyWinRE` | `bcdboot`, `reagentc` | + +A few hundred lines of PowerShell. We would also get to fix the traps documented +in `docs/PESETUP-INTERNALS.md`: substring first-match-wins driver matching, a +family filter that knows only Latitude/OptiPlex/Precision, and a driver miss +being a **warning** rather than a failure. + +**The cost is ownership, not effort.** `Release.json` points at MCL auto-update +URLs, so GE expects this media to self-update. Diverging means tracking their +format changes indefinitely, plus a likely compliance conversation about the +mandated imaging tool. Hence: run the 40-second test first. + +### 7.3 What we would keep either way + +Image-time preflight stays valuable regardless: secure boot (PESetup hard-fails +without it), >=120 GB disk (`MinRequiredSpaceWithoutCompression`), driver match +for the model, media age (30-day expiry). Already built as +`playbook/scripts/preflight.ps1`. + +--- + +## 8. Enrollment + +### 8.1 How it works today + +- Six MCL packages on the media: `PERS` / `SH` / `LOAN` x `GCCH` / `RoW`, plus + the SFLD package for shopfloor +- `Start-BulkEnrollOrchestrator.ps1` asks tenant + purpose via + `Select-PCConfig.ps1` (a GUI), then applies the matching package +- `Insert-BPRTToPPKG.ps1` fetches an **encrypted token table** from + `mcl.dwcdn.geaerospace.com`, decrypts with a static passphrase + (PBKDF2 310k / SHA-256 / AES-CBC), matches on **Tenant AND Purpose**, injects + the token and stamps the real expiry into the filename + (`Exp_XXXXXXXX` -> `Exp_yyyyMMdd`) +- The package itself sets `F%SERIAL%` and + performs the Entra join +- A human then assigns the device category in Intune + +As of 2026-08-06 the tenant/purpose choice can be pre-seeded at the PXE menu +(`C:\Enrollment\pc-config.txt`, consumed by a shim at the path the orchestrator +already calls). Injection deliberately stays at first logon - it needs internet. + +### 8.2 The constraint that shapes pre-imaged stock + +The BPRT token is stamped into the package with an expiry. A PC imaged, powered +off, and left in a cupboard for three months has a **dead token** and will never +join - and the failure looks identical to "not on the production network yet", +because `Entra ID Joined: false` right after imaging is normal. + +Three options, pick deliberately: + +1. **Join at imaging time** - requires the production network at the bench + (which dual-homing could provide), and pre-imaged stock is then already joined +2. **Join at first power-on** - cleaner logistically, but stock has a shelf life + bounded by token expiry +3. **Re-inject on demand** - first boot detects a dead token and fetches a fresh + one; needs the machine to reach the token blob + +Whichever is chosen, the *silent* failure must be closed: a check that +distinguishes "token expired" from "no network yet". + +--- + +## 9. Ownership model + +Four times on 2026-08-06 two systems owned one thing and the last writer won +silently. A clean slate is the moment to fix that. See `docs/OWNERSHIP.md`. + +| Concern | Owner | Not | +|---|---|---| +| Computer name | the provisioning package (`F%SERIAL%`) | any script in this repo | +| Drive letters during imaging | PESetup (`W:`, hardcoded 9 places) | anything running `diskpart` concurrently | +| Entra enrollment | the package + orchestrator | `run-enrollment.ps1` | +| Kiosk / display targets | GE-Enforce scope | `site-config.json` (backstop only) | +| Imaging status | PXE server | the collector | +| Inventory | collector API | the PXE server | +| Installed app state | GE-Enforce reports | anything else | +| Job queue | **ShopDB** (owns assets) | the PXE webapp (executor only) | + +--- + +## 10. Phased delivery + +Each phase is useful standing alone. No phase requires the next. + +**Phase 0 - settle the unknowns (hours)** +- The 40-second PESetup autostart test +- Decide: one image or three +- Decide: join at imaging time, at first power-on, or re-inject + +**Phase 1 - visibility (small)** +- WinPE reports serial / MAC / model on boot; "awaiting assignment" list in the webapp +- Relay imaging status to ShopDB over the GE side +- *Useful immediately: you can see what is booting and what it did.* + +**Phase 2 - dual-home safely (small, gating)** +- Interface-scope UFW, bind Samba, bring up `enp0s31f6` +- Nothing else proceeds safely until this is done + +**Phase 3 - job queue** +- Job model + API in ShopDB (create / claim / complete), with the on-screen + confirmation code +- WinPE polls once where the menu is today; falls back to the menu if no job + +**Phase 4 - first-boot selection** +- Selection UI, server-backed lists, POST the choice, re-runnable +- Retire the WinPE sub-menus for subtype and tool association + +**Phase 5 - per-machine records** +- Import the CSVs and `.reg` files into ShopDB with payload refs +- Retire `bay-config.csv`, `cmm-bay-config.csv`, and staging all 147 `.reg` files + to every bay + +**Phase 6 - thin the image** +- Remove per-type media (`_media/`), the staging block, the Office ppkg + variants, the CMM bay picker, the WaxTrace ISO cherry-pick +- WinPE's job becomes: partition, apply, join, install enforce client, reboot + +--- + +## 11. Risks + +| Risk | Why it matters | Mitigation | +|---|---|---| +| Remote-triggered wipe hits the wrong machine | destructive, irreversible | on-screen confirmation code; job may only target a waiting machine; log against operator and asset | +| ShopDB becomes a single point of failure | today a bay configures from the share with ShopDB down | GE-Enforce ETag / last-known-good cache; explicit failure UI, never a silent stall | +| Dual-homing exposes the PXE server | SSH `pxe/pxe`, SMB, TFTP currently `ALLOW IN Anywhere` | Phase 2 gates everything | +| BPRT expiry on shelf stock | silent, indistinguishable from "no network yet" | pick a join strategy (§8.2) and add an explicit expiry check | +| Selection UI becomes the new single point of correctness | wrong subtype = wrong PC-DMIS, wrong bay config | re-runnable, server-corrected, audited | +| Re-implementing PESetup | ongoing format tracking + compliance | run the autostart test first | +| Big payloads over the corporate network | PC-DMIS, FormTracePak ISOs (~2 GB each), Keyence | GE-Enforce supports `smb` / `http` / `inline` per entry - decide per payload | +| Three images drift apart | drift caused the 2026-08-06 outage | prefer one image; `share-drift.py` guards what remains | + +--- + +## 12. Open questions + +1. One image or three? What genuinely differs between them? +2. Join at imaging time, at first power-on, or re-inject on demand? +3. Does PESetup auto-start after 31 seconds? +4. Is replacing the GE-mandated imaging tool acceptable, if it comes to that? +5. Does the selection UI run as SYSTEM (no LAPS password), or as an admin user? +6. Which payloads stay on SMB and which move to HTTPS? +7. Who operates the job queue day to day - is "Create this PC" a technician + action, or does it need approval? + +--- + +## 13. References + +- `docs/PESETUP-INTERNALS.md` - decompiled imaging tool behaviour +- `docs/OWNERSHIP.md` - who owns what, with the 2026-08-06 collisions +- `scripts/preflight.py`, `scripts/lint-unattend.py`, + `scripts/lint-driver-catalogue.py`, `scripts/share-drift.py` +- `playbook/scripts/preflight.ps1` - bay-side checks +- shopdb-flask: `plugins/geenforce/seed_display_scope.py` - the pattern this + proposal generalises