The kexec-initrd build copied a fixed set of module directories plus
drivers/net/mii.ko. Ubuntu ships that as mii.ko.zst, so the -e test
skipped it, and usbnet, r8152 and cdc_ether then failed to load with
"Unknown symbol mii_*": USB-C to RJ45 adapters never appeared (seen
2026-10-05, Dell USB-C adapter). igb lacked dca and i2c-algo-bit the
same way.
Walk the host modules.dep from the copied modules and copy every
dependency (73 on 6.8.0-71). Also copy rtl_nic, tigon, bnx2, bnx2x and
e100 firmware, which those drivers request at probe.
The initrd has no udev, so only modules init names get loaded, and
xhci_pci is a module in the Ubuntu kernel. USB 3 / USB-C ports were
therefore dead in the UEFI Blancco path and a Dell USB-C to RJ45 adapter
never appeared; init took the onboard e1000e port with no cable.
- modprobe xhci_pci and xhci_pci_renesas before the NIC drivers
- bring up interfaces inside the carrier wait too, since USB NICs
enumerate seconds after onboard ports
- on no-link, print each interface's driver and the NIC/firmware/link
kernel messages before dropping to a shell
Live kexec-initrd.img also now carries the NIC firmware set from the
Blancco 7.20 early cpio (rtl_nic, tigon, bnx2x, ...) under /lib/firmware.
The playbook build does not add it yet.
The UEFI init took the first interface in /sys/class/net and put
172.16.9.250 on it. On a PC with a second port, dock or USB NIC that can
be the unplugged one; wget then sat at "Connecting to 172.16.9.1:80"
forever (seen 2026-10-05 on a Dell, serial GGX6BN4). Bring every wired
interface up and use the first with carrier; with no link anywhere,
list the interfaces and drop to a shell. wget calls get -T 30 so a
stalled transfer fails loudly instead of hanging.
Live kexec-initrd.img was rebuilt with only init swapped (backup
kexec-initrd.img.bak-20261005-pre-carrier).
7.20 licenses through the Blancco cloud, and on the air-gapped PXE LAN
the only route out is WiFi INTERNETACCESS. The legacy BIOS cmdline still
blacklisted iwlwifi/iwlmvm, so BIOS boots could never license. btusb
stays blacklisted; rd.udev.timeout=10 bounds the udev hang iwlwifi
caused on some Intel machines. Live GetPxeScript.aspx changed to match
(backup .bak-20261005-pre-bios-wifi).
/reports/export?range=1m|3m|6m|12m&format=csv|zip. CSV has one row per
erased drive (system and drive serials, standard, result, times, licence
consumption id). ZIP holds the original XML/PDF reports plus summary.csv.
Reports are selected by the date inside each XML, falling back to mtime
only when it is missing. Exports are audit-logged.
playbook/blancco-preferences.xml is now the vendor 7.20 preferences
(schema 4.20, "Non_GEAE_Wired" profile) with only our site overrides:
SMB report share on 172.16.9.1 with auto_backup, WiFi INTERNETACCESS for
cloud licensing (wired stays up for SMB), manual as the fallback process,
bootable report, screensaver 86400. workflow default=true is kept: the
cloud workflow erases unattended, as live 7.15 already did. No custom
fields, matching both the vendor image and live 7.15.
prepare-boot-tools.sh:
- 7.20 initramfs is an uncompressed early cpio (NIC firmware, itself
.xz files) followed by the XZ archive. The old patch decompressed from
byte 0, failed, warned and carried on, so PXE boots would drop network
after copytoram. Walk the cpio to its trailer and patch the XZ part.
- Patch exactly one copytoram match or fail. The looser fallback pattern
matches five hooks in 7.20.
- config.img now carries playbook/blancco-preferences.xml, so the BIOS
(native kernel) and UEFI (HTTP) paths read the same settings. Refuses
to build if the playbook preferences are for another Blancco version.
- The old global replace() also set the 802.1x, WiFi and repeater
passwords to "blancco". Gone with the above.
- Keep preferences.xml size by refilling the NUL gap before the trailing
filler comment (7.20), or trimming the comment (7.15).
- OUT_DIR can be overridden to stage a build outside boot-tools/.
Run-ShopfloorSetup and Stage-Dispatcher top AutoLogonCount up to 10 so
SupportUser survives vendor-forced reboots. Sysinternals Autologon, which
lockdown uses to switch the bay to ShopFloor, leaves the counter in place,
so ShopFloor inherited the remainder. Windows counts it down on every
autologon and at 0 sets AutoAdminLogon=0: the bay boots to a ShopFloor
password prompt a few reboots after imaging.
Delete the counter once Winlogon shows ShopFloor:
- Monitor-IntuneProgress.ps1, on the poll that sees the handover
- SetShopfloorAutoLogon.bat, after sfld_autologon.ps1
- Force-Lockdown.bat, on success
The SupportUser top-ups stay. Tested in the win11 VM: the Monitor block
removes the counter for ShopFloor, keeps it for SupportUser, and is a
no-op when clean; both batch files leave ShopFloor autologon on with no
counter.
Three bugs, all found by actually running the thing rather than reading it.
1. `--e10` with no value did nothing. Making the flag optional-valued gave it
an empty-string constant, and the code deciding whether to drive the
controller used a truth test - so the documented `--e10` form printed the
full translation and quietly skipped the hardware. No error, because
nothing failed. Now gated on `is not None`, with a comment saying why.
2. A missing part program dumped a traceback. The top-level handler tidied
e10 errors and re-raised everything else. Now a clean message that lists
the part programs it can actually see, which also covers Windows path
confusion.
3. Errors could vanish when piped. stdout is block-buffered and stderr is
not, so the message printed before the output it referred to and a `| tail`
dropped it entirely - looking like a silent failure. stdout is flushed
before writing an error.
Serial access moves behind open_port(), with a termios backend on POSIX and a
Win32 backend reached through ctypes. Deliberately no pyserial: these tools
have to run on locked-down, air-gapped shopfloor machines where installing a
package is a project of its own.
Verified on a real controller from both a Linux laptop and a Windows 11 PC.
Three things that made this harder than it should have been, all now handled:
- CreateFileW returns a HANDLE, which is 64-bit. Without an explicit restype
ctypes truncates it to c_int. The open then appears to succeed and every
read and write silently operates on a bogus handle, which looks exactly
like a dead cable. Every kernel32 call now declares argtypes and restype.
- Defaulting to a fixed COM number was actively harmful. A wrong-but-real
port opens without error and answers nothing, so defaulting to COM3 while
the adapter was on COM4 produced the same silent symptom. Windows now
auto-picks only when exactly one port looks like a USB virtual COM port,
and otherwise refuses and lists what it found.
- COM10 and above need the \\.\ prefix, which is harmless on COM1-9, so it
is always applied.
Adds two diagnostics, because "no response" has too many possible causes to
guess at:
ports every serial port that actually exists. On Windows this reads
SERIALCOMM from the registry, so a port absent here does not exist
now, whatever Device Manager showed last week.
doctor walks the layers - discovery, open, write, read - and reports which
one fails, including a warning when the configured port is an
onboard one while a USB adapter is present.
Port selection is -d, then $E10_PORT, then the safe auto-pick. Friendlier
errors on both platforms for a missing port or a permissions problem.
Groundwork for replacing a Telesis pinstamp with a SIC Marking e10 while
keeping the existing eDNC/MarkTelesis operator workflow. MarkTelesis has a
hardcoded marker-type enum and eDNC is serial-only, so the plan is a
translator that impersonates a Telesis controller on a COM port. These are
the pieces that translator will be built from, all verified against a real
e10 (a P63, firmware "E10 P63 (20140924)").
scripts/e10-client.py
e10-V6 protocol client. Marking path and file transfer both use the BINARY
protocol: the text protocol uses space as its delimiter, so it cannot carry
a field cleared to a single space nor the space runs in real marked text,
and it has no checksum. Includes fault decoding against the manual's fig 1,
and a guard that refuses to overflow a variable's declared buffer.
scripts/make-e10-pattern.py
Builds marking files with DECLARED variables. The controller's own protocol
cannot create one: NEWFILE/INSERTTEXTLINE/SAVEFILE writes the reference but
leaves the variable count at zero, and SETVAR then fails. Template-based,
copying through byte-for-byte everything the format notes do not explain.
scripts/simulate-ednc.py
Parses a part program, substitutes tokens and emits the byte-exact frames
MarkTelesis would send, then translates them for the e10 and can drive one.
The wire format came from disassembling MarkTelesis.exe rather than from a
capture: frame builder at 0x401cf0, and the type letters at their call
sites. Note there is no station ID, unlike the Backup470 dialect.
scripts/fake-telesis.py
Answers as a Telesis controller so MarkTelesis will hold a conversation
long enough to be recorded, with switches to force NAK and timeout paths.
reference/e10/
52-file snapshot off the bench controller, used as the corpus for reversing
the MARK_V5.0 format and for testing without hardware. ConfigCtrl is
excluded and gitignored - it stores the supervisor password in cleartext.
GE part programs are excluded for the same reason.
Three hardware findings worth knowing:
- The controller TRUNCATES SILENTLY to a variable's declared buffer size
while SETVAR still answers ok, and GETVAR echoes the full value back so
it cannot detect this. Guarded client-side against the size in the file.
- It refuses to mark when a referenced variable is unset, which makes an
unset field fail loudly rather than marking the previous part's data.
- Setting variables does not modify the stored file, so backups will not
churn on every marked part.
Every failure of Get-TelesisPortConfig was filed as "no Telesis marker
configured on this PC" and exited 0. That is correct for the many PCs which are
not marker bays, and wrong for a bay whose registry key EXISTS and could not be
read: a half-configured marker was silently never backed up again, which is
exactly the failure shape this feature was built to prevent. The absent-key case
now throws a distinguishable exception and stays quiet; anything else is logged
every cycle and counted as a skip.
Get-TelesisFileList was unbounded. The receiver drives that loop, so a
controller repeating a name or answering noise on a marginal cable would spin
for ever - and not harmlessly, because it holds the MARKING port while
GE-Enforce waits on the script with no timeout, so one chatty controller wedges
the enforcement cycle and the operator cannot mark a part. Now capped by count,
by elapsed time, and by repeat detection, all of which THROW: a partial listing
presented as a complete snapshot would be worse than failing.
Baud and Data Bits parse with TryParse. Note the narrower reason than it looks:
[int]'' is 0 in PowerShell, so empty and non-numeric values were already handled
by the zero-check. What was not handled is a numeric string too large for Int32,
which throws on the cast and would have been filed as "not a marker bay".
Verified in the win11 VM as SYSTEM, against both the old and new module: absent
key stays quiet, a key without Port Id is loud, garbage and out-of-range baud
both fall back to 9600, a real 19200 is still honoured, a repeating controller
stops at once and endless unique names stop at the cap.
The snapshot path stored against a marker read "W:\backup\partmarker\...". That
mapping exists only inside the SYSTEM logon session, only while the GE-Enforce
cycle is running, and only on that one bay - so the path recorded in ShopDB
resolved to nothing for the person most likely to need it.
Writing still goes through the drive letter, deliberately: that is the
connection GE-Enforce authenticated, and reaching the same server by another
name costs a second authentication or fails outright with error 1219. Only the
recorded path changes.
Resolving a mapped drive to its share needs all three of DisplayRoot, Root and
the WMI ProviderName, because which one is populated depends on how the drive
was made: net use mappings (what GE-Enforce creates) fill DisplayRoot, while a
New-PSDrive to a UNC leaves DisplayRoot EMPTY and puts the share in Root. A
local drive and an already-UNC path both fall through untouched.
The script had no CmdletBinding, so PowerShell dropped unrecognised named
arguments into $args instead of failing. A typo in a manifest entry Args -
"-WindowStartMinutes 45" - would have been ignored silently, and the capture
would have run at the wrong time with nothing in the log to explain it. That
is exactly how an hour went into debugging a time gate that was working.
Also adds WindowStartMinute so the window can open at :45 rather than on the
hour. A shift change is a moment, not an hour, and the first run of the day is
easiest to watch when it lands when somebody expects it.
Both entry files described the wrong scope. Checking the live share mirror:
Backup-PartMarker.ps1 belongs in common\scripts, not the partmarker scope,
because it imports ..\lib\TelesisMarker.psm1 and ..\lib\ShopdbBackupClient.psm1
and those live in common\lib. Backup-NtlarsSettings.ps1 sits there for the same
reason, and its entry already lists gea-shopfloor-partmarker in PCTypes -
scoping is by PCTypes, not by which directory the entry lives in. As written it
would have failed its imports on the first cycle.
eMxInfo healing is a NEW entry in the partmarker scope rather than a PCTypes
edit: those entries live in the collections and nocollections scopes, which a
marker bay never evaluates. The site already duplicates the config and the cmd
per scope, so markers follow that pattern - at the cost of the yearly rotation
now touching three copies, which the entry says out loud.
The cmd template also carried a stale source path. The deployed cmd reads
..\configs\eMxInfo.txt; the repo template still said the file sat beside it,
which would have exited 1 on every run.
Shopfloor PCs intermittently lose S: and ask for credentials while navigating
the identical UNC path in Explorer works fine. Four different causes produce
that same appearance - a persistent reconnect that raced the network, a stored
credential whose target string does not cover the name form in use, two
identities contending for one server, or a password that rotated - and each
needs a different fix. From the desk they are indistinguishable.
Test-ShareAccess.ps1 collects what separates them: reachability of both name
forms, the mappings and the account each remembers, the exact credential target
strings, Kerberos tickets, live SMB sessions and the identities holding them,
and the SMBClient/Connectivity log. Then it names the causes the evidence
supports rather than leaving a wall of output to interpret.
Read-only by construction: it maps nothing, deletes nothing, stores no
credential, and prints usernames and a password-present flag but never a
password, because its log lands where techs read it. Safe on a bay while
somebody is working.
It reports which identity it is running as, because mappings and SMB sessions
are per logon session: run as the user to explain S:, as SYSTEM to explain W:
and the backup scripts. A drive missing in one is genuinely absent in the
other, and mistaking that for a fault is easy.
Also derives the marker backup's share root from the script's own location
rather than a hardcoded UNC. GE-Enforce mounts the share as W: with SFLD
credentials, and reaching the same server by a different name than the mount
used costs a second authentication - or fails outright with 1219 - for reasons
the error text never mentions. The script's own path is the one handle
guaranteed to name the connection that already exists.
This runs as SYSTEM, which authenticates to the network as the machine account.
A SYSTEM process reaching a UNC path without credentials gets an access-denied
that Test-Path reports as "not found", so a permissions problem reads as a
missing directory and sends the next person hunting the wrong thing. The
backups registry already carries that warning; the client had not acted on it.
On a write failure the log now names the likely cause, and the two conditions
that make it work: the entry must run under the GE-Enforce cycle, since the
share is mounted only for its duration and all SYSTEM processes share the logon
session that holds the connection, and the share ACL has to admit computer
accounts.
Verified against an unreachable UNC on a bay with a real controller: capture
succeeded, the write failed, the diagnostic was logged and the cycle was
counted as a missed backup.
A part marker has no machine number of its own - the device is what a backup
belongs to, and C:\Enrollment\asset-id.txt is what names it. Reading that file
inside the shared machine-number lookup looked like the tidy place for it and
was wrong: marker bays run eDNC too, so Backup-NtlarsSettings.ps1 reaches the
same lookup, and it would have started posting a machine's DNC settings under
the marker's asset number where today it exits quietly. The marker script asks
for the fallback explicitly instead, which is what -MachineNumberFallback is
for.
Also from review:
Returning a byte array bare lets the pipeline enumerate it, so an EMPTY file
came back as $null, ComputeHash threw, and with ErrorActionPreference=Continue
the manifest entry silently kept the previous file's hash - a corrupted
manifest stored as a good revision. Empty parameter files are plausible.
Reading registry values directly under StrictMode throws when one is absent, so
the "no Port Id" guard was unreachable and a partially configured marker bay
reported itself as having no marker at all, permanently and quietly. Values are
now read through a presence check with the fleet's own defaults.
The outside-window quiet state embedded the hour, so the say-once module said
it hourly. Command replies now have their checksum verified like data blocks
do, since a corrupted listing entry would otherwise be trusted and a file
requested under a name the controller does not have. A controller that
genuinely holds nothing is no longer reported as a failed capture. A snapshot
that could not be written to the share now counts as a missed backup like the
rest, and the header no longer claims orphaned snapshots get tidied - they are
kept on purpose, because a timed-out post may still have been recorded.
The eMxInfo entry on the share is scoped to collections and nocollections, so
a marker bay never heals. Marker bays run eDNC as well - 01-eDNC.ps1 installs
it before the Mark MSI - so the same vendor installer clobbers the same file
there, with nothing to restore it.
The fix is a scoping change rather than a new mechanism: add the marker PCType
to the existing entry. The paths it already writes are the right ones, because
mxTransactionDll.dll in the Mark overlay reads \Dnc\Server Files\eMxInfo.txt -
the Mark suite uses the DNC install, not its own directory.
The cmd variant additionally refreshes the copy that imaging drops into the
Mark directory. Nothing reads that path, so it is cosmetic; it is there so a
healed bay and a freshly imaged one are identical, rather than differing in a
way that costs somebody an afternoon later.
Scoped to gea-shopfloor-partmarker so it reaches marker bays only. The script
checks the DNC\Mark registry itself as well, so a mis-scoped entry exits
quietly on an ordinary PC rather than erroring every five minutes.
The entry records what the share layout has to provide, because the script
resolves its two modules relative to its own location and one of them,
ShopdbBackupClient.psm1, is shared with the NTLARS backup and has changed. It
also records the schedule and the settings a bay needs, since all three are
places where a deployment can look successful and quietly do nothing.
Also counts a failed capture as a missed backup, not just a busy port. Port
contention resolves itself; a controller that will not answer does not, and
the two are worth telling apart in the count that reaches ShopDB.
Telesis pinstamp controllers have never been backed up. The vendor's own
utilities, Backup470 and Backup420, are GUIs with no command line, so nothing
could capture a marker on a schedule - and a controller swap meant retyping
patterns from memory.
TelesisMarker.psm1 speaks the protocol directly, reversed from a capture of
Backup470 against a TMC470 and verified against hardware: a pattern pulled by
this code is byte-identical to the one the vendor tool saves, and a pattern
written back survives a power cycle. A TMC420 was captured the same way and
differs in exactly one respect - it exposes no parameter files - so one
implementation covers both families, with content classes chosen from a Type I
status probe rather than a per-model dialect.
Backup-PartMarker.ps1 captures everything the controller holds into a
timestamped snapshot on the share and records it in ShopDB against the marker
asset. Snapshots rather than a mirrored directory, because a mirror forces a
choice between deleting the only copy of a retired pattern and claiming a
pattern that no longer exists; a snapshot has neither problem, and the manifest
hash means an unchanged controller writes nothing.
The marker's serial port is shared with eDNC and is exclusive-open, so captures
happen only at shift changes - 14:00 normally, when someone is on site to fix a
problem, with 06:00 and 22:00 engaging only once the backup is overdue. The
capture yields the port within a second if the marker application starts
mid-run, and cycles missed to a busy port are counted and carried into the next
snapshot, since posting a no-op to keep the health badge green would hide
exactly the failure the badge exists to report.
TYPE 'R' deletes a file on the controller, immediately and without
confirmation. It is documented in the module header and deliberately not
implemented.
A bay comes off the line unable to build a printer queue. Set-ShopdbPrinters.ps1
creates queues against drivers that are already present and skips any that are
not, so a freshly imaged PC silently gets no printers until some later
enforcement cycle tops it up.
The two universals are 100 MB and resolve the overwhelming majority of the
fleet's printers. The four device-specific drivers are another 126 MB and are
deliberately NOT here - they belong only on bays that own that hardware, and
those bays pick them up from the runtime manifest.
NO WAVE GATE on these entries, unlike the runtime ones. Waves exist because ~300
bays share one runtime share and GE-Enforce spreads them across only five
minutes; imaging runs a handful of bays at a time on the isolated PXE LAN that
already moves multi-GB WIMs, so there is no stampede to prevent.
DETECTION IS THE SPOOLER'S OWN KEY, not pnputil, and the difference is not
cosmetic. Installing a printer driver is two steps: pnputil /add-driver stages
the package into the driver store, Add-PrinterDriver registers it with the
spooler. Get-PrinterDriver - which is what Set-ShopdbPrinters consults - reads
the spooler. Measured on a Win11 bay: after Remove-PrinterDriver the spooler
reports the driver gone while pnputil still lists the package, so a pnputil-
detected entry would report compliant forever while every queue kept skipping. A
broken bay that looks healthy is the worst outcome available here.
The payload is not in git, matching every other installer under preinstall/:
stage printer-drivers/ onto the PXE share alongside the rest.
The EAP-PEAP and Office comments outlived the entries they labelled, which
would send the next reader looking for commands that are not there. The
surviving wireless comment said 'Wi-Fi profiles' plural when only
INTERNETACCESS remains; it now records why the other two went and that
wlansvc has to be started first.
INTERNETACCESS has never been deployed. wlansvc ships Manual/Stopped on
Windows 11, nothing in the unattend started it, and `netsh wlan add profile`
refuses to run without it:
The Wireless AutoConfig Service (wlansvc) is not running.
exit=1
Reproduced in the win11 VM. So specialize was adding no wireless profile at
all, and every bay reached OOBE with no known network - which is exactly the
wifi/network selection the tech has been clicking through.
Fixed by starting the service first. `net start`, NOT `sc start`: sc is
asynchronous and returns START_PENDING, and the VM showed netsh losing that
race and still reporting the service as not running. The command is idempotent
and always exits 0, so a machine with no wireless hardware is unaffected.
With the profile genuinely landing, HideWirelessSetupInOOBE is now true.
Dropped EAP-PEAP and BLUESSO from specialize while here. Both are installed by
the MCL ppkg - Install-GEWirelessSettings.ps1, with byte-identical EAP-PEAP.msi
(sha d90f2489045b6fdb) and BLUESSO.xml (sha d8d886db37a40054) - and BLUESSO
could never have bootstrapped anything: hidden SSID, connectionMode=manual,
eapGtc with passwordFromUser. INTERNETACCESS is the bootstrap path and it is
WPA2PSK with connectionMode=auto, so it needs no EAP module and no certificate.
The eight GE certificates at Orders 4-11 are also duplicated by the ppkg (as
Certificates_*.provxml runtime entries) and are left in place for now: that is
a separate cut, and the case for removing them rests on the pre-ppkg token
fetch being publicly signed, which is true today (mcl.dwcdn.geaerospace.com is
issued by Google Trust Services) but is not ours to depend on.
lint-unattend.py passes clean on all three live files and all three repo copies.
The shortcut fix from e844ff3 has been live in git and absent from the floor
since 2026-08-06. site-config.json exists TWICE on the share, from one repo
source: enrollment/shopfloor-setup/ and enrollment/config/. Only the second one
is staged to a bay - startnet copies Y:\config\site-config.json to
W:\Enrollment\site-config.json - and it was the stale one.
So every bay imaged in the last two weeks came up without the Plant Apps startup
item and without the Defect_Tracker taskbar pin, while the drift report showed
site-config.json in sync, because it was reading the copy nothing consumes. A
green check on the wrong file is worse than no check.
Deployed the repo copy over it (backup on the server at
~/backups/site-config.json.bak-20260819) and marked BOTH destinations git-owned,
so neither can go stale behind the other.
The shopfloor unattend is reconciled the other way round. Live was 87 lines ahead
of the repo - the default-user startup-delay removal, the Windows Update
disables, the removable-media block that stops PPKG auto-detection at OOBE, and
the run-enrollment.ps1 path fix from C:\ to C:\Enrollment. The repo copy was a
201-line fossil. LIVE WINS: these files boot machines, and pushing the repo copy
over them is exactly the 2026-08-06 outage that prompted this tool. Adopted live
into the repo verbatim (lint clean) rather than merging by hand.
Also fixed a pair that could never pass: the engineer unattend was compared
against playbook/FlatUnattendW10.xml, which is the STANDARD answer file, so it
reported DIFFERS permanently. 6f86c81 added FlatUnattendW10-engineer.xml but did
not repoint the pair at it. A permanently red row is one nobody reads, which is
how the site-config gap stayed invisible next to it.
All twelve pairs are now git-owned and in sync, and the gate has nothing left
classified as "known bad" to hide behind.
Zscaler: Install-Zscaler.ps1 ships in all three MCL packages with
Zscaler-windows-4.7.0.317-installer-x64.msi bundled. It is tenant-aware
(geaerospace.com for GCCH, geaerospaceglobal.com for RoW, gegovdev.us for QA)
and passes enableFips=1.
Our FirstLogonCommands entry installed 4.5.0.337, two minor versions older,
without enableFips, and with userDomain hardcoded to geaerospace.com - wrong
for a RoW bay, which startnet's tenant sub-menu can produce. Two msiexec runs
against the same product at one logon also risk 1618. Removed, same reasoning
as Office.
THE BIGGER FIND, while removing it: this repo template was badly out of date
with the share, and the playbook deploys it with force: yes.
Missing here but present live: specialize Orders 39-48 - the bulk-enrollment
orchestrator registration (Order 39, Register-BulkEnrollTask-Standard.ps1), the
Default User startup-delay and Start-menu tweaks, the two NoAutoUpdate keys,
the 'vanilla' Shift+F10 fallback alias, and the removable-media block that stops
the PPKG being auto-detected at OOBE. Also missing: FirstLogonCommands Orders
4-10 (wuauserv disable, Default User hive edits, removable-media unblock).
A playbook run would therefore have stripped enrollment from every gea-standard
and gea-engineer bay. Both templates are now reconciled against the live share.
Engineer gets its own template. The two files differ only by FirstLogonCommands
Order 11, which installs the Engineering Layer - so deploying one shared file to
both, as the playbook did, silently removed that entry from gea-engineer on
every run. Split into FlatUnattendW10.xml and FlatUnattendW10-engineer.xml with
an explicit per-type loop.
lint-unattend.py passes clean on all three live files and all three repo copies.
FirstLogonCommands Order 1 was `shutdown -a`, and it protected nothing.
The reboot it was aimed at is the one provtool schedules when the provisioning
package is applied - and Start-BulkEnrollOrchestrator.ps1 already aborts that
itself, on the line straight after provtool returns, with its own `shutdown /a`
and the comment "Abort the reboot that the provisioning package scheduled".
Our copy runs at logon, before the orchestrator has applied the package, so at
the moment it executes there is nothing pending to abort.
The reboot that actually interrupts first-logon work is the orchestrator's own
`shutdown /r /t 0 /f` at the end of its run. That one is immediate, so it has
no pending-shutdown window for `shutdown /a` to cancel, and it is deliberate -
the Entra ID join needs it.
So the entry was a no-op that read like protection. It cost real time this week:
its presence was taken as evidence that first-logon work was being guarded when
it was not.
Removing it changes no behaviour. What it was reaching for - first-logon work
surviving a mid-logon reboot - needs resumability instead, which is what the
retired Office payload now does and what zscaler.bat still lacks.
Order 2 (zscaler) is left numbered 2; FirstLogonCommands execute in ascending
Order and gaps are legal, so renumbering would be churn for no gain.
Live gea-standard and gea-engineer updated to match. lint-unattend.py passes
clean on all three live files and both repo copies. gea-shopfloor never had
this entry.
Install-Office.ps1 ships inside all three MCL provisioning packages
(GCCH_Prod_MCL-PERS v2.15, -SH v2.10, -LOAN v1.3 and the RoW equivalents). It
moves M365 Office x86.zip out of C:\Deploy\Applications\BPRT, extracts it to
C:\WCDApps\Office and runs setup.exe /configure against the vendor's own
Configuration-x86.xml.
The unattend's FirstLogonCommands Order 3 was a second ODT run at that same
first logon. Click-to-Run is single-instance, so the two conflicted - which is
what "Office is not installing properly" looked like on both image types.
They also disagreed on what to install. The vendor package uses
O365ProPlusRetail, excludes Access/Groove/Lync/OneDrive/Teams, matches the OS
language, carries RemoveMSI and AppSettings, sets SharedComputerLicensing=1 for
the SH purpose and drops Outlook for LOAN. Ours used O365ProPlusNoTeamsRetail,
en-us, no RemoveMSI, no purpose awareness, and hardcoded
SharedComputerLicensing=0 - which would have broken shared computer activation
on every SH machine.
Removing the entry here matters as much as removing it live: the playbook
deploys this file to both image types with force: yes, so a playbook run would
have put the duplicate straight back.
The share payload is kept but retired, with a README recording why and what to
check before ever re-enabling it. Its install.bat and gea.xml were repaired
first (stage out of C:\Deploy ahead of the orchestrator's DeleteDeployDir task,
pin the ODT build to match v32.cab so no CDN lookup is attempted on the
isolated LAN, and register a self-deleting resume task since FirstLogonCommands
run only once) - VM-tested, then left in place unreferenced.
Live gea-standard and gea-engineer unattends updated to match; lint-unattend.py
passes clean on all three live files and both repo copies.
The first draft identified every bay by PC name, taken from a 12 June map and
the machine list's Shop Floor PC Name column. Both are hand-built snapshots: a
PC that has since been replaced or re-imaged answers to a different name while
the machine it sits at has not moved. Following those names would eventually
mean renaming the wrong bay.
Machine number leads every table now, hostnames are marked as "where the fault
was last seen", and the reader is pointed at what the bay itself knows - the
eDNC registry first, then the enrollment file, then UDC - with ShopDB as the
live PC-to-machine record since the collector maintains it.
Also notes that either CSV should be generated from ShopDB rather than
hand-maintained, for the same reason.
The rename fix only changes what future renumbers do. Nine bays are already in
the state it prevents, the 3.25.3 adapter is staged but not on the SFLD share,
and two questions - whether DMG Mori needs a bundle at all, and whether the v2
manifest ever received the MTConnect entries - are unanswered.
All of it needs a machine the dev box cannot reach, so it is written as steps
to follow rather than something to run here: exact hashes to check before
copying, per-bay numbers from the hostname map, verification after each change,
and rollback for the share copy.
Notes the two things not proven on real hardware: that stopping the Makino
Adapter Manager keeps it from rewriting config on shutdown, and that the four
Fanuc bays with no agent device name are a different fault that a rename would
not fix.
Set-MachineNumber rewrote one Devices.xml per variant and restarted the agent.
That is right for Okuma and eDNC, which keep the device name in that one file -
checked across 7 Okuma and 15 eDNC bay captures. Fanuc and Makino keep it in
several, and editing only the agent's copy leaves the adapter streaming under
the old identity.
It has already happened. Seven of thirty-four captured Fanuc bays have the
agent on the right machine number and the adapter still on the bay it was
imaged from: 4007 on a 7801 Toshulin, 3031 on a 7804, 2005 against 2006 on a
dual-spindle pair. Both captured Makino bays have Devices.xml on the machine
number and the other three files still on MAKINO-1.
Update-MTConnectVariantName renames across a declared set of files with the
owning services stopped first. Makino needs that ordering more than most: the
vendor's guide says the Adapter Manager rewrites the COMPLETE configuration
from memory as it stops, so an edit made while it runs is discarded, and that
the name must be identical in every file or the agent may not start.
It collects every name in play before rewriting rather than discovering one.
A bay half-renamed by the old code carries two at once, and converging on the
one that happened to be found leaves the other behind - which is the state this
is meant to end, not reproduce. Nothing is written if a service will not stop,
and whatever was stopped is started again.
Tested on Windows 11 against copies of the real captures: the 7801 bay (adapter
4007) converges to 7801 in two files and leaves the correct agent alone; the
7502 Makino bay converges all four; second runs report no changes. Service
ordering verified by process id, using stand-in services - sc.exe fakes named
after the real ones are not real services, so the service list is a parameter
defaulting to the production names.
Does not fix the bays already in this state; they need a run each.
Sixteen files that run on every shopfloor PC existed only on the SFLD share.
The cost showed up while debugging the NTLARS backup: the script that posts to
ShopDB could not be read, reviewed or diffed, so its behaviour was inferred
from log output for most of a day. It turned out to hold a silent fallback that
had been governing the whole fleet for months.
Imported as-is from tsgwp00525-v2, no edits:
lib/ShopdbBackupClient.psm1 the shared backup client
scripts/Backup-NtlarsSettings.ps1 converted to use it
scripts/Set-ShopdbCollectorKey.ps1 collector credential delivery
scripts/Test-RegExport.ps1 exercises the .reg codec with mocks
scripts/Set-EventSaver*.ps1 kiosk power / screensaver / disable
scripts/Setup-OpenText.* OpenText install + toolbar
scripts/Migrate-PCType.ps1, Select-KioskType.ps1, Set-FmsHostsEntry.ps1,
scripts/ensure-vnc-firewall.ps1, Install-AcroReader.cmd, Install-Oracle11r2.cmd
lib/Install-FromManifest.ps1 is also updated from the share, which was 37 lines
AHEAD of this repo and purely additive: the Add-EnforceResult reporting added
during the kiosk API cutover, done live and never committed back. Nothing was
removed.
Checked for embedded secrets before committing; there are none.
Set-ShopdbCollectorKey deliberately reads its token from a sibling file on the
share rather than holding it, so the script is safe to track.
The share remains what actually runs. This makes it reviewable, and makes the
next drift visible as a diff rather than a surprise.
The prune drops a *.log by LastWriteTime. A log written to a fixed filename is
appended every cycle, so its LastWriteTime is always now, it can never be older
than any cutoff, and it grows without limit. ntlars-backup.log, eventsaver.log
and shopdb-collector-key.log all do this. On one part-marker PC eventsaver.log
had reached 11,000 lines and ntlars-backup.log 3,234, of which 3,217 were the
same "Throttled" line - the four events that mattered were unfindable without
grep -v.
Undated logs are now rolled to <name>-YYYYMMDD.log before the prune runs, which
stops them being written to and lets them age out on the existing 30 day
retention. The owning scripts need no change: they append with Add-Content or
Tee-Object, which recreate a missing file on the next write. This matters
because those three scripts are live-share artifacts, not in this repo.
Rolled under the date of its FIRST LINE, so the stamp matches the contents.
That is also the only trustworthy signal: CreationTime is not, because NTFS
file tunneling gives a recreated file the old creation time when it reappears
within 15 seconds, so a busy log keyed on that would look stale the instant it
rolled and would roll again every cycle.
A log with no parseable timestamp is left alone until it passes 5MB, so an
unrecognised format still cannot grow forever. Empty files and already-stamped
files are skipped. Rolling onto an existing target appends rather than
overwriting, so a second roll on one day loses neither side.
Verified against real files under pwsh, including the append branch and the
owner recreating the file afterwards. The first attempt keyed on CreationTime
and a second used [datetime]::TryParse with an untyped $null, which throws "no
overload" and would have made the whole roll a silent no-op; the ref is now
pre-typed.
New section 9a - five design rules, each from a bug fixed on 2026-08-06 and each
cheap to honour in a new design but expensive to retrofit:
Absent and empty are different. Four scripts tested config as
"not null AND Count > 0", so an explicitly empty list fell through to a
hardcoded default - a kiosk configured with "desktopApps": [] was given the
full machine-tool set. Configuring "none" produced "everything".
Deny by default. preinstall.json had 16 of 21 entries at PCTypes ['*'] and the
SMB manifest 18 of 25 unfiltered. Both filters worked perfectly; they just were
not applied.
Gate at the point of action. Call-site gates on PC type were bypassed by the
finalization phase, silently.
The image carries configuration. WJ Shopfloor.lnk is inside the WIM, so no
pipeline change removes it.
The ppkg is a hard boundary. Chrome, RealVNC, Tanium, CyberArk and the rest come
from the package - thinning the image does not thin them.
Also corrects a premise in section 4.2: a first-boot selection UI cannot assume
the API is reachable, because ShopDB needs the AESFMA wifi SSID and the imaging
LAN has no route to it. Points at Install-ShopdbKiosk-WhenOnline.ps1 as the
reusable shape - arm a task at imaging, do nothing until the API answers, act
once, verify, self-remove.
Plus two risks, the diagnostics collector in the reference map, and a gotchas
section: HardwareDriver.json defeats ConvertFrom-Json on every PowerShell
version, Compress-Archive caps at 2 GB against an 8 GB package, and PowerShell
Core normalises backslashes on Linux so bay scripts can be tested here.
The driver check - the most valuable check in this script - has never actually
run. It died on:
Driver check failed: Cannot convert the JSON string because a dictionary that
was converted from the string contains the duplicated keys 'FileName' and
'FileName'
HardwareDriver.json carries both casings of the same fields - fileName and
FileName, destinationDir and DestinationDir. ConvertFrom-Json treats object keys
case-insensitively and refuses the document. -AsHashtable would handle it but
that is PowerShell 6+, and WinPE runs 5.1. Confirmed it throws on PowerShell 7
too, so no version of ConvertFrom-Json can read this file as-is.
Pulls the four needed fields out of each entry by regex instead, preferring the
lowercase key and falling back to the capitalised one, and unescaping the
backslashes in destinationDir.
Tested against the real 44-entry catalogue, all three outcomes:
OptiPlex Micro 7020, pack present -> OK, win11_optiplexd13mlk7020_a09.zip
same model, pack removed -> FAIL, names the missing path
Surface Laptop 7 -> FAIL, no pack matches
Worth noting the check was failing SAFE - a WARN that reads like a tooling
glitch rather than a missing driver pack. It would have stayed invisible until a
bay imaged with no NIC.
The post-fix capture from 579C144 reported three entries as unknown(0x04) -
SecurityHealth, RtkAudUService, WavesSvc. The decoder only knew 02/06 enabled and
03/07 disabled. 04 is also enabled and 05 also disabled; without them the report
says 'unknown' for entries that are perfectly ordinary.
PREINSTALL SCOPE
Sixteen entries carried PCTypes ['*'], so a Display kiosk installed Adobe
Acrobat, OpenText HostExplorer, WJF Defect Tracker, the shopfloor serial drivers
and twelve legacy VC++ redistributables. Those are now scoped to the nine
machine-tool types.
A Display now installs 3 of 21 entries:
PowerShell 7.5.4 SFLD-ApplyDSCConfig runs pwsh.exe 7
VC++ Redistributable 2015-2022 x64 modern x64 runtime Edge/WebView2/PS7 lean on
Display Kiosk App already Display-scoped
Kept as wildcards deliberately - removing the x64 runtime risks breaking the
kiosk itself, and PS7 is a hard dependency of the DSC task. Everything else a
kiosk has no use for. Edited as targeted text replacement, 16 line pairs, so the
file's formatting and its long _comment blocks are untouched.
THE LAST OF THE EMPTY-MEANS-DEFAULT BUGS
Configure-PC.ps1 and 07-TaskbarLayout.ps1 had the same test as the two fixed
earlier: $null -ne $cfg -AND .Count -gt 0, so an explicitly empty list fell
through to a hardcoded shopfloor default. Configure-PC is the one that actually
bit - it runs in the finalization phase and put these in the all-users Startup
folder on 579C144, timestamped mid-image:
Defect Tracker.lnk 16:46:03
Plant Apps.lnk 16:46:03
Plant Apps launching msedge --new-window on a kiosk. eDNC was in the same
hardcoded list and only escaped because its Test-Path guard found no DncMain.exe.
08-EdgeDefaultBrowser.ps1 had it too, with a Plant Apps + WJ Shopfloor + Dashboard
tab fallback. Harmless today because the Display profiles configure a real tab,
but one edit away from biting.
Get-ProfileValue returns $null only when a key is absent from BOTH the profile
and site-config, so $null is the only honest "not configured" signal and an
empty array means what it says.
VERIFIED against the post-fix capture: the S: mapper Run key and the machine
number prompt task are both gone, and 'ShopDB Kiosk Bootstrap' registered
correctly with boot and time triggers.
Also: the collector's StartupApproved decoder now handles 0x04/0x05, which that
same capture surfaced as unknown.
Still baked into the WIM and unaffected by any of this: WJ Shopfloor.lnk, dated
April, sitting in the all-users Startup folder.
Three fixes from the 579C144 diagnostics, all the same shape: a Display kiosk
being treated as a machine-tool bay.
desktopApps: configuring "none" produced "everything"
"Display-Dashboard": { "desktopApps": [] }
06-OrganizeDesktop.ps1 tested $null -ne $cfgApps -AND .Count -gt 0, so an
explicitly empty list fell through to the hardcoded fallback and the kiosk was
given UDC, eDNC, NTLARS, WJ Shopfloor and Defect_Tracker shortcuts.
Get-ProfileValue returns $null only when the key is ABSENT from both the profile
and site-config, so $null is the real "not configured" signal and an empty array
means what it says. startupItems carried the same test - harmless today because
its else branch has no fallback list, but commented so the two cannot drift.
Machine-number prompt on a machine with no machine number
Register-CheckMachineNumberTask.ps1 armed 'Prompt Machine Number' (AtLogOn,
BUILTIN\Users) whenever the number was the 9999 placeholder - which is always
true on a Display, because startnet only collects a real number for machine-tool
types. A logon dialog on a kiosk with no keyboard. Now skips PC types that have
no machine number by design, and clears any stale task.
S: mapper on a share-less PC
Displays are Entra-joined with local accounts and no SFLD credentials, so mapping
S: can only fail, once per logon, forever. Run-ShopfloorSetup already gated the
CALL on $noEnforceTypes, but the bay registered it anyway at 15:07:55 with no
"Skipping" line in the log - so something in the finalization phase reaches the
registrar past that gate. Rather than chase the caller, the registrar now gates
itself and removes a stale Run entry. The call-site gate stays; this makes the
outcome correct regardless of who invokes it.
That bypass is worth understanding separately - the same pattern would defeat any
call-site gate in the finalization phase.
All three parse clean and are deployed byte-identical to the share.
The bootstrap looked for C:\Enrollment\kiosk\Install-ShopdbKiosk.ps1, but
startnet copies the type-specific tree to
C:\Enrollment\shopfloor-setup\gea-shopfloor-display\ - so the payload lands
beside the bootstrap, not under C:\Enrollment.
As written it would have logged "Installer not staged" on every 15-minute cycle
and never installed anything, with no error anywhere. Exactly the silent-failure
shape this whole day has been about, and it would have looked like the API being
unreachable.
Resolves via a -KioskRoot parameter defaulting to kiosk\ next to the script, so
the bootstrap and its payload travel together regardless of where the tree is
staged. display-type.txt still comes from EnrollmentRoot, which is correct - that
one really is at C:\Enrollment.
A PXE-imaged display ends up with no GE-Enforce client at all. Confirmed on
579C144, 2026-08-06:
is the client installed? NOT FOUND
scheduled tasks that would run it? NONE
Not a broken configuration - nothing had ever tried. Install-ShopdbKiosk.ps1
downloads itself from {BaseUrl}/installers/kiosk over HTTPS, and ShopDB is only
reachable after the bay joins the AESFMA wifi SSID, so it cannot run during
imaging. Nothing was arranged to run it afterwards.
09-Setup-Display.ps1 now registers 'ShopDB Kiosk Bootstrap' as a SYSTEM task at
boot and every 15 minutes. Install-ShopdbKiosk-WhenOnline.ps1 does nothing until
ShopDB answers, then runs the vendor installer once, verifies BaseUrl and the
enforce task exist, deletes the staged key and unregisters itself. A bay cabled
up days later still self-configures.
It deliberately does not reimplement the installer - it waits, invokes, verifies
and cleans up, so the vendor script can be replaced wholesale without touching
this.
The key file is LABELLED (collector=, fetch=) rather than positional. The two
tokens are not interchangeable and a mix-up is silent: a fetch token in the
collector slot leaves asset reporting broken while everything looks configured.
A bare unlabelled line is ignored rather than guessed at. Missing keys are not
fatal - the fetch token is unnecessary on an IP-allowlisted subnet, and the
installer skips the asset-report task rather than failing.
Staged on the share: the bootstrap, the vendor installer under kiosk\, and the
updated display setup. The key itself is NOT staged yet - see the commit
discussion; it needs to be labelled with which scope it carries first.
Three defects found by the first real run, on 579C144.
16 GB collection, zip failed. run-enrollment harvests
C:\ProgramData\Microsoft\Provisioning into C:\Logs\PPKG, so the 8 GB
provisioning package existed TWICE under the trees being copied. Compress-Archive
cannot exceed 2 GB and died with "stream was too long". Copies now exclude
*.ppkg, *.wim and *.iso and cap individual files at 100 MB.
Base-URL detection matched the wrong value. The name pattern accepted anything
containing "base", so it picked up baseVersion=2.0.2 from
HKLM:\SOFTWARE\GE\SFLD\Credentials and then probed "2.0.2/api/docs" - making
both UNREACHABLE lines meaningless. Now matches the name exactly and requires the
value to start with http.
Unreachable was reported as a bare failure. ShopDB is only reachable once the bay
has joined the AESFMA wifi SSID, so on the imaging LAN or plain wired an
unreachable result is EXPECTED. It now says so, rather than looking like a fault.
The run still answered the question it was written for: no enforce client is
installed and nothing is scheduled to call the API, so this bay was never going
to reach GE-Enforce regardless of network.
Disabling a startup item through Task Manager or Settings does not remove the Run
key or the Startup shortcut - Windows writes a flag into StartupApproved instead.
So the collector would have listed those entries as present with no indication
they had been switched off, and a bay where someone had already tidied up would
look identical to one where nothing autostarts.
Reads all five StartupApproved keys, including Run32 for the WOW64 view, and
decodes the first byte: 02/06 enabled, 03/07 disabled.
This matters for the current investigation because turning an item off is a
per-machine workaround - the next imaged bay installs and starts it again. The
fix is not installing it, which needs the app-to-autostart mapping this section
preserves.
Every imaging investigation so far has meant walking to a bay and hand-copying
files, and each time discovering another thing we wished we had grabbed at the
same moment. This takes the lot. Read-only; changes nothing.
Collects: computer name (active AND pending, which is how you tell whether the
package's H<serial> -> F<serial> rename has landed or is still queued for the
next reboot), dsregcmd enrollment state, the package self-checks, installed
applications, everything that starts by itself, kiosk shortcut targets, Edge
policy, GE-Enforce config, drivers, disk, network, provisioning sessions, and the
whole of C:\Logs, Panther and the provisioning data plus the diagnostics evtx.
Two things it does deliberately:
Autostart is captured across all four surfaces - Run/RunOnce in BOTH registry
views, all Startup folders, non-Microsoft scheduled tasks, and auto-start
services outside C:\Windows. That combination is what identifies which installer
planted a given autostart entry. A 32-bit installer's Run key lands under
Wow6432Node where 64-bit tooling never looks, which is exactly how the old
Dashboard/Lobby autostart survived an earlier purge.
It TESTS the ShopDB GE-Enforce API rather than just reporting config. Config on
disk proves nothing - a client can be present, configured, and never once
succeed. It reports whether a client exists at all, whether anything is scheduled
to run it, the configured base URL, and then actually probes the endpoint. A 401
is a good result: it proves DNS, routing and TLS work and the service answered.
Only a timeout or DNS failure means unreachable. No token is sent.
Deliberately avoids Win32_Product - querying it triggers an MSI reconfigure of
every installed product, which is slow and can change the machine.
Run it BEFORE lockdown. What it captures is known-CURRENT, not known-good: a bay
straight off the line carries applications that should not be there, because
preinstall.json entries without a PCTypes filter install everywhere. The point is
to have an exact record of what imaging really produces so the unnecessary items
can be identified and filtered. After lockdown you cannot tell whether something
is absent because lockdown removed it or because imaging never installed it.
Staged on the enrollment share alongside the other shopfloor-setup scripts.
REFERENCES (section 13, was six lines, now a map)
Where every referenced thing lives, so the document is usable by someone without
today's context: this repo, the PXE server share, shopdb-flask, the on-bay log
locations, and the evidence bundles each claim came from. Plus re-derivation
recipes - pulling startnet out of boot.wim, extracting customizations.xml from an
8 GB .ppkg without unpacking it, reading the provisioning evtx, and the two
validation commands - and the 2026-08-06 commit list.
LTSC AUTOSTART - CORRECTED
Section 7.1 previously argued PESetup might already auto-start: AutoStartCountDown
is 31, disableauto.json is absent, and the operator on 579C144 clicked Next at 11
seconds, so nobody had waited long enough to find out.
Per the team's operational experience that is wrong for our media. LTSC builds
cannot autostart - the SELECT OS confirmation is mandatory regardless of the
countdown, and every image we produce is Windows 11 LTSC 24H2.
So the countdown is not a route to zero-touch. That promotes re-implementing the
imaging step from a contingency to the actual decision, and the risk table and
open questions now frame it as "re-implement, or accept one click per build" -
zero-touch being the only thing that requires it. The hands-off test is kept, but
only to confirm and record the behaviour first-hand.
Design proposal, nothing built. Two ideas in one architecture: the image becomes
thin (partition, apply, join Entra, install the enforce client - everything else
is data pulled from the API and enforced by GE-Enforce), and imaging becomes a
job rather than a menu (open a machine in ShopDB, click Create this PC, pick the
zeroed box waiting in WinPE, and it images start to finish reporting progress
onto that machine's record).
Every claim about current behaviour is sourced from 2026-08-06 evidence -
decompiled binaries, live logs off bay 579C144, and the live share - and where
something is inferred it says so.
Covers: why the current early-decision chain fails silently (five worked examples
from one day); what already exists to build on; per-machine configuration records
replacing 147 .reg files and two CSVs, referencing payloads rather than embedding
them; the dual-NIC design with the firewall work that gates it; whether PESetup
needs replacing at all; the BPRT token shelf-life constraint on pre-imaged stock;
an ownership table; six delivery phases each useful alone; risks; and seven open
questions.
Two findings in here are worth acting on before any of it:
disableauto.json is ABSENT from the media and AutoStartCountDown is 31, while
the operator on 579C144 clicked Next at 11 seconds. PESetup may already
auto-start and nobody has waited long enough to find out. One bay, hands off
the keyboard, count to 40.
dnsmasq is already safely bound (listen-address + bind-dynamic) so DHCP will
not leak onto a GE-side NIC - but Samba binds to all interfaces and every ufw
rule is ALLOW IN Anywhere, including SSH on a pxe/pxe account. That work gates
dual-homing.
The PXE server address appeared 14 times and the share credentials 13, so moving
the server or rotating the account meant editing every net use in the file and
rebuilding boot.wim - with no way to tell from a diff whether one had been
missed. Now set once at the top as PXESERVER / SHAREUSER / SHAREPASS.
Exactly one literal address remains, the definition on line 6. Prose in REM
comments is left alone so the paths stay readable.
This also reaches the post-boot self-heal for free. startnet already writes
C:\Enrollment\fetch-source.txt, and both Fetch-StagingPayload.ps1 and
Verify-And-Heal-Staging.ps1 read it (line1=UNC, line2=user, line3=pass) with
their own hardcoded values only as a fallback. Confirmed on 579C144:
[13:30:35] [INFO] fetch-source.txt: UNC=\172.16.9.1\enrollment user=pxe-upload
[13:30:36] [INFO] Mounting \172.16.9.1\enrollment as Z: (attempt 1/5)...
so the seeded file wins and the heal follows PXESERVER automatically. My earlier
claim that the heal scripts were hardcoded was wrong - they were already
config-driven and startnet was already seeding them. The hardcoding was all in
startnet itself.
Verified: set on line 13, first use on line 20; parens balance; every goto
resolves; 1005 CRLF lines with no bare LF; the deployed copy contains exactly one
literal address. boot.wim md5 7a9b832b.
Dell and GE manifests spell the same folder inconsistently - OptiPlex vs
Optiplex. Windows does not care, but the Samba share sits on a case-sensitive
filesystem, so a blind mkdir -p created a SECOND tree and the drivers split
between them.
That is exactly how the OptiPlex Micro 7020 pack went missing: the manifest asked
for OptiPlex/D13MLK while the 3.2 GB zip sat in Optiplex/D13MLK. PESetup found no
pack, logged a warning rather than an error, and the bay imaged with no network
drivers - so DNS failed at first boot and bulk enrollment could not reach the
CDN. Symptoms three steps from the cause.
mkdir_ci walks the path one component at a time and reuses whatever is already
there whatever its case, creating only genuinely new components. Callers must use
the RETURNED path, since it may differ in case from the requested one. Falls back
to plain mkdir -p rather than skipping a download if the resolve fails.
Both spellings still exist in the live catalogues; scripts/lint-driver-catalogue.py
reports a case mismatch as an error so the pair cannot silently drift again.
PC PURPOSE / TENANT FOR STANDARD AND ENGINEER
The capability was already there - the media carries all six MCL packages
(PERS/SH/LOAN x GCCH/RoW) and Start-BulkEnrollOrchestrator.ps1 picks one from
whatever Select-PCConfig.ps1 returns. The problem was WHEN it asks: a GUI at
first logon, so an otherwise unattended build stops at a dialog until someone
walks over.
Now startnet asks after Standard/Engineer is chosen, while the tech who knows
what the machine is for is standing at it, and writes {Tenant}_{Purpose} to
C:\Enrollment\pc-config.txt. A shim at the path the orchestrator already calls
returns that value and falls through to the vendor dialog when it is absent or
malformed - so "ask at first logon" stays available and bays imaged before this
are unaffected. The vendor script is preserved as Select-PCConfig-vendor.ps1;
replace THAT when the vendor ships a new one.
The tenant is not cosmetic. Insert-BPRTToPPKG.ps1 matches the bulk token on
Tenant AND Purpose, so this selects which token gets injected.
ONLY THE CHOICE IS PRE-SEEDED, deliberately. Injection stays at first logon
because it fetches an encrypted token table from mcl.dwcdn.geaerospace.com,
unreachable from the isolated imaging LAN - which is also why routing MCL
packages through run-enrollment.ps1 would not work: provtool would get a package
whose token is still the Exp_XXXXXXXX placeholder and join nothing.
Standard/engineer previously skipped staging entirely (no ppkg, no PCTYPE), so
the staging gate now also admits PCCONFIG.
3D PARTS KIOSK
The display submenu offered Dashboard and Lobby only. The GE-Enforce display
scope has always had a third entry - 3DPrintRoom -> /shopdb/parts-kiosk - so a
parts kiosk could only be set by editing display-type.txt by hand after imaging.
Added as option 3. The value is a KEY into that map, so the spelling matches
exactly.
Verified: startnet parens balance, every goto resolves, 994 CRLF lines with no
bare LF; shim parses clean. Deployed - boot.wim 4d16c946, shim staged with the
vendor dialog preserved.
OWNERSHIP.md
Every expensive bug in this pipeline has had one shape: two systems setting the
same thing, last writer winning silently. Four happened on 2026-08-06 alone -
computer name (package vs run-enrollment), drive letters (PESetup vs a volume
finder), enrollment (package vs an at-logon -ManualFallback task that syspreped
finished machines), kiosk URLs (GE-Enforce vs site-config). Each is written down
with the evidence so the next person deletes a writer instead of adding one.
share-drift.py
The share is production and the repo is meant to describe it, but drift runs both
ways: live hand-edits nobody committed, and repo fixes never deployed. The
unattend outage lived only on the share while the repo copy was fine, and nothing
compared them.
Each mapped pair is classified. git-owned means the repo wins and the pair must
match - those fail the run. unreconciled means the two have genuinely diverged
and nobody has decided; reported, not failed. The unattends are unreconciled on
purpose: live is ~17 KB against ~12 KB in the repo, so a blind push would regress
production. Reads over SSH via base64 so BOM and CRLF survive the hop.
First run: 8 git-owned pairs all match, 4 known-unreconciled.
Run-ShopfloorSetup.ps1
Corrects a comment that was actively misleading. It claimed shopfloor PCs are
"vanilla by design" and that the orchestrator runs -ManualFallback to skip BPRT
injection and the package entirely. Shopfloor bays DO enrol - the SFLD package
joins Entra with its BPRT token and a human assigns the device category in
Intune. -ManualFallback runs sysprep /oobe /reboot, which is why wiring it to an
at-logon task destroyed the deployment chain.
The absent Entra wait is still correct, for a different reason: at that point the
bay is on the isolated PXE LAN with no route to Entra (579C144 held 172.16.9.81
and 172.24.19.142, neither in the production ranges). sync_intune retries until
the tech re-cables. "Entra ID Joined: false" right after imaging is normal.
TWO SCRIPTS, DIFFERENT AUDIENCES
playbook/scripts/preflight.ps1 runs at the bay, called by startnet once the
media is mapped. It checks the four things that come straight out of PESetup's
own behaviour:
secure boot GatherDataSelection fails outright when SecurebootEnabled != 1
disk >= 120GB MinRequiredSpaceWithoutCompression is 128849018880
driver match reimplements GetDriverByModel - family filter, untrimmed
comma-separated substring tokens, first match wins - and checks
the pack is actually on the media
media age media expires 30 days after build; Media.tag's timestamp is the
local proxy
The driver check is the one that earns it. A miss is only a WARNING to PESetup,
so the bay images with no NIC, DNS fails at first boot, and enrollment cannot
reach the CDN - a symptom three steps removed from the cause. Advisory by
design: it reports and pauses on a blocker, the tech decides. Lives on the
enrollment share so it can be fixed without rebuilding boot.wim.
scripts/preflight.py runs on the server before a build day and aggregates
everything already built - driver catalogue lint, unattend lint, per-PCTYPE
media view verify - plus a new advisory firmware-coverage check that lists
catalogued models with no BIOS models.txt entry. That last one is how the
OptiPlex 7020 family sat uncovered: 127 catalogued models, 58 covered today.
First run: driver catalogues clean, all three unattends clean, firmware coverage
advisory only. PREFLIGHT PASSED.
Verified: both scripts parse clean (PowerShell parser / python), startnet parens
balance, every goto resolves, 915 CRLF lines with no bare LF. Deployed -
boot.wim md5 99fd3132, preflight.ps1 on the share.
Three changes aimed at the same failure mode: a bay that images green and is
silently unusable, diagnosed only by walking over and copying files off by hand.
VERIFY STAGING (startnet, at :pctype_done)
Checks pc-type.txt, Run-ShopfloorSetup.ps1, shopfloor-setup/common and
shopfloor-setup/<PCTYPE> exist on the applied volume before anything depends on
them, retries the small trees once, and prints a loud banner if the retry does
not fix it. The enrollment package is checked but never blindly re-pulled - it is
8 GB, so a miss is reported instead.
Done here because a tech is still at the machine: a short copy found in WinPE
costs 30 seconds, the same copy found at first logon costs 20 minutes, and found
never costs a rebuild. Four Display bays sat green at stage 2 for weeks.
HARVEST LOGS (startnet, after PESetup exits)
Collects X:\*.log, the generated X:\Unattend.xml, PESetup's own log from the
target's Panther directory and winpe-staging.log into
\<server>\enrollment\imaging-logs\<serial>\, plus a build-context.txt naming
PCTYPE, PPKG, machine number and media. All of it was being discarded at reboot.
Runs after PESetup exits so the logs are final, which means re-mapping Y: since
cleanup already dropped it. Best-effort throughout - a bay must never fail to
reboot because a log copy failed.
W: WAIT CAP 20 -> 45 MINUTES
270 polls instead of 120. Fine on NVMe either way, but a WIM apply can exceed 20
minutes on slow media, and the failure mode is the os_not_found banner plus
nothing staged. The loop still exits the moment the SYSTEM hive appears.
RUN-ENROLLMENT RUNS ONCE
Marker at C:\Enrollment\.ppkg-applied, written on exit 0 and also on 0x800700B7
ERROR_ALREADY_EXISTS. Observed running twice on 579C144; the second pass
re-applied a pending rename over the package's own and otherwise did nothing.
Verified: startnet parens balance, every goto resolves, 899 CRLF lines with no
bare LF; run-enrollment parses clean under the PowerShell parser. Deployed -
boot.wim md5 159c2a4d, live run-enrollment dce9d50a.
