OWNERSHIP.md
Every expensive bug in this pipeline has had one shape: two systems setting the
same thing, last writer winning silently. Four happened on 2026-08-06 alone -
computer name (package vs run-enrollment), drive letters (PESetup vs a volume
finder), enrollment (package vs an at-logon -ManualFallback task that syspreped
finished machines), kiosk URLs (GE-Enforce vs site-config). Each is written down
with the evidence so the next person deletes a writer instead of adding one.
share-drift.py
The share is production and the repo is meant to describe it, but drift runs both
ways: live hand-edits nobody committed, and repo fixes never deployed. The
unattend outage lived only on the share while the repo copy was fine, and nothing
compared them.
Each mapped pair is classified. git-owned means the repo wins and the pair must
match - those fail the run. unreconciled means the two have genuinely diverged
and nobody has decided; reported, not failed. The unattends are unreconciled on
purpose: live is ~17 KB against ~12 KB in the repo, so a blind push would regress
production. Reads over SSH via base64 so BOM and CRLF survive the hop.
First run: 8 git-owned pairs all match, 4 known-unreconciled.
Run-ShopfloorSetup.ps1
Corrects a comment that was actively misleading. It claimed shopfloor PCs are
"vanilla by design" and that the orchestrator runs -ManualFallback to skip BPRT
injection and the package entirely. Shopfloor bays DO enrol - the SFLD package
joins Entra with its BPRT token and a human assigns the device category in
Intune. -ManualFallback runs sysprep /oobe /reboot, which is why wiring it to an
at-logon task destroyed the deployment chain.
The absent Entra wait is still correct, for a different reason: at that point the
bay is on the isolated PXE LAN with no route to Entra (579C144 held 172.16.9.81
and 172.24.19.142, neither in the production ranges). sync_intune retries until
the tech re-cables. "Entra ID Joined: false" right after imaging is normal.
TWO SCRIPTS, DIFFERENT AUDIENCES
playbook/scripts/preflight.ps1 runs at the bay, called by startnet once the
media is mapped. It checks the four things that come straight out of PESetup's
own behaviour:
secure boot GatherDataSelection fails outright when SecurebootEnabled != 1
disk >= 120GB MinRequiredSpaceWithoutCompression is 128849018880
driver match reimplements GetDriverByModel - family filter, untrimmed
comma-separated substring tokens, first match wins - and checks
the pack is actually on the media
media age media expires 30 days after build; Media.tag's timestamp is the
local proxy
The driver check is the one that earns it. A miss is only a WARNING to PESetup,
so the bay images with no NIC, DNS fails at first boot, and enrollment cannot
reach the CDN - a symptom three steps removed from the cause. Advisory by
design: it reports and pauses on a blocker, the tech decides. Lives on the
enrollment share so it can be fixed without rebuilding boot.wim.
scripts/preflight.py runs on the server before a build day and aggregates
everything already built - driver catalogue lint, unattend lint, per-PCTYPE
media view verify - plus a new advisory firmware-coverage check that lists
catalogued models with no BIOS models.txt entry. That last one is how the
OptiPlex 7020 family sat uncovered: 127 catalogued models, 58 covered today.
First run: driver catalogues clean, all three unattends clean, firmware coverage
advisory only. PREFLIGHT PASSED.
Verified: both scripts parse clean (PowerShell parser / python), startnet parens
balance, every goto resolves, 915 CRLF lines with no bare LF. Deployed -
boot.wim md5 99fd3132, preflight.ps1 on the share.
Well-formed XML is not enough. Windows Setup validates against a schema and one
bad value invalidates the WHOLE answer file for its pass, so the machine stops at
a dialog with nothing configured and the only clue is an XPath buried in
C:\Windows\Panther\setupact.log.
On 2026-08-06 every shopfloor, standard and engineer build was failing exactly
that way:
/settings/RunSynchronous/RunSynchronousCommand/[Order="16"]/Path
Description = Value is invalid. hrResult = 0x80220005 pass = specialize
An inlined "powershell.exe -Command ..." had grown to 676 characters in a field
capped at 259, and had been broken since the previous evening. Every check here
is mechanical and would have caught it before a bay was booted:
path-too-long RunSynchronousCommand/Path > 259
cmdline-too-long SynchronousCommand/CommandLine > 1024
description-too-long Description > 256
duplicate-element a once-only element appearing twice
not-well-formed XML does not parse
bom UTF-8 BOM (the live files have none, and python's
utf-8-sig silently ADDS one when writing them back)
unknown-token %token% PESetup will not substitute; %WINDIR% and the
other shell variables are excluded or it cries wolf
Reads over SSH via base64 so a BOM or CRLF survives the hop unchanged. Exits
non-zero on any ERROR so it can gate a deploy.
Immediately found a second live defect the manual review had missed: a 329-char
Description on gea-engineer Order 11, which would have failed the oobeSystem pass
once specialize started passing. All three live image types and both repo copies
now lint clean.
CopyPackages copies the whole of <media>\Deploy\Applications to the target,
recursively, with no manifest and no filtering, so every bay built from the
shared gea-shopfloor media receives every shopfloor payload whatever its type.
The obvious fix - repoint a path inside the shared media as each bay picks its
type - races. Bays image concurrently and CopyPackages is fail-fast, so moving a
path while another bay is mid-copy either hands it the wrong payload or fails its
imaging outright, intermittently and unreproducibly.
Give each type its own media directory instead, built once and never mutated.
Everything in it is a symlink, so a type costs inodes rather than gigabytes and
the shared media stays untouched while bays read it. Samba already serves this
pattern: follow symlinks and wide links are on, and every image root reaches
Sources, Operating Systems, Out-of-box Drivers and Packages the same way.
Types come from the enrollment share's menu.json, the file the WinPE picker
already renders, so the media set cannot drift from the boot menu. Deliberately
NOT mirrored: the FlatUnattendW10.xml.pre-* backups and stray logs beside the
real unattend - the view is the clean set PESetup actually reads. Each type is
built into a scratch dir and swapped, so a bay mapping the path mid-refresh sees
the old tree or the new one, never a half-built one.
Applied on 172.16.9.1: 10 types, 190 links, all 60 critical paths resolve. An
SMB client sees ordinary directories and files - Sources, Control,
FlatUnattendW10.xml at 18078 bytes, HardwareDriver.json at 46113 - which is the
behaviour that actually matters, since PESetup reads this over SMB from WinPE.
Note for anyone extending this: passing the server-side script on stdin to
"echo pxe | sudo -S bash -s" is a silent no-op. Bash inherits the password pipe,
reads EOF, runs nothing and exits 0. It travels as a heredoc for that reason.
A driver miss is only a warning: GetDriverByModel returns null, PESetup logs
"driver for [MODEL] not found" and images the machine anyway. The bay comes up
with no NIC and no WiFi, DNS fails at first boot, and bulk enrollment cannot
reach the CDN - symptoms far enough from the cause that the OptiPlex Micro 7020
pack sat missing and the Display MicroPC failures were blamed on a drive letter.
Reimplements the matcher from the decompiled source (docs/PESETUP-INTERNALS.md)
and reports what silently breaks it:
virtual-platform one such entry flips the tool into virtual-only mode and
hard-fails every physical machine
empty-token a trailing comma yields "", and Contains("") is true for
every model, so that entry swallows the catalogue
token-whitespace Split(',') does not trim, so " OptiPlex 3010" needs the
space present in the model string too
case-mismatch the share is case-sensitive; Optiplex vs OptiPlex splits the
tree and the pack is never found
missing-zip referenced pack absent
family-mismatch the family filter runs first, so a token whose line
contradicts the family field can never match
shadowed first match wins, so a later entry may be unreachable
duplicate-token osId is not part of the match, so a win10 pack can land on a
win11 build purely by ordering
--models resolves real WMI model strings through the same code, which is the
check that actually predicts a no-driver build. Exits non-zero on ERROR or
CRITICAL so it can gate a deploy.
Verified both ways: a synthetic catalogue carrying each defect reports all eight
and exits 1; the three live catalogues on 172.16.9.1 come back clean at 0. The
file listing needs find -L and the same anchoring as destinationDir - without
either, every zip check silently passes.
parse_bios_catalog kept the 'latest' BIOS per model with a string compare, so e.g. '1.20.1' > '1.9.0' was False and it wrongly retained the older 1.9.0. Added _ver_tuple() and compare tuples of ints so the genuinely newest firmware wins.
Single-site bay-stuck issue at WJ: GE Intune Report IP script filters
Get-NetIPAddress on StartsWith("10.") and posts everything matching
to the GE Tines webhook. Bays at WJ get the PXE LAN 10.9.100.x IP
captured and reported -> GE backend tags bays as on a non-corp 10.x
subnet -> dynamic group eligibility for SFLD policy never matches.
Other GE sites work because their PXE LANs aren't on 10.x at all.
Renumber PXE LAN to RFC1918 172.16.9.0/24 so the GE filter naturally
skips wired PXE addresses without any disable-NIC dance.
Server-side already in flight (netplan dual-bound, dnsmasq scope +
boot URL repointed, blancco preferences + grub.cfg + iPXE GetPxeScript
all sed'd to 172.16.9.1). This commit is the playbook / scripts /
docs side: 109 hits across 35 files sed'd in one shot.
After this lands + boot.wim is rebuilt + bays renumber off DHCP,
the 10.9.100.1 binding will be dropped from netplan as the final
cleanup step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related fixes:
1. Hard-coded BIOS push path was /srv/samba/enrollment/BIOS, which does
not exist on the live PXE server. Real path is the shared
/srv/samba/winpeapps/_shared/BIOS/ where check-bios.cmd lives and
playbook task pxe_server_setup.yml:485 deploys Flash64W.exe + the
per-model BIOS .exe files.
2. Generated models.txt was 2-column (ModelSubstring|BIOSFile) but
check-bios.cmd reads tokens=1,2,3 with delims=| and uses field 3
for the version compare. Without the 3rd column, the version-skip
logic never engages and every imaged PC re-flashes BIOS on every
boot. Now writes 3-column (ModelSubstring|BIOSFile|Version).
3. The script overwrote the live models.txt with only the entries it
touched in the current run. Live had 50+ entries; a single-model
run wiped the other 49. Now prints the lines and asks the operator
to merge them into playbook/shopfloor-setup/BIOS/models.txt and
re-deploy via scripts/deploy-bios.sh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/diagnostics/Capture-LockdownState.ps1 captures Windows endpoint
state at three lifecycle checkpoints so the deltas isolate which phase
delivered (or failed to deliver) each component:
- pre-category - PPKG-enrolled, no Intune category yet
- post-category - category-driven assignments arrived, pre-lockdown
- post-lockdown - kiosk + autologon + AppLocker fully landed
Bumped from the previous 2-stage (pre/post) version. Legacy 'pre'/'post'
aliases preserved.
Captures additions driven by the SFLD-DSC v2.0.2 post-mortem:
- IMECache file listing (catches missing sastoken.txt)
- DSCDeployment.log + version.txt copied from C:\pc\
- SFLD\DSC payload listing
- C:\Logs\BPRT\ runtime state (criticalChecks.json, packageInfo.json)
- C:\WCDApps\ deploy verification
- Windows\Provisioning\Diagnostics copy
- Tasks-RunHistory.csv with LastRunTime + LastTaskResult per task
- DeviceManagement-Events.csv (MDM 429s, AAD token failures)
- Provisioning-Events.csv (PPKG runtime errors)
- MDM-Certificates.csv (enrollment cert health)
scripts/diagnostics/snapshot-runbook.txt: step-by-step ops guide
covering when to fire each stage, where output lands, how to ship it
back via image-upload share, and which files to compare first when
diffing.
Shebang was \`#!/usr/bin/env python3\r\` which errored with
\`/usr/bin/env: python3\r: No such file or directory\`. Pure line-ending
fix, no content change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates per-type enforcers (CMM, Keyence, Machine, Common, Acrobat)
into one dispatcher driven by pc-type.txt + site-config and a share-side
manifest layout. Same share is now the single source of truth for routine
software updates without re-imaging.
Runtime:
common/GE-Enforce.ps1 SYSTEM scheduled task. Reads
common/manifest.json plus optional
<pcType>/manifest.json and
<pcType-subType>/manifest.json.
Dispatches each entry through the lib.
Writes _outputs/logs/<hostname>/status.json
on the share after each cycle for fleet
monitoring.
common/Register-GEEnforce.ps1 Task registration. Triggers: AtLogOn +
every 5 min (jittered per-PC from
hostname hash) + daily at 05:45,
13:45, 21:45 EST shift windows.
Unregisters legacy per-type tasks on
install so the two coexist at most for
the duration of a single enforce cycle.
common/Deploy-GEEnforce.ps1 Retrofit helper for already-imaged PCs
(admin-run; copies runtime + registers
task + optional immediate trigger).
Library (common/lib/Install-FromManifest.ps1):
- New Type values: PS1, BAT, File, Registry, INF
- New DetectionMethod values: Always, MarkerFile, ValueMatches, pnputil
- TargetHostnames filter (exact + -like wildcards, ANDed with PCTypes)
- Schema version check (logs WARN on manifest newer than lib MAJOR)
- Auto-writes MarkerFile on successful one-shot PS1/BAT/CMD runs
- MSI log scan on failure surfaces meaningful install errors
- Lib version bumped 2.0 -> 2.1 for TargetHostnames
Observability:
common/monitor-fleet-status.py Scans _outputs/logs/*/status.json for
stale check-ins, failed scopes, and
version drift. Respects scope (dir-name),
PCTypes, and TargetHostnames filters so
entries excluded from a PC do not
false-flag as drift.
Regression harness:
common/test/ Parameterized VM harness + README
covering every action type plus
rollback, bad/missing SFLD creds, and
schema versioning.
Imaging integration:
Run-ShopfloorSetup.ps1 now stages GE-Enforce.ps1 and lib to
C:\Program Files\GE\Shopfloor\ and invokes Register-GEEnforce.ps1
at the end of setup. Legacy Register-CommonEnforce invocation is
kept for the transition; it and the legacy per-type enforcer files
are dead code once Register-GEEnforce runs and will be removed in a
dedicated cleanup pass.
Standard-Machine manifest:
eDNC entry bumped 6.4.3 -> 6.4.5. DetectionValue pinned to the
4-part FileVersion 6.4.5.0 verified against a fresh install in the
Win11 analyzer VM. UDC DetectionValue pinned to 1.0.34 (registry
stores 3-part for UDC; verified live).
scripts/mirror-from-gold.sh:
Restructured around share-root rsyncs (one pass per Samba share)
to close gaps in the prior per-subdir layout: winpeapps/_shared/
Applications (7.5 GB of Adobe + fonts + Java + Office + OpenText
+ printdrivers + wireless + Zscaler), additional winpeapps image
types, and enrollment flat-layout root files. Adds
--skip-clonezilla and --skip-reports.
Verified end-to-end in the Win11 analyzer VM:
- Every action Type and DetectionMethod round-tripped
- PCTypes filter (Oracle excluded on Shopfloor, Firefox included
on Shopfloor and DESKTOP-*, excluded elsewhere)
- TargetHostnames filter (exact, wildcard, no-match)
- Upgrade path: XML hash bump + fleet re-copy
- Rollback path: history-archive restore propagates via enforcer,
fleet converges back without per-PC intervention
- Status writeback + monitor script drift detection
- Graceful degradation on bad creds, missing creds, share
unreachable (all exit 0, log clearly, retry next cycle)
Not in this commit (follow-ups):
- Retire legacy per-type *-Enforce.ps1 files and simplify
09-Setup-*.ps1 scripts (coordinated multi-file cleanup)
- Stage 2b: InUseCheck close-and-reopen, ApplyMode gating,
UpdateWindow, .apply-now.txt sentinel, BITS pre-staging,
1618 mutex retry, PostInstallCheck, Uninstall action
- Management app (manifest CRUD + deploy + rollback + fleet view)
- ShopFloor autologon persistence bug (deferred for next imaging
attempt with live registry evidence)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- playbook/startnet.cmd + startnet-template.cmd: after preinstall staging,
xcopy Y:\pre-install\udc-backups to W:\PreInstall\udc-backups so UDC
settings JSONs are available during image deployment. Harvested from
live gold where this block existed but was never committed.
- scripts/mirror-from-gold.sh: update source paths to current taxonomy
layout (pre-install/, installers-post/, blancco/, config/) and add
ppkgs/, scripts/, shopfloor-setup/ sections. Added --delete for exact
mirror semantics. Used to seed the spare PXE server at 10.9.100.2 on
2026-04-16 from gold at 10.9.100.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new docs cover what the system is (boot chain, services, shares,
enrollment layout, data flow) and what to change per site (every
hardcoded value, where it lives, secrets handling).
scripts/mirror-from-gold.sh replicates content from an existing PXE
server (Operating Systems, drivers, packages, custom installers, BIOS,
PCDMIS, Blancco custom image, site-config) onto a freshly-installed PXE
server. Translates the legacy flat enrollment layout on the source into
the reorganized taxonomy (ppkgs/, pre-install/installers/, installers-
post/cmm/, blancco/, config/) on the destination. Tolerates rsync
exit 23 (permission-denied subdirs like the OpenText W10shortcuts dir
that is pxe-upload-group-only on legacy servers).
Workstation reorganization:
- All build/deploy/helper scripts moved into scripts/ (paths updated to use
REPO_ROOT instead of SCRIPT_DIR so they resolve sibling dirs from the new
depth)
- New config/ directory placeholder for site-specific overrides
- Removed stale: mok-keys/, test-vm.sh, test-lab.sh, setup-guide-original.txt,
unattend/ (duplicate of moved playbook/FlatUnattendW10.xml)
- README.md and SETUP.md structure listings updated, dead "Testing with KVM"
section removed
- .claude/ gitignored
Enrollment share internal taxonomy (forward-looking; existing servers
unaffected since they keep their current boot.wim with flat paths):
- Single SMB share kept (WinPE only mounts one Y: drive), but content now
organised into ppkgs/, scripts/, config/, shopfloor-setup/, pre-install/{bios,
installers}, installers-post/cmm/, blancco/, logs/
- README.md deployed to share root explaining each subdir
- New playbook tasks deploy site-config.json + wait-for-internet.ps1 +
migrate-to-wifi.ps1 explicitly (were ad-hoc on legacy servers)
- BIOS subdir moved into pre-install/bios/, preinstall/ renamed to pre-install/
- startnet.cmd + startnet-template.cmd updated with new Y:\subdir\ paths
- Bumped GCCH PPKG references v4.9 -> v4.10
Blancco USB-build fixes (so next fresh USB install boots Blancco end-to-end
without the manual fixup we did against GOLD):
- grub-blancco.cfg: kernel/initrd switched HTTP -> TFTP (GRUB's HTTP module
times out on multi-MB files); added modprobe.blacklist=iwlwifi,iwlmvm,btusb
(WiFi drivers hang udev on Intel business PCs)
- grubx64.efi rebuilt from updated cfg
- Playbook task added to create /srv/tftp/blancco/ symlinks pointing at the
HTTP-served binaries
run-enrollment.ps1: OOBEComplete is now set AFTER PPKG install (Win11 22H2+
hangs indefinitely if OOBEComplete is set before the bulk-enrollment PPKG runs).
Also includes deploy-bios.sh / pull-bios.sh / busybox-static / models.txt
that were sitting untracked at the repo root.
