Sites no longer need either MAC in site-vars.
pxe_nic_mac empty: before anything changes, the playbook runs
pxe-pick-nic --auto. It takes the one wired port with a link, accepts it
only if pxe-nic-check calls it isolated (no LLDP/CDP/802.1X, no other
DHCP server, at most 10 hosts heard in 65 s), and pins it. The choice is
saved in /etc/pxe-server/nic-pick.conf and never re-guessed at boot.
When it cannot tell (no link, two links, not isolated) nothing changes,
and the console banner says why and to run sudo pxe-pick-nic: pull the
imaging cable and plug it back in, and the port that blinked is the
imaging one. Both helpers are installed before the pick, so that works
after a refused first boot. The site's corp domains and allowed
switches are passed to the check.
corp_nic_mac empty: every other wired port (en*, eth*) is a catch-all
DHCP client, so GE can be plugged in at install or later.
- dnsmasq stays pinned with interface=pxe0. A new check refuses any
dhcp-range outside the imaging subnet, and proxy ranges.
- The catch-all ports ignore offers from 172.16.9.1.
- UFW allows SSH and 9009 by source on any interface, and keeps those
rules when the old any-interface rules are removed.
- pxe_nic_name must not look like en*/eth* in this mode.
pxe-netplan-apply.sh takes 3 or 5 args. Without a pinned GE NIC it:
- renames stray names (the old pxe0 after a re-pick, a leftover corp0)
to enx<mac>, so the catch-all takes them, and restores them on revert;
- checks that every NIC that had a DHCP address before has one after,
tracked by MAC.
With both MACs set (the WJ server), netplan, the dnsmasq pin and UFW
render byte-for-byte as before.
Tested in the pxetest VMs:
- Fresh install with no MACs and the GE link down: the imaging port was
picked and pinned as pxe0 with 172.16.9.1, and dnsmasq bound DHCP to
pxe0 only. GE plugged in afterwards got a lease, SSH worked, and the
guard reported pxe0 isolated.
- --auto refuses with two links.
- The interactive pick works by pulling and replugging the link.
- A 3-arg re-apply is UNCHANGED.
Unit tests (fake sysfs) cover the re-pick and pinned-to-catch-all renames.
A freshly installed server could not build standard or engineer: about
8 GB they install from existed only on the running server.
scripts/capture-site-content.sh pulls it over SSH into the gitignored
payload:
- _shared/Applications/extra (Office, Adobe, OpenText, Java, fonts, Wi-Fi,
Zscaler), without pxe-status/ (the playbook deploys that);
- each image's own Deploy/Applications files and folders (GE CA
certificates, the Engineering Layer payload), without the repo-managed
Install-EngineeringLayer.ps1 and without stale BPRT copies;
- the links images keep there, as links.txt;
- fresh bios-staging (*.exe plus models.txt, which the Dell update
rewrites), boot.wim, and the known Blancco file set.
It dry-runs by default, compares by checksum (a Blancco config.img had the
same size and time as the stale copy), writes SHA256SUMS, and refuses what
FAT32 cannot hold: files of 4 GiB or more, Windows-illegal names, and
case-only name clashes.
build-usb.sh copies site-content/ unsplit and refuses to build without it
(unless --skip-site-content), or when SHA256SUMS is missing or does not
cover every file. The playbook checks SHA256SUMS, then merges the content
into winpeapps and recreates the links. It only links to folders that
exist, so it never creates an empty _shared BPRT, which would stop the
first MCL import from seeding it. Link lines are validated before anything
is copied.
pull-bios.sh is gone. It read a path the server no longer uses, with a
hard-coded password.
Small Control JSONs have arrived empty or wrong more than once, and the old
check compared sizes only and skipped missing files. A copy of
Deploy/Control on the dev PC still holds two 0-byte JSONs.
Upload-Image.ps1 (tested on Windows PowerShell 5.1 in the win11 VM):
- refuses an unfinished MCL cache: exactly one <OS>-Production.tag (MCL
writes it last), no Control file newer than it, selected OS matches,
every Control/Tools JSON non-empty and valid
- checks payload against the SHA-256 MCL's own Control JSONs publish. MCL
resumes downloads onto old files (FileMode.Append) and then trusts any
file of the right size: the September junk-tail corruption. Offers to
delete the bad cache files so MCL downloads them again
- hashes every file into a manifest, robocopies, POSTs the manifest
- re-sends bad small files over HTTP (no proxy), large ones by deleting
the bad copy and robocopy /J (robocopy skips a same-size, same-time file
as "modified" even with /IS /IT)
Webapp:
- services/upload_verify.py hashes the upload on the server's own disk in
a background thread; reports missing, differing, bad-in-MCL-cache and
stray files; PUT /api/upload/file writes temp, checks hash, renames
- import of the upload folder only after a passed verification, with
every file unchanged (size, mtime, ctime, inode), no stray files, and
the target the OS allows (LTSC: Shop Floor only; else Standard or
Engineer; unknown OS: none). Import, verify and PUT share one lock
- paths matched case-insensitively against the disk, as Samba does
- import page shows the upload, MCL run, verification and stray files
with a remove button; record of each import kept on the image
Reviewed adversarially; all 11 confirmed findings fixed with tests.
42 new unit tests. VM test: clean upload; same-size same-mtime damage on
3 files repaired in one round; stray file blocks import until removed;
Shop Floor refused for a Pro upload; import records and clears the
folder; missing tag, 0-byte JSON, junk-head WIM and late Control file
each refused with nothing uploaded.
Current Clonezilla is amd64-only. The menu now checks the CPU with iPXE's
cpuid --ext 29 (long mode): 64-bit CPUs get the current release (through
Debian's shim under Secure Boot), 32-bit-only CPUs (Pentium 4 without
EM64T, Pentium M, early Atom) get Clonezilla 3.2.0-5 i686, the last
32-bit release, from clonezilla-i686/. Without a cpuid command the 32-bit
build is used, which also runs on 64-bit CPUs; UEFI always gets the
64-bit build. A 'Clonezilla Live 32-bit' item forces it.
prepare-boot-tools.sh downloads it, the playbook deploys it.
Clonezilla Live runs a Debian-signed kernel whose modules (usb-storage,
uas, exfat) are signed by the same build key, so the menu now starts it
through Debian's Microsoft-signed shim with the iPXE shim command, the
same way Blancco uses Ubuntu's. prepare-boot-tools.sh extracts that shim
from the Clonezilla release (EFI/boot/bootx64.efi -> clonezilla/
shimx64.efi) and warns when the shim is not Microsoft-signed or the
kernel not Debian-signed.
Memtest86+ is unsigned and needed Secure Boot off; memory tests now use
the Dell built-in diagnostics (F12 at power-on, Diagnostics), noted in
the menu. The playbook removes the old memtest web dir. Nothing on the
menu needs Secure Boot off any more; the 'disable Secure Boot' screen
remains only as the fallback when a signed path fails.
H1 corp_admin_cidrs (UFW 22/9009 on the corp NIC from these only; refuses
when it would lock out the current SSH client) and corp_ssh_password_auth
(sshd drop-in, passwords only from the imaging subnet). Defaults keep the
current behaviour and print a loud warning.
H2 pxe-netplan-apply.sh: pre-flight refusal (pxe-MAC NIC holds the default
route, a DHCP lease or the route to an SSH client), run from the USB
before anything changes; post-apply checks (names, imaging IPv4, corp
IPv4 back within 30 s, SSH route) and a 120 s systemd-run auto-revert.
H3 restore() puts the previous NIC names back (two-phase rename via
temporary names, so swaps work) before generate + apply.
H4 a failed rename is fatal (restore + exit 1); the play drops the old
any-interface UFW rules only after verifying both NICs and their
per-interface SSH rules.
H5 first-boot checks the playbook exit code, retries at next boot (max 5),
writes /var/lib/pxe-firstboot.failed and /etc/issue + /etc/motd banners.
M1 build-usb.sh verifies iPXE files against the playbook pins, every wheel
against its own package's hashes, drops the unpinned pip fallback,
requires --site-vars or --no-site-vars.
M2 webapp restarts when the code, wheels or unit changed.
M3 ignore-carrier on the imaging NIC; ip_nonlocal_bind kept for the
apache/networkd boot race.
M4 menu suffix must match blancco_kver; missing kernel with no earlier
build is fatal; a live menu with the staged Blancco test entry is not
overwritten.
M5 build-proxmox-iso.sh is unsupported (hard exit); use build-usb --image.
M6 deploy-unattend.sh: password via SSHPASS env and sudo stdin only,
%q-quoted remote args, sha256 check of the live file; lint-unattend.py
reads PXE_SERVER/PXE_USER/PXE_PASS and uses sshpass -e.
M7 the playbook lints the unattend files before the force: yes copies
(lint-unattend.py ships on CIDATA as scripts/); pre-commit ACMR.
M8 Blancco initramfs build: no ignore_errors, pipefail, trap cleanup.
Also: Send-ImagingStage.ps1 deploy, carrier hook install tasks
(pxe-nic-carrier, networkd-dispatcher), L1 imaging-active hold
(allow_disruptive) and no iPXE downgrade below an Updates-page install,
L2 iso-volid, L3 kver pin read from the playbook, L4 cp -rL, L5 refuse
>= 4 GiB WinPE files, L6 PIP_DIR wipe guard, L7 per-package wheel hashes,
no online fallback, L8 check-updates.py (apt_pkg/cached compare, all
components, exit 2 on no debs / bad answers).
- autoinstall: dead mirror + fallback offline-install, so the installed
base is exactly the ISO's; late-commands run offline-apt-install.sh on
CIDATA packages/ (local apt repo, one transaction, errors fail the
install, log /var/log/offline-apt-install.log); first boot no longer
runs dpkg -i; real archive sources written back at the end.
- download-packages.sh --iso: unpacks the ISO's default install source
into a docker image (pxe-iso-base:<version>) and downloads there: all
updates of the base, the explicit list (zip included) and the newest
pool packages with their closure; writes Packages + explicit.list and
checks the set offline in a fresh base container (--network none).
- offline-apt-install.sh: new; also used by that check (--simulate).
- build-usb.sh: ships Packages, explicit.list and the script; refuses a
set without them.
- tests/vm: pxetest-client-nosb (Secure Boot off, e1000e, SATA), one
client at a time.
Clonezilla 3.2.1-6 is gone from the mirrors (the download failed and the
unzip aborted the script before Memtest). Memtest moved its downloads to
download/v<ver>/, and 8.x names the x86_64 UEFI binary mt86p_<ver>_x86_64
with no .efi suffix; the script now finds it.
- webapp/requirements.txt pins all eight wheels with sha256, at the fixed
releases (lxml 6.1.3 for GHSA-vfmq-68hx-4jfw / PYSEC-2026-87, werkzeug
3.1.9 for GHSA-g6x2-hccm-hh4m).
- download-packages.sh fetches exactly that set for Python 3.12 / x86_64
into an emptied folder; PIP_DIR is honoured inside the container (own
mount), files are handed back to the calling user, USE_DOCKER=1 forces
the container on a noble host, an unknown package name now prints apt's
error instead of failing silently, and linux-modules-extra-7.0.* is gone
(it does not exist; 7.0 ships those modules in linux-modules).
- Playbook: the wheel installer refuses any wheel whose sha256 is not
pinned, needs exactly one wheel per pin, and removes an older installed
version by its RECORD first. The webapp copy is a full rsync every run
(was cp with creates: app.py, which never updated an install).
- scripts/check-updates.py: offline debs against noble security/updates
and the pins against PyPI + osv.dev; exit 1 when something needs
updating. Run it before building a USB.
Tested: installer in a Python 3.12 venv (clean install, upgrade over lxml
6.0.4 + werkzeug 3.1.8 leaves no old dist-info, re-run is a no-op, a stray
wheel is refused); check-updates.py flags the April bundle (94 security
updates) and passes the pins.
- Copy the whole webapp/ tree (minus __pycache__) in build-usb.sh and
build-proxmox-iso.sh, so config, services/, views/ and
blancco_evidence_monthly.py reach the server.
- sfdisk --append failure now aborts; the CIDATA partition is the one
starting exactly at the computed sector, and nothing that starts
before it is ever formatted (the old fallback hit the ISO's EFI
partition).
- Volume id and El Torito load sizes come from the ISO
(xorriso -report_el_torito as_mkisofs); grub.cfg must contain
autoinstall after patching.
- --image FILE SIZE|auto builds the same layout in a file without root
(mkfs.vfat -C + mtools, sfdisk on the file). Plus --payload-root,
--skip-enrollment/-bios/-drivers/-boot-tools, --yes.
- tests/vm: libvirt harness (pxetest-server, pxetest-client, isolated
pxetest-imaging network) with fetch-iso/build-image/create/start/
status/stop/destroy.
Every server has an imaging NIC (static, the only one that serves
DHCP/TFTP/HTTP boot/SMB) and a GE corporate NIC (DHCP client, SSH and the
webapp on 9009). The playbook guessed the PXE NIC and the installer put
172.16.9.1/24 on every en* NIC, so a two-NIC box could answer on the GE
network.
- New per-site vars pxe_nic_mac / corp_nic_mac (required, from
playbook/site-vars.yml or -e; template site-vars.example.yml), stable
names pxe_nic_name / corp_nic_name (default pxe0 / corp0, overridable),
pxe_server_ip, pxe_subnet, DHCP range. pxe_imaging_mac / pxe_ge_mac are
derived from them. The play stops before touching anything if a MAC is
unset, malformed, duplicated or not on the box (lists the box's NICs).
- Heuristic NIC detection and the single-NIC netplan task replaced by a
two-NIC netplan template (match macaddress + set-name), installed by
pxe-netplan-apply.sh: validate in a scratch root, back up, generate,
rename, apply, restore the previous config on any failure. cloud-init
network config disabled so it cannot rewrite it.
- dnsmasq: /etc/dnsmasq.d/pxe-nic-only.conf from the vars (interface=,
except-interface=, no-dhcp-interface=); dnsmasq.conf IP/range from vars.
- UFW per interface: imaging NIC 22, 67/udp, 69/udp, 80, 4433, 445, 139,
137/udp, 138/udp, 9009; GE NIC 22 and 9009 only. Old any-interface rules
and the stale 9011 rule removed after the new ones are in.
- sysctl drop-in: IPv4/IPv6 forwarding off; ip_nonlocal_bind so Apache
can bind the imaging IP before the address is up.
- Samba [global]: interfaces = 127.0.0.0/8 <imaging NIC>, bind interfaces
only. Apache 80 and 4433 on the imaging IP only; 9009 stays on all (TODO:
the webapp has no auth).
- autoinstall: DHCP only on en*, no static 172.16.9.1; apt geoip off and
offline-install fallback now that the corp NIC may have a lease.
- Blancco UEFI/Secure Boot: blancco_kver (7.0.0-34-generic) drives the
kernel, initramfs and modules builds instead of the running kernel;
outputs vmlinuz-ubuntu-70, kexec-initrd-70.img, kmod-<kver>.tar.gz as the
menu and blancco-init.sh expect; creates: guards keyed on the full kver.
Ubuntu's shimx64.efi.signed published as blancco/shimx64.efi.
download-packages.sh adds shim-signed, linux-generic-hwe-24.04 and the
pinned linux-image/modules/modules-extra for BLANCCO_KVER.
Rendered with the live MACs and names, netplan generate output is
identical to the live config; pxe-nic-only.conf and the Samba binding
match live directive for directive.
a94c203 added a RunSynchronous Path of 417 chars to both std/eng answer
files. Windows caps it at 259 and then rejects the WHOLE answer file, so
an Engineer build stopped at 'Windows could not parse or process unattend
answer file'. The live files were restored to their pre-a94c203 versions
the same day.
- The status steps now fit: Send-ImagingStage.ps1 -Persist copies itself
out of C:\Deploy (no cmd copy chain), short PowerShell switches, default
arguments left out. Longest Path is 168 chars; lint-unattend passes.
NOT deployed: these files must go live only through deploy-unattend.sh.
- scripts/deploy-unattend.sh: the only route for an answer file to a PXE
server. Lints first and refuses on any error, backs up, swaps
atomically, lints the live copy and restores the backup if that fails.
- .githooks/pre-commit (git config core.hooksPath .githooks): refuses a
commit whose staged unattend fails the linter. Verified by planting a
352-char Path (blocked) and against the exact a94c203 file (rejected).
- Boot menu: also accept SecureBoot read as hex 01, and show the raw values
in the warning so a wrong read can be diagnosed from the screen.
Live since 2026-10-14: dnsmasq hands every UEFI client ipxe-shim.efi
(iPXE's shim, dual-signed under Microsoft UEFI CA 2011 + 2023), which
loads ipxe.efi from the TFTP root, now the iPXE v2.0.0 snponly build
(firmware network and USB drivers; the full build kills USB keyboards).
The per-MAC test tags and folders are gone; the old Broadcom-signed
1.21.1 build stays as ipxe-bcm.efi, the rollback.
- playbook: pinned sha256 vars, the new dnsmasq default line, and a
verified, non-optional task that installs ipxe-shim.efi, ipxe.efi
(snponly), ipxe.pxe and ipxe-bcm.efi (was: copy ipxe.efi and the never
served undionly.kpxe, errors ignored).
- scripts/fetch-ipxe.sh downloads the pinned tarball and shim into
boot-files/ for the USB build; README/SETUP list the step.
- iPXE page: ipxe.efi is staged from snponly.efi, a build that drives USB
itself is refused, and the snponly-shim/snponly root files are gone.
Tested: the playbook task against the real release files (installs the
live hashes; a tampered file stops it before anything is replaced),
ansible --syntax-check, staging, and the full-build refusal.
sbsigntool, zip and curl, the Zscaler root CA in the system store (GitHub is
TLS-intercepted on the corporate NIC; downloads.dell.com is not), and
/var/lib/pxe-webapp/updates.
Updates group:
- Dell drivers and BIOS: check Dell's catalogs against all three driver maps
and models.txt, download with a shared speed cap, repack, preview, apply
with backups and rollback.
- iPXE boot files: check GitHub for the signed Secure Boot set, verify the
sha256 and each file's signer, install atomically, roll back. Includes the
by-hand procedure.
- Blancco: upload a release, stage it, test-boot it from the menu, activate,
roll back.
- BPRT sync: validate a batch uploaded from the MediaCreator Lite cache
(scripts/windows/Sync-BPRT.ps1 or a zip), show the diff, swap with the site
overlay files kept.
Images: one sidebar entry per image with Overview, Drivers, Unattend and OS and
Packages tabs. Every save shows a diff first, keeps the file's format, makes a
backup and can be undone. Driver lint runs inline.
Enrollment: upload standard/engineer MCL packages, one per tenant/purpose,
with archive and restore. Hardware coverage counts engineer as Win11.
playbook/blancco-preferences.xml is now the vendor 7.20 preferences
(schema 4.20, "Non_GEAE_Wired" profile) with only our site overrides:
SMB report share on 172.16.9.1 with auto_backup, WiFi INTERNETACCESS for
cloud licensing (wired stays up for SMB), manual as the fallback process,
bootable report, screensaver 86400. workflow default=true is kept: the
cloud workflow erases unattended, as live 7.15 already did. No custom
fields, matching both the vendor image and live 7.15.
prepare-boot-tools.sh:
- 7.20 initramfs is an uncompressed early cpio (NIC firmware, itself
.xz files) followed by the XZ archive. The old patch decompressed from
byte 0, failed, warned and carried on, so PXE boots would drop network
after copytoram. Walk the cpio to its trailer and patch the XZ part.
- Patch exactly one copytoram match or fail. The looser fallback pattern
matches five hooks in 7.20.
- config.img now carries playbook/blancco-preferences.xml, so the BIOS
(native kernel) and UEFI (HTTP) paths read the same settings. Refuses
to build if the playbook preferences are for another Blancco version.
- The old global replace() also set the 802.1x, WiFi and repeater
passwords to "blancco". Gone with the above.
- Keep preferences.xml size by refilling the NUL gap before the trailing
filler comment (7.20), or trimming the comment (7.15).
- OUT_DIR can be overridden to stage a build outside boot-tools/.
Three bugs, all found by actually running the thing rather than reading it.
1. `--e10` with no value did nothing. Making the flag optional-valued gave it
an empty-string constant, and the code deciding whether to drive the
controller used a truth test - so the documented `--e10` form printed the
full translation and quietly skipped the hardware. No error, because
nothing failed. Now gated on `is not None`, with a comment saying why.
2. A missing part program dumped a traceback. The top-level handler tidied
e10 errors and re-raised everything else. Now a clean message that lists
the part programs it can actually see, which also covers Windows path
confusion.
3. Errors could vanish when piped. stdout is block-buffered and stderr is
not, so the message printed before the output it referred to and a `| tail`
dropped it entirely - looking like a silent failure. stdout is flushed
before writing an error.
Serial access moves behind open_port(), with a termios backend on POSIX and a
Win32 backend reached through ctypes. Deliberately no pyserial: these tools
have to run on locked-down, air-gapped shopfloor machines where installing a
package is a project of its own.
Verified on a real controller from both a Linux laptop and a Windows 11 PC.
Three things that made this harder than it should have been, all now handled:
- CreateFileW returns a HANDLE, which is 64-bit. Without an explicit restype
ctypes truncates it to c_int. The open then appears to succeed and every
read and write silently operates on a bogus handle, which looks exactly
like a dead cable. Every kernel32 call now declares argtypes and restype.
- Defaulting to a fixed COM number was actively harmful. A wrong-but-real
port opens without error and answers nothing, so defaulting to COM3 while
the adapter was on COM4 produced the same silent symptom. Windows now
auto-picks only when exactly one port looks like a USB virtual COM port,
and otherwise refuses and lists what it found.
- COM10 and above need the \\.\ prefix, which is harmless on COM1-9, so it
is always applied.
Adds two diagnostics, because "no response" has too many possible causes to
guess at:
ports every serial port that actually exists. On Windows this reads
SERIALCOMM from the registry, so a port absent here does not exist
now, whatever Device Manager showed last week.
doctor walks the layers - discovery, open, write, read - and reports which
one fails, including a warning when the configured port is an
onboard one while a USB adapter is present.
Port selection is -d, then $E10_PORT, then the safe auto-pick. Friendlier
errors on both platforms for a missing port or a permissions problem.
Groundwork for replacing a Telesis pinstamp with a SIC Marking e10 while
keeping the existing eDNC/MarkTelesis operator workflow. MarkTelesis has a
hardcoded marker-type enum and eDNC is serial-only, so the plan is a
translator that impersonates a Telesis controller on a COM port. These are
the pieces that translator will be built from, all verified against a real
e10 (a P63, firmware "E10 P63 (20140924)").
scripts/e10-client.py
e10-V6 protocol client. Marking path and file transfer both use the BINARY
protocol: the text protocol uses space as its delimiter, so it cannot carry
a field cleared to a single space nor the space runs in real marked text,
and it has no checksum. Includes fault decoding against the manual's fig 1,
and a guard that refuses to overflow a variable's declared buffer.
scripts/make-e10-pattern.py
Builds marking files with DECLARED variables. The controller's own protocol
cannot create one: NEWFILE/INSERTTEXTLINE/SAVEFILE writes the reference but
leaves the variable count at zero, and SETVAR then fails. Template-based,
copying through byte-for-byte everything the format notes do not explain.
scripts/simulate-ednc.py
Parses a part program, substitutes tokens and emits the byte-exact frames
MarkTelesis would send, then translates them for the e10 and can drive one.
The wire format came from disassembling MarkTelesis.exe rather than from a
capture: frame builder at 0x401cf0, and the type letters at their call
sites. Note there is no station ID, unlike the Backup470 dialect.
scripts/fake-telesis.py
Answers as a Telesis controller so MarkTelesis will hold a conversation
long enough to be recorded, with switches to force NAK and timeout paths.
reference/e10/
52-file snapshot off the bench controller, used as the corpus for reversing
the MARK_V5.0 format and for testing without hardware. ConfigCtrl is
excluded and gitignored - it stores the supervisor password in cleartext.
GE part programs are excluded for the same reason.
Three hardware findings worth knowing:
- The controller TRUNCATES SILENTLY to a variable's declared buffer size
while SETVAR still answers ok, and GETVAR echoes the full value back so
it cannot detect this. Guarded client-side against the size in the file.
- It refuses to mark when a referenced variable is unset, which makes an
unset field fail loudly rather than marking the previous part's data.
- Setting variables does not modify the stored file, so backups will not
churn on every marked part.
The shortcut fix from e844ff3 has been live in git and absent from the floor
since 2026-08-06. site-config.json exists TWICE on the share, from one repo
source: enrollment/shopfloor-setup/ and enrollment/config/. Only the second one
is staged to a bay - startnet copies Y:\config\site-config.json to
W:\Enrollment\site-config.json - and it was the stale one.
So every bay imaged in the last two weeks came up without the Plant Apps startup
item and without the Defect_Tracker taskbar pin, while the drift report showed
site-config.json in sync, because it was reading the copy nothing consumes. A
green check on the wrong file is worse than no check.
Deployed the repo copy over it (backup on the server at
~/backups/site-config.json.bak-20260819) and marked BOTH destinations git-owned,
so neither can go stale behind the other.
The shopfloor unattend is reconciled the other way round. Live was 87 lines ahead
of the repo - the default-user startup-delay removal, the Windows Update
disables, the removable-media block that stops PPKG auto-detection at OOBE, and
the run-enrollment.ps1 path fix from C:\ to C:\Enrollment. The repo copy was a
201-line fossil. LIVE WINS: these files boot machines, and pushing the repo copy
over them is exactly the 2026-08-06 outage that prompted this tool. Adopted live
into the repo verbatim (lint clean) rather than merging by hand.
Also fixed a pair that could never pass: the engineer unattend was compared
against playbook/FlatUnattendW10.xml, which is the STANDARD answer file, so it
reported DIFFERS permanently. 6f86c81 added FlatUnattendW10-engineer.xml but did
not repoint the pair at it. A permanently red row is one nobody reads, which is
how the site-config gap stayed invisible next to it.
All twelve pairs are now git-owned and in sync, and the gate has nothing left
classified as "known bad" to hide behind.
Dell and GE manifests spell the same folder inconsistently - OptiPlex vs
Optiplex. Windows does not care, but the Samba share sits on a case-sensitive
filesystem, so a blind mkdir -p created a SECOND tree and the drivers split
between them.
That is exactly how the OptiPlex Micro 7020 pack went missing: the manifest asked
for OptiPlex/D13MLK while the 3.2 GB zip sat in Optiplex/D13MLK. PESetup found no
pack, logged a warning rather than an error, and the bay imaged with no network
drivers - so DNS failed at first boot and bulk enrollment could not reach the
CDN. Symptoms three steps from the cause.
mkdir_ci walks the path one component at a time and reuses whatever is already
there whatever its case, creating only genuinely new components. Callers must use
the RETURNED path, since it may differ in case from the requested one. Falls back
to plain mkdir -p rather than skipping a download if the resolve fails.
Both spellings still exist in the live catalogues; scripts/lint-driver-catalogue.py
reports a case mismatch as an error so the pair cannot silently drift again.
OWNERSHIP.md
Every expensive bug in this pipeline has had one shape: two systems setting the
same thing, last writer winning silently. Four happened on 2026-08-06 alone -
computer name (package vs run-enrollment), drive letters (PESetup vs a volume
finder), enrollment (package vs an at-logon -ManualFallback task that syspreped
finished machines), kiosk URLs (GE-Enforce vs site-config). Each is written down
with the evidence so the next person deletes a writer instead of adding one.
share-drift.py
The share is production and the repo is meant to describe it, but drift runs both
ways: live hand-edits nobody committed, and repo fixes never deployed. The
unattend outage lived only on the share while the repo copy was fine, and nothing
compared them.
Each mapped pair is classified. git-owned means the repo wins and the pair must
match - those fail the run. unreconciled means the two have genuinely diverged
and nobody has decided; reported, not failed. The unattends are unreconciled on
purpose: live is ~17 KB against ~12 KB in the repo, so a blind push would regress
production. Reads over SSH via base64 so BOM and CRLF survive the hop.
First run: 8 git-owned pairs all match, 4 known-unreconciled.
Run-ShopfloorSetup.ps1
Corrects a comment that was actively misleading. It claimed shopfloor PCs are
"vanilla by design" and that the orchestrator runs -ManualFallback to skip BPRT
injection and the package entirely. Shopfloor bays DO enrol - the SFLD package
joins Entra with its BPRT token and a human assigns the device category in
Intune. -ManualFallback runs sysprep /oobe /reboot, which is why wiring it to an
at-logon task destroyed the deployment chain.
The absent Entra wait is still correct, for a different reason: at that point the
bay is on the isolated PXE LAN with no route to Entra (579C144 held 172.16.9.81
and 172.24.19.142, neither in the production ranges). sync_intune retries until
the tech re-cables. "Entra ID Joined: false" right after imaging is normal.
TWO SCRIPTS, DIFFERENT AUDIENCES
playbook/scripts/preflight.ps1 runs at the bay, called by startnet once the
media is mapped. It checks the four things that come straight out of PESetup's
own behaviour:
secure boot GatherDataSelection fails outright when SecurebootEnabled != 1
disk >= 120GB MinRequiredSpaceWithoutCompression is 128849018880
driver match reimplements GetDriverByModel - family filter, untrimmed
comma-separated substring tokens, first match wins - and checks
the pack is actually on the media
media age media expires 30 days after build; Media.tag's timestamp is the
local proxy
The driver check is the one that earns it. A miss is only a WARNING to PESetup,
so the bay images with no NIC, DNS fails at first boot, and enrollment cannot
reach the CDN - a symptom three steps removed from the cause. Advisory by
design: it reports and pauses on a blocker, the tech decides. Lives on the
enrollment share so it can be fixed without rebuilding boot.wim.
scripts/preflight.py runs on the server before a build day and aggregates
everything already built - driver catalogue lint, unattend lint, per-PCTYPE
media view verify - plus a new advisory firmware-coverage check that lists
catalogued models with no BIOS models.txt entry. That last one is how the
OptiPlex 7020 family sat uncovered: 127 catalogued models, 58 covered today.
First run: driver catalogues clean, all three unattends clean, firmware coverage
advisory only. PREFLIGHT PASSED.
Verified: both scripts parse clean (PowerShell parser / python), startnet parens
balance, every goto resolves, 915 CRLF lines with no bare LF. Deployed -
boot.wim md5 99fd3132, preflight.ps1 on the share.
Well-formed XML is not enough. Windows Setup validates against a schema and one
bad value invalidates the WHOLE answer file for its pass, so the machine stops at
a dialog with nothing configured and the only clue is an XPath buried in
C:\Windows\Panther\setupact.log.
On 2026-08-06 every shopfloor, standard and engineer build was failing exactly
that way:
/settings/RunSynchronous/RunSynchronousCommand/[Order="16"]/Path
Description = Value is invalid. hrResult = 0x80220005 pass = specialize
An inlined "powershell.exe -Command ..." had grown to 676 characters in a field
capped at 259, and had been broken since the previous evening. Every check here
is mechanical and would have caught it before a bay was booted:
path-too-long RunSynchronousCommand/Path > 259
cmdline-too-long SynchronousCommand/CommandLine > 1024
description-too-long Description > 256
duplicate-element a once-only element appearing twice
not-well-formed XML does not parse
bom UTF-8 BOM (the live files have none, and python's
utf-8-sig silently ADDS one when writing them back)
unknown-token %token% PESetup will not substitute; %WINDIR% and the
other shell variables are excluded or it cries wolf
Reads over SSH via base64 so a BOM or CRLF survives the hop unchanged. Exits
non-zero on any ERROR so it can gate a deploy.
Immediately found a second live defect the manual review had missed: a 329-char
Description on gea-engineer Order 11, which would have failed the oobeSystem pass
once specialize started passing. All three live image types and both repo copies
now lint clean.
CopyPackages copies the whole of <media>\Deploy\Applications to the target,
recursively, with no manifest and no filtering, so every bay built from the
shared gea-shopfloor media receives every shopfloor payload whatever its type.
The obvious fix - repoint a path inside the shared media as each bay picks its
type - races. Bays image concurrently and CopyPackages is fail-fast, so moving a
path while another bay is mid-copy either hands it the wrong payload or fails its
imaging outright, intermittently and unreproducibly.
Give each type its own media directory instead, built once and never mutated.
Everything in it is a symlink, so a type costs inodes rather than gigabytes and
the shared media stays untouched while bays read it. Samba already serves this
pattern: follow symlinks and wide links are on, and every image root reaches
Sources, Operating Systems, Out-of-box Drivers and Packages the same way.
Types come from the enrollment share's menu.json, the file the WinPE picker
already renders, so the media set cannot drift from the boot menu. Deliberately
NOT mirrored: the FlatUnattendW10.xml.pre-* backups and stray logs beside the
real unattend - the view is the clean set PESetup actually reads. Each type is
built into a scratch dir and swapped, so a bay mapping the path mid-refresh sees
the old tree or the new one, never a half-built one.
Applied on 172.16.9.1: 10 types, 190 links, all 60 critical paths resolve. An
SMB client sees ordinary directories and files - Sources, Control,
FlatUnattendW10.xml at 18078 bytes, HardwareDriver.json at 46113 - which is the
behaviour that actually matters, since PESetup reads this over SMB from WinPE.
Note for anyone extending this: passing the server-side script on stdin to
"echo pxe | sudo -S bash -s" is a silent no-op. Bash inherits the password pipe,
reads EOF, runs nothing and exits 0. It travels as a heredoc for that reason.
A driver miss is only a warning: GetDriverByModel returns null, PESetup logs
"driver for [MODEL] not found" and images the machine anyway. The bay comes up
with no NIC and no WiFi, DNS fails at first boot, and bulk enrollment cannot
reach the CDN - symptoms far enough from the cause that the OptiPlex Micro 7020
pack sat missing and the Display MicroPC failures were blamed on a drive letter.
Reimplements the matcher from the decompiled source (docs/PESETUP-INTERNALS.md)
and reports what silently breaks it:
virtual-platform one such entry flips the tool into virtual-only mode and
hard-fails every physical machine
empty-token a trailing comma yields "", and Contains("") is true for
every model, so that entry swallows the catalogue
token-whitespace Split(',') does not trim, so " OptiPlex 3010" needs the
space present in the model string too
case-mismatch the share is case-sensitive; Optiplex vs OptiPlex splits the
tree and the pack is never found
missing-zip referenced pack absent
family-mismatch the family filter runs first, so a token whose line
contradicts the family field can never match
shadowed first match wins, so a later entry may be unreachable
duplicate-token osId is not part of the match, so a win10 pack can land on a
win11 build purely by ordering
--models resolves real WMI model strings through the same code, which is the
check that actually predicts a no-driver build. Exits non-zero on ERROR or
CRITICAL so it can gate a deploy.
Verified both ways: a synthetic catalogue carrying each defect reports all eight
and exits 1; the three live catalogues on 172.16.9.1 come back clean at 0. The
file listing needs find -L and the same anchoring as destinationDir - without
either, every zip check silently passes.
parse_bios_catalog kept the 'latest' BIOS per model with a string compare, so e.g. '1.20.1' > '1.9.0' was False and it wrongly retained the older 1.9.0. Added _ver_tuple() and compare tuples of ints so the genuinely newest firmware wins.
Single-site bay-stuck issue at WJ: GE Intune Report IP script filters
Get-NetIPAddress on StartsWith("10.") and posts everything matching
to the GE Tines webhook. Bays at WJ get the PXE LAN 10.9.100.x IP
captured and reported -> GE backend tags bays as on a non-corp 10.x
subnet -> dynamic group eligibility for SFLD policy never matches.
Other GE sites work because their PXE LANs aren't on 10.x at all.
Renumber PXE LAN to RFC1918 172.16.9.0/24 so the GE filter naturally
skips wired PXE addresses without any disable-NIC dance.
Server-side already in flight (netplan dual-bound, dnsmasq scope +
boot URL repointed, blancco preferences + grub.cfg + iPXE GetPxeScript
all sed'd to 172.16.9.1). This commit is the playbook / scripts /
docs side: 109 hits across 35 files sed'd in one shot.
After this lands + boot.wim is rebuilt + bays renumber off DHCP,
the 10.9.100.1 binding will be dropped from netplan as the final
cleanup step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related fixes:
1. Hard-coded BIOS push path was /srv/samba/enrollment/BIOS, which does
not exist on the live PXE server. Real path is the shared
/srv/samba/winpeapps/_shared/BIOS/ where check-bios.cmd lives and
playbook task pxe_server_setup.yml:485 deploys Flash64W.exe + the
per-model BIOS .exe files.
2. Generated models.txt was 2-column (ModelSubstring|BIOSFile) but
check-bios.cmd reads tokens=1,2,3 with delims=| and uses field 3
for the version compare. Without the 3rd column, the version-skip
logic never engages and every imaged PC re-flashes BIOS on every
boot. Now writes 3-column (ModelSubstring|BIOSFile|Version).
3. The script overwrote the live models.txt with only the entries it
touched in the current run. Live had 50+ entries; a single-model
run wiped the other 49. Now prints the lines and asks the operator
to merge them into playbook/shopfloor-setup/BIOS/models.txt and
re-deploy via scripts/deploy-bios.sh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/diagnostics/Capture-LockdownState.ps1 captures Windows endpoint
state at three lifecycle checkpoints so the deltas isolate which phase
delivered (or failed to deliver) each component:
- pre-category - PPKG-enrolled, no Intune category yet
- post-category - category-driven assignments arrived, pre-lockdown
- post-lockdown - kiosk + autologon + AppLocker fully landed
Bumped from the previous 2-stage (pre/post) version. Legacy 'pre'/'post'
aliases preserved.
Captures additions driven by the SFLD-DSC v2.0.2 post-mortem:
- IMECache file listing (catches missing sastoken.txt)
- DSCDeployment.log + version.txt copied from C:\pc\
- SFLD\DSC payload listing
- C:\Logs\BPRT\ runtime state (criticalChecks.json, packageInfo.json)
- C:\WCDApps\ deploy verification
- Windows\Provisioning\Diagnostics copy
- Tasks-RunHistory.csv with LastRunTime + LastTaskResult per task
- DeviceManagement-Events.csv (MDM 429s, AAD token failures)
- Provisioning-Events.csv (PPKG runtime errors)
- MDM-Certificates.csv (enrollment cert health)
scripts/diagnostics/snapshot-runbook.txt: step-by-step ops guide
covering when to fire each stage, where output lands, how to ship it
back via image-upload share, and which files to compare first when
diffing.
Shebang was \`#!/usr/bin/env python3\r\` which errored with
\`/usr/bin/env: python3\r: No such file or directory\`. Pure line-ending
fix, no content change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates per-type enforcers (CMM, Keyence, Machine, Common, Acrobat)
into one dispatcher driven by pc-type.txt + site-config and a share-side
manifest layout. Same share is now the single source of truth for routine
software updates without re-imaging.
Runtime:
common/GE-Enforce.ps1 SYSTEM scheduled task. Reads
common/manifest.json plus optional
<pcType>/manifest.json and
<pcType-subType>/manifest.json.
Dispatches each entry through the lib.
Writes _outputs/logs/<hostname>/status.json
on the share after each cycle for fleet
monitoring.
common/Register-GEEnforce.ps1 Task registration. Triggers: AtLogOn +
every 5 min (jittered per-PC from
hostname hash) + daily at 05:45,
13:45, 21:45 EST shift windows.
Unregisters legacy per-type tasks on
install so the two coexist at most for
the duration of a single enforce cycle.
common/Deploy-GEEnforce.ps1 Retrofit helper for already-imaged PCs
(admin-run; copies runtime + registers
task + optional immediate trigger).
Library (common/lib/Install-FromManifest.ps1):
- New Type values: PS1, BAT, File, Registry, INF
- New DetectionMethod values: Always, MarkerFile, ValueMatches, pnputil
- TargetHostnames filter (exact + -like wildcards, ANDed with PCTypes)
- Schema version check (logs WARN on manifest newer than lib MAJOR)
- Auto-writes MarkerFile on successful one-shot PS1/BAT/CMD runs
- MSI log scan on failure surfaces meaningful install errors
- Lib version bumped 2.0 -> 2.1 for TargetHostnames
Observability:
common/monitor-fleet-status.py Scans _outputs/logs/*/status.json for
stale check-ins, failed scopes, and
version drift. Respects scope (dir-name),
PCTypes, and TargetHostnames filters so
entries excluded from a PC do not
false-flag as drift.
Regression harness:
common/test/ Parameterized VM harness + README
covering every action type plus
rollback, bad/missing SFLD creds, and
schema versioning.
Imaging integration:
Run-ShopfloorSetup.ps1 now stages GE-Enforce.ps1 and lib to
C:\Program Files\GE\Shopfloor\ and invokes Register-GEEnforce.ps1
at the end of setup. Legacy Register-CommonEnforce invocation is
kept for the transition; it and the legacy per-type enforcer files
are dead code once Register-GEEnforce runs and will be removed in a
dedicated cleanup pass.
Standard-Machine manifest:
eDNC entry bumped 6.4.3 -> 6.4.5. DetectionValue pinned to the
4-part FileVersion 6.4.5.0 verified against a fresh install in the
Win11 analyzer VM. UDC DetectionValue pinned to 1.0.34 (registry
stores 3-part for UDC; verified live).
scripts/mirror-from-gold.sh:
Restructured around share-root rsyncs (one pass per Samba share)
to close gaps in the prior per-subdir layout: winpeapps/_shared/
Applications (7.5 GB of Adobe + fonts + Java + Office + OpenText
+ printdrivers + wireless + Zscaler), additional winpeapps image
types, and enrollment flat-layout root files. Adds
--skip-clonezilla and --skip-reports.
Verified end-to-end in the Win11 analyzer VM:
- Every action Type and DetectionMethod round-tripped
- PCTypes filter (Oracle excluded on Shopfloor, Firefox included
on Shopfloor and DESKTOP-*, excluded elsewhere)
- TargetHostnames filter (exact, wildcard, no-match)
- Upgrade path: XML hash bump + fleet re-copy
- Rollback path: history-archive restore propagates via enforcer,
fleet converges back without per-PC intervention
- Status writeback + monitor script drift detection
- Graceful degradation on bad creds, missing creds, share
unreachable (all exit 0, log clearly, retry next cycle)
Not in this commit (follow-ups):
- Retire legacy per-type *-Enforce.ps1 files and simplify
09-Setup-*.ps1 scripts (coordinated multi-file cleanup)
- Stage 2b: InUseCheck close-and-reopen, ApplyMode gating,
UpdateWindow, .apply-now.txt sentinel, BITS pre-staging,
1618 mutex retry, PostInstallCheck, Uninstall action
- Management app (manifest CRUD + deploy + rollback + fleet view)
- ShopFloor autologon persistence bug (deferred for next imaging
attempt with live registry evidence)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- playbook/startnet.cmd + startnet-template.cmd: after preinstall staging,
xcopy Y:\pre-install\udc-backups to W:\PreInstall\udc-backups so UDC
settings JSONs are available during image deployment. Harvested from
live gold where this block existed but was never committed.
- scripts/mirror-from-gold.sh: update source paths to current taxonomy
layout (pre-install/, installers-post/, blancco/, config/) and add
ppkgs/, scripts/, shopfloor-setup/ sections. Added --delete for exact
mirror semantics. Used to seed the spare PXE server at 10.9.100.2 on
2026-04-16 from gold at 10.9.100.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new docs cover what the system is (boot chain, services, shares,
enrollment layout, data flow) and what to change per site (every
hardcoded value, where it lives, secrets handling).
scripts/mirror-from-gold.sh replicates content from an existing PXE
server (Operating Systems, drivers, packages, custom installers, BIOS,
PCDMIS, Blancco custom image, site-config) onto a freshly-installed PXE
server. Translates the legacy flat enrollment layout on the source into
the reorganized taxonomy (ppkgs/, pre-install/installers/, installers-
post/cmm/, blancco/, config/) on the destination. Tolerates rsync
exit 23 (permission-denied subdirs like the OpenText W10shortcuts dir
that is pxe-upload-group-only on legacy servers).
Workstation reorganization:
- All build/deploy/helper scripts moved into scripts/ (paths updated to use
REPO_ROOT instead of SCRIPT_DIR so they resolve sibling dirs from the new
depth)
- New config/ directory placeholder for site-specific overrides
- Removed stale: mok-keys/, test-vm.sh, test-lab.sh, setup-guide-original.txt,
unattend/ (duplicate of moved playbook/FlatUnattendW10.xml)
- README.md and SETUP.md structure listings updated, dead "Testing with KVM"
section removed
- .claude/ gitignored
Enrollment share internal taxonomy (forward-looking; existing servers
unaffected since they keep their current boot.wim with flat paths):
- Single SMB share kept (WinPE only mounts one Y: drive), but content now
organised into ppkgs/, scripts/, config/, shopfloor-setup/, pre-install/{bios,
installers}, installers-post/cmm/, blancco/, logs/
- README.md deployed to share root explaining each subdir
- New playbook tasks deploy site-config.json + wait-for-internet.ps1 +
migrate-to-wifi.ps1 explicitly (were ad-hoc on legacy servers)
- BIOS subdir moved into pre-install/bios/, preinstall/ renamed to pre-install/
- startnet.cmd + startnet-template.cmd updated with new Y:\subdir\ paths
- Bumped GCCH PPKG references v4.9 -> v4.10
Blancco USB-build fixes (so next fresh USB install boots Blancco end-to-end
without the manual fixup we did against GOLD):
- grub-blancco.cfg: kernel/initrd switched HTTP -> TFTP (GRUB's HTTP module
times out on multi-MB files); added modprobe.blacklist=iwlwifi,iwlmvm,btusb
(WiFi drivers hang udev on Intel business PCs)
- grubx64.efi rebuilt from updated cfg
- Playbook task added to create /srv/tftp/blancco/ symlinks pointing at the
HTTP-served binaries
run-enrollment.ps1: OOBEComplete is now set AFTER PPKG install (Win11 22H2+
hangs indefinitely if OOBEComplete is set before the bulk-enrollment PPKG runs).
Also includes deploy-bios.sh / pull-bios.sh / busybox-static / models.txt
that were sitting untracked at the repo root.
