Everything West Jefferson specific moved out to a separate repo
(pxe-site-wj): NTLARS/UDC/heat-treat backups keyed by WJ machine numbers,
CMM and Wax/Trace bay lists, WJ OpenText profiles, site-config.json, the
WJ preinstall list, and the WJ shopfloor PC types (Collections,
NoCollections, Common, Keyence, CMM, Genspect, HeatTreat, Wax/Trace,
Display, PartMarker) with GE-Enforce, machine numbers and the desktop
layout. The part-marker bench tools moved to their own repo. Dead
scripts removed: build-proxmox-iso, deploy-bios, sync_hardware_models,
mirror-from-gold, pull-bios, startnet-template.cmd.
Shopfloor is one generic type, gea-shopfloor-standard, with today's
timing:
- PPKG at first logon (unattend Order 10, unchanged).
- Install-AppList.ps1 runs C:\PreInstall\preinstall.json during setup.
- A new SYSTEM task, GE Shopfloor Post-PPKG, runs postppkg.json once
after the post-PPKG reboot. It also clears AutoLogonCount after the
ShopFloor handover (the Users-level monitor can be denied that HKLM
write) and removes the Intune Sync task when done.
- Both lists ship empty. A site adds entries and installers.
- WinPE choice 3 goes straight to the PPKG menu. Staging, fetch and heal
carry only the generic payload. The webapp's shopfloor sub-menu editor
is gone, and the imaging tiles name the new stages.
- The playbook deploys the empty lists only if missing. It now deploys
the PowerShell 7 MSI the shopfloor unattend installs, which it never
did before.
Blancco site settings move into the webapp (Updates > Blancco releases):
business location, licensing Wi-Fi SSID and password. They are rendered
into the served preferences.xml and applied to every staged release, and
are seeded from site-vars (site_name, blancco_wifi_ssid,
blancco_wifi_password) only while none exist. The repo XML carries no
site values; the plaintext Wi-Fi password is gone from it.
New site_name and site_timezone settings: report and evidence-pack site
name, server clock and webapp time zone. No more West Jefferson defaults
in webapp/config.py or the dashboard title. Sync-BPRT.ps1 gets the
server address filled in at download instead of WJ's GE address.
USB on Windows:
- scripts/Build-PxeUsb.ps1 runs the same checks as build-usb.sh. It
writes one FAT32 CIDATA partition (ISO files plus patched grub.cfg plus
payload) with diskpart and reads every file back by SHA-256.
- Tested with Windows PowerShell 5.1 in the win11 VM, and the layout
installs and first-boots in the pxetest VM.
- scripts/stage-usb-share.sh stages the build folder on a share.
- site-content/ is optional for both builders.
KB docs/kb/How_To_Deploy_PXE_Server_New_Site.html: new-site deployment,
Windows only, with a site-vars.yml guide.
Checks: webapp tests pass (5 suites); the unattend lint and playbook
syntax checks pass; every .ps1 parses; there are no WJ hosts, IPs, names
or PC types in playbook/, webapp/, scripts/ or autoinstall/.
Sites no longer need either MAC in site-vars.
pxe_nic_mac empty: before anything changes, the playbook runs
pxe-pick-nic --auto. It takes the one wired port with a link, accepts it
only if pxe-nic-check calls it isolated (no LLDP/CDP/802.1X, no other
DHCP server, at most 10 hosts heard in 65 s), and pins it. The choice is
saved in /etc/pxe-server/nic-pick.conf and never re-guessed at boot.
When it cannot tell (no link, two links, not isolated) nothing changes,
and the console banner says why and to run sudo pxe-pick-nic: pull the
imaging cable and plug it back in, and the port that blinked is the
imaging one. Both helpers are installed before the pick, so that works
after a refused first boot. The site's corp domains and allowed
switches are passed to the check.
corp_nic_mac empty: every other wired port (en*, eth*) is a catch-all
DHCP client, so GE can be plugged in at install or later.
- dnsmasq stays pinned with interface=pxe0. A new check refuses any
dhcp-range outside the imaging subnet, and proxy ranges.
- The catch-all ports ignore offers from 172.16.9.1.
- UFW allows SSH and 9009 by source on any interface, and keeps those
rules when the old any-interface rules are removed.
- pxe_nic_name must not look like en*/eth* in this mode.
pxe-netplan-apply.sh takes 3 or 5 args. Without a pinned GE NIC it:
- renames stray names (the old pxe0 after a re-pick, a leftover corp0)
to enx<mac>, so the catch-all takes them, and restores them on revert;
- checks that every NIC that had a DHCP address before has one after,
tracked by MAC.
With both MACs set (the WJ server), netplan, the dnsmasq pin and UFW
render byte-for-byte as before.
Tested in the pxetest VMs:
- Fresh install with no MACs and the GE link down: the imaging port was
picked and pinned as pxe0 with 172.16.9.1, and dnsmasq bound DHCP to
pxe0 only. GE plugged in afterwards got a lease, SSH worked, and the
guard reported pxe0 isolated.
- --auto refuses with two links.
- The interactive pick works by pulling and replugging the link.
- A 3-arg re-apply is UNCHANGED.
Unit tests (fake sysfs) cover the re-pick and pinned-to-catch-all renames.
A freshly installed server could not build standard or engineer: about
8 GB they install from existed only on the running server.
scripts/capture-site-content.sh pulls it over SSH into the gitignored
payload:
- _shared/Applications/extra (Office, Adobe, OpenText, Java, fonts, Wi-Fi,
Zscaler), without pxe-status/ (the playbook deploys that);
- each image's own Deploy/Applications files and folders (GE CA
certificates, the Engineering Layer payload), without the repo-managed
Install-EngineeringLayer.ps1 and without stale BPRT copies;
- the links images keep there, as links.txt;
- fresh bios-staging (*.exe plus models.txt, which the Dell update
rewrites), boot.wim, and the known Blancco file set.
It dry-runs by default, compares by checksum (a Blancco config.img had the
same size and time as the stale copy), writes SHA256SUMS, and refuses what
FAT32 cannot hold: files of 4 GiB or more, Windows-illegal names, and
case-only name clashes.
build-usb.sh copies site-content/ unsplit and refuses to build without it
(unless --skip-site-content), or when SHA256SUMS is missing or does not
cover every file. The playbook checks SHA256SUMS, then merges the content
into winpeapps and recreates the links. It only links to folders that
exist, so it never creates an empty _shared BPRT, which would stop the
first MCL import from seeding it. Link lines are validated before anything
is copied.
pull-bios.sh is gone. It read a path the server no longer uses, with a
hard-coded password.
An MCL import merged Deploy/Applications with rsync -a, which replaced each
image's BPRT symlink with a plain copy of the vendor folder. That copy has
no site overlay, so standard/engineer enrollment lost
Register-BulkEnrollTask-Standard.ps1 (specialize Order 39).
Now Deploy/Applications is imported first and without BPRT:
- where _shared/Applications/BPRT exists it is left alone (the BPRT page
keeps it current) and the image link is put back;
- on a box with none yet, the imported BPRT seeds it: vendor
Select-PCConfig.ps1 kept as -vendor, our overlay added, BOM fix applied;
- the import refuses before anything moves when that folder would not work
(vendor scripts missing, or no overlay copy to add).
The two Register scripts join Select-PCConfig.ps1 in the repo. The playbook
already syncs shopfloor-setup to the enrollment share, so
config.BPRT_OVERLAY_DIR finds all three there; overlay_sources now falls
back to it for every overlay file, not only the shim.
Engineer builds left the employee without "Configure my Engineering
Layer". First logon ran C:\Deploy\Applications\extra_engineering\
Install-EngineeringLayer.ps1, and the script itself lived in C:\Deploy:
std/eng have no autologon, so a PC that rebooted before the employee
signed in had lost C:\Deploy to the orchestrator's DeleteDeployDir task,
and nothing installed, silently.
- engineer unattend, specialize Order 37 (free slot): run the script with
-StageOnly, copying the payload and the script to C:\PreInstall\engineering
while C:\Deploy exists (Path 119 chars)
- first logon Order 11: run it from C:\PreInstall\engineering, falling back
to C:\Deploy if staging did not happen
- the script (now in the repo, playbook/engineering/; it lived only on the
server): after the install, restore the Public desktop icon if it is
missing and the PC is not configured yet (no C:\TMP\DUU_Complete.log), and
register Active Setup so each later user gets the icon on their own
desktop when the Public one is gone. v3.1 puts the icon on the Public
desktop; clicking it (configure, from GE's V: share) deletes it for all
- playbook deploys the script to gea-engineer extra_engineering; the 650 MB
payload stays site content (documented in SITE-CUSTOMIZATION.md)
Tested in the win11 VM with the exact unattend strings and a stand-in
installer that skips the icon: specialize staged payload and script;
C:\Deploy deleted; first logon installed from the staged copy, restored the
icon and registered Active Setup. The per-user step as a new user: icon on
their desktop when the PC is unconfigured and the Public icon is gone; none
when configured or when the Public icon is there. Lint clean. Not deployed:
unattends go out only through scripts/deploy-unattend.sh.
Everything after OOBE needs the employee to log in, which happens off the
PXE network, so the floor waited forever at step 3 of 5. Now 3 steps:
WinPE, Windows setup installing apps, ready at OOBE (succeeded).
- both unattends: specialize Order 49 posts step 3 as succeeded
(Path 182 chars; lint clean). Only that line changed.
- Send-ImagingStage.ps1: StageTotal 3; steps above the total (the old
first-logon steps 4 and 5 still in the unattends) are logged and not
posted, so they cannot reopen a finished build
- winpe-status-push.ps1: total 3 for gea-standard / gea-engineer
- floor tile step 3: "Ready at OOBE"
Tested in the win11 VM against the test server: steps 2 and 3 post,
step 3 shows succeeded, step 4 is skipped. Not deployed: the unattends
go out only through scripts/deploy-unattend.sh.
- Deploy shopfloor-setup with rsync instead of the copy module: the
~4 GB tree took about 4 minutes on every re-run because copy checksums
each file. rsync transfers only changes; changed only when it did.
- iPXE drops a trailing space, so 'Machine number:' ran into the typed
digits. The space now comes from a hex 20 setting.
- tests/vm: pxetest-xp is a permanent client (SeaBIOS, qemu32 32-bit CPU,
rtl8139, 1 GiB, IDE disk) for the i686 Clonezilla path; start.sh xp.
Only one client may run at a time across all three.
Boot menu: "Clonezilla: back up this PC" and "restore this PC" ask for the
machine number and boot Clonezilla (64-bit with Debian shim, or 32-bit
i686 by cpuid) with ocs_repository=smb3://clonezilla:<pw>@server/clonezilla/,
ocs_prerun fetching clonezilla-scripts/pxe-cz.sh and ocs_live_run running
it. The script validates the machine number, saves
<machine>-<YYYY-MM-DD-HHMM> (server time), writes pxe-meta.txt, runs
ocs-chkimg, and restores only that machine's images with -k1 -r after a
typed YES. The old entry is "Clonezilla: manual".
Playbook: clonezilla Samba user for the [clonezilla] share only, password
from clonezilla_smb_password (validated, filled into the menu
placeholder), scripts installed to the web root.
Webapp: Clonezilla backups page lists image folders and zips grouped by
machine number with serial/model/version/check, serial mismatch flag,
delete, move to another machine, streamed tar download, disk space bar
and an explicit keep-last-N retention with preview. Tests and a VM test
plan added.
Current Clonezilla is amd64-only. The menu now checks the CPU with iPXE's
cpuid --ext 29 (long mode): 64-bit CPUs get the current release (through
Debian's shim under Secure Boot), 32-bit-only CPUs (Pentium 4 without
EM64T, Pentium M, early Atom) get Clonezilla 3.2.0-5 i686, the last
32-bit release, from clonezilla-i686/. Without a cpuid command the 32-bit
build is used, which also runs on 64-bit CPUs; UEFI always gets the
64-bit build. A 'Clonezilla Live 32-bit' item forces it.
prepare-boot-tools.sh downloads it, the playbook deploys it.
Clonezilla Live runs a Debian-signed kernel whose modules (usb-storage,
uas, exfat) are signed by the same build key, so the menu now starts it
through Debian's Microsoft-signed shim with the iPXE shim command, the
same way Blancco uses Ubuntu's. prepare-boot-tools.sh extracts that shim
from the Clonezilla release (EFI/boot/bootx64.efi -> clonezilla/
shimx64.efi) and warns when the shim is not Microsoft-signed or the
kernel not Debian-signed.
Memtest86+ is unsigned and needed Secure Boot off; memory tests now use
the Dell built-in diagnostics (F12 at power-on, Diagnostics), noted in
the menu. The playbook removes the old memtest web dir. Nothing on the
menu needs Secure Boot off any more; the 'disable Secure Boot' screen
remains only as the fallback when a signed path fails.
H1 corp_admin_cidrs (UFW 22/9009 on the corp NIC from these only; refuses
when it would lock out the current SSH client) and corp_ssh_password_auth
(sshd drop-in, passwords only from the imaging subnet). Defaults keep the
current behaviour and print a loud warning.
H2 pxe-netplan-apply.sh: pre-flight refusal (pxe-MAC NIC holds the default
route, a DHCP lease or the route to an SSH client), run from the USB
before anything changes; post-apply checks (names, imaging IPv4, corp
IPv4 back within 30 s, SSH route) and a 120 s systemd-run auto-revert.
H3 restore() puts the previous NIC names back (two-phase rename via
temporary names, so swaps work) before generate + apply.
H4 a failed rename is fatal (restore + exit 1); the play drops the old
any-interface UFW rules only after verifying both NICs and their
per-interface SSH rules.
H5 first-boot checks the playbook exit code, retries at next boot (max 5),
writes /var/lib/pxe-firstboot.failed and /etc/issue + /etc/motd banners.
M1 build-usb.sh verifies iPXE files against the playbook pins, every wheel
against its own package's hashes, drops the unpinned pip fallback,
requires --site-vars or --no-site-vars.
M2 webapp restarts when the code, wheels or unit changed.
M3 ignore-carrier on the imaging NIC; ip_nonlocal_bind kept for the
apache/networkd boot race.
M4 menu suffix must match blancco_kver; missing kernel with no earlier
build is fatal; a live menu with the staged Blancco test entry is not
overwritten.
M5 build-proxmox-iso.sh is unsupported (hard exit); use build-usb --image.
M6 deploy-unattend.sh: password via SSHPASS env and sudo stdin only,
%q-quoted remote args, sha256 check of the live file; lint-unattend.py
reads PXE_SERVER/PXE_USER/PXE_PASS and uses sshpass -e.
M7 the playbook lints the unattend files before the force: yes copies
(lint-unattend.py ships on CIDATA as scripts/); pre-commit ACMR.
M8 Blancco initramfs build: no ignore_errors, pipefail, trap cleanup.
Also: Send-ImagingStage.ps1 deploy, carrier hook install tasks
(pxe-nic-carrier, networkd-dispatcher), L1 imaging-active hold
(allow_disruptive) and no iPXE downgrade below an Updates-page install,
L2 iso-volid, L3 kver pin read from the playbook, L4 cp -rL, L5 refuse
>= 4 GiB WinPE files, L6 PIP_DIR wipe guard, L7 per-package wheel hashes,
no online fallback, L8 check-updates.py (apt_pkg/cached compare, all
components, exit 2 on no debs / bad answers).
- pxe-nic-check: count 802.1X only from an authenticator (EAPOL type 0
with EAP Request/Success/Failure). iPXE sends EAPOL-Start on every NIC
open and Windows sends Start/Response, so a PXE-booting bay made the
verdict 'ge' and the watch stopped dnsmasq for the floor.
- pxe-nic-check: join the STP/PAE/LLDP/CDP/PVST+ multicast groups on
every capture socket, so CDP-only ports are heard on a plain NIC.
- pxe-nic-check --watch: if dnsmasq was (re)started inside the window,
re-check (~6 s) before stopping it; skip when already stopped.
New --state-cmd and --print-imaging.
- pxe-nic-carrier (+ pxe-nic-carrier.service): networkd-dispatcher hook
for the imaging NIC only. Carrier loss stops dnsmasq; carrier gain
restarts it through its guard if it was running or the hook stopped it.
- pxe-dhcp-hook.sh: clear only SMB (tcp 445/139) conntrack entries and
sockets, not all traffic for the client, which raced iPXE's first
HTTP fetch.
- GetPxeScript.aspx: every boot line has a failure branch (iPXE 2.0
accepts an unsigned EFI kernel and fails at boot under Secure Boot,
which aborted the script).
- Tests: client EAPOL, packet_mreq packing, watch race, shell helpers on
stub PATH, lab scenarios eapol-client, br-eapol-client, race, carrier.
- webapp/requirements.txt pins all eight wheels with sha256, at the fixed
releases (lxml 6.1.3 for GHSA-vfmq-68hx-4jfw / PYSEC-2026-87, werkzeug
3.1.9 for GHSA-g6x2-hccm-hh4m).
- download-packages.sh fetches exactly that set for Python 3.12 / x86_64
into an emptied folder; PIP_DIR is honoured inside the container (own
mount), files are handed back to the calling user, USE_DOCKER=1 forces
the container on a noble host, an unknown package name now prints apt's
error instead of failing silently, and linux-modules-extra-7.0.* is gone
(it does not exist; 7.0 ships those modules in linux-modules).
- Playbook: the wheel installer refuses any wheel whose sha256 is not
pinned, needs exactly one wheel per pin, and removes an older installed
version by its RECORD first. The webapp copy is a full rsync every run
(was cp with creates: app.py, which never updated an install).
- scripts/check-updates.py: offline debs against noble security/updates
and the pins against PyPI + osv.dev; exit 1 when something needs
updating. Run it before building a USB.
Tested: installer in a Python 3.12 venv (clean install, upgrade over lxml
6.0.4 + werkzeug 3.1.8 leaves no old dist-info, re-run is a no-op, a stray
wheel is refused); check-updates.py flags the April bundle (94 security
updates) and passes the pins.
Every server has an imaging NIC (static, the only one that serves
DHCP/TFTP/HTTP boot/SMB) and a GE corporate NIC (DHCP client, SSH and the
webapp on 9009). The playbook guessed the PXE NIC and the installer put
172.16.9.1/24 on every en* NIC, so a two-NIC box could answer on the GE
network.
- New per-site vars pxe_nic_mac / corp_nic_mac (required, from
playbook/site-vars.yml or -e; template site-vars.example.yml), stable
names pxe_nic_name / corp_nic_name (default pxe0 / corp0, overridable),
pxe_server_ip, pxe_subnet, DHCP range. pxe_imaging_mac / pxe_ge_mac are
derived from them. The play stops before touching anything if a MAC is
unset, malformed, duplicated or not on the box (lists the box's NICs).
- Heuristic NIC detection and the single-NIC netplan task replaced by a
two-NIC netplan template (match macaddress + set-name), installed by
pxe-netplan-apply.sh: validate in a scratch root, back up, generate,
rename, apply, restore the previous config on any failure. cloud-init
network config disabled so it cannot rewrite it.
- dnsmasq: /etc/dnsmasq.d/pxe-nic-only.conf from the vars (interface=,
except-interface=, no-dhcp-interface=); dnsmasq.conf IP/range from vars.
- UFW per interface: imaging NIC 22, 67/udp, 69/udp, 80, 4433, 445, 139,
137/udp, 138/udp, 9009; GE NIC 22 and 9009 only. Old any-interface rules
and the stale 9011 rule removed after the new ones are in.
- sysctl drop-in: IPv4/IPv6 forwarding off; ip_nonlocal_bind so Apache
can bind the imaging IP before the address is up.
- Samba [global]: interfaces = 127.0.0.0/8 <imaging NIC>, bind interfaces
only. Apache 80 and 4433 on the imaging IP only; 9009 stays on all (TODO:
the webapp has no auth).
- autoinstall: DHCP only on en*, no static 172.16.9.1; apt geoip off and
offline-install fallback now that the corp NIC may have a lease.
- Blancco UEFI/Secure Boot: blancco_kver (7.0.0-34-generic) drives the
kernel, initramfs and modules builds instead of the running kernel;
outputs vmlinuz-ubuntu-70, kexec-initrd-70.img, kmod-<kver>.tar.gz as the
menu and blancco-init.sh expect; creates: guards keyed on the full kver.
Ubuntu's shimx64.efi.signed published as blancco/shimx64.efi.
download-packages.sh adds shim-signed, linux-generic-hwe-24.04 and the
pinned linux-image/modules/modules-extra for BLANCCO_KVER.
Rendered with the live MACs and names, netplan generate output is
identical to the live config; pxe-nic-only.conf and the Samba binding
match live directive for directive.
A PC whose imaging failed and is re-imaged gets the same IP, and the
server still held state from the dead run, so net use failed (System
error 53). The April fix (18537ac) did not cover it:
- The dnsmasq hook only acted on 'add'/'del'. A re-image comes back to the
existing lease, which dnsmasq reports as 'old', and only when the
hostname/MAC changed. Now dnsmasq sets script-on-renewal, and the hook
clears state on 'old' when the request comes from PXE firmware or iPXE
(vendor class PXEClient), i.e. the machine just network-booted. A
Windows lease renewal (MSFT 5.0) is left alone.
- The 1 h conntrack timeout never applied after a reboot: systemd-sysctl
runs before ufw loads nf_conntrack, so the keys did not exist and the
5-day default stayed. /etc/modules-load.d/pxe-conntrack.conf loads it
first.
- smb-soft-reset.sh flushed neighbours on br-pxe, which no longer exists.
Live 2026-10-14. Hook tested on the server with an unused IP: network-boot
renewal clears, Windows renewal and no vendor class do not, add clears.
pxe-nic-check classifies each NIC from what it hears: LLDP/CDP/802.1X
from managed switches, foreign DHCP offers and their domain (one
DHCPDISCOVER on the imaging NIC; the GE NIC's networkd lease is read
instead), STP and ARP hosts for information. A kernel BPF filter keeps
bulk SMB traffic out of the capture loop; bridge ports are listened on
too, since LLDP and EAPOL stop at the port.
- dnsmasq drop-in runs --guard as ExecStartPre: exit 1 on a ge or
foreign-dhcp imaging verdict, fail open (loudly) on bugs or no link
- pxe-nic-check.timer runs --watch every 5 min: stops dnsmasq on a bad
imaging verdict, never starts it
- /run/pxe-nic-status.json (0644, atomic) for the webapp
- playbook installs it all and writes /etc/pxe-server/nic.conf from
pxe_imaging_mac, pxe_ge_mac, pxe_corp_domain_suffixes,
pxe_nic_allowed_switches
- unit tests plus a no-root network-namespace lab (veth pairs, a raw
socket DHCP/LLDP peer and real dnsmasq); docs/pxe-nic-guard.md has the
operator notes and the two floor tests
a94c203 added a RunSynchronous Path of 417 chars to both std/eng answer
files. Windows caps it at 259 and then rejects the WHOLE answer file, so
an Engineer build stopped at 'Windows could not parse or process unattend
answer file'. The live files were restored to their pre-a94c203 versions
the same day.
- The status steps now fit: Send-ImagingStage.ps1 -Persist copies itself
out of C:\Deploy (no cmd copy chain), short PowerShell switches, default
arguments left out. Longest Path is 168 chars; lint-unattend passes.
NOT deployed: these files must go live only through deploy-unattend.sh.
- scripts/deploy-unattend.sh: the only route for an answer file to a PXE
server. Lints first and refuses on any error, backs up, swaps
atomically, lints the live copy and restores the backup if that fails.
- .githooks/pre-commit (git config core.hooksPath .githooks): refuses a
commit whose staged unattend fails the linter. Verified by planting a
352-char Path (blocked) and against the exact a94c203 file (rejected).
- Boot menu: also accept SecureBoot read as hex 01, and show the raw values
in the warning so a wrong read can be diagnosed from the screen.
- winpe-status-push.ps1: check-bios.cmd stages the firmware update before
the imaging menu, but startnet's 'BIOS firmware update' push only comes
at the end of WinPE, so builds showed a plain 'Applying image' for the
whole apply. The first push now reports the staged update when
X:\bios-fired.flag exists, and the floor shows 'Applying image - BIOS
update staged' (do not power off during the next reboot). Also: CIM
fallback for the serial where wmic is missing, and standard/engineer
always report 5 steps.
- Network pill: cdc_ncm (e.g. the WisdPI RTL8157 5G adapter) keeps link
speed in a 32-bit bits/s field, so 5 Gb/s reads 705 Mb/s; show the real
speed with a note.
Tested in the win11 VM (with and without the BIOS flag, and the late
startnet push).
Live since 2026-10-14 for every client (all UEFI clients now run iPXE 2.0):
- WinPE skips the 'Secure Boot must be ENABLED' warning and its 5 s wait
when iPXE reads Secure Boot as on (${efi/SecureBoot:uint8}); otherwise,
or on an iPXE that cannot read it, it warns as before.
- Blancco and Blancco debug start through Ubuntu's shim (iPXE shim
command), so they run with Secure Boot on; an iPXE without the command
boots the kernel directly as before. The separate Secure Boot TEST entry
is gone.
- Blancco has its own menu section; only Clonezilla and Memtest stay under
'Secure Boot must be DISABLED'.
The playbook now installs playbook/ipxe/GetPxeScript.aspx instead of an
inline copy that had drifted from the live menu. Checked that the Blancco
Releases page still parses it (one item blancco, :blancco_efi block).
Live since 2026-10-14: dnsmasq hands every UEFI client ipxe-shim.efi
(iPXE's shim, dual-signed under Microsoft UEFI CA 2011 + 2023), which
loads ipxe.efi from the TFTP root, now the iPXE v2.0.0 snponly build
(firmware network and USB drivers; the full build kills USB keyboards).
The per-MAC test tags and folders are gone; the old Broadcom-signed
1.21.1 build stays as ipxe-bcm.efi, the rollback.
- playbook: pinned sha256 vars, the new dnsmasq default line, and a
verified, non-optional task that installs ipxe-shim.efi, ipxe.efi
(snponly), ipxe.pxe and ipxe-bcm.efi (was: copy ipxe.efi and the never
served undionly.kpxe, errors ignored).
- scripts/fetch-ipxe.sh downloads the pinned tarball and shim into
boot-files/ for the USB build; README/SETUP list the step.
- iPXE page: ipxe.efi is staged from snponly.efi, a build that drives USB
itself is refused, and the snponly-shim/snponly root files are gone.
Tested: the playbook task against the real release files (installs the
live hashes; a tampered file stops it before anything is replaced),
ansible --syntax-check, staging, and the full-build refusal.
The floor's 8 stages are the shopfloor pipeline, and only shopfloor-setup
reports after WinPE, so every gea-standard / gea-engineer build sat at
1/8 forever with no type.
- winpe-status-push.ps1 reads the image from the Z: share mapping when no
PCTYPE is set and reports those builds as 5 steps.
- Send-ImagingStage.ps1 (deployed to _shared\Applications\extra\pxe-status)
posts steps 2-5 from the unattend: start and end of specialize, first
logon, and setup complete (succeeded). It is copied to
C:\ProgramData\PxeStatus first, because the bulk-enrollment task deletes
C:\Deploy before first logon. Best-effort: always exits 0.
- Floor rows use Standard/Engineer labels and show the image as the kind.
Tested as SYSTEM in the win11 VM (PowerShell 5.1): the exact unattend
command line, stages 3-5 against a test server, a dead port (exit 0), and
the image regex.
'24250' also matched the Dell 24 All-in-One EC24250, which needs a different
BIOS (Dell_EC27250_EC24250). The three exact models are the ones the combined
file is built for. Found by the webapp's Dell check (ambiguous match).
FileVersion detection on goCMM.exe so an older 2.x does not satisfy it. Same
NSIS installer and /S switch. Deployed to installers-post/cmm and the share's
shopfloor-setup copy; 2.12.3 moved to installers-post/cmm-archive so it is not
copied to every CMM image.
Per-bay backups were taken under goCMM 1.1, so goCMM.reg restores
Installation Directory = the 1.1 folder. goCMM 2.x never reads it, but the 2.x
installer sees it as '1.1 installed' and shows a dialog /S cannot answer,
which hung GE-Enforce on CMM11 and CMM12. Kept when 1.1 really is installed.
sbsigntool, zip and curl, the Zscaler root CA in the system store (GitHub is
TLS-intercepted on the corporate NIC; downloads.dell.com is not), and
/var/lib/pxe-webapp/updates.
Scanned against Dell's per-model catalogs (CatalogIndexPC, 2026-10-09).
About 40 lines were behind, nine pointed at another platform's BIOS
(Latitude 7450 -> OptiPlex 7450, Latitude 5530/5540 -> Precision
5530/5540, Precision 5490/7770/7780 -> OptiPlex AIO, Precision 5550 ->
Latitude 5550, OptiPlex 7400/7410/7420 AIO -> Latitude 7x00/7x10/7x20),
and '7320 2-in-1 Detachable' never matched the WMI name. Specific lines
now precede general ones (5430 Rugged, 7090 Ultra, 3000 Thin Client)
because check-bios.cmd takes the first substring match.
The gea-standard/gea-engineer path (purpose -> tenant -> enroll_staged)
never mapped the enrollment share, so winpe-status-push.ps1 and
preflight.ps1 silently skipped those builds. Map it at :end if absent.
The hardware report went out from winpe-status-push.ps1 on the
enrollment share (Y:), but startnet.cmd never maps Y: on the
gea-standard / gea-engineer path (choices 1/2 go purpose -> tenant ->
enroll_staged), so those builds never reported. The same gap means
neither the status push nor preflight.ps1's driver check has ever run
for standard/engineer images.
The BIOS share is mapped by every WinPE boot before the menu, and
check-bios.cmd lives on it, so it now calls hardware-report.ps1 from
the same folder (best effort, output discarded). winpe-status-push.ps1
goes back to status only, so shopfloor bays are not counted twice. The
playbook deploys hardware-report.ps1 with the other BIOS files.
WinPE now reports each machine's WMI model, manufacturer, Dell system
SKU and BIOS version to /hardware/seen (winpe-status-push.ps1, best
effort, wmic only). The webapp records every model seen and checks it
against what the server has:
- each image's Deploy/Control/HardwareDriver.json, matched exactly as
PESetup's GetDriverByModel does (as reimplemented in
lint-driver-catalogue.py), plus whether the pack's zip is on disk;
- the WinPE BIOS manifest (substring match like check-bios.cmd).
/hardware lists every model with per-image driver status and BIOS
status. A model missing a Win11 pack or a BIOS entry becomes item 5 on
the Technician To-Do (and the banner count) until the pack/BIOS is added
or it is dismissed. Virtual machines are ignored.
Also deploys the repo winpe-status-push.ps1 over the live copy, which
predated the -CurrentStage/-StageIndex parameters startnet.cmd passes
for the BIOS-flash stage, so that status post was failing.
Webapp: config.DISPLAY_TZ (default America/New_York) and a local_dt
filter that renders datetimes, ISO strings and Unix timestamps as e.g.
"2026-10-05 13:54 EDT", DST included. Used for the ServiceNow tick list,
technician to-do, imaging tiles and detail page; timestamp_fmt (reports,
backups, enrollment) goes through it too instead of relying on the
process TZ. CSV report_date is Eastern with its offset. Evidence-pack
months are Eastern calendar months, so a wipe late on the 31st lands in
that month rather than the next.
Blancco: init points the new root's /etc/localtime at America/New_York
so the GUI clock is local. Report timestamps keep their explicit
offset; report.utc_offset stays 0 since a fixed offset cannot follow DST.
Report file names stay UTC, as Blancco's own Save names them.
The initrd's busybox has the tr applet but no /bin/tr link, so the
quiet boot printed "/init: line 140: tr: not found" and the stage 1/2
dashboard posts were skipped. sed is linked.
First normal (non-debug) run stuck at 6/7 "Report sent to cloud": outside
Blancco's debug mode Albus does not log erasureFinishedForEntity, Report
saving success or "Auto process reached the final step", which the
parser had been taken from a debug-mode journal.
A watcher thread now reads each new /reports/*.xml (auto mode always
saves there) once it is stable and posts 7: "Wipe complete" (succeeded)
or "Erasure FAILED: <model serial, ...>" (failed). Judged per drive, so
a failed Purge with a successful Clear fallback on the same drive is a
wipe. --outcome FILE prints the verdict for a report.
Automates the Blancco parts of the onsite-support CMMC playbook:
- ENT-056/158 evidence pack: /reports/evidence?month=YYYY-MM returns
ENT-056_<Site>_<YYYY-MM>.zip (summary.csv, README, each report's XML
and PDF). pxe-blancco-evidence.timer saves last month's pack to the
reports share's evidence/ folder on the 1st (Persistent).
- ENT-045/159 ServiceNow tick list: /reports/servicenow lists recent
reports as "Wiped" or "Blancco Wipe failed" with a pre-written
destruction note, copy buttons and Done (CSRF-protected).
- Cloud upload: blancco-progress.py posts each report's upload result
("Sending report document id" -> "Report sent to MC") to
/reports/cloud-status (CSRF-exempt like /imaging/status); the Reports
page and the CSVs show it, so a wipe missing from BMP is visible.
- CSV gains failure_reason (exception messages, failed steps, error
counts), business_location and cloud_upload.
- Preferences: business_location "West Jefferson" (config.BLANCCO_SITE)
and pdf+xml reports; the report copier now copies the PDF too, named
after its XML. .part and hidden files are not listed as reports.
State (cloud confirmations, ServiceNow ticks) is a JSON file at
/var/lib/pxe-webapp/blancco-state.json, off the report share.
Each Blancco PC now gets a tile on /imaging, keyed by BIOS serial with
pctype "blancco", using the /imaging/status endpoint the WinPE clients
already post to (stage_total 7):
1 Downloading Blancco, 2 Starting Blancco (blancco-init.sh, wget)
3 Waiting for licence / Licence OK (N left)
4 Erasing: step N - <step> [NN%]
5 Erasure finished / Report saved
6 Report sent to cloud
7 Done (succeeded), or Erasure FAILED (failed, sticks)
playbook/blancco-progress.py follows the AlbusSeven journal inside the
Blancco rootfs and maps its lines to those stages; patterns taken from
a real 7.20 auto-mode run and replayed with --replay/--dry-run, plus a
synthetic failed step. Init fetches it from the web root at boot and
installs it as blancco-progress.service; the playbook deploys it there.
The report copier adds a "Report copied to share" log line. Stage 1
resets a finished tile for the same serial. All posts are best effort.
process=auto erases unattended but Albus forces auto_backup off outside
workflow/manual, so every report needed a manual Save to reach the
share. Debug SSH on a live 7.20 run showed Albus does keep each report
at /reports/<document_id>.xml ("Report saving success", "CRC verified")
and uploads it to MC, with /mnt/share already mounted.
Install blancco-report-sync as a systemd service in the new root: every
10s copy each finished /reports/*.xml (older than 5s, not yet copied)
to /mnt/share as YYYYMMDD_HHMMSS_report.xml (UTC, as a manual Save
names it), via a .part file and rename. Verified live: the auto-run
report landed as 20261005_164520_report.xml.
With blancco.debug=1 the init now also creates /etc/debug.environment in
the new root (Blancco's isDebugMode, so 2_sshd starts sshd) and installs
root's authorized_keys from http://<server>/blancco/debug_authorized_keys.
Root is locked and sshd is key-only, so normal boots stay closed and a
debug boot without that file on the server stays closed too.
Used to find where Albus keeps the finished report in process=auto, which
never auto-backs-up to the share. Live GetPxeScript.aspx has a matching
"Blancco Drive Eraser (debug / SSH)" item (UEFI, 7.0 kernel, verbose);
not in the playbook menu.
process=workflow failed on the first real run: "failed to fetch the
default workflow ... there is no default workflow for user
api.wipecheck@ge.com". The GE Blancco console needs a default workflow
assigned to that account. Until then run auto, as live 7.15 did:
unattended erasure once licensed, report to the cloud automatically,
SMB share copy via Save (auto_backup is forced off outside
workflow/manual).
Live 7.15 ran process=auto: it erased unattended but Albus forces
auto_backup off outside workflow/manual ("... not workflow or manual:
Report auto backup must be disabled"), so every report needed a manual
Save to reach the share. 7.20 was set to manual, which honours
auto_backup but waits at "Select devices" (seen on the first real run).
workflow is the only process that both runs unattended (the account's
default MC workflow, workflow default=true, fetched over WiFi) and
auto-backs-up the report ("Start file watcher for auto backup of
workflow process"). Live preferences.xml and config.img updated
(backups .bak-20261005-pre-workflow).
On a native boot config.img is an initrd and archiso leaves its files in
/run/initramfs, where /opt/scripts/inc/1_reportKey, 2_certificate,
1_config and 2_flexible_container install report.key, ca.pem,
hybrid.ids and the flexible container. The Ubuntu-kernel UEFI path never
created /run/initramfs, so 1_reportKey wrote an empty /albus/report.key
and the rest were skipped (true since 7.15). Fetch config.img, unpack it
on a tmpfs mounted at the new root's /run (systemd keeps an existing
/run), and fail loudly if report.key is missing.
First real 7.20 run (Dell Pro 5 14, Panther Lake, 7.0 kernel) licensed
and erased, but did not start the cloud workflow on its own or save the
report without a click; the missing report key is the leading suspect.
- Fetch kmod-$KVER.tar.gz first, falling back to kmod.tar.gz, so more
than one boot kernel can share the web root. Live now boots the UEFI
path on Ubuntu's 24.04 HWE 7.0.0-34-generic kernel: Panther Lake
laptops (Dell Pro 14, AX211 on CNVi) need iwlwifi firmware API 100,
which the 6.8 driver cannot load. 6.8 stays deployed as the rollback.
- The 10s DefaultTimeoutStopSec did not reach the /mnt/share mount: a
mount unit's timeout is TimeoutSec=, defaulting to the start timeout.
Add a mnt-share.mount drop-in with TimeoutSec=10s, LazyUnmount=yes and
ForceUnmount=yes.
NOT in the playbook yet: it builds the initrd and kmod tarball from the
server's running kernel (6.8). The 7.0 artifacts on 172.16.9.1 were
built off-box; installing linux-generic-hwe-24.04 on the server is what
makes a playbook run reproduce them.
The UEFI loader ran under set -x and printed modprobe -v, ip link and
ip addr, and neither kernel cmdline had quiet, so the screen scrolled
hundreds of lines before the Blancco GUI. Now:
- quiet loglevel=3 on both Blancco kernel lines (also quiets systemd)
- init clears the screen and prints one line per step
- blancco.debug=1 on the cmdline restores the old trace
- failure paths still print their diagnostics and drop to a shell
Live GetPxeScript.aspx changed to match (backup
.bak-20261005-pre-quiet) and kexec-initrd.img rebuilt.
Restarting Blancco after an erasure stalled on "A stop job is running
for /mnt/share": the network goes down before the SMB report share is
unmounted, and the CIFS umount waits out the 90s default. Reports are
written long before shutdown, so set DefaultTimeoutStopSec=10s in the
switched-to rootfs.
The UEFI Blancco path boots the Ubuntu 6.8 kernel on Blancco's rootfs, so
iwlwifi loads firmware from the Blancco image. 7.20 dropped the API 86
ucode for AX201/AX210/AX211 and ships only API 89, which the 6.8 driver
cannot use: WiFi never came up, so no cloud licence (seen 2026-10-05).
kmod.tar.gz now also carries the host's lib/firmware/iwlwifi-* (183
files, .zst, which the kernel prefers over the image's .xz). Files only:
on the Blancco rootfs lib is a symlink to usr/lib. Live tarball rebuilt
(backup kmod.tar.gz.bak-20261005-pre-iwlfw); the task's creates: guard
means an existing server needs the file removed to pick this up.
The kexec-initrd build copied a fixed set of module directories plus
drivers/net/mii.ko. Ubuntu ships that as mii.ko.zst, so the -e test
skipped it, and usbnet, r8152 and cdc_ether then failed to load with
"Unknown symbol mii_*": USB-C to RJ45 adapters never appeared (seen
2026-10-05, Dell USB-C adapter). igb lacked dca and i2c-algo-bit the
same way.
Walk the host modules.dep from the copied modules and copy every
dependency (73 on 6.8.0-71). Also copy rtl_nic, tigon, bnx2, bnx2x and
e100 firmware, which those drivers request at probe.
The initrd has no udev, so only modules init names get loaded, and
xhci_pci is a module in the Ubuntu kernel. USB 3 / USB-C ports were
therefore dead in the UEFI Blancco path and a Dell USB-C to RJ45 adapter
never appeared; init took the onboard e1000e port with no cable.
- modprobe xhci_pci and xhci_pci_renesas before the NIC drivers
- bring up interfaces inside the carrier wait too, since USB NICs
enumerate seconds after onboard ports
- on no-link, print each interface's driver and the NIC/firmware/link
kernel messages before dropping to a shell
Live kexec-initrd.img also now carries the NIC firmware set from the
Blancco 7.20 early cpio (rtl_nic, tigon, bnx2x, ...) under /lib/firmware.
The playbook build does not add it yet.
The UEFI init took the first interface in /sys/class/net and put
172.16.9.250 on it. On a PC with a second port, dock or USB NIC that can
be the unplugged one; wget then sat at "Connecting to 172.16.9.1:80"
forever (seen 2026-10-05 on a Dell, serial GGX6BN4). Bring every wired
interface up and use the first with carrier; with no link anywhere,
list the interfaces and drop to a shell. wget calls get -T 30 so a
stalled transfer fails loudly instead of hanging.
Live kexec-initrd.img was rebuilt with only init swapped (backup
kexec-initrd.img.bak-20261005-pre-carrier).
