Every server has an imaging NIC (static, the only one that serves
DHCP/TFTP/HTTP boot/SMB) and a GE corporate NIC (DHCP client, SSH and the
webapp on 9009). The playbook guessed the PXE NIC and the installer put
172.16.9.1/24 on every en* NIC, so a two-NIC box could answer on the GE
network.
- New per-site vars pxe_nic_mac / corp_nic_mac (required, from
playbook/site-vars.yml or -e; template site-vars.example.yml), stable
names pxe_nic_name / corp_nic_name (default pxe0 / corp0, overridable),
pxe_server_ip, pxe_subnet, DHCP range. pxe_imaging_mac / pxe_ge_mac are
derived from them. The play stops before touching anything if a MAC is
unset, malformed, duplicated or not on the box (lists the box's NICs).
- Heuristic NIC detection and the single-NIC netplan task replaced by a
two-NIC netplan template (match macaddress + set-name), installed by
pxe-netplan-apply.sh: validate in a scratch root, back up, generate,
rename, apply, restore the previous config on any failure. cloud-init
network config disabled so it cannot rewrite it.
- dnsmasq: /etc/dnsmasq.d/pxe-nic-only.conf from the vars (interface=,
except-interface=, no-dhcp-interface=); dnsmasq.conf IP/range from vars.
- UFW per interface: imaging NIC 22, 67/udp, 69/udp, 80, 4433, 445, 139,
137/udp, 138/udp, 9009; GE NIC 22 and 9009 only. Old any-interface rules
and the stale 9011 rule removed after the new ones are in.
- sysctl drop-in: IPv4/IPv6 forwarding off; ip_nonlocal_bind so Apache
can bind the imaging IP before the address is up.
- Samba [global]: interfaces = 127.0.0.0/8 <imaging NIC>, bind interfaces
only. Apache 80 and 4433 on the imaging IP only; 9009 stays on all (TODO:
the webapp has no auth).
- autoinstall: DHCP only on en*, no static 172.16.9.1; apt geoip off and
offline-install fallback now that the corp NIC may have a lease.
- Blancco UEFI/Secure Boot: blancco_kver (7.0.0-34-generic) drives the
kernel, initramfs and modules builds instead of the running kernel;
outputs vmlinuz-ubuntu-70, kexec-initrd-70.img, kmod-<kver>.tar.gz as the
menu and blancco-init.sh expect; creates: guards keyed on the full kver.
Ubuntu's shimx64.efi.signed published as blancco/shimx64.efi.
download-packages.sh adds shim-signed, linux-generic-hwe-24.04 and the
pinned linux-image/modules/modules-extra for BLANCCO_KVER.
Rendered with the live MACs and names, netplan generate output is
identical to the live config; pxe-nic-only.conf and the Samba binding
match live directive for directive.
A PC whose imaging failed and is re-imaged gets the same IP, and the
server still held state from the dead run, so net use failed (System
error 53). The April fix (18537ac) did not cover it:
- The dnsmasq hook only acted on 'add'/'del'. A re-image comes back to the
existing lease, which dnsmasq reports as 'old', and only when the
hostname/MAC changed. Now dnsmasq sets script-on-renewal, and the hook
clears state on 'old' when the request comes from PXE firmware or iPXE
(vendor class PXEClient), i.e. the machine just network-booted. A
Windows lease renewal (MSFT 5.0) is left alone.
- The 1 h conntrack timeout never applied after a reboot: systemd-sysctl
runs before ufw loads nf_conntrack, so the keys did not exist and the
5-day default stayed. /etc/modules-load.d/pxe-conntrack.conf loads it
first.
- smb-soft-reset.sh flushed neighbours on br-pxe, which no longer exists.
Live 2026-10-14. Hook tested on the server with an unused IP: network-boot
renewal clears, Windows renewal and no vendor class do not, add clears.
pxe-nic-check classifies each NIC from what it hears: LLDP/CDP/802.1X
from managed switches, foreign DHCP offers and their domain (one
DHCPDISCOVER on the imaging NIC; the GE NIC's networkd lease is read
instead), STP and ARP hosts for information. A kernel BPF filter keeps
bulk SMB traffic out of the capture loop; bridge ports are listened on
too, since LLDP and EAPOL stop at the port.
- dnsmasq drop-in runs --guard as ExecStartPre: exit 1 on a ge or
foreign-dhcp imaging verdict, fail open (loudly) on bugs or no link
- pxe-nic-check.timer runs --watch every 5 min: stops dnsmasq on a bad
imaging verdict, never starts it
- /run/pxe-nic-status.json (0644, atomic) for the webapp
- playbook installs it all and writes /etc/pxe-server/nic.conf from
pxe_imaging_mac, pxe_ge_mac, pxe_corp_domain_suffixes,
pxe_nic_allowed_switches
- unit tests plus a no-root network-namespace lab (veth pairs, a raw
socket DHCP/LLDP peer and real dnsmasq); docs/pxe-nic-guard.md has the
operator notes and the two floor tests
Single-site bay-stuck issue at WJ: GE Intune Report IP script filters
Get-NetIPAddress on StartsWith("10.") and posts everything matching
to the GE Tines webhook. Bays at WJ get the PXE LAN 10.9.100.x IP
captured and reported -> GE backend tags bays as on a non-corp 10.x
subnet -> dynamic group eligibility for SFLD policy never matches.
Other GE sites work because their PXE LANs aren't on 10.x at all.
Renumber PXE LAN to RFC1918 172.16.9.0/24 so the GE filter naturally
skips wired PXE addresses without any disable-NIC dance.
Server-side already in flight (netplan dual-bound, dnsmasq scope +
boot URL repointed, blancco preferences + grub.cfg + iPXE GetPxeScript
all sed'd to 172.16.9.1). This commit is the playbook / scripts /
docs side: 109 hits across 35 files sed'd in one shot.
After this lands + boot.wim is rebuilt + bays renumber off DHCP,
the 10.9.100.1 binding will be dropped from netplan as the final
cleanup step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes that go together so a re-image never hits "System error 53":
1. dnsmasq dhcp-script hook (playbook/pxe-server-helpers/pxe-dhcp-hook.sh)
Fires on every add/del lease event. Runs conntrack -D and ss -K for the
client IP so any stale ESTABLISHED SMB session from a previous boot is
cleared before the client reconnects. Runs as root (dnsmasq default).
Wired into /etc/dnsmasq.conf via dhcp-script= directive in the playbook.
2. One-source PPKG (playbook/startnet.cmd + startnet-template.cmd)
The 5 per-Office PPKG copies were bit-for-bit identical; only the
filename differs because BPRT parses Office and Region out of the name.
Store one source file (e.g. GCCH_Prod_SFLD_v4.11.ppkg) and construct
the BPRT-tagged target filename at menu-selection time from variables:
SOURCE_PPKG / PPKG_VER / PPKG_EXP / REGION / OFFICE
copy /Y "Y:\ppkgs\%SOURCE_PPKG%" "W:\Enrollment\%PPKG%"
Bumped PPKG_VER v4.10 -> v4.11 and PPKG_EXP 20260430 -> 20270430.
Saves ~30G on disk per version.
3. run-enrollment.ps1 already committed in 5a9c3db uses provtool.exe
directly (no PowerShell cmdlet 180s timeout). Included here because it
is part of the same end-to-end PPKG path.
WinPE clients re-imaging the same machine hit "System error 53 -
network path not found" on the second attempt. systemctl restart smbd
did not help; only a full server power cycle cleared the state.
Root cause is kernel nf_conntrack: the default TCP ESTABLISHED timeout
is 5 days (432000s), so a session from the first WinPE run whose
client rebooted abnormally leaves an ASSURED ESTABLISHED entry that
ufw's state-tracking rules then mis-classify the new SYN against.
Fix applied in three layers:
- /etc/sysctl.d/99-pxe-conntrack.conf drops TCP ESTABLISHED timeout
to 1 hour and shortens the half-closed states to 30s each.
- smb.conf gains socket options TCP_NODELAY SO_KEEPALIVE IPTOS_LOWDELAY
plus keepalive = 30 and deadtime = 5. Active sessions refresh the
conntrack timer every 30s via keepalives so they never age out;
dead ones expire in an hour.
- /usr/local/sbin/smb-diag.sh snapshots kernel + Samba state for
remote diagnosis; /usr/local/sbin/smb-soft-reset.sh walks a
progressive recovery (nmbd/smbd restart, conntrack flush, arp
flush, ss -K) as an alternative to power-cycling.
conntrack package added to download-packages.sh and playbook verify
list so the offline .deb bundle ships with it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
