ge-aerospace/pxe-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'fix-boot' into pxe-webapp-redesign

Browse Source
This commit is contained in:
cproudlock
2026-10-15 14:27:35 -04:00
parent 606659bf07 21ff95af78
commit b20e6be629
13 changed files with 1022 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
docs/pxe-nic-guard.md
View File

@@ -22,6 +22,9 @@ The NIC guard catches those.
| `/etc/pxe-server/nic.conf` | which NIC is which | | `/etc/pxe-server/nic.conf` | which NIC is which |
| `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start| | `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start|
| `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min | | `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min |
| `/usr/local/sbin/pxe-nic-carrier` | re-check on imaging cable moves |
| `/etc/networkd-dispatcher/<state>.d/50-pxe-nic-carrier` | calls it on link changes |
| `/etc/systemd/system/pxe-nic-carrier.service` | the re-check after carrier gain |
| `/run/pxe-nic-status.json` | last result, read by the webapp | | `/run/pxe-nic-status.json` | last result, read by the webapp |
Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml` Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml`
@@ -33,11 +36,18 @@ from the first two). The same MACs pin the NICs in netplan
## How it decides ## How it decides
It listens on each NIC (raw socket, sends nothing) for: It listens on each NIC (raw socket, sends nothing; it joins the STP,
802.1X, LLDP, CDP and PVST+ multicast groups so a NIC that is not
promiscuous still hears them) for:
- LLDP and CDP: managed switches announce themselves. GE switches do; the - LLDP and CDP: managed switches announce themselves. GE switches do; the
unmanaged imaging switch does not. unmanaged imaging switch does not.
- 802.1X EAPOL: GE ports ask for port authentication. - 802.1X EAPOL from an authenticator: GE ports ask for port
authentication (EAP Request, Success or Failure). EAPOL that clients
send is ignored: iPXE sends three EAPOL-Starts every time it opens a
NIC and Windows sends Starts and EAP Responses, so a bay PXE-booting on
the imaging switch is not mistaken for a GE port. Those frames show up
only as "N client EAPOL frames ignored".
- DHCP OFFER/ACK from any other server. - DHCP OFFER/ACK from any other server.
- STP and the count of distinct ARP hosts: reported, never decisive. - STP and the count of distinct ARP hosts: reported, never decisive.
@@ -70,7 +80,19 @@ never allowed.
about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no
link exits 0 (fail open) with an error line in the journal. link exits 0 (fail open) with an error line in the journal.
- `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs - `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs
`systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq. `systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq. If
dnsmasq was (re)started during the 60 s window (an operator, or the
carrier hook, restarted it and its guard passed), the watch re-checks
for about 6 s and stops dnsmasq only if that still says blocked; if
dnsmasq is already stopped it only logs.
- `pxe-nic-carrier` - the networkd-dispatcher hook. Only the imaging NIC
(or its bridge ports) counts; GE NIC events are ignored. Carrier lost:
dnsmasq is stopped at once if it was running. Carrier back: after 2 s,
`pxe-nic-carrier.service` restarts dnsmasq if it was running or the hook
stopped it, so the guard checks the new network before dnsmasq serves
again; dnsmasq that an operator or the watch stopped stays stopped.
A bridge with several ports stops dnsmasq when any one of them loses
carrier. Log: `journalctl -t pxe-nic-carrier`.
Every start or restart of dnsmasq runs the guard, so `systemctl restart Every start or restart of dnsmasq runs the guard, so `systemctl restart
dnsmasq` and the `@reboot` cron restart take about 6 s longer than before. dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
@@ -81,6 +103,7 @@ dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
cat /run/pxe-nic-status.json # last guard/watch result cat /run/pxe-nic-status.json # last guard/watch result
journalctl -u dnsmasq -b | grep pxe-nic-check journalctl -u dnsmasq -b | grep pxe-nic-check
journalctl -u pxe-nic-check -b journalctl -u pxe-nic-check -b
journalctl -t pxe-nic-carrier -b # cable unplug / replug
## dnsmasq will not start ## dnsmasq will not start
@@ -94,7 +117,7 @@ dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
## Floor tests (run with the floor idle) ## Floor tests (run with the floor idle)
Both tests stop imaging for a few minutes. Have console or GE-side SSH All three tests stop imaging for a few minutes. Have console or GE-side SSH
access to the server; do not rely on the imaging network for the session. access to the server; do not rely on the imaging network for the session.
### (a) Rogue DHCP on the imaging switch: expect the alarm ### (a) Rogue DHCP on the imaging switch: expect the alarm
@@ -126,6 +149,10 @@ dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE.
dnsmasq; the guard would still block it, but do not test that way). dnsmasq; the guard would still block it, but do not test that way).
2. Move the IMAGING NIC's cable from the imaging switch into a live GE 2. Move the IMAGING NIC's cable from the imaging switch into a live GE
network port. Wait for link: `ip -br link` shows the imaging NIC `UP`. network port. Wait for link: `ip -br link` shows the imaging NIC `UP`.
The carrier hook sees the link come up and runs the guard once (one
DHCPDISCOVER) to refresh the status file; with dnsmasq stopped it does
not start it. `journalctl -t pxe-nic-carrier -n 5` shows `dnsmasq was
not running, left stopped`.
3. Run the guard by hand first: 3. Run the guard by hand first:
`sudo pxe-nic-check --guard; echo "exit $?"` `sudo pxe-nic-check --guard; echo "exit $?"`
Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP, Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP,
@@ -142,6 +169,17 @@ dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE.
shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq` shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq`
starts. PXE-boot one client to confirm. starts. PXE-boot one client to confirm.
### (c) Imaging cable unplugged and replugged: expect stop, re-check, start
1. dnsmasq running, imaging NIC `isolated`.
2. Unplug the imaging NIC's cable. Within a second
`journalctl -t pxe-nic-carrier -n 5` shows `carrier lost on imaging NIC
... stopped dnsmasq` and `systemctl is-active dnsmasq` is `inactive`.
3. Plug it back into the imaging switch. About 10 s later the journal
shows `guard passed, dnsmasq (re)started` and dnsmasq is `active`.
Had it gone into a GE port instead, the guard refuses: the journal
shows `dnsmasq NOT started` and dnsmasq stays down.
## Status file ## Status file
`/run/pxe-nic-status.json`, mode 0644, replaced atomically: `/run/pxe-nic-status.json`, mode 0644, replaced atomically:

27
playbook/ipxe/GetPxeScript.aspx
View File

@@ -37,13 +37,23 @@ initrd http://${server}/win11/EFI/Microsoft/Boot/BCD EFI/Microsoft/Boot/BC
initrd http://${server}/win11/EFI/Boot/bootx64.efi EFI/Boot/bootx64.efi initrd http://${server}/win11/EFI/Boot/bootx64.efi EFI/Boot/bootx64.efi
initrd http://${server}/win11/Boot/boot.sdi Boot/boot.sdi initrd http://${server}/win11/Boot/boot.sdi Boot/boot.sdi
initrd http://${server}/win11/sources/boot.wim sources/boot.wim initrd http://${server}/win11/sources/boot.wim sources/boot.wim
boot # A command that fails without || aborts the whole iPXE script (the PC
# drops to firmware with no explanation), so every boot line has one.
boot || goto winpe_failed
:winpe_failed
echo
echo Windows PE boot FAILED (see the error above).
echo Check that the PXE server is up and the win11 boot files are staged.
prompt Press any key to return to menu... && goto menu || goto menu
:clonezilla :clonezilla
set base http://${server}/clonezilla set base http://${server}/clonezilla
kernel ${base}/vmlinuz boot=live username=user union=overlay config components noswap edd=on nomodeset nodmraid locales= keyboard-layouts= ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_batch=no net.ifnames=0 nosplash noprompt fetch=${base}/filesystem.squashfs || goto secureboot_warn kernel ${base}/vmlinuz boot=live username=user union=overlay config components noswap edd=on nomodeset nodmraid locales= keyboard-layouts= ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_batch=no net.ifnames=0 nosplash noprompt fetch=${base}/filesystem.squashfs || goto secureboot_warn
initrd ${base}/initrd.img initrd ${base}/initrd.img || goto boot_failed
boot # iPXE 2.0 accepts an unsigned EFI kernel (PE fallback) and fails only
# here under Secure Boot.
boot || goto secureboot_warn
:blancco :blancco
iseq ${platform} pcbios && goto blancco_bios || goto blancco_efi iseq ${platform} pcbios && goto blancco_bios || goto blancco_efi
@@ -55,7 +65,7 @@ initrd --name intel-ucode.img http://${server}/blancco/intel-ucode.img || goto b
initrd --name amd-ucode.img http://${server}/blancco/amd-ucode.img || goto blancco_bios_failed initrd --name amd-ucode.img http://${server}/blancco/amd-ucode.img || goto blancco_bios_failed
initrd --name config.img http://${server}/blancco/config.img || goto blancco_bios_failed initrd --name config.img http://${server}/blancco/config.img || goto blancco_bios_failed
initrd --name initramfs-bde-linux.img http://${server}/blancco/initramfs-bde-linux.img || goto blancco_bios_failed initrd --name initramfs-bde-linux.img http://${server}/blancco/initramfs-bde-linux.img || goto blancco_bios_failed
boot boot || goto blancco_bios_failed
:blancco_bios_failed :blancco_bios_failed
echo echo
echo Blancco BIOS boot FAILED. echo Blancco BIOS boot FAILED.
@@ -91,7 +101,14 @@ chain http://${server}/blancco/grubx64.efi || goto secureboot_warn
:memtest :memtest
kernel http://${server}/memtest/memtest.efi || goto secureboot_warn kernel http://${server}/memtest/memtest.efi || goto secureboot_warn
boot # iPXE 2.0 accepts an unsigned EFI image (PE fallback) and fails only
# here under Secure Boot.
boot || goto secureboot_warn
:boot_failed
echo
echo Download FAILED (see the error above). Check the PXE server and retry.
prompt Press any key to return to menu... && goto menu || goto menu
:secureboot_warn :secureboot_warn
echo echo

30
playbook/pxe-server-helpers/pxe-dhcp-hook.sh
View File

@@ -3,10 +3,15 @@
# pxe-dhcp-hook.sh - dnsmasq dhcp-script hook. # pxe-dhcp-hook.sh - dnsmasq dhcp-script hook.
# #
# Runs every time a PXE client gets/changes/releases a DHCP lease on # Runs every time a PXE client gets/changes/releases a DHCP lease on
# 172.16.9.0/24. Flushes conntrack entries and drops any lingering # 172.16.9.0/24. Flushes that client's SMB (TCP 445 and 139) conntrack
# TCP sockets for that client IP. Prevents stale server-side state from # entries and drops its lingering SMB sockets. Prevents stale server-side
# causing "System error 53 - network path not found" when a WinPE client # state from causing "System error 53 - network path not found" when a
# re-images the same machine without a clean SMB session teardown. # WinPE client re-images the same machine without a clean SMB session
# teardown.
#
# SMB only, on purpose: the lease that fires this hook is iPXE's, and its
# first HTTP fetch of the boot menu is racing this script. Killing every
# socket or conntrack entry for the client IP broke that fetch.
# #
# dnsmasq calls this with argv: # dnsmasq calls this with argv:
# $1 = action (add | old | del) # $1 = action (add | old | del)
@@ -26,6 +31,9 @@
set +e set +e
# dnsmasq's environment may lack sbin; keep any PATH it gave us first.
PATH="${PATH:+$PATH:}/usr/sbin:/usr/bin:/sbin:/bin"
action="$1" action="$1"
mac="$2" mac="$2"
ip="$3" ip="$3"
@@ -34,11 +42,15 @@ host="${4:-}"
[ -z "$ip" ] && exit 0 [ -z "$ip" ] && exit 0
clear_state() { clear_state() {
/usr/sbin/conntrack -D -s "$ip" >/dev/null 2>&1 local port
/usr/sbin/conntrack -D -d "$ip" >/dev/null 2>&1 for port in 445 139; do
/usr/bin/ss -K "dst = $ip" >/dev/null 2>&1 # client -> our share, and anything the other way round
/usr/bin/ss -K "src = $ip" >/dev/null 2>&1 conntrack -D -p tcp -s "$ip" --dport "$port" >/dev/null 2>&1
logger -t pxe-dhcp-hook "cleared stale state for $1 $ip ($mac${host:+ $host})" conntrack -D -p tcp -d "$ip" --sport "$port" >/dev/null 2>&1
done
# our SMB sockets (local port 445/139) whose peer is this client
ss -K -t "( sport = :445 or sport = :139 ) and dst = $ip" >/dev/null 2>&1
logger -t pxe-dhcp-hook "cleared stale SMB state for $1 $ip ($mac${host:+ $host})"
} }
case "$action" in case "$action" in

145
playbook/pxe-server-helpers/pxe-nic-carrier Executable file
View File

@@ -0,0 +1,145 @@
#!/bin/bash
#
# pxe-nic-carrier - re-check the imaging NIC whenever its cable moves.
#
# The dnsmasq guard (pxe-nic-check --guard) runs only when dnsmasq starts,
# and the watch timer only every 5 min. If the imaging cable is moved into
# a GE port while dnsmasq runs, dnsmasq would answer DHCP on the GE LAN
# until the next watch. This hook closes that gap:
#
# carrier lost dnsmasq is stopped at once (if it was running) and a
# marker remembers that this hook stopped it.
# carrier back pxe-nic-carrier.service runs "pxe-nic-carrier --gain":
# after a 2 s settle, if dnsmasq is running or the marker
# is set, "systemctl restart dnsmasq" - its ExecStartPre
# guard checks the NIC while dnsmasq is down and refuses
# to start it on the GE network. Otherwise it only runs
# the guard to refresh /run/pxe-nic-status.json. dnsmasq
# that an operator or the watch stopped stays stopped.
#
# Only the imaging NIC (and, for a bridge, its ports) counts, as resolved
# by "pxe-nic-check --print-imaging" from /etc/pxe-server/nic.conf or the
# dnsmasq interface= pin. Events on any other NIC (the GE NIC) exit at
# once and touch nothing. Link state is read from sysfs, not from the
# event name, and remembered per interface, so the same transition seen
# twice (carrier, then routable) acts once.
#
# Called by networkd-dispatcher with IFACE and STATE in the environment
# (installed as /etc/networkd-dispatcher/<state>.d/50-pxe-nic-carrier for
# carrier, degraded, degraded-carrier, enslaved, routable, no-carrier and
# off), or by hand: pxe-nic-carrier IFACE. Logs: journalctl -t pxe-nic-carrier
#
# The dispatcher waits for this script, so the event path never runs a
# check itself: loss is two systemctl calls, gain hands off to the service.
set -u
CHECK="${PXE_NIC_CHECK:-/usr/local/sbin/pxe-nic-check}"
SYS_NET="${PXE_SYS_NET:-/sys/class/net}"
RUN="${PXE_NIC_CARRIER_RUN:-/run/pxe-nic-carrier}"
SYSTEMCTL="${PXE_SYSTEMCTL:-systemctl}"
SETTLE="${PXE_NIC_CARRIER_SETTLE:-2}"
GAIN_UNIT="pxe-nic-carrier.service"
TAG="pxe-nic-carrier"
PATH="${PATH:+$PATH:}/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin"
log() { # log PRIORITY MESSAGE
logger -t "$TAG" -p "daemon.$1" -- "$2" 2>/dev/null || \
echo "$TAG: $2" >&2
}
imaging_ifaces() {
"$CHECK" --print-imaging 2>/dev/null
}
carrier_of() { # 1 if the interface has carrier, else 0 (down reads EINVAL)
local c
c=$(cat "$SYS_NET/$1/carrier" 2>/dev/null) || c=0
[ "$c" = 1 ] && echo 1 || echo 0
}
any_carrier() { # any imaging interface or port with carrier
local i
for i in "$@"; do
[ "$(carrier_of "$i")" = 1 ] && return 0
done
return 1
}
dnsmasq_running() { # active, starting or reloading
case "$("$SYSTEMCTL" is-active dnsmasq 2>/dev/null)" in
active|activating|reloading) return 0 ;;
esac
return 1
}
on_event() {
local iface="$1" list=() match="" i now prev
mapfile -t list < <(imaging_ifaces)
for i in "${list[@]}"; do
[ -n "$i" ] && [ "$i" = "$iface" ] && match=1
done
[ -n "$match" ] || exit 0 # not the imaging NIC: hands off
mkdir -p "$RUN" || exit 0
exec 9>"$RUN/event.lock"
flock -w 10 9 || log warning "lock busy, handling $iface anyway"
now=$(carrier_of "$iface")
prev=$(cat "$RUN/link.$iface" 2>/dev/null) || prev=unknown
echo "$now" > "$RUN/link.$iface"
[ "$now" = "$prev" ] && exit 0 # same state again: already handled
if [ "$now" = 0 ]; then
# an in-flight gain check must not restart dnsmasq after this
"$SYSTEMCTL" stop "$GAIN_UNIT" >/dev/null 2>&1
if dnsmasq_running; then
: > "$RUN/stopped-dnsmasq"
"$SYSTEMCTL" stop --no-block dnsmasq
log warning "carrier lost on imaging NIC $iface (${STATE:-event}): stopped dnsmasq; it restarts after the guard passes when the link returns"
else
log info "carrier lost on imaging NIC $iface (${STATE:-event}); dnsmasq not running"
fi
else
log notice "carrier up on imaging NIC $iface (${STATE:-event}, was $prev): re-checking before dnsmasq serves"
"$SYSTEMCTL" restart --no-block "$GAIN_UNIT"
fi
exit 0
}
on_gain() {
local list=() rc
sleep "$SETTLE"
mapfile -t list < <(imaging_ifaces)
if [ "${#list[@]}" -eq 0 ] || [ -z "${list[0]}" ]; then
log err "imaging NIC unknown, nothing to re-check"
exit 0
fi
if ! any_carrier "${list[@]}"; then
log info "imaging NIC ${list[0]} lost carrier again, nothing to do"
exit 0
fi
if dnsmasq_running || [ -e "$RUN/stopped-dnsmasq" ]; then
# restart = stop, then the ExecStartPre guard with dnsmasq down
"$SYSTEMCTL" restart dnsmasq
rc=$?
rm -f "$RUN/stopped-dnsmasq"
if [ "$rc" -eq 0 ]; then
log notice "imaging NIC ${list[0]}: guard passed, dnsmasq (re)started"
else
log crit "imaging NIC ${list[0]}: dnsmasq NOT started (guard refused or start failed, rc $rc); see journalctl -u dnsmasq. Fix the cabling, then systemctl start dnsmasq"
fi
else
"$CHECK" --guard >/dev/null 2>&1
rc=$?
log info "imaging NIC ${list[0]}: dnsmasq was not running, left stopped (guard rc $rc, status refreshed)"
fi
exit 0
}
case "${1:-}" in
--gain) on_gain ;;
-h|--help) sed -n '2,36p' "$0"; exit 0 ;;
*) on_event "${1:-${IFACE:-}}" ;;
esac

16
playbook/pxe-server-helpers/pxe-nic-carrier.service Normal file
View File

@@ -0,0 +1,16 @@
# Started (never enabled) by pxe-nic-carrier, the networkd-dispatcher
# hook, when the imaging NIC gets carrier back. Re-checks the imaging NIC
# before dnsmasq serves again: if dnsmasq was running (or the hook stopped
# it on carrier loss), "systemctl restart dnsmasq" so its ExecStartPre
# guard decides; otherwise it only refreshes /run/pxe-nic-status.json.
# The hook restarts this unit on every carrier gain and stops it on a
# carrier loss, so the newest event wins.
[Unit]
Description=PXE NIC carrier change: re-check the imaging NIC before dnsmasq serves
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pxe-nic-carrier --gain
TimeoutStartSec=120

209
playbook/pxe-server-helpers/pxe-nic-check
View File

@@ -32,7 +32,9 @@ Verdicts per interface:
no-link no carrier no-link no carrier
isolated no foreign DHCP server, no LLDP/CDP/EAPOL seen isolated no foreign DHCP server, no LLDP/CDP/EAPOL seen
ge foreign DHCP with a corp domain (opt 15 / 119), or any ge foreign DHCP with a corp domain (opt 15 / 119), or any
LLDP, CDP or EAPOL seen (unmanaged switches send none) LLDP, CDP or authenticator EAPOL seen (unmanaged switches
send none). EAPOL from clients (iPXE and Windows send
EAPOL-Start and EAP-Response) is ignored.
foreign-dhcp a non-own DHCP server without a corp domain foreign-dhcp a non-own DHCP server without a corp domain
error the check itself failed error the check itself failed
@@ -46,9 +48,14 @@ Modes:
exit 0 = safe to serve, exit 1 = DO NOT serve (ge or exit 0 = safe to serve, exit 1 = DO NOT serve (ge or
foreign-dhcp). Internal errors and no link exit 0 with a loud foreign-dhcp). Internal errors and no link exit 0 with a loud
log line: fail-open on bugs, fail-closed only on detection. log line: fail-open on bugs, fail-closed only on detection.
--print-imaging
the imaging NIC (and its bridge ports), for pxe-nic-carrier.
--watch systemd timer. 60 s window + probe, writes the status file, --watch systemd timer. 60 s window + probe, writes the status file,
and on a bad imaging verdict runs "systemctl stop dnsmasq" and on a bad imaging verdict runs "systemctl stop dnsmasq"
and logs CRIT. It never starts dnsmasq. and logs CRIT. It never starts dnsmasq. If dnsmasq was
(re)started inside the window (its guard passed after part
of the evidence was heard), it re-checks for ~6 s first and
stops dnsmasq only if that still says blocked.
Config: /etc/pxe-server/nic.conf (INI, [nic] section optional) Config: /etc/pxe-server/nic.conf (INI, [nic] section optional)
@@ -96,6 +103,8 @@ STATUS_PATH = "/run/pxe-nic-status.json"
DEFAULT_SUFFIXES = "geaerospace.net, ge.com" DEFAULT_SUFFIXES = "geaerospace.net, ge.com"
DEFAULT_OWN_IP = "172.16.9.1" DEFAULT_OWN_IP = "172.16.9.1"
STOP_CMD = "systemctl stop dnsmasq" STOP_CMD = "systemctl stop dnsmasq"
STATE_CMD = ("systemctl show -p ActiveState "
"-p ActiveEnterTimestampMonotonic dnsmasq")
ETH_P_ALL = 0x0003 ETH_P_ALL = 0x0003
PACKET_OUTGOING = 4 PACKET_OUTGOING = 4
@@ -329,6 +338,23 @@ def parse_bootp(udp_payload):
} }
EAPOL_TYPE_EAP = 0
EAP_AUTHENTICATOR_CODES = (1, 3, 4) # Request, Success, Failure
def eapol_from_authenticator(payload):
"""True only for EAPOL that an 802.1X authenticator (a switch port)
sends: EAPOL type 0 (EAP-Packet) carrying EAP code 1 Request, 3 Success
or 4 Failure. Supplicants send EAPOL-Start (type 1), Logoff (type 2)
and EAP Responses (code 2): iPXE sends three EAPOL-Starts every time
it opens a NIC, and Windows dot3svc sends Starts and Responses, so a
bay PXE-booting next to us must not read as a GE switch port."""
if len(payload) < 5:
return False
return payload[1] == EAPOL_TYPE_EAP \
and payload[4] in EAP_AUTHENTICATOR_CODES
def parse_dhcp_frame(etype, payload): def parse_dhcp_frame(etype, payload):
"""IPv4/UDP 67->68 server message -> bootp dict (+ src ip) or None.""" """IPv4/UDP 67->68 server message -> bootp dict (+ src ip) or None."""
if etype != ETH_IP or len(payload) < 20: if etype != ETH_IP or len(payload) < 20:
@@ -352,7 +378,8 @@ class Observations:
def __init__(self): def __init__(self):
self.lldp = {} # (chassis, port) -> dict self.lldp = {} # (chassis, port) -> dict
self.cdp = {} # (device, port) -> dict self.cdp = {} # (device, port) -> dict
self.eapol = False self.eapol = False # EAPOL from an authenticator (a switch)
self.eapol_client = 0 # supplicant EAPOL frames, ignored
self.stp = False self.stp = False
self.arp_macs = set() self.arp_macs = set()
self.dhcp = {} # (server, src_mac) -> dict self.dhcp = {} # (server, src_mac) -> dict
@@ -371,7 +398,10 @@ class Observations:
self.lldp[(info["chassis"], info["port"])] = info self.lldp[(info["chassis"], info["port"])] = info
return return
if etype == ETH_EAPOL: if etype == ETH_EAPOL:
if eapol_from_authenticator(payload):
self.eapol = True self.eapol = True
else:
self.eapol_client += 1
return return
if etype <= 1500: # 802.3 length field: LLC frames if etype <= 1500: # 802.3 length field: LLC frames
if dst == MAC_CDP: if dst == MAC_CDP:
@@ -763,6 +793,45 @@ def attach_bpf(sock):
# ---------------------------------------------------------------- capture # ---------------------------------------------------------------- capture
SOL_PACKET = 263
PACKET_ADD_MEMBERSHIP = 1
PACKET_MR_MULTICAST = 0
# Link-local groups the NIC's hardware filter drops unless someone joins
# them: STP, EAPOL (PAE), LLDP, CDP/VTP/DTP, PVST+. Without them a plain
# (non-promiscuous) NIC on a CDP-only Cisco port hears nothing.
CAPTURE_GROUPS = ["01:80:c2:00:00:00", "01:80:c2:00:00:03",
"01:80:c2:00:00:0e", "01:00:0c:cc:cc:cc",
"01:00:0c:cc:cc:cd"]
def packet_mreq(ifindex, mac):
"""struct packet_mreq {int mr_ifindex; unsigned short mr_type,
mr_alen; unsigned char mr_address[8];} for PACKET_MR_MULTICAST."""
addr = bytes.fromhex(norm_mac(mac).replace(":", ""))
if len(addr) != 6:
raise ValueError("bad MAC %r" % mac)
return struct.pack("iHH8s", ifindex, PACKET_MR_MULTICAST, len(addr),
addr)
def join_groups(sock, ifname, groups=CAPTURE_GROUPS):
"""Join each group on this socket's interface. The kernel drops the
memberships when the socket closes. Returns the groups that failed."""
failed = []
try:
ifindex = socket.if_nametoindex(ifname)
except OSError:
return list(groups)
for g in groups:
try:
sock.setsockopt(SOL_PACKET, PACKET_ADD_MEMBERSHIP,
packet_mreq(ifindex, g))
except (OSError, ValueError):
failed.append(g)
return failed
def capture(targets, window, probe_ifaces, own_macs, probe_window=6.0): def capture(targets, window, probe_ifaces, own_macs, probe_window=6.0):
"""targets: {report_iface: [capture ifaces]}. Listens on all of them """targets: {report_iface: [capture ifaces]}. Listens on all of them
for `window` s (at least probe_window after the probe). Returns for `window` s (at least probe_window after the probe). Returns
@@ -783,6 +852,10 @@ def capture(targets, window, probe_ifaces, own_macs, probe_window=6.0):
log("no BPF filter on %s (%s), parsing every frame" log("no BPF filter on %s (%s), parsing every frame"
% (cap, e), "warning") % (cap, e), "warning")
s.bind((cap, ETH_P_ALL)) s.bind((cap, ETH_P_ALL))
bad = join_groups(s, cap)
if bad:
log("could not join %s on %s; a quiet switch may be "
"missed" % (", ".join(bad), cap), "warning")
s.setblocking(False) s.setblocking(False)
socks[s] = name socks[s] = name
except OSError as e: except OSError as e:
@@ -883,6 +956,10 @@ def classify(link_up, error, obs, dhcp_servers, suffixes, window,
evidence.append(txt) evidence.append(txt)
if obs and obs.eapol: if obs and obs.eapol:
evidence.append("802.1X EAPOL seen (port authentication)") evidence.append("802.1X EAPOL seen (port authentication)")
if obs and obs.eapol_client:
evidence.append("%d client EAPOL frames ignored (supplicants such "
"as iPXE EAPOL-Start, informational)"
% obs.eapol_client)
if obs and obs.stp: if obs and obs.stp:
evidence.append("STP BPDUs seen (informational)") evidence.append("STP BPDUs seen (informational)")
@@ -1135,6 +1212,81 @@ def summary(status):
return "\n".join(lines) return "\n".join(lines)
def parse_unit_state(text):
"""`systemctl show -p ActiveState -p ActiveEnterTimestampMonotonic`
output -> (active_state or None, active_enter_us or None)."""
props = {}
for line in (text or "").splitlines():
if "=" in line:
k, v = line.split("=", 1)
props[k.strip()] = v.strip()
state = props.get("ActiveState") or None
try:
since = int(props.get("ActiveEnterTimestampMonotonic", ""))
except ValueError:
since = None
return state, since
def dnsmasq_state(cmd):
try:
res = subprocess.run(cmd, shell=True, capture_output=True,
text=True, timeout=10)
except (OSError, subprocess.SubprocessError):
return None, None
if res.returncode != 0:
return None, None
return parse_unit_state(res.stdout)
def watch_decision(state, since_us, window_start_us):
"""What --watch does with a 'blocked' verdict, given dnsmasq's
ActiveState / ActiveEnterTimestampMonotonic and the CLOCK_MONOTONIC
microsecond the watch's capture began. -> "stop", "skip" (already
stopped) or "recheck" (dnsmasq was (re)started inside the window, so
its own guard passed after part of our evidence was heard; confirm
with a fresh short check before stopping it)."""
if state is None:
return "stop" # cannot tell: stopping is the safe side
if state in ("inactive", "failed", "deactivating"):
return "skip"
if state in ("activating", "reloading"):
return "recheck" # its guard is running right now
if since_us and since_us > window_start_us:
return "recheck"
return "stop"
def stop_dnsmasq(cmd):
try:
rc = subprocess.run(cmd, shell=True, timeout=60).returncode
if rc != 0:
log("'%s' exited %d" % (cmd, rc), "crit")
except (OSError, subprocess.SubprocessError) as e:
log("could not stop dnsmasq: %s" % e, "crit")
def recheck(args, cfg):
"""A guard-style check (imaging NIC, short window + probe), written to
the status file. -> (state, reason). An internal error counts as
blocked: the 60 s watch already saw the bad evidence."""
sub = argparse.Namespace(**vars(args))
sub.mode = "guard"
sub.window = 5.0
sub.probe = True
try:
status, _ = run_check(sub, cfg)
except Exception as e:
return "blocked", "re-check failed (%s); keeping the watch verdict" \
% e
if args.write:
try:
write_status(args.write, status)
except OSError as e:
log("could not write %s: %s" % (args.write, e), "err")
return status["dhcp_guard"]["state"], status["dhcp_guard"]["reason"]
def main(argv=None): def main(argv=None):
ap = argparse.ArgumentParser( ap = argparse.ArgumentParser(
prog=PROG, prog=PROG,
@@ -1149,6 +1301,10 @@ def main(argv=None):
mode.add_argument("--watch", dest="mode", action="store_const", mode.add_argument("--watch", dest="mode", action="store_const",
const="watch", help="timer mode: stop dnsmasq on a " const="watch", help="timer mode: stop dnsmasq on a "
"bad imaging verdict") "bad imaging verdict")
mode.add_argument("--print-imaging", dest="mode", action="store_const",
const="print-imaging", help="print the imaging NIC "
"and, for a bridge, its ports (one per line; nothing "
"if unknown), then exit. Used by pxe-nic-carrier.")
ap.add_argument("--json", action="store_true", ap.add_argument("--json", action="store_true",
help="print the status JSON") help="print the status JSON")
ap.add_argument("--write", metavar="PATH", nargs="?", const=STATUS_PATH, ap.add_argument("--write", metavar="PATH", nargs="?", const=STATUS_PATH,
@@ -1171,6 +1327,9 @@ def main(argv=None):
ap.add_argument("--lease-dir", default=LEASE_DIR) ap.add_argument("--lease-dir", default=LEASE_DIR)
ap.add_argument("--stop-cmd", default=STOP_CMD, ap.add_argument("--stop-cmd", default=STOP_CMD,
help="command --watch runs to stop dnsmasq") help="command --watch runs to stop dnsmasq")
ap.add_argument("--state-cmd", default=STATE_CMD,
help="command --watch runs to read dnsmasq's "
"ActiveState and ActiveEnterTimestampMonotonic")
args = ap.parse_args(argv) args = ap.parse_args(argv)
args.mode = args.mode or "check" args.mode = args.mode or "check"
if args.window is None: if args.window is None:
@@ -1180,6 +1339,22 @@ def main(argv=None):
if args.mode in ("guard", "watch") and not args.write: if args.mode in ("guard", "watch") and not args.write:
args.write = STATUS_PATH args.write = STATUS_PATH
if args.mode == "print-imaging":
try:
cfg = load_config(args.config, args.dnsmasq_nic_conf)
ifaces = all_ifaces()
imaging, _ge, _notes = resolve_roles(cfg, ifaces, iface_addrs(),
args.lease_dir)
except Exception as e:
log("could not resolve the imaging NIC: %s" % e, "err")
return 2
if imaging:
print("\n".join([imaging] + (bridge_ports(imaging)
if is_bridge(imaging) else [])))
return 0
# CLOCK_MONOTONIC, the clock of ActiveEnterTimestampMonotonic
window_start_us = int(time.clock_gettime(time.CLOCK_MONOTONIC) * 1e6)
try: try:
cfg = load_config(args.config, args.dnsmasq_nic_conf) cfg = load_config(args.config, args.dnsmasq_nic_conf)
status, imaging = run_check(args, cfg) status, imaging = run_check(args, cfg)
@@ -1217,15 +1392,25 @@ def main(argv=None):
return 0 return 0
if args.mode == "watch": if args.mode == "watch":
if state == "blocked": if state == "blocked":
log("imaging NIC is NOT isolated, stopping dnsmasq: " + reason, dstate, since = dnsmasq_state(args.state_cmd)
"crit") what = watch_decision(dstate, since, window_start_us)
try: if what == "recheck":
rc = subprocess.run(args.stop_cmd, shell=True, log("dnsmasq was (re)started during this watch window "
timeout=60).returncode "(its guard passed); re-checking before stopping it: "
if rc != 0: + reason, "warning")
log("'%s' exited %d" % (args.stop_cmd, rc), "crit") state, reason = recheck(args, cfg)
except (OSError, subprocess.SubprocessError) as e: if state == "blocked":
log("could not stop dnsmasq: %s" % e, "crit") what = "stop"
else:
log("re-check says %s, leaving dnsmasq running: %s"
% (state, reason), "notice")
if what == "skip":
log("imaging NIC is NOT isolated, dnsmasq already %s: %s"
% (dstate, reason), "crit")
elif what == "stop":
log("imaging NIC is NOT isolated, stopping dnsmasq: "
+ reason, "crit")
stop_dnsmasq(args.stop_cmd)
elif state == "unknown": elif state == "unknown":
log("watch could not verify the imaging NIC: " + reason, "err") log("watch could not verify the imaging NIC: " + reason, "err")
else: else:

3
playbook/pxe-server-helpers/pxe-nic-check.service
View File

@@ -3,7 +3,8 @@
# /run/pxe-nic-status.json for the webapp, and STOPS dnsmasq if the imaging # /run/pxe-nic-status.json for the webapp, and STOPS dnsmasq if the imaging
# NIC turns out to be on the GE network or next to another DHCP server. # NIC turns out to be on the GE network or next to another DHCP server.
# It never starts dnsmasq: after fixing the cabling, start it by hand # It never starts dnsmasq: after fixing the cabling, start it by hand
# (systemctl start dnsmasq), which re-runs the guard. # (systemctl start dnsmasq), which re-runs the guard. If dnsmasq was
# restarted inside the window it re-checks (~6 s) before stopping it.
[Unit] [Unit]
Description=PXE NIC watch: keep dnsmasq off the GE corporate network Description=PXE NIC watch: keep dnsmasq off the GE corporate network

14
tests/pxe-nic-check/README.md
View File

@@ -12,6 +12,15 @@ Synthesized LLDP, CDP, EAPOL, STP and ARP frames and DHCPOFFER/ACK packets
cloned-PXE-server handling; networkd lease parsing; config fallbacks; the cloned-PXE-server handling; networkd lease parsing; config fallbacks; the
dnsmasq_bound logic; the status JSON shape and atomic 0644 write. dnsmasq_bound logic; the status JSON shape and atomic 0644 write.
## Shell helpers
`test_helpers.py` runs `pxe-dhcp-hook.sh` (add/del/old with PXEClient,
MSFT 5.0 and no vendor class: only SMB 445/139 conntrack entries and
sockets are cleared) and `pxe-nic-carrier` (loss stops dnsmasq, gain
hands off once, GE NIC events touch nothing, bridge ports count) with
stub `ss`, `conntrack`, `logger`, `systemctl` and `pxe-nic-check` on a
fake PATH and a fake `/sys/class/net`. Part of the unit run above.
## Namespace lab ## Namespace lab
python3 tests/pxe-nic-check/lab.py # all scenarios, ~5 min python3 tests/pxe-nic-check/lab.py # all scenarios, ~5 min
@@ -38,6 +47,7 @@ namespace cannot drop privileges; scenarios are skipped without it).
| not-corp-domain | DHCP domain notgeaerospace.net | foreign-dhcp | 1 | | not-corp-domain | DHCP domain notgeaerospace.net | foreign-dhcp | 1 |
| lldp-only | LLDP | ge | 1 | | lldp-only | LLDP | ge | 1 |
| eapol-only | 802.1X EAPOL | ge | 1 | | eapol-only | 802.1X EAPOL | ge | 1 |
| eapol-client | EAPOL-Start + EAP-Response, like a PXE-booting bay | isolated | 0 |
| own-server | DHCP with server id 172.16.9.1 from a local MAC | isolated | 0 | | own-server | DHCP with server id 172.16.9.1 from a local MAC | isolated | 0 |
| cloned-pxe | DHCP with server id 172.16.9.1, foreign MAC | foreign-dhcp | 1 | | cloned-pxe | DHCP with server id 172.16.9.1, foreign MAC | foreign-dhcp | 1 |
| no-link | peer end down | no-link | 0 | | no-link | peer end down | no-link | 0 |
@@ -46,5 +56,9 @@ namespace cannot drop privileges; scenarios are skipped without it).
| own-dnsmasq | real dnsmasq serving the imaging NIC itself | isolated, dnsmasq_bound true | 0 | | own-dnsmasq | real dnsmasq serving the imaging NIC itself | isolated, dnsmasq_bound true | 0 |
| br-* | same, imaging NIC is a bridge (like br-pxe), named in nic.conf by a port's MAC | | | | br-* | same, imaging NIC is a bridge (like br-pxe), named in nic.conf by a port's MAC | | |
`rogue`, `lldp-only` and `br-lldp` also run `--watch` as if dnsmasq had
been restarted inside the window: it must log the re-check and still stop.
Every watch gets `--state-cmd` (a printf standing in for `systemctl show`).
The GE NIC (g0) is checked from a fake networkd lease with domain The GE NIC (g0) is checked from a fake networkd lease with domain
test.geaerospace.net and must read `ge` in every scenario. test.geaerospace.net and must read `ge` in every scenario.

18
tests/pxe-nic-check/frames.py
View File

@@ -66,13 +66,25 @@ def cdp(src, device="WJ-CORE1.wjs.geaerospace.net", port="GigabitEthernet1/0/3",
+ llc + llc
def eapol(src): def eapol(src, code=1):
# EAPOL v2, type 0 (EAP packet), EAP Request/Identity """EAPOL v2, type 0 (EAP packet). code 1 Request/Identity (what a
eap = bytes([1, 1]) + struct.pack("!H", 5) + bytes([1]) switch port sends), 2 Response/Identity (a supplicant), 3 Success,
4 Failure."""
if code in (1, 2):
eap = bytes([code, 1]) + struct.pack("!H", 5) + bytes([1])
else:
eap = bytes([code, 1]) + struct.pack("!H", 4)
body = bytes([2, 0]) + struct.pack("!H", len(eap)) + eap body = bytes([2, 0]) + struct.pack("!H", len(eap)) + eap
return eth("01:80:c2:00:00:03", src, 0x888E, body) return eth("01:80:c2:00:00:03", src, 0x888E, body)
def eapol_start(src, eapol_type=1):
"""Supplicant EAPOL-Start (type 1) or Logoff (type 2), empty body:
iPXE sends three Starts each time it opens a NIC."""
body = bytes([2, eapol_type]) + struct.pack("!H", 0)
return eth("01:80:c2:00:00:03", src, 0x888E, body)
def stp(src): def stp(src):
bpdu = b"\x00\x00\x00\x00\x00" + b"\x80\x00" + mac(src) + b"\x00" * 23 bpdu = b"\x00\x00\x00\x00\x00" + b"\x80\x00" + mac(src) + b"\x00" * 23
llc = b"\x42\x42\x03" + bpdu llc = b"\x42\x42\x03" + bpdu

112
tests/pxe-nic-check/lab.py
View File

@@ -34,6 +34,8 @@ HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.normpath(os.path.join( SCRIPT = os.path.normpath(os.path.join(
HERE, "..", "..", "playbook", "pxe-server-helpers", "pxe-nic-check")) HERE, "..", "..", "playbook", "pxe-server-helpers", "pxe-nic-check"))
PEER = os.path.join(HERE, "lab_peer.py") PEER = os.path.join(HERE, "lab_peer.py")
CARRIER = os.path.normpath(os.path.join(
HERE, "..", "..", "playbook", "pxe-server-helpers", "pxe-nic-carrier"))
ENTRY_KEYS = ["iface", "mac", "role", "link", "speed_mbps", "addrs", ENTRY_KEYS = ["iface", "mac", "role", "link", "speed_mbps", "addrs",
"verdict", "evidence", "dhcp_servers", "switch", "arp_hosts", "verdict", "evidence", "dhcp_servers", "switch", "arp_hosts",
@@ -56,6 +58,10 @@ SCENARIOS = [
("lldp-only", ["--lldp", "WJ-IDF2"], "ge", 1, ("lldp-only", ["--lldp", "WJ-IDF2"], "ge", 1,
"LLDP from WJ-IDF2 port Gi1/0/14"), "LLDP from WJ-IDF2 port Gi1/0/14"),
("eapol-only", ["--eapol"], "ge", 1, "EAPOL"), ("eapol-only", ["--eapol"], "ge", 1, "EAPOL"),
# a bay PXE-booting on the imaging switch: iPXE EAPOL-Start and a
# Windows EAP-Response must not read as a GE switch port
("eapol-client", ["--eapol-start"], "isolated", 0,
"client EAPOL frames ignored"),
("own-server", ["--dhcp", "172.16.9.1"], "isolated", 0, ("own-server", ["--dhcp", "172.16.9.1"], "isolated", 0,
"own DHCP server 172.16.9.1 answered (ignored)"), "own DHCP server 172.16.9.1 answered (ignored)"),
("cloned-pxe", ["--dhcp", "172.16.9.1", "--mac", "02:cc:00:00:00:01"], ("cloned-pxe", ["--dhcp", "172.16.9.1", "--mac", "02:cc:00:00:00:01"],
@@ -77,6 +83,8 @@ BRIDGE_SCENARIOS = [
("br-isolated", None, "isolated", 0, "no DHCP server, no switch"), ("br-isolated", None, "isolated", 0, "no DHCP server, no switch"),
("br-lldp", ["--lldp", "WJ-IDF2"], "ge", 1, "LLDP from WJ-IDF2"), ("br-lldp", ["--lldp", "WJ-IDF2"], "ge", 1, "LLDP from WJ-IDF2"),
("br-eapol", ["--eapol"], "ge", 1, "EAPOL"), ("br-eapol", ["--eapol"], "ge", 1, "EAPOL"),
("br-eapol-client", ["--eapol-start"], "isolated", 0,
"client EAPOL frames ignored"),
("br-rogue", ["--dhcp", "10.99.0.1"], "foreign-dhcp", 1, ("br-rogue", ["--dhcp", "10.99.0.1"], "foreign-dhcp", 1,
"DHCP from 10.99.0.1, no domain"), "DHCP from 10.99.0.1, no domain"),
("br-fake-ge", ["--dhcp", "10.134.48.1", "--domain", ("br-fake-ge", ["--dhcp", "10.134.48.1", "--domain",
@@ -85,6 +93,17 @@ BRIDGE_SCENARIOS = [
] ]
# Scenarios that also run --watch as if an operator restarted dnsmasq in
# the middle of the window: the watch must re-check, and still stop it.
RACE = {"rogue", "lldp-only", "br-lldp"}
# What `systemctl show` would say: dnsmasq active since boot, or since
# just now (inside the watch window).
STATE_EARLY = "printf 'ActiveState=active\\nActiveEnterTimestampMonotonic=1\\n'"
STATE_LATE = ("printf 'ActiveState=active\\n"
"ActiveEnterTimestampMonotonic=%d\\n'" % (2 ** 62))
def sh(*cmd): def sh(*cmd):
subprocess.run(cmd, check=True) subprocess.run(cmd, check=True)
@@ -252,18 +271,97 @@ def scenario(name, spec, want, want_rc, needle, bound, conf, leases, tmp,
# 3. watch, with a marker file standing in for systemctl stop # 3. watch, with a marker file standing in for systemctl stop
marker = os.path.join(tmp, name + ".stopped") marker = os.path.join(tmp, name + ".stopped")
r = run_script(["--watch", "--window", "3", "--write", status, r = run_script(["--watch", "--window", "3", "--write", status,
"--stop-cmd", "touch " + marker], conf, leases) "--stop-cmd", "touch " + marker,
"--state-cmd", STATE_EARLY], conf, leases)
stopped = os.path.exists(marker) stopped = os.path.exists(marker)
if stopped != bool(want_rc): if stopped != bool(want_rc):
fails.append("watch stop called=%s (want %s): %s" fails.append("watch stop called=%s (want %s): %s"
% (stopped, bool(want_rc), r.stderr.strip())) % (stopped, bool(want_rc), r.stderr.strip()))
if want_rc and "<2>" not in r.stderr and "CRIT" not in r.stderr: if want_rc and "<2>" not in r.stderr and "CRIT" not in r.stderr:
fails.append("watch did not log CRIT") fails.append("watch did not log CRIT")
# 4. watch racing an operator restart: re-check, still stop
if name in RACE:
marker2 = marker + ".race"
r = run_script(["--watch", "--window", "3", "--write", status,
"--stop-cmd", "touch " + marker2,
"--state-cmd", STATE_LATE], conf, leases)
if "re-checking" not in r.stderr:
fails.append("race: watch did not re-check: %s"
% r.stderr.strip())
if not os.path.exists(marker2):
fails.append("race: re-check did not stop dnsmasq: %s"
% r.stderr.strip())
if "could not join" in r.stderr:
fails.append("multicast join failed: %s" % r.stderr.strip())
finally: finally:
stop_peer(p, spec) stop_peer(p, spec)
return fails, im return fails, im
def carrier_lab(conf, leases, tmp):
"""pxe-nic-carrier against real sysfs carrier in the namespace: pull
the far end of v0's veth, plug it back, and poke it with GE NIC
events. systemctl is a stub that records argv and says dnsmasq is
active; pxe-nic-check --print-imaging is the real script."""
fails = []
stub = os.path.join(tmp, "carrier-stub")
os.makedirs(stub, exist_ok=True)
calls = os.path.join(stub, "calls")
with open(os.path.join(stub, "systemctl"), "w") as f:
f.write('#!/bin/sh\necho "$*" >> %s\n'
'[ "$1" = is-active ] && echo active\nexit 0\n' % calls)
with open(os.path.join(stub, "check"), "w") as f:
f.write('#!/bin/sh\nexec %s %s --config %s --dnsmasq-nic-conf '
'/nonexistent --lease-dir %s "$@"\n'
% (sys.executable, SCRIPT, conf, leases))
with open(os.path.join(stub, "logger"), "w") as f: # not the host log
f.write('#!/bin/sh\necho "$*" >> %s/log\n' % stub)
for n in ("systemctl", "check", "logger"):
os.chmod(os.path.join(stub, n), 0o755)
env = dict(os.environ, PATH=stub + ":" + os.environ.get("PATH", ""),
PXE_SYSTEMCTL=os.path.join(stub, "systemctl"),
PXE_NIC_CHECK=os.path.join(stub, "check"),
PXE_NIC_CARRIER_RUN=os.path.join(stub, "run"))
def event(iface, state):
if os.path.exists(calls):
os.unlink(calls)
subprocess.run(["bash", CARRIER], env=dict(env, IFACE=iface,
STATE=state),
timeout=30, check=True)
if not os.path.exists(calls):
return []
with open(calls) as f:
return f.read().splitlines()
event("v0", "routable") # first sight: up
sh("ip", "link", "set", "v1", "down")
time.sleep(0.5)
got = event("v0", "no-carrier")
if "stop --no-block dnsmasq" not in got:
fails.append("carrier loss did not stop dnsmasq: %s" % got)
sh("ip", "link", "set", "v0", "down") # admin down: EINVAL
got = event("v0", "off")
if got:
fails.append("repeat loss not idempotent: %s" % got)
sh("ip", "link", "set", "v0", "up")
sh("ip", "link", "set", "v1", "up")
time.sleep(1.5)
got = event("v0", "carrier")
if "restart --no-block pxe-nic-carrier.service" not in got:
fails.append("carrier gain did not hand off: %s" % got)
if event("v0", "routable"):
fails.append("repeat gain not idempotent")
sh("ip", "link", "set", "g1", "down")
time.sleep(0.5)
got = event("g0", "no-carrier")
sh("ip", "link", "set", "g1", "up")
if got:
fails.append("GE NIC event touched systemctl: %s" % got)
time.sleep(1.0)
return fails
def inside(selected): def inside(selected):
tmp = tempfile.mkdtemp(prefix="pxe-nic-lab-") tmp = tempfile.mkdtemp(prefix="pxe-nic-lab-")
conf, conf_br, leases = setup(tmp) conf, conf_br, leases = setup(tmp)
@@ -293,6 +391,18 @@ def inside(selected):
print(" " + f, flush=True) print(" " + f, flush=True)
if im and not fails: if im and not fails:
print(" " + " | ".join(im["evidence"]), flush=True) print(" " + " | ".join(im["evidence"]), flush=True)
if not selected or "carrier" in selected:
t0 = time.monotonic()
try:
fails = carrier_lab(conf, leases, tmp)
except Exception as e:
fails = ["exception: %r" % e]
state = "FAIL" if fails else "PASS"
results.append(("carrier", state, fails))
print("%s %-16s pxe-nic-carrier on real carrier (%.0f s)"
% (state, "carrier", time.monotonic() - t0), flush=True)
for f in fails:
print(" " + f, flush=True)
n_pass = sum(1 for r in results if r[1] == "PASS") n_pass = sum(1 for r in results if r[1] == "PASS")
n_fail = sum(1 for r in results if r[1] == "FAIL") n_fail = sum(1 for r in results if r[1] == "FAIL")
n_skip = sum(1 for r in results if r[1] == "SKIP") n_skip = sum(1 for r in results if r[1] == "SKIP")

6
tests/pxe-nic-check/lab_peer.py
View File

@@ -10,6 +10,8 @@ plays a switch and/or a DHCP server with raw AF_PACKET sockets:
--domain NAME option 15 in the offer --domain NAME option 15 in the offer
--lldp SYSNAME send an LLDP frame every second --lldp SYSNAME send an LLDP frame every second
--eapol send an EAPOL Request/Identity every second --eapol send an EAPOL Request/Identity every second
--eapol-start send EAPOL-Start + EAP-Response like a PXE-booting
client (iPXE, Windows dot3svc) every second
--mac MAC source MAC for the frames (default: the iface's MAC) --mac MAC source MAC for the frames (default: the iface's MAC)
Never run this outside the lab netns: it is a rogue DHCP server. Never run this outside the lab netns: it is a rogue DHCP server.
@@ -34,6 +36,7 @@ def main():
ap.add_argument("--domain") ap.add_argument("--domain")
ap.add_argument("--lldp") ap.add_argument("--lldp")
ap.add_argument("--eapol", action="store_true") ap.add_argument("--eapol", action="store_true")
ap.add_argument("--eapol-start", action="store_true")
ap.add_argument("--mac") ap.add_argument("--mac")
ap.add_argument("--duration", type=float, default=120) ap.add_argument("--duration", type=float, default=120)
args = ap.parse_args() args = ap.parse_args()
@@ -58,6 +61,9 @@ def main():
vlan=309)) vlan=309))
if args.eapol: if args.eapol:
s.send(F.eapol(mac)) s.send(F.eapol(mac))
if args.eapol_start:
s.send(F.eapol_start(mac))
s.send(F.eapol(mac, code=2))
next_beacon = now + 1.0 next_beacon = now + 1.0
r, _, _ = select.select([s], [], [], 0.2) r, _, _ = select.select([s], [], [], 0.2)
if not r: if not r:

290
tests/pxe-nic-check/test_helpers.py Normal file
View File

@@ -0,0 +1,290 @@
"""
Tests for the shell helpers next to pxe-nic-check: pxe-dhcp-hook.sh (the
dnsmasq dhcp-script) and pxe-nic-carrier (the networkd-dispatcher hook).
No root, no network: every command they run (ss, conntrack, logger,
systemctl, pxe-nic-check) is a stub on a fake PATH that records its argv.
python3 -m unittest discover -s tests/pxe-nic-check -v
"""
import os
import shutil
import subprocess
import tempfile
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
HELPERS = os.path.normpath(os.path.join(HERE, "..", "..", "playbook",
"pxe-server-helpers"))
HOOK = os.path.join(HELPERS, "pxe-dhcp-hook.sh")
CARRIER = os.path.join(HELPERS, "pxe-nic-carrier")
# Each stub appends "name arg1 arg2 ..." (args joined by \x1f) to $CALLS.
STUB = r"""#!/bin/bash
{ printf '%s' "$(basename "$0")"; for a in "$@"; do printf '\x1f%s' "$a"; done
printf '\n'; } >> "$CALLS"
"""
SYSTEMCTL = STUB + r"""
case "$1" in
is-active) s=$(cat "$STUBDIR/dnsmasq.state" 2>/dev/null || echo active)
echo "$s"; [ "$s" = active ] ;;
restart) [ "$2" = dnsmasq ] && exit "$(cat "$STUBDIR/restart.rc" 2>/dev/null || echo 0)"
exit 0 ;;
*) exit 0 ;;
esac
"""
CHECK = STUB + r"""
[ "$1" = --print-imaging ] && cat "$STUBDIR/imaging" 2>/dev/null
exit 0
"""
class Stubs:
def __init__(self):
self.dir = tempfile.mkdtemp(prefix="pxe-helper-test-")
self.bin = os.path.join(self.dir, "bin")
os.mkdir(self.bin)
self.calls = os.path.join(self.dir, "calls")
for name, body in (("ss", STUB), ("conntrack", STUB),
("logger", STUB), ("systemctl", SYSTEMCTL),
("pxe-nic-check", CHECK)):
self.write(os.path.join(self.bin, name), body, 0o755)
self.sys = os.path.join(self.dir, "sys")
self.run_dir = os.path.join(self.dir, "run")
def write(self, path, text, mode=0o644):
with open(path, "w") as f:
f.write(text)
os.chmod(path, mode)
def env(self, **extra):
env = {"PATH": self.bin + ":/usr/bin:/bin", "CALLS": self.calls,
"STUBDIR": self.dir,
"PXE_NIC_CHECK": os.path.join(self.bin, "pxe-nic-check"),
"PXE_SYSTEMCTL": os.path.join(self.bin, "systemctl"),
"PXE_SYS_NET": self.sys,
"PXE_NIC_CARRIER_RUN": self.run_dir,
"PXE_NIC_CARRIER_SETTLE": "0"}
env.update(extra)
return env
def get_calls(self, name=None):
try:
with open(self.calls) as f:
lines = f.read().splitlines()
except OSError:
return []
out = [ln.split("\x1f") for ln in lines]
return [c for c in out if name is None or c[0] == name]
def reset(self):
if os.path.exists(self.calls):
os.unlink(self.calls)
def link(self, iface, carrier):
d = os.path.join(self.sys, iface)
os.makedirs(d, exist_ok=True)
path = os.path.join(d, "carrier")
if carrier is None: # admin down: reading carrier fails
if os.path.exists(path):
os.unlink(path)
else:
self.write(path, "%d\n" % carrier)
def cleanup(self):
shutil.rmtree(self.dir, ignore_errors=True)
class DhcpHook(unittest.TestCase):
def setUp(self):
self.s = Stubs()
def tearDown(self):
self.s.cleanup()
def run_hook(self, action, vendor=None):
env = self.s.env()
if vendor is not None:
env["DNSMASQ_VENDOR_CLASS"] = vendor
r = subprocess.run(["bash", HOOK, action, "02:aa:00:00:00:05",
"172.16.9.50", "bay3105"], env=env,
capture_output=True, text=True, timeout=10)
self.assertEqual(r.returncode, 0, r.stderr)
return self.s.get_calls()
def assert_smb_only(self, calls):
ct = [c[1:] for c in calls if c[0] == "conntrack"]
self.assertEqual(sorted(ct), sorted([
["-D", "-p", "tcp", "-s", "172.16.9.50", "--dport", "445"],
["-D", "-p", "tcp", "-d", "172.16.9.50", "--sport", "445"],
["-D", "-p", "tcp", "-s", "172.16.9.50", "--dport", "139"],
["-D", "-p", "tcp", "-d", "172.16.9.50", "--sport", "139"],
]))
ss = [c[1:] for c in calls if c[0] == "ss"]
self.assertEqual(ss, [["-K", "-t", "( sport = :445 or sport = :139 )"
" and dst = 172.16.9.50"]])
logs = [c for c in calls if c[0] == "logger"]
self.assertEqual(len(logs), 1)
self.assertIn("SMB", logs[0][-1])
def test_add_and_del_clear_smb_state_whatever_the_vendor(self):
for action in ("add", "del"):
for vendor in ("PXEClient:Arch:00007:UNDI:003016",
"MSFT 5.0", None):
self.s.reset()
self.assert_smb_only(self.run_hook(action, vendor))
def test_old_only_for_network_boot(self):
self.s.reset()
calls = self.run_hook("old", "PXEClient:Arch:00007:UNDI:003016")
self.assert_smb_only(calls)
self.assertIn("network boot, PXEClient:Arch:00007",
[c for c in calls if c[0] == "logger"][0][-1])
for vendor in ("MSFT 5.0", None, ""):
self.s.reset()
self.assertEqual(self.run_hook("old", vendor), [], vendor)
def test_never_clears_all_traffic(self):
calls = self.run_hook("add", "PXEClient")
for c in calls:
if c[0] == "conntrack":
self.assertIn("-p", c)
self.assertIn("tcp", c)
if c[0] == "ss":
self.assertIn("sport = :445", c[-1])
def test_no_ip_does_nothing(self):
r = subprocess.run(["bash", HOOK, "add", "02:aa:00:00:00:05", ""],
env=self.s.env(), capture_output=True, timeout=10)
self.assertEqual(r.returncode, 0)
self.assertEqual(self.s.get_calls(), [])
class Carrier(unittest.TestCase):
def setUp(self):
self.s = Stubs()
self.s.write(os.path.join(self.s.dir, "imaging"), "pxe0\n")
self.s.link("pxe0", 1)
self.s.link("corp0", 1)
def tearDown(self):
self.s.cleanup()
def event(self, iface, state="routable"):
r = subprocess.run(["bash", CARRIER], env=self.s.env(
IFACE=iface, STATE=state), capture_output=True, text=True,
timeout=20)
self.assertEqual(r.returncode, 0, r.stderr)
return self.s.get_calls()
def gain(self):
r = subprocess.run(["bash", CARRIER, "--gain"], env=self.s.env(),
capture_output=True, text=True, timeout=20)
self.assertEqual(r.returncode, 0, r.stderr)
return self.s.get_calls()
def systemctl(self, calls):
return [c[1:] for c in calls if c[0] == "systemctl"]
def dnsmasq(self, state):
self.s.write(os.path.join(self.s.dir, "dnsmasq.state"), state + "\n")
def marker(self):
return os.path.exists(os.path.join(self.s.run_dir, "stopped-dnsmasq"))
def test_ge_nic_is_never_touched(self):
for carrier in (0, 1, None, 1):
self.s.reset()
self.s.link("corp0", carrier)
calls = self.event("corp0", "no-carrier")
self.assertEqual([c[0] for c in calls], ["pxe-nic-check"])
self.assertFalse(os.path.exists(os.path.join(self.s.run_dir,
"link.corp0")))
def test_unknown_imaging_nic_does_nothing(self):
self.s.write(os.path.join(self.s.dir, "imaging"), "")
calls = self.event("pxe0", "no-carrier")
self.assertEqual([c[0] for c in calls], ["pxe-nic-check"])
def test_loss_stops_running_dnsmasq_and_marks(self):
self.event("pxe0", "routable") # first sight: up
self.s.reset()
self.s.link("pxe0", 0)
sc = self.systemctl(self.event("pxe0", "no-carrier"))
self.assertIn(["stop", "pxe-nic-carrier.service"], sc)
self.assertIn(["stop", "--no-block", "dnsmasq"], sc)
self.assertLess(sc.index(["stop", "pxe-nic-carrier.service"]),
sc.index(["stop", "--no-block", "dnsmasq"]))
self.assertTrue(self.marker())
# same state again (off after no-carrier): idempotent, no calls
self.s.reset()
self.s.link("pxe0", None)
self.assertEqual(self.systemctl(self.event("pxe0", "off")), [])
def test_loss_with_dnsmasq_stopped_leaves_no_marker(self):
self.dnsmasq("inactive")
self.s.link("pxe0", 0)
sc = self.systemctl(self.event("pxe0", "no-carrier"))
self.assertNotIn(["stop", "--no-block", "dnsmasq"], sc)
self.assertFalse(self.marker())
def test_gain_hands_off_once(self):
self.s.link("pxe0", 0)
self.event("pxe0", "no-carrier")
self.s.reset()
self.s.link("pxe0", 1)
sc = self.systemctl(self.event("pxe0", "carrier"))
self.assertEqual(sc[-1], ["restart", "--no-block",
"pxe-nic-carrier.service"])
self.assertNotIn("dnsmasq", [x for c in sc for x in c])
self.s.reset() # routable right after carrier
self.assertEqual(self.systemctl(self.event("pxe0", "routable")), [])
def test_bridge_port_events_count(self):
self.s.write(os.path.join(self.s.dir, "imaging"), "br-pxe\npxe0\n")
self.s.link("br-pxe", 1)
self.event("pxe0", "enslaved")
self.s.reset()
self.s.link("pxe0", 0)
sc = self.systemctl(self.event("pxe0", "no-carrier"))
self.assertIn(["stop", "--no-block", "dnsmasq"], sc)
def test_gain_restarts_dnsmasq_through_its_guard(self):
os.makedirs(self.s.run_dir)
self.s.write(os.path.join(self.s.run_dir, "stopped-dnsmasq"), "")
self.dnsmasq("inactive") # the hook stopped it
sc = self.systemctl(self.gain())
self.assertIn(["restart", "dnsmasq"], sc)
self.assertFalse(self.marker())
def test_gain_restarts_running_dnsmasq(self):
# loss never seen (fast swap, boot): dnsmasq running, no marker
sc = self.systemctl(self.gain())
self.assertIn(["restart", "dnsmasq"], sc)
def test_gain_logs_crit_when_guard_refuses(self):
self.s.write(os.path.join(self.s.dir, "restart.rc"), "1\n")
calls = self.gain()
logs = [c for c in calls if c[0] == "logger"]
self.assertTrue(any("daemon.crit" in c and "NOT started" in c[-1]
for c in logs), logs)
def test_gain_leaves_operator_stopped_dnsmasq_alone(self):
self.dnsmasq("inactive")
calls = self.gain()
self.assertNotIn(["restart", "dnsmasq"], self.systemctl(calls))
self.assertNotIn(["start", "dnsmasq"], self.systemctl(calls))
self.assertIn(["pxe-nic-check", "--guard"],
[c for c in calls if c[0] == "pxe-nic-check"])
def test_gain_does_nothing_if_link_dropped_again(self):
self.s.link("pxe0", 0)
calls = self.gain()
self.assertEqual(self.systemctl(calls), [])
self.assertNotIn(["pxe-nic-check", "--guard"], calls)
if __name__ == "__main__":
unittest.main()

140
tests/pxe-nic-check/test_pxe_nic_check.py
View File

@@ -82,6 +82,21 @@ class Frames(unittest.TestCase):
o = fed(F.eapol(SW)) o = fed(F.eapol(SW))
self.assertTrue(o.eapol) self.assertTrue(o.eapol)
def test_eapol_only_from_authenticator(self):
client = "02:aa:00:00:00:07"
for name, fr, want in [
("request", F.eapol(SW, code=1), True),
("success", F.eapol(SW, code=3), True),
("failure", F.eapol(SW, code=4), True),
("response", F.eapol(client, code=2), False),
("start", F.eapol_start(client), False),
("logoff", F.eapol_start(client, eapol_type=2), False),
("truncated", F.eapol(SW)[:17], False),
]:
o = fed(fr)
self.assertEqual(o.eapol, want, name)
self.assertEqual(o.eapol_client, 0 if want else 1, name)
def test_stp(self): def test_stp(self):
o = fed(F.stp(SW)) o = fed(F.stp(SW))
self.assertTrue(o.stp) self.assertTrue(o.stp)
@@ -207,6 +222,113 @@ class Bpf(unittest.TestCase):
self.assertFalse(run_bpf(insns, fr), name) self.assertFalse(run_bpf(insns, fr), name)
class Multicast(unittest.TestCase):
def test_packet_mreq_packing(self):
import struct
raw = M.packet_mreq(7, "01:80:C2:00:00:0E")
self.assertEqual(len(raw), 16) # sizeof(struct packet_mreq)
idx, typ, alen, addr = struct.unpack("iHH8s", raw)
self.assertEqual((idx, typ, alen), (7, M.PACKET_MR_MULTICAST, 6))
self.assertEqual(addr, bytes.fromhex("0180c200000e") + b"\0\0")
self.assertEqual(M.SOL_PACKET, 263)
self.assertEqual(M.PACKET_ADD_MEMBERSHIP, 1)
with self.assertRaises(ValueError):
M.packet_mreq(1, "01:80:c2")
def test_groups(self):
self.assertEqual(sorted(M.CAPTURE_GROUPS), sorted([
"01:80:c2:00:00:00", "01:80:c2:00:00:03", "01:80:c2:00:00:0e",
"01:00:0c:cc:cc:cc", "01:00:0c:cc:cc:cd"]))
def test_join_groups_calls_setsockopt_per_group(self):
calls = []
class FakeSock:
def setsockopt(self, level, opt, val):
calls.append((level, opt, val))
if val[8:14] == bytes.fromhex("01000ccccccd"):
raise OSError("nope")
import socket
real = socket.if_nametoindex
socket.if_nametoindex = lambda name: 3
try:
failed = M.join_groups(FakeSock(), "eth9")
finally:
socket.if_nametoindex = real
self.assertEqual(failed, ["01:00:0c:cc:cc:cd"])
self.assertEqual(len(calls), 5)
for level, opt, val in calls:
self.assertEqual((level, opt), (263, 1))
self.assertEqual(val[:4], M.packet_mreq(3, "01:80:c2:00:00:00")[:4])
self.assertEqual(M.join_groups(FakeSock(), "no-such-if-xyz"),
M.CAPTURE_GROUPS)
class WatchRace(unittest.TestCase):
def test_parse_unit_state(self):
self.assertEqual(M.parse_unit_state(
"ActiveState=active\nActiveEnterTimestampMonotonic=123456\n"),
("active", 123456))
self.assertEqual(M.parse_unit_state(
"ActiveEnterTimestampMonotonic=0\nActiveState=inactive\n"),
("inactive", 0))
self.assertEqual(M.parse_unit_state(""), (None, None))
def test_decision(self):
start = 1000000
self.assertEqual(M.watch_decision("active", 500, start), "stop")
self.assertEqual(M.watch_decision("active", start + 1, start),
"recheck")
self.assertEqual(M.watch_decision("activating", 0, start), "recheck")
self.assertEqual(M.watch_decision("inactive", 0, start), "skip")
self.assertEqual(M.watch_decision("failed", 0, start), "skip")
self.assertEqual(M.watch_decision(None, None, start), "stop")
self.assertEqual(M.watch_decision("active", None, start), "stop")
def _main_watch(self, state_out, recheck_state):
d = tempfile.mkdtemp()
marker = os.path.join(d, "stopped")
saved = M.run_check, M.recheck
blocked = {"checked_at": "x", "duration_s": 1, "interfaces": [],
"dhcp_guard": {"state": "blocked", "reason": "LLDP"}}
rechecked = []
M.run_check = lambda args, cfg: (blocked, None)
def fake_recheck(args, cfg):
rechecked.append(1)
return recheck_state, "re-check"
M.recheck = fake_recheck
try:
rc = M.main(["--watch", "--config", "/nonexistent",
"--write", os.path.join(d, "s.json"),
"--stop-cmd", "touch " + marker,
"--state-cmd", "printf '%s'" % state_out])
finally:
M.run_check, M.recheck = saved
self.assertEqual(rc, 0)
return os.path.exists(marker), bool(rechecked)
def test_watch_stops_when_dnsmasq_predates_window(self):
self.assertEqual(self._main_watch(
"ActiveState=active\nActiveEnterTimestampMonotonic=1\n",
"allowed"), (True, False))
def test_watch_rechecks_after_operator_restart(self):
late = "ActiveState=active\nActiveEnterTimestampMonotonic=%d\n" \
% (2 ** 62)
self.assertEqual(self._main_watch(late, "allowed"), (False, True))
self.assertEqual(self._main_watch(late, "blocked"), (True, True))
def test_watch_skips_stopped_dnsmasq(self):
self.assertEqual(self._main_watch(
"ActiveState=inactive\nActiveEnterTimestampMonotonic=0\n",
"allowed"), (False, False))
def test_watch_stops_when_state_unknown(self):
self.assertEqual(self._main_watch("", "allowed"), (True, False))
class OwnServer(unittest.TestCase): class OwnServer(unittest.TestCase):
def test_own_needs_own_ip_and_local_mac(self): def test_own_needs_own_ip_and_local_mac(self):
o = M.Observations() o = M.Observations()
@@ -284,6 +406,24 @@ class Verdicts(unittest.TestCase):
M.allowed_list("IMG-SW1")) M.allowed_list("IMG-SW1"))
self.assertEqual(v, "ge") self.assertEqual(v, "ge")
def test_client_eapol_stays_isolated(self):
# a bay PXE-booting next to us: iPXE sends 3 EAPOL-Starts on every
# NIC open, Windows dot3svc sends Starts and EAP-Responses
client = "02:aa:00:00:00:07"
starts = [F.eapol_start(client)] * 3
v, ev = self.run_case(starts)
self.assertEqual(v, "isolated", ev)
self.assertTrue(any("3 client EAPOL frames ignored" in e
for e in ev), ev)
v, ev = self.run_case([F.eapol(client, code=2)])
self.assertEqual(v, "isolated", ev)
v, ev = self.run_case(starts + [F.eapol(client, code=2)])
self.assertEqual(v, "isolated", ev)
# a switch port asking for identity is still decisive
v, ev = self.run_case(starts + [F.eapol(SW, code=1)])
self.assertEqual(v, "ge", ev)
self.assertIn("802.1X EAPOL seen (port authentication)", ev)
def test_no_link_and_error(self): def test_no_link_and_error(self):
self.assertEqual(self.run_case([], link=False)[0], "no-link") self.assertEqual(self.run_case([], link=False)[0], "no-link")
self.assertEqual(self.run_case([], error="boom")[0], "error") self.assertEqual(self.run_case([], error="boom")[0], "error")