diff --git a/docs/pxe-nic-guard.md b/docs/pxe-nic-guard.md index 916c595..b5dfcb8 100644 --- a/docs/pxe-nic-guard.md +++ b/docs/pxe-nic-guard.md @@ -22,6 +22,9 @@ The NIC guard catches those. | `/etc/pxe-server/nic.conf` | which NIC is which | | `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start| | `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min | +| `/usr/local/sbin/pxe-nic-carrier` | re-check on imaging cable moves | +| `/etc/networkd-dispatcher/.d/50-pxe-nic-carrier` | calls it on link changes | +| `/etc/systemd/system/pxe-nic-carrier.service` | the re-check after carrier gain | | `/run/pxe-nic-status.json` | last result, read by the webapp | Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml` @@ -33,11 +36,18 @@ from the first two). The same MACs pin the NICs in netplan ## How it decides -It listens on each NIC (raw socket, sends nothing) for: +It listens on each NIC (raw socket, sends nothing; it joins the STP, +802.1X, LLDP, CDP and PVST+ multicast groups so a NIC that is not +promiscuous still hears them) for: - LLDP and CDP: managed switches announce themselves. GE switches do; the unmanaged imaging switch does not. -- 802.1X EAPOL: GE ports ask for port authentication. +- 802.1X EAPOL from an authenticator: GE ports ask for port + authentication (EAP Request, Success or Failure). EAPOL that clients + send is ignored: iPXE sends three EAPOL-Starts every time it opens a + NIC and Windows sends Starts and EAP Responses, so a bay PXE-booting on + the imaging switch is not mistaken for a GE port. Those frames show up + only as "N client EAPOL frames ignored". - DHCP OFFER/ACK from any other server. - STP and the count of distinct ARP hosts: reported, never decisive. @@ -70,7 +80,19 @@ never allowed. about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no link exits 0 (fail open) with an error line in the journal. - `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs - `systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq. + `systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq. If + dnsmasq was (re)started during the 60 s window (an operator, or the + carrier hook, restarted it and its guard passed), the watch re-checks + for about 6 s and stops dnsmasq only if that still says blocked; if + dnsmasq is already stopped it only logs. +- `pxe-nic-carrier` - the networkd-dispatcher hook. Only the imaging NIC + (or its bridge ports) counts; GE NIC events are ignored. Carrier lost: + dnsmasq is stopped at once if it was running. Carrier back: after 2 s, + `pxe-nic-carrier.service` restarts dnsmasq if it was running or the hook + stopped it, so the guard checks the new network before dnsmasq serves + again; dnsmasq that an operator or the watch stopped stays stopped. + A bridge with several ports stops dnsmasq when any one of them loses + carrier. Log: `journalctl -t pxe-nic-carrier`. Every start or restart of dnsmasq runs the guard, so `systemctl restart dnsmasq` and the `@reboot` cron restart take about 6 s longer than before. @@ -81,6 +103,7 @@ dnsmasq` and the `@reboot` cron restart take about 6 s longer than before. cat /run/pxe-nic-status.json # last guard/watch result journalctl -u dnsmasq -b | grep pxe-nic-check journalctl -u pxe-nic-check -b + journalctl -t pxe-nic-carrier -b # cable unplug / replug ## dnsmasq will not start @@ -94,7 +117,7 @@ dnsmasq` and the `@reboot` cron restart take about 6 s longer than before. ## Floor tests (run with the floor idle) -Both tests stop imaging for a few minutes. Have console or GE-side SSH +All three tests stop imaging for a few minutes. Have console or GE-side SSH access to the server; do not rely on the imaging network for the session. ### (a) Rogue DHCP on the imaging switch: expect the alarm @@ -126,6 +149,10 @@ dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE. dnsmasq; the guard would still block it, but do not test that way). 2. Move the IMAGING NIC's cable from the imaging switch into a live GE network port. Wait for link: `ip -br link` shows the imaging NIC `UP`. + The carrier hook sees the link come up and runs the guard once (one + DHCPDISCOVER) to refresh the status file; with dnsmasq stopped it does + not start it. `journalctl -t pxe-nic-carrier -n 5` shows `dnsmasq was + not running, left stopped`. 3. Run the guard by hand first: `sudo pxe-nic-check --guard; echo "exit $?"` Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP, @@ -142,6 +169,17 @@ dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE. shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq` starts. PXE-boot one client to confirm. +### (c) Imaging cable unplugged and replugged: expect stop, re-check, start + +1. dnsmasq running, imaging NIC `isolated`. +2. Unplug the imaging NIC's cable. Within a second + `journalctl -t pxe-nic-carrier -n 5` shows `carrier lost on imaging NIC + ... stopped dnsmasq` and `systemctl is-active dnsmasq` is `inactive`. +3. Plug it back into the imaging switch. About 10 s later the journal + shows `guard passed, dnsmasq (re)started` and dnsmasq is `active`. + Had it gone into a GE port instead, the guard refuses: the journal + shows `dnsmasq NOT started` and dnsmasq stays down. + ## Status file `/run/pxe-nic-status.json`, mode 0644, replaced atomically: diff --git a/playbook/ipxe/GetPxeScript.aspx b/playbook/ipxe/GetPxeScript.aspx index 34f2299..a3d380c 100644 --- a/playbook/ipxe/GetPxeScript.aspx +++ b/playbook/ipxe/GetPxeScript.aspx @@ -37,13 +37,23 @@ initrd http://${server}/win11/EFI/Microsoft/Boot/BCD EFI/Microsoft/Boot/BC initrd http://${server}/win11/EFI/Boot/bootx64.efi EFI/Boot/bootx64.efi initrd http://${server}/win11/Boot/boot.sdi Boot/boot.sdi initrd http://${server}/win11/sources/boot.wim sources/boot.wim -boot +# A command that fails without || aborts the whole iPXE script (the PC +# drops to firmware with no explanation), so every boot line has one. +boot || goto winpe_failed + +:winpe_failed +echo +echo Windows PE boot FAILED (see the error above). +echo Check that the PXE server is up and the win11 boot files are staged. +prompt Press any key to return to menu... && goto menu || goto menu :clonezilla set base http://${server}/clonezilla kernel ${base}/vmlinuz boot=live username=user union=overlay config components noswap edd=on nomodeset nodmraid locales= keyboard-layouts= ocs_live_run="ocs-live-general" ocs_live_extra_param="" ocs_live_batch=no net.ifnames=0 nosplash noprompt fetch=${base}/filesystem.squashfs || goto secureboot_warn -initrd ${base}/initrd.img -boot +initrd ${base}/initrd.img || goto boot_failed +# iPXE 2.0 accepts an unsigned EFI kernel (PE fallback) and fails only +# here under Secure Boot. +boot || goto secureboot_warn :blancco iseq ${platform} pcbios && goto blancco_bios || goto blancco_efi @@ -55,7 +65,7 @@ initrd --name intel-ucode.img http://${server}/blancco/intel-ucode.img || goto b initrd --name amd-ucode.img http://${server}/blancco/amd-ucode.img || goto blancco_bios_failed initrd --name config.img http://${server}/blancco/config.img || goto blancco_bios_failed initrd --name initramfs-bde-linux.img http://${server}/blancco/initramfs-bde-linux.img || goto blancco_bios_failed -boot +boot || goto blancco_bios_failed :blancco_bios_failed echo echo Blancco BIOS boot FAILED. @@ -91,7 +101,14 @@ chain http://${server}/blancco/grubx64.efi || goto secureboot_warn :memtest kernel http://${server}/memtest/memtest.efi || goto secureboot_warn -boot +# iPXE 2.0 accepts an unsigned EFI image (PE fallback) and fails only +# here under Secure Boot. +boot || goto secureboot_warn + +:boot_failed +echo +echo Download FAILED (see the error above). Check the PXE server and retry. +prompt Press any key to return to menu... && goto menu || goto menu :secureboot_warn echo diff --git a/playbook/pxe-server-helpers/pxe-dhcp-hook.sh b/playbook/pxe-server-helpers/pxe-dhcp-hook.sh index 6eb72da..917819d 100755 --- a/playbook/pxe-server-helpers/pxe-dhcp-hook.sh +++ b/playbook/pxe-server-helpers/pxe-dhcp-hook.sh @@ -3,10 +3,15 @@ # pxe-dhcp-hook.sh - dnsmasq dhcp-script hook. # # Runs every time a PXE client gets/changes/releases a DHCP lease on -# 172.16.9.0/24. Flushes conntrack entries and drops any lingering -# TCP sockets for that client IP. Prevents stale server-side state from -# causing "System error 53 - network path not found" when a WinPE client -# re-images the same machine without a clean SMB session teardown. +# 172.16.9.0/24. Flushes that client's SMB (TCP 445 and 139) conntrack +# entries and drops its lingering SMB sockets. Prevents stale server-side +# state from causing "System error 53 - network path not found" when a +# WinPE client re-images the same machine without a clean SMB session +# teardown. +# +# SMB only, on purpose: the lease that fires this hook is iPXE's, and its +# first HTTP fetch of the boot menu is racing this script. Killing every +# socket or conntrack entry for the client IP broke that fetch. # # dnsmasq calls this with argv: # $1 = action (add | old | del) @@ -26,6 +31,9 @@ set +e +# dnsmasq's environment may lack sbin; keep any PATH it gave us first. +PATH="${PATH:+$PATH:}/usr/sbin:/usr/bin:/sbin:/bin" + action="$1" mac="$2" ip="$3" @@ -34,11 +42,15 @@ host="${4:-}" [ -z "$ip" ] && exit 0 clear_state() { - /usr/sbin/conntrack -D -s "$ip" >/dev/null 2>&1 - /usr/sbin/conntrack -D -d "$ip" >/dev/null 2>&1 - /usr/bin/ss -K "dst = $ip" >/dev/null 2>&1 - /usr/bin/ss -K "src = $ip" >/dev/null 2>&1 - logger -t pxe-dhcp-hook "cleared stale state for $1 $ip ($mac${host:+ $host})" + local port + for port in 445 139; do + # client -> our share, and anything the other way round + conntrack -D -p tcp -s "$ip" --dport "$port" >/dev/null 2>&1 + conntrack -D -p tcp -d "$ip" --sport "$port" >/dev/null 2>&1 + done + # our SMB sockets (local port 445/139) whose peer is this client + ss -K -t "( sport = :445 or sport = :139 ) and dst = $ip" >/dev/null 2>&1 + logger -t pxe-dhcp-hook "cleared stale SMB state for $1 $ip ($mac${host:+ $host})" } case "$action" in diff --git a/playbook/pxe-server-helpers/pxe-nic-carrier b/playbook/pxe-server-helpers/pxe-nic-carrier new file mode 100755 index 0000000..8e810fc --- /dev/null +++ b/playbook/pxe-server-helpers/pxe-nic-carrier @@ -0,0 +1,145 @@ +#!/bin/bash +# +# pxe-nic-carrier - re-check the imaging NIC whenever its cable moves. +# +# The dnsmasq guard (pxe-nic-check --guard) runs only when dnsmasq starts, +# and the watch timer only every 5 min. If the imaging cable is moved into +# a GE port while dnsmasq runs, dnsmasq would answer DHCP on the GE LAN +# until the next watch. This hook closes that gap: +# +# carrier lost dnsmasq is stopped at once (if it was running) and a +# marker remembers that this hook stopped it. +# carrier back pxe-nic-carrier.service runs "pxe-nic-carrier --gain": +# after a 2 s settle, if dnsmasq is running or the marker +# is set, "systemctl restart dnsmasq" - its ExecStartPre +# guard checks the NIC while dnsmasq is down and refuses +# to start it on the GE network. Otherwise it only runs +# the guard to refresh /run/pxe-nic-status.json. dnsmasq +# that an operator or the watch stopped stays stopped. +# +# Only the imaging NIC (and, for a bridge, its ports) counts, as resolved +# by "pxe-nic-check --print-imaging" from /etc/pxe-server/nic.conf or the +# dnsmasq interface= pin. Events on any other NIC (the GE NIC) exit at +# once and touch nothing. Link state is read from sysfs, not from the +# event name, and remembered per interface, so the same transition seen +# twice (carrier, then routable) acts once. +# +# Called by networkd-dispatcher with IFACE and STATE in the environment +# (installed as /etc/networkd-dispatcher/.d/50-pxe-nic-carrier for +# carrier, degraded, degraded-carrier, enslaved, routable, no-carrier and +# off), or by hand: pxe-nic-carrier IFACE. Logs: journalctl -t pxe-nic-carrier +# +# The dispatcher waits for this script, so the event path never runs a +# check itself: loss is two systemctl calls, gain hands off to the service. + +set -u + +CHECK="${PXE_NIC_CHECK:-/usr/local/sbin/pxe-nic-check}" +SYS_NET="${PXE_SYS_NET:-/sys/class/net}" +RUN="${PXE_NIC_CARRIER_RUN:-/run/pxe-nic-carrier}" +SYSTEMCTL="${PXE_SYSTEMCTL:-systemctl}" +SETTLE="${PXE_NIC_CARRIER_SETTLE:-2}" +GAIN_UNIT="pxe-nic-carrier.service" +TAG="pxe-nic-carrier" + +PATH="${PATH:+$PATH:}/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin" + +log() { # log PRIORITY MESSAGE + logger -t "$TAG" -p "daemon.$1" -- "$2" 2>/dev/null || \ + echo "$TAG: $2" >&2 +} + +imaging_ifaces() { + "$CHECK" --print-imaging 2>/dev/null +} + +carrier_of() { # 1 if the interface has carrier, else 0 (down reads EINVAL) + local c + c=$(cat "$SYS_NET/$1/carrier" 2>/dev/null) || c=0 + [ "$c" = 1 ] && echo 1 || echo 0 +} + +any_carrier() { # any imaging interface or port with carrier + local i + for i in "$@"; do + [ "$(carrier_of "$i")" = 1 ] && return 0 + done + return 1 +} + +dnsmasq_running() { # active, starting or reloading + case "$("$SYSTEMCTL" is-active dnsmasq 2>/dev/null)" in + active|activating|reloading) return 0 ;; + esac + return 1 +} + +on_event() { + local iface="$1" list=() match="" i now prev + mapfile -t list < <(imaging_ifaces) + for i in "${list[@]}"; do + [ -n "$i" ] && [ "$i" = "$iface" ] && match=1 + done + [ -n "$match" ] || exit 0 # not the imaging NIC: hands off + + mkdir -p "$RUN" || exit 0 + exec 9>"$RUN/event.lock" + flock -w 10 9 || log warning "lock busy, handling $iface anyway" + + now=$(carrier_of "$iface") + prev=$(cat "$RUN/link.$iface" 2>/dev/null) || prev=unknown + echo "$now" > "$RUN/link.$iface" + [ "$now" = "$prev" ] && exit 0 # same state again: already handled + + if [ "$now" = 0 ]; then + # an in-flight gain check must not restart dnsmasq after this + "$SYSTEMCTL" stop "$GAIN_UNIT" >/dev/null 2>&1 + if dnsmasq_running; then + : > "$RUN/stopped-dnsmasq" + "$SYSTEMCTL" stop --no-block dnsmasq + log warning "carrier lost on imaging NIC $iface (${STATE:-event}): stopped dnsmasq; it restarts after the guard passes when the link returns" + else + log info "carrier lost on imaging NIC $iface (${STATE:-event}); dnsmasq not running" + fi + else + log notice "carrier up on imaging NIC $iface (${STATE:-event}, was $prev): re-checking before dnsmasq serves" + "$SYSTEMCTL" restart --no-block "$GAIN_UNIT" + fi + exit 0 +} + +on_gain() { + local list=() rc + sleep "$SETTLE" + mapfile -t list < <(imaging_ifaces) + if [ "${#list[@]}" -eq 0 ] || [ -z "${list[0]}" ]; then + log err "imaging NIC unknown, nothing to re-check" + exit 0 + fi + if ! any_carrier "${list[@]}"; then + log info "imaging NIC ${list[0]} lost carrier again, nothing to do" + exit 0 + fi + if dnsmasq_running || [ -e "$RUN/stopped-dnsmasq" ]; then + # restart = stop, then the ExecStartPre guard with dnsmasq down + "$SYSTEMCTL" restart dnsmasq + rc=$? + rm -f "$RUN/stopped-dnsmasq" + if [ "$rc" -eq 0 ]; then + log notice "imaging NIC ${list[0]}: guard passed, dnsmasq (re)started" + else + log crit "imaging NIC ${list[0]}: dnsmasq NOT started (guard refused or start failed, rc $rc); see journalctl -u dnsmasq. Fix the cabling, then systemctl start dnsmasq" + fi + else + "$CHECK" --guard >/dev/null 2>&1 + rc=$? + log info "imaging NIC ${list[0]}: dnsmasq was not running, left stopped (guard rc $rc, status refreshed)" + fi + exit 0 +} + +case "${1:-}" in + --gain) on_gain ;; + -h|--help) sed -n '2,36p' "$0"; exit 0 ;; + *) on_event "${1:-${IFACE:-}}" ;; +esac diff --git a/playbook/pxe-server-helpers/pxe-nic-carrier.service b/playbook/pxe-server-helpers/pxe-nic-carrier.service new file mode 100644 index 0000000..728fa49 --- /dev/null +++ b/playbook/pxe-server-helpers/pxe-nic-carrier.service @@ -0,0 +1,16 @@ +# Started (never enabled) by pxe-nic-carrier, the networkd-dispatcher +# hook, when the imaging NIC gets carrier back. Re-checks the imaging NIC +# before dnsmasq serves again: if dnsmasq was running (or the hook stopped +# it on carrier loss), "systemctl restart dnsmasq" so its ExecStartPre +# guard decides; otherwise it only refreshes /run/pxe-nic-status.json. +# The hook restarts this unit on every carrier gain and stops it on a +# carrier loss, so the newest event wins. + +[Unit] +Description=PXE NIC carrier change: re-check the imaging NIC before dnsmasq serves +After=network.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/pxe-nic-carrier --gain +TimeoutStartSec=120 diff --git a/playbook/pxe-server-helpers/pxe-nic-check b/playbook/pxe-server-helpers/pxe-nic-check index 964901e..0cb8469 100755 --- a/playbook/pxe-server-helpers/pxe-nic-check +++ b/playbook/pxe-server-helpers/pxe-nic-check @@ -32,7 +32,9 @@ Verdicts per interface: no-link no carrier isolated no foreign DHCP server, no LLDP/CDP/EAPOL seen ge foreign DHCP with a corp domain (opt 15 / 119), or any - LLDP, CDP or EAPOL seen (unmanaged switches send none) + LLDP, CDP or authenticator EAPOL seen (unmanaged switches + send none). EAPOL from clients (iPXE and Windows send + EAPOL-Start and EAP-Response) is ignored. foreign-dhcp a non-own DHCP server without a corp domain error the check itself failed @@ -46,9 +48,14 @@ Modes: exit 0 = safe to serve, exit 1 = DO NOT serve (ge or foreign-dhcp). Internal errors and no link exit 0 with a loud log line: fail-open on bugs, fail-closed only on detection. + --print-imaging + the imaging NIC (and its bridge ports), for pxe-nic-carrier. --watch systemd timer. 60 s window + probe, writes the status file, and on a bad imaging verdict runs "systemctl stop dnsmasq" - and logs CRIT. It never starts dnsmasq. + and logs CRIT. It never starts dnsmasq. If dnsmasq was + (re)started inside the window (its guard passed after part + of the evidence was heard), it re-checks for ~6 s first and + stops dnsmasq only if that still says blocked. Config: /etc/pxe-server/nic.conf (INI, [nic] section optional) @@ -96,6 +103,8 @@ STATUS_PATH = "/run/pxe-nic-status.json" DEFAULT_SUFFIXES = "geaerospace.net, ge.com" DEFAULT_OWN_IP = "172.16.9.1" STOP_CMD = "systemctl stop dnsmasq" +STATE_CMD = ("systemctl show -p ActiveState " + "-p ActiveEnterTimestampMonotonic dnsmasq") ETH_P_ALL = 0x0003 PACKET_OUTGOING = 4 @@ -329,6 +338,23 @@ def parse_bootp(udp_payload): } +EAPOL_TYPE_EAP = 0 +EAP_AUTHENTICATOR_CODES = (1, 3, 4) # Request, Success, Failure + + +def eapol_from_authenticator(payload): + """True only for EAPOL that an 802.1X authenticator (a switch port) + sends: EAPOL type 0 (EAP-Packet) carrying EAP code 1 Request, 3 Success + or 4 Failure. Supplicants send EAPOL-Start (type 1), Logoff (type 2) + and EAP Responses (code 2): iPXE sends three EAPOL-Starts every time + it opens a NIC, and Windows dot3svc sends Starts and Responses, so a + bay PXE-booting next to us must not read as a GE switch port.""" + if len(payload) < 5: + return False + return payload[1] == EAPOL_TYPE_EAP \ + and payload[4] in EAP_AUTHENTICATOR_CODES + + def parse_dhcp_frame(etype, payload): """IPv4/UDP 67->68 server message -> bootp dict (+ src ip) or None.""" if etype != ETH_IP or len(payload) < 20: @@ -352,7 +378,8 @@ class Observations: def __init__(self): self.lldp = {} # (chassis, port) -> dict self.cdp = {} # (device, port) -> dict - self.eapol = False + self.eapol = False # EAPOL from an authenticator (a switch) + self.eapol_client = 0 # supplicant EAPOL frames, ignored self.stp = False self.arp_macs = set() self.dhcp = {} # (server, src_mac) -> dict @@ -371,7 +398,10 @@ class Observations: self.lldp[(info["chassis"], info["port"])] = info return if etype == ETH_EAPOL: - self.eapol = True + if eapol_from_authenticator(payload): + self.eapol = True + else: + self.eapol_client += 1 return if etype <= 1500: # 802.3 length field: LLC frames if dst == MAC_CDP: @@ -763,6 +793,45 @@ def attach_bpf(sock): # ---------------------------------------------------------------- capture +SOL_PACKET = 263 +PACKET_ADD_MEMBERSHIP = 1 +PACKET_MR_MULTICAST = 0 + +# Link-local groups the NIC's hardware filter drops unless someone joins +# them: STP, EAPOL (PAE), LLDP, CDP/VTP/DTP, PVST+. Without them a plain +# (non-promiscuous) NIC on a CDP-only Cisco port hears nothing. +CAPTURE_GROUPS = ["01:80:c2:00:00:00", "01:80:c2:00:00:03", + "01:80:c2:00:00:0e", "01:00:0c:cc:cc:cc", + "01:00:0c:cc:cc:cd"] + + +def packet_mreq(ifindex, mac): + """struct packet_mreq {int mr_ifindex; unsigned short mr_type, + mr_alen; unsigned char mr_address[8];} for PACKET_MR_MULTICAST.""" + addr = bytes.fromhex(norm_mac(mac).replace(":", "")) + if len(addr) != 6: + raise ValueError("bad MAC %r" % mac) + return struct.pack("iHH8s", ifindex, PACKET_MR_MULTICAST, len(addr), + addr) + + +def join_groups(sock, ifname, groups=CAPTURE_GROUPS): + """Join each group on this socket's interface. The kernel drops the + memberships when the socket closes. Returns the groups that failed.""" + failed = [] + try: + ifindex = socket.if_nametoindex(ifname) + except OSError: + return list(groups) + for g in groups: + try: + sock.setsockopt(SOL_PACKET, PACKET_ADD_MEMBERSHIP, + packet_mreq(ifindex, g)) + except (OSError, ValueError): + failed.append(g) + return failed + + def capture(targets, window, probe_ifaces, own_macs, probe_window=6.0): """targets: {report_iface: [capture ifaces]}. Listens on all of them for `window` s (at least probe_window after the probe). Returns @@ -783,6 +852,10 @@ def capture(targets, window, probe_ifaces, own_macs, probe_window=6.0): log("no BPF filter on %s (%s), parsing every frame" % (cap, e), "warning") s.bind((cap, ETH_P_ALL)) + bad = join_groups(s, cap) + if bad: + log("could not join %s on %s; a quiet switch may be " + "missed" % (", ".join(bad), cap), "warning") s.setblocking(False) socks[s] = name except OSError as e: @@ -883,6 +956,10 @@ def classify(link_up, error, obs, dhcp_servers, suffixes, window, evidence.append(txt) if obs and obs.eapol: evidence.append("802.1X EAPOL seen (port authentication)") + if obs and obs.eapol_client: + evidence.append("%d client EAPOL frames ignored (supplicants such " + "as iPXE EAPOL-Start, informational)" + % obs.eapol_client) if obs and obs.stp: evidence.append("STP BPDUs seen (informational)") @@ -1135,6 +1212,81 @@ def summary(status): return "\n".join(lines) +def parse_unit_state(text): + """`systemctl show -p ActiveState -p ActiveEnterTimestampMonotonic` + output -> (active_state or None, active_enter_us or None).""" + props = {} + for line in (text or "").splitlines(): + if "=" in line: + k, v = line.split("=", 1) + props[k.strip()] = v.strip() + state = props.get("ActiveState") or None + try: + since = int(props.get("ActiveEnterTimestampMonotonic", "")) + except ValueError: + since = None + return state, since + + +def dnsmasq_state(cmd): + try: + res = subprocess.run(cmd, shell=True, capture_output=True, + text=True, timeout=10) + except (OSError, subprocess.SubprocessError): + return None, None + if res.returncode != 0: + return None, None + return parse_unit_state(res.stdout) + + +def watch_decision(state, since_us, window_start_us): + """What --watch does with a 'blocked' verdict, given dnsmasq's + ActiveState / ActiveEnterTimestampMonotonic and the CLOCK_MONOTONIC + microsecond the watch's capture began. -> "stop", "skip" (already + stopped) or "recheck" (dnsmasq was (re)started inside the window, so + its own guard passed after part of our evidence was heard; confirm + with a fresh short check before stopping it).""" + if state is None: + return "stop" # cannot tell: stopping is the safe side + if state in ("inactive", "failed", "deactivating"): + return "skip" + if state in ("activating", "reloading"): + return "recheck" # its guard is running right now + if since_us and since_us > window_start_us: + return "recheck" + return "stop" + + +def stop_dnsmasq(cmd): + try: + rc = subprocess.run(cmd, shell=True, timeout=60).returncode + if rc != 0: + log("'%s' exited %d" % (cmd, rc), "crit") + except (OSError, subprocess.SubprocessError) as e: + log("could not stop dnsmasq: %s" % e, "crit") + + +def recheck(args, cfg): + """A guard-style check (imaging NIC, short window + probe), written to + the status file. -> (state, reason). An internal error counts as + blocked: the 60 s watch already saw the bad evidence.""" + sub = argparse.Namespace(**vars(args)) + sub.mode = "guard" + sub.window = 5.0 + sub.probe = True + try: + status, _ = run_check(sub, cfg) + except Exception as e: + return "blocked", "re-check failed (%s); keeping the watch verdict" \ + % e + if args.write: + try: + write_status(args.write, status) + except OSError as e: + log("could not write %s: %s" % (args.write, e), "err") + return status["dhcp_guard"]["state"], status["dhcp_guard"]["reason"] + + def main(argv=None): ap = argparse.ArgumentParser( prog=PROG, @@ -1149,6 +1301,10 @@ def main(argv=None): mode.add_argument("--watch", dest="mode", action="store_const", const="watch", help="timer mode: stop dnsmasq on a " "bad imaging verdict") + mode.add_argument("--print-imaging", dest="mode", action="store_const", + const="print-imaging", help="print the imaging NIC " + "and, for a bridge, its ports (one per line; nothing " + "if unknown), then exit. Used by pxe-nic-carrier.") ap.add_argument("--json", action="store_true", help="print the status JSON") ap.add_argument("--write", metavar="PATH", nargs="?", const=STATUS_PATH, @@ -1171,6 +1327,9 @@ def main(argv=None): ap.add_argument("--lease-dir", default=LEASE_DIR) ap.add_argument("--stop-cmd", default=STOP_CMD, help="command --watch runs to stop dnsmasq") + ap.add_argument("--state-cmd", default=STATE_CMD, + help="command --watch runs to read dnsmasq's " + "ActiveState and ActiveEnterTimestampMonotonic") args = ap.parse_args(argv) args.mode = args.mode or "check" if args.window is None: @@ -1180,6 +1339,22 @@ def main(argv=None): if args.mode in ("guard", "watch") and not args.write: args.write = STATUS_PATH + if args.mode == "print-imaging": + try: + cfg = load_config(args.config, args.dnsmasq_nic_conf) + ifaces = all_ifaces() + imaging, _ge, _notes = resolve_roles(cfg, ifaces, iface_addrs(), + args.lease_dir) + except Exception as e: + log("could not resolve the imaging NIC: %s" % e, "err") + return 2 + if imaging: + print("\n".join([imaging] + (bridge_ports(imaging) + if is_bridge(imaging) else []))) + return 0 + + # CLOCK_MONOTONIC, the clock of ActiveEnterTimestampMonotonic + window_start_us = int(time.clock_gettime(time.CLOCK_MONOTONIC) * 1e6) try: cfg = load_config(args.config, args.dnsmasq_nic_conf) status, imaging = run_check(args, cfg) @@ -1217,15 +1392,25 @@ def main(argv=None): return 0 if args.mode == "watch": if state == "blocked": - log("imaging NIC is NOT isolated, stopping dnsmasq: " + reason, - "crit") - try: - rc = subprocess.run(args.stop_cmd, shell=True, - timeout=60).returncode - if rc != 0: - log("'%s' exited %d" % (args.stop_cmd, rc), "crit") - except (OSError, subprocess.SubprocessError) as e: - log("could not stop dnsmasq: %s" % e, "crit") + dstate, since = dnsmasq_state(args.state_cmd) + what = watch_decision(dstate, since, window_start_us) + if what == "recheck": + log("dnsmasq was (re)started during this watch window " + "(its guard passed); re-checking before stopping it: " + + reason, "warning") + state, reason = recheck(args, cfg) + if state == "blocked": + what = "stop" + else: + log("re-check says %s, leaving dnsmasq running: %s" + % (state, reason), "notice") + if what == "skip": + log("imaging NIC is NOT isolated, dnsmasq already %s: %s" + % (dstate, reason), "crit") + elif what == "stop": + log("imaging NIC is NOT isolated, stopping dnsmasq: " + + reason, "crit") + stop_dnsmasq(args.stop_cmd) elif state == "unknown": log("watch could not verify the imaging NIC: " + reason, "err") else: diff --git a/playbook/pxe-server-helpers/pxe-nic-check.service b/playbook/pxe-server-helpers/pxe-nic-check.service index dc22d7b..d9464be 100644 --- a/playbook/pxe-server-helpers/pxe-nic-check.service +++ b/playbook/pxe-server-helpers/pxe-nic-check.service @@ -3,7 +3,8 @@ # /run/pxe-nic-status.json for the webapp, and STOPS dnsmasq if the imaging # NIC turns out to be on the GE network or next to another DHCP server. # It never starts dnsmasq: after fixing the cabling, start it by hand -# (systemctl start dnsmasq), which re-runs the guard. +# (systemctl start dnsmasq), which re-runs the guard. If dnsmasq was +# restarted inside the window it re-checks (~6 s) before stopping it. [Unit] Description=PXE NIC watch: keep dnsmasq off the GE corporate network diff --git a/tests/pxe-nic-check/README.md b/tests/pxe-nic-check/README.md index 8d020c8..fe6f3fa 100644 --- a/tests/pxe-nic-check/README.md +++ b/tests/pxe-nic-check/README.md @@ -12,6 +12,15 @@ Synthesized LLDP, CDP, EAPOL, STP and ARP frames and DHCPOFFER/ACK packets cloned-PXE-server handling; networkd lease parsing; config fallbacks; the dnsmasq_bound logic; the status JSON shape and atomic 0644 write. +## Shell helpers + +`test_helpers.py` runs `pxe-dhcp-hook.sh` (add/del/old with PXEClient, +MSFT 5.0 and no vendor class: only SMB 445/139 conntrack entries and +sockets are cleared) and `pxe-nic-carrier` (loss stops dnsmasq, gain +hands off once, GE NIC events touch nothing, bridge ports count) with +stub `ss`, `conntrack`, `logger`, `systemctl` and `pxe-nic-check` on a +fake PATH and a fake `/sys/class/net`. Part of the unit run above. + ## Namespace lab python3 tests/pxe-nic-check/lab.py # all scenarios, ~5 min @@ -38,6 +47,7 @@ namespace cannot drop privileges; scenarios are skipped without it). | not-corp-domain | DHCP domain notgeaerospace.net | foreign-dhcp | 1 | | lldp-only | LLDP | ge | 1 | | eapol-only | 802.1X EAPOL | ge | 1 | +| eapol-client | EAPOL-Start + EAP-Response, like a PXE-booting bay | isolated | 0 | | own-server | DHCP with server id 172.16.9.1 from a local MAC | isolated | 0 | | cloned-pxe | DHCP with server id 172.16.9.1, foreign MAC | foreign-dhcp | 1 | | no-link | peer end down | no-link | 0 | @@ -46,5 +56,9 @@ namespace cannot drop privileges; scenarios are skipped without it). | own-dnsmasq | real dnsmasq serving the imaging NIC itself | isolated, dnsmasq_bound true | 0 | | br-* | same, imaging NIC is a bridge (like br-pxe), named in nic.conf by a port's MAC | | | +`rogue`, `lldp-only` and `br-lldp` also run `--watch` as if dnsmasq had +been restarted inside the window: it must log the re-check and still stop. +Every watch gets `--state-cmd` (a printf standing in for `systemctl show`). + The GE NIC (g0) is checked from a fake networkd lease with domain test.geaerospace.net and must read `ge` in every scenario. diff --git a/tests/pxe-nic-check/frames.py b/tests/pxe-nic-check/frames.py index 8d3dab7..5c0dd15 100644 --- a/tests/pxe-nic-check/frames.py +++ b/tests/pxe-nic-check/frames.py @@ -66,13 +66,25 @@ def cdp(src, device="WJ-CORE1.wjs.geaerospace.net", port="GigabitEthernet1/0/3", + llc -def eapol(src): - # EAPOL v2, type 0 (EAP packet), EAP Request/Identity - eap = bytes([1, 1]) + struct.pack("!H", 5) + bytes([1]) +def eapol(src, code=1): + """EAPOL v2, type 0 (EAP packet). code 1 Request/Identity (what a + switch port sends), 2 Response/Identity (a supplicant), 3 Success, + 4 Failure.""" + if code in (1, 2): + eap = bytes([code, 1]) + struct.pack("!H", 5) + bytes([1]) + else: + eap = bytes([code, 1]) + struct.pack("!H", 4) body = bytes([2, 0]) + struct.pack("!H", len(eap)) + eap return eth("01:80:c2:00:00:03", src, 0x888E, body) +def eapol_start(src, eapol_type=1): + """Supplicant EAPOL-Start (type 1) or Logoff (type 2), empty body: + iPXE sends three Starts each time it opens a NIC.""" + body = bytes([2, eapol_type]) + struct.pack("!H", 0) + return eth("01:80:c2:00:00:03", src, 0x888E, body) + + def stp(src): bpdu = b"\x00\x00\x00\x00\x00" + b"\x80\x00" + mac(src) + b"\x00" * 23 llc = b"\x42\x42\x03" + bpdu diff --git a/tests/pxe-nic-check/lab.py b/tests/pxe-nic-check/lab.py index a57bbd0..50fa9c4 100755 --- a/tests/pxe-nic-check/lab.py +++ b/tests/pxe-nic-check/lab.py @@ -34,6 +34,8 @@ HERE = os.path.dirname(os.path.abspath(__file__)) SCRIPT = os.path.normpath(os.path.join( HERE, "..", "..", "playbook", "pxe-server-helpers", "pxe-nic-check")) PEER = os.path.join(HERE, "lab_peer.py") +CARRIER = os.path.normpath(os.path.join( + HERE, "..", "..", "playbook", "pxe-server-helpers", "pxe-nic-carrier")) ENTRY_KEYS = ["iface", "mac", "role", "link", "speed_mbps", "addrs", "verdict", "evidence", "dhcp_servers", "switch", "arp_hosts", @@ -56,6 +58,10 @@ SCENARIOS = [ ("lldp-only", ["--lldp", "WJ-IDF2"], "ge", 1, "LLDP from WJ-IDF2 port Gi1/0/14"), ("eapol-only", ["--eapol"], "ge", 1, "EAPOL"), + # a bay PXE-booting on the imaging switch: iPXE EAPOL-Start and a + # Windows EAP-Response must not read as a GE switch port + ("eapol-client", ["--eapol-start"], "isolated", 0, + "client EAPOL frames ignored"), ("own-server", ["--dhcp", "172.16.9.1"], "isolated", 0, "own DHCP server 172.16.9.1 answered (ignored)"), ("cloned-pxe", ["--dhcp", "172.16.9.1", "--mac", "02:cc:00:00:00:01"], @@ -77,6 +83,8 @@ BRIDGE_SCENARIOS = [ ("br-isolated", None, "isolated", 0, "no DHCP server, no switch"), ("br-lldp", ["--lldp", "WJ-IDF2"], "ge", 1, "LLDP from WJ-IDF2"), ("br-eapol", ["--eapol"], "ge", 1, "EAPOL"), + ("br-eapol-client", ["--eapol-start"], "isolated", 0, + "client EAPOL frames ignored"), ("br-rogue", ["--dhcp", "10.99.0.1"], "foreign-dhcp", 1, "DHCP from 10.99.0.1, no domain"), ("br-fake-ge", ["--dhcp", "10.134.48.1", "--domain", @@ -85,6 +93,17 @@ BRIDGE_SCENARIOS = [ ] +# Scenarios that also run --watch as if an operator restarted dnsmasq in +# the middle of the window: the watch must re-check, and still stop it. +RACE = {"rogue", "lldp-only", "br-lldp"} + +# What `systemctl show` would say: dnsmasq active since boot, or since +# just now (inside the watch window). +STATE_EARLY = "printf 'ActiveState=active\\nActiveEnterTimestampMonotonic=1\\n'" +STATE_LATE = ("printf 'ActiveState=active\\n" + "ActiveEnterTimestampMonotonic=%d\\n'" % (2 ** 62)) + + def sh(*cmd): subprocess.run(cmd, check=True) @@ -252,18 +271,97 @@ def scenario(name, spec, want, want_rc, needle, bound, conf, leases, tmp, # 3. watch, with a marker file standing in for systemctl stop marker = os.path.join(tmp, name + ".stopped") r = run_script(["--watch", "--window", "3", "--write", status, - "--stop-cmd", "touch " + marker], conf, leases) + "--stop-cmd", "touch " + marker, + "--state-cmd", STATE_EARLY], conf, leases) stopped = os.path.exists(marker) if stopped != bool(want_rc): fails.append("watch stop called=%s (want %s): %s" % (stopped, bool(want_rc), r.stderr.strip())) if want_rc and "<2>" not in r.stderr and "CRIT" not in r.stderr: fails.append("watch did not log CRIT") + # 4. watch racing an operator restart: re-check, still stop + if name in RACE: + marker2 = marker + ".race" + r = run_script(["--watch", "--window", "3", "--write", status, + "--stop-cmd", "touch " + marker2, + "--state-cmd", STATE_LATE], conf, leases) + if "re-checking" not in r.stderr: + fails.append("race: watch did not re-check: %s" + % r.stderr.strip()) + if not os.path.exists(marker2): + fails.append("race: re-check did not stop dnsmasq: %s" + % r.stderr.strip()) + if "could not join" in r.stderr: + fails.append("multicast join failed: %s" % r.stderr.strip()) finally: stop_peer(p, spec) return fails, im +def carrier_lab(conf, leases, tmp): + """pxe-nic-carrier against real sysfs carrier in the namespace: pull + the far end of v0's veth, plug it back, and poke it with GE NIC + events. systemctl is a stub that records argv and says dnsmasq is + active; pxe-nic-check --print-imaging is the real script.""" + fails = [] + stub = os.path.join(tmp, "carrier-stub") + os.makedirs(stub, exist_ok=True) + calls = os.path.join(stub, "calls") + with open(os.path.join(stub, "systemctl"), "w") as f: + f.write('#!/bin/sh\necho "$*" >> %s\n' + '[ "$1" = is-active ] && echo active\nexit 0\n' % calls) + with open(os.path.join(stub, "check"), "w") as f: + f.write('#!/bin/sh\nexec %s %s --config %s --dnsmasq-nic-conf ' + '/nonexistent --lease-dir %s "$@"\n' + % (sys.executable, SCRIPT, conf, leases)) + with open(os.path.join(stub, "logger"), "w") as f: # not the host log + f.write('#!/bin/sh\necho "$*" >> %s/log\n' % stub) + for n in ("systemctl", "check", "logger"): + os.chmod(os.path.join(stub, n), 0o755) + env = dict(os.environ, PATH=stub + ":" + os.environ.get("PATH", ""), + PXE_SYSTEMCTL=os.path.join(stub, "systemctl"), + PXE_NIC_CHECK=os.path.join(stub, "check"), + PXE_NIC_CARRIER_RUN=os.path.join(stub, "run")) + + def event(iface, state): + if os.path.exists(calls): + os.unlink(calls) + subprocess.run(["bash", CARRIER], env=dict(env, IFACE=iface, + STATE=state), + timeout=30, check=True) + if not os.path.exists(calls): + return [] + with open(calls) as f: + return f.read().splitlines() + + event("v0", "routable") # first sight: up + sh("ip", "link", "set", "v1", "down") + time.sleep(0.5) + got = event("v0", "no-carrier") + if "stop --no-block dnsmasq" not in got: + fails.append("carrier loss did not stop dnsmasq: %s" % got) + sh("ip", "link", "set", "v0", "down") # admin down: EINVAL + got = event("v0", "off") + if got: + fails.append("repeat loss not idempotent: %s" % got) + sh("ip", "link", "set", "v0", "up") + sh("ip", "link", "set", "v1", "up") + time.sleep(1.5) + got = event("v0", "carrier") + if "restart --no-block pxe-nic-carrier.service" not in got: + fails.append("carrier gain did not hand off: %s" % got) + if event("v0", "routable"): + fails.append("repeat gain not idempotent") + sh("ip", "link", "set", "g1", "down") + time.sleep(0.5) + got = event("g0", "no-carrier") + sh("ip", "link", "set", "g1", "up") + if got: + fails.append("GE NIC event touched systemctl: %s" % got) + time.sleep(1.0) + return fails + + def inside(selected): tmp = tempfile.mkdtemp(prefix="pxe-nic-lab-") conf, conf_br, leases = setup(tmp) @@ -293,6 +391,18 @@ def inside(selected): print(" " + f, flush=True) if im and not fails: print(" " + " | ".join(im["evidence"]), flush=True) + if not selected or "carrier" in selected: + t0 = time.monotonic() + try: + fails = carrier_lab(conf, leases, tmp) + except Exception as e: + fails = ["exception: %r" % e] + state = "FAIL" if fails else "PASS" + results.append(("carrier", state, fails)) + print("%s %-16s pxe-nic-carrier on real carrier (%.0f s)" + % (state, "carrier", time.monotonic() - t0), flush=True) + for f in fails: + print(" " + f, flush=True) n_pass = sum(1 for r in results if r[1] == "PASS") n_fail = sum(1 for r in results if r[1] == "FAIL") n_skip = sum(1 for r in results if r[1] == "SKIP") diff --git a/tests/pxe-nic-check/lab_peer.py b/tests/pxe-nic-check/lab_peer.py index 842009f..ac52918 100755 --- a/tests/pxe-nic-check/lab_peer.py +++ b/tests/pxe-nic-check/lab_peer.py @@ -10,6 +10,8 @@ plays a switch and/or a DHCP server with raw AF_PACKET sockets: --domain NAME option 15 in the offer --lldp SYSNAME send an LLDP frame every second --eapol send an EAPOL Request/Identity every second + --eapol-start send EAPOL-Start + EAP-Response like a PXE-booting + client (iPXE, Windows dot3svc) every second --mac MAC source MAC for the frames (default: the iface's MAC) Never run this outside the lab netns: it is a rogue DHCP server. @@ -34,6 +36,7 @@ def main(): ap.add_argument("--domain") ap.add_argument("--lldp") ap.add_argument("--eapol", action="store_true") + ap.add_argument("--eapol-start", action="store_true") ap.add_argument("--mac") ap.add_argument("--duration", type=float, default=120) args = ap.parse_args() @@ -58,6 +61,9 @@ def main(): vlan=309)) if args.eapol: s.send(F.eapol(mac)) + if args.eapol_start: + s.send(F.eapol_start(mac)) + s.send(F.eapol(mac, code=2)) next_beacon = now + 1.0 r, _, _ = select.select([s], [], [], 0.2) if not r: diff --git a/tests/pxe-nic-check/test_helpers.py b/tests/pxe-nic-check/test_helpers.py new file mode 100644 index 0000000..a92a266 --- /dev/null +++ b/tests/pxe-nic-check/test_helpers.py @@ -0,0 +1,290 @@ +""" +Tests for the shell helpers next to pxe-nic-check: pxe-dhcp-hook.sh (the +dnsmasq dhcp-script) and pxe-nic-carrier (the networkd-dispatcher hook). +No root, no network: every command they run (ss, conntrack, logger, +systemctl, pxe-nic-check) is a stub on a fake PATH that records its argv. + + python3 -m unittest discover -s tests/pxe-nic-check -v +""" + +import os +import shutil +import subprocess +import tempfile +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +HELPERS = os.path.normpath(os.path.join(HERE, "..", "..", "playbook", + "pxe-server-helpers")) +HOOK = os.path.join(HELPERS, "pxe-dhcp-hook.sh") +CARRIER = os.path.join(HELPERS, "pxe-nic-carrier") + +# Each stub appends "name arg1 arg2 ..." (args joined by \x1f) to $CALLS. +STUB = r"""#!/bin/bash +{ printf '%s' "$(basename "$0")"; for a in "$@"; do printf '\x1f%s' "$a"; done + printf '\n'; } >> "$CALLS" +""" + +SYSTEMCTL = STUB + r""" +case "$1" in + is-active) s=$(cat "$STUBDIR/dnsmasq.state" 2>/dev/null || echo active) + echo "$s"; [ "$s" = active ] ;; + restart) [ "$2" = dnsmasq ] && exit "$(cat "$STUBDIR/restart.rc" 2>/dev/null || echo 0)" + exit 0 ;; + *) exit 0 ;; +esac +""" + +CHECK = STUB + r""" +[ "$1" = --print-imaging ] && cat "$STUBDIR/imaging" 2>/dev/null +exit 0 +""" + + +class Stubs: + def __init__(self): + self.dir = tempfile.mkdtemp(prefix="pxe-helper-test-") + self.bin = os.path.join(self.dir, "bin") + os.mkdir(self.bin) + self.calls = os.path.join(self.dir, "calls") + for name, body in (("ss", STUB), ("conntrack", STUB), + ("logger", STUB), ("systemctl", SYSTEMCTL), + ("pxe-nic-check", CHECK)): + self.write(os.path.join(self.bin, name), body, 0o755) + self.sys = os.path.join(self.dir, "sys") + self.run_dir = os.path.join(self.dir, "run") + + def write(self, path, text, mode=0o644): + with open(path, "w") as f: + f.write(text) + os.chmod(path, mode) + + def env(self, **extra): + env = {"PATH": self.bin + ":/usr/bin:/bin", "CALLS": self.calls, + "STUBDIR": self.dir, + "PXE_NIC_CHECK": os.path.join(self.bin, "pxe-nic-check"), + "PXE_SYSTEMCTL": os.path.join(self.bin, "systemctl"), + "PXE_SYS_NET": self.sys, + "PXE_NIC_CARRIER_RUN": self.run_dir, + "PXE_NIC_CARRIER_SETTLE": "0"} + env.update(extra) + return env + + def get_calls(self, name=None): + try: + with open(self.calls) as f: + lines = f.read().splitlines() + except OSError: + return [] + out = [ln.split("\x1f") for ln in lines] + return [c for c in out if name is None or c[0] == name] + + def reset(self): + if os.path.exists(self.calls): + os.unlink(self.calls) + + def link(self, iface, carrier): + d = os.path.join(self.sys, iface) + os.makedirs(d, exist_ok=True) + path = os.path.join(d, "carrier") + if carrier is None: # admin down: reading carrier fails + if os.path.exists(path): + os.unlink(path) + else: + self.write(path, "%d\n" % carrier) + + def cleanup(self): + shutil.rmtree(self.dir, ignore_errors=True) + + +class DhcpHook(unittest.TestCase): + def setUp(self): + self.s = Stubs() + + def tearDown(self): + self.s.cleanup() + + def run_hook(self, action, vendor=None): + env = self.s.env() + if vendor is not None: + env["DNSMASQ_VENDOR_CLASS"] = vendor + r = subprocess.run(["bash", HOOK, action, "02:aa:00:00:00:05", + "172.16.9.50", "bay3105"], env=env, + capture_output=True, text=True, timeout=10) + self.assertEqual(r.returncode, 0, r.stderr) + return self.s.get_calls() + + def assert_smb_only(self, calls): + ct = [c[1:] for c in calls if c[0] == "conntrack"] + self.assertEqual(sorted(ct), sorted([ + ["-D", "-p", "tcp", "-s", "172.16.9.50", "--dport", "445"], + ["-D", "-p", "tcp", "-d", "172.16.9.50", "--sport", "445"], + ["-D", "-p", "tcp", "-s", "172.16.9.50", "--dport", "139"], + ["-D", "-p", "tcp", "-d", "172.16.9.50", "--sport", "139"], + ])) + ss = [c[1:] for c in calls if c[0] == "ss"] + self.assertEqual(ss, [["-K", "-t", "( sport = :445 or sport = :139 )" + " and dst = 172.16.9.50"]]) + logs = [c for c in calls if c[0] == "logger"] + self.assertEqual(len(logs), 1) + self.assertIn("SMB", logs[0][-1]) + + def test_add_and_del_clear_smb_state_whatever_the_vendor(self): + for action in ("add", "del"): + for vendor in ("PXEClient:Arch:00007:UNDI:003016", + "MSFT 5.0", None): + self.s.reset() + self.assert_smb_only(self.run_hook(action, vendor)) + + def test_old_only_for_network_boot(self): + self.s.reset() + calls = self.run_hook("old", "PXEClient:Arch:00007:UNDI:003016") + self.assert_smb_only(calls) + self.assertIn("network boot, PXEClient:Arch:00007", + [c for c in calls if c[0] == "logger"][0][-1]) + for vendor in ("MSFT 5.0", None, ""): + self.s.reset() + self.assertEqual(self.run_hook("old", vendor), [], vendor) + + def test_never_clears_all_traffic(self): + calls = self.run_hook("add", "PXEClient") + for c in calls: + if c[0] == "conntrack": + self.assertIn("-p", c) + self.assertIn("tcp", c) + if c[0] == "ss": + self.assertIn("sport = :445", c[-1]) + + def test_no_ip_does_nothing(self): + r = subprocess.run(["bash", HOOK, "add", "02:aa:00:00:00:05", ""], + env=self.s.env(), capture_output=True, timeout=10) + self.assertEqual(r.returncode, 0) + self.assertEqual(self.s.get_calls(), []) + + +class Carrier(unittest.TestCase): + def setUp(self): + self.s = Stubs() + self.s.write(os.path.join(self.s.dir, "imaging"), "pxe0\n") + self.s.link("pxe0", 1) + self.s.link("corp0", 1) + + def tearDown(self): + self.s.cleanup() + + def event(self, iface, state="routable"): + r = subprocess.run(["bash", CARRIER], env=self.s.env( + IFACE=iface, STATE=state), capture_output=True, text=True, + timeout=20) + self.assertEqual(r.returncode, 0, r.stderr) + return self.s.get_calls() + + def gain(self): + r = subprocess.run(["bash", CARRIER, "--gain"], env=self.s.env(), + capture_output=True, text=True, timeout=20) + self.assertEqual(r.returncode, 0, r.stderr) + return self.s.get_calls() + + def systemctl(self, calls): + return [c[1:] for c in calls if c[0] == "systemctl"] + + def dnsmasq(self, state): + self.s.write(os.path.join(self.s.dir, "dnsmasq.state"), state + "\n") + + def marker(self): + return os.path.exists(os.path.join(self.s.run_dir, "stopped-dnsmasq")) + + def test_ge_nic_is_never_touched(self): + for carrier in (0, 1, None, 1): + self.s.reset() + self.s.link("corp0", carrier) + calls = self.event("corp0", "no-carrier") + self.assertEqual([c[0] for c in calls], ["pxe-nic-check"]) + self.assertFalse(os.path.exists(os.path.join(self.s.run_dir, + "link.corp0"))) + + def test_unknown_imaging_nic_does_nothing(self): + self.s.write(os.path.join(self.s.dir, "imaging"), "") + calls = self.event("pxe0", "no-carrier") + self.assertEqual([c[0] for c in calls], ["pxe-nic-check"]) + + def test_loss_stops_running_dnsmasq_and_marks(self): + self.event("pxe0", "routable") # first sight: up + self.s.reset() + self.s.link("pxe0", 0) + sc = self.systemctl(self.event("pxe0", "no-carrier")) + self.assertIn(["stop", "pxe-nic-carrier.service"], sc) + self.assertIn(["stop", "--no-block", "dnsmasq"], sc) + self.assertLess(sc.index(["stop", "pxe-nic-carrier.service"]), + sc.index(["stop", "--no-block", "dnsmasq"])) + self.assertTrue(self.marker()) + # same state again (off after no-carrier): idempotent, no calls + self.s.reset() + self.s.link("pxe0", None) + self.assertEqual(self.systemctl(self.event("pxe0", "off")), []) + + def test_loss_with_dnsmasq_stopped_leaves_no_marker(self): + self.dnsmasq("inactive") + self.s.link("pxe0", 0) + sc = self.systemctl(self.event("pxe0", "no-carrier")) + self.assertNotIn(["stop", "--no-block", "dnsmasq"], sc) + self.assertFalse(self.marker()) + + def test_gain_hands_off_once(self): + self.s.link("pxe0", 0) + self.event("pxe0", "no-carrier") + self.s.reset() + self.s.link("pxe0", 1) + sc = self.systemctl(self.event("pxe0", "carrier")) + self.assertEqual(sc[-1], ["restart", "--no-block", + "pxe-nic-carrier.service"]) + self.assertNotIn("dnsmasq", [x for c in sc for x in c]) + self.s.reset() # routable right after carrier + self.assertEqual(self.systemctl(self.event("pxe0", "routable")), []) + + def test_bridge_port_events_count(self): + self.s.write(os.path.join(self.s.dir, "imaging"), "br-pxe\npxe0\n") + self.s.link("br-pxe", 1) + self.event("pxe0", "enslaved") + self.s.reset() + self.s.link("pxe0", 0) + sc = self.systemctl(self.event("pxe0", "no-carrier")) + self.assertIn(["stop", "--no-block", "dnsmasq"], sc) + + def test_gain_restarts_dnsmasq_through_its_guard(self): + os.makedirs(self.s.run_dir) + self.s.write(os.path.join(self.s.run_dir, "stopped-dnsmasq"), "") + self.dnsmasq("inactive") # the hook stopped it + sc = self.systemctl(self.gain()) + self.assertIn(["restart", "dnsmasq"], sc) + self.assertFalse(self.marker()) + + def test_gain_restarts_running_dnsmasq(self): + # loss never seen (fast swap, boot): dnsmasq running, no marker + sc = self.systemctl(self.gain()) + self.assertIn(["restart", "dnsmasq"], sc) + + def test_gain_logs_crit_when_guard_refuses(self): + self.s.write(os.path.join(self.s.dir, "restart.rc"), "1\n") + calls = self.gain() + logs = [c for c in calls if c[0] == "logger"] + self.assertTrue(any("daemon.crit" in c and "NOT started" in c[-1] + for c in logs), logs) + + def test_gain_leaves_operator_stopped_dnsmasq_alone(self): + self.dnsmasq("inactive") + calls = self.gain() + self.assertNotIn(["restart", "dnsmasq"], self.systemctl(calls)) + self.assertNotIn(["start", "dnsmasq"], self.systemctl(calls)) + self.assertIn(["pxe-nic-check", "--guard"], + [c for c in calls if c[0] == "pxe-nic-check"]) + + def test_gain_does_nothing_if_link_dropped_again(self): + self.s.link("pxe0", 0) + calls = self.gain() + self.assertEqual(self.systemctl(calls), []) + self.assertNotIn(["pxe-nic-check", "--guard"], calls) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/pxe-nic-check/test_pxe_nic_check.py b/tests/pxe-nic-check/test_pxe_nic_check.py index fc57a4b..cc4b31d 100644 --- a/tests/pxe-nic-check/test_pxe_nic_check.py +++ b/tests/pxe-nic-check/test_pxe_nic_check.py @@ -82,6 +82,21 @@ class Frames(unittest.TestCase): o = fed(F.eapol(SW)) self.assertTrue(o.eapol) + def test_eapol_only_from_authenticator(self): + client = "02:aa:00:00:00:07" + for name, fr, want in [ + ("request", F.eapol(SW, code=1), True), + ("success", F.eapol(SW, code=3), True), + ("failure", F.eapol(SW, code=4), True), + ("response", F.eapol(client, code=2), False), + ("start", F.eapol_start(client), False), + ("logoff", F.eapol_start(client, eapol_type=2), False), + ("truncated", F.eapol(SW)[:17], False), + ]: + o = fed(fr) + self.assertEqual(o.eapol, want, name) + self.assertEqual(o.eapol_client, 0 if want else 1, name) + def test_stp(self): o = fed(F.stp(SW)) self.assertTrue(o.stp) @@ -207,6 +222,113 @@ class Bpf(unittest.TestCase): self.assertFalse(run_bpf(insns, fr), name) +class Multicast(unittest.TestCase): + def test_packet_mreq_packing(self): + import struct + raw = M.packet_mreq(7, "01:80:C2:00:00:0E") + self.assertEqual(len(raw), 16) # sizeof(struct packet_mreq) + idx, typ, alen, addr = struct.unpack("iHH8s", raw) + self.assertEqual((idx, typ, alen), (7, M.PACKET_MR_MULTICAST, 6)) + self.assertEqual(addr, bytes.fromhex("0180c200000e") + b"\0\0") + self.assertEqual(M.SOL_PACKET, 263) + self.assertEqual(M.PACKET_ADD_MEMBERSHIP, 1) + with self.assertRaises(ValueError): + M.packet_mreq(1, "01:80:c2") + + def test_groups(self): + self.assertEqual(sorted(M.CAPTURE_GROUPS), sorted([ + "01:80:c2:00:00:00", "01:80:c2:00:00:03", "01:80:c2:00:00:0e", + "01:00:0c:cc:cc:cc", "01:00:0c:cc:cc:cd"])) + + def test_join_groups_calls_setsockopt_per_group(self): + calls = [] + + class FakeSock: + def setsockopt(self, level, opt, val): + calls.append((level, opt, val)) + if val[8:14] == bytes.fromhex("01000ccccccd"): + raise OSError("nope") + + import socket + real = socket.if_nametoindex + socket.if_nametoindex = lambda name: 3 + try: + failed = M.join_groups(FakeSock(), "eth9") + finally: + socket.if_nametoindex = real + self.assertEqual(failed, ["01:00:0c:cc:cc:cd"]) + self.assertEqual(len(calls), 5) + for level, opt, val in calls: + self.assertEqual((level, opt), (263, 1)) + self.assertEqual(val[:4], M.packet_mreq(3, "01:80:c2:00:00:00")[:4]) + self.assertEqual(M.join_groups(FakeSock(), "no-such-if-xyz"), + M.CAPTURE_GROUPS) + + +class WatchRace(unittest.TestCase): + def test_parse_unit_state(self): + self.assertEqual(M.parse_unit_state( + "ActiveState=active\nActiveEnterTimestampMonotonic=123456\n"), + ("active", 123456)) + self.assertEqual(M.parse_unit_state( + "ActiveEnterTimestampMonotonic=0\nActiveState=inactive\n"), + ("inactive", 0)) + self.assertEqual(M.parse_unit_state(""), (None, None)) + + def test_decision(self): + start = 1000000 + self.assertEqual(M.watch_decision("active", 500, start), "stop") + self.assertEqual(M.watch_decision("active", start + 1, start), + "recheck") + self.assertEqual(M.watch_decision("activating", 0, start), "recheck") + self.assertEqual(M.watch_decision("inactive", 0, start), "skip") + self.assertEqual(M.watch_decision("failed", 0, start), "skip") + self.assertEqual(M.watch_decision(None, None, start), "stop") + self.assertEqual(M.watch_decision("active", None, start), "stop") + + def _main_watch(self, state_out, recheck_state): + d = tempfile.mkdtemp() + marker = os.path.join(d, "stopped") + saved = M.run_check, M.recheck + blocked = {"checked_at": "x", "duration_s": 1, "interfaces": [], + "dhcp_guard": {"state": "blocked", "reason": "LLDP"}} + rechecked = [] + M.run_check = lambda args, cfg: (blocked, None) + + def fake_recheck(args, cfg): + rechecked.append(1) + return recheck_state, "re-check" + M.recheck = fake_recheck + try: + rc = M.main(["--watch", "--config", "/nonexistent", + "--write", os.path.join(d, "s.json"), + "--stop-cmd", "touch " + marker, + "--state-cmd", "printf '%s'" % state_out]) + finally: + M.run_check, M.recheck = saved + self.assertEqual(rc, 0) + return os.path.exists(marker), bool(rechecked) + + def test_watch_stops_when_dnsmasq_predates_window(self): + self.assertEqual(self._main_watch( + "ActiveState=active\nActiveEnterTimestampMonotonic=1\n", + "allowed"), (True, False)) + + def test_watch_rechecks_after_operator_restart(self): + late = "ActiveState=active\nActiveEnterTimestampMonotonic=%d\n" \ + % (2 ** 62) + self.assertEqual(self._main_watch(late, "allowed"), (False, True)) + self.assertEqual(self._main_watch(late, "blocked"), (True, True)) + + def test_watch_skips_stopped_dnsmasq(self): + self.assertEqual(self._main_watch( + "ActiveState=inactive\nActiveEnterTimestampMonotonic=0\n", + "allowed"), (False, False)) + + def test_watch_stops_when_state_unknown(self): + self.assertEqual(self._main_watch("", "allowed"), (True, False)) + + class OwnServer(unittest.TestCase): def test_own_needs_own_ip_and_local_mac(self): o = M.Observations() @@ -284,6 +406,24 @@ class Verdicts(unittest.TestCase): M.allowed_list("IMG-SW1")) self.assertEqual(v, "ge") + def test_client_eapol_stays_isolated(self): + # a bay PXE-booting next to us: iPXE sends 3 EAPOL-Starts on every + # NIC open, Windows dot3svc sends Starts and EAP-Responses + client = "02:aa:00:00:00:07" + starts = [F.eapol_start(client)] * 3 + v, ev = self.run_case(starts) + self.assertEqual(v, "isolated", ev) + self.assertTrue(any("3 client EAPOL frames ignored" in e + for e in ev), ev) + v, ev = self.run_case([F.eapol(client, code=2)]) + self.assertEqual(v, "isolated", ev) + v, ev = self.run_case(starts + [F.eapol(client, code=2)]) + self.assertEqual(v, "isolated", ev) + # a switch port asking for identity is still decisive + v, ev = self.run_case(starts + [F.eapol(SW, code=1)]) + self.assertEqual(v, "ge", ev) + self.assertIn("802.1X EAPOL seen (port authentication)", ev) + def test_no_link_and_error(self): self.assertEqual(self.run_case([], link=False)[0], "no-link") self.assertEqual(self.run_case([], error="boom")[0], "error")