From b11e6782978cb5087bac8b9e30b0290ca88ba0ed Mon Sep 17 00:00:00 2001 From: cproudlock Date: Fri, 16 Oct 2026 15:56:47 -0400 Subject: [PATCH] NICs: pick the imaging port at install, GE can go into any other port Sites no longer need either MAC in site-vars. pxe_nic_mac empty: before anything changes, the playbook runs pxe-pick-nic --auto. It takes the one wired port with a link, accepts it only if pxe-nic-check calls it isolated (no LLDP/CDP/802.1X, no other DHCP server, at most 10 hosts heard in 65 s), and pins it. The choice is saved in /etc/pxe-server/nic-pick.conf and never re-guessed at boot. When it cannot tell (no link, two links, not isolated) nothing changes, and the console banner says why and to run sudo pxe-pick-nic: pull the imaging cable and plug it back in, and the port that blinked is the imaging one. Both helpers are installed before the pick, so that works after a refused first boot. The site's corp domains and allowed switches are passed to the check. corp_nic_mac empty: every other wired port (en*, eth*) is a catch-all DHCP client, so GE can be plugged in at install or later. - dnsmasq stays pinned with interface=pxe0. A new check refuses any dhcp-range outside the imaging subnet, and proxy ranges. - The catch-all ports ignore offers from 172.16.9.1. - UFW allows SSH and 9009 by source on any interface, and keeps those rules when the old any-interface rules are removed. - pxe_nic_name must not look like en*/eth* in this mode. pxe-netplan-apply.sh takes 3 or 5 args. Without a pinned GE NIC it: - renames stray names (the old pxe0 after a re-pick, a leftover corp0) to enx, so the catch-all takes them, and restores them on revert; - checks that every NIC that had a DHCP address before has one after, tracked by MAC. With both MACs set (the WJ server), netplan, the dnsmasq pin and UFW render byte-for-byte as before. Tested in the pxetest VMs: - Fresh install with no MACs and the GE link down: the imaging port was picked and pinned as pxe0 with 172.16.9.1, and dnsmasq bound DHCP to pxe0 only. GE plugged in afterwards got a lease, SSH worked, and the guard reported pxe0 isolated. - --auto refuses with two links. - The interactive pick works by pulling and replugging the link. - A 3-arg re-apply is UNCHANGED. Unit tests (fake sysfs) cover the re-pick and pinned-to-catch-all renames. --- SETUP.md | 12 +- autoinstall/user-data | 8 +- docs/SITE-CUSTOMIZATION.md | 42 ++- docs/pxe-nic-guard.md | 6 +- .../pxe-server-helpers/pxe-netplan-apply.sh | 152 +++++++-- playbook/pxe-server-helpers/pxe-pick-nic | 297 ++++++++++++++++++ playbook/pxe_server_setup.yml | 246 +++++++++++++-- playbook/site-vars.example.yml | 20 +- .../templates/dnsmasq-pxe-nic-only.conf.j2 | 6 + playbook/templates/netplan-pxe.yaml.j2 | 32 ++ scripts/build-usb.sh | 30 +- tests/vm/build-image.sh | 31 +- 12 files changed, 774 insertions(+), 108 deletions(-) create mode 100755 playbook/pxe-server-helpers/pxe-pick-nic diff --git a/SETUP.md b/SETUP.md index 7177a3f..ce1deb7 100644 --- a/SETUP.md +++ b/SETUP.md @@ -144,9 +144,15 @@ This creates a bootable USB with: ### Step 5: Connect to Isolated Network -Both NICs are cabled before the install: the imaging NIC (`pxe_nic_mac`) -to the isolated switch for PXE clients, the corp NIC (`corp_nic_mac`) to -the GE network. Nothing needs to move after the install. +During the install plug in ONLY the imaging switch (GE unplugged). With +`pxe_nic_mac` empty in site-vars, the first boot picks the one wired port +with a link, checks it is isolated (no GE switch, no 802.1X, no other DHCP +server) and pins it. Then plug the GE cable into any other port: it becomes +a DHCP client for SSH and the webapp, and dnsmasq never serves it. If the +first boot cannot tell which port is the imaging one, the console says so: +log in and run `sudo pxe-pick-nic`, pull the imaging cable and plug it back +in. Sites that prefer to pin both ports by MAC set `pxe_nic_mac` and +`corp_nic_mac` instead; both cables can then be in from the start. ### Step 6: Import WinPE Content (if not bundled in Step 3) diff --git a/autoinstall/user-data b/autoinstall/user-data index 6dc13d8..eeda477 100644 --- a/autoinstall/user-data +++ b/autoinstall/user-data @@ -14,8 +14,9 @@ autoinstall: # the imaging LAN) stays without one. No static 172.16.9.1 here: matched # by name it would land on EVERY NIC, the GE one included, and the box # would answer as 172.16.9.1 on the corporate network. First boot's - # playbook replaces this file with the two-NIC config pinned by MAC - # (pxe_nic_mac / corp_nic_mac in playbook/site-vars.yml). optional: the + # playbook replaces this file with the imaging NIC pinned by MAC + # (pxe_nic_mac in playbook/site-vars.yml, or picked by its link when + # empty; corp_nic_mac optional). optional: the # install and boot never wait for a lease (air-gapped is fine). network: version: 2 @@ -139,9 +140,12 @@ autoinstall: if mount "$CIDATA_DEV" /mnt/usb; then if [ -f /mnt/usb/playbook/pxe_server_setup.yml ]; then cd /mnt/usb/playbook + rm -f "$STATE/nic-hint" ansible-playbook -i localhost, -c local pxe_server_setup.yml RC=$? WHY="ansible-playbook exit code $RC" + # pxe-pick-nic leaves the reason it could not pick the imaging NIC + [ "$RC" -ne 0 ] && [ -s "$STATE/nic-hint" ] && WHY="$(head -1 "$STATE/nic-hint")" cd / else WHY="no playbook/pxe_server_setup.yml on CIDATA" diff --git a/docs/SITE-CUSTOMIZATION.md b/docs/SITE-CUSTOMIZATION.md index 8746b50..ea39b49 100644 --- a/docs/SITE-CUSTOMIZATION.md +++ b/docs/SITE-CUSTOMIZATION.md @@ -20,21 +20,38 @@ contribute a `config/sites/.yaml` template back to the repo. ### Network -Every server has two NICs: the imaging NIC (isolated imaging LAN, static -172.16.9.1, the only NIC that serves DHCP/TFTP/HTTP boot/SMB) and the GE -corporate NIC (DHCP client; SSH and the webapp on 9009 only). Put both MACs -in `playbook/site-vars.yml` (copy `playbook/site-vars.example.yml`) before -building the USB: `pxe_nic_mac`, `corp_nic_mac`. The installer brings the -network up with DHCP only; the first-boot playbook stops before changing -anything if either MAC is unset or missing, then pins both NICs by MAC in -netplan (names `pxe0` / `corp0`, overridable), dnsmasq, UFW and Samba. +The imaging NIC (isolated imaging LAN, static 172.16.9.1, the only NIC that +serves DHCP/TFTP/HTTP boot/SMB) is pinned by MAC. The GE corporate side (DHCP +client; SSH and the webapp on 9009 only) is optional. Both MACs in +`playbook/site-vars.yml` (copy `playbook/site-vars.example.yml`) may be left +empty: + +- `pxe_nic_mac` empty: the first boot picks the imaging NIC itself + (`pxe-pick-nic --auto`). It takes the one wired port with a link, and only + if `pxe-nic-check` calls it isolated, then pins it; the choice is in + `/etc/pxe-server/nic-pick.conf` and is never re-guessed at boot. So during + the install plug in ONLY the imaging switch. When it cannot tell (no link, + two links, or that port sees GE) it changes nothing and the console says + to run `sudo pxe-pick-nic`: pull the imaging cable and plug it back in, + and the port that blinked is the imaging one. Run it again after swapping + a dead USB NIC, then re-run the playbook. +- `corp_nic_mac` empty: every other wired port (`en*`, `eth*`) is a DHCP + client, so the GE cable can go into any of them, at install or later. + dnsmasq is pinned to the imaging NIC (`interface=`) and serves only the + imaging subnet, and those ports ignore DHCP offers from 172.16.9.1. UFW + allows SSH and 9009 by source, not by interface. +- `corp_nic_mac` set: that port alone is the GE NIC, named `corp0` (as on + the WJ server). + +The first-boot playbook pins the result in netplan (names `pxe0` / `corp0`, +overridable), dnsmasq, UFW and Samba. | Value | Default | Where it lives | |-------------------|----------------------|--------------------------------------------------------------------------------| | PXE server IP | 172.16.9.1 | `playbook/pxe_server_setup.yml` (dnsmasq config, iPXE script, samba conf, webapp env), `playbook/startnet.cmd` (mount paths), `boot-tools/blancco/grub-blancco.cfg` (TFTP/HTTP URLs) | | PXE subnet | 172.16.9.0/24 | Same as above, plus `playbook/pxe_server_setup.yml` (UFW rules) | | DHCP range | 172.16.9.10-100 | `playbook/site-vars.yml` (`pxe_dhcp_start`, `pxe_dhcp_end`) | -| NIC MACs / names | (required) / pxe0, corp0 | `playbook/site-vars.yml` (`pxe_nic_mac`, `corp_nic_mac`, `pxe_nic_name`, `corp_nic_name`) | +| NIC MACs / names | (picked at install) / pxe0, corp0 | `playbook/site-vars.yml` (`pxe_nic_mac`, `corp_nic_mac`, `pxe_nic_name`, `corp_nic_name`) | | Hostname | pxeserver | `autoinstall/user-data` (identity.hostname) | ### Access from the GE corporate network @@ -46,7 +63,7 @@ warning on every run while that is so. Set these in `playbook/site-vars.yml`: | Var | Default | Effect | |--------------------------|---------|--------| -| `corp_admin_cidrs` | `[]` | List of admin subnets/hosts, e.g. `["10.134.48.0/24", "3.12.34.56/32"]`. When non-empty, UFW allows 22 and 9009 on `corp_nic_name` from these only and drops the any-source rules. The play refuses (nothing changed) if an entry is invalid or your own SSH session is not inside one of them. The imaging NIC keeps its rules. | +| `corp_admin_cidrs` | `[]` | List of admin subnets/hosts, e.g. `["10.134.48.0/24", "3.12.34.56/32"]`. When non-empty, UFW allows 22 and 9009 from these only (on `corp_nic_name` when `corp_nic_mac` is set, else on any interface) and drops the any-source rules. The play refuses (nothing changed) if an entry is invalid or your own SSH session is not inside one of them. The imaging NIC keeps its rules. | | `corp_ssh_password_auth` | `true` | `false` adds `/etc/ssh/sshd_config.d/10-pxe-corp-auth.conf`: `PasswordAuthentication no`, re-allowed by a `Match Address` block for `pxe_subnet` and localhost. Keys only from GE. The play refuses while `pxe` has no `~/.ssh/authorized_keys`, and removes the drop-in again if `sshd -t` rejects it. | | `allow_disruptive` | `false` | Re-runs while imaging is active (SMB sessions on 445, or DHCP leases granted in the last `pxe_busy_window_min` = 30 min) do not bounce the NICs or restart dnsmasq/Apache/Samba; a needed network change stops the play. `-e allow_disruptive=true` overrides. | @@ -58,8 +75,9 @@ desks' subnet, then (after `ssh-copy-id`) `corp_ssh_password_auth: false`. The network step refuses before changing anything when the NIC that `pxe_nic_mac` names holds the default route, a DHCP lease, or the route to your SSH client (swapped MACs or cables). When it does apply, it arms a -120 s auto-revert timer (`pxe-netplan-revert-*`), checks both names, the -imaging address and that the corp NIC gets its IPv4 back within 30 s, and +120 s auto-revert timer (`pxe-netplan-revert-*`), checks the names, the +imaging address and that the corp NIC (or, without one pinned, every other +port that had an address) gets its IPv4 back within 30 s, and restores the previous netplan config AND NIC names on any failure. Backups: `/var/backups/netplan-/`. diff --git a/docs/pxe-nic-guard.md b/docs/pxe-nic-guard.md index b5dfcb8..7c61c76 100644 --- a/docs/pxe-nic-guard.md +++ b/docs/pxe-nic-guard.md @@ -32,7 +32,11 @@ Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml` `pxe_nic_allowed_switches`; `pxe_imaging_mac` / `pxe_ge_mac` are derived from the first two). The same MACs pin the NICs in netplan (`playbook/templates/netplan-pxe.yaml.j2`, names `pxe0` / `corp0`) and in -`/etc/dnsmasq.d/pxe-nic-only.conf`. +`/etc/dnsmasq.d/pxe-nic-only.conf`. With `pxe_nic_mac` empty the playbook +picks the imaging NIC at install time with `pxe-pick-nic --auto`, which +runs this check on the one port with a link (`--iface X --probe`, 65 s) +and accepts only "isolated". With `corp_nic_mac` empty `ge_mac` is blank +and the GE NIC is whichever other port holds a DHCP lease. ## How it decides diff --git a/playbook/pxe-server-helpers/pxe-netplan-apply.sh b/playbook/pxe-server-helpers/pxe-netplan-apply.sh index 0532346..c4a4011 100755 --- a/playbook/pxe-server-helpers/pxe-netplan-apply.sh +++ b/playbook/pxe-server-helpers/pxe-netplan-apply.sh @@ -1,12 +1,17 @@ #!/bin/bash -# pxe-netplan-apply.sh - install the two-NIC netplan config without ever -# leaving the box unreachable on the corporate NIC. +# pxe-netplan-apply.sh - install the PXE server's netplan config without +# ever leaving the box unreachable on the corporate side. # # Usage: -# pxe-netplan-apply.sh STAGED_YAML PXE_NAME PXE_MAC CORP_NAME CORP_MAC -# pxe-netplan-apply.sh --preflight STAGED_YAML PXE_NAME PXE_MAC CORP_NAME CORP_MAC +# pxe-netplan-apply.sh STAGED_YAML PXE_NAME PXE_MAC [CORP_NAME CORP_MAC] +# pxe-netplan-apply.sh --preflight STAGED_YAML PXE_NAME PXE_MAC [CORP_NAME CORP_MAC] # pxe-netplan-apply.sh --revert BACKUP_DIR (the auto-revert timer) # +# Without CORP_NAME/CORP_MAC (corp_nic_mac empty in the playbook) only the +# imaging NIC is pinned and renamed; the other wired ports are catch-all +# DHCP clients that keep their names. Every pre-flight refusal about the +# imaging NIC applies the same. +# # Env (set by pxe_server_setup.yml): # PXE_SUBNET imaging LAN, e.g. 172.16.9.0/24 (SSH peers inside it may # legitimately come in on the imaging NIC) @@ -128,14 +133,16 @@ ssh_peers() { preflight() { local pif cif peer dev bad=0 pif=$(iface_of "$PXE_MAC") - cif=$(iface_of "$CORP_MAC") if [ -z "$pif" ]; then log "REFUSED: no physical NIC has pxe_nic_mac $PXE_MAC" return 1 fi - if [ -z "$cif" ]; then - log "REFUSED: no physical NIC has corp_nic_mac $CORP_MAC" - return 1 + if [ -n "$CORP_MAC" ]; then + cif=$(iface_of "$CORP_MAC") + if [ -z "$cif" ]; then + log "REFUSED: no physical NIC has corp_nic_mac $CORP_MAC" + return 1 + fi fi if default_route_devs | grep -qxF "$pif"; then log "REFUSED: $pif (pxe_nic_mac $PXE_MAC) holds the default route" @@ -156,8 +163,8 @@ preflight() { bad=1 done < <(ssh_peers) if [ "$bad" -ne 0 ]; then - log "The NIC named by pxe_nic_mac looks like the GE corporate one: pxe_nic_mac" - log "and corp_nic_mac are probably swapped, or the cables are. It would become" + log "The NIC named by pxe_nic_mac looks like the GE corporate one: the MACs are" + log "probably swapped, or the cables are. It would become" log "static ${PXE_SUBNET:-imaging} with no gateway and cut this box off. Nothing was changed." log "NICs now: $(for d in "$SYSFS_NET"/*; do [ -e "$d/device" ] && printf '%s=%s ' "$(basename "$d")" "$(cat "$d/address" 2>/dev/null)"; done)" return 1 @@ -181,32 +188,38 @@ link_rename() { return 1 } -# set_names MAC1 NAME1 MAC2 NAME2: give each MAC its name. A NIC that must +# set_names MAC1 NAME1 [MAC2 NAME2]: give each MAC its name. A NIC that must # change goes through a temporary name first, so a swap (pxe0 <-> corp0) # works in both directions. Temporary names carry this run's PID, so they -# never collide with a name left over from an earlier (failed) pass. +# never collide with a name left over from an earlier (failed) pass. A pair +# with an empty MAC or name is skipped (no pinned GE NIC). set_names() { - local -a macs=("$1" "$3") want=("$2" "$4") cur=() + local -a macs=() want=() cur=() local i + while [ $# -ge 2 ]; do + if [ -n "$1" ] && [ -n "$2" ]; then + macs+=("$1"); want+=("$2") + fi + shift 2 + done RENAME_SEQ=$(( ${RENAME_SEQ:-0} + 1 )) - for i in 0 1; do - [ -n "${want[i]}" ] || return 0 + for i in "${!macs[@]}"; do cur[i]=$(iface_of "${macs[i]}") if [ -z "${cur[i]}" ]; then log "ERROR: no NIC with MAC ${macs[i]}" return 1 fi done - for i in 0 1; do + for i in "${!macs[@]}"; do [ "${cur[i]}" = "${want[i]}" ] && continue link_rename "${cur[i]}" "pxt$$-$RENAME_SEQ$i" || return 1 cur[i]=pxt$$-$RENAME_SEQ$i done - for i in 0 1; do + for i in "${!macs[@]}"; do [ "${cur[i]}" = "${want[i]}" ] && continue link_rename "${cur[i]}" "${want[i]}" || return 1 done - for i in 0 1; do + for i in "${!macs[@]}"; do if [ "$(iface_of "${macs[i]}")" != "${want[i]}" ]; then log "ERROR: MAC ${macs[i]} is not named ${want[i]} after renaming" return 1 @@ -216,8 +229,49 @@ set_names() { } names_ok() { - [ "$(iface_of "$PXE_MAC")" = "$PXE_NAME" ] && + [ "$(iface_of "$PXE_MAC")" = "$PXE_NAME" ] || return 1 + if [ -n "$CORP_MAC" ]; then [ "$(iface_of "$CORP_MAC")" = "$CORP_NAME" ] + else + [ -z "$(stray_names)" ] + fi +} + +# No pinned GE NIC: "MAC NAME" for every other physical NIC whose name the +# catch-all (en*, eth*) would not match - the previous imaging NIC still +# called pxe0 after a re-pick, or corp0 from a pinned config. Each gets the +# kernel-style MAC name enx, so it becomes a catch-all DHCP client and +# pxe0 is free for the new imaging NIC. +stray_names() { + local d n m + [ -z "$CORP_MAC" ] || return 0 + for d in "$SYSFS_NET"/*; do + [ -e "$d/device" ] || continue + n=${d##*/} + m=$(cat "$d/address" 2>/dev/null) + [ "$m" = "$PXE_MAC" ] && continue + [ "$(cat "$d/type" 2>/dev/null)" = 1 ] || continue + [ -e "$d/wireless" ] || [ -e "$d/phy80211" ] || [ -e "$d/bridge" ] && continue + case "$n" in en*|eth*) continue ;; esac + echo "$m enx${m//:/}" + done + return 0 +} + +# MACs of the physical NICs other than the imaging one that hold a DHCP +# (dynamic) IPv4: the GE side. They must have one again after the apply, +# or an admin on the GE side may have been cut off. Static addresses (the +# previous imaging NIC's 172.16.9.1) do not count. By MAC: names change. +other_v4_macs() { + local d n m + for d in "$SYSFS_NET"/*; do + [ -e "$d/device" ] || continue + n=${d##*/} + m=$(cat "$d/address" 2>/dev/null) + [ "$m" = "$PXE_MAC" ] && continue + ip -4 -o addr show dev "$n" scope global dynamic 2>/dev/null | grep -q ' inet ' && echo "$m" + done + return 0 } # --- backup / restore -------------------------------------------------------- @@ -239,6 +293,7 @@ write_state() { echo "ORIG_PXE_IF=$ORIG_PXE_IF" echo "CORP_MAC=$CORP_MAC" echo "ORIG_CORP_IF=$ORIG_CORP_IF" + echo "EXTRA=$EXTRA" } >"$BACKUP.state" } @@ -249,6 +304,7 @@ read_state() { NETPLAN_DIR) [[ "$v" =~ ^/[A-Za-z0-9/._-]*$ ]] && NETPLAN_DIR=$v ;; PXE_MAC|CORP_MAC) [[ "$v" =~ ^([0-9a-f]{2}:){5}[0-9a-f]{2}$ ]] && printf -v "$k" '%s' "$v" ;; ORIG_PXE_IF|ORIG_CORP_IF) [[ "$v" =~ ^[A-Za-z0-9._-]{0,15}$ ]] && printf -v "$k" '%s' "$v" ;; + EXTRA) [[ "$v" =~ ^(([0-9a-f]{2}:){5}[0-9a-f]{2}=[A-Za-z0-9._-]{1,15},?)*$ ]] && EXTRA=$v ;; esac done <"$BACKUP.state" TARGET=$NETPLAN_DIR/50-cloud-init.yaml @@ -268,8 +324,11 @@ restore() { log "restoring the previous config from $BACKUP" find "$NETPLAN_DIR" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.yml' \) -delete cp -a "$BACKUP/." "$NETPLAN_DIR/" - if [ -n "${ORIG_PXE_IF:-}" ] && [ -n "${ORIG_CORP_IF:-}" ]; then - set_names "$PXE_MAC" "$ORIG_PXE_IF" "$CORP_MAC" "$ORIG_CORP_IF" || + if [ -n "${ORIG_PXE_IF:-}" ]; then + local -a back=() + local pair + for pair in ${EXTRA//,/ }; do back+=("${pair%%=*}" "${pair#*=}"); done + set_names "$PXE_MAC" "$ORIG_PXE_IF" "$CORP_MAC" "${ORIG_CORP_IF:-}" "${back[@]}" || { log "ERROR: could not put the previous NIC names back"; rc=1; } fi netplan generate && netplan apply || { log "ERROR: netplan apply of the restored config failed"; rc=1; } @@ -300,7 +359,7 @@ case "${1:-}" in log "revert: $BACKUP already committed or reverted; nothing to do" exit 0 fi - PXE_MAC="" CORP_MAC="" ORIG_PXE_IF="" ORIG_CORP_IF="" + PXE_MAC="" CORP_MAC="" ORIG_PXE_IF="" ORIG_CORP_IF="" EXTRA="" read_state log "revert: the apply was not confirmed within its window" restore @@ -308,15 +367,16 @@ case "${1:-}" in ;; esac -if [ $# -ne 5 ]; then - echo "usage: $0 [--preflight] STAGED_YAML PXE_NAME PXE_MAC CORP_NAME CORP_MAC" >&2 +if [ $# -ne 3 ] && [ $# -ne 5 ]; then + echo "usage: $0 [--preflight] STAGED_YAML PXE_NAME PXE_MAC [CORP_NAME CORP_MAC]" >&2 exit 2 fi STAGED=$1 PXE_NAME=$2 PXE_MAC=${3,,} -CORP_NAME=$4 -CORP_MAC=${5,,} +CORP_NAME=${4:-} +CORP_MAC=${5:-} +CORP_MAC=${CORP_MAC,,} # 1. validate in a scratch root SCRATCH=$(mktemp -d) @@ -330,7 +390,7 @@ fi # 2. already in place? if cmp -s "$STAGED" "$TARGET" && [ -z "$(other_files)" ] && names_ok; then - log "netplan already in place, $PXE_NAME and $CORP_NAME present" + log "netplan already in place, $PXE_NAME${CORP_NAME:+ and $CORP_NAME} present" echo UNCHANGED exit 0 fi @@ -350,14 +410,26 @@ flock -w 60 9 || { log "ERROR: another pxe-netplan-apply holds $LOCK_FILE; nothi # 4. back up, record names, arm the auto-revert ORIG_PXE_IF=$(iface_of "$PXE_MAC") -ORIG_CORP_IF=$(iface_of "$CORP_MAC") +ORIG_CORP_IF="" +[ -n "$CORP_MAC" ] && ORIG_CORP_IF=$(iface_of "$CORP_MAC") CORP_HAD_V4=0 -has_v4 "$ORIG_CORP_IF" && CORP_HAD_V4=1 +[ -n "$ORIG_CORP_IF" ] && has_v4 "$ORIG_CORP_IF" && CORP_HAD_V4=1 +OTHERS_V4="" +[ -z "$CORP_MAC" ] && OTHERS_V4=$(other_v4_macs) +# NICs to move off a name the catch-all would not match, recorded with +# their current names for the revert ("mac=name,..."). +EXTRA="" +FREE=() +while read -r m newname; do + [ -n "$m" ] || continue + EXTRA="${EXTRA:+$EXTRA,}$m=$(iface_of "$m")" + FREE+=("$m" "$newname") +done < <(stray_names) BACKUP=$BACKUP_ROOT/netplan-$(date +%Y%m%d-%H%M%S) mkdir -p "$BACKUP" cp -a "$NETPLAN_DIR/." "$BACKUP/" write_state -log "previous config saved in $BACKUP (NICs: $ORIG_PXE_IF, $ORIG_CORP_IF)" +log "previous config saved in $BACKUP (NICs: $ORIG_PXE_IF${ORIG_CORP_IF:+, $ORIG_CORP_IF})" REVERT_UNIT="" if [ "$REVERT_AFTER" -gt 0 ] 2>/dev/null && command -v systemd-run >/dev/null; then @@ -389,20 +461,32 @@ netplan generate || fail "netplan generate failed" # 6. stable names now, so dnsmasq/UFW/Samba (which use them) work before # the next reboot. Does not depend on the netplan version renaming. -set_names "$PXE_MAC" "$PXE_NAME" "$CORP_MAC" "$CORP_NAME" || fail "NIC rename failed" +[ ${#FREE[@]} -gt 0 ] && log "freeing names for the catch-all: $EXTRA" +set_names "$PXE_MAC" "$PXE_NAME" "$CORP_MAC" "$CORP_NAME" "${FREE[@]}" || fail "NIC rename failed" # 7. apply netplan apply || fail "netplan apply failed" # 8. check -names_ok || fail "$PXE_NAME / $CORP_NAME are not on MACs $PXE_MAC / $CORP_MAC after apply" +names_ok || fail "$PXE_NAME${CORP_NAME:+ / $CORP_NAME} not on MAC $PXE_MAC${CORP_MAC:+ / $CORP_MAC} after apply" ok=0 for _ in $(seq 1 "$PXE_WAIT"); do if has_v4 "$PXE_NAME"; then ok=1; break; fi sleep 1 done [ "$ok" -eq 1 ] || fail "$PXE_NAME has no IPv4 address after ${PXE_WAIT}s" -if [ "$CORP_HAD_V4" -eq 1 ]; then +for m in $OTHERS_V4; do + ok=0 + for _ in $(seq 1 "$CORP_WAIT"); do + n=$(iface_of "$m") + if [ -n "$n" ] && has_v4 "$n"; then ok=1; break; fi + sleep 1 + done + [ "$ok" -eq 1 ] || fail "the NIC with MAC $m had a DHCP IPv4 before and has none after ${CORP_WAIT}s" +done +if [ -z "$CORP_MAC" ]; then + : +elif [ "$CORP_HAD_V4" -eq 1 ]; then ok=0 for _ in $(seq 1 "$CORP_WAIT"); do if has_v4 "$CORP_NAME"; then ok=1; break; fi @@ -427,6 +511,6 @@ touch "$BACKUP.committed" cancel_revert trap - INT TERM ip -br addr show dev "$PXE_NAME" 2>/dev/null -ip -br addr show dev "$CORP_NAME" 2>/dev/null +[ -n "$CORP_NAME" ] && ip -br addr show dev "$CORP_NAME" 2>/dev/null echo CHANGED exit 0 diff --git a/playbook/pxe-server-helpers/pxe-pick-nic b/playbook/pxe-server-helpers/pxe-pick-nic new file mode 100755 index 0000000..9eb9ac0 --- /dev/null +++ b/playbook/pxe-server-helpers/pxe-pick-nic @@ -0,0 +1,297 @@ +#!/bin/bash +# pxe-pick-nic - choose the imaging NIC by its cable, not by its MAC. +# +# Usage: +# sudo pxe-pick-nic interactive: pull the imaging switch cable and +# plug it back in; the port that blinked is it +# pxe-pick-nic --auto for the first-boot playbook: the one wired +# port with a link, if it is isolated +# pxe-pick-nic --show print the saved choice +# +# The choice is saved in /etc/pxe-server/nic-pick.conf (imaging_mac=...). +# The playbook uses it when site-vars.yml leaves pxe_nic_mac empty, and pins +# that MAC from then on: the box never re-guesses at boot, so a moved cable +# can never move 172.16.9.1 and DHCP onto another port. +# +# A port is accepted only when pxe-nic-check calls it "isolated": no GE +# switch (LLDP/CDP), no 802.1X port authentication, and no other DHCP +# server answering one DHCPDISCOVER. It listens CHECK_WINDOW seconds +# (default 65: a Cisco switch sends CDP every 60 s). --auto also refuses a +# port where more than ARP_MAX (default 10) hosts were heard: an imaging +# switch at install time is nearly silent, a GE access VLAN is not, and +# that holds even when the switch sends no LLDP/CDP and the probe is lost +# (STP still listening). Anything else is refused and nothing is saved. +# The check uses the site's corp_domain_suffixes and allowed_switches +# (PXE_PICK_SUFFIXES / PXE_PICK_ALLOWED from the playbook, else +# /etc/pxe-server/nic.conf), so a managed imaging switch can be allowed. +# +# Install rule for sites: during the install plug in ONLY the imaging +# switch. Plug the GE cable into any other port afterwards; the catch-all +# netplan entry makes it a DHCP client and dnsmasq never serves it. +# +# Exit: 0 chosen (or --show found one), 1 refused / could not choose, +# 2 usage. + +set -u + +CONF=${PXE_PICK_CONF:-/etc/pxe-server/nic-pick.conf} +HINT=${PXE_PICK_HINT:-/var/lib/pxe-firstboot/nic-hint} +SYSFS_NET=${SYSFS_NET:-/sys/class/net} +NIC_CHECK=${PXE_NIC_CHECK:-/usr/local/sbin/pxe-nic-check} +CHECK_WINDOW=${CHECK_WINDOW:-65} +ARP_MAX=${PXE_PICK_ARP_MAX:-10} +NIC_CONF=${PXE_NIC_CONF:-/etc/pxe-server/nic.conf} +SETTLE=${PXE_PICK_SETTLE:-8} +WAIT=${PXE_PICK_WAIT:-180} + +log() { echo "pxe-pick-nic: $*"; } + +# Physical wired Ethernet ports: a device behind them, ARP type 1, not +# wireless, not a bridge. +wired() { + local d n + for d in "$SYSFS_NET"/*; do + n=${d##*/} + [ -e "$d/device" ] || continue + [ "$(cat "$d/type" 2>/dev/null)" = 1 ] || continue + [ -e "$d/wireless" ] || [ -e "$d/phy80211" ] && continue + [ -e "$d/bridge" ] && continue + echo "$n" + done +} + +mac_of() { cat "$SYSFS_NET/$1/address" 2>/dev/null; } + +# 1 = link, 0 = none. carrier is unreadable while the port is admin-down. +carrier() { [ "$(cat "$SYSFS_NET/$1/carrier" 2>/dev/null)" = 1 ] && echo 1 || echo 0; } + +# Bring every wired port up (no address is added) so its link can be seen. +links_up() { + local n up=0 + for n in $(wired); do + if ! ip link show dev "$n" 2>/dev/null | grep -q '[<,]UP[,>]'; then + ip link set dev "$n" up 2>/dev/null && up=1 + fi + done + [ "$up" -eq 1 ] && sleep "$SETTLE" + return 0 +} + +table() { + local n + for n in $(wired); do + printf ' %-16s %s link %s\n' "$n" "$(mac_of "$n")" \ + "$([ "$(carrier "$n")" = 1 ] && echo up || echo down)" + done +} + +# A pxe-nic-check config with only the site settings that matter here +# (never the MAC keys: the port under test is given with --iface). +check_config() { + local f=$1 suffixes=${PXE_PICK_SUFFIXES:-} allowed=${PXE_PICK_ALLOWED:-} + if [ -z "$suffixes$allowed" ] && [ -f "$NIC_CONF" ]; then + suffixes=$(sed -n 's/^[[:space:]]*corp_domain_suffixes[[:space:]]*=[[:space:]]*//p' "$NIC_CONF" | tail -1) + allowed=$(sed -n 's/^[[:space:]]*allowed_switches[[:space:]]*=[[:space:]]*//p' "$NIC_CONF" | tail -1) + fi + { + echo "[nic]" + [ -n "$suffixes" ] && echo "corp_domain_suffixes = $suffixes" + [ -n "$allowed" ] && echo "allowed_switches = $allowed" + } >"$f" +} + +# check IFACE [strict]: prints the verdict, returns 0 only for "isolated" +# (with strict, also only when at most ARP_MAX hosts were heard). +check() { + local out verdict arp why cfg + if [ ! -f "$NIC_CHECK" ]; then + log "ERROR: $NIC_CHECK not found" + return 1 + fi + cfg=$(mktemp) + check_config "$cfg" + log "checking $1 is isolated: listening ${CHECK_WINDOW}s for GE switches and DHCP servers..." + out=$(python3 "$NIC_CHECK" --iface "$1" --probe --window "$CHECK_WINDOW" --json --config "$cfg" 2>/dev/null) + rm -f "$cfg" + read -r verdict arp why < <(python3 -c ' +import json, sys +try: + e = json.load(sys.stdin)["interfaces"][0] + ev = "; ".join(x for x in e["evidence"] if not x.endswith("ARP hosts seen")) + print(e["verdict"], int(e.get("arp_hosts") or 0), ev or "-") +except Exception as ex: + print("error", 0, "pxe-nic-check gave no usable result: %s" % ex) +' <<<"$out") + if [ "$verdict" = isolated ]; then + if [ "$arp" -gt "$ARP_MAX" ]; then + log "$1: isolated by its switch and DHCP, but $arp hosts were heard (more than $ARP_MAX)" + if [ "${2:-}" = strict ]; then + log "REFUSED: a busy LAN is not an imaging switch at install time; it may be a GE port" + log "with LLDP/CDP off. Pick it by its cable (sudo pxe-pick-nic) or set pxe_nic_mac." + return 1 + fi + log "WARNING: make sure this is the imaging switch, not a GE port with LLDP/CDP turned off." + else + log "$1: isolated (no GE switch, no 802.1X, no other DHCP server, $arp hosts heard)" + fi + return 0 + fi + log "$1: verdict $verdict: $why" + case "$verdict" in + ge) log "That port is on the GE network (or a managed switch not in allowed_switches). It must never be the imaging port." ;; + foreign-dhcp) log "Another DHCP server answers on that port (a second PXE server, or a routed LAN)." ;; + no-link) log "That port lost its link during the check." ;; + esac + return 1 +} + +save() { + local n=$1 mac + mac=$(mac_of "$n") + mkdir -p "$(dirname "$CONF")" + { + echo "# Written by pxe-pick-nic $(date -Is). The imaging NIC; the playbook" + echo "# pins it when site-vars.yml leaves pxe_nic_mac empty." + echo "imaging_mac=$mac" + echo "picked_iface=$n" + } >"$CONF.tmp" && mv "$CONF.tmp" "$CONF" + rm -f "$HINT" + log "saved: imaging NIC = $n ($mac) in $CONF" +} + +hint() { + mkdir -p "$(dirname "$HINT")" 2>/dev/null + { printf '%s\n' "$*" >"$HINT"; } 2>/dev/null +} + +saved_mac() { + sed -n 's/^imaging_mac=\(\([0-9a-f]\{2\}:\)\{5\}[0-9a-f]\{2\}\)$/\1/p' "$CONF" 2>/dev/null | tail -1 +} + +# --- modes --------------------------------------------------------------------- + +mode_show() { + local mac + mac=$(saved_mac) + [ -n "$mac" ] || { log "no saved choice in $CONF"; return 1; } + echo "$mac" +} + +mode_auto() { + local mac n with=() names + mac=$(saved_mac) + if [ -n "$mac" ]; then + for n in $(wired); do + if [ "$(mac_of "$n")" = "$mac" ]; then + log "using the saved choice: $n ($mac)" + echo "$mac" + return 0 + fi + done + log "the saved imaging NIC $mac is not on this box any more" + hint "The saved imaging NIC $mac is gone (dongle replaced?). Log in and run: sudo pxe-pick-nic" + return 1 + fi + links_up + for n in $(wired); do + [ "$(carrier "$n")" = 1 ] && with+=("$n") + done + log "wired ports:" + table + if [ ${#with[@]} -eq 0 ]; then + log "REFUSED: no wired port has a link. Plug the imaging switch into the port to use." + hint "Could not pick the imaging port: no port has a link. Plug in the imaging switch, then log in and run: sudo pxe-pick-nic" + return 1 + fi + if [ ${#with[@]} -gt 1 ]; then + names="${with[*]}" + log "REFUSED: ${#with[@]} ports have a link ($names). During the install only the imaging switch may be plugged in." + hint "Could not pick the imaging port: ${#with[@]} ports have a link ($names). Log in and run: sudo pxe-pick-nic" + return 1 + fi + n=${with[0]} + if ! check "$n" strict; then + hint "Could not pick the imaging port: $n is not isolated (see /var/log/first-boot.log). Plug only the imaging switch in, then log in and run: sudo pxe-pick-nic" + return 1 + fi + save "$n" >&2 + mac_of "$n" +} + +# Wait until exactly one port changes its link; print "NAME NEWSTATE". +wait_change() { + local -A before=() + local n t=0 changed=() + for n in $(wired); do before[$n]=$(carrier "$n"); done + while [ "$t" -lt $((WAIT * 2)) ]; do + sleep 0.5 + t=$((t + 1)) + changed=() + for n in "${!before[@]}"; do + [ "$(carrier "$n")" != "${before[$n]}" ] && changed+=("$n") + done + if [ ${#changed[@]} -eq 1 ]; then + sleep 1 # a flapping connector settles + n=${changed[0]} + echo "$n $(carrier "$n")" + return 0 + fi + if [ ${#changed[@]} -gt 1 ]; then + log "more than one port changed (${changed[*]}); move one cable at a time" >&2 + return 1 + fi + done + log "nothing changed in ${WAIT}s" >&2 + return 1 +} + +mode_interactive() { + local n state again ans + [ -t 0 ] || { log "interactive mode needs a terminal (use --auto from scripts)"; return 2; } + [ "$(id -u)" -eq 0 ] || { log "run it as root: sudo pxe-pick-nic"; return 2; } + links_up + echo + echo "Pick the imaging NIC: the port cabled to the isolated imaging switch." + echo "Wired ports now:" + table + echo + echo "Pull the imaging switch cable out of this server now" + echo "(or, if it is not plugged in yet, plug it in now). Waiting up to ${WAIT}s..." + read -r n state < <(wait_change) || return 1 + [ -n "${n:-}" ] || return 1 + if [ "$state" = 0 ]; then + echo "$n lost its link. Plug the cable back into the SAME port..." + read -r again state < <(wait_change) || return 1 + if [ "${again:-}" != "$n" ] || [ "$state" != 1 ]; then + log "expected $n to come back up, saw ${again:-nothing} $state; try again" + return 1 + fi + fi + sleep "$SETTLE" + echo "Imaging port: $n ($(mac_of "$n"))" + check "$n" || { log "nothing saved"; return 1; } + read -rp "Use $n ($(mac_of "$n")) as the imaging NIC? [y/N] " ans + [[ "$ans" =~ ^[Yy]$ ]] || { log "nothing saved"; return 1; } + save "$n" + if [ -e /var/lib/pxe-firstboot.failed ] && [ -x /opt/first-boot.sh ]; then + read -rp "Run the server setup again now? [Y/n] " ans + if [[ ! "$ans" =~ ^[Nn]$ ]]; then + rm -f /var/lib/pxe-firstboot/attempts + /opt/first-boot.sh 2>&1 | tee -a /var/log/first-boot.log + return "${PIPESTATUS[0]}" + fi + echo "Run it later with: sudo rm -f /var/lib/pxe-firstboot/attempts; sudo /opt/first-boot.sh" + else + echo "Saved. If the server is already set up, re-run the playbook from the" + echo "installer USB to move the imaging config onto this port." + fi + return 0 +} + +case "${1:-}" in + --auto) mode_auto ;; + --show) mode_show ;; + "") mode_interactive ;; + -h|--help) sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "usage: $0 [--auto | --show]" >&2; exit 2 ;; +esac diff --git a/playbook/pxe_server_setup.yml b/playbook/pxe_server_setup.yml index 7447f70..bc65e0c 100644 --- a/playbook/pxe_server_setup.yml +++ b/playbook/pxe_server_setup.yml @@ -73,19 +73,30 @@ # image roots (on a rebuild the _media views may not exist yet). zscaler_ca_share_glob: "{{ samba_share }}/_media/*/Deploy/Applications/ZscalerCommercialCertificate-2048-SHA256.crt {{ samba_share }}/gea-*/Deploy/Applications/ZscalerCommercialCertificate-2048-SHA256.crt" # --- The two NICs (site standard) --- - # Every server has two NICs: the imaging NIC (isolated imaging LAN, - # static pxe_server_ip, the ONLY NIC that serves DHCP/TFTP/HTTP boot/SMB) - # and the GE corporate NIC (DHCP client; SSH and the webapp on 9009 for - # admins, outbound updates through Zscaler). Both MACs are REQUIRED: - # set them in playbook/site-vars.yml on the USB (copy - # site-vars.example.yml) or with -e. The playbook stops before it - # touches anything if either is unset or not on this box. Read them - # with: ip -br link + # The imaging NIC (isolated imaging LAN, static pxe_server_ip, the ONLY + # NIC that serves DHCP/TFTP/HTTP boot/SMB) is pinned by MAC. Leave + # pxe_nic_mac empty and the first run picks it (pxe-pick-nic --auto): + # the one wired port with a link, accepted only if pxe-nic-check calls + # it isolated, saved in /etc/pxe-server/nic-pick.conf and pinned from + # then on. So during the install plug in ONLY the imaging switch. If it + # cannot tell, it stops and the console says to run sudo pxe-pick-nic + # (pull the imaging cable and plug it back in). + # + # The GE corporate NIC (DHCP client; SSH and the webapp on 9009) is + # optional: + # corp_nic_mac empty (default): every other wired port (en*, eth*) is + # a DHCP client, so GE can be plugged into any of them, then or + # later. dnsmasq never serves them (interface= pins it to the + # imaging NIC) and they ignore our own DHCP offers. + # corp_nic_mac set: that one port is pinned and named corp_nic_name, + # as on the WJ server; other ports stay unconfigured. pxe_nic_mac: "" corp_nic_mac: "" # Stable names, set by netplan (match macaddress + set-name). dnsmasq, - # UFW and Samba refer to these. A site may keep its existing names - # (e.g. pxe_nic_name: enx34c8d6b11010, corp_nic_name: enp0s31f6). + # UFW and Samba refer to these. With corp_nic_mac empty they must not + # look like en*/eth* (the catch-all would match them by name). With both + # MACs pinned a site may keep its existing names (WJ: pxe_nic_name: + # enx34c8d6b11010, corp_nic_name: enp0s31f6). pxe_nic_name: "pxe0" corp_nic_name: "corp0" # Imaging LAN addressing. NOTE: 172.16.9.1 is also hard-coded in the @@ -101,6 +112,7 @@ corp_route_metric: 100 # NIC guard (pxe-nic-check): dnsmasq must never serve DHCP/PXE on the GE # corporate network. These are derived from pxe_nic_mac / corp_nic_mac + # (pxe_ge_mac empty: the guard takes the NIC holding a DHCP lease) # (one source of truth) and written to /etc/pxe-server/nic.conf; do not # set them separately (the playbook fails if they disagree). A MAC of a # bridge port resolves to the bridge. @@ -210,29 +222,77 @@ - Tools tasks: - # --- The two NICs: validate, then pin by MAC (netplan) --- + # --- The NICs: pick / validate, then pin by MAC (netplan) --- # Replaces the old guess (first en* without 172.16.9.1 / the default - # route), which on a two-NIC box could pick the GE NIC. + # route), which on a two-NIC box could pick the GE NIC. The pick runs + # from the USB copies, before anything on the box changes, and only + # ever at install time: once pinned, nothing re-guesses at boot. + # Installed first (they change no network config): when the pick + # refuses, the play stops here and the console tells the operator to + # run sudo pxe-pick-nic, which needs both on the box (first-boot.sh + # unmounts the USB). + - name: "NICs: install pxe-nic-check and pxe-pick-nic before the pick" + copy: + src: "{{ usb_mount }}/pxe-server-helpers/{{ item }}" + dest: "/usr/local/sbin/{{ item }}" + mode: '0755' + loop: [pxe-nic-check, pxe-pick-nic] + + - name: "NICs: pick the imaging NIC (pxe_nic_mac is not set in site-vars)" + command: /usr/local/sbin/pxe-pick-nic --auto + environment: + PXE_PICK_SUFFIXES: "{{ pxe_corp_domain_suffixes }}" + PXE_PICK_ALLOWED: "{{ pxe_nic_allowed_switches }}" + register: nic_pick + changed_when: false + failed_when: false + when: pxe_nic_mac | length == 0 + + - name: "NICs: pick result" + debug: + msg: "{{ nic_pick.stdout_lines }}" + when: pxe_nic_mac | length == 0 + + - name: "NICs: the imaging NIC could not be picked" + fail: + msg: >- + Could not pick the imaging NIC, nothing was changed: + {{ nic_pick.stdout_lines | select('search', 'REFUSED|verdict|gone|ERROR') | join(' ') }} + During the install plug in ONLY the imaging switch, or log in and + run: sudo pxe-pick-nic (pull the imaging cable and plug it back in). + Or set pxe_nic_mac in playbook/site-vars.yml. + when: + - pxe_nic_mac | length == 0 + - nic_pick.rc != 0 or not (nic_pick.stdout_lines | last | default('') is match('^([0-9a-f]{2}:){5}[0-9a-f]{2}$')) + + - name: "NICs: pin the picked imaging NIC" + set_fact: + pxe_nic_mac: "{{ nic_pick.stdout_lines | last }}" + when: pxe_nic_mac | length == 0 + - name: "NICs: check the per-site NIC vars" assert: that: - - pxe_nic_mac | length > 0 - - corp_nic_mac | length > 0 - pxe_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$') - - corp_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$') + - corp_nic_mac | length == 0 or corp_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$') - pxe_nic_mac | lower != corp_nic_mac | lower - pxe_imaging_mac | lower == pxe_nic_mac | lower - pxe_ge_mac | lower == corp_nic_mac | lower - pxe_nic_name | length > 0 and pxe_nic_name | length <= 15 - corp_nic_name | length > 0 and corp_nic_name | length <= 15 - pxe_nic_name != corp_nic_name + - corp_nic_mac | length > 0 or not (pxe_nic_name is match('^(en|eth)')) fail_msg: >- - Set pxe_nic_mac (imaging NIC) and corp_nic_mac (GE corporate NIC) - to two different MACs (aa:bb:cc:dd:ee:ff) in playbook/site-vars.yml - (copy site-vars.example.yml) or with -e, then re-run. Do not set - pxe_imaging_mac / pxe_ge_mac yourself; they follow these. Names - (pxe_nic_name, corp_nic_name) must differ and be 1-15 chars. - Now: pxe_nic_mac='{{ pxe_nic_mac }}' corp_nic_mac='{{ corp_nic_mac }}'. + pxe_nic_mac (imaging NIC) must be a MAC (aa:bb:cc:dd:ee:ff); + corp_nic_mac (GE NIC) empty or a different MAC. Set them in + playbook/site-vars.yml (copy site-vars.example.yml) or with -e, + then re-run. Do not set pxe_imaging_mac / pxe_ge_mac yourself; + they follow these (an -e pxe_nic_mac="" overrides the pick too: + drop it). Names (pxe_nic_name, corp_nic_name) must differ + and be 1-15 chars, and with corp_nic_mac empty pxe_nic_name must + not start with en or eth (the catch-all for the other ports + matches those). Now: pxe_nic_mac='{{ pxe_nic_mac }}' + corp_nic_mac='{{ corp_nic_mac }}' pxe_nic_name='{{ pxe_nic_name }}'. quiet: yes - name: "NICs: list the physical NICs on this box" @@ -244,15 +304,15 @@ register: nic_list changed_when: false - - name: "NICs: both MACs must be on this box" + - name: "NICs: the pinned MACs must be on this box" vars: box_macs: "{{ nic_list.stdout_lines | map('regex_replace', '^.* ', '') | map('lower') | list }}" assert: that: - pxe_nic_mac | lower in box_macs - - corp_nic_mac | lower in box_macs + - corp_nic_mac | length == 0 or corp_nic_mac | lower in box_macs fail_msg: >- - {{ 'pxe_nic_mac ' ~ pxe_nic_mac ~ ' is not on this box. ' if pxe_nic_mac | lower not in box_macs else '' }}{{ 'corp_nic_mac ' ~ corp_nic_mac ~ ' is not on this box. ' if corp_nic_mac | lower not in box_macs else '' }}Nothing was changed (the network config would have left that NIC + {{ 'pxe_nic_mac ' ~ pxe_nic_mac ~ ' is not on this box. ' if pxe_nic_mac | lower not in box_macs else '' }}{{ 'corp_nic_mac ' ~ corp_nic_mac ~ ' is not on this box. ' if corp_nic_mac and corp_nic_mac | lower not in box_macs else '' }}Nothing was changed (the network config would have left that NIC unconfigured). Physical NICs here (name MAC): {{ nic_list.stdout_lines | join('; ') }}. Fix playbook/site-vars.yml and re-run. @@ -396,7 +456,7 @@ bash {{ usb_mount }}/pxe-server-helpers/pxe-netplan-apply.sh --preflight /run/pxe-server-netplan.check.yaml {{ pxe_nic_name }} {{ pxe_nic_mac | lower }} - {{ corp_nic_name }} {{ corp_nic_mac | lower }} + {{ (corp_nic_name ~ ' ' ~ corp_nic_mac | lower) if corp_nic_mac else '' }} environment: PXE_SUBNET: "{{ pxe_subnet }}" PXE_SSH_CLIENT: "{{ pxe_ssh_client }}" @@ -439,11 +499,57 @@ dest: /etc/pxe-server-netplan.yaml mode: '0600' + # The catch-all ports (corp_nic_mac empty) are DHCP clients on whatever + # they are cabled to. Cabled to the imaging switch by mistake, they would + # take an address from our own dnsmasq and put a second interface in the + # imaging subnet; DenyList makes them ignore offers from pxe_server_ip. + - name: "NICs: drop-in folders for the catch-all ports" + file: + path: "/etc/systemd/network/10-netplan-{{ item }}.network.d" + state: directory + mode: '0755' + loop: [zz-en, zz-eth] + when: corp_nic_mac | length == 0 + + - name: "NICs: catch-all ports ignore our own DHCP offers" + copy: + dest: "/etc/systemd/network/10-netplan-{{ item }}.network.d/90-pxe-deny-own-dhcp.conf" + mode: '0644' + content: | + # Managed by pxe_server_setup.yml. Never take a lease from this + # server's own dnsmasq (an imaging switch on a non-imaging port). + [DHCPv4] + DenyList={{ pxe_server_ip }} + loop: [zz-en, zz-eth] + when: corp_nic_mac | length == 0 + + - name: "NICs: no catch-all drop-ins with a pinned GE NIC" + file: + path: "/etc/systemd/network/10-netplan-{{ item }}.network.d" + state: absent + loop: [zz-en, zz-eth] + when: corp_nic_mac | length > 0 + + - name: "NICs: create /etc/pxe-server" + file: + path: /etc/pxe-server + state: directory + mode: '0755' + + - name: "NICs: record the pinned imaging NIC for pxe-pick-nic --show" + copy: + dest: /etc/pxe-server/nic-pick.conf + mode: '0644' + content: | + # Written by pxe_server_setup.yml. The imaging NIC the playbook pins. + imaging_mac={{ pxe_nic_mac | lower }} + when: nic_pick is not skipped and nic_pick.rc == 0 + - name: "NICs: validate, install and apply netplan (rolls back on failure)" command: >- /usr/local/sbin/pxe-netplan-apply.sh /etc/pxe-server-netplan.yaml {{ pxe_nic_name }} {{ pxe_nic_mac | lower }} - {{ corp_nic_name }} {{ corp_nic_mac | lower }} + {{ (corp_nic_name ~ ' ' ~ corp_nic_mac | lower) if corp_nic_mac else '' }} environment: PXE_SUBNET: "{{ pxe_subnet }}" PXE_SSH_CLIENT: "{{ pxe_ssh_client }}" @@ -520,7 +626,7 @@ listen-address={{ pxe_server_ip }} bind-dynamic # Interface pinning (serve {{ pxe_nic_name }} only, never - # {{ corp_nic_name }}) is in /etc/dnsmasq.d/pxe-nic-only.conf. + # {{ corp_nic_name if corp_nic_mac else 'the other ports' }}) is in /etc/dnsmasq.d/pxe-nic-only.conf. dhcp-range={{ pxe_dhcp_start }},{{ pxe_dhcp_end }},{{ pxe_dhcp_lease }} # No default gateway (option 3) and no DNS (option 6) handed out: # the PXE network is isolated and the PXE server does not forward @@ -576,6 +682,37 @@ mode: '0644' backup: yes + # The second layer under interface=: dnsmasq offers only on an interface + # with an address inside a dhcp-range. With every range in the imaging + # subnet, a GE port (10.x) matches none even if the pin were lost. A + # proxy range or one outside the subnet would remove that layer. + - name: "dnsmasq: every dhcp-range is in the imaging subnet, none is proxy" + shell: | + set -u + bad=0 + # Files the dnsmasq conf-dir skips (dot files, ~ backups, .dpkg-*, + # .bak) are skipped here too, so a backup never refuses the run. + FILES=(/etc/dnsmasq.conf) + for f in /etc/dnsmasq.d/*; do + b=${f##*/} + case "$b" in .*|*~|*.dpkg-*|*.bak|*.bak.*) continue ;; esac + [ -f "$f" ] && FILES+=("$f") + done + while IFS= read -r line; do + range=${line#*dhcp-range=} + first=$(printf '%s' "$range" | tr ',' '\n' | grep -Em1 '^[0-9]+(\.[0-9]+){3}$') + if printf '%s' "$range" | grep -qi proxy; then + echo "REFUSED proxy DHCP: $line" >&2; bad=1 + elif [ -z "$first" ] || ! python3 -c 'import ipaddress,sys; sys.exit(ipaddress.ip_address(sys.argv[1]) not in ipaddress.ip_network(sys.argv[2]))' "$first" "{{ pxe_subnet }}"; then + echo "REFUSED dhcp-range outside {{ pxe_subnet }}: $line" >&2; bad=1 + fi + done < <(grep -Hs '^[[:space:]]*dhcp-range=' "${FILES[@]}" | sed 's/[[:space:]]*#.*//') + [ "$bad" -eq 0 ] || exit 1 + echo "dhcp-range ok" + args: + executable: /bin/bash + changed_when: false + # --- NIC guard: keep dnsmasq off the GE corporate network --- # pxe-nic-check classifies each NIC from what it hears (LLDP/CDP/802.1X # from managed switches, foreign DHCP offers and their domain). The @@ -1927,7 +2064,21 @@ loop: "{{ ufw_corp_rules }}" loop_control: label: "{{ corp_nic_name }} {{ item.port }}/{{ item.proto }}" - when: corp_admin_cidrs | length == 0 + when: corp_admin_cidrs | length == 0 and corp_nic_mac | length > 0 + + # No pinned GE NIC (corp_nic_mac empty): the GE port has no fixed name, + # so SSH and the webapp are allowed by source, on any interface. The + # imaging NIC already allows both, so this opens nothing new there. + - name: "UFW: allow SSH and the webapp from all of GE (no pinned GE NIC)" + ufw: + rule: allow + direction: in + port: "{{ item.port }}" + proto: "{{ item.proto }}" + loop: "{{ ufw_corp_rules }}" + loop_control: + label: "any {{ item.port }}/{{ item.proto }}" + when: corp_admin_cidrs | length == 0 and corp_nic_mac | length == 0 - name: "UFW: WARNING, SSH and the root webapp are open to all of GE" debug: @@ -1935,7 +2086,7 @@ - "!!! WARNING: corp_admin_cidrs is empty. SSH (password auth, shared" - "!!! 'pxe' password) and the webapp on 9009 (no login, runs as root)" - "!!! accept connections from ANY host on the GE corporate network via" - - "!!! {{ corp_nic_name }}. Set corp_admin_cidrs in playbook/site-vars.yml" + - "!!! {{ corp_nic_name if corp_nic_mac else 'any GE port' }}. Set corp_admin_cidrs in playbook/site-vars.yml" - "!!! to the admin subnets (docs/SITE-CUSTOMIZATION.md) and re-run." when: corp_admin_cidrs | length == 0 @@ -1950,7 +2101,19 @@ loop: "{{ ufw_corp_rules | product(corp_admin_cidrs) | list }}" loop_control: label: "{{ corp_nic_name }} from {{ item.1 }} {{ item.0.port }}/{{ item.0.proto }}" - when: corp_admin_cidrs | length > 0 + when: corp_admin_cidrs | length > 0 and corp_nic_mac | length > 0 + + - name: "UFW: allow SSH and the webapp from corp_admin_cidrs only (no pinned GE NIC)" + ufw: + rule: allow + direction: in + from_ip: "{{ item.1 }}" + port: "{{ item.0.port }}" + proto: "{{ item.0.proto }}" + loop: "{{ ufw_corp_rules | product(corp_admin_cidrs) | list }}" + loop_control: + label: "any from {{ item.1 }} {{ item.0.port }}/{{ item.0.proto }}" + when: corp_admin_cidrs | length > 0 and corp_nic_mac | length == 0 - name: "UFW: drop the any-source rules on the GE corporate NIC (restricted now)" ufw: @@ -1963,7 +2126,7 @@ loop: "{{ ufw_corp_rules }}" loop_control: label: "{{ corp_nic_name }} any {{ item.port }}/{{ item.proto }}" - when: corp_admin_cidrs | length > 0 + when: corp_admin_cidrs | length > 0 and corp_nic_mac | length > 0 # Only drop the old any-interface rules (allow 22 among them) once both # named NICs exist on the right MACs and the per-interface SSH rules are @@ -1975,29 +2138,46 @@ shell: | set -u bad=0 - for pair in "{{ pxe_nic_name }}={{ pxe_nic_mac | lower }}" "{{ corp_nic_name }}={{ corp_nic_mac | lower }}"; do + PAIRS=("{{ pxe_nic_name }}={{ pxe_nic_mac | lower }}") + NAMES=("{{ pxe_nic_name }}") + {% if corp_nic_mac %} + PAIRS+=("{{ corp_nic_name }}={{ corp_nic_mac | lower }}") + NAMES+=("{{ corp_nic_name }}") + {% endif %} + for pair in "${PAIRS[@]}"; do n=${pair%%=*}; m=${pair#*=} if [ "$(cat "/sys/class/net/$n/address" 2>/dev/null)" != "$m" ]; then echo "ERROR: interface $n with MAC $m is not present" >&2; bad=1 fi done added=$(ufw show added 2>/dev/null) - for n in "{{ pxe_nic_name }}" "{{ corp_nic_name }}"; do + for n in "${NAMES[@]}"; do if ! grep -E "allow in on $n( |$)" <<<"$added" | grep -qE 'port 22 proto tcp'; then echo "ERROR: no UFW 'allow in on $n ... port 22 proto tcp' rule" >&2; bad=1 fi done + {% if not corp_nic_mac %} + # No pinned GE NIC: GE-side SSH is an any-interface rule, by source. + if ! grep -E '^ufw allow' <<<"$added" | grep -v ' on ' | grep -qE '(port 22( |$)| 22/tcp( |$))'; then + echo "ERROR: no any-interface UFW rule for port 22 (GE side)" >&2; bad=1 + fi + {% endif %} [ "$bad" -eq 0 ] || { echo "the old any-interface rules are kept" >&2; exit 1; } echo "both NICs present, per-interface SSH rules in place" register: ufw_verify changed_when: false + # With no pinned GE NIC and corp_admin_cidrs empty, "allow 22" and + # "allow 9009" on any interface ARE the GE-side rules: keep them. - name: "UFW: remove the old any-interface rules" ufw: rule: allow port: "{{ item.port }}" proto: "{{ item.proto }}" delete: yes + when: >- + corp_nic_mac | length > 0 or corp_admin_cidrs | length > 0 + or item.port not in (ufw_corp_rules | map(attribute='port') | list) loop: - { port: "22", proto: tcp } - { port: "67", proto: udp } diff --git a/playbook/site-vars.example.yml b/playbook/site-vars.example.yml index b1d19bd..0328ce2 100644 --- a/playbook/site-vars.example.yml +++ b/playbook/site-vars.example.yml @@ -4,16 +4,22 @@ # loads it on first boot; -e on the command line still overrides it. # site-vars.yml is git-ignored: each site's MACs stay with that site. # -# REQUIRED: the two NICs, by MAC (ip -br link on the box, or the label). -# imaging NIC = cable to the isolated imaging switch (serves DHCP/PXE/SMB) -# corp NIC = cable to the GE corporate network (SSH + webapp 9009 only) -# The playbook changes nothing on the network until both are set, differ, -# and are both present on the box. +# The NICs, by MAC (ip -br link on the box, or the label). Both optional: +# pxe_nic_mac imaging NIC = cable to the isolated imaging switch (serves +# DHCP/PXE/SMB). Empty: the first boot picks the one wired +# port with a link, if pxe-nic-check calls it isolated, and +# pins it. So during the install plug in ONLY the imaging +# switch. If it cannot tell, the console says to log in and +# run: sudo pxe-pick-nic (pull the imaging cable, plug it in). +# corp_nic_mac GE corporate NIC (SSH + webapp 9009). Empty: every other +# wired port is a DHCP client, so GE can be plugged into any +# of them, now or later. Set: only that port, named corp0. pxe_nic_mac: "" # e.g. "34:c8:d6:b1:10:10" corp_nic_mac: "" # e.g. "20:88:10:af:0d:e5" -# Optional: stable interface names (default pxe0 / corp0). Keep a box's -# existing names instead, e.g. on the WJ server: +# Optional: stable interface names (default pxe0 / corp0). With corp_nic_mac +# empty, pxe_nic_name must not start with en or eth. With both MACs set a +# box may keep its existing names, e.g. on the WJ server: # pxe_nic_name: enx34c8d6b11010 # corp_nic_name: enp0s31f6 diff --git a/playbook/templates/dnsmasq-pxe-nic-only.conf.j2 b/playbook/templates/dnsmasq-pxe-nic-only.conf.j2 index f2f747d..2d3c4be 100644 --- a/playbook/templates/dnsmasq-pxe-nic-only.conf.j2 +++ b/playbook/templates/dnsmasq-pxe-nic-only.conf.j2 @@ -4,5 +4,11 @@ # port=0 / listen-address / bind-dynamic in /etc/dnsmasq.conf and the # pxe-nic-check guard (docs/pxe-nic-guard.md). interface={{ pxe_nic_name }} +{% if corp_nic_mac %} except-interface={{ corp_nic_name }} no-dhcp-interface={{ corp_nic_name }} +{% else %} +# No pinned GE NIC (corp_nic_mac empty): every other wired port is a DHCP +# client with no fixed name. interface= alone keeps dnsmasq off them, and +# its only dhcp-range is the imaging subnet, so a GE address matches none. +{% endif %} diff --git a/playbook/templates/netplan-pxe.yaml.j2 b/playbook/templates/netplan-pxe.yaml.j2 index 1c0ecf3..6e5e3a6 100644 --- a/playbook/templates/netplan-pxe.yaml.j2 +++ b/playbook/templates/netplan-pxe.yaml.j2 @@ -3,6 +3,7 @@ # replaced on the next playbook run; the previous file is kept under # /var/backups/netplan-/. # +{% if corp_nic_mac %} # Two NICs, each matched by MAC and given a stable name: # pxe-lan imaging LAN, static, no gateway, no DNS. The ONLY NIC that # serves DHCP/TFTP/HTTP boot/SMB. ignore-carrier: the address @@ -11,6 +12,24 @@ # Apache without {{ pxe_server_ip }} (networkd # ConfigureWithoutCarrier= + IgnoreCarrierLoss=). # ge-corp GE corporate network, DHCP client (SSH + webapp 9009, updates). +{% else %} +# The imaging NIC pinned by MAC, every other wired port a DHCP client: +# pxe-lan imaging LAN, matched by MAC and given a stable name. Static, +# no gateway, no DNS. The ONLY NIC that serves DHCP/TFTP/HTTP +# boot/SMB. ignore-carrier: the address is put on (and kept) +# without link, so a switch that is off or a cable pulled at +# boot does not leave dnsmasq, Samba and Apache without +# {{ pxe_server_ip }} (networkd ConfigureWithoutCarrier= + +# IgnoreCarrierLoss=). +# zz-en, zz-eth every other wired port: DHCP client, so the GE cable +# can go into any of them, at install or later. No set-name. +# networkd applies the first matching .network file in name +# order: 10-netplan-pxe-lan sorts before 10-netplan-zz-*, and +# {{ pxe_nic_name }} does not look like en*/eth* (the playbook +# checks), so the imaging NIC never falls into these. dnsmasq +# never serves them (interface= pins it to {{ pxe_nic_name }}), +# and a drop-in makes them ignore offers from {{ pxe_server_ip }}. +{% endif %} network: version: 2 renderer: networkd @@ -25,6 +44,7 @@ network: optional: true addresses: - {{ pxe_server_ip }}/{{ pxe_prefix }} +{% if corp_nic_mac %} ge-corp: match: macaddress: "{{ corp_nic_mac | lower }}" @@ -32,3 +52,15 @@ network: dhcp4: true dhcp4-overrides: route-metric: {{ corp_route_metric }} +{% else %} +{% for id, glob in [('zz-en', 'en*'), ('zz-eth', 'eth*')] %} + {{ id }}: + match: + name: "{{ glob }}" + dhcp4: true + dhcp6: false + optional: true + dhcp4-overrides: + route-metric: {{ corp_route_metric }} +{% endfor %} +{% endif %} diff --git a/scripts/build-usb.sh b/scripts/build-usb.sh index a5ec14b..60fd4e5 100755 --- a/scripts/build-usb.sh +++ b/scripts/build-usb.sh @@ -34,12 +34,12 @@ # Engineering Layer...: scripts/capture-site-content.sh). # Without it a new server cannot build standard or # engineer until that content is copied over by hand. -# --site-vars FILE put FILE on CIDATA as playbook/site-vars.yml (the -# per-site NIC MACs; see playbook/site-vars.example.yml) -# --no-site-vars build without site vars (the first-boot playbook then -# stops until they are set on the server). One of the -# two is required; playbook/site-vars.yml in the repo -# is never copied implicitly. +# --site-vars FILE put FILE on CIDATA as playbook/site-vars.yml +# (per-site values; see playbook/site-vars.example.yml) +# --no-site-vars build without site vars (defaults; the first boot +# picks the imaging NIC by its link). One of the two +# is required; playbook/site-vars.yml in the repo is +# never copied implicitly. # --yes do not ask for confirmation (device mode still asks # YES for a non-removable device) # @@ -205,8 +205,9 @@ if [ -n "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 1 ]; then exit 1 fi if [ -z "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 0 ]; then - echo "ERROR: give --site-vars FILE (the server's NIC MACs, see playbook/site-vars.example.yml)" - echo " or --no-site-vars to build a USB whose first boot stops until they are set." + echo "ERROR: give --site-vars FILE (per-site values, see playbook/site-vars.example.yml)" + echo " or --no-site-vars (the first boot then picks the imaging NIC itself and uses" + echo " the defaults for everything else)." exit 1 fi if [ -n "$SITE_VARS" ]; then @@ -214,9 +215,18 @@ if [ -n "$SITE_VARS" ]; then echo "ERROR: --site-vars file not found: $SITE_VARS" exit 1 fi + # Both MACs are optional: an empty pxe_nic_mac makes the first boot pick + # the imaging NIC by its link (only the imaging switch plugged in), an + # empty corp_nic_mac makes every other port a DHCP client. One that is + # set must be a MAC. for v in pxe_nic_mac corp_nic_mac; do - if ! grep -qE "^$v:[[:space:]]*[\"']?([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}[\"']?[[:space:]]*(#.*)?$" "$SITE_VARS"; then - echo "ERROR: $SITE_VARS does not set $v to a MAC (aa:bb:cc:dd:ee:ff)" + line=$(grep -E "^$v:" "$SITE_VARS" | tail -1) + [ -n "$line" ] || continue + if grep -qE "^$v:[[:space:]]*([\"'][\"'])?[[:space:]]*(#.*)?$" <<<"$line"; then + continue + fi + if ! grep -qE "^$v:[[:space:]]*[\"']?([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}[\"']?[[:space:]]*(#.*)?$" <<<"$line"; then + echo "ERROR: $SITE_VARS sets $v to something that is not a MAC (aa:bb:cc:dd:ee:ff) or empty" exit 1 fi done diff --git a/tests/vm/build-image.sh b/tests/vm/build-image.sh index f4a0384..ee4e311 100755 --- a/tests/vm/build-image.sh +++ b/tests/vm/build-image.sh @@ -2,18 +2,26 @@ # Build the installer USB as a disk image FILE with build-usb.sh image mode, # then upload it into the libvirt 'default' pool as volume pxetest-usb.img. # -# Usage: tests/vm/build-image.sh [--with-enrollment] [--with-boot-tools] [--size SIZE] -# Default skips enrollment/, bios-staging/, drivers-staging/ and boot-tools/ -# to keep the image small; SIZE defaults to "auto". +# Usage: tests/vm/build-image.sh [--with-enrollment] [--with-boot-tools] +# [--with-site-content] [--pick-nic] [--size SIZE] +# Default skips enrollment/, bios-staging/, drivers-staging/, boot-tools/ +# and site-content/ to keep the image small; SIZE defaults to "auto". +# --pick-nic leaves both MACs out of site-vars: the first boot picks the +# imaging NIC by its link and every other port is a DHCP client (start the +# server with its GE NIC unplugged: virsh domif-setlink pxetest-server +# 52:54:00:9e:10:01 down --config). # Env: PXETEST_PAYLOAD_ROOT (where offline-packages/, boot-files/... live) source "$(dirname "$0")/common.sh" SIZE=auto -SKIPS=(--skip-enrollment --skip-bios --skip-drivers --skip-boot-tools) +SKIPS=(--skip-enrollment --skip-bios --skip-drivers --skip-boot-tools --skip-site-content) +PICK_NIC=0 while [ $# -gt 0 ]; do case "$1" in --with-enrollment) SKIPS=("${SKIPS[@]/--skip-enrollment}"); shift ;; --with-boot-tools) SKIPS=("${SKIPS[@]/--skip-boot-tools}"); shift ;; + --with-site-content) SKIPS=("${SKIPS[@]/--skip-site-content}"); shift ;; + --pick-nic) PICK_NIC=1; shift ;; --size) SIZE="$2"; shift 2 ;; *) die "unknown option $1" ;; esac @@ -43,13 +51,24 @@ fi # Site vars for the VM "site": pin the NICs by the fixed MACs from common.sh. SITE_VARS="$PXETEST_CACHE/site-vars.yml" -cat >"$SITE_VARS" <"$SITE_VARS" <"$SITE_VARS" <