NICs: pick the imaging port at install, GE can go into any other port

Sites no longer need either MAC in site-vars.

pxe_nic_mac empty: before anything changes, the playbook runs
pxe-pick-nic --auto. It takes the one wired port with a link, accepts it
only if pxe-nic-check calls it isolated (no LLDP/CDP/802.1X, no other
DHCP server, at most 10 hosts heard in 65 s), and pins it. The choice is
saved in /etc/pxe-server/nic-pick.conf and never re-guessed at boot.
When it cannot tell (no link, two links, not isolated) nothing changes,
and the console banner says why and to run sudo pxe-pick-nic: pull the
imaging cable and plug it back in, and the port that blinked is the
imaging one. Both helpers are installed before the pick, so that works
after a refused first boot. The site's corp domains and allowed
switches are passed to the check.

corp_nic_mac empty: every other wired port (en*, eth*) is a catch-all
DHCP client, so GE can be plugged in at install or later.
- dnsmasq stays pinned with interface=pxe0. A new check refuses any
  dhcp-range outside the imaging subnet, and proxy ranges.
- The catch-all ports ignore offers from 172.16.9.1.
- UFW allows SSH and 9009 by source on any interface, and keeps those
  rules when the old any-interface rules are removed.
- pxe_nic_name must not look like en*/eth* in this mode.

pxe-netplan-apply.sh takes 3 or 5 args. Without a pinned GE NIC it:
- renames stray names (the old pxe0 after a re-pick, a leftover corp0)
  to enx<mac>, so the catch-all takes them, and restores them on revert;
- checks that every NIC that had a DHCP address before has one after,
  tracked by MAC.

With both MACs set (the WJ server), netplan, the dnsmasq pin and UFW
render byte-for-byte as before.

Tested in the pxetest VMs:
- Fresh install with no MACs and the GE link down: the imaging port was
  picked and pinned as pxe0 with 172.16.9.1, and dnsmasq bound DHCP to
  pxe0 only. GE plugged in afterwards got a lease, SSH worked, and the
  guard reported pxe0 isolated.
- --auto refuses with two links.
- The interactive pick works by pulling and replugging the link.
- A 3-arg re-apply is UNCHANGED.

Unit tests (fake sysfs) cover the re-pick and pinned-to-catch-all renames.
This commit is contained in:
cproudlock
2026-10-16 15:56:47 -04:00
parent 9af1c6d89a
commit b11e678297
12 changed files with 774 additions and 108 deletions

View File

@@ -73,19 +73,30 @@
# image roots (on a rebuild the _media views may not exist yet).
zscaler_ca_share_glob: "{{ samba_share }}/_media/*/Deploy/Applications/ZscalerCommercialCertificate-2048-SHA256.crt {{ samba_share }}/gea-*/Deploy/Applications/ZscalerCommercialCertificate-2048-SHA256.crt"
# --- The two NICs (site standard) ---
# Every server has two NICs: the imaging NIC (isolated imaging LAN,
# static pxe_server_ip, the ONLY NIC that serves DHCP/TFTP/HTTP boot/SMB)
# and the GE corporate NIC (DHCP client; SSH and the webapp on 9009 for
# admins, outbound updates through Zscaler). Both MACs are REQUIRED:
# set them in playbook/site-vars.yml on the USB (copy
# site-vars.example.yml) or with -e. The playbook stops before it
# touches anything if either is unset or not on this box. Read them
# with: ip -br link
# The imaging NIC (isolated imaging LAN, static pxe_server_ip, the ONLY
# NIC that serves DHCP/TFTP/HTTP boot/SMB) is pinned by MAC. Leave
# pxe_nic_mac empty and the first run picks it (pxe-pick-nic --auto):
# the one wired port with a link, accepted only if pxe-nic-check calls
# it isolated, saved in /etc/pxe-server/nic-pick.conf and pinned from
# then on. So during the install plug in ONLY the imaging switch. If it
# cannot tell, it stops and the console says to run sudo pxe-pick-nic
# (pull the imaging cable and plug it back in).
#
# The GE corporate NIC (DHCP client; SSH and the webapp on 9009) is
# optional:
# corp_nic_mac empty (default): every other wired port (en*, eth*) is
# a DHCP client, so GE can be plugged into any of them, then or
# later. dnsmasq never serves them (interface= pins it to the
# imaging NIC) and they ignore our own DHCP offers.
# corp_nic_mac set: that one port is pinned and named corp_nic_name,
# as on the WJ server; other ports stay unconfigured.
pxe_nic_mac: ""
corp_nic_mac: ""
# Stable names, set by netplan (match macaddress + set-name). dnsmasq,
# UFW and Samba refer to these. A site may keep its existing names
# (e.g. pxe_nic_name: enx34c8d6b11010, corp_nic_name: enp0s31f6).
# UFW and Samba refer to these. With corp_nic_mac empty they must not
# look like en*/eth* (the catch-all would match them by name). With both
# MACs pinned a site may keep its existing names (WJ: pxe_nic_name:
# enx34c8d6b11010, corp_nic_name: enp0s31f6).
pxe_nic_name: "pxe0"
corp_nic_name: "corp0"
# Imaging LAN addressing. NOTE: 172.16.9.1 is also hard-coded in the
@@ -101,6 +112,7 @@
corp_route_metric: 100
# NIC guard (pxe-nic-check): dnsmasq must never serve DHCP/PXE on the GE
# corporate network. These are derived from pxe_nic_mac / corp_nic_mac
# (pxe_ge_mac empty: the guard takes the NIC holding a DHCP lease)
# (one source of truth) and written to /etc/pxe-server/nic.conf; do not
# set them separately (the playbook fails if they disagree). A MAC of a
# bridge port resolves to the bridge.
@@ -210,29 +222,77 @@
- Tools
tasks:
# --- The two NICs: validate, then pin by MAC (netplan) ---
# --- The NICs: pick / validate, then pin by MAC (netplan) ---
# Replaces the old guess (first en* without 172.16.9.1 / the default
# route), which on a two-NIC box could pick the GE NIC.
# route), which on a two-NIC box could pick the GE NIC. The pick runs
# from the USB copies, before anything on the box changes, and only
# ever at install time: once pinned, nothing re-guesses at boot.
# Installed first (they change no network config): when the pick
# refuses, the play stops here and the console tells the operator to
# run sudo pxe-pick-nic, which needs both on the box (first-boot.sh
# unmounts the USB).
- name: "NICs: install pxe-nic-check and pxe-pick-nic before the pick"
copy:
src: "{{ usb_mount }}/pxe-server-helpers/{{ item }}"
dest: "/usr/local/sbin/{{ item }}"
mode: '0755'
loop: [pxe-nic-check, pxe-pick-nic]
- name: "NICs: pick the imaging NIC (pxe_nic_mac is not set in site-vars)"
command: /usr/local/sbin/pxe-pick-nic --auto
environment:
PXE_PICK_SUFFIXES: "{{ pxe_corp_domain_suffixes }}"
PXE_PICK_ALLOWED: "{{ pxe_nic_allowed_switches }}"
register: nic_pick
changed_when: false
failed_when: false
when: pxe_nic_mac | length == 0
- name: "NICs: pick result"
debug:
msg: "{{ nic_pick.stdout_lines }}"
when: pxe_nic_mac | length == 0
- name: "NICs: the imaging NIC could not be picked"
fail:
msg: >-
Could not pick the imaging NIC, nothing was changed:
{{ nic_pick.stdout_lines | select('search', 'REFUSED|verdict|gone|ERROR') | join(' ') }}
During the install plug in ONLY the imaging switch, or log in and
run: sudo pxe-pick-nic (pull the imaging cable and plug it back in).
Or set pxe_nic_mac in playbook/site-vars.yml.
when:
- pxe_nic_mac | length == 0
- nic_pick.rc != 0 or not (nic_pick.stdout_lines | last | default('') is match('^([0-9a-f]{2}:){5}[0-9a-f]{2}$'))
- name: "NICs: pin the picked imaging NIC"
set_fact:
pxe_nic_mac: "{{ nic_pick.stdout_lines | last }}"
when: pxe_nic_mac | length == 0
- name: "NICs: check the per-site NIC vars"
assert:
that:
- pxe_nic_mac | length > 0
- corp_nic_mac | length > 0
- pxe_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$')
- corp_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$')
- corp_nic_mac | length == 0 or corp_nic_mac is match('^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$')
- pxe_nic_mac | lower != corp_nic_mac | lower
- pxe_imaging_mac | lower == pxe_nic_mac | lower
- pxe_ge_mac | lower == corp_nic_mac | lower
- pxe_nic_name | length > 0 and pxe_nic_name | length <= 15
- corp_nic_name | length > 0 and corp_nic_name | length <= 15
- pxe_nic_name != corp_nic_name
- corp_nic_mac | length > 0 or not (pxe_nic_name is match('^(en|eth)'))
fail_msg: >-
Set pxe_nic_mac (imaging NIC) and corp_nic_mac (GE corporate NIC)
to two different MACs (aa:bb:cc:dd:ee:ff) in playbook/site-vars.yml
(copy site-vars.example.yml) or with -e, then re-run. Do not set
pxe_imaging_mac / pxe_ge_mac yourself; they follow these. Names
(pxe_nic_name, corp_nic_name) must differ and be 1-15 chars.
Now: pxe_nic_mac='{{ pxe_nic_mac }}' corp_nic_mac='{{ corp_nic_mac }}'.
pxe_nic_mac (imaging NIC) must be a MAC (aa:bb:cc:dd:ee:ff);
corp_nic_mac (GE NIC) empty or a different MAC. Set them in
playbook/site-vars.yml (copy site-vars.example.yml) or with -e,
then re-run. Do not set pxe_imaging_mac / pxe_ge_mac yourself;
they follow these (an -e pxe_nic_mac="" overrides the pick too:
drop it). Names (pxe_nic_name, corp_nic_name) must differ
and be 1-15 chars, and with corp_nic_mac empty pxe_nic_name must
not start with en or eth (the catch-all for the other ports
matches those). Now: pxe_nic_mac='{{ pxe_nic_mac }}'
corp_nic_mac='{{ corp_nic_mac }}' pxe_nic_name='{{ pxe_nic_name }}'.
quiet: yes
- name: "NICs: list the physical NICs on this box"
@@ -244,15 +304,15 @@
register: nic_list
changed_when: false
- name: "NICs: both MACs must be on this box"
- name: "NICs: the pinned MACs must be on this box"
vars:
box_macs: "{{ nic_list.stdout_lines | map('regex_replace', '^.* ', '') | map('lower') | list }}"
assert:
that:
- pxe_nic_mac | lower in box_macs
- corp_nic_mac | lower in box_macs
- corp_nic_mac | length == 0 or corp_nic_mac | lower in box_macs
fail_msg: >-
{{ 'pxe_nic_mac ' ~ pxe_nic_mac ~ ' is not on this box. ' if pxe_nic_mac | lower not in box_macs else '' }}{{ 'corp_nic_mac ' ~ corp_nic_mac ~ ' is not on this box. ' if corp_nic_mac | lower not in box_macs else '' }}Nothing was changed (the network config would have left that NIC
{{ 'pxe_nic_mac ' ~ pxe_nic_mac ~ ' is not on this box. ' if pxe_nic_mac | lower not in box_macs else '' }}{{ 'corp_nic_mac ' ~ corp_nic_mac ~ ' is not on this box. ' if corp_nic_mac and corp_nic_mac | lower not in box_macs else '' }}Nothing was changed (the network config would have left that NIC
unconfigured). Physical NICs here (name MAC):
{{ nic_list.stdout_lines | join('; ') }}.
Fix playbook/site-vars.yml and re-run.
@@ -396,7 +456,7 @@
bash {{ usb_mount }}/pxe-server-helpers/pxe-netplan-apply.sh --preflight
/run/pxe-server-netplan.check.yaml
{{ pxe_nic_name }} {{ pxe_nic_mac | lower }}
{{ corp_nic_name }} {{ corp_nic_mac | lower }}
{{ (corp_nic_name ~ ' ' ~ corp_nic_mac | lower) if corp_nic_mac else '' }}
environment:
PXE_SUBNET: "{{ pxe_subnet }}"
PXE_SSH_CLIENT: "{{ pxe_ssh_client }}"
@@ -439,11 +499,57 @@
dest: /etc/pxe-server-netplan.yaml
mode: '0600'
# The catch-all ports (corp_nic_mac empty) are DHCP clients on whatever
# they are cabled to. Cabled to the imaging switch by mistake, they would
# take an address from our own dnsmasq and put a second interface in the
# imaging subnet; DenyList makes them ignore offers from pxe_server_ip.
- name: "NICs: drop-in folders for the catch-all ports"
file:
path: "/etc/systemd/network/10-netplan-{{ item }}.network.d"
state: directory
mode: '0755'
loop: [zz-en, zz-eth]
when: corp_nic_mac | length == 0
- name: "NICs: catch-all ports ignore our own DHCP offers"
copy:
dest: "/etc/systemd/network/10-netplan-{{ item }}.network.d/90-pxe-deny-own-dhcp.conf"
mode: '0644'
content: |
# Managed by pxe_server_setup.yml. Never take a lease from this
# server's own dnsmasq (an imaging switch on a non-imaging port).
[DHCPv4]
DenyList={{ pxe_server_ip }}
loop: [zz-en, zz-eth]
when: corp_nic_mac | length == 0
- name: "NICs: no catch-all drop-ins with a pinned GE NIC"
file:
path: "/etc/systemd/network/10-netplan-{{ item }}.network.d"
state: absent
loop: [zz-en, zz-eth]
when: corp_nic_mac | length > 0
- name: "NICs: create /etc/pxe-server"
file:
path: /etc/pxe-server
state: directory
mode: '0755'
- name: "NICs: record the pinned imaging NIC for pxe-pick-nic --show"
copy:
dest: /etc/pxe-server/nic-pick.conf
mode: '0644'
content: |
# Written by pxe_server_setup.yml. The imaging NIC the playbook pins.
imaging_mac={{ pxe_nic_mac | lower }}
when: nic_pick is not skipped and nic_pick.rc == 0
- name: "NICs: validate, install and apply netplan (rolls back on failure)"
command: >-
/usr/local/sbin/pxe-netplan-apply.sh /etc/pxe-server-netplan.yaml
{{ pxe_nic_name }} {{ pxe_nic_mac | lower }}
{{ corp_nic_name }} {{ corp_nic_mac | lower }}
{{ (corp_nic_name ~ ' ' ~ corp_nic_mac | lower) if corp_nic_mac else '' }}
environment:
PXE_SUBNET: "{{ pxe_subnet }}"
PXE_SSH_CLIENT: "{{ pxe_ssh_client }}"
@@ -520,7 +626,7 @@
listen-address={{ pxe_server_ip }}
bind-dynamic
# Interface pinning (serve {{ pxe_nic_name }} only, never
# {{ corp_nic_name }}) is in /etc/dnsmasq.d/pxe-nic-only.conf.
# {{ corp_nic_name if corp_nic_mac else 'the other ports' }}) is in /etc/dnsmasq.d/pxe-nic-only.conf.
dhcp-range={{ pxe_dhcp_start }},{{ pxe_dhcp_end }},{{ pxe_dhcp_lease }}
# No default gateway (option 3) and no DNS (option 6) handed out:
# the PXE network is isolated and the PXE server does not forward
@@ -576,6 +682,37 @@
mode: '0644'
backup: yes
# The second layer under interface=: dnsmasq offers only on an interface
# with an address inside a dhcp-range. With every range in the imaging
# subnet, a GE port (10.x) matches none even if the pin were lost. A
# proxy range or one outside the subnet would remove that layer.
- name: "dnsmasq: every dhcp-range is in the imaging subnet, none is proxy"
shell: |
set -u
bad=0
# Files the dnsmasq conf-dir skips (dot files, ~ backups, .dpkg-*,
# .bak) are skipped here too, so a backup never refuses the run.
FILES=(/etc/dnsmasq.conf)
for f in /etc/dnsmasq.d/*; do
b=${f##*/}
case "$b" in .*|*~|*.dpkg-*|*.bak|*.bak.*) continue ;; esac
[ -f "$f" ] && FILES+=("$f")
done
while IFS= read -r line; do
range=${line#*dhcp-range=}
first=$(printf '%s' "$range" | tr ',' '\n' | grep -Em1 '^[0-9]+(\.[0-9]+){3}$')
if printf '%s' "$range" | grep -qi proxy; then
echo "REFUSED proxy DHCP: $line" >&2; bad=1
elif [ -z "$first" ] || ! python3 -c 'import ipaddress,sys; sys.exit(ipaddress.ip_address(sys.argv[1]) not in ipaddress.ip_network(sys.argv[2]))' "$first" "{{ pxe_subnet }}"; then
echo "REFUSED dhcp-range outside {{ pxe_subnet }}: $line" >&2; bad=1
fi
done < <(grep -Hs '^[[:space:]]*dhcp-range=' "${FILES[@]}" | sed 's/[[:space:]]*#.*//')
[ "$bad" -eq 0 ] || exit 1
echo "dhcp-range ok"
args:
executable: /bin/bash
changed_when: false
# --- NIC guard: keep dnsmasq off the GE corporate network ---
# pxe-nic-check classifies each NIC from what it hears (LLDP/CDP/802.1X
# from managed switches, foreign DHCP offers and their domain). The
@@ -1927,7 +2064,21 @@
loop: "{{ ufw_corp_rules }}"
loop_control:
label: "{{ corp_nic_name }} {{ item.port }}/{{ item.proto }}"
when: corp_admin_cidrs | length == 0
when: corp_admin_cidrs | length == 0 and corp_nic_mac | length > 0
# No pinned GE NIC (corp_nic_mac empty): the GE port has no fixed name,
# so SSH and the webapp are allowed by source, on any interface. The
# imaging NIC already allows both, so this opens nothing new there.
- name: "UFW: allow SSH and the webapp from all of GE (no pinned GE NIC)"
ufw:
rule: allow
direction: in
port: "{{ item.port }}"
proto: "{{ item.proto }}"
loop: "{{ ufw_corp_rules }}"
loop_control:
label: "any {{ item.port }}/{{ item.proto }}"
when: corp_admin_cidrs | length == 0 and corp_nic_mac | length == 0
- name: "UFW: WARNING, SSH and the root webapp are open to all of GE"
debug:
@@ -1935,7 +2086,7 @@
- "!!! WARNING: corp_admin_cidrs is empty. SSH (password auth, shared"
- "!!! 'pxe' password) and the webapp on 9009 (no login, runs as root)"
- "!!! accept connections from ANY host on the GE corporate network via"
- "!!! {{ corp_nic_name }}. Set corp_admin_cidrs in playbook/site-vars.yml"
- "!!! {{ corp_nic_name if corp_nic_mac else 'any GE port' }}. Set corp_admin_cidrs in playbook/site-vars.yml"
- "!!! to the admin subnets (docs/SITE-CUSTOMIZATION.md) and re-run."
when: corp_admin_cidrs | length == 0
@@ -1950,7 +2101,19 @@
loop: "{{ ufw_corp_rules | product(corp_admin_cidrs) | list }}"
loop_control:
label: "{{ corp_nic_name }} from {{ item.1 }} {{ item.0.port }}/{{ item.0.proto }}"
when: corp_admin_cidrs | length > 0
when: corp_admin_cidrs | length > 0 and corp_nic_mac | length > 0
- name: "UFW: allow SSH and the webapp from corp_admin_cidrs only (no pinned GE NIC)"
ufw:
rule: allow
direction: in
from_ip: "{{ item.1 }}"
port: "{{ item.0.port }}"
proto: "{{ item.0.proto }}"
loop: "{{ ufw_corp_rules | product(corp_admin_cidrs) | list }}"
loop_control:
label: "any from {{ item.1 }} {{ item.0.port }}/{{ item.0.proto }}"
when: corp_admin_cidrs | length > 0 and corp_nic_mac | length == 0
- name: "UFW: drop the any-source rules on the GE corporate NIC (restricted now)"
ufw:
@@ -1963,7 +2126,7 @@
loop: "{{ ufw_corp_rules }}"
loop_control:
label: "{{ corp_nic_name }} any {{ item.port }}/{{ item.proto }}"
when: corp_admin_cidrs | length > 0
when: corp_admin_cidrs | length > 0 and corp_nic_mac | length > 0
# Only drop the old any-interface rules (allow 22 among them) once both
# named NICs exist on the right MACs and the per-interface SSH rules are
@@ -1975,29 +2138,46 @@
shell: |
set -u
bad=0
for pair in "{{ pxe_nic_name }}={{ pxe_nic_mac | lower }}" "{{ corp_nic_name }}={{ corp_nic_mac | lower }}"; do
PAIRS=("{{ pxe_nic_name }}={{ pxe_nic_mac | lower }}")
NAMES=("{{ pxe_nic_name }}")
{% if corp_nic_mac %}
PAIRS+=("{{ corp_nic_name }}={{ corp_nic_mac | lower }}")
NAMES+=("{{ corp_nic_name }}")
{% endif %}
for pair in "${PAIRS[@]}"; do
n=${pair%%=*}; m=${pair#*=}
if [ "$(cat "/sys/class/net/$n/address" 2>/dev/null)" != "$m" ]; then
echo "ERROR: interface $n with MAC $m is not present" >&2; bad=1
fi
done
added=$(ufw show added 2>/dev/null)
for n in "{{ pxe_nic_name }}" "{{ corp_nic_name }}"; do
for n in "${NAMES[@]}"; do
if ! grep -E "allow in on $n( |$)" <<<"$added" | grep -qE 'port 22 proto tcp'; then
echo "ERROR: no UFW 'allow in on $n ... port 22 proto tcp' rule" >&2; bad=1
fi
done
{% if not corp_nic_mac %}
# No pinned GE NIC: GE-side SSH is an any-interface rule, by source.
if ! grep -E '^ufw allow' <<<"$added" | grep -v ' on ' | grep -qE '(port 22( |$)| 22/tcp( |$))'; then
echo "ERROR: no any-interface UFW rule for port 22 (GE side)" >&2; bad=1
fi
{% endif %}
[ "$bad" -eq 0 ] || { echo "the old any-interface rules are kept" >&2; exit 1; }
echo "both NICs present, per-interface SSH rules in place"
register: ufw_verify
changed_when: false
# With no pinned GE NIC and corp_admin_cidrs empty, "allow 22" and
# "allow 9009" on any interface ARE the GE-side rules: keep them.
- name: "UFW: remove the old any-interface rules"
ufw:
rule: allow
port: "{{ item.port }}"
proto: "{{ item.proto }}"
delete: yes
when: >-
corp_nic_mac | length > 0 or corp_admin_cidrs | length > 0
or item.port not in (ufw_corp_rules | map(attribute='port') | list)
loop:
- { port: "22", proto: tcp }
- { port: "67", proto: udp }