ead3314f58
Every server has an imaging NIC (static, the only one that serves DHCP/TFTP/HTTP boot/SMB) and a GE corporate NIC (DHCP client, SSH and the webapp on 9009). The playbook guessed the PXE NIC and the installer put 172.16.9.1/24 on every en* NIC, so a two-NIC box could answer on the GE network. - New per-site vars pxe_nic_mac / corp_nic_mac (required, from playbook/site-vars.yml or -e; template site-vars.example.yml), stable names pxe_nic_name / corp_nic_name (default pxe0 / corp0, overridable), pxe_server_ip, pxe_subnet, DHCP range. pxe_imaging_mac / pxe_ge_mac are derived from them. The play stops before touching anything if a MAC is unset, malformed, duplicated or not on the box (lists the box's NICs). - Heuristic NIC detection and the single-NIC netplan task replaced by a two-NIC netplan template (match macaddress + set-name), installed by pxe-netplan-apply.sh: validate in a scratch root, back up, generate, rename, apply, restore the previous config on any failure. cloud-init network config disabled so it cannot rewrite it. - dnsmasq: /etc/dnsmasq.d/pxe-nic-only.conf from the vars (interface=, except-interface=, no-dhcp-interface=); dnsmasq.conf IP/range from vars. - UFW per interface: imaging NIC 22, 67/udp, 69/udp, 80, 4433, 445, 139, 137/udp, 138/udp, 9009; GE NIC 22 and 9009 only. Old any-interface rules and the stale 9011 rule removed after the new ones are in. - sysctl drop-in: IPv4/IPv6 forwarding off; ip_nonlocal_bind so Apache can bind the imaging IP before the address is up. - Samba [global]: interfaces = 127.0.0.0/8 <imaging NIC>, bind interfaces only. Apache 80 and 4433 on the imaging IP only; 9009 stays on all (TODO: the webapp has no auth). - autoinstall: DHCP only on en*, no static 172.16.9.1; apt geoip off and offline-install fallback now that the corp NIC may have a lease. - Blancco UEFI/Secure Boot: blancco_kver (7.0.0-34-generic) drives the kernel, initramfs and modules builds instead of the running kernel; outputs vmlinuz-ubuntu-70, kexec-initrd-70.img, kmod-<kver>.tar.gz as the menu and blancco-init.sh expect; creates: guards keyed on the full kver. Ubuntu's shimx64.efi.signed published as blancco/shimx64.efi. download-packages.sh adds shim-signed, linux-generic-hwe-24.04 and the pinned linux-image/modules/modules-extra for BLANCCO_KVER. Rendered with the live MACs and names, netplan generate output is identical to the live config; pxe-nic-only.conf and the Samba binding match live directive for directive.
162 lines
8.3 KiB
Markdown
162 lines
8.3 KiB
Markdown
# PXE NIC guard
|
|
|
|
The PXE server's DHCP/PXE (dnsmasq) must never serve on the GE corporate
|
|
network. Each server has two NICs:
|
|
|
|
| Role | Network | Address |
|
|
|---------|-----------------------------|----------------------------------|
|
|
| imaging | isolated imaging switch | static 172.16.9.1/24, dnsmasq |
|
|
| ge | GE corporate network | DHCP client (systemd-networkd) |
|
|
|
|
`/etc/dnsmasq.d/pxe-nic-only.conf` pins dnsmasq to the imaging NIC by name.
|
|
That stops dnsmasq answering on the GE NIC, but not on the imaging NIC when
|
|
that NIC itself ends up on the wrong network: a cable swap, an imaging
|
|
switch uplinked into a GE port, or a second PXE server on the same switch.
|
|
The NIC guard catches those.
|
|
|
|
## What is installed
|
|
|
|
| File | Purpose |
|
|
|-----------------------------------------------------------|---------------------------------|
|
|
| `/usr/local/sbin/pxe-nic-check` | the classifier |
|
|
| `/etc/pxe-server/nic.conf` | which NIC is which |
|
|
| `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start|
|
|
| `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min |
|
|
| `/run/pxe-nic-status.json` | last result, read by the webapp |
|
|
|
|
Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml`
|
|
(vars `pxe_nic_mac`, `corp_nic_mac`, `pxe_corp_domain_suffixes`,
|
|
`pxe_nic_allowed_switches`; `pxe_imaging_mac` / `pxe_ge_mac` are derived
|
|
from the first two). The same MACs pin the NICs in netplan
|
|
(`playbook/templates/netplan-pxe.yaml.j2`, names `pxe0` / `corp0`) and in
|
|
`/etc/dnsmasq.d/pxe-nic-only.conf`.
|
|
|
|
## How it decides
|
|
|
|
It listens on each NIC (raw socket, sends nothing) for:
|
|
|
|
- LLDP and CDP: managed switches announce themselves. GE switches do; the
|
|
unmanaged imaging switch does not.
|
|
- 802.1X EAPOL: GE ports ask for port authentication.
|
|
- DHCP OFFER/ACK from any other server.
|
|
- STP and the count of distinct ARP hosts: reported, never decisive.
|
|
|
|
On the imaging NIC it also sends one DHCPDISCOVER and collects offers for
|
|
6 s. On the GE NIC it reads the networkd lease instead (no traffic).
|
|
|
|
| Verdict | Meaning |
|
|
|----------------|------------------------------------------------------------|
|
|
| `isolated` | no foreign DHCP, no LLDP/CDP/EAPOL |
|
|
| `ge` | LLDP, CDP or EAPOL seen, or a DHCP server whose domain ends in a corp suffix (`geaerospace.net`, `ge.com`; whole labels, so `notgeaerospace.net` does not count) |
|
|
| `foreign-dhcp` | another DHCP server, no corp domain |
|
|
| `no-link` | no carrier |
|
|
| `error` | the check itself failed |
|
|
|
|
Healthy: imaging NIC `isolated` (or `no-link`), GE NIC `ge`.
|
|
|
|
Our own dnsmasq never counts. An offer whose server id is our own address
|
|
but comes from a MAC that is not ours is a second PXE server and DOES count.
|
|
|
|
If the imaging switch is itself a managed switch that sends LLDP/CDP, put
|
|
its LLDP system name, CDP device id or chassis MAC in `allowed_switches`
|
|
(`pxe_nic_allowed_switches`), or the guard will block dnsmasq. 802.1X is
|
|
never allowed.
|
|
|
|
## Modes
|
|
|
|
- `pxe-nic-check` - check every NIC (60 s), print a summary. `--json`,
|
|
`--write`, `--probe` (imaging DHCP probe), `--iface NAME`, `--window S`.
|
|
- `pxe-nic-check --guard` - dnsmasq `ExecStartPre`. Imaging NIC only,
|
|
about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no
|
|
link exits 0 (fail open) with an error line in the journal.
|
|
- `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs
|
|
`systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq.
|
|
|
|
Every start or restart of dnsmasq runs the guard, so `systemctl restart
|
|
dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
|
|
|
|
## Reading it
|
|
|
|
sudo pxe-nic-check # full summary, 60 s
|
|
cat /run/pxe-nic-status.json # last guard/watch result
|
|
journalctl -u dnsmasq -b | grep pxe-nic-check
|
|
journalctl -u pxe-nic-check -b
|
|
|
|
## dnsmasq will not start
|
|
|
|
1. `journalctl -u dnsmasq -b | grep pxe-nic-check` shows the evidence, e.g.
|
|
`REFUSING to start dnsmasq: imaging NIC br-pxe verdict ge: LLDP from
|
|
WJ-IDF2 port Gi1/0/14`.
|
|
2. Check the imaging cabling: the imaging NIC must go to the imaging
|
|
switch only, and that switch must not be uplinked anywhere.
|
|
3. `sudo pxe-nic-check` until the imaging NIC reads `isolated`.
|
|
4. `sudo systemctl start dnsmasq` (runs the guard again).
|
|
|
|
## Floor tests (run with the floor idle)
|
|
|
|
Both tests stop imaging for a few minutes. Have console or GE-side SSH
|
|
access to the server; do not rely on the imaging network for the session.
|
|
|
|
### (a) Rogue DHCP on the imaging switch: expect the alarm
|
|
|
|
1. Confirm the baseline: `sudo pxe-nic-check --probe` shows the imaging
|
|
NIC `isolated` and `dnsmasq_bound` true.
|
|
2. Plug a device that runs a DHCP server into a free port of the IMAGING
|
|
switch (a spare home router with DHCP on, LAN port only, nothing in its
|
|
WAN port; or a laptop running dnsmasq).
|
|
3. Run the watch now instead of waiting up to 5 min:
|
|
`sudo systemctl start pxe-nic-check.service`
|
|
4. Expect:
|
|
- `journalctl -u pxe-nic-check -n 20` has a CRIT line `imaging NIC is
|
|
NOT isolated, stopping dnsmasq: ... verdict foreign-dhcp: DHCP from
|
|
192.168.x.1, ...` (or `ge` if the router sends LLDP).
|
|
- `systemctl is-active dnsmasq` is `inactive`.
|
|
- `/run/pxe-nic-status.json` has `"dhcp_guard": {"state": "blocked", ...}`.
|
|
5. Prove the guard too: `sudo systemctl start dnsmasq` must FAIL, and
|
|
`journalctl -u dnsmasq -n 20` shows `REFUSING to start dnsmasq`.
|
|
6. Unplug the rogue device. `sudo pxe-nic-check --probe` shows `isolated`.
|
|
`sudo systemctl start dnsmasq` starts. PXE-boot one client to confirm.
|
|
|
|
### (b) Cable swap into a GE port: expect the guard to refuse dnsmasq
|
|
|
|
dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE.
|
|
|
|
1. `sudo systemctl stop dnsmasq`; `systemctl is-active dnsmasq` must say
|
|
`inactive`. Do not reboot during the test (the `@reboot` cron restarts
|
|
dnsmasq; the guard would still block it, but do not test that way).
|
|
2. Move the IMAGING NIC's cable from the imaging switch into a live GE
|
|
network port. Wait for link: `ip -br link` shows the imaging NIC `UP`.
|
|
3. Run the guard by hand first:
|
|
`sudo pxe-nic-check --guard; echo "exit $?"`
|
|
Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP,
|
|
802.1X, or `DHCP from 10.x.x.x, domain ....geaerospace.net`.
|
|
This sends one DHCPDISCOVER on the GE port, the same as any PC being
|
|
plugged in. To send nothing, use `--no-probe --window 60` instead and
|
|
rely on LLDP/CDP/802.1X (switches announce every 30-60 s).
|
|
**If it exits 0, STOP: do not start dnsmasq. Move the cable back and
|
|
send the output of `sudo pxe-nic-check --probe --json`.**
|
|
4. Only after step 3 exited 1: `sudo systemctl start dnsmasq`. Expect
|
|
`Job for dnsmasq.service failed`; `systemctl is-active dnsmasq` says
|
|
`failed`; `journalctl -u dnsmasq -n 20` shows `REFUSING`.
|
|
5. Move the cable back to the imaging switch. `sudo pxe-nic-check --probe`
|
|
shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq`
|
|
starts. PXE-boot one client to confirm.
|
|
|
|
## Status file
|
|
|
|
`/run/pxe-nic-status.json`, mode 0644, replaced atomically:
|
|
|
|
{"checked_at": "2026-10-14T16:47:09-04:00", "duration_s": 60,
|
|
"dhcp_guard": {"state": "allowed"|"blocked"|"unknown", "reason": "..."},
|
|
"interfaces": [{"iface": "br-pxe", "mac": "aa:bb:..", "role": "imaging"|"ge"|"other",
|
|
"link": "up"|"down", "speed_mbps": 5000, "addrs": ["172.16.9.1/24"],
|
|
"verdict": "isolated"|"ge"|"foreign-dhcp"|"no-link"|"error", "evidence": ["..."],
|
|
"dhcp_servers": [{"server": "10.1.2.3", "domain": "x.geaerospace.net", "own": false}],
|
|
"switch": {"lldp": [{"system": "...", "port": "...", "vlan": null}],
|
|
"cdp": [{"device": "...", "port": "..."}], "eapol": false, "stp": false},
|
|
"arp_hosts": 3, "dnsmasq_bound": true}]}
|
|
|
|
`dnsmasq_bound` is null when it cannot be told (no `ss`). A bridge
|
|
(br-pxe) is reported as one interface; its ports are listened on too,
|
|
because LLDP and 802.1X frames stop at the port and never reach the bridge.
|