NIC guard: dnsmasq never serves DHCP/PXE on the GE network
pxe-nic-check classifies each NIC from what it hears: LLDP/CDP/802.1X from managed switches, foreign DHCP offers and their domain (one DHCPDISCOVER on the imaging NIC; the GE NIC's networkd lease is read instead), STP and ARP hosts for information. A kernel BPF filter keeps bulk SMB traffic out of the capture loop; bridge ports are listened on too, since LLDP and EAPOL stop at the port. - dnsmasq drop-in runs --guard as ExecStartPre: exit 1 on a ge or foreign-dhcp imaging verdict, fail open (loudly) on bugs or no link - pxe-nic-check.timer runs --watch every 5 min: stops dnsmasq on a bad imaging verdict, never starts it - /run/pxe-nic-status.json (0644, atomic) for the webapp - playbook installs it all and writes /etc/pxe-server/nic.conf from pxe_imaging_mac, pxe_ge_mac, pxe_corp_domain_suffixes, pxe_nic_allowed_switches - unit tests plus a no-root network-namespace lab (veth pairs, a raw socket DHCP/LLDP peer and real dnsmasq); docs/pxe-nic-guard.md has the operator notes and the two floor tests
This commit is contained in:
cproudlock
158
docs/pxe-nic-guard.md
Normal file
158
docs/pxe-nic-guard.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# PXE NIC guard
|
||||
|
||||
The PXE server's DHCP/PXE (dnsmasq) must never serve on the GE corporate
|
||||
network. Each server has two NICs:
|
||||
|
||||
| Role | Network | Address |
|
||||
|---------|-----------------------------|----------------------------------|
|
||||
| imaging | isolated imaging switch | static 172.16.9.1/24, dnsmasq |
|
||||
| ge | GE corporate network | DHCP client (systemd-networkd) |
|
||||
|
||||
`/etc/dnsmasq.d/pxe-nic-only.conf` pins dnsmasq to the imaging NIC by name.
|
||||
That stops dnsmasq answering on the GE NIC, but not on the imaging NIC when
|
||||
that NIC itself ends up on the wrong network: a cable swap, an imaging
|
||||
switch uplinked into a GE port, or a second PXE server on the same switch.
|
||||
The NIC guard catches those.
|
||||
|
||||
## What is installed
|
||||
|
||||
| File | Purpose |
|
||||
|-----------------------------------------------------------|---------------------------------|
|
||||
| `/usr/local/sbin/pxe-nic-check` | the classifier |
|
||||
| `/etc/pxe-server/nic.conf` | which NIC is which |
|
||||
| `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start|
|
||||
| `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min |
|
||||
| `/run/pxe-nic-status.json` | last result, read by the webapp |
|
||||
|
||||
Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml`
|
||||
(vars `pxe_imaging_mac`, `pxe_ge_mac`, `pxe_corp_domain_suffixes`,
|
||||
`pxe_nic_allowed_switches`).
|
||||
|
||||
## How it decides
|
||||
|
||||
It listens on each NIC (raw socket, sends nothing) for:
|
||||
|
||||
- LLDP and CDP: managed switches announce themselves. GE switches do; the
|
||||
unmanaged imaging switch does not.
|
||||
- 802.1X EAPOL: GE ports ask for port authentication.
|
||||
- DHCP OFFER/ACK from any other server.
|
||||
- STP and the count of distinct ARP hosts: reported, never decisive.
|
||||
|
||||
On the imaging NIC it also sends one DHCPDISCOVER and collects offers for
|
||||
6 s. On the GE NIC it reads the networkd lease instead (no traffic).
|
||||
|
||||
| Verdict | Meaning |
|
||||
|----------------|------------------------------------------------------------|
|
||||
| `isolated` | no foreign DHCP, no LLDP/CDP/EAPOL |
|
||||
| `ge` | LLDP, CDP or EAPOL seen, or a DHCP server whose domain ends in a corp suffix (`geaerospace.net`, `ge.com`; whole labels, so `notgeaerospace.net` does not count) |
|
||||
| `foreign-dhcp` | another DHCP server, no corp domain |
|
||||
| `no-link` | no carrier |
|
||||
| `error` | the check itself failed |
|
||||
|
||||
Healthy: imaging NIC `isolated` (or `no-link`), GE NIC `ge`.
|
||||
|
||||
Our own dnsmasq never counts. An offer whose server id is our own address
|
||||
but comes from a MAC that is not ours is a second PXE server and DOES count.
|
||||
|
||||
If the imaging switch is itself a managed switch that sends LLDP/CDP, put
|
||||
its LLDP system name, CDP device id or chassis MAC in `allowed_switches`
|
||||
(`pxe_nic_allowed_switches`), or the guard will block dnsmasq. 802.1X is
|
||||
never allowed.
|
||||
|
||||
## Modes
|
||||
|
||||
- `pxe-nic-check` - check every NIC (60 s), print a summary. `--json`,
|
||||
`--write`, `--probe` (imaging DHCP probe), `--iface NAME`, `--window S`.
|
||||
- `pxe-nic-check --guard` - dnsmasq `ExecStartPre`. Imaging NIC only,
|
||||
about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no
|
||||
link exits 0 (fail open) with an error line in the journal.
|
||||
- `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs
|
||||
`systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq.
|
||||
|
||||
Every start or restart of dnsmasq runs the guard, so `systemctl restart
|
||||
dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
|
||||
|
||||
## Reading it
|
||||
|
||||
sudo pxe-nic-check # full summary, 60 s
|
||||
cat /run/pxe-nic-status.json # last guard/watch result
|
||||
journalctl -u dnsmasq -b | grep pxe-nic-check
|
||||
journalctl -u pxe-nic-check -b
|
||||
|
||||
## dnsmasq will not start
|
||||
|
||||
1. `journalctl -u dnsmasq -b | grep pxe-nic-check` shows the evidence, e.g.
|
||||
`REFUSING to start dnsmasq: imaging NIC br-pxe verdict ge: LLDP from
|
||||
WJ-IDF2 port Gi1/0/14`.
|
||||
2. Check the imaging cabling: the imaging NIC must go to the imaging
|
||||
switch only, and that switch must not be uplinked anywhere.
|
||||
3. `sudo pxe-nic-check` until the imaging NIC reads `isolated`.
|
||||
4. `sudo systemctl start dnsmasq` (runs the guard again).
|
||||
|
||||
## Floor tests (run with the floor idle)
|
||||
|
||||
Both tests stop imaging for a few minutes. Have console or GE-side SSH
|
||||
access to the server; do not rely on the imaging network for the session.
|
||||
|
||||
### (a) Rogue DHCP on the imaging switch: expect the alarm
|
||||
|
||||
1. Confirm the baseline: `sudo pxe-nic-check --probe` shows the imaging
|
||||
NIC `isolated` and `dnsmasq_bound` true.
|
||||
2. Plug a device that runs a DHCP server into a free port of the IMAGING
|
||||
switch (a spare home router with DHCP on, LAN port only, nothing in its
|
||||
WAN port; or a laptop running dnsmasq).
|
||||
3. Run the watch now instead of waiting up to 5 min:
|
||||
`sudo systemctl start pxe-nic-check.service`
|
||||
4. Expect:
|
||||
- `journalctl -u pxe-nic-check -n 20` has a CRIT line `imaging NIC is
|
||||
NOT isolated, stopping dnsmasq: ... verdict foreign-dhcp: DHCP from
|
||||
192.168.x.1, ...` (or `ge` if the router sends LLDP).
|
||||
- `systemctl is-active dnsmasq` is `inactive`.
|
||||
- `/run/pxe-nic-status.json` has `"dhcp_guard": {"state": "blocked", ...}`.
|
||||
5. Prove the guard too: `sudo systemctl start dnsmasq` must FAIL, and
|
||||
`journalctl -u dnsmasq -n 20` shows `REFUSING to start dnsmasq`.
|
||||
6. Unplug the rogue device. `sudo pxe-nic-check --probe` shows `isolated`.
|
||||
`sudo systemctl start dnsmasq` starts. PXE-boot one client to confirm.
|
||||
|
||||
### (b) Cable swap into a GE port: expect the guard to refuse dnsmasq
|
||||
|
||||
dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE.
|
||||
|
||||
1. `sudo systemctl stop dnsmasq`; `systemctl is-active dnsmasq` must say
|
||||
`inactive`. Do not reboot during the test (the `@reboot` cron restarts
|
||||
dnsmasq; the guard would still block it, but do not test that way).
|
||||
2. Move the IMAGING NIC's cable from the imaging switch into a live GE
|
||||
network port. Wait for link: `ip -br link` shows the imaging NIC `UP`.
|
||||
3. Run the guard by hand first:
|
||||
`sudo pxe-nic-check --guard; echo "exit $?"`
|
||||
Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP,
|
||||
802.1X, or `DHCP from 10.x.x.x, domain ....geaerospace.net`.
|
||||
This sends one DHCPDISCOVER on the GE port, the same as any PC being
|
||||
plugged in. To send nothing, use `--no-probe --window 60` instead and
|
||||
rely on LLDP/CDP/802.1X (switches announce every 30-60 s).
|
||||
**If it exits 0, STOP: do not start dnsmasq. Move the cable back and
|
||||
send the output of `sudo pxe-nic-check --probe --json`.**
|
||||
4. Only after step 3 exited 1: `sudo systemctl start dnsmasq`. Expect
|
||||
`Job for dnsmasq.service failed`; `systemctl is-active dnsmasq` says
|
||||
`failed`; `journalctl -u dnsmasq -n 20` shows `REFUSING`.
|
||||
5. Move the cable back to the imaging switch. `sudo pxe-nic-check --probe`
|
||||
shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq`
|
||||
starts. PXE-boot one client to confirm.
|
||||
|
||||
## Status file
|
||||
|
||||
`/run/pxe-nic-status.json`, mode 0644, replaced atomically:
|
||||
|
||||
{"checked_at": "2026-10-14T16:47:09-04:00", "duration_s": 60,
|
||||
"dhcp_guard": {"state": "allowed"|"blocked"|"unknown", "reason": "..."},
|
||||
"interfaces": [{"iface": "br-pxe", "mac": "aa:bb:..", "role": "imaging"|"ge"|"other",
|
||||
"link": "up"|"down", "speed_mbps": 5000, "addrs": ["172.16.9.1/24"],
|
||||
"verdict": "isolated"|"ge"|"foreign-dhcp"|"no-link"|"error", "evidence": ["..."],
|
||||
"dhcp_servers": [{"server": "10.1.2.3", "domain": "x.geaerospace.net", "own": false}],
|
||||
"switch": {"lldp": [{"system": "...", "port": "...", "vlan": null}],
|
||||
"cdp": [{"device": "...", "port": "..."}], "eapol": false, "stp": false},
|
||||
"arp_hosts": 3, "dnsmasq_bound": true}]}
|
||||
|
||||
`dnsmasq_bound` is null when it cannot be told (no `ss`). A bridge
|
||||
(br-pxe) is reported as one interface; its ports are listened on too,
|
||||
because LLDP and 802.1X frames stop at the port and never reach the bridge.
|
||||
Reference in New Issue
Block a user