ge-aerospace/pxe-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
120553c9759e16660f8e595a2341f21c065fcedb
pxe-server/docs/pxe-nic-guard.md
cproudlock b11e678297 NICs: pick the imaging port at install, GE can go into any other port 
Sites no longer need either MAC in site-vars.

pxe_nic_mac empty: before anything changes, the playbook runs
pxe-pick-nic --auto. It takes the one wired port with a link, accepts it
only if pxe-nic-check calls it isolated (no LLDP/CDP/802.1X, no other
DHCP server, at most 10 hosts heard in 65 s), and pins it. The choice is
saved in /etc/pxe-server/nic-pick.conf and never re-guessed at boot.
When it cannot tell (no link, two links, not isolated) nothing changes,
and the console banner says why and to run sudo pxe-pick-nic: pull the
imaging cable and plug it back in, and the port that blinked is the
imaging one. Both helpers are installed before the pick, so that works
after a refused first boot. The site's corp domains and allowed
switches are passed to the check.

corp_nic_mac empty: every other wired port (en*, eth*) is a catch-all
DHCP client, so GE can be plugged in at install or later.
- dnsmasq stays pinned with interface=pxe0. A new check refuses any
  dhcp-range outside the imaging subnet, and proxy ranges.
- The catch-all ports ignore offers from 172.16.9.1.
- UFW allows SSH and 9009 by source on any interface, and keeps those
  rules when the old any-interface rules are removed.
- pxe_nic_name must not look like en*/eth* in this mode.

pxe-netplan-apply.sh takes 3 or 5 args. Without a pinned GE NIC it:
- renames stray names (the old pxe0 after a re-pick, a leftover corp0)
  to enx<mac>, so the catch-all takes them, and restores them on revert;
- checks that every NIC that had a DHCP address before has one after,
  tracked by MAC.

With both MACs set (the WJ server), netplan, the dnsmasq pin and UFW
render byte-for-byte as before.

Tested in the pxetest VMs:
- Fresh install with no MACs and the GE link down: the imaging port was
  picked and pinned as pxe0 with 172.16.9.1, and dnsmasq bound DHCP to
  pxe0 only. GE plugged in afterwards got a lease, SSH worked, and the
  guard reported pxe0 isolated.
- --auto refuses with two links.
- The interactive pick works by pulling and replugging the link.
- A 3-arg re-apply is UNCHANGED.

Unit tests (fake sysfs) cover the re-pick and pinned-to-catch-all renames.
2026-10-16 15:56:47 -04:00

204 lines
11 KiB
Markdown
Raw Blame History

# PXE NIC guard
The PXE server's DHCP/PXE (dnsmasq) must never serve on the GE corporate
network. Each server has two NICs:
| Role | Network | Address |
|---------|-----------------------------|----------------------------------|
| imaging | isolated imaging switch | static 172.16.9.1/24, dnsmasq |
| ge | GE corporate network | DHCP client (systemd-networkd) |
`/etc/dnsmasq.d/pxe-nic-only.conf` pins dnsmasq to the imaging NIC by name.
That stops dnsmasq answering on the GE NIC, but not on the imaging NIC when
that NIC itself ends up on the wrong network: a cable swap, an imaging
switch uplinked into a GE port, or a second PXE server on the same switch.
The NIC guard catches those.
## What is installed
| File | Purpose |
|-----------------------------------------------------------|---------------------------------|
| `/usr/local/sbin/pxe-nic-check` | the classifier |
| `/etc/pxe-server/nic.conf` | which NIC is which |
| `/etc/systemd/system/dnsmasq.service.d/nic-guard.conf` | guard before every dnsmasq start|
| `/etc/systemd/system/pxe-nic-check.{service,timer}` | watch every 5 min |
| `/usr/local/sbin/pxe-nic-carrier` | re-check on imaging cable moves |
| `/etc/networkd-dispatcher/<state>.d/50-pxe-nic-carrier` | calls it on link changes |
| `/etc/systemd/system/pxe-nic-carrier.service` | the re-check after carrier gain |
| `/run/pxe-nic-status.json` | last result, read by the webapp |
Sources: `playbook/pxe-server-helpers/`, installed by `pxe_server_setup.yml`
(vars `pxe_nic_mac`, `corp_nic_mac`, `pxe_corp_domain_suffixes`,
`pxe_nic_allowed_switches`; `pxe_imaging_mac` / `pxe_ge_mac` are derived
from the first two). The same MACs pin the NICs in netplan
(`playbook/templates/netplan-pxe.yaml.j2`, names `pxe0` / `corp0`) and in
`/etc/dnsmasq.d/pxe-nic-only.conf`. With `pxe_nic_mac` empty the playbook
picks the imaging NIC at install time with `pxe-pick-nic --auto`, which
runs this check on the one port with a link (`--iface X --probe`, 65 s)
and accepts only "isolated". With `corp_nic_mac` empty `ge_mac` is blank
and the GE NIC is whichever other port holds a DHCP lease.
## How it decides
It listens on each NIC (raw socket, sends nothing; it joins the STP,
802.1X, LLDP, CDP and PVST+ multicast groups so a NIC that is not
promiscuous still hears them) for:
- LLDP and CDP: managed switches announce themselves. GE switches do; the
unmanaged imaging switch does not.
- 802.1X EAPOL from an authenticator: GE ports ask for port
authentication (EAP Request, Success or Failure). EAPOL that clients
send is ignored: iPXE sends three EAPOL-Starts every time it opens a
NIC and Windows sends Starts and EAP Responses, so a bay PXE-booting on
the imaging switch is not mistaken for a GE port. Those frames show up
only as "N client EAPOL frames ignored".
- DHCP OFFER/ACK from any other server.
- STP and the count of distinct ARP hosts: reported, never decisive.
On the imaging NIC it also sends one DHCPDISCOVER and collects offers for
6 s. On the GE NIC it reads the networkd lease instead (no traffic).
| Verdict | Meaning |
|----------------|------------------------------------------------------------|
| `isolated` | no foreign DHCP, no LLDP/CDP/EAPOL |
| `ge` | LLDP, CDP or EAPOL seen, or a DHCP server whose domain ends in a corp suffix (`geaerospace.net`, `ge.com`; whole labels, so `notgeaerospace.net` does not count) |
| `foreign-dhcp` | another DHCP server, no corp domain |
| `no-link` | no carrier |
| `error` | the check itself failed |
Healthy: imaging NIC `isolated` (or `no-link`), GE NIC `ge`.
Our own dnsmasq never counts. An offer whose server id is our own address
but comes from a MAC that is not ours is a second PXE server and DOES count.
If the imaging switch is itself a managed switch that sends LLDP/CDP, put
its LLDP system name, CDP device id or chassis MAC in `allowed_switches`
(`pxe_nic_allowed_switches`), or the guard will block dnsmasq. 802.1X is
never allowed.
## Modes
- `pxe-nic-check` - check every NIC (60 s), print a summary. `--json`,
`--write`, `--probe` (imaging DHCP probe), `--iface NAME`, `--window S`.
- `pxe-nic-check --guard` - dnsmasq `ExecStartPre`. Imaging NIC only,
about 6 s. Exit 1 = do not serve: dnsmasq does not start. A bug or no
link exits 0 (fail open) with an error line in the journal.
- `pxe-nic-check --watch` - the timer. On a bad imaging verdict it runs
`systemctl stop dnsmasq` and logs CRIT. It never starts dnsmasq. If
dnsmasq was (re)started during the 60 s window (an operator, or the
carrier hook, restarted it and its guard passed), the watch re-checks
for about 6 s and stops dnsmasq only if that still says blocked; if
dnsmasq is already stopped it only logs.
- `pxe-nic-carrier` - the networkd-dispatcher hook. Only the imaging NIC
(or its bridge ports) counts; GE NIC events are ignored. Carrier lost:
dnsmasq is stopped at once if it was running. Carrier back: after 2 s,
`pxe-nic-carrier.service` restarts dnsmasq if it was running or the hook
stopped it, so the guard checks the new network before dnsmasq serves
again; dnsmasq that an operator or the watch stopped stays stopped.
A bridge with several ports stops dnsmasq when any one of them loses
carrier. Log: `journalctl -t pxe-nic-carrier`.
Every start or restart of dnsmasq runs the guard, so `systemctl restart
dnsmasq` and the `@reboot` cron restart take about 6 s longer than before.
## Reading it
sudo pxe-nic-check # full summary, 60 s
cat /run/pxe-nic-status.json # last guard/watch result
journalctl -u dnsmasq -b | grep pxe-nic-check
journalctl -u pxe-nic-check -b
journalctl -t pxe-nic-carrier -b # cable unplug / replug
## dnsmasq will not start
1. `journalctl -u dnsmasq -b | grep pxe-nic-check` shows the evidence, e.g.
`REFUSING to start dnsmasq: imaging NIC br-pxe verdict ge: LLDP from
WJ-IDF2 port Gi1/0/14`.
2. Check the imaging cabling: the imaging NIC must go to the imaging
switch only, and that switch must not be uplinked anywhere.
3. `sudo pxe-nic-check` until the imaging NIC reads `isolated`.
4. `sudo systemctl start dnsmasq` (runs the guard again).
## Floor tests (run with the floor idle)
All three tests stop imaging for a few minutes. Have console or GE-side SSH
access to the server; do not rely on the imaging network for the session.
### (a) Rogue DHCP on the imaging switch: expect the alarm
1. Confirm the baseline: `sudo pxe-nic-check --probe` shows the imaging
NIC `isolated` and `dnsmasq_bound` true.
2. Plug a device that runs a DHCP server into a free port of the IMAGING
switch (a spare home router with DHCP on, LAN port only, nothing in its
WAN port; or a laptop running dnsmasq).
3. Run the watch now instead of waiting up to 5 min:
`sudo systemctl start pxe-nic-check.service`
4. Expect:
- `journalctl -u pxe-nic-check -n 20` has a CRIT line `imaging NIC is
NOT isolated, stopping dnsmasq: ... verdict foreign-dhcp: DHCP from
192.168.x.1, ...` (or `ge` if the router sends LLDP).
- `systemctl is-active dnsmasq` is `inactive`.
- `/run/pxe-nic-status.json` has `"dhcp_guard": {"state": "blocked", ...}`.
5. Prove the guard too: `sudo systemctl start dnsmasq` must FAIL, and
`journalctl -u dnsmasq -n 20` shows `REFUSING to start dnsmasq`.
6. Unplug the rogue device. `sudo pxe-nic-check --probe` shows `isolated`.
`sudo systemctl start dnsmasq` starts. PXE-boot one client to confirm.
### (b) Cable swap into a GE port: expect the guard to refuse dnsmasq
dnsmasq must be stopped BEFORE the cable moves, so it never serves on GE.
1. `sudo systemctl stop dnsmasq`; `systemctl is-active dnsmasq` must say
`inactive`. Do not reboot during the test (the `@reboot` cron restarts
dnsmasq; the guard would still block it, but do not test that way).
2. Move the IMAGING NIC's cable from the imaging switch into a live GE
network port. Wait for link: `ip -br link` shows the imaging NIC `UP`.
The carrier hook sees the link come up and runs the guard once (one
DHCPDISCOVER) to refresh the status file; with dnsmasq stopped it does
not start it. `journalctl -t pxe-nic-carrier -n 5` shows `dnsmasq was
not running, left stopped`.
3. Run the guard by hand first:
`sudo pxe-nic-check --guard; echo "exit $?"`
Expect `exit 1` and a `REFUSING to start dnsmasq` line naming LLDP/CDP,
802.1X, or `DHCP from 10.x.x.x, domain ....geaerospace.net`.
This sends one DHCPDISCOVER on the GE port, the same as any PC being
plugged in. To send nothing, use `--no-probe --window 60` instead and
rely on LLDP/CDP/802.1X (switches announce every 30-60 s).
**If it exits 0, STOP: do not start dnsmasq. Move the cable back and
send the output of `sudo pxe-nic-check --probe --json`.**
4. Only after step 3 exited 1: `sudo systemctl start dnsmasq`. Expect
`Job for dnsmasq.service failed`; `systemctl is-active dnsmasq` says
`failed`; `journalctl -u dnsmasq -n 20` shows `REFUSING`.
5. Move the cable back to the imaging switch. `sudo pxe-nic-check --probe`
shows the imaging NIC `isolated`. `sudo systemctl start dnsmasq`
starts. PXE-boot one client to confirm.
### (c) Imaging cable unplugged and replugged: expect stop, re-check, start
1. dnsmasq running, imaging NIC `isolated`.
2. Unplug the imaging NIC's cable. Within a second
`journalctl -t pxe-nic-carrier -n 5` shows `carrier lost on imaging NIC
... stopped dnsmasq` and `systemctl is-active dnsmasq` is `inactive`.
3. Plug it back into the imaging switch. About 10 s later the journal
shows `guard passed, dnsmasq (re)started` and dnsmasq is `active`.
Had it gone into a GE port instead, the guard refuses: the journal
shows `dnsmasq NOT started` and dnsmasq stays down.
## Status file
`/run/pxe-nic-status.json`, mode 0644, replaced atomically:
{"checked_at": "2026-10-14T16:47:09-04:00", "duration_s": 60,
"dhcp_guard": {"state": "allowed"|"blocked"|"unknown", "reason": "..."},
"interfaces": [{"iface": "br-pxe", "mac": "aa:bb:..", "role": "imaging"|"ge"|"other",
"link": "up"|"down", "speed_mbps": 5000, "addrs": ["172.16.9.1/24"],
"verdict": "isolated"|"ge"|"foreign-dhcp"|"no-link"|"error", "evidence": ["..."],
"dhcp_servers": [{"server": "10.1.2.3", "domain": "x.geaerospace.net", "own": false}],
"switch": {"lldp": [{"system": "...", "port": "...", "vlan": null}],
"cdp": [{"device": "...", "port": "..."}], "eapol": false, "stp": false},
"arp_hosts": 3, "dnsmasq_bound": true}]}
`dnsmasq_bound` is null when it cannot be told (no `ss`). A bridge
(br-pxe) is reported as one interface; its ports are listened on too,
because LLDP and 802.1X frames stop at the port and never reach the bridge.
Reference in New Issue View Git Blame Copy Permalink