ge-aerospace/pxe-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build/playbook re-run safety: corp access, netplan rollback, first-boot retry

Browse Source 
H1  corp_admin_cidrs (UFW 22/9009 on the corp NIC from these only; refuses
    when it would lock out the current SSH client) and corp_ssh_password_auth
    (sshd drop-in, passwords only from the imaging subnet). Defaults keep the
    current behaviour and print a loud warning.
H2  pxe-netplan-apply.sh: pre-flight refusal (pxe-MAC NIC holds the default
    route, a DHCP lease or the route to an SSH client), run from the USB
    before anything changes; post-apply checks (names, imaging IPv4, corp
    IPv4 back within 30 s, SSH route) and a 120 s systemd-run auto-revert.
H3  restore() puts the previous NIC names back (two-phase rename via
    temporary names, so swaps work) before generate + apply.
H4  a failed rename is fatal (restore + exit 1); the play drops the old
    any-interface UFW rules only after verifying both NICs and their
    per-interface SSH rules.
H5  first-boot checks the playbook exit code, retries at next boot (max 5),
    writes /var/lib/pxe-firstboot.failed and /etc/issue + /etc/motd banners.
M1  build-usb.sh verifies iPXE files against the playbook pins, every wheel
    against its own package's hashes, drops the unpinned pip fallback,
    requires --site-vars or --no-site-vars.
M2  webapp restarts when the code, wheels or unit changed.
M3  ignore-carrier on the imaging NIC; ip_nonlocal_bind kept for the
    apache/networkd boot race.
M4  menu suffix must match blancco_kver; missing kernel with no earlier
    build is fatal; a live menu with the staged Blancco test entry is not
    overwritten.
M5  build-proxmox-iso.sh is unsupported (hard exit); use build-usb --image.
M6  deploy-unattend.sh: password via SSHPASS env and sudo stdin only,
    %q-quoted remote args, sha256 check of the live file; lint-unattend.py
    reads PXE_SERVER/PXE_USER/PXE_PASS and uses sshpass -e.
M7  the playbook lints the unattend files before the force: yes copies
    (lint-unattend.py ships on CIDATA as scripts/); pre-commit ACMR.
M8  Blancco initramfs build: no ignore_errors, pipefail, trap cleanup.
Also: Send-ImagingStage.ps1 deploy, carrier hook install tasks
(pxe-nic-carrier, networkd-dispatcher), L1 imaging-active hold
(allow_disruptive) and no iPXE downgrade below an Updates-page install,
L2 iso-volid, L3 kver pin read from the playbook, L4 cp -rL, L5 refuse
>= 4 GiB WinPE files, L6 PIP_DIR wipe guard, L7 per-package wheel hashes,
no online fallback, L8 check-updates.py (apt_pkg/cached compare, all
components, exit 2 on no debs / bad answers).
This commit is contained in:
cproudlock
2026-10-15 14:32:23 -04:00
parent 53d39bfc50
commit 0b1625ed87
15 changed files with 1401 additions and 614 deletions
Download Patch File Download Diff File Expand all files Collapse all files

629
playbook/pxe_server_setup.yml
View File

@@ -30,6 +30,7 @@
- ansible
- wimtools
- conntrack
- networkd-dispatcher # carrier hook (pxe-nic-carrier)
- sbsigntool # sbverify: iPXE update signature checks
- p7zip-full # 7z: Dell driver packs, catalog cabs, Blancco ISO
- zip # driver zips PESetup unpacks
@@ -62,6 +63,9 @@
ipxe_shim_sha256: "5eecca2780bd49c900565e124516a1bd666ec5e012825f34991b6ba1ef2fa6cf" # ipxe/shim ipxe-16.1 ipxe-shimx64.efi
ipxe_snponly_sha256: "b1e67c3e4a1e8708ddfd0079ad4505e3a02245acb55ee9a95437ab3c507be82a" # x86_64-sb/snponly.efi
ipxe_bios_sha256: "0fb21c695c82fd70d4f89732ddcd285063776e2bbd71de9a2c39e9cfe4b180b8" # x86_64/ipxe.pxe
# Release the pins above come from. A re-run does not replace files the
# webapp's Updates page installed from a NEWER release (installed.json).
ipxe_release_tag: "v2.0.0"
# Zscaler Root CA: GE's Zscaler re-signs GitHub TLS, so the iPXE update
# check fails without it. Repo copy first, else the copy on the image share.
zscaler_ca_src: "{{ playbook_dir }}/certs/zscaler-root-ca.crt"
@@ -126,6 +130,37 @@
ufw_corp_rules:
- { port: "22", proto: tcp }
- { port: "9009", proto: tcp }
# Who may reach SSH and the webapp (9009, no login, runs as root) on the
# GE corporate NIC. Empty (the default) = all of GE, as before, with a
# loud warning every run. Set to the admin subnets/hosts, e.g.
# ["10.134.48.0/24", "3.12.34.56/32"]: UFW then allows 22 and 9009 on
# the corp NIC from these only. The play refuses to restrict if your own
# SSH session is not inside one of them. See docs/SITE-CUSTOMIZATION.md.
corp_admin_cidrs: []
# false: sshd refuses password logins except from the imaging subnet
# and localhost (keys only from the GE side). Default true keeps the
# shared 'pxe' password working from GE, so nobody is locked out; the
# play refuses false while the pxe user has no authorized_keys.
corp_ssh_password_auth: true
# Re-run safety: when imaging is in progress (SMB sessions open on 445
# or DHCP leases granted in the last pxe_busy_window_min minutes) a
# re-run does not bounce the NICs or restart dnsmasq/Apache/Samba.
# A needed network change then stops the play with a message;
# -e allow_disruptive=true overrides.
allow_disruptive: false
pxe_busy_window_min: 30
# networkd-dispatcher states the carrier hook (pxe-nic-carrier) runs in.
carrier_hook_states:
- carrier.d
- degraded.d
- degraded-carrier.d
- enslaved.d
- routable.d
- no-carrier.d
- off.d
# The admin's SSH client (empty when not run over SSH, or when sudo
# dropped SSH_CONNECTION; the checks also read sshd's live peers).
pxe_ssh_client: "{{ (lookup('env', 'SSH_CONNECTION') | default('', true)).split(' ')[0] }}"
# --- Blancco UEFI / Secure Boot path ---
# Blancco's UEFI entries boot a Canonical-signed Ubuntu HWE kernel
# (through Ubuntu's shim when Secure Boot is on), then switch_root into
@@ -136,6 +171,13 @@
# major+minor ("7.0.0-34-generic" -> "70"), as GetPxeScript.aspx expects:
# vmlinuz-ubuntu-70, kexec-initrd-70.img; modules are
# kmod-<blancco_kver>.tar.gz (blancco-init.sh fetches kmod-$(uname -r)).
# The pin lives in three places; this var is the source:
# 1. here (blancco_kver)
# 2. scripts/download-packages.sh reads its BLANCCO_KVER default from
# this line (packages linux-image/modules-<kver>)
# 3. playbook/ipxe/GetPxeScript.aspx names the files by suffix
# (vmlinuz-ubuntu-70, kexec-initrd-70.img); the play fails if that
# suffix is not blancco_ksuffix.
blancco_kver: "7.0.0-34-generic"
blancco_ksuffix: "{{ blancco_kver.split('.')[0] }}{{ blancco_kver.split('.')[1] }}"
image_types:
@@ -208,6 +250,145 @@
Fix playbook/site-vars.yml and re-run.
quiet: yes
# --- Re-run safety (nothing below changes the box until these pass) ---
- name: "Re-run safety: is imaging in progress? (SMB sessions, recent DHCP leases)"
args:
executable: /bin/bash
shell: |
smb=$(ss -Htn state established '( sport = :445 )' 2>/dev/null | wc -l)
recent=0
L=/var/lib/misc/dnsmasq.leases
lease="{{ pxe_dhcp_lease }}"
case "$lease" in
*[0-9]d) secs=$(( ${lease%d} * 86400 )) ;;
*[0-9]h) secs=$(( ${lease%h} * 3600 )) ;;
*[0-9]m) secs=$(( ${lease%m} * 60 )) ;;
*[0-9]s) secs=${lease%s} ;;
*[0-9]) secs=$lease ;;
*) secs=0 ;;
esac
if [ -f "$L" ] && [ "$secs" -gt 0 ]; then
# Field 1 is the expiry; granted = expiry - lease time.
recent=$(awk -v now="$(date +%s)" -v lease="$secs" -v win="{{ (pxe_busy_window_min | int) * 60 }}" \
'$1 > 0 && ($1 - lease) > (now - win) {n++} END {print n + 0}' "$L")
fi
echo "$smb $recent"
register: pxe_busy
changed_when: false
- name: "Re-run safety: decide"
set_fact:
pxe_busy_smb: "{{ pxe_busy.stdout.split()[0] | int }}"
pxe_busy_leases: "{{ pxe_busy.stdout.split()[1] | int }}"
pxe_hold_disruptive: "{{ ((pxe_busy.stdout.split()[0] | int) > 0 or (pxe_busy.stdout.split()[1] | int) > 0) and not (allow_disruptive | bool) }}"
- name: "Re-run safety: imaging in progress, disruptive steps are held"
debug:
msg: >-
{{ pxe_busy_smb }} SMB session(s) open and {{ pxe_busy_leases }}
DHCP lease(s) granted in the last {{ pxe_busy_window_min }} min:
imaging looks active. This run will NOT bounce the NICs or restart
dnsmasq/Apache/Samba (a needed network change stops the play).
Re-run when idle, or with -e allow_disruptive=true.
when: pxe_hold_disruptive | bool
- name: "Corp NIC: corp_admin_cidrs must be valid and include your own SSH session"
# Restricting SSH on the GE NIC to subnets that do not contain the
# admin running this play would lock them out on the next login.
when: corp_admin_cidrs | length > 0
environment:
PXE_SSH_CLIENT: "{{ pxe_ssh_client }}"
CIDRS: "{{ corp_admin_cidrs | join(' ') }}"
PXE_SUBNET: "{{ pxe_subnet }}"
args:
executable: /bin/bash
shell: |
peers=$( { echo "${PXE_SSH_CLIENT:-}"; ss -Htn state established '( sport = :22 )' 2>/dev/null | awk '{print $NF}'; } )
python3 - "$peers" <<'PY'
import ipaddress, os, re, sys
try:
nets = [ipaddress.ip_network(c, strict=False) for c in os.environ["CIDRS"].split()]
except ValueError as ex:
sys.exit(f"REFUSED: corp_admin_cidrs has an invalid entry: {ex}")
local = [ipaddress.ip_network(os.environ["PXE_SUBNET"], strict=False),
ipaddress.ip_network("127.0.0.0/8"), ipaddress.ip_network("::1/128")]
bad = []
def parse(raw):
for cand in (raw, re.sub(r":\d+$", "", raw)):
cand = cand.strip("[]")
cand = cand[7:] if cand.startswith("::ffff:") else cand
try:
return ipaddress.ip_address(cand)
except ValueError:
pass
return None
for raw in sys.argv[1].split():
ip = parse(raw)
if ip is None:
continue
if any(ip in n for n in local) or any(ip in n for n in nets):
continue
bad.append(str(ip))
if bad:
sys.exit("REFUSED: SSH client(s) %s are not inside corp_admin_cidrs %s; restricting "
"would lock them out. Add their subnet, or run from inside one. Nothing changed."
% (", ".join(sorted(set(bad))), os.environ["CIDRS"]))
print("corp_admin_cidrs cover every current SSH client")
PY
changed_when: false
- name: "SSH: keys required before password logins are turned off on the GE side"
when: not (corp_ssh_password_auth | bool)
shell: |
home=$(getent passwd pxe | cut -d: -f6)
if [ -n "$home" ] && [ -s "$home/.ssh/authorized_keys" ] &&
grep -qE '^(ssh-|ecdsa-|sk-)' "$home/.ssh/authorized_keys"; then
echo "pxe has $(grep -cE '^(ssh-|ecdsa-|sk-)' "$home/.ssh/authorized_keys") key(s)"
else
echo "REFUSED: corp_ssh_password_auth is false but pxe has no ~/.ssh/authorized_keys;" >&2
echo "the GE side would be locked out. Add a key (ssh-copy-id pxe@<corp ip>) first." >&2
exit 1
fi
changed_when: false
- name: "NICs: render the netplan config for the pre-flight check (in /run, not installed)"
template:
src: templates/netplan-pxe.yaml.j2
dest: /run/pxe-server-netplan.check.yaml
mode: '0600'
changed_when: false
- name: "NICs: pre-flight (refuses swapped MACs / mis-cabled NICs before anything changes)"
# Refuses when the NIC pxe_nic_mac names holds the default route, a
# DHCP lease or the route to an SSH client: it would become static
# with no gateway and cut the box off. Runs from the USB copy, so
# nothing on the box has been touched yet.
command: >-
bash {{ usb_mount }}/pxe-server-helpers/pxe-netplan-apply.sh --preflight
/run/pxe-server-netplan.check.yaml
{{ pxe_nic_name }} {{ pxe_nic_mac | lower }}
{{ corp_nic_name }} {{ corp_nic_mac | lower }}
environment:
PXE_SUBNET: "{{ pxe_subnet }}"
PXE_SSH_CLIENT: "{{ pxe_ssh_client }}"
register: netplan_preflight
changed_when: false
- name: "NICs: pre-flight result"
debug:
msg: "{{ netplan_preflight.stdout_lines }}"
- name: "NICs: a network change is needed but imaging is in progress"
fail:
msg: >-
The netplan config / NIC names must change, which bounces the NICs,
but imaging is active ({{ pxe_busy_smb }} SMB sessions,
{{ pxe_busy_leases }} recent leases). Nothing was changed. Re-run
when idle, or with -e allow_disruptive=true.
when:
- pxe_hold_disruptive | bool
- netplan_preflight.stdout_lines | last | default('') == 'NEEDED'
- name: "NICs: stop cloud-init from rewriting the network config"
copy:
dest: /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
@@ -234,6 +415,9 @@
/usr/local/sbin/pxe-netplan-apply.sh /etc/pxe-server-netplan.yaml
{{ pxe_nic_name }} {{ pxe_nic_mac | lower }}
{{ corp_nic_name }} {{ corp_nic_mac | lower }}
environment:
PXE_SUBNET: "{{ pxe_subnet }}"
PXE_SSH_CLIENT: "{{ pxe_ssh_client }}"
register: netplan_apply
changed_when: netplan_apply.stdout_lines | last | default('') == 'CHANGED'
@@ -252,9 +436,12 @@
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.default.forwarding = 0
# Apache listens on {{ pxe_server_ip }} only (ports 80/4433). Let it
# bind before networkd has put the address on {{ pxe_nic_name }} (no
# carrier at boot, or apache starting first), or apache2 fails to
# start and takes the webapp on 9009 down with it.
# bind before networkd has put the address on {{ pxe_nic_name }}, or
# apache2 fails to start and takes the webapp on 9009 down with it.
# The netplan config sets ignore-carrier on {{ pxe_nic_name }}, so a
# missing link no longer keeps the address off; this stays for the
# boot-order race (apache2 is ordered only after network.target,
# not after networkd has configured the address).
net.ipv4.ip_nonlocal_bind = 1
register: pxe_net_sysctl
@@ -416,6 +603,62 @@
- pxe-nic-check.service
- pxe-nic-check.timer
# Carrier hook: networkd-dispatcher runs pxe-nic-carrier on every link
# state change; it starts pxe-nic-carrier.service, which stops dnsmasq
# when the imaging link drops and starts it again (through the
# ExecStartPre guard) when it comes back. pxe-nic-check, its service
# and pxe-dhcp-hook.sh are (re)copied by the tasks above/below.
- name: "Carrier hook: networkd-dispatcher installed (offline repo)"
apt:
name: networkd-dispatcher
state: present
- name: "Carrier hook: deploy pxe-nic-carrier"
copy:
src: "{{ usb_mount }}/pxe-server-helpers/pxe-nic-carrier"
dest: /usr/local/sbin/pxe-nic-carrier
owner: root
group: root
mode: '0755'
- name: "Carrier hook: install pxe-nic-carrier.service (started by the hook, not enabled)"
copy:
src: "{{ usb_mount }}/pxe-server-helpers/pxe-nic-carrier.service"
dest: /etc/systemd/system/pxe-nic-carrier.service
mode: '0644'
register: carrier_unit
- name: "Carrier hook: systemd daemon-reload"
systemd:
daemon_reload: yes
when: carrier_unit is changed
- name: "Carrier hook: networkd-dispatcher state directories"
file:
path: "/etc/networkd-dispatcher/{{ item }}"
state: directory
owner: root
group: root
mode: '0755'
loop: "{{ carrier_hook_states }}"
notify: restart networkd-dispatcher
- name: "Carrier hook: link the hook into each state"
file:
src: /usr/local/sbin/pxe-nic-carrier
dest: "/etc/networkd-dispatcher/{{ item }}/50-pxe-nic-carrier"
state: link
owner: root
group: root
loop: "{{ carrier_hook_states }}"
notify: restart networkd-dispatcher
- name: "Carrier hook: networkd-dispatcher enabled and running"
systemd:
name: networkd-dispatcher
enabled: yes
state: started
- name: "NIC guard: enable watch timer"
systemd:
name: pxe-nic-check.timer
@@ -458,6 +701,47 @@
- blancco
- memtest
- name: "Menu: the Blancco file names in the menu must match blancco_kver"
# GetPxeScript.aspx names the UEFI Blancco files by suffix
# (vmlinuz-ubuntu-70, kexec-initrd-70.img); the play builds them from
# blancco_kver. A mismatch boots a file that is never built.
shell: |
M="{{ usb_mount }}/ipxe/GetPxeScript.aspx"
found=$(grep -oE '(vmlinuz-ubuntu|kexec-initrd)-[0-9]+' "$M" | sed 's/.*-//' | sort -u | tr '\n' ' ')
echo "menu suffixes: ${found:-none}"
for x in $found; do
if [ "$x" != "{{ blancco_ksuffix }}" ]; then
echo "REFUSED: $M references Blancco files with suffix $x but blancco_kver" >&2
echo "{{ blancco_kver }} builds suffix {{ blancco_ksuffix }}. Change them together" >&2
echo "(playbook var blancco_kver, scripts/download-packages.sh, the menu)." >&2
exit 1
fi
done
register: menu_ksuffix
changed_when: false
- name: "Menu: does the live menu carry a staged Blancco test entry?"
# The Blancco Releases page (webapp/services/blancco_update.py) adds a
# test entry between these markers; overwriting the menu would drop it
# while the page still shows it as added.
shell: |
F="{{ web_root }}/Altiris/iPXE/GetPxeScript.aspx"
[ -f "$F" ] || exit 0
if grep -qF '# BEGIN blancco-staged test entry' "$F" || grep -qE '^item[[:space:]]+blancco_staged[[:space:]]' "$F"; then
echo STAGED-TEST
fi
register: menu_test_entry
changed_when: false
- name: "Menu: NOT replacing the live menu (Blancco test entry present)"
debug:
msg: >-
WARNING: {{ web_root }}/Altiris/iPXE/GetPxeScript.aspx carries the
staged Blancco test entry (Updates > Blancco). The menu was NOT
updated from {{ usb_mount }}/ipxe/GetPxeScript.aspx. Remove the
test entry on that page (or promote the release), then re-run.
when: "'STAGED-TEST' in menu_test_entry.stdout"
- name: "Install GetPxeScript.aspx (iPXE boot menu)"
# The live menu, tracked as a file (playbook/ipxe/GetPxeScript.aspx).
# WinPE skips the Secure Boot warning when iPXE reads Secure Boot as on
@@ -470,6 +754,7 @@
dest: "{{ web_root }}/Altiris/iPXE/GetPxeScript.aspx"
mode: '0644'
backup: yes
when: "'STAGED-TEST' not in menu_test_entry.stdout"
# Boot files (80) and the iPXE menu (4433) are served on the imaging IP
# only, never on the GE NIC. 9009 (webapp) stays on all addresses: admins
@@ -700,6 +985,20 @@
group: pxe
mode: '0775'
# Imaging stage reporter: the bays run it from the shared Applications
# tree and post their stage to the webapp's /imaging dashboard.
- name: "Imaging status: create _shared/Applications/extra/pxe-status"
file:
path: "{{ samba_share }}/_shared/Applications/extra/pxe-status"
state: directory
mode: '0755'
- name: "Imaging status: deploy Send-ImagingStage.ps1"
copy:
src: "{{ usb_mount }}/imaging-status/Send-ImagingStage.ps1"
dest: "{{ samba_share }}/_shared/Applications/extra/pxe-status/Send-ImagingStage.ps1"
mode: '0644'
- name: "Deploy BIOS check script + manifest to winpeapps/_shared/BIOS/"
# Path matches what startnet.cmd reads at WinPE boot:
# net use B: \\172.16.9.1\winpeapps\_shared
@@ -916,6 +1215,27 @@
force: no
loop: "{{ image_types }}"
- name: "Unattend: lint the answer files before they replace the live ones"
# A bad answer file stops every bay imaged with it at "Windows could
# not parse or process unattend answer file". The copies below use
# force: yes, so this gate is the only check. scripts/lint-unattend.py
# rides on the USB as scripts/ next to playbook/ (build-usb.sh), and
# is in the repo at the same relative path.
args:
executable: /bin/bash
shell: |
for L in "{{ usb_mount }}/../scripts/lint-unattend.py" "{{ playbook_dir }}/../scripts/lint-unattend.py"; do
if [ -f "$L" ]; then
exec python3 "$L" --local \
"{{ usb_mount }}/FlatUnattendW10-shopfloor.xml" \
"{{ usb_mount }}/FlatUnattendW10.xml" \
"{{ usb_mount }}/FlatUnattendW10-engineer.xml"
fi
done
echo "ERROR: scripts/lint-unattend.py not found next to {{ usb_mount }} or {{ playbook_dir }}" >&2
exit 1
changed_when: false
- name: "Deploy shopfloor unattend.xml template"
copy:
src: "{{ usb_mount }}/FlatUnattendW10-shopfloor.xml"
@@ -1006,12 +1326,49 @@
check "$W/ipxeboot/x86_64-sb/snponly.efi" "{{ ipxe_snponly_sha256 }}"
check "$W/ipxeboot/x86_64/ipxe.pxe" "{{ ipxe_bios_sha256 }}"
put() { install -m 0644 -o root -g root "$1" "$T/.$2.new" && mv -f "$T/.$2.new" "$T/$2"; }
put "$U/ipxe-shimx64.efi" ipxe-shim.efi
put "$W/ipxeboot/x86_64-sb/snponly.efi" ipxe.efi
put "$W/ipxeboot/x86_64/ipxe.pxe" ipxe.pxe
# Never downgrade what the webapp's Updates page installed from a
# newer release than ipxe_release_tag: a file whose live sha256 still
# matches its installed.json record with a newer tag is kept. The
# shim and ipxe.efi go together (the shim must trust ipxe.efi's CA).
AHEAD=$(python3 - /var/lib/pxe-webapp/updates/ipxe/installed.json "$T" "{{ ipxe_release_tag }}" <<'PY'
import hashlib, json, os, re, sys
log, tftp, pin = sys.argv[1:4]
def ver(t):
m = re.search(r"(\d+)\.(\d+)(?:\.(\d+))?", t or "")
return tuple(int(x or 0) for x in m.groups()) if m else None
try:
data = json.load(open(log))
except (OSError, ValueError):
data = {}
data = data if isinstance(data, dict) else {}
for name in ("ipxe-shim.efi", "ipxe.efi", "ipxe.pxe"):
rec, path = data.get(name), os.path.join(tftp, name)
if not isinstance(rec, dict) or not os.path.isfile(path):
continue
if hashlib.sha256(open(path, "rb").read()).hexdigest() != rec.get("sha256"):
continue
a, b = ver(rec.get("tag")), ver(pin)
if a and b and a > b:
print(f"{name}={rec.get('tag')}")
PY
)
done=""
if grep -qE '^(ipxe-shim\.efi|ipxe\.efi)=' <<<"$AHEAD"; then
echo "KEPT ipxe-shim.efi + ipxe.efi: the Updates page installed a newer release ($(grep -E '^ipxe(-shim)?\.efi=' <<<"$AHEAD" | tr '\n' ' ')) than the playbook's {{ ipxe_release_tag }}"
else
put "$U/ipxe-shimx64.efi" ipxe-shim.efi
put "$W/ipxeboot/x86_64-sb/snponly.efi" ipxe.efi
done="ipxe-shim.efi, ipxe.efi (snponly)"
fi
if grep -qE '^ipxe\.pxe=' <<<"$AHEAD"; then
echo "KEPT ipxe.pxe: the Updates page installed $(grep -E '^ipxe\.pxe=' <<<"$AHEAD"), newer than {{ ipxe_release_tag }}"
else
put "$W/ipxeboot/x86_64/ipxe.pxe" ipxe.pxe
done="${done:+$done, }ipxe.pxe"
fi
# Rollback boot file: the old Broadcom-signed build (boot-files/ipxe.efi).
if [ -f "$U/ipxe.efi" ]; then put "$U/ipxe.efi" ipxe-bcm.efi; fi
echo "installed ipxe-shim.efi, ipxe.efi (snponly), ipxe.pxe$( [ -f "$U/ipxe.efi" ] && echo ', ipxe-bcm.efi')"
if [ -f "$U/ipxe.efi" ]; then put "$U/ipxe.efi" ipxe-bcm.efi; done="${done:+$done, }ipxe-bcm.efi"; fi
echo "installed: ${done:-nothing}"
args:
executable: /bin/bash
register: ipxe_chain
@@ -1049,17 +1406,42 @@
set_fact:
blancco_kernel_ok: "{{ blancco_kernel_files.results | map(attribute='stat.exists') | min }}"
- name: "Blancco: warn when the Blancco kernel is missing"
- name: "Blancco: are the UEFI Blancco files already served?"
stat:
path: "{{ web_root }}/blancco/{{ item }}"
loop:
- "vmlinuz-ubuntu-{{ blancco_ksuffix }}"
- "kexec-initrd-{{ blancco_ksuffix }}.img"
register: blancco_served
when: not blancco_kernel_ok
# The menu boots these files (checked above: same suffix). Without the
# kernel they cannot be built; with no earlier build in place, every
# UEFI Blancco boot would fail, so that is fatal, not a warning.
- name: "Blancco: the kernel is missing and the menu's files were never built"
fail:
msg: >-
/boot/vmlinuz-{{ blancco_kver }} or /lib/modules/{{ blancco_kver }}
is missing, and {{ web_root }}/blancco has no
vmlinuz-ubuntu-{{ blancco_ksuffix }} /
kexec-initrd-{{ blancco_ksuffix }}.img from an earlier run, but the
iPXE menu boots them. Install linux-image-{{ blancco_kver }} +
linux-modules-{{ blancco_kver }} (offline packages,
scripts/download-packages.sh) or set blancco_kver to an installed
version (ls /lib/modules) AND change the menu to match, then re-run.
when:
- not blancco_kernel_ok
- not (blancco_served.results | map(attribute='stat.exists') | min)
- name: "Blancco: warn when the Blancco kernel is missing (earlier build kept)"
debug:
msg: >-
WARNING: /boot/vmlinuz-{{ blancco_kver }} or
/lib/modules/{{ blancco_kver }} is missing, so the Blancco UEFI
files (vmlinuz-ubuntu-{{ blancco_ksuffix }},
kexec-initrd-{{ blancco_ksuffix }}.img,
kmod-{{ blancco_kver }}.tar.gz) are NOT built. Install
files are NOT rebuilt; the copies from an earlier run stay in
place (they may not match blancco_kver). Install
linux-image-{{ blancco_kver }} + linux-modules-{{ blancco_kver }}
(scripts/download-packages.sh, BLANCCO_KVER) or set blancco_kver
to an installed version (ls /lib/modules), then re-run.
and re-run.
when: not blancco_kernel_ok
- name: "Build Blancco PXE initramfs (kexec-initrd-{{ blancco_ksuffix }}.img)"
@@ -1079,8 +1461,9 @@
creates: "{{ web_root }}/blancco/.kexec-initrd-{{ blancco_ksuffix }}.{{ blancco_kver }}.built"
when: blancco_kernel_ok
shell: |
set -e
set -eo pipefail
WORK=$(mktemp -d)
trap 'rm -rf "$WORK" "$OUT.tmp"' EXIT
KVER="{{ blancco_kver }}"
OUT="{{ web_root }}/blancco/kexec-initrd-{{ blancco_ksuffix }}.img"
mkdir -p "$WORK"/{bin,lib/modules/$KVER/kernel,lib64,sbin,usr/share/udhcpc,etc,run,proc,sys,dev}
@@ -1172,11 +1555,9 @@
find . | cpio -o -H newc --quiet | gzip -1 > "$OUT.tmp"
mv -f "$OUT.tmp" "$OUT"
cd /
rm -rf "$WORK"
rm -f "{{ web_root }}"/blancco/.kexec-initrd-{{ blancco_ksuffix }}.*.built
touch "{{ web_root }}/blancco/.kexec-initrd-{{ blancco_ksuffix }}.{{ blancco_kver }}.built"
echo "Built $(basename "$OUT") for $KVER: $(stat -c %s "$OUT") bytes"
ignore_errors: yes
- name: "Copy Ubuntu kernel for Blancco PXE boot (vmlinuz-ubuntu-{{ blancco_ksuffix }})"
copy:
@@ -1312,15 +1693,25 @@
when: usb_images_dir.stat.exists
- name: "Restart and enable services"
# Held while imaging is active (see "Re-run safety"): started +
# enabled only, so a WinPE client mid-copy keeps its SMB session.
systemd:
name: "{{ item }}"
state: restarted
state: "{{ 'started' if pxe_hold_disruptive | bool else 'restarted' }}"
enabled: yes
loop:
- dnsmasq
- apache2
- smbd
- name: "Services NOT restarted (imaging in progress)"
debug:
msg: >-
dnsmasq, apache2 and smbd were not restarted because imaging is
active; config changes from this run apply at their next restart.
Re-run when idle (or -e allow_disruptive=true) to restart them.
when: pxe_hold_disruptive | bool
# Firewall per interface. Imaging NIC: everything imaging needs. GE
# corporate NIC: SSH and the webapp only. New rules go in first, then the
# old any-interface rules are removed, so SSH is never closed mid-run.
@@ -1335,7 +1726,7 @@
loop_control:
label: "{{ pxe_nic_name }} {{ item.port }}/{{ item.proto }}"
- name: "UFW: allow SSH and the webapp on the GE corporate NIC"
- name: "UFW: allow SSH and the webapp on the GE corporate NIC (from all of GE)"
ufw:
rule: allow
direction: in
@@ -1345,6 +1736,70 @@
loop: "{{ ufw_corp_rules }}"
loop_control:
label: "{{ corp_nic_name }} {{ item.port }}/{{ item.proto }}"
when: corp_admin_cidrs | length == 0
- name: "UFW: WARNING, SSH and the root webapp are open to all of GE"
debug:
msg:
- "!!! WARNING: corp_admin_cidrs is empty. SSH (password auth, shared"
- "!!! 'pxe' password) and the webapp on 9009 (no login, runs as root)"
- "!!! accept connections from ANY host on the GE corporate network via"
- "!!! {{ corp_nic_name }}. Set corp_admin_cidrs in playbook/site-vars.yml"
- "!!! to the admin subnets (docs/SITE-CUSTOMIZATION.md) and re-run."
when: corp_admin_cidrs | length == 0
- name: "UFW: allow SSH and the webapp on the GE corporate NIC from corp_admin_cidrs only"
ufw:
rule: allow
direction: in
interface: "{{ corp_nic_name }}"
from_ip: "{{ item.1 }}"
port: "{{ item.0.port }}"
proto: "{{ item.0.proto }}"
loop: "{{ ufw_corp_rules | product(corp_admin_cidrs) | list }}"
loop_control:
label: "{{ corp_nic_name }} from {{ item.1 }} {{ item.0.port }}/{{ item.0.proto }}"
when: corp_admin_cidrs | length > 0
- name: "UFW: drop the any-source rules on the GE corporate NIC (restricted now)"
ufw:
rule: allow
direction: in
interface: "{{ corp_nic_name }}"
port: "{{ item.port }}"
proto: "{{ item.proto }}"
delete: yes
loop: "{{ ufw_corp_rules }}"
loop_control:
label: "{{ corp_nic_name }} any {{ item.port }}/{{ item.proto }}"
when: corp_admin_cidrs | length > 0
# Only drop the old any-interface rules (allow 22 among them) once both
# named NICs exist on the right MACs and the per-interface SSH rules are
# in UFW. Otherwise those rules would name interfaces that are not
# there and removing the old ones would close SSH.
- name: "UFW: verify both named NICs and their per-interface rules"
args:
executable: /bin/bash
shell: |
set -u
bad=0
for pair in "{{ pxe_nic_name }}={{ pxe_nic_mac | lower }}" "{{ corp_nic_name }}={{ corp_nic_mac | lower }}"; do
n=${pair%%=*}; m=${pair#*=}
if [ "$(cat "/sys/class/net/$n/address" 2>/dev/null)" != "$m" ]; then
echo "ERROR: interface $n with MAC $m is not present" >&2; bad=1
fi
done
added=$(ufw show added 2>/dev/null)
for n in "{{ pxe_nic_name }}" "{{ corp_nic_name }}"; do
if ! grep -E "allow in on $n( |$)" <<<"$added" | grep -qE 'port 22 proto tcp'; then
echo "ERROR: no UFW 'allow in on $n ... port 22 proto tcp' rule" >&2; bad=1
fi
done
[ "$bad" -eq 0 ] || { echo "the old any-interface rules are kept" >&2; exit 1; }
echo "both NICs present, per-interface SSH rules in place"
register: ufw_verify
changed_when: false
- name: "UFW: remove the old any-interface rules"
ufw:
@@ -1377,6 +1832,50 @@
state: enabled
policy: deny
# --- SSH on the GE NIC: password logins (corp_ssh_password_auth) ---
# The drop-in sorts before 50-cloud-init.conf (autoinstall allow-pw), and
# sshd keeps the first value it reads. "Match all" ends the block so the
# rest of sshd_config is not scoped to it.
- name: "SSH: password logins only from the imaging LAN (corp_ssh_password_auth: false)"
copy:
dest: /etc/ssh/sshd_config.d/10-pxe-corp-auth.conf
mode: '0644'
content: |
# Managed by pxe_server_setup.yml (corp_ssh_password_auth: false).
# Keys only from the GE corporate side; the shared password still
# works from the imaging LAN and localhost.
PasswordAuthentication no
KbdInteractiveAuthentication no
Match Address {{ pxe_subnet }},127.0.0.0/8,::1
PasswordAuthentication yes
Match all
register: sshd_dropin_add
when: not (corp_ssh_password_auth | bool)
- name: "SSH: password logins allowed everywhere (corp_ssh_password_auth: true)"
file:
path: /etc/ssh/sshd_config.d/10-pxe-corp-auth.conf
state: absent
register: sshd_dropin_del
when: corp_ssh_password_auth | bool
- name: "SSH: check the sshd config, reload it"
when: (sshd_dropin_add is changed) or (sshd_dropin_del is changed)
block:
- name: "SSH: sshd -t"
command: /usr/sbin/sshd -t
changed_when: false
- name: "SSH: reload sshd"
command: systemctl try-reload-or-restart ssh.service
rescue:
- name: "SSH: bad config, remove the drop-in again"
file:
path: /etc/ssh/sshd_config.d/10-pxe-corp-auth.conf
state: absent
- name: "SSH: fail"
fail:
msg: "sshd -t rejected the config with the corp auth drop-in; it was removed and sshd not reloaded."
- name: "Schedule dnsmasq restart 15s after reboot"
copy:
dest: /etc/cron.d/dnsmasq-restart
@@ -1447,19 +1946,23 @@
- name: "Copy webapp from USB"
# The whole tree (app, config, services, views, templates, static), every
# run, so a re-run with a newer USB updates the app. Was: cp with
# creates: app.py, which never updated an existing install.
# creates: app.py, which never updated an existing install. -i lists
# what changed, so the service restarts only then.
shell: |
set -e
set -eo pipefail
test -f "{{ usb_root }}/webapp/app.py"
rsync -a --delete --exclude __pycache__ "{{ usb_root }}/webapp/" /opt/pxe-webapp/
rsync -ai --delete --exclude __pycache__ "{{ usb_root }}/webapp/" /opt/pxe-webapp/
args:
executable: /bin/bash
register: webapp_rsync
changed_when: webapp_rsync.stdout | trim | length > 0
- name: "Install webapp Python dependencies (offline wheels)"
args:
executable: /bin/bash
shell: |
# Find the pip-wheels directory on the CIDATA mount
# Find the pip-wheels directory on the CIDATA mount. No online
# fallback: an unpinned pip install from PyPI is never wanted here.
export WHEEL_DIR=""
for d in "{{ usb_root }}/pip-wheels" "{{ usb_mount }}/pip-wheels"; do
if [ -d "$d" ] && compgen -G "$d/*.whl" > /dev/null; then
@@ -1467,41 +1970,50 @@
break
fi
done
if [ -n "$WHEEL_DIR" ]; then
# Install wheels directly using Python zipfile (bypasses pip entirely)
# This avoids the pip3/distutils incompatibility between Ubuntu 22.04 debs and Python 3.12
export REQS="{{ usb_root }}/webapp/requirements.txt"
python3 << 'PYEOF'
# Installs only the pinned wheels: the sha256 of every wheel must be in
# webapp/requirements.txt, and each pinned package must have exactly one
# wheel. An older installed version of a package is removed first (by its
# RECORD), so an upgrade does not leave the old files beside the new.
if [ -z "$WHEEL_DIR" ]; then
echo "ERROR: no pip-wheels/*.whl under {{ usb_root }} or {{ usb_mount }}." >&2
echo "Build them with scripts/download-packages.sh and rebuild the USB." >&2
exit 1
fi
# Install wheels directly using Python zipfile (bypasses pip entirely)
# This avoids the pip3/distutils incompatibility between Ubuntu 22.04 debs and Python 3.12
export REQS="{{ usb_root }}/webapp/requirements.txt"
python3 << 'PYEOF'
# Installs only the pinned wheels: the sha256 of each wheel must be one of
# the hashes pinned for THAT package in webapp/requirements.txt, and
# each pinned package must have exactly one wheel. A package already
# installed at the pinned version is left alone; an older installed
# version is removed first (by its RECORD), so an upgrade does not
# leave the old files beside the new. Last line CHANGED / UNCHANGED.
import zipfile, sysconfig, glob, os, re, sys, hashlib, csv, shutil
norm = lambda n: re.sub(r'[-_.]+', '_', n).lower()
site = sysconfig.get_path('platlib')
wheel_dir = os.environ['WHEEL_DIR']
reqs = open(os.environ['REQS']).read()
pins, hashes = {}, set()
pins, hashes = {}, {}
for m in re.finditer(r'^([A-Za-z0-9_.-]+)==([^\s\\]+)((?:\s*\\?\s*--hash=sha256:[0-9a-f]{64})+)', reqs, re.M):
name = re.sub(r'[-_.]+', '_', m.group(1)).lower()
name = norm(m.group(1))
pins[name] = m.group(2)
hashes.update(re.findall(r'sha256:([0-9a-f]{64})', m.group(3)))
hashes[name] = set(re.findall(r'sha256:([0-9a-f]{64})', m.group(3)))
wheels = sorted(glob.glob(os.path.join(wheel_dir, '*.whl')))
seen = {}
for whl in wheels:
h = hashlib.sha256(open(whl, 'rb').read()).hexdigest()
name = re.sub(r'[-_.]+', '_', os.path.basename(whl).split('-')[0]).lower()
if h not in hashes or name not in pins:
sys.exit(f'REFUSED: {os.path.basename(whl)} is not a pinned wheel (sha256 {h[:16]}...)')
parts = os.path.basename(whl).split('-')
name = norm(parts[0])
if name not in pins or h not in hashes[name] or parts[1] != pins[name]:
sys.exit(f'REFUSED: {os.path.basename(whl)} is not the pinned wheel of its package (sha256 {h[:16]}...)')
seen.setdefault(name, []).append(whl)
bad = [n for n in pins if len(seen.get(n, [])) != 1]
if bad:
sys.exit(f'REFUSED: need exactly one wheel for each pinned package; check: {", ".join(bad)}')
changed = False
for name, (whl,) in sorted(seen.items()):
for old in glob.glob(os.path.join(site, '*.dist-info')):
stem = os.path.basename(old)[:-len('.dist-info')]
oname = re.sub(r'[-_.]+', '_', stem.rsplit('-', 1)[0]).lower()
if oname != name:
continue
current = [d for d in glob.glob(os.path.join(site, '*.dist-info'))
if norm(os.path.basename(d)[:-len('.dist-info')].rsplit('-', 1)[0]) == name]
if [os.path.basename(d)[:-len('.dist-info')].rsplit('-', 1)[1] for d in current] == [pins[name]]:
continue
for old in current:
rec = os.path.join(old, 'RECORD')
if os.path.isfile(rec):
for row in csv.reader(open(rec)):
@@ -1514,13 +2026,12 @@
print(f' Installing {os.path.basename(whl)}')
with zipfile.ZipFile(whl) as z:
z.extractall(site)
changed = True
print('All pinned wheels installed to ' + site)
print('CHANGED' if changed else 'UNCHANGED')
PYEOF
else
# Fallback: try system pip (works if system has internet and compatible pip)
python3 -m pip install --break-system-packages flask lxml 2>/dev/null ||
pip3 install --break-system-packages flask lxml
fi
register: webapp_wheels
changed_when: webapp_wheels.stdout_lines | last | default('') == 'CHANGED'
- name: "Create systemd service for PXE webapp"
copy:
@@ -1551,6 +2062,7 @@
[Install]
WantedBy=multi-user.target
register: webapp_unit
- name: "Enable and start PXE webapp service"
systemd:
@@ -1558,6 +2070,17 @@
state: started
enabled: yes
daemon_reload: yes
register: webapp_started
- name: "Restart PXE webapp (new code, wheels or unit on this run)"
# A re-run used to leave the old code running until the next reboot.
systemd:
name: pxe-webapp
state: restarted
daemon_reload: yes
when:
- webapp_rsync is changed or webapp_wheels is changed or webapp_unit is changed
- webapp_started is not changed
# Monthly Blancco CMMC evidence pack (ENT-056/158): on the 1st, save last
# month's ZIP to the reports share's evidence/ folder for the tech to copy
@@ -1602,9 +2125,8 @@
dest: /etc/apache2/sites-available/pxe-server.conf
content: |
# 9009 listens on every address: admins use the webapp from the GE
# corporate NIC. TODO: the webapp has NO authentication; anyone on
# the GE network who can reach this box can drive it. Add auth (or
# restrict 9009 on the corp NIC to an admin subnet in UFW).
# corporate NIC. The webapp has NO authentication: restrict 9009
# (and 22) on the corp NIC with corp_admin_cidrs (UFW). TODO: auth.
Listen 9009
<VirtualHost *:80>
DocumentRoot {{ web_root }}
@@ -1657,3 +2179,8 @@
handlers:
- name: "reload sysctl"
command: sysctl --system
- name: "restart networkd-dispatcher"
systemd:
name: networkd-dispatcher
state: restarted