ge-aerospace/pxe-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build/playbook re-run safety: corp access, netplan rollback, first-boot retry

Browse Source 
H1  corp_admin_cidrs (UFW 22/9009 on the corp NIC from these only; refuses
    when it would lock out the current SSH client) and corp_ssh_password_auth
    (sshd drop-in, passwords only from the imaging subnet). Defaults keep the
    current behaviour and print a loud warning.
H2  pxe-netplan-apply.sh: pre-flight refusal (pxe-MAC NIC holds the default
    route, a DHCP lease or the route to an SSH client), run from the USB
    before anything changes; post-apply checks (names, imaging IPv4, corp
    IPv4 back within 30 s, SSH route) and a 120 s systemd-run auto-revert.
H3  restore() puts the previous NIC names back (two-phase rename via
    temporary names, so swaps work) before generate + apply.
H4  a failed rename is fatal (restore + exit 1); the play drops the old
    any-interface UFW rules only after verifying both NICs and their
    per-interface SSH rules.
H5  first-boot checks the playbook exit code, retries at next boot (max 5),
    writes /var/lib/pxe-firstboot.failed and /etc/issue + /etc/motd banners.
M1  build-usb.sh verifies iPXE files against the playbook pins, every wheel
    against its own package's hashes, drops the unpinned pip fallback,
    requires --site-vars or --no-site-vars.
M2  webapp restarts when the code, wheels or unit changed.
M3  ignore-carrier on the imaging NIC; ip_nonlocal_bind kept for the
    apache/networkd boot race.
M4  menu suffix must match blancco_kver; missing kernel with no earlier
    build is fatal; a live menu with the staged Blancco test entry is not
    overwritten.
M5  build-proxmox-iso.sh is unsupported (hard exit); use build-usb --image.
M6  deploy-unattend.sh: password via SSHPASS env and sudo stdin only,
    %q-quoted remote args, sha256 check of the live file; lint-unattend.py
    reads PXE_SERVER/PXE_USER/PXE_PASS and uses sshpass -e.
M7  the playbook lints the unattend files before the force: yes copies
    (lint-unattend.py ships on CIDATA as scripts/); pre-commit ACMR.
M8  Blancco initramfs build: no ignore_errors, pipefail, trap cleanup.
Also: Send-ImagingStage.ps1 deploy, carrier hook install tasks
(pxe-nic-carrier, networkd-dispatcher), L1 imaging-active hold
(allow_disruptive) and no iPXE downgrade below an Updates-page install,
L2 iso-volid, L3 kver pin read from the playbook, L4 cp -rL, L5 refuse
>= 4 GiB WinPE files, L6 PIP_DIR wipe guard, L7 per-package wheel hashes,
no online fallback, L8 check-updates.py (apt_pkg/cached compare, all
components, exit 2 on no debs / bad answers).
This commit is contained in:
cproudlock
2026-10-15 14:32:23 -04:00
parent 53d39bfc50
commit 0b1625ed87
15 changed files with 1401 additions and 614 deletions
Download Patch File Download Diff File Expand all files Collapse all files

193
scripts/build-usb.sh
View File

@@ -32,9 +32,18 @@
# --skip-boot-tools do not copy boot-tools/
# --site-vars FILE put FILE on CIDATA as playbook/site-vars.yml (the
# per-site NIC MACs; see playbook/site-vars.example.yml)
# --no-site-vars build without site vars (the first-boot playbook then
# stops until they are set on the server). One of the
# two is required; playbook/site-vars.yml in the repo
# is never copied implicitly.
# --yes do not ask for confirmation (device mode still asks
# YES for a non-removable device)
#
# Refuses to build when a payload does not match its pin: the iPXE files in
# boot-files/ against the playbook's ipxe_*_sha256, every wheel in
# pip-wheels/ against its own package's hashes in webapp/requirements.txt,
# and the ISO against the volume id offline-packages/ was resolved for.
#
# WARNING: device mode ERASES the target USB device.
set -euo pipefail
@@ -48,8 +57,8 @@ usage() {
echo "Usage: sudo $0 [options] /dev/sdX /path/to/ubuntu-24.04.iso [/path/to/winpe-images]"
echo " $0 [options] --image out.img SIZE|auto /path/to/ubuntu-24.04.iso [/path/to/winpe-images]"
echo ""
echo " Options: --payload-root DIR, --site-vars FILE, --skip-enrollment, --skip-bios,"
echo " --skip-drivers, --skip-boot-tools, --yes"
echo " Options: --payload-root DIR, --site-vars FILE | --no-site-vars, --skip-enrollment,"
echo " --skip-bios, --skip-drivers, --skip-boot-tools, --yes"
echo ""
echo " The optional last argument is the path to WinPE deployment content"
echo " (e.g., the mounted Media Creator LITE USB). If provided, the images"
@@ -70,6 +79,7 @@ SKIP_DRIVERS=0
SKIP_BOOT_TOOLS=0
ASSUME_YES=0
SITE_VARS=""
NO_SITE_VARS=0
POSITIONAL=()
while [ $# -gt 0 ]; do
case "$1" in
@@ -82,6 +92,7 @@ while [ $# -gt 0 ]; do
--site-vars)
[ $# -ge 2 ] || { usage; exit 1; }
SITE_VARS="$2"; shift 2 ;;
--no-site-vars) NO_SITE_VARS=1; shift ;;
--skip-enrollment) SKIP_ENROLLMENT=1; shift ;;
--skip-bios) SKIP_BIOS=1; shift ;;
--skip-drivers) SKIP_DRIVERS=1; shift ;;
@@ -181,10 +192,112 @@ for f in "$OFFLINE_PKG_DIR/Packages" "$OFFLINE_PKG_DIR/explicit.list" "$REPO_ROO
fi
done
if [ -n "$SITE_VARS" ] && [ ! -f "$SITE_VARS" ]; then
echo "ERROR: --site-vars file not found: $SITE_VARS"
if [ -n "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 1 ]; then
echo "ERROR: give --site-vars FILE or --no-site-vars, not both"
exit 1
fi
if [ -z "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 0 ]; then
echo "ERROR: give --site-vars FILE (the server's NIC MACs, see playbook/site-vars.example.yml)"
echo " or --no-site-vars to build a USB whose first boot stops until they are set."
exit 1
fi
if [ -n "$SITE_VARS" ]; then
if [ ! -f "$SITE_VARS" ]; then
echo "ERROR: --site-vars file not found: $SITE_VARS"
exit 1
fi
for v in pxe_nic_mac corp_nic_mac; do
if ! grep -qE "^$v:[[:space:]]*[\"']?([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}[\"']?[[:space:]]*(#.*)?$" "$SITE_VARS"; then
echo "ERROR: $SITE_VARS does not set $v to a MAC (aa:bb:cc:dd:ee:ff)"
exit 1
fi
done
fi
# --- Payload pins: refuse to build a USB the server would reject ---
# iPXE chain: the same sha256 pins the playbook checks (single source: the
# ipxe_*_sha256 vars in playbook/pxe_server_setup.yml).
pin_of() {
sed -n "s/^[[:space:]]*$1:[[:space:]]*\"\([0-9a-f]\{64\}\)\".*/\1/p" "$PLAYBOOK_DIR/pxe_server_setup.yml" | head -1
}
sha_ok() { # sha_ok FILE PIN
[ -f "$1" ] && [ "$(sha256sum <"$1" | cut -d' ' -f1)" = "$2" ]
}
for var in ipxe_tarball_sha256 ipxe_shim_sha256 ipxe_snponly_sha256 ipxe_bios_sha256; do
[ -n "$(pin_of "$var")" ] || { echo "ERROR: no $var pin in playbook/pxe_server_setup.yml"; exit 1; }
done
if ! sha_ok "$BOOT_FILES_DIR/ipxeboot.tar.gz" "$(pin_of ipxe_tarball_sha256)"; then
echo "ERROR: $BOOT_FILES_DIR/ipxeboot.tar.gz is missing or does not match ipxe_tarball_sha256"
echo " (scripts/fetch-ipxe.sh fetches the pinned release)"
exit 1
fi
if ! sha_ok "$BOOT_FILES_DIR/ipxe-shimx64.efi" "$(pin_of ipxe_shim_sha256)"; then
echo "ERROR: $BOOT_FILES_DIR/ipxe-shimx64.efi is missing or does not match ipxe_shim_sha256"
exit 1
fi
for pair in "ipxeboot/x86_64-sb/snponly.efi ipxe_snponly_sha256" "ipxeboot/x86_64/ipxe.pxe ipxe_bios_sha256"; do
member=${pair% *}; var=${pair#* }
got=$(tar -xzOf "$BOOT_FILES_DIR/ipxeboot.tar.gz" "$member" 2>/dev/null | sha256sum | cut -d' ' -f1)
if [ "$got" != "$(pin_of "$var")" ]; then
echo "ERROR: $member in ipxeboot.tar.gz does not match $var"
exit 1
fi
done
echo "iPXE chain matches the playbook pins"
# Wheels: every wheel must be the pinned version of its package with one
# of THAT package's hashes, and each pinned package needs exactly one. No
# download fallback: the playbook refuses anything else.
if ! compgen -G "$PIP_WHEELS_DIR/*.whl" >/dev/null; then
echo "ERROR: no wheels in $PIP_WHEELS_DIR; build them with scripts/download-packages.sh"
exit 1
fi
python3 - "$PIP_WHEELS_DIR" "$WEBAPP_DIR/requirements.txt" <<'PY' || exit 1
import glob, hashlib, os, re, sys
wheel_dir, reqs = sys.argv[1], open(sys.argv[2]).read()
norm = lambda n: re.sub(r"[-_.]+", "_", n).lower()
pins, hashes = {}, {}
for m in re.finditer(r"^([A-Za-z0-9_.-]+)==([^\s\\]+)((?:\s*\\?\s*--hash=sha256:[0-9a-f]{64})+)", reqs, re.M):
pins[norm(m.group(1))] = m.group(2)
hashes[norm(m.group(1))] = set(re.findall(r"sha256:([0-9a-f]{64})", m.group(3)))
if not pins:
sys.exit("ERROR: no hashed pins in webapp/requirements.txt")
seen, bad = {}, []
for whl in sorted(glob.glob(os.path.join(wheel_dir, "*.whl"))):
base = os.path.basename(whl)
name, ver = norm(base.split("-")[0]), base.split("-")[1]
h = hashlib.sha256(open(whl, "rb").read()).hexdigest()
if name not in pins:
bad.append(f"{base}: {name} is not pinned")
elif ver != pins[name]:
bad.append(f"{base}: version {ver}, pinned {pins[name]}")
elif h not in hashes[name]:
bad.append(f"{base}: sha256 {h[:16]}... is not a pinned hash of {name}")
seen.setdefault(name, []).append(base)
bad += [f"{n}: {len(seen.get(n, []))} wheels (need exactly 1)" for n in pins if len(seen.get(n, [])) != 1]
others = [os.path.basename(p) for p in glob.glob(os.path.join(wheel_dir, "*")) if not p.endswith(".whl")]
bad += [f"{o}: not a wheel" for o in others]
if bad:
print("ERROR: pip-wheels/ does not match webapp/requirements.txt:")
print("\n".join(" " + b for b in bad))
sys.exit(1)
print(f"{len(seen)} wheels match their pinned hashes")
PY
# ISO: the offline set is resolved against one ISO's base system.
ISO_VOLID_NOW=$(xorriso -indev "$ISO_PATH" -pvd_info 2>/dev/null | sed -n 's/^Volume Id *: *//p' | head -1)
if [ -f "$OFFLINE_PKG_DIR/iso-volid" ]; then
SET_VOLID=$(head -1 "$OFFLINE_PKG_DIR/iso-volid")
if [ "$SET_VOLID" != "$ISO_VOLID_NOW" ]; then
echo "ERROR: offline-packages/ was built for ISO '$SET_VOLID', this ISO is '$ISO_VOLID_NOW'."
echo " Rebuild it: scripts/download-packages.sh --iso $ISO_PATH $OFFLINE_PKG_DIR"
exit 1
fi
echo "offline-packages/ matches ISO $ISO_VOLID_NOW"
else
echo "WARNING: $OFFLINE_PKG_DIR/iso-volid missing (set built by an older download-packages.sh);"
echo " cannot check it was resolved against $ISO_VOLID_NOW."
fi
# Verify required source files exist
for f in "$AUTOINSTALL_DIR/user-data" "$AUTOINSTALL_DIR/meta-data" "$PLAYBOOK_DIR/pxe_server_setup.yml"; do
@@ -466,7 +579,7 @@ t_mkdir() {
t_cp() {
local dest="$1"; shift
if [ "$IMAGE_MODE" -eq 0 ]; then
cp -r "$@" "$MOUNT_POINT/$dest/"
cp -rL "$@" "$MOUNT_POINT/$dest/"
else
mcopy -i "$CIDATA_IMG" -s -Q -D o "$@" "::/${dest:+$dest/}"
fi
@@ -510,19 +623,27 @@ fi
t_cp "" "$REPO_ROOT/scripts/offline-apt-install.sh"
echo " Copied $DEB_COUNT .deb packages + Packages + explicit.list to packages/"
# Copy playbook directory
t_cp "" "$PLAYBOOK_DIR"
# Copy playbook directory, without a repo-local site-vars.yml: the site
# vars on the USB are exactly --site-vars (or none with --no-site-vars).
mkdir -p "$WORK/stage"
cp -rL "$PLAYBOOK_DIR" "$WORK/stage/playbook"
rm -f "$WORK/stage/playbook/site-vars.yml"
if [ -n "$SITE_VARS" ]; then
cp "$SITE_VARS" "$WORK/stage/playbook/site-vars.yml"
fi
t_cp "" "$WORK/stage/playbook"
rm -rf "$WORK/stage"
echo " Copied playbook/"
if [ -n "$SITE_VARS" ]; then
mkdir -p "$WORK/sv"
cp "$SITE_VARS" "$WORK/sv/site-vars.yml"
t_cp playbook "$WORK/sv/site-vars.yml"
rm -rf "$WORK/sv"
echo " Copied $SITE_VARS as playbook/site-vars.yml"
elif [ ! -f "$PLAYBOOK_DIR/site-vars.yml" ]; then
echo " WARNING: no playbook/site-vars.yml (and no --site-vars): the playbook"
echo " stops until pxe_nic_mac / corp_nic_mac are set on the server."
else
echo " WARNING: --no-site-vars: the first-boot playbook stops until"
echo " pxe_nic_mac / corp_nic_mac are set on the server."
fi
# The unattend linter the playbook runs before it replaces the answer files
# ({{ usb_mount }}/../scripts/lint-unattend.py).
t_mkdir scripts
t_cp scripts "$REPO_ROOT/scripts/lint-unattend.py"
# Copy the whole webapp tree (app.py imports config, services/, views/;
# the Blancco evidence timer runs blancco_evidence_monthly.py), minus caches.
@@ -536,24 +657,10 @@ if [ -d "$WEBAPP_DIR" ]; then
echo " Copied webapp/"
fi
# Copy pip wheels for offline Flask install
if [ ! -d "$PIP_WHEELS_DIR" ]; then
echo " pip-wheels/ not found - downloading now..."
mkdir -p "$PIP_WHEELS_DIR"
if pip3 download -d "$PIP_WHEELS_DIR" flask lxml 2>/dev/null; then
echo " Downloaded pip wheels successfully."
else
echo " WARNING: Failed to download pip wheels (no internet?)"
echo " The PXE server will need internet to install Flask later,"
echo " or manually copy wheels to pip-wheels/ and rebuild."
rmdir "$PIP_WHEELS_DIR" 2>/dev/null || true
fi
fi
if [ -d "$PIP_WHEELS_DIR" ]; then
t_cp "" "$PIP_WHEELS_DIR"
WHEEL_COUNT=$(find -L "$PIP_WHEELS_DIR" -name '*.whl' | wc -l)
echo " Copied pip-wheels/ ($WHEEL_COUNT wheels)"
fi
# Copy the pinned pip wheels (checked against requirements.txt above)
t_cp "" "$PIP_WHEELS_DIR"
WHEEL_COUNT=$(find -L "$PIP_WHEELS_DIR" -name '*.whl' | wc -l)
echo " Copied pip-wheels/ ($WHEEL_COUNT wheels)"
# Copy WinPE boot files (wimboot, boot.wim, BCD, ipxe.efi, etc.) to the root
if [ -d "$BOOT_FILES_DIR" ]; then
@@ -643,8 +750,17 @@ fi
# Optionally copy WinPE deployment images
if [ -n "$WINPE_SOURCE" ] && [ -d "$WINPE_SOURCE" ]; then
echo " Copying WinPE deployment content from $WINPE_SOURCE..."
# FAT32 cannot hold a file of 4 GiB or more, and the playbook's import
# (cp -rn) does not reassemble split parts, so such a file stops here.
BIG=$(find -L "$WINPE_SOURCE" -type f -size +4194303k 2>/dev/null)
if [ -n "$BIG" ]; then
echo "ERROR: WinPE content has files of 4 GiB or more, which FAT32 (CIDATA) cannot hold:"
printf '%s\n' "$BIG" | sed 's/^/ /'
echo " Import those images through the webapp (Images > Import) instead."
exit 1
fi
t_mkdir images
t_cp images "$WINPE_SOURCE"/* || true
t_cp images "$WINPE_SOURCE"/*
IMG_SIZE=$(du -shL "$WINPE_SOURCE" | cut -f1)
echo " Copied WinPE images ($IMG_SIZE)"
elif [ -n "$WINPE_SOURCE" ]; then
@@ -689,9 +805,12 @@ if [ "$IMAGE_MODE" -eq 1 ]; then
echo " or attach it to a VM as a USB disk (see tests/vm/README.md)."
else
echo "Next steps:"
echo " 1. Insert USB into target machine"
echo " 2. Boot from USB (F12 / boot menu)"
echo " 3. Ubuntu will auto-install and configure the PXE server"
echo " 4. After reboot, move the NIC to the isolated PXE network"
echo " 1. Cable both NICs: the imaging NIC (pxe_nic_mac) to the isolated"
echo " imaging switch, the corp NIC (corp_nic_mac) to the GE network"
echo " 2. Insert USB into the target machine, boot from it (F12 / boot menu)"
echo " 3. Ubuntu auto-installs; on first boot the playbook pins the NICs by"
echo " MAC (imaging = static 172.16.9.1, corp = DHCP) and sets up PXE"
echo " 4. Check /var/log/first-boot.log; a failure shows a banner at login"
echo " and on the console, and first boot retries at the next reboot"
fi
echo ""