ge-aerospace/pxe-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build/playbook re-run safety: corp access, netplan rollback, first-boot retry

Browse Source 
H1  corp_admin_cidrs (UFW 22/9009 on the corp NIC from these only; refuses
    when it would lock out the current SSH client) and corp_ssh_password_auth
    (sshd drop-in, passwords only from the imaging subnet). Defaults keep the
    current behaviour and print a loud warning.
H2  pxe-netplan-apply.sh: pre-flight refusal (pxe-MAC NIC holds the default
    route, a DHCP lease or the route to an SSH client), run from the USB
    before anything changes; post-apply checks (names, imaging IPv4, corp
    IPv4 back within 30 s, SSH route) and a 120 s systemd-run auto-revert.
H3  restore() puts the previous NIC names back (two-phase rename via
    temporary names, so swaps work) before generate + apply.
H4  a failed rename is fatal (restore + exit 1); the play drops the old
    any-interface UFW rules only after verifying both NICs and their
    per-interface SSH rules.
H5  first-boot checks the playbook exit code, retries at next boot (max 5),
    writes /var/lib/pxe-firstboot.failed and /etc/issue + /etc/motd banners.
M1  build-usb.sh verifies iPXE files against the playbook pins, every wheel
    against its own package's hashes, drops the unpinned pip fallback,
    requires --site-vars or --no-site-vars.
M2  webapp restarts when the code, wheels or unit changed.
M3  ignore-carrier on the imaging NIC; ip_nonlocal_bind kept for the
    apache/networkd boot race.
M4  menu suffix must match blancco_kver; missing kernel with no earlier
    build is fatal; a live menu with the staged Blancco test entry is not
    overwritten.
M5  build-proxmox-iso.sh is unsupported (hard exit); use build-usb --image.
M6  deploy-unattend.sh: password via SSHPASS env and sudo stdin only,
    %q-quoted remote args, sha256 check of the live file; lint-unattend.py
    reads PXE_SERVER/PXE_USER/PXE_PASS and uses sshpass -e.
M7  the playbook lints the unattend files before the force: yes copies
    (lint-unattend.py ships on CIDATA as scripts/); pre-commit ACMR.
M8  Blancco initramfs build: no ignore_errors, pipefail, trap cleanup.
Also: Send-ImagingStage.ps1 deploy, carrier hook install tasks
(pxe-nic-carrier, networkd-dispatcher), L1 imaging-active hold
(allow_disruptive) and no iPXE downgrade below an Updates-page install,
L2 iso-volid, L3 kver pin read from the playbook, L4 cp -rL, L5 refuse
>= 4 GiB WinPE files, L6 PIP_DIR wipe guard, L7 per-package wheel hashes,
no online fallback, L8 check-updates.py (apt_pkg/cached compare, all
components, exit 2 on no debs / bad answers).
This commit is contained in:
cproudlock
2026-10-15 14:32:23 -04:00
parent 53d39bfc50
commit 0b1625ed87
15 changed files with 1401 additions and 614 deletions
Download Patch File Download Diff File Expand all files Collapse all files

376
scripts/build-proxmox-iso.sh
View File

@@ -1,360 +1,28 @@
#!/bin/bash
#
# build-proxmox-iso.sh — Build a self-contained PXE server installer ISO for Proxmox
# build-proxmox-iso.sh - UNSUPPORTED, kept only to say what to use instead.
#
# Repackages the Ubuntu 24.04 Server ISO with:
# - Autoinstall configuration (zero-touch install)
# - All offline .deb packages and Python wheels
# - Ansible playbook, Flask webapp, and boot tools
# This script built a second, hand-maintained copy of the installer
# (its own late-commands and first-boot script). It drifted from
# autoinstall/user-data: it kept the dead-mirror apt block but dropped the
# step that writes the real archive sources back, installed the debs with
# "dpkg -i ... || true" instead of the checked offline-apt-install.sh path,
# had no site vars (so the two-NIC playbook always stopped), and no
# first-boot failure handling. Rather than keep two installers in step,
# there is one: build-usb.sh. Its image mode needs no root and gives a disk
# image a Proxmox VM boots directly:
#
# The resulting ISO can be uploaded to Proxmox, attached to a VM, and booted.
# Ubuntu auto-installs, then first-boot configures all PXE services automatically.
# scripts/build-usb.sh --site-vars site-vars.yml --image pxe-usb.img auto \
# ubuntu-24.04.x-live-server-amd64.iso
# # on the Proxmox host:
# qm importdisk <vmid> pxe-usb.img <storage>
# # attach it as a second disk (or USB), boot order: system disk, then it
#
# Usage:
# ./build-proxmox-iso.sh /path/to/ubuntu-24.04-live-server-amd64.iso [output.iso]
#
# Prerequisites (on build workstation):
# sudo apt install xorriso p7zip-full
#
# Before building, run:
# ./download-packages.sh (downloads offline .debs + pip wheels)
# ./prepare-boot-tools.sh ... (extracts Clonezilla, Blancco, Memtest)
# The VM needs two NICs (imaging bridge with no other DHCP server, and the
# corp/GE bridge); put their MACs in site-vars.yml. tests/vm/ does the same
# with libvirt.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "$0")"/.. && pwd)"
AUTOINSTALL_DIR="$REPO_ROOT/autoinstall"
PLAYBOOK_DIR="$REPO_ROOT/playbook"
OFFLINE_PKG_DIR="$REPO_ROOT/offline-packages"
WEBAPP_DIR="$REPO_ROOT/webapp"
PIP_WHEELS_DIR="$REPO_ROOT/pip-wheels"
BOOT_TOOLS_DIR="$REPO_ROOT/boot-tools"
# --- Validate arguments ---
if [ $# -lt 1 ]; then
echo "Usage: $0 /path/to/ubuntu-24.04-live-server-amd64.iso [output.iso]"
echo ""
echo " Creates a self-contained ISO for deploying the PXE server in Proxmox."
echo " The ISO auto-installs Ubuntu and configures all PXE services."
echo ""
echo "Prerequisites:"
echo " sudo apt install xorriso p7zip-full"
exit 1
fi
UBUNTU_ISO="$(realpath "$1")"
OUTPUT_ISO="${2:-$REPO_ROOT/pxe-server-proxmox.iso}"
# --- Validate prerequisites ---
echo "============================================"
echo "PXE Server Proxmox ISO Builder"
echo "============================================"
echo ""
MISSING_CMDS=()
for cmd in xorriso 7z; do
if ! command -v "$cmd" &>/dev/null; then
MISSING_CMDS+=("$cmd")
fi
done
if [ ${#MISSING_CMDS[@]} -gt 0 ]; then
echo "ERROR: Missing required tools: ${MISSING_CMDS[*]}"
echo "Install with: sudo apt install xorriso p7zip-full"
exit 1
fi
if [ ! -f "$UBUNTU_ISO" ]; then
echo "ERROR: ISO not found at $UBUNTU_ISO"
exit 1
fi
# Quick sanity check: ensure it looks like an Ubuntu Server ISO
ISO_CONTENTS=$(7z l "$UBUNTU_ISO" 2>&1) || true
if ! echo "$ISO_CONTENTS" | grep -q "casper/vmlinuz"; then
echo "ERROR: Does not appear to be an Ubuntu Server ISO (missing casper/vmlinuz)"
exit 1
fi
if [ ! -f "$AUTOINSTALL_DIR/user-data" ]; then
echo "ERROR: user-data not found at $AUTOINSTALL_DIR/user-data"
exit 1
fi
if [ ! -f "$PLAYBOOK_DIR/pxe_server_setup.yml" ]; then
echo "ERROR: pxe_server_setup.yml not found at $PLAYBOOK_DIR/"
exit 1
fi
echo "Ubuntu ISO : $UBUNTU_ISO"
echo "Output ISO : $OUTPUT_ISO"
echo "Source Dir : $REPO_ROOT"
echo ""
# --- Setup work directory with cleanup trap ---
WORK_DIR=$(mktemp -d)
cleanup() { rm -rf "$WORK_DIR"; }
trap cleanup EXIT
EXTRACT_DIR="$WORK_DIR/iso"
BOOT_IMG_DIR="$WORK_DIR/BOOT"
# --- Step 1: Extract Ubuntu ISO ---
echo "[1/6] Extracting Ubuntu ISO..."
7z x -o"$EXTRACT_DIR" "$UBUNTU_ISO" -y >/dev/null 2>&1
# 7z extracts [BOOT] directory containing EFI images needed for rebuild
# Move it out so it doesn't end up in the final ISO filesystem
if [ -d "$EXTRACT_DIR/[BOOT]" ]; then
mv "$EXTRACT_DIR/[BOOT]" "$BOOT_IMG_DIR"
echo " Extracted boot images for BIOS + UEFI"
else
echo "ERROR: [BOOT] directory not found in extracted ISO"
echo " The Ubuntu ISO may be corrupted or an unsupported version."
exit 1
fi
# Ensure files are writable (ISO extraction may set read-only)
chmod -R u+w "$EXTRACT_DIR"
# --- Step 2: Generate autoinstall user-data ---
echo "[2/6] Generating autoinstall configuration..."
mkdir -p "$EXTRACT_DIR/server"
touch "$EXTRACT_DIR/server/meta-data"
# Reuse the common sections (identity, network, storage, SSH) from existing user-data
# and replace late-commands with ISO-specific versions
sed '/^ late-commands:/,$d' "$AUTOINSTALL_DIR/user-data" > "$EXTRACT_DIR/server/user-data"
# Append ISO-specific late-commands
cat >> "$EXTRACT_DIR/server/user-data" << 'LATE_COMMANDS'
late-commands:
# Copy project files from ISO (/cdrom/pxe-data/) to the installed system
- mkdir -p /target/opt/pxe-setup
- cp -r /cdrom/pxe-data/packages /target/opt/pxe-setup/ 2>/dev/null || true
- cp -r /cdrom/pxe-data/playbook /target/opt/pxe-setup/ 2>/dev/null || true
- cp -r /cdrom/pxe-data/webapp /target/opt/pxe-setup/ 2>/dev/null || true
- cp -r /cdrom/pxe-data/pip-wheels /target/opt/pxe-setup/ 2>/dev/null || true
- cp -r /cdrom/pxe-data/boot-tools /target/opt/pxe-setup/ 2>/dev/null || true
# Copy boot files (wimboot, boot.wim, BCD, ipxe.efi, etc.) from pxe-data root
- sh -c 'for f in /cdrom/pxe-data/*; do [ -f "$f" ] && cp "$f" /target/opt/pxe-setup/; done' || true
# Install deb packages in target chroot
- |
curtin in-target --target=/target -- bash -c '
if compgen -G "/opt/pxe-setup/packages/*.deb" > /dev/null; then
dpkg -i /opt/pxe-setup/packages/*.deb 2>/dev/null || true
dpkg -i /opt/pxe-setup/packages/*.deb 2>/dev/null || true
if command -v nmcli >/dev/null; then
systemctl enable NetworkManager
fi
fi
'
# Create first-boot script (reads from local /opt/pxe-setup/)
- |
curtin in-target --target=/target -- bash -c '
cat <<"EOF" > /opt/first-boot.sh
#!/bin/bash
SRC=/opt/pxe-setup
# Install all offline .deb packages
if compgen -G "$SRC/packages/*.deb" > /dev/null; then
dpkg -i $SRC/packages/*.deb 2>/dev/null || true
dpkg -i $SRC/packages/*.deb 2>/dev/null || true
fi
# Run the Ansible playbook (override USB paths to local source)
if [ -f $SRC/playbook/pxe_server_setup.yml ]; then
cd $SRC/playbook
ansible-playbook -i localhost, -c local pxe_server_setup.yml \
-e usb_root=$SRC -e usb_mount=$SRC/playbook
fi
# Disable rc.local to prevent rerunning
sed -i "s|^/opt/first-boot.sh.*|# &|" /etc/rc.local
lvextend -r -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv || true
# Clean up large setup files to save disk space
rm -rf $SRC/packages $SRC/pip-wheels $SRC/boot-tools
rm -f $SRC/boot.wim $SRC/boot.sdi $SRC/bootx64.efi $SRC/wimboot $SRC/ipxe.efi $SRC/BCD $SRC/boot.stl
EOF
'
- curtin in-target --target=/target -- chmod +x /opt/first-boot.sh
# Create rc.local to run first-boot on next startup
- |
curtin in-target --target=/target -- bash -c '
cat <<"EOF" > /etc/rc.local
#!/bin/bash
/opt/first-boot.sh > /var/log/first-boot.log 2>&1 &
exit 0
EOF
'
- curtin in-target --target=/target -- chmod +x /etc/rc.local
user-data:
disable_root: false
refresh-installer:
update: no
LATE_COMMANDS
echo " Generated server/user-data and server/meta-data"
# --- Step 3: Copy project files to pxe-data/ ---
echo "[3/6] Copying project files to ISO..."
PXE_DATA="$EXTRACT_DIR/pxe-data"
mkdir -p "$PXE_DATA"
# Offline .deb packages
if [ -d "$OFFLINE_PKG_DIR" ]; then
mkdir -p "$PXE_DATA/packages"
DEB_COUNT=0
for deb in "$OFFLINE_PKG_DIR"/*.deb; do
if [ -f "$deb" ]; then
cp "$deb" "$PXE_DATA/packages/"
DEB_COUNT=$((DEB_COUNT + 1))
fi
done
echo " Copied $DEB_COUNT .deb packages"
else
echo " WARNING: No offline-packages/ directory. Run download-packages.sh first."
fi
# Ansible playbook
mkdir -p "$PXE_DATA/playbook"
cp "$PLAYBOOK_DIR/"* "$PXE_DATA/playbook/" 2>/dev/null || true
echo " Copied playbook/"
# Flask webapp
if [ -d "$WEBAPP_DIR" ]; then
# Whole tree: app.py imports config, services/ and views/, and the
# Blancco evidence timer runs blancco_evidence_monthly.py.
rm -rf "$PXE_DATA/webapp"
cp -r "$WEBAPP_DIR" "$PXE_DATA/webapp"
find "$PXE_DATA/webapp" -name __pycache__ -type d -prune -exec rm -rf {} +
find "$PXE_DATA/webapp" -name '*.py[co]' -delete
echo " Copied webapp/"
fi
# Python wheels
if [ ! -d "$PIP_WHEELS_DIR" ]; then
echo " pip-wheels/ not found — downloading now..."
mkdir -p "$PIP_WHEELS_DIR"
if pip3 download -d "$PIP_WHEELS_DIR" flask lxml 2>/dev/null; then
echo " Downloaded pip wheels successfully."
else
echo " WARNING: Failed to download pip wheels (no internet?)"
rmdir "$PIP_WHEELS_DIR" 2>/dev/null || true
fi
fi
if [ -d "$PIP_WHEELS_DIR" ]; then
cp -r "$PIP_WHEELS_DIR" "$PXE_DATA/pip-wheels"
WHEEL_COUNT=$(find "$PIP_WHEELS_DIR" -name '*.whl' | wc -l)
echo " Copied pip-wheels/ ($WHEEL_COUNT wheels)"
fi
# WinPE boot files (wimboot, boot.wim, BCD, ipxe.efi, etc.)
BOOT_FILES_DIR="$REPO_ROOT/boot-files"
if [ -d "$BOOT_FILES_DIR" ]; then
BOOT_FILE_COUNT=0
for bf in "$BOOT_FILES_DIR"/*; do
if [ -f "$bf" ]; then
cp "$bf" "$PXE_DATA/"
BOOT_FILE_COUNT=$((BOOT_FILE_COUNT + 1))
fi
done
BOOT_FILES_SIZE=$(du -sh "$BOOT_FILES_DIR" | cut -f1)
echo " Copied $BOOT_FILE_COUNT boot files ($BOOT_FILES_SIZE) — wimboot, boot.wim, ipxe.efi, etc."
else
echo " WARNING: No boot-files/ found (copy WinPE boot files from Media Creator)"
fi
# Boot tools (Clonezilla, Blancco, Memtest)
if [ -d "$BOOT_TOOLS_DIR" ]; then
cp -r "$BOOT_TOOLS_DIR" "$PXE_DATA/boot-tools"
TOOLS_SIZE=$(du -sh "$PXE_DATA/boot-tools" | cut -f1)
echo " Copied boot-tools/ ($TOOLS_SIZE)"
else
echo " No boot-tools/ found (run prepare-boot-tools.sh first)"
fi
# --- Step 4: Modify GRUB for autoinstall ---
echo "[4/6] Configuring autoinstall boot..."
GRUB_CFG="$EXTRACT_DIR/boot/grub/grub.cfg"
if [ ! -f "$GRUB_CFG" ]; then
echo "ERROR: boot/grub/grub.cfg not found in extracted ISO"
exit 1
fi
# Add autoinstall kernel parameter with nocloud datasource pointing to /cdrom/server/
# The semicolon must be escaped as \; in GRUB (it's a command separator)
# Apply to both regular and HWE kernels
sed -i 's|/casper/vmlinuz\b|/casper/vmlinuz autoinstall ds=nocloud\\;s=/cdrom/server/|g' "$GRUB_CFG"
sed -i 's|/casper/hwe-vmlinuz\b|/casper/hwe-vmlinuz autoinstall ds=nocloud\\;s=/cdrom/server/|g' "$GRUB_CFG"
# Reduce timeout for automatic boot (1 second instead of default 30)
sed -i 's/set timeout=.*/set timeout=1/' "$GRUB_CFG"
echo " Modified GRUB: autoinstall enabled, timeout=1s"
# --- Step 5: Rebuild ISO ---
echo "[5/6] Rebuilding ISO (this may take a few minutes)..."
# Verify required boot images exist
EFI_IMG="$BOOT_IMG_DIR/2-Boot-NoEmul.img"
if [ ! -f "$EFI_IMG" ]; then
echo "ERROR: EFI boot image not found at $EFI_IMG"
exit 1
fi
if [ ! -f "$EXTRACT_DIR/boot/grub/i386-pc/eltorito.img" ]; then
echo "ERROR: BIOS boot image not found at boot/grub/i386-pc/eltorito.img"
exit 1
fi
xorriso -as mkisofs -r \
-V 'PXE-SERVER' \
-o "$OUTPUT_ISO" \
--grub2-mbr --interval:local_fs:0s-15s:zero_mbrpt,zero_gpt:"$UBUNTU_ISO" \
--protective-msdos-label \
-partition_cyl_align off \
-partition_offset 16 \
--mbr-force-bootable \
-append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b "$EFI_IMG" \
-appended_part_as_gpt \
-iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7 \
-c '/boot.catalog' \
-b '/boot/grub/i386-pc/eltorito.img' \
-no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info \
-eltorito-alt-boot \
-e '--interval:appended_partition_2:::' \
-no-emul-boot \
"$EXTRACT_DIR"
# --- Step 6: Done ---
echo "[6/6] Cleaning up..."
ISO_SIZE=$(du -sh "$OUTPUT_ISO" | cut -f1)
echo ""
echo "============================================"
echo "Proxmox ISO build complete!"
echo "============================================"
echo ""
echo "Output: $OUTPUT_ISO ($ISO_SIZE)"
echo ""
echo "Proxmox deployment:"
echo " 1. Upload ISO to Proxmox storage (Datacenter -> Storage -> ISO Images)"
echo " 2. Create a new VM:"
echo " - BIOS: OVMF (UEFI) — or SeaBIOS (both work)"
echo " - Memory: 4096 MB"
echo " - CPU: 2+ cores"
echo " - Disk: 40+ GB (VirtIO SCSI)"
echo " - Network: Bridge connected to isolated PXE network"
echo " 3. Attach ISO as CD-ROM and start the VM"
echo " 4. Ubuntu auto-installs (~10-15 minutes, zero interaction)"
echo " 5. After reboot, first-boot configures all PXE services"
echo " 6. Access webapp at http://172.16.9.1:9009"
echo ""
echo "NOTE: The VM's network bridge must be connected to your isolated PXE"
echo " network. The server will use static IP 172.16.9.1/24."
echo ""
echo "build-proxmox-iso.sh is no longer supported (it drifted from autoinstall/user-data)." >&2
echo "Use: scripts/build-usb.sh --site-vars FILE --image pxe-usb.img auto UBUNTU.iso" >&2
echo "then on Proxmox: qm importdisk <vmid> pxe-usb.img <storage> (see the header of this script)." >&2
exit 1

193
scripts/build-usb.sh
View File

@@ -32,9 +32,18 @@
# --skip-boot-tools do not copy boot-tools/
# --site-vars FILE put FILE on CIDATA as playbook/site-vars.yml (the
# per-site NIC MACs; see playbook/site-vars.example.yml)
# --no-site-vars build without site vars (the first-boot playbook then
# stops until they are set on the server). One of the
# two is required; playbook/site-vars.yml in the repo
# is never copied implicitly.
# --yes do not ask for confirmation (device mode still asks
# YES for a non-removable device)
#
# Refuses to build when a payload does not match its pin: the iPXE files in
# boot-files/ against the playbook's ipxe_*_sha256, every wheel in
# pip-wheels/ against its own package's hashes in webapp/requirements.txt,
# and the ISO against the volume id offline-packages/ was resolved for.
#
# WARNING: device mode ERASES the target USB device.
set -euo pipefail
@@ -48,8 +57,8 @@ usage() {
echo "Usage: sudo $0 [options] /dev/sdX /path/to/ubuntu-24.04.iso [/path/to/winpe-images]"
echo " $0 [options] --image out.img SIZE|auto /path/to/ubuntu-24.04.iso [/path/to/winpe-images]"
echo ""
echo " Options: --payload-root DIR, --site-vars FILE, --skip-enrollment, --skip-bios,"
echo " --skip-drivers, --skip-boot-tools, --yes"
echo " Options: --payload-root DIR, --site-vars FILE | --no-site-vars, --skip-enrollment,"
echo " --skip-bios, --skip-drivers, --skip-boot-tools, --yes"
echo ""
echo " The optional last argument is the path to WinPE deployment content"
echo " (e.g., the mounted Media Creator LITE USB). If provided, the images"
@@ -70,6 +79,7 @@ SKIP_DRIVERS=0
SKIP_BOOT_TOOLS=0
ASSUME_YES=0
SITE_VARS=""
NO_SITE_VARS=0
POSITIONAL=()
while [ $# -gt 0 ]; do
case "$1" in
@@ -82,6 +92,7 @@ while [ $# -gt 0 ]; do
--site-vars)
[ $# -ge 2 ] || { usage; exit 1; }
SITE_VARS="$2"; shift 2 ;;
--no-site-vars) NO_SITE_VARS=1; shift ;;
--skip-enrollment) SKIP_ENROLLMENT=1; shift ;;
--skip-bios) SKIP_BIOS=1; shift ;;
--skip-drivers) SKIP_DRIVERS=1; shift ;;
@@ -181,10 +192,112 @@ for f in "$OFFLINE_PKG_DIR/Packages" "$OFFLINE_PKG_DIR/explicit.list" "$REPO_ROO
fi
done
if [ -n "$SITE_VARS" ] && [ ! -f "$SITE_VARS" ]; then
echo "ERROR: --site-vars file not found: $SITE_VARS"
if [ -n "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 1 ]; then
echo "ERROR: give --site-vars FILE or --no-site-vars, not both"
exit 1
fi
if [ -z "$SITE_VARS" ] && [ "$NO_SITE_VARS" -eq 0 ]; then
echo "ERROR: give --site-vars FILE (the server's NIC MACs, see playbook/site-vars.example.yml)"
echo " or --no-site-vars to build a USB whose first boot stops until they are set."
exit 1
fi
if [ -n "$SITE_VARS" ]; then
if [ ! -f "$SITE_VARS" ]; then
echo "ERROR: --site-vars file not found: $SITE_VARS"
exit 1
fi
for v in pxe_nic_mac corp_nic_mac; do
if ! grep -qE "^$v:[[:space:]]*[\"']?([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}[\"']?[[:space:]]*(#.*)?$" "$SITE_VARS"; then
echo "ERROR: $SITE_VARS does not set $v to a MAC (aa:bb:cc:dd:ee:ff)"
exit 1
fi
done
fi
# --- Payload pins: refuse to build a USB the server would reject ---
# iPXE chain: the same sha256 pins the playbook checks (single source: the
# ipxe_*_sha256 vars in playbook/pxe_server_setup.yml).
pin_of() {
sed -n "s/^[[:space:]]*$1:[[:space:]]*\"\([0-9a-f]\{64\}\)\".*/\1/p" "$PLAYBOOK_DIR/pxe_server_setup.yml" | head -1
}
sha_ok() { # sha_ok FILE PIN
[ -f "$1" ] && [ "$(sha256sum <"$1" | cut -d' ' -f1)" = "$2" ]
}
for var in ipxe_tarball_sha256 ipxe_shim_sha256 ipxe_snponly_sha256 ipxe_bios_sha256; do
[ -n "$(pin_of "$var")" ] || { echo "ERROR: no $var pin in playbook/pxe_server_setup.yml"; exit 1; }
done
if ! sha_ok "$BOOT_FILES_DIR/ipxeboot.tar.gz" "$(pin_of ipxe_tarball_sha256)"; then
echo "ERROR: $BOOT_FILES_DIR/ipxeboot.tar.gz is missing or does not match ipxe_tarball_sha256"
echo " (scripts/fetch-ipxe.sh fetches the pinned release)"
exit 1
fi
if ! sha_ok "$BOOT_FILES_DIR/ipxe-shimx64.efi" "$(pin_of ipxe_shim_sha256)"; then
echo "ERROR: $BOOT_FILES_DIR/ipxe-shimx64.efi is missing or does not match ipxe_shim_sha256"
exit 1
fi
for pair in "ipxeboot/x86_64-sb/snponly.efi ipxe_snponly_sha256" "ipxeboot/x86_64/ipxe.pxe ipxe_bios_sha256"; do
member=${pair% *}; var=${pair#* }
got=$(tar -xzOf "$BOOT_FILES_DIR/ipxeboot.tar.gz" "$member" 2>/dev/null | sha256sum | cut -d' ' -f1)
if [ "$got" != "$(pin_of "$var")" ]; then
echo "ERROR: $member in ipxeboot.tar.gz does not match $var"
exit 1
fi
done
echo "iPXE chain matches the playbook pins"
# Wheels: every wheel must be the pinned version of its package with one
# of THAT package's hashes, and each pinned package needs exactly one. No
# download fallback: the playbook refuses anything else.
if ! compgen -G "$PIP_WHEELS_DIR/*.whl" >/dev/null; then
echo "ERROR: no wheels in $PIP_WHEELS_DIR; build them with scripts/download-packages.sh"
exit 1
fi
python3 - "$PIP_WHEELS_DIR" "$WEBAPP_DIR/requirements.txt" <<'PY' || exit 1
import glob, hashlib, os, re, sys
wheel_dir, reqs = sys.argv[1], open(sys.argv[2]).read()
norm = lambda n: re.sub(r"[-_.]+", "_", n).lower()
pins, hashes = {}, {}
for m in re.finditer(r"^([A-Za-z0-9_.-]+)==([^\s\\]+)((?:\s*\\?\s*--hash=sha256:[0-9a-f]{64})+)", reqs, re.M):
pins[norm(m.group(1))] = m.group(2)
hashes[norm(m.group(1))] = set(re.findall(r"sha256:([0-9a-f]{64})", m.group(3)))
if not pins:
sys.exit("ERROR: no hashed pins in webapp/requirements.txt")
seen, bad = {}, []
for whl in sorted(glob.glob(os.path.join(wheel_dir, "*.whl"))):
base = os.path.basename(whl)
name, ver = norm(base.split("-")[0]), base.split("-")[1]
h = hashlib.sha256(open(whl, "rb").read()).hexdigest()
if name not in pins:
bad.append(f"{base}: {name} is not pinned")
elif ver != pins[name]:
bad.append(f"{base}: version {ver}, pinned {pins[name]}")
elif h not in hashes[name]:
bad.append(f"{base}: sha256 {h[:16]}... is not a pinned hash of {name}")
seen.setdefault(name, []).append(base)
bad += [f"{n}: {len(seen.get(n, []))} wheels (need exactly 1)" for n in pins if len(seen.get(n, [])) != 1]
others = [os.path.basename(p) for p in glob.glob(os.path.join(wheel_dir, "*")) if not p.endswith(".whl")]
bad += [f"{o}: not a wheel" for o in others]
if bad:
print("ERROR: pip-wheels/ does not match webapp/requirements.txt:")
print("\n".join(" " + b for b in bad))
sys.exit(1)
print(f"{len(seen)} wheels match their pinned hashes")
PY
# ISO: the offline set is resolved against one ISO's base system.
ISO_VOLID_NOW=$(xorriso -indev "$ISO_PATH" -pvd_info 2>/dev/null | sed -n 's/^Volume Id *: *//p' | head -1)
if [ -f "$OFFLINE_PKG_DIR/iso-volid" ]; then
SET_VOLID=$(head -1 "$OFFLINE_PKG_DIR/iso-volid")
if [ "$SET_VOLID" != "$ISO_VOLID_NOW" ]; then
echo "ERROR: offline-packages/ was built for ISO '$SET_VOLID', this ISO is '$ISO_VOLID_NOW'."
echo " Rebuild it: scripts/download-packages.sh --iso $ISO_PATH $OFFLINE_PKG_DIR"
exit 1
fi
echo "offline-packages/ matches ISO $ISO_VOLID_NOW"
else
echo "WARNING: $OFFLINE_PKG_DIR/iso-volid missing (set built by an older download-packages.sh);"
echo " cannot check it was resolved against $ISO_VOLID_NOW."
fi
# Verify required source files exist
for f in "$AUTOINSTALL_DIR/user-data" "$AUTOINSTALL_DIR/meta-data" "$PLAYBOOK_DIR/pxe_server_setup.yml"; do
@@ -466,7 +579,7 @@ t_mkdir() {
t_cp() {
local dest="$1"; shift
if [ "$IMAGE_MODE" -eq 0 ]; then
cp -r "$@" "$MOUNT_POINT/$dest/"
cp -rL "$@" "$MOUNT_POINT/$dest/"
else
mcopy -i "$CIDATA_IMG" -s -Q -D o "$@" "::/${dest:+$dest/}"
fi
@@ -510,19 +623,27 @@ fi
t_cp "" "$REPO_ROOT/scripts/offline-apt-install.sh"
echo " Copied $DEB_COUNT .deb packages + Packages + explicit.list to packages/"
# Copy playbook directory
t_cp "" "$PLAYBOOK_DIR"
# Copy playbook directory, without a repo-local site-vars.yml: the site
# vars on the USB are exactly --site-vars (or none with --no-site-vars).
mkdir -p "$WORK/stage"
cp -rL "$PLAYBOOK_DIR" "$WORK/stage/playbook"
rm -f "$WORK/stage/playbook/site-vars.yml"
if [ -n "$SITE_VARS" ]; then
cp "$SITE_VARS" "$WORK/stage/playbook/site-vars.yml"
fi
t_cp "" "$WORK/stage/playbook"
rm -rf "$WORK/stage"
echo " Copied playbook/"
if [ -n "$SITE_VARS" ]; then
mkdir -p "$WORK/sv"
cp "$SITE_VARS" "$WORK/sv/site-vars.yml"
t_cp playbook "$WORK/sv/site-vars.yml"
rm -rf "$WORK/sv"
echo " Copied $SITE_VARS as playbook/site-vars.yml"
elif [ ! -f "$PLAYBOOK_DIR/site-vars.yml" ]; then
echo " WARNING: no playbook/site-vars.yml (and no --site-vars): the playbook"
echo " stops until pxe_nic_mac / corp_nic_mac are set on the server."
else
echo " WARNING: --no-site-vars: the first-boot playbook stops until"
echo " pxe_nic_mac / corp_nic_mac are set on the server."
fi
# The unattend linter the playbook runs before it replaces the answer files
# ({{ usb_mount }}/../scripts/lint-unattend.py).
t_mkdir scripts
t_cp scripts "$REPO_ROOT/scripts/lint-unattend.py"
# Copy the whole webapp tree (app.py imports config, services/, views/;
# the Blancco evidence timer runs blancco_evidence_monthly.py), minus caches.
@@ -536,24 +657,10 @@ if [ -d "$WEBAPP_DIR" ]; then
echo " Copied webapp/"
fi
# Copy pip wheels for offline Flask install
if [ ! -d "$PIP_WHEELS_DIR" ]; then
echo " pip-wheels/ not found - downloading now..."
mkdir -p "$PIP_WHEELS_DIR"
if pip3 download -d "$PIP_WHEELS_DIR" flask lxml 2>/dev/null; then
echo " Downloaded pip wheels successfully."
else
echo " WARNING: Failed to download pip wheels (no internet?)"
echo " The PXE server will need internet to install Flask later,"
echo " or manually copy wheels to pip-wheels/ and rebuild."
rmdir "$PIP_WHEELS_DIR" 2>/dev/null || true
fi
fi
if [ -d "$PIP_WHEELS_DIR" ]; then
t_cp "" "$PIP_WHEELS_DIR"
WHEEL_COUNT=$(find -L "$PIP_WHEELS_DIR" -name '*.whl' | wc -l)
echo " Copied pip-wheels/ ($WHEEL_COUNT wheels)"
fi
# Copy the pinned pip wheels (checked against requirements.txt above)
t_cp "" "$PIP_WHEELS_DIR"
WHEEL_COUNT=$(find -L "$PIP_WHEELS_DIR" -name '*.whl' | wc -l)
echo " Copied pip-wheels/ ($WHEEL_COUNT wheels)"
# Copy WinPE boot files (wimboot, boot.wim, BCD, ipxe.efi, etc.) to the root
if [ -d "$BOOT_FILES_DIR" ]; then
@@ -643,8 +750,17 @@ fi
# Optionally copy WinPE deployment images
if [ -n "$WINPE_SOURCE" ] && [ -d "$WINPE_SOURCE" ]; then
echo " Copying WinPE deployment content from $WINPE_SOURCE..."
# FAT32 cannot hold a file of 4 GiB or more, and the playbook's import
# (cp -rn) does not reassemble split parts, so such a file stops here.
BIG=$(find -L "$WINPE_SOURCE" -type f -size +4194303k 2>/dev/null)
if [ -n "$BIG" ]; then
echo "ERROR: WinPE content has files of 4 GiB or more, which FAT32 (CIDATA) cannot hold:"
printf '%s\n' "$BIG" | sed 's/^/ /'
echo " Import those images through the webapp (Images > Import) instead."
exit 1
fi
t_mkdir images
t_cp images "$WINPE_SOURCE"/* || true
t_cp images "$WINPE_SOURCE"/*
IMG_SIZE=$(du -shL "$WINPE_SOURCE" | cut -f1)
echo " Copied WinPE images ($IMG_SIZE)"
elif [ -n "$WINPE_SOURCE" ]; then
@@ -689,9 +805,12 @@ if [ "$IMAGE_MODE" -eq 1 ]; then
echo " or attach it to a VM as a USB disk (see tests/vm/README.md)."
else
echo "Next steps:"
echo " 1. Insert USB into target machine"
echo " 2. Boot from USB (F12 / boot menu)"
echo " 3. Ubuntu will auto-install and configure the PXE server"
echo " 4. After reboot, move the NIC to the isolated PXE network"
echo " 1. Cable both NICs: the imaging NIC (pxe_nic_mac) to the isolated"
echo " imaging switch, the corp NIC (corp_nic_mac) to the GE network"
echo " 2. Insert USB into the target machine, boot from it (F12 / boot menu)"
echo " 3. Ubuntu auto-installs; on first boot the playbook pins the NICs by"
echo " MAC (imaging = static 172.16.9.1, corp = DHCP) and sets up PXE"
echo " 4. Check /var/log/first-boot.log; a failure shows a banner at login"
echo " and on the console, and first boot retries at the next reboot"
fi
echo ""

60
scripts/check-updates.py
View File

@@ -2,18 +2,21 @@
"""check-updates.py - is the offline bundle current? Run before building a USB.
* offline-packages/*.deb against Ubuntu 24.04 (noble, noble-updates,
noble-security; main + universe): which have a newer version, and which
noble-security; main, restricted, universe, multiverse): which have a
newer version, and which
of those are security updates.
* webapp/requirements.txt pins against PyPI (newer release?) and osv.dev
(known vulnerabilities in the pinned version).
Exit 0 = nothing outstanding; 1 = security updates or vulnerable pins found
(the details are printed); 2 = could not check (no network, bad input).
(the details are printed); 2 = could not check (no network, bad or missing
input such as a folder with no .debs, an unreadable index or API answer).
Needs internet (Ubuntu archive, pypi.org, api.osv.dev); read-only.
Usage: scripts/check-updates.py [--debs DIR] [--reqs FILE] [--all]
"""
import argparse
import functools
import glob
import gzip
import json
@@ -26,7 +29,11 @@ import urllib.request
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ARCHIVE = "http://archive.ubuntu.com/ubuntu/dists"
SUITES = ("noble", "noble-updates", "noble-security")
COMPONENTS = ("main", "universe")
COMPONENTS = ("main", "restricted", "universe", "multiverse")
class CheckError(Exception):
"""Could not check (exit 2)."""
def fetch(url, timeout=60):
@@ -34,21 +41,39 @@ def fetch(url, timeout=60):
return r.read()
try:
import apt_pkg # python3-apt: in-process comparison, no fork per pair
apt_pkg.init_system()
except Exception: # not installed: fall back to dpkg, cached
apt_pkg = None
@functools.lru_cache(maxsize=None)
def newer(a, b):
"""True if Debian version a > b."""
if a == b:
return False
if apt_pkg is not None:
return apt_pkg.version_compare(a, b) > 0
return subprocess.run(["dpkg", "--compare-versions", a, "gt", b]).returncode == 0
def ubuntu_index():
"""({package: newest version}, {package: newest security version})."""
def ubuntu_index(wanted):
"""({package: newest version}, {package: newest security version}),
for the package names in wanted only."""
best, sec = {}, {}
for suite in SUITES:
for comp in COMPONENTS:
text = gzip.decompress(fetch(f"{ARCHIVE}/{suite}/{comp}/binary-amd64/Packages.gz")).decode("utf-8", "replace")
try:
text = gzip.decompress(fetch(f"{ARCHIVE}/{suite}/{comp}/binary-amd64/Packages.gz")).decode("utf-8", "replace")
except (OSError, EOFError) as ex:
raise CheckError(f"{suite}/{comp} index: {ex}")
pkg = None
for line in text.splitlines():
if line.startswith("Package: "):
pkg = line[9:].strip()
if pkg not in wanted:
pkg = None
elif line.startswith("Version: ") and pkg:
v = line[9:].strip()
if pkg not in best or newer(v, best[pkg]):
@@ -61,9 +86,9 @@ def ubuntu_index():
def check_debs(deb_dir, show_all):
debs = glob.glob(os.path.join(deb_dir, "*.deb"))
if not debs:
print(f"no .deb files in {deb_dir}: nothing to check (a failed download?)")
return 1
best, sec = ubuntu_index()
raise CheckError(f"no .deb files in {deb_dir}: nothing to check (a failed download?)")
names = {os.path.basename(d).split("_")[0] for d in debs}
best, sec = ubuntu_index(names)
stale, security = [], []
for d in debs:
parts = os.path.basename(d)[:-4].split("_")
@@ -82,14 +107,23 @@ def check_debs(deb_dir, show_all):
def check_pins(reqs):
pins = re.findall(r"^([A-Za-z0-9_.-]+)==([^\s\\]+)", open(reqs).read(), re.M)
try:
text = open(reqs).read()
except OSError as ex:
raise CheckError(f"{reqs}: {ex}")
pins = re.findall(r"^([A-Za-z0-9_.-]+)==([^\s\\]+)", text, re.M)
if not pins:
print(f"no pinned packages in {reqs}")
return 0
q = {"queries": [{"package": {"name": n, "ecosystem": "PyPI"}, "version": v} for n, v in pins]}
req = urllib.request.Request("https://api.osv.dev/v1/querybatch", data=json.dumps(q).encode(),
headers={"Content-Type": "application/json"})
results = json.load(urllib.request.urlopen(req, timeout=60))["results"]
try:
results = json.load(urllib.request.urlopen(req, timeout=60))["results"]
if not isinstance(results, list) or len(results) != len(pins):
raise ValueError("osv.dev returned %r results for %d pins" % (type(results).__name__, len(pins)))
except (ValueError, KeyError, TypeError) as ex:
raise CheckError(f"osv.dev answer: {ex}")
bad = 0
print(f"Python pins in {os.path.relpath(reqs, ROOT)}:")
for (n, v), r in zip(pins, results):
@@ -97,7 +131,7 @@ def check_pins(reqs):
latest = json.loads(fetch(f"https://pypi.org/pypi/{n}/json", 30))["info"]["version"]
except Exception: # report, keep going
latest = "?"
vulns = [x["id"] for x in r.get("vulns", [])]
vulns = [x.get("id", "?") for x in (r or {}).get("vulns", [])]
bad += bool(vulns)
flag = "VULNERABLE" if vulns else ("newer " if latest not in (v, "?") else "ok ")
print(f" {flag} {n:14} {v:10} latest {latest:10} {', '.join(vulns)}")
@@ -112,7 +146,7 @@ def main():
a = ap.parse_args()
try:
problems = check_debs(a.debs, a.all) + check_pins(a.reqs)
except OSError as ex:
except (OSError, CheckError) as ex:
print(f"could not check: {ex}", file=sys.stderr)
return 2
print("PASSED: nothing outstanding" if not problems else f"ATTENTION: {problems} item(s) need updating")

82
scripts/deploy-unattend.sh
View File

@@ -4,14 +4,17 @@
# server. A bad answer file does not fail at copy time: every bay imaged
# with it stops at "Windows could not parse or process unattend answer
# file" and has to be re-imaged. So this refuses to copy unless the
# linter passes, backs the live file up, swaps it in atomically, and lints
# the copy on the server afterwards.
# linter passes, backs the live file up, swaps it in atomically, checks
# the live copy is byte-identical (sha256) to the linted one, and lints the
# copy on the server afterwards. Any check failing restores the backup.
#
# Usage: scripts/deploy-unattend.sh <image-type> <local FlatUnattendW10.xml>
# e.g. scripts/deploy-unattend.sh gea-engineer playbook/FlatUnattendW10-engineer.xml
#
# Env: PXE_SERVER (default 172.16.9.1), PXE_USER (default pxe); the
# remote password is read from PXE_PASS or prompted for.
# remote password is read from PXE_PASS or prompted for. It never appears
# on a command line (local or remote): sshpass gets it from SSHPASS, sudo
# on the server from stdin.
set -euo pipefail
@@ -27,34 +30,75 @@ STAMP="$(date +%Y%m%d-%H%M%S)"
case "$IMAGE" in
*[!A-Za-z0-9._-]*|"") echo "bad image type: $IMAGE" >&2; exit 2 ;;
esac
case "$SERVER$USER_" in
*[!A-Za-z0-9._@:-]*) echo "bad PXE_SERVER / PXE_USER" >&2; exit 2 ;;
esac
[ -f "$SRC" ] || { echo "no such file: $SRC" >&2; exit 2; }
echo "== 1/3 lint $SRC"
echo "== 1/4 lint $SRC"
if ! python3 "$LINT" --local "$SRC"; then
echo "REFUSED: the linter failed; nothing was copied." >&2
exit 1
fi
LOCAL_SHA=$(sha256sum <"$SRC" | cut -d' ' -f1)
if [ -z "${PXE_PASS:-}" ]; then
read -rs -p "password for $USER_@$SERVER: " PXE_PASS; echo
fi
export SSHPASS="$PXE_PASS"
export SSHPASS="$PXE_PASS" PXE_PASS PXE_USER="$USER_" PXE_SERVER="$SERVER"
SSH=(sshpass -e ssh -o StrictHostKeyChecking=no -o ConnectTimeout=8 "$USER_@$SERVER")
echo "== 2/3 back up and swap $DEST on $SERVER"
sshpass -e scp -q -o StrictHostKeyChecking=no "$SRC" "$USER_@$SERVER:/tmp/unattend-$STAMP.xml"
"${SSH[@]}" "set -e; P() { echo \"\$SSHPASS_REMOTE\" | sudo -S -p '' \"\$@\"; }; SSHPASS_REMOTE='$PXE_PASS'
[ -f '$DEST' ] || { echo 'no live file at $DEST' >&2; exit 1; }
P cp -a '$DEST' '$DEST.bak-$STAMP'
P install -m 644 -o root -g root /tmp/unattend-$STAMP.xml '$DEST.new'
P mv -f '$DEST.new' '$DEST'
rm -f /tmp/unattend-$STAMP.xml
echo \"backup: $DEST.bak-$STAMP\"; sha256sum '$DEST'"
# remote SCRIPT ARGS...: run SCRIPT under bash on the server with ARGS. The
# first stdin line is the password; the script reads it into $P and hands
# it to "sudo -S" through a pipe (printf is a builtin: not in ps). Every
# word is %q-quoted for the remote shell.
remote() {
local script=$1; shift
local cmd
cmd="bash -c $(printf '%q' "$script") _$(printf ' %q' "$@")"
printf '%s\n' "$PXE_PASS" | "${SSH[@]}" "$cmd"
}
# Prologue every remote script starts with.
PRE='set -euo pipefail; IFS= read -r P; S() { printf "%s\n" "$P" | sudo -k -S -p "" "$@"; };'
echo "== 3/3 lint the live copy"
if ! python3 "$LINT" --server "$SERVER" --image "$IMAGE"; then
echo "LIVE FILE FAILS LINT: restoring the backup." >&2
"${SSH[@]}" "echo '$PXE_PASS' | sudo -S -p '' cp -a '$DEST.bak-$STAMP' '$DEST'"
echo "== 2/4 upload to $SERVER"
TMP=$(remote "$PRE"' mktemp /tmp/unattend.XXXXXX')
case "$TMP" in /tmp/unattend.*) ;; *) echo "remote mktemp failed: $TMP" >&2; exit 1 ;; esac
sshpass -e scp -q -o StrictHostKeyChecking=no "$SRC" "$USER_@$SERVER:$TMP"
echo "== 3/4 back up and swap $DEST"
OUT=$(remote "$PRE"'
DEST=$1 TMP=$2 STAMP=$3 WANT=$4
[ -f "$DEST" ] || { echo "no live file at $DEST" >&2; rm -f "$TMP"; exit 1; }
GOT=$(sha256sum <"$TMP" | cut -d" " -f1)
[ "$GOT" = "$WANT" ] || { echo "upload corrupted: $GOT != $WANT" >&2; rm -f "$TMP"; exit 1; }
S cp -a "$DEST" "$DEST.bak-$STAMP"
S install -m 644 -o root -g root "$TMP" "$DEST.new"
S mv -f "$DEST.new" "$DEST"
rm -f "$TMP"
echo "backup: $DEST.bak-$STAMP"
echo "live-sha256: $(sha256sum <"$DEST" | cut -d" " -f1)"' "$DEST" "$TMP" "$STAMP" "$LOCAL_SHA")
echo "$OUT"
rollback() {
echo "$1: restoring the backup." >&2
remote "$PRE"' S cp -a "$1.bak-$2" "$1"' "$DEST" "$STAMP"
exit 1
}
LIVE_SHA=$(sed -n 's/^live-sha256: //p' <<<"$OUT")
if [ "$LIVE_SHA" != "$LOCAL_SHA" ]; then
rollback "LIVE FILE sha256 $LIVE_SHA != linted copy $LOCAL_SHA"
fi
echo "Deployed $SRC to $IMAGE."
echo "== 4/4 lint the live copy"
if ! python3 "$LINT" --server "$SERVER" --image "$IMAGE" --sha256 >"${TMPDIR:-/tmp}/lint-live-$STAMP.log" 2>&1; then
cat "${TMPDIR:-/tmp}/lint-live-$STAMP.log" >&2
rollback "LIVE FILE FAILS LINT"
fi
cat "${TMPDIR:-/tmp}/lint-live-$STAMP.log"
if ! grep -q "^sha256 $LOCAL_SHA $IMAGE\$" "${TMPDIR:-/tmp}/lint-live-$STAMP.log"; then
rollback "the linter read back a different file than the one deployed"
fi
rm -f "${TMPDIR:-/tmp}/lint-live-$STAMP.log"
echo "Deployed $SRC to $IMAGE (sha256 $LOCAL_SHA)."

27
scripts/download-packages.sh
View File

@@ -44,13 +44,19 @@ set -euo pipefail
SCRIPT_PATH="$(readlink -f "$0")"
REPO_DIR="$(cd "$(dirname "$SCRIPT_PATH")"/.. && pwd)"
# Kernel the Blancco UEFI path boots (playbook var blancco_kver; keep the two
# in step). The playbook builds vmlinuz-ubuntu-70, kexec-initrd-70.img and
# Kernel the Blancco UEFI path boots. The default is read from the playbook
# var blancco_kver (the one source; see its comment for the other places).
# The playbook builds vmlinuz-ubuntu-70, kexec-initrd-70.img and
# kmod-<kver>.tar.gz from this kernel's /boot and /lib/modules, and the
# Canonical-signed image is what Ubuntu's shim starts with Secure Boot on.
# The versioned packages pin it; the HWE metapackage keeps the box on the
# HWE line. Override: BLANCCO_KVER=7.0.0-40-generic ./download-packages.sh
BLANCCO_KVER="${BLANCCO_KVER:-7.0.0-34-generic}"
# (then set blancco_kver to match, or the playbook will not find it).
if [ -z "${BLANCCO_KVER:-}" ]; then
BLANCCO_KVER=$(sed -n 's/^[[:space:]]*blancco_kver:[[:space:]]*"\([^"]*\)".*/\1/p' \
"$REPO_DIR/playbook/pxe_server_setup.yml" | head -1)
fi
[ -n "$BLANCCO_KVER" ] || { echo "ERROR: no blancco_kver in playbook/pxe_server_setup.yml (set BLANCCO_KVER)" >&2; exit 1; }
# Packages installed by the Ansible playbook (pxe_server_setup.yml)
PLAYBOOK_PACKAGES=(
@@ -66,6 +72,7 @@ PLAYBOOK_PACKAGES=(
grub-efi-amd64-bin
grub-common
conntrack
networkd-dispatcher # carrier hook (pxe-nic-carrier); in the base, pinned here anyway
busybox-static
zstd
cpio
@@ -300,6 +307,9 @@ docker run --rm \
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
"$TAG" bash /repo/scripts/download-packages.sh
# The ISO this set was resolved against; build-usb.sh refuses another one.
printf '%s\n' "$VOLID" >"$OUT_DIR_ABS/iso-volid"
# --- 3. offline check: resolve exactly as the server will, no network ---
echo "[3/4] Offline check on a fresh base container (no network)..."
docker run --rm --network none \
@@ -310,8 +320,15 @@ docker run --rm --network none \
echo "[4/4] Downloading Python wheels for the webapp..."
# Exactly the pinned, hashed set in webapp/requirements.txt, for the server's
# Python 3.12 on x86_64. The folder is emptied first: the playbook installs
# every wheel in it and refuses any wheel that is not pinned.
find "$PIP_DIR_HOST" -mindepth 1 -delete
# every wheel in it and refuses any wheel that is not pinned. Only a folder
# of wheels is emptied: a PIP_DIR pointing anywhere else is refused.
NOT_WHEELS=$(find "$PIP_DIR_HOST" -mindepth 1 \( ! -name '*.whl' -o ! -type f \) -print -quit)
if [ -n "$NOT_WHEELS" ]; then
echo "ERROR: $PIP_DIR_HOST holds more than *.whl files (e.g. $NOT_WHEELS); refusing to empty it." >&2
echo " Point PIP_DIR at a wheels-only folder." >&2
exit 1
fi
find "$PIP_DIR_HOST" -mindepth 1 -maxdepth 1 -name '*.whl' -type f -delete
docker run --rm -v "$REPO_DIR:/repo:ro" -v "$PIP_DIR_HOST:/pipout" \
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" ubuntu:24.04 bash -euc '
apt-get update -qq >/dev/null

28
scripts/lint-unattend.py
View File

@@ -38,18 +38,24 @@ Usage:
Exits non-zero when any ERROR is found, so it can gate a deploy.
Remote mode reads PXE_SERVER (default 172.16.9.1), PXE_USER (default pxe)
and PXE_PASS (default pxe) from the environment. The password reaches
sshpass through its environment (sshpass -e), never on a command line.
Requires: sshpass (remote mode only)
"""
import argparse
import os
import re
import shlex
import subprocess
import sys
import xml.dom.minidom
PXE_HOST = "172.16.9.1"
PXE_USER = "pxe"
PXE_PASS = "pxe"
PXE_HOST = os.environ.get("PXE_SERVER") or "172.16.9.1"
PXE_USER = os.environ.get("PXE_USER") or "pxe"
PXE_PASS = os.environ.get("PXE_PASS") or "pxe"
IMAGE_BASE = "/srv/samba/winpeapps"
UNATTEND_REL = "Deploy/FlatUnattendW10.xml"
@@ -76,15 +82,16 @@ ERROR, WARN = "ERROR", "WARN"
def ssh_cmd(host, cmd):
env = dict(os.environ, SSHPASS=PXE_PASS)
return subprocess.run(
["sshpass", "-p", PXE_PASS, "ssh", "-o", "StrictHostKeyChecking=no",
["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
"-o", "LogLevel=ERROR", f"{PXE_USER}@{host}", cmd],
capture_output=True, text=True)
capture_output=True, text=True, env=env)
def read_remote_bytes(host, path):
"""Base64 in transit so a BOM or CRLF survives the hop intact."""
r = ssh_cmd(host, "base64 -w0 '%s' 2>/dev/null" % path)
r = ssh_cmd(host, "base64 -w0 %s 2>/dev/null" % shlex.quote(path))
if r.returncode != 0 or not r.stdout.strip():
return None
import base64
@@ -165,9 +172,13 @@ def main():
parser = argparse.ArgumentParser(description="Validate unattend answer files.")
parser.add_argument("--server", default=PXE_HOST, help=f"PXE server (default: {PXE_HOST})")
parser.add_argument("--image", help="single image type instead of all")
parser.add_argument("--sha256", action="store_true",
help="also print the sha256 of each file linted (deploy-unattend.sh compares it)")
parser.add_argument("--local", nargs="+", metavar="FILE", help="lint local files instead")
parser.add_argument("--quiet", action="store_true", help="suppress WARN lines")
args = parser.parse_args()
if args.image and not re.fullmatch(r"[A-Za-z0-9._-]+", args.image):
sys.exit("ERROR: bad image type %r" % args.image)
findings = []
targets = []
@@ -180,7 +191,7 @@ def main():
images = [args.image]
else:
r = ssh_cmd(args.server,
"find '%s' -maxdepth 1 -mindepth 1 -type d -printf '%%f\\n'" % IMAGE_BASE)
"find %s -maxdepth 1 -mindepth 1 -type d -printf '%%f\\n'" % shlex.quote(IMAGE_BASE))
if r.returncode != 0:
sys.exit("ERROR: cannot list %s: %s" % (IMAGE_BASE, r.stderr.strip()))
images = sorted(n for n in r.stdout.split() if not n.startswith("_"))
@@ -190,6 +201,9 @@ def main():
for name, data in targets:
check(name, data, findings)
if args.sha256 and data is not None:
import hashlib
print("sha256 %s %s" % (hashlib.sha256(data).hexdigest(), name))
print("Linted %d unattend file(s)\n" % len(targets))
shown = [f for f in findings if not (args.quiet and f[0] == WARN)]