Capture site content from a live server so a new one can build every image

A freshly installed server could not build standard or engineer: about
8 GB they install from existed only on the running server.
scripts/capture-site-content.sh pulls it over SSH into the gitignored
payload:
- _shared/Applications/extra (Office, Adobe, OpenText, Java, fonts, Wi-Fi,
  Zscaler), without pxe-status/ (the playbook deploys that);
- each image's own Deploy/Applications files and folders (GE CA
  certificates, the Engineering Layer payload), without the repo-managed
  Install-EngineeringLayer.ps1 and without stale BPRT copies;
- the links images keep there, as links.txt;
- fresh bios-staging (*.exe plus models.txt, which the Dell update
  rewrites), boot.wim, and the known Blancco file set.

It dry-runs by default, compares by checksum (a Blancco config.img had the
same size and time as the stale copy), writes SHA256SUMS, and refuses what
FAT32 cannot hold: files of 4 GiB or more, Windows-illegal names, and
case-only name clashes.

build-usb.sh copies site-content/ unsplit and refuses to build without it
(unless --skip-site-content), or when SHA256SUMS is missing or does not
cover every file. The playbook checks SHA256SUMS, then merges the content
into winpeapps and recreates the links. It only links to folders that
exist, so it never creates an empty _shared BPRT, which would stop the
first MCL import from seeding it. Link lines are validated before anything
is copied.

pull-bios.sh is gone. It read a path the server no longer uses, with a
hard-coded password.

.gitignore

@@ -129,3 +129,6 @@ reference/e10/partprograms/
 
 # Per-site playbook values (NIC MACs); template: playbook/site-vars.example.yml
 playbook/site-vars.yml
+
+# Site content pulled from a live server (scripts/capture-site-content.sh)
+site-content/

README.md

@@ -180,7 +180,7 @@ pxe-server/
 │   ├── download-packages.sh      # Downloads offline .debs + pip wheels
 │   ├── download-drivers.py       # Downloads Dell drivers directly from dell.com
 │   ├── deploy-bios.sh            # Pushes BIOS updates to enrollment share
-│   ├── pull-bios.sh              # Pulls BIOS binaries from upstream cache
+│   ├── capture-site-content.sh   # Pulls site content + fresh payload from a live server
 │   ├── sync_hardware_models.py   # Syncs hardware model configs across images
 │   ├── Upload-Image.ps1          # Windows: upload MCL cache to PXE via SMB
 │   └── Download-Drivers.ps1      # Windows: download hardware drivers from GE CDN

SETUP.md

@@ -86,6 +86,17 @@ to `pip-wheels/`. About 500 debs (~1.6 GB) + 8 wheels.
 
 Extracts boot files for Blancco and Clonezilla (64-bit with Debian's signed shim, so it boots with Secure Boot on, plus the last 32-bit release for XP-era PCs) into the `boot-tools/` directory. Automatically patches Blancco's `config.img` to auto-save erasure reports to the PXE server's Samba share.
 
+### Step 2b: Capture site content from a running server
+
+```bash
+./scripts/capture-site-content.sh            # dry run: lists what would change
+./scripts/capture-site-content.sh --apply    # pull it
+```
+
+Pulls what neither the repo nor the MCL image import brings, from a working PXE server (default 172.16.9.1, SSH key login as `pxe`): `_shared/Applications/extra` (Office, Adobe, OpenText, Java, fonts, Wi-Fi, Zscaler), the GE CA certificates and the Engineering Layer payload in each image's `Deploy/Applications`, and the links images keep there, into `site-content/`. It also refreshes `bios-staging/` (BIOS files and `models.txt`), `boot-files/boot.wim` and the Blancco files in `boot-tools/blancco/` from the server. It writes `site-content/SHA256SUMS`, which the playbook checks before restoring anything, and refuses content a FAT32 USB cannot hold. Without it a new server cannot build Standard or Engineer (`build-usb.sh` refuses unless you pass `--skip-site-content`).
+
+The first MCL image import on the new server then seeds `_shared/Applications/BPRT`, adding our overlay scripts from the repo.
+
 ### Step 3: Build the USB
 
 ```bash
@@ -234,7 +245,7 @@ pxe-server/
 │   ├── download-packages.sh      # Downloads all offline packages
 │   ├── download-drivers.py       # Downloads Dell drivers from dell.com
 │   ├── deploy-bios.sh            # Pushes BIOS updates to enrollment share
-│   ├── pull-bios.sh              # Pulls BIOS binaries from upstream cache
+│   ├── capture-site-content.sh   # Pulls site content + fresh payload from a live server
 │   ├── sync_hardware_models.py   # Syncs hardware model configs across images
 │   ├── Upload-Image.ps1          # Windows: upload MCL cache to PXE via SMB
 │   └── Download-Drivers.ps1      # Windows: download hardware drivers from GE CDN

playbook/pxe_server_setup.yml

@@ -1093,6 +1093,82 @@
         dest: "{{ samba_share }}/gea-engineer/Deploy/Applications/extra_engineering/Install-EngineeringLayer.ps1"
         mode: '0644'
 
+    # Site content (scripts/capture-site-content.sh on the build PC): what
+    # neither the repo nor the MCL image import brings - _shared/Applications/
+    # extra (Office, Adobe, OpenText, Java, fonts, Wi-Fi, Zscaler), the GE CA
+    # certificates and the Engineering Layer payload in each image's
+    # Deploy/Applications, and the links images keep there. Without it a new
+    # server cannot build standard or engineer. Checked against SHA256SUMS
+    # first: a bad copy stops the play instead of imaging PCs from it.
+    - name: "Site content: check the USB copy against SHA256SUMS"
+      shell: |
+        set -e
+        SC="{{ usb_root }}/site-content"
+        if [ ! -d "$SC" ]; then echo "SKIP: no site-content on the USB"; exit 0; fi
+        cd "$SC"
+        sha256sum --quiet --strict -c SHA256SUMS
+        LISTED=$(grep -c . SHA256SUMS)
+        FOUND=$(find . -type f ! -name SHA256SUMS | wc -l)
+        if [ "$LISTED" -ne "$FOUND" ]; then
+          echo "site-content holds $FOUND files, SHA256SUMS lists $LISTED" >&2; exit 1
+        fi
+        echo "VERIFIED $LISTED files"
+      args:
+        executable: /bin/bash
+      register: site_content_check
+      changed_when: false
+
+    - name: "Site content: restore into winpeapps (merge, nothing deleted)"
+      when: "'VERIFIED' in site_content_check.stdout"
+      shell: |
+        set -euo pipefail
+        SC="{{ usb_root }}/site-content"
+        S="{{ samba_share }}"
+        LINES=""
+        [ -f "$SC/links.txt" ] && LINES=$(grep -v '^#' "$SC/links.txt" | grep . || true)
+        # Check every link line before anything is copied.
+        while IFS=$'\t' read -r img name target; do
+          [ -n "$img" ] || continue
+          case "$img/$name/$target" in *..*) echo "REFUSED odd link line: $img $name $target" >&2; exit 1 ;; esac
+          case "$img$name" in */*) echo "REFUSED odd link line: $img $name" >&2; exit 1 ;; esac
+          case "$target" in _shared/*) ;; *) echo "REFUSED link outside _shared: $target" >&2; exit 1 ;; esac
+        done <<< "$LINES"
+        # Count files written (^>f), not folders whose time was touched.
+        if [ -d "$SC/shared" ]; then
+          N=$(rsync -rt --chmod=D0755,F0755 -i "$SC/shared/" "$S/_shared/" | { grep -c '^>f' || true; })
+          echo "shared: wrote $N files"
+        fi
+        if [ -d "$SC/images" ]; then
+          for d in "$SC/images"/*/; do
+            img=$(basename "$d")
+            mkdir -p "$S/$img/Deploy"
+            N=$(rsync -rt --chmod=D0755,F0755 -i "$d" "$S/$img/Deploy/" | { grep -c '^>f' || true; })
+            echo "$img: wrote $N files"
+          done
+        fi
+        # Links into _shared, as the captured server had them. Only to a
+        # folder that exists: an empty _shared/Applications/BPRT made here
+        # would stop the first MCL import from seeding it (bprt_sync).
+        while IFS=$'\t' read -r img name target; do
+          [ -n "$img" ] || continue
+          link="$S/$img/Deploy/Applications/$name"
+          if [ ! -d "$S/$target" ]; then echo "link later: $img/$name -> $target (not there yet)"; continue; fi
+          if [ -L "$link" ] && [ "$(readlink "$link")" = "$S/$target" ]; then continue; fi
+          if [ -e "$link" ] && [ ! -L "$link" ]; then echo "KEPT real folder $link (not replaced by a link)"; continue; fi
+          mkdir -p "$S/$img/Deploy/Applications"
+          ln -sfn "$S/$target" "$link"
+          echo "link: $img/$name -> $target"
+        done <<< "$LINES"
+      args:
+        executable: /bin/bash
+      register: site_content_restore
+      changed_when: site_content_restore.stdout is search('wrote [1-9]|^link:', multiline=True)
+
+    - name: "Site content: what was restored"
+      debug:
+        var: site_content_restore.stdout_lines
+      when: site_content_restore.stdout_lines is defined
+
     - name: "Deploy BIOS check script + manifest to winpeapps/_shared/BIOS/"
       # Path matches what startnet.cmd reads at WinPE boot:
       #   net use B: \\172.16.9.1\winpeapps\_shared
@@ -1121,6 +1197,9 @@
       shell: >
         if [ -d "{{ usb_root }}/bios" ]; then
           cp -f {{ usb_root }}/bios/*.exe /srv/samba/winpeapps/_shared/BIOS/ 2>/dev/null || true
+          # models.txt from the captured server pairs models with these files;
+          # the repo copy installed above is older (the Dell update rewrites it).
+          [ -f "{{ usb_root }}/bios/models.txt" ] && cp -f "{{ usb_root }}/bios/models.txt" /srv/samba/winpeapps/_shared/BIOS/models.txt
           count=$(find /srv/samba/winpeapps/_shared/BIOS -name '*.exe' | wc -l)
           echo "Deployed $count BIOS binaries"
         else

scripts/build-usb.sh

@@ -24,12 +24,16 @@
 #   --image FILE SIZE   build a disk image file instead of writing a device
 #   --payload-root DIR  take the gitignored payload dirs (offline-packages/,
 #                       pip-wheels/, boot-files/, boot-tools/, enrollment/,
-#                       bios-staging/, drivers-staging/) from DIR instead of
+#                       bios-staging/, drivers-staging/, site-content/) from DIR instead of
 #                       the repo root (e.g. a main checkout next to a worktree)
 #   --skip-enrollment   do not copy enrollment/
 #   --skip-bios         do not copy bios-staging/
 #   --skip-drivers      do not copy drivers-staging/
 #   --skip-boot-tools   do not copy boot-tools/
+#   --skip-site-content do not copy site-content/ (Office, Adobe, certs,
+#                       Engineering Layer...: scripts/capture-site-content.sh).
+#                       Without it a new server cannot build standard or
+#                       engineer until that content is copied over by hand.
 #   --site-vars FILE    put FILE on CIDATA as playbook/site-vars.yml (the
 #                       per-site NIC MACs; see playbook/site-vars.example.yml)
 #   --no-site-vars      build without site vars (the first-boot playbook then
@@ -58,7 +62,7 @@ usage() {
     echo "       $0 [options] --image out.img SIZE|auto /path/to/ubuntu-24.04.iso [/path/to/winpe-images]"
     echo ""
     echo "  Options: --payload-root DIR, --site-vars FILE | --no-site-vars, --skip-enrollment,"
-    echo "           --skip-bios, --skip-drivers, --skip-boot-tools, --yes"
+    echo "           --skip-bios, --skip-drivers, --skip-boot-tools, --skip-site-content, --yes"
     echo ""
     echo "  The optional last argument is the path to WinPE deployment content"
     echo "  (e.g., the mounted Media Creator LITE USB). If provided, the images"
@@ -77,6 +81,7 @@ SKIP_ENROLLMENT=0
 SKIP_BIOS=0
 SKIP_DRIVERS=0
 SKIP_BOOT_TOOLS=0
+SKIP_SITE_CONTENT=0
 ASSUME_YES=0
 SITE_VARS=""
 NO_SITE_VARS=0
@@ -97,6 +102,7 @@ while [ $# -gt 0 ]; do
         --skip-bios) SKIP_BIOS=1; shift ;;
         --skip-drivers) SKIP_DRIVERS=1; shift ;;
         --skip-boot-tools) SKIP_BOOT_TOOLS=1; shift ;;
+        --skip-site-content) SKIP_SITE_CONTENT=1; shift ;;
         --yes|-y) ASSUME_YES=1; shift ;;
         -h|--help) usage; exit 0 ;;
         --) shift; POSITIONAL+=("$@"); break ;;
@@ -124,10 +130,12 @@ BOOT_TOOLS_DIR="$PAYLOAD_ROOT/boot-tools"
 ENROLLMENT_DIR="$PAYLOAD_ROOT/enrollment"
 BIOS_DIR="$PAYLOAD_ROOT/bios-staging"
 DRIVERS_DIR="$PAYLOAD_ROOT/drivers-staging"
+SITE_CONTENT_DIR="$PAYLOAD_ROOT/site-content"
 [ "$SKIP_ENROLLMENT" -eq 1 ] && ENROLLMENT_DIR=""
 [ "$SKIP_BIOS" -eq 1 ] && BIOS_DIR=""
 [ "$SKIP_DRIVERS" -eq 1 ] && DRIVERS_DIR=""
 [ "$SKIP_BOOT_TOOLS" -eq 1 ] && BOOT_TOOLS_DIR=""
+[ "$SKIP_SITE_CONTENT" -eq 1 ] && SITE_CONTENT_DIR=""
 
 # FAT32 has a 4GB max file size; files larger than this are split into
 # chunks that the playbook reassembles with `cat`.
@@ -299,6 +307,34 @@ else
     echo "         cannot check it was resolved against $ISO_VOLID_NOW."
 fi
 
+# site-content/ is restored file by file against its SHA256SUMS, so it is
+# never split: a file FAT32 cannot hold refuses the build instead.
+if [ -n "$SITE_CONTENT_DIR" ]; then
+    if [ ! -d "$SITE_CONTENT_DIR" ]; then
+        echo "ERROR: no site-content/ in $PAYLOAD_ROOT. Run scripts/capture-site-content.sh --apply"
+        echo "       (or pass --skip-site-content: the new server then lacks Office, Adobe, certs"
+        echo "       and the Engineering Layer until they are copied by hand)."
+        exit 1
+    fi
+    if [ ! -f "$SITE_CONTENT_DIR/SHA256SUMS" ]; then
+        echo "ERROR: site-content/SHA256SUMS missing. Re-run scripts/capture-site-content.sh --apply"
+        exit 1
+    fi
+    SC_BIG=$(find -L "$SITE_CONTENT_DIR" -type f -size +4194303k)
+    if [ -n "$SC_BIG" ]; then
+        echo "ERROR: site-content/ has files of 4 GiB or more (FAT32 cannot hold them):"
+        printf '%s\n' "$SC_BIG" | sed 's/^/    /'
+        exit 1
+    fi
+    SC_LISTED=$(grep -c . "$SITE_CONTENT_DIR/SHA256SUMS")
+    SC_FOUND=$(find -L "$SITE_CONTENT_DIR" -type f ! -name SHA256SUMS | wc -l)
+    if [ "$SC_LISTED" -ne "$SC_FOUND" ]; then
+        echo "ERROR: site-content/ holds $SC_FOUND files but SHA256SUMS lists $SC_LISTED."
+        echo "       Re-run scripts/capture-site-content.sh --apply"
+        exit 1
+    fi
+fi
+
 # Verify required source files exist
 for f in "$AUTOINSTALL_DIR/user-data" "$AUTOINSTALL_DIR/meta-data" "$PLAYBOOK_DIR/pxe_server_setup.yml"; do
     if [ ! -f "$f" ]; then
@@ -455,7 +491,7 @@ if [ "$IMAGE_MODE" -eq 1 ]; then
     PAYLOAD_BYTES=$(src_bytes "$AUTOINSTALL_DIR/user-data" "$AUTOINSTALL_DIR/meta-data" \
         "$OFFLINE_PKG_DIR" "$PLAYBOOK_DIR" "$WEBAPP_DIR" "$PIP_WHEELS_DIR" \
         "$BOOT_FILES_DIR" "$BOOT_TOOLS_DIR" "$ENROLLMENT_DIR" "$BIOS_DIR" \
-        "$DRIVERS_DIR" "$WINPE_SOURCE")
+        "$DRIVERS_DIR" "$SITE_CONTENT_DIR" "$WINPE_SOURCE")
     if [ "$IMAGE_SIZE_ARG" = "auto" ]; then
         # payload + 10% + 256 MiB FAT/cluster slack, rounded up to 1 MiB
         CIDATA_BYTES=$(( PAYLOAD_BYTES + PAYLOAD_BYTES / 10 + 256 * 1024 * 1024 ))
@@ -710,6 +746,15 @@ else
     echo "    No enrollment/ directory found (PPKGs can be uploaded via webapp later)"
 fi
 
+# Copy the site content (capture-site-content.sh); the playbook checks
+# SHA256SUMS before it restores anything.
+if [ -n "$SITE_CONTENT_DIR" ]; then
+    t_cp "" "$SITE_CONTENT_DIR"
+    echo "    Copied site-content/ ($(du -shL "$SITE_CONTENT_DIR" | cut -f1), $SC_FOUND files)"
+else
+    echo "    Skipped site-content/ (--skip-site-content)"
+fi
+
 # Copy BIOS update binaries if staged
 if [ -n "$BIOS_DIR" ] && [ -d "$BIOS_DIR" ] && [ "$(ls -A "$BIOS_DIR" 2>/dev/null)" ]; then
     echo "    Copying BIOS update binaries from bios-staging/..."

scripts/capture-site-content.sh

@@ -0,0 +1,237 @@
+#!/bin/bash
+#
+# capture-site-content.sh - Pull the content a PXE server needs but neither
+# the repo nor the MCL image import provides, into the gitignored USB payload,
+# so a freshly installed server can build every image from the USB alone.
+#
+# Run on the build PC (not the server), with the imaging LAN reachable:
+#   scripts/capture-site-content.sh                 show what would change
+#   scripts/capture-site-content.sh --apply         pull it
+#
+# What it pulls (live path -> payload dir):
+#   winpeapps/_shared/Applications/extra/        -> site-content/shared/Applications/extra/
+#       Office, Adobe, OpenText, Java, fonts, Wi-Fi profile, print drivers,
+#       Zscaler: every unattend installs from C:\Deploy\Applications\extra.
+#       pxe-status/ is left out (the playbook deploys Send-ImagingStage.ps1).
+#   winpeapps/<image>/Deploy/Applications/ top-level files (GE CA and Zscaler
+#       certificates) and real folders (extra_engineering)
+#                                                -> site-content/images/<image>/Applications/
+#       Install-EngineeringLayer.ps1 is left out (repo playbook/engineering/).
+#   the links each image keeps in Deploy/Applications (extra, BPRT)
+#                                                -> site-content/links.txt
+#   winpeapps/_shared/BIOS/*.exe + models.txt    -> bios-staging/
+#       models.txt pairs each model with its file; the webapp's Dell update
+#       rewrites it, so the live copy, not the repo's, matches the binaries.
+#   www/win11/sources/boot.wim                   -> boot-files/boot.wim
+#   www/blancco/<the files already in boot-tools/blancco/>
+#                                                -> boot-tools/blancco/
+#       only that known set: kexec-initrd, kmod, preferences.xml and the rest
+#       are built by the playbook on the server.
+#
+# Not pulled, on purpose: BPRT (the first MCL import seeds it, plus our
+# overlay from the repo), Operating Systems / Packages / drivers (the MCL
+# import and Upload-Image.ps1 -IncludeDrivers bring those), Blancco reports,
+# Clonezilla backups and imaging logs (records, not setup).
+#
+# Then it writes site-content/SHA256SUMS (the playbook checks it before it
+# restores anything), and refuses content a FAT32 USB cannot hold: a file of
+# 4 GiB or more, a name Windows rejects, or two names that differ only in case.
+#
+# Needs SSH key login as pxe on the server (files there are world-readable).
+# Folders are mirrored (--delete): what the server no longer has goes from
+# the payload too. Nothing on the server is changed.
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")"/.. && pwd)"
+SERVER="172.16.9.1"
+USER_="pxe"
+PAYLOAD_ROOT="$REPO_ROOT"
+APPLY=0
+
+usage() {
+    echo "Usage: $0 [--apply] [--server IP] [--user NAME] [--payload-root DIR]"
+    echo "  Without --apply it only shows what would change."
+}
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --apply) APPLY=1; shift ;;
+        --server) SERVER="${2:?}"; shift 2 ;;
+        --user) USER_="${2:?}"; shift 2 ;;
+        --payload-root) PAYLOAD_ROOT="$(cd "${2:?}" && pwd)"; shift 2 ;;
+        -h|--help) usage; exit 0 ;;
+        *) usage; exit 1 ;;
+    esac
+done
+
+SHARE="${CAPTURE_SHARE:-/srv/samba/winpeapps}"   # overridable for tests only
+WWW="${CAPTURE_WWW:-/var/www/html}"
+SITE="$PAYLOAD_ROOT/site-content"
+BIOS="$PAYLOAD_ROOT/bios-staging"
+BOOT_FILES="$PAYLOAD_ROOT/boot-files"
+BLANCCO="$PAYLOAD_ROOT/boot-tools/blancco"
+SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=10)
+REMOTE="$USER_@$SERVER"
+
+DRY=(-n)
+[ "$APPLY" -eq 1 ] && DRY=()
+
+# rsync -a over ssh. --checksum: the same size and time is not proof of the
+# same content (a Blancco config.img differed that way). -L: a symlink in a
+# pulled folder becomes its file (FAT32 has no links). Itemized, so the dry
+# run lists every change.
+pull() {
+    rsync -aL --checksum --mkpath --delete --itemize-changes --human-readable --stats \
+        -e "ssh ${SSH_OPTS[*]}" "${DRY[@]}" "$@" | summarize
+}
+
+# Itemized lines are noisy for 7 GB of Office; print counts and the first few.
+summarize() {
+    awk '
+        /^[<>ch.*]/ || /^\*deleting/ { n++; if (n <= 8) print "      " $0; next }
+        /^Number of (regular )?files transferred|^Total transferred file size|^Total file size/ { s = s "      " $0 "\n" }
+        END { if (n > 8) printf "      ... %d changes in all\n", n; else if (n == 0) print "      (no changes)"; printf "%s", s }'
+}
+
+echo "Server : $REMOTE"
+echo "Payload: $PAYLOAD_ROOT"
+[ "$APPLY" -eq 1 ] && echo "Mode   : APPLY" || echo "Mode   : dry run (add --apply to pull)"
+echo
+
+ssh "${SSH_OPTS[@]}" "$REMOTE" true || { echo "ERROR: cannot ssh to $REMOTE (key login needed)" >&2; exit 1; }
+
+# --- what the server has ------------------------------------------------------
+# One remote listing: images with Deploy/Applications, and per image its
+# top-level entries as "image<TAB>kind<TAB>name<TAB>link target".
+LISTING=$(ssh "${SSH_OPTS[@]}" "$REMOTE" "
+    cd '$SHARE' || exit 1
+    for apps in */Deploy/Applications; do
+        img=\${apps%%/*}
+        case \"\$img\" in _*) continue ;; esac
+        [ -d \"\$apps\" ] || continue
+        for e in \"\$apps\"/* \"\$apps\"/.[!.]*; do
+            [ -e \"\$e\" ] || [ -L \"\$e\" ] || continue
+            n=\${e##*/}
+            if [ -L \"\$e\" ]; then printf '%s\tlink\t%s\t%s\n' \"\$img\" \"\$n\" \"\$(readlink \"\$e\")\"
+            elif [ -d \"\$e\" ]; then printf '%s\tdir\t%s\t\n' \"\$img\" \"\$n\"
+            else printf '%s\tfile\t%s\t\n' \"\$img\" \"\$n\"; fi
+        done
+    done")
+
+IMAGES=$(printf '%s\n' "$LISTING" | cut -f1 | sort -u | grep -v '^$' || true)
+[ -n "$IMAGES" ] || { echo "ERROR: no image on $SERVER has Deploy/Applications" >&2; exit 1; }
+
+# Links into _shared are recreated by the playbook. Anything else is reported.
+LINKS=""
+while IFS=$'\t' read -r img kind name target; do
+    [ "$kind" = link ] || continue
+    case "$target" in
+        "$SHARE"/_shared/*) LINKS+="$img	$name	${target#"$SHARE"/}"$'\n' ;;
+        *) echo "WARNING: $img/Deploy/Applications/$name links outside _shared ($target); not captured" ;;
+    esac
+done <<< "$LISTING"
+
+# --- 1. _shared/Applications/extra -------------------------------------------
+echo "[1/5] _shared/Applications/extra -> site-content/shared/Applications/extra/"
+pull --exclude=/pxe-status/ "$REMOTE:$SHARE/_shared/Applications/extra/" "$SITE/shared/Applications/extra/"
+
+# --- 2. per-image Applications content ---------------------------------------
+echo "[2/5] per-image Deploy/Applications files and folders -> site-content/images/"
+for img in $IMAGES; do
+    names=$(printf '%s\n' "$LISTING" | awk -F'\t' -v i="$img" '$1 == i && ($2 == "file" || $2 == "dir") { print $3 }')
+    # BPRT as a real folder is an old import's copy of the vendor folder: not ours.
+    names=$(printf '%s\n' "$names" | grep -vix 'bprt' || true)
+    if [ -z "$names" ]; then
+        echo "    $img: nothing (links only)"
+        [ "$APPLY" -eq 1 ] && rm -rf "$SITE/images/$img"
+        continue
+    fi
+    echo "    $img: $(printf '%s\n' "$names" | tr '\n' ' ')"
+    filter=()
+    while IFS= read -r n; do
+        [ -n "$n" ] && filter+=(--include="/$n" --include="/$n/***")
+    done <<< "$names"
+    pull --exclude=Install-EngineeringLayer.ps1 "${filter[@]}" --exclude='*' \
+        --delete-excluded "$REMOTE:$SHARE/$img/Deploy/Applications/" "$SITE/images/$img/Applications/"
+done
+
+# --- 3. BIOS --------------------------------------------------------------------
+echo "[3/5] _shared/BIOS (*.exe, models.txt) -> bios-staging/"
+pull --include='/*.exe' --include=/models.txt --exclude='*' --delete-excluded \
+    "$REMOTE:$SHARE/_shared/BIOS/" "$BIOS/"
+
+# --- 4. boot.wim -----------------------------------------------------------------
+echo "[4/5] win11/sources/boot.wim -> boot-files/boot.wim"
+pull "$REMOTE:$WWW/win11/sources/boot.wim" "$BOOT_FILES/boot.wim"
+
+# --- 5. Blancco (the known set only) ---------------------------------------------
+echo "[5/5] blancco (files already in boot-tools/blancco/) -> boot-tools/blancco/"
+if [ -d "$BLANCCO" ] && [ -n "$(ls -A "$BLANCCO")" ]; then
+    filter=()
+    for e in "$BLANCCO"/*; do
+        n=${e##*/}
+        filter+=(--include="/$n")
+        [ -d "$e" ] && filter+=(--include="/$n/***")
+    done
+    pull "${filter[@]}" --exclude='*' "$REMOTE:$WWW/blancco/" "$BLANCCO/"
+else
+    echo "    SKIP: boot-tools/blancco/ is empty (run scripts/prepare-boot-tools.sh first)"
+fi
+
+if [ "$APPLY" -ne 1 ]; then
+    echo
+    echo "Dry run: nothing written. Re-run with --apply to pull."
+    exit 0
+fi
+
+# --- links, checks, checksums ---------------------------------------------------
+printf '# image<TAB>name in Deploy/Applications<TAB>link target under winpeapps/\n%s' "$LINKS" > "$SITE/links.txt"
+
+echo
+echo "Checking what a FAT32 USB can hold..."
+BAD=0
+BIG=$(find "$SITE" "$BIOS" "$BOOT_FILES/boot.wim" -type f -size +4194303k 2>/dev/null || true)
+if [ -n "$BIG" ]; then
+    echo "ERROR: files of 4 GiB or more (FAT32 cannot hold them):"; printf '%s\n' "$BIG" | sed 's/^/    /'; BAD=1
+fi
+ILLEGAL=$(cd "$SITE" && find . -name '*[<>:"|?*\\]*' 2>/dev/null || true)
+if [ -n "$ILLEGAL" ]; then
+    echo "ERROR: names Windows/FAT32 rejects:"; printf '%s\n' "$ILLEGAL" | sed 's/^/    /'; BAD=1
+fi
+CASE=$(cd "$SITE" && find . | sort -f | uniq -di || true)
+if [ -n "$CASE" ]; then
+    echo "ERROR: names that differ only in case (one would overwrite the other):"
+    printf '%s\n' "$CASE" | sed 's/^/    /'; BAD=1
+fi
+[ "$BAD" -eq 0 ] || exit 1
+echo "    ok"
+
+echo "Writing site-content/SHA256SUMS..."
+# Written beside site-content, not in it, so the list never names itself.
+( cd "$SITE" && find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum ) > "$PAYLOAD_ROOT/.site-content.SHA256SUMS.tmp"
+mv "$PAYLOAD_ROOT/.site-content.SHA256SUMS.tmp" "$SITE/SHA256SUMS"
+
+echo
+echo "Summary:"
+printf '    site-content/  %s, %s files\n' "$(du -sh "$SITE" | cut -f1)" "$(grep -c . "$SITE/SHA256SUMS")"
+printf '    links.txt      %s links\n' "$(grep -vc '^#' "$SITE/links.txt" || true)"
+printf '    bios-staging/  %s, %s .exe\n' "$(du -sh "$BIOS" | cut -f1)" "$(find "$BIOS" -name '*.exe' | wc -l)"
+printf '    boot.wim       %s\n' "$(du -h "$BOOT_FILES/boot.wim" | cut -f1)"
+
+# Files the repo tracks that the server has a different copy of: review and
+# commit (or revert) before building the USB.
+cd "$REPO_ROOT"
+if [ "$PAYLOAD_ROOT" = "$REPO_ROOT" ] && ! git diff --quiet -- boot-tools/blancco/grub-blancco.cfg 2>/dev/null; then
+    echo
+    echo "NOTE: boot-tools/blancco/grub-blancco.cfg (tracked) now holds the server's copy:"
+    git --no-pager diff --stat -- boot-tools/blancco/grub-blancco.cfg
+fi
+for f in check-bios.cmd hardware-report.ps1; do
+    if ssh "${SSH_OPTS[@]}" "$REMOTE" "cat '$SHARE/_shared/BIOS/$f'" 2>/dev/null \
+            | cmp -s - "playbook/shopfloor-setup/BIOS/$f"; then :; else
+        echo "NOTE: the server's _shared/BIOS/$f differs from playbook/shopfloor-setup/BIOS/$f (the repo copy is installed)"
+    fi
+done
+echo
+echo "Done. Build the USB next (scripts/build-usb.sh copies site-content/)."

scripts/deploy-bios.sh

@@ -21,7 +21,7 @@ SCP="sshpass -p $PXE_PASS scp -o StrictHostKeyChecking=no -o ConnectTimeout=10"
 
 # Verify sources exist
 if [ ! -d "$BIOS_DIR" ] || [ -z "$(ls -A "$BIOS_DIR" 2>/dev/null)" ]; then
-    echo "ERROR: bios-staging/ is empty or missing. Run ./pull-bios.sh first."
+    echo "ERROR: bios-staging/ is empty or missing. Run scripts/capture-site-content.sh --apply first."
     exit 1
 fi
 

scripts/pull-bios.sh

@@ -1,21 +0,0 @@
-#!/bin/bash
-# pull-bios.sh - Pull BIOS update binaries from prod PXE server to bios-staging/
-# Run this with the USB NIC plugged in, before building the USB.
-
-set -e
-
-REPO_ROOT="$(cd "$(dirname "$0")"/.. && pwd)"
-DEST="$REPO_ROOT/bios-staging"
-PXE_SERVER="172.16.9.1"
-PXE_USER="pxe"
-PXE_PASS="pxe"
-
-mkdir -p "$DEST"
-
-echo "Pulling BIOS binaries from $PXE_SERVER..."
-sshpass -p "$PXE_PASS" scp -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
-    "$PXE_USER@$PXE_SERVER:/srv/samba/enrollment/BIOS/*.exe" "$DEST/"
-
-COUNT=$(find "$DEST" -name '*.exe' | wc -l)
-SIZE=$(du -sh "$DEST" | cut -f1)
-echo "Done: $COUNT files ($SIZE) in bios-staging/"