From 68df59e1173e6d8bb9f376a9c82b35102daaa5a2 Mon Sep 17 00:00:00 2001 From: cproudlock Date: Thu, 6 Aug 2026 14:28:16 -0400 Subject: [PATCH] Record who owns what, and report repo-vs-share drift OWNERSHIP.md Every expensive bug in this pipeline has had one shape: two systems setting the same thing, last writer winning silently. Four happened on 2026-08-06 alone - computer name (package vs run-enrollment), drive letters (PESetup vs a volume finder), enrollment (package vs an at-logon -ManualFallback task that syspreped finished machines), kiosk URLs (GE-Enforce vs site-config). Each is written down with the evidence so the next person deletes a writer instead of adding one. share-drift.py The share is production and the repo is meant to describe it, but drift runs both ways: live hand-edits nobody committed, and repo fixes never deployed. The unattend outage lived only on the share while the repo copy was fine, and nothing compared them. Each mapped pair is classified. git-owned means the repo wins and the pair must match - those fail the run. unreconciled means the two have genuinely diverged and nobody has decided; reported, not failed. The unattends are unreconciled on purpose: live is ~17 KB against ~12 KB in the repo, so a blind push would regress production. Reads over SSH via base64 so BOM and CRLF survive the hop. First run: 8 git-owned pairs all match, 4 known-unreconciled. Run-ShopfloorSetup.ps1 Corrects a comment that was actively misleading. It claimed shopfloor PCs are "vanilla by design" and that the orchestrator runs -ManualFallback to skip BPRT injection and the package entirely. Shopfloor bays DO enrol - the SFLD package joins Entra with its BPRT token and a human assigns the device category in Intune. -ManualFallback runs sysprep /oobe /reboot, which is why wiring it to an at-logon task destroyed the deployment chain. The absent Entra wait is still correct, for a different reason: at that point the bay is on the isolated PXE LAN with no route to Entra (579C144 held 172.16.9.81 and 172.24.19.142, neither in the production ranges). sync_intune retries until the tech re-cables. "Entra ID Joined: false" right after imaging is normal. --- docs/OWNERSHIP.md | 95 +++++++++++ .../shopfloor-setup/Run-ShopfloorSetup.ps1 | 48 +++++- scripts/share-drift.py | 153 ++++++++++++++++++ 3 files changed, 293 insertions(+), 3 deletions(-) create mode 100644 docs/OWNERSHIP.md create mode 100755 scripts/share-drift.py diff --git a/docs/OWNERSHIP.md b/docs/OWNERSHIP.md new file mode 100644 index 0000000..de8b5d9 --- /dev/null +++ b/docs/OWNERSHIP.md @@ -0,0 +1,95 @@ +# Who owns what + +Every bug that cost a day in this pipeline has had the same shape: two systems +both setting the same thing, and whichever ran last silently won. This file says +who owns each concern so the next person deletes the loser instead of adding a +third writer. + +All four examples below are real, from 2026-08-06. + +## Computer name - the PPKG owns it + +The provisioning package declares: + +```xml +F%SERIAL% +``` + +so a bay comes up as `F`. + +`run-enrollment.ps1` used to also run `Rename-Computer -NewName "E$serial"`. +Both are pending renames; last writer wins at reboot. The script ran twice, and +its second run landed after the package had already queued `F579C144`, so the +bay came up `E579C144` with the package reporting no errors at all. + +**Rule:** nothing in this repo renames a machine. If the naming convention +changes, it changes in the package. + +## Drive letters - PESetup owns them + +PESetup hardcodes `W:` in nine places - every copy destination, both DISM +offline sessions, `bcdboot`, `reagentc` - and creates it during its own disk +preparation. + +`startnet.cmd` briefly had a volume finder that scanned for "the applied +Windows volume" and `diskpart`-assigned it to `W:`. On a re-image it found the +*previous* install, relabelled a partition PESetup was about to erase, and every +staging copy failed into a volume that no longer existed. + +**Rule:** wait for `W:\Windows\System32\config\system` - the hive only exists +once the WIM apply has written it. Never run `diskpart` while PESetup is +running. + +See `docs/PESETUP-INTERNALS.md`. + +## Enrollment - the PPKG owns it, the orchestrator drives reboots + +The SFLD package joins Entra using the BPRT token in +`0__Accounts_Azure.provxml`. A human then assigns the device category in Intune. + +The vendor's `Start-BulkEnrollOrchestrator.ps1` has two branches. The normal one +aborts the package's own reboot, registers `AutoSecondReboot`, and drives the +Entra join to completion. The `-ManualFallback` one runs `sysprep /oobe /reboot` +- it is an interactive escape hatch for handing a machine back to OOBE, meant to +be triggered by a person. + +The shopfloor unattend registered `-ManualFallback` as an at-logon scheduled +task. So OOBE completed, autologon fired, and four seconds later the machine +syspreped itself back to OOBE, losing the deployment chain permanently. + +**Rule:** shopfloor bays enrol. Never wire `-ManualFallback` to anything +automatic. + +## Kiosk URLs - GE-Enforce owns them + +`plugins/geenforce/seed_display_scope.py` in shopdb-flask is authoritative: + +| display-type.txt | route | +|---|---| +| `Dashboard` | `/shopdb/shopfloor` | +| `Lobby` | `/shopdb/tv` | +| `3DPrintRoom` | `/shopdb/parts-kiosk` | + +Its dispatcher prefers the server-side role from Settings > Dashboard Defaults +(resolved by device IP), falls back to `C:\Enrollment\display-type.txt`, writes +the Startup shortcut itself, and **sweeps** any shortcut matching +`shopfloor-dashboard` or `/shopdb/`. + +`site-config.json` also carries `edgeHomepage` per display type. Those are a +backstop for the window before the kiosk installer runs - a stale value there +gets deleted on the next enforce cycle rather than honoured. + +**Rule:** if a kiosk points at the wrong page, fix the GE-Enforce scope first. +Keep `site-config.json` correct, but do not expect it to win. + +## Repo vs share + +The share is production; the repo is meant to describe it. Drift runs both ways - +live hand-edits nobody committed, and repo fixes never deployed. + +`scripts/share-drift.py` classifies every mapped pair as `git-owned` (repo wins, +safe to push) or `unreconciled` (diverged, nobody has decided). It reports and +never writes. Run it before a build day; `scripts/preflight.py` covers the rest. + +The unattends are `unreconciled` on purpose: the live copies are ~17 KB and the +repo copies ~12 KB, so pushing the repo would regress production. diff --git a/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 b/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 index a7fbf2a..880d135 100644 --- a/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 +++ b/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 @@ -62,7 +62,7 @@ cmd /c "shutdown /a 2>nul" *>$null # done work and continues from where it left off. # # Also top up AutoLogonCount so the SupportUser autologon budget -# (LogonCount=7 from unattend XML) survives extra unplanned reboots. +# (LogonCount=12 from unattend XML) survives extra unplanned reboots. $selfResumeKey = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce' $selfResumeName = 'ResumeRunShopfloorSetup' $selfResumeCmd = 'powershell.exe -NoProfile -ExecutionPolicy Bypass -File "' + $PSCommandPath + '"' @@ -94,9 +94,51 @@ $enrollDir = "C:\Enrollment" $typeFile = Join-Path $enrollDir "pc-type.txt" $setupDir = Join-Path $enrollDir "shopfloor-setup" +# NOTE: there is deliberately NO wait for an Entra join here - but NOT because +# shopfloor bays skip enrollment. They do enrol: the SFLD provisioning package +# joins Entra using the BPRT token it carries, and a human then assigns the +# device category in Intune. +# +# The wait is absent because the join cannot happen yet. At this point the bay is +# still on the isolated PXE LAN with no route to Entra - observed on 579C144 +# 2026-08-06, holding 172.16.9.81 and 172.24.19.142, neither in the production +# ranges. sync_intune retries every 30 seconds until the tech re-cables to +# production, which is the right place to wait. An earlier version blocked here +# for 45 minutes and then warned about a failure that had not happened. +# "Entra ID Joined: false" in C:\Logs\BPRT\criticalChecks.json straight after +# imaging is therefore NORMAL, not a fault. +# +# CORRECTION (2026-08-06): this comment previously claimed shopfloor PCs are +# "vanilla by design" and that the orchestrator runs with -ManualFallback to skip +# BPRT injection and the package entirely. That was wrong and dangerous. +# -ManualFallback runs sysprep /oobe /reboot, so wiring it to an at-logon task +# syspreped finished machines seconds after autologon and destroyed the +# deployment chain. See docs/OWNERSHIP.md. + if (-not (Test-Path $typeFile)) { - Write-Host "No pc-type.txt found - skipping shopfloor setup." - exit 0 + # A missing pc-type.txt means one of two very different things, and the old + # blanket "skip + exit 0" hid the bad one for weeks: four Display bays sat + # at imaging stage 2 with a green exit code and nobody noticed. + # - no C:\Enrollment at all -> this machine was never staged by WinPE + # (pre-imaging, or the staging block never ran). That is a FAILURE on a + # machine that has clearly just been imaged, so say so loudly. + # - C:\Enrollment exists but no pc-type.txt -> staging ran and the write + # failed. Also a failure. + $stagingLog = Join-Path $enrollDir 'winpe-staging.log' + $detail = if (Test-Path $enrollDir) { + "C:\Enrollment exists but pc-type.txt is missing - WinPE staging ran but did not write it. Check $stagingLog." + } else { + "C:\Enrollment does not exist - WinPE staging never ran. The Windows volume was probably not found in startnet.cmd, so pc-type.txt, the enrollment package and shopfloor-setup were ALL skipped." + } + Write-Host "" + Write-Host "================================================================" + Write-Host " FAILED: no pc-type.txt at $typeFile" + Write-Host " $detail" + Write-Host " Shopfloor setup cannot run. This PC is imaged but NOT configured." + Write-Host "================================================================" + Write-Host "" + Report-Stage -Stage 'Run-ShopfloorSetup: FAILED - no pc-type.txt' -Index 2 -Status 'failed' -Error_ $detail + exit 1 } $pcType = (Get-Content $typeFile -First 1).Trim() diff --git a/scripts/share-drift.py b/scripts/share-drift.py new file mode 100755 index 0000000..f746a88 --- /dev/null +++ b/scripts/share-drift.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +""" +share-drift.py - report where the repo and the live share disagree. + +WHY +The share is production. The repo is meant to describe it. On 2026-08-06 an +unattend edit made directly on the share broke every shopfloor, standard and +engineer build for a day - while the repo copy was fine the whole time. Nobody +could have known, because nothing compared them. + +Worse, drift runs both directions. Some live files are AHEAD of the repo +(hand-edits nobody committed) and some repo files are ahead of live (fixes never +deployed). Blindly pushing the repo over the share would have overwritten a +working 17 KB unattend with a stale 12 KB one. + +So this tool REPORTS by default and never writes. Each pair is classified: + + git-owned the repo is the source of truth. Safe to push. + unreconciled the two have diverged and nobody has decided which wins. NOT safe + to push. Reconcile by hand, then move the pair to git-owned. + +Usage: + ./share-drift.py # status table + ./share-drift.py --diff # show the actual differences + ./share-drift.py --only unattend # filter by substring + +Exit non-zero if any git-owned pair differs, so it can gate a deploy. An +unreconciled pair that differs is reported but does not fail - that is a known +state, not a regression. +""" + +import argparse +import base64 +import difflib +import hashlib +import os +import subprocess +import sys + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PXE_HOST = "172.16.9.1" +PXE_USER = "pxe" +PXE_PASS = "pxe" + +GIT_OWNED, UNRECONCILED = "git-owned", "unreconciled" + +# (repo path, live path, ownership) +PAIRS = [ + ("playbook/shopfloor-setup/run-enrollment.ps1", + "/srv/samba/enrollment/scripts/run-enrollment.ps1", GIT_OWNED), + ("playbook/scripts/preflight.ps1", + "/srv/samba/enrollment/scripts/preflight.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Run-ShopfloorSetup.ps1", + "/srv/samba/enrollment/shopfloor-setup/Run-ShopfloorSetup.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Verify-And-Heal-Staging.ps1", + "/srv/samba/enrollment/shopfloor-setup/Verify-And-Heal-Staging.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Fetch-StagingPayload.ps1", + "/srv/samba/enrollment/shopfloor-setup/Fetch-StagingPayload.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/site-config.json", + "/srv/samba/enrollment/shopfloor-setup/site-config.json", GIT_OWNED), + ("playbook/shopfloor-setup/BIOS/models.txt", + "/srv/samba/winpeapps/_shared/BIOS/models.txt", GIT_OWNED), + ("playbook/shopfloor-setup/BIOS/check-bios.cmd", + "/srv/samba/winpeapps/_shared/BIOS/check-bios.cmd", GIT_OWNED), + + # The config/ copy is what startnet stages to C:\Enrollment\site-config.json + # and it has its own edit history - it carried the dead tsgwp00524 host that + # the repo copy never had. Same filename, different lineage. + ("playbook/shopfloor-setup/site-config.json", + "/srv/samba/enrollment/config/site-config.json", UNRECONCILED), + + # The live unattends are the ones that boot machines and they are FAR ahead + # of the repo copies (17 KB vs 12 KB). Pushing the repo over them would + # regress production. Reconcile before promoting to git-owned. + ("playbook/FlatUnattendW10-shopfloor.xml", + "/srv/samba/winpeapps/gea-shopfloor/Deploy/FlatUnattendW10.xml", UNRECONCILED), + ("playbook/FlatUnattendW10.xml", + "/srv/samba/winpeapps/gea-standard/Deploy/FlatUnattendW10.xml", UNRECONCILED), + ("playbook/FlatUnattendW10.xml", + "/srv/samba/winpeapps/gea-engineer/Deploy/FlatUnattendW10.xml", UNRECONCILED), +] + + +def ssh(cmd): + return subprocess.run( + ["sshpass", "-p", PXE_PASS, "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "LogLevel=ERROR", f"{PXE_USER}@{PXE_HOST}", cmd], + capture_output=True, text=True) + + +def live_bytes(path): + """base64 in transit so BOM and CRLF survive the hop unchanged.""" + r = ssh("base64 -w0 '%s' 2>/dev/null" % path) + if r.returncode != 0 or not r.stdout.strip(): + return None + return base64.b64decode(r.stdout.strip()) + + +def repo_bytes(path): + try: + with open(os.path.join(REPO, path), "rb") as f: + return f.read() + except OSError: + return None + + +def short(b): + return hashlib.md5(b).hexdigest()[:10] if b is not None else "MISSING" + + +def main(): + ap = argparse.ArgumentParser(description="Report repo vs live share drift.") + ap.add_argument("--diff", action="store_true", help="show the actual differences") + ap.add_argument("--only", help="only pairs whose paths contain this substring") + args = ap.parse_args() + + blocking = 0 + known = 0 + print("%-13s %-44s %-11s %-11s" % ("OWNERSHIP", "LIVE PATH", "REPO", "LIVE")) + print("-" * 84) + + for repo_path, live_path, owner in PAIRS: + if args.only and args.only not in repo_path and args.only not in live_path: + continue + rb, lb = repo_bytes(repo_path), live_bytes(live_path) + same = rb is not None and lb is not None and rb == lb + tag = "same" if same else "DIFFERS" + if not same: + if owner == GIT_OWNED: + blocking += 1 + else: + known += 1 + print("%-13s %-44s %-11s %-11s %s" + % (owner, live_path.replace("/srv/samba/", ""), short(rb), short(lb), tag)) + + if args.diff and not same and rb is not None and lb is not None: + a = rb.decode("utf-8", "replace").replace("\r\n", "\n").splitlines() + b = lb.decode("utf-8", "replace").replace("\r\n", "\n").splitlines() + for line in list(difflib.unified_diff(a, b, "repo", "live", lineterm="", n=1))[:40]: + print(" " + line) + print() + + print("-" * 84) + print("%d git-owned pair(s) out of sync, %d known-unreconciled" % (blocking, known)) + if blocking: + print("FAILED: a git-owned file differs from the share. Deploy it or commit the live version.") + return 1 + print("PASSED: everything git-owned matches the share.") + return 0 + + +if __name__ == "__main__": + sys.exit(main())