diff --git a/docs/OWNERSHIP.md b/docs/OWNERSHIP.md new file mode 100644 index 0000000..de8b5d9 --- /dev/null +++ b/docs/OWNERSHIP.md @@ -0,0 +1,95 @@ +# Who owns what + +Every bug that cost a day in this pipeline has had the same shape: two systems +both setting the same thing, and whichever ran last silently won. This file says +who owns each concern so the next person deletes the loser instead of adding a +third writer. + +All four examples below are real, from 2026-08-06. + +## Computer name - the PPKG owns it + +The provisioning package declares: + +```xml +F%SERIAL% +``` + +so a bay comes up as `F`. + +`run-enrollment.ps1` used to also run `Rename-Computer -NewName "E$serial"`. +Both are pending renames; last writer wins at reboot. The script ran twice, and +its second run landed after the package had already queued `F579C144`, so the +bay came up `E579C144` with the package reporting no errors at all. + +**Rule:** nothing in this repo renames a machine. If the naming convention +changes, it changes in the package. + +## Drive letters - PESetup owns them + +PESetup hardcodes `W:` in nine places - every copy destination, both DISM +offline sessions, `bcdboot`, `reagentc` - and creates it during its own disk +preparation. + +`startnet.cmd` briefly had a volume finder that scanned for "the applied +Windows volume" and `diskpart`-assigned it to `W:`. On a re-image it found the +*previous* install, relabelled a partition PESetup was about to erase, and every +staging copy failed into a volume that no longer existed. + +**Rule:** wait for `W:\Windows\System32\config\system` - the hive only exists +once the WIM apply has written it. Never run `diskpart` while PESetup is +running. + +See `docs/PESETUP-INTERNALS.md`. + +## Enrollment - the PPKG owns it, the orchestrator drives reboots + +The SFLD package joins Entra using the BPRT token in +`0__Accounts_Azure.provxml`. A human then assigns the device category in Intune. + +The vendor's `Start-BulkEnrollOrchestrator.ps1` has two branches. The normal one +aborts the package's own reboot, registers `AutoSecondReboot`, and drives the +Entra join to completion. The `-ManualFallback` one runs `sysprep /oobe /reboot` +- it is an interactive escape hatch for handing a machine back to OOBE, meant to +be triggered by a person. + +The shopfloor unattend registered `-ManualFallback` as an at-logon scheduled +task. So OOBE completed, autologon fired, and four seconds later the machine +syspreped itself back to OOBE, losing the deployment chain permanently. + +**Rule:** shopfloor bays enrol. Never wire `-ManualFallback` to anything +automatic. + +## Kiosk URLs - GE-Enforce owns them + +`plugins/geenforce/seed_display_scope.py` in shopdb-flask is authoritative: + +| display-type.txt | route | +|---|---| +| `Dashboard` | `/shopdb/shopfloor` | +| `Lobby` | `/shopdb/tv` | +| `3DPrintRoom` | `/shopdb/parts-kiosk` | + +Its dispatcher prefers the server-side role from Settings > Dashboard Defaults +(resolved by device IP), falls back to `C:\Enrollment\display-type.txt`, writes +the Startup shortcut itself, and **sweeps** any shortcut matching +`shopfloor-dashboard` or `/shopdb/`. + +`site-config.json` also carries `edgeHomepage` per display type. Those are a +backstop for the window before the kiosk installer runs - a stale value there +gets deleted on the next enforce cycle rather than honoured. + +**Rule:** if a kiosk points at the wrong page, fix the GE-Enforce scope first. +Keep `site-config.json` correct, but do not expect it to win. + +## Repo vs share + +The share is production; the repo is meant to describe it. Drift runs both ways - +live hand-edits nobody committed, and repo fixes never deployed. + +`scripts/share-drift.py` classifies every mapped pair as `git-owned` (repo wins, +safe to push) or `unreconciled` (diverged, nobody has decided). It reports and +never writes. Run it before a build day; `scripts/preflight.py` covers the rest. + +The unattends are `unreconciled` on purpose: the live copies are ~17 KB and the +repo copies ~12 KB, so pushing the repo would regress production. diff --git a/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 b/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 index a7fbf2a..880d135 100644 --- a/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 +++ b/playbook/shopfloor-setup/Run-ShopfloorSetup.ps1 @@ -62,7 +62,7 @@ cmd /c "shutdown /a 2>nul" *>$null # done work and continues from where it left off. # # Also top up AutoLogonCount so the SupportUser autologon budget -# (LogonCount=7 from unattend XML) survives extra unplanned reboots. +# (LogonCount=12 from unattend XML) survives extra unplanned reboots. $selfResumeKey = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce' $selfResumeName = 'ResumeRunShopfloorSetup' $selfResumeCmd = 'powershell.exe -NoProfile -ExecutionPolicy Bypass -File "' + $PSCommandPath + '"' @@ -94,9 +94,51 @@ $enrollDir = "C:\Enrollment" $typeFile = Join-Path $enrollDir "pc-type.txt" $setupDir = Join-Path $enrollDir "shopfloor-setup" +# NOTE: there is deliberately NO wait for an Entra join here - but NOT because +# shopfloor bays skip enrollment. They do enrol: the SFLD provisioning package +# joins Entra using the BPRT token it carries, and a human then assigns the +# device category in Intune. +# +# The wait is absent because the join cannot happen yet. At this point the bay is +# still on the isolated PXE LAN with no route to Entra - observed on 579C144 +# 2026-08-06, holding 172.16.9.81 and 172.24.19.142, neither in the production +# ranges. sync_intune retries every 30 seconds until the tech re-cables to +# production, which is the right place to wait. An earlier version blocked here +# for 45 minutes and then warned about a failure that had not happened. +# "Entra ID Joined: false" in C:\Logs\BPRT\criticalChecks.json straight after +# imaging is therefore NORMAL, not a fault. +# +# CORRECTION (2026-08-06): this comment previously claimed shopfloor PCs are +# "vanilla by design" and that the orchestrator runs with -ManualFallback to skip +# BPRT injection and the package entirely. That was wrong and dangerous. +# -ManualFallback runs sysprep /oobe /reboot, so wiring it to an at-logon task +# syspreped finished machines seconds after autologon and destroyed the +# deployment chain. See docs/OWNERSHIP.md. + if (-not (Test-Path $typeFile)) { - Write-Host "No pc-type.txt found - skipping shopfloor setup." - exit 0 + # A missing pc-type.txt means one of two very different things, and the old + # blanket "skip + exit 0" hid the bad one for weeks: four Display bays sat + # at imaging stage 2 with a green exit code and nobody noticed. + # - no C:\Enrollment at all -> this machine was never staged by WinPE + # (pre-imaging, or the staging block never ran). That is a FAILURE on a + # machine that has clearly just been imaged, so say so loudly. + # - C:\Enrollment exists but no pc-type.txt -> staging ran and the write + # failed. Also a failure. + $stagingLog = Join-Path $enrollDir 'winpe-staging.log' + $detail = if (Test-Path $enrollDir) { + "C:\Enrollment exists but pc-type.txt is missing - WinPE staging ran but did not write it. Check $stagingLog." + } else { + "C:\Enrollment does not exist - WinPE staging never ran. The Windows volume was probably not found in startnet.cmd, so pc-type.txt, the enrollment package and shopfloor-setup were ALL skipped." + } + Write-Host "" + Write-Host "================================================================" + Write-Host " FAILED: no pc-type.txt at $typeFile" + Write-Host " $detail" + Write-Host " Shopfloor setup cannot run. This PC is imaged but NOT configured." + Write-Host "================================================================" + Write-Host "" + Report-Stage -Stage 'Run-ShopfloorSetup: FAILED - no pc-type.txt' -Index 2 -Status 'failed' -Error_ $detail + exit 1 } $pcType = (Get-Content $typeFile -First 1).Trim() diff --git a/scripts/share-drift.py b/scripts/share-drift.py new file mode 100755 index 0000000..f746a88 --- /dev/null +++ b/scripts/share-drift.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +""" +share-drift.py - report where the repo and the live share disagree. + +WHY +The share is production. The repo is meant to describe it. On 2026-08-06 an +unattend edit made directly on the share broke every shopfloor, standard and +engineer build for a day - while the repo copy was fine the whole time. Nobody +could have known, because nothing compared them. + +Worse, drift runs both directions. Some live files are AHEAD of the repo +(hand-edits nobody committed) and some repo files are ahead of live (fixes never +deployed). Blindly pushing the repo over the share would have overwritten a +working 17 KB unattend with a stale 12 KB one. + +So this tool REPORTS by default and never writes. Each pair is classified: + + git-owned the repo is the source of truth. Safe to push. + unreconciled the two have diverged and nobody has decided which wins. NOT safe + to push. Reconcile by hand, then move the pair to git-owned. + +Usage: + ./share-drift.py # status table + ./share-drift.py --diff # show the actual differences + ./share-drift.py --only unattend # filter by substring + +Exit non-zero if any git-owned pair differs, so it can gate a deploy. An +unreconciled pair that differs is reported but does not fail - that is a known +state, not a regression. +""" + +import argparse +import base64 +import difflib +import hashlib +import os +import subprocess +import sys + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PXE_HOST = "172.16.9.1" +PXE_USER = "pxe" +PXE_PASS = "pxe" + +GIT_OWNED, UNRECONCILED = "git-owned", "unreconciled" + +# (repo path, live path, ownership) +PAIRS = [ + ("playbook/shopfloor-setup/run-enrollment.ps1", + "/srv/samba/enrollment/scripts/run-enrollment.ps1", GIT_OWNED), + ("playbook/scripts/preflight.ps1", + "/srv/samba/enrollment/scripts/preflight.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Run-ShopfloorSetup.ps1", + "/srv/samba/enrollment/shopfloor-setup/Run-ShopfloorSetup.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Verify-And-Heal-Staging.ps1", + "/srv/samba/enrollment/shopfloor-setup/Verify-And-Heal-Staging.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/Fetch-StagingPayload.ps1", + "/srv/samba/enrollment/shopfloor-setup/Fetch-StagingPayload.ps1", GIT_OWNED), + ("playbook/shopfloor-setup/site-config.json", + "/srv/samba/enrollment/shopfloor-setup/site-config.json", GIT_OWNED), + ("playbook/shopfloor-setup/BIOS/models.txt", + "/srv/samba/winpeapps/_shared/BIOS/models.txt", GIT_OWNED), + ("playbook/shopfloor-setup/BIOS/check-bios.cmd", + "/srv/samba/winpeapps/_shared/BIOS/check-bios.cmd", GIT_OWNED), + + # The config/ copy is what startnet stages to C:\Enrollment\site-config.json + # and it has its own edit history - it carried the dead tsgwp00524 host that + # the repo copy never had. Same filename, different lineage. + ("playbook/shopfloor-setup/site-config.json", + "/srv/samba/enrollment/config/site-config.json", UNRECONCILED), + + # The live unattends are the ones that boot machines and they are FAR ahead + # of the repo copies (17 KB vs 12 KB). Pushing the repo over them would + # regress production. Reconcile before promoting to git-owned. + ("playbook/FlatUnattendW10-shopfloor.xml", + "/srv/samba/winpeapps/gea-shopfloor/Deploy/FlatUnattendW10.xml", UNRECONCILED), + ("playbook/FlatUnattendW10.xml", + "/srv/samba/winpeapps/gea-standard/Deploy/FlatUnattendW10.xml", UNRECONCILED), + ("playbook/FlatUnattendW10.xml", + "/srv/samba/winpeapps/gea-engineer/Deploy/FlatUnattendW10.xml", UNRECONCILED), +] + + +def ssh(cmd): + return subprocess.run( + ["sshpass", "-p", PXE_PASS, "ssh", "-o", "StrictHostKeyChecking=no", + "-o", "LogLevel=ERROR", f"{PXE_USER}@{PXE_HOST}", cmd], + capture_output=True, text=True) + + +def live_bytes(path): + """base64 in transit so BOM and CRLF survive the hop unchanged.""" + r = ssh("base64 -w0 '%s' 2>/dev/null" % path) + if r.returncode != 0 or not r.stdout.strip(): + return None + return base64.b64decode(r.stdout.strip()) + + +def repo_bytes(path): + try: + with open(os.path.join(REPO, path), "rb") as f: + return f.read() + except OSError: + return None + + +def short(b): + return hashlib.md5(b).hexdigest()[:10] if b is not None else "MISSING" + + +def main(): + ap = argparse.ArgumentParser(description="Report repo vs live share drift.") + ap.add_argument("--diff", action="store_true", help="show the actual differences") + ap.add_argument("--only", help="only pairs whose paths contain this substring") + args = ap.parse_args() + + blocking = 0 + known = 0 + print("%-13s %-44s %-11s %-11s" % ("OWNERSHIP", "LIVE PATH", "REPO", "LIVE")) + print("-" * 84) + + for repo_path, live_path, owner in PAIRS: + if args.only and args.only not in repo_path and args.only not in live_path: + continue + rb, lb = repo_bytes(repo_path), live_bytes(live_path) + same = rb is not None and lb is not None and rb == lb + tag = "same" if same else "DIFFERS" + if not same: + if owner == GIT_OWNED: + blocking += 1 + else: + known += 1 + print("%-13s %-44s %-11s %-11s %s" + % (owner, live_path.replace("/srv/samba/", ""), short(rb), short(lb), tag)) + + if args.diff and not same and rb is not None and lb is not None: + a = rb.decode("utf-8", "replace").replace("\r\n", "\n").splitlines() + b = lb.decode("utf-8", "replace").replace("\r\n", "\n").splitlines() + for line in list(difflib.unified_diff(a, b, "repo", "live", lineterm="", n=1))[:40]: + print(" " + line) + print() + + print("-" * 84) + print("%d git-owned pair(s) out of sync, %d known-unreconciled" % (blocking, known)) + if blocking: + print("FAILED: a git-owned file differs from the share. Deploy it or commit the live version.") + return 1 + print("PASSED: everything git-owned matches the share.") + return 0 + + +if __name__ == "__main__": + sys.exit(main())