geenforce: shadow runs inside the enforce cycle, not on its own clock
Test bays reported 0 installed / 0 skipped / 0 failed and nothing was being processed. The share is a TRANSIENT mount: GE-Enforce.ps1 maps it for the length of its cycle and unmounts at the end - mounted 12:10:02, unmounted 12:10:27 on FB9TP7V3. The shadow task ran on its own 15-minute schedule, so it woke with the drive gone, handed the engine a path that no longer resolved, and the summary zero-filled. A silent nothing, indistinguishable from a healthy no-op, which is why it read as "not processing" rather than as a failure. Register-ShopdbShadow.ps1 is replaced by Invoke-ShopdbShadow.ps1, which runs AS a manifest entry and invokes the runner directly. The share is mounted because the enforce cycle is what called us; there is one cadence instead of two; and there is no task to register, drift, or heal. It also unregisters the superseded task, so bays that already carry it clean themselves up rather than keeping a run that cannot see the share and overwrites the real result with 0/0/0. The runner gains the Test-Path guard that should have been there: an unreachable -ShareManifestPath now logs, writes an event, and reports a real failure naming the reason, instead of running the engine against nothing. That silent zero is what made this expensive to find. Verified on the win11 VM with a substituted drive removed between runs - the same call gives real counts while mapped and the guard when not. The earlier task self-heal is moot now; it was a fix at the wrong layer, and testing against a local path is what hid a mount I had already noted was dynamic.
This commit is contained in:
cproudlock
@@ -131,6 +131,29 @@ try {
|
||||
if ($ShadowMode -and $ShareManifestPath) {
|
||||
# Shadow: install from the share exactly as today (no payload resolve,
|
||||
# no common merge - the share already carries its own common scope).
|
||||
#
|
||||
# GUARDED, because the share is a TRANSIENT mount. GE-Enforce.ps1 maps it
|
||||
# for the length of its cycle and unmounts at the end, so a caller that
|
||||
# runs on its own schedule finds the drive gone. Handing the engine a
|
||||
# path on a dead drive returned nothing usable and the summary zero-filled
|
||||
# to 0 installed / 0 skipped / 0 failed - a silent nothing that reads
|
||||
# exactly like a healthy no-op. Shadow must run INSIDE the enforce cycle;
|
||||
# this makes the alternative loud instead of invisible.
|
||||
if (-not (Test-Path -LiteralPath $ShareManifestPath)) {
|
||||
$reason = "share manifest not reachable at $ShareManifestPath " +
|
||||
'(is the share still mounted? shadow must run inside the ' +
|
||||
'enforce cycle)'
|
||||
Write-Log $reason 'WARN'
|
||||
Write-ShopdbEventLog -Message ("GE-Enforce shadow could not reach the share manifest for scope '$Scope': $reason") -EntryType 'Error' -EventId 1002
|
||||
try {
|
||||
$failReport = New-ShopdbReport -Scope $Scope -AppliedVersion 0 -Summary @{
|
||||
Installed = 0; Skipped = 0; Failed = 1; Filtered = 0; EnforcerVersion = '2.6'
|
||||
Results = @(@{ Name = '(share-manifest)'; Action = 'failed'; Message = $reason })
|
||||
}
|
||||
Send-ShopdbReport -Config $config -Report $failReport | Out-Null
|
||||
} catch {}
|
||||
exit 0
|
||||
}
|
||||
$manifestToRun = $ShareManifestPath
|
||||
} else {
|
||||
# Optional common-scope inheritance (OFF by default; displays are
|
||||
|
||||
85
plugins/geenforce/client/Invoke-ShopdbShadow.ps1
Normal file
85
plugins/geenforce/client/Invoke-ShopdbShadow.ps1
Normal file
@@ -0,0 +1,85 @@
|
||||
# Invoke-ShopdbShadow.ps1 -- run one shopdb SHADOW cycle, from the manifest.
|
||||
#
|
||||
# Shadow = fetch the shopdb manifest, diff it against the share manifest, report
|
||||
# the cycle to shopdb. The engine still installs FROM THE SHARE, so behaviour is
|
||||
# unchanged. It is the observable step before any cutover.
|
||||
#
|
||||
# WHY THIS RUNS AS A MANIFEST ENTRY AND NOT A SCHEDULED TASK
|
||||
#
|
||||
# The first version registered a separate 15-minute task. That cannot work:
|
||||
# GE-Enforce.ps1 mounts the SFLD share for the length of its own cycle and
|
||||
# unmounts it at the end (mounted 12:10:02, unmounted 12:10:27). A task on its
|
||||
# own schedule therefore wakes up with the drive gone, hands the engine a path
|
||||
# that no longer resolves, and reports 0 installed / 0 skipped / 0 failed - a
|
||||
# silent nothing indistinguishable from a healthy no-op. It also meant two
|
||||
# cadences that could drift apart, and a registration that had to be healed.
|
||||
#
|
||||
# Running here removes all of it: the share is mounted because the enforce cycle
|
||||
# is what invoked us, the cadence is the fleet's own, and there is no task.
|
||||
#
|
||||
# Runs as SYSTEM under GE-Enforce, from an entry gated to the test bays.
|
||||
# Fail-safe: exits 0 on every path - shadowing must never stop a bay enforcing.
|
||||
|
||||
$ErrorActionPreference = 'Continue'
|
||||
|
||||
$InstallDir = 'C:\Program Files\GE\Shopfloor'
|
||||
$BaseUrl = 'https://tsgwp00525.wjs.geaerospace.net/shopdb'
|
||||
$LegacyTask = 'ShopDB GE-Enforce (shadow)'
|
||||
|
||||
function Write-ShadowLog {
|
||||
# To a file as well as the host: the engine records only "ps1: <path>" and
|
||||
# an exit code for a PS1 entry, so Write-Host reaches nothing, and with the
|
||||
# fail-safe exit 0 a silent early-out looks exactly like success.
|
||||
param([string]$Message)
|
||||
$line = "[{0}] [shadow] {1}" -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message
|
||||
Write-Host $line
|
||||
try {
|
||||
$dir = 'C:\Logs\Shopfloor'
|
||||
if (-not (Test-Path $dir)) { New-Item -ItemType Directory -Path $dir -Force | Out-Null }
|
||||
Add-Content -LiteralPath (Join-Path $dir ('shadow-{0}.log' -f (Get-Date -Format yyyyMMdd))) -Value $line
|
||||
} catch { }
|
||||
}
|
||||
|
||||
try {
|
||||
# Retire the scheduled task the earlier version left behind. Bays that got
|
||||
# it would otherwise keep firing a run that cannot see the share, reporting
|
||||
# 0/0/0 over the top of the real result from this one.
|
||||
$stale = Get-ScheduledTask -TaskName $LegacyTask -ErrorAction SilentlyContinue
|
||||
if ($stale) {
|
||||
Unregister-ScheduledTask -TaskName $LegacyTask -Confirm:$false -ErrorAction SilentlyContinue
|
||||
Write-ShadowLog "removed the superseded '$LegacyTask' task (it ran outside the share mount)."
|
||||
}
|
||||
|
||||
# Scope is this script's own directory name, never hardcoded: the same file
|
||||
# ships from more than one scope and a wrong value would shadow the wrong
|
||||
# manifest silently.
|
||||
$scopeDir = Split-Path -Parent $PSScriptRoot
|
||||
$scope = Split-Path -Leaf $scopeDir
|
||||
$shareManifest = Join-Path $scopeDir 'manifest.json'
|
||||
|
||||
$runner = Join-Path $InstallDir 'Invoke-ShopdbEnforce.ps1'
|
||||
$engine = Join-Path $InstallDir 'lib\Install-FromManifest.ps1'
|
||||
foreach ($required in @($runner, $engine, $shareManifest)) {
|
||||
if (-not (Test-Path -LiteralPath $required)) {
|
||||
Write-ShadowLog "MISSING $required - skipping this cycle."
|
||||
exit 0
|
||||
}
|
||||
}
|
||||
|
||||
# BaseUrl only when it differs, so a hand-set value is not churned.
|
||||
$regPath = 'HKLM:\SOFTWARE\GE\ShopDB'
|
||||
if (-not (Test-Path $regPath)) { New-Item -Path $regPath -Force | Out-Null }
|
||||
if ((Get-ItemProperty -Path $regPath -Name BaseUrl -ErrorAction SilentlyContinue).BaseUrl -ne $BaseUrl) {
|
||||
Set-ItemProperty -Path $regPath -Name BaseUrl -Value $BaseUrl
|
||||
Write-ShadowLog "BaseUrl set to $BaseUrl"
|
||||
}
|
||||
|
||||
Write-ShadowLog "shadowing $scope against $shareManifest"
|
||||
& $runner -Scope $scope -EnginePath $engine -ShadowMode -ShareManifestPath $shareManifest
|
||||
Write-ShadowLog "cycle complete (runner exit $LASTEXITCODE)."
|
||||
exit 0
|
||||
}
|
||||
catch {
|
||||
Write-ShadowLog "FAILED: $_"
|
||||
exit 0
|
||||
}
|
||||
@@ -1,129 +0,0 @@
|
||||
# Register-ShopdbShadow.ps1 -- put this bay into shopdb SHADOW mode.
|
||||
#
|
||||
# Shadow mode = fetch the shopdb manifest, diff it against the share manifest,
|
||||
# report the cycle to shopdb, and install FROM THE SHARE exactly as today. Zero
|
||||
# behaviour change. It is the observable step before any cutover.
|
||||
#
|
||||
# Runs as SYSTEM under GE-Enforce, from a manifest entry gated to one hostname.
|
||||
# Idempotent: re-registers the task each cycle so drift self-heals, and writes
|
||||
# BaseUrl only when it differs.
|
||||
#
|
||||
# WHY the share manifest is read through the mounted drive: GE-Enforce.ps1
|
||||
# mounts the SFLD share with SFLD credentials before invoking the engine, and
|
||||
# this script runs inside that window. SYSTEM has no standing access to the UNC
|
||||
# path, so the mounted drive is the only path that resolves. The drive letter is
|
||||
# not fixed, so it is derived from where this script is running rather than
|
||||
# hardcoded.
|
||||
|
||||
$ErrorActionPreference = 'Continue'
|
||||
|
||||
$TaskName = 'ShopDB GE-Enforce (shadow)'
|
||||
$InstallDir = 'C:\Program Files\GE\Shopfloor'
|
||||
$BaseUrl = 'https://tsgwp00525.wjs.geaerospace.net/shopdb'
|
||||
# Scope is NOT hardcoded: this script is shipped by more than one scope
|
||||
# (collections and nocollections today), and a wrong value here would shadow the
|
||||
# wrong manifest silently. It runs from <drive>:\<scope>\shopdb-client, so the
|
||||
# directory it sits under IS the scope name - the same derivation used below for
|
||||
# the share manifest, so the two cannot disagree.
|
||||
$Scope = Split-Path -Leaf (Split-Path -Parent $PSScriptRoot)
|
||||
|
||||
function Write-ShadowLog {
|
||||
# Write-Host ALONE is not enough: the engine records only "ps1: <path>" and
|
||||
# the exit code for a PS1 entry, so nothing this script says reaches the
|
||||
# enforce log. Combined with the fail-safe `exit 0` on every path, a silent
|
||||
# early-out was indistinguishable from success - which is exactly how the
|
||||
# never-firing task went unnoticed. Write to a file as well so the next
|
||||
# failure is answerable from disk.
|
||||
param([string]$Message)
|
||||
$line = "[{0}] [shadow-setup] {1}" -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message
|
||||
Write-Host $line
|
||||
try {
|
||||
$dir = 'C:\Logs\Shopfloor'
|
||||
if (-not (Test-Path $dir)) { New-Item -ItemType Directory -Path $dir -Force | Out-Null }
|
||||
Add-Content -LiteralPath (Join-Path $dir ('shadow-setup-{0}.log' -f (Get-Date -Format yyyyMMdd))) -Value $line
|
||||
} catch { }
|
||||
}
|
||||
|
||||
try {
|
||||
# --- BaseUrl (the client reads this; no token needed on an allowlisted subnet)
|
||||
$regPath = 'HKLM:\SOFTWARE\GE\ShopDB'
|
||||
if (-not (Test-Path $regPath)) { New-Item -Path $regPath -Force | Out-Null }
|
||||
$current = (Get-ItemProperty -Path $regPath -Name BaseUrl -ErrorAction SilentlyContinue).BaseUrl
|
||||
if ($current -ne $BaseUrl) {
|
||||
Set-ItemProperty -Path $regPath -Name BaseUrl -Value $BaseUrl
|
||||
Write-ShadowLog "BaseUrl set to $BaseUrl"
|
||||
}
|
||||
|
||||
$runner = Join-Path $InstallDir 'Invoke-ShopdbEnforce.ps1'
|
||||
$engine = Join-Path $InstallDir 'lib\Install-FromManifest.ps1'
|
||||
foreach ($p in @($runner, $engine)) {
|
||||
if (-not (Test-Path -LiteralPath $p)) {
|
||||
Write-ShadowLog "MISSING $p - the File entries have not landed yet; will retry next cycle."
|
||||
exit 0 # fail-safe: never break the bay, the entry re-runs
|
||||
}
|
||||
}
|
||||
|
||||
# --- the share manifest this scope is enforced from, via the mounted drive.
|
||||
# $PSScriptRoot is <drive>:\<scope>\shopdb-client, so its grandparent is the
|
||||
# scope dir. Deriving it keeps this correct whatever letter GE-Enforce mounted.
|
||||
$scopeDir = Split-Path -Parent $PSScriptRoot
|
||||
$shareManifest = Join-Path $scopeDir 'manifest.json'
|
||||
if (-not (Test-Path -LiteralPath $shareManifest)) {
|
||||
Write-ShadowLog "share manifest not found at $shareManifest - not registering."
|
||||
exit 0
|
||||
}
|
||||
|
||||
$arguments = '-NoProfile -ExecutionPolicy Bypass -File "{0}" -Scope "{1}" -EnginePath "{2}" -ShadowMode -ShareManifestPath "{3}"' `
|
||||
-f $runner, $Scope, $engine, $shareManifest
|
||||
|
||||
# ONLY register when it is missing or its arguments changed.
|
||||
#
|
||||
# Registering unconditionally is what stopped this working the first time.
|
||||
# A `-Once -At (Get-Date)` trigger does NOT fire immediately: the first run
|
||||
# is start-boundary PLUS the repetition interval, so 15 minutes out. This
|
||||
# entry is DetectionMethod=Always and runs every enforce cycle, 5 minutes
|
||||
# apart, and each Register-ScheduledTask -Force reset the start boundary to
|
||||
# "now" - pushing the first run back to +15 before the previous +15 could
|
||||
# elapse. 5 < 15, so the task sat at Ready with LastTaskResult 267011
|
||||
# (SCHED_S_TASK_HAS_NOT_RUN) indefinitely. Measured on the win11 VM.
|
||||
$existing = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
||||
if ($existing) {
|
||||
$currentArgs = ($existing.Actions | Select-Object -First 1).Arguments
|
||||
if ($currentArgs -eq $arguments) {
|
||||
# Correct arguments, but a bay provisioned by the BROKEN version of
|
||||
# this script carries a task that was reset every cycle and so has
|
||||
# never run. Leaving it alone would strand exactly the bays that hit
|
||||
# the bug. Kick it once; after that LastRunTime is set and this is a
|
||||
# no-op forever.
|
||||
$info = $existing | Get-ScheduledTaskInfo
|
||||
$neverran = ($null -eq $info.LastRunTime) -or
|
||||
($info.LastRunTime -lt (Get-Date '2000-01-01'))
|
||||
if ($neverran) {
|
||||
Write-ShadowLog ("task exists but has never run (LastTaskResult $($info.LastTaskResult)) - starting it once.")
|
||||
Start-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
||||
}
|
||||
return # schedule is correct; do not reset the start boundary
|
||||
}
|
||||
Write-ShadowLog 'task arguments changed - re-registering.'
|
||||
}
|
||||
|
||||
$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument $arguments
|
||||
# RepetitionInterval ALONE - passing RepetitionDuration serializes to a
|
||||
# Duration the Task Scheduler schema rejects.
|
||||
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Minutes 15)
|
||||
$principal = New-ScheduledTaskPrincipal -UserId 'NT AUTHORITY\SYSTEM' -RunLevel Highest
|
||||
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable
|
||||
|
||||
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger `
|
||||
-Principal $principal -Settings $settings -Force | Out-Null
|
||||
# Kick it once rather than waiting out the first 15-minute interval, so a
|
||||
# freshly provisioned bay reports on this cycle instead of the next.
|
||||
Start-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
||||
Write-ShadowLog "registered '$TaskName' (every 15 min), shadowing $shareManifest"
|
||||
exit 0
|
||||
}
|
||||
catch {
|
||||
# Fail-safe: a broken setup script must never stop the bay enforcing.
|
||||
Write-ShadowLog "FAILED: $_"
|
||||
exit 0
|
||||
}
|
||||
Reference in New Issue
Block a user