Verified MCL uploads: hash every file, repair over HTTP, gate the import
Small Control JSONs have arrived empty or wrong more than once, and the old check compared sizes only and skipped missing files. A copy of Deploy/Control on the dev PC still holds two 0-byte JSONs. Upload-Image.ps1 (tested on Windows PowerShell 5.1 in the win11 VM): - refuses an unfinished MCL cache: exactly one <OS>-Production.tag (MCL writes it last), no Control file newer than it, selected OS matches, every Control/Tools JSON non-empty and valid - checks payload against the SHA-256 MCL's own Control JSONs publish. MCL resumes downloads onto old files (FileMode.Append) and then trusts any file of the right size: the September junk-tail corruption. Offers to delete the bad cache files so MCL downloads them again - hashes every file into a manifest, robocopies, POSTs the manifest - re-sends bad small files over HTTP (no proxy), large ones by deleting the bad copy and robocopy /J (robocopy skips a same-size, same-time file as "modified" even with /IS /IT) Webapp: - services/upload_verify.py hashes the upload on the server's own disk in a background thread; reports missing, differing, bad-in-MCL-cache and stray files; PUT /api/upload/file writes temp, checks hash, renames - import of the upload folder only after a passed verification, with every file unchanged (size, mtime, ctime, inode), no stray files, and the target the OS allows (LTSC: Shop Floor only; else Standard or Engineer; unknown OS: none). Import, verify and PUT share one lock - paths matched case-insensitively against the disk, as Samba does - import page shows the upload, MCL run, verification and stray files with a remove button; record of each import kept on the image Reviewed adversarially; all 11 confirmed findings fixed with tests. 42 new unit tests. VM test: clean upload; same-size same-mtime damage on 3 files repaired in one round; stray file blocks import until removed; Shop Floor refused for a Pro upload; import records and clears the folder; missing tag, 0-byte JSON, junk-head WIM and late Control file each refused with nothing uploaded.
This commit is contained in:
cproudlock
@@ -1,8 +1,25 @@
|
|||||||
#
|
#
|
||||||
# Upload-Image.ps1 — Copy MCL cached image to the PXE server
|
# Upload-Image.ps1 - Copy the Media Creator Lite cache to the PXE server,
|
||||||
|
# then PROVE every file arrived intact.
|
||||||
#
|
#
|
||||||
# Reads user_selections.json to upload only the selected OS, matching
|
# 1. Checks the MCL cache before sending anything:
|
||||||
# packages, and config files. Drivers are EXCLUDED by default.
|
# - exactly one <OS>-Production.tag at the cache root (MCL writes it only
|
||||||
|
# when a run finishes; none = an unfinished or interrupted cache)
|
||||||
|
# - no Control file newer than that tag (MCL still writing / crashed)
|
||||||
|
# - the OS in OperatingSystem.json is the one in user_selections.json
|
||||||
|
# - every Control / Tools JSON is non-empty and parses
|
||||||
|
# - every WIM, update and driver pack hashes to the SHA-256 MCL's own
|
||||||
|
# Control JSONs publish (MCL resumes downloads onto old files and then
|
||||||
|
# trusts any file of the right size, so a cache can hold junk)
|
||||||
|
# 2. Hashes every file it will send into upload-manifest.json.
|
||||||
|
# 3. Copies with robocopy as before (SMB share \\<server>\image-upload).
|
||||||
|
# 4. POSTs the manifest to the PXE webapp, which hashes the files on the
|
||||||
|
# server's own disk (no Windows SMB cache in the way) and lists any file
|
||||||
|
# that is missing or differs.
|
||||||
|
# 5. Re-sends those files over HTTP (small files; the server writes a temp
|
||||||
|
# file, checks the hash, then renames it) or robocopy /J /IS (large
|
||||||
|
# files), and verifies again, up to -MaxRounds times.
|
||||||
|
# The webapp's Image Import only accepts an upload that passed step 4.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# .\Upload-Image.ps1 (selected OS + packages, no drivers)
|
# .\Upload-Image.ps1 (selected OS + packages, no drivers)
|
||||||
@@ -10,19 +27,23 @@
|
|||||||
# .\Upload-Image.ps1 -CachePath "D:\MCL\Cache" (custom cache location)
|
# .\Upload-Image.ps1 -CachePath "D:\MCL\Cache" (custom cache location)
|
||||||
# .\Upload-Image.ps1 -Server 172.16.9.1 (custom server IP)
|
# .\Upload-Image.ps1 -Server 172.16.9.1 (custom server IP)
|
||||||
#
|
#
|
||||||
# After upload, use the PXE webapp (http://172.16.9.1:9009) to import
|
# Exit codes: 0 verified, 1 failed (details printed), 2 MCL cache not usable.
|
||||||
# the uploaded content into the desired image type.
|
|
||||||
#
|
#
|
||||||
|
|
||||||
param(
|
param(
|
||||||
[string]$CachePath = "C:\ProgramData\GEAerospace\MediaCreator\Cache",
|
[string]$CachePath = "C:\ProgramData\GEAerospace\MediaCreator\Cache",
|
||||||
[string]$Server = "172.16.9.1",
|
[string]$Server = "172.16.9.1",
|
||||||
|
[int]$WebPort = 9009,
|
||||||
[string]$User = "pxe-upload",
|
[string]$User = "pxe-upload",
|
||||||
[string]$Pass = "pxe",
|
[string]$Pass = "pxe",
|
||||||
|
[int]$MaxRounds = 3,
|
||||||
[switch]$IncludeDrivers
|
[switch]$IncludeDrivers
|
||||||
)
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
$Share = "\\$Server\image-upload"
|
$Share = "\\$Server\image-upload"
|
||||||
|
$ApiBase = "http://${Server}:$WebPort"
|
||||||
|
$ExcludedDeployDirs = @("Operating Systems", "Out-of-box Drivers", "Packages")
|
||||||
|
|
||||||
function Format-Size {
|
function Format-Size {
|
||||||
param([long]$Bytes)
|
param([long]$Bytes)
|
||||||
@@ -33,7 +54,79 @@ function Format-Size {
|
|||||||
|
|
||||||
function Resolve-DestDir {
|
function Resolve-DestDir {
|
||||||
param([string]$Dir)
|
param([string]$Dir)
|
||||||
return ($Dir -replace '^\*destinationdir\*\\?', '')
|
$d = $Dir -replace '^\*destinationdir\*\\?', ''
|
||||||
|
return (($d -replace '^[A-Za-z]:\\', '').Trim('\'))
|
||||||
|
}
|
||||||
|
|
||||||
|
function Write-Step {
|
||||||
|
param([string]$Text)
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "========================================" -ForegroundColor Cyan
|
||||||
|
Write-Host $Text -ForegroundColor Cyan
|
||||||
|
Write-Host "========================================" -ForegroundColor Cyan
|
||||||
|
}
|
||||||
|
|
||||||
|
function Stop-Cache {
|
||||||
|
# The cache itself is unusable: nothing was sent.
|
||||||
|
param([string[]]$Lines)
|
||||||
|
Write-Host ""
|
||||||
|
foreach ($l in $Lines) { Write-Host "ERROR: $l" -ForegroundColor Red }
|
||||||
|
Write-Host "Nothing was uploaded." -ForegroundColor Red
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
function Read-JsonFile {
|
||||||
|
param([string]$Path)
|
||||||
|
return (Get-Content -LiteralPath $Path -Raw -Encoding UTF8 | ConvertFrom-Json)
|
||||||
|
}
|
||||||
|
|
||||||
|
# HTTP to the webapp with HttpWebRequest and NO proxy: the work PC's system
|
||||||
|
# proxy (Zscaler) must not see traffic to the isolated imaging LAN.
|
||||||
|
# Returns @{ Status = <int>; Body = <string> }; never throws on HTTP errors.
|
||||||
|
function Invoke-Pxe {
|
||||||
|
param([string]$Method, [string]$Path, [byte[]]$Body, [string]$InFile,
|
||||||
|
[string]$ContentType = "application/json", [int]$TimeoutSec = 120)
|
||||||
|
$req = [System.Net.HttpWebRequest]::Create("$ApiBase$Path")
|
||||||
|
$req.Method = $Method
|
||||||
|
$req.Proxy = $null
|
||||||
|
$req.Timeout = $TimeoutSec * 1000
|
||||||
|
$req.ReadWriteTimeout = $TimeoutSec * 1000
|
||||||
|
$req.KeepAlive = $false
|
||||||
|
if ($Body -or $InFile) {
|
||||||
|
$req.ContentType = $ContentType
|
||||||
|
if ($InFile) { $req.ContentLength = (Get-Item -LiteralPath $InFile).Length }
|
||||||
|
else { $req.ContentLength = $Body.Length }
|
||||||
|
try { $rs = $req.GetRequestStream() } catch { return @{ Status = 0; Body = $_.Exception.Message } }
|
||||||
|
try {
|
||||||
|
if ($InFile) {
|
||||||
|
$fs = [System.IO.File]::OpenRead($InFile)
|
||||||
|
try { $fs.CopyTo($rs) } finally { $fs.Dispose() }
|
||||||
|
} else {
|
||||||
|
$rs.Write($Body, 0, $Body.Length)
|
||||||
|
}
|
||||||
|
} finally { $rs.Dispose() }
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$resp = $req.GetResponse()
|
||||||
|
} catch {
|
||||||
|
# A .NET method call wraps the WebException in a
|
||||||
|
# MethodInvocationException: look at the inner one.
|
||||||
|
$ex = $_.Exception
|
||||||
|
while ($ex -and -not ($ex -is [System.Net.WebException])) { $ex = $ex.InnerException }
|
||||||
|
if (-not $ex -or -not $ex.Response) { return @{ Status = 0; Body = $_.Exception.Message } }
|
||||||
|
$resp = $ex.Response
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
$sr = New-Object System.IO.StreamReader($resp.GetResponseStream())
|
||||||
|
$text = $sr.ReadToEnd()
|
||||||
|
$sr.Dispose()
|
||||||
|
return @{ Status = [int]$resp.StatusCode; Body = $text }
|
||||||
|
} finally { $resp.Close() }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ErrorText {
|
||||||
|
param($Resp)
|
||||||
|
try { return ($Resp.Body | ConvertFrom-Json).error } catch { return $Resp.Body }
|
||||||
}
|
}
|
||||||
|
|
||||||
Write-Host ""
|
Write-Host ""
|
||||||
@@ -43,17 +136,60 @@ Write-Host "========================================" -ForegroundColor Cyan
|
|||||||
Write-Host ""
|
Write-Host ""
|
||||||
|
|
||||||
# --- Validate source paths ---
|
# --- Validate source paths ---
|
||||||
|
$CachePath = $CachePath.TrimEnd('\')
|
||||||
$DeployPath = Join-Path $CachePath "Deploy"
|
$DeployPath = Join-Path $CachePath "Deploy"
|
||||||
$ToolsPath = Join-Path (Split-Path $CachePath -Parent) "Tools"
|
$ToolsPath = Join-Path (Split-Path $CachePath -Parent) "Tools"
|
||||||
if (-not (Test-Path $ToolsPath -PathType Container)) {
|
if (-not (Test-Path $ToolsPath -PathType Container)) {
|
||||||
$ToolsPath = "C:\ProgramData\GEAerospace\MediaCreator\Tools"
|
$ToolsPath = "C:\ProgramData\GEAerospace\MediaCreator\Tools"
|
||||||
}
|
}
|
||||||
$SourcesZip = Join-Path $CachePath "Boot\Sources.zip"
|
$SourcesZip = Join-Path $CachePath "Boot\Sources.zip"
|
||||||
|
$ControlPath = Join-Path $DeployPath "Control"
|
||||||
|
|
||||||
if (-not (Test-Path $DeployPath -PathType Container)) {
|
if (-not (Test-Path $DeployPath -PathType Container)) {
|
||||||
Write-Host "ERROR: Deploy directory not found at $DeployPath" -ForegroundColor Red
|
Write-Host "ERROR: Deploy directory not found at $DeployPath" -ForegroundColor Red
|
||||||
Write-Host " .\Upload-Image.ps1 -CachePath ""D:\Path\To\Cache""" -ForegroundColor Yellow
|
Write-Host " .\Upload-Image.ps1 -CachePath ""D:\Path\To\Cache""" -ForegroundColor Yellow
|
||||||
exit 1
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- MCL finished? One tag at the cache root, written at the very end ---
|
||||||
|
$tags = @(Get-ChildItem -LiteralPath $CachePath -Filter "*.tag" -File -ErrorAction SilentlyContinue)
|
||||||
|
if ($tags.Count -eq 0) {
|
||||||
|
Stop-Cache @("No MCL completion tag (*.tag) in $CachePath.",
|
||||||
|
"MCL writes it only when a run finishes: the cache is incomplete or the run was interrupted.",
|
||||||
|
"Run Media Creator Lite again and let it finish.")
|
||||||
|
}
|
||||||
|
if ($tags.Count -gt 1) {
|
||||||
|
Stop-Cache (@("More than one MCL tag in ${CachePath}:") + ($tags | ForEach-Object { " $($_.Name)" }) +
|
||||||
|
@("MCL leaves exactly one. Run Media Creator Lite again."))
|
||||||
|
}
|
||||||
|
$tag = $tags[0]
|
||||||
|
$tagBytes = [System.IO.File]::ReadAllBytes($tag.FullName)
|
||||||
|
if ($tagBytes.Length -lt 8) { Stop-Cache @("MCL tag $($tag.Name) is damaged ($($tagBytes.Length) bytes). Run MCL again.") }
|
||||||
|
# MCL writes DateTime.Now.Ticks (local time) into the tag (TagFactory).
|
||||||
|
try { $tagTime = New-Object DateTime ([BitConverter]::ToInt64($tagBytes, 0)) }
|
||||||
|
catch { Stop-Cache @("MCL tag $($tag.Name) holds no valid time. Run MCL again.") }
|
||||||
|
$tagOs = $tag.BaseName -replace '-(Production|QA)$', ''
|
||||||
|
|
||||||
|
$late = @(Get-ChildItem -LiteralPath $ControlPath -File -ErrorAction SilentlyContinue |
|
||||||
|
Where-Object { $_.LastWriteTime -gt $tagTime.AddSeconds(5) })
|
||||||
|
if ($late.Count -gt 0) {
|
||||||
|
Stop-Cache (@("These Control files are newer than the MCL finish tag ($tagTime):") +
|
||||||
|
($late | ForEach-Object { " $($_.Name) $($_.LastWriteTime)" }) +
|
||||||
|
@("MCL was still writing or crashed. Run Media Creator Lite again and let it finish."))
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Every Control / Tools JSON must be non-empty and parse ---
|
||||||
|
$badJson = @()
|
||||||
|
foreach ($dir in @($ControlPath, $ToolsPath)) {
|
||||||
|
if (-not (Test-Path $dir -PathType Container)) { continue }
|
||||||
|
foreach ($j in Get-ChildItem -LiteralPath $dir -Filter "*.json" -File -Recurse) {
|
||||||
|
if ($j.Length -eq 0) { $badJson += "$($j.FullName) (0 bytes)"; continue }
|
||||||
|
try { $null = Read-JsonFile $j.FullName } catch { $badJson += "$($j.FullName) (not valid JSON)" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ($badJson.Count -gt 0) {
|
||||||
|
Stop-Cache (@("Broken JSON in the MCL cache:") + ($badJson | ForEach-Object { " $_" }) +
|
||||||
|
@("Delete the cache folder ($CachePath) and run Media Creator Lite again."))
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- Parse user_selections.json ---
|
# --- Parse user_selections.json ---
|
||||||
@@ -61,49 +197,60 @@ $SelectionsFile = Join-Path $ToolsPath "user_selections.json"
|
|||||||
if (-not (Test-Path $SelectionsFile)) {
|
if (-not (Test-Path $SelectionsFile)) {
|
||||||
Write-Host "ERROR: user_selections.json not found at $SelectionsFile" -ForegroundColor Red
|
Write-Host "ERROR: user_selections.json not found at $SelectionsFile" -ForegroundColor Red
|
||||||
Write-Host " Run Media Creator Lite first to create a configuration." -ForegroundColor Yellow
|
Write-Host " Run Media Creator Lite first to create a configuration." -ForegroundColor Yellow
|
||||||
exit 1
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
$selections = (Get-Content $SelectionsFile -Raw | ConvertFrom-Json)[0]
|
$selections = (Read-JsonFile $SelectionsFile)[0]
|
||||||
$selectedOsId = $selections.OperatingSystemSelection
|
$selectedOsId = $selections.OperatingSystemSelection
|
||||||
$selectedModelIds = @($selections.HardwareModelSelection | ForEach-Object { $_.Id } | Select-Object -Unique)
|
$selectedModelIds = @($selections.HardwareModelSelection | ForEach-Object { $_.Id } | Select-Object -Unique)
|
||||||
|
|
||||||
# --- Parse control JSONs ---
|
# --- Parse control JSONs ---
|
||||||
$ControlPath = Join-Path $DeployPath "Control"
|
|
||||||
|
|
||||||
$osJsonFile = Join-Path $ControlPath "OperatingSystem.json"
|
$osJsonFile = Join-Path $ControlPath "OperatingSystem.json"
|
||||||
$driverJsonFile = Join-Path $ControlPath "HardwareDriver.json"
|
$driverJsonFile = Join-Path $ControlPath "HardwareDriver.json"
|
||||||
$pkgJsonFile = Join-Path $ControlPath "packages.json"
|
$pkgJsonFile = Join-Path $ControlPath "packages.json"
|
||||||
|
$langJsonFile = Join-Path $ControlPath "LanguagePacks.json"
|
||||||
|
|
||||||
if (-not (Test-Path $osJsonFile)) {
|
if (-not (Test-Path $osJsonFile)) {
|
||||||
Write-Host "ERROR: OperatingSystem.json not found in $ControlPath" -ForegroundColor Red
|
Write-Host "ERROR: OperatingSystem.json not found in $ControlPath" -ForegroundColor Red
|
||||||
exit 1
|
exit 2
|
||||||
}
|
}
|
||||||
|
|
||||||
$osJson = Get-Content $osJsonFile -Raw | ConvertFrom-Json
|
$osJson = @(Read-JsonFile $osJsonFile)
|
||||||
$driverJson = if (Test-Path $driverJsonFile) { Get-Content $driverJsonFile -Raw | ConvertFrom-Json } else { @() }
|
$driverJson = if (Test-Path $driverJsonFile) { @(Read-JsonFile $driverJsonFile) } else { @() }
|
||||||
$pkgJson = if (Test-Path $pkgJsonFile) { Get-Content $pkgJsonFile -Raw | ConvertFrom-Json } else { @() }
|
$pkgJson = if (Test-Path $pkgJsonFile) { @(Read-JsonFile $pkgJsonFile) } else { @() }
|
||||||
|
$langJson = if (Test-Path $langJsonFile) { @(Read-JsonFile $langJsonFile) } else { @() }
|
||||||
|
|
||||||
|
# --- The cached OS must be the selected one, and the one the tag names ---
|
||||||
|
$matchedOs = @($osJson | Where-Object { $_.operatingSystemVersion.id -eq [int]$selectedOsId })
|
||||||
|
if ($matchedOs.Count -eq 0) {
|
||||||
|
Stop-Cache @("OperatingSystem.json does not hold the OS selected in user_selections.json (ID $selectedOsId).",
|
||||||
|
"Run Media Creator Lite again.")
|
||||||
|
}
|
||||||
|
$osv = $matchedOs[0].operatingSystemVersion
|
||||||
|
if (-not $tagOs.StartsWith($osv.productName, [StringComparison]::OrdinalIgnoreCase)) {
|
||||||
|
Stop-Cache @("The MCL tag says '$tagOs' but OperatingSystem.json holds '$($osv.productName)'.",
|
||||||
|
"The cache is a mix of two runs. Run Media Creator Lite again.")
|
||||||
|
}
|
||||||
|
$isLtsc = $osv.productName -match 'LTSC'
|
||||||
|
$suggested = if ($isLtsc) { "gea-shopfloor" } else { "standard-or-engineer" }
|
||||||
|
|
||||||
# --- Resolve selections to paths ---
|
# --- Resolve selections to paths ---
|
||||||
|
|
||||||
# OS: match OperatingSystemSelection ID to OperatingSystem.json entries
|
|
||||||
$matchedOs = @($osJson | Where-Object { $_.operatingSystemVersion.id -eq [int]$selectedOsId })
|
|
||||||
$osDirs = @()
|
$osDirs = @()
|
||||||
$osTotalSize = [long]0
|
$osTotalSize = [long]0
|
||||||
foreach ($os in $matchedOs) {
|
foreach ($os in $matchedOs) {
|
||||||
$rel = Resolve-DestDir $os.operatingSystemVersion.wim.DestinationDir
|
$osDirs += Resolve-DestDir $os.operatingSystemVersion.wim.DestinationDir
|
||||||
$osDirs += $rel
|
|
||||||
$osTotalSize += $os.operatingSystemVersion.wim.size
|
$osTotalSize += $os.operatingSystemVersion.wim.size
|
||||||
}
|
}
|
||||||
|
|
||||||
# Packages: enabled + matching OS ID
|
# Packages: enabled + matching OS ID (osId is a comma string in the new schema)
|
||||||
$matchedPkgs = @($pkgJson | Where-Object { $_.aOsIds -contains $selectedOsId -and $_.enabled -eq 1 })
|
function Get-OsIds { param($e) if ($e.aOsIds) { return @($e.aOsIds) } return @("$($e.osId)" -split ',' | ForEach-Object { $_.Trim() }) }
|
||||||
|
$matchedPkgs = @($pkgJson | Where-Object { (Get-OsIds $_) -contains "$selectedOsId" -and $_.enabled -eq 1 })
|
||||||
$pkgTotalSize = [long]0
|
$pkgTotalSize = [long]0
|
||||||
foreach ($pkg in $matchedPkgs) { $pkgTotalSize += $pkg.size }
|
foreach ($pkg in $matchedPkgs) { $pkgTotalSize += $pkg.size }
|
||||||
|
|
||||||
# Drivers: match selected model IDs (family) + OS ID, deduplicate by path
|
# Drivers: match selected model IDs (family) + OS ID, deduplicate by path
|
||||||
$allMatchingDrivers = @($driverJson | Where-Object {
|
$allMatchingDrivers = @($driverJson | Where-Object {
|
||||||
$selectedModelIds -contains $_.family -and $_.aOsIds -contains $selectedOsId
|
$selectedModelIds -contains $_.family -and (Get-OsIds $_) -contains "$selectedOsId"
|
||||||
})
|
})
|
||||||
$allDriverDirSet = [ordered]@{}
|
$allDriverDirSet = [ordered]@{}
|
||||||
foreach ($drv in $allMatchingDrivers) {
|
foreach ($drv in $allMatchingDrivers) {
|
||||||
@@ -123,122 +270,232 @@ if ($IncludeDrivers) {
|
|||||||
|
|
||||||
# --- Display upload plan ---
|
# --- Display upload plan ---
|
||||||
Write-Host " Cache: $CachePath"
|
Write-Host " Cache: $CachePath"
|
||||||
|
Write-Host " MCL run: $tagOs, finished $tagTime" -ForegroundColor Green
|
||||||
Write-Host " Server: $Server"
|
Write-Host " Server: $Server"
|
||||||
|
Write-Host " Target: $(if ($isLtsc) { 'Shop Floor (LTSC)' } else { 'Standard or Engineer (pick it in the webapp)' })"
|
||||||
Write-Host ""
|
Write-Host ""
|
||||||
Write-Host " Upload Plan (from user_selections.json):" -ForegroundColor Cyan
|
Write-Host " Upload Plan (from user_selections.json):" -ForegroundColor Cyan
|
||||||
Write-Host " ------------------------------------------"
|
Write-Host " ------------------------------------------"
|
||||||
|
Write-Host " OS: $($osv.marketingName) $($osv.productName) ($(Format-Size $osTotalSize))" -ForegroundColor Green
|
||||||
if ($matchedOs.Count -gt 0) {
|
|
||||||
$osName = $matchedOs[0].operatingSystemVersion.marketingName
|
|
||||||
Write-Host " OS: $osName ($(Format-Size $osTotalSize))" -ForegroundColor Green
|
|
||||||
} else {
|
|
||||||
Write-Host " OS: No match for selection ID $selectedOsId" -ForegroundColor Red
|
|
||||||
}
|
|
||||||
|
|
||||||
Write-Host " Packages: $($matchedPkgs.Count) update(s) ($(Format-Size $pkgTotalSize))" -ForegroundColor Green
|
Write-Host " Packages: $($matchedPkgs.Count) update(s) ($(Format-Size $pkgTotalSize))" -ForegroundColor Green
|
||||||
|
|
||||||
if ($IncludeDrivers) {
|
if ($IncludeDrivers) {
|
||||||
Write-Host " Drivers: $($driverDirs.Count) model(s) ($(Format-Size $driverTotalSize))" -ForegroundColor Green
|
Write-Host " Drivers: $($driverDirs.Count) model(s) ($(Format-Size $driverTotalSize))" -ForegroundColor Green
|
||||||
} else {
|
} else {
|
||||||
Write-Host " Drivers: SKIPPED -- $allDriverCount available, use -IncludeDrivers" -ForegroundColor Yellow
|
Write-Host " Drivers: SKIPPED -- $allDriverCount available, use -IncludeDrivers" -ForegroundColor Yellow
|
||||||
}
|
}
|
||||||
|
|
||||||
Write-Host " Control: Always included" -ForegroundColor Gray
|
Write-Host " Control: Always included" -ForegroundColor Gray
|
||||||
Write-Host " Tools: $(if (Test-Path $ToolsPath) { 'Yes' } else { 'Not found' })" -ForegroundColor $(if (Test-Path $ToolsPath) { "Gray" } else { "Yellow" })
|
Write-Host " Tools: $(if (Test-Path $ToolsPath) { 'Yes' } else { 'Not found' })" -ForegroundColor $(if (Test-Path $ToolsPath) { "Gray" } else { "Yellow" })
|
||||||
Write-Host " Sources: $(if (Test-Path $SourcesZip) { 'Yes (from Boot\Sources.zip)' } else { 'Not found' })" -ForegroundColor $(if (Test-Path $SourcesZip) { "Gray" } else { "Yellow" })
|
Write-Host " Sources: $(if (Test-Path $SourcesZip) { 'Yes (from Boot\Sources.zip)' } else { 'Not found' })" -ForegroundColor $(if (Test-Path $SourcesZip) { "Gray" } else { "Yellow" })
|
||||||
Write-Host ""
|
Write-Host ""
|
||||||
|
|
||||||
# --- Connect to SMB share ---
|
# --- Sources: extract now so it can be hashed with everything else ---
|
||||||
Write-Host "Connecting to $Share ..." -ForegroundColor Gray
|
$TempExtract = $null
|
||||||
net use $Share /delete 2>$null | Out-Null
|
$TempSources = $null
|
||||||
|
if (Test-Path $SourcesZip) {
|
||||||
|
$TempExtract = Join-Path $env:TEMP "SourcesExtract"
|
||||||
|
Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue
|
||||||
|
Expand-Archive $SourcesZip -DestinationPath $TempExtract -Force
|
||||||
|
# Handle nested Sources folder (zip may contain Sources/ at root)
|
||||||
|
$TempSources = $TempExtract
|
||||||
|
if ((Test-Path (Join-Path $TempExtract "Sources")) -and -not (Test-Path (Join-Path $TempExtract "Diskpart"))) {
|
||||||
|
$TempSources = Join-Path $TempExtract "Sources"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$netResult = net use $Share /user:$User $Pass 2>&1
|
# --- Everything that will be sent: local file + path on the share ---
|
||||||
|
$sendList = New-Object System.Collections.Generic.List[object]
|
||||||
|
function Add-Tree {
|
||||||
|
param([string]$LocalDir, [string]$RelPrefix, [string[]]$SkipDirs = @())
|
||||||
|
if (-not (Test-Path -LiteralPath $LocalDir -PathType Container)) { return }
|
||||||
|
$base = (Get-Item -LiteralPath $LocalDir).FullName.TrimEnd('\')
|
||||||
|
foreach ($f in Get-ChildItem -LiteralPath $base -Recurse -File -Force) {
|
||||||
|
$rel = $f.FullName.Substring($base.Length + 1)
|
||||||
|
if ($SkipDirs.Count -gt 0) {
|
||||||
|
# robocopy /XD <name> skips that folder name at any depth
|
||||||
|
$segs = $rel.Split('\')
|
||||||
|
$skip = $false
|
||||||
|
for ($i = 0; $i -lt $segs.Count - 1; $i++) { if ($SkipDirs -contains $segs[$i]) { $skip = $true; break } }
|
||||||
|
if ($skip) { continue }
|
||||||
|
}
|
||||||
|
$sendList.Add([pscustomobject]@{ Local = $f.FullName; Rel = (Join-Path $RelPrefix $rel); Size = $f.Length })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Add-Tree $DeployPath "Deploy" $ExcludedDeployDirs
|
||||||
|
foreach ($d in $osDirs) { Add-Tree (Join-Path $CachePath $d) $d }
|
||||||
|
foreach ($pkg in $matchedPkgs) {
|
||||||
|
$pdir = Resolve-DestDir $pkg.destinationDir
|
||||||
|
$rel = if ($pdir) { Join-Path $pdir $pkg.fileName } else { $pkg.fileName }
|
||||||
|
$local = Join-Path $CachePath $rel
|
||||||
|
if (Test-Path -LiteralPath $local -PathType Leaf) {
|
||||||
|
# File name as robocopy will write it (disk case), not as the JSON spells it
|
||||||
|
$item = Get-Item -LiteralPath $local
|
||||||
|
$rel = Join-Path (Split-Path $rel -Parent) $item.Name
|
||||||
|
$sendList.Add([pscustomobject]@{ Local = $item.FullName; Rel = $rel; Size = $item.Length })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
foreach ($d in $driverDirs) { Add-Tree (Join-Path $CachePath $d) $d }
|
||||||
|
Add-Tree $ToolsPath "Tools"
|
||||||
|
if ($TempSources) { Add-Tree $TempSources "Sources" }
|
||||||
|
|
||||||
|
# The same file can be reached twice (an OS folder outside the excluded
|
||||||
|
# Deploy dirs, nested driver folders, two packages naming one file); a
|
||||||
|
# duplicate path would make the server refuse the whole manifest.
|
||||||
|
$seenRel = @{}
|
||||||
|
$dedup = New-Object System.Collections.Generic.List[object]
|
||||||
|
foreach ($f in $sendList) {
|
||||||
|
$k = ($f.Rel -replace '\\', '/').ToLower()
|
||||||
|
if (-not $seenRel.ContainsKey($k)) { $seenRel[$k] = $true; $dedup.Add($f) }
|
||||||
|
}
|
||||||
|
$sendList = $dedup
|
||||||
|
|
||||||
|
# --- Hash everything; check payload against MCL's published hashes ---
|
||||||
|
Write-Step "Hashing $($sendList.Count) file(s) ($(Format-Size (($sendList | Measure-Object Size -Sum).Sum)))"
|
||||||
|
$mclHash = @{}
|
||||||
|
function Add-Mcl { param($Dest, $Name, $Hash)
|
||||||
|
# Only a real SHA-256 (as the server checks): anything else is not ours to judge
|
||||||
|
if (-not ($Dest -and $Name -and ("$Hash" -match '^[0-9a-fA-F]{64}$'))) { return }
|
||||||
|
$dd = Resolve-DestDir $Dest
|
||||||
|
$key = if ($dd) { Join-Path $dd $Name } else { "$Name" }
|
||||||
|
$mclHash[$key.ToLower()] = "$Hash".ToLower() }
|
||||||
|
foreach ($e in $osJson) {
|
||||||
|
$w = $e.operatingSystemVersion.wim
|
||||||
|
$n = if ($w.fileName) { $w.fileName } else { "install.wim" }
|
||||||
|
Add-Mcl $w.destinationDir $n $w.hash
|
||||||
|
}
|
||||||
|
foreach ($e in @($driverJson) + @($pkgJson)) { if ($e) { Add-Mcl $e.destinationDir $e.fileName $e.hash } }
|
||||||
|
foreach ($e in $langJson) { foreach ($lf in @($e.files)) { if ($lf) { Add-Mcl $lf.destFolder $lf.fileName $lf.hash } } }
|
||||||
|
|
||||||
|
$totalBytes = [long](($sendList | Measure-Object Size -Sum).Sum)
|
||||||
|
$doneBytes = [long]0
|
||||||
|
$badCache = @()
|
||||||
|
$i = 0
|
||||||
|
foreach ($f in $sendList) {
|
||||||
|
$i++
|
||||||
|
$pct = if ($totalBytes -gt 0) { [int](100 * $doneBytes / $totalBytes) } else { 100 }
|
||||||
|
Write-Progress -Activity "Hashing" -Status "$i/$($sendList.Count) $($f.Rel)" -PercentComplete $pct
|
||||||
|
$f | Add-Member -NotePropertyName Sha256 -NotePropertyValue (Get-FileHash -LiteralPath $f.Local -Algorithm SHA256).Hash.ToLower()
|
||||||
|
$doneBytes += $f.Size
|
||||||
|
$want = $mclHash[$f.Rel.ToLower()]
|
||||||
|
if ($want -and $want -ne $f.Sha256) { $badCache += $f }
|
||||||
|
}
|
||||||
|
Write-Progress -Activity "Hashing" -Completed
|
||||||
|
if ($badCache.Count -gt 0) {
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "These cache files do not match the hash MCL publishes for them (damaged download):" -ForegroundColor Red
|
||||||
|
foreach ($b in $badCache) { Write-Host " $($b.Rel)" -ForegroundColor Red }
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "MCL re-downloads a MISSING file correctly, but trusts a damaged one of the right size." -ForegroundColor Yellow
|
||||||
|
$ans = Read-Host "Delete these $($badCache.Count) file(s) from the MCL cache so MCL downloads them again? (Y/N)"
|
||||||
|
if ($ans -match '^[Yy]') {
|
||||||
|
foreach ($b in $badCache) { Remove-Item -LiteralPath $b.Local -Force; Write-Host " deleted $($b.Local)" -ForegroundColor Gray }
|
||||||
|
Write-Host "Now run Media Creator Lite again, then this script." -ForegroundColor Yellow
|
||||||
|
}
|
||||||
|
if ($TempExtract) { Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue }
|
||||||
|
Write-Host "Nothing was uploaded." -ForegroundColor Red
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
Write-Host " All $($sendList.Count) file(s) hashed; payload matches MCL's published hashes." -ForegroundColor Green
|
||||||
|
|
||||||
|
$pcName = ($env:COMPUTERNAME -replace '[^A-Za-z0-9._-]', '')
|
||||||
|
$uploadId = "{0}-{1}" -f (Get-Date -Format "yyyyMMdd-HHmmss"), $pcName
|
||||||
|
if ($uploadId.Length -gt 80) { $uploadId = $uploadId.Substring(0, 80) }
|
||||||
|
$manifest = [ordered]@{
|
||||||
|
manifest_version = 1
|
||||||
|
upload_id = $uploadId
|
||||||
|
created = (Get-Date -Format "yyyy-MM-ddTHH:mm:sszzz")
|
||||||
|
source_pc = $pcName
|
||||||
|
mcl = [ordered]@{
|
||||||
|
tag = $tag.Name
|
||||||
|
finished = $tagTime.ToString("yyyy-MM-ddTHH:mm:ss")
|
||||||
|
os_id = [int]$osv.id
|
||||||
|
os_name = "$($osv.productName)"
|
||||||
|
os_version = "$($osv.versionNumber)"
|
||||||
|
build = "$($osv.buildNumber)"
|
||||||
|
suggested_target = $suggested
|
||||||
|
}
|
||||||
|
files = @($sendList | ForEach-Object { [ordered]@{ path = ($_.Rel -replace '\\', '/'); size = [long]$_.Size; sha256 = $_.Sha256 } })
|
||||||
|
}
|
||||||
|
$manifestBytes = [System.Text.Encoding]::UTF8.GetBytes(($manifest | ConvertTo-Json -Depth 6 -Compress))
|
||||||
|
|
||||||
|
# --- Is the webapp reachable? (before spending time on the copy) ---
|
||||||
|
$ping = Invoke-Pxe -Method GET -Path "/api/upload/verify" -TimeoutSec 15
|
||||||
|
if ($ping.Status -ne 200) {
|
||||||
|
Write-Host "ERROR: the PXE webapp at $ApiBase does not answer ($($ping.Status) $($ping.Body))." -ForegroundColor Red
|
||||||
|
Write-Host " The upload cannot be verified without it. Check the server, then run this again." -ForegroundColor Yellow
|
||||||
|
if ($TempExtract) { Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue }
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- Connect to SMB share ---
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "Connecting to $Share ..." -ForegroundColor Gray
|
||||||
|
# net use goes through cmd: in Windows PowerShell 5.1 a native command's
|
||||||
|
# redirected stderr becomes an error record, which ErrorAction Stop turns
|
||||||
|
# into a crash ("The network connection could not be found").
|
||||||
|
cmd /c "net use ""$Share"" /delete /y >nul 2>&1" | Out-Null
|
||||||
|
|
||||||
|
$netResult = cmd /c "net use ""$Share"" /user:$User ""$Pass"" 2>&1"
|
||||||
if ($LASTEXITCODE -ne 0) {
|
if ($LASTEXITCODE -ne 0) {
|
||||||
Write-Host "ERROR: Could not connect to $Share" -ForegroundColor Red
|
Write-Host "ERROR: Could not connect to $Share" -ForegroundColor Red
|
||||||
Write-Host $netResult -ForegroundColor Red
|
Write-Host $netResult -ForegroundColor Red
|
||||||
Write-Host ""
|
Write-Host ""
|
||||||
Write-Host "Make sure:" -ForegroundColor Yellow
|
Write-Host "Make sure:" -ForegroundColor Yellow
|
||||||
Write-Host " - The PXE server is running at $Server" -ForegroundColor Yellow
|
Write-Host " - The PXE server is running at $Server" -ForegroundColor Yellow
|
||||||
Write-Host " - This PC is on the 10.9.100.x network" -ForegroundColor Yellow
|
Write-Host " - This PC is on the 172.16.9.x imaging network" -ForegroundColor Yellow
|
||||||
Write-Host " - Samba is running on the PXE server" -ForegroundColor Yellow
|
Write-Host " - Samba is running on the PXE server" -ForegroundColor Yellow
|
||||||
|
if ($TempExtract) { Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue }
|
||||||
exit 1
|
exit 1
|
||||||
}
|
}
|
||||||
Write-Host "Connected." -ForegroundColor Green
|
Write-Host "Connected." -ForegroundColor Green
|
||||||
Write-Host ""
|
|
||||||
|
|
||||||
$failed = $false
|
$failed = $false
|
||||||
$stepNum = 0
|
$stepNum = 0
|
||||||
$totalSteps = 1 # Deploy base always
|
$totalSteps = 2 # Deploy base + OS always
|
||||||
if ($matchedOs.Count -gt 0) { $totalSteps++ }
|
|
||||||
if ($matchedPkgs.Count -gt 0) { $totalSteps++ }
|
if ($matchedPkgs.Count -gt 0) { $totalSteps++ }
|
||||||
if ($IncludeDrivers -and $driverDirs.Count -gt 0) { $totalSteps++ }
|
if ($IncludeDrivers -and $driverDirs.Count -gt 0) { $totalSteps++ }
|
||||||
if (Test-Path $ToolsPath -PathType Container) { $totalSteps++ }
|
if (Test-Path $ToolsPath -PathType Container) { $totalSteps++ }
|
||||||
if (Test-Path $SourcesZip) { $totalSteps++ }
|
if ($TempSources) { $totalSteps++ }
|
||||||
|
|
||||||
|
# robocopy exit codes 8 and up are failures
|
||||||
|
$rcArgs = @("/R:3", "/W:5", "/NP", "/ETA")
|
||||||
|
|
||||||
# --- Step: Deploy base (Control, Applications, config -- skip big dirs) ---
|
# --- Step: Deploy base (Control, Applications, config -- skip big dirs) ---
|
||||||
$stepNum++
|
$stepNum++
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
Write-Step "[$stepNum/$totalSteps] Copying Deploy\ base (Control, Applications, config) ..."
|
||||||
Write-Host "[$stepNum/$totalSteps] Copying Deploy\ base (Control, Applications, config) ..." -ForegroundColor Cyan
|
robocopy $DeployPath "$Share\Deploy" /E /XD @ExcludedDeployDirs @rcArgs
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: Deploy base copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
|
|
||||||
robocopy $DeployPath "$Share\Deploy" /E /XD "Operating Systems" "Out-of-box Drivers" "Packages" /R:3 /W:5 /NP /ETA
|
|
||||||
if ($LASTEXITCODE -ge 8) {
|
|
||||||
Write-Host "ERROR: Deploy base copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Step: Operating System ---
|
# --- Step: Operating System ---
|
||||||
if ($matchedOs.Count -gt 0) {
|
$stepNum++
|
||||||
$stepNum++
|
Write-Step "[$stepNum/$totalSteps] Copying Operating System ..."
|
||||||
Write-Host ""
|
foreach ($osDir in $osDirs) {
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
$src = Join-Path $CachePath $osDir
|
||||||
Write-Host "[$stepNum/$totalSteps] Copying Operating System ..." -ForegroundColor Cyan
|
if (Test-Path -LiteralPath $src -PathType Container) {
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
Write-Host " $osDir" -ForegroundColor Gray
|
||||||
|
robocopy $src (Join-Path $Share $osDir) /E @rcArgs
|
||||||
foreach ($osDir in $osDirs) {
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: OS copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
$src = Join-Path $CachePath $osDir
|
} else {
|
||||||
$dst = Join-Path $Share $osDir
|
Write-Host " SKIPPED (not cached): $osDir" -ForegroundColor Yellow
|
||||||
if (Test-Path $src -PathType Container) {
|
|
||||||
Write-Host " $osDir" -ForegroundColor Gray
|
|
||||||
robocopy $src $dst /E /R:3 /W:5 /NP /ETA
|
|
||||||
if ($LASTEXITCODE -ge 8) {
|
|
||||||
Write-Host "ERROR: OS copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
Write-Host " SKIPPED (not cached): $osDir" -ForegroundColor Yellow
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- Step: Packages ---
|
# --- Step: Packages ---
|
||||||
if ($matchedPkgs.Count -gt 0) {
|
if ($matchedPkgs.Count -gt 0) {
|
||||||
$stepNum++
|
$stepNum++
|
||||||
Write-Host ""
|
Write-Step "[$stepNum/$totalSteps] Copying Packages ..."
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
Write-Host "[$stepNum/$totalSteps] Copying Packages ..." -ForegroundColor Cyan
|
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
|
|
||||||
# Group packages by destination directory for efficient robocopy
|
|
||||||
$pkgGroups = [ordered]@{}
|
$pkgGroups = [ordered]@{}
|
||||||
foreach ($pkg in $matchedPkgs) {
|
foreach ($pkg in $matchedPkgs) {
|
||||||
$rel = Resolve-DestDir $pkg.destinationDir
|
$rel = Resolve-DestDir $pkg.destinationDir
|
||||||
if (-not $pkgGroups.Contains($rel)) { $pkgGroups[$rel] = @() }
|
if (-not $pkgGroups.Contains($rel)) { $pkgGroups[$rel] = @() }
|
||||||
$pkgGroups[$rel] += $pkg.fileName
|
$pkgGroups[$rel] += $pkg.fileName
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach ($dir in $pkgGroups.Keys) {
|
foreach ($dir in $pkgGroups.Keys) {
|
||||||
$src = Join-Path $CachePath $dir
|
$src = Join-Path $CachePath $dir
|
||||||
$dst = Join-Path $Share $dir
|
if (Test-Path -LiteralPath $src -PathType Container) {
|
||||||
$files = $pkgGroups[$dir]
|
foreach ($f in $pkgGroups[$dir]) { Write-Host " $f" -ForegroundColor Gray }
|
||||||
if (Test-Path $src -PathType Container) {
|
$robocopyArgs = @($src, (Join-Path $Share $dir)) + $pkgGroups[$dir] + $rcArgs
|
||||||
foreach ($f in $files) { Write-Host " $f" -ForegroundColor Gray }
|
|
||||||
$robocopyArgs = @($src, $dst) + $files + @("/R:3", "/W:5", "/NP", "/ETA")
|
|
||||||
& robocopy @robocopyArgs
|
& robocopy @robocopyArgs
|
||||||
if ($LASTEXITCODE -ge 8) {
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: Package copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
Write-Host "ERROR: Package copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
Write-Host " SKIPPED (not cached): $dir" -ForegroundColor Yellow
|
Write-Host " SKIPPED (not cached): $dir" -ForegroundColor Yellow
|
||||||
}
|
}
|
||||||
@@ -248,23 +505,15 @@ if ($matchedPkgs.Count -gt 0) {
|
|||||||
# --- Step: Drivers (only with -IncludeDrivers) ---
|
# --- Step: Drivers (only with -IncludeDrivers) ---
|
||||||
if ($IncludeDrivers -and $driverDirs.Count -gt 0) {
|
if ($IncludeDrivers -and $driverDirs.Count -gt 0) {
|
||||||
$stepNum++
|
$stepNum++
|
||||||
Write-Host ""
|
Write-Step "[$stepNum/$totalSteps] Copying Drivers ($($driverDirs.Count) model(s)) ..."
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
Write-Host "[$stepNum/$totalSteps] Copying Drivers ($($driverDirs.Count) model(s)) ..." -ForegroundColor Cyan
|
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
|
|
||||||
$drvCopied = 0
|
$drvCopied = 0
|
||||||
foreach ($drvDir in $driverDirs) {
|
foreach ($drvDir in $driverDirs) {
|
||||||
$drvCopied++
|
$drvCopied++
|
||||||
$src = Join-Path $CachePath $drvDir
|
$src = Join-Path $CachePath $drvDir
|
||||||
$dst = Join-Path $Share $drvDir
|
if (Test-Path -LiteralPath $src -PathType Container) {
|
||||||
if (Test-Path $src -PathType Container) {
|
|
||||||
Write-Host " [$drvCopied/$($driverDirs.Count)] $drvDir" -ForegroundColor Gray
|
Write-Host " [$drvCopied/$($driverDirs.Count)] $drvDir" -ForegroundColor Gray
|
||||||
robocopy $src $dst /E /R:3 /W:5 /NP /ETA
|
robocopy $src (Join-Path $Share $drvDir) /E @rcArgs
|
||||||
if ($LASTEXITCODE -ge 8) {
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: Driver copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
Write-Host "ERROR: Driver copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
Write-Host " SKIPPED (not cached): $drvDir" -ForegroundColor Yellow
|
Write-Host " SKIPPED (not cached): $drvDir" -ForegroundColor Yellow
|
||||||
}
|
}
|
||||||
@@ -274,106 +523,118 @@ if ($IncludeDrivers -and $driverDirs.Count -gt 0) {
|
|||||||
# --- Step: Tools ---
|
# --- Step: Tools ---
|
||||||
if (Test-Path $ToolsPath -PathType Container) {
|
if (Test-Path $ToolsPath -PathType Container) {
|
||||||
$stepNum++
|
$stepNum++
|
||||||
Write-Host ""
|
Write-Step "[$stepNum/$totalSteps] Copying Tools\ ..."
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
robocopy $ToolsPath "$Share\Tools" /E @rcArgs
|
||||||
Write-Host "[$stepNum/$totalSteps] Copying Tools\ ..." -ForegroundColor Cyan
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: Tools copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
|
|
||||||
robocopy $ToolsPath "$Share\Tools" /E /R:3 /W:5 /NP /ETA
|
|
||||||
if ($LASTEXITCODE -ge 8) {
|
|
||||||
Write-Host "ERROR: Tools copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- Step: Sources ---
|
# --- Step: Sources ---
|
||||||
$TempSources = $null
|
if ($TempSources) {
|
||||||
if (Test-Path $SourcesZip) {
|
|
||||||
$stepNum++
|
$stepNum++
|
||||||
Write-Host ""
|
Write-Step "[$stepNum/$totalSteps] Copying Sources\ ..."
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
robocopy $TempSources "$Share\Sources" /E @rcArgs
|
||||||
Write-Host "[$stepNum/$totalSteps] Extracting and copying Sources\ ..." -ForegroundColor Cyan
|
if ($LASTEXITCODE -ge 8) { Write-Host "ERROR: Sources copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red; $failed = $true }
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
|
||||||
|
|
||||||
$TempExtract = Join-Path $env:TEMP "SourcesExtract"
|
|
||||||
Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue
|
|
||||||
Expand-Archive $SourcesZip -DestinationPath $TempExtract -Force
|
|
||||||
|
|
||||||
# Handle nested Sources folder (zip may contain Sources/ at root)
|
|
||||||
$TempSources = $TempExtract
|
|
||||||
if ((Test-Path (Join-Path $TempExtract "Sources")) -and -not (Test-Path (Join-Path $TempExtract "Diskpart"))) {
|
|
||||||
$TempSources = Join-Path $TempExtract "Sources"
|
|
||||||
}
|
|
||||||
|
|
||||||
robocopy $TempSources "$Share\Sources" /E /R:3 /W:5 /NP /ETA
|
|
||||||
if ($LASTEXITCODE -ge 8) {
|
|
||||||
Write-Host "ERROR: Sources copy failed (exit code $LASTEXITCODE)" -ForegroundColor Red
|
|
||||||
$failed = $true
|
|
||||||
}
|
|
||||||
|
|
||||||
Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# --- Verify small files (SMB write-cache workaround) ---
|
# --- Verify on the server; re-send what is wrong ---
|
||||||
Write-Host ""
|
$byRel = @{}
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
foreach ($f in $sendList) { $byRel[($f.Rel -replace '\\', '/').ToLower()] = $f }
|
||||||
Write-Host "Verifying small files ..." -ForegroundColor Cyan
|
$verified = $false
|
||||||
Write-Host "========================================" -ForegroundColor Cyan
|
$result = $null
|
||||||
|
for ($round = 1; $round -le $MaxRounds + 1; $round++) {
|
||||||
|
Write-Step "Verifying on the server (round $round) ..."
|
||||||
|
$r = Invoke-Pxe -Method POST -Path "/api/upload/verify" -Body $manifestBytes -TimeoutSec 120
|
||||||
|
if ($r.Status -ne 202) {
|
||||||
|
Write-Host "ERROR: verify request refused ($($r.Status)): $(Get-ErrorText $r)" -ForegroundColor Red
|
||||||
|
break
|
||||||
|
}
|
||||||
|
$deadline = (Get-Date).AddHours(2)
|
||||||
|
$postedAt = Get-Date
|
||||||
|
do {
|
||||||
|
if ((Get-Date) -gt $deadline) { $result = $null; break }
|
||||||
|
if ($result -and $result.upload_id -ne $uploadId -and ((Get-Date) - $postedAt).TotalSeconds -gt 120) {
|
||||||
|
Write-Host "ERROR: the server is working on another upload ($($result.upload_id)). Run this again later." -ForegroundColor Red
|
||||||
|
$result = $null; break
|
||||||
|
}
|
||||||
|
Start-Sleep -Seconds 2
|
||||||
|
$s = Invoke-Pxe -Method GET -Path "/api/upload/verify" -TimeoutSec 30
|
||||||
|
$result = $null
|
||||||
|
if ($s.Status -eq 200) { $result = $s.Body | ConvertFrom-Json }
|
||||||
|
if ($result -and $result.state -eq "running" -and $result.total_bytes -gt 0) {
|
||||||
|
Write-Progress -Activity "Server is hashing the upload" -Status "$($result.done_files)/$($result.total_files) files" `
|
||||||
|
-PercentComplete ([Math]::Min(100, [int](100 * $result.done_bytes / $result.total_bytes)))
|
||||||
|
}
|
||||||
|
} while (-not $result -or $result.state -eq "running" -or $result.upload_id -ne $uploadId)
|
||||||
|
Write-Progress -Activity "Server is hashing the upload" -Completed
|
||||||
|
|
||||||
$fixCount = 0
|
if (-not $result) { Write-Host "ERROR: the server did not finish verifying within 2 hours." -ForegroundColor Red; break }
|
||||||
$verifyPairs = @(
|
if ($result.state -eq "error") { Write-Host "ERROR: the server could not verify: $($result.error)" -ForegroundColor Red; break }
|
||||||
@{ Local = (Join-Path $DeployPath "Control"); Remote = "$Share\Deploy\Control" }
|
if ($result.ok) { $verified = $true; break }
|
||||||
)
|
|
||||||
if (Test-Path $ToolsPath -PathType Container) {
|
|
||||||
$verifyPairs += @{ Local = $ToolsPath; Remote = "$Share\Tools" }
|
|
||||||
}
|
|
||||||
|
|
||||||
foreach ($pair in $verifyPairs) {
|
foreach ($b in @($result.bad_in_cache)) { if ($b) { Write-Host " BAD IN MCL CACHE: $($b.path) ($($b.reason))" -ForegroundColor Red } }
|
||||||
$localDir = $pair.Local
|
if (@($result.bad_in_cache).Count -gt 0) { break } # re-sending a bad source cannot help
|
||||||
$remoteDir = $pair.Remote
|
|
||||||
if (-not (Test-Path $localDir -PathType Container)) { continue }
|
|
||||||
|
|
||||||
Get-ChildItem -Path $localDir -Recurse -File -ErrorAction SilentlyContinue |
|
$redo = @(@($result.missing) + @($result.mismatch | ForEach-Object { $_.path }) | Where-Object { $_ })
|
||||||
Where-Object { $_.Length -gt 0 -and $_.Length -lt 1MB } |
|
Write-Host " $($redo.Count) file(s) missing or different on the server." -ForegroundColor Yellow
|
||||||
ForEach-Object {
|
if ($round -gt $MaxRounds) { break }
|
||||||
$rel = $_.FullName.Substring($localDir.Length)
|
foreach ($rel in $redo) {
|
||||||
$dstFile = Join-Path $remoteDir $rel
|
$f = $byRel[$rel.ToLower()]
|
||||||
if (Test-Path $dstFile) {
|
if (-not $f) { continue }
|
||||||
$dstSize = (Get-Item $dstFile).Length
|
if ($f.Size -le [long]$result.small_max) {
|
||||||
if ($dstSize -ne $_.Length) {
|
$q = "?upload_id=$([uri]::EscapeDataString($uploadId))&path=$([uri]::EscapeDataString($rel))"
|
||||||
Write-Host " Fixing: $rel ($dstSize -> $($_.Length) bytes)" -ForegroundColor Yellow
|
$p = Invoke-Pxe -Method PUT -Path "/api/upload/file$q" -InFile $f.Local -ContentType "application/octet-stream" -TimeoutSec 300
|
||||||
$bytes = [System.IO.File]::ReadAllBytes($_.FullName)
|
if ($p.Status -eq 200) { Write-Host " re-sent over HTTP: $rel" -ForegroundColor Green }
|
||||||
[System.IO.File]::WriteAllBytes($dstFile, $bytes)
|
else { Write-Host " HTTP re-send failed ($($p.Status)) for ${rel}: $(Get-ErrorText $p)" -ForegroundColor Red }
|
||||||
$fixCount++
|
} else {
|
||||||
}
|
$dstDir = Join-Path $Share (Split-Path ($f.Rel) -Parent)
|
||||||
|
$dstFile = Join-Path $dstDir (Split-Path $f.Local -Leaf)
|
||||||
|
Write-Host " re-copying (delete, then robocopy /J): $rel" -ForegroundColor Gray
|
||||||
|
# robocopy calls a same-size, same-time file "same" or "modified"
|
||||||
|
# and skips it, even with /IS /IT. Delete the bad copy so it is a
|
||||||
|
# new file; /IS /IT /IM stay as a backstop if the delete fails.
|
||||||
|
Remove-Item -LiteralPath $dstFile -Force -ErrorAction SilentlyContinue
|
||||||
|
$rc = robocopy (Split-Path $f.Local -Parent) $dstDir (Split-Path $f.Local -Leaf) /J /IS /IT /IM @rcArgs
|
||||||
|
if ($LASTEXITCODE -ge 8 -or $LASTEXITCODE -eq 0) {
|
||||||
|
# 0 = nothing copied: robocopy skipped it again
|
||||||
|
Write-Host " robocopy did not copy $rel (exit $LASTEXITCODE):" -ForegroundColor Red
|
||||||
|
$rc | Where-Object { $_ -match '\S' } | Select-Object -Last 8 | ForEach-Object { Write-Host " $_" }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if ($fixCount -eq 0) {
|
# --- Disconnect and clean up ---
|
||||||
Write-Host " All files verified OK." -ForegroundColor Green
|
cmd /c "net use ""$Share"" /delete /y >nul 2>&1" | Out-Null
|
||||||
} else {
|
if ($TempExtract) { Remove-Item -Recurse -Force $TempExtract -ErrorAction SilentlyContinue }
|
||||||
Write-Host " Fixed $fixCount file(s)." -ForegroundColor Yellow
|
|
||||||
}
|
|
||||||
|
|
||||||
# --- Disconnect ---
|
|
||||||
net use $Share /delete 2>$null | Out-Null
|
|
||||||
|
|
||||||
# --- Summary ---
|
# --- Summary ---
|
||||||
Write-Host ""
|
Write-Host ""
|
||||||
if ($failed) {
|
if ($failed -and $verified) {
|
||||||
Write-Host "========================================" -ForegroundColor Red
|
Write-Host " (robocopy reported errors on the first pass; the files it missed were re-sent and verified.)" -ForegroundColor Yellow
|
||||||
Write-Host " Upload completed with errors." -ForegroundColor Red
|
|
||||||
Write-Host "========================================" -ForegroundColor Red
|
|
||||||
} else {
|
|
||||||
Write-Host "========================================" -ForegroundColor Green
|
|
||||||
Write-Host " Upload complete!" -ForegroundColor Green
|
|
||||||
Write-Host "========================================" -ForegroundColor Green
|
|
||||||
}
|
}
|
||||||
Write-Host ""
|
if ($verified) {
|
||||||
Write-Host "Next steps:" -ForegroundColor Cyan
|
Write-Host "========================================" -ForegroundColor Green
|
||||||
Write-Host " 1. Open the PXE webapp: http://$Server`:9009" -ForegroundColor White
|
Write-Host " Upload complete: all $($sendList.Count) file(s) verified on the server." -ForegroundColor Green
|
||||||
Write-Host " 2. Go to Image Import" -ForegroundColor White
|
Write-Host "========================================" -ForegroundColor Green
|
||||||
Write-Host " 3. Select source 'image-upload' and target image type" -ForegroundColor White
|
if ([int]$result.extra_count -gt 0) {
|
||||||
Write-Host " 4. Click Import" -ForegroundColor White
|
Write-Host ""
|
||||||
Write-Host ""
|
Write-Host " NOTE: the upload folder also holds $($result.extra_count) file(s) from an earlier upload." -ForegroundColor Yellow
|
||||||
|
Write-Host " Import is blocked until you remove them (button on the Image Import page)." -ForegroundColor Yellow
|
||||||
|
}
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "Next steps:" -ForegroundColor Cyan
|
||||||
|
Write-Host " 1. Open the PXE webapp: http://${Server}:$WebPort/images/import" -ForegroundColor White
|
||||||
|
Write-Host " 2. Target: $(if ($isLtsc) { 'Shop Floor (the only choice for LTSC)' } else { 'Standard or Engineer' })" -ForegroundColor White
|
||||||
|
Write-Host " 3. Click Start import" -ForegroundColor White
|
||||||
|
Write-Host ""
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
Write-Host "========================================" -ForegroundColor Red
|
||||||
|
Write-Host " Upload NOT verified." -ForegroundColor Red
|
||||||
|
Write-Host "========================================" -ForegroundColor Red
|
||||||
|
if ($result) {
|
||||||
|
foreach ($m in @($result.missing)) { if ($m) { Write-Host " missing: $m" -ForegroundColor Red } }
|
||||||
|
foreach ($m in @($result.mismatch)) { if ($m) { Write-Host " differs: $($m.path) ($($m.reason))" -ForegroundColor Red } }
|
||||||
|
}
|
||||||
|
Write-Host "The webapp will not import this upload. Run the script again." -ForegroundColor Yellow
|
||||||
|
exit 1
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ from services import (
|
|||||||
nic_status,
|
nic_status,
|
||||||
system,
|
system,
|
||||||
unattend,
|
unattend,
|
||||||
|
upload_verify,
|
||||||
wim,
|
wim,
|
||||||
)
|
)
|
||||||
from services.audit import audit
|
from services.audit import audit
|
||||||
@@ -217,11 +218,32 @@ def images_import():
|
|||||||
flash(f"Source path does not exist: {source}", "danger")
|
flash(f"Source path does not exist: {source}", "danger")
|
||||||
return redirect(url_for("images_import"))
|
return redirect(url_for("images_import"))
|
||||||
|
|
||||||
|
# Network uploads must be the verified Upload-Image.ps1 upload: every
|
||||||
|
# file hashed on this disk against the PC's manifest, nothing stray.
|
||||||
|
up_real = os.path.realpath(config.UPLOAD_DIR)
|
||||||
|
src_real = os.path.realpath(source)
|
||||||
|
is_upload = src_real == up_real or src_real.startswith(up_real + "/")
|
||||||
|
if is_upload:
|
||||||
|
if src_real != up_real:
|
||||||
|
flash("Import the whole upload folder; its manifest covers all of it.", "danger")
|
||||||
|
return redirect(url_for("images_import"))
|
||||||
|
# Held through gate, move and clean-up: no verify or PUT may
|
||||||
|
# touch the folder while its files are being moved.
|
||||||
|
if not upload_verify.try_lock():
|
||||||
|
flash("Import refused: a verification or file upload is running. Try again.", "danger")
|
||||||
|
return redirect(url_for("images_import"))
|
||||||
|
ok, reason, _info = upload_verify.gate(target, locked=True)
|
||||||
|
if not ok:
|
||||||
|
upload_verify.unlock()
|
||||||
|
flash(f"Import refused: {reason}", "danger")
|
||||||
|
return redirect(url_for("images_import"))
|
||||||
|
|
||||||
root = fs.image_root(target)
|
root = fs.image_root(target)
|
||||||
dest = fs.deploy_path(target)
|
dest = fs.deploy_path(target)
|
||||||
try:
|
try:
|
||||||
os.makedirs(dest, exist_ok=True)
|
os.makedirs(dest, exist_ok=True)
|
||||||
src_items = os.listdir(source)
|
# The manifest / verify state stay behind (finish_import clears them).
|
||||||
|
src_items = [i for i in os.listdir(source) if not upload_verify.is_reserved(i)]
|
||||||
|
|
||||||
# Move files from network upload to save disk space; copy from USB.
|
# Move files from network upload to save disk space; copy from USB.
|
||||||
# deploy.sync_tree uses rsync --checksum: unchanged files are
|
# deploy.sync_tree uses rsync --checksum: unchanged files are
|
||||||
@@ -265,18 +287,32 @@ def images_import():
|
|||||||
media_tag = os.path.join(control_dir, "Media.tag")
|
media_tag = os.path.join(control_dir, "Media.tag")
|
||||||
Path(media_tag).touch()
|
Path(media_tag).touch()
|
||||||
|
|
||||||
audit("IMAGE_IMPORT", f"{source} -> {target}")
|
if is_upload:
|
||||||
|
rec = upload_verify.finish_import(root, target)
|
||||||
|
audit("IMAGE_IMPORT", f"{source} -> {target} (verified upload {rec.get('upload_id')})")
|
||||||
|
else:
|
||||||
|
audit("IMAGE_IMPORT", f"{source} -> {target} (USB, not verified)")
|
||||||
flash(
|
flash(
|
||||||
f"Successfully imported content to {config.FRIENDLY_NAMES.get(target, target)}.",
|
f"Successfully imported content to {config.FRIENDLY_NAMES.get(target, target)}.",
|
||||||
"success",
|
"success",
|
||||||
)
|
)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
flash(f"Import failed: {exc}", "danger")
|
flash(f"Import failed: {exc}", "danger")
|
||||||
|
finally:
|
||||||
|
if is_upload:
|
||||||
|
upload_verify.unlock()
|
||||||
|
|
||||||
return redirect(url_for("images_import"))
|
return redirect(url_for("images_import"))
|
||||||
|
|
||||||
|
up_ok, up_reason, up_info = upload_verify.gate()
|
||||||
|
last_imports = {it: upload_verify.last_import(fs.image_root(it)) for it in config.IMAGE_TYPES}
|
||||||
return render_template(
|
return render_template(
|
||||||
"import.html",
|
"import.html",
|
||||||
|
upload_ok=up_ok,
|
||||||
|
upload_reason=up_reason,
|
||||||
|
upload_info=up_info,
|
||||||
|
last_imports=last_imports,
|
||||||
|
upload_dir=config.UPLOAD_DIR,
|
||||||
usb_mounts=usb_mounts,
|
usb_mounts=usb_mounts,
|
||||||
upload_sources=upload_sources,
|
upload_sources=upload_sources,
|
||||||
images=image_list,
|
images=image_list,
|
||||||
@@ -1872,6 +1908,46 @@ def audit_log():
|
|||||||
# Routes - JSON API
|
# Routes - JSON API
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@app.route("/images/import/remove-extras", methods=["POST"])
|
||||||
|
def images_import_remove_extras():
|
||||||
|
try:
|
||||||
|
n = upload_verify.remove_extras()
|
||||||
|
except upload_verify.UploadError as ex:
|
||||||
|
flash(f"Could not remove: {ex}", "danger")
|
||||||
|
return redirect(url_for("images_import"))
|
||||||
|
audit("UPLOAD_REMOVE_EXTRAS", f"{n} file(s) not in the manifest removed")
|
||||||
|
flash(f"Removed {n} file(s) that were not part of this upload.", "success")
|
||||||
|
return redirect(url_for("images_import"))
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/upload/verify", methods=["GET", "POST"])
|
||||||
|
def api_upload_verify():
|
||||||
|
# CSRF-exempt machine-to-machine endpoint: Upload-Image.ps1 POSTs its
|
||||||
|
# manifest after robocopy, then polls GET until the hashing is done.
|
||||||
|
if request.method == "GET":
|
||||||
|
return jsonify(upload_verify.public_state())
|
||||||
|
data = request.get_json(silent=True)
|
||||||
|
try:
|
||||||
|
st = upload_verify.start_verify(data)
|
||||||
|
except upload_verify.UploadError as ex:
|
||||||
|
return jsonify({"error": str(ex)}), ex.status
|
||||||
|
audit("UPLOAD_VERIFY", f"upload {st.get('upload_id')}: {st.get('total_files')} file(s)")
|
||||||
|
return jsonify(st), 202
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/api/upload/file", methods=["PUT"])
|
||||||
|
def api_upload_file():
|
||||||
|
# Second channel for files SMB delivered wrong: one manifest file per
|
||||||
|
# PUT, hash-checked before it replaces anything.
|
||||||
|
try:
|
||||||
|
res = upload_verify.receive_file(request.args.get("upload_id", ""), request.args.get("path", ""),
|
||||||
|
request.stream, request.content_length)
|
||||||
|
except upload_verify.UploadError as ex:
|
||||||
|
return jsonify({"error": str(ex)}), ex.status
|
||||||
|
audit("UPLOAD_FILE_RESENT", f"{res['path']} ({res['size']} bytes) over HTTP")
|
||||||
|
return jsonify(res), 200
|
||||||
|
|
||||||
|
|
||||||
@app.route("/api/services")
|
@app.route("/api/services")
|
||||||
def api_services():
|
def api_services():
|
||||||
services = {s: system.service_status(s) for s in ("dnsmasq", "apache2", "smbd")}
|
services = {s: system.service_status(s) for s in ("dnsmasq", "apache2", "smbd")}
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ CSRF_EXEMPT_PATHS = {
|
|||||||
"/imaging/status",
|
"/imaging/status",
|
||||||
"/reports/cloud-status",
|
"/reports/cloud-status",
|
||||||
"/hardware/seen",
|
"/hardware/seen",
|
||||||
|
"/api/upload/verify", # Upload-Image.ps1 (PUT /api/upload/file is not a POST)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
708
webapp/services/upload_verify.py
Normal file
708
webapp/services/upload_verify.py
Normal file
@@ -0,0 +1,708 @@
|
|||||||
|
"""Verified MCL uploads: prove every file in the upload folder is the one
|
||||||
|
the Windows PC sent, repair bad ones over HTTP, and gate the import.
|
||||||
|
|
||||||
|
Upload-Image.ps1 (work PC) hashes every file it copies into
|
||||||
|
upload-manifest.json and POSTs that manifest here after robocopy. A
|
||||||
|
background thread then hashes the files on THIS disk (no SMB client cache
|
||||||
|
in the way) and records which are missing, which differ, and which were
|
||||||
|
already bad in the MCL cache (their hash on the PC does not match the hash
|
||||||
|
MCL's own Control JSONs publish). The script re-sends the bad small files
|
||||||
|
with PUT, which writes them to a temp name, checks the hash and renames
|
||||||
|
them into place.
|
||||||
|
|
||||||
|
The import page only imports an upload folder whose last verification
|
||||||
|
passed and whose files have not changed since (size, mtime, ctime and
|
||||||
|
inode of every file), with no stray files from an earlier upload mixed in.
|
||||||
|
The import holds the same lock as verification and PUT, so neither can
|
||||||
|
run while files are being moved.
|
||||||
|
|
||||||
|
Paths are matched against the disk case-insensitively, as Windows and
|
||||||
|
Samba do: the manifest's case comes from MCL's JSON, the disk's from
|
||||||
|
whatever robocopy created first.
|
||||||
|
|
||||||
|
State files live at the top of the upload folder and are never imported:
|
||||||
|
upload-manifest.json the manifest (written by the POST, not by SMB)
|
||||||
|
.upload-verify.json progress / last result (small; polled)
|
||||||
|
.upload-verify-files.json per-file hash cache and snapshot (gate only)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import secrets
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
import config
|
||||||
|
|
||||||
|
MANIFEST = "upload-manifest.json"
|
||||||
|
STATE = ".upload-verify.json"
|
||||||
|
DETAIL = ".upload-verify-files.json"
|
||||||
|
TOP_FILES = (MANIFEST, STATE, DETAIL)
|
||||||
|
PART_RE = re.compile(r"^\..+\.part-[0-9a-f]{16}$")
|
||||||
|
|
||||||
|
SMALL_MAX = 64 * 1024 * 1024 # PUT limit; bigger files go back to robocopy
|
||||||
|
MAX_FILES = 200_000
|
||||||
|
UPLOAD_ID_RE = re.compile(r"^[A-Za-z0-9._-]{1,80}$")
|
||||||
|
SHA_RE = re.compile(r"^[0-9a-fA-F]{64}$")
|
||||||
|
DRIVE_RE = re.compile(r"^[A-Za-z]:[\\/]")
|
||||||
|
CHUNK = 1024 * 1024
|
||||||
|
|
||||||
|
_lock = threading.Lock() # one verify, PUT or import at a time
|
||||||
|
|
||||||
|
|
||||||
|
class UploadError(ValueError):
|
||||||
|
"""Bad request (400) unless .status says otherwise."""
|
||||||
|
|
||||||
|
def __init__(self, msg, status=400):
|
||||||
|
super().__init__(msg)
|
||||||
|
self.status = status
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def exclusive(what):
|
||||||
|
"""Hold the upload lock or fail at once with 409."""
|
||||||
|
if not _lock.acquire(blocking=False):
|
||||||
|
raise UploadError(f"busy: a verification, file upload or import is running ({what} refused)", 409)
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
_lock.release()
|
||||||
|
|
||||||
|
|
||||||
|
# --- paths ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def root():
|
||||||
|
return config.UPLOAD_DIR
|
||||||
|
|
||||||
|
|
||||||
|
def _now():
|
||||||
|
return datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")
|
||||||
|
|
||||||
|
|
||||||
|
def is_reserved(rel):
|
||||||
|
"""Only our own files at the top of the folder (and their temp files).
|
||||||
|
A '.part-' file deeper down is debris and counts as stray."""
|
||||||
|
if "/" in rel:
|
||||||
|
return False
|
||||||
|
if rel in TOP_FILES:
|
||||||
|
return True
|
||||||
|
m = PART_RE.match(rel)
|
||||||
|
return bool(m) and any(rel.startswith(f".{n}.part-") for n in TOP_FILES)
|
||||||
|
|
||||||
|
|
||||||
|
def norm_path(p):
|
||||||
|
"""Manifest path -> 'Deploy/Control/x.json'. Raises on anything that
|
||||||
|
could leave the upload folder."""
|
||||||
|
if not isinstance(p, str) or not p or len(p) > 1024 or "\x00" in p:
|
||||||
|
raise UploadError("bad path")
|
||||||
|
p = p.replace("\\", "/")
|
||||||
|
if p.startswith("/") or DRIVE_RE.match(p) or ":" in p:
|
||||||
|
raise UploadError(f"absolute path not allowed: {p}")
|
||||||
|
parts = p.split("/")
|
||||||
|
if any(s in ("", ".", "..") for s in parts):
|
||||||
|
raise UploadError(f"bad path: {p}")
|
||||||
|
if is_reserved(p) or PART_RE.match(parts[-1]):
|
||||||
|
raise UploadError(f"reserved name: {p}")
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def safe_join(rel, base=None):
|
||||||
|
"""Absolute path of rel inside base, refusing symlink escapes."""
|
||||||
|
base = os.path.realpath(base or root())
|
||||||
|
full = os.path.join(base, *rel.split("/"))
|
||||||
|
real = os.path.realpath(full)
|
||||||
|
if real != full or not real.startswith(base + os.sep):
|
||||||
|
# realpath differs: a symlink somewhere on the way. Refuse it.
|
||||||
|
raise UploadError(f"path leaves the upload folder or uses a link: {rel}")
|
||||||
|
return full
|
||||||
|
|
||||||
|
|
||||||
|
def scan(base=None):
|
||||||
|
"""Walk the upload folder once.
|
||||||
|
Returns (files, links): files = {rel_lower: [rel, ...]} for regular
|
||||||
|
files; links = rels of symlinks and other non-regular entries, which
|
||||||
|
are never part of an upload."""
|
||||||
|
base = base or root()
|
||||||
|
files, links = {}, []
|
||||||
|
for dp, dns, fns in os.walk(base):
|
||||||
|
for d in list(dns):
|
||||||
|
if os.path.islink(os.path.join(dp, d)):
|
||||||
|
links.append(os.path.relpath(os.path.join(dp, d), base).replace(os.sep, "/"))
|
||||||
|
dns.remove(d)
|
||||||
|
dns.sort()
|
||||||
|
for fn in sorted(fns):
|
||||||
|
full = os.path.join(dp, fn)
|
||||||
|
rel = os.path.relpath(full, base).replace(os.sep, "/")
|
||||||
|
if is_reserved(rel):
|
||||||
|
continue
|
||||||
|
if os.path.islink(full) or not os.path.isfile(full):
|
||||||
|
links.append(rel)
|
||||||
|
continue
|
||||||
|
files.setdefault(rel.lower(), []).append(rel)
|
||||||
|
return files, links
|
||||||
|
|
||||||
|
|
||||||
|
def _key(st):
|
||||||
|
"""What 'unchanged' means: robocopy keeps the source mtime, so mtime and
|
||||||
|
size alone cannot tell a re-copy from the bad copy before it. ctime is
|
||||||
|
set by the kernel on every write and cannot be set from userland."""
|
||||||
|
return [st.st_size, st.st_mtime_ns, st.st_ctime_ns, st.st_ino]
|
||||||
|
|
||||||
|
|
||||||
|
# --- manifest ---------------------------------------------------------------
|
||||||
|
|
||||||
|
def parse_manifest(data):
|
||||||
|
"""Validate a manifest dict; return it with normalised paths and
|
||||||
|
lower-case hashes."""
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
raise UploadError("manifest must be a JSON object")
|
||||||
|
if data.get("manifest_version") != 1:
|
||||||
|
raise UploadError("unsupported manifest_version (want 1)")
|
||||||
|
uid = data.get("upload_id")
|
||||||
|
if not isinstance(uid, str) or not UPLOAD_ID_RE.match(uid):
|
||||||
|
raise UploadError("bad upload_id")
|
||||||
|
files = data.get("files")
|
||||||
|
if not isinstance(files, list) or not files:
|
||||||
|
raise UploadError("manifest has no files")
|
||||||
|
if len(files) > MAX_FILES:
|
||||||
|
raise UploadError("too many files")
|
||||||
|
out, seen = [], set()
|
||||||
|
for f in files:
|
||||||
|
if not isinstance(f, dict):
|
||||||
|
raise UploadError("bad file entry")
|
||||||
|
p = norm_path(f.get("path"))
|
||||||
|
size, sha = f.get("size"), f.get("sha256")
|
||||||
|
if not isinstance(size, int) or isinstance(size, bool) or size < 0:
|
||||||
|
raise UploadError(f"bad size for {p}")
|
||||||
|
if not isinstance(sha, str) or not SHA_RE.match(sha):
|
||||||
|
raise UploadError(f"bad sha256 for {p}")
|
||||||
|
if p.lower() in seen:
|
||||||
|
raise UploadError(f"duplicate path: {p}")
|
||||||
|
seen.add(p.lower())
|
||||||
|
out.append({"path": p, "size": size, "sha256": sha.lower()})
|
||||||
|
mcl = data.get("mcl") if isinstance(data.get("mcl"), dict) else {}
|
||||||
|
clean = {"manifest_version": 1, "upload_id": uid, "files": out, "mcl": mcl}
|
||||||
|
for k in ("created", "source_pc"):
|
||||||
|
if isinstance(data.get(k), str):
|
||||||
|
clean[k] = data[k][:200]
|
||||||
|
return clean
|
||||||
|
|
||||||
|
|
||||||
|
def _read_json(path):
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
return json.loads(f.read().decode("utf-8-sig"))
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path, obj, indent=1):
|
||||||
|
"""Atomic: temp file in the same folder, fsync, rename."""
|
||||||
|
d = os.path.dirname(path)
|
||||||
|
tmp = os.path.join(d, f".{os.path.basename(path)}.part-{secrets.token_hex(8)}")
|
||||||
|
with open(tmp, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(obj, f, indent=indent)
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
|
def load_manifest():
|
||||||
|
"""(manifest, raw_sha) or (None, None) when there is none."""
|
||||||
|
p = os.path.join(root(), MANIFEST)
|
||||||
|
if not os.path.isfile(p):
|
||||||
|
return None, None
|
||||||
|
with open(p, "rb") as f:
|
||||||
|
raw = f.read()
|
||||||
|
try:
|
||||||
|
m = parse_manifest(json.loads(raw.decode("utf-8-sig")))
|
||||||
|
except (ValueError, UnicodeDecodeError):
|
||||||
|
return None, None
|
||||||
|
return m, hashlib.sha256(raw).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def load_state():
|
||||||
|
try:
|
||||||
|
return _read_json(os.path.join(root(), STATE))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def load_detail():
|
||||||
|
try:
|
||||||
|
return _read_json(os.path.join(root(), DETAIL))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _chown_like_parent(path):
|
||||||
|
"""Match the share's owner (Samba 'force user = pxe') so a later SMB
|
||||||
|
re-copy can overwrite what the webapp (root) wrote."""
|
||||||
|
try:
|
||||||
|
st = os.stat(os.path.dirname(path))
|
||||||
|
os.chown(path, st.st_uid, st.st_gid)
|
||||||
|
except (OSError, AttributeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# --- MCL's own hashes and OS ------------------------------------------------
|
||||||
|
|
||||||
|
def _mcl_rel(dest_dir, file_name):
|
||||||
|
"""MCL destination ('*destinationdir*\\Deploy\\Packages\\X' or
|
||||||
|
'C:\\Deploy\\Languages\\Y') + file name -> 'Deploy/Packages/X/name'."""
|
||||||
|
d = (dest_dir or "").replace("\\", "/")
|
||||||
|
d = re.sub(r"^\*destinationdir\*/?", "", d, flags=re.I)
|
||||||
|
d = re.sub(r"^[A-Za-z]:/", "", d).strip("/")
|
||||||
|
return f"{d}/{file_name}" if d else file_name
|
||||||
|
|
||||||
|
|
||||||
|
def _control_list(base, name):
|
||||||
|
try:
|
||||||
|
d = _read_json(os.path.join(base, "Deploy", "Control", name))
|
||||||
|
return d if isinstance(d, list) else []
|
||||||
|
except (OSError, ValueError, UnicodeDecodeError):
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def mcl_expected(base=None):
|
||||||
|
"""{path_lower: (path, sha256_lower)} for every payload file the
|
||||||
|
Control JSONs in this upload describe. Only real SHA-256 values."""
|
||||||
|
base = base or root()
|
||||||
|
exp = {}
|
||||||
|
|
||||||
|
def add(dest, name, sha):
|
||||||
|
if not (dest and name and isinstance(sha, str) and SHA_RE.match(sha)):
|
||||||
|
return
|
||||||
|
rel = _mcl_rel(dest, name)
|
||||||
|
exp[rel.lower()] = (rel, sha.lower())
|
||||||
|
|
||||||
|
for e in _control_list(base, "OperatingSystem.json"):
|
||||||
|
wim = ((e or {}).get("operatingSystemVersion") or {}).get("wim") or {}
|
||||||
|
add(wim.get("destinationDir") or wim.get("DestinationDir"),
|
||||||
|
wim.get("fileName") or wim.get("FileName") or "install.wim", wim.get("hash") or wim.get("Hash"))
|
||||||
|
for fname in ("HardwareDriver.json", "packages.json"):
|
||||||
|
for e in _control_list(base, fname):
|
||||||
|
if isinstance(e, dict):
|
||||||
|
add(e.get("destinationDir") or e.get("DestinationDir"),
|
||||||
|
e.get("fileName") or e.get("FileName"), e.get("hash") or e.get("Hash"))
|
||||||
|
for e in _control_list(base, "LanguagePacks.json"):
|
||||||
|
for lf in (e or {}).get("files") or []:
|
||||||
|
if isinstance(lf, dict):
|
||||||
|
add(lf.get("destFolder"), lf.get("fileName"), lf.get("hash"))
|
||||||
|
return exp
|
||||||
|
|
||||||
|
|
||||||
|
def detected_os(manifest=None, base=None):
|
||||||
|
"""The OS the upload carries: the OperatingSystem.json entry whose id
|
||||||
|
is the selected one (manifest mcl.os_id, else Tools/user_selections.json).
|
||||||
|
A single entry is taken as is. None when it cannot be told."""
|
||||||
|
base = base or root()
|
||||||
|
entries = [e.get("operatingSystemVersion") for e in _control_list(base, "OperatingSystem.json")
|
||||||
|
if isinstance(e, dict) and isinstance(e.get("operatingSystemVersion"), dict)]
|
||||||
|
if not entries:
|
||||||
|
return None
|
||||||
|
want = ((manifest or {}).get("mcl") or {}).get("os_id")
|
||||||
|
if want is None:
|
||||||
|
try:
|
||||||
|
sel = _read_json(os.path.join(base, "Tools", "user_selections.json"))
|
||||||
|
want = sel[0].get("OperatingSystemSelection")
|
||||||
|
except (OSError, ValueError, KeyError, IndexError, TypeError, AttributeError):
|
||||||
|
want = None
|
||||||
|
if want is not None:
|
||||||
|
v = next((e for e in entries if str(e.get("id")) == str(want)), None)
|
||||||
|
else:
|
||||||
|
v = entries[0] if len(entries) == 1 else None
|
||||||
|
if not v or not v.get("productName"):
|
||||||
|
return None
|
||||||
|
return {"id": v.get("id"), "name": v.get("productName", ""),
|
||||||
|
"version": v.get("versionNumber", ""), "build": v.get("buildNumber", "")}
|
||||||
|
|
||||||
|
|
||||||
|
def allowed_targets(os_info):
|
||||||
|
"""LTSC is always Shop Floor; anything else is Standard or Engineer.
|
||||||
|
Unknown OS: nothing (fail closed)."""
|
||||||
|
if not os_info:
|
||||||
|
return []
|
||||||
|
if "ltsc" in (os_info.get("name") or "").lower():
|
||||||
|
return [t for t in config.IMAGE_TYPES if "shopfloor" in t]
|
||||||
|
return [t for t in config.IMAGE_TYPES if "shopfloor" not in t]
|
||||||
|
|
||||||
|
|
||||||
|
# --- verification -----------------------------------------------------------
|
||||||
|
|
||||||
|
def _sha256(path, progress=None):
|
||||||
|
h = hashlib.sha256()
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
while True:
|
||||||
|
b = f.read(CHUNK)
|
||||||
|
if not b:
|
||||||
|
break
|
||||||
|
h.update(b)
|
||||||
|
if progress:
|
||||||
|
progress(len(b))
|
||||||
|
return h.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def extras(manifest, base=None, disk=None):
|
||||||
|
"""Files (and links) in the upload folder that this upload did not send."""
|
||||||
|
files, links = disk or scan(base)
|
||||||
|
want = {f["path"].lower() for f in manifest["files"]}
|
||||||
|
out = list(links)
|
||||||
|
for low, rels in files.items():
|
||||||
|
if low not in want:
|
||||||
|
out.extend(rels)
|
||||||
|
elif len(rels) > 1:
|
||||||
|
out.extend(rels[1:]) # same name in two cases: one is stray
|
||||||
|
return sorted(out)
|
||||||
|
|
||||||
|
|
||||||
|
def run_verify(manifest, manifest_sha, base=None):
|
||||||
|
"""Hash every manifest file on disk; write and return the result.
|
||||||
|
Call with the lock held."""
|
||||||
|
base = base or root()
|
||||||
|
state_path = os.path.join(base, STATE)
|
||||||
|
cache = {}
|
||||||
|
prev = load_detail() if base == root() else {}
|
||||||
|
if prev.get("upload_id") == manifest["upload_id"]:
|
||||||
|
cache = prev.get("hashes") or {}
|
||||||
|
|
||||||
|
files = manifest["files"]
|
||||||
|
total_bytes = sum(f["size"] for f in files)
|
||||||
|
st = {"state": "running", "upload_id": manifest["upload_id"], "manifest_sha": manifest_sha,
|
||||||
|
"started": _now(), "total_files": len(files), "total_bytes": total_bytes,
|
||||||
|
"done_files": 0, "done_bytes": 0}
|
||||||
|
_write_json(state_path, st)
|
||||||
|
last = [time.monotonic()]
|
||||||
|
|
||||||
|
def tick(n):
|
||||||
|
st["done_bytes"] += n
|
||||||
|
if time.monotonic() - last[0] > 2:
|
||||||
|
last[0] = time.monotonic()
|
||||||
|
_write_json(state_path, st)
|
||||||
|
|
||||||
|
disk = scan(base)
|
||||||
|
missing, mismatch, empty_json, hashes, snap = [], [], [], {}, {}
|
||||||
|
for f in files:
|
||||||
|
rel = f["path"]
|
||||||
|
on_disk = disk[0].get(rel.lower()) or []
|
||||||
|
if not on_disk:
|
||||||
|
missing.append(rel)
|
||||||
|
st["done_bytes"] += f["size"]
|
||||||
|
st["done_files"] += 1
|
||||||
|
continue
|
||||||
|
if len(on_disk) > 1:
|
||||||
|
mismatch.append({"path": rel, "reason": "two files differ only in case: " + ", ".join(on_disk)})
|
||||||
|
st["done_bytes"] += f["size"]
|
||||||
|
st["done_files"] += 1
|
||||||
|
continue
|
||||||
|
actual = on_disk[0]
|
||||||
|
try:
|
||||||
|
p = safe_join(actual, base)
|
||||||
|
except UploadError:
|
||||||
|
missing.append(rel)
|
||||||
|
st["done_files"] += 1
|
||||||
|
continue
|
||||||
|
s = os.stat(p)
|
||||||
|
k = _key(s)
|
||||||
|
if s.st_size != f["size"]:
|
||||||
|
mismatch.append({"path": rel, "reason": f"size {s.st_size}, expected {f['size']}"})
|
||||||
|
st["done_bytes"] += f["size"]
|
||||||
|
else:
|
||||||
|
c = cache.get(actual)
|
||||||
|
if c and c[:4] == k and c[4] == f["sha256"]:
|
||||||
|
sha = c[4] # same file, already proven right
|
||||||
|
st["done_bytes"] += f["size"]
|
||||||
|
else:
|
||||||
|
sha = _sha256(p, tick)
|
||||||
|
if sha != f["sha256"]:
|
||||||
|
mismatch.append({"path": rel, "reason": "content differs"})
|
||||||
|
else:
|
||||||
|
hashes[actual] = k + [sha] # only proven-good files are cached
|
||||||
|
snap[actual] = k
|
||||||
|
low = rel.lower()
|
||||||
|
if s.st_size == 0 and low.endswith(".json") and low.startswith(("deploy/control/", "tools/")):
|
||||||
|
empty_json.append(rel)
|
||||||
|
st["done_files"] += 1
|
||||||
|
|
||||||
|
# Bad at the source: the PC's hash is not the one MCL publishes.
|
||||||
|
bad_cache = []
|
||||||
|
by_lower = {f["path"].lower(): f for f in files}
|
||||||
|
for low, (_mrel, msha) in mcl_expected(base).items():
|
||||||
|
f = by_lower.get(low)
|
||||||
|
if f and f["sha256"] != msha:
|
||||||
|
bad_cache.append({"path": f["path"], "reason": "hash differs from MCL's Control JSON"})
|
||||||
|
for rel in empty_json:
|
||||||
|
bad_cache.append({"path": rel, "reason": "empty Control JSON"})
|
||||||
|
|
||||||
|
ext = extras(manifest, base, disk)
|
||||||
|
ok = not (missing or mismatch or bad_cache)
|
||||||
|
os_info = detected_os(manifest, base) if ok else None
|
||||||
|
allowed = allowed_targets(os_info) if ok else []
|
||||||
|
result = {
|
||||||
|
"state": "done", "ok": ok, "importable": ok and not ext and bool(allowed),
|
||||||
|
"upload_id": manifest["upload_id"], "manifest_sha": manifest_sha,
|
||||||
|
"started": st["started"], "finished": _now(),
|
||||||
|
"total_files": len(files), "total_bytes": total_bytes,
|
||||||
|
"missing": missing,
|
||||||
|
"mismatch": mismatch,
|
||||||
|
"bad_in_cache": bad_cache,
|
||||||
|
"extra": ext[:500], "extra_count": len(ext),
|
||||||
|
"os": os_info,
|
||||||
|
"allowed_targets": allowed,
|
||||||
|
"small_max": SMALL_MAX,
|
||||||
|
}
|
||||||
|
_write_json(os.path.join(base, DETAIL), {"upload_id": manifest["upload_id"],
|
||||||
|
"manifest_sha": manifest_sha,
|
||||||
|
"snapshot": snap, "hashes": hashes}, indent=None)
|
||||||
|
_write_json(state_path, result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def start_verify(data):
|
||||||
|
"""Store the POSTed manifest and start hashing in the background.
|
||||||
|
Returns the public state."""
|
||||||
|
m = parse_manifest(data)
|
||||||
|
if not _lock.acquire(blocking=False):
|
||||||
|
raise UploadError("busy: a verification, file upload or import is running", 409)
|
||||||
|
try:
|
||||||
|
base = root()
|
||||||
|
os.makedirs(base, exist_ok=True)
|
||||||
|
raw = json.dumps(data, indent=1).encode("utf-8")
|
||||||
|
mpath = os.path.join(base, MANIFEST)
|
||||||
|
tmp = os.path.join(base, f".{MANIFEST}.part-{secrets.token_hex(8)}")
|
||||||
|
with open(tmp, "wb") as f:
|
||||||
|
f.write(raw)
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
os.replace(tmp, mpath)
|
||||||
|
_chown_like_parent(mpath)
|
||||||
|
msha = hashlib.sha256(raw).hexdigest()
|
||||||
|
_write_json(os.path.join(base, STATE),
|
||||||
|
{"state": "running", "upload_id": m["upload_id"], "manifest_sha": msha,
|
||||||
|
"started": _now(), "total_files": len(m["files"]),
|
||||||
|
"total_bytes": sum(f["size"] for f in m["files"]), "done_files": 0,
|
||||||
|
"done_bytes": 0})
|
||||||
|
except BaseException:
|
||||||
|
_lock.release()
|
||||||
|
raise
|
||||||
|
|
||||||
|
def work():
|
||||||
|
try:
|
||||||
|
run_verify(m, msha)
|
||||||
|
except Exception as ex: # report, never leave "running" behind
|
||||||
|
try:
|
||||||
|
_write_json(os.path.join(root(), STATE),
|
||||||
|
{"state": "error", "ok": False, "importable": False,
|
||||||
|
"upload_id": m["upload_id"], "error": str(ex), "finished": _now()})
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
_lock.release()
|
||||||
|
|
||||||
|
threading.Thread(target=work, name="upload-verify", daemon=True).start()
|
||||||
|
return public_state()
|
||||||
|
|
||||||
|
|
||||||
|
def public_state():
|
||||||
|
s = load_state()
|
||||||
|
return s if s else {"state": "none"}
|
||||||
|
|
||||||
|
|
||||||
|
# --- second channel: PUT one small file -------------------------------------
|
||||||
|
|
||||||
|
def receive_file(upload_id, rel, stream, length):
|
||||||
|
"""Write one manifest file from an HTTP body. Temp name, hash check,
|
||||||
|
rename. Returns {"path", "sha256", "size"}."""
|
||||||
|
rel = norm_path(rel)
|
||||||
|
with exclusive("file upload"):
|
||||||
|
m, _ = load_manifest()
|
||||||
|
if not m:
|
||||||
|
raise UploadError("no manifest: POST /api/upload/verify first", 409)
|
||||||
|
if upload_id != m["upload_id"]:
|
||||||
|
raise UploadError("upload_id does not match the current manifest", 409)
|
||||||
|
entry = next((f for f in m["files"] if f["path"].lower() == rel.lower()), None)
|
||||||
|
if not entry:
|
||||||
|
raise UploadError(f"not in the manifest: {rel}", 404)
|
||||||
|
if entry["size"] > SMALL_MAX:
|
||||||
|
raise UploadError(f"{entry['path']} is over {SMALL_MAX // (1024 * 1024)} MB: re-copy it with robocopy", 413)
|
||||||
|
if length is None or length != entry["size"]:
|
||||||
|
raise UploadError(f"body is {length} bytes, manifest says {entry['size']}")
|
||||||
|
# Replace the file that is there, whatever its case; else create it,
|
||||||
|
# reusing existing folders case-insensitively (as Samba does).
|
||||||
|
on_disk = scan()[0].get(entry["path"].lower()) or []
|
||||||
|
if len(on_disk) > 1:
|
||||||
|
raise UploadError(f"two files differ only in case: {', '.join(on_disk)}; remove one", 409)
|
||||||
|
target_rel = on_disk[0] if on_disk else _resolve_dirs(entry["path"])
|
||||||
|
dest = safe_join(target_rel)
|
||||||
|
d = os.path.dirname(dest)
|
||||||
|
tmp = os.path.join(d, f".{os.path.basename(dest)}.part-{secrets.token_hex(8)}")
|
||||||
|
h, n = hashlib.sha256(), 0
|
||||||
|
try:
|
||||||
|
with open(tmp, "xb") as f:
|
||||||
|
while n <= entry["size"]:
|
||||||
|
b = stream.read(min(CHUNK, entry["size"] + 1 - n))
|
||||||
|
if not b:
|
||||||
|
break
|
||||||
|
n += len(b)
|
||||||
|
h.update(b)
|
||||||
|
f.write(b)
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
if n != entry["size"]:
|
||||||
|
raise UploadError(f"received {n} bytes, expected {entry['size']}")
|
||||||
|
if h.hexdigest() != entry["sha256"]:
|
||||||
|
raise UploadError(f"hash of the received {entry['path']} does not match the manifest", 422)
|
||||||
|
if os.path.isdir(dest) and not os.path.islink(dest):
|
||||||
|
raise UploadError(f"a folder is in the way: {target_rel}", 409)
|
||||||
|
_chown_like_parent(tmp)
|
||||||
|
os.chmod(tmp, 0o664)
|
||||||
|
os.replace(tmp, dest)
|
||||||
|
finally:
|
||||||
|
if os.path.exists(tmp):
|
||||||
|
os.remove(tmp)
|
||||||
|
return {"path": entry["path"], "written_as": target_rel, "sha256": entry["sha256"], "size": n}
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_dirs(rel):
|
||||||
|
"""Map rel's folders onto existing ones case-insensitively, creating
|
||||||
|
what is missing (refusing links). Returns the path to write."""
|
||||||
|
cur = os.path.realpath(root())
|
||||||
|
out = []
|
||||||
|
for part in rel.split("/")[:-1]:
|
||||||
|
hit = None
|
||||||
|
try:
|
||||||
|
for name in os.listdir(cur):
|
||||||
|
if name.lower() == part.lower():
|
||||||
|
hit = name
|
||||||
|
break
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
name = hit or part
|
||||||
|
nxt = os.path.join(cur, name)
|
||||||
|
if os.path.islink(nxt):
|
||||||
|
raise UploadError(f"link in path: {rel}")
|
||||||
|
if not os.path.isdir(nxt):
|
||||||
|
os.mkdir(nxt)
|
||||||
|
_chown_like_parent(nxt)
|
||||||
|
cur = nxt
|
||||||
|
out.append(name)
|
||||||
|
out.append(rel.split("/")[-1])
|
||||||
|
return "/".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
# --- import gate ------------------------------------------------------------
|
||||||
|
|
||||||
|
def gate(target=None, locked=False):
|
||||||
|
"""(ok, reason, info) for importing the upload folder into target.
|
||||||
|
info carries what the import page shows. Pass locked=True when the
|
||||||
|
caller already holds the lock (the import itself)."""
|
||||||
|
m, msha = load_manifest()
|
||||||
|
info = {"manifest": None, "state": public_state()}
|
||||||
|
if not m:
|
||||||
|
return False, ("No upload manifest. Upload with scripts\\Upload-Image.ps1, which "
|
||||||
|
"checks every file; files copied by hand cannot be verified."), info
|
||||||
|
info["manifest"] = {k: m.get(k) for k in ("upload_id", "created", "source_pc", "mcl")}
|
||||||
|
info["manifest"]["files"] = len(m["files"])
|
||||||
|
info["manifest"]["bytes"] = sum(f["size"] for f in m["files"])
|
||||||
|
s = load_state()
|
||||||
|
if (not locked and _lock.locked()) or (s and s.get("state") == "running"):
|
||||||
|
return False, "A verification, file upload or import is running.", info
|
||||||
|
if not s or s.get("state") != "done":
|
||||||
|
return False, "This upload has not been verified yet (run Upload-Image.ps1 to the end).", info
|
||||||
|
if s.get("upload_id") != m["upload_id"] or s.get("manifest_sha") != msha:
|
||||||
|
return False, "The manifest changed after the last verification; verify again.", info
|
||||||
|
if not s.get("ok"):
|
||||||
|
n = len(s.get("missing", [])) + len(s.get("mismatch", [])) + len(s.get("bad_in_cache", []))
|
||||||
|
return False, f"The last verification found {n} bad file(s).", info
|
||||||
|
det = load_detail()
|
||||||
|
if det.get("upload_id") != m["upload_id"] or det.get("manifest_sha") != msha:
|
||||||
|
return False, "Verification details are missing; verify again.", info
|
||||||
|
snap = det.get("snapshot") or {}
|
||||||
|
if len(snap) != len(m["files"]):
|
||||||
|
return False, "Verification details are incomplete; verify again.", info
|
||||||
|
for rel, key in snap.items():
|
||||||
|
try:
|
||||||
|
st = os.stat(safe_join(rel))
|
||||||
|
except (OSError, UploadError):
|
||||||
|
return False, f"{rel} is gone since verification; verify again.", info
|
||||||
|
if _key(st) != key:
|
||||||
|
return False, f"{rel} changed since verification; verify again.", info
|
||||||
|
ext = extras(m)
|
||||||
|
info["extra"] = ext[:50]
|
||||||
|
info["extra_count"] = len(ext)
|
||||||
|
if ext:
|
||||||
|
return False, (f"{len(ext)} file(s) in the upload folder are not part of this upload "
|
||||||
|
"(left from an earlier one). Remove them first."), info
|
||||||
|
allowed = s.get("allowed_targets") or []
|
||||||
|
info["allowed_targets"] = allowed
|
||||||
|
info["os"] = s.get("os")
|
||||||
|
if not allowed:
|
||||||
|
return False, "Cannot tell which OS this upload carries (OperatingSystem.json), so it has no safe target.", info
|
||||||
|
if target is not None and target not in allowed:
|
||||||
|
return False, f"This upload carries {(s.get('os') or {}).get('name', 'an unknown OS')}; it can only go to: {', '.join(allowed)}.", info
|
||||||
|
return True, "", info
|
||||||
|
|
||||||
|
|
||||||
|
def remove_extras():
|
||||||
|
"""Delete files and links in the upload folder that are not in the
|
||||||
|
manifest. Returns how many were removed."""
|
||||||
|
with exclusive("remove"):
|
||||||
|
m, _ = load_manifest()
|
||||||
|
if not m:
|
||||||
|
raise UploadError("no manifest")
|
||||||
|
n = 0
|
||||||
|
for rel in extras(m):
|
||||||
|
p = os.path.join(root(), *rel.split("/"))
|
||||||
|
if os.path.islink(p) or os.path.isfile(p):
|
||||||
|
os.remove(p)
|
||||||
|
n += 1
|
||||||
|
for dp, _dns, _fns in os.walk(root(), topdown=False):
|
||||||
|
if dp != root():
|
||||||
|
try:
|
||||||
|
os.rmdir(dp)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
def import_record(target):
|
||||||
|
"""What gets stored on the image after a verified import."""
|
||||||
|
m, _ = load_manifest()
|
||||||
|
s = load_state() or {}
|
||||||
|
return {"upload_id": m["upload_id"] if m else None, "imported": _now(), "target": target,
|
||||||
|
"source_pc": (m or {}).get("source_pc"), "mcl": (m or {}).get("mcl"),
|
||||||
|
"os": s.get("os"), "files": len(m["files"]) if m else 0}
|
||||||
|
|
||||||
|
|
||||||
|
def finish_import(image_root, target):
|
||||||
|
"""After the import moved the files: keep a record on the image, then
|
||||||
|
clear the manifest and state so the folder starts fresh. Call with
|
||||||
|
the lock held."""
|
||||||
|
rec = import_record(target)
|
||||||
|
try:
|
||||||
|
_write_json(os.path.join(image_root, ".mcl-import.json"), rec)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
for name in TOP_FILES:
|
||||||
|
try:
|
||||||
|
os.remove(os.path.join(root(), name))
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
return rec
|
||||||
|
|
||||||
|
|
||||||
|
def last_import(image_root):
|
||||||
|
try:
|
||||||
|
return _read_json(os.path.join(image_root, ".mcl-import.json"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def try_lock():
|
||||||
|
"""For the import route: take the lock without waiting (True if held)."""
|
||||||
|
return _lock.acquire(blocking=False)
|
||||||
|
|
||||||
|
|
||||||
|
def unlock():
|
||||||
|
_lock.release()
|
||||||
@@ -21,6 +21,12 @@
|
|||||||
.imp-actions { display: flex; flex-wrap: wrap; gap: 10px; align-items: center; }
|
.imp-actions { display: flex; flex-wrap: wrap; gap: 10px; align-items: center; }
|
||||||
.imp-empty { display: flex; flex-direction: column; gap: 10px; align-items: flex-start; font-size: 14px; color: var(--text-2); line-height: 1.6; }
|
.imp-empty { display: flex; flex-direction: column; gap: 10px; align-items: flex-start; font-size: 14px; color: var(--text-2); line-height: 1.6; }
|
||||||
.imp-empty p { margin: 0; }
|
.imp-empty p { margin: 0; }
|
||||||
|
.upv-rows { display: flex; flex-direction: column; gap: 6px; font-size: 14px; }
|
||||||
|
.upv-row { display: grid; grid-template-columns: 92px minmax(0, 1fr); gap: 10px; align-items: center; }
|
||||||
|
.upv-row .k { color: var(--muted); font-size: 13px; }
|
||||||
|
.upv-bad summary { cursor: pointer; font-size: 13px; font-weight: 600; }
|
||||||
|
.upv-bad ul { margin: 8px 0 0; padding-left: 18px; display: flex; flex-direction: column; gap: 3px; font-size: 13px; }
|
||||||
|
.upv-bad li { min-width: 0; }
|
||||||
|
|
||||||
.ig { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.3fr) 92px 112px auto; align-items: center; gap: 0 14px;
|
.ig { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.3fr) 92px 112px auto; align-items: center; gap: 0 14px;
|
||||||
padding: 11px 24px; font-size: 14px; border-top: 1px solid var(--line); }
|
padding: 11px 24px; font-size: 14px; border-top: 1px solid var(--line); }
|
||||||
@@ -50,40 +56,81 @@
|
|||||||
<section class="panel imp-panel" aria-labelledby="imp-up-h">
|
<section class="panel imp-panel" aria-labelledby="imp-up-h">
|
||||||
<div>
|
<div>
|
||||||
<h2 id="imp-up-h">From the network upload share</h2>
|
<h2 id="imp-up-h">From the network upload share</h2>
|
||||||
<p class="lead">Files copied over SMB to <span class="mono">\\172.16.9.1\image-upload</span>. They are moved, not copied.</p>
|
<p class="lead">Uploaded by <span class="mono">Upload-Image.ps1</span> to <span class="mono">\\172.16.9.1\image-upload</span> and checked file by file. They are moved, not copied.</p>
|
||||||
</div>
|
</div>
|
||||||
{% if upload_sources %}
|
{% set man = upload_info.manifest %}
|
||||||
|
{% set st = upload_info.state %}
|
||||||
|
{% if man %}
|
||||||
|
<div class="upv-rows">
|
||||||
|
<div class="upv-row"><span class="k">Upload</span><span class="tip" data-tip="{{ man.upload_id }}{% if man.created %} created {{ man.created }}{% endif %}"><span class="trunc mono">{{ man.upload_id }}</span></span></div>
|
||||||
|
{% if man.mcl %}
|
||||||
|
<div class="upv-row"><span class="k">MCL cache</span><span class="tip" data-tip="{{ man.mcl.tag or '' }}"><span class="trunc">{{ man.mcl.os_name or 'Unknown OS' }}{% if man.mcl.os_version %} {{ man.mcl.os_version }}{% endif %}, finished {{ (man.mcl.finished or '?')|replace('T', ' ') }}</span></span></div>
|
||||||
|
{% endif %}
|
||||||
|
<div class="upv-row"><span class="k">Files</span><span>{{ man.files }} ({{ man.bytes|filesizeformat(true) }})</span></div>
|
||||||
|
<div class="upv-row"><span class="k">Verified</span><span>
|
||||||
|
{% if st.state == 'running' %}{{ ui.chip('Checking ' ~ st.done_files ~ '/' ~ st.total_files, 'warn') }}
|
||||||
|
{% elif st.state == 'done' and st.ok %}{{ ui.chip('Every file matches', 'ok') }} <span class="sub">{{ (st.finished or '')|replace('T', ' ') }}</span>
|
||||||
|
{% elif st.state == 'done' %}{{ ui.chip((st.missing|length + st.mismatch|length + st.bad_in_cache|length) ~ ' bad file(s)', 'err') }}
|
||||||
|
{% elif st.state == 'error' %}{{ ui.chip('Check failed', 'err') }} <span class="sub">{{ st.error }}</span>
|
||||||
|
{% else %}{{ ui.chip('Not verified', 'off') }}{% endif %}
|
||||||
|
</span></div>
|
||||||
|
</div>
|
||||||
|
{% if st.state == 'done' and not st.ok %}
|
||||||
|
<details class="upv-bad"><summary>Bad files</summary>
|
||||||
|
<ul>
|
||||||
|
{% for f in st.missing %}<li class="tip" data-tip="{{ f }}"><span class="trunc mono">Missing: {{ f }}</span></li>{% endfor %}
|
||||||
|
{% for f in st.mismatch %}<li class="tip" data-tip="{{ f.path }}: {{ f.reason }}"><span class="trunc mono">Differs: {{ f.path }}</span></li>{% endfor %}
|
||||||
|
{% for f in st.bad_in_cache %}<li class="tip" data-tip="{{ f.path }}: {{ f.reason }}"><span class="trunc mono">Bad in MCL cache: {{ f.path }}</span></li>{% endfor %}
|
||||||
|
</ul>
|
||||||
|
<p class="sub">Run Upload-Image.ps1 again: it re-sends the files that differ. Files bad in the MCL cache must be downloaded again by MCL.</p>
|
||||||
|
</details>
|
||||||
|
{% endif %}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if upload_info.extra_count %}
|
||||||
|
<div class="notice"><span><strong>{{ upload_info.extra_count }} file(s) in the upload folder are not part of this upload</strong>, left from an earlier one. They would be imported with it.</span></div>
|
||||||
|
<details class="upv-bad"><summary>Show them</summary><ul>
|
||||||
|
{% for f in upload_info.extra %}<li class="tip" data-tip="{{ f }}"><span class="trunc mono">{{ f }}</span></li>{% endfor %}
|
||||||
|
{% if upload_info.extra_count > upload_info.extra|length %}<li class="sub">and {{ upload_info.extra_count - upload_info.extra|length }} more</li>{% endif %}
|
||||||
|
</ul></details>
|
||||||
|
<form method="POST" action="{{ url_for('images_import_remove_extras') }}" onsubmit="return confirm('Delete the {{ upload_info.extra_count }} file(s) that are not in this upload\'s manifest?');">
|
||||||
|
<input type="hidden" name="_csrf_token" value="{{ csrf_token() }}">
|
||||||
|
<button type="submit" class="ui-btn">Remove files not in this upload</button>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if upload_ok %}
|
||||||
<form method="POST" id="uploadImportForm">
|
<form method="POST" id="uploadImportForm">
|
||||||
<input type="hidden" name="_csrf_token" value="{{ csrf_token() }}">
|
<input type="hidden" name="_csrf_token" value="{{ csrf_token() }}">
|
||||||
<div class="imp-field">
|
<input type="hidden" name="source" value="{{ upload_dir }}">
|
||||||
<label for="uploadSource">Source</label>
|
|
||||||
<select class="field" name="source" id="uploadSource" required>
|
|
||||||
<option value="">Select an upload folder</option>
|
|
||||||
{% for src in upload_sources %}
|
|
||||||
<option value="{{ src }}">{{ src }}</option>
|
|
||||||
{% endfor %}
|
|
||||||
</select>
|
|
||||||
</div>
|
|
||||||
<div class="imp-field">
|
<div class="imp-field">
|
||||||
<label for="uploadTarget">Target image</label>
|
<label for="uploadTarget">Target image</label>
|
||||||
|
{% set allowed = upload_info.allowed_targets %}
|
||||||
<select class="field" name="target" id="uploadTarget" required>
|
<select class="field" name="target" id="uploadTarget" required>
|
||||||
<option value="">Select the image to import into</option>
|
{% if allowed|length > 1 %}<option value="">Select the image to import into</option>{% endif %}
|
||||||
{% for it in image_types %}
|
{% for it in allowed %}
|
||||||
<option value="{{ it }}">{{ friendly_names[it] }} ({{ it }})</option>
|
{% set li = last_imports.get(it) %}
|
||||||
|
{% set older = li and li.mcl and man.mcl and li.mcl.finished and man.mcl.finished and man.mcl.finished < li.mcl.finished %}
|
||||||
|
<option value="{{ it }}" data-older="{{ '1' if older else '' }}" data-last="{{ (li.mcl.finished if li and li.mcl else '')|replace('T', ' ') }}">{{ friendly_names[it] }} ({{ it }})</option>
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
</select>
|
</select>
|
||||||
<span class="sub">Content goes into the image's Deploy folder. Out-of-box Drivers are pooled once in a shared folder and linked from each image to save disk space.</span>
|
<span class="sub">{% if upload_info.os %}This upload carries {{ upload_info.os.name }}{% if upload_info.os.version %} {{ upload_info.os.version }}{% endif %}, so it can only go to the image(s) listed.{% endif %}
|
||||||
|
Out-of-box Drivers are pooled once in a shared folder and linked from each image.</span>
|
||||||
</div>
|
</div>
|
||||||
|
<div class="notice" id="olderWarn" hidden><span><strong>This MCL cache is older than the one last imported into that image</strong> (<span id="olderWhen"></span>).</span></div>
|
||||||
<div class="notice"><span><strong>The files are moved out of the upload folder</strong>, so the source is used up.
|
<div class="notice"><span><strong>The files are moved out of the upload folder</strong>, so the source is used up.
|
||||||
Files with the same name in the image are overwritten. Large images take several minutes.</span></div>
|
Files with the same name in the image are overwritten. Large images take several minutes.</span></div>
|
||||||
<div class="imp-actions">
|
<div class="imp-actions">
|
||||||
<button type="submit" class="ui-btn pri" id="uploadImportBtn">Start import</button>
|
<button type="submit" class="ui-btn pri" id="uploadImportBtn">Start import</button>
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
|
{% elif upload_sources %}
|
||||||
|
<div class="notice"><span><strong>Import blocked:</strong> {{ upload_reason }}</span></div>
|
||||||
|
<div class="imp-actions"><button type="button" class="ui-btn" onclick="location.reload()">Refresh</button></div>
|
||||||
{% else %}
|
{% else %}
|
||||||
<div class="imp-empty">
|
<div class="imp-empty">
|
||||||
{{ ui.chip('Nothing uploaded', 'off') }}
|
{{ ui.chip('Nothing uploaded', 'off') }}
|
||||||
<p>Map <span class="mono">\\172.16.9.1\image-upload</span> on your Windows PC and copy the Deploy folder contents there, then refresh.</p>
|
<p>On the PC with the Media Creator Lite cache, run <span class="mono">scripts\Upload-Image.ps1</span>. It copies the cache here and checks every file.</p>
|
||||||
<button type="button" class="ui-btn" onclick="location.reload()">Refresh</button>
|
<button type="button" class="ui-btn" onclick="location.reload()">Refresh</button>
|
||||||
</div>
|
</div>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
@@ -152,6 +199,15 @@
|
|||||||
{% block extra_scripts %}
|
{% block extra_scripts %}
|
||||||
<script>
|
<script>
|
||||||
document.addEventListener('DOMContentLoaded', function() {
|
document.addEventListener('DOMContentLoaded', function() {
|
||||||
|
var tsel = document.getElementById('uploadTarget');
|
||||||
|
function olderCheck() {
|
||||||
|
var w = document.getElementById('olderWarn');
|
||||||
|
if (!tsel || !w) return;
|
||||||
|
var o = tsel.options[tsel.selectedIndex];
|
||||||
|
w.hidden = !(o && o.dataset.older);
|
||||||
|
document.getElementById('olderWhen').textContent = o && o.dataset.last ? 'last import finished ' + o.dataset.last : '';
|
||||||
|
}
|
||||||
|
if (tsel) { tsel.addEventListener('change', olderCheck); olderCheck(); }
|
||||||
['importForm', 'uploadImportForm'].forEach(function(formId) {
|
['importForm', 'uploadImportForm'].forEach(function(formId) {
|
||||||
var form = document.getElementById(formId);
|
var form = document.getElementById(formId);
|
||||||
if (form) {
|
if (form) {
|
||||||
|
|||||||
521
webapp/tests/test_upload_verify.py
Normal file
521
webapp/tests/test_upload_verify.py
Normal file
@@ -0,0 +1,521 @@
|
|||||||
|
"""Verified MCL uploads: manifest checks, server-side hashing, the HTTP
|
||||||
|
second channel, and the import gate.
|
||||||
|
|
||||||
|
Run from the webapp folder:
|
||||||
|
PYTHONPATH=.:<deps> python -m unittest tests.test_upload_verify -v
|
||||||
|
|
||||||
|
Every config path points into a temp folder, so nothing on the host is
|
||||||
|
touched. The planted failures are the real ones: a 0-byte Control JSON,
|
||||||
|
a file with the right size and the wrong content (the September MCL
|
||||||
|
cache corruption), a file SMB never delivered, and a symlink escape.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import unittest
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
_T = tempfile.mkdtemp(prefix="pxe-upv-test-")
|
||||||
|
for _k, _sub in {"SAMBA_SHARE": "samba", "CLONEZILLA_SHARE": "cz", "BLANCCO_REPORTS": "br",
|
||||||
|
"ENROLLMENT_SHARE": "enr", "WEB_ROOT": "www", "IMAGING_DIR": "img", "UPDATES_DIR": "upd",
|
||||||
|
"TFTP_ROOT": "tftp", "BIOS_DIR": "bios", "UPLOAD_DIR": "upl", "SAMBA_LOG_DIR": "smblog"}.items():
|
||||||
|
os.environ.setdefault(_k, os.path.join(_T, _sub))
|
||||||
|
os.makedirs(os.environ[_k], exist_ok=True)
|
||||||
|
for _k in ("AUDIT_LOG", "HARDWARE_STATE", "BLANCCO_STATE", "DNSMASQ_LEASES", "APACHE_ACCESS_LOG",
|
||||||
|
"DNSMASQ_SYSLOG", "NIC_STATUS_FILE", "NIC_CONF", "DNSMASQ_NIC_CONF", "DNSMASQ_CONF", "CLONEZILLA_STATE"):
|
||||||
|
os.environ.setdefault(_k, os.path.join(_T, _k.lower()))
|
||||||
|
os.makedirs(os.path.join(os.environ["ENROLLMENT_SHARE"], "ppkgs"), exist_ok=True)
|
||||||
|
|
||||||
|
WEBAPP = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
sys.path.insert(0, WEBAPP)
|
||||||
|
os.chdir(WEBAPP)
|
||||||
|
|
||||||
|
import config # noqa: E402
|
||||||
|
import app as A # noqa: E402
|
||||||
|
from services import upload_verify as UV # noqa: E402
|
||||||
|
|
||||||
|
WIM = b"W" * 4096
|
||||||
|
WIM_DIR = "Deploy/Operating Systems/Windows 11.0.26100.0 Pro x64/September 9, 2025"
|
||||||
|
|
||||||
|
|
||||||
|
def sha(b):
|
||||||
|
return hashlib.sha256(b).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def os_json(product="Windows 11 Professional", wim=WIM):
|
||||||
|
return json.dumps([{"operatingSystemVersion": {
|
||||||
|
"productName": product, "versionNumber": "24H2", "buildNumber": "26100", "id": 21,
|
||||||
|
"wim": {"hash": sha(wim).upper(), "size": len(wim),
|
||||||
|
"destinationDir": "*destinationdir*\\" + WIM_DIR.replace("/", "\\")}}}]).encode()
|
||||||
|
|
||||||
|
|
||||||
|
class Base(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.tmp = tempfile.mkdtemp(prefix="upv-", dir=_T)
|
||||||
|
self.up = os.path.join(self.tmp, "upload")
|
||||||
|
self.share = os.path.join(self.tmp, "samba")
|
||||||
|
os.makedirs(self.up)
|
||||||
|
os.makedirs(self.share)
|
||||||
|
self.patches = [mock.patch.object(config, "UPLOAD_DIR", self.up),
|
||||||
|
mock.patch.object(config, "SAMBA_SHARE", self.share),
|
||||||
|
mock.patch.object(config, "SHARED_DIR", os.path.join(self.share, "_shared"))]
|
||||||
|
for p in self.patches:
|
||||||
|
p.start()
|
||||||
|
self.files = {}
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
for p in self.patches:
|
||||||
|
p.stop()
|
||||||
|
shutil.rmtree(self.tmp, ignore_errors=True)
|
||||||
|
|
||||||
|
def put(self, rel, data, on_server=True):
|
||||||
|
"""data = what the PC has; it goes into the manifest, and (unless
|
||||||
|
on_server is False) onto the server too."""
|
||||||
|
self.files[rel] = data
|
||||||
|
if on_server:
|
||||||
|
p = os.path.join(self.up, *rel.split("/"))
|
||||||
|
os.makedirs(os.path.dirname(p), exist_ok=True)
|
||||||
|
with open(p, "wb") as f:
|
||||||
|
f.write(data)
|
||||||
|
|
||||||
|
def standard_upload(self, product="Windows 11 Professional"):
|
||||||
|
self.put("Deploy/Control/OperatingSystem.json", os_json(product))
|
||||||
|
self.put("Deploy/Control/packages.json", b"[]")
|
||||||
|
self.put("Tools/user_selections.json", b'[{"OperatingSystemSelection": "21"}]')
|
||||||
|
self.put(f"{WIM_DIR}/install.wim", WIM)
|
||||||
|
|
||||||
|
def manifest(self, uid="20261016-1200-PC1", **extra):
|
||||||
|
m = {"manifest_version": 1, "upload_id": uid, "source_pc": "PC1",
|
||||||
|
"mcl": {"os_name": "Windows 11 Professional", "finished": "2026-10-16T11:00:00"},
|
||||||
|
"files": [{"path": r, "size": len(d), "sha256": sha(d)} for r, d in self.files.items()]}
|
||||||
|
m.update(extra)
|
||||||
|
return m
|
||||||
|
|
||||||
|
def verify(self, m=None):
|
||||||
|
"""Synchronous verify the way the POST stores it."""
|
||||||
|
m = m or self.manifest()
|
||||||
|
raw = json.dumps(m, indent=1).encode()
|
||||||
|
with open(os.path.join(self.up, UV.MANIFEST), "wb") as f:
|
||||||
|
f.write(raw)
|
||||||
|
return UV.run_verify(UV.parse_manifest(m), hashlib.sha256(raw).hexdigest())
|
||||||
|
|
||||||
|
|
||||||
|
class ManifestTests(Base):
|
||||||
|
def test_bad_paths_rejected(self):
|
||||||
|
for p in ["../x", "a/../../x", "/etc/passwd", "C:/Windows/x", "C:\\x", "a//b", "./a",
|
||||||
|
"a\\..\\..\\b", UV.MANIFEST, UV.STATE, "a/.x.json.part-0123456789abcdef", "", "a\x00b"]:
|
||||||
|
with self.assertRaises(UV.UploadError, msg=p):
|
||||||
|
UV.norm_path(p)
|
||||||
|
|
||||||
|
def test_backslashes_normalised(self):
|
||||||
|
self.assertEqual(UV.norm_path("Deploy\\Control\\os.json"), "Deploy/Control/os.json")
|
||||||
|
|
||||||
|
def test_bad_manifests_rejected(self):
|
||||||
|
ok = {"path": "a.json", "size": 1, "sha256": "0" * 64}
|
||||||
|
for m in [None, [], {"manifest_version": 2, "upload_id": "x", "files": [ok]},
|
||||||
|
{"manifest_version": 1, "upload_id": "../x", "files": [ok]},
|
||||||
|
{"manifest_version": 1, "upload_id": "x", "files": []},
|
||||||
|
{"manifest_version": 1, "upload_id": "x", "files": [dict(ok, sha256="zz")]},
|
||||||
|
{"manifest_version": 1, "upload_id": "x", "files": [dict(ok, size=-1)]},
|
||||||
|
{"manifest_version": 1, "upload_id": "x", "files": [dict(ok, size=True)]},
|
||||||
|
{"manifest_version": 1, "upload_id": "x", "files": [ok, dict(ok, path="A.JSON")]}]:
|
||||||
|
with self.assertRaises(UV.UploadError, msg=repr(m)):
|
||||||
|
UV.parse_manifest(m)
|
||||||
|
|
||||||
|
def test_mcl_rel_mapping(self):
|
||||||
|
self.assertEqual(UV._mcl_rel("*destinationdir*\\Deploy\\Packages\\X", "a.msu"), "Deploy/Packages/X/a.msu")
|
||||||
|
self.assertEqual(UV._mcl_rel("C:\\Deploy\\Languages\\Y", "b.cab"), "Deploy/Languages/Y/b.cab")
|
||||||
|
|
||||||
|
|
||||||
|
class VerifyTests(Base):
|
||||||
|
def test_good_upload_passes(self):
|
||||||
|
self.standard_upload()
|
||||||
|
r = self.verify()
|
||||||
|
self.assertTrue(r["ok"], r)
|
||||||
|
self.assertTrue(r["importable"])
|
||||||
|
self.assertEqual(r["allowed_targets"], ["gea-standard", "gea-engineer"])
|
||||||
|
self.assertEqual(len(UV.load_detail()["snapshot"]), 4)
|
||||||
|
|
||||||
|
def test_ltsc_is_shopfloor_only(self):
|
||||||
|
self.standard_upload(product="Windows 11 LTSC")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual(r["allowed_targets"], ["gea-shopfloor"])
|
||||||
|
|
||||||
|
def test_missing_file_caught(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.put("Deploy/Control/HardwareDriver.json", b"[]", on_server=False)
|
||||||
|
r = self.verify()
|
||||||
|
self.assertFalse(r["ok"])
|
||||||
|
self.assertEqual(r["missing"], ["Deploy/Control/HardwareDriver.json"])
|
||||||
|
|
||||||
|
def test_zero_byte_json_on_server_caught(self):
|
||||||
|
# SMB left the file at 0 bytes: size differs, so it is a mismatch.
|
||||||
|
self.standard_upload()
|
||||||
|
open(os.path.join(self.up, "Deploy", "Control", "packages.json"), "wb").close()
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual([m["path"] for m in r["mismatch"]], ["Deploy/Control/packages.json"])
|
||||||
|
|
||||||
|
def test_same_size_wrong_content_caught(self):
|
||||||
|
# The case the old size-only check could never see.
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, "Deploy", "Control", "packages.json"), "wb") as f:
|
||||||
|
f.write(b"{}")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual(r["mismatch"], [{"path": "Deploy/Control/packages.json", "reason": "content differs"}])
|
||||||
|
|
||||||
|
def test_zero_byte_json_in_cache_caught(self):
|
||||||
|
# Broken at the source: the PC's own copy is empty, so the hashes
|
||||||
|
# agree, but an empty Control JSON is never right.
|
||||||
|
self.standard_upload()
|
||||||
|
self.put("Deploy/Control/boot.json", b"")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertFalse(r["ok"])
|
||||||
|
self.assertEqual(r["bad_in_cache"], [{"path": "Deploy/Control/boot.json", "reason": "empty Control JSON"}])
|
||||||
|
|
||||||
|
def test_september_pattern_caught(self):
|
||||||
|
# MCL resumed onto an older file: right size, junk content, on the
|
||||||
|
# PC. Manifest and server agree; MCL's published hash does not.
|
||||||
|
self.standard_upload()
|
||||||
|
self.put(f"{WIM_DIR}/install.wim", b"X" * 100 + WIM[100:])
|
||||||
|
r = self.verify()
|
||||||
|
self.assertFalse(r["ok"])
|
||||||
|
self.assertEqual([b["path"] for b in r["bad_in_cache"]], [f"{WIM_DIR}/install.wim"])
|
||||||
|
self.assertEqual(r["mismatch"], [])
|
||||||
|
|
||||||
|
def test_mcl_path_case_insensitive(self):
|
||||||
|
# Windows paths ignore case; MCL's JSON and the folder may differ.
|
||||||
|
self.standard_upload()
|
||||||
|
del self.files[f"{WIM_DIR}/install.wim"]
|
||||||
|
os.remove(os.path.join(self.up, *WIM_DIR.split("/"), "install.wim"))
|
||||||
|
self.put(f"{WIM_DIR}/INSTALL.WIM", b"Z" * len(WIM))
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual([b["path"] for b in r["bad_in_cache"]], [f"{WIM_DIR}/INSTALL.WIM"])
|
||||||
|
|
||||||
|
def test_extras_block_import_not_verify(self):
|
||||||
|
self.standard_upload()
|
||||||
|
os.makedirs(os.path.join(self.up, "Deploy", "Old"))
|
||||||
|
with open(os.path.join(self.up, "Deploy", "Old", "stale.json"), "w") as f:
|
||||||
|
f.write("{}")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertTrue(r["ok"])
|
||||||
|
self.assertFalse(r["importable"])
|
||||||
|
self.assertEqual(r["extra"], ["Deploy/Old/stale.json"])
|
||||||
|
|
||||||
|
def test_symlink_escape_refused(self):
|
||||||
|
self.standard_upload()
|
||||||
|
outside = os.path.join(self.tmp, "outside")
|
||||||
|
os.makedirs(outside)
|
||||||
|
with open(os.path.join(outside, "x.json"), "wb") as f:
|
||||||
|
f.write(b"[]")
|
||||||
|
os.symlink(outside, os.path.join(self.up, "Evil"))
|
||||||
|
self.put("Evil/x.json", b"[]", on_server=False)
|
||||||
|
r = self.verify()
|
||||||
|
self.assertIn("Evil/x.json", r["missing"])
|
||||||
|
self.assertIn("Evil", r["extra"]) # the link itself blocks the import
|
||||||
|
|
||||||
|
def test_second_round_reuses_hashes(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.verify()
|
||||||
|
with mock.patch.object(UV, "_sha256", side_effect=AssertionError("rehashed")):
|
||||||
|
r = self.verify()
|
||||||
|
self.assertTrue(r["ok"])
|
||||||
|
|
||||||
|
def rewrite_keep_mtime(self, rel, data):
|
||||||
|
"""What robocopy does: new content, source mtime kept."""
|
||||||
|
p = os.path.join(self.up, *rel.split("/"))
|
||||||
|
st = os.stat(p)
|
||||||
|
time.sleep(0.01)
|
||||||
|
with open(p, "wb") as f:
|
||||||
|
f.write(data)
|
||||||
|
os.utime(p, ns=(st.st_atime_ns, st.st_mtime_ns))
|
||||||
|
|
||||||
|
def test_bad_copy_then_good_recopy_same_mtime(self):
|
||||||
|
# Review finding 2: a re-copied large file must be hashed again even
|
||||||
|
# though robocopy gives it the same size and mtime as the bad copy.
|
||||||
|
self.standard_upload()
|
||||||
|
wim_rel = f"{WIM_DIR}/install.wim"
|
||||||
|
self.rewrite_keep_mtime(wim_rel, b"Q" * len(WIM))
|
||||||
|
self.assertEqual([m["path"] for m in self.verify()["mismatch"]], [wim_rel])
|
||||||
|
self.rewrite_keep_mtime(wim_rel, WIM)
|
||||||
|
self.assertTrue(self.verify()["ok"])
|
||||||
|
|
||||||
|
def test_good_file_corrupted_in_place_same_mtime(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.assertTrue(self.verify()["ok"])
|
||||||
|
self.rewrite_keep_mtime("Deploy/Control/packages.json", b"{}")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual([m["path"] for m in r["mismatch"]], ["Deploy/Control/packages.json"])
|
||||||
|
|
||||||
|
def test_os_picked_by_selected_id(self):
|
||||||
|
# Review finding 3: LTSC listed first, Pro (21) selected.
|
||||||
|
self.standard_upload()
|
||||||
|
ltsc = json.loads(os_json("Windows 11 LTSC"))[0]
|
||||||
|
ltsc["operatingSystemVersion"]["id"] = 20
|
||||||
|
pro = json.loads(os_json())[0]
|
||||||
|
self.put("Deploy/Control/OperatingSystem.json", json.dumps([ltsc, pro]).encode())
|
||||||
|
r = self.verify(self.manifest(mcl={"os_id": 21}))
|
||||||
|
self.assertEqual(r["allowed_targets"], ["gea-standard", "gea-engineer"])
|
||||||
|
r = self.verify(self.manifest(mcl={"os_id": 20}))
|
||||||
|
self.assertEqual(r["allowed_targets"], ["gea-shopfloor"])
|
||||||
|
|
||||||
|
def test_unknown_os_fails_closed(self):
|
||||||
|
# Review finding 4: valid JSON, wrong shape.
|
||||||
|
self.standard_upload()
|
||||||
|
self.put("Deploy/Control/OperatingSystem.json", b'{"x": 1}')
|
||||||
|
r = self.verify()
|
||||||
|
self.assertTrue(r["ok"])
|
||||||
|
self.assertEqual(r["allowed_targets"], [])
|
||||||
|
self.assertFalse(r["importable"])
|
||||||
|
ok, reason, _ = UV.gate("gea-standard")
|
||||||
|
self.assertFalse(ok)
|
||||||
|
self.assertIn("Cannot tell which OS", reason)
|
||||||
|
|
||||||
|
def test_nested_part_file_is_stray(self):
|
||||||
|
# Review finding 5: debris from a killed PUT must not ride along.
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, "Deploy", "Control", ".x.msu.part-0123456789abcdef"), "wb") as f:
|
||||||
|
f.write(b"junk")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertEqual(r["extra"], ["Deploy/Control/.x.msu.part-0123456789abcdef"])
|
||||||
|
self.assertEqual(UV.remove_extras(), 1)
|
||||||
|
|
||||||
|
def test_case_differs_from_manifest(self):
|
||||||
|
# Review finding 8: manifest says packages.json, disk has Packages.json.
|
||||||
|
self.standard_upload()
|
||||||
|
os.rename(os.path.join(self.up, "Deploy", "Control", "packages.json"),
|
||||||
|
os.path.join(self.up, "Deploy", "Control", "Packages.json"))
|
||||||
|
r = self.verify()
|
||||||
|
self.assertTrue(r["ok"], r)
|
||||||
|
self.assertEqual(r["extra"], [])
|
||||||
|
self.assertTrue(UV.gate("gea-standard")[0])
|
||||||
|
|
||||||
|
def test_two_cases_on_disk(self):
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, "Deploy", "Control", "PACKAGES.json"), "wb") as f:
|
||||||
|
f.write(b"[]")
|
||||||
|
r = self.verify()
|
||||||
|
self.assertFalse(r["ok"])
|
||||||
|
self.assertIn("differ only in case", r["mismatch"][0]["reason"])
|
||||||
|
|
||||||
|
|
||||||
|
class ReceiveTests(Base):
|
||||||
|
def setUp(self):
|
||||||
|
super().setUp()
|
||||||
|
self.standard_upload()
|
||||||
|
self.put("Deploy/Control/HardwareDriver.json", b'[{"x": 1}]', on_server=False)
|
||||||
|
self.m = self.manifest()
|
||||||
|
self.verify(self.m)
|
||||||
|
self.uid = self.m["upload_id"]
|
||||||
|
|
||||||
|
def send(self, rel, data, uid=None, length=None):
|
||||||
|
return UV.receive_file(uid or self.uid, rel, io.BytesIO(data), len(data) if length is None else length)
|
||||||
|
|
||||||
|
def test_resend_repairs_and_verify_passes(self):
|
||||||
|
r = self.send("Deploy/Control/HardwareDriver.json", b'[{"x": 1}]')
|
||||||
|
self.assertEqual(r["size"], 10)
|
||||||
|
self.assertTrue(self.verify(self.m)["ok"])
|
||||||
|
|
||||||
|
def test_wrong_hash_leaves_file_alone(self):
|
||||||
|
p = os.path.join(self.up, "Deploy", "Control", "packages.json")
|
||||||
|
with self.assertRaises(UV.UploadError) as cm:
|
||||||
|
self.send("Deploy/Control/packages.json", b"{}")
|
||||||
|
self.assertEqual(cm.exception.status, 422)
|
||||||
|
with open(p, "rb") as f:
|
||||||
|
self.assertEqual(f.read(), b"[]")
|
||||||
|
self.assertEqual([n for n in os.listdir(os.path.dirname(p)) if ".part-" in n], [])
|
||||||
|
|
||||||
|
def test_wrong_upload_id(self):
|
||||||
|
with self.assertRaises(UV.UploadError) as cm:
|
||||||
|
self.send("Deploy/Control/packages.json", b"[]", uid="other")
|
||||||
|
self.assertEqual(cm.exception.status, 409)
|
||||||
|
|
||||||
|
def test_not_in_manifest(self):
|
||||||
|
with self.assertRaises(UV.UploadError) as cm:
|
||||||
|
self.send("Deploy/Control/new.json", b"[]")
|
||||||
|
self.assertEqual(cm.exception.status, 404)
|
||||||
|
|
||||||
|
def test_length_must_match(self):
|
||||||
|
with self.assertRaises(UV.UploadError):
|
||||||
|
self.send("Deploy/Control/packages.json", b"[]", length=99)
|
||||||
|
|
||||||
|
def test_traversal_refused(self):
|
||||||
|
with self.assertRaises(UV.UploadError):
|
||||||
|
self.send("../../etc/x", b"[]")
|
||||||
|
|
||||||
|
def test_big_file_goes_back_to_robocopy(self):
|
||||||
|
with mock.patch.object(UV, "SMALL_MAX", 100):
|
||||||
|
with self.assertRaises(UV.UploadError) as cm:
|
||||||
|
self.send(f"{WIM_DIR}/install.wim", WIM)
|
||||||
|
self.assertEqual(cm.exception.status, 413)
|
||||||
|
|
||||||
|
def test_put_reuses_existing_case(self):
|
||||||
|
# The folder on disk is "deploy/control" (lower case): the PUT must
|
||||||
|
# write into it, not create a second "Deploy/Control".
|
||||||
|
os.rename(os.path.join(self.up, "Deploy"), os.path.join(self.up, "deploy"))
|
||||||
|
os.rename(os.path.join(self.up, "deploy", "Control"), os.path.join(self.up, "deploy", "control"))
|
||||||
|
r = self.send("Deploy/Control/HardwareDriver.json", b'[{"x": 1}]')
|
||||||
|
self.assertEqual(r["written_as"], "deploy/control/HardwareDriver.json")
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(self.up, "Deploy")))
|
||||||
|
self.assertTrue(self.verify(self.m)["ok"])
|
||||||
|
|
||||||
|
def test_put_refused_while_locked(self):
|
||||||
|
self.assertTrue(UV.try_lock())
|
||||||
|
try:
|
||||||
|
with self.assertRaises(UV.UploadError) as cm:
|
||||||
|
self.send("Deploy/Control/HardwareDriver.json", b'[{"x": 1}]')
|
||||||
|
self.assertEqual(cm.exception.status, 409)
|
||||||
|
finally:
|
||||||
|
UV.unlock()
|
||||||
|
|
||||||
|
def test_link_in_parent_refused(self):
|
||||||
|
outside = os.path.join(self.tmp, "outside2")
|
||||||
|
os.makedirs(outside)
|
||||||
|
shutil.rmtree(os.path.join(self.up, "Tools"))
|
||||||
|
os.symlink(outside, os.path.join(self.up, "Tools"))
|
||||||
|
with self.assertRaises(UV.UploadError):
|
||||||
|
self.send("Tools/user_selections.json", self.files["Tools/user_selections.json"])
|
||||||
|
self.assertEqual(os.listdir(outside), [])
|
||||||
|
|
||||||
|
|
||||||
|
class GateTests(Base):
|
||||||
|
def test_no_manifest(self):
|
||||||
|
ok, reason, _ = UV.gate("gea-standard")
|
||||||
|
self.assertFalse(ok)
|
||||||
|
self.assertIn("Upload-Image.ps1", reason)
|
||||||
|
|
||||||
|
def test_unverified(self):
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, UV.MANIFEST), "w") as f:
|
||||||
|
json.dump(self.manifest(), f)
|
||||||
|
ok, reason, _ = UV.gate("gea-standard")
|
||||||
|
self.assertFalse(ok)
|
||||||
|
self.assertIn("not been verified", reason)
|
||||||
|
|
||||||
|
def test_pass_then_changed_file(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.verify()
|
||||||
|
self.assertTrue(UV.gate("gea-standard")[0])
|
||||||
|
self.assertFalse(UV.gate("gea-shopfloor")[0])
|
||||||
|
p = os.path.join(self.up, "Deploy", "Control", "packages.json")
|
||||||
|
time.sleep(0.01)
|
||||||
|
with open(p, "wb") as f:
|
||||||
|
f.write(b"[]")
|
||||||
|
ok, reason, _ = UV.gate("gea-standard")
|
||||||
|
self.assertFalse(ok)
|
||||||
|
self.assertIn("changed since verification", reason)
|
||||||
|
|
||||||
|
def test_manifest_swapped_after_verify(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.verify()
|
||||||
|
with open(os.path.join(self.up, UV.MANIFEST), "w") as f:
|
||||||
|
json.dump(self.manifest(), f) # same content, different bytes
|
||||||
|
self.assertFalse(UV.gate("gea-standard")[0])
|
||||||
|
|
||||||
|
def test_extras_then_removed(self):
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, "stale.txt"), "w") as f:
|
||||||
|
f.write("old")
|
||||||
|
self.verify()
|
||||||
|
ok, reason, info = UV.gate("gea-standard")
|
||||||
|
self.assertFalse(ok)
|
||||||
|
self.assertEqual(info["extra_count"], 1)
|
||||||
|
self.assertEqual(UV.remove_extras(), 1)
|
||||||
|
self.assertTrue(UV.gate("gea-standard")[0])
|
||||||
|
|
||||||
|
|
||||||
|
class RouteTests(Base):
|
||||||
|
def setUp(self):
|
||||||
|
super().setUp()
|
||||||
|
A.app.config["TESTING"] = True
|
||||||
|
self.c = A.app.test_client()
|
||||||
|
|
||||||
|
def wait(self):
|
||||||
|
for _ in range(200):
|
||||||
|
s = self.c.get("/api/upload/verify").get_json()
|
||||||
|
if s.get("state") in ("done", "error"):
|
||||||
|
return s
|
||||||
|
time.sleep(0.02)
|
||||||
|
self.fail("verify never finished")
|
||||||
|
|
||||||
|
def test_full_flow_through_http(self):
|
||||||
|
self.standard_upload()
|
||||||
|
self.put("Deploy/Control/HardwareDriver.json", b'[{"x": 1}]', on_server=False)
|
||||||
|
m = self.manifest()
|
||||||
|
r = self.c.post("/api/upload/verify", json=m)
|
||||||
|
self.assertEqual(r.status_code, 202, r.get_data(as_text=True))
|
||||||
|
s = self.wait()
|
||||||
|
self.assertEqual(s["missing"], ["Deploy/Control/HardwareDriver.json"])
|
||||||
|
r = self.c.put("/api/upload/file", query_string={"upload_id": m["upload_id"],
|
||||||
|
"path": "Deploy/Control/HardwareDriver.json"}, data=b'[{"x": 1}]')
|
||||||
|
self.assertEqual(r.status_code, 200, r.get_data(as_text=True))
|
||||||
|
self.assertEqual(self.c.post("/api/upload/verify", json=m).status_code, 202)
|
||||||
|
s = self.wait()
|
||||||
|
self.assertTrue(s["ok"] and s["importable"], s)
|
||||||
|
|
||||||
|
# Import: refused into the wrong image, accepted into the right one.
|
||||||
|
with self.c.session_transaction() as sess:
|
||||||
|
sess["_csrf_token"] = "t"
|
||||||
|
r = self.c.post("/images/import", data={"_csrf_token": "t", "source": self.up, "target": "gea-shopfloor"})
|
||||||
|
self.assertFalse(os.path.isdir(os.path.join(self.share, "gea-shopfloor", "Deploy", "Control")))
|
||||||
|
r = self.c.post("/images/import", data={"_csrf_token": "t", "source": self.up, "target": "gea-standard"})
|
||||||
|
self.assertEqual(r.status_code, 302)
|
||||||
|
img = os.path.join(self.share, "gea-standard")
|
||||||
|
self.assertTrue(os.path.isfile(os.path.join(img, "Deploy", "Control", "HardwareDriver.json")))
|
||||||
|
with open(os.path.join(img, ".mcl-import.json")) as f:
|
||||||
|
rec = json.load(f)
|
||||||
|
self.assertEqual(rec["upload_id"], m["upload_id"])
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(img, UV.MANIFEST)))
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(img, UV.STATE)))
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(self.up, UV.MANIFEST)))
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(img, UV.DETAIL)))
|
||||||
|
|
||||||
|
def test_bad_manifest_400(self):
|
||||||
|
r = self.c.post("/api/upload/verify", json={"manifest_version": 1, "upload_id": "x",
|
||||||
|
"files": [{"path": "../x", "size": 1, "sha256": "0" * 64}]})
|
||||||
|
self.assertEqual(r.status_code, 400)
|
||||||
|
|
||||||
|
def test_unverified_upload_refused_by_import(self):
|
||||||
|
self.standard_upload()
|
||||||
|
with self.c.session_transaction() as sess:
|
||||||
|
sess["_csrf_token"] = "t"
|
||||||
|
self.c.post("/images/import", data={"_csrf_token": "t", "source": self.up, "target": "gea-standard"})
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(self.share, "gea-standard", "Deploy", "Control")))
|
||||||
|
|
||||||
|
def test_import_refused_while_locked(self):
|
||||||
|
# Review finding 7: no import while a verify or PUT holds the lock.
|
||||||
|
self.standard_upload()
|
||||||
|
self.verify()
|
||||||
|
with self.c.session_transaction() as sess:
|
||||||
|
sess["_csrf_token"] = "t"
|
||||||
|
self.assertTrue(UV.try_lock())
|
||||||
|
try:
|
||||||
|
self.c.post("/images/import", data={"_csrf_token": "t", "source": self.up, "target": "gea-standard"})
|
||||||
|
finally:
|
||||||
|
UV.unlock()
|
||||||
|
self.assertFalse(os.path.exists(os.path.join(self.share, "gea-standard", "Deploy", "Control")))
|
||||||
|
self.assertTrue(os.path.exists(os.path.join(self.up, UV.MANIFEST)))
|
||||||
|
self.assertTrue(UV.try_lock()) # released again
|
||||||
|
UV.unlock()
|
||||||
|
|
||||||
|
def test_import_page_renders(self):
|
||||||
|
self.standard_upload()
|
||||||
|
with open(os.path.join(self.up, "stale.txt"), "w") as f:
|
||||||
|
f.write("old")
|
||||||
|
self.verify()
|
||||||
|
r = self.c.get("/images/import")
|
||||||
|
self.assertEqual(r.status_code, 200)
|
||||||
|
body = r.get_data(as_text=True)
|
||||||
|
self.assertIn("not part of this upload", body)
|
||||||
|
|
||||||
|
|
||||||
|
import atexit # noqa: E402
|
||||||
|
atexit.register(shutil.rmtree, _T, True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user