feat(installer): lock the third-party payload, and build on Windows without Bash
The bundle carries ~40 wheels, a Python installer and two MSIs. All of them run as SYSTEM on the target server, and nothing verified any of them. A missing wheelhouse printed MISSING and the script still exited 0, so an empty bundle compiled into a shippable installer and the failure surfaced on an air-gapped server with no way to fix it. bundle-lock.json now records that payload exactly - sha256 and byte size per file - and verification is set equality: a missing file, an unexpected extra file, or changed content all fail. Both builders check it and refuse to produce an unverified bundle; the lock ships inside the bundle and shopdb-install.ps1 re-checks it on the server before running any of it. This is deliberately a layer above requirements.txt hashes. pip lists every artifact of a pinned version (cffi 2.1.0 alone has 100 hashes), so it proves a wheel is genuine, not that it is the wheel this bundle was built and tested with; it ignores extra files in the wheelhouse; and it covers none of the executables. refresh-bundle-lock.ps1 regenerates the lock but refuses to overwrite one until the operator has seen the diff, because the commit is the review - it is the only place a change to what runs as SYSTEM becomes visible to a human. build-installer.ps1 is the whole build natively on Windows, so a work PC needs no Bash. It shares the plugin closure resolver with build-site.sh. Both builders now copy the installer scripts from the repository. They were copied from a downloads folder, so the logic that shipped was not the logic that was committed and the build worked on exactly one machine. Two verifiers exist because PowerShell is the only thing guaranteed present on the target server, while the Linux builder should not need pwsh. tests/test_bundle_lock.py runs both against the same fixtures and fails if they disagree.
This commit is contained in:
@@ -19,27 +19,122 @@ so a separate repo would drift out of step with the thing it installs.
|
||||
| `build-installer.sh` | Stages the bundle from a site profile. |
|
||||
| `make-branding.py` | Generates wizard artwork and the icon from `frontend/public/*.svg`. |
|
||||
| `*.bmp`, `shopdb.ico` | Generated artwork, committed so a Windows build box needs no Python. |
|
||||
| `build-installer.ps1` | The same build, natively on Windows. No Bash needed. |
|
||||
| `bundle-lock.json` | The exact third-party payload this installer ships. Reviewed by commit. |
|
||||
| `bundle-lock.ps1` | Creates and checks that lock. Also runs on the target server. |
|
||||
| `refresh-bundle-lock.ps1` | Regenerates the lock, after showing what changed. |
|
||||
| `verify_bundle_lock.py` | The same check for the Bash builder, so Linux needs no pwsh. |
|
||||
|
||||
## Building
|
||||
|
||||
Two builders, same result. Use whichever machine you are on - `build-installer.ps1`
|
||||
does the whole job natively so a Windows work PC needs no Bash.
|
||||
|
||||
```bash
|
||||
# 1. Stage the bundle for a site's plugin set.
|
||||
# Linux / WSL
|
||||
./build-installer.sh ../../site-profile.example.json
|
||||
```
|
||||
|
||||
# 2. Add the pieces that cannot be built on Linux:
|
||||
# bundle/wheels/ ~47 cp314 win_amd64 wheels, built ON Windows:
|
||||
# pip download -r requirements.txt -d wheels --only-binary=:all:
|
||||
# bundle/python/ python-3.14.x-amd64.exe
|
||||
# bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
|
||||
# bundle/mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)
|
||||
```powershell
|
||||
# Windows
|
||||
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
|
||||
```
|
||||
|
||||
# 3. Compile on Windows.
|
||||
Both stage the app tree, build the SPA twice, write `plugins.iss`, copy the
|
||||
installer scripts **from this directory**, and then verify the third-party
|
||||
payload against `bundle-lock.json`. They exit non-zero if it does not match.
|
||||
|
||||
The payload itself is added by hand, and no longer needs Windows to produce:
|
||||
|
||||
```
|
||||
bundle/wheels/ pip download -r requirements.txt --only-binary=:all: \
|
||||
--platform win_amd64 --python-version 314 \
|
||||
--implementation cp --abi cp314 -d wheels
|
||||
bundle/python/ python-3.14.x-amd64.exe
|
||||
bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
|
||||
bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below)
|
||||
bundle/mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)
|
||||
```
|
||||
|
||||
Then compile on Windows:
|
||||
|
||||
```
|
||||
iscc ShopDBFlask.iss
|
||||
```
|
||||
|
||||
**Inno Setup 6.6.0 or newer.** The wizard uses the built-in `windows11` custom
|
||||
style, which earlier versions reject. The script fails at compile time with that
|
||||
sentence rather than with a bare "WizardStyle is invalid".
|
||||
|
||||
The wheelhouse is **cp314-locked**. A different Python minor version means a
|
||||
different wheelhouse; the installer will not use a Python it did not install.
|
||||
|
||||
## What is pinned, and where
|
||||
|
||||
Three layers, because no one of them covers the whole problem.
|
||||
|
||||
| Layer | Covers | Enforced |
|
||||
|---|---|---|
|
||||
| `requirements.txt` sha256 per package | every wheel is genuinely what upstream published | `pip --require-hashes` at install; aborts on mismatch |
|
||||
| `bundle-lock.json` | the EXACT payload: wheels, Python installer, MSIs | both builders, and again on the server before anything runs |
|
||||
| git | the application tree | code review |
|
||||
|
||||
`--require-hashes` alone is not enough. pip lists every artifact of a pinned
|
||||
version - `cffi 2.1.0` has 100 hashes - so it proves the wheel is genuine, not
|
||||
that it is the wheel this bundle was built and tested with. It also ignores
|
||||
extra files in the wheelhouse, and says nothing about the Python installer or
|
||||
the MSIs, all of which run as SYSTEM on the target server.
|
||||
|
||||
So `bundle-lock.json` records an exact file set with a sha256 and a byte size
|
||||
each, and verification is **set equality**: a missing file, an unexpected extra
|
||||
file, or changed content all fail. There is no install-time override.
|
||||
|
||||
The lockfile is `--universal`, so one file serves Linux (dev, Docker, CI) and the
|
||||
Windows wheelhouse. A Linux-only resolve had silently omitted `colorama`, a
|
||||
win32-only dependency of `click` - which in hash-checking mode is a hard error
|
||||
rather than a quiet omission.
|
||||
|
||||
### Changing what ships
|
||||
|
||||
```powershell
|
||||
.\refresh-bundle-lock.ps1 # show what changed, write nothing
|
||||
.\refresh-bundle-lock.ps1 -Yes # write it
|
||||
```
|
||||
|
||||
Then **commit `bundle-lock.json`**. That commit is the review - it is the only
|
||||
place a change to what runs as SYSTEM on a customer's server becomes visible to
|
||||
a human. `refresh-bundle-lock.ps1` refuses to overwrite an existing lock until
|
||||
you have seen the diff, for that reason.
|
||||
|
||||
To stage a bundle before its lock exists: `ALLOW_UNLOCKED=1` (Bash) or
|
||||
`-AllowUnlocked` (PowerShell). Bundles built that way must not be shipped.
|
||||
|
||||
Verifying a live server, months later and offline:
|
||||
|
||||
```powershell
|
||||
shopdb-admin.ps1 verify
|
||||
```
|
||||
|
||||
## Client IP addresses
|
||||
|
||||
IIS does not set `X-Forwarded-For` on its own, and HttpPlatformHandler connects
|
||||
from loopback. Without a rule, **every client reads as 127.0.0.1** - so the
|
||||
GE-Enforce IP allowlist, the dashboard's visitor-location lookup and per-host
|
||||
login rate limiting all stop working, silently.
|
||||
|
||||
The wizard asks, because the two answers are mutually exclusive:
|
||||
|
||||
- **Clients connect directly** (`-ClientIpSource direct`, the default) - installs
|
||||
URL Rewrite from the bundle and sets `X-Forwarded-For` from `REMOTE_ADDR`.
|
||||
Overwriting the header is what stops a client spoofing its own.
|
||||
- **A proxy sits in front** (`-ClientIpSource proxy`) - leaves the rule off.
|
||||
Behind ARR or a load balancer `REMOTE_ADDR` is the *proxy*, so applying the
|
||||
rule would discard the real client IP.
|
||||
|
||||
An existing `web.config` is never overwritten - it is the one file on a server
|
||||
that legitimately carries hand-edits. The installer reports what it found
|
||||
instead.
|
||||
|
||||
## Deployment methods
|
||||
|
||||
Chosen in the wizard, and the bundle carries a SPA build for each because Vite
|
||||
|
||||
268
deploy/windows/installer/build-installer.ps1
Normal file
268
deploy/windows/installer/build-installer.ps1
Normal file
@@ -0,0 +1,268 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Stage a lean per-site installer bundle on Windows, and verify its third-party
|
||||
payload against bundle-lock.json.
|
||||
|
||||
.DESCRIPTION
|
||||
The Windows equivalent of build-installer.sh, for a work PC with no Bash. It
|
||||
does the whole job natively - it does NOT shell out to build-site.sh - so the
|
||||
only tools needed are Python, Node and (to compile) Inno Setup.
|
||||
|
||||
What it does, in order:
|
||||
|
||||
1. Resolves the site's plugin closure from its profile.
|
||||
2. Builds the SPA twice, because Vite compiles the base path in and it
|
||||
therefore cannot be chosen at install time: once for /<alias> (subpath
|
||||
deployment) and once for / (its own site).
|
||||
3. Stages the backend tree: core, the chosen plugins only, and the runtime
|
||||
files a deployable tree needs.
|
||||
4. Writes plugins.iss so the wizard's plugin page matches the payload.
|
||||
5. Copies the installer scripts from THIS directory.
|
||||
6. Verifies wheels\, python\ and the MSIs against bundle-lock.json, and
|
||||
FAILS if the bundle is not exactly what the lock describes.
|
||||
|
||||
Step 6 is the point of the script. A bundle that does not match its lock is
|
||||
not shipped, and "the wheelhouse was missing" is caught here rather than by
|
||||
an operator halfway through installing on a server with no network.
|
||||
|
||||
.PARAMETER Profile
|
||||
Path to the site profile (see deploy\site-profile.example.json).
|
||||
|
||||
.PARAMETER RepoRoot
|
||||
The repository. Defaults to four levels up from this script, which is correct
|
||||
for a normal checkout.
|
||||
|
||||
.PARAMETER SkipFrontend
|
||||
Reuse the SPA builds already staged in the bundle. For iterating on the
|
||||
installer itself, where two Vite builds per run is most of the wall clock.
|
||||
Never use it for a bundle you intend to ship.
|
||||
|
||||
.PARAMETER AllowUnlocked
|
||||
Stage the bundle and report payload problems WITHOUT failing. For assembling
|
||||
a bundle before its lock exists. A bundle built this way must not be shipped;
|
||||
run refresh-bundle-lock.ps1, commit the lock, then build again without this.
|
||||
|
||||
.EXAMPLE
|
||||
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
|
||||
.\build-installer.ps1 -Profile C:\sites\wjf.json -SkipFrontend
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $Profile,
|
||||
[string] $RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path,
|
||||
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
|
||||
[string] $SubpathAlias = 'shopdb',
|
||||
[switch] $SkipFrontend,
|
||||
[switch] $AllowUnlocked
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
|
||||
|
||||
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
|
||||
function Step { param($m) Write-Host ''; Write-Host "==> $m" -ForegroundColor Cyan }
|
||||
function Die { param($m, $fix = '') Write-Host ''; Write-Host " $m" -ForegroundColor Red
|
||||
if ($fix) { Write-Host " $fix" -ForegroundColor Yellow }; exit 1 }
|
||||
|
||||
function Invoke-Tool {
|
||||
# Runs a build tool and stops on a non-zero exit. npm writes progress to
|
||||
# stderr on SUCCESS, so stderr alone must never be treated as failure.
|
||||
param([string] $Exe, [string[]] $Arguments, [string] $WorkDir, [string] $What)
|
||||
Push-Location $WorkDir
|
||||
try {
|
||||
$prev = $ErrorActionPreference
|
||||
$ErrorActionPreference = 'Continue'
|
||||
& $Exe @Arguments 2>&1 | ForEach-Object { Say " $_" 'DarkGray' }
|
||||
$code = $LASTEXITCODE
|
||||
$ErrorActionPreference = $prev
|
||||
if ($code -ne 0) { Die "$What failed (exit $code)" }
|
||||
} finally { Pop-Location }
|
||||
}
|
||||
|
||||
if (-not (Test-Path $Profile)) { Die "profile not found: $Profile" }
|
||||
if (-not (Test-Path $RepoRoot)) { Die "repo not found: $RepoRoot" }
|
||||
$Profile = (Resolve-Path $Profile).Path
|
||||
$RepoRoot = (Resolve-Path $RepoRoot).Path
|
||||
$AppOut = Join-Path $BundleRoot 'app'
|
||||
|
||||
$python = (Get-Command python -ErrorAction SilentlyContinue)
|
||||
if (-not $python) { $python = Get-Command py -ErrorAction SilentlyContinue }
|
||||
if (-not $python) { Die 'Python not found on PATH' 'Install Python 3.14 and re-open the shell.' }
|
||||
$npm = Get-Command npm.cmd -ErrorAction SilentlyContinue
|
||||
if (-not $npm -and -not $SkipFrontend) { Die 'npm not found on PATH' 'Install Node, or pass -SkipFrontend to reuse the staged SPA.' }
|
||||
|
||||
Say ''
|
||||
Say " repo : $RepoRoot"
|
||||
Say " profile : $Profile"
|
||||
Say " bundle : $BundleRoot"
|
||||
|
||||
# --- 1. plugin closure ------------------------------------------------------
|
||||
# Same resolver the Linux builder uses, so both produce the same set from one
|
||||
# profile instead of two implementations of the closure rules.
|
||||
Step 'Resolving plugin closure'
|
||||
$closure = (& $python.Source (Join-Path $RepoRoot 'scripts\resolve_plugin_closure.py') $Profile $RepoRoot)
|
||||
if ($LASTEXITCODE -ne 0 -or -not $closure) { Die 'could not resolve the plugin closure from the profile' }
|
||||
$closure = $closure.Trim()
|
||||
Say " $closure" 'White'
|
||||
|
||||
# --- 2. frontend ------------------------------------------------------------
|
||||
$frontend = Join-Path $RepoRoot 'frontend'
|
||||
$subStaged = Join-Path $BundleRoot 'spa-subpath'
|
||||
$rootStaged= Join-Path $BundleRoot 'spa-root'
|
||||
|
||||
if ($SkipFrontend) {
|
||||
if (-not (Test-Path $subStaged) -or -not (Test-Path $rootStaged)) {
|
||||
Die '-SkipFrontend was passed but no SPA build is staged' 'Run once without it.'
|
||||
}
|
||||
Say ''
|
||||
Say ' Reusing the staged SPA builds (-SkipFrontend). NOT shippable if the frontend changed.' 'Yellow'
|
||||
} else {
|
||||
Step "Building the SPA for /$SubpathAlias/"
|
||||
$env:SITE_PLUGINS = $closure
|
||||
$env:VITE_BASE_PATH = "/$SubpathAlias/"
|
||||
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'subpath frontend build' }
|
||||
finally { Remove-Item Env:\VITE_BASE_PATH -ErrorAction SilentlyContinue }
|
||||
Remove-Item $subStaged -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Copy-Item (Join-Path $frontend 'dist') $subStaged -Recurse -Force
|
||||
# The alias is fixed at BUILD time; the installer reads this and refuses to
|
||||
# publish under a different one rather than serving a page that cannot load
|
||||
# its own assets.
|
||||
Set-Content -Path (Join-Path $subStaged '.alias') -Value $SubpathAlias -Encoding ASCII
|
||||
|
||||
# Root build LAST, so frontend\dist is left in the state a developer expects.
|
||||
Step 'Building the SPA for /'
|
||||
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'root frontend build' }
|
||||
finally { Remove-Item Env:\SITE_PLUGINS -ErrorAction SilentlyContinue }
|
||||
Remove-Item $rootStaged -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Copy-Item (Join-Path $frontend 'dist') $rootStaged -Recurse -Force
|
||||
}
|
||||
|
||||
# --- 3. backend tree --------------------------------------------------------
|
||||
Step "Staging the application tree"
|
||||
Remove-Item $AppOut -Recurse -Force -ErrorAction SilentlyContinue
|
||||
New-Item -ItemType Directory -Path (Join-Path $AppOut 'plugins') -Force | Out-Null
|
||||
|
||||
Copy-Item (Join-Path $RepoRoot 'shopdb') $AppOut -Recurse -Force
|
||||
foreach ($name in $closure.Split(',')) {
|
||||
$src = Join-Path $RepoRoot ('plugins\' + $name.Trim())
|
||||
if (-not (Test-Path $src)) { Die "plugin in the closure is not on disk: $name" }
|
||||
Copy-Item $src (Join-Path $AppOut 'plugins') -Recurse -Force
|
||||
}
|
||||
|
||||
# Runtime files a deployable tree needs beyond the Python packages. Without
|
||||
# these the tree imports but cannot be run or migrated.
|
||||
foreach ($f in @('wsgi.py', 'requirements.txt')) {
|
||||
Copy-Item (Join-Path $RepoRoot $f) $AppOut -Force
|
||||
}
|
||||
Copy-Item (Join-Path $RepoRoot 'migrations') $AppOut -Recurse -Force
|
||||
# ONLY web.config, not all of deploy\. The bundle is staged INSIDE deploy\, so
|
||||
# copying the whole tree would recurse into its own output, and the rest of
|
||||
# deploy\ is installer source that has no business on an application server.
|
||||
# shopdb-install.ps1 reads it from exactly this path.
|
||||
$cfgSrc = Join-Path $RepoRoot 'deploy\windows\web.config'
|
||||
if (Test-Path $cfgSrc) {
|
||||
New-Item -ItemType Directory -Path (Join-Path $AppOut 'deploy\windows') -Force | Out-Null
|
||||
Copy-Item $cfgSrc (Join-Path $AppOut 'deploy\windows') -Force
|
||||
}
|
||||
|
||||
# Stage the profile INTO the tree: `flask plugin apply-profile` at provisioning
|
||||
# reads the same profile the tree was staged from, so the installed plugin set
|
||||
# and the shipped plugin code cannot drift.
|
||||
Copy-Item $Profile (Join-Path $AppOut 'site-profile.json') -Force
|
||||
|
||||
# The installer expects frontend\dist and frontend\dist-subpath.
|
||||
$feOut = Join-Path $AppOut 'frontend'
|
||||
New-Item -ItemType Directory -Path $feOut -Force | Out-Null
|
||||
Copy-Item $rootStaged (Join-Path $feOut 'dist') -Recurse -Force
|
||||
Copy-Item $subStaged (Join-Path $feOut 'dist-subpath') -Recurse -Force
|
||||
|
||||
Get-ChildItem $AppOut -Recurse -Directory -Filter '__pycache__' -ErrorAction SilentlyContinue |
|
||||
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Get-ChildItem $AppOut -Recurse -File -Filter '*.pyc' -ErrorAction SilentlyContinue |
|
||||
Remove-Item -Force -ErrorAction SilentlyContinue
|
||||
|
||||
# --- 4. plugins.iss ---------------------------------------------------------
|
||||
Step 'Writing plugins.iss'
|
||||
$shipped = (Get-ChildItem (Join-Path $AppOut 'plugins') -Directory | Select-Object -ExpandProperty Name) -join ','
|
||||
$aliasBuilt = ''
|
||||
$aliasFile = Join-Path $feOut 'dist-subpath\.alias'
|
||||
if (Test-Path $aliasFile) { $aliasBuilt = (Get-Content $aliasFile -TotalCount 1).Trim() }
|
||||
@"
|
||||
; GENERATED by build-installer.ps1 - do not edit.
|
||||
; The plugins present in bundle\app\plugins at build time.
|
||||
#define AvailablePlugins "$shipped"
|
||||
; The alias the subpath SPA was built for, or empty if this bundle has no
|
||||
; subpath build - in which case the wizard must not offer that option.
|
||||
#define SubpathAlias "$aliasBuilt"
|
||||
"@ | Set-Content -Path (Join-Path $PSScriptRoot 'plugins.iss') -Encoding ASCII
|
||||
Say " $shipped" 'White'
|
||||
|
||||
# --- 5. installer scripts ---------------------------------------------------
|
||||
# From THIS directory, which is the reviewed copy under version control. They
|
||||
# used to be copied from a downloads folder, so the logic that shipped was not
|
||||
# the logic that was committed and the build only worked on one machine.
|
||||
Step 'Copying installer scripts'
|
||||
foreach ($f in @('shopdb-install.ps1', 'shopdb-preflight.ps1', 'bundle-lock.ps1')) {
|
||||
$src = Join-Path $PSScriptRoot $f
|
||||
if (-not (Test-Path $src)) { Die "installer script missing from the repo: $f" }
|
||||
Copy-Item $src $BundleRoot -Force
|
||||
Say " $f"
|
||||
}
|
||||
|
||||
# --- 6. payload verification ------------------------------------------------
|
||||
Step 'Verifying the third-party payload against bundle-lock.json'
|
||||
$lockPath = Join-Path $PSScriptRoot 'bundle-lock.json'
|
||||
$lock = Read-BundleLock $lockPath
|
||||
if (-not $lock) {
|
||||
$msg = "no bundle-lock.json at $lockPath"
|
||||
if ($AllowUnlocked) { Say " $msg - continuing because -AllowUnlocked was passed" 'Yellow' }
|
||||
else {
|
||||
Die $msg @'
|
||||
Add the wheels and installers to the bundle, then:
|
||||
.\refresh-bundle-lock.ps1 (review the list)
|
||||
.\refresh-bundle-lock.ps1 -Yes (write it)
|
||||
and commit bundle-lock.json. Pass -AllowUnlocked to stage a bundle without one -
|
||||
it must not be shipped.
|
||||
'@
|
||||
}
|
||||
} else {
|
||||
$problems = Test-BundleLock -BundleRoot $BundleRoot -Lock $lock
|
||||
if ($problems.Count -eq 0) {
|
||||
Say (" payload matches the lock ({0}, {1})" -f `
|
||||
(Get-JsonProperty $lock 'pythontag' 'unknown'), (Get-JsonProperty $lock 'platform' 'unknown')) 'Green'
|
||||
# Ships WITH the bundle: the installer re-checks the payload on the
|
||||
# target server before running any of it, so tampering between build and
|
||||
# install is caught too.
|
||||
Copy-Item $lockPath $BundleRoot -Force
|
||||
} else {
|
||||
Say ''
|
||||
foreach ($p in $problems) { Say " $p" 'Red' }
|
||||
Say ''
|
||||
if ($AllowUnlocked) {
|
||||
Say ' -AllowUnlocked: continuing anyway. DO NOT SHIP this bundle.' 'Yellow'
|
||||
} else {
|
||||
Die ("{0} payload problem(s) - the bundle is not what the lock describes" -f $problems.Count) @'
|
||||
Either the payload is wrong (fix the bundle) or it changed on purpose (run
|
||||
refresh-bundle-lock.ps1, read the diff, and commit the new lock).
|
||||
'@
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- summary ----------------------------------------------------------------
|
||||
Write-Host ''
|
||||
Say " Bundle staged at: $BundleRoot" 'White'
|
||||
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'mysql')) {
|
||||
$p = Join-Path $BundleRoot $d
|
||||
if (Test-Path $p) {
|
||||
$mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB)
|
||||
Say (" {0,-22} {1,8:N1} MB" -f $d, $mb)
|
||||
} else {
|
||||
Say (" {0,-22} {1}" -f $d, 'absent') 'DarkGray'
|
||||
}
|
||||
}
|
||||
Write-Host ''
|
||||
Say ' Compile with: iscc ShopDBFlask.iss' 'White'
|
||||
Say ' (Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)' 'DarkGray'
|
||||
Write-Host ''
|
||||
@@ -59,26 +59,71 @@ cat > "$HERE/plugins.iss" <<EOF
|
||||
EOF
|
||||
echo " $PLUGINS"
|
||||
|
||||
# From THIS directory, which is the reviewed copy under version control. These
|
||||
# used to be copied from $HOME/Downloads, so the installer logic that shipped was
|
||||
# not the logic that was committed, and the build only worked on one machine.
|
||||
echo "==> Copying installer scripts"
|
||||
mkdir -p "$BUNDLE"
|
||||
cp "$HOME/Downloads/shopdb-install.ps1" "$HOME/Downloads/shopdb-preflight.ps1" "$BUNDLE/"
|
||||
for f in shopdb-install.ps1 shopdb-preflight.ps1 bundle-lock.ps1; do
|
||||
[ -f "$HERE/$f" ] || { echo "installer script missing from the repo: $f"; exit 1; }
|
||||
cp "$HERE/$f" "$BUNDLE/"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Bundle staged at: $BUNDLE"
|
||||
for d in app wheels python httpplatformhandler mysql; do
|
||||
for d in app wheels python httpplatformhandler urlrewrite mysql; do
|
||||
if [ -d "$BUNDLE/$d" ]; then
|
||||
printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)"
|
||||
else
|
||||
printf ' %-20s MISSING\n' "$d"
|
||||
printf ' %-20s absent\n' "$d"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
echo " plugins shipped: $(ls "$BUNDLE/app/plugins" 2>/dev/null | tr '\n' ' ')"
|
||||
|
||||
# --- payload verification ---------------------------------------------------
|
||||
# The third-party payload is the part git does not record: the wheels, the Python
|
||||
# installer and the MSIs that run as SYSTEM on the target server. It must be
|
||||
# EXACTLY what bundle-lock.json describes - no missing file, no stale extra wheel
|
||||
# left over from a previous build, no changed content - or this is not a bundle
|
||||
# anyone reviewed. Previously a missing wheelhouse printed MISSING and the script
|
||||
# still exited 0, so an empty bundle compiled into a shippable installer and the
|
||||
# failure surfaced on an air-gapped server with no way to fix it.
|
||||
#
|
||||
# ALLOW_UNLOCKED=1 downgrades this to a warning, for assembling a bundle before
|
||||
# its lock exists. A bundle built that way must not be shipped.
|
||||
echo ""
|
||||
echo "Missing pieces must be added by hand before compiling:"
|
||||
echo " wheels\\ built ON Windows (cp314 win_amd64), ~47 wheels"
|
||||
echo " python\\ python-3.14.6-amd64.exe"
|
||||
echo " httpplatformhandler\\ httpPlatformHandler_amd64.msi"
|
||||
echo " mysql\\ mysql-8.0.46-winx64.msi (bundled-database option only)"
|
||||
echo "==> Verifying the third-party payload against bundle-lock.json"
|
||||
if python3 "$HERE/verify_bundle_lock.py" "$BUNDLE" "$HERE/bundle-lock.json"; then
|
||||
echo " payload matches the lock"
|
||||
# Ships WITH the bundle: the installer re-checks the payload on the target
|
||||
# server before running any of it, so tampering between build and install is
|
||||
# caught too.
|
||||
cp "$HERE/bundle-lock.json" "$BUNDLE/"
|
||||
elif [ "${ALLOW_UNLOCKED:-0}" = "1" ]; then
|
||||
echo ""
|
||||
echo " ALLOW_UNLOCKED=1: continuing anyway. DO NOT SHIP this bundle."
|
||||
else
|
||||
echo ""
|
||||
echo " The bundle is not what the lock describes."
|
||||
echo ""
|
||||
echo " Add the missing pieces by hand:"
|
||||
echo " wheels/ pip download -r requirements.txt --only-binary=:all: \\"
|
||||
echo " --platform win_amd64 --python-version 314 \\"
|
||||
echo " --implementation cp --abi cp314 -d wheels"
|
||||
echo " python/ python-3.14.x-amd64.exe"
|
||||
echo " httpplatformhandler/ httpPlatformHandler_amd64.msi"
|
||||
echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)"
|
||||
echo " mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)"
|
||||
echo ""
|
||||
echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:"
|
||||
echo " pwsh ./refresh-bundle-lock.ps1 # review the diff"
|
||||
echo " pwsh ./refresh-bundle-lock.ps1 -Yes # write it"
|
||||
echo ""
|
||||
echo " To stage a bundle before its lock exists: ALLOW_UNLOCKED=1 $0 ..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Then compile on Windows: iscc ShopDBFlask.iss"
|
||||
echo "(Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)"
|
||||
|
||||
192
deploy/windows/installer/bundle-lock.ps1
Normal file
192
deploy/windows/installer/bundle-lock.ps1
Normal file
@@ -0,0 +1,192 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Hash manifest for the installer's third-party payload: create it, and check a
|
||||
bundle against it.
|
||||
|
||||
.DESCRIPTION
|
||||
Dot-source this. It defines two functions and runs nothing on its own:
|
||||
|
||||
New-BundleLock hash a staged bundle and return the lock object
|
||||
Test-BundleLock compare a staged bundle against a lock, return problems
|
||||
|
||||
WHAT IT COVERS, and why pip's own hash checking is not enough.
|
||||
|
||||
requirements.txt carries a sha256 for every wheel, so pip refuses an artifact
|
||||
upstream did not publish. Three gaps remain, and all three are what actually
|
||||
goes wrong with a hand-assembled offline bundle:
|
||||
|
||||
1. pip lists EVERY artifact of a pinned version - cffi 2.1.0 alone has 100
|
||||
hashes. It proves the wheel is genuine, not that it is the wheel this
|
||||
bundle was built and tested with.
|
||||
2. pip ignores extra files in the wheelhouse. A stale wheel left behind by
|
||||
a previous build sits there unnoticed until a resolve picks it up.
|
||||
3. pip says nothing about the rest of the payload - the Python installer,
|
||||
the HttpPlatformHandler MSI, URL Rewrite, MySQL. Those are executables
|
||||
that run as SYSTEM on the target server and were, until this file, the
|
||||
only unverified thing the installer would run.
|
||||
|
||||
So the lock records an exact file set with a sha256 and a byte size each, and
|
||||
verification is SET EQUALITY: a missing file, an unexpected extra file, or a
|
||||
changed file all fail. Nothing is skipped and nothing is "close enough".
|
||||
|
||||
The app tree is deliberately NOT covered. It is built from the repository on
|
||||
every run and changes with every commit; hashing it would make the lock churn
|
||||
constantly and train everyone to regenerate it without reading it. Git is the
|
||||
record for the app tree. This file is the record for everything that comes
|
||||
from outside the repository.
|
||||
|
||||
.NOTES
|
||||
Stock Windows PowerShell 5.1, and also runs under pwsh on Linux so the Bash
|
||||
builder can call the same checker.
|
||||
#>
|
||||
|
||||
# Payload directories under the bundle root. 'required' means the installer
|
||||
# cannot work without it; the optional ones are per-deployment choices, and an
|
||||
# absent optional directory is fine. A PRESENT directory is always checked in
|
||||
# full, optional or not.
|
||||
$script:BundlePayloads = @(
|
||||
@{ Name = 'wheels'; Required = $true; What = 'Python wheels for the offline install' }
|
||||
@{ Name = 'python'; Required = $true; What = 'the Python installer' }
|
||||
@{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' }
|
||||
@{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' }
|
||||
@{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' }
|
||||
)
|
||||
|
||||
function Get-JsonProperty {
|
||||
# shopdb-install.ps1 runs under Set-StrictMode 2.0, where reading a property
|
||||
# that does not exist on a PSCustomObject THROWS instead of returning $null.
|
||||
# A truncated or hand-edited bundle-lock.json would therefore blow up with
|
||||
# "Property 'payloads' cannot be found" rather than saying what is wrong with
|
||||
# the lock. Every read of parsed JSON goes through here.
|
||||
param($Object, [string] $Name, $Default = $null)
|
||||
if ($null -eq $Object) { return $Default }
|
||||
$prop = $Object.PSObject.Properties[$Name]
|
||||
if ($null -eq $prop) { return $Default }
|
||||
return $prop.Value
|
||||
}
|
||||
|
||||
function Get-FileDigest {
|
||||
param([string] $Path)
|
||||
$sha = [System.Security.Cryptography.SHA256]::Create()
|
||||
$stream = [System.IO.File]::OpenRead($Path)
|
||||
try { return (-join ($sha.ComputeHash($stream) | ForEach-Object { $_.ToString('x2') })) }
|
||||
finally { $stream.Dispose(); $sha.Dispose() }
|
||||
}
|
||||
|
||||
function Get-PayloadFiles {
|
||||
# Every file in the directory, keyed by its path RELATIVE to that directory
|
||||
# with forward slashes, so a lock generated on Windows reads the same from
|
||||
# the Bash builder.
|
||||
param([string] $Dir)
|
||||
$out = @{}
|
||||
if (-not (Test-Path $Dir)) { return $out }
|
||||
$root = (Resolve-Path $Dir).Path.TrimEnd('\', '/')
|
||||
foreach ($f in (Get-ChildItem $Dir -Recurse -File)) {
|
||||
$rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/')
|
||||
$out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length }
|
||||
}
|
||||
return $out
|
||||
}
|
||||
|
||||
function New-BundleLock {
|
||||
<#
|
||||
Hash a staged bundle. The caller writes the result to bundle-lock.json;
|
||||
this returns the object so a caller can diff it against the committed lock
|
||||
before overwriting anything.
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $BundleRoot,
|
||||
[string] $PythonTag = 'cp314',
|
||||
[string] $Platform = 'win_amd64'
|
||||
)
|
||||
$payloads = @{}
|
||||
foreach ($p in $script:BundlePayloads) {
|
||||
$dir = Join-Path $BundleRoot $p.Name
|
||||
if (-not (Test-Path $dir)) {
|
||||
if ($p.Required) { throw ("payload directory is missing: {0} ({1})" -f $p.Name, $p.What) }
|
||||
continue
|
||||
}
|
||||
$files = Get-PayloadFiles $dir
|
||||
if ($files.Count -eq 0 -and $p.Required) {
|
||||
throw ("payload directory is empty: {0} ({1})" -f $p.Name, $p.What)
|
||||
}
|
||||
$payloads[$p.Name] = @{ required = $p.Required; files = $files }
|
||||
}
|
||||
return [ordered]@{
|
||||
schema = 1
|
||||
generated = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
|
||||
pythontag = $PythonTag
|
||||
platform = $Platform
|
||||
note = 'Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.'
|
||||
payloads = $payloads
|
||||
}
|
||||
}
|
||||
|
||||
function Test-BundleLock {
|
||||
<#
|
||||
Compare a staged bundle against a lock. Returns an array of problem
|
||||
strings - EMPTY means the bundle is exactly what the lock describes.
|
||||
|
||||
Returning problems rather than throwing is deliberate: an operator fixing a
|
||||
wheelhouse wants the whole list at once, not one failure per rebuild.
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $BundleRoot,
|
||||
[Parameter(Mandatory = $true)] $Lock
|
||||
)
|
||||
$problems = @()
|
||||
$payloads = Get-JsonProperty $Lock 'payloads'
|
||||
if ($null -eq $payloads) { return @('bundle-lock.json has no "payloads" section') }
|
||||
|
||||
foreach ($p in $script:BundlePayloads) {
|
||||
$name = $p.Name
|
||||
$dir = Join-Path $BundleRoot $name
|
||||
$present = Test-Path $dir
|
||||
$entry = Get-JsonProperty $payloads $name
|
||||
$locked = ($null -ne $entry)
|
||||
|
||||
if (-not $locked) {
|
||||
# Not in the lock at all. An unlocked directory that exists is a
|
||||
# payload nobody reviewed, which is exactly what this is here to stop.
|
||||
if ($present) { $problems += "$name/ is present but is not in bundle-lock.json - regenerate the lock" }
|
||||
elseif ($p.Required) { $problems += "$name/ is required but is in neither the bundle nor the lock" }
|
||||
continue
|
||||
}
|
||||
if (-not $present) {
|
||||
if ($p.Required -or (Get-JsonProperty $entry 'required' $false)) {
|
||||
$problems += "$name/ is in the lock but missing from the bundle ($($p.What))"
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
$expected = @{}
|
||||
$lockedFiles = Get-JsonProperty $entry 'files'
|
||||
if ($null -ne $lockedFiles) {
|
||||
foreach ($prop in $lockedFiles.PSObject.Properties) { $expected[$prop.Name] = $prop.Value }
|
||||
}
|
||||
$actual = Get-PayloadFiles $dir
|
||||
|
||||
# Sorted so the report reads the same way twice, and so it matches the
|
||||
# order verify_bundle_lock.py produces.
|
||||
foreach ($rel in ($expected.Keys | Sort-Object)) {
|
||||
if (-not $actual.ContainsKey($rel)) { $problems += "$name/$rel is in the lock but missing from the bundle"; continue }
|
||||
if ($actual[$rel].sha256 -ne $expected[$rel].sha256) {
|
||||
$problems += "$name/$rel does NOT match the lock (expected sha256 $($expected[$rel].sha256.Substring(0,12))..., got $($actual[$rel].sha256.Substring(0,12))...)"
|
||||
} elseif ([int64] $actual[$rel].size -ne [int64] $expected[$rel].size) {
|
||||
# Cannot happen for a matching sha256, so it means the lock itself
|
||||
# was hand-edited. Say so rather than passing it.
|
||||
$problems += "$name/$rel size disagrees with the lock - the lock has been edited by hand"
|
||||
}
|
||||
}
|
||||
foreach ($rel in ($actual.Keys | Sort-Object)) {
|
||||
if (-not $expected.ContainsKey($rel)) { $problems += "$name/$rel is in the bundle but NOT in the lock (unexpected extra file)" }
|
||||
}
|
||||
}
|
||||
return $problems
|
||||
}
|
||||
|
||||
function Read-BundleLock {
|
||||
param([Parameter(Mandatory = $true)] [string] $Path)
|
||||
if (-not (Test-Path $Path)) { return $null }
|
||||
return (Get-Content $Path -Raw | ConvertFrom-Json)
|
||||
}
|
||||
@@ -150,7 +150,7 @@ def main():
|
||||
for s, name in [(55, "wizard-small.bmp"), (64, "wizard-small@125.bmp"),
|
||||
(138, "wizard-small@250.bmp")]:
|
||||
small(s).save(OUT / name, "BMP"); made.append((name, f"{s}x{s}"))
|
||||
icon(OUT / "shopdb.ico"); made.append(("shopdb.ico", "multi-res"))
|
||||
icon(OUT / "shopdb.ico"); made.append(("shopdb.ico", "multi-size"))
|
||||
for name, dims in made:
|
||||
print(f" {name:<26} {dims:<10} {(OUT / name).stat().st_size // 1024} KB")
|
||||
|
||||
|
||||
110
deploy/windows/installer/refresh-bundle-lock.ps1
Normal file
110
deploy/windows/installer/refresh-bundle-lock.ps1
Normal file
@@ -0,0 +1,110 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Regenerate bundle-lock.json from the staged bundle, after showing what would
|
||||
change.
|
||||
|
||||
.DESCRIPTION
|
||||
Run this on the machine that assembled the wheelhouse, once the bundle holds
|
||||
the payload you intend to ship:
|
||||
|
||||
bundle\wheels\ the wheels, built with the matching Python
|
||||
bundle\python\ the Python installer
|
||||
bundle\httpplatformhandler\ the IIS module MSI
|
||||
bundle\urlrewrite\ URL Rewrite MSI (optional)
|
||||
bundle\mysql\ MySQL MSI (optional)
|
||||
|
||||
Then COMMIT the resulting bundle-lock.json. That commit is the review: it is
|
||||
the only place a change to what runs as SYSTEM on a customer's server becomes
|
||||
visible to a human. A lock regenerated and committed without reading the diff
|
||||
provides nothing, so this refuses to overwrite an existing lock until you
|
||||
have seen the change and passed -Yes.
|
||||
|
||||
.EXAMPLE
|
||||
.\refresh-bundle-lock.ps1 # show the diff, write nothing
|
||||
.\refresh-bundle-lock.ps1 -Yes # write it
|
||||
|
||||
.NOTES
|
||||
Building the wheelhouse itself no longer requires Windows. From any machine:
|
||||
pip download -r requirements.txt -d wheels --only-binary=:all: `
|
||||
--platform win_amd64 --python-version 314 --implementation cp --abi cp314
|
||||
Do it wherever you like; this script records what came out.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
|
||||
[string] $LockPath = (Join-Path $PSScriptRoot 'bundle-lock.json'),
|
||||
[string] $PythonTag = 'cp314',
|
||||
[string] $Platform = 'win_amd64',
|
||||
[switch] $Yes
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
|
||||
|
||||
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
|
||||
|
||||
if (-not (Test-Path $BundleRoot)) {
|
||||
Say "bundle not found: $BundleRoot" 'Red'
|
||||
Say 'Stage it first with build-installer.ps1 (or build-installer.sh), then add' 'Yellow'
|
||||
Say 'the wheels and installers by hand.' 'Yellow'
|
||||
exit 1
|
||||
}
|
||||
|
||||
Say ''
|
||||
Say " Hashing $BundleRoot" 'Cyan'
|
||||
$new = New-BundleLock -BundleRoot $BundleRoot -PythonTag $PythonTag -Platform $Platform
|
||||
|
||||
foreach ($name in $new.payloads.Keys) {
|
||||
Say (" {0,-22} {1,4} files" -f $name, $new.payloads[$name].files.Count)
|
||||
}
|
||||
|
||||
$old = Read-BundleLock $LockPath
|
||||
if (-not $old) {
|
||||
Say ''
|
||||
Say ' No existing lock - this will be the first one.' 'Yellow'
|
||||
} else {
|
||||
# Diff by file, per payload, so the operator sees exactly which artifacts
|
||||
# changed rather than "the lock is different".
|
||||
Say ''
|
||||
Say ' Changes against the committed lock:' 'Cyan'
|
||||
$changes = 0
|
||||
foreach ($name in $new.payloads.Keys) {
|
||||
$oldFiles = @{}
|
||||
if ($old.payloads.PSObject.Properties.Name -contains $name) {
|
||||
foreach ($p in $old.payloads.$name.files.PSObject.Properties) { $oldFiles[$p.Name] = $p.Value.sha256 }
|
||||
}
|
||||
$newFiles = $new.payloads[$name].files
|
||||
foreach ($rel in ($newFiles.Keys | Sort-Object)) {
|
||||
if (-not $oldFiles.ContainsKey($rel)) { Say " + $name/$rel" 'Green'; $changes++ }
|
||||
elseif ($oldFiles[$rel] -ne $newFiles[$rel].sha256) { Say " ~ $name/$rel (content changed)" 'Yellow'; $changes++ }
|
||||
}
|
||||
foreach ($rel in ($oldFiles.Keys | Sort-Object)) {
|
||||
if (-not $newFiles.ContainsKey($rel)) { Say " - $name/$rel" 'Red'; $changes++ }
|
||||
}
|
||||
}
|
||||
foreach ($p in $old.payloads.PSObject.Properties.Name) {
|
||||
if (-not $new.payloads.Contains($p)) { Say " - $p/ (whole payload gone)" 'Red'; $changes++ }
|
||||
}
|
||||
if ($changes -eq 0) {
|
||||
Say ' none - the bundle already matches the lock' 'Green'
|
||||
exit 0
|
||||
}
|
||||
Say ''
|
||||
Say (" {0} change(s)." -f $changes) 'White'
|
||||
}
|
||||
|
||||
if (-not $Yes) {
|
||||
Say ''
|
||||
Say ' Nothing written. Read the list above, then re-run with -Yes.' 'Yellow'
|
||||
Say ' Commit the resulting bundle-lock.json - that commit IS the review.' 'Yellow'
|
||||
exit 2
|
||||
}
|
||||
|
||||
# ConvertTo-Json defaults to a depth of 2, which silently flattens the per-file
|
||||
# entries into "System.Collections.Hashtable" strings and produces a lock that
|
||||
# verifies against nothing.
|
||||
$new | ConvertTo-Json -Depth 8 | Set-Content -Path $LockPath -Encoding UTF8
|
||||
Say ''
|
||||
Say " Written: $LockPath" 'Green'
|
||||
Say ' Commit it.' 'Green'
|
||||
119
deploy/windows/installer/verify_bundle_lock.py
Normal file
119
deploy/windows/installer/verify_bundle_lock.py
Normal file
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check a staged installer bundle against bundle-lock.json.
|
||||
|
||||
Prints one line per problem and exits non-zero if there are any. Exits 0 only
|
||||
when the bundle's third-party payload is EXACTLY what the lock describes: no
|
||||
missing file, no unexpected extra file, no changed content.
|
||||
|
||||
Why this exists alongside bundle-lock.ps1, which does the same job:
|
||||
|
||||
- bundle-lock.ps1 is canonical. It runs at INSTALL time on the target server,
|
||||
where PowerShell is the only thing guaranteed to be present - Python is not
|
||||
installed until stage 2, and verifying the payload after running part of it
|
||||
would defeat the purpose.
|
||||
- This file lets the Linux builder (build-installer.sh) do the same check
|
||||
without adding pwsh as a build dependency.
|
||||
|
||||
The two are kept honest by tests/test_bundle_lock.py, which runs BOTH against
|
||||
the same fixtures and fails if they disagree.
|
||||
|
||||
Usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Must match $script:BundlePayloads in bundle-lock.ps1.
|
||||
PAYLOADS = [
|
||||
('wheels', True, 'Python wheels for the offline install'),
|
||||
('python', True, 'the Python installer'),
|
||||
('httpplatformhandler', True, 'the IIS module that launches waitress'),
|
||||
('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'),
|
||||
('mysql', False, 'MySQL, for the bundled-database option'),
|
||||
]
|
||||
|
||||
|
||||
def digest(path):
|
||||
sha = hashlib.sha256()
|
||||
with open(path, 'rb') as fh:
|
||||
for chunk in iter(lambda: fh.read(1024 * 1024), b''):
|
||||
sha.update(chunk)
|
||||
return sha.hexdigest()
|
||||
|
||||
|
||||
def payload_files(directory):
|
||||
"""Every file under the directory, keyed by forward-slashed relative path."""
|
||||
found = {}
|
||||
if not os.path.isdir(directory):
|
||||
return found
|
||||
for root, _dirs, files in os.walk(directory):
|
||||
for name in files:
|
||||
full = os.path.join(root, name)
|
||||
rel = os.path.relpath(full, directory).replace(os.sep, '/')
|
||||
found[rel] = {'sha256': digest(full), 'size': os.path.getsize(full)}
|
||||
return found
|
||||
|
||||
|
||||
def verify(bundle_root, lock):
|
||||
problems = []
|
||||
locked = lock.get('payloads')
|
||||
if not locked:
|
||||
return ['bundle-lock.json has no "payloads" section']
|
||||
|
||||
for name, required, what in PAYLOADS:
|
||||
directory = os.path.join(bundle_root, name)
|
||||
present = os.path.isdir(directory)
|
||||
if name not in locked:
|
||||
if present:
|
||||
problems.append(
|
||||
'%s/ is present but is not in bundle-lock.json - regenerate the lock' % name)
|
||||
elif required:
|
||||
problems.append(
|
||||
'%s/ is required but is in neither the bundle nor the lock' % name)
|
||||
continue
|
||||
if not present:
|
||||
if required or locked[name].get('required'):
|
||||
problems.append('%s/ is in the lock but missing from the bundle (%s)' % (name, what))
|
||||
continue
|
||||
|
||||
expected = locked[name].get('files', {})
|
||||
actual = payload_files(directory)
|
||||
for rel, want in sorted(expected.items()):
|
||||
got = actual.get(rel)
|
||||
if got is None:
|
||||
problems.append('%s/%s is in the lock but missing from the bundle' % (name, rel))
|
||||
elif got['sha256'] != want['sha256']:
|
||||
problems.append(
|
||||
'%s/%s does NOT match the lock (expected sha256 %s..., got %s...)'
|
||||
% (name, rel, want['sha256'][:12], got['sha256'][:12]))
|
||||
elif int(got['size']) != int(want['size']):
|
||||
# Impossible for a matching sha256, so the lock was hand-edited.
|
||||
problems.append(
|
||||
'%s/%s size disagrees with the lock - the lock has been edited by hand'
|
||||
% (name, rel))
|
||||
for rel in sorted(actual):
|
||||
if rel not in expected:
|
||||
problems.append(
|
||||
'%s/%s is in the bundle but NOT in the lock (unexpected extra file)'
|
||||
% (name, rel))
|
||||
return problems
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 3:
|
||||
sys.exit('usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>')
|
||||
bundle_root, lock_path = sys.argv[1], sys.argv[2]
|
||||
if not os.path.exists(lock_path):
|
||||
print('no bundle-lock.json at %s' % lock_path)
|
||||
return 1
|
||||
with open(lock_path) as fh:
|
||||
lock = json.load(fh)
|
||||
problems = verify(bundle_root, lock)
|
||||
for problem in problems:
|
||||
print(problem)
|
||||
return 1 if problems else 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
199
tests/test_bundle_lock.py
Normal file
199
tests/test_bundle_lock.py
Normal file
@@ -0,0 +1,199 @@
|
||||
"""The installer bundle lock: fail-closed behaviour, and parity between the two
|
||||
implementations that read it.
|
||||
|
||||
There are two verifiers on purpose - bundle-lock.ps1 runs at install time on a
|
||||
server where PowerShell is the only thing present, verify_bundle_lock.py runs in
|
||||
the Linux builder without adding pwsh as a build dependency. Two implementations
|
||||
of one rule drift. These tests run BOTH against the same fixtures and fail if
|
||||
they disagree, so the drift shows up here rather than as a bundle that one of
|
||||
them waves through.
|
||||
|
||||
The PowerShell half is skipped where pwsh is absent; the Python half always runs.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
INSTALLER = Path(__file__).resolve().parents[1] / 'deploy' / 'windows' / 'installer'
|
||||
VERIFY_PY = INSTALLER / 'verify_bundle_lock.py'
|
||||
LOCK_PS1 = INSTALLER / 'bundle-lock.ps1'
|
||||
PWSH = shutil.which('pwsh') or shutil.which('powershell')
|
||||
|
||||
pytestmark = pytest.mark.skipif(not VERIFY_PY.exists(), reason='installer not in this tree')
|
||||
|
||||
|
||||
def build_bundle(root):
|
||||
"""A minimal bundle holding every REQUIRED payload directory."""
|
||||
for name, files in (
|
||||
('wheels', {'alembic-1.18.4-py3-none-any.whl': b'wheel-a',
|
||||
'cffi-2.1.0-cp314-cp314-win_amd64.whl': b'wheel-b'}),
|
||||
('python', {'python-3.14.6-amd64.exe': b'py'}),
|
||||
('httpplatformhandler', {'httpPlatformHandler_amd64.msi': b'hph'}),
|
||||
):
|
||||
directory = root / name
|
||||
directory.mkdir(parents=True)
|
||||
for filename, content in files.items():
|
||||
(directory / filename).write_bytes(content)
|
||||
return root
|
||||
|
||||
|
||||
def write_lock(bundle, lock_path):
|
||||
"""Generate the lock the same way refresh-bundle-lock.ps1 does."""
|
||||
sys.path.insert(0, str(INSTALLER))
|
||||
import verify_bundle_lock as verifier
|
||||
|
||||
payloads = {}
|
||||
for name, required, _what in verifier.PAYLOADS:
|
||||
directory = bundle / name
|
||||
if not directory.is_dir():
|
||||
continue
|
||||
payloads[name] = {'required': required, 'files': verifier.payload_files(str(directory))}
|
||||
lock_path.write_text(json.dumps({
|
||||
'schema': 1, 'generated': '2026-08-03T00:00:00Z',
|
||||
'pythontag': 'cp314', 'platform': 'win_amd64', 'payloads': payloads,
|
||||
}))
|
||||
|
||||
|
||||
def check_python(bundle, lock_path):
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(VERIFY_PY), str(bundle), str(lock_path)],
|
||||
capture_output=True, text=True)
|
||||
problems = [line for line in result.stdout.splitlines() if line.strip()]
|
||||
# Exit status and output must agree, or a caller that checks only one of them
|
||||
# gets a different answer from a caller that checks the other.
|
||||
assert (result.returncode != 0) == bool(problems), result.stdout
|
||||
return problems
|
||||
|
||||
|
||||
def check_powershell(bundle, lock_path):
|
||||
script = (
|
||||
". '%s'; $p = Test-BundleLock -BundleRoot '%s' -Lock (Read-BundleLock '%s'); "
|
||||
"if ($p) { $p -join \"`n\" }" % (LOCK_PS1, bundle, lock_path))
|
||||
result = subprocess.run([PWSH, '-NoProfile', '-Command', script],
|
||||
capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return [line for line in result.stdout.splitlines() if line.strip()]
|
||||
|
||||
|
||||
def check_both(bundle, lock_path):
|
||||
"""Returns the Python verdict, asserting PowerShell reports the same set.
|
||||
|
||||
Compared as sets: both sort their output, but PowerShell's Sort-Object is
|
||||
culture-aware and Python's sorted() is ordinal, so the two can order the same
|
||||
findings differently. What must never differ is WHICH problems are found.
|
||||
"""
|
||||
problems = check_python(bundle, lock_path)
|
||||
if PWSH and LOCK_PS1.exists():
|
||||
assert sorted(check_powershell(bundle, lock_path)) == sorted(problems), (
|
||||
'verify_bundle_lock.py and bundle-lock.ps1 disagree')
|
||||
return problems
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def locked(tmp_path):
|
||||
bundle = build_bundle(tmp_path / 'bundle')
|
||||
lock = tmp_path / 'bundle-lock.json'
|
||||
write_lock(bundle, lock)
|
||||
return bundle, lock
|
||||
|
||||
|
||||
def test_untouched_bundle_passes(locked):
|
||||
bundle, lock = locked
|
||||
assert check_both(bundle, lock) == []
|
||||
|
||||
|
||||
def test_extra_file_fails(locked):
|
||||
"""The gap pip's own hash checking leaves: a stale wheel nobody asked for."""
|
||||
bundle, lock = locked
|
||||
(bundle / 'wheels' / 'stale-0.1-py3-none-any.whl').write_bytes(b'left over')
|
||||
problems = check_both(bundle, lock)
|
||||
assert any('stale-0.1' in p and 'NOT in the lock' in p for p in problems)
|
||||
|
||||
|
||||
def test_altered_file_fails(locked):
|
||||
bundle, lock = locked
|
||||
(bundle / 'wheels' / 'cffi-2.1.0-cp314-cp314-win_amd64.whl').write_bytes(b'swapped')
|
||||
problems = check_both(bundle, lock)
|
||||
assert any('does NOT match the lock' in p for p in problems)
|
||||
|
||||
|
||||
def test_missing_file_fails(locked):
|
||||
bundle, lock = locked
|
||||
(bundle / 'wheels' / 'alembic-1.18.4-py3-none-any.whl').unlink()
|
||||
problems = check_both(bundle, lock)
|
||||
assert any('missing from the bundle' in p for p in problems)
|
||||
|
||||
|
||||
def test_missing_required_payload_fails(locked):
|
||||
bundle, lock = locked
|
||||
shutil.rmtree(bundle / 'python')
|
||||
problems = check_both(bundle, lock)
|
||||
assert any(p.startswith('python/') for p in problems)
|
||||
|
||||
|
||||
def test_unlocked_payload_directory_fails(locked):
|
||||
"""An optional payload dropped in after the lock was made is still unreviewed."""
|
||||
bundle, lock = locked
|
||||
(bundle / 'mysql').mkdir()
|
||||
(bundle / 'mysql' / 'mysql-8.0.46-winx64.msi').write_bytes(b'msi')
|
||||
problems = check_both(bundle, lock)
|
||||
assert any('mysql/ is present but is not in bundle-lock.json' in p for p in problems)
|
||||
|
||||
|
||||
def test_absent_optional_payload_is_fine(locked):
|
||||
bundle, lock = locked
|
||||
assert not (bundle / 'mysql').exists()
|
||||
assert check_both(bundle, lock) == []
|
||||
|
||||
|
||||
def test_missing_lock_fails_closed(locked):
|
||||
"""No lock is a failure, not a pass. The build must not fall through to 'fine'."""
|
||||
bundle, lock = locked
|
||||
lock.unlink()
|
||||
result = subprocess.run([sys.executable, str(VERIFY_PY), str(bundle), str(lock)],
|
||||
capture_output=True, text=True)
|
||||
assert result.returncode != 0
|
||||
assert 'no bundle-lock.json' in result.stdout
|
||||
|
||||
|
||||
def test_malformed_lock_reports_rather_than_crashing(locked):
|
||||
"""A truncated lock must produce a sentence, not a stack trace.
|
||||
|
||||
shopdb-install.ps1 runs under Set-StrictMode 2.0, where reading a property
|
||||
that is not there throws. Both verifiers have to survive a lock missing the
|
||||
section they read first.
|
||||
"""
|
||||
bundle, lock = locked
|
||||
lock.write_text(json.dumps({'schema': 1, 'pythontag': 'cp314'}))
|
||||
problems = check_both(bundle, lock)
|
||||
assert problems == ['bundle-lock.json has no "payloads" section']
|
||||
|
||||
|
||||
def test_lock_missing_files_section_is_not_a_silent_pass(locked):
|
||||
"""A payload entry with no file list must not read as 'nothing expected, fine'."""
|
||||
bundle, lock = locked
|
||||
data = json.loads(lock.read_text())
|
||||
del data['payloads']['wheels']['files']
|
||||
lock.write_text(json.dumps(data))
|
||||
problems = check_both(bundle, lock)
|
||||
assert problems, 'an entry with no file list let every wheel through'
|
||||
assert all('wheels/' in p for p in problems)
|
||||
|
||||
|
||||
def test_payload_list_matches_powershell():
|
||||
"""Both files enumerate the payload directories. Same names, same required
|
||||
flags, same order - a directory that is required in one and optional in the
|
||||
other means the builder and the installer disagree about what may be absent.
|
||||
"""
|
||||
sys.path.insert(0, str(INSTALLER))
|
||||
import verify_bundle_lock as verifier
|
||||
|
||||
declared = re.findall(r"Name\s*=\s*'([a-z]+)'\s*;\s*Required\s*=\s*\$(true|false)",
|
||||
LOCK_PS1.read_text())
|
||||
assert declared, 'could not find $script:BundlePayloads in bundle-lock.ps1'
|
||||
assert declared == [(name, str(required).lower()) for name, required, _ in verifier.PAYLOADS]
|
||||
Reference in New Issue
Block a user