ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release 0.8.0: Windows installer, twelve fixes from a real Server 2019 install, operator docs

Browse Source
This commit is contained in:
Cameron Proudlock
2026-08-05 07:36:43 -04:00
parent 208a572fd7
commit d8e3280dfc
27 changed files with 3189 additions and 396 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
CHANGELOG.md
View File

@@ -10,7 +10,71 @@ ADR-007 and ADR-002.
## [Unreleased] ## [Unreleased]
### Added ## [0.8.0] - 2026-08-05
First release to carry the Windows installer. Everything below shipped after
v0.7.0 was tagged, and a complete install was exercised end to end on Windows
Server 2019 before this release was cut.
### Fixed - installer, from a real Server 2019 install
- `packaging` was imported by the plugin loader but declared nowhere. It reached
development and CI only as a dependency of pytest, so the whole suite passed
while a virtual environment built from `requirements.txt` alone - which is
exactly what the installer builds - could not import the application at all.
`tests/test_runtime_dependencies.py` now fails on any runtime import that is
not a declared dependency.
- IIS returned 500.52 before the application was ever launched. `web.config`
declared `<allowedServerVariables>` for the X-Forwarded-For rule, and that
section ships `overrideModeDefault="Deny"`, so the whole file was rejected.
The installer now permits the single variable at server level instead of
unlocking the section for every site on the machine, and repairs a
`web.config` an earlier build had made unusable.
- The config unlock ran before the application it unlocks existed, so the scoped
form could never succeed on a first install and every install silently fell
back to granting handler delegation server-wide.
- The stage 5 smoke test discarded the status code and error page it had already
received, reporting "site did not return 200" for a fault IIS had named. It
now records both, plus the tail of the application log.
- Database dumps were readable by every authenticated user: a directory created
under ProgramData inherits `Users:RX`, and the owner-only ACL was applied only
when the installer itself created it.
- The uninstaller ran the 32-bit PowerShell, which cannot see IIS, so the site,
pool and application survived a "successful" uninstall pointing at a deleted
directory.
- Uninstall matched applications by alias alone and would remove an unrelated
application of the same name under another site.
- Wizard input reached a command line unchecked: unvalidated ports, a drive-root
path that escaped its own quote, and a password written as ANSI but read back
as UTF-8, which reported a correct non-ASCII password as wrong.
- Plugin deregistration could never succeed - it omitted `--yes` against a
command that prompts - while the plugin directory was deleted regardless.
- Preflight rows were drawn past the bottom of the panel and silently vanished;
the failures loop had no cap at all.
### Changed - installer behaviour
- "Is this a re-run of my install?" is answered from a durable install record
rather than inferred from the state of the machine. Nothing on a server says
who created its database tables, so a retry after a failed first install was
taken for an upgrade of a working system: it demanded a mandatory backup of a
database its own failed attempt had written, then refused to prune tables it
had created minutes earlier. During unfinished first provisioning the backup
is advisory and prune may force; on an established install both are unchanged.
- The installer offers every bundled plugin, so one build serves any site
instead of one build per plugin profile.
### Added - operator documentation and diagnostics
- `docs/UPDATES-WINDOWS.md` - what operators should expect from future updates,
bug fixes and security releases, including downtime, what is preserved, and
the effect on other sites sharing the same IIS server.
- `docs/RELEASING-WINDOWS.md` - how to build and release, and the two known gaps.
- `deploy/windows/shopdb-diagnose.py` - collects what IIS answers, the config
lock state, the application logs and the ACLs in one pass, scrubbing secrets
before writing anything.
### Added - the installer itself
- Air-gapped Windows installer (`deploy/windows/installer/`). One `.exe` per - Air-gapped Windows installer (`deploy/windows/installer/`). One `.exe` per
site, built from that site's plugin profile, containing Python, the wheels, site, built from that site's plugin profile, containing Python, the wheels,

20
deploy/site-profile-universal.json Normal file
View File

@@ -0,0 +1,20 @@
{
"site": "universal",
"plugins": [
"computers",
"employees",
"geenforce",
"knowledgebase",
"machines",
"measuringtools",
"network",
"notifications",
"printedparts",
"printers",
"slides",
"usb",
"warranty"
],
"locked": [],
"_comment": "The profile the released Windows installer is built from. Every bundled plugin that carries a manifest, so ONE exe serves any site: the wizard offers all of them and the operator ticks what that site uses. Plugins left unticked are never installed, and 'flask plugin prune-schema' drops their tables at provisioning (ADR-014). 'applications' is deliberately absent - it is manifest-less core and always ships. Build with: deploy/windows/installer/build-installer.sh deploy/site-profile-universal.json <repo>. Use site-profile.example.json instead only when a site genuinely needs a lean build; see ADR-013."
}

6
deploy/windows/installer/README.md
View File

@@ -53,7 +53,9 @@ bundle/wheels/ pip download -r requirements.txt --only-binary=:all
bundle/python/ python-3.14.x-amd64.exe bundle/python/ python-3.14.x-amd64.exe
bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below) bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below)
bundle/mysql/ mysql-8.0.x-winx64.msi (bundled-database option only) bundle/vcredist/ VC_redist.x64.exe (MySQL will not install without it)
bundle/mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)
bundle/mysqlclient/ mysql.exe, mysqldump.exe + libcrypto/libssl (backups)
``` ```
Then compile on Windows: Then compile on Windows:
@@ -235,7 +237,7 @@ database is an upgrade, and treating it as fresh would drop tables.
## Testing notes ## Testing notes
Verified end to end on Windows Server 2025 against both a bundled MySQL 8.0 and Verified end to end on Windows Server 2025 against both a bundled MySQL 8.4 LTS and
an existing MySQL 5.6: fresh install, upgrade, re-run idempotency, failure and an existing MySQL 5.6: fresh install, upgrade, re-run idempotency, failure and
rollback, uninstall, and both deployment methods including switching between rollback, uninstall, and both deployment methods including switching between
them. them.

211
deploy/windows/installer/ShopDBFlask.iss
View File

@@ -4,7 +4,7 @@
; ;
; DESIGN NOTE - why this is a thin wrapper and not a reimplementation: ; DESIGN NOTE - why this is a thin wrapper and not a reimplementation:
; every install action lives in shopdb-install.ps1, which is verified end to end ; every install action lives in shopdb-install.ps1, which is verified end to end
; on Windows Server 2025 against both a bundled MySQL 8.0 and an existing MySQL ; on Windows Server 2025 against both a bundled MySQL 8.4 LTS and an existing MySQL
; 5.6. Reimplementing any of it in Pascal Script would create a second code path ; 5.6. Reimplementing any of it in Pascal Script would create a second code path
; that nobody tests. This file only collects operator input, runs the stages in ; that nobody tests. This file only collects operator input, runs the stages in
; order, and reports which one failed. ; order, and reports which one failed.
@@ -181,6 +181,7 @@ Filename: "{win}\Sysnative\WindowsPowerShell\v1.0\powershell.exe"; \
var var
DbChoicePage: TInputOptionWizardPage; DbChoicePage: TInputOptionWizardPage;
DbDetailsPage: TInputQueryWizardPage; DbDetailsPage: TInputQueryWizardPage;
DbCredsPage: TInputQueryWizardPage;
SitePage: TInputQueryWizardPage; SitePage: TInputQueryWizardPage;
PreflightPage: TWizardPage; PreflightPage: TWizardPage;
BannerPanel: TPanel; BannerPanel: TPanel;
@@ -402,33 +403,40 @@ begin
'choose the second option - installing a second server would collide on ' + 'choose the second option - installing a second server would collide on ' +
'port 3306.', 'port 3306.',
True, False); True, False);
DbChoicePage.Add('Install the bundled MySQL 8.0 (new servers only)'); DbChoicePage.Add('Install the bundled MySQL 8.4 LTS (new servers only)');
DbChoicePage.Add('Use a MySQL server this site already runs'); DbChoicePage.Add('Use a MySQL server this site already runs');
DbChoicePage.SelectedValueIndex := DB_EXISTING; DbChoicePage.SelectedValueIndex := DB_EXISTING;
DbDetailsPage := CreateInputQueryPage(DbChoicePage.ID, DbDetailsPage := CreateInputQueryPage(DbChoicePage.ID,
'Existing database', 'Connection details', 'Existing database', 'Connection details',
'The database and application user must already exist. Your DBA creates ' + 'The database and application user must already exist - the installer does ' +
'them; the installer does not, so that it never needs administrative ' + 'not create them, so it never needs administrative rights on your database ' +
'rights on your database server. If they need the exact statements:' + #13#10 + 'server. The exact SQL to give your DBA is in the INSTALL-WINDOWS guide.');
' CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;' + #13#10 +
' CREATE USER ''shopdb''@''%'' IDENTIFIED BY ''<password>'';' + #13#10 +
' GRANT ALL PRIVILEGES ON shopdb_flask.* TO ''shopdb''@''%'';' + #13#10 +
'The utf8mb4 charset matters - the default on older servers is latin1.');
DbDetailsPage.Add('Host:', False); DbDetailsPage.Add('Host:', False);
DbDetailsPage.Add('Port:', False); DbDetailsPage.Add('Port:', False);
DbDetailsPage.Add('Database:', False); DbDetailsPage.Add('Database:', False);
DbDetailsPage.Add('Username:', False);
DbDetailsPage.Add('Password:', True);
DbDetailsPage.Values[0] := '127.0.0.1'; DbDetailsPage.Values[0] := '127.0.0.1';
DbDetailsPage.Values[1] := '3306'; DbDetailsPage.Values[1] := '3306';
DbDetailsPage.Values[2] := 'shopdb_flask'; DbDetailsPage.Values[2] := 'shopdb_flask';
DbDetailsPage.Values[3] := 'shopdb';
// SPLIT ACROSS TWO PAGES on purpose. CreateInputQueryPage stacks its fields
// below the description and does not scroll or shrink: with five fields the
// last one is drawn past the bottom of the surface and simply does not appear.
// Trimming the description bought one field back and lost the next, which is
// guessing at a pixel budget that varies with DPI and font. Three fields and
// two fields both fit under any reasonable description, at any scaling.
DbCredsPage := CreateInputQueryPage(DbDetailsPage.ID,
'Existing database', 'Sign-in',
'The account the application uses to reach that database. It needs full ' +
'rights on it, and nothing outside it.');
DbCredsPage.Add('Username:', False);
DbCredsPage.Add('Password:', True);
DbCredsPage.Values[0] := 'shopdb';
// How the application is published. Offered only when the bundle actually // How the application is published. Offered only when the bundle actually
// carries a subpath SPA build - Vite compiles the base path in, so this can // carries a subpath SPA build - Vite compiles the base path in, so this can
// never be a pure runtime switch. // never be a pure runtime switch.
DeployPage := CreateInputOptionPage(DbDetailsPage.ID, DeployPage := CreateInputOptionPage(DbCredsPage.ID,
'Address', 'How should people reach ShopDB-Flask?', 'Address', 'How should people reach ShopDB-Flask?',
'Both options serve the same application. The second needs no new DNS name ' 'Both options serve the same application. The second needs no new DNS name '
+ 'and no port number, because it rides this server''s existing address.', + 'and no port number, because it rides this server''s existing address.',
@@ -647,7 +655,7 @@ end;
// If nothing is wrong the page collapses to a single reassuring statement. // If nothing is wrong the page collapses to a single reassuring statement.
procedure RenderPreflight(Lines: TArrayOfString; var HasBlockers: Boolean); procedure RenderPreflight(Lines: TArrayOfString; var HasBlockers: Boolean);
var var
I, Y, Shown, Fails, Warns, Passes: Integer; I, Y, Shown, Fails, Warns, Passes, RowHeight, MaxRows, Hidden: Integer;
Parts: TArrayOfString; Parts: TArrayOfString;
Check, Detail, Fix: String; Check, Detail, Fix: String;
begin begin
@@ -655,6 +663,16 @@ begin
Y := ScaleY(4); Y := ScaleY(4);
Shown := 0; Shown := 0;
// How many rows actually FIT, measured, not guessed. The old cap of 6 bore no
// relation to the panel, which holds roughly three: at four or five notes the
// rows below were drawn past the bottom edge and silently vanished - taking
// with them the warning that the bundled MySQL collides on port 3306, which is
// exactly what the operator needs before the very next page.
RowHeight := ScaleY(46) + ScaleY(ROW_GAP);
if RowHeight < 1 then RowHeight := 1;
MaxRows := DetailPanel.Height div RowHeight;
if MaxRows < 1 then MaxRows := 1;
// Blockers first, then warnings - a second pass rather than one, so severity // Blockers first, then warnings - a second pass rather than one, so severity
// ordering does not depend on the order the checks happen to run in. // ordering does not depend on the order the checks happen to run in.
for I := 0 to GetArrayLength(Lines) - 1 do for I := 0 to GetArrayLength(Lines) - 1 do
@@ -668,6 +686,9 @@ begin
for I := 0 to GetArrayLength(Lines) - 1 do for I := 0 to GetArrayLength(Lines) - 1 do
begin begin
// Blockers are capped too. Without this the FAIL loop drew every failure,
// however many, straight off the bottom of the panel.
if Shown >= MaxRows then Break;
Parts := StringSplit(Lines[I], ['|'], stAll); Parts := StringSplit(Lines[I], ['|'], stAll);
if GetArrayLength(Parts) < 4 then Continue; if GetArrayLength(Parts) < 4 then Continue;
if Parts[0] <> 'FAIL' then Continue; if Parts[0] <> 'FAIL' then Continue;
@@ -682,7 +703,7 @@ begin
for I := 0 to GetArrayLength(Lines) - 1 do for I := 0 to GetArrayLength(Lines) - 1 do
begin begin
if Shown >= 6 then Break; if Shown >= MaxRows then Break;
Parts := StringSplit(Lines[I], ['|'], stAll); Parts := StringSplit(Lines[I], ['|'], stAll);
if GetArrayLength(Parts) < 4 then Continue; if GetArrayLength(Parts) < 4 then Continue;
if Parts[0] <> 'WARN' then Continue; if Parts[0] <> 'WARN' then Continue;
@@ -719,8 +740,13 @@ begin
+ 'verified. Nothing needs your attention.'); + 'verified. Nothing needs your attention.');
end; end;
if (Shown >= 6) and (Warns > 6 - Fails) then // Count what was actually left out rather than inferring it from the old cap,
FooterText.Caption := 'Some notes are not shown. The full check is in the install log.' // which under-reported: with 2 failures and 5 warnings it claimed everything
// was shown while four rows had been dropped.
Hidden := (Fails + Warns) - Shown;
if Hidden > 0 then
FooterText.Caption := IntToStr(Hidden) + ' further item(s) are not shown here. '
+ 'All of them are in the install log.'
else else
FooterText.Caption := IntToStr(Passes) + ' checks passed. Nothing has been changed on this server.'; FooterText.Caption := IntToStr(Passes) + ' checks passed. Nothing has been changed on this server.';
end; end;
@@ -829,10 +855,76 @@ begin
Log('[shopdb] pre-fill base directory: ' + Result); Log('[shopdb] pre-fill base directory: ' + Result);
end; end;
// Everything typed in the wizard ends up on a command line that
// CommandLineToArgvW parses, and reaches PowerShell parameters typed [int] or
// [string]. A blank, a space, or a trailing backslash therefore has to be caught
// HERE - by the time the script sees it, the arguments have already shifted.
function IsValidPort(const S: String): Boolean;
var
I, N: Integer;
begin
Result := False;
if (S = '') or (Length(S) > 5) then Exit;
for I := 1 to Length(S) do
if (S[I] < '0') or (S[I] > '9') then Exit; // also rejects spaces and signs
N := StrToIntDef(S, -1);
Result := (N >= 1) and (N <= 65535);
end;
function IsAsciiOnly(const S: String): Boolean;
var
I: Integer;
begin
Result := True;
for I := 1 to Length(S) do
if Ord(S[I]) > 126 then
begin
Result := False;
Exit;
end;
end;
// A path that ends in a backslash - a drive root such as D:\ - closes the
// argument with \" , which CommandLineToArgvW reads as an ESCAPED quote. The
// argument never terminates and every argument after it shifts one place.
// Doubling the backslash inside the quotes is the documented way out.
function QuotePathArg(const P: String): String;
begin
if (P <> '') and (P[Length(P)] = '\') then
Result := '"' + P + '\' + '"'
else
Result := '"' + P + '"';
end;
function NextButtonClick(CurPageID: Integer): Boolean; function NextButtonClick(CurPageID: Integer): Boolean;
begin begin
Result := True; Result := True;
// A drive root is refused outright rather than quoted around. Uninstall
// removes the application directory recursively, so accepting D:\ here would
// mean uninstalling ShopDB-Flask wipes the whole drive.
if CurPageID = wpSelectDir then
begin
if Length(WizardDirValue) <= 3 then
begin
MsgBox('Choose a folder rather than a whole drive.' + #13#10#13#10
+ 'Removing ShopDB-Flask deletes the folder it was installed into, '
+ 'so installing to a drive root would delete everything on that '
+ 'drive when it is uninstalled.', mbError, MB_OK);
Result := False;
end;
end;
if CurPageID = SitePage.ID then
begin
if not IsValidPort(SitePage.Values[1]) then
begin
MsgBox('Enter the port as a number between 1 and 65535.', mbError, MB_OK);
Result := False;
end;
end;
if CurPageID = wpWelcome then if CurPageID = wpWelcome then
RunPreflight; RunPreflight;
@@ -865,8 +957,34 @@ begin
MsgBox('Enter the database host.', mbError, MB_OK); MsgBox('Enter the database host.', mbError, MB_OK);
Result := False; Result := False;
end end
else if (DbDetailsPage.Values[4] = '') and else if not IsValidPort(DbDetailsPage.Values[1]) then
(not FileExists(InstalledDir + '.env')) then begin
MsgBox('Enter the database port as a number between 1 and 65535.'
+ #13#10#13#10 + 'MySQL uses 3306 unless it was changed.',
mbError, MB_OK);
Result := False;
end;
end;
end;
if CurPageID = DbCredsPage.ID then
begin
if not UseBundledDb then
begin
if DbCredsPage.Values[0] = '' then
begin
MsgBox('Enter the username the application connects with.', mbError, MB_OK);
Result := False;
end
// Blank is allowed when the password is already ON DISK in a form the
// installer can read: .env from a previous install, or the ACL'd .dbpass
// handoff stage 0 leaves when it created the database itself. Stage 2
// consumes .dbpass on its own, so demanding the password here blocked the
// operator on something the installer already had - and sent them looking
// for a generated secret they were never meant to handle.
else if (DbCredsPage.Values[1] = '') and
(not FileExists(InstalledDir + '.env')) and
(not FileExists(InstalledDir + '.dbpass')) then
begin begin
// Only required on a FRESH install. On an upgrade, blank means "keep the // Only required on a FRESH install. On an upgrade, blank means "keep the
// password already in .env", so the operator never has to know it. // password already in .env", so the operator never has to know it.
@@ -913,6 +1031,13 @@ begin
if (CurPageID = DbDetailsPage.ID) and (not DbPageReady) then if (CurPageID = DbDetailsPage.ID) and (not DbPageReady) then
begin begin
DbPageReady := True; DbPageReady := True;
// A stage-0 handoff means the database was created by a previous run of this
// installer and the password is already on disk. Nobody should be asked to
// find or retype a secret the installer generated.
if FileExists(InstalledDir + '.dbpass') then
DbCredsPage.SubCaptionLabel.Caption :=
'This server already has a database created by this installer. Leave the '
+ 'password blank and it will be used automatically.';
if LoadStringsFromFile(InstalledDir + '.env', Lines) then if LoadStringsFromFile(InstalledDir + '.env', Lines) then
for I := 0 to GetArrayLength(Lines) - 1 do for I := 0 to GetArrayLength(Lines) - 1 do
if Pos('DATABASE_URL=', Lines[I]) = 1 then if Pos('DATABASE_URL=', Lines[I]) = 1 then
@@ -924,7 +1049,7 @@ begin
Creds := Copy(Existing, 1, Pos('@', Existing) - 1); Creds := Copy(Existing, 1, Pos('@', Existing) - 1);
Rest := Copy(Existing, Pos('@', Existing) + 1, Length(Existing)); Rest := Copy(Existing, Pos('@', Existing) + 1, Length(Existing));
if Pos(':', Creds) > 0 then if Pos(':', Creds) > 0 then
DbDetailsPage.Values[3] := Copy(Creds, 1, Pos(':', Creds) - 1); DbCredsPage.Values[0] := Copy(Creds, 1, Pos(':', Creds) - 1);
if Pos(':', Rest) > 0 then if Pos(':', Rest) > 0 then
begin begin
DbDetailsPage.Values[0] := Copy(Rest, 1, Pos(':', Rest) - 1); DbDetailsPage.Values[0] := Copy(Rest, 1, Pos(':', Rest) - 1);
@@ -940,10 +1065,13 @@ begin
end; end;
// Password intentionally left blank: blank means "keep the current // Password intentionally left blank: blank means "keep the current
// one", so an upgrade never needs the operator to know it. // one", so an upgrade never needs the operator to know it.
DbDetailsPage.Values[4] := ''; DbCredsPage.Values[1] := '';
DbDetailsPage.SubCaptionLabel.Caption := DbDetailsPage.SubCaptionLabel.Caption :=
'These are the settings this server is using now. Leave the password ' 'These are the settings this server is using now.';
+ 'blank to keep the current one.'; // The hint belongs where the password field actually is - it moved to
// the next page when this one was split.
DbCredsPage.SubCaptionLabel.Caption :=
'Leave the password blank to keep the one this server already uses.';
end; end;
end; end;
@@ -994,7 +1122,7 @@ begin
end; end;
// The bundled path generates its own credentials, so asking for them would be // The bundled path generates its own credentials, so asking for them would be
// meaningless - and any value typed here would be silently ignored. // meaningless - and any value typed here would be silently ignored.
Result := (PageID = DbDetailsPage.ID) and UseBundledDb; Result := ((PageID = DbDetailsPage.ID) or (PageID = DbCredsPage.ID)) and UseBundledDb;
end; end;
// Runs from ssPostInstall, NOT PrepareToInstall. // Runs from ssPostInstall, NOT PrepareToInstall.
@@ -1019,22 +1147,31 @@ var
Cmd: String; Cmd: String;
begin begin
if not FileExists(LnkPath) then Exit; if not FileExists(LnkPath) then Exit;
// THREE quotes each side, not four. Four is an escaped quote twice over, which
// keeps the whole expression inside one Pascal literal: LnkPath was never
// interpolated, PowerShell got the bare word "LnkPath", failed to parse, and
// no shortcut ever received the elevation flag. Self-elevation hid it - the
// only symptom was a second window after an extra prompt.
Cmd := '-NoProfile -ExecutionPolicy Bypass -Command "' Cmd := '-NoProfile -ExecutionPolicy Bypass -Command "'
+ '$p='''' + LnkPath + ''''; ' + '$p=''' + LnkPath + '''; '
+ '$b=[IO.File]::ReadAllBytes($p); ' + '$b=[IO.File]::ReadAllBytes($p); '
+ '$b[21]=$b[21] -bor 0x20; ' + '$b[21]=$b[21] -bor 0x20; '
+ '[IO.File]::WriteAllBytes($p,$b)"'; + '[IO.File]::WriteAllBytes($p,$b)"';
Exec(PowerShellPath, Cmd, '', SW_HIDE, ewWaitUntilTerminated, ResultCode); Exec(PowerShellPath, Cmd, '', SW_HIDE, ewWaitUntilTerminated, ResultCode);
if ResultCode <> 0 then
Log('[shopdb] could not mark ' + LnkPath + ' run-as-administrator (exit '
+ IntToStr(ResultCode) + '); the console will self-elevate instead');
end; end;
function RunInstallStages: String; function RunInstallStages: String;
var var
ResultCode: Integer; ResultCode: Integer;
PwFile, Args, Common: String; PwFile, Args, Common: String;
PwLines: TArrayOfString;
begin begin
Result := ''; Result := '';
Common := '-BundleRoot "' + ExpandConstant('{tmp}\shopdb-bundle') + '"' + Common := '-BundleRoot ' + QuotePathArg(ExpandConstant('{tmp}\shopdb-bundle')) +
' -AppRoot "' + ExpandConstant('{app}') + '"' + ' -AppRoot ' + QuotePathArg(ExpandConstant('{app}')) +
' -SiteHost "' + SitePage.Values[0] + '"' + ' -SiteHost "' + SitePage.Values[0] + '"' +
' -SitePort ' + SitePage.Values[1] + ' -SitePort ' + SitePage.Values[1] +
' -OnFailure never' + ' -OnFailure never' +
@@ -1070,10 +1207,16 @@ begin
// PowerShell transcripts. The installer shreds the file after reading it. // PowerShell transcripts. The installer shreds the file after reading it.
// No password typed on an upgrade means "leave .env alone" - so send no // No password typed on an upgrade means "leave .env alone" - so send no
// password file, and the installer keeps the existing DATABASE_URL. // password file, and the installer keeps the existing DATABASE_URL.
if DbDetailsPage.Values[4] <> '' then if DbCredsPage.Values[1] <> '' then
begin begin
PwFile := ExpandConstant('{tmp}\dbpw.txt'); PwFile := ExpandConstant('{tmp}\dbpw.txt');
SaveStringToFile(PwFile, DbDetailsPage.Values[4] + #13#10, False); // UTF-8, no BOM. SaveStringToFile writes an AnsiString, but the installer
// reads this back with -Encoding UTF8 - so any non-ASCII character in the
// password came out mangled and the database rejected a password that was
// typed correctly, reported as "wrong username or password".
SetArrayLength(PwLines, 1);
PwLines[0] := DbCredsPage.Values[1];
SaveStringsToUTF8FileWithoutBOM(PwFile, PwLines, False);
end end
else else
PwFile := ''; PwFile := '';
@@ -1081,7 +1224,7 @@ begin
' -DbHost "' + DbDetailsPage.Values[0] + '"' + ' -DbHost "' + DbDetailsPage.Values[0] + '"' +
' -DbPort ' + DbDetailsPage.Values[1] + ' -DbPort ' + DbDetailsPage.Values[1] +
' -DbName "' + DbDetailsPage.Values[2] + '"' + ' -DbName "' + DbDetailsPage.Values[2] + '"' +
' -DbUser "' + DbDetailsPage.Values[3] + '"'; ' -DbUser "' + DbCredsPage.Values[0] + '"';
if PwFile <> '' then Args := Args + ' -DbPasswordFile "' + PwFile + '"'; if PwFile <> '' then Args := Args + ' -DbPasswordFile "' + PwFile + '"';
end; end;
@@ -1169,6 +1312,12 @@ end;
; Delegate to the same tested script rather than duplicating removal logic here. ; Delegate to the same tested script rather than duplicating removal logic here.
; It removes the site, app pool, firewall rule and application directory, and ; It removes the site, app pool, firewall rule and application directory, and
; deliberately does NOT drop the database or uninstall MySQL. ; deliberately does NOT drop the database or uninstall MySQL.
Filename: "powershell.exe"; \ ;
; Sysnative, for the same reason as the [Run] entry above: the uninstaller is a
; 32-bit process, so a bare "powershell.exe" is resolved through WOW64 to the
; 32-bit PowerShell, which cannot see the IIS provider at all. Removal then
; skipped the site, application pool and application - leaving them pointing at
; a directory it HAD deleted - while Windows reported a clean uninstall.
Filename: "{win}\Sysnative\WindowsPowerShell\v1.0\powershell.exe"; \
Parameters: "-NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-install.ps1"" -Stage uninstall -BundleRoot ""{app}"" -AppRoot ""{app}"" -OnFailure never"; \ Parameters: "-NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-install.ps1"" -Stage uninstall -BundleRoot ""{app}"" -AppRoot ""{app}"" -OnFailure never"; \
RunOnceId: "ShopDBFlaskUninstall"; Flags: waituntilterminated runhidden RunOnceId: "ShopDBFlaskUninstall"; Flags: waituntilterminated runhidden

2
deploy/windows/installer/build-installer.ps1
View File

@@ -286,7 +286,7 @@ refresh-bundle-lock.ps1, read the diff, and commit the new lock).
# --- summary ---------------------------------------------------------------- # --- summary ----------------------------------------------------------------
Write-Host '' Write-Host ''
Say " Bundle staged at: $BundleRoot" 'White' Say " Bundle staged at: $BundleRoot" 'White'
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'mysql')) { foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'vcredist', 'mysqlclient', 'mysql')) {
$p = Join-Path $BundleRoot $d $p = Join-Path $BundleRoot $d
if (Test-Path $p) { if (Test-Path $p) {
$mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB) $mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB)

5
deploy/windows/installer/build-installer.sh
View File

@@ -82,7 +82,7 @@ done
echo "" echo ""
echo "Bundle staged at: $BUNDLE" echo "Bundle staged at: $BUNDLE"
for d in app wheels python httpplatformhandler urlrewrite mysql; do for d in app wheels python httpplatformhandler urlrewrite vcredist mysqlclient mysql; do
if [ -d "$BUNDLE/$d" ]; then if [ -d "$BUNDLE/$d" ]; then
printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)" printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)"
else else
@@ -125,7 +125,8 @@ else
echo " python/ python-3.14.x-amd64.exe" echo " python/ python-3.14.x-amd64.exe"
echo " httpplatformhandler/ httpPlatformHandler_amd64.msi" echo " httpplatformhandler/ httpPlatformHandler_amd64.msi"
echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)" echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)"
echo " mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)" echo " vcredist/ VC_redist.x64.exe (MySQL requires it)"
echo " mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)"
echo "" echo ""
echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:" echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:"
echo " pwsh ./refresh-bundle-lock.ps1 # review the diff" echo " pwsh ./refresh-bundle-lock.ps1 # review the diff"

411
deploy/windows/installer/bundle-lock.json
View File

@@ -1,197 +1,240 @@
{ {
"schema": 1, "schema": 1,
"generated": "2026-08-03T15:45:11Z", "generated": "2026-08-04T23:11:22Z",
"pythontag": "cp314", "pythontag": "cp314",
"platform": "win_amd64", "platform": "win_amd64",
"note": "Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.", "note": "Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.",
"payloads": { "payloads": {
"wheels": {
"required": true,
"files": {
"jinja2-3.1.6-py3-none-any.whl": {
"size": 134899,
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67"
},
"itsdangerous-2.2.0-py3-none-any.whl": {
"size": 16234,
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef"
},
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
"size": 187937,
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb"
},
"email_validator-2.3.0-py3-none-any.whl": {
"size": 35604,
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4"
},
"urllib3-2.7.0-py3-none-any.whl": {
"size": 131087,
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897"
},
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
"size": 22698,
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb"
},
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
"size": 25125,
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0"
},
"pycparser-3.0-py3-none-any.whl": {
"size": 48172,
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992"
},
"requests-2.33.1-py3-none-any.whl": {
"size": 64947,
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a"
},
"blinker-1.9.0-py3-none-any.whl": {
"size": 8458,
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc"
},
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
"size": 239835,
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033"
},
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
"size": 16582,
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258"
},
"alembic-1.18.4-py3-none-any.whl": {
"size": 263893,
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a"
},
"pyjwt-2.12.1-py3-none-any.whl": {
"size": 29726,
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c"
},
"mako-1.3.12-py3-none-any.whl": {
"size": 78521,
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9"
},
"waitress-3.0.2-py3-none-any.whl": {
"size": 56232,
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e"
},
"colorama-0.4.6-py2.py3-none-any.whl": {
"size": 25335,
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
},
"certifi-2026.4.22-py3-none-any.whl": {
"size": 135707,
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a"
},
"cachelib-0.13.0-py3-none-any.whl": {
"size": 20914,
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516"
},
"pymysql-1.1.3-py3-none-any.whl": {
"size": 45356,
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289"
},
"werkzeug-3.1.8-py3-none-any.whl": {
"size": 226459,
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50"
},
"idna-3.13-py3-none-any.whl": {
"size": 68629,
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3"
},
"python_dotenv-1.2.2-py3-none-any.whl": {
"size": 22101,
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a"
},
"flask_marshmallow-1.5.0-py3-none-any.whl": {
"size": 12161,
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6"
},
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
"size": 15341,
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581"
},
"flask_cors-6.0.2-py3-none-any.whl": {
"size": 13257,
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a"
},
"flask_caching-2.4.0-py3-none-any.whl": {
"size": 28727,
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486"
},
"tzdata-2026.3-py2.py3-none-any.whl": {
"size": 348168,
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931"
},
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
"size": 159634,
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f"
},
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
"size": 2144204,
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5"
},
"marshmallow-4.3.0-py3-none-any.whl": {
"size": 49148,
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46"
},
"dnspython-2.8.0-py3-none-any.whl": {
"size": 331094,
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af"
},
"typing_extensions-4.15.0-py3-none-any.whl": {
"size": 44614,
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"
},
"flask-3.1.3-py3-none-any.whl": {
"size": 103424,
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c"
},
"tabulate-0.10.0-py3-none-any.whl": {
"size": 39814,
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3"
},
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
"size": 18197576,
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49"
},
"Flask_Migrate-4.1.0-py3-none-any.whl": {
"size": 21237,
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d"
},
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
"size": 3840395,
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30"
},
"click-8.3.3-py3-none-any.whl": {
"size": 110502,
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613"
}
}
},
"python": {
"required": true,
"files": {
"python-3.14.6-amd64.exe": {
"size": 30774112,
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd"
}
}
},
"urlrewrite": {
"required": false,
"files": {
"rewrite_amd64_en-US.msi": {
"size": 6078464,
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a"
}
}
},
"httpplatformhandler": { "httpplatformhandler": {
"required": true,
"files": { "files": {
"httpPlatformHandler_amd64.msi": { "httpPlatformHandler_amd64.msi": {
"size": 557056, "sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef",
"sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef" "size": 557056
} }
} },
"required": true
},
"python": {
"files": {
"python-3.14.6-amd64.exe": {
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd",
"size": 30774112
}
},
"required": true
},
"vcredist": {
"files": {
"VC_redist.x64.exe": {
"sha256": "cc0ff0eb1dc3f5188ae6300faef32bf5beeba4bdd6e8e445a9184072096b713b",
"size": 25635768
}
},
"required": false
},
"mysql": {
"files": {
"mysql-8.4.6-winx64.msi": {
"sha256": "868885b2e221409f0170729bb30b2b2ce83440a1a4c735d19b3c225b51045762",
"size": 135081984
}
},
"required": false
},
"wheels": {
"files": {
"email_validator-2.3.0-py3-none-any.whl": {
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4",
"size": 35604
},
"cachelib-0.13.0-py3-none-any.whl": {
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516",
"size": 20914
},
"flask_caching-2.4.0-py3-none-any.whl": {
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486",
"size": 28727
},
"typing_extensions-4.15.0-py3-none-any.whl": {
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548",
"size": 44614
},
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033",
"size": 239835
},
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f",
"size": 159634
},
"colorama-0.4.6-py2.py3-none-any.whl": {
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6",
"size": 25335
},
"Flask_Migrate-4.1.0-py3-none-any.whl": {
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d",
"size": 21237
},
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581",
"size": 15341
},
"flask_marshmallow-1.5.0-py3-none-any.whl": {
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6",
"size": 12161
},
"tzdata-2026.3-py2.py3-none-any.whl": {
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931",
"size": 348168
},
"python_dotenv-1.2.2-py3-none-any.whl": {
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a",
"size": 22101
},
"dnspython-2.8.0-py3-none-any.whl": {
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af",
"size": 331094
},
"pymysql-1.1.3-py3-none-any.whl": {
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289",
"size": 45356
},
"packaging-26.3-py3-none-any.whl": {
"sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c",
"size": 129956
},
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb",
"size": 22698
},
"urllib3-2.7.0-py3-none-any.whl": {
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897",
"size": 131087
},
"marshmallow-4.3.0-py3-none-any.whl": {
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46",
"size": 49148
},
"waitress-3.0.2-py3-none-any.whl": {
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e",
"size": 56232
},
"idna-3.13-py3-none-any.whl": {
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3",
"size": 68629
},
"flask-3.1.3-py3-none-any.whl": {
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c",
"size": 103424
},
"werkzeug-3.1.8-py3-none-any.whl": {
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50",
"size": 226459
},
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0",
"size": 25125
},
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258",
"size": 16582
},
"alembic-1.18.4-py3-none-any.whl": {
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a",
"size": 263893
},
"tabulate-0.10.0-py3-none-any.whl": {
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3",
"size": 39814
},
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb",
"size": 187937
},
"jinja2-3.1.6-py3-none-any.whl": {
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67",
"size": 134899
},
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30",
"size": 3840395
},
"certifi-2026.4.22-py3-none-any.whl": {
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a",
"size": 135707
},
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5",
"size": 2144204
},
"blinker-1.9.0-py3-none-any.whl": {
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc",
"size": 8458
},
"pycparser-3.0-py3-none-any.whl": {
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992",
"size": 48172
},
"flask_cors-6.0.2-py3-none-any.whl": {
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a",
"size": 13257
},
"requests-2.33.1-py3-none-any.whl": {
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a",
"size": 64947
},
"pyjwt-2.12.1-py3-none-any.whl": {
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c",
"size": 29726
},
"click-8.3.3-py3-none-any.whl": {
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613",
"size": 110502
},
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49",
"size": 18197576
},
"mako-1.3.12-py3-none-any.whl": {
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9",
"size": 78521
},
"itsdangerous-2.2.0-py3-none-any.whl": {
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef",
"size": 16234
}
},
"required": true
},
"urlrewrite": {
"files": {
"rewrite_amd64_en-US.msi": {
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a",
"size": 6078464
}
},
"required": false
},
"mysqlclient": {
"files": {
"libcrypto-3-x64.dll": {
"sha256": "e84ec61fc1b07a3c899ae39c1dc3f38591cdd310c043a388e7ab40a6828aa297",
"size": 5187216
},
"mysqldump.exe": {
"sha256": "329410b7a8ae3d68d38e6f5b781610770854d3f11b975060e231dae62c20dc46",
"size": 7142528
},
"mysql.exe": {
"sha256": "bd6880253c1853fc17b1e19129d6e5d9f9c86a2dccf2c7c9e6b878a10baa9770",
"size": 7177336
},
"libssl-3-x64.dll": {
"sha256": "e24fcd6c8ec31a14c97f520b5b960ed360c55ddf0935040eff297e511697633e",
"size": 788616
}
},
"required": false
} }
} }
} }

42
deploy/windows/installer/bundle-lock.ps1
View File

@@ -50,6 +50,7 @@ $script:BundlePayloads = @(
@{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' } @{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' }
@{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' } @{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' }
@{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' } @{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' }
@{ Name = 'vcredist'; Required = $false; What = 'the Visual C++ runtime MySQL requires' }
@{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' } @{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' }
) )
@@ -75,14 +76,35 @@ function Get-FileDigest {
} }
function Get-PayloadFiles { function Get-PayloadFiles {
# Every file in the directory, keyed by its path RELATIVE to that directory <#
# with forward slashes, so a lock generated on Windows reads the same from Every file in the directory, keyed by its path RELATIVE to that directory
# the Bash builder. with forward slashes, so a lock generated on Windows reads the same from
the Bash builder.
The root comes from Get-Item, NOT Resolve-Path, and the prefix is checked
before it is trimmed. Both matter, and a real install proved it:
Inno extracts the bundle under C:\Users\ADMINI~1\AppData\Local\Temp\... -
an 8.3 SHORT path. Resolve-Path kept that short form while Get-ChildItem
returned the long one (Administrator), so the root was five characters
shorter than the prefix it was slicing off. Every relative path came out
mangled - 'wheels/heels/flask.whl' - and the verifier reported all 96 files
as simultaneously missing and unexpected. The payload was fine; the
comparison was not.
Get-Item and Get-ChildItem go through the same provider, so their path
forms agree. The StartsWith guard means that if they ever disagree again
this fails loudly instead of inventing paths.
#>
param([string] $Dir) param([string] $Dir)
$out = @{} $out = @{}
if (-not (Test-Path $Dir)) { return $out } if (-not (Test-Path $Dir)) { return $out }
$root = (Resolve-Path $Dir).Path.TrimEnd('\', '/') $rootItem = Get-Item -LiteralPath $Dir
foreach ($f in (Get-ChildItem $Dir -Recurse -File)) { $root = $rootItem.FullName.TrimEnd('\', '/')
foreach ($f in (Get-ChildItem -LiteralPath $rootItem.FullName -Recurse -File)) {
if (-not $f.FullName.StartsWith($root, [System.StringComparison]::OrdinalIgnoreCase)) {
throw ("cannot place '{0}' beneath '{1}' - path forms disagree (8.3 short name?)" -f $f.FullName, $root)
}
$rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/') $rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/')
$out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length } $out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length }
} }
@@ -137,7 +159,9 @@ function Test-BundleLock {
) )
$problems = @() $problems = @()
$payloads = Get-JsonProperty $Lock 'payloads' $payloads = Get-JsonProperty $Lock 'payloads'
if ($null -eq $payloads) { return @('bundle-lock.json has no "payloads" section') } # ,@(...) for the same reason as the final return: a bare one-element array
# unrolls to a string, and the caller's .Count then measures the wrong thing.
if ($null -eq $payloads) { return ,@('bundle-lock.json has no "payloads" section') }
foreach ($p in $script:BundlePayloads) { foreach ($p in $script:BundlePayloads) {
$name = $p.Name $name = $p.Name
@@ -184,7 +208,11 @@ function Test-BundleLock {
} }
} }
$problems += Test-WheelhouseCoversRequirements -BundleRoot $BundleRoot $problems += Test-WheelhouseCoversRequirements -BundleRoot $BundleRoot
return $problems # The comma keeps this an ARRAY through the return. Without it PowerShell
# unrolls an empty result to $null and a single result to a bare string, and
# every caller that asks for .Count then behaves differently depending on how
# many problems there happen to be.
return ,$problems
} }
function Test-WheelhouseCoversRequirements { function Test-WheelhouseCoversRequirements {

57
deploy/windows/installer/shopdb-admin.ps1
View File

@@ -195,6 +195,8 @@ function Get-DbParts {
function Find-MysqlClient { function Find-MysqlClient {
$candidates = @( $candidates = @(
'C:\MySQL84\bin\mysql.exe',
'C:\Program Files\MySQL\MySQL Server 8.4\bin\mysql.exe',
'C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe', 'C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe',
'C:\mysql56\bin\mysql.exe' 'C:\mysql56\bin\mysql.exe'
) + @(Get-ChildItem 'C:\Program Files\MySQL' -Filter mysql.exe -Recurse -EA SilentlyContinue | ) + @(Get-ChildItem 'C:\Program Files\MySQL' -Filter mysql.exe -Recurse -EA SilentlyContinue |
@@ -312,10 +314,38 @@ function Stop-App {
function Restart-App { function Restart-App {
Head 'Restarting' Head 'Restarting'
# Recycle rather than stop/start: it drains existing requests instead of # A recycle CANNOT start something that is stopped - it is a no-op on a
# cutting them off, and it is what a config change actually needs. # stopped pool. Restarting after a failed upgrade, which is exactly when the
& $AppCmd recycle apppool /apppool.name:$AppPool 2>&1 | ForEach-Object { Say " $_" } # pool is stopped, therefore did nothing and then reported "did not respond"
Start-Sleep -Seconds 2 # in red, as though the application were broken. Start it first when needed.
Import-Module WebAdministration -ErrorAction SilentlyContinue
$started = $true
try {
if (Test-Path "IIS:\AppPools\$AppPool") {
$started = ((Get-Item "IIS:\AppPools\$AppPool").State -eq 'Started')
}
} catch { }
if (-not $started) {
Say ' the application pool is stopped; starting it' 'Yellow'
Start-WebAppPool -Name $AppPool -ErrorAction SilentlyContinue
Start-Sleep -Seconds 3
} else {
# Recycle rather than stop/start: it drains existing requests instead of
# cutting them off, and it is what a config change actually needs.
& $AppCmd recycle apppool /apppool.name:$AppPool 2>&1 | ForEach-Object { Say " $_" }
if ($LASTEXITCODE -ne 0) { Say " recycle reported exit $LASTEXITCODE" 'Yellow' }
Start-Sleep -Seconds 2
}
# A stopped SITE answers nothing however healthy the pool is.
try {
$siteNow = Get-Website -Name $SiteName -ErrorAction SilentlyContinue
if ($siteNow -and $siteNow.State -ne 'Started') {
Say ' the site is stopped; starting it' 'Yellow'
Start-Website -Name $SiteName -ErrorAction SilentlyContinue
Start-Sleep -Seconds 2
}
} catch { }
try { try {
$r = Invoke-WebRequest -Uri ((Get-Deployment).BaseUrl + '/') -UseBasicParsing -TimeoutSec 30 $r = Invoke-WebRequest -Uri ((Get-Deployment).BaseUrl + '/') -UseBasicParsing -TimeoutSec 30
Say (" back up (HTTP {0})" -f $r.StatusCode) 'Green' Say (" back up (HTTP {0})" -f $r.StatusCode) 'Green'
@@ -370,9 +400,28 @@ function Backup-Db {
Head 'Database backup' Head 'Database backup'
$db = Get-DbParts $db = Get-DbParts
if (-not $db) { Say ' no .env found - cannot determine the database' 'Red'; return } if (-not $db) { Say ' no .env found - cannot determine the database' 'Red'; return }
$usingDefault = -not $Dest
if (-not $Dest) { $Dest = 'C:\ProgramData\ShopDB-Flask\backups' } if (-not $Dest) { $Dest = 'C:\ProgramData\ShopDB-Flask\backups' }
if (-not (Test-Path $Dest)) { New-Item -ItemType Directory -Path $Dest -Force | Out-Null } if (-not (Test-Path $Dest)) { New-Item -ItemType Directory -Path $Dest -Force | Out-Null }
# A dump holds every row, including the users table and its password hashes.
# A directory created under ProgramData INHERITS Users:RX, so those hashes
# were readable by every authenticated user on the server whenever this
# command created the directory rather than the installer.
#
# Re-applied on every backup, not only on creation, because this may be
# repairing a directory made by an earlier version.
#
# Only for the default location. A path the operator named is theirs, and
# silently rewriting its ACL is not this command's business - say so instead.
if ($usingDefault) {
& icacls.exe $Dest '/inheritance:r' `
'/grant' 'BUILTIN\Administrators:(OI)(CI)(F)' `
'/grant' 'NT AUTHORITY\SYSTEM:(OI)(CI)(F)' 2>&1 | Out-Null
} else {
Say ' note: this dump contains password hashes - check who can read that directory' 'Yellow'
}
$mysql = Find-MysqlClient $mysql = Find-MysqlClient
if (-not $mysql) { Say ' mysql client not found' 'Red'; return } if (-not $mysql) { Say ' mysql client not found' 'Red'; return }
$dump = Join-Path (Split-Path $mysql -Parent) 'mysqldump.exe' $dump = Join-Path (Split-Path $mysql -Parent) 'mysqldump.exe'

811
deploy/windows/installer/shopdb-install.ps1
View File

File diff suppressed because it is too large Load Diff

31
deploy/windows/installer/shopdb-preflight.ps1
View File

@@ -221,8 +221,12 @@ Invoke-Check 'Network' 'Site port' {
Add-Result 'Network' 'Site port' 'INFO' ` Add-Result 'Network' 'Site port' 'INFO' `
"TCP $SitePort is used by the existing $SiteName site - this will be upgraded in place" "TCP $SitePort is used by the existing $SiteName site - this will be upgraded in place"
} else { } else {
Add-Result 'Network' 'Site port' 'FAIL' "TCP $SitePort is in use$owner" ` # WARN, not FAIL. This check runs before the operator has reached the
"Choose a different port with -SitePort, or stop the listener." # Address page, so it is testing the DEFAULT port, not necessarily
# the one they intend to use. Blocking here would refuse an install
# over a conflict the very next page lets them resolve.
Add-Result 'Network' 'Site port' 'WARN' "TCP $SitePort is in use$owner" `
"Pick a different port on the Address page later in this wizard, or stop whatever is holding it."
} }
} }
} }
@@ -266,8 +270,13 @@ Invoke-Check 'IIS' 'HttpPlatformHandler' {
if ($modules -match 'httpPlatformHandler') { if ($modules -match 'httpPlatformHandler') {
Add-Result 'IIS' 'HttpPlatformHandler' 'PASS' 'installed' Add-Result 'IIS' 'HttpPlatformHandler' 'PASS' 'installed'
} else { } else {
Add-Result 'IIS' 'HttpPlatformHandler' 'FAIL' 'not installed' ` # NOT a blocker: the MSI is in the bundle and stage 4 installs it. This
'Install httpPlatformHandler_amd64.msi from the bundle. IIS cannot launch waitress without it.' # was a FAIL, which - once the preflight page started blocking on any
# failure - stopped the wizard dead over something the installer was
# about to do by itself, with no way forward but to go and install it by
# hand. Nothing the installer SUPPLIES may be a blocker.
Add-Result 'IIS' 'HttpPlatformHandler' 'INFO' 'not installed yet' `
'The installer installs it from the bundle. No action needed.'
} }
} }
@@ -350,7 +359,7 @@ Invoke-Check 'MySQL' 'Service' {
$svcs = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^MySQL' -or $_.DisplayName -match 'MySQL' }) $svcs = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^MySQL' -or $_.DisplayName -match 'MySQL' })
if ($svcs.Count -eq 0) { if ($svcs.Count -eq 0) {
Add-Result 'MySQL' 'Service' 'INFO' 'no MySQL service found' ` Add-Result 'MySQL' 'Service' 'INFO' 'no MySQL service found' `
'Bundled MySQL 8.0 is the appropriate choice on this box.' 'Bundled MySQL 8.4 LTS is the appropriate choice on this box.'
return return
} }
foreach ($s in $svcs) { foreach ($s in $svcs) {
@@ -378,7 +387,7 @@ Invoke-Check 'MySQL' 'Backup client' {
$found = '' $found = ''
foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'), foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'),
(Join-Path $AppRoot 'mysqlclient'), (Join-Path $AppRoot 'mysqlclient'),
'C:\Program Files\MySQL', 'C:\mysql56\bin', 'C:\MySQL84', 'C:\Program Files\MySQL', 'C:\mysql56\bin',
'C:\Program Files (x86)\MySQL')) { 'C:\Program Files (x86)\MySQL')) {
if (-not (Test-Path $root)) { continue } if (-not (Test-Path $root)) { continue }
$hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue | $hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue |
@@ -389,7 +398,7 @@ Invoke-Check 'MySQL' 'Backup client' {
Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)" Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)"
} else { } else {
Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' ` Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' `
'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. Add mysqlclient\ to the installer bundle, or install the MySQL client on this server.' 'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. It is not on this server yet - the installer stages its own copy, so this normally resolves itself during installation.'
} }
} }
@@ -430,7 +439,13 @@ Invoke-Check 'MySQL' 'Version and config' {
Add-Result 'MySQL' '5.6 index flags' 'WARN' 'all three present in my.ini' ` Add-Result 'MySQL' '5.6 index flags' 'WARN' 'all three present in my.ini' `
'Present in the file only. They take effect after a MySQL RESTART, which interrupts the classic ASP app. Confirm with SHOW VARIABLES or flask db-utils preflight.' 'Present in the file only. They take effect after a MySQL RESTART, which interrupts the classic ASP app. Confirm with SHOW VARIABLES or flask db-utils preflight.'
} else { } else {
Add-Result 'MySQL' '5.6 index flags' 'FAIL' ("missing: " + ($missing -join ', ')) ` # WARN, not FAIL. This inspects the LOCAL MySQL, which may not be the
# database the operator is about to install against - a bundled 8.4,
# or a remote server. Blocking the wizard here refused an install
# over a server that had nothing to do with it. Stage 3 runs
# 'flask db-utils preflight' against the database actually chosen,
# which is the check that can genuinely block.
Add-Result 'MySQL' '5.6 index flags' 'WARN' ("missing: " + ($missing -join ', ')) `
"Add to [mysqld] in $ini and restart MySQL, or 'flask db upgrade' fails with error 1071. NOTE: restarting interrupts the classic ASP app." "Add to [mysqld] in $ini and restart MySQL, or 'flask db upgrade' fails with error 1071. NOTE: restarting interrupts the classic ASP app."
} }
} }

130
deploy/windows/installer/tests/test-install-state.ps1 Normal file
View File

@@ -0,0 +1,130 @@
# Tests the installer's durable install record.
#
# The record answers one question: "is this a re-run of MY install?". Getting it
# wrong is destructive in one direction - answering "first provisioning" for a
# server holding real data lets prune-schema --force loose on it - and a dead
# end in the other, which is what it was built to fix.
#
# Run directly: pwsh -File test-install-state.ps1
# pytest runs it through tests/test_installer_state.py wherever pwsh exists.
Set-StrictMode -Version 2.0
$ErrorActionPreference = 'Stop'
$Here = Split-Path -Parent $MyInvocation.MyCommand.Path
$Installer = Join-Path (Split-Path -Parent $Here) 'shopdb-install.ps1'
if (-not (Test-Path $Installer)) { throw "installer not found at $Installer" }
$Sandbox = Join-Path ([System.IO.Path]::GetTempPath()) ('shopdb-state-' + [System.Guid]::NewGuid().ToString('N'))
# Load ONLY the state block, so the rest of the installer does not run. Bounded
# by the same markers the installer uses, and it fails loudly if either moves
# rather than silently testing nothing.
$lines = Get-Content $Installer
$start = ($lines | Select-String -Pattern '^\$script:StateFile ' | Select-Object -First 1)
$end = ($lines | Select-String -Pattern '^function Invoke-Native' | Select-Object -First 1)
if (-not $start -or -not $end) { throw 'could not locate the install-record block in shopdb-install.ps1' }
$block = $lines[($start.LineNumber - 1)..($end.LineNumber - 2)] -join "`n"
if ($block -notmatch 'function Test-FirstProvisioning') { throw 'extracted block does not contain the state functions' }
# Stand-ins for the surrounding script.
$script:Created = New-Object System.Collections.ArrayList
function Write-Log { param($m, $l = 'INFO') }
function Protect-File { param([string] $Path, [switch] $Directory) }
function Test-SamePath {
param([string] $A, [string] $B)
if (-not $A) { return -not $B }
return ($A.TrimEnd('\', '/').ToLowerInvariant() -eq $B.TrimEnd('\', '/').ToLowerInvariant())
}
if (-not $env:ProgramData) { $env:ProgramData = [System.IO.Path]::GetTempPath() }
Invoke-Expression $block
$script:Failures = 0
function Check {
param([string] $Name, $Expected, $Actual)
if ($Expected -eq $Actual) {
Write-Host (" PASS " + $Name)
} else {
Write-Host (" FAIL {0}: expected [{1}], got [{2}]" -f $Name, $Expected, $Actual)
$script:Failures++
}
}
function Reset-Case {
param([switch] $Stamped)
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
New-Item -ItemType Directory -Path $Sandbox -Force | Out-Null
$script:AppRoot = $Sandbox
$script:StateFile = Join-Path $Sandbox 'install-state.json'
$script:InstallState = $null
if ($Stamped) { Set-Content (Join-Path $Sandbox '.installed-version') '0.7.0' }
}
Write-Host 'brand new server'
Reset-Case
Initialize-InstallState
Check 'first provisioning' $true (Test-FirstProvisioning)
Check 'record written' $true (Test-Path $script:StateFile)
Write-Host 'retry of a failed first install'
$script:InstallState = $null
Initialize-InstallState
Check 'still first provisioning' $true (Test-FirstProvisioning)
Write-Host 'genuine upgrade after a completed install'
Complete-InstallState
$script:InstallState = $null
Initialize-InstallState
Check 'no longer first provisioning' $false (Test-FirstProvisioning)
Write-Host 'install predating the record - must take the safe side'
Reset-Case -Stamped
Initialize-InstallState
Check 'treated as established' $false (Test-FirstProvisioning)
Write-Host 'record naming a different directory'
Reset-Case
Initialize-InstallState
$other = Get-Content $script:StateFile -Raw | ConvertFrom-Json
$other.approot = 'C:\somewhere-else'
($other | ConvertTo-Json -Depth 6) | Set-Content $script:StateFile
$script:InstallState = $null
Check 'foreign record rejected' $null (Get-InstallState)
Write-Host 'corrupt record'
Reset-Case
Set-Content $script:StateFile '{ this is not json'
$script:InstallState = $null
Check 'corrupt record rejected' $null (Get-InstallState)
Reset-Case
Set-Content $script:StateFile '{ this is not json'
Initialize-InstallState
Check 'recovers and rewrites' $true ($null -ne $script:InstallState)
Write-Host 'created items survive a crash'
Reset-Case
Initialize-InstallState
Track 'site' 'shopdb-flask'
Track 'apppool' 'shopdbflask'
Track 'site' 'shopdb-flask'
$script:InstallState = $null
Initialize-InstallState
Check 'site remembered' 'shopdb-flask' ((Get-CreatedItems 'site') -join ',')
Check 'apppool remembered' 'shopdbflask' ((Get-CreatedItems 'apppool') -join ',')
# A zero-length array returned from a function unrolls to $null, and $null.Count
# is fatal under StrictMode. This is the check that catches losing the comma.
Check 'unknown kind is an empty array' 0 ((Get-CreatedItems 'firewall')).Count
Write-Host 'no record at all'
$script:InstallState = $null
Check 'empty list, not null' 0 ((Get-CreatedItems 'site')).Count
Check 'not first provisioning' $false (Test-FirstProvisioning)
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
if ($script:Failures -gt 0) {
Write-Host ("{0} check(s) failed" -f $script:Failures)
exit 1
}
Write-Host 'all checks passed'
exit 0

1
deploy/windows/installer/verify_bundle_lock.py
View File

@@ -32,6 +32,7 @@ PAYLOADS = [
('httpplatformhandler', True, 'the IIS module that launches waitress'), ('httpplatformhandler', True, 'the IIS module that launches waitress'),
('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'), ('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'),
('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'), ('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'),
('vcredist', False, 'the Visual C++ runtime MySQL requires'),
('mysql', False, 'MySQL, for the bundled-database option'), ('mysql', False, 'MySQL, for the bundled-database option'),
] ]

422
deploy/windows/shopdb-diagnose.py Normal file
View File

@@ -0,0 +1,422 @@
"""Collect everything needed to diagnose a ShopDB-Flask stage 5 failure.
Stage 5 is the smoke test: the installer asks IIS for the site and expects 200.
When it does not get one, the cause is always in one of four places, and this
script reads all four in one pass so the answer arrives in a single round trip:
1. What IIS actually answers, and with which status code and error page.
2. Whether the config sections httpPlatformHandler needs are unlocked.
3. Whether the app-pool identity can read the app and run its venv.
4. Whether the app itself imports and starts.
Run it ON THE SERVER, as Administrator:
C:\\Python314\\python.exe shopdb-diagnose.py
It writes shopdb-diagnose-<timestamp>.txt next to itself and prints the path.
Send that file back.
SECRETS: the report never contains them. Values from .env (database password,
SECRET_KEY, JWT_SECRET_KEY) are read first, then scrubbed out of every section
of the report before it is written, including command output and tracebacks
that might quote them.
Standard library only, so it runs on the bundled runtime or any system Python.
"""
import os
import re
import socket
import subprocess
import sys
import time
from datetime import datetime
APP_ROOT = os.environ.get('SHOPDB_APPROOT', r'C:\shopdb-flask')
ALIAS = 'shopdb'
TIMEOUT = 25
# Filled from .env, then scrubbed from the whole report.
SECRETS = []
WINDIR = os.environ.get('WINDIR', r'C:\Windows')
# Sysnative gives a 32-bit process the real 64-bit System32. Harmless on 64-bit.
APPCMD_CANDIDATES = [
os.path.join(WINDIR, 'Sysnative', 'inetsrv', 'appcmd.exe'),
os.path.join(WINDIR, 'System32', 'inetsrv', 'appcmd.exe'),
]
def find_appcmd():
for path in APPCMD_CANDIDATES:
if os.path.isfile(path):
return path
return None
class Report(object):
def __init__(self):
self.chunks = []
def head(self, title):
self.chunks.append('\n' + '=' * 72 + '\n' + title + '\n' + '=' * 72)
def line(self, text=''):
self.chunks.append(str(text))
def block(self, title, body):
self.chunks.append('\n--- %s ---' % title)
if body is None or str(body).strip() == '':
self.chunks.append('(no output)')
else:
self.chunks.append(str(body).rstrip())
def text(self):
return '\n'.join(self.chunks) + '\n'
def run(cmd, timeout=60):
"""Run a command, return combined output. Never raises."""
try:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, shell=False)
out, _ = proc.communicate(timeout=timeout)
text = out.decode('utf-8', 'replace') if out else ''
return '[exit %s]\n%s' % (proc.returncode, text)
except subprocess.TimeoutExpired:
try:
proc.kill()
except Exception:
pass
return '[TIMED OUT after %ss]' % timeout
except Exception as exc:
return '[could not run: %s]' % exc
def powershell(script, timeout=90):
exe = os.path.join(WINDIR, 'Sysnative', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
if not os.path.isfile(exe):
exe = 'powershell.exe'
return run([exe, '-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', script],
timeout=timeout)
def load_secrets():
"""Read .env so its secret VALUES can be scrubbed from the report."""
env_path = os.path.join(APP_ROOT, '.env')
found = {}
if not os.path.isfile(env_path):
return found, None
try:
with open(env_path, 'r', encoding='utf-8', errors='replace') as handle:
raw = handle.read()
except Exception as exc:
return found, '[could not read .env: %s]' % exc
for line in raw.splitlines():
line = line.strip()
if not line or line.startswith('#') or '=' not in line:
continue
key, value = line.split('=', 1)
key, value = key.strip(), value.strip().strip('"').strip("'")
found[key] = value
if not value:
continue
upper = key.upper()
if 'SECRET' in upper or 'PASSWORD' in upper or 'TOKEN' in upper or 'KEY' in upper:
SECRETS.append(value)
if upper == 'DATABASE_URL':
# mysql+pymysql://user:PASSWORD@host/db -- the password only.
match = re.match(r'^[^:]+://([^:@/]+):([^@]+)@', value)
if match:
SECRETS.append(match.group(2))
return found, raw
def scrub(text):
"""Remove every known secret value from the report."""
for secret in SECRETS:
if secret and len(secret) >= 4:
text = text.replace(secret, '<REDACTED>')
# Catch a password inside any URL that did not come from .env.
text = re.sub(r'(://[^:@/\s]+:)[^@\s]+(@)', r'\1<REDACTED>\2', text)
return text
def http_probe(url):
"""Fetch a URL, returning status, headers and body even for an error page."""
import urllib.error
import urllib.request
started = time.time()
try:
request = urllib.request.Request(url, headers={'User-Agent': 'shopdb-diagnose'})
with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
body = response.read(4000).decode('utf-8', 'replace')
return {'status': response.status, 'reason': response.reason,
'headers': dict(response.headers), 'body': body,
'seconds': time.time() - started}
except urllib.error.HTTPError as exc:
body = ''
try:
body = exc.read(4000).decode('utf-8', 'replace')
except Exception:
pass
return {'status': exc.code, 'reason': exc.reason,
'headers': dict(exc.headers or {}), 'body': body,
'seconds': time.time() - started}
except Exception as exc:
return {'status': None, 'reason': '%s: %s' % (type(exc).__name__, exc),
'headers': {}, 'body': '', 'seconds': time.time() - started}
def summarise_iis_error(body):
"""Pull the meaningful bits out of an IIS error page."""
if not body:
return None
import html
# style and script blocks first: their contents survive plain tag stripping
# and drag CSS into the summary.
flat = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', body)
flat = re.sub(r'(?s)<!--.*?-->', ' ', flat)
flat = re.sub(r'(?s)<[^>]*>', ' ', flat)
# Any unterminated tag left by the 4000-byte body truncation.
flat = re.sub(r'(?s)<[^>]*$', ' ', flat)
flat = html.unescape(flat)
flat = re.sub(r'\s+', ' ', flat).strip()
hints = []
for pattern in (r'\b\d{3}\.\d+\b', r'0x[0-9a-fA-F]{8}',
r'Error Code[^.]{0,80}', r'Config (?:Error|File)[^.]{0,120}',
r'Requested URL[^.]{0,120}', r'Physical Path[^.]{0,120}'):
for match in re.findall(pattern, flat):
cleaned = re.sub(r'\s+', ' ', match).strip(' :-')
if cleaned and cleaned not in hints:
hints.append(cleaned)
return {'flat': flat[:1200], 'hints': hints}
def main():
report = Report()
stamp = datetime.now().strftime('%Y%m%d-%H%M%S')
env_values, env_raw = load_secrets()
report.line('ShopDB-Flask stage 5 diagnostic')
report.line('generated %s' % datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
report.line('host %s' % socket.gethostname())
report.line('app root %s' % APP_ROOT)
report.line('python %s' % sys.version.replace('\n', ' '))
report.line('process is %d-bit' % (64 if sys.maxsize > 2 ** 32 else 32))
# ---------------------------------------------------------------- 1. HTTP
report.head('1. WHAT IIS ANSWERS')
report.line('This is the single most important section. The status code names')
report.line('the fault: 500.19 = config locked, 502.3/503 = the app did not')
report.line('start, 404 = application or handler mapping missing.')
hostname = socket.gethostname()
targets = [
'http://localhost/%s/' % ALIAS,
'http://127.0.0.1/%s/' % ALIAS,
'http://[::1]/%s/' % ALIAS,
'http://%s/%s/' % (hostname, ALIAS),
'http://localhost/',
]
for url in targets:
result = http_probe(url)
report.line('\n%s' % url)
report.line(' status : %s %s' % (result['status'], result['reason']))
report.line(' time : %.1fs' % result['seconds'])
server = result['headers'].get('Server')
if server:
report.line(' server : %s' % server)
summary = summarise_iis_error(result['body'])
if summary and summary['hints']:
report.line(' hints : %s' % ' | '.join(summary['hints'][:8]))
if summary and summary['flat']:
report.line(' body : %s' % summary['flat'][:600])
# localhost resolving to ::1 first has bitten this install before.
report.block('name resolution for localhost', run(
['nslookup', 'localhost'], timeout=20))
# ------------------------------------------------------------- 2. IIS state
report.head('2. IIS STATE')
appcmd = find_appcmd()
if not appcmd:
report.line('appcmd.exe NOT FOUND - is the IIS role installed?')
else:
report.line('appcmd: %s' % appcmd)
report.block('sites', run([appcmd, 'list', 'sites']))
report.block('applications', run([appcmd, 'list', 'apps']))
report.block('app pools', run([appcmd, 'list', 'apppools']))
report.block('worker processes (empty means nothing is running)',
run([appcmd, 'list', 'wp']))
report.block('modules: httpPlatformHandler present?',
run([appcmd, 'list', 'modules']))
# overrideMode tells us whether the unlock actually took effect.
#
# allowedServerVariables is in this list because it caused a 500.52 that
# the first two sections could not explain: it is Deny by default, so an
# <allowedServerVariables> block in the app's web.config is rejected
# before httpPlatformHandler runs. Checking only the sections we unlock
# would have missed the one we do not.
for section in ('system.webServer/handlers',
'system.webServer/httpPlatform',
'system.webServer/rewrite/allowedServerVariables',
'system.webServer/rewrite/rules'):
report.block('lock state of %s' % section,
run([appcmd, 'list', 'config', '/section:%s' % section,
'/text:*']))
report.block('W3SVC / WAS services', powershell(
"Get-Service W3SVC,WAS | Format-Table Name,Status,StartType -AutoSize | Out-String"))
report.block('listeners on port 80', powershell(
"Get-NetTCPConnection -LocalPort 80 -State Listen -EA SilentlyContinue | "
"Format-Table LocalAddress,LocalPort,OwningProcess -AutoSize | Out-String"))
report.block('app pool detail', powershell(
"Import-Module WebAdministration -EA SilentlyContinue; "
"Get-Item IIS:\\AppPools\\shopdbflask -EA SilentlyContinue | "
"Select-Object name,state,managedRuntimeVersion,enable32BitAppOnWin64,"
"@{n='identity';e={$_.processModel.identityType}} | Format-List | Out-String"))
# --------------------------------------------------------- 3. app + config
report.head('3. APPLICATION AND CONFIG')
web_config = os.path.join(APP_ROOT, 'web.config')
if os.path.isfile(web_config):
try:
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
report.block('web.config', handle.read())
except Exception as exc:
report.block('web.config', '[could not read: %s]' % exc)
else:
report.block('web.config', 'MISSING at %s' % web_config)
if env_raw is None:
report.block('.env', 'MISSING at %s' % os.path.join(APP_ROOT, '.env'))
else:
# Keys and non-secret values only. Secret values are scrubbed anyway.
lines = []
for key in sorted(env_values):
upper = key.upper()
secretish = ('SECRET' in upper or 'PASSWORD' in upper
or 'TOKEN' in upper or 'KEY' in upper
or upper == 'DATABASE_URL')
if secretish:
lines.append('%s = <set, %d chars>' % (key, len(env_values[key])))
else:
lines.append('%s = %s' % (key, env_values[key]))
report.block('.env (secret values withheld)', '\n'.join(lines))
venv_python = os.path.join(APP_ROOT, 'venv', 'Scripts', 'python.exe')
report.line('\nvenv python exists: %s' % os.path.isfile(venv_python))
if os.path.isfile(venv_python):
# The exact failure stage 3 used to hit. Proves the app imports.
report.block('venv: import shopdb', run(
[venv_python, '-c',
'import shopdb; print("import OK"); '
'app = shopdb.create_app(); print("create_app OK")'], timeout=120))
report.block('venv: waitress present', run(
[venv_python, '-c', 'import waitress; print(waitress.__version__)'],
timeout=60))
# What httpPlatformHandler is told to launch, and whether it exists.
if os.path.isfile(web_config):
try:
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
raw = handle.read()
match = re.search(r'processPath\s*=\s*"([^"]+)"', raw)
args = re.search(r'arguments\s*=\s*"([^"]*)"', raw)
if match:
path = os.path.expandvars(match.group(1))
report.line('\nhttpPlatform processPath : %s' % match.group(1))
report.line(' resolved : %s' % path)
report.line(' exists : %s' % os.path.isfile(path))
if args:
report.line('httpPlatform arguments : %s' % args.group(1))
except Exception as exc:
report.line('[could not parse web.config: %s]' % exc)
# ------------------------------------------------------------- 4. app logs
report.head('4. APPLICATION LOGS')
log_dir = os.path.join(APP_ROOT, 'logs')
if not os.path.isdir(log_dir):
report.line('MISSING: %s' % log_dir)
else:
entries = []
for name in sorted(os.listdir(log_dir)):
full = os.path.join(log_dir, name)
try:
entries.append((os.path.getmtime(full), full, name,
os.path.getsize(full)))
except OSError:
pass
if not entries:
report.line('%s is EMPTY.' % log_dir)
report.line('No stdout log at all means httpPlatformHandler never')
report.line('launched python - look at the pool identity and ACLs.')
for _, full, name, size in sorted(entries, reverse=True)[:5]:
if size == 0:
report.block('%s (0 bytes)' % name,
'EMPTY - python was launched but wrote nothing.')
continue
try:
with open(full, 'r', encoding='utf-8', errors='replace') as handle:
tail = handle.readlines()[-60:]
report.block('%s (%d bytes, last 60 lines)' % (name, size),
''.join(tail))
except Exception as exc:
report.block(name, '[could not read: %s]' % exc)
# ---------------------------------------------------------------- 5. ACLs
report.head('5. PERMISSIONS')
report.line('The pool runs as "IIS AppPool\\shopdbflask". It needs RX on the')
report.line('tree, Modify on logs and instance, and Read on .env.')
for target in (APP_ROOT, log_dir, os.path.join(APP_ROOT, '.env'),
os.path.join(APP_ROOT, 'venv', 'Scripts')):
if os.path.exists(target):
report.block('icacls %s' % target, run(['icacls', target], timeout=40))
# ------------------------------------------------------------ 6. event log
report.head('6. EVENT LOG')
report.block('recent application errors', powershell(
"Get-WinEvent -FilterHashtable @{LogName='Application';"
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
"Where-Object { $_.ProviderName -match 'HttpPlatform|IIS|W3SVC|WAS|\\.NET' "
"-or $_.LevelDisplayName -eq 'Error' } | Select-Object -First 25 "
"TimeCreated,ProviderName,LevelDisplayName,Message | Format-List | Out-String",
timeout=180))
report.block('system log: WAS / W3SVC', powershell(
"Get-WinEvent -FilterHashtable @{LogName='System';"
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
"Where-Object { $_.ProviderName -match 'WAS|W3SVC|HTTP' } | "
"Select-Object -First 20 TimeCreated,ProviderName,LevelDisplayName,Message | "
"Format-List | Out-String", timeout=180))
# ------------------------------------------------------------------ write
body = scrub(report.text())
out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'shopdb-diagnose-%s.txt' % stamp)
try:
with open(out_path, 'w', encoding='utf-8') as handle:
handle.write(body)
except Exception:
out_path = os.path.join(os.environ.get('TEMP', r'C:\Windows\Temp'),
'shopdb-diagnose-%s.txt' % stamp)
with open(out_path, 'w', encoding='utf-8') as handle:
handle.write(body)
print('')
print('Report written to:')
print(' %s' % out_path)
print('')
print('%d secret value(s) were scrubbed from it.' % len(SECRETS))
print('Send that file back.')
return 0
if __name__ == '__main__':
sys.exit(main())

23
deploy/windows/web.config
View File

@@ -67,15 +67,28 @@
It ships DISABLED because it needs the URL Rewrite module; enabled without It ships DISABLED because it needs the URL Rewrite module; enabled without
it, IIS returns HTTP 500.19 ("configuration section not well-formed"). it, IIS returns HTTP 500.19 ("configuration section not well-formed").
There is deliberately NO <allowedServerVariables> block below. Setting a
server variable requires that variable to be allowed, but the section
system.webServer/rewrite/allowedServerVariables ships with
overrideModeDefault="Deny", so declaring it in an application's own
web.config is refused outright: IIS answered 500.52 with error 0x80070021,
"this configuration section cannot be used at this path", BEFORE it ever
reached httpPlatformHandler - so python was never launched and the stdout
log stayed empty, which looks like an application fault and is not one.
The installer instead allows the single variable at server level, which
grants exactly HTTP_X_FORWARDED_FOR rather than unlocking the section and
letting every site on the machine declare arbitrary server variables.
The installer handles both: -ClientIpSource direct installs URL Rewrite The installer handles both: -ClientIpSource direct installs URL Rewrite
from the bundle and enables this; -ClientIpSource proxy leaves it alone. from the bundle, allows the variable, and enables this; -ClientIpSource
By hand: install URL Rewrite, then delete the two marker lines below. proxy leaves it alone. By hand: install URL Rewrite, run
appcmd set config /section:system.webServer/rewrite/allowedServerVariables ^
/+"[name='HTTP_X_FORWARDED_FOR']" /commit:apphost
then delete the two marker lines below.
--> -->
<!-- SHOPDB-CLIENTIP-BEGIN <!-- SHOPDB-CLIENTIP-BEGIN
<rewrite> <rewrite>
<allowedServerVariables>
<add name="HTTP_X_FORWARDED_FOR" />
</allowedServerVariables>
<rules> <rules>
<rule name="Set X-Forwarded-For" stopProcessing="false"> <rule name="Set X-Forwarded-For" stopProcessing="false">
<match url=".*" /> <match url=".*" />

2
frontend/package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "shopdb-frontend", "name": "shopdb-frontend",
"version": "0.7.0", "version": "0.8.0",
"private": true, "private": true,
"type": "module", "type": "module",
"scripts": { "scripts": {

28
pytest.ini Normal file
View File

@@ -0,0 +1,28 @@
[pytest]
# Tests live in tests/. Saying so is not cosmetic.
#
# deploy/windows/installer/bundle/ is BUILD OUTPUT: build-installer.sh stages a
# complete copy of the application there, plugins included. Without this,
# collection walked into it and imported every plugin a second time under a
# second path, so SQLAlchemy raised "Table 'printeditems' is already defined for
# this MetaData instance" and the whole suite failed to collect - on a tree with
# nothing wrong in it, purely because somebody had built an installer first.
#
# It only started biting when the bundle grew from four plugins to thirteen,
# which is the sort of delay that makes this look like a code fault rather than
# a stale artefact.
testpaths = tests
# Belt and braces: even an explicit path or an IDE run must not descend into
# build output or a staged copy of the tree.
norecursedirs =
.git
venv
.venv
node_modules
bundle
dist
build
frontend/dist
frontend/dist-subpath
*.egg-info

6
requirements.in
View File

@@ -18,6 +18,12 @@ click>=8.1
python-dotenv>=1.0 python-dotenv>=1.0
tabulate>=0.9 tabulate>=0.9
# Plugin contract version checking (shopdb/plugins/loader.py).
# Declared explicitly because it reaches dev and CI only as a pytest
# dependency: every test passes while a production venv, which has no pytest,
# cannot import shopdb at all.
packaging>=23.0
# HTTP/API clients # HTTP/API clients
requests>=2.31 requests>=2.31

77
requirements.txt
View File

@@ -15,31 +15,29 @@
# #
# Regenerate with the command above. Do NOT hand-edit: waitress and tzdata were # Regenerate with the command above. Do NOT hand-edit: waitress and tzdata were
# hand-added here once and vanished on the next compile. # hand-added here once and vanished on the next compile.
#
# Only DIRECT requirements belong in requirements.in. A runtime import that is
# satisfied in dev only because some test dependency happens to pull it in will
# pass every test and then fail on a production venv: `packaging` reached this
# project through pytest, and the Windows install could not import shopdb at
# all until it was declared. tests/test_runtime_dependencies.py guards that.
alembic==1.18.4 \ alembic==1.18.4 \
--hash=sha256:a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a \ --hash=sha256:a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a \
--hash=sha256:cb6e1fd84b6174ab8dbb2329f86d631ba9559dd78df550b57804d607672cedbc --hash=sha256:cb6e1fd84b6174ab8dbb2329f86d631ba9559dd78df550b57804d607672cedbc
# via # via flask-migrate
# -r requirements.in
# flask-migrate
blinker==1.9.0 \ blinker==1.9.0 \
--hash=sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf \ --hash=sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf \
--hash=sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc --hash=sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc
# via # via flask
# -r requirements.in
# flask
cachelib==0.13.0 \ cachelib==0.13.0 \
--hash=sha256:209d8996e3c57595bee274ff97116d1d73c4980b2fd9a34c7846cd07fd2e1a48 \ --hash=sha256:209d8996e3c57595bee274ff97116d1d73c4980b2fd9a34c7846cd07fd2e1a48 \
--hash=sha256:8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516 --hash=sha256:8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516
# via # via flask-caching
# -r requirements.in
# flask-caching
certifi==2026.4.22 \ certifi==2026.4.22 \
--hash=sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a \ --hash=sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a \
--hash=sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580 --hash=sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580
# via # via requests
# -r requirements.in cffi==2.1.0 ; platform_python_implementation != 'PyPy' \
# requests
cffi==2.1.0 \
--hash=sha256:02cb7ff33ded4f1532476731f89ede53e2e488a8e6205515a82144246ffa7dcc \ --hash=sha256:02cb7ff33ded4f1532476731f89ede53e2e488a8e6205515a82144246ffa7dcc \
--hash=sha256:03e9810d18c646077e501f661b682fbf5dee4676048527ca3cffe66faa9960dd \ --hash=sha256:03e9810d18c646077e501f661b682fbf5dee4676048527ca3cffe66faa9960dd \
--hash=sha256:0520e1f4c35f44e209cbbb421b67eec42e6a157f59444dfb6058874ff3610e5d \ --hash=sha256:0520e1f4c35f44e209cbbb421b67eec42e6a157f59444dfb6058874ff3610e5d \
@@ -140,9 +138,7 @@ cffi==2.1.0 \
--hash=sha256:f9912624a0c0b834b7520d7769b3644453aabc0a7e1c839da7359f050750e9bc \ --hash=sha256:f9912624a0c0b834b7520d7769b3644453aabc0a7e1c839da7359f050750e9bc \
--hash=sha256:fb62edb5bb52cca65fab91a63afa7561607120d26090a7e8fda6fb9f064726da \ --hash=sha256:fb62edb5bb52cca65fab91a63afa7561607120d26090a7e8fda6fb9f064726da \
--hash=sha256:ff067a8d8d880e7809e4ac88eb009bb848870115317b306666502ccad30b147f --hash=sha256:ff067a8d8d880e7809e4ac88eb009bb848870115317b306666502ccad30b147f
# via # via cryptography
# -r requirements.in
# cryptography
charset-normalizer==3.4.7 \ charset-normalizer==3.4.7 \
--hash=sha256:007d05ec7321d12a40227aae9e2bc6dca73f3cb21058999a1df9e193555a9dcc \ --hash=sha256:007d05ec7321d12a40227aae9e2bc6dca73f3cb21058999a1df9e193555a9dcc \
--hash=sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c \ --hash=sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c \
@@ -273,9 +269,7 @@ charset-normalizer==3.4.7 \
--hash=sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6 \ --hash=sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6 \
--hash=sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79 \ --hash=sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79 \
--hash=sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464 --hash=sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464
# via # via requests
# -r requirements.in
# requests
click==8.3.3 \ click==8.3.3 \
--hash=sha256:398329ad4837b2ff7cbe1dd166a4c0f8900c3ca3a218de04466f38f6497f18a2 \ --hash=sha256:398329ad4837b2ff7cbe1dd166a4c0f8900c3ca3a218de04466f38f6497f18a2 \
--hash=sha256:a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613 --hash=sha256:a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613
@@ -337,9 +331,7 @@ cryptography==50.0.0 \
dnspython==2.8.0 \ dnspython==2.8.0 \
--hash=sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af \ --hash=sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af \
--hash=sha256:181d3c6996452cb1189c4046c61599b84a5a86e099562ffde77d26984ff26d0f --hash=sha256:181d3c6996452cb1189c4046c61599b84a5a86e099562ffde77d26984ff26d0f
# via # via email-validator
# -r requirements.in
# email-validator
email-validator==2.3.0 \ email-validator==2.3.0 \
--hash=sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4 \ --hash=sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4 \
--hash=sha256:9fc05c37f2f6cf439ff414f8fc46d917929974a82244c20eb10231ba60c54426 --hash=sha256:9fc05c37f2f6cf439ff414f8fc46d917929974a82244c20eb10231ba60c54426
@@ -381,7 +373,7 @@ flask-sqlalchemy==3.1.1 \
# via # via
# -r requirements.in # -r requirements.in
# flask-migrate # flask-migrate
greenlet==3.5.0 \ greenlet==3.5.0 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' \
--hash=sha256:0ecec963079cd58cbd14723582384f11f166fd58883c15dcbfb342e0bc9b5846 \ --hash=sha256:0ecec963079cd58cbd14723582384f11f166fd58883c15dcbfb342e0bc9b5846 \
--hash=sha256:0ed006e4b86c59de7467eb2601cd1b77b5a7d657d1ee55e30fe30d76451edba4 \ --hash=sha256:0ed006e4b86c59de7467eb2601cd1b77b5a7d657d1ee55e30fe30d76451edba4 \
--hash=sha256:0ff251e9a0279522e62f6176412869395a64ddf2b5c5f782ff609a8216a4e662 \ --hash=sha256:0ff251e9a0279522e62f6176412869395a64ddf2b5c5f782ff609a8216a4e662 \
@@ -441,34 +433,25 @@ greenlet==3.5.0 \
--hash=sha256:f35807464c4c58c55f0d31dfa83c541a5615d825c2fe3d2b95360cf7c4e3c0a8 \ --hash=sha256:f35807464c4c58c55f0d31dfa83c541a5615d825c2fe3d2b95360cf7c4e3c0a8 \
--hash=sha256:f8c30c2225f40dd76c50790f0eb3b5c7c18431efb299e2782083e1981feed243 \ --hash=sha256:f8c30c2225f40dd76c50790f0eb3b5c7c18431efb299e2782083e1981feed243 \
--hash=sha256:fa94cb2288681e3a11645958f1871d48ee9211bd2f66628fdace505927d6e564 --hash=sha256:fa94cb2288681e3a11645958f1871d48ee9211bd2f66628fdace505927d6e564
# via # via sqlalchemy
# -r requirements.in
# sqlalchemy
idna==3.13 \ idna==3.13 \
--hash=sha256:585ea8fe5d69b9181ec1afba340451fba6ba764af97026f92a91d4eef164a242 \ --hash=sha256:585ea8fe5d69b9181ec1afba340451fba6ba764af97026f92a91d4eef164a242 \
--hash=sha256:892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3 --hash=sha256:892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3
# via # via
# -r requirements.in
# email-validator # email-validator
# requests # requests
itsdangerous==2.2.0 \ itsdangerous==2.2.0 \
--hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef \ --hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef \
--hash=sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173 --hash=sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173
# via # via flask
# -r requirements.in
# flask
jinja2==3.1.6 \ jinja2==3.1.6 \
--hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \ --hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \
--hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67 --hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67
# via # via flask
# -r requirements.in
# flask
mako==1.3.12 \ mako==1.3.12 \
--hash=sha256:8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9 \ --hash=sha256:8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9 \
--hash=sha256:9f778e93289bd410bb35daadeb4fc66d95a746f0b75777b942088b7fd7af550a --hash=sha256:9f778e93289bd410bb35daadeb4fc66d95a746f0b75777b942088b7fd7af550a
# via # via alembic
# -r requirements.in
# alembic
markupsafe==3.0.3 \ markupsafe==3.0.3 \
--hash=sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f \ --hash=sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f \
--hash=sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a \ --hash=sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a \
@@ -560,7 +543,6 @@ markupsafe==3.0.3 \
--hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \ --hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \
--hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50 --hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50
# via # via
# -r requirements.in
# flask # flask
# jinja2 # jinja2
# mako # mako
@@ -569,7 +551,6 @@ marshmallow==4.3.0 \
--hash=sha256:46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46 \ --hash=sha256:46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46 \
--hash=sha256:fb43c53b3fe240b8f6af37223d6ef1636f927ad9bea8ab323afad95dff090880 --hash=sha256:fb43c53b3fe240b8f6af37223d6ef1636f927ad9bea8ab323afad95dff090880
# via # via
# -r requirements.in
# flask-marshmallow # flask-marshmallow
# marshmallow-sqlalchemy # marshmallow-sqlalchemy
marshmallow-sqlalchemy==1.5.0 \ marshmallow-sqlalchemy==1.5.0 \
@@ -604,18 +585,18 @@ mysql-connector-python==9.7.0 \
--hash=sha256:ea4e05ab864ab8b0b5066d885d89ca51e5ba1320dba520c20a6f9388a8eca022 \ --hash=sha256:ea4e05ab864ab8b0b5066d885d89ca51e5ba1320dba520c20a6f9388a8eca022 \
--hash=sha256:ee90c5f44f706f012be17f03f6ad158ff96e7f2dcc077896fe4537d3d28b3cf4 --hash=sha256:ee90c5f44f706f012be17f03f6ad158ff96e7f2dcc077896fe4537d3d28b3cf4
# via -r requirements.in # via -r requirements.in
pycparser==3.0 \ packaging==26.3 \
--hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
# via -r requirements.in
pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy' \
--hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \ --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \
--hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992 --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992
# via # via cffi
# -r requirements.in
# cffi
pyjwt==2.12.1 \ pyjwt==2.12.1 \
--hash=sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c \ --hash=sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c \
--hash=sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b --hash=sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b
# via # via flask-jwt-extended
# -r requirements.in
# flask-jwt-extended
pymysql==1.1.3 \ pymysql==1.1.3 \
--hash=sha256:8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289 \ --hash=sha256:8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289 \
--hash=sha256:e70ebf2047a4edf6138cf79c68ad418ef620af65900aa585c5e8bfc95044d43a --hash=sha256:e70ebf2047a4edf6138cf79c68ad418ef620af65900aa585c5e8bfc95044d43a
@@ -693,7 +674,6 @@ sqlalchemy==2.0.49 \
--hash=sha256:ec44cfa7ef1a728e88ad41674de50f6db8cfdb3e2af84af86e0041aaf02d43d0 \ --hash=sha256:ec44cfa7ef1a728e88ad41674de50f6db8cfdb3e2af84af86e0041aaf02d43d0 \
--hash=sha256:fb37f15714ec2652d574f021d479e78cd4eb9d04396dca36568fdfffb3487982 --hash=sha256:fb37f15714ec2652d574f021d479e78cd4eb9d04396dca36568fdfffb3487982
# via # via
# -r requirements.in
# alembic # alembic
# flask-sqlalchemy # flask-sqlalchemy
# marshmallow-sqlalchemy # marshmallow-sqlalchemy
@@ -705,7 +685,6 @@ typing-extensions==4.15.0 \
--hash=sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466 \ --hash=sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466 \
--hash=sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548 --hash=sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548
# via # via
# -r requirements.in
# alembic # alembic
# sqlalchemy # sqlalchemy
tzdata==2026.3 \ tzdata==2026.3 \
@@ -715,9 +694,7 @@ tzdata==2026.3 \
urllib3==2.7.0 \ urllib3==2.7.0 \
--hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \ --hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \
--hash=sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897 --hash=sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897
# via # via requests
# -r requirements.in
# requests
waitress==3.0.2 \ waitress==3.0.2 \
--hash=sha256:682aaaf2af0c44ada4abfb70ded36393f0e307f4ab9456a215ce0020baefc31f \ --hash=sha256:682aaaf2af0c44ada4abfb70ded36393f0e307f4ab9456a215ce0020baefc31f \
--hash=sha256:c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e --hash=sha256:c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e

20
scripts/gen_openapi.py
View File

@@ -20,6 +20,21 @@ OUT = os.path.join(REPO, 'docs', 'openapi.json')
VERBS = ('get', 'post', 'put', 'patch', 'delete', 'head', 'options') VERBS = ('get', 'post', 'put', 'patch', 'delete', 'head', 'options')
def _product_version():
"""Read __version__ out of shopdb/__init__.py without importing the app.
Importing shopdb here would pull in Flask, the plugin loader and a database
configuration, none of which this generator needs.
"""
here = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
init = os.path.join(here, 'shopdb', '__init__.py')
with open(init, 'r', encoding='utf-8') as handle:
match = re.search(r"^__version__ = '([^']+)'", handle.read(), re.M)
if not match:
raise SystemExit('could not read __version__ from ' + init)
return match.group(1)
def _norm_path(p): def _norm_path(p):
# Flask <int:id> / <id> -> OpenAPI {id} # Flask <int:id> / <id> -> OpenAPI {id}
return re.sub(r'<(?:[^:>]+:)?([^>]+)>', r'{\1}', p or '') return re.sub(r'<(?:[^:>]+:)?([^>]+)>', r'{\1}', p or '')
@@ -68,7 +83,10 @@ def build(surfaces):
'openapi': '3.1.0', 'openapi': '3.1.0',
'info': { 'info': {
'title': 'ShopDB Flask API', 'title': 'ShopDB Flask API',
'version': '0.7.0', # Read from the code, not restated. A hardcoded copy here had
# already drifted a release behind, and the same mistake in the
# installer script shipped an exe stamped with the wrong version.
'version': _product_version(),
'description': ( 'description': (
'Asset-management API (core + plugins). Responses use a ' 'Asset-management API (core + plugins). Responses use a '
'`success_response` envelope: `{status, data, meta}`. Auth: Bearer ' '`success_response` envelope: `{status, data, meta}`. Auth: Bearer '

5
shopdb/__init__.py
View File

@@ -42,7 +42,7 @@ __contract_version__ = '0.15.0'
# plugin-contract version above are distinct series with independent # plugin-contract version above are distinct series with independent
# bump rules. Not part of the shopdb.api contract surface, so it is # bump rules. Not part of the shopdb.api contract surface, so it is
# not re-exported there. # not re-exported there.
__version__ = '0.7.0' __version__ = '0.8.0'
def create_app(config_name: str = None) -> Flask: def create_app(config_name: str = None) -> Flask:
@@ -172,12 +172,13 @@ def register_blueprints(app: Flask):
def register_cli_commands(app: Flask): def register_cli_commands(app: Flask):
"""Register Flask CLI commands.""" """Register Flask CLI commands."""
from .plugins.cli import plugin_cli from .plugins.cli import plugin_cli
from .cli import db_cli, seed_cli, relationships_cli from .cli import db_cli, seed_cli, relationships_cli, csv_cli
app.cli.add_command(plugin_cli) app.cli.add_command(plugin_cli)
app.cli.add_command(db_cli) app.cli.add_command(db_cli)
app.cli.add_command(seed_cli) app.cli.add_command(seed_cli)
app.cli.add_command(relationships_cli) app.cli.add_command(relationships_cli)
app.cli.add_command(csv_cli)
def register_error_handlers(app: Flask): def register_error_handlers(app: Flask):

157
shopdb/cli/__init__.py
View File

@@ -825,3 +825,160 @@ def seed_demo_clear(yes):
click.echo(click.style( click.echo(click.style(
f"Removed {len(demo_ids)} demo assets and " f"Removed {len(demo_ids)} demo assets and "
f"{parts_deleted} sample parts.", fg='green')) f"{parts_deleted} sample parts.", fg='green'))
@click.group('csv')
def csv_cli():
"""Load a site's starting data from CSV files."""
pass
@csv_cli.command('templates')
@click.option('--out', 'outdir', default='csv-templates',
help='Directory to write the templates into.')
@with_appcontext
def csv_templates(outdir):
"""Write a CSV template per importable table, generated from the schema.
Generated rather than kept by hand: a maintained template set drifts on the
next migration and does so silently, since the file still looks correct.
"""
import os
from shopdb.core.services.csvimport import IMPORTABLE, generate_template
if not os.path.isdir(outdir):
os.makedirs(outdir)
for tablename in IMPORTABLE:
path = os.path.join(outdir, tablename + '.csv')
with open(path, 'w', encoding='utf-8') as handle:
handle.write(generate_template(tablename))
click.echo(' %s' % path)
readme = os.path.join(outdir, 'README.txt')
with open(readme, 'w', encoding='utf-8') as handle:
handle.write(
'ShopDB-Flask import templates\n'
'=============================\n\n'
'Generated from the live database schema. Every column here exists;\n'
'every required column is marked.\n\n'
'Fill in the ones you need - you do not need all of them.\n\n'
'Foreign keys take a NAME or a numeric id. Write the name:\n'
' locationid -> Building 1 Bay 3\n'
' vendorid -> Haas Automation\n'
'The importer resolves it, and tells you which row and column to fix\n'
'if the name is not found.\n\n'
'Import the whole folder at once and order is handled for you:\n\n'
' flask csv import --dir . (checks only, changes nothing)\n'
' flask csv import --dir . --commit (applies)\n\n'
'Nothing is written unless every row passes, so a mistake on line 400\n'
'does not leave 399 rows half-imported.\n\n'
'User accounts are deliberately not importable here: passwords do not\n'
'belong in a spreadsheet.\n')
click.echo(' %s' % readme)
click.echo('')
click.echo(click.style('%d templates written to %s' % (len(IMPORTABLE), outdir),
fg='green', bold=True))
@csv_cli.command('import')
@click.option('--file', 'path', default=None, help='One CSV file.')
@click.option('--dir', 'directory', default=None,
help='A folder of CSVs, imported in dependency order.')
@click.option('--table', 'tablename', default=None,
help='Target table. Defaults to the file name.')
@click.option('--commit', is_flag=True, default=False,
help='Apply the changes. Without this, nothing is written.')
@with_appcontext
def csv_import(path, directory, tablename, commit):
"""Validate CSVs and, with --commit, load them.
Dry run by default. The report is the same either way, so what you review is
what you get.
"""
import os
from shopdb.extensions import db
from sqlalchemy.exc import SQLAlchemyError
from shopdb.core.services.csvimport import (
ImportError_, Resolver, dependency_order, import_csv, table_from_filename)
if not path and not directory:
raise click.UsageError('give --file or --dir')
jobs = []
if path:
jobs.append((tablename or table_from_filename(os.path.basename(path)), path))
else:
found = {}
for name in os.listdir(directory):
if not name.lower().endswith('.csv'):
continue
found[table_from_filename(name)] = os.path.join(directory, name)
ordered = dependency_order(list(found))
unknown = sorted(set(found) - set(ordered))
for name in ordered:
jobs.append((name, found[name]))
if unknown:
click.echo(click.style(
'skipping (not importable): %s' % ', '.join(unknown), fg='yellow'))
if not jobs:
click.echo('nothing to do - no CSV files found')
return
click.echo(click.style(
'Checking %d file(s)%s' % (len(jobs), '' if commit else ' - DRY RUN, nothing will be written'),
bold=True))
click.echo('')
resolver = Resolver()
results = []
failed = False
for name, filepath in jobs:
with open(filepath, 'r', encoding='utf-8-sig') as handle:
text = handle.read()
try:
# Every file is applied inside ONE transaction, so a failure part way
# through a folder rolls the whole run back rather than leaving the
# site half-populated.
result = import_csv(name, text, resolver=resolver, commit=commit)
except ImportError_ as exc:
click.echo(click.style(' %-18s %s' % (name, exc), fg='red'))
failed = True
continue
except SQLAlchemyError as exc:
# Anything the database itself refuses. The operator gets the cause
# in one line rather than a traceback they cannot act on.
db.session.rollback()
click.echo(click.style(' %-18s database error: %s'
% (name, str(exc).split(chr(10))[0]), fg='red'))
failed = True
continue
results.append(result)
colour = 'green' if result.ok else 'red'
click.echo(click.style(' ' + result.summary(), fg=colour))
for problem in result.problems[:20]:
click.echo(click.style(' %s' % problem, fg='red'))
if len(result.problems) > 20:
click.echo(click.style(' ... and %d more' % (len(result.problems) - 20), fg='red'))
if not result.ok:
failed = True
click.echo('')
if failed:
db.session.rollback()
click.echo(click.style('Nothing was imported. Fix the problems above and run again.',
fg='red', bold=True))
raise SystemExit(1)
total_new = sum(r.created for r in results)
total_upd = sum(r.updated for r in results)
if commit:
db.session.commit()
click.echo(click.style('Imported: %d new, %d updated.' % (total_new, total_upd),
fg='green', bold=True))
else:
db.session.rollback()
click.echo(click.style(
'Looks good: %d would be created, %d updated.' % (total_new, total_upd),
fg='green', bold=True))
click.echo('Run again with --commit to apply.')

500
shopdb/core/services/csvimport.py Normal file
View File

@@ -0,0 +1,500 @@
"""Bulk-load a site's starting data from CSV files.
WHY THIS EXISTS. Adopting a site means getting its existing asset register into
ShopDB. The HTTP import surface (the IMPORT-API wiki page) is the right tool when
there is a source system to read from and someone able to script against it. A
sister site with a spreadsheet and no developer needs something else, and that
is the common case.
THE DESIGN PROBLEM, which a hand-written template set gets wrong. A CSV row for
an asset has to say where the asset is. The database stores `locationid`, an
integer. Nobody types integers into a spreadsheet correctly, and requiring it
means the operator must first import locations, read back the generated ids, and
paste them into the asset sheet. That is not a workflow anyone completes.
So every foreign key here accepts EITHER a numeric id OR the referenced row's
name, and names are resolved at import time:
assetnumber,assettypeid,locationid,businessunitid
CMM-01,Measuring Tool,Building 1 Bay 3,Quality
The column keeps its database name - the convention in CONTRIBUTING.md - while
the VALUE is whatever the operator actually knows. A name that does not resolve
is a row error naming the column, the value, and where to fix it, not a foreign
key constraint violation from three layers down.
IDEMPOTENCE. Each table declares a natural key (`assetnumber`, `locationname`,
...). Re-importing a file updates the matched rows rather than duplicating them,
so a site can correct its spreadsheet and run it again - which they will.
SAFETY. Validation is a separate pass over the whole file BEFORE anything is
written, and the default is a dry run. An import either applies completely or
not at all.
"""
from __future__ import annotations
import csv
import io
import re
from collections import OrderedDict
from sqlalchemy import inspect as sqla_inspect
from shopdb.extensions import db
class ImportError_(Exception):
"""A problem with the request itself, not with a row."""
# The tables a site may load, in dependency order: a table never appears before
# something it references. `label` is the human-facing column - the one an
# operator would write in a spreadsheet, and the one used to resolve this table
# when another table points at it. `natural` is what makes a re-import an update
# rather than a duplicate.
#
# Deliberately NOT every table. Transactional and derived data (audit rows,
# reports, enforcement history) has no business arriving by spreadsheet, and
# offering it would invite someone to try.
IMPORTABLE = OrderedDict([
('assetstatuses', {'label': 'status', 'natural': 'status'}),
('assettypes', {'label': 'assettype', 'natural': 'assettype'}),
('locationtypes', {'label': 'locationtype', 'natural': 'locationtype'}),
('modeltypes', {'label': 'modeltype', 'natural': 'modeltype'}),
('computertypes', {'label': 'computertype', 'natural': 'computertype'}),
('machinetypes', {'label': 'machinetype', 'natural': 'machinetype'}),
('businessunits', {'label': 'businessunit', 'natural': 'businessunit'}),
('locations', {'label': 'locationname', 'natural': 'locationname'}),
('vendors', {'label': 'vendor', 'natural': 'vendor'}),
('models', {'label': 'modelnumber', 'natural': 'modelnumber'}),
('operatingsystems', {'label': 'osname', 'natural': 'osname'}),
('assets', {'label': 'assetnumber', 'natural': 'assetnumber'}),
('computers', {'label': 'hostname', 'natural': 'assetid'}),
('machines', {'label': None, 'natural': 'assetid'}),
])
# Columns a CSV may never set: surrogate keys the database owns, and audit
# stamps. Accepting an id from a spreadsheet is how two sites end up with
# colliding primary keys the first time anyone merges data.
NEVER_IMPORT = {'createddate', 'modifieddate'}
# Credentials are never taken from a file. A CSV that carries password hashes
# gets mailed around, and one that carries plaintext is worse. Users are created
# with a random password and must change it at first login.
PASSWORD_COLUMNS = {'passwordhash', 'password'}
TRUE_VALUES = {'1', 'true', 'yes', 'y', 't'}
FALSE_VALUES = {'0', 'false', 'no', 'n', 'f', ''}
def table_for(name):
table = db.metadata.tables.get(name)
if table is None:
raise ImportError_(
"no such table: %s\nImportable: %s" % (name, ', '.join(IMPORTABLE)))
return table
def assert_schema_ready(tablename):
"""Fail with a sentence, not a traceback, when the schema is not there yet.
Running this against a database that has never been migrated is an easy
mistake - a fresh site, or the wrong DATABASE_URL - and SQLAlchemy's answer
is a wall of stack frames ending in 'no such table'. The operator needs to
be told to run the migrations.
"""
# The SESSION's connection, not db.engine. Taking a separate connection ends
# up rolling back the session's in-flight transaction under a shared-
# connection pool, which silently discarded rows that had just been written.
inspector = sqla_inspect(db.session.connection())
if not inspector.has_table(tablename):
raise ImportError_(
"the '%s' table does not exist in this database.\n"
" Run 'flask db upgrade' first, and check DATABASE_URL points at the\n"
" site you meant." % tablename)
def importable_columns(tablename):
"""Columns an operator may set, in a stable order.
Primary keys are excluded - they are the database's - EXCEPT where the key
is also a foreign key, as on the subtype tables where `assetid` both
identifies the row and points at its asset.
"""
table = table_for(tablename)
primary = {c.name for c in table.primary_key}
out = []
for column in table.columns:
if column.name in NEVER_IMPORT:
continue
if column.name in primary and not column.foreign_keys:
continue
if column.name in PASSWORD_COLUMNS:
continue
out.append(column)
return out
def required_columns(tablename):
primary = {c.name for c in table_for(tablename).primary_key}
return [
c.name for c in importable_columns(tablename)
if not c.nullable and c.name not in primary
and c.default is None and c.server_default is None
]
def _fk_target(column):
"""(tablename, idcolumn) this column points at, or None."""
if not column.foreign_keys:
return None
target = list(column.foreign_keys)[0].column
return target.table.name, target.name
def _label_column(tablename):
"""The human-readable column for a table, used to resolve references to it.
Derived where possible so this does not need maintaining: the schema names
reference tables' label column after the table itself (assettypes.assettype,
locationtypes.locationtype). IMPORTABLE overrides where that rule does not
hold.
"""
spec = IMPORTABLE.get(tablename)
if spec and spec.get('label'):
return spec['label']
table = db.metadata.tables.get(tablename)
if table is None:
return None
singular = tablename[:-1] if tablename.endswith('s') else tablename
for candidate in (singular, singular + 'name', tablename + 'name'):
if candidate in table.columns:
return candidate
return None
def _coerce(column, raw):
"""Turn a CSV string into something the column will accept."""
text = (raw or '').strip()
typename = column.type.__class__.__name__.upper()
if text == '':
return None
if 'BOOL' in typename:
low = text.lower()
if low in TRUE_VALUES:
return True
if low in FALSE_VALUES:
return False
raise ValueError("expected 1 or 0, got '%s'" % text)
if 'INT' in typename:
try:
return int(text)
except ValueError:
raise ValueError("expected a whole number, got '%s'" % text)
if 'FLOAT' in typename or 'NUMERIC' in typename or 'DECIMAL' in typename:
try:
return float(text)
except ValueError:
raise ValueError("expected a number, got '%s'" % text)
if 'DATE' in typename or 'TIME' in typename:
from datetime import datetime
for fmt in ('%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%S', '%Y-%m-%d', '%d/%m/%Y', '%m/%d/%Y'):
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
raise ValueError("expected a date like 2026-08-04, got '%s'" % text)
length = getattr(column.type, 'length', None)
if length and len(text) > length:
raise ValueError("longer than the column allows (%d > %d characters)" % (len(text), length))
return text
class Resolver:
"""Turns names into ids, remembering what it has already looked up.
Rows created earlier in the SAME import are resolvable, so one file can
reference another that was loaded moments ago without a commit in between.
"""
def __init__(self):
self._cache = {}
def resolve(self, tablename, idcolumn, value):
if value is None:
return None
# A number is taken at face value: exports round-trip, and some sites do
# know their ids.
if isinstance(value, int) or (isinstance(value, str) and value.strip().isdigit()):
return int(value)
label = _label_column(tablename)
if not label:
raise ValueError(
"cannot look up %s by name - give the numeric %s instead" % (tablename, idcolumn))
key = (tablename, str(value).strip().lower())
if key in self._cache:
return self._cache[key]
table = db.metadata.tables[tablename]
row = db.session.execute(
table.select().where(
db.func.lower(db.func.trim(table.c[label])) == str(value).strip().lower())
).first()
if row is None:
raise ValueError(
"nothing in %s is named '%s' - add it to %s.csv, or import that file first"
% (tablename, value, tablename))
found = getattr(row, idcolumn)
self._cache[key] = found
return found
def remember(self, tablename, name, rowid):
if name is None:
return
self._cache[(tablename, str(name).strip().lower())] = rowid
class RowProblem(object):
def __init__(self, line, column, message):
self.line = line
self.column = column
self.message = message
def __str__(self):
where = "line %d" % self.line
if self.column:
where += ", column '%s'" % self.column
return "%s: %s" % (where, self.message)
class ImportResult(object):
def __init__(self, tablename):
self.tablename = tablename
self.created = 0
self.updated = 0
self.problems = []
self.skipped = 0
@property
def ok(self):
return not self.problems
def summary(self):
return "%s: %d new, %d updated, %d problem(s)" % (
self.tablename, self.created, self.updated, len(self.problems))
def read_rows(text):
"""Rows as dicts, with the sample/comment lines the templates carry removed."""
reader = csv.reader(io.StringIO(text))
header = None
rows = []
for number, fields in enumerate(reader, 1):
if not fields or all(not f.strip() for f in fields):
continue
if fields[0].lstrip().startswith('#'):
continue
if header is None:
# Strip a UTF-8 BOM if one survived: Excel writes it, and a header
# named '\ufeffassetnumber' would match nothing.
header = [f.strip().lstrip('\ufeff') for f in fields]
continue
# A short row is padded rather than rejected: trailing empty cells are
# routinely dropped by spreadsheet exports.
padded = list(fields) + [''] * (len(header) - len(fields))
rows.append((number, dict(zip(header, padded))))
if header is None:
raise ImportError_('the file has no header row')
return header, rows
def import_csv(tablename, text, resolver=None, commit=False):
"""Validate a CSV against one table and write it into the current transaction.
Validation runs over EVERY row first. Nothing is written unless the whole
file is clean, so a typo on line 400 does not leave 399 rows imported.
Writes always go into the SESSION; this never commits. The caller commits or
rolls back, which is what makes a dry run over a whole folder work: assets
can resolve the locations the previous file just created, because they are
really there inside the transaction, and the rollback removes them again.
Skipping the writes on a dry run instead made every cross-file reference
fail, which is the one thing a folder-wide check exists to verify.
`commit` is retained only for callers that want the old explicit read.
"""
table = table_for(tablename)
spec = IMPORTABLE.get(tablename)
if spec is None:
raise ImportError_(
"%s is not importable.\nImportable: %s" % (tablename, ', '.join(IMPORTABLE)))
assert_schema_ready(tablename)
resolver = resolver or Resolver()
result = ImportResult(tablename)
header, rows = read_rows(text)
allowed = {c.name: c for c in importable_columns(tablename)}
unknown = [h for h in header if h not in allowed]
if unknown:
known = ', '.join(sorted(allowed))
result.problems.append(RowProblem(
1, None, "unknown column(s): %s\n this table accepts: %s" % (', '.join(unknown), known)))
return result
missing = [c for c in required_columns(tablename) if c not in header]
if missing:
result.problems.append(RowProblem(
1, None, "required column(s) missing: %s" % ', '.join(missing)))
return result
natural = spec['natural']
if natural not in header:
result.problems.append(RowProblem(
1, None, "the '%s' column identifies each row and must be present" % natural))
return result
prepared = []
for line, raw in rows:
values = {}
failed = False
for name, column in allowed.items():
if name not in raw:
continue
target = _fk_target(column)
# RESOLVE BEFORE COERCING. A foreign key column is an integer, so
# coercing first rejects every name with "expected a whole number" -
# which defeats the entire point of accepting names. Resolution
# handles the numeric case itself.
if target:
text = (raw[name] or '').strip()
if text == '':
values[name] = None
continue
try:
values[name] = resolver.resolve(target[0], target[1], text)
except ValueError as exc:
result.problems.append(RowProblem(line, name, str(exc)))
failed = True
continue
try:
values[name] = _coerce(column, raw[name])
except ValueError as exc:
result.problems.append(RowProblem(line, name, str(exc)))
failed = True
if failed:
continue
if values.get(natural) in (None, ''):
result.problems.append(RowProblem(line, natural, 'must have a value'))
continue
prepared.append((line, values))
if not result.ok:
return result
# Second pass: decide insert vs update. Counted even on a dry run, because
# "this will change 4000 rows" is the number an operator needs before saying
# yes.
label = _label_column(tablename)
for line, values in prepared:
existing = db.session.execute(
table.select().where(table.c[natural] == values[natural])).first()
if existing is None:
result.created += 1
inserted = db.session.execute(table.insert().values(**values))
newid = inserted.inserted_primary_key[0] if inserted.inserted_primary_key else None
if label and label in values:
resolver.remember(tablename, values[label], newid)
else:
result.updated += 1
db.session.execute(
table.update().where(table.c[natural] == values[natural]).values(**values))
if label and label in values:
primary = list(table.primary_key)[0].name
resolver.remember(tablename, values[label], getattr(existing, primary))
return result
def generate_template(tablename):
"""A header row plus a commented example, built from the live schema.
Generated rather than maintained: a hand-written template set drifts on the
next migration, and silently - the file still looks right.
"""
columns = importable_columns(tablename)
required = set(required_columns(tablename))
header = [c.name for c in columns]
notes = []
for column in columns:
bits = []
bits.append('REQUIRED' if column.name in required else 'optional')
target = _fk_target(column)
if target:
label = _label_column(target[0])
if label:
bits.append("name from %s.csv (or a numeric id)" % target[0])
else:
bits.append("numeric %s id" % target[0])
else:
typename = column.type.__class__.__name__.upper()
if 'BOOL' in typename:
bits.append('1 or 0')
elif 'DATE' in typename or 'TIME' in typename:
bits.append('YYYY-MM-DD')
elif getattr(column.type, 'length', None):
bits.append('text, max %d' % column.type.length)
notes.append("# %-24s %s" % (column.name, ', '.join(bits)))
spec = IMPORTABLE[tablename]
lines = [
"# %s.csv - generated from the ShopDB schema, do not hand-maintain" % tablename,
"#",
"# Rows are matched on '%s'. Re-importing updates a matching row rather" % spec['natural'],
"# than creating a second one, so it is safe to correct this file and run again.",
"#",
"# Columns:",
] + notes + [
"#",
"# Lines starting with # are ignored. Delete the example row before importing.",
','.join(header),
'# ' + ','.join(_example(c, required) for c in columns),
]
return '\n'.join(lines) + '\n'
def _example(column, required):
target = _fk_target(column)
if target:
label = _label_column(target[0])
return ('<%s name>' % target[0][:-1]) if label else '<%s id>' % target[0]
typename = column.type.__class__.__name__.upper()
if 'BOOL' in typename:
return '1'
if 'DATE' in typename or 'TIME' in typename:
return '2026-08-04'
if 'INT' in typename or 'FLOAT' in typename or 'NUMERIC' in typename:
return '0'
return '<%s>' % column.name
def dependency_order(names):
"""The given tables, ordered so a table never precedes what it references."""
known = [t for t in IMPORTABLE if t in set(names)]
return known
def table_from_filename(filename):
stem = re.sub(r'\.csv$', '', filename.strip(), flags=re.I)
stem = stem.rsplit('/', 1)[-1].rsplit('\\', 1)[-1].lower()
return stem

236
tests/test_csv_import.py Normal file
View File

@@ -0,0 +1,236 @@
"""Bulk CSV import: name resolution, all-or-nothing safety, and template truth.
The behaviour under test is what makes a spreadsheet import usable by someone
who does not have the database in front of them. In particular: a foreign key
column accepts the NAME of the thing it points at. Requiring numeric ids means
importing a file, reading back generated ids, and pasting them into the next
file - a workflow nobody finishes.
"""
import pytest
from shopdb.core.services.csvimport import (
IMPORTABLE, ImportError_, Resolver, generate_template, import_csv,
importable_columns, required_columns,
)
from shopdb.extensions import db as _db
REFERENCE = [
('assettypes', 'assettype,description\nMachine,Shop floor\nMeasuring Tool,Gage lab\n'),
('assetstatuses', 'status,description\nActive,In service\nRetired,Out\n'),
('locationtypes', 'locationtype\nBay\nRoom\n'),
]
def seed_reference(resolver):
for table, text in REFERENCE:
result = import_csv(table, text, resolver=resolver, commit=True)
assert result.ok, [str(p) for p in result.problems]
result = import_csv(
'locations',
'locationname,building,locationtypeid\nBay 3,Building 1,Bay\nGage Lab,Building 2,Room\n',
resolver=resolver, commit=True)
assert result.ok, [str(p) for p in result.problems]
def count(table):
return _db.session.execute(_db.text('select count(*) from %s' % table)).scalar()
def test_foreign_keys_resolve_by_name(db):
"""The headline behaviour: write 'Gage Lab', not locationid=2."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets',
'assetnumber,name,assettypeid,statusid,locationid\n'
'CMM-01,Zeiss,Measuring Tool,Active,Gage Lab\n',
resolver=resolver, commit=True)
assert result.ok, [str(p) for p in result.problems]
row = _db.session.execute(_db.text(
'select t.assettype, s.status, l.locationname from assets a '
'join assettypes t on t.assettypeid = a.assettypeid '
'join assetstatuses s on s.statusid = a.statusid '
'join locations l on l.locationid = a.locationid')).fetchone()
assert tuple(row) == ('Measuring Tool', 'Active', 'Gage Lab')
def test_numeric_ids_still_work(db):
"""Exports round-trip, and some sites genuinely know their ids."""
resolver = Resolver()
seed_reference(resolver)
typeid = _db.session.execute(_db.text(
"select assettypeid from assettypes where assettype = 'Machine'")).scalar()
result = import_csv(
'assets', 'assetnumber,assettypeid\nMILL-1,%d\n' % typeid,
resolver=resolver, commit=True)
assert result.ok, [str(p) for p in result.problems]
assert count('assets') == 1
def test_unknown_name_names_the_column_and_the_value(db):
"""The error has to be actionable by someone holding a spreadsheet."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets',
'assetnumber,assettypeid,locationid\nX-1,Machine,Bay 9\n',
resolver=resolver, commit=False)
assert not result.ok
message = str(result.problems[0])
assert "locationid" in message
assert "Bay 9" in message
assert "locations.csv" in message
def test_reimport_updates_rather_than_duplicating(db):
"""Sites correct their spreadsheet and run it again. That must not double up."""
resolver = Resolver()
seed_reference(resolver)
text = 'assetnumber,name,assettypeid\nCMM-01,Zeiss,Machine\n'
first = import_csv('assets', text, resolver=resolver, commit=True)
assert (first.created, first.updated) == (1, 0)
renamed = 'assetnumber,name,assettypeid\nCMM-01,Zeiss Contura,Machine\n'
second = import_csv('assets', renamed, resolver=resolver, commit=True)
assert (second.created, second.updated) == (0, 1)
assert count('assets') == 1
assert _db.session.execute(_db.text(
"select name from assets where assetnumber = 'CMM-01'")).scalar() == 'Zeiss Contura'
def test_one_bad_row_writes_nothing(db):
"""A typo on the last line must not leave the earlier lines imported."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets',
'assetnumber,assettypeid\n'
'GOOD-1,Machine\n'
'GOOD-2,Machine\n'
'BAD-3,NoSuchType\n',
resolver=resolver, commit=True)
assert not result.ok
assert count('assets') == 0, 'rows were written despite a failure'
def test_unknown_column_is_rejected_with_the_accepted_list(db):
"""The failure mode of a hand-written template: invented columns."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'vendors', 'vendor,zipcode\nDell,78682\n', resolver=resolver, commit=False)
assert not result.ok
message = str(result.problems[0])
assert 'zipcode' in message
assert 'this table accepts' in message
def test_missing_required_column_is_rejected(db):
result = import_csv('assets', 'name\nNo asset number\n', commit=False)
assert not result.ok
assert 'assetnumber' in str(result.problems[0])
def test_booleans_take_1_and_0_and_reject_prose(db):
resolver = Resolver()
seed_reference(resolver)
ok = import_csv('assets', 'assetnumber,assettypeid,isactive\nA-1,Machine,0\n',
resolver=resolver, commit=True)
assert ok.ok, [str(p) for p in ok.problems]
bad = import_csv('assets', 'assetnumber,assettypeid,isactive\nA-2,Machine,maybe\n',
resolver=resolver, commit=False)
assert not bad.ok
assert 'isactive' in str(bad.problems[0])
def test_comment_and_blank_lines_are_ignored(db):
"""The templates ship a commented example. Importing one unedited should be
a no-op, not a row of literal placeholder text."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets',
'# a comment\nassetnumber,assettypeid\n\n# <assetnumber>,<type>\nREAL-1,Machine\n',
resolver=resolver, commit=True)
assert result.ok, [str(p) for p in result.problems]
assert result.created == 1
def test_passwords_are_never_importable():
"""A CSV carrying password hashes gets mailed around; one carrying plaintext
is worse. Neither column may be set from a file."""
for table in IMPORTABLE:
names = {c.name for c in importable_columns(table)}
assert 'passwordhash' not in names
assert 'password' not in names
def test_generated_templates_only_contain_real_columns(db):
"""The regression that motivated all of this.
A hand-written template set had invented columns on 7 of 11 tables and named
a table that does not exist. Generating from the schema is what makes that
impossible - so assert the generator actually does it.
"""
for table in IMPORTABLE:
text = generate_template(table)
header = [line for line in text.splitlines() if line and not line.startswith('#')][0]
real = {c.name for c in importable_columns(table)}
for column in header.split(','):
assert column in real, '%s.csv offers a column that does not exist: %s' % (table, column)
def test_every_template_includes_its_required_columns(db):
"""A template that omits a required column produces a file that cannot import."""
for table in IMPORTABLE:
header = [line for line in generate_template(table).splitlines()
if line and not line.startswith('#')][0].split(',')
for column in required_columns(table):
assert column in header, '%s.csv is missing required column %s' % (table, column)
def test_importable_tables_all_exist(db):
"""A registry entry naming a table that is not in the schema would fail only
when someone tried to use it."""
for table in IMPORTABLE:
assert table in _db.metadata.tables, '%s is registered but not in the schema' % table
def test_unimportable_table_is_refused(db):
with pytest.raises(ImportError_):
import_csv('auditlog', 'x\n1\n', commit=False)
def test_dry_run_reports_counts_then_rolls_back(db):
"""The number an operator needs before saying yes - and no trace afterwards.
Writes go into the transaction even on a dry run, which is what lets a later
file resolve names an earlier one created. The rollback is what makes it a
dry run.
"""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets', 'assetnumber,assettypeid\nD-1,Machine\nD-2,Machine\n',
resolver=resolver)
assert result.ok
assert result.created == 2
_db.session.rollback()
assert count('assets') == 0
def test_a_dry_run_over_several_files_resolves_across_them(db):
"""The whole-folder check: assets must resolve locations that only exist
because an earlier file in the same run created them."""
resolver = Resolver()
seed_reference(resolver)
result = import_csv(
'assets',
'assetnumber,assettypeid,locationid\nX-1,Machine,Bay 3\n',
resolver=resolver)
assert result.ok, [str(p) for p in result.problems]
_db.session.rollback()
assert count('assets') == 0

51
tests/test_installer_defaults.py
View File

@@ -53,3 +53,54 @@ def test_every_wizard_exclusion_is_a_real_plugin():
for name in wizard_excludes(): for name in wizard_excludes():
assert (PLUGINS / name / 'manifest.json').exists(), ( assert (PLUGINS / name / 'manifest.json').exists(), (
'%s is excluded by the wizard but has no manifest' % name) '%s is excluded by the wizard but has no manifest' % name)
# The preflight blocks the wizard on any FAIL. That is correct for something the
# operator must go and fix, and wrong for anything the installer carries in its
# own bundle - there, blocking stops the wizard over something it was about to do
# itself, with no way forward.
BUNDLE_SUPPLIED = [
'HttpPlatformHandler', # httpplatformhandler\*.msi, installed by stage 4
'URL Rewrite', # urlrewrite\*.msi, installed by stage 4
]
PREFLIGHT = REPO / 'deploy' / 'windows' / 'installer' / 'shopdb-preflight.ps1'
def test_nothing_the_bundle_supplies_is_a_blocker():
"""Regression: HttpPlatformHandler was a FAIL, so a server without it could
not get past the preflight page - to install the very thing that was
missing."""
text = PREFLIGHT.read_text(encoding='utf-8', errors='replace')
offenders = []
for line in text.splitlines():
if "'FAIL'" not in line:
continue
for component in BUNDLE_SUPPLIED:
if component.lower() in line.lower():
offenders.append(line.strip())
assert not offenders, (
'the installer supplies these, so they must not block the wizard:\n %s'
% '\n '.join(offenders))
def test_installer_controlled_paths_have_no_spaces():
"""msiexec and bootstrapper command lines split on spaces.
The installer passes these as PROPERTY=value / TargetDir=value on a command
line. A value containing a space has to be quoted as PROPERTY="C:\\..." -
quoting the whole token instead makes the tool reject the line and print its
usage dialog, which is what 'the bundled MySQL database could not be
installed' looked like from the outside. The Python target was already
space-free for this exact reason; MySQL was not.
"""
text = PREFLIGHT.parent.joinpath('shopdb-install.ps1').read_text(
encoding='utf-8', errors='replace')
offenders = []
for line in text.splitlines():
match = re.match(r"\s*\[string\]\s*\$(Mysql\w*|PyTarget)\s*=\s*'([^']+)'", line)
if match and ' ' in match.group(2):
offenders.append('%s = %s' % (match.group(1), match.group(2)))
assert not offenders, (
'these are passed on a command line and must not contain spaces:\n %s'
% '\n '.join(offenders))

106
tests/test_installer_state.py Normal file
View File

@@ -0,0 +1,106 @@
"""Guards on the installer's durable install record.
The record replaced a set of guesses. "Is this a re-run of MY install?" used to
be answered from whatever the machine looked like - a MySQL service exists, the
database has tables, the site exists - and none of those say who created them.
A retry after a failed first install was therefore taken for an upgrade of a
working system, which dead-ended the retry the wizard invites the operator to
run.
Getting it wrong is asymmetric. Answering "still first provisioning" for a
server holding real data would let `prune-schema --force` drop live tables. The
behavioural checks live in the PowerShell harness next to the installer, run
here when pwsh is available; the text assertions below hold everywhere, because
CI does not necessarily have pwsh and these are the invariants worth failing a
build over.
"""
import shutil
import subprocess
from pathlib import Path
import pytest
REPO = Path(__file__).resolve().parent.parent
INSTALLER = REPO / 'deploy' / 'windows' / 'installer' / 'shopdb-install.ps1'
HARNESS = REPO / 'deploy' / 'windows' / 'installer' / 'tests' / 'test-install-state.ps1'
pytestmark = pytest.mark.skipif(
not INSTALLER.is_file(),
reason='installer not in this checkout')
def installer_text():
return INSTALLER.read_text(encoding='utf-8', errors='replace')
def test_prune_force_is_gated_on_first_provisioning():
"""--force drops tables that contain rows.
It is correct on a first install, where core migrations seed a few plugin
reference tables, and destructive on an upgrade, where those rows are the
site's data. The gate must therefore test BOTH "database was empty" and the
install record, never the database alone.
"""
text = installer_text()
assert "'prune-schema','--yes','--force'" in text, 'the forcing call moved or was renamed'
guard = "if ((-not $script:DbWasEmpty) -and (-not $firstRun)) {"
assert guard in text, (
'the prune gate no longer consults the install record. Forcing on the '
'strength of an empty database alone is what made every retry of a '
'failed first install fail at stage 3.')
def test_first_provisioning_comes_from_the_record_not_the_database():
text = installer_text()
assert 'function Test-FirstProvisioning' in text
assert '$firstRun = Test-FirstProvisioning' in text, (
'stage 3 must ask the record, not infer from table counts')
def test_missing_record_takes_the_safe_side():
"""An install predating the record has a version stamp and probably data.
Treating that as a first run would arm prune --force against it.
"""
text = installer_text()
assert "'predates-this-record'" in text, (
'the fallback for an unrecorded but stamped install is gone; without it '
'an established server can be classified as first provisioning')
def test_created_items_returns_an_array_when_empty():
"""A zero-length array returned from a PowerShell function unrolls to $null.
$null.Count is then a terminating error under Set-StrictMode -Version 2.0 -
the same fault that made bundle verification fail on every install. The
leading comma is what prevents it.
"""
text = installer_text()
assert 'return ,@()' in text, 'Get-CreatedItems lost its comma protection'
assert 'return ,@(Get-StateValue $created $Kind' in text, (
'the populated return path lost its comma protection')
def test_record_is_written_before_any_stage_runs():
"""Written at the START of provisioning, not the end.
A run that dies halfway is exactly the run whose retry needs the record.
"""
text = installer_text()
assert "if ($Stage -ne 'uninstall') { Initialize-InstallState }" in text
start = text.index('Initialize-InstallState }')
switch = text.index('switch ($Stage)', start - 400)
assert start < switch, 'the record must be initialised before the stage switch'
@pytest.mark.skipif(shutil.which('pwsh') is None, reason='pwsh not installed')
def test_install_state_behaviour():
"""Run the PowerShell harness: retries, upgrades, corrupt and foreign records."""
assert HARNESS.is_file(), 'the PowerShell harness is missing'
result = subprocess.run(
['pwsh', '-NoProfile', '-File', str(HARNESS)],
capture_output=True, text=True, timeout=300)
assert result.returncode == 0, (
'install-record harness failed:\n' + result.stdout + result.stderr)
assert 'all checks passed' in result.stdout

159
tests/test_runtime_dependencies.py Normal file
View File

@@ -0,0 +1,159 @@
"""Every third-party module the app imports at runtime must be a DECLARED dependency.
The bug this exists to prevent: `shopdb/plugins/loader.py` imports `packaging`,
which was never listed in requirements.in. Locally and in CI that import
resolved anyway, because pytest depends on packaging -- so all 1159 tests
passed. The Windows installer builds its venv from requirements.txt alone, with
no test dependencies, so `import shopdb` raised ModuleNotFoundError and the
whole application failed to start on a customer server.
Test dependencies are exactly the blind spot: they are present everywhere the
test suite runs and absent everywhere it does not.
The check is static (AST) rather than a real import, so it covers plugins that
are not installed at this site and code paths no test exercises.
"""
import ast
import sys
from pathlib import Path
import pytest
try:
from importlib.metadata import packages_distributions
except ImportError: # pragma: no cover - Python < 3.10
packages_distributions = None
REPO = Path(__file__).resolve().parent.parent
SCANNED = ('shopdb', 'plugins', 'scripts')
# First-party top-level names. Imports of these resolve from the source tree.
FIRST_PARTY = {'shopdb', 'plugins', 'scripts', 'migrations', 'tests', 'wsgi', 'conftest'}
# Modules deliberately imported without being declared. Each needs a reason.
ALLOWED_UNDECLARED = {
# Provided by the interpreter's own bundled installer tooling, and only
# touched by developer scripts that never run on a deployed site.
'pip',
'setuptools',
'pkg_resources',
}
def _normalise(name):
"""PEP 503 normalisation, so Flask_Migrate and flask-migrate compare equal."""
return name.lower().replace('_', '-').replace('.', '-')
def _declared_distributions():
"""Distribution names pinned in requirements.txt."""
text = (REPO / 'requirements.txt').read_text(encoding='utf-8')
declared = set()
for line in text.splitlines():
line = line.strip()
if not line or line.startswith('#') or line.startswith('--'):
continue
# "name==1.2.3 ; marker \" -> "name"
head = line.split('==', 1)[0].split(';', 1)[0].strip()
if head:
declared.add(_normalise(head))
return declared
def _iter_source_files():
for top in SCANNED:
root = REPO / top
if not root.is_dir():
continue
for path in root.rglob('*.py'):
parts = set(path.parts)
if {'node_modules', 'venv', '.venv', 'build'} & parts:
continue
yield path
def _imported_top_level_modules(path):
"""Top-level module names imported unconditionally by `path`.
Imports nested inside a `try:` are skipped: that is how this codebase spells
an optional dependency, and a missing one is handled rather than fatal.
Imports inside functions are still counted -- a deferred import of an
undeclared package fails just as hard, only later.
"""
try:
tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))
except SyntaxError:
pytest.fail('could not parse {}'.format(path.relative_to(REPO)))
guarded = set()
for node in ast.walk(tree):
if isinstance(node, ast.Try):
for child in ast.walk(node):
guarded.add(id(child))
found = set()
for node in ast.walk(tree):
if id(node) in guarded:
continue
if isinstance(node, ast.Import):
for alias in node.names:
found.add(alias.name.split('.')[0])
elif isinstance(node, ast.ImportFrom):
# level > 0 is a relative (first-party) import.
if node.level == 0 and node.module:
found.add(node.module.split('.')[0])
return found
@pytest.mark.skipif(packages_distributions is None, reason='needs Python 3.10+')
def test_runtime_imports_are_declared_dependencies():
declared = _declared_distributions()
assert 'flask' in declared, 'requirements.txt did not parse as expected'
module_to_dists = packages_distributions()
stdlib = getattr(sys, 'stdlib_module_names', frozenset())
offenders = {}
for path in _iter_source_files():
for module in _imported_top_level_modules(path):
if module in FIRST_PARTY or module in stdlib or module in ALLOWED_UNDECLARED:
continue
if module.startswith('_'):
continue
dists = module_to_dists.get(module)
if not dists:
# Not installed in this environment, so its distribution cannot
# be identified. Absence here is not evidence of a problem.
continue
if not any(_normalise(d) in declared for d in dists):
rel = str(path.relative_to(REPO))
offenders.setdefault(
'{} (from {})'.format(module, '/'.join(sorted(dists))), []
).append(rel)
if offenders:
lines = []
for module, files in sorted(offenders.items()):
shown = sorted(files)[:3]
more = '' if len(files) <= 3 else ' (+{} more)'.format(len(files) - 3)
lines.append(' {}\n {}{}'.format(module, ', '.join(shown), more))
pytest.fail(
'Imported at runtime but not declared in requirements.txt.\n'
'These resolve here only because a test dependency supplies them; a\n'
'production venv has no test dependencies and will fail to import.\n'
'Add them to requirements.in and recompile.\n\n' + '\n'.join(lines)
)
def test_packaging_is_declared():
"""Regression: the specific import that broke the Windows install.
Kept separate from the sweep above because that one can only flag a module
it can attribute to an installed distribution -- an environment without
packaging would skip it silently, which is precisely the shipping case.
"""
assert 'packaging' in _declared_distributions(), (
'shopdb/plugins/loader.py imports packaging at module scope. Without it '
'declared, `import shopdb` fails on any venv built from requirements.txt.'
)