Release 0.8.0: Windows installer, twelve fixes from a real Server 2019 install, operator docs
This commit is contained in:
66
CHANGELOG.md
66
CHANGELOG.md
@@ -10,7 +10,71 @@ ADR-007 and ADR-002.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
## [0.8.0] - 2026-08-05
|
||||
|
||||
First release to carry the Windows installer. Everything below shipped after
|
||||
v0.7.0 was tagged, and a complete install was exercised end to end on Windows
|
||||
Server 2019 before this release was cut.
|
||||
|
||||
### Fixed - installer, from a real Server 2019 install
|
||||
|
||||
- `packaging` was imported by the plugin loader but declared nowhere. It reached
|
||||
development and CI only as a dependency of pytest, so the whole suite passed
|
||||
while a virtual environment built from `requirements.txt` alone - which is
|
||||
exactly what the installer builds - could not import the application at all.
|
||||
`tests/test_runtime_dependencies.py` now fails on any runtime import that is
|
||||
not a declared dependency.
|
||||
- IIS returned 500.52 before the application was ever launched. `web.config`
|
||||
declared `<allowedServerVariables>` for the X-Forwarded-For rule, and that
|
||||
section ships `overrideModeDefault="Deny"`, so the whole file was rejected.
|
||||
The installer now permits the single variable at server level instead of
|
||||
unlocking the section for every site on the machine, and repairs a
|
||||
`web.config` an earlier build had made unusable.
|
||||
- The config unlock ran before the application it unlocks existed, so the scoped
|
||||
form could never succeed on a first install and every install silently fell
|
||||
back to granting handler delegation server-wide.
|
||||
- The stage 5 smoke test discarded the status code and error page it had already
|
||||
received, reporting "site did not return 200" for a fault IIS had named. It
|
||||
now records both, plus the tail of the application log.
|
||||
- Database dumps were readable by every authenticated user: a directory created
|
||||
under ProgramData inherits `Users:RX`, and the owner-only ACL was applied only
|
||||
when the installer itself created it.
|
||||
- The uninstaller ran the 32-bit PowerShell, which cannot see IIS, so the site,
|
||||
pool and application survived a "successful" uninstall pointing at a deleted
|
||||
directory.
|
||||
- Uninstall matched applications by alias alone and would remove an unrelated
|
||||
application of the same name under another site.
|
||||
- Wizard input reached a command line unchecked: unvalidated ports, a drive-root
|
||||
path that escaped its own quote, and a password written as ANSI but read back
|
||||
as UTF-8, which reported a correct non-ASCII password as wrong.
|
||||
- Plugin deregistration could never succeed - it omitted `--yes` against a
|
||||
command that prompts - while the plugin directory was deleted regardless.
|
||||
- Preflight rows were drawn past the bottom of the panel and silently vanished;
|
||||
the failures loop had no cap at all.
|
||||
|
||||
### Changed - installer behaviour
|
||||
|
||||
- "Is this a re-run of my install?" is answered from a durable install record
|
||||
rather than inferred from the state of the machine. Nothing on a server says
|
||||
who created its database tables, so a retry after a failed first install was
|
||||
taken for an upgrade of a working system: it demanded a mandatory backup of a
|
||||
database its own failed attempt had written, then refused to prune tables it
|
||||
had created minutes earlier. During unfinished first provisioning the backup
|
||||
is advisory and prune may force; on an established install both are unchanged.
|
||||
- The installer offers every bundled plugin, so one build serves any site
|
||||
instead of one build per plugin profile.
|
||||
|
||||
### Added - operator documentation and diagnostics
|
||||
|
||||
- `docs/UPDATES-WINDOWS.md` - what operators should expect from future updates,
|
||||
bug fixes and security releases, including downtime, what is preserved, and
|
||||
the effect on other sites sharing the same IIS server.
|
||||
- `docs/RELEASING-WINDOWS.md` - how to build and release, and the two known gaps.
|
||||
- `deploy/windows/shopdb-diagnose.py` - collects what IIS answers, the config
|
||||
lock state, the application logs and the ACLs in one pass, scrubbing secrets
|
||||
before writing anything.
|
||||
|
||||
### Added - the installer itself
|
||||
|
||||
- Air-gapped Windows installer (`deploy/windows/installer/`). One `.exe` per
|
||||
site, built from that site's plugin profile, containing Python, the wheels,
|
||||
|
||||
20
deploy/site-profile-universal.json
Normal file
20
deploy/site-profile-universal.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"site": "universal",
|
||||
"plugins": [
|
||||
"computers",
|
||||
"employees",
|
||||
"geenforce",
|
||||
"knowledgebase",
|
||||
"machines",
|
||||
"measuringtools",
|
||||
"network",
|
||||
"notifications",
|
||||
"printedparts",
|
||||
"printers",
|
||||
"slides",
|
||||
"usb",
|
||||
"warranty"
|
||||
],
|
||||
"locked": [],
|
||||
"_comment": "The profile the released Windows installer is built from. Every bundled plugin that carries a manifest, so ONE exe serves any site: the wizard offers all of them and the operator ticks what that site uses. Plugins left unticked are never installed, and 'flask plugin prune-schema' drops their tables at provisioning (ADR-014). 'applications' is deliberately absent - it is manifest-less core and always ships. Build with: deploy/windows/installer/build-installer.sh deploy/site-profile-universal.json <repo>. Use site-profile.example.json instead only when a site genuinely needs a lean build; see ADR-013."
|
||||
}
|
||||
@@ -53,7 +53,9 @@ bundle/wheels/ pip download -r requirements.txt --only-binary=:all
|
||||
bundle/python/ python-3.14.x-amd64.exe
|
||||
bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
|
||||
bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below)
|
||||
bundle/mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)
|
||||
bundle/vcredist/ VC_redist.x64.exe (MySQL will not install without it)
|
||||
bundle/mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)
|
||||
bundle/mysqlclient/ mysql.exe, mysqldump.exe + libcrypto/libssl (backups)
|
||||
```
|
||||
|
||||
Then compile on Windows:
|
||||
@@ -235,7 +237,7 @@ database is an upgrade, and treating it as fresh would drop tables.
|
||||
|
||||
## Testing notes
|
||||
|
||||
Verified end to end on Windows Server 2025 against both a bundled MySQL 8.0 and
|
||||
Verified end to end on Windows Server 2025 against both a bundled MySQL 8.4 LTS and
|
||||
an existing MySQL 5.6: fresh install, upgrade, re-run idempotency, failure and
|
||||
rollback, uninstall, and both deployment methods including switching between
|
||||
them.
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
;
|
||||
; DESIGN NOTE - why this is a thin wrapper and not a reimplementation:
|
||||
; every install action lives in shopdb-install.ps1, which is verified end to end
|
||||
; on Windows Server 2025 against both a bundled MySQL 8.0 and an existing MySQL
|
||||
; on Windows Server 2025 against both a bundled MySQL 8.4 LTS and an existing MySQL
|
||||
; 5.6. Reimplementing any of it in Pascal Script would create a second code path
|
||||
; that nobody tests. This file only collects operator input, runs the stages in
|
||||
; order, and reports which one failed.
|
||||
@@ -181,6 +181,7 @@ Filename: "{win}\Sysnative\WindowsPowerShell\v1.0\powershell.exe"; \
|
||||
var
|
||||
DbChoicePage: TInputOptionWizardPage;
|
||||
DbDetailsPage: TInputQueryWizardPage;
|
||||
DbCredsPage: TInputQueryWizardPage;
|
||||
SitePage: TInputQueryWizardPage;
|
||||
PreflightPage: TWizardPage;
|
||||
BannerPanel: TPanel;
|
||||
@@ -402,33 +403,40 @@ begin
|
||||
'choose the second option - installing a second server would collide on ' +
|
||||
'port 3306.',
|
||||
True, False);
|
||||
DbChoicePage.Add('Install the bundled MySQL 8.0 (new servers only)');
|
||||
DbChoicePage.Add('Install the bundled MySQL 8.4 LTS (new servers only)');
|
||||
DbChoicePage.Add('Use a MySQL server this site already runs');
|
||||
DbChoicePage.SelectedValueIndex := DB_EXISTING;
|
||||
|
||||
DbDetailsPage := CreateInputQueryPage(DbChoicePage.ID,
|
||||
'Existing database', 'Connection details',
|
||||
'The database and application user must already exist. Your DBA creates ' +
|
||||
'them; the installer does not, so that it never needs administrative ' +
|
||||
'rights on your database server. If they need the exact statements:' + #13#10 +
|
||||
' CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;' + #13#10 +
|
||||
' CREATE USER ''shopdb''@''%'' IDENTIFIED BY ''<password>'';' + #13#10 +
|
||||
' GRANT ALL PRIVILEGES ON shopdb_flask.* TO ''shopdb''@''%'';' + #13#10 +
|
||||
'The utf8mb4 charset matters - the default on older servers is latin1.');
|
||||
'The database and application user must already exist - the installer does ' +
|
||||
'not create them, so it never needs administrative rights on your database ' +
|
||||
'server. The exact SQL to give your DBA is in the INSTALL-WINDOWS guide.');
|
||||
DbDetailsPage.Add('Host:', False);
|
||||
DbDetailsPage.Add('Port:', False);
|
||||
DbDetailsPage.Add('Database:', False);
|
||||
DbDetailsPage.Add('Username:', False);
|
||||
DbDetailsPage.Add('Password:', True);
|
||||
DbDetailsPage.Values[0] := '127.0.0.1';
|
||||
DbDetailsPage.Values[1] := '3306';
|
||||
DbDetailsPage.Values[2] := 'shopdb_flask';
|
||||
DbDetailsPage.Values[3] := 'shopdb';
|
||||
|
||||
// SPLIT ACROSS TWO PAGES on purpose. CreateInputQueryPage stacks its fields
|
||||
// below the description and does not scroll or shrink: with five fields the
|
||||
// last one is drawn past the bottom of the surface and simply does not appear.
|
||||
// Trimming the description bought one field back and lost the next, which is
|
||||
// guessing at a pixel budget that varies with DPI and font. Three fields and
|
||||
// two fields both fit under any reasonable description, at any scaling.
|
||||
DbCredsPage := CreateInputQueryPage(DbDetailsPage.ID,
|
||||
'Existing database', 'Sign-in',
|
||||
'The account the application uses to reach that database. It needs full ' +
|
||||
'rights on it, and nothing outside it.');
|
||||
DbCredsPage.Add('Username:', False);
|
||||
DbCredsPage.Add('Password:', True);
|
||||
DbCredsPage.Values[0] := 'shopdb';
|
||||
|
||||
// How the application is published. Offered only when the bundle actually
|
||||
// carries a subpath SPA build - Vite compiles the base path in, so this can
|
||||
// never be a pure runtime switch.
|
||||
DeployPage := CreateInputOptionPage(DbDetailsPage.ID,
|
||||
DeployPage := CreateInputOptionPage(DbCredsPage.ID,
|
||||
'Address', 'How should people reach ShopDB-Flask?',
|
||||
'Both options serve the same application. The second needs no new DNS name '
|
||||
+ 'and no port number, because it rides this server''s existing address.',
|
||||
@@ -647,7 +655,7 @@ end;
|
||||
// If nothing is wrong the page collapses to a single reassuring statement.
|
||||
procedure RenderPreflight(Lines: TArrayOfString; var HasBlockers: Boolean);
|
||||
var
|
||||
I, Y, Shown, Fails, Warns, Passes: Integer;
|
||||
I, Y, Shown, Fails, Warns, Passes, RowHeight, MaxRows, Hidden: Integer;
|
||||
Parts: TArrayOfString;
|
||||
Check, Detail, Fix: String;
|
||||
begin
|
||||
@@ -655,6 +663,16 @@ begin
|
||||
Y := ScaleY(4);
|
||||
Shown := 0;
|
||||
|
||||
// How many rows actually FIT, measured, not guessed. The old cap of 6 bore no
|
||||
// relation to the panel, which holds roughly three: at four or five notes the
|
||||
// rows below were drawn past the bottom edge and silently vanished - taking
|
||||
// with them the warning that the bundled MySQL collides on port 3306, which is
|
||||
// exactly what the operator needs before the very next page.
|
||||
RowHeight := ScaleY(46) + ScaleY(ROW_GAP);
|
||||
if RowHeight < 1 then RowHeight := 1;
|
||||
MaxRows := DetailPanel.Height div RowHeight;
|
||||
if MaxRows < 1 then MaxRows := 1;
|
||||
|
||||
// Blockers first, then warnings - a second pass rather than one, so severity
|
||||
// ordering does not depend on the order the checks happen to run in.
|
||||
for I := 0 to GetArrayLength(Lines) - 1 do
|
||||
@@ -668,6 +686,9 @@ begin
|
||||
|
||||
for I := 0 to GetArrayLength(Lines) - 1 do
|
||||
begin
|
||||
// Blockers are capped too. Without this the FAIL loop drew every failure,
|
||||
// however many, straight off the bottom of the panel.
|
||||
if Shown >= MaxRows then Break;
|
||||
Parts := StringSplit(Lines[I], ['|'], stAll);
|
||||
if GetArrayLength(Parts) < 4 then Continue;
|
||||
if Parts[0] <> 'FAIL' then Continue;
|
||||
@@ -682,7 +703,7 @@ begin
|
||||
|
||||
for I := 0 to GetArrayLength(Lines) - 1 do
|
||||
begin
|
||||
if Shown >= 6 then Break;
|
||||
if Shown >= MaxRows then Break;
|
||||
Parts := StringSplit(Lines[I], ['|'], stAll);
|
||||
if GetArrayLength(Parts) < 4 then Continue;
|
||||
if Parts[0] <> 'WARN' then Continue;
|
||||
@@ -719,8 +740,13 @@ begin
|
||||
+ 'verified. Nothing needs your attention.');
|
||||
end;
|
||||
|
||||
if (Shown >= 6) and (Warns > 6 - Fails) then
|
||||
FooterText.Caption := 'Some notes are not shown. The full check is in the install log.'
|
||||
// Count what was actually left out rather than inferring it from the old cap,
|
||||
// which under-reported: with 2 failures and 5 warnings it claimed everything
|
||||
// was shown while four rows had been dropped.
|
||||
Hidden := (Fails + Warns) - Shown;
|
||||
if Hidden > 0 then
|
||||
FooterText.Caption := IntToStr(Hidden) + ' further item(s) are not shown here. '
|
||||
+ 'All of them are in the install log.'
|
||||
else
|
||||
FooterText.Caption := IntToStr(Passes) + ' checks passed. Nothing has been changed on this server.';
|
||||
end;
|
||||
@@ -829,10 +855,76 @@ begin
|
||||
Log('[shopdb] pre-fill base directory: ' + Result);
|
||||
end;
|
||||
|
||||
// Everything typed in the wizard ends up on a command line that
|
||||
// CommandLineToArgvW parses, and reaches PowerShell parameters typed [int] or
|
||||
// [string]. A blank, a space, or a trailing backslash therefore has to be caught
|
||||
// HERE - by the time the script sees it, the arguments have already shifted.
|
||||
|
||||
function IsValidPort(const S: String): Boolean;
|
||||
var
|
||||
I, N: Integer;
|
||||
begin
|
||||
Result := False;
|
||||
if (S = '') or (Length(S) > 5) then Exit;
|
||||
for I := 1 to Length(S) do
|
||||
if (S[I] < '0') or (S[I] > '9') then Exit; // also rejects spaces and signs
|
||||
N := StrToIntDef(S, -1);
|
||||
Result := (N >= 1) and (N <= 65535);
|
||||
end;
|
||||
|
||||
function IsAsciiOnly(const S: String): Boolean;
|
||||
var
|
||||
I: Integer;
|
||||
begin
|
||||
Result := True;
|
||||
for I := 1 to Length(S) do
|
||||
if Ord(S[I]) > 126 then
|
||||
begin
|
||||
Result := False;
|
||||
Exit;
|
||||
end;
|
||||
end;
|
||||
|
||||
// A path that ends in a backslash - a drive root such as D:\ - closes the
|
||||
// argument with \" , which CommandLineToArgvW reads as an ESCAPED quote. The
|
||||
// argument never terminates and every argument after it shifts one place.
|
||||
// Doubling the backslash inside the quotes is the documented way out.
|
||||
function QuotePathArg(const P: String): String;
|
||||
begin
|
||||
if (P <> '') and (P[Length(P)] = '\') then
|
||||
Result := '"' + P + '\' + '"'
|
||||
else
|
||||
Result := '"' + P + '"';
|
||||
end;
|
||||
|
||||
function NextButtonClick(CurPageID: Integer): Boolean;
|
||||
begin
|
||||
Result := True;
|
||||
|
||||
// A drive root is refused outright rather than quoted around. Uninstall
|
||||
// removes the application directory recursively, so accepting D:\ here would
|
||||
// mean uninstalling ShopDB-Flask wipes the whole drive.
|
||||
if CurPageID = wpSelectDir then
|
||||
begin
|
||||
if Length(WizardDirValue) <= 3 then
|
||||
begin
|
||||
MsgBox('Choose a folder rather than a whole drive.' + #13#10#13#10
|
||||
+ 'Removing ShopDB-Flask deletes the folder it was installed into, '
|
||||
+ 'so installing to a drive root would delete everything on that '
|
||||
+ 'drive when it is uninstalled.', mbError, MB_OK);
|
||||
Result := False;
|
||||
end;
|
||||
end;
|
||||
|
||||
if CurPageID = SitePage.ID then
|
||||
begin
|
||||
if not IsValidPort(SitePage.Values[1]) then
|
||||
begin
|
||||
MsgBox('Enter the port as a number between 1 and 65535.', mbError, MB_OK);
|
||||
Result := False;
|
||||
end;
|
||||
end;
|
||||
|
||||
if CurPageID = wpWelcome then
|
||||
RunPreflight;
|
||||
|
||||
@@ -865,8 +957,34 @@ begin
|
||||
MsgBox('Enter the database host.', mbError, MB_OK);
|
||||
Result := False;
|
||||
end
|
||||
else if (DbDetailsPage.Values[4] = '') and
|
||||
(not FileExists(InstalledDir + '.env')) then
|
||||
else if not IsValidPort(DbDetailsPage.Values[1]) then
|
||||
begin
|
||||
MsgBox('Enter the database port as a number between 1 and 65535.'
|
||||
+ #13#10#13#10 + 'MySQL uses 3306 unless it was changed.',
|
||||
mbError, MB_OK);
|
||||
Result := False;
|
||||
end;
|
||||
end;
|
||||
end;
|
||||
|
||||
if CurPageID = DbCredsPage.ID then
|
||||
begin
|
||||
if not UseBundledDb then
|
||||
begin
|
||||
if DbCredsPage.Values[0] = '' then
|
||||
begin
|
||||
MsgBox('Enter the username the application connects with.', mbError, MB_OK);
|
||||
Result := False;
|
||||
end
|
||||
// Blank is allowed when the password is already ON DISK in a form the
|
||||
// installer can read: .env from a previous install, or the ACL'd .dbpass
|
||||
// handoff stage 0 leaves when it created the database itself. Stage 2
|
||||
// consumes .dbpass on its own, so demanding the password here blocked the
|
||||
// operator on something the installer already had - and sent them looking
|
||||
// for a generated secret they were never meant to handle.
|
||||
else if (DbCredsPage.Values[1] = '') and
|
||||
(not FileExists(InstalledDir + '.env')) and
|
||||
(not FileExists(InstalledDir + '.dbpass')) then
|
||||
begin
|
||||
// Only required on a FRESH install. On an upgrade, blank means "keep the
|
||||
// password already in .env", so the operator never has to know it.
|
||||
@@ -913,6 +1031,13 @@ begin
|
||||
if (CurPageID = DbDetailsPage.ID) and (not DbPageReady) then
|
||||
begin
|
||||
DbPageReady := True;
|
||||
// A stage-0 handoff means the database was created by a previous run of this
|
||||
// installer and the password is already on disk. Nobody should be asked to
|
||||
// find or retype a secret the installer generated.
|
||||
if FileExists(InstalledDir + '.dbpass') then
|
||||
DbCredsPage.SubCaptionLabel.Caption :=
|
||||
'This server already has a database created by this installer. Leave the '
|
||||
+ 'password blank and it will be used automatically.';
|
||||
if LoadStringsFromFile(InstalledDir + '.env', Lines) then
|
||||
for I := 0 to GetArrayLength(Lines) - 1 do
|
||||
if Pos('DATABASE_URL=', Lines[I]) = 1 then
|
||||
@@ -924,7 +1049,7 @@ begin
|
||||
Creds := Copy(Existing, 1, Pos('@', Existing) - 1);
|
||||
Rest := Copy(Existing, Pos('@', Existing) + 1, Length(Existing));
|
||||
if Pos(':', Creds) > 0 then
|
||||
DbDetailsPage.Values[3] := Copy(Creds, 1, Pos(':', Creds) - 1);
|
||||
DbCredsPage.Values[0] := Copy(Creds, 1, Pos(':', Creds) - 1);
|
||||
if Pos(':', Rest) > 0 then
|
||||
begin
|
||||
DbDetailsPage.Values[0] := Copy(Rest, 1, Pos(':', Rest) - 1);
|
||||
@@ -940,10 +1065,13 @@ begin
|
||||
end;
|
||||
// Password intentionally left blank: blank means "keep the current
|
||||
// one", so an upgrade never needs the operator to know it.
|
||||
DbDetailsPage.Values[4] := '';
|
||||
DbCredsPage.Values[1] := '';
|
||||
DbDetailsPage.SubCaptionLabel.Caption :=
|
||||
'These are the settings this server is using now. Leave the password '
|
||||
+ 'blank to keep the current one.';
|
||||
'These are the settings this server is using now.';
|
||||
// The hint belongs where the password field actually is - it moved to
|
||||
// the next page when this one was split.
|
||||
DbCredsPage.SubCaptionLabel.Caption :=
|
||||
'Leave the password blank to keep the one this server already uses.';
|
||||
end;
|
||||
end;
|
||||
|
||||
@@ -994,7 +1122,7 @@ begin
|
||||
end;
|
||||
// The bundled path generates its own credentials, so asking for them would be
|
||||
// meaningless - and any value typed here would be silently ignored.
|
||||
Result := (PageID = DbDetailsPage.ID) and UseBundledDb;
|
||||
Result := ((PageID = DbDetailsPage.ID) or (PageID = DbCredsPage.ID)) and UseBundledDb;
|
||||
end;
|
||||
|
||||
// Runs from ssPostInstall, NOT PrepareToInstall.
|
||||
@@ -1019,22 +1147,31 @@ var
|
||||
Cmd: String;
|
||||
begin
|
||||
if not FileExists(LnkPath) then Exit;
|
||||
// THREE quotes each side, not four. Four is an escaped quote twice over, which
|
||||
// keeps the whole expression inside one Pascal literal: LnkPath was never
|
||||
// interpolated, PowerShell got the bare word "LnkPath", failed to parse, and
|
||||
// no shortcut ever received the elevation flag. Self-elevation hid it - the
|
||||
// only symptom was a second window after an extra prompt.
|
||||
Cmd := '-NoProfile -ExecutionPolicy Bypass -Command "'
|
||||
+ '$p='''' + LnkPath + ''''; '
|
||||
+ '$p=''' + LnkPath + '''; '
|
||||
+ '$b=[IO.File]::ReadAllBytes($p); '
|
||||
+ '$b[21]=$b[21] -bor 0x20; '
|
||||
+ '[IO.File]::WriteAllBytes($p,$b)"';
|
||||
Exec(PowerShellPath, Cmd, '', SW_HIDE, ewWaitUntilTerminated, ResultCode);
|
||||
if ResultCode <> 0 then
|
||||
Log('[shopdb] could not mark ' + LnkPath + ' run-as-administrator (exit '
|
||||
+ IntToStr(ResultCode) + '); the console will self-elevate instead');
|
||||
end;
|
||||
|
||||
function RunInstallStages: String;
|
||||
var
|
||||
ResultCode: Integer;
|
||||
PwFile, Args, Common: String;
|
||||
PwLines: TArrayOfString;
|
||||
begin
|
||||
Result := '';
|
||||
Common := '-BundleRoot "' + ExpandConstant('{tmp}\shopdb-bundle') + '"' +
|
||||
' -AppRoot "' + ExpandConstant('{app}') + '"' +
|
||||
Common := '-BundleRoot ' + QuotePathArg(ExpandConstant('{tmp}\shopdb-bundle')) +
|
||||
' -AppRoot ' + QuotePathArg(ExpandConstant('{app}')) +
|
||||
' -SiteHost "' + SitePage.Values[0] + '"' +
|
||||
' -SitePort ' + SitePage.Values[1] +
|
||||
' -OnFailure never' +
|
||||
@@ -1070,10 +1207,16 @@ begin
|
||||
// PowerShell transcripts. The installer shreds the file after reading it.
|
||||
// No password typed on an upgrade means "leave .env alone" - so send no
|
||||
// password file, and the installer keeps the existing DATABASE_URL.
|
||||
if DbDetailsPage.Values[4] <> '' then
|
||||
if DbCredsPage.Values[1] <> '' then
|
||||
begin
|
||||
PwFile := ExpandConstant('{tmp}\dbpw.txt');
|
||||
SaveStringToFile(PwFile, DbDetailsPage.Values[4] + #13#10, False);
|
||||
// UTF-8, no BOM. SaveStringToFile writes an AnsiString, but the installer
|
||||
// reads this back with -Encoding UTF8 - so any non-ASCII character in the
|
||||
// password came out mangled and the database rejected a password that was
|
||||
// typed correctly, reported as "wrong username or password".
|
||||
SetArrayLength(PwLines, 1);
|
||||
PwLines[0] := DbCredsPage.Values[1];
|
||||
SaveStringsToUTF8FileWithoutBOM(PwFile, PwLines, False);
|
||||
end
|
||||
else
|
||||
PwFile := '';
|
||||
@@ -1081,7 +1224,7 @@ begin
|
||||
' -DbHost "' + DbDetailsPage.Values[0] + '"' +
|
||||
' -DbPort ' + DbDetailsPage.Values[1] +
|
||||
' -DbName "' + DbDetailsPage.Values[2] + '"' +
|
||||
' -DbUser "' + DbDetailsPage.Values[3] + '"';
|
||||
' -DbUser "' + DbCredsPage.Values[0] + '"';
|
||||
if PwFile <> '' then Args := Args + ' -DbPasswordFile "' + PwFile + '"';
|
||||
end;
|
||||
|
||||
@@ -1169,6 +1312,12 @@ end;
|
||||
; Delegate to the same tested script rather than duplicating removal logic here.
|
||||
; It removes the site, app pool, firewall rule and application directory, and
|
||||
; deliberately does NOT drop the database or uninstall MySQL.
|
||||
Filename: "powershell.exe"; \
|
||||
;
|
||||
; Sysnative, for the same reason as the [Run] entry above: the uninstaller is a
|
||||
; 32-bit process, so a bare "powershell.exe" is resolved through WOW64 to the
|
||||
; 32-bit PowerShell, which cannot see the IIS provider at all. Removal then
|
||||
; skipped the site, application pool and application - leaving them pointing at
|
||||
; a directory it HAD deleted - while Windows reported a clean uninstall.
|
||||
Filename: "{win}\Sysnative\WindowsPowerShell\v1.0\powershell.exe"; \
|
||||
Parameters: "-NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-install.ps1"" -Stage uninstall -BundleRoot ""{app}"" -AppRoot ""{app}"" -OnFailure never"; \
|
||||
RunOnceId: "ShopDBFlaskUninstall"; Flags: waituntilterminated runhidden
|
||||
|
||||
@@ -286,7 +286,7 @@ refresh-bundle-lock.ps1, read the diff, and commit the new lock).
|
||||
# --- summary ----------------------------------------------------------------
|
||||
Write-Host ''
|
||||
Say " Bundle staged at: $BundleRoot" 'White'
|
||||
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'mysql')) {
|
||||
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'vcredist', 'mysqlclient', 'mysql')) {
|
||||
$p = Join-Path $BundleRoot $d
|
||||
if (Test-Path $p) {
|
||||
$mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB)
|
||||
|
||||
@@ -82,7 +82,7 @@ done
|
||||
|
||||
echo ""
|
||||
echo "Bundle staged at: $BUNDLE"
|
||||
for d in app wheels python httpplatformhandler urlrewrite mysql; do
|
||||
for d in app wheels python httpplatformhandler urlrewrite vcredist mysqlclient mysql; do
|
||||
if [ -d "$BUNDLE/$d" ]; then
|
||||
printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)"
|
||||
else
|
||||
@@ -125,7 +125,8 @@ else
|
||||
echo " python/ python-3.14.x-amd64.exe"
|
||||
echo " httpplatformhandler/ httpPlatformHandler_amd64.msi"
|
||||
echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)"
|
||||
echo " mysql/ mysql-8.0.x-winx64.msi (bundled-database option only)"
|
||||
echo " vcredist/ VC_redist.x64.exe (MySQL requires it)"
|
||||
echo " mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)"
|
||||
echo ""
|
||||
echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:"
|
||||
echo " pwsh ./refresh-bundle-lock.ps1 # review the diff"
|
||||
|
||||
@@ -1,197 +1,240 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"generated": "2026-08-03T15:45:11Z",
|
||||
"generated": "2026-08-04T23:11:22Z",
|
||||
"pythontag": "cp314",
|
||||
"platform": "win_amd64",
|
||||
"note": "Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.",
|
||||
"payloads": {
|
||||
"wheels": {
|
||||
"required": true,
|
||||
"files": {
|
||||
"jinja2-3.1.6-py3-none-any.whl": {
|
||||
"size": 134899,
|
||||
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67"
|
||||
},
|
||||
"itsdangerous-2.2.0-py3-none-any.whl": {
|
||||
"size": 16234,
|
||||
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef"
|
||||
},
|
||||
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
|
||||
"size": 187937,
|
||||
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb"
|
||||
},
|
||||
"email_validator-2.3.0-py3-none-any.whl": {
|
||||
"size": 35604,
|
||||
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4"
|
||||
},
|
||||
"urllib3-2.7.0-py3-none-any.whl": {
|
||||
"size": 131087,
|
||||
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897"
|
||||
},
|
||||
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
|
||||
"size": 22698,
|
||||
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb"
|
||||
},
|
||||
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
|
||||
"size": 25125,
|
||||
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0"
|
||||
},
|
||||
"pycparser-3.0-py3-none-any.whl": {
|
||||
"size": 48172,
|
||||
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992"
|
||||
},
|
||||
"requests-2.33.1-py3-none-any.whl": {
|
||||
"size": 64947,
|
||||
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a"
|
||||
},
|
||||
"blinker-1.9.0-py3-none-any.whl": {
|
||||
"size": 8458,
|
||||
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc"
|
||||
},
|
||||
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
|
||||
"size": 239835,
|
||||
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033"
|
||||
},
|
||||
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
|
||||
"size": 16582,
|
||||
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258"
|
||||
},
|
||||
"alembic-1.18.4-py3-none-any.whl": {
|
||||
"size": 263893,
|
||||
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a"
|
||||
},
|
||||
"pyjwt-2.12.1-py3-none-any.whl": {
|
||||
"size": 29726,
|
||||
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c"
|
||||
},
|
||||
"mako-1.3.12-py3-none-any.whl": {
|
||||
"size": 78521,
|
||||
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9"
|
||||
},
|
||||
"waitress-3.0.2-py3-none-any.whl": {
|
||||
"size": 56232,
|
||||
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e"
|
||||
},
|
||||
"colorama-0.4.6-py2.py3-none-any.whl": {
|
||||
"size": 25335,
|
||||
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
|
||||
},
|
||||
"certifi-2026.4.22-py3-none-any.whl": {
|
||||
"size": 135707,
|
||||
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a"
|
||||
},
|
||||
"cachelib-0.13.0-py3-none-any.whl": {
|
||||
"size": 20914,
|
||||
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516"
|
||||
},
|
||||
"pymysql-1.1.3-py3-none-any.whl": {
|
||||
"size": 45356,
|
||||
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289"
|
||||
},
|
||||
"werkzeug-3.1.8-py3-none-any.whl": {
|
||||
"size": 226459,
|
||||
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50"
|
||||
},
|
||||
"idna-3.13-py3-none-any.whl": {
|
||||
"size": 68629,
|
||||
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3"
|
||||
},
|
||||
"python_dotenv-1.2.2-py3-none-any.whl": {
|
||||
"size": 22101,
|
||||
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a"
|
||||
},
|
||||
"flask_marshmallow-1.5.0-py3-none-any.whl": {
|
||||
"size": 12161,
|
||||
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6"
|
||||
},
|
||||
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
|
||||
"size": 15341,
|
||||
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581"
|
||||
},
|
||||
"flask_cors-6.0.2-py3-none-any.whl": {
|
||||
"size": 13257,
|
||||
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a"
|
||||
},
|
||||
"flask_caching-2.4.0-py3-none-any.whl": {
|
||||
"size": 28727,
|
||||
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486"
|
||||
},
|
||||
"tzdata-2026.3-py2.py3-none-any.whl": {
|
||||
"size": 348168,
|
||||
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931"
|
||||
},
|
||||
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
|
||||
"size": 159634,
|
||||
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f"
|
||||
},
|
||||
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
|
||||
"size": 2144204,
|
||||
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5"
|
||||
},
|
||||
"marshmallow-4.3.0-py3-none-any.whl": {
|
||||
"size": 49148,
|
||||
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46"
|
||||
},
|
||||
"dnspython-2.8.0-py3-none-any.whl": {
|
||||
"size": 331094,
|
||||
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af"
|
||||
},
|
||||
"typing_extensions-4.15.0-py3-none-any.whl": {
|
||||
"size": 44614,
|
||||
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548"
|
||||
},
|
||||
"flask-3.1.3-py3-none-any.whl": {
|
||||
"size": 103424,
|
||||
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c"
|
||||
},
|
||||
"tabulate-0.10.0-py3-none-any.whl": {
|
||||
"size": 39814,
|
||||
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3"
|
||||
},
|
||||
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
|
||||
"size": 18197576,
|
||||
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49"
|
||||
},
|
||||
"Flask_Migrate-4.1.0-py3-none-any.whl": {
|
||||
"size": 21237,
|
||||
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d"
|
||||
},
|
||||
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
|
||||
"size": 3840395,
|
||||
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30"
|
||||
},
|
||||
"click-8.3.3-py3-none-any.whl": {
|
||||
"size": 110502,
|
||||
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613"
|
||||
}
|
||||
}
|
||||
},
|
||||
"python": {
|
||||
"required": true,
|
||||
"files": {
|
||||
"python-3.14.6-amd64.exe": {
|
||||
"size": 30774112,
|
||||
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd"
|
||||
}
|
||||
}
|
||||
},
|
||||
"urlrewrite": {
|
||||
"required": false,
|
||||
"files": {
|
||||
"rewrite_amd64_en-US.msi": {
|
||||
"size": 6078464,
|
||||
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a"
|
||||
}
|
||||
}
|
||||
},
|
||||
"httpplatformhandler": {
|
||||
"required": true,
|
||||
"files": {
|
||||
"httpPlatformHandler_amd64.msi": {
|
||||
"size": 557056,
|
||||
"sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef"
|
||||
"sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef",
|
||||
"size": 557056
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"python": {
|
||||
"files": {
|
||||
"python-3.14.6-amd64.exe": {
|
||||
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd",
|
||||
"size": 30774112
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"vcredist": {
|
||||
"files": {
|
||||
"VC_redist.x64.exe": {
|
||||
"sha256": "cc0ff0eb1dc3f5188ae6300faef32bf5beeba4bdd6e8e445a9184072096b713b",
|
||||
"size": 25635768
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"mysql": {
|
||||
"files": {
|
||||
"mysql-8.4.6-winx64.msi": {
|
||||
"sha256": "868885b2e221409f0170729bb30b2b2ce83440a1a4c735d19b3c225b51045762",
|
||||
"size": 135081984
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"wheels": {
|
||||
"files": {
|
||||
"email_validator-2.3.0-py3-none-any.whl": {
|
||||
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4",
|
||||
"size": 35604
|
||||
},
|
||||
"cachelib-0.13.0-py3-none-any.whl": {
|
||||
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516",
|
||||
"size": 20914
|
||||
},
|
||||
"flask_caching-2.4.0-py3-none-any.whl": {
|
||||
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486",
|
||||
"size": 28727
|
||||
},
|
||||
"typing_extensions-4.15.0-py3-none-any.whl": {
|
||||
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548",
|
||||
"size": 44614
|
||||
},
|
||||
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033",
|
||||
"size": 239835
|
||||
},
|
||||
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f",
|
||||
"size": 159634
|
||||
},
|
||||
"colorama-0.4.6-py2.py3-none-any.whl": {
|
||||
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6",
|
||||
"size": 25335
|
||||
},
|
||||
"Flask_Migrate-4.1.0-py3-none-any.whl": {
|
||||
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d",
|
||||
"size": 21237
|
||||
},
|
||||
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581",
|
||||
"size": 15341
|
||||
},
|
||||
"flask_marshmallow-1.5.0-py3-none-any.whl": {
|
||||
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6",
|
||||
"size": 12161
|
||||
},
|
||||
"tzdata-2026.3-py2.py3-none-any.whl": {
|
||||
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931",
|
||||
"size": 348168
|
||||
},
|
||||
"python_dotenv-1.2.2-py3-none-any.whl": {
|
||||
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a",
|
||||
"size": 22101
|
||||
},
|
||||
"dnspython-2.8.0-py3-none-any.whl": {
|
||||
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af",
|
||||
"size": 331094
|
||||
},
|
||||
"pymysql-1.1.3-py3-none-any.whl": {
|
||||
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289",
|
||||
"size": 45356
|
||||
},
|
||||
"packaging-26.3-py3-none-any.whl": {
|
||||
"sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c",
|
||||
"size": 129956
|
||||
},
|
||||
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
|
||||
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb",
|
||||
"size": 22698
|
||||
},
|
||||
"urllib3-2.7.0-py3-none-any.whl": {
|
||||
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897",
|
||||
"size": 131087
|
||||
},
|
||||
"marshmallow-4.3.0-py3-none-any.whl": {
|
||||
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46",
|
||||
"size": 49148
|
||||
},
|
||||
"waitress-3.0.2-py3-none-any.whl": {
|
||||
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e",
|
||||
"size": 56232
|
||||
},
|
||||
"idna-3.13-py3-none-any.whl": {
|
||||
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3",
|
||||
"size": 68629
|
||||
},
|
||||
"flask-3.1.3-py3-none-any.whl": {
|
||||
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c",
|
||||
"size": 103424
|
||||
},
|
||||
"werkzeug-3.1.8-py3-none-any.whl": {
|
||||
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50",
|
||||
"size": 226459
|
||||
},
|
||||
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
|
||||
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0",
|
||||
"size": 25125
|
||||
},
|
||||
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
|
||||
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258",
|
||||
"size": 16582
|
||||
},
|
||||
"alembic-1.18.4-py3-none-any.whl": {
|
||||
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a",
|
||||
"size": 263893
|
||||
},
|
||||
"tabulate-0.10.0-py3-none-any.whl": {
|
||||
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3",
|
||||
"size": 39814
|
||||
},
|
||||
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb",
|
||||
"size": 187937
|
||||
},
|
||||
"jinja2-3.1.6-py3-none-any.whl": {
|
||||
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67",
|
||||
"size": 134899
|
||||
},
|
||||
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
|
||||
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30",
|
||||
"size": 3840395
|
||||
},
|
||||
"certifi-2026.4.22-py3-none-any.whl": {
|
||||
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a",
|
||||
"size": 135707
|
||||
},
|
||||
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5",
|
||||
"size": 2144204
|
||||
},
|
||||
"blinker-1.9.0-py3-none-any.whl": {
|
||||
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc",
|
||||
"size": 8458
|
||||
},
|
||||
"pycparser-3.0-py3-none-any.whl": {
|
||||
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992",
|
||||
"size": 48172
|
||||
},
|
||||
"flask_cors-6.0.2-py3-none-any.whl": {
|
||||
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a",
|
||||
"size": 13257
|
||||
},
|
||||
"requests-2.33.1-py3-none-any.whl": {
|
||||
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a",
|
||||
"size": 64947
|
||||
},
|
||||
"pyjwt-2.12.1-py3-none-any.whl": {
|
||||
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c",
|
||||
"size": 29726
|
||||
},
|
||||
"click-8.3.3-py3-none-any.whl": {
|
||||
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613",
|
||||
"size": 110502
|
||||
},
|
||||
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49",
|
||||
"size": 18197576
|
||||
},
|
||||
"mako-1.3.12-py3-none-any.whl": {
|
||||
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9",
|
||||
"size": 78521
|
||||
},
|
||||
"itsdangerous-2.2.0-py3-none-any.whl": {
|
||||
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef",
|
||||
"size": 16234
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"urlrewrite": {
|
||||
"files": {
|
||||
"rewrite_amd64_en-US.msi": {
|
||||
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a",
|
||||
"size": 6078464
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"mysqlclient": {
|
||||
"files": {
|
||||
"libcrypto-3-x64.dll": {
|
||||
"sha256": "e84ec61fc1b07a3c899ae39c1dc3f38591cdd310c043a388e7ab40a6828aa297",
|
||||
"size": 5187216
|
||||
},
|
||||
"mysqldump.exe": {
|
||||
"sha256": "329410b7a8ae3d68d38e6f5b781610770854d3f11b975060e231dae62c20dc46",
|
||||
"size": 7142528
|
||||
},
|
||||
"mysql.exe": {
|
||||
"sha256": "bd6880253c1853fc17b1e19129d6e5d9f9c86a2dccf2c7c9e6b878a10baa9770",
|
||||
"size": 7177336
|
||||
},
|
||||
"libssl-3-x64.dll": {
|
||||
"sha256": "e24fcd6c8ec31a14c97f520b5b960ed360c55ddf0935040eff297e511697633e",
|
||||
"size": 788616
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,6 +50,7 @@ $script:BundlePayloads = @(
|
||||
@{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' }
|
||||
@{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' }
|
||||
@{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' }
|
||||
@{ Name = 'vcredist'; Required = $false; What = 'the Visual C++ runtime MySQL requires' }
|
||||
@{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' }
|
||||
)
|
||||
|
||||
@@ -75,14 +76,35 @@ function Get-FileDigest {
|
||||
}
|
||||
|
||||
function Get-PayloadFiles {
|
||||
# Every file in the directory, keyed by its path RELATIVE to that directory
|
||||
# with forward slashes, so a lock generated on Windows reads the same from
|
||||
# the Bash builder.
|
||||
<#
|
||||
Every file in the directory, keyed by its path RELATIVE to that directory
|
||||
with forward slashes, so a lock generated on Windows reads the same from
|
||||
the Bash builder.
|
||||
|
||||
The root comes from Get-Item, NOT Resolve-Path, and the prefix is checked
|
||||
before it is trimmed. Both matter, and a real install proved it:
|
||||
|
||||
Inno extracts the bundle under C:\Users\ADMINI~1\AppData\Local\Temp\... -
|
||||
an 8.3 SHORT path. Resolve-Path kept that short form while Get-ChildItem
|
||||
returned the long one (Administrator), so the root was five characters
|
||||
shorter than the prefix it was slicing off. Every relative path came out
|
||||
mangled - 'wheels/heels/flask.whl' - and the verifier reported all 96 files
|
||||
as simultaneously missing and unexpected. The payload was fine; the
|
||||
comparison was not.
|
||||
|
||||
Get-Item and Get-ChildItem go through the same provider, so their path
|
||||
forms agree. The StartsWith guard means that if they ever disagree again
|
||||
this fails loudly instead of inventing paths.
|
||||
#>
|
||||
param([string] $Dir)
|
||||
$out = @{}
|
||||
if (-not (Test-Path $Dir)) { return $out }
|
||||
$root = (Resolve-Path $Dir).Path.TrimEnd('\', '/')
|
||||
foreach ($f in (Get-ChildItem $Dir -Recurse -File)) {
|
||||
$rootItem = Get-Item -LiteralPath $Dir
|
||||
$root = $rootItem.FullName.TrimEnd('\', '/')
|
||||
foreach ($f in (Get-ChildItem -LiteralPath $rootItem.FullName -Recurse -File)) {
|
||||
if (-not $f.FullName.StartsWith($root, [System.StringComparison]::OrdinalIgnoreCase)) {
|
||||
throw ("cannot place '{0}' beneath '{1}' - path forms disagree (8.3 short name?)" -f $f.FullName, $root)
|
||||
}
|
||||
$rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/')
|
||||
$out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length }
|
||||
}
|
||||
@@ -137,7 +159,9 @@ function Test-BundleLock {
|
||||
)
|
||||
$problems = @()
|
||||
$payloads = Get-JsonProperty $Lock 'payloads'
|
||||
if ($null -eq $payloads) { return @('bundle-lock.json has no "payloads" section') }
|
||||
# ,@(...) for the same reason as the final return: a bare one-element array
|
||||
# unrolls to a string, and the caller's .Count then measures the wrong thing.
|
||||
if ($null -eq $payloads) { return ,@('bundle-lock.json has no "payloads" section') }
|
||||
|
||||
foreach ($p in $script:BundlePayloads) {
|
||||
$name = $p.Name
|
||||
@@ -184,7 +208,11 @@ function Test-BundleLock {
|
||||
}
|
||||
}
|
||||
$problems += Test-WheelhouseCoversRequirements -BundleRoot $BundleRoot
|
||||
return $problems
|
||||
# The comma keeps this an ARRAY through the return. Without it PowerShell
|
||||
# unrolls an empty result to $null and a single result to a bare string, and
|
||||
# every caller that asks for .Count then behaves differently depending on how
|
||||
# many problems there happen to be.
|
||||
return ,$problems
|
||||
}
|
||||
|
||||
function Test-WheelhouseCoversRequirements {
|
||||
|
||||
@@ -195,6 +195,8 @@ function Get-DbParts {
|
||||
|
||||
function Find-MysqlClient {
|
||||
$candidates = @(
|
||||
'C:\MySQL84\bin\mysql.exe',
|
||||
'C:\Program Files\MySQL\MySQL Server 8.4\bin\mysql.exe',
|
||||
'C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe',
|
||||
'C:\mysql56\bin\mysql.exe'
|
||||
) + @(Get-ChildItem 'C:\Program Files\MySQL' -Filter mysql.exe -Recurse -EA SilentlyContinue |
|
||||
@@ -312,10 +314,38 @@ function Stop-App {
|
||||
|
||||
function Restart-App {
|
||||
Head 'Restarting'
|
||||
# Recycle rather than stop/start: it drains existing requests instead of
|
||||
# cutting them off, and it is what a config change actually needs.
|
||||
& $AppCmd recycle apppool /apppool.name:$AppPool 2>&1 | ForEach-Object { Say " $_" }
|
||||
Start-Sleep -Seconds 2
|
||||
# A recycle CANNOT start something that is stopped - it is a no-op on a
|
||||
# stopped pool. Restarting after a failed upgrade, which is exactly when the
|
||||
# pool is stopped, therefore did nothing and then reported "did not respond"
|
||||
# in red, as though the application were broken. Start it first when needed.
|
||||
Import-Module WebAdministration -ErrorAction SilentlyContinue
|
||||
$started = $true
|
||||
try {
|
||||
if (Test-Path "IIS:\AppPools\$AppPool") {
|
||||
$started = ((Get-Item "IIS:\AppPools\$AppPool").State -eq 'Started')
|
||||
}
|
||||
} catch { }
|
||||
if (-not $started) {
|
||||
Say ' the application pool is stopped; starting it' 'Yellow'
|
||||
Start-WebAppPool -Name $AppPool -ErrorAction SilentlyContinue
|
||||
Start-Sleep -Seconds 3
|
||||
} else {
|
||||
# Recycle rather than stop/start: it drains existing requests instead of
|
||||
# cutting them off, and it is what a config change actually needs.
|
||||
& $AppCmd recycle apppool /apppool.name:$AppPool 2>&1 | ForEach-Object { Say " $_" }
|
||||
if ($LASTEXITCODE -ne 0) { Say " recycle reported exit $LASTEXITCODE" 'Yellow' }
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
|
||||
# A stopped SITE answers nothing however healthy the pool is.
|
||||
try {
|
||||
$siteNow = Get-Website -Name $SiteName -ErrorAction SilentlyContinue
|
||||
if ($siteNow -and $siteNow.State -ne 'Started') {
|
||||
Say ' the site is stopped; starting it' 'Yellow'
|
||||
Start-Website -Name $SiteName -ErrorAction SilentlyContinue
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
} catch { }
|
||||
try {
|
||||
$r = Invoke-WebRequest -Uri ((Get-Deployment).BaseUrl + '/') -UseBasicParsing -TimeoutSec 30
|
||||
Say (" back up (HTTP {0})" -f $r.StatusCode) 'Green'
|
||||
@@ -370,9 +400,28 @@ function Backup-Db {
|
||||
Head 'Database backup'
|
||||
$db = Get-DbParts
|
||||
if (-not $db) { Say ' no .env found - cannot determine the database' 'Red'; return }
|
||||
$usingDefault = -not $Dest
|
||||
if (-not $Dest) { $Dest = 'C:\ProgramData\ShopDB-Flask\backups' }
|
||||
if (-not (Test-Path $Dest)) { New-Item -ItemType Directory -Path $Dest -Force | Out-Null }
|
||||
|
||||
# A dump holds every row, including the users table and its password hashes.
|
||||
# A directory created under ProgramData INHERITS Users:RX, so those hashes
|
||||
# were readable by every authenticated user on the server whenever this
|
||||
# command created the directory rather than the installer.
|
||||
#
|
||||
# Re-applied on every backup, not only on creation, because this may be
|
||||
# repairing a directory made by an earlier version.
|
||||
#
|
||||
# Only for the default location. A path the operator named is theirs, and
|
||||
# silently rewriting its ACL is not this command's business - say so instead.
|
||||
if ($usingDefault) {
|
||||
& icacls.exe $Dest '/inheritance:r' `
|
||||
'/grant' 'BUILTIN\Administrators:(OI)(CI)(F)' `
|
||||
'/grant' 'NT AUTHORITY\SYSTEM:(OI)(CI)(F)' 2>&1 | Out-Null
|
||||
} else {
|
||||
Say ' note: this dump contains password hashes - check who can read that directory' 'Yellow'
|
||||
}
|
||||
|
||||
$mysql = Find-MysqlClient
|
||||
if (-not $mysql) { Say ' mysql client not found' 'Red'; return }
|
||||
$dump = Join-Path (Split-Path $mysql -Parent) 'mysqldump.exe'
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -221,8 +221,12 @@ Invoke-Check 'Network' 'Site port' {
|
||||
Add-Result 'Network' 'Site port' 'INFO' `
|
||||
"TCP $SitePort is used by the existing $SiteName site - this will be upgraded in place"
|
||||
} else {
|
||||
Add-Result 'Network' 'Site port' 'FAIL' "TCP $SitePort is in use$owner" `
|
||||
"Choose a different port with -SitePort, or stop the listener."
|
||||
# WARN, not FAIL. This check runs before the operator has reached the
|
||||
# Address page, so it is testing the DEFAULT port, not necessarily
|
||||
# the one they intend to use. Blocking here would refuse an install
|
||||
# over a conflict the very next page lets them resolve.
|
||||
Add-Result 'Network' 'Site port' 'WARN' "TCP $SitePort is in use$owner" `
|
||||
"Pick a different port on the Address page later in this wizard, or stop whatever is holding it."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -266,8 +270,13 @@ Invoke-Check 'IIS' 'HttpPlatformHandler' {
|
||||
if ($modules -match 'httpPlatformHandler') {
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'PASS' 'installed'
|
||||
} else {
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'FAIL' 'not installed' `
|
||||
'Install httpPlatformHandler_amd64.msi from the bundle. IIS cannot launch waitress without it.'
|
||||
# NOT a blocker: the MSI is in the bundle and stage 4 installs it. This
|
||||
# was a FAIL, which - once the preflight page started blocking on any
|
||||
# failure - stopped the wizard dead over something the installer was
|
||||
# about to do by itself, with no way forward but to go and install it by
|
||||
# hand. Nothing the installer SUPPLIES may be a blocker.
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'INFO' 'not installed yet' `
|
||||
'The installer installs it from the bundle. No action needed.'
|
||||
}
|
||||
}
|
||||
|
||||
@@ -350,7 +359,7 @@ Invoke-Check 'MySQL' 'Service' {
|
||||
$svcs = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^MySQL' -or $_.DisplayName -match 'MySQL' })
|
||||
if ($svcs.Count -eq 0) {
|
||||
Add-Result 'MySQL' 'Service' 'INFO' 'no MySQL service found' `
|
||||
'Bundled MySQL 8.0 is the appropriate choice on this box.'
|
||||
'Bundled MySQL 8.4 LTS is the appropriate choice on this box.'
|
||||
return
|
||||
}
|
||||
foreach ($s in $svcs) {
|
||||
@@ -378,7 +387,7 @@ Invoke-Check 'MySQL' 'Backup client' {
|
||||
$found = ''
|
||||
foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'),
|
||||
(Join-Path $AppRoot 'mysqlclient'),
|
||||
'C:\Program Files\MySQL', 'C:\mysql56\bin',
|
||||
'C:\MySQL84', 'C:\Program Files\MySQL', 'C:\mysql56\bin',
|
||||
'C:\Program Files (x86)\MySQL')) {
|
||||
if (-not (Test-Path $root)) { continue }
|
||||
$hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue |
|
||||
@@ -389,7 +398,7 @@ Invoke-Check 'MySQL' 'Backup client' {
|
||||
Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)"
|
||||
} else {
|
||||
Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' `
|
||||
'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. Add mysqlclient\ to the installer bundle, or install the MySQL client on this server.'
|
||||
'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. It is not on this server yet - the installer stages its own copy, so this normally resolves itself during installation.'
|
||||
}
|
||||
}
|
||||
|
||||
@@ -430,7 +439,13 @@ Invoke-Check 'MySQL' 'Version and config' {
|
||||
Add-Result 'MySQL' '5.6 index flags' 'WARN' 'all three present in my.ini' `
|
||||
'Present in the file only. They take effect after a MySQL RESTART, which interrupts the classic ASP app. Confirm with SHOW VARIABLES or flask db-utils preflight.'
|
||||
} else {
|
||||
Add-Result 'MySQL' '5.6 index flags' 'FAIL' ("missing: " + ($missing -join ', ')) `
|
||||
# WARN, not FAIL. This inspects the LOCAL MySQL, which may not be the
|
||||
# database the operator is about to install against - a bundled 8.4,
|
||||
# or a remote server. Blocking the wizard here refused an install
|
||||
# over a server that had nothing to do with it. Stage 3 runs
|
||||
# 'flask db-utils preflight' against the database actually chosen,
|
||||
# which is the check that can genuinely block.
|
||||
Add-Result 'MySQL' '5.6 index flags' 'WARN' ("missing: " + ($missing -join ', ')) `
|
||||
"Add to [mysqld] in $ini and restart MySQL, or 'flask db upgrade' fails with error 1071. NOTE: restarting interrupts the classic ASP app."
|
||||
}
|
||||
}
|
||||
|
||||
130
deploy/windows/installer/tests/test-install-state.ps1
Normal file
130
deploy/windows/installer/tests/test-install-state.ps1
Normal file
@@ -0,0 +1,130 @@
|
||||
# Tests the installer's durable install record.
|
||||
#
|
||||
# The record answers one question: "is this a re-run of MY install?". Getting it
|
||||
# wrong is destructive in one direction - answering "first provisioning" for a
|
||||
# server holding real data lets prune-schema --force loose on it - and a dead
|
||||
# end in the other, which is what it was built to fix.
|
||||
#
|
||||
# Run directly: pwsh -File test-install-state.ps1
|
||||
# pytest runs it through tests/test_installer_state.py wherever pwsh exists.
|
||||
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$Here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$Installer = Join-Path (Split-Path -Parent $Here) 'shopdb-install.ps1'
|
||||
if (-not (Test-Path $Installer)) { throw "installer not found at $Installer" }
|
||||
|
||||
$Sandbox = Join-Path ([System.IO.Path]::GetTempPath()) ('shopdb-state-' + [System.Guid]::NewGuid().ToString('N'))
|
||||
|
||||
# Load ONLY the state block, so the rest of the installer does not run. Bounded
|
||||
# by the same markers the installer uses, and it fails loudly if either moves
|
||||
# rather than silently testing nothing.
|
||||
$lines = Get-Content $Installer
|
||||
$start = ($lines | Select-String -Pattern '^\$script:StateFile ' | Select-Object -First 1)
|
||||
$end = ($lines | Select-String -Pattern '^function Invoke-Native' | Select-Object -First 1)
|
||||
if (-not $start -or -not $end) { throw 'could not locate the install-record block in shopdb-install.ps1' }
|
||||
$block = $lines[($start.LineNumber - 1)..($end.LineNumber - 2)] -join "`n"
|
||||
if ($block -notmatch 'function Test-FirstProvisioning') { throw 'extracted block does not contain the state functions' }
|
||||
|
||||
# Stand-ins for the surrounding script.
|
||||
$script:Created = New-Object System.Collections.ArrayList
|
||||
function Write-Log { param($m, $l = 'INFO') }
|
||||
function Protect-File { param([string] $Path, [switch] $Directory) }
|
||||
function Test-SamePath {
|
||||
param([string] $A, [string] $B)
|
||||
if (-not $A) { return -not $B }
|
||||
return ($A.TrimEnd('\', '/').ToLowerInvariant() -eq $B.TrimEnd('\', '/').ToLowerInvariant())
|
||||
}
|
||||
if (-not $env:ProgramData) { $env:ProgramData = [System.IO.Path]::GetTempPath() }
|
||||
|
||||
Invoke-Expression $block
|
||||
|
||||
$script:Failures = 0
|
||||
function Check {
|
||||
param([string] $Name, $Expected, $Actual)
|
||||
if ($Expected -eq $Actual) {
|
||||
Write-Host (" PASS " + $Name)
|
||||
} else {
|
||||
Write-Host (" FAIL {0}: expected [{1}], got [{2}]" -f $Name, $Expected, $Actual)
|
||||
$script:Failures++
|
||||
}
|
||||
}
|
||||
|
||||
function Reset-Case {
|
||||
param([switch] $Stamped)
|
||||
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
|
||||
New-Item -ItemType Directory -Path $Sandbox -Force | Out-Null
|
||||
$script:AppRoot = $Sandbox
|
||||
$script:StateFile = Join-Path $Sandbox 'install-state.json'
|
||||
$script:InstallState = $null
|
||||
if ($Stamped) { Set-Content (Join-Path $Sandbox '.installed-version') '0.7.0' }
|
||||
}
|
||||
|
||||
Write-Host 'brand new server'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
Check 'first provisioning' $true (Test-FirstProvisioning)
|
||||
Check 'record written' $true (Test-Path $script:StateFile)
|
||||
|
||||
Write-Host 'retry of a failed first install'
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'still first provisioning' $true (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'genuine upgrade after a completed install'
|
||||
Complete-InstallState
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'no longer first provisioning' $false (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'install predating the record - must take the safe side'
|
||||
Reset-Case -Stamped
|
||||
Initialize-InstallState
|
||||
Check 'treated as established' $false (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'record naming a different directory'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
$other = Get-Content $script:StateFile -Raw | ConvertFrom-Json
|
||||
$other.approot = 'C:\somewhere-else'
|
||||
($other | ConvertTo-Json -Depth 6) | Set-Content $script:StateFile
|
||||
$script:InstallState = $null
|
||||
Check 'foreign record rejected' $null (Get-InstallState)
|
||||
|
||||
Write-Host 'corrupt record'
|
||||
Reset-Case
|
||||
Set-Content $script:StateFile '{ this is not json'
|
||||
$script:InstallState = $null
|
||||
Check 'corrupt record rejected' $null (Get-InstallState)
|
||||
Reset-Case
|
||||
Set-Content $script:StateFile '{ this is not json'
|
||||
Initialize-InstallState
|
||||
Check 'recovers and rewrites' $true ($null -ne $script:InstallState)
|
||||
|
||||
Write-Host 'created items survive a crash'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
Track 'site' 'shopdb-flask'
|
||||
Track 'apppool' 'shopdbflask'
|
||||
Track 'site' 'shopdb-flask'
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'site remembered' 'shopdb-flask' ((Get-CreatedItems 'site') -join ',')
|
||||
Check 'apppool remembered' 'shopdbflask' ((Get-CreatedItems 'apppool') -join ',')
|
||||
# A zero-length array returned from a function unrolls to $null, and $null.Count
|
||||
# is fatal under StrictMode. This is the check that catches losing the comma.
|
||||
Check 'unknown kind is an empty array' 0 ((Get-CreatedItems 'firewall')).Count
|
||||
|
||||
Write-Host 'no record at all'
|
||||
$script:InstallState = $null
|
||||
Check 'empty list, not null' 0 ((Get-CreatedItems 'site')).Count
|
||||
Check 'not first provisioning' $false (Test-FirstProvisioning)
|
||||
|
||||
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
|
||||
if ($script:Failures -gt 0) {
|
||||
Write-Host ("{0} check(s) failed" -f $script:Failures)
|
||||
exit 1
|
||||
}
|
||||
Write-Host 'all checks passed'
|
||||
exit 0
|
||||
@@ -32,6 +32,7 @@ PAYLOADS = [
|
||||
('httpplatformhandler', True, 'the IIS module that launches waitress'),
|
||||
('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'),
|
||||
('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'),
|
||||
('vcredist', False, 'the Visual C++ runtime MySQL requires'),
|
||||
('mysql', False, 'MySQL, for the bundled-database option'),
|
||||
]
|
||||
|
||||
|
||||
422
deploy/windows/shopdb-diagnose.py
Normal file
422
deploy/windows/shopdb-diagnose.py
Normal file
@@ -0,0 +1,422 @@
|
||||
"""Collect everything needed to diagnose a ShopDB-Flask stage 5 failure.
|
||||
|
||||
Stage 5 is the smoke test: the installer asks IIS for the site and expects 200.
|
||||
When it does not get one, the cause is always in one of four places, and this
|
||||
script reads all four in one pass so the answer arrives in a single round trip:
|
||||
|
||||
1. What IIS actually answers, and with which status code and error page.
|
||||
2. Whether the config sections httpPlatformHandler needs are unlocked.
|
||||
3. Whether the app-pool identity can read the app and run its venv.
|
||||
4. Whether the app itself imports and starts.
|
||||
|
||||
Run it ON THE SERVER, as Administrator:
|
||||
|
||||
C:\\Python314\\python.exe shopdb-diagnose.py
|
||||
|
||||
It writes shopdb-diagnose-<timestamp>.txt next to itself and prints the path.
|
||||
Send that file back.
|
||||
|
||||
SECRETS: the report never contains them. Values from .env (database password,
|
||||
SECRET_KEY, JWT_SECRET_KEY) are read first, then scrubbed out of every section
|
||||
of the report before it is written, including command output and tracebacks
|
||||
that might quote them.
|
||||
|
||||
Standard library only, so it runs on the bundled runtime or any system Python.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
|
||||
APP_ROOT = os.environ.get('SHOPDB_APPROOT', r'C:\shopdb-flask')
|
||||
ALIAS = 'shopdb'
|
||||
TIMEOUT = 25
|
||||
|
||||
# Filled from .env, then scrubbed from the whole report.
|
||||
SECRETS = []
|
||||
|
||||
WINDIR = os.environ.get('WINDIR', r'C:\Windows')
|
||||
# Sysnative gives a 32-bit process the real 64-bit System32. Harmless on 64-bit.
|
||||
APPCMD_CANDIDATES = [
|
||||
os.path.join(WINDIR, 'Sysnative', 'inetsrv', 'appcmd.exe'),
|
||||
os.path.join(WINDIR, 'System32', 'inetsrv', 'appcmd.exe'),
|
||||
]
|
||||
|
||||
|
||||
def find_appcmd():
|
||||
for path in APPCMD_CANDIDATES:
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
class Report(object):
|
||||
def __init__(self):
|
||||
self.chunks = []
|
||||
|
||||
def head(self, title):
|
||||
self.chunks.append('\n' + '=' * 72 + '\n' + title + '\n' + '=' * 72)
|
||||
|
||||
def line(self, text=''):
|
||||
self.chunks.append(str(text))
|
||||
|
||||
def block(self, title, body):
|
||||
self.chunks.append('\n--- %s ---' % title)
|
||||
if body is None or str(body).strip() == '':
|
||||
self.chunks.append('(no output)')
|
||||
else:
|
||||
self.chunks.append(str(body).rstrip())
|
||||
|
||||
def text(self):
|
||||
return '\n'.join(self.chunks) + '\n'
|
||||
|
||||
|
||||
def run(cmd, timeout=60):
|
||||
"""Run a command, return combined output. Never raises."""
|
||||
try:
|
||||
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, shell=False)
|
||||
out, _ = proc.communicate(timeout=timeout)
|
||||
text = out.decode('utf-8', 'replace') if out else ''
|
||||
return '[exit %s]\n%s' % (proc.returncode, text)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
proc.kill()
|
||||
except Exception:
|
||||
pass
|
||||
return '[TIMED OUT after %ss]' % timeout
|
||||
except Exception as exc:
|
||||
return '[could not run: %s]' % exc
|
||||
|
||||
|
||||
def powershell(script, timeout=90):
|
||||
exe = os.path.join(WINDIR, 'Sysnative', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
|
||||
if not os.path.isfile(exe):
|
||||
exe = 'powershell.exe'
|
||||
return run([exe, '-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', script],
|
||||
timeout=timeout)
|
||||
|
||||
|
||||
def load_secrets():
|
||||
"""Read .env so its secret VALUES can be scrubbed from the report."""
|
||||
env_path = os.path.join(APP_ROOT, '.env')
|
||||
found = {}
|
||||
if not os.path.isfile(env_path):
|
||||
return found, None
|
||||
try:
|
||||
with open(env_path, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
raw = handle.read()
|
||||
except Exception as exc:
|
||||
return found, '[could not read .env: %s]' % exc
|
||||
|
||||
for line in raw.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith('#') or '=' not in line:
|
||||
continue
|
||||
key, value = line.split('=', 1)
|
||||
key, value = key.strip(), value.strip().strip('"').strip("'")
|
||||
found[key] = value
|
||||
if not value:
|
||||
continue
|
||||
upper = key.upper()
|
||||
if 'SECRET' in upper or 'PASSWORD' in upper or 'TOKEN' in upper or 'KEY' in upper:
|
||||
SECRETS.append(value)
|
||||
if upper == 'DATABASE_URL':
|
||||
# mysql+pymysql://user:PASSWORD@host/db -- the password only.
|
||||
match = re.match(r'^[^:]+://([^:@/]+):([^@]+)@', value)
|
||||
if match:
|
||||
SECRETS.append(match.group(2))
|
||||
return found, raw
|
||||
|
||||
|
||||
def scrub(text):
|
||||
"""Remove every known secret value from the report."""
|
||||
for secret in SECRETS:
|
||||
if secret and len(secret) >= 4:
|
||||
text = text.replace(secret, '<REDACTED>')
|
||||
# Catch a password inside any URL that did not come from .env.
|
||||
text = re.sub(r'(://[^:@/\s]+:)[^@\s]+(@)', r'\1<REDACTED>\2', text)
|
||||
return text
|
||||
|
||||
|
||||
def http_probe(url):
|
||||
"""Fetch a URL, returning status, headers and body even for an error page."""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
started = time.time()
|
||||
try:
|
||||
request = urllib.request.Request(url, headers={'User-Agent': 'shopdb-diagnose'})
|
||||
with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
|
||||
body = response.read(4000).decode('utf-8', 'replace')
|
||||
return {'status': response.status, 'reason': response.reason,
|
||||
'headers': dict(response.headers), 'body': body,
|
||||
'seconds': time.time() - started}
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = ''
|
||||
try:
|
||||
body = exc.read(4000).decode('utf-8', 'replace')
|
||||
except Exception:
|
||||
pass
|
||||
return {'status': exc.code, 'reason': exc.reason,
|
||||
'headers': dict(exc.headers or {}), 'body': body,
|
||||
'seconds': time.time() - started}
|
||||
except Exception as exc:
|
||||
return {'status': None, 'reason': '%s: %s' % (type(exc).__name__, exc),
|
||||
'headers': {}, 'body': '', 'seconds': time.time() - started}
|
||||
|
||||
|
||||
def summarise_iis_error(body):
|
||||
"""Pull the meaningful bits out of an IIS error page."""
|
||||
if not body:
|
||||
return None
|
||||
import html
|
||||
|
||||
# style and script blocks first: their contents survive plain tag stripping
|
||||
# and drag CSS into the summary.
|
||||
flat = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', body)
|
||||
flat = re.sub(r'(?s)<!--.*?-->', ' ', flat)
|
||||
flat = re.sub(r'(?s)<[^>]*>', ' ', flat)
|
||||
# Any unterminated tag left by the 4000-byte body truncation.
|
||||
flat = re.sub(r'(?s)<[^>]*$', ' ', flat)
|
||||
flat = html.unescape(flat)
|
||||
flat = re.sub(r'\s+', ' ', flat).strip()
|
||||
|
||||
hints = []
|
||||
for pattern in (r'\b\d{3}\.\d+\b', r'0x[0-9a-fA-F]{8}',
|
||||
r'Error Code[^.]{0,80}', r'Config (?:Error|File)[^.]{0,120}',
|
||||
r'Requested URL[^.]{0,120}', r'Physical Path[^.]{0,120}'):
|
||||
for match in re.findall(pattern, flat):
|
||||
cleaned = re.sub(r'\s+', ' ', match).strip(' :-')
|
||||
if cleaned and cleaned not in hints:
|
||||
hints.append(cleaned)
|
||||
return {'flat': flat[:1200], 'hints': hints}
|
||||
|
||||
|
||||
def main():
|
||||
report = Report()
|
||||
stamp = datetime.now().strftime('%Y%m%d-%H%M%S')
|
||||
|
||||
env_values, env_raw = load_secrets()
|
||||
|
||||
report.line('ShopDB-Flask stage 5 diagnostic')
|
||||
report.line('generated %s' % datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
|
||||
report.line('host %s' % socket.gethostname())
|
||||
report.line('app root %s' % APP_ROOT)
|
||||
report.line('python %s' % sys.version.replace('\n', ' '))
|
||||
report.line('process is %d-bit' % (64 if sys.maxsize > 2 ** 32 else 32))
|
||||
|
||||
# ---------------------------------------------------------------- 1. HTTP
|
||||
report.head('1. WHAT IIS ANSWERS')
|
||||
report.line('This is the single most important section. The status code names')
|
||||
report.line('the fault: 500.19 = config locked, 502.3/503 = the app did not')
|
||||
report.line('start, 404 = application or handler mapping missing.')
|
||||
|
||||
hostname = socket.gethostname()
|
||||
targets = [
|
||||
'http://localhost/%s/' % ALIAS,
|
||||
'http://127.0.0.1/%s/' % ALIAS,
|
||||
'http://[::1]/%s/' % ALIAS,
|
||||
'http://%s/%s/' % (hostname, ALIAS),
|
||||
'http://localhost/',
|
||||
]
|
||||
for url in targets:
|
||||
result = http_probe(url)
|
||||
report.line('\n%s' % url)
|
||||
report.line(' status : %s %s' % (result['status'], result['reason']))
|
||||
report.line(' time : %.1fs' % result['seconds'])
|
||||
server = result['headers'].get('Server')
|
||||
if server:
|
||||
report.line(' server : %s' % server)
|
||||
summary = summarise_iis_error(result['body'])
|
||||
if summary and summary['hints']:
|
||||
report.line(' hints : %s' % ' | '.join(summary['hints'][:8]))
|
||||
if summary and summary['flat']:
|
||||
report.line(' body : %s' % summary['flat'][:600])
|
||||
|
||||
# localhost resolving to ::1 first has bitten this install before.
|
||||
report.block('name resolution for localhost', run(
|
||||
['nslookup', 'localhost'], timeout=20))
|
||||
|
||||
# ------------------------------------------------------------- 2. IIS state
|
||||
report.head('2. IIS STATE')
|
||||
appcmd = find_appcmd()
|
||||
if not appcmd:
|
||||
report.line('appcmd.exe NOT FOUND - is the IIS role installed?')
|
||||
else:
|
||||
report.line('appcmd: %s' % appcmd)
|
||||
report.block('sites', run([appcmd, 'list', 'sites']))
|
||||
report.block('applications', run([appcmd, 'list', 'apps']))
|
||||
report.block('app pools', run([appcmd, 'list', 'apppools']))
|
||||
report.block('worker processes (empty means nothing is running)',
|
||||
run([appcmd, 'list', 'wp']))
|
||||
report.block('modules: httpPlatformHandler present?',
|
||||
run([appcmd, 'list', 'modules']))
|
||||
# overrideMode tells us whether the unlock actually took effect.
|
||||
#
|
||||
# allowedServerVariables is in this list because it caused a 500.52 that
|
||||
# the first two sections could not explain: it is Deny by default, so an
|
||||
# <allowedServerVariables> block in the app's web.config is rejected
|
||||
# before httpPlatformHandler runs. Checking only the sections we unlock
|
||||
# would have missed the one we do not.
|
||||
for section in ('system.webServer/handlers',
|
||||
'system.webServer/httpPlatform',
|
||||
'system.webServer/rewrite/allowedServerVariables',
|
||||
'system.webServer/rewrite/rules'):
|
||||
report.block('lock state of %s' % section,
|
||||
run([appcmd, 'list', 'config', '/section:%s' % section,
|
||||
'/text:*']))
|
||||
|
||||
report.block('W3SVC / WAS services', powershell(
|
||||
"Get-Service W3SVC,WAS | Format-Table Name,Status,StartType -AutoSize | Out-String"))
|
||||
report.block('listeners on port 80', powershell(
|
||||
"Get-NetTCPConnection -LocalPort 80 -State Listen -EA SilentlyContinue | "
|
||||
"Format-Table LocalAddress,LocalPort,OwningProcess -AutoSize | Out-String"))
|
||||
report.block('app pool detail', powershell(
|
||||
"Import-Module WebAdministration -EA SilentlyContinue; "
|
||||
"Get-Item IIS:\\AppPools\\shopdbflask -EA SilentlyContinue | "
|
||||
"Select-Object name,state,managedRuntimeVersion,enable32BitAppOnWin64,"
|
||||
"@{n='identity';e={$_.processModel.identityType}} | Format-List | Out-String"))
|
||||
|
||||
# --------------------------------------------------------- 3. app + config
|
||||
report.head('3. APPLICATION AND CONFIG')
|
||||
|
||||
web_config = os.path.join(APP_ROOT, 'web.config')
|
||||
if os.path.isfile(web_config):
|
||||
try:
|
||||
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
report.block('web.config', handle.read())
|
||||
except Exception as exc:
|
||||
report.block('web.config', '[could not read: %s]' % exc)
|
||||
else:
|
||||
report.block('web.config', 'MISSING at %s' % web_config)
|
||||
|
||||
if env_raw is None:
|
||||
report.block('.env', 'MISSING at %s' % os.path.join(APP_ROOT, '.env'))
|
||||
else:
|
||||
# Keys and non-secret values only. Secret values are scrubbed anyway.
|
||||
lines = []
|
||||
for key in sorted(env_values):
|
||||
upper = key.upper()
|
||||
secretish = ('SECRET' in upper or 'PASSWORD' in upper
|
||||
or 'TOKEN' in upper or 'KEY' in upper
|
||||
or upper == 'DATABASE_URL')
|
||||
if secretish:
|
||||
lines.append('%s = <set, %d chars>' % (key, len(env_values[key])))
|
||||
else:
|
||||
lines.append('%s = %s' % (key, env_values[key]))
|
||||
report.block('.env (secret values withheld)', '\n'.join(lines))
|
||||
|
||||
venv_python = os.path.join(APP_ROOT, 'venv', 'Scripts', 'python.exe')
|
||||
report.line('\nvenv python exists: %s' % os.path.isfile(venv_python))
|
||||
if os.path.isfile(venv_python):
|
||||
# The exact failure stage 3 used to hit. Proves the app imports.
|
||||
report.block('venv: import shopdb', run(
|
||||
[venv_python, '-c',
|
||||
'import shopdb; print("import OK"); '
|
||||
'app = shopdb.create_app(); print("create_app OK")'], timeout=120))
|
||||
report.block('venv: waitress present', run(
|
||||
[venv_python, '-c', 'import waitress; print(waitress.__version__)'],
|
||||
timeout=60))
|
||||
|
||||
# What httpPlatformHandler is told to launch, and whether it exists.
|
||||
if os.path.isfile(web_config):
|
||||
try:
|
||||
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
raw = handle.read()
|
||||
match = re.search(r'processPath\s*=\s*"([^"]+)"', raw)
|
||||
args = re.search(r'arguments\s*=\s*"([^"]*)"', raw)
|
||||
if match:
|
||||
path = os.path.expandvars(match.group(1))
|
||||
report.line('\nhttpPlatform processPath : %s' % match.group(1))
|
||||
report.line(' resolved : %s' % path)
|
||||
report.line(' exists : %s' % os.path.isfile(path))
|
||||
if args:
|
||||
report.line('httpPlatform arguments : %s' % args.group(1))
|
||||
except Exception as exc:
|
||||
report.line('[could not parse web.config: %s]' % exc)
|
||||
|
||||
# ------------------------------------------------------------- 4. app logs
|
||||
report.head('4. APPLICATION LOGS')
|
||||
log_dir = os.path.join(APP_ROOT, 'logs')
|
||||
if not os.path.isdir(log_dir):
|
||||
report.line('MISSING: %s' % log_dir)
|
||||
else:
|
||||
entries = []
|
||||
for name in sorted(os.listdir(log_dir)):
|
||||
full = os.path.join(log_dir, name)
|
||||
try:
|
||||
entries.append((os.path.getmtime(full), full, name,
|
||||
os.path.getsize(full)))
|
||||
except OSError:
|
||||
pass
|
||||
if not entries:
|
||||
report.line('%s is EMPTY.' % log_dir)
|
||||
report.line('No stdout log at all means httpPlatformHandler never')
|
||||
report.line('launched python - look at the pool identity and ACLs.')
|
||||
for _, full, name, size in sorted(entries, reverse=True)[:5]:
|
||||
if size == 0:
|
||||
report.block('%s (0 bytes)' % name,
|
||||
'EMPTY - python was launched but wrote nothing.')
|
||||
continue
|
||||
try:
|
||||
with open(full, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
tail = handle.readlines()[-60:]
|
||||
report.block('%s (%d bytes, last 60 lines)' % (name, size),
|
||||
''.join(tail))
|
||||
except Exception as exc:
|
||||
report.block(name, '[could not read: %s]' % exc)
|
||||
|
||||
# ---------------------------------------------------------------- 5. ACLs
|
||||
report.head('5. PERMISSIONS')
|
||||
report.line('The pool runs as "IIS AppPool\\shopdbflask". It needs RX on the')
|
||||
report.line('tree, Modify on logs and instance, and Read on .env.')
|
||||
for target in (APP_ROOT, log_dir, os.path.join(APP_ROOT, '.env'),
|
||||
os.path.join(APP_ROOT, 'venv', 'Scripts')):
|
||||
if os.path.exists(target):
|
||||
report.block('icacls %s' % target, run(['icacls', target], timeout=40))
|
||||
|
||||
# ------------------------------------------------------------ 6. event log
|
||||
report.head('6. EVENT LOG')
|
||||
report.block('recent application errors', powershell(
|
||||
"Get-WinEvent -FilterHashtable @{LogName='Application';"
|
||||
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
|
||||
"Where-Object { $_.ProviderName -match 'HttpPlatform|IIS|W3SVC|WAS|\\.NET' "
|
||||
"-or $_.LevelDisplayName -eq 'Error' } | Select-Object -First 25 "
|
||||
"TimeCreated,ProviderName,LevelDisplayName,Message | Format-List | Out-String",
|
||||
timeout=180))
|
||||
report.block('system log: WAS / W3SVC', powershell(
|
||||
"Get-WinEvent -FilterHashtable @{LogName='System';"
|
||||
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
|
||||
"Where-Object { $_.ProviderName -match 'WAS|W3SVC|HTTP' } | "
|
||||
"Select-Object -First 20 TimeCreated,ProviderName,LevelDisplayName,Message | "
|
||||
"Format-List | Out-String", timeout=180))
|
||||
|
||||
# ------------------------------------------------------------------ write
|
||||
body = scrub(report.text())
|
||||
out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
'shopdb-diagnose-%s.txt' % stamp)
|
||||
try:
|
||||
with open(out_path, 'w', encoding='utf-8') as handle:
|
||||
handle.write(body)
|
||||
except Exception:
|
||||
out_path = os.path.join(os.environ.get('TEMP', r'C:\Windows\Temp'),
|
||||
'shopdb-diagnose-%s.txt' % stamp)
|
||||
with open(out_path, 'w', encoding='utf-8') as handle:
|
||||
handle.write(body)
|
||||
|
||||
print('')
|
||||
print('Report written to:')
|
||||
print(' %s' % out_path)
|
||||
print('')
|
||||
print('%d secret value(s) were scrubbed from it.' % len(SECRETS))
|
||||
print('Send that file back.')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -67,15 +67,28 @@
|
||||
It ships DISABLED because it needs the URL Rewrite module; enabled without
|
||||
it, IIS returns HTTP 500.19 ("configuration section not well-formed").
|
||||
|
||||
There is deliberately NO <allowedServerVariables> block below. Setting a
|
||||
server variable requires that variable to be allowed, but the section
|
||||
system.webServer/rewrite/allowedServerVariables ships with
|
||||
overrideModeDefault="Deny", so declaring it in an application's own
|
||||
web.config is refused outright: IIS answered 500.52 with error 0x80070021,
|
||||
"this configuration section cannot be used at this path", BEFORE it ever
|
||||
reached httpPlatformHandler - so python was never launched and the stdout
|
||||
log stayed empty, which looks like an application fault and is not one.
|
||||
|
||||
The installer instead allows the single variable at server level, which
|
||||
grants exactly HTTP_X_FORWARDED_FOR rather than unlocking the section and
|
||||
letting every site on the machine declare arbitrary server variables.
|
||||
|
||||
The installer handles both: -ClientIpSource direct installs URL Rewrite
|
||||
from the bundle and enables this; -ClientIpSource proxy leaves it alone.
|
||||
By hand: install URL Rewrite, then delete the two marker lines below.
|
||||
from the bundle, allows the variable, and enables this; -ClientIpSource
|
||||
proxy leaves it alone. By hand: install URL Rewrite, run
|
||||
appcmd set config /section:system.webServer/rewrite/allowedServerVariables ^
|
||||
/+"[name='HTTP_X_FORWARDED_FOR']" /commit:apphost
|
||||
then delete the two marker lines below.
|
||||
-->
|
||||
<!-- SHOPDB-CLIENTIP-BEGIN
|
||||
<rewrite>
|
||||
<allowedServerVariables>
|
||||
<add name="HTTP_X_FORWARDED_FOR" />
|
||||
</allowedServerVariables>
|
||||
<rules>
|
||||
<rule name="Set X-Forwarded-For" stopProcessing="false">
|
||||
<match url=".*" />
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "shopdb-frontend",
|
||||
"version": "0.7.0",
|
||||
"version": "0.8.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
||||
28
pytest.ini
Normal file
28
pytest.ini
Normal file
@@ -0,0 +1,28 @@
|
||||
[pytest]
|
||||
# Tests live in tests/. Saying so is not cosmetic.
|
||||
#
|
||||
# deploy/windows/installer/bundle/ is BUILD OUTPUT: build-installer.sh stages a
|
||||
# complete copy of the application there, plugins included. Without this,
|
||||
# collection walked into it and imported every plugin a second time under a
|
||||
# second path, so SQLAlchemy raised "Table 'printeditems' is already defined for
|
||||
# this MetaData instance" and the whole suite failed to collect - on a tree with
|
||||
# nothing wrong in it, purely because somebody had built an installer first.
|
||||
#
|
||||
# It only started biting when the bundle grew from four plugins to thirteen,
|
||||
# which is the sort of delay that makes this look like a code fault rather than
|
||||
# a stale artefact.
|
||||
testpaths = tests
|
||||
|
||||
# Belt and braces: even an explicit path or an IDE run must not descend into
|
||||
# build output or a staged copy of the tree.
|
||||
norecursedirs =
|
||||
.git
|
||||
venv
|
||||
.venv
|
||||
node_modules
|
||||
bundle
|
||||
dist
|
||||
build
|
||||
frontend/dist
|
||||
frontend/dist-subpath
|
||||
*.egg-info
|
||||
@@ -18,6 +18,12 @@ click>=8.1
|
||||
python-dotenv>=1.0
|
||||
tabulate>=0.9
|
||||
|
||||
# Plugin contract version checking (shopdb/plugins/loader.py).
|
||||
# Declared explicitly because it reaches dev and CI only as a pytest
|
||||
# dependency: every test passes while a production venv, which has no pytest,
|
||||
# cannot import shopdb at all.
|
||||
packaging>=23.0
|
||||
|
||||
# HTTP/API clients
|
||||
requests>=2.31
|
||||
|
||||
|
||||
@@ -15,31 +15,29 @@
|
||||
#
|
||||
# Regenerate with the command above. Do NOT hand-edit: waitress and tzdata were
|
||||
# hand-added here once and vanished on the next compile.
|
||||
#
|
||||
# Only DIRECT requirements belong in requirements.in. A runtime import that is
|
||||
# satisfied in dev only because some test dependency happens to pull it in will
|
||||
# pass every test and then fail on a production venv: `packaging` reached this
|
||||
# project through pytest, and the Windows install could not import shopdb at
|
||||
# all until it was declared. tests/test_runtime_dependencies.py guards that.
|
||||
alembic==1.18.4 \
|
||||
--hash=sha256:a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a \
|
||||
--hash=sha256:cb6e1fd84b6174ab8dbb2329f86d631ba9559dd78df550b57804d607672cedbc
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask-migrate
|
||||
# via flask-migrate
|
||||
blinker==1.9.0 \
|
||||
--hash=sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf \
|
||||
--hash=sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask
|
||||
# via flask
|
||||
cachelib==0.13.0 \
|
||||
--hash=sha256:209d8996e3c57595bee274ff97116d1d73c4980b2fd9a34c7846cd07fd2e1a48 \
|
||||
--hash=sha256:8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask-caching
|
||||
# via flask-caching
|
||||
certifi==2026.4.22 \
|
||||
--hash=sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a \
|
||||
--hash=sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580
|
||||
# via
|
||||
# -r requirements.in
|
||||
# requests
|
||||
cffi==2.1.0 \
|
||||
# via requests
|
||||
cffi==2.1.0 ; platform_python_implementation != 'PyPy' \
|
||||
--hash=sha256:02cb7ff33ded4f1532476731f89ede53e2e488a8e6205515a82144246ffa7dcc \
|
||||
--hash=sha256:03e9810d18c646077e501f661b682fbf5dee4676048527ca3cffe66faa9960dd \
|
||||
--hash=sha256:0520e1f4c35f44e209cbbb421b67eec42e6a157f59444dfb6058874ff3610e5d \
|
||||
@@ -140,9 +138,7 @@ cffi==2.1.0 \
|
||||
--hash=sha256:f9912624a0c0b834b7520d7769b3644453aabc0a7e1c839da7359f050750e9bc \
|
||||
--hash=sha256:fb62edb5bb52cca65fab91a63afa7561607120d26090a7e8fda6fb9f064726da \
|
||||
--hash=sha256:ff067a8d8d880e7809e4ac88eb009bb848870115317b306666502ccad30b147f
|
||||
# via
|
||||
# -r requirements.in
|
||||
# cryptography
|
||||
# via cryptography
|
||||
charset-normalizer==3.4.7 \
|
||||
--hash=sha256:007d05ec7321d12a40227aae9e2bc6dca73f3cb21058999a1df9e193555a9dcc \
|
||||
--hash=sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c \
|
||||
@@ -273,9 +269,7 @@ charset-normalizer==3.4.7 \
|
||||
--hash=sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6 \
|
||||
--hash=sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79 \
|
||||
--hash=sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464
|
||||
# via
|
||||
# -r requirements.in
|
||||
# requests
|
||||
# via requests
|
||||
click==8.3.3 \
|
||||
--hash=sha256:398329ad4837b2ff7cbe1dd166a4c0f8900c3ca3a218de04466f38f6497f18a2 \
|
||||
--hash=sha256:a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613
|
||||
@@ -337,9 +331,7 @@ cryptography==50.0.0 \
|
||||
dnspython==2.8.0 \
|
||||
--hash=sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af \
|
||||
--hash=sha256:181d3c6996452cb1189c4046c61599b84a5a86e099562ffde77d26984ff26d0f
|
||||
# via
|
||||
# -r requirements.in
|
||||
# email-validator
|
||||
# via email-validator
|
||||
email-validator==2.3.0 \
|
||||
--hash=sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4 \
|
||||
--hash=sha256:9fc05c37f2f6cf439ff414f8fc46d917929974a82244c20eb10231ba60c54426
|
||||
@@ -381,7 +373,7 @@ flask-sqlalchemy==3.1.1 \
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask-migrate
|
||||
greenlet==3.5.0 \
|
||||
greenlet==3.5.0 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64' \
|
||||
--hash=sha256:0ecec963079cd58cbd14723582384f11f166fd58883c15dcbfb342e0bc9b5846 \
|
||||
--hash=sha256:0ed006e4b86c59de7467eb2601cd1b77b5a7d657d1ee55e30fe30d76451edba4 \
|
||||
--hash=sha256:0ff251e9a0279522e62f6176412869395a64ddf2b5c5f782ff609a8216a4e662 \
|
||||
@@ -441,34 +433,25 @@ greenlet==3.5.0 \
|
||||
--hash=sha256:f35807464c4c58c55f0d31dfa83c541a5615d825c2fe3d2b95360cf7c4e3c0a8 \
|
||||
--hash=sha256:f8c30c2225f40dd76c50790f0eb3b5c7c18431efb299e2782083e1981feed243 \
|
||||
--hash=sha256:fa94cb2288681e3a11645958f1871d48ee9211bd2f66628fdace505927d6e564
|
||||
# via
|
||||
# -r requirements.in
|
||||
# sqlalchemy
|
||||
# via sqlalchemy
|
||||
idna==3.13 \
|
||||
--hash=sha256:585ea8fe5d69b9181ec1afba340451fba6ba764af97026f92a91d4eef164a242 \
|
||||
--hash=sha256:892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3
|
||||
# via
|
||||
# -r requirements.in
|
||||
# email-validator
|
||||
# requests
|
||||
itsdangerous==2.2.0 \
|
||||
--hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef \
|
||||
--hash=sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask
|
||||
# via flask
|
||||
jinja2==3.1.6 \
|
||||
--hash=sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d \
|
||||
--hash=sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask
|
||||
# via flask
|
||||
mako==1.3.12 \
|
||||
--hash=sha256:8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9 \
|
||||
--hash=sha256:9f778e93289bd410bb35daadeb4fc66d95a746f0b75777b942088b7fd7af550a
|
||||
# via
|
||||
# -r requirements.in
|
||||
# alembic
|
||||
# via alembic
|
||||
markupsafe==3.0.3 \
|
||||
--hash=sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f \
|
||||
--hash=sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a \
|
||||
@@ -560,7 +543,6 @@ markupsafe==3.0.3 \
|
||||
--hash=sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a \
|
||||
--hash=sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask
|
||||
# jinja2
|
||||
# mako
|
||||
@@ -569,7 +551,6 @@ marshmallow==4.3.0 \
|
||||
--hash=sha256:46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46 \
|
||||
--hash=sha256:fb43c53b3fe240b8f6af37223d6ef1636f927ad9bea8ab323afad95dff090880
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask-marshmallow
|
||||
# marshmallow-sqlalchemy
|
||||
marshmallow-sqlalchemy==1.5.0 \
|
||||
@@ -604,18 +585,18 @@ mysql-connector-python==9.7.0 \
|
||||
--hash=sha256:ea4e05ab864ab8b0b5066d885d89ca51e5ba1320dba520c20a6f9388a8eca022 \
|
||||
--hash=sha256:ee90c5f44f706f012be17f03f6ad158ff96e7f2dcc077896fe4537d3d28b3cf4
|
||||
# via -r requirements.in
|
||||
pycparser==3.0 \
|
||||
packaging==26.3 \
|
||||
--hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \
|
||||
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
|
||||
# via -r requirements.in
|
||||
pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy' \
|
||||
--hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \
|
||||
--hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992
|
||||
# via
|
||||
# -r requirements.in
|
||||
# cffi
|
||||
# via cffi
|
||||
pyjwt==2.12.1 \
|
||||
--hash=sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c \
|
||||
--hash=sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b
|
||||
# via
|
||||
# -r requirements.in
|
||||
# flask-jwt-extended
|
||||
# via flask-jwt-extended
|
||||
pymysql==1.1.3 \
|
||||
--hash=sha256:8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289 \
|
||||
--hash=sha256:e70ebf2047a4edf6138cf79c68ad418ef620af65900aa585c5e8bfc95044d43a
|
||||
@@ -693,7 +674,6 @@ sqlalchemy==2.0.49 \
|
||||
--hash=sha256:ec44cfa7ef1a728e88ad41674de50f6db8cfdb3e2af84af86e0041aaf02d43d0 \
|
||||
--hash=sha256:fb37f15714ec2652d574f021d479e78cd4eb9d04396dca36568fdfffb3487982
|
||||
# via
|
||||
# -r requirements.in
|
||||
# alembic
|
||||
# flask-sqlalchemy
|
||||
# marshmallow-sqlalchemy
|
||||
@@ -705,7 +685,6 @@ typing-extensions==4.15.0 \
|
||||
--hash=sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466 \
|
||||
--hash=sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548
|
||||
# via
|
||||
# -r requirements.in
|
||||
# alembic
|
||||
# sqlalchemy
|
||||
tzdata==2026.3 \
|
||||
@@ -715,9 +694,7 @@ tzdata==2026.3 \
|
||||
urllib3==2.7.0 \
|
||||
--hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \
|
||||
--hash=sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897
|
||||
# via
|
||||
# -r requirements.in
|
||||
# requests
|
||||
# via requests
|
||||
waitress==3.0.2 \
|
||||
--hash=sha256:682aaaf2af0c44ada4abfb70ded36393f0e307f4ab9456a215ce0020baefc31f \
|
||||
--hash=sha256:c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e
|
||||
|
||||
@@ -20,6 +20,21 @@ OUT = os.path.join(REPO, 'docs', 'openapi.json')
|
||||
VERBS = ('get', 'post', 'put', 'patch', 'delete', 'head', 'options')
|
||||
|
||||
|
||||
def _product_version():
|
||||
"""Read __version__ out of shopdb/__init__.py without importing the app.
|
||||
|
||||
Importing shopdb here would pull in Flask, the plugin loader and a database
|
||||
configuration, none of which this generator needs.
|
||||
"""
|
||||
here = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
init = os.path.join(here, 'shopdb', '__init__.py')
|
||||
with open(init, 'r', encoding='utf-8') as handle:
|
||||
match = re.search(r"^__version__ = '([^']+)'", handle.read(), re.M)
|
||||
if not match:
|
||||
raise SystemExit('could not read __version__ from ' + init)
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def _norm_path(p):
|
||||
# Flask <int:id> / <id> -> OpenAPI {id}
|
||||
return re.sub(r'<(?:[^:>]+:)?([^>]+)>', r'{\1}', p or '')
|
||||
@@ -68,7 +83,10 @@ def build(surfaces):
|
||||
'openapi': '3.1.0',
|
||||
'info': {
|
||||
'title': 'ShopDB Flask API',
|
||||
'version': '0.7.0',
|
||||
# Read from the code, not restated. A hardcoded copy here had
|
||||
# already drifted a release behind, and the same mistake in the
|
||||
# installer script shipped an exe stamped with the wrong version.
|
||||
'version': _product_version(),
|
||||
'description': (
|
||||
'Asset-management API (core + plugins). Responses use a '
|
||||
'`success_response` envelope: `{status, data, meta}`. Auth: Bearer '
|
||||
|
||||
@@ -42,7 +42,7 @@ __contract_version__ = '0.15.0'
|
||||
# plugin-contract version above are distinct series with independent
|
||||
# bump rules. Not part of the shopdb.api contract surface, so it is
|
||||
# not re-exported there.
|
||||
__version__ = '0.7.0'
|
||||
__version__ = '0.8.0'
|
||||
|
||||
|
||||
def create_app(config_name: str = None) -> Flask:
|
||||
@@ -172,12 +172,13 @@ def register_blueprints(app: Flask):
|
||||
def register_cli_commands(app: Flask):
|
||||
"""Register Flask CLI commands."""
|
||||
from .plugins.cli import plugin_cli
|
||||
from .cli import db_cli, seed_cli, relationships_cli
|
||||
from .cli import db_cli, seed_cli, relationships_cli, csv_cli
|
||||
|
||||
app.cli.add_command(plugin_cli)
|
||||
app.cli.add_command(db_cli)
|
||||
app.cli.add_command(seed_cli)
|
||||
app.cli.add_command(relationships_cli)
|
||||
app.cli.add_command(csv_cli)
|
||||
|
||||
|
||||
def register_error_handlers(app: Flask):
|
||||
|
||||
@@ -825,3 +825,160 @@ def seed_demo_clear(yes):
|
||||
click.echo(click.style(
|
||||
f"Removed {len(demo_ids)} demo assets and "
|
||||
f"{parts_deleted} sample parts.", fg='green'))
|
||||
|
||||
|
||||
@click.group('csv')
|
||||
def csv_cli():
|
||||
"""Load a site's starting data from CSV files."""
|
||||
pass
|
||||
|
||||
|
||||
@csv_cli.command('templates')
|
||||
@click.option('--out', 'outdir', default='csv-templates',
|
||||
help='Directory to write the templates into.')
|
||||
@with_appcontext
|
||||
def csv_templates(outdir):
|
||||
"""Write a CSV template per importable table, generated from the schema.
|
||||
|
||||
Generated rather than kept by hand: a maintained template set drifts on the
|
||||
next migration and does so silently, since the file still looks correct.
|
||||
"""
|
||||
import os
|
||||
from shopdb.core.services.csvimport import IMPORTABLE, generate_template
|
||||
|
||||
if not os.path.isdir(outdir):
|
||||
os.makedirs(outdir)
|
||||
for tablename in IMPORTABLE:
|
||||
path = os.path.join(outdir, tablename + '.csv')
|
||||
with open(path, 'w', encoding='utf-8') as handle:
|
||||
handle.write(generate_template(tablename))
|
||||
click.echo(' %s' % path)
|
||||
|
||||
readme = os.path.join(outdir, 'README.txt')
|
||||
with open(readme, 'w', encoding='utf-8') as handle:
|
||||
handle.write(
|
||||
'ShopDB-Flask import templates\n'
|
||||
'=============================\n\n'
|
||||
'Generated from the live database schema. Every column here exists;\n'
|
||||
'every required column is marked.\n\n'
|
||||
'Fill in the ones you need - you do not need all of them.\n\n'
|
||||
'Foreign keys take a NAME or a numeric id. Write the name:\n'
|
||||
' locationid -> Building 1 Bay 3\n'
|
||||
' vendorid -> Haas Automation\n'
|
||||
'The importer resolves it, and tells you which row and column to fix\n'
|
||||
'if the name is not found.\n\n'
|
||||
'Import the whole folder at once and order is handled for you:\n\n'
|
||||
' flask csv import --dir . (checks only, changes nothing)\n'
|
||||
' flask csv import --dir . --commit (applies)\n\n'
|
||||
'Nothing is written unless every row passes, so a mistake on line 400\n'
|
||||
'does not leave 399 rows half-imported.\n\n'
|
||||
'User accounts are deliberately not importable here: passwords do not\n'
|
||||
'belong in a spreadsheet.\n')
|
||||
click.echo(' %s' % readme)
|
||||
click.echo('')
|
||||
click.echo(click.style('%d templates written to %s' % (len(IMPORTABLE), outdir),
|
||||
fg='green', bold=True))
|
||||
|
||||
|
||||
@csv_cli.command('import')
|
||||
@click.option('--file', 'path', default=None, help='One CSV file.')
|
||||
@click.option('--dir', 'directory', default=None,
|
||||
help='A folder of CSVs, imported in dependency order.')
|
||||
@click.option('--table', 'tablename', default=None,
|
||||
help='Target table. Defaults to the file name.')
|
||||
@click.option('--commit', is_flag=True, default=False,
|
||||
help='Apply the changes. Without this, nothing is written.')
|
||||
@with_appcontext
|
||||
def csv_import(path, directory, tablename, commit):
|
||||
"""Validate CSVs and, with --commit, load them.
|
||||
|
||||
Dry run by default. The report is the same either way, so what you review is
|
||||
what you get.
|
||||
"""
|
||||
import os
|
||||
from shopdb.extensions import db
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from shopdb.core.services.csvimport import (
|
||||
ImportError_, Resolver, dependency_order, import_csv, table_from_filename)
|
||||
|
||||
if not path and not directory:
|
||||
raise click.UsageError('give --file or --dir')
|
||||
|
||||
jobs = []
|
||||
if path:
|
||||
jobs.append((tablename or table_from_filename(os.path.basename(path)), path))
|
||||
else:
|
||||
found = {}
|
||||
for name in os.listdir(directory):
|
||||
if not name.lower().endswith('.csv'):
|
||||
continue
|
||||
found[table_from_filename(name)] = os.path.join(directory, name)
|
||||
ordered = dependency_order(list(found))
|
||||
unknown = sorted(set(found) - set(ordered))
|
||||
for name in ordered:
|
||||
jobs.append((name, found[name]))
|
||||
if unknown:
|
||||
click.echo(click.style(
|
||||
'skipping (not importable): %s' % ', '.join(unknown), fg='yellow'))
|
||||
|
||||
if not jobs:
|
||||
click.echo('nothing to do - no CSV files found')
|
||||
return
|
||||
|
||||
click.echo(click.style(
|
||||
'Checking %d file(s)%s' % (len(jobs), '' if commit else ' - DRY RUN, nothing will be written'),
|
||||
bold=True))
|
||||
click.echo('')
|
||||
|
||||
resolver = Resolver()
|
||||
results = []
|
||||
failed = False
|
||||
for name, filepath in jobs:
|
||||
with open(filepath, 'r', encoding='utf-8-sig') as handle:
|
||||
text = handle.read()
|
||||
try:
|
||||
# Every file is applied inside ONE transaction, so a failure part way
|
||||
# through a folder rolls the whole run back rather than leaving the
|
||||
# site half-populated.
|
||||
result = import_csv(name, text, resolver=resolver, commit=commit)
|
||||
except ImportError_ as exc:
|
||||
click.echo(click.style(' %-18s %s' % (name, exc), fg='red'))
|
||||
failed = True
|
||||
continue
|
||||
except SQLAlchemyError as exc:
|
||||
# Anything the database itself refuses. The operator gets the cause
|
||||
# in one line rather than a traceback they cannot act on.
|
||||
db.session.rollback()
|
||||
click.echo(click.style(' %-18s database error: %s'
|
||||
% (name, str(exc).split(chr(10))[0]), fg='red'))
|
||||
failed = True
|
||||
continue
|
||||
results.append(result)
|
||||
colour = 'green' if result.ok else 'red'
|
||||
click.echo(click.style(' ' + result.summary(), fg=colour))
|
||||
for problem in result.problems[:20]:
|
||||
click.echo(click.style(' %s' % problem, fg='red'))
|
||||
if len(result.problems) > 20:
|
||||
click.echo(click.style(' ... and %d more' % (len(result.problems) - 20), fg='red'))
|
||||
if not result.ok:
|
||||
failed = True
|
||||
|
||||
click.echo('')
|
||||
if failed:
|
||||
db.session.rollback()
|
||||
click.echo(click.style('Nothing was imported. Fix the problems above and run again.',
|
||||
fg='red', bold=True))
|
||||
raise SystemExit(1)
|
||||
|
||||
total_new = sum(r.created for r in results)
|
||||
total_upd = sum(r.updated for r in results)
|
||||
if commit:
|
||||
db.session.commit()
|
||||
click.echo(click.style('Imported: %d new, %d updated.' % (total_new, total_upd),
|
||||
fg='green', bold=True))
|
||||
else:
|
||||
db.session.rollback()
|
||||
click.echo(click.style(
|
||||
'Looks good: %d would be created, %d updated.' % (total_new, total_upd),
|
||||
fg='green', bold=True))
|
||||
click.echo('Run again with --commit to apply.')
|
||||
|
||||
500
shopdb/core/services/csvimport.py
Normal file
500
shopdb/core/services/csvimport.py
Normal file
@@ -0,0 +1,500 @@
|
||||
"""Bulk-load a site's starting data from CSV files.
|
||||
|
||||
WHY THIS EXISTS. Adopting a site means getting its existing asset register into
|
||||
ShopDB. The HTTP import surface (the IMPORT-API wiki page) is the right tool when
|
||||
there is a source system to read from and someone able to script against it. A
|
||||
sister site with a spreadsheet and no developer needs something else, and that
|
||||
is the common case.
|
||||
|
||||
THE DESIGN PROBLEM, which a hand-written template set gets wrong. A CSV row for
|
||||
an asset has to say where the asset is. The database stores `locationid`, an
|
||||
integer. Nobody types integers into a spreadsheet correctly, and requiring it
|
||||
means the operator must first import locations, read back the generated ids, and
|
||||
paste them into the asset sheet. That is not a workflow anyone completes.
|
||||
|
||||
So every foreign key here accepts EITHER a numeric id OR the referenced row's
|
||||
name, and names are resolved at import time:
|
||||
|
||||
assetnumber,assettypeid,locationid,businessunitid
|
||||
CMM-01,Measuring Tool,Building 1 Bay 3,Quality
|
||||
|
||||
The column keeps its database name - the convention in CONTRIBUTING.md - while
|
||||
the VALUE is whatever the operator actually knows. A name that does not resolve
|
||||
is a row error naming the column, the value, and where to fix it, not a foreign
|
||||
key constraint violation from three layers down.
|
||||
|
||||
IDEMPOTENCE. Each table declares a natural key (`assetnumber`, `locationname`,
|
||||
...). Re-importing a file updates the matched rows rather than duplicating them,
|
||||
so a site can correct its spreadsheet and run it again - which they will.
|
||||
|
||||
SAFETY. Validation is a separate pass over the whole file BEFORE anything is
|
||||
written, and the default is a dry run. An import either applies completely or
|
||||
not at all.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
from collections import OrderedDict
|
||||
|
||||
from sqlalchemy import inspect as sqla_inspect
|
||||
|
||||
from shopdb.extensions import db
|
||||
|
||||
|
||||
class ImportError_(Exception):
|
||||
"""A problem with the request itself, not with a row."""
|
||||
|
||||
|
||||
# The tables a site may load, in dependency order: a table never appears before
|
||||
# something it references. `label` is the human-facing column - the one an
|
||||
# operator would write in a spreadsheet, and the one used to resolve this table
|
||||
# when another table points at it. `natural` is what makes a re-import an update
|
||||
# rather than a duplicate.
|
||||
#
|
||||
# Deliberately NOT every table. Transactional and derived data (audit rows,
|
||||
# reports, enforcement history) has no business arriving by spreadsheet, and
|
||||
# offering it would invite someone to try.
|
||||
IMPORTABLE = OrderedDict([
|
||||
('assetstatuses', {'label': 'status', 'natural': 'status'}),
|
||||
('assettypes', {'label': 'assettype', 'natural': 'assettype'}),
|
||||
('locationtypes', {'label': 'locationtype', 'natural': 'locationtype'}),
|
||||
('modeltypes', {'label': 'modeltype', 'natural': 'modeltype'}),
|
||||
('computertypes', {'label': 'computertype', 'natural': 'computertype'}),
|
||||
('machinetypes', {'label': 'machinetype', 'natural': 'machinetype'}),
|
||||
('businessunits', {'label': 'businessunit', 'natural': 'businessunit'}),
|
||||
('locations', {'label': 'locationname', 'natural': 'locationname'}),
|
||||
('vendors', {'label': 'vendor', 'natural': 'vendor'}),
|
||||
('models', {'label': 'modelnumber', 'natural': 'modelnumber'}),
|
||||
('operatingsystems', {'label': 'osname', 'natural': 'osname'}),
|
||||
('assets', {'label': 'assetnumber', 'natural': 'assetnumber'}),
|
||||
('computers', {'label': 'hostname', 'natural': 'assetid'}),
|
||||
('machines', {'label': None, 'natural': 'assetid'}),
|
||||
])
|
||||
|
||||
# Columns a CSV may never set: surrogate keys the database owns, and audit
|
||||
# stamps. Accepting an id from a spreadsheet is how two sites end up with
|
||||
# colliding primary keys the first time anyone merges data.
|
||||
NEVER_IMPORT = {'createddate', 'modifieddate'}
|
||||
|
||||
# Credentials are never taken from a file. A CSV that carries password hashes
|
||||
# gets mailed around, and one that carries plaintext is worse. Users are created
|
||||
# with a random password and must change it at first login.
|
||||
PASSWORD_COLUMNS = {'passwordhash', 'password'}
|
||||
|
||||
TRUE_VALUES = {'1', 'true', 'yes', 'y', 't'}
|
||||
FALSE_VALUES = {'0', 'false', 'no', 'n', 'f', ''}
|
||||
|
||||
|
||||
def table_for(name):
|
||||
table = db.metadata.tables.get(name)
|
||||
if table is None:
|
||||
raise ImportError_(
|
||||
"no such table: %s\nImportable: %s" % (name, ', '.join(IMPORTABLE)))
|
||||
return table
|
||||
|
||||
|
||||
def assert_schema_ready(tablename):
|
||||
"""Fail with a sentence, not a traceback, when the schema is not there yet.
|
||||
|
||||
Running this against a database that has never been migrated is an easy
|
||||
mistake - a fresh site, or the wrong DATABASE_URL - and SQLAlchemy's answer
|
||||
is a wall of stack frames ending in 'no such table'. The operator needs to
|
||||
be told to run the migrations.
|
||||
"""
|
||||
# The SESSION's connection, not db.engine. Taking a separate connection ends
|
||||
# up rolling back the session's in-flight transaction under a shared-
|
||||
# connection pool, which silently discarded rows that had just been written.
|
||||
inspector = sqla_inspect(db.session.connection())
|
||||
if not inspector.has_table(tablename):
|
||||
raise ImportError_(
|
||||
"the '%s' table does not exist in this database.\n"
|
||||
" Run 'flask db upgrade' first, and check DATABASE_URL points at the\n"
|
||||
" site you meant." % tablename)
|
||||
|
||||
|
||||
def importable_columns(tablename):
|
||||
"""Columns an operator may set, in a stable order.
|
||||
|
||||
Primary keys are excluded - they are the database's - EXCEPT where the key
|
||||
is also a foreign key, as on the subtype tables where `assetid` both
|
||||
identifies the row and points at its asset.
|
||||
"""
|
||||
table = table_for(tablename)
|
||||
primary = {c.name for c in table.primary_key}
|
||||
out = []
|
||||
for column in table.columns:
|
||||
if column.name in NEVER_IMPORT:
|
||||
continue
|
||||
if column.name in primary and not column.foreign_keys:
|
||||
continue
|
||||
if column.name in PASSWORD_COLUMNS:
|
||||
continue
|
||||
out.append(column)
|
||||
return out
|
||||
|
||||
|
||||
def required_columns(tablename):
|
||||
primary = {c.name for c in table_for(tablename).primary_key}
|
||||
return [
|
||||
c.name for c in importable_columns(tablename)
|
||||
if not c.nullable and c.name not in primary
|
||||
and c.default is None and c.server_default is None
|
||||
]
|
||||
|
||||
|
||||
def _fk_target(column):
|
||||
"""(tablename, idcolumn) this column points at, or None."""
|
||||
if not column.foreign_keys:
|
||||
return None
|
||||
target = list(column.foreign_keys)[0].column
|
||||
return target.table.name, target.name
|
||||
|
||||
|
||||
def _label_column(tablename):
|
||||
"""The human-readable column for a table, used to resolve references to it.
|
||||
|
||||
Derived where possible so this does not need maintaining: the schema names
|
||||
reference tables' label column after the table itself (assettypes.assettype,
|
||||
locationtypes.locationtype). IMPORTABLE overrides where that rule does not
|
||||
hold.
|
||||
"""
|
||||
spec = IMPORTABLE.get(tablename)
|
||||
if spec and spec.get('label'):
|
||||
return spec['label']
|
||||
table = db.metadata.tables.get(tablename)
|
||||
if table is None:
|
||||
return None
|
||||
singular = tablename[:-1] if tablename.endswith('s') else tablename
|
||||
for candidate in (singular, singular + 'name', tablename + 'name'):
|
||||
if candidate in table.columns:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _coerce(column, raw):
|
||||
"""Turn a CSV string into something the column will accept."""
|
||||
text = (raw or '').strip()
|
||||
typename = column.type.__class__.__name__.upper()
|
||||
|
||||
if text == '':
|
||||
return None
|
||||
|
||||
if 'BOOL' in typename:
|
||||
low = text.lower()
|
||||
if low in TRUE_VALUES:
|
||||
return True
|
||||
if low in FALSE_VALUES:
|
||||
return False
|
||||
raise ValueError("expected 1 or 0, got '%s'" % text)
|
||||
|
||||
if 'INT' in typename:
|
||||
try:
|
||||
return int(text)
|
||||
except ValueError:
|
||||
raise ValueError("expected a whole number, got '%s'" % text)
|
||||
|
||||
if 'FLOAT' in typename or 'NUMERIC' in typename or 'DECIMAL' in typename:
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
raise ValueError("expected a number, got '%s'" % text)
|
||||
|
||||
if 'DATE' in typename or 'TIME' in typename:
|
||||
from datetime import datetime
|
||||
for fmt in ('%Y-%m-%d %H:%M:%S', '%Y-%m-%dT%H:%M:%S', '%Y-%m-%d', '%d/%m/%Y', '%m/%d/%Y'):
|
||||
try:
|
||||
return datetime.strptime(text, fmt)
|
||||
except ValueError:
|
||||
continue
|
||||
raise ValueError("expected a date like 2026-08-04, got '%s'" % text)
|
||||
|
||||
length = getattr(column.type, 'length', None)
|
||||
if length and len(text) > length:
|
||||
raise ValueError("longer than the column allows (%d > %d characters)" % (len(text), length))
|
||||
return text
|
||||
|
||||
|
||||
class Resolver:
|
||||
"""Turns names into ids, remembering what it has already looked up.
|
||||
|
||||
Rows created earlier in the SAME import are resolvable, so one file can
|
||||
reference another that was loaded moments ago without a commit in between.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._cache = {}
|
||||
|
||||
def resolve(self, tablename, idcolumn, value):
|
||||
if value is None:
|
||||
return None
|
||||
# A number is taken at face value: exports round-trip, and some sites do
|
||||
# know their ids.
|
||||
if isinstance(value, int) or (isinstance(value, str) and value.strip().isdigit()):
|
||||
return int(value)
|
||||
|
||||
label = _label_column(tablename)
|
||||
if not label:
|
||||
raise ValueError(
|
||||
"cannot look up %s by name - give the numeric %s instead" % (tablename, idcolumn))
|
||||
|
||||
key = (tablename, str(value).strip().lower())
|
||||
if key in self._cache:
|
||||
return self._cache[key]
|
||||
|
||||
table = db.metadata.tables[tablename]
|
||||
row = db.session.execute(
|
||||
table.select().where(
|
||||
db.func.lower(db.func.trim(table.c[label])) == str(value).strip().lower())
|
||||
).first()
|
||||
if row is None:
|
||||
raise ValueError(
|
||||
"nothing in %s is named '%s' - add it to %s.csv, or import that file first"
|
||||
% (tablename, value, tablename))
|
||||
found = getattr(row, idcolumn)
|
||||
self._cache[key] = found
|
||||
return found
|
||||
|
||||
def remember(self, tablename, name, rowid):
|
||||
if name is None:
|
||||
return
|
||||
self._cache[(tablename, str(name).strip().lower())] = rowid
|
||||
|
||||
|
||||
class RowProblem(object):
|
||||
def __init__(self, line, column, message):
|
||||
self.line = line
|
||||
self.column = column
|
||||
self.message = message
|
||||
|
||||
def __str__(self):
|
||||
where = "line %d" % self.line
|
||||
if self.column:
|
||||
where += ", column '%s'" % self.column
|
||||
return "%s: %s" % (where, self.message)
|
||||
|
||||
|
||||
class ImportResult(object):
|
||||
def __init__(self, tablename):
|
||||
self.tablename = tablename
|
||||
self.created = 0
|
||||
self.updated = 0
|
||||
self.problems = []
|
||||
self.skipped = 0
|
||||
|
||||
@property
|
||||
def ok(self):
|
||||
return not self.problems
|
||||
|
||||
def summary(self):
|
||||
return "%s: %d new, %d updated, %d problem(s)" % (
|
||||
self.tablename, self.created, self.updated, len(self.problems))
|
||||
|
||||
|
||||
def read_rows(text):
|
||||
"""Rows as dicts, with the sample/comment lines the templates carry removed."""
|
||||
reader = csv.reader(io.StringIO(text))
|
||||
header = None
|
||||
rows = []
|
||||
for number, fields in enumerate(reader, 1):
|
||||
if not fields or all(not f.strip() for f in fields):
|
||||
continue
|
||||
if fields[0].lstrip().startswith('#'):
|
||||
continue
|
||||
if header is None:
|
||||
# Strip a UTF-8 BOM if one survived: Excel writes it, and a header
|
||||
# named '\ufeffassetnumber' would match nothing.
|
||||
header = [f.strip().lstrip('\ufeff') for f in fields]
|
||||
continue
|
||||
# A short row is padded rather than rejected: trailing empty cells are
|
||||
# routinely dropped by spreadsheet exports.
|
||||
padded = list(fields) + [''] * (len(header) - len(fields))
|
||||
rows.append((number, dict(zip(header, padded))))
|
||||
if header is None:
|
||||
raise ImportError_('the file has no header row')
|
||||
return header, rows
|
||||
|
||||
|
||||
def import_csv(tablename, text, resolver=None, commit=False):
|
||||
"""Validate a CSV against one table and write it into the current transaction.
|
||||
|
||||
Validation runs over EVERY row first. Nothing is written unless the whole
|
||||
file is clean, so a typo on line 400 does not leave 399 rows imported.
|
||||
|
||||
Writes always go into the SESSION; this never commits. The caller commits or
|
||||
rolls back, which is what makes a dry run over a whole folder work: assets
|
||||
can resolve the locations the previous file just created, because they are
|
||||
really there inside the transaction, and the rollback removes them again.
|
||||
Skipping the writes on a dry run instead made every cross-file reference
|
||||
fail, which is the one thing a folder-wide check exists to verify.
|
||||
|
||||
`commit` is retained only for callers that want the old explicit read.
|
||||
"""
|
||||
table = table_for(tablename)
|
||||
spec = IMPORTABLE.get(tablename)
|
||||
if spec is None:
|
||||
raise ImportError_(
|
||||
"%s is not importable.\nImportable: %s" % (tablename, ', '.join(IMPORTABLE)))
|
||||
|
||||
assert_schema_ready(tablename)
|
||||
resolver = resolver or Resolver()
|
||||
result = ImportResult(tablename)
|
||||
header, rows = read_rows(text)
|
||||
|
||||
allowed = {c.name: c for c in importable_columns(tablename)}
|
||||
unknown = [h for h in header if h not in allowed]
|
||||
if unknown:
|
||||
known = ', '.join(sorted(allowed))
|
||||
result.problems.append(RowProblem(
|
||||
1, None, "unknown column(s): %s\n this table accepts: %s" % (', '.join(unknown), known)))
|
||||
return result
|
||||
|
||||
missing = [c for c in required_columns(tablename) if c not in header]
|
||||
if missing:
|
||||
result.problems.append(RowProblem(
|
||||
1, None, "required column(s) missing: %s" % ', '.join(missing)))
|
||||
return result
|
||||
|
||||
natural = spec['natural']
|
||||
if natural not in header:
|
||||
result.problems.append(RowProblem(
|
||||
1, None, "the '%s' column identifies each row and must be present" % natural))
|
||||
return result
|
||||
|
||||
prepared = []
|
||||
for line, raw in rows:
|
||||
values = {}
|
||||
failed = False
|
||||
for name, column in allowed.items():
|
||||
if name not in raw:
|
||||
continue
|
||||
target = _fk_target(column)
|
||||
# RESOLVE BEFORE COERCING. A foreign key column is an integer, so
|
||||
# coercing first rejects every name with "expected a whole number" -
|
||||
# which defeats the entire point of accepting names. Resolution
|
||||
# handles the numeric case itself.
|
||||
if target:
|
||||
text = (raw[name] or '').strip()
|
||||
if text == '':
|
||||
values[name] = None
|
||||
continue
|
||||
try:
|
||||
values[name] = resolver.resolve(target[0], target[1], text)
|
||||
except ValueError as exc:
|
||||
result.problems.append(RowProblem(line, name, str(exc)))
|
||||
failed = True
|
||||
continue
|
||||
try:
|
||||
values[name] = _coerce(column, raw[name])
|
||||
except ValueError as exc:
|
||||
result.problems.append(RowProblem(line, name, str(exc)))
|
||||
failed = True
|
||||
|
||||
if failed:
|
||||
continue
|
||||
if values.get(natural) in (None, ''):
|
||||
result.problems.append(RowProblem(line, natural, 'must have a value'))
|
||||
continue
|
||||
prepared.append((line, values))
|
||||
|
||||
if not result.ok:
|
||||
return result
|
||||
|
||||
# Second pass: decide insert vs update. Counted even on a dry run, because
|
||||
# "this will change 4000 rows" is the number an operator needs before saying
|
||||
# yes.
|
||||
label = _label_column(tablename)
|
||||
for line, values in prepared:
|
||||
existing = db.session.execute(
|
||||
table.select().where(table.c[natural] == values[natural])).first()
|
||||
if existing is None:
|
||||
result.created += 1
|
||||
inserted = db.session.execute(table.insert().values(**values))
|
||||
newid = inserted.inserted_primary_key[0] if inserted.inserted_primary_key else None
|
||||
if label and label in values:
|
||||
resolver.remember(tablename, values[label], newid)
|
||||
else:
|
||||
result.updated += 1
|
||||
db.session.execute(
|
||||
table.update().where(table.c[natural] == values[natural]).values(**values))
|
||||
if label and label in values:
|
||||
primary = list(table.primary_key)[0].name
|
||||
resolver.remember(tablename, values[label], getattr(existing, primary))
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def generate_template(tablename):
|
||||
"""A header row plus a commented example, built from the live schema.
|
||||
|
||||
Generated rather than maintained: a hand-written template set drifts on the
|
||||
next migration, and silently - the file still looks right.
|
||||
"""
|
||||
columns = importable_columns(tablename)
|
||||
required = set(required_columns(tablename))
|
||||
header = [c.name for c in columns]
|
||||
|
||||
notes = []
|
||||
for column in columns:
|
||||
bits = []
|
||||
bits.append('REQUIRED' if column.name in required else 'optional')
|
||||
target = _fk_target(column)
|
||||
if target:
|
||||
label = _label_column(target[0])
|
||||
if label:
|
||||
bits.append("name from %s.csv (or a numeric id)" % target[0])
|
||||
else:
|
||||
bits.append("numeric %s id" % target[0])
|
||||
else:
|
||||
typename = column.type.__class__.__name__.upper()
|
||||
if 'BOOL' in typename:
|
||||
bits.append('1 or 0')
|
||||
elif 'DATE' in typename or 'TIME' in typename:
|
||||
bits.append('YYYY-MM-DD')
|
||||
elif getattr(column.type, 'length', None):
|
||||
bits.append('text, max %d' % column.type.length)
|
||||
notes.append("# %-24s %s" % (column.name, ', '.join(bits)))
|
||||
|
||||
spec = IMPORTABLE[tablename]
|
||||
lines = [
|
||||
"# %s.csv - generated from the ShopDB schema, do not hand-maintain" % tablename,
|
||||
"#",
|
||||
"# Rows are matched on '%s'. Re-importing updates a matching row rather" % spec['natural'],
|
||||
"# than creating a second one, so it is safe to correct this file and run again.",
|
||||
"#",
|
||||
"# Columns:",
|
||||
] + notes + [
|
||||
"#",
|
||||
"# Lines starting with # are ignored. Delete the example row before importing.",
|
||||
','.join(header),
|
||||
'# ' + ','.join(_example(c, required) for c in columns),
|
||||
]
|
||||
return '\n'.join(lines) + '\n'
|
||||
|
||||
|
||||
def _example(column, required):
|
||||
target = _fk_target(column)
|
||||
if target:
|
||||
label = _label_column(target[0])
|
||||
return ('<%s name>' % target[0][:-1]) if label else '<%s id>' % target[0]
|
||||
typename = column.type.__class__.__name__.upper()
|
||||
if 'BOOL' in typename:
|
||||
return '1'
|
||||
if 'DATE' in typename or 'TIME' in typename:
|
||||
return '2026-08-04'
|
||||
if 'INT' in typename or 'FLOAT' in typename or 'NUMERIC' in typename:
|
||||
return '0'
|
||||
return '<%s>' % column.name
|
||||
|
||||
|
||||
def dependency_order(names):
|
||||
"""The given tables, ordered so a table never precedes what it references."""
|
||||
known = [t for t in IMPORTABLE if t in set(names)]
|
||||
return known
|
||||
|
||||
|
||||
def table_from_filename(filename):
|
||||
stem = re.sub(r'\.csv$', '', filename.strip(), flags=re.I)
|
||||
stem = stem.rsplit('/', 1)[-1].rsplit('\\', 1)[-1].lower()
|
||||
return stem
|
||||
236
tests/test_csv_import.py
Normal file
236
tests/test_csv_import.py
Normal file
@@ -0,0 +1,236 @@
|
||||
"""Bulk CSV import: name resolution, all-or-nothing safety, and template truth.
|
||||
|
||||
The behaviour under test is what makes a spreadsheet import usable by someone
|
||||
who does not have the database in front of them. In particular: a foreign key
|
||||
column accepts the NAME of the thing it points at. Requiring numeric ids means
|
||||
importing a file, reading back generated ids, and pasting them into the next
|
||||
file - a workflow nobody finishes.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from shopdb.core.services.csvimport import (
|
||||
IMPORTABLE, ImportError_, Resolver, generate_template, import_csv,
|
||||
importable_columns, required_columns,
|
||||
)
|
||||
from shopdb.extensions import db as _db
|
||||
|
||||
|
||||
REFERENCE = [
|
||||
('assettypes', 'assettype,description\nMachine,Shop floor\nMeasuring Tool,Gage lab\n'),
|
||||
('assetstatuses', 'status,description\nActive,In service\nRetired,Out\n'),
|
||||
('locationtypes', 'locationtype\nBay\nRoom\n'),
|
||||
]
|
||||
|
||||
|
||||
def seed_reference(resolver):
|
||||
for table, text in REFERENCE:
|
||||
result = import_csv(table, text, resolver=resolver, commit=True)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
result = import_csv(
|
||||
'locations',
|
||||
'locationname,building,locationtypeid\nBay 3,Building 1,Bay\nGage Lab,Building 2,Room\n',
|
||||
resolver=resolver, commit=True)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
|
||||
|
||||
def count(table):
|
||||
return _db.session.execute(_db.text('select count(*) from %s' % table)).scalar()
|
||||
|
||||
|
||||
def test_foreign_keys_resolve_by_name(db):
|
||||
"""The headline behaviour: write 'Gage Lab', not locationid=2."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets',
|
||||
'assetnumber,name,assettypeid,statusid,locationid\n'
|
||||
'CMM-01,Zeiss,Measuring Tool,Active,Gage Lab\n',
|
||||
resolver=resolver, commit=True)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
|
||||
row = _db.session.execute(_db.text(
|
||||
'select t.assettype, s.status, l.locationname from assets a '
|
||||
'join assettypes t on t.assettypeid = a.assettypeid '
|
||||
'join assetstatuses s on s.statusid = a.statusid '
|
||||
'join locations l on l.locationid = a.locationid')).fetchone()
|
||||
assert tuple(row) == ('Measuring Tool', 'Active', 'Gage Lab')
|
||||
|
||||
|
||||
def test_numeric_ids_still_work(db):
|
||||
"""Exports round-trip, and some sites genuinely know their ids."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
typeid = _db.session.execute(_db.text(
|
||||
"select assettypeid from assettypes where assettype = 'Machine'")).scalar()
|
||||
result = import_csv(
|
||||
'assets', 'assetnumber,assettypeid\nMILL-1,%d\n' % typeid,
|
||||
resolver=resolver, commit=True)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
assert count('assets') == 1
|
||||
|
||||
|
||||
def test_unknown_name_names_the_column_and_the_value(db):
|
||||
"""The error has to be actionable by someone holding a spreadsheet."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets',
|
||||
'assetnumber,assettypeid,locationid\nX-1,Machine,Bay 9\n',
|
||||
resolver=resolver, commit=False)
|
||||
assert not result.ok
|
||||
message = str(result.problems[0])
|
||||
assert "locationid" in message
|
||||
assert "Bay 9" in message
|
||||
assert "locations.csv" in message
|
||||
|
||||
|
||||
def test_reimport_updates_rather_than_duplicating(db):
|
||||
"""Sites correct their spreadsheet and run it again. That must not double up."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
text = 'assetnumber,name,assettypeid\nCMM-01,Zeiss,Machine\n'
|
||||
first = import_csv('assets', text, resolver=resolver, commit=True)
|
||||
assert (first.created, first.updated) == (1, 0)
|
||||
|
||||
renamed = 'assetnumber,name,assettypeid\nCMM-01,Zeiss Contura,Machine\n'
|
||||
second = import_csv('assets', renamed, resolver=resolver, commit=True)
|
||||
assert (second.created, second.updated) == (0, 1)
|
||||
assert count('assets') == 1
|
||||
assert _db.session.execute(_db.text(
|
||||
"select name from assets where assetnumber = 'CMM-01'")).scalar() == 'Zeiss Contura'
|
||||
|
||||
|
||||
def test_one_bad_row_writes_nothing(db):
|
||||
"""A typo on the last line must not leave the earlier lines imported."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets',
|
||||
'assetnumber,assettypeid\n'
|
||||
'GOOD-1,Machine\n'
|
||||
'GOOD-2,Machine\n'
|
||||
'BAD-3,NoSuchType\n',
|
||||
resolver=resolver, commit=True)
|
||||
assert not result.ok
|
||||
assert count('assets') == 0, 'rows were written despite a failure'
|
||||
|
||||
|
||||
def test_unknown_column_is_rejected_with_the_accepted_list(db):
|
||||
"""The failure mode of a hand-written template: invented columns."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'vendors', 'vendor,zipcode\nDell,78682\n', resolver=resolver, commit=False)
|
||||
assert not result.ok
|
||||
message = str(result.problems[0])
|
||||
assert 'zipcode' in message
|
||||
assert 'this table accepts' in message
|
||||
|
||||
|
||||
def test_missing_required_column_is_rejected(db):
|
||||
result = import_csv('assets', 'name\nNo asset number\n', commit=False)
|
||||
assert not result.ok
|
||||
assert 'assetnumber' in str(result.problems[0])
|
||||
|
||||
|
||||
def test_booleans_take_1_and_0_and_reject_prose(db):
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
ok = import_csv('assets', 'assetnumber,assettypeid,isactive\nA-1,Machine,0\n',
|
||||
resolver=resolver, commit=True)
|
||||
assert ok.ok, [str(p) for p in ok.problems]
|
||||
|
||||
bad = import_csv('assets', 'assetnumber,assettypeid,isactive\nA-2,Machine,maybe\n',
|
||||
resolver=resolver, commit=False)
|
||||
assert not bad.ok
|
||||
assert 'isactive' in str(bad.problems[0])
|
||||
|
||||
|
||||
def test_comment_and_blank_lines_are_ignored(db):
|
||||
"""The templates ship a commented example. Importing one unedited should be
|
||||
a no-op, not a row of literal placeholder text."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets',
|
||||
'# a comment\nassetnumber,assettypeid\n\n# <assetnumber>,<type>\nREAL-1,Machine\n',
|
||||
resolver=resolver, commit=True)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
assert result.created == 1
|
||||
|
||||
|
||||
def test_passwords_are_never_importable():
|
||||
"""A CSV carrying password hashes gets mailed around; one carrying plaintext
|
||||
is worse. Neither column may be set from a file."""
|
||||
for table in IMPORTABLE:
|
||||
names = {c.name for c in importable_columns(table)}
|
||||
assert 'passwordhash' not in names
|
||||
assert 'password' not in names
|
||||
|
||||
|
||||
def test_generated_templates_only_contain_real_columns(db):
|
||||
"""The regression that motivated all of this.
|
||||
|
||||
A hand-written template set had invented columns on 7 of 11 tables and named
|
||||
a table that does not exist. Generating from the schema is what makes that
|
||||
impossible - so assert the generator actually does it.
|
||||
"""
|
||||
for table in IMPORTABLE:
|
||||
text = generate_template(table)
|
||||
header = [line for line in text.splitlines() if line and not line.startswith('#')][0]
|
||||
real = {c.name for c in importable_columns(table)}
|
||||
for column in header.split(','):
|
||||
assert column in real, '%s.csv offers a column that does not exist: %s' % (table, column)
|
||||
|
||||
|
||||
def test_every_template_includes_its_required_columns(db):
|
||||
"""A template that omits a required column produces a file that cannot import."""
|
||||
for table in IMPORTABLE:
|
||||
header = [line for line in generate_template(table).splitlines()
|
||||
if line and not line.startswith('#')][0].split(',')
|
||||
for column in required_columns(table):
|
||||
assert column in header, '%s.csv is missing required column %s' % (table, column)
|
||||
|
||||
|
||||
def test_importable_tables_all_exist(db):
|
||||
"""A registry entry naming a table that is not in the schema would fail only
|
||||
when someone tried to use it."""
|
||||
for table in IMPORTABLE:
|
||||
assert table in _db.metadata.tables, '%s is registered but not in the schema' % table
|
||||
|
||||
|
||||
def test_unimportable_table_is_refused(db):
|
||||
with pytest.raises(ImportError_):
|
||||
import_csv('auditlog', 'x\n1\n', commit=False)
|
||||
|
||||
|
||||
def test_dry_run_reports_counts_then_rolls_back(db):
|
||||
"""The number an operator needs before saying yes - and no trace afterwards.
|
||||
|
||||
Writes go into the transaction even on a dry run, which is what lets a later
|
||||
file resolve names an earlier one created. The rollback is what makes it a
|
||||
dry run.
|
||||
"""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets', 'assetnumber,assettypeid\nD-1,Machine\nD-2,Machine\n',
|
||||
resolver=resolver)
|
||||
assert result.ok
|
||||
assert result.created == 2
|
||||
_db.session.rollback()
|
||||
assert count('assets') == 0
|
||||
|
||||
|
||||
def test_a_dry_run_over_several_files_resolves_across_them(db):
|
||||
"""The whole-folder check: assets must resolve locations that only exist
|
||||
because an earlier file in the same run created them."""
|
||||
resolver = Resolver()
|
||||
seed_reference(resolver)
|
||||
result = import_csv(
|
||||
'assets',
|
||||
'assetnumber,assettypeid,locationid\nX-1,Machine,Bay 3\n',
|
||||
resolver=resolver)
|
||||
assert result.ok, [str(p) for p in result.problems]
|
||||
_db.session.rollback()
|
||||
assert count('assets') == 0
|
||||
@@ -53,3 +53,54 @@ def test_every_wizard_exclusion_is_a_real_plugin():
|
||||
for name in wizard_excludes():
|
||||
assert (PLUGINS / name / 'manifest.json').exists(), (
|
||||
'%s is excluded by the wizard but has no manifest' % name)
|
||||
|
||||
|
||||
# The preflight blocks the wizard on any FAIL. That is correct for something the
|
||||
# operator must go and fix, and wrong for anything the installer carries in its
|
||||
# own bundle - there, blocking stops the wizard over something it was about to do
|
||||
# itself, with no way forward.
|
||||
BUNDLE_SUPPLIED = [
|
||||
'HttpPlatformHandler', # httpplatformhandler\*.msi, installed by stage 4
|
||||
'URL Rewrite', # urlrewrite\*.msi, installed by stage 4
|
||||
]
|
||||
|
||||
PREFLIGHT = REPO / 'deploy' / 'windows' / 'installer' / 'shopdb-preflight.ps1'
|
||||
|
||||
|
||||
def test_nothing_the_bundle_supplies_is_a_blocker():
|
||||
"""Regression: HttpPlatformHandler was a FAIL, so a server without it could
|
||||
not get past the preflight page - to install the very thing that was
|
||||
missing."""
|
||||
text = PREFLIGHT.read_text(encoding='utf-8', errors='replace')
|
||||
offenders = []
|
||||
for line in text.splitlines():
|
||||
if "'FAIL'" not in line:
|
||||
continue
|
||||
for component in BUNDLE_SUPPLIED:
|
||||
if component.lower() in line.lower():
|
||||
offenders.append(line.strip())
|
||||
assert not offenders, (
|
||||
'the installer supplies these, so they must not block the wizard:\n %s'
|
||||
% '\n '.join(offenders))
|
||||
|
||||
|
||||
def test_installer_controlled_paths_have_no_spaces():
|
||||
"""msiexec and bootstrapper command lines split on spaces.
|
||||
|
||||
The installer passes these as PROPERTY=value / TargetDir=value on a command
|
||||
line. A value containing a space has to be quoted as PROPERTY="C:\\..." -
|
||||
quoting the whole token instead makes the tool reject the line and print its
|
||||
usage dialog, which is what 'the bundled MySQL database could not be
|
||||
installed' looked like from the outside. The Python target was already
|
||||
space-free for this exact reason; MySQL was not.
|
||||
"""
|
||||
text = PREFLIGHT.parent.joinpath('shopdb-install.ps1').read_text(
|
||||
encoding='utf-8', errors='replace')
|
||||
offenders = []
|
||||
for line in text.splitlines():
|
||||
match = re.match(r"\s*\[string\]\s*\$(Mysql\w*|PyTarget)\s*=\s*'([^']+)'", line)
|
||||
if match and ' ' in match.group(2):
|
||||
offenders.append('%s = %s' % (match.group(1), match.group(2)))
|
||||
assert not offenders, (
|
||||
'these are passed on a command line and must not contain spaces:\n %s'
|
||||
% '\n '.join(offenders))
|
||||
|
||||
106
tests/test_installer_state.py
Normal file
106
tests/test_installer_state.py
Normal file
@@ -0,0 +1,106 @@
|
||||
"""Guards on the installer's durable install record.
|
||||
|
||||
The record replaced a set of guesses. "Is this a re-run of MY install?" used to
|
||||
be answered from whatever the machine looked like - a MySQL service exists, the
|
||||
database has tables, the site exists - and none of those say who created them.
|
||||
A retry after a failed first install was therefore taken for an upgrade of a
|
||||
working system, which dead-ended the retry the wizard invites the operator to
|
||||
run.
|
||||
|
||||
Getting it wrong is asymmetric. Answering "still first provisioning" for a
|
||||
server holding real data would let `prune-schema --force` drop live tables. The
|
||||
behavioural checks live in the PowerShell harness next to the installer, run
|
||||
here when pwsh is available; the text assertions below hold everywhere, because
|
||||
CI does not necessarily have pwsh and these are the invariants worth failing a
|
||||
build over.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
INSTALLER = REPO / 'deploy' / 'windows' / 'installer' / 'shopdb-install.ps1'
|
||||
HARNESS = REPO / 'deploy' / 'windows' / 'installer' / 'tests' / 'test-install-state.ps1'
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not INSTALLER.is_file(),
|
||||
reason='installer not in this checkout')
|
||||
|
||||
|
||||
def installer_text():
|
||||
return INSTALLER.read_text(encoding='utf-8', errors='replace')
|
||||
|
||||
|
||||
def test_prune_force_is_gated_on_first_provisioning():
|
||||
"""--force drops tables that contain rows.
|
||||
|
||||
It is correct on a first install, where core migrations seed a few plugin
|
||||
reference tables, and destructive on an upgrade, where those rows are the
|
||||
site's data. The gate must therefore test BOTH "database was empty" and the
|
||||
install record, never the database alone.
|
||||
"""
|
||||
text = installer_text()
|
||||
assert "'prune-schema','--yes','--force'" in text, 'the forcing call moved or was renamed'
|
||||
guard = "if ((-not $script:DbWasEmpty) -and (-not $firstRun)) {"
|
||||
assert guard in text, (
|
||||
'the prune gate no longer consults the install record. Forcing on the '
|
||||
'strength of an empty database alone is what made every retry of a '
|
||||
'failed first install fail at stage 3.')
|
||||
|
||||
|
||||
def test_first_provisioning_comes_from_the_record_not_the_database():
|
||||
text = installer_text()
|
||||
assert 'function Test-FirstProvisioning' in text
|
||||
assert '$firstRun = Test-FirstProvisioning' in text, (
|
||||
'stage 3 must ask the record, not infer from table counts')
|
||||
|
||||
|
||||
def test_missing_record_takes_the_safe_side():
|
||||
"""An install predating the record has a version stamp and probably data.
|
||||
|
||||
Treating that as a first run would arm prune --force against it.
|
||||
"""
|
||||
text = installer_text()
|
||||
assert "'predates-this-record'" in text, (
|
||||
'the fallback for an unrecorded but stamped install is gone; without it '
|
||||
'an established server can be classified as first provisioning')
|
||||
|
||||
|
||||
def test_created_items_returns_an_array_when_empty():
|
||||
"""A zero-length array returned from a PowerShell function unrolls to $null.
|
||||
|
||||
$null.Count is then a terminating error under Set-StrictMode -Version 2.0 -
|
||||
the same fault that made bundle verification fail on every install. The
|
||||
leading comma is what prevents it.
|
||||
"""
|
||||
text = installer_text()
|
||||
assert 'return ,@()' in text, 'Get-CreatedItems lost its comma protection'
|
||||
assert 'return ,@(Get-StateValue $created $Kind' in text, (
|
||||
'the populated return path lost its comma protection')
|
||||
|
||||
|
||||
def test_record_is_written_before_any_stage_runs():
|
||||
"""Written at the START of provisioning, not the end.
|
||||
|
||||
A run that dies halfway is exactly the run whose retry needs the record.
|
||||
"""
|
||||
text = installer_text()
|
||||
assert "if ($Stage -ne 'uninstall') { Initialize-InstallState }" in text
|
||||
start = text.index('Initialize-InstallState }')
|
||||
switch = text.index('switch ($Stage)', start - 400)
|
||||
assert start < switch, 'the record must be initialised before the stage switch'
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which('pwsh') is None, reason='pwsh not installed')
|
||||
def test_install_state_behaviour():
|
||||
"""Run the PowerShell harness: retries, upgrades, corrupt and foreign records."""
|
||||
assert HARNESS.is_file(), 'the PowerShell harness is missing'
|
||||
result = subprocess.run(
|
||||
['pwsh', '-NoProfile', '-File', str(HARNESS)],
|
||||
capture_output=True, text=True, timeout=300)
|
||||
assert result.returncode == 0, (
|
||||
'install-record harness failed:\n' + result.stdout + result.stderr)
|
||||
assert 'all checks passed' in result.stdout
|
||||
159
tests/test_runtime_dependencies.py
Normal file
159
tests/test_runtime_dependencies.py
Normal file
@@ -0,0 +1,159 @@
|
||||
"""Every third-party module the app imports at runtime must be a DECLARED dependency.
|
||||
|
||||
The bug this exists to prevent: `shopdb/plugins/loader.py` imports `packaging`,
|
||||
which was never listed in requirements.in. Locally and in CI that import
|
||||
resolved anyway, because pytest depends on packaging -- so all 1159 tests
|
||||
passed. The Windows installer builds its venv from requirements.txt alone, with
|
||||
no test dependencies, so `import shopdb` raised ModuleNotFoundError and the
|
||||
whole application failed to start on a customer server.
|
||||
|
||||
Test dependencies are exactly the blind spot: they are present everywhere the
|
||||
test suite runs and absent everywhere it does not.
|
||||
|
||||
The check is static (AST) rather than a real import, so it covers plugins that
|
||||
are not installed at this site and code paths no test exercises.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
try:
|
||||
from importlib.metadata import packages_distributions
|
||||
except ImportError: # pragma: no cover - Python < 3.10
|
||||
packages_distributions = None
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
SCANNED = ('shopdb', 'plugins', 'scripts')
|
||||
|
||||
# First-party top-level names. Imports of these resolve from the source tree.
|
||||
FIRST_PARTY = {'shopdb', 'plugins', 'scripts', 'migrations', 'tests', 'wsgi', 'conftest'}
|
||||
|
||||
# Modules deliberately imported without being declared. Each needs a reason.
|
||||
ALLOWED_UNDECLARED = {
|
||||
# Provided by the interpreter's own bundled installer tooling, and only
|
||||
# touched by developer scripts that never run on a deployed site.
|
||||
'pip',
|
||||
'setuptools',
|
||||
'pkg_resources',
|
||||
}
|
||||
|
||||
|
||||
def _normalise(name):
|
||||
"""PEP 503 normalisation, so Flask_Migrate and flask-migrate compare equal."""
|
||||
return name.lower().replace('_', '-').replace('.', '-')
|
||||
|
||||
|
||||
def _declared_distributions():
|
||||
"""Distribution names pinned in requirements.txt."""
|
||||
text = (REPO / 'requirements.txt').read_text(encoding='utf-8')
|
||||
declared = set()
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith('#') or line.startswith('--'):
|
||||
continue
|
||||
# "name==1.2.3 ; marker \" -> "name"
|
||||
head = line.split('==', 1)[0].split(';', 1)[0].strip()
|
||||
if head:
|
||||
declared.add(_normalise(head))
|
||||
return declared
|
||||
|
||||
|
||||
def _iter_source_files():
|
||||
for top in SCANNED:
|
||||
root = REPO / top
|
||||
if not root.is_dir():
|
||||
continue
|
||||
for path in root.rglob('*.py'):
|
||||
parts = set(path.parts)
|
||||
if {'node_modules', 'venv', '.venv', 'build'} & parts:
|
||||
continue
|
||||
yield path
|
||||
|
||||
|
||||
def _imported_top_level_modules(path):
|
||||
"""Top-level module names imported unconditionally by `path`.
|
||||
|
||||
Imports nested inside a `try:` are skipped: that is how this codebase spells
|
||||
an optional dependency, and a missing one is handled rather than fatal.
|
||||
Imports inside functions are still counted -- a deferred import of an
|
||||
undeclared package fails just as hard, only later.
|
||||
"""
|
||||
try:
|
||||
tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))
|
||||
except SyntaxError:
|
||||
pytest.fail('could not parse {}'.format(path.relative_to(REPO)))
|
||||
|
||||
guarded = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Try):
|
||||
for child in ast.walk(node):
|
||||
guarded.add(id(child))
|
||||
|
||||
found = set()
|
||||
for node in ast.walk(tree):
|
||||
if id(node) in guarded:
|
||||
continue
|
||||
if isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
found.add(alias.name.split('.')[0])
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
# level > 0 is a relative (first-party) import.
|
||||
if node.level == 0 and node.module:
|
||||
found.add(node.module.split('.')[0])
|
||||
return found
|
||||
|
||||
|
||||
@pytest.mark.skipif(packages_distributions is None, reason='needs Python 3.10+')
|
||||
def test_runtime_imports_are_declared_dependencies():
|
||||
declared = _declared_distributions()
|
||||
assert 'flask' in declared, 'requirements.txt did not parse as expected'
|
||||
|
||||
module_to_dists = packages_distributions()
|
||||
stdlib = getattr(sys, 'stdlib_module_names', frozenset())
|
||||
|
||||
offenders = {}
|
||||
for path in _iter_source_files():
|
||||
for module in _imported_top_level_modules(path):
|
||||
if module in FIRST_PARTY or module in stdlib or module in ALLOWED_UNDECLARED:
|
||||
continue
|
||||
if module.startswith('_'):
|
||||
continue
|
||||
dists = module_to_dists.get(module)
|
||||
if not dists:
|
||||
# Not installed in this environment, so its distribution cannot
|
||||
# be identified. Absence here is not evidence of a problem.
|
||||
continue
|
||||
if not any(_normalise(d) in declared for d in dists):
|
||||
rel = str(path.relative_to(REPO))
|
||||
offenders.setdefault(
|
||||
'{} (from {})'.format(module, '/'.join(sorted(dists))), []
|
||||
).append(rel)
|
||||
|
||||
if offenders:
|
||||
lines = []
|
||||
for module, files in sorted(offenders.items()):
|
||||
shown = sorted(files)[:3]
|
||||
more = '' if len(files) <= 3 else ' (+{} more)'.format(len(files) - 3)
|
||||
lines.append(' {}\n {}{}'.format(module, ', '.join(shown), more))
|
||||
pytest.fail(
|
||||
'Imported at runtime but not declared in requirements.txt.\n'
|
||||
'These resolve here only because a test dependency supplies them; a\n'
|
||||
'production venv has no test dependencies and will fail to import.\n'
|
||||
'Add them to requirements.in and recompile.\n\n' + '\n'.join(lines)
|
||||
)
|
||||
|
||||
|
||||
def test_packaging_is_declared():
|
||||
"""Regression: the specific import that broke the Windows install.
|
||||
|
||||
Kept separate from the sweep above because that one can only flag a module
|
||||
it can attribute to an installed distribution -- an environment without
|
||||
packaging would skip it silently, which is precisely the shipping case.
|
||||
"""
|
||||
assert 'packaging' in _declared_distributions(), (
|
||||
'shopdb/plugins/loader.py imports packaging at module scope. Without it '
|
||||
'declared, `import shopdb` fails on any venv built from requirements.txt.'
|
||||
)
|
||||
Reference in New Issue
Block a user