From aea2905de0f4d1508397833ef1f1e1128833d9c9 Mon Sep 17 00:00:00 2001 From: cproudlock Date: Mon, 3 Aug 2026 14:39:38 -0400 Subject: [PATCH] fix(installer): stop it lying, stop it leaking, and make it findable Nine fixes from a review of the installer against its actual audience: DT leads at sister sites who are not Windows, IIS or Python specialists and who will lean on an AI assistant to get through it. TRUTHFULNESS. The preflight was advisory - an operator read 'IIS is not installed', pressed Next, answered five more pages and the install died partway through with Python already on the box. The results page now blocks while anything is failing, repaints on every run instead of latching after the first, and offers 'Check again' so a fixed problem does not mean starting over. On failure the wizard said 'Nothing was left running', which is false in every path because the stages run with -OnFailure never: it now says the server is part-configured, that re-running is safe, and how to remove it. The final page no longer reads 'ShopDB-Flask is ready' after a failed install. SECRETS. The generated MySQL root password went to Write-Host in a process the wizard runs hidden - so nobody saw it - and stdout is forwarded into the setup log operators are told to send to support, so it was permanently recorded for everyone who did not need it. It now goes to an ACL'd file. Database dumps, which contain every user password hash, landed in a ProgramData directory readable by every user on the box; the directory is now locked at creation. UPGRADES ON REMOTE-DATABASE SITES. mysqldump was looked for only under local MySQL install paths, so a site whose database is on another host silently skipped every pre-upgrade backup - after stage 2 had already stopped the pool and replaced the tree. Find-MysqlTool now prefers a client shipped in the bundle, stage 2 stages it onto the server, preflight reports when it is missing, and mysqlclient\ is an optional locked payload. UNINSTALL. A subpath install is an IIS Application, not a site; removing only the site left the application pointing at a deleted directory, so the parent site - at West Jefferson, the live classic ASP - served 503 on that path forever while Add/Remove Programs reported success. Uninstall now reads MOUNT_PATH and removes the application. The firewall rule was created as "$SiteName $SitePort" and removed as the literal 'ShopDB-Flask 8090', which matches nothing. DAY-2 TOOLING. Every shortcut now passes -AppRoot and -SitePort, and the console forwards them through its own elevation and 32-bit relaunches instead of discarding them - a non-default directory or port made it report a healthy site as broken, from a shortcut the installer wrote. 'Open ShopDB-Flask' resolved to a hardcoded localhost:8090 that was wrong for every subpath install; it now asks the console, which reads the address the installer recorded, and no longer demands administrator to open a browser. SMOKE TEST. The parent-site port lookup filtered for an http binding and defaulted to 80, so an https-only parent site failed a working install with a red dialog. DOCS AND /api/docs. The installer was invisible: nothing in docs/, README.md or CLAUDE.md mentioned it, so a DT lead or their assistant landed on the manual IIS runbook and hand-built the very server the installer then refuses to upgrade. docs/INSTALL-WINDOWS.md and docs/OPERATE-WINDOWS.md are now the canonical route, the two manual runbooks are bannered as reference-only, README and CLAUDE.md route by target, and llms.txt tells an assistant which document to follow and to ask for 'check -Json' before diagnosing. Both ship on the server, along with openapi.json and llms.txt - without those the self-hosted /api/docs was broken on every installed box, which matters most to the sites least able to debug it. Stage 5 now checks it actually serves. shopdb-admin.ps1 gains 'check -Json': one structured, secret-free block covering version, publishing method, IIS state, HTTP reachability, database, Python version, plugins and errors. That is the cheapest useful answer to 'the operator will ask an LLM' - it works with no infrastructure, which a install-time MCP server could not. --- CLAUDE.md | 1 + README.md | 11 +- deploy/windows/installer/ShopDBFlask.iss | 256 +++++++++++++----- deploy/windows/installer/build-installer.ps1 | 13 + deploy/windows/installer/bundle-lock.ps1 | 1 + deploy/windows/installer/shopdb-admin.ps1 | 132 ++++++++- deploy/windows/installer/shopdb-install.ps1 | 154 ++++++++++- deploy/windows/installer/shopdb-preflight.ps1 | 25 ++ .../windows/installer/verify_bundle_lock.py | 1 + docs/BACKUP-RESTORE.md | 32 +++ docs/DEPLOY-WINDOWS-IIS.md | 8 + docs/INSTALL-WINDOWS-IIS.md | 8 + docs/INSTALL-WINDOWS.md | 201 ++++++++++++++ docs/OPERATE-WINDOWS.md | 195 +++++++++++++ docs/UPGRADE.md | 24 ++ docs/llms.txt | 23 ++ scripts/build-site.sh | 16 ++ 17 files changed, 1004 insertions(+), 97 deletions(-) create mode 100644 docs/INSTALL-WINDOWS.md create mode 100644 docs/OPERATE-WINDOWS.md diff --git a/CLAUDE.md b/CLAUDE.md index 846c7fd..4a4bc3b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,6 +50,7 @@ Refactor phases 0-5 landed; phase 6 (multi-site distribution readiness) largely - 13 bundled plugins all satisfy contract: computers, employees, geenforce, knowledgebase, machines, measuringtools, network, notifications, printedparts, printers, slides, usb, warranty - Core Alembic chain: baseline `68b3947ae14f` -> head `7d26_settings_description_text` (33 core migrations). Each plugin owns its own chain (ADR-008); deploy runs `flask db upgrade` then `flask plugin upgrade-all`. Reproducible + idempotent from empty (env.py relaxes session sql_mode so the chain runs on strict MySQL 8). - Lean per-site builds (ADR-013 + ADR-014): `scripts/build-site.sh` (backend) + `SITE_PLUGINS` via `scripts/stage-frontend.mjs` (frontend) ship only chosen plugins; `flask plugin prune-schema` drops non-installed plugins' tables at provisioning. Sidebar nav / settings / Displays all gate on staged routes. Manifest-less `plugins//frontend/` dirs (e.g. `applications`) are core and always ship. +- Windows sites install from a single air-gapped installer `.exe` built per site from its plugin profile (`deploy/windows/installer/`, built by `build-installer.sh` or `build-installer.ps1`). Operator docs: `docs/INSTALL-WINDOWS.md` + `docs/OPERATE-WINDOWS.md` - these are canonical for a NEW site. `docs/INSTALL-WINDOWS-IIS.md` and `docs/DEPLOY-WINDOWS-IIS.md` are the MANUAL procedure, kept for hand-built servers only. The installer verifies its third-party payload against `bundle-lock.json` and installs wheels with `pip --require-hashes`; every build stages a CycloneDX SBOM (`sbom.cdx.json`) onto the server. - Legacy import: `docs/IMPORT-API.md` is the schema-agnostic import contract; `docs/IMPORT-ADOPTION.md` + `docs/PILOT-DEPLOY.md` cover adopting a site; `scripts/site_imports/wjf/` is the West Jefferson reference loader (all 16 stages, validated end-to-end including on a Windows + MySQL 8 VM). - API is migration-complete: an admin PAT + docs/IMPORT-API.md let a script import the whole legacy DB (X-Import-Mode preserves timestamps). - Pre-1.0 framework; sister sites should pin tight `core_version` ranges until contract reaches 1.0 diff --git a/README.md b/README.md index 378ac33..b0535ba 100644 --- a/README.md +++ b/README.md @@ -174,8 +174,15 @@ token plus [docs/IMPORT-API.md](docs/IMPORT-API.md) drive the whole migration through documented endpoints (`X-Import-Mode` preserves original timestamps). `scripts/site_imports/wjf/` is the West Jefferson reference loader. -For the full per-site deployment runbook see [docs/DEPLOY.md](docs/DEPLOY.md); -for every environment variable and Setting key see [docs/CONFIG.md](docs/CONFIG.md). +### Which deployment route + +| Target | Use | +|---|---| +| **Windows Server + IIS** (how sister sites run) | **[docs/INSTALL-WINDOWS.md](docs/INSTALL-WINDOWS.md)** - one installer `.exe`, offline, no manual IIS work. Day 2: [docs/OPERATE-WINDOWS.md](docs/OPERATE-WINDOWS.md) | +| Linux / Docker, air-gapped | [docs/DEPLOY-AIRGAP.md](docs/DEPLOY-AIRGAP.md) | +| Linux / Docker, connected | [docs/DEPLOY.md](docs/DEPLOY.md) | + +For every environment variable and Setting key see [docs/CONFIG.md](docs/CONFIG.md). ## Configuration diff --git a/deploy/windows/installer/ShopDBFlask.iss b/deploy/windows/installer/ShopDBFlask.iss index c23daab..17c81f7 100644 --- a/deploy/windows/installer/ShopDBFlask.iss +++ b/deploy/windows/installer/ShopDBFlask.iss @@ -93,7 +93,10 @@ Name: "english"; MessagesFile: "compiler:Default.isl" WelcomeLabel1=Set up ShopDB-Flask on this server ; Short, and says the two things an operator actually wants up front: it will not ; need the internet, and it will tell them before it changes anything. -WelcomeLabel2=Everything needed is included in this installer. No internet connection is used at any point.%n%nYou will be asked four short questions, and nothing on this server is changed until you reach the final confirmation.%n%nThe next page checks that this server is ready. It only reads - it changes nothing. +; The installer supplies Python, the wheels, the IIS modules and optionally MySQL. +; It does NOT install the IIS Web Server role itself - that is a Windows feature, +; and the check on the next page confirms it is present rather than adding it. +WelcomeLabel2=Everything this application needs is included - Python, its packages, the IIS modules and optionally MySQL. No internet connection is used at any point.%n%nThis server must already have the IIS Web Server role installed. The next page checks that, and everything else this needs, before anything is changed.%n%nYou will then be asked a few short questions. Nothing on this server is changed until the final confirmation. ClickNext=Click Next to check this server. FinishedHeadingLabel=ShopDB-Flask is ready SetupAppTitle=ShopDB-Flask Setup @@ -131,18 +134,28 @@ Source: "swatch-bad.bmp"; Flags: dontcopy [Icons] ; A folder rather than loose icons: this is a server tool, not a desktop app. +; +; EVERY shortcut passes -AppRoot and -SitePort. The console defaults to +; C:\shopdb-flask and port 8090, so on any install that chose a different +; directory or port it looked in the wrong place and reported a perfectly healthy +; site as broken - from a Start Menu shortcut the installer wrote itself. Name: "{group}\ShopDB-Flask Console"; Filename: "powershell.exe"; \ - Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"""; \ + Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" -AppRoot ""{app}"" -SitePort {code:SitePortValue}"; \ WorkingDir: "{app}"; IconFilename: "{app}\shopdb.ico"; \ Comment: "Status, restart, logs and backups" -Name: "{group}\Open ShopDB-Flask"; Filename: "http://localhost:8090/login"; \ - IconFilename: "{app}\shopdb.ico"; Comment: "Open the application in a browser" +; Resolves the address from .env at click time rather than baking one in. The +; literal http://localhost:8090/login this used to carry was wrong for every +; subpath install and every non-default port, with no operator mistake involved. +Name: "{group}\Open ShopDB-Flask"; Filename: "powershell.exe"; \ + Parameters: "-NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File ""{app}\shopdb-admin.ps1"" open -AppRoot ""{app}"" -SitePort {code:SitePortValue}"; \ + WorkingDir: "{app}"; IconFilename: "{app}\shopdb.ico"; \ + Comment: "Open the application in a browser" Name: "{group}\Restart ShopDB-Flask"; Filename: "powershell.exe"; \ - Parameters: "-NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" restart"; \ + Parameters: "-NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" restart -AppRoot ""{app}"" -SitePort {code:SitePortValue}"; \ WorkingDir: "{app}"; IconFilename: "{app}\shopdb.ico"; \ Comment: "Recycle the application pool" Name: "{group}\Back up the database"; Filename: "powershell.exe"; \ - Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" backup"; \ + Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" backup -AppRoot ""{app}"" -SitePort {code:SitePortValue}"; \ WorkingDir: "{app}"; IconFilename: "{app}\shopdb.ico"; \ Comment: "Write a .sql dump to ProgramData" @@ -155,7 +168,7 @@ Name: "{group}\Back up the database"; Filename: "powershell.exe"; \ ; System32 from a 32-bit process. The script also self-corrects, but launching ; it correctly avoids a visible relaunch. Filename: "{win}\Sysnative\WindowsPowerShell\v1.0\powershell.exe"; \ - Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"""; \ + Parameters: "-NoExit -NoProfile -ExecutionPolicy Bypass -File ""{app}\shopdb-admin.ps1"" -AppRoot ""{app}"" -SitePort {code:SitePortValue}"; \ Description: "Open the ShopDB-Flask console"; \ Flags: postinstall skipifsilent unchecked @@ -171,6 +184,12 @@ var BannerSub: TNewStaticText; DetailPanel: TPanel; FooterText: TNewStaticText; + RecheckButton: TNewButton; + // Set by the preflight render. The wizard REFUSES to leave the results page + // while this is True: the check used to be advisory, so a server missing IIS + // sailed through every page and failed in the middle of installing, having + // already put Python on the box. + PreflightBlocked: Boolean; PreflightDone: Boolean; // Shown while the preflight runs. Without it, clicking Next appears to hang: // the check takes a few seconds, runs hidden, and gives no sign of life. @@ -268,6 +287,11 @@ begin Result.Bitmap.LoadFromFile(ExpandConstant('{tmp}\') + Swatch); end; +// Defined further down, next to the preflight rendering they drive. Declared here +// because InitializeWizard wires RecheckClick to a button before that point. +procedure RunPreflight; forward; +procedure RecheckClick(Sender: TObject); forward; + procedure InitializeWizard; var I: Integer; @@ -324,10 +348,22 @@ begin FooterText.Parent := PreflightPage.Surface; FooterText.Left := 0; FooterText.Top := PreflightPage.SurfaceHeight - ScaleY(16); - FooterText.Width := PreflightPage.SurfaceWidth; + FooterText.Width := PreflightPage.SurfaceWidth - ScaleX(84); FooterText.AutoSize := False; FooterText.Caption := ''; + // A blocking page needs a way forward that is not "cancel the installer". The + // operator fixes what the page told them to fix - installs the IIS role, frees + // the port - and re-checks without starting over. + RecheckButton := TNewButton.Create(WizardForm); + RecheckButton.Parent := PreflightPage.Surface; + RecheckButton.Width := ScaleX(78); + RecheckButton.Height := ScaleY(23); + RecheckButton.Left := PreflightPage.SurfaceWidth - RecheckButton.Width; + RecheckButton.Top := PreflightPage.SurfaceHeight - ScaleY(23); + RecheckButton.Caption := 'Check again'; + RecheckButton.OnClick := @RecheckClick; + CheckingPage := CreateOutputProgressPage('Checking this server', 'Reading the current configuration. Nothing is being changed.'); @@ -664,6 +700,94 @@ begin FooterText.Caption := IntToStr(Passes) + ' checks passed. Nothing has been changed on this server.'; end; +// Run the read-only check and paint its results. Called from Next on the welcome +// page, and again whenever the operator fixes something and presses Check again. +// +// Stage 1 changes nothing, so running it before the operator commits to anything +// is free, and it catches the blockers that are expensive later: IIS absent, +// locked config sections, the port in use, MySQL 5.6 missing its index flags. +procedure RunPreflight; +var + I, ResultCode: Integer; + Output: TExecOutput; + PreflightScript, Detail: String; + HasBlockers: Boolean; +begin + // The bundle is NOT extracted yet at wizard time - [Files] is processed during + // the install step. Pull just this one script out of the compressed payload. + ExtractTemporaryFile('shopdb-preflight.ps1'); + PreflightScript := ExpandConstant('{tmp}\shopdb-preflight.ps1'); + + // Visible feedback for the few seconds the check takes. SW_HIDE means the + // operator sees nothing at all otherwise, and a frozen wizard reads as a + // crash. ProgressBar.Style := npbstMarquee because the duration is unknown - + // a bar that sits at 0% is worse than no bar. + CheckingPage.SetText('Looking at IIS, disk, ports and the database...', ''); + CheckingPage.SetProgress(0, 0); + CheckingPage.ProgressBar.Style := npbstMarquee; + CheckingPage.Show; + try + // ExecAndCaptureOutput (6.4.0+) hands back stdout and stderr as string arrays, + // so output never touches disk. That deletes BOTH bugs this page shipped with: + // - Exec() has no shell, so "> file" went to PowerShell as a literal + // argument and no file was ever written (page came up empty); + // - PowerShell 5.1 writes UTF-16LE, and LoadStringFromFile reads bytes as + // ANSI, so the page rendered as garbage with a null between characters. + // Neither failure mode can recur now: there is no file and no encoding step. + // 64-bit PowerShell - see PowerShellPath. The IIS checks fail without it. + ExecAndCaptureOutput(PowerShellPath, + '-NoProfile -ExecutionPolicy Bypass -File "' + PreflightScript + '" -Delimited', + '', SW_HIDE, ewWaitUntilTerminated, ResultCode, Output); + + // Every run repaints from scratch. The old code rendered ONCE and latched, + // so a re-check could not have shown a different answer even if the operator + // had fixed everything. + while DetailPanel.ControlCount > 0 do + DetailPanel.Controls[0].Free; + + PreflightDone := True; + if (not Output.Error) and (GetArrayLength(Output.StdOut) > 0) then + begin + RenderPreflight(Output.StdOut, HasBlockers); + PreflightBlocked := HasBlockers; + end + else + begin + // stderr is captured SEPARATELY, so a failure can report what actually + // went wrong instead of only an exit code. + Detail := ''; + for I := 0 to GetArrayLength(Output.StdErr) - 1 do + Detail := Detail + Output.StdErr[I] + ' '; + if Trim(Detail) = '' then Detail := 'No error output was produced.'; + SetSwatch(BannerBar, 'swatch-warn.bmp'); + BannerText.Caption := 'Could not check this server'; + BannerSub.Caption := 'Nothing has been verified. Continuing is a risk.'; + AddRow(ScaleY(4), 'swatch-warn.bmp', + 'Check did not run (exit ' + IntToStr(ResultCode) + ')', Detail); + FooterText.Caption := 'Nothing has been changed on this server.'; + // A check that could not RUN is not a check that PASSED. It does not block - + // there is no evidence of a problem - but it must not read like a green light. + PreflightBlocked := False; + end; + finally + CheckingPage.Hide; + end; +end; + +procedure RecheckClick(Sender: TObject); +begin + RunPreflight; +end; + +// The port the operator chose, for the [Icons]/[Run] entries. A subpath install +// is reached on the parent site's port, and the console works that out from +// MOUNT_PATH in .env, so the value only has to be right for the own-site case. +function SitePortValue(Param: String): String; +begin + Result := SitePage.Values[1]; + if Trim(Result) = '' then Result := '8090'; +end; + // Where an EXISTING install would be, for the pre-fill reads. // // Not WizardDirValue() alone: these pages are inserted after wpWelcome and so @@ -681,75 +805,26 @@ begin end; function NextButtonClick(CurPageID: Integer): Boolean; -var - ResultCode, I: Integer; - Output: TExecOutput; - PreflightScript, Detail: String; - HasBlockers: Boolean; begin Result := True; if CurPageID = wpWelcome then + RunPreflight; + + // The results page BLOCKS while anything is failing. It used to be advisory: + // the operator read "IIS is not installed", pressed Next, answered five more + // pages, and the install then died partway through with Python already on the + // box. Refusing here costs them nothing - the server is untouched at this + // point - and the page already says what to do about each failure. + if CurPageID = PreflightPage.ID then begin - // Stage 1 is read-only, so running it before the operator commits to - // anything is free and catches the common blockers (IIS absent, locked - // config sections, port in use, MySQL 5.6 missing its index flags). - // The bundle is NOT extracted yet at wizard time - [Files] is processed during - // the install step. Pull just this one script out of the compressed payload. - ExtractTemporaryFile('shopdb-preflight.ps1'); - PreflightScript := ExpandConstant('{tmp}\shopdb-preflight.ps1'); - - // Visible feedback for the few seconds the check takes. SW_HIDE means the - // operator sees nothing at all otherwise, and a frozen wizard reads as a - // crash. ProgressBar.Style := npbstMarquee because the duration is unknown - - // a bar that sits at 0% is worse than no bar. - CheckingPage.SetText('Looking at IIS, disk, ports and the database...', ''); - CheckingPage.SetProgress(0, 0); - CheckingPage.ProgressBar.Style := npbstMarquee; - CheckingPage.Show; - try - - // ExecAndCaptureOutput (6.4.0+) hands back stdout and stderr as string arrays, - // so output never touches disk. That deletes BOTH bugs this page shipped with: - // - Exec() has no shell, so "> file" went to PowerShell as a literal - // argument and no file was ever written (page came up empty); - // - PowerShell 5.1 writes UTF-16LE, and LoadStringFromFile reads bytes as - // ANSI, so the page rendered as garbage with a null between characters. - // Neither failure mode can recur now: there is no file and no encoding step. - // 64-bit PowerShell - see PowerShellPath. The IIS checks fail without it. - ExecAndCaptureOutput(PowerShellPath, - '-NoProfile -ExecutionPolicy Bypass -File "' + PreflightScript + '" -Delimited', - '', SW_HIDE, ewWaitUntilTerminated, ResultCode, Output); - // Build the rows once. Going Back and Next again must not stack a second set - // of controls on top of the first. - if not PreflightDone then + if PreflightBlocked then begin - PreflightDone := True; - if (not Output.Error) and (GetArrayLength(Output.StdOut) > 0) then - begin - RenderPreflight(Output.StdOut, HasBlockers); - if HasBlockers then - MsgBox('This server is not ready. The problems are listed on the next ' - + 'page, each with what to do about it.', mbError, MB_OK); - end - else - begin - // stderr is captured SEPARATELY, so a failure can report what actually - // went wrong instead of only an exit code. - Detail := ''; - for I := 0 to GetArrayLength(Output.StdErr) - 1 do - Detail := Detail + Output.StdErr[I] + ' '; - if Trim(Detail) = '' then Detail := 'No error output was produced.'; - SetSwatch(BannerBar, 'swatch-warn.bmp'); - BannerText.Caption := 'Could not check this server'; - BannerSub.Caption := 'Installation can continue, but nothing has been verified.'; - AddRow(ScaleY(4), 'swatch-warn.bmp', - 'Check did not run (exit ' + IntToStr(ResultCode) + ')', Detail); - FooterText.Caption := 'Nothing has been changed on this server.'; - end; - end; - finally - CheckingPage.Hide; + MsgBox('This server is not ready yet.' + #13#10#13#10 + + 'Each problem above says what to do about it. Fix them, then choose ' + + '"Check again".' + #13#10#13#10 + + 'Nothing has been changed on this server.', mbError, MB_OK); + Result := False; end; end; @@ -992,8 +1067,19 @@ begin else Result := 'The installation could not be completed (exit ' + IntToStr(ResultCode) + ').'; + // NOT "nothing was left running". The stages run with -OnFailure never, so + // nothing is rolled back: whatever had been created by the failing point is + // still there. Claiming otherwise sent operators away believing the server + // was clean when it was half-configured, and the next thing they did was + // install again on top of it. Result := Result + #13#10#13#10 - + 'Nothing was left running. Full details are in:' + #13#10 + + 'This server has been part-configured. Whatever had been done ' + + 'before the failure is still in place.' + #13#10#13#10 + + 'What to do:' + #13#10 + + ' - Fix the cause above, then run this installer again. Re-running ' + + 'is safe and picks up where it left off.' + #13#10 + + ' - Or remove it entirely from Settings > Apps.' + #13#10#13#10 + + 'Full details, including everything that was created, are in:' + #13#10 + ExpandConstant('{log}'); end; end; @@ -1021,14 +1107,36 @@ begin // "Unknown preprocessor directive" before Pascal parsing happens. Keep the // concatenation operator at the start of continuation lines instead. if Failure <> '' then - MsgBox(Failure, mbCriticalError, MB_OK) + begin + MsgBox(Failure, mbCriticalError, MB_OK); + // The final page is headed "ShopDB-Flask is ready" from [Messages]. After a + // failed install that is the last thing the operator reads, and it + // contradicts the error box they just dismissed. Setup cannot be made to + // fail from here, so at least stop it claiming success. + WizardForm.FinishedHeadingLabel.Caption := 'ShopDB-Flask is NOT installed'; + WizardForm.FinishedLabel.Caption := + 'The installation did not complete. This server has been part-configured.' + + #13#10#13#10 + + 'Fix the problem reported above and run this installer again - re-running ' + + 'is safe. Or remove it from Settings > Apps.' + + #13#10#13#10 + + 'The log is at ' + ExpandConstant('{log}'); + end else + begin + // Recorded so the Start Menu shortcut and the console open the address this + // install actually serves, rather than a compile-time guess. + SaveStringToFile(ExpandConstant('{app}\.installed-url'), FinalUrl, False); MsgBox('ShopDB-Flask is installed.' + #13#10#13#10 + 'Open ' + FinalUrl + #13#10#13#10 + 'With no user in the database that page offers to create the first ' - + 'administrator and then runs the setup wizard.', + + 'administrator and then runs the setup wizard.' + + #13#10#13#10 + + 'Day-to-day: use the ShopDB-Flask Console in the Start Menu, or read ' + + ExpandConstant('{app}\docs\OPERATE-WINDOWS.md'), mbInformation, MB_OK); + end; end; end; diff --git a/deploy/windows/installer/build-installer.ps1 b/deploy/windows/installer/build-installer.ps1 index ccef0db..5a0f672 100644 --- a/deploy/windows/installer/build-installer.ps1 +++ b/deploy/windows/installer/build-installer.ps1 @@ -175,6 +175,19 @@ Step 'Generating SBOM' -o (Join-Path $AppOut 'sbom.cdx.json') | ForEach-Object { Say " $_" 'White' } if ($LASTEXITCODE -ne 0) { Die 'SBOM generation failed' } +# Docs the running site serves, plus the runbooks an air-gapped server has no +# other way to reach. Without openapi.json and llms.txt the self-hosted /api/docs +# page is broken on every installed server. +Step 'Staging docs' +$docsOut = Join-Path $AppOut 'docs' +New-Item -ItemType Directory -Path $docsOut -Force | Out-Null +foreach ($doc in @('openapi.json', 'llms.txt', 'api-inventory.json', + 'INSTALL-WINDOWS.md', 'OPERATE-WINDOWS.md', + 'BACKUP-RESTORE.md', 'UPGRADE.md')) { + $src = Join-Path $RepoRoot ('docs\' + $doc) + if (Test-Path $src) { Copy-Item $src $docsOut -Force; Say " $doc" } +} + # Stage the profile INTO the tree: `flask plugin apply-profile` at provisioning # reads the same profile the tree was staged from, so the installed plugin set # and the shipped plugin code cannot drift. diff --git a/deploy/windows/installer/bundle-lock.ps1 b/deploy/windows/installer/bundle-lock.ps1 index a1e862f..7cb317e 100644 --- a/deploy/windows/installer/bundle-lock.ps1 +++ b/deploy/windows/installer/bundle-lock.ps1 @@ -49,6 +49,7 @@ $script:BundlePayloads = @( @{ Name = 'python'; Required = $true; What = 'the Python installer' } @{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' } @{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' } + @{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' } @{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' } ) diff --git a/deploy/windows/installer/shopdb-admin.ps1 b/deploy/windows/installer/shopdb-admin.ps1 index 03d783b..c616fea 100644 --- a/deploy/windows/installer/shopdb-admin.ps1 +++ b/deploy/windows/installer/shopdb-admin.ps1 @@ -25,6 +25,11 @@ param( 'check','sessions','plugins','add-plugin','verify','uninstall')] [string] $Command = 'menu', [string] $Path = '', + # Machine-readable output for 'check'. The people running this are expected to + # ask an AI assistant for help, and pasting a screenshot of a console into a + # chat window loses most of what matters. One structured blob they can paste + # gives the assistant real state to reason about instead of guesses. + [switch] $Json, [string] $AppRoot = 'C:\shopdb-flask', [string] $SiteName = 'shopdb-flask', [string] $AppPool = 'shopdbflask', @@ -46,13 +51,27 @@ $AppCmd = Join-Path $env:windir 'System32\inetsrv\appcmd.exe' # than fix every launcher, relaunch under the native PowerShell. 'Sysnative' is # the alias that lets a 32-bit process reach the real System32, and it exists # ONLY for 32-bit processes - hence the guard. + +# Every relaunch below must carry the ORIGINAL arguments through. They used to be +# dropped, so a console started from the Start Menu with -AppRoot D:\shopdb +# relaunched itself with the C:\shopdb-flask default and reported a healthy +# install as missing - and the operator had done nothing wrong. +function Get-ForwardedArgs { + $forward = @('-NoProfile', '-ExecutionPolicy', 'Bypass', + '-File', ('"' + $PSCommandPath + '"'), $Command) + if ($Path) { $forward += @('-Path', ('"' + $Path + '"')) } + if ($AppRoot) { $forward += @('-AppRoot', ('"' + $AppRoot + '"')) } + if ($SiteName) { $forward += @('-SiteName', ('"' + $SiteName + '"')) } + if ($AppPool) { $forward += @('-AppPool', ('"' + $AppPool + '"')) } + if ($SitePort) { $forward += @('-SitePort', $SitePort) } + if ($Json) { $forward += '-Json' } + return $forward +} + if ([Environment]::Is64BitOperatingSystem -and -not [Environment]::Is64BitProcess) { $native = Join-Path $env:windir 'Sysnative\WindowsPowerShell\v1.0\powershell.exe' if (Test-Path $native) { - $relaunch = @('-NoProfile', '-ExecutionPolicy', 'Bypass', - '-File', ('"' + $PSCommandPath + '"'), $Command) - if ($Path) { $relaunch += @('-Path', ('"' + $Path + '"')) } - Start-Process -FilePath $native -ArgumentList $relaunch -Wait -NoNewWindow + Start-Process -FilePath $native -ArgumentList (Get-ForwardedArgs) -Wait -NoNewWindow return } } @@ -65,13 +84,14 @@ if ([Environment]::Is64BitOperatingSystem -and -not [Environment]::Is64BitProces # instead of reporting a false state. $identity = [Security.Principal.WindowsIdentity]::GetCurrent() $principal = New-Object Security.Principal.WindowsPrincipal($identity) -if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { +# 'open' just launches a browser. Prompting for administrator to do that trains +# people to click through UAC, and the Start Menu shortcut uses this command. +if ($Command -ne 'open' -and + -not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { Write-Host '' Write-Host ' Administrator rights are needed to read IIS state.' -ForegroundColor Yellow Write-Host ' Re-launching elevated - approve the prompt.' -ForegroundColor Yellow - $argList = @('-NoExit', '-NoProfile', '-ExecutionPolicy', 'Bypass', - '-File', ('"' + $PSCommandPath + '"'), $Command) - if ($Path) { $argList += @('-Path', ('"' + $Path + '"')) } + $argList = @('-NoExit') + (Get-ForwardedArgs) try { Start-Process -FilePath 'powershell.exe' -ArgumentList $argList -Verb RunAs | Out-Null } catch { @@ -326,6 +346,14 @@ function Show-Logs { } function Open-Site { + # The installer records the address it actually published at. Prefer it: this + # command runs unelevated, and .env is ACL'd, so working the address out from + # MOUNT_PATH and IIS may not be possible from here. + $recorded = Join-Path $AppRoot '.installed-url' + if (Test-Path $recorded) { + $u = (Get-Content $recorded -TotalCount 1).Trim() + if ($u) { Start-Process $u; Say " opened $u" 'Green'; return } + } $deploy = Get-Deployment if ($deploy.Subpath) { if ($deploy.Port -eq 80) { $u = "http://{0}/{1}/login" -f $env:COMPUTERNAME, $deploy.Alias } @@ -334,7 +362,7 @@ function Open-Site { $u = "http://{0}:{1}/login" -f $env:COMPUTERNAME, $SitePort } Start-Process $u - Say ' opened in your browser' 'Green' + Say " opened $u" 'Green' } function Backup-Db { @@ -385,7 +413,90 @@ function Backup-Db { } } +function Get-CheckState { + <# + Everything an outside reader needs to reason about this server, gathered + without changing anything. Secrets are NEVER included: the database password + lives in DATABASE_URL and this reports the host, port, name and user only. + #> + $deploy = Get-Deployment + $db = Get-DbParts + $state = [ordered]@{ + collected = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') + computername = $env:COMPUTERNAME + approot = $AppRoot + installed = (Test-Path (Join-Path $AppRoot 'shopdb\__init__.py')) + version = '' + publishedas = if ($deploy.Subpath) { "subpath /$($deploy.Alias) under '$($deploy.Parent)'" } else { "own site '$SiteName' on port $SitePort" } + baseurl = $deploy.BaseUrl + apppool = 'unknown' + poolstate = 'unknown' + sitestate = 'unknown' + responding = $false + httpstatus = 0 + database = $null + pythonversion = '' + plugins = @() + sbomcomponents = 0 + errors = @() + } + foreach ($pair in @(@('version', '.installed-version'))) { + $f = Join-Path $AppRoot $pair[1] + if (Test-Path $f) { $state[$pair[0]] = (Get-Content $f -TotalCount 1).Trim() } + } + $state.apppool = $AppPool + try { + Import-Module WebAdministration -ErrorAction Stop + if (Test-Path "IIS:\AppPools\$AppPool") { $state.poolstate = (Get-Item "IIS:\AppPools\$AppPool").State.ToString() } + if ($deploy.Subpath) { + $app = Get-WebApplication -Site $deploy.Parent -Name $deploy.Alias -ErrorAction SilentlyContinue + $state.sitestate = if ($app) { 'application present' } else { 'application MISSING' } + } else { + $s = Get-Website -Name $SiteName -ErrorAction SilentlyContinue + $state.sitestate = if ($s) { $s.State.ToString() } else { 'site MISSING' } + } + } catch { $state.errors += "IIS: $($_.Exception.Message)" } + + try { + $r = Invoke-WebRequest -Uri ($deploy.BaseUrl + '/') -UseBasicParsing -TimeoutSec 20 + $state.responding = $true + $state.httpstatus = [int] $r.StatusCode + } catch { $state.errors += "HTTP: $($_.Exception.Message)" } + + if ($db) { + $state.database = [ordered]@{ host = $db.Host; port = $db.Port; name = $db.Name; user = $db.User; reachable = $false } + $mysql = Find-MysqlClient + if ($mysql) { + $q = "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='$($db.Name)';" + $n = $q | & $mysql "-u$($db.User)" "-p$($db.Pass)" "-h$($db.Host)" "-P$($db.Port)" -N 2>$null + if ($LASTEXITCODE -eq 0) { $state.database.reachable = $true; $state.database.tables = [int] $n } + } else { $state.errors += 'no mysql client found, database not probed' } + } else { $state.errors += 'no .env found' } + + $py = Join-Path $AppRoot 'venv\Scripts\python.exe' + if (Test-Path $py) { $state.pythonversion = (& $py -c "import sys; print('%d.%d.%d' % sys.version_info[:3])" 2>$null | Select-Object -First 1) } + + $dir = Join-Path $AppRoot 'plugins' + if (Test-Path $dir) { + $state.plugins = @(Get-ChildItem $dir -Directory -EA SilentlyContinue | + Where-Object { Test-Path (Join-Path $_.FullName 'manifest.json') } | + Select-Object -ExpandProperty Name) + } + $sbom = Join-Path $AppRoot 'sbom.cdx.json' + if (Test-Path $sbom) { + try { $state.sbomcomponents = (Get-Content $sbom -Raw | ConvertFrom-Json).components.Count } catch { } + } + return $state +} + function Invoke-Check { + if ($Json) { + # ONLY the JSON goes to stdout, so it can be redirected to a file or piped + # without a banner in the middle of the document. + Get-CheckState | ConvertTo-Json -Depth 6 + return + } + Head 'Health check' $flask = Join-Path $AppRoot 'venv\Scripts\flask.exe' if (-not (Test-Path $flask)) { Say ' application not installed' 'Red'; return } @@ -393,6 +504,9 @@ function Invoke-Check { $env:FLASK_APP = 'shopdb' try { & $flask db-utils preflight 2>&1 | ForEach-Object { Say " $_" } } finally { Pop-Location } + Say '' + Say ' For help from an AI assistant, paste the output of:' 'DarkGray' + Say ' shopdb-admin.ps1 check -Json' 'White' } function Show-Sessions { diff --git a/deploy/windows/installer/shopdb-install.ps1 b/deploy/windows/installer/shopdb-install.ps1 index 5c46b35..8125f79 100644 --- a/deploy/windows/installer/shopdb-install.ps1 +++ b/deploy/windows/installer/shopdb-install.ps1 @@ -396,6 +396,19 @@ function Compare-Version { function Find-MysqlTool { param([string] $Name) # mysql.exe or mysqldump.exe + # The BUNDLE first, then the install directory, then the local server. + # + # A site whose MySQL lives on another host has no client installed here at + # all, so mysqldump was never found - which meant every upgrade skipped the + # pre-upgrade backup, and it did so AFTER stage 2 had stopped the pool and + # replaced the application tree. Shipping the client makes the backup work + # on a remote-database site, which is the case that needs it most. + $bundled = @( + (Join-Path $BundleRoot ('mysqlclient\' + $Name)), + (Join-Path $AppRoot ('mysqlclient\' + $Name)) + ) + foreach ($b in $bundled) { if (Test-Path $b) { return $b } } + $roots = @('C:\Program Files\MySQL', 'C:\mysql56\bin', 'C:\Program Files (x86)\MySQL') foreach ($r in $roots) { if (Test-Path $r) { @@ -468,8 +481,14 @@ function Backup-Database { $dump = Find-MysqlTool 'mysqldump.exe' if (-not $dump) { Write-Log 'mysqldump not found; skipping backup' 'WARN'; return '' } + # A dump contains every row, including the users table and its password + # hashes. ProgramData is readable by every user on the box by default, so the + # directory is locked to Administrators and SYSTEM the moment it is created. $dir = Join-Path $env:ProgramData 'ShopDB-Flask\backups' - if (-not (Test-Path $dir)) { New-Item -ItemType Directory -Path $dir -Force | Out-Null } + if (-not (Test-Path $dir)) { + New-Item -ItemType Directory -Path $dir -Force | Out-Null + Protect-File $dir + } $file = Join-Path $dir ("{0}-{1}-{2}.sql" -f $db.Name, $Reason, (Get-Date -Format 'yyyyMMdd-HHmmss')) Write-Log "backing up $($db.Name) before migrating" @@ -749,11 +768,28 @@ FLUSH PRIVILEGES; Write-Log "database $DbName and user $DbUser created" 'OK' Write-Log "app password written to $handoff (stage 2 consumes and deletes it)" - Write-Host '' - Write-Host ' MySQL root password (shown ONCE, not stored anywhere):' -ForegroundColor Yellow - Write-Host (" {0}" -f $rootPass) -ForegroundColor Yellow - Write-Host ' Write it down now. It cannot be recovered.' -ForegroundColor Yellow - Write-Host '' + # NOT Write-Host. Under the wizard this process runs hidden, so nobody ever + # saw this - and every line of stdout is forwarded into the setup log that + # operators are told to send to support. The generated root password was + # therefore invisible to the person who needed it and permanently recorded + # for everyone who did not. + # + # Write it to a file only Administrators and SYSTEM can read, and let the + # wizard tell the operator where it is. + $rootFile = Join-Path $env:ProgramData 'ShopDB-Flask\mysql-root-password.txt' + $rootDir = Split-Path $rootFile -Parent + if (-not (Test-Path $rootDir)) { New-Item -ItemType Directory -Path $rootDir -Force | Out-Null } + New-Item -ItemType File -Force -Path $rootFile | Out-Null + Protect-File $rootFile + Set-Content -Path $rootFile -Value @( + 'MySQL root password for this server, generated during installation.', + 'It is not recorded anywhere else and cannot be recovered.', + 'Move it into your password manager, then delete this file.', + '', + $rootPass + ) -Encoding UTF8 + Write-Log "MySQL root password written to $rootFile (Administrators and SYSTEM only)" 'OK' + Write-Log 'MYSQLROOTFILE:' + $rootFile Write-Log 'stage 0 complete' 'OK' } @@ -900,6 +936,17 @@ a page that cannot load its own assets. Rebuild with scripts/build-site.sh if (-not $WhatIfOnly) { $bundledLock = Join-Path $BundleRoot 'bundle-lock.json' if (Test-Path $bundledLock) { Copy-Item $bundledLock $AppRoot -Force } + # The MySQL client stays on the server. The bundle is extracted to a temp + # directory and deleted when Setup exits, so a copy that lived only there + # would leave `shopdb-admin backup` with nothing to run on a site whose + # database is on another host. + $bundledClient = Join-Path $BundleRoot 'mysqlclient' + if (Test-Path $bundledClient) { + $target = Join-Path $AppRoot 'mysqlclient' + if (-not (Test-Path $target)) { New-Item -ItemType Directory -Path $target -Force | Out-Null } + Copy-Item (Join-Path $bundledClient '*') $target -Recurse -Force + Write-Log 'MySQL client staged for backups' 'OK' + } } foreach ($sub in @('logs','instance')) { $p = Join-Path $AppRoot $sub @@ -1659,15 +1706,44 @@ function Invoke-Stage5 { # requesting http://localhost:8090/ would test a site that does not exist. if ($MountAlias) { $alias = $MountAlias.Trim('/') - $parentPort = 80 + # Prefer an http binding, but fall back to https. A parent site published + # ONLY over https - which is normal, and which the operator did nothing + # wrong to have - left this defaulting to port 80, requesting a URL that + # answers nothing, and failing a working install with a red dialog. + $parentScheme = 'http' + $parentPort = 0 try { - $b = (Get-Website -Name $ParentSite -ErrorAction SilentlyContinue).bindings.Collection | - Where-Object { $_.protocol -eq 'http' } | Select-Object -First 1 + $bindings = (Get-Website -Name $ParentSite -ErrorAction SilentlyContinue).bindings.Collection + $b = $bindings | Where-Object { $_.protocol -eq 'http' } | Select-Object -First 1 + if (-not $b) { + $b = $bindings | Where-Object { $_.protocol -eq 'https' } | Select-Object -First 1 + if ($b) { $parentScheme = 'https' } + } if ($b -and ($b.bindingInformation -match '^[^:]*:(\d+):')) { $parentPort = [int]$Matches[1] } } catch { } - $targets = @("http://localhost:$parentPort/$alias/") + if ($parentPort -eq 0) { $parentPort = if ($parentScheme -eq 'https') { 443 } else { 80 } } + + $targets = @("{0}://localhost:{1}/{2}/" -f $parentScheme, $parentPort, $alias) if ($hostName -and $hostName -ne 'localhost') { - $targets += "http://{0}:{1}/{2}/" -f $hostName, $parentPort, $alias + $targets += "{0}://{1}:{2}/{3}/" -f $parentScheme, $hostName, $parentPort, $alias + } + # An https parent almost certainly has a certificate for its real name, + # not for 'localhost', and a certificate complaint is not an application + # fault. Accept any certificate for the duration of the smoke test only. + if ($parentScheme -eq 'https') { + Write-Log 'parent site is https-only; certificate validation is skipped for this check' 'WARN' + try { + Add-Type -TypeDefinition @' +using System.Net; +using System.Security.Cryptography.X509Certificates; +public class ShopdbSmokeTestCertPolicy : ICertificatePolicy { + public bool CheckValidationResult(ServicePoint sp, X509Certificate cert, WebRequest req, int problem) { + return true; + } +} +'@ -ErrorAction SilentlyContinue + [System.Net.ServicePointManager]::CertificatePolicy = New-Object ShopdbSmokeTestCertPolicy + } catch { } } } else { $targets = @("http://localhost:$SitePort/") @@ -1707,6 +1783,21 @@ function Invoke-Stage5 { } } + # The self-hosted API documentation. It is the thing an operator - or the + # assistant an operator is asking - reaches for on a server with no internet, + # and it is served from a file that has to have been staged into the build. + # A missing file makes /api/docs render an empty page, which nobody notices + # until it is needed. Warn, never fail: the application itself is fine. + $base = $targets[0].TrimEnd('/') + try { + $d = Invoke-WebRequest -Uri "$base/api/docs/openapi.json" -UseBasicParsing -TimeoutSec 20 + if ($d.StatusCode -eq 200) { Write-Log 'API documentation is being served at /api/docs' 'OK' } + else { Write-Log "/api/docs/openapi.json returned $($d.StatusCode)" 'WARN' } + } catch { + Write-Log '/api/docs is NOT working - docs/openapi.json was not staged into this build' 'WARN' + Write-Log ' the site runs fine; offline API reference and LLM assistance will not' 'WARN' + } + # The install is proven working, so the stage-0 handoff is no longer the only # copy of the generated password (.env has it and the app is running on it). # Safe to shred now, and only now - see the note in stage 2. @@ -1806,6 +1897,34 @@ function Invoke-Uninstall { Import-Module WebAdministration -ErrorAction SilentlyContinue + # A SUBPATH install has no site of its own - it is an Application under an + # existing one. Removing only the site left the application in place, pointed + # at a directory this is about to delete, so the parent site (at West + # Jefferson, the live classic ASP) served 503 on that path forever and + # Add/Remove Programs reported success. + # + # The alias is whatever MOUNT_PATH says, which is the same value wsgi.py + # mounts on, so it cannot disagree with how the app was actually published. + $mount = '' + $envFile = Join-Path $AppRoot '.env' + if (Test-Path $envFile) { + $line = Get-Content $envFile -ErrorAction SilentlyContinue | + Where-Object { $_ -like 'MOUNT_PATH=*' } | Select-Object -First 1 + if ($line) { $mount = $line.Substring('MOUNT_PATH='.Length).Trim().Trim('/') } + } + if ($mount) { + $removed = $false + foreach ($site in (Get-Website -ErrorAction SilentlyContinue)) { + $app = Get-WebApplication -Site $site.Name -Name $mount -ErrorAction SilentlyContinue + if ($app) { + Remove-WebApplication -Site $site.Name -Name $mount -ErrorAction SilentlyContinue + Write-Log "removed application /$mount from site '$($site.Name)'" 'OK' + $removed = $true + } + } + if (-not $removed) { Write-Log "application /$mount not present" } + } + if (Get-Website -Name $SiteName -ErrorAction SilentlyContinue) { Remove-Website -Name $SiteName -ErrorAction SilentlyContinue Write-Log "removed site $SiteName" 'OK' @@ -1820,11 +1939,22 @@ function Invoke-Uninstall { Write-Log "removed app pool $AppPool" 'OK' } else { Write-Log "app pool $AppPool not present" } - $rule = "ShopDB-Flask $SitePort" + # Stage 4 creates the rule as "$SiteName $SitePort". This looked for the + # literal "ShopDB-Flask 8090", which is not the same string as the default + # "shopdb-flask 8090" and matches nothing at all on a non-default port, so + # the rule outlived the uninstall. Build the name the same way stage 4 does, + # and sweep any rule left by a differently-ported install of the same site. + $rule = "$SiteName $SitePort" if (Get-NetFirewallRule -DisplayName $rule -ErrorAction SilentlyContinue) { Remove-NetFirewallRule -DisplayName $rule -ErrorAction SilentlyContinue Write-Log "removed firewall rule '$rule'" 'OK' } else { Write-Log "firewall rule '$rule' not present" } + Get-NetFirewallRule -ErrorAction SilentlyContinue | + Where-Object { $_.DisplayName -like "$SiteName *" -and $_.DisplayName -ne $rule } | + ForEach-Object { + Remove-NetFirewallRule -DisplayName $_.DisplayName -ErrorAction SilentlyContinue + Write-Log "removed leftover firewall rule '$($_.DisplayName)'" 'OK' + } if (Test-Path $AppRoot) { # .env holds the database password in plaintext by design (the app reads it diff --git a/deploy/windows/installer/shopdb-preflight.ps1 b/deploy/windows/installer/shopdb-preflight.ps1 index 3e8285b..dfaaeee 100644 --- a/deploy/windows/installer/shopdb-preflight.ps1 +++ b/deploy/windows/installer/shopdb-preflight.ps1 @@ -368,6 +368,31 @@ Invoke-Check 'MySQL' 'Port 3306' { } } +Invoke-Check 'MySQL' 'Backup client' { + # mysqldump is what takes the mandatory pre-upgrade backup. Without it every + # upgrade skips the backup - and skips it AFTER the application pool has been + # stopped and the tree replaced, so the site is down and there is nothing to + # restore from. A site whose database is on another server typically has no + # MySQL client installed here at all, which is exactly the case that needs it. + $names = @('mysqldump.exe') + $found = '' + foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'), + (Join-Path $AppRoot 'mysqlclient'), + 'C:\Program Files\MySQL', 'C:\mysql56\bin', + 'C:\Program Files (x86)\MySQL')) { + if (-not (Test-Path $root)) { continue } + $hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue | + Select-Object -First 1 + if ($hit) { $found = $hit.FullName; break } + } + if ($found) { + Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)" + } else { + Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' ` + 'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. Add mysqlclient\ to the installer bundle, or install the MySQL client on this server.' + } +} + Invoke-Check 'MySQL' 'Version and config' { # Find mysqld.exe via the service binary path; read the version and locate my.ini. $svc = Get-CimInstance Win32_Service -ErrorAction SilentlyContinue | diff --git a/deploy/windows/installer/verify_bundle_lock.py b/deploy/windows/installer/verify_bundle_lock.py index ad8fc2b..80634f0 100644 --- a/deploy/windows/installer/verify_bundle_lock.py +++ b/deploy/windows/installer/verify_bundle_lock.py @@ -31,6 +31,7 @@ PAYLOADS = [ ('python', True, 'the Python installer'), ('httpplatformhandler', True, 'the IIS module that launches waitress'), ('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'), + ('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'), ('mysql', False, 'MySQL, for the bundled-database option'), ] diff --git a/docs/BACKUP-RESTORE.md b/docs/BACKUP-RESTORE.md index 45f171f..197a16e 100644 --- a/docs/BACKUP-RESTORE.md +++ b/docs/BACKUP-RESTORE.md @@ -142,8 +142,40 @@ version both are no-ops. - `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .` should return a `VALIDATION_ERROR`, not a 500. +## Windows sites (installer-built) + +On a server installed from the Windows installer, everything above is wrapped by +the operator console. Do not run mysqldump by hand: + +```powershell +cd C:\shopdb-flask +.\shopdb-admin.ps1 backup # C:\ProgramData\ShopDB-Flask\backups +.\shopdb-admin.ps1 backup D:\backups +``` + +The dump is verified complete before it is reported as good; a truncated one is +deleted rather than left to be discovered when it is needed. An upgrade takes its +own backup automatically before touching the schema, and restores from it if a +migration fails. + +Two Windows-specific notes: + +- The backup directory is locked to Administrators and SYSTEM, because a dump + contains every row including user password hashes. Keep it that way. +- `mysqldump` must be present. It ships with the bundled-database option; a site + using a remote MySQL needs `mysqlclient\` in its installer bundle, or the + pre-upgrade backup is skipped. `shopdb-admin.ps1 check` reports this. + +Restoring is the standard `mysql < dump.sql`, then +`.\shopdb-admin.ps1 restart`. Also restore `C:\shopdb-flask\instance\` if you are +rebuilding a server - it holds uploaded branding and map blueprints, which the +database does not. + +See [OPERATE-WINDOWS.md](OPERATE-WINDOWS.md). + ## See also - [DEPLOY.md](DEPLOY.md) - first-time deploy +- [INSTALL-WINDOWS.md](INSTALL-WINDOWS.md) - Windows Server install - [UPGRADE.md](UPGRADE.md) - upgrade procedure (back up first) - [CONFIG.md](CONFIG.md) - environment variables and Setting keys diff --git a/docs/DEPLOY-WINDOWS-IIS.md b/docs/DEPLOY-WINDOWS-IIS.md index 3c36736..ac71dfe 100644 --- a/docs/DEPLOY-WINDOWS-IIS.md +++ b/docs/DEPLOY-WINDOWS-IIS.md @@ -1,5 +1,13 @@ # Deploy shopdb-flask to Windows IIS (MySQL 5.6) +> **Not the route for a new site.** Sister sites install from the Windows +> installer — one `.exe`, no manual IIS work: **[INSTALL-WINDOWS.md](INSTALL-WINDOWS.md)**. +> +> This is the **manual** procedure for the West Jefferson server, which was built +> by hand against its existing MySQL 5.6 and predates the installer. Keep it for +> that box. + + Runbook for standing up a single-site instance on the production Windows Server that already runs the classic ASP shopdb, using IIS + HttpPlatformHandler + waitress, against the existing MySQL 5.6. This is the test-instance path; keep diff --git a/docs/INSTALL-WINDOWS-IIS.md b/docs/INSTALL-WINDOWS-IIS.md index 15c78a0..8ba14da 100644 --- a/docs/INSTALL-WINDOWS-IIS.md +++ b/docs/INSTALL-WINDOWS-IIS.md @@ -1,5 +1,13 @@ # ShopDB - Windows + IIS install runbook +> **Not the route for a new site.** Sister sites install from the Windows +> installer — one `.exe`, no manual IIS work: **[INSTALL-WINDOWS.md](INSTALL-WINDOWS.md)**. +> +> This document is the **manual** procedure, kept for reference and for +> hand-built servers that predate the installer. Note that the installer will not +> adopt a server built this way without `-AdoptExisting`, on purpose. + + A step-by-step, **tested** install for a new site on Windows Server / Windows 11 with IIS in front of the Flask app (HttpPlatformHandler -> waitress), backed by MySQL. This runbook was validated end to end on a win11 + IIS + MySQL 5.6 box. diff --git a/docs/INSTALL-WINDOWS.md b/docs/INSTALL-WINDOWS.md new file mode 100644 index 0000000..0005980 --- /dev/null +++ b/docs/INSTALL-WINDOWS.md @@ -0,0 +1,201 @@ +# Install ShopDB-Flask on Windows Server + +**This is the route for a new site.** You run one `.exe`, answer a few questions, +and get a working application. Nothing here needs an internet connection, and you +do not need to know IIS, Python or MySQL. + +If you are looking after an existing hand-built server, see +[DEPLOY-WINDOWS-IIS.md](DEPLOY-WINDOWS-IIS.md) instead — that is the manual +procedure, and the installer will not adopt a server it did not build. + +--- + +## Before you start + +You need **four things**. The installer supplies everything else. + +| | What | How to check | +|---|---|---| +| 1 | Windows Server 2019 or newer | `winver` | +| 2 | The **IIS Web Server role** installed | Server Manager → Manage → Add Roles → Web Server (IIS). Or run the PowerShell below. | +| 3 | Administrator rights on the box | Right-click PowerShell → "Run as administrator" works | +| 4 | A decision about the database — see [Which database?](#which-database) | — | + +Installing IIS, if it is missing (this needs no internet): + +```powershell +Install-WindowsFeature -Name Web-Server -IncludeManagementTools +``` + +The installer **checks all of this before it changes anything**, and it will not +let you continue until the check passes. You do not have to get it right first +time. + +### Which database? + +Two options. Pick before you start, because they ask different questions. + +- **Use the bundled MySQL** — the installer puts MySQL 8.0 on this server and + creates the database for you. Choose this when the server has no database + today. Simplest option, nothing to arrange in advance. +- **Use an existing MySQL** — the database already exists somewhere, and you have + a hostname, a database name, a username and a password for it. Choose this if + your site already runs MySQL, or a DBA looks after it. + +If you are unsure: if nobody has given you database credentials, you want the +bundled option. + +--- + +## Installing + +1. Copy the installer `.exe` onto the server. It is one file and needs no + network. +2. **Right-click it → Run as administrator.** Without this it cannot configure + IIS, and it will tell you so. +3. Work through the wizard. The pages are: + +| Page | What it wants | If unsure | +|---|---|---| +| **Server check** | Nothing — it reports what it found | Fix anything red, then "Check again". You cannot continue while something is red, and nothing has been changed yet. | +| **Features** | Which parts of the product this site uses | The defaults are fine. You can add more later; removing needs a new installer. | +| **Database** | Bundled or existing — see above | Bundled | +| **Database details** | Host, port, name, user, password | Only asked for the existing-database option | +| **Address** | How people reach the site | See [Own address or subpath?](#own-address-or-subpath) | +| **Client addresses** | Whether a proxy sits in front | See [Client addresses](#client-addresses) | +| **Location** | Where to install | `C:\shopdb-flask` is fine | + +4. The install takes a few minutes. Most of it is Python and the database schema. +5. At the end you get the address to open. **Write it down** — it is also on the + Start Menu as "Open ShopDB-Flask". + +### Own address or subpath? + +- **Its own address** — `http://yourserver:8090/`. Choose this on a server that + is not already running a website. Simplest. +- **Under this server's existing address** — `http://yourserver/shopdb/`. Choose + this when the server already serves something else and you do not want a second + port or a new DNS name. This is what West Jefferson uses. + +You cannot change your mind later without re-running the installer, because the +web interface has the address compiled into it. + +### Client addresses + +The application records who connects, and some features decide what to show based +on it. The wizard asks one question: + +- **Clients connect to this server directly** — the normal answer. Pick this + unless you know otherwise. +- **A proxy or load balancer sits in front** — pick this only if your network + team has told you traffic reaches this server through something else first. + +Getting this wrong is not dangerous, but the site will record every visitor as +coming from the server itself, and features that depend on location will not +work. It can be changed later by re-running the installer. + +--- + +## First login + +Open the address the installer gave you. With no users in the database yet, the +page offers to **create the first administrator**, then runs a short setup wizard +for site details, features and the floor map. + +That first account is a normal administrator account. Use a real password — +this is the account that creates everyone else. + +--- + +## Did it work? + +From the Start Menu, open **ShopDB-Flask Console** and pick option 1, or: + +```powershell +cd C:\shopdb-flask +.\shopdb-admin.ps1 status +``` + +You want to see the site started, the pool started, and `responding : yes`. + +Day-to-day tasks — restarting, backups, logs, upgrades — are in +[OPERATE-WINDOWS.md](OPERATE-WINDOWS.md). + +--- + +## If the install fails + +**The server is left part-configured.** Whatever had been done before the failure +is still there. That is deliberate: it means re-running is able to pick up where +it stopped. + +1. Read the error. It names the cause and what to do about it. +2. Fix that, then **run the same installer again**. Re-running is safe — it skips + what is already done and does not touch your database or `.env`. +3. If you would rather start clean, remove it from **Settings → Apps** first. + +The full log is at: + +``` +C:\ProgramData\ShopDB-Flask\logs\shopdb-install-.log +``` + +It records every step, including everything that was created. Send this if you +need help. + +### Getting help from an AI assistant + +These installs are often done with an assistant open in another window. Give it +real state rather than a description: + +```powershell +.\shopdb-admin.ps1 check -Json +``` + +That prints one structured block covering the version, how the site is published, +IIS state, database reachability, Python version, installed features and any +errors. Paste it in. **It contains no passwords.** The install log is also safe +to share — the installer keeps secrets out of it deliberately. + +Offline API reference for this server is served at `/api/docs` on the site +itself, and `docs\` in the install directory holds these runbooks. + +--- + +## Upgrading + +Run a newer installer over the top. It: + +- backs the database up first, **verifies the dump is complete**, and refuses to + continue if it cannot; +- restores from that backup if the schema migration fails; +- refuses to install an **older** build over a newer one; +- keeps your `.env`, your data and your `web.config`. + +Nothing else is required. See [UPGRADE.md](UPGRADE.md). + +> **Before your first upgrade:** confirm `mysqldump` is available — the console's +> health check reports it. Without it the pre-upgrade backup is skipped, and that +> is the one you would want if a migration went wrong. It ships with the bundled +> database option; for an existing remote database, ask for `mysqlclient\` to be +> included in your installer bundle. + +--- + +## Removing it + +**Settings → Apps → ShopDB-Flask**, or Add/Remove Programs. + +That removes the website, the application pool, the firewall rule and the +application directory. It deliberately **does not** drop the database and does +not uninstall MySQL, so your data survives. + +Take a backup first: `.\shopdb-admin.ps1 backup` + +--- + +## Notes for the person who builds the installer + +Building a bundle for a site is a separate job, documented in +[../deploy/windows/installer/README.md](../deploy/windows/installer/README.md). +Sites receive a finished `.exe`; they do not build one. diff --git a/docs/OPERATE-WINDOWS.md b/docs/OPERATE-WINDOWS.md new file mode 100644 index 0000000..9e5c3f2 --- /dev/null +++ b/docs/OPERATE-WINDOWS.md @@ -0,0 +1,195 @@ +# Running ShopDB-Flask on Windows Server + +Day-to-day operation of a site installed with the Windows installer. If you are +installing for the first time, start with [INSTALL-WINDOWS.md](INSTALL-WINDOWS.md). + +Everything here goes through one tool, installed alongside the application: + +``` +C:\shopdb-flask\shopdb-admin.ps1 +``` + +The Start Menu folder **ShopDB-Flask** has shortcuts for the common tasks. Run it +with no arguments for a menu, or pass a command directly. It needs +Administrator — it will ask, except for `open`. + +--- + +## The commands + +| Command | What it does | Safe at any time | +|---|---|---| +| `status` | Is it published, running, responding; database and table count | yes | +| `restart` | Recycles the application pool. **Use this after any config change.** | yes — drains requests rather than cutting them off | +| `stop` / `start` | Takes the site down / brings it back | yes, but `stop` makes it unavailable | +| `logs` | Last lines of the application and install logs | yes | +| `check` | Full health check | yes | +| `check -Json` | The same, machine-readable — see [Getting help](#getting-help) | yes | +| `verify` | Which build this is, and whether what is installed still matches it | yes | +| `sessions` | IIS worker processes and memory | yes | +| `plugins` | Which features are installed, which are available | yes | +| `add-plugin -Path ` | Turns on a feature this build ships | changes the site; restarts it | +| `backup [-Path ]` | Writes a verified `.sql` dump | yes, but see below | +| `open` | Opens the site in a browser | yes | + +Examples: + +```powershell +.\shopdb-admin.ps1 status +.\shopdb-admin.ps1 restart +.\shopdb-admin.ps1 backup D:\backups +.\shopdb-admin.ps1 verify -Path leaflet +``` + +--- + +## Backups + +```powershell +.\shopdb-admin.ps1 backup +``` + +Writes to `C:\ProgramData\ShopDB-Flask\backups` unless you pass a directory. The +dump is **verified complete** before it is reported as good — a truncated backup +is deleted rather than left to be discovered later. + +Two things to know: + +- **The dump contains everything, including user password hashes.** The directory + is locked to Administrators and SYSTEM. Keep it that way, and treat copies as + sensitive. +- **Store it off this server.** A backup on the server does not survive the + server. + +An upgrade takes its own backup automatically, before it touches the schema. + +Restore, and the Linux/Docker equivalents, are in +[BACKUP-RESTORE.md](BACKUP-RESTORE.md). + +--- + +## Upgrading + +Run a newer installer over the top. Nothing else. It backs up first, refuses to +go backwards, and restores if a migration fails. See +[UPGRADE.md](UPGRADE.md). + +--- + +## Adding a feature + +```powershell +.\shopdb-admin.ps1 plugins # what is here +.\shopdb-admin.ps1 add-plugin -Path warranty # turn one on +``` + +Only features **shipped in this build** can be added. Each site's installer is +built for that site's chosen feature set, so a feature nobody asked for is not on +the server at all — adding it means a new installer built from an updated +profile. `plugins` shows you which is which. + +--- + +## When something is wrong + +Work down this list. + +**1. Is it actually down?** + +```powershell +.\shopdb-admin.ps1 status +``` + +`responding : NO` with the pool `Started` usually means the application failed to +start, not that IIS is broken. + +**2. What does it say?** + +```powershell +.\shopdb-admin.ps1 logs +``` + +Application logs are in `C:\shopdb-flask\logs`, install logs in +`C:\ProgramData\ShopDB-Flask\logs`. + +**3. Try a restart.** It fixes anything that is a stuck worker, and tells you +immediately if it is not: + +```powershell +.\shopdb-admin.ps1 restart +``` + +**4. Check the database is reachable** — `status` reports the host and whether it +could count tables. A site that starts but shows no data is usually a database +problem, not an application one. + +**5. Confirm nothing has drifted:** + +```powershell +.\shopdb-admin.ps1 verify +``` + +This flags packages that no longer match what shipped — which usually means +somebody ran a `pip install` on the server by hand. + +--- + +## Getting help + +Give an assistant real state rather than describing the symptom: + +```powershell +.\shopdb-admin.ps1 check -Json +``` + +One structured block: version, how the site is published, IIS and pool state, +whether it responds, database host and reachability, Python version, installed +features, and any errors. **It contains no passwords** and is safe to paste into +a chat window or a ticket. + +The install log is also safe to share — secrets are deliberately kept out of it. + +Offline reference on the server itself: + +- `/api/docs` on the site — the full API reference, self-hosted, no internet. +- `C:\shopdb-flask\docs\` — these runbooks. +- `C:\shopdb-flask\sbom.cdx.json` — every component this build contains. + +--- + +## Answering "are we affected by this vulnerability?" + +The server carries its own bill of materials, so this does not need the build box +or an internet connection: + +```powershell +.\shopdb-admin.ps1 verify -Path +``` + +It reports whether the component is here, at what version, and whether it +actually **ships** or is only used to build the software. Example: + +``` +matches for 'leaflet': + leaflet 1.9.4 SHIPPED +``` + +Nothing found means this server does not carry it. + +--- + +## Where things live + +| | | +|---|---| +| Application | `C:\shopdb-flask` | +| Configuration and secrets | `C:\shopdb-flask\.env` (locked down — do not loosen) | +| Application logs | `C:\shopdb-flask\logs` | +| Install logs | `C:\ProgramData\ShopDB-Flask\logs` | +| Backups | `C:\ProgramData\ShopDB-Flask\backups` | +| Bill of materials | `C:\shopdb-flask\sbom.cdx.json` | +| Which build this is | `C:\shopdb-flask\.installed-version` | + +If the bundled MySQL was installed, its generated root password was written once +to `C:\ProgramData\ShopDB-Flask\mysql-root-password.txt`. **Move it into your +password manager and delete that file.** It cannot be recovered. diff --git a/docs/UPGRADE.md b/docs/UPGRADE.md index f27dfc0..5fd0e89 100644 --- a/docs/UPGRADE.md +++ b/docs/UPGRADE.md @@ -111,9 +111,33 @@ print(Setting.query.filter(Setting.key.like('map_blueprint%')).all()) PY ``` +## Windows sites (installer-built) + +Run a newer installer `.exe` over the existing install. That is the whole +procedure - none of the manual steps above apply. + +It backs the database up first and verifies the dump, applies the core and plugin +migrations, restores from that backup if a migration fails, and refuses to +install an older build over a newer one. Your `.env`, your data and your +`web.config` are kept. + +Before the first upgrade, confirm `mysqldump` is available +(`.\shopdb-admin.ps1 check`). Without it the pre-upgrade backup is skipped, which +is the one you would want if a migration went wrong. + +Afterwards: + +```powershell +.\shopdb-admin.ps1 status +.\shopdb-admin.ps1 verify +``` + +See [OPERATE-WINDOWS.md](OPERATE-WINDOWS.md). + ## See also - [BACKUP-RESTORE.md](BACKUP-RESTORE.md) - what to back up and how to restore +- [INSTALL-WINDOWS.md](INSTALL-WINDOWS.md) - Windows Server install - [CONFIG.md](CONFIG.md) - environment variables and Setting keys - [DEPLOY.md](DEPLOY.md) - first-time deploy runbook - `CHANGELOG.md` - what changed in each release diff --git a/docs/llms.txt b/docs/llms.txt index f1def7a..8cc1fbc 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -5,6 +5,29 @@ network devices, measuring tools, applications, knowledge base, USB, warranties) for GE Aerospace sites. This file is the quick entrypoint; the full machine spec is the OpenAPI at `/api/docs/openapi.json` (browse it at `/api/docs`). +## Installing and operating this server (read this before answering "how do I install/fix it") +The people running these servers are often not Windows, IIS or Python +specialists, and they ask assistants for help. Two documents are authoritative: +- `docs/INSTALL-WINDOWS.md` - installing a NEW site. One offline installer `.exe`. +- `docs/OPERATE-WINDOWS.md` - restart, logs, backups, upgrades, troubleshooting. +Both ship in `docs/` inside the install directory on every installed server. + +Do NOT walk someone through `docs/INSTALL-WINDOWS-IIS.md` or +`docs/DEPLOY-WINDOWS-IIS.md` for a new site. Those are the MANUAL procedure, kept +only for hand-built servers that predate the installer; following them produces a +server the installer then refuses to upgrade. + +Day-2 operations all go through `shopdb-admin.ps1` in the install directory +(default `C:\shopdb-flask`): `status`, `restart`, `logs`, `check`, `verify`, +`backup`, `plugins`, `open`. Before diagnosing anything, ask for the output of +`shopdb-admin.ps1 check -Json` - it reports version, publishing method, IIS and +pool state, HTTP reachability, database host and reachability, Python version, +installed plugins and errors, and it contains no secrets. `verify -Path ` +answers "does this server carry component X" from the on-box CycloneDX SBOM. + +Python is 3.14 and the wheelhouse is locked to it; an upgrade against a venv +built by a different minor version is refused by design. + ## Base URL Prod (West Jefferson): `https://tsgwp00525.wjs.geaerospace.net/shopdb` All API paths are under `/api` (e.g. `/api/assets`). Dev: `http://localhost:5001`. diff --git a/scripts/build-site.sh b/scripts/build-site.sh index 7d3b4c1..4fcb40a 100755 --- a/scripts/build-site.sh +++ b/scripts/build-site.sh @@ -86,6 +86,22 @@ fi echo "==> Generating SBOM ..." python3 "$REPO/scripts/generate_sbom.py" "$REPO" -o "$OUT/sbom.cdx.json" +# Docs the RUNNING SITE serves or the operator needs on the box. Without +# openapi.json and llms.txt the self-hosted /api/docs page is broken on every +# installed server - which matters most for the sites least able to debug it, and +# for the operators who will point an assistant at their own instance. The +# Windows runbooks ship because an air-gapped server has no other way to reach +# them; docs/ is not otherwise staged, so shipping the whole tree would be noise. +echo "==> Staging docs ..." +mkdir -p "$OUT/docs" +for doc in openapi.json llms.txt api-inventory.json \ + INSTALL-WINDOWS.md OPERATE-WINDOWS.md BACKUP-RESTORE.md UPGRADE.md; do + [ -f "$REPO/docs/$doc" ] && cp -a "$REPO/docs/$doc" "$OUT/docs/" +done +# Never let a missing optional doc fail the build - the loop's last test governs +# the exit status under `set -e`. +true + # Stage the profile INTO the tree. This is what makes the set self-describing: # `flask plugin apply-profile` at provisioning time reads the same profile the # tree was staged from, so the installed plugin set and the shipped plugin code