367bc56a6db3cbb0d115acaeb46ded4faaf40331
429 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
367bc56a6d |
Ship the equipment catalog so a new site does not start empty
`flask seed reference-data` wrote a dozen generic model types and no vendors or models at all, so adopting this platform began by retyping a catalog another site had already spent a year building. That is the largest single obstacle to standing a new facility up. scripts/export_catalog.py dumps the catalog from a live instance to shopdb/data/catalog.json, and `flask seed catalog` loads it. What travels: vendors 53, models 128, modelsupplies 146, modeltypes 35, machinetypes 21, computertypes 10, printertypes 9, networkdevicetypes 5, locationtypes 11, operatingsystems 14, measuringtooltypes 8, notificationtypes 3, accessprotocols 3 The 146 printer supplies are the most useful part after the models themselves: every toner, drum and maintenance kit with its part number, colour, capacity tier and page yield, already matched to the right model, instead of somebody reading them off spent cartridges. IDEMPOTENT and ADDITIVE. Records match on a natural key - a vendor by name, a model by vendor plus model number, a supply by model plus part number - so a second run adds nothing, and it never updates or deletes: a site that corrected a description or pointed a model at its own photo keeps its version. Catalog only. No assets, locations, employees, business units or anything with a serial number: nobody wants one plant's machines appearing at another. Vendor contact details are excluded too, since a rep's name and number belong to whoever holds that relationship. supportteams, printerdrivers and customfields are site-specific and deliberately absent. Models and supplies reference their vendor by NAME rather than id, because ids differ between databases and an id-keyed catalog would silently attach part numbers to the wrong printer. The installer offers it as a tick-box on a new "Starter data" page, defaulting to on, passing -SeedCatalog to stage 3. Offered rather than assumed: a site that machines nothing does not want 21 machine types cluttering its dropdowns. Verified by loading into an empty database and running twice: every group populated on the first pass, "Catalog already present, nothing to add" on the second. |
||
|
cproudlock
|
53c1f6476c |
Pick a network device's map position, and stop hardcoding one site's label prefix
Two things a second site ran into. The network device form asked for the map position as two raw numbers, so placing a device meant reading coordinates off another screen and typing them in. Machines, PCs and printers have had a "Set Location on Map" picker all along, and the network API already accepted mapx and mapy - only the form was missing. Same picker, same modal. The 3D parts kiosk hardcoded 'WJ' as the prefix shown before the number box, with a comment inviting whoever needed something else to edit the source. That is West Jefferson's gage-lab tag format and nobody else's, so another site's operators were told to expect letters that are not on their labels. It is now printedparts_label_prefix, set in Settings, defaulting to EMPTY - a site that has not set one sees no prefix rather than inheriting another site's convention. West Jefferson sets it to WJ once. The kiosk hides the prefix entirely when unset and falls back to no prefix if the setting cannot be read, because a cosmetic hint must never stop a kiosk working. Not to be confused with printedparts_code_prefix, which mints item codes like 3DP0042 and was already configurable. That is the code we generate; this is the tag already printed on the label. |
||
|
cproudlock
|
b44108f70c |
Run one slideshow timer, not two, and honour each slide's own duration
A site added two slides to the lobby display and it never changed between them. onMounted awaited fetchSlides, which starts the slideshow itself once it has more than one slide, and then started it AGAIN unconditionally. Two timer chains ran, and the second assignment to slideTimer lost the handle to the first, so nothing could ever cancel it. Both fired about ten seconds later, milliseconds apart, and each advanced one slide. With exactly two slides that is 0 -> 1 -> 0 every cycle: the display looked frozen. With three or more it advanced by two and merely skipped one, which is why this survived so long - and why adding a third slide would have appeared to "fix" it. onMounted no longer starts it; fetchSlides owns that. scheduleNextSlide also cancels any pending timer before setting a new one, so a future double-call replaces the chain rather than leaking an untracked one. While here: the feed has always sent a per-slide duration and the display ignored it, hardcoding ten seconds, so a slide set to hold for a minute changed after ten. It now uses the slide's own value, and the progress bar animates over that same duration instead of finishing early and sitting full. |
||
|
cproudlock
|
ead5bd8f58 |
Give the console a repair verb, and something real to check
A server whose migrations or seeds never finished does not fail politely. Most
pages answer 500 and settings endpoints answer 404 for keys that were never
created, which reads as a broken application rather than an unfinished install.
One site spent a morning being debugged that way.
`shopdb-admin.ps1 repair` runs what stage 3 of the installer runs: db upgrade,
plugin upgrade-all, and the three seeds. Every step is idempotent, so running it
on a healthy server changes nothing, and each step runs independently so one
failure does not silently skip the rest.
`check` now says so before anyone has to infer it:
THIS SERVER IS NOT FULLY PROVISIONED
- seed data is missing (permissions, settings or reference data)
Most pages will answer 500 until this is fixed. Run:
shopdb-admin.ps1 repair
That needs a real test to sit on, so `flask db-utils seed-state` reports each
seed group and exits non-zero when any is missing. Verified by emptying the
settings table inside a transaction: MISSING, exit 1, rollback clean. Without it
the console check would have looked reassuring while testing nothing - an older
build with no such command reports UNKNOWN rather than healthy, for the same
reason.
|
||
|
cproudlock
|
705dd771bd |
Store a blank optional unique field as NULL, and answer a duplicate with 409
A site reported "internal server error" adding a second business unit. It was reproducible: create one with a blank code, create another with a blank code, 500. A column that is unique and nullable accepts any number of NULLs - that is what makes "optional but unique" work - and exactly ONE empty string. The form sent '', so the first blank code saved and every one after it collided with it. The field showed no asterisk because it genuinely is optional; the database just behaved as though it were not. This is not specific to business units. A dozen columns across core and the plugins are unique and nullable - asset numbers, hostnames, item codes, subnet names, gage-lab tags - and each was one blank form away from the same 500. Fixing them an endpoint at a time would have left the next to be found by a user, so a before_flush listener normalises blank to NULL on any unique nullable text column. Listening on Session rather than on individual mappers covers plugin models imported later, and avoids mapper-event semantics that differ between SQLAlchemy versions. A genuine duplicate is now a 409 with a readable message rather than a bare 500 with a traceback in the log: reusing a code that is taken is the caller's mistake, not a server fault. Verified against the development database: three business units with blank codes all save, the blank stores as NULL, and a real duplicate code returns 409. |
||
|
cproudlock
|
85ff25462e |
Reset to page one when a filter changes, and let the catalog carry a real type
Two unrelated things found while looking at blank printer types. Selecting a filter while past page one returned an empty list. The filter asked the server for page 5 of a result set that now had one page, and the screen said nothing matched. useListQuery already resets the page - setSearch and setExtra both do - but the filter dropdowns bypassed it and called the loader directly. Nine list pages now route through applyFilter, which calls setPage(1) when it needs to and loads directly when already on page one, so the composable's URL watcher does not also fire and fetch twice. scripts/retype_models.py addresses why printer types cannot be derived. The catalog types every printer model "Printer": true, and useless, since it does not say whether the product is a laser, a plotter or a label printer. That answer is a property of the model - every VersaLink C405 is a laser MFP - but nothing recorded it, so nothing could derive it. Recording it on the MODEL means the existing backfill fills every printer by exact name match, and a printer added later inherits the right type the moment its model is chosen. It exports the models needing a decision to CSV with a type suggested from the model number, a person corrects the column, and applying it is a dry run unless given --commit. A suggested type is refused unless it already exists in that asset class's own vocabulary, which is what keeps the later name match working. The suggestion order matters and got this wrong first time: a generic plotter pattern matched "Zebra ZT411" and filed a label printer as a plotter. Brands now come before generic patterns, and the review step exists precisely because a confident wrong guess would type every asset using that model. Verified on the development database: 24 printer models need a decision, 22 got a sensible suggestion, applying them let all 42 printers match a printertype by name, and the transaction rolled back cleanly. |
||
|
cproudlock
|
e22322dcc9 |
Show model type in the machines list
The machine's own type is blank for the 134 machines that came from the classic ASP database on machinetypeid=1, a LocationOnly placeholder the import refuses to carry across as a real subtype. The catalog model knows what those machines are, and its type is populated, so the column reads modeltypename under a heading that says so. Where both values exist they are identical - all 262 machines in the development database match exactly - so nothing is lost by showing the one that is reliably filled in. This does not fix the underlying gap. A null machinetypeid also excludes a machine from the map's subtype filter and drops its marker to the default colour, and no column heading affects that. Only populating machinetypeid does, which is what the backfill script is for. |
||
|
cproudlock
|
cb18d170cf |
Say whose type it is
Two fields on the same page were both labelled "Type": the asset's own, and the catalog model's. Only one of them was vague. "Model type" already says exactly what it is; the bare "Type" did not say whose. So the unqualified one is the one that changes. No new vocabulary, and "Model type" reads correctly against it: Type -> Machine Type (machines) Type -> PC Type (computers) Type -> Printer Type (printers) Type -> Device Type (network devices) Left alone everywhere the word is not ambiguous - measuring tools, subnets, VLANs, notifications, supply types and the manifest editor have no model type on screen to be confused with. This is a labelling change only. It does not address the blank type column on machines imported from the classic ASP database, which is a data gap the backfill script fills; renaming a column heading was never going to put values in it. |
||
|
cproudlock
|
24266146d8 |
Show the model's type only when it differs from the asset's own
Adding a "Model type" row next to "Type" put two rows in the Hardware section that read identically. They come from different tables - modeltypes is the catalog-wide list spanning every kind of asset, machinetypes is machine-only - but the names line up in practice: all 262 machines in the development database match exactly, which is the same fact that makes the type backfill safe. So the row now appears only when the two disagree, which is the case worth seeing: a model catalogued as one thing fitted to an asset recorded as another. When they agree it says nothing and is hidden. Applied to machines, PCs, printers and network devices, each compared against its own type table. |
||
|
cproudlock
|
58b460fe3d |
Backfill an asset's type from its model, by exact name only
Correcting an earlier judgement. I said the model's type could not be used to fill an asset's type, because modeltypes is the catalog-wide list covering every kind of asset - it holds "Access Point", "Camera" and "Desktop PC" - and only about two thirds of its names exist as machine types. That is true across the whole catalog and misleading in practice. Restricted to the models an asset class actually uses, the picture is different: all 262 machines in the development database map exactly, because the non-machine entries are never used by machines. The blanks on the machines list are rows whose type the database could already have supplied. So the backfill now fills the type as well, under a rule that cannot mistype anything: exact name match or nothing. A model type with no identically named entry in the asset's own type table is reported with a count and left untouched, so somebody can decide rather than have a guess written into their data. The same shape covers computers, printers and network devices, each against its own type table. Verified against the development database by nulling one machine's type inside a transaction: it was detected as fillable, the proposal read "LocationOnly" -> "LocationOnly", the update restored exactly the original id, and the rollback left the row unchanged. Still a dry run unless given --commit, and a table missing the model column is skipped, so it runs against a server whose network migration is not yet applied. |
||
|
cproudlock
|
3f320fcc8b |
Derive an asset's vendor from its catalog model, and show the model's own type
An asset that carries a model but no vendor was showing a blank the database could already answer: the model records its vendor, and both sides reference the same vendors table. Machines, PCs, printers and network devices now fall back to it. The fallback is FLAGGED, not merged silently. to_dict sets vendorfrommodel and the detail pages render "(from model)" beside the value, because the record itself is still empty: the edit form shows an empty vendor box, and a page implying the vendor is stored would be lying about where it came from. The model's type is exposed under its own name, modeltypename, and shown as a separate "Model type" row. It is deliberately NOT used to fill in the asset's own type. modeltypes is the catalog-wide list covering every kind of asset - it holds "Access Point", "Camera" and "Desktop PC" alongside the machine entries - so it is a different taxonomy from machinetypes. Only about two thirds of the names overlap, and mapping one onto the other would mistype the remainder, with the failure mode being a machine labelled "Desktop PC". scripts/backfill_vendor_from_model.py writes the derived vendor down for real, since the display fallback leaves reports that read vendorid still seeing nothing. It is a dry run unless given --commit, fills only rows where the asset's vendor is NULL and the model names one, and never overwrites a vendor somebody chose. It skips a table lacking either column, so it runs against a server whose network migration has not been applied yet. Verified against the development database by nulling one machine's vendor inside a transaction: it was detected as fillable, restored to exactly its original value, and the rollback left the row untouched. FLASK_ENV is not forced by the script. The app already reads it from .env, and overriding it demanded a SECRET_KEY the environment had no reason to supply. |
||
|
cproudlock
|
f8c4246483 |
Fix model photo upload, and give network devices the model link the page assumed
Three faults around vendor-model photos, found while looking at why an uploaded image did not appear. Saving a model was blocked after uploading a photo. The Image URL field was type="url", and an upload sets it to an application path such as /api/models/image/model-120.png. Native url validation demands an absolute URL with a scheme, so the browser refused to submit the form with "Please enter a URL" for a value the page had just written itself. The field is now type="text", which is what it always needed to be: it holds either a full web address or a path on this server. documentationurl stays type="url". The upload button did not appear when adding a model, only when editing one. That was deliberate - the photo is stored as model-<id>.<ext>, so it cannot be sent before the record has an id - but it reads as a missing feature, and the hint explaining it was easy to miss. A photo chosen while creating is now held and uploaded as soon as the model is saved, and it is dropped if the dialog is cancelled, so it cannot land on the next model created in the same session. Network devices could never show a photo. NetworkDeviceDetail.vue binds its hero image to networkdevice.imageurl, but networkdevices carried only vendorid, with no link to a catalog model, so nothing could populate it - a feature that looked present and could not work. Machines, PCs and printers have carried modelnumberid since July. This adds the same column and relationship, the to_dict branch that exposes modelname and imageurl, the field on the API, and a Model selector on the form so the link can actually be set. The migration is guarded the same way employees0002photo is: on a fresh database the tables come from the SQLAlchemy models, which already declare the column, so an unconditional add fails with "duplicate column name". The foreign key is created only on databases that can add one by ALTER; routing it through batch_alter_table made Alembic's column sort raise "Circular dependency detected" on the fresh-database test. Deploying this needs `flask db upgrade` and `flask plugin upgrade-all` on the server, not just a file copy. |
||
|
cproudlock
|
92a90fcec6 |
Document publishing the installer as a release asset
The exe is attached to a release, never committed: most forges reject files over 100 MB inside a repository while allowing release assets far larger, and a committed binary would sit in every future clone forever. Also records that tags must be pushed explicitly. A plain push of the branch does not carry them, so a release had nothing to hang off. |
||
|
cproudlock
|
89e880afc3 |
Release 0.8.0
The Windows installer has never shipped under a version: v0.7.0 was tagged before any of it existed, so every build handed out so far stamped a server with 0.7.0. Two servers running different builds were indistinguishable, and the installer logged each upgrade as "same version already installed" rather than recording what changed. This cuts the release that fixes that. 0.8.0 rather than a patch: the air-gapped installer is a new capability, and pre-1.0 semantic versioning puts that in the minor slot (ADR-007). CHANGELOG gains a 0.8.0 section covering the twelve defects a real Windows Server 2019 install surfaced, the move from inferring "is this a re-run of my install?" to recording it, and the operator documentation. deploy/site-profile-universal.json is now in the repository. Released builds were being produced from a profile in a temporary directory, so the next release could not have been reproduced once that file was cleaned up. docs/RELEASING-WINDOWS.md points at the committed profile and says why. scripts/gen_openapi.py reads __version__ out of shopdb/__init__.py instead of restating it. Its hardcoded copy had already drifted a release behind, which is the same mistake that once shipped an installer stamped with the wrong version.v0.8.0 |
||
|
cproudlock
|
fb53161578 |
Answer "is this a re-run of my install?" from a record, not from the machine
The installer inferred that question from whatever the server happened to look like: a MySQL service exists, the database has tables, the site exists, the venv exists. None of those record who created them. A retry after a failed first install was therefore taken for an upgrade of somebody else's working system, which produced two dead ends on exactly the retry the wizard invites: stage 3 demanded a mandatory backup of a database its own failed attempt had written, and then refused to prune tables it had created minutes earlier, because core migration 7d05 seeds access protocols owned by the computers plugin and any profile without that plugin hit the refusal every single time. An install record at ProgramData\ShopDB-Flask\install-state.json answers it instead. It is written when provisioning STARTS rather than when it finishes, because the run that dies halfway is precisely the run whose retry needs it, and it records what this installer created as it goes, so a crashed run no longer leaves the next one guessing from the machine. During unfinished first provisioning the pre-migration backup becomes advisory and prune may force, since every row present was written by an earlier attempt of the same install. On an established install both stay exactly as they were. The classification is deliberately asymmetric: an install predating this record carries a version stamp and probably real data, so it is treated as established and keeps the mandatory backup. Guessing "first run" there would arm prune --force against live tables. Get-CreatedItems comma-protects its return. A zero-length array returned from a PowerShell function unrolls to $null, and $null.Count is fatal under StrictMode 2.0 - the same fault that made bundle verification fail on every install earlier. The harness caught it before it shipped. Tests: deploy/windows/installer/tests/test-install-state.ps1 exercises new servers, retries, completed installs, unrecorded-but-stamped installs, records naming another directory, corrupt records, and persistence across a crash. tests/test_installer_state.py runs it wherever pwsh exists and asserts the invariants as text everywhere else. Both were confirmed to fail when the prune gate or the comma protection is removed. pytest.ini stops collection walking into deploy/windows/installer/bundle, which is build output holding a complete second copy of the application. Importing every plugin twice made SQLAlchemy refuse a redefined table and the whole suite fail to collect, on a tree with nothing wrong in it, purely because an installer had been built first. It surfaced only when the bundle grew from four plugins to thirteen. |
||
|
cproudlock
|
412c2dc877 |
Record the code-signing decision
Waiting for a certificate from the organisation's own certificate authority rather than buying one from a public CA. Every server this installer runs on is centrally managed and already trusts that root, so an internally issued Authenticode certificate removes the unknown-publisher warning exactly where it matters; a public certificate would buy trust on machines this software never reaches. Notes the interim measure that costs nothing: publish the SHA-256 through a channel separate from the installer, since a hash beside the file is only as trustworthy as write access to that location. Wording avoids naming internal infrastructure, since docs/ is published. |
||
|
cproudlock
|
1c04ff28b9 |
Close the remaining installer review findings
Eleven findings, grouped by the root cause each belongs to. Wizard input reaching a command line unchecked (ShopDBFlask.iss). Port fields were spliced in bare and arrive as [int] parameters, so a blank or mistyped port shifted every argument after it; both port fields are now validated as 1-65535 digits. A path ending in a backslash, which is what a drive root looks like, ended its argument with \" and CommandLineToArgvW read that as an escaped quote, so paths are now quoted through a helper that doubles the trailing backslash. A drive root is refused outright as well: uninstall deletes the application directory recursively, so installing to D:\ would have wiped the drive on removal. The password handoff was written with SaveStringToFile, which writes an AnsiString, and read back as UTF-8, so a correct non-ASCII password was reported as wrong; it now goes out as UTF-8 without a BOM. Launching without checking the result. Plugin deregistration invoked "flask plugin uninstall" without --yes, and the command carries a click confirmation_option that aborts with exit 1 when nothing can answer the prompt, so it could never once have succeeded; the bare 2>&1 under EAP Stop then turned that into a terminating error which the catch downgraded to a warning while the plugin directory was deleted regardless. It now passes --yes, brackets the error preference, restores the location in a finally, and keeps the code on disk unless deregistration actually succeeded. MarkShortcutRunAs had four quotes where it needed three, which kept the whole command inside one Pascal literal so LnkPath was never interpolated and no shortcut ever got the elevation flag; its exit code is now logged too. Comparing IIS physical paths as raw strings. IIS stores the path as typed, so it may carry environment variables or a trailing backslash. A Test-SamePath helper now normalises both sides. That closes a real hazard in uninstall, which matched applications on alias alone and would remove an unrelated application of the same name under another site, unattended, since -OnFailure never suppresses the confirmation. Accepting existing IIS state without reconciling it. "Site already exists" took the site however it was, so re-running with a different port left the old binding while CORS_ORIGINS, the firewall rule and the smoke test all used the new one, failing a working server. It now refuses with both ports named rather than silently re-binding, and refuses a site of that name serving a different directory. Preflight rows drawn past the panel. The failures loop had no cap at all and the warnings loop capped at 6, a number unrelated to the panel, which holds about three rows. The cap is now measured from the panel height, applies to both loops, and the footer counts what was actually left out instead of inferring it. Also: a failed upgrade now says the application pool is still stopped and how to start it, rather than only "part-configured", since stage 2 stops a pool that was serving. It is deliberately not restarted automatically, because after a stage 3 failure the deployed code and the schema may disagree. shopdb-admin Restart-App starts a stopped pool or site instead of recycling, which is a no-op on a stopped pool and then reported the application as unresponsive. A dead Write-Log line that parsed as three arguments is gone, and a preflight warning no longer tells the operator to add a directory to a compiled exe. |
||
|
cproudlock
|
ce521e84a5 |
Lock down backup directory ACLs, and let the uninstaller reach IIS
Two findings from the installer review, both of which fail silently. Database dumps were readable by every authenticated user. A directory created under ProgramData inherits BUILTIN\Users:(I)(OI)(CI)(RX), and a dump contains every row including the users table and its password hashes. The installer applied an owner-only ACL, but only in the branch that CREATED the directory, so a directory created first by the console (shopdb-admin.ps1 backup) kept the inherited permissions and the installer could never repair it. The ACL is now re-applied on every run rather than only on creation, and the grants are made inheritable with (OI)(CI) so dumps written into the directory later are covered too. shopdb-admin.ps1 applies the same hardening for the default location, and for an operator-named path says the dump holds password hashes rather than silently rewriting the ACL of a directory that is theirs. Verified on Windows: before, the directory carried BUILTIN\Users:(I)(OI)(CI) (RX); after, only SYSTEM and Administrators, and a file created inside inherits exactly those two. Without (OI)(CI) that file would not have been covered. The uninstaller could not remove anything in IIS. [UninstallRun] launched a bare "powershell.exe", and the Inno uninstaller is a 32-bit process, so WOW64 resolved it to the 32-bit PowerShell, which cannot see the IIS provider. The site, application pool and application survived, pointing at a directory that HAD been deleted, while Windows reported a clean uninstall. It now uses the same Sysnative path as the [Run] entry, which was the last unshielded launch site in the file. |
||
|
cproudlock
|
1d73bd477e |
Document what future Windows releases look like, for operators and for builders
Two audiences, two documents. Both were only in people's heads. UPDATES-WINDOWS.md is for whoever runs a server: updates arrive as one self-contained exe, an update takes two to four minutes, the site is down for that time, .env and data and any hand-edited web.config are kept, unticking a feature never removes it, the database is backed up and verified first, and a downgrade is refused because migrations only go forwards. It covers both kinds of security release, application and third-party, and explains that the CycloneDX inventory staged on every server is what answers a published vulnerability question. It also says plainly that the exe is not signed and the checksum is the integrity check to rely on today. It answers one question the existing docs did not address at all: the effect on other sites sharing the same IIS server. The application pool is isolated and the configuration is scoped to its own path, so other sites keep their own handlers. What IS shared gets named rather than glossed: installing the IIS modules and writing server-level configuration recycles application pools across the server, which can drop requests in flight and clears in-memory session state, though IIS is never stopped and no iisreset is issued. The two IIS modules and the single permitted rewrite server variable are machine-wide and stay behind on uninstall, deliberately, since another site may have come to depend on them. The bundled database option collides on port 3306 with an existing MySQL. RELEASING-WINDOWS.md is for whoever builds releases: the three kinds of change and the commands for each, why bundle-lock.json must be committed, the two dependency traps that have each already cost a release, which generated files must never be hand-edited, and the pre-release checks. It records the two known gaps honestly - no code signing, and compiling still requires Windows and a person. UPGRADE.md and OPERATE-WINDOWS.md link to the operator document. |
||
|
cproudlock
|
aeee210cf6 |
Repair a web.config that an earlier build made unusable
Stage 4 deliberately leaves an existing web.config alone, because operators put real changes in it: extra MIME maps, a /installers location, bindings, a proxy-specific rule. Overwriting reverts those silently. That rule had no exception, and earlier builds of this installer wrote an <allowedServerVariables> block which is fatal on its own: the section is Deny by default, so IIS rejects the entire file with 500.52 before httpPlatformHandler runs. Any server already installed would therefore keep the broken file forever, with re-running the fixed installer powerless to help, since the first thing stage 4 does is decline to touch it. Strip just that element, keeping every other edit, and only when it contains nothing besides the variable this installer adds. A block holding anything else is somebody's deliberate change and is left alone with a warning. The previous file is copied to web.config.before-xff-fix first. Exercised against four inputs: the file earlier builds wrote, which is repaired and still parses as XML with the rewrite rule intact; a block with an operator-added variable, which is left unchanged; an empty block, which is the $null.Count trap under Set-StrictMode 2.0 and is why the filter is wrapped in @(); and an already-correct file, which is a no-op. |
||
|
cproudlock
|
95b0b77c13 |
Allow HTTP_X_FORWARDED_FOR at server level instead of declaring it per-application
The stage 5 smoke test failure was a locked config section, but not one of the two the installer unlocks. A diagnostic collected from the server returned: HTTP 500.52 - URL Rewrite Module Error Module RewriteModule, Handler httpplatformhandler Error Code 0x80070021 Config Error: This configuration section cannot be used at this path. Config File: \\?\C:\shopdb-flask\web.config handlers and httpPlatform were both overrideMode Allow and locked false, so the unlock had worked. The section at fault was a third one, system.webServer/rewrite/allowedServerVariables, which ships overrideModeDefault="Deny". web.config declared <allowedServerVariables> locally for the X-Forwarded-For rule, and IIS rejects that declaration outright, failing the entire configuration before httpPlatformHandler ran. python was therefore never launched and C:\shopdb-flask\logs stayed empty, which reads as a dead application or a permissions fault and is neither. Unlocking the section would let every site on the machine declare arbitrary server variables. The installer now adds the single variable to the server-level allow list, checking first because a duplicate add is an error, and web.config no longer declares it. The rewrite rule is unchanged. Verified by applying the installer's own uncommenting to the template and parsing the result: one rewrite element, no allowedServerVariables, the rule still setting HTTP_X_FORWARDED_FOR from REMOTE_ADDR. shopdb-diagnose.py checked only the two sections the installer unlocks, so it could not have named this one; the IIS error page did. It now reports the lock state of the rewrite sections as well. |
||
|
cproudlock
|
10ee3a3c58 |
Add a stage 5 diagnostic collector
The stage 5 smoke test failing tells us only that IIS did not return 200. The cause is in one of four places, and finding out which has taken a round trip per guess. This gathers all four in one pass and writes a single report. It records what IIS actually answers on localhost, 127.0.0.1, ::1 and the machine name, including the status code and the parsed text of the IIS error page; the site, application, pool and module state from appcmd, plus the override state of the two config sections httpPlatformHandler needs; the contents of web.config and the resolved httpPlatform processPath; whether the venv can import shopdb and call create_app; the application logs, separating a missing log from an empty one; the ACLs the pool identity depends on; and recent HttpPlatform, WAS and W3SVC event log entries. Secrets never reach the report. Values are read from .env first, then scrubbed from every section before the file is written, which covers command output and tracebacks that might quote them. A password embedded in any connection URL is also masked whether or not it came from .env. Standard library only, so it runs on the bundled runtime or any system Python. Verified end to end on a Windows VM: it correctly reported a 404 with the IIS error code for an absent application, and that localhost resolves to ::1 first. |
||
|
cproudlock
|
97391cdee4 |
Unlock IIS config after the application exists, and report why the smoke test failed
Two defects found in the stage 4 and stage 5 logs from a Windows Server 2019 install. The scoped config unlock ran before the thing it unlocks existed. appcmd resolves its location argument against applicationHost.config, but the unlock was issued from the ACL block, ahead of New-WebApplication. On a first install "Default Web Site/shopdb" is not there yet, so appcmd returned 80070003, "the system cannot find the path specified", and the code fell through to unlocking the section for the entire machine. That fallback exists for servers which refuse the scoped form; it was instead the only path a first install could take, so every install silently granted handler delegation server-wide. Moving the block below site and application creation lets the scoped unlock work. The smoke test discarded the diagnosis. Invoke-WebRequest raises on any non-2xx, and the catch block kept nothing from the exception, so a fault IIS had already identified by status code was reported as "site did not return 200 ... check the logs". It now records the status code and the text of the IIS error page, and prints the tail of the HttpPlatform stdout log, which is where a Python traceback lands. It also distinguishes a missing log from an empty one: the first means the pool never launched python, the second that python started and wrote nothing. A non-200 that did not raise, such as a redirect, skipped the retry delay, so the loop could spend all twelve attempts at once and report a timeout without having waited. |
||
|
cproudlock
|
a352a21a10 |
Declare packaging as a runtime dependency
shopdb/plugins/loader.py imports packaging.specifiers and packaging.version at module scope, but packaging was never listed in requirements.in. It was present in every development and CI environment as a transitive dependency of pytest, so the full suite passed while a venv built from requirements.txt alone could not import shopdb at all. The Windows installer builds exactly such a venv, so stage 3 failed on a customer server with ModuleNotFoundError: No module named 'packaging', after the runtime and all wheels had installed successfully. Add packaging to requirements.in, recompile the hashed lockfile, and add the wheel to the offline wheelhouse with the matching bundle-lock entry. The recompile also picked up newer uv formatting: inline environment markers on cffi and greenlet and shorter "via" comments. The pinned distribution set and every existing hash are unchanged. tests/test_runtime_dependencies.py guards the general case by scanning shopdb/, plugins/ and scripts/ for unconditional third-party imports and asserting each maps to a distribution pinned in requirements.txt. Test dependencies are the blind spot for this class of failure, since they are present wherever the suite runs and absent wherever it does not. |
||
|
cproudlock
|
f6b621d126 |
fix(installer): keep -Wait, and use the stage-0 handoff even when .env exists
Two defects from a Server 2019 run that got further than any before it - the payload verified against the lock on a real server for the first time. EMPTY EXIT CODE. "Python install failed (exit )" on an install that had actually worked. Dropping -Wait to make -TimeoutSec enforceable left $p.ExitCode unreadable: PowerShell only reliably populates it on a waited process. -Wait is restored and the trade is now explicit - exit codes are load-bearing here, 1639 vs 1603 vs 3010 is the entire diagnosis, and a bounded wait is not worth losing them for. -TimeoutSec is advisory: logged as an expected duration so a hang is identifiable, not enforced. The code is also read defensively now, and an unreadable one fails loudly rather than being taken for success. That timeout has never worked - -Wait made the block dead code from the start - so nothing is lost that was ever there. Trying to fix it broke something that was working, which was the wrong trade to make silently. STALE .env PREFERRED OVER A GOOD HANDOFF. The .dbpass fallback sat in the else of "if .env exists", so it was consulted only when .env was absent. A part-finished install HAS an .env, holding whatever password stage 2 last wrote; if stage 0 has since regenerated the credential, .env is stale and .dbpass is correct - and the installer preferred the stale one, giving "Access denied" with the right password sitting unread on disk. The handoff is now applied before the branch, so it covers both, and only when .env points at the local server so it can never redirect a site whose database lives elsewhere. |
||
|
cproudlock
|
21110b86eb |
fix(installer): clear the retry path, which is the path everyone is actually on
Five defects from the Windows-defect review, each confirmed against the code
before changing it. Four of the five only fire on a RE-RUN - and after eight
attempts a re-run is the normal case, not an edge case, which is exactly why
they survived.
Invoke-Native, three defects in one function:
- Any non-zero exit was failure. 3010 and 1641 mean "done, reboot required",
and the VC++ redistributable returns 3010 on a server with a pending file
rename - an ordinary state on a freshly patched box. It is now an accepted
outcome for the installers that can report it, logged as a warning so the
operator knows a reboot is owed.
- -Wait blocks inside Start-Process until the child exits, so the -TimeoutSec
block below it could never run. Every timeout on every MSI was decorative.
The wait is now bounded here, followed by a parameterless WaitForExit so the
redirected output is flushed before it is read.
- The Python bootstrapper ran /quiet with no /norestart, free to reboot the
server mid-install.
Stage 0 refused to run when a MySQL service existed - including the MySQL84 it
had registered itself. Every bundled-database retry dead-ended while the wizard
promised that re-running was safe. A foreign MySQL still blocks; ours is started
if stopped, and the create-the-server block is skipped. It also no longer tries
to bootstrap through a root account whose password it set on the previous run:
with the handoff present there is nothing to do, and without it there is no safe
automatic recovery, so it says what to do instead of guessing.
Stage 4's appcmd unlock used '2>&1' under $ErrorActionPreference = 'Stop', which
turns any appcmd stderr into a terminating error - so the exit-code test and the
server-wide fallback, the whole reason the block exists, were unreachable, and
the stage aborted after Python, the venv, the schema and the ACLs had been
changed.
Stage 3 ran prune-schema and treated its refusal as a failure. Refusing is the
designed outcome when a table holds rows, signalled with SystemExit(1), so
Invoke-Native killed the stage and the reporting written to explain the refusal
was unreachable. Core migration 7d05 seeds access protocols owned by the
computers plugin, so any profile omitting computers hit this on every retry.
The preflight's MySQL 5.6 index-flag check is a warning, not a blocker. It
inspects the LOCAL MySQL, which may not be the database being installed against;
stage 3 checks the one actually chosen. Same class as the HttpPlatformHandler
blocker fixed earlier.
|
||
|
cproudlock
|
5f350179b1 |
fix(installer): undo a fix applied twice, and load the checker at script scope
A review of the installer for Windows-only defect classes found seven live issues. These two would have stopped the next attempt on any server. DOUBLE-APPLIED GUARD. Yesterday's $null.Count fix was applied at BOTH ends: Test-BundleLock returns ,$problems, and the call site also wrapped it in @(). The comma already hands the array back intact, so the extra @() nests it and .Count becomes 1 regardless of how many problems there are. Every install would have failed with "the bundle does not match bundle-lock.json (1 problem(s))" on a byte-perfect payload. Applying the same guard at both ends was worse than applying it at neither. Verified in a Windows VM against a real bundle: clean 0, tampered 1, restored 0. DOT-SOURCE SCOPE. bundle-lock.ps1 was dot-sourced INSIDE Assert-BundleIntegrity, which loads it into that function's scope - every helper it defines disappears when the function returns. Assert-BundleIntegrity itself worked; the next caller, Get-WheelhousePythonTag, died with "The term 'Get-JsonProperty' is not recognized". It only fires where a venv already exists, so greenfield was fine and every retry after a part-completed install was not. Now loaded once at script scope, guarded so the stages that run without a bundle still work. Both were confirmed by running them rather than by reading: the nesting with a three-case pwsh test, the scoping with a minimal repro. |
||
|
cproudlock
|
14fedcee4c |
fix(installer): a clean payload crashed stage 2
Reported from Server 2019: "The property 'Count' cannot be found on this object" immediately into stage 2. Test-BundleLock returns an array of problems, and an EMPTY array means the payload is exactly right. PowerShell unrolls a zero-element return into $null, and under Set-StrictMode 2.0 $null.Count throws - so the branch that runs when everything is correct was the one that could not run. Every failing bundle got past it fine, which is why nothing caught it until the 8.3 path fix made verification succeed for the first time on a real server. Fixed at both ends: the call site wraps in @(), and Test-BundleLock returns ,$problems so no caller can be handed $null or a bare string depending on how many problems there happen to be. The other .Count uses in this file were already @()-wrapped and are unaffected. |
||
|
cproudlock
|
fe091e751a |
fix(installer): the bundled database rejected connections it should have accepted
Two defects in the stage 0 bootstrap, both surfacing as "Access denied" on a server where the operator was holding the correct password. CREATE USER IF NOT EXISTS is a no-op on an existing user - it does NOT change the password. Stage 0 generates a fresh password every run and overwrites .dbpass with it unconditionally, so any path that re-runs the bootstrap over an existing account left the handoff holding a password the server had never been told. ALTER USER now follows each CREATE, so the stored password and the handoff always agree. The user was also only created for 'localhost' and '127.0.0.1'. On current Windows, 'localhost' resolves to the IPv6 loopback FIRST, so an operator who types localhost rather than 127.0.0.1 arrives as '<user>'@'::1' - an account that did not exist - and MySQL answers "Access denied" naming a host they never typed. The ::1 account is now created and granted alongside the other two. Note the datadir guard means the first defect could not fire on a straightforward re-run - stage 0 refuses a non-empty data directory before reaching the bootstrap. It was still wrong, and reachable once the directory has been cleared by hand, which is what the failure message tells operators to do. |
||
|
cproudlock
|
189a474082 |
fix(installer): do not demand a password the installer already holds
Asked why the password box does not pre-fill from .dbpass. It should not - but it should not have been demanding a password either. .dbpass is the ACL'd handoff stage 0 writes when it creates the database itself, and stage 2 already reads it automatically when no password is supplied. The wizard, though, required a password whenever .env was absent, without checking for the handoff. On a server where stage 0 had completed but stage 2 had not - which is exactly what a partly-failed install leaves - the operator was blocked on a secret the installer already had, and sent hunting for a generated password they were never meant to handle. Blank is now accepted when either .env or .dbpass is present, and the sign-in page says so when it sees a handoff. Deliberately NOT pre-filled into the password box, for two reasons. It is the only copy of a generated password, so round-tripping it through a UI control and back out through a temporary password file adds exposure for no benefit - stage 2 reads the file directly. And .dbpass belongs to the BUNDLED database; on the existing-database page the operator is pointing at someone else's server, where a locally generated password is simply the wrong answer. |
||
|
cproudlock
|
5f18ca27a1 |
fix(installer): split the database page so every field is reachable
Trimming the description brought the Username box back and left Password off the bottom. CreateInputQueryPage stacks its fields below the description and neither scrolls nor shrinks, so a field that does not fit is drawn past the surface and simply never appears - no error, no scrollbar. Sizing the description against a pixel budget that varies with DPI and font scaling is guesswork, and it had now failed twice. Connection details (host, port, database) and sign-in (username, password) are now two pages of three and two fields. Both fit under any reasonable description, at any scaling, without anyone having to estimate. The upgrade hint about leaving the password blank moves to the sign-in page, where the password field actually is. ShouldSkipPage hides both pages for the bundled-database option, and the stage arguments read the values from their new homes. |
||
|
cproudlock
|
e650eb0220 |
fix(installer): database page lost its Username and Password boxes
Reported from the Server 2019 test: the Existing database page rendered a truncated "Username:" label and no input boxes at all below it, so there was no way to enter connection details. CreateInputQueryPage lays its fields out BELOW the description text. The description had grown to include a five-line CREATE DATABASE / CREATE USER / GRANT block, added so a DBA could be handed the exact SQL. With five fields underneath, the last two fell past the bottom of the page surface, where they are simply not drawn - no error, no scrollbar, just missing controls. The description is back to three lines. The SQL moves to docs/INSTALL-WINDOWS.md, which is where someone would look for it anyway and where it can be copied without being retyped from a wizard page. Wizard page descriptions are a fixed budget: anything long enough to be worth reading twice belongs in the guide, not on the page. |
||
|
cproudlock
|
c5797bb339 |
fix(installer): payload verification broke on 8.3 short paths
A Server 2019 install reported all 96 payload files as simultaneously missing and unexpected, with mangled names - wheels/heels/flask.whl, python/ython/python-3.14.6-amd64.exe, mysqlclient/lient/mysql.exe. Exactly five characters of each directory name survived, which is the difference between ADMINI~1 and Administrator. Inno extracts the bundle under C:\Users\ADMINI~1\AppData\Local\Temp\..., an 8.3 SHORT path. Resolve-Path kept that short form while Get-ChildItem returned the long one, so the root was five characters shorter than the prefix being sliced off every FullName, and every relative key came out wrong. The payload was correct; the comparison was not - the verifier refused a perfectly good bundle. The root now comes from Get-Item, which goes through the same provider as Get-ChildItem so their path forms agree, and the prefix is checked with StartsWith before being trimmed. If the two ever disagree again this throws instead of inventing paths. Verified against the real failure mode rather than assumed: running the check through C:\SHOPDB~3\bundle in a Windows VM now passes. Nothing on Linux or in a normally-pathed Windows directory could have caught this - the short name only appears under a profile directory long enough to need one, which is where Setup extracts. |
||
|
cproudlock
|
263ae8e3b4 |
fix(installer): install the Visual C++ runtime before MySQL, and log the MSI
Second failure from the Server 2019 test. The previous fix worked - msiexec went from exit 1639 (ERROR_INVALID_COMMAND_LINE, which is why it printed its usage dialog) to exit 1603 (ERROR_INSTALL_FAILURE), so the command line parses now and the MSI itself is failing. It failed in 1.1 seconds. An MSI that dies that fast has not begun installing; it has failed a launch condition. MySQL 8.4 requires the Visual C++ redistributable and a bare Windows Server does not ship it - the same runtime mysql.exe and mysqldump.exe import, which was visible when their DLL dependencies were trimmed and went unnoticed. Stage 0 now installs VC_redist.x64.exe from the bundle before touching MySQL, skipping it when vcruntime140.dll is already present, and fails with a sentence naming the requirement if the redistributable is absent from the bundle altogether. vcredist\ is an optional locked payload. msiexec also gets /l*v now. A bare 1603 names neither the failing action nor the reason, and it is the most common MySQL install failure - diagnosing this one took a launch-condition inference rather than a log. The MSI log lands beside the installer's own in ProgramData, so the next failure is readable instead of guessed at. |
||
|
cproudlock
|
8d0afc40d3 |
fix(installer): bundled MySQL install failed on a malformed msiexec command line
Reported from a Windows Server 2019 test: a "Windows Installer" dialog listing every msiexec /Option appeared, then the wizard reported that the bundled MySQL database could not be installed. That dialog is msiexec's usage help - it prints it when the command line does not parse - so the install never started. Cause: $MysqlRoot defaulted to 'C:\Program Files\MySQL\MySQL Server 8.4', which contains spaces. Invoke-Native wraps any argument containing whitespace in quotes, producing "INSTALLDIR=C:\Program Files\...". msiexec takes public properties as PROPERTY=value and expects the VALUE quoted - INSTALLDIR="C:\Program Files\..." - so it rejected the line, printed usage, and exited non-zero. This file already carried the rule, next to the Python target: "Never put a space in a path this installer controls." I broke it setting the 8.4 path. Two fixes, because one of them alone leaves the trap in place: - $MysqlRoot is now C:\MySQL84, space-free like C:\Python314. The MySQL client search paths in the installer, the preflight and the operator console all look there first, keeping backups working against the bundled server. - Invoke-Native now quotes PROPERTY=value correctly, so passing a spaced path explicitly no longer produces an unparseable command line. tests/test_installer_defaults.py fails if an installer-controlled path default ever contains a space again. |
||
|
cproudlock
|
5321649e02 |
fix(installer): stop blocking the wizard on things the installer itself installs
The preflight page began refusing to continue while any check was failing, which is right for something the operator must go and fix. HttpPlatformHandler was marked FAIL when absent - so on a server without it the wizard stopped dead, telling the operator the server was not ready, over a module the bundle carries and stage 4 installs a few pages later. The only way forward was to go and install by hand the exact thing the installer was about to install. It is now INFO: reported, not blocking, matching how URL Rewrite is already handled. Nothing the installer SUPPLIES may block the wizard, and tests/test_installer_defaults.py now fails if that rule is broken again. The site-port conflict check is downgraded from FAIL to WARN for the same class of reason: it runs before the operator reaches the Address page, so it tests the DEFAULT port rather than the one they intend to use, and blocking refuses an install over a conflict the very next page lets them resolve. Genuine blockers are unchanged - no IIS, no WebAdministration, wrong Windows edition or architecture, no disk, and the MySQL 5.6 index flags. Those the operator really does have to fix first. |
||
|
cproudlock
|
4a8bd138a9 |
feat(import): load a site's data from spreadsheets
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
|
||
|
cproudlock
|
f72813ed9c |
feat(installer): bundle the database - MySQL 8.4 LTS, not 8.0
The bundled-database option could not actually be built. Stage 0 looks for mysql\mysql-8.0.x-winx64.msi, and Oracle no longer publishes a standalone server MSI for 8.0 - every 8.0.x returns 404. What remains for 8.0 is the MySQL Installer bundle, which is an installer-manager: 'msiexec /i INSTALLDIR=' would install THAT rather than a database, and stage 0 would then fail on a missing mysqld.exe. MySQL 8.0 also reached end of life in April 2026, so bundling it would have put an unsupported database on every new site. 8.4 LTS still ships the standalone MSI (129MB, which is what the '125MB' note in stage 0 was written against) and is supported into 2032. Defaults follow it: install root MySQL Server 8.4, service MySQL84. The operator console still looks for an 8.0 install path as a fallback, for sites already running one. Also bundles mysqlclient\ - mysql.exe and mysqldump.exe with the two OpenSSL DLLs they actually import, 20MB rather than the 51MB of debug and auth-plugin libraries the archive ships. Stage 2 stages it onto the server, so a site whose database is on ANOTHER host can still take the pre-upgrade backup that every upgrade depends on. That was the gap the preflight had started warning about. Bundle is now 221MB. |
||
|
cproudlock
|
8d9d1d3439 |
test(docs): skip the publishability gate where there is no docs/ to check
The GitHub backend job failed on test_there_are_docs_to_check, correctly. docs/ is stripped from the published repository - it lives in the wiki on that side - so on the mirror the glob matched nothing and the guard fired exactly as designed. An absent docs/ and a glob that silently matches nothing in a tree that HAS docs are different conditions, and the test conflated them. The module now skips when the directory is not there at all, and the guard still fails when it is there and empty. Verified all three ways: 7 pass here, 7 skip in a docs-less checkout, and the guard still fails against a docs/ containing no markdown. The gate has to ship rather than be excluded from publication, because the published tree is where the GitHub CI that would catch a regression runs. |
||
|
cproudlock
|
2073d0dbe8 |
build(export): purge stale generated paths from the publication tree
rsync --exclude also PROTECTS a path from --delete, so anything that reached the publication tree before its exclude existed stayed there permanently - invisible to the sync and surfacing only as a scrub-gate failure. That cost two rounds of 'add the exclude, still fails' on the installer bundle and again on .pytest_cache. The generated paths are now purged before the sync, so adding an exclude is sufficient on its own. tests/test_docs_publishable.py assembles its search terms from fragments: a file containing the literal strings the scrub greps for tripped that scrub on itself. Excluding the file from publication would have removed the check from the repository it protects. |
||
|
cproudlock
|
ee083ea80e |
docs: stop publishing internal references to a public wiki
docs/ is excluded from the code bundle and its scrub gate, because it goes to the GitHub wiki instead - via a generator that has no gate at all. So the one part of the repository written in prose, by people, about internal infrastructure, was the one part nothing checked. What was reaching a public wiki: the internal git server's URL and hostname, .gitea workflow paths, developer home directories in the GE-Enforce cutover reference, and a dev database root password inside a copy-pasteable command in the import guide. All replaced with neutral equivalents. tests/test_docs_publishable.py is now the gate, at the source, in CI - a wiki page cannot be un-published, so catching this after the fact is not good enough. PROJECT-REVIEW.md also referred to internal tooling by name throughout; those references are generalised. It remains an internal candid assessment of this project that is nonetheless published, which is worth a separate decision. |
||
|
cproudlock
|
bec138f5ac |
build(export): keep the installer's staged bundle out of publication
The sync walks the working tree rather than git, so deploy/windows/installer/ bundle came through despite being gitignored - about 100MB of build output containing a copy of the whole application tree, the wheels and the vendor installers. Its copies of config.py and requirements.txt then tripped the scrub gate, which is the only reason it was noticed. Excluded along with the installer's other generated files. Note that rsync --exclude also protects a path from --delete, so a copy already in the publication tree has to be removed by hand once. |
||
|
cproudlock
|
e158cb21f9 |
deps: keep build-machine paths out of the lockfiles
The '# via' annotations recorded the absolute path of the temporary file the lockfile was compiled from, which is meaningless to anyone else and does not belong in a published artifact. They now read 'requirements.in', which is where these requirements actually come from. Pins and hashes are unchanged - verified by a hash-checked dry-run install. |
||
|
cproudlock
|
2c415a1712 |
fix(installer): correct a false security claim, and clear the should-fix list
CLIENT IP / SPOOFABILITY. docs/geenforce-api-cutover.md claimed that removing the IIS rewrite rule made the allowlist fail closed and that it does NOT become spoofable. The opposite is true. IIS never sets X-Forwarded-For on its own; the rule is the only thing that does. Remove it and IIS still forwards whatever X-Forwarded-For the CALLER sent, waitress trusts it because it arrives from 127.0.0.1, and remote_addr becomes attacker-controlled - so a token-less caller can fetch manifests from anywhere on the network. The document and the _trusted_client_ip docstring now say so, waitress runs with --trusted-proxy-count=1, and stage 5 checks the rule is actually live rather than assuming it. The wizard question is rephrased to something an operator can verify with their network team instead of guessing at. NON-ASCII. The style gate only ever checked .py/.vue/.js/.ts, so documentation accumulated em-dashes, arrows and box-drawing characters against this repo's own convention - including in files added this week. Cleaned, and the gate now uses INCLUDES_ALL so Markdown, JSON and YAML are covered. PLUGIN DEFAULTS. The wizard pre-ticked measuringtools and printedparts, both of which ship default_enabled=false, so every site taking the defaults installed and enabled them against their manifests. Inno has no JSON parser so the list must be hardcoded, but tests/test_installer_defaults.py now fails when it drifts. UPGRADES. The payload copy merges, so a plugin dropped from a site's profile kept its code forever - which defeats a lean build and leaves core's optional-import guards succeeding for a plugin the site no longer has. Stale plugin directories are now deregistered and removed before the copy. add-plugin used 'plugin install', which for the five default_enabled=false plugins left them installed but DISABLED - and printed a green success line anyway. It now goes through apply-profile, and the success line is gated on the exit code. Invoke-Flask records its own exit status, because $LASTEXITCODE keeps a stale value when flask.exe is missing and no native command runs. CHARSET. The utf8mb4 compiler hook lived inline in migrations/env.py, so it covered the CORE chain only: plugin baselines inherited the server default, which on a latin1 server means two charsets in one database. It is now shopdb/utils/mysql_charset.py, imported by both, and preflight reports the database's default charset. BACKUP HONESTY. The dump was described as 'all of your asset data'. Uploaded branding and floor-map images live in instance\ on disk, not in the database, so a restore from the .sql alone comes back with no map. backup now archives instance\ alongside it and says both are needed. VERSIONING. AppVersion was hardcoded at 0.9.0 while the product, the frontend and the newest tag said 0.7.0 - and 0.9.0 collides with a retired contract version. Both builders now generate version.iss from shopdb/__init__.py. Smaller: rollback overwrites .env before deleting it, as uninstall already did; appcmd unlocks are scoped to this site's location rather than server-wide, with the wide unlock as a fallback; DEVELOPMENT-SETUP says Python 3.14; the README plugin list gains printedparts; prune-schema --force is documented as first-provisioning-only; HTTPS is documented as not-the-default with the steps to add it; the DBA SQL is on the wizard's database page; the features page says unticking does not remove an installed feature; and the installer README states that bundle-lock cannot vouch for the exe itself - that needs signing or an out-of-band hash, neither of which is wired up. |
||
|
cproudlock
|
aea2905de0 |
fix(installer): stop it lying, stop it leaking, and make it findable
Nine fixes from a review of the installer against its actual audience: DT leads at sister sites who are not Windows, IIS or Python specialists and who will lean on an AI assistant to get through it. TRUTHFULNESS. The preflight was advisory - an operator read 'IIS is not installed', pressed Next, answered five more pages and the install died partway through with Python already on the box. The results page now blocks while anything is failing, repaints on every run instead of latching after the first, and offers 'Check again' so a fixed problem does not mean starting over. On failure the wizard said 'Nothing was left running', which is false in every path because the stages run with -OnFailure never: it now says the server is part-configured, that re-running is safe, and how to remove it. The final page no longer reads 'ShopDB-Flask is ready' after a failed install. SECRETS. The generated MySQL root password went to Write-Host in a process the wizard runs hidden - so nobody saw it - and stdout is forwarded into the setup log operators are told to send to support, so it was permanently recorded for everyone who did not need it. It now goes to an ACL'd file. Database dumps, which contain every user password hash, landed in a ProgramData directory readable by every user on the box; the directory is now locked at creation. UPGRADES ON REMOTE-DATABASE SITES. mysqldump was looked for only under local MySQL install paths, so a site whose database is on another host silently skipped every pre-upgrade backup - after stage 2 had already stopped the pool and replaced the tree. Find-MysqlTool now prefers a client shipped in the bundle, stage 2 stages it onto the server, preflight reports when it is missing, and mysqlclient\ is an optional locked payload. UNINSTALL. A subpath install is an IIS Application, not a site; removing only the site left the application pointing at a deleted directory, so the parent site - at West Jefferson, the live classic ASP - served 503 on that path forever while Add/Remove Programs reported success. Uninstall now reads MOUNT_PATH and removes the application. The firewall rule was created as "$SiteName $SitePort" and removed as the literal 'ShopDB-Flask 8090', which matches nothing. DAY-2 TOOLING. Every shortcut now passes -AppRoot and -SitePort, and the console forwards them through its own elevation and 32-bit relaunches instead of discarding them - a non-default directory or port made it report a healthy site as broken, from a shortcut the installer wrote. 'Open ShopDB-Flask' resolved to a hardcoded localhost:8090 that was wrong for every subpath install; it now asks the console, which reads the address the installer recorded, and no longer demands administrator to open a browser. SMOKE TEST. The parent-site port lookup filtered for an http binding and defaulted to 80, so an https-only parent site failed a working install with a red dialog. DOCS AND /api/docs. The installer was invisible: nothing in docs/, README.md or CLAUDE.md mentioned it, so a DT lead or their assistant landed on the manual IIS runbook and hand-built the very server the installer then refuses to upgrade. docs/INSTALL-WINDOWS.md and docs/OPERATE-WINDOWS.md are now the canonical route, the two manual runbooks are bannered as reference-only, README and CLAUDE.md route by target, and llms.txt tells an assistant which document to follow and to ask for 'check -Json' before diagnosing. Both ship on the server, along with openapi.json and llms.txt - without those the self-hosted /api/docs was broken on every installed box, which matters most to the sites least able to debug it. Stage 5 now checks it actually serves. shopdb-admin.ps1 gains 'check -Json': one structured, secret-free block covering version, publishing method, IIS state, HTTP reachability, database, Python version, plugins and errors. That is the cheapest useful answer to 'the operator will ask an LLM' - it works with no infrastructure, which a install-time MCP server could not. |
||
|
cproudlock
|
e58f376643 |
fix(installer): the adopt-existing guard never fired
Stage 4 decided whether the IIS objects it was about to reconcile were its own by testing for .installed-version. Stage 2 writes that file, and stage 2 always runs first in a '-Stage all' install - so by the time the guard looked, the stamp it had just written made every server look like one this installer built, including the hand-built ones the guard exists to protect. Stage 2 now records whether a stamp was present BEFORE it writes its own, and stage 4 reads that observation. Running stage 4 alone still tests the file, which is correct there: no stage 2 has run to disturb it. Found by review, not by test - the guard has no coverage, because exercising it needs a live IIS. |
||
|
cproudlock
|
3606d8d696 |
feat(sbom): ship a CycloneDX bill of materials with every build
An air-gapped site cannot be scanned from anywhere else, so when a CVE lands the only way to answer 'is that component here, and at what version' was to RDP in and go looking. The frontend was the real blind spot: nothing recorded which version of leaflet, dompurify, jspdf or html2canvas ends up inside the compiled SPA. scripts/generate_sbom.py emits CycloneDX 1.6 covering both ecosystems - every pin in requirements.txt with the sha256 the installer enforces, and every package in package-lock.json. Build-only npm packages are marked scope 'excluded' rather than dropped, so 'not here' stays distinguishable from 'not looked for'. Dependency edges are real: uv's '# via' comments give the Python graph and package-lock gives the npm one. Hand-rolled rather than cyclonedx-py plus cyclonedx-npm because both inputs are already pinned and committed - this is a format translation, not a scan - and because the build box may be a work PC with nothing but Python and Node. It is deterministic by construction: same inputs, byte-identical output, so regenerating does not churn. Staged into the application tree by both builders, so it installs onto the server with the app. shopdb-admin.ps1 verify reports it and searches it by component name, which is the question actually being asked. Packages appearing at several depths in package-lock (node_modules/vite and node_modules/vitest/node_modules/vite) are merged, and a copy reachable outside the dev tree makes the component count as shipped. Emitting both produced duplicate bom-refs, which CycloneDX forbids and scanners reject; getting the dev merge backwards would have hidden a shipped package from a CVE search. Not covered by bundle-lock.json on purpose: its provenance is git, not the third-party payload. |
||
|
cproudlock
|
1bf3cb2e1c |
feat(installer): refuse to damage an installation it did not create
Two guards for a server deployed by hand, which the West Jefferson production box is. An existing venv is reused, which is right for a repair or an upgrade of an install this made, and wrong when the venv belongs to a different Python. The wheelhouse is tagged for one minor version, so pip finds no candidate for the compiled packages and dies partway through - after Python has been installed and the application tree replaced. The two versions are now compared up front and the run stops with both numbers and what to do about it. Switching deployment method removes the other method's IIS artifact. That is correct when this installer owns both and dangerous when it does not: a wrong -MountAlias would call Remove-WebApplication on a live mount with no prompt and no error, and the first sign would be the site returning 404. It now refuses unless a version stamp shows this installer made the install, or -AdoptExisting is passed, and the refusal lists exactly what it would have removed. |
||
|
cproudlock
|
13d831eb90 |
feat(installer): require the wheelhouse to satisfy requirements.txt, and lock the real payload
The lock records what IS in the wheelhouse, not what the application NEEDS, so an incomplete wheelhouse was locked, blessed and shipped - and only failed on an air-gapped server. That is not hypothetical. Assembling the wheelhouse anywhere other than Windows silently omits colorama, a win32-only dependency of click, because pip evaluates environment markers against the machine doing the downloading rather than the machine being targeted. The bundle built here was short exactly that one wheel. Both verifiers now cross-check wheels/ against the staged requirements.txt, ignoring markers, since a requirement guarded by sys_platform == 'win32' is precisely the one that must be present. Names are normalised to PEP 427 wheel form, so mysql-connector-python matches mysql_connector_python. bundle-lock.json is the first real lock: 42 files, cp314/win_amd64 - 39 wheels, Python 3.14.6, HttpPlatformHandler 1.2 and URL Rewrite. MySQL is absent and optional; a site choosing the bundled-database option adds it and re-locks. The naming gate now skips the installer's build output. It contains a staged copy of the application plus a second SPA build under dist-subpath, which --exclude-dir=dist does not match, so a staged bundle failed the gate on vendored minified JS nobody in this repository wrote. |
||
|
cproudlock
|
44237b5cbd |
feat(installer): bundle URL Rewrite, ask where client IPs come from, verify installs
IIS does not set X-Forwarded-For on its own and HttpPlatformHandler connects from loopback, so without a rewrite rule every client reads as 127.0.0.1. The GE-Enforce IP allowlist, the dashboard visitor-location lookup and per-host login rate limiting all stop working, silently. The rule needed URL Rewrite, which the installer told operators to download - from an air-gapped server. URL Rewrite now ships in the bundle, and the wizard asks which case applies, because the two answers are mutually exclusive. Directly exposed: install it and set X-Forwarded-For from REMOTE_ADDR, which is what stops a client spoofing its own. Behind a proxy: leave the rule off, since REMOTE_ADDR is the proxy and applying it would discard the real client IP. The rule is enabled by deleting two explicit marker lines rather than by a regex over the surrounding comment, so editing that prose cannot silently disable it. An existing web.config is no longer overwritten. It is the one file on a server that legitimately carries hand-edits, and replacing it reverted them without a word - on a server where the X-Forwarded-For rule had been enabled by hand, that alone would have turned the GE-Enforce IP allowlist off. The installer reports what it found instead. pip now runs with --require-hashes and --only-binary=:all:. Hash-checking is requested explicitly rather than inferred from the lockfile, so shipping an unhashed requirements.txt fails loudly instead of quietly dropping the check. shopdb-admin.ps1 gains a verify command: which bundle this server was installed from, and whether the installed packages still match what shipped. The .iss states its compiler floor. WizardStyle uses the built-in windows11 custom style, which needs Inno Setup 6.6.0; older compilers now fail with that sentence rather than 'WizardStyle is invalid'. |