wj
90 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
a46058a4dd |
Close the last three restore-gate gaps, and make the gate test the real gate
Three defects that all shared a shape: the restore path looked checked and was not. An unreadable data revision was treated as a legacy set. backup.sh discards the mysql error when reading alembic_version, so a set could be written with an empty datarev; the compatibility check then fell through to the code-to-code head comparison - the very check datarev was introduced to replace - and badged the set restorable on no evidence at all. A manifest that CARRIES the key and left it empty is now refused, while one with no key at all is still the genuine legacy case and keeps the fallback. backup.sh records the failure in run history rather than passing it over in a log line, since the set it just wrote cannot be restored. Dumps were bound to the schema they came from. mysqldump --databases bakes CREATE DATABASE and USE <name> into the file, so restoring onto a site whose MYSQL_DATABASE differs loaded every row into the dump's schema, left the live database untouched, and reported success. New sets are dumped without it and load into the schema named at restore time. Existing sets are still bound, so the gate reads which schema they name and refuses a mismatch instead of restoring beside the live data. Also dropped --no-tablespaces into the main dump: it needs the PROCESS privilege the app user does not have on MySQL 8. The gate's own test carried a copy of the gate. test-restore-datarev.sh re-implemented the predicate in its own shell functions, so it would have stayed green through any drift in restore.sh - including the plugin-chain hole fixed in the previous commit, which it could never have caught. restore.sh now takes RESTORE_CHECK_ONLY, which stops it immediately after the gates having touched nothing, plus overridable paths for the migration and plugin trees. The test drives the shipped script and asserts on its exit code, so the verdict under test IS the verdict that ships. Confirmed non-vacuous by planting a violation: disabling the plugin revision check turns the suite red. Nine cases now, up from five, covering both plugin-chain directions, a plugin the build does not have, and the schema binding. Moving the schema check earlier was prompted by that test: it had been placed after the safety copy, so it needed a live database to reach a verdict it can reach for free. It sits with the other gates now and stops before anything is written. Round-tripped against a real stack after the dump format change: backup, mutate, restore, data back to backup-time state, and the new dumps confirmed to carry no schema name. Suite 2607 passed / 9 skipped. Scope: product |
||
|
cproudlock
|
e2705d732a |
Gate a restore on every schema chain, not just the core one
The compatibility gate read alembic_version and nothing else, but plugin schemas move independently of core. Roll the code back to an older plugin revision and the core head can be identical, so a dump carrying plugin tables the code has never heard of passed the gate, loaded - destroying what was there - and only then failed in 'flask plugin upgrade-all' with "Can't locate revision". A failed restore that has already replaced what it was restoring over is the worst outcome this feature can produce, and it was reachable from an ordinary rollback. Backups now stamp the applied revision of every alembic_version_<plugin> table alongside the core one, read from the dumped database the same way. restore.sh checks each against that plugin's own versions directory and refuses before touching anything, naming the plugin and the revision to deploy past. A plugin in the dump that this build does not have at all is refused for the same reason. The list endpoint's compatibility badge asks the same question, so the UI cannot offer a set the restore will reject. Sets taken before the stamp existed carry no dataplugins and keep their old core-only verdict. Their plugin revisions are unknowable, and making every existing set unrestorable to close this would be a worse trade than the gap. Proven against a real stack, reproducing the exact case: core head identical, endpoints data stamped at a revision the build does not have. The restore refused, the live row was still there afterwards, and it stopped early enough that the pre-restore safety copy was never needed. The same set with its true revision restored normally. Suite 2605 passed / 9 skipped, with the gate covered both ways plus the legacy fallback. Scope: product |
||
|
cproudlock
|
fa6b99bf91 |
Finish a restore without a browser, from the set that was asked for
Closes the three defects in the imported restore path, at Ben's request. Each was proven against a real stack before and after: two backup sets with different data and different instance files, then a restore of the OLDER one, which is the case that was broken. A restore no longer depends on a tab staying open. The backup sidecar has no Flask, so it loads the dump and leaves a sentinel for the api container to run the migrations. The only thing that ever acted on that sentinel was the poll loop of the modal that queued the restore, so closing the tab - or letting a token expire - during the minutes a load takes left the data at the old revision under new code, every query on a newer column failing, and nothing retrying. A watcher in the api container now claims and runs it. The endpoint no longer runs the upgrade at all; it only makes sure something is pending, which is also how a failed upgrade is retried. That takes Alembic out of the request path, where it could outrun the gunicorn worker timeout and be killed mid-chain, leaving a half-applied revision the next attempt trips over. The claim is a rename, because rename(2) is atomic: several workers and a browser poll can race the same sentinel and exactly one wins. A failed upgrade parks itself as .upgrade-failed rather than re-arming, since putting it back would rerun the same broken chain on every poll forever. Instance files come from the set being restored. The restic call asked for 'latest' regardless of which set was named, so restoring anything but the newest paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json - a mismatch nothing downstream could detect. Now scoped by the set's own tag, which backup.sh has always written. The tree is replaced rather than restored over, so files created after the snapshot are gone afterwards: a restore has to put the site back to the snapshot, not merge into it. Done as a staged swap rather than 'restic restore --delete --target /', because that aims a delete pass at the container root and the cost of getting its filter wrong is the whole filesystem. Nothing destructive happens before a copy is taken. The dump carries DROP TABLE per table with no transaction around it, and the instance tree is replaced wholesale, so a wrong set or a failure part-way through left nothing to go back to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball and a README saying what it is and how to load it - and refuses to touch anything if that copy cannot be written. Two more found while proving the above, both of which stopped the overlay working at all: * Every script under scripts/backup/ was committed non-executable, while every other .sh in the repo is 755. The compose entrypoints are /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup and watcher containers could not start from a clean checkout on any machine, and nothing could fix it at runtime. * Dockerfile.backup could not build without the corporate CA staged. A plain COPY of the gitignored cert fails outright; the trailing-glob form does too, because BuildKit fails a COPY whose pattern matches nothing - which is exactly the case being handled. Copying the directory and testing for the file inside it is the form that works both ways. Verified by building the image with no cert present. Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic claim, the parked failure and the endpoint no longer running the upgrade. Scope: product |
||
|
cproudlock
|
83190422c9 |
Act on the review of the wilm import
Six findings from a four-agent review of the imported work, plus the contract bump the import owed. Nothing here changes what the imported features do. The contract version was not bumped for a change that widens what a plugin may declare. The asset-panel vocabulary gained two tab groups (serviceprocess, reports), a datetime column format, a map.body key and a tabs section that renders as a table. 0.29.0 bumped for exactly this class of change when the optional group was added, so this is 0.31.0 with a ledger row. All four are optional and resolve server-side when absent, so a plugin declaring none of them is unaffected. Site Backups now answers honestly where it has nothing behind it. The overlay is a compose profile: a separate container writes sets onto a shared volume and the api container only reads them. A Windows/IIS install runs no such container and the default root resolves to a drive-letter path nothing creates, so Back Up Now either failed on PermissionError or reported a run queued that would never happen, under a banner reading "No verified backup yet" - which describes a backup that failed, not a feature that was never installed. The three write routes refuse with the same 404 shape the configuration-policy routes use for a disabled feature, the list and status routes report whether a store exists, and the page says so instead of offering buttons. Two listAll callers still read a response that is not there. listAll resolves to a plain array through fetchAllPages, so response.data.data throws: PC Types settings rendered empty, and in Configuration Policies the throw happened inside the shared try, so PC types, applications, the action catalog and monitors were all left unassigned by one bad read. The agent package upload cannot succeed on IIS as shipped. MAX_CONTENT_LENGTH went to 256 MB to admit the ~69 MB tray binary, but IIS request filtering caps a body at 30,000,000 bytes and answers 413 before waitress sees the request, so web.config needs the matching requestLimits. The backup image could not be built by anyone outside a TLS-intercepting network. Dockerfile.backup did a plain COPY of a gitignored corporate root CA, and a COPY of a missing file fails the build. The trailing glob makes it a no-op when the cert is absent. endpointstates.domain was the one new string column written without a clamp. Under STRICT_TRANS_TABLES an over-long value raises Data too long and aborts the whole check-in, so a single malformed report would stop a machine reporting telemetry and events entirely. Not fixed here, and left with upstream: the restore path finalizes its schema upgrade only from a browser poll loop, so closing the tab mid-restore leaves the data behind the code; it restores instance files from the newest snapshot rather than the set that was asked for; and it takes no safety dump before the destructive load. All three are in the Docker-only overlay, which this site does not run, and none can be validated here. Scope: product |
||
|
cproudlock
|
6e36df841e |
Import the wilm endpoint telemetry, PC Detail rework and backup overlay
Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at
|
||
|
cproudlock
|
45acaf0b8f |
Resolve an asset type by name, and send a map position with its level
Wave 5, the asset forms. Both defects are the same mistake in different clothes: a form deciding something it should have been told. Four forms carried a hardcoded asset-type id - const COMPUTER_ASSETTYPEID = 2, and the same for machines, printers and network devices - to tell CustomFieldsInputs which type's fields to render. Those ids come from seed order. On the instance they were written against, computer was 2; on a freshly seeded database computer is 1 and 2 is network_device. So the PC form asked for another type's custom fields, rendered them, and dropped everything typed into them behind a success toast, because the values were keyed to a type the record does not have. Verified in a browser against a correctly built dist: the form now requests assettypeid=1 on an instance where computer is 1. The name is the stable thing - assettype is a fixed vocabulary and the API already publishes the mapping - so composables/assetTypeIds.js resolves it once per page load. A failed load leaves it undefined, which renders no custom-field panel: a missing panel is honest, a guessed id is what this replaces. Three of the five forms that emit a map position sent mapx and mapy without levelid. ADR-017 is explicit that a position without its level cannot be drawn on the right floor plan, so a picked pin saved a null level and rendered nowhere - while the form displayed "Position: x, y" back, because it kept the value it had never sent. The ADR-017 gate should have caught that and did not: it globbed '*.py' only, so it policed what the server emits and never what the browser sends. It now covers .vue and .js too, and accepts both quoting styles. Verified by planting: drop levelid from PCForm again and the gate names that exact line. Five composable tests, including a fixture with a DIFFERENT seed order - the case the constants got wrong. Still open from this area, and deliberately not in this commit: the PC form writes an INHERITED location back as the record's own. Fixing it needs the API to distinguish an owned locationid from a derived one, which is a contract change rather than a form fix. Scope: product |
||
|
cproudlock
|
377cc69264 |
Make a partly-undone import retryable, and refuse a plan it cannot deliver
Five findings from the review of the WINC import, plus the rendering each fix needed to be usable. An undo that hit a conflict wrote status 'undone' anyway, and undo_batch returns immediately for an undone batch. So the one entry that needed a retry was the only one that could never get it: the human resolves the conflict, clicks Undo, and gets a no-op. A conflicted undo is now 'partially-undone', undoneat is stamped only when the batch is genuinely finished, and the batches table offers "Retry undo" for that status. The column was String(16) and the new value is exactly 16 characters, so migration 7d38 widens it to 32 rather than rely on an exact fit. _UNDO_REFERENCES listed only assets, so a business unit this batch created was hard-deleted even when something else still pointed at it. Two of the three missing referrers are real FKs (dashboarddefaults.businessunitid, businessunits.parentid): MySQL raises and the whole undo rolls back, so the batch cannot be undone at all. The third, notifications.businessunitid, has no FK, so the delete succeeded and left a notification pointing at an id that no longer exists - silently, on both engines. All four are listed and the guard is tested through the route. The reference guard also queried tables that need not exist. A lean site drops the tables of plugins it does not install (ADR-014), and notifications is plugin-owned, so on such a site the query raised 1146, rolled the undo back and left the batch permanently un-undoable. Absent tables are skipped; a table that does not exist cannot hold a reference. _PACKET_GROUPS reports on location, but apply_plan only ever creates business units - it sets locationid only when the location already resolved. A plan needing a new location asked the admin to acknowledge it, got the tick, wrote nothing, and reported nothing. build_packet now takes the groups the writer can actually create; anything else needed is 'uncreatable' and blocks at plan time, where the admin can act on it, rather than dying at commit on a message about acknowledging something the screen had no control for. The writer refuses the same case independently. Both the console packet and the import page name the blocking groups: setting blocking without printing the reason would leave the banner pointing at nothing. The web undo route caught ValueError only, so refusing a batch belonging to another importer surfaced as a 500. That refusal is new too: the journal is a general primitive and undo_batch does not filter on kind, so any other importer's batches were reversible through this endpoint, carrying WINC's references map - which lists none of that importer's tables, making every reference check pass and every delete unconditional. Nine tests. The two guards were verified by planting violations: dropping dashboarddefaults from the references map, and passing creatable=None, each fail the test written for them. Scope: product |
||
|
cproudlock
|
405a2ef919 |
Cap request bodies, close a hole in the gate, fix to_dict and two settings pages
Six fixes that were ours regardless of what the sister site does. Four came out of our own reviews; two you reported from the settings pages. MAX_CONTENT_LENGTH was set nowhere. Every upload route caps its own payload, but each of those checks runs after the body has arrived: the first touch of request.form or request.files makes Werkzeug parse the whole multipart and spool the file to a temp file, so a route can reject the content and still have paid for the transfer and the disk. Three separate reviews found that shape in three different routes, which makes it a property of the app rather than of any handler. 128 MB, clearing the 64 MB agent binary with room for the envelope. The naming gate excluded devnotes with --exclude-dir, which matches a directory of that NAME at any depth - so a force-added plugins/<plugin>/devnotes/ was invisible to every check in the file, ADR-015 included. Replaced with a path anchored post-filter. Verified by planting the same site literal in root devnotes/ AND in plugins/endpoints/devnotes/: the gate now reports exactly one, the nested one. Before, it reported none. BaseModel.to_dict walked __table__.columns and published every one, so a model without an override published whatever it carried - including columns added later by someone not thinking about this method. User has no override, so User.to_dict() returned its bcrypt hash. Nothing calls it in a response path today, which is luck rather than design. A blanket allowlist would empty the output of all 34 models relying on the default, so this pins a floor instead: named secret-bearing columns are never emitted, and a model may extend the set. Eight tests, including a sweep over every mapped model. ActionPicker's Esc handlers had .prevent but not .stop, so the event bubbled to the wrapper and closed the whole picker instead of returning to the monitor list. The branding page reported "error loading image" on the site and badge logos. Two bugs stacked. The load failure was real: the preview used the stored value as a raw src, and that value is root-relative, so under /shopdb or /ops it resolved against the SERVER root and 404d - siteSettings.js wraps every logo in withBase() and this page was the one place that did not. With that fixed two of three were still invisible, because the shipped GE mark is fill="white" for the dark sidebar and was rendering white-on-white. The preview tile now sits on a mid-tone checker so artwork of any colour reads against it. The floor-map levels table ran off the right with its cells crushed - nine columns of inputs, no scroll wrapper, and .data-table is styled nowhere. My first attempt reached for the global .table-container and made it WORSE: that class also sets white-space: nowrap, and a full sentence of hint text lives in one of the cells, so nowrap forced the table wider still. It now scrolls in a local wrapper, the hint wraps, and the name input has a width instead of collapsing to one character. Suite 2377, frontend 324, gate green. Scope: product |
||
|
cproudlock
|
28269e429a |
Import the WINC batches CLI and its README
One commit from the sister site, purely additive: a read-only `batches` command
so an operator working from the shell can find a batchid to undo, and a README
covering the four WINC commands. Two new files under scripts/site_imports/winc/,
nothing existing touched.
Verified rather than assumed: it changes none of the files fixed in
|
||
|
cproudlock
|
13be0ef90f |
Import the wilm agent store and WINC adoption work, with four fixes
Eleven commits from the sister site: a reversible Maximo/WINC machine-adoption import, a versioned agent binary store with upload and rollback, remote-BuildKit CI, tray version reporting, and UI fixes. Applied as a patch again - the published mirror rewrites hashes, so the branches have unrelated histories and git refuses to merge them. Reviewed by three agents before merging. Four things fixed here. THE PUBLISH WORKFLOW WAS DEAD. The new remote-BuildKit step in publish-images.yml sits at column 0 instead of indented under steps:, so the whole file is invalid YAML and GitHub would refuse it. Their entire GHCR distribution channel - which Start-ShopDBDev.ps1 and the compose files consume - was broken at the source and clearly never run. Indented; all five workflows now parse. THE IMPORT'S APPROVAL CHECK COULD NOT FAIL. commit() built the plan and the approval packet from the same upload and then asked the writer whether they matched, so packet_matches_plan compared a hash against itself. The documented contract - "a human approves a specific, frozen plan, and the writer can prove nothing drifted" - did not exist in the web path: an admin could preview file A and commit file B, gated only by the needs-create group NAMES they ticked. The preview's planhash is now required at commit and checked against a freshly rebuilt plan. Their tests committed without previewing, which is what let this through; they go through the real flow now, and three new tests pin it - all confirmed to fail with the check stubbed out. endpoints.agentpackage QUIETLY GREW. It used to mean "wrap the staged exe in a ZIP"; with the upload routes it also means "choose the executable every future package installs and runs as SYSTEM". Before this, staging needed filesystem access to the server. The permission description still said only "generate and download a package", so an admin granting it could not see what they were granting. Reworded. NOTHING RECORDED WHAT WAS STAGED, OR BY WHOM. The manifest held version, size and timestamp; re-uploading a version replaces the bytes in place under the same label. A sha256 of the stored file is now recorded, and the upload is audit-logged with it - after an incident there was previously no answer to who put that binary on the fleet. Also documented the nine new routes in docs/api-inventory.json (that gate cannot run on the mirror, so it falls to us at promotion time again), regenerated the spec and the project map, and kept our own docs where they conflicted: docs/ is scrubbed from the mirror, so their DEV-BUILD-TEST.md still had the dev-be channel mapping we corrected, and their PLUGIN-HOOKS.md was a new 45-line file rather than our 737-line canonical one. Their journal section is appended to ours. Contract version comes across as 0.30.0 for a genuinely additive surface. Suite 2369, frontend 324, gate green. Scope: product |
||
|
cproudlock
|
88a683dad3 |
Act on the five-agent review of the wilm import
Findings the reviewers verified, fixed here. The larger cluster they found is NOT fixed and is listed at the end. Frontend. reportImageUrl is called from the template, so it ran on every render with no in-flight guard: a report whose fetch was pending was re-requested, and because each resolution reassigns the map the whole grid re-rendered and re-fired the rest, costing up to N-squared requests for N thumbnails. Two blobs resolving for one key orphaned the first object URL where revoke could not reach it, and a failure was never recorded so a broken image retried forever. Guarded, deduplicated, failures remembered. Three CSS variables did not exist, so their hardcoded fallbacks always won: --bg-alt (the "No screenshot" tile, near-black in light mode), and --warning-text/-bg/-border (dark khaki on a dark modal). Replaced with real tokens. The DEV badge's literal #c62828 is var(--danger). Installer and launcher. The generated site installer ran docker compose with no -p, so the project name defaulted to the package folder - which carries the version. Installing the next version created a new project with new empty volumes and the site came up looking wiped, old data stranded under the old project. Pinned to shopdb-<channel>. Start-ShopDBDev deleted any existing -TargetDir that lacked a .git, recursively and without asking. For a script handed round as "save it anywhere and run it", a mistyped -TargetDir pointed at Documents would have destroyed it. It now only reclaims a directory that is empty or carries this launcher's own marker, and refuses otherwise. Dev secrets came from Get-Random, which is System.Random: a 32-bit seed, and all four values off one stream, so recovering one recovers the rest. They sign JWTs for a stack a pilot may point at real imported data. CSPRNG now, and the API port binds to loopback like the DB rather than every interface. Two of my own from earlier in this session. The operator-report size check sat below request.form, and the first access to that parses and spools the whole multipart - so it ran after the thing it prevents. Moved above. And the channel remap missed build-channel-installer.ps1 entirely, where it set BOTH channels to main: a "stable" package built from the trunk, the exact mislabel the remap was for. stable now resolves the newest v* tag. Also removed a zero-byte docs/ShopDB-Planner-Tasks-v2.csv that rode in. NOT FIXED - needs a decision, and it is the biggest thing the review found. Migrations 0006/0007 freeze the originating site's fleet convention into product seed: a shipped policy literally named "RPCA Shopfloor App Monitors", a hardcoded 'Shopfloor PC' type deciding who gets the shipped watch-list, and shipped rows the API refuses to edit or deactivate - including their launch paths. A site whose eDNC is on another drive cannot correct it and gets permanent false drift. Those are stamped migrations, so this gets harder to undo at every install. See the review notes before anyone upgrades a second site through 0006-0007. Suite 2314 passing, frontend 316, all three PowerShell files parse. Scope: product |
||
|
cproudlock
|
2e42972f44 |
Map the image channels to how this repo ships, and disarm auto-publish
The channel work assumed the branch model it was written against: upstream, main is a release branch and dev-be is the trunk. Neither holds here. main IS our trunk - 212 commits past v0.11.3 - and a release is a tag (ADR-007). dev-be does not exist at all, so the edge half would never have fired and the launchers would have cloned a missing branch. Taken literally it would have published our unreleased trunk to every site as :stable, which is the opposite of what the word promises to someone who has no way to check. Remapped throughout: tag v* -> :stable the thing that was actually released main -> :edge the trunk, DEV badge lit publish-images.yml keys the channel on github.ref_type = tag rather than on a branch name, so the mapping cannot silently drift back. Start-ShopDBDev.ps1, dev-bootstrap.ps1 and build-channel-installer.ps1 resolve stable to the newest v* tag via git ls-remote --sort=-v:refname, warning and falling back to main if a repo has no tags yet. DEV-BUILD-TEST.md follows. The push: trigger is commented out. Automatic publishing waits on someone confirming the GHCR package exists under this owner and deciding who may pull it; workflow_dispatch works meanwhile. Turning it on is one uncomment. All three scripts parse clean under pwsh, and the tag-selection expression was run against this repo rather than eyeballed. Per the standing rule, the runtime paths still want a pass in the win11 VM before anyone leans on them. Scope: product |
||
|
cproudlock
|
27b2817fd7 |
Import the wilm endpoint-monitor and channel-distribution work
Everything on origin/wilm that is not on the published main, applied as one
change because the published history has rewritten hashes: per-commit
cherry-pick conflicts against our main even though the squashed diff applies
clean.
Contents: endpoint monitor catalog + policy attachment + warn-only drift
reconciler, operator issue reports from the tray, agent launch-path allowlist,
and the GHCR channel-image distribution path for dev workstations.
NOT READY FOR main. Open items:
- 10 new routes are absent from docs/api-inventory.json, so
test_api_inventory_parity fails. That gate SKIPS on the published mirror
(the test ships, its data file is scrubbed), which is why wilm never saw it.
- dev-bootstrap.ps1 and Start-ShopDBDev.ps1 hardcode the ge-aero GitHub org.
- plugins/endpoints/plugin.py carries a whitespace-only re-indent.
Scope: product
|
||
benjamin.baker
|
7c09f1e4ca |
Add a standalone endpoint agent uninstaller
Reverses everything the installer and site-agent generator create: stops and deletes the service, kills agent/tray processes, removes the tray Run key and the HKLM\SOFTWARE\GE\ShopDB credential hive in both registry views, drops the Defender path exclusion, and deletes the install directory and ProgramData tree. Every step is best-effort and idempotent so it is safe to re-run and safe on a partially installed host. |
||
|
benjamin.baker
|
1fa005eb8a |
Green up
Tracking docs/openapi.json (so the app can serve the spec at runtime) makes docs/ exist on the published GitHub mirror, which defeated the skipif(not DOCS.is_dir()) guard in the docs publish/version tests: they then ran against a prose-less checkout and failed on CI. Key the skip off docs/*.md prose instead, matching the canonical published-checkout definition in scripts/check-gates-ran.sh. Also: the OpenAPI generator emitted a trailing-slash path (/api/docs/), which a strict proxy like IIS treats as a redirecting resource a subpath-mounted client cannot follow; normalise trailing slashes in _norm_path and correct the committed spec. And allow docx (python-docx) as an undeclared import: it is used only by scripts/docs/generate_docx.py, a developer tool that never runs on a deployed site. |
||
|
cproudlock
|
96a4a38c8c |
Make the legacy import safe to resume, and aimable without editing it
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed. |
||
|
cproudlock
|
b8fc2e9e33 |
Upgrade the eight shipped dependencies carrying advisories
scan_sbom.py reported 50 advisory matches against shipped components, 16 of them at or above HIGH. This clears all of them: the scan now reports 0 shipped findings and the gate passes. pyjwt 2.12.1 -> 2.13.0 idna 3.13 -> 3.19 axios 1.13.2 -> 1.20.0 (29 advisories) postcss 8.5.6 -> 8.5.26 dompurify 3.4.11 -> 3.4.14 nanoid 3.3.11 -> 3.3.18 follow-redirects 1.15.11 -> 1.16.0 form-data 4.0.5 -> 4.0.6 Every one is an in-major bump inside a range the manifests already allowed, so this is a lockfile refresh rather than a migration. The Python side was compiled with --upgrade-package for each name so nothing else moved; the diff is two version lines and their hashes. nanoid is worth recording: it is transitive under postcss at ^3.3.11, and its advisories are fixed on both the 3.x and 5.x branches. 3.3.18 is the fix that is actually reachable. The declared floors for axios and dompurify were raised to the versions in the lockfile. They were ^1.6.0 and ^3.4.11, which still nominally admitted the vulnerable range - npm resolves to the highest match so an install was never going to pick one, but the manifest should state the minimum safe version rather than rely on that. axios 1.20 adds optional proxy-agent support, which pulls https-proxy-agent and agent-base into the tree. Both are Node-only and neither reaches the browser bundle; the build was checked for them. The main chunk grew 256kB to 271kB (85.7kB to 90.5kB gzipped) from the rest of the axios delta. The PyJWT entry in the allowlist was removed rather than kept. It stopped matching the moment the upgrade landed, and an allowlist that accumulates entries for advisories that no longer apply is one nobody rereads. The reasoning is preserved as the worked example in docs/DEPENDENCY-SECURITY.md. Verified: full suite 2161 passed, frontend 273 passed, build clean, npm audit --omit=dev reports 0 vulnerabilities. |
||
|
cproudlock
|
1cc2a6b782 |
Recommend the smallest upgrade that clears an advisory, not the highest
The rollup took the highest fix version across a component's advisories. That is wrong whenever an advisory is patched on more than one release branch. nanoid is the case that exposed it. Both advisories against it are fixed on two branches - 3.3.16 and 5.1.16, 3.3.18 and 5.1.6 - and the rollup recommended 5.1.16 with a MAJOR warning. But nanoid is transitive, pulled in by postcss at `^3.3.11`, a range no 5.x release can satisfy. The recommendation was not merely a bigger job than necessary, it was unreachable: the actual target is 3.3.18, a patch inside the existing range. Per advisory the target is now the lowest published fix above the version in use; per component it is the highest of those, since one upgrade has to satisfy every advisory against it. Fixes on older branches are skipped rather than offered as a downgrade - axios advisories list 0.30.x fixes that would have been a downgrade from 1.13.2. This also gives the MAJOR label meaning. It now fires only when no fix exists on the current branch, rather than whenever some branch happened to be higher. |
||
|
cproudlock
|
794bb4668f |
Add vulnerability scanning on top of the SBOM
The SBOM was generated and installed onto every server, and nothing read it. "Are we exposed to this advisory" was still a person going and looking, which meant it happened when somebody remembered to. scripts/scan_sbom.py queries OSV.dev by purl - no account, no API key, no agent, no vulnerability database to maintain on a box that cannot reach the internet. The SBOM is already the input OSV wants, so this is a lookup against what shipped rather than a second scan of the tree that could disagree with it. Three decisions worth recording: Shipped and build-only findings are both reported, but only shipped ones can fail a build. Two thirds of the components here are build tooling the SBOM already marks scope 'excluded', and the highest-severity match in the whole tree today is a CRITICAL against the test runner. A gate that fails a release over that trains everybody to bypass it, and then it catches nothing. Severity is taken from the advisory, never derived. Where there is only a CVSS vector the report says UNKNOWN, which is shown and never gates. A guessed score is worse than an honest gap because somebody would act on it. The allowlist is a record rather than a mute button: each entry carries the reason, the accepter, the date, and what would invalidate the reasoning. The first entry is a worked example - a PyJWT algorithm-confusion advisory that needs both mixed algorithm families and a raw JSON JWK as the key, and this application configures neither. The report ends with a per-component upgrade rollup, since that is the part somebody acts on. A target on a different major version is labelled rather than listed beside patch bumps as though the two cost the same. Current state: 8 shipped components have fixes available, none of them exploitable through a path this application opens today. |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
f4b17d0fa9 |
Prove the gates ran, and ship the spec the app serves
Two faults with one cause: docs/ is excluded from publication because the prose
becomes the wiki, and two things under docs/ are not prose.
shopdb/core/api/docs.py reads docs/openapi.json AT RUNTIME to serve /api/docs.
Excluding it meant every site installed from the published repo shipped the
ReDoc page, loaded it, and got {"error":"openapi.json not generated"}. The spec
now publishes - it is a product file that happens to live under docs/. The prose
still goes to the wiki and nowhere else, and docs/api-inventory.json stays out,
because nothing reads it at runtime.
The second fault is the reason this was worth chasing. Three gate modules skip
themselves when the file they read is absent, which is correct in a published
checkout and silently wrong on GitHub, which runs against exactly that checkout.
The modules skipped, CI reported green, and an outside branch added nine
undocumented routes and passed. A gate that stops running is worse than no gate,
because nobody looks at it again.
scripts/check-gates-ran.sh asserts each gate ran, with two lists: the ones that
must run everywhere, and the ones needing files publication deliberately leaves
behind - required here, allowed to be absent there. A `gates` job runs it in CI.
Proven in both directions: green in each checkout, and red when the file a gate
reads is removed.
Also purge instance/plugins.json from the publication tree. It was never
published - it is untracked there - but an old copy had sat in that tree for
months, protected from --delete by its own exclude, and it made the published
repo fail its own suite locally: 13 plugins enabled where the code expects 16.
A publication tree that cannot pass its own tests cannot be verified before
shipping. That is the exact trap the comment at the top of the script warns
about, which had claimed a second victim.
Also: concurrency on the workflow, so a second push supersedes the first rather
than queueing behind it holding a self-hosted runner.
|
||
|
cproudlock
|
024751896e |
Keep the site-literal patterns out of the published copy
The comment explaining the concatenation fix spelled the example hostname out in full, which tripped the publication scrub - the same trap the surrounding comment warns about, since the scrub greps this file for the very strings the patterns are assembled from. Reworded with a neutral example. Re-verified after the reword by planting a violation and watching the gate fail. |
||
|
cproudlock
|
dffa2082be |
Backups: stop storing credentials, and fix a dedup chain that never matched
The NTLARS/DNC registry carries an FTP password in cleartext and it was stored verbatim. That put it in the backuprevisions table, on the revision panel which renders values, and in the download route which hands back a ready-to-import .reg - so anyone holding backups.view could read it off a web page. Redaction happens at PARSE, so the secret never reaches the database at all. Redacting on display would have left it at rest and one query away. What is deliberately NOT done is dropping the value silently. This codec already refuses to discard an unparseable value, precisely because a backup that looks complete and restores a machine with a setting missing is the worst outcome. A redacted value keeps its name, is listed in the projection, and renders as a COMMENT naming what must be set by hand. Emitting a placeholder as the password would produce a machine that looks restored and cannot connect, and the technician would go debugging the network. scripts/redact_backup_secrets.py scrubs revisions recorded before this, dry run by default. It leaves contenthash alone: the hash identifies the configuration a bay reported, so rewriting it would make every bay's next post look like a change and insert a duplicate revision across the fleet. Separately, sourcehostname is now normalized at ingest. samesource() maps '' to IS NULL for the query while the insert stored the raw '', so rows written by a client posting an empty hostname could never match their own next post: every post inserted, the table grew without bound, and the history filled with identical entries nobody could read. Whitespace did the same thing more quietly. |
||
|
cproudlock
|
3c68ac2850 |
ADR-015: the site-literal check was matching nothing, and had been all along
The patterns are assembled from fragments so the published script does not
contain the strings it greps for. They used the apostrophe-escape form
('a'\''b'), which EMBEDS a literal apostrophe, instead of the concatenation form
('a''b'), which joins them. Every pattern therefore carried a stray quote in the
middle and could never match a line of real code, so the check passed everything
from the day it was written.
Verified the only way that means anything: plant a violation, watch it fail,
revert. Fixed, it immediately found 25 real hits.
Among them, in seven places, the default ServiceNow URLs - one organisation's QA
tenant, shipped with the integration ENABLED. Any site that had not configured
ServiceNow still rendered ticket links, pointing its technicians at a tenant
they cannot reach; an adopting site would have been pointed into somebody else's
instance entirely. There is no sensible cross-site default for "which ServiceNow
is yours", so the URLs ship blank and the integration ships off, matching the
Zabbix block that got this right.
A named SaaS tenant is as site-specific as a hostname, so it joins the patterns.
Colocated .spec.js files are excluded the same way tests/ already is: sample data
naming a facility is not a shipped literal.
|
||
|
cproudlock
|
315d4f46ce |
Repair backup revisions that recorded a mapped drive
Snapshots written before the client was fixed stored "W:\backup\partmarker\..." as their location. That mapping exists only inside the SYSTEM logon session, only during an enforcement cycle, and only on the bay that wrote it, so the path recorded against a marker resolved to nothing for the person most likely to open it. The script rewrites backuprevisions.sharepath and nothing else - no revision is created, deleted, re-filed or re-hashed, and no file on the share moves, since the bytes are already where the UNC will point. Dry run by default. It matches on the drive letter alone and leaves everything else untouched, including paths that are already UNC and paths on another drive. This repairs a specific known mistake rather than normalising whatever it can reach, which is the version of this script that would eventually rewrite something it should not have. rewrite() is kept pure so the decision of WHICH paths to touch is testable without an app context; the tests cover the already-UNC, other-drive, empty, case-difference and trailing-separator cases. |
||
|
cproudlock
|
7d7caba56b |
Let a maintenance script find the database the site actually uses
Running `python scripts/export_catalog.py` on a deployed site died with 1045 "Access denied for user 'root'@'localhost'", naming a user that site does not have. Nothing loads .env for a plain script run. wsgi.py calls load_dotenv() for the served app and the flask CLI loads .env for itself, so both reach the real database; a bare `python scripts/...` gets neither. DATABASE_URL then stays unset, FLASK_ENV defaults to development, and config.py falls back to a hardcoded root@localhost URL. The read-only failure above is the harmless version. Six of these seven scripts WRITE - backfill, reclassify, retype, collapse, refile, import - and on any box where those fallback root credentials happen to resolve, they would have silently rewritten the wrong database with no error at all. That is also why this survived: the dev .env points at root, so the fallback matched what the developer expected and nothing ever looked wrong here. All seven now load .env from the repo root, resolved from __file__ rather than the working directory, since these are run by hand during a deployment and rarely from the repo root. tests/test_maintenance_scripts.py pins it for any future script: every script that calls create_app must call load_dotenv, before create_app rather than after it, and with an explicit path rather than a cwd-relative search. |
||
|
cproudlock
|
c6c806667e |
Run the database version the rest of the product already recommends
INSTALL-WINDOWS-IIS.md has said MySQL 8.4 LTS is standard for new installs since 8.0 reached end of life in April 2026, while both compose files and the offline bundler still pinned 8.0. A site reading the Windows runbook and a site reading the Docker one were being told to run different servers, and the migration page written this week sent people onto the dead one. Verified against a real server rather than by editing a tag: 8.4.11, core chain plus five plugin chains applied clean, 66 tables at a single utf8mb4_unicode_ci collation, six alembic version tables. The image's PyMySQL authenticates against 8.4's caching_sha2_password, which is what requirements.in already pins cryptography for. Existing servers need one thing done FIRST: 8.4 removes mysql_native_password, so an account created on 5.6 or 5.7 must be moved to caching_sha2_password before the upgrade or it cannot authenticate afterwards. In-place also has no downgrade path, and 5.7 cannot reach 8.4 in one hop. For databases this size a dump into a fresh 8.4 server is the better trade: same outage, and the old server stays as the rollback. Air-gapped sites need a fresh offline bundle, because the tarball carries the MySQL image alongside the app image. Also here, found by having it bite during that verification: the db healthcheck pinged over the unix socket, and the entrypoint's init pass answers on the socket while running the server with --skip-networking. The probe therefore reported healthy DURING init, which is what `depends_on: service_healthy` gates api and migrate on. A ping passed at 8 seconds and the next query failed because the server was mid-restart. Probing 127.0.0.1 keeps it red until the real server is listening. |
||
|
cproudlock
|
2d09fa3201 |
Collect what bays actually have, separately from what they are told to have
ShopDB knew what a bay SHOULD have and nothing about what it DOES. Adding the observed half makes a rollout a review instead of a typing exercise: the floor reports itself in, you look, and you adopt. The collection uses the mechanism that already exists rather than a new one. POST /api/collector/printers dispatches to the printers plugin's apply_collector_payload, the same ADR-006 hook the computers and backups plugins implement. New client script, new plugin-owned table, no new transport and no new credential. OBSERVED AND ASSIGNED STAY APART, and that is the point rather than a detail. A collector report can never write an assignment row: _reconcile_edges is the only function that writes usesprinter/defaultprinter, it has two call sites, and both are authenticated routes a human calls. If a drifted bay's own state were allowed to become what it is told to install, every configuration error would become permanent the next time that PC checked in. Seeding an assignment from observed state is explicit - POST /assignments/seed-from-observed - because a rollout adopts many machines at once. It routes through the same _reconcile_edges as the editor, so there is one write path with two doors, and a queue matching no known printer is REFUSED rather than guessed into an assignment. That last rule is the lesson from the measuring tools: adopting on a weak key produced 43 duplicate instruments. Two fixes on top of what the agents built. The replace deleted a host's previous rows by exact case-folded name while the read path treats a short name and its FQDN as one machine, so a PC that changed spelling appeared to hold every queue twice - which reads as drift that is not there. And the client sent 'reportedat' where the declared schema said 'observedat'. Also here: the legacy loader now imports machines.printerid, the classic system's record of each machine's default printer, which it silently dropped - the production import would have lost every one. And Set-ShopdbPrinters.ps1 finally registers the per-user logon task, staging Apply-ShopdbDefaultPrinter.ps1 to C:\ProgramData first because the share it lives on is mounted only during the enforcement cycle and the task runs at logon when it is gone. VALIDATED ON WINDOWS 11 (build 26200), not just on Linux pwsh, which parses these scripts happily and executes none of the spooler branches. The reporter: posts a correct payload with the X-API-Key header; resolves BaseUrl and CollectorKey from HKLM when given no arguments; suppresses the virtual queues by port; resolves port addresses; and reads the CONSOLE USER's default out of HKU rather than SYSTEM's own, which is a different and usually wrong answer. Two results matter more than the rest. With the spooler stopped, both the cmdlet and the CIM path fail and the script posts NOTHING - verified against a capture server that recorded zero requests, where an empty list would instead have erased that host's observed rows and read as a bay that lost its printers. A genuinely empty host still posts [], because that is a real and different fact. The logon task registers as the Users group at Limited, and falls back to the well-known SID S-1-5-32-545 when the group name will not resolve, as it will not on localised Windows. It was then run with the source directory RENAMED AWAY, to stand in for the share being unmounted, and it still moved the user's default - which is the whole reason the script is staged to C:\ProgramData rather than run from where it lives. The guarantees against damage were re-checked rather than assumed: an empty assignment changes nothing, an unreachable server changes nothing, -WhatIfOnly leaves no queue, no task, no staged file and no registry value behind, and a drifted queue is repointed IN PLACE with Set-Printer so whoever has it as their default keeps it. Not covered by any of this: the driver-staging path, which needs a real vendor package rather than the class drivers a VM ships with. |
||
|
cproudlock
|
f34b9ca710 |
Carry the level everywhere a position is drawn, and gate it per occurrence
The hover mini-map said "This asset has a position (2835, 1410) but no level" for every asset in the product. When 0.11.0 gave LocationMapTooltip a levelid prop, NONE of its seven call sites were taught to pass one - printer, machine and PC detail pages, the toner report, enforcement reports, the warranty chip and the dashboard cards - so the component correctly reported a missing level and the preview never drew. Two payloads behind those views also emitted mapx/mapy with no level: the toner report and the enforcement report. The map PDF export had the ORIGINAL bug still in it: it plotted every filtered asset onto the sheet, so exporting the ground floor printed second-floor markers on it. Worse than on screen, because nobody can correct a sheet once it has been printed and carried onto the floor. It now exports only the level being viewed. The legacy import loader sent mapleft/maptop with no level at three call sites. That loader is the one still to run against production, and every marker it created would have been undrawable. It now resolves the site's default level - the legacy schema predates levels and has one floor plan, so that is what its coordinates mean. THE GATE MISSED ALL OF THIS because it asked whether a FILE mentions 'levelid', not whether each position does: one module emitted 'mapx' six times and 'levelid' once and passed. It now checks per occurrence, covers scripts/ as well as shopdb/ and plugins/, and fails any Vue file that binds tooltip coordinates without :levelid. Both new rules were confirmed to fail the build against planted violations before being relied on. Printer QR labels: the asset number is no longer printed. A label now reads name (8201-HPLaserJetPro), QR, FQDN, then IP. The name falls back to the assetnumber because that is where sites actually keep it - every printer here has an empty name field, so preferring the Windows queue name alone would have printed a blank line on every label. |
||
|
cproudlock
|
dd503be4ba |
Release 0.11.1
A patch release. The pagination-cap fixes, and the documentation that missed the 0.11.0 installer by a few minutes. docs/UPGRADE.md is one of the documents the Windows installer puts on the server, and its buildings-and-levels section was committed after 0.11.0 was tagged. So 0.11.0 shipped the page an operator reads WHILE upgrading without the guidance about that upgrade - which matters most on an air-gapped box, where there is no other way to reach it. The installer now also ships FLOOR-MAP.md for the same reason the Windows runbooks ship. No application code changed on the server side and the plugin contract stays at 0.20.0. The version and the changelog are the release; the detail is in the entry. |
||
|
cproudlock
|
3324dbd91e |
Buildings and levels for the floor map, and make every identifier searchable
The map was one picture of one floor. A second floor was added, the blueprint changed size, and machines moved, so a position now records WHICH DRAWING its coordinates belong to. Buildings and levels (ADR-017). Each level owns its blueprint per theme and its own native pixel size; assets.mapx/mapy are pixels of assets.levelid, not of the site. A position whose level is unknown renders "level unknown" and is never drawn on the default level, because a marker on the wrong floor plan looks entirely correct while pointing at the wrong place. Repositioning in bulk: filter by unplaced, needs-review or level, search, place, confirm. Landmark recalibration solves the transform PER AXIS from landmark pairs and never from image dimensions - the canvas grew taller without rescaling, so a dimension-derived scale would stretch Y by 1.57 and be wrong everywhere. It defaults to a dry run, reports what would land off the drawing, snapshots before applying, and clears mapverifiedat because a transform is a guess awaiting review. Snapshots restore, including the level and the review state, and a restore snapshots first so an undo is undoable. Search: gaugelabreference was matched only for measuring tools and maintenancereference was matched nowhere at all, for any asset type, while Settings happily offers both identifiers on machines and PCs. A tag an operator is told to record has to be findable or it is a write-only field. USB devices and printed items were unreachable from search entirely - neither is an asset, so the generic asset search could not see them and no searcher existed; they now match on serial, asset tag, label, bin code and gage-lab tag, honouring isactive, with Settings toggles and result labels to match. The retired-application rule was half a rule: GET /api/knowledgebase hid articles whose topic application is retired while global search still returned them and printed the retired application as the subject. A filter is only real if every path that reaches the row applies it. Contract to 0.20.0 (additive): Asset gained levelid and mapverifiedat, Location gained levelid, and resolve_asset_position returns the levelid belonging to whichever source supplied the coordinates. The five plugins that write a map position are re-pinned. The install-list text format gained levelid as a NINTH field, appended, because the shipped Pascal installer reads fields 0-7 by index. That installer still compiles in one drawing's dimensions and bundles one blueprint, so its map is accurate for the default level only; /api/maplevels is deliberately unauthenticated so it can read both at runtime once rebuilt. Recorded in PRINTER-INSTALLER.md section 6 along with the other known gaps. Migration 7d33 converts an existing single-map site into one building and one default level carrying the old map_* settings, then assigns every placed asset and location to it. Nothing moves on screen. Old settings rows are kept so a rollback still finds them. Verified end to end on MySQL 5.6 from a production-shaped database. |
||
|
cproudlock
|
e7b8933588 |
Stop the publication scrub failing on the rules that enforce it
The export gate now carries the site patterns, and three kinds of file tripped it - two legitimately, one by construction. Two test files held real internal subnets as fixtures. They are documentation ranges now, which test the same logic and disclose nothing. The project-map generator and a changelog entry named a file that is excluded from publication, so a public reader was pointed at something they cannot see. Both now describe what happened without naming it. And the naming script has to CONTAIN the site patterns in order to grep for them, so written literally the rule's own definition fails the gate that enforces it. The patterns are assembled from fragments, the same trick the docs publishability test already uses for the same reason. Verified the hard way: a planted literal is still caught, so the fragmentation did not quietly turn the rule into one that matches nothing - which is the obvious way for this fix to have gone wrong. |
||
|
cproudlock
|
ad335cfc9e |
api: document the twenty-two routes that were serving traffic in silence
The inventory is hand-maintained, and should stay that way - its value is the prose, and what an endpoint is FOR cannot be derived from the code. An audit of all 372 documented operations found zero phantom routes and zero wrong parameter names, so the maintenance is in good order. What hand-maintenance cannot do is notice a route somebody added. Twenty-two were undocumented: the entire backups plugin surface, every one of the dashboard card endpoints added with contract 0.19.0, the GE-Enforce publish preflight, the employee SSO resolver, the protocol update verbs, and the four /api/docs routes - so the spec did not describe how to fetch the spec. Coverage is now a test. It walks the live url_map and fails when a served route has no entry, which means adding an endpoint includes describing it, in the same commit, while the author still knows what it is for. The reverse direction is checked too: a documented route that no longer exists sends a reader to a 404. Writing that test found one more thing. The inventory writes multi-verb routes as "PUT|PATCH", and neither the parity check nor the generator split on the pipe - so those operations were absent from the published spec entirely, with nothing reporting it. The spec now carries all 394 operations the code serves, which is the first time the two numbers have matched. The generator's own docstring claimed the inventory could be regenerated. It cannot; nothing generates it. That sentence is why nobody noticed it was falling behind. |
||
|
cproudlock
|
36b0265668 |
openapi: emit a spec a machine can actually use
The generated spec carried no `parameters` and no `requestBody` on any of its 372 operations. That is invalid OpenAPI 3.1, and the practical cost was worse than the formal one: the MCP server builds its tools from this file, so every tool had an empty input schema and silently dropped whatever the caller passed. A request for one asset returned the list, and nothing anywhere reported an error. All 118 templated paths now declare their path parameters, typed from the Flask converter that named them, and write verbs declare a JSON body. The body is an open object carrying the prose description rather than an invented schema. The inventory describes bodies in sentences, and a field list this generator guessed at would be worse than none - but "an object, described here" is the difference between a client that can send a body and one that cannot send anything. Security was wrong on 123 operations. `jwt-optional` means "works logged out, returns more logged in", which OpenAPI expresses as the empty requirement alongside the scheme; publishing them as bearer-required told every reader that a public endpoint needs a token. Responses were one hardcoded 200, so a generated client had no idea a call could fail. Every operation now documents the error envelope - and the envelope itself is a defined schema, because its error nests under `data.error` rather than at the top level, which is the single thing people get wrong when writing against this API. 95 summaries were cut at 120 characters mid-word, which is what a tool picker shows a user as the whole description of a call. They now end on a word. Tests pin the shape rather than the prose. One of them contradicted an older test that REQUIRED the contract version as a literal in PLUGIN-HOOKS.md - the same copying that left nine documents stale - so that test now asserts the doc points at the generated map instead. |
||
|
cproudlock
|
05be4c4489 |
docs: fix the examples that fail if you paste them
Six procedures that could not be followed as written.
Eighty-nine curl examples single-quoted `Authorization: Bearer $TOKEN`, so the
shell never expanded it and the server answered 422 "Not enough segments". Nine
more did the same with X-API-Key. The other 129 examples in the same file
already used double quotes, so this was drift rather than a convention, and the
spec regenerated from it carried the fault onward.
The GE-Enforce report example put a `//` comment inside a JSON body. The server
parses with silent=True, so it saw `{}` and answered "hostname is required"
about a body that plainly has one - the worst kind of error message, one that
sends the reader to the wrong field entirely.
The IIS install ran `flask db upgrade` and a per-plugin install loop but never
`flask plugin upgrade-all`, leaving every plugin's own chain unapplied. That is
precisely the 1054 "Unknown column" a deploy then hits somewhere else, days
later, on the page that uses the new column.
The pilot runbook looped `flask plugin enable` over plugins that were not yet
installed; enable refuses those, so on a fresh database it exited 1 on every
iteration and enabled nothing. ADR-013 had already recorded that defect.
`apply-profile` installs and enables in dependency order, which is what the step
was reaching for.
DEPLOY-WINDOWS-IIS named a profile file that does not exist; the shipped ones do.
And PLUGIN-EXTERNAL-REPO never mentioned PLUGIN_TABLE_OWNERS, while the
migration engine raises for any plugin missing from it - so the guide's own
step 5 fails for any external plugin that owns a table. That the registry lives
in the framework repo is deliberate, so the guide now says so, and says what it
costs: a table-owning external plugin is a two-repository change, and a plugin
that owns no tables avoids it entirely.
|
||
|
cproudlock
|
4995456136 |
docs: take one site's name, hosts and paths off the public wiki
The publishability gate caught internal tooling names and developer paths but nothing site-specific, so roughly sixty leaks reached the wiki: the site name in ten documents, real fleet hostnames in the collector and GE-Enforce examples, an internal database name through the whole import guide, imaging-share paths, and a maintainer's username as the Deciders line of every ADR and inside a generated curl example. None of it is a security matter on an air-gapped fleet. It matters because these pages are read by engineers at other plants, and a document that names one site throughout reads as that site's notes rather than a product's documentation - which is exactly what it then gets treated as. Examples now use neutral hostnames, the site is "the reference site" where the distinction carries meaning, and ADRs are decided by "ShopDB maintainers". The gate carries all of these patterns, so the next one fails a build. Two documents leave docs/ because they were never written for an outside reader. PROJECT-REVIEW.md is an internal health memo pinned to a commit from July, whose headline finding (an untracked playbook) has since been fixed - it is history, and git holds it. PILOT-DEPLOY.md is one site's own cutover runbook, complete with a "re-measure before publishing" placeholder; it moves next to the loader it belongs to, in scripts/site_imports/wjf/. ADR-015 is AMENDED rather than rewritten. Its enforcement section still said report-only and its backlog still listed hardcodes that are now cleared, which left the record contradicting itself. The amendment says what changed and why the report-only period ended; the original text stays, because what the decision looked like when it was taken is the part worth keeping. Also corrects llms.txt's response envelope, which had errors at the top level and pagination at meta.total. Both are nested one deeper, so anything written against that description read undefined on every error it tried to handle. |
||
|
cproudlock
|
c648bdf560 |
labels: one module knows how to draw a code, seven views stop guessing
Three core pages and four plugin pages each imported qrcode and jsbarcode directly, and each carried its own answer to the same questions: what margin, what width, which error correction, how big a module must be before a scanner can read it. The answers had already drifted - margin 0 in one place and 2 in another, width 150 against 160 - and on a label that is the difference between a sticker that scans and one that does not. frontend/src/utils/codes.js owns it now: the label-stock presets, the quiet-zone and margin defaults, CODE128 with no printed value, and the printer-resolution arithmetic that only the Tech Tools generator had. A view passes what is specific to its own label and nothing else - MachineBadge still asks for CODE39, because the badge readers predate the shop-floor scanners and decode nothing else, and that is exactly the kind of thing a call site should say out loud. views/print/qrLogo.js is folded in rather than left as a second half-shared helper that only some of the pages reached into. The check script now fails a build that imports either library outside that module. Without it this re-forks within a month: the next label page starts by copying the nearest existing one, which is how it happened the first time. Tests cover the part no amount of looking at a screen verifies - a QR that looks fine at 96 dpi on a monitor can be unreadable at 203 dpi on half-inch stock. |
||
|
cproudlock
|
8f9805d964 |
Generate the facts that go stale, and record the fortnight in the changelog
CLAUDE.md is read at the start of every session and claimed contract 0.16.0 and 1159 tests while the code was at 0.18.0 and 1567, along with a plugin count and a chain head that were both wrong. Hand-written facts about a moving codebase are wrong within a fortnight, and a reader who cannot tell which lines are stale has to re-derive all of them. So they are derived. scripts/gen_project_map.py reads the versions, the plugin inventory, every Alembic chain head, the ADR index with status and the size of the codebase out of the code, and --check fails a build where the committed map no longer matches. CLAUDE.md now points at it, leads with the multi-site rule, and lists the gates to run instead of restating the conventions they enforce. The changelog's Unreleased section covered 8 of the 46 commits since 0.9.0 and had no Changed, Fixed or Security sections at all. It now carries the whole fortnight, including both contract bumps - which had never been recorded even though ADR-002 makes contract versions their own series. |
||
|
cproudlock
|
035419fa51 |
ADR-015: stop shipping one site's values, and make the rule a gate
The scanner has been reporting the same count for weeks, which is what a rule that only prints becomes. It now FAILS the build, and it looks where the leaks actually were: PowerShell, the installer, the seeds, generated JSON, the frontend - case-insensitively, across plugins, shopdb, scripts, deploy, tools. A line that is deliberate declares itself with an ADR-015-OK marker and a reason, so the claim is visible in review instead of tolerated in silence. What it found, fixed here: - The shadow client wrote one site's ShopDB URL into HKLM whenever the registry disagreed. At the site it was written for that reads as healing drift; anywhere else it overwrites the site's own address on every enforce cycle, and the site cannot win because the cycle repeats. The bay's value now wins, an explicit -BaseUrl seeds it, and with neither there is nothing honest to write, so it says so and skips. - The kiosk dispatcher fell back to one plant's host when HKLM was unset, so a kiosk elsewhere quietly opened a server it has no business reaching. The fallback is now this site's site_base_url, baked in at seed time, and the dispatcher refuses rather than guessing when neither is set. Its legacy shortcut matcher derives the host from that URL instead of naming one. - The OpenAPI generator hardcoded a production hostname into every spec it generated, which then published to a public wiki. The relative mount is the only server it can honestly name; a site passes its own by environment. - Placeholders and examples in the UI and the client help offered real internal subnets and a real production URL. They now use documentation ranges. Both publication gates - the export scrub and the docs publishability test - carry the site patterns, which neither did. One plant's hostname, FQDN and internal networks are out of the documentation and the generated specs. Comments naming the reference site are reworded rather than deleted: the reasoning is worth keeping, the plant name is not what makes it true. |
||
|
cproudlock
|
13e6e039fe |
ADR-015: where a site's own data is allowed to live
ShopDB is a product several sites adopt, developed at one site against that site's live fleet, so every feature arrives carrying West Jefferson's hostnames, networks and share paths. A scan of plugins/ and shopdb/ returns 19 site literals, of which about 8 are a shipped default or a hardcode rather than a comment: a kiosk URL baked into a PowerShell payload a sister site cannot override, a WJ UNC path as the backup share default, and device.geaerospace.net shipped as a SETTING DEFAULT - inherited silently by anyone who adopts. The mechanisms already existed; the rule did not, so each value landed wherever was convenient. The rule: a setting with a NEUTRAL default first, a site-namespaced directory second, seed data third. Blank must be a working state. Fleet-wide vocabulary such as the gea-shopfloor-* imaging types is not site data and is out of scope - it already has a per-site override. Enforced by an addition to check-naming-and-style.sh, which CI already runs, in REPORT-ONLY mode. Making a 19-item backlog a hard gate the day it lands blocks every unrelated commit until someone clears it; SITE_LITERALS_ENFORCE=1 turns it into a gate once the listed hardcodes are done. The ADR carries that backlog, including that the display FQDN domain is defined three times in three files. |
||
|
cproudlock
|
300f9123d2 |
backups: re-file historical revisions onto the part markers they came from
Backups collected before markers became assets were filed against the operation number the PC reported. The collector now files a marker PC's backup against its marker, but only for backups collected since; everything already stored stayed where it was, so the new marker assets read as empty while the operation holds a mixed history. The attribution is exact rather than a guess. A revision records the PC it was read from, and the collector records which marker each PC drives, so the move is that mapping applied. Revisions with no source hostname stay put, since nothing says which marker they came from. Moving a revision can duplicate one the collector has already written against the marker, so the script collapses those as part of the move, keeping the earliest so the date a config was first seen survives. Dry run by default. |
||
|
cproudlock
|
d429c882b4 |
backups: a revision chain belongs to a PC, not just a machine
Dedup compared a posted config against the latest revision for the ASSET, which is only correct when a machine number means one PC. Several PCs share one here: the part markers on 0613, 0615 and WJPRT are separate devices, differing by COM port, filed under one machine number. Each marker's post therefore differed from whichever marker had posted last, nothing ever deduped, and the table grew by one row per PC per collection cycle. A chain is now (asset, kind, source hostname). An unchanged config is a no-op again, and each PC keeps its own history against the machine. NULL sources - rows written before the column was populated, and hand-loaded ones - form their own chain via IS NULL; `column == None` never matches in SQL, so without that those rows would have re-posted forever. Two consumers assumed the old key and are fixed with it. Retention pruned per asset, so a busy marker's revisions could evict a quiet marker's only backup; it now prunes each chain separately, protecting the newest and oldest of each. The revision diff compared against the previous revision on the machine, which across two markers reported one device's COM port as a change on the other; it now compares within the source's own chain. scripts/collapse_duplicate_backup_revisions.py cleans up what the old rule wrote. It removes only a revision whose hash repeats the one before it in the same chain - rows the fixed code would never have written - and keeps every genuine change, every chain's newest and oldest, and every source. Dry run by default. Its --report mode explains what grew each chain, which separates a legitimately shared machine number from two PCs wrongly carrying the same one, and from a value inside the config that changes on its own. |
||
|
cproudlock
|
bb4169cd8c |
backups: keep the seed script's docstring site-neutral
The import script is bundled in a multi-site product, so its usage examples should not name one site's share layout or a dev box path. |
||
|
cproudlock
|
fca775c737 |
backups plugin: per-asset config backups with revision history
Adds a kind-pluggable backups plugin. Configuration captured from a PC is filed against the MACHINE it controls, with a revision history and download back to the native format. NTLARS/DNC is the first kind. Settings live in the controlling PC's registry but describe the machine, so revisions attach to the machine's asset and carry no foreign key to the PC: history survives a PC being replaced or deleted, and sourcehostname records the handover. Storage splits by kind. Parseable kinds store a dialect-neutral JSON projection in ShopDB and re-render on download; opaque vendor formats (part marker and similar) keep their bytes on the SFLD share with ShopDB holding metadata and the UNC pointer. Two .reg dialects exist in the wild: NTLARS's own Save... export omits the WOW6432Node path segment, scripted exports include it. Parsing strips whichever root matched, so a stored revision commits to neither and download offers both (NTLARS Load... by default, WOW6432Node for direct reg import). Getting this backwards is silent, so the dedup hash deliberately excludes sourcedialect and both dialects of one config dedup to a single revision. Dedup is load-bearing: the collector runs every GE-Enforce cycle across the fleet, so a revision is inserted only when the content hash differs from that asset's latest for that kind. A freshly imaged PC opens NTLARS with a blank General tab. Recording that would make an empty config the newest revision exactly when someone needs the last good one, so a blank MachineNo is rejected rather than accepted as a change. Two of the 320 known-good backups on the share already have that shape. DNC Info card summarises the latest revision on the machine page: General (Cnc, NcIF, HostType), eFocas, Serial, NTSHR when populated (only 18 of 147 machines), and MARK when the machine is a marker. MARK is gated on Cnc=MARKER or the ShopDB machine type, not on the MARK key having content: MARK carries serial defaults on 145 of 147 machines and DncPatterns reads YES on 103 including ordinary lathes, so neither identifies a marker. The info card is owned by the kind (BackupKind.infopanel/buildinfo) and served by a generic endpoint, so the expected successor to DNC ships its own card by adding a class rather than changing the plugin or the panel wiring. Also: schedule and retention settings with a prune that never drops the newest or the oldest revision, and scripts/import_ntlars_backups.py to seed history from the existing per-machine .reg files (144 of 147 resolve to assets). Codec verified against all 320 real backups: round-trips clean through both dialects. Bay-side generation verified on Windows against reg.exe export. |
||
|
cproudlock
|
367bc56a6d |
Ship the equipment catalog so a new site does not start empty
`flask seed reference-data` wrote a dozen generic model types and no vendors or models at all, so adopting this platform began by retyping a catalog another site had already spent a year building. That is the largest single obstacle to standing a new facility up. scripts/export_catalog.py dumps the catalog from a live instance to shopdb/data/catalog.json, and `flask seed catalog` loads it. What travels: vendors 53, models 128, modelsupplies 146, modeltypes 35, machinetypes 21, computertypes 10, printertypes 9, networkdevicetypes 5, locationtypes 11, operatingsystems 14, measuringtooltypes 8, notificationtypes 3, accessprotocols 3 The 146 printer supplies are the most useful part after the models themselves: every toner, drum and maintenance kit with its part number, colour, capacity tier and page yield, already matched to the right model, instead of somebody reading them off spent cartridges. IDEMPOTENT and ADDITIVE. Records match on a natural key - a vendor by name, a model by vendor plus model number, a supply by model plus part number - so a second run adds nothing, and it never updates or deletes: a site that corrected a description or pointed a model at its own photo keeps its version. Catalog only. No assets, locations, employees, business units or anything with a serial number: nobody wants one plant's machines appearing at another. Vendor contact details are excluded too, since a rep's name and number belong to whoever holds that relationship. supportteams, printerdrivers and customfields are site-specific and deliberately absent. Models and supplies reference their vendor by NAME rather than id, because ids differ between databases and an id-keyed catalog would silently attach part numbers to the wrong printer. The installer offers it as a tick-box on a new "Starter data" page, defaulting to on, passing -SeedCatalog to stage 3. Offered rather than assumed: a site that machines nothing does not want 21 machine types cluttering its dropdowns. Verified by loading into an empty database and running twice: every group populated on the first pass, "Catalog already present, nothing to add" on the second. |
||
|
cproudlock
|
85ff25462e |
Reset to page one when a filter changes, and let the catalog carry a real type
Two unrelated things found while looking at blank printer types. Selecting a filter while past page one returned an empty list. The filter asked the server for page 5 of a result set that now had one page, and the screen said nothing matched. useListQuery already resets the page - setSearch and setExtra both do - but the filter dropdowns bypassed it and called the loader directly. Nine list pages now route through applyFilter, which calls setPage(1) when it needs to and loads directly when already on page one, so the composable's URL watcher does not also fire and fetch twice. scripts/retype_models.py addresses why printer types cannot be derived. The catalog types every printer model "Printer": true, and useless, since it does not say whether the product is a laser, a plotter or a label printer. That answer is a property of the model - every VersaLink C405 is a laser MFP - but nothing recorded it, so nothing could derive it. Recording it on the MODEL means the existing backfill fills every printer by exact name match, and a printer added later inherits the right type the moment its model is chosen. It exports the models needing a decision to CSV with a type suggested from the model number, a person corrects the column, and applying it is a dry run unless given --commit. A suggested type is refused unless it already exists in that asset class's own vocabulary, which is what keeps the later name match working. The suggestion order matters and got this wrong first time: a generic plotter pattern matched "Zebra ZT411" and filed a label printer as a plotter. Brands now come before generic patterns, and the review step exists precisely because a confident wrong guess would type every asset using that model. Verified on the development database: 24 printer models need a decision, 22 got a sensible suggestion, applying them let all 42 printers match a printertype by name, and the transaction rolled back cleanly. |
||
|
cproudlock
|
58b460fe3d |
Backfill an asset's type from its model, by exact name only
Correcting an earlier judgement. I said the model's type could not be used to fill an asset's type, because modeltypes is the catalog-wide list covering every kind of asset - it holds "Access Point", "Camera" and "Desktop PC" - and only about two thirds of its names exist as machine types. That is true across the whole catalog and misleading in practice. Restricted to the models an asset class actually uses, the picture is different: all 262 machines in the development database map exactly, because the non-machine entries are never used by machines. The blanks on the machines list are rows whose type the database could already have supplied. So the backfill now fills the type as well, under a rule that cannot mistype anything: exact name match or nothing. A model type with no identically named entry in the asset's own type table is reported with a count and left untouched, so somebody can decide rather than have a guess written into their data. The same shape covers computers, printers and network devices, each against its own type table. Verified against the development database by nulling one machine's type inside a transaction: it was detected as fillable, the proposal read "LocationOnly" -> "LocationOnly", the update restored exactly the original id, and the rollback left the row unchanged. Still a dry run unless given --commit, and a table missing the model column is skipped, so it runs against a server whose network migration is not yet applied. |
||
|
cproudlock
|
3f320fcc8b |
Derive an asset's vendor from its catalog model, and show the model's own type
An asset that carries a model but no vendor was showing a blank the database could already answer: the model records its vendor, and both sides reference the same vendors table. Machines, PCs, printers and network devices now fall back to it. The fallback is FLAGGED, not merged silently. to_dict sets vendorfrommodel and the detail pages render "(from model)" beside the value, because the record itself is still empty: the edit form shows an empty vendor box, and a page implying the vendor is stored would be lying about where it came from. The model's type is exposed under its own name, modeltypename, and shown as a separate "Model type" row. It is deliberately NOT used to fill in the asset's own type. modeltypes is the catalog-wide list covering every kind of asset - it holds "Access Point", "Camera" and "Desktop PC" alongside the machine entries - so it is a different taxonomy from machinetypes. Only about two thirds of the names overlap, and mapping one onto the other would mistype the remainder, with the failure mode being a machine labelled "Desktop PC". scripts/backfill_vendor_from_model.py writes the derived vendor down for real, since the display fallback leaves reports that read vendorid still seeing nothing. It is a dry run unless given --commit, fills only rows where the asset's vendor is NULL and the model names one, and never overwrites a vendor somebody chose. It skips a table lacking either column, so it runs against a server whose network migration has not been applied yet. Verified against the development database by nulling one machine's vendor inside a transaction: it was detected as fillable, restored to exactly its original value, and the rollback left the row untouched. FLASK_ENV is not forced by the script. The app already reads it from .env, and overriding it demanded a SECRET_KEY the environment had no reason to supply. |
||
|
cproudlock
|
89e880afc3 |
Release 0.8.0
The Windows installer has never shipped under a version: v0.7.0 was tagged before any of it existed, so every build handed out so far stamped a server with 0.7.0. Two servers running different builds were indistinguishable, and the installer logged each upgrade as "same version already installed" rather than recording what changed. This cuts the release that fixes that. 0.8.0 rather than a patch: the air-gapped installer is a new capability, and pre-1.0 semantic versioning puts that in the minor slot (ADR-007). CHANGELOG gains a 0.8.0 section covering the twelve defects a real Windows Server 2019 install surfaced, the move from inferring "is this a re-run of my install?" to recording it, and the operator documentation. deploy/site-profile-universal.json is now in the repository. Released builds were being produced from a profile in a temporary directory, so the next release could not have been reproduced once that file was cleaned up. docs/RELEASING-WINDOWS.md points at the committed profile and says why. scripts/gen_openapi.py reads __version__ out of shopdb/__init__.py instead of restating it. Its hardcoded copy had already drifted a release behind, which is the same mistake that once shipped an installer stamped with the wrong version. |
||
|
cproudlock
|
2c415a1712 |
fix(installer): correct a false security claim, and clear the should-fix list
CLIENT IP / SPOOFABILITY. docs/geenforce-api-cutover.md claimed that removing the IIS rewrite rule made the allowlist fail closed and that it does NOT become spoofable. The opposite is true. IIS never sets X-Forwarded-For on its own; the rule is the only thing that does. Remove it and IIS still forwards whatever X-Forwarded-For the CALLER sent, waitress trusts it because it arrives from 127.0.0.1, and remote_addr becomes attacker-controlled - so a token-less caller can fetch manifests from anywhere on the network. The document and the _trusted_client_ip docstring now say so, waitress runs with --trusted-proxy-count=1, and stage 5 checks the rule is actually live rather than assuming it. The wizard question is rephrased to something an operator can verify with their network team instead of guessing at. NON-ASCII. The style gate only ever checked .py/.vue/.js/.ts, so documentation accumulated em-dashes, arrows and box-drawing characters against this repo's own convention - including in files added this week. Cleaned, and the gate now uses INCLUDES_ALL so Markdown, JSON and YAML are covered. PLUGIN DEFAULTS. The wizard pre-ticked measuringtools and printedparts, both of which ship default_enabled=false, so every site taking the defaults installed and enabled them against their manifests. Inno has no JSON parser so the list must be hardcoded, but tests/test_installer_defaults.py now fails when it drifts. UPGRADES. The payload copy merges, so a plugin dropped from a site's profile kept its code forever - which defeats a lean build and leaves core's optional-import guards succeeding for a plugin the site no longer has. Stale plugin directories are now deregistered and removed before the copy. add-plugin used 'plugin install', which for the five default_enabled=false plugins left them installed but DISABLED - and printed a green success line anyway. It now goes through apply-profile, and the success line is gated on the exit code. Invoke-Flask records its own exit status, because $LASTEXITCODE keeps a stale value when flask.exe is missing and no native command runs. CHARSET. The utf8mb4 compiler hook lived inline in migrations/env.py, so it covered the CORE chain only: plugin baselines inherited the server default, which on a latin1 server means two charsets in one database. It is now shopdb/utils/mysql_charset.py, imported by both, and preflight reports the database's default charset. BACKUP HONESTY. The dump was described as 'all of your asset data'. Uploaded branding and floor-map images live in instance\ on disk, not in the database, so a restore from the .sql alone comes back with no map. backup now archives instance\ alongside it and says both are needed. VERSIONING. AppVersion was hardcoded at 0.9.0 while the product, the frontend and the newest tag said 0.7.0 - and 0.9.0 collides with a retired contract version. Both builders now generate version.iss from shopdb/__init__.py. Smaller: rollback overwrites .env before deleting it, as uninstall already did; appcmd unlocks are scoped to this site's location rather than server-wide, with the wide unlock as a fallback; DEVELOPMENT-SETUP says Python 3.14; the README plugin list gains printedparts; prune-schema --force is documented as first-provisioning-only; HTTPS is documented as not-the-default with the steps to add it; the DBA SQL is on the wizard's database page; the features page says unticking does not remove an installed feature; and the installer README states that bundle-lock cannot vouch for the exe itself - that needs signing or an out-of-band hash, neither of which is wired up. |