dev
223 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
a46058a4dd |
Close the last three restore-gate gaps, and make the gate test the real gate
Three defects that all shared a shape: the restore path looked checked and was not. An unreadable data revision was treated as a legacy set. backup.sh discards the mysql error when reading alembic_version, so a set could be written with an empty datarev; the compatibility check then fell through to the code-to-code head comparison - the very check datarev was introduced to replace - and badged the set restorable on no evidence at all. A manifest that CARRIES the key and left it empty is now refused, while one with no key at all is still the genuine legacy case and keeps the fallback. backup.sh records the failure in run history rather than passing it over in a log line, since the set it just wrote cannot be restored. Dumps were bound to the schema they came from. mysqldump --databases bakes CREATE DATABASE and USE <name> into the file, so restoring onto a site whose MYSQL_DATABASE differs loaded every row into the dump's schema, left the live database untouched, and reported success. New sets are dumped without it and load into the schema named at restore time. Existing sets are still bound, so the gate reads which schema they name and refuses a mismatch instead of restoring beside the live data. Also dropped --no-tablespaces into the main dump: it needs the PROCESS privilege the app user does not have on MySQL 8. The gate's own test carried a copy of the gate. test-restore-datarev.sh re-implemented the predicate in its own shell functions, so it would have stayed green through any drift in restore.sh - including the plugin-chain hole fixed in the previous commit, which it could never have caught. restore.sh now takes RESTORE_CHECK_ONLY, which stops it immediately after the gates having touched nothing, plus overridable paths for the migration and plugin trees. The test drives the shipped script and asserts on its exit code, so the verdict under test IS the verdict that ships. Confirmed non-vacuous by planting a violation: disabling the plugin revision check turns the suite red. Nine cases now, up from five, covering both plugin-chain directions, a plugin the build does not have, and the schema binding. Moving the schema check earlier was prompted by that test: it had been placed after the safety copy, so it needed a live database to reach a verdict it can reach for free. It sits with the other gates now and stops before anything is written. Round-tripped against a real stack after the dump format change: backup, mutate, restore, data back to backup-time state, and the new dumps confirmed to carry no schema name. Suite 2607 passed / 9 skipped. Scope: product |
||
|
cproudlock
|
e2705d732a |
Gate a restore on every schema chain, not just the core one
The compatibility gate read alembic_version and nothing else, but plugin schemas move independently of core. Roll the code back to an older plugin revision and the core head can be identical, so a dump carrying plugin tables the code has never heard of passed the gate, loaded - destroying what was there - and only then failed in 'flask plugin upgrade-all' with "Can't locate revision". A failed restore that has already replaced what it was restoring over is the worst outcome this feature can produce, and it was reachable from an ordinary rollback. Backups now stamp the applied revision of every alembic_version_<plugin> table alongside the core one, read from the dumped database the same way. restore.sh checks each against that plugin's own versions directory and refuses before touching anything, naming the plugin and the revision to deploy past. A plugin in the dump that this build does not have at all is refused for the same reason. The list endpoint's compatibility badge asks the same question, so the UI cannot offer a set the restore will reject. Sets taken before the stamp existed carry no dataplugins and keep their old core-only verdict. Their plugin revisions are unknowable, and making every existing set unrestorable to close this would be a worse trade than the gap. Proven against a real stack, reproducing the exact case: core head identical, endpoints data stamped at a revision the build does not have. The restore refused, the live row was still there afterwards, and it stopped early enough that the pre-restore safety copy was never needed. The same set with its true revision restored normally. Suite 2605 passed / 9 skipped, with the gate covered both ways plus the legacy fallback. Scope: product |
||
|
cproudlock
|
fa6b99bf91 |
Finish a restore without a browser, from the set that was asked for
Closes the three defects in the imported restore path, at Ben's request. Each was proven against a real stack before and after: two backup sets with different data and different instance files, then a restore of the OLDER one, which is the case that was broken. A restore no longer depends on a tab staying open. The backup sidecar has no Flask, so it loads the dump and leaves a sentinel for the api container to run the migrations. The only thing that ever acted on that sentinel was the poll loop of the modal that queued the restore, so closing the tab - or letting a token expire - during the minutes a load takes left the data at the old revision under new code, every query on a newer column failing, and nothing retrying. A watcher in the api container now claims and runs it. The endpoint no longer runs the upgrade at all; it only makes sure something is pending, which is also how a failed upgrade is retried. That takes Alembic out of the request path, where it could outrun the gunicorn worker timeout and be killed mid-chain, leaving a half-applied revision the next attempt trips over. The claim is a rename, because rename(2) is atomic: several workers and a browser poll can race the same sentinel and exactly one wins. A failed upgrade parks itself as .upgrade-failed rather than re-arming, since putting it back would rerun the same broken chain on every poll forever. Instance files come from the set being restored. The restic call asked for 'latest' regardless of which set was named, so restoring anything but the newest paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json - a mismatch nothing downstream could detect. Now scoped by the set's own tag, which backup.sh has always written. The tree is replaced rather than restored over, so files created after the snapshot are gone afterwards: a restore has to put the site back to the snapshot, not merge into it. Done as a staged swap rather than 'restic restore --delete --target /', because that aims a delete pass at the container root and the cost of getting its filter wrong is the whole filesystem. Nothing destructive happens before a copy is taken. The dump carries DROP TABLE per table with no transaction around it, and the instance tree is replaced wholesale, so a wrong set or a failure part-way through left nothing to go back to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball and a README saying what it is and how to load it - and refuses to touch anything if that copy cannot be written. Two more found while proving the above, both of which stopped the overlay working at all: * Every script under scripts/backup/ was committed non-executable, while every other .sh in the repo is 755. The compose entrypoints are /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup and watcher containers could not start from a clean checkout on any machine, and nothing could fix it at runtime. * Dockerfile.backup could not build without the corporate CA staged. A plain COPY of the gitignored cert fails outright; the trailing-glob form does too, because BuildKit fails a COPY whose pattern matches nothing - which is exactly the case being handled. Copying the directory and testing for the file inside it is the form that works both ways. Verified by building the image with no cert present. Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic claim, the parked failure and the endpoint no longer running the upgrade. Scope: product |
||
|
cproudlock
|
83190422c9 |
Act on the review of the wilm import
Six findings from a four-agent review of the imported work, plus the contract bump the import owed. Nothing here changes what the imported features do. The contract version was not bumped for a change that widens what a plugin may declare. The asset-panel vocabulary gained two tab groups (serviceprocess, reports), a datetime column format, a map.body key and a tabs section that renders as a table. 0.29.0 bumped for exactly this class of change when the optional group was added, so this is 0.31.0 with a ledger row. All four are optional and resolve server-side when absent, so a plugin declaring none of them is unaffected. Site Backups now answers honestly where it has nothing behind it. The overlay is a compose profile: a separate container writes sets onto a shared volume and the api container only reads them. A Windows/IIS install runs no such container and the default root resolves to a drive-letter path nothing creates, so Back Up Now either failed on PermissionError or reported a run queued that would never happen, under a banner reading "No verified backup yet" - which describes a backup that failed, not a feature that was never installed. The three write routes refuse with the same 404 shape the configuration-policy routes use for a disabled feature, the list and status routes report whether a store exists, and the page says so instead of offering buttons. Two listAll callers still read a response that is not there. listAll resolves to a plain array through fetchAllPages, so response.data.data throws: PC Types settings rendered empty, and in Configuration Policies the throw happened inside the shared try, so PC types, applications, the action catalog and monitors were all left unassigned by one bad read. The agent package upload cannot succeed on IIS as shipped. MAX_CONTENT_LENGTH went to 256 MB to admit the ~69 MB tray binary, but IIS request filtering caps a body at 30,000,000 bytes and answers 413 before waitress sees the request, so web.config needs the matching requestLimits. The backup image could not be built by anyone outside a TLS-intercepting network. Dockerfile.backup did a plain COPY of a gitignored corporate root CA, and a COPY of a missing file fails the build. The trailing glob makes it a no-op when the cert is absent. endpointstates.domain was the one new string column written without a clamp. Under STRICT_TRANS_TABLES an over-long value raises Data too long and aborts the whole check-in, so a single malformed report would stop a machine reporting telemetry and events entirely. Not fixed here, and left with upstream: the restore path finalizes its schema upgrade only from a browser poll loop, so closing the tab mid-restore leaves the data behind the code; it restores instance files from the newest snapshot rather than the set that was asked for; and it takes no safety dump before the destructive load. All three are in the Docker-only overlay, which this site does not run, and none can be validated here. Scope: product |
||
|
cproudlock
|
6e36df841e |
Import the wilm endpoint telemetry, PC Detail rework and backup overlay
Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at
|
||
|
cproudlock
|
a6775c1da1 |
docs: ADR-018 supersedes ADR-005, metrology instruments are machines
ADR-005 split machines from measuring tools on the grounds that they are two asset classes. It named the metrology population as CMMs, vision systems, wax trace and Genspect; the plugin was then built against a docstring naming calipers, micrometers and thread gages. Those are not the same population, and the difference is the whole decision: bay-mounted capital equipment with a controlling PC on one side, hand tools in a drawer on the other. Sixteen months on, measuringtools holds zero rows against 204 machines. The metrology inventory waiting to be entered is 134 instruments - 35 CMM, 32 form tracer, 23 vision, 18 wax, 14 measuring, 8 Genspect, 2 eddy, 1 spline - every one bay-mounted with a controlling PC, and not one hand gage among them. CMM is already a machinetype, and Part Marker is already a machine while Form Tracer is a measuring tool, so the line is already drawn somewhere no operator can predict. Retires the plugin and the measuring_tool asset type. Calibration is not lost: it moves onto machines as four nullable columns with status still derived at read, which is a gain rather than a port because a machine could not record calibration at all before. ADR-005's one concrete objection - that "show me all measuring tools" would need an enumeration of type names - is answered with machinetypes.ismetrology, so it stays a single-column filter. Taken now because measuringtools is empty. After the 134 rows are entered they carry relationships, audit history, map positions and collector links, and the migration stops being a drop. Scope: product |
||
|
cproudlock
|
349e8c89bd |
docs: EventSaver has no compiled-in folder fallback any more
The adoption guide told an adopting site that a missing EventSaver.ini makes the binary fall back to "the reference site's file server". That was true of EventSaver.cs until the ADR-015 fix set Folder to empty, and EVENTSAVER.md was updated then while this page was not, so the two documents disagreed about what a missing ini does. Current behaviour is that it shows nothing, which is the point: failing visibly beats displaying another site's slides. Says that, and adds the case the old text was accidentally describing - a .scr compiled before the fix still carries the path, so a binary somebody was handed is not the same as one they built. Scope: product |
||
|
cproudlock
|
cbdfcbabb7 |
One badge vocabulary, and images a browser can actually load
Two operator-visible defects and three doc corrections, closing the tail of the 2026-09-03 run. THE BADGE. The two kiosks read the same physical badges and disagreed. The parts kiosk understood both shapes a scanner emits - the bare SSO and the PayNo wrapper 0<digits>BZ - while the USB kiosk compared the raw string to an SSO, so the wrapper resolved to nobody. An operator whose keyboard-wedge scanner emits it could take a 3D-printed part and could not check out a drive, with the same badge, at two cabinets a few metres apart. The shape now lives in shopdb/utils/badges.py on the contract surface, because a plugin may not import another plugin and because a badge is a property of the site rather than of a cabinet. Both kiosks use it, and both store the DIGITS rather than the string that arrived: one row per spelling makes a custody history that cannot be joined to itself. That closed a subtler hole in the same place. str.isdigit() is true for Arabic-Indic and other decimal forms, and int() parses them, so a badge written in those characters was accepted, matched a REAL employee, and was then stored as an SSO string nothing else matches. ASCII digits only. An empty scan and an unreadable one now get different answers, because they are different problems for the person at the cabinet. THE IMAGES. /api/applications/image/<filename> and /api/models/image/<filename> required a permission, and a browser fetching <img src="..."> sends no Authorization header - so the gate did not protect the file, it guaranteed the picture never appeared. Both are ungated now, like the four sibling image routes (branding logo, map blueprint, employee photo, part picture), and registered in PUBLIC_ENDPOINTS with that reasoning. /api/printers/lookup?fqdn= never matched anything: both arguments were compared against Communication.ipaddress, so a hostname lookup 404d on printers whose hostname was on the record. An FQDN now matches the hostname, with a fallback to the short name. The slide manager gained the per-slide hold time. The PATCH route and the feed have always carried `seconds`; nothing called it, so the one documented per-slide control was curl-only. Docs: API-REFERENCE listed the employee lookups and the whole notifications surface as unauthenticated when they answer 401, and called both kiosks "deliberately open" when they are off by default and refuse anything without the site's device token. The installer download says applications.download, which is what the code requires. api-inventory.json said printers/lookup was jwt-optional. Fourteen tests. The badge rule verified by planting the old per-kiosk behaviour: three fail, one in each kiosk. Scope: product |
||
|
cproudlock
|
1d2bbb3630 |
One key for audit retention, and stop the SAML page claiming a feature exists
Wave 6: two settings pages that told an administrator the site was configured when it was not. That is worse than a missing feature, because nobody goes looking for it. Audit retention had TWO keys. shopdb/core/services/auditretention.py reads `audit_retentiondays` (category site, default 0). A second `audit_retention_days` was seeded under category audit with a default of 90, and the Audit & Logging page wrote to that one - so an administrator set a period, reloaded, saw it persisted, and nothing ever consulted it. Retention stayed off. The page now writes the key that is read, the duplicate is out of the seed list, and migration 7d39 carries a value an operator actually set on the dead key over to the live one before deleting it: their intent was "keep 30 days", and the fix should not silently discard it. The page's copy also claimed logs "will be automatically purged", which nothing does - purging is manual, from the site settings page or `flask audit purge --apply`. The Authentication page offered a full SAML configuration - metadata URL, entity id, ACS URL, auto-create, admin group - and reported "Configured - SAML login available" once two fields held text. There is no SAML anywhere in the codebase: no login route, no ACS, no assertion handling. SAML IS planned, so the settings stay and the page keeps them; what changes is that it now says plainly that nothing consumes them yet, and the status line never claims availability. The dangerous control there was "Allow Local Login". Turning it off did nothing at all, so an administrator could believe they had closed local password login while it stayed wide open. It is locked on until there is another way in, and says why. CONFIG.md marks the whole block reserved rather than describing it as working configuration. Scope: product |
||
|
cproudlock
|
13be0ef90f |
Import the wilm agent store and WINC adoption work, with four fixes
Eleven commits from the sister site: a reversible Maximo/WINC machine-adoption import, a versioned agent binary store with upload and rollback, remote-BuildKit CI, tray version reporting, and UI fixes. Applied as a patch again - the published mirror rewrites hashes, so the branches have unrelated histories and git refuses to merge them. Reviewed by three agents before merging. Four things fixed here. THE PUBLISH WORKFLOW WAS DEAD. The new remote-BuildKit step in publish-images.yml sits at column 0 instead of indented under steps:, so the whole file is invalid YAML and GitHub would refuse it. Their entire GHCR distribution channel - which Start-ShopDBDev.ps1 and the compose files consume - was broken at the source and clearly never run. Indented; all five workflows now parse. THE IMPORT'S APPROVAL CHECK COULD NOT FAIL. commit() built the plan and the approval packet from the same upload and then asked the writer whether they matched, so packet_matches_plan compared a hash against itself. The documented contract - "a human approves a specific, frozen plan, and the writer can prove nothing drifted" - did not exist in the web path: an admin could preview file A and commit file B, gated only by the needs-create group NAMES they ticked. The preview's planhash is now required at commit and checked against a freshly rebuilt plan. Their tests committed without previewing, which is what let this through; they go through the real flow now, and three new tests pin it - all confirmed to fail with the check stubbed out. endpoints.agentpackage QUIETLY GREW. It used to mean "wrap the staged exe in a ZIP"; with the upload routes it also means "choose the executable every future package installs and runs as SYSTEM". Before this, staging needed filesystem access to the server. The permission description still said only "generate and download a package", so an admin granting it could not see what they were granting. Reworded. NOTHING RECORDED WHAT WAS STAGED, OR BY WHOM. The manifest held version, size and timestamp; re-uploading a version replaces the bytes in place under the same label. A sha256 of the stored file is now recorded, and the upload is audit-logged with it - after an incident there was previously no answer to who put that binary on the fleet. Also documented the nine new routes in docs/api-inventory.json (that gate cannot run on the mirror, so it falls to us at promotion time again), regenerated the spec and the project map, and kept our own docs where they conflicted: docs/ is scrubbed from the mirror, so their DEV-BUILD-TEST.md still had the dev-be channel mapping we corrected, and their PLUGIN-HOOKS.md was a new 45-line file rather than our 737-line canonical one. Their journal section is appended to ours. Contract version comes across as 0.30.0 for a genuinely additive surface. Suite 2369, frontend 324, gate green. Scope: product |
||
|
cproudlock
|
e98e15259e |
Unpick one site's conventions from the shipped bundle, and bind the launch-path
check to the file it actually runs Two clusters the five-agent review found. Both are about a product that another plant has to be able to install. ADR-015: the shipped monitor bundle was the originating plant's fleet convention frozen into product seed. The business policy was seeded as "RPCA Shopfloor App Monitors". RPCA is that plant's facility designator, the row is source='shipped', and the API refused to rename it - so every adopting site would carry another plant's label at the top of its policy list forever. 0007 now seeds the neutral name, and 0010 renames it on installs that already stamped 0007, guarded so it cannot collide with a policy a site made itself. Downgrade is one-way on purpose: putting another site's designator back is not a state worth reaching. Which PC type the bundle attaches to was the literal 'Shopfloor PC'. A plant whose type is "Shop Floor Workstation" silently got no shipped monitors at all, and one using that name for something else got all of them. It is a setting now, defaulting to the seeded name so nothing changes here, and a blank value attaches the bundle to nothing - the right neutral answer for a site that does not want it. Shipped rows refused every edit, including isactive and launchpath. A plant whose eDNC is on D:, or that runs no UDC, could not correct the path or switch the monitor off, and then carried permanent false drift for it. ADR-015 allows seed data BECAUSE a site can edit it afterwards. Paths and the active flag are now editable; kind and displayname stay fixed, because that pair is the identity 0007's bundle matches on. Also corrected two false claims in 0007's docstring: the policy tables belong to the computers plugin, not the core baseline, and a guarded no-op does NOT reconcile later - Alembic stamps the revision either way, so the bundle would simply never seed. LaunchPathPolicy was purely lexical. GetFullPath is string normalization that never touches the disk, so the root check constrained the NAME while File.Exists and CreateProcess followed reparse points to somewhere else. A junction under a writable directory in an approved root - creating one needs no privilege, only write access to the parent, and vendor trees under Program Files are not reliably locked down - turned WebUI monitor-edit rights into code execution as the console operator. It now resolves the real target, leaf and directory chain both, and re-checks THAT against the roots, rejecting if resolution fails. Alternate data streams passed the whole policy on net8.0: an interior colon no longer throws in GetFullPath, so "readme.txt:payload.exe" reported an .exe extension, passed the root check, and existed. Rejected explicitly now. Six tests added for the shipped-row and pctype behaviour. The Core project compiles clean. The launch-path change has NOT been exercised against real junctions, ADS or 8.3 names - that needs the win11 VM, and per the reviewer it wants a planted-violation suite there rather than my say-so. Still open from the review: monitor revocation does not take effect until agent restart, ActionPicker is keyboard-inoperable, the lightbox has no focus trap, one attached monitor grants generic any-target process control, and screenshots are retained unencrypted in the user profile. Scope: product |
||
|
cproudlock
|
88a683dad3 |
Act on the five-agent review of the wilm import
Findings the reviewers verified, fixed here. The larger cluster they found is NOT fixed and is listed at the end. Frontend. reportImageUrl is called from the template, so it ran on every render with no in-flight guard: a report whose fetch was pending was re-requested, and because each resolution reassigns the map the whole grid re-rendered and re-fired the rest, costing up to N-squared requests for N thumbnails. Two blobs resolving for one key orphaned the first object URL where revoke could not reach it, and a failure was never recorded so a broken image retried forever. Guarded, deduplicated, failures remembered. Three CSS variables did not exist, so their hardcoded fallbacks always won: --bg-alt (the "No screenshot" tile, near-black in light mode), and --warning-text/-bg/-border (dark khaki on a dark modal). Replaced with real tokens. The DEV badge's literal #c62828 is var(--danger). Installer and launcher. The generated site installer ran docker compose with no -p, so the project name defaulted to the package folder - which carries the version. Installing the next version created a new project with new empty volumes and the site came up looking wiped, old data stranded under the old project. Pinned to shopdb-<channel>. Start-ShopDBDev deleted any existing -TargetDir that lacked a .git, recursively and without asking. For a script handed round as "save it anywhere and run it", a mistyped -TargetDir pointed at Documents would have destroyed it. It now only reclaims a directory that is empty or carries this launcher's own marker, and refuses otherwise. Dev secrets came from Get-Random, which is System.Random: a 32-bit seed, and all four values off one stream, so recovering one recovers the rest. They sign JWTs for a stack a pilot may point at real imported data. CSPRNG now, and the API port binds to loopback like the DB rather than every interface. Two of my own from earlier in this session. The operator-report size check sat below request.form, and the first access to that parses and spools the whole multipart - so it ran after the thing it prevents. Moved above. And the channel remap missed build-channel-installer.ps1 entirely, where it set BOTH channels to main: a "stable" package built from the trunk, the exact mislabel the remap was for. stable now resolves the newest v* tag. Also removed a zero-byte docs/ShopDB-Planner-Tasks-v2.csv that rode in. NOT FIXED - needs a decision, and it is the biggest thing the review found. Migrations 0006/0007 freeze the originating site's fleet convention into product seed: a shipped policy literally named "RPCA Shopfloor App Monitors", a hardcoded 'Shopfloor PC' type deciding who gets the shipped watch-list, and shipped rows the API refuses to edit or deactivate - including their launch paths. A site whose eDNC is on another drive cannot correct it and gets permanent false drift. Those are stamped migrations, so this gets harder to undo at every install. See the review notes before anyone upgrades a second site through 0006-0007. Suite 2314 passing, frontend 316, all three PowerShell files parse. Scope: product |
||
|
cproudlock
|
b5540e631c |
Merge the wilm endpoint-monitor and channel-distribution work into main
Six commits from wilm-import: the squashed import, route documentation, the channel remap onto our tag-and-trunk model, the project map, parallel test support, and the four smaller review findings. Merged rather than cherry-picked: the published mirror rewrites hashes, so the commits do not apply one at a time against our history even though the squashed diff applies clean. Scope: product # Conflicts: # docs/PROJECT-MAP.md |
||
|
cproudlock
|
28bf7ea1bf |
Regenerate the project map for the imported plugins and tests
Scope: product |
||
|
cproudlock
|
2e42972f44 |
Map the image channels to how this repo ships, and disarm auto-publish
The channel work assumed the branch model it was written against: upstream, main is a release branch and dev-be is the trunk. Neither holds here. main IS our trunk - 212 commits past v0.11.3 - and a release is a tag (ADR-007). dev-be does not exist at all, so the edge half would never have fired and the launchers would have cloned a missing branch. Taken literally it would have published our unreleased trunk to every site as :stable, which is the opposite of what the word promises to someone who has no way to check. Remapped throughout: tag v* -> :stable the thing that was actually released main -> :edge the trunk, DEV badge lit publish-images.yml keys the channel on github.ref_type = tag rather than on a branch name, so the mapping cannot silently drift back. Start-ShopDBDev.ps1, dev-bootstrap.ps1 and build-channel-installer.ps1 resolve stable to the newest v* tag via git ls-remote --sort=-v:refname, warning and falling back to main if a repo has no tags yet. DEV-BUILD-TEST.md follows. The push: trigger is commented out. Automatic publishing waits on someone confirming the GHCR package exists under this owner and deciding who may pull it; workflow_dispatch works meanwhile. Turning it on is one uncomment. All three scripts parse clean under pwsh, and the tag-selection expression was run against this repo rather than eyeballed. Per the standing rule, the runtime paths still want a pass in the win11 VM before anyone leans on them. Scope: product |
||
|
cproudlock
|
56ed89bb7b |
Document the ten new endpoint routes and undo the stray re-indent
Two of the three things holding the wilm import out of main. The monitor CRUD, the two resolver reads and the three operator-report routes had no entry in docs/api-inventory.json, so test_api_inventory_parity failed. The gate cannot run on the published mirror - the test ships but its data file is scrubbed, and it skips on the file's absence - which is why the branch that added the routes never saw it. Cost lands here at promotion time instead. /api/endpoints is registered both with and without a trailing slash and both forms are served, so both are described rather than guessing which one counts. Spec regenerated: 353 paths, 504 operations. plugins/endpoints/plugin.py had a contiguous run of continuation lines shifted by one space. Python accepted it and git diff -w showed nothing, but it put deletions in a diff that is otherwise purely additive. Restored; the file now differs from the published main by additions only. Scope: product |
||
|
cproudlock
|
3587a35ad6 |
Regenerate the project map
Test count had drifted from the tree. The map is generated; a stale one fails scripts/gen_project_map.py --check in CI. Scope: product |
||
|
cproudlock
|
27b2817fd7 |
Import the wilm endpoint-monitor and channel-distribution work
Everything on origin/wilm that is not on the published main, applied as one
change because the published history has rewritten hashes: per-commit
cherry-pick conflicts against our main even though the squashed diff applies
clean.
Contents: endpoint monitor catalog + policy attachment + warn-only drift
reconciler, operator issue reports from the tray, agent launch-path allowlist,
and the GHCR channel-image distribution path for dev workstations.
NOT READY FOR main. Open items:
- 10 new routes are absent from docs/api-inventory.json, so
test_api_inventory_parity fails. That gate SKIPS on the published mirror
(the test ships, its data file is scrubbed), which is why wilm never saw it.
- dev-bootstrap.ps1 and Start-ShopDBDev.ps1 hardcode the ge-aero GitHub org.
- plugins/endpoints/plugin.py carries a whitespace-only re-indent.
Scope: product
|
||
|
cproudlock
|
2feb569686 |
Rename Metrics to Monitors and move it second
"Metrics" promised numbers. The tab holds tracked services, tracked processes and network interfaces - things being WATCHED and their current state, only one of which is a measurement. Monitors is what the panels actually are, and it matches the word the agent's own config already uses for them. Moved to second, ahead of Software. What is being watched on this box is the question asked right after "what is it", and well before its application inventory - which is reference data somebody goes looking for deliberately. Tab order is now Details, Monitors, Software, Activity. Contract 0.29.0 keeps its number. It is unreleased, so the group vocabulary it introduces is still being settled and nothing can yet depend on the old name. Both the server and the client fall back to 'details' for a group they do not recognise, so a site that deploys these halves out of step gets a misplaced card rather than a missing one. Scope: product |
||
|
cproudlock
|
2122c3b45c |
Tab the PC detail page, and let a panel say which tab it belongs to
The page had grown to roughly ten cards in one flow, and three more surfaces were wanted on it: installed software, live metrics, and the service controls. Stacking those into the same column would have buried what the page is for. Tabs were mocked and rejected on 2026-08-28, on the grounds that the strip duplicated the sidebar and cost hero space. That objection is answered by where the strip sits: the hero and the triage band stay above it and never move, so "is this machine OK" is still answered without a click, and the strip partitions the detail underneath rather than navigating anywhere. WHY A PANEL DECLARES ITS OWN GROUP Most cards on a detail page are not the page's. They arrive through the ADR-010 get_asset_panels hook, so a page that sorted panels into tabs by knowing their ids would defeat the hook it is built on. Panels now carry an optional group instead, and the page only knows the five group names. Resolution happens server-side, in the asset-panels route, so that an absent or misspelled group becomes 'details' before the client ever sees it. The client renders known groups only; an unrecognised value that survived to it would be a panel that exists and is displayed nowhere. Contract 0.28.0 -> 0.29.0, additive - every existing plugin keeps working untouched, and the endpoints status panel is deliberately left ungrouped so the default path is exercised by a shipped plugin and not only by tests. A tab is offered only when something fills it. An empty tab reads as broken rather than as "this site does not collect that", so the strip is built from the groups that actually returned data, and a tab that disappears on refetch falls back to Details rather than leaving the page blank. AssetDetailTabs is core, not a PCDetail component: machines, printers, network devices and measuring tools all have the same shape, and the shared asset-detail shell they need starts here. Scope: product |
||
|
cproudlock
|
df083776b3 |
Offer the search box to whoever can actually search
The sidebar box was drawn for anyone signed in, on the reasoning that offering it to a signed-out reader gives them a control that can only answer 401. That stopped being the right test when the guest role became able to hold assets.view: the box was hidden from exactly the visitor the grant was reopened for, and a guest could reach search only by typing /search?q= by hand. The client cannot decide this. Its hasPermission getter reads the signed-in user's permission list and a guest has no user at all, so asking it about a guest answers no for everyone. get_navigation already had the problem and already solved it by letting the server decide, so capabilities does the same thing and reports what the CALLER holds. It reports booleans about the caller and never what the guest role holds in general, so it tells an anonymous reader nothing the rendered page does not. Public because it is the pre-login call that decides whether the box is drawn at all. Members who lack assets.view now lose a box that only ever 401'd for them. On a failed call the box stays hidden rather than falling back to the old signed-in test, which would put it back in front of those same members. Scope: product |
||
|
cproudlock
|
d9eb7a74db | Regenerate the project map after promotion | ||
|
cproudlock
|
dbc43162c5 |
Pin the approved content digest on an endpoint command
Delivery descriptors for run_script and update_app are resolved when the
agent claims a command, not when an operator queues it, and the command
row recorded no digest: the enqueue path resolved the descriptor purely
to validate it and threw the result away.
So what an operator approved and what an endpoint runs were two separate
lookups against mutable content, minutes or hours apart. Anyone able to
replace a script body or an installer package could do so in the gap.
The claim then re-resolves, the descriptor faithfully describes the new
bytes, the agent verifies correctly, and it runs as LocalSystem. The
audit trail names the operator who queued the original.
The gate on the verb and the gate on the payload were also different
grants: pushing needs endpoints.deploy, but replacing what gets pushed
needs only applications.edit.
pinnedsha256 records the digest as it stood at enqueue, and the claim
refuses a descriptor that has moved, failing the push with an audit row
instead of shipping it. Null for actions that deliver no content, and
for rows queued before this migration, which claim as before. A
descriptor that vanished entirely is not treated as tampering: that path
already sends the command through undescribed so the agent fails closed.
Separately, a result may now only finalize a command that is actually
running. That needs no agent change and closes two things: reporting on
a command that was never claimed, and overwriting a recorded failure
with a success.
The agent now sends its hostname on the result callback so the server
can check it owns the command it is finalizing. The shared collector key
does not say who is calling, so without it any agent can post a result
for any command id. The server treats the parameter as optional for
agents that predate this and enforces it when present, so there is no
flag day. This is a floor, not a fix: only a per-endpoint credential
makes the caller's identity provable, and that is tracked separately.
The .NET side is unbuilt here and needs a build plus a win11 VM run.
Scope: product
(cherry picked from commit
|
||
|
cproudlock
|
ccd8221adf |
Key the client address on the connection, not a header the caller sends
client_ip() read the first X-Forwarded-For hop unconditionally. The
login rate limit and the account lockout are keyed on that value and the
audit log records it, so rotating the header per request bought an
unbounded budget and an untraceable trail.
This walked around TRUSTED_PROXY_COUNT, which exists to decide exactly
this. The app leaves ProxyFix off by default and says why, ten lines of
comment away: ProxyFix trusts X-Forwarded-*, a client can send those
itself, and the login limiter is keyed on the address. Then client_ip
read the header anyway, so the stated protection never applied.
Resolution is now from remote_addr, which ProxyFix has already filled
from the header wherever a proxy is genuinely configured. Both cases end
at the same trustworthy source, and the port still comes off because ARR
forwards clientip:port and ProxyFix passes it through verbatim.
geenforce and tools had each hand-rolled the same mistake for their own
limiters. Rather than duplicate security-relevant logic a third time,
client_ip and strip_port join the shopdb.api surface (contract 0.28.0).
geenforce's _trusted_client_ip, which already refused the raw header for
its IP allowlist, is untouched and was the model for this.
The stability table was also missing rows for 0.26.0 and 0.27.0.
INSTALL-WINDOWS-IIS.md never mentioned TRUSTED_PROXY_COUNT, so a site
installed from it runs behind IIS with the setting at 0: every request
looks like it came from IIS, the whole site shares one rate-limit
bucket, and the audit log records the proxy for every action. It now
says so, and says not to set it when nothing is in front.
Three test modules simulated separate callers by setting
X-Forwarded-For, which is the header a real attacker rotates to escape
the limit. One of them, test_client_ip_prefers_forwarded_first_hop_
stripped, asserted the defect outright. The suite could not have caught
this. They now drive REMOTE_ADDR, with a regression guard that forging
the header buys nothing.
Scope: product
(cherry picked from commit
|
||
|
cproudlock
|
5f27f81e88 |
Clear the review blockers on the merged wilm work
A multi-agent review of the merged branch produced 29 findings that survived
adversarial verification. This clears the blockers and the four worth fixing
before it reaches main. The design held up well: path-traversal defence on all
four serving routes is clean, the agent's SHA-256 check fails closed, enqueue
authority is proportionate and audited, and no site literals or credential leaks
in the new C#.
THE TESTS NEVER RAN
pytest.ini sets testpaths = tests, so plugins/endpoints/tests/ was collected ZERO
times. Fifty-five lines of delivery assertions were dead code, and run directly
five of them failed. plugins/printedparts/tests/ had been invisible for longer.
That is the worst shape a gap can take: the suite is green, the coverage is
imaginary, and the collector auth on a download route could be deleted without
anything going red.
Both files moved under tests/. tests/test_no_hidden_tests.py now fails if a test
file appears anywhere pytest will not look; it caught a staged copy inside the
installer bundle on its first run, and was verified by planting a violation.
The five failures were stale expectations, and they split by cause. His: the
applications panel made six panels where his test said five, and 'applications'
belongs in KNOWN_PAYLOAD_KEYS now that it is persisted - his own test contradicted
his own feature. Ours: we renamed the events card to activity, moved the
fleet-wide cards to endpoints.fleet, and made the list route a bare route('')
which Flask rejects unless the blueprint is mounted under a prefix.
FOUR FIXES
The snapshot table's key was a 32-bit autoincrement on a table rewritten
wholesale every check-in, so it climbs at (endpoints x apps) per report and
exhausts on MySQL. Widened while the table is unreleased. Widening it broke every
insert on SQLite, which only autoincrements a column typed exactly INTEGER
PRIMARY KEY, so it carries a dialect variant - and nothing caught either fault,
because no test inserted into that table at all. One does now.
Ingest truncated displayname but not displayversion or publisher. On strict MySQL
an over-long registry value fails the whole check-in transaction, so one machine
with a malformed Uninstall entry goes silent entirely. SQLite does not enforce
lengths, so the suite would never have shown it.
update_app's version: branch pinned AppVersion.sha256, but that route resolves the
version to its APPLICATION and serves application.installpath - so the digest
described an artifact the route never sends, and where it was null the agent had
nothing to verify and failed open. It now pins the file actually served.
The same branch ended the delivery URL at the id, so an MSI arrived with no
extension and could not be routed through msiexec. It carries the filename now.
Agent packaging moved off require_role('admin') to a new endpoints.agentpackage
permission. The generated ZIP carries a live collector key, so this is a heavier
grant than deploying to one endpoint - and a role cannot be given to one person
for one job the way a permission can.
Full suite 2231 passed (2198 before, plus the 22 that never ran and 11 new),
frontend 314.
|
||
benjamin.baker
|
1fa005eb8a |
Green up
Tracking docs/openapi.json (so the app can serve the spec at runtime) makes docs/ exist on the published GitHub mirror, which defeated the skipif(not DOCS.is_dir()) guard in the docs publish/version tests: they then ran against a prose-less checkout and failed on CI. Key the skip off docs/*.md prose instead, matching the canonical published-checkout definition in scripts/check-gates-ran.sh. Also: the OpenAPI generator emitted a trailing-slash path (/api/docs/), which a strict proxy like IIS treats as a redirecting resource a subpath-mounted client cannot follow; normalise trailing slashes in _norm_path and correct the committed spec. And allow docx (python-docx) as an undeclared import: it is used only by scripts/docs/generate_docx.py, a developer tool that never runs on a deployed site. |
||
|
cproudlock
|
7a8acbbcb5 | Regenerate the project map after promotion | ||
|
cproudlock
|
13c46af5ce |
Rebuild the dashboard around what needs a person
The board was one card per source in a fixed grid, and both halves of that hurt. Every source got the same small box whatever it had to say. Warranties showed 5 of 50 behind "and 45 more"; the crash card truncated at 4; a one-row card sat beside a four-row card and left a hole. Nothing had room, which is the complaint that started this: not enough space to relay the data. And a box per source flattened priority. An enforcement failure and a warranty notice looked equally urgent because each owned a card, while the largest type on the page belonged to counts that are true every day and need nobody. WHAT THE PAGE IS NOW A severity rollup, one worst-first queue of everything that needs a person, and a row of context beneath it. The queue merges every critical and warning row and sorts them, so reading order is work order. It is full width, which is where the room came from. Rows are capped per source, because fifty expiring warranties must not bury two crashed PCs, and what did not fit is named per source with a link to it - a bare "and 45 more" that leads nowhere is worse than saying nothing. The rollup counts ROWS, not cards, and filters the board in place rather than navigating away, so the counts and the rows cannot disagree. Severity, position and permission were already declared per card and already sorted on. The old layout threw that away; this only stops it. WHAT WENT, AND WHY IT SHOULD NOT COME BACK Four asset tiles plus an "All assets" tile: machines + PCs + printers + network IS all assets, so the page showed a total beside its own addends at the same weight. It is now a line under a rule, where a total belongs. The printer strip. It carried the printer count a third time - the tile row and "Total Printers" were already the same number - and "Online" equalled total, so three tiles held one fact. A genuinely low cartridge already reaches the queue through the printers-supplies card, which is the moment anyone wants it. Three cards whose whole job was to display 0. Recent Devices: the largest block on the page, listing five machines that were all "In Use" with business unit "TBD". Recency is not a reason to look at something. CONTRACT 0.26.0: 'shortname' Cards may declare a short label for their rows in the merged queue. Their 'title' is written as a heading - "Endpoints with crashes or shutdowns" - and as a row badge it ran wider than the hostname it qualified, putting the badge rather than the asset at the start of every line. CSS-truncating the title was tried first and was worse: four consecutive rows reading "ENDPOINTS WITH CR..." say nothing at all. Optional and defaults to the title, so a plugin that does not declare one is unaffected. Additive field, minor bump. ALSO: "silent 1 days" A meta suffix is written for the plural case and the card cannot know the value. One row saying it was a shrug; the merged queue puts every quiet endpoint on one screen, so it said it five times in a column and read as broken. Now singularised at exactly 1, on the first word, keeping what follows: "1 event (48h)", "silent 1 day". The test asserting ' ms' survives is what caught the first version turning it into ' m'. DashboardRow.vue is extracted because the queue and the context panels render the same row, and the map tooltip, link fallbacks and chips are too easy to get subtly different in a second copy. Verified on screen against live data. Full suite 2178 passed, frontend 301. |
||
|
cproudlock
|
70e909054f |
Document the endpoints plugin in the catalogue
docs/PLUGINS.md listed sixteen of the seventeen bundled plugins. The missing one was endpoints - the newest, and the one another site is most likely to ask about, since it is the Fleet console and the live panels on a PC detail page. The row records the two things that are not guessable from the name: it depends on computers because an endpoint is telemetry ABOUT a PC asset rather than a second record of it, and its four permissions are split deliberately. Read is grantable to guests by the .view suffix rule; fleet, command and restart are not, and the plugin declares no override, so the command channel cannot be opened to anonymous callers by ticking a box. Ingest is called out as NOT being a route here - agent reports arrive on the generic collector route under a service token scoped collector.ingest.endpoints. Somebody looking for the ingest endpoint in this plugin will not find one. applications/ stays absent on purpose: it has no manifest and no plugin.py, so per the manifest-less rule it is core frontend surface that always ships, not a plugin a site can choose. |
||
|
cproudlock
|
b8fc2e9e33 |
Upgrade the eight shipped dependencies carrying advisories
scan_sbom.py reported 50 advisory matches against shipped components, 16 of them at or above HIGH. This clears all of them: the scan now reports 0 shipped findings and the gate passes. pyjwt 2.12.1 -> 2.13.0 idna 3.13 -> 3.19 axios 1.13.2 -> 1.20.0 (29 advisories) postcss 8.5.6 -> 8.5.26 dompurify 3.4.11 -> 3.4.14 nanoid 3.3.11 -> 3.3.18 follow-redirects 1.15.11 -> 1.16.0 form-data 4.0.5 -> 4.0.6 Every one is an in-major bump inside a range the manifests already allowed, so this is a lockfile refresh rather than a migration. The Python side was compiled with --upgrade-package for each name so nothing else moved; the diff is two version lines and their hashes. nanoid is worth recording: it is transitive under postcss at ^3.3.11, and its advisories are fixed on both the 3.x and 5.x branches. 3.3.18 is the fix that is actually reachable. The declared floors for axios and dompurify were raised to the versions in the lockfile. They were ^1.6.0 and ^3.4.11, which still nominally admitted the vulnerable range - npm resolves to the highest match so an install was never going to pick one, but the manifest should state the minimum safe version rather than rely on that. axios 1.20 adds optional proxy-agent support, which pulls https-proxy-agent and agent-base into the tree. Both are Node-only and neither reaches the browser bundle; the build was checked for them. The main chunk grew 256kB to 271kB (85.7kB to 90.5kB gzipped) from the rest of the axios delta. The PyJWT entry in the allowlist was removed rather than kept. It stopped matching the moment the upgrade landed, and an allowlist that accumulates entries for advisories that no longer apply is one nobody rereads. The reasoning is preserved as the worked example in docs/DEPENDENCY-SECURITY.md. Verified: full suite 2161 passed, frontend 273 passed, build clean, npm audit --omit=dev reports 0 vulnerabilities. |
||
|
cproudlock
|
1cc2a6b782 |
Recommend the smallest upgrade that clears an advisory, not the highest
The rollup took the highest fix version across a component's advisories. That is wrong whenever an advisory is patched on more than one release branch. nanoid is the case that exposed it. Both advisories against it are fixed on two branches - 3.3.16 and 5.1.16, 3.3.18 and 5.1.6 - and the rollup recommended 5.1.16 with a MAJOR warning. But nanoid is transitive, pulled in by postcss at `^3.3.11`, a range no 5.x release can satisfy. The recommendation was not merely a bigger job than necessary, it was unreachable: the actual target is 3.3.18, a patch inside the existing range. Per advisory the target is now the lowest published fix above the version in use; per component it is the highest of those, since one upgrade has to satisfy every advisory against it. Fixes on older branches are skipped rather than offered as a downgrade - axios advisories list 0.30.x fixes that would have been a downgrade from 1.13.2. This also gives the MAJOR label meaning. It now fires only when no fix exists on the current branch, rather than whenever some branch happened to be higher. |
||
|
cproudlock
|
c4e2db4755 |
Route the dependency-security doc from START-HERE
START-HERE routes by the question somebody arrived with, and 'are we exposed to this CVE' had no entry - the SBOM was only mentioned in the Windows update doc, which is not where anybody asking that would look. |
||
|
cproudlock
|
794bb4668f |
Add vulnerability scanning on top of the SBOM
The SBOM was generated and installed onto every server, and nothing read it. "Are we exposed to this advisory" was still a person going and looking, which meant it happened when somebody remembered to. scripts/scan_sbom.py queries OSV.dev by purl - no account, no API key, no agent, no vulnerability database to maintain on a box that cannot reach the internet. The SBOM is already the input OSV wants, so this is a lookup against what shipped rather than a second scan of the tree that could disagree with it. Three decisions worth recording: Shipped and build-only findings are both reported, but only shipped ones can fail a build. Two thirds of the components here are build tooling the SBOM already marks scope 'excluded', and the highest-severity match in the whole tree today is a CRITICAL against the test runner. A gate that fails a release over that trains everybody to bypass it, and then it catches nothing. Severity is taken from the advisory, never derived. Where there is only a CVSS vector the report says UNKNOWN, which is shown and never gates. A guessed score is worse than an honest gap because somebody would act on it. The allowlist is a record rather than a mute button: each entry carries the reason, the accepter, the date, and what would invalidate the reasoning. The first entry is a worked example - a PyJWT algorithm-confusion advisory that needs both mixed algorithm families and a raw JSON JWK as the key, and this application configures neither. The report ends with a per-component upgrade rollup, since that is the part somebody acts on. A target on a different major version is labelled rather than listed beside patch bumps as though the two cost the same. Current state: 8 shipped components have fixes available, none of them exploitable through a path this application opens today. |
||
|
cproudlock
|
0baed4cc80 |
Close the gaps against the design: critical tile, filters, policy card, action placement
Four things the mockup showed and the first cut did not have.
FLEET
- A 'with critical events' tile, from the stability route that existed and
nothing called. Clicking it narrows to rows whose health is not ok - client
side, because the health rollup is assembled per row after pagination, so
it narrows the PAGE rather than the fleet. Worth knowing before the count
is read as fleet-wide.
- PC type and Location filters. The list route already accepted both; there
was simply no control. Both use listAll, not list: perpage is clamped to
100 server-side, so a long location list would lose its tail in a dropdown
while still looking complete.
- The machine cell shows number and name together where they differ, so
'3105 Okuma' rather than a bare number.
PC DETAIL
- A Policy card, contributed by the computers plugin. It matters because the
effective policy already narrows the command allowlist, so without it the
reason a control is unavailable on a given PC is written down nowhere.
Provenance, so it sits in the reference column, not the triage band.
Its route returns an EMPTY field list rather than the feature-gate 404 when
policies are disabled. The panel contract hides a card with no fields, so
it disappears on sites that have not enabled the feature - a 404 would have
rendered a broken card on most of them. That was flagged in the design
review as a snag; this is the fix.
- Refresh Status and Restart Computer moved out of the page header into the
triage band. They act on the MACHINE; Print Label, Edit, Retire and Back to
List act on the RECORD, and mixing them put the most consequential button
on the page - it halts whatever machine the PC serves - between printing a
sticker and editing a form. In the band it sits beside the liveness chip
and the machine it would take down.
The band takes them through an 'actions' slot rather than knowing about PC
verbs, so a measuring tool can put 'Calibrate' there without the core
component learning anything about computers.
Full suite 2124 passed, vitest 260 passed, gates green, spec at 333 paths.
|
||
|
cproudlock
|
414a7243d7 |
Merge events and commands into one Activity feed, and lead with live state
Two changes to the PC page, both about what a responder meets first. LIVE STATE FIRST. The contributed panels moved to the head of the card flow. The grid is a two-column FLOW (column-count, with display:contents on the column wrappers), so a card cannot be pinned to a side and only document order decides where it lands - putting the panels first fills the left column with endpoint status, services, processes, interfaces and activity, and leaves identity, hardware, network and location to the right. Panels still self-order among themselves by declared position and still self-hide when no plugin answers for the asset type. ONE ACTIVITY FEED replaces the Recent events card. Events and commands answer the same question - what happened to this box - and reading them as two lists meant interleaving by eye. GET /api/endpoints/asset/<id>/activity merges them newest-first, capped at 15. Both are keyed by assetid, so it is two indexed reads and a merge sort rather than a join, and it renders through the existing ADR-010 list renderer with no bespoke frontend work. A failed command is marked critical, because that is the row somebody needs to notice. Not collapsed by default, unlike the card it replaces. The core AuditLog is deliberately NOT a third source: its rows carry the action as the entity name and the asset only inside a details string, so including it would mean parsing prose to find out which machine a row belongs to. Requester names are withheld from callers who cannot themselves issue commands, via caller_holds. A read-only viewer - a guest, on a site that grants this panel - gets the history without a roster of who restarted what, and the lookup is skipped entirely rather than fetched and discarded. requestedby holds a user ID, so names resolve through one batched query, not one per row. Verified on dev telemetry: a queued command shows as 'Collect now / Pending by admin' above the Windows events, ordered by time. Full suite 2124 passed, vitest 260 passed, naming/contract/parity gates green, spec regenerated at 332 paths. |
||
|
cproudlock
|
54f0857e13 |
Split the fleet roster from per-endpoint reads, and stop offering controls nobody can use
Twelve endpoints routes shared one permission, which made a reasonable policy impossible to express. A site that wants signed-out visitors to see live status on a PC page had to hand them the entire fleet - including every logged-in username on the floor - because the per-asset panels and the fleet-wide list were gated on the same string. endpoints.fleet now gates the list and the three dashboard routes. endpoints.view keeps the per-asset reads: status, services, processes, interfaces, events, command history. The name is load-bearing. guest_grantable_permissions() opens any '.view' or '.read' permission to the guest role, so calling this one 'endpoints.fleet' rather than 'endpoints.fleetview' is what keeps the roster off the public surface while per-asset telemetry stays grantable. Verified both ways. Also: the services panel emitted Start/Stop row actions to every reader, because row_actions_for() never consulted the caller. Clicking was always refused server-side, so nothing was exploitable, but showing a read-only viewer controls over a shop-floor PC is the wrong signal - and it stops being cosmetic the moment a site grants that panel to guests. That needed a contract addition rather than a local check: caller_holds() on shopdb.api answers whether THIS caller holds a permission without refusing the request, which is the complement to require_permission for a surface that serves everyone but should only offer what the caller can act on. Guests are included, since a signed-out visitor holds whatever the guest role was granted. Reaching into shopdb.utils.authz instead would repeat the ADR-002 violation the contract gate caught earlier in this branch. Contract 0.24.0 -> 0.25.0. DEPLOYMENT: endpoints.fleet is a new permission. Run 'flask seed permissions' and grant it to the roles that should keep seeing the Fleet page. Until then that page is admin-only, since admins bypass every check - a safe direction, but not an automatic one. Full suite 2124 passed, naming and contract gates green, map and spec regenerated. |
||
|
cproudlock
|
af42ba3589 |
Register the endpoints list without a trailing slash, and stop hiding its errors
An endpoint reported telemetry correctly for hours while the Fleet page showed
"No endpoints found". The PC detail page displayed its panels the whole time, so
ingest was plainly working; only the list was empty.
The list route was the one route in 331 paths registered with a trailing slash:
@endpoints_bp.route('/', methods=['GET']) -> /api/endpoints/
The browser client calls it without one, so Flask answered a 308 redirect to an
ABSOLUTE url. On dev that resolves and nobody notices. Behind the IIS subpath it
cannot: with no ProxyFix, Werkzeug builds that Location from the scheme and host
it believes it has, so it emitted http:// and dropped the /shopdb prefix. A
browser will not follow an https page to http, the request failed, and the
page's catch block rendered the empty state.
Every other route in the plugin worked throughout, because none of them carries
a trailing slash - the detail panels call /endpoints/asset/<id>/status and were
never redirected. Detail works, list empty is the signature of this bug.
Registered as '' now, so the rule is /api/endpoints. That is what
CONTRIBUTING.md's API section spells and what every other list route in the
product already did. The inventory and the generated spec are updated to match.
Also here:
- tests/test_core/test_no_redirecting_routes.py fails the build if any /api
rule is registered with a trailing slash and strict slashes. Verified by
planting the violation: it named the offending route, then passed once
reverted. It takes the shared app fixture rather than building its own
application - doing that rebinds SQLAlchemy and leaves the NEXT module
without tables, which showed up as three unrelated "no such table: settings"
failures visible only in a full run.
- The Fleet page now renders "Could not load endpoints (HTTP nnn). This is a
failed request, not an empty fleet." A list that reports a 403 and an empty
fleet identically is what made this take as long as it did.
- TRUSTED_PROXY_COUNT (default 0) applies ProxyFix when set. Off by default on
purpose: it makes X-Forwarded-* trusted, and a client can send those itself,
while the login rate limit and account lockout are keyed on the client
address. Set it to 1 on an instance behind IIS.
Full suite 2124 passed, vitest 260 passed, naming and docs gates green.
|
||
|
cproudlock
|
ae800772d1 |
Make the dev MySQL 5.6 index-prefix requirement actually reachable
The printers observedqueues migration failed with error 1071 on a 5.6 dev server whose file format was still Antelope. The unique index it needs is (hostname, queuename), two utf8mb4 VARCHAR(255) columns, which is 2040 bytes against a 767-byte cap. The compiler hook in shopdb/utils/mysql_charset.py was already emitting ROW_FORMAT=DYNAMIC, which raises that cap to 3072. What is easy to miss is that Antelope makes 5.6 IGNORE the row format and build a Compact table with a warning rather than an error, so the tables look created and keep the old limit. docs/DEPLOY.md has required innodb_large_prefix plus Barracuda for pre-5.7 servers all along; nothing pointed a developer hitting 1071 at it. So the troubleshooting row that said 'use MySQL 8 for dev' now gives the fix, names the silent Compact downgrade, and says such tables stay capped until they are rebuilt with ALTER TABLE ... ROW_FORMAT=DYNAMIC. DATABASE_URL also gains ?charset=utf8mb4, which DEPLOY.md already requires and this environment did not have. |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
8953ce8210 |
Ticking a section off crosses out the section
It struck the heading and left everything underneath looking untouched, which is the opposite of what somebody ticking "this section is done" means. The strike now covers every block under that heading, down to the next heading of the same or higher rank - so a top-level section takes its sub-sections with it. That needed the rank, and the HTML importer was throwing it away: h2, h3 and h4 all arrived as a flat 'heading', so a parent heading had no children to cross out. It was the visible half of a real loss - a converted article kept its words and lost its shape. Headings now carry their level and render at it, and a guide written before this reads as flat, exactly as it did. Code blocks, tables and images inside a finished section fade but are not struck. A line through a command makes it unreadable, and re-reading the command is usually the reason to look back at a step already done. The five imported guides were re-imported so they carry their structure. |
||
|
cproudlock
|
afce24379c |
Reading anything needs a permission
A sweep of the URL map found 165 GET routes that answered anybody, with no token at all: the whole asset inventory, every machine, PC, printer and network device, the USB checkout history and who is holding which drive, the staff directory with its photographs, every core report. Guest access had been argued about one feature-flag at a time while the front door stood open. Each of those now carries the read permission it belongs to. That also makes the guest role mean something: a site can open exactly the reads it chooses, and nothing follows by accident. Thirty-three endpoints stay public, and each is named with its reason rather than left to be inferred: the app shell and its bootstrap, the settings allowlist (the handler already filters an anonymous caller down to branding and a few keys), images fetched by <img src> which carries no Authorization header, what a display kiosk renders before anybody signs in, the parts and USB kiosks, the client scripts that run as SYSTEM before a user session exists, and a secure-send link whose unguessable token IS the authorisation. tests/test_core/test_public_read_surface.py is the standing gate, the companion to the mutating-route sweep that has guarded writes for months. A new public read fails CI until somebody adds it to that list in front of a reviewer. Four further reads need a login but no permission, because the handler decides from who you are - your own user record, your own API tokens, the role catalogue the grid renders - and the same test proves each still refuses an anonymous caller. Verified not vacuous by ungating one route and watching the sweep name it. The employees plugin gained employees.view; the directory is names, SSOs, photographs and reporting lines, and it had no permission of any kind. Around ninety tests asserted payload shapes against endpoints that answered anybody. They are about what the answer says, not about who may ask, so they are signed in now rather than the gate being loosened. The ones that assert an anonymous caller is refused were left exactly as they were. |
||
|
cproudlock
|
2331c03664 |
Tick a guide's sections off as you work through it
Only checklist items could be ticked, which answers "what have I done" but not the question a long procedure actually raises: where was I. Somebody looks away to do the thing the guide asked for and looks back at a wall of text. Every step and every heading now carries a checkbox while reading. Ticking one strikes it through and fades it rather than hiding it, because somebody scanning back up the page has to still be able to read what they did. The number is struck through with it, but the step's own "Step 3" label is not - that is the address of the thing, not part of what was done. Headings count, not just steps. A guide converted from an HTML article often has fifty headings and no steps at all, and those are exactly the guides long enough to lose your place in. That also means the card's "N steps to tick" stops reading as a broken feature on a guide that is entirely tickable. Reading only, as the checklists already were: in the editor a tick would be an edit, and an author clicking around their own draft would quietly change what the next reader sees as done. Progress stays per browser and per version - shopfloor PCs share a login, so one technician's ticks must not greet the next person on that bay, and step 4 of version 3 is not step 4 of version 2. |
||
|
cproudlock
|
151d7c4f44 |
Anyone not signed in holds the guest role
Whether somebody with no account may see something was being answered one feature at a time: a securesend_allow_guests setting, an enable flag per kiosk, and very nearly a third for guides - each with its own settings page, its own vocabulary and its own hand-written branch inside a route. Every one of them answered a permissions question, and this app already administers permissions per role. So a caller with no token is not nobody: they hold the seeded `guest` role. require_permission consults it when there is no identity, which is the entire mechanism. The role is seeded holding NOTHING, so an upgrade opens no door; a site opens one by ticking a box in the same grid it uses for every other role, and closes it the same way. Revocation is read on every request rather than cached, because taking guest access away is what somebody does the moment they realise what they opened, and it has to bite immediately. The redundant @jwt_required() above require_permission is gone from 184 routes. It was belt-and-braces before; now it is actively wrong, because it refuses a request with no token before the permission check can have an opinion, so a grant would appear to do nothing. What a guest may hold is bounded. Any .view or .read permission is grantable. A WRITE has to be blessed by whoever owns it, through the new get_guest_grantable_permissions hook (contract 0.21.0), and Secure Send is the only case that exists: sending a password to a vendor is the one write this product means to offer somebody who never signs in. Everything else is refused by the API and greyed out in the grid rather than warned about - a checkbox that quietly makes deleting an asset anonymous is not a warning anybody reads twice. A guest write records no creator: the trail says when and from where, never who, and that is the reason for the bar. GE-Enforce and Slides gained read permissions so they can be opened to guests without handing out the manage right. Their read routes accept either through require_any_permission, so nothing an existing manager could reach has moved. The navigation endpoint now returns only what the caller can actually reach. Without that a signed-out visitor gets a sidebar of the whole app where every link bounces off the login page, which reads as a broken site rather than as a closed door. A nav item may name several permissions; holding any one is enough, which is what GE-Enforce and Slides need now that a viewer and a manager hold different strings. Two things this found on the way: current_user is a LocalProxy, so `current_user is None` is False for a request with nobody behind it - harmless under a required JWT, an AttributeError the moment verification became optional. And an image pasted into a guide before its first save was never attached to it, so it was visible to editors and invisible to every reader; saving now attaches it, and `flask guides fix-images` repairs what is stored. The guides guest setting written earlier today is gone. Granting guides.view to the guest role replaces it, which is the point. |
||
|
cproudlock
|
8531559f22 |
A guide can be carried to another site
Guides are written on one site and needed on all of them, and the sites that matter are air-gapped from wherever the writing happens. Until now a procedure was stranded in the database it was typed into: the HTML importer could rebuild one from its original article, but anything authored in the editor had nowhere to go. flask guides export writes one .guide.json per guide, with every image the blocks reference carried inside it as base64. One file on purpose - a bundle that needs its images copied alongside it is a bundle that will eventually arrive without them. flask guides import reads them back, a directory at a time, and refuses anything that is not a bundle before it writes a row rather than half-importing a stray .json sitting in the same folder. What travels is the content: title, summary, tags, blocks, images. What does not is ids, revision history, authorship and publication state, because those belong to whichever database issued them - an imported guide starts as a draft at revision 1, owned by whoever ran the import. The SLUG does travel: it is the address somebody pastes into a work order, and the same guide should answer to the same name on every site. --replace updates the guide already carrying that slug instead of creating a second one, which is how a corrected guide is re-shipped. It drops the stored images first: the incoming bundle is the whole truth for that guide, and a guide re-shipped monthly would otherwise accumulate orphans nothing points at. Bundles refuse a version they are too old to read rather than importing the parts they recognise. A newer site's guide is not a guide this one can be trusted to render. Proven by round-tripping a five-guide export back into the same site, including one carrying two PNG screenshots, and reading it in a browser. |
||
|
cproudlock
|
c8efadefba |
An HTML article can become a guide
A site adopting this app arrives holding procedures already written: styled HTML pages that get mailed around, opened off a share, and edited by whoever still has the file. Those pages are exactly what a guide is for, and retyping one into the editor is how a good procedure stays on a share forever. flask guides import-html reads the page and produces blocks. It recognises the vocabulary those pages actually use - a step box with its own number, an alert in one of three colours, a code block with a copy button, a checkbox list, a table with a header row - and maps each to the block that already renders it, lifting embedded images into guideimages so a document stops carrying its screenshots as base64. Page furniture goes: the header, the footer, and the table of contents a guide builds for itself from its own headings. The conversion is lossy on purpose. Colour, fonts, width and nested layout are dropped, because storing blocks rather than markup is what makes dark mode, printing and tickable steps work at all. So an import is always a DRAFT and the default is a dry run that reports what it would produce and writes nothing. Blocks go through the same validator the API uses: an import must not be able to store something the editor could not have produced. The parser is tolerant, because real articles are not well-formed. An unclosed <p> is normal HTML and the browser closes it at the next block element; without the same rule one unclosed tag swallows the rest of the article into a single paragraph, which is what it did before the test caught it. Proven end to end against four real articles, including one carrying two PNG screenshots, read back in a browser. |
||
|
cproudlock
|
f59aca5a1b |
USB check-in / check-out audit, as a report rather than a paged list
CMMC removable-media tracking is an audit obligation, and answering "who had this drive, and when" meant paging through the check-in/out list by hand. GET /api/usb/reports/audit returns the events - a take and a return are separate rows, because a count cannot answer the question an audit actually asks - with summary figures riding along for the people who only want the shape of it, and a CSV for the people who want it in a spreadsheet. The window is named rather than a day count: today, last month, six months, a year, all time. Nobody should have to work out that a year is 365. Both directory modes answer in one voice. The external path reads the cmmc_usb checkinoutlog; self-hosted expands one usbcheckouts row into its two ends, so a returned drive is distinguishable from one still out, and the window filter runs in SQL against either end of the loan - a drive taken before the window and returned inside it is a check-in the window has to show. Rows come back from the mode-specific half; the CSV, the summary and the response shape are built once in the route. Ordering is total in both modes. Timestamp alone ties for two events in the same second, and the same query then returns them in a different order each run, which reads as the log being unstable. |
||
|
cproudlock
|
95ad4f1ef3 |
Docs: the USB kiosk, its endpoints and why the list costs a badge
Changelog, the plugin table, the three kiosk endpoints in the API inventory, and regenerated openapi.json / PROJECT-MAP. |
||
|
cproudlock
|
90c3c3102b |
Docs: guides in the indexes, the stalest-kind rule, and the changelog
Guides was shipped without an entry in PLUGINS.md or START-HERE, so a site adopting the product had no way to find out it exists. BACKUP-KINDS.md described a host's badge as coming from the most recently confirmed kind. It has not worked that way since the fleet report was reworked: the freshest revision per (host, kind) is taken, and then the STALEST of those kinds decides, because reporting the freshest would let a healthy NTLARS backup mask a part-marker capture that stopped a month ago. Stalest among kinds that can be JUDGED - a kind never confirmed is unmeasurable, not stale, and treating it as the worst value made a host with a working backup show no badge at all. The backups row also still advertised UDC and CMM kinds that do not exist. api-inventory, openapi.json and PROJECT-MAP regenerated. |
||
|
cproudlock
|
580c0d0c80 | Record the tab title and Guides packaging fixes in the changelog | ||
|
cproudlock
|
bd0e9695e5 |
Document the guides API, and move Guides beside Slides
The repo already enforces that every served route appears in docs/api-inventory.json, and the guides endpoints did not - so the parity test was failing. They are now listed with auth, params and a working example each, beside knowledgebase because the two get read for the same reason. docs/GUIDES.md covers what the inventory cannot: the block schema and why it is blocks rather than HTML, the id rule that makes a tick rememberable, what the server refuses and why validation happens at the door, the working-copy / revision / published-pointer split that stops a live procedure changing under somebody following it, and a worked example of creating and publishing a guide from a script. It also states plainly what does not exist yet - no importer, orphaned images never pruned - and what converting existing documents will actually be like: structured HTML and Markdown map onto block types nearly one for one, while a plain-text runbook lands as paragraphs and needs marking up by hand afterwards. Generated ids have to stay stable across re-imports, or every re-import resets readers' progress. Guides moves to position 8 in the sidebar, beside Slides, rather than down among the reference lists in Information. A guide is something somebody goes and does something with, not a catalogue they look things up in. |