dev
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
377cc69264 |
Make a partly-undone import retryable, and refuse a plan it cannot deliver
Five findings from the review of the WINC import, plus the rendering each fix needed to be usable. An undo that hit a conflict wrote status 'undone' anyway, and undo_batch returns immediately for an undone batch. So the one entry that needed a retry was the only one that could never get it: the human resolves the conflict, clicks Undo, and gets a no-op. A conflicted undo is now 'partially-undone', undoneat is stamped only when the batch is genuinely finished, and the batches table offers "Retry undo" for that status. The column was String(16) and the new value is exactly 16 characters, so migration 7d38 widens it to 32 rather than rely on an exact fit. _UNDO_REFERENCES listed only assets, so a business unit this batch created was hard-deleted even when something else still pointed at it. Two of the three missing referrers are real FKs (dashboarddefaults.businessunitid, businessunits.parentid): MySQL raises and the whole undo rolls back, so the batch cannot be undone at all. The third, notifications.businessunitid, has no FK, so the delete succeeded and left a notification pointing at an id that no longer exists - silently, on both engines. All four are listed and the guard is tested through the route. The reference guard also queried tables that need not exist. A lean site drops the tables of plugins it does not install (ADR-014), and notifications is plugin-owned, so on such a site the query raised 1146, rolled the undo back and left the batch permanently un-undoable. Absent tables are skipped; a table that does not exist cannot hold a reference. _PACKET_GROUPS reports on location, but apply_plan only ever creates business units - it sets locationid only when the location already resolved. A plan needing a new location asked the admin to acknowledge it, got the tick, wrote nothing, and reported nothing. build_packet now takes the groups the writer can actually create; anything else needed is 'uncreatable' and blocks at plan time, where the admin can act on it, rather than dying at commit on a message about acknowledging something the screen had no control for. The writer refuses the same case independently. Both the console packet and the import page name the blocking groups: setting blocking without printing the reason would leave the banner pointing at nothing. The web undo route caught ValueError only, so refusing a batch belonging to another importer surfaced as a 500. That refusal is new too: the journal is a general primitive and undo_batch does not filter on kind, so any other importer's batches were reversible through this endpoint, carrying WINC's references map - which lists none of that importer's tables, making every reference check pass and every delete unconditional. Nine tests. The two guards were verified by planting violations: dropping dashboarddefaults from the references map, and passing creatable=None, each fail the test written for them. Scope: product |
||
|
cproudlock
|
13be0ef90f |
Import the wilm agent store and WINC adoption work, with four fixes
Eleven commits from the sister site: a reversible Maximo/WINC machine-adoption import, a versioned agent binary store with upload and rollback, remote-BuildKit CI, tray version reporting, and UI fixes. Applied as a patch again - the published mirror rewrites hashes, so the branches have unrelated histories and git refuses to merge them. Reviewed by three agents before merging. Four things fixed here. THE PUBLISH WORKFLOW WAS DEAD. The new remote-BuildKit step in publish-images.yml sits at column 0 instead of indented under steps:, so the whole file is invalid YAML and GitHub would refuse it. Their entire GHCR distribution channel - which Start-ShopDBDev.ps1 and the compose files consume - was broken at the source and clearly never run. Indented; all five workflows now parse. THE IMPORT'S APPROVAL CHECK COULD NOT FAIL. commit() built the plan and the approval packet from the same upload and then asked the writer whether they matched, so packet_matches_plan compared a hash against itself. The documented contract - "a human approves a specific, frozen plan, and the writer can prove nothing drifted" - did not exist in the web path: an admin could preview file A and commit file B, gated only by the needs-create group NAMES they ticked. The preview's planhash is now required at commit and checked against a freshly rebuilt plan. Their tests committed without previewing, which is what let this through; they go through the real flow now, and three new tests pin it - all confirmed to fail with the check stubbed out. endpoints.agentpackage QUIETLY GREW. It used to mean "wrap the staged exe in a ZIP"; with the upload routes it also means "choose the executable every future package installs and runs as SYSTEM". Before this, staging needed filesystem access to the server. The permission description still said only "generate and download a package", so an admin granting it could not see what they were granting. Reworded. NOTHING RECORDED WHAT WAS STAGED, OR BY WHOM. The manifest held version, size and timestamp; re-uploading a version replaces the bytes in place under the same label. A sha256 of the stored file is now recorded, and the upload is audit-logged with it - after an incident there was previously no answer to who put that binary on the fleet. Also documented the nine new routes in docs/api-inventory.json (that gate cannot run on the mirror, so it falls to us at promotion time again), regenerated the spec and the project map, and kept our own docs where they conflicted: docs/ is scrubbed from the mirror, so their DEV-BUILD-TEST.md still had the dev-be channel mapping we corrected, and their PLUGIN-HOOKS.md was a new 45-line file rather than our 737-line canonical one. Their journal section is appended to ours. Contract version comes across as 0.30.0 for a genuinely additive surface. Suite 2369, frontend 324, gate green. Scope: product |
||
|
cproudlock
|
259fe0de8f |
Read a numeric natural key as a name before reading it as an id
An Azure device import died part-way with a foreign-key error, and the crash was the lucky outcome. The Azure adapter routes a device that already exists to its real asset by writing that asset's assetnumber into the assetid column of the CSV it generates, expecting the engine to look it up. The resolver instead takes any all-digit string at face value as a literal id. This site numbers operations 0NNN, so numeric assetnumbers are ordinary here: `0704` was read as id 704, which does not exist, hence the error. `0600` would have been read as id 600 - a printer - and bound a PC to it silently, reporting success. Two changes, both needed. The resolver now tries the label first for an all-digit string and only falls back to reading it as an id when nothing is named that. The face-value behaviour exists for a real reason - exports round-trip and carry real ids, as its original comment says - so it stays as the fallback rather than being removed. Somebody writing `0600` in a column keyed on assetnumber means the asset called 0600, and binding the row to an unrelated record is far worse than failing. _existing_hostname_map now hands over the assetid rather than the assetnumber. Giving the resolver the id it is going to treat the value as makes the passthrough correct by construction instead of correct by luck. Verified against a restored copy of the production database: 234 devices, 221 matching an existing computer and updating in place, 13 new, no problems. The same run previously failed on the foreign key. Neither half was wrong alone. The passthrough dates from the original CSV import and the hostname map from the wilm merge months later; they only meet at a site whose natural keys are numeric. Scope: product |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
9d96006eae |
Audit log retention: a way to purge, off by default
The audit log is append-only and nothing ever removed a row, so it grows for as long as a site runs. This site reached tens of thousands of pages, which is past the point of being useful: nobody pages through that to find anything. Retention is OPT-IN and the default keeps everything. An upgrade that silently started deleting a site's audit history would be indefensible, so audit_retentiondays defaults to 0 and nothing is removed until somebody sets a number. The Site & Facility card shows the log's size and exactly how many entries a purge would delete BEFORE anything is pressed. Deleting happens in committed batches. That is not an optimisation: one DELETE over a million rows builds a single enormous transaction and holds locks across the table while it runs, which on this server means blocking the collectors posting inventory. The API caps its batch count so the request cannot outlive a proxy timeout, and a run that stops early reports how much is left rather than claiming it finished - a purge reporting success over 900k remaining rows would be the same silent half-failure this codebase keeps getting bitten by. `flask audit stats` and `flask audit purge` for the server, the latter reporting by default and deleting only with --apply, with no batch cap because it is not behind a proxy and a purge that stops halfway leaves the operator guessing. audit.purge is a separate permission from audit.view. Reading the log and destroying it are different acts, and plenty of people need only the first. Chunking selects a page of ids and deletes those rather than using DELETE ... LIMIT, which is MySQL-only while the suite runs on SQLite. |
||
|
cproudlock
|
fa0f6d7ebc |
Record what nobody can find, and a spelling corrector that refuses to guess
Two halves of one question - how to handle a misspelled search - deliberately kept separate, because only one of them should be turned on today. WHAT NOBODY CAN FIND IS NOT RECORDED ANYWHERE. A search returning zero results is the only evidence of the gap between what people look for and what is there, and it vanished. Logged now at INFO with a stable prefix, so a week of it greps into a list. That list is what should decide whether correction is worth wiring in: multi-word matching and cross-field matching both changed in the last day, so a good share of what used to fail may already be found. The failures left over might be typos, or vocabulary nobody has entered, or records that genuinely do not exist - and each wants a different answer. THE CORRECTOR IS BUILT AND NOT WIRED IN. shopdb/core/services/spellfix.py, with tests, ready to attach in about ten lines once there is evidence about what to attach it to. NOT SOUNDEX, which was the obvious candidate. MySQL has it and SQLite does not, and the suite runs on SQLite while production runs MySQL - so a SOUNDEX() in a query is either an error in every test or a production path no test executes. It is also wrong for this data: soundex is English-name phonetics, four characters wide, and it DISCARDS DIGITS, so CSF16 and CSF17 hash identically. Half of what people search here is an identifier. So: character distance in Python, same behaviour on both dialects. THE VOCABULARY IS THE DATA. No dictionary holds Genspect, Telesis, Keyence or wax-trace. Terms come from the columns being searched, which also means a vendor added this morning is correctable this morning. TWO GUARDS, and they are the point rather than a detail. Digits must match EXACTLY: CSF16 to CSF17 is one edit and a different bay, so anything carrying digits is either right or not correctable - while cfs16 to csf16 still works, because the guard is on the digits and not on identifiers wholesale. And the first character must match, since typos land mid-word far more often than on the first key, which costs almost no recall. Distance is Damerau-Levenshtein so a transposition costs one edit rather than two - Keyecne for Keyence is the commonest error there is, and plain Levenshtein pushes it past the threshold on short words. Allowed distance scales with length. A tie returns NOTHING: two equally good candidates means there is no answer, and offering either implies a confidence that is not there. It suggests; it never rewrites. Silently searching for something else is how somebody orders the wrong cartridge. 13 tests, weighted toward the refusals, because those are the cases where being wrong costs something. |
||
|
cproudlock
|
4a8bd138a9 |
feat(import): load a site's data from spreadsheets
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
|
||
|
cproudlock
|
cd353b6432 |
Review safe-polish: docs accuracy, dead imports, no-emoji, geenforce robustness
From the full multi-agent review (0 high, 7 medium, 17 low findings). Applies the mechanical, low-risk items; design/policy findings left for a decision. Docs accuracy: CLAUDE.md contract 0.10.0 -> 0.11.0 and both stale Alembic head citations -> 7d24_customfield_searchable / 31 migrations; Dockerfile bundled- plugin comment fixed (drop nonexistent "equipment", add machines + measuringtools, count eleven). Style/naming (LOCKED rules): remove a CSS-escaped pushpin emoji before location search results (no-emoji policy); rename ManifestEditor shareRoot -> shareroot (variable mirrors the API field verbatim). Dead code: remove confirmed-unused imports across ~20 modules (require_role/ require_permission scaffold residue, stray db/Vendor/Model/current_user/Optional/ error_response); drop unused build_scope import + a stale GEENFORCE_API_KEY docstring clause in geenforce. Migration files left untouched. Correctness: geenforce ingest robustness - record_enforcement_report now 400s on a non-dict counts / non-list results instead of 500; _apply_app_link ignores a non-numeric appid per its docstring instead of 500. Regression tests added. Backend query.get sweep finished: auth.py refresh -> db.session.get (last one). 910 backend tests pass; pyflakes clean; naming green; frontend build green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
b130ef43f3 |
Add the dualpath-as-single-machine site toggle
Most facilities consider a Dualpath pair one physical dual-bay machine. New site setting dualpath_single_machine (default on): the machines list, dashboard counts, machines-by-type report, and floor map collapse each pair to its primary bay (lower assetnumber), with combined 2007 / 2008 labels; pagination totals stay honest. Detail pages remain per-bay and always show a dual-bay sibling banner linking the partner. Pair resolution lives in core services and joins the plugin contract surface (0.8.0 -> 0.9.0). On the WJ dataset: 31 pairs collapse, machine counts 262 -> 231, map 470 assets. Toggle verified live in both states, left on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
cproudlock
|
60641161d5 |
Second-pass review fixes: kill last hardcoded creds, wire get_services, dedup
Verification audit (re-run of the 6 skill lenses) confirmed the prior fixes hold and surfaced a few misses: Security (HIGH): - search.py _check_smart_redirect still opened a raw pymysql connection with root/rootpassword (reachable on any 9-digit SSO query). Now uses the shared env-backed employee_connection helper. - Deleted dead shopdb/core/services/employee_service.py (zero importers; carried another root/rootpassword literal). No hardcoded credentials remain in app logic; config.py dev defaults stay gated by ProductionConfig.validate. Dead hook: - get_services was implemented by the printers plugin but had no consumer (docs claimed otherwise). Added PluginManager.get_service(name) that resolves a service from enabled plugins; updated PLUGIN-HOOKS.md. Tests: - search disabled-plugin exclusion (the high-value gap): enabled plugin's hostname appears, disabled plugin's hostname drops out (searched by a hostname distinct from assetnumber so only the gated domain can match). - get_service consumer test (unknown name -> None). Simplify: - Extract the triplicated GE_LOGO_SVG + loadLogo + drawLogoOverlay into shared frontend/src/views/print/qrLogo.js (renderQrDataUrl); both QR views use it. - applications.py: lift the misplaced pagination import to the top; drop unused Computer unpacking in the 3 endpoints that only touch ComputerInstalledApp. 154 tests pass, naming/style green, app boots, QR render verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
c8b325d2e7 |
Add USB, Notifications, Network plugins and reusable EmployeeSearch component
New Plugins: - USB plugin: Device checkout/checkin with employee lookup, checkout history - Notifications plugin: Announcements with types, scheduling, shopfloor display - Network plugin: Network device management with subnets and VLANs - Equipment and Computers plugins: Asset type separation Frontend: - EmployeeSearch component: Reusable employee lookup with autocomplete - USB views: List, detail, checkout/checkin modals - Notifications views: List, form with recognition mode - Network views: Device list, detail, form - Calendar view with FullCalendar integration - Shopfloor and TV dashboard views - Reports index page - Map editor for asset positioning - Light/dark mode fixes for map tooltips Backend: - Employee search API with external lookup service - Collector API for PowerShell data collection - Reports API endpoints - Slides API for TV dashboard - Fixed AppVersion model (removed BaseModel inheritance) - Added checkout_name column to usbcheckouts table Styling: - Unified detail page styles - Improved pagination (page numbers instead of prev/next) - Dark/light mode theme improvements Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
cproudlock
|
30dd65674d |
Initial commit: Shop Database Flask Application
Flask backend with Vue 3 frontend for shop floor machine management. Includes database schema export for MySQL shopdb_flask database. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |