96a4a38c8cf7415ffc6ede22c2ae0b7cd715e8f3
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
96a4a38c8c |
Make the legacy import safe to resume, and aimable without editing it
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed. |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
f34b9ca710 |
Carry the level everywhere a position is drawn, and gate it per occurrence
The hover mini-map said "This asset has a position (2835, 1410) but no level" for every asset in the product. When 0.11.0 gave LocationMapTooltip a levelid prop, NONE of its seven call sites were taught to pass one - printer, machine and PC detail pages, the toner report, enforcement reports, the warranty chip and the dashboard cards - so the component correctly reported a missing level and the preview never drew. Two payloads behind those views also emitted mapx/mapy with no level: the toner report and the enforcement report. The map PDF export had the ORIGINAL bug still in it: it plotted every filtered asset onto the sheet, so exporting the ground floor printed second-floor markers on it. Worse than on screen, because nobody can correct a sheet once it has been printed and carried onto the floor. It now exports only the level being viewed. The legacy import loader sent mapleft/maptop with no level at three call sites. That loader is the one still to run against production, and every marker it created would have been undrawable. It now resolves the site's default level - the legacy schema predates levels and has one floor plan, so that is what its coordinates mean. THE GATE MISSED ALL OF THIS because it asked whether a FILE mentions 'levelid', not whether each position does: one module emitted 'mapx' six times and 'levelid' once and passed. It now checks per occurrence, covers scripts/ as well as shopdb/ and plugins/, and fails any Vue file that binds tooltip coordinates without :levelid. Both new rules were confirmed to fail the build against planted violations before being relied on. Printer QR labels: the asset number is no longer printed. A label now reads name (8201-HPLaserJetPro), QR, FQDN, then IP. The name falls back to the assetnumber because that is where sites actually keep it - every printer here has an empty name field, so preferring the Windows queue name alone would have printed a blank line on every label. |
||
|
cproudlock
|
4a74a1a405 |
WJ import loader: mark setup_complete (a full import is the setup)
A freshly-imported DB left setup_complete unset, so the admin was bounced into the first-run wizard even though the instance is fully populated. The harness now sets setup_complete=true (it already mints the admin), so an imported instance goes straight to the app. |
||
|
cproudlock
|
6c4bd20e01 |
WJ import loader: catalog + assets-hub stages (the machineid->assetid crosswalk)
Adds the keystone stages to the reference loader. catalog: seeds modeltypes (from classic machinetypes + category), the per-plugin asset subtypes routed by machinetype (machines/network/measuringtool types), computer subtypes (from pctype), the 5-row controllertypes vendor/model split, and the models catalog - each with a persisted legacy->new crosswalk. Verified: modeltypes 31, computertypes 12, models 118. assets (the hub): fans classic machines out to the right endpoint by machinetypeid (+ the pctype metrology override), applying the resolved decisions - assetnumber = machinenumber else hostname, skip 9999, skip duplicate machinenumbers, LocationOnly/printer/USB routed out. Persists the machineid -> assetid crosswalk every downstream stage needs. Verified against a fresh scratch target: 884 assets (computer 623, machine 68, network 58, measuringtool 135), zero endpoint errors, idempotent re-run (stays 884). Skips: location 158, dup 69, other 53, 9999 1. Harness now runs each plugin's idempotent on_install so the AssetType rows exist (a DB built with plugin upgrade-all instead of a fresh install lacks them, and the create routes 500 without them). 409-resolve lookups page through per_page. Remaining stages: communications (primary IP fold - source is the communications table, not machines.ipaddress1), applications/installs, warranties, notifications, KB, subnets/VLANs, usb, verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
b11a6f26d8 |
WJ legacy-import reference loader: harness + reference/employees stages
Layer 2 of the import design (see the memory + scratchpad/IMPORT-PLAN.md): a SITE-SPECIFIC reference loader that maps WJ's classic-ASP schema onto the maintained, schema-agnostic IMPORT-API contract. Other sites copy the pattern against their own source DB; nobody runs this loader as-is. Harness (scripts/site_imports/wjf/harness.py): builds the app against the current DATABASE_URL (point it at a throwaway import DB), mints an unscoped admin PAT in-process, and drives the real import endpoints through the app test client with Authorization: Bearer + X-Import-Mode - exercising the same routes/authz/validation an HTTP client would, no running server needed. Read-only pymysql access to the three scratch source DBs; legacy-id -> new-id crosswalks persist to JSON so a crashed run resumes and later stages resolve FKs. Stages implemented + verified idempotent against a fresh scratch target (shopdb_flask_import): reference (vendors 46, businessunits 13, operatingsystems 11) and employees (directory 415, re-run updated-not-duplicated). Remaining stages (models, applications, assets hub + crosswalk, dependents, network, usb, verify) are stubbed with the same shape; README documents the adoption playbook. idmap.json is generated state (gitignored). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |