dev
23 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
cproudlock
|
377cc69264 |
Make a partly-undone import retryable, and refuse a plan it cannot deliver
Five findings from the review of the WINC import, plus the rendering each fix needed to be usable. An undo that hit a conflict wrote status 'undone' anyway, and undo_batch returns immediately for an undone batch. So the one entry that needed a retry was the only one that could never get it: the human resolves the conflict, clicks Undo, and gets a no-op. A conflicted undo is now 'partially-undone', undoneat is stamped only when the batch is genuinely finished, and the batches table offers "Retry undo" for that status. The column was String(16) and the new value is exactly 16 characters, so migration 7d38 widens it to 32 rather than rely on an exact fit. _UNDO_REFERENCES listed only assets, so a business unit this batch created was hard-deleted even when something else still pointed at it. Two of the three missing referrers are real FKs (dashboarddefaults.businessunitid, businessunits.parentid): MySQL raises and the whole undo rolls back, so the batch cannot be undone at all. The third, notifications.businessunitid, has no FK, so the delete succeeded and left a notification pointing at an id that no longer exists - silently, on both engines. All four are listed and the guard is tested through the route. The reference guard also queried tables that need not exist. A lean site drops the tables of plugins it does not install (ADR-014), and notifications is plugin-owned, so on such a site the query raised 1146, rolled the undo back and left the batch permanently un-undoable. Absent tables are skipped; a table that does not exist cannot hold a reference. _PACKET_GROUPS reports on location, but apply_plan only ever creates business units - it sets locationid only when the location already resolved. A plan needing a new location asked the admin to acknowledge it, got the tick, wrote nothing, and reported nothing. build_packet now takes the groups the writer can actually create; anything else needed is 'uncreatable' and blocks at plan time, where the admin can act on it, rather than dying at commit on a message about acknowledging something the screen had no control for. The writer refuses the same case independently. Both the console packet and the import page name the blocking groups: setting blocking without printing the reason would leave the banner pointing at nothing. The web undo route caught ValueError only, so refusing a batch belonging to another importer surfaced as a 500. That refusal is new too: the journal is a general primitive and undo_batch does not filter on kind, so any other importer's batches were reversible through this endpoint, carrying WINC's references map - which lists none of that importer's tables, making every reference check pass and every delete unconditional. Nine tests. The two guards were verified by planting violations: dropping dashboarddefaults from the references map, and passing creatable=None, each fail the test written for them. Scope: product |
||
|
cproudlock
|
28269e429a |
Import the WINC batches CLI and its README
One commit from the sister site, purely additive: a read-only `batches` command
so an operator working from the shell can find a batchid to undo, and a README
covering the four WINC commands. Two new files under scripts/site_imports/winc/,
nothing existing touched.
Verified rather than assumed: it changes none of the files fixed in
|
||
|
cproudlock
|
13be0ef90f |
Import the wilm agent store and WINC adoption work, with four fixes
Eleven commits from the sister site: a reversible Maximo/WINC machine-adoption import, a versioned agent binary store with upload and rollback, remote-BuildKit CI, tray version reporting, and UI fixes. Applied as a patch again - the published mirror rewrites hashes, so the branches have unrelated histories and git refuses to merge them. Reviewed by three agents before merging. Four things fixed here. THE PUBLISH WORKFLOW WAS DEAD. The new remote-BuildKit step in publish-images.yml sits at column 0 instead of indented under steps:, so the whole file is invalid YAML and GitHub would refuse it. Their entire GHCR distribution channel - which Start-ShopDBDev.ps1 and the compose files consume - was broken at the source and clearly never run. Indented; all five workflows now parse. THE IMPORT'S APPROVAL CHECK COULD NOT FAIL. commit() built the plan and the approval packet from the same upload and then asked the writer whether they matched, so packet_matches_plan compared a hash against itself. The documented contract - "a human approves a specific, frozen plan, and the writer can prove nothing drifted" - did not exist in the web path: an admin could preview file A and commit file B, gated only by the needs-create group NAMES they ticked. The preview's planhash is now required at commit and checked against a freshly rebuilt plan. Their tests committed without previewing, which is what let this through; they go through the real flow now, and three new tests pin it - all confirmed to fail with the check stubbed out. endpoints.agentpackage QUIETLY GREW. It used to mean "wrap the staged exe in a ZIP"; with the upload routes it also means "choose the executable every future package installs and runs as SYSTEM". Before this, staging needed filesystem access to the server. The permission description still said only "generate and download a package", so an admin granting it could not see what they were granting. Reworded. NOTHING RECORDED WHAT WAS STAGED, OR BY WHOM. The manifest held version, size and timestamp; re-uploading a version replaces the bytes in place under the same label. A sha256 of the stored file is now recorded, and the upload is audit-logged with it - after an incident there was previously no answer to who put that binary on the fleet. Also documented the nine new routes in docs/api-inventory.json (that gate cannot run on the mirror, so it falls to us at promotion time again), regenerated the spec and the project map, and kept our own docs where they conflicted: docs/ is scrubbed from the mirror, so their DEV-BUILD-TEST.md still had the dev-be channel mapping we corrected, and their PLUGIN-HOOKS.md was a new 45-line file rather than our 737-line canonical one. Their journal section is appended to ours. Contract version comes across as 0.30.0 for a genuinely additive surface. Suite 2369, frontend 324, gate green. Scope: product |
||
|
cproudlock
|
96a4a38c8c |
Make the legacy import safe to resume, and aimable without editing it
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed. |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
2d09fa3201 |
Collect what bays actually have, separately from what they are told to have
ShopDB knew what a bay SHOULD have and nothing about what it DOES. Adding the observed half makes a rollout a review instead of a typing exercise: the floor reports itself in, you look, and you adopt. The collection uses the mechanism that already exists rather than a new one. POST /api/collector/printers dispatches to the printers plugin's apply_collector_payload, the same ADR-006 hook the computers and backups plugins implement. New client script, new plugin-owned table, no new transport and no new credential. OBSERVED AND ASSIGNED STAY APART, and that is the point rather than a detail. A collector report can never write an assignment row: _reconcile_edges is the only function that writes usesprinter/defaultprinter, it has two call sites, and both are authenticated routes a human calls. If a drifted bay's own state were allowed to become what it is told to install, every configuration error would become permanent the next time that PC checked in. Seeding an assignment from observed state is explicit - POST /assignments/seed-from-observed - because a rollout adopts many machines at once. It routes through the same _reconcile_edges as the editor, so there is one write path with two doors, and a queue matching no known printer is REFUSED rather than guessed into an assignment. That last rule is the lesson from the measuring tools: adopting on a weak key produced 43 duplicate instruments. Two fixes on top of what the agents built. The replace deleted a host's previous rows by exact case-folded name while the read path treats a short name and its FQDN as one machine, so a PC that changed spelling appeared to hold every queue twice - which reads as drift that is not there. And the client sent 'reportedat' where the declared schema said 'observedat'. Also here: the legacy loader now imports machines.printerid, the classic system's record of each machine's default printer, which it silently dropped - the production import would have lost every one. And Set-ShopdbPrinters.ps1 finally registers the per-user logon task, staging Apply-ShopdbDefaultPrinter.ps1 to C:\ProgramData first because the share it lives on is mounted only during the enforcement cycle and the task runs at logon when it is gone. VALIDATED ON WINDOWS 11 (build 26200), not just on Linux pwsh, which parses these scripts happily and executes none of the spooler branches. The reporter: posts a correct payload with the X-API-Key header; resolves BaseUrl and CollectorKey from HKLM when given no arguments; suppresses the virtual queues by port; resolves port addresses; and reads the CONSOLE USER's default out of HKU rather than SYSTEM's own, which is a different and usually wrong answer. Two results matter more than the rest. With the spooler stopped, both the cmdlet and the CIM path fail and the script posts NOTHING - verified against a capture server that recorded zero requests, where an empty list would instead have erased that host's observed rows and read as a bay that lost its printers. A genuinely empty host still posts [], because that is a real and different fact. The logon task registers as the Users group at Limited, and falls back to the well-known SID S-1-5-32-545 when the group name will not resolve, as it will not on localised Windows. It was then run with the source directory RENAMED AWAY, to stand in for the share being unmounted, and it still moved the user's default - which is the whole reason the script is staged to C:\ProgramData rather than run from where it lives. The guarantees against damage were re-checked rather than assumed: an empty assignment changes nothing, an unreachable server changes nothing, -WhatIfOnly leaves no queue, no task, no staged file and no registry value behind, and a drifted queue is repointed IN PLACE with Set-Printer so whoever has it as their default keeps it. Not covered by any of this: the driver-staging path, which needs a real vendor package rather than the class drivers a VM ships with. |
||
|
cproudlock
|
f34b9ca710 |
Carry the level everywhere a position is drawn, and gate it per occurrence
The hover mini-map said "This asset has a position (2835, 1410) but no level" for every asset in the product. When 0.11.0 gave LocationMapTooltip a levelid prop, NONE of its seven call sites were taught to pass one - printer, machine and PC detail pages, the toner report, enforcement reports, the warranty chip and the dashboard cards - so the component correctly reported a missing level and the preview never drew. Two payloads behind those views also emitted mapx/mapy with no level: the toner report and the enforcement report. The map PDF export had the ORIGINAL bug still in it: it plotted every filtered asset onto the sheet, so exporting the ground floor printed second-floor markers on it. Worse than on screen, because nobody can correct a sheet once it has been printed and carried onto the floor. It now exports only the level being viewed. The legacy import loader sent mapleft/maptop with no level at three call sites. That loader is the one still to run against production, and every marker it created would have been undrawable. It now resolves the site's default level - the legacy schema predates levels and has one floor plan, so that is what its coordinates mean. THE GATE MISSED ALL OF THIS because it asked whether a FILE mentions 'levelid', not whether each position does: one module emitted 'mapx' six times and 'levelid' once and passed. It now checks per occurrence, covers scripts/ as well as shopdb/ and plugins/, and fails any Vue file that binds tooltip coordinates without :levelid. Both new rules were confirmed to fail the build against planted violations before being relied on. Printer QR labels: the asset number is no longer printed. A label now reads name (8201-HPLaserJetPro), QR, FQDN, then IP. The name falls back to the assetnumber because that is where sites actually keep it - every printer here has an empty name field, so preferring the Windows queue name alone would have printed a blank line on every label. |
||
|
cproudlock
|
05be4c4489 |
docs: fix the examples that fail if you paste them
Six procedures that could not be followed as written.
Eighty-nine curl examples single-quoted `Authorization: Bearer $TOKEN`, so the
shell never expanded it and the server answered 422 "Not enough segments". Nine
more did the same with X-API-Key. The other 129 examples in the same file
already used double quotes, so this was drift rather than a convention, and the
spec regenerated from it carried the fault onward.
The GE-Enforce report example put a `//` comment inside a JSON body. The server
parses with silent=True, so it saw `{}` and answered "hostname is required"
about a body that plainly has one - the worst kind of error message, one that
sends the reader to the wrong field entirely.
The IIS install ran `flask db upgrade` and a per-plugin install loop but never
`flask plugin upgrade-all`, leaving every plugin's own chain unapplied. That is
precisely the 1054 "Unknown column" a deploy then hits somewhere else, days
later, on the page that uses the new column.
The pilot runbook looped `flask plugin enable` over plugins that were not yet
installed; enable refuses those, so on a fresh database it exited 1 on every
iteration and enabled nothing. ADR-013 had already recorded that defect.
`apply-profile` installs and enables in dependency order, which is what the step
was reaching for.
DEPLOY-WINDOWS-IIS named a profile file that does not exist; the shipped ones do.
And PLUGIN-EXTERNAL-REPO never mentioned PLUGIN_TABLE_OWNERS, while the
migration engine raises for any plugin missing from it - so the guide's own
step 5 fails for any external plugin that owns a table. That the registry lives
in the framework repo is deliberate, so the guide now says so, and says what it
costs: a table-owning external plugin is a two-repository change, and a plugin
that owns no tables avoids it entirely.
|
||
|
cproudlock
|
4995456136 |
docs: take one site's name, hosts and paths off the public wiki
The publishability gate caught internal tooling names and developer paths but nothing site-specific, so roughly sixty leaks reached the wiki: the site name in ten documents, real fleet hostnames in the collector and GE-Enforce examples, an internal database name through the whole import guide, imaging-share paths, and a maintainer's username as the Deciders line of every ADR and inside a generated curl example. None of it is a security matter on an air-gapped fleet. It matters because these pages are read by engineers at other plants, and a document that names one site throughout reads as that site's notes rather than a product's documentation - which is exactly what it then gets treated as. Examples now use neutral hostnames, the site is "the reference site" where the distinction carries meaning, and ADRs are decided by "ShopDB maintainers". The gate carries all of these patterns, so the next one fails a build. Two documents leave docs/ because they were never written for an outside reader. PROJECT-REVIEW.md is an internal health memo pinned to a commit from July, whose headline finding (an untracked playbook) has since been fixed - it is history, and git holds it. PILOT-DEPLOY.md is one site's own cutover runbook, complete with a "re-measure before publishing" placeholder; it moves next to the loader it belongs to, in scripts/site_imports/wjf/. ADR-015 is AMENDED rather than rewritten. Its enforcement section still said report-only and its backlog still listed hardcodes that are now cleared, which left the record contradicting itself. The amendment says what changed and why the report-only period ended; the original text stays, because what the decision looked like when it was taken is the part worth keeping. Also corrects llms.txt's response envelope, which had errors at the top level and pagination at meta.total. Both are nested one deeper, so anything written against that description read undefined on every error it tried to handle. |
||
|
cproudlock
|
3ba09ac9f3 |
Fix bit(1) import coercion and subpath login redirect; add GitHub export script
Loader: bool() on pymysql bit(1) bytes is always true - isinstallable and isshopfloor imported as 1 for every row; route through _truthy_bit. The employee source DB is now optional (shopdb-only imports). Frontend: under a subpath mount the 401 interceptor stored the browser path (mount base included) as the login redirect and the router applied its base again (/ops/ops). New stripBase() keeps redirects base-free. tools/export-github.sh automates the publication flow: prune + scrub + commit into ~/projects/shopdb-flask-pub and emit a transfer bundle. |
||
|
cproudlock
|
7d7862f4b5 |
Neutral wording in import docs; portable tool output paths
Import-surface docs and docstrings describe the automation as a migration script; status-doc references in CHANGELOG/ADR-009/ROADMAP point at repo files. Screenshot/verify tools write to /tmp/shopdb-shots (created on import) instead of a machine-specific directory. |
||
|
cproudlock
|
9e2544a687 |
wjf loader: import active machines only (isactive=1)
Classic ASP keeps retired machines in the machines table as history (isactive=0); every other stage already filters isactive=1, but the assets hub, metrology, locations and the verify count read machines unfiltered, so ~240 retired units (incl. all G-prefix hostnames and 61 dead metrology PCs that each synthesized a phantom measuring tool) landed as live assets. Add the isactive=1 filter to those four queries. Downstream stages resolve via the id crosswalk, so warranties /comms/relationships/installs for retired machines now drop automatically. |
||
|
cproudlock
|
4a74a1a405 |
WJ import loader: mark setup_complete (a full import is the setup)
A freshly-imported DB left setup_complete unset, so the admin was bounced into the first-run wizard even though the instance is fully populated. The harness now sets setup_complete=true (it already mints the admin), so an imported instance goes straight to the app. |
||
|
cproudlock
|
9c909af66d |
WJ import loader: metrology PCs are computers that control a synthesized tool
Routing by pctype wrongly swept ~105 metrology PCs (CMM/Genspect/Keyence/Wax) into measuringtools - but a PC that drives an instrument is still a computer; the physical CMM/gauge is the tool. Dropped the pctype override: those PCs now import as computers (measuringtools drops to the 48 real instrument rows). Classic has no separate tool row for a metrology PC, so they'd be orphaned. New metrology stage synthesizes a measuring-tool asset per metrology PC (typed by its pctype: CMM / Form Tracer / Vision System / Genspect) and a Controls relationship PC -> tool, mirroring what the runtime collector does. Result: computers 663->751, measuringtools = 48 real + 88 synthesized (each linked to its controlling PC), 88 new Controls relationships, no orphans. |
||
|
cproudlock
|
49d311b56e |
WJ import loader: map application link + documentation path
The applications stage dropped applicationlink and documentationpath, so the app detail's "Launch Application" and "Documentation" links were always empty. Map both from the classic applications table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
c4690da262 |
WJ import loader: resolve notification employee SSOs to names
Notifications imported with only employeesso, leaving employeename null - the model displays "employeename or employeesso", so recognition/training cards showed a bare SSO instead of a name. Build an SSO -> "First Last" map from the employee source and populate employeename (comma-separated SSOs -> joined names). SSOs not in the directory (former/non-WJF) stay null and fall back to the SSO, as before. Verified on the import DB: 261 notifications re-imported, names resolved (Brandon Saltz, Jon Kolkmann, ...). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
009ac9f9ef |
Import docs: adoption playbook + superseded-mappers note + loader status
Follow-up to the mapper retirement (the new docs missed the prior commit's staging). Adds docs/IMPORT-ADOPTION.md (two-layer import story + stage/crosswalk guidance), scripts/migration/README.md (dir superseded, points at the API + loader), and updates the WJ loader README to complete status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
af9afc15c5 |
WJ import loader: printers stage + Controlled-By relationship reversal
printers: printers come from the printers TABLE (not the machines hub), so a
dedicated stage - assetnumber synthesized PRN-{printerid}, IP folded via the
create route, host machineid resolved to a location when it is a LocationOnly
row. Skips inactive. 50 of 56 imported.
relationships: a "Controlled By" edge now flips to the forward Controls
direction (source/target swapped, mapped to the Controls type) instead of
importing a redundant inverse type.
Final fresh full run: 983 assets (933 machines-hub + 50 printers), zero endpoint
errors, all 15 stages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
cproudlock
|
99ad6c0ebb |
WJ import loader: tail stages (locations, relationships, subnets, usb, verify)
Completes the loader end to end. Verified on a fresh scratch target, zero endpoint errors: - locations: the 24 islocationonly rows -> core Locations (crosswalk machineid -> locationid). - relationships: 93 active edges imported (206 of 299 dropped because an endpoint became a Location / was skipped / dedup-lost); types folded onto the seeded canonical set; dedup on (source,target,type). - subnets: 37 (full CIDR reconstructed as INET_NTOA(ipstart)+suffix; VLANs lookup-or-create by number; 3 duplicate CIDRs first-wins-skipped). - usb: 18 cmmc devices + 232 check-in/out events, paired with per-device open state so unpaired log rows do not 400. Needs the usb plugin enabled + usb directory mode selfhosted. - verify: source-vs-target row-count audit (assets 1167->933 by the skip rules, applications 121=121, employees 415=415, KB 342->341). Full pipeline default runs all 14 stages in order. NOTE: the import target needs every bundled plugin enabled (usb ships disabled in this dev registry - enable it before importing). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
40a89b360a |
WJ import loader: route LocationOnly by the islocationonly bit, not machinetypeid
Of the 158 machinetypeid=1 rows, only 24 carry the islocationonly bit (real named areas: DT Office, IT Closet, Materials, ...). The other 134 are active, modelled shop machines just left untyped - routing all 158 to Locations dropped those 134 real assets. Route on the bit instead; the 134 untyped rows import as machines with a null subtype (machinetypeid=1 is not a real machine subtype, so catalog skips seeding one). Also process asset routes in richness order (computer > measuringtool > network > machine) so on a duplicate machinenumber the PC - which carries installs + IP a bare untyped machine does not - wins first-come. Result on the scratch target: 933 assets (computer 663, machine 76, network 58, measuringtool 136), 24 locations (was mis-routing 158), installs 850 (was 653 - PCs no longer lose their numbers to bare machines), warranties 464, comms 461. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
6d79e469fa |
WJ import loader: dependent-entity stages (comms, apps/installs, warranty, notif, KB)
All consume the machineid->assetid crosswalk from the assets hub. Verified against the scratch target, zero endpoint errors: - communications: 435 primary IPs folded onto assets. No bulk endpoint exists, so this is the plan's documented direct-ORM gap (reads the source communications table where comstypeid=1 AND isprimary=1, not machines.ipaddress1 which is empty). - applications: supportteams 45, applications 121 (colliding names dedup via the unique-appname 409-resolve), appversions 47, installs 653 (machineid -> assetid -> computerid; only computer assets take installs). - warranties: 424 linked, vendor hardcoded Dell (source has none). - notifications: types 6, notifications 261 (2099 sentinel endtime clamped). - knowledgebase: 341 (appid resolved through the applications name map). Inactive rows skipped everywhere per the decisions. Remaining loader stages: locations (the 158 LocationOnly rows), relationships (301 active edges), subnets/VLANs, usb (cmmc pairing), verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
6c4bd20e01 |
WJ import loader: catalog + assets-hub stages (the machineid->assetid crosswalk)
Adds the keystone stages to the reference loader. catalog: seeds modeltypes (from classic machinetypes + category), the per-plugin asset subtypes routed by machinetype (machines/network/measuringtool types), computer subtypes (from pctype), the 5-row controllertypes vendor/model split, and the models catalog - each with a persisted legacy->new crosswalk. Verified: modeltypes 31, computertypes 12, models 118. assets (the hub): fans classic machines out to the right endpoint by machinetypeid (+ the pctype metrology override), applying the resolved decisions - assetnumber = machinenumber else hostname, skip 9999, skip duplicate machinenumbers, LocationOnly/printer/USB routed out. Persists the machineid -> assetid crosswalk every downstream stage needs. Verified against a fresh scratch target: 884 assets (computer 623, machine 68, network 58, measuringtool 135), zero endpoint errors, idempotent re-run (stays 884). Skips: location 158, dup 69, other 53, 9999 1. Harness now runs each plugin's idempotent on_install so the AssetType rows exist (a DB built with plugin upgrade-all instead of a fresh install lacks them, and the create routes 500 without them). 409-resolve lookups page through per_page. Remaining stages: communications (primary IP fold - source is the communications table, not machines.ipaddress1), applications/installs, warranties, notifications, KB, subnets/VLANs, usb, verify. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
b11a6f26d8 |
WJ legacy-import reference loader: harness + reference/employees stages
Layer 2 of the import design (see the memory + scratchpad/IMPORT-PLAN.md): a SITE-SPECIFIC reference loader that maps WJ's classic-ASP schema onto the maintained, schema-agnostic IMPORT-API contract. Other sites copy the pattern against their own source DB; nobody runs this loader as-is. Harness (scripts/site_imports/wjf/harness.py): builds the app against the current DATABASE_URL (point it at a throwaway import DB), mints an unscoped admin PAT in-process, and drives the real import endpoints through the app test client with Authorization: Bearer + X-Import-Mode - exercising the same routes/authz/validation an HTTP client would, no running server needed. Read-only pymysql access to the three scratch source DBs; legacy-id -> new-id crosswalks persist to JSON so a crashed run resumes and later stages resolve FKs. Stages implemented + verified idempotent against a fresh scratch target (shopdb_flask_import): reference (vendors 46, businessunits 13, operatingsystems 11) and employees (directory 415, re-run updated-not-duplicated). Remaining stages (models, applications, assets hub + crosswalk, dependents, network, usb, verify) are stubbed with the same shape; README documents the adoption playbook. idmap.json is generated state (gitignored). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |