96a4a38c8c
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed.
WJ classic-ASP import loader (reference implementation)
This is site-specific reference glue, not product code. It maps West
Jefferson's classic-ASP shopdb / cmmc_usb / wjf_employees schema onto the
maintained, schema-agnostic import contract in the IMPORT-API wiki page.
Every adopting site has its own source database. Nobody else runs this loader. Instead, copy the pattern:
- Point the harness at your source DB(s) (edit
harness.Source). - Write per-entity stages that read your tables and POST to the same
the IMPORT-API wiki page endpoints with
Authorization: Bearer <admin PAT>andX-Import-Mode: true. - Persist legacy-id -> new-id crosswalks (see
harness.IdMap) so later stages resolve foreign keys and a crashed run resumes.
The import API is the stable contract; loaders are per-site. The mapping itself is the bulk of the work: enumerate the source tables against this contract and write one stage per entity (this loader is the worked example).
Running (against a THROWAWAY import database)
DATABASE_URL='mysql+pymysql://root:PW@127.0.0.1:3306/shopdb_flask_import?charset=utf8mb4' \
venv/bin/python -m scripts.site_imports.wjf.run --stages reference,employees
Prereqs: a fresh target DB built with flask db upgrade + flask plugin upgrade-all + flask seed permissions/settings/reference-data, and the three
source dumps loaded into scratch DBs (shopdb_src, cmmc_usb_src,
wjf_employees_src). See scratchpad/IMPORT-PLAN.md for the full mapping,
resolved decisions, and remaining stages.
Status
All 17 stages built: reference, employees, catalog, assets (the hub +
machineid->assetid crosswalk), locations, printers, metrology,
communications, applications (+ installs), warranties, notifications,
knowledgebase, relationships, defaultprinters, subnets, usb, verify.
The counts below are from the 2026-07-13 run and are STALE. Two routing
changes landed after them - the isactive=1 filter and the metrology routing
rework - so the asset mix has moved substantially. They are kept only as a
shape, not as a checklist: re-measure from the next full run before quoting
them anywhere, and do not verify a fresh import against them.
Last full run (2026-07-13, superseded): 983 assets (computer 663, machine 76, network 58, measuring-tool 136, printer 50), 24 locations, 415 employees, 850 installs, 461 primary IPs, 464 warranties, 261 notifications, 341 KB, 93 relationships, 37 subnets, 18 USB devices + 232 events.
Known gaps, none of them blocking a scratch run:
warranties,notificationsandknowledgebasePOST without a duplicate check on the API side, so resuming ACROSS one of those stages doubles its rows. Resuming at or after them is safe. Re-run into a clean target when in doubt.- There is no
--dry-runand no guard on the target: the harness mints an admin PAT and setssetup_completein whateverDATABASE_URLpoints at. Check the target before running. - The
verifystage counts targetassetsagainst sourcemachines, which is apples to oranges - the target also holds printers and synthesized metrology tools.
Known follow-ups: general asset locationid is null (no source column outside
printers); the ~146 duplicate machinenumbers are first-wins-skipped by decision.