534f179ee6
main was 16 days and 259 commits behind the line the product is actually built on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It was neither current nor curated - just stale. This brings it forward and, at the same time, makes it mean something: main is what a receiving site installs, and nothing else. WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty - plus applications, which is manifest-less core and always ships. The only manifest dependency inside that set is endpoints on computers, and both are in it, so the closure is self-contained. WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb stay on the site line. So do the Windows installer tree, the VM rehearsal helpers, the agent tooling config, one site's data loader, the internal audit and findings records, and the planner spreadsheet. 1,135 tracked files against 1,438. A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped: deploy/site-profile-universal.json still named all seventeen plugins. It is the file docs/DEPLOY.md tells a site to provision from, so a fresh install would have failed at `flask plugin apply-profile` with PluginError - the first thing a receiving site does, broken by the branch meant to serve them. It now names the twelve that exist, and its comment says the two must agree. agent/ was never refreshed. Whoever populated this branch copied shopdb/, plugins/, frontend/ and tests/ from the site line and left the agent tree at its September state: 22 C# and PowerShell sources missing, including AgentCredentials, EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the site line had deleted still present. The old tree is internally consistent, so it compiles, and nothing in CI builds it - every gate was green over a server that speaks enrollment and per-PC credentials to a bundled agent source that cannot. Resynced whole. DECISIONS THIS ENCODES, rather than defers: - The plugin-authoring lab builds printedparts, whose manifest depends on employees, so it could not stay. The lab remains on the site line and main ships no build-along tutorial. That is a real loss for an adopting developer and is recorded here rather than discovered later. - Recognition, Training and Recertification notifications are no longer seeded. Each splits one notification into a card per employee and renders a photo, which needs a plugin this branch does not bundle. Seeding them would offer an author a type that cannot resolve anybody. A site that installs employees gets them from that plugin's own seed. - Windows/IIS is documented as the legacy path throughout, and the README leads with Docker, because that is what a receiving site gets. The API description is regenerated for THIS branch - 347 paths and 483 operations, down from 415 and 573 - so the published contract describes what is here rather than what the site line serves. docs/CONFIG.md lost the two measuring-tool label keys the code no longer declares, which two lanes had decided opposite ways. Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run rather than skip, and every index line count is derived from the file it names.
4.4 KiB
Building a per-site import loader
Four import pages. Shaping the loader: this page. The HTTP wire contract it drives: IMPORT-API, which this page assumes you have read and does not restate. Spreadsheet and no developer: CSV-IMPORT. Source is the classic-ASP shopdb: IMPORT-LEGACY-ASP.
Two layers, and why the split holds
Every adopting site has its own source database, and it will not match another site's schema. So the import is split:
- The import API is the stable contract. Whatever your source looks like, you create records through the same documented REST endpoints. This layer is the product; it is schema-agnostic and documented once, in IMPORT-API.
- A per-site loader is thin glue. It reads your source database and POSTs to those endpoints. Nobody runs another site's loader - you copy the pattern.
Keeping the two apart is what stops one site's judgement calls from becoming everyone's contract.
The shape of a loader
The reference implementation lives under scripts/site_imports/<site>/. Read it
alongside this page.
harness.py- builds the app against the targetDATABASE_URL, mints an unscoped admin PAT in-process, and drives the real endpoints through the app test client withAuthorization: Bearer <pat>+X-Import-Mode: true. This exercises the same routes, authorization and validation an HTTP client would, with no running server needed. It also holds read-only access to the source DB and a JSONIdMapof legacy-id to new-id crosswalks.run.py- orderedstage_*functions. Each reads a slice of the source, POSTs it, and records the crosswalk later stages resolve foreign keys against.
Post-import fixups that re-point existing assets (for example
scripts/reclassify_servers_to_network.py, which moved servers imported as PCs
to network devices in place) belong in the site loader's verify stage, never in
the API layer.
The crosswalk is the keystone
Reference and lookup tables first so foreign keys resolve, then the entity hub, then dependents, then links:
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
-> dependents (installs, warranties, notifications, ...) -> relationships
Capture every legacy id to new id as you create rows, and resolve foreign keys through it in later stages. New autoincrement ids will not match the source's. The full ordering rule is section 4 of IMPORT-API.md.
Producing the mapping
You do not have to hand-derive the source-to-target mapping. Point an agent-assisted workflow at a source database plus the API contract and it emits a per-table mapping (source columns to endpoint fields, transforms, what is importable and what is out of scope) plus a loader skeleton. That is the repeatable onboarding path, and IMPORT-LEGACY-ASP.md is a finished example of the output.
Running it, against a THROWAWAY import database
- Build a fresh target:
flask db upgrade,flask plugin upgrade-all, thenflask seed permissions/settings/reference-data. Enable every bundled plugin you need - some ship disabled, and a plugin's routes only register when it is enabled at app start. - Load your source dump into a scratch database the loader can read.
- Run the loader stages in order, dry-running and spot-checking as you go.
- Verify: row-count and foreign-key-resolution audit against the source, then a UI spot-check - log in and eyeball the lists, the map, a detail page.
- Only then point a real instance at the imported database.
What a mature loader ends up handling
From the reference implementation, as a checklist of what to expect:
- Fanning one source "machine" table out to several asset types (computer/machine/network/measuring-tool) by a routing rule, with the duplicate, placeholder and skip decisions applied.
- Synthesizing a natural key where the source lacks one (printers to
PRN-{id}). - Folding a primary IP onto an asset, pairing a check-in/out event log into checkouts, deduping colliding names, reversing an inverse relationship type.
- The handful of narrow gaps the API cannot cover (there is no bulk communications endpoint, for instance) handled as documented direct-ORM writes.