534f179ee6
main was 16 days and 259 commits behind the line the product is actually built on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It was neither current nor curated - just stale. This brings it forward and, at the same time, makes it mean something: main is what a receiving site installs, and nothing else. WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty - plus applications, which is manifest-less core and always ships. The only manifest dependency inside that set is endpoints on computers, and both are in it, so the closure is self-contained. WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb stay on the site line. So do the Windows installer tree, the VM rehearsal helpers, the agent tooling config, one site's data loader, the internal audit and findings records, and the planner spreadsheet. 1,135 tracked files against 1,438. A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped: deploy/site-profile-universal.json still named all seventeen plugins. It is the file docs/DEPLOY.md tells a site to provision from, so a fresh install would have failed at `flask plugin apply-profile` with PluginError - the first thing a receiving site does, broken by the branch meant to serve them. It now names the twelve that exist, and its comment says the two must agree. agent/ was never refreshed. Whoever populated this branch copied shopdb/, plugins/, frontend/ and tests/ from the site line and left the agent tree at its September state: 22 C# and PowerShell sources missing, including AgentCredentials, EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the site line had deleted still present. The old tree is internally consistent, so it compiles, and nothing in CI builds it - every gate was green over a server that speaks enrollment and per-PC credentials to a bundled agent source that cannot. Resynced whole. DECISIONS THIS ENCODES, rather than defers: - The plugin-authoring lab builds printedparts, whose manifest depends on employees, so it could not stay. The lab remains on the site line and main ships no build-along tutorial. That is a real loss for an adopting developer and is recorded here rather than discovered later. - Recognition, Training and Recertification notifications are no longer seeded. Each splits one notification into a card per employee and renders a photo, which needs a plugin this branch does not bundle. Seeding them would offer an author a type that cannot resolve anybody. A site that installs employees gets them from that plugin's own seed. - Windows/IIS is documented as the legacy path throughout, and the README leads with Docker, because that is what a receiving site gets. The API description is regenerated for THIS branch - 347 paths and 483 operations, down from 415 and 573 - so the published contract describes what is here rather than what the site line serves. docs/CONFIG.md lost the two measuring-tool label keys the code no longer declares, which two lanes had decided opposite ways. Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run rather than skip, and every index line count is derived from the file it names.
97 lines
4.4 KiB
Markdown
97 lines
4.4 KiB
Markdown
<!-- docmeta
|
|
purpose: explanation
|
|
audience: adopter, developer
|
|
status: current
|
|
summary: How to build the per-site loader that reads your source database and drives the import API.
|
|
loadwith: IMPORT-API.md
|
|
-->
|
|
|
|
# Building a per-site import loader
|
|
|
|
**Four import pages.** Shaping the loader: this page. The HTTP wire contract it
|
|
drives: [IMPORT-API](IMPORT-API.md), which this page assumes you have read and
|
|
does not restate. Spreadsheet and no developer: [CSV-IMPORT](CSV-IMPORT.md).
|
|
Source is the classic-ASP shopdb: [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md).
|
|
|
|
## Two layers, and why the split holds
|
|
|
|
Every adopting site has its own source database, and it will not match another
|
|
site's schema. So the import is split:
|
|
|
|
1. **The import API is the stable contract.** Whatever your source looks like,
|
|
you create records through the same documented REST endpoints. This layer is
|
|
the product; it is schema-agnostic and documented once, in IMPORT-API.
|
|
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
|
|
to those endpoints. Nobody runs another site's loader - you copy the pattern.
|
|
|
|
Keeping the two apart is what stops one site's judgement calls from becoming
|
|
everyone's contract.
|
|
|
|
## The shape of a loader
|
|
|
|
The reference implementation lives under `scripts/site_imports/<site>/`. Read it
|
|
alongside this page.
|
|
|
|
- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
|
|
unscoped admin PAT in-process, and drives the real endpoints through the app
|
|
test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
|
|
exercises the same routes, authorization and validation an HTTP client would,
|
|
with no running server needed. It also holds read-only access to the source DB
|
|
and a JSON `IdMap` of legacy-id to new-id crosswalks.
|
|
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
|
|
POSTs it, and records the crosswalk later stages resolve foreign keys against.
|
|
|
|
Post-import fixups that re-point existing assets (for example
|
|
`scripts/reclassify_servers_to_network.py`, which moved servers imported as PCs
|
|
to network devices in place) belong in the site loader's verify stage, never in
|
|
the API layer.
|
|
|
|
### The crosswalk is the keystone
|
|
|
|
Reference and lookup tables first so foreign keys resolve, then the entity hub,
|
|
then dependents, then links:
|
|
|
|
```
|
|
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
|
|
-> dependents (installs, warranties, notifications, ...) -> relationships
|
|
```
|
|
|
|
Capture every legacy id to new id as you create rows, and resolve foreign keys
|
|
through it in later stages. New autoincrement ids will not match the source's.
|
|
The full ordering rule is section 4 of [IMPORT-API.md](IMPORT-API.md).
|
|
|
|
## Producing the mapping
|
|
|
|
You do not have to hand-derive the source-to-target mapping. Point an
|
|
agent-assisted workflow at a source database plus the API contract and it emits
|
|
a per-table mapping (source columns to endpoint fields, transforms, what is
|
|
importable and what is out of scope) plus a loader skeleton. That is the
|
|
repeatable onboarding path, and
|
|
[IMPORT-LEGACY-ASP.md](IMPORT-LEGACY-ASP.md) is a finished example of the output.
|
|
|
|
## Running it, against a THROWAWAY import database
|
|
|
|
1. Build a fresh target: `flask db upgrade`, `flask plugin upgrade-all`, then
|
|
`flask seed permissions/settings/reference-data`. Enable every bundled plugin
|
|
you need - some ship disabled, and a plugin's routes only register when it is
|
|
enabled at app start.
|
|
2. Load your source dump into a scratch database the loader can read.
|
|
3. Run the loader stages in order, dry-running and spot-checking as you go.
|
|
4. Verify: row-count and foreign-key-resolution audit against the source, then a
|
|
UI spot-check - log in and eyeball the lists, the map, a detail page.
|
|
5. Only then point a real instance at the imported database.
|
|
|
|
## What a mature loader ends up handling
|
|
|
|
From the reference implementation, as a checklist of what to expect:
|
|
|
|
- Fanning one source "machine" table out to several asset types
|
|
(computer/machine/network/measuring-tool) by a routing rule, with the
|
|
duplicate, placeholder and skip decisions applied.
|
|
- Synthesizing a natural key where the source lacks one (printers to `PRN-{id}`).
|
|
- Folding a primary IP onto an asset, pairing a check-in/out event log into
|
|
checkouts, deduping colliding names, reversing an inverse relationship type.
|
|
- The handful of narrow gaps the API cannot cover (there is no bulk
|
|
communications endpoint, for instance) handled as documented direct-ORM
|
|
writes.
|