# Building a per-site import loader **Four import pages.** Shaping the loader: this page. The HTTP wire contract it drives: [IMPORT-API](IMPORT-API.md), which this page assumes you have read and does not restate. Spreadsheet and no developer: [CSV-IMPORT](CSV-IMPORT.md). Source is the classic-ASP shopdb: [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md). ## Two layers, and why the split holds Every adopting site has its own source database, and it will not match another site's schema. So the import is split: 1. **The import API is the stable contract.** Whatever your source looks like, you create records through the same documented REST endpoints. This layer is the product; it is schema-agnostic and documented once, in IMPORT-API. 2. **A per-site loader is thin glue.** It reads *your* source database and POSTs to those endpoints. Nobody runs another site's loader - you copy the pattern. Keeping the two apart is what stops one site's judgement calls from becoming everyone's contract. ## The shape of a loader The reference implementation lives under `scripts/site_imports//`. Read it alongside this page. - `harness.py` - builds the app against the target `DATABASE_URL`, mints an unscoped admin PAT in-process, and drives the real endpoints through the app test client with `Authorization: Bearer ` + `X-Import-Mode: true`. This exercises the same routes, authorization and validation an HTTP client would, with no running server needed. It also holds read-only access to the source DB and a JSON `IdMap` of legacy-id to new-id crosswalks. - `run.py` - ordered `stage_*` functions. Each reads a slice of the source, POSTs it, and records the crosswalk later stages resolve foreign keys against. Post-import fixups that re-point existing assets (for example `scripts/reclassify_servers_to_network.py`, which moved servers imported as PCs to network devices in place) belong in the site loader's verify stage, never in the API layer. ### The crosswalk is the keystone Reference and lookup tables first so foreign keys resolve, then the entity hub, then dependents, then links: ``` reference -> catalog -> assets (persist the source-id -> assetid crosswalk) -> dependents (installs, warranties, notifications, ...) -> relationships ``` Capture every legacy id to new id as you create rows, and resolve foreign keys through it in later stages. New autoincrement ids will not match the source's. The full ordering rule is section 4 of [IMPORT-API.md](IMPORT-API.md). ## Producing the mapping You do not have to hand-derive the source-to-target mapping. Point an agent-assisted workflow at a source database plus the API contract and it emits a per-table mapping (source columns to endpoint fields, transforms, what is importable and what is out of scope) plus a loader skeleton. That is the repeatable onboarding path, and [IMPORT-LEGACY-ASP.md](IMPORT-LEGACY-ASP.md) is a finished example of the output. ## Running it, against a THROWAWAY import database 1. Build a fresh target: `flask db upgrade`, `flask plugin upgrade-all`, then `flask seed permissions/settings/reference-data`. Enable every bundled plugin you need - some ship disabled, and a plugin's routes only register when it is enabled at app start. 2. Load your source dump into a scratch database the loader can read. 3. Run the loader stages in order, dry-running and spot-checking as you go. 4. Verify: row-count and foreign-key-resolution audit against the source, then a UI spot-check - log in and eyeball the lists, the map, a detail page. 5. Only then point a real instance at the imported database. ## What a mature loader ends up handling From the reference implementation, as a checklist of what to expect: - Fanning one source "machine" table out to several asset types (computer/machine/network/measuring-tool) by a routing rule, with the duplicate, placeholder and skip decisions applied. - Synthesizing a natural key where the source lacks one (printers to `PRN-{id}`). - Folding a primary IP onto an asset, pairing a check-in/out event log into checkouts, deduping colliding names, reversing an inverse relationship type. - The handful of narrow gaps the API cannot cover (there is no bulk communications endpoint, for instance) handled as documented direct-ORM writes.