4a8bd138a9
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
102 lines
4.6 KiB
Markdown
102 lines
4.6 KiB
Markdown
# Importing a site's legacy data
|
|
|
|
## Two routes in, and which one you want
|
|
|
|
**If the site has a spreadsheet and no developer**, use the CSV import. It is
|
|
the common case, and it needs nothing beyond the templates:
|
|
|
|
```bash
|
|
flask csv templates --out csv-templates # generated from the live schema
|
|
# fill them in
|
|
flask csv import --dir csv-templates # checks only, changes nothing
|
|
flask csv import --dir csv-templates --commit
|
|
```
|
|
|
|
Foreign keys take a NAME, not an id - write `Bay 3`, not `locationid=7`. The
|
|
importer resolves them, including across files in the same run, and a name it
|
|
cannot find is reported with the line, the column and the value. Nothing is
|
|
written unless every row passes, and re-running an edited file updates rows
|
|
rather than duplicating them. See [CSV-IMPORT.md](CSV-IMPORT.md).
|
|
|
|
**If the site has a source database to read from**, and someone able to script
|
|
against it, the HTTP import API below is the better tool: it carries the whole
|
|
history, preserves original timestamps, and handles relationships the CSV set
|
|
does not model.
|
|
|
|
---
|
|
|
|
Every adopting site has its own source database - it will not match another
|
|
site's schema. So the import is split in two layers:
|
|
|
|
1. **The import API is the stable contract** (`docs/IMPORT-API.md`). Whatever
|
|
your source looks like, you create flask records through the same documented
|
|
REST endpoints, authenticated with an admin PAT and the `X-Import-Mode`
|
|
header (which preserves legacy timestamps). This layer is the product; it is
|
|
schema-agnostic.
|
|
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
|
|
to those endpoints. Nobody runs another site's loader - you copy the pattern.
|
|
|
|
The West Jefferson loader in `scripts/site_imports/wjf/` is reference
|
|
implementation #1. Read it alongside this guide.
|
|
|
|
## The shape of a loader
|
|
|
|
- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
|
|
unscoped admin PAT in-process, and drives the real endpoints through the app
|
|
test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
|
|
exercises the same routes/authz/validation an HTTP client would, no running
|
|
server needed. It also holds read-only access to the source DB and a JSON
|
|
`IdMap` of legacy-id -> new-id crosswalks.
|
|
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
|
|
POSTs it, and records the crosswalk later stages resolve foreign keys against.
|
|
|
|
Post-import fixups that re-point existing assets (example:
|
|
`scripts/reclassify_servers_to_network.py`, servers imported as PCs moved to
|
|
network devices in place) belong in the site loader's verify stage, not in the
|
|
stable API layer.
|
|
|
|
### Stage order matters
|
|
|
|
Reference/lookup tables first (so foreign keys resolve), then the entity hub,
|
|
then dependents, then links:
|
|
|
|
```
|
|
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
|
|
-> dependents (installs, warranties, notifications, ...) -> relationships
|
|
```
|
|
|
|
The **crosswalk is the keystone**: capture every legacy id -> new id as you
|
|
create rows, and resolve foreign keys through it in later stages. New
|
|
autoincrement ids will not match the source's.
|
|
|
|
## Producing the mapping
|
|
|
|
You do not have to hand-derive the source -> target mapping. Point the
|
|
agent-assisted workflow at a source database plus this API contract and it emits
|
|
a per-table mapping (source columns -> endpoint fields, transforms, what is
|
|
importable vs out of scope) and a loader skeleton. That is the repeatable
|
|
onboarding path.
|
|
|
|
## Running (against a THROWAWAY import database)
|
|
|
|
1. Build a fresh target: `flask db upgrade` + `flask plugin upgrade-all` +
|
|
`flask seed permissions/settings/reference-data`. Enable every bundled plugin
|
|
you need (some ship disabled; a plugin's routes only register when it is
|
|
enabled at app start).
|
|
2. Load your source dump into a scratch DB the loader can read.
|
|
3. Run the loader stages in order, dry-running / spot-checking as you go.
|
|
4. Verify: row-count + foreign-key-resolution audit against the source, then a
|
|
UI spot-check (log in, eyeball the lists / map / a detail page).
|
|
5. Only then point a real instance at the imported database.
|
|
|
|
## What the WJ loader demonstrates
|
|
|
|
- Fanning one legacy "machine" table out to the flask asset types
|
|
(computer/machine/network/measuring-tool) by a routing rule, with the
|
|
duplicate/placeholder/skip decisions applied.
|
|
- Synthesizing a natural key when the source lacks one (printers -> `PRN-{id}`).
|
|
- Folding a primary IP onto an asset, pairing a check-in/out event log into
|
|
checkouts, deduping colliding names, reversing an inverse relationship type.
|
|
- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications
|
|
endpoint) handled as documented direct-ORM writes.
|