feat(import): load a site's data from spreadsheets
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
This commit is contained in:
@@ -1,5 +1,30 @@
|
||||
# Importing a site's legacy data
|
||||
|
||||
## Two routes in, and which one you want
|
||||
|
||||
**If the site has a spreadsheet and no developer**, use the CSV import. It is
|
||||
the common case, and it needs nothing beyond the templates:
|
||||
|
||||
```bash
|
||||
flask csv templates --out csv-templates # generated from the live schema
|
||||
# fill them in
|
||||
flask csv import --dir csv-templates # checks only, changes nothing
|
||||
flask csv import --dir csv-templates --commit
|
||||
```
|
||||
|
||||
Foreign keys take a NAME, not an id - write `Bay 3`, not `locationid=7`. The
|
||||
importer resolves them, including across files in the same run, and a name it
|
||||
cannot find is reported with the line, the column and the value. Nothing is
|
||||
written unless every row passes, and re-running an edited file updates rows
|
||||
rather than duplicating them. See [CSV-IMPORT.md](CSV-IMPORT.md).
|
||||
|
||||
**If the site has a source database to read from**, and someone able to script
|
||||
against it, the HTTP import API below is the better tool: it carries the whole
|
||||
history, preserves original timestamps, and handles relationships the CSV set
|
||||
does not model.
|
||||
|
||||
---
|
||||
|
||||
Every adopting site has its own source database - it will not match another
|
||||
site's schema. So the import is split in two layers:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user