feat(import): load a site's data from spreadsheets
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
This commit is contained in:
142
docs/CSV-IMPORT.md
Normal file
142
docs/CSV-IMPORT.md
Normal file
@@ -0,0 +1,142 @@
|
||||
# Loading a site's data from spreadsheets
|
||||
|
||||
For getting a new site's starting data in when you have a spreadsheet rather
|
||||
than a source database to script against. No developer needed.
|
||||
|
||||
If the site *does* have a source system worth reading, the HTTP import API is
|
||||
the better tool - see [IMPORT-ADOPTION.md](IMPORT-ADOPTION.md).
|
||||
|
||||
---
|
||||
|
||||
## The short version
|
||||
|
||||
```bash
|
||||
cd C:\shopdb-flask # or your install directory
|
||||
venv\Scripts\flask csv templates --out csv-templates
|
||||
```
|
||||
|
||||
Fill in the templates. Then:
|
||||
|
||||
```bash
|
||||
venv\Scripts\flask csv import --dir csv-templates
|
||||
```
|
||||
|
||||
That **checks only** and changes nothing. It tells you what it would create and
|
||||
what is wrong. When you are happy:
|
||||
|
||||
```bash
|
||||
venv\Scripts\flask csv import --dir csv-templates --commit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Write names, not numbers
|
||||
|
||||
This is the part that makes the difference. Every column that points at another
|
||||
table accepts the **name** of the thing:
|
||||
|
||||
```
|
||||
assetnumber,name,assettypeid,statusid,locationid
|
||||
CMM-01,Zeiss Contura,Measuring Tool,Active,Gage Lab
|
||||
MILL-07,Haas VF-2,Machine,Active,Bay 3
|
||||
```
|
||||
|
||||
`assettypeid` gets `Measuring Tool`. `locationid` gets `Gage Lab`. You never
|
||||
have to import a file, read back the numbers it generated, and paste them into
|
||||
the next one.
|
||||
|
||||
The column keeps its database name so it matches the rest of the system, but
|
||||
the value is whatever you actually know. Numeric ids still work if you have
|
||||
them - useful when re-importing something this system exported.
|
||||
|
||||
Names resolve **across files in the same run**, so `assets.csv` can reference a
|
||||
location that only exists because `locations.csv` was loaded moments earlier.
|
||||
|
||||
## Nothing is half-imported
|
||||
|
||||
Every row is checked before anything is written. If one row is wrong, nothing is
|
||||
written at all - you fix the file and run it again. A mistake on line 400 never
|
||||
leaves 399 rows loaded.
|
||||
|
||||
## Running it twice is safe
|
||||
|
||||
Each file is matched on a natural key - `assetnumber` for assets, `locationname`
|
||||
for locations, and so on. Re-importing an edited file **updates** those rows
|
||||
rather than creating second copies. Correcting a spreadsheet and re-running is
|
||||
the expected workflow, not a mistake.
|
||||
|
||||
## What the errors look like
|
||||
|
||||
```
|
||||
assets: 0 new, 0 updated, 1 problem(s)
|
||||
line 4, column 'locationid': nothing in locations is named 'Bay 9'
|
||||
- add it to locations.csv, or import that file first
|
||||
```
|
||||
|
||||
Line, column, value, and what to do. Not a foreign key constraint violation.
|
||||
|
||||
---
|
||||
|
||||
## What you can import
|
||||
|
||||
Fourteen tables, in the order the importer handles them. You only need the ones
|
||||
you have; skip any file you do not care about.
|
||||
|
||||
| Order | File | Matched on |
|
||||
|---|---|---|
|
||||
| 1 | `assetstatuses.csv` | `status` |
|
||||
| 2 | `assettypes.csv` | `assettype` |
|
||||
| 3 | `locationtypes.csv` | `locationtype` |
|
||||
| 4 | `modeltypes.csv` | `modeltype` |
|
||||
| 5 | `computertypes.csv` | `computertype` |
|
||||
| 6 | `machinetypes.csv` | `machinetype` |
|
||||
| 7 | `businessunits.csv` | `businessunit` |
|
||||
| 8 | `locations.csv` | `locationname` |
|
||||
| 9 | `vendors.csv` | `vendor` |
|
||||
| 10 | `models.csv` | `modelnumber` |
|
||||
| 11 | `operatingsystems.csv` | `osname` |
|
||||
| 12 | `assets.csv` | `assetnumber` |
|
||||
| 13 | `computers.csv` | `assetid` |
|
||||
| 14 | `machines.csv` | `assetid` |
|
||||
|
||||
`--dir` handles the order for you. Use `--file` with `--table` for one file.
|
||||
|
||||
**User accounts are deliberately not importable.** Passwords do not belong in a
|
||||
spreadsheet, in either direction. Create the first administrator through the
|
||||
first-run page and the rest in the application.
|
||||
|
||||
## The templates are generated, not maintained
|
||||
|
||||
`flask csv templates` builds them from the live database schema each time. Every
|
||||
column offered exists; every required one is marked; every foreign key says
|
||||
which file it refers to.
|
||||
|
||||
This matters because the alternative does not work. A hand-written template set
|
||||
was tried, and it had invented columns on seven of eleven tables and named a
|
||||
table that does not exist - while looking entirely plausible. Templates that are
|
||||
generated cannot drift from the schema, and a test fails the build if they ever
|
||||
do.
|
||||
|
||||
## Editing the files
|
||||
|
||||
- **UTF-8**, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
|
||||
- Lines starting with `#` are ignored, so the notes and the example row in each
|
||||
template can stay where they are.
|
||||
- Booleans are `1` or `0`.
|
||||
- Dates are `YYYY-MM-DD` (`2026-08-04`). `YYYY-MM-DD HH:MM:SS` also works, as do
|
||||
`DD/MM/YYYY` and `MM/DD/YYYY`.
|
||||
- Leave a cell **empty** for "no value". Not `NULL`, not `N/A`.
|
||||
- Quote anything containing a comma: `"Bay 3, North"`.
|
||||
|
||||
## If it will not run
|
||||
|
||||
**"the 'assets' table does not exist in this database"** - the schema has not
|
||||
been created. Run `flask db upgrade` first, and check `DATABASE_URL` points at
|
||||
the site you meant.
|
||||
|
||||
**"unknown column(s): ..."** - a column that does not exist, usually from an
|
||||
older template. Regenerate with `flask csv templates`; the message lists what
|
||||
the table does accept.
|
||||
|
||||
**"required column(s) missing: ..."** - a column that must be present has been
|
||||
deleted from the header. Regenerate and copy your data across.
|
||||
@@ -1,5 +1,30 @@
|
||||
# Importing a site's legacy data
|
||||
|
||||
## Two routes in, and which one you want
|
||||
|
||||
**If the site has a spreadsheet and no developer**, use the CSV import. It is
|
||||
the common case, and it needs nothing beyond the templates:
|
||||
|
||||
```bash
|
||||
flask csv templates --out csv-templates # generated from the live schema
|
||||
# fill them in
|
||||
flask csv import --dir csv-templates # checks only, changes nothing
|
||||
flask csv import --dir csv-templates --commit
|
||||
```
|
||||
|
||||
Foreign keys take a NAME, not an id - write `Bay 3`, not `locationid=7`. The
|
||||
importer resolves them, including across files in the same run, and a name it
|
||||
cannot find is reported with the line, the column and the value. Nothing is
|
||||
written unless every row passes, and re-running an edited file updates rows
|
||||
rather than duplicating them. See [CSV-IMPORT.md](CSV-IMPORT.md).
|
||||
|
||||
**If the site has a source database to read from**, and someone able to script
|
||||
against it, the HTTP import API below is the better tool: it carries the whole
|
||||
history, preserves original timestamps, and handles relationships the CSV set
|
||||
does not model.
|
||||
|
||||
---
|
||||
|
||||
Every adopting site has its own source database - it will not match another
|
||||
site's schema. So the import is split in two layers:
|
||||
|
||||
|
||||
@@ -28,6 +28,19 @@ answers "does this server carry component X" from the on-box CycloneDX SBOM.
|
||||
Python is 3.14 and the wheelhouse is locked to it; an upgrade against a venv
|
||||
built by a different minor version is refused by design.
|
||||
|
||||
## Bulk-loading a site's data
|
||||
Two routes, and the right answer depends on what the site has:
|
||||
- SPREADSHEET, no developer (the common case): `flask csv templates --out <dir>`
|
||||
generates templates FROM THE LIVE SCHEMA, then `flask csv import --dir <dir>`
|
||||
checks and `--commit` applies. Foreign keys accept the NAME of the referenced
|
||||
row ('Bay 3'), not a numeric id, and resolve across files in one run. Dry run
|
||||
is the default; nothing is written unless every row passes; re-importing an
|
||||
edited file updates rather than duplicates. See `docs/CSV-IMPORT.md`.
|
||||
- A SOURCE DATABASE to script against: the HTTP import API, `docs/IMPORT-API.md`
|
||||
and `docs/IMPORT-ADOPTION.md`.
|
||||
Do NOT hand-write CSV templates - generate them. User accounts are deliberately
|
||||
not CSV-importable.
|
||||
|
||||
## Base URL
|
||||
Prod (West Jefferson): `https://tsgwp00525.wjs.geaerospace.net/shopdb`
|
||||
All API paths are under `/api` (e.g. `<base>/api/assets`). Dev: `http://localhost:5001`.
|
||||
|
||||
Reference in New Issue
Block a user