4a8bd138a9
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
4.9 KiB
Loading a site's data from spreadsheets
For getting a new site's starting data in when you have a spreadsheet rather than a source database to script against. No developer needed.
If the site does have a source system worth reading, the HTTP import API is the better tool - see IMPORT-ADOPTION.md.
The short version
cd C:\shopdb-flask # or your install directory
venv\Scripts\flask csv templates --out csv-templates
Fill in the templates. Then:
venv\Scripts\flask csv import --dir csv-templates
That checks only and changes nothing. It tells you what it would create and what is wrong. When you are happy:
venv\Scripts\flask csv import --dir csv-templates --commit
Write names, not numbers
This is the part that makes the difference. Every column that points at another table accepts the name of the thing:
assetnumber,name,assettypeid,statusid,locationid
CMM-01,Zeiss Contura,Measuring Tool,Active,Gage Lab
MILL-07,Haas VF-2,Machine,Active,Bay 3
assettypeid gets Measuring Tool. locationid gets Gage Lab. You never
have to import a file, read back the numbers it generated, and paste them into
the next one.
The column keeps its database name so it matches the rest of the system, but the value is whatever you actually know. Numeric ids still work if you have them - useful when re-importing something this system exported.
Names resolve across files in the same run, so assets.csv can reference a
location that only exists because locations.csv was loaded moments earlier.
Nothing is half-imported
Every row is checked before anything is written. If one row is wrong, nothing is written at all - you fix the file and run it again. A mistake on line 400 never leaves 399 rows loaded.
Running it twice is safe
Each file is matched on a natural key - assetnumber for assets, locationname
for locations, and so on. Re-importing an edited file updates those rows
rather than creating second copies. Correcting a spreadsheet and re-running is
the expected workflow, not a mistake.
What the errors look like
assets: 0 new, 0 updated, 1 problem(s)
line 4, column 'locationid': nothing in locations is named 'Bay 9'
- add it to locations.csv, or import that file first
Line, column, value, and what to do. Not a foreign key constraint violation.
What you can import
Fourteen tables, in the order the importer handles them. You only need the ones you have; skip any file you do not care about.
| Order | File | Matched on |
|---|---|---|
| 1 | assetstatuses.csv |
status |
| 2 | assettypes.csv |
assettype |
| 3 | locationtypes.csv |
locationtype |
| 4 | modeltypes.csv |
modeltype |
| 5 | computertypes.csv |
computertype |
| 6 | machinetypes.csv |
machinetype |
| 7 | businessunits.csv |
businessunit |
| 8 | locations.csv |
locationname |
| 9 | vendors.csv |
vendor |
| 10 | models.csv |
modelnumber |
| 11 | operatingsystems.csv |
osname |
| 12 | assets.csv |
assetnumber |
| 13 | computers.csv |
assetid |
| 14 | machines.csv |
assetid |
--dir handles the order for you. Use --file with --table for one file.
User accounts are deliberately not importable. Passwords do not belong in a spreadsheet, in either direction. Create the first administrator through the first-run page and the rest in the application.
The templates are generated, not maintained
flask csv templates builds them from the live database schema each time. Every
column offered exists; every required one is marked; every foreign key says
which file it refers to.
This matters because the alternative does not work. A hand-written template set was tried, and it had invented columns on seven of eleven tables and named a table that does not exist - while looking entirely plausible. Templates that are generated cannot drift from the schema, and a test fails the build if they ever do.
Editing the files
- UTF-8, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
- Lines starting with
#are ignored, so the notes and the example row in each template can stay where they are. - Booleans are
1or0. - Dates are
YYYY-MM-DD(2026-08-04).YYYY-MM-DD HH:MM:SSalso works, as doDD/MM/YYYYandMM/DD/YYYY. - Leave a cell empty for "no value". Not
NULL, notN/A. - Quote anything containing a comma:
"Bay 3, North".
If it will not run
"the 'assets' table does not exist in this database" - the schema has not
been created. Run flask db upgrade first, and check DATABASE_URL points at
the site you meant.
"unknown column(s): ..." - a column that does not exist, usually from an
older template. Regenerate with flask csv templates; the message lists what
the table does accept.
"required column(s) missing: ..." - a column that must be present has been deleted from the header. Regenerate and copy your data across.