4a8bd138a9
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
143 lines
4.9 KiB
Markdown
143 lines
4.9 KiB
Markdown
# Loading a site's data from spreadsheets
|
|
|
|
For getting a new site's starting data in when you have a spreadsheet rather
|
|
than a source database to script against. No developer needed.
|
|
|
|
If the site *does* have a source system worth reading, the HTTP import API is
|
|
the better tool - see [IMPORT-ADOPTION.md](IMPORT-ADOPTION.md).
|
|
|
|
---
|
|
|
|
## The short version
|
|
|
|
```bash
|
|
cd C:\shopdb-flask # or your install directory
|
|
venv\Scripts\flask csv templates --out csv-templates
|
|
```
|
|
|
|
Fill in the templates. Then:
|
|
|
|
```bash
|
|
venv\Scripts\flask csv import --dir csv-templates
|
|
```
|
|
|
|
That **checks only** and changes nothing. It tells you what it would create and
|
|
what is wrong. When you are happy:
|
|
|
|
```bash
|
|
venv\Scripts\flask csv import --dir csv-templates --commit
|
|
```
|
|
|
|
---
|
|
|
|
## Write names, not numbers
|
|
|
|
This is the part that makes the difference. Every column that points at another
|
|
table accepts the **name** of the thing:
|
|
|
|
```
|
|
assetnumber,name,assettypeid,statusid,locationid
|
|
CMM-01,Zeiss Contura,Measuring Tool,Active,Gage Lab
|
|
MILL-07,Haas VF-2,Machine,Active,Bay 3
|
|
```
|
|
|
|
`assettypeid` gets `Measuring Tool`. `locationid` gets `Gage Lab`. You never
|
|
have to import a file, read back the numbers it generated, and paste them into
|
|
the next one.
|
|
|
|
The column keeps its database name so it matches the rest of the system, but
|
|
the value is whatever you actually know. Numeric ids still work if you have
|
|
them - useful when re-importing something this system exported.
|
|
|
|
Names resolve **across files in the same run**, so `assets.csv` can reference a
|
|
location that only exists because `locations.csv` was loaded moments earlier.
|
|
|
|
## Nothing is half-imported
|
|
|
|
Every row is checked before anything is written. If one row is wrong, nothing is
|
|
written at all - you fix the file and run it again. A mistake on line 400 never
|
|
leaves 399 rows loaded.
|
|
|
|
## Running it twice is safe
|
|
|
|
Each file is matched on a natural key - `assetnumber` for assets, `locationname`
|
|
for locations, and so on. Re-importing an edited file **updates** those rows
|
|
rather than creating second copies. Correcting a spreadsheet and re-running is
|
|
the expected workflow, not a mistake.
|
|
|
|
## What the errors look like
|
|
|
|
```
|
|
assets: 0 new, 0 updated, 1 problem(s)
|
|
line 4, column 'locationid': nothing in locations is named 'Bay 9'
|
|
- add it to locations.csv, or import that file first
|
|
```
|
|
|
|
Line, column, value, and what to do. Not a foreign key constraint violation.
|
|
|
|
---
|
|
|
|
## What you can import
|
|
|
|
Fourteen tables, in the order the importer handles them. You only need the ones
|
|
you have; skip any file you do not care about.
|
|
|
|
| Order | File | Matched on |
|
|
|---|---|---|
|
|
| 1 | `assetstatuses.csv` | `status` |
|
|
| 2 | `assettypes.csv` | `assettype` |
|
|
| 3 | `locationtypes.csv` | `locationtype` |
|
|
| 4 | `modeltypes.csv` | `modeltype` |
|
|
| 5 | `computertypes.csv` | `computertype` |
|
|
| 6 | `machinetypes.csv` | `machinetype` |
|
|
| 7 | `businessunits.csv` | `businessunit` |
|
|
| 8 | `locations.csv` | `locationname` |
|
|
| 9 | `vendors.csv` | `vendor` |
|
|
| 10 | `models.csv` | `modelnumber` |
|
|
| 11 | `operatingsystems.csv` | `osname` |
|
|
| 12 | `assets.csv` | `assetnumber` |
|
|
| 13 | `computers.csv` | `assetid` |
|
|
| 14 | `machines.csv` | `assetid` |
|
|
|
|
`--dir` handles the order for you. Use `--file` with `--table` for one file.
|
|
|
|
**User accounts are deliberately not importable.** Passwords do not belong in a
|
|
spreadsheet, in either direction. Create the first administrator through the
|
|
first-run page and the rest in the application.
|
|
|
|
## The templates are generated, not maintained
|
|
|
|
`flask csv templates` builds them from the live database schema each time. Every
|
|
column offered exists; every required one is marked; every foreign key says
|
|
which file it refers to.
|
|
|
|
This matters because the alternative does not work. A hand-written template set
|
|
was tried, and it had invented columns on seven of eleven tables and named a
|
|
table that does not exist - while looking entirely plausible. Templates that are
|
|
generated cannot drift from the schema, and a test fails the build if they ever
|
|
do.
|
|
|
|
## Editing the files
|
|
|
|
- **UTF-8**, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
|
|
- Lines starting with `#` are ignored, so the notes and the example row in each
|
|
template can stay where they are.
|
|
- Booleans are `1` or `0`.
|
|
- Dates are `YYYY-MM-DD` (`2026-08-04`). `YYYY-MM-DD HH:MM:SS` also works, as do
|
|
`DD/MM/YYYY` and `MM/DD/YYYY`.
|
|
- Leave a cell **empty** for "no value". Not `NULL`, not `N/A`.
|
|
- Quote anything containing a comma: `"Bay 3, North"`.
|
|
|
|
## If it will not run
|
|
|
|
**"the 'assets' table does not exist in this database"** - the schema has not
|
|
been created. Run `flask db upgrade` first, and check `DATABASE_URL` points at
|
|
the site you meant.
|
|
|
|
**"unknown column(s): ..."** - a column that does not exist, usually from an
|
|
older template. Regenerate with `flask csv templates`; the message lists what
|
|
the table does accept.
|
|
|
|
**"required column(s) missing: ..."** - a column that must be present has been
|
|
deleted from the header. Regenerate and copy your data across.
|