# Importing a site's legacy data

## Two routes in, and which one you want

**If the site has a spreadsheet and no developer**, use the CSV import. It is
the common case, and it needs nothing beyond the templates:

```bash
flask csv templates --out csv-templates    # generated from the live schema
# fill them in
flask csv import --dir csv-templates       # checks only, changes nothing
flask csv import --dir csv-templates --commit
```

Foreign keys take a NAME, not an id - write `Bay 3`, not `locationid=7`. The
importer resolves them, including across files in the same run, and a name it
cannot find is reported with the line, the column and the value. Nothing is
written unless every row passes, and re-running an edited file updates rows
rather than duplicating them. See [CSV-IMPORT.md](CSV-IMPORT.md).

**If the site has a source database to read from**, and someone able to script
against it, the HTTP import API below is the better tool: it carries the whole
history, preserves original timestamps, and handles relationships the CSV set
does not model.

---

Every adopting site has its own source database - it will not match another
site's schema. So the import is split in two layers:

1. **The import API is the stable contract** (`docs/IMPORT-API.md`). Whatever
   your source looks like, you create flask records through the same documented
   REST endpoints, authenticated with an admin PAT and the `X-Import-Mode`
   header (which preserves legacy timestamps). This layer is the product; it is
   schema-agnostic.
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
   to those endpoints. Nobody runs another site's loader - you copy the pattern.

The the reference site loader in `scripts/site_imports/wjf/` is reference
implementation #1. Read it alongside this guide.

## The shape of a loader

- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
  unscoped admin PAT in-process, and drives the real endpoints through the app
  test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
  exercises the same routes/authz/validation an HTTP client would, no running
  server needed. It also holds read-only access to the source DB and a JSON
  `IdMap` of legacy-id -> new-id crosswalks.
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
  POSTs it, and records the crosswalk later stages resolve foreign keys against.

Post-import fixups that re-point existing assets (example:
`scripts/reclassify_servers_to_network.py`, servers imported as PCs moved to
network devices in place) belong in the site loader's verify stage, not in the
stable API layer.

### Stage order matters

Reference/lookup tables first (so foreign keys resolve), then the entity hub,
then dependents, then links:

```
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
  -> dependents (installs, warranties, notifications, ...) -> relationships
```

The **crosswalk is the keystone**: capture every legacy id -> new id as you
create rows, and resolve foreign keys through it in later stages. New
autoincrement ids will not match the source's.

## Producing the mapping

You do not have to hand-derive the source -> target mapping. Point the
agent-assisted workflow at a source database plus this API contract and it emits
a per-table mapping (source columns -> endpoint fields, transforms, what is
importable vs out of scope) and a loader skeleton. That is the repeatable
onboarding path.

## Running (against a THROWAWAY import database)

1. Build a fresh target: `flask db upgrade` + `flask plugin upgrade-all` +
   `flask seed permissions/settings/reference-data`. Enable every bundled plugin
   you need (some ship disabled; a plugin's routes only register when it is
   enabled at app start).
2. Load your source dump into a scratch DB the loader can read.
3. Run the loader stages in order, dry-running / spot-checking as you go.
4. Verify: row-count + foreign-key-resolution audit against the source, then a
   UI spot-check (log in, eyeball the lists / map / a detail page).
5. Only then point a real instance at the imported database.

## What the reference loader demonstrates

- Fanning one legacy "machine" table out to the flask asset types
  (computer/machine/network/measuring-tool) by a routing rule, with the
  duplicate/placeholder/skip decisions applied.
- Synthesizing a natural key when the source lacks one (printers -> `PRN-{id}`).
- Folding a primary IP onto an asset, pairing a check-in/out event log into
  checkouts, deduping colliding names, reversing an inverse relationship type.
- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications
  endpoint) handled as documented direct-ORM writes.