shopdb-flask/docs/IMPORT-API.md

<!-- docmeta
purpose: reference
audience: developer, integrator, adopter
status: current
summary: The stable HTTP import contract - PAT auth, X-Import-Mode timestamps, and the lookup-then-upsert recipe.
loadwith:
-->

# Import API: loading a site's data through HTTP

**Four import pages.** The HTTP wire contract: this page. Spreadsheet and no
developer: [CSV-IMPORT](CSV-IMPORT.md). How to shape a per-site loader around
this contract: [IMPORT-ADOPTION](IMPORT-ADOPTION.md). Source is the classic-ASP
shopdb: [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) on top of this page.

This page is schema-agnostic: it says how to authenticate, how to preserve a
legacy row's real timestamps, and how to make a run rerunnable. It does NOT map
anybody's source columns - that is the loader's job.

Every row is created through a documented endpoint, so authorization,
validation, auditing and plugin hooks run exactly as they do for a human
operator. No direct writes to the application database are needed or wanted.

Contents: [1 Authentication](#1-authentication) |
[2 Import mode](#2-import-mode-the-x-import-mode-header) |
[3 Reference-data seed](#3-reference-data-seed) |
[4 Order of operations](#4-order-of-operations) |
[5 The idempotent upsert recipe](#5-the-idempotent-upsert-recipe) |
[6 Verifying a run](#6-verifying-a-run)

---

## 1. Authentication

Examples on this page use `http://localhost:5001`, a development instance.
Substitute your own base URL; every import call targets `/api/...` on it.

Every write needs authentication, and import mode additionally needs an admin.

A large import can outlast a login JWT: `access_token` expires after one hour,
so a long run dies mid-import with 401s. Use a **personal API token (PAT)**
instead. A PAT never expires (unless you set an expiry), acts as the user that
created it, and is sent exactly like a JWT: `Authorization: Bearer
shopdb_pat_...`. Create one as an admin, from Settings > API Tokens or the API:

```bash
# Bootstrap: a short login JWT is fine just to mint the long-lived PAT.
JWT=$(curl -s http://localhost:5001/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"<admin>","password":"<password>"}' | jq -r '.data.access_token')

# The full secret (shopdb_pat_...) is returned ONCE. Save it now.
curl -s http://localhost:5001/api/apitokens \
  -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/json' \
  -d '{"name":"legacy import runner"}' | jq -r '.data.secret'
```

Revoke it from the same Settings page (or `DELETE /api/apitokens/<id>`) when
the import is done. Minting one needs the `apitokens.create` permission, which
admins have by default.

Use an **unscoped** token. A token may optionally carry a scopes list that
limits it to specific permissions; a scoped token suspends the admin bypass and
is denied on role-gated endpoints AND on import mode, so it cannot run an
import. Leave the "Restrict permissions" option off, which is the default.

## 2. Import mode: the `X-Import-Mode` header

By default the server stamps `createddate`/`modifieddate` to "now" on every
create and update, which would erase a migrated row's real history. To preserve
it, send the request header:

```
X-Import-Mode: true
```

When (and only when) the caller is an admin AND that header is present:

- create/update endpoints on timestamped entities accept optional
  `createddate` and `modifieddate` fields in the JSON body and store them
  verbatim (naive UTC). Both `2020-01-05T12:00:00` (ISO) and the legacy
  `2020-01-05 12:00:00` (MySQL) forms are parsed. A bare `2020-01-05` works too.

Without the header, or for a non-admin caller, those fields are silently ignored
and the server behaves exactly as it does normally. This is enforced centrally
by `shopdb/utils/import_mode.py` (`import_mode_active`, `apply_import_timestamps`,
`parse_import_datetime`), exposed on the plugin contract surface `shopdb.api`.

Timestamped entities that honor `createddate`/`modifieddate`: assets (every
type plugin), vendors, models, modeltypes, businessunits, locations, operating
systems, applications, knowledge base, asset relationships.

Entities that carry history in domain fields instead (createddate passthrough is
a no-op there, by design): notifications (`starttime`/`endtime`), warranties
(`startdate`/`enddate`/`lastcheckeddate`). Set those fields directly in the
payload; they are already accepted.

## 3. Reference-data seed

Before importing, seed the reference tables that have no CRUD endpoint of their
own (communication types such as IP/Serial/USB, default statuses, canonical
relationship types, permissions, settings):

```bash
flask seed permissions
flask seed settings
flask seed reference-data
```

`communicationtypes` is populated here, so a primary-IP mapping can resolve
`comtype='IP'`. There is no CRUD endpoint for it; the seed is the only way in.

---

## 4. Order of operations

Import in dependency order so foreign keys always resolve. Each step is a
lookup-then-upsert loop (section 5); rerunning any step is safe.

1. **Reference and lookup data first**
   1. Vendors (`/api/vendors`)
   2. Model types (`/api/modeltypes`)
   3. Models (`/api/models`) - needs vendors and model types
   4. Business units (`/api/businessunits`)
   5. Location types, then locations (`/api/locations/types`, `/api/locations`)
   6. Operating systems (`/api/operatingsystems`)
   7. Asset statuses (`/api/assets/statuses`)
   8. Relationship types (`/api/assets/relationshiptypes`)
   9. Notification types (`/api/notifications/types`)
   10. Per-plugin subtypes: computer, machine, printer and network-device types
   11. Support teams and their contacts (`/api/supportteams`,
       `/api/supportteams/{id}/contacts`) - BEFORE applications, because
       `applications.supportteamid` points at a team
   12. Applications (`/api/applications`) and their versions
2. **Assets, per type.** Each create makes the core Asset row plus its
   extension: computers, machines, printers, network devices.
3. **Communications.** The primary IP is set through the asset payload's
   `ipaddress` field in step 2. There is no bulk-communications endpoint, so
   secondary interfaces have no target; carry them as custom fields or defer
   them.
4. **Relationships** (`/api/assets/relationships`): needs both endpoint assets
   and the relationship types to exist already.
5. **Installed applications** (`/api/computers/{id}/apps`): needs computers and
   applications.
6. **Knowledge base, notifications and warranties**, including backdated
   history.
7. **Custom fields** for any source column with no home in the target schema:
   `POST /api/customfields` once per field, then
   `PUT /api/customfields/asset/{assetid}`. Custom-field values are not
   timestamped, so they carry no history.

### A map position needs its level

`mapx`/`mapy` are absolute pixels in the native coordinate space of ONE level
(ADR-017), so an import that sends coordinates without `levelid` produces
markers the map renders as "level unknown" rather than drawing them on the
default level. A source that predates levels has one floor plan, so the right
value is that site's default level:

```bash
curl -s http://server/api/maplevels | python -c "import json,sys; print(json.load(sys.stdin)['data']['defaultlevelid'])"
```

Send that `levelid` on every row carrying coordinates. If an import already ran
without it, the levels migration's backfill is not retroactive - assign the rows
with `POST /api/mappositions/positions`, which requires `levelid` per row for
exactly this reason. Loading and recalibrating the plans themselves is
[FLOOR-MAP.md](FLOOR-MAP.md).

---

## 5. The idempotent upsert recipe

The endpoints are NOT upserts. The idempotent unit is a two-step recipe that
composes with import mode:

1. **Look up** the row by its natural key using the exact-match list filter.
2. If found, **PUT** to update it; if not, **POST** to create it.

Each import-relevant list endpoint has an exact-match filter for its natural key
(added for exactly this purpose):

| entity | lookup |
|---|---|
| assets (all 5 plugins) | `GET /api/{plugin}?assetnumber=<n>` |
| vendors | `GET /api/vendors?vendor=<name>` |
| models | `GET /api/models?modelnumber=<m>&vendor=<vendorid>` |
| model types | `GET /api/modeltypes?modeltype=<name>` |
| business units | `GET /api/businessunits?businessunit=<name>` |
| locations | `GET /api/locations?locationname=<name>` |
| operating systems | `GET /api/operatingsystems?osname=<name>` |
| applications | `GET /api/applications?appname=<name>` |
| knowledge base | `GET /api/knowledgebase?linkurl=<url>` |
| warranties | `GET /api/warranty?servicetag=<tag>&vendor=<name>` |
| notifications | `GET /api/notifications?ticketnumber=<t>` |

### Worked example

The loop, with the two headers that matter and a dry run. A single bad row
reports and keeps going: aborting a multi-hour run on row 4000 is worse than
finishing with a list of failures.

```python
import os
import requests

BASE = "http://localhost:5001"


class ImportClient:
    def __init__(self, token=None, dryrun=False):
        self.session = requests.Session()
        self.dryrun = dryrun
        self.session.headers.update({
            # A PAT (shopdb_pat_...) does not expire like a login JWT, so it
            # survives a long import. X-Import-Mode makes the createddate /
            # modifieddate passthrough take effect.
            "Authorization": f"Bearer {token or os.environ['SHOPDB_TOKEN']}",
            "X-Import-Mode": "true",
        })

    def lookup(self, path, params):
        resp = self.session.get(f"{BASE}{path}", params=params)
        resp.raise_for_status()
        rows = resp.json().get("data") or []
        return rows[0] if rows else None

    def upsert(self, path, idfield, lookupparams, payload):
        """Lookup by natural key; PUT if found, else POST. Returns the row."""
        existing = self.lookup(path, lookupparams)
        if self.dryrun:
            print(f"[dry-run] {'PUT' if existing else 'POST'} {path} {lookupparams}")
            return existing or payload
        if existing:
            resp = self.session.put(f"{BASE}{path}/{existing[idfield]}", json=payload)
        else:
            resp = self.session.post(f"{BASE}{path}", json=payload)
        if resp.status_code >= 400:
            print(f"ERROR {resp.status_code} {path}: {resp.text[:200]}")
            return None
        return resp.json()["data"]


# One entity, in the order of section 4. Legacy history is preserved because
# X-Import-Mode is set.
client.upsert(
    "/api/vendors",
    idfield="vendorid",
    lookupparams={"vendor": row["vendor"]},
    payload={"vendor": row["vendor"],
             "createddate": row.get("dateadded"),
             "modifieddate": row.get("lastupdated")},
)
```

Keep a `legacy_id -> new_id` dictionary for every entity as you import it; you
need it to remap foreign keys (a machine's `businessunitid`, a checkout's
`machineid`, a relationship's `machineid`/`related_machineid`, and so on).

---

## 6. Verifying a run

Compare counts after each phase. The new side exposes a total in the pagination
metadata, so one call per entity answers it:

```bash
curl -s "http://localhost:5001/api/vendors?per_page=1" \
  -H "Authorization: Bearer <token>" | jq '.meta.pagination.total'
```

Count the same thing on the source however that source is queried, and
investigate any gap. Where your loader routes one source table to several target
plugins, the source-side count has to be filtered the same way the routing was,
or the two numbers are not measuring the same set.

Beyond counts: resolve-rate on foreign keys (nothing silently landed NULL), then
a UI spot-check - log in, open a list, the map, and one detail page.

The wider API surface and its auth model is the generated spec at `GET /api/docs`
(`docs/openapi.json`); [llms.txt](llms.txt) is the short version of the envelope
and the auth model.