shopdb-flask/docs/IMPORT-API.md

Import API: loading a site's data through HTTP

Four import pages. The HTTP wire contract: this page. Spreadsheet and no developer: CSV-IMPORT. How to shape a per-site loader around this contract: IMPORT-ADOPTION. Source is the classic-ASP shopdb: IMPORT-LEGACY-ASP on top of this page.

This page is schema-agnostic: it says how to authenticate, how to preserve a legacy row's real timestamps, and how to make a run rerunnable. It does NOT map anybody's source columns - that is the loader's job.

Every row is created through a documented endpoint, so authorization, validation, auditing and plugin hooks run exactly as they do for a human operator. No direct writes to the application database are needed or wanted.

Contents: 1 Authentication | 2 Import mode | 3 Reference-data seed | 4 Order of operations | 5 The idempotent upsert recipe | 6 Verifying a run

---

1. Authentication

Examples on this page use http://localhost:5001, a development instance. Substitute your own base URL; every import call targets /api/... on it.

Every write needs authentication, and import mode additionally needs an admin.

A large import can outlast a login JWT: access_token expires after one hour, so a long run dies mid-import with 401s. Use a personal API token (PAT) instead. A PAT never expires (unless you set an expiry), acts as the user that created it, and is sent exactly like a JWT: Authorization: Bearer shopdb_pat_.... Create one as an admin, from Settings > API Tokens or the API:

# Bootstrap: a short login JWT is fine just to mint the long-lived PAT.
JWT=$(curl -s http://localhost:5001/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"<admin>","password":"<password>"}' | jq -r '.data.access_token')

# The full secret (shopdb_pat_...) is returned ONCE. Save it now.
curl -s http://localhost:5001/api/apitokens \
  -H "Authorization: Bearer $JWT" \
  -H 'Content-Type: application/json' \
  -d '{"name":"legacy import runner"}' | jq -r '.data.secret'

Revoke it from the same Settings page (or DELETE /api/apitokens/<id>) when the import is done. Minting one needs the apitokens.create permission, which admins have by default.

Use an unscoped token. A token may optionally carry a scopes list that limits it to specific permissions; a scoped token suspends the admin bypass and is denied on role-gated endpoints AND on import mode, so it cannot run an import. Leave the "Restrict permissions" option off, which is the default.

2. Import mode: the X-Import-Mode header

By default the server stamps createddate/modifieddate to "now" on every create and update, which would erase a migrated row's real history. To preserve it, send the request header:

X-Import-Mode: true

When (and only when) the caller is an admin AND that header is present:

• create/update endpoints on timestamped entities accept optional createddate and modifieddate fields in the JSON body and store them verbatim (naive UTC). Both 2020-01-05T12:00:00 (ISO) and the legacy 2020-01-05 12:00:00 (MySQL) forms are parsed. A bare 2020-01-05 works too.

Without the header, or for a non-admin caller, those fields are silently ignored and the server behaves exactly as it does normally. This is enforced centrally by shopdb/utils/import_mode.py (import_mode_active, apply_import_timestamps, parse_import_datetime), exposed on the plugin contract surface shopdb.api.

Timestamped entities that honor createddate/modifieddate: assets (every type plugin), vendors, models, modeltypes, businessunits, locations, operating systems, applications, knowledge base, asset relationships.

Entities that carry history in domain fields instead (createddate passthrough is a no-op there, by design): notifications (starttime/endtime), warranties (startdate/enddate/lastcheckeddate). Set those fields directly in the payload; they are already accepted.

3. Reference-data seed

Before importing, seed the reference tables that have no CRUD endpoint of their own (communication types such as IP/Serial/USB, default statuses, canonical relationship types, permissions, settings):

flask seed permissions
flask seed settings
flask seed reference-data

communicationtypes is populated here, so a primary-IP mapping can resolve comtype='IP'. There is no CRUD endpoint for it; the seed is the only way in.

---

4. Order of operations

Import in dependency order so foreign keys always resolve. Each step is a lookup-then-upsert loop (section 5); rerunning any step is safe.

1. Reference and lookup data first
   1. Vendors (/api/vendors)
   2. Model types (/api/modeltypes)
   3. Models (/api/models) - needs vendors and model types
   4. Business units (/api/businessunits)
   5. Location types, then locations (/api/locations/types, /api/locations)
   6. Operating systems (/api/operatingsystems)
   7. Asset statuses (/api/assets/statuses)
   8. Relationship types (/api/assets/relationshiptypes)
   9. Notification types (/api/notifications/types)
   10. Per-plugin subtypes: computer, machine, printer and network-device types
   11. Support teams and their contacts (/api/supportteams, /api/supportteams/{id}/contacts) - BEFORE applications, because applications.supportteamid points at a team
   12. Applications (/api/applications) and their versions
2. Assets, per type. Each create makes the core Asset row plus its extension: computers, machines, printers, network devices.
3. Communications. The primary IP is set through the asset payload's ipaddress field in step 2. There is no bulk-communications endpoint, so secondary interfaces have no target; carry them as custom fields or defer them.
4. Relationships (/api/assets/relationships): needs both endpoint assets and the relationship types to exist already.
5. Installed applications (/api/computers/{id}/apps): needs computers and applications.
6. Knowledge base, notifications and warranties, including backdated history.
7. Custom fields for any source column with no home in the target schema: POST /api/customfields once per field, then PUT /api/customfields/asset/{assetid}. Custom-field values are not timestamped, so they carry no history.

A map position needs its level

mapx/mapy are absolute pixels in the native coordinate space of ONE level (ADR-017), so an import that sends coordinates without levelid produces markers the map renders as "level unknown" rather than drawing them on the default level. A source that predates levels has one floor plan, so the right value is that site's default level:

curl -s http://server/api/maplevels | python -c "import json,sys; print(json.load(sys.stdin)['data']['defaultlevelid'])"

Send that levelid on every row carrying coordinates. If an import already ran without it, the levels migration's backfill is not retroactive - assign the rows with POST /api/mappositions/positions, which requires levelid per row for exactly this reason. Loading and recalibrating the plans themselves is FLOOR-MAP.md.

---

5. The idempotent upsert recipe

The endpoints are NOT upserts. The idempotent unit is a two-step recipe that composes with import mode:

1. Look up the row by its natural key using the exact-match list filter.
2. If found, PUT to update it; if not, POST to create it.

Each import-relevant list endpoint has an exact-match filter for its natural key (added for exactly this purpose):

entity	lookup
assets (all 5 plugins)	GET /api/{plugin}?assetnumber=<n>
vendors	GET /api/vendors?vendor=<name>
models	GET /api/models?modelnumber=<m>&vendor=<vendorid>
model types	GET /api/modeltypes?modeltype=<name>
business units	GET /api/businessunits?businessunit=<name>
locations	GET /api/locations?locationname=<name>
operating systems	GET /api/operatingsystems?osname=<name>
applications	GET /api/applications?appname=<name>
knowledge base	GET /api/knowledgebase?linkurl=<url>
warranties	GET /api/warranty?servicetag=<tag>&vendor=<name>
notifications	GET /api/notifications?ticketnumber=<t>

Worked example

The loop, with the two headers that matter and a dry run. A single bad row reports and keeps going: aborting a multi-hour run on row 4000 is worse than finishing with a list of failures.

import os
import requests

BASE = "http://localhost:5001"


class ImportClient:
    def __init__(self, token=None, dryrun=False):
        self.session = requests.Session()
        self.dryrun = dryrun
        self.session.headers.update({
            # A PAT (shopdb_pat_...) does not expire like a login JWT, so it
            # survives a long import. X-Import-Mode makes the createddate /
            # modifieddate passthrough take effect.
            "Authorization": f"Bearer {token or os.environ['SHOPDB_TOKEN']}",
            "X-Import-Mode": "true",
        })

    def lookup(self, path, params):
        resp = self.session.get(f"{BASE}{path}", params=params)
        resp.raise_for_status()
        rows = resp.json().get("data") or []
        return rows[0] if rows else None

    def upsert(self, path, idfield, lookupparams, payload):
        """Lookup by natural key; PUT if found, else POST. Returns the row."""
        existing = self.lookup(path, lookupparams)
        if self.dryrun:
            print(f"[dry-run] {'PUT' if existing else 'POST'} {path} {lookupparams}")
            return existing or payload
        if existing:
            resp = self.session.put(f"{BASE}{path}/{existing[idfield]}", json=payload)
        else:
            resp = self.session.post(f"{BASE}{path}", json=payload)
        if resp.status_code >= 400:
            print(f"ERROR {resp.status_code} {path}: {resp.text[:200]}")
            return None
        return resp.json()["data"]


# One entity, in the order of section 4. Legacy history is preserved because
# X-Import-Mode is set.
client.upsert(
    "/api/vendors",
    idfield="vendorid",
    lookupparams={"vendor": row["vendor"]},
    payload={"vendor": row["vendor"],
             "createddate": row.get("dateadded"),
             "modifieddate": row.get("lastupdated")},
)

Keep a legacy_id -> new_id dictionary for every entity as you import it; you need it to remap foreign keys (a machine's businessunitid, a checkout's machineid, a relationship's machineid/related_machineid, and so on).

---

6. Verifying a run

Compare counts after each phase. The new side exposes a total in the pagination metadata, so one call per entity answers it:

curl -s "http://localhost:5001/api/vendors?per_page=1" \
  -H "Authorization: Bearer <token>" | jq '.meta.pagination.total'

Count the same thing on the source however that source is queried, and investigate any gap. Where your loader routes one source table to several target plugins, the source-side count has to be filtered the same way the routing was, or the two numbers are not measuring the same set.

Beyond counts: resolve-rate on foreign keys (nothing silently landed NULL), then a UI spot-check - log in, open a list, the map, and one detail page.

The wider API surface and its auth model is the generated spec at GET /api/docs (docs/openapi.json); llms.txt is the short version of the envelope and the auth model.