diff --git a/docs/IMPORT-ADOPTION.md b/docs/IMPORT-ADOPTION.md new file mode 100644 index 0000000..834f001 --- /dev/null +++ b/docs/IMPORT-ADOPTION.md @@ -0,0 +1,71 @@ +# Importing a site's legacy data + +Every adopting site has its own source database - it will not match another +site's schema. So the import is split in two layers: + +1. **The import API is the stable contract** (`docs/IMPORT-API.md`). Whatever + your source looks like, you create flask records through the same documented + REST endpoints, authenticated with an admin PAT and the `X-Import-Mode` + header (which preserves legacy timestamps). This layer is the product; it is + schema-agnostic. +2. **A per-site loader is thin glue.** It reads *your* source database and POSTs + to those endpoints. Nobody runs another site's loader - you copy the pattern. + +The West Jefferson loader in `scripts/site_imports/wjf/` is reference +implementation #1. Read it alongside this guide. + +## The shape of a loader + +- `harness.py` - builds the app against the target `DATABASE_URL`, mints an + unscoped admin PAT in-process, and drives the real endpoints through the app + test client with `Authorization: Bearer ` + `X-Import-Mode: true`. This + exercises the same routes/authz/validation an HTTP client would, no running + server needed. It also holds read-only access to the source DB and a JSON + `IdMap` of legacy-id -> new-id crosswalks. +- `run.py` - ordered `stage_*` functions. Each reads a slice of the source, + POSTs it, and records the crosswalk later stages resolve foreign keys against. + +### Stage order matters + +Reference/lookup tables first (so foreign keys resolve), then the entity hub, +then dependents, then links: + +``` +reference -> catalog -> assets (persist the source-id -> assetid crosswalk) + -> dependents (installs, warranties, notifications, ...) -> relationships +``` + +The **crosswalk is the keystone**: capture every legacy id -> new id as you +create rows, and resolve foreign keys through it in later stages. New +autoincrement ids will not match the source's. + +## Producing the mapping + +You do not have to hand-derive the source -> target mapping. Point the +agent-assisted workflow at a source database plus this API contract and it emits +a per-table mapping (source columns -> endpoint fields, transforms, what is +importable vs out of scope) and a loader skeleton. That is the repeatable +onboarding path. + +## Running (against a THROWAWAY import database) + +1. Build a fresh target: `flask db upgrade` + `flask plugin upgrade-all` + + `flask seed permissions/settings/reference-data`. Enable every bundled plugin + you need (some ship disabled; a plugin's routes only register when it is + enabled at app start). +2. Load your source dump into a scratch DB the loader can read. +3. Run the loader stages in order, dry-running / spot-checking as you go. +4. Verify: row-count + foreign-key-resolution audit against the source, then a + UI spot-check (log in, eyeball the lists / map / a detail page). +5. Only then point a real instance at the imported database. + +## What the WJ loader demonstrates + +- Fanning one legacy "machine" table out to the flask asset types + (computer/machine/network/measuring-tool) by a routing rule, with the + duplicate/placeholder/skip decisions applied. +- Synthesizing a natural key when the source lacks one (printers -> `PRN-{id}`). +- Folding a primary IP onto an asset, pairing a check-in/out event log into + checkouts, deduping colliding names, reversing an inverse relationship type. +- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications + endpoint) handled as documented direct-ORM writes. diff --git a/scripts/migration/README.md b/scripts/migration/README.md new file mode 100644 index 0000000..7caca36 --- /dev/null +++ b/scripts/migration/README.md @@ -0,0 +1,24 @@ +# scripts/migration (superseded) + +The direct-SQL migrators that used to live here (`migrate_assets.py`, +`migrate_communications.py`, `migrate_notifications.py`, `migrate_usb.py`, +`run_migration.py`, `verify_migration.py`) and `scripts/import_from_mysql.py` +were **removed** - they had drifted badly out of sync with the current schema +(they targeted a nonexistent `equipment` table, the retired `Machine` model, and +columns that no longer exist) and were actively misleading. + +## Use the import API + a site loader instead + +Legacy data is imported through the maintained, schema-agnostic contract in +[`docs/IMPORT-API.md`](../../docs/IMPORT-API.md) (create through real endpoints +with an admin PAT + `X-Import-Mode`). Each adopting site writes a thin loader +that reads its own source database and drives that API. + +- Reference implementation: [`scripts/site_imports/wjf/`](../site_imports/wjf/) + (West Jefferson classic-ASP -> flask). +- Adoption playbook: [`docs/IMPORT-ADOPTION.md`](../../docs/IMPORT-ADOPTION.md). + +## What's kept here + +- `fix_legacy_schema.sql` - one-time SQL fixups against a legacy source DB. +- `one-offs/` - individual one-shot SQL snippets (see its README). diff --git a/scripts/site_imports/wjf/README.md b/scripts/site_imports/wjf/README.md index 9ec2ad0..9a580eb 100644 --- a/scripts/site_imports/wjf/README.md +++ b/scripts/site_imports/wjf/README.md @@ -33,13 +33,16 @@ resolved decisions, and remaining stages. ## Status -- **Implemented + verified idempotent:** `reference` (vendors, businessunits, - operatingsystems), `employees` (directory bulk upsert; photos deferred). -- **TODO stages:** `models`, `applications`, `assets` (the hub - fan machines - out by type, persist the machineid->assetid crosswalk), `dependents` - (installs, warranties, notifications, KB), `network` (+ subnets/VLANs), `usb` - (cmmc device + checkinout pairing), `verify`. +Complete - all 15 stages built and verified end-to-end against a fresh scratch +target (zero endpoint errors): `reference`, `employees`, `catalog`, `assets` +(the hub + machineid->assetid crosswalk), `locations`, `printers`, +`communications`, `applications` (+ installs), `warranties`, `notifications`, +`knowledgebase`, `relationships`, `subnets`, `usb`, `verify`. -The harness (PAT auth, import-mode, id-map persistence, endpoint error capture) -is proven; the remaining stages are additional `stage_*` functions in `run.py` -following the same shape. +Last full run: 983 assets (computer 663, machine 76, network 58, measuring-tool +136, printer 50), 24 locations, 415 employees, 850 installs, 461 primary IPs, +464 warranties, 261 notifications, 341 KB, 93 relationships, 37 subnets, 18 USB +devices + 232 events. UI spot-check passed (computer list + shop-floor map). + +Known follow-ups: general asset `locationid` is null (no source column outside +printers); the ~146 duplicate machinenumbers are first-wins-skipped by decision.