WJ legacy-import reference loader: harness + reference/employees stages
Some checks failed
CI / backend (push) Successful in 1m38s
CI / naming (push) Successful in 2s
CI / frontend (push) Successful in 9s
CI / migrations-mysql (push) Failing after 8s

Layer 2 of the import design (see the memory + scratchpad/IMPORT-PLAN.md): a
SITE-SPECIFIC reference loader that maps WJ's classic-ASP schema onto the
maintained, schema-agnostic IMPORT-API contract. Other sites copy the pattern
against their own source DB; nobody runs this loader as-is.

Harness (scripts/site_imports/wjf/harness.py): builds the app against the
current DATABASE_URL (point it at a throwaway import DB), mints an unscoped
admin PAT in-process, and drives the real import endpoints through the app test
client with Authorization: Bearer + X-Import-Mode - exercising the same
routes/authz/validation an HTTP client would, no running server needed.
Read-only pymysql access to the three scratch source DBs; legacy-id -> new-id
crosswalks persist to JSON so a crashed run resumes and later stages resolve FKs.

Stages implemented + verified idempotent against a fresh scratch target
(shopdb_flask_import): reference (vendors 46, businessunits 13, operatingsystems
11) and employees (directory 415, re-run updated-not-duplicated). Remaining
stages (models, applications, assets hub + crosswalk, dependents, network, usb,
verify) are stubbed with the same shape; README documents the adoption playbook.

idmap.json is generated state (gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
cproudlock
2026-07-13 11:20:58 -04:00
parent 012718f0fd
commit b11a6f26d8
6 changed files with 337 additions and 0 deletions

View File

@@ -0,0 +1,45 @@
# WJ classic-ASP import loader (reference implementation)
This is **site-specific reference glue**, not product code. It maps West
Jefferson's classic-ASP `shopdb` / `cmmc_usb` / `wjf_employees` schema onto the
maintained, schema-agnostic import contract in `docs/IMPORT-API.md`.
**Every adopting site has its own source database.** Nobody else runs this
loader. Instead, copy the pattern:
1. Point the harness at your source DB(s) (edit `harness.Source`).
2. Write per-entity stages that read your tables and POST to the same
`docs/IMPORT-API.md` endpoints with `Authorization: Bearer <admin PAT>` and
`X-Import-Mode: true`.
3. Persist legacy-id -> new-id crosswalks (see `harness.IdMap`) so later stages
resolve foreign keys and a crashed run resumes.
The import API is the stable contract; loaders are per-site. The mapping itself
can be produced with the agent-assisted workflow (point Fable/Opus agents at a
source DB + this contract -> they emit the mapping + a loader skeleton).
## Running (against a THROWAWAY import database)
```bash
DATABASE_URL='mysql+pymysql://root:PW@127.0.0.1:3306/shopdb_flask_import?charset=utf8mb4' \
venv/bin/python -m scripts.site_imports.wjf.run --stages reference,employees
```
Prereqs: a fresh target DB built with `flask db upgrade` + `flask plugin
upgrade-all` + `flask seed permissions/settings/reference-data`, and the three
source dumps loaded into scratch DBs (`shopdb_src`, `cmmc_usb_src`,
`wjf_employees_src`). See `scratchpad/IMPORT-PLAN.md` for the full mapping,
resolved decisions, and remaining stages.
## Status
- **Implemented + verified idempotent:** `reference` (vendors, businessunits,
operatingsystems), `employees` (directory bulk upsert; photos deferred).
- **TODO stages:** `models`, `applications`, `assets` (the hub - fan machines
out by type, persist the machineid->assetid crosswalk), `dependents`
(installs, warranties, notifications, KB), `network` (+ subnets/VLANs), `usb`
(cmmc device + checkinout pairing), `verify`.
The harness (PAT auth, import-mode, id-map persistence, endpoint error capture)
is proven; the remaining stages are additional `stage_*` functions in `run.py`
following the same shape.