WJ legacy-import reference loader: harness + reference/employees stages
Layer 2 of the import design (see the memory + scratchpad/IMPORT-PLAN.md): a SITE-SPECIFIC reference loader that maps WJ's classic-ASP schema onto the maintained, schema-agnostic IMPORT-API contract. Other sites copy the pattern against their own source DB; nobody runs this loader as-is. Harness (scripts/site_imports/wjf/harness.py): builds the app against the current DATABASE_URL (point it at a throwaway import DB), mints an unscoped admin PAT in-process, and drives the real import endpoints through the app test client with Authorization: Bearer + X-Import-Mode - exercising the same routes/authz/validation an HTTP client would, no running server needed. Read-only pymysql access to the three scratch source DBs; legacy-id -> new-id crosswalks persist to JSON so a crashed run resumes and later stages resolve FKs. Stages implemented + verified idempotent against a fresh scratch target (shopdb_flask_import): reference (vendors 46, businessunits 13, operatingsystems 11) and employees (directory 415, re-run updated-not-duplicated). Remaining stages (models, applications, assets hub + crosswalk, dependents, network, usb, verify) are stubbed with the same shape; README documents the adoption playbook. idmap.json is generated state (gitignored). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
45
scripts/site_imports/wjf/README.md
Normal file
45
scripts/site_imports/wjf/README.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# WJ classic-ASP import loader (reference implementation)
|
||||
|
||||
This is **site-specific reference glue**, not product code. It maps West
|
||||
Jefferson's classic-ASP `shopdb` / `cmmc_usb` / `wjf_employees` schema onto the
|
||||
maintained, schema-agnostic import contract in `docs/IMPORT-API.md`.
|
||||
|
||||
**Every adopting site has its own source database.** Nobody else runs this
|
||||
loader. Instead, copy the pattern:
|
||||
|
||||
1. Point the harness at your source DB(s) (edit `harness.Source`).
|
||||
2. Write per-entity stages that read your tables and POST to the same
|
||||
`docs/IMPORT-API.md` endpoints with `Authorization: Bearer <admin PAT>` and
|
||||
`X-Import-Mode: true`.
|
||||
3. Persist legacy-id -> new-id crosswalks (see `harness.IdMap`) so later stages
|
||||
resolve foreign keys and a crashed run resumes.
|
||||
|
||||
The import API is the stable contract; loaders are per-site. The mapping itself
|
||||
can be produced with the agent-assisted workflow (point Fable/Opus agents at a
|
||||
source DB + this contract -> they emit the mapping + a loader skeleton).
|
||||
|
||||
## Running (against a THROWAWAY import database)
|
||||
|
||||
```bash
|
||||
DATABASE_URL='mysql+pymysql://root:PW@127.0.0.1:3306/shopdb_flask_import?charset=utf8mb4' \
|
||||
venv/bin/python -m scripts.site_imports.wjf.run --stages reference,employees
|
||||
```
|
||||
|
||||
Prereqs: a fresh target DB built with `flask db upgrade` + `flask plugin
|
||||
upgrade-all` + `flask seed permissions/settings/reference-data`, and the three
|
||||
source dumps loaded into scratch DBs (`shopdb_src`, `cmmc_usb_src`,
|
||||
`wjf_employees_src`). See `scratchpad/IMPORT-PLAN.md` for the full mapping,
|
||||
resolved decisions, and remaining stages.
|
||||
|
||||
## Status
|
||||
|
||||
- **Implemented + verified idempotent:** `reference` (vendors, businessunits,
|
||||
operatingsystems), `employees` (directory bulk upsert; photos deferred).
|
||||
- **TODO stages:** `models`, `applications`, `assets` (the hub - fan machines
|
||||
out by type, persist the machineid->assetid crosswalk), `dependents`
|
||||
(installs, warranties, notifications, KB), `network` (+ subnets/VLANs), `usb`
|
||||
(cmmc device + checkinout pairing), `verify`.
|
||||
|
||||
The harness (PAT auth, import-mode, id-map persistence, endpoint error capture)
|
||||
is proven; the remaining stages are additional `stage_*` functions in `run.py`
|
||||
following the same shape.
|
||||
Reference in New Issue
Block a user