ge-aerospace/shopdb-flask

Fork 0

Files

History

cproudlock 99ad6c0ebb

CI / backend (push) Successful in 1m38s

Details

CI / naming (push) Successful in 1s

Details

CI / frontend (push) Successful in 7s

Details

CI / migrations-mysql (push) Failing after 7s

Details

WJ import loader: tail stages (locations, relationships, subnets, usb, verify)

Completes the loader end to end. Verified on a fresh scratch target, zero
endpoint errors:

- locations: the 24 islocationonly rows -> core Locations (crosswalk machineid
  -> locationid).
- relationships: 93 active edges imported (206 of 299 dropped because an
  endpoint became a Location / was skipped / dedup-lost); types folded onto the
  seeded canonical set; dedup on (source,target,type).
- subnets: 37 (full CIDR reconstructed as INET_NTOA(ipstart)+suffix; VLANs
  lookup-or-create by number; 3 duplicate CIDRs first-wins-skipped).
- usb: 18 cmmc devices + 232 check-in/out events, paired with per-device open
  state so unpaired log rows do not 400. Needs the usb plugin enabled + usb
  directory mode selfhosted.
- verify: source-vs-target row-count audit (assets 1167->933 by the skip rules,
  applications 121=121, employees 415=415, KB 342->341).

Full pipeline default runs all 14 stages in order. NOTE: the import target needs
every bundled plugin enabled (usb ships disabled in this dev registry - enable
it before importing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-13 12:05:33 -04:00

__init__.py

WJ legacy-import reference loader: harness + reference/employees stages

2026-07-13 11:20:58 -04:00

harness.py

WJ import loader: catalog + assets-hub stages (the machineid->assetid crosswalk)

2026-07-13 11:37:11 -04:00

README.md

WJ legacy-import reference loader: harness + reference/employees stages

2026-07-13 11:20:58 -04:00

run.py

WJ import loader: tail stages (locations, relationships, subnets, usb, verify)

2026-07-13 12:05:33 -04:00

README.md

WJ classic-ASP import loader (reference implementation)

This is site-specific reference glue, not product code. It maps West Jefferson's classic-ASP shopdb / cmmc_usb / wjf_employees schema onto the maintained, schema-agnostic import contract in docs/IMPORT-API.md.

Every adopting site has its own source database. Nobody else runs this loader. Instead, copy the pattern:

Point the harness at your source DB(s) (edit harness.Source).
Write per-entity stages that read your tables and POST to the same docs/IMPORT-API.md endpoints with Authorization: Bearer <admin PAT> and X-Import-Mode: true.
Persist legacy-id -> new-id crosswalks (see harness.IdMap) so later stages resolve foreign keys and a crashed run resumes.

The import API is the stable contract; loaders are per-site. The mapping itself can be produced with the agent-assisted workflow (point Fable/Opus agents at a source DB + this contract -> they emit the mapping + a loader skeleton).

Running (against a THROWAWAY import database)

DATABASE_URL='mysql+pymysql://root:PW@127.0.0.1:3306/shopdb_flask_import?charset=utf8mb4' \
  venv/bin/python -m scripts.site_imports.wjf.run --stages reference,employees

Prereqs: a fresh target DB built with flask db upgrade + flask plugin upgrade-all + flask seed permissions/settings/reference-data, and the three source dumps loaded into scratch DBs (shopdb_src, cmmc_usb_src, wjf_employees_src). See scratchpad/IMPORT-PLAN.md for the full mapping, resolved decisions, and remaining stages.

Status

Implemented + verified idempotent: reference (vendors, businessunits, operatingsystems), employees (directory bulk upsert; photos deferred).
TODO stages: models, applications, assets (the hub - fan machines out by type, persist the machineid->assetid crosswalk), dependents (installs, warranties, notifications, KB), network (+ subnets/VLANs), usb (cmmc device + checkinout pairing), verify.

The harness (PAT auth, import-mode, id-map persistence, endpoint error capture) is proven; the remaining stages are additional stage_* functions in run.py following the same shape.