4995456136
The publishability gate caught internal tooling names and developer paths but nothing site-specific, so roughly sixty leaks reached the wiki: the site name in ten documents, real fleet hostnames in the collector and GE-Enforce examples, an internal database name through the whole import guide, imaging-share paths, and a maintainer's username as the Deciders line of every ADR and inside a generated curl example. None of it is a security matter on an air-gapped fleet. It matters because these pages are read by engineers at other plants, and a document that names one site throughout reads as that site's notes rather than a product's documentation - which is exactly what it then gets treated as. Examples now use neutral hostnames, the site is "the reference site" where the distinction carries meaning, and ADRs are decided by "ShopDB maintainers". The gate carries all of these patterns, so the next one fails a build. Two documents leave docs/ because they were never written for an outside reader. PROJECT-REVIEW.md is an internal health memo pinned to a commit from July, whose headline finding (an untracked playbook) has since been fixed - it is history, and git holds it. PILOT-DEPLOY.md is one site's own cutover runbook, complete with a "re-measure before publishing" placeholder; it moves next to the loader it belongs to, in scripts/site_imports/wjf/. ADR-015 is AMENDED rather than rewritten. Its enforcement section still said report-only and its backlog still listed hardcodes that are now cleared, which left the record contradicting itself. The amendment says what changed and why the report-only period ended; the original text stays, because what the decision looked like when it was taken is the part worth keeping. Also corrects llms.txt's response envelope, which had errors at the top level and pagination at meta.total. Both are nested one deeper, so anything written against that description read undefined on every error it tried to handle.
4.6 KiB
Importing a site's legacy data
Two routes in, and which one you want
If the site has a spreadsheet and no developer, use the CSV import. It is the common case, and it needs nothing beyond the templates:
flask csv templates --out csv-templates # generated from the live schema
# fill them in
flask csv import --dir csv-templates # checks only, changes nothing
flask csv import --dir csv-templates --commit
Foreign keys take a NAME, not an id - write Bay 3, not locationid=7. The
importer resolves them, including across files in the same run, and a name it
cannot find is reported with the line, the column and the value. Nothing is
written unless every row passes, and re-running an edited file updates rows
rather than duplicating them. See CSV-IMPORT.md.
If the site has a source database to read from, and someone able to script against it, the HTTP import API below is the better tool: it carries the whole history, preserves original timestamps, and handles relationships the CSV set does not model.
Every adopting site has its own source database - it will not match another site's schema. So the import is split in two layers:
- The import API is the stable contract (
docs/IMPORT-API.md). Whatever your source looks like, you create flask records through the same documented REST endpoints, authenticated with an admin PAT and theX-Import-Modeheader (which preserves legacy timestamps). This layer is the product; it is schema-agnostic. - A per-site loader is thin glue. It reads your source database and POSTs to those endpoints. Nobody runs another site's loader - you copy the pattern.
The the reference site loader in scripts/site_imports/wjf/ is reference
implementation #1. Read it alongside this guide.
The shape of a loader
harness.py- builds the app against the targetDATABASE_URL, mints an unscoped admin PAT in-process, and drives the real endpoints through the app test client withAuthorization: Bearer <pat>+X-Import-Mode: true. This exercises the same routes/authz/validation an HTTP client would, no running server needed. It also holds read-only access to the source DB and a JSONIdMapof legacy-id -> new-id crosswalks.run.py- orderedstage_*functions. Each reads a slice of the source, POSTs it, and records the crosswalk later stages resolve foreign keys against.
Post-import fixups that re-point existing assets (example:
scripts/reclassify_servers_to_network.py, servers imported as PCs moved to
network devices in place) belong in the site loader's verify stage, not in the
stable API layer.
Stage order matters
Reference/lookup tables first (so foreign keys resolve), then the entity hub, then dependents, then links:
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
-> dependents (installs, warranties, notifications, ...) -> relationships
The crosswalk is the keystone: capture every legacy id -> new id as you create rows, and resolve foreign keys through it in later stages. New autoincrement ids will not match the source's.
Producing the mapping
You do not have to hand-derive the source -> target mapping. Point the agent-assisted workflow at a source database plus this API contract and it emits a per-table mapping (source columns -> endpoint fields, transforms, what is importable vs out of scope) and a loader skeleton. That is the repeatable onboarding path.
Running (against a THROWAWAY import database)
- Build a fresh target:
flask db upgrade+flask plugin upgrade-all+flask seed permissions/settings/reference-data. Enable every bundled plugin you need (some ship disabled; a plugin's routes only register when it is enabled at app start). - Load your source dump into a scratch DB the loader can read.
- Run the loader stages in order, dry-running / spot-checking as you go.
- Verify: row-count + foreign-key-resolution audit against the source, then a UI spot-check (log in, eyeball the lists / map / a detail page).
- Only then point a real instance at the imported database.
What the reference loader demonstrates
- Fanning one legacy "machine" table out to the flask asset types (computer/machine/network/measuring-tool) by a routing rule, with the duplicate/placeholder/skip decisions applied.
- Synthesizing a natural key when the source lacks one (printers ->
PRN-{id}). - Folding a primary IP onto an asset, pairing a check-in/out event log into checkouts, deduping colliding names, reversing an inverse relationship type.
- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications endpoint) handled as documented direct-ORM writes.