4995456136
The publishability gate caught internal tooling names and developer paths but nothing site-specific, so roughly sixty leaks reached the wiki: the site name in ten documents, real fleet hostnames in the collector and GE-Enforce examples, an internal database name through the whole import guide, imaging-share paths, and a maintainer's username as the Deciders line of every ADR and inside a generated curl example. None of it is a security matter on an air-gapped fleet. It matters because these pages are read by engineers at other plants, and a document that names one site throughout reads as that site's notes rather than a product's documentation - which is exactly what it then gets treated as. Examples now use neutral hostnames, the site is "the reference site" where the distinction carries meaning, and ADRs are decided by "ShopDB maintainers". The gate carries all of these patterns, so the next one fails a build. Two documents leave docs/ because they were never written for an outside reader. PROJECT-REVIEW.md is an internal health memo pinned to a commit from July, whose headline finding (an untracked playbook) has since been fixed - it is history, and git holds it. PILOT-DEPLOY.md is one site's own cutover runbook, complete with a "re-measure before publishing" placeholder; it moves next to the loader it belongs to, in scripts/site_imports/wjf/. ADR-015 is AMENDED rather than rewritten. Its enforcement section still said report-only and its backlog still listed hardcodes that are now cleared, which left the record contradicting itself. The amendment says what changed and why the report-only period ended; the original text stays, because what the decision looked like when it was taken is the part worth keeping. Also corrects llms.txt's response envelope, which had errors at the top level and pagination at meta.total. Both are nested one deeper, so anything written against that description read undefined on every error it tried to handle.
102 lines
4.6 KiB
Markdown
102 lines
4.6 KiB
Markdown
# Importing a site's legacy data
|
|
|
|
## Two routes in, and which one you want
|
|
|
|
**If the site has a spreadsheet and no developer**, use the CSV import. It is
|
|
the common case, and it needs nothing beyond the templates:
|
|
|
|
```bash
|
|
flask csv templates --out csv-templates # generated from the live schema
|
|
# fill them in
|
|
flask csv import --dir csv-templates # checks only, changes nothing
|
|
flask csv import --dir csv-templates --commit
|
|
```
|
|
|
|
Foreign keys take a NAME, not an id - write `Bay 3`, not `locationid=7`. The
|
|
importer resolves them, including across files in the same run, and a name it
|
|
cannot find is reported with the line, the column and the value. Nothing is
|
|
written unless every row passes, and re-running an edited file updates rows
|
|
rather than duplicating them. See [CSV-IMPORT.md](CSV-IMPORT.md).
|
|
|
|
**If the site has a source database to read from**, and someone able to script
|
|
against it, the HTTP import API below is the better tool: it carries the whole
|
|
history, preserves original timestamps, and handles relationships the CSV set
|
|
does not model.
|
|
|
|
---
|
|
|
|
Every adopting site has its own source database - it will not match another
|
|
site's schema. So the import is split in two layers:
|
|
|
|
1. **The import API is the stable contract** (`docs/IMPORT-API.md`). Whatever
|
|
your source looks like, you create flask records through the same documented
|
|
REST endpoints, authenticated with an admin PAT and the `X-Import-Mode`
|
|
header (which preserves legacy timestamps). This layer is the product; it is
|
|
schema-agnostic.
|
|
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
|
|
to those endpoints. Nobody runs another site's loader - you copy the pattern.
|
|
|
|
The the reference site loader in `scripts/site_imports/wjf/` is reference
|
|
implementation #1. Read it alongside this guide.
|
|
|
|
## The shape of a loader
|
|
|
|
- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
|
|
unscoped admin PAT in-process, and drives the real endpoints through the app
|
|
test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
|
|
exercises the same routes/authz/validation an HTTP client would, no running
|
|
server needed. It also holds read-only access to the source DB and a JSON
|
|
`IdMap` of legacy-id -> new-id crosswalks.
|
|
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
|
|
POSTs it, and records the crosswalk later stages resolve foreign keys against.
|
|
|
|
Post-import fixups that re-point existing assets (example:
|
|
`scripts/reclassify_servers_to_network.py`, servers imported as PCs moved to
|
|
network devices in place) belong in the site loader's verify stage, not in the
|
|
stable API layer.
|
|
|
|
### Stage order matters
|
|
|
|
Reference/lookup tables first (so foreign keys resolve), then the entity hub,
|
|
then dependents, then links:
|
|
|
|
```
|
|
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
|
|
-> dependents (installs, warranties, notifications, ...) -> relationships
|
|
```
|
|
|
|
The **crosswalk is the keystone**: capture every legacy id -> new id as you
|
|
create rows, and resolve foreign keys through it in later stages. New
|
|
autoincrement ids will not match the source's.
|
|
|
|
## Producing the mapping
|
|
|
|
You do not have to hand-derive the source -> target mapping. Point the
|
|
agent-assisted workflow at a source database plus this API contract and it emits
|
|
a per-table mapping (source columns -> endpoint fields, transforms, what is
|
|
importable vs out of scope) and a loader skeleton. That is the repeatable
|
|
onboarding path.
|
|
|
|
## Running (against a THROWAWAY import database)
|
|
|
|
1. Build a fresh target: `flask db upgrade` + `flask plugin upgrade-all` +
|
|
`flask seed permissions/settings/reference-data`. Enable every bundled plugin
|
|
you need (some ship disabled; a plugin's routes only register when it is
|
|
enabled at app start).
|
|
2. Load your source dump into a scratch DB the loader can read.
|
|
3. Run the loader stages in order, dry-running / spot-checking as you go.
|
|
4. Verify: row-count + foreign-key-resolution audit against the source, then a
|
|
UI spot-check (log in, eyeball the lists / map / a detail page).
|
|
5. Only then point a real instance at the imported database.
|
|
|
|
## What the reference loader demonstrates
|
|
|
|
- Fanning one legacy "machine" table out to the flask asset types
|
|
(computer/machine/network/measuring-tool) by a routing rule, with the
|
|
duplicate/placeholder/skip decisions applied.
|
|
- Synthesizing a natural key when the source lacks one (printers -> `PRN-{id}`).
|
|
- Folding a primary IP onto an asset, pairing a check-in/out event log into
|
|
checkouts, deduping colliding names, reversing an inverse relationship type.
|
|
- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications
|
|
endpoint) handled as documented direct-ORM writes.
|