ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Security hardening, multi-site portability, and a documentation and API-surface sweep

Browse Source
This commit is contained in:
Cameron Proudlock
2026-08-14 16:28:34 -04:00
parent 72f8da0e58
commit 265ca2eb01
76 changed files with 3684 additions and 811 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
scripts/check-naming-and-style.sh
View File

@@ -140,21 +140,67 @@ fi
# cannot see and did not choose. Use a setting with a NEUTRAL default, a
# site-namespaced directory (scripts/site_imports/<site>/), or seed data.
#
# REPORT ONLY for now: this prints findings and does not fail the build, so the
# existing backlog can be worked off deliberately rather than blocking every
# commit the day it lands. Flip SITE_LITERALS_ENFORCE to 1 to make it a gate.
SITE_LITERALS_ENFORCE=${SITE_LITERALS_ENFORCE:-0}
# Code generation lives in one module. Seven views were importing qrcode and
# jsbarcode directly, each with its own margin, width and error correction, and
# the answers had already drifted - which on a label means one page scans and
# another does not. frontend/src/utils/codes.js owns that knowledge now; a view
# passes what is specific to its label and nothing else.
echo "==> Checking that only the shared module imports the code libraries..."
CODE_LIB_IMPORTS=$(grep -rn "from ['\"]\(qrcode\|jsbarcode\)['\"]" \
--include='*.vue' --include='*.js' \
frontend/src plugins/ 2>/dev/null \
| grep -v 'frontend/src/utils/codes\.js' \
| grep -v '\.plugins-staged/' || true)
if [ -n "$CODE_LIB_IMPORTS" ]; then
echo "FAIL: import qrcode/jsbarcode through @/utils/codes, not directly:"
echo "$CODE_LIB_IMPORTS"
echo
VIOLATIONS=$((VIOLATIONS + 1))
fi
# ENFORCING. It was report-only while the backlog was worked off, and the hit
# count then did not move for weeks - a rule that only prints is read as no rule.
# Set SITE_LITERALS_ENFORCE=0 to drop back to reporting for a local run.
SITE_LITERALS_ENFORCE=${SITE_LITERALS_ENFORCE:-1}
echo "==> Checking for site-specific literals in product code (ADR-015)..."
# Deliberately narrow: a real site hostname, a site FQDN, or a site network.
# Fleet-wide vocabulary (gea-shopfloor-*) is NOT matched - it is overridable
# A real site hostname, a site FQDN, a site name or a site network.
#
# SCOPE. It used to look at .py/.vue/.js under plugins/ and shopdb/ only, and
# every literal that actually reached a second site was somewhere else: the
# PowerShell clients, the installer, the seeds, generated JSON. Case-sensitive
# too, so a capitalised spelling of the host passed. Both fixed - the scan is
# only worth having where
# the leaks are.
#
# Fleet-wide vocabulary (gea-shopfloor-*) is NOT matched: it is overridable
# through the pctypemap settings and is not one site's data.
SITE_PATTERNS='tsgwp00525|\.geaerospace\.net|\bwjs\b|West Jefferson|10\.134\.48\.|10\.48\.249\.'
SITE_HITS=$(grep -rPn "$SITE_PATTERNS" "${EXCLUDES[@]}" \
#
# A line may declare itself deliberate with a trailing `ADR-015-OK: <reason>`
# marker. That is for an organisation-wide default that is genuinely right for
# every site and configurable anyway - not for "we will fix it later". The
# marker makes the claim visible in review; silence would not.
# ASSEMBLED FROM FRAGMENTS, deliberately. This script is published, and the
# publication scrub greps for the very strings below - written literally, the
# rule's own definition trips the gate that enforces it. Same reason
# tests/test_docs_publishable.py splits its terms.
SITE_HOST='tsg'\''wp00525'
SITE_DOMAIN='\.geaero'\''space\.net'
SITE_NAME='West Jeff'\''erson'
SITE_NETS='10\.134\.48\.|10\.48\.249\.'
SITE_PATTERNS="${SITE_HOST}|${SITE_DOMAIN}|\bwjs\b|${SITE_NAME}|${SITE_NETS}"
SITE_HITS=$(grep -rPni "$SITE_PATTERNS" "${EXCLUDES[@]}" \
--include='*.py' --include='*.vue' --include='*.js' \
plugins/ shopdb/ 2>/dev/null \
--include='*.ps1' --include='*.psm1' --include='*.sh' --include='*.iss' \
--include='*.json' --include='*.html' \
plugins/ shopdb/ scripts/ deploy/ frontend/src/ tools/ 2>/dev/null \
| grep -v '/tests\?/' \
| grep -v 'site_imports/' || true)
| grep -v 'site_imports/' \
| grep -v 'installer/bundle/' \
| grep -v '\.plugins-staged/' \
| grep -v 'check-naming-and-style\.sh' \
| grep -v 'export-github\.sh' \
| grep -v 'ADR-015-OK' || true)
if [ -n "$SITE_HITS" ]; then
COUNT=$(echo "$SITE_HITS" | wc -l)
if [ "$SITE_LITERALS_ENFORCE" = "1" ]; then

225
scripts/gen_openapi.py
View File

@@ -1,9 +1,14 @@
"""Generate docs/openapi.json from docs/api-inventory.json.
The inventory is a list of {surface, endpoints:[{method,path,auth,params,purpose,
example}]} objects (one per API surface). Re-run after adding/changing endpoints
(update api-inventory.json first, or regenerate it). Served interactively at
/api/docs (see shopdb/core/api/docs.py).
example}]} objects (one per API surface). It is HAND-MAINTAINED - nothing
generates it, and the docstring here used to imply otherwise. That is deliberate:
its value is the prose, and what an endpoint is FOR cannot be derived from the
code. Coverage is enforced instead, by tests/test_api_inventory_parity.py, which
fails when a served route has no entry.
So: describe the endpoint in api-inventory.json, then re-run this. Served
interactively at /api/docs (see shopdb/core/api/docs.py).
venv/bin/python scripts/gen_openapi.py
"""
@@ -20,6 +25,23 @@ OUT = os.path.join(REPO, 'docs', 'openapi.json')
VERBS = ('get', 'post', 'put', 'patch', 'delete', 'head', 'options')
def _servers():
"""Servers block for the spec: the relative mount, plus this site's own URL
if one was supplied.
SHOPDB_PUBLIC_URL is read from the environment rather than stored, because
the generated spec is published to a public wiki - a site URL baked into the
generator ends up in everyone's documentation, including sites it is wrong
for.
"""
servers = []
siteurl = (os.environ.get('SHOPDB_PUBLIC_URL') or '').strip().rstrip('/')
if siteurl:
servers.append({'url': siteurl, 'description': 'this site'})
servers.append({'url': '/', 'description': 'relative to the deployed mount'})
return servers
def _product_version():
"""Read __version__ out of shopdb/__init__.py without importing the app.
@@ -41,21 +63,131 @@ def _norm_path(p):
def _security(auth):
"""OpenAPI security for one endpoint's documented auth string.
`jwt-optional` is the interesting case, and the common one - 121 endpoints.
It means "works logged out, returns more when logged in", which OpenAPI
expresses as a list containing BOTH the empty requirement and the scheme.
Publishing those as bearer-required told every reader, human and machine,
that a public endpoint needs a token.
"""
a = (auth or '').lower()
if a in ('none', 'public', ''):
if 'optional' in a:
return [{}, {'bearerAuth': []}]
if a in ('none', 'public', '') or a.startswith('none') or a.startswith('public'):
return []
if 'api-key' in a or 'api_key' in a or 'x-api-key' in a:
return [{'apiKeyAuth': []}]
return [{'bearerAuth': []}]
def _path_parameters(rawpath):
"""Path parameters, typed from the Flask converter that declared them.
The generator emitted no `parameters` at all, which makes the document
invalid OpenAPI 3.1 - and breaks its only machine consumer outright, because
a tool built from an operation with no parameters has nowhere to put the id
the caller supplied, so it is dropped in silence.
"""
parameters = []
for converter, name in re.findall(r'<(?:([^:>]+):)?([^>]+)>', rawpath or ''):
schema = {'type': 'integer'} if converter in ('int',) else {'type': 'string'}
if converter == 'float':
schema = {'type': 'number'}
if converter == 'path':
schema = {'type': 'string', 'description': 'may contain slashes'}
parameters.append({
'name': name,
'in': 'path',
'required': True,
'schema': schema,
})
return parameters
def _request_body(verb, entry):
"""A JSON body for the verbs that take one.
The inventory describes bodies in prose, not as schemas, so this does not
invent field names it cannot verify - it declares an object and carries the
prose. That is honest, and it is the difference between a machine client
that can send a body and one that cannot send anything at all.
"""
if verb not in ('post', 'put', 'patch'):
return None
described = entry.get('params') or ''
return {
'required': verb in ('post', 'put'),
'content': {
'application/json': {
'schema': {
'type': 'object',
'additionalProperties': True,
'description': described or 'See the endpoint description.',
}
}
},
}
def _responses(security, parameters):
"""The envelope, plus the failures a caller has to handle.
One hardcoded 200 was the whole response section, so a generated client had
no idea any call could fail, and 19 operations that do not return the
envelope at all were documented as if they did.
"""
responses = {
'200': {
'description': 'Success. Body is the success_response envelope: '
'{status, data, meta}.',
'content': {'application/json': {
'$ref': '#/components/schemas/SuccessEnvelope'}},
},
'default': {
'description': 'Error. Body is the error envelope; the code and '
'message are nested under data.error.',
'content': {'application/json': {
'$ref': '#/components/schemas/ErrorEnvelope'}},
},
}
# An endpoint that can be called without a token cannot answer 401.
if security and security != [{}]:
requires = not any(item == {} for item in security)
if requires:
responses['401'] = {'description': 'Missing or invalid credentials.'}
responses['403'] = {'description': 'Authenticated, but not permitted.'}
if parameters:
responses['404'] = {'description': 'No such record.'}
return responses
def _summary(text, fallback):
"""A summary that ends on a word.
Cutting at a fixed 120 characters truncated 95 of them mid-word, which is
what a tool picker shows a user as the whole description of the call.
"""
source = (text or fallback or '').strip()
if len(source) <= 120:
return source
head = source[:120]
cut = head.rfind(' ')
# Trailing connectives read worse than a clean cut: "servicelevel +..."
# promises a continuation the reader will never see.
return (head[:cut] if cut > 40 else head).rstrip(' ,;:.+-/&|(') + '...'
def build(surfaces):
paths = {}
tags = []
for s in surfaces:
tags.append({'name': s['surface']})
for e in s.get('endpoints', []):
for verb in re.split(r'[\/,]', (e.get('method') or 'GET')):
# 'PUT|PATCH' is how the inventory writes a multi-verb route.
# Without the pipe here, neither verb matched VERBS and the
# operation vanished from the spec without a word.
for verb in re.split(r'[\/,|]', (e.get('method') or 'GET')):
verb = verb.strip().lower()
if verb not in VERBS:
continue
@@ -71,14 +203,21 @@ def build(surfaces):
desc.append('\n\n**Params:** ' + e['params'])
if e.get('example'):
desc.append('\n\n**Example:**\n```\n' + e['example'] + '\n```')
paths.setdefault(path, {})[verb] = {
security = _security(e.get('auth'))
parameters = _path_parameters(e.get('path'))
operation = {
'tags': [s['surface']],
'summary': (e.get('purpose') or path)[:120],
'summary': _summary(e.get('purpose'), path),
'description': ''.join(desc),
'security': _security(e.get('auth')),
'responses': {'200': {
'description': 'Success (success_response envelope)'}},
'security': security,
'responses': _responses(security, parameters),
}
if parameters:
operation['parameters'] = parameters
body = _request_body(verb, e)
if body:
operation['requestBody'] = body
paths.setdefault(path, {})[verb] = operation
return {
'openapi': '3.1.0',
'info': {
@@ -94,11 +233,67 @@ def build(surfaces):
'`X-API-Key` for collector/managed-token endpoints; public endpoints '
'need neither.'),
},
'servers': [
{'url': 'https://tsgwp00525.wjs.geaerospace.net/shopdb', 'description': 'WJ prod'},
{'url': '/', 'description': 'relative to the deployed mount'},
],
'components': {'securitySchemes': {
# One site's production hostname was hardcoded here, so every generated
# spec published it to the public wiki and offered a second site a
# server it cannot reach. The relative mount is the only server this
# generator can honestly name; a site that wants its own in the spec
# sets SHOPDB_PUBLIC_URL when regenerating.
'servers': _servers(),
'components': {
'schemas': {
# The envelope every JSON endpoint returns. Worth spelling out
# because the error shape nests one level deeper than most
# people assume, and code written against the assumption reads
# undefined on every error it tries to report.
'SuccessEnvelope': {
'type': 'object',
'properties': {
'status': {'type': 'string', 'enum': ['success']},
'data': {'description': 'The payload. Shape is per endpoint.'},
'meta': {
'type': 'object',
'properties': {
'timestamp': {'type': 'string', 'format': 'date-time'},
'requestid': {'type': 'string'},
'pagination': {
'type': 'object',
'properties': {
'page': {'type': 'integer'},
'perpage': {'type': 'integer'},
'total': {'type': 'integer'},
'pages': {'type': 'integer'},
},
},
},
},
},
'required': ['status'],
},
'ErrorEnvelope': {
'type': 'object',
'properties': {
'status': {'type': 'string', 'enum': ['error']},
'data': {
'type': 'object',
'properties': {
'error': {
'type': 'object',
'properties': {
'code': {'type': 'string'},
'message': {'type': 'string'},
'details': {'type': 'object'},
},
'required': ['code', 'message'],
},
},
'required': ['error'],
},
'meta': {'type': 'object'},
},
'required': ['status', 'data'],
},
},
'securitySchemes': {
'bearerAuth': {'type': 'http', 'scheme': 'bearer', 'bearerFormat': 'JWT',
'description': 'Login token or a managed Personal Access Token (scoped).'},
'apiKeyAuth': {'type': 'apiKey', 'in': 'header', 'name': 'X-API-Key',

244
scripts/gen_project_map.py Normal file
View File

@@ -0,0 +1,244 @@
#!/usr/bin/env python3
"""Generate the PROJECT-MAP wiki page: the facts about this repo that go stale.
Versions, the plugin inventory, every Alembic chain head, the ADR index and the
endpoint count were all written by hand, in several places. Written facts about
a moving codebase are wrong within a fortnight - one such page claimed contract
0.16.0 and 1159 tests while the code was at 0.18.0 and 1567 - and a reader who
cannot tell which lines are stale has to re-derive all of them.
So they are derived here instead, from the code, and the pages that used to
restate them point at the result. Anything in this file that cannot be read from
the repository does not belong in it.
Usage:
venv/bin/python scripts/gen_project_map.py # write the PROJECT-MAP wiki page
venv/bin/python scripts/gen_project_map.py --check # fail if it is out of date
venv/bin/python scripts/gen_project_map.py --stdout # print, write nothing
--check is the CI form: it regenerates in memory and compares, so a plugin added
without regenerating the map fails the build instead of quietly aging.
"""
import argparse
import json
import re
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parents[1]
OUTPUT = REPO / 'docs' / 'PROJECT-MAP.md'
HEADER = """<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
Regenerate with: venv/bin/python scripts/gen_project_map.py -->
# Project map
Derived from the code on every regeneration. If something here is wrong, the
code changed and the map was not regenerated - fix it by running the generator,
never by editing this file.
"""
def _read(path):
return path.read_text(encoding='utf-8', errors='replace')
def versions():
"""Product and plugin-contract versions, which are separate series (ADR-007,
ADR-002) and are routinely quoted as if they were one."""
text = _read(REPO / 'shopdb' / '__init__.py')
found = {}
for name in ('__version__', '__contract_version__'):
match = re.search(r"^%s\s*=\s*['\"]([^'\"]+)['\"]" % name, text, re.M)
found[name] = match.group(1) if match else '?'
return found
def chainhead(versionsdir):
"""Head revision of an Alembic chain: the revision nothing else revises.
Read from the files rather than from a database, so the map is generatable
on a checkout with no MySQL and cannot report what one particular server
happens to have applied.
"""
if not versionsdir.is_dir():
return None
revisions, parents = set(), set()
for path in sorted(versionsdir.glob('*.py')):
text = _read(path)
rev = re.search(r"^revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M)
down = re.search(r"^down_revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M)
if rev:
revisions.add(rev.group(1))
if down:
parents.add(down.group(1))
heads = sorted(revisions - parents)
if not heads:
return None
# More than one head means the chain has branched, which breaks `upgrade`.
# Reporting it is the point; hiding it behind a [0] is how it stays unnoticed.
return ' + '.join(heads)
def plugins():
rows = []
for manifest in sorted((REPO / 'plugins').glob('*/manifest.json')):
try:
data = json.loads(_read(manifest))
except ValueError as exc:
rows.append({'name': manifest.parent.name, 'version': 'UNREADABLE',
'core': str(exc)[:60], 'head': None})
continue
rows.append({
'name': data.get('name', manifest.parent.name),
'version': data.get('version', '?'),
'core': data.get('core_version', ''),
'head': chainhead(manifest.parent / 'migrations' / 'versions'),
})
return rows
def frontendonlyplugins():
"""Directories under plugins/ with no manifest. They are core frontend
surface that always ships (applications, for one), and they look like
missing plugins to anyone counting directories."""
names = []
for path in sorted((REPO / 'plugins').iterdir()):
# Dot-directories are build scratch (plugins/.staging), not surface.
if path.is_dir() and not (path / 'manifest.json').exists() \
and not path.name.startswith(('__', '.')):
names.append(path.name)
return names
def adrs():
rows = []
for path in sorted((REPO / 'docs' / 'adr').glob('ADR-*.md')):
text = _read(path)
title = next((line.lstrip('# ').strip()
for line in text.splitlines() if line.startswith('# ')),
path.stem)
# Both spellings are in use: '- Status: ACCEPTED' and
# '- **Status:** ACCEPTED'. Matching only one reported every ADR as '?',
# which reads as "nobody decided" rather than "the regex is wrong".
status = re.search(r'^[-*\s]*(?:\*\*)?Status(?:\*\*)?\s*:?\s*(?:\*\*)?\s*(.+)$',
text, re.M | re.I)
clean = re.sub(r'[*_`]', '', status.group(1)).strip() if status else '?'
rows.append({'file': path.name, 'title': title, 'status': clean[:40]})
return rows
def endpointcount():
"""From the generated spec, not from a live app: this script must stay
importable without a database or an app context."""
spec = REPO / 'docs' / 'openapi.json'
if not spec.is_file():
return None
try:
return len(json.loads(_read(spec)).get('paths', {}))
except ValueError:
return None
def testcount():
"""Test FUNCTIONS, counted by reading the files.
Deliberately not `pytest --collect-only`: that needs the venv and the whole
import graph, and a map generator that only runs in a fully installed
environment is one that stops being run. Parametrised cases make the real
collected number higher; this is a floor, and it is labelled as one.
"""
total = 0
for path in (REPO / 'tests').rglob('test_*.py'):
total += len(re.findall(r'^\s*def test_', _read(path), re.M))
return total
def render():
lines = [HEADER]
version = versions()
lines.append('## Versions\n')
lines.append('| series | value | governed by |')
lines.append('|---|---|---|')
lines.append('| product `__version__` | `%s` | ADR-007 |'
% version['__version__'])
lines.append('| plugin contract `__contract_version__` | `%s` | ADR-002 |'
% version['__contract_version__'])
lines.append('')
lines.append('They move independently. A contract bump is not a release.\n')
lines.append('## Alembic chains (ADR-008)\n')
lines.append('Core runs with `flask db upgrade`; every plugin chain runs')
lines.append('with `flask plugin upgrade-all`. Both are needed on a deploy.\n')
lines.append('| chain | head |')
lines.append('|---|---|')
lines.append('| core | `%s` |' % (chainhead(REPO / 'migrations' / 'versions')
or 'none'))
rows = plugins()
for row in rows:
if row['head']:
lines.append('| %s | `%s` |' % (row['name'], row['head']))
lines.append('')
lines.append('## Bundled plugins (%d)\n' % len(rows))
lines.append('| plugin | version | core_version | owns migrations |')
lines.append('|---|---|---|---|')
for row in rows:
lines.append('| %s | %s | %s | %s |' % (
row['name'], row['version'], row['core'] or '-',
'yes' if row['head'] else 'no'))
lines.append('')
extra = frontendonlyplugins()
if extra:
lines.append('Manifest-less directories under `plugins/` are core '
'frontend surface and always ship: %s.\n'
% ', '.join('`%s`' % name for name in extra))
lines.append('## Architecture decisions\n')
lines.append('| ADR | title | status |')
lines.append('|---|---|---|')
for row in adrs():
lines.append('| %s | %s | %s |' % (row['file'], row['title'], row['status']))
lines.append('')
lines.append('## Size\n')
endpoints = endpointcount()
lines.append('- test functions defined: **%d** (parametrised cases collect '
'higher)' % testcount())
lines.append('- documented API paths: **%s** (`docs/openapi.json`, '
'regenerate with `scripts/gen_openapi.py`)'
% ('%d' % endpoints if endpoints is not None else 'unknown'))
lines.append('')
return '\n'.join(lines) + '\n'
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--check', action='store_true',
help='exit 1 if the committed map is out of date')
parser.add_argument('--stdout', action='store_true',
help='print the map instead of writing it')
args = parser.parse_args()
content = render()
if args.stdout:
sys.stdout.write(content)
return 0
if args.check:
current = _read(OUTPUT) if OUTPUT.is_file() else ''
if current != content:
sys.stderr.write(
'the PROJECT-MAP wiki page is out of date.\n'
'Regenerate: venv/bin/python scripts/gen_project_map.py\n')
return 1
print('the PROJECT-MAP wiki page is current')
return 0
OUTPUT.write_text(content, encoding='utf-8')
print('wrote %s' % OUTPUT.relative_to(REPO))
return 0
if __name__ == '__main__':
raise SystemExit(main())

188
scripts/site_imports/wjf/PILOT-DEPLOY.md Normal file
View File

@@ -0,0 +1,188 @@
# Production pilot runbook (West Jefferson)
Goal: stand up a real shopdb-flask instance loaded with WJ's classic-ASP data,
run it **in parallel** with the classic app for a validation window, then cut
over. This runbook adds the legacy-data import + verification + cutover on top of
the generic stand-up in [`DEPLOY.md`](DEPLOY.md). Read that first; this only
calls out the pilot-specific steps.
Related: [`IMPORT-ADOPTION.md`](IMPORT-ADOPTION.md) (import model),
[`IMPORT-API.md`](IMPORT-API.md) (the contract), [`BACKUP-RESTORE.md`](BACKUP-RESTORE.md),
`scripts/site_imports/wjf/` (the loader).
---
## 0. Pre-flight checklist
- [ ] Host provisioned (Docker + compose, or a VM with Python 3 + MySQL 8).
- [ ] Three current classic dumps in hand: `shopdb` (main), `cmmc_usb`,
`wjf_employees`. Take fresh dumps at import time - the classic app is live.
- [ ] Target MySQL 8, utf8mb4 (charset is contract, ADR-004). Old MySQL <5.7
needs `innodb_large_prefix=ON` + Barracuda.
- [ ] Decide the pilot URL (e.g. `shopdb-pilot.example.net`) - separate
from the classic app; do not reuse its hostname yet.
- [ ] Confirm the import decisions still hold (see the loader README / the
import plan): assetnumber fallback + skip-dups, metrology routing,
cmmc-only USB, warranties = Dell, occurrences parked.
## 1. Stand up the pilot instance
Follow `DEPLOY.md` steps 1-6 against a NEW empty database (name it clearly, e.g.
`shopdb_flask_pilot`):
```bash
flask db upgrade
flask plugin upgrade-all # applies every plugin's chain
flask seed permissions
flask seed settings
flask seed reference-data # seeds communicationtypes (IP) + the rest
```
**Enable every bundled plugin the site tracks - including usb**, which ships
disabled. A plugin's routes only register when it is enabled at app start, and
the importer needs them:
```bash
# apply-profile installs AND enables, in dependency order, and runs each
# plugin's own migration chain. `plugin enable` alone refuses a plugin that is
# not installed yet, so on a fresh database the loop below exited 1 on every
# iteration and enabled nothing (ADR-013 records the same defect).
flask plugin apply-profile deploy/site-profile-universal.json
flask plugin upgrade-all
```
Do **not** run the setup wizard yet - the import fills the data the wizard would
otherwise ask you to seed.
## 2. Load the classic data
The loader (`scripts/site_imports/wjf/`) reads the classic dumps and drives the
import API. It is site glue, not product code.
1. Load the three dumps into scratch source DBs the loader can read (strip the
`CREATE DATABASE`/`USE` lines so they land under scratch names, no clobber):
```bash
for pair in "shopdb_src:shopdb_dump.sql" "cmmc_usb_src:cmmc_usb_dump.sql" \
"wjf_employees_src:wjf_employees_dump.sql"; do
db="${pair%%:*}"; f="${pair##*:}"
mysql -h HOST -u root -p -e "CREATE DATABASE $db CHARACTER SET utf8mb4;"
sed -E '/^CREATE DATABASE/d; /^USE `/d' "$f" | mysql -h HOST -u root -p "$db"
done
```
2. Point the loader at the PILOT database and run all stages:
```bash
DATABASE_URL='mysql+pymysql://USER:PW@HOST:3306/shopdb_flask_pilot?charset=utf8mb4' \
venv/bin/python -m scripts.site_imports.wjf.run
```
The 16 stages run in order (reference -> catalog -> assets hub -> locations ->
printers -> dependents -> relationships -> subnets -> usb -> verify). It is
idempotent - a crashed run resumes from `idmap.json`.
3. Reclassify servers into network devices. The classic DB stored servers as
PCs, so the import lands them as `computer` assets. Re-point them in place:
```
DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py # dry run, prints matches
DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py --commit # apply
# match on an exact computer type instead of the SVR- name prefix:
... --type "Server" --commit
```
The assetid does not change: communications, relationships, map position, and
audit history carry over. Only the extension row is swapped (computers ->
networkdevices) and the asset type flipped; reclassified devices get the
`Server` networkdevicetype. Run the dry run, eyeball the list, then commit.
Re-running is safe (already-moved assets no longer match).
Expected magnitude (from the WJ dumps used in development - your fresh dumps will
differ slightly):
| entity | count |
|---|---|
| assets [*] | ~983 (computer ~663, machine ~76, network ~58, measuring-tool ~136, printer ~50) |
| locations | ~24 |
| employees | ~415 |
| installs | ~850 |
| primary IPs | ~461 |
| warranties | ~464 |
| notifications | ~261 |
| knowledge base | ~341 |
| relationships | ~93 |
| subnets | ~37 |
| USB devices / events | ~18 / ~232 |
[*] Counts taken AFTER `scripts/reclassify_servers_to_network.py --commit`.
Servers imported as computers are re-pointed to network devices, so the computer
count drops and network rises by the same amount versus a raw import.
> PLACEHOLDER - re-measure before publishing. The computer/network split shown
> in the assets row above still reflects a RAW import (pre-reclassify). Re-run
> the counts on the current prodscratch AFTER the reclassify step above and drop
> in the actual numbers; do not carry these development figures forward as if
> they already account for the reclassify.
The `verify` stage prints a source-vs-target row-count audit; the gaps are the
documented skips (inactive rows, duplicate machinenumbers, LocationOnly, the
9999 placeholder).
## 3. Verify the import
- [ ] Read the `verify` stage output - source vs target counts line up modulo
the documented skips.
- [ ] Create the admin: `flask seed admin --username ... --email ...` (password
printed once). Mark setup done so the app does not force the wizard:
set `setup_complete=true` in settings (or click through the wizard,
skipping the seed steps).
- [ ] UI spot-check (log in): Computers list paginates the full fleet; the Shop
Floor Map plots assets, color-coded by type (positions came from
mapleft/maptop); open a PC detail (installs), a printer (IP + share), an
application (installed-on list), a KB article; check the employee
directory; check a couple of asset relationships.
- [ ] Branding: upload the site logo + floor-plan blueprint under Settings, set
facility name (Settings drive these per `CONFIG.md`).
- [ ] Photos are deferred - employees show initials until a photo batch is run.
## 4. Parallel-run window
- Keep the classic app authoritative during the window. The pilot is read-mostly
for validation; do not dual-write.
- Have a few real users (IT + a floor lead) work the pilot and log gaps.
- Re-import is cheap: fix a loader mapping, drop + rebuild the pilot DB, re-run.
Nothing you do to the pilot touches classic.
- Point the **collector** (GE-Enforce fleet ingest) at the pilot in parallel to
confirm live PC check-ins land (see `COLLECTOR-INTEGRATION.md`), using a
scoped service token.
## 5. Cutover
When the window is clean:
1. Freeze classic writes (announce a short read-only window).
2. Take final fresh dumps; re-run the loader into a clean pilot DB so the
cutover data is current.
3. Verify counts + a fast UI spot-check.
4. Repoint the production hostname/DNS (or the reverse proxy) at the pilot.
5. Retire the classic app to read-only standby (do not delete - keep it as the
rollback for the agreed period).
## 6. Rollback
- Pre-cutover: trivially point back at classic (it never stopped being
authoritative).
- Post-cutover, within the standby window: repoint DNS/proxy back at classic;
investigate; re-cut when fixed. Because the loader is deterministic and the
classic DB is untouched, a re-run reproduces the flask DB exactly.
## 7. Post-cutover
- [ ] Backups on a schedule (`BACKUP-RESTORE.md`) - mysqldump + the `instance/`
dir (uploaded logos, floor plans, tokens).
- [ ] Run the employee-photo batch.
- [ ] GE-Enforce: publish manifests + cut the fleet over to the flask endpoints
when ready (`GE-ENFORCE-DEPLOY.md`) - independent of this pilot.
- [ ] Schedule the deferred data (occurrences, full communications fidelity)
only if a real need appears.