ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reciprocal controls links, dead chip links, and a resumable legacy import

Browse Source 
The machine page now shows the PC that controls it. The endpoint was correct all
along and returned both directions; AssetTriageBand discarded every peer whose
type was not 'machine' and was mounted only on the PC page. The counterpart type,
label and relationship types are now props.

Fixed while adopting it more widely: every band chip linked to /assets/<id>, and
there is no such route. The resolver that AssetRelationships.vue kept privately
is now frontend/src/utils/assetRoute.js, used by both - and its type lookup is
case-insensitive, because seed data capitalises the type names while this site's
database does not.

The legacy import is now safe to resume. The 409-resolve asked for
per_page=10000 against a server that clamps to 100, so it silently searched the
first 100 rows and dropped links past that on every re-run. The crosswalk is now
bound to the database it describes, so one carried across a rebuild is refused
rather than attaching rows to the wrong assets. Source database names are
resolved from the environment, so the loader can be aimed without editing it.

Full suite 2178 passed, frontend 280 passed.
This commit is contained in:
Cameron Proudlock
2026-08-29 19:09:06 -04:00
parent a8f8580738
commit 8666e0271c
14 changed files with 701 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
scripts/site_imports/wjf/run.py
View File

@@ -9,16 +9,19 @@ Stages are idempotent (re-running resolves existing rows by natural key), so a
crashed run resumes. See scripts/site_imports/wjf/harness.py for the contract
this drives, and scratchpad/IMPORT-PLAN.md for the full mapping + decisions.
Implemented: reference (vendors, businessunits, operatingsystems), employees.
TODO stages (assets hub + dependents + network + usb) are stubbed - they need
the machineid->assetid crosswalk this harness persists.
All 17 stages are implemented; STAGES at the bottom of this file is the list
and main()'s --stages default is the order they must run in. The assets hub
comes before everything that references an asset, because it is what builds the
machineid->assetid crosswalk the harness persists.
"""
import argparse
import sys
from urllib.parse import quote
import pymysql
from .harness import Harness
from .harness import Harness, Source, StaleIdMapError
def _id_of(row, idfield):
@@ -33,6 +36,76 @@ def _id_of(row, idfield):
return None
# The server clamps any page size to MAX_PAGE_SIZE (shopdb/config.py), which
# is 100. Asking for more is not an error and not a warning - it just silently
# returns 100 rows, which is why the 409-resolve below pages instead.
PAGE_SIZE = 100
# A resolve that walks more pages than this is a bug in the caller, not a big
# table. Bounded so a paging mistake cannot spin forever against a live server.
MAX_RESOLVE_PAGES = 200
def _rows_of(payload):
"""The list out of a list-endpoint response, whichever shape it took."""
if isinstance(payload, list):
return payload
if isinstance(payload, dict):
items = payload.get('items')
if isinstance(items, list):
return items
return []
def _resolve_existing(h, lookup, unique_field, wanted, idfield):
"""Find an existing row by its natural key. Returns its id, or None.
This used to ask for ?per_page=10000 and read the single response. The
server clamps per_page to 100, so it only ever searched the first 100 rows
and returned None for everything after that. The failure was silent and
downstream: _upsert returned None, the id never reached the crosswalk, and
the run finished "successfully" having quietly dropped a model's link on
every machine that used it. The target already holds more than 100 models
and more than 100 applications, so this fired on any re-run or resume - the
exact case the loader documents as safe.
Two passes. Some endpoints expose an exact natural-key filter for this
(?modelnumber= on models, ?appname= on applications, ?vendor= on vendors);
an endpoint without one ignores the unknown query parameter and returns its
normal first page, so the paged walk that follows is what finds the row.
"""
wanted_key = str(wanted).strip().lower()
separator = '&' if '?' in lookup else '?'
def match(rows):
for row in rows:
if str(row.get(unique_field, '')).strip().lower() == wanted_key:
return _id_of(row, idfield)
return None
# Pass 1: the exact filter, where the endpoint has one.
exact = '%s%s%s=%s&per_page=%d' % (
lookup, separator, unique_field, quote(str(wanted)), PAGE_SIZE)
_, data = h.get(exact)
found = match(_rows_of(data))
if found is not None:
return found
# Pass 2: every page, in order.
for page in range(1, MAX_RESOLVE_PAGES + 1):
_, data = h.get('%s%sper_page=%d&page=%d'
% (lookup, separator, PAGE_SIZE, page))
rows = _rows_of(data)
if not rows:
return None
found = match(rows)
if found is not None:
return found
if len(rows) < PAGE_SIZE:
return None
return None
def _upsert(h, path, payload, unique_field, idfield, list_path=None):
"""POST payload; on 409 resolve the existing row by its unique field.
Returns the new/existing id, or None."""
@@ -40,14 +113,8 @@ def _upsert(h, path, payload, unique_field, idfield, list_path=None):
if status in (200, 201):
return _id_of(data, idfield)
if status == 409:
lookup = list_path or path
lookup += ('&' if '?' in lookup else '?') + 'per_page=10000'
_, rows = h.get(lookup)
items = rows if isinstance(rows, list) else rows.get('items', rows)
for row in (items or []):
if str(row.get(unique_field, '')).strip().lower() == \
str(payload[unique_field]).strip().lower():
return _id_of(row, idfield)
return _resolve_existing(h, list_path or path, unique_field,
payload[unique_field], idfield)
return None
@@ -418,10 +485,22 @@ def stage_metrology(h):
'businessunitid': h.ids.get('businessunit', m['businessunitid']),
'mapx': m['mapleft'], 'mapy': m['maptop'],
'levelid': h.defaultlevelid})
if status not in (200, 201):
if status in (200, 201):
tool_assetid = _id_of(data, 'assetid')
made += 1
elif status == 409:
# Already imported by an earlier run. Resolve it rather than
# skipping: `continue` here left the crosswalk without this tool,
# so the Controls edge below was never made and a resumed run
# produced metrology PCs driving nothing. stage_printers already
# had this branch; this stage was written later and missed it.
tool_assetid = _resolve_existing(
h, '/api/measuringtools', 'assetnumber',
f"MT-{m['machineid']}", 'assetid')
if not tool_assetid:
continue
else:
continue
tool_assetid = _id_of(data, 'assetid')
made += 1
if tool_assetid and controls_id:
h.ids.put('metrologytool', m['machineid'], tool_assetid)
st, _ = h.post('/api/assets/relationships', {
@@ -906,10 +985,27 @@ def main():
'metrology,communications,applications,warranties,notifications,'
'knowledgebase,relationships,defaultprinters,subnets,usb,verify',
help='comma list of stages to run')
parser.add_argument(
'--idmap', default=None,
help='path to the legacy-id crosswalk (default: idmap.json beside '
'this script). One per target database - a crosswalk from '
'another database maps to the wrong assets.')
args = parser.parse_args()
h = Harness()
print(f'== WJ import loader (target: {h.app.config["SQLALCHEMY_DATABASE_URI"].split("/")[-1]}) ==')
try:
h = Harness(idmap_path=args.idmap)
except StaleIdMapError as error:
print('REFUSED: %s' % error)
return 1
target = h.app.config['SQLALCHEMY_DATABASE_URI'].split('/')[-1]
print(f'== WJ import loader (target: {target}) ==')
# Say which sources this run reads. They are overridable now, so "it
# imported nothing" is usually "it read the wrong database".
for logical in ('shopdb_src', 'cmmc_usb_src', 'wjf_employees_src'):
real = Source.resolve(logical)
if real != logical:
print(f' source {logical} -> {real}')
try:
for name in args.stages.split(','):
name = name.strip()
@@ -922,9 +1018,15 @@ def main():
print(f'\n {len(h.errors)} endpoint error(s):')
for path, code, _payload, data in h.errors[:10]:
print(f' {code} {path}: {data}')
# An endpoint error is not a crash, but it did leave rows unimported,
# and a run that exits 0 after 200 of them looks like a success in a
# deploy log.
if h.errors:
return 1
finally:
h.close()
return 0
if __name__ == '__main__':
main()
sys.exit(main())