Make the legacy import safe to resume, and aimable without editing it
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed.
This commit is contained in:
cproudlock
@@ -14,6 +14,7 @@ crashed run resumes and later stages resolve foreign keys.
|
||||
"""
|
||||
|
||||
import json
|
||||
import uuid
|
||||
import os
|
||||
import pymysql
|
||||
|
||||
@@ -30,26 +31,63 @@ def _source_password():
|
||||
with open(os.path.abspath(env)) as handle:
|
||||
for line in handle:
|
||||
if line.startswith('DATABASE_URL='):
|
||||
# Not every server runs as root, and 'root:' then simply is not
|
||||
# there: split()[1] raised IndexError, which read as a crash
|
||||
# rather than the clear message two lines down.
|
||||
if 'root:' not in line:
|
||||
continue
|
||||
after = line.split('root:', 1)[1]
|
||||
return after.split('@', 1)[0]
|
||||
raise RuntimeError('could not resolve source DB password')
|
||||
|
||||
|
||||
class Source:
|
||||
"""Read-only access to the three scratch source databases."""
|
||||
"""Read-only access to the three source databases.
|
||||
|
||||
def __init__(self, host='127.0.0.1', port=3306, user='root', password=None):
|
||||
The stage functions name a source by its LOGICAL name - 'shopdb_src',
|
||||
'cmmc_usb_src', 'wjf_employees_src' - and this class resolves each to a
|
||||
real database. That indirection is the point: the names were hardcoded at
|
||||
29 call sites, so pointing the loader at a differently-named copy (the
|
||||
prodscratch restore, or a site's own dump) meant editing run.py, which is
|
||||
not something anybody should do to aim a loader. The logical names stay in
|
||||
the stages where they read as documentation; the mapping lives here.
|
||||
|
||||
Everything is environment-overridable, defaulting to the scratch layout:
|
||||
|
||||
SOURCE_DB_MAIN default shopdb_src
|
||||
SOURCE_DB_USB default cmmc_usb_src
|
||||
SOURCE_DB_EMPLOYEES default wjf_employees_src
|
||||
SOURCE_DB_HOST/PORT/USER/PASSWORD
|
||||
"""
|
||||
|
||||
# logical name -> (env var, default real name)
|
||||
ALIASES = {
|
||||
'shopdb_src': ('SOURCE_DB_MAIN', 'shopdb_src'),
|
||||
'cmmc_usb_src': ('SOURCE_DB_USB', 'cmmc_usb_src'),
|
||||
'wjf_employees_src': ('SOURCE_DB_EMPLOYEES', 'wjf_employees_src'),
|
||||
}
|
||||
|
||||
def __init__(self, host=None, port=None, user=None, password=None):
|
||||
self.password = password or _source_password()
|
||||
self.host, self.port, self.user = host, port, user
|
||||
self.host = host or os.environ.get('SOURCE_DB_HOST', '127.0.0.1')
|
||||
self.port = int(port or os.environ.get('SOURCE_DB_PORT', 3306))
|
||||
self.user = user or os.environ.get('SOURCE_DB_USER', 'root')
|
||||
self._conns = {}
|
||||
|
||||
@classmethod
|
||||
def resolve(cls, name):
|
||||
"""The real database behind a logical source name."""
|
||||
env, default = cls.ALIASES.get(name, (None, name))
|
||||
return os.environ.get(env, default) if env else name
|
||||
|
||||
def db(self, name):
|
||||
if name not in self._conns:
|
||||
self._conns[name] = pymysql.connect(
|
||||
real = self.resolve(name)
|
||||
if real not in self._conns:
|
||||
self._conns[real] = pymysql.connect(
|
||||
host=self.host, port=self.port, user=self.user,
|
||||
password=self.password, database=name, charset='utf8mb4',
|
||||
password=self.password, database=real, charset='utf8mb4',
|
||||
cursorclass=pymysql.cursors.DictCursor)
|
||||
return self._conns[name]
|
||||
return self._conns[real]
|
||||
|
||||
def rows(self, dbname, sql, args=None):
|
||||
with self.db(dbname).cursor() as cur:
|
||||
@@ -57,15 +95,62 @@ class Source:
|
||||
return cur.fetchall()
|
||||
|
||||
|
||||
class IdMap:
|
||||
"""Persisted legacy-id -> new-id crosswalks, one namespace per entity."""
|
||||
class StaleIdMapError(RuntimeError):
|
||||
"""The crosswalk on disk was built against a different target database."""
|
||||
|
||||
def __init__(self, path):
|
||||
|
||||
class IdMap:
|
||||
"""Persisted legacy-id -> new-id crosswalks, one namespace per entity.
|
||||
|
||||
BOUND TO ONE TARGET DATABASE. The crosswalk holds this site's legacy ids
|
||||
mapped to assetids in a specific database, and those assetids mean nothing
|
||||
anywhere else. Two ways that used to go wrong, both silent:
|
||||
|
||||
- Delete idmap.json and re-run: every create 409s, nothing resolves into
|
||||
the crosswalk, and warranties, installs, relationships and printer
|
||||
assignments all quietly import as ~0 rows.
|
||||
- Keep idmap.json across a drop-and-rebuild of the target: the legacy ids
|
||||
now point at assetids belonging to the OLD database, so warranties and
|
||||
printer assignments attach to whatever asset happens to hold that id
|
||||
now. Wrong data, no error, and the runbook's two "drop and rebuild"
|
||||
steps never said to delete the file.
|
||||
|
||||
The bind is a marker row in the target's own settings table, so it dies
|
||||
with the data: a rebuild produces a new marker, and the mismatch is caught
|
||||
before a single row is written. The database NAME is not enough on its own,
|
||||
because a rebuild keeps it.
|
||||
"""
|
||||
|
||||
MARKER_KEY = 'siteimport_targetinstance'
|
||||
|
||||
def __init__(self, path, target=None):
|
||||
self.path = path
|
||||
self.target = target
|
||||
self._data = {}
|
||||
if os.path.exists(path):
|
||||
with open(path) as handle:
|
||||
self._data = json.load(handle)
|
||||
self._check_target()
|
||||
|
||||
def _check_target(self):
|
||||
if not self.target:
|
||||
return
|
||||
seen = self._data.get('_target')
|
||||
if seen is None:
|
||||
# Written before this check existed. Adopt it rather than refuse:
|
||||
# refusing would strand a crosswalk that is probably fine, and the
|
||||
# next run is bound properly.
|
||||
self._data['_target'] = self.target
|
||||
self.save()
|
||||
return
|
||||
if seen != self.target:
|
||||
raise StaleIdMapError(
|
||||
'%s was built against target %s, but this run targets %s.\n'
|
||||
'Those assetids belong to a different database and using them '
|
||||
'would attach imported rows to the wrong assets.\n'
|
||||
'Delete the file to start a fresh crosswalk, or point '
|
||||
'--idmap at the one for this target.'
|
||||
% (self.path, seen, self.target))
|
||||
|
||||
def put(self, namespace, legacy, new):
|
||||
self._data.setdefault(namespace, {})[str(legacy)] = new
|
||||
@@ -78,7 +163,8 @@ class IdMap:
|
||||
return str(legacy) in self._data.get(namespace, {})
|
||||
|
||||
def count(self, namespace):
|
||||
return len(self._data.get(namespace, {}))
|
||||
entry = self._data.get(namespace, {})
|
||||
return len(entry) if isinstance(entry, dict) else 0
|
||||
|
||||
def save(self):
|
||||
with open(self.path, 'w') as handle:
|
||||
@@ -97,11 +183,38 @@ class Harness:
|
||||
self._ensure_asset_types()
|
||||
self.secret = self._mint_admin_pat()
|
||||
default = os.path.join(os.path.dirname(__file__), 'idmap.json')
|
||||
self.ids = IdMap(idmap_path or default)
|
||||
self.ids = IdMap(idmap_path or default, target=self._target_instance())
|
||||
self.source = Source()
|
||||
self.errors = []
|
||||
self._defaultlevelid = None
|
||||
|
||||
def _target_instance(self):
|
||||
"""A stable id for THIS target database, created once and stored in it.
|
||||
|
||||
Kept in the target's own settings table so it dies when the data does.
|
||||
A drop-and-rebuild yields a new id, which is what lets IdMap catch a
|
||||
crosswalk carried over from the previous database - the failure that
|
||||
silently attaches imported rows to whatever asset now holds that id.
|
||||
|
||||
The database name alone would not do: a rebuild keeps the name.
|
||||
"""
|
||||
from shopdb.core.models import Setting
|
||||
existing = Setting.query.filter_by(key=IdMap.MARKER_KEY).first()
|
||||
if existing and existing.value:
|
||||
return existing.value
|
||||
marker = uuid.uuid4().hex
|
||||
if existing:
|
||||
existing.value = marker
|
||||
else:
|
||||
db.session.add(Setting(
|
||||
key=IdMap.MARKER_KEY, value=marker, valuetype='string',
|
||||
category='import',
|
||||
description='Identifies this database to the site-import '
|
||||
'crosswalk. Created by the loader; do not copy it '
|
||||
'between databases.'))
|
||||
db.session.commit()
|
||||
return marker
|
||||
|
||||
@property
|
||||
def defaultlevelid(self):
|
||||
"""The level imported map positions belong to (ADR-017).
|
||||
|
||||
Reference in New Issue
Block a user