ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Browse Source 
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
This commit is contained in:
cproudlock
2026-08-28 16:17:17 -04:00
parent 93ca7587ce
commit 784964c907
149 changed files with 17545 additions and 627 deletions
Download Patch File Download Diff File Expand all files Collapse all files

363
shopdb/core/services/csvimport.py
View File

@@ -1,7 +1,7 @@
"""Bulk-load a site's starting data from CSV files.
WHY THIS EXISTS. Adopting a site means getting its existing asset register into
ShopDB. The HTTP import surface (docs/IMPORT-API.md) is the right tool when
ShopDB. The HTTP import surface (the IMPORT-API wiki page) is the right tool when
there is a source system to read from and someone able to script against it. A
sister site with a spreadsheet and no developer needs something else, and that
is the common case.
@@ -34,6 +34,7 @@ not at all.
from __future__ import annotations
import csv
import difflib
import io
import re
from collections import OrderedDict
@@ -73,6 +74,74 @@ IMPORTABLE = OrderedDict([
('machines', {'label': None, 'natural': 'assetid'}),
])
# A small, INTERNALLY CONSISTENT worked example per table, in import order. A
# site can download the set, see the pattern, and keep it as a starter workbook:
# every name referenced by a later file (Bay 3, Dell, Computer, ...) is defined
# in an earlier one, so the whole set imports cleanly as a demo before a single
# real row is typed. They are emitted as LIVE ROWS, not comments: importing a
# freshly downloaded template writes PC-001, CNC-014, Dell and Bay 3, and
# updates any existing In Use / Retired status row it matches on. The emitted
# instructions say so, and test_template_example_set_imports_cleanly depends on
# it. Delete the example rows before importing real data. Values avoid commas on
# purpose - the generator writes them without CSV quoting.
EXAMPLE_ROWS = OrderedDict([
('assetstatuses', [
{'status': 'In Use', 'description': 'Actively deployed', 'color': '#22c55e', 'isactive': '1'},
{'status': 'In Storage', 'description': 'Spare - not deployed', 'color': '#eab308', 'isactive': '1'},
{'status': 'Retired', 'description': 'Removed from service', 'color': '#ef4444', 'isactive': '1'},
]),
('assettypes', [
{'assettype': 'Computer', 'pluginname': 'computers', 'tablename': 'computers', 'description': 'PCs - servers - workstations', 'icon': 'laptop', 'color': '#3b82f6', 'isactive': '1'},
{'assettype': 'Machine', 'pluginname': 'machines', 'tablename': 'machines', 'description': 'Shop-floor machines', 'icon': 'cpu', 'color': '#8b5cf6', 'isactive': '1'},
]),
('locationtypes', [
{'locationtype': 'Building', 'description': 'A physical building', 'color': '#64748b', 'isactive': '1'},
{'locationtype': 'Bay', 'description': 'A shop-floor bay', 'color': '#0ea5e9', 'isactive': '1'},
]),
('modeltypes', [
{'modeltype': 'Desktop', 'category': 'Computer', 'description': 'Desktop PC chassis', 'icon': 'desktop', 'isactive': '1'},
{'modeltype': 'CNC Machine', 'category': 'Machine', 'description': 'CNC machining center', 'icon': 'cpu', 'isactive': '1'},
]),
('computertypes', [
{'computertype': 'Shopfloor PC', 'description': 'PC on the shop floor', 'icon': 'monitor', 'color': '#3b82f6', 'isactive': '1'},
{'computertype': 'Engineer Workstation', 'description': 'CAD/CAM workstation', 'icon': 'laptop', 'color': '#6366f1', 'isactive': '1'},
]),
('machinetypes', [
{'machinetype': 'CNC Mill', 'description': 'Computer numerical control mill', 'icon': 'cpu', 'color': '#8b5cf6', 'isactive': '1'},
{'machinetype': 'CMM', 'description': 'Coordinate measuring machine', 'icon': 'ruler', 'color': '#14b8a6', 'isactive': '1'},
]),
('businessunits', [
{'businessunit': 'Machining', 'code': 'MACH', 'description': 'Machining department', 'isactive': '1'},
{'businessunit': 'Quality', 'code': 'QUAL', 'description': 'Quality and metrology', 'isactive': '1'},
]),
('locations', [
{'locationname': 'Building 1', 'building': 'Building 1', 'floor': '1', 'description': 'Main manufacturing building', 'locationtypeid': 'Building', 'isactive': '1'},
{'locationname': 'Bay 3', 'building': 'Building 1', 'floor': '1', 'room': 'Machining bay 3', 'description': 'Machining bay 3', 'locationtypeid': 'Bay', 'isactive': '1'},
]),
('vendors', [
{'vendor': 'Dell', 'description': 'Computer hardware', 'website': 'https://www.dell.com', 'supportphone': '800-555-0100', 'supportemail': 'support@dell.com', 'isactive': '1'},
{'vendor': 'Haas Automation', 'description': 'CNC machine tools', 'website': 'https://www.haascnc.com', 'supportphone': '800-555-0199', 'isactive': '1'},
]),
('models', [
{'modelnumber': 'OptiPlex 7010', 'modeltypeid': 'Desktop', 'vendorid': 'Dell', 'description': 'Small form factor desktop', 'isactive': '1'},
{'modelnumber': 'VF-2', 'modeltypeid': 'CNC Machine', 'vendorid': 'Haas Automation', 'description': '40-taper vertical mill', 'isactive': '1'},
]),
('operatingsystems', [
{'osname': 'Windows 11 (24H2)', 'osversion': '24H2', 'architecture': 'x64', 'endoflife': '2027-10-12', 'isactive': '1'},
{'osname': 'Windows 10 (22H2)', 'osversion': '22H2', 'architecture': 'x64', 'endoflife': '2025-10-14', 'isactive': '1'},
]),
('assets', [
{'assetnumber': 'PC-001', 'name': 'Shopfloor PC 001', 'serialnumber': 'SN-PC-001', 'assettypeid': 'Computer', 'statusid': 'In Use', 'locationid': 'Bay 3', 'businessunitid': 'Machining', 'isactive': '1'},
{'assetnumber': 'CNC-014', 'name': 'Haas VF-2 #14', 'serialnumber': 'SN-CNC-014', 'assettypeid': 'Machine', 'statusid': 'In Use', 'locationid': 'Bay 3', 'businessunitid': 'Machining', 'isactive': '1'},
]),
('computers', [
{'assetid': 'PC-001', 'computertypeid': 'Shopfloor PC', 'hostname': 'WS-PC-001', 'osid': 'Windows 11 (24H2)', 'vendorid': 'Dell', 'modelnumberid': 'OptiPlex 7010', 'loggedinuser': 'jdoe', 'lastreporteddate': '2026-08-01', 'lastboottime': '2026-08-01', 'isactive': '1', 'machineassetnumber': 'CNC-014'},
]),
('machines', [
{'assetid': 'CNC-014', 'machinetypeid': 'CNC Mill', 'vendorid': 'Haas Automation', 'modelnumberid': 'VF-2', 'requiresmanualconfig': '0', 'islocationonly': '0', 'lastmaintenancedate': '2026-06-01', 'nextmaintenancedate': '2026-12-01', 'maintenanceintervaldays': '180', 'isactive': '1'},
]),
])
# Columns a CSV may never set: surrogate keys the database owns, and audit
# stamps. Accepting an id from a spreadsheet is how two sites end up with
# colliding primary keys the first time anyone merges data.
@@ -86,6 +155,128 @@ PASSWORD_COLUMNS = {'passwordhash', 'password'}
TRUE_VALUES = {'1', 'true', 'yes', 'y', 't'}
FALSE_VALUES = {'0', 'false', 'no', 'n', 'f', ''}
# Reconciliation: when a name does not match a lookup row exactly, the resolver
# may still bind it to an existing row that is close enough. Only lookup tables
# take part - the small reference tables an operator picks values from. Rows that
# ARE the data being imported (assets, computers, machines) and the fast-moving
# operatingsystems list are excluded, so a typo there is a new row, not a silent
# merge onto something unrelated.
RECONCILABLE_TABLES = {
'assetstatuses', 'assettypes', 'locationtypes', 'modeltypes',
'computertypes', 'machinetypes', 'businessunits', 'locations',
'vendors', 'models',
}
# Corporate suffixes stripped before comparison so 'Dell Inc.' reconciles to
# 'Dell'. Lowercase, punctuation already removed by the time these are checked.
CORPORATE_SUFFIXES = {
'inc', 'incorporated', 'llc', 'llp', 'ltd', 'limited', 'corp',
'corporation', 'co', 'company', 'gmbh', 'ag', 'sa', 'plc',
}
# Default similarity a fuzzy match must reach, 0..1. Also the value the settings
# default is seeded with; 0 disables reconciliation entirely. Fuzzy matching is
# opt-in: it ships OFF (0) because a high similarity score is precisely the
# dangerous case - 'Bay 310' and 'Bay 311' score 0.857 and would silently merge.
# A site turns it on deliberately once it understands its own vocabulary.
DEFAULT_RECONCILE_THRESHOLD = 0.0
# -- plugin import hooks (inversion of control) ---------------------------------
#
# A spreadsheet is the wrong place to type an integer foreign key, and it is
# also the wrong place to split a relationship across two files: a site that
# knows "this PC drives machine 3015" wants to write that number in the PC row,
# not hand-build an assetrelationships sheet keyed on ids it never sees.
#
# But the machine link is a computers-plugin concept, and this module is core:
# it must not import a plugin. So a plugin REGISTERS itself here instead. It
# declares a VIRTUAL COLUMN - a header this table accepts but never writes to
# the database - and a HOOK that runs once per written row to turn that column's
# value into whatever the plugin means by it (here, a PC->machine controls link).
# Core stays ignorant of what the column is for; the plugin owns the meaning.
_VIRTUAL_COLUMNS = {} # tablename -> {columnname: description}
_ROW_HOOKS = {} # tablename -> [hook, ...]
def register_import_hook(tablename, virtual_columns, hook):
"""Let a plugin own virtual column(s) on an importable table.
virtual_columns: either a sequence of header names, or a mapping of
name -> one-line description. Each is accepted for ``tablename`` but never
written - it is pulled out of the row and handed to the hook instead of the
table. A description, when given, is shown in the generated template's
column notes so an operator knows what to type.
hook(rowvalues, virtualvalues, line) -> list[RowWarning]: called once per
written row, AFTER the row is inserted/updated, so its id exists. It runs
inside the caller's transaction - which a dry run rolls back - so it must
not commit, and must not do anything a rollback cannot undo (no emails). A
hook reports trouble by returning RowWarning, never by raising: a bad
machine number is an amber note beside a PC that still imports, not a failed
file.
Idempotent by contract: registering the same (table, column, hook) twice is
a no-op, so a plugin re-init during tests or a reload does not double-fire.
"""
cols = _VIRTUAL_COLUMNS.setdefault(tablename, {})
if isinstance(virtual_columns, dict):
for name, description in virtual_columns.items():
cols[name] = description or ''
else:
for name in virtual_columns:
cols.setdefault(name, '')
hooks = _ROW_HOOKS.setdefault(tablename, [])
# Dedupe by the underlying function's qualified name, not by the bound
# method: a plugin re-init (a new app context in the test suite, a reload in
# prod) makes a fresh instance whose bound method is != the old one, and a
# symlinked plugin can even be imported under two module paths, so an
# identity check would append a second hook and the row would be processed -
# and warned about - more than once.
def _identity(fn):
func = getattr(fn, '__func__', fn)
return getattr(func, '__qualname__', repr(func))
if _identity(hook) not in [_identity(existing) for existing in hooks]:
hooks.append(hook)
def virtual_columns_for(tablename):
"""Header names this table accepts but does not store. Never None."""
return set(_VIRTUAL_COLUMNS.get(tablename, ()))
def virtual_column_notes(tablename):
"""(name, description) for each virtual column, in registration order."""
return list(_VIRTUAL_COLUMNS.get(tablename, {}).items())
def normalize_label(value):
"""Fold a label to its comparison form: lowercase, no punctuation, no
corporate suffix, single-spaced. 'Dell Inc.' and 'DELL' both become 'dell'.
"""
text = (value or '').strip().lower()
text = re.sub(r'[^a-z0-9]+', ' ', text).strip()
if not text:
return ''
words = [w for w in text.split(' ') if w]
while len(words) > 1 and words[-1] in CORPORATE_SUFFIXES:
words.pop()
return ' '.join(words)
def digit_runs(value):
"""Return the ordered list of digit runs in a normalized label.
'bay 310' -> ['310'], 'precision 3660' -> ['3660'], 'dell' -> []. Two labels
that share a fuzzy word shape but differ in any digit run ('bay 310' vs
'bay 311') must NOT reconcile: a single differing digit is a small fraction
of a long string, so SequenceMatcher scores it above threshold even though
the labels name two distinct things. Comparing digit runs exactly closes
that gap while still fuzzing the surrounding words.
"""
return re.findall(r'\d+', value or '')
def table_for(name):
table = db.metadata.tables.get(name)
@@ -221,10 +412,17 @@ class Resolver:
Rows created earlier in the SAME import are resolvable, so one file can
reference another that was loaded moments ago without a commit in between.
When a name does not match exactly and the table is reconcilable, a
normalized then fuzzy comparison against the existing rows may bind it to a
close-enough match. Every such binding is recorded so the caller can report
it for review; nothing is merged silently.
"""
def __init__(self):
def __init__(self, reconcile_threshold=0.0):
self._cache = {}
self.reconcile_threshold = reconcile_threshold or 0.0
self.events = []
def resolve(self, tablename, idcolumn, value):
if value is None:
@@ -249,6 +447,10 @@ class Resolver:
db.func.lower(db.func.trim(table.c[label])) == str(value).strip().lower())
).first()
if row is None:
found = self._reconcile(tablename, idcolumn, label, table, value)
if found is not None:
self._cache[key] = found
return found
raise ValueError(
"nothing in %s is named '%s' - add it to %s.csv, or import that file first"
% (tablename, value, tablename))
@@ -256,6 +458,59 @@ class Resolver:
self._cache[key] = found
return found
def _reconcile(self, tablename, idcolumn, label, table, value):
"""Bind value to the closest existing row when within threshold.
Returns the matched id and records an event, or None when reconciliation
is off, the table is not reconcilable, or nothing is close enough.
"""
if self.reconcile_threshold <= 0 or tablename not in RECONCILABLE_TABLES:
return None
wanted = normalize_label(value)
if not wanted:
return None
wanted_digits = digit_runs(wanted)
best_id = None
best_label = None
best_score = 0.0
for existing in db.session.execute(table.select()):
candidate = getattr(existing, label, None)
normalized = normalize_label(candidate)
if not normalized:
continue
if normalized == wanted:
score = 1.0
else:
# Never fuzzy-match two labels whose digit runs differ. 'Bay 310'
# and 'Bay 311' share every word and score 0.857, but the digits
# name distinct things; refuse rather than silently merge. The
# normalized-exact tier above (score 1.0) is unaffected.
if digit_runs(normalized) != wanted_digits:
continue
score = difflib.SequenceMatcher(None, wanted, normalized).ratio()
if score > best_score:
best_score = score
best_id = getattr(existing, idcolumn)
best_label = candidate
if best_id is not None and best_score >= self.reconcile_threshold:
self.events.append({
'table': tablename,
'input': str(value).strip(),
'matched': best_label,
'score': round(best_score, 3),
})
return best_id
return None
def take_events(self):
"""Return and clear reconciliations recorded since the last call."""
events = self.events
self.events = []
return events
def remember(self, tablename, name, rowid):
if name is None:
return
@@ -275,6 +530,32 @@ class RowProblem(object):
return "%s: %s" % (where, self.message)
class RowWarning(object):
"""A non-blocking, amber-tier finding.
Errors (RowProblem) stop the whole import; a RowWarning does NOT - the row
it belongs to still imports. It exists so a caveat (an unresolved or
conflicting machine link, say) is loud and REVIEWABLE without failing an
otherwise-clean file. Every warning names WHERE it happened, WHY, and the
single ACTION that fixes it, so a cold user can correct it in place - the
convention IT Glue / RMM imports train.
"""
def __init__(self, line, column, reason, action, subject=None, target=None):
self.line = line
self.column = column
self.reason = reason # what is wrong, in plain terms
self.action = action # the one corrective step to take
self.subject = subject # the row's actor (e.g. the PC hostname)
self.target = target # what it referenced (e.g. the machine number)
def __str__(self):
where = "line %d" % self.line
if self.column:
where += ", column '%s'" % self.column
return "%s: %s - %s" % (where, self.reason, self.action)
class ImportResult(object):
def __init__(self, tablename):
self.tablename = tablename
@@ -282,14 +563,19 @@ class ImportResult(object):
self.updated = 0
self.problems = []
self.skipped = 0
self.reconciliations = []
self.warnings = []
@property
def ok(self):
# Warnings are amber, not red: a file with only warnings is still ok to
# commit. Only problems (errors) block.
return not self.problems
def summary(self):
return "%s: %d new, %d updated, %d problem(s)" % (
self.tablename, self.created, self.updated, len(self.problems))
return "%s: %d new, %d updated, %d problem(s), %d warning(s)" % (
self.tablename, self.created, self.updated,
len(self.problems), len(self.warnings))
def read_rows(text):
@@ -343,9 +629,10 @@ def import_csv(tablename, text, resolver=None, commit=False):
header, rows = read_rows(text)
allowed = {c.name: c for c in importable_columns(tablename)}
unknown = [h for h in header if h not in allowed]
virtual = virtual_columns_for(tablename)
unknown = [h for h in header if h not in allowed and h not in virtual]
if unknown:
known = ', '.join(sorted(allowed))
known = ', '.join(sorted(set(allowed) | virtual))
result.problems.append(RowProblem(
1, None, "unknown column(s): %s\n this table accepts: %s" % (', '.join(unknown), known)))
return result
@@ -365,7 +652,17 @@ def import_csv(tablename, text, resolver=None, commit=False):
prepared = []
for line, raw in rows:
values = {}
virtualvalues = {}
failed = False
# Virtual columns belong to a plugin hook, not the table. Pull them out
# first so they never reach an insert/update, and stash them for the
# post-write hook. Whitespace-trimmed; an empty cell is dropped so the
# hook only sees values a row actually carried.
for name in virtual:
if name in raw:
text = (raw[name] or '').strip()
if text != '':
virtualvalues[name] = text
for name, column in allowed.items():
if name not in raw:
continue
@@ -381,6 +678,10 @@ def import_csv(tablename, text, resolver=None, commit=False):
continue
try:
values[name] = resolver.resolve(target[0], target[1], text)
for event in resolver.take_events():
event['line'] = line
event['column'] = name
result.reconciliations.append(event)
except ValueError as exc:
result.problems.append(RowProblem(line, name, str(exc)))
failed = True
@@ -396,16 +697,18 @@ def import_csv(tablename, text, resolver=None, commit=False):
if values.get(natural) in (None, ''):
result.problems.append(RowProblem(line, natural, 'must have a value'))
continue
prepared.append((line, values))
prepared.append((line, values, virtualvalues))
if not result.ok:
return result
hooks = _ROW_HOOKS.get(tablename, ())
# Second pass: decide insert vs update. Counted even on a dry run, because
# "this will change 4000 rows" is the number an operator needs before saying
# yes.
label = _label_column(tablename)
for line, values in prepared:
for line, values, virtualvalues in prepared:
existing = db.session.execute(
table.select().where(table.c[natural] == values[natural])).first()
if existing is None:
@@ -422,6 +725,15 @@ def import_csv(tablename, text, resolver=None, commit=False):
primary = list(table.primary_key)[0].name
resolver.remember(tablename, values[label], getattr(existing, primary))
# Fire plugin hooks AFTER the write, so the row's id exists. Only when
# the row actually carried a virtual value - a hook that has nothing to
# act on should never run. Warnings are collected, never raised: an
# unresolved machine number is amber, not a failed file.
if hooks and virtualvalues:
for hook in hooks:
result.warnings.extend(
hook(values, virtualvalues, line) or [])
return result
@@ -456,24 +768,55 @@ def generate_template(tablename):
bits.append('text, max %d' % column.type.length)
notes.append("# %-24s %s" % (column.name, ', '.join(bits)))
# Virtual columns a plugin owns: accepted on this table but written by the
# plugin's hook, not the schema. Appended after the real columns so the
# header carries them and the notes explain what they do.
virtualnames = []
for name, description in virtual_column_notes(tablename):
virtualnames.append(name)
header.append(name)
note = 'optional, ' + description if description else 'optional'
notes.append("# %-24s %s" % (name, note))
spec = IMPORTABLE[tablename]
examples = EXAMPLE_ROWS.get(tablename, [])
exampleorder = [c.name for c in columns] + virtualnames
lines = [
"# %s.csv - generated from the ShopDB schema, do not hand-maintain" % tablename,
"#",
"# Rows are matched on '%s'. Re-importing updates a matching row rather" % spec['natural'],
"# than creating a second one, so it is safe to correct this file and run again.",
"#",
"# HOW TO USE THIS TEMPLATE:",
"# 1. Delete everything above the header row (every line starting with #).",
"# 2. Fill in your own rows using the example row(s) below as a guide.",
"# 3. DELETE THE EXAMPLE DATA before you import - the example rows are",
"# live data and WILL be imported if you leave them in.",
"#",
"# Columns:",
] + notes + [
"#",
"# Lines starting with # are ignored. Delete the example row before importing.",
"# --- header and example rows follow. Keep the header; replace the examples. ---",
','.join(header),
'# ' + ','.join(_example(c, required) for c in columns),
] + [
','.join(_example_value(row.get(name, '')) for name in exampleorder)
for row in examples
]
return '\n'.join(lines) + '\n'
def _example_value(value):
"""A single example cell. Commas would break the unquoted CSV the template
writes, so they are replaced with a dash; example data is authored without
them, this is only a guard.
"""
return str(value).replace(',', ' -')
def _example(column, required):
"""Kept for callers/tests that ask for a single synthetic cell. The template
generator now emits the curated EXAMPLE_ROWS instead.
"""
target = _fk_target(column)
if target:
label = _label_column(target[0])

325
shopdb/core/services/importsources.py Normal file
View File

@@ -0,0 +1,325 @@
"""Foreign import-source adapters.
The CSV import engine in csvimport.py speaks exactly one dialect: the canonical
per-table files (assets.csv, computers.csv, vendors.csv, ...). Foreign exports -
an Entra/Azure device-group member list, a Thor PC inventory dump - do not match
that dialect and never will; they are owned by other systems.
Rather than teach the engine every foreign schema, each adapter here TRANSFORMS
a foreign export into the canonical files and hands them back as
{tablename: csv_text}. The caller then runs those through the ordinary
import_csv path, so dry-run/commit, dependency ordering, and reconciliation all
work unchanged.
An adapter declares a header signature so an upload can be auto-detected, and a
transform() that yields the canonical files. Adapters never touch the database.
"""
import csv
import io
class ImportSourceError(Exception):
"""A foreign file could not be transformed (bad header, empty, etc.)."""
# Canonical defaults for a discovered PC. Azure carries only a hostname, so
# every device it creates is a computer that is In Use until something richer
# (Thor, the collector) says otherwise.
DEFAULT_ASSETTYPE = 'computer'
DEFAULT_STATUS = 'In Use'
def _read(text):
"""Parse CSV text into (header, list-of-dicts), tolerating a UTF-8 BOM."""
stream = io.StringIO(text)
reader = csv.DictReader(stream)
header = [h.strip() for h in (reader.fieldnames or [])]
rows = []
for raw in reader:
rows.append({(k.strip() if k else k): (v or '').strip()
for k, v in raw.items()})
return header, rows
def _write(header, rows):
"""Render canonical rows back to CSV text with a trailing newline."""
out = io.StringIO()
writer = csv.DictWriter(out, fieldnames=header, extrasaction='ignore')
writer.writeheader()
for row in rows:
writer.writerow(row)
return out.getvalue()
def _has_columns(header, required):
lowered = {h.lower() for h in header}
return all(col.lower() in lowered for col in required)
class AzureDeviceGroupSource:
"""Entra/Azure device-group member export.
Columns: displayName, objectType, mail, userType, id, deviceId,
userPrincipalName, isUser, isGroup, isGuest. Only device rows carry signal;
displayName is the hostname and deviceId is the Entra device GUID. The export
has no model, OS, location or user data, so this creates/updates minimal
asset + computer rows keyed on hostname.
options:
computertype - optional name stamped onto every computer's computertypeid
(e.g. 'Shopfloor PC' for a SHP group, 'Inspection PC' for
BUS). Resolved by the engine against computertypes.
category - optional Entra device category / group membership label
stamped onto every computer's azuredevicecategory (e.g.
SFLD_DYN_Device_Group_<Site>_BUS_NC_COUNTRY:USA).
hostname_map - optional {lowercased hostname: existing assetnumber}. A
device whose displayName matches an existing computer's
hostname is routed to that asset and updated in place
(no new asset row); unmatched devices create a new
asset + computer keyed on the hostname.
"""
key = 'azure'
label = 'Azure device group'
signature = ('displayName', 'deviceId', 'objectType')
@classmethod
def matches(cls, header):
return _has_columns(header, cls.signature)
@classmethod
def transform(cls, text, options=None):
options = options or {}
computertype = (options.get('computertype') or '').strip()
category = (options.get('category') or '').strip()
hostname_map = options.get('hostname_map') or {}
header, rows = _read(text)
if not _has_columns(header, cls.signature):
raise ImportSourceError(
'not an Azure device-group export (expected displayName, '
'deviceId, objectType columns)')
assets = []
computers = []
seen = set()
for row in rows:
objecttype = (row.get('objectType') or '').strip().lower()
if objecttype and objecttype != 'device':
continue
hostname = (row.get('displayName') or '').strip()
if not hostname or hostname.lower() in seen:
continue
seen.add(hostname.lower())
# Align on hostname: if this device already exists in ShopDB, point
# the computer row at its real asset so the engine updates it in
# place. Only create a new asset when there is no hostname match.
existing_assetnumber = hostname_map.get(hostname.lower())
computer = {
'assetid': existing_assetnumber if existing_assetnumber is not None else hostname,
'hostname': hostname,
'azuredeviceid': (row.get('deviceId') or '').strip(),
'azureobjectid': (row.get('id') or '').strip(),
}
if computertype:
computer['computertypeid'] = computertype
if category:
computer['azuredevicecategory'] = category
computers.append(computer)
if existing_assetnumber is None:
assets.append({
'assetnumber': hostname,
'name': hostname,
'assettypeid': DEFAULT_ASSETTYPE,
'statusid': DEFAULT_STATUS,
})
if not computers:
raise ImportSourceError('no device rows found in the Azure export')
asset_header = ['assetnumber', 'name', 'assettypeid', 'statusid']
computer_header = ['assetid', 'hostname', 'azuredeviceid', 'azureobjectid']
if computertype:
computer_header.append('computertypeid')
if category:
computer_header.append('azuredevicecategory')
produced = {
'computers': _write(computer_header, computers),
}
# Only emit an assets file when there are new devices to create; an
# empty assets.csv would just be an extra clean no-op table.
if assets:
produced['assets'] = _write(asset_header, assets)
return produced
class ThorSource:
"""Thor PC-inventory export (internal GE Aerospace tool).
A wide, fixed-format PC inventory. Maps the columns ShopDB owns into
canonical rows: hostname, vendor (BIOS Manf.), model, operating system,
logged-in user, last-reported and last-boot timestamps, and location. The
network columns (MAC, switch, VLAN, IP) and the machine 'Attached' link are
out of scope for this first pass.
options:
computertype - optional name stamped onto every computer's computertypeid.
Resolved by the engine against computertypes.
hostname_map - optional {lowercased hostname: existing assetnumber}. A
host whose Host Name matches an existing computer's hostname
is routed to that asset and updated in place (no new asset
row); unmatched hosts create a new asset + computer keyed on
the hostname.
"""
key = 'thor'
label = 'Thor PC inventory'
signature = ('Host Name', 'BIOS Manf.', 'OS Version')
@classmethod
def matches(cls, header):
return _has_columns(header, cls.signature)
@classmethod
def transform(cls, text, options=None):
options = options or {}
computertype = (options.get('computertype') or '').strip()
hostname_map = options.get('hostname_map') or {}
header, rows = _read(text)
if not _has_columns(header, cls.signature):
raise ImportSourceError(
'not a Thor export (expected Host Name, BIOS Manf., OS Version '
'columns)')
vendors = {}
models = {}
operatingsystems = {}
assets = []
computers = []
seen = set()
for row in rows:
hostname = (row.get('Host Name') or '').strip()
if not hostname or hostname.lower() in seen:
continue
seen.add(hostname.lower())
vendor = (row.get('BIOS Manf.') or '').strip()
model = (row.get('Model') or '').strip()
osname = cls._osname(row)
location = (row.get('Location') or '').strip()
loggedinuser = cls._logon(row)
if vendor:
vendors[vendor.lower()] = vendor
if model:
models[model.lower()] = model
if osname:
operatingsystems[osname.lower()] = osname
# Align on hostname: if this host already exists in ShopDB, point
# the computer row at its real asset so the engine updates it in
# place. Only create a new asset when there is no hostname match.
existing_assetnumber = hostname_map.get(hostname.lower())
if existing_assetnumber is None:
assets.append({
'assetnumber': hostname,
'name': hostname,
'assettypeid': DEFAULT_ASSETTYPE,
'statusid': DEFAULT_STATUS,
'locationid': location,
})
computer = {
'assetid': existing_assetnumber if existing_assetnumber is not None else hostname,
'hostname': hostname,
'osid': osname,
'vendorid': vendor,
'modelnumberid': model,
'loggedinuser': loggedinuser,
'lastreporteddate': cls._date(row.get('Last Updated')),
'lastboottime': cls._date(row.get('Boot Time')),
}
if computertype:
computer['computertypeid'] = computertype
computers.append(computer)
if not computers:
raise ImportSourceError('no host rows found in the Thor export')
out = {}
if vendors:
out['vendors'] = _write(['vendor'],
[{'vendor': v} for v in vendors.values()])
if models:
out['models'] = _write(['modelnumber'],
[{'modelnumber': m} for m in models.values()])
if operatingsystems:
out['operatingsystems'] = _write(
['osname'], [{'osname': o} for o in operatingsystems.values()])
asset_header = ['assetnumber', 'name', 'assettypeid', 'statusid', 'locationid']
computer_header = ['assetid', 'hostname', 'osid', 'vendorid',
'modelnumberid', 'loggedinuser', 'lastreporteddate',
'lastboottime']
if computertype:
computer_header.append('computertypeid')
out['computers'] = _write(computer_header, computers)
# Only emit an assets file when there are new hosts to create; an empty
# assets.csv would just be an extra clean no-op table.
if assets:
out['assets'] = _write(asset_header, assets)
return out
@staticmethod
def _osname(row):
"""Combine OS Version and OS ID into a single OS name.
Thor splits the OS into a build ('10.0.26100') and a marketing id
('24H2'); neither alone is meaningful, so join them.
"""
version = (row.get('OS Version') or '').strip()
osid = (row.get('OS ID') or '').strip()
if version and osid:
return '%s (%s)' % (version, osid)
return version or osid
@staticmethod
def _logon(row):
"""Take the account name out of a DOMAIN\\user or HOST\\user LOGON."""
logon = (row.get('LOGON') or '').strip()
if '\\' in logon:
return logon.rsplit('\\', 1)[-1]
return logon
@staticmethod
def _date(value):
"""Normalize a Thor timestamp to a form the engine's date coercion
accepts, or empty when it cannot be parsed. Leaves already-ISO values
alone; the engine tries several formats itself.
"""
return (value or '').strip()
SOURCES = {
AzureDeviceGroupSource.key: AzureDeviceGroupSource,
ThorSource.key: ThorSource,
}
def get_source(key):
source = SOURCES.get((key or '').strip().lower())
if source is None:
raise ImportSourceError(
"unknown import source '%s' (known: %s)"
% (key, ', '.join(sorted(SOURCES))))
return source
def detect_source(header):
"""Return the adapter whose signature matches this header, or None."""
for source in SOURCES.values():
if source.matches(header):
return source
return None