ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Browse Source 
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
This commit is contained in:
cproudlock
2026-08-28 16:17:17 -04:00
parent 93ca7587ce
commit 784964c907
149 changed files with 17545 additions and 627 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
shopdb/__init__.py
View File

@@ -57,7 +57,31 @@ from .plugins import plugin_manager
# permission / empty / position. BREAKING for any plugin still using the old
# shape, which is why it is recorded here: the change itself shipped earlier
# without a bump, and a contract that changes silently is not a contract.
__contract_version__ = '0.21.0'
# 0.21.0: added require_any_permission beside require_permission and require_role
# on shopdb.api so a plugin can gate a route on holding ANY of several
# permissions (an OR check) instead of one. Part of the read-permission rework
# where nav items and GET routes declare the permission they need. Additive name
# on the authz surface, minor bump.
# 0.22.0: added register_import_hook to shopdb.api so a plugin can own a VIRTUAL
# COLUMN on an importable CSV table - a header the bulk importer accepts but
# never writes - and a post-write row hook that turns that value into a plugin
# concept. The computers plugin uses it for a machineassetnumber column on
# computers.csv that builds the PC->machine controls link, so a site writes the
# machine number in the PC row instead of hand-building a relationships sheet.
# Core stays ignorant of the column's meaning. Additive name on the import
# surface, minor bump.
# 0.23.0: added mint_service_token, list_service_tokens and revoke_service_tokens
# to shopdb.api so a plugin can provision, list, and revoke a single-scope
# managed service token (an endpoint agent key) without importing core token
# internals. The endpoints plugin uses them for a UI-managed, regenerable
# collector.ingest.endpoints key whose revocation is an instant kill switch.
# Additive names on the surface, minor bump.
# 0.24.0: added collector_authorized to shopdb.api so a plugin can gate an
# agent-facing route that lives OUTSIDE the generic collector blueprint (the
# endpoints command claim/result long-poll callbacks) on the same per-plugin
# collector key + managed-token check the ingest route uses, without importing
# core collector internals. Additive name on the surface, minor bump.
__contract_version__ = '0.24.0'
# Product release version (see ADR-007). The product version and the
# plugin-contract version above are distinct series with independent

36
shopdb/api/__init__.py
View File

@@ -75,8 +75,28 @@ from shopdb.utils.authz import require_permission, require_any_permission, requi
# internals.
from shopdb.utils.apitoken_auth import (
service_token_authorized, authorized_service_token,
mint_service_token, list_service_tokens, revoke_service_tokens,
)
# Collector-token authorization for agent-facing plugin routes that live outside
# the generic collector blueprint (e.g. the endpoints command claim/result
# callbacks). Reuses the same per-plugin collector key + managed-token check the
# ingest route uses, so a plugin never imports core collector internals.
def collector_authorized(pluginname: str):
"""Authorize an agent-facing route on a plugin's collector token.
Returns None when the caller presented a valid collector credential for
this plugin (the per-plugin env key or a collector.ingest[.<plugin>]-scoped
managed token), else the error_response to return. Lets a plugin gate an
agent callback on the same key and kill switch as its ingest route without
reaching into core collector internals.
Imported lazily: shopdb.core.api imports back into shopdb.api (send_upload),
so a module-level import here would be a circular import at load time.
"""
from shopdb.core.api.collector import _check_collector_auth, _plugin_api_key
return _check_collector_auth(_plugin_api_key(pluginname), pluginname)
# Serving an uploaded file safely. A plugin that serves user-supplied bytes must
# not have to remember the headers that keep an SVG or an HTML file from running
# as script in this origin - see shopdb/utils/uploads.py.
@@ -95,6 +115,15 @@ from shopdb.core.services.dualpath import (
dualpath_single_machine_enabled,
)
# Bulk-import extension point: a plugin registers a virtual column on an
# importable CSV table plus a post-write row hook, so a relationship a site
# would otherwise hand-build (a PC's machine number -> a controls link) can be
# written in the row itself. Core never learns what the column means.
from shopdb.core.services.csvimport import (
register_import_hook,
RowWarning,
)
# Legacy employee directory lookup (read-only) used by notifications
from shopdb.utils.employee_db import employee_connection
from shopdb.utils.mailer import send_email, send_alert, send_webhook
@@ -261,6 +290,9 @@ __all__ = [
'resolve_asset_position',
'resolve_dualpath_pairs',
'dualpath_single_machine_enabled',
# Bulk-import extension point
'register_import_hook',
'RowWarning',
# Infrastructure
'db',
'cache',
@@ -302,6 +334,10 @@ __all__ = [
'require_role',
'service_token_authorized',
'authorized_service_token',
'mint_service_token',
'list_service_tokens',
'revoke_service_tokens',
'collector_authorized',
'SupportTeam',
# Serving uploads
'send_upload',

2
shopdb/config.py
View File

@@ -134,7 +134,7 @@ class DevelopmentConfig(Config):
SQLALCHEMY_DATABASE_URI = os.environ.get(
'DATABASE_URL',
'mysql+pymysql://root:rootpassword@127.0.0.1:3306/shopdb_flask',
'mysql+pymysql://root:changeme@127.0.0.1:3306/shopdb_flask',
)

61
shopdb/core/api/collector.py
View File

@@ -39,17 +39,56 @@ def _computer_models():
return None
def _token_is_collector(token):
"""True when this managed token is a collector service token: scoped for
collector.ingest AND its owner actually holds that permission (admins do)."""
def _collector_scopes_for(pluginname=None):
"""The scope names that authorize a collector request.
The generic scope `collector.ingest` authorizes every collector. A plugin
MAY also mint a token scoped to `collector.ingest.<pluginname>` so that
rotating/revoking it stops only that plugin's collector traffic (endpoints
uses this for its regenerable agent kill switch). Both are accepted.
"""
scopes = {COLLECTOR_SCOPE}
if pluginname:
scopes.add(f'{COLLECTOR_SCOPE}.{pluginname}')
return scopes
def _env_fallback_disabled(pluginname=None):
"""True when a plugin has switched OFF the bootstrap env-key fallback.
Once an operator provisions a UI-managed collector token, the plugin sets
its `plugin.<pluginname>.envkeyfallback` setting to false so the old .env
key stops authorizing. That turns key revocation into a real kill switch:
rotating or revoking the managed token cannot be bypassed by a lingering env
key. Absent
setting = fallback stays ON (back-compat, so nothing breaks on upgrade).
"""
if not pluginname:
return False
from shopdb.core.models.setting import Setting
key = f'plugin.{pluginname}.envkeyfallback'
return Setting.get(key, default=True) is False
def _token_is_collector(token, pluginname=None):
"""True when this managed token is a collector service token: scoped for a
collector scope AND its owner actually holds that permission (admins do).
Accepts the generic `collector.ingest` or, when pluginname is given, the
plugin-specific `collector.ingest.<pluginname>` scope.
"""
from shopdb.core.models import User
scopelist = token.scopelist
if not scopelist or COLLECTOR_SCOPE not in scopelist:
if not scopelist:
return False
allowed = _collector_scopes_for(pluginname)
held = [s for s in scopelist if s in allowed]
if not held:
return False
user = db.session.get(User, token.userid)
return (user is not None and user.isactive
and user.haspermission(COLLECTOR_SCOPE))
and any(user.haspermission(s) for s in held))
def _collector_managed_token(api_key):
@@ -73,24 +112,28 @@ def _collector_managed_token(api_key):
return None
def _check_collector_auth(expected_key):
def _check_collector_auth(expected_key, pluginname=None):
"""Authorize a collector request. Returns None when authorized, else the
error_response to return.
Accepts a collector-scoped managed token (Bearer or X-API-Key) OR the env
key via X-API-Key (legacy/bootstrap fallback). Env keys stay supported so
nothing breaks; a valid managed token works even with no env key set.
When pluginname is given, a token scoped to collector.ingest.<pluginname>
also authorizes (in addition to the generic collector.ingest scope).
"""
from shopdb.utils.apitoken_auth import touch_apitoken_lastused
api_key = request.headers.get('X-API-Key')
token = _collector_managed_token(api_key)
if token is not None and _token_is_collector(token):
if token is not None and _token_is_collector(token, pluginname):
touch_apitoken_lastused(token)
return None
if expected_key and api_key == expected_key:
if expected_key and api_key == expected_key \
and not _env_fallback_disabled(pluginname):
return None
# Fail-closed only on pure server misconfiguration: no env key AND the
@@ -200,7 +243,7 @@ def generic_collect(pluginname):
# Per-plugin env key (fallback) OR a collector-scoped managed token. Header
# only (querystring fallback dropped, see require_api_key).
error = _check_collector_auth(_plugin_api_key(pluginname))
error = _check_collector_auth(_plugin_api_key(pluginname), pluginname)
if error is not None:
return error

9
shopdb/core/api/models.py
View File

@@ -8,7 +8,7 @@ from flask_jwt_extended import jwt_required
from werkzeug.utils import secure_filename
from shopdb.extensions import db
from shopdb.core.models import Model
from shopdb.core.models import Model, ModelType
from shopdb.utils.responses import (
success_response,
error_response,
@@ -56,6 +56,12 @@ def list_models():
if modeltype_id := request.args.get('modeltype', type=int):
query = query.filter(Model.modeltypeid == modeltype_id)
# Scope to a model-type category (Equipment, PC, Network, Printer) so a
# subtype form only offers models that make sense for it. Models with no
# type are excluded, since a null category cannot satisfy the filter.
if category := request.args.get('category'):
query = query.join(ModelType).filter(ModelType.category == category)
# Exact-match natural-key lookup for idempotent import. Natural key is
# modelnumber + vendor; pair this with ?vendor=<id> to disambiguate.
if exactmodelnumber := request.args.get('modelnumber'):
@@ -73,6 +79,7 @@ def list_models():
d = m.to_dict()
d['vendor'] = m.vendor.vendor if m.vendor else None
d['modeltype'] = m.modeltype.modeltype if m.modeltype else None
d['category'] = m.modeltype.category if m.modeltype else None
data.append(d)
return paginated_response(data, page, per_page, total)

325
shopdb/core/api/settings.py
View File

@@ -1,10 +1,12 @@
"""Settings API routes."""
import os
import re
from flask import Blueprint, request, current_app
from flask_jwt_extended import jwt_required, get_jwt_identity
from werkzeug.utils import secure_filename
from sqlalchemy import select as sa_select
from shopdb.extensions import db, cache
from shopdb.core.models import Setting, AuditLog
@@ -241,6 +243,315 @@ def serve_branding_logo(filename):
return send_upload(_branding_dir(), filename)
def _asset_import_dir():
"""Configured Asset Collector CSV location, or '' when unset."""
setting = Setting.query.filter_by(key='asset_import_csv_location').first()
return (setting.value or '').strip() if setting else ''
# The collector script that gets served (with the site's CSV path baked in) from
# the Import Site Assets page. Copied into the image under scripts/ (see
# Dockerfile), so it is always available next to the running API.
_COLLECTOR_SCRIPT = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))),
'scripts', 'site_imports', 'assetcollector', 'Collect-ShopDB.ps1')
def _embed_store_path(script_text, path):
"""Return the collector script with $NetworkStoreRoot set to path.
The script ships with a BLANK default on that line; rewriting it means an
operator downloads a script already pointed at this facility's CSV store and
does not have to edit PowerShell by hand. A blank path leaves the shipped
blank default untouched, so a site that has not configured
asset_import_csv_location keeps everything local instead of inheriting
another facility's share.
"""
if not path:
return script_text
literal = path.replace("'", "''")
replacement = "$NetworkStoreRoot = '%s'" % literal
# A function replacement is used deliberately: a UNC path is full of
# backslashes, and a string replacement runs them through re's escape
# processing (so \s, \w, a trailing \1 all raise or corrupt). A callable
# returns the text verbatim.
return re.sub(
r"(?m)^\$NetworkStoreRoot\s*=.*$",
lambda _match: replacement,
script_text,
count=1)
@settings_bp.route('/asset-import/script', methods=['GET'])
@require_role('admin')
def download_collector_script():
"""Download Collect-ShopDB.ps1 with this site's CSV path embedded."""
try:
with open(_COLLECTOR_SCRIPT, 'r', encoding='utf-8') as handle:
script_text = handle.read()
except OSError:
return error_response(ErrorCodes.NOT_FOUND,
'Collector script is not available on the server',
http_code=404)
script_text = _embed_store_path(script_text, _asset_import_dir())
response = current_app.response_class(script_text, mimetype='text/plain')
response.headers['Content-Disposition'] = 'attachment; filename=Collect-ShopDB.ps1'
return response
@settings_bp.route('/asset-import/templates', methods=['GET'])
@require_role('admin')
def list_import_templates():
"""List the importable CSV tables in the order they must be uploaded."""
from shopdb.core.services.csvimport import IMPORTABLE
tables = [{'table': name, 'filename': '%s.csv' % name}
for name in IMPORTABLE]
return success_response({'tables': tables})
@settings_bp.route('/asset-import/templates/<tablename>', methods=['GET'])
@require_role('admin')
def download_import_template(tablename):
"""Download a generated CSV template for a single importable table."""
from shopdb.core.services.csvimport import (
IMPORTABLE, generate_template, ImportError_)
if tablename not in IMPORTABLE:
return error_response(ErrorCodes.NOT_FOUND,
'No such importable table: %s' % tablename,
http_code=404)
try:
template_text = generate_template(tablename)
except ImportError_ as exc:
return error_response(ErrorCodes.VALIDATION_ERROR, str(exc))
response = current_app.response_class(template_text, mimetype='text/csv')
response.headers['Content-Disposition'] = (
'attachment; filename=%s.csv' % tablename)
return response
def _existing_hostname_map():
"""Case-insensitive hostname -> assetnumber map for existing computers.
Azure device-group exports align on hostname (their displayName), while the
import engine keys assets on assetnumber. This lets the Azure adapter route
a matched device to its real asset so the import updates in place.
"""
assets = db.metadata.tables['assets']
computers = db.metadata.tables['computers']
query = sa_select(
computers.c.hostname, assets.c.assetnumber
).select_from(
computers.join(assets, computers.c.assetid == assets.c.assetid)
).where(computers.c.hostname.isnot(None))
mapping = {}
for hostname, assetnumber in db.session.execute(query):
if hostname:
mapping[hostname.strip().lower()] = assetnumber
return mapping
@settings_bp.route('/asset-import/import', methods=['POST'])
@require_role('admin')
def import_asset_csvs():
"""Validate uploaded CSVs and, when commit=true, load them.
Mirrors the 'flask csv import' CLI: every file runs through import_csv inside
ONE transaction with a shared resolver, so a computers.csv can resolve the
assets.csv uploaded alongside it. A dry run (the default) rolls the whole
transaction back and reports what WOULD change; commit=true persists it. The
whole run is all-or-nothing - if any file has a row problem, nothing is
written even on a commit.
A 'source' form field selects a foreign adapter (azure, thor); it defaults
to 'native' where uploads are already canonical per-table files. Foreign
uploads are transformed into canonical files first, then run identically.
"""
from sqlalchemy.exc import SQLAlchemyError
from shopdb.core.services.csvimport import (
ImportError_, Resolver, dependency_order, import_csv, table_from_filename)
from shopdb.core.services.importsources import (
ImportSourceError, get_source)
uploads = request.files.getlist('files')
if not uploads:
return error_response(ErrorCodes.VALIDATION_ERROR, 'No CSV files were uploaded')
commit = str(request.form.get('commit', '')).strip().lower() in (
'1', 'true', 'yes', 'on')
# A foreign source (azure, thor) is transformed into canonical files before
# import. 'native' (or nothing) means the files already ARE canonical.
source_key = (request.form.get('source') or 'native').strip().lower()
source = None
if source_key and source_key != 'native':
try:
source = get_source(source_key)
except ImportSourceError as exc:
return error_response(ErrorCodes.VALIDATION_ERROR, str(exc))
source_options = {
'computertype': (request.form.get('computertype') or '').strip(),
'category': (request.form.get('category') or '').strip(),
}
# Azure and Thor align on hostname: their device name is the ShopDB
# computers.hostname, not the assetnumber. Hand the adapter a
# case-insensitive hostname -> existing assetnumber map so devices that
# already exist update in place (keyed to their real asset) instead of
# minting a duplicate asset.
if source is not None and getattr(source, 'key', None) in ('azure', 'thor'):
source_options['hostname_map'] = _existing_hostname_map()
texts = {}
filenames = {}
if source is not None:
# One foreign export becomes several canonical files. Each upload is
# transformed and its canonical outputs merged; a later file's table
# wins, matching the native one-file-per-table rule.
for upload in uploads:
name = upload.filename or ''
if not name.lower().endswith('.csv'):
continue
try:
raw = upload.read().decode('utf-8-sig')
except (UnicodeDecodeError, OSError):
return error_response(ErrorCodes.VALIDATION_ERROR,
'Could not read %s as UTF-8 text' % name)
try:
produced = source.transform(raw, options=source_options)
except ImportSourceError as exc:
return error_response(ErrorCodes.VALIDATION_ERROR,
'%s: %s' % (name, str(exc)))
for table, csv_text in produced.items():
texts[table] = csv_text
filenames[table] = '%s (from %s)' % (table, name)
else:
# Map each upload to a table by filename. A later file with the same
# table name wins, matching the folder-import behaviour.
for upload in uploads:
name = upload.filename or ''
if not name.lower().endswith('.csv'):
continue
table = table_from_filename(name)
try:
texts[table] = upload.read().decode('utf-8-sig')
except (UnicodeDecodeError, OSError):
return error_response(ErrorCodes.VALIDATION_ERROR,
'Could not read %s as UTF-8 text' % name)
filenames[table] = name
if not texts:
return error_response(ErrorCodes.VALIDATION_ERROR,
'None of the uploaded files were CSV files')
ordered = dependency_order(list(texts))
skipped = sorted(set(texts) - set(ordered))
reconcile_setting = Setting.query.filter_by(
key='asset_import_reconcile_threshold').first()
try:
reconcile_threshold = float(reconcile_setting.value) if reconcile_setting and reconcile_setting.value else 0.0
except (TypeError, ValueError):
reconcile_threshold = 0.0
resolver = Resolver(reconcile_threshold=reconcile_threshold)
files_report = []
problem_count = 0
warning_count = 0
reconciled_count = 0
request_error = None
try:
for table in ordered:
try:
result = import_csv(table, texts[table], resolver=resolver, commit=False)
except ImportError_ as exc:
request_error = '%s: %s' % (filenames[table], str(exc))
break
except SQLAlchemyError as exc:
db.session.rollback()
request_error = '%s: database error: %s' % (
filenames[table], str(exc).split(chr(10))[0])
break
problems = [
{'line': p.line, 'column': p.column, 'message': p.message}
for p in result.problems
]
# Amber tier: a warning is a finding beside a row that still
# imported (an unresolved machine link, a contested bay), NOT a
# reason to fail the file. Reported alongside problems so the UI can
# show Errors / Warnings / Info distinctly, but never gates commit.
warnings = [
{'line': w.line, 'column': w.column, 'reason': w.reason,
'action': w.action, 'subject': w.subject, 'target': w.target}
for w in result.warnings
]
problem_count += len(problems)
warning_count += len(warnings)
reconciled_count += len(result.reconciliations)
files_report.append({
'table': table,
'filename': filenames[table],
'created': result.created,
'updated': result.updated,
'problems': problems,
'warnings': warnings,
'reconciliations': result.reconciliations,
})
applied = False
if request_error is not None:
db.session.rollback()
elif problem_count > 0:
# A row problem anywhere fails the whole run, so nothing is written
# even when commit was asked for.
db.session.rollback()
elif commit:
# One audit row for the whole run, written INSIDE the same
# transaction so it cannot survive a rollback or go missing from a
# commit. Every other write in this app is auditable; a bulk import
# that creates thousands of rows was the one path where nobody could
# answer "who loaded this, and when".
AuditLog.log(
'imported', 'SiteAssets',
entityname=', '.join(f['table'] for f in files_report) or 'none',
changes={
'files': [f['table'] for f in files_report],
'created': sum(f['created'] for f in files_report),
'updated': sum(f['updated'] for f in files_report),
'reconciled': reconciled_count,
})
db.session.commit()
applied = True
else:
db.session.rollback()
except Exception:
db.session.rollback()
raise
total_created = sum(f['created'] for f in files_report)
total_updated = sum(f['updated'] for f in files_report)
return success_response({
'committed': applied,
'requested_commit': commit,
'request_error': request_error,
'files': files_report,
'skipped': skipped,
'problem_count': problem_count,
'warning_count': warning_count,
'reconciled_count': reconciled_count,
'reconcile_threshold': reconcile_threshold,
'total_created': total_created,
'total_updated': total_updated,
})
@settings_bp.route('', methods=['GET'])
@jwt_required(optional=True)
def list_settings():
@@ -568,6 +879,20 @@ def build_default_settings():
'category': 'site',
'description': 'Email domain appended to a contact SSO to build email/Teams links (sso@domain). Blank disables contact action buttons.'
},
{
'key': 'asset_import_csv_location',
'value': '',
'valuetype': 'string',
'category': 'site',
'description': r'Shared network location where the Asset Collector script deposits its CSVs and where ShopDB reads them for import (e.g. \\server.fqdn\share\folder). This path is baked into the downloadable collector script. Blank = not configured.'
},
{
'key': 'asset_import_reconcile_threshold',
'value': '0',
'valuetype': 'string',
'category': 'site',
'description': 'Similarity (0 to 1) a fuzzy lookup match must reach for the asset importer to reconcile a near-miss name (e.g. "Dell Inc." to an existing "Dell") onto an existing vendor, model, type, status, business unit, or location. Reconciled matches are applied but reported for review. 0 (the default) disables fuzzy reconciliation - names must match exactly after normalization. Labels whose digit runs differ (e.g. "Bay 310" vs "Bay 311") are never fuzzy-matched. Operating systems and asset rows are never reconciled.'
},
]
# Site branding. Each instance (ADR-004) can replace the shipped GE defaults

363
shopdb/core/services/csvimport.py
View File

@@ -1,7 +1,7 @@
"""Bulk-load a site's starting data from CSV files.
WHY THIS EXISTS. Adopting a site means getting its existing asset register into
ShopDB. The HTTP import surface (docs/IMPORT-API.md) is the right tool when
ShopDB. The HTTP import surface (the IMPORT-API wiki page) is the right tool when
there is a source system to read from and someone able to script against it. A
sister site with a spreadsheet and no developer needs something else, and that
is the common case.
@@ -34,6 +34,7 @@ not at all.
from __future__ import annotations
import csv
import difflib
import io
import re
from collections import OrderedDict
@@ -73,6 +74,74 @@ IMPORTABLE = OrderedDict([
('machines', {'label': None, 'natural': 'assetid'}),
])
# A small, INTERNALLY CONSISTENT worked example per table, in import order. A
# site can download the set, see the pattern, and keep it as a starter workbook:
# every name referenced by a later file (Bay 3, Dell, Computer, ...) is defined
# in an earlier one, so the whole set imports cleanly as a demo before a single
# real row is typed. They are emitted as LIVE ROWS, not comments: importing a
# freshly downloaded template writes PC-001, CNC-014, Dell and Bay 3, and
# updates any existing In Use / Retired status row it matches on. The emitted
# instructions say so, and test_template_example_set_imports_cleanly depends on
# it. Delete the example rows before importing real data. Values avoid commas on
# purpose - the generator writes them without CSV quoting.
EXAMPLE_ROWS = OrderedDict([
('assetstatuses', [
{'status': 'In Use', 'description': 'Actively deployed', 'color': '#22c55e', 'isactive': '1'},
{'status': 'In Storage', 'description': 'Spare - not deployed', 'color': '#eab308', 'isactive': '1'},
{'status': 'Retired', 'description': 'Removed from service', 'color': '#ef4444', 'isactive': '1'},
]),
('assettypes', [
{'assettype': 'Computer', 'pluginname': 'computers', 'tablename': 'computers', 'description': 'PCs - servers - workstations', 'icon': 'laptop', 'color': '#3b82f6', 'isactive': '1'},
{'assettype': 'Machine', 'pluginname': 'machines', 'tablename': 'machines', 'description': 'Shop-floor machines', 'icon': 'cpu', 'color': '#8b5cf6', 'isactive': '1'},
]),
('locationtypes', [
{'locationtype': 'Building', 'description': 'A physical building', 'color': '#64748b', 'isactive': '1'},
{'locationtype': 'Bay', 'description': 'A shop-floor bay', 'color': '#0ea5e9', 'isactive': '1'},
]),
('modeltypes', [
{'modeltype': 'Desktop', 'category': 'Computer', 'description': 'Desktop PC chassis', 'icon': 'desktop', 'isactive': '1'},
{'modeltype': 'CNC Machine', 'category': 'Machine', 'description': 'CNC machining center', 'icon': 'cpu', 'isactive': '1'},
]),
('computertypes', [
{'computertype': 'Shopfloor PC', 'description': 'PC on the shop floor', 'icon': 'monitor', 'color': '#3b82f6', 'isactive': '1'},
{'computertype': 'Engineer Workstation', 'description': 'CAD/CAM workstation', 'icon': 'laptop', 'color': '#6366f1', 'isactive': '1'},
]),
('machinetypes', [
{'machinetype': 'CNC Mill', 'description': 'Computer numerical control mill', 'icon': 'cpu', 'color': '#8b5cf6', 'isactive': '1'},
{'machinetype': 'CMM', 'description': 'Coordinate measuring machine', 'icon': 'ruler', 'color': '#14b8a6', 'isactive': '1'},
]),
('businessunits', [
{'businessunit': 'Machining', 'code': 'MACH', 'description': 'Machining department', 'isactive': '1'},
{'businessunit': 'Quality', 'code': 'QUAL', 'description': 'Quality and metrology', 'isactive': '1'},
]),
('locations', [
{'locationname': 'Building 1', 'building': 'Building 1', 'floor': '1', 'description': 'Main manufacturing building', 'locationtypeid': 'Building', 'isactive': '1'},
{'locationname': 'Bay 3', 'building': 'Building 1', 'floor': '1', 'room': 'Machining bay 3', 'description': 'Machining bay 3', 'locationtypeid': 'Bay', 'isactive': '1'},
]),
('vendors', [
{'vendor': 'Dell', 'description': 'Computer hardware', 'website': 'https://www.dell.com', 'supportphone': '800-555-0100', 'supportemail': 'support@dell.com', 'isactive': '1'},
{'vendor': 'Haas Automation', 'description': 'CNC machine tools', 'website': 'https://www.haascnc.com', 'supportphone': '800-555-0199', 'isactive': '1'},
]),
('models', [
{'modelnumber': 'OptiPlex 7010', 'modeltypeid': 'Desktop', 'vendorid': 'Dell', 'description': 'Small form factor desktop', 'isactive': '1'},
{'modelnumber': 'VF-2', 'modeltypeid': 'CNC Machine', 'vendorid': 'Haas Automation', 'description': '40-taper vertical mill', 'isactive': '1'},
]),
('operatingsystems', [
{'osname': 'Windows 11 (24H2)', 'osversion': '24H2', 'architecture': 'x64', 'endoflife': '2027-10-12', 'isactive': '1'},
{'osname': 'Windows 10 (22H2)', 'osversion': '22H2', 'architecture': 'x64', 'endoflife': '2025-10-14', 'isactive': '1'},
]),
('assets', [
{'assetnumber': 'PC-001', 'name': 'Shopfloor PC 001', 'serialnumber': 'SN-PC-001', 'assettypeid': 'Computer', 'statusid': 'In Use', 'locationid': 'Bay 3', 'businessunitid': 'Machining', 'isactive': '1'},
{'assetnumber': 'CNC-014', 'name': 'Haas VF-2 #14', 'serialnumber': 'SN-CNC-014', 'assettypeid': 'Machine', 'statusid': 'In Use', 'locationid': 'Bay 3', 'businessunitid': 'Machining', 'isactive': '1'},
]),
('computers', [
{'assetid': 'PC-001', 'computertypeid': 'Shopfloor PC', 'hostname': 'WS-PC-001', 'osid': 'Windows 11 (24H2)', 'vendorid': 'Dell', 'modelnumberid': 'OptiPlex 7010', 'loggedinuser': 'jdoe', 'lastreporteddate': '2026-08-01', 'lastboottime': '2026-08-01', 'isactive': '1', 'machineassetnumber': 'CNC-014'},
]),
('machines', [
{'assetid': 'CNC-014', 'machinetypeid': 'CNC Mill', 'vendorid': 'Haas Automation', 'modelnumberid': 'VF-2', 'requiresmanualconfig': '0', 'islocationonly': '0', 'lastmaintenancedate': '2026-06-01', 'nextmaintenancedate': '2026-12-01', 'maintenanceintervaldays': '180', 'isactive': '1'},
]),
])
# Columns a CSV may never set: surrogate keys the database owns, and audit
# stamps. Accepting an id from a spreadsheet is how two sites end up with
# colliding primary keys the first time anyone merges data.
@@ -86,6 +155,128 @@ PASSWORD_COLUMNS = {'passwordhash', 'password'}
TRUE_VALUES = {'1', 'true', 'yes', 'y', 't'}
FALSE_VALUES = {'0', 'false', 'no', 'n', 'f', ''}
# Reconciliation: when a name does not match a lookup row exactly, the resolver
# may still bind it to an existing row that is close enough. Only lookup tables
# take part - the small reference tables an operator picks values from. Rows that
# ARE the data being imported (assets, computers, machines) and the fast-moving
# operatingsystems list are excluded, so a typo there is a new row, not a silent
# merge onto something unrelated.
RECONCILABLE_TABLES = {
'assetstatuses', 'assettypes', 'locationtypes', 'modeltypes',
'computertypes', 'machinetypes', 'businessunits', 'locations',
'vendors', 'models',
}
# Corporate suffixes stripped before comparison so 'Dell Inc.' reconciles to
# 'Dell'. Lowercase, punctuation already removed by the time these are checked.
CORPORATE_SUFFIXES = {
'inc', 'incorporated', 'llc', 'llp', 'ltd', 'limited', 'corp',
'corporation', 'co', 'company', 'gmbh', 'ag', 'sa', 'plc',
}
# Default similarity a fuzzy match must reach, 0..1. Also the value the settings
# default is seeded with; 0 disables reconciliation entirely. Fuzzy matching is
# opt-in: it ships OFF (0) because a high similarity score is precisely the
# dangerous case - 'Bay 310' and 'Bay 311' score 0.857 and would silently merge.
# A site turns it on deliberately once it understands its own vocabulary.
DEFAULT_RECONCILE_THRESHOLD = 0.0
# -- plugin import hooks (inversion of control) ---------------------------------
#
# A spreadsheet is the wrong place to type an integer foreign key, and it is
# also the wrong place to split a relationship across two files: a site that
# knows "this PC drives machine 3015" wants to write that number in the PC row,
# not hand-build an assetrelationships sheet keyed on ids it never sees.
#
# But the machine link is a computers-plugin concept, and this module is core:
# it must not import a plugin. So a plugin REGISTERS itself here instead. It
# declares a VIRTUAL COLUMN - a header this table accepts but never writes to
# the database - and a HOOK that runs once per written row to turn that column's
# value into whatever the plugin means by it (here, a PC->machine controls link).
# Core stays ignorant of what the column is for; the plugin owns the meaning.
_VIRTUAL_COLUMNS = {} # tablename -> {columnname: description}
_ROW_HOOKS = {} # tablename -> [hook, ...]
def register_import_hook(tablename, virtual_columns, hook):
"""Let a plugin own virtual column(s) on an importable table.
virtual_columns: either a sequence of header names, or a mapping of
name -> one-line description. Each is accepted for ``tablename`` but never
written - it is pulled out of the row and handed to the hook instead of the
table. A description, when given, is shown in the generated template's
column notes so an operator knows what to type.
hook(rowvalues, virtualvalues, line) -> list[RowWarning]: called once per
written row, AFTER the row is inserted/updated, so its id exists. It runs
inside the caller's transaction - which a dry run rolls back - so it must
not commit, and must not do anything a rollback cannot undo (no emails). A
hook reports trouble by returning RowWarning, never by raising: a bad
machine number is an amber note beside a PC that still imports, not a failed
file.
Idempotent by contract: registering the same (table, column, hook) twice is
a no-op, so a plugin re-init during tests or a reload does not double-fire.
"""
cols = _VIRTUAL_COLUMNS.setdefault(tablename, {})
if isinstance(virtual_columns, dict):
for name, description in virtual_columns.items():
cols[name] = description or ''
else:
for name in virtual_columns:
cols.setdefault(name, '')
hooks = _ROW_HOOKS.setdefault(tablename, [])
# Dedupe by the underlying function's qualified name, not by the bound
# method: a plugin re-init (a new app context in the test suite, a reload in
# prod) makes a fresh instance whose bound method is != the old one, and a
# symlinked plugin can even be imported under two module paths, so an
# identity check would append a second hook and the row would be processed -
# and warned about - more than once.
def _identity(fn):
func = getattr(fn, '__func__', fn)
return getattr(func, '__qualname__', repr(func))
if _identity(hook) not in [_identity(existing) for existing in hooks]:
hooks.append(hook)
def virtual_columns_for(tablename):
"""Header names this table accepts but does not store. Never None."""
return set(_VIRTUAL_COLUMNS.get(tablename, ()))
def virtual_column_notes(tablename):
"""(name, description) for each virtual column, in registration order."""
return list(_VIRTUAL_COLUMNS.get(tablename, {}).items())
def normalize_label(value):
"""Fold a label to its comparison form: lowercase, no punctuation, no
corporate suffix, single-spaced. 'Dell Inc.' and 'DELL' both become 'dell'.
"""
text = (value or '').strip().lower()
text = re.sub(r'[^a-z0-9]+', ' ', text).strip()
if not text:
return ''
words = [w for w in text.split(' ') if w]
while len(words) > 1 and words[-1] in CORPORATE_SUFFIXES:
words.pop()
return ' '.join(words)
def digit_runs(value):
"""Return the ordered list of digit runs in a normalized label.
'bay 310' -> ['310'], 'precision 3660' -> ['3660'], 'dell' -> []. Two labels
that share a fuzzy word shape but differ in any digit run ('bay 310' vs
'bay 311') must NOT reconcile: a single differing digit is a small fraction
of a long string, so SequenceMatcher scores it above threshold even though
the labels name two distinct things. Comparing digit runs exactly closes
that gap while still fuzzing the surrounding words.
"""
return re.findall(r'\d+', value or '')
def table_for(name):
table = db.metadata.tables.get(name)
@@ -221,10 +412,17 @@ class Resolver:
Rows created earlier in the SAME import are resolvable, so one file can
reference another that was loaded moments ago without a commit in between.
When a name does not match exactly and the table is reconcilable, a
normalized then fuzzy comparison against the existing rows may bind it to a
close-enough match. Every such binding is recorded so the caller can report
it for review; nothing is merged silently.
"""
def __init__(self):
def __init__(self, reconcile_threshold=0.0):
self._cache = {}
self.reconcile_threshold = reconcile_threshold or 0.0
self.events = []
def resolve(self, tablename, idcolumn, value):
if value is None:
@@ -249,6 +447,10 @@ class Resolver:
db.func.lower(db.func.trim(table.c[label])) == str(value).strip().lower())
).first()
if row is None:
found = self._reconcile(tablename, idcolumn, label, table, value)
if found is not None:
self._cache[key] = found
return found
raise ValueError(
"nothing in %s is named '%s' - add it to %s.csv, or import that file first"
% (tablename, value, tablename))
@@ -256,6 +458,59 @@ class Resolver:
self._cache[key] = found
return found
def _reconcile(self, tablename, idcolumn, label, table, value):
"""Bind value to the closest existing row when within threshold.
Returns the matched id and records an event, or None when reconciliation
is off, the table is not reconcilable, or nothing is close enough.
"""
if self.reconcile_threshold <= 0 or tablename not in RECONCILABLE_TABLES:
return None
wanted = normalize_label(value)
if not wanted:
return None
wanted_digits = digit_runs(wanted)
best_id = None
best_label = None
best_score = 0.0
for existing in db.session.execute(table.select()):
candidate = getattr(existing, label, None)
normalized = normalize_label(candidate)
if not normalized:
continue
if normalized == wanted:
score = 1.0
else:
# Never fuzzy-match two labels whose digit runs differ. 'Bay 310'
# and 'Bay 311' share every word and score 0.857, but the digits
# name distinct things; refuse rather than silently merge. The
# normalized-exact tier above (score 1.0) is unaffected.
if digit_runs(normalized) != wanted_digits:
continue
score = difflib.SequenceMatcher(None, wanted, normalized).ratio()
if score > best_score:
best_score = score
best_id = getattr(existing, idcolumn)
best_label = candidate
if best_id is not None and best_score >= self.reconcile_threshold:
self.events.append({
'table': tablename,
'input': str(value).strip(),
'matched': best_label,
'score': round(best_score, 3),
})
return best_id
return None
def take_events(self):
"""Return and clear reconciliations recorded since the last call."""
events = self.events
self.events = []
return events
def remember(self, tablename, name, rowid):
if name is None:
return
@@ -275,6 +530,32 @@ class RowProblem(object):
return "%s: %s" % (where, self.message)
class RowWarning(object):
"""A non-blocking, amber-tier finding.
Errors (RowProblem) stop the whole import; a RowWarning does NOT - the row
it belongs to still imports. It exists so a caveat (an unresolved or
conflicting machine link, say) is loud and REVIEWABLE without failing an
otherwise-clean file. Every warning names WHERE it happened, WHY, and the
single ACTION that fixes it, so a cold user can correct it in place - the
convention IT Glue / RMM imports train.
"""
def __init__(self, line, column, reason, action, subject=None, target=None):
self.line = line
self.column = column
self.reason = reason # what is wrong, in plain terms
self.action = action # the one corrective step to take
self.subject = subject # the row's actor (e.g. the PC hostname)
self.target = target # what it referenced (e.g. the machine number)
def __str__(self):
where = "line %d" % self.line
if self.column:
where += ", column '%s'" % self.column
return "%s: %s - %s" % (where, self.reason, self.action)
class ImportResult(object):
def __init__(self, tablename):
self.tablename = tablename
@@ -282,14 +563,19 @@ class ImportResult(object):
self.updated = 0
self.problems = []
self.skipped = 0
self.reconciliations = []
self.warnings = []
@property
def ok(self):
# Warnings are amber, not red: a file with only warnings is still ok to
# commit. Only problems (errors) block.
return not self.problems
def summary(self):
return "%s: %d new, %d updated, %d problem(s)" % (
self.tablename, self.created, self.updated, len(self.problems))
return "%s: %d new, %d updated, %d problem(s), %d warning(s)" % (
self.tablename, self.created, self.updated,
len(self.problems), len(self.warnings))
def read_rows(text):
@@ -343,9 +629,10 @@ def import_csv(tablename, text, resolver=None, commit=False):
header, rows = read_rows(text)
allowed = {c.name: c for c in importable_columns(tablename)}
unknown = [h for h in header if h not in allowed]
virtual = virtual_columns_for(tablename)
unknown = [h for h in header if h not in allowed and h not in virtual]
if unknown:
known = ', '.join(sorted(allowed))
known = ', '.join(sorted(set(allowed) | virtual))
result.problems.append(RowProblem(
1, None, "unknown column(s): %s\n this table accepts: %s" % (', '.join(unknown), known)))
return result
@@ -365,7 +652,17 @@ def import_csv(tablename, text, resolver=None, commit=False):
prepared = []
for line, raw in rows:
values = {}
virtualvalues = {}
failed = False
# Virtual columns belong to a plugin hook, not the table. Pull them out
# first so they never reach an insert/update, and stash them for the
# post-write hook. Whitespace-trimmed; an empty cell is dropped so the
# hook only sees values a row actually carried.
for name in virtual:
if name in raw:
text = (raw[name] or '').strip()
if text != '':
virtualvalues[name] = text
for name, column in allowed.items():
if name not in raw:
continue
@@ -381,6 +678,10 @@ def import_csv(tablename, text, resolver=None, commit=False):
continue
try:
values[name] = resolver.resolve(target[0], target[1], text)
for event in resolver.take_events():
event['line'] = line
event['column'] = name
result.reconciliations.append(event)
except ValueError as exc:
result.problems.append(RowProblem(line, name, str(exc)))
failed = True
@@ -396,16 +697,18 @@ def import_csv(tablename, text, resolver=None, commit=False):
if values.get(natural) in (None, ''):
result.problems.append(RowProblem(line, natural, 'must have a value'))
continue
prepared.append((line, values))
prepared.append((line, values, virtualvalues))
if not result.ok:
return result
hooks = _ROW_HOOKS.get(tablename, ())
# Second pass: decide insert vs update. Counted even on a dry run, because
# "this will change 4000 rows" is the number an operator needs before saying
# yes.
label = _label_column(tablename)
for line, values in prepared:
for line, values, virtualvalues in prepared:
existing = db.session.execute(
table.select().where(table.c[natural] == values[natural])).first()
if existing is None:
@@ -422,6 +725,15 @@ def import_csv(tablename, text, resolver=None, commit=False):
primary = list(table.primary_key)[0].name
resolver.remember(tablename, values[label], getattr(existing, primary))
# Fire plugin hooks AFTER the write, so the row's id exists. Only when
# the row actually carried a virtual value - a hook that has nothing to
# act on should never run. Warnings are collected, never raised: an
# unresolved machine number is amber, not a failed file.
if hooks and virtualvalues:
for hook in hooks:
result.warnings.extend(
hook(values, virtualvalues, line) or [])
return result
@@ -456,24 +768,55 @@ def generate_template(tablename):
bits.append('text, max %d' % column.type.length)
notes.append("# %-24s %s" % (column.name, ', '.join(bits)))
# Virtual columns a plugin owns: accepted on this table but written by the
# plugin's hook, not the schema. Appended after the real columns so the
# header carries them and the notes explain what they do.
virtualnames = []
for name, description in virtual_column_notes(tablename):
virtualnames.append(name)
header.append(name)
note = 'optional, ' + description if description else 'optional'
notes.append("# %-24s %s" % (name, note))
spec = IMPORTABLE[tablename]
examples = EXAMPLE_ROWS.get(tablename, [])
exampleorder = [c.name for c in columns] + virtualnames
lines = [
"# %s.csv - generated from the ShopDB schema, do not hand-maintain" % tablename,
"#",
"# Rows are matched on '%s'. Re-importing updates a matching row rather" % spec['natural'],
"# than creating a second one, so it is safe to correct this file and run again.",
"#",
"# HOW TO USE THIS TEMPLATE:",
"# 1. Delete everything above the header row (every line starting with #).",
"# 2. Fill in your own rows using the example row(s) below as a guide.",
"# 3. DELETE THE EXAMPLE DATA before you import - the example rows are",
"# live data and WILL be imported if you leave them in.",
"#",
"# Columns:",
] + notes + [
"#",
"# Lines starting with # are ignored. Delete the example row before importing.",
"# --- header and example rows follow. Keep the header; replace the examples. ---",
','.join(header),
'# ' + ','.join(_example(c, required) for c in columns),
] + [
','.join(_example_value(row.get(name, '')) for name in exampleorder)
for row in examples
]
return '\n'.join(lines) + '\n'
def _example_value(value):
"""A single example cell. Commas would break the unquoted CSV the template
writes, so they are replaced with a dash; example data is authored without
them, this is only a guard.
"""
return str(value).replace(',', ' -')
def _example(column, required):
"""Kept for callers/tests that ask for a single synthetic cell. The template
generator now emits the curated EXAMPLE_ROWS instead.
"""
target = _fk_target(column)
if target:
label = _label_column(target[0])

325
shopdb/core/services/importsources.py Normal file
View File

@@ -0,0 +1,325 @@
"""Foreign import-source adapters.
The CSV import engine in csvimport.py speaks exactly one dialect: the canonical
per-table files (assets.csv, computers.csv, vendors.csv, ...). Foreign exports -
an Entra/Azure device-group member list, a Thor PC inventory dump - do not match
that dialect and never will; they are owned by other systems.
Rather than teach the engine every foreign schema, each adapter here TRANSFORMS
a foreign export into the canonical files and hands them back as
{tablename: csv_text}. The caller then runs those through the ordinary
import_csv path, so dry-run/commit, dependency ordering, and reconciliation all
work unchanged.
An adapter declares a header signature so an upload can be auto-detected, and a
transform() that yields the canonical files. Adapters never touch the database.
"""
import csv
import io
class ImportSourceError(Exception):
"""A foreign file could not be transformed (bad header, empty, etc.)."""
# Canonical defaults for a discovered PC. Azure carries only a hostname, so
# every device it creates is a computer that is In Use until something richer
# (Thor, the collector) says otherwise.
DEFAULT_ASSETTYPE = 'computer'
DEFAULT_STATUS = 'In Use'
def _read(text):
"""Parse CSV text into (header, list-of-dicts), tolerating a UTF-8 BOM."""
stream = io.StringIO(text)
reader = csv.DictReader(stream)
header = [h.strip() for h in (reader.fieldnames or [])]
rows = []
for raw in reader:
rows.append({(k.strip() if k else k): (v or '').strip()
for k, v in raw.items()})
return header, rows
def _write(header, rows):
"""Render canonical rows back to CSV text with a trailing newline."""
out = io.StringIO()
writer = csv.DictWriter(out, fieldnames=header, extrasaction='ignore')
writer.writeheader()
for row in rows:
writer.writerow(row)
return out.getvalue()
def _has_columns(header, required):
lowered = {h.lower() for h in header}
return all(col.lower() in lowered for col in required)
class AzureDeviceGroupSource:
"""Entra/Azure device-group member export.
Columns: displayName, objectType, mail, userType, id, deviceId,
userPrincipalName, isUser, isGroup, isGuest. Only device rows carry signal;
displayName is the hostname and deviceId is the Entra device GUID. The export
has no model, OS, location or user data, so this creates/updates minimal
asset + computer rows keyed on hostname.
options:
computertype - optional name stamped onto every computer's computertypeid
(e.g. 'Shopfloor PC' for a SHP group, 'Inspection PC' for
BUS). Resolved by the engine against computertypes.
category - optional Entra device category / group membership label
stamped onto every computer's azuredevicecategory (e.g.
SFLD_DYN_Device_Group_<Site>_BUS_NC_COUNTRY:USA).
hostname_map - optional {lowercased hostname: existing assetnumber}. A
device whose displayName matches an existing computer's
hostname is routed to that asset and updated in place
(no new asset row); unmatched devices create a new
asset + computer keyed on the hostname.
"""
key = 'azure'
label = 'Azure device group'
signature = ('displayName', 'deviceId', 'objectType')
@classmethod
def matches(cls, header):
return _has_columns(header, cls.signature)
@classmethod
def transform(cls, text, options=None):
options = options or {}
computertype = (options.get('computertype') or '').strip()
category = (options.get('category') or '').strip()
hostname_map = options.get('hostname_map') or {}
header, rows = _read(text)
if not _has_columns(header, cls.signature):
raise ImportSourceError(
'not an Azure device-group export (expected displayName, '
'deviceId, objectType columns)')
assets = []
computers = []
seen = set()
for row in rows:
objecttype = (row.get('objectType') or '').strip().lower()
if objecttype and objecttype != 'device':
continue
hostname = (row.get('displayName') or '').strip()
if not hostname or hostname.lower() in seen:
continue
seen.add(hostname.lower())
# Align on hostname: if this device already exists in ShopDB, point
# the computer row at its real asset so the engine updates it in
# place. Only create a new asset when there is no hostname match.
existing_assetnumber = hostname_map.get(hostname.lower())
computer = {
'assetid': existing_assetnumber if existing_assetnumber is not None else hostname,
'hostname': hostname,
'azuredeviceid': (row.get('deviceId') or '').strip(),
'azureobjectid': (row.get('id') or '').strip(),
}
if computertype:
computer['computertypeid'] = computertype
if category:
computer['azuredevicecategory'] = category
computers.append(computer)
if existing_assetnumber is None:
assets.append({
'assetnumber': hostname,
'name': hostname,
'assettypeid': DEFAULT_ASSETTYPE,
'statusid': DEFAULT_STATUS,
})
if not computers:
raise ImportSourceError('no device rows found in the Azure export')
asset_header = ['assetnumber', 'name', 'assettypeid', 'statusid']
computer_header = ['assetid', 'hostname', 'azuredeviceid', 'azureobjectid']
if computertype:
computer_header.append('computertypeid')
if category:
computer_header.append('azuredevicecategory')
produced = {
'computers': _write(computer_header, computers),
}
# Only emit an assets file when there are new devices to create; an
# empty assets.csv would just be an extra clean no-op table.
if assets:
produced['assets'] = _write(asset_header, assets)
return produced
class ThorSource:
"""Thor PC-inventory export (internal GE Aerospace tool).
A wide, fixed-format PC inventory. Maps the columns ShopDB owns into
canonical rows: hostname, vendor (BIOS Manf.), model, operating system,
logged-in user, last-reported and last-boot timestamps, and location. The
network columns (MAC, switch, VLAN, IP) and the machine 'Attached' link are
out of scope for this first pass.
options:
computertype - optional name stamped onto every computer's computertypeid.
Resolved by the engine against computertypes.
hostname_map - optional {lowercased hostname: existing assetnumber}. A
host whose Host Name matches an existing computer's hostname
is routed to that asset and updated in place (no new asset
row); unmatched hosts create a new asset + computer keyed on
the hostname.
"""
key = 'thor'
label = 'Thor PC inventory'
signature = ('Host Name', 'BIOS Manf.', 'OS Version')
@classmethod
def matches(cls, header):
return _has_columns(header, cls.signature)
@classmethod
def transform(cls, text, options=None):
options = options or {}
computertype = (options.get('computertype') or '').strip()
hostname_map = options.get('hostname_map') or {}
header, rows = _read(text)
if not _has_columns(header, cls.signature):
raise ImportSourceError(
'not a Thor export (expected Host Name, BIOS Manf., OS Version '
'columns)')
vendors = {}
models = {}
operatingsystems = {}
assets = []
computers = []
seen = set()
for row in rows:
hostname = (row.get('Host Name') or '').strip()
if not hostname or hostname.lower() in seen:
continue
seen.add(hostname.lower())
vendor = (row.get('BIOS Manf.') or '').strip()
model = (row.get('Model') or '').strip()
osname = cls._osname(row)
location = (row.get('Location') or '').strip()
loggedinuser = cls._logon(row)
if vendor:
vendors[vendor.lower()] = vendor
if model:
models[model.lower()] = model
if osname:
operatingsystems[osname.lower()] = osname
# Align on hostname: if this host already exists in ShopDB, point
# the computer row at its real asset so the engine updates it in
# place. Only create a new asset when there is no hostname match.
existing_assetnumber = hostname_map.get(hostname.lower())
if existing_assetnumber is None:
assets.append({
'assetnumber': hostname,
'name': hostname,
'assettypeid': DEFAULT_ASSETTYPE,
'statusid': DEFAULT_STATUS,
'locationid': location,
})
computer = {
'assetid': existing_assetnumber if existing_assetnumber is not None else hostname,
'hostname': hostname,
'osid': osname,
'vendorid': vendor,
'modelnumberid': model,
'loggedinuser': loggedinuser,
'lastreporteddate': cls._date(row.get('Last Updated')),
'lastboottime': cls._date(row.get('Boot Time')),
}
if computertype:
computer['computertypeid'] = computertype
computers.append(computer)
if not computers:
raise ImportSourceError('no host rows found in the Thor export')
out = {}
if vendors:
out['vendors'] = _write(['vendor'],
[{'vendor': v} for v in vendors.values()])
if models:
out['models'] = _write(['modelnumber'],
[{'modelnumber': m} for m in models.values()])
if operatingsystems:
out['operatingsystems'] = _write(
['osname'], [{'osname': o} for o in operatingsystems.values()])
asset_header = ['assetnumber', 'name', 'assettypeid', 'statusid', 'locationid']
computer_header = ['assetid', 'hostname', 'osid', 'vendorid',
'modelnumberid', 'loggedinuser', 'lastreporteddate',
'lastboottime']
if computertype:
computer_header.append('computertypeid')
out['computers'] = _write(computer_header, computers)
# Only emit an assets file when there are new hosts to create; an empty
# assets.csv would just be an extra clean no-op table.
if assets:
out['assets'] = _write(asset_header, assets)
return out
@staticmethod
def _osname(row):
"""Combine OS Version and OS ID into a single OS name.
Thor splits the OS into a build ('10.0.26100') and a marketing id
('24H2'); neither alone is meaningful, so join them.
"""
version = (row.get('OS Version') or '').strip()
osid = (row.get('OS ID') or '').strip()
if version and osid:
return '%s (%s)' % (version, osid)
return version or osid
@staticmethod
def _logon(row):
"""Take the account name out of a DOMAIN\\user or HOST\\user LOGON."""
logon = (row.get('LOGON') or '').strip()
if '\\' in logon:
return logon.rsplit('\\', 1)[-1]
return logon
@staticmethod
def _date(value):
"""Normalize a Thor timestamp to a form the engine's date coercion
accepts, or empty when it cannot be parsed. Leaves already-ISO values
alone; the engine tries several formats itself.
"""
return (value or '').strip()
SOURCES = {
AzureDeviceGroupSource.key: AzureDeviceGroupSource,
ThorSource.key: ThorSource,
}
def get_source(key):
source = SOURCES.get((key or '').strip().lower())
if source is None:
raise ImportSourceError(
"unknown import source '%s' (known: %s)"
% (key, ', '.join(sorted(SOURCES))))
return source
def detect_source(header):
"""Return the adapter whose signature matches this header, or None."""
for source in SOURCES.values():
if source.matches(header):
return source
return None

1761
shopdb/data/catalog.json
View File

File diff suppressed because it is too large Load Diff

5
shopdb/plugins/alembic_template.py
View File

@@ -49,8 +49,11 @@ logger = logging.getLogger('alembic.env.plugin')
PLUGIN_TABLE_OWNERS: dict[str, Iterable[str]] = {
'backups': ('backuprevisions',),
'computers': ('computertypes', 'computers', 'computerinstalledapps',
'accessprotocols', 'computeraccess'),
'accessprotocols', 'computeraccess',
'configurationpolicies', 'configurationpolicyactions',
'businesspolicies', 'businesspolicyitems'),
'employees': ('directoryemployees',),
'endpoints': ('endpointstates', 'endpointevents', 'endpointcommands'),
'geenforce': ('manifestscopes', 'manifestentries', 'manifestentrypctypes',
'manifestentryhostnames', 'manifestentrymachinenumbers',
'manifestinusechecks', 'manifestinusecheckprocesses',

6
shopdb/plugins/templates/README.md.tmpl
View File

@@ -17,7 +17,7 @@ This plugin was generated by `flask plugin new $name`. It satisfies the framewor
| You want to... | Do this |
|---|---|
| Add a hook (search, navigation, dashboard widget) | Override the method in `${Name}Plugin`. See `docs/PLUGIN-HOOKS.md`. |
| Add a hook (search, navigation, dashboard widget) | Override the method in `${Name}Plugin`. See the PLUGIN-HOOKS wiki page. |
| Accept external collector data | Override `get_collector_schema()` to return a JSON Schema. See ADR-006. |
| Add another model | Create `models/<other>.py`, export it in `models/__init__.py`, return it in `get_models()`. |
| Add a CLI command | Override `get_cli_commands()` returning a list of Click commands. |
@@ -37,7 +37,7 @@ pytest plugins/$name/tests/
## References
- `docs/PLUGIN-HOOKS.md` - canonical hook reference
- `docs/PLUGIN-QUICKSTART.md` - 30-minute walkthrough
- the PLUGIN-HOOKS wiki page - canonical hook reference
- the PLUGIN-QUICKSTART wiki page - 30-minute walkthrough
- `migrations/adr/ADR-001-asset-as-platform-contract.md` - the platform contract
- `migrations/adr/ADR-002-plugin-versioning.md` - versioning rules

2
shopdb/plugins/templates/schemas/__init__.py.tmpl
View File

@@ -2,5 +2,5 @@
Add schema classes here when you need request/response validation
beyond the simple to_dict() output. The framework wires marshmallow
into the response helpers; see docs/PLUGIN-HOOKS.md for details.
into the response helpers; see the PLUGIN-HOOKS wiki page for details.
"""

76
shopdb/utils/apitoken_auth.py
View File

@@ -107,6 +107,82 @@ def authorized_service_token(scope):
return token
def mint_service_token(scope, name, owner_userid, revoke_existing=True):
"""Mint a managed service token scoped to exactly ONE permission.
The contract-surface way for a plugin to provision an unattended service
credential (an endpoint agent key, a collector key) without importing core
token internals. The returned secret is the ONLY time the full secret is
available - callers reveal it once and store only the prefix afterwards.
scope the single permission the token authorizes (e.g.
collector.ingest.endpoints). The owner must hold it.
name human label shown in the token list UI.
owner_userid the acting admin who owns (and can be audited for) the key.
revoke_existing when True (default), every currently-active token scoped to
EXACTLY this one permission is deactivated first, so
regenerating is a hard cutover: the old agent key stops
working the instant the new one is minted.
Returns (token, secret). Commits.
"""
from shopdb.core.models import User
owner = db.session.get(User, owner_userid)
if owner is None or not owner.isactive or not owner.haspermission(scope):
raise PermissionError(
'Token owner must be an active user holding ' + scope)
if revoke_existing:
for existing in list_service_tokens(scope):
existing.isactive = False
secret = ApiToken.generate_secret()
token = ApiToken(
userid=owner_userid,
name=name,
tokenprefix=ApiToken.prefix_of(secret),
tokenhash=ApiToken.hash_secret(secret),
)
token.scopelist = [scope]
db.session.add(token)
db.session.commit()
return token, secret
def list_service_tokens(scope, active_only=True):
"""Return managed tokens scoped to EXACTLY the single permission `scope`.
Used by a plugin settings page to render the current agent-key state
(prefix, created, last-used) without exposing token internals. Filters in
Python because scopes are stored as a JSON blob, not a queryable column.
"""
query = ApiToken.query
if active_only:
query = query.filter_by(isactive=True)
matches = []
for token in query.order_by(ApiToken.createddate.desc()).all():
scopelist = token.scopelist
if scopelist and len(scopelist) == 1 and scopelist[0] == scope:
matches.append(token)
return matches
def revoke_service_tokens(scope):
"""Deactivate every active managed token scoped to EXACTLY `scope`.
The kill switch: revoking the endpoint agent key stops all endpoint-agent
traffic authorized by a managed token. Returns the count revoked. Commits.
"""
revoked = 0
for token in list_service_tokens(scope):
token.isactive = False
revoked += 1
if revoked:
db.session.commit()
return revoked
def service_token_authorized(scope):
"""True when the current request carries a managed token scoped for `scope`
whose owner is active and holds that permission. Accepts X-API-Key or a

2
shopdb/utils/import_mode.py
View File

@@ -10,7 +10,7 @@ needs two things the normal API withholds:
Both are gated. The caller must BOTH be an admin AND send the request header
`X-Import-Mode: true`. Outside import mode every helper here is a no-op, so
wiring a call into a normal create/update path never changes behavior for
regular users. See docs/IMPORT-API.md for the operator manual.
regular users. See the IMPORT-API wiki page for the operator manual.
"""
from datetime import datetime, timezone