Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
This commit is contained in:
cproudlock
2026-08-28 16:17:17 -04:00
parent 93ca7587ce
commit 784964c907
149 changed files with 17545 additions and 627 deletions

View File

@@ -39,17 +39,56 @@ def _computer_models():
return None
def _token_is_collector(token):
"""True when this managed token is a collector service token: scoped for
collector.ingest AND its owner actually holds that permission (admins do)."""
def _collector_scopes_for(pluginname=None):
"""The scope names that authorize a collector request.
The generic scope `collector.ingest` authorizes every collector. A plugin
MAY also mint a token scoped to `collector.ingest.<pluginname>` so that
rotating/revoking it stops only that plugin's collector traffic (endpoints
uses this for its regenerable agent kill switch). Both are accepted.
"""
scopes = {COLLECTOR_SCOPE}
if pluginname:
scopes.add(f'{COLLECTOR_SCOPE}.{pluginname}')
return scopes
def _env_fallback_disabled(pluginname=None):
"""True when a plugin has switched OFF the bootstrap env-key fallback.
Once an operator provisions a UI-managed collector token, the plugin sets
its `plugin.<pluginname>.envkeyfallback` setting to false so the old .env
key stops authorizing. That turns key revocation into a real kill switch:
rotating or revoking the managed token cannot be bypassed by a lingering env
key. Absent
setting = fallback stays ON (back-compat, so nothing breaks on upgrade).
"""
if not pluginname:
return False
from shopdb.core.models.setting import Setting
key = f'plugin.{pluginname}.envkeyfallback'
return Setting.get(key, default=True) is False
def _token_is_collector(token, pluginname=None):
"""True when this managed token is a collector service token: scoped for a
collector scope AND its owner actually holds that permission (admins do).
Accepts the generic `collector.ingest` or, when pluginname is given, the
plugin-specific `collector.ingest.<pluginname>` scope.
"""
from shopdb.core.models import User
scopelist = token.scopelist
if not scopelist or COLLECTOR_SCOPE not in scopelist:
if not scopelist:
return False
allowed = _collector_scopes_for(pluginname)
held = [s for s in scopelist if s in allowed]
if not held:
return False
user = db.session.get(User, token.userid)
return (user is not None and user.isactive
and user.haspermission(COLLECTOR_SCOPE))
and any(user.haspermission(s) for s in held))
def _collector_managed_token(api_key):
@@ -73,24 +112,28 @@ def _collector_managed_token(api_key):
return None
def _check_collector_auth(expected_key):
def _check_collector_auth(expected_key, pluginname=None):
"""Authorize a collector request. Returns None when authorized, else the
error_response to return.
Accepts a collector-scoped managed token (Bearer or X-API-Key) OR the env
key via X-API-Key (legacy/bootstrap fallback). Env keys stay supported so
nothing breaks; a valid managed token works even with no env key set.
When pluginname is given, a token scoped to collector.ingest.<pluginname>
also authorizes (in addition to the generic collector.ingest scope).
"""
from shopdb.utils.apitoken_auth import touch_apitoken_lastused
api_key = request.headers.get('X-API-Key')
token = _collector_managed_token(api_key)
if token is not None and _token_is_collector(token):
if token is not None and _token_is_collector(token, pluginname):
touch_apitoken_lastused(token)
return None
if expected_key and api_key == expected_key:
if expected_key and api_key == expected_key \
and not _env_fallback_disabled(pluginname):
return None
# Fail-closed only on pure server misconfiguration: no env key AND the
@@ -200,7 +243,7 @@ def generic_collect(pluginname):
# Per-plugin env key (fallback) OR a collector-scoped managed token. Header
# only (querystring fallback dropped, see require_api_key).
error = _check_collector_auth(_plugin_api_key(pluginname))
error = _check_collector_auth(_plugin_api_key(pluginname), pluginname)
if error is not None:
return error

View File

@@ -8,7 +8,7 @@ from flask_jwt_extended import jwt_required
from werkzeug.utils import secure_filename
from shopdb.extensions import db
from shopdb.core.models import Model
from shopdb.core.models import Model, ModelType
from shopdb.utils.responses import (
success_response,
error_response,
@@ -56,6 +56,12 @@ def list_models():
if modeltype_id := request.args.get('modeltype', type=int):
query = query.filter(Model.modeltypeid == modeltype_id)
# Scope to a model-type category (Equipment, PC, Network, Printer) so a
# subtype form only offers models that make sense for it. Models with no
# type are excluded, since a null category cannot satisfy the filter.
if category := request.args.get('category'):
query = query.join(ModelType).filter(ModelType.category == category)
# Exact-match natural-key lookup for idempotent import. Natural key is
# modelnumber + vendor; pair this with ?vendor=<id> to disambiguate.
if exactmodelnumber := request.args.get('modelnumber'):
@@ -73,6 +79,7 @@ def list_models():
d = m.to_dict()
d['vendor'] = m.vendor.vendor if m.vendor else None
d['modeltype'] = m.modeltype.modeltype if m.modeltype else None
d['category'] = m.modeltype.category if m.modeltype else None
data.append(d)
return paginated_response(data, page, per_page, total)

View File

@@ -1,10 +1,12 @@
"""Settings API routes."""
import os
import re
from flask import Blueprint, request, current_app
from flask_jwt_extended import jwt_required, get_jwt_identity
from werkzeug.utils import secure_filename
from sqlalchemy import select as sa_select
from shopdb.extensions import db, cache
from shopdb.core.models import Setting, AuditLog
@@ -241,6 +243,315 @@ def serve_branding_logo(filename):
return send_upload(_branding_dir(), filename)
def _asset_import_dir():
"""Configured Asset Collector CSV location, or '' when unset."""
setting = Setting.query.filter_by(key='asset_import_csv_location').first()
return (setting.value or '').strip() if setting else ''
# The collector script that gets served (with the site's CSV path baked in) from
# the Import Site Assets page. Copied into the image under scripts/ (see
# Dockerfile), so it is always available next to the running API.
_COLLECTOR_SCRIPT = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))),
'scripts', 'site_imports', 'assetcollector', 'Collect-ShopDB.ps1')
def _embed_store_path(script_text, path):
"""Return the collector script with $NetworkStoreRoot set to path.
The script ships with a BLANK default on that line; rewriting it means an
operator downloads a script already pointed at this facility's CSV store and
does not have to edit PowerShell by hand. A blank path leaves the shipped
blank default untouched, so a site that has not configured
asset_import_csv_location keeps everything local instead of inheriting
another facility's share.
"""
if not path:
return script_text
literal = path.replace("'", "''")
replacement = "$NetworkStoreRoot = '%s'" % literal
# A function replacement is used deliberately: a UNC path is full of
# backslashes, and a string replacement runs them through re's escape
# processing (so \s, \w, a trailing \1 all raise or corrupt). A callable
# returns the text verbatim.
return re.sub(
r"(?m)^\$NetworkStoreRoot\s*=.*$",
lambda _match: replacement,
script_text,
count=1)
@settings_bp.route('/asset-import/script', methods=['GET'])
@require_role('admin')
def download_collector_script():
"""Download Collect-ShopDB.ps1 with this site's CSV path embedded."""
try:
with open(_COLLECTOR_SCRIPT, 'r', encoding='utf-8') as handle:
script_text = handle.read()
except OSError:
return error_response(ErrorCodes.NOT_FOUND,
'Collector script is not available on the server',
http_code=404)
script_text = _embed_store_path(script_text, _asset_import_dir())
response = current_app.response_class(script_text, mimetype='text/plain')
response.headers['Content-Disposition'] = 'attachment; filename=Collect-ShopDB.ps1'
return response
@settings_bp.route('/asset-import/templates', methods=['GET'])
@require_role('admin')
def list_import_templates():
"""List the importable CSV tables in the order they must be uploaded."""
from shopdb.core.services.csvimport import IMPORTABLE
tables = [{'table': name, 'filename': '%s.csv' % name}
for name in IMPORTABLE]
return success_response({'tables': tables})
@settings_bp.route('/asset-import/templates/<tablename>', methods=['GET'])
@require_role('admin')
def download_import_template(tablename):
"""Download a generated CSV template for a single importable table."""
from shopdb.core.services.csvimport import (
IMPORTABLE, generate_template, ImportError_)
if tablename not in IMPORTABLE:
return error_response(ErrorCodes.NOT_FOUND,
'No such importable table: %s' % tablename,
http_code=404)
try:
template_text = generate_template(tablename)
except ImportError_ as exc:
return error_response(ErrorCodes.VALIDATION_ERROR, str(exc))
response = current_app.response_class(template_text, mimetype='text/csv')
response.headers['Content-Disposition'] = (
'attachment; filename=%s.csv' % tablename)
return response
def _existing_hostname_map():
"""Case-insensitive hostname -> assetnumber map for existing computers.
Azure device-group exports align on hostname (their displayName), while the
import engine keys assets on assetnumber. This lets the Azure adapter route
a matched device to its real asset so the import updates in place.
"""
assets = db.metadata.tables['assets']
computers = db.metadata.tables['computers']
query = sa_select(
computers.c.hostname, assets.c.assetnumber
).select_from(
computers.join(assets, computers.c.assetid == assets.c.assetid)
).where(computers.c.hostname.isnot(None))
mapping = {}
for hostname, assetnumber in db.session.execute(query):
if hostname:
mapping[hostname.strip().lower()] = assetnumber
return mapping
@settings_bp.route('/asset-import/import', methods=['POST'])
@require_role('admin')
def import_asset_csvs():
"""Validate uploaded CSVs and, when commit=true, load them.
Mirrors the 'flask csv import' CLI: every file runs through import_csv inside
ONE transaction with a shared resolver, so a computers.csv can resolve the
assets.csv uploaded alongside it. A dry run (the default) rolls the whole
transaction back and reports what WOULD change; commit=true persists it. The
whole run is all-or-nothing - if any file has a row problem, nothing is
written even on a commit.
A 'source' form field selects a foreign adapter (azure, thor); it defaults
to 'native' where uploads are already canonical per-table files. Foreign
uploads are transformed into canonical files first, then run identically.
"""
from sqlalchemy.exc import SQLAlchemyError
from shopdb.core.services.csvimport import (
ImportError_, Resolver, dependency_order, import_csv, table_from_filename)
from shopdb.core.services.importsources import (
ImportSourceError, get_source)
uploads = request.files.getlist('files')
if not uploads:
return error_response(ErrorCodes.VALIDATION_ERROR, 'No CSV files were uploaded')
commit = str(request.form.get('commit', '')).strip().lower() in (
'1', 'true', 'yes', 'on')
# A foreign source (azure, thor) is transformed into canonical files before
# import. 'native' (or nothing) means the files already ARE canonical.
source_key = (request.form.get('source') or 'native').strip().lower()
source = None
if source_key and source_key != 'native':
try:
source = get_source(source_key)
except ImportSourceError as exc:
return error_response(ErrorCodes.VALIDATION_ERROR, str(exc))
source_options = {
'computertype': (request.form.get('computertype') or '').strip(),
'category': (request.form.get('category') or '').strip(),
}
# Azure and Thor align on hostname: their device name is the ShopDB
# computers.hostname, not the assetnumber. Hand the adapter a
# case-insensitive hostname -> existing assetnumber map so devices that
# already exist update in place (keyed to their real asset) instead of
# minting a duplicate asset.
if source is not None and getattr(source, 'key', None) in ('azure', 'thor'):
source_options['hostname_map'] = _existing_hostname_map()
texts = {}
filenames = {}
if source is not None:
# One foreign export becomes several canonical files. Each upload is
# transformed and its canonical outputs merged; a later file's table
# wins, matching the native one-file-per-table rule.
for upload in uploads:
name = upload.filename or ''
if not name.lower().endswith('.csv'):
continue
try:
raw = upload.read().decode('utf-8-sig')
except (UnicodeDecodeError, OSError):
return error_response(ErrorCodes.VALIDATION_ERROR,
'Could not read %s as UTF-8 text' % name)
try:
produced = source.transform(raw, options=source_options)
except ImportSourceError as exc:
return error_response(ErrorCodes.VALIDATION_ERROR,
'%s: %s' % (name, str(exc)))
for table, csv_text in produced.items():
texts[table] = csv_text
filenames[table] = '%s (from %s)' % (table, name)
else:
# Map each upload to a table by filename. A later file with the same
# table name wins, matching the folder-import behaviour.
for upload in uploads:
name = upload.filename or ''
if not name.lower().endswith('.csv'):
continue
table = table_from_filename(name)
try:
texts[table] = upload.read().decode('utf-8-sig')
except (UnicodeDecodeError, OSError):
return error_response(ErrorCodes.VALIDATION_ERROR,
'Could not read %s as UTF-8 text' % name)
filenames[table] = name
if not texts:
return error_response(ErrorCodes.VALIDATION_ERROR,
'None of the uploaded files were CSV files')
ordered = dependency_order(list(texts))
skipped = sorted(set(texts) - set(ordered))
reconcile_setting = Setting.query.filter_by(
key='asset_import_reconcile_threshold').first()
try:
reconcile_threshold = float(reconcile_setting.value) if reconcile_setting and reconcile_setting.value else 0.0
except (TypeError, ValueError):
reconcile_threshold = 0.0
resolver = Resolver(reconcile_threshold=reconcile_threshold)
files_report = []
problem_count = 0
warning_count = 0
reconciled_count = 0
request_error = None
try:
for table in ordered:
try:
result = import_csv(table, texts[table], resolver=resolver, commit=False)
except ImportError_ as exc:
request_error = '%s: %s' % (filenames[table], str(exc))
break
except SQLAlchemyError as exc:
db.session.rollback()
request_error = '%s: database error: %s' % (
filenames[table], str(exc).split(chr(10))[0])
break
problems = [
{'line': p.line, 'column': p.column, 'message': p.message}
for p in result.problems
]
# Amber tier: a warning is a finding beside a row that still
# imported (an unresolved machine link, a contested bay), NOT a
# reason to fail the file. Reported alongside problems so the UI can
# show Errors / Warnings / Info distinctly, but never gates commit.
warnings = [
{'line': w.line, 'column': w.column, 'reason': w.reason,
'action': w.action, 'subject': w.subject, 'target': w.target}
for w in result.warnings
]
problem_count += len(problems)
warning_count += len(warnings)
reconciled_count += len(result.reconciliations)
files_report.append({
'table': table,
'filename': filenames[table],
'created': result.created,
'updated': result.updated,
'problems': problems,
'warnings': warnings,
'reconciliations': result.reconciliations,
})
applied = False
if request_error is not None:
db.session.rollback()
elif problem_count > 0:
# A row problem anywhere fails the whole run, so nothing is written
# even when commit was asked for.
db.session.rollback()
elif commit:
# One audit row for the whole run, written INSIDE the same
# transaction so it cannot survive a rollback or go missing from a
# commit. Every other write in this app is auditable; a bulk import
# that creates thousands of rows was the one path where nobody could
# answer "who loaded this, and when".
AuditLog.log(
'imported', 'SiteAssets',
entityname=', '.join(f['table'] for f in files_report) or 'none',
changes={
'files': [f['table'] for f in files_report],
'created': sum(f['created'] for f in files_report),
'updated': sum(f['updated'] for f in files_report),
'reconciled': reconciled_count,
})
db.session.commit()
applied = True
else:
db.session.rollback()
except Exception:
db.session.rollback()
raise
total_created = sum(f['created'] for f in files_report)
total_updated = sum(f['updated'] for f in files_report)
return success_response({
'committed': applied,
'requested_commit': commit,
'request_error': request_error,
'files': files_report,
'skipped': skipped,
'problem_count': problem_count,
'warning_count': warning_count,
'reconciled_count': reconciled_count,
'reconcile_threshold': reconcile_threshold,
'total_created': total_created,
'total_updated': total_updated,
})
@settings_bp.route('', methods=['GET'])
@jwt_required(optional=True)
def list_settings():
@@ -568,6 +879,20 @@ def build_default_settings():
'category': 'site',
'description': 'Email domain appended to a contact SSO to build email/Teams links (sso@domain). Blank disables contact action buttons.'
},
{
'key': 'asset_import_csv_location',
'value': '',
'valuetype': 'string',
'category': 'site',
'description': r'Shared network location where the Asset Collector script deposits its CSVs and where ShopDB reads them for import (e.g. \\server.fqdn\share\folder). This path is baked into the downloadable collector script. Blank = not configured.'
},
{
'key': 'asset_import_reconcile_threshold',
'value': '0',
'valuetype': 'string',
'category': 'site',
'description': 'Similarity (0 to 1) a fuzzy lookup match must reach for the asset importer to reconcile a near-miss name (e.g. "Dell Inc." to an existing "Dell") onto an existing vendor, model, type, status, business unit, or location. Reconciled matches are applied but reported for review. 0 (the default) disables fuzzy reconciliation - names must match exactly after normalization. Labels whose digit runs differ (e.g. "Bay 310" vs "Bay 311") are never fuzzy-matched. Operating systems and asset rows are never reconciled.'
},
]
# Site branding. Each instance (ADR-004) can replace the shipped GE defaults