ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Browse Source 
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
This commit is contained in:
cproudlock
2026-08-28 16:17:17 -04:00
parent 93ca7587ce
commit 784964c907
149 changed files with 17545 additions and 627 deletions
Download Patch File Download Diff File Expand all files Collapse all files

358
plugins/computers/plugin.py
View File

@@ -12,8 +12,16 @@ from shopdb.plugins.base import BasePlugin, PluginMeta
from shopdb.api import db, AssetType
from .models import Computer, ComputerType, ComputerInstalledApp, AccessProtocol, ComputerAccess
from .models import (
ConfigurationPolicy,
ConfigurationPolicyAction,
BusinessPolicy,
BusinessPolicyItem,
)
from .api import computers_bp
from .policyflag import policy_flag_enabled as _policy_flag_enabled
logger = logging.getLogger(__name__)
# Marker stamped on PC->measuringtool "controls" links this collector creates
@@ -32,6 +40,14 @@ MEASURINGTOOL_LINK_ORIGIN = 'collector:measuringtool'
# touched.
MACHINE_LINK_ORIGIN = 'collector:machine'
# Marker stamped on the PC->machine link built from a CSV import's machine
# number column. A DISTINCT origin from the collector's so the two provenances
# stay separable: an import establishes a link once from a spreadsheet, the
# collector maintains it continuously from the floor, and a site may use one,
# the other, or both. Same archive discipline - only rows carrying this label
# are touched by a re-import.
MACHINE_LINK_IMPORT_ORIGIN = 'import:machine'
# Marker stamped on both links a part-marker PC produces: PC controls marker,
# and marker partof the operation it serves. Same archive discipline again.
PARTMARKER_LINK_ORIGIN = 'collector:partmarker'
@@ -84,12 +100,229 @@ class ComputersPlugin(BasePlugin):
def get_models(self) -> List[Type]:
"""Return list of SQLAlchemy model classes."""
return [Computer, ComputerType, ComputerInstalledApp, AccessProtocol, ComputerAccess]
return [Computer, ComputerType, ComputerInstalledApp, AccessProtocol, ComputerAccess,
ConfigurationPolicy, ConfigurationPolicyAction, BusinessPolicy, BusinessPolicyItem]
def init_app(self, app: Flask, db_instance) -> None:
"""Initialize plugin with Flask app."""
self._register_import_hook()
logger.info(f"Computers plugin initialized (v{self.meta.version})")
# -- bulk-import machine link -------------------------------------------
def _register_import_hook(self):
"""Own a `machineassetnumber` column on computers.csv.
A site with a spreadsheet knows "this PC drives machine 3015" and wants
to write that number in the PC row, not hand-build an assetrelationships
sheet keyed on ids it never sees. The core importer stays ignorant of
what the column means: it accepts the header, keeps it out of the
computers write, and hands the value to this hook after each PC row is
written, so the PC's asset id exists. Registration is idempotent, so a
plugin reload or a test re-init does not double-fire.
"""
from shopdb.api import register_import_hook
register_import_hook(
'computers',
{'machineassetnumber':
'the asset number of the machine this PC drives (from '
'machines.csv); builds the PC-controls-machine link. A number '
'that has no machine, or one already driven by another PC, is '
'reported as a warning and skipped - the PC still imports.'},
self._import_machine_link)
def _import_machine_link(self, rowvalues, virtualvalues, line):
"""Build a PC->machine controls link from an import row's machine number.
Runs inside the importer's transaction - a dry run rolls it back - so it
reuses the collector's mature _sync_machine_link rather than inventing a
second linker, but under the import origin (so an import never archives
a collector's rows and vice versa) and in preview mode (so a bulk import
never emails an alert per row; the findings are the report's amber
warnings instead).
Returns RowWarning list, never raises: a machine number that resolves to
no asset is an amber note beside a PC that still imports, not a failed
file. Every warning names the PC, the number, and the one fix.
"""
from shopdb.api import RowWarning
machinenumber = (virtualvalues.get('machineassetnumber') or '').strip()
if not machinenumber:
return []
assetid = rowvalues.get('assetid')
comp = Computer.query.filter_by(assetid=assetid).first() if assetid else None
if comp is None:
# The PC row was written but its Computer extension is not readable
# here - nothing to link to. Amber, not fatal.
return [RowWarning(
line, 'machineassetnumber',
reason='PC row not found to link to machine {!r}'.format(
machinenumber),
action='re-run the import; if it recurs, check the computers '
'row imported',
target=machinenumber)]
hostname = comp.hostname or (assetid and str(assetid)) or '?'
# IN-FILE FAN-OUT GUARD. Two rows in one computers.csv both naming the
# same machine is a spreadsheet mistake - a copied cell, a wrong
# workstation number - not a swap. But _incumbent_has_yielded reads a
# freshly-imported PC (no lastreporteddate) as having yielded, so
# without this the SECOND row would silently archive the first row's
# link and steal the machine, and the file would import clean with the
# wrong PC on the bay. So the first claim on a machine this run wins and
# is remembered; a later row for the same machine is warned and skipped.
# The claim set lives on db.session.info, which Flask-SQLAlchemy resets
# at request teardown, so it is scoped to exactly this one import and
# never leaks into the next.
machine = self._asset_by_number(machinenumber)
if machine is not None and machine.assetid == assetid:
# The number names this very PC. Happens on a PC first seen before
# the machine-number convention, whose own asset number IS the
# machine number. Skip - a PC does not control itself.
return [RowWarning(
line, 'machineassetnumber',
reason='machine number {!r} is this PC\'s own asset number; '
'not linked'.format(machinenumber),
action='give this PC a hostname-based asset number, or point '
'machineassetnumber at the machine it drives',
subject=hostname, target=machinenumber)]
if machine is not None:
# A location-only machine is a virtual bay marker, not a real
# machine, so nothing controls it. Skip rather than build a link
# that would read as a PC driving a location.
skip = self._location_only_skip(
machine, hostname, machinenumber, line)
if skip is not None:
return [skip]
if machine is not None and machine.assetid != assetid:
# ALREADY-CONTROLS-DIFFERENT. Unlike the collector - which reads a
# new machine number as a PC moved to a new bay and silently
# re-bays it - an import naming a machine when this PC already
# actively controls a DIFFERENT one is far more likely a wrong
# workstation number in the sheet than a real move. Warn and skip
# rather than archive a link a person may have set on purpose.
other = self._active_controlled_machine(assetid, machine.assetid)
if other is not None:
return [RowWarning(
line, 'machineassetnumber',
reason='{} already controls machine {}; not moved to {}'
.format(hostname, other, machine.assetnumber),
action='if this PC really moved, clear the existing link '
'first; otherwise correct the machine number on '
'this row',
subject=hostname, target=machinenumber)]
if machine is not None and machine.assetid != assetid:
claimed = db.session.info.setdefault('import_machine_claims', {})
# Keyed by machine asset id -> (claimant PC asset id, its hostname).
# The id is what the guard compares on (the same PC re-stating its
# own link is not a conflict); the hostname is only for the message.
firstclaimant = claimed.get(machine.assetid)
if firstclaimant is not None and firstclaimant[0] != assetid:
return [RowWarning(
line, 'machineassetnumber',
reason='machine {} was already claimed by {} earlier in '
'this import; {} not linked'.format(
machine.assetnumber, firstclaimant[1], hostname),
action='a machine drives one PC - correct the machine '
'number on one of these rows so each PC names its '
'own machine',
subject=hostname, target=machinenumber)]
claimed[machine.assetid] = (assetid, hostname)
# _sync_machine_link appends plain strings to its warnings list. Collect
# them and translate to structured, actionable RowWarnings so the UI can
# show the PC, the machine number, and the corrective step.
rawwarnings = []
self._sync_machine_link(
comp, machinenumber, rawwarnings,
origin_label=MACHINE_LINK_IMPORT_ORIGIN, preview=True)
out = []
for text in rawwarnings:
out.append(RowWarning(
line, 'machineassetnumber',
reason=text,
action=self._machine_link_action(text, machinenumber),
subject=hostname, target=machinenumber))
return out
@staticmethod
def _active_controlled_machine(pcassetid, excludeassetid):
"""Asset number of a machine this PC actively controls under the import
origin, other than excludeassetid, or None.
Scoped to MACHINE_LINK_IMPORT_ORIGIN so an import only reasons about its
own links: a collector link or a hand-made one is the collector's or a
person's to move, not the import's to block on.
"""
from shopdb.api import AssetRelationship, RelationshipType, Asset
controls = RelationshipType.query.filter_by(
relationshiptype='controls').first()
if not controls:
return None
rel = AssetRelationship.query.filter(
AssetRelationship.sourceassetid == pcassetid,
AssetRelationship.relationshiptypeid == controls.relationshiptypeid,
AssetRelationship.label == MACHINE_LINK_IMPORT_ORIGIN,
AssetRelationship.targetassetid != excludeassetid,
AssetRelationship.isactive.is_(True),
).first()
if rel is None:
return None
asset = db.session.get(Asset, rel.targetassetid)
return asset.assetnumber if asset else str(rel.targetassetid)
@staticmethod
def _location_only_skip(machine, hostname, machinenumber, line):
"""A RowWarning if the resolved asset is a location-only marker, else None.
A machine flagged islocationonly is a virtual bay, not a real machine,
so a PC does not control it. Guarded behind the machines plugin being
installed - a site without it cannot have a location-only machine, and
the import must not fail because the model is absent.
"""
from shopdb.api import RowWarning
try:
from plugins.machines.models import Machine
except ImportError:
return None
row = Machine.query.filter_by(assetid=machine.assetid).first()
if row is None or not row.islocationonly:
return None
return RowWarning(
line, 'machineassetnumber',
reason='machine {} is a location-only marker, not a real machine; '
'not linked'.format(machine.assetnumber),
action='point machineassetnumber at a real machine, or clear it '
'for this PC',
subject=hostname, target=machinenumber)
@staticmethod
def _machine_link_action(text, machinenumber):
"""The one corrective step for a machine-link warning, by its shape."""
lowered = text.lower()
if 'no asset for machine number' in lowered:
return ('create the machine asset with number {!r} (import '
'machines.csv first), or correct the number on this PC '
'row'.format(machinenumber))
if 'still held by' in lowered:
return ('confirm which PC drives this machine; retire or move the '
'old PC off In Use to hand the machine over')
if 'taken over from' in lowered:
return ('expected on a PC swap; set the old PC to Retired, '
'Inventory or In Repair')
if 'relationship type missing' in lowered:
return 'run flask seed reference-data, then re-import'
return 'review this PC and its machine number'
# -- ADR-006 collector contract -----------------------------------------
def get_collector_schema(self) -> Optional[Dict]:
@@ -173,6 +406,36 @@ class ComputersPlugin(BasePlugin):
'site\'s URL and ranges',
'position': 26,
},
] + self._policy_settings_cards()
def _policy_settings_cards(self):
"""Policy admin cards, shown only while the feature flag is on.
Gating the cards on the switch keeps the settings menu honest: the
Configuration/Business Policy screens appear exactly when the feature is
enabled and vanish on rollback, matching the 404 the routes return.
"""
if not self._configpolicies_enabled_safe():
return []
return [
{
'group': 'Computers',
'to': '/settings/configuration-policies',
'icon': 'shield',
'title': 'Configuration policies',
'description': 'Reusable rules binding endpoint actions to PC '
'types and applications',
'position': 27,
},
{
'group': 'Computers',
'to': '/settings/business-policies',
'icon': 'layers',
'title': 'Business policies',
'description': 'Ordered collections of configuration policies '
'assigned by priority',
'position': 28,
},
]
def get_settings_defaults(self) -> List[dict]:
@@ -214,8 +477,41 @@ class ComputersPlugin(BasePlugin):
'Off while machine numbers are shared between '
'PCs; the collector response still warns.',
},
{
'key': 'computers_configpolicies_enabled',
'value': 'false',
'valuetype': 'boolean',
'category': 'computers',
'description': 'Master switch for RMM-style Configuration and '
'Business Policies. Off by default: models exist '
'but the CRUD routes, WebUI, and endpoint '
'enforcement stay inert so the whole feature can '
'be rolled back without a schema change.',
},
]
def _configpolicies_enabled(self):
"""Whether the Configuration/Business Policy feature is turned on.
Off by default. This is the single exit gate for the policy scope: the
tables and models load regardless (SQLAlchemy needs them), but every
route, panel, and enforcement path checks this first and no-ops while it
is false, so the feature can be disabled without a migration.
"""
return _policy_flag_enabled()
def _configpolicies_enabled_safe(self):
"""Flag read that never raises when the settings table is absent.
get_settings_cards can be called in contexts without a database (the
plugin contract tests build hooks against a bare app), so a missing
settings table must read as "off" rather than propagate.
"""
try:
return _policy_flag_enabled()
except Exception:
return False
def _machinelink_alerts_enabled(self):
"""Whether the PC-to-machine alerts may be sent.
@@ -534,7 +830,8 @@ class ComputersPlugin(BasePlugin):
'with POST /api/printers/assignments/seed-from-observed.'
% ' and '.join(sent))
def _sync_machine_link(self, comp, machinenumber, warnings):
def _sync_machine_link(self, comp, machinenumber, warnings,
origin_label=MACHINE_LINK_ORIGIN, preview=False):
"""Link a PC to the machine it drives, from the reported machine number.
Until this existed the machine number was collected and then discarded,
@@ -556,6 +853,15 @@ class ComputersPlugin(BasePlugin):
cadence, alerting on every pass. The incumbent therefore keeps the
machine while it is still alive, and the challenger is recorded as a
dormant link. See _incumbent_has_yielded for what alive means.
origin_label separates provenance: the collector stamps
MACHINE_LINK_ORIGIN, a CSV import stamps MACHINE_LINK_IMPORT_ORIGIN, and
each only ever archives its own rows so the two sources coexist.
preview=True is for a dry-run import: DB writes still happen in the
caller's transaction (which it rolls back), but the alert fan-out - the
one side effect a rollback cannot undo - is suppressed so a preview
never emails anyone.
"""
from shopdb.api import AssetRelationship, RelationshipType, Asset
@@ -587,7 +893,7 @@ class ComputersPlugin(BasePlugin):
mine = AssetRelationship.query.filter(
AssetRelationship.sourceassetid == pcasset.assetid,
AssetRelationship.relationshiptypeid == controls.relationshiptypeid,
AssetRelationship.label == MACHINE_LINK_ORIGIN,
AssetRelationship.label == origin_label,
).all()
link = None
@@ -597,11 +903,23 @@ class ComputersPlugin(BasePlugin):
else:
rel.isactive = False
# The unique constraint on assetrelationships is (source, target, type)
# and does NOT include label, so an import (import:machine) and the
# collector (collector:machine) cannot each hold their OWN row for the
# same PC->machine control. If our label-scoped lookup found nothing but
# a row for this exact triple exists under the other provenance, adopt
# it rather than insert a duplicate that would violate the constraint.
if link is None:
link = AssetRelationship.query.filter_by(
sourceassetid=pcasset.assetid,
targetassetid=machine.assetid,
relationshiptypeid=controls.relationshiptypeid).first()
# Whoever actively holds this machine now, if it is not this PC.
held = AssetRelationship.query.filter(
AssetRelationship.targetassetid == machine.assetid,
AssetRelationship.relationshiptypeid == controls.relationshiptypeid,
AssetRelationship.label == MACHINE_LINK_ORIGIN,
AssetRelationship.label == origin_label,
AssetRelationship.sourceassetid != pcasset.assetid,
AssetRelationship.isactive.is_(True),
).all()
@@ -624,9 +942,10 @@ class ComputersPlugin(BasePlugin):
sourceassetid=pcasset.assetid,
targetassetid=machine.assetid,
relationshiptypeid=controls.relationshiptypeid,
label=MACHINE_LINK_ORIGIN,
label=origin_label,
isactive=False))
self._alert_machine_contested(names, comp, machine)
if not preview:
self._alert_machine_contested(names, comp, machine)
else:
link.isactive = False
return [{'assetid': machine.assetid,
@@ -640,7 +959,7 @@ class ComputersPlugin(BasePlugin):
sourceassetid=pcasset.assetid,
targetassetid=machine.assetid,
relationshiptypeid=controls.relationshiptypeid,
label=MACHINE_LINK_ORIGIN,
label=origin_label,
isactive=True)
db.session.add(link)
else:
@@ -654,7 +973,8 @@ class ComputersPlugin(BasePlugin):
warnings.append(
'machine {} was taken over from {}; check that PC'.format(
machine.assetnumber, oldname))
self._alert_pc_superseded(oldname, comp, machine)
if not preview:
self._alert_pc_superseded(oldname, comp, machine)
return [{'assetid': machine.assetid,
'machinenumber': machine.assetnumber,
@@ -757,8 +1077,8 @@ class ComputersPlugin(BasePlugin):
#
# A row made by hand carries neither label and is deliberately NOT
# counted: a person's link is not the collector's to archive.
ourlabels = (label, MACHINE_LINK_ORIGIN, MEASURINGTOOL_LINK_ORIGIN,
PARTMARKER_LINK_ORIGIN)
ourlabels = (label, MACHINE_LINK_ORIGIN, MACHINE_LINK_IMPORT_ORIGIN,
MEASURINGTOOL_LINK_ORIGIN, PARTMARKER_LINK_ORIGIN)
others = AssetRelationship.query.filter(
AssetRelationship.targetassetid == deviceasset.assetid,
AssetRelationship.relationshiptypeid == controls.relationshiptypeid,
@@ -1310,15 +1630,17 @@ class ComputersPlugin(BasePlugin):
sourceassetid=pcasset.assetid,
targetassetid=adopted.assetid,
relationshiptypeid=controls.relationshiptypeid).first()
if claimed is not None and claimed.label == MACHINE_LINK_ORIGIN:
if claimed is not None and claimed.label in (
MACHINE_LINK_ORIGIN, MACHINE_LINK_IMPORT_ORIGIN):
# LEAVE IT ALONE. On a CMM the instrument IS the reported bay,
# so the machine sync has already made the row for this exact
# triple - and it finds its own rows BY that label. Relabelling
# hid the link from it, so the next cycle built a second row for
# the same triple and MySQL rejected it: the PC reported 200,
# then 500 forever after. The row's lifecycle (including staying
# dormant while a bay is contested) belongs to the machine sync;
# adoption only needs the identity, which it has.
# so the machine sync (or a CSV import's machine link) has
# already made the row for this exact triple - and each finds
# its own rows BY that label. Relabelling hid the link from it,
# so the next cycle built a second row for the same triple and
# MySQL rejected it: the PC reported 200, then 500 forever
# after. The row's lifecycle (including staying dormant while a
# bay is contested) belongs to the machine sync; adoption only
# needs the identity, which it has.
pass
elif claimed is not None:
# Relabelling is what adoption MEANS for any other row: the link