Files

cproudlock 784964c907 Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.

2026-08-28 16:17:17 -04:00

4.2 KiB

Raw Blame History

Collect-ShopDB.ps1 Overview

Overview

Purpose: A unified, read-only "site adoption" collector that runs on any Windows PC, gathers facts about that PC plus any attached eDNC/DNC/UDC machine, and merges them into a shared set of importable CSVs so a site can build its ShopDB inventory one PC at a time.
Compatibility: PowerShell 5.1+; CSV headers are aligned to the ShopDB importer (flask csv import --dir).

Configuration and destinations

Two-stage flow: a fast local stage folder (%ProgramData%\ShopDB\intake) where CSVs are built/merged, and an optional network store (a file share) that the whole site accumulates into.
Command-line params -OutputRoot and -PublishRoot override the in-script CONFIG block.
Other params: -AssetNumber (defaults to hostname), -AssetType/-ComputerType/-MachineType (resolved by name), explicit -RegPath/-JsonPath machine source overrides, and -SkipMachine/-SkipApps switches for partial runs.

Data model (the stores)

Six importable CSVs, each with a defined header order and natural key: vendors.csv, models.csv, operatingsystems.csv, assets.csv, computers.csv, machines.csv.
Column names follow ShopDB conventions (e.g., assetid, modelnumberid, controllervendorid, lastreporteddate).

PC collection (`Get-PcFacts`, read-only)

Uses CIM/WMI (Win32_ComputerSystem, Win32_OperatingSystem, Win32_BIOS) to gather hostname, vendor/manufacturer, model, OS caption, serial number, last boot time, and logged-in user.
Enumerates network interfaces (hardware adapters only: alias, IPv4, MAC).
Enumerates installed applications from the registry Uninstall keys (name, version, publisher), skipping system components - unless -SkipApps is set.

Machine (eDNC/DNC/UDC) collection

Resolve-DncSource searches in priority order: explicit JSON path -> explicit .reg path -> live registry (HKLM\...\GE Aircraft Engines\DNC) -> local .reg export files -> local udc_settings_*.json files.
Three parsers normalize the different source formats into one shape: ConvertFrom-RegExport (parses exported .reg files including dword decoding), ConvertFrom-LiveRegistry, and ConvertFrom-UdcJson.
Extracts machine number, site, control type, DNC type, and connection detail (TCP address/port for ethernet, COM port/baud for serial).
Resolve-ControllerVendor maps control type / NcIF to a controller vendor (e.g., eFOCAS -> FANUC, Siemens, Okuma).

Merge / upsert logic (`Merge-CsvStore`)

Field-level UPSERT keyed on each table's natural key: existing rows are matched case-insensitively, and empty incoming cells never overwrite previously captured values - so partial or repeated runs fill gaps without wiping earlier data.
CSV writing is UTF-8 without BOM, with proper quoting/escaping.

Sidecar files (phase-2 data CSV can't carry)

Per-PC sidecars\computer_<assetnumber>.json: interfaces + installed apps.
Per-machine sidecars\machine_<machinenumber>.json: PC-to-machine link (relationship "controls"), DNC/control details, and connection transport info.

Publishing and output

Optionally mirrors the whole local store to the network share (Copy-Item -Recurse -Force), with a warning if run elevated (network writes should run as standard user).
Prints a summary and the follow-up dry-run command: flask csv import --dir "<OutputRoot>".

Relevance to an "Asset Collector" package

This script is a solid template for one collector in a broader package. Notable patterns worth generalizing across other collectors (existing databases, Intune/Azure exports, etc.):

Normalized intermediate schema (the six CSV stores + JSON sidecars) that all sources converge on, decoupling collection from import.
Idempotent field-level merge so multiple heterogeneous sources can contribute to the same asset without clobbering each other.
Source-priority resolution and per-format parsers - the same shape could wrap an Intune/Azure Get-MgDevice export or a SQL query.
Machines, PCs, Applications are already modeled here; an Intune/Azure source would map naturally onto computers.csv + the installed-apps sidecar, while a MES/CNC database could feed machines.csv.

4.2 KiB Raw Blame History