Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
This commit is contained in:
cproudlock
2026-08-28 16:17:17 -04:00
parent 93ca7587ce
commit 784964c907
149 changed files with 17545 additions and 627 deletions

View File

@@ -0,0 +1,594 @@
#Requires -Version 5.1
[CmdletBinding()]
param(
# Where CSVs are collected/accumulated. Overrides the in-script CONFIG block
# below. See the CONFIG block for the easy place to set this permanently.
[Parameter(Mandatory = $false)]
[string]$OutputRoot,
# Optional network share to mirror the store into after a run. Overrides the
# in-script CONFIG block below.
[Parameter(Mandatory = $false)]
[string]$PublishRoot,
# Business identifier for this PC (workstation number). Defaults to hostname.
[Parameter(Mandatory = $false)]
[string]$AssetNumber,
# Value written into assets.assettypeid for the PC (resolved by name).
[Parameter(Mandatory = $false)]
[string]$AssetType = 'Computer',
# Value written into computers.computertypeid (resolved by name).
[Parameter(Mandatory = $false)]
[string]$ComputerType,
# Value written into machines.machinetypeid (resolved by name).
[Parameter(Mandatory = $false)]
[string]$MachineType,
# Explicit eDNC/UDC source overrides. When omitted, standard on-asset
# locations are searched.
[Parameter(Mandatory = $false)]
[string]$RegPath,
[Parameter(Mandatory = $false)]
[string]$JsonPath,
# Skip switches for partial runs.
[switch]$SkipMachine,
[switch]$SkipApps
)
<#
.SYNOPSIS
Unified ShopDB site-adoption collector. Runs on any PC, collects the PC and any
attached eDNC/DNC machine, and UPSERTS the results into a shared set of
importable CSVs so a site can build its inventory PC by PC.
.DESCRIPTION
Accumulating store under OutputRoot:
vendors.csv, models.csv, operatingsystems.csv, assets.csv,
computers.csv, machines.csv
Rows are merged field-by-field on each table's natural key (a later run fills
gaps and refreshes values without wiping data an earlier run captured).
Phase-2 data that CSV cannot carry is written per entity under OutputRoot\sidecars:
computer_<assetnumber>.json (interfaces, installed apps)
machine_<machinenumber>.json (PC<->Machine link, connection detail)
Read-only against the machine. Local reads may run elevated; publishing to a
share runs in the current user context only.
.NOTES
PS 5.1 compatible. Headers match the ShopDB importer (see CSV-SCHEMA.md).
#>
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
# =============================================================================
# CONFIG - SET YOUR CSV DESTINATIONS HERE
# -----------------------------------------------------------------------------
# Two-stage flow so you can collect locally (possibly elevated) and then append
# the captured rows to a CSV store that lives on a network share:
#
# 1) LocalStageRoot Fast, always-writable local folder where CSVs are built
# and merged. Collection can run elevated here.
# 2) NetworkStoreRoot The shared CSV store the whole site accumulates into and
# that you later point 'flask csv import --dir' at. Leave
# blank ('') to keep everything local. When set, the merged
# store is copied here at the end of the run. Network
# writes must happen as a STANDARD user - do NOT run the
# publish step elevated.
#
# Example network path: '\\fileserver\shopdb$\intake'
#
# Command-line -OutputRoot / -PublishRoot still override these values.
# =============================================================================
$LocalStageRoot = Join-Path $env:ProgramData 'ShopDB\intake'
# Ship blank so a site that has not configured its share keeps everything local
# instead of publishing to another site's path. Set this to your own share (or
# pass -PublishRoot), e.g. '\\fileserver\shopdb$\intake'. The download endpoint
# fills this in from asset_import_csv_location when the site has set it.
$NetworkStoreRoot = ''
# Resolve effective destinations (command-line params win over CONFIG above).
if (-not $OutputRoot) { $OutputRoot = $LocalStageRoot }
if (-not $PublishRoot) { $PublishRoot = $NetworkStoreRoot }
# --- Store definitions (header order + natural key per importable table) -------
$script:Stores = [ordered]@{
'vendors.csv' = @{ key = 'vendor'; header = @('vendor') }
'models.csv' = @{ key = 'modelnumber'; header = @('modelnumber','vendorid','modeltypeid') }
'operatingsystems.csv' = @{ key = 'osname'; header = @('osname') }
'assets.csv' = @{ key = 'assetnumber'; header = @('assetnumber','assettypeid','name','statusid','serialnumber','gaugelabreference','maintenancereference','locationid','businessunitid','notes') }
'computers.csv' = @{ key = 'assetid'; header = @('assetid','hostname','computertypeid','osid','vendorid','modelnumberid','loggedinuser','lastreporteddate','lastboottime') }
'machines.csv' = @{ key = 'assetid'; header = @('assetid','machinetypeid','vendorid','modelnumberid','controllervendorid','controllermodelid','requiresmanualconfig','islocationonly','notes') }
}
# --- Shared helpers -----------------------------------------------------------
function ConvertTo-CsvField {
param([AllowNull()]$Value)
if ($null -eq $Value) { return '' }
$text = [string]$Value
if ($text -match '[",\r\n]') { return '"' + ($text -replace '"', '""') + '"' }
return $text
}
function Get-Prop {
param([AllowNull()]$Object, [Parameter(Mandatory = $true)][string]$Name)
if ($null -eq $Object) { return $null }
$prop = $Object.PSObject.Properties[$Name]
if ($null -eq $prop) { return $null }
return $prop.Value
}
function Get-NonEmpty {
param([AllowNull()]$Value)
if ($null -ne $Value -and -not [string]::IsNullOrWhiteSpace([string]$Value)) {
return ([string]$Value).Trim()
}
return $null
}
function Get-FirstNonEmpty {
param([object[]]$Values)
foreach ($v in $Values) { $n = Get-NonEmpty $v; if ($n) { return $n } }
return $null
}
function Format-ShopDbDate {
param([AllowNull()]$Value)
if ($null -eq $Value) { return $null }
try { return ([datetime]$Value).ToString('yyyy-MM-dd HH:mm:ss') } catch { return $null }
}
function Write-Utf8NoBom {
param([string]$Path, [string]$Content)
$encoding = New-Object System.Text.UTF8Encoding($false)
[System.IO.File]::WriteAllText($Path, $Content, $encoding)
}
# Field-level upsert into a persistent CSV store, keyed by natural key. Existing
# values are kept when the incoming cell is empty, so a partial run never blanks
# out data an earlier run captured.
function Merge-CsvStore {
param(
[Parameter(Mandatory = $true)][string]$Path,
[Parameter(Mandatory = $true)][string[]]$Header,
[Parameter(Mandatory = $true)][string]$Key,
[Parameter(Mandatory = $true)][AllowNull()][object[]]$IncomingRows
)
$index = [ordered]@{}
if (Test-Path -LiteralPath $Path) {
foreach ($existing in @(Import-Csv -LiteralPath $Path)) {
$keyVal = Get-NonEmpty (Get-Prop $existing $Key)
if (-not $keyVal) { continue }
$index[$keyVal.ToLowerInvariant()] = $existing
}
}
$changed = 0
foreach ($row in $IncomingRows) {
if ($null -eq $row) { continue }
$keyVal = Get-NonEmpty (Get-Prop $row $Key)
if (-not $keyVal) { continue }
$lc = $keyVal.ToLowerInvariant()
if ($index.Contains($lc)) {
$target = $index[$lc]
foreach ($col in $Header) {
$incoming = Get-NonEmpty (Get-Prop $row $col)
if ($incoming) {
if ($target.PSObject.Properties[$col]) { $target.$col = $incoming }
else { $target | Add-Member -NotePropertyName $col -NotePropertyValue $incoming -Force }
}
}
} else {
$fresh = [ordered]@{}
foreach ($col in $Header) { $fresh[$col] = Get-NonEmpty (Get-Prop $row $col) }
$index[$lc] = [pscustomobject]$fresh
}
$changed++
}
$lines = New-Object System.Collections.Generic.List[string]
$lines.Add(($Header -join ','))
foreach ($stored in $index.Values) {
$cells = foreach ($col in $Header) { ConvertTo-CsvField (Get-Prop $stored $col) }
$lines.Add(($cells -join ','))
}
Write-Utf8NoBom -Path $Path -Content (($lines -join "`r`n") + "`r`n")
return $changed
}
# --- eDNC/UDC parsing (registry, .reg export, UDC settings json) --------------
function Resolve-ControllerVendor {
param([AllowNull()][string]$ControlType, [AllowNull()][string]$NcIF)
$ct = ('' + $ControlType).Trim().ToUpperInvariant()
$nc = ('' + $NcIF).Trim().ToUpperInvariant()
if ($nc -eq 'EFOCAS' -or $ct -eq 'EFOCAS') { return 'FANUC' }
if ($ct -eq 'SIEMENS') { return 'Siemens' }
if ($ct -eq 'OKUMA') { return 'Okuma' }
return $null
}
function Read-RegExport {
param([Parameter(Mandatory = $true)][string]$Path)
$lines = @(Get-Content -LiteralPath $Path -Encoding Unicode -ErrorAction Stop)
$firstMeaningful = ($lines | Where-Object { $_ -match '\S' } | Select-Object -First 1)
if (-not ($firstMeaningful -match '^(REGEDIT4|Windows Registry Editor Version)')) {
$lines = @(Get-Content -LiteralPath $Path -Encoding Default -ErrorAction Stop)
}
$data = @{}
$currentKey = $null
foreach ($raw in $lines) {
$line = $raw.Trim()
if ([string]::IsNullOrWhiteSpace($line)) { continue }
if ($line -match '^(REGEDIT4|Windows Registry Editor Version)') { continue }
if ($line -match '^\s*;') { continue }
if ($line -match '^\[(?<key>[^\]]+)\]$') {
$currentKey = $Matches['key']
if (-not $data.ContainsKey($currentKey)) { $data[$currentKey] = @{} }
continue
}
if (-not $currentKey) { continue }
if ($line -match '^"(?<name>[^"]+)"=(?<rhs>.+)$') {
$name = $Matches['name']
$rhs = $Matches['rhs'].Trim()
if ($rhs -match '^"(?<s>.*)"$') { $val = $Matches['s'] }
elseif ($rhs -match '^dword:(?<hex>[0-9a-fA-F]{8})$') { $val = [Convert]::ToInt32($Matches['hex'], 16) }
else { $val = $rhs }
$data[$currentKey][$name] = $val
}
}
return $data
}
function Get-RegExportValue {
param([hashtable]$Reg, [string]$KeySuffix, [string]$Name)
foreach ($key in $Reg.Keys) {
if ($key -match ('\\' + [regex]::Escape($KeySuffix) + '$')) {
if ($Reg[$key].ContainsKey($Name)) { return $Reg[$key][$Name] }
}
}
return $null
}
function ConvertFrom-RegExport {
param([hashtable]$Reg)
[ordered]@{
source = 'reg'
machinenumber = Get-NonEmpty (Get-RegExportValue $Reg 'General' 'MachineNo')
site = Get-NonEmpty (Get-RegExportValue $Reg 'General' 'Site')
controltype = Get-NonEmpty (Get-RegExportValue $Reg 'General' 'Cnc')
ncif = Get-NonEmpty (Get-RegExportValue $Reg 'General' 'NcIF')
dnctype = 'eDNC'
tcpaddress = Get-NonEmpty (Get-RegExportValue $Reg 'eFocas' 'IpAddr')
tcpport = Get-NonEmpty (Get-RegExportValue $Reg 'eFocas' 'SocketNo')
comport = Get-NonEmpty (Get-RegExportValue $Reg 'PPDCS' 'Port Id')
baudrate = Get-NonEmpty (Get-RegExportValue $Reg 'PPDCS' 'Baud')
}
}
function Get-RegistryValue {
param([string]$Path, [string]$Name)
if (-not (Test-Path -LiteralPath $Path)) { return $null }
return Get-Prop (Get-ItemProperty -LiteralPath $Path -ErrorAction SilentlyContinue) $Name
}
function ConvertFrom-LiveRegistry {
param([string]$Base)
if (-not (Test-Path -LiteralPath $Base)) { return $null }
$general = Join-Path $Base 'General'
$efocas = Join-Path $Base 'eFocas'
$ppdcs = Join-Path $Base 'PPDCS'
[ordered]@{
source = 'registry'
machinenumber = Get-NonEmpty (Get-RegistryValue $general 'MachineNo')
site = Get-NonEmpty (Get-RegistryValue $general 'Site')
controltype = Get-NonEmpty (Get-RegistryValue $general 'Cnc')
ncif = Get-NonEmpty (Get-RegistryValue $general 'NcIF')
dnctype = 'eDNC'
tcpaddress = Get-NonEmpty (Get-RegistryValue $efocas 'IpAddr')
tcpport = Get-NonEmpty (Get-RegistryValue $efocas 'SocketNo')
comport = Get-NonEmpty (Get-RegistryValue $ppdcs 'Port Id')
baudrate = Get-NonEmpty (Get-RegistryValue $ppdcs 'Baud')
}
}
function ConvertFrom-UdcJson {
param([string]$Path)
$obj = Get-Content -LiteralPath $Path -Raw -ErrorAction Stop | ConvertFrom-Json
$general = Get-Prop $obj 'GeneralSettings'
$ethernet = Get-Prop $obj 'EthernetSettings'
$serial = Get-Prop $obj 'SerialSettings'
[ordered]@{
source = 'json'
machinenumber = Get-NonEmpty (Get-Prop $general 'MachineNumber')
site = Get-NonEmpty (Get-Prop $general 'Site')
controltype = Get-NonEmpty (Get-Prop $general 'ControlType')
ncif = $null
dnctype = Get-NonEmpty (Get-Prop $general 'DncType')
tcpaddress = Get-NonEmpty (Get-Prop $ethernet 'TCPAddress')
tcpport = Get-NonEmpty (Get-Prop $ethernet 'TCPPort')
comport = Get-NonEmpty (Get-Prop $serial 'ComPort')
baudrate = Get-NonEmpty (Get-Prop $serial 'BaudRate')
}
}
function Find-FirstFile {
param([string[]]$Roots, [string]$Filter)
foreach ($root in $Roots) {
if (-not $root -or -not (Test-Path -LiteralPath $root)) { continue }
$hit = Get-ChildItem -LiteralPath $root -Filter $Filter -File -ErrorAction SilentlyContinue |
Sort-Object LastWriteTime -Descending | Select-Object -First 1
if ($hit) { return $hit.FullName }
}
return $null
}
function Resolve-DncSource {
param([string]$RegPath, [string]$JsonPath)
if ($JsonPath -and (Test-Path -LiteralPath $JsonPath)) {
Write-Host (' machine source: JSON (explicit) ' + $JsonPath) -ForegroundColor DarkGray
return ConvertFrom-UdcJson -Path $JsonPath
}
if ($RegPath -and (Test-Path -LiteralPath $RegPath)) {
Write-Host (' machine source: .reg (explicit) ' + $RegPath) -ForegroundColor DarkGray
return ConvertFrom-RegExport -Reg (Read-RegExport -Path $RegPath)
}
foreach ($base in @(
'HKLM:\SOFTWARE\GE Aircraft Engines\DNC',
'HKLM:\SOFTWARE\WOW6432Node\GE Aircraft Engines\DNC')) {
$fromReg = ConvertFrom-LiveRegistry -Base $base
if ($fromReg -and $fromReg.machinenumber) {
Write-Host (' machine source: live registry ' + $base) -ForegroundColor DarkGray
return $fromReg
}
}
$localReg = Find-FirstFile -Roots @('C:\Program Files (x86)\Dnc', (Join-Path $env:SystemDrive 'GE Aerospace')) -Filter '*.reg'
if ($localReg) {
Write-Host (' machine source: local .reg ' + $localReg) -ForegroundColor DarkGray
return ConvertFrom-RegExport -Reg (Read-RegExport -Path $localReg)
}
$localJson = Find-FirstFile -Roots @((Join-Path $env:SystemDrive 'GE Aerospace'), 'C:\Program Files (x86)\UDC', 'C:\Program Files (x86)\Dnc') -Filter 'udc_settings_*.json'
if ($localJson) {
Write-Host (' machine source: local UDC json ' + $localJson) -ForegroundColor DarkGray
return ConvertFrom-UdcJson -Path $localJson
}
return $null
}
# --- PC collection (read-only) ------------------------------------------------
function Get-PcFacts {
$computerSystem = Get-CimInstance Win32_ComputerSystem -ErrorAction SilentlyContinue
$operatingSys = Get-CimInstance Win32_OperatingSystem -ErrorAction SilentlyContinue
$bios = Get-CimInstance Win32_BIOS -ErrorAction SilentlyContinue
$interfaces = @()
try {
foreach ($cfg in @(Get-NetIPConfiguration -ErrorAction SilentlyContinue)) {
$adapter = $null
try { $adapter = Get-NetAdapter -InterfaceAlias $cfg.InterfaceAlias -ErrorAction SilentlyContinue } catch {}
if (-not $adapter -or (Get-Prop $adapter 'HardwareInterface') -ne $true) { continue }
$ipv4 = @(Get-Prop $cfg 'IPv4Address') | Select-Object -First 1
$interfaces += [ordered]@{
interfacealias = [string]$cfg.InterfaceAlias
ipaddress = if ($ipv4) { [string](Get-Prop $ipv4 'IPAddress') } else { $null }
macaddress = [string](Get-Prop $adapter 'MacAddress')
}
}
} catch {}
$installedApps = @()
if (-not $SkipApps) {
foreach ($keyPath in @(
'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*',
'HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*')) {
foreach ($entry in @(Get-ItemProperty -Path $keyPath -ErrorAction SilentlyContinue)) {
$displayName = Get-NonEmpty (Get-Prop $entry 'DisplayName')
if (-not $displayName) { continue }
if ((Get-Prop $entry 'SystemComponent') -eq 1) { continue }
$installedApps += [ordered]@{
appname = $displayName
installedversion = Get-NonEmpty (Get-Prop $entry 'DisplayVersion')
publisher = Get-NonEmpty (Get-Prop $entry 'Publisher')
}
}
}
$installedApps = @($installedApps | Sort-Object { $_.appname } -Unique)
}
[ordered]@{
hostname = Get-FirstNonEmpty @($env:COMPUTERNAME, (Get-Prop $computerSystem 'Name'))
vendor = Get-NonEmpty (Get-Prop $computerSystem 'Manufacturer')
model = Get-NonEmpty (Get-Prop $computerSystem 'Model')
osname = Get-NonEmpty (Get-Prop $operatingSys 'Caption')
serial = Get-FirstNonEmpty @((Get-Prop $bios 'SerialNumber'), (Get-Prop $computerSystem 'SerialNumber'))
lastboot = Format-ShopDbDate (Get-Prop $operatingSys 'LastBootUpTime')
loggeduser = Get-NonEmpty (Get-Prop $computerSystem 'UserName')
interfaces = $interfaces
installedapps = $installedApps
}
}
# --- Main ---------------------------------------------------------------------
Write-Host ''
Write-Host ' ShopDB unified collector' -ForegroundColor Cyan
Write-Host (' ' + ('-' * 56)) -ForegroundColor DarkGray
if (-not (Test-Path -LiteralPath $OutputRoot)) { New-Item -ItemType Directory -Path $OutputRoot -Force | Out-Null }
$sidecarDir = Join-Path $OutputRoot 'sidecars'
if (-not (Test-Path -LiteralPath $sidecarDir)) { New-Item -ItemType Directory -Path $sidecarDir -Force | Out-Null }
$collectedAt = (Get-Date).ToString('yyyy-MM-dd HH:mm:ss')
$pc = Get-PcFacts
if (-not $AssetNumber) { $AssetNumber = $pc.hostname }
# Row sets keyed to the shared stores.
$vendorRows = @()
$modelRows = @()
$osRows = @()
$assetRows = @()
$computerRows = @()
$machineRows = @()
if ($pc.vendor) { $vendorRows += [pscustomobject]@{ vendor = $pc.vendor } }
if ($pc.model) { $modelRows += [pscustomobject]@{ modelnumber = $pc.model; vendorid = $pc.vendor; modeltypeid = $null } }
if ($pc.osname) { $osRows += [pscustomobject]@{ osname = $pc.osname } }
$assetRows += [pscustomobject]@{
assetnumber = $AssetNumber; assettypeid = $AssetType; name = $pc.hostname; statusid = $null
serialnumber = $pc.serial; gaugelabreference = $null; maintenancereference = $null
locationid = $null; businessunitid = $null
notes = 'PC collected ' + $collectedAt
}
$computerRows += [pscustomobject]@{
assetid = $AssetNumber; hostname = $pc.hostname; computertypeid = $ComputerType; osid = $pc.osname
vendorid = $pc.vendor; modelnumberid = $pc.model; loggedinuser = $pc.loggeduser
lastreporteddate = $collectedAt; lastboottime = $pc.lastboot
}
# Machine (best-effort).
$machine = $null
if (-not $SkipMachine) {
try { $machine = Resolve-DncSource -RegPath $RegPath -JsonPath $JsonPath } catch { $machine = $null }
}
if ($machine -and $machine.machinenumber) {
$controllerVendor = Resolve-ControllerVendor -ControlType $machine.controltype -NcIF $machine.ncif
$transport = if ($machine.tcpaddress) { 'ethernet' } elseif ($machine.comport) { 'serial' } else { 'unknown' }
$noteBits = @("DNC=$($machine.dnctype)")
if ($machine.controltype) { $noteBits += "control=$($machine.controltype)" }
$noteBits += "transport=$transport"
$machineNote = 'eDNC collected ' + $collectedAt + '; ' + ($noteBits -join '; ')
if ($controllerVendor) { $vendorRows += [pscustomobject]@{ vendor = $controllerVendor } }
$assetRows += [pscustomobject]@{
assetnumber = $machine.machinenumber; assettypeid = 'Machine'; name = $machine.machinenumber; statusid = $null
serialnumber = $null; gaugelabreference = $null; maintenancereference = $null
locationid = $null; businessunitid = $null; notes = $machineNote
}
$machineRows += [pscustomobject]@{
assetid = $machine.machinenumber; machinetypeid = $MachineType; vendorid = $null; modelnumberid = $null
controllervendorid = $controllerVendor; controllermodelid = $null
requiresmanualconfig = $null; islocationonly = $null; notes = $machineNote
}
$link = [ordered]@{
pcassetnumber = $AssetNumber; machineassetnumber = $machine.machinenumber; relationship = 'controls'
dnctype = $machine.dnctype; controltype = $machine.controltype; controllervendor = $controllerVendor
site = $machine.site; source = $machine.source; collectedat = $collectedAt
connection = [ordered]@{
transport = $transport; tcpaddress = $machine.tcpaddress; tcpport = $machine.tcpport
comport = $machine.comport; baudrate = $machine.baudrate
}
}
Write-Utf8NoBom -Path (Join-Path $sidecarDir ('machine_' + $machine.machinenumber + '.json')) -Content ($link | ConvertTo-Json -Depth 6)
Write-Host (' linked machine ' + $machine.machinenumber + ' (' + $controllerVendor + ', ' + $transport + ')') -ForegroundColor DarkGray
} elseif (-not $SkipMachine) {
Write-Host ' no eDNC/DNC/UDC machine config found on this PC' -ForegroundColor DarkGray
}
# --- Upsert into the shared store --------------------------------------------
$rowsByStore = @{
'vendors.csv' = $vendorRows
'models.csv' = $modelRows
'operatingsystems.csv' = $osRows
'assets.csv' = $assetRows
'computers.csv' = $computerRows
'machines.csv' = $machineRows
}
Write-Host ''
Write-Host (' Store: ' + $OutputRoot) -ForegroundColor Green
foreach ($fileName in $script:Stores.Keys) {
$spec = $script:Stores[$fileName]
$rows = @($rowsByStore[$fileName])
if ($rows.Count -eq 0) { continue }
$path = Join-Path $OutputRoot $fileName
$n = Merge-CsvStore -Path $path -Header $spec.header -Key $spec.key -IncomingRows $rows
Write-Host (' merged ' + $n + ' row(s) into ' + $fileName) -ForegroundColor DarkGray
}
# PC phase-2 sidecar.
$pcSidecar = [ordered]@{
assetnumber = $AssetNumber; hostname = $pc.hostname; collectedat = $collectedAt
interfaces = $pc.interfaces; installedapps = $pc.installedapps
}
Write-Utf8NoBom -Path (Join-Path $sidecarDir ('computer_' + $AssetNumber + '.json')) -Content ($pcSidecar | ConvertTo-Json -Depth 6)
# --- Publish (standard-user network mirror, optional) -------------------------
if ($PublishRoot) {
if (([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(
[Security.Principal.WindowsBuiltInRole]::Administrator)) {
Write-Host ''
Write-Host ' WARNING: publishing while elevated. Network writes should run as the standard user.' -ForegroundColor Yellow
}
if (-not (Test-Path -LiteralPath $PublishRoot)) { New-Item -ItemType Directory -Path $PublishRoot -Force | Out-Null }
# Merge THIS PC's rows into the shared store rather than overwriting it. A
# blind copy would erase every other PC's rows (issue: the store is meant to
# accumulate site-wide). Merge-CsvStore upserts field-by-field, so a second
# PC's run adds its rows and refreshes shared ones without dropping the rest.
# A lock file serializes concurrent publishers so two PCs cannot read-modify-
# write the same store CSV at once and lose each other's rows.
$lockPath = Join-Path $PublishRoot '.publish.lock'
$lockStream = $null
for ($attempt = 0; $attempt -lt 60; $attempt++) {
try {
$lockStream = [System.IO.File]::Open(
$lockPath, [System.IO.FileMode]::OpenOrCreate,
[System.IO.FileAccess]::ReadWrite, [System.IO.FileShare]::None)
break
} catch {
Start-Sleep -Milliseconds 500
}
}
if ($null -eq $lockStream) {
Write-Host ''
Write-Host (' WARNING: could not acquire publish lock at ' + $lockPath + '; skipping network merge.') -ForegroundColor Yellow
} else {
try {
foreach ($fileName in $script:Stores.Keys) {
$spec = $script:Stores[$fileName]
$localPath = Join-Path $OutputRoot $fileName
if (-not (Test-Path -LiteralPath $localPath)) { continue }
$localRows = @(Import-Csv -LiteralPath $localPath)
if ($localRows.Count -eq 0) { continue }
$sharePath = Join-Path $PublishRoot $fileName
$n = Merge-CsvStore -Path $sharePath -Header $spec.header -Key $spec.key -IncomingRows $localRows
Write-Host (' merged ' + $n + ' row(s) into ' + $fileName + ' on share') -ForegroundColor DarkGray
}
# Sidecars are uniquely named per asset, so a straight copy is safe and
# nothing is overwritten across PCs.
$localSidecars = Join-Path $OutputRoot 'sidecars'
if (Test-Path -LiteralPath $localSidecars) {
$shareSidecars = Join-Path $PublishRoot 'sidecars'
if (-not (Test-Path -LiteralPath $shareSidecars)) { New-Item -ItemType Directory -Path $shareSidecars -Force | Out-Null }
Copy-Item -Path (Join-Path $localSidecars '*') -Destination $shareSidecars -Recurse -Force
}
} finally {
$lockStream.Close()
$lockStream.Dispose()
Remove-Item -LiteralPath $lockPath -Force -ErrorAction SilentlyContinue
}
Write-Host ''
Write-Host (' Published store to: ' + $PublishRoot) -ForegroundColor Green
}
}
Write-Host ''
Write-Host ' Done. Dry-run validate the accumulated store with:' -ForegroundColor Cyan
Write-Host (' flask csv import --dir "' + $OutputRoot + '"') -ForegroundColor White
Write-Host ''

View File

@@ -0,0 +1,91 @@
# Collect-ShopDB.ps1 Overview
## Overview
- **Purpose**: A unified, read-only "site adoption" collector that runs on any
Windows PC, gathers facts about that PC plus any attached eDNC/DNC/UDC machine,
and merges them into a shared set of importable CSVs so a site can build its
ShopDB inventory one PC at a time.
- **Compatibility**: PowerShell 5.1+; CSV headers are aligned to the ShopDB
importer (`flask csv import --dir`).
## Configuration and destinations
- **Two-stage flow**: a fast local stage folder (`%ProgramData%\ShopDB\intake`)
where CSVs are built/merged, and an optional network store (a file share) that
the whole site accumulates into.
- Command-line params `-OutputRoot` and `-PublishRoot` override the in-script
CONFIG block.
- Other params: `-AssetNumber` (defaults to hostname),
`-AssetType`/`-ComputerType`/`-MachineType` (resolved by name), explicit
`-RegPath`/`-JsonPath` machine source overrides, and `-SkipMachine`/`-SkipApps`
switches for partial runs.
## Data model (the stores)
- Six importable CSVs, each with a defined header order and natural key:
`vendors.csv`, `models.csv`, `operatingsystems.csv`, `assets.csv`,
`computers.csv`, `machines.csv`.
- Column names follow ShopDB conventions (e.g., `assetid`, `modelnumberid`,
`controllervendorid`, `lastreporteddate`).
## PC collection (`Get-PcFacts`, read-only)
- Uses CIM/WMI (`Win32_ComputerSystem`, `Win32_OperatingSystem`, `Win32_BIOS`)
to gather hostname, vendor/manufacturer, model, OS caption, serial number,
last boot time, and logged-in user.
- Enumerates **network interfaces** (hardware adapters only: alias, IPv4, MAC).
- Enumerates **installed applications** from the registry Uninstall keys (name,
version, publisher), skipping system components - unless `-SkipApps` is set.
## Machine (eDNC/DNC/UDC) collection
- `Resolve-DncSource` searches in priority order: explicit JSON path -> explicit
.reg path -> live registry (`HKLM\...\GE Aircraft Engines\DNC`) -> local `.reg`
export files -> local `udc_settings_*.json` files.
- Three parsers normalize the different source formats into one shape:
`ConvertFrom-RegExport` (parses exported `.reg` files including dword decoding),
`ConvertFrom-LiveRegistry`, and `ConvertFrom-UdcJson`.
- Extracts machine number, site, control type, DNC type, and connection detail
(TCP address/port for ethernet, COM port/baud for serial).
- `Resolve-ControllerVendor` maps control type / NcIF to a controller vendor
(e.g., eFOCAS -> FANUC, Siemens, Okuma).
## Merge / upsert logic (`Merge-CsvStore`)
- Field-level UPSERT keyed on each table's natural key: existing rows are matched
case-insensitively, and empty incoming cells never overwrite previously captured
values - so partial or repeated runs fill gaps without wiping earlier data.
- CSV writing is UTF-8 without BOM, with proper quoting/escaping.
## Sidecar files (phase-2 data CSV can't carry)
- Per-PC `sidecars\computer_<assetnumber>.json`: interfaces + installed apps.
- Per-machine `sidecars\machine_<machinenumber>.json`: PC-to-machine link
(relationship "controls"), DNC/control details, and connection transport info.
## Publishing and output
- Optionally mirrors the whole local store to the network share
(`Copy-Item -Recurse -Force`), with a warning if run elevated (network writes
should run as standard user).
- Prints a summary and the follow-up dry-run command:
`flask csv import --dir "<OutputRoot>"`.
---
## Relevance to an "Asset Collector" package
This script is a solid template for one collector in a broader package. Notable
patterns worth generalizing across other collectors (existing databases,
Intune/Azure exports, etc.):
- **Normalized intermediate schema** (the six CSV stores + JSON sidecars) that
all sources converge on, decoupling collection from import.
- **Idempotent field-level merge** so multiple heterogeneous sources can
contribute to the same asset without clobbering each other.
- **Source-priority resolution** and per-format parsers - the same shape could
wrap an Intune/Azure `Get-MgDevice` export or a SQL query.
- **Machines, PCs, Applications** are already modeled here; an Intune/Azure
source would map naturally onto `computers.csv` + the installed-apps sidecar,
while a MES/CNC database could feed `machines.csv`.

View File

@@ -0,0 +1,5 @@
assetnumber,name,serialnumber,assettypeid,statusid,locationid,businessunitid,notes
SAMPLE-PC-01,Shopfloor PC 01,DELLSN0001,computer,In Use,A1,Gage Lab,Sample collected device
SAMPLE-PC-02,CMM PC 02,DELLSN0002,computer,In Use,A1,Gage Lab,Sample collected device
SAMPLE-MC-01,Zeiss CMM 01,ZEISSSN0001,machine,In Use,A1,Gage Lab,Sample collected device attached to SAMPLE-PC-02
SAMPLE-MC-02,Haas VF-2SS 02,HAASSN0002,machine,In Use,A1,Gage Lab,Sample collected standalone CNC
1 assetnumber name serialnumber assettypeid statusid locationid businessunitid notes
2 SAMPLE-PC-01 Shopfloor PC 01 DELLSN0001 computer In Use A1 Gage Lab Sample collected device
3 SAMPLE-PC-02 CMM PC 02 DELLSN0002 computer In Use A1 Gage Lab Sample collected device
4 SAMPLE-MC-01 Zeiss CMM 01 ZEISSSN0001 machine In Use A1 Gage Lab Sample collected device attached to SAMPLE-PC-02
5 SAMPLE-MC-02 Haas VF-2SS 02 HAASSN0002 machine In Use A1 Gage Lab Sample collected standalone CNC

View File

@@ -0,0 +1,3 @@
assetid,computertypeid,hostname,osid,vendorid,modelnumberid,loggedinuser
SAMPLE-PC-01,Shopfloor PC,WKS-SAMPLE01,Windows 10 Enterprise,Dell,OptiPlex 7090,EXAMPLE\operator1
SAMPLE-PC-02,CMM PC,WKS-SAMPLE02,Windows 11 Enterprise,Dell,Precision 3660,EXAMPLE\operator2
1 assetid computertypeid hostname osid vendorid modelnumberid loggedinuser
2 SAMPLE-PC-01 Shopfloor PC WKS-SAMPLE01 Windows 10 Enterprise Dell OptiPlex 7090 EXAMPLE\operator1
3 SAMPLE-PC-02 CMM PC WKS-SAMPLE02 Windows 11 Enterprise Dell Precision 3660 EXAMPLE\operator2

View File

@@ -0,0 +1,3 @@
assetid,machinetypeid,vendorid,modelnumberid,requiresmanualconfig,islocationonly
SAMPLE-MC-01,CMM,Zeiss,CONTURA G2,0,0
SAMPLE-MC-02,CNC,Haas Automation,VF-2SS,0,0
1 assetid machinetypeid vendorid modelnumberid requiresmanualconfig islocationonly
2 SAMPLE-MC-01 CMM Zeiss CONTURA G2 0 0
3 SAMPLE-MC-02 CNC Haas Automation VF-2SS 0 0

View File

@@ -0,0 +1,5 @@
modelnumber,vendorid
OptiPlex 7090,Dell
Precision 3660,Dell
VF-2SS,Haas Automation
CONTURA G2,Zeiss
1 modelnumber vendorid
2 OptiPlex 7090 Dell
3 Precision 3660 Dell
4 VF-2SS Haas Automation
5 CONTURA G2 Zeiss

View File

@@ -0,0 +1,3 @@
osname
Windows 10 Enterprise
Windows 11 Enterprise
1 osname
2 Windows 10 Enterprise
3 Windows 11 Enterprise

View File

@@ -0,0 +1,4 @@
vendor
Dell
Haas Automation
Zeiss
1 vendor
2 Dell
3 Haas Automation
4 Zeiss