Add site asset CSV import with fuzzy reconciliation
Sites seed inventory from the Asset Collector script or spreadsheet exports, but lookup names drift between sources (Dell vs Dell Inc.), which would otherwise create duplicate vendors/models/types on every import. The importer now normalizes and fuzzy-matches near-miss references onto existing lookup rows within an operator-set similarity threshold, auto-applying the match while reporting it for review so nothing merges silently. Operating systems and asset rows are never reconciled. Adds the dedicated Import Site Assets page with script/template download and a multi-file dry-run/commit workflow.
This commit is contained in:
benjamin.baker
91
scripts/site_imports/assetcollector/overview.md
Normal file
91
scripts/site_imports/assetcollector/overview.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Collect-ShopDB.ps1 Overview
|
||||
|
||||
## Overview
|
||||
|
||||
- **Purpose**: A unified, read-only "site adoption" collector that runs on any
|
||||
Windows PC, gathers facts about that PC plus any attached eDNC/DNC/UDC machine,
|
||||
and merges them into a shared set of importable CSVs so a site can build its
|
||||
ShopDB inventory one PC at a time.
|
||||
- **Compatibility**: PowerShell 5.1+; CSV headers are aligned to the ShopDB
|
||||
importer (`flask csv import --dir`).
|
||||
|
||||
## Configuration and destinations
|
||||
|
||||
- **Two-stage flow**: a fast local stage folder (`%ProgramData%\ShopDB\intake`)
|
||||
where CSVs are built/merged, and an optional network store (a file share) that
|
||||
the whole site accumulates into.
|
||||
- Command-line params `-OutputRoot` and `-PublishRoot` override the in-script
|
||||
CONFIG block.
|
||||
- Other params: `-AssetNumber` (defaults to hostname),
|
||||
`-AssetType`/`-ComputerType`/`-MachineType` (resolved by name), explicit
|
||||
`-RegPath`/`-JsonPath` machine source overrides, and `-SkipMachine`/`-SkipApps`
|
||||
switches for partial runs.
|
||||
|
||||
## Data model (the stores)
|
||||
|
||||
- Six importable CSVs, each with a defined header order and natural key:
|
||||
`vendors.csv`, `models.csv`, `operatingsystems.csv`, `assets.csv`,
|
||||
`computers.csv`, `machines.csv`.
|
||||
- Column names follow ShopDB conventions (e.g., `assetid`, `modelnumberid`,
|
||||
`controllervendorid`, `lastreporteddate`).
|
||||
|
||||
## PC collection (`Get-PcFacts`, read-only)
|
||||
|
||||
- Uses CIM/WMI (`Win32_ComputerSystem`, `Win32_OperatingSystem`, `Win32_BIOS`)
|
||||
to gather hostname, vendor/manufacturer, model, OS caption, serial number,
|
||||
last boot time, and logged-in user.
|
||||
- Enumerates **network interfaces** (hardware adapters only: alias, IPv4, MAC).
|
||||
- Enumerates **installed applications** from the registry Uninstall keys (name,
|
||||
version, publisher), skipping system components - unless `-SkipApps` is set.
|
||||
|
||||
## Machine (eDNC/DNC/UDC) collection
|
||||
|
||||
- `Resolve-DncSource` searches in priority order: explicit JSON path -> explicit
|
||||
.reg path -> live registry (`HKLM\...\GE Aircraft Engines\DNC`) -> local `.reg`
|
||||
export files -> local `udc_settings_*.json` files.
|
||||
- Three parsers normalize the different source formats into one shape:
|
||||
`ConvertFrom-RegExport` (parses exported `.reg` files including dword decoding),
|
||||
`ConvertFrom-LiveRegistry`, and `ConvertFrom-UdcJson`.
|
||||
- Extracts machine number, site, control type, DNC type, and connection detail
|
||||
(TCP address/port for ethernet, COM port/baud for serial).
|
||||
- `Resolve-ControllerVendor` maps control type / NcIF to a controller vendor
|
||||
(e.g., eFOCAS -> FANUC, Siemens, Okuma).
|
||||
|
||||
## Merge / upsert logic (`Merge-CsvStore`)
|
||||
|
||||
- Field-level UPSERT keyed on each table's natural key: existing rows are matched
|
||||
case-insensitively, and empty incoming cells never overwrite previously captured
|
||||
values - so partial or repeated runs fill gaps without wiping earlier data.
|
||||
- CSV writing is UTF-8 without BOM, with proper quoting/escaping.
|
||||
|
||||
## Sidecar files (phase-2 data CSV can't carry)
|
||||
|
||||
- Per-PC `sidecars\computer_<assetnumber>.json`: interfaces + installed apps.
|
||||
- Per-machine `sidecars\machine_<machinenumber>.json`: PC-to-machine link
|
||||
(relationship "controls"), DNC/control details, and connection transport info.
|
||||
|
||||
## Publishing and output
|
||||
|
||||
- Optionally mirrors the whole local store to the network share
|
||||
(`Copy-Item -Recurse -Force`), with a warning if run elevated (network writes
|
||||
should run as standard user).
|
||||
- Prints a summary and the follow-up dry-run command:
|
||||
`flask csv import --dir "<OutputRoot>"`.
|
||||
|
||||
---
|
||||
|
||||
## Relevance to an "Asset Collector" package
|
||||
|
||||
This script is a solid template for one collector in a broader package. Notable
|
||||
patterns worth generalizing across other collectors (existing databases,
|
||||
Intune/Azure exports, etc.):
|
||||
|
||||
- **Normalized intermediate schema** (the six CSV stores + JSON sidecars) that
|
||||
all sources converge on, decoupling collection from import.
|
||||
- **Idempotent field-level merge** so multiple heterogeneous sources can
|
||||
contribute to the same asset without clobbering each other.
|
||||
- **Source-priority resolution** and per-format parsers - the same shape could
|
||||
wrap an Intune/Azure `Get-MgDevice` export or a SQL query.
|
||||
- **Machines, PCs, Applications** are already modeled here; an Intune/Azure
|
||||
source would map naturally onto `computers.csv` + the installed-apps sidecar,
|
||||
while a MES/CNC database could feed `machines.csv`.
|
||||
Reference in New Issue
Block a user