shopdb-flask/plugins/endpoints/DESIGN-DECISIONS-agent.md

# Agent Design Decisions (running preference list)

Server-side decisions made while building the `endpoints` plugin that the
future C# agent (and the ServiceNow ticketing surface) must honor. This is a
living document: add to it as the agent build raises questions. ASCII only, no
em-dashes, no emojis, per repo style.

## Scope of this document

The `endpoints` plugin (server) is the authoritative contract. The C# agent is
a separate deliverable in its own solution/repo. Anything the agent sends must
match what this plugin ingests; when the two disagree, this plugin wins and the
agent changes.

## 1. Transport and identity

- Agent posts one JSON object per host to `POST /api/collector/endpoints`
  (the generic ADR-006 collector route, auto-registered by the core loader).
- Auth is a collector-scoped credential in the request HEADER, not a user JWT
  and not a querystring. Server reads `COLLECTOR_API_KEY_ENDPOINTS` first, then
  falls back to `COLLECTOR_API_KEY`. Provision the key the same way the
  computers reporter does: out of band, e.g. `HKLM:\SOFTWARE\GE\ShopDB
  CollectorKey`. The key is never baked into a distributed script/binary.
- Identity field is `hostname` (case-insensitive). The server resolves it to an
  existing computer asset via the `computers` plugin. In STEADY STATE an unknown
  hostname is REJECTED with a validation error, never auto-created by this
  plugin. Decision: the fleet is provisioned in ShopDB first (imports/Azure
  device sync) OR adopted at first contact via the delegation shim in section
  1a; the `endpoints` plugin itself does not get to invent assets.

## 1a. First-contact adoption (delegation shim, not a design reversal)

The agent installer's first run reaches a PC that may not exist in ShopDB yet.
Rather than letting `endpoints` create assets (which would break ADR-001 and the
section 1 rule that this plugin never invents assets), first contact DELEGATES
adoption to the `computers` collector, which is already the sole authoritative
creator of computer Assets and PC rows.

Decision:

- The `endpoints` plugin NEVER constructs an `Asset` or a `Computer` row itself.
  Section 1 stands. This is a clarification, not a reversal.
- When `apply_collector_payload` cannot resolve the hostname AND the payload
  opts in to adoption (see the `adopt` flag below), the plugin calls the
  `computers` plugin's `apply_collector_payload` with the PC-identifying subset
  of the payload, flushes, then re-resolves the hostname and proceeds with
  telemetry. All of this runs inside the ONE transaction the generic collector
  dispatcher owns (`shopdb/core/api/collector.py`): a `ValueError` still rolls
  the whole thing back, so a half-adopted PC can never be committed.
- Steady-state reports send `adopt=false` (or omit it once a site chooses strict
  mode); only the installer's first run sends `adopt=true`. After adoption the
  strict "must already resolve" invariant holds for every subsequent report.

First-contact payload (the PC-identifying subset the shim forwards to the
`computers` collector - every field except `hostname` is optional but recommended
so the adopted PC is not a bare shell):

- `hostname` (required), `loggedinuser`, `serialnumber`, `pctype`,
  `machinenumber` (only when the PC is bound to a machine; `''`/`'9999'` are
  treated as none by the computers collector).

Net effect for the agent:

- ONE POST to `/api/collector/endpoints` with `adopt=true` on first run creates
  the Asset + PC (via the shim) and writes the first `EndpointState` in the same
  commit, so the endpoint appears in the UI immediately (no second call, no
  separate enrollment endpoint).
- OPEN: whether to keep `adopt` per-report or promote it to a site setting
  (strict vs. self-adopting fleet). Default proposed: adoption allowed, agent
  sends `adopt=true` only on first run.

## 2. Report shape (current server contract)

Top-level payload fields the server currently reads (see
`plugins/endpoints/plugin.py` `get_collector_schema`):

- `hostname` (required), `loggedinuser`, `agentversion`
- Metrics: `lastboottime` (ISO-8601), `uptimeseconds`, `idleseconds`,
  `cpupercent`, `memorypercent`
- Inventory arrays (stored as latest-snapshot JSON, not history):
  `services`, `ports`, `interfaces`, `scheduledtasks`
- `events`: array of curated Windows Event Log records (see section 4)

Decision: inventory and metrics are a SNAPSHOT overwrite (one `EndpointState`
row per asset), not time-series. The agent sends the full current list each
cycle; the server does not diff or version them. If per-sample history is later
required, it goes in a new append-only table and the contract stays additive.

## 3. Reporting rhythm

- Default report interval: 15 minutes (`report_interval_minutes` in the plugin
  manifest settings; a site setting can override). Agent design should make the
  interval configurable and jittered to avoid a fleet-wide thundering herd.
- There is NO separate heartbeat. `lastreportedat` is stamped on every report;
  "quiet" endpoints are those past the quiet-report window (default 24h, site
  setting `endpoints_quietreporthours`). Agent does not need a distinct ping.

## 4. Curated Windows Event catalog (LOCKED to the server list)

The server stores ONLY these IDs; anything else in `events` is ignored (not an
error). Source of truth: `CURATED_EVENT_IDS` in `plugins/endpoints/plugin.py`.

| Event ID | Channel      | Meaning                                    |
|----------|--------------|--------------------------------------------|
| 4624     | Security     | Successful logon                           |
| 4625     | Security     | Failed logon                               |
| 4634     | Security     | Logoff                                     |
| 4672     | Security     | Special privileges assigned to new logon   |
| 4697     | Security     | Service installed                          |
| 4698     | Security     | Scheduled task created                     |
| 41       | System       | Kernel-Power, unexpected shutdown          |
| 4618     | Security     | Monitored security event pattern           |
| 55       | System       | NTFS file system corruption                |
| 7011     | System       | Service Control Manager timeout            |
| 10016    | System       | DCOM permission error                      |
| 1001     | Application  | Application error / bugcheck report        |

- Per event, the agent sends: `eventid` (required), `eventchannel`,
  `eventsource`, `eventtime` (ISO-8601, event time not report time),
  `eventmessage`, and `fields` (an object of extracted structured values, e.g.
  target account on a 4625). `fields` is stored as `payloadjson`.
- OPEN QUESTION for the agent build: which IDs need structured `fields` parsing
  vs. raw message pass-through. Start with pass-through message; add structured
  parsing per ID as alerting rules demand it. The teammate flagged event parsing
  as the hard part; keep the catalog fixed and small until baselined.
- Scheduled-task history being disabled on some hosts means 4698 and task
  "last run" may be sparse; the agent should read current task state directly,
  not rely on event history alone.

## 5. Monitored services/processes

Known targets to watch (non-exhaustive, confirm during agent build): UDC, eDnc,
Intune-related services. Server stores whatever list the agent sends in
`servicesjson` without a fixed schema yet. Decision deferred: once the watched
set is final, consider promoting it to a server-known enum for alerting.

## 6. ServiceNow ticketing surface (future, creds confirmed available)

Not built server-side yet. Recorded here so the agent and a future server
endpoint agree:

- Interactive C# piece (system-tray/desktop launcher) owns the 5-click-max flow:
  network precheck -> issue dropdown -> optional screenshot -> submit.
- Fixed dropdown reasons (from the source conversation): Can't logon to CSF;
  Data collections down; Can't download program; Machine communication failing;
  Network drives down; Other (free text).
- Screenshot saved to local `Pictures\Screenshots` before submission.
- OPEN: whether ticket submission goes direct to the ServiceNow API from the
  agent, or is proxied through a ShopDB server endpoint (preferred: proxy, so
  ServiceNow creds live server-side and the agent carries only its collector
  key). Decide before wiring the integration.

## 7. Runtime split (from the planning conversation)

- Interactive bits (tray launcher, ticketing, screenshot): C# / .NET, likely a
  Windows service + tray app.
- Collections MAY be locally hosted PowerShell scripts the service invokes,
  where that is simpler than reimplementing WMI/CIM queries in C#.
- Code signing and Intune deployment are agent-side concerns, out of scope for
  the server plugin but must be planned before fleet rollout.

### 7a. Tray deployment and on-demand "Send Status" (built)

The tray icon is a PER-USER process, not a service. A Windows service runs in
session 0 and cannot draw a tray icon in the operator's session, so the tray
ships as a second executable (`shopdb-endpoint-tray.exe`) that the installer
copies next to `shopdbagent.exe`, registers for logon autostart via the HKCU
Run key (`ShopDbEndpointTray`), and launches for the current user. Because the
Run key is per-user, the installer must be run from the operator's own session;
`-NoTray` yields a headless, service-only install.

DECISION - "Send Status now" reuses `--once`, it does not reimplement collection.
The tray's Send Status action shells out to `shopdbagent.exe --once`, the same
production collect-and-report path the service uses on its cadence. Consequences:
the tray holds NO collector key (the agent exe reads it from registry/appsettings
itself), there is one collection code path to maintain, and an operator can force
the ShopDB monitor page to refresh immediately instead of waiting for the next
cadence tick. The tray surfaces the agent's exit code as a balloon tip (0 =
sent, non-zero = check the rolling agent log). Note: `--once` honors
`adoptonfirstrun`, so an on-demand send can carry `adopt=true`; the server dedups
on hostname, so this is safe.

The ServiceNow ticket action stays a scaffold (section 6). A future desktop
shortcut to the same tray exe is the intended explicitly-launchable entry point
for that workflow; nothing about the Send Status wiring blocks it.

## 8. Web UI display (server-side, built)

The collected data surfaces through ShopDB's existing declaration-driven UI
contract (ADR-010). The plugin declares WHAT to show; generic core components
render it. No plugin-specific Vue components were written. Audience is DT
support / IT, NOT the shop-floor operator (the operator's surface is the
agent-side ticket tool).

Surfaces implemented:

- Per-PC asset panels (computer assets only), on the PC detail page:
  - Endpoint status (keyvalue): logged-in user, uptime, idle, last boot,
    CPU/memory %, agent version, last reported. Feeds
    `GET /api/endpoints/asset/{assetid}/status`.
  - Services and processes (list + badge): monitored services with a green/red
    run-state badge. Feeds `.../services`.
  - Recent events (list + severity badge): last 10 curated events. Feeds
    `.../recentevents`.
- Dashboard exception cards (hide when empty):
  - Endpoints gone quiet (warning): `.../dashboard/quiet`.
  - Endpoints with crashes or shutdowns (critical): `.../dashboard/stability`,
    bounded to a 48h window and the critical event catalog.

DECISION - freshness and cost. There is NO live streaming and NO polling. The
freshness ceiling is the agent report cadence (~15 min), so a websocket/poll
buys nothing but load. Data is fetched on page/dashboard load only, matching the
existing computers-quiet card. Asset panels are single-asset indexed reads
(negligible). Dashboard cards run once per dashboard load, cap at 50 rows, and
the stability card is time-bounded so it does not grow with total history. If
fleet-wide aggregation ever gets heavy, memoize dashboard aggregates with the
core `cache` helper for ~60s - documented lever, not built yet.

Agent implication: the agent does not drive any UI refresh. Its only job is to
report on its rhythm; the UI reflects whatever the last report wrote. The
`services` payload should send a per-service run STATE (e.g. running/stopped) so
the badge can color it; a missing/unknown state renders as a gray Unknown badge.

## 9. Shop-floor collection knowledge base (reference, NOT a schema change)

This section is a body of reference, not a change request. Nothing here alters
the shape of ShopDB as it stands. It records WHAT a shop-floor endpoint really
is, WHICH signals identify it, and WHY each signal matters to the enterprise, so
that whoever grows the collector understands the stakes behind each field before
touching the wire contract. The source material is the site's own extant
tooling: `scripts/site_imports/assetcollector/Collect-ShopDB.ps1` (the mature
CSV collector), the migration/assessment flow under `scripts/.temp`
(`Assess-Orchestrator.ps1`, `Setup_ShopPC.ps1`, `Setup_InspectionPC.ps1`,
`eDncRegToUDC.ps1`, `machinetypeselector.bat`), and the eDNC/UDC/MTConnect config
formats those scripts read. Treat it as the "why" that the terse contract in
sections 1 through 8 cannot carry.

### 9.0 Why this matters beyond one plugin

This collector is a wedge into an enterprise-wide body of knowledge. A shop-floor
PC is never just a PC: it is the human-facing half of a machine tool, the thing
that feeds NC programs to a controller and streams data back off it. Getting its
identity, hardware, network, and application picture right lets ShopDB answer
questions that today require a person to walk the floor: which controller does
this box talk to, is its data path healthy, is the hardware under warranty, what
is it actually running. Every signal below earns its place by answering one of
those questions.

### 9.1 The endpoint taxonomy (how the floor classifies its own PCs)

`machinetypeselector.bat` plus the two `Setup_*` scripts show the site already
thinks in three roles. The collector should be able to DERIVE the role from
on-box evidence rather than ask an operator:

- Shop PC: runs UDC (Universal Data Collector) and/or eDNC, wired to one CNC
  machine (FANUC / Siemens / Okuma controller). This is the ~2/3 majority.
- Inspection PC: metrology stack. Marked by `C:\GE Aerospace\Genspect\genspect.true`
  (a flag file `Setup_InspectionPC.ps1` writes) and/or `C:\GE Aerospace\Gagecal`
  (GageCal 6.4).
- Non-shop-floor PC: office box, no machine link, minimal shop apps.

Role is derivable from application/marker presence (section 9.4), which is a more
honest signal than the raw WMI `Model` string the current `pctype` probe sends.

### 9.2 Hardware facts, and WHY

`Collect-ShopDB.ps1 :: Get-PcFacts` reads `Win32_ComputerSystem`,
`Win32_OperatingSystem`, and `Win32_BIOS` for: vendor/manufacturer, model, OS
caption, serial (BIOS serial first, ComputerSystem serial as fallback), last
boot, logged-in user.

Why hardware is not cosmetic:

- Dell Warranty API linkage. Vendor + model family + service tag (BIOS serial)
  is exactly the tuple the Dell Warranty API keys on. Capturing it lets ShopDB
  resolve warranty status and hardware line remotely, so a support tech can
  diagnose a hardware-level fault (failing disk, EOL platform, out-of-warranty
  chassis) without physically finding the machine. The serial is the join key;
  the model family is the diagnostic context.
- Remote hardware diagnosis generally. Knowing the exact hardware line turns
  "some PC is flaky" into "this is a known-bad model/BIOS combination," which is
  an enterprise-scale pattern once the fleet reports uniformly.

Collection note: prefer BIOS `SerialNumber` (the Dell service tag) and capture
manufacturer + model as distinct fields, since the warranty join needs the raw
service tag, not a normalized `pctype`.

### 9.3 Network interfaces, and WHY (the second-NIC reality)

`Get-PcFacts` and `Assess-Orchestrator.ps1 :: Get-EthernetNetworkSnapshot`
enumerate hardware adapters only and capture alias, IPv4, MAC, gateway, and
subnet (the orchestrator even reconstructs the mask from prefix length).

Why the interface list is first-class, not trivia:

- The second interface IS the machine link. Upwards of two-thirds of shop-floor
  PCs have a SECOND network interface with a STATIC, manually mapped IP address
  whose sole purpose is to talk to the related machine's onboard IPS
  (industrial PC / controller network). That private NIC is physical evidence of
  the PC-to-machine relationship and often the only place the machine-side
  address is recorded. Losing it loses the wiring diagram of the floor.
- MAC as stable identity. MAC is the most stable hardware identifier across
  re-image and rename, which is why every extant script grabs it. It is the
  durable anchor when hostname/asset number churn during migration.

Collection note: send ALL hardware interfaces (alias, IPv4, MAC, and where
available gateway/subnet), not just the primary. The primary/gateway NIC is the
corporate LAN; the gatewayless static NIC is very likely the machine IPS link and
is the interesting one.

### 9.4 Installed applications, and WHY (the extrapolation engine)

`Get-PcFacts` reads the registry Uninstall keys (name, version, publisher,
skipping `SystemComponent`). On these boxes the installed-app set is small and
highly diagnostic: they primarily run a handful of known apps.

Why the app inventory is the richest signal we have:

- Apps imply role (section 9.1): UDC/eDNC => Shop PC; Genspect/GageCal =>
  Inspection PC; neither => office.
- Apps imply the MACHINE binding. This is the crux. If UDC or eDNC (or MTConnect)
  is installed, its config file carries the four-digit number ShopDB calls the
  Machine Number, and from it we can extrapolate the ShopDB `machines` entry the
  PC should be related to. The same value appears under different names across
  the app generations:
  - eDNC `.reg` / live registry: `MachineNo` (under
    `HKLM\SOFTWARE\[WOW6432Node\]GE Aircraft Engines\DNC\General`).
  - UDC `udc_settings_*.json`: `GeneralSettings.MachineNumber`.
  - MTConnect: workstation / workstation ID / MachineNo, carried in the
    adapter/agent config under the fixed directory tree documented in section
    9.9.
  Treat "MachineNo", "workstation", "workstation ID", and "MachineNumber" as the
  same concept: the ShopDB Machine Number. Version and publisher are worth
  keeping too (they date the app generation, which tells you which config format
  to parse).

Collection note: the app inventory is not just an asset list; it is the pointer
that tells the collector WHICH config file to open next to find the machine
binding.

### 9.5 The machine binding (DNC/UDC/MTConnect config), and WHY

`Collect-ShopDB.ps1 :: Resolve-DncSource` resolves the machine config with a
SOURCE-PRIORITY chain, then three parsers normalize the formats into one shape:

- Priority: explicit JSON path -> explicit `.reg` path -> live registry
  (`HKLM\...\GE Aircraft Engines\DNC`) -> local `.reg` export -> local
  `udc_settings_*.json`. This ordering makes one probe work across both the
  eDNC-era and UDC-era boxes.
- Parsers: `ConvertFrom-RegExport` (parses exported `.reg`, including
  `dword:` decoding), `ConvertFrom-LiveRegistry`, `ConvertFrom-UdcJson`.
- Extracted: machine number, site, control type, DNC type, and connection detail
  (TCP address/port for ethernet; COM port/baud for serial).
- Controller vendor is DERIVED, not stored raw: `Resolve-ControllerVendor` maps
  `NcIF = EFOCAS -> FANUC`, `Cnc = SIEMENS -> Siemens`, `Cnc = OKUMA -> Okuma`.
  `eDncRegToUDC.ps1` confirms the same mapping and the eFocas/PPDCS key layout.

Why it matters: this binding is what upgrades a bare endpoint into "the PC that
controls machine 1234 over eFOCAS at 10.x.x.x." That relationship is the point of
the whole exercise; the hardware and app facts are supporting evidence for it.

### 9.6 Printers, and WHY

`Assess-Orchestrator.ps1 :: Get-PrinterSnapshot` / `Get-PrinterPortSnapshot`
capture name, driver, port, host address, shared/published, and the default
flag, filtering out virtual outputs (Print to PDF, XPS, Fax, OneNote). Shop and
inspection PCs are tied to specific physical/label printers, so the printer set
is part of a workstation's operational identity and a common migration break
point (a replaced PC that lost its label printer is a floor stoppage).

### 9.7 Patterns from the extant tooling worth carrying forward

Two engineering habits in these scripts are worth honoring in the collector,
because they encode hard-won migration lessons:

- Field-level idempotent merge (`Merge-CsvStore`): an empty incoming cell NEVER
  overwrites a previously captured value; later partial runs fill gaps without
  wiping earlier data. The server's endpoints ingest already mirrors this instinct
  (`if 'field' in payload`), and it should stay that way: a partial report means
  "leave the rest alone," not "clear it."
- Normalized intermediate shape + sidecars: heterogeneous sources (registry,
  `.reg`, JSON, WMI) all converge on one normalized record before import, with
  phase-2 data (interfaces, installed apps, PC-to-machine link) written as JSON
  sidecars. The same decoupling lets a future non-agent source (Intune/Azure
  `Get-MgDevice`, a MES/CNC database) feed the same model without a new pipeline.

### 9.9 MTConnect footprint, and WHY (three parallel stacks per box)

Source: `scripts/.temp/Install_MTConnect_ExisDir.ps1`,
`Install_MTConnect_ExisDir_BatConvert.ps1`, its usage guide
`Guide-Install_MTConnect_ExisDir_BatConvert.md`, and
`Firewall_Rules_MTConnect_All.bat`. MTConnect is the standardized data-off-the-
machine layer, and the install scripts reveal its exact on-disk shape, which is
what a collector needs to detect it reliably.

The critical structural fact: an endpoint can host up to THREE parallel MTConnect
stacks, one per data path, each an Adapter (talks to the machine) plus an Agent
(serves the MTConnect HTTP feed):

- Machine-direct: `C:\MTConnect\Adapter` + `C:\MTConnect\Agent` (Agent exe
  `C:\MTConnect\Agent\Agent.exe`).
- eDNC-fed: `C:\MTConnect_eDNC\Adapter` + `C:\MTConnect_eDNC\Agent`
  (`...\Agent\agent.exe`). The eDNC adapter is `DNCAdapter.exe` driven by
  `DNCAdapter.ini`, tying this path back to the eDNC config world in section 9.5.
- UDC-fed: `C:\ProgramData\udc\mtconnect_udc\agent` (`...\agent\agent.exe`,
  launched via `run.bat`). This binds MTConnect to the UDC install and therefore
  to the same `udc_settings_*.json` that carries `GeneralSettings.MachineNumber`.

Why the collector should care about these paths:

- Presence is role/path evidence. Which of the three trees exist tells you HOW the
  machine's data is being lifted (direct vs eDNC vs UDC), which is a real
  operational distinction, not a cosmetic one.
- The MTConnect Agent is a network service. The firewall bat and the
  `Ensure-FirewallProgramRule` calls open inbound rules named
  `Allow_MTConnect_Machine`, `Allow_MTConnect_UDC`, `Allow_MTConnect_eDNC` for
  each `Agent.exe`. That means each agent listens on an HTTP port, so its live
  MTConnect feed (which publishes a device UUID / station identity) is itself a
  future data source the collector could probe, not just a process to count.
- Machine Number lives here too. Because the eDNC path is `DNCAdapter.ini` and the
  UDC path is the UDC settings JSON, the MTConnect footprint is a second route to
  the same four-digit Machine Number described in 9.4/9.5, cross-checkable against
  the eDNC/UDC values.

Collection note: detect MTConnect by the three directory trees, capture which
Agent exes exist and whether their service/process is running (section 5 already
covers MTConnect as a monitored service), and record the data path
(direct/eDNC/UDC).
(`Install_MTConnect_ExisDir_BatConvert.ps1`) is a deployment concern, not a
collection concern, but it explains why these `install.bat`/`start.bat` files
were rewritten to self-elevate without a `pause`: so they can run unattended
under DSC on locked-down machines. Do not treat a self-elevating batch as
tampering.

### 9.8 Boundary (explicit)

None of section 9 is a ShopDB schema proposal. Where a signal above has no home
in the current model, that is fine: it stays reference until a separate, deliberate
decision (its own ADR / contract bump) decides to store it. The value here is
comprehension: anyone extending the collector should know that hardware feeds
warranty, the second NIC is the machine link, and the installed apps are the
thread that leads to the Machine Number. Build the agent with that understanding;
do not let this document, by itself, move the schema.