Files

benjamin.baker d68c6db9ab fix endpoints plugin CI failures

Migration tests knew no endpoints head revision, so the parametrized guards raised KeyError. Register it in EXPECTED_HEAD_REVISION.

EndpointState declared lastreportedat both index=True and an explicit db.Index over the same column, and the baseline migration named single-column indexes differently from what create_all builds. On a pre-existing table the name-keyed guard could not see the create_all index and added a duplicate, tripping the idempotency guard. Drop the redundant declaration and rename migration indexes to ix_<table>_<column> to match.

endpoints was absent from the universal site profile, so the installer would not offer it and prune-schema would drop its tables at provisioning. Add it.

The agent design record lived under docs/, which is stripped to the public wiki and normally absent from the tree; a lone file there un-skipped the wiki-publish guards. Move it into the plugin.

2026-08-27 07:10:43 -04:00

8.4 KiB

Raw Blame History

Agent Design Decisions (running preference list)

Server-side decisions made while building the endpoints plugin that the future C# agent (and the ServiceNow ticketing surface) must honor. This is a living document: add to it as the agent build raises questions. ASCII only, no em-dashes, no emojis, per repo style.

Scope of this document

The endpoints plugin (server) is the authoritative contract. The C# agent is a separate deliverable in its own solution/repo. Anything the agent sends must match what this plugin ingests; when the two disagree, this plugin wins and the agent changes.

1. Transport and identity

Agent posts one JSON object per host to POST /api/collector/endpoints (the generic ADR-006 collector route, auto-registered by the core loader).
Auth is a collector-scoped credential in the request HEADER, not a user JWT and not a querystring. Server reads COLLECTOR_API_KEY_ENDPOINTS first, then falls back to COLLECTOR_API_KEY. Provision the key the same way the computers reporter does: out of band, e.g. HKLM:\SOFTWARE\GE\ShopDB CollectorKey. The key is never baked into a distributed script/binary.
Identity field is hostname (case-insensitive). The server resolves it to an existing computer asset via the computers plugin. An unknown hostname is REJECTED with a validation error, never auto-created. Decision: the fleet is provisioned in ShopDB first (imports/Azure device sync); the agent does not get to invent assets. Revisit only if bulk self-registration becomes a need.

2. Report shape (current server contract)

Top-level payload fields the server currently reads (see plugins/endpoints/plugin.py get_collector_schema):

hostname (required), loggedinuser, agentversion
Metrics: lastboottime (ISO-8601), uptimeseconds, idleseconds, cpupercent, memorypercent
Inventory arrays (stored as latest-snapshot JSON, not history): services, ports, interfaces, scheduledtasks
events: array of curated Windows Event Log records (see section 4)

Decision: inventory and metrics are a SNAPSHOT overwrite (one EndpointState row per asset), not time-series. The agent sends the full current list each cycle; the server does not diff or version them. If per-sample history is later required, it goes in a new append-only table and the contract stays additive.

3. Reporting rhythm

Default report interval: 15 minutes (report_interval_minutes in the plugin manifest settings; a site setting can override). Agent design should make the interval configurable and jittered to avoid a fleet-wide thundering herd.
There is NO separate heartbeat. lastreportedat is stamped on every report; "quiet" endpoints are those past the quiet-report window (default 24h, site setting endpoints_quietreporthours). Agent does not need a distinct ping.

4. Curated Windows Event catalog (LOCKED to the server list)

The server stores ONLY these IDs; anything else in events is ignored (not an error). Source of truth: CURATED_EVENT_IDS in plugins/endpoints/plugin.py.

Event ID	Channel	Meaning
4624	Security	Successful logon
4625	Security	Failed logon
4634	Security	Logoff
4672	Security	Special privileges assigned to new logon
4697	Security	Service installed
4698	Security	Scheduled task created
41	System	Kernel-Power, unexpected shutdown
4618	Security	Monitored security event pattern
55	System	NTFS file system corruption
7011	System	Service Control Manager timeout
10016	System	DCOM permission error
1001	Application	Application error / bugcheck report

Per event, the agent sends: eventid (required), eventchannel, eventsource, eventtime (ISO-8601, event time not report time), eventmessage, and fields (an object of extracted structured values, e.g. target account on a 4625). fields is stored as payloadjson.
OPEN QUESTION for the agent build: which IDs need structured fields parsing vs. raw message pass-through. Start with pass-through message; add structured parsing per ID as alerting rules demand it. The teammate flagged event parsing as the hard part; keep the catalog fixed and small until baselined.
Scheduled-task history being disabled on some hosts means 4698 and task "last run" may be sparse; the agent should read current task state directly, not rely on event history alone.

5. Monitored services/processes

Known targets to watch (non-exhaustive, confirm during agent build): UDC, eDnc, Intune-related services. Server stores whatever list the agent sends in servicesjson without a fixed schema yet. Decision deferred: once the watched set is final, consider promoting it to a server-known enum for alerting.

6. ServiceNow ticketing surface (future, creds confirmed available)

Not built server-side yet. Recorded here so the agent and a future server endpoint agree:

Interactive C# piece (system-tray/desktop launcher) owns the 5-click-max flow: network precheck -> issue dropdown -> optional screenshot -> submit.
Fixed dropdown reasons (from the source conversation): Can't logon to CSF; Data collections down; Can't download program; Machine communication failing; Network drives down; Other (free text).
Screenshot saved to local Pictures\Screenshots before submission.
OPEN: whether ticket submission goes direct to the ServiceNow API from the agent, or is proxied through a ShopDB server endpoint (preferred: proxy, so ServiceNow creds live server-side and the agent carries only its collector key). Decide before wiring the integration.

7. Runtime split (from the planning conversation)

Interactive bits (tray launcher, ticketing, screenshot): C# / .NET, likely a Windows service + tray app.
Collections MAY be locally hosted PowerShell scripts the service invokes, where that is simpler than reimplementing WMI/CIM queries in C#.
Code signing and Intune deployment are agent-side concerns, out of scope for the server plugin but must be planned before fleet rollout.

8. Web UI display (server-side, built)

The collected data surfaces through ShopDB's existing declaration-driven UI contract (ADR-010). The plugin declares WHAT to show; generic core components render it. No plugin-specific Vue components were written. Audience is DT support / IT, NOT the shop-floor operator (the operator's surface is the agent-side ticket tool).

Surfaces implemented:

Per-PC asset panels (computer assets only), on the PC detail page:
- Endpoint status (keyvalue): logged-in user, uptime, idle, last boot, CPU/memory %, agent version, last reported. Feeds GET /api/endpoints/asset/{assetid}/status.
- Services and processes (list + badge): monitored services with a green/red run-state badge. Feeds .../services.
- Recent events (list + severity badge): last 10 curated events. Feeds .../recentevents.
Dashboard exception cards (hide when empty):
- Endpoints gone quiet (warning): .../dashboard/quiet.
- Endpoints with crashes or shutdowns (critical): .../dashboard/stability, bounded to a 48h window and the critical event catalog.

DECISION - freshness and cost. There is NO live streaming and NO polling. The freshness ceiling is the agent report cadence (~15 min), so a websocket/poll buys nothing but load. Data is fetched on page/dashboard load only, matching the existing computers-quiet card. Asset panels are single-asset indexed reads (negligible). Dashboard cards run once per dashboard load, cap at 50 rows, and the stability card is time-bounded so it does not grow with total history. If fleet-wide aggregation ever gets heavy, memoize dashboard aggregates with the core cache helper for ~60s - documented lever, not built yet.

Agent implication: the agent does not drive any UI refresh. Its only job is to report on its rhythm; the UI reflects whatever the last report wrote. The services payload should send a per-service run STATE (e.g. running/stopped) so the badge can color it; a missing/unknown state renders as a gray Unknown badge.

8.4 KiB Raw Blame History