862b290545
Curated events were re-read from the newest 25 entries per channel on every collect, and the server appended whatever arrived. One real event was therefore stored again on every cadence tick for as long as it stayed in that window, so event counts measured how long a box had been up rather than what had happened to it. The agent now keeps a per-channel watermark of the last Windows event record id it reported, persisted under ProgramData so a service restart does not replay history, and sends eventrecordid with each event. The server dedupes on (assetid, eventchannel, eventrecordid) behind a unique index, so a replay from an older agent cannot double-store either. Six inventory keys the agent had been sending all along - bios, disks, domain, memory, osinfo, processor - were parsed, logged as unpersisted, and dropped. They now have columns and are written on check-in, along with user context, pending reboot, and the cpupercent/memorypercent/ idleseconds fields that were declared in the contract but always null. Session-scoped data cannot be read from the agent's LocalSystem session 0 at all: mapped drives and user-owned listeners do not exist there. A probe runs in the interactive user's session and hands results back through a spool file. It is not yet producing output on a live endpoint; the column and panel are in place and read empty until that is fixed. Also bounds the user-context script wait. It was WaitForSingleObject with INFINITE, so a hung script pinned the command worker permanently.
352 lines
16 KiB
Markdown
352 lines
16 KiB
Markdown
# Endpoint agent contract (C# build reference)
|
|
|
|
Companion to DESIGN-DECISIONS-agent.md. This is the concrete wire contract the
|
|
future C# agent targets, derived from `EndpointsPlugin.get_collector_schema()`
|
|
and the generic collector dispatcher in `shopdb/core/api/collector.py`. When the
|
|
schema changes, update this file in the same change.
|
|
|
|
## 1. Transport
|
|
|
|
- Method/URL: `POST {baseurl}/api/collector/endpoints`
|
|
- Auth header: `X-API-Key: <collector key>` (Bearer token also accepted). The
|
|
key is a UI-managed, endpoints-only service token minted from Settings ->
|
|
Endpoint agent key (section 6). A bootstrap `.env` collector key still works
|
|
until the first UI key is generated, then it is switched off (section 6). No
|
|
querystring fallback.
|
|
- Body: `application/json`, a single flat object (NOT an array). One POST per
|
|
report per endpoint.
|
|
- Identity field: `hostname` (required, case-insensitive).
|
|
|
|
## 2. Request payload
|
|
|
|
Only `hostname` is required. Every other field is optional; the server upserts
|
|
only the keys that are present, so a lightweight report can omit inventory
|
|
arrays. `adopt` is first-contact only.
|
|
|
|
```jsonc
|
|
{
|
|
"hostname": "SHOP-PC-014", // required, resolved to computers.hostname
|
|
"adopt": true, // first run ONLY; delegates PC creation to computers plugin
|
|
"loggedinuser": "j.smith",
|
|
"serialnumber": "5CD1234ABC", // adoption subset (forwarded to computers)
|
|
"pctype": "CNC Controller", // adoption subset
|
|
"machinenumber": "12345", // adoption subset; "" or "9999" == none
|
|
"agentversion": "1.0.0",
|
|
"trayversion": "1.0.0", // file version of the interactive tray exe, omitted if no tray
|
|
"lastboottime": "2025-01-14T06:12:00Z", // ISO-8601 UTC
|
|
"uptimeseconds": 86400,
|
|
"idleseconds": 120,
|
|
"cpupercent": 12.5,
|
|
"memorypercent": 47.0,
|
|
"services": [ /* opaque array, stored as JSON */ ],
|
|
"ports": [ /* opaque array */ ],
|
|
"interfaces": [
|
|
{
|
|
"alias": "Intel(R) Ethernet Connection", // adapter description
|
|
"ipv4": "10.20.30.40", // "" if none
|
|
"mac": "AA:BB:CC:DD:EE:FF",
|
|
"gateway": "10.20.30.1", // "" for the static machine-IPS NIC
|
|
"subnet": "255.255.255.0",
|
|
"dhcp": false
|
|
}
|
|
],
|
|
"scheduledtasks": [ /* opaque array */ ],
|
|
"disks": [
|
|
{
|
|
"drive": "C:",
|
|
"label": "OS",
|
|
"filesystem": "NTFS",
|
|
"sizebytes": 511000000000,
|
|
"freebytes": 128000000000,
|
|
"usedbytes": 383000000000,
|
|
"freepercent": 25.0 // precomputed; a threshold is a direct compare
|
|
}
|
|
],
|
|
"bios": { /* opaque object: vendor, version, release date */ },
|
|
"memory": { /* opaque object: totalbytes, freebytes */ },
|
|
"processor": { /* opaque object: name, cores, logicalprocessors, maxclockmhz */ },
|
|
"osinfo": { /* opaque object: build, version, install date */ },
|
|
"domain": "shopfloor.example.com", // domain or workgroup membership
|
|
"pendingreboot": false, // omit when unknown; absent != false
|
|
"usercontext": { // omitted entirely when nobody is signed in
|
|
"netuse": [
|
|
{
|
|
"localname": "Z:",
|
|
"remotename": "\\\\fileserver\\programs",
|
|
"status": "OK",
|
|
"username": "SHOPFLOOR\\operator",
|
|
"persistent": true
|
|
}
|
|
],
|
|
"userlisteners": [
|
|
{ "localaddress": "0.0.0.0", "localport": 7878, "processid": 4512, "processname": "DncMain" }
|
|
],
|
|
"idleseconds": 45,
|
|
"username": "operator",
|
|
"collectedat": "2025-01-14T06:12:00Z"
|
|
},
|
|
"events": [
|
|
{
|
|
"eventid": 6008, // required per item; non-catalog IDs are dropped, not errored
|
|
"eventrecordid": 184412, // Windows record id; dedupe key, see notes
|
|
"eventchannel": "System",
|
|
"eventsource": "EventLog",
|
|
"eventtime": "2025-01-14T06:11:58Z",
|
|
"eventmessage": "The previous system shutdown was unexpected.",
|
|
"fields": { "any": "extra" } // opaque object, stored as JSON
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Notes:
|
|
- `adopt=true` is sent ONLY by the installer's first run. Steady-state reports
|
|
omit it (or send `false`). After adoption, an unknown hostname is a hard
|
|
validation error again (DESIGN-DECISIONS-agent.md 1a).
|
|
- The adoption subset (`serialnumber`, `pctype`, `machinenumber`, plus
|
|
`loggedinuser`) is forwarded to the `computers` collector so the newly created
|
|
PC is not a bare shell. Send them on first contact.
|
|
- `services`/`ports`/`interfaces`/`scheduledtasks` are stored verbatim as JSON;
|
|
the server does not validate their inner shape yet. Keep them stable so the UI
|
|
can render them later. `interfaces` reports EVERY hardware NIC (not just the
|
|
primary) so the gatewayless static NIC that links a PC to its machine's IPS is
|
|
preserved (DESIGN-DECISIONS-agent.md 9.3); each entry is
|
|
`{alias, ipv4, mac, gateway, subnet, dhcp}`.
|
|
- `machinenumber` is discovered on-box by the agent from the eDNC registry
|
|
(`HKLM\SOFTWARE\[WOW6432Node\]GE Aircraft Engines\DNC\General:MachineNo`) or
|
|
the UDC `udc_settings_*.json` (`GeneralSettings.MachineNumber`), in that order
|
|
(DESIGN-DECISIONS-agent.md 9.4/9.5). `""` or `"9999"` means none.
|
|
- `eventrecordid` is the Windows event record id: monotonic and unique within a
|
|
channel on a machine. The agent uses it as a per-channel harvest watermark and
|
|
the server enforces a unique index on
|
|
`(assetid, eventchannel, eventrecordid)`, so one real event is stored exactly
|
|
once. It is nullable: an older agent omits it, and MySQL permits repeated
|
|
NULLs in a unique index, so those reports keep the previous append behavior
|
|
rather than failing ingest. Send it whenever the source record exposes one.
|
|
- `disks`/`bios`/`memory`/`processor`/`osinfo`/`usercontext` are stored verbatim
|
|
as JSON snapshots on the endpoint state, newest-wins. `domain` and
|
|
`pendingreboot` are scalar columns because the fleet view filters and alerts
|
|
on them.
|
|
- OMITTING a key is not the same as sending an empty value. Every one of these
|
|
fields is applied only when the key is PRESENT, so a report that omits it
|
|
preserves the last-known value. Send `null`/`[]` only when you mean "this is
|
|
genuinely empty now"; omit when you simply did not collect it this cycle.
|
|
- `pendingreboot` follows that rule strictly: omit it when the check failed. A
|
|
literal `false` asserts the endpoint is confirmed NOT holding a reboot.
|
|
- `usercontext` is gathered by a probe the agent runs inside the signed-in
|
|
user's session, because mapped drives, user-owned listeners and idle time do
|
|
not exist in the agent's own LocalSystem session 0. It is omitted when no one
|
|
is signed in, which is a normal unattended state, not a fault.
|
|
- `ports` carries LISTENING sockets only. The full connection table is available
|
|
on demand via the `network_connections` command instead, because established
|
|
connections churn constantly and would change every report for no diagnostic
|
|
gain.
|
|
|
|
## 3. Response
|
|
|
|
Success (HTTP 200), ADR-006 envelope:
|
|
|
|
```jsonc
|
|
{
|
|
"status": "ok",
|
|
"action": "created", // "created" first time, "updated" thereafter
|
|
"assetid": 4821,
|
|
"identityvalue": "SHOP-PC-014",
|
|
"warnings": [
|
|
"adopted new computer for hostname: 'SHOP-PC-014'" // present only on adoption
|
|
]
|
|
}
|
|
```
|
|
|
|
Validation failure (HTTP 400) - e.g. unknown hostname with `adopt` absent:
|
|
|
|
```jsonc
|
|
{ "status": "error", "error": { "code": "VALIDATION_ERROR",
|
|
"message": "could not resolve hostname to a computer: 'SHOP-PC-014'" } }
|
|
```
|
|
|
|
Agent handling: treat non-200 as retryable except a `VALIDATION_ERROR` that
|
|
names an unresolved hostname on a NON-first-contact report - that means the PC
|
|
was removed server-side; surface it, do not spin.
|
|
|
|
## 4. C# DTOs
|
|
|
|
Match the JSON keys EXACTLY (lowercase, no underscores) to satisfy the naming
|
|
rule; use attributes so C# properties can stay PascalCase without a camelCase
|
|
serializer changing the wire keys.
|
|
|
|
```csharp
|
|
using System.Text.Json.Serialization;
|
|
|
|
public sealed class EndpointReport
|
|
{
|
|
[JsonPropertyName("hostname")] public string Hostname { get; set; } = "";
|
|
[JsonPropertyName("adopt")] public bool? Adopt { get; set; }
|
|
[JsonPropertyName("loggedinuser")] public string? LoggedInUser { get; set; }
|
|
[JsonPropertyName("serialnumber")] public string? SerialNumber { get; set; }
|
|
[JsonPropertyName("pctype")] public string? PcType { get; set; }
|
|
[JsonPropertyName("machinenumber")] public string? MachineNumber { get; set; }
|
|
[JsonPropertyName("agentversion")] public string? AgentVersion { get; set; }
|
|
[JsonPropertyName("trayversion")] public string? TrayVersion { get; set; }
|
|
[JsonPropertyName("lastboottime")] public DateTimeOffset? LastBootTime { get; set; }
|
|
[JsonPropertyName("uptimeseconds")] public long? UptimeSeconds { get; set; }
|
|
[JsonPropertyName("idleseconds")] public long? IdleSeconds { get; set; }
|
|
[JsonPropertyName("cpupercent")] public double? CpuPercent { get; set; }
|
|
[JsonPropertyName("memorypercent")] public double? MemoryPercent { get; set; }
|
|
[JsonPropertyName("services")] public object[]? Services { get; set; }
|
|
[JsonPropertyName("ports")] public object[]? Ports { get; set; }
|
|
[JsonPropertyName("interfaces")] public object[]? Interfaces { get; set; }
|
|
[JsonPropertyName("scheduledtasks")]public object[]? ScheduledTasks { get; set; }
|
|
[JsonPropertyName("disks")] public object[]? Disks { get; set; }
|
|
[JsonPropertyName("bios")] public object? Bios { get; set; }
|
|
[JsonPropertyName("memory")] public object? Memory { get; set; }
|
|
[JsonPropertyName("processor")] public object? Processor { get; set; }
|
|
[JsonPropertyName("osinfo")] public object? OsInfo { get; set; }
|
|
[JsonPropertyName("domain")] public string? Domain { get; set; }
|
|
[JsonPropertyName("pendingreboot")] public bool? PendingReboot { get; set; }
|
|
[JsonPropertyName("usercontext")] public object? UserContext { get; set; }
|
|
[JsonPropertyName("applications")] public object[]? Applications { get; set; }
|
|
[JsonPropertyName("events")] public EndpointEventRecord[]? Events { get; set; }
|
|
}
|
|
|
|
public sealed class EndpointEventRecord
|
|
{
|
|
[JsonPropertyName("eventid")] public int EventId { get; set; }
|
|
[JsonPropertyName("eventrecordid")] public long? EventRecordId { get; set; }
|
|
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
|
|
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
|
|
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
|
|
[JsonPropertyName("eventmessage")] public string? EventMessage { get; set; }
|
|
[JsonPropertyName("fields")] public Dictionary<string, object>? Fields { get; set; }
|
|
}
|
|
|
|
public sealed class CollectorResponse
|
|
{
|
|
[JsonPropertyName("status")] public string Status { get; set; } = "";
|
|
[JsonPropertyName("action")] public string? Action { get; set; }
|
|
[JsonPropertyName("assetid")] public int? AssetId { get; set; }
|
|
[JsonPropertyName("identityvalue")] public string? IdentityValue { get; set; }
|
|
[JsonPropertyName("warnings")] public string[]? Warnings { get; set; }
|
|
}
|
|
```
|
|
|
|
Serializer settings: `DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull`
|
|
so omitted fields are not sent (server upserts only present keys). Send UTC ISO-8601.
|
|
|
|
## 5. "Build site agent" settings flow (build cadence)
|
|
|
|
Goal: an operator in ShopDB Settings produces an instance-locked `.exe` that
|
|
knows its server URL and its collector key, needing no manual config on the
|
|
client.
|
|
|
|
1. Settings -> "Build site agent": operator picks/creates a collector-scoped
|
|
managed token (the same token accepted by `X-API-Key` on
|
|
`/api/collector/endpoints`). This binds the build to one ShopDB instance.
|
|
2. Server stamps a small config (base URL + token + reporting cadence) into the
|
|
agent build artifact. Instance-locked = the token only works against that
|
|
server, and the server can revoke it to kill a leaked build.
|
|
3. First run on a client:
|
|
- Collect the ShopDB PowerShell asset-collector data set (minus the CSV
|
|
emit): hostname, logged-in user, serial number, pc type, machine number,
|
|
boot time, plus the curated event tail.
|
|
- POST once with `adopt=true`. The endpoints plugin delegates PC creation to
|
|
the `computers` collector, then writes the first `EndpointState` in the
|
|
same commit. The endpoint appears in the Fleet UI immediately - no second
|
|
call, no separate enrollment endpoint.
|
|
4. Subsequent runs POST on the configured cadence with `adopt` omitted; strict
|
|
hostname resolution applies from then on.
|
|
|
|
Open item (from DESIGN-DECISIONS-agent.md 1a): whether `adopt` stays a
|
|
per-report flag or is promoted to a site setting (strict vs self-adopting
|
|
fleet). Default proposed: agent sends `adopt=true` on first run only.
|
|
|
|
## 6. UI-managed agent key (kill switch)
|
|
|
|
The endpoint agent authenticates with a single UI-managed key scoped to ONLY
|
|
`collector.ingest.endpoints`, so rotating or revoking it stops all
|
|
endpoint-agent traffic without affecting the computers or geenforce collectors.
|
|
|
|
- Settings -> Endpoint agent key (admin only) generates the key. The full secret
|
|
is shown ONCE at generation and stored only as a hash afterwards; configure it
|
|
as the agent `X-API-Key`.
|
|
- Generating a key is a hard cutover: any previous endpoint key is revoked in the
|
|
same request, and the bootstrap `.env` fallback for endpoints is switched off
|
|
(`plugin.endpoints.envkeyfallback` -> false), so the managed key becomes the
|
|
sole credential.
|
|
- Regenerate to rotate (old key dies immediately). Revoke to stop all
|
|
endpoint-agent reporting until a new key is generated - the kill switch for a
|
|
leaked agent.
|
|
- Backend: `GET/POST /api/endpoints/agentkey`, `POST
|
|
/api/endpoints/agentkey/regenerate`, `DELETE /api/endpoints/agentkey`. These
|
|
use the `mint_service_token`/`list_service_tokens`/`revoke_service_tokens`
|
|
contract helpers (shopdb.api, contract 0.22.0).
|
|
|
|
The bootstrap `.env` key (`COLLECTOR_API_KEY[_ENDPOINTS]`) remains only as a
|
|
first-run path before a UI key exists; it cannot be the runtime kill switch
|
|
because container env is injected at boot and cannot be rotated live.
|
|
|
|
## 7. Command channel (WebUI -> agent)
|
|
|
|
The WebUI can send an endpoint an ALLOWLISTED command (never an arbitrary
|
|
command string). Delivery is agent-initiated long-poll, so shop-floor PCs open
|
|
no inbound port: the agent makes an outbound request the server holds open until
|
|
a command is queued, runs it locally, and posts the result back. All three
|
|
routes reuse the same endpoints collector key (section 6) via `X-API-Key`, so
|
|
the command channel shares the kill switch and adds no second secret.
|
|
|
|
Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
|
|
|
|
- `mtconnect_start` - run the MTConnect UDC agent start script
|
|
(`C:\ProgramData\UDC\MTConnect_UDC\Agent\run.bat`).
|
|
- `mtconnect_stop` - stop the running MTConnect UDC agent process.
|
|
- `mtconnect_status` - report whether the MTConnect UDC agent is running.
|
|
- `network_connections` - return the full TCP connection table with owning
|
|
process ids and names. Read-only diagnostic. The result is capped at 500
|
|
connections with a `truncated` flag in `detail`, because an oversized payload
|
|
rejected by the server returns nothing at all and a partial answer is more
|
|
useful than a failed one. Standard reports carry listening ports only.
|
|
|
|
### 7.1 Claim the next command (long-poll)
|
|
|
|
`GET {baseurl}/api/endpoints/commands/next?hostname=<hostname>`
|
|
|
|
- Auth: `X-API-Key` (endpoints collector key).
|
|
- The server holds the request open up to ~25s. If a command is queued it is
|
|
atomically claimed (marked running) and returned; only one poll ever claims a
|
|
given command.
|
|
- Command queued (HTTP 200):
|
|
|
|
```jsonc
|
|
{ "status": "ok", "data": { "endpointcommandid": 91, "action": "mtconnect_start" } }
|
|
```
|
|
|
|
- Nothing queued within the hold window: HTTP 204 (no body). Re-poll.
|
|
|
|
### 7.2 Report a command result
|
|
|
|
`POST {baseurl}/api/endpoints/commands/<endpointcommandid>/result`
|
|
|
|
- Auth: `X-API-Key` (endpoints collector key).
|
|
- Body:
|
|
|
|
```jsonc
|
|
{
|
|
"succeeded": true, // required; false marks the command failed
|
|
"message": "MTConnect started", // short summary, optional (<= 2000 chars)
|
|
"mtconnectrunning": true, // optional; observed run state after the action
|
|
"detail": { "any": "extra" } // optional structured detail, stored as JSON
|
|
}
|
|
```
|
|
|
|
- When `mtconnectrunning` is present the server caches it on the endpoint's
|
|
`EndpointState` (`mtconnectrunning` + `mtconnectcheckedat`) so the WebUI toggle
|
|
shows a real state between polls. Every `mtconnect_*` action SHOULD probe and
|
|
report the resulting run state so the toggle never lies.
|
|
|
|
### 7.3 Enqueue (WebUI side, for reference)
|
|
|
|
`POST /api/endpoints/<assetid>/commands` with `{ "action": "mtconnect_start" }`,
|
|
gated by user JWT + `endpoints.command`. Only one command may be in flight per
|
|
endpoint (a second enqueue returns HTTP 409 until the first finalizes).
|
|
|