Files
shopdb-flask/plugins/endpoints/AGENT-CONTRACT.md
cproudlock 784964c907 Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
2026-08-28 16:17:17 -04:00

275 lines
12 KiB
Markdown

# Endpoint agent contract (C# build reference)
Companion to DESIGN-DECISIONS-agent.md. This is the concrete wire contract the
future C# agent targets, derived from `EndpointsPlugin.get_collector_schema()`
and the generic collector dispatcher in `shopdb/core/api/collector.py`. When the
schema changes, update this file in the same change.
## 1. Transport
- Method/URL: `POST {baseurl}/api/collector/endpoints`
- Auth header: `X-API-Key: <collector key>` (Bearer token also accepted). The
key is a UI-managed, endpoints-only service token minted from Settings ->
Endpoint agent key (section 6). A bootstrap `.env` collector key still works
until the first UI key is generated, then it is switched off (section 6). No
querystring fallback.
- Body: `application/json`, a single flat object (NOT an array). One POST per
report per endpoint.
- Identity field: `hostname` (required, case-insensitive).
## 2. Request payload
Only `hostname` is required. Every other field is optional; the server upserts
only the keys that are present, so a lightweight report can omit inventory
arrays. `adopt` is first-contact only.
```jsonc
{
"hostname": "SHOP-PC-014", // required, resolved to computers.hostname
"adopt": true, // first run ONLY; delegates PC creation to computers plugin
"loggedinuser": "j.smith",
"serialnumber": "5CD1234ABC", // adoption subset (forwarded to computers)
"pctype": "CNC Controller", // adoption subset
"machinenumber": "12345", // adoption subset; "" or "9999" == none
"agentversion": "1.0.0",
"lastboottime": "2025-01-14T06:12:00Z", // ISO-8601 UTC
"uptimeseconds": 86400,
"idleseconds": 120,
"cpupercent": 12.5,
"memorypercent": 47.0,
"services": [ /* opaque array, stored as JSON */ ],
"ports": [ /* opaque array */ ],
"interfaces": [
{
"alias": "Intel(R) Ethernet Connection", // adapter description
"ipv4": "10.20.30.40", // "" if none
"mac": "AA:BB:CC:DD:EE:FF",
"gateway": "10.20.30.1", // "" for the static machine-IPS NIC
"subnet": "255.255.255.0",
"dhcp": false
}
],
"scheduledtasks": [ /* opaque array */ ],
"events": [
{
"eventid": 6008, // required per item; non-catalog IDs are dropped, not errored
"eventchannel": "System",
"eventsource": "EventLog",
"eventtime": "2025-01-14T06:11:58Z",
"eventmessage": "The previous system shutdown was unexpected.",
"fields": { "any": "extra" } // opaque object, stored as JSON
}
]
}
```
Notes:
- `adopt=true` is sent ONLY by the installer's first run. Steady-state reports
omit it (or send `false`). After adoption, an unknown hostname is a hard
validation error again (DESIGN-DECISIONS-agent.md 1a).
- The adoption subset (`serialnumber`, `pctype`, `machinenumber`, plus
`loggedinuser`) is forwarded to the `computers` collector so the newly created
PC is not a bare shell. Send them on first contact.
- `services`/`ports`/`interfaces`/`scheduledtasks` are stored verbatim as JSON;
the server does not validate their inner shape yet. Keep them stable so the UI
can render them later. `interfaces` reports EVERY hardware NIC (not just the
primary) so the gatewayless static NIC that links a PC to its machine's IPS is
preserved (DESIGN-DECISIONS-agent.md 9.3); each entry is
`{alias, ipv4, mac, gateway, subnet, dhcp}`.
- `machinenumber` is discovered on-box by the agent from the eDNC registry
(`HKLM\SOFTWARE\[WOW6432Node\]GE Aircraft Engines\DNC\General:MachineNo`) or
the UDC `udc_settings_*.json` (`GeneralSettings.MachineNumber`), in that order
(DESIGN-DECISIONS-agent.md 9.4/9.5). `""` or `"9999"` means none.
## 3. Response
Success (HTTP 200), ADR-006 envelope:
```jsonc
{
"status": "ok",
"action": "created", // "created" first time, "updated" thereafter
"assetid": 4821,
"identityvalue": "SHOP-PC-014",
"warnings": [
"adopted new computer for hostname: 'SHOP-PC-014'" // present only on adoption
]
}
```
Validation failure (HTTP 400) - e.g. unknown hostname with `adopt` absent:
```jsonc
{ "status": "error", "error": { "code": "VALIDATION_ERROR",
"message": "could not resolve hostname to a computer: 'SHOP-PC-014'" } }
```
Agent handling: treat non-200 as retryable except a `VALIDATION_ERROR` that
names an unresolved hostname on a NON-first-contact report - that means the PC
was removed server-side; surface it, do not spin.
## 4. C# DTOs
Match the JSON keys EXACTLY (lowercase, no underscores) to satisfy the naming
rule; use attributes so C# properties can stay PascalCase without a camelCase
serializer changing the wire keys.
```csharp
using System.Text.Json.Serialization;
public sealed class EndpointReport
{
[JsonPropertyName("hostname")] public string Hostname { get; set; } = "";
[JsonPropertyName("adopt")] public bool? Adopt { get; set; }
[JsonPropertyName("loggedinuser")] public string? LoggedInUser { get; set; }
[JsonPropertyName("serialnumber")] public string? SerialNumber { get; set; }
[JsonPropertyName("pctype")] public string? PcType { get; set; }
[JsonPropertyName("machinenumber")] public string? MachineNumber { get; set; }
[JsonPropertyName("agentversion")] public string? AgentVersion { get; set; }
[JsonPropertyName("lastboottime")] public DateTimeOffset? LastBootTime { get; set; }
[JsonPropertyName("uptimeseconds")] public long? UptimeSeconds { get; set; }
[JsonPropertyName("idleseconds")] public long? IdleSeconds { get; set; }
[JsonPropertyName("cpupercent")] public double? CpuPercent { get; set; }
[JsonPropertyName("memorypercent")] public double? MemoryPercent { get; set; }
[JsonPropertyName("services")] public object[]? Services { get; set; }
[JsonPropertyName("ports")] public object[]? Ports { get; set; }
[JsonPropertyName("interfaces")] public object[]? Interfaces { get; set; }
[JsonPropertyName("scheduledtasks")]public object[]? ScheduledTasks { get; set; }
[JsonPropertyName("events")] public EndpointEventRecord[]? Events { get; set; }
}
public sealed class EndpointEventRecord
{
[JsonPropertyName("eventid")] public int EventId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
[JsonPropertyName("eventmessage")] public string? EventMessage { get; set; }
[JsonPropertyName("fields")] public Dictionary<string, object>? Fields { get; set; }
}
public sealed class CollectorResponse
{
[JsonPropertyName("status")] public string Status { get; set; } = "";
[JsonPropertyName("action")] public string? Action { get; set; }
[JsonPropertyName("assetid")] public int? AssetId { get; set; }
[JsonPropertyName("identityvalue")] public string? IdentityValue { get; set; }
[JsonPropertyName("warnings")] public string[]? Warnings { get; set; }
}
```
Serializer settings: `DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull`
so omitted fields are not sent (server upserts only present keys). Send UTC ISO-8601.
## 5. "Build site agent" settings flow (build cadence)
Goal: an operator in ShopDB Settings produces an instance-locked `.exe` that
knows its server URL and its collector key, needing no manual config on the
client.
1. Settings -> "Build site agent": operator picks/creates a collector-scoped
managed token (the same token accepted by `X-API-Key` on
`/api/collector/endpoints`). This binds the build to one ShopDB instance.
2. Server stamps a small config (base URL + token + reporting cadence) into the
agent build artifact. Instance-locked = the token only works against that
server, and the server can revoke it to kill a leaked build.
3. First run on a client:
- Collect the ShopDB PowerShell asset-collector data set (minus the CSV
emit): hostname, logged-in user, serial number, pc type, machine number,
boot time, plus the curated event tail.
- POST once with `adopt=true`. The endpoints plugin delegates PC creation to
the `computers` collector, then writes the first `EndpointState` in the
same commit. The endpoint appears in the Fleet UI immediately - no second
call, no separate enrollment endpoint.
4. Subsequent runs POST on the configured cadence with `adopt` omitted; strict
hostname resolution applies from then on.
Open item (from DESIGN-DECISIONS-agent.md 1a): whether `adopt` stays a
per-report flag or is promoted to a site setting (strict vs self-adopting
fleet). Default proposed: agent sends `adopt=true` on first run only.
## 6. UI-managed agent key (kill switch)
The endpoint agent authenticates with a single UI-managed key scoped to ONLY
`collector.ingest.endpoints`, so rotating or revoking it stops all
endpoint-agent traffic without affecting the computers or geenforce collectors.
- Settings -> Endpoint agent key (admin only) generates the key. The full secret
is shown ONCE at generation and stored only as a hash afterwards; configure it
as the agent `X-API-Key`.
- Generating a key is a hard cutover: any previous endpoint key is revoked in the
same request, and the bootstrap `.env` fallback for endpoints is switched off
(`plugin.endpoints.envkeyfallback` -> false), so the managed key becomes the
sole credential.
- Regenerate to rotate (old key dies immediately). Revoke to stop all
endpoint-agent reporting until a new key is generated - the kill switch for a
leaked agent.
- Backend: `GET/POST /api/endpoints/agentkey`, `POST
/api/endpoints/agentkey/regenerate`, `DELETE /api/endpoints/agentkey`. These
use the `mint_service_token`/`list_service_tokens`/`revoke_service_tokens`
contract helpers (shopdb.api, contract 0.22.0).
The bootstrap `.env` key (`COLLECTOR_API_KEY[_ENDPOINTS]`) remains only as a
first-run path before a UI key exists; it cannot be the runtime kill switch
because container env is injected at boot and cannot be rotated live.
## 7. Command channel (WebUI -> agent)
The WebUI can send an endpoint an ALLOWLISTED command (never an arbitrary
command string). Delivery is agent-initiated long-poll, so shop-floor PCs open
no inbound port: the agent makes an outbound request the server holds open until
a command is queued, runs it locally, and posts the result back. All three
routes reuse the same endpoints collector key (section 6) via `X-API-Key`, so
the command channel shares the kill switch and adds no second secret.
Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
- `mtconnect_start` - run the MTConnect UDC agent start script
(`C:\ProgramData\UDC\MTConnect_UDC\Agent\run.bat`).
- `mtconnect_stop` - stop the running MTConnect UDC agent process.
- `mtconnect_status` - report whether the MTConnect UDC agent is running.
### 7.1 Claim the next command (long-poll)
`GET {baseurl}/api/endpoints/commands/next?hostname=<hostname>`
- Auth: `X-API-Key` (endpoints collector key).
- The server holds the request open up to ~25s. If a command is queued it is
atomically claimed (marked running) and returned; only one poll ever claims a
given command.
- Command queued (HTTP 200):
```jsonc
{ "status": "ok", "data": { "endpointcommandid": 91, "action": "mtconnect_start" } }
```
- Nothing queued within the hold window: HTTP 204 (no body). Re-poll.
### 7.2 Report a command result
`POST {baseurl}/api/endpoints/commands/<endpointcommandid>/result`
- Auth: `X-API-Key` (endpoints collector key).
- Body:
```jsonc
{
"succeeded": true, // required; false marks the command failed
"message": "MTConnect started", // short summary, optional (<= 2000 chars)
"mtconnectrunning": true, // optional; observed run state after the action
"detail": { "any": "extra" } // optional structured detail, stored as JSON
}
```
- When `mtconnectrunning` is present the server caches it on the endpoint's
`EndpointState` (`mtconnectrunning` + `mtconnectcheckedat`) so the WebUI toggle
shows a real state between polls. Every `mtconnect_*` action SHOULD probe and
report the resulting run state so the toggle never lies.
### 7.3 Enqueue (WebUI side, for reference)
`POST /api/endpoints/<assetid>/commands` with `{ "action": "mtconnect_start" }`,
gated by user JWT + `endpoints.command`. Only one command may be in flight per
endpoint (a second enqueue returns HTTP 409 until the first finalizes).