ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Endpoint telemetry, PC Detail rework and a checked backup/restore path

Browse Source
This commit is contained in:
Cameron Proudlock
2026-09-06 18:53:11 -04:00
parent 32b8745d10
commit 993353dfc0
97 changed files with 13340 additions and 756 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
plugins/endpoints/AGENT-CONTRACT.md
View File

@@ -10,8 +10,8 @@ schema changes, update this file in the same change.
- Method/URL: `POST {baseurl}/api/collector/endpoints`
- Auth header: `X-API-Key: <collector key>` (Bearer token also accepted). The
key is a UI-managed, endpoints-only service token minted from Settings ->
Endpoint agent key (section 6). A bootstrap `.env` collector key still works
until the first UI key is generated, then it is switched off (section 6). No
Endpoint agent key (section 7). A bootstrap `.env` collector key still works
until the first UI key is generated, then it is switched off (section 7). No
querystring fallback.
- Body: `application/json`, a single flat object (NOT an array). One POST per
report per endpoint.
@@ -51,9 +51,69 @@ arrays. `adopt` is first-contact only.
}
],
"scheduledtasks": [ /* opaque array */ ],
"disks": [
{
"drive": "C:",
"label": "OS",
"filesystem": "NTFS",
"sizebytes": 511000000000,
"freebytes": 128000000000,
"usedbytes": 383000000000,
"freepercent": 25.0 // precomputed; a threshold is a direct compare
}
],
"bios": { /* opaque object: vendor, version, release date */ },
"memory": { /* opaque object: totalbytes, freebytes */ },
"processor": { /* opaque object: name, cores, logicalprocessors, maxclockmhz */ },
"osinfo": { /* opaque object: build, version, install date */ },
"domain": "shopfloor.example.com", // domain or workgroup membership
"pendingreboot": false, // omit when unknown; absent != false
"rebootreasons": [], // [] = checked, nothing pending; see notes
"bootinfo": {
"hiberbootenabled": false, // fast startup CONFIGURED
"lastboottype": "Cold" // what the last boot DID: Cold|Fast startup|Resume|Unknown
},
"defender": { // persisted, deliberately not surfaced
"serviceenabled": true,
"realtimeenabled": true,
"signatureagedays": 0, // real 0 = updated today; null = did not report
"antispywareagedays": 0,
"quickscanagedays": 0
},
"diskhealth": [ // per PHYSICAL device; see notes
{
"deviceid": "0",
"model": "Samsung SSD 990 PRO 1TB",
"mediatype": "SSD",
"bustype": "NVMe",
"healthstatus": "Healthy",
"predictfailure": false, // precomputed; null = drive did not answer
"source": "physicaldisk" // or "diskdrive"
}
],
"usercontext": { // omitted entirely when nobody is signed in
"netuse": [
{
"localname": "Z:",
"remotename": "\\\\fileserver\\programs",
"status": "OK",
"username": "SHOPFLOOR\\operator",
"persistent": true
}
],
"userlisteners": [
{ "localaddress": "0.0.0.0", "localport": 7878, "processid": 4512, "processname": "DncMain" }
],
"idleseconds": 45,
"username": "operator",
"elevated": false, // token elevation of the collecting process;
// a spool stamped true is REJECTED (see 5.3)
"collectedat": "2025-01-14T06:12:00Z"
},
"events": [
{
"eventid": 6008, // required per item; non-catalog IDs are dropped, not errored
"eventrecordid": 184412, // Windows record id; dedupe key, see notes
"eventchannel": "System",
"eventsource": "EventLog",
"eventtime": "2025-01-14T06:11:58Z",
@@ -81,6 +141,71 @@ Notes:
(`HKLM\SOFTWARE\[WOW6432Node\]GE Aircraft Engines\DNC\General:MachineNo`) or
the UDC `udc_settings_*.json` (`GeneralSettings.MachineNumber`), in that order
(DESIGN-DECISIONS-agent.md 9.4/9.5). `""` or `"9999"` means none.
- `eventrecordid` is the Windows event record id: monotonic and unique within a
channel on a machine. The agent uses it as a per-channel harvest watermark and
the server enforces a unique index on
`(assetid, eventchannel, eventrecordid)`, so one real event is stored exactly
once. It is nullable: an older agent omits it, and MySQL permits repeated
NULLs in a unique index, so those reports keep the previous append behavior
rather than failing ingest. Send it whenever the source record exposes one.
- `disks`/`bios`/`memory`/`processor`/`osinfo`/`usercontext` are stored verbatim
as JSON snapshots on the endpoint state, newest-wins. `domain` and
`pendingreboot` are scalar columns because the fleet view filters and alerts
on them.
- OMITTING a key is not the same as sending an empty value. Every one of these
fields is applied only when the key is PRESENT, so a report that omits it
preserves the last-known value. Send `null`/`[]` only when you mean "this is
genuinely empty now"; omit when you simply did not collect it this cycle.
- `pendingreboot` follows that rule strictly: omit it when the check failed. A
literal `false` asserts the endpoint is confirmed NOT holding a reboot.
- `rebootreasons` is NOT redundant with `pendingreboot`. That key is a single
bool OR-ed from three registry signals and cannot say which fired, and the
difference is operational: `pendingfilerename` means files are staged for
replacement on next boot, so the box is mid-servicing, while a lone
`componentbasedservicing` or `windowsupdate` key can sit set for weeks on a
machine nobody has rebooted. Same boolean, very different urgency. An EMPTY
array means checked and nothing pending; OMIT the key when the check failed.
- `bootinfo` carries two distinct facts that must not be collapsed:
`hiberbootenabled` is whether fast startup is CONFIGURED (registry, always
readable) and `lastboottype` is what the last boot ACTUALLY did (Kernel-Boot
event 27). The registry cannot tell you what the last boot did, and the event
may have aged out of the log, so neither substitutes for the other. This
exists because uptime LIES under fast startup: a box reports days of uptime
after what the operator experienced as a shutdown.
- `defender` is stored but deliberately NOT surfaced. Defender produces known
false positives in this environment, so wiring it to a card now would train
operators to ignore that card. Signature/scan ages are in DAYS; a real `0`
means updated today, so send `null` when Defender did not report rather than
defaulting to zero.
- `interfaces` entries gained `linkspeedbps` and `fullduplex`, paired on by MAC
from `MSFT_NetAdapter`. Both are `null` when link state was unreadable or the
adapter is disconnected - an unknown duplex must not read as half-duplex.
- `diskhealth` is per PHYSICAL device and is NOT the same list as `disks`, which
is per volume. One physical drive can back several volumes or none the
operator sees, so the two do not share a key and are not merged.
- `diskhealth` carries BOTH agent sources and the server does not reconcile
them. `source: "physicaldisk"` is MSFT_PhysicalDisk, the only source that
reports NVMe at all - the classic ATA SMART class
(`MSStorageDriver_FailurePredictStatus`) returns zero instances on an NVMe
machine, so an agent collecting only that would report an empty health list on
modern hardware and look like "no problems". `source: "diskdrive"` is
Win32_DiskDrive.Status, covering ATA/SATA and hosts where the Storage
namespace is unavailable. When both answer for one drive and disagree, both
rows are shown; that disagreement is the signal.
- `predictfailure` is precomputed by the agent so an alert is a boolean test
rather than a string match against a Windows enum the server would have to
keep in sync. `null` means the drive did not answer: unknown, NOT healthy.
`healthstatus` keeps the human-readable source value alongside it.
- `usercontext` is gathered by a probe run inside the signed-in operator's
session by the `ShopDB user-context probe` scheduled task (and, opportunistically,
by the tray), because mapped drives, user-owned listeners and idle time
do not exist in the agent's own LocalSystem session 0. It is omitted when no
one is signed in, which is a normal unattended state, not a fault. The
security-context split that makes this work is normative; see section 5.
- `ports` carries LISTENING sockets only. The full connection table is available
on demand via the `network_connections` command instead, because established
connections churn constantly and would change every report for no diagnostic
gain.
## 3. Response
@@ -137,13 +262,23 @@ public sealed class EndpointReport
[JsonPropertyName("ports")] public object[]? Ports { get; set; }
[JsonPropertyName("interfaces")] public object[]? Interfaces { get; set; }
[JsonPropertyName("scheduledtasks")]public object[]? ScheduledTasks { get; set; }
[JsonPropertyName("disks")] public object[]? Disks { get; set; }
[JsonPropertyName("bios")] public object? Bios { get; set; }
[JsonPropertyName("memory")] public object? Memory { get; set; }
[JsonPropertyName("processor")] public object? Processor { get; set; }
[JsonPropertyName("osinfo")] public object? OsInfo { get; set; }
[JsonPropertyName("domain")] public string? Domain { get; set; }
[JsonPropertyName("pendingreboot")] public bool? PendingReboot { get; set; }
[JsonPropertyName("usercontext")] public object? UserContext { get; set; }
[JsonPropertyName("applications")] public object[]? Applications { get; set; }
[JsonPropertyName("events")] public EndpointEventRecord[]? Events { get; set; }
}
public sealed class EndpointEventRecord
{
[JsonPropertyName("eventid")] public int EventId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventid")] public int EventId { get; set; }
[JsonPropertyName("eventrecordid")] public long? EventRecordId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
[JsonPropertyName("eventmessage")] public string? EventMessage { get; set; }
@@ -163,7 +298,102 @@ public sealed class CollectorResponse
Serializer settings: `DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull`
so omitted fields are not sent (server upserts only present keys). Send UTC ISO-8601.
## 5. "Build site agent" settings flow (build cadence)
## 5. Security contexts, spool handoff and log destinations (NORMATIVE)
Three processes run as three different principals. Confusing them is what makes
`usercontext` silently empty, so the boundaries are fixed by contract.
| Process | Runs as | May write to |
|---|---|---|
| Service (`shopdbagent.exe`) | LocalSystem, session 0 | `%ProgramData%\ShopDB\EndpointAgent\logs\<date>-agent.log` |
| Tray + probe (`shopdb-endpoint-tray.exe`, `shopdbagent.exe --collect-usercontext`) | `ShopFloor` operator, interactive session, UNELEVATED (task principal `S-1-5-32-545` at `RunLevel Limited`) | `C:\Users\ShopFloor\ShopDB\` and `C:\Users\ShopFloor\ShopDB\Logs\<date>-trayagent.log` |
| Installer (`install.ps1`, generated by `_install_script()` in `plugins/endpoints/api/siteagent.py`) | Elevated admin, one-shot | `%ProgramData%\ShopDB\Logs\<datetime>-shopdbinstall.log` and the append-only ledger `%ProgramData%\ShopDB\Logs\shopdbagent-installs.log` |
Rules:
1. **The operator cannot write to `C:\Program Files` or to the SYSTEM-owned
`%ProgramData%\ShopDB\EndpointAgent\logs`.** Anything running in the user's
session MUST log under `C:\Users\ShopFloor\ShopDB\Logs`. The file log sink
degrades to a silent no-op when it cannot create its directory, so violating
this does not raise an error - it discards the log line and the failure
becomes invisible.
2. **User context is collected in the operator's own logon session**, because
mapped drives, user-owned listeners and idle time are per-session and do not
exist in session 0. The service MUST NOT try to descend into the session to
collect them. The authoritative collector is the `ShopDB user-context probe`
scheduled task, registered by the installer against the `BUILTIN\Users` group
(`S-1-5-32-545`) at `RunLevel Limited`, triggered at logon and every 30
minutes. The tray also probes on its own cadence and writes the same spool;
the two are belt-and-braces and the last writer wins.
3. **The probe MUST NOT run elevated, and a spool stamped `elevated` is
REJECTED.** Drive letters are a property of the logon token, not the machine:
an elevated process gets a different token to the operator's interactive one
and enumerates a different, usually empty, drive table. This was measured on a
live endpoint, where the same two shares reported `Unavailable` from an
elevated shell and `OK` from the operator's, in both `Win32_NetworkConnection`
and `Get-SmbMapping`. `--collect-usercontext` therefore refuses to run when
elevated (exit code 2) rather than producing a confident wrong answer, and the
installer MUST NOT start the tray or probe as a child process - it is
elevated, and `explorer.exe` de-elevates without changing ACCOUNT, so an
install run by an admin produced a probe running as that admin. It MUST
instead reach the operator's session through a scheduled task principal.
`RunLevel Limited` on the task is load-bearing for the same reason; a task
promoted to `Highest` reproduces the bug while still looking registered.
4. **The probe WRITES the spool; the service only READS it.** The spool PERSISTS
and is overwritten by each probe run. The service MUST NOT delete it: the
collector's last-known-good cache is in-memory only, so consuming the file
meant a service restart blanked mapped drives until the next probe cycle.
Staleness is bounded by the in-file `collectedat` stamp (max age 90 minutes),
not by deletion, so a probe that has stopped running still stops re-reporting
stale drives - it just does so on a timer instead of on a read.
5. **The spool location is fixed at `C:\Users\ShopFloor\ShopDB\usercontext.json`,
and writer and reader MUST derive it from the same helper.** Every shop-floor
endpoint uses the same local `ShopFloor` account. Resolving the path through
`WTSQueryUserToken` or through `%USERPROFILE%` is forbidden: the former is
privileged and endpoint protection blocks it, the latter is wrong under a
brokered or re-parented tray launch, and in both cases the writer and reader
disagree about the location with no error on either side.
6. **The probe MUST verify the spool exists after writing it.** A zero exit code
from `powershell.exe` proves the interpreter started, not that a file was
produced; success is reported only when the file is present on disk.
7. **A spool not stamped `ShopFloor` is REJECTED.**
collecting `username` and the service discards any spool whose stamp is
missing or different. Drive mappings from the wrong session are worse than
no drive mappings, so this fails loudly rather than degrading quietly.
8. **Losing user context is never fatal.** If the `ShopFloor` profile is absent
or unwritable, `usercontext` is omitted and telemetry, commands, inventory
and the command channel continue unaffected.
9. **`--collect-usercontext` MUST NOT touch privileged resources.** It runs as
the operator and only writes into that operator's own profile, so it is
handled before the service's bootstrap logger, fault monitor and config load.
It MUST NOT open the SYSTEM-owned log directory, register an EventLog source,
or read `CollectorKey` from HKLM - those are precisely the calls an EPM broker
intercepts, and the registry read would put the collector secret in a process
that has no need of it. It therefore also works on a host whose CollectorKey
has not been provisioned yet.
10. **The installer provisions** `C:\Users\ShopFloor\ShopDB\` and its `Logs`
subdirectory with `ShopFloor: Modify` before the probe first runs.
11. **Every install appends one line to the install ledger**, including
reinstalls and repeat installs of the same version, recording timestamp,
host, installing user, service version, tray version, install directory and
the user-context task state (`ok:<principal>/<runlevel>`, `UNVERIFIED:...`,
`FAILED:...` or `skipped`). The task is the single point of failure for
session-scoped collection, so its registration outcome MUST be recorded
where it survives the install transcript.
12. **Task registration MUST be verified by reading the task back**, not inferred
from `Register-ScheduledTask` returning without error. Policy or a
management agent can rewrite or drop a task after registration, and a task
that exists but runs with the wrong principal or run level looks healthy in
the scheduler UI while producing wrong data.
13. **All log filenames use `<stamp>-<component>.log`** so service, tray and
installer logs sort chronologically and collect with one date-leading glob.
The three log destinations are deliberately separate: an install is one-shot and
elevated, the service log is a rolling runtime log, and the tray log is the only
one written from the operator's session. Merging them makes an install
indistinguishable from runtime behaviour when reading a failure back.
## 6. "Build site agent" settings flow (build cadence)
Goal: an operator in ShopDB Settings produces an instance-locked `.exe` that
knows its server URL and its collector key, needing no manual config on the
@@ -190,7 +420,7 @@ Open item (from DESIGN-DECISIONS-agent.md 1a): whether `adopt` stays a
per-report flag or is promoted to a site setting (strict vs self-adopting
fleet). Default proposed: agent sends `adopt=true` on first run only.
## 6. UI-managed agent key (kill switch)
## 7. UI-managed agent key (kill switch)
The endpoint agent authenticates with a single UI-managed key scoped to ONLY
`collector.ingest.endpoints`, so rotating or revoking it stops all
@@ -215,13 +445,13 @@ The bootstrap `.env` key (`COLLECTOR_API_KEY[_ENDPOINTS]`) remains only as a
first-run path before a UI key exists; it cannot be the runtime kill switch
because container env is injected at boot and cannot be rotated live.
## 7. Command channel (WebUI -> agent)
## 8. Command channel (WebUI -> agent)
The WebUI can send an endpoint an ALLOWLISTED command (never an arbitrary
command string). Delivery is agent-initiated long-poll, so shop-floor PCs open
no inbound port: the agent makes an outbound request the server holds open until
a command is queued, runs it locally, and posts the result back. All three
routes reuse the same endpoints collector key (section 6) via `X-API-Key`, so
routes reuse the same endpoints collector key (section 7) via `X-API-Key`, so
the command channel shares the kill switch and adds no second secret.
Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
@@ -230,6 +460,11 @@ Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
(`C:\ProgramData\UDC\MTConnect_UDC\Agent\run.bat`).
- `mtconnect_stop` - stop the running MTConnect UDC agent process.
- `mtconnect_status` - report whether the MTConnect UDC agent is running.
- `network_connections` - return the full TCP connection table with owning
process ids and names. Read-only diagnostic. The result is capped at 500
connections with a `truncated` flag in `detail`, because an oversized payload
rejected by the server returns nothing at all and a partial answer is more
useful than a failed one. Standard reports carry listening ports only.
### 7.1 Claim the next command (long-poll)