ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Import the wilm endpoint telemetry, PC Detail rework and backup overlay

Browse Source 
Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at 7905917c. wilm rides the publication lineage and shares no
ancestor with wj, so this is a filtered tree patch rather than a merge:
94 files, and every path the publication export strips (docs/, tools/, mcp/,
CLAUDE.md, start-*.sh, .gitea/, the three excluded tests) is held back, because
a raw diff would delete each of them from this tree.

What comes across:

* Endpoint telemetry, phase 2. Physical disk health from two sources, NIC link
  state, boot type, Defender health, processor/memory/chassis inventory, the
  listening TCP set, mapped network drives and Windows Update history. Five
  additive migrations, 0012 through 0016, off the 0011 head.
* Mapped drives are collected by a probe running in the interactive user's
  session, because they do not exist in the agent's session 0. Two Windows
  sources can disagree about one drive and the worse status wins, so a
  half-broken share is never rendered as healthy.
* Windows events are reported once each, keyed on the channel's own record id,
  instead of being re-sent whenever a report overlapped the last one.
* PC Detail rework: Details cards flow, the hand-written Network card gives way
  to a full address history, Storage and disk health merge into one card with
  two sections, Operator Reports and Corporate Tools get their own tabs.
* An opt-in restic backup overlay with a third container, restore gated on the
  dump's own data revision rather than the code head, and backup/restore history.
* The agent's version is read from the binary's PE version rather than trusted
  from the csproj, and the Fleet view gains an Agent Version column.
* The enqueue route now asks caller_holds instead of the owner's permissions,
  which closes the scoped-token elevation on run_script, update_app and
  system_restart.

Held back deliberately:

* agent/installer/install-shopdbagent.ps1 stays deleted, as upstream deleted it.
  The installer that reaches a bay is generated by _install_script() in
  plugins/endpoints/api/siteagent.py; the standalone script had stopped being
  shipped and was collecting fixes no endpoint ever ran.
* The .gitignore hunk adding docs/*.md. Correct on the publication lineage,
  where docs/ becomes the wiki and its absence is how the gates detect a
  published checkout. Here it would silently ignore every new doc.
* Their deletions of docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md. Upstream
  those were stubs; here PLUGIN-HOOKS.md is the 31KB document CLAUDE.md points
  readers at.
* Five zero-byte files committed by an accidental `git add -A`, a run log from
  the author's own machine, and docs/notes/, which is working scratch that says
  so in its own header and belongs in devnotes/ on this side.

Carried with the import so it lands green: the 17 new routes are described in
docs/api-inventory.json with the spec and project map regenerated, and two new
frontend specs read fixtures through fileURLToPath(URL) which node rejects under
jsdom's own URL global.

Suite 2591 passed / 9 skipped; frontend 391 passed; naming, style and doc gates
clean.

Scope: product
This commit is contained in:
cproudlock
2026-09-06 16:47:24 -04:00
parent 9947a477b8
commit 6e36df841e
96 changed files with 12395 additions and 761 deletions
Download Patch File Download Diff File Expand all files Collapse all files

883
plugins/computers/frontend/views/PCDetail.vue
View File

File diff suppressed because it is too large Load Diff

12
plugins/computers/frontend/views/PCForm.vue
View File

@@ -573,12 +573,14 @@ onMounted(async () => {
operatingsystemsApi.list({ perpage: 100 }),
computersApi.protocols.list(),
businessunitsApi.listAll(),
// Handles its own failure: a site without printers still gets a form.
])
pcTypes.value = ptRes.data.data || []
statuses.value = statusRes.data.data || []
vendors.value = vendorRes.data.data || []
// listAll() pages past the server's 100-row cap and returns the ROWS, not
// an axios response, so these are read directly. Reading .data.data here
// throws and takes the whole form down with it.
vendors.value = vendorRes || []
models.value = allModels
locations.value = locRes || []
operatingsystems.value = osRes.data.data || []
@@ -636,7 +638,11 @@ onMounted(async () => {
}
} catch (err) {
console.error('Error loading data:', err)
error.value = 'Failed to load data'
// A client-side TypeError has no err.response, so apiError() would report
// only the fallback and hide which field broke. Name it either way.
error.value = err?.response
? apiError(err, 'Failed to load data')
: `Failed to load data: ${err?.message || err}`
} finally {
loading.value = false
}

251
plugins/endpoints/AGENT-CONTRACT.md
View File

@@ -10,8 +10,8 @@ schema changes, update this file in the same change.
- Method/URL: `POST {baseurl}/api/collector/endpoints`
- Auth header: `X-API-Key: <collector key>` (Bearer token also accepted). The
key is a UI-managed, endpoints-only service token minted from Settings ->
Endpoint agent key (section 6). A bootstrap `.env` collector key still works
until the first UI key is generated, then it is switched off (section 6). No
Endpoint agent key (section 7). A bootstrap `.env` collector key still works
until the first UI key is generated, then it is switched off (section 7). No
querystring fallback.
- Body: `application/json`, a single flat object (NOT an array). One POST per
report per endpoint.
@@ -51,9 +51,69 @@ arrays. `adopt` is first-contact only.
}
],
"scheduledtasks": [ /* opaque array */ ],
"disks": [
{
"drive": "C:",
"label": "OS",
"filesystem": "NTFS",
"sizebytes": 511000000000,
"freebytes": 128000000000,
"usedbytes": 383000000000,
"freepercent": 25.0 // precomputed; a threshold is a direct compare
}
],
"bios": { /* opaque object: vendor, version, release date */ },
"memory": { /* opaque object: totalbytes, freebytes */ },
"processor": { /* opaque object: name, cores, logicalprocessors, maxclockmhz */ },
"osinfo": { /* opaque object: build, version, install date */ },
"domain": "shopfloor.example.com", // domain or workgroup membership
"pendingreboot": false, // omit when unknown; absent != false
"rebootreasons": [], // [] = checked, nothing pending; see notes
"bootinfo": {
"hiberbootenabled": false, // fast startup CONFIGURED
"lastboottype": "Cold" // what the last boot DID: Cold|Fast startup|Resume|Unknown
},
"defender": { // persisted, deliberately not surfaced
"serviceenabled": true,
"realtimeenabled": true,
"signatureagedays": 0, // real 0 = updated today; null = did not report
"antispywareagedays": 0,
"quickscanagedays": 0
},
"diskhealth": [ // per PHYSICAL device; see notes
{
"deviceid": "0",
"model": "Samsung SSD 990 PRO 1TB",
"mediatype": "SSD",
"bustype": "NVMe",
"healthstatus": "Healthy",
"predictfailure": false, // precomputed; null = drive did not answer
"source": "physicaldisk" // or "diskdrive"
}
],
"usercontext": { // omitted entirely when nobody is signed in
"netuse": [
{
"localname": "Z:",
"remotename": "\\\\fileserver\\programs",
"status": "OK",
"username": "SHOPFLOOR\\operator",
"persistent": true
}
],
"userlisteners": [
{ "localaddress": "0.0.0.0", "localport": 7878, "processid": 4512, "processname": "DncMain" }
],
"idleseconds": 45,
"username": "operator",
"elevated": false, // token elevation of the collecting process;
// a spool stamped true is REJECTED (see 5.3)
"collectedat": "2025-01-14T06:12:00Z"
},
"events": [
{
"eventid": 6008, // required per item; non-catalog IDs are dropped, not errored
"eventrecordid": 184412, // Windows record id; dedupe key, see notes
"eventchannel": "System",
"eventsource": "EventLog",
"eventtime": "2025-01-14T06:11:58Z",
@@ -81,6 +141,71 @@ Notes:
(`HKLM\SOFTWARE\[WOW6432Node\]GE Aircraft Engines\DNC\General:MachineNo`) or
the UDC `udc_settings_*.json` (`GeneralSettings.MachineNumber`), in that order
(DESIGN-DECISIONS-agent.md 9.4/9.5). `""` or `"9999"` means none.
- `eventrecordid` is the Windows event record id: monotonic and unique within a
channel on a machine. The agent uses it as a per-channel harvest watermark and
the server enforces a unique index on
`(assetid, eventchannel, eventrecordid)`, so one real event is stored exactly
once. It is nullable: an older agent omits it, and MySQL permits repeated
NULLs in a unique index, so those reports keep the previous append behavior
rather than failing ingest. Send it whenever the source record exposes one.
- `disks`/`bios`/`memory`/`processor`/`osinfo`/`usercontext` are stored verbatim
as JSON snapshots on the endpoint state, newest-wins. `domain` and
`pendingreboot` are scalar columns because the fleet view filters and alerts
on them.
- OMITTING a key is not the same as sending an empty value. Every one of these
fields is applied only when the key is PRESENT, so a report that omits it
preserves the last-known value. Send `null`/`[]` only when you mean "this is
genuinely empty now"; omit when you simply did not collect it this cycle.
- `pendingreboot` follows that rule strictly: omit it when the check failed. A
literal `false` asserts the endpoint is confirmed NOT holding a reboot.
- `rebootreasons` is NOT redundant with `pendingreboot`. That key is a single
bool OR-ed from three registry signals and cannot say which fired, and the
difference is operational: `pendingfilerename` means files are staged for
replacement on next boot, so the box is mid-servicing, while a lone
`componentbasedservicing` or `windowsupdate` key can sit set for weeks on a
machine nobody has rebooted. Same boolean, very different urgency. An EMPTY
array means checked and nothing pending; OMIT the key when the check failed.
- `bootinfo` carries two distinct facts that must not be collapsed:
`hiberbootenabled` is whether fast startup is CONFIGURED (registry, always
readable) and `lastboottype` is what the last boot ACTUALLY did (Kernel-Boot
event 27). The registry cannot tell you what the last boot did, and the event
may have aged out of the log, so neither substitutes for the other. This
exists because uptime LIES under fast startup: a box reports days of uptime
after what the operator experienced as a shutdown.
- `defender` is stored but deliberately NOT surfaced. Defender produces known
false positives in this environment, so wiring it to a card now would train
operators to ignore that card. Signature/scan ages are in DAYS; a real `0`
means updated today, so send `null` when Defender did not report rather than
defaulting to zero.
- `interfaces` entries gained `linkspeedbps` and `fullduplex`, paired on by MAC
from `MSFT_NetAdapter`. Both are `null` when link state was unreadable or the
adapter is disconnected - an unknown duplex must not read as half-duplex.
- `diskhealth` is per PHYSICAL device and is NOT the same list as `disks`, which
is per volume. One physical drive can back several volumes or none the
operator sees, so the two do not share a key and are not merged.
- `diskhealth` carries BOTH agent sources and the server does not reconcile
them. `source: "physicaldisk"` is MSFT_PhysicalDisk, the only source that
reports NVMe at all - the classic ATA SMART class
(`MSStorageDriver_FailurePredictStatus`) returns zero instances on an NVMe
machine, so an agent collecting only that would report an empty health list on
modern hardware and look like "no problems". `source: "diskdrive"` is
Win32_DiskDrive.Status, covering ATA/SATA and hosts where the Storage
namespace is unavailable. When both answer for one drive and disagree, both
rows are shown; that disagreement is the signal.
- `predictfailure` is precomputed by the agent so an alert is a boolean test
rather than a string match against a Windows enum the server would have to
keep in sync. `null` means the drive did not answer: unknown, NOT healthy.
`healthstatus` keeps the human-readable source value alongside it.
- `usercontext` is gathered by a probe run inside the signed-in operator's
session by the `ShopDB user-context probe` scheduled task (and, opportunistically,
by the tray), because mapped drives, user-owned listeners and idle time
do not exist in the agent's own LocalSystem session 0. It is omitted when no
one is signed in, which is a normal unattended state, not a fault. The
security-context split that makes this work is normative; see section 5.
- `ports` carries LISTENING sockets only. The full connection table is available
on demand via the `network_connections` command instead, because established
connections churn constantly and would change every report for no diagnostic
gain.
## 3. Response
@@ -137,13 +262,23 @@ public sealed class EndpointReport
[JsonPropertyName("ports")] public object[]? Ports { get; set; }
[JsonPropertyName("interfaces")] public object[]? Interfaces { get; set; }
[JsonPropertyName("scheduledtasks")]public object[]? ScheduledTasks { get; set; }
[JsonPropertyName("disks")] public object[]? Disks { get; set; }
[JsonPropertyName("bios")] public object? Bios { get; set; }
[JsonPropertyName("memory")] public object? Memory { get; set; }
[JsonPropertyName("processor")] public object? Processor { get; set; }
[JsonPropertyName("osinfo")] public object? OsInfo { get; set; }
[JsonPropertyName("domain")] public string? Domain { get; set; }
[JsonPropertyName("pendingreboot")] public bool? PendingReboot { get; set; }
[JsonPropertyName("usercontext")] public object? UserContext { get; set; }
[JsonPropertyName("applications")] public object[]? Applications { get; set; }
[JsonPropertyName("events")] public EndpointEventRecord[]? Events { get; set; }
}
public sealed class EndpointEventRecord
{
[JsonPropertyName("eventid")] public int EventId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventid")] public int EventId { get; set; }
[JsonPropertyName("eventrecordid")] public long? EventRecordId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
[JsonPropertyName("eventmessage")] public string? EventMessage { get; set; }
@@ -163,7 +298,102 @@ public sealed class CollectorResponse
Serializer settings: `DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull`
so omitted fields are not sent (server upserts only present keys). Send UTC ISO-8601.
## 5. "Build site agent" settings flow (build cadence)
## 5. Security contexts, spool handoff and log destinations (NORMATIVE)
Three processes run as three different principals. Confusing them is what makes
`usercontext` silently empty, so the boundaries are fixed by contract.
| Process | Runs as | May write to |
|---|---|---|
| Service (`shopdbagent.exe`) | LocalSystem, session 0 | `%ProgramData%\ShopDB\EndpointAgent\logs\<date>-agent.log` |
| Tray + probe (`shopdb-endpoint-tray.exe`, `shopdbagent.exe --collect-usercontext`) | `ShopFloor` operator, interactive session, UNELEVATED (task principal `S-1-5-32-545` at `RunLevel Limited`) | `C:\Users\ShopFloor\ShopDB\` and `C:\Users\ShopFloor\ShopDB\Logs\<date>-trayagent.log` |
| Installer (`install.ps1`, generated by `_install_script()` in `plugins/endpoints/api/siteagent.py`) | Elevated admin, one-shot | `%ProgramData%\ShopDB\Logs\<datetime>-shopdbinstall.log` and the append-only ledger `%ProgramData%\ShopDB\Logs\shopdbagent-installs.log` |
Rules:
1. **The operator cannot write to `C:\Program Files` or to the SYSTEM-owned
`%ProgramData%\ShopDB\EndpointAgent\logs`.** Anything running in the user's
session MUST log under `C:\Users\ShopFloor\ShopDB\Logs`. The file log sink
degrades to a silent no-op when it cannot create its directory, so violating
this does not raise an error - it discards the log line and the failure
becomes invisible.
2. **User context is collected in the operator's own logon session**, because
mapped drives, user-owned listeners and idle time are per-session and do not
exist in session 0. The service MUST NOT try to descend into the session to
collect them. The authoritative collector is the `ShopDB user-context probe`
scheduled task, registered by the installer against the `BUILTIN\Users` group
(`S-1-5-32-545`) at `RunLevel Limited`, triggered at logon and every 30
minutes. The tray also probes on its own cadence and writes the same spool;
the two are belt-and-braces and the last writer wins.
3. **The probe MUST NOT run elevated, and a spool stamped `elevated` is
REJECTED.** Drive letters are a property of the logon token, not the machine:
an elevated process gets a different token to the operator's interactive one
and enumerates a different, usually empty, drive table. This was measured on a
live endpoint, where the same two shares reported `Unavailable` from an
elevated shell and `OK` from the operator's, in both `Win32_NetworkConnection`
and `Get-SmbMapping`. `--collect-usercontext` therefore refuses to run when
elevated (exit code 2) rather than producing a confident wrong answer, and the
installer MUST NOT start the tray or probe as a child process - it is
elevated, and `explorer.exe` de-elevates without changing ACCOUNT, so an
install run by an admin produced a probe running as that admin. It MUST
instead reach the operator's session through a scheduled task principal.
`RunLevel Limited` on the task is load-bearing for the same reason; a task
promoted to `Highest` reproduces the bug while still looking registered.
4. **The probe WRITES the spool; the service only READS it.** The spool PERSISTS
and is overwritten by each probe run. The service MUST NOT delete it: the
collector's last-known-good cache is in-memory only, so consuming the file
meant a service restart blanked mapped drives until the next probe cycle.
Staleness is bounded by the in-file `collectedat` stamp (max age 90 minutes),
not by deletion, so a probe that has stopped running still stops re-reporting
stale drives - it just does so on a timer instead of on a read.
5. **The spool location is fixed at `C:\Users\ShopFloor\ShopDB\usercontext.json`,
and writer and reader MUST derive it from the same helper.** Every shop-floor
endpoint uses the same local `ShopFloor` account. Resolving the path through
`WTSQueryUserToken` or through `%USERPROFILE%` is forbidden: the former is
privileged and endpoint protection blocks it, the latter is wrong under a
brokered or re-parented tray launch, and in both cases the writer and reader
disagree about the location with no error on either side.
6. **The probe MUST verify the spool exists after writing it.** A zero exit code
from `powershell.exe` proves the interpreter started, not that a file was
produced; success is reported only when the file is present on disk.
7. **A spool not stamped `ShopFloor` is REJECTED.**
collecting `username` and the service discards any spool whose stamp is
missing or different. Drive mappings from the wrong session are worse than
no drive mappings, so this fails loudly rather than degrading quietly.
8. **Losing user context is never fatal.** If the `ShopFloor` profile is absent
or unwritable, `usercontext` is omitted and telemetry, commands, inventory
and the command channel continue unaffected.
9. **`--collect-usercontext` MUST NOT touch privileged resources.** It runs as
the operator and only writes into that operator's own profile, so it is
handled before the service's bootstrap logger, fault monitor and config load.
It MUST NOT open the SYSTEM-owned log directory, register an EventLog source,
or read `CollectorKey` from HKLM - those are precisely the calls an EPM broker
intercepts, and the registry read would put the collector secret in a process
that has no need of it. It therefore also works on a host whose CollectorKey
has not been provisioned yet.
10. **The installer provisions** `C:\Users\ShopFloor\ShopDB\` and its `Logs`
subdirectory with `ShopFloor: Modify` before the probe first runs.
11. **Every install appends one line to the install ledger**, including
reinstalls and repeat installs of the same version, recording timestamp,
host, installing user, service version, tray version, install directory and
the user-context task state (`ok:<principal>/<runlevel>`, `UNVERIFIED:...`,
`FAILED:...` or `skipped`). The task is the single point of failure for
session-scoped collection, so its registration outcome MUST be recorded
where it survives the install transcript.
12. **Task registration MUST be verified by reading the task back**, not inferred
from `Register-ScheduledTask` returning without error. Policy or a
management agent can rewrite or drop a task after registration, and a task
that exists but runs with the wrong principal or run level looks healthy in
the scheduler UI while producing wrong data.
13. **All log filenames use `<stamp>-<component>.log`** so service, tray and
installer logs sort chronologically and collect with one date-leading glob.
The three log destinations are deliberately separate: an install is one-shot and
elevated, the service log is a rolling runtime log, and the tray log is the only
one written from the operator's session. Merging them makes an install
indistinguishable from runtime behaviour when reading a failure back.
## 6. "Build site agent" settings flow (build cadence)
Goal: an operator in ShopDB Settings produces an instance-locked `.exe` that
knows its server URL and its collector key, needing no manual config on the
@@ -190,7 +420,7 @@ Open item (from DESIGN-DECISIONS-agent.md 1a): whether `adopt` stays a
per-report flag or is promoted to a site setting (strict vs self-adopting
fleet). Default proposed: agent sends `adopt=true` on first run only.
## 6. UI-managed agent key (kill switch)
## 7. UI-managed agent key (kill switch)
The endpoint agent authenticates with a single UI-managed key scoped to ONLY
`collector.ingest.endpoints`, so rotating or revoking it stops all
@@ -215,13 +445,13 @@ The bootstrap `.env` key (`COLLECTOR_API_KEY[_ENDPOINTS]`) remains only as a
first-run path before a UI key exists; it cannot be the runtime kill switch
because container env is injected at boot and cannot be rotated live.
## 7. Command channel (WebUI -> agent)
## 8. Command channel (WebUI -> agent)
The WebUI can send an endpoint an ALLOWLISTED command (never an arbitrary
command string). Delivery is agent-initiated long-poll, so shop-floor PCs open
no inbound port: the agent makes an outbound request the server holds open until
a command is queued, runs it locally, and posts the result back. All three
routes reuse the same endpoints collector key (section 6) via `X-API-Key`, so
routes reuse the same endpoints collector key (section 7) via `X-API-Key`, so
the command channel shares the kill switch and adds no second secret.
Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
@@ -230,6 +460,11 @@ Allowlisted actions (server source of truth: `plugins/endpoints/commands.py`):
(`C:\ProgramData\UDC\MTConnect_UDC\Agent\run.bat`).
- `mtconnect_stop` - stop the running MTConnect UDC agent process.
- `mtconnect_status` - report whether the MTConnect UDC agent is running.
- `network_connections` - return the full TCP connection table with owning
process ids and names. Read-only diagnostic. The result is capped at 500
connections with a `truncated` flag in `detail`, because an oversized payload
rejected by the server returns nothing at all and a partial answer is more
useful than a failed one. Standard reports carry listening ports only.
### 7.1 Claim the next command (long-poll)

37
plugins/endpoints/api/agentupload.py
View File

@@ -26,10 +26,18 @@ from shopdb.api import (success_response, error_response, ErrorCodes, AuditLog,
from .routes import endpoints_bp
from . import agentstore
# Hard cap on an uploaded agent binary. The real agent is a few MB; this is a
# generous ceiling that still refuses an operator who points at a giant wrong
# file before it is buffered fully into memory-staged storage.
_MAX_UPLOAD_BYTES = 64 * 1024 * 1024
# Hard cap on an uploaded agent binary. This is a generous ceiling that still
# refuses an operator who points at a giant wrong file before it is buffered
# fully into memory-staged storage.
#
# 128 MB, not 64: the self-contained single-file tray publishes at ~69 MB
# (it carries the whole .NET runtime plus WinForms), which the old 64 MB cap
# rejected outright. The upload failed on SIZE, but the operator saw only a
# version-detection failure, because the size check returns before the PE is
# ever parsed. Config.MAX_CONTENT_LENGTH must stay above this - it bounds the
# whole multipart envelope, so a cap equal to it would reject a legal file on
# the boundary bytes alone.
_MAX_UPLOAD_BYTES = 128 * 1024 * 1024
@endpoints_bp.route('/siteagent/binaries/<binkey>', methods=['GET'])
@@ -69,11 +77,6 @@ def upload_agent_binary(binkey):
ErrorCodes.VALIDATION_ERROR,
'binkey must be one of: agent, tray',
http_code=400)
if not agentstore.is_valid_version(version):
return error_response(
ErrorCodes.VALIDATION_ERROR,
'version must be a dotted-numeric string such as 1.6.3',
http_code=400)
upload = request.files.get('file')
if upload is None or not upload.filename:
@@ -94,6 +97,22 @@ def upload_agent_binary(binkey):
'uploaded file is not a Windows executable',
http_code=400)
# Gate the version at the PE, not at the raw input. Product versions carry
# semver build metadata (1.5.0+abcdef...) that is not a legal directory
# name, so the operator's typed string is only a hint. The dotted-numeric
# FILE version stamped in the binary's PE resource is the authority; fall
# back to it whenever the typed value is missing or not dotted-numeric, and
# only reject when neither yields a usable version.
if not agentstore.is_valid_version(version):
detected = agentstore.read_pe_file_version(data)
if agentstore.is_valid_version(detected):
version = detected
else:
return error_response(
ErrorCodes.VALIDATION_ERROR,
'version must be a dotted-numeric string such as 1.6.3',
http_code=400)
result = agentstore.store_version(binkey, version, data,
make_current=makecurrent)

767
plugins/endpoints/api/routes.py
View File

@@ -87,6 +87,81 @@ def _format_uptime(seconds):
return ' '.join(parts)
def _state_json(state, attr):
"""Parse one of the EndpointState inventory JSON blobs.
These columns are written straight from the agent snapshot, so a partial
or malformed blob from one bad check-in must not take the whole panel
down. Anything unparseable reads as absent.
"""
raw = getattr(state, attr, None) if state else None
if not raw:
return None
try:
return json.loads(raw)
except (TypeError, ValueError):
return None
def _format_bytes(value):
"""Byte count -> 'N.N GB'. Sizes here are disks and RAM, never sub-MB."""
if value is None:
return None
try:
value = float(value)
except (TypeError, ValueError):
return None
for unit in ('B', 'KB', 'MB', 'GB', 'TB'):
if abs(value) < 1024.0 or unit == 'TB':
return '{:.1f} {}'.format(value, unit) if unit != 'B' else '{:.0f} B'.format(value)
value /= 1024.0
return None
def _format_agent_date(value):
"""Agent timestamps are ISO strings, not datetimes; show the date only.
A BIOS release date or an OS install date is only ever read at day
resolution, and the agent sends a full 7-digit-fraction UTC stamp.
"""
if not value:
return None
text = str(value)
return text[:10] if len(text) >= 10 else text
# A Windows event message runs to several hundred words. It is shown in an
# expander that is closed by default, so the cap here only exists to keep one
# pathological message from bloating the payload - not to fit a line. Blank
# runs are collapsed but real line breaks are kept, because the body is read
# as a block once opened.
_EVENT_BODY_LIMIT = 4000
def _event_body(event):
"""Readable message body for one curated event, bounded and normalized.
Windows messages arrive padded with long runs of blank lines that render
as ragged gaps. Runs of blank lines collapse to one and trailing spaces
go, but single newlines survive so the text keeps its shape when opened.
"""
text = (getattr(event, 'eventmessage', None) or '').strip()
if not text:
return None
lines = [line.rstrip() for line in text.splitlines()]
cleaned = []
for line in lines:
if not line and cleaned and not cleaned[-1]:
continue
cleaned.append(line)
text = '\n'.join(cleaned).strip()
if not text:
return None
if len(text) <= _EVENT_BODY_LIMIT:
return text
return text[:_EVENT_BODY_LIMIT].rstrip() + '...'
# =============================================================================
# Endpoint state
# =============================================================================
@@ -419,6 +494,60 @@ def _drift_for_state(state):
return parsed if isinstance(parsed, list) else []
def _dell_updates_summary(state):
"""Parse the cached Dell Command | Update summary off an EndpointState.
dellupdatesjson is written on check-in from the shopdb-dellupdates agent
utility, which reads the log the \\ShopDB-DellCommandScan task produces.
Null means the endpoint has never reported a scan. Tolerant of a malformed
blob so a bad cache never breaks the panel.
"""
raw = getattr(state, 'dellupdatesjson', None) if state else None
if not raw:
return None
try:
parsed = json.loads(raw)
except (TypeError, ValueError):
return None
return parsed if isinstance(parsed, dict) else None
def _dell_updates_value(state):
"""Render the informational 'Pending Dell updates' status value.
Informational only - this row never claims an endpoint needs attention,
because a pending optional driver is not a support problem. Three states are
kept distinct on purpose, and all three are VISIBLE:
never scanned -> 'Never scanned'. Deliberately not 'No': those are
different facts, and claiming 'No' would tell an operator
the machine is current when nobody has ever checked. It
is also not hidden - a missing row reads as a broken
panel, so the unknown state says so out loud.
scanned clean -> 'No'.
pending -> 'Yes (n)', with the count because one optional update and
fourteen urgent ones warrant different urgency.
"""
summary = _dell_updates_summary(state)
if not summary:
return 'Never scanned'
status = summary.get('status')
if status == 'error':
# The scan log exists but could not be read. A different problem from
# never having scanned, and worth saying so rather than implying the
# endpoint was never checked.
return 'Unknown (scan unreadable)'
if status != 'ok':
return 'Never scanned'
try:
count = int(summary.get('count') or 0)
except (TypeError, ValueError):
return 'Unknown (scan unreadable)'
if count <= 0:
return 'No'
return 'Yes ({})'.format(count)
@endpoints_bp.route('/asset/<int:asset_id>/status', methods=['GET'])
@require_permission('endpoints.view')
def panel_status(asset_id: int):
@@ -432,24 +561,34 @@ def panel_status(asset_id: int):
if not state:
return success_response({'fields': []})
def field(label, value, mono=False):
return {'label': label, 'value': value, 'mono': mono}
def field(label, value, mono=False, fmt=None):
entry = {'label': label, 'value': value, 'mono': mono}
if fmt:
entry['format'] = fmt
return entry
fields = [
field('Logged in user', state.loggedinuser),
field('Uptime', _format_uptime(state.uptimeseconds)),
field('Idle', _format_uptime(state.idleseconds)),
field('Last boot',
state.lastboottime.isoformat() + 'Z' if state.lastboottime else None),
state.lastboottime.isoformat() + 'Z' if state.lastboottime else None,
fmt='datetime'),
field('CPU %', state.cpupercent),
field('Memory %', state.memorypercent),
field('Agent', state.agentversion, mono=True),
field('Tray', state.trayversion, mono=True),
field('Last reported',
state.lastreportedat.isoformat() + 'Z' if state.lastreportedat else None),
state.lastreportedat.isoformat() + 'Z' if state.lastreportedat else None,
fmt='datetime'),
]
fields = [f for f in fields if f['value'] not in (None, '')]
# Informational only, and appended after the null-filter above because it
# always renders - including an explicit "Never scanned", which a hidden row
# would leave looking like a broken panel.
fields.append(field('Pending Dell updates', _dell_updates_value(state)))
# Warn-only reconciler summary: monitors site policy expects to be running
# that the last check-in did not report as running. Surfaced as a status
# field so an operator sees the drift on the panel; nothing is auto-started.
@@ -465,6 +604,492 @@ def panel_status(asset_id: int):
return success_response({'fields': fields})
@endpoints_bp.route('/asset/<int:asset_id>/hardware', methods=['GET'])
@require_permission('endpoints.view')
def panel_hardware(asset_id: int):
"""Keyvalue panel: processor, memory and system chassis inventory.
Only reported by agent 1.7.0 and later. An older agent leaves these
columns null, so the panel yields no fields and hides itself rather than
rendering a card full of blanks.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
if not state:
return success_response({'fields': []})
processor = _state_json(state, 'processorjson') or {}
memory = _state_json(state, 'memoryjson') or {}
fields = [
{'label': 'Processor', 'value': processor.get('name')},
{'label': 'Cores', 'value': processor.get('cores')},
{'label': 'Logical processors', 'value': processor.get('logicalprocessors')},
{'label': 'Max clock',
'value': '{} MHz'.format(processor['maxclockmhz'])
if processor.get('maxclockmhz') else None},
{'label': 'Total memory', 'value': _format_bytes(memory.get('totalbytes'))},
{'label': 'Free memory', 'value': _format_bytes(memory.get('freebytes'))},
]
fields = [f for f in fields if f['value'] not in (None, '')]
return success_response({'fields': fields})
# Wire values to operator-readable text. The wire keys are stable contract;
# these labels are not, so they live here rather than in the agent.
_REBOOT_REASON_LABELS = {
'componentbasedservicing': 'servicing',
'windowsupdate': 'Windows Update',
'pendingfilerename': 'files staged for replacement',
}
@endpoints_bp.route('/asset/<int:asset_id>/osinfo', methods=['GET'])
@require_permission('endpoints.view')
def panel_osinfo(asset_id: int):
"""Keyvalue panel: operating system, firmware and domain membership.
OS and BIOS are one card on purpose: both answer "what is this box
running", and split across two cards each would be three short rows.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
if not state:
return success_response({'fields': []})
osinfo = _state_json(state, 'osinfojson') or {}
bios = _state_json(state, 'biosjson') or {}
bootinfo = _state_json(state, 'bootinfojson') or {}
pendingreboot = None
if state.pendingreboot is not None:
pendingreboot = 'Yes' if state.pendingreboot else 'No'
# Say WHICH signal fired when we know. A staged file rename means the
# box is mid-servicing; a lone servicing key can sit set for weeks on a
# machine nobody rebooted. Same "Yes", very different urgency.
reasons = _state_json(state, 'rebootreasonsjson')
if state.pendingreboot and isinstance(reasons, list) and reasons:
pendingreboot = '{} ({})'.format(
pendingreboot, ', '.join(_REBOOT_REASON_LABELS.get(r, r)
for r in reasons if isinstance(r, str)))
# Only worth a row when fast startup actually applies. On a box with it
# disabled and a cold last boot there is nothing to warn about, and a
# permanent "Cold" row is noise on a card that is already dense.
boottype = bootinfo.get('lastboottype')
if boottype in (None, '', 'Cold', 'Unknown'):
boottype = None
elif boottype == 'Fast startup':
# Uptime is measured from the resume, not the last true boot, so it
# understates how long the kernel has actually been up. Said plainly
# because this is the field that makes uptime read wrong.
boottype = 'Fast startup (uptime not since cold boot)'
fields = [
{'label': 'OS version', 'value': osinfo.get('version'), 'mono': True},
{'label': 'Build', 'value': osinfo.get('build'), 'mono': True},
{'label': 'Architecture', 'value': osinfo.get('architecture')},
{'label': 'Installed', 'value': _format_agent_date(osinfo.get('installdate'))},
{'label': 'Domain', 'value': state.domain},
{'label': 'BIOS vendor', 'value': bios.get('vendor')},
{'label': 'BIOS version', 'value': bios.get('version'), 'mono': True},
{'label': 'BIOS date', 'value': _format_agent_date(bios.get('releasedate'))},
{'label': 'Last boot', 'value': boottype},
{'label': 'Pending reboot', 'value': pendingreboot},
]
fields = [f for f in fields
if f['value'] not in (None, '')]
return success_response({'fields': fields})
def _storage_rows(disks):
"""Turn the agent's volume list into display rows.
freepercent is computed by the agent from 1.7.0; for a snapshot written
by an older agent it is derived here from the byte counts so the column
is not empty for the one field an operator actually scans.
"""
if not isinstance(disks, list):
return []
rows = []
for disk in disks:
if not isinstance(disk, dict):
continue
freepercent = disk.get('freepercent')
if freepercent is None:
size = disk.get('sizebytes') or 0
free = disk.get('freebytes')
if size and free is not None:
freepercent = round(free * 100.0 / size, 1)
rows.append({
'drive': disk.get('drive'),
'label': disk.get('label') or '',
'filesystem': disk.get('filesystem'),
'size': _format_bytes(disk.get('sizebytes')),
'free': _format_bytes(disk.get('freebytes')),
'freepercent': '{}%'.format(freepercent) if freepercent is not None else None,
})
return rows
_STORAGE_COLUMNS = [
{'key': 'drive', 'label': 'Drive'},
{'key': 'label', 'label': 'Label'},
{'key': 'filesystem', 'label': 'Filesystem'},
{'key': 'size', 'label': 'Size'},
{'key': 'free', 'label': 'Free'},
{'key': 'freepercent', 'label': 'Free %'},
]
_DISKHEALTH_COLUMNS = [
{'key': 'model', 'label': 'Drive'},
{'key': 'mediatype', 'label': 'Media'},
{'key': 'bustype', 'label': 'Bus'},
{'key': 'healthstatus', 'label': 'Health'},
{'key': 'verdict', 'label': 'Verdict'},
{'key': 'source', 'label': 'Source'},
]
@endpoints_bp.route('/asset/<int:asset_id>/storage', methods=['GET'])
@require_permission('endpoints.view')
def panel_storage(asset_id: int):
"""Tabs panel: volume capacity and physical drive health as two sections.
One card because an operator looking at a full disk and an operator looking
at a dying disk are usually the same person in the same minute. TWO
SECTIONS because the two do not share a key: storage is per volume (C:, D:)
and health is per physical device, and one drive can back several volumes
or none the operator sees. Nothing is reconciled across the sections -
this is not the mapped-drive merge, where two sources described the same
drive and the worse status won.
Both sections are emitted even when empty so the section keeps its own
empty message; a dropped section would take its tab with it and read as
"no disks" rather than "nothing reported".
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
disks = _state_json(state, 'disksjson') if state else None
drives = _state_json(state, 'diskhealthjson') if state else None
return success_response({'sections': [
{
'label': 'Volumes',
'columns': _STORAGE_COLUMNS,
'rows': _storage_rows(disks),
'empty': 'No disks reported.',
},
{
'label': 'Disk health',
'columns': _DISKHEALTH_COLUMNS,
'rows': _diskhealth_rows(drives),
'empty': 'No disk health reported.',
},
]})
def _diskhealth_rows(drives):
"""Turn the agent's disk health list into display rows, worst first.
Both agent sources are kept rather than merged. MSFT_PhysicalDisk is the
only source that reports NVMe at all, and Win32_DiskDrive covers hardware
and hosts where the Storage namespace is unavailable. When both answer for
the same hardware, two rows that disagree is the useful signal; collapsing
them would hide exactly the case worth looking at.
"""
if not isinstance(drives, list):
return []
rows = []
for drive in drives:
if not isinstance(drive, dict):
continue
predictfailure = drive.get('predictfailure')
if predictfailure is None:
verdict = 'Unknown'
elif predictfailure:
verdict = 'Failure predicted'
else:
verdict = 'OK'
rows.append({
'deviceid': drive.get('deviceid'),
'model': drive.get('model') or '',
'mediatype': drive.get('mediatype') or '',
'bustype': drive.get('bustype') or '',
'healthstatus': drive.get('healthstatus') or 'Unknown',
'verdict': verdict,
'source': drive.get('source') or '',
})
# Failure first, then unknown, then OK. Unknown outranks OK because a drive
# that stopped answering is a weaker claim of health than one that said OK.
# An unrecognised verdict sorts as unknown rather than last, so a future
# verdict string cannot bury itself under the healthy drives.
order = {'Failure predicted': 0, 'Unknown': 1, 'OK': 2}
rows.sort(key=lambda row: (order.get(row['verdict'], 1),
row['deviceid'] or ''))
return rows
@endpoints_bp.route('/asset/<int:asset_id>/diskhealth', methods=['GET'])
@require_permission('endpoints.view')
def panel_diskhealth(asset_id: int):
"""Physical drive health rows.
No longer backs a panel of its own - the Storage card renders it as a
section. Kept as a route because it is a stable read for callers outside
the panel renderer and removing it would break them for a UI-only change.
Rows are ordered so anything predicting failure sorts first: a failing
drive must not be below the fold on a machine with several disks.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
drives = _state_json(state, 'diskhealthjson') if state else None
return success_response(_diskhealth_rows(drives))
@endpoints_bp.route('/asset/<int:asset_id>/listeners', methods=['GET'])
@require_permission('endpoints.view')
def panel_listeners(asset_id: int):
"""Table panel: TCP sockets in LISTEN, with the owning process.
The listening set is the box's exposed surface, so it is sorted by port
to make an unexpected one easy to spot. The full connection table is a
deliberately on-demand command, not standard telemetry.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
ports = _state_json(state, 'portsjson') if state else None
if not isinstance(ports, list):
return success_response([])
rows = []
for entry in ports:
if not isinstance(entry, dict):
continue
# Sample-data rows predate the agent listener collector and carry a
# flat port/process shape; read both rather than showing them blank.
port = entry.get('localport', entry.get('port'))
rows.append({
'localport': port,
'localaddress': entry.get('localaddress'),
'protocol': (entry.get('protocol') or 'tcp').upper(),
'processname': entry.get('processname') or entry.get('process'),
'processid': entry.get('processid'),
})
rows.sort(key=lambda r: (r['localport'] is None, r['localport']))
return success_response(rows)
def _drive_letter(value):
"""Normalise a drive to its bare letter so the two sources can be paired.
net use reports 'Z:' and Get-SmbMapping reports 'Z:' or occasionally a
trailing separator, so compare on the letter alone.
"""
text = (value or '').strip().rstrip('\\/')
return text.upper() or None
# Worst-first. The panel shows the worse of the two sources for a drive, so a
# share that one source calls OK and the other calls Unavailable is reported as
# Unavailable: a drive that is broken in either view is broken for the operator.
#
# WARNING - ranks must stay DISTINCT across statuses that can genuinely differ.
# _worse_drive_status resolves by min(), which breaks a tie by argument order,
# so two statuses sharing a rank make the verdict depend on which SOURCE
# happened to report which - the same drive reads healthy or broken based on
# enumeration order. Deliberate synonyms are fine (disconnected/unavailable,
# ok/connected) because either answer is the same answer. Adding a NEW status at
# an existing rank is not. test_which_source_reported_a_status_does_not_change
# _the_verdict in tests/test_plugins/test_endpoints_mapped_drives.py asserts
# both directions of each pair and is what catches this; the single-direction
# tests stay green through it.
_DRIVE_STATUS_RANK = {
'unavailable': 0,
'disconnected': 0,
'paused': 1,
'reconnecting': 2,
'ok': 3,
'connected': 3,
}
def _worse_drive_status(left, right):
"""Return whichever status is worse, preferring a known one over None.
An unrecognised status ranks as Reconnecting, deliberately below OK: a
status we cannot interpret must not be assumed healthy.
"""
candidates = [s for s in (left, right) if s]
if not candidates:
return None
unknown_rank = _DRIVE_STATUS_RANK['reconnecting']
return min(
candidates,
key=lambda s: _DRIVE_STATUS_RANK.get(str(s).strip().lower(), unknown_rank)
)
def _mapped_drive_rows(context):
"""Merge the two mapped-drive views into one structured row per letter.
Windows exposes mapped drives through Win32_NetworkConnection (the 'net use'
table) and through Get-SmbMapping, and they do not always agree - a share can
be Unavailable in one and OK or absent in the other. The probe collects both
without reconciling them, so the disagreement survives to here.
A drive present in only one source is still a drive: it is reported with the
status that source gave it, rather than dropped for lack of a second opinion.
WARNING - three behaviours here are load-bearing and easy to break while
"simplifying". They are covered by tests/test_plugins/test_endpoints_mapped
_drives.py; if you touch this, read those first:
1. Worse-status-wins. Never report the healthier of the two sources. A
drive that either source calls Unavailable IS unavailable to the
operator, and rendering it OK sends someone to a share that is down.
2. Single-source drives are kept, not dropped. Get-SmbMapping is absent on
some builds and session types; requiring agreement would silently hide
every mapped drive on those machines.
3. Unknown statuses rank as Reconnecting, not as OK. An unrecognised
string must never be optimistically treated as healthy.
Changing any of these makes a broken drive look fine, which is a silent
failure - the card stays green and nobody is told.
"""
drives = {}
netuse = context.get('netuse')
if isinstance(netuse, list):
for entry in netuse:
if not isinstance(entry, dict):
continue
letter = _drive_letter(entry.get('localname') or entry.get('local'))
if not letter:
continue
drives[letter] = {
'remote': entry.get('remotename') or entry.get('remote') or '',
'status': entry.get('status'),
}
smbmapping = context.get('smbmapping')
if isinstance(smbmapping, list):
for entry in smbmapping:
if not isinstance(entry, dict):
continue
letter = _drive_letter(entry.get('localpath'))
if not letter:
continue
existing = drives.get(letter)
if existing:
existing['status'] = _worse_drive_status(
existing.get('status'), entry.get('status'))
existing['remote'] = existing['remote'] or entry.get('remotepath') or ''
else:
drives[letter] = {
'remote': entry.get('remotepath') or '',
'status': entry.get('status'),
}
rows = []
for letter in sorted(drives):
drive = drives[letter]
# A drive neither source gave a status for is Unknown, never blank. An
# unannotated row reads as healthy, so a failed check would render as a
# working drive - the one outcome this card must never produce.
status = (drive.get('status') or '').strip() or 'Unknown'
healthy = status.lower() in ('ok', 'connected')
rows.append({
'drive': letter,
'remote': (drive.get('remote') or '').strip(),
'statuslabel': status,
# Only unhealthy drives are coloured. Green on every working drive
# makes the broken one just another badge in a row of badges; the
# card's job is to make the exception findable at a glance.
'statecolor': 'green' if healthy else 'red',
})
return rows
def _mapped_drives(context):
"""The mapped drives as one-line labels, for the user context card.
Built on _mapped_drive_rows so the summary line and the card can never
disagree about a drive's status.
"""
labels = []
for row in _mapped_drive_rows(context):
label = '{} {}'.format(row['drive'], row['remote']).strip()
# Only the problem states are annotated. Tagging every healthy drive
# with "(OK)" would bury the one that is not.
if row['statecolor'] != 'green':
label = '{} ({})'.format(label, row['statuslabel'])
labels.append(label)
return labels
def _mapped_drives_unhealthy_first(rows):
"""Order drive rows so anything not healthy sorts first.
A user with a dozen mappings should not have to hunt for the broken one.
Sorting only on health keeps it stable: drive-letter order survives within
each group, so the list does not reshuffle between refreshes.
"""
return sorted(rows, key=lambda row: row['statecolor'] == 'green')
@endpoints_bp.route('/asset/<int:asset_id>/mappeddrives', methods=['GET'])
@require_permission('endpoints.view')
def panel_mappeddrives(asset_id: int):
"""List panel: mapped network drives with a health badge each.
Session-scoped: mapped drives do not exist in the agent's session 0, so
these come from the interactive user probe. Nothing reported when nobody
is signed in, which is a valid state and not a failure.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
context = _state_json(state, 'usercontextjson') if state else None
if not isinstance(context, dict) or not context:
return success_response([])
return success_response(_mapped_drives_unhealthy_first(_mapped_drive_rows(context)))
@endpoints_bp.route('/asset/<int:asset_id>/usercontext', methods=['GET'])
@require_permission('endpoints.view')
def panel_usercontext(asset_id: int):
"""Keyvalue panel: session-scoped data from the interactive user probe.
Mapped drives and user-owned listeners do not exist in the agent's
session 0, so they are collected by a probe run in the logged-in user's
session. Nothing is reported when no one is signed in, which is a valid
state and not a failure.
Mapped drives arrive from two sources that can disagree; the worse of the
two statuses is shown so a half-broken share is never rendered as healthy.
"""
state = EndpointState.query.filter_by(assetid=asset_id).first()
if not state:
return success_response({'fields': []})
context = _state_json(state, 'usercontextjson')
if not isinstance(context, dict) or not context:
return success_response({'fields': []})
drives = _mapped_drives(context)
listeners = context.get('userlisteners')
listenercount = len(listeners) if isinstance(listeners, list) else None
fields = [
{'label': 'Probed user', 'value': context.get('username')},
{'label': 'Collected', 'value': context.get('collectedat')},
{'label': 'Idle', 'value': _format_uptime(context.get('idleseconds'))},
{'label': 'Mapped drives',
'value': ', '.join(drives) if drives else None, 'mono': True},
{'label': 'User listeners', 'value': listenercount},
]
fields = [f for f in fields if f['value'] not in (None, '')]
return success_response({'fields': fields})
@endpoints_bp.route('/asset/<int:asset_id>/telemetrymeta', methods=['GET'])
@require_permission('endpoints.view')
def panel_telemetry_meta(asset_id: int):
@@ -723,10 +1348,18 @@ def panel_activity(asset_id: int):
rows = []
for event in events:
# The provider name is what distinguishes two events sharing an ID,
# and the message body is the line an operator actually reads. Both
# are already stored; the feed just was not showing them.
source = event.eventsource or event.eventchannel
detail = 'Windows event {}'.format(event.eventid)
if source:
detail = '{} - {}'.format(detail, source)
rows.append({
'title': event_title(event.eventid),
'severity': event_severity(event.eventid),
'detail': 'Windows event {}'.format(event.eventid),
'detail': detail,
'body': _event_body(event),
'eventtime': event.eventtime,
})
@@ -742,7 +1375,9 @@ def panel_activity(asset_id: int):
'title': '{}{}'.format(label, target),
# A failed command is the one an operator needs to notice.
'severity': 'critical' if command.status == EndpointCommand.STATUS_FAILED else 'info',
'detail': '{}{}'.format((command.status or 'queued').title(), who),
'detail': '{}{}{}'.format('Command - ', (command.status or 'queued').title(), who),
# A failed command's reason is the whole point of noticing it.
'body': (command.resultmessage or '').strip() or None,
'eventtime': command.createdat,
})
@@ -768,10 +1403,37 @@ def panel_activity(asset_id: int):
'severity': r['severity'],
'severitycolor': colours.get(r['severity'], 'gray'),
'detail': r['detail'],
'body': r['body'],
'eventtime': r['eventtime'].isoformat() + 'Z' if r['eventtime'] else None,
} for r in rows[:15]])
def _link_label(speedbps, fullduplex):
"""Render NIC link speed and duplex as one cell.
Null speed means the agent could not read link state OR the adapter is
disconnected - verified on a live box that a disconnected NIC returns empty
values. Either way we do not know, so it reads '-' rather than 0 Mb.
Half duplex is called out explicitly because that is the actionable case: a
NIC that negotiated half duplex on a switch port is a cable or port fault
that otherwise presents as "the machine is slow sometimes". Full duplex is
the normal case and gets no annotation, so the only thing standing out in
the column is the thing worth looking at.
"""
if not speedbps:
return '-'
if speedbps >= 1000000000:
speed = '{:g} Gb'.format(speedbps / 1000000000)
else:
speed = '{:g} Mb'.format(speedbps / 1000000)
if fullduplex is False:
return '{} half duplex'.format(speed)
return speed
@endpoints_bp.route('/asset/<int:asset_id>/interfaces', methods=['GET'])
@require_permission('endpoints.view')
def panel_interfaces(asset_id: int):
@@ -809,6 +1471,99 @@ def panel_interfaces(asset_id: int):
# A gatewayless NIC with an address is the machine link; the gateway
# NIC is the corporate LAN.
'role': 'Machine link' if (ipv4 and not gateway) else 'LAN',
'link': _link_label(entry.get('linkspeedbps'),
entry.get('fullduplex')),
})
return success_response(rows)
# Columns for the two cards carved out of the old hand-written Network card on
# the Details tab. That card listed current addresses with a Primary badge and
# nothing else, which undersold what the rows actually hold.
_IPHISTORY_COLUMNS = [
{'key': 'ipaddress', 'label': 'IP address'},
{'key': 'macaddress', 'label': 'MAC'},
{'key': 'status', 'label': 'Status'},
{'key': 'firstseen', 'label': 'First seen'},
{'key': 'lastchanged', 'label': 'Last changed'},
]
def _isostamp(value):
"""ISO-8601 with a Z suffix, matching every other timestamp this file emits.
Formatting is left to the client so it can render in the viewer's locale
and timezone; returning a preformatted string here would pin every reader
to the server's.
"""
return value.isoformat() + 'Z' if value else None
def _communication_rows(asset_id: int):
"""Every Communication row for an asset, newest change first.
These rows are an ACCIDENTAL history rather than a designed one.
_record_interfaces in the computers plugin matches an incoming NIC on MAC
then on IP and inserts when neither matches, and it never deletes - so an
address the machine has not held for a year is still sitting here next to
the current one. That is genuinely useful (it answers "what was this box on
before?"), so it is surfaced as history rather than cleaned away.
"""
from shopdb.api import Communication
# MySQL has no NULLS LAST, so sort on an explicit "is null" key first.
# Rows that never got a modifieddate sink to the bottom on every dialect.
return (Communication.query
.filter_by(assetid=asset_id)
.order_by(Communication.modifieddate.is_(None),
Communication.modifieddate.desc(),
Communication.communicationid.desc())
.all())
def _iphistory_status(comm):
"""What this row IS, in the vocabulary somebody troubleshooting would use.
Primary is the address the collector last picked as this box's main one.
APIPA is called out by name because a 169.254.x.x address is not an address
the machine was given - it is what Windows assigns when DHCP failed, so its
presence in the list is a record of a past network fault, not a past
subnet.
"""
address = (comm.ipaddress or '').strip()
if address.startswith('169.254.'):
return 'APIPA (no DHCP)'
if comm.isprimary:
return 'Primary'
if comm.ismachinenetwork:
return 'Machine network'
return 'Previous'
@endpoints_bp.route('/asset/<int:asset_id>/iphistory', methods=['GET'])
@require_permission('endpoints.view')
def panel_iphistory(asset_id: int):
"""Table panel: every address this asset has held, newest change first.
Replaces the hand-written Network card that used to sit on the Details tab
and show only the current addresses. Dates come from the row's own
lifecycle columns, so they are labelled for what they are: 'Last changed'
is when a field on the row last moved, NOT when the address was last seen.
An address the machine still holds and reports unchanged does not update
modifieddate, so calling that column 'Last seen' would be a lie the data
cannot support.
"""
rows = []
for comm in _communication_rows(asset_id):
address = (comm.ipaddress or '').strip()
if not address:
continue
rows.append({
'ipaddress': address,
'macaddress': (comm.macaddress or '').strip() or '-',
'status': _iphistory_status(comm),
'firstseen': _isostamp(comm.createddate),
'lastchanged': _isostamp(comm.modifieddate),
})
return success_response(rows)

237
plugins/endpoints/api/siteagent.py
View File

@@ -136,6 +136,12 @@ _BASE_APPSETTINGS = {
'UDC.exe': 'C:\\Program Files\\UDC\\UDC.exe',
'agent.exe': 'C:\\ProgramData\\udc\\mtconnect_udc\\agent\\run.bat',
},
# Approved scheduled task paths for scheduledtask_run. Shipped empty, which
# means the agent falls back to ScheduledTaskPolicy.DefaultApprovedTaskPaths
# (the Corporate Tools set). Listing entries here REPLACES those defaults, so
# a site can narrow or widen the set without an agent rebuild. Anything not
# matched fails closed.
'approvedtaskpaths': [],
}
_README = """ShopDB endpoint agent (site build)
@@ -217,7 +223,19 @@ param(
[string] $ServiceName = "ShopDbEndpointAgent",
[string] $TrayExeName = "shopdb-endpoint-tray.exe",
[string] $CertFileName = {certfile},
[switch] $NoTray
[switch] $NoTray,
# Scheduled task that runs the user-context probe INSIDE the operator's own
# logon session. Mapped drives are per logon token, so this is the only
# reliable way to collect them; see the registration block below.
[string] $UserContextTaskName = "ShopDB user-context probe",
[int] $UserContextTaskMinutes = 30,
# The account the probe task runs as. Must match the account whose profile
# the agent reads the spool from (FileLoggerProvider.ShopFloorUserName), or
# the probe writes session data the service never looks at.
[string] $UserContextTaskUser = "ShopFloor",
[switch] $NoUserContextTask
)
$ErrorActionPreference = "Stop"
@@ -362,10 +380,175 @@ if (-not $NoTray) {{
}}
catch {{ Write-Warning "could not create Start Menu shortcut: $($_.Exception.Message)" }}
# Launch the tray once now so it appears without waiting for the next
# logon. Single-instance guard makes a duplicate launch a no-op.
try {{ Start-Process -FilePath $installedTray -WorkingDirectory $InstallDir }}
catch {{ Write-Warning "could not start tray: $($_.Exception.Message)" }}
# Deliberately NOT launched here. This installer runs elevated, so
# Start-Process would hand the tray an ADMIN token, and drive letters
# are per logon token: the probe would then enumerate the installing
# admin's drive table, not the operator's. Measured on a live endpoint,
# the same two shares reported Unavailable from the elevated shell and
# OK from the operator's own session, in both Win32_NetworkConnection
# and Get-SmbMapping. Launching via explorer.exe does not fix this
# either - it de-elevates but does NOT change account.
#
# The Run key above is the correct autostart: it fires at a genuine
# logon, in the operator's own session. Collection no longer depends on
# the tray at all - the scheduled task below owns it.
Write-Host " tray will start at next logon (not launched from this elevated install)."
}}
}}
# 2b. Register the user-context probe as a scheduled task owned by the SHOPFLOOR
# USER. This is what makes mapped drives, user listeners and idle time
# collectable: it is the only mechanism that starts a process in the operator's
# own session with the operator's own token.
#
# RunLevel Limited is load-bearing, not a hardening nicety - the unelevated
# token is precisely what sees the operator's mapped drives.
#
# UserId, NOT GroupId. A group-principal task is registered against the group
# rather than against any one account, and a member of that group cannot invoke
# it on demand: Start-ScheduledTask from the ShopFloor session returns access
# denied, so the task only ever ran on its own trigger. That is exactly the bug
# where "Refresh Status" appeared to do nothing. Registering to the specific
# account makes the task startable by that account, which is what both the tray
# and any manual retry depend on. The group form is kept only as a fallback for
# a host where the account cannot be resolved - it still collects on a timer,
# it just cannot be kicked on demand.
$taskState = "skipped"
if (-not $NoUserContextTask) {{
Write-Host " registering user-context probe task ..."
$taskAgentExe = Join-Path $InstallDir "shopdbagent.exe"
$taskRegistered = $false
$taskDetail = "no principal accepted"
try {{
$taskAction = New-ScheduledTaskAction -Execute $taskAgentExe -Argument "--collect-usercontext"
# AtLogOn covers the normal case. The repeating trigger matters just as
# much: AtLogOn does NOT fire for a session that is already signed in,
# which is every install onto a machine someone is currently using.
$taskTriggers = @(New-ScheduledTaskTrigger -AtLogOn)
if ($UserContextTaskMinutes -gt 0) {{
$taskTriggers += New-ScheduledTaskTrigger -Once -At (Get-Date) -RepetitionInterval (New-TimeSpan -Minutes $UserContextTaskMinutes)
}}
$taskSettings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Minutes 5)
# Try the machine-qualified account first, then the bare name. Interactive
# LogonType so the task joins the existing interactive session instead of
# creating a separate one with its own (empty) drive table.
$taskCandidates = @(
@{{ Kind = "user"; Id = "$env:COMPUTERNAME\\$UserContextTaskUser" }},
@{{ Kind = "user"; Id = $UserContextTaskUser }},
@{{ Kind = "group"; Id = "S-1-5-32-545" }},
@{{ Kind = "group"; Id = "BUILTIN\\Users" }}
)
foreach ($candidate in $taskCandidates) {{
try {{
if ($candidate.Kind -eq "user") {{
$taskPrincipal = New-ScheduledTaskPrincipal -UserId $candidate.Id -LogonType Interactive -RunLevel Limited -ErrorAction Stop
}}
else {{
$taskPrincipal = New-ScheduledTaskPrincipal -GroupId $candidate.Id -RunLevel Limited -ErrorAction Stop
}}
Register-ScheduledTask -TaskName $UserContextTaskName -Action $taskAction -Trigger $taskTriggers -Principal $taskPrincipal -Settings $taskSettings -Force -ErrorAction Stop | Out-Null
$taskRegistered = $true
$taskDetail = "$($candidate.Kind):$($candidate.Id)"
break
}}
catch {{ $taskDetail = $_.Exception.Message }}
}}
}}
catch {{ $taskDetail = "could not build the task: $($_.Exception.Message)" }}
if ($taskRegistered) {{
# Read the task back rather than trusting Register-ScheduledTask to have
# returned cleanly: policy or a management agent can rewrite or drop a
# task after registration, and a task that exists but runs elevated
# looks healthy in the scheduler UI while producing wrong data.
$verified = Get-ScheduledTask -TaskName $UserContextTaskName -ErrorAction SilentlyContinue
if (-not $verified) {{
$taskState = "UNVERIFIED:task not found after registration"
Write-Warning " user-context task registered but cannot be read back."
}}
elseif ([string] $verified.Principal.RunLevel -eq "Highest") {{
$taskState = "UNVERIFIED:elevated"
Write-Warning " user-context task runs ELEVATED; mapped drives would be collected from the wrong token."
}}
elseif ($verified.Principal.GroupId) {{
# Registered, will collect on its triggers, but cannot be started on
# demand by the operator - so "Refresh Status" will fall back to the
# timer. Surfaced rather than reported as ok, because the symptom
# (stale session data that never responds to a refresh) is otherwise
# indistinguishable from a broken probe.
$taskState = "DEGRADED:group principal $($verified.Principal.GroupId)"
Write-Warning " user-context task fell back to a GROUP principal; on-demand refresh will not work, only the $UserContextTaskMinutes-minute interval."
}}
else {{
$taskState = "ok:$($verified.Principal.UserId)/$($verified.Principal.RunLevel)"
Write-Host " user-context task registered and verified: $taskState"
}}
# AtLogOn will not fire for an already-signed-in session, so kick it once
# rather than leaving a fresh install blind until the repeat interval.
try {{
Start-ScheduledTask -TaskName $UserContextTaskName -ErrorAction Stop
Write-Host " user-context task started once for the current session."
}}
catch {{ Write-Warning " could not start the task now: $($_.Exception.Message); it runs at next logon or interval." }}
}}
else {{
# Not fatal: the service still checks in, only the session-scoped fields
# are missing, and the ledger records why.
$taskState = "FAILED:$taskDetail"
Write-Warning " could not register the user-context task: $taskDetail"
Write-Warning " mapped drives and idle time will not be collected."
}}
}}
# 2c. Start the tray in the operator's session.
#
# The Run key registered above only fires at LOGON, so installing onto a machine
# someone is already signed in to leaves no tray until they sign out and back in.
# The installer cannot start it directly - it is elevated, and the tray must run
# unelevated in the user's own session - so we borrow the same mechanism that
# solves this for the probe: a transient scheduled task with a Limited group
# principal, run once and then deleted.
#
# Registered without a trigger; it exists only to be invoked immediately. The
# tray is single-instance, so if one is somehow already running this is a no-op.
if (-not $NoTray -and (Test-Path (Join-Path $InstallDir $TrayExeName))) {{
$launchTaskName = "ShopDB tray first launch"
try {{
$launchExe = Join-Path $InstallDir $TrayExeName
$launchAction = New-ScheduledTaskAction -Execute $launchExe -WorkingDirectory $InstallDir
$launchSettings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries
$launched = $false
foreach ($groupId in @("S-1-5-32-545", "BUILTIN\\Users")) {{
try {{
$launchPrincipal = New-ScheduledTaskPrincipal -GroupId $groupId -RunLevel Limited -ErrorAction Stop
Register-ScheduledTask -TaskName $launchTaskName -Action $launchAction -Principal $launchPrincipal -Settings $launchSettings -Force -ErrorAction Stop | Out-Null
Start-ScheduledTask -TaskName $launchTaskName -ErrorAction Stop
$launched = $true
break
}}
catch {{ }}
}}
# Give the task a moment to spawn the tray before removing it: deleting a
# task that is still starting can kill the process it just launched.
if ($launched) {{
Start-Sleep -Seconds 3
Write-Host " tray started in the operator's session."
}}
else {{
Write-Warning " could not start the tray now; it will start at next logon via the Run key."
}}
# Transient by design: this is a one-shot launcher, not an autostart.
# The Run key owns autostart, and leaving this behind would mean two
# things racing to start the same single-instance tray at every logon.
try {{ Unregister-ScheduledTask -TaskName $launchTaskName -Confirm:$false -ErrorAction Stop }}
catch {{ Write-Warning " could not remove the transient launch task '$launchTaskName'; remove it manually." }}
}}
catch {{
Write-Warning " could not start the tray now: $($_.Exception.Message); it will start at next logon."
}}
}}
@@ -397,15 +580,22 @@ Write-Host " service is running." -ForegroundColor Green
$trayBase2 = [System.IO.Path]::GetFileNameWithoutExtension($TrayExeName)
if (-not $NoTray -and (Test-Path (Join-Path $InstallDir $TrayExeName))) {{
$trayOk = $false
for ($attempt = 1; $attempt -le 5; $attempt++) {{
if (Get-Process -Name $trayBase2 -ErrorAction SilentlyContinue) {{ $trayOk = $true; break }}
Write-Host " tray not running yet (attempt $attempt/5); relaunching ..."
try {{ Start-Process -FilePath (Join-Path $InstallDir $TrayExeName) -WorkingDirectory $InstallDir }} catch {{ }}
# Report only; never relaunch from here. This block used to Start-Process the
# tray when it was not found running, which would hand it this installer's
# ADMIN token and reintroduce the wrong-drive-table bug. The launch task
# above is the only place the tray is started during install, and it starts
# it correctly - unelevated, in the operator's session.
$trayFound = $false
for ($attempt = 1; $attempt -le 3; $attempt++) {{
if (Get-Process -Name $trayBase2 -ErrorAction SilentlyContinue) {{ $trayFound = $true; break }}
Start-Sleep -Seconds 2
}}
if ($trayOk) {{ Write-Host " tray is running." -ForegroundColor Green }}
else {{ Write-Warning "tray did not confirm running; it will retry at next logon via the Run key." }}
if ($trayFound) {{
Write-Host " tray is running." -ForegroundColor Green
}}
else {{
Write-Warning "tray did not confirm running; it will start at next logon via the Run key."
}}
}}
# 5. Preserve the install artifacts (minus the secret-bearing install.ps1) under
@@ -432,6 +622,29 @@ if (Test-Path $installSelf) {{
}}
Write-Host "[6/6] Done." -ForegroundColor Green
# Append one line per install, including reinstalls and repeat installs of the
# same version. The user-context task state is recorded because it is the single
# point of failure for session-scoped collection: if mapped drives go missing
# later, this answers "was the task ever registered on this host, and as what"
# without needing the endpoint back. The ledger is a record, not a gate - a
# failure to append must never fail an otherwise-good install.
try {{
$ledgerDir = Join-Path $env:ProgramData "ShopDB\\Logs"
New-Item -ItemType Directory -Force -Path $ledgerDir | Out-Null
$ledgerPath = Join-Path $ledgerDir "shopdbagent-installs.log"
$ledgerAgentVersion = "unknown"
if (Test-Path $installedAgent) {{ $ledgerAgentVersion = (Get-Item $installedAgent).VersionInfo.FileVersion }}
$ledgerTrayVersion = "not-installed"
$ledgerTrayPath = Join-Path $InstallDir $TrayExeName
if (Test-Path $ledgerTrayPath) {{ $ledgerTrayVersion = (Get-Item $ledgerTrayPath).VersionInfo.FileVersion }}
$ledgerLine = "{{0}} host={{1}} installer={{2}} service={{3}} tray={{4}} dir={{5}} usercontexttask={{6}}" -f (Get-Date -Format "yyyy-MM-dd HH:mm:ss zzz"), $env:COMPUTERNAME, $env:USERNAME, $ledgerAgentVersion, $ledgerTrayVersion, $InstallDir, $taskState
Add-Content -Path $ledgerPath -Value $ledgerLine -Encoding UTF8
Write-Host "Install ledger: $ledgerPath"
Write-Host " $ledgerLine" -ForegroundColor DarkGray
}}
catch {{ Write-Warning "could not append to the install ledger: $($_.Exception.Message)" }}
if (Test-Path $uninstaller) {{
Write-Host "Uninstaller: $uninstaller" -ForegroundColor Cyan
Write-Host " To remove the agent, run that script from an elevated PowerShell."

117
plugins/endpoints/commands.py
View File

@@ -77,12 +77,36 @@ COMMAND_ALLOWLIST = {
'label': 'Collect now',
'description': 'Force the endpoint agent to report fresh telemetry off-cadence.',
},
'network_connections': {
'label': 'Show network connections',
'description': 'Return the full TCP connection table with owning processes. '
'Read-only diagnostic; the standard report carries only '
'listening ports because established connections churn.',
},
'scheduledtask_run': {
'label': 'Run scheduled task',
'description': 'Start an already-registered Windows scheduled task on '
'the endpoint by its task path (for example '
'\\ReportIP). The agent does not create, modify, or '
'define tasks - it only starts one that already exists, '
'and only when the path clears the agent-side approved '
'task allowlist. The task runs in whatever context its '
'own registered definition specifies, not the agent\'s.',
'targetkind': 'scheduledtask',
},
'run_script': {
'label': 'Run script',
'description': 'Run a ShopDB-hosted script on the endpoint. The agent '
'fetches the script body by digest using its collector '
'credential and runs it in the script\'s configured '
'context (system or the logged-on user).',
'context (system or the logged-on user). CAUTION: a '
'script sent this way runs as SYSTEM unless its '
'descriptor pins the user context, so this is arbitrary '
'code execution on the endpoint by design. Grant '
'endpoints.deploy only to the most trusted operators, '
'and gate the UI behind an explicit confirm the way '
'system_restart is gated (see confirmRestart in '
'PCDetail.vue).',
'targetkind': 'script',
},
'update_app': {
@@ -96,11 +120,76 @@ COMMAND_ALLOWLIST = {
}
# Dell Command | Update: TWO utilities, deliberately two separate actions.
# A scan reports what is available and changes nothing; an update installs and
# can reboot. Collapsing them into one parameterized action would make the
# destructive one reachable by editing a field, so they stay distinct and the
# UI can gate them differently.
#
# Both are driven through scheduledtask_run against ShopDB-owned task
# definitions (\ShopDB-DellCommandScan and \ShopDB-DellCommandUpdate, already in
# ScheduledTaskPolicy's shipped defaults) rather than a bespoke verb. The task
# definition - not the agent - pins the dcu-cli.exe path, its arguments, and its
# output path, which is why no new agent code is needed to run them.
#
# OUTPUT PATH CONVENTION. A ShopDB-initiated scan MUST write its report under
# %ProgramData%\ShopDB\EndpointAgent\ (the same SYSTEM-owned root as the agent
# logs), NOT to Dell's default location. Corporate-managed DCU tasks write to
# the default; keeping ShopDB's output separate is the only thing that makes
# "did ShopDB ask for this scan, or did corporate" answerable when the results
# are collected. The path lives in the task definition and is generated
# server-side, so it can change without touching the agent.
#
# STILL TO DO - the results collector. Firing DCU is the easy half. schtasks
# /run returns as soon as the task STARTS, so a successful scheduledtask_run
# means "scan started", never "scan finished" - the command result cannot carry
# findings. Reporting available updates back into ShopDB needs a collector that
# reads the report from the path above and folds it into the telemetry payload.
# That is a separate change from this one; do not infer from a green command
# result that the data path exists.
# Reserved seam - one-click mapped-drive remediation. NOT implemented; this is a
# placement note so the next person adds it here rather than inventing a new
# transport. When it lands it will re-establish a Disconnected mapping surfaced
# by the user-context card:
#
# 'remap_drive': {
# 'label': 'Reconnect mapped drive',
# 'description': 'Re-establish a disconnected network drive mapping.',
# 'targetkind': 'drive',
# },
#
# It MUST execute in the interactive USER session. A SYSTEM-context remap creates
# a mapping in SYSTEM's own drive table that the operator cannot see, so the card
# would keep reporting the drive as broken while the "fix" silently succeeded
# somewhere nobody is looking. An ELEVATED remap fails the same way, because UAC
# token separation gives the elevated context a different drive table - the same
# reason the probe itself runs unelevated from the tray.
#
# CommandExecutor.RunScriptInUserSession is the existing path for this.
# Actions that demand a permission stricter than the base endpoints.command.
# The enqueue route requires the caller to ALSO hold the mapped permission, so a
# destructive action (a remote reboot) is a heavier grant than a routine toggle.
# NOTE FOR THE ROLES/PERMISSIONS WORK: scheduledtask_run is deliberately mapped
# onto the EXISTING endpoints.deploy grant rather than introducing a new one, so
# the wire is pre-built and needs no migration. Be aware this WIDENS what
# endpoints.deploy means: it previously covered run_script + update_app, and now
# also covers starting approved scheduled tasks. If that bundle is too coarse,
# the clean split is a separate endpoints.corptools permission mapped here - no
# other code needs to change, because the enqueue route reads this table.
#
# Suggested operator-facing wording for the combined grant:
# "Caution: Operator can run scheduled tasks, update applications, and affect
# system-level changes to the endpoint."
# That wording is accurate for scheduledtask_run and update_app. It UNDERSTATES
# run_script, which is arbitrary SYSTEM code execution - if run_script stays in
# this bundle, say so plainly in the role description.
ELEVATED_ACTION_PERMISSIONS = {
'system_restart': 'endpoints.restart',
'scheduledtask_run': 'endpoints.deploy',
'run_script': 'endpoints.deploy',
'update_app': 'endpoints.deploy',
}
@@ -215,6 +304,24 @@ def monitor_id_from_action(action):
return None
# Actions gated by ROLE ONLY, never by per-endpoint Configuration Policy.
#
# These are the Corporate Tools actions. They start already-registered,
# agent-allowlisted tasks and are not a per-machine configuration concern, so a
# policy that never mentions them must not remove them. Policy exists to shape
# what a given PC TYPE exposes (which monitors, which service primitives); it is
# not the authority on whether an admin may run a corporate tool.
#
# Without this exemption the intersection at the end of ALLOWED_ACTIONS_FOR
# silently drops these actions on any endpoint whose effective policy is
# non-empty, producing "Action not permitted by policy for this endpoint" even
# for a caller holding endpoints.deploy. The real gate for these stays
# ELEVATED_ACTION_PERMISSIONS (role), enforced in the enqueue route.
POLICY_EXEMPT_ACTIONS = frozenset({
'scheduledtask_run',
})
def ALLOWED_ACTIONS_FOR(pctype=None, assetid=None):
"""The action names an endpoint may run.
@@ -223,8 +330,9 @@ def ALLOWED_ACTIONS_FOR(pctype=None, assetid=None):
result is narrowed to the actions the endpoint's effective policy permits -
this is the allowlist-gating half of the policy design. With the feature off
(or no assetid), it returns the full base allowlist, so nothing changes until
a site opts in. Policy resolution is imported lazily to keep this module free
of a hard dependency on the computers plugin.
a site opts in. Actions in POLICY_EXEMPT_ACTIONS are never narrowed away;
they are gated by role alone. Policy resolution is imported lazily to keep
this module free of a hard dependency on the computers plugin.
"""
base = set(COMMAND_ALLOWLIST.keys())
if assetid is None:
@@ -247,6 +355,9 @@ def ALLOWED_ACTIONS_FOR(pctype=None, assetid=None):
# "not permitted by policy". Expand each monitor:<id> to its kind's control
# actions before intersecting.
allowed = allowed | _monitor_control_actions(allowed)
# Role-gated actions are added back unconditionally: policy narrows the
# per-endpoint surface, but it does not own the corporate tools.
allowed = allowed | POLICY_EXEMPT_ACTIONS
# Never widen past the code-level allowlist: policy can only gate within it.
return base & allowed

7
plugins/endpoints/eventcatalog.py
View File

@@ -24,7 +24,14 @@ EVENT_CATALOG = {
55: ('File system corruption', 'critical', 'System'),
7011: ('Service timeout', 'critical', 'System'),
10016: ('DCOM permission error', 'warning', 'System'),
# Windows Error Reporting trio on the Application channel. 1000 is the
# faulting-application record (app name, version, faulting module, exception
# code) and is the one that says what actually broke; 1001 is the WER bucket
# record Windows writes alongside it; 1002 is a hang rather than a crash, so
# it is a warning - the app is unresponsive, not gone.
1000: ('Application error', 'critical', 'Application'),
1001: ('Application crash', 'critical', 'Application'),
1002: ('Application hang', 'warning', 'Application'),
}
# The IDs a tech should see surfaced fleet-wide on the dashboard stability card.

40
plugins/endpoints/frontend/views/AgentKeySettings.vue
View File

@@ -319,9 +319,9 @@
<div
v-for="binary in binaries"
:key="binary.key"
class="binary-block"
class="card binary-block"
>
<h4>{{ binary.label }}</h4>
<h4 class="binary-title">{{ binary.label }}</h4>
<p class="field-hint">
Upload {{ binary.name }} updated EXE. Uploading here will update the
binaries available for your "Build Site Agent" install package.
@@ -355,7 +355,7 @@
<button
class="btn btn-secondary"
type="button"
:disabled="binary.busy || !binary.version || !binary.file"
:disabled="binary.busy || binary.detecting || !binary.file"
@click="uploadBinary(binary.key)"
>
{{ binary.busy ? 'Uploading...' : 'Upload current release' }}
@@ -370,8 +370,9 @@
Upload legacy build for rollback
</label>
<p class="field-warning">
WARNING: confirm agent version before uploading from the EXE
properties and enter the version number.
The version is read from the EXE itself. Leave this blank to use the
binary's own file version, or enter a dotted-numeric override only
when the build carries no readable version.
</p>
<div class="binary-upload">
<input
@@ -389,7 +390,7 @@
<button
class="btn btn-secondary"
type="button"
:disabled="binary.busy || !binary.legacyversion || !binary.legacyfile"
:disabled="binary.busy || !binary.legacyfile"
@click="uploadLegacy(binary.key)"
>
{{ binary.busy ? 'Uploading...' : 'Upload legacy build' }}
@@ -756,7 +757,8 @@ async function detectVersion(binkey, file) {
const form = new FormData()
form.append('file', file)
const response = await api.post(
`/endpoints/siteagent/binaries/${binkey}/detect`, form)
`/endpoints/siteagent/binaries/${binkey}/detect`, form,
{ headers: { 'Content-Type': 'multipart/form-data' } })
return response.data.data?.version || ''
}
@@ -789,7 +791,7 @@ function onLegacyFileChange(binkey, event) {
async function uploadBinary(binkey) {
const binary = binaryFor(binkey)
if (!binary || !binary.file || !binary.version) return
if (!binary || !binary.file) return
binary.busy = true
binary.message = ''
binary.error = ''
@@ -798,7 +800,8 @@ async function uploadBinary(binkey) {
form.append('version', binary.version)
form.append('file', binary.file)
const response = await api.post(
`/endpoints/siteagent/binaries/${binkey}`, form)
`/endpoints/siteagent/binaries/${binkey}`, form,
{ headers: { 'Content-Type': 'multipart/form-data' } })
const data = response.data.data || {}
applyBinaryStatus(binkey, data.current, data.versions)
binary.version = ''
@@ -817,7 +820,7 @@ async function uploadBinary(binkey) {
async function uploadLegacy(binkey) {
const binary = binaryFor(binkey)
if (!binary || !binary.legacyfile || !binary.legacyversion) return
if (!binary || !binary.legacyfile) return
binary.busy = true
binary.message = ''
binary.error = ''
@@ -827,7 +830,8 @@ async function uploadLegacy(binkey) {
form.append('file', binary.legacyfile)
form.append('makecurrent', 'false')
const response = await api.post(
`/endpoints/siteagent/binaries/${binkey}`, form)
`/endpoints/siteagent/binaries/${binkey}`, form,
{ headers: { 'Content-Type': 'multipart/form-data' } })
const data = response.data.data || {}
applyBinaryStatus(binkey, data.current, data.versions)
binary.legacyversion = ''
@@ -906,10 +910,18 @@ function editConfiguration() {
.settings-section > h3 { margin-bottom: 0.25rem; }
.info-list { margin: 0.25rem 0 0; padding-left: 1.25rem; }
.info-list li { margin-bottom: 0.5rem; }
/* Each agent is its own nested card inside the collapsible, so the two
binaries read as distinct units instead of one continuous wall of
controls separated by a hairline rule. */
.binary-block {
margin-top: 1.25rem;
padding-top: 1rem;
border-top: 1px solid var(--color-border, #333);
margin-top: 1rem;
padding: 1rem;
}
.binary-block:first-of-type { margin-top: 0.75rem; }
.binary-title {
margin: 0 0 0.5rem;
padding-bottom: 0.5rem;
border-bottom: 1px solid var(--color-border, #333);
}
.binary-upload {
display: flex;

4
plugins/endpoints/frontend/views/EndpointsList.vue
View File

@@ -91,6 +91,7 @@
<th>Machine</th>
<th>PC Type</th>
<th>Health</th>
<th>Agent Version</th>
<th>Logged-in User</th>
<th>Uptime</th>
<th>Last Reported</th>
@@ -123,12 +124,13 @@
class="health-pill" :class="item.health.level">{{ item.health.label }}</span>
<span v-else>-</span>
</td>
<td>{{ item.agentversion || '-' }}</td>
<td>{{ item.loggedinuser || '-' }}</td>
<td>{{ item.uptimedisplay || '-' }}</td>
<td>{{ formatRelative(item.lastreportedat) }}</td>
</tr>
<tr v-if="endpoints.length === 0">
<td :colspan="cancommand ? 9 : 8" style="text-align: center; color: var(--text-light);">
<td :colspan="cancommand ? 10 : 9" style="text-align: center; color: var(--text-light);">
No endpoints found
</td>
</tr>

80
plugins/endpoints/migrations/versions/0012_endpoint_eventrecordid.py Normal file
View File

@@ -0,0 +1,80 @@
"""Dedup curated Windows events on the Windows event record id.
Adds endpointevents.eventrecordid and a unique index over
(assetid, eventchannel, eventrecordid).
Why: the agent used to re-read the newest window of every curated channel on
each cadence tick, and this ingest appended those rows without a dedup check, so
one real event was stored once per report for as long as it stayed in that
window - up to 96 identical rows a day. The agent now carries a per-channel
watermark keyed on the Windows record id, which is monotonic and unique within a
channel on a machine; this index is the server-side guarantee that survives an
agent reinstall, a restored image, or two overlapping reports, none of which
preserve the agent's watermark.
Nullable on purpose: an agent older than this field reports no record id, and
MySQL permits repeated NULLs inside a unique index, so those rows keep the
previous append-only behavior instead of failing ingest.
Existing rows are left alone. They all predate the column and so carry NULL,
which cannot collide with the new index. Purging the historical duplicates is a
separate, deliberate data cleanup - not something a schema migration should do
irreversibly.
Idempotent: both the column-add and the index-create check the live schema
first, so a db.create_all() site and a re-run both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0012eventrecordid'
down_revision = 'endpoints0011trayversion'
branch_labels = None
depends_on = None
INDEX_NAME = 'idx_endpointevents_assetrecord'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def _has_index(table_name, index_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(index['name'] == index_name
for index in inspector.get_indexes(table_name))
def upgrade():
if not _has_column('endpointevents', 'eventrecordid'):
op.add_column(
'endpointevents',
sa.Column('eventrecordid', sa.BigInteger(), nullable=True))
if not _has_index('endpointevents', INDEX_NAME):
op.create_index(
INDEX_NAME,
'endpointevents',
['assetid', 'eventchannel', 'eventrecordid'],
unique=True)
def downgrade():
if _has_index('endpointevents', INDEX_NAME):
op.drop_index(INDEX_NAME, table_name='endpointevents')
if _has_column('endpointevents', 'eventrecordid'):
op.drop_column('endpointevents', 'eventrecordid')

93
plugins/endpoints/migrations/versions/0013_endpoint_inventory_columns.py Normal file
View File

@@ -0,0 +1,93 @@
"""Persist the hardware, OS, and session inventory the agent already reports.
Adds to endpointstates: disksjson, biosjson, memoryjson, processorjson,
osinfojson, usercontextjson, domain, and pendingreboot.
Why now: the agent has been sending disks, bios, memory, processor, osinfo and
domain on every report for several versions. The ingest logged them as unknown
payload keys and threw them away, which was the intended staging step - the
shapes are now proven against real endpoint traffic, so they get columns.
usercontext and pendingreboot are new in this same change.
Shape follows the existing servicesjson/portsjson convention: newest snapshot
per endpoint as MEDIUMTEXT, not one row per sample. The agent resends the full
structure every cycle and operators ask "what is in this PC now", not "what was
in it in March". History would be pure write amplification on a fleet-sized
table.
domain and pendingreboot are real columns rather than JSON because they are
single values the fleet view filters, groups, and alerts on; burying a scalar in
a blob makes it unqueryable.
All columns are nullable. Every existing row predates them, and there is no
correct backfill value - a machine whose reboot state has never been reported is
NOT the same as one confirmed to have no reboot pending, and inventing False
would silently create a fleet of "healthy" endpoints.
Idempotent: each column-add checks the live schema first, so a db.create_all()
site and a re-run both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0013inventorycolumns'
down_revision = 'endpoints0012eventrecordid'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
# MEDIUMTEXT. An installed-application or disk list on a busy engineering
# workstation comfortably exceeds the 64KB TEXT ceiling, and a silent truncation
# there would corrupt the JSON rather than merely shorten it.
_SNAPSHOT_COLUMNS = (
'disksjson',
'biosjson',
'memoryjson',
'processorjson',
'osinfojson',
'usercontextjson',
)
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
for column_name in _SNAPSHOT_COLUMNS:
if not _has_column(TABLE_NAME, column_name):
op.add_column(
TABLE_NAME,
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
if not _has_column(TABLE_NAME, 'domain'):
op.add_column(
TABLE_NAME,
sa.Column('domain', sa.String(length=255), nullable=True))
if not _has_column(TABLE_NAME, 'pendingreboot'):
op.add_column(
TABLE_NAME,
sa.Column('pendingreboot', sa.Boolean(), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, 'pendingreboot'):
op.drop_column(TABLE_NAME, 'pendingreboot')
if _has_column(TABLE_NAME, 'domain'):
op.drop_column(TABLE_NAME, 'domain')
for column_name in reversed(_SNAPSHOT_COLUMNS):
if _has_column(TABLE_NAME, column_name):
op.drop_column(TABLE_NAME, column_name)

56
plugins/endpoints/migrations/versions/0014_endpoint_diskhealth.py Normal file
View File

@@ -0,0 +1,56 @@
"""Add diskhealthjson to endpointstates.
Per-drive health, kept separate from disksjson. That column answers "how full
is this drive"; this one answers "is this drive dying". Merging them would mean
a capacity collection failure and a health collection failure could not be told
apart, and the two come from different WMI sources with different failure modes.
JSON rather than scalar columns because a machine has a variable number of
drives and each carries its own verdict, and because the agent reports the same
drive from two sources - MSFT_PhysicalDisk for NVMe and Win32_DiskDrive for
ATA/SATA - which the server keeps side by side rather than collapsing.
MEDIUMTEXT for consistency with the other snapshot columns on this table.
Nullable with no backfill. A drive whose health has never been reported is not
a healthy drive, and defaulting to an empty list would read as "no disks have
problems" on every endpoint that predates this column.
Idempotent: the column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0014diskhealth'
down_revision = 'endpoints0013inventorycolumns'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
COLUMN_NAME = 'diskhealthjson'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
if not _has_column(TABLE_NAME, COLUMN_NAME):
op.add_column(
TABLE_NAME,
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, COLUMN_NAME):
op.drop_column(TABLE_NAME, COLUMN_NAME)

78
plugins/endpoints/migrations/versions/0015_endpoint_system_telemetry.py Normal file
View File

@@ -0,0 +1,78 @@
"""Add rebootreasonsjson, bootinfojson and defenderjson to endpointstates.
Phase 2 system-side telemetry. All three are collect-and-persist; none is
surfaced in the UI by this change.
rebootreasonsjson is NOT redundant with the existing pendingreboot column. That
column is a single boolean OR-ed from three registry signals, so it cannot say
which one fired, and the difference is operationally significant: a
PendingFileRenameOperations entry means files are staged for replacement on the
next boot, so the machine is mid-servicing, whereas a lone Component Based
Servicing or Windows Update key can sit set for weeks on a box nobody has
rebooted. Same boolean, very different urgency.
bootinfojson exists because uptime lies under fast startup - a machine reports
days of uptime after what the operator experienced as a shutdown, and a "reboot"
that was really a hybrid resume does not clear the driver or kernel state a
support call is usually trying to clear.
defenderjson is stored deliberately unsurfaced. Defender is known to produce
false positives in this environment, so wiring it to a card now would train
operators to ignore that card; persisting it starts the history accumulating for
whenever someone is ready to interpret it.
JSON rather than scalars: reboot reasons is a variable-length list, and the boot
and Defender payloads are small related groups that are read together and would
otherwise be six sparse columns.
MEDIUMTEXT for consistency with the other snapshot columns on this table.
Nullable with no backfill. Null means never reported, which is deliberately
distinct from an empty list (checked, nothing pending) - defaulting the reboot
reasons to [] would assert every pre-existing endpoint was confirmed clean.
Idempotent: each column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0015systemtelemetry'
down_revision = 'endpoints0014diskhealth'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
_COLUMNS = (
'rebootreasonsjson',
'bootinfojson',
'defenderjson',
)
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
for column_name in _COLUMNS:
if not _has_column(TABLE_NAME, column_name):
op.add_column(
TABLE_NAME,
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
def downgrade():
for column_name in reversed(_COLUMNS):
if _has_column(TABLE_NAME, column_name):
op.drop_column(TABLE_NAME, column_name)

51
plugins/endpoints/migrations/versions/0016_endpoint_dellupdates.py Normal file
View File

@@ -0,0 +1,51 @@
"""Add dellupdatesjson to endpointstates.
Stores the pending Dell Command | Update summary reported by the
shopdb-dellupdates agent utility. MEDIUMTEXT for consistency with the other
snapshot columns on this table.
Nullable with no backfill. Null means the endpoint has never reported a Dell
scan result, which is deliberately distinct from a stored payload whose status
is 'ok' with an empty list (scanned, nothing pending). The status card renders
those two states differently, so defaulting this to an empty summary would
assert every pre-existing endpoint had been scanned and found clean.
Idempotent: the column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0016dellupdates'
down_revision = 'endpoints0015systemtelemetry'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
COLUMN_NAME = 'dellupdatesjson'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
if not _has_column(TABLE_NAME, COLUMN_NAME):
op.add_column(
TABLE_NAME,
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, COLUMN_NAME):
op.drop_column(TABLE_NAME, COLUMN_NAME)

74
plugins/endpoints/models/endpoint.py
View File

@@ -80,6 +80,65 @@ class EndpointState(db.Model):
interfacesjson = db.Column(db.Text(length=16777215), nullable=True)
scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True)
# Hardware and OS inventory, same newest-snapshot rule as above. These
# arrived on the wire well before this and were logged as unknown keys, so
# the shapes are already proven against real endpoint traffic.
disksjson = db.Column(db.Text(length=16777215), nullable=True)
biosjson = db.Column(db.Text(length=16777215), nullable=True)
memoryjson = db.Column(db.Text(length=16777215), nullable=True)
processorjson = db.Column(db.Text(length=16777215), nullable=True)
osinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Session-scoped data collected by a probe running as the signed-in user
# (mapped drives, user-owned listeners). Null when nobody is signed in,
# which is a normal unattended state rather than a fault.
usercontextjson = db.Column(db.Text(length=16777215), nullable=True)
# Windows domain or workgroup membership. A scalar, not JSON: it is a single
# value the fleet view filters and groups on.
domain = db.Column(db.String(255), nullable=True)
# True when the endpoint is holding a reboot. Null = never reported, which
# is deliberately distinct from a confident False.
pendingreboot = db.Column(db.Boolean, nullable=True)
# Per-drive health from the agent's two disk sources. JSON because a machine
# has a variable number of drives and each carries its own verdict. Kept
# separate from disksjson: that is capacity, this is whether the hardware is
# dying. Null = never reported.
diskhealthjson = db.Column(db.Text(length=16777215), nullable=True)
# Which signal(s) drove pendingreboot. That column is a single bool OR-ed
# from three sources and so cannot say WHY, and the difference matters: a
# pendingfilerename means files are staged for replacement on next boot, so
# the box is mid-servicing, while a lone servicing key can sit set for weeks
# on a machine nobody rebooted. Empty list = checked, nothing pending.
# Null = never reported or the check failed.
rebootreasonsjson = db.Column(db.Text(length=16777215), nullable=True)
# Fast startup configuration and what the last boot actually did. Kept
# because uptime is not trustworthy under fast startup: a box can report
# days of uptime after what the operator experienced as a shutdown, and a
# "reboot" that was a hybrid resume does not clear the state a support call
# is usually trying to clear.
bootinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Defender health. Persisted but deliberately NOT surfaced anywhere yet:
# this environment produces known false positives, and wiring it to a card
# now would train operators to ignore that card. Stored so the history is
# already accumulating when someone is ready to interpret it.
defenderjson = db.Column(db.Text(length=16777215), nullable=True)
# Pending Dell Command | Update items, produced by the shopdb-dellupdates
# agent utility from the log the \ShopDB-DellCommandScan task writes. Only
# the count drives the status card; the short list is stored so the panel
# can show what is actually pending without a second round trip.
#
# A null column and a stored payload with status 'unknown' both mean "never
# scanned", which stays distinct from a scan that found nothing pending. The
# status card relies on that distinction - see _dell_updates_field.
dellupdatesjson = db.Column(db.Text(length=16777215), nullable=True)
# Warn-only reconciler output, refreshed on every agent check-in in the same
# commit as servicesjson. Each entry names a policy/shipped-attached monitor
# the endpoint is expected to be running but the latest snapshot does not
@@ -116,6 +175,8 @@ class EndpointState(db.Model):
'idleseconds': self.idleseconds,
'cpupercent': self.cpupercent,
'memorypercent': self.memorypercent,
'domain': self.domain,
'pendingreboot': self.pendingreboot,
'mtconnectrunning': self.mtconnectrunning,
'mtconnectcheckedat': _utciso(self.mtconnectcheckedat),
'lastreportedat': _utciso(self.lastreportedat),
@@ -255,6 +316,16 @@ class EndpointEvent(db.Model):
eventchannel = db.Column(db.String(50), nullable=True)
eventsource = db.Column(db.String(255), nullable=True)
# The Windows event record id: monotonic and unique within its channel on a
# given machine. This is the dedup key. The agent also uses it as its
# harvest watermark, so in normal operation a record is sent once; the
# unique index below is what guarantees it even when an agent is reinstalled
# (losing its watermark) or two reports overlap. Nullable because an agent
# older than this field still reports, and MySQL permits repeated NULLs in a
# unique index, so those degrade to the previous append-only behavior rather
# than failing.
eventrecordid = db.Column(db.BigInteger, nullable=True)
# When the event happened on the endpoint (from the record), not when it
# reached ShopDB.
eventtime = db.Column(db.DateTime, nullable=True, index=True)
@@ -269,6 +340,8 @@ class EndpointEvent(db.Model):
__table_args__ = (
db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'),
db.Index('idx_endpointevents_assetrecord',
'assetid', 'eventchannel', 'eventrecordid', unique=True),
)
def __repr__(self):
@@ -279,6 +352,7 @@ class EndpointEvent(db.Model):
'endpointeventid': self.endpointeventid,
'assetid': self.assetid,
'eventid': self.eventid,
'eventrecordid': self.eventrecordid,
'eventchannel': self.eventchannel,
'eventsource': self.eventsource,
'eventtime': _utciso(self.eventtime),

372
plugins/endpoints/plugin.py
View File

@@ -52,6 +52,9 @@ KNOWN_PAYLOAD_KEYS = frozenset({
'agentversion', 'trayversion', 'lastboottime', 'uptimeseconds', 'idleseconds',
'cpupercent', 'memorypercent', 'services', 'ports', 'interfaces',
'scheduledtasks', 'applications', 'events',
'disks', 'bios', 'memory', 'processor', 'osinfo', 'domain',
'pendingreboot', 'usercontext', 'diskhealth',
'rebootreasons', 'bootinfo', 'defender', 'dellupdates',
})
@@ -179,12 +182,32 @@ class EndpointsPlugin(BasePlugin):
# remote reboot is a strictly heavier grant than a routine toggle.
('endpoints.restart',
'Restart endpoint computers (destructive)', 'endpoints'),
# Elevated grant for software delivery (run_script, update_app).
# Elevated grant for software delivery (run_script, update_app) and
# for starting approved scheduled tasks (scheduledtask_run).
# Required IN ADDITION to endpoints.command by the enqueue route, so
# running a staged script or pushing an installer is a strictly
# heavier grant than a routine service/process toggle.
#
# NOTE FOR THE ROLES WORK - this wire is pre-built for you.
# scheduledtask_run was deliberately mapped onto this EXISTING grant
# rather than minting a new permission, so nothing needs migrating.
# Be aware that WIDENS what endpoints.deploy means: it used to cover
# run_script + update_app, and now also covers starting any task on
# the agent's approved-task allowlist. Splitting it out is a one-line
# change (add endpoints.corptools here and remap it in
# commands.ELEVATED_ACTION_PERMISSIONS) - the enqueue route reads
# that table, so no other code changes.
#
# If you do split it, note the three are NOT equal risk:
# scheduledtask_run only starts something already registered on the
# endpoint and is allowlisted agent-side; update_app runs a pinned
# installer; run_script is arbitrary code that runs as SYSTEM unless
# its descriptor pins the user context. run_script is the reason this
# grant's description has to stay blunt.
('endpoints.deploy',
'Run staged scripts and push application installers to endpoints', 'endpoints'),
'Run staged scripts, start approved scheduled tasks, and push '
'application installers to endpoints (scripts run as SYSTEM; '
'grant only to fully trusted operators)', 'endpoints'),
# Packaging the agent is a heavier grant than deploying to one, and
# it is not the same job either: the generated ZIP carries a live
# collector key stamped into its install script, so whoever holds
@@ -391,7 +414,9 @@ class EndpointsPlugin(BasePlugin):
'position': 12,
# Service state is what is being watched; the Start/Stop
# controls belong beside it, not in a tab of their own.
'group': 'monitors',
# Lives in serviceprocess rather than monitors: that tab is the
# box's own health, this is what runs on it.
'group': 'serviceprocess',
},
{
'id': 'endpoints-tracked-processes',
@@ -431,7 +456,7 @@ class EndpointsPlugin(BasePlugin):
'empty': 'No monitored processes reported.',
'position': 13,
# observed process state, not something you act on here
'group': 'monitors',
'group': 'serviceprocess',
},
{
# One feed instead of two cards. Events and commands answer the
@@ -472,6 +497,11 @@ class EndpointsPlugin(BasePlugin):
{'key': 'detail'},
{'key': 'eventtime', 'label': 'At', 'format': 'date'},
],
# Expands in place on click. The message body is several
# hundred words of Windows boilerplate around the one or
# two lines that matter, so it stays collapsed until asked
# for rather than pushing every other row off the card.
'body': 'body',
},
'empty': 'Nothing recorded for this endpoint yet.',
'position': 20,
@@ -493,10 +523,46 @@ class EndpointsPlugin(BasePlugin):
{'key': 'subnet', 'label': 'Subnet'},
{'key': 'dhcp', 'label': 'Assignment'},
{'key': 'role', 'label': 'Role'},
{'key': 'link', 'label': 'Link'},
],
'empty': 'No network interfaces reported.',
'position': 16,
# adapter state is watched, not something you act on
# adapter state is watched, not something you act on.
# Declared in monitors but RENDERED at the top of the Details
# tab (PCDetail.vue lifts it by id and excludes it here): which
# adapter is up and what address it holds is asked at the same
# moment as "what is this box".
'group': 'monitors',
},
{
# What this box has been on the network, carved out of the
# hand-written Network card that used to sit on Details and
# show only current addresses with a Primary badge.
#
# The history is ACCIDENTAL: _record_interfaces matches on MAC
# then IP and never deletes, so old addresses accumulate. That
# turned out to be worth reading rather than worth cleaning
# away, so it is presented as what it already is.
'id': 'endpoints-iphistory',
'title': 'IP History',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/iphistory',
'render': 'table',
'collapsible': True,
'columns': [
{'key': 'ipaddress', 'label': 'IP address', 'mono': True},
{'key': 'macaddress', 'label': 'MAC', 'mono': True},
{'key': 'status', 'label': 'Status'},
{'key': 'firstseen', 'label': 'First seen',
'format': 'date'},
# NOT "last seen": modifieddate only moves when a field on
# the row changes, so an address still held and reported
# unchanged keeps an old stamp here.
{'key': 'lastchanged', 'label': 'Last changed',
'format': 'date'},
],
'empty': 'No addresses on record for this endpoint.',
'position': 16.5,
'group': 'monitors',
},
{
@@ -524,10 +590,123 @@ class EndpointsPlugin(BasePlugin):
# the installed-software inventory
'group': 'software',
},
{
'id': 'endpoints-hardware',
'title': 'Hardware',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/hardware',
'render': 'keyvalue',
'collapsible': True,
'empty': 'No hardware inventory reported.',
'position': 14,
'group': 'monitors',
},
{
'id': 'endpoints-osinfo',
'title': 'Operating system',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/osinfo',
'render': 'keyvalue',
'collapsible': True,
'empty': 'No OS inventory reported.',
'position': 15,
'group': 'monitors',
},
{
# Session-scoped, so it sits with the other things watched
# rather than acted on. The user context card still carries a
# one-line summary; this is the same data with room to show a
# remote path and a per-drive badge, and both are built from
# _mapped_drive_rows so they cannot disagree.
#
# Room reserved here for the future one-click reconnect: the
# list render already supports a row action, so adding it is a
# map/commandendpoint change rather than a new card.
'id': 'endpoints-mappeddrives',
'title': 'Mapped drives',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/mappeddrives',
'render': 'list',
'collapsible': True,
'status': {
'key': 'statecolor',
'levels': {
'green': 'ok',
'yellow': 'warning',
'red': 'critical',
},
},
'map': {
'title': 'drive',
'meta': [{'key': 'remote', 'mono': True}],
'badge': {'label': 'statuslabel', 'color': 'statecolor'},
},
'empty': 'No mapped drives reported.',
'position': 21,
'group': 'monitors',
},
{
# Volumes and physical drive health in one card, as two
# sections. VISUAL GROUPING ONLY: the two tables share no key
# (storage is per volume, health is per physical device) and
# nothing is reconciled across them. They are one card because
# the operator reading a full disk and the one reading a dying
# disk are usually the same person in the same minute.
'id': 'endpoints-storage',
'title': 'Storage',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/storage',
'render': 'tabs',
'collapsible': True,
'empty': 'No disks reported.',
'position': 17,
'group': 'monitors',
},
{
# The listening set is the box's exposed network surface. The
# full connection table is deliberately not here: it is an
# on-demand command, because a per-cycle capture of every
# connection is a lot of rows nobody reads.
'id': 'endpoints-listeners',
'title': 'Listening ports',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/listeners',
'render': 'table',
'collapsible': True,
'columns': [
{'key': 'localport', 'label': 'Port'},
{'key': 'protocol', 'label': 'Protocol'},
{'key': 'localaddress', 'label': 'Address'},
{'key': 'processname', 'label': 'Process'},
{'key': 'processid', 'label': 'PID'},
],
'empty': 'No listening ports reported.',
'position': 19,
'group': 'monitors',
},
{
# Session-scoped data collected by a probe in the interactive
# user's session, so it is absent whenever nobody is signed in.
# Last on the page: it is the least reliably present card, and
# an empty one at the bottom reads as "not reported" rather
# than pushing the populated cards down.
'id': 'endpoints-usercontext',
'title': 'User context',
'assettypes': ['computer'],
'endpoint': '/api/endpoints/asset/{assetid}/usercontext',
'render': 'keyvalue',
'collapsible': True,
'empty': 'No user session data reported.',
'position': 40,
'group': 'monitors',
},
{
# Operator-submitted issue reports: a free-text note and an
# optional screenshot the operator captured from the tray. Sits
# in the Monitors tab beside the other watch-and-observe panels.
# optional screenshot the operator captured from the tray. Has
# a tab of its own rather than sitting in System Monitors: that
# tab is machine-collected health, this is a human telling you
# something, and the thumbnail grid is far bulkier than the
# fact cards it used to sit between.
# A custom render carries the thumbnail grid and the expanding
# lightbox with left/right cycling the generic modes cannot.
'id': 'endpoints-operator-reports',
@@ -539,7 +718,7 @@ class EndpointsPlugin(BasePlugin):
'collapsible': True,
'empty': 'No operator reports for this endpoint.',
'position': 18,
'group': 'monitors',
'group': 'reports',
},
]
@@ -636,6 +815,80 @@ class EndpointsPlugin(BasePlugin):
'type': 'array',
'description': 'Scheduled task inventory + last run.',
},
'disks': {
'type': 'array',
'description': 'Fixed local disks: {drive, label, '
'filesystem, sizebytes, freebytes, '
'usedbytes, freepercent}. Newest '
'snapshot, replaced on every report.',
},
'bios': {
'type': 'object',
'description': 'Firmware identity: vendor, version, '
'release date.',
},
'memory': {
'type': 'object',
'description': 'Physical memory totals in bytes.',
},
'processor': {
'type': 'object',
'description': 'CPU identity, core and logical '
'processor counts, max clock.',
},
'osinfo': {
'type': 'object',
'description': 'OS build, version, and install date.',
},
'domain': {
'type': 'string',
'description': 'Windows domain or workgroup membership.',
},
'pendingreboot': {
'type': 'boolean',
'description': 'True when the endpoint is holding a '
'reboot. Absent means never reported, '
'which is not the same as false.',
},
'diskhealth': {
'type': 'array',
'description': 'Per-drive health from two sources: '
'physicaldisk (the only one reporting '
'NVMe) and diskdrive (ATA/SATA and '
'fallback). Each entry carries a '
'precomputed predictfailure; null there '
'means the drive did not answer.',
},
'rebootreasons': {
'type': 'array',
'description': 'Which signal(s) drove pendingreboot: '
'componentbasedservicing, windowsupdate, '
'pendingfilerename. Empty array means '
'checked and nothing pending; omit when '
'the check failed.',
},
'bootinfo': {
'type': 'object',
'description': 'hiberbootenabled (fast startup is '
'configured) and lastboottype (what the '
'last boot actually did). Uptime is not '
'trustworthy when fast startup is in '
'play.',
},
'defender': {
'type': 'object',
'description': 'Defender service/realtime state and '
'signature and scan ages in days. '
'Persisted but not surfaced.',
},
'usercontext': {
'type': 'object',
'description': 'Session-scoped data collected by a probe '
'running as the signed-in user: mapped '
'drives (netuse), user-owned listeners, '
'and idle seconds. Absent when nobody is '
'signed in.',
},
'applications': {
'type': 'array',
'description': 'Installed-software SBOM from both '
@@ -663,6 +916,12 @@ class EndpointsPlugin(BasePlugin):
'required': ['eventid'],
'properties': {
'eventid': {'type': 'integer'},
'eventrecordid': {
'type': 'integer',
'description': 'Windows event record id; '
'unique per channel per '
'machine, used to dedup.',
},
'eventchannel': {'type': 'string'},
'eventsource': {'type': 'string'},
'eventtime': {'type': 'string',
@@ -846,6 +1105,48 @@ class EndpointsPlugin(BasePlugin):
if 'scheduledtasks' in payload:
state.scheduledtasksjson = _as_json(payload.get('scheduledtasks'))
# Hardware/OS inventory. Same presence guard as above: a report that
# omits a key must not wipe the last-known value. These change rarely
# (a RAM upgrade, a BIOS flash), so a transient collection failure on
# one cycle should leave the previous snapshot standing rather than
# blank the asset record.
if 'disks' in payload:
state.disksjson = _as_json(payload.get('disks'))
if 'bios' in payload:
state.biosjson = _as_json(payload.get('bios'))
if 'memory' in payload:
state.memoryjson = _as_json(payload.get('memory'))
if 'processor' in payload:
state.processorjson = _as_json(payload.get('processor'))
if 'osinfo' in payload:
state.osinfojson = _as_json(payload.get('osinfo'))
if 'domain' in payload:
state.domain = payload.get('domain')
if 'pendingreboot' in payload:
state.pendingreboot = payload.get('pendingreboot')
if 'diskhealth' in payload:
state.diskhealthjson = _as_json(payload.get('diskhealth'))
if 'rebootreasons' in payload:
state.rebootreasonsjson = _as_json(payload.get('rebootreasons'))
if 'bootinfo' in payload:
state.bootinfojson = _as_json(payload.get('bootinfo'))
if 'defender' in payload:
state.defenderjson = _as_json(payload.get('defender'))
# Presence-guarded like the probe output below: the agent omits the key
# entirely when the collector utility is not installed, and overwriting
# to null there would discard a still-valid scan result from before the
# utility was removed.
if 'dellupdates' in payload:
state.dellupdatesjson = _as_json(payload.get('dellupdates'))
# Session-scoped probe output. The agent repeats its last result between
# probe runs and omits the key entirely when nobody is signed in, so the
# presence guard preserves the last known mapped drives across an
# unattended period instead of flapping the field to null.
if 'usercontext' in payload:
state.usercontextjson = _as_json(payload.get('usercontext'))
# Installed-software SBOM: stored as a per-asset SNAPSHOT in its own
# table (endpointapplications), not a JSON blob on the state, so it can
# later join the applications catalog ("Managed Applications"). Only
@@ -897,6 +1198,19 @@ class EndpointsPlugin(BasePlugin):
hostname, len(unknown_keys), sorted(unknown_keys), preview_text)
stored_events = 0
skipped_events = 0
# Records the agent already sent must not be appended a second time.
# The agent carries a per-channel watermark so the steady state is
# "only what is new", but a reinstalled agent, a restored image, or two
# overlapping reports can all replay records. Resolve that here rather
# than relying on the unique index to raise, because an IntegrityError
# would abort the whole collector transaction and lose the state upsert
# along with it.
#
# One query for the whole batch, not one per record: a seeding agent
# sends up to the per-channel cap at once.
candidates = []
for record in (payload.get('events') or []):
try:
eventid = int(record.get('eventid'))
@@ -906,10 +1220,45 @@ class EndpointsPlugin(BasePlugin):
if eventid not in CURATED_EVENT_IDS:
# Outside the catalog: ignored on purpose, not an error.
continue
try:
eventrecordid = record.get('eventrecordid')
eventrecordid = (int(eventrecordid)
if eventrecordid is not None else None)
except (TypeError, ValueError):
eventrecordid = None
candidates.append((eventid, eventrecordid, record))
keyed = [(record.get('eventchannel'), eventrecordid)
for _e, eventrecordid, record in candidates
if eventrecordid is not None]
already_stored = set()
if keyed:
rows = (db.session.query(EndpointEvent.eventchannel,
EndpointEvent.eventrecordid)
.filter(EndpointEvent.assetid == assetid,
EndpointEvent.eventrecordid.in_(
[recordid for _c, recordid in keyed]))
.all())
already_stored = {(channel, recordid) for channel, recordid in rows}
for eventid, eventrecordid, record in candidates:
eventchannel = record.get('eventchannel')
if eventrecordid is not None:
key = (eventchannel, eventrecordid)
if key in already_stored:
skipped_events += 1
continue
# Guard against the same record appearing twice inside one
# payload as well as against rows already in the table.
already_stored.add(key)
db.session.add(EndpointEvent(
assetid=assetid,
eventid=eventid,
eventchannel=record.get('eventchannel'),
eventrecordid=eventrecordid,
eventchannel=eventchannel,
eventsource=record.get('eventsource'),
eventtime=_parse_dt(record.get('eventtime')),
eventmessage=record.get('eventmessage'),
@@ -917,6 +1266,11 @@ class EndpointsPlugin(BasePlugin):
))
stored_events += 1
if skipped_events:
logger.debug(
'endpoint %s replayed %d already-stored event record(s)',
hostname, skipped_events)
# flush, not commit: the collector dispatcher owns the transaction.
db.session.flush()

8
plugins/machines/frontend/views/MachineForm.vue
View File

@@ -591,14 +591,14 @@ onMounted(async () => {
machineTypes.value = typesRes.data.data || []
statuses.value = statusRes.data.data || []
vendors.value = vendorRes.data.data || []
locations.value = locRes || []
models.value = allModels
businessunits.value = buRes.data.data || []
// listAll resolves to the ARRAY, not a response: fetchAllPages already
// unwrapped every page. Reading .data.data off it threw, and the whole
// parallel load went to the catch - so every dropdown on this form came up
// empty and the machine's own values never loaded.
vendors.value = vendorRes || []
locations.value = locRes || []
models.value = allModels
businessunits.value = buRes || []
pcs.value = pcsRes || []
// Load relationship types separately

5
plugins/measuringtools/frontend/views/MeasuringToolForm.vue
View File

@@ -187,8 +187,11 @@ onMounted(async () => {
])
types.value = typesResponse.data.data || []
statuses.value = statusResponse.data.data || []
// listAll resolves to the ARRAY, not a response: fetchAllPages already
// unwrapped every page. Reading .data.data off it throws and takes the
// whole parallel load into the catch.
locations.value = locResponse || []
businessunits.value = buResponse.data.data || []
businessunits.value = buResponse || []
const assetType = (assetTypesResponse.data.data || []).find(a => a.assettype === 'measuring_tool')
assettypeid.value = assetType ? assetType.assettypeid : null