ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Import the wilm endpoint telemetry, PC Detail rework and backup overlay

Browse Source 
Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at 7905917c. wilm rides the publication lineage and shares no
ancestor with wj, so this is a filtered tree patch rather than a merge:
94 files, and every path the publication export strips (docs/, tools/, mcp/,
CLAUDE.md, start-*.sh, .gitea/, the three excluded tests) is held back, because
a raw diff would delete each of them from this tree.

What comes across:

* Endpoint telemetry, phase 2. Physical disk health from two sources, NIC link
  state, boot type, Defender health, processor/memory/chassis inventory, the
  listening TCP set, mapped network drives and Windows Update history. Five
  additive migrations, 0012 through 0016, off the 0011 head.
* Mapped drives are collected by a probe running in the interactive user's
  session, because they do not exist in the agent's session 0. Two Windows
  sources can disagree about one drive and the worse status wins, so a
  half-broken share is never rendered as healthy.
* Windows events are reported once each, keyed on the channel's own record id,
  instead of being re-sent whenever a report overlapped the last one.
* PC Detail rework: Details cards flow, the hand-written Network card gives way
  to a full address history, Storage and disk health merge into one card with
  two sections, Operator Reports and Corporate Tools get their own tabs.
* An opt-in restic backup overlay with a third container, restore gated on the
  dump's own data revision rather than the code head, and backup/restore history.
* The agent's version is read from the binary's PE version rather than trusted
  from the csproj, and the Fleet view gains an Agent Version column.
* The enqueue route now asks caller_holds instead of the owner's permissions,
  which closes the scoped-token elevation on run_script, update_app and
  system_restart.

Held back deliberately:

* agent/installer/install-shopdbagent.ps1 stays deleted, as upstream deleted it.
  The installer that reaches a bay is generated by _install_script() in
  plugins/endpoints/api/siteagent.py; the standalone script had stopped being
  shipped and was collecting fixes no endpoint ever ran.
* The .gitignore hunk adding docs/*.md. Correct on the publication lineage,
  where docs/ becomes the wiki and its absence is how the gates detect a
  published checkout. Here it would silently ignore every new doc.
* Their deletions of docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md. Upstream
  those were stubs; here PLUGIN-HOOKS.md is the 31KB document CLAUDE.md points
  readers at.
* Five zero-byte files committed by an accidental `git add -A`, a run log from
  the author's own machine, and docs/notes/, which is working scratch that says
  so in its own header and belongs in devnotes/ on this side.

Carried with the import so it lands green: the 17 new routes are described in
docs/api-inventory.json with the spec and project map regenerated, and two new
frontend specs read fixtures through fileURLToPath(URL) which node rejects under
jsdom's own URL global.

Suite 2591 passed / 9 skipped; frontend 391 passed; naming, style and doc gates
clean.

Scope: product
This commit is contained in:
cproudlock
2026-09-06 16:47:24 -04:00
parent 9947a477b8
commit 6e36df841e
96 changed files with 12395 additions and 761 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
plugins/endpoints/models/endpoint.py
View File

@@ -80,6 +80,65 @@ class EndpointState(db.Model):
interfacesjson = db.Column(db.Text(length=16777215), nullable=True)
scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True)
# Hardware and OS inventory, same newest-snapshot rule as above. These
# arrived on the wire well before this and were logged as unknown keys, so
# the shapes are already proven against real endpoint traffic.
disksjson = db.Column(db.Text(length=16777215), nullable=True)
biosjson = db.Column(db.Text(length=16777215), nullable=True)
memoryjson = db.Column(db.Text(length=16777215), nullable=True)
processorjson = db.Column(db.Text(length=16777215), nullable=True)
osinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Session-scoped data collected by a probe running as the signed-in user
# (mapped drives, user-owned listeners). Null when nobody is signed in,
# which is a normal unattended state rather than a fault.
usercontextjson = db.Column(db.Text(length=16777215), nullable=True)
# Windows domain or workgroup membership. A scalar, not JSON: it is a single
# value the fleet view filters and groups on.
domain = db.Column(db.String(255), nullable=True)
# True when the endpoint is holding a reboot. Null = never reported, which
# is deliberately distinct from a confident False.
pendingreboot = db.Column(db.Boolean, nullable=True)
# Per-drive health from the agent's two disk sources. JSON because a machine
# has a variable number of drives and each carries its own verdict. Kept
# separate from disksjson: that is capacity, this is whether the hardware is
# dying. Null = never reported.
diskhealthjson = db.Column(db.Text(length=16777215), nullable=True)
# Which signal(s) drove pendingreboot. That column is a single bool OR-ed
# from three sources and so cannot say WHY, and the difference matters: a
# pendingfilerename means files are staged for replacement on next boot, so
# the box is mid-servicing, while a lone servicing key can sit set for weeks
# on a machine nobody rebooted. Empty list = checked, nothing pending.
# Null = never reported or the check failed.
rebootreasonsjson = db.Column(db.Text(length=16777215), nullable=True)
# Fast startup configuration and what the last boot actually did. Kept
# because uptime is not trustworthy under fast startup: a box can report
# days of uptime after what the operator experienced as a shutdown, and a
# "reboot" that was a hybrid resume does not clear the state a support call
# is usually trying to clear.
bootinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Defender health. Persisted but deliberately NOT surfaced anywhere yet:
# this environment produces known false positives, and wiring it to a card
# now would train operators to ignore that card. Stored so the history is
# already accumulating when someone is ready to interpret it.
defenderjson = db.Column(db.Text(length=16777215), nullable=True)
# Pending Dell Command | Update items, produced by the shopdb-dellupdates
# agent utility from the log the \ShopDB-DellCommandScan task writes. Only
# the count drives the status card; the short list is stored so the panel
# can show what is actually pending without a second round trip.
#
# A null column and a stored payload with status 'unknown' both mean "never
# scanned", which stays distinct from a scan that found nothing pending. The
# status card relies on that distinction - see _dell_updates_field.
dellupdatesjson = db.Column(db.Text(length=16777215), nullable=True)
# Warn-only reconciler output, refreshed on every agent check-in in the same
# commit as servicesjson. Each entry names a policy/shipped-attached monitor
# the endpoint is expected to be running but the latest snapshot does not
@@ -116,6 +175,8 @@ class EndpointState(db.Model):
'idleseconds': self.idleseconds,
'cpupercent': self.cpupercent,
'memorypercent': self.memorypercent,
'domain': self.domain,
'pendingreboot': self.pendingreboot,
'mtconnectrunning': self.mtconnectrunning,
'mtconnectcheckedat': _utciso(self.mtconnectcheckedat),
'lastreportedat': _utciso(self.lastreportedat),
@@ -255,6 +316,16 @@ class EndpointEvent(db.Model):
eventchannel = db.Column(db.String(50), nullable=True)
eventsource = db.Column(db.String(255), nullable=True)
# The Windows event record id: monotonic and unique within its channel on a
# given machine. This is the dedup key. The agent also uses it as its
# harvest watermark, so in normal operation a record is sent once; the
# unique index below is what guarantees it even when an agent is reinstalled
# (losing its watermark) or two reports overlap. Nullable because an agent
# older than this field still reports, and MySQL permits repeated NULLs in a
# unique index, so those degrade to the previous append-only behavior rather
# than failing.
eventrecordid = db.Column(db.BigInteger, nullable=True)
# When the event happened on the endpoint (from the record), not when it
# reached ShopDB.
eventtime = db.Column(db.DateTime, nullable=True, index=True)
@@ -269,6 +340,8 @@ class EndpointEvent(db.Model):
__table_args__ = (
db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'),
db.Index('idx_endpointevents_assetrecord',
'assetid', 'eventchannel', 'eventrecordid', unique=True),
)
def __repr__(self):
@@ -279,6 +352,7 @@ class EndpointEvent(db.Model):
'endpointeventid': self.endpointeventid,
'assetid': self.assetid,
'eventid': self.eventid,
'eventrecordid': self.eventrecordid,
'eventchannel': self.eventchannel,
'eventsource': self.eventsource,
'eventtime': _utciso(self.eventtime),