Report each Windows event once, and persist the inventory already sent

Curated events were re-read from the newest 25 entries per channel on every
collect, and the server appended whatever arrived. One real event was
therefore stored again on every cadence tick for as long as it stayed in
that window, so event counts measured how long a box had been up rather
than what had happened to it.

The agent now keeps a per-channel watermark of the last Windows event
record id it reported, persisted under ProgramData so a service restart
does not replay history, and sends eventrecordid with each event. The
server dedupes on (assetid, eventchannel, eventrecordid) behind a unique
index, so a replay from an older agent cannot double-store either.

Six inventory keys the agent had been sending all along - bios, disks,
domain, memory, osinfo, processor - were parsed, logged as unpersisted,
and dropped. They now have columns and are written on check-in, along
with user context, pending reboot, and the cpupercent/memorypercent/
idleseconds fields that were declared in the contract but always null.

Session-scoped data cannot be read from the agent's LocalSystem session 0
at all: mapped drives and user-owned listeners do not exist there. A probe
runs in the interactive user's session and hands results back through a
spool file. It is not yet producing output on a live endpoint; the column
and panel are in place and read empty until that is fixed.

Also bounds the user-context script wait. It was WaitForSingleObject with
INFINITE, so a hung script pinned the command worker permanently.
This commit is contained in:
benjamin.baker
2026-09-04 19:09:12 -04:00
parent 3714b4bc99
commit 862b290545
18 changed files with 1633 additions and 39 deletions

View File

@@ -80,6 +80,28 @@ class EndpointState(db.Model):
interfacesjson = db.Column(db.Text(length=16777215), nullable=True)
scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True)
# Hardware and OS inventory, same newest-snapshot rule as above. These
# arrived on the wire well before this and were logged as unknown keys, so
# the shapes are already proven against real endpoint traffic.
disksjson = db.Column(db.Text(length=16777215), nullable=True)
biosjson = db.Column(db.Text(length=16777215), nullable=True)
memoryjson = db.Column(db.Text(length=16777215), nullable=True)
processorjson = db.Column(db.Text(length=16777215), nullable=True)
osinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Session-scoped data collected by a probe running as the signed-in user
# (mapped drives, user-owned listeners). Null when nobody is signed in,
# which is a normal unattended state rather than a fault.
usercontextjson = db.Column(db.Text(length=16777215), nullable=True)
# Windows domain or workgroup membership. A scalar, not JSON: it is a single
# value the fleet view filters and groups on.
domain = db.Column(db.String(255), nullable=True)
# True when the endpoint is holding a reboot. Null = never reported, which
# is deliberately distinct from a confident False.
pendingreboot = db.Column(db.Boolean, nullable=True)
# Warn-only reconciler output, refreshed on every agent check-in in the same
# commit as servicesjson. Each entry names a policy/shipped-attached monitor
# the endpoint is expected to be running but the latest snapshot does not
@@ -116,6 +138,8 @@ class EndpointState(db.Model):
'idleseconds': self.idleseconds,
'cpupercent': self.cpupercent,
'memorypercent': self.memorypercent,
'domain': self.domain,
'pendingreboot': self.pendingreboot,
'mtconnectrunning': self.mtconnectrunning,
'mtconnectcheckedat': _utciso(self.mtconnectcheckedat),
'lastreportedat': _utciso(self.lastreportedat),
@@ -255,6 +279,16 @@ class EndpointEvent(db.Model):
eventchannel = db.Column(db.String(50), nullable=True)
eventsource = db.Column(db.String(255), nullable=True)
# The Windows event record id: monotonic and unique within its channel on a
# given machine. This is the dedup key. The agent also uses it as its
# harvest watermark, so in normal operation a record is sent once; the
# unique index below is what guarantees it even when an agent is reinstalled
# (losing its watermark) or two reports overlap. Nullable because an agent
# older than this field still reports, and MySQL permits repeated NULLs in a
# unique index, so those degrade to the previous append-only behavior rather
# than failing.
eventrecordid = db.Column(db.BigInteger, nullable=True)
# When the event happened on the endpoint (from the record), not when it
# reached ShopDB.
eventtime = db.Column(db.DateTime, nullable=True, index=True)
@@ -269,6 +303,8 @@ class EndpointEvent(db.Model):
__table_args__ = (
db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'),
db.Index('idx_endpointevents_assetrecord',
'assetid', 'eventchannel', 'eventrecordid', unique=True),
)
def __repr__(self):
@@ -279,6 +315,7 @@ class EndpointEvent(db.Model):
'endpointeventid': self.endpointeventid,
'assetid': self.assetid,
'eventid': self.eventid,
'eventrecordid': self.eventrecordid,
'eventchannel': self.eventchannel,
'eventsource': self.eventsource,
'eventtime': _utciso(self.eventtime),