04616fce8b
Fleet ranked endpoints by raw critical-event counts, so a machine with a noisy but harmless event log outranked one with a mapped drive down or a blocking service stopped. Volume is not severity, and the aggregation ran across the whole fleet on every page load. Rank operational blockers above degraded conditions above event noise, and cache the verdict per asset so Fleet reads a row instead of recomputing 48h of history. Transitions are written only when a verdict changes, capped at 50 rows per asset. Absence of a policy binding must never suppress a verdict: drives and events are not policy-gated, and services fall back to a built-in default impact when no mapping applies. Event 1001 drops to informational because the WER bucket pairs with 1000 and double-counted every crash. Parallelize the Fleet first-paint loaders that were awaiting serially.
268 lines
11 KiB
Python
268 lines
11 KiB
Python
"""Endpoint health rollup: cached verdict, transition history, impact mapping.
|
|
|
|
Three tables, one job: decide what the Fleet Health column says about an
|
|
endpoint, decide it ONCE when the data changes, and keep enough history to
|
|
answer "when did this break".
|
|
|
|
Why this exists at all. The verdict used to be computed on every Fleet page
|
|
load by aggregating 48h of endpointevents fleet-wide, which cost more the
|
|
longer the fleet ran, and it ranked a raw Windows event count above every
|
|
other signal. A box with a crash loop reported "Application crash x5861" while
|
|
its dead eDNC service - the thing that actually stops work on the floor - was
|
|
structurally unreachable, because the event branch returned first.
|
|
|
|
The model here separates two things the old rollup conflated:
|
|
|
|
events Point-in-time facts. They happened, they are counted, they age
|
|
out of the window on their own.
|
|
conditions A stopped service or a disconnected drive is not an event, it is
|
|
a STATE with a duration. It opens, it persists, and it is only
|
|
over when something observes it to be over.
|
|
|
|
Ranking a condition against an event count is comparing incompatible things.
|
|
Conditions are pinned until explicitly resolved; events are ranked beneath
|
|
them. That is the whole reason the Health column can be trusted now.
|
|
"""
|
|
|
|
from datetime import datetime
|
|
|
|
from shopdb.api import db
|
|
|
|
|
|
def _utciso(value):
|
|
"""Naive-UTC datetime -> ISO string marked UTC, or None."""
|
|
return value.isoformat() + 'Z' if value else None
|
|
|
|
|
|
# Operational impact, in the site's terms rather than the OS's. Windows calls a
|
|
# crash severe; the floor calls a disconnected drive a stoppage. These disagree,
|
|
# and the floor wins - which is why this vocabulary is not the event catalog's
|
|
# 'critical'/'warning'/'info'.
|
|
IMPACT_BLOCKER = 'blocker'
|
|
IMPACT_DEGRADED = 'degraded'
|
|
IMPACT_INFORMATIONAL = 'informational'
|
|
|
|
IMPACT_CLASSES = (IMPACT_BLOCKER, IMPACT_DEGRADED, IMPACT_INFORMATIONAL)
|
|
|
|
# Lower sorts worse, matching _DRIVE_STATUS_RANK's existing convention so the
|
|
# two ladders in this plugin do not read in opposite directions.
|
|
IMPACT_RANK = {
|
|
IMPACT_BLOCKER: 0,
|
|
IMPACT_DEGRADED: 1,
|
|
IMPACT_INFORMATIONAL: 2,
|
|
}
|
|
|
|
# Where a verdict came from. Kept alongside sourceref so the UI can deep-link
|
|
# back to the monitor or the event that produced the label.
|
|
SOURCE_DRIVE = 'drive'
|
|
SOURCE_SERVICE = 'service'
|
|
SOURCE_PROCESS = 'process'
|
|
SOURCE_EVENT = 'event'
|
|
SOURCE_NONE = 'none'
|
|
|
|
|
|
class EndpointHealthState(db.Model):
|
|
"""The current health verdict for one endpoint - the Fleet page's only read.
|
|
|
|
Cached deliberately. The Fleet list joins this and reads it directly; it
|
|
does not recompute anything per row. Written in the SAME commit as the
|
|
check-in that produced it, so the cached verdict and the telemetry it
|
|
reflects can never disagree (the ledger pattern this codebase already uses
|
|
for quantityonhand).
|
|
"""
|
|
|
|
__tablename__ = 'endpointhealthstates'
|
|
|
|
endpointhealthstateid = db.Column(db.Integer, primary_key=True)
|
|
|
|
# One row per asset. Unique rather than a plain index: two live verdicts
|
|
# for one endpoint is not a state this table should be able to represent.
|
|
assetid = db.Column(
|
|
db.Integer,
|
|
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
|
nullable=False,
|
|
unique=True,
|
|
)
|
|
|
|
# Display vocabulary, matching what the Fleet renderer already understands:
|
|
# 'ok', 'warning', 'critical', 'unknown'.
|
|
level = db.Column(db.String(16), nullable=False, default='unknown')
|
|
|
|
# The phrase an operator reads ("eDNC stopped", "S: disconnected"). A
|
|
# PHRASE, not a score - it has to say what to do.
|
|
label = db.Column(db.String(255), nullable=True)
|
|
|
|
# Null for an ok/unknown verdict: nothing is wrong, so nothing has impact.
|
|
impactclass = db.Column(db.String(16), nullable=True)
|
|
|
|
sourcekind = db.Column(db.String(16), nullable=False, default=SOURCE_NONE)
|
|
|
|
# Monitor id, drive letter or Windows event id, depending on sourcekind.
|
|
# Free text because those identifier spaces have nothing in common.
|
|
sourceref = db.Column(db.String(64), nullable=True)
|
|
|
|
# When the CURRENT condition opened. Survives re-evaluation: a service that
|
|
# has been down for three days keeps its original firstseenat, so the UI can
|
|
# say how long rather than only that it is down now.
|
|
firstseenat = db.Column(db.DateTime, nullable=True)
|
|
|
|
# Last time an agent explicitly observed this endpoint. Drives staleness:
|
|
# a pinned blocker on an endpoint that stopped reporting is neither
|
|
# resolvable nor trustworthy, and must not render as present tense.
|
|
lastconfirmedat = db.Column(db.DateTime, nullable=True)
|
|
|
|
lastevaluatedat = db.Column(db.DateTime, nullable=False,
|
|
default=datetime.utcnow)
|
|
|
|
# Transitions inside the trailing window. A service flapping ten times an
|
|
# hour is a different fault from one cleanly down, and dispatching someone
|
|
# to a service that has already restarted twice wastes a trip.
|
|
flapcount = db.Column(db.Integer, nullable=False, default=0)
|
|
|
|
__table_args__ = (
|
|
db.Index('idx_endpointhealthstates_asset', 'assetid'),
|
|
db.Index('idx_endpointhealthstates_impactclass', 'impactclass'),
|
|
)
|
|
|
|
def __repr__(self):
|
|
return f"<EndpointHealthState asset={self.assetid} {self.level}>"
|
|
|
|
def to_dict(self):
|
|
return {
|
|
'assetid': self.assetid,
|
|
'level': self.level,
|
|
'label': self.label,
|
|
'impactclass': self.impactclass,
|
|
'sourcekind': self.sourcekind,
|
|
'sourceref': self.sourceref,
|
|
'firstseenat': _utciso(self.firstseenat),
|
|
'lastconfirmedat': _utciso(self.lastconfirmedat),
|
|
'lastevaluatedat': _utciso(self.lastevaluatedat),
|
|
'flapcount': self.flapcount,
|
|
}
|
|
|
|
|
|
class EndpointHealthTransition(db.Model):
|
|
"""One verdict CHANGE for an endpoint. Written on change, never on sample.
|
|
|
|
Storing a row per check-in would make the last 50 rows cover a few minutes
|
|
and say nothing. Storing a row per change makes 50 rows cover months on a
|
|
stable endpoint, compresses "nothing happened" to zero cost, and answers
|
|
the question operators actually ask - when did this break - directly rather
|
|
than by inferring it from a gap between samples.
|
|
|
|
It also makes flapping measurable: 50 transitions in an hour and 50 over a
|
|
year are the same row count and obviously different faults.
|
|
"""
|
|
|
|
__tablename__ = 'endpointhealthtransitions'
|
|
|
|
endpointhealthtransitionid = db.Column(db.Integer, primary_key=True)
|
|
|
|
assetid = db.Column(
|
|
db.Integer,
|
|
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
|
nullable=False,
|
|
)
|
|
|
|
level = db.Column(db.String(16), nullable=False)
|
|
label = db.Column(db.String(255), nullable=True)
|
|
impactclass = db.Column(db.String(16), nullable=True)
|
|
sourcekind = db.Column(db.String(16), nullable=False, default=SOURCE_NONE)
|
|
sourceref = db.Column(db.String(64), nullable=True)
|
|
|
|
openedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
|
|
|
|
# Null means STILL OPEN. Retention never evicts an unresolved row: at 50
|
|
# deep a long-running blocker's opening transition would otherwise age out
|
|
# behind a burst of unrelated churn, deleting the record of why the thing
|
|
# currently on screen is on screen.
|
|
resolvedat = db.Column(db.DateTime, nullable=True)
|
|
|
|
__table_args__ = (
|
|
db.Index('idx_endpointhealthtransitions_assetopened',
|
|
'assetid', 'openedat'),
|
|
db.Index('idx_endpointhealthtransitions_assetresolved',
|
|
'assetid', 'resolvedat'),
|
|
)
|
|
|
|
def __repr__(self):
|
|
return (f"<EndpointHealthTransition asset={self.assetid} "
|
|
f"{self.level} opened={self.openedat}>")
|
|
|
|
def to_dict(self):
|
|
return {
|
|
'endpointhealthtransitionid': self.endpointhealthtransitionid,
|
|
'assetid': self.assetid,
|
|
'level': self.level,
|
|
'label': self.label,
|
|
'impactclass': self.impactclass,
|
|
'sourcekind': self.sourcekind,
|
|
'sourceref': self.sourceref,
|
|
'openedat': _utciso(self.openedat),
|
|
'resolvedat': _utciso(self.resolvedat),
|
|
}
|
|
|
|
|
|
class EndpointMonitorImpact(db.Model):
|
|
"""How much a given monitor matters ON A GIVEN PC TYPE.
|
|
|
|
eDNC stopped on a shopfloor PC halts work; the same service stopped on a
|
|
lobby display is noise. Impact is therefore a property of the monitor IN
|
|
CONTEXT, not of the monitor and not of the event type - which is why it
|
|
cannot live on endpointmonitors and could never have lived in the event
|
|
catalog.
|
|
|
|
Why a plugin-owned side table rather than a column on the policy action:
|
|
monitors bind to PC types through configurationpolicyactions, which is a
|
|
CORE table shared by every plugin. Adding an endpoints-specific impact
|
|
column there would be a contract-surface change imposed on unrelated
|
|
plugins. This keeps the concept inside the plugin that owns it.
|
|
|
|
ABSENCE OF A ROW IS THE DEFAULT, and that is load-bearing. An endpoint with
|
|
no policies, no PC type, or no monitors at all resolves to the built-in
|
|
default impact for its source kind - the same verdict the old rollup gave.
|
|
The no-policy path is the NORMAL path here, not a special case.
|
|
"""
|
|
|
|
__tablename__ = 'endpointmonitorimpacts'
|
|
|
|
endpointmonitorimpactid = db.Column(db.Integer, primary_key=True)
|
|
|
|
endpointmonitorid = db.Column(
|
|
db.Integer,
|
|
db.ForeignKey('endpointmonitors.endpointmonitorid',
|
|
ondelete='CASCADE'),
|
|
nullable=False,
|
|
)
|
|
|
|
# Plain integer, not a ForeignKey: computertypes belongs to the computers
|
|
# plugin and owns its own migration chain. A cross-plugin FK would couple
|
|
# the two chains' ordering for no benefit the unique constraint below does
|
|
# not already provide.
|
|
computertypeid = db.Column(db.Integer, nullable=False)
|
|
|
|
impactclass = db.Column(db.String(16), nullable=False,
|
|
default=IMPACT_DEGRADED)
|
|
|
|
createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
|
|
|
|
__table_args__ = (
|
|
db.UniqueConstraint('endpointmonitorid', 'computertypeid',
|
|
name='uq_endpointmonitorimpacts_monitortype'),
|
|
db.Index('idx_endpointmonitorimpacts_computertypeid', 'computertypeid'),
|
|
)
|
|
|
|
def __repr__(self):
|
|
return (f"<EndpointMonitorImpact monitor={self.endpointmonitorid} "
|
|
f"pctype={self.computertypeid} {self.impactclass}>")
|
|
|
|
def to_dict(self):
|
|
return {
|
|
'endpointmonitorimpactid': self.endpointmonitorimpactid,
|
|
'endpointmonitorid': self.endpointmonitorid,
|
|
'computertypeid': self.computertypeid,
|
|
'impactclass': self.impactclass,
|
|
'createdat': _utciso(self.createdat),
|
|
}
|