ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make main the lean product a site receives

Browse Source 
main was 16 days and 259 commits behind the line the product is actually built
on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It
was neither current nor curated - just stale. This brings it forward and, at the
same time, makes it mean something: main is what a receiving site installs, and
nothing else.

WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides,
knowledgebase, machines, network, notifications, printers, slides, tools,
warranty - plus applications, which is manifest-less core and always ships. The
only manifest dependency inside that set is endpoints on computers, and both are
in it, so the closure is self-contained.

WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb
stay on the site line. So do the Windows installer tree, the VM rehearsal
helpers, the agent tooling config, one site's data loader, the internal audit
and findings records, and the planner spreadsheet. 1,135 tracked files against
1,438.

A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped:

deploy/site-profile-universal.json still named all seventeen plugins. It is the
file docs/DEPLOY.md tells a site to provision from, so a fresh install would
have failed at `flask plugin apply-profile` with PluginError - the first thing a
receiving site does, broken by the branch meant to serve them. It now names the
twelve that exist, and its comment says the two must agree.

agent/ was never refreshed. Whoever populated this branch copied shopdb/,
plugins/, frontend/ and tests/ from the site line and left the agent tree at its
September state: 22 C# and PowerShell sources missing, including AgentCredentials,
EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the
site line had deleted still present. The old tree is internally consistent, so it
compiles, and nothing in CI builds it - every gate was green over a server that
speaks enrollment and per-PC credentials to a bundled agent source that cannot.
Resynced whole.

DECISIONS THIS ENCODES, rather than defers:

- The plugin-authoring lab builds printedparts, whose manifest depends on
  employees, so it could not stay. The lab remains on the site line and main
  ships no build-along tutorial. That is a real loss for an adopting developer
  and is recorded here rather than discovered later.
- Recognition, Training and Recertification notifications are no longer seeded.
  Each splits one notification into a card per employee and renders a photo,
  which needs a plugin this branch does not bundle. Seeding them would offer an
  author a type that cannot resolve anybody. A site that installs employees gets
  them from that plugin's own seed.
- Windows/IIS is documented as the legacy path throughout, and the README leads
  with Docker, because that is what a receiving site gets.

The API description is regenerated for THIS branch - 347 paths and 483
operations, down from 415 and 573 - so the published contract describes what is
here rather than what the site line serves. docs/CONFIG.md lost the two
measuring-tool label keys the code no longer declares, which two lanes had
decided opposite ways.

Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run
rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
cproudlock
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
plugins/endpoints/models/__init__.py
View File

@@ -2,11 +2,23 @@
from .endpoint import (
EndpointState, EndpointEvent, EndpointCommand, EndpointApplication,
EndpointLock, EndpointLogCollection,
)
from .monitor import EndpointMonitor
from .operatorreport import OperatorReport
from .credential import (
EndpointCredential, EndpointEnrollRequest, EndpointMigrationWave,
)
from .bootstrap import EndpointBootstrapToken
from .health import (
EndpointHealthState, EndpointHealthTransition, EndpointMonitorImpact,
)
__all__ = [
'EndpointState', 'EndpointEvent', 'EndpointCommand', 'EndpointApplication',
'EndpointLock', 'EndpointLogCollection',
'EndpointMonitor', 'OperatorReport',
'EndpointCredential', 'EndpointEnrollRequest', 'EndpointMigrationWave',
'EndpointBootstrapToken',
'EndpointHealthState', 'EndpointHealthTransition', 'EndpointMonitorImpact',
]

78
plugins/endpoints/models/bootstrap.py Normal file
View File

@@ -0,0 +1,78 @@
"""One-line bootstrap install tokens.
An admin mints a short-lived token in the UI and hands a tech one command. The
token is only ever an authorization to FETCH the bootstrap script and the
keyless agent package: it carries no fleet key, provisions no credential, and
the PC it installs still has to be approved in Settings > Endpoints >
Enrollment before it can report anything.
Only the sha256 of the secret is stored, plus a short display prefix, the same
shape as EndpointCredential. A token is spent per package download (usecount
against maxuses) and dies at expiresat or when an admin revokes it.
"""
from datetime import datetime
from shopdb.api import db
def _utciso(value):
return value.isoformat() + 'Z' if value else None
class EndpointBootstrapToken(db.Model):
"""A short-lived authorization to fetch the bootstrap installer."""
__tablename__ = 'endpointbootstraptokens'
tokenid = db.Column(db.Integer, primary_key=True)
tokenhash = db.Column(db.CHAR(64), nullable=False, unique=True)
tokenprefix = db.Column(db.String(12), nullable=False)
createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
createdby = db.Column(db.String(64), nullable=True)
expiresat = db.Column(db.DateTime, nullable=False)
maxuses = db.Column(db.Integer, nullable=False, default=1)
usecount = db.Column(db.Integer, nullable=False, default=0,
server_default='0')
revokedat = db.Column(db.DateTime, nullable=True)
lastusedat = db.Column(db.DateTime, nullable=True)
lastusedip = db.Column(db.String(45), nullable=True)
def isusable(self, now):
"""Whether this token may still serve a download at `now`."""
if self.revokedat is not None:
return False
if self.expiresat is None or self.expiresat <= now:
return False
return (self.usecount or 0) < (self.maxuses or 0)
def state(self, now):
if self.revokedat is not None:
return 'revoked'
if self.expiresat is None or self.expiresat <= now:
return 'expired'
if (self.usecount or 0) >= (self.maxuses or 0):
return 'exhausted'
return 'active'
def to_dict(self, now=None):
now = now or datetime.utcnow()
used = self.usecount or 0
maximum = self.maxuses or 0
return {
'tokenid': self.tokenid,
'tokenprefix': self.tokenprefix,
'createdat': _utciso(self.createdat),
'createdby': self.createdby,
'expiresat': _utciso(self.expiresat),
'maxuses': maximum,
'usecount': used,
'remaining': max(maximum - used, 0),
'revokedat': _utciso(self.revokedat),
'lastusedat': _utciso(self.lastusedat),
'lastusedip': self.lastusedip,
'state': self.state(now),
}
def __repr__(self):
return f'<EndpointBootstrapToken {self.tokenprefix}>'

165
plugins/endpoints/models/credential.py Normal file
View File

@@ -0,0 +1,165 @@
"""Per-PC endpoint credentials, enrollment requests and migration waves.
EndpointCredential
One row per asset. Holds only the sha256 of the secret the
agent was issued, plus a short display prefix. Revoking sets
revokedat; re-issuing after a revoke UPDATES the same row, so
an asset never carries two credentials. The row therefore also
says that this PC has enrolled at least once, which is how
agentauth tells a revoked PC from one that never enrolled:
plugin.endpoints.revokeddeniessharedkey refuses the shared key
to the first and never to the second. reenrollblocked keeps
a revoked PC from enrolling again (rolling it back to an agent
that only knows the shared key, or a compromised PC) until an
admin allows it.
EndpointEnrollRequest
One row per (hostname, serial number) that asked to enroll -
the endpoint's identity, and the reason every contact from one
PC lands on one row. The agent's enrollment nonce is recorded
on first sight and is what an admin approval is bound to, so a
second machine claiming the same hostname and serial cannot
collect the approved credential. A REJECTED row binds to the
identity instead: any nonce from that hostname and serial stays
rejected, so a reinstall cannot argue its way back into the
queue. Only an admin approving the request clears it.
EndpointMigrationWave
An admin action that pushed the agent upgrade to a set of PCs.
Progress is computed live from credentials and commands; only
the membership is stored.
"""
from datetime import datetime
from shopdb.api import db
def _utciso(value):
return value.isoformat() + 'Z' if value else None
class EndpointCredential(db.Model):
"""The credential issued to one endpoint (hash only)."""
__tablename__ = 'endpointcredentials'
ENROLLED_AUTO = 'auto'
ENROLLED_APPROVED = 'approved'
credentialid = db.Column(db.Integer, primary_key=True)
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
unique=True,
)
credentialhash = db.Column(db.CHAR(64), nullable=False, unique=True)
credentialprefix = db.Column(db.String(12), nullable=False)
enrolledvia = db.Column(db.String(16), nullable=False)
issuedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
lastusedat = db.Column(db.DateTime, nullable=True)
revokedat = db.Column(db.DateTime, nullable=True)
revokedby = db.Column(db.String(64), nullable=True)
reenrollblocked = db.Column(db.Boolean, nullable=False, default=False,
server_default=db.false())
reenrollblockedreason = db.Column(db.String(64), nullable=True)
@property
def isactive(self):
return self.revokedat is None
def to_dict(self):
return {
'credentialid': self.credentialid,
'assetid': self.assetid,
'credentialprefix': self.credentialprefix,
'enrolledvia': self.enrolledvia,
'issuedat': _utciso(self.issuedat),
'lastusedat': _utciso(self.lastusedat),
'revokedat': _utciso(self.revokedat),
'revokedby': self.revokedby,
'reenrollblocked': bool(self.reenrollblocked),
'reenrollblockedreason': self.reenrollblockedreason,
}
def __repr__(self):
return f'<EndpointCredential asset={self.assetid} {self.credentialprefix}>'
class EndpointEnrollRequest(db.Model):
"""A PC asking for a credential."""
__tablename__ = 'endpointenrollrequests'
STATUS_PENDING = 'pending'
STATUS_APPROVED = 'approved'
STATUS_REJECTED = 'rejected'
STATUS_ISSUED = 'issued'
requestid = db.Column(db.Integer, primary_key=True)
hostname = db.Column(db.String(255), nullable=False)
serialnumber = db.Column(db.String(128), nullable=False)
deviceid = db.Column(db.String(64), nullable=True)
enrollnonce = db.Column(db.String(128), nullable=False)
agentversion = db.Column(db.String(32), nullable=True)
status = db.Column(db.String(16), nullable=False, default=STATUS_PENDING)
reason = db.Column(db.String(64), nullable=True)
matchedassetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='SET NULL'),
nullable=True,
)
sourceip = db.Column(db.String(45), nullable=True)
requestedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
lastseenat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
decidedat = db.Column(db.DateTime, nullable=True)
decidedby = db.Column(db.String(64), nullable=True)
# Prefix lengths keep the key under 767 bytes on utf8mb4 (4 bytes a char).
__table_args__ = (
db.Index('idx_endpointenrollrequests_hostserial', 'hostname',
'serialnumber',
mysql_length={'hostname': 128, 'serialnumber': 63}),
)
def to_dict(self):
return {
'requestid': self.requestid,
'hostname': self.hostname,
'serialnumber': self.serialnumber,
'deviceid': self.deviceid,
'agentversion': self.agentversion,
'reason': self.reason,
'status': self.status,
'matchedassetid': self.matchedassetid,
'requestedat': _utciso(self.requestedat),
'lastseenat': _utciso(self.lastseenat),
'decidedat': _utciso(self.decidedat),
'decidedby': self.decidedby,
'sourceip': self.sourceip,
}
def __repr__(self):
return f'<EndpointEnrollRequest {self.hostname!r} {self.status}>'
class EndpointMigrationWave(db.Model):
"""One batch of agent upgrades pushed to move PCs onto credentials."""
__tablename__ = 'endpointmigrationwaves'
waveid = db.Column(db.Integer, primary_key=True)
createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
createdby = db.Column(db.String(64), nullable=True)
# Comma list of asset ids. TEXT, not JSON: MySQL 5.6.
assetids = db.Column(db.Text, nullable=False, default='')
notes = db.Column(db.String(255), nullable=True)
@property
def assetidlist(self):
return [int(part) for part in (self.assetids or '').split(',')
if part.strip().isdigit()]
def __repr__(self):
return f'<EndpointMigrationWave {self.waveid}>'

334
plugins/endpoints/models/endpoint.py
View File

@@ -16,13 +16,23 @@ Two shapes, one per concern:
crash) so this stays a readable security/stability history
rather than a raw event firehose.
EndpointLogCollection
A plaintext log file collected from an endpoint on operator
demand. Kept out of EndpointCommand.resultjson because it is
bulky, needs its own per-asset retention cap, and has to be
addressable by id for the download route.
Both link to the core Asset via assetid (FK, ON DELETE CASCADE), so an endpoint
is the same physical thing as its Computer/Asset - no duplicate identity. The
agent reports a hostname; the plugin resolves it to the existing Computer.
"""
import json
import re
from datetime import datetime
import sqlalchemy as sa
from shopdb.api import db
@@ -80,6 +90,65 @@ class EndpointState(db.Model):
interfacesjson = db.Column(db.Text(length=16777215), nullable=True)
scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True)
# Hardware and OS inventory, same newest-snapshot rule as above. These
# arrived on the wire well before this and were logged as unknown keys, so
# the shapes are already proven against real endpoint traffic.
disksjson = db.Column(db.Text(length=16777215), nullable=True)
biosjson = db.Column(db.Text(length=16777215), nullable=True)
memoryjson = db.Column(db.Text(length=16777215), nullable=True)
processorjson = db.Column(db.Text(length=16777215), nullable=True)
osinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Session-scoped data collected by a probe running as the signed-in user
# (mapped drives, user-owned listeners). Null when nobody is signed in,
# which is a normal unattended state rather than a fault.
usercontextjson = db.Column(db.Text(length=16777215), nullable=True)
# Windows domain or workgroup membership. A scalar, not JSON: it is a single
# value the fleet view filters and groups on.
domain = db.Column(db.String(255), nullable=True)
# True when the endpoint is holding a reboot. Null = never reported, which
# is deliberately distinct from a confident False.
pendingreboot = db.Column(db.Boolean, nullable=True)
# Per-drive health from the agent's two disk sources. JSON because a machine
# has a variable number of drives and each carries its own verdict. Kept
# separate from disksjson: that is capacity, this is whether the hardware is
# dying. Null = never reported.
diskhealthjson = db.Column(db.Text(length=16777215), nullable=True)
# Which signal(s) drove pendingreboot. That column is a single bool OR-ed
# from three sources and so cannot say WHY, and the difference matters: a
# pendingfilerename means files are staged for replacement on next boot, so
# the box is mid-servicing, while a lone servicing key can sit set for weeks
# on a machine nobody rebooted. Empty list = checked, nothing pending.
# Null = never reported or the check failed.
rebootreasonsjson = db.Column(db.Text(length=16777215), nullable=True)
# Fast startup configuration and what the last boot actually did. Kept
# because uptime is not trustworthy under fast startup: a box can report
# days of uptime after what the operator experienced as a shutdown, and a
# "reboot" that was a hybrid resume does not clear the state a support call
# is usually trying to clear.
bootinfojson = db.Column(db.Text(length=16777215), nullable=True)
# Defender health. Persisted but deliberately NOT surfaced anywhere yet:
# this environment produces known false positives, and wiring it to a card
# now would train operators to ignore that card. Stored so the history is
# already accumulating when someone is ready to interpret it.
defenderjson = db.Column(db.Text(length=16777215), nullable=True)
# Pending Dell Command | Update items, produced by the shopdb-dellupdates
# agent utility from the log the \ShopDB-DellCommandScan task writes. Only
# the count drives the status card; the short list is stored so the panel
# can show what is actually pending without a second round trip.
#
# A null column and a stored payload with status 'unknown' both mean "never
# scanned", which stays distinct from a scan that found nothing pending. The
# status card relies on that distinction - see _dell_updates_field.
dellupdatesjson = db.Column(db.Text(length=16777215), nullable=True)
# Warn-only reconciler output, refreshed on every agent check-in in the same
# commit as servicesjson. Each entry names a policy/shipped-attached monitor
# the endpoint is expected to be running but the latest snapshot does not
@@ -116,6 +185,8 @@ class EndpointState(db.Model):
'idleseconds': self.idleseconds,
'cpupercent': self.cpupercent,
'memorypercent': self.memorypercent,
'domain': self.domain,
'pendingreboot': self.pendingreboot,
'mtconnectrunning': self.mtconnectrunning,
'mtconnectcheckedat': _utciso(self.mtconnectcheckedat),
'lastreportedat': _utciso(self.lastreportedat),
@@ -214,8 +285,16 @@ class EndpointCommand(db.Model):
def __repr__(self):
return f"<EndpointCommand {self.action} asset={self.assetid} {self.status}>"
def to_dict(self):
return {
def to_dict(self, includedetail=False):
"""Serialise a command for the operator views.
resultjson is opt-in. It is agent-reported structured detail, and for a
script it carries the evidence that a green badge alone does not: the
exit code, the context it ran as and the directory it really ran in. On
failure the agent also puts the captured output in there, so it is not
served to every list that only needs a status.
"""
data = {
'endpointcommandid': self.endpointcommandid,
'assetid': self.assetid,
'action': self.action,
@@ -228,6 +307,24 @@ class EndpointCommand(db.Model):
'claimedat': _utciso(self.claimedat),
'completedat': _utciso(self.completedat),
}
if includedetail:
data['resultdetail'] = self._resultdetail()
return data
def _resultdetail(self):
"""resultjson as a dict, or None.
Stored as text, so a malformed or truncated payload is possible. A
detail panel is diagnostics: failing to parse it must not break the
command list that surrounds it.
"""
if not self.resultjson:
return None
try:
parsed = json.loads(self.resultjson)
except (TypeError, ValueError):
return None
return parsed if isinstance(parsed, dict) else None
class EndpointEvent(db.Model):
@@ -255,6 +352,16 @@ class EndpointEvent(db.Model):
eventchannel = db.Column(db.String(50), nullable=True)
eventsource = db.Column(db.String(255), nullable=True)
# The Windows event record id: monotonic and unique within its channel on a
# given machine. This is the dedup key. The agent also uses it as its
# harvest watermark, so in normal operation a record is sent once; the
# unique index below is what guarantees it even when an agent is reinstalled
# (losing its watermark) or two reports overlap. Nullable because an agent
# older than this field still reports, and MySQL permits repeated NULLs in a
# unique index, so those degrade to the previous append-only behavior rather
# than failing.
eventrecordid = db.Column(db.BigInteger, nullable=True)
# When the event happened on the endpoint (from the record), not when it
# reached ShopDB.
eventtime = db.Column(db.DateTime, nullable=True, index=True)
@@ -269,6 +376,8 @@ class EndpointEvent(db.Model):
__table_args__ = (
db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'),
db.Index('idx_endpointevents_assetrecord',
'assetid', 'eventchannel', 'eventrecordid', unique=True),
)
def __repr__(self):
@@ -279,6 +388,7 @@ class EndpointEvent(db.Model):
'endpointeventid': self.endpointeventid,
'assetid': self.assetid,
'eventid': self.eventid,
'eventrecordid': self.eventrecordid,
'eventchannel': self.eventchannel,
'eventsource': self.eventsource,
'eventtime': _utciso(self.eventtime),
@@ -354,8 +464,15 @@ class EndpointApplication(db.Model):
# When this snapshot row was written (the report that produced it).
recordedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
# SHA256 fingerprint for delta-based updates: hash of displayname, displayversion,
# publisher. Allows efficient comparison to detect changes without comparing all
# columns. Computed by agent and sent in payload. Nullable for backward compatibility
# during migration - backfill script will populate existing rows.
appfingerprint = db.Column(db.String(64), nullable=True)
__table_args__ = (
db.Index('idx_endpointapplications_assetname', 'assetid', 'displayname'),
db.Index('idx_endpointapplications_fingerprint', 'assetid', 'appfingerprint', unique=True),
)
def __repr__(self):
@@ -371,3 +488,216 @@ class EndpointApplication(db.Model):
'appid': self.appid,
'recordedat': _utciso(self.recordedat),
}
class EndpointLock(db.Model):
"""An admin-only quarantine refusing operator commands on one endpoint.
Deliberately NOT the same thing as a PolicyActionHold, despite the similar
name. The two sit on different axes and collapsing them would recreate the
axis-mixing this refactor exists to undo:
PolicyActionHold policy-resolution axis. Suppresses ONE desired-state
action so reconciliation stops fighting an admin.
Per-action, several may stand at once, and it EXPIRES.
EndpointLock caller-authority axis. Refuses commands from non-admins
on this endpoint regardless of policy or permission.
Per-endpoint, one at a time, and INDEFINITE.
Indefinite is the point. A lock represents a physical-world condition - a
machine mid-repair, a validated part running, a box that must not reboot
during a customer audit - and those do not end on a timer. A TTL here would
quietly unlock a machine someone is still working inside, which is the exact
accident the lock exists to prevent. Stale locks are surfaced in the UI
instead of being auto-cleared.
Released rows are kept and marked rather than deleted: the record of who
quarantined a machine, why, and who lifted it is the audit trail, and it is
most wanted precisely when the lock is no longer there to explain itself.
"""
__tablename__ = 'endpointlocks'
endpointlockid = db.Column(db.Integer, primary_key=True)
# Not unique at the schema level: released rows accumulate per asset and a
# unique constraint would make the second lock in an endpoint's life fail.
# The one-active-lock invariant is enforced on write in PLACE_ENDPOINT_LOCK.
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
index=True,
)
reason = db.Column(db.Text, nullable=False)
# Who set it. The id goes null if the account is deleted, but the username
# snapshot survives: a quarantine that forgets who imposed it is unusable at
# review time, which is the only time anyone reads it.
lockedbyuserid = db.Column(
db.Integer,
db.ForeignKey('users.userid', ondelete='SET NULL'),
nullable=True,
)
lockedbyusername = db.Column(db.String(100), nullable=True)
lockedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
# Null while the lock stands. Set on release, which is what makes this row
# inactive - there is no isactive flag to drift out of step with them.
releasedat = db.Column(db.DateTime, nullable=True)
releasedbyuserid = db.Column(
db.Integer,
db.ForeignKey('users.userid', ondelete='SET NULL'),
nullable=True,
)
releasedbyusername = db.Column(db.String(100), nullable=True)
__table_args__ = (
# Serves the hot path: "is there an active lock on this asset", asked on
# every enqueue and every detail page load.
db.Index('idx_endpointlocks_assetactive', 'assetid', 'releasedat'),
)
def __repr__(self):
state = 'released' if self.releasedat else 'active'
return f"<EndpointLock asset={self.assetid} {state}>"
def to_dict(self):
return {
'endpointlockid': self.endpointlockid,
'assetid': self.assetid,
'reason': self.reason,
'lockedbyuserid': self.lockedbyuserid,
'lockedbyusername': self.lockedbyusername,
'lockedat': _utciso(self.lockedat),
'releasedat': _utciso(self.releasedat),
'releasedbyusername': self.releasedbyusername,
}
class EndpointLogCollection(db.Model):
"""One plaintext log file collected from an endpoint by an operator.
Collected content lives here rather than in EndpointCommand.resultjson for
three reasons. It is bulky and would be dragged along by every command
history list that only wants a status; it needs its own retention rule
(RETENTION_PER_ASSET), which a command row cannot express without deleting
the audit record of the command itself; and the download route needs to
serve bytes, which means it needs a row it can address by id rather than a
JSON field it has to re-parse.
Retention is deliberately PER ASSET. A global cap would let one busy machine
evict another machine's history, and the history is read per machine on the
PC detail page, so a global cap would make the panel unreliable exactly
where it is used. Pruning happens on insert in the result route.
The content stored here is a TAIL, not necessarily the whole file - see
istruncated. A log that has run for months is not something to load into a
browser pane, and the agent bounds what it sends.
"""
__tablename__ = 'endpointlogcollections'
# How many collections to keep per asset. The operator asked for 20: enough
# to compare a handful of runs against each other, few enough that stored
# log text does not grow without bound across a fleet.
RETENTION_PER_ASSET = 20
endpointlogcollectionid = db.Column(db.Integer, primary_key=True)
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
index=True,
)
# The command that produced this, kept for audit: it ties the collected
# bytes back to who queued the collection and when it ran. SET NULL rather
# than CASCADE so pruning or expiring a command never silently destroys a
# log an operator is still looking at.
endpointcommandid = db.Column(
db.Integer,
db.ForeignKey('endpointcommands.endpointcommandid', ondelete='SET NULL'),
nullable=True,
)
# The path as the AGENT resolved it, which is not always the path the
# operator typed (environment variables get expanded endpoint-side). Storing
# the resolved one is what makes the history honest about what was read.
sourcepath = db.Column(db.String(512), nullable=False)
content = db.Column(db.Text(length=16777215), nullable=True)
# Size of the file ON DISK, not the size of content. When the tail is
# truncated these differ, and that difference is the whole point: it tells
# the operator they are looking at the end of something larger.
filesizebytes = db.Column(db.BigInteger, nullable=True)
contentbytes = db.Column(db.Integer, nullable=True)
linecount = db.Column(db.Integer, nullable=True)
istruncated = db.Column(db.Boolean, nullable=False,
default=False, server_default=sa.false())
# Last-write time reported by the endpoint. A log that has not changed since
# the previous collection is the fastest way to see an app is not logging.
filemodifiedat = db.Column(db.DateTime, nullable=True)
# Username snapshot alongside the id: the id alone becomes unreadable if the
# user is later removed, and this history is read long after collection.
collectedbyuserid = db.Column(
db.Integer,
db.ForeignKey('users.userid', ondelete='SET NULL'),
nullable=True,
)
collectedbyusername = db.Column(db.String(100), nullable=True)
collectedat = db.Column(db.DateTime, nullable=False,
default=datetime.utcnow, index=True)
__table_args__ = (
# Serves both hot paths: listing an asset's history newest-first, and
# finding the oldest rows to prune on insert.
db.Index('idx_endpointlogcollections_assetcollected',
'assetid', 'collectedat'),
)
def __repr__(self):
return (f"<EndpointLogCollection asset={self.assetid} "
f"{self.sourcepath}>")
@property
def filename(self):
"""A download filename derived from the collected path."""
raw = (self.sourcepath or '').replace('/', '\\')
base = raw.rsplit('\\', 1)[-1].strip() or 'collected.log'
# Keep it to characters that survive a Content-Disposition header and a
# Windows save dialog without quoting games.
safe = re.sub(r'[^A-Za-z0-9._-]', '_', base)[:100]
stamp = (self.collectedat or datetime.utcnow()).strftime('%Y%m%d-%H%M%S')
return f'{stamp}_{safe}'
def to_dict(self, includecontent=False):
"""Serialise a collection.
Content is opt-in for the same reason resultjson is on EndpointCommand:
the history list renders 20 rows and none of them need the log body,
which may be hundreds of kilobytes each.
"""
data = {
'endpointlogcollectionid': self.endpointlogcollectionid,
'assetid': self.assetid,
'endpointcommandid': self.endpointcommandid,
'sourcepath': self.sourcepath,
'filesizebytes': self.filesizebytes,
'contentbytes': self.contentbytes,
'linecount': self.linecount,
'istruncated': bool(self.istruncated),
'filemodifiedat': _utciso(self.filemodifiedat),
'collectedbyuserid': self.collectedbyuserid,
'collectedbyusername': self.collectedbyusername,
'collectedat': _utciso(self.collectedat),
'filename': self.filename,
}
if includecontent:
data['content'] = self.content or ''
return data

267
plugins/endpoints/models/health.py Normal file
View File

@@ -0,0 +1,267 @@
"""Endpoint health rollup: cached verdict, transition history, impact mapping.
Three tables, one job: decide what the Fleet Health column says about an
endpoint, decide it ONCE when the data changes, and keep enough history to
answer "when did this break".
Why this exists at all. The verdict used to be computed on every Fleet page
load by aggregating 48h of endpointevents fleet-wide, which cost more the
longer the fleet ran, and it ranked a raw Windows event count above every
other signal. A box with a crash loop reported "Application crash x5861" while
its dead eDNC service - the thing that actually stops work on the floor - was
structurally unreachable, because the event branch returned first.
The model here separates two things the old rollup conflated:
events Point-in-time facts. They happened, they are counted, they age
out of the window on their own.
conditions A stopped service or a disconnected drive is not an event, it is
a STATE with a duration. It opens, it persists, and it is only
over when something observes it to be over.
Ranking a condition against an event count is comparing incompatible things.
Conditions are pinned until explicitly resolved; events are ranked beneath
them. That is the whole reason the Health column can be trusted now.
"""
from datetime import datetime
from shopdb.api import db
def _utciso(value):
"""Naive-UTC datetime -> ISO string marked UTC, or None."""
return value.isoformat() + 'Z' if value else None
# Operational impact, in the site's terms rather than the OS's. Windows calls a
# crash severe; the floor calls a disconnected drive a stoppage. These disagree,
# and the floor wins - which is why this vocabulary is not the event catalog's
# 'critical'/'warning'/'info'.
IMPACT_BLOCKER = 'blocker'
IMPACT_DEGRADED = 'degraded'
IMPACT_INFORMATIONAL = 'informational'
IMPACT_CLASSES = (IMPACT_BLOCKER, IMPACT_DEGRADED, IMPACT_INFORMATIONAL)
# Lower sorts worse, matching _DRIVE_STATUS_RANK's existing convention so the
# two ladders in this plugin do not read in opposite directions.
IMPACT_RANK = {
IMPACT_BLOCKER: 0,
IMPACT_DEGRADED: 1,
IMPACT_INFORMATIONAL: 2,
}
# Where a verdict came from. Kept alongside sourceref so the UI can deep-link
# back to the monitor or the event that produced the label.
SOURCE_DRIVE = 'drive'
SOURCE_SERVICE = 'service'
SOURCE_PROCESS = 'process'
SOURCE_EVENT = 'event'
SOURCE_NONE = 'none'
class EndpointHealthState(db.Model):
"""The current health verdict for one endpoint - the Fleet page's only read.
Cached deliberately. The Fleet list joins this and reads it directly; it
does not recompute anything per row. Written in the SAME commit as the
check-in that produced it, so the cached verdict and the telemetry it
reflects can never disagree (the ledger pattern this codebase already uses
for quantityonhand).
"""
__tablename__ = 'endpointhealthstates'
endpointhealthstateid = db.Column(db.Integer, primary_key=True)
# One row per asset. Unique rather than a plain index: two live verdicts
# for one endpoint is not a state this table should be able to represent.
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
unique=True,
)
# Display vocabulary, matching what the Fleet renderer already understands:
# 'ok', 'warning', 'critical', 'unknown'.
level = db.Column(db.String(16), nullable=False, default='unknown')
# The phrase an operator reads ("eDNC stopped", "S: disconnected"). A
# PHRASE, not a score - it has to say what to do.
label = db.Column(db.String(255), nullable=True)
# Null for an ok/unknown verdict: nothing is wrong, so nothing has impact.
impactclass = db.Column(db.String(16), nullable=True)
sourcekind = db.Column(db.String(16), nullable=False, default=SOURCE_NONE)
# Monitor id, drive letter or Windows event id, depending on sourcekind.
# Free text because those identifier spaces have nothing in common.
sourceref = db.Column(db.String(64), nullable=True)
# When the CURRENT condition opened. Survives re-evaluation: a service that
# has been down for three days keeps its original firstseenat, so the UI can
# say how long rather than only that it is down now.
firstseenat = db.Column(db.DateTime, nullable=True)
# Last time an agent explicitly observed this endpoint. Drives staleness:
# a pinned blocker on an endpoint that stopped reporting is neither
# resolvable nor trustworthy, and must not render as present tense.
lastconfirmedat = db.Column(db.DateTime, nullable=True)
lastevaluatedat = db.Column(db.DateTime, nullable=False,
default=datetime.utcnow)
# Transitions inside the trailing window. A service flapping ten times an
# hour is a different fault from one cleanly down, and dispatching someone
# to a service that has already restarted twice wastes a trip.
flapcount = db.Column(db.Integer, nullable=False, default=0)
__table_args__ = (
db.Index('idx_endpointhealthstates_asset', 'assetid'),
db.Index('idx_endpointhealthstates_impactclass', 'impactclass'),
)
def __repr__(self):
return f"<EndpointHealthState asset={self.assetid} {self.level}>"
def to_dict(self):
return {
'assetid': self.assetid,
'level': self.level,
'label': self.label,
'impactclass': self.impactclass,
'sourcekind': self.sourcekind,
'sourceref': self.sourceref,
'firstseenat': _utciso(self.firstseenat),
'lastconfirmedat': _utciso(self.lastconfirmedat),
'lastevaluatedat': _utciso(self.lastevaluatedat),
'flapcount': self.flapcount,
}
class EndpointHealthTransition(db.Model):
"""One verdict CHANGE for an endpoint. Written on change, never on sample.
Storing a row per check-in would make the last 50 rows cover a few minutes
and say nothing. Storing a row per change makes 50 rows cover months on a
stable endpoint, compresses "nothing happened" to zero cost, and answers
the question operators actually ask - when did this break - directly rather
than by inferring it from a gap between samples.
It also makes flapping measurable: 50 transitions in an hour and 50 over a
year are the same row count and obviously different faults.
"""
__tablename__ = 'endpointhealthtransitions'
endpointhealthtransitionid = db.Column(db.Integer, primary_key=True)
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
)
level = db.Column(db.String(16), nullable=False)
label = db.Column(db.String(255), nullable=True)
impactclass = db.Column(db.String(16), nullable=True)
sourcekind = db.Column(db.String(16), nullable=False, default=SOURCE_NONE)
sourceref = db.Column(db.String(64), nullable=True)
openedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
# Null means STILL OPEN. Retention never evicts an unresolved row: at 50
# deep a long-running blocker's opening transition would otherwise age out
# behind a burst of unrelated churn, deleting the record of why the thing
# currently on screen is on screen.
resolvedat = db.Column(db.DateTime, nullable=True)
__table_args__ = (
db.Index('idx_endpointhealthtransitions_assetopened',
'assetid', 'openedat'),
db.Index('idx_endpointhealthtransitions_assetresolved',
'assetid', 'resolvedat'),
)
def __repr__(self):
return (f"<EndpointHealthTransition asset={self.assetid} "
f"{self.level} opened={self.openedat}>")
def to_dict(self):
return {
'endpointhealthtransitionid': self.endpointhealthtransitionid,
'assetid': self.assetid,
'level': self.level,
'label': self.label,
'impactclass': self.impactclass,
'sourcekind': self.sourcekind,
'sourceref': self.sourceref,
'openedat': _utciso(self.openedat),
'resolvedat': _utciso(self.resolvedat),
}
class EndpointMonitorImpact(db.Model):
"""How much a given monitor matters ON A GIVEN PC TYPE.
eDNC stopped on a shopfloor PC halts work; the same service stopped on a
lobby display is noise. Impact is therefore a property of the monitor IN
CONTEXT, not of the monitor and not of the event type - which is why it
cannot live on endpointmonitors and could never have lived in the event
catalog.
Why a plugin-owned side table rather than a column on the policy action:
monitors bind to PC types through configurationpolicyactions, which is a
CORE table shared by every plugin. Adding an endpoints-specific impact
column there would be a contract-surface change imposed on unrelated
plugins. This keeps the concept inside the plugin that owns it.
ABSENCE OF A ROW IS THE DEFAULT, and that is load-bearing. An endpoint with
no policies, no PC type, or no monitors at all resolves to the built-in
default impact for its source kind - the same verdict the old rollup gave.
The no-policy path is the NORMAL path here, not a special case.
"""
__tablename__ = 'endpointmonitorimpacts'
endpointmonitorimpactid = db.Column(db.Integer, primary_key=True)
endpointmonitorid = db.Column(
db.Integer,
db.ForeignKey('endpointmonitors.endpointmonitorid',
ondelete='CASCADE'),
nullable=False,
)
# Plain integer, not a ForeignKey: computertypes belongs to the computers
# plugin and owns its own migration chain. A cross-plugin FK would couple
# the two chains' ordering for no benefit the unique constraint below does
# not already provide.
computertypeid = db.Column(db.Integer, nullable=False)
impactclass = db.Column(db.String(16), nullable=False,
default=IMPACT_DEGRADED)
createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
__table_args__ = (
db.UniqueConstraint('endpointmonitorid', 'computertypeid',
name='uq_endpointmonitorimpacts_monitortype'),
db.Index('idx_endpointmonitorimpacts_computertypeid', 'computertypeid'),
)
def __repr__(self):
return (f"<EndpointMonitorImpact monitor={self.endpointmonitorid} "
f"pctype={self.computertypeid} {self.impactclass}>")
def to_dict(self):
return {
'endpointmonitorimpactid': self.endpointmonitorimpactid,
'endpointmonitorid': self.endpointmonitorid,
'computertypeid': self.computertypeid,
'impactclass': self.impactclass,
'createdat': _utciso(self.createdat),
}