ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add log collection, script output capture, and timestamp fixes

Browse Source 
Log Collector: a new collect_log action reads a plaintext log file from a
known path and returns its tail for viewing and download. Rides the existing
endpoints.updateapp grant rather than inventing one, since collecting a log is
the same trust level as pushing an application. Collected logs land in their
own table with per-asset retention of 20 instead of overloading the command
audit row, and the agent refuses UNC, traversal, wildcards, directories and
non-text content independently of the server check.

collect_log is POLICY_EXEMPT. Every existing Configuration Policy predates the
action and so cannot name it, and the intersection in ALLOWED_ACTIONS_FOR drops
anything unnamed - so every policied endpoint refused log collection outright,
including for admins, who saw a permission-shaped error for a grant they held.
Exemption from policy is not exemption from permission: the elevated grant
still applies, and a regression test asserts both halves so a future edit
cannot quietly loosen the grant while "fixing" the policy gate.

Script output capture: successful runs can return output, gated per script by
captureoutput. Failures always returned it; success never did, which left a
green badge as the only evidence a run produced.

Timestamps: Activity and Operator Reports rendered date-only because the shared
formatter's 'date' branch drops the clock time. Run Scripts additionally
labels its timestamp, since a bare time meant "completed" on a terminal row but
"queued" on an in-flight one - indistinguishable, and the wrong reading.

Agent 1.16.0. Minor: collect_log is a new action name, so an older agent asked
for it fails closed on the unknown-action default.
This commit is contained in:
benjamin.baker
2026-09-09 19:23:43 -04:00
parent 494b837bae
commit b49fb60c0c
19 changed files with 2207 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
plugins/endpoints/models/endpoint.py
View File

@@ -16,13 +16,23 @@ Two shapes, one per concern:
crash) so this stays a readable security/stability history
rather than a raw event firehose.
EndpointLogCollection
A plaintext log file collected from an endpoint on operator
demand. Kept out of EndpointCommand.resultjson because it is
bulky, needs its own per-asset retention cap, and has to be
addressable by id for the download route.
Both link to the core Asset via assetid (FK, ON DELETE CASCADE), so an endpoint
is the same physical thing as its Computer/Asset - no duplicate identity. The
agent reports a hostname; the plugin resolves it to the existing Computer.
"""
import json
import re
from datetime import datetime
import sqlalchemy as sa
from shopdb.api import db
@@ -275,8 +285,16 @@ class EndpointCommand(db.Model):
def __repr__(self):
return f"<EndpointCommand {self.action} asset={self.assetid} {self.status}>"
def to_dict(self):
return {
def to_dict(self, includedetail=False):
"""Serialise a command for the operator views.
resultjson is opt-in. It is agent-reported structured detail, and for a
script it carries the evidence that a green badge alone does not: the
exit code, the context it ran as and the directory it really ran in. On
failure the agent also puts the captured output in there, so it is not
served to every list that only needs a status.
"""
data = {
'endpointcommandid': self.endpointcommandid,
'assetid': self.assetid,
'action': self.action,
@@ -289,6 +307,24 @@ class EndpointCommand(db.Model):
'claimedat': _utciso(self.claimedat),
'completedat': _utciso(self.completedat),
}
if includedetail:
data['resultdetail'] = self._resultdetail()
return data
def _resultdetail(self):
"""resultjson as a dict, or None.
Stored as text, so a malformed or truncated payload is possible. A
detail panel is diagnostics: failing to parse it must not break the
command list that surrounds it.
"""
if not self.resultjson:
return None
try:
parsed = json.loads(self.resultjson)
except (TypeError, ValueError):
return None
return parsed if isinstance(parsed, dict) else None
class EndpointEvent(db.Model):
@@ -531,3 +567,130 @@ class EndpointLock(db.Model):
'releasedat': _utciso(self.releasedat),
'releasedbyusername': self.releasedbyusername,
}
class EndpointLogCollection(db.Model):
"""One plaintext log file collected from an endpoint by an operator.
Collected content lives here rather than in EndpointCommand.resultjson for
three reasons. It is bulky and would be dragged along by every command
history list that only wants a status; it needs its own retention rule
(RETENTION_PER_ASSET), which a command row cannot express without deleting
the audit record of the command itself; and the download route needs to
serve bytes, which means it needs a row it can address by id rather than a
JSON field it has to re-parse.
Retention is deliberately PER ASSET. A global cap would let one busy machine
evict another machine's history, and the history is read per machine on the
PC detail page, so a global cap would make the panel unreliable exactly
where it is used. Pruning happens on insert in the result route.
The content stored here is a TAIL, not necessarily the whole file - see
istruncated. A log that has run for months is not something to load into a
browser pane, and the agent bounds what it sends.
"""
__tablename__ = 'endpointlogcollections'
# How many collections to keep per asset. The operator asked for 20: enough
# to compare a handful of runs against each other, few enough that stored
# log text does not grow without bound across a fleet.
RETENTION_PER_ASSET = 20
endpointlogcollectionid = db.Column(db.Integer, primary_key=True)
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
nullable=False,
index=True,
)
# The command that produced this, kept for audit: it ties the collected
# bytes back to who queued the collection and when it ran. SET NULL rather
# than CASCADE so pruning or expiring a command never silently destroys a
# log an operator is still looking at.
endpointcommandid = db.Column(
db.Integer,
db.ForeignKey('endpointcommands.endpointcommandid', ondelete='SET NULL'),
nullable=True,
)
# The path as the AGENT resolved it, which is not always the path the
# operator typed (environment variables get expanded endpoint-side). Storing
# the resolved one is what makes the history honest about what was read.
sourcepath = db.Column(db.String(512), nullable=False)
content = db.Column(db.Text(length=16777215), nullable=True)
# Size of the file ON DISK, not the size of content. When the tail is
# truncated these differ, and that difference is the whole point: it tells
# the operator they are looking at the end of something larger.
filesizebytes = db.Column(db.BigInteger, nullable=True)
contentbytes = db.Column(db.Integer, nullable=True)
linecount = db.Column(db.Integer, nullable=True)
istruncated = db.Column(db.Boolean, nullable=False,
default=False, server_default=sa.false())
# Last-write time reported by the endpoint. A log that has not changed since
# the previous collection is the fastest way to see an app is not logging.
filemodifiedat = db.Column(db.DateTime, nullable=True)
# Username snapshot alongside the id: the id alone becomes unreadable if the
# user is later removed, and this history is read long after collection.
collectedbyuserid = db.Column(
db.Integer,
db.ForeignKey('users.userid', ondelete='SET NULL'),
nullable=True,
)
collectedbyusername = db.Column(db.String(100), nullable=True)
collectedat = db.Column(db.DateTime, nullable=False,
default=datetime.utcnow, index=True)
__table_args__ = (
# Serves both hot paths: listing an asset's history newest-first, and
# finding the oldest rows to prune on insert.
db.Index('idx_endpointlogcollections_assetcollected',
'assetid', 'collectedat'),
)
def __repr__(self):
return (f"<EndpointLogCollection asset={self.assetid} "
f"{self.sourcepath}>")
@property
def filename(self):
"""A download filename derived from the collected path."""
raw = (self.sourcepath or '').replace('/', '\\')
base = raw.rsplit('\\', 1)[-1].strip() or 'collected.log'
# Keep it to characters that survive a Content-Disposition header and a
# Windows save dialog without quoting games.
safe = re.sub(r'[^A-Za-z0-9._-]', '_', base)[:100]
stamp = (self.collectedat or datetime.utcnow()).strftime('%Y%m%d-%H%M%S')
return f'{stamp}_{safe}'
def to_dict(self, includecontent=False):
"""Serialise a collection.
Content is opt-in for the same reason resultjson is on EndpointCommand:
the history list renders 20 rows and none of them need the log body,
which may be hundreds of kilobytes each.
"""
data = {
'endpointlogcollectionid': self.endpointlogcollectionid,
'assetid': self.assetid,
'endpointcommandid': self.endpointcommandid,
'sourcepath': self.sourcepath,
'filesizebytes': self.filesizebytes,
'contentbytes': self.contentbytes,
'linecount': self.linecount,
'istruncated': bool(self.istruncated),
'filemodifiedat': _utciso(self.filemodifiedat),
'collectedbyuserid': self.collectedbyuserid,
'collectedbyusername': self.collectedbyusername,
'collectedat': _utciso(self.collectedat),
'filename': self.filename,
}
if includecontent:
data['content'] = self.content or ''
return data