add endpoints plugin with fleet list view and monitoring drafts
Server-side RMM telemetry plugin: current-snapshot endpointstates plus append-only curated endpointevents, resolved to existing Computer assets by hostname. Adds ADR-010 asset panels (status, services, recent events), dashboard quiet and stability cards, and a minimal Fleet list view that links into existing PC detail pages. Registers the activity sidebar icon and a dev-only sample seeder. Ingest arrives on the generic collector route; the future C# agent design record lives in docs/DESIGN-DECISIONS-agent.md.
This commit is contained in:
155
docs/DESIGN-DECISIONS-agent.md
Normal file
155
docs/DESIGN-DECISIONS-agent.md
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
# Agent Design Decisions (running preference list)
|
||||||
|
|
||||||
|
Server-side decisions made while building the `endpoints` plugin that the
|
||||||
|
future C# agent (and the ServiceNow ticketing surface) must honor. This is a
|
||||||
|
living document: add to it as the agent build raises questions. ASCII only, no
|
||||||
|
em-dashes, no emojis, per repo style.
|
||||||
|
|
||||||
|
## Scope of this document
|
||||||
|
|
||||||
|
The `endpoints` plugin (server) is the authoritative contract. The C# agent is
|
||||||
|
a separate deliverable in its own solution/repo. Anything the agent sends must
|
||||||
|
match what this plugin ingests; when the two disagree, this plugin wins and the
|
||||||
|
agent changes.
|
||||||
|
|
||||||
|
## 1. Transport and identity
|
||||||
|
|
||||||
|
- Agent posts one JSON object per host to `POST /api/collector/endpoints`
|
||||||
|
(the generic ADR-006 collector route, auto-registered by the core loader).
|
||||||
|
- Auth is a collector-scoped credential in the request HEADER, not a user JWT
|
||||||
|
and not a querystring. Server reads `COLLECTOR_API_KEY_ENDPOINTS` first, then
|
||||||
|
falls back to `COLLECTOR_API_KEY`. Provision the key the same way the
|
||||||
|
computers reporter does: out of band, e.g. `HKLM:\SOFTWARE\GE\ShopDB
|
||||||
|
CollectorKey`. The key is never baked into a distributed script/binary.
|
||||||
|
- Identity field is `hostname` (case-insensitive). The server resolves it to an
|
||||||
|
existing computer asset via the `computers` plugin. An unknown hostname is
|
||||||
|
REJECTED with a validation error, never auto-created. Decision: the fleet is
|
||||||
|
provisioned in ShopDB first (imports/Azure device sync); the agent does not
|
||||||
|
get to invent assets. Revisit only if bulk self-registration becomes a need.
|
||||||
|
|
||||||
|
## 2. Report shape (current server contract)
|
||||||
|
|
||||||
|
Top-level payload fields the server currently reads (see
|
||||||
|
`plugins/endpoints/plugin.py` `get_collector_schema`):
|
||||||
|
|
||||||
|
- `hostname` (required), `loggedinuser`, `agentversion`
|
||||||
|
- Metrics: `lastboottime` (ISO-8601), `uptimeseconds`, `idleseconds`,
|
||||||
|
`cpupercent`, `memorypercent`
|
||||||
|
- Inventory arrays (stored as latest-snapshot JSON, not history):
|
||||||
|
`services`, `ports`, `interfaces`, `scheduledtasks`
|
||||||
|
- `events`: array of curated Windows Event Log records (see section 4)
|
||||||
|
|
||||||
|
Decision: inventory and metrics are a SNAPSHOT overwrite (one `EndpointState`
|
||||||
|
row per asset), not time-series. The agent sends the full current list each
|
||||||
|
cycle; the server does not diff or version them. If per-sample history is later
|
||||||
|
required, it goes in a new append-only table and the contract stays additive.
|
||||||
|
|
||||||
|
## 3. Reporting rhythm
|
||||||
|
|
||||||
|
- Default report interval: 15 minutes (`report_interval_minutes` in the plugin
|
||||||
|
manifest settings; a site setting can override). Agent design should make the
|
||||||
|
interval configurable and jittered to avoid a fleet-wide thundering herd.
|
||||||
|
- There is NO separate heartbeat. `lastreportedat` is stamped on every report;
|
||||||
|
"quiet" endpoints are those past the quiet-report window (default 24h, site
|
||||||
|
setting `endpoints_quietreporthours`). Agent does not need a distinct ping.
|
||||||
|
|
||||||
|
## 4. Curated Windows Event catalog (LOCKED to the server list)
|
||||||
|
|
||||||
|
The server stores ONLY these IDs; anything else in `events` is ignored (not an
|
||||||
|
error). Source of truth: `CURATED_EVENT_IDS` in `plugins/endpoints/plugin.py`.
|
||||||
|
|
||||||
|
| Event ID | Channel | Meaning |
|
||||||
|
|----------|--------------|--------------------------------------------|
|
||||||
|
| 4624 | Security | Successful logon |
|
||||||
|
| 4625 | Security | Failed logon |
|
||||||
|
| 4634 | Security | Logoff |
|
||||||
|
| 4672 | Security | Special privileges assigned to new logon |
|
||||||
|
| 4697 | Security | Service installed |
|
||||||
|
| 4698 | Security | Scheduled task created |
|
||||||
|
| 41 | System | Kernel-Power, unexpected shutdown |
|
||||||
|
| 4618 | Security | Monitored security event pattern |
|
||||||
|
| 55 | System | NTFS file system corruption |
|
||||||
|
| 7011 | System | Service Control Manager timeout |
|
||||||
|
| 10016 | System | DCOM permission error |
|
||||||
|
| 1001 | Application | Application error / bugcheck report |
|
||||||
|
|
||||||
|
- Per event, the agent sends: `eventid` (required), `eventchannel`,
|
||||||
|
`eventsource`, `eventtime` (ISO-8601, event time not report time),
|
||||||
|
`eventmessage`, and `fields` (an object of extracted structured values, e.g.
|
||||||
|
target account on a 4625). `fields` is stored as `payloadjson`.
|
||||||
|
- OPEN QUESTION for the agent build: which IDs need structured `fields` parsing
|
||||||
|
vs. raw message pass-through. Start with pass-through message; add structured
|
||||||
|
parsing per ID as alerting rules demand it. The teammate flagged event parsing
|
||||||
|
as the hard part; keep the catalog fixed and small until baselined.
|
||||||
|
- Scheduled-task history being disabled on some hosts means 4698 and task
|
||||||
|
"last run" may be sparse; the agent should read current task state directly,
|
||||||
|
not rely on event history alone.
|
||||||
|
|
||||||
|
## 5. Monitored services/processes
|
||||||
|
|
||||||
|
Known targets to watch (non-exhaustive, confirm during agent build): UDC, eDnc,
|
||||||
|
Intune-related services. Server stores whatever list the agent sends in
|
||||||
|
`servicesjson` without a fixed schema yet. Decision deferred: once the watched
|
||||||
|
set is final, consider promoting it to a server-known enum for alerting.
|
||||||
|
|
||||||
|
## 6. ServiceNow ticketing surface (future, creds confirmed available)
|
||||||
|
|
||||||
|
Not built server-side yet. Recorded here so the agent and a future server
|
||||||
|
endpoint agree:
|
||||||
|
|
||||||
|
- Interactive C# piece (system-tray/desktop launcher) owns the 5-click-max flow:
|
||||||
|
network precheck -> issue dropdown -> optional screenshot -> submit.
|
||||||
|
- Fixed dropdown reasons (from the source conversation): Can't logon to CSF;
|
||||||
|
Data collections down; Can't download program; Machine communication failing;
|
||||||
|
Network drives down; Other (free text).
|
||||||
|
- Screenshot saved to local `Pictures\Screenshots` before submission.
|
||||||
|
- OPEN: whether ticket submission goes direct to the ServiceNow API from the
|
||||||
|
agent, or is proxied through a ShopDB server endpoint (preferred: proxy, so
|
||||||
|
ServiceNow creds live server-side and the agent carries only its collector
|
||||||
|
key). Decide before wiring the integration.
|
||||||
|
|
||||||
|
## 7. Runtime split (from the planning conversation)
|
||||||
|
|
||||||
|
- Interactive bits (tray launcher, ticketing, screenshot): C# / .NET, likely a
|
||||||
|
Windows service + tray app.
|
||||||
|
- Collections MAY be locally hosted PowerShell scripts the service invokes,
|
||||||
|
where that is simpler than reimplementing WMI/CIM queries in C#.
|
||||||
|
- Code signing and Intune deployment are agent-side concerns, out of scope for
|
||||||
|
the server plugin but must be planned before fleet rollout.
|
||||||
|
|
||||||
|
## 8. Web UI display (server-side, built)
|
||||||
|
|
||||||
|
The collected data surfaces through ShopDB's existing declaration-driven UI
|
||||||
|
contract (ADR-010). The plugin declares WHAT to show; generic core components
|
||||||
|
render it. No plugin-specific Vue components were written. Audience is DT
|
||||||
|
support / IT, NOT the shop-floor operator (the operator's surface is the
|
||||||
|
agent-side ticket tool).
|
||||||
|
|
||||||
|
Surfaces implemented:
|
||||||
|
|
||||||
|
- Per-PC asset panels (computer assets only), on the PC detail page:
|
||||||
|
- Endpoint status (keyvalue): logged-in user, uptime, idle, last boot,
|
||||||
|
CPU/memory %, agent version, last reported. Feeds
|
||||||
|
`GET /api/endpoints/asset/{assetid}/status`.
|
||||||
|
- Services and processes (list + badge): monitored services with a green/red
|
||||||
|
run-state badge. Feeds `.../services`.
|
||||||
|
- Recent events (list + severity badge): last 10 curated events. Feeds
|
||||||
|
`.../recentevents`.
|
||||||
|
- Dashboard exception cards (hide when empty):
|
||||||
|
- Endpoints gone quiet (warning): `.../dashboard/quiet`.
|
||||||
|
- Endpoints with crashes or shutdowns (critical): `.../dashboard/stability`,
|
||||||
|
bounded to a 48h window and the critical event catalog.
|
||||||
|
|
||||||
|
DECISION - freshness and cost. There is NO live streaming and NO polling. The
|
||||||
|
freshness ceiling is the agent report cadence (~15 min), so a websocket/poll
|
||||||
|
buys nothing but load. Data is fetched on page/dashboard load only, matching the
|
||||||
|
existing computers-quiet card. Asset panels are single-asset indexed reads
|
||||||
|
(negligible). Dashboard cards run once per dashboard load, cap at 50 rows, and
|
||||||
|
the stability card is time-bounded so it does not grow with total history. If
|
||||||
|
fleet-wide aggregation ever gets heavy, memoize dashboard aggregates with the
|
||||||
|
core `cache` helper for ~60s - documented lever, not built yet.
|
||||||
|
|
||||||
|
Agent implication: the agent does not drive any UI refresh. Its only job is to
|
||||||
|
report on its rhythm; the UI reflects whatever the last report wrote. The
|
||||||
|
`services` payload should send a per-service run STATE (e.g. running/stopped) so
|
||||||
|
the badge can color it; a missing/unknown state renders as a gray Unknown badge.
|
||||||
@@ -146,6 +146,16 @@ export const machinesApi = {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Endpoints API (plugin) - fleet telemetry read surface
|
||||||
|
export const endpointsApi = {
|
||||||
|
list(params = {}) {
|
||||||
|
return api.get('/endpoints', { params })
|
||||||
|
},
|
||||||
|
get(assetId) {
|
||||||
|
return api.get(`/endpoints/${assetId}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Computers API (plugin)
|
// Computers API (plugin)
|
||||||
export const computersApi = {
|
export const computersApi = {
|
||||||
list(params = {}) {
|
list(params = {}) {
|
||||||
|
|||||||
@@ -109,7 +109,7 @@ import ToastHost from '../components/ToastHost.vue'
|
|||||||
import {
|
import {
|
||||||
Sun, Moon, LayoutDashboard, Calendar, Map, Cog, Monitor,
|
Sun, Moon, LayoutDashboard, Calendar, Map, Cog, Monitor,
|
||||||
Printer, Globe, Usb, AppWindow, BookOpen, BarChart3, Bell, Image, ShieldCheck, Ruler,
|
Printer, Globe, Usb, AppWindow, BookOpen, BarChart3, Bell, Image, ShieldCheck, Ruler,
|
||||||
Box, KeyRound, LogOut, Wrench
|
Box, KeyRound, LogOut, Wrench, Activity
|
||||||
} from 'lucide-vue-next'
|
} from 'lucide-vue-next'
|
||||||
import { useAuthStore } from '../stores/auth'
|
import { useAuthStore } from '../stores/auth'
|
||||||
import { currentTheme, toggleTheme } from '../stores/theme'
|
import { currentTheme, toggleTheme } from '../stores/theme'
|
||||||
@@ -178,6 +178,7 @@ const iconMap = {
|
|||||||
'ruler': Ruler,
|
'ruler': Ruler,
|
||||||
'box': Box,
|
'box': Box,
|
||||||
'wrench': Wrench,
|
'wrench': Wrench,
|
||||||
|
'activity': Activity,
|
||||||
}
|
}
|
||||||
|
|
||||||
// Default navigation (used as fallback if API fails)
|
// Default navigation (used as fallback if API fails)
|
||||||
|
|||||||
5
plugins/endpoints/__init__.py
Normal file
5
plugins/endpoints/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
"""Endpoints plugin package."""
|
||||||
|
|
||||||
|
from .plugin import EndpointsPlugin
|
||||||
|
|
||||||
|
__all__ = ['EndpointsPlugin']
|
||||||
5
plugins/endpoints/api/__init__.py
Normal file
5
plugins/endpoints/api/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
"""Endpoints plugin API."""
|
||||||
|
|
||||||
|
from .routes import endpoints_bp
|
||||||
|
|
||||||
|
__all__ = ['endpoints_bp']
|
||||||
350
plugins/endpoints/api/routes.py
Normal file
350
plugins/endpoints/api/routes.py
Normal file
@@ -0,0 +1,350 @@
|
|||||||
|
"""Endpoints plugin API endpoints (read + dashboard).
|
||||||
|
|
||||||
|
Ingest is NOT here: agent reports arrive on the generic collector route
|
||||||
|
POST /api/collector/endpoints (ADR-006), authorized by a collector-scoped
|
||||||
|
service token, and are handled by EndpointsPlugin.apply_collector_payload.
|
||||||
|
These routes are the human-facing read surface, gated by user JWT + RBAC.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
from flask import Blueprint, request
|
||||||
|
from flask_jwt_extended import jwt_required
|
||||||
|
|
||||||
|
from shopdb.api import (
|
||||||
|
db, Asset, Setting, success_response, error_response, paginated_response,
|
||||||
|
ErrorCodes, get_pagination_params, paginate_query, require_permission,
|
||||||
|
)
|
||||||
|
|
||||||
|
from ..models import EndpointState, EndpointEvent
|
||||||
|
from ..eventcatalog import CRITICAL_EVENT_IDS, event_severity, event_title
|
||||||
|
|
||||||
|
endpoints_bp = Blueprint('endpoints', __name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _naive_utcnow():
|
||||||
|
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
|
def _format_uptime(seconds):
|
||||||
|
"""Whole-seconds uptime -> 'Nd Nh Nm', or None."""
|
||||||
|
if seconds is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
seconds = int(seconds)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
days, rem = divmod(seconds, 86400)
|
||||||
|
hours, rem = divmod(rem, 3600)
|
||||||
|
minutes = rem // 60
|
||||||
|
parts = []
|
||||||
|
if days:
|
||||||
|
parts.append('{}d'.format(days))
|
||||||
|
if hours:
|
||||||
|
parts.append('{}h'.format(hours))
|
||||||
|
parts.append('{}m'.format(minutes))
|
||||||
|
return ' '.join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Endpoint state
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
@endpoints_bp.route('/', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def list_endpoints():
|
||||||
|
"""List reported endpoint states.
|
||||||
|
|
||||||
|
Joins through Computer so each row carries computerid and can deep-link
|
||||||
|
into the existing PC detail page (/pcs/{computerid}). Endpoint state only
|
||||||
|
exists for hostnames resolved to a Computer, so the join is not lossy.
|
||||||
|
"""
|
||||||
|
from plugins.computers.models import Computer
|
||||||
|
|
||||||
|
page, per_page = get_pagination_params(request)
|
||||||
|
|
||||||
|
query = (db.session.query(EndpointState, Computer.computerid)
|
||||||
|
.join(Asset, Asset.assetid == EndpointState.assetid)
|
||||||
|
.join(Computer, Computer.assetid == EndpointState.assetid)
|
||||||
|
.filter(Asset.isactive.is_(True)))
|
||||||
|
|
||||||
|
if search := request.args.get('search'):
|
||||||
|
query = query.filter(EndpointState.hostname.ilike(f'%{search}%'))
|
||||||
|
|
||||||
|
query = query.order_by(EndpointState.lastreportedat.desc())
|
||||||
|
|
||||||
|
items, total = paginate_query(query, page, per_page)
|
||||||
|
data = []
|
||||||
|
for state, computerid in items:
|
||||||
|
row = state.to_dict()
|
||||||
|
row['computerid'] = computerid
|
||||||
|
row['uptimedisplay'] = _format_uptime(state.uptimeseconds)
|
||||||
|
data.append(row)
|
||||||
|
|
||||||
|
return paginated_response(data, page, per_page, total)
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/<int:asset_id>', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def get_endpoint(asset_id: int):
|
||||||
|
"""Get the current state for one endpoint by asset id."""
|
||||||
|
state = EndpointState.query.filter_by(assetid=asset_id).first()
|
||||||
|
|
||||||
|
if not state:
|
||||||
|
return error_response(
|
||||||
|
ErrorCodes.NOT_FOUND,
|
||||||
|
f'No endpoint state for asset {asset_id}',
|
||||||
|
http_code=404,
|
||||||
|
)
|
||||||
|
|
||||||
|
return success_response(state.to_dict())
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/<int:asset_id>/events', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def list_endpoint_events(asset_id: int):
|
||||||
|
"""List curated Windows events for one endpoint, newest first."""
|
||||||
|
page, per_page = get_pagination_params(request)
|
||||||
|
|
||||||
|
query = EndpointEvent.query.filter_by(assetid=asset_id)
|
||||||
|
|
||||||
|
if eventid := request.args.get('eventid'):
|
||||||
|
try:
|
||||||
|
query = query.filter(EndpointEvent.eventid == int(eventid))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return error_response(ErrorCodes.VALIDATION_ERROR,
|
||||||
|
'eventid must be an integer')
|
||||||
|
|
||||||
|
query = query.order_by(EndpointEvent.eventtime.desc())
|
||||||
|
|
||||||
|
items, total = paginate_query(query, page, per_page)
|
||||||
|
data = [e.to_dict() for e in items]
|
||||||
|
|
||||||
|
return paginated_response(data, page, per_page, total)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Asset panels (ADR-010) - shaped for the generic PluginAssetPanels renderer
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
@endpoints_bp.route('/asset/<int:asset_id>/status', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def panel_status(asset_id: int):
|
||||||
|
"""Keyvalue panel: the endpoint's current status snapshot.
|
||||||
|
|
||||||
|
Single indexed lookup on endpointstates.assetid. Returns the {fields:[...]}
|
||||||
|
shape the keyvalue renderer expects; an absent state yields no fields so the
|
||||||
|
panel hides itself.
|
||||||
|
"""
|
||||||
|
state = EndpointState.query.filter_by(assetid=asset_id).first()
|
||||||
|
if not state:
|
||||||
|
return success_response({'fields': []})
|
||||||
|
|
||||||
|
def field(label, value, mono=False):
|
||||||
|
return {'label': label, 'value': value, 'mono': mono}
|
||||||
|
|
||||||
|
fields = [
|
||||||
|
field('Logged in user', state.loggedinuser),
|
||||||
|
field('Uptime', _format_uptime(state.uptimeseconds)),
|
||||||
|
field('Idle', _format_uptime(state.idleseconds)),
|
||||||
|
field('Last boot',
|
||||||
|
state.lastboottime.isoformat() + 'Z' if state.lastboottime else None),
|
||||||
|
field('CPU %', state.cpupercent),
|
||||||
|
field('Memory %', state.memorypercent),
|
||||||
|
field('Agent', state.agentversion, mono=True),
|
||||||
|
field('Last reported',
|
||||||
|
state.lastreportedat.isoformat() + 'Z' if state.lastreportedat else None),
|
||||||
|
]
|
||||||
|
fields = [f for f in fields if f['value'] not in (None, '')]
|
||||||
|
return success_response({'fields': fields})
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/asset/<int:asset_id>/services', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def panel_services(asset_id: int):
|
||||||
|
"""List panel: monitored services/processes with a run-state badge per row.
|
||||||
|
|
||||||
|
Reshapes the latest servicesjson snapshot into rows the list renderer maps
|
||||||
|
to name + status badge. Tolerant of a missing or unexpected snapshot shape:
|
||||||
|
an unknown state renders as an Unknown badge rather than failing.
|
||||||
|
"""
|
||||||
|
state = EndpointState.query.filter_by(assetid=asset_id).first()
|
||||||
|
if not state or not state.servicesjson:
|
||||||
|
return success_response([])
|
||||||
|
|
||||||
|
try:
|
||||||
|
services = json.loads(state.servicesjson)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return success_response([])
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for entry in (services or []):
|
||||||
|
if isinstance(entry, dict):
|
||||||
|
name = entry.get('name') or entry.get('service') or 'Unknown'
|
||||||
|
raw = (entry.get('state') or entry.get('status') or '').strip().lower()
|
||||||
|
else:
|
||||||
|
name, raw = str(entry), ''
|
||||||
|
running = raw in ('running', 'started', 'up', 'ok', 'active')
|
||||||
|
rows.append({
|
||||||
|
'name': name,
|
||||||
|
'statelabel': 'Running' if running else (raw.title() or 'Unknown'),
|
||||||
|
'statecolor': 'green' if running else ('red' if raw else 'gray'),
|
||||||
|
})
|
||||||
|
return success_response(rows)
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/asset/<int:asset_id>/recentevents', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def panel_recent_events(asset_id: int):
|
||||||
|
"""List panel: the endpoint's most recent curated events, newest first.
|
||||||
|
|
||||||
|
Indexed range scan on (assetid, eventtime), capped small - this is the
|
||||||
|
at-a-glance history card, not the full report (that is the events route
|
||||||
|
with pagination).
|
||||||
|
"""
|
||||||
|
events = (EndpointEvent.query
|
||||||
|
.filter_by(assetid=asset_id)
|
||||||
|
.order_by(EndpointEvent.eventtime.desc())
|
||||||
|
.limit(10)
|
||||||
|
.all())
|
||||||
|
|
||||||
|
rows = [{
|
||||||
|
'title': event_title(e.eventid),
|
||||||
|
'severity': event_severity(e.eventid),
|
||||||
|
'eventtime': e.eventtime.isoformat() + 'Z' if e.eventtime else None,
|
||||||
|
} for e in events]
|
||||||
|
return success_response(rows)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Dashboard
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
@endpoints_bp.route('/dashboard/summary', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def dashboard_summary():
|
||||||
|
"""High-level fleet counts for the endpoints dashboard."""
|
||||||
|
total = db.session.query(EndpointState).join(
|
||||||
|
Asset, Asset.assetid == EndpointState.assetid
|
||||||
|
).filter(Asset.isactive.is_(True)).count()
|
||||||
|
|
||||||
|
hours = _quiet_hours()
|
||||||
|
cutoff = _naive_utcnow() - timedelta(hours=hours)
|
||||||
|
|
||||||
|
reporting = db.session.query(EndpointState).join(
|
||||||
|
Asset, Asset.assetid == EndpointState.assetid
|
||||||
|
).filter(
|
||||||
|
Asset.isactive.is_(True),
|
||||||
|
EndpointState.lastreportedat.isnot(None),
|
||||||
|
EndpointState.lastreportedat >= cutoff,
|
||||||
|
).count()
|
||||||
|
|
||||||
|
return success_response({
|
||||||
|
'total': total,
|
||||||
|
'reporting': reporting,
|
||||||
|
'quiet': total - reporting,
|
||||||
|
'quiethours': hours,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/dashboard/quiet', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def dashboard_quiet():
|
||||||
|
"""Endpoints that reported before, then went quiet past the window.
|
||||||
|
|
||||||
|
Never-reported endpoints are excluded: those are records without an agent,
|
||||||
|
not agents that stopped. Silence is the only signal - there is no heartbeat
|
||||||
|
separate from the report.
|
||||||
|
"""
|
||||||
|
hours = _quiet_hours()
|
||||||
|
cutoff = _naive_utcnow() - timedelta(hours=hours)
|
||||||
|
now = _naive_utcnow()
|
||||||
|
|
||||||
|
from plugins.computers.models import Computer
|
||||||
|
|
||||||
|
rows = (db.session.query(EndpointState, Computer.computerid)
|
||||||
|
.join(Asset, Asset.assetid == EndpointState.assetid)
|
||||||
|
.join(Computer, Computer.assetid == EndpointState.assetid)
|
||||||
|
.filter(Asset.isactive.is_(True),
|
||||||
|
EndpointState.lastreportedat.isnot(None),
|
||||||
|
EndpointState.lastreportedat < cutoff)
|
||||||
|
.all())
|
||||||
|
|
||||||
|
out = [{
|
||||||
|
'computerid': computerid,
|
||||||
|
'assetid': s.assetid,
|
||||||
|
'hostname': s.hostname,
|
||||||
|
'lastreported': s.lastreportedat.isoformat() + 'Z',
|
||||||
|
'quietdays': (now - s.lastreportedat).days,
|
||||||
|
} for s, computerid in rows]
|
||||||
|
|
||||||
|
out.sort(key=lambda r: -r['quietdays'])
|
||||||
|
return success_response(out[:50])
|
||||||
|
|
||||||
|
|
||||||
|
@endpoints_bp.route('/dashboard/stability', methods=['GET'])
|
||||||
|
@jwt_required()
|
||||||
|
@require_permission('endpoints.view')
|
||||||
|
def dashboard_stability():
|
||||||
|
"""Endpoints with recent crash/shutdown/corruption events.
|
||||||
|
|
||||||
|
Bounded to a fixed 48h window and the critical event catalog, capped at 50
|
||||||
|
rows: cost is a range scan on eventtime filtered to a short id list, not a
|
||||||
|
scan of all history. Joins through Computer so each row links to the PC
|
||||||
|
detail page.
|
||||||
|
"""
|
||||||
|
from plugins.computers.models import Computer
|
||||||
|
|
||||||
|
if not CRITICAL_EVENT_IDS:
|
||||||
|
return success_response([])
|
||||||
|
|
||||||
|
cutoff = _naive_utcnow() - timedelta(hours=48)
|
||||||
|
|
||||||
|
rows = (db.session.query(
|
||||||
|
Computer.computerid,
|
||||||
|
EndpointState.hostname,
|
||||||
|
db.func.count(EndpointEvent.endpointeventid),
|
||||||
|
db.func.max(EndpointEvent.eventtime),
|
||||||
|
)
|
||||||
|
.select_from(EndpointEvent)
|
||||||
|
.join(Asset, Asset.assetid == EndpointEvent.assetid)
|
||||||
|
.join(Computer, Computer.assetid == EndpointEvent.assetid)
|
||||||
|
.outerjoin(EndpointState,
|
||||||
|
EndpointState.assetid == EndpointEvent.assetid)
|
||||||
|
.filter(Asset.isactive.is_(True),
|
||||||
|
EndpointEvent.eventid.in_(CRITICAL_EVENT_IDS),
|
||||||
|
EndpointEvent.eventtime.isnot(None),
|
||||||
|
EndpointEvent.eventtime >= cutoff)
|
||||||
|
.group_by(Computer.computerid, EndpointState.hostname)
|
||||||
|
.all())
|
||||||
|
|
||||||
|
out = [{
|
||||||
|
'computerid': computerid,
|
||||||
|
'hostname': hostname,
|
||||||
|
'eventcount': count,
|
||||||
|
'lastevent': lasttime.isoformat() + 'Z' if lasttime else None,
|
||||||
|
} for computerid, hostname, count, lasttime in rows]
|
||||||
|
|
||||||
|
out.sort(key=lambda r: -r['eventcount'])
|
||||||
|
return success_response(out[:50])
|
||||||
|
|
||||||
|
|
||||||
|
def _quiet_hours() -> int:
|
||||||
|
"""Quiet-report window in hours, from a site setting, default 24."""
|
||||||
|
hours = 24
|
||||||
|
setting = Setting.query.filter_by(key='endpoints_quietreporthours').first()
|
||||||
|
if setting and (setting.value or '').strip():
|
||||||
|
try:
|
||||||
|
hours = int(setting.value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return hours
|
||||||
46
plugins/endpoints/eventcatalog.py
Normal file
46
plugins/endpoints/eventcatalog.py
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
"""Curated Windows event catalog: id -> human title and severity.
|
||||||
|
|
||||||
|
Single server-side source of truth shared by the recent-events asset panel and
|
||||||
|
the stability dashboard card, so a 4625 reads as "Failed logon (warning)" in
|
||||||
|
both places without either restating the mapping. The set of IDs here MUST stay
|
||||||
|
in step with CURATED_EVENT_IDS in plugin.py; a test asserts they match.
|
||||||
|
|
||||||
|
Severity vocabulary matches the dashboard card renderer: 'critical', 'warning',
|
||||||
|
'info'. The IDs marked critical are the ones an operator/tech should act on
|
||||||
|
without being asked - unexpected shutdown, crash, filesystem corruption, SCM
|
||||||
|
timeout. Logon activity is informational history unless it is a failure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# id -> (title, severity, channel)
|
||||||
|
EVENT_CATALOG = {
|
||||||
|
4624: ('Successful logon', 'info', 'Security'),
|
||||||
|
4625: ('Failed logon', 'warning', 'Security'),
|
||||||
|
4634: ('Logoff', 'info', 'Security'),
|
||||||
|
4672: ('Special privileges assigned', 'info', 'Security'),
|
||||||
|
4697: ('Service installed', 'warning', 'Security'),
|
||||||
|
4698: ('Scheduled task created', 'warning', 'Security'),
|
||||||
|
41: ('Unexpected shutdown', 'critical', 'System'),
|
||||||
|
4618: ('Monitored security event', 'warning', 'Security'),
|
||||||
|
55: ('File system corruption', 'critical', 'System'),
|
||||||
|
7011: ('Service timeout', 'critical', 'System'),
|
||||||
|
10016: ('DCOM permission error', 'warning', 'System'),
|
||||||
|
1001: ('Application crash', 'critical', 'Application'),
|
||||||
|
}
|
||||||
|
|
||||||
|
# The IDs a tech should see surfaced fleet-wide on the dashboard stability card.
|
||||||
|
CRITICAL_EVENT_IDS = tuple(
|
||||||
|
eventid for eventid, (_t, severity, _c) in EVENT_CATALOG.items()
|
||||||
|
if severity == 'critical'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def event_title(eventid):
|
||||||
|
"""Human title for a curated event id, or a generic fallback."""
|
||||||
|
entry = EVENT_CATALOG.get(eventid)
|
||||||
|
return entry[0] if entry else 'Event {}'.format(eventid)
|
||||||
|
|
||||||
|
|
||||||
|
def event_severity(eventid):
|
||||||
|
"""Severity ('critical'|'warning'|'info') for a curated event id."""
|
||||||
|
entry = EVENT_CATALOG.get(eventid)
|
||||||
|
return entry[1] if entry else 'info'
|
||||||
20
plugins/endpoints/frontend/routes.js
Normal file
20
plugins/endpoints/frontend/routes.js
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
/**
|
||||||
|
* Endpoints plugin routes
|
||||||
|
*
|
||||||
|
* Groundwork for the fleet view. There is no endpoint-specific detail route
|
||||||
|
* yet: rows link into the existing PC detail page (/pcs/:id) so telemetry
|
||||||
|
* stays interoperable with the computers plugin instead of duplicating it.
|
||||||
|
*/
|
||||||
|
export default [
|
||||||
|
{
|
||||||
|
path: 'endpoints',
|
||||||
|
name: 'endpoints',
|
||||||
|
component: () => import('./views/EndpointsList.vue'),
|
||||||
|
meta: { plugin: 'endpoints' }
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
// No full-screen routes: every view in this plugin renders inside
|
||||||
|
// AppLayout. Declared so routes.gen.js can read `.toplevel` off the module
|
||||||
|
// without Rollup warning about a missing export.
|
||||||
|
export const toplevel = []
|
||||||
153
plugins/endpoints/frontend/views/EndpointsList.vue
Normal file
153
plugins/endpoints/frontend/views/EndpointsList.vue
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
<template>
|
||||||
|
<div>
|
||||||
|
<div class="page-header">
|
||||||
|
<h2>Fleet</h2>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Filters -->
|
||||||
|
<div class="filters">
|
||||||
|
<input
|
||||||
|
v-model="search"
|
||||||
|
type="text"
|
||||||
|
class="form-control"
|
||||||
|
placeholder="Search endpoints..."
|
||||||
|
@input="debouncedSearch"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<div v-if="loading" class="loading">Loading...</div>
|
||||||
|
|
||||||
|
<template v-else>
|
||||||
|
<div class="table-container">
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Hostname</th>
|
||||||
|
<th>Logged-in User</th>
|
||||||
|
<th>Uptime</th>
|
||||||
|
<th>CPU</th>
|
||||||
|
<th>Memory</th>
|
||||||
|
<th>Last Reported</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr
|
||||||
|
v-for="item in endpoints"
|
||||||
|
:key="item.assetid"
|
||||||
|
class="clickable-row"
|
||||||
|
:class="{ disabled: !item.computerid }"
|
||||||
|
@click="openPc(item)"
|
||||||
|
>
|
||||||
|
<td>{{ item.hostname || '-' }}</td>
|
||||||
|
<td>{{ item.loggedinuser || '-' }}</td>
|
||||||
|
<td>{{ item.uptimedisplay || '-' }}</td>
|
||||||
|
<td>{{ formatPercent(item.cpupercent) }}</td>
|
||||||
|
<td>{{ formatPercent(item.memorypercent) }}</td>
|
||||||
|
<td>{{ formatTime(item.lastreportedat) }}</td>
|
||||||
|
</tr>
|
||||||
|
<tr v-if="endpoints.length === 0">
|
||||||
|
<td colspan="6" style="text-align: center; color: var(--text-light);">
|
||||||
|
No endpoints found
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Pagination -->
|
||||||
|
<PaginationBar
|
||||||
|
:page="page"
|
||||||
|
:totalPages="totalPages"
|
||||||
|
:perPage="perPage"
|
||||||
|
@update:page="goToPage"
|
||||||
|
@update:perPage="changePerPage"
|
||||||
|
/>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup>
|
||||||
|
import { ref, onMounted } from 'vue'
|
||||||
|
import { useRouter } from 'vue-router'
|
||||||
|
import { endpointsApi } from '@/api'
|
||||||
|
import PaginationBar from '@/components/PaginationBar.vue'
|
||||||
|
import { useListQuery } from '@/composables/listQuery'
|
||||||
|
|
||||||
|
const router = useRouter()
|
||||||
|
const endpoints = ref([])
|
||||||
|
const loading = ref(true)
|
||||||
|
const { page, search, setPage, setSearch } = useListQuery({ onChange: loadEndpoints })
|
||||||
|
const totalPages = ref(1)
|
||||||
|
const perPage = ref(20)
|
||||||
|
|
||||||
|
let searchTimeout = null
|
||||||
|
|
||||||
|
onMounted(() => {
|
||||||
|
loadEndpoints()
|
||||||
|
})
|
||||||
|
|
||||||
|
async function loadEndpoints() {
|
||||||
|
loading.value = true
|
||||||
|
try {
|
||||||
|
const params = {
|
||||||
|
page: page.value,
|
||||||
|
perpage: perPage.value
|
||||||
|
}
|
||||||
|
if (search.value) params.search = search.value
|
||||||
|
|
||||||
|
const response = await endpointsApi.list(params)
|
||||||
|
endpoints.value = response.data.data || []
|
||||||
|
totalPages.value = response.data.meta?.pagination?.totalpages || response.data.meta?.pagination?.total_pages || 1
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Error loading endpoints:', error)
|
||||||
|
} finally {
|
||||||
|
loading.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Interoperate with PC Details instead of a bespoke detail page: a row only
|
||||||
|
// navigates when it resolved to a computerid.
|
||||||
|
function openPc(item) {
|
||||||
|
if (item.computerid) {
|
||||||
|
router.push(`/pcs/${item.computerid}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatPercent(value) {
|
||||||
|
if (value === null || value === undefined) return '-'
|
||||||
|
return `${Math.round(value)}%`
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatTime(value) {
|
||||||
|
if (!value) return '-'
|
||||||
|
return new Date(value).toLocaleString()
|
||||||
|
}
|
||||||
|
|
||||||
|
function debouncedSearch() {
|
||||||
|
clearTimeout(searchTimeout)
|
||||||
|
searchTimeout = setTimeout(() => {
|
||||||
|
setSearch(search.value)
|
||||||
|
loadEndpoints()
|
||||||
|
}, 300)
|
||||||
|
}
|
||||||
|
|
||||||
|
function goToPage(p) {
|
||||||
|
setPage(p)
|
||||||
|
loadEndpoints()
|
||||||
|
}
|
||||||
|
|
||||||
|
function changePerPage(newPerPage) {
|
||||||
|
perPage.value = newPerPage
|
||||||
|
setPage(1)
|
||||||
|
loadEndpoints()
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.clickable-row.disabled {
|
||||||
|
cursor: default;
|
||||||
|
opacity: 0.6;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
21
plugins/endpoints/manifest.json
Normal file
21
plugins/endpoints/manifest.json
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"name": "endpoints",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"description": "Endpoint RMM telemetry: metrics, services, inventory, and curated Windows event records reported by the ShopDB agent",
|
||||||
|
"author": "ShopDB Team",
|
||||||
|
"dependencies": ["computers"],
|
||||||
|
"core_version": ">=0.20.0,<1.0.0",
|
||||||
|
"api_prefix": "/api/endpoints",
|
||||||
|
"provides": {
|
||||||
|
"features": [
|
||||||
|
"endpoint_metrics",
|
||||||
|
"service_state",
|
||||||
|
"inventory_snapshot",
|
||||||
|
"event_records"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"settings": {
|
||||||
|
"report_interval_minutes": 15,
|
||||||
|
"quiet_report_hours": 24
|
||||||
|
}
|
||||||
|
}
|
||||||
15
plugins/endpoints/migrations/env.py
Normal file
15
plugins/endpoints/migrations/env.py
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
"""Alembic environment for the endpoints plugin migration chain.
|
||||||
|
|
||||||
|
Delegates to the shared runner in shopdb.plugins.alembic_template, which filters
|
||||||
|
the metadata to this plugin's tables and drives Alembic against the per-plugin
|
||||||
|
version table alembic_version_endpoints. See ADR-008 for the ownership model.
|
||||||
|
This plugin is NEW: its 0001 baseline really CREATES its tables, because the
|
||||||
|
core chain never built them.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
|
||||||
|
os.environ['PLUGIN_NAME'] = 'endpoints'
|
||||||
|
|
||||||
|
from shopdb.plugins.alembic_template import run_migrations # noqa: E402
|
||||||
|
|
||||||
|
run_migrations()
|
||||||
24
plugins/endpoints/migrations/script.py.mako
Normal file
24
plugins/endpoints/migrations/script.py.mako
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
"""${message}
|
||||||
|
|
||||||
|
Revision ID: ${up_revision}
|
||||||
|
Revises: ${down_revision | comma,n}
|
||||||
|
Create Date: ${create_date}
|
||||||
|
|
||||||
|
"""
|
||||||
|
from alembic import op
|
||||||
|
import sqlalchemy as sa
|
||||||
|
${imports if imports else ""}
|
||||||
|
|
||||||
|
# revision identifiers, used by Alembic.
|
||||||
|
revision = ${repr(up_revision)}
|
||||||
|
down_revision = ${repr(down_revision)}
|
||||||
|
branch_labels = ${repr(branch_labels)}
|
||||||
|
depends_on = ${repr(depends_on)}
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade():
|
||||||
|
${upgrades if upgrades else "pass"}
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade():
|
||||||
|
${downgrades if downgrades else "pass"}
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""endpoints plugin baseline (real create).
|
||||||
|
|
||||||
|
A NEW plugin: the core chain never knew about endpointstates or endpointevents,
|
||||||
|
so this per-plugin chain is their sole authoritative creator.
|
||||||
|
|
||||||
|
Emits explicit Alembic ops rather than using the create_plugin_tables helper:
|
||||||
|
the helper builds a per-plugin MetaData filtered to the plugin's own tables, so
|
||||||
|
the foreign key to the core assets table cannot resolve at CreateTable-compile
|
||||||
|
time (NoReferencedTableError). Tables inherit the connection's default charset,
|
||||||
|
matching how the core chain creates its own.
|
||||||
|
"""
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
from shopdb.plugins.alembic_template import (
|
||||||
|
create_index_if_missing,
|
||||||
|
create_table_if_missing,
|
||||||
|
)
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
|
||||||
|
# revision identifiers, used by Alembic.
|
||||||
|
revision = 'endpoints0001baseline'
|
||||||
|
down_revision = None
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade():
|
||||||
|
create_table_if_missing(
|
||||||
|
'endpointstates',
|
||||||
|
sa.Column('endpointstateid', sa.Integer(), nullable=False),
|
||||||
|
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||||
|
sa.Column('hostname', sa.String(length=100), nullable=True),
|
||||||
|
sa.Column('loggedinuser', sa.String(length=100), nullable=True),
|
||||||
|
sa.Column('agentversion', sa.String(length=40), nullable=True),
|
||||||
|
sa.Column('lastboottime', sa.DateTime(), nullable=True),
|
||||||
|
sa.Column('uptimeseconds', sa.BigInteger(), nullable=True),
|
||||||
|
sa.Column('idleseconds', sa.BigInteger(), nullable=True),
|
||||||
|
sa.Column('cpupercent', sa.Float(), nullable=True),
|
||||||
|
sa.Column('memorypercent', sa.Float(), nullable=True),
|
||||||
|
sa.Column('servicesjson', sa.Text(length=16777215), nullable=True),
|
||||||
|
sa.Column('portsjson', sa.Text(length=16777215), nullable=True),
|
||||||
|
sa.Column('interfacesjson', sa.Text(length=16777215), nullable=True),
|
||||||
|
sa.Column('scheduledtasksjson', sa.Text(length=16777215),
|
||||||
|
nullable=True),
|
||||||
|
sa.Column('lastreportedat', sa.DateTime(), nullable=True),
|
||||||
|
sa.Column('createdat', sa.DateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||||
|
ondelete='CASCADE'),
|
||||||
|
sa.PrimaryKeyConstraint('endpointstateid'),
|
||||||
|
sa.UniqueConstraint('assetid'),
|
||||||
|
)
|
||||||
|
create_index_if_missing('idx_endpointstates_lastreportedat',
|
||||||
|
'endpointstates', ['lastreportedat'])
|
||||||
|
|
||||||
|
create_table_if_missing(
|
||||||
|
'endpointevents',
|
||||||
|
sa.Column('endpointeventid', sa.Integer(), nullable=False),
|
||||||
|
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||||
|
sa.Column('eventid', sa.Integer(), nullable=False),
|
||||||
|
sa.Column('eventchannel', sa.String(length=50), nullable=True),
|
||||||
|
sa.Column('eventsource', sa.String(length=255), nullable=True),
|
||||||
|
sa.Column('eventtime', sa.DateTime(), nullable=True),
|
||||||
|
sa.Column('eventmessage', sa.Text(), nullable=True),
|
||||||
|
sa.Column('payloadjson', sa.Text(length=16777215), nullable=True),
|
||||||
|
sa.Column('recordedat', sa.DateTime(), nullable=False),
|
||||||
|
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||||
|
ondelete='CASCADE'),
|
||||||
|
sa.PrimaryKeyConstraint('endpointeventid'),
|
||||||
|
)
|
||||||
|
create_index_if_missing('ixendpointeventsassetid', 'endpointevents',
|
||||||
|
['assetid'])
|
||||||
|
create_index_if_missing('ixendpointeventseventid', 'endpointevents',
|
||||||
|
['eventid'])
|
||||||
|
create_index_if_missing('ixendpointeventseventtime', 'endpointevents',
|
||||||
|
['eventtime'])
|
||||||
|
# Alerting/history reads by "this endpoint's events over time", so the
|
||||||
|
# composite is the index that actually gets used.
|
||||||
|
create_index_if_missing('idx_endpointevents_assetevent', 'endpointevents',
|
||||||
|
['assetid', 'eventtime'])
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade():
|
||||||
|
op.drop_index('idx_endpointevents_assetevent',
|
||||||
|
table_name='endpointevents')
|
||||||
|
op.drop_index('ixendpointeventseventtime', table_name='endpointevents')
|
||||||
|
op.drop_index('ixendpointeventseventid', table_name='endpointevents')
|
||||||
|
op.drop_index('ixendpointeventsassetid', table_name='endpointevents')
|
||||||
|
op.drop_table('endpointevents')
|
||||||
|
|
||||||
|
op.drop_index('idx_endpointstates_lastreportedat',
|
||||||
|
table_name='endpointstates')
|
||||||
|
op.drop_table('endpointstates')
|
||||||
5
plugins/endpoints/models/__init__.py
Normal file
5
plugins/endpoints/models/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
"""Endpoints plugin models."""
|
||||||
|
|
||||||
|
from .endpoint import EndpointState, EndpointEvent
|
||||||
|
|
||||||
|
__all__ = ['EndpointState', 'EndpointEvent']
|
||||||
163
plugins/endpoints/models/endpoint.py
Normal file
163
plugins/endpoints/models/endpoint.py
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
"""Endpoint RMM telemetry models.
|
||||||
|
|
||||||
|
Two shapes, one per concern:
|
||||||
|
|
||||||
|
EndpointState The cached CURRENT snapshot for an asset - one row per asset,
|
||||||
|
overwritten in place on every agent report. Holds the fast-
|
||||||
|
moving metrics (uptime, idle, last boot, logged-in user) and
|
||||||
|
the latest inventory snapshots (services/processes, open ports,
|
||||||
|
interfaces, scheduled tasks) as serialized JSON. This is the
|
||||||
|
cached-current-value half of the ledger pattern: it moves in
|
||||||
|
the SAME commit as any EndpointEvent rows the report carries.
|
||||||
|
|
||||||
|
EndpointEvent Append-only record of a curated Windows Event Log entry. The
|
||||||
|
agent only forwards a fixed catalog of IDs (logon/logoff,
|
||||||
|
privilege use, service install, unexpected shutdown, DCOM,
|
||||||
|
crash) so this stays a readable security/stability history
|
||||||
|
rather than a raw event firehose.
|
||||||
|
|
||||||
|
Both link to the core Asset via assetid (FK, ON DELETE CASCADE), so an endpoint
|
||||||
|
is the same physical thing as its Computer/Asset - no duplicate identity. The
|
||||||
|
agent reports a hostname; the plugin resolves it to the existing Computer.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from shopdb.api import db
|
||||||
|
|
||||||
|
|
||||||
|
def _utciso(value):
|
||||||
|
"""Naive-UTC datetime -> ISO string marked UTC, or None.
|
||||||
|
|
||||||
|
Every datetime this plugin stores is naive UTC. Serializing without the Z
|
||||||
|
leaves the receiver to guess, and JavaScript guesses browser-local.
|
||||||
|
"""
|
||||||
|
return value.isoformat() + 'Z' if value else None
|
||||||
|
|
||||||
|
|
||||||
|
class EndpointState(db.Model):
|
||||||
|
"""Latest reported state for a single endpoint (one row per asset)."""
|
||||||
|
|
||||||
|
__tablename__ = 'endpointstates'
|
||||||
|
|
||||||
|
endpointstateid = db.Column(db.Integer, primary_key=True)
|
||||||
|
|
||||||
|
# The asset this state belongs to. Unique: exactly one current snapshot per
|
||||||
|
# endpoint. Resolved from the reported hostname against the computers plugin.
|
||||||
|
assetid = db.Column(
|
||||||
|
db.Integer,
|
||||||
|
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
||||||
|
unique=True,
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Read-only view of the owning asset. No cascade or backref: the asset side
|
||||||
|
# must not gain a dependency on this plugin.
|
||||||
|
asset = db.relationship('Asset', lazy='joined', viewonly=True)
|
||||||
|
|
||||||
|
hostname = db.Column(db.String(100), nullable=True)
|
||||||
|
loggedinuser = db.Column(db.String(100), nullable=True)
|
||||||
|
agentversion = db.Column(db.String(40), nullable=True)
|
||||||
|
|
||||||
|
# Fast-moving metrics
|
||||||
|
lastboottime = db.Column(db.DateTime, nullable=True)
|
||||||
|
uptimeseconds = db.Column(db.BigInteger, nullable=True)
|
||||||
|
idleseconds = db.Column(db.BigInteger, nullable=True)
|
||||||
|
cpupercent = db.Column(db.Float, nullable=True)
|
||||||
|
memorypercent = db.Column(db.Float, nullable=True)
|
||||||
|
|
||||||
|
# Latest inventory snapshots, serialized JSON (MEDIUMTEXT). Kept as the
|
||||||
|
# newest snapshot rather than one row per sample: the agent reports the full
|
||||||
|
# list each cycle and history of these lists is not what operators ask for.
|
||||||
|
servicesjson = db.Column(db.Text(length=16777215), nullable=True)
|
||||||
|
portsjson = db.Column(db.Text(length=16777215), nullable=True)
|
||||||
|
interfacesjson = db.Column(db.Text(length=16777215), nullable=True)
|
||||||
|
scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True)
|
||||||
|
|
||||||
|
# When the agent last successfully reported. There is no heartbeat separate
|
||||||
|
# from the report, so silence past the quiet-hours window is the only
|
||||||
|
# evidence an endpoint has stopped checking in.
|
||||||
|
lastreportedat = db.Column(db.DateTime, nullable=True, index=True)
|
||||||
|
|
||||||
|
createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
|
||||||
|
|
||||||
|
__table_args__ = (
|
||||||
|
db.Index('idx_endpointstates_lastreportedat', 'lastreportedat'),
|
||||||
|
)
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return f"<EndpointState {self.hostname or self.assetid}>"
|
||||||
|
|
||||||
|
def to_dict(self):
|
||||||
|
return {
|
||||||
|
'endpointstateid': self.endpointstateid,
|
||||||
|
'assetid': self.assetid,
|
||||||
|
'hostname': self.hostname,
|
||||||
|
'loggedinuser': self.loggedinuser,
|
||||||
|
'agentversion': self.agentversion,
|
||||||
|
'lastboottime': _utciso(self.lastboottime),
|
||||||
|
'uptimeseconds': self.uptimeseconds,
|
||||||
|
'idleseconds': self.idleseconds,
|
||||||
|
'cpupercent': self.cpupercent,
|
||||||
|
'memorypercent': self.memorypercent,
|
||||||
|
'lastreportedat': _utciso(self.lastreportedat),
|
||||||
|
'createdat': _utciso(self.createdat),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class EndpointEvent(db.Model):
|
||||||
|
"""A single curated Windows Event Log record forwarded by the agent."""
|
||||||
|
|
||||||
|
__tablename__ = 'endpointevents'
|
||||||
|
|
||||||
|
endpointeventid = db.Column(db.Integer, primary_key=True)
|
||||||
|
|
||||||
|
assetid = db.Column(
|
||||||
|
db.Integer,
|
||||||
|
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
||||||
|
nullable=False,
|
||||||
|
index=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
asset = db.relationship('Asset', lazy='joined', viewonly=True)
|
||||||
|
|
||||||
|
# The Windows Event ID (for example 4625 failed logon, 41 unexpected
|
||||||
|
# shutdown, 1001 application crash). Indexed because alerting queries filter
|
||||||
|
# on it.
|
||||||
|
eventid = db.Column(db.Integer, nullable=False, index=True)
|
||||||
|
|
||||||
|
# Event Log channel the record came from: Security, System, Application.
|
||||||
|
eventchannel = db.Column(db.String(50), nullable=True)
|
||||||
|
eventsource = db.Column(db.String(255), nullable=True)
|
||||||
|
|
||||||
|
# When the event happened on the endpoint (from the record), not when it
|
||||||
|
# reached ShopDB.
|
||||||
|
eventtime = db.Column(db.DateTime, nullable=True, index=True)
|
||||||
|
|
||||||
|
eventmessage = db.Column(db.Text, nullable=True)
|
||||||
|
|
||||||
|
# Structured fields the agent extracted for this ID (for example the target
|
||||||
|
# account on a 4625). Advisory index only, not part of dedup.
|
||||||
|
payloadjson = db.Column(db.Text(length=16777215), nullable=True)
|
||||||
|
|
||||||
|
recordedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
|
||||||
|
|
||||||
|
__table_args__ = (
|
||||||
|
db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'),
|
||||||
|
)
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return f"<EndpointEvent {self.eventid} asset={self.assetid}>"
|
||||||
|
|
||||||
|
def to_dict(self):
|
||||||
|
return {
|
||||||
|
'endpointeventid': self.endpointeventid,
|
||||||
|
'assetid': self.assetid,
|
||||||
|
'eventid': self.eventid,
|
||||||
|
'eventchannel': self.eventchannel,
|
||||||
|
'eventsource': self.eventsource,
|
||||||
|
'eventtime': _utciso(self.eventtime),
|
||||||
|
'eventmessage': self.eventmessage,
|
||||||
|
'recordedat': _utciso(self.recordedat),
|
||||||
|
}
|
||||||
376
plugins/endpoints/plugin.py
Normal file
376
plugins/endpoints/plugin.py
Normal file
@@ -0,0 +1,376 @@
|
|||||||
|
"""Endpoints plugin main class.
|
||||||
|
|
||||||
|
Endpoint RMM telemetry reported by the ShopDB agent. Accepts collector input
|
||||||
|
over HTTPS per ADR-006: the agent posts a per-host payload to
|
||||||
|
POST /api/collector/endpoints, the plugin resolves the reported hostname to an
|
||||||
|
existing computer asset, overwrites that asset's cached EndpointState, and
|
||||||
|
appends any curated Windows events the report carried - all in one commit, so
|
||||||
|
the cached snapshot and the events it summarizes never disagree.
|
||||||
|
|
||||||
|
Identity is the hostname, resolved against the computers plugin. An unresolved
|
||||||
|
hostname is reported as a ValueError (surfaced to the caller as a validation
|
||||||
|
error), never silently dropped: an unknown host is a real gap between the fleet
|
||||||
|
and ShopDB, the same discipline the backups collector uses for machine numbers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List, Optional, Type
|
||||||
|
|
||||||
|
from flask import Flask, Blueprint
|
||||||
|
|
||||||
|
from shopdb.plugins.base import BasePlugin, PluginMeta
|
||||||
|
from shopdb.api import db
|
||||||
|
|
||||||
|
from .api import endpoints_bp
|
||||||
|
from .eventcatalog import EVENT_CATALOG
|
||||||
|
from .models import EndpointState, EndpointEvent
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# The fixed catalog of Windows Event IDs the agent forwards. Derived from
|
||||||
|
# EVENT_CATALOG (eventcatalog.py) so the ingest filter and the display titles
|
||||||
|
# can never drift. IDs outside this set are ignored on ingest rather than
|
||||||
|
# stored, keeping endpointevents a readable security/stability history instead
|
||||||
|
# of a raw event firehose.
|
||||||
|
CURATED_EVENT_IDS = frozenset(EVENT_CATALOG)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_dt(value):
|
||||||
|
"""Best-effort parse of an ISO-8601 string to naive UTC, or None."""
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
if isinstance(value, datetime):
|
||||||
|
return value.replace(tzinfo=None)
|
||||||
|
text = str(value).strip().replace('Z', '+00:00')
|
||||||
|
try:
|
||||||
|
parsed = datetime.fromisoformat(text)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
if parsed.tzinfo is not None:
|
||||||
|
parsed = parsed.astimezone(tz=None).replace(tzinfo=None)
|
||||||
|
return parsed
|
||||||
|
|
||||||
|
|
||||||
|
def _as_json(value):
|
||||||
|
"""Serialize a list/dict to JSON text, or None when nothing was sent."""
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
return json.dumps(value, separators=(',', ':'), sort_keys=True)
|
||||||
|
|
||||||
|
|
||||||
|
class EndpointsPlugin(BasePlugin):
|
||||||
|
"""Endpoint RMM telemetry ingest plus read/dashboard surface."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._manifest = self._load_manifest()
|
||||||
|
|
||||||
|
def _load_manifest(self) -> Dict:
|
||||||
|
manifest_path = Path(__file__).parent / 'manifest.json'
|
||||||
|
if manifest_path.exists():
|
||||||
|
with open(manifest_path, 'r') as f:
|
||||||
|
return json.load(f)
|
||||||
|
return {}
|
||||||
|
|
||||||
|
@property
|
||||||
|
def meta(self) -> PluginMeta:
|
||||||
|
return PluginMeta(
|
||||||
|
name=self._manifest.get('name', 'endpoints'),
|
||||||
|
version=self._manifest.get('version', '0.1.0'),
|
||||||
|
description=self._manifest.get(
|
||||||
|
'description', 'Endpoint RMM telemetry'),
|
||||||
|
author=self._manifest.get('author', 'ShopDB Team'),
|
||||||
|
dependencies=self._manifest.get('dependencies', ['computers']),
|
||||||
|
core_version=self._manifest.get('core_version', '>=0.20.0,<1.0.0'),
|
||||||
|
api_prefix=self._manifest.get('api_prefix', '/api/endpoints'),
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_blueprint(self) -> Optional[Blueprint]:
|
||||||
|
return endpoints_bp
|
||||||
|
|
||||||
|
def get_models(self) -> List[Type]:
|
||||||
|
return [EndpointState, EndpointEvent]
|
||||||
|
|
||||||
|
def get_permissions(self) -> List:
|
||||||
|
"""RBAC for the human-facing read routes.
|
||||||
|
|
||||||
|
View only. Ingest is not covered here: the collector path authorizes on
|
||||||
|
a collector.ingest-scoped service token, not a user JWT. There is no
|
||||||
|
write/delete permission because operators do not edit telemetry - the
|
||||||
|
agent is the sole author, and history is pruned by policy, not by hand.
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
('endpoints.view', 'View endpoint telemetry', 'endpoints'),
|
||||||
|
]
|
||||||
|
|
||||||
|
def get_navigation_items(self) -> List[Dict]:
|
||||||
|
"""Sidebar entry for the fleet list view.
|
||||||
|
|
||||||
|
Sits just after the PCs item (position 15) because the fleet view is
|
||||||
|
a telemetry lens over the same computers.
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
'name': 'Fleet',
|
||||||
|
'icon': 'activity',
|
||||||
|
'route': '/endpoints',
|
||||||
|
'position': 16,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
def init_app(self, app: Flask, db_instance) -> None:
|
||||||
|
logger.info('Endpoints plugin initialized (v%s)', self.meta.version)
|
||||||
|
|
||||||
|
# ---- ADR-010 UI hooks -----------------------------------------------
|
||||||
|
|
||||||
|
def get_asset_panels(self) -> List[Dict]:
|
||||||
|
"""Per-PC detail panels, computer assets only.
|
||||||
|
|
||||||
|
Three panels feeding the generic renderer: the current status snapshot
|
||||||
|
(keyvalue), monitored services with a run-state badge (list), and the
|
||||||
|
most recent curated events (list). Each is a single-asset indexed read,
|
||||||
|
so the cost is the same profile as the warranty/backups panels. Panels
|
||||||
|
hide themselves when their endpoint returns nothing.
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
'id': 'endpoints-status',
|
||||||
|
'title': 'Endpoint status',
|
||||||
|
'assettypes': ['computer'],
|
||||||
|
'endpoint': '/api/endpoints/asset/{assetid}/status',
|
||||||
|
'render': 'keyvalue',
|
||||||
|
'empty': 'No agent report yet.',
|
||||||
|
'position': 10,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'id': 'endpoints-services',
|
||||||
|
'title': 'Services and processes',
|
||||||
|
'assettypes': ['computer'],
|
||||||
|
'endpoint': '/api/endpoints/asset/{assetid}/services',
|
||||||
|
'render': 'list',
|
||||||
|
'map': {
|
||||||
|
'title': 'name',
|
||||||
|
'badge': {'label': 'statelabel', 'color': 'statecolor'},
|
||||||
|
},
|
||||||
|
'empty': 'No monitored services reported.',
|
||||||
|
'position': 12,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'id': 'endpoints-events',
|
||||||
|
'title': 'Recent events',
|
||||||
|
'assettypes': ['computer'],
|
||||||
|
'endpoint': '/api/endpoints/asset/{assetid}/recentevents',
|
||||||
|
'render': 'list',
|
||||||
|
'map': {
|
||||||
|
'title': 'title',
|
||||||
|
'badge': {
|
||||||
|
'label': 'severity',
|
||||||
|
'labelmap': {
|
||||||
|
'critical': 'Critical',
|
||||||
|
'warning': 'Warning',
|
||||||
|
'info': 'Info',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
'meta': [
|
||||||
|
{'key': 'eventtime', 'label': 'At', 'format': 'date'},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
'empty': 'No recent events.',
|
||||||
|
'position': 14,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
def get_dashboard_widgets(self) -> List[Dict]:
|
||||||
|
"""Fleet-wide exception cards. Run once on dashboard load, not per PC.
|
||||||
|
|
||||||
|
Both cards hide when empty and cap their rows, so a healthy fleet adds
|
||||||
|
nothing to the dashboard and an unhealthy one adds a bounded list.
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
'id': 'endpoints-quiet',
|
||||||
|
'viewall': '/pcs',
|
||||||
|
'title': 'Endpoints gone quiet',
|
||||||
|
'endpoint': '/api/endpoints/dashboard/quiet',
|
||||||
|
'render': 'exceptions',
|
||||||
|
'severity': 'warning',
|
||||||
|
'permission': 'endpoints.view',
|
||||||
|
'empty': 'hide',
|
||||||
|
'position': 22,
|
||||||
|
'map': {
|
||||||
|
'title': 'hostname',
|
||||||
|
'detail': '',
|
||||||
|
'meta': [{'key': 'quietdays', 'label': 'silent',
|
||||||
|
'suffix': ' days'}],
|
||||||
|
'link': '/pcs/{computerid}',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'id': 'endpoints-stability',
|
||||||
|
'viewall': '/pcs',
|
||||||
|
'title': 'Endpoints with crashes or shutdowns',
|
||||||
|
'endpoint': '/api/endpoints/dashboard/stability',
|
||||||
|
'render': 'exceptions',
|
||||||
|
'severity': 'critical',
|
||||||
|
'permission': 'endpoints.view',
|
||||||
|
'empty': 'hide',
|
||||||
|
'position': 16,
|
||||||
|
'map': {
|
||||||
|
'title': 'hostname',
|
||||||
|
'detail': '',
|
||||||
|
'meta': [{'key': 'eventcount', 'suffix': ' events (48h)'}],
|
||||||
|
'link': '/pcs/{computerid}',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
# ---- ADR-006 collector contract -------------------------------------
|
||||||
|
|
||||||
|
def get_collector_schema(self) -> Optional[Dict]:
|
||||||
|
return {
|
||||||
|
'identityfield': 'hostname',
|
||||||
|
'fields': {
|
||||||
|
'type': 'object',
|
||||||
|
'required': ['hostname'],
|
||||||
|
'properties': {
|
||||||
|
'hostname': {
|
||||||
|
'type': 'string',
|
||||||
|
'description': 'Endpoint hostname. Resolved against '
|
||||||
|
'the computers plugin (computers.hostname).',
|
||||||
|
},
|
||||||
|
'loggedinuser': {'type': 'string'},
|
||||||
|
'agentversion': {'type': 'string'},
|
||||||
|
'lastboottime': {'type': 'string', 'format': 'date-time'},
|
||||||
|
'uptimeseconds': {'type': 'integer'},
|
||||||
|
'idleseconds': {'type': 'integer'},
|
||||||
|
'cpupercent': {'type': 'number'},
|
||||||
|
'memorypercent': {'type': 'number'},
|
||||||
|
'services': {
|
||||||
|
'type': 'array',
|
||||||
|
'description': 'Monitored services/processes and their '
|
||||||
|
'run state (UDC, eDnc, Intune, ...).',
|
||||||
|
},
|
||||||
|
'ports': {
|
||||||
|
'type': 'array',
|
||||||
|
'description': 'Open listening ports.',
|
||||||
|
},
|
||||||
|
'interfaces': {
|
||||||
|
'type': 'array',
|
||||||
|
'description': 'Network interface information.',
|
||||||
|
},
|
||||||
|
'scheduledtasks': {
|
||||||
|
'type': 'array',
|
||||||
|
'description': 'Scheduled task inventory + last run.',
|
||||||
|
},
|
||||||
|
'events': {
|
||||||
|
'type': 'array',
|
||||||
|
'description': 'Curated Windows Event Log records. Only '
|
||||||
|
'IDs in the server catalog are stored.',
|
||||||
|
'items': {
|
||||||
|
'type': 'object',
|
||||||
|
'required': ['eventid'],
|
||||||
|
'properties': {
|
||||||
|
'eventid': {'type': 'integer'},
|
||||||
|
'eventchannel': {'type': 'string'},
|
||||||
|
'eventsource': {'type': 'string'},
|
||||||
|
'eventtime': {'type': 'string',
|
||||||
|
'format': 'date-time'},
|
||||||
|
'eventmessage': {'type': 'string'},
|
||||||
|
'fields': {'type': 'object'},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def apply_collector_payload(self, payload: Dict) -> Dict:
|
||||||
|
"""Idempotently upsert one endpoint's state and append its events.
|
||||||
|
|
||||||
|
The cached EndpointState and any EndpointEvent rows are written in the
|
||||||
|
same transaction (the dispatcher owns the commit), so a reader never
|
||||||
|
sees a snapshot without the events that produced it.
|
||||||
|
"""
|
||||||
|
from plugins.computers.models import Computer
|
||||||
|
|
||||||
|
warnings: List[str] = []
|
||||||
|
|
||||||
|
hostname = (payload.get('hostname') or '').strip()
|
||||||
|
if not hostname:
|
||||||
|
raise ValueError('hostname is required')
|
||||||
|
|
||||||
|
computer = (Computer.query
|
||||||
|
.filter(db.func.lower(Computer.hostname) == hostname.lower())
|
||||||
|
.first())
|
||||||
|
if computer is None:
|
||||||
|
raise ValueError(
|
||||||
|
'could not resolve hostname to a computer: {!r}'.format(hostname))
|
||||||
|
|
||||||
|
assetid = computer.assetid
|
||||||
|
|
||||||
|
state = EndpointState.query.filter_by(assetid=assetid).first()
|
||||||
|
action = 'updated'
|
||||||
|
if state is None:
|
||||||
|
state = EndpointState(assetid=assetid)
|
||||||
|
db.session.add(state)
|
||||||
|
action = 'created'
|
||||||
|
|
||||||
|
state.hostname = hostname
|
||||||
|
if 'loggedinuser' in payload:
|
||||||
|
state.loggedinuser = payload.get('loggedinuser')
|
||||||
|
if 'agentversion' in payload:
|
||||||
|
state.agentversion = payload.get('agentversion')
|
||||||
|
if 'lastboottime' in payload:
|
||||||
|
state.lastboottime = _parse_dt(payload.get('lastboottime'))
|
||||||
|
if 'uptimeseconds' in payload:
|
||||||
|
state.uptimeseconds = payload.get('uptimeseconds')
|
||||||
|
if 'idleseconds' in payload:
|
||||||
|
state.idleseconds = payload.get('idleseconds')
|
||||||
|
if 'cpupercent' in payload:
|
||||||
|
state.cpupercent = payload.get('cpupercent')
|
||||||
|
if 'memorypercent' in payload:
|
||||||
|
state.memorypercent = payload.get('memorypercent')
|
||||||
|
if 'services' in payload:
|
||||||
|
state.servicesjson = _as_json(payload.get('services'))
|
||||||
|
if 'ports' in payload:
|
||||||
|
state.portsjson = _as_json(payload.get('ports'))
|
||||||
|
if 'interfaces' in payload:
|
||||||
|
state.interfacesjson = _as_json(payload.get('interfaces'))
|
||||||
|
if 'scheduledtasks' in payload:
|
||||||
|
state.scheduledtasksjson = _as_json(payload.get('scheduledtasks'))
|
||||||
|
|
||||||
|
state.lastreportedat = datetime.utcnow()
|
||||||
|
|
||||||
|
stored_events = 0
|
||||||
|
for record in (payload.get('events') or []):
|
||||||
|
try:
|
||||||
|
eventid = int(record.get('eventid'))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
warnings.append('skipped event with non-integer eventid')
|
||||||
|
continue
|
||||||
|
if eventid not in CURATED_EVENT_IDS:
|
||||||
|
# Outside the catalog: ignored on purpose, not an error.
|
||||||
|
continue
|
||||||
|
db.session.add(EndpointEvent(
|
||||||
|
assetid=assetid,
|
||||||
|
eventid=eventid,
|
||||||
|
eventchannel=record.get('eventchannel'),
|
||||||
|
eventsource=record.get('eventsource'),
|
||||||
|
eventtime=_parse_dt(record.get('eventtime')),
|
||||||
|
eventmessage=record.get('eventmessage'),
|
||||||
|
payloadjson=_as_json(record.get('fields')),
|
||||||
|
))
|
||||||
|
stored_events += 1
|
||||||
|
|
||||||
|
# flush, not commit: the collector dispatcher owns the transaction.
|
||||||
|
db.session.flush()
|
||||||
|
|
||||||
|
return {
|
||||||
|
'action': action,
|
||||||
|
'assetid': assetid,
|
||||||
|
'identityvalue': hostname,
|
||||||
|
'storedevents': stored_events,
|
||||||
|
'warnings': warnings,
|
||||||
|
}
|
||||||
92
plugins/endpoints/tests/test_plugin.py
Normal file
92
plugins/endpoints/tests/test_plugin.py
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
"""Endpoints plugin smoke + collector-contract tests.
|
||||||
|
|
||||||
|
Asserts the plugin loads cleanly, satisfies the framework contract, and that
|
||||||
|
its ADR-006 collector schema is well formed. Extend with DB-backed ingest tests
|
||||||
|
as the plugin grows.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from flask import Blueprint
|
||||||
|
|
||||||
|
from plugins.endpoints.plugin import EndpointsPlugin, CURATED_EVENT_IDS
|
||||||
|
from plugins.endpoints.eventcatalog import EVENT_CATALOG, CRITICAL_EVENT_IDS
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_plugin_meta_is_valid():
|
||||||
|
"""meta returns a PluginMeta with the expected name and prefix."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
assert plugin.meta.name == 'endpoints'
|
||||||
|
assert plugin.meta.api_prefix == '/api/endpoints'
|
||||||
|
assert 'computers' in plugin.meta.dependencies
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_plugin_get_blueprint_returns_blueprint():
|
||||||
|
"""get_blueprint returns a Flask Blueprint, not None."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
assert isinstance(plugin.get_blueprint(), Blueprint)
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_plugin_get_models_returns_models():
|
||||||
|
"""get_models returns the two telemetry tables."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
models = plugin.get_models()
|
||||||
|
assert len(models) >= 2
|
||||||
|
for model in models:
|
||||||
|
assert hasattr(model, '__tablename__')
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_collector_schema_identity_is_hostname():
|
||||||
|
"""ADR-006 schema declares hostname as the identity field."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
schema = plugin.get_collector_schema()
|
||||||
|
assert schema is not None
|
||||||
|
assert schema['identityfield'] == 'hostname'
|
||||||
|
assert 'hostname' in schema['fields']['required']
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_curated_event_ids_include_core_security_ids():
|
||||||
|
"""The curated catalog covers the agreed logon/stability IDs."""
|
||||||
|
for eventid in (4624, 4625, 4634, 4672, 4697, 4698, 41, 4618, 55, 7011,
|
||||||
|
10016, 1001):
|
||||||
|
assert eventid in CURATED_EVENT_IDS
|
||||||
|
|
||||||
|
|
||||||
|
def test_curated_ids_match_the_event_catalog():
|
||||||
|
"""Ingest filter and display catalog cannot drift apart."""
|
||||||
|
assert CURATED_EVENT_IDS == frozenset(EVENT_CATALOG)
|
||||||
|
|
||||||
|
|
||||||
|
def test_critical_event_ids_are_a_subset_of_the_catalog():
|
||||||
|
"""Dashboard stability card only queries known critical IDs."""
|
||||||
|
assert set(CRITICAL_EVENT_IDS)
|
||||||
|
assert set(CRITICAL_EVENT_IDS).issubset(set(EVENT_CATALOG))
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_asset_panels_are_computer_scoped():
|
||||||
|
"""Every asset panel targets computer assets only, with an endpoint."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
panels = plugin.get_asset_panels()
|
||||||
|
assert len(panels) == 3
|
||||||
|
for panel in panels:
|
||||||
|
assert panel['assettypes'] == ['computer']
|
||||||
|
assert panel['endpoint'].startswith('/api/endpoints/asset/{assetid}')
|
||||||
|
assert panel['render'] in ('keyvalue', 'list')
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_dashboard_widgets_declared():
|
||||||
|
"""Quiet + stability exception cards are declared and hide when empty."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
widgets = plugin.get_dashboard_widgets()
|
||||||
|
ids = {w['id'] for w in widgets}
|
||||||
|
assert {'endpoints-quiet', 'endpoints-stability'} <= ids
|
||||||
|
for widget in widgets:
|
||||||
|
assert widget['render'] == 'exceptions'
|
||||||
|
assert widget['empty'] == 'hide'
|
||||||
|
assert widget['permission'] == 'endpoints.view'
|
||||||
|
|
||||||
|
|
||||||
|
def test_endpoints_navigation_item_points_at_fleet_view():
|
||||||
|
"""The fleet list view is exposed as a single /endpoints nav item."""
|
||||||
|
plugin = EndpointsPlugin()
|
||||||
|
items = plugin.get_navigation_items()
|
||||||
|
routes = {item['route'] for item in items}
|
||||||
|
assert '/endpoints' in routes
|
||||||
239
scripts/dev/seed_endpoints_sample.py
Normal file
239
scripts/dev/seed_endpoints_sample.py
Normal file
@@ -0,0 +1,239 @@
|
|||||||
|
"""Seed sample endpoint telemetry for the Fleet module (DEV ONLY).
|
||||||
|
|
||||||
|
Attaches EndpointState snapshots plus a handful of curated EndpointEvents to
|
||||||
|
existing Computer assets so the fleet list, the PC-detail asset panels
|
||||||
|
(status / services / recent events), and the dashboard quiet + stability cards
|
||||||
|
have something to render before a real agent exists.
|
||||||
|
|
||||||
|
Sample rows are marked with a sentinel agentversion ("0.0.0-sample") so they can
|
||||||
|
be listed and cleared without touching real agent data.
|
||||||
|
|
||||||
|
DEV ONLY: this file lives under scripts/dev/ and must never ship in a
|
||||||
|
production image (see devnotes/live-container-and-testing-workflow.md). Do not
|
||||||
|
import it from application code.
|
||||||
|
|
||||||
|
Usage (in the API container):
|
||||||
|
docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py
|
||||||
|
docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py --clear
|
||||||
|
docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py --limit 8
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
# Repo root on sys.path so `shopdb` and `plugins` import when run as a script.
|
||||||
|
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')))
|
||||||
|
|
||||||
|
from shopdb import create_app # noqa: E402
|
||||||
|
from shopdb.api import db # noqa: E402
|
||||||
|
|
||||||
|
SAMPLE_MARKER = '0.0.0-sample'
|
||||||
|
|
||||||
|
|
||||||
|
def _naive_utcnow():
|
||||||
|
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
|
def _services(*running_then_stopped):
|
||||||
|
"""Build a services snapshot. Positional pairs are (name, state)."""
|
||||||
|
return [{'name': name, 'state': state} for name, state in running_then_stopped]
|
||||||
|
|
||||||
|
|
||||||
|
def _ports(*pairs):
|
||||||
|
return [{'port': port, 'process': process} for port, process in pairs]
|
||||||
|
|
||||||
|
|
||||||
|
def _interfaces(ip, mac):
|
||||||
|
return [{'name': 'Ethernet', 'ip': ip, 'mac': mac}]
|
||||||
|
|
||||||
|
|
||||||
|
# Per-machine telemetry shapes. minutesago drives whether a machine reads as
|
||||||
|
# fresh, stale, or quiet; a couple are pushed past a plausible quiet window so
|
||||||
|
# the dashboard quiet card has rows. Everything is offline-plausible because
|
||||||
|
# there is no live agent.
|
||||||
|
SAMPLE_PROFILES = [
|
||||||
|
{
|
||||||
|
'loggedinuser': 'CORP\\jmartin',
|
||||||
|
'agentversion': SAMPLE_MARKER,
|
||||||
|
'minutesago': 3,
|
||||||
|
'uptimeseconds': 4 * 86400 + 6 * 3600,
|
||||||
|
'idleseconds': 12 * 60,
|
||||||
|
'cpupercent': 14.0,
|
||||||
|
'memorypercent': 46.0,
|
||||||
|
'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'),
|
||||||
|
('WinRM', 'Running'), ('MSSQLSERVER', 'Running')),
|
||||||
|
'ports': _ports((3389, 'svchost'), (5985, 'winrm'), (445, 'System')),
|
||||||
|
'interfaces': _interfaces('10.20.4.11', '00:1A:2B:3C:4D:01'),
|
||||||
|
'events': [(4624, -8), (4672, -8), (4698, -120)],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'loggedinuser': 'CORP\\operator2',
|
||||||
|
'agentversion': SAMPLE_MARKER,
|
||||||
|
'minutesago': 9,
|
||||||
|
'uptimeseconds': 2 * 3600 + 40 * 60,
|
||||||
|
'idleseconds': 3 * 60,
|
||||||
|
'cpupercent': 61.0,
|
||||||
|
'memorypercent': 72.0,
|
||||||
|
'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'),
|
||||||
|
('WinRM', 'Stopped')),
|
||||||
|
'ports': _ports((3389, 'svchost'), (445, 'System')),
|
||||||
|
'interfaces': _interfaces('10.20.4.27', '00:1A:2B:3C:4D:02'),
|
||||||
|
# Recent unexpected shutdown -> shows on the stability card.
|
||||||
|
'events': [(41, -50), (1001, -30), (4625, -12)],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'loggedinuser': None,
|
||||||
|
'agentversion': SAMPLE_MARKER,
|
||||||
|
'minutesago': 22,
|
||||||
|
'uptimeseconds': 11 * 86400,
|
||||||
|
'idleseconds': 9 * 3600,
|
||||||
|
'cpupercent': 3.0,
|
||||||
|
'memorypercent': 38.0,
|
||||||
|
'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'),
|
||||||
|
('WinRM', 'Running')),
|
||||||
|
'ports': _ports((3389, 'svchost'), (5985, 'winrm')),
|
||||||
|
'interfaces': _interfaces('10.20.4.44', '00:1A:2B:3C:4D:03'),
|
||||||
|
'events': [(4624, -600), (7011, -180)],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'loggedinuser': 'CORP\\nightshift',
|
||||||
|
'agentversion': SAMPLE_MARKER,
|
||||||
|
# Quiet: no report in over a day.
|
||||||
|
'minutesago': 1500,
|
||||||
|
'uptimeseconds': 30 * 60,
|
||||||
|
'idleseconds': 30 * 60,
|
||||||
|
'cpupercent': 1.0,
|
||||||
|
'memorypercent': 29.0,
|
||||||
|
'services': _services(('Spooler', 'Stopped'), ('W32Time', 'Running')),
|
||||||
|
'ports': _ports((445, 'System'),),
|
||||||
|
'interfaces': _interfaces('10.20.4.58', '00:1A:2B:3C:4D:04'),
|
||||||
|
'events': [(55, -1600), (41, -1580)],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
'loggedinuser': 'CORP\\qa_kiosk',
|
||||||
|
'agentversion': SAMPLE_MARKER,
|
||||||
|
'minutesago': 6,
|
||||||
|
'uptimeseconds': 6 * 3600,
|
||||||
|
'idleseconds': 45 * 60,
|
||||||
|
'cpupercent': 22.0,
|
||||||
|
'memorypercent': 55.0,
|
||||||
|
'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'),
|
||||||
|
('WinRM', 'Running'), ('BITS', 'Running')),
|
||||||
|
'ports': _ports((3389, 'svchost'), (5985, 'winrm'), (135, 'rpc')),
|
||||||
|
'interfaces': _interfaces('10.20.4.63', '00:1A:2B:3C:4D:05'),
|
||||||
|
'events': [(4624, -20), (10016, -90), (4697, -240)],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _clear(EndpointState, EndpointEvent):
|
||||||
|
sample_states = EndpointState.query.filter_by(agentversion=SAMPLE_MARKER).all()
|
||||||
|
asset_ids = [s.assetid for s in sample_states]
|
||||||
|
events_removed = 0
|
||||||
|
if asset_ids:
|
||||||
|
events_removed = (EndpointEvent.query
|
||||||
|
.filter(EndpointEvent.assetid.in_(asset_ids))
|
||||||
|
.delete(synchronize_session=False))
|
||||||
|
states_removed = (EndpointState.query
|
||||||
|
.filter_by(agentversion=SAMPLE_MARKER)
|
||||||
|
.delete(synchronize_session=False))
|
||||||
|
db.session.commit()
|
||||||
|
print('Cleared {} sample state(s) and {} sample event(s).'
|
||||||
|
.format(states_removed, events_removed))
|
||||||
|
|
||||||
|
|
||||||
|
def _seed(limit):
|
||||||
|
from plugins.computers.models import Computer
|
||||||
|
from plugins.endpoints.models import EndpointState, EndpointEvent
|
||||||
|
from plugins.endpoints.eventcatalog import EVENT_CATALOG
|
||||||
|
|
||||||
|
computers = (Computer.query
|
||||||
|
.filter(Computer.assetid.isnot(None))
|
||||||
|
.order_by(Computer.computerid.asc())
|
||||||
|
.limit(limit)
|
||||||
|
.all())
|
||||||
|
if not computers:
|
||||||
|
print('No Computer assets found to attach sample endpoints to. '
|
||||||
|
'Seed the computers plugin first.')
|
||||||
|
return
|
||||||
|
|
||||||
|
now = _naive_utcnow()
|
||||||
|
created, updated, events_added = 0, 0, 0
|
||||||
|
|
||||||
|
for index, computer in enumerate(computers):
|
||||||
|
profile = SAMPLE_PROFILES[index % len(SAMPLE_PROFILES)]
|
||||||
|
reported = now - timedelta(minutes=profile['minutesago'])
|
||||||
|
boot = reported - timedelta(seconds=profile['uptimeseconds'])
|
||||||
|
|
||||||
|
state = EndpointState.query.filter_by(assetid=computer.assetid).first()
|
||||||
|
if state and state.agentversion not in (None, SAMPLE_MARKER):
|
||||||
|
# Never stomp a real agent's snapshot.
|
||||||
|
continue
|
||||||
|
if not state:
|
||||||
|
state = EndpointState(assetid=computer.assetid, createdat=now)
|
||||||
|
db.session.add(state)
|
||||||
|
created += 1
|
||||||
|
else:
|
||||||
|
updated += 1
|
||||||
|
|
||||||
|
state.hostname = computer.hostname or 'HOST-{}'.format(computer.computerid)
|
||||||
|
state.loggedinuser = profile['loggedinuser']
|
||||||
|
state.agentversion = SAMPLE_MARKER
|
||||||
|
state.lastboottime = boot
|
||||||
|
state.uptimeseconds = profile['uptimeseconds']
|
||||||
|
state.idleseconds = profile['idleseconds']
|
||||||
|
state.cpupercent = profile['cpupercent']
|
||||||
|
state.memorypercent = profile['memorypercent']
|
||||||
|
state.servicesjson = json.dumps(profile['services'])
|
||||||
|
state.portsjson = json.dumps(profile['ports'])
|
||||||
|
state.interfacesjson = json.dumps(profile['interfaces'])
|
||||||
|
state.scheduledtasksjson = json.dumps([])
|
||||||
|
state.lastreportedat = reported
|
||||||
|
|
||||||
|
# Refresh this asset's sample events so re-running stays idempotent.
|
||||||
|
EndpointEvent.query.filter_by(assetid=computer.assetid).delete(
|
||||||
|
synchronize_session=False)
|
||||||
|
for eventid, offset_minutes in profile['events']:
|
||||||
|
title, severity, channel = EVENT_CATALOG.get(
|
||||||
|
eventid, ('Event {}'.format(eventid), 'info', 'System'))
|
||||||
|
event = EndpointEvent(
|
||||||
|
assetid=computer.assetid,
|
||||||
|
eventid=eventid,
|
||||||
|
eventchannel=channel,
|
||||||
|
eventsource='sample-seed',
|
||||||
|
eventtime=reported + timedelta(minutes=offset_minutes),
|
||||||
|
eventmessage='{} (sample)'.format(title),
|
||||||
|
payloadjson=json.dumps({'sample': True, 'severity': severity}),
|
||||||
|
recordedat=now,
|
||||||
|
)
|
||||||
|
db.session.add(event)
|
||||||
|
events_added += 1
|
||||||
|
|
||||||
|
db.session.commit()
|
||||||
|
print('Seeded sample endpoints: {} created, {} updated, {} event(s) added '
|
||||||
|
'across {} computer(s).'
|
||||||
|
.format(created, updated, events_added, len(computers)))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument('--clear', action='store_true',
|
||||||
|
help='Remove sample-marked endpoint states and their events.')
|
||||||
|
parser.add_argument('--limit', type=int, default=5,
|
||||||
|
help='Max number of computers to attach sample endpoints to.')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
app = create_app(os.environ.get('FLASK_ENV', 'production'))
|
||||||
|
with app.app_context():
|
||||||
|
from plugins.endpoints.models import EndpointState, EndpointEvent
|
||||||
|
if args.clear:
|
||||||
|
_clear(EndpointState, EndpointEvent)
|
||||||
|
else:
|
||||||
|
_seed(args.limit)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user