diff --git a/docs/DESIGN-DECISIONS-agent.md b/docs/DESIGN-DECISIONS-agent.md new file mode 100644 index 00000000..67e2da4b --- /dev/null +++ b/docs/DESIGN-DECISIONS-agent.md @@ -0,0 +1,155 @@ +# Agent Design Decisions (running preference list) + +Server-side decisions made while building the `endpoints` plugin that the +future C# agent (and the ServiceNow ticketing surface) must honor. This is a +living document: add to it as the agent build raises questions. ASCII only, no +em-dashes, no emojis, per repo style. + +## Scope of this document + +The `endpoints` plugin (server) is the authoritative contract. The C# agent is +a separate deliverable in its own solution/repo. Anything the agent sends must +match what this plugin ingests; when the two disagree, this plugin wins and the +agent changes. + +## 1. Transport and identity + +- Agent posts one JSON object per host to `POST /api/collector/endpoints` + (the generic ADR-006 collector route, auto-registered by the core loader). +- Auth is a collector-scoped credential in the request HEADER, not a user JWT + and not a querystring. Server reads `COLLECTOR_API_KEY_ENDPOINTS` first, then + falls back to `COLLECTOR_API_KEY`. Provision the key the same way the + computers reporter does: out of band, e.g. `HKLM:\SOFTWARE\GE\ShopDB + CollectorKey`. The key is never baked into a distributed script/binary. +- Identity field is `hostname` (case-insensitive). The server resolves it to an + existing computer asset via the `computers` plugin. An unknown hostname is + REJECTED with a validation error, never auto-created. Decision: the fleet is + provisioned in ShopDB first (imports/Azure device sync); the agent does not + get to invent assets. Revisit only if bulk self-registration becomes a need. + +## 2. Report shape (current server contract) + +Top-level payload fields the server currently reads (see +`plugins/endpoints/plugin.py` `get_collector_schema`): + +- `hostname` (required), `loggedinuser`, `agentversion` +- Metrics: `lastboottime` (ISO-8601), `uptimeseconds`, `idleseconds`, + `cpupercent`, `memorypercent` +- Inventory arrays (stored as latest-snapshot JSON, not history): + `services`, `ports`, `interfaces`, `scheduledtasks` +- `events`: array of curated Windows Event Log records (see section 4) + +Decision: inventory and metrics are a SNAPSHOT overwrite (one `EndpointState` +row per asset), not time-series. The agent sends the full current list each +cycle; the server does not diff or version them. If per-sample history is later +required, it goes in a new append-only table and the contract stays additive. + +## 3. Reporting rhythm + +- Default report interval: 15 minutes (`report_interval_minutes` in the plugin + manifest settings; a site setting can override). Agent design should make the + interval configurable and jittered to avoid a fleet-wide thundering herd. +- There is NO separate heartbeat. `lastreportedat` is stamped on every report; + "quiet" endpoints are those past the quiet-report window (default 24h, site + setting `endpoints_quietreporthours`). Agent does not need a distinct ping. + +## 4. Curated Windows Event catalog (LOCKED to the server list) + +The server stores ONLY these IDs; anything else in `events` is ignored (not an +error). Source of truth: `CURATED_EVENT_IDS` in `plugins/endpoints/plugin.py`. + +| Event ID | Channel | Meaning | +|----------|--------------|--------------------------------------------| +| 4624 | Security | Successful logon | +| 4625 | Security | Failed logon | +| 4634 | Security | Logoff | +| 4672 | Security | Special privileges assigned to new logon | +| 4697 | Security | Service installed | +| 4698 | Security | Scheduled task created | +| 41 | System | Kernel-Power, unexpected shutdown | +| 4618 | Security | Monitored security event pattern | +| 55 | System | NTFS file system corruption | +| 7011 | System | Service Control Manager timeout | +| 10016 | System | DCOM permission error | +| 1001 | Application | Application error / bugcheck report | + +- Per event, the agent sends: `eventid` (required), `eventchannel`, + `eventsource`, `eventtime` (ISO-8601, event time not report time), + `eventmessage`, and `fields` (an object of extracted structured values, e.g. + target account on a 4625). `fields` is stored as `payloadjson`. +- OPEN QUESTION for the agent build: which IDs need structured `fields` parsing + vs. raw message pass-through. Start with pass-through message; add structured + parsing per ID as alerting rules demand it. The teammate flagged event parsing + as the hard part; keep the catalog fixed and small until baselined. +- Scheduled-task history being disabled on some hosts means 4698 and task + "last run" may be sparse; the agent should read current task state directly, + not rely on event history alone. + +## 5. Monitored services/processes + +Known targets to watch (non-exhaustive, confirm during agent build): UDC, eDnc, +Intune-related services. Server stores whatever list the agent sends in +`servicesjson` without a fixed schema yet. Decision deferred: once the watched +set is final, consider promoting it to a server-known enum for alerting. + +## 6. ServiceNow ticketing surface (future, creds confirmed available) + +Not built server-side yet. Recorded here so the agent and a future server +endpoint agree: + +- Interactive C# piece (system-tray/desktop launcher) owns the 5-click-max flow: + network precheck -> issue dropdown -> optional screenshot -> submit. +- Fixed dropdown reasons (from the source conversation): Can't logon to CSF; + Data collections down; Can't download program; Machine communication failing; + Network drives down; Other (free text). +- Screenshot saved to local `Pictures\Screenshots` before submission. +- OPEN: whether ticket submission goes direct to the ServiceNow API from the + agent, or is proxied through a ShopDB server endpoint (preferred: proxy, so + ServiceNow creds live server-side and the agent carries only its collector + key). Decide before wiring the integration. + +## 7. Runtime split (from the planning conversation) + +- Interactive bits (tray launcher, ticketing, screenshot): C# / .NET, likely a + Windows service + tray app. +- Collections MAY be locally hosted PowerShell scripts the service invokes, + where that is simpler than reimplementing WMI/CIM queries in C#. +- Code signing and Intune deployment are agent-side concerns, out of scope for + the server plugin but must be planned before fleet rollout. + +## 8. Web UI display (server-side, built) + +The collected data surfaces through ShopDB's existing declaration-driven UI +contract (ADR-010). The plugin declares WHAT to show; generic core components +render it. No plugin-specific Vue components were written. Audience is DT +support / IT, NOT the shop-floor operator (the operator's surface is the +agent-side ticket tool). + +Surfaces implemented: + +- Per-PC asset panels (computer assets only), on the PC detail page: + - Endpoint status (keyvalue): logged-in user, uptime, idle, last boot, + CPU/memory %, agent version, last reported. Feeds + `GET /api/endpoints/asset/{assetid}/status`. + - Services and processes (list + badge): monitored services with a green/red + run-state badge. Feeds `.../services`. + - Recent events (list + severity badge): last 10 curated events. Feeds + `.../recentevents`. +- Dashboard exception cards (hide when empty): + - Endpoints gone quiet (warning): `.../dashboard/quiet`. + - Endpoints with crashes or shutdowns (critical): `.../dashboard/stability`, + bounded to a 48h window and the critical event catalog. + +DECISION - freshness and cost. There is NO live streaming and NO polling. The +freshness ceiling is the agent report cadence (~15 min), so a websocket/poll +buys nothing but load. Data is fetched on page/dashboard load only, matching the +existing computers-quiet card. Asset panels are single-asset indexed reads +(negligible). Dashboard cards run once per dashboard load, cap at 50 rows, and +the stability card is time-bounded so it does not grow with total history. If +fleet-wide aggregation ever gets heavy, memoize dashboard aggregates with the +core `cache` helper for ~60s - documented lever, not built yet. + +Agent implication: the agent does not drive any UI refresh. Its only job is to +report on its rhythm; the UI reflects whatever the last report wrote. The +`services` payload should send a per-service run STATE (e.g. running/stopped) so +the badge can color it; a missing/unknown state renders as a gray Unknown badge. diff --git a/frontend/src/api/index.js b/frontend/src/api/index.js index 45db2f13..f323ff10 100644 --- a/frontend/src/api/index.js +++ b/frontend/src/api/index.js @@ -146,6 +146,16 @@ export const machinesApi = { } } +// Endpoints API (plugin) - fleet telemetry read surface +export const endpointsApi = { + list(params = {}) { + return api.get('/endpoints', { params }) + }, + get(assetId) { + return api.get(`/endpoints/${assetId}`) + } +} + // Computers API (plugin) export const computersApi = { list(params = {}) { diff --git a/frontend/src/views/AppLayout.vue b/frontend/src/views/AppLayout.vue index 19b4a8ce..d110bf8f 100644 --- a/frontend/src/views/AppLayout.vue +++ b/frontend/src/views/AppLayout.vue @@ -109,7 +109,7 @@ import ToastHost from '../components/ToastHost.vue' import { Sun, Moon, LayoutDashboard, Calendar, Map, Cog, Monitor, Printer, Globe, Usb, AppWindow, BookOpen, BarChart3, Bell, Image, ShieldCheck, Ruler, - Box, KeyRound, LogOut, Wrench + Box, KeyRound, LogOut, Wrench, Activity } from 'lucide-vue-next' import { useAuthStore } from '../stores/auth' import { currentTheme, toggleTheme } from '../stores/theme' @@ -178,6 +178,7 @@ const iconMap = { 'ruler': Ruler, 'box': Box, 'wrench': Wrench, + 'activity': Activity, } // Default navigation (used as fallback if API fails) diff --git a/plugins/endpoints/__init__.py b/plugins/endpoints/__init__.py new file mode 100644 index 00000000..f33cbea9 --- /dev/null +++ b/plugins/endpoints/__init__.py @@ -0,0 +1,5 @@ +"""Endpoints plugin package.""" + +from .plugin import EndpointsPlugin + +__all__ = ['EndpointsPlugin'] diff --git a/plugins/endpoints/api/__init__.py b/plugins/endpoints/api/__init__.py new file mode 100644 index 00000000..6ad50266 --- /dev/null +++ b/plugins/endpoints/api/__init__.py @@ -0,0 +1,5 @@ +"""Endpoints plugin API.""" + +from .routes import endpoints_bp + +__all__ = ['endpoints_bp'] diff --git a/plugins/endpoints/api/routes.py b/plugins/endpoints/api/routes.py new file mode 100644 index 00000000..650ad040 --- /dev/null +++ b/plugins/endpoints/api/routes.py @@ -0,0 +1,350 @@ +"""Endpoints plugin API endpoints (read + dashboard). + +Ingest is NOT here: agent reports arrive on the generic collector route +POST /api/collector/endpoints (ADR-006), authorized by a collector-scoped +service token, and are handled by EndpointsPlugin.apply_collector_payload. +These routes are the human-facing read surface, gated by user JWT + RBAC. +""" + +import json +from datetime import datetime, timedelta, timezone + +from flask import Blueprint, request +from flask_jwt_extended import jwt_required + +from shopdb.api import ( + db, Asset, Setting, success_response, error_response, paginated_response, + ErrorCodes, get_pagination_params, paginate_query, require_permission, +) + +from ..models import EndpointState, EndpointEvent +from ..eventcatalog import CRITICAL_EVENT_IDS, event_severity, event_title + +endpoints_bp = Blueprint('endpoints', __name__) + + +def _naive_utcnow(): + return datetime.now(timezone.utc).replace(tzinfo=None) + + +def _format_uptime(seconds): + """Whole-seconds uptime -> 'Nd Nh Nm', or None.""" + if seconds is None: + return None + try: + seconds = int(seconds) + except (TypeError, ValueError): + return None + days, rem = divmod(seconds, 86400) + hours, rem = divmod(rem, 3600) + minutes = rem // 60 + parts = [] + if days: + parts.append('{}d'.format(days)) + if hours: + parts.append('{}h'.format(hours)) + parts.append('{}m'.format(minutes)) + return ' '.join(parts) + + +# ============================================================================= +# Endpoint state +# ============================================================================= + +@endpoints_bp.route('/', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def list_endpoints(): + """List reported endpoint states. + + Joins through Computer so each row carries computerid and can deep-link + into the existing PC detail page (/pcs/{computerid}). Endpoint state only + exists for hostnames resolved to a Computer, so the join is not lossy. + """ + from plugins.computers.models import Computer + + page, per_page = get_pagination_params(request) + + query = (db.session.query(EndpointState, Computer.computerid) + .join(Asset, Asset.assetid == EndpointState.assetid) + .join(Computer, Computer.assetid == EndpointState.assetid) + .filter(Asset.isactive.is_(True))) + + if search := request.args.get('search'): + query = query.filter(EndpointState.hostname.ilike(f'%{search}%')) + + query = query.order_by(EndpointState.lastreportedat.desc()) + + items, total = paginate_query(query, page, per_page) + data = [] + for state, computerid in items: + row = state.to_dict() + row['computerid'] = computerid + row['uptimedisplay'] = _format_uptime(state.uptimeseconds) + data.append(row) + + return paginated_response(data, page, per_page, total) + + +@endpoints_bp.route('/', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def get_endpoint(asset_id: int): + """Get the current state for one endpoint by asset id.""" + state = EndpointState.query.filter_by(assetid=asset_id).first() + + if not state: + return error_response( + ErrorCodes.NOT_FOUND, + f'No endpoint state for asset {asset_id}', + http_code=404, + ) + + return success_response(state.to_dict()) + + +@endpoints_bp.route('//events', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def list_endpoint_events(asset_id: int): + """List curated Windows events for one endpoint, newest first.""" + page, per_page = get_pagination_params(request) + + query = EndpointEvent.query.filter_by(assetid=asset_id) + + if eventid := request.args.get('eventid'): + try: + query = query.filter(EndpointEvent.eventid == int(eventid)) + except (TypeError, ValueError): + return error_response(ErrorCodes.VALIDATION_ERROR, + 'eventid must be an integer') + + query = query.order_by(EndpointEvent.eventtime.desc()) + + items, total = paginate_query(query, page, per_page) + data = [e.to_dict() for e in items] + + return paginated_response(data, page, per_page, total) + + +# ============================================================================= +# Asset panels (ADR-010) - shaped for the generic PluginAssetPanels renderer +# ============================================================================= + +@endpoints_bp.route('/asset//status', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def panel_status(asset_id: int): + """Keyvalue panel: the endpoint's current status snapshot. + + Single indexed lookup on endpointstates.assetid. Returns the {fields:[...]} + shape the keyvalue renderer expects; an absent state yields no fields so the + panel hides itself. + """ + state = EndpointState.query.filter_by(assetid=asset_id).first() + if not state: + return success_response({'fields': []}) + + def field(label, value, mono=False): + return {'label': label, 'value': value, 'mono': mono} + + fields = [ + field('Logged in user', state.loggedinuser), + field('Uptime', _format_uptime(state.uptimeseconds)), + field('Idle', _format_uptime(state.idleseconds)), + field('Last boot', + state.lastboottime.isoformat() + 'Z' if state.lastboottime else None), + field('CPU %', state.cpupercent), + field('Memory %', state.memorypercent), + field('Agent', state.agentversion, mono=True), + field('Last reported', + state.lastreportedat.isoformat() + 'Z' if state.lastreportedat else None), + ] + fields = [f for f in fields if f['value'] not in (None, '')] + return success_response({'fields': fields}) + + +@endpoints_bp.route('/asset//services', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def panel_services(asset_id: int): + """List panel: monitored services/processes with a run-state badge per row. + + Reshapes the latest servicesjson snapshot into rows the list renderer maps + to name + status badge. Tolerant of a missing or unexpected snapshot shape: + an unknown state renders as an Unknown badge rather than failing. + """ + state = EndpointState.query.filter_by(assetid=asset_id).first() + if not state or not state.servicesjson: + return success_response([]) + + try: + services = json.loads(state.servicesjson) + except (TypeError, ValueError): + return success_response([]) + + rows = [] + for entry in (services or []): + if isinstance(entry, dict): + name = entry.get('name') or entry.get('service') or 'Unknown' + raw = (entry.get('state') or entry.get('status') or '').strip().lower() + else: + name, raw = str(entry), '' + running = raw in ('running', 'started', 'up', 'ok', 'active') + rows.append({ + 'name': name, + 'statelabel': 'Running' if running else (raw.title() or 'Unknown'), + 'statecolor': 'green' if running else ('red' if raw else 'gray'), + }) + return success_response(rows) + + +@endpoints_bp.route('/asset//recentevents', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def panel_recent_events(asset_id: int): + """List panel: the endpoint's most recent curated events, newest first. + + Indexed range scan on (assetid, eventtime), capped small - this is the + at-a-glance history card, not the full report (that is the events route + with pagination). + """ + events = (EndpointEvent.query + .filter_by(assetid=asset_id) + .order_by(EndpointEvent.eventtime.desc()) + .limit(10) + .all()) + + rows = [{ + 'title': event_title(e.eventid), + 'severity': event_severity(e.eventid), + 'eventtime': e.eventtime.isoformat() + 'Z' if e.eventtime else None, + } for e in events] + return success_response(rows) + + +# ============================================================================= +# Dashboard +# ============================================================================= + +@endpoints_bp.route('/dashboard/summary', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def dashboard_summary(): + """High-level fleet counts for the endpoints dashboard.""" + total = db.session.query(EndpointState).join( + Asset, Asset.assetid == EndpointState.assetid + ).filter(Asset.isactive.is_(True)).count() + + hours = _quiet_hours() + cutoff = _naive_utcnow() - timedelta(hours=hours) + + reporting = db.session.query(EndpointState).join( + Asset, Asset.assetid == EndpointState.assetid + ).filter( + Asset.isactive.is_(True), + EndpointState.lastreportedat.isnot(None), + EndpointState.lastreportedat >= cutoff, + ).count() + + return success_response({ + 'total': total, + 'reporting': reporting, + 'quiet': total - reporting, + 'quiethours': hours, + }) + + +@endpoints_bp.route('/dashboard/quiet', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def dashboard_quiet(): + """Endpoints that reported before, then went quiet past the window. + + Never-reported endpoints are excluded: those are records without an agent, + not agents that stopped. Silence is the only signal - there is no heartbeat + separate from the report. + """ + hours = _quiet_hours() + cutoff = _naive_utcnow() - timedelta(hours=hours) + now = _naive_utcnow() + + from plugins.computers.models import Computer + + rows = (db.session.query(EndpointState, Computer.computerid) + .join(Asset, Asset.assetid == EndpointState.assetid) + .join(Computer, Computer.assetid == EndpointState.assetid) + .filter(Asset.isactive.is_(True), + EndpointState.lastreportedat.isnot(None), + EndpointState.lastreportedat < cutoff) + .all()) + + out = [{ + 'computerid': computerid, + 'assetid': s.assetid, + 'hostname': s.hostname, + 'lastreported': s.lastreportedat.isoformat() + 'Z', + 'quietdays': (now - s.lastreportedat).days, + } for s, computerid in rows] + + out.sort(key=lambda r: -r['quietdays']) + return success_response(out[:50]) + + +@endpoints_bp.route('/dashboard/stability', methods=['GET']) +@jwt_required() +@require_permission('endpoints.view') +def dashboard_stability(): + """Endpoints with recent crash/shutdown/corruption events. + + Bounded to a fixed 48h window and the critical event catalog, capped at 50 + rows: cost is a range scan on eventtime filtered to a short id list, not a + scan of all history. Joins through Computer so each row links to the PC + detail page. + """ + from plugins.computers.models import Computer + + if not CRITICAL_EVENT_IDS: + return success_response([]) + + cutoff = _naive_utcnow() - timedelta(hours=48) + + rows = (db.session.query( + Computer.computerid, + EndpointState.hostname, + db.func.count(EndpointEvent.endpointeventid), + db.func.max(EndpointEvent.eventtime), + ) + .select_from(EndpointEvent) + .join(Asset, Asset.assetid == EndpointEvent.assetid) + .join(Computer, Computer.assetid == EndpointEvent.assetid) + .outerjoin(EndpointState, + EndpointState.assetid == EndpointEvent.assetid) + .filter(Asset.isactive.is_(True), + EndpointEvent.eventid.in_(CRITICAL_EVENT_IDS), + EndpointEvent.eventtime.isnot(None), + EndpointEvent.eventtime >= cutoff) + .group_by(Computer.computerid, EndpointState.hostname) + .all()) + + out = [{ + 'computerid': computerid, + 'hostname': hostname, + 'eventcount': count, + 'lastevent': lasttime.isoformat() + 'Z' if lasttime else None, + } for computerid, hostname, count, lasttime in rows] + + out.sort(key=lambda r: -r['eventcount']) + return success_response(out[:50]) + + +def _quiet_hours() -> int: + """Quiet-report window in hours, from a site setting, default 24.""" + hours = 24 + setting = Setting.query.filter_by(key='endpoints_quietreporthours').first() + if setting and (setting.value or '').strip(): + try: + hours = int(setting.value) + except (TypeError, ValueError): + pass + return hours diff --git a/plugins/endpoints/eventcatalog.py b/plugins/endpoints/eventcatalog.py new file mode 100644 index 00000000..5f6c8903 --- /dev/null +++ b/plugins/endpoints/eventcatalog.py @@ -0,0 +1,46 @@ +"""Curated Windows event catalog: id -> human title and severity. + +Single server-side source of truth shared by the recent-events asset panel and +the stability dashboard card, so a 4625 reads as "Failed logon (warning)" in +both places without either restating the mapping. The set of IDs here MUST stay +in step with CURATED_EVENT_IDS in plugin.py; a test asserts they match. + +Severity vocabulary matches the dashboard card renderer: 'critical', 'warning', +'info'. The IDs marked critical are the ones an operator/tech should act on +without being asked - unexpected shutdown, crash, filesystem corruption, SCM +timeout. Logon activity is informational history unless it is a failure. +""" + +# id -> (title, severity, channel) +EVENT_CATALOG = { + 4624: ('Successful logon', 'info', 'Security'), + 4625: ('Failed logon', 'warning', 'Security'), + 4634: ('Logoff', 'info', 'Security'), + 4672: ('Special privileges assigned', 'info', 'Security'), + 4697: ('Service installed', 'warning', 'Security'), + 4698: ('Scheduled task created', 'warning', 'Security'), + 41: ('Unexpected shutdown', 'critical', 'System'), + 4618: ('Monitored security event', 'warning', 'Security'), + 55: ('File system corruption', 'critical', 'System'), + 7011: ('Service timeout', 'critical', 'System'), + 10016: ('DCOM permission error', 'warning', 'System'), + 1001: ('Application crash', 'critical', 'Application'), +} + +# The IDs a tech should see surfaced fleet-wide on the dashboard stability card. +CRITICAL_EVENT_IDS = tuple( + eventid for eventid, (_t, severity, _c) in EVENT_CATALOG.items() + if severity == 'critical' +) + + +def event_title(eventid): + """Human title for a curated event id, or a generic fallback.""" + entry = EVENT_CATALOG.get(eventid) + return entry[0] if entry else 'Event {}'.format(eventid) + + +def event_severity(eventid): + """Severity ('critical'|'warning'|'info') for a curated event id.""" + entry = EVENT_CATALOG.get(eventid) + return entry[1] if entry else 'info' diff --git a/plugins/endpoints/frontend/routes.js b/plugins/endpoints/frontend/routes.js new file mode 100644 index 00000000..1d51f0e2 --- /dev/null +++ b/plugins/endpoints/frontend/routes.js @@ -0,0 +1,20 @@ +/** + * Endpoints plugin routes + * + * Groundwork for the fleet view. There is no endpoint-specific detail route + * yet: rows link into the existing PC detail page (/pcs/:id) so telemetry + * stays interoperable with the computers plugin instead of duplicating it. + */ +export default [ + { + path: 'endpoints', + name: 'endpoints', + component: () => import('./views/EndpointsList.vue'), + meta: { plugin: 'endpoints' } + } +] + +// No full-screen routes: every view in this plugin renders inside +// AppLayout. Declared so routes.gen.js can read `.toplevel` off the module +// without Rollup warning about a missing export. +export const toplevel = [] diff --git a/plugins/endpoints/frontend/views/EndpointsList.vue b/plugins/endpoints/frontend/views/EndpointsList.vue new file mode 100644 index 00000000..802ccbef --- /dev/null +++ b/plugins/endpoints/frontend/views/EndpointsList.vue @@ -0,0 +1,153 @@ + + + + + diff --git a/plugins/endpoints/manifest.json b/plugins/endpoints/manifest.json new file mode 100644 index 00000000..3b8c3fdf --- /dev/null +++ b/plugins/endpoints/manifest.json @@ -0,0 +1,21 @@ +{ + "name": "endpoints", + "version": "0.1.0", + "description": "Endpoint RMM telemetry: metrics, services, inventory, and curated Windows event records reported by the ShopDB agent", + "author": "ShopDB Team", + "dependencies": ["computers"], + "core_version": ">=0.20.0,<1.0.0", + "api_prefix": "/api/endpoints", + "provides": { + "features": [ + "endpoint_metrics", + "service_state", + "inventory_snapshot", + "event_records" + ] + }, + "settings": { + "report_interval_minutes": 15, + "quiet_report_hours": 24 + } +} diff --git a/plugins/endpoints/migrations/env.py b/plugins/endpoints/migrations/env.py new file mode 100644 index 00000000..15f32823 --- /dev/null +++ b/plugins/endpoints/migrations/env.py @@ -0,0 +1,15 @@ +"""Alembic environment for the endpoints plugin migration chain. + +Delegates to the shared runner in shopdb.plugins.alembic_template, which filters +the metadata to this plugin's tables and drives Alembic against the per-plugin +version table alembic_version_endpoints. See ADR-008 for the ownership model. +This plugin is NEW: its 0001 baseline really CREATES its tables, because the +core chain never built them. +""" +import os + +os.environ['PLUGIN_NAME'] = 'endpoints' + +from shopdb.plugins.alembic_template import run_migrations # noqa: E402 + +run_migrations() diff --git a/plugins/endpoints/migrations/script.py.mako b/plugins/endpoints/migrations/script.py.mako new file mode 100644 index 00000000..4f44da1d --- /dev/null +++ b/plugins/endpoints/migrations/script.py.mako @@ -0,0 +1,24 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +# revision identifiers, used by Alembic. +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + + +def upgrade(): + ${upgrades if upgrades else "pass"} + + +def downgrade(): + ${downgrades if downgrades else "pass"} diff --git a/plugins/endpoints/migrations/versions/0001_endpoints_baseline.py b/plugins/endpoints/migrations/versions/0001_endpoints_baseline.py new file mode 100644 index 00000000..7e80055d --- /dev/null +++ b/plugins/endpoints/migrations/versions/0001_endpoints_baseline.py @@ -0,0 +1,93 @@ +"""endpoints plugin baseline (real create). + +A NEW plugin: the core chain never knew about endpointstates or endpointevents, +so this per-plugin chain is their sole authoritative creator. + +Emits explicit Alembic ops rather than using the create_plugin_tables helper: +the helper builds a per-plugin MetaData filtered to the plugin's own tables, so +the foreign key to the core assets table cannot resolve at CreateTable-compile +time (NoReferencedTableError). Tables inherit the connection's default charset, +matching how the core chain creates its own. +""" +from alembic import op + +from shopdb.plugins.alembic_template import ( + create_index_if_missing, + create_table_if_missing, +) +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = 'endpoints0001baseline' +down_revision = None +branch_labels = None +depends_on = None + + +def upgrade(): + create_table_if_missing( + 'endpointstates', + sa.Column('endpointstateid', sa.Integer(), nullable=False), + sa.Column('assetid', sa.Integer(), nullable=False), + sa.Column('hostname', sa.String(length=100), nullable=True), + sa.Column('loggedinuser', sa.String(length=100), nullable=True), + sa.Column('agentversion', sa.String(length=40), nullable=True), + sa.Column('lastboottime', sa.DateTime(), nullable=True), + sa.Column('uptimeseconds', sa.BigInteger(), nullable=True), + sa.Column('idleseconds', sa.BigInteger(), nullable=True), + sa.Column('cpupercent', sa.Float(), nullable=True), + sa.Column('memorypercent', sa.Float(), nullable=True), + sa.Column('servicesjson', sa.Text(length=16777215), nullable=True), + sa.Column('portsjson', sa.Text(length=16777215), nullable=True), + sa.Column('interfacesjson', sa.Text(length=16777215), nullable=True), + sa.Column('scheduledtasksjson', sa.Text(length=16777215), + nullable=True), + sa.Column('lastreportedat', sa.DateTime(), nullable=True), + sa.Column('createdat', sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'], + ondelete='CASCADE'), + sa.PrimaryKeyConstraint('endpointstateid'), + sa.UniqueConstraint('assetid'), + ) + create_index_if_missing('idx_endpointstates_lastreportedat', + 'endpointstates', ['lastreportedat']) + + create_table_if_missing( + 'endpointevents', + sa.Column('endpointeventid', sa.Integer(), nullable=False), + sa.Column('assetid', sa.Integer(), nullable=False), + sa.Column('eventid', sa.Integer(), nullable=False), + sa.Column('eventchannel', sa.String(length=50), nullable=True), + sa.Column('eventsource', sa.String(length=255), nullable=True), + sa.Column('eventtime', sa.DateTime(), nullable=True), + sa.Column('eventmessage', sa.Text(), nullable=True), + sa.Column('payloadjson', sa.Text(length=16777215), nullable=True), + sa.Column('recordedat', sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'], + ondelete='CASCADE'), + sa.PrimaryKeyConstraint('endpointeventid'), + ) + create_index_if_missing('ixendpointeventsassetid', 'endpointevents', + ['assetid']) + create_index_if_missing('ixendpointeventseventid', 'endpointevents', + ['eventid']) + create_index_if_missing('ixendpointeventseventtime', 'endpointevents', + ['eventtime']) + # Alerting/history reads by "this endpoint's events over time", so the + # composite is the index that actually gets used. + create_index_if_missing('idx_endpointevents_assetevent', 'endpointevents', + ['assetid', 'eventtime']) + + +def downgrade(): + op.drop_index('idx_endpointevents_assetevent', + table_name='endpointevents') + op.drop_index('ixendpointeventseventtime', table_name='endpointevents') + op.drop_index('ixendpointeventseventid', table_name='endpointevents') + op.drop_index('ixendpointeventsassetid', table_name='endpointevents') + op.drop_table('endpointevents') + + op.drop_index('idx_endpointstates_lastreportedat', + table_name='endpointstates') + op.drop_table('endpointstates') diff --git a/plugins/endpoints/models/__init__.py b/plugins/endpoints/models/__init__.py new file mode 100644 index 00000000..f641e95b --- /dev/null +++ b/plugins/endpoints/models/__init__.py @@ -0,0 +1,5 @@ +"""Endpoints plugin models.""" + +from .endpoint import EndpointState, EndpointEvent + +__all__ = ['EndpointState', 'EndpointEvent'] diff --git a/plugins/endpoints/models/endpoint.py b/plugins/endpoints/models/endpoint.py new file mode 100644 index 00000000..7dd0e654 --- /dev/null +++ b/plugins/endpoints/models/endpoint.py @@ -0,0 +1,163 @@ +"""Endpoint RMM telemetry models. + +Two shapes, one per concern: + + EndpointState The cached CURRENT snapshot for an asset - one row per asset, + overwritten in place on every agent report. Holds the fast- + moving metrics (uptime, idle, last boot, logged-in user) and + the latest inventory snapshots (services/processes, open ports, + interfaces, scheduled tasks) as serialized JSON. This is the + cached-current-value half of the ledger pattern: it moves in + the SAME commit as any EndpointEvent rows the report carries. + + EndpointEvent Append-only record of a curated Windows Event Log entry. The + agent only forwards a fixed catalog of IDs (logon/logoff, + privilege use, service install, unexpected shutdown, DCOM, + crash) so this stays a readable security/stability history + rather than a raw event firehose. + +Both link to the core Asset via assetid (FK, ON DELETE CASCADE), so an endpoint +is the same physical thing as its Computer/Asset - no duplicate identity. The +agent reports a hostname; the plugin resolves it to the existing Computer. +""" + +from datetime import datetime + +from shopdb.api import db + + +def _utciso(value): + """Naive-UTC datetime -> ISO string marked UTC, or None. + + Every datetime this plugin stores is naive UTC. Serializing without the Z + leaves the receiver to guess, and JavaScript guesses browser-local. + """ + return value.isoformat() + 'Z' if value else None + + +class EndpointState(db.Model): + """Latest reported state for a single endpoint (one row per asset).""" + + __tablename__ = 'endpointstates' + + endpointstateid = db.Column(db.Integer, primary_key=True) + + # The asset this state belongs to. Unique: exactly one current snapshot per + # endpoint. Resolved from the reported hostname against the computers plugin. + assetid = db.Column( + db.Integer, + db.ForeignKey('assets.assetid', ondelete='CASCADE'), + unique=True, + nullable=False, + index=True, + ) + + # Read-only view of the owning asset. No cascade or backref: the asset side + # must not gain a dependency on this plugin. + asset = db.relationship('Asset', lazy='joined', viewonly=True) + + hostname = db.Column(db.String(100), nullable=True) + loggedinuser = db.Column(db.String(100), nullable=True) + agentversion = db.Column(db.String(40), nullable=True) + + # Fast-moving metrics + lastboottime = db.Column(db.DateTime, nullable=True) + uptimeseconds = db.Column(db.BigInteger, nullable=True) + idleseconds = db.Column(db.BigInteger, nullable=True) + cpupercent = db.Column(db.Float, nullable=True) + memorypercent = db.Column(db.Float, nullable=True) + + # Latest inventory snapshots, serialized JSON (MEDIUMTEXT). Kept as the + # newest snapshot rather than one row per sample: the agent reports the full + # list each cycle and history of these lists is not what operators ask for. + servicesjson = db.Column(db.Text(length=16777215), nullable=True) + portsjson = db.Column(db.Text(length=16777215), nullable=True) + interfacesjson = db.Column(db.Text(length=16777215), nullable=True) + scheduledtasksjson = db.Column(db.Text(length=16777215), nullable=True) + + # When the agent last successfully reported. There is no heartbeat separate + # from the report, so silence past the quiet-hours window is the only + # evidence an endpoint has stopped checking in. + lastreportedat = db.Column(db.DateTime, nullable=True, index=True) + + createdat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow) + + __table_args__ = ( + db.Index('idx_endpointstates_lastreportedat', 'lastreportedat'), + ) + + def __repr__(self): + return f"" + + def to_dict(self): + return { + 'endpointstateid': self.endpointstateid, + 'assetid': self.assetid, + 'hostname': self.hostname, + 'loggedinuser': self.loggedinuser, + 'agentversion': self.agentversion, + 'lastboottime': _utciso(self.lastboottime), + 'uptimeseconds': self.uptimeseconds, + 'idleseconds': self.idleseconds, + 'cpupercent': self.cpupercent, + 'memorypercent': self.memorypercent, + 'lastreportedat': _utciso(self.lastreportedat), + 'createdat': _utciso(self.createdat), + } + + +class EndpointEvent(db.Model): + """A single curated Windows Event Log record forwarded by the agent.""" + + __tablename__ = 'endpointevents' + + endpointeventid = db.Column(db.Integer, primary_key=True) + + assetid = db.Column( + db.Integer, + db.ForeignKey('assets.assetid', ondelete='CASCADE'), + nullable=False, + index=True, + ) + + asset = db.relationship('Asset', lazy='joined', viewonly=True) + + # The Windows Event ID (for example 4625 failed logon, 41 unexpected + # shutdown, 1001 application crash). Indexed because alerting queries filter + # on it. + eventid = db.Column(db.Integer, nullable=False, index=True) + + # Event Log channel the record came from: Security, System, Application. + eventchannel = db.Column(db.String(50), nullable=True) + eventsource = db.Column(db.String(255), nullable=True) + + # When the event happened on the endpoint (from the record), not when it + # reached ShopDB. + eventtime = db.Column(db.DateTime, nullable=True, index=True) + + eventmessage = db.Column(db.Text, nullable=True) + + # Structured fields the agent extracted for this ID (for example the target + # account on a 4625). Advisory index only, not part of dedup. + payloadjson = db.Column(db.Text(length=16777215), nullable=True) + + recordedat = db.Column(db.DateTime, nullable=False, default=datetime.utcnow) + + __table_args__ = ( + db.Index('idx_endpointevents_assetevent', 'assetid', 'eventtime'), + ) + + def __repr__(self): + return f"" + + def to_dict(self): + return { + 'endpointeventid': self.endpointeventid, + 'assetid': self.assetid, + 'eventid': self.eventid, + 'eventchannel': self.eventchannel, + 'eventsource': self.eventsource, + 'eventtime': _utciso(self.eventtime), + 'eventmessage': self.eventmessage, + 'recordedat': _utciso(self.recordedat), + } diff --git a/plugins/endpoints/plugin.py b/plugins/endpoints/plugin.py new file mode 100644 index 00000000..35d71cc8 --- /dev/null +++ b/plugins/endpoints/plugin.py @@ -0,0 +1,376 @@ +"""Endpoints plugin main class. + +Endpoint RMM telemetry reported by the ShopDB agent. Accepts collector input +over HTTPS per ADR-006: the agent posts a per-host payload to +POST /api/collector/endpoints, the plugin resolves the reported hostname to an +existing computer asset, overwrites that asset's cached EndpointState, and +appends any curated Windows events the report carried - all in one commit, so +the cached snapshot and the events it summarizes never disagree. + +Identity is the hostname, resolved against the computers plugin. An unresolved +hostname is reported as a ValueError (surfaced to the caller as a validation +error), never silently dropped: an unknown host is a real gap between the fleet +and ShopDB, the same discipline the backups collector uses for machine numbers. +""" + +import json +import logging +from datetime import datetime +from pathlib import Path +from typing import Dict, List, Optional, Type + +from flask import Flask, Blueprint + +from shopdb.plugins.base import BasePlugin, PluginMeta +from shopdb.api import db + +from .api import endpoints_bp +from .eventcatalog import EVENT_CATALOG +from .models import EndpointState, EndpointEvent + +logger = logging.getLogger(__name__) + +# The fixed catalog of Windows Event IDs the agent forwards. Derived from +# EVENT_CATALOG (eventcatalog.py) so the ingest filter and the display titles +# can never drift. IDs outside this set are ignored on ingest rather than +# stored, keeping endpointevents a readable security/stability history instead +# of a raw event firehose. +CURATED_EVENT_IDS = frozenset(EVENT_CATALOG) + + +def _parse_dt(value): + """Best-effort parse of an ISO-8601 string to naive UTC, or None.""" + if not value: + return None + if isinstance(value, datetime): + return value.replace(tzinfo=None) + text = str(value).strip().replace('Z', '+00:00') + try: + parsed = datetime.fromisoformat(text) + except ValueError: + return None + if parsed.tzinfo is not None: + parsed = parsed.astimezone(tz=None).replace(tzinfo=None) + return parsed + + +def _as_json(value): + """Serialize a list/dict to JSON text, or None when nothing was sent.""" + if value is None: + return None + return json.dumps(value, separators=(',', ':'), sort_keys=True) + + +class EndpointsPlugin(BasePlugin): + """Endpoint RMM telemetry ingest plus read/dashboard surface.""" + + def __init__(self): + self._manifest = self._load_manifest() + + def _load_manifest(self) -> Dict: + manifest_path = Path(__file__).parent / 'manifest.json' + if manifest_path.exists(): + with open(manifest_path, 'r') as f: + return json.load(f) + return {} + + @property + def meta(self) -> PluginMeta: + return PluginMeta( + name=self._manifest.get('name', 'endpoints'), + version=self._manifest.get('version', '0.1.0'), + description=self._manifest.get( + 'description', 'Endpoint RMM telemetry'), + author=self._manifest.get('author', 'ShopDB Team'), + dependencies=self._manifest.get('dependencies', ['computers']), + core_version=self._manifest.get('core_version', '>=0.20.0,<1.0.0'), + api_prefix=self._manifest.get('api_prefix', '/api/endpoints'), + ) + + def get_blueprint(self) -> Optional[Blueprint]: + return endpoints_bp + + def get_models(self) -> List[Type]: + return [EndpointState, EndpointEvent] + + def get_permissions(self) -> List: + """RBAC for the human-facing read routes. + + View only. Ingest is not covered here: the collector path authorizes on + a collector.ingest-scoped service token, not a user JWT. There is no + write/delete permission because operators do not edit telemetry - the + agent is the sole author, and history is pruned by policy, not by hand. + """ + return [ + ('endpoints.view', 'View endpoint telemetry', 'endpoints'), + ] + + def get_navigation_items(self) -> List[Dict]: + """Sidebar entry for the fleet list view. + + Sits just after the PCs item (position 15) because the fleet view is + a telemetry lens over the same computers. + """ + return [ + { + 'name': 'Fleet', + 'icon': 'activity', + 'route': '/endpoints', + 'position': 16, + }, + ] + + def init_app(self, app: Flask, db_instance) -> None: + logger.info('Endpoints plugin initialized (v%s)', self.meta.version) + + # ---- ADR-010 UI hooks ----------------------------------------------- + + def get_asset_panels(self) -> List[Dict]: + """Per-PC detail panels, computer assets only. + + Three panels feeding the generic renderer: the current status snapshot + (keyvalue), monitored services with a run-state badge (list), and the + most recent curated events (list). Each is a single-asset indexed read, + so the cost is the same profile as the warranty/backups panels. Panels + hide themselves when their endpoint returns nothing. + """ + return [ + { + 'id': 'endpoints-status', + 'title': 'Endpoint status', + 'assettypes': ['computer'], + 'endpoint': '/api/endpoints/asset/{assetid}/status', + 'render': 'keyvalue', + 'empty': 'No agent report yet.', + 'position': 10, + }, + { + 'id': 'endpoints-services', + 'title': 'Services and processes', + 'assettypes': ['computer'], + 'endpoint': '/api/endpoints/asset/{assetid}/services', + 'render': 'list', + 'map': { + 'title': 'name', + 'badge': {'label': 'statelabel', 'color': 'statecolor'}, + }, + 'empty': 'No monitored services reported.', + 'position': 12, + }, + { + 'id': 'endpoints-events', + 'title': 'Recent events', + 'assettypes': ['computer'], + 'endpoint': '/api/endpoints/asset/{assetid}/recentevents', + 'render': 'list', + 'map': { + 'title': 'title', + 'badge': { + 'label': 'severity', + 'labelmap': { + 'critical': 'Critical', + 'warning': 'Warning', + 'info': 'Info', + }, + }, + 'meta': [ + {'key': 'eventtime', 'label': 'At', 'format': 'date'}, + ], + }, + 'empty': 'No recent events.', + 'position': 14, + }, + ] + + def get_dashboard_widgets(self) -> List[Dict]: + """Fleet-wide exception cards. Run once on dashboard load, not per PC. + + Both cards hide when empty and cap their rows, so a healthy fleet adds + nothing to the dashboard and an unhealthy one adds a bounded list. + """ + return [ + { + 'id': 'endpoints-quiet', + 'viewall': '/pcs', + 'title': 'Endpoints gone quiet', + 'endpoint': '/api/endpoints/dashboard/quiet', + 'render': 'exceptions', + 'severity': 'warning', + 'permission': 'endpoints.view', + 'empty': 'hide', + 'position': 22, + 'map': { + 'title': 'hostname', + 'detail': '', + 'meta': [{'key': 'quietdays', 'label': 'silent', + 'suffix': ' days'}], + 'link': '/pcs/{computerid}', + }, + }, + { + 'id': 'endpoints-stability', + 'viewall': '/pcs', + 'title': 'Endpoints with crashes or shutdowns', + 'endpoint': '/api/endpoints/dashboard/stability', + 'render': 'exceptions', + 'severity': 'critical', + 'permission': 'endpoints.view', + 'empty': 'hide', + 'position': 16, + 'map': { + 'title': 'hostname', + 'detail': '', + 'meta': [{'key': 'eventcount', 'suffix': ' events (48h)'}], + 'link': '/pcs/{computerid}', + }, + }, + ] + + # ---- ADR-006 collector contract ------------------------------------- + + def get_collector_schema(self) -> Optional[Dict]: + return { + 'identityfield': 'hostname', + 'fields': { + 'type': 'object', + 'required': ['hostname'], + 'properties': { + 'hostname': { + 'type': 'string', + 'description': 'Endpoint hostname. Resolved against ' + 'the computers plugin (computers.hostname).', + }, + 'loggedinuser': {'type': 'string'}, + 'agentversion': {'type': 'string'}, + 'lastboottime': {'type': 'string', 'format': 'date-time'}, + 'uptimeseconds': {'type': 'integer'}, + 'idleseconds': {'type': 'integer'}, + 'cpupercent': {'type': 'number'}, + 'memorypercent': {'type': 'number'}, + 'services': { + 'type': 'array', + 'description': 'Monitored services/processes and their ' + 'run state (UDC, eDnc, Intune, ...).', + }, + 'ports': { + 'type': 'array', + 'description': 'Open listening ports.', + }, + 'interfaces': { + 'type': 'array', + 'description': 'Network interface information.', + }, + 'scheduledtasks': { + 'type': 'array', + 'description': 'Scheduled task inventory + last run.', + }, + 'events': { + 'type': 'array', + 'description': 'Curated Windows Event Log records. Only ' + 'IDs in the server catalog are stored.', + 'items': { + 'type': 'object', + 'required': ['eventid'], + 'properties': { + 'eventid': {'type': 'integer'}, + 'eventchannel': {'type': 'string'}, + 'eventsource': {'type': 'string'}, + 'eventtime': {'type': 'string', + 'format': 'date-time'}, + 'eventmessage': {'type': 'string'}, + 'fields': {'type': 'object'}, + }, + }, + }, + }, + }, + } + + def apply_collector_payload(self, payload: Dict) -> Dict: + """Idempotently upsert one endpoint's state and append its events. + + The cached EndpointState and any EndpointEvent rows are written in the + same transaction (the dispatcher owns the commit), so a reader never + sees a snapshot without the events that produced it. + """ + from plugins.computers.models import Computer + + warnings: List[str] = [] + + hostname = (payload.get('hostname') or '').strip() + if not hostname: + raise ValueError('hostname is required') + + computer = (Computer.query + .filter(db.func.lower(Computer.hostname) == hostname.lower()) + .first()) + if computer is None: + raise ValueError( + 'could not resolve hostname to a computer: {!r}'.format(hostname)) + + assetid = computer.assetid + + state = EndpointState.query.filter_by(assetid=assetid).first() + action = 'updated' + if state is None: + state = EndpointState(assetid=assetid) + db.session.add(state) + action = 'created' + + state.hostname = hostname + if 'loggedinuser' in payload: + state.loggedinuser = payload.get('loggedinuser') + if 'agentversion' in payload: + state.agentversion = payload.get('agentversion') + if 'lastboottime' in payload: + state.lastboottime = _parse_dt(payload.get('lastboottime')) + if 'uptimeseconds' in payload: + state.uptimeseconds = payload.get('uptimeseconds') + if 'idleseconds' in payload: + state.idleseconds = payload.get('idleseconds') + if 'cpupercent' in payload: + state.cpupercent = payload.get('cpupercent') + if 'memorypercent' in payload: + state.memorypercent = payload.get('memorypercent') + if 'services' in payload: + state.servicesjson = _as_json(payload.get('services')) + if 'ports' in payload: + state.portsjson = _as_json(payload.get('ports')) + if 'interfaces' in payload: + state.interfacesjson = _as_json(payload.get('interfaces')) + if 'scheduledtasks' in payload: + state.scheduledtasksjson = _as_json(payload.get('scheduledtasks')) + + state.lastreportedat = datetime.utcnow() + + stored_events = 0 + for record in (payload.get('events') or []): + try: + eventid = int(record.get('eventid')) + except (TypeError, ValueError): + warnings.append('skipped event with non-integer eventid') + continue + if eventid not in CURATED_EVENT_IDS: + # Outside the catalog: ignored on purpose, not an error. + continue + db.session.add(EndpointEvent( + assetid=assetid, + eventid=eventid, + eventchannel=record.get('eventchannel'), + eventsource=record.get('eventsource'), + eventtime=_parse_dt(record.get('eventtime')), + eventmessage=record.get('eventmessage'), + payloadjson=_as_json(record.get('fields')), + )) + stored_events += 1 + + # flush, not commit: the collector dispatcher owns the transaction. + db.session.flush() + + return { + 'action': action, + 'assetid': assetid, + 'identityvalue': hostname, + 'storedevents': stored_events, + 'warnings': warnings, + } diff --git a/plugins/endpoints/tests/test_plugin.py b/plugins/endpoints/tests/test_plugin.py new file mode 100644 index 00000000..5440411d --- /dev/null +++ b/plugins/endpoints/tests/test_plugin.py @@ -0,0 +1,92 @@ +"""Endpoints plugin smoke + collector-contract tests. + +Asserts the plugin loads cleanly, satisfies the framework contract, and that +its ADR-006 collector schema is well formed. Extend with DB-backed ingest tests +as the plugin grows. +""" + +from flask import Blueprint + +from plugins.endpoints.plugin import EndpointsPlugin, CURATED_EVENT_IDS +from plugins.endpoints.eventcatalog import EVENT_CATALOG, CRITICAL_EVENT_IDS + + +def test_endpoints_plugin_meta_is_valid(): + """meta returns a PluginMeta with the expected name and prefix.""" + plugin = EndpointsPlugin() + assert plugin.meta.name == 'endpoints' + assert plugin.meta.api_prefix == '/api/endpoints' + assert 'computers' in plugin.meta.dependencies + + +def test_endpoints_plugin_get_blueprint_returns_blueprint(): + """get_blueprint returns a Flask Blueprint, not None.""" + plugin = EndpointsPlugin() + assert isinstance(plugin.get_blueprint(), Blueprint) + + +def test_endpoints_plugin_get_models_returns_models(): + """get_models returns the two telemetry tables.""" + plugin = EndpointsPlugin() + models = plugin.get_models() + assert len(models) >= 2 + for model in models: + assert hasattr(model, '__tablename__') + + +def test_endpoints_collector_schema_identity_is_hostname(): + """ADR-006 schema declares hostname as the identity field.""" + plugin = EndpointsPlugin() + schema = plugin.get_collector_schema() + assert schema is not None + assert schema['identityfield'] == 'hostname' + assert 'hostname' in schema['fields']['required'] + + +def test_endpoints_curated_event_ids_include_core_security_ids(): + """The curated catalog covers the agreed logon/stability IDs.""" + for eventid in (4624, 4625, 4634, 4672, 4697, 4698, 41, 4618, 55, 7011, + 10016, 1001): + assert eventid in CURATED_EVENT_IDS + + +def test_curated_ids_match_the_event_catalog(): + """Ingest filter and display catalog cannot drift apart.""" + assert CURATED_EVENT_IDS == frozenset(EVENT_CATALOG) + + +def test_critical_event_ids_are_a_subset_of_the_catalog(): + """Dashboard stability card only queries known critical IDs.""" + assert set(CRITICAL_EVENT_IDS) + assert set(CRITICAL_EVENT_IDS).issubset(set(EVENT_CATALOG)) + + +def test_endpoints_asset_panels_are_computer_scoped(): + """Every asset panel targets computer assets only, with an endpoint.""" + plugin = EndpointsPlugin() + panels = plugin.get_asset_panels() + assert len(panels) == 3 + for panel in panels: + assert panel['assettypes'] == ['computer'] + assert panel['endpoint'].startswith('/api/endpoints/asset/{assetid}') + assert panel['render'] in ('keyvalue', 'list') + + +def test_endpoints_dashboard_widgets_declared(): + """Quiet + stability exception cards are declared and hide when empty.""" + plugin = EndpointsPlugin() + widgets = plugin.get_dashboard_widgets() + ids = {w['id'] for w in widgets} + assert {'endpoints-quiet', 'endpoints-stability'} <= ids + for widget in widgets: + assert widget['render'] == 'exceptions' + assert widget['empty'] == 'hide' + assert widget['permission'] == 'endpoints.view' + + +def test_endpoints_navigation_item_points_at_fleet_view(): + """The fleet list view is exposed as a single /endpoints nav item.""" + plugin = EndpointsPlugin() + items = plugin.get_navigation_items() + routes = {item['route'] for item in items} + assert '/endpoints' in routes diff --git a/scripts/dev/seed_endpoints_sample.py b/scripts/dev/seed_endpoints_sample.py new file mode 100644 index 00000000..db2c4baf --- /dev/null +++ b/scripts/dev/seed_endpoints_sample.py @@ -0,0 +1,239 @@ +"""Seed sample endpoint telemetry for the Fleet module (DEV ONLY). + +Attaches EndpointState snapshots plus a handful of curated EndpointEvents to +existing Computer assets so the fleet list, the PC-detail asset panels +(status / services / recent events), and the dashboard quiet + stability cards +have something to render before a real agent exists. + +Sample rows are marked with a sentinel agentversion ("0.0.0-sample") so they can +be listed and cleared without touching real agent data. + +DEV ONLY: this file lives under scripts/dev/ and must never ship in a +production image (see devnotes/live-container-and-testing-workflow.md). Do not +import it from application code. + +Usage (in the API container): + docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py + docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py --clear + docker exec shopdb-flask-wilm-api-1 python scripts/dev/seed_endpoints_sample.py --limit 8 +""" + +import argparse +import json +import os +import sys +from datetime import datetime, timedelta, timezone + +# Repo root on sys.path so `shopdb` and `plugins` import when run as a script. +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))) + +from shopdb import create_app # noqa: E402 +from shopdb.api import db # noqa: E402 + +SAMPLE_MARKER = '0.0.0-sample' + + +def _naive_utcnow(): + return datetime.now(timezone.utc).replace(tzinfo=None) + + +def _services(*running_then_stopped): + """Build a services snapshot. Positional pairs are (name, state).""" + return [{'name': name, 'state': state} for name, state in running_then_stopped] + + +def _ports(*pairs): + return [{'port': port, 'process': process} for port, process in pairs] + + +def _interfaces(ip, mac): + return [{'name': 'Ethernet', 'ip': ip, 'mac': mac}] + + +# Per-machine telemetry shapes. minutesago drives whether a machine reads as +# fresh, stale, or quiet; a couple are pushed past a plausible quiet window so +# the dashboard quiet card has rows. Everything is offline-plausible because +# there is no live agent. +SAMPLE_PROFILES = [ + { + 'loggedinuser': 'CORP\\jmartin', + 'agentversion': SAMPLE_MARKER, + 'minutesago': 3, + 'uptimeseconds': 4 * 86400 + 6 * 3600, + 'idleseconds': 12 * 60, + 'cpupercent': 14.0, + 'memorypercent': 46.0, + 'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'), + ('WinRM', 'Running'), ('MSSQLSERVER', 'Running')), + 'ports': _ports((3389, 'svchost'), (5985, 'winrm'), (445, 'System')), + 'interfaces': _interfaces('10.20.4.11', '00:1A:2B:3C:4D:01'), + 'events': [(4624, -8), (4672, -8), (4698, -120)], + }, + { + 'loggedinuser': 'CORP\\operator2', + 'agentversion': SAMPLE_MARKER, + 'minutesago': 9, + 'uptimeseconds': 2 * 3600 + 40 * 60, + 'idleseconds': 3 * 60, + 'cpupercent': 61.0, + 'memorypercent': 72.0, + 'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'), + ('WinRM', 'Stopped')), + 'ports': _ports((3389, 'svchost'), (445, 'System')), + 'interfaces': _interfaces('10.20.4.27', '00:1A:2B:3C:4D:02'), + # Recent unexpected shutdown -> shows on the stability card. + 'events': [(41, -50), (1001, -30), (4625, -12)], + }, + { + 'loggedinuser': None, + 'agentversion': SAMPLE_MARKER, + 'minutesago': 22, + 'uptimeseconds': 11 * 86400, + 'idleseconds': 9 * 3600, + 'cpupercent': 3.0, + 'memorypercent': 38.0, + 'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'), + ('WinRM', 'Running')), + 'ports': _ports((3389, 'svchost'), (5985, 'winrm')), + 'interfaces': _interfaces('10.20.4.44', '00:1A:2B:3C:4D:03'), + 'events': [(4624, -600), (7011, -180)], + }, + { + 'loggedinuser': 'CORP\\nightshift', + 'agentversion': SAMPLE_MARKER, + # Quiet: no report in over a day. + 'minutesago': 1500, + 'uptimeseconds': 30 * 60, + 'idleseconds': 30 * 60, + 'cpupercent': 1.0, + 'memorypercent': 29.0, + 'services': _services(('Spooler', 'Stopped'), ('W32Time', 'Running')), + 'ports': _ports((445, 'System'),), + 'interfaces': _interfaces('10.20.4.58', '00:1A:2B:3C:4D:04'), + 'events': [(55, -1600), (41, -1580)], + }, + { + 'loggedinuser': 'CORP\\qa_kiosk', + 'agentversion': SAMPLE_MARKER, + 'minutesago': 6, + 'uptimeseconds': 6 * 3600, + 'idleseconds': 45 * 60, + 'cpupercent': 22.0, + 'memorypercent': 55.0, + 'services': _services(('Spooler', 'Running'), ('W32Time', 'Running'), + ('WinRM', 'Running'), ('BITS', 'Running')), + 'ports': _ports((3389, 'svchost'), (5985, 'winrm'), (135, 'rpc')), + 'interfaces': _interfaces('10.20.4.63', '00:1A:2B:3C:4D:05'), + 'events': [(4624, -20), (10016, -90), (4697, -240)], + }, +] + + +def _clear(EndpointState, EndpointEvent): + sample_states = EndpointState.query.filter_by(agentversion=SAMPLE_MARKER).all() + asset_ids = [s.assetid for s in sample_states] + events_removed = 0 + if asset_ids: + events_removed = (EndpointEvent.query + .filter(EndpointEvent.assetid.in_(asset_ids)) + .delete(synchronize_session=False)) + states_removed = (EndpointState.query + .filter_by(agentversion=SAMPLE_MARKER) + .delete(synchronize_session=False)) + db.session.commit() + print('Cleared {} sample state(s) and {} sample event(s).' + .format(states_removed, events_removed)) + + +def _seed(limit): + from plugins.computers.models import Computer + from plugins.endpoints.models import EndpointState, EndpointEvent + from plugins.endpoints.eventcatalog import EVENT_CATALOG + + computers = (Computer.query + .filter(Computer.assetid.isnot(None)) + .order_by(Computer.computerid.asc()) + .limit(limit) + .all()) + if not computers: + print('No Computer assets found to attach sample endpoints to. ' + 'Seed the computers plugin first.') + return + + now = _naive_utcnow() + created, updated, events_added = 0, 0, 0 + + for index, computer in enumerate(computers): + profile = SAMPLE_PROFILES[index % len(SAMPLE_PROFILES)] + reported = now - timedelta(minutes=profile['minutesago']) + boot = reported - timedelta(seconds=profile['uptimeseconds']) + + state = EndpointState.query.filter_by(assetid=computer.assetid).first() + if state and state.agentversion not in (None, SAMPLE_MARKER): + # Never stomp a real agent's snapshot. + continue + if not state: + state = EndpointState(assetid=computer.assetid, createdat=now) + db.session.add(state) + created += 1 + else: + updated += 1 + + state.hostname = computer.hostname or 'HOST-{}'.format(computer.computerid) + state.loggedinuser = profile['loggedinuser'] + state.agentversion = SAMPLE_MARKER + state.lastboottime = boot + state.uptimeseconds = profile['uptimeseconds'] + state.idleseconds = profile['idleseconds'] + state.cpupercent = profile['cpupercent'] + state.memorypercent = profile['memorypercent'] + state.servicesjson = json.dumps(profile['services']) + state.portsjson = json.dumps(profile['ports']) + state.interfacesjson = json.dumps(profile['interfaces']) + state.scheduledtasksjson = json.dumps([]) + state.lastreportedat = reported + + # Refresh this asset's sample events so re-running stays idempotent. + EndpointEvent.query.filter_by(assetid=computer.assetid).delete( + synchronize_session=False) + for eventid, offset_minutes in profile['events']: + title, severity, channel = EVENT_CATALOG.get( + eventid, ('Event {}'.format(eventid), 'info', 'System')) + event = EndpointEvent( + assetid=computer.assetid, + eventid=eventid, + eventchannel=channel, + eventsource='sample-seed', + eventtime=reported + timedelta(minutes=offset_minutes), + eventmessage='{} (sample)'.format(title), + payloadjson=json.dumps({'sample': True, 'severity': severity}), + recordedat=now, + ) + db.session.add(event) + events_added += 1 + + db.session.commit() + print('Seeded sample endpoints: {} created, {} updated, {} event(s) added ' + 'across {} computer(s).' + .format(created, updated, events_added, len(computers))) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--clear', action='store_true', + help='Remove sample-marked endpoint states and their events.') + parser.add_argument('--limit', type=int, default=5, + help='Max number of computers to attach sample endpoints to.') + args = parser.parse_args() + + app = create_app(os.environ.get('FLASK_ENV', 'production')) + with app.app_context(): + from plugins.endpoints.models import EndpointState, EndpointEvent + if args.clear: + _clear(EndpointState, EndpointEvent) + else: + _seed(args.limit) + + +if __name__ == '__main__': + main()