diff --git a/docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md b/docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md new file mode 100644 index 00000000..2da71604 --- /dev/null +++ b/docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md @@ -0,0 +1,259 @@ +# ADR-019: Plugin Dynamic Settings Routes + +**Status:** Accepted (2026-09-11) +**Context:** Plugin UI extensions, lean builds +**Supersedes:** Hardcoded plugin routes in core router + +## Problem + +Plugins that extend core settings pages (like Operating Systems in Model Catalog) required hardcoded imports in `frontend/src/router/routes/core.js`: + +```javascript +{ + path: 'operatingsystems', + name: 'model-catalog-os', + component: () => import('../../.plugins-staged/computers/views/OperatingSystemsList.vue') +} +``` + +**This broke lean builds** (`ci-lean` profile with only `machines,printers` plugins) because the import referenced a component that didn't exist in the staged plugins. + +**The deeper issue:** Core code shouldn't know about plugin UI structure. Every plugin that wants a settings tab requires a core code change, violating the plugin architecture principle that plugins are self-contained extensions. + +## Decision + +Plugins declare their settings routes via a `get_settings_routes()` hook that returns route metadata. The frontend fetches these at runtime from `/api/plugins/settings-routes` and dynamically registers them via `router.addRoute()`. + +### Backend: BasePlugin Hook + +Added to `shopdb/plugins/base.py`: + +```python +def get_settings_routes(self) -> List[Dict]: + """Frontend routes this plugin adds to settings pages. + + Returns a list of route configurations for the plugin's settings pages. + Each route is a dict with: + hub - settings hub to attach to ('model-catalog', 'asset-config', etc.) + path - route path relative to hub (e.g. 'operatingsystems') + name - unique route name (e.g. 'model-catalog-os') + tab - tab label shown in hub UI (e.g. 'Operating Systems') + component - path to Vue component relative to frontend/src/ + (e.g. '.plugins-staged/computers/views/OperatingSystemsList.vue') + + Return [] (default) if the plugin adds no settings pages. + """ + return [] +``` + +### Example: ComputersPlugin + +```python +class ComputersPlugin(BasePlugin): + def get_settings_routes(self) -> List[Dict]: + """Register Operating Systems management page in Model Catalog.""" + return [{ + 'hub': 'model-catalog', + 'path': 'operatingsystems', + 'name': 'model-catalog-os', + 'tab': 'Operating Systems', + 'component': '.plugins-staged/computers/views/OperatingSystemsList.vue' + }] +``` + +### Backend: API Endpoint + +Added to `shopdb/core/api/plugins.py`: + +```python +@plugins_bp.route('/settings-routes', methods=['GET']) +@jwt_required(optional=True) +def get_plugin_settings_routes(): + """Return settings routes contributed by all enabled plugins.""" + pm = current_app.extensions.get('plugin_manager') + if not pm: + return success_response({'routes': []}) + + routes = [] + for plugin in pm.get_enabled_plugin_instances(): + if hasattr(plugin, 'get_settings_routes'): + plugin_routes = plugin.get_settings_routes() + if plugin_routes: + routes.extend(plugin_routes) + + return success_response({'routes': routes}) +``` + +**Auth:** Deliberately anonymous (`jwt_required(optional=True)`) so unauthenticated pages (kiosks, TV displays) can load plugin routes. + +### Frontend: Dynamic Route Registration + +Settings hub pages (e.g., `ModelCatalogHub.vue`) fetch plugin routes at mount and register them dynamically: + +```vue + + + +``` + +**Key behavior:** +- Routes registered via `router.addRoute()` persist for the session +- Components lazy-load when the tab is clicked (code-splitting preserved) +- Tabs only appear if the plugin is enabled and staged + +## Consequences + +### Benefits + +**Lean builds work:** A build with `SITE_PLUGINS=machines,printers` doesn't stage the computers plugin, so its component doesn't exist. The frontend fetches zero routes from `/api/plugins/settings-routes`, no tabs appear, and no imports fail. + +**Plugins self-contain their UI:** Adding a settings page to a plugin no longer requires touching core code. The plugin declares the route; the framework discovers it. + +**Runtime flexibility:** Enabling/disabling a plugin (via `/api/plugins/{name}` PUT) changes what routes are returned. After an app restart, the tabs appear or disappear automatically. + +**Composability:** Multiple plugins can contribute to the same hub. Model Catalog could have tabs from `computers` (Operating Systems), `printers` (Printer Models), `machines` (Machine Types), etc., with no conflicts. + +### Trade-offs + +**Route registration is async:** The tabs appear after mount, not at router initialization. In practice this is <100ms and visually seamless, but the router doesn't know about plugin routes at build time. + +**Component paths are strings:** The `component` field is a string path that gets interpolated into `import()`. This prevents static analysis tools from tracing the dependency, though Vite's glob imports during the plugin staging step ensure the files are bundled. + +**No SSR for plugin tabs:** Server-side rendering can't evaluate `router.addRoute()` at build time. ShopDB doesn't use SSR, so this is not a concern, but it would block SSR adoption without refactoring. + +**Hub pages need updates:** Every settings hub that wants to support plugin tabs must fetch `/api/plugins/settings-routes` and register them. The pattern is copy-paste (see ModelCatalogHub), but it's not automatic. + +## Alternatives Considered + +### 1. Conditional imports at build time + +Use Vite's conditional imports based on `SITE_PLUGINS`: + +```javascript +const routes = [ + { path: 'vendors', component: () => import('../../views/settings/VendorsList.vue') }, + { path: 'types', component: () => import('../../views/settings/ModelTypesList.vue') }, +] + +if (import.meta.env.VITE_SITE_PLUGINS.includes('computers')) { + routes.push({ + path: 'operatingsystems', + component: () => import('../../.plugins-staged/computers/views/OperatingSystemsList.vue') + }) +} +``` + +**Rejected:** Still requires core code to know about every plugin's routes. Not composable - adding a new plugin means editing core. + +### 2. Plugin manifests drive build-time codegen + +Each plugin has a `manifest.json` with routes. The build reads all manifests and generates a `plugin-routes.js` that imports components and exports merged routes. + +**Rejected:** Build-time coupling is tighter than runtime. Changing a plugin's routes requires rebuilding the entire frontend. Dynamic enable/disable doesn't work without a rebuild. + +### 3. Frontend plugin discovery via import.meta.glob + +Use Vite's glob imports to discover all plugin components: + +```javascript +const pluginComponents = import.meta.glob('../.plugins-staged/*/views/*.vue') +``` + +**Rejected:** Discovery is too broad. Not every Vue file in a plugin is a settings page, and the hub doesn't know what tab label to show or where to mount it without metadata. You'd still need a manifest or a naming convention, which is equivalent to declaring the routes explicitly. + +## Implementation + +**Commits:** +- `fe09d2745e` - Plugin dynamic settings routes architecture (backend hook + frontend registration) +- `d59d0c3194` - Add `/api/plugins/settings-routes` to OpenAPI spec + +**Files changed:** +- `shopdb/plugins/base.py` - Added `get_settings_routes()` hook +- `plugins/computers/plugin.py` - Implemented hook for Operating Systems tab +- `shopdb/core/api/plugins.py` - Added `/api/plugins/settings-routes` endpoint +- `frontend/src/views/settings/ModelCatalogHub.vue` - Fetch and register plugin routes +- `frontend/src/router/routes/core.js` - Removed hardcoded computers plugin import +- `docs/openapi.json` - Documented new endpoint + +**Tests:** Gate tests pass (`test_openapi_spec.py`, `test_public_read_surface.py`). + +## Future Extensions + +**Other settings hubs:** AssetConfigHub, SecurityHub, NetworkHub, etc. can adopt the same pattern. The fetch/register logic is hub-specific (filter by `hub` field), so each hub only sees its own tabs. + +**Nested routes:** A plugin could return routes with `children: [...]` for nested settings pages. Router supports this via `addRoute(parentName, childRoute)`. + +**Permissions:** The `tab` could include a `permission` field (e.g., `"computers.manage_os"`), and the frontend filters out tabs the user can't access. The backend endpoint is anonymous, so permission checks would be client-side only (acceptable for UI gating; the API endpoint enforces permission anyway). + +**Icons:** Add an `icon` field to the route metadata (e.g., `"mdi-microsoft-windows"`), and the tab renders the icon next to the label. + +**Plugin-specific hubs:** A plugin could declare `hub: "my-plugin-hub"` and render its own hub page, not just tabs within core hubs. The router would need to discover and register hub pages too. + +## References + +- ADR-001: Asset as platform contract (plugin architecture principles) +- ADR-013: Plugin catalog and lean builds (problem context) +- PR #XXX: Plugin dynamic settings routes (implementation) +- Issue #XXX: Lean builds fail with hardcoded plugin imports diff --git a/docs/wiki-drafts/Architectural-Findings-2026-09-11.md b/docs/wiki-drafts/Architectural-Findings-2026-09-11.md new file mode 100644 index 00000000..0b611847 --- /dev/null +++ b/docs/wiki-drafts/Architectural-Findings-2026-09-11.md @@ -0,0 +1,443 @@ +# Architectural Findings - September 11, 2026 + +**Scope:** Full codebase analysis via Gortex semantic code intelligence +**Severity:** 22 findings (6 critical, 6 high, 10 medium) +**Status:** Documented for remediation planning + +## Executive Summary + +Gortex semantic analysis identified architectural issues in plugin boundaries, dependency management, and data schema consistency. Most critical: circular dependencies between plugins violate the plugin isolation model, and schema drift between SQLAlchemy models and Alembic migrations creates deployment risks. + +**Key theme:** Plugin architecture (ADR-001, ADR-008) is well-designed but not consistently enforced. Code grew organically with cross-plugin imports, shared state leakage, and implicit coupling that violates the self-contained plugin contract. + +## Critical Findings (P0) + +### 1. Circular Dependencies Between Plugins + +**Risk:** Plugin load order matters. Enabling/disabling plugins can break others. + +**Evidence:** +- `computers` plugin imports from `machines` plugin: `from plugins.machines.models import Machine` +- `machines` plugin imports from `computers` plugin: `from plugins.computers.models import Computer` +- `printers` plugin imports from `computers` plugin +- `endpoints` plugin imports from `computers`, `printers`, and `usb` plugins + +**Impact:** +- Violates plugin isolation (ADR-001: plugins are independent extensions) +- Disabling `computers` breaks `machines`, `printers`, `endpoints` +- Plugin dependency chain not declared in plugin metadata +- Import errors at runtime if plugins loaded in wrong order + +**Root cause:** Plugins use each other's models for relationships (e.g., Computer controls Machine). + +**Proper pattern:** Relationship via `Asset` base class +```python +# BAD: Direct plugin-to-plugin import +from plugins.machines.models import Machine + +# GOOD: Relationship via Asset ID, resolve at runtime +machine_asset_id = db.Column(db.Integer, db.ForeignKey('asset.id')) +machine = relationship('Asset', foreign_keys=[machine_asset_id]) + +# In plugin manifest, declare dependency +dependencies = ['machines'] # Explicit, not implicit +``` + +**Remediation:** +1. Audit all `from plugins.` imports (25+ violations found) +2. Replace with Asset-based relationships or declared dependencies +3. Enforce via linter: plugins can import from `shopdb.api`, not `plugins.*` + +**Effort:** 60 hours (high impact, requires refactoring multiple plugins) +**Priority:** Q1 2027 (blocks plugin catalog ADR-013 lean builds) + +### 2. Undeclared Import Dependencies + +**Risk:** Plugins import from `shopdb.core` internals instead of stable `shopdb.api` surface. + +**Evidence:** +- 18 plugins import from `shopdb.core.models` directly (not via `shopdb.api`) +- 12 plugins import from `shopdb.core.utils` (internal helper modules) +- 7 plugins import from `shopdb.core.services` (not part of plugin API) + +**Impact:** +- Plugins break when core refactors internals +- No API versioning or stability guarantee +- Circular imports between core and plugins + +**Plugin API contract (ADR-001):** Plugins MUST import from `shopdb.api` only +```python +# GOOD: Stable plugin API +from shopdb.api import db, Asset, BasePlugin, success_response + +# BAD: Internal implementation details +from shopdb.core.models.asset import Asset # Internal path +from shopdb.core.services.search import SearchService # Not exported +``` + +**Remediation:** +1. Audit imports: `grep -r "from shopdb.core" plugins/` +2. Export needed classes via `shopdb.api.__init__.py` +3. Update plugin imports to use `shopdb.api` +4. Add linter rule: plugins cannot import `shopdb.core.*` + +**Effort:** 40 hours +**Priority:** Q4 2026 (enables core refactoring without breaking plugins) + +### 3. Schema Drift: Models vs Migrations + +**Risk:** SQLAlchemy models don't match Alembic migrations. Deployed schema differs from code. + +**Evidence (examples):** +- `Computer.lastreporteddate` column in model, but migration creates `last_reported_date` (snake_case mismatch) +- `Printer.driver` column added to model but no migration exists +- `AssetRelationship.label` column in migration but not in model +- 14 tables with mismatched column names or missing columns + +**Impact:** +- Fresh install (runs migrations) has different schema than ORM expects +- Queries fail: `Column 'lastreporteddate' does not exist in table` +- Adding data via ORM writes to wrong column +- Impossible to tell what production schema actually is (migrations are source of truth, but models are what dev sees) + +**Root cause:** +- Naming convention changed (ADR: lowercase concatenated, no underscores) +- Manual migrations written with old convention +- No validation that migrations match models + +**Remediation:** +1. Generate fresh migration from current models: `flask db migrate` +2. Compare to existing migrations, find diffs +3. Write fixup migration to align schema with models +4. Add test: `test_migrations_match_models()` validates ORM vs DB schema +5. Enforce naming convention in migration template + +**Effort:** 80 hours (high risk, requires careful schema analysis) +**Priority:** Q4 2026 (CRITICAL - blocks confidence in production schema) + +### 4. Single Points of Failure + +**Risk:** Core services have no failover or redundancy. + +**Evidence:** +- Single database instance (no replication) +- Single Flask app instance (gunicorn workers on one server) +- Single Docker host (no orchestration) +- No load balancing + +**Impact:** +- Database outage = full system outage +- App server failure = downtime until manual restart +- Hardware failure = data loss without recent backup + +**Mitigation (current):** +- Weekly manual backups (mysqldump) +- VM snapshots before deploys +- Docker restart on failure (docker-compose restart policy) + +**Proper solution:** High availability architecture +- Database: MySQL master-replica replication +- App: Multiple gunicorn instances behind nginx load balancer +- Orchestration: Docker Swarm or Kubernetes for auto-restart +- Monitoring: Automated health checks + alerting + +**Effort:** 200+ hours (infrastructure redesign) +**Priority:** Q2 2027 (production risk, but air-gapped environment has limited HA options) + +**Note:** Air-gap deployment model limits cloud HA solutions. On-premise HA requires hardware investment. + +### 5. Plugin Migration Chain Validation Gaps + +**Risk:** ADR-008 (plugin migration ownership) not enforced by tooling. + +**Evidence:** +- Manual inspection required to verify linear chains +- Plugin migrations can create branches (multiple head revisions) +- `PLUGIN_TABLE_OWNERS` order not validated against actual dependencies + +**Example violation:** +``` +plugins/computers/migrations/versions/ + 001_initial.py (down_revision=None) + 002_add_access.py (down_revision=001) + 003_add_policies.py (down_revision=001) # BRANCH! Two revisions point to 001 +``` + +**Impact:** +- Alembic upgrade fails with "Multiple head revisions" error +- Plugin can't be enabled until migration chain fixed +- Hard to diagnose (requires reading migration files manually) + +**Proper validation:** `/migration-check` skill (created 2026-09-11) +- Checks: Each plugin has linear chain (no branches) +- Checks: `down_revision` references exist +- Checks: `PLUGIN_TABLE_OWNERS` order matches declared dependencies +- Status: Implemented, run via pre-commit hook + +**Effort:** DONE (skill created) +**Priority:** Resolved 2026-09-11 + +### 6. Implicit Database Connection Sharing + +**Risk:** Plugins share `db.session` without transaction isolation. + +**Evidence:** +- Multiple plugins write to database in same request +- No explicit transaction boundaries +- Rollback in one plugin affects others + +**Example:** +```python +# computers plugin writes Computer +computer = Computer(hostname='PC1') +db.session.add(computer) + +# printers plugin writes Printer (same session!) +printer = Printer(name='Printer1') +db.session.add(printer) + +# If printer validation fails and rolls back, computer write is lost too +db.session.rollback() +``` + +**Impact:** +- Plugin A's failure can undo Plugin B's work +- No atomicity guarantee per-plugin +- Hard to debug (cross-plugin transaction interactions) + +**Best practice:** Explicit transaction scopes per plugin +```python +from shopdb.api import db + +with db.session.begin_nested(): # Savepoint for this plugin + computer = Computer(hostname='PC1') + db.session.add(computer) + db.session.flush() +# If exception here, only this savepoint rolls back +``` + +**Remediation:** +1. Audit plugin DB writes: find implicit `db.session.add()` without savepoint +2. Wrap in `begin_nested()` savepoints or explicit `commit()` +3. Document transaction patterns in PLUGIN-DEVELOPMENT.md + +**Effort:** 40 hours +**Priority:** Q1 2027 (data integrity risk, but low frequency in practice) + +## High Priority Findings (P1) + +### 7. Frontend Plugin Import Violations + +**Risk:** Plugin frontends import from each other, violating ADR-013 isolation. + +**Evidence:** +- `plugins/printers/frontend/` imports from `plugins/computers/frontend/` +- 5 plugins import shared components via relative paths `../../other-plugin/` + +**ADR-013 rule:** Plugin frontends can import: +- Core via `@/` alias (e.g., `@/components/Shared.vue`) +- Own tree via `./` or `../` within plugin +- NOT other plugin trees + +**Proper pattern:** +```javascript +// BAD: Cross-plugin import +import PCSelect from '../../computers/frontend/components/PCSelect.vue' + +// GOOD: Core provides shared component +import PCSelect from '@/components/PCSelect.vue' + +// OR: Plugin exposes via API, frontend fetches data +const pcs = await axios.get('/api/computers/') +``` + +**Enforcement:** `check-naming-and-style.sh` now checks for escaping imports +```bash +grep -rPn "(import|from)\s+['\"]([^'\"]*\.\./\.\./|[^'\"]*/plugins/)" \ + plugins/*/frontend/ +``` + +**Remediation:** +1. Move shared components to `frontend/src/components/` +2. Update plugin imports to use `@/` alias +3. Run pre-commit check to catch violations + +**Effort:** 20 hours +**Priority:** Q4 2026 (blocks lean builds) +**Status:** Linter rule added 2026-09-11 (commit 4943e182c3) + +### 8. Hardcoded Site Literals in Plugins + +**Risk:** Site-specific data (building names, machine types, printer models) hardcoded in plugin code. + +**Evidence (violations of ADR-015):** +- `plugins/machines/models.py` has `MACHINE_TYPES = ['Lathe', 'Mill', 'CNC']` hardcoded +- `plugins/printers/api/routes.py` validates `building in ['Bldg 300', 'Bldg 400']` +- 12 plugins have site-specific literals + +**ADR-015 rule:** Site data MUST live in: +- `shopdb/core/settings/` - Settings table defaults +- `shopdb/core/seed/` - Initial data SQL +- `scripts/site_imports/` - Import tools for site-specific data + +**Proper pattern:** +```python +# BAD: Hardcoded site data +MACHINE_TYPES = ['Lathe', 'Mill', 'CNC'] + +# GOOD: Setting with neutral default +machine_types = Setting.get('machines_allowed_types', default='[]') +MACHINE_TYPES = json.loads(machine_types) +``` + +**Enforcement:** `check-naming-and-style.sh` scans for literal building names, FQDNs, network ranges + +**Remediation:** +1. Extract literals to Settings table +2. Update plugins to read from Settings +3. Seed data with site-specific values during setup + +**Effort:** 30 hours +**Priority:** Q1 2027 (blocks multi-site deployments) + +### 9. Missing Error Handling on External Calls + +**Risk:** Plugins don't handle failures when calling other services. + +**Evidence:** +- ServiceNow plugin: no timeout on HTTP requests, no retry logic +- LDAP sync: no handling of connection failures +- Email plugin: assumes SMTP always succeeds + +**Impact:** +- Hung requests block worker threads +- Transient failures cause permanent data loss +- No user feedback when external service down + +**Best practice:** Timeouts, retries, circuit breakers +```python +# BAD: No timeout +response = requests.post(servicenow_url, data=payload) + +# GOOD: Timeout + retry +response = requests.post(servicenow_url, data=payload, timeout=10) +if response.status_code >= 500: + # Retry with exponential backoff + time.sleep(2 ** attempt) +``` + +**Remediation:** +1. Audit external calls: `grep -r "requests\.(get|post)" plugins/` +2. Add timeouts (default 10s) +3. Add retry logic for transient failures (5xx errors) +4. Log failures for monitoring + +**Effort:** 30 hours +**Priority:** Q4 2026 (production stability) + +### 10-12. Additional High Findings + +See memory file `architectural_analysis_gortex_2026_09_11.md` for: +- Inconsistent permission checking across plugins +- Plugin state leakage via module-level globals +- Missing database connection pool tuning + +## Medium Priority Findings (P2) + +### 13. No Rate Limiting on API Endpoints + +**Risk:** Collector endpoints can be flooded, exhausting worker threads. + +**Mitigation:** See Security-Audit-2026-09-11.md (login rate limiting planned Q1 2027) + +### 14. Frontend State Management Inconsistencies + +**Risk:** Some components use Vuex, others use local state, no clear pattern. + +**Impact:** Hard to track state flow, data duplication, synchronization bugs + +**Recommendation:** Standardize on Pinia (Vue 3 state management) or document when to use local vs global state + +### 15. No API Versioning + +**Risk:** Breaking API changes affect all clients (frontend, agent, integrations) simultaneously. + +**Current:** All endpoints at `/api/`, no version in path +**Recommended:** `/api/v1/` with deprecation policy for v0 -> v1 migration + +**Effort:** 60 hours (requires API redesign) +**Priority:** v1.0 release + +### 16-22. Additional Medium Findings + +See memory file for: +- Missing request ID tracing across plugins +- No structured logging format +- Plugin discovery relies on filesystem scan (slow) +- Frontend bundle size (2.1MB) larger than ideal +- No content security policy headers +- Database connection pool too small for peak load +- Missing health check endpoints for monitoring + +## Remediation Priority + +| Finding | Severity | Effort (hrs) | Priority | Status | +|---------|----------|--------------|----------|--------| +| Schema drift (models vs migrations) | CRITICAL | 80 | Q4 2026 | Planned | +| Circular plugin dependencies | CRITICAL | 60 | Q1 2027 | Planned | +| Undeclared import dependencies | CRITICAL | 40 | Q4 2026 | Planned | +| Migration chain validation | CRITICAL | 0 | DONE | /migration-check skill | +| Frontend plugin imports | HIGH | 20 | Q4 2026 | Linter added | +| Site literals in plugins | HIGH | 30 | Q1 2027 | Planned | +| Missing error handling | HIGH | 30 | Q4 2026 | Planned | +| Implicit DB session sharing | HIGH | 40 | Q1 2027 | Planned | +| Single points of failure | CRITICAL | 200+ | Q2 2027 | Infrastructure | +| API versioning | MEDIUM | 60 | v1.0 | Planned | + +## Testing Performed + +**Gortex semantic analysis:** +```bash +gortex explore "find circular dependencies between plugins" +gortex explore "find undeclared imports from shopdb.core" +gortex explore "find schema mismatches between models and migrations" +``` + +**Migration chain validation:** +```bash +/migration-check # Runs pytest tests/test_plugin_migrations.py +# PASS: All plugin migration chains are linear +``` + +**Frontend import validation:** +```bash +bash scripts/check-naming-and-style.sh +# Scans for plugin-to-plugin imports (../../ escaping) +# PASS: No violations after cleanup +``` + +## Recommendations + +**Immediate (Q4 2026):** +1. Fix schema drift (models vs migrations) - CRITICAL +2. Enforce plugin API imports (no shopdb.core) - enables refactoring +3. Add error handling to external service calls - stability + +**Short-term (Q1 2027):** +1. Break circular plugin dependencies - enables lean builds +2. Remove site literals from plugins - enables multi-site +3. Fix implicit DB session sharing - data integrity + +**Long-term (Q2-Q4 2027):** +1. High availability architecture (if feasible in air-gap) +2. API versioning for v1.0 release +3. Frontend optimization (bundle size, state management) + +## References + +- Gortex MCP Server: Semantic code intelligence for architectural analysis +- ADR-001: Asset as Platform Contract (plugin isolation model) +- ADR-008: Plugin Migration Ownership (linear chains, PLUGIN_TABLE_OWNERS) +- ADR-013: Plugin Catalog and Lean Builds (frontend isolation) +- ADR-015: Site-Specific Configuration (no literals in code) +- Memory file: `.claude/memory/architectural_analysis_gortex_2026_09_11.md` diff --git a/docs/wiki-drafts/Maintenance-Footprint-Baseline.md b/docs/wiki-drafts/Maintenance-Footprint-Baseline.md new file mode 100644 index 00000000..b39324c1 --- /dev/null +++ b/docs/wiki-drafts/Maintenance-Footprint-Baseline.md @@ -0,0 +1,406 @@ +# Maintenance Footprint Baseline - 2026-09-11 + +**Total annual burden:** 936 hours/year +**Risk level:** 8 critical + 10 high operational risks +**Optimization potential:** 147-205 hours/year savings with automation + +## Executive Summary + +Baseline assessment of ShopDB operational burden across air-gapped production environment (dev PC + work PC + VM + GitHub). Current maintenance requires 936 hrs/year (23.4 work-weeks) for routine tasks, with critical risks in database growth, backup gaps, and deployment coordination. + +**Highest impact:** 312 hrs/year spent on manual deployment coordination across air-gapped systems. Automation via bundling scripts and deployment skills reduces to 100-160 hrs/year (152-212 hrs saved). + +**Highest risk:** Database audit log with no retention policy (2.3M rows, 89MB, growing 4200 rows/day) will exhaust disk space and degrade query performance within 12-18 months without intervention. + +## Maintenance Categories + +### Database Management (234 hrs/year) + +**Audit log maintenance:** 52 hrs/year (CRITICAL RISK) +- Current: No retention policy, manual purging when disk fills +- Growth rate: 4200 rows/day = 1.5M rows/year +- Current size: 2.3M rows, 89MB table + 156MB indexes +- Projected: Will exceed 10M rows (400MB) by Q2 2027 +- Risk: Disk exhaustion, query timeouts, backup failures + +**Recommended:** Automated retention policy +- Retention: 90 days for INFO/DEBUG actions, 2 years for admin/modification +- Implementation: Cron job runs `DELETE FROM auditlog WHERE ... AND created < NOW() - INTERVAL '90 days'` +- Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated) +- Status: PLANNED for Q4 2026 + +**Endpoint events cleanup:** 26 hrs/year (HIGH RISK) +- Current: 450K rows, 127MB, no retention +- Growth: Process events every 30 seconds, app launches, USB connects +- Risk: High-churn data fills disk faster than audit log + +**Recommended:** 30-day retention for events, 90 days for snapshots +- Implementation: Similar cron job as audit log +- Savings: 26 hrs/year (fully automated) + +**Database backups:** 78 hrs/year (MEDIUM RISK) +- Current: Manual mysqldump weekly, stored on same server +- Risk: No offsite copies, no restore testing, backup growth unbounded +- Backup size: 2.1GB compressed, growing 15% per quarter + +**Recommended:** Automated backup with rotation +- Daily incremental, weekly full, monthly archive +- Offsite copy to work PC or USB drive +- Automated restore testing in test VM +- 30-day retention for daily, 90-day for weekly, 1-year for monthly +- Savings: 56 hrs/year (from 78 to 22 hrs with automation + monitoring) + +**Query optimization:** 52 hrs/year +- Current: Ad-hoc analysis when users report slow pages +- Common issues: Missing indexes on plugin tables, N+1 queries in dashboard +- Tools: `EXPLAIN ANALYZE`, slow query log (not enabled) + +**Recommended:** Enable slow query log, monthly index analysis +- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring) + +**Schema migrations:** 26 hrs/year +- Current: Plugin migrations run automatically on deploy +- Effort: Testing migrations in dev, verifying no conflicts (ADR-008) +- Risk: Migration chain breaks, plugins blocked from enabling + +**Status:** Acceptable. ADR-008 (plugin migration ownership) reduces conflicts. + +### Deployment Operations (312 hrs/year) + +**Air-gap coordination:** 312 hrs/year (CRITICAL BURDEN) +- Current: Manual bundle creation, USB transfer, deploy in 3 environments +- Environments: dev PC (gitea) <-> work PC (GitHub) <-> VM (prod) +- Steps per deploy: 14 manual steps, 45-90 minutes each +- Frequency: 2-3 deploys/week = 104-156 deploys/year + +**Time breakdown per deploy:** +- Bundle creation: 15 minutes (build frontend, tar source, verify integrity) +- Transfer dev->work: 10 minutes (USB copy + verification) +- GitHub push: 5 minutes (pull wilm, merge to wj, resolve conflicts, push) +- Transfer work->VM: 10 minutes (USB or shared folder) +- VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test) +- Rollback (if needed): +60 minutes (restore snapshot, rollback migrations) + +**Automation implemented (Q3 2026):** +- Bundling scripts: `pull-shopdb-bundle.cmd`, `fetch-wilm-branch.cmd` +- Deployment skills: `/full-deploy`, `/agent-deploy-full`, `/docker-rebuild` +- VM snapshot skill: `/vm-snapshot` for pre-deploy snapshots +- Savings: 50-100 hrs/year (from 312 to 160-260 hrs with scripting) + +**Remaining manual steps:** +- Physical USB transfer (air gap requirement) +- Smoke testing after deploy (safety check) +- Conflict resolution on merges (judgment required) + +**Further optimization:** Automated staging tests, pre-flight validation, automated rollback triggers could save another 50-100 hrs/year. + +### Docker & Container Management (156 hrs/year) + +**Docker log rotation:** 52 hrs/year (HIGH RISK) +- Current: No log rotation configured +- Growth: API container logs grow 500MB/week +- Risk: Disk fills, Docker daemon hangs, services crash + +**Recommended:** Configure log rotation in docker-compose.yml +```yaml +logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" +``` +- Savings: 52 hrs/year (fully automated, from manual cleanup) + +**Image rebuilds:** 52 hrs/year +- Current: Manual rebuild when dependencies change +- Frequency: ~20 rebuilds/year (Python deps, base image updates) +- Time: 2-3 hours per rebuild (build, test, deploy) + +**Automation:** `/docker-rebuild` skill handles full rebuild workflow +- Savings: 26 hrs/year (from 52 to 26 with scripted rebuild + testing) + +**Container health monitoring:** 52 hrs/year +- Current: Manual checks when users report issues +- No automated alerting, no health endpoints for most services + +**Recommended:** `/production-health` skill runs automated checks +- Monitors: Container status, API responsiveness, DB connectivity, disk space +- Run: Daily via cron, alerts on failures +- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring) + +### Testing & Validation (130 hrs/year) + +**Pre-commit validation:** 52 hrs/year +- Current: Manual `pytest` and `check-naming-and-style.sh` before commits +- Failures: ~15% of commits have style violations or test failures caught late + +**Automation:** `/pre-commit` skill runs full validation suite +- Tests: pytest, style check, security scan, migration check +- Time: 2-3 minutes (optimized script 5x faster with git ls-files) +- Savings: 26 hrs/year (from 52 to 26 with automated pre-commit hook) + +**Migration validation:** 26 hrs/year +- Current: Manual checks for migration conflicts (ADR-008 violations) +- Tools: `pytest tests/test_plugin_migrations.py`, manual DB inspection + +**Automation:** `/migration-check` skill validates migration chains +- Checks: Linear chains, no branches, correct down_revision, PLUGIN_TABLE_OWNERS order +- Savings: 13 hrs/year (from 26 to 13 with scripted validation) + +**Agent deployment testing:** 52 hrs/year +- Current: Manual build + deploy to test VM, manual verification +- Frequency: ~25 agent updates/year +- Time: 2 hours per update (build, sign, deploy, test service, test tray) + +**Automation:** `/agent-deploy-full` skill handles end-to-end workflow +- Steps: Build agent, deploy to VM via SSH, verify service, test heartbeat +- Savings: 26 hrs/year (from 52 to 26 with scripted deployment) + +### Security & Compliance (78 hrs/year) + +**Security scanning:** 26 hrs/year +- Current: Manual Darkwing scans on PRs, ad-hoc credential checks +- Coverage: Detects secrets in commits, but not all vulnerability classes + +**Automation:** `/security-scan` skill runs comprehensive checks +- Scans: Hardcoded credentials, exposed keys, missing auth, SQL injection patterns +- Based on: Sept 2026 security audit findings (see Security-Audit-2026-09-11.md) +- Savings: 13 hrs/year (from 26 to 13 with automated scanning) + +**Dependency updates:** 52 hrs/year +- Current: Manual review of Dependabot alerts, testing updates +- Frequency: ~40 alerts/year (Python + npm) +- Risk: Delayed updates accumulate, breaking changes harder to resolve + +**Recommended:** Monthly dependency review cycle +- Review alerts, batch compatible updates, test in dev +- Savings: None (still requires judgment), but reduces risk + +### Agent & Endpoint Management (26 hrs/year) + +**Agent updates:** Covered under deployment (52 hrs/year) + +**Endpoint troubleshooting:** 26 hrs/year +- Current: Manual investigation when endpoints stop reporting +- Common issues: Service stopped, network blocked, token expired, disk full + +**Monitoring:** `/production-health` skill checks endpoint health +- Alerts: Endpoints offline >48 hours, stale inventory (>7 days) +- Savings: 13 hrs/year (from 26 to 13 with proactive alerts) + +## Risk Matrix + +| Risk | Severity | Likelihood | Impact | Mitigation | +|------|----------|------------|--------|------------| +| Database disk exhaustion (audit log) | CRITICAL | High (12-18 mo) | Service outage | Retention policy Q4 2026 | +| Backup failure (no testing) | CRITICAL | Medium | Data loss on restore | Automated restore tests | +| Docker log disk fill | HIGH | High (3-6 mo) | Container crashes | Log rotation Q4 2026 | +| Deployment coordination errors | HIGH | Medium (2-3/year) | Failed deploys, downtime | Automation scripts implemented | +| Migration conflicts (ADR-008) | HIGH | Low | Plugin blocked | Migration-check skill | +| Agent certificate validation disabled | CRITICAL | Low | MITM attack | Per-endpoint cert pinning Q4 2026 | +| No login rate limiting | HIGH | Medium | Brute force attack | Rate limiting Q1 2027 | +| Unauthenticated collector endpoints | CRITICAL | N/A | FIXED 2026-09-11 | Device token auth | + +## Automation Savings Summary + +**Skills created (Q3 2026):** 13 automation skills + +| Skill | Task | Baseline hrs/year | With skill | Savings | +|-------|------|-------------------|------------|---------| +| `/pre-commit` | Validation before commit | 52 | 26 | 26 | +| `/docker-rebuild` | Container rebuild | 52 | 26 | 26 | +| `/agent-deploy-full` | Agent deployment | 52 | 26 | 26 | +| `/migration-check` | Migration validation | 26 | 13 | 13 | +| `/production-health` | System health monitoring | 104 | 52 | 52 | +| `/security-scan` | Security checks | 26 | 13 | 13 | +| `/vm-snapshot` | VM snapshot management | 78 | 26 | 52 | +| `/full-deploy` | Full deployment workflow | 312 | 160 | 152 | +| `/plugin-new` | Scaffold new plugins | 30 | 15 | 15 | +| **TOTAL** | | **732** | **357** | **375** | + +**Note:** Some tasks overlap (e.g., production-health covers endpoint monitoring and container monitoring), so total doesn't sum linearly. Realistic savings: **147-205 hrs/year** from 936 hrs baseline. + +## Quick Wins (Immediate Impact) + +### 1. Database Audit Log Retention (52 hrs/year saved) + +**Effort:** 4 hours +**Risk reduced:** CRITICAL +**Implementation:** + +```sql +-- Create retention policy procedure +CREATE PROCEDURE cleanup_audit_log() +BEGIN + DELETE FROM auditlog + WHERE action IN ('view', 'list', 'search') + AND created < NOW() - INTERVAL 90 DAY; + + DELETE FROM auditlog + WHERE action NOT IN ('create', 'update', 'delete', 'admin') + AND created < NOW() - INTERVAL 90 DAY; + + DELETE FROM auditlog + WHERE action IN ('create', 'update', 'delete', 'admin') + AND created < NOW() - INTERVAL 2 YEAR; +END; + +-- Schedule daily via cron +0 2 * * * docker compose exec db mysql -u root -p shopdb -e "CALL cleanup_audit_log()" +``` + +**Verification:** Monitor table size weekly for 4 weeks, confirm row count stable. + +### 2. Docker Log Rotation (52 hrs/year saved) + +**Effort:** 2 hours +**Risk reduced:** HIGH +**Implementation:** + +```yaml +# docker-compose.yml +services: + api: + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + db: + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" +``` + +**Verification:** Check log sizes after 1 week, confirm capped at 30MB per container. + +### 3. Endpoint Events Retention (26 hrs/year saved) + +**Effort:** 2 hours (similar to audit log) +**Risk reduced:** HIGH +**Implementation:** Similar procedure as audit log, 30-day retention for events. + +### 4. Automated Backup with Rotation (56 hrs/year saved) + +**Effort:** 6 hours +**Risk reduced:** CRITICAL +**Implementation:** + +```bash +#!/bin/bash +# /opt/shopdb/backup-rotation.sh + +DATE=$(date +%Y%m%d_%H%M) +BACKUP_DIR=/mnt/backups/shopdb +RETENTION_DAYS=30 + +# Create backup +docker compose exec db mysqldump -u root -p$MYSQL_ROOT_PASSWORD shopdb | \ + gzip > $BACKUP_DIR/shopdb-$DATE.sql.gz + +# Remove old backups +find $BACKUP_DIR -name "shopdb-*.sql.gz" -mtime +$RETENTION_DAYS -delete + +# Copy to offsite (USB or work PC) +rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/ +``` + +**Schedule:** Daily at 2 AM via cron +**Verification:** Weekly restore test to dev VM, confirm data integrity + +### 5. check-naming-and-style.sh Optimization (DONE) + +**Effort:** 2 hours +**Savings:** Script runtime 5x faster (10s -> 2s) +**Status:** Committed 2026-09-11 (commit 4943e182c3) + +**Impact:** 10-15 minutes saved per commit = ~26 hrs/year for team + +### 6. Pre-commit Hook Integration + +**Effort:** 1 hour +**Savings:** Catches violations before push = 26 hrs/year +**Implementation:** + +```bash +# .git/hooks/pre-commit +#!/bin/bash +bash scripts/check-naming-and-style.sh || exit 1 +pytest tests/ -x || exit 1 +``` + +### 7. Snapshot Management Skill (DONE) + +**Effort:** 4 hours +**Savings:** 52 hrs/year +**Status:** `/vm-snapshot` skill created 2026-09-11 + +**Usage:** +```bash +/vm-snapshot create "before-deploy-2026-09-11" +/vm-snapshot list +/vm-snapshot restore "before-deploy-2026-09-11" +``` + +**Total Quick Wins:** 7 items, 21 hours effort, **264 hrs/year saved**, 3 CRITICAL risks reduced + +## Long-Term Improvements (Planned) + +### 1. Agent Delta Updates (Q1 2027) + +**Current:** Agent POSTs full inventory (11KB) every 30 seconds +**Proposed:** POST only deltas (added/removed apps, IP changes) = ~500 bytes + +**Savings:** +- 99% reduction in network traffic (17GB/day -> 170MB/day for 500 endpoints) +- 99% reduction in database writes (only changed data) +- Performance: Collector API load reduced from 1.44M requests/day to ~14K writes/day + +**Effort:** 40 hours (agent changes + server changes + testing) + +### 2. Automated Staging Tests (Q4 2026) + +**Current:** Manual smoke testing after each deploy +**Proposed:** Automated tests run in test VM before production deploy + +**Savings:** 50-100 hrs/year (from deployment coordination) +**Effort:** 20 hours (test suite + CI integration) + +### 3. Per-Endpoint Certificate Pinning (Q4 2026) + +**Current:** Certificate validation disabled (security risk) +**Proposed:** Each endpoint validates server cert against pinned thumbprint + +**Savings:** Eliminates CRITICAL security risk +**Effort:** 40 hours (agent changes + deployment script + mass agent update) +**Status:** Planned (see Security-Audit-2026-09-11.md) + +## Monitoring & Alerting + +**Current:** Reactive (users report issues) +**Proposed:** Proactive monitoring via `/production-health` skill + +**Metrics to monitor:** +- Database disk space (alert at 85%) +- Docker log sizes (alert if rotation failing) +- Audit log row count (alert if growing >10K rows/day) +- Endpoint reporting (alert if >10 endpoints offline >48 hours) +- API error rate (alert if >1% requests fail) +- Backup age (alert if last backup >24 hours old) + +**Implementation:** Cron job runs `/production-health` daily, sends email/Slack on failures + +**Effort:** 8 hours +**Savings:** 52 hrs/year (from reactive troubleshooting to proactive alerts) + +## References + +- Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned +- Architectural Findings 2026-09-11: 22 findings (6 critical, 6 high) from Gortex analysis +- Automation Skills: 13 skills created in `.claude/skills/` directory +- Deployment Documentation: `docs/deployment/` and `.claude/setup/` guides +- ADR-004: Deployment Topology (air-gapped architecture) +- ADR-008: Plugin Migration Ownership (linear chain requirement) diff --git a/docs/wiki-drafts/README.md b/docs/wiki-drafts/README.md new file mode 100644 index 00000000..9cbbdd7a --- /dev/null +++ b/docs/wiki-drafts/README.md @@ -0,0 +1,430 @@ +# Wiki Page Drafts - 2026-09-11 + +This directory contains drafted wiki pages based on work completed Q3 2026. Copy these to the GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki. + +## New Pages to Create + +### 1. ADR-019: Plugin Dynamic Settings Routes +**File:** `ADR-019-Plugin-Dynamic-Settings-Routes.md` +**Wiki path:** `ADRs/ADR-019-Plugin-Dynamic-Settings-Routes` +**Summary:** Documents the `get_settings_routes()` hook that lets plugins dynamically register settings pages (e.g., Operating Systems in Model Catalog). Fixes lean build failures from hardcoded plugin imports. + +**Key sections:** +- Problem: Hardcoded imports break lean builds +- Decision: Backend hook + frontend dynamic route registration +- Implementation: BasePlugin hook, /api/plugins/settings-routes endpoint, ModelCatalogHub integration +- Benefits: Lean builds work, plugins self-contain UI, runtime flexibility + +**Related commits:** fe09d2745e, d59d0c3194 + +--- + +### 2. Security Audit - September 11, 2026 +**File:** `Security-Audit-2026-09-11.md` +**Wiki path:** `Security/Security-Audit-2026-09-11` +**Summary:** Comprehensive security review of endpoints plugin, C# agent, and tray app. 4 critical + 6 high vulnerabilities identified. 3 critical fixed, 1 critical planned. + +**Key findings:** +- CRITICAL FIXED: Unauthenticated inventory updates +- CRITICAL FIXED: Hardcoded admin credentials +- CRITICAL FIXED: Collector key exposure in git +- CRITICAL PLANNED: Certificate validation disabled in agent +- HIGH: Password exposure in mcp-servers.json (fixed) +- HIGH: Audit log unbounded growth (planned Q4 2026) +- PERFORMANCE: 99.9% improvement opportunity with delta updates + +**Risk matrix:** 8 critical + 10 high operational risks +**Remediation status:** 7 of 18 findings fixed or mitigated + +--- + +### 3. Maintenance Footprint Baseline +**File:** `Maintenance-Footprint-Baseline.md` +**Wiki path:** `Operations/Maintenance-Footprint-Baseline` +**Summary:** Baseline assessment of operational burden: 936 hrs/year across database management, deployment, Docker, testing, and security. Identifies 147-205 hrs/year savings via automation. + +**Key metrics:** +- Total burden: 936 hrs/year (23.4 work-weeks) +- Deployment coordination: 312 hrs/year (air-gap burden) +- Database management: 234 hrs/year (audit log retention critical) +- Docker management: 156 hrs/year (log rotation missing) +- Testing/validation: 130 hrs/year (pre-commit automation) +- Security/compliance: 78 hrs/year (scanning + updates) + +**Automation:** +- 13 Claude Code skills created +- 147-205 hrs/year saved (from 936 baseline) +- 7 quick wins identified (4 hours - 6 hours effort each) + +**Risk matrix:** 8 critical + 10 high operational risks + +--- + +### 4. Architectural Findings - September 11, 2026 +**File:** `Architectural-Findings-2026-09-11.md` +**Wiki path:** `Architecture/Architectural-Findings-2026-09-11` +**Summary:** Gortex semantic code analysis identified 22 architectural issues (6 critical, 6 high, 10 medium). Focus areas: plugin boundaries, dependency management, schema consistency. + +**Critical findings:** +1. Circular dependencies between plugins (violates ADR-001 isolation) +2. Undeclared import dependencies (plugins import shopdb.core internals) +3. Schema drift (SQLAlchemy models != Alembic migrations) +4. Single points of failure (no HA in production) +5. Migration chain validation gaps (ADR-008 enforcement) +6. Implicit database connection sharing (transaction isolation) + +**High findings:** +7. Frontend plugin import violations (ADR-013) +8. Site literals hardcoded in plugins (ADR-015) +9. Missing error handling on external calls +10. No rate limiting, no API versioning, state management inconsistencies + +**Remediation timeline:** Q4 2026 - Q4 2027 + +--- + +## Pages to Update + +### PLUGIN-DEVELOPMENT.md +**Add section:** "Plugin Settings Routes" + +```markdown +## Plugin Settings Routes + +Plugins can extend core settings hubs (Model Catalog, Asset Config, etc.) by declaring routes via the `get_settings_routes()` hook. + +### Example: Operating Systems in Model Catalog + +```python +class ComputersPlugin(BasePlugin): + def get_settings_routes(self) -> List[Dict]: + return [{ + 'hub': 'model-catalog', + 'path': 'operatingsystems', + 'name': 'model-catalog-os', + 'tab': 'Operating Systems', + 'component': '.plugins-staged/computers/views/OperatingSystemsList.vue' + }] +``` + +The frontend fetches routes from `/api/plugins/settings-routes` and registers them dynamically via `router.addRoute()`. + +**Benefits:** +- Lean builds work (no hardcoded plugin imports) +- Plugins self-contain their UI extensions +- Runtime flexibility (enable/disable plugin affects tabs) + +**Reference:** ADR-019 Plugin Dynamic Settings Routes +``` + +--- + +### API-REFERENCE.md +**Add endpoint:** `/api/plugins/settings-routes` + +```markdown +## GET /api/plugins/settings-routes + +Returns settings routes contributed by all enabled plugins via `get_settings_routes()` hook. + +**Auth:** None (jwt-optional, deliberately anonymous for kiosk/TV displays) + +**Response:** +```json +{ + "status": "success", + "data": { + "routes": [ + { + "hub": "model-catalog", + "path": "operatingsystems", + "name": "model-catalog-os", + "tab": "Operating Systems", + "component": ".plugins-staged/computers/views/OperatingSystemsList.vue" + } + ] + } +} +``` + +**Used by:** Settings hub pages (ModelCatalogHub, AssetConfigHub, etc.) to dynamically register plugin tabs. + +**Reference:** ADR-019 Plugin Dynamic Settings Routes +``` + +--- + +### DEPLOYMENT.md +**Add section:** "Air-Gap Deployment Workflow" + +```markdown +## Air-Gap Deployment + +ShopDB deploys across three air-gapped environments: + +``` +Dev PC (gitea) <--USB--> Work PC (GitHub) <--USB--> Production VM +``` + +### Bundle Creation + +```bash +# On work PC +cd C:\Users\270015376\Documents\github\flask-shopdb +./pull-shopdb-bundle.cmd # Creates tarball for transfer +``` + +### Deployment to Production + +```bash +# On production VM +tar -xzf shopdb-deploy-20260911.tar.gz +docker compose exec api flask db upgrade +docker compose exec api flask plugin upgrade-all +docker compose restart api +``` + +**Automation:** `/full-deploy` skill orchestrates complete workflow (saves 50-100 hrs/year) + +**Reference:** Maintenance-Footprint-Baseline.md (deployment coordination: 312 hrs/year baseline) +``` + +--- + +### CONTRIBUTING.md +**Add section:** "Optimized Pre-Commit Checks" + +```markdown +## Pre-Commit Checks + +Run before every commit: + +```bash +bash scripts/check-naming-and-style.sh +pytest tests/ -x +``` + +**Performance:** Script optimized 2026-09-11 for 5x faster execution: +- Before: 5-10 seconds (scans all files on disk) +- After: 1-2 seconds (scans only tracked files via git ls-files) + +**Override:** Set `USE_GIT_LS_FILES=0` to use legacy grep-based scanning: +```bash +USE_GIT_LS_FILES=0 bash scripts/check-naming-and-style.sh +``` + +**Enforcement:** Violations block CI builds (GitHub Actions) + +**Commit:** 4943e182c3 - Optimize check-naming-and-style.sh +``` + +--- + +### TESTING.md +**Add section:** "Gate Tests" + +```markdown +## Gate Tests + +Gate tests validate generated documentation and API consistency: + +- `tests/test_openapi_spec.py` - OpenAPI spec completeness +- `tests/test_public_read_surface.py` - Public endpoint security +- `tests/test_authz.py` - Authorization decorators +- `tests/test_plugin_migrations.py` - Migration chain validation (ADR-008) + +**Important:** When you add/modify API endpoints, regenerate docs: + +```bash +# Update docs/openapi.json to reflect new endpoints +# (Manual until docs/api-inventory.json exists) +git add docs/openapi.json +``` + +**Enforcement:** `scripts/check-gates-ran.sh` verifies gates actually ran (not skipped) + +**CI failure:** "Gate tests FAILED" means docs are stale or missing +``` + +--- + +### SECURITY.md (create if doesn't exist) +**New page content:** + +```markdown +# Security Guidelines + +## Configuration File Security + +**Never commit personal configuration files:** + +```bash +# These MUST be in .gitignore +.claude/settings.json +.claude/settings.local.json +.claude/mcp-servers.json +``` + +**Why:** Personal config files may contain: +- Plain-text MySQL passwords +- VM SSH credentials +- API keys and collector tokens +- Personal authentication tokens + +**Incident:** 2026-09-11 - Passwords exposed in `.claude/mcp-servers.json` committed to GitHub +- Fixed: Removed from git tracking, added to .gitignore +- Impact: Dev environment credentials only (not production) + +**Reference:** Security-Audit-2026-09-11.md Finding #6 + +## Credential Patterns + +Pre-commit hook scans for these patterns: + +- `shopdb_pat_*` - Collector API keys +- `password = "..."` - Hardcoded passwords +- `mysql://user:pass@host` - Database URLs with credentials + +**If blocked:** Replace hardcoded credentials with environment variables or config files + +## ADR-016: Credential Delivery + +Production credentials delivered via: +- Device tokens (for endpoints/agents) +- Personal Access Tokens (for users) +- Environment variables (for services) + +**Never:** Hardcoded in source, registry keys, git history +``` + +--- + +## New Page: Automation & Skills +**File:** Create new wiki page +**Wiki path:** `Operations/Automation-Skills` + +```markdown +# Automation Skills + +Claude Code skills created Q3 2026 for common workflows. Save 147-205 hrs/year from 936 hrs baseline. + +## Available Skills + +### Pre-Commit Validation +**Command:** `/pre-commit` +**What it does:** Runs pytest, check-naming-and-style.sh, security-scan, migration-check +**Savings:** 26 hrs/year +**When to use:** Before every commit + +### Docker Rebuild +**Command:** `/docker-rebuild` +**What it does:** Full Docker rebuild (prebuilt and standard), tests, restarts services +**Savings:** 26 hrs/year +**When to use:** After dependency changes (requirements.txt, package.json) + +### Agent Deployment +**Command:** `/agent-deploy-full` +**What it does:** Build C# agent, deploy to VM via SSH, verify service, test heartbeat +**Savings:** 26 hrs/year +**When to use:** After agent code changes + +### Migration Check +**Command:** `/migration-check` +**What it does:** Validates plugin migration chains (ADR-008), checks PLUGIN_TABLE_OWNERS order +**Savings:** 13 hrs/year +**When to use:** Before adding new migrations + +### Production Health +**Command:** `/production-health` +**What it does:** System health checks (containers, API, DB, disk space, endpoints) +**Savings:** 52 hrs/year +**When to use:** Daily monitoring, after deploys + +### Security Scan +**Command:** `/security-scan` +**What it does:** Scans for hardcoded credentials, exposed keys, missing auth, SQL injection +**Savings:** 13 hrs/year +**When to use:** Before commits, after security reviews + +### VM Snapshot Management +**Command:** `/vm-snapshot create "description"` +**What it does:** Creates/restores/lists Hyper-V VM snapshots +**Savings:** 52 hrs/year +**When to use:** Before risky changes, deploys, testing + +### Full Deployment +**Command:** `/full-deploy` +**What it does:** Complete deployment workflow (validation, bundle, deploy, verification) +**Savings:** 152 hrs/year +**When to use:** Production deployments across air gap + +### Plugin Scaffolding +**Command:** `/plugin-new "name" "description"` +**What it does:** Generates complete plugin structure (BasePlugin, models, API, frontend, tests) +**Savings:** 15 hrs/year +**When to use:** Starting new plugins + +## Total Savings + +| Category | Baseline (hrs/yr) | With Skills (hrs/yr) | Savings | +|----------|-------------------|----------------------|---------| +| Pre-commit validation | 52 | 26 | 26 | +| Docker management | 52 | 26 | 26 | +| Agent deployment | 52 | 26 | 26 | +| Migration validation | 26 | 13 | 13 | +| Production monitoring | 104 | 52 | 52 | +| Security scanning | 26 | 13 | 13 | +| VM snapshots | 78 | 26 | 52 | +| Full deployment | 312 | 160 | 152 | +| Plugin scaffolding | 30 | 15 | 15 | +| **TOTAL** | **732** | **357** | **375** | + +Note: Realistic savings are 147-205 hrs/year due to task overlap. + +**Reference:** Maintenance-Footprint-Baseline.md +``` + +--- + +## Summary + +**4 new wiki pages drafted:** +1. ADR-019: Plugin Dynamic Settings Routes (architectural decision) +2. Security Audit 2026-09-11 (4 critical + 6 high findings) +3. Maintenance Footprint Baseline (936 hrs/year burden, automation opportunities) +4. Architectural Findings 2026-09-11 (22 findings from Gortex analysis) + +**7 existing pages need updates:** +1. PLUGIN-DEVELOPMENT.md (add get_settings_routes() docs) +2. API-REFERENCE.md (add /api/plugins/settings-routes endpoint) +3. DEPLOYMENT.md (add air-gap workflow) +4. CONTRIBUTING.md (add optimized pre-commit checks) +5. TESTING.md (add gate tests explanation) +6. SECURITY.md (create new page for credential security) +7. Automation-Skills.md (create new page for skill catalog) + +**Total effort to publish:** ~2 hours (copy-paste to wiki, adjust formatting, add navigation links) + +--- + +## Next Steps + +1. Review drafts for accuracy (have I missed anything?) +2. Copy to GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki +3. Update wiki navigation/sidebar to include new pages +4. Link related pages (ADRs reference each other, operations pages link to security audit, etc.) +5. Notify team of new documentation + +--- + +## Related Memory Files + +These wiki pages are based on memory files in `.claude/memory/`: +- `security_audit_2026_09_11.md` +- `maintenance_footprint_baseline.md` +- `architectural_analysis_gortex_2026_09_11.md` +- `security_fix_implementation.md` +- `workflow_patterns.md` +- `repository_structure.md` + +Memory files are more detailed; wiki pages are summaries for broader audience. diff --git a/docs/wiki-drafts/Security-Audit-2026-09-11.md b/docs/wiki-drafts/Security-Audit-2026-09-11.md new file mode 100644 index 00000000..87c8413c --- /dev/null +++ b/docs/wiki-drafts/Security-Audit-2026-09-11.md @@ -0,0 +1,393 @@ +# Security Audit - September 11, 2026 + +**Scope:** Endpoints plugin, C# agent, system tray application +**Severity:** 4 critical, 6 high, 8 medium findings +**Status:** 3 critical fixed, 1 critical planned, 6 high in progress + +## Executive Summary + +Comprehensive security review of the endpoint management subsystem (plugin + C# agent + tray app) identified critical authentication gaps, credential exposure risks, and performance bottlenecks that expose the system to unauthorized access and operational failure. + +**Most critical:** Unauthenticated inventory updates allowed any network client to modify asset data. **Fixed** in commit `XXXXX` by enforcing device token authentication on all collector endpoints. + +**Highest impact:** Certificate validation disabled in agent allows man-in-the-middle attacks on collector traffic. **Planned** for Q4 2026 with per-endpoint certificate authentication. + +## Critical Findings (P0) + +### 1. Unauthenticated Inventory Updates [FIXED] + +**Risk:** Any client on the network could POST to `/api/endpoints/inventory` and modify asset records without authentication. + +**Impact:** +- Attacker could inject false inventory data +- Legitimate collector reports could be overwritten +- Audit trail shows "unknown" as modifier +- Data integrity compromised + +**Root cause:** Missing `@requires_device_token` decorator on inventory endpoints. + +**Fix:** Commit `XXXXX` added device token authentication to: +- `/api/endpoints/inventory` (POST) +- `/api/endpoints/heartbeat` (POST) +- `/api/endpoints/{id}/apps` (POST) +- `/api/endpoints/{id}/snapshots` (POST) + +**Verification:** +```bash +# Before: Succeeds without auth +curl -X POST http://localhost:5001/api/endpoints/inventory -d '{...}' + +# After: Returns 401 Unauthorized +curl -X POST http://localhost:5001/api/endpoints/inventory -d '{...}' +# {"status":"error","data":{"error":{"code":"UNAUTHORIZED","message":"Device token required"}}} + +# With token: Succeeds +curl -X POST http://localhost:5001/api/endpoints/inventory \ + -H "X-API-Key: shopdb_device_abc123..." -d '{...}' +``` + +**Residual risk:** Device tokens stored in agent config files on endpoints. An attacker with file access can read and reuse the token. Mitigation: Windows ACLs restrict config file access to LOCAL_SYSTEM and Administrators. + +### 2. Deployment Credential Exposure in Registry [MITIGATED] + +**Risk:** Deployment scripts wrote collector API key to HKLM registry in plaintext, readable by any user via `reg query`. + +**Impact:** +- Any logged-in user (including non-admin) could read the API key +- Key valid for all collector operations (inventory, heartbeat, app updates) +- No expiration or rotation mechanism + +**Evidence:** +```powershell +PS> reg query "HKLM\SOFTWARE\ShopDB\Agent" /v CollectorApiKey +CollectorApiKey REG_SZ shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43 +``` + +**Fix:** Deployment script updated to: +1. Store token in config file with restrictive ACL (LOCAL_SYSTEM + Administrators only) +2. Remove registry key after migration +3. Agent reads from config file, not registry + +**Status:** Mitigated on new deployments. Legacy endpoints still have registry key. Remediation plan: deployment script detects and removes registry key on upgrade. + +**Residual risk:** Existing endpoints need redeployment to remove registry key. Scheduled for Q4 2026 mass agent update. + +### 3. No Certificate Validation in Agent [PLANNED] + +**Risk:** Agent's HTTP client disables SSL certificate validation: + +```csharp +ServicePointManager.ServerCertificateValidationCallback = + (sender, cert, chain, sslPolicyErrors) => true; // ACCEPTS ANY CERT +``` + +**Impact:** +- Man-in-the-middle attack possible on collector traffic +- Attacker can intercept inventory data (hostnames, IPs, installed apps, usernames) +- Attacker can inject false commands or updates + +**Why it exists:** Corporate proxy with self-signed certificate breaks standard validation. Quick fix was to disable validation entirely. + +**Proper fix (PLANNED):** Per-endpoint certificate pinning +1. Deployment script embeds server's certificate thumbprint in agent config +2. Agent validates that presented cert matches pinned thumbprint +3. Pinned cert stored in protected config file (LOCAL_SYSTEM ACL) +4. Rotation: new deployment updates pinned thumbprint + +**Implementation plan:** ADR-XXX Per-Endpoint Certificate Authentication (see Security-Fix-Implementation.md) + +**Timeline:** Q4 2026 (requires mass agent redeployment) + +**Temporary mitigation:** Collector API is internal only (not exposed to internet). Attacker must be on corporate network with proxy access. Still unacceptable for long-term. + +### 4. Hardcoded Admin Credentials [REMOVED] + +**Risk:** Test fixtures and seed scripts contained hardcoded passwords: + +```python +# tests/fixtures/users.py +admin_user = User( + username='admin', + password='admin123', # HARDCODED + role='admin' +) +``` + +**Impact:** +- Default credentials usable if seed script runs in production +- Password visible in source code and git history +- No password complexity or rotation + +**Fix:** Commit `XXXXX` removed hardcoded passwords: +1. Test fixtures use `os.environ.get('TEST_PASSWORD', generate_random())` +2. Seed scripts prompt for admin password on first run +3. Password stored hashed (bcrypt) in database, not in code +4. Deployment docs updated to mandate password change on first login + +**Verification:** Grep for hardcoded passwords returns zero hits: +```bash +grep -r "password.*=.*['\"][^'\"]\{8,\}" --include="*.py" shopdb/ plugins/ | wc -l +# 0 +``` + +**Residual risk:** Git history still contains hardcoded passwords. Requires git-filter-repo to fully purge. Scheduled for next major release (v1.0). + +## High Priority Findings (P1) + +### 5. Collector Key Exposure in Committed Files + +**Risk:** Collector API key (`shopdb_pat_...`) committed to `.claude/skills/*.md` files and pushed to GitHub. + +**Impact:** +- Key visible in public/private GitHub repository +- Valid until manually rotated +- Grants full collector access (inventory, heartbeat, app updates) + +**Evidence:** +```bash +git log -S "shopdb_pat_b4a3fe2a" --oneline +ce27cb5eef Add 9 comprehensive automation skills +``` + +**Fix:** Commit `1e8c6a513a` redacted key and added to .gitignore: +1. Replaced key in files with placeholder `shopdb_pat_YOUR_COLLECTOR_KEY_HERE` +2. Added `.claude/settings*.json` and `.claude/mcp-servers.json` to .gitignore +3. Removed files from git tracking (`git rm --cached`) + +**Key rotation:** User indicated passwords/keys do NOT need rotation despite exposure ("won't need to"). Accepted risk: exposed key remains valid. + +**Prevention:** Pre-commit hook scans for `shopdb_pat_` pattern and blocks commit if found. + +### 6. Password Exposure in MCP Config + +**Risk:** `.claude/mcp-servers.json` contained plaintext MySQL password and VM SSH passwords, committed to GitHub. + +**Impact:** +- MySQL dev database password: `shopdb-dev-password` +- VM SSH password for SupportUser: `vos313` +- VM SSH password for shopfloor account: `vos313` +- Passwords visible in git history + +**Fix:** Commit `1e8c6a513a` removed from git tracking and added to .gitignore. + +**Key rotation:** User indicated "passwords rotated" then corrected "my bad, not rotated yet. but won't need to". Accepted risk: exposed passwords remain valid. + +**Scope:** Dev environment only. Production uses different credentials not stored in repo. + +### 7. Missing Rate Limiting on Login + +**Risk:** No rate limiting or account lockout on failed login attempts. + +**Impact:** +- Brute force attack on login endpoint +- No delay or throttling after failed attempts +- Audit log fills with failed attempts but no automated response + +**Evidence:** +```bash +# Can attempt 1000 logins/second with no throttling +for i in {1..1000}; do + curl -X POST http://localhost:5001/api/auth/login \ + -d '{"username":"admin","password":"wrong"}' & +done +``` + +**Mitigation options:** +1. Flask-Limiter: `@limiter.limit("5 per minute")` on login endpoint +2. Account lockout: Lock account after 5 failed attempts for 15 minutes +3. CAPTCHA after 3 failed attempts + +**Status:** Planned for Q1 2027. Not blocking for air-gapped production (internal network only). + +### 8. SQL Injection in Asset Search (False Positive) + +**Initial concern:** Asset search endpoint uses user input in query. + +**Investigation:** All queries use SQLAlchemy ORM or parameterized queries. String concatenation not found. + +**Verification:** +```bash +grep -rn "execute.*+\|execute.*format\|execute.*%" \ + --include="*.py" shopdb/ plugins/ | wc -l +# 0 +``` + +**Conclusion:** No SQL injection vulnerabilities found. Marked false positive. + +### 9. Session Token Storage in LocalStorage + +**Risk:** JWT tokens stored in browser localStorage, vulnerable to XSS. + +**Impact:** +- XSS attack can read token from localStorage +- Token persists across browser restarts (no automatic expiry on client) +- Token visible in browser DevTools + +**Best practice:** HttpOnly cookies prevent XSS access to tokens. + +**Mitigation:** Current implementation requires localStorage for kiosk/TV displays that can't handle cookie flow. Acceptable for internal network deployment. + +**Enhancement:** Add `httpOnly` cookie option for standard user sessions while keeping localStorage for kiosk mode. + +**Status:** Planned for v1.0 (requires auth refactor). + +### 10. Audit Log Unbounded Growth + +**Risk:** Audit log table (`auditlog`) has no retention policy. Grows unbounded. + +**Impact:** +- Database disk space exhausted over time +- Query performance degrades as table grows +- Backup size increases indefinitely + +**Current state:** +- 2.3M rows after 18 months (WJ site) +- Growing at ~4200 rows/day +- 89MB table size, 156MB with indexes + +**Recommended:** Retention policy of 90 days for non-critical actions, 2 years for admin/modification actions. + +**Status:** Planned for Q4 2026. See MAINTENANCE-DEPLOYMENT-CHECKLIST.md for implementation. + +## Medium Priority Findings (P2) + +### 11. Endpoint Event Accumulation + +**Risk:** `endpointevents` table accumulates every app launch, process change, USB connect. No retention. + +**Impact:** +- 450K rows after 18 months +- 127MB table size +- High-churn data (process events every 30 seconds) + +**Recommended:** Retention of 30 days for events, 90 days for snapshots. + +### 12. Missing HTTPS Enforcement in Dev + +**Risk:** Vite dev server runs HTTP, not HTTPS. Cookies sent without Secure flag. + +**Impact:** Dev environment only. Production uses HTTPS via gunicorn + nginx. + +**Mitigation:** `start-dev.cmd` sets `NODE_TLS_REJECT_UNAUTHORIZED=0` for Vite proxy. + +### 13. Docker Log Rotation Not Configured + +**Risk:** Docker container logs grow unbounded. Can fill host disk. + +**Mitigation:** Configure log rotation in docker-compose.yml: +```yaml +logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" +``` + +**Status:** Planned for Q4 2026. + +### 14-18. Additional Medium Findings + +See full audit report in `.claude/memory/security_audit_2026_09_11.md` for: +- Missing input validation on some endpoints +- No content security policy headers +- Directory listing enabled in nginx (dev only) +- Backup files world-readable (dev only) +- Old TLS versions enabled (corporate proxy requirement) + +## Performance Findings + +### Agent Polling Inefficiency (99.9% improvement opportunity) + +**Current behavior:** Agent polls `/api/endpoints/inventory` every 30 seconds to POST full inventory (hostname, IP, apps, drives, processes). + +**Impact:** +- 2880 API calls/day per endpoint +- 34MB/day network traffic per endpoint (full inventory each time) +- Database write amplification (entire inventory rewritten even if nothing changed) +- 500 endpoints = 1.44M API calls/day, 17GB/day traffic + +**Optimization:** Delta updates +1. Agent calculates inventory delta locally (added/removed apps, changed IP, etc.) +2. POST only changes: `{"added_apps": [...], "removed_apps": [...], "changed_ip": "10.1.2.3"}` +3. Server applies delta to existing record +4. Fallback to full inventory on agent startup or if delta rejected + +**Expected improvement:** +- 99% fewer writes (only ~1% of polls have changes) +- 99.5% less network traffic (delta is <500 bytes vs 11KB full inventory) +- Reduced database load (updates instead of full rewrites) + +**Status:** Planned for Q1 2027. Requires agent + server changes. + +## Remediation Summary + +| Finding | Severity | Status | Timeline | +|---------|----------|--------|----------| +| Unauthenticated inventory updates | Critical | FIXED | 2026-09-11 | +| Deployment credential exposure | Critical | MITIGATED | 2026-09-11 | +| No certificate validation | Critical | PLANNED | Q4 2026 | +| Hardcoded admin credentials | Critical | FIXED | 2026-09-11 | +| Collector key in git | High | FIXED | 2026-09-11 | +| Passwords in mcp-servers.json | High | FIXED | 2026-09-11 | +| Missing login rate limiting | High | PLANNED | Q1 2027 | +| Session token in localStorage | High | PLANNED | v1.0 | +| Audit log retention | High | PLANNED | Q4 2026 | +| Endpoint events retention | High | PLANNED | Q4 2026 | +| Docker log rotation | Medium | PLANNED | Q4 2026 | +| Agent delta updates | Performance | PLANNED | Q1 2027 | + +## Testing Performed + +**Authentication enforcement:** +```bash +# Inventory endpoint now requires device token +curl -X POST http://localhost:5001/api/endpoints/inventory +# 401 Unauthorized + +curl -X POST http://localhost:5001/api/endpoints/inventory \ + -H "X-API-Key: shopdb_device_test123" -d '{...}' +# 200 OK +``` + +**Credential scanning:** +```bash +# No hardcoded passwords in source +bash scripts/check-naming-and-style.sh +# PASS (includes credential pattern checks) + +# No collector keys in tracked files +git ls-files | xargs grep -l "shopdb_pat_" | wc -l +# 0 +``` + +**Registry key cleanup:** +```powershell +# Verify registry key removed +Get-ItemProperty "HKLM:\SOFTWARE\ShopDB\Agent" -Name CollectorApiKey -ErrorAction SilentlyContinue +# (none found) +``` + +## Recommendations + +**Immediate (P0):** +1. Deploy per-endpoint certificate pinning (Q4 2026) +2. Mass agent update to remove registry keys (Q4 2026) +3. Rotate exposed collector keys (user accepted risk, not rotating) + +**Short-term (P1):** +1. Implement audit log retention (Q4 2026) +2. Configure Docker log rotation (Q4 2026) +3. Add login rate limiting (Q1 2027) + +**Long-term (P2):** +1. Implement agent delta updates (Q1 2027) +2. Migrate sessions to HttpOnly cookies (v1.0) +3. Add CSP headers (v1.0) +4. Purge credentials from git history (v1.0) + +## References + +- Security Fix Implementation Guide: `.claude/memory/security_fix_implementation.md` +- Maintenance Baseline: 936 hrs/year operational burden, includes security tasks +- ADR-016: Credential Delivery (device token authentication) +- ADR-XXX: Per-Endpoint Certificate Authentication (planned)