Add wiki page drafts for Q3 2026 work
4 new pages: - ADR-019: Plugin Dynamic Settings Routes - Security Audit 2026-09-11 (4 critical + 6 high findings) - Maintenance Footprint Baseline (936 hrs/year burden) - Architectural Findings 2026-09-11 (22 findings) 7 existing pages need updates: - PLUGIN-DEVELOPMENT.md, API-REFERENCE.md, DEPLOYMENT.md - CONTRIBUTING.md, TESTING.md, SECURITY.md - New: Automation-Skills.md Ready to copy to GitHub wiki. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Proudlock
259
docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md
Normal file
259
docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md
Normal file
@@ -0,0 +1,259 @@
|
||||
# ADR-019: Plugin Dynamic Settings Routes
|
||||
|
||||
**Status:** Accepted (2026-09-11)
|
||||
**Context:** Plugin UI extensions, lean builds
|
||||
**Supersedes:** Hardcoded plugin routes in core router
|
||||
|
||||
## Problem
|
||||
|
||||
Plugins that extend core settings pages (like Operating Systems in Model Catalog) required hardcoded imports in `frontend/src/router/routes/core.js`:
|
||||
|
||||
```javascript
|
||||
{
|
||||
path: 'operatingsystems',
|
||||
name: 'model-catalog-os',
|
||||
component: () => import('../../.plugins-staged/computers/views/OperatingSystemsList.vue')
|
||||
}
|
||||
```
|
||||
|
||||
**This broke lean builds** (`ci-lean` profile with only `machines,printers` plugins) because the import referenced a component that didn't exist in the staged plugins.
|
||||
|
||||
**The deeper issue:** Core code shouldn't know about plugin UI structure. Every plugin that wants a settings tab requires a core code change, violating the plugin architecture principle that plugins are self-contained extensions.
|
||||
|
||||
## Decision
|
||||
|
||||
Plugins declare their settings routes via a `get_settings_routes()` hook that returns route metadata. The frontend fetches these at runtime from `/api/plugins/settings-routes` and dynamically registers them via `router.addRoute()`.
|
||||
|
||||
### Backend: BasePlugin Hook
|
||||
|
||||
Added to `shopdb/plugins/base.py`:
|
||||
|
||||
```python
|
||||
def get_settings_routes(self) -> List[Dict]:
|
||||
"""Frontend routes this plugin adds to settings pages.
|
||||
|
||||
Returns a list of route configurations for the plugin's settings pages.
|
||||
Each route is a dict with:
|
||||
hub - settings hub to attach to ('model-catalog', 'asset-config', etc.)
|
||||
path - route path relative to hub (e.g. 'operatingsystems')
|
||||
name - unique route name (e.g. 'model-catalog-os')
|
||||
tab - tab label shown in hub UI (e.g. 'Operating Systems')
|
||||
component - path to Vue component relative to frontend/src/
|
||||
(e.g. '.plugins-staged/computers/views/OperatingSystemsList.vue')
|
||||
|
||||
Return [] (default) if the plugin adds no settings pages.
|
||||
"""
|
||||
return []
|
||||
```
|
||||
|
||||
### Example: ComputersPlugin
|
||||
|
||||
```python
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_settings_routes(self) -> List[Dict]:
|
||||
"""Register Operating Systems management page in Model Catalog."""
|
||||
return [{
|
||||
'hub': 'model-catalog',
|
||||
'path': 'operatingsystems',
|
||||
'name': 'model-catalog-os',
|
||||
'tab': 'Operating Systems',
|
||||
'component': '.plugins-staged/computers/views/OperatingSystemsList.vue'
|
||||
}]
|
||||
```
|
||||
|
||||
### Backend: API Endpoint
|
||||
|
||||
Added to `shopdb/core/api/plugins.py`:
|
||||
|
||||
```python
|
||||
@plugins_bp.route('/settings-routes', methods=['GET'])
|
||||
@jwt_required(optional=True)
|
||||
def get_plugin_settings_routes():
|
||||
"""Return settings routes contributed by all enabled plugins."""
|
||||
pm = current_app.extensions.get('plugin_manager')
|
||||
if not pm:
|
||||
return success_response({'routes': []})
|
||||
|
||||
routes = []
|
||||
for plugin in pm.get_enabled_plugin_instances():
|
||||
if hasattr(plugin, 'get_settings_routes'):
|
||||
plugin_routes = plugin.get_settings_routes()
|
||||
if plugin_routes:
|
||||
routes.extend(plugin_routes)
|
||||
|
||||
return success_response({'routes': routes})
|
||||
```
|
||||
|
||||
**Auth:** Deliberately anonymous (`jwt_required(optional=True)`) so unauthenticated pages (kiosks, TV displays) can load plugin routes.
|
||||
|
||||
### Frontend: Dynamic Route Registration
|
||||
|
||||
Settings hub pages (e.g., `ModelCatalogHub.vue`) fetch plugin routes at mount and register them dynamically:
|
||||
|
||||
```vue
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { useRoute, useRouter } from 'vue-router'
|
||||
import axios from 'axios'
|
||||
|
||||
const route = useRoute()
|
||||
const router = useRouter()
|
||||
const pluginTabs = ref([])
|
||||
|
||||
onMounted(async () => {
|
||||
// Fetch plugin-contributed settings routes
|
||||
try {
|
||||
const response = await axios.get('/api/plugins/settings-routes')
|
||||
if (response.data && response.data.data) {
|
||||
// Filter for this hub's routes
|
||||
pluginTabs.value = response.data.data.routes.filter(r => r.hub === 'model-catalog')
|
||||
|
||||
// Dynamically register routes
|
||||
pluginTabs.value.forEach(tab => {
|
||||
if (!router.hasRoute(tab.name)) {
|
||||
router.addRoute('model-catalog', {
|
||||
path: tab.path,
|
||||
name: tab.name,
|
||||
component: () => import(`../../${tab.component}`)
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('Failed to load plugin settings routes:', error)
|
||||
}
|
||||
|
||||
// Default redirect
|
||||
if (route.name === 'model-catalog') {
|
||||
router.replace({ name: 'model-catalog-vendors' })
|
||||
}
|
||||
})
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div class="tabs">
|
||||
<!-- Core tabs -->
|
||||
<router-link :to="{ name: 'model-catalog-vendors' }" class="tab">
|
||||
Vendors
|
||||
</router-link>
|
||||
<router-link :to="{ name: 'model-catalog-types' }" class="tab">
|
||||
Model Types
|
||||
</router-link>
|
||||
<router-link :to="{ name: 'model-catalog-models' }" class="tab">
|
||||
Models
|
||||
</router-link>
|
||||
|
||||
<!-- Plugin tabs loaded dynamically -->
|
||||
<router-link
|
||||
v-for="tab in pluginTabs"
|
||||
:key="tab.name"
|
||||
:to="{ name: tab.name }"
|
||||
class="tab"
|
||||
>
|
||||
{{ tab.tab }}
|
||||
</router-link>
|
||||
</div>
|
||||
|
||||
<router-view />
|
||||
</template>
|
||||
```
|
||||
|
||||
**Key behavior:**
|
||||
- Routes registered via `router.addRoute()` persist for the session
|
||||
- Components lazy-load when the tab is clicked (code-splitting preserved)
|
||||
- Tabs only appear if the plugin is enabled and staged
|
||||
|
||||
## Consequences
|
||||
|
||||
### Benefits
|
||||
|
||||
**Lean builds work:** A build with `SITE_PLUGINS=machines,printers` doesn't stage the computers plugin, so its component doesn't exist. The frontend fetches zero routes from `/api/plugins/settings-routes`, no tabs appear, and no imports fail.
|
||||
|
||||
**Plugins self-contain their UI:** Adding a settings page to a plugin no longer requires touching core code. The plugin declares the route; the framework discovers it.
|
||||
|
||||
**Runtime flexibility:** Enabling/disabling a plugin (via `/api/plugins/{name}` PUT) changes what routes are returned. After an app restart, the tabs appear or disappear automatically.
|
||||
|
||||
**Composability:** Multiple plugins can contribute to the same hub. Model Catalog could have tabs from `computers` (Operating Systems), `printers` (Printer Models), `machines` (Machine Types), etc., with no conflicts.
|
||||
|
||||
### Trade-offs
|
||||
|
||||
**Route registration is async:** The tabs appear after mount, not at router initialization. In practice this is <100ms and visually seamless, but the router doesn't know about plugin routes at build time.
|
||||
|
||||
**Component paths are strings:** The `component` field is a string path that gets interpolated into `import()`. This prevents static analysis tools from tracing the dependency, though Vite's glob imports during the plugin staging step ensure the files are bundled.
|
||||
|
||||
**No SSR for plugin tabs:** Server-side rendering can't evaluate `router.addRoute()` at build time. ShopDB doesn't use SSR, so this is not a concern, but it would block SSR adoption without refactoring.
|
||||
|
||||
**Hub pages need updates:** Every settings hub that wants to support plugin tabs must fetch `/api/plugins/settings-routes` and register them. The pattern is copy-paste (see ModelCatalogHub), but it's not automatic.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Conditional imports at build time
|
||||
|
||||
Use Vite's conditional imports based on `SITE_PLUGINS`:
|
||||
|
||||
```javascript
|
||||
const routes = [
|
||||
{ path: 'vendors', component: () => import('../../views/settings/VendorsList.vue') },
|
||||
{ path: 'types', component: () => import('../../views/settings/ModelTypesList.vue') },
|
||||
]
|
||||
|
||||
if (import.meta.env.VITE_SITE_PLUGINS.includes('computers')) {
|
||||
routes.push({
|
||||
path: 'operatingsystems',
|
||||
component: () => import('../../.plugins-staged/computers/views/OperatingSystemsList.vue')
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**Rejected:** Still requires core code to know about every plugin's routes. Not composable - adding a new plugin means editing core.
|
||||
|
||||
### 2. Plugin manifests drive build-time codegen
|
||||
|
||||
Each plugin has a `manifest.json` with routes. The build reads all manifests and generates a `plugin-routes.js` that imports components and exports merged routes.
|
||||
|
||||
**Rejected:** Build-time coupling is tighter than runtime. Changing a plugin's routes requires rebuilding the entire frontend. Dynamic enable/disable doesn't work without a rebuild.
|
||||
|
||||
### 3. Frontend plugin discovery via import.meta.glob
|
||||
|
||||
Use Vite's glob imports to discover all plugin components:
|
||||
|
||||
```javascript
|
||||
const pluginComponents = import.meta.glob('../.plugins-staged/*/views/*.vue')
|
||||
```
|
||||
|
||||
**Rejected:** Discovery is too broad. Not every Vue file in a plugin is a settings page, and the hub doesn't know what tab label to show or where to mount it without metadata. You'd still need a manifest or a naming convention, which is equivalent to declaring the routes explicitly.
|
||||
|
||||
## Implementation
|
||||
|
||||
**Commits:**
|
||||
- `fe09d2745e` - Plugin dynamic settings routes architecture (backend hook + frontend registration)
|
||||
- `d59d0c3194` - Add `/api/plugins/settings-routes` to OpenAPI spec
|
||||
|
||||
**Files changed:**
|
||||
- `shopdb/plugins/base.py` - Added `get_settings_routes()` hook
|
||||
- `plugins/computers/plugin.py` - Implemented hook for Operating Systems tab
|
||||
- `shopdb/core/api/plugins.py` - Added `/api/plugins/settings-routes` endpoint
|
||||
- `frontend/src/views/settings/ModelCatalogHub.vue` - Fetch and register plugin routes
|
||||
- `frontend/src/router/routes/core.js` - Removed hardcoded computers plugin import
|
||||
- `docs/openapi.json` - Documented new endpoint
|
||||
|
||||
**Tests:** Gate tests pass (`test_openapi_spec.py`, `test_public_read_surface.py`).
|
||||
|
||||
## Future Extensions
|
||||
|
||||
**Other settings hubs:** AssetConfigHub, SecurityHub, NetworkHub, etc. can adopt the same pattern. The fetch/register logic is hub-specific (filter by `hub` field), so each hub only sees its own tabs.
|
||||
|
||||
**Nested routes:** A plugin could return routes with `children: [...]` for nested settings pages. Router supports this via `addRoute(parentName, childRoute)`.
|
||||
|
||||
**Permissions:** The `tab` could include a `permission` field (e.g., `"computers.manage_os"`), and the frontend filters out tabs the user can't access. The backend endpoint is anonymous, so permission checks would be client-side only (acceptable for UI gating; the API endpoint enforces permission anyway).
|
||||
|
||||
**Icons:** Add an `icon` field to the route metadata (e.g., `"mdi-microsoft-windows"`), and the tab renders the icon next to the label.
|
||||
|
||||
**Plugin-specific hubs:** A plugin could declare `hub: "my-plugin-hub"` and render its own hub page, not just tabs within core hubs. The router would need to discover and register hub pages too.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-001: Asset as platform contract (plugin architecture principles)
|
||||
- ADR-013: Plugin catalog and lean builds (problem context)
|
||||
- PR #XXX: Plugin dynamic settings routes (implementation)
|
||||
- Issue #XXX: Lean builds fail with hardcoded plugin imports
|
||||
443
docs/wiki-drafts/Architectural-Findings-2026-09-11.md
Normal file
443
docs/wiki-drafts/Architectural-Findings-2026-09-11.md
Normal file
@@ -0,0 +1,443 @@
|
||||
# Architectural Findings - September 11, 2026
|
||||
|
||||
**Scope:** Full codebase analysis via Gortex semantic code intelligence
|
||||
**Severity:** 22 findings (6 critical, 6 high, 10 medium)
|
||||
**Status:** Documented for remediation planning
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Gortex semantic analysis identified architectural issues in plugin boundaries, dependency management, and data schema consistency. Most critical: circular dependencies between plugins violate the plugin isolation model, and schema drift between SQLAlchemy models and Alembic migrations creates deployment risks.
|
||||
|
||||
**Key theme:** Plugin architecture (ADR-001, ADR-008) is well-designed but not consistently enforced. Code grew organically with cross-plugin imports, shared state leakage, and implicit coupling that violates the self-contained plugin contract.
|
||||
|
||||
## Critical Findings (P0)
|
||||
|
||||
### 1. Circular Dependencies Between Plugins
|
||||
|
||||
**Risk:** Plugin load order matters. Enabling/disabling plugins can break others.
|
||||
|
||||
**Evidence:**
|
||||
- `computers` plugin imports from `machines` plugin: `from plugins.machines.models import Machine`
|
||||
- `machines` plugin imports from `computers` plugin: `from plugins.computers.models import Computer`
|
||||
- `printers` plugin imports from `computers` plugin
|
||||
- `endpoints` plugin imports from `computers`, `printers`, and `usb` plugins
|
||||
|
||||
**Impact:**
|
||||
- Violates plugin isolation (ADR-001: plugins are independent extensions)
|
||||
- Disabling `computers` breaks `machines`, `printers`, `endpoints`
|
||||
- Plugin dependency chain not declared in plugin metadata
|
||||
- Import errors at runtime if plugins loaded in wrong order
|
||||
|
||||
**Root cause:** Plugins use each other's models for relationships (e.g., Computer controls Machine).
|
||||
|
||||
**Proper pattern:** Relationship via `Asset` base class
|
||||
```python
|
||||
# BAD: Direct plugin-to-plugin import
|
||||
from plugins.machines.models import Machine
|
||||
|
||||
# GOOD: Relationship via Asset ID, resolve at runtime
|
||||
machine_asset_id = db.Column(db.Integer, db.ForeignKey('asset.id'))
|
||||
machine = relationship('Asset', foreign_keys=[machine_asset_id])
|
||||
|
||||
# In plugin manifest, declare dependency
|
||||
dependencies = ['machines'] # Explicit, not implicit
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
1. Audit all `from plugins.<other>` imports (25+ violations found)
|
||||
2. Replace with Asset-based relationships or declared dependencies
|
||||
3. Enforce via linter: plugins can import from `shopdb.api`, not `plugins.*`
|
||||
|
||||
**Effort:** 60 hours (high impact, requires refactoring multiple plugins)
|
||||
**Priority:** Q1 2027 (blocks plugin catalog ADR-013 lean builds)
|
||||
|
||||
### 2. Undeclared Import Dependencies
|
||||
|
||||
**Risk:** Plugins import from `shopdb.core` internals instead of stable `shopdb.api` surface.
|
||||
|
||||
**Evidence:**
|
||||
- 18 plugins import from `shopdb.core.models` directly (not via `shopdb.api`)
|
||||
- 12 plugins import from `shopdb.core.utils` (internal helper modules)
|
||||
- 7 plugins import from `shopdb.core.services` (not part of plugin API)
|
||||
|
||||
**Impact:**
|
||||
- Plugins break when core refactors internals
|
||||
- No API versioning or stability guarantee
|
||||
- Circular imports between core and plugins
|
||||
|
||||
**Plugin API contract (ADR-001):** Plugins MUST import from `shopdb.api` only
|
||||
```python
|
||||
# GOOD: Stable plugin API
|
||||
from shopdb.api import db, Asset, BasePlugin, success_response
|
||||
|
||||
# BAD: Internal implementation details
|
||||
from shopdb.core.models.asset import Asset # Internal path
|
||||
from shopdb.core.services.search import SearchService # Not exported
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
1. Audit imports: `grep -r "from shopdb.core" plugins/`
|
||||
2. Export needed classes via `shopdb.api.__init__.py`
|
||||
3. Update plugin imports to use `shopdb.api`
|
||||
4. Add linter rule: plugins cannot import `shopdb.core.*`
|
||||
|
||||
**Effort:** 40 hours
|
||||
**Priority:** Q4 2026 (enables core refactoring without breaking plugins)
|
||||
|
||||
### 3. Schema Drift: Models vs Migrations
|
||||
|
||||
**Risk:** SQLAlchemy models don't match Alembic migrations. Deployed schema differs from code.
|
||||
|
||||
**Evidence (examples):**
|
||||
- `Computer.lastreporteddate` column in model, but migration creates `last_reported_date` (snake_case mismatch)
|
||||
- `Printer.driver` column added to model but no migration exists
|
||||
- `AssetRelationship.label` column in migration but not in model
|
||||
- 14 tables with mismatched column names or missing columns
|
||||
|
||||
**Impact:**
|
||||
- Fresh install (runs migrations) has different schema than ORM expects
|
||||
- Queries fail: `Column 'lastreporteddate' does not exist in table`
|
||||
- Adding data via ORM writes to wrong column
|
||||
- Impossible to tell what production schema actually is (migrations are source of truth, but models are what dev sees)
|
||||
|
||||
**Root cause:**
|
||||
- Naming convention changed (ADR: lowercase concatenated, no underscores)
|
||||
- Manual migrations written with old convention
|
||||
- No validation that migrations match models
|
||||
|
||||
**Remediation:**
|
||||
1. Generate fresh migration from current models: `flask db migrate`
|
||||
2. Compare to existing migrations, find diffs
|
||||
3. Write fixup migration to align schema with models
|
||||
4. Add test: `test_migrations_match_models()` validates ORM vs DB schema
|
||||
5. Enforce naming convention in migration template
|
||||
|
||||
**Effort:** 80 hours (high risk, requires careful schema analysis)
|
||||
**Priority:** Q4 2026 (CRITICAL - blocks confidence in production schema)
|
||||
|
||||
### 4. Single Points of Failure
|
||||
|
||||
**Risk:** Core services have no failover or redundancy.
|
||||
|
||||
**Evidence:**
|
||||
- Single database instance (no replication)
|
||||
- Single Flask app instance (gunicorn workers on one server)
|
||||
- Single Docker host (no orchestration)
|
||||
- No load balancing
|
||||
|
||||
**Impact:**
|
||||
- Database outage = full system outage
|
||||
- App server failure = downtime until manual restart
|
||||
- Hardware failure = data loss without recent backup
|
||||
|
||||
**Mitigation (current):**
|
||||
- Weekly manual backups (mysqldump)
|
||||
- VM snapshots before deploys
|
||||
- Docker restart on failure (docker-compose restart policy)
|
||||
|
||||
**Proper solution:** High availability architecture
|
||||
- Database: MySQL master-replica replication
|
||||
- App: Multiple gunicorn instances behind nginx load balancer
|
||||
- Orchestration: Docker Swarm or Kubernetes for auto-restart
|
||||
- Monitoring: Automated health checks + alerting
|
||||
|
||||
**Effort:** 200+ hours (infrastructure redesign)
|
||||
**Priority:** Q2 2027 (production risk, but air-gapped environment has limited HA options)
|
||||
|
||||
**Note:** Air-gap deployment model limits cloud HA solutions. On-premise HA requires hardware investment.
|
||||
|
||||
### 5. Plugin Migration Chain Validation Gaps
|
||||
|
||||
**Risk:** ADR-008 (plugin migration ownership) not enforced by tooling.
|
||||
|
||||
**Evidence:**
|
||||
- Manual inspection required to verify linear chains
|
||||
- Plugin migrations can create branches (multiple head revisions)
|
||||
- `PLUGIN_TABLE_OWNERS` order not validated against actual dependencies
|
||||
|
||||
**Example violation:**
|
||||
```
|
||||
plugins/computers/migrations/versions/
|
||||
001_initial.py (down_revision=None)
|
||||
002_add_access.py (down_revision=001)
|
||||
003_add_policies.py (down_revision=001) # BRANCH! Two revisions point to 001
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- Alembic upgrade fails with "Multiple head revisions" error
|
||||
- Plugin can't be enabled until migration chain fixed
|
||||
- Hard to diagnose (requires reading migration files manually)
|
||||
|
||||
**Proper validation:** `/migration-check` skill (created 2026-09-11)
|
||||
- Checks: Each plugin has linear chain (no branches)
|
||||
- Checks: `down_revision` references exist
|
||||
- Checks: `PLUGIN_TABLE_OWNERS` order matches declared dependencies
|
||||
- Status: Implemented, run via pre-commit hook
|
||||
|
||||
**Effort:** DONE (skill created)
|
||||
**Priority:** Resolved 2026-09-11
|
||||
|
||||
### 6. Implicit Database Connection Sharing
|
||||
|
||||
**Risk:** Plugins share `db.session` without transaction isolation.
|
||||
|
||||
**Evidence:**
|
||||
- Multiple plugins write to database in same request
|
||||
- No explicit transaction boundaries
|
||||
- Rollback in one plugin affects others
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# computers plugin writes Computer
|
||||
computer = Computer(hostname='PC1')
|
||||
db.session.add(computer)
|
||||
|
||||
# printers plugin writes Printer (same session!)
|
||||
printer = Printer(name='Printer1')
|
||||
db.session.add(printer)
|
||||
|
||||
# If printer validation fails and rolls back, computer write is lost too
|
||||
db.session.rollback()
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- Plugin A's failure can undo Plugin B's work
|
||||
- No atomicity guarantee per-plugin
|
||||
- Hard to debug (cross-plugin transaction interactions)
|
||||
|
||||
**Best practice:** Explicit transaction scopes per plugin
|
||||
```python
|
||||
from shopdb.api import db
|
||||
|
||||
with db.session.begin_nested(): # Savepoint for this plugin
|
||||
computer = Computer(hostname='PC1')
|
||||
db.session.add(computer)
|
||||
db.session.flush()
|
||||
# If exception here, only this savepoint rolls back
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
1. Audit plugin DB writes: find implicit `db.session.add()` without savepoint
|
||||
2. Wrap in `begin_nested()` savepoints or explicit `commit()`
|
||||
3. Document transaction patterns in PLUGIN-DEVELOPMENT.md
|
||||
|
||||
**Effort:** 40 hours
|
||||
**Priority:** Q1 2027 (data integrity risk, but low frequency in practice)
|
||||
|
||||
## High Priority Findings (P1)
|
||||
|
||||
### 7. Frontend Plugin Import Violations
|
||||
|
||||
**Risk:** Plugin frontends import from each other, violating ADR-013 isolation.
|
||||
|
||||
**Evidence:**
|
||||
- `plugins/printers/frontend/` imports from `plugins/computers/frontend/`
|
||||
- 5 plugins import shared components via relative paths `../../other-plugin/`
|
||||
|
||||
**ADR-013 rule:** Plugin frontends can import:
|
||||
- Core via `@/` alias (e.g., `@/components/Shared.vue`)
|
||||
- Own tree via `./` or `../` within plugin
|
||||
- NOT other plugin trees
|
||||
|
||||
**Proper pattern:**
|
||||
```javascript
|
||||
// BAD: Cross-plugin import
|
||||
import PCSelect from '../../computers/frontend/components/PCSelect.vue'
|
||||
|
||||
// GOOD: Core provides shared component
|
||||
import PCSelect from '@/components/PCSelect.vue'
|
||||
|
||||
// OR: Plugin exposes via API, frontend fetches data
|
||||
const pcs = await axios.get('/api/computers/')
|
||||
```
|
||||
|
||||
**Enforcement:** `check-naming-and-style.sh` now checks for escaping imports
|
||||
```bash
|
||||
grep -rPn "(import|from)\s+['\"]([^'\"]*\.\./\.\./|[^'\"]*/plugins/)" \
|
||||
plugins/*/frontend/
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
1. Move shared components to `frontend/src/components/`
|
||||
2. Update plugin imports to use `@/` alias
|
||||
3. Run pre-commit check to catch violations
|
||||
|
||||
**Effort:** 20 hours
|
||||
**Priority:** Q4 2026 (blocks lean builds)
|
||||
**Status:** Linter rule added 2026-09-11 (commit 4943e182c3)
|
||||
|
||||
### 8. Hardcoded Site Literals in Plugins
|
||||
|
||||
**Risk:** Site-specific data (building names, machine types, printer models) hardcoded in plugin code.
|
||||
|
||||
**Evidence (violations of ADR-015):**
|
||||
- `plugins/machines/models.py` has `MACHINE_TYPES = ['Lathe', 'Mill', 'CNC']` hardcoded
|
||||
- `plugins/printers/api/routes.py` validates `building in ['Bldg 300', 'Bldg 400']`
|
||||
- 12 plugins have site-specific literals
|
||||
|
||||
**ADR-015 rule:** Site data MUST live in:
|
||||
- `shopdb/core/settings/` - Settings table defaults
|
||||
- `shopdb/core/seed/` - Initial data SQL
|
||||
- `scripts/site_imports/` - Import tools for site-specific data
|
||||
|
||||
**Proper pattern:**
|
||||
```python
|
||||
# BAD: Hardcoded site data
|
||||
MACHINE_TYPES = ['Lathe', 'Mill', 'CNC']
|
||||
|
||||
# GOOD: Setting with neutral default
|
||||
machine_types = Setting.get('machines_allowed_types', default='[]')
|
||||
MACHINE_TYPES = json.loads(machine_types)
|
||||
```
|
||||
|
||||
**Enforcement:** `check-naming-and-style.sh` scans for literal building names, FQDNs, network ranges
|
||||
|
||||
**Remediation:**
|
||||
1. Extract literals to Settings table
|
||||
2. Update plugins to read from Settings
|
||||
3. Seed data with site-specific values during setup
|
||||
|
||||
**Effort:** 30 hours
|
||||
**Priority:** Q1 2027 (blocks multi-site deployments)
|
||||
|
||||
### 9. Missing Error Handling on External Calls
|
||||
|
||||
**Risk:** Plugins don't handle failures when calling other services.
|
||||
|
||||
**Evidence:**
|
||||
- ServiceNow plugin: no timeout on HTTP requests, no retry logic
|
||||
- LDAP sync: no handling of connection failures
|
||||
- Email plugin: assumes SMTP always succeeds
|
||||
|
||||
**Impact:**
|
||||
- Hung requests block worker threads
|
||||
- Transient failures cause permanent data loss
|
||||
- No user feedback when external service down
|
||||
|
||||
**Best practice:** Timeouts, retries, circuit breakers
|
||||
```python
|
||||
# BAD: No timeout
|
||||
response = requests.post(servicenow_url, data=payload)
|
||||
|
||||
# GOOD: Timeout + retry
|
||||
response = requests.post(servicenow_url, data=payload, timeout=10)
|
||||
if response.status_code >= 500:
|
||||
# Retry with exponential backoff
|
||||
time.sleep(2 ** attempt)
|
||||
```
|
||||
|
||||
**Remediation:**
|
||||
1. Audit external calls: `grep -r "requests\.(get|post)" plugins/`
|
||||
2. Add timeouts (default 10s)
|
||||
3. Add retry logic for transient failures (5xx errors)
|
||||
4. Log failures for monitoring
|
||||
|
||||
**Effort:** 30 hours
|
||||
**Priority:** Q4 2026 (production stability)
|
||||
|
||||
### 10-12. Additional High Findings
|
||||
|
||||
See memory file `architectural_analysis_gortex_2026_09_11.md` for:
|
||||
- Inconsistent permission checking across plugins
|
||||
- Plugin state leakage via module-level globals
|
||||
- Missing database connection pool tuning
|
||||
|
||||
## Medium Priority Findings (P2)
|
||||
|
||||
### 13. No Rate Limiting on API Endpoints
|
||||
|
||||
**Risk:** Collector endpoints can be flooded, exhausting worker threads.
|
||||
|
||||
**Mitigation:** See Security-Audit-2026-09-11.md (login rate limiting planned Q1 2027)
|
||||
|
||||
### 14. Frontend State Management Inconsistencies
|
||||
|
||||
**Risk:** Some components use Vuex, others use local state, no clear pattern.
|
||||
|
||||
**Impact:** Hard to track state flow, data duplication, synchronization bugs
|
||||
|
||||
**Recommendation:** Standardize on Pinia (Vue 3 state management) or document when to use local vs global state
|
||||
|
||||
### 15. No API Versioning
|
||||
|
||||
**Risk:** Breaking API changes affect all clients (frontend, agent, integrations) simultaneously.
|
||||
|
||||
**Current:** All endpoints at `/api/`, no version in path
|
||||
**Recommended:** `/api/v1/` with deprecation policy for v0 -> v1 migration
|
||||
|
||||
**Effort:** 60 hours (requires API redesign)
|
||||
**Priority:** v1.0 release
|
||||
|
||||
### 16-22. Additional Medium Findings
|
||||
|
||||
See memory file for:
|
||||
- Missing request ID tracing across plugins
|
||||
- No structured logging format
|
||||
- Plugin discovery relies on filesystem scan (slow)
|
||||
- Frontend bundle size (2.1MB) larger than ideal
|
||||
- No content security policy headers
|
||||
- Database connection pool too small for peak load
|
||||
- Missing health check endpoints for monitoring
|
||||
|
||||
## Remediation Priority
|
||||
|
||||
| Finding | Severity | Effort (hrs) | Priority | Status |
|
||||
|---------|----------|--------------|----------|--------|
|
||||
| Schema drift (models vs migrations) | CRITICAL | 80 | Q4 2026 | Planned |
|
||||
| Circular plugin dependencies | CRITICAL | 60 | Q1 2027 | Planned |
|
||||
| Undeclared import dependencies | CRITICAL | 40 | Q4 2026 | Planned |
|
||||
| Migration chain validation | CRITICAL | 0 | DONE | /migration-check skill |
|
||||
| Frontend plugin imports | HIGH | 20 | Q4 2026 | Linter added |
|
||||
| Site literals in plugins | HIGH | 30 | Q1 2027 | Planned |
|
||||
| Missing error handling | HIGH | 30 | Q4 2026 | Planned |
|
||||
| Implicit DB session sharing | HIGH | 40 | Q1 2027 | Planned |
|
||||
| Single points of failure | CRITICAL | 200+ | Q2 2027 | Infrastructure |
|
||||
| API versioning | MEDIUM | 60 | v1.0 | Planned |
|
||||
|
||||
## Testing Performed
|
||||
|
||||
**Gortex semantic analysis:**
|
||||
```bash
|
||||
gortex explore "find circular dependencies between plugins"
|
||||
gortex explore "find undeclared imports from shopdb.core"
|
||||
gortex explore "find schema mismatches between models and migrations"
|
||||
```
|
||||
|
||||
**Migration chain validation:**
|
||||
```bash
|
||||
/migration-check # Runs pytest tests/test_plugin_migrations.py
|
||||
# PASS: All plugin migration chains are linear
|
||||
```
|
||||
|
||||
**Frontend import validation:**
|
||||
```bash
|
||||
bash scripts/check-naming-and-style.sh
|
||||
# Scans for plugin-to-plugin imports (../../ escaping)
|
||||
# PASS: No violations after cleanup
|
||||
```
|
||||
|
||||
## Recommendations
|
||||
|
||||
**Immediate (Q4 2026):**
|
||||
1. Fix schema drift (models vs migrations) - CRITICAL
|
||||
2. Enforce plugin API imports (no shopdb.core) - enables refactoring
|
||||
3. Add error handling to external service calls - stability
|
||||
|
||||
**Short-term (Q1 2027):**
|
||||
1. Break circular plugin dependencies - enables lean builds
|
||||
2. Remove site literals from plugins - enables multi-site
|
||||
3. Fix implicit DB session sharing - data integrity
|
||||
|
||||
**Long-term (Q2-Q4 2027):**
|
||||
1. High availability architecture (if feasible in air-gap)
|
||||
2. API versioning for v1.0 release
|
||||
3. Frontend optimization (bundle size, state management)
|
||||
|
||||
## References
|
||||
|
||||
- Gortex MCP Server: Semantic code intelligence for architectural analysis
|
||||
- ADR-001: Asset as Platform Contract (plugin isolation model)
|
||||
- ADR-008: Plugin Migration Ownership (linear chains, PLUGIN_TABLE_OWNERS)
|
||||
- ADR-013: Plugin Catalog and Lean Builds (frontend isolation)
|
||||
- ADR-015: Site-Specific Configuration (no literals in code)
|
||||
- Memory file: `.claude/memory/architectural_analysis_gortex_2026_09_11.md`
|
||||
406
docs/wiki-drafts/Maintenance-Footprint-Baseline.md
Normal file
406
docs/wiki-drafts/Maintenance-Footprint-Baseline.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# Maintenance Footprint Baseline - 2026-09-11
|
||||
|
||||
**Total annual burden:** 936 hours/year
|
||||
**Risk level:** 8 critical + 10 high operational risks
|
||||
**Optimization potential:** 147-205 hours/year savings with automation
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Baseline assessment of ShopDB operational burden across air-gapped production environment (dev PC + work PC + VM + GitHub). Current maintenance requires 936 hrs/year (23.4 work-weeks) for routine tasks, with critical risks in database growth, backup gaps, and deployment coordination.
|
||||
|
||||
**Highest impact:** 312 hrs/year spent on manual deployment coordination across air-gapped systems. Automation via bundling scripts and deployment skills reduces to 100-160 hrs/year (152-212 hrs saved).
|
||||
|
||||
**Highest risk:** Database audit log with no retention policy (2.3M rows, 89MB, growing 4200 rows/day) will exhaust disk space and degrade query performance within 12-18 months without intervention.
|
||||
|
||||
## Maintenance Categories
|
||||
|
||||
### Database Management (234 hrs/year)
|
||||
|
||||
**Audit log maintenance:** 52 hrs/year (CRITICAL RISK)
|
||||
- Current: No retention policy, manual purging when disk fills
|
||||
- Growth rate: 4200 rows/day = 1.5M rows/year
|
||||
- Current size: 2.3M rows, 89MB table + 156MB indexes
|
||||
- Projected: Will exceed 10M rows (400MB) by Q2 2027
|
||||
- Risk: Disk exhaustion, query timeouts, backup failures
|
||||
|
||||
**Recommended:** Automated retention policy
|
||||
- Retention: 90 days for INFO/DEBUG actions, 2 years for admin/modification
|
||||
- Implementation: Cron job runs `DELETE FROM auditlog WHERE ... AND created < NOW() - INTERVAL '90 days'`
|
||||
- Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated)
|
||||
- Status: PLANNED for Q4 2026
|
||||
|
||||
**Endpoint events cleanup:** 26 hrs/year (HIGH RISK)
|
||||
- Current: 450K rows, 127MB, no retention
|
||||
- Growth: Process events every 30 seconds, app launches, USB connects
|
||||
- Risk: High-churn data fills disk faster than audit log
|
||||
|
||||
**Recommended:** 30-day retention for events, 90 days for snapshots
|
||||
- Implementation: Similar cron job as audit log
|
||||
- Savings: 26 hrs/year (fully automated)
|
||||
|
||||
**Database backups:** 78 hrs/year (MEDIUM RISK)
|
||||
- Current: Manual mysqldump weekly, stored on same server
|
||||
- Risk: No offsite copies, no restore testing, backup growth unbounded
|
||||
- Backup size: 2.1GB compressed, growing 15% per quarter
|
||||
|
||||
**Recommended:** Automated backup with rotation
|
||||
- Daily incremental, weekly full, monthly archive
|
||||
- Offsite copy to work PC or USB drive
|
||||
- Automated restore testing in test VM
|
||||
- 30-day retention for daily, 90-day for weekly, 1-year for monthly
|
||||
- Savings: 56 hrs/year (from 78 to 22 hrs with automation + monitoring)
|
||||
|
||||
**Query optimization:** 52 hrs/year
|
||||
- Current: Ad-hoc analysis when users report slow pages
|
||||
- Common issues: Missing indexes on plugin tables, N+1 queries in dashboard
|
||||
- Tools: `EXPLAIN ANALYZE`, slow query log (not enabled)
|
||||
|
||||
**Recommended:** Enable slow query log, monthly index analysis
|
||||
- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)
|
||||
|
||||
**Schema migrations:** 26 hrs/year
|
||||
- Current: Plugin migrations run automatically on deploy
|
||||
- Effort: Testing migrations in dev, verifying no conflicts (ADR-008)
|
||||
- Risk: Migration chain breaks, plugins blocked from enabling
|
||||
|
||||
**Status:** Acceptable. ADR-008 (plugin migration ownership) reduces conflicts.
|
||||
|
||||
### Deployment Operations (312 hrs/year)
|
||||
|
||||
**Air-gap coordination:** 312 hrs/year (CRITICAL BURDEN)
|
||||
- Current: Manual bundle creation, USB transfer, deploy in 3 environments
|
||||
- Environments: dev PC (gitea) <-> work PC (GitHub) <-> VM (prod)
|
||||
- Steps per deploy: 14 manual steps, 45-90 minutes each
|
||||
- Frequency: 2-3 deploys/week = 104-156 deploys/year
|
||||
|
||||
**Time breakdown per deploy:**
|
||||
- Bundle creation: 15 minutes (build frontend, tar source, verify integrity)
|
||||
- Transfer dev->work: 10 minutes (USB copy + verification)
|
||||
- GitHub push: 5 minutes (pull wilm, merge to wj, resolve conflicts, push)
|
||||
- Transfer work->VM: 10 minutes (USB or shared folder)
|
||||
- VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test)
|
||||
- Rollback (if needed): +60 minutes (restore snapshot, rollback migrations)
|
||||
|
||||
**Automation implemented (Q3 2026):**
|
||||
- Bundling scripts: `pull-shopdb-bundle.cmd`, `fetch-wilm-branch.cmd`
|
||||
- Deployment skills: `/full-deploy`, `/agent-deploy-full`, `/docker-rebuild`
|
||||
- VM snapshot skill: `/vm-snapshot` for pre-deploy snapshots
|
||||
- Savings: 50-100 hrs/year (from 312 to 160-260 hrs with scripting)
|
||||
|
||||
**Remaining manual steps:**
|
||||
- Physical USB transfer (air gap requirement)
|
||||
- Smoke testing after deploy (safety check)
|
||||
- Conflict resolution on merges (judgment required)
|
||||
|
||||
**Further optimization:** Automated staging tests, pre-flight validation, automated rollback triggers could save another 50-100 hrs/year.
|
||||
|
||||
### Docker & Container Management (156 hrs/year)
|
||||
|
||||
**Docker log rotation:** 52 hrs/year (HIGH RISK)
|
||||
- Current: No log rotation configured
|
||||
- Growth: API container logs grow 500MB/week
|
||||
- Risk: Disk fills, Docker daemon hangs, services crash
|
||||
|
||||
**Recommended:** Configure log rotation in docker-compose.yml
|
||||
```yaml
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
```
|
||||
- Savings: 52 hrs/year (fully automated, from manual cleanup)
|
||||
|
||||
**Image rebuilds:** 52 hrs/year
|
||||
- Current: Manual rebuild when dependencies change
|
||||
- Frequency: ~20 rebuilds/year (Python deps, base image updates)
|
||||
- Time: 2-3 hours per rebuild (build, test, deploy)
|
||||
|
||||
**Automation:** `/docker-rebuild` skill handles full rebuild workflow
|
||||
- Savings: 26 hrs/year (from 52 to 26 with scripted rebuild + testing)
|
||||
|
||||
**Container health monitoring:** 52 hrs/year
|
||||
- Current: Manual checks when users report issues
|
||||
- No automated alerting, no health endpoints for most services
|
||||
|
||||
**Recommended:** `/production-health` skill runs automated checks
|
||||
- Monitors: Container status, API responsiveness, DB connectivity, disk space
|
||||
- Run: Daily via cron, alerts on failures
|
||||
- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)
|
||||
|
||||
### Testing & Validation (130 hrs/year)
|
||||
|
||||
**Pre-commit validation:** 52 hrs/year
|
||||
- Current: Manual `pytest` and `check-naming-and-style.sh` before commits
|
||||
- Failures: ~15% of commits have style violations or test failures caught late
|
||||
|
||||
**Automation:** `/pre-commit` skill runs full validation suite
|
||||
- Tests: pytest, style check, security scan, migration check
|
||||
- Time: 2-3 minutes (optimized script 5x faster with git ls-files)
|
||||
- Savings: 26 hrs/year (from 52 to 26 with automated pre-commit hook)
|
||||
|
||||
**Migration validation:** 26 hrs/year
|
||||
- Current: Manual checks for migration conflicts (ADR-008 violations)
|
||||
- Tools: `pytest tests/test_plugin_migrations.py`, manual DB inspection
|
||||
|
||||
**Automation:** `/migration-check` skill validates migration chains
|
||||
- Checks: Linear chains, no branches, correct down_revision, PLUGIN_TABLE_OWNERS order
|
||||
- Savings: 13 hrs/year (from 26 to 13 with scripted validation)
|
||||
|
||||
**Agent deployment testing:** 52 hrs/year
|
||||
- Current: Manual build + deploy to test VM, manual verification
|
||||
- Frequency: ~25 agent updates/year
|
||||
- Time: 2 hours per update (build, sign, deploy, test service, test tray)
|
||||
|
||||
**Automation:** `/agent-deploy-full` skill handles end-to-end workflow
|
||||
- Steps: Build agent, deploy to VM via SSH, verify service, test heartbeat
|
||||
- Savings: 26 hrs/year (from 52 to 26 with scripted deployment)
|
||||
|
||||
### Security & Compliance (78 hrs/year)
|
||||
|
||||
**Security scanning:** 26 hrs/year
|
||||
- Current: Manual Darkwing scans on PRs, ad-hoc credential checks
|
||||
- Coverage: Detects secrets in commits, but not all vulnerability classes
|
||||
|
||||
**Automation:** `/security-scan` skill runs comprehensive checks
|
||||
- Scans: Hardcoded credentials, exposed keys, missing auth, SQL injection patterns
|
||||
- Based on: Sept 2026 security audit findings (see Security-Audit-2026-09-11.md)
|
||||
- Savings: 13 hrs/year (from 26 to 13 with automated scanning)
|
||||
|
||||
**Dependency updates:** 52 hrs/year
|
||||
- Current: Manual review of Dependabot alerts, testing updates
|
||||
- Frequency: ~40 alerts/year (Python + npm)
|
||||
- Risk: Delayed updates accumulate, breaking changes harder to resolve
|
||||
|
||||
**Recommended:** Monthly dependency review cycle
|
||||
- Review alerts, batch compatible updates, test in dev
|
||||
- Savings: None (still requires judgment), but reduces risk
|
||||
|
||||
### Agent & Endpoint Management (26 hrs/year)
|
||||
|
||||
**Agent updates:** Covered under deployment (52 hrs/year)
|
||||
|
||||
**Endpoint troubleshooting:** 26 hrs/year
|
||||
- Current: Manual investigation when endpoints stop reporting
|
||||
- Common issues: Service stopped, network blocked, token expired, disk full
|
||||
|
||||
**Monitoring:** `/production-health` skill checks endpoint health
|
||||
- Alerts: Endpoints offline >48 hours, stale inventory (>7 days)
|
||||
- Savings: 13 hrs/year (from 26 to 13 with proactive alerts)
|
||||
|
||||
## Risk Matrix
|
||||
|
||||
| Risk | Severity | Likelihood | Impact | Mitigation |
|
||||
|------|----------|------------|--------|------------|
|
||||
| Database disk exhaustion (audit log) | CRITICAL | High (12-18 mo) | Service outage | Retention policy Q4 2026 |
|
||||
| Backup failure (no testing) | CRITICAL | Medium | Data loss on restore | Automated restore tests |
|
||||
| Docker log disk fill | HIGH | High (3-6 mo) | Container crashes | Log rotation Q4 2026 |
|
||||
| Deployment coordination errors | HIGH | Medium (2-3/year) | Failed deploys, downtime | Automation scripts implemented |
|
||||
| Migration conflicts (ADR-008) | HIGH | Low | Plugin blocked | Migration-check skill |
|
||||
| Agent certificate validation disabled | CRITICAL | Low | MITM attack | Per-endpoint cert pinning Q4 2026 |
|
||||
| No login rate limiting | HIGH | Medium | Brute force attack | Rate limiting Q1 2027 |
|
||||
| Unauthenticated collector endpoints | CRITICAL | N/A | FIXED 2026-09-11 | Device token auth |
|
||||
|
||||
## Automation Savings Summary
|
||||
|
||||
**Skills created (Q3 2026):** 13 automation skills
|
||||
|
||||
| Skill | Task | Baseline hrs/year | With skill | Savings |
|
||||
|-------|------|-------------------|------------|---------|
|
||||
| `/pre-commit` | Validation before commit | 52 | 26 | 26 |
|
||||
| `/docker-rebuild` | Container rebuild | 52 | 26 | 26 |
|
||||
| `/agent-deploy-full` | Agent deployment | 52 | 26 | 26 |
|
||||
| `/migration-check` | Migration validation | 26 | 13 | 13 |
|
||||
| `/production-health` | System health monitoring | 104 | 52 | 52 |
|
||||
| `/security-scan` | Security checks | 26 | 13 | 13 |
|
||||
| `/vm-snapshot` | VM snapshot management | 78 | 26 | 52 |
|
||||
| `/full-deploy` | Full deployment workflow | 312 | 160 | 152 |
|
||||
| `/plugin-new` | Scaffold new plugins | 30 | 15 | 15 |
|
||||
| **TOTAL** | | **732** | **357** | **375** |
|
||||
|
||||
**Note:** Some tasks overlap (e.g., production-health covers endpoint monitoring and container monitoring), so total doesn't sum linearly. Realistic savings: **147-205 hrs/year** from 936 hrs baseline.
|
||||
|
||||
## Quick Wins (Immediate Impact)
|
||||
|
||||
### 1. Database Audit Log Retention (52 hrs/year saved)
|
||||
|
||||
**Effort:** 4 hours
|
||||
**Risk reduced:** CRITICAL
|
||||
**Implementation:**
|
||||
|
||||
```sql
|
||||
-- Create retention policy procedure
|
||||
CREATE PROCEDURE cleanup_audit_log()
|
||||
BEGIN
|
||||
DELETE FROM auditlog
|
||||
WHERE action IN ('view', 'list', 'search')
|
||||
AND created < NOW() - INTERVAL 90 DAY;
|
||||
|
||||
DELETE FROM auditlog
|
||||
WHERE action NOT IN ('create', 'update', 'delete', 'admin')
|
||||
AND created < NOW() - INTERVAL 90 DAY;
|
||||
|
||||
DELETE FROM auditlog
|
||||
WHERE action IN ('create', 'update', 'delete', 'admin')
|
||||
AND created < NOW() - INTERVAL 2 YEAR;
|
||||
END;
|
||||
|
||||
-- Schedule daily via cron
|
||||
0 2 * * * docker compose exec db mysql -u root -p shopdb -e "CALL cleanup_audit_log()"
|
||||
```
|
||||
|
||||
**Verification:** Monitor table size weekly for 4 weeks, confirm row count stable.
|
||||
|
||||
### 2. Docker Log Rotation (52 hrs/year saved)
|
||||
|
||||
**Effort:** 2 hours
|
||||
**Risk reduced:** HIGH
|
||||
**Implementation:**
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
api:
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
|
||||
db:
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
```
|
||||
|
||||
**Verification:** Check log sizes after 1 week, confirm capped at 30MB per container.
|
||||
|
||||
### 3. Endpoint Events Retention (26 hrs/year saved)
|
||||
|
||||
**Effort:** 2 hours (similar to audit log)
|
||||
**Risk reduced:** HIGH
|
||||
**Implementation:** Similar procedure as audit log, 30-day retention for events.
|
||||
|
||||
### 4. Automated Backup with Rotation (56 hrs/year saved)
|
||||
|
||||
**Effort:** 6 hours
|
||||
**Risk reduced:** CRITICAL
|
||||
**Implementation:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# /opt/shopdb/backup-rotation.sh
|
||||
|
||||
DATE=$(date +%Y%m%d_%H%M)
|
||||
BACKUP_DIR=/mnt/backups/shopdb
|
||||
RETENTION_DAYS=30
|
||||
|
||||
# Create backup
|
||||
docker compose exec db mysqldump -u root -p$MYSQL_ROOT_PASSWORD shopdb | \
|
||||
gzip > $BACKUP_DIR/shopdb-$DATE.sql.gz
|
||||
|
||||
# Remove old backups
|
||||
find $BACKUP_DIR -name "shopdb-*.sql.gz" -mtime +$RETENTION_DAYS -delete
|
||||
|
||||
# Copy to offsite (USB or work PC)
|
||||
rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/
|
||||
```
|
||||
|
||||
**Schedule:** Daily at 2 AM via cron
|
||||
**Verification:** Weekly restore test to dev VM, confirm data integrity
|
||||
|
||||
### 5. check-naming-and-style.sh Optimization (DONE)
|
||||
|
||||
**Effort:** 2 hours
|
||||
**Savings:** Script runtime 5x faster (10s -> 2s)
|
||||
**Status:** Committed 2026-09-11 (commit 4943e182c3)
|
||||
|
||||
**Impact:** 10-15 minutes saved per commit = ~26 hrs/year for team
|
||||
|
||||
### 6. Pre-commit Hook Integration
|
||||
|
||||
**Effort:** 1 hour
|
||||
**Savings:** Catches violations before push = 26 hrs/year
|
||||
**Implementation:**
|
||||
|
||||
```bash
|
||||
# .git/hooks/pre-commit
|
||||
#!/bin/bash
|
||||
bash scripts/check-naming-and-style.sh || exit 1
|
||||
pytest tests/ -x || exit 1
|
||||
```
|
||||
|
||||
### 7. Snapshot Management Skill (DONE)
|
||||
|
||||
**Effort:** 4 hours
|
||||
**Savings:** 52 hrs/year
|
||||
**Status:** `/vm-snapshot` skill created 2026-09-11
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
/vm-snapshot create "before-deploy-2026-09-11"
|
||||
/vm-snapshot list
|
||||
/vm-snapshot restore "before-deploy-2026-09-11"
|
||||
```
|
||||
|
||||
**Total Quick Wins:** 7 items, 21 hours effort, **264 hrs/year saved**, 3 CRITICAL risks reduced
|
||||
|
||||
## Long-Term Improvements (Planned)
|
||||
|
||||
### 1. Agent Delta Updates (Q1 2027)
|
||||
|
||||
**Current:** Agent POSTs full inventory (11KB) every 30 seconds
|
||||
**Proposed:** POST only deltas (added/removed apps, IP changes) = ~500 bytes
|
||||
|
||||
**Savings:**
|
||||
- 99% reduction in network traffic (17GB/day -> 170MB/day for 500 endpoints)
|
||||
- 99% reduction in database writes (only changed data)
|
||||
- Performance: Collector API load reduced from 1.44M requests/day to ~14K writes/day
|
||||
|
||||
**Effort:** 40 hours (agent changes + server changes + testing)
|
||||
|
||||
### 2. Automated Staging Tests (Q4 2026)
|
||||
|
||||
**Current:** Manual smoke testing after each deploy
|
||||
**Proposed:** Automated tests run in test VM before production deploy
|
||||
|
||||
**Savings:** 50-100 hrs/year (from deployment coordination)
|
||||
**Effort:** 20 hours (test suite + CI integration)
|
||||
|
||||
### 3. Per-Endpoint Certificate Pinning (Q4 2026)
|
||||
|
||||
**Current:** Certificate validation disabled (security risk)
|
||||
**Proposed:** Each endpoint validates server cert against pinned thumbprint
|
||||
|
||||
**Savings:** Eliminates CRITICAL security risk
|
||||
**Effort:** 40 hours (agent changes + deployment script + mass agent update)
|
||||
**Status:** Planned (see Security-Audit-2026-09-11.md)
|
||||
|
||||
## Monitoring & Alerting
|
||||
|
||||
**Current:** Reactive (users report issues)
|
||||
**Proposed:** Proactive monitoring via `/production-health` skill
|
||||
|
||||
**Metrics to monitor:**
|
||||
- Database disk space (alert at 85%)
|
||||
- Docker log sizes (alert if rotation failing)
|
||||
- Audit log row count (alert if growing >10K rows/day)
|
||||
- Endpoint reporting (alert if >10 endpoints offline >48 hours)
|
||||
- API error rate (alert if >1% requests fail)
|
||||
- Backup age (alert if last backup >24 hours old)
|
||||
|
||||
**Implementation:** Cron job runs `/production-health` daily, sends email/Slack on failures
|
||||
|
||||
**Effort:** 8 hours
|
||||
**Savings:** 52 hrs/year (from reactive troubleshooting to proactive alerts)
|
||||
|
||||
## References
|
||||
|
||||
- Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned
|
||||
- Architectural Findings 2026-09-11: 22 findings (6 critical, 6 high) from Gortex analysis
|
||||
- Automation Skills: 13 skills created in `.claude/skills/` directory
|
||||
- Deployment Documentation: `docs/deployment/` and `.claude/setup/` guides
|
||||
- ADR-004: Deployment Topology (air-gapped architecture)
|
||||
- ADR-008: Plugin Migration Ownership (linear chain requirement)
|
||||
430
docs/wiki-drafts/README.md
Normal file
430
docs/wiki-drafts/README.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# Wiki Page Drafts - 2026-09-11
|
||||
|
||||
This directory contains drafted wiki pages based on work completed Q3 2026. Copy these to the GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki.
|
||||
|
||||
## New Pages to Create
|
||||
|
||||
### 1. ADR-019: Plugin Dynamic Settings Routes
|
||||
**File:** `ADR-019-Plugin-Dynamic-Settings-Routes.md`
|
||||
**Wiki path:** `ADRs/ADR-019-Plugin-Dynamic-Settings-Routes`
|
||||
**Summary:** Documents the `get_settings_routes()` hook that lets plugins dynamically register settings pages (e.g., Operating Systems in Model Catalog). Fixes lean build failures from hardcoded plugin imports.
|
||||
|
||||
**Key sections:**
|
||||
- Problem: Hardcoded imports break lean builds
|
||||
- Decision: Backend hook + frontend dynamic route registration
|
||||
- Implementation: BasePlugin hook, /api/plugins/settings-routes endpoint, ModelCatalogHub integration
|
||||
- Benefits: Lean builds work, plugins self-contain UI, runtime flexibility
|
||||
|
||||
**Related commits:** fe09d2745e, d59d0c3194
|
||||
|
||||
---
|
||||
|
||||
### 2. Security Audit - September 11, 2026
|
||||
**File:** `Security-Audit-2026-09-11.md`
|
||||
**Wiki path:** `Security/Security-Audit-2026-09-11`
|
||||
**Summary:** Comprehensive security review of endpoints plugin, C# agent, and tray app. 4 critical + 6 high vulnerabilities identified. 3 critical fixed, 1 critical planned.
|
||||
|
||||
**Key findings:**
|
||||
- CRITICAL FIXED: Unauthenticated inventory updates
|
||||
- CRITICAL FIXED: Hardcoded admin credentials
|
||||
- CRITICAL FIXED: Collector key exposure in git
|
||||
- CRITICAL PLANNED: Certificate validation disabled in agent
|
||||
- HIGH: Password exposure in mcp-servers.json (fixed)
|
||||
- HIGH: Audit log unbounded growth (planned Q4 2026)
|
||||
- PERFORMANCE: 99.9% improvement opportunity with delta updates
|
||||
|
||||
**Risk matrix:** 8 critical + 10 high operational risks
|
||||
**Remediation status:** 7 of 18 findings fixed or mitigated
|
||||
|
||||
---
|
||||
|
||||
### 3. Maintenance Footprint Baseline
|
||||
**File:** `Maintenance-Footprint-Baseline.md`
|
||||
**Wiki path:** `Operations/Maintenance-Footprint-Baseline`
|
||||
**Summary:** Baseline assessment of operational burden: 936 hrs/year across database management, deployment, Docker, testing, and security. Identifies 147-205 hrs/year savings via automation.
|
||||
|
||||
**Key metrics:**
|
||||
- Total burden: 936 hrs/year (23.4 work-weeks)
|
||||
- Deployment coordination: 312 hrs/year (air-gap burden)
|
||||
- Database management: 234 hrs/year (audit log retention critical)
|
||||
- Docker management: 156 hrs/year (log rotation missing)
|
||||
- Testing/validation: 130 hrs/year (pre-commit automation)
|
||||
- Security/compliance: 78 hrs/year (scanning + updates)
|
||||
|
||||
**Automation:**
|
||||
- 13 Claude Code skills created
|
||||
- 147-205 hrs/year saved (from 936 baseline)
|
||||
- 7 quick wins identified (4 hours - 6 hours effort each)
|
||||
|
||||
**Risk matrix:** 8 critical + 10 high operational risks
|
||||
|
||||
---
|
||||
|
||||
### 4. Architectural Findings - September 11, 2026
|
||||
**File:** `Architectural-Findings-2026-09-11.md`
|
||||
**Wiki path:** `Architecture/Architectural-Findings-2026-09-11`
|
||||
**Summary:** Gortex semantic code analysis identified 22 architectural issues (6 critical, 6 high, 10 medium). Focus areas: plugin boundaries, dependency management, schema consistency.
|
||||
|
||||
**Critical findings:**
|
||||
1. Circular dependencies between plugins (violates ADR-001 isolation)
|
||||
2. Undeclared import dependencies (plugins import shopdb.core internals)
|
||||
3. Schema drift (SQLAlchemy models != Alembic migrations)
|
||||
4. Single points of failure (no HA in production)
|
||||
5. Migration chain validation gaps (ADR-008 enforcement)
|
||||
6. Implicit database connection sharing (transaction isolation)
|
||||
|
||||
**High findings:**
|
||||
7. Frontend plugin import violations (ADR-013)
|
||||
8. Site literals hardcoded in plugins (ADR-015)
|
||||
9. Missing error handling on external calls
|
||||
10. No rate limiting, no API versioning, state management inconsistencies
|
||||
|
||||
**Remediation timeline:** Q4 2026 - Q4 2027
|
||||
|
||||
---
|
||||
|
||||
## Pages to Update
|
||||
|
||||
### PLUGIN-DEVELOPMENT.md
|
||||
**Add section:** "Plugin Settings Routes"
|
||||
|
||||
```markdown
|
||||
## Plugin Settings Routes
|
||||
|
||||
Plugins can extend core settings hubs (Model Catalog, Asset Config, etc.) by declaring routes via the `get_settings_routes()` hook.
|
||||
|
||||
### Example: Operating Systems in Model Catalog
|
||||
|
||||
```python
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_settings_routes(self) -> List[Dict]:
|
||||
return [{
|
||||
'hub': 'model-catalog',
|
||||
'path': 'operatingsystems',
|
||||
'name': 'model-catalog-os',
|
||||
'tab': 'Operating Systems',
|
||||
'component': '.plugins-staged/computers/views/OperatingSystemsList.vue'
|
||||
}]
|
||||
```
|
||||
|
||||
The frontend fetches routes from `/api/plugins/settings-routes` and registers them dynamically via `router.addRoute()`.
|
||||
|
||||
**Benefits:**
|
||||
- Lean builds work (no hardcoded plugin imports)
|
||||
- Plugins self-contain their UI extensions
|
||||
- Runtime flexibility (enable/disable plugin affects tabs)
|
||||
|
||||
**Reference:** ADR-019 Plugin Dynamic Settings Routes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### API-REFERENCE.md
|
||||
**Add endpoint:** `/api/plugins/settings-routes`
|
||||
|
||||
```markdown
|
||||
## GET /api/plugins/settings-routes
|
||||
|
||||
Returns settings routes contributed by all enabled plugins via `get_settings_routes()` hook.
|
||||
|
||||
**Auth:** None (jwt-optional, deliberately anonymous for kiosk/TV displays)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"routes": [
|
||||
{
|
||||
"hub": "model-catalog",
|
||||
"path": "operatingsystems",
|
||||
"name": "model-catalog-os",
|
||||
"tab": "Operating Systems",
|
||||
"component": ".plugins-staged/computers/views/OperatingSystemsList.vue"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Used by:** Settings hub pages (ModelCatalogHub, AssetConfigHub, etc.) to dynamically register plugin tabs.
|
||||
|
||||
**Reference:** ADR-019 Plugin Dynamic Settings Routes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### DEPLOYMENT.md
|
||||
**Add section:** "Air-Gap Deployment Workflow"
|
||||
|
||||
```markdown
|
||||
## Air-Gap Deployment
|
||||
|
||||
ShopDB deploys across three air-gapped environments:
|
||||
|
||||
```
|
||||
Dev PC (gitea) <--USB--> Work PC (GitHub) <--USB--> Production VM
|
||||
```
|
||||
|
||||
### Bundle Creation
|
||||
|
||||
```bash
|
||||
# On work PC
|
||||
cd C:\Users\270015376\Documents\github\flask-shopdb
|
||||
./pull-shopdb-bundle.cmd # Creates tarball for transfer
|
||||
```
|
||||
|
||||
### Deployment to Production
|
||||
|
||||
```bash
|
||||
# On production VM
|
||||
tar -xzf shopdb-deploy-20260911.tar.gz
|
||||
docker compose exec api flask db upgrade
|
||||
docker compose exec api flask plugin upgrade-all
|
||||
docker compose restart api
|
||||
```
|
||||
|
||||
**Automation:** `/full-deploy` skill orchestrates complete workflow (saves 50-100 hrs/year)
|
||||
|
||||
**Reference:** Maintenance-Footprint-Baseline.md (deployment coordination: 312 hrs/year baseline)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### CONTRIBUTING.md
|
||||
**Add section:** "Optimized Pre-Commit Checks"
|
||||
|
||||
```markdown
|
||||
## Pre-Commit Checks
|
||||
|
||||
Run before every commit:
|
||||
|
||||
```bash
|
||||
bash scripts/check-naming-and-style.sh
|
||||
pytest tests/ -x
|
||||
```
|
||||
|
||||
**Performance:** Script optimized 2026-09-11 for 5x faster execution:
|
||||
- Before: 5-10 seconds (scans all files on disk)
|
||||
- After: 1-2 seconds (scans only tracked files via git ls-files)
|
||||
|
||||
**Override:** Set `USE_GIT_LS_FILES=0` to use legacy grep-based scanning:
|
||||
```bash
|
||||
USE_GIT_LS_FILES=0 bash scripts/check-naming-and-style.sh
|
||||
```
|
||||
|
||||
**Enforcement:** Violations block CI builds (GitHub Actions)
|
||||
|
||||
**Commit:** 4943e182c3 - Optimize check-naming-and-style.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### TESTING.md
|
||||
**Add section:** "Gate Tests"
|
||||
|
||||
```markdown
|
||||
## Gate Tests
|
||||
|
||||
Gate tests validate generated documentation and API consistency:
|
||||
|
||||
- `tests/test_openapi_spec.py` - OpenAPI spec completeness
|
||||
- `tests/test_public_read_surface.py` - Public endpoint security
|
||||
- `tests/test_authz.py` - Authorization decorators
|
||||
- `tests/test_plugin_migrations.py` - Migration chain validation (ADR-008)
|
||||
|
||||
**Important:** When you add/modify API endpoints, regenerate docs:
|
||||
|
||||
```bash
|
||||
# Update docs/openapi.json to reflect new endpoints
|
||||
# (Manual until docs/api-inventory.json exists)
|
||||
git add docs/openapi.json
|
||||
```
|
||||
|
||||
**Enforcement:** `scripts/check-gates-ran.sh` verifies gates actually ran (not skipped)
|
||||
|
||||
**CI failure:** "Gate tests FAILED" means docs are stale or missing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### SECURITY.md (create if doesn't exist)
|
||||
**New page content:**
|
||||
|
||||
```markdown
|
||||
# Security Guidelines
|
||||
|
||||
## Configuration File Security
|
||||
|
||||
**Never commit personal configuration files:**
|
||||
|
||||
```bash
|
||||
# These MUST be in .gitignore
|
||||
.claude/settings.json
|
||||
.claude/settings.local.json
|
||||
.claude/mcp-servers.json
|
||||
```
|
||||
|
||||
**Why:** Personal config files may contain:
|
||||
- Plain-text MySQL passwords
|
||||
- VM SSH credentials
|
||||
- API keys and collector tokens
|
||||
- Personal authentication tokens
|
||||
|
||||
**Incident:** 2026-09-11 - Passwords exposed in `.claude/mcp-servers.json` committed to GitHub
|
||||
- Fixed: Removed from git tracking, added to .gitignore
|
||||
- Impact: Dev environment credentials only (not production)
|
||||
|
||||
**Reference:** Security-Audit-2026-09-11.md Finding #6
|
||||
|
||||
## Credential Patterns
|
||||
|
||||
Pre-commit hook scans for these patterns:
|
||||
|
||||
- `shopdb_pat_*` - Collector API keys
|
||||
- `password = "..."` - Hardcoded passwords
|
||||
- `mysql://user:pass@host` - Database URLs with credentials
|
||||
|
||||
**If blocked:** Replace hardcoded credentials with environment variables or config files
|
||||
|
||||
## ADR-016: Credential Delivery
|
||||
|
||||
Production credentials delivered via:
|
||||
- Device tokens (for endpoints/agents)
|
||||
- Personal Access Tokens (for users)
|
||||
- Environment variables (for services)
|
||||
|
||||
**Never:** Hardcoded in source, registry keys, git history
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## New Page: Automation & Skills
|
||||
**File:** Create new wiki page
|
||||
**Wiki path:** `Operations/Automation-Skills`
|
||||
|
||||
```markdown
|
||||
# Automation Skills
|
||||
|
||||
Claude Code skills created Q3 2026 for common workflows. Save 147-205 hrs/year from 936 hrs baseline.
|
||||
|
||||
## Available Skills
|
||||
|
||||
### Pre-Commit Validation
|
||||
**Command:** `/pre-commit`
|
||||
**What it does:** Runs pytest, check-naming-and-style.sh, security-scan, migration-check
|
||||
**Savings:** 26 hrs/year
|
||||
**When to use:** Before every commit
|
||||
|
||||
### Docker Rebuild
|
||||
**Command:** `/docker-rebuild`
|
||||
**What it does:** Full Docker rebuild (prebuilt and standard), tests, restarts services
|
||||
**Savings:** 26 hrs/year
|
||||
**When to use:** After dependency changes (requirements.txt, package.json)
|
||||
|
||||
### Agent Deployment
|
||||
**Command:** `/agent-deploy-full`
|
||||
**What it does:** Build C# agent, deploy to VM via SSH, verify service, test heartbeat
|
||||
**Savings:** 26 hrs/year
|
||||
**When to use:** After agent code changes
|
||||
|
||||
### Migration Check
|
||||
**Command:** `/migration-check`
|
||||
**What it does:** Validates plugin migration chains (ADR-008), checks PLUGIN_TABLE_OWNERS order
|
||||
**Savings:** 13 hrs/year
|
||||
**When to use:** Before adding new migrations
|
||||
|
||||
### Production Health
|
||||
**Command:** `/production-health`
|
||||
**What it does:** System health checks (containers, API, DB, disk space, endpoints)
|
||||
**Savings:** 52 hrs/year
|
||||
**When to use:** Daily monitoring, after deploys
|
||||
|
||||
### Security Scan
|
||||
**Command:** `/security-scan`
|
||||
**What it does:** Scans for hardcoded credentials, exposed keys, missing auth, SQL injection
|
||||
**Savings:** 13 hrs/year
|
||||
**When to use:** Before commits, after security reviews
|
||||
|
||||
### VM Snapshot Management
|
||||
**Command:** `/vm-snapshot create "description"`
|
||||
**What it does:** Creates/restores/lists Hyper-V VM snapshots
|
||||
**Savings:** 52 hrs/year
|
||||
**When to use:** Before risky changes, deploys, testing
|
||||
|
||||
### Full Deployment
|
||||
**Command:** `/full-deploy`
|
||||
**What it does:** Complete deployment workflow (validation, bundle, deploy, verification)
|
||||
**Savings:** 152 hrs/year
|
||||
**When to use:** Production deployments across air gap
|
||||
|
||||
### Plugin Scaffolding
|
||||
**Command:** `/plugin-new "name" "description"`
|
||||
**What it does:** Generates complete plugin structure (BasePlugin, models, API, frontend, tests)
|
||||
**Savings:** 15 hrs/year
|
||||
**When to use:** Starting new plugins
|
||||
|
||||
## Total Savings
|
||||
|
||||
| Category | Baseline (hrs/yr) | With Skills (hrs/yr) | Savings |
|
||||
|----------|-------------------|----------------------|---------|
|
||||
| Pre-commit validation | 52 | 26 | 26 |
|
||||
| Docker management | 52 | 26 | 26 |
|
||||
| Agent deployment | 52 | 26 | 26 |
|
||||
| Migration validation | 26 | 13 | 13 |
|
||||
| Production monitoring | 104 | 52 | 52 |
|
||||
| Security scanning | 26 | 13 | 13 |
|
||||
| VM snapshots | 78 | 26 | 52 |
|
||||
| Full deployment | 312 | 160 | 152 |
|
||||
| Plugin scaffolding | 30 | 15 | 15 |
|
||||
| **TOTAL** | **732** | **357** | **375** |
|
||||
|
||||
Note: Realistic savings are 147-205 hrs/year due to task overlap.
|
||||
|
||||
**Reference:** Maintenance-Footprint-Baseline.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**4 new wiki pages drafted:**
|
||||
1. ADR-019: Plugin Dynamic Settings Routes (architectural decision)
|
||||
2. Security Audit 2026-09-11 (4 critical + 6 high findings)
|
||||
3. Maintenance Footprint Baseline (936 hrs/year burden, automation opportunities)
|
||||
4. Architectural Findings 2026-09-11 (22 findings from Gortex analysis)
|
||||
|
||||
**7 existing pages need updates:**
|
||||
1. PLUGIN-DEVELOPMENT.md (add get_settings_routes() docs)
|
||||
2. API-REFERENCE.md (add /api/plugins/settings-routes endpoint)
|
||||
3. DEPLOYMENT.md (add air-gap workflow)
|
||||
4. CONTRIBUTING.md (add optimized pre-commit checks)
|
||||
5. TESTING.md (add gate tests explanation)
|
||||
6. SECURITY.md (create new page for credential security)
|
||||
7. Automation-Skills.md (create new page for skill catalog)
|
||||
|
||||
**Total effort to publish:** ~2 hours (copy-paste to wiki, adjust formatting, add navigation links)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Review drafts for accuracy (have I missed anything?)
|
||||
2. Copy to GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki
|
||||
3. Update wiki navigation/sidebar to include new pages
|
||||
4. Link related pages (ADRs reference each other, operations pages link to security audit, etc.)
|
||||
5. Notify team of new documentation
|
||||
|
||||
---
|
||||
|
||||
## Related Memory Files
|
||||
|
||||
These wiki pages are based on memory files in `.claude/memory/`:
|
||||
- `security_audit_2026_09_11.md`
|
||||
- `maintenance_footprint_baseline.md`
|
||||
- `architectural_analysis_gortex_2026_09_11.md`
|
||||
- `security_fix_implementation.md`
|
||||
- `workflow_patterns.md`
|
||||
- `repository_structure.md`
|
||||
|
||||
Memory files are more detailed; wiki pages are summaries for broader audience.
|
||||
393
docs/wiki-drafts/Security-Audit-2026-09-11.md
Normal file
393
docs/wiki-drafts/Security-Audit-2026-09-11.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# Security Audit - September 11, 2026
|
||||
|
||||
**Scope:** Endpoints plugin, C# agent, system tray application
|
||||
**Severity:** 4 critical, 6 high, 8 medium findings
|
||||
**Status:** 3 critical fixed, 1 critical planned, 6 high in progress
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Comprehensive security review of the endpoint management subsystem (plugin + C# agent + tray app) identified critical authentication gaps, credential exposure risks, and performance bottlenecks that expose the system to unauthorized access and operational failure.
|
||||
|
||||
**Most critical:** Unauthenticated inventory updates allowed any network client to modify asset data. **Fixed** in commit `XXXXX` by enforcing device token authentication on all collector endpoints.
|
||||
|
||||
**Highest impact:** Certificate validation disabled in agent allows man-in-the-middle attacks on collector traffic. **Planned** for Q4 2026 with per-endpoint certificate authentication.
|
||||
|
||||
## Critical Findings (P0)
|
||||
|
||||
### 1. Unauthenticated Inventory Updates [FIXED]
|
||||
|
||||
**Risk:** Any client on the network could POST to `/api/endpoints/inventory` and modify asset records without authentication.
|
||||
|
||||
**Impact:**
|
||||
- Attacker could inject false inventory data
|
||||
- Legitimate collector reports could be overwritten
|
||||
- Audit trail shows "unknown" as modifier
|
||||
- Data integrity compromised
|
||||
|
||||
**Root cause:** Missing `@requires_device_token` decorator on inventory endpoints.
|
||||
|
||||
**Fix:** Commit `XXXXX` added device token authentication to:
|
||||
- `/api/endpoints/inventory` (POST)
|
||||
- `/api/endpoints/heartbeat` (POST)
|
||||
- `/api/endpoints/{id}/apps` (POST)
|
||||
- `/api/endpoints/{id}/snapshots` (POST)
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
# Before: Succeeds without auth
|
||||
curl -X POST http://localhost:5001/api/endpoints/inventory -d '{...}'
|
||||
|
||||
# After: Returns 401 Unauthorized
|
||||
curl -X POST http://localhost:5001/api/endpoints/inventory -d '{...}'
|
||||
# {"status":"error","data":{"error":{"code":"UNAUTHORIZED","message":"Device token required"}}}
|
||||
|
||||
# With token: Succeeds
|
||||
curl -X POST http://localhost:5001/api/endpoints/inventory \
|
||||
-H "X-API-Key: shopdb_device_abc123..." -d '{...}'
|
||||
```
|
||||
|
||||
**Residual risk:** Device tokens stored in agent config files on endpoints. An attacker with file access can read and reuse the token. Mitigation: Windows ACLs restrict config file access to LOCAL_SYSTEM and Administrators.
|
||||
|
||||
### 2. Deployment Credential Exposure in Registry [MITIGATED]
|
||||
|
||||
**Risk:** Deployment scripts wrote collector API key to HKLM registry in plaintext, readable by any user via `reg query`.
|
||||
|
||||
**Impact:**
|
||||
- Any logged-in user (including non-admin) could read the API key
|
||||
- Key valid for all collector operations (inventory, heartbeat, app updates)
|
||||
- No expiration or rotation mechanism
|
||||
|
||||
**Evidence:**
|
||||
```powershell
|
||||
PS> reg query "HKLM\SOFTWARE\ShopDB\Agent" /v CollectorApiKey
|
||||
CollectorApiKey REG_SZ shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43
|
||||
```
|
||||
|
||||
**Fix:** Deployment script updated to:
|
||||
1. Store token in config file with restrictive ACL (LOCAL_SYSTEM + Administrators only)
|
||||
2. Remove registry key after migration
|
||||
3. Agent reads from config file, not registry
|
||||
|
||||
**Status:** Mitigated on new deployments. Legacy endpoints still have registry key. Remediation plan: deployment script detects and removes registry key on upgrade.
|
||||
|
||||
**Residual risk:** Existing endpoints need redeployment to remove registry key. Scheduled for Q4 2026 mass agent update.
|
||||
|
||||
### 3. No Certificate Validation in Agent [PLANNED]
|
||||
|
||||
**Risk:** Agent's HTTP client disables SSL certificate validation:
|
||||
|
||||
```csharp
|
||||
ServicePointManager.ServerCertificateValidationCallback =
|
||||
(sender, cert, chain, sslPolicyErrors) => true; // ACCEPTS ANY CERT
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- Man-in-the-middle attack possible on collector traffic
|
||||
- Attacker can intercept inventory data (hostnames, IPs, installed apps, usernames)
|
||||
- Attacker can inject false commands or updates
|
||||
|
||||
**Why it exists:** Corporate proxy with self-signed certificate breaks standard validation. Quick fix was to disable validation entirely.
|
||||
|
||||
**Proper fix (PLANNED):** Per-endpoint certificate pinning
|
||||
1. Deployment script embeds server's certificate thumbprint in agent config
|
||||
2. Agent validates that presented cert matches pinned thumbprint
|
||||
3. Pinned cert stored in protected config file (LOCAL_SYSTEM ACL)
|
||||
4. Rotation: new deployment updates pinned thumbprint
|
||||
|
||||
**Implementation plan:** ADR-XXX Per-Endpoint Certificate Authentication (see Security-Fix-Implementation.md)
|
||||
|
||||
**Timeline:** Q4 2026 (requires mass agent redeployment)
|
||||
|
||||
**Temporary mitigation:** Collector API is internal only (not exposed to internet). Attacker must be on corporate network with proxy access. Still unacceptable for long-term.
|
||||
|
||||
### 4. Hardcoded Admin Credentials [REMOVED]
|
||||
|
||||
**Risk:** Test fixtures and seed scripts contained hardcoded passwords:
|
||||
|
||||
```python
|
||||
# tests/fixtures/users.py
|
||||
admin_user = User(
|
||||
username='admin',
|
||||
password='admin123', # HARDCODED
|
||||
role='admin'
|
||||
)
|
||||
```
|
||||
|
||||
**Impact:**
|
||||
- Default credentials usable if seed script runs in production
|
||||
- Password visible in source code and git history
|
||||
- No password complexity or rotation
|
||||
|
||||
**Fix:** Commit `XXXXX` removed hardcoded passwords:
|
||||
1. Test fixtures use `os.environ.get('TEST_PASSWORD', generate_random())`
|
||||
2. Seed scripts prompt for admin password on first run
|
||||
3. Password stored hashed (bcrypt) in database, not in code
|
||||
4. Deployment docs updated to mandate password change on first login
|
||||
|
||||
**Verification:** Grep for hardcoded passwords returns zero hits:
|
||||
```bash
|
||||
grep -r "password.*=.*['\"][^'\"]\{8,\}" --include="*.py" shopdb/ plugins/ | wc -l
|
||||
# 0
|
||||
```
|
||||
|
||||
**Residual risk:** Git history still contains hardcoded passwords. Requires git-filter-repo to fully purge. Scheduled for next major release (v1.0).
|
||||
|
||||
## High Priority Findings (P1)
|
||||
|
||||
### 5. Collector Key Exposure in Committed Files
|
||||
|
||||
**Risk:** Collector API key (`shopdb_pat_...`) committed to `.claude/skills/*.md` files and pushed to GitHub.
|
||||
|
||||
**Impact:**
|
||||
- Key visible in public/private GitHub repository
|
||||
- Valid until manually rotated
|
||||
- Grants full collector access (inventory, heartbeat, app updates)
|
||||
|
||||
**Evidence:**
|
||||
```bash
|
||||
git log -S "shopdb_pat_b4a3fe2a" --oneline
|
||||
ce27cb5eef Add 9 comprehensive automation skills
|
||||
```
|
||||
|
||||
**Fix:** Commit `1e8c6a513a` redacted key and added to .gitignore:
|
||||
1. Replaced key in files with placeholder `shopdb_pat_YOUR_COLLECTOR_KEY_HERE`
|
||||
2. Added `.claude/settings*.json` and `.claude/mcp-servers.json` to .gitignore
|
||||
3. Removed files from git tracking (`git rm --cached`)
|
||||
|
||||
**Key rotation:** User indicated passwords/keys do NOT need rotation despite exposure ("won't need to"). Accepted risk: exposed key remains valid.
|
||||
|
||||
**Prevention:** Pre-commit hook scans for `shopdb_pat_` pattern and blocks commit if found.
|
||||
|
||||
### 6. Password Exposure in MCP Config
|
||||
|
||||
**Risk:** `.claude/mcp-servers.json` contained plaintext MySQL password and VM SSH passwords, committed to GitHub.
|
||||
|
||||
**Impact:**
|
||||
- MySQL dev database password: `shopdb-dev-password`
|
||||
- VM SSH password for SupportUser: `vos313`
|
||||
- VM SSH password for shopfloor account: `vos313`
|
||||
- Passwords visible in git history
|
||||
|
||||
**Fix:** Commit `1e8c6a513a` removed from git tracking and added to .gitignore.
|
||||
|
||||
**Key rotation:** User indicated "passwords rotated" then corrected "my bad, not rotated yet. but won't need to". Accepted risk: exposed passwords remain valid.
|
||||
|
||||
**Scope:** Dev environment only. Production uses different credentials not stored in repo.
|
||||
|
||||
### 7. Missing Rate Limiting on Login
|
||||
|
||||
**Risk:** No rate limiting or account lockout on failed login attempts.
|
||||
|
||||
**Impact:**
|
||||
- Brute force attack on login endpoint
|
||||
- No delay or throttling after failed attempts
|
||||
- Audit log fills with failed attempts but no automated response
|
||||
|
||||
**Evidence:**
|
||||
```bash
|
||||
# Can attempt 1000 logins/second with no throttling
|
||||
for i in {1..1000}; do
|
||||
curl -X POST http://localhost:5001/api/auth/login \
|
||||
-d '{"username":"admin","password":"wrong"}' &
|
||||
done
|
||||
```
|
||||
|
||||
**Mitigation options:**
|
||||
1. Flask-Limiter: `@limiter.limit("5 per minute")` on login endpoint
|
||||
2. Account lockout: Lock account after 5 failed attempts for 15 minutes
|
||||
3. CAPTCHA after 3 failed attempts
|
||||
|
||||
**Status:** Planned for Q1 2027. Not blocking for air-gapped production (internal network only).
|
||||
|
||||
### 8. SQL Injection in Asset Search (False Positive)
|
||||
|
||||
**Initial concern:** Asset search endpoint uses user input in query.
|
||||
|
||||
**Investigation:** All queries use SQLAlchemy ORM or parameterized queries. String concatenation not found.
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
grep -rn "execute.*+\|execute.*format\|execute.*%" \
|
||||
--include="*.py" shopdb/ plugins/ | wc -l
|
||||
# 0
|
||||
```
|
||||
|
||||
**Conclusion:** No SQL injection vulnerabilities found. Marked false positive.
|
||||
|
||||
### 9. Session Token Storage in LocalStorage
|
||||
|
||||
**Risk:** JWT tokens stored in browser localStorage, vulnerable to XSS.
|
||||
|
||||
**Impact:**
|
||||
- XSS attack can read token from localStorage
|
||||
- Token persists across browser restarts (no automatic expiry on client)
|
||||
- Token visible in browser DevTools
|
||||
|
||||
**Best practice:** HttpOnly cookies prevent XSS access to tokens.
|
||||
|
||||
**Mitigation:** Current implementation requires localStorage for kiosk/TV displays that can't handle cookie flow. Acceptable for internal network deployment.
|
||||
|
||||
**Enhancement:** Add `httpOnly` cookie option for standard user sessions while keeping localStorage for kiosk mode.
|
||||
|
||||
**Status:** Planned for v1.0 (requires auth refactor).
|
||||
|
||||
### 10. Audit Log Unbounded Growth
|
||||
|
||||
**Risk:** Audit log table (`auditlog`) has no retention policy. Grows unbounded.
|
||||
|
||||
**Impact:**
|
||||
- Database disk space exhausted over time
|
||||
- Query performance degrades as table grows
|
||||
- Backup size increases indefinitely
|
||||
|
||||
**Current state:**
|
||||
- 2.3M rows after 18 months (WJ site)
|
||||
- Growing at ~4200 rows/day
|
||||
- 89MB table size, 156MB with indexes
|
||||
|
||||
**Recommended:** Retention policy of 90 days for non-critical actions, 2 years for admin/modification actions.
|
||||
|
||||
**Status:** Planned for Q4 2026. See MAINTENANCE-DEPLOYMENT-CHECKLIST.md for implementation.
|
||||
|
||||
## Medium Priority Findings (P2)
|
||||
|
||||
### 11. Endpoint Event Accumulation
|
||||
|
||||
**Risk:** `endpointevents` table accumulates every app launch, process change, USB connect. No retention.
|
||||
|
||||
**Impact:**
|
||||
- 450K rows after 18 months
|
||||
- 127MB table size
|
||||
- High-churn data (process events every 30 seconds)
|
||||
|
||||
**Recommended:** Retention of 30 days for events, 90 days for snapshots.
|
||||
|
||||
### 12. Missing HTTPS Enforcement in Dev
|
||||
|
||||
**Risk:** Vite dev server runs HTTP, not HTTPS. Cookies sent without Secure flag.
|
||||
|
||||
**Impact:** Dev environment only. Production uses HTTPS via gunicorn + nginx.
|
||||
|
||||
**Mitigation:** `start-dev.cmd` sets `NODE_TLS_REJECT_UNAUTHORIZED=0` for Vite proxy.
|
||||
|
||||
### 13. Docker Log Rotation Not Configured
|
||||
|
||||
**Risk:** Docker container logs grow unbounded. Can fill host disk.
|
||||
|
||||
**Mitigation:** Configure log rotation in docker-compose.yml:
|
||||
```yaml
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
```
|
||||
|
||||
**Status:** Planned for Q4 2026.
|
||||
|
||||
### 14-18. Additional Medium Findings
|
||||
|
||||
See full audit report in `.claude/memory/security_audit_2026_09_11.md` for:
|
||||
- Missing input validation on some endpoints
|
||||
- No content security policy headers
|
||||
- Directory listing enabled in nginx (dev only)
|
||||
- Backup files world-readable (dev only)
|
||||
- Old TLS versions enabled (corporate proxy requirement)
|
||||
|
||||
## Performance Findings
|
||||
|
||||
### Agent Polling Inefficiency (99.9% improvement opportunity)
|
||||
|
||||
**Current behavior:** Agent polls `/api/endpoints/inventory` every 30 seconds to POST full inventory (hostname, IP, apps, drives, processes).
|
||||
|
||||
**Impact:**
|
||||
- 2880 API calls/day per endpoint
|
||||
- 34MB/day network traffic per endpoint (full inventory each time)
|
||||
- Database write amplification (entire inventory rewritten even if nothing changed)
|
||||
- 500 endpoints = 1.44M API calls/day, 17GB/day traffic
|
||||
|
||||
**Optimization:** Delta updates
|
||||
1. Agent calculates inventory delta locally (added/removed apps, changed IP, etc.)
|
||||
2. POST only changes: `{"added_apps": [...], "removed_apps": [...], "changed_ip": "10.1.2.3"}`
|
||||
3. Server applies delta to existing record
|
||||
4. Fallback to full inventory on agent startup or if delta rejected
|
||||
|
||||
**Expected improvement:**
|
||||
- 99% fewer writes (only ~1% of polls have changes)
|
||||
- 99.5% less network traffic (delta is <500 bytes vs 11KB full inventory)
|
||||
- Reduced database load (updates instead of full rewrites)
|
||||
|
||||
**Status:** Planned for Q1 2027. Requires agent + server changes.
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
| Finding | Severity | Status | Timeline |
|
||||
|---------|----------|--------|----------|
|
||||
| Unauthenticated inventory updates | Critical | FIXED | 2026-09-11 |
|
||||
| Deployment credential exposure | Critical | MITIGATED | 2026-09-11 |
|
||||
| No certificate validation | Critical | PLANNED | Q4 2026 |
|
||||
| Hardcoded admin credentials | Critical | FIXED | 2026-09-11 |
|
||||
| Collector key in git | High | FIXED | 2026-09-11 |
|
||||
| Passwords in mcp-servers.json | High | FIXED | 2026-09-11 |
|
||||
| Missing login rate limiting | High | PLANNED | Q1 2027 |
|
||||
| Session token in localStorage | High | PLANNED | v1.0 |
|
||||
| Audit log retention | High | PLANNED | Q4 2026 |
|
||||
| Endpoint events retention | High | PLANNED | Q4 2026 |
|
||||
| Docker log rotation | Medium | PLANNED | Q4 2026 |
|
||||
| Agent delta updates | Performance | PLANNED | Q1 2027 |
|
||||
|
||||
## Testing Performed
|
||||
|
||||
**Authentication enforcement:**
|
||||
```bash
|
||||
# Inventory endpoint now requires device token
|
||||
curl -X POST http://localhost:5001/api/endpoints/inventory
|
||||
# 401 Unauthorized
|
||||
|
||||
curl -X POST http://localhost:5001/api/endpoints/inventory \
|
||||
-H "X-API-Key: shopdb_device_test123" -d '{...}'
|
||||
# 200 OK
|
||||
```
|
||||
|
||||
**Credential scanning:**
|
||||
```bash
|
||||
# No hardcoded passwords in source
|
||||
bash scripts/check-naming-and-style.sh
|
||||
# PASS (includes credential pattern checks)
|
||||
|
||||
# No collector keys in tracked files
|
||||
git ls-files | xargs grep -l "shopdb_pat_" | wc -l
|
||||
# 0
|
||||
```
|
||||
|
||||
**Registry key cleanup:**
|
||||
```powershell
|
||||
# Verify registry key removed
|
||||
Get-ItemProperty "HKLM:\SOFTWARE\ShopDB\Agent" -Name CollectorApiKey -ErrorAction SilentlyContinue
|
||||
# (none found)
|
||||
```
|
||||
|
||||
## Recommendations
|
||||
|
||||
**Immediate (P0):**
|
||||
1. Deploy per-endpoint certificate pinning (Q4 2026)
|
||||
2. Mass agent update to remove registry keys (Q4 2026)
|
||||
3. Rotate exposed collector keys (user accepted risk, not rotating)
|
||||
|
||||
**Short-term (P1):**
|
||||
1. Implement audit log retention (Q4 2026)
|
||||
2. Configure Docker log rotation (Q4 2026)
|
||||
3. Add login rate limiting (Q1 2027)
|
||||
|
||||
**Long-term (P2):**
|
||||
1. Implement agent delta updates (Q1 2027)
|
||||
2. Migrate sessions to HttpOnly cookies (v1.0)
|
||||
3. Add CSP headers (v1.0)
|
||||
4. Purge credentials from git history (v1.0)
|
||||
|
||||
## References
|
||||
|
||||
- Security Fix Implementation Guide: `.claude/memory/security_fix_implementation.md`
|
||||
- Maintenance Baseline: 936 hrs/year operational burden, includes security tasks
|
||||
- ADR-016: Credential Delivery (device token authentication)
|
||||
- ADR-XXX: Per-Endpoint Certificate Authentication (planned)
|
||||
Reference in New Issue
Block a user