shopdb-flask/docs/wiki-drafts/Maintenance-Footprint-Baseline.md

Maintenance Footprint Baseline - 2026-09-11

Total annual burden: 936 hours/year (estimate, see the note below) Risk level: originally stated as 8 critical + 10 high operational risks Optimization potential: 147-205 hours/year savings with automation Last checked against code: 2026-09-18, branch gh-wj at 35d4fcd41

About the numbers (added 2026-09-18)

Every hour figure on this page is an estimate with no recorded measurement behind it. No time log, ticket export or deploy log is cited anywhere, and none was found in the repository. Treat the hours as an ordering of where effort goes, not as data. The same applies to the row counts and growth rates in the Database Management section: no query output is cited and none could be taken from a repository checkout.

What CAN be checked from the repository is which of the recommendations have since been built. Several have. The sections below are annotated accordingly, dated 2026-09-18, and the stale risk-matrix rows are corrected.

What the tree actually contains (2026-09-18)

Thing	Count	How counted
Plugins	18 directories, 17 with a manifest.json	ls -d plugins/*/; applications carries no manifest
Plugin migration revisions	70 across 17 chains	ls plugins/*/migrations/versions/*.py
Largest single chain	endpoints, 24 revisions	one root, one head, no branch
Core migration revisions	58	ls migrations/versions/*.py
Test files	223	find tests -name "test_*.py"
Top-level scripts	29 (.sh/.py/.ps1/.mjs), plus scripts/backup, scripts/dev, scripts/migration, scripts/loadtest, scripts/site_imports	ls scripts/
Automation skills	13	ls <assistant-config>/skills/*.md minus README

The "13 automation skills" claim later on this page is the one headline figure that checks out exactly.

Executive Summary

Baseline assessment of ShopDB operational burden across air-gapped production environment (dev PC + work PC + VM + GitHub). Current maintenance requires 936 hrs/year (23.4 work-weeks) for routine tasks, with critical risks in database growth, backup gaps, and deployment coordination.

Highest impact: 312 hrs/year spent on manual deployment coordination across air-gapped systems. Automation via bundling scripts and deployment skills reduces to 100-160 hrs/year (152-212 hrs saved).

Highest risk: Database audit log with no retention policy (2.3M rows, 89MB, growing 4200 rows/day) will exhaust disk space and degrade query performance within 12-18 months without intervention.

Maintenance Categories

Database Management (234 hrs/year)

Audit log maintenance: 52 hrs/year (CRITICAL RISK)

• Current: No retention policy, manual purging when disk fills
• Growth rate: 4200 rows/day = 1.5M rows/year
• Current size: 2.3M rows, 89MB table + 156MB indexes
• Projected: Will exceed 10M rows (400MB) by Q2 2027
• Risk: Disk exhaustion, query timeouts, backup failures

Recommended: Automated retention policy

• Retention: 90 days for INFO/DEBUG actions, 2 years for admin/modification
• Implementation: Cron job runs DELETE FROM auditlog WHERE ... AND created < NOW() - INTERVAL '90 days'
• Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated)
• Status: PLANNED for Q4 2026

2026-09-18 update: half built, and the half that is built is manual. Retention exists as a service and a CLI, not as a cron job and not as a stored procedure. shopdb/core/services/auditretention.py provides configureddays(), stats() and purge(); flask audit stats (shopdb/cli/__init__.py:1891) reports size against the configured window and flask audit purge (:1911) deletes, with a dry-run preview first and --days to override for one run. The window is the audit_retentiondays setting (shopdb/core/api/settings.py:1049), and its default is '0', which means keep everything forever: nothing is deleted until an operator sets a value. There is no scheduler in the application and no cron entry in the repository, so the purge runs when somebody runs it. The remaining work is a value in Settings

Site & Facility and a host cron line, not code.

Endpoint events cleanup: 26 hrs/year (HIGH RISK)

• Current: 450K rows, 127MB, no retention
• Growth: Process events every 30 seconds, app launches, USB connects
• Risk: High-churn data fills disk faster than audit log

Recommended: 30-day retention for events, 90 days for snapshots

• Implementation: Similar cron job as audit log
• Savings: 26 hrs/year (fully automated)

2026-09-18 update: partly built, per table rather than globally. Operator reports carry a retention window defaulting to 7 days (plugins/endpoints/api/routes.py:3619-3620, endpoints.operatorreport.retentiondays, 0 or less meaning keep forever), and endpoint health transitions are trimmed to a high-water mark on write, keeping every unresolved row regardless of age (plugins/endpoints/services/health.py:326). The row counts and the 127MB figure above were never sourced and are not verifiable from a checkout.

Database backups: 78 hrs/year (MEDIUM RISK)

• Current (as stated 2026-09-11): Manual mysqldump weekly, stored on same server
• Risk: No offsite copies, no restore testing, backup growth unbounded
• Backup size: 2.1GB compressed, growing 15% per quarter (unsourced)

2026-09-18 update: STALE. This is built, and it is what the recommendation below asks for. scripts/backup/ holds backup.sh, restore.sh, status.sh, watch.sh, simulate-restore.sh and test-restore-datarev.sh, plus a Dockerfile.backup and a compose file. Retention is enforced by restic: scripts/backup/backup.sh:134-141 runs restic forget --keep-daily 5 --keep-weekly 4 --keep-monthly 6 --prune. Destinations are configured in Settings > Backups and include local, SMB and AWS S3, so offsite is a setting rather than a manual copy. Each backup set carries db.sql.gz, the instance/ tree and a manifest.json of schema revisions, and restore validates schema compatibility before it runs. A restore-simulation harness is in the tree. Backup credential redaction landed in the same commit as audit retention (scripts/redact_backup_secrets.py).

Recommended: Automated backup with rotation

• Daily incremental, weekly full, monthly archive
• Offsite copy to work PC or USB drive
• Automated restore testing in test VM
• 30-day retention for daily, 90-day for weekly, 1-year for monthly
• Savings: 56 hrs/year (from 78 to 22 hrs with automation + monitoring)

Query optimization: 52 hrs/year

• Current: Ad-hoc analysis when users report slow pages
• Common issues: Missing indexes on plugin tables, N+1 queries in dashboard
• Tools: EXPLAIN ANALYZE, slow query log (not enabled)

Recommended: Enable slow query log, monthly index analysis

• Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

Schema migrations: 26 hrs/year

• Current: Plugin migrations run automatically on deploy
• Effort: Testing migrations in dev, verifying no conflicts (ADR-008)
• Risk: Migration chain breaks, plugins blocked from enabling

Status: Acceptable. ADR-008 (plugin migration ownership) reduces conflicts.

Deployment Operations (312 hrs/year)

Air-gap coordination: 312 hrs/year (CRITICAL BURDEN)

• Current: Manual bundle creation, USB transfer, deploy in 3 environments
• Environments: dev PC (the internal git server) <-> work PC (GitHub) <-> VM (prod)
• Steps per deploy: 14 manual steps, 45-90 minutes each
• Frequency: 2-3 deploys/week = 104-156 deploys/year

Time breakdown per deploy:

• Bundle creation: 15 minutes (build frontend, tar source, verify integrity)
• Transfer dev->work: 10 minutes (USB copy + verification)
• GitHub push: 5 minutes (pull wilm, merge to wj, resolve conflicts, push)
• Transfer work->VM: 10 minutes (USB or shared folder)
• VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test)
• Rollback (if needed): +60 minutes (restore snapshot, rollback migrations)

Automation implemented (Q3 2026):

• Bundling scripts: pull-shopdb-bundle.cmd, fetch-wilm-branch.cmd
• Deployment skills: /full-deploy, /agent-deploy-full, /docker-rebuild
• VM snapshot skill: /vm-snapshot for pre-deploy snapshots
• Savings: 50-100 hrs/year (from 312 to 160-260 hrs with scripting)

Remaining manual steps:

• Physical USB transfer (air gap requirement)
• Smoke testing after deploy (safety check)
• Conflict resolution on merges (judgment required)

Further optimization: Automated staging tests, pre-flight validation, automated rollback triggers could save another 50-100 hrs/year.

Docker & Container Management (156 hrs/year)

Docker log rotation: 52 hrs/year (HIGH RISK)

• Current: No log rotation configured
• Growth: API container logs grow 500MB/week
• Risk: Disk fills, Docker daemon hangs, services crash

Recommended: Configure log rotation in docker-compose.yml

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

• Savings: 52 hrs/year (fully automated, from manual cleanup)

2026-09-18 update: STILL OPEN, and still the cheapest item on this page. grep -c logging docker-compose.yml returns 0: there is no logging: block on any service. Three services do carry restart: unless-stopped (docker-compose.yml:24, :104, :120) and three carry healthcheck: blocks (:38, :112, :219), so the container lifecycle is managed; log size is not. The 500MB/week growth figure was never sourced.

Image rebuilds: 52 hrs/year

• Current: Manual rebuild when dependencies change
• Frequency: ~20 rebuilds/year (Python deps, base image updates)
• Time: 2-3 hours per rebuild (build, test, deploy)

Automation: /docker-rebuild skill handles full rebuild workflow

• Savings: 26 hrs/year (from 52 to 26 with scripted rebuild + testing)

Container health monitoring: 52 hrs/year

• Current: Manual checks when users report issues
• No automated alerting, no health endpoints for most services

Recommended: /production-health skill runs automated checks

• Monitors: Container status, API responsiveness, DB connectivity, disk space
• Run: Daily via cron, alerts on failures
• Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

Testing & Validation (130 hrs/year)

Pre-commit validation: 52 hrs/year

• Current: Manual pytest and check-naming-and-style.sh before commits
• Failures: ~15% of commits have style violations or test failures caught late

Automation: /pre-commit skill runs full validation suite

• Tests: pytest, style check, security scan, migration check
• Time: 2-3 minutes (optimized script 5x faster with git ls-files)
• Savings: 26 hrs/year (from 52 to 26 with automated pre-commit hook)

Migration validation: 26 hrs/year

• Current: Manual checks for migration conflicts (ADR-008 violations)
• Tools: pytest tests/test_plugin_migrations.py, manual DB inspection

Automation: /migration-check skill validates migration chains

• Checks: Linear chains, no branches, correct down_revision, PLUGIN_TABLE_OWNERS order
• Savings: 13 hrs/year (from 26 to 13 with scripted validation)

Agent deployment testing: 52 hrs/year

• Current: Manual build + deploy to test VM, manual verification
• Frequency: ~25 agent updates/year
• Time: 2 hours per update (build, sign, deploy, test service, test tray)

Automation: /agent-deploy-full skill handles end-to-end workflow

• Steps: Build agent, deploy to VM via SSH, verify service, test heartbeat
• Savings: 26 hrs/year (from 52 to 26 with scripted deployment)

Security & Compliance (78 hrs/year)

Security scanning: 26 hrs/year

• Current: Manual Darkwing scans on PRs, ad-hoc credential checks
• Coverage: Detects secrets in commits, but not all vulnerability classes

Automation: /security-scan skill runs comprehensive checks

• Scans: Hardcoded credentials, exposed keys, missing auth, SQL injection patterns
• Based on: Sept 2026 security audit findings (see Security-Audit-2026-09-11.md)
• Savings: 13 hrs/year (from 26 to 13 with automated scanning)

Dependency updates: 52 hrs/year

• Current: Manual review of Dependabot alerts, testing updates
• Frequency: ~40 alerts/year (Python + npm)
• Risk: Delayed updates accumulate, breaking changes harder to resolve

Recommended: Monthly dependency review cycle

• Review alerts, batch compatible updates, test in dev
• Savings: None (still requires judgment), but reduces risk

Agent & Endpoint Management (26 hrs/year)

Agent updates: Covered under deployment (52 hrs/year)

Endpoint troubleshooting: 26 hrs/year

• Current: Manual investigation when endpoints stop reporting
• Common issues: Service stopped, network blocked, token expired, disk full

Monitoring: /production-health skill checks endpoint health

• Alerts: Endpoints offline >48 hours, stale inventory (>7 days)
• Savings: 13 hrs/year (from 26 to 13 with proactive alerts)

Risk Matrix

Risk	Severity	Likelihood	Impact	Mitigation
Database disk exhaustion (audit log)	CRITICAL	High (12-18 mo)	Service outage	Retention policy Q4 2026
Backup failure (no testing)	CRITICAL	Medium	Data loss on restore	Automated restore tests
Docker log disk fill	HIGH	High (3-6 mo)	Container crashes	Log rotation Q4 2026
Deployment coordination errors	HIGH	Medium (2-3/year)	Failed deploys, downtime	Automation scripts implemented
Migration conflicts (ADR-008)	HIGH	Low	Plugin blocked	Migration-check skill
Agent certificate validation disabled	CORRECTED 2026-09-18	n/a	n/a	Built. See note below
No login rate limiting	CORRECTED 2026-09-18	n/a	n/a	Built. See note below
Unauthenticated collector endpoints	CORRECTED 2026-09-18	n/a	n/a	Per-PC endpoint credentials, not device tokens. See note below
Docker log disk fill	HIGH	unmeasured	Container crashes	STILL OPEN, no logging: block in docker-compose.yml

Risk matrix corrections (2026-09-18)

Agent certificate validation. Not disabled. AgentHttpFactory is the only HttpClient construction site in the agent, every channel runs ServerPinSet.Validate, and Validate has no path returning true for an unmatched pin (agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs:291-331). A failed registry re-read returns null and keeps the existing pins rather than silently unpinning (:352-386). The three SYSTEM PowerShell reporters refuse to send at all when ServerPins is set and the check cannot be enabled (plugins/computers/client/Report-AssetToShopDB.ps1:228-247, and the same block in Report-PrintersToShopDB.ps1 and upgrade-shopdbagent.ps1:436-447). The one ServerCertificateValidationCallback = {$true} left in the tree is test-vm-connectivity.ps1:44, a developer connectivity probe.

Login rate limiting. Built, and not "Q1 2027": shopdb/core/api/auth.py:35 _login_ratelimited(), keyed per host at :52, applied at :80, with tests/test_core/test_auth_ratelimit.py over it. Note the counter lives in per-process SimpleCache, so at four gunicorn workers the effective budget is four times the configured maximum.

Collector authentication. The mechanism is a per-PC endpoint credential, not a device token and not ADR-016. An enrolled PC sends X-Endpoint-Credential (plugins/endpoints/api/agentauth.py:1-18); the server stores only a sha256 hash plus a 12-character display prefix (plugins/endpoints/models/credential.py); once a PC holds an active credential the shared key is refused for it (plugins/endpoints/api/agentauth.py:129-134). ADR-016's own mechanism - credentials.fetch, GET /api/collector/credentials, Fernet at rest - is not built: grepping shopdb/ and plugins/ for credentials.fetch, collector/credentials and Fernet returns zero hits, matching ADR-016's own "Implementation status, 2026-08-14". The rollout runbook is plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md.

One boundary the runbook does not state: turning the shared key off does not close the four legacy core collector routes. /pc (shopdb/core/api/collector.py:349), /apps (:387), /heartbeat (:459) and /bulk (:494) are decorated @require_api_key, which at :154-164 checks only COLLECTOR_API_KEY or a collector.ingest-scoped token and never consults agent_shared_key_refused. Anything still holding the env shared key can rewrite logged-in user, last boot time, serial number and installed-app inventory for any hostname. Regenerating the key in the UI (plugins/endpoints/api/routes.py:1864-1884) mints a new managed token and clears the env fallback flag but does not change COLLECTOR_API_KEY, and _env_fallback_disabled(None) returns False (collector.py:60-74), so the legacy routes keep honouring the env key. Rotating COLLECTOR_API_KEY in .env is the step that closes it.

Automation Savings Summary

Skills created (Q3 2026): 13 automation skills

Skill	Task	Baseline hrs/year	With skill	Savings
/pre-commit	Validation before commit	52	26	26
/docker-rebuild	Container rebuild	52	26	26
/agent-deploy-full	Agent deployment	52	26	26
/migration-check	Migration validation	26	13	13
/production-health	System health monitoring	104	52	52
/security-scan	Security checks	26	13	13
/vm-snapshot	VM snapshot management	78	26	52
/full-deploy	Full deployment workflow	312	160	152
/plugin-new	Scaffold new plugins	30	15	15
TOTAL		732	357	375

Note: Some tasks overlap (e.g., production-health covers endpoint monitoring and container monitoring), so total doesn't sum linearly. Realistic savings: 147-205 hrs/year from 936 hrs baseline.

Quick Wins (Immediate Impact)

1. Database Audit Log Retention (52 hrs/year saved)

Effort: 4 hours
Risk reduced: CRITICAL
Implementation:

-- Create retention policy procedure
CREATE PROCEDURE cleanup_audit_log()
BEGIN
  DELETE FROM auditlog 
  WHERE action IN ('view', 'list', 'search') 
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action NOT IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 2 YEAR;
END;

-- Schedule daily via cron
0 2 * * * docker compose exec db mysql -u root -p shopdb -e "CALL cleanup_audit_log()"

Verification: Monitor table size weekly for 4 weeks, confirm row count stable.

2. Docker Log Rotation (52 hrs/year saved)

Effort: 2 hours
Risk reduced: HIGH
Implementation:

# docker-compose.yml
services:
  api:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
  
  db:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Verification: Check log sizes after 1 week, confirm capped at 30MB per container.

3. Endpoint Events Retention (26 hrs/year saved)

Effort: 2 hours (similar to audit log)
Risk reduced: HIGH
Implementation: Similar procedure as audit log, 30-day retention for events.

4. Automated Backup with Rotation (56 hrs/year saved)

Effort: 6 hours
Risk reduced: CRITICAL
Implementation:

#!/bin/bash
# /opt/shopdb/backup-rotation.sh

DATE=$(date +%Y%m%d_%H%M)
BACKUP_DIR=/mnt/backups/shopdb
RETENTION_DAYS=30

# Create backup
docker compose exec db mysqldump -u root -p$MYSQL_ROOT_PASSWORD shopdb | \
  gzip > $BACKUP_DIR/shopdb-$DATE.sql.gz

# Remove old backups
find $BACKUP_DIR -name "shopdb-*.sql.gz" -mtime +$RETENTION_DAYS -delete

# Copy to offsite (USB or work PC)
rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/

Schedule: Daily at 2 AM via cron
Verification: Weekly restore test to dev VM, confirm data integrity

5. check-naming-and-style.sh Optimization (DONE)

Status: In the tree and on by default. USE_GIT_LS_FILES defaults to 1 (scripts/check-naming-and-style.sh:24) and the script takes the git ls-files -z path whenever it is inside a work tree (:26, :159, :183, :218), filtering out migrations/versions/, frontend/dist/, staticdocs/ and .plugins-staged/. Setting USE_GIT_LS_FILES=0 falls back to recursive grep with --exclude-dir for non-git checkouts. The whole script exits 0 on gh-wj at 35d4fcd41.

The 5x / 10s-to-2s claim is unsourced and no timing was recorded for this page. The savings line below it follows from it and is equally unsourced.

Coverage note added 2026-09-18: the root-anchored ^migrations/versions/ filter does not exclude plugins/*/migrations/versions/, so plugin migrations ARE scanned. They are not caught by the column rule, though: that rule greps ^\s+[a-z]+_[a-z_]+\s*=\s*db\.Column (:236), which only matches a model attribute assignment, while migrations write sa.Column('name', ...). The column name is created in the one file the naming gate cannot read.

6. Pre-commit Hook Integration

Effort: 1 hour
Savings: Catches violations before push = 26 hrs/year
Implementation:

# .git/hooks/pre-commit
#!/bin/bash
bash scripts/check-naming-and-style.sh || exit 1
pytest tests/ -x || exit 1

7. Snapshot Management Skill (DONE)

Effort: 4 hours
Savings: 52 hrs/year
Status: /vm-snapshot skill created 2026-09-11

Usage:

/vm-snapshot create "before-deploy-2026-09-11"
/vm-snapshot list
/vm-snapshot restore "before-deploy-2026-09-11"

Total Quick Wins: 7 items, 21 hours effort, 264 hrs/year saved, 3 CRITICAL risks reduced

Long-Term Improvements (Planned)

1. Agent Delta Updates - BUILT (2026-09-18 update)

Original text: "Agent POSTs full inventory (11KB) every 30 seconds. Proposed: POST only deltas." Both halves are now wrong.

The cadence was never 30 seconds. The agent default is 900 seconds with up to 60 seconds of jitter: agent/ShopDbEndpointAgent/appsettings.json ("cadenceseconds": 900, "jitterseconds": 60) and agent/ShopDbEndpointAgent.Core/AgentConfig.cs:14 and :31. The 17GB/day and 1.44M-requests/day figures were derived from the wrong interval and should not be quoted.

The delta is implemented, on the server side. The agent still reports the full installed-software list each cycle, but each entry carries a SHA-256 fingerprint of displayname|displayversion|publisher, and _replace_applications (plugins/endpoints/plugin.py:1122-1140) compares fingerprint sets instead of delete-all-insert-all: adds for new fingerprints, deletes for departed ones, unchanged rows untouched (:1277-1289). The docstring states the effect as roughly 100 database operations per check-in down to about 2. It is guarded: fingerprint format validation, a delete-count check that warns rather than aborting, a memory cap above 5000 apps, a legacy fall-back for agents that send no fingerprints, and collision reconciliation inside a savepoint (:1318, :1387). The unique (assetid, appfingerprint) index was added NULL-filled by migration plugins/endpoints/migrations/versions/0020_*, so index creation cannot fail on a populated production table.

Remaining network-side work, if anyone wants it, is making the agent send only the changed entries rather than the full list. The database win is already taken.

2. Automated Staging Tests (Q4 2026)

Current: Manual smoke testing after each deploy
Proposed: Automated tests run in test VM before production deploy

Savings: 50-100 hrs/year (from deployment coordination)
Effort: 20 hours (test suite + CI integration)

3. Per-Endpoint Certificate Pinning - BUILT (2026-09-18 update)

Pins are SPKI SHA-256, validated as 64 lowercase hex on the way in, enforced on every agent channel, and owned by the registry rather than by appsettings.json (plugins/endpoints/client/Set-ShopDBServerPins.ps1 writes HKLM:\SOFTWARE\GE\ShopDB ServerPins; the precedence is stated in plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md, section "The registry wins"). A CA-flagged certificate is pinned but deliberately NOT imported into LocalMachine\Root, and the build warns the admin through an X-ShopDB-Build-Warning header. See the risk-matrix correction above for the enforcement call sites.

Monitoring & Alerting

Current: Reactive (users report issues)
Proposed: Proactive monitoring via /production-health skill

Metrics to monitor:

• Database disk space (alert at 85%)
• Docker log sizes (alert if rotation failing)
• Audit log row count (alert if growing >10K rows/day)
• Endpoint reporting (alert if >10 endpoints offline >48 hours)
• API error rate (alert if >1% requests fail)
• Backup age (alert if last backup >24 hours old)

Implementation: Cron job runs /production-health daily, sends email/Slack on failures

Effort: 8 hours
Savings: 52 hrs/year (from reactive troubleshooting to proactive alerts)

References

• Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned
• Architectural Findings 2026-09-11: re-checked 2026-09-18; most of the 22 findings are refuted or stale, see that page for the verdict on each
• ADR-Conformance-2026-09-18: which ADRs were checked in the 2026-09-18 review and which hold
• Automation Skills: 13 skills created in the assistant skills directory
• Deployment Documentation: docs/deployment/ and the assistant setup guides
• ADR-004: Deployment Topology (air-gapped architecture)
• ADR-008: Plugin Migration Ownership (linear chain requirement)