Files
shopdb-flask/.claude/maintenance/MAINTENANCE-DEPLOYMENT-CHECKLIST.md
Proudlock bd48814021 Reorganize repository structure and clean up documentation
Clean up and organize the flask-shopdb repository for better
maintainability and discoverability:

## Changes

- Update .gitignore for build artifacts (publish/, data/, 122M total)
- Reorganize .claude/ into 6 subdirectories with README navigation
- Consolidate overlapping MCP setup guides into MCP-QUICK-START.md
- Move deployment docs from root to docs/deployment/
- Archive dated status reports to .claude/archived/
- Delete temporary files (logs, backups, empty reports, ~50K)

## New Structure

.claude/ organized into:
- setup/ - MCP, VM, Hyper-V, GitHub setup guides + scripts (17 files)
- plans/ - Feature plans and protocols (5 files)
- maintenance/ - Deployment checklists and reports (4 files)
- archived/ - Historical documentation (9 files)
- skills/ - Claude Code automation (5 files)
- workflows/ - Security audit workflow (1 file)

docs/deployment/ created with 6 deployment guides:
- LOCAL-DEV-SETUP.md, DOCKER-SETUP.md, DOCKER-PREBUILT-APPROACH.md
- PRODUCTION-READINESS.md, DEPLOY-TO-VM.md, README.md

Root directory cleaned to essential docs only:
- CLAUDE.md, README.md, CONTRIBUTING.md, CHANGELOG.md

Benefits:
- Clear navigation via README.md in each subdirectory
- No duplicate/overlapping documentation
- Large artifacts properly gitignored (won't bloat repository)
- Historical context preserved in archived/

See .claude/CLEANUP-SUMMARY.md for complete details.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-09-11 15:45:05 -04:00

832 lines
22 KiB
Markdown

# ShopDB Maintenance & Deployment Checklist
**Generated:** 2026-09-11
**Source:** Maintenance Footprint Analysis (936 hrs/year baseline)
## Quick Reference
**Risk Profile:** HIGH
**Annual Maintenance:** 936 hours (0.45 FTE)
**Critical Risks:** 8
**Quick Wins Available:** 7 (22-35 hours total effort)
---
## IMMEDIATE ACTIONS (Next 30 Days)
### Priority 1: Data Growth Bomb (6 months to disk exhaustion)
#### ☐ Configure Audit Log Retention (4 hours)
**Owner:** Developer + Admin
**Effort:** 2 hours config + 2 hours testing
```bash
# Test on copy of production database first
docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40 --dry-run
# Add to crontab (daily at 2 AM)
0 2 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40
```
**Validation:**
- Before: `SELECT COUNT(*) FROM auditlogs;` (expect millions)
- After 30 days: Should stabilize under 1M rows
- Backup size reduction: 60-80%
#### ☐ Configure Endpoint Event Retention (8 hours)
**Owner:** Developer
**Effort:** 4 hours dev + 2 hours testing + 2 hours deploy
**Implementation:**
1. Add Flask CLI command to `plugins/endpoints/cli.py`:
```python
@endpoints_cli.command('prune-events')
@click.option('--days', default=90, help='Retain events for N days')
@click.option('--batch-size', default=10000, help='Delete batch size')
@click.option('--dry-run', is_flag=True, help='Show what would be deleted')
def prune_events(days, batch_size, dry_run):
"""Prune endpoint events older than N days."""
cutoff = datetime.utcnow() - timedelta(days=days)
if dry_run:
count = db.session.query(func.count(EndpointEvent.id)).filter(
EndpointEvent.eventtime < cutoff
).scalar()
click.echo(f"Would delete {count:,} events older than {cutoff.date()}")
return
deleted = 0
while True:
batch = db.session.query(EndpointEvent).filter(
EndpointEvent.eventtime < cutoff
).limit(batch_size).all()
if not batch:
break
for event in batch:
db.session.delete(event)
db.session.commit()
deleted += len(batch)
click.echo(f"Deleted {deleted:,} events...")
click.echo(f"Total deleted: {deleted:,} events")
```
2. Add to crontab (daily at 3 AM):
```bash
0 3 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask endpoints prune-events --days 90
```
**Validation:**
- Test on copy first: `flask endpoints prune-events --days 90 --dry-run`
- Monitor table size: `SELECT COUNT(*) FROM endpointevents;`
- Should stabilize under 500K rows
#### ☐ Configure Docker Log Rotation (2 hours)
**Owner:** Admin
**Effort:** 1 hour config + 1 hour verification
**Method 1: daemon.json (applies to all containers)**
```json
// C:\ProgramData\Docker\config\daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}
```
**Method 2: docker-compose.yml (per-service)**
```yaml
services:
api:
logging:
driver: json-file
options:
max-size: "10m"
max-file: "5"
```
**After changing daemon.json:**
```powershell
Restart-Service docker
docker compose -f docker-compose.prebuilt.yml restart
```
**Validation:**
```powershell
# Check log file sizes
Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
Select-Object Name, @{N='SizeMB';E={[math]::Round($_.Length/1MB,2)}} |
Sort-Object SizeMB -Descending
```
### Priority 2: Backup and Recovery (currently MISSING)
#### ☐ Enable Backup Profile (6 hours)
**Owner:** Admin + Developer
**Effort:** 3 hours setup + 3 hours restore test
**Setup:**
1. Generate secure backup password:
```powershell
# Generate 32-character password
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 32 | % {[char]$_})
```
2. Add to `.env`:
```bash
RESTIC_PASSWORD=<generated-password-here>
RESTIC_REPOSITORY=/backup/shopdb
BACKUP_RETENTION_DAYS=90
BACKUP_RETENTION_WEEKS=12
BACKUP_RETENTION_MONTHS=18
```
3. Enable backup service:
```bash
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
```
4. Initialize repository (first time only):
```bash
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic init
```
5. Add to crontab (daily at 1 AM):
```bash
0 1 * * * cd /path/to/shopdb && docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup
```
**Validation:**
```bash
# List snapshots
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots
# Test restore to temp location
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
```
**Quarterly Restore Test:**
1. Restore to test environment
2. Start services on test environment
3. Verify database integrity: `flask db current`
4. Spot-check data (assets, endpoints, users)
5. Run migrations to latest: `flask db upgrade`
6. Document restore time and any issues
#### ☐ Add Disk Space Monitoring (4 hours)
**Owner:** Admin
**Effort:** 2 hours script + 2 hours integration
**PowerShell Script** (`scripts/monitor-disk-space.ps1`):
```powershell
# Disk space monitoring for ShopDB
$HostDiskThreshold = 85
$DbVolumeThreshold = 40 # GB
$InstanceVolumeThreshold = 15 # GB
$AlertEmail = "shopdb-alerts@example.com"
# Check host disk
$disk = Get-PSDrive C | Select-Object @{N='Used%';E={[math]::Round(($_.Used/($_.Used+$_.Free))*100,1)}}
if ($disk.'Used%' -gt $HostDiskThreshold) {
Send-MailMessage -To $AlertEmail -Subject "ALERT: Host disk ${disk.'Used%'}% full" `
-Body "Host C: drive is ${disk.'Used%'}% full (threshold: ${HostDiskThreshold}%)"
}
# Check Docker volumes
$dbVolume = docker volume inspect shopdb_db_data | ConvertFrom-Json
$dbSizeGB = [math]::Round((Get-ChildItem $dbVolume.Mountpoint -Recurse |
Measure-Object -Property Length -Sum).Sum / 1GB, 2)
if ($dbSizeGB -gt $DbVolumeThreshold) {
Send-MailMessage -To $AlertEmail -Subject "ALERT: Database volume ${dbSizeGB}GB" `
-Body "db_data volume is ${dbSizeGB}GB (threshold: ${DbVolumeThreshold}GB)"
}
# Check container log sizes
$logSizeMB = (Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
Measure-Object -Property Length -Sum).Sum / 1MB
if ($logSizeMB -gt 1000) {
Send-MailMessage -To $AlertEmail -Subject "ALERT: Container logs ${logSizeMB}MB" `
-Body "Container logs total ${logSizeMB}MB. Configure log rotation."
}
# Log results
Add-Content "C:\ShopDB\logs\disk-monitor.log" "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),${disk.'Used%'},${dbSizeGB},${logSizeMB}"
```
**Task Scheduler** (daily at 6 AM):
```powershell
$action = New-ScheduledTaskAction -Execute "powershell.exe" `
-Argument "-File C:\ShopDB\scripts\monitor-disk-space.ps1"
$trigger = New-ScheduledTaskTrigger -Daily -At 6am
Register-ScheduledTask -TaskName "ShopDB-DiskMonitor" -Action $action -Trigger $trigger
```
### Priority 3: Production Stability (bleeding-edge runtime)
#### ☐ Downgrade Python to 3.11 LTS (8 hours)
**Owner:** Developer
**Effort:** 2 hours rebuild + 4 hours pilot + 2 hours bundle
**Steps:**
1. Edit `Dockerfile` line 101:
```dockerfile
# FROM python:3.14-slim
FROM python:3.11-slim
```
2. Edit `Dockerfile.prebuilt` line 1:
```dockerfile
# FROM python:3.14-slim
FROM python:3.11-slim
```
3. Rebuild image:
```bash
docker compose -f docker-compose.prebuilt.yml build --no-cache
```
4. Test on pilot site for 1 week:
- Deploy to ops environment
- Monitor for runtime errors
- Verify all plugins load
- Run full test suite: `pytest -v`
5. Create bundle for production:
```bash
.\pull-shopdb-bundle.cmd wj
```
6. Deploy to production sites
**Validation:**
```bash
# Verify Python version in container
docker compose -f docker-compose.prebuilt.yml run --rm api python --version
# Should output: Python 3.11.x
# Check for deprecation warnings in logs
docker compose -f docker-compose.prebuilt.yml logs api | grep -i deprecated
```
---
## MONITORING SETUP (First Week)
### Daily Checks (5 minutes)
#### ☐ Endpoint Health Dashboard
```sql
-- Check endpoint reporting status
SELECT
COUNT(*) as total_endpoints,
SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) as reported_24h,
ROUND(100.0 * SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_healthy
FROM assets
WHERE assettype = 'endpoint';
```
**Alert if:** pct_healthy < 95%
#### ☐ Backup Verification
```bash
# Check last backup timestamp
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots --last
# Should show snapshot within last 24 hours
```
**Alert if:** No snapshot in last 24 hours
#### ☐ Docker Service Health
```bash
docker ps --filter name=shopdb --format "table {{.Names}}\t{{.Status}}\t{{.State}}"
```
**Alert if:** Any service not "Up" and "healthy"
#### ☐ Database Disk Space
```powershell
# Quick check
docker volume inspect shopdb_db_data | ConvertFrom-Json |
Select-Object @{N='Mountpoint';E={$_.Mountpoint}},
@{N='SizeGB';E={[math]::Round((Get-ChildItem $_.Mountpoint -Recurse |
Measure-Object -Property Length -Sum).Sum / 1GB, 2)}}
```
**Alert if:** >40GB (or >85% of allocated disk)
#### ☐ Critical Application Errors
```bash
# Check for OperationalError, IntegrityError, 500 responses
docker compose -f docker-compose.prebuilt.yml logs --since 24h api |
grep -E "OperationalError|IntegrityError|HTTP/1.1\" 500"
```
**Alert if:** More than 10 errors in 24 hours
### Weekly Checks (15 minutes)
#### ☐ Audit Log Size
```sql
SELECT
COUNT(*) as total_rows,
MIN(created) as oldest_entry,
MAX(created) as newest_entry,
DATEDIFF(MAX(created), MIN(created)) as retention_days
FROM auditlogs;
```
**Alert if:** total_rows > 1M and retention_days > 90
#### ☐ Failed Login Attempts
```sql
SELECT
username,
failed_attempts,
lastfailedlogin,
isactive
FROM users
WHERE failed_attempts > 0
ORDER BY failed_attempts DESC
LIMIT 20;
```
**Alert if:** Any account with >10 failed attempts in 24 hours
#### ☐ Endpoint Events Accumulation
```sql
SELECT
COUNT(*) as total_events,
MIN(eventtime) as oldest_event,
MAX(eventtime) as newest_event
FROM endpointevents;
```
**Alert if:** total_events > 500K
#### ☐ Security Advisories
```bash
cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json
```
**Review:** HIGH severity findings in shipped components (exclude dev-only)
#### ☐ Database Performance
```sql
-- Check for slow queries (if slow query log enabled)
SELECT
query_time,
lock_time,
rows_examined,
sql_text
FROM mysql.slow_log
WHERE query_time > 5
ORDER BY query_time DESC
LIMIT 10;
```
**Alert if:** Regular queries taking >5 seconds
### Monthly Checks (1 hour)
#### ☐ PAT Usage Audit
```sql
SELECT
t.tokenid,
u.username,
t.lastusedat,
t.scope,
DATEDIFF(NOW(), t.lastusedat) as days_since_use
FROM apitokens t
JOIN users u ON t.userid = u.userid
WHERE t.lastusedat < NOW() - INTERVAL 90 DAY
OR t.lastusedat IS NULL
ORDER BY days_since_use DESC;
```
**Action:** Revoke stale tokens (not used in 90+ days)
#### ☐ Certificate Expiration Check
```bash
openssl x509 -in ./certs/server.crt -noout -enddate
```
**Alert if:** <60 days until expiry
#### ☐ Backup Restoration Test
```bash
# Restore to test environment
docker compose -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
# Start test stack
cd /restore-test
docker compose -f docker-compose.prebuilt.yml up -d
# Verify database
docker compose exec api flask db current
# Should show: 7d51_endpoint_logcollections (head)
```
**Document:** Restore time, data integrity, any issues
#### ☐ Plugin Version Audit
```bash
# Compare plugin registry to disk versions
docker compose run --rm api python -c "
from shopdb.extensions import plugin_registry
for name, plugin in plugin_registry.plugins.items():
print(f'{name}: {plugin.version}')
"
```
**Alert if:** Version mismatch between registry and MANIFEST.json
#### ☐ Configuration Drift Review
```bash
# Compare .env files across sites
# dev, ops, prod should have consistent:
# - CORS_ORIGINS
# - TRUSTED_PROXY_COUNT
# - Collector API keys
# - JWT secret keys
```
**Document:** Any drift and reason
### Quarterly Checks (4 hours)
#### ☐ Full DR Drill
**Scenario:** Complete site loss, restore from backup
**Steps:**
1. Simulate failure (stop all containers, optionally remove volumes)
2. Restore from latest backup
3. Redeploy containers
4. Verify 700 endpoints reconnect within 15 minutes
5. Spot-check all features:
- User login
- Asset search
- Script execution
- Report generation
6. Document restore time and issues
**Target:** Full recovery in <2 hours
#### ☐ Security Log Audit
```sql
-- Check for anomalous patterns
SELECT
userid,
actiontype,
COUNT(*) as action_count,
MIN(created) as first_action,
MAX(created) as last_action
FROM auditlogs
WHERE created > NOW() - INTERVAL 90 DAY
AND (
actiontype IN ('user.delete', 'permission.grant', 'setting.update')
OR HOUR(created) NOT BETWEEN 6 AND 22 -- Off-hours access
)
GROUP BY userid, actiontype
HAVING action_count > 5
ORDER BY action_count DESC;
```
**Review:** Off-hours access, bulk deletes, permission changes
#### ☐ Dependency Vulnerability Scan
```bash
# Run OSV.dev scan
cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json > vuln-report.txt
# Review findings
grep -E "HIGH|CRITICAL" vuln-report.txt | grep -v "dev-only"
```
**Action:** Plan upgrade path for HIGH findings in shipped components
---
## AUTOMATED ALERTS (Configure Once)
### Critical Alerts (immediate notification)
#### ☐ Database Disk >85% Full
**Method:** Daily disk monitor script (see Priority 2)
**Recipient:** Admin + Developer
**Action:** Implement retention policies or expand disk
#### ☐ Host Disk >85% Full
**Method:** Daily disk monitor script
**Recipient:** Admin
**Action:** Clean logs, check Docker volumes, expand disk
#### ☐ Backup Failed or >24h Old
**Method:** Check restic last snapshot timestamp
**Recipient:** Admin
**Action:** Investigate backup service logs, verify disk space
#### ☐ >10 Endpoints Offline >48 Hours
**Method:** Query assets table daily
**Recipient:** Admin
**Action:** Check network connectivity, agent service status
#### ☐ Application Error Rate >1% of Requests
**Method:** Parse gunicorn access logs
**Recipient:** Developer
**Action:** Check Flask logs for exceptions, review recent changes
#### ☐ Certificate Expiring in <30 Days
**Method:** Daily openssl check
**Recipient:** Admin
**Action:** Generate new self-signed cert or renew
### Warning Alerts (daily summary)
#### ☐ Audit Log >1M Rows Without Retention
**Method:** Daily SQL query
**Recipient:** Admin
**Action:** Configure audit log purge job
#### ☐ Endpoint Events >500K Rows
**Method:** Daily SQL query
**Recipient:** Admin
**Action:** Configure event retention
#### ☐ Failed Login Rate >10/min from Single IP
**Method:** Parse Flask auth logs
**Recipient:** Admin
**Action:** Review for brute force attempt, consider IP block
---
## MAINTENANCE SCHEDULE
### Daily (5 min)
- Endpoint health dashboard check
- Backup verification
- Docker service health
- Database disk space
- Critical error review
### Weekly (15 min)
- Audit log size check
- Failed login review
- Endpoint events accumulation
- Security advisories scan
- Database performance metrics
### Monthly (1 hour)
- PAT usage audit
- Certificate expiration check
- Backup restoration test
- Plugin version audit
- Configuration drift review
### Quarterly (4 hours)
- Full DR drill
- Security log audit
- Dependency vulnerability scan
**Total Annual Scheduled Effort:**
- Daily: 365 * 5 min = 30 hours
- Weekly: 52 * 15 min = 13 hours
- Monthly: 12 * 1 hour = 12 hours
- Quarterly: 4 * 4 hours = 16 hours
- **Subtotal: 71 hours/year**
**Full Baseline (from analysis): 936 hours/year**
- Scheduled monitoring: 71 hours (8%)
- Planned maintenance: 600 hours (64%)
- Ad-hoc incident response: 265 hours (28%)
---
## DEPLOYMENT RUNBOOK
### Pre-Deployment Checklist
#### ☐ Create Backup
```bash
# Take manual snapshot before deploy
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic backup /app/instance --tag pre-deploy
```
#### ☐ Test Bundle on Pilot Site
- Deploy to ops environment first
- Verify all services start (db, migrate, api)
- Check migration logs for errors
- Spot-check core features
- Monitor for 24 hours
#### ☐ Review Migration Chain
```bash
# Check what migrations will run
docker compose run --rm api flask db heads
docker compose run --rm api flask db current
```
#### ☐ Verify Disk Space
```bash
# Ensure enough space for migration and new code
df -h /var/lib/docker
```
### Deployment Steps
#### 1. Stop Services
```bash
docker compose -f docker-compose.prebuilt.yml down
```
#### 2. Extract Bundle
```bash
# Transfer bundle across air gap
# Extract to deployment directory
tar -xzf shopdb-bundle-wj-<date>.tar.gz
```
#### 3. Pull Latest Code
```bash
cd /path/to/shopdb
git pull origin wj
# Or: git bundle unbundle ../shopdb-wj.bundle
```
#### 4. Rebuild Images
```bash
docker compose -f docker-compose.prebuilt.yml build --no-cache
```
#### 5. Start Stack (migrations run automatically)
```bash
docker compose -f docker-compose.prebuilt.yml up -d
```
#### 6. Monitor Startup
```bash
# Watch logs in real-time
docker compose -f docker-compose.prebuilt.yml logs -f migrate
# Verify migrations completed
docker compose -f docker-compose.prebuilt.yml logs migrate | tail -20
# Check services healthy
docker ps
```
#### 7. Verify Application
```bash
# Check database schema version
docker compose run --rm api flask db current
# Test core endpoints
curl https://localhost:5001/api/health
curl https://localhost:5001/api/assets/types
# Check UI loads
# Navigate to https://<hostname>:5173
```
### Rollback Procedure
#### If Migration Fails:
```bash
# Stop services
docker compose -f docker-compose.prebuilt.yml down
# Restore from pre-deploy backup
docker compose -f docker-compose.backup.yml run --rm backup restic restore <snapshot-id> --target /restore
# Copy restored instance/ directory
cp -r /restore/app/instance /app/instance
# Revert code to previous version
git checkout <previous-commit>
# Rebuild and start
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml up -d
```
#### If Application Fails Post-Deploy:
```bash
# Check logs first
docker compose logs api | grep -E "ERROR|CRITICAL"
# If unfixable, rollback code
git checkout <previous-commit>
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml restart api
```
### Post-Deployment Verification
#### ☐ All Services Healthy
```bash
docker ps --filter name=shopdb
# Should show: db (healthy), api (healthy), migrate (exited 0)
```
#### ☐ Endpoints Checking In
```sql
SELECT COUNT(*) FROM assets
WHERE assettype='endpoint'
AND lastreportedat > NOW() - INTERVAL 30 MINUTE;
```
**Expect:** >95% of endpoints
#### ☐ No Critical Errors in Logs
```bash
docker compose logs --since 1h api | grep -i critical
```
**Expect:** Empty or expected warnings only
#### ☐ UI Loads and Functions
- Login works
- Asset search returns results
- Can navigate to endpoint details
- Scripts tab loads
#### ☐ Document Deployment
```markdown
## Deployment Log: 2026-09-11
- **Branch:** wj
- **Commit:** 0b252db7d2
- **Migrations:** 7d51_endpoint_logcollections (head)
- **Downtime:** 5 minutes
- **Issues:** None
- **Endpoint Recovery:** 98% within 15 minutes
```
---
## SKILL REQUIREMENTS
### Operator (20% of maintenance)
- Run daily monitoring scripts
- Verify backup completion
- Check disk space
- Restart services if needed
- Basic Windows/Docker troubleshooting
### Admin (25% of maintenance)
- Configure cron jobs / Task Scheduler
- Manage Docker Compose deployments
- User/permission management
- Review logs for security issues
- Certificate management
### Developer (40% of maintenance)
- Dependency upgrades
- Plugin updates
- Troubleshoot application errors
- Database performance tuning
- Implement retention policies
- Security patch development
### Architect (15% of maintenance)
- Major version upgrades
- DR drill planning
- Security audit response
- Capacity planning
- Architecture decisions
---
## CRITICAL COMPETENCIES REQUIRED
1. **Air-gap bundle workflow** - Create git bundles, transfer across three gaps, verify checksums
2. **Database migration troubleshooting** - Alembic per-plugin chains, PLUGIN_TABLE_OWNERS, MySQL DDL behavior
3. **Docker Compose production operations** - Multi-service stack, volume persistence, logging drivers
4. **Flask + SQLAlchemy performance tuning** - Connection pools, query optimization, retention policies
5. **Authentication and authorization** - JWT tokens, PATs, role permissions, collector key security
6. **Plugin architecture** - Dependency graph, contract versions, enable/disable coordination
7. **Security audit response** - Translate findings to remediation plans, prioritize risks
---
## KNOWN LIMITATIONS
### Cannot Be Automated
- Air-gap bundle transfer (physical media required)
- Certificate renewal (self-signed, manual process)
- Plugin dependency ordering (requires developer judgment)
- Security audit triage (requires threat modeling)
### Requires Developer Skills
- Migration failures (manual schema reconciliation)
- Plugin contract violations (code changes needed)
- Performance degradation (query optimization, indexing)
- Security vulnerability remediation (code patches)
### Incident Response Lag
- Bundle creation + 3 air gaps = 2-3 day minimum for critical fixes
- No rapid patching for zero-day vulnerabilities
- Cannot roll back failed migrations easily (MySQL DDL auto-commits)
---
## REFERENCES
- [[maintenance-footprint-baseline]] - Full 936-hour analysis
- [[security-audit-2026-09-11]] - Security findings driving Priority 1-3
- [[security-fix-implementation]] - Long-term security roadmap
- `docker-compose.backup.yml` - Backup service configuration
- `scripts/scan_sbom.py` - Vulnerability scanning tool