shopdb-flask/.claude/maintenance/MAINTENANCE-DEPLOYMENT-CHECKLIST.md

ShopDB Maintenance & Deployment Checklist

Generated: 2026-09-11
Source: Maintenance Footprint Analysis (936 hrs/year baseline)

Quick Reference

Risk Profile: HIGH
Annual Maintenance: 936 hours (0.45 FTE)
Critical Risks: 8
Quick Wins Available: 7 (22-35 hours total effort)

---

IMMEDIATE ACTIONS (Next 30 Days)

Priority 1: Data Growth Bomb (6 months to disk exhaustion)

[ ] Configure Audit Log Retention (4 hours)

Owner: Developer + Admin
Effort: 2 hours config + 2 hours testing

# Test on copy of production database first
docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40 --dry-run

# Add to crontab (daily at 2 AM)
0 2 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40

Validation:

• Before: SELECT COUNT(*) FROM auditlogs; (expect millions)
• After 30 days: Should stabilize under 1M rows
• Backup size reduction: 60-80%

[ ] Configure Endpoint Event Retention (8 hours)

Owner: Developer
Effort: 4 hours dev + 2 hours testing + 2 hours deploy

Implementation:

1. Add Flask CLI command to plugins/endpoints/cli.py:

@endpoints_cli.command('prune-events')
@click.option('--days', default=90, help='Retain events for N days')
@click.option('--batch-size', default=10000, help='Delete batch size')
@click.option('--dry-run', is_flag=True, help='Show what would be deleted')
def prune_events(days, batch_size, dry_run):
    """Prune endpoint events older than N days."""
    cutoff = datetime.utcnow() - timedelta(days=days)
    
    if dry_run:
        count = db.session.query(func.count(EndpointEvent.id)).filter(
            EndpointEvent.eventtime < cutoff
        ).scalar()
        click.echo(f"Would delete {count:,} events older than {cutoff.date()}")
        return
    
    deleted = 0
    while True:
        batch = db.session.query(EndpointEvent).filter(
            EndpointEvent.eventtime < cutoff
        ).limit(batch_size).all()
        
        if not batch:
            break
        
        for event in batch:
            db.session.delete(event)
        
        db.session.commit()
        deleted += len(batch)
        click.echo(f"Deleted {deleted:,} events...")
    
    click.echo(f"Total deleted: {deleted:,} events")

2. Add to crontab (daily at 3 AM):

0 3 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask endpoints prune-events --days 90

Validation:

• Test on copy first: flask endpoints prune-events --days 90 --dry-run
• Monitor table size: SELECT COUNT(*) FROM endpointevents;
• Should stabilize under 500K rows

[ ] Configure Docker Log Rotation (2 hours)

Owner: Admin
Effort: 1 hour config + 1 hour verification

Method 1: daemon.json (applies to all containers)

// C:\ProgramData\Docker\config\daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Method 2: docker-compose.yml (per-service)

services:
  api:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"

After changing daemon.json:

Restart-Service docker
docker compose -f docker-compose.prebuilt.yml restart

Validation:

# Check log file sizes
Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" | 
  Select-Object Name, @{N='SizeMB';E={[math]::Round($_.Length/1MB,2)}} | 
  Sort-Object SizeMB -Descending

Priority 2: Backup and Recovery (currently MISSING)

[ ] Enable Backup Profile (6 hours)

Owner: Admin + Developer
Effort: 3 hours setup + 3 hours restore test

Setup:

1. Generate secure backup password:

# Generate 32-character password
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 32 | % {[char]$_})

2. Add to .env:

RESTIC_PASSWORD=<generated-password-here>
RESTIC_REPOSITORY=/backup/shopdb
BACKUP_RETENTION_DAYS=90
BACKUP_RETENTION_WEEKS=12
BACKUP_RETENTION_MONTHS=18

3. Enable backup service:

docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d

4. Initialize repository (first time only):

docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic init

5. Add to crontab (daily at 1 AM):

0 1 * * * cd /path/to/shopdb && docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup

Validation:

# List snapshots
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots

# Test restore to temp location
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test

Quarterly Restore Test:

1. Restore to test environment
2. Start services on test environment
3. Verify database integrity: flask db current
4. Spot-check data (assets, endpoints, users)
5. Run migrations to latest: flask db upgrade
6. Document restore time and any issues

[ ] Add Disk Space Monitoring (4 hours)

Owner: Admin
Effort: 2 hours script + 2 hours integration

PowerShell Script (scripts/monitor-disk-space.ps1):

# Disk space monitoring for ShopDB
$HostDiskThreshold = 85
$DbVolumeThreshold = 40  # GB
$InstanceVolumeThreshold = 15  # GB
$AlertEmail = "shopdb-alerts@example.com"

# Check host disk
$disk = Get-PSDrive C | Select-Object @{N='Used%';E={[math]::Round(($_.Used/($_.Used+$_.Free))*100,1)}}
if ($disk.'Used%' -gt $HostDiskThreshold) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Host disk ${disk.'Used%'}% full" `
        -Body "Host C: drive is ${disk.'Used%'}% full (threshold: ${HostDiskThreshold}%)"
}

# Check Docker volumes
$dbVolume = docker volume inspect shopdb_db_data | ConvertFrom-Json
$dbSizeGB = [math]::Round((Get-ChildItem $dbVolume.Mountpoint -Recurse | 
    Measure-Object -Property Length -Sum).Sum / 1GB, 2)

if ($dbSizeGB -gt $DbVolumeThreshold) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Database volume ${dbSizeGB}GB" `
        -Body "db_data volume is ${dbSizeGB}GB (threshold: ${DbVolumeThreshold}GB)"
}

# Check container log sizes
$logSizeMB = (Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" | 
    Measure-Object -Property Length -Sum).Sum / 1MB

if ($logSizeMB -gt 1000) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Container logs ${logSizeMB}MB" `
        -Body "Container logs total ${logSizeMB}MB. Configure log rotation."
}

# Log results
Add-Content "C:\ShopDB\logs\disk-monitor.log" "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),${disk.'Used%'},${dbSizeGB},${logSizeMB}"

Task Scheduler (daily at 6 AM):

$action = New-ScheduledTaskAction -Execute "powershell.exe" `
    -Argument "-File C:\ShopDB\scripts\monitor-disk-space.ps1"
$trigger = New-ScheduledTaskTrigger -Daily -At 6am
Register-ScheduledTask -TaskName "ShopDB-DiskMonitor" -Action $action -Trigger $trigger

Priority 3: Production Stability (bleeding-edge runtime)

[ ] Downgrade Python to 3.11 LTS (8 hours)

Owner: Developer
Effort: 2 hours rebuild + 4 hours pilot + 2 hours bundle

Steps:

1. Edit Dockerfile line 101:

# FROM python:3.14-slim
FROM python:3.11-slim

2. Edit Dockerfile.prebuilt line 1:

# FROM python:3.14-slim
FROM python:3.11-slim

3. Rebuild image:

docker compose -f docker-compose.prebuilt.yml build --no-cache

4. Test on pilot site for 1 week:

   • Deploy to ops environment
   • Monitor for runtime errors
   • Verify all plugins load
   • Run full test suite: pytest -v

5. Create bundle for production:

.\pull-shopdb-bundle.cmd wj

6. Deploy to production sites

Validation:

# Verify Python version in container
docker compose -f docker-compose.prebuilt.yml run --rm api python --version
# Should output: Python 3.11.x

# Check for deprecation warnings in logs
docker compose -f docker-compose.prebuilt.yml logs api | grep -i deprecated

---

MONITORING SETUP (First Week)

Daily Checks (5 minutes)

[ ] Endpoint Health Dashboard

-- Check endpoint reporting status
SELECT 
  COUNT(*) as total_endpoints,
  SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) as reported_24h,
  ROUND(100.0 * SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_healthy
FROM assets
WHERE assettype = 'endpoint';

Alert if: pct_healthy < 95%

[ ] Backup Verification

# Check last backup timestamp
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots --last

# Should show snapshot within last 24 hours

Alert if: No snapshot in last 24 hours

[ ] Docker Service Health

docker ps --filter name=shopdb --format "table {{.Names}}\t{{.Status}}\t{{.State}}"

Alert if: Any service not "Up" and "healthy"

[ ] Database Disk Space

# Quick check
docker volume inspect shopdb_db_data | ConvertFrom-Json | 
  Select-Object @{N='Mountpoint';E={$_.Mountpoint}}, 
                @{N='SizeGB';E={[math]::Round((Get-ChildItem $_.Mountpoint -Recurse | 
                  Measure-Object -Property Length -Sum).Sum / 1GB, 2)}}

Alert if: >40GB (or >85% of allocated disk)

[ ] Critical Application Errors

# Check for OperationalError, IntegrityError, 500 responses
docker compose -f docker-compose.prebuilt.yml logs --since 24h api | 
  grep -E "OperationalError|IntegrityError|HTTP/1.1\" 500"

Alert if: More than 10 errors in 24 hours

Weekly Checks (15 minutes)

[ ] Audit Log Size

SELECT 
  COUNT(*) as total_rows,
  MIN(created) as oldest_entry,
  MAX(created) as newest_entry,
  DATEDIFF(MAX(created), MIN(created)) as retention_days
FROM auditlogs;

Alert if: total_rows > 1M and retention_days > 90

[ ] Failed Login Attempts

SELECT 
  username,
  failed_attempts,
  lastfailedlogin,
  isactive
FROM users
WHERE failed_attempts > 0
ORDER BY failed_attempts DESC
LIMIT 20;

Alert if: Any account with >10 failed attempts in 24 hours

[ ] Endpoint Events Accumulation

SELECT 
  COUNT(*) as total_events,
  MIN(eventtime) as oldest_event,
  MAX(eventtime) as newest_event
FROM endpointevents;

Alert if: total_events > 500K

[ ] Security Advisories

cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json

Review: HIGH severity findings in shipped components (exclude dev-only)

[ ] Database Performance

-- Check for slow queries (if slow query log enabled)
SELECT 
  query_time,
  lock_time,
  rows_examined,
  sql_text
FROM mysql.slow_log
WHERE query_time > 5
ORDER BY query_time DESC
LIMIT 10;

Alert if: Regular queries taking >5 seconds

Monthly Checks (1 hour)

[ ] PAT Usage Audit

SELECT 
  t.tokenid,
  u.username,
  t.lastusedat,
  t.scope,
  DATEDIFF(NOW(), t.lastusedat) as days_since_use
FROM apitokens t
JOIN users u ON t.userid = u.userid
WHERE t.lastusedat < NOW() - INTERVAL 90 DAY
   OR t.lastusedat IS NULL
ORDER BY days_since_use DESC;

Action: Revoke stale tokens (not used in 90+ days)

[ ] Certificate Expiration Check

openssl x509 -in ./certs/server.crt -noout -enddate

Alert if: <60 days until expiry

[ ] Backup Restoration Test

# Restore to test environment
docker compose -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test

# Start test stack
cd /restore-test
docker compose -f docker-compose.prebuilt.yml up -d

# Verify database
docker compose exec api flask db current
# Should show: 7d51_endpoint_logcollections (head)

Document: Restore time, data integrity, any issues

[ ] Plugin Version Audit

# Compare plugin registry to disk versions
docker compose run --rm api python -c "
from shopdb.extensions import plugin_registry
for name, plugin in plugin_registry.plugins.items():
    print(f'{name}: {plugin.version}')
"

Alert if: Version mismatch between registry and MANIFEST.json

[ ] Configuration Drift Review

# Compare .env files across sites
# dev, ops, prod should have consistent:
# - CORS_ORIGINS
# - TRUSTED_PROXY_COUNT
# - Collector API keys
# - JWT secret keys

Document: Any drift and reason

Quarterly Checks (4 hours)

[ ] Full DR Drill

Scenario: Complete site loss, restore from backup

Steps:

1. Simulate failure (stop all containers, optionally remove volumes)
2. Restore from latest backup
3. Redeploy containers
4. Verify 700 endpoints reconnect within 15 minutes
5. Spot-check all features:
   • User login
   • Asset search
   • Script execution
   • Report generation
6. Document restore time and issues

Target: Full recovery in <2 hours

[ ] Security Log Audit

-- Check for anomalous patterns
SELECT 
  userid,
  actiontype,
  COUNT(*) as action_count,
  MIN(created) as first_action,
  MAX(created) as last_action
FROM auditlogs
WHERE created > NOW() - INTERVAL 90 DAY
  AND (
    actiontype IN ('user.delete', 'permission.grant', 'setting.update')
    OR HOUR(created) NOT BETWEEN 6 AND 22  -- Off-hours access
  )
GROUP BY userid, actiontype
HAVING action_count > 5
ORDER BY action_count DESC;

Review: Off-hours access, bulk deletes, permission changes

[ ] Dependency Vulnerability Scan

# Run OSV.dev scan
cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json > vuln-report.txt

# Review findings
grep -E "HIGH|CRITICAL" vuln-report.txt | grep -v "dev-only"

Action: Plan upgrade path for HIGH findings in shipped components

---

AUTOMATED ALERTS (Configure Once)

Critical Alerts (immediate notification)

[ ] Database Disk >85% Full

Method: Daily disk monitor script (see Priority 2)
Recipient: Admin + Developer
Action: Implement retention policies or expand disk

[ ] Host Disk >85% Full

Method: Daily disk monitor script
Recipient: Admin
Action: Clean logs, check Docker volumes, expand disk

[ ] Backup Failed or >24h Old

Method: Check restic last snapshot timestamp
Recipient: Admin
Action: Investigate backup service logs, verify disk space

[ ] >10 Endpoints Offline >48 Hours

Method: Query assets table daily
Recipient: Admin
Action: Check network connectivity, agent service status

[ ] Application Error Rate >1% of Requests

Method: Parse gunicorn access logs
Recipient: Developer
Action: Check Flask logs for exceptions, review recent changes

[ ] Certificate Expiring in <30 Days

Method: Daily openssl check
Recipient: Admin
Action: Generate new self-signed cert or renew

Warning Alerts (daily summary)

[ ] Audit Log >1M Rows Without Retention

Method: Daily SQL query
Recipient: Admin
Action: Configure audit log purge job

[ ] Endpoint Events >500K Rows

Method: Daily SQL query
Recipient: Admin
Action: Configure event retention

[ ] Failed Login Rate >10/min from Single IP

Method: Parse Flask auth logs
Recipient: Admin
Action: Review for brute force attempt, consider IP block

---

MAINTENANCE SCHEDULE

Daily (5 min)

• Endpoint health dashboard check
• Backup verification
• Docker service health
• Database disk space
• Critical error review

Weekly (15 min)

• Audit log size check
• Failed login review
• Endpoint events accumulation
• Security advisories scan
• Database performance metrics

Monthly (1 hour)

• PAT usage audit
• Certificate expiration check
• Backup restoration test
• Plugin version audit
• Configuration drift review

Quarterly (4 hours)

• Full DR drill
• Security log audit
• Dependency vulnerability scan

Total Annual Scheduled Effort:

• Daily: 365 * 5 min = 30 hours
• Weekly: 52 * 15 min = 13 hours
• Monthly: 12 * 1 hour = 12 hours
• Quarterly: 4 * 4 hours = 16 hours
• Subtotal: 71 hours/year

Full Baseline (from analysis): 936 hours/year

• Scheduled monitoring: 71 hours (8%)
• Planned maintenance: 600 hours (64%)
• Ad-hoc incident response: 265 hours (28%)

---

DEPLOYMENT RUNBOOK

Pre-Deployment Checklist

[ ] Create Backup

# Take manual snapshot before deploy
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic backup /app/instance --tag pre-deploy

[ ] Test Bundle on Pilot Site

• Deploy to ops environment first
• Verify all services start (db, migrate, api)
• Check migration logs for errors
• Spot-check core features
• Monitor for 24 hours

[ ] Review Migration Chain

# Check what migrations will run
docker compose run --rm api flask db heads
docker compose run --rm api flask db current

[ ] Verify Disk Space

# Ensure enough space for migration and new code
df -h /var/lib/docker

Deployment Steps

1. Stop Services

docker compose -f docker-compose.prebuilt.yml down

2. Extract Bundle

# Transfer bundle across air gap
# Extract to deployment directory
tar -xzf shopdb-bundle-wj-<date>.tar.gz

3. Pull Latest Code

cd /path/to/shopdb
git pull origin wj
# Or: git bundle unbundle ../shopdb-wj.bundle

4. Rebuild Images

docker compose -f docker-compose.prebuilt.yml build --no-cache

5. Start Stack (migrations run automatically)

docker compose -f docker-compose.prebuilt.yml up -d

6. Monitor Startup

# Watch logs in real-time
docker compose -f docker-compose.prebuilt.yml logs -f migrate

# Verify migrations completed
docker compose -f docker-compose.prebuilt.yml logs migrate | tail -20

# Check services healthy
docker ps

7. Verify Application

# Check database schema version
docker compose run --rm api flask db current

# Test core endpoints
curl https://localhost:5001/api/health
curl https://localhost:5001/api/assets/types

# Check UI loads
# Navigate to https://<hostname>:5173

Rollback Procedure

If Migration Fails:

# Stop services
docker compose -f docker-compose.prebuilt.yml down

# Restore from pre-deploy backup
docker compose -f docker-compose.backup.yml run --rm backup restic restore <snapshot-id> --target /restore

# Copy restored instance/ directory
cp -r /restore/app/instance /app/instance

# Revert code to previous version
git checkout <previous-commit>

# Rebuild and start
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml up -d

If Application Fails Post-Deploy:

# Check logs first
docker compose logs api | grep -E "ERROR|CRITICAL"

# If unfixable, rollback code
git checkout <previous-commit>
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml restart api

Post-Deployment Verification

[ ] All Services Healthy

docker ps --filter name=shopdb
# Should show: db (healthy), api (healthy), migrate (exited 0)

[ ] Endpoints Checking In

SELECT COUNT(*) FROM assets 
WHERE assettype='endpoint' 
  AND lastreportedat > NOW() - INTERVAL 30 MINUTE;

Expect: >95% of endpoints

[ ] No Critical Errors in Logs

docker compose logs --since 1h api | grep -i critical

Expect: Empty or expected warnings only

[ ] UI Loads and Functions

• Login works
• Asset search returns results
• Can navigate to endpoint details
• Scripts tab loads

[ ] Document Deployment

## Deployment Log: 2026-09-11

- **Branch:** wj
- **Commit:** 0b252db7d2
- **Migrations:** 7d51_endpoint_logcollections (head)
- **Downtime:** 5 minutes
- **Issues:** None
- **Endpoint Recovery:** 98% within 15 minutes

---

SKILL REQUIREMENTS

Operator (20% of maintenance)

• Run daily monitoring scripts
• Verify backup completion
• Check disk space
• Restart services if needed
• Basic Windows/Docker troubleshooting

Admin (25% of maintenance)

• Configure cron jobs / Task Scheduler
• Manage Docker Compose deployments
• User/permission management
• Review logs for security issues
• Certificate management

Developer (40% of maintenance)

• Dependency upgrades
• Plugin updates
• Troubleshoot application errors
• Database performance tuning
• Implement retention policies
• Security patch development

Architect (15% of maintenance)

• Major version upgrades
• DR drill planning
• Security audit response
• Capacity planning
• Architecture decisions

---

CRITICAL COMPETENCIES REQUIRED

1. Air-gap bundle workflow - Create git bundles, transfer across three gaps, verify checksums
2. Database migration troubleshooting - Alembic per-plugin chains, PLUGIN_TABLE_OWNERS, MySQL DDL behavior
3. Docker Compose production operations - Multi-service stack, volume persistence, logging drivers
4. Flask + SQLAlchemy performance tuning - Connection pools, query optimization, retention policies
5. Authentication and authorization - JWT tokens, PATs, role permissions, collector key security
6. Plugin architecture - Dependency graph, contract versions, enable/disable coordination
7. Security audit response - Translate findings to remediation plans, prioritize risks

---

KNOWN LIMITATIONS

Cannot Be Automated

• Air-gap bundle transfer (physical media required)
• Certificate renewal (self-signed, manual process)
• Plugin dependency ordering (requires developer judgment)
• Security audit triage (requires threat modeling)

Requires Developer Skills

• Migration failures (manual schema reconciliation)
• Plugin contract violations (code changes needed)
• Performance degradation (query optimization, indexing)
• Security vulnerability remediation (code patches)

Incident Response Lag

• Bundle creation + 3 air gaps = 2-3 day minimum for critical fixes
• No rapid patching for zero-day vulnerabilities
• Cannot roll back failed migrations easily (MySQL DDL auto-commits)

---

REFERENCES

• maintenance-footprint-baseline - Full 936-hour analysis
• security-audit-2026-09-11 - Security findings driving Priority 1-3
• security-fix-implementation - Long-term security roadmap
• docker-compose.backup.yml - Backup service configuration
• scripts/scan_sbom.py - Vulnerability scanning tool