# ShopDB Maintenance & Deployment Checklist
**Generated:** 2026-09-11  
**Source:** Maintenance Footprint Analysis (936 hrs/year baseline)

## Quick Reference

**Risk Profile:** HIGH  
**Annual Maintenance:** 936 hours (0.45 FTE)  
**Critical Risks:** 8  
**Quick Wins Available:** 7 (22-35 hours total effort)

---

## IMMEDIATE ACTIONS (Next 30 Days)

### Priority 1: Data Growth Bomb (6 months to disk exhaustion)

#### ☐ Configure Audit Log Retention (4 hours)
**Owner:** Developer + Admin  
**Effort:** 2 hours config + 2 hours testing

```bash
# Test on copy of production database first
docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40 --dry-run

# Add to crontab (daily at 2 AM)
0 2 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40
```

**Validation:**
- Before: `SELECT COUNT(*) FROM auditlogs;` (expect millions)
- After 30 days: Should stabilize under 1M rows
- Backup size reduction: 60-80%

#### ☐ Configure Endpoint Event Retention (8 hours)
**Owner:** Developer  
**Effort:** 4 hours dev + 2 hours testing + 2 hours deploy

**Implementation:**
1. Add Flask CLI command to `plugins/endpoints/cli.py`:
```python
@endpoints_cli.command('prune-events')
@click.option('--days', default=90, help='Retain events for N days')
@click.option('--batch-size', default=10000, help='Delete batch size')
@click.option('--dry-run', is_flag=True, help='Show what would be deleted')
def prune_events(days, batch_size, dry_run):
    """Prune endpoint events older than N days."""
    cutoff = datetime.utcnow() - timedelta(days=days)
    
    if dry_run:
        count = db.session.query(func.count(EndpointEvent.id)).filter(
            EndpointEvent.eventtime < cutoff
        ).scalar()
        click.echo(f"Would delete {count:,} events older than {cutoff.date()}")
        return
    
    deleted = 0
    while True:
        batch = db.session.query(EndpointEvent).filter(
            EndpointEvent.eventtime < cutoff
        ).limit(batch_size).all()
        
        if not batch:
            break
        
        for event in batch:
            db.session.delete(event)
        
        db.session.commit()
        deleted += len(batch)
        click.echo(f"Deleted {deleted:,} events...")
    
    click.echo(f"Total deleted: {deleted:,} events")
```

2. Add to crontab (daily at 3 AM):
```bash
0 3 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask endpoints prune-events --days 90
```

**Validation:**
- Test on copy first: `flask endpoints prune-events --days 90 --dry-run`
- Monitor table size: `SELECT COUNT(*) FROM endpointevents;`
- Should stabilize under 500K rows

#### ☐ Configure Docker Log Rotation (2 hours)
**Owner:** Admin  
**Effort:** 1 hour config + 1 hour verification

**Method 1: daemon.json (applies to all containers)**
```json
// C:\ProgramData\Docker\config\daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}
```

**Method 2: docker-compose.yml (per-service)**
```yaml
services:
  api:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
```

**After changing daemon.json:**
```powershell
Restart-Service docker
docker compose -f docker-compose.prebuilt.yml restart
```

**Validation:**
```powershell
# Check log file sizes
Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" | 
  Select-Object Name, @{N='SizeMB';E={[math]::Round($_.Length/1MB,2)}} | 
  Sort-Object SizeMB -Descending
```

### Priority 2: Backup and Recovery (currently MISSING)

#### ☐ Enable Backup Profile (6 hours)
**Owner:** Admin + Developer  
**Effort:** 3 hours setup + 3 hours restore test

**Setup:**
1. Generate secure backup password:
```powershell
# Generate 32-character password
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 32 | % {[char]$_})
```

2. Add to `.env`:
```bash
RESTIC_PASSWORD=<generated-password-here>
RESTIC_REPOSITORY=/backup/shopdb
BACKUP_RETENTION_DAYS=90
BACKUP_RETENTION_WEEKS=12
BACKUP_RETENTION_MONTHS=18
```

3. Enable backup service:
```bash
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
```

4. Initialize repository (first time only):
```bash
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic init
```

5. Add to crontab (daily at 1 AM):
```bash
0 1 * * * cd /path/to/shopdb && docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup
```

**Validation:**
```bash
# List snapshots
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots

# Test restore to temp location
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
```

**Quarterly Restore Test:**
1. Restore to test environment
2. Start services on test environment
3. Verify database integrity: `flask db current`
4. Spot-check data (assets, endpoints, users)
5. Run migrations to latest: `flask db upgrade`
6. Document restore time and any issues

#### ☐ Add Disk Space Monitoring (4 hours)
**Owner:** Admin  
**Effort:** 2 hours script + 2 hours integration

**PowerShell Script** (`scripts/monitor-disk-space.ps1`):
```powershell
# Disk space monitoring for ShopDB
$HostDiskThreshold = 85
$DbVolumeThreshold = 40  # GB
$InstanceVolumeThreshold = 15  # GB
$AlertEmail = "shopdb-alerts@example.com"

# Check host disk
$disk = Get-PSDrive C | Select-Object @{N='Used%';E={[math]::Round(($_.Used/($_.Used+$_.Free))*100,1)}}
if ($disk.'Used%' -gt $HostDiskThreshold) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Host disk ${disk.'Used%'}% full" `
        -Body "Host C: drive is ${disk.'Used%'}% full (threshold: ${HostDiskThreshold}%)"
}

# Check Docker volumes
$dbVolume = docker volume inspect shopdb_db_data | ConvertFrom-Json
$dbSizeGB = [math]::Round((Get-ChildItem $dbVolume.Mountpoint -Recurse | 
    Measure-Object -Property Length -Sum).Sum / 1GB, 2)

if ($dbSizeGB -gt $DbVolumeThreshold) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Database volume ${dbSizeGB}GB" `
        -Body "db_data volume is ${dbSizeGB}GB (threshold: ${DbVolumeThreshold}GB)"
}

# Check container log sizes
$logSizeMB = (Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" | 
    Measure-Object -Property Length -Sum).Sum / 1MB

if ($logSizeMB -gt 1000) {
    Send-MailMessage -To $AlertEmail -Subject "ALERT: Container logs ${logSizeMB}MB" `
        -Body "Container logs total ${logSizeMB}MB. Configure log rotation."
}

# Log results
Add-Content "C:\ShopDB\logs\disk-monitor.log" "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),${disk.'Used%'},${dbSizeGB},${logSizeMB}"
```

**Task Scheduler** (daily at 6 AM):
```powershell
$action = New-ScheduledTaskAction -Execute "powershell.exe" `
    -Argument "-File C:\ShopDB\scripts\monitor-disk-space.ps1"
$trigger = New-ScheduledTaskTrigger -Daily -At 6am
Register-ScheduledTask -TaskName "ShopDB-DiskMonitor" -Action $action -Trigger $trigger
```

### Priority 3: Production Stability (bleeding-edge runtime)

#### ☐ Downgrade Python to 3.11 LTS (8 hours)
**Owner:** Developer  
**Effort:** 2 hours rebuild + 4 hours pilot + 2 hours bundle

**Steps:**
1. Edit `Dockerfile` line 101:
```dockerfile
# FROM python:3.14-slim
FROM python:3.11-slim
```

2. Edit `Dockerfile.prebuilt` line 1:
```dockerfile
# FROM python:3.14-slim
FROM python:3.11-slim
```

3. Rebuild image:
```bash
docker compose -f docker-compose.prebuilt.yml build --no-cache
```

4. Test on pilot site for 1 week:
   - Deploy to ops environment
   - Monitor for runtime errors
   - Verify all plugins load
   - Run full test suite: `pytest -v`

5. Create bundle for production:
```bash
.\pull-shopdb-bundle.cmd wj
```

6. Deploy to production sites

**Validation:**
```bash
# Verify Python version in container
docker compose -f docker-compose.prebuilt.yml run --rm api python --version
# Should output: Python 3.11.x

# Check for deprecation warnings in logs
docker compose -f docker-compose.prebuilt.yml logs api | grep -i deprecated
```

---

## MONITORING SETUP (First Week)

### Daily Checks (5 minutes)

#### ☐ Endpoint Health Dashboard
```sql
-- Check endpoint reporting status
SELECT 
  COUNT(*) as total_endpoints,
  SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) as reported_24h,
  ROUND(100.0 * SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_healthy
FROM assets
WHERE assettype = 'endpoint';
```
**Alert if:** pct_healthy < 95%

#### ☐ Backup Verification
```bash
# Check last backup timestamp
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots --last

# Should show snapshot within last 24 hours
```
**Alert if:** No snapshot in last 24 hours

#### ☐ Docker Service Health
```bash
docker ps --filter name=shopdb --format "table {{.Names}}\t{{.Status}}\t{{.State}}"
```
**Alert if:** Any service not "Up" and "healthy"

#### ☐ Database Disk Space
```powershell
# Quick check
docker volume inspect shopdb_db_data | ConvertFrom-Json | 
  Select-Object @{N='Mountpoint';E={$_.Mountpoint}}, 
                @{N='SizeGB';E={[math]::Round((Get-ChildItem $_.Mountpoint -Recurse | 
                  Measure-Object -Property Length -Sum).Sum / 1GB, 2)}}
```
**Alert if:** >40GB (or >85% of allocated disk)

#### ☐ Critical Application Errors
```bash
# Check for OperationalError, IntegrityError, 500 responses
docker compose -f docker-compose.prebuilt.yml logs --since 24h api | 
  grep -E "OperationalError|IntegrityError|HTTP/1.1\" 500"
```
**Alert if:** More than 10 errors in 24 hours

### Weekly Checks (15 minutes)

#### ☐ Audit Log Size
```sql
SELECT 
  COUNT(*) as total_rows,
  MIN(created) as oldest_entry,
  MAX(created) as newest_entry,
  DATEDIFF(MAX(created), MIN(created)) as retention_days
FROM auditlogs;
```
**Alert if:** total_rows > 1M and retention_days > 90

#### ☐ Failed Login Attempts
```sql
SELECT 
  username,
  failed_attempts,
  lastfailedlogin,
  isactive
FROM users
WHERE failed_attempts > 0
ORDER BY failed_attempts DESC
LIMIT 20;
```
**Alert if:** Any account with >10 failed attempts in 24 hours

#### ☐ Endpoint Events Accumulation
```sql
SELECT 
  COUNT(*) as total_events,
  MIN(eventtime) as oldest_event,
  MAX(eventtime) as newest_event
FROM endpointevents;
```
**Alert if:** total_events > 500K

#### ☐ Security Advisories
```bash
cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json
```
**Review:** HIGH severity findings in shipped components (exclude dev-only)

#### ☐ Database Performance
```sql
-- Check for slow queries (if slow query log enabled)
SELECT 
  query_time,
  lock_time,
  rows_examined,
  sql_text
FROM mysql.slow_log
WHERE query_time > 5
ORDER BY query_time DESC
LIMIT 10;
```
**Alert if:** Regular queries taking >5 seconds

### Monthly Checks (1 hour)

#### ☐ PAT Usage Audit
```sql
SELECT 
  t.tokenid,
  u.username,
  t.lastusedat,
  t.scope,
  DATEDIFF(NOW(), t.lastusedat) as days_since_use
FROM apitokens t
JOIN users u ON t.userid = u.userid
WHERE t.lastusedat < NOW() - INTERVAL 90 DAY
   OR t.lastusedat IS NULL
ORDER BY days_since_use DESC;
```
**Action:** Revoke stale tokens (not used in 90+ days)

#### ☐ Certificate Expiration Check
```bash
openssl x509 -in ./certs/server.crt -noout -enddate
```
**Alert if:** <60 days until expiry

#### ☐ Backup Restoration Test
```bash
# Restore to test environment
docker compose -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test

# Start test stack
cd /restore-test
docker compose -f docker-compose.prebuilt.yml up -d

# Verify database
docker compose exec api flask db current
# Should show: 7d51_endpoint_logcollections (head)
```
**Document:** Restore time, data integrity, any issues

#### ☐ Plugin Version Audit
```bash
# Compare plugin registry to disk versions
docker compose run --rm api python -c "
from shopdb.extensions import plugin_registry
for name, plugin in plugin_registry.plugins.items():
    print(f'{name}: {plugin.version}')
"
```
**Alert if:** Version mismatch between registry and MANIFEST.json

#### ☐ Configuration Drift Review
```bash
# Compare .env files across sites
# dev, ops, prod should have consistent:
# - CORS_ORIGINS
# - TRUSTED_PROXY_COUNT
# - Collector API keys
# - JWT secret keys
```
**Document:** Any drift and reason

### Quarterly Checks (4 hours)

#### ☐ Full DR Drill
**Scenario:** Complete site loss, restore from backup

**Steps:**
1. Simulate failure (stop all containers, optionally remove volumes)
2. Restore from latest backup
3. Redeploy containers
4. Verify 700 endpoints reconnect within 15 minutes
5. Spot-check all features:
   - User login
   - Asset search
   - Script execution
   - Report generation
6. Document restore time and issues

**Target:** Full recovery in <2 hours

#### ☐ Security Log Audit
```sql
-- Check for anomalous patterns
SELECT 
  userid,
  actiontype,
  COUNT(*) as action_count,
  MIN(created) as first_action,
  MAX(created) as last_action
FROM auditlogs
WHERE created > NOW() - INTERVAL 90 DAY
  AND (
    actiontype IN ('user.delete', 'permission.grant', 'setting.update')
    OR HOUR(created) NOT BETWEEN 6 AND 22  -- Off-hours access
  )
GROUP BY userid, actiontype
HAVING action_count > 5
ORDER BY action_count DESC;
```
**Review:** Off-hours access, bulk deletes, permission changes

#### ☐ Dependency Vulnerability Scan
```bash
# Run OSV.dev scan
cd /path/to/shopdb
python scripts/scan_sbom.py sbom.cdx.json > vuln-report.txt

# Review findings
grep -E "HIGH|CRITICAL" vuln-report.txt | grep -v "dev-only"
```
**Action:** Plan upgrade path for HIGH findings in shipped components

---

## AUTOMATED ALERTS (Configure Once)

### Critical Alerts (immediate notification)

#### ☐ Database Disk >85% Full
**Method:** Daily disk monitor script (see Priority 2)  
**Recipient:** Admin + Developer  
**Action:** Implement retention policies or expand disk

#### ☐ Host Disk >85% Full
**Method:** Daily disk monitor script  
**Recipient:** Admin  
**Action:** Clean logs, check Docker volumes, expand disk

#### ☐ Backup Failed or >24h Old
**Method:** Check restic last snapshot timestamp  
**Recipient:** Admin  
**Action:** Investigate backup service logs, verify disk space

#### ☐ >10 Endpoints Offline >48 Hours
**Method:** Query assets table daily  
**Recipient:** Admin  
**Action:** Check network connectivity, agent service status

#### ☐ Application Error Rate >1% of Requests
**Method:** Parse gunicorn access logs  
**Recipient:** Developer  
**Action:** Check Flask logs for exceptions, review recent changes

#### ☐ Certificate Expiring in <30 Days
**Method:** Daily openssl check  
**Recipient:** Admin  
**Action:** Generate new self-signed cert or renew

### Warning Alerts (daily summary)

#### ☐ Audit Log >1M Rows Without Retention
**Method:** Daily SQL query  
**Recipient:** Admin  
**Action:** Configure audit log purge job

#### ☐ Endpoint Events >500K Rows
**Method:** Daily SQL query  
**Recipient:** Admin  
**Action:** Configure event retention

#### ☐ Failed Login Rate >10/min from Single IP
**Method:** Parse Flask auth logs  
**Recipient:** Admin  
**Action:** Review for brute force attempt, consider IP block

---

## MAINTENANCE SCHEDULE

### Daily (5 min)
- Endpoint health dashboard check
- Backup verification
- Docker service health
- Database disk space
- Critical error review

### Weekly (15 min)
- Audit log size check
- Failed login review
- Endpoint events accumulation
- Security advisories scan
- Database performance metrics

### Monthly (1 hour)
- PAT usage audit
- Certificate expiration check
- Backup restoration test
- Plugin version audit
- Configuration drift review

### Quarterly (4 hours)
- Full DR drill
- Security log audit
- Dependency vulnerability scan

**Total Annual Scheduled Effort:**
- Daily: 365 * 5 min = 30 hours
- Weekly: 52 * 15 min = 13 hours
- Monthly: 12 * 1 hour = 12 hours
- Quarterly: 4 * 4 hours = 16 hours
- **Subtotal: 71 hours/year**

**Full Baseline (from analysis): 936 hours/year**
- Scheduled monitoring: 71 hours (8%)
- Planned maintenance: 600 hours (64%)
- Ad-hoc incident response: 265 hours (28%)

---

## DEPLOYMENT RUNBOOK

### Pre-Deployment Checklist

#### ☐ Create Backup
```bash
# Take manual snapshot before deploy
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic backup /app/instance --tag pre-deploy
```

#### ☐ Test Bundle on Pilot Site
- Deploy to ops environment first
- Verify all services start (db, migrate, api)
- Check migration logs for errors
- Spot-check core features
- Monitor for 24 hours

#### ☐ Review Migration Chain
```bash
# Check what migrations will run
docker compose run --rm api flask db heads
docker compose run --rm api flask db current
```

#### ☐ Verify Disk Space
```bash
# Ensure enough space for migration and new code
df -h /var/lib/docker
```

### Deployment Steps

#### 1. Stop Services
```bash
docker compose -f docker-compose.prebuilt.yml down
```

#### 2. Extract Bundle
```bash
# Transfer bundle across air gap
# Extract to deployment directory
tar -xzf shopdb-bundle-wj-<date>.tar.gz
```

#### 3. Pull Latest Code
```bash
cd /path/to/shopdb
git pull origin wj
# Or: git bundle unbundle ../shopdb-wj.bundle
```

#### 4. Rebuild Images
```bash
docker compose -f docker-compose.prebuilt.yml build --no-cache
```

#### 5. Start Stack (migrations run automatically)
```bash
docker compose -f docker-compose.prebuilt.yml up -d
```

#### 6. Monitor Startup
```bash
# Watch logs in real-time
docker compose -f docker-compose.prebuilt.yml logs -f migrate

# Verify migrations completed
docker compose -f docker-compose.prebuilt.yml logs migrate | tail -20

# Check services healthy
docker ps
```

#### 7. Verify Application
```bash
# Check database schema version
docker compose run --rm api flask db current

# Test core endpoints
curl https://localhost:5001/api/health
curl https://localhost:5001/api/assets/types

# Check UI loads
# Navigate to https://<hostname>:5173
```

### Rollback Procedure

#### If Migration Fails:

```bash
# Stop services
docker compose -f docker-compose.prebuilt.yml down

# Restore from pre-deploy backup
docker compose -f docker-compose.backup.yml run --rm backup restic restore <snapshot-id> --target /restore

# Copy restored instance/ directory
cp -r /restore/app/instance /app/instance

# Revert code to previous version
git checkout <previous-commit>

# Rebuild and start
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml up -d
```

#### If Application Fails Post-Deploy:

```bash
# Check logs first
docker compose logs api | grep -E "ERROR|CRITICAL"

# If unfixable, rollback code
git checkout <previous-commit>
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml restart api
```

### Post-Deployment Verification

#### ☐ All Services Healthy
```bash
docker ps --filter name=shopdb
# Should show: db (healthy), api (healthy), migrate (exited 0)
```

#### ☐ Endpoints Checking In
```sql
SELECT COUNT(*) FROM assets 
WHERE assettype='endpoint' 
  AND lastreportedat > NOW() - INTERVAL 30 MINUTE;
```
**Expect:** >95% of endpoints

#### ☐ No Critical Errors in Logs
```bash
docker compose logs --since 1h api | grep -i critical
```
**Expect:** Empty or expected warnings only

#### ☐ UI Loads and Functions
- Login works
- Asset search returns results
- Can navigate to endpoint details
- Scripts tab loads

#### ☐ Document Deployment
```markdown
## Deployment Log: 2026-09-11

- **Branch:** wj
- **Commit:** 0b252db7d2
- **Migrations:** 7d51_endpoint_logcollections (head)
- **Downtime:** 5 minutes
- **Issues:** None
- **Endpoint Recovery:** 98% within 15 minutes
```

---

## SKILL REQUIREMENTS

### Operator (20% of maintenance)
- Run daily monitoring scripts
- Verify backup completion
- Check disk space
- Restart services if needed
- Basic Windows/Docker troubleshooting

### Admin (25% of maintenance)
- Configure cron jobs / Task Scheduler
- Manage Docker Compose deployments
- User/permission management
- Review logs for security issues
- Certificate management

### Developer (40% of maintenance)
- Dependency upgrades
- Plugin updates
- Troubleshoot application errors
- Database performance tuning
- Implement retention policies
- Security patch development

### Architect (15% of maintenance)
- Major version upgrades
- DR drill planning
- Security audit response
- Capacity planning
- Architecture decisions

---

## CRITICAL COMPETENCIES REQUIRED

1. **Air-gap bundle workflow** - Create git bundles, transfer across three gaps, verify checksums
2. **Database migration troubleshooting** - Alembic per-plugin chains, PLUGIN_TABLE_OWNERS, MySQL DDL behavior
3. **Docker Compose production operations** - Multi-service stack, volume persistence, logging drivers
4. **Flask + SQLAlchemy performance tuning** - Connection pools, query optimization, retention policies
5. **Authentication and authorization** - JWT tokens, PATs, role permissions, collector key security
6. **Plugin architecture** - Dependency graph, contract versions, enable/disable coordination
7. **Security audit response** - Translate findings to remediation plans, prioritize risks

---

## KNOWN LIMITATIONS

### Cannot Be Automated
- Air-gap bundle transfer (physical media required)
- Certificate renewal (self-signed, manual process)
- Plugin dependency ordering (requires developer judgment)
- Security audit triage (requires threat modeling)

### Requires Developer Skills
- Migration failures (manual schema reconciliation)
- Plugin contract violations (code changes needed)
- Performance degradation (query optimization, indexing)
- Security vulnerability remediation (code patches)

### Incident Response Lag
- Bundle creation + 3 air gaps = 2-3 day minimum for critical fixes
- No rapid patching for zero-day vulnerabilities
- Cannot roll back failed migrations easily (MySQL DDL auto-commits)

---

## REFERENCES

- [[maintenance-footprint-baseline]] - Full 936-hour analysis
- [[security-audit-2026-09-11]] - Security findings driving Priority 1-3
- [[security-fix-implementation]] - Long-term security roadmap
- `docker-compose.backup.yml` - Backup service configuration
- `scripts/scan_sbom.py` - Vulnerability scanning tool