Clean up and organize the flask-shopdb repository for better maintainability and discoverability: ## Changes - Update .gitignore for build artifacts (publish/, data/, 122M total) - Reorganize .claude/ into 6 subdirectories with README navigation - Consolidate overlapping MCP setup guides into MCP-QUICK-START.md - Move deployment docs from root to docs/deployment/ - Archive dated status reports to .claude/archived/ - Delete temporary files (logs, backups, empty reports, ~50K) ## New Structure .claude/ organized into: - setup/ - MCP, VM, Hyper-V, GitHub setup guides + scripts (17 files) - plans/ - Feature plans and protocols (5 files) - maintenance/ - Deployment checklists and reports (4 files) - archived/ - Historical documentation (9 files) - skills/ - Claude Code automation (5 files) - workflows/ - Security audit workflow (1 file) docs/deployment/ created with 6 deployment guides: - LOCAL-DEV-SETUP.md, DOCKER-SETUP.md, DOCKER-PREBUILT-APPROACH.md - PRODUCTION-READINESS.md, DEPLOY-TO-VM.md, README.md Root directory cleaned to essential docs only: - CLAUDE.md, README.md, CONTRIBUTING.md, CHANGELOG.md Benefits: - Clear navigation via README.md in each subdirectory - No duplicate/overlapping documentation - Large artifacts properly gitignored (won't bloat repository) - Historical context preserved in archived/ See .claude/CLEANUP-SUMMARY.md for complete details. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
832 lines
22 KiB
Markdown
832 lines
22 KiB
Markdown
# ShopDB Maintenance & Deployment Checklist
|
|
**Generated:** 2026-09-11
|
|
**Source:** Maintenance Footprint Analysis (936 hrs/year baseline)
|
|
|
|
## Quick Reference
|
|
|
|
**Risk Profile:** HIGH
|
|
**Annual Maintenance:** 936 hours (0.45 FTE)
|
|
**Critical Risks:** 8
|
|
**Quick Wins Available:** 7 (22-35 hours total effort)
|
|
|
|
---
|
|
|
|
## IMMEDIATE ACTIONS (Next 30 Days)
|
|
|
|
### Priority 1: Data Growth Bomb (6 months to disk exhaustion)
|
|
|
|
#### ☐ Configure Audit Log Retention (4 hours)
|
|
**Owner:** Developer + Admin
|
|
**Effort:** 2 hours config + 2 hours testing
|
|
|
|
```bash
|
|
# Test on copy of production database first
|
|
docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40 --dry-run
|
|
|
|
# Add to crontab (daily at 2 AM)
|
|
0 2 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40
|
|
```
|
|
|
|
**Validation:**
|
|
- Before: `SELECT COUNT(*) FROM auditlogs;` (expect millions)
|
|
- After 30 days: Should stabilize under 1M rows
|
|
- Backup size reduction: 60-80%
|
|
|
|
#### ☐ Configure Endpoint Event Retention (8 hours)
|
|
**Owner:** Developer
|
|
**Effort:** 4 hours dev + 2 hours testing + 2 hours deploy
|
|
|
|
**Implementation:**
|
|
1. Add Flask CLI command to `plugins/endpoints/cli.py`:
|
|
```python
|
|
@endpoints_cli.command('prune-events')
|
|
@click.option('--days', default=90, help='Retain events for N days')
|
|
@click.option('--batch-size', default=10000, help='Delete batch size')
|
|
@click.option('--dry-run', is_flag=True, help='Show what would be deleted')
|
|
def prune_events(days, batch_size, dry_run):
|
|
"""Prune endpoint events older than N days."""
|
|
cutoff = datetime.utcnow() - timedelta(days=days)
|
|
|
|
if dry_run:
|
|
count = db.session.query(func.count(EndpointEvent.id)).filter(
|
|
EndpointEvent.eventtime < cutoff
|
|
).scalar()
|
|
click.echo(f"Would delete {count:,} events older than {cutoff.date()}")
|
|
return
|
|
|
|
deleted = 0
|
|
while True:
|
|
batch = db.session.query(EndpointEvent).filter(
|
|
EndpointEvent.eventtime < cutoff
|
|
).limit(batch_size).all()
|
|
|
|
if not batch:
|
|
break
|
|
|
|
for event in batch:
|
|
db.session.delete(event)
|
|
|
|
db.session.commit()
|
|
deleted += len(batch)
|
|
click.echo(f"Deleted {deleted:,} events...")
|
|
|
|
click.echo(f"Total deleted: {deleted:,} events")
|
|
```
|
|
|
|
2. Add to crontab (daily at 3 AM):
|
|
```bash
|
|
0 3 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask endpoints prune-events --days 90
|
|
```
|
|
|
|
**Validation:**
|
|
- Test on copy first: `flask endpoints prune-events --days 90 --dry-run`
|
|
- Monitor table size: `SELECT COUNT(*) FROM endpointevents;`
|
|
- Should stabilize under 500K rows
|
|
|
|
#### ☐ Configure Docker Log Rotation (2 hours)
|
|
**Owner:** Admin
|
|
**Effort:** 1 hour config + 1 hour verification
|
|
|
|
**Method 1: daemon.json (applies to all containers)**
|
|
```json
|
|
// C:\ProgramData\Docker\config\daemon.json
|
|
{
|
|
"log-driver": "json-file",
|
|
"log-opts": {
|
|
"max-size": "10m",
|
|
"max-file": "5"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Method 2: docker-compose.yml (per-service)**
|
|
```yaml
|
|
services:
|
|
api:
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "5"
|
|
```
|
|
|
|
**After changing daemon.json:**
|
|
```powershell
|
|
Restart-Service docker
|
|
docker compose -f docker-compose.prebuilt.yml restart
|
|
```
|
|
|
|
**Validation:**
|
|
```powershell
|
|
# Check log file sizes
|
|
Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
|
|
Select-Object Name, @{N='SizeMB';E={[math]::Round($_.Length/1MB,2)}} |
|
|
Sort-Object SizeMB -Descending
|
|
```
|
|
|
|
### Priority 2: Backup and Recovery (currently MISSING)
|
|
|
|
#### ☐ Enable Backup Profile (6 hours)
|
|
**Owner:** Admin + Developer
|
|
**Effort:** 3 hours setup + 3 hours restore test
|
|
|
|
**Setup:**
|
|
1. Generate secure backup password:
|
|
```powershell
|
|
# Generate 32-character password
|
|
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 32 | % {[char]$_})
|
|
```
|
|
|
|
2. Add to `.env`:
|
|
```bash
|
|
RESTIC_PASSWORD=<generated-password-here>
|
|
RESTIC_REPOSITORY=/backup/shopdb
|
|
BACKUP_RETENTION_DAYS=90
|
|
BACKUP_RETENTION_WEEKS=12
|
|
BACKUP_RETENTION_MONTHS=18
|
|
```
|
|
|
|
3. Enable backup service:
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
|
|
```
|
|
|
|
4. Initialize repository (first time only):
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic init
|
|
```
|
|
|
|
5. Add to crontab (daily at 1 AM):
|
|
```bash
|
|
0 1 * * * cd /path/to/shopdb && docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup
|
|
```
|
|
|
|
**Validation:**
|
|
```bash
|
|
# List snapshots
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots
|
|
|
|
# Test restore to temp location
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
|
|
```
|
|
|
|
**Quarterly Restore Test:**
|
|
1. Restore to test environment
|
|
2. Start services on test environment
|
|
3. Verify database integrity: `flask db current`
|
|
4. Spot-check data (assets, endpoints, users)
|
|
5. Run migrations to latest: `flask db upgrade`
|
|
6. Document restore time and any issues
|
|
|
|
#### ☐ Add Disk Space Monitoring (4 hours)
|
|
**Owner:** Admin
|
|
**Effort:** 2 hours script + 2 hours integration
|
|
|
|
**PowerShell Script** (`scripts/monitor-disk-space.ps1`):
|
|
```powershell
|
|
# Disk space monitoring for ShopDB
|
|
$HostDiskThreshold = 85
|
|
$DbVolumeThreshold = 40 # GB
|
|
$InstanceVolumeThreshold = 15 # GB
|
|
$AlertEmail = "shopdb-alerts@example.com"
|
|
|
|
# Check host disk
|
|
$disk = Get-PSDrive C | Select-Object @{N='Used%';E={[math]::Round(($_.Used/($_.Used+$_.Free))*100,1)}}
|
|
if ($disk.'Used%' -gt $HostDiskThreshold) {
|
|
Send-MailMessage -To $AlertEmail -Subject "ALERT: Host disk ${disk.'Used%'}% full" `
|
|
-Body "Host C: drive is ${disk.'Used%'}% full (threshold: ${HostDiskThreshold}%)"
|
|
}
|
|
|
|
# Check Docker volumes
|
|
$dbVolume = docker volume inspect shopdb_db_data | ConvertFrom-Json
|
|
$dbSizeGB = [math]::Round((Get-ChildItem $dbVolume.Mountpoint -Recurse |
|
|
Measure-Object -Property Length -Sum).Sum / 1GB, 2)
|
|
|
|
if ($dbSizeGB -gt $DbVolumeThreshold) {
|
|
Send-MailMessage -To $AlertEmail -Subject "ALERT: Database volume ${dbSizeGB}GB" `
|
|
-Body "db_data volume is ${dbSizeGB}GB (threshold: ${DbVolumeThreshold}GB)"
|
|
}
|
|
|
|
# Check container log sizes
|
|
$logSizeMB = (Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
|
|
Measure-Object -Property Length -Sum).Sum / 1MB
|
|
|
|
if ($logSizeMB -gt 1000) {
|
|
Send-MailMessage -To $AlertEmail -Subject "ALERT: Container logs ${logSizeMB}MB" `
|
|
-Body "Container logs total ${logSizeMB}MB. Configure log rotation."
|
|
}
|
|
|
|
# Log results
|
|
Add-Content "C:\ShopDB\logs\disk-monitor.log" "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),${disk.'Used%'},${dbSizeGB},${logSizeMB}"
|
|
```
|
|
|
|
**Task Scheduler** (daily at 6 AM):
|
|
```powershell
|
|
$action = New-ScheduledTaskAction -Execute "powershell.exe" `
|
|
-Argument "-File C:\ShopDB\scripts\monitor-disk-space.ps1"
|
|
$trigger = New-ScheduledTaskTrigger -Daily -At 6am
|
|
Register-ScheduledTask -TaskName "ShopDB-DiskMonitor" -Action $action -Trigger $trigger
|
|
```
|
|
|
|
### Priority 3: Production Stability (bleeding-edge runtime)
|
|
|
|
#### ☐ Downgrade Python to 3.11 LTS (8 hours)
|
|
**Owner:** Developer
|
|
**Effort:** 2 hours rebuild + 4 hours pilot + 2 hours bundle
|
|
|
|
**Steps:**
|
|
1. Edit `Dockerfile` line 101:
|
|
```dockerfile
|
|
# FROM python:3.14-slim
|
|
FROM python:3.11-slim
|
|
```
|
|
|
|
2. Edit `Dockerfile.prebuilt` line 1:
|
|
```dockerfile
|
|
# FROM python:3.14-slim
|
|
FROM python:3.11-slim
|
|
```
|
|
|
|
3. Rebuild image:
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml build --no-cache
|
|
```
|
|
|
|
4. Test on pilot site for 1 week:
|
|
- Deploy to ops environment
|
|
- Monitor for runtime errors
|
|
- Verify all plugins load
|
|
- Run full test suite: `pytest -v`
|
|
|
|
5. Create bundle for production:
|
|
```bash
|
|
.\pull-shopdb-bundle.cmd wj
|
|
```
|
|
|
|
6. Deploy to production sites
|
|
|
|
**Validation:**
|
|
```bash
|
|
# Verify Python version in container
|
|
docker compose -f docker-compose.prebuilt.yml run --rm api python --version
|
|
# Should output: Python 3.11.x
|
|
|
|
# Check for deprecation warnings in logs
|
|
docker compose -f docker-compose.prebuilt.yml logs api | grep -i deprecated
|
|
```
|
|
|
|
---
|
|
|
|
## MONITORING SETUP (First Week)
|
|
|
|
### Daily Checks (5 minutes)
|
|
|
|
#### ☐ Endpoint Health Dashboard
|
|
```sql
|
|
-- Check endpoint reporting status
|
|
SELECT
|
|
COUNT(*) as total_endpoints,
|
|
SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) as reported_24h,
|
|
ROUND(100.0 * SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_healthy
|
|
FROM assets
|
|
WHERE assettype = 'endpoint';
|
|
```
|
|
**Alert if:** pct_healthy < 95%
|
|
|
|
#### ☐ Backup Verification
|
|
```bash
|
|
# Check last backup timestamp
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots --last
|
|
|
|
# Should show snapshot within last 24 hours
|
|
```
|
|
**Alert if:** No snapshot in last 24 hours
|
|
|
|
#### ☐ Docker Service Health
|
|
```bash
|
|
docker ps --filter name=shopdb --format "table {{.Names}}\t{{.Status}}\t{{.State}}"
|
|
```
|
|
**Alert if:** Any service not "Up" and "healthy"
|
|
|
|
#### ☐ Database Disk Space
|
|
```powershell
|
|
# Quick check
|
|
docker volume inspect shopdb_db_data | ConvertFrom-Json |
|
|
Select-Object @{N='Mountpoint';E={$_.Mountpoint}},
|
|
@{N='SizeGB';E={[math]::Round((Get-ChildItem $_.Mountpoint -Recurse |
|
|
Measure-Object -Property Length -Sum).Sum / 1GB, 2)}}
|
|
```
|
|
**Alert if:** >40GB (or >85% of allocated disk)
|
|
|
|
#### ☐ Critical Application Errors
|
|
```bash
|
|
# Check for OperationalError, IntegrityError, 500 responses
|
|
docker compose -f docker-compose.prebuilt.yml logs --since 24h api |
|
|
grep -E "OperationalError|IntegrityError|HTTP/1.1\" 500"
|
|
```
|
|
**Alert if:** More than 10 errors in 24 hours
|
|
|
|
### Weekly Checks (15 minutes)
|
|
|
|
#### ☐ Audit Log Size
|
|
```sql
|
|
SELECT
|
|
COUNT(*) as total_rows,
|
|
MIN(created) as oldest_entry,
|
|
MAX(created) as newest_entry,
|
|
DATEDIFF(MAX(created), MIN(created)) as retention_days
|
|
FROM auditlogs;
|
|
```
|
|
**Alert if:** total_rows > 1M and retention_days > 90
|
|
|
|
#### ☐ Failed Login Attempts
|
|
```sql
|
|
SELECT
|
|
username,
|
|
failed_attempts,
|
|
lastfailedlogin,
|
|
isactive
|
|
FROM users
|
|
WHERE failed_attempts > 0
|
|
ORDER BY failed_attempts DESC
|
|
LIMIT 20;
|
|
```
|
|
**Alert if:** Any account with >10 failed attempts in 24 hours
|
|
|
|
#### ☐ Endpoint Events Accumulation
|
|
```sql
|
|
SELECT
|
|
COUNT(*) as total_events,
|
|
MIN(eventtime) as oldest_event,
|
|
MAX(eventtime) as newest_event
|
|
FROM endpointevents;
|
|
```
|
|
**Alert if:** total_events > 500K
|
|
|
|
#### ☐ Security Advisories
|
|
```bash
|
|
cd /path/to/shopdb
|
|
python scripts/scan_sbom.py sbom.cdx.json
|
|
```
|
|
**Review:** HIGH severity findings in shipped components (exclude dev-only)
|
|
|
|
#### ☐ Database Performance
|
|
```sql
|
|
-- Check for slow queries (if slow query log enabled)
|
|
SELECT
|
|
query_time,
|
|
lock_time,
|
|
rows_examined,
|
|
sql_text
|
|
FROM mysql.slow_log
|
|
WHERE query_time > 5
|
|
ORDER BY query_time DESC
|
|
LIMIT 10;
|
|
```
|
|
**Alert if:** Regular queries taking >5 seconds
|
|
|
|
### Monthly Checks (1 hour)
|
|
|
|
#### ☐ PAT Usage Audit
|
|
```sql
|
|
SELECT
|
|
t.tokenid,
|
|
u.username,
|
|
t.lastusedat,
|
|
t.scope,
|
|
DATEDIFF(NOW(), t.lastusedat) as days_since_use
|
|
FROM apitokens t
|
|
JOIN users u ON t.userid = u.userid
|
|
WHERE t.lastusedat < NOW() - INTERVAL 90 DAY
|
|
OR t.lastusedat IS NULL
|
|
ORDER BY days_since_use DESC;
|
|
```
|
|
**Action:** Revoke stale tokens (not used in 90+ days)
|
|
|
|
#### ☐ Certificate Expiration Check
|
|
```bash
|
|
openssl x509 -in ./certs/server.crt -noout -enddate
|
|
```
|
|
**Alert if:** <60 days until expiry
|
|
|
|
#### ☐ Backup Restoration Test
|
|
```bash
|
|
# Restore to test environment
|
|
docker compose -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
|
|
|
|
# Start test stack
|
|
cd /restore-test
|
|
docker compose -f docker-compose.prebuilt.yml up -d
|
|
|
|
# Verify database
|
|
docker compose exec api flask db current
|
|
# Should show: 7d51_endpoint_logcollections (head)
|
|
```
|
|
**Document:** Restore time, data integrity, any issues
|
|
|
|
#### ☐ Plugin Version Audit
|
|
```bash
|
|
# Compare plugin registry to disk versions
|
|
docker compose run --rm api python -c "
|
|
from shopdb.extensions import plugin_registry
|
|
for name, plugin in plugin_registry.plugins.items():
|
|
print(f'{name}: {plugin.version}')
|
|
"
|
|
```
|
|
**Alert if:** Version mismatch between registry and MANIFEST.json
|
|
|
|
#### ☐ Configuration Drift Review
|
|
```bash
|
|
# Compare .env files across sites
|
|
# dev, ops, prod should have consistent:
|
|
# - CORS_ORIGINS
|
|
# - TRUSTED_PROXY_COUNT
|
|
# - Collector API keys
|
|
# - JWT secret keys
|
|
```
|
|
**Document:** Any drift and reason
|
|
|
|
### Quarterly Checks (4 hours)
|
|
|
|
#### ☐ Full DR Drill
|
|
**Scenario:** Complete site loss, restore from backup
|
|
|
|
**Steps:**
|
|
1. Simulate failure (stop all containers, optionally remove volumes)
|
|
2. Restore from latest backup
|
|
3. Redeploy containers
|
|
4. Verify 700 endpoints reconnect within 15 minutes
|
|
5. Spot-check all features:
|
|
- User login
|
|
- Asset search
|
|
- Script execution
|
|
- Report generation
|
|
6. Document restore time and issues
|
|
|
|
**Target:** Full recovery in <2 hours
|
|
|
|
#### ☐ Security Log Audit
|
|
```sql
|
|
-- Check for anomalous patterns
|
|
SELECT
|
|
userid,
|
|
actiontype,
|
|
COUNT(*) as action_count,
|
|
MIN(created) as first_action,
|
|
MAX(created) as last_action
|
|
FROM auditlogs
|
|
WHERE created > NOW() - INTERVAL 90 DAY
|
|
AND (
|
|
actiontype IN ('user.delete', 'permission.grant', 'setting.update')
|
|
OR HOUR(created) NOT BETWEEN 6 AND 22 -- Off-hours access
|
|
)
|
|
GROUP BY userid, actiontype
|
|
HAVING action_count > 5
|
|
ORDER BY action_count DESC;
|
|
```
|
|
**Review:** Off-hours access, bulk deletes, permission changes
|
|
|
|
#### ☐ Dependency Vulnerability Scan
|
|
```bash
|
|
# Run OSV.dev scan
|
|
cd /path/to/shopdb
|
|
python scripts/scan_sbom.py sbom.cdx.json > vuln-report.txt
|
|
|
|
# Review findings
|
|
grep -E "HIGH|CRITICAL" vuln-report.txt | grep -v "dev-only"
|
|
```
|
|
**Action:** Plan upgrade path for HIGH findings in shipped components
|
|
|
|
---
|
|
|
|
## AUTOMATED ALERTS (Configure Once)
|
|
|
|
### Critical Alerts (immediate notification)
|
|
|
|
#### ☐ Database Disk >85% Full
|
|
**Method:** Daily disk monitor script (see Priority 2)
|
|
**Recipient:** Admin + Developer
|
|
**Action:** Implement retention policies or expand disk
|
|
|
|
#### ☐ Host Disk >85% Full
|
|
**Method:** Daily disk monitor script
|
|
**Recipient:** Admin
|
|
**Action:** Clean logs, check Docker volumes, expand disk
|
|
|
|
#### ☐ Backup Failed or >24h Old
|
|
**Method:** Check restic last snapshot timestamp
|
|
**Recipient:** Admin
|
|
**Action:** Investigate backup service logs, verify disk space
|
|
|
|
#### ☐ >10 Endpoints Offline >48 Hours
|
|
**Method:** Query assets table daily
|
|
**Recipient:** Admin
|
|
**Action:** Check network connectivity, agent service status
|
|
|
|
#### ☐ Application Error Rate >1% of Requests
|
|
**Method:** Parse gunicorn access logs
|
|
**Recipient:** Developer
|
|
**Action:** Check Flask logs for exceptions, review recent changes
|
|
|
|
#### ☐ Certificate Expiring in <30 Days
|
|
**Method:** Daily openssl check
|
|
**Recipient:** Admin
|
|
**Action:** Generate new self-signed cert or renew
|
|
|
|
### Warning Alerts (daily summary)
|
|
|
|
#### ☐ Audit Log >1M Rows Without Retention
|
|
**Method:** Daily SQL query
|
|
**Recipient:** Admin
|
|
**Action:** Configure audit log purge job
|
|
|
|
#### ☐ Endpoint Events >500K Rows
|
|
**Method:** Daily SQL query
|
|
**Recipient:** Admin
|
|
**Action:** Configure event retention
|
|
|
|
#### ☐ Failed Login Rate >10/min from Single IP
|
|
**Method:** Parse Flask auth logs
|
|
**Recipient:** Admin
|
|
**Action:** Review for brute force attempt, consider IP block
|
|
|
|
---
|
|
|
|
## MAINTENANCE SCHEDULE
|
|
|
|
### Daily (5 min)
|
|
- Endpoint health dashboard check
|
|
- Backup verification
|
|
- Docker service health
|
|
- Database disk space
|
|
- Critical error review
|
|
|
|
### Weekly (15 min)
|
|
- Audit log size check
|
|
- Failed login review
|
|
- Endpoint events accumulation
|
|
- Security advisories scan
|
|
- Database performance metrics
|
|
|
|
### Monthly (1 hour)
|
|
- PAT usage audit
|
|
- Certificate expiration check
|
|
- Backup restoration test
|
|
- Plugin version audit
|
|
- Configuration drift review
|
|
|
|
### Quarterly (4 hours)
|
|
- Full DR drill
|
|
- Security log audit
|
|
- Dependency vulnerability scan
|
|
|
|
**Total Annual Scheduled Effort:**
|
|
- Daily: 365 * 5 min = 30 hours
|
|
- Weekly: 52 * 15 min = 13 hours
|
|
- Monthly: 12 * 1 hour = 12 hours
|
|
- Quarterly: 4 * 4 hours = 16 hours
|
|
- **Subtotal: 71 hours/year**
|
|
|
|
**Full Baseline (from analysis): 936 hours/year**
|
|
- Scheduled monitoring: 71 hours (8%)
|
|
- Planned maintenance: 600 hours (64%)
|
|
- Ad-hoc incident response: 265 hours (28%)
|
|
|
|
---
|
|
|
|
## DEPLOYMENT RUNBOOK
|
|
|
|
### Pre-Deployment Checklist
|
|
|
|
#### ☐ Create Backup
|
|
```bash
|
|
# Take manual snapshot before deploy
|
|
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic backup /app/instance --tag pre-deploy
|
|
```
|
|
|
|
#### ☐ Test Bundle on Pilot Site
|
|
- Deploy to ops environment first
|
|
- Verify all services start (db, migrate, api)
|
|
- Check migration logs for errors
|
|
- Spot-check core features
|
|
- Monitor for 24 hours
|
|
|
|
#### ☐ Review Migration Chain
|
|
```bash
|
|
# Check what migrations will run
|
|
docker compose run --rm api flask db heads
|
|
docker compose run --rm api flask db current
|
|
```
|
|
|
|
#### ☐ Verify Disk Space
|
|
```bash
|
|
# Ensure enough space for migration and new code
|
|
df -h /var/lib/docker
|
|
```
|
|
|
|
### Deployment Steps
|
|
|
|
#### 1. Stop Services
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml down
|
|
```
|
|
|
|
#### 2. Extract Bundle
|
|
```bash
|
|
# Transfer bundle across air gap
|
|
# Extract to deployment directory
|
|
tar -xzf shopdb-bundle-wj-<date>.tar.gz
|
|
```
|
|
|
|
#### 3. Pull Latest Code
|
|
```bash
|
|
cd /path/to/shopdb
|
|
git pull origin wj
|
|
# Or: git bundle unbundle ../shopdb-wj.bundle
|
|
```
|
|
|
|
#### 4. Rebuild Images
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml build --no-cache
|
|
```
|
|
|
|
#### 5. Start Stack (migrations run automatically)
|
|
```bash
|
|
docker compose -f docker-compose.prebuilt.yml up -d
|
|
```
|
|
|
|
#### 6. Monitor Startup
|
|
```bash
|
|
# Watch logs in real-time
|
|
docker compose -f docker-compose.prebuilt.yml logs -f migrate
|
|
|
|
# Verify migrations completed
|
|
docker compose -f docker-compose.prebuilt.yml logs migrate | tail -20
|
|
|
|
# Check services healthy
|
|
docker ps
|
|
```
|
|
|
|
#### 7. Verify Application
|
|
```bash
|
|
# Check database schema version
|
|
docker compose run --rm api flask db current
|
|
|
|
# Test core endpoints
|
|
curl https://localhost:5001/api/health
|
|
curl https://localhost:5001/api/assets/types
|
|
|
|
# Check UI loads
|
|
# Navigate to https://<hostname>:5173
|
|
```
|
|
|
|
### Rollback Procedure
|
|
|
|
#### If Migration Fails:
|
|
|
|
```bash
|
|
# Stop services
|
|
docker compose -f docker-compose.prebuilt.yml down
|
|
|
|
# Restore from pre-deploy backup
|
|
docker compose -f docker-compose.backup.yml run --rm backup restic restore <snapshot-id> --target /restore
|
|
|
|
# Copy restored instance/ directory
|
|
cp -r /restore/app/instance /app/instance
|
|
|
|
# Revert code to previous version
|
|
git checkout <previous-commit>
|
|
|
|
# Rebuild and start
|
|
docker compose -f docker-compose.prebuilt.yml build
|
|
docker compose -f docker-compose.prebuilt.yml up -d
|
|
```
|
|
|
|
#### If Application Fails Post-Deploy:
|
|
|
|
```bash
|
|
# Check logs first
|
|
docker compose logs api | grep -E "ERROR|CRITICAL"
|
|
|
|
# If unfixable, rollback code
|
|
git checkout <previous-commit>
|
|
docker compose -f docker-compose.prebuilt.yml build
|
|
docker compose -f docker-compose.prebuilt.yml restart api
|
|
```
|
|
|
|
### Post-Deployment Verification
|
|
|
|
#### ☐ All Services Healthy
|
|
```bash
|
|
docker ps --filter name=shopdb
|
|
# Should show: db (healthy), api (healthy), migrate (exited 0)
|
|
```
|
|
|
|
#### ☐ Endpoints Checking In
|
|
```sql
|
|
SELECT COUNT(*) FROM assets
|
|
WHERE assettype='endpoint'
|
|
AND lastreportedat > NOW() - INTERVAL 30 MINUTE;
|
|
```
|
|
**Expect:** >95% of endpoints
|
|
|
|
#### ☐ No Critical Errors in Logs
|
|
```bash
|
|
docker compose logs --since 1h api | grep -i critical
|
|
```
|
|
**Expect:** Empty or expected warnings only
|
|
|
|
#### ☐ UI Loads and Functions
|
|
- Login works
|
|
- Asset search returns results
|
|
- Can navigate to endpoint details
|
|
- Scripts tab loads
|
|
|
|
#### ☐ Document Deployment
|
|
```markdown
|
|
## Deployment Log: 2026-09-11
|
|
|
|
- **Branch:** wj
|
|
- **Commit:** 0b252db7d2
|
|
- **Migrations:** 7d51_endpoint_logcollections (head)
|
|
- **Downtime:** 5 minutes
|
|
- **Issues:** None
|
|
- **Endpoint Recovery:** 98% within 15 minutes
|
|
```
|
|
|
|
---
|
|
|
|
## SKILL REQUIREMENTS
|
|
|
|
### Operator (20% of maintenance)
|
|
- Run daily monitoring scripts
|
|
- Verify backup completion
|
|
- Check disk space
|
|
- Restart services if needed
|
|
- Basic Windows/Docker troubleshooting
|
|
|
|
### Admin (25% of maintenance)
|
|
- Configure cron jobs / Task Scheduler
|
|
- Manage Docker Compose deployments
|
|
- User/permission management
|
|
- Review logs for security issues
|
|
- Certificate management
|
|
|
|
### Developer (40% of maintenance)
|
|
- Dependency upgrades
|
|
- Plugin updates
|
|
- Troubleshoot application errors
|
|
- Database performance tuning
|
|
- Implement retention policies
|
|
- Security patch development
|
|
|
|
### Architect (15% of maintenance)
|
|
- Major version upgrades
|
|
- DR drill planning
|
|
- Security audit response
|
|
- Capacity planning
|
|
- Architecture decisions
|
|
|
|
---
|
|
|
|
## CRITICAL COMPETENCIES REQUIRED
|
|
|
|
1. **Air-gap bundle workflow** - Create git bundles, transfer across three gaps, verify checksums
|
|
2. **Database migration troubleshooting** - Alembic per-plugin chains, PLUGIN_TABLE_OWNERS, MySQL DDL behavior
|
|
3. **Docker Compose production operations** - Multi-service stack, volume persistence, logging drivers
|
|
4. **Flask + SQLAlchemy performance tuning** - Connection pools, query optimization, retention policies
|
|
5. **Authentication and authorization** - JWT tokens, PATs, role permissions, collector key security
|
|
6. **Plugin architecture** - Dependency graph, contract versions, enable/disable coordination
|
|
7. **Security audit response** - Translate findings to remediation plans, prioritize risks
|
|
|
|
---
|
|
|
|
## KNOWN LIMITATIONS
|
|
|
|
### Cannot Be Automated
|
|
- Air-gap bundle transfer (physical media required)
|
|
- Certificate renewal (self-signed, manual process)
|
|
- Plugin dependency ordering (requires developer judgment)
|
|
- Security audit triage (requires threat modeling)
|
|
|
|
### Requires Developer Skills
|
|
- Migration failures (manual schema reconciliation)
|
|
- Plugin contract violations (code changes needed)
|
|
- Performance degradation (query optimization, indexing)
|
|
- Security vulnerability remediation (code patches)
|
|
|
|
### Incident Response Lag
|
|
- Bundle creation + 3 air gaps = 2-3 day minimum for critical fixes
|
|
- No rapid patching for zero-day vulnerabilities
|
|
- Cannot roll back failed migrations easily (MySQL DDL auto-commits)
|
|
|
|
---
|
|
|
|
## REFERENCES
|
|
|
|
- [[maintenance-footprint-baseline]] - Full 936-hour analysis
|
|
- [[security-audit-2026-09-11]] - Security findings driving Priority 1-3
|
|
- [[security-fix-implementation]] - Long-term security roadmap
|
|
- `docker-compose.backup.yml` - Backup service configuration
|
|
- `scripts/scan_sbom.py` - Vulnerability scanning tool
|