Reorganize repository structure and clean up documentation
Clean up and organize the flask-shopdb repository for better maintainability and discoverability: ## Changes - Update .gitignore for build artifacts (publish/, data/, 122M total) - Reorganize .claude/ into 6 subdirectories with README navigation - Consolidate overlapping MCP setup guides into MCP-QUICK-START.md - Move deployment docs from root to docs/deployment/ - Archive dated status reports to .claude/archived/ - Delete temporary files (logs, backups, empty reports, ~50K) ## New Structure .claude/ organized into: - setup/ - MCP, VM, Hyper-V, GitHub setup guides + scripts (17 files) - plans/ - Feature plans and protocols (5 files) - maintenance/ - Deployment checklists and reports (4 files) - archived/ - Historical documentation (9 files) - skills/ - Claude Code automation (5 files) - workflows/ - Security audit workflow (1 file) docs/deployment/ created with 6 deployment guides: - LOCAL-DEV-SETUP.md, DOCKER-SETUP.md, DOCKER-PREBUILT-APPROACH.md - PRODUCTION-READINESS.md, DEPLOY-TO-VM.md, README.md Root directory cleaned to essential docs only: - CLAUDE.md, README.md, CONTRIBUTING.md, CHANGELOG.md Benefits: - Clear navigation via README.md in each subdirectory - No duplicate/overlapping documentation - Large artifacts properly gitignored (won't bloat repository) - Historical context preserved in archived/ See .claude/CLEANUP-SUMMARY.md for complete details. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Proudlock
831
.claude/maintenance/MAINTENANCE-DEPLOYMENT-CHECKLIST.md
Normal file
831
.claude/maintenance/MAINTENANCE-DEPLOYMENT-CHECKLIST.md
Normal file
@@ -0,0 +1,831 @@
|
||||
# ShopDB Maintenance & Deployment Checklist
|
||||
**Generated:** 2026-09-11
|
||||
**Source:** Maintenance Footprint Analysis (936 hrs/year baseline)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
**Risk Profile:** HIGH
|
||||
**Annual Maintenance:** 936 hours (0.45 FTE)
|
||||
**Critical Risks:** 8
|
||||
**Quick Wins Available:** 7 (22-35 hours total effort)
|
||||
|
||||
---
|
||||
|
||||
## IMMEDIATE ACTIONS (Next 30 Days)
|
||||
|
||||
### Priority 1: Data Growth Bomb (6 months to disk exhaustion)
|
||||
|
||||
#### ☐ Configure Audit Log Retention (4 hours)
|
||||
**Owner:** Developer + Admin
|
||||
**Effort:** 2 hours config + 2 hours testing
|
||||
|
||||
```bash
|
||||
# Test on copy of production database first
|
||||
docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40 --dry-run
|
||||
|
||||
# Add to crontab (daily at 2 AM)
|
||||
0 2 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask audit purge --days 90 --batch-size 5000 --max-batches 40
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- Before: `SELECT COUNT(*) FROM auditlogs;` (expect millions)
|
||||
- After 30 days: Should stabilize under 1M rows
|
||||
- Backup size reduction: 60-80%
|
||||
|
||||
#### ☐ Configure Endpoint Event Retention (8 hours)
|
||||
**Owner:** Developer
|
||||
**Effort:** 4 hours dev + 2 hours testing + 2 hours deploy
|
||||
|
||||
**Implementation:**
|
||||
1. Add Flask CLI command to `plugins/endpoints/cli.py`:
|
||||
```python
|
||||
@endpoints_cli.command('prune-events')
|
||||
@click.option('--days', default=90, help='Retain events for N days')
|
||||
@click.option('--batch-size', default=10000, help='Delete batch size')
|
||||
@click.option('--dry-run', is_flag=True, help='Show what would be deleted')
|
||||
def prune_events(days, batch_size, dry_run):
|
||||
"""Prune endpoint events older than N days."""
|
||||
cutoff = datetime.utcnow() - timedelta(days=days)
|
||||
|
||||
if dry_run:
|
||||
count = db.session.query(func.count(EndpointEvent.id)).filter(
|
||||
EndpointEvent.eventtime < cutoff
|
||||
).scalar()
|
||||
click.echo(f"Would delete {count:,} events older than {cutoff.date()}")
|
||||
return
|
||||
|
||||
deleted = 0
|
||||
while True:
|
||||
batch = db.session.query(EndpointEvent).filter(
|
||||
EndpointEvent.eventtime < cutoff
|
||||
).limit(batch_size).all()
|
||||
|
||||
if not batch:
|
||||
break
|
||||
|
||||
for event in batch:
|
||||
db.session.delete(event)
|
||||
|
||||
db.session.commit()
|
||||
deleted += len(batch)
|
||||
click.echo(f"Deleted {deleted:,} events...")
|
||||
|
||||
click.echo(f"Total deleted: {deleted:,} events")
|
||||
```
|
||||
|
||||
2. Add to crontab (daily at 3 AM):
|
||||
```bash
|
||||
0 3 * * * docker compose -f docker-compose.prebuilt.yml run --rm api flask endpoints prune-events --days 90
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
- Test on copy first: `flask endpoints prune-events --days 90 --dry-run`
|
||||
- Monitor table size: `SELECT COUNT(*) FROM endpointevents;`
|
||||
- Should stabilize under 500K rows
|
||||
|
||||
#### ☐ Configure Docker Log Rotation (2 hours)
|
||||
**Owner:** Admin
|
||||
**Effort:** 1 hour config + 1 hour verification
|
||||
|
||||
**Method 1: daemon.json (applies to all containers)**
|
||||
```json
|
||||
// C:\ProgramData\Docker\config\daemon.json
|
||||
{
|
||||
"log-driver": "json-file",
|
||||
"log-opts": {
|
||||
"max-size": "10m",
|
||||
"max-file": "5"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Method 2: docker-compose.yml (per-service)**
|
||||
```yaml
|
||||
services:
|
||||
api:
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "5"
|
||||
```
|
||||
|
||||
**After changing daemon.json:**
|
||||
```powershell
|
||||
Restart-Service docker
|
||||
docker compose -f docker-compose.prebuilt.yml restart
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
```powershell
|
||||
# Check log file sizes
|
||||
Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
|
||||
Select-Object Name, @{N='SizeMB';E={[math]::Round($_.Length/1MB,2)}} |
|
||||
Sort-Object SizeMB -Descending
|
||||
```
|
||||
|
||||
### Priority 2: Backup and Recovery (currently MISSING)
|
||||
|
||||
#### ☐ Enable Backup Profile (6 hours)
|
||||
**Owner:** Admin + Developer
|
||||
**Effort:** 3 hours setup + 3 hours restore test
|
||||
|
||||
**Setup:**
|
||||
1. Generate secure backup password:
|
||||
```powershell
|
||||
# Generate 32-character password
|
||||
-join ((48..57) + (65..90) + (97..122) | Get-Random -Count 32 | % {[char]$_})
|
||||
```
|
||||
|
||||
2. Add to `.env`:
|
||||
```bash
|
||||
RESTIC_PASSWORD=<generated-password-here>
|
||||
RESTIC_REPOSITORY=/backup/shopdb
|
||||
BACKUP_RETENTION_DAYS=90
|
||||
BACKUP_RETENTION_WEEKS=12
|
||||
BACKUP_RETENTION_MONTHS=18
|
||||
```
|
||||
|
||||
3. Enable backup service:
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
|
||||
```
|
||||
|
||||
4. Initialize repository (first time only):
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic init
|
||||
```
|
||||
|
||||
5. Add to crontab (daily at 1 AM):
|
||||
```bash
|
||||
0 1 * * * cd /path/to/shopdb && docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup
|
||||
```
|
||||
|
||||
**Validation:**
|
||||
```bash
|
||||
# List snapshots
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots
|
||||
|
||||
# Test restore to temp location
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
|
||||
```
|
||||
|
||||
**Quarterly Restore Test:**
|
||||
1. Restore to test environment
|
||||
2. Start services on test environment
|
||||
3. Verify database integrity: `flask db current`
|
||||
4. Spot-check data (assets, endpoints, users)
|
||||
5. Run migrations to latest: `flask db upgrade`
|
||||
6. Document restore time and any issues
|
||||
|
||||
#### ☐ Add Disk Space Monitoring (4 hours)
|
||||
**Owner:** Admin
|
||||
**Effort:** 2 hours script + 2 hours integration
|
||||
|
||||
**PowerShell Script** (`scripts/monitor-disk-space.ps1`):
|
||||
```powershell
|
||||
# Disk space monitoring for ShopDB
|
||||
$HostDiskThreshold = 85
|
||||
$DbVolumeThreshold = 40 # GB
|
||||
$InstanceVolumeThreshold = 15 # GB
|
||||
$AlertEmail = "shopdb-alerts@example.com"
|
||||
|
||||
# Check host disk
|
||||
$disk = Get-PSDrive C | Select-Object @{N='Used%';E={[math]::Round(($_.Used/($_.Used+$_.Free))*100,1)}}
|
||||
if ($disk.'Used%' -gt $HostDiskThreshold) {
|
||||
Send-MailMessage -To $AlertEmail -Subject "ALERT: Host disk ${disk.'Used%'}% full" `
|
||||
-Body "Host C: drive is ${disk.'Used%'}% full (threshold: ${HostDiskThreshold}%)"
|
||||
}
|
||||
|
||||
# Check Docker volumes
|
||||
$dbVolume = docker volume inspect shopdb_db_data | ConvertFrom-Json
|
||||
$dbSizeGB = [math]::Round((Get-ChildItem $dbVolume.Mountpoint -Recurse |
|
||||
Measure-Object -Property Length -Sum).Sum / 1GB, 2)
|
||||
|
||||
if ($dbSizeGB -gt $DbVolumeThreshold) {
|
||||
Send-MailMessage -To $AlertEmail -Subject "ALERT: Database volume ${dbSizeGB}GB" `
|
||||
-Body "db_data volume is ${dbSizeGB}GB (threshold: ${DbVolumeThreshold}GB)"
|
||||
}
|
||||
|
||||
# Check container log sizes
|
||||
$logSizeMB = (Get-ChildItem "C:\ProgramData\Docker\containers\*\*-json.log" |
|
||||
Measure-Object -Property Length -Sum).Sum / 1MB
|
||||
|
||||
if ($logSizeMB -gt 1000) {
|
||||
Send-MailMessage -To $AlertEmail -Subject "ALERT: Container logs ${logSizeMB}MB" `
|
||||
-Body "Container logs total ${logSizeMB}MB. Configure log rotation."
|
||||
}
|
||||
|
||||
# Log results
|
||||
Add-Content "C:\ShopDB\logs\disk-monitor.log" "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss'),${disk.'Used%'},${dbSizeGB},${logSizeMB}"
|
||||
```
|
||||
|
||||
**Task Scheduler** (daily at 6 AM):
|
||||
```powershell
|
||||
$action = New-ScheduledTaskAction -Execute "powershell.exe" `
|
||||
-Argument "-File C:\ShopDB\scripts\monitor-disk-space.ps1"
|
||||
$trigger = New-ScheduledTaskTrigger -Daily -At 6am
|
||||
Register-ScheduledTask -TaskName "ShopDB-DiskMonitor" -Action $action -Trigger $trigger
|
||||
```
|
||||
|
||||
### Priority 3: Production Stability (bleeding-edge runtime)
|
||||
|
||||
#### ☐ Downgrade Python to 3.11 LTS (8 hours)
|
||||
**Owner:** Developer
|
||||
**Effort:** 2 hours rebuild + 4 hours pilot + 2 hours bundle
|
||||
|
||||
**Steps:**
|
||||
1. Edit `Dockerfile` line 101:
|
||||
```dockerfile
|
||||
# FROM python:3.14-slim
|
||||
FROM python:3.11-slim
|
||||
```
|
||||
|
||||
2. Edit `Dockerfile.prebuilt` line 1:
|
||||
```dockerfile
|
||||
# FROM python:3.14-slim
|
||||
FROM python:3.11-slim
|
||||
```
|
||||
|
||||
3. Rebuild image:
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml build --no-cache
|
||||
```
|
||||
|
||||
4. Test on pilot site for 1 week:
|
||||
- Deploy to ops environment
|
||||
- Monitor for runtime errors
|
||||
- Verify all plugins load
|
||||
- Run full test suite: `pytest -v`
|
||||
|
||||
5. Create bundle for production:
|
||||
```bash
|
||||
.\pull-shopdb-bundle.cmd wj
|
||||
```
|
||||
|
||||
6. Deploy to production sites
|
||||
|
||||
**Validation:**
|
||||
```bash
|
||||
# Verify Python version in container
|
||||
docker compose -f docker-compose.prebuilt.yml run --rm api python --version
|
||||
# Should output: Python 3.11.x
|
||||
|
||||
# Check for deprecation warnings in logs
|
||||
docker compose -f docker-compose.prebuilt.yml logs api | grep -i deprecated
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MONITORING SETUP (First Week)
|
||||
|
||||
### Daily Checks (5 minutes)
|
||||
|
||||
#### ☐ Endpoint Health Dashboard
|
||||
```sql
|
||||
-- Check endpoint reporting status
|
||||
SELECT
|
||||
COUNT(*) as total_endpoints,
|
||||
SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) as reported_24h,
|
||||
ROUND(100.0 * SUM(CASE WHEN lastreportedat > NOW() - INTERVAL 24 HOUR THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_healthy
|
||||
FROM assets
|
||||
WHERE assettype = 'endpoint';
|
||||
```
|
||||
**Alert if:** pct_healthy < 95%
|
||||
|
||||
#### ☐ Backup Verification
|
||||
```bash
|
||||
# Check last backup timestamp
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic snapshots --last
|
||||
|
||||
# Should show snapshot within last 24 hours
|
||||
```
|
||||
**Alert if:** No snapshot in last 24 hours
|
||||
|
||||
#### ☐ Docker Service Health
|
||||
```bash
|
||||
docker ps --filter name=shopdb --format "table {{.Names}}\t{{.Status}}\t{{.State}}"
|
||||
```
|
||||
**Alert if:** Any service not "Up" and "healthy"
|
||||
|
||||
#### ☐ Database Disk Space
|
||||
```powershell
|
||||
# Quick check
|
||||
docker volume inspect shopdb_db_data | ConvertFrom-Json |
|
||||
Select-Object @{N='Mountpoint';E={$_.Mountpoint}},
|
||||
@{N='SizeGB';E={[math]::Round((Get-ChildItem $_.Mountpoint -Recurse |
|
||||
Measure-Object -Property Length -Sum).Sum / 1GB, 2)}}
|
||||
```
|
||||
**Alert if:** >40GB (or >85% of allocated disk)
|
||||
|
||||
#### ☐ Critical Application Errors
|
||||
```bash
|
||||
# Check for OperationalError, IntegrityError, 500 responses
|
||||
docker compose -f docker-compose.prebuilt.yml logs --since 24h api |
|
||||
grep -E "OperationalError|IntegrityError|HTTP/1.1\" 500"
|
||||
```
|
||||
**Alert if:** More than 10 errors in 24 hours
|
||||
|
||||
### Weekly Checks (15 minutes)
|
||||
|
||||
#### ☐ Audit Log Size
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) as total_rows,
|
||||
MIN(created) as oldest_entry,
|
||||
MAX(created) as newest_entry,
|
||||
DATEDIFF(MAX(created), MIN(created)) as retention_days
|
||||
FROM auditlogs;
|
||||
```
|
||||
**Alert if:** total_rows > 1M and retention_days > 90
|
||||
|
||||
#### ☐ Failed Login Attempts
|
||||
```sql
|
||||
SELECT
|
||||
username,
|
||||
failed_attempts,
|
||||
lastfailedlogin,
|
||||
isactive
|
||||
FROM users
|
||||
WHERE failed_attempts > 0
|
||||
ORDER BY failed_attempts DESC
|
||||
LIMIT 20;
|
||||
```
|
||||
**Alert if:** Any account with >10 failed attempts in 24 hours
|
||||
|
||||
#### ☐ Endpoint Events Accumulation
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) as total_events,
|
||||
MIN(eventtime) as oldest_event,
|
||||
MAX(eventtime) as newest_event
|
||||
FROM endpointevents;
|
||||
```
|
||||
**Alert if:** total_events > 500K
|
||||
|
||||
#### ☐ Security Advisories
|
||||
```bash
|
||||
cd /path/to/shopdb
|
||||
python scripts/scan_sbom.py sbom.cdx.json
|
||||
```
|
||||
**Review:** HIGH severity findings in shipped components (exclude dev-only)
|
||||
|
||||
#### ☐ Database Performance
|
||||
```sql
|
||||
-- Check for slow queries (if slow query log enabled)
|
||||
SELECT
|
||||
query_time,
|
||||
lock_time,
|
||||
rows_examined,
|
||||
sql_text
|
||||
FROM mysql.slow_log
|
||||
WHERE query_time > 5
|
||||
ORDER BY query_time DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
**Alert if:** Regular queries taking >5 seconds
|
||||
|
||||
### Monthly Checks (1 hour)
|
||||
|
||||
#### ☐ PAT Usage Audit
|
||||
```sql
|
||||
SELECT
|
||||
t.tokenid,
|
||||
u.username,
|
||||
t.lastusedat,
|
||||
t.scope,
|
||||
DATEDIFF(NOW(), t.lastusedat) as days_since_use
|
||||
FROM apitokens t
|
||||
JOIN users u ON t.userid = u.userid
|
||||
WHERE t.lastusedat < NOW() - INTERVAL 90 DAY
|
||||
OR t.lastusedat IS NULL
|
||||
ORDER BY days_since_use DESC;
|
||||
```
|
||||
**Action:** Revoke stale tokens (not used in 90+ days)
|
||||
|
||||
#### ☐ Certificate Expiration Check
|
||||
```bash
|
||||
openssl x509 -in ./certs/server.crt -noout -enddate
|
||||
```
|
||||
**Alert if:** <60 days until expiry
|
||||
|
||||
#### ☐ Backup Restoration Test
|
||||
```bash
|
||||
# Restore to test environment
|
||||
docker compose -f docker-compose.backup.yml run --rm backup restic restore latest --target /restore-test
|
||||
|
||||
# Start test stack
|
||||
cd /restore-test
|
||||
docker compose -f docker-compose.prebuilt.yml up -d
|
||||
|
||||
# Verify database
|
||||
docker compose exec api flask db current
|
||||
# Should show: 7d51_endpoint_logcollections (head)
|
||||
```
|
||||
**Document:** Restore time, data integrity, any issues
|
||||
|
||||
#### ☐ Plugin Version Audit
|
||||
```bash
|
||||
# Compare plugin registry to disk versions
|
||||
docker compose run --rm api python -c "
|
||||
from shopdb.extensions import plugin_registry
|
||||
for name, plugin in plugin_registry.plugins.items():
|
||||
print(f'{name}: {plugin.version}')
|
||||
"
|
||||
```
|
||||
**Alert if:** Version mismatch between registry and MANIFEST.json
|
||||
|
||||
#### ☐ Configuration Drift Review
|
||||
```bash
|
||||
# Compare .env files across sites
|
||||
# dev, ops, prod should have consistent:
|
||||
# - CORS_ORIGINS
|
||||
# - TRUSTED_PROXY_COUNT
|
||||
# - Collector API keys
|
||||
# - JWT secret keys
|
||||
```
|
||||
**Document:** Any drift and reason
|
||||
|
||||
### Quarterly Checks (4 hours)
|
||||
|
||||
#### ☐ Full DR Drill
|
||||
**Scenario:** Complete site loss, restore from backup
|
||||
|
||||
**Steps:**
|
||||
1. Simulate failure (stop all containers, optionally remove volumes)
|
||||
2. Restore from latest backup
|
||||
3. Redeploy containers
|
||||
4. Verify 700 endpoints reconnect within 15 minutes
|
||||
5. Spot-check all features:
|
||||
- User login
|
||||
- Asset search
|
||||
- Script execution
|
||||
- Report generation
|
||||
6. Document restore time and issues
|
||||
|
||||
**Target:** Full recovery in <2 hours
|
||||
|
||||
#### ☐ Security Log Audit
|
||||
```sql
|
||||
-- Check for anomalous patterns
|
||||
SELECT
|
||||
userid,
|
||||
actiontype,
|
||||
COUNT(*) as action_count,
|
||||
MIN(created) as first_action,
|
||||
MAX(created) as last_action
|
||||
FROM auditlogs
|
||||
WHERE created > NOW() - INTERVAL 90 DAY
|
||||
AND (
|
||||
actiontype IN ('user.delete', 'permission.grant', 'setting.update')
|
||||
OR HOUR(created) NOT BETWEEN 6 AND 22 -- Off-hours access
|
||||
)
|
||||
GROUP BY userid, actiontype
|
||||
HAVING action_count > 5
|
||||
ORDER BY action_count DESC;
|
||||
```
|
||||
**Review:** Off-hours access, bulk deletes, permission changes
|
||||
|
||||
#### ☐ Dependency Vulnerability Scan
|
||||
```bash
|
||||
# Run OSV.dev scan
|
||||
cd /path/to/shopdb
|
||||
python scripts/scan_sbom.py sbom.cdx.json > vuln-report.txt
|
||||
|
||||
# Review findings
|
||||
grep -E "HIGH|CRITICAL" vuln-report.txt | grep -v "dev-only"
|
||||
```
|
||||
**Action:** Plan upgrade path for HIGH findings in shipped components
|
||||
|
||||
---
|
||||
|
||||
## AUTOMATED ALERTS (Configure Once)
|
||||
|
||||
### Critical Alerts (immediate notification)
|
||||
|
||||
#### ☐ Database Disk >85% Full
|
||||
**Method:** Daily disk monitor script (see Priority 2)
|
||||
**Recipient:** Admin + Developer
|
||||
**Action:** Implement retention policies or expand disk
|
||||
|
||||
#### ☐ Host Disk >85% Full
|
||||
**Method:** Daily disk monitor script
|
||||
**Recipient:** Admin
|
||||
**Action:** Clean logs, check Docker volumes, expand disk
|
||||
|
||||
#### ☐ Backup Failed or >24h Old
|
||||
**Method:** Check restic last snapshot timestamp
|
||||
**Recipient:** Admin
|
||||
**Action:** Investigate backup service logs, verify disk space
|
||||
|
||||
#### ☐ >10 Endpoints Offline >48 Hours
|
||||
**Method:** Query assets table daily
|
||||
**Recipient:** Admin
|
||||
**Action:** Check network connectivity, agent service status
|
||||
|
||||
#### ☐ Application Error Rate >1% of Requests
|
||||
**Method:** Parse gunicorn access logs
|
||||
**Recipient:** Developer
|
||||
**Action:** Check Flask logs for exceptions, review recent changes
|
||||
|
||||
#### ☐ Certificate Expiring in <30 Days
|
||||
**Method:** Daily openssl check
|
||||
**Recipient:** Admin
|
||||
**Action:** Generate new self-signed cert or renew
|
||||
|
||||
### Warning Alerts (daily summary)
|
||||
|
||||
#### ☐ Audit Log >1M Rows Without Retention
|
||||
**Method:** Daily SQL query
|
||||
**Recipient:** Admin
|
||||
**Action:** Configure audit log purge job
|
||||
|
||||
#### ☐ Endpoint Events >500K Rows
|
||||
**Method:** Daily SQL query
|
||||
**Recipient:** Admin
|
||||
**Action:** Configure event retention
|
||||
|
||||
#### ☐ Failed Login Rate >10/min from Single IP
|
||||
**Method:** Parse Flask auth logs
|
||||
**Recipient:** Admin
|
||||
**Action:** Review for brute force attempt, consider IP block
|
||||
|
||||
---
|
||||
|
||||
## MAINTENANCE SCHEDULE
|
||||
|
||||
### Daily (5 min)
|
||||
- Endpoint health dashboard check
|
||||
- Backup verification
|
||||
- Docker service health
|
||||
- Database disk space
|
||||
- Critical error review
|
||||
|
||||
### Weekly (15 min)
|
||||
- Audit log size check
|
||||
- Failed login review
|
||||
- Endpoint events accumulation
|
||||
- Security advisories scan
|
||||
- Database performance metrics
|
||||
|
||||
### Monthly (1 hour)
|
||||
- PAT usage audit
|
||||
- Certificate expiration check
|
||||
- Backup restoration test
|
||||
- Plugin version audit
|
||||
- Configuration drift review
|
||||
|
||||
### Quarterly (4 hours)
|
||||
- Full DR drill
|
||||
- Security log audit
|
||||
- Dependency vulnerability scan
|
||||
|
||||
**Total Annual Scheduled Effort:**
|
||||
- Daily: 365 * 5 min = 30 hours
|
||||
- Weekly: 52 * 15 min = 13 hours
|
||||
- Monthly: 12 * 1 hour = 12 hours
|
||||
- Quarterly: 4 * 4 hours = 16 hours
|
||||
- **Subtotal: 71 hours/year**
|
||||
|
||||
**Full Baseline (from analysis): 936 hours/year**
|
||||
- Scheduled monitoring: 71 hours (8%)
|
||||
- Planned maintenance: 600 hours (64%)
|
||||
- Ad-hoc incident response: 265 hours (28%)
|
||||
|
||||
---
|
||||
|
||||
## DEPLOYMENT RUNBOOK
|
||||
|
||||
### Pre-Deployment Checklist
|
||||
|
||||
#### ☐ Create Backup
|
||||
```bash
|
||||
# Take manual snapshot before deploy
|
||||
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml run --rm backup restic backup /app/instance --tag pre-deploy
|
||||
```
|
||||
|
||||
#### ☐ Test Bundle on Pilot Site
|
||||
- Deploy to ops environment first
|
||||
- Verify all services start (db, migrate, api)
|
||||
- Check migration logs for errors
|
||||
- Spot-check core features
|
||||
- Monitor for 24 hours
|
||||
|
||||
#### ☐ Review Migration Chain
|
||||
```bash
|
||||
# Check what migrations will run
|
||||
docker compose run --rm api flask db heads
|
||||
docker compose run --rm api flask db current
|
||||
```
|
||||
|
||||
#### ☐ Verify Disk Space
|
||||
```bash
|
||||
# Ensure enough space for migration and new code
|
||||
df -h /var/lib/docker
|
||||
```
|
||||
|
||||
### Deployment Steps
|
||||
|
||||
#### 1. Stop Services
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml down
|
||||
```
|
||||
|
||||
#### 2. Extract Bundle
|
||||
```bash
|
||||
# Transfer bundle across air gap
|
||||
# Extract to deployment directory
|
||||
tar -xzf shopdb-bundle-wj-<date>.tar.gz
|
||||
```
|
||||
|
||||
#### 3. Pull Latest Code
|
||||
```bash
|
||||
cd /path/to/shopdb
|
||||
git pull origin wj
|
||||
# Or: git bundle unbundle ../shopdb-wj.bundle
|
||||
```
|
||||
|
||||
#### 4. Rebuild Images
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml build --no-cache
|
||||
```
|
||||
|
||||
#### 5. Start Stack (migrations run automatically)
|
||||
```bash
|
||||
docker compose -f docker-compose.prebuilt.yml up -d
|
||||
```
|
||||
|
||||
#### 6. Monitor Startup
|
||||
```bash
|
||||
# Watch logs in real-time
|
||||
docker compose -f docker-compose.prebuilt.yml logs -f migrate
|
||||
|
||||
# Verify migrations completed
|
||||
docker compose -f docker-compose.prebuilt.yml logs migrate | tail -20
|
||||
|
||||
# Check services healthy
|
||||
docker ps
|
||||
```
|
||||
|
||||
#### 7. Verify Application
|
||||
```bash
|
||||
# Check database schema version
|
||||
docker compose run --rm api flask db current
|
||||
|
||||
# Test core endpoints
|
||||
curl https://localhost:5001/api/health
|
||||
curl https://localhost:5001/api/assets/types
|
||||
|
||||
# Check UI loads
|
||||
# Navigate to https://<hostname>:5173
|
||||
```
|
||||
|
||||
### Rollback Procedure
|
||||
|
||||
#### If Migration Fails:
|
||||
|
||||
```bash
|
||||
# Stop services
|
||||
docker compose -f docker-compose.prebuilt.yml down
|
||||
|
||||
# Restore from pre-deploy backup
|
||||
docker compose -f docker-compose.backup.yml run --rm backup restic restore <snapshot-id> --target /restore
|
||||
|
||||
# Copy restored instance/ directory
|
||||
cp -r /restore/app/instance /app/instance
|
||||
|
||||
# Revert code to previous version
|
||||
git checkout <previous-commit>
|
||||
|
||||
# Rebuild and start
|
||||
docker compose -f docker-compose.prebuilt.yml build
|
||||
docker compose -f docker-compose.prebuilt.yml up -d
|
||||
```
|
||||
|
||||
#### If Application Fails Post-Deploy:
|
||||
|
||||
```bash
|
||||
# Check logs first
|
||||
docker compose logs api | grep -E "ERROR|CRITICAL"
|
||||
|
||||
# If unfixable, rollback code
|
||||
git checkout <previous-commit>
|
||||
docker compose -f docker-compose.prebuilt.yml build
|
||||
docker compose -f docker-compose.prebuilt.yml restart api
|
||||
```
|
||||
|
||||
### Post-Deployment Verification
|
||||
|
||||
#### ☐ All Services Healthy
|
||||
```bash
|
||||
docker ps --filter name=shopdb
|
||||
# Should show: db (healthy), api (healthy), migrate (exited 0)
|
||||
```
|
||||
|
||||
#### ☐ Endpoints Checking In
|
||||
```sql
|
||||
SELECT COUNT(*) FROM assets
|
||||
WHERE assettype='endpoint'
|
||||
AND lastreportedat > NOW() - INTERVAL 30 MINUTE;
|
||||
```
|
||||
**Expect:** >95% of endpoints
|
||||
|
||||
#### ☐ No Critical Errors in Logs
|
||||
```bash
|
||||
docker compose logs --since 1h api | grep -i critical
|
||||
```
|
||||
**Expect:** Empty or expected warnings only
|
||||
|
||||
#### ☐ UI Loads and Functions
|
||||
- Login works
|
||||
- Asset search returns results
|
||||
- Can navigate to endpoint details
|
||||
- Scripts tab loads
|
||||
|
||||
#### ☐ Document Deployment
|
||||
```markdown
|
||||
## Deployment Log: 2026-09-11
|
||||
|
||||
- **Branch:** wj
|
||||
- **Commit:** 0b252db7d2
|
||||
- **Migrations:** 7d51_endpoint_logcollections (head)
|
||||
- **Downtime:** 5 minutes
|
||||
- **Issues:** None
|
||||
- **Endpoint Recovery:** 98% within 15 minutes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SKILL REQUIREMENTS
|
||||
|
||||
### Operator (20% of maintenance)
|
||||
- Run daily monitoring scripts
|
||||
- Verify backup completion
|
||||
- Check disk space
|
||||
- Restart services if needed
|
||||
- Basic Windows/Docker troubleshooting
|
||||
|
||||
### Admin (25% of maintenance)
|
||||
- Configure cron jobs / Task Scheduler
|
||||
- Manage Docker Compose deployments
|
||||
- User/permission management
|
||||
- Review logs for security issues
|
||||
- Certificate management
|
||||
|
||||
### Developer (40% of maintenance)
|
||||
- Dependency upgrades
|
||||
- Plugin updates
|
||||
- Troubleshoot application errors
|
||||
- Database performance tuning
|
||||
- Implement retention policies
|
||||
- Security patch development
|
||||
|
||||
### Architect (15% of maintenance)
|
||||
- Major version upgrades
|
||||
- DR drill planning
|
||||
- Security audit response
|
||||
- Capacity planning
|
||||
- Architecture decisions
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL COMPETENCIES REQUIRED
|
||||
|
||||
1. **Air-gap bundle workflow** - Create git bundles, transfer across three gaps, verify checksums
|
||||
2. **Database migration troubleshooting** - Alembic per-plugin chains, PLUGIN_TABLE_OWNERS, MySQL DDL behavior
|
||||
3. **Docker Compose production operations** - Multi-service stack, volume persistence, logging drivers
|
||||
4. **Flask + SQLAlchemy performance tuning** - Connection pools, query optimization, retention policies
|
||||
5. **Authentication and authorization** - JWT tokens, PATs, role permissions, collector key security
|
||||
6. **Plugin architecture** - Dependency graph, contract versions, enable/disable coordination
|
||||
7. **Security audit response** - Translate findings to remediation plans, prioritize risks
|
||||
|
||||
---
|
||||
|
||||
## KNOWN LIMITATIONS
|
||||
|
||||
### Cannot Be Automated
|
||||
- Air-gap bundle transfer (physical media required)
|
||||
- Certificate renewal (self-signed, manual process)
|
||||
- Plugin dependency ordering (requires developer judgment)
|
||||
- Security audit triage (requires threat modeling)
|
||||
|
||||
### Requires Developer Skills
|
||||
- Migration failures (manual schema reconciliation)
|
||||
- Plugin contract violations (code changes needed)
|
||||
- Performance degradation (query optimization, indexing)
|
||||
- Security vulnerability remediation (code patches)
|
||||
|
||||
### Incident Response Lag
|
||||
- Bundle creation + 3 air gaps = 2-3 day minimum for critical fixes
|
||||
- No rapid patching for zero-day vulnerabilities
|
||||
- Cannot roll back failed migrations easily (MySQL DDL auto-commits)
|
||||
|
||||
---
|
||||
|
||||
## REFERENCES
|
||||
|
||||
- [[maintenance-footprint-baseline]] - Full 936-hour analysis
|
||||
- [[security-audit-2026-09-11]] - Security findings driving Priority 1-3
|
||||
- [[security-fix-implementation]] - Long-term security roadmap
|
||||
- `docker-compose.backup.yml` - Backup service configuration
|
||||
- `scripts/scan_sbom.py` - Vulnerability scanning tool
|
||||
Reference in New Issue
Block a user