shopdb-flask/.claude/skills/production-health.md

---
name: production-health
description: Comprehensive health check of running system - services, plugins, database, API response times, logs
---

# Production Health Check

Complete system health assessment for site lead confidence before/after deployments.

## What This Skill Does

Comprehensive health validation:
1. Check all services running (Docker containers)
2. Verify all plugins loaded and active
3. Check recent logs for errors
4. Test login flow works
5. Verify endpoints reporting
6. Check database connectivity and performance
7. Test API response times
8. Validate critical workflows

## When to Use This Skill

**Before deployment:**
- Baseline current health
- Ensure system stable before changes

**After deployment:**
- Verify deployment succeeded
- Catch issues before users report them

**Regular monitoring:**
- Daily health check
- After maintenance windows
- When investigating user reports

## Workflow

### 1. Check Docker Services

```bash
docker compose ps
```

**Expected containers:**
- flask-shopdb-api-1: Up (healthy)
- flask-shopdb-db-1: Up (healthy)

**What to flag:**
- Container stopped
- Container restarting
- Health check failing
- High restart count

```bash
# Check uptime
docker compose ps --format "table {{.Service}}\t{{.Status}}"

# Check restart count
docker inspect flask-shopdb-api-1 --format='{{.RestartCount}}'
```

**Warning threshold:**
- Restart count > 0 (investigate why it restarted)
- Uptime < 5 minutes (recently restarted)

### 2. Verify Plugin Loading

```bash
docker compose exec api flask plugin list
```

**Expected output:**
- All plugins listed
- Enabled plugins show as active
- No import errors

**For each plugin, check:**
- Status: enabled/disabled
- Import succeeded (no errors)
- Version number
- Last updated timestamp

**What to flag:**
```
WARNING: Plugin 'endpoints' failed to load
  Error: ImportError: cannot import name 'Asset'
  Impact: Endpoint functionality unavailable
  Action: Check plugin code, verify dependencies
```

### 3. Check Recent Logs for Errors

Scan last 100 lines of API logs:

```bash
docker compose logs api --tail 100
```

**Pattern match for issues:**
- `ERROR` - Application errors
- `CRITICAL` - Critical failures
- `Exception` - Unhandled exceptions
- `Traceback` - Python stack traces
- `500 Internal Server Error` - API failures
- `Connection refused` - Database connectivity issues

**Categorize errors:**
- **Critical**: Service crashes, database down, security breaches
- **High**: Plugin failures, API 500s, data corruption
- **Medium**: Performance warnings, deprecation notices
- **Low**: Info messages with ERROR level (false positives)

**Report top errors:**
```
Recent errors in logs (last 100 lines):
1. ERROR [endpoints] Failed to update endpoint status (count: 5)
   Last seen: 2 minutes ago
   Impact: Endpoint status may be stale

2. WARNING [sqlalchemy] Connection pool exhausted (count: 2)
   Last seen: 15 minutes ago
   Impact: Slow API responses

No critical errors detected.
```

### 4. Test Database Connectivity

**Connection test:**
```bash
docker compose exec api flask shell -c "from shopdb.api import db; print(db.session.execute('SELECT 1').scalar())"
```

Expected: `1`

**Connection pool status:**
```bash
docker compose exec api flask shell -c "from shopdb.api import db; print(db.engine.pool.status())"
```

**What to check:**
- Pool size vs. active connections
- Connection timeout rate
- Average query time

**Query performance test:**
```bash
# Test simple query speed
time docker compose exec api flask shell -c "from shopdb.api import db, Asset; Asset.query.count()"
```

**Thresholds:**
- < 100ms: Good
- 100-500ms: Acceptable
- > 500ms: Investigate (table scans, missing indexes)

### 5. Test API Endpoints

**Health check:**
```bash
curl -k https://localhost:5001/api/health
```

Expected: `{"status": "healthy"}` in < 100ms

**Authentication endpoint:**
```bash
curl -k https://localhost:5001/api/auth/check
```

Expected: 401 or 200 (depending on session)

**Core API endpoints:**
```bash
# Test key endpoints (with timing)
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/assets
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/employees
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/plugins
```

**Response time thresholds:**
- < 200ms: Good
- 200-500ms: Acceptable
- 500ms-1s: Slow (investigate)
- > 1s: Performance issue

### 6. Test Login Flow

Automated login test:

```bash
# POST login request
curl -k -X POST https://localhost:5001/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"testadmin","password":"admin123"}' \
  -c /tmp/cookies.txt

# Verify session cookie received
grep session /tmp/cookies.txt
```

**Expected:**
- 200 OK response
- Session cookie set
- User data returned (id, username, roles)

**What to flag:**
- 500 error (authentication broken)
- 401 error with correct credentials (database issue)
- No session cookie (session middleware broken)
- Slow response > 1s (database slow)

### 7. Verify Endpoints Reporting

Check that endpoint agent is reporting:

```bash
# Count active endpoints
docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; print(f'Active endpoints: {Endpoint.query.filter_by(status=\"active\").count()}')"

# Check recent heartbeats (last 10 min)
docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; from datetime import datetime, timedelta; recent = Endpoint.query.filter(Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)).count(); print(f'Recent heartbeats: {recent}')"
```

**What to check:**
- At least 1 endpoint active (VM should be reporting)
- Recent heartbeats > 0 (agent connected in last 10 min)
- No endpoints with status 'error'

**If no endpoints reporting:**
- Check agent service on VM
- Verify network connectivity (firewall)
- Check API collector authentication

### 8. Check Frontend Accessibility

**Production build (Flask serves):**
```bash
curl -k -I https://localhost:5001/ | head -5
```

Expected: 200 OK, Content-Type: text/html

**Vite dev server (if running):**
```bash
curl -I http://localhost:5173/ | head -5
```

Expected: 200 OK

**Browser tests:**
- Load https://localhost:5001
- Check for JavaScript errors in console (F12)
- Verify sidebar loads
- Check plugin panels accessible

### 9. Validate Critical Workflows

Test end-to-end workflows:

**Asset creation workflow:**
```bash
# Create test asset via API
curl -k -X POST https://localhost:5001/api/assets \
  -H "Content-Type: application/json" \
  -b /tmp/cookies.txt \
  -d '{"name":"test-asset-health-check","asset_type":"test"}'
```

Expected: 201 Created

**Asset retrieval:**
```bash
curl -k https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt
```

Expected: 200 OK, asset data returned

**Cleanup test asset:**
```bash
curl -k -X DELETE https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt
```

### 10. Check System Resources

**Docker resource usage:**
```bash
docker stats --no-stream
```

**What to check:**
- CPU usage < 80% (unless under load)
- Memory usage < 80% of limit
- No memory leaks (usage trending up over time)

**Disk space:**
```bash
docker system df
```

**Thresholds:**
- Images: < 10GB acceptable
- Containers: < 1GB acceptable
- Volumes: Depends on data size, monitor growth

### 11. Summary Report

**[X] System healthy:**
```
Production health check: HEALTHY

Services:
[X] Docker containers: 2/2 running (uptime: 3 days)
[X] Flask API: Responding in 45ms avg
[X] Database: Connected, pool healthy (12/20 connections)

Plugins:
[X] All 12 plugins loaded successfully
[X] No import errors

Endpoints:
[X] 1 active endpoint (VM)
[X] Last heartbeat: 2 minutes ago

Performance:
[X] API response times: 50-200ms (good)
[X] Database queries: 20-80ms (good)
[X] CPU usage: 12% (low)
[X] Memory usage: 45% (normal)

Logs:
[X] No critical errors in last 100 lines
WARNING: 2 warnings (connection pool exhausted)

System is healthy and ready for production use.

Minor action items:
- Investigate connection pool exhaustion (may need to increase pool size)
```

**WARNING: Issues detected:**
```
Production health check: DEGRADED

CRITICAL ISSUES:
1. Database connection failing
   Error: could not connect to server
   Impact: All API requests failing with 500 errors
   Action: Check database service, verify credentials

2. Plugin 'endpoints' failed to load
   Error: ImportError: cannot import name 'Asset'
   Impact: Endpoint functionality unavailable
   Action: Check recent code changes, verify shopdb.api imports

HIGH PRIORITY:
3. API response times degraded (800ms avg, was 100ms)
   Impact: Slow user experience
   Action: Check database indexes, query performance

4. 15 endpoint agents not reporting (last seen > 1 hour)
   Impact: Stale endpoint data
   Action: Check agent service on VMs, network connectivity

WARNINGS:
5. Container restarted 3 times in last hour
   Action: Check logs for crash cause

System is DEGRADED - address critical issues before deployment!

Detailed logs: /tmp/health-check-{timestamp}.log
```

## Troubleshooting

**API not responding:**
- Check Docker: `docker compose ps`
- Check logs: `docker compose logs api --tail 50`
- Restart if needed: `docker compose restart api`

**Database connection issues:**
- Check database running: `docker compose ps db`
- Check credentials: .env file
- Test connection: `docker compose exec db mysql -u root -p`

**Slow API performance:**
- Check database query times
- Look for missing indexes: `EXPLAIN SELECT ...`
- Check for table scans in logs

**Plugins not loading:**
- Check import errors in logs
- Verify dependencies installed
- Check for circular imports

## Related

- `/docker-rebuild` - Rebuild if services unhealthy
- `/pre-commit` - Pre-deployment validation
- `/migration-check` - Database migration validation
- [[maintenance-footprint-baseline]] - Baseline metrics for comparison