shopdb-flask/.claude/skills/production-health.md

name, description

name	description
production-health	Comprehensive health check of running system - services, plugins, database, API response times, logs

Production Health Check

Complete system health assessment for site lead confidence before/after deployments.

What This Skill Does

Comprehensive health validation:

1. Check all services running (Docker containers)
2. Verify all plugins loaded and active
3. Check recent logs for errors
4. Test login flow works
5. Verify endpoints reporting
6. Check database connectivity and performance
7. Test API response times
8. Validate critical workflows

When to Use This Skill

Before deployment:

• Baseline current health
• Ensure system stable before changes

After deployment:

• Verify deployment succeeded
• Catch issues before users report them

Regular monitoring:

• Daily health check
• After maintenance windows
• When investigating user reports

Workflow

1. Check Docker Services

docker compose ps

Expected containers:

• flask-shopdb-api-1: Up (healthy)
• flask-shopdb-db-1: Up (healthy)

What to flag:

• Container stopped
• Container restarting
• Health check failing
• High restart count

# Check uptime
docker compose ps --format "table {{.Service}}\t{{.Status}}"

# Check restart count
docker inspect flask-shopdb-api-1 --format='{{.RestartCount}}'

Warning threshold:

• Restart count > 0 (investigate why it restarted)
• Uptime < 5 minutes (recently restarted)

2. Verify Plugin Loading

docker compose exec api flask plugin list

Expected output:

• All plugins listed
• Enabled plugins show as active
• No import errors

For each plugin, check:

• Status: enabled/disabled
• Import succeeded (no errors)
• Version number
• Last updated timestamp

What to flag:

WARNING: Plugin 'endpoints' failed to load
  Error: ImportError: cannot import name 'Asset'
  Impact: Endpoint functionality unavailable
  Action: Check plugin code, verify dependencies

3. Check Recent Logs for Errors

Scan last 100 lines of API logs:

docker compose logs api --tail 100

Pattern match for issues:

• ERROR - Application errors
• CRITICAL - Critical failures
• Exception - Unhandled exceptions
• Traceback - Python stack traces
• 500 Internal Server Error - API failures
• Connection refused - Database connectivity issues

Categorize errors:

• Critical: Service crashes, database down, security breaches
• High: Plugin failures, API 500s, data corruption
• Medium: Performance warnings, deprecation notices
• Low: Info messages with ERROR level (false positives)

Report top errors:

Recent errors in logs (last 100 lines):
1. ERROR [endpoints] Failed to update endpoint status (count: 5)
   Last seen: 2 minutes ago
   Impact: Endpoint status may be stale

2. WARNING [sqlalchemy] Connection pool exhausted (count: 2)
   Last seen: 15 minutes ago
   Impact: Slow API responses

No critical errors detected.

4. Test Database Connectivity

Connection test:

docker compose exec api flask shell -c "from shopdb.api import db; print(db.session.execute('SELECT 1').scalar())"

Expected: 1

Connection pool status:

docker compose exec api flask shell -c "from shopdb.api import db; print(db.engine.pool.status())"

What to check:

• Pool size vs. active connections
• Connection timeout rate
• Average query time

Query performance test:

# Test simple query speed
time docker compose exec api flask shell -c "from shopdb.api import db, Asset; Asset.query.count()"

Thresholds:

• < 100ms: Good
• 100-500ms: Acceptable

• 500ms: Investigate (table scans, missing indexes)

5. Test API Endpoints

Health check:

curl -k https://localhost:5001/api/health

Expected: {"status": "healthy"} in < 100ms

Authentication endpoint:

curl -k https://localhost:5001/api/auth/check

Expected: 401 or 200 (depending on session)

Core API endpoints:

# Test key endpoints (with timing)
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/assets
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/employees
curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/plugins

Response time thresholds:

• < 200ms: Good
• 200-500ms: Acceptable
• 500ms-1s: Slow (investigate)

• 1s: Performance issue

6. Test Login Flow

Automated login test:

# POST login request
curl -k -X POST https://localhost:5001/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"testadmin","password":"admin123"}' \
  -c /tmp/cookies.txt

# Verify session cookie received
grep session /tmp/cookies.txt

Expected:

• 200 OK response
• Session cookie set
• User data returned (id, username, roles)

What to flag:

• 500 error (authentication broken)
• 401 error with correct credentials (database issue)
• No session cookie (session middleware broken)
• Slow response > 1s (database slow)

7. Verify Endpoints Reporting

Check that endpoint agent is reporting:

# Count active endpoints
docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; print(f'Active endpoints: {Endpoint.query.filter_by(status=\"active\").count()}')"

# Check recent heartbeats (last 10 min)
docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; from datetime import datetime, timedelta; recent = Endpoint.query.filter(Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)).count(); print(f'Recent heartbeats: {recent}')"

What to check:

• At least 1 endpoint active (VM should be reporting)
• Recent heartbeats > 0 (agent connected in last 10 min)
• No endpoints with status 'error'

If no endpoints reporting:

• Check agent service on VM
• Verify network connectivity (firewall)
• Check API collector authentication

8. Check Frontend Accessibility

Production build (Flask serves):

curl -k -I https://localhost:5001/ | head -5

Expected: 200 OK, Content-Type: text/html

Vite dev server (if running):

curl -I http://localhost:5173/ | head -5

Expected: 200 OK

Browser tests:

• Load https://localhost:5001
• Check for JavaScript errors in console (F12)
• Verify sidebar loads
• Check plugin panels accessible

9. Validate Critical Workflows

Test end-to-end workflows:

Asset creation workflow:

# Create test asset via API
curl -k -X POST https://localhost:5001/api/assets \
  -H "Content-Type: application/json" \
  -b /tmp/cookies.txt \
  -d '{"name":"test-asset-health-check","asset_type":"test"}'

Expected: 201 Created

Asset retrieval:

curl -k https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt

Expected: 200 OK, asset data returned

Cleanup test asset:

curl -k -X DELETE https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt

10. Check System Resources

Docker resource usage:

docker stats --no-stream

What to check:

• CPU usage < 80% (unless under load)
• Memory usage < 80% of limit
• No memory leaks (usage trending up over time)

Disk space:

docker system df

Thresholds:

• Images: < 10GB acceptable
• Containers: < 1GB acceptable
• Volumes: Depends on data size, monitor growth

11. Summary Report

[X] System healthy:

Production health check: HEALTHY

Services:
[X] Docker containers: 2/2 running (uptime: 3 days)
[X] Flask API: Responding in 45ms avg
[X] Database: Connected, pool healthy (12/20 connections)

Plugins:
[X] All 12 plugins loaded successfully
[X] No import errors

Endpoints:
[X] 1 active endpoint (VM)
[X] Last heartbeat: 2 minutes ago

Performance:
[X] API response times: 50-200ms (good)
[X] Database queries: 20-80ms (good)
[X] CPU usage: 12% (low)
[X] Memory usage: 45% (normal)

Logs:
[X] No critical errors in last 100 lines
WARNING: 2 warnings (connection pool exhausted)

System is healthy and ready for production use.

Minor action items:
- Investigate connection pool exhaustion (may need to increase pool size)

WARNING: Issues detected:

Production health check: DEGRADED

CRITICAL ISSUES:
1. Database connection failing
   Error: could not connect to server
   Impact: All API requests failing with 500 errors
   Action: Check database service, verify credentials

2. Plugin 'endpoints' failed to load
   Error: ImportError: cannot import name 'Asset'
   Impact: Endpoint functionality unavailable
   Action: Check recent code changes, verify shopdb.api imports

HIGH PRIORITY:
3. API response times degraded (800ms avg, was 100ms)
   Impact: Slow user experience
   Action: Check database indexes, query performance

4. 15 endpoint agents not reporting (last seen > 1 hour)
   Impact: Stale endpoint data
   Action: Check agent service on VMs, network connectivity

WARNINGS:
5. Container restarted 3 times in last hour
   Action: Check logs for crash cause

System is DEGRADED - address critical issues before deployment!

Detailed logs: /tmp/health-check-{timestamp}.log

Troubleshooting

API not responding:

• Check Docker: docker compose ps
• Check logs: docker compose logs api --tail 50
• Restart if needed: docker compose restart api

Database connection issues:

• Check database running: docker compose ps db
• Check credentials: .env file
• Test connection: docker compose exec db mysql -u root -p

Slow API performance:

• Check database query times
• Look for missing indexes: EXPLAIN SELECT ...
• Check for table scans in logs

Plugins not loading:

• Check import errors in logs
• Verify dependencies installed
• Check for circular imports

Related

• /docker-rebuild - Rebuild if services unhealthy
• /pre-commit - Pre-deployment validation
• /migration-check - Database migration validation
• maintenance-footprint-baseline - Baseline metrics for comparison