Add 9 comprehensive automation skills and enforce tool usage

Create complete automation toolkit with 9 new skills covering all major workflows, saving 147-205 hrs/year from 936 hrs/year baseline. ## New Skills (Time Savings) Quick wins: - pre-commit: Complete validation workflow (8-16 hrs/year) - docker-rebuild: Full rebuild after changes (8 hrs/year) - agent-deploy-full: Complete agent deployment (3 hrs/year) Safety and quality: - migration-check: ADR-008 validation (prevents critical errors) - production-health: System health monitoring (deployment confidence) - security-scan: Sept 2026 audit compliance (prevent regressions) Advanced workflows: - vm-snapshot: Hyper-V snapshot management (78 hrs/year) - plugin-new: Scaffold new plugins (30-60 min per plugin) - full-deploy: Air-gap deployment orchestration (50-100 hrs/year) ## Tool Engagement Enforcement Updated settings.json to load TOOL-ENGAGEMENT-PROTOCOL.md in every session: - Protocol now always in context (cannot forget to check) - MCP tool requirements visible before starting work - Enforces coordinator.announce_work, Gortex usage, skill invocation ## Code Cleanup Removed unicode symbols from all skill files per CLAUDE.md: - No emojis (replaced with text) - No unicode checkmarks/arrows (replaced with ASCII) - No em-dashes (replaced with hyphens) - No box-drawing characters (replaced with +, -, |) - Plain ASCII only Total: 13 skills (4 existing + 9 new), all documentation plain ASCII Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-09-11 16:19:21 -04:00
parent bd48814021
commit ce27cb5eef
12 changed files with 3784 additions and 27 deletions
--- a/.claude/skills/production-health.md
+++ b/.claude/skills/production-health.md
@@ -0,0 +1,402 @@
+---
+name: production-health
+description: Comprehensive health check of running system - services, plugins, database, API response times, logs
+---
+
+# Production Health Check
+
+Complete system health assessment for site lead confidence before/after deployments.
+
+## What This Skill Does
+
+Comprehensive health validation:
+1. Check all services running (Docker containers)
+2. Verify all plugins loaded and active
+3. Check recent logs for errors
+4. Test login flow works
+5. Verify endpoints reporting
+6. Check database connectivity and performance
+7. Test API response times
+8. Validate critical workflows
+
+## When to Use This Skill
+
+**Before deployment:**
+- Baseline current health
+- Ensure system stable before changes
+
+**After deployment:**
+- Verify deployment succeeded
+- Catch issues before users report them
+
+**Regular monitoring:**
+- Daily health check
+- After maintenance windows
+- When investigating user reports
+
+## Workflow
+
+### 1. Check Docker Services
+
+```bash
+docker compose ps
+```
+
+**Expected containers:**
+- flask-shopdb-api-1: Up (healthy)
+- flask-shopdb-db-1: Up (healthy)
+
+**What to flag:**
+- Container stopped
+- Container restarting
+- Health check failing
+- High restart count
+
+```bash
+# Check uptime
+docker compose ps --format "table {{.Service}}\t{{.Status}}"
+
+# Check restart count
+docker inspect flask-shopdb-api-1 --format='{{.RestartCount}}'
+```
+
+**Warning threshold:**
+- Restart count > 0 (investigate why it restarted)
+- Uptime < 5 minutes (recently restarted)
+
+### 2. Verify Plugin Loading
+
+```bash
+docker compose exec api flask plugin list
+```
+
+**Expected output:**
+- All plugins listed
+- Enabled plugins show as active
+- No import errors
+
+**For each plugin, check:**
+- Status: enabled/disabled
+- Import succeeded (no errors)
+- Version number
+- Last updated timestamp
+
+**What to flag:**
+```
+WARNING: Plugin 'endpoints' failed to load
+  Error: ImportError: cannot import name 'Asset'
+  Impact: Endpoint functionality unavailable
+  Action: Check plugin code, verify dependencies
+```
+
+### 3. Check Recent Logs for Errors
+
+Scan last 100 lines of API logs:
+
+```bash
+docker compose logs api --tail 100
+```
+
+**Pattern match for issues:**
+- `ERROR` - Application errors
+- `CRITICAL` - Critical failures
+- `Exception` - Unhandled exceptions
+- `Traceback` - Python stack traces
+- `500 Internal Server Error` - API failures
+- `Connection refused` - Database connectivity issues
+
+**Categorize errors:**
+- **Critical**: Service crashes, database down, security breaches
+- **High**: Plugin failures, API 500s, data corruption
+- **Medium**: Performance warnings, deprecation notices
+- **Low**: Info messages with ERROR level (false positives)
+
+**Report top errors:**
+```
+Recent errors in logs (last 100 lines):
+1. ERROR [endpoints] Failed to update endpoint status (count: 5)
+   Last seen: 2 minutes ago
+   Impact: Endpoint status may be stale
+
+2. WARNING [sqlalchemy] Connection pool exhausted (count: 2)
+   Last seen: 15 minutes ago
+   Impact: Slow API responses
+
+No critical errors detected.
+```
+
+### 4. Test Database Connectivity
+
+**Connection test:**
+```bash
+docker compose exec api flask shell -c "from shopdb.api import db; print(db.session.execute('SELECT 1').scalar())"
+```
+
+Expected: `1`
+
+**Connection pool status:**
+```bash
+docker compose exec api flask shell -c "from shopdb.api import db; print(db.engine.pool.status())"
+```
+
+**What to check:**
+- Pool size vs. active connections
+- Connection timeout rate
+- Average query time
+
+**Query performance test:**
+```bash
+# Test simple query speed
+time docker compose exec api flask shell -c "from shopdb.api import db, Asset; Asset.query.count()"
+```
+
+**Thresholds:**
+- < 100ms: Good
+- 100-500ms: Acceptable
+- > 500ms: Investigate (table scans, missing indexes)
+
+### 5. Test API Endpoints
+
+**Health check:**
+```bash
+curl -k https://localhost:5001/api/health
+```
+
+Expected: `{"status": "healthy"}` in < 100ms
+
+**Authentication endpoint:**
+```bash
+curl -k https://localhost:5001/api/auth/check
+```
+
+Expected: 401 or 200 (depending on session)
+
+**Core API endpoints:**
+```bash
+# Test key endpoints (with timing)
+curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/assets
+curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/employees
+curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/plugins
+```
+
+**Response time thresholds:**
+- < 200ms: Good
+- 200-500ms: Acceptable
+- 500ms-1s: Slow (investigate)
+- > 1s: Performance issue
+
+### 6. Test Login Flow
+
+Automated login test:
+
+```bash
+# POST login request
+curl -k -X POST https://localhost:5001/api/auth/login \
+  -H "Content-Type: application/json" \
+  -d '{"username":"testadmin","password":"admin123"}' \
+  -c /tmp/cookies.txt
+
+# Verify session cookie received
+grep session /tmp/cookies.txt
+```
+
+**Expected:**
+- 200 OK response
+- Session cookie set
+- User data returned (id, username, roles)
+
+**What to flag:**
+- 500 error (authentication broken)
+- 401 error with correct credentials (database issue)
+- No session cookie (session middleware broken)
+- Slow response > 1s (database slow)
+
+### 7. Verify Endpoints Reporting
+
+Check that endpoint agent is reporting:
+
+```bash
+# Count active endpoints
+docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; print(f'Active endpoints: {Endpoint.query.filter_by(status=\"active\").count()}')"
+
+# Check recent heartbeats (last 10 min)
+docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; from datetime import datetime, timedelta; recent = Endpoint.query.filter(Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)).count(); print(f'Recent heartbeats: {recent}')"
+```
+
+**What to check:**
+- At least 1 endpoint active (VM should be reporting)
+- Recent heartbeats > 0 (agent connected in last 10 min)
+- No endpoints with status 'error'
+
+**If no endpoints reporting:**
+- Check agent service on VM
+- Verify network connectivity (firewall)
+- Check API collector authentication
+
+### 8. Check Frontend Accessibility
+
+**Production build (Flask serves):**
+```bash
+curl -k -I https://localhost:5001/ | head -5
+```
+
+Expected: 200 OK, Content-Type: text/html
+
+**Vite dev server (if running):**
+```bash
+curl -I http://localhost:5173/ | head -5
+```
+
+Expected: 200 OK
+
+**Browser tests:**
+- Load https://localhost:5001
+- Check for JavaScript errors in console (F12)
+- Verify sidebar loads
+- Check plugin panels accessible
+
+### 9. Validate Critical Workflows
+
+Test end-to-end workflows:
+
+**Asset creation workflow:**
+```bash
+# Create test asset via API
+curl -k -X POST https://localhost:5001/api/assets \
+  -H "Content-Type: application/json" \
+  -b /tmp/cookies.txt \
+  -d '{"name":"test-asset-health-check","asset_type":"test"}'
+```
+
+Expected: 201 Created
+
+**Asset retrieval:**
+```bash
+curl -k https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt
+```
+
+Expected: 200 OK, asset data returned
+
+**Cleanup test asset:**
+```bash
+curl -k -X DELETE https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt
+```
+
+### 10. Check System Resources
+
+**Docker resource usage:**
+```bash
+docker stats --no-stream
+```
+
+**What to check:**
+- CPU usage < 80% (unless under load)
+- Memory usage < 80% of limit
+- No memory leaks (usage trending up over time)
+
+**Disk space:**
+```bash
+docker system df
+```
+
+**Thresholds:**
+- Images: < 10GB acceptable
+- Containers: < 1GB acceptable
+- Volumes: Depends on data size, monitor growth
+
+### 11. Summary Report
+
+**[X] System healthy:**
+```
+Production health check: HEALTHY
+
+Services:
+[X] Docker containers: 2/2 running (uptime: 3 days)
+[X] Flask API: Responding in 45ms avg
+[X] Database: Connected, pool healthy (12/20 connections)
+
+Plugins:
+[X] All 12 plugins loaded successfully
+[X] No import errors
+
+Endpoints:
+[X] 1 active endpoint (VM)
+[X] Last heartbeat: 2 minutes ago
+
+Performance:
+[X] API response times: 50-200ms (good)
+[X] Database queries: 20-80ms (good)
+[X] CPU usage: 12% (low)
+[X] Memory usage: 45% (normal)
+
+Logs:
+[X] No critical errors in last 100 lines
+WARNING: 2 warnings (connection pool exhausted)
+
+System is healthy and ready for production use.
+
+Minor action items:
+- Investigate connection pool exhaustion (may need to increase pool size)
+```
+
+**WARNING: Issues detected:**
+```
+Production health check: DEGRADED
+
+CRITICAL ISSUES:
+1. Database connection failing
+   Error: could not connect to server
+   Impact: All API requests failing with 500 errors
+   Action: Check database service, verify credentials
+
+2. Plugin 'endpoints' failed to load
+   Error: ImportError: cannot import name 'Asset'
+   Impact: Endpoint functionality unavailable
+   Action: Check recent code changes, verify shopdb.api imports
+
+HIGH PRIORITY:
+3. API response times degraded (800ms avg, was 100ms)
+   Impact: Slow user experience
+   Action: Check database indexes, query performance
+
+4. 15 endpoint agents not reporting (last seen > 1 hour)
+   Impact: Stale endpoint data
+   Action: Check agent service on VMs, network connectivity
+
+WARNINGS:
+5. Container restarted 3 times in last hour
+   Action: Check logs for crash cause
+
+System is DEGRADED - address critical issues before deployment!
+
+Detailed logs: /tmp/health-check-{timestamp}.log
+```
+
+## Troubleshooting
+
+**API not responding:**
+- Check Docker: `docker compose ps`
+- Check logs: `docker compose logs api --tail 50`
+- Restart if needed: `docker compose restart api`
+
+**Database connection issues:**
+- Check database running: `docker compose ps db`
+- Check credentials: .env file
+- Test connection: `docker compose exec db mysql -u root -p`
+
+**Slow API performance:**
+- Check database query times
+- Look for missing indexes: `EXPLAIN SELECT ...`
+- Check for table scans in logs
+
+**Plugins not loading:**
+- Check import errors in logs
+- Verify dependencies installed
+- Check for circular imports
+
+## Related
+
+- `/docker-rebuild` - Rebuild if services unhealthy
+- `/pre-commit` - Pre-deployment validation
+- `/migration-check` - Database migration validation
+- [[maintenance-footprint-baseline]] - Baseline metrics for comparison