From ce27cb5eef6f2063c758dc5ab5e25588761f0679 Mon Sep 17 00:00:00 2001 From: Proudlock <270015376@geaerospace.com> Date: Fri, 11 Sep 2026 16:19:21 -0400 Subject: [PATCH] Add 9 comprehensive automation skills and enforce tool usage Create complete automation toolkit with 9 new skills covering all major workflows, saving 147-205 hrs/year from 936 hrs/year baseline. ## New Skills (Time Savings) Quick wins: - pre-commit: Complete validation workflow (8-16 hrs/year) - docker-rebuild: Full rebuild after changes (8 hrs/year) - agent-deploy-full: Complete agent deployment (3 hrs/year) Safety and quality: - migration-check: ADR-008 validation (prevents critical errors) - production-health: System health monitoring (deployment confidence) - security-scan: Sept 2026 audit compliance (prevent regressions) Advanced workflows: - vm-snapshot: Hyper-V snapshot management (78 hrs/year) - plugin-new: Scaffold new plugins (30-60 min per plugin) - full-deploy: Air-gap deployment orchestration (50-100 hrs/year) ## Tool Engagement Enforcement Updated settings.json to load TOOL-ENGAGEMENT-PROTOCOL.md in every session: - Protocol now always in context (cannot forget to check) - MCP tool requirements visible before starting work - Enforces coordinator.announce_work, Gortex usage, skill invocation ## Code Cleanup Removed unicode symbols from all skill files per CLAUDE.md: - No emojis (replaced with text) - No unicode checkmarks/arrows (replaced with ASCII) - No em-dashes (replaced with hyphens) - No box-drawing characters (replaced with +, -, |) - Plain ASCII only Total: 13 skills (4 existing + 9 new), all documentation plain ASCII Co-Authored-By: Claude Sonnet 4.5 --- .claude/settings.json | 3 +- .claude/skills/README.md | 230 ++++++++- .claude/skills/agent-deploy-full.md | 286 +++++++++++ .claude/skills/docker-rebuild.md | 248 ++++++++++ .claude/skills/full-deploy.md | 577 +++++++++++++++++++++++ .claude/skills/merge-wilm.md | 4 +- .claude/skills/migration-check.md | 328 +++++++++++++ .claude/skills/plugin-new.md | 707 ++++++++++++++++++++++++++++ .claude/skills/pre-commit.md | 186 ++++++++ .claude/skills/production-health.md | 402 ++++++++++++++++ .claude/skills/security-scan.md | 445 +++++++++++++++++ .claude/skills/vm-snapshot.md | 395 ++++++++++++++++ 12 files changed, 3784 insertions(+), 27 deletions(-) create mode 100644 .claude/skills/agent-deploy-full.md create mode 100644 .claude/skills/docker-rebuild.md create mode 100644 .claude/skills/full-deploy.md create mode 100644 .claude/skills/migration-check.md create mode 100644 .claude/skills/plugin-new.md create mode 100644 .claude/skills/pre-commit.md create mode 100644 .claude/skills/production-health.md create mode 100644 .claude/skills/security-scan.md create mode 100644 .claude/skills/vm-snapshot.md diff --git a/.claude/settings.json b/.claude/settings.json index 4f4e3e1e..60ee38eb 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -6,7 +6,8 @@ }, "context": { "alwaysInclude": [ - ".claude/INDEX.md" + ".claude/INDEX.md", + ".claude/plans/TOOL-ENGAGEMENT-PROTOCOL.md" ] }, "hooks": { diff --git a/.claude/skills/README.md b/.claude/skills/README.md index ce7464be..5ef15662 100644 --- a/.claude/skills/README.md +++ b/.claude/skills/README.md @@ -1,22 +1,125 @@ # ShopDB Skills -Custom slash commands for common workflows. +Custom slash commands for common workflows. **13 skills total** - comprehensive automation toolkit. -## Available Skills +## Quick Reference by Category -### `/build-agent` -Build and publish agent and tray executables to Documents/shopdb-builds folder. +**Development & Testing:** +- `/pre-commit` - Complete pre-commit validation +- `/test-changes` - Quick test suite +- `/docker-rebuild` - Rebuild Docker after changes -**When to use:** After making changes to agent or tray C# code, before deployment. +**Deployment:** +- `/full-deploy` - Complete air-gap deployment workflow +- `/agent-deploy-full` - Build, deploy, verify C# agent +- `/deploy-frontend` - Deploy Vue to production +- `/merge-wilm` - Merge upstream branch + +**Infrastructure:** +- `/vm-snapshot` - Hyper-V snapshot management +- `/production-health` - System health check +- `/migration-check` - Database migration validation +- `/security-scan` - Security vulnerability scan + +**Development Tools:** +- `/plugin-new` - Scaffold new plugin +- `/build-agent` - Build C# executables + +--- + +## Development & Testing Skills + +### `/pre-commit` NEW +**Complete pre-commit validation workflow - tests, style checks, build verification, ADR compliance** + +**When to use:** Before EVERY commit to wj branch or creating PR. **What it does:** -- Compiles both executables -- Creates dated folder in Documents -- Generates VERSION.txt with release notes -- Copies .exe files ready for deployment +- Runs full test suite (pytest -v) +- Checks naming and style conventions +- Builds frontend to catch errors +- Verifies both URLs work (Vite + Flask) +- Scans for hardcoded site data (ADR-015) +- Validates migration linearity (ADR-008) +- Provides pass/fail summary + +**Time saved:** 8-16 hrs/year + +### `/test-changes` +**Run pytest and frontend build checks before committing (lighter than /pre-commit)** + +**When to use:** Quick validation during development. + +**What it does:** +- Runs pytest for backend +- Builds frontend to catch errors +- Runs style/naming checker +- Provides targeted test commands + +### `/docker-rebuild` NEW +**Complete Docker rebuild workflow - build frontend, rebuild image, run migrations, verify health** + +**When to use:** After merging branches, dependency changes, or Dockerfile modifications. + +**What it does:** +- Builds frontend (prebuilt approach) +- Rebuilds Docker image +- Restarts containers cleanly +- Runs database migrations +- Verifies services healthy +- Tests both URLs work + +**Time saved:** 8 hrs/year + +--- + +## Deployment Skills + +### `/full-deploy` NEW +**Complete deployment workflow for air-gapped production - saves 50-100 hrs/year from 312 hrs coordination burden** + +**When to use:** Deploying to production (wj branch) or ops (dev branch). + +**What it does:** +- Pre-deploy validation (tests, health, security, migrations) +- Creates deployment bundle +- Deploys to dev PC first (staging test) +- Generates production deployment runbook +- Guides through production deployment +- Post-deploy validation +- Provides rollback plan if issues + +**Time saved:** 50-100 hrs/year (from 312 hrs/year burden) + +### `/agent-deploy-full` NEW +**Complete agent deployment workflow - build, test, deploy to VM, verify endpoint reporting** + +**When to use:** Deploying C# agent or tray updates to VM. + +**What it does:** +- Builds agent and tray executables +- Tests locally (quick sanity check) +- Deploys to VM via SSH +- Restarts Windows service +- Verifies agent reports to API +- Checks endpoint appears in UI +- Monitors logs for errors + +**Time saved:** 3 hrs/year + +### `/deploy-frontend` +**Build frontend and deploy to prod server via Y drive** + +**When to use:** After making Vue/frontend changes that need to go to production. + +**What it does:** +- Runs production build with correct VITE_BASE_PATH +- Copies dist/ to Y: drive +- Provides instructions for IIS restart +- Verifies build output ### `/merge-wilm` -Merge latest wilm branch commits into wj with proper conflict resolution. +**Merge latest wilm branch commits into wj with proper conflict resolution** **When to use:** Periodically to bring wilm improvements into wj prod branch. @@ -27,27 +130,106 @@ Merge latest wilm branch commits into wj with proper conflict resolution. - Coordinates agent/tray versions - Creates merge commit -### `/deploy-frontend` -Build frontend and deploy to prod server via Y drive. +### `/build-agent` +**Build and publish agent and tray executables to Documents/shopdb-builds folder** -**When to use:** After making Vue/frontend changes that need to go to production. +**When to use:** After making changes to agent or tray C# code, before deployment. **What it does:** -- Runs production build with correct VITE_BASE_PATH -- Copies dist/ to Y: drive -- Provides instructions for IIS restart -- Verifies build output +- Compiles both executables +- Creates dated folder in Documents +- Generates VERSION.txt with release notes +- Copies .exe files ready for deployment -### `/test-changes` -Run pytest and frontend build checks before committing. +--- -**When to use:** Before claiming work is complete or pushing commits. +## Infrastructure Skills + +### `/vm-snapshot` NEW +**Create, restore, list, and manage Hyper-V VM snapshots** + +**When to use:** Before risky changes, for baseline snapshots, weekly cleanup. **What it does:** -- Runs pytest for backend -- Builds frontend to catch errors -- Runs style/naming checker -- Provides targeted test commands +- Creates snapshot with descriptive name +- Lists all snapshots with dates and sizes +- Restores to specific snapshot +- Deletes old snapshots (cleanup) +- Verifies VM state after restore + +**Time saved:** 78 hrs/year + +### `/production-health` NEW +**Comprehensive health check of running system - services, plugins, database, API response times, logs** + +**When to use:** Before/after deployments, daily monitoring, investigating issues. + +**What it does:** +- Checks all Docker services running +- Verifies all plugins loaded +- Scans recent logs for errors +- Tests login flow works +- Verifies endpoints reporting +- Checks database connectivity and performance +- Tests API response times +- Validates critical workflows + +**Value:** Deployment confidence, early issue detection + +### `/migration-check` NEW +**Deep validation of plugin migrations for ADR-008 compliance - check linearity, dependencies, foreign keys** + +**When to use:** Before committing migrations, before production deployment. + +**What it does:** +- Verifies each plugin migration chain is linear +- Checks for circular dependencies between plugins +- Validates foreign key references exist +- Tests migrations up AND down +- Verifies PLUGIN_TABLE_OWNERS coordination +- Checks for duplicate version numbers +- Identifies orphaned migrations + +**Value:** Prevents critical ADR-008 violations in production + +### `/security-scan` NEW +**Run security checks from Sept 2026 audit - scan for hardcoded credentials, exposed keys, missing authentication** + +**When to use:** Before commits, before deployments, after merging external code. + +**What it does:** +- Scans for hardcoded credentials and API keys +- Checks for collector keys in committed files +- Verifies certificate validation enabled +- Checks authentication on sensitive endpoints +- Scans for SQL injection vulnerabilities +- Verifies HTTPS enforcement +- Checks for sensitive data in logs +- Validates input sanitization + +**Value:** Prevents security regressions from Sept 2026 audit + +--- + +## Development Tools + +### `/plugin-new` NEW +**Scaffold new ShopDB plugin following ADRs - directory structure, BasePlugin boilerplate, migrations, Vue component** + +**When to use:** Creating a new plugin. + +**What it does:** +- Creates complete plugin directory structure +- Generates BasePlugin subclass with manifest +- Creates SQLAlchemy model (Asset polymorphism) +- Creates Flask API blueprint (CRUD endpoints) +- Initializes Alembic migrations +- Creates Vue frontend component +- Generates tests +- Creates README with checklist +- Adds to PLUGIN_TABLE_OWNERS + +**Time saved:** 30-60 min per new plugin ## How Skills Work diff --git a/.claude/skills/agent-deploy-full.md b/.claude/skills/agent-deploy-full.md new file mode 100644 index 00000000..6997a5b1 --- /dev/null +++ b/.claude/skills/agent-deploy-full.md @@ -0,0 +1,286 @@ +--- +name: agent-deploy-full +description: Complete agent deployment workflow - build, test, deploy to VM, verify endpoint reporting +--- + +# Full Agent Deployment Workflow + +Build C# agent and tray, deploy to VM, restart service, and verify endpoint appears in ShopDB UI. + +## What This Skill Does + +End-to-end agent deployment: +1. Build agent and tray executables +2. Test executables can start locally +3. Deploy to VM via SSH +4. Restart Windows service +5. Verify agent reports to API +6. Check endpoint appears in UI +7. Monitor logs for errors + +## Prerequisites + +Before running this workflow: +- Flask API running at https://192.168.100.1:5001 +- VM accessible via SSH (192.168.100.10) +- SSH key configured: ~/.ssh/vm_key +- VM users: SupportUser (admin) for service, shopfloor for testing + +## Workflow + +### 1. Build Agent and Tray + +Call existing build skill: + +```bash +# Invoke /build-agent skill or manually: +cd agent +dotnet publish ShopDbEndpointAgent/ShopDbEndpointAgent.csproj -c Release -o ../publish/agent +dotnet publish ShopDbEndpointAgent.Tray/ShopDbEndpointAgent.Tray.csproj -c Release -o ../publish/tray +cd .. +``` + +**Verify build:** +- publish/agent/shopdbagent.exe exists (~35MB) +- publish/tray/shopdb-endpoint-tray.exe exists (~69MB) +- Both are self-contained (include .NET runtime) + +If build fails: +- Check for compilation errors +- Verify .NET SDK installed (dotnet --version) +- Check for missing dependencies + +### 2. Test Locally (Quick Sanity Check) + +Test agent can start without crashing: + +```bash +cd publish/agent +./shopdbagent.exe --help +cd ../.. +``` + +**Expected:** Help text shows, no exceptions + +**Don't run full agent locally** - it will try to install as Windows service + +### 3. Stop VM Service + +Stop existing service before deploying new version: + +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Stop-Service ShopDBEndpointAgent -ErrorAction SilentlyContinue" +``` + +**Check status:** +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-Service ShopDBEndpointAgent" +``` + +Expected: Status = Stopped + +### 4. Deploy to VM + +Copy new executables to VM: + +```bash +# Agent +scp -i ~/.ssh/vm_key -r publish/agent/* SupportUser@192.168.100.10:C:/ProgramData/ShopDB/EndpointAgent/ + +# Tray +scp -i ~/.ssh/vm_key -r publish/tray/* SupportUser@192.168.100.10:C:/ProgramData/ShopDB/EndpointAgent/ +``` + +**Verify copy:** +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-ChildItem C:/ProgramData/ShopDB/EndpointAgent/ -Name" +``` + +Should see: +- shopdbagent.exe +- shopdb-endpoint-tray.exe +- Various .dll files + +### 5. Verify Configuration + +Check agent configuration in registry: + +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-ItemProperty HKLM:\SOFTWARE\GE\ShopDB" +``` + +**Required settings:** +- `BaseUrl` = https://192.168.100.1:5001 +- `CollectorKey` = shopdb_pat_YOUR_COLLECTOR_KEY_HERE + +If missing or wrong: +```bash +# Set BaseUrl +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "reg add HKLM\SOFTWARE\GE\ShopDB /v BaseUrl /t REG_SZ /d https://192.168.100.1:5001 /f" + +# Set CollectorKey +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "reg add HKLM\SOFTWARE\GE\ShopDB /v CollectorKey /t REG_SZ /d shopdb_pat_YOUR_COLLECTOR_KEY_HERE /f" +``` + +### 6. Start Service + +Restart the Windows service: + +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Start-Service ShopDBEndpointAgent" +``` + +**Verify service started:** +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-Service ShopDBEndpointAgent" +``` + +Expected: Status = Running + +If service won't start: +- Check logs (step 7) +- Verify registry configuration +- Check .NET runtime version compatibility + +### 7. Monitor Logs + +Check agent logs for errors: + +```bash +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-Content C:/ProgramData/ShopDB/EndpointAgent/logs/latest.log -Tail 30" +``` + +**What to look for:** +- [X] "Service started successfully" +- [X] "Connected to API at https://192.168.100.1:5001" +- [X] "Inventory collection started" +- [ ] Connection errors +- [ ] Authentication failures +- [ ] Exceptions + +Common issues: +- **Connection refused**: Flask API not running or firewall blocking +- **401 Unauthorized**: CollectorKey mismatch +- **Certificate error**: Agent not configured to accept self-signed cert + +### 8. Verify API Connection + +Check agent made initial connection: + +```bash +# Check Flask logs for collector POST +docker compose logs api --tail 50 | grep -i "collector\|endpoint" +``` + +**Expected:** POST /api/collectors/heartbeat or similar + +### 9. Check Endpoint in UI + +Navigate to endpoints page and verify: + +**URL:** https://192.168.100.1:5001/endpoints + +**Should see:** +- Endpoint with hostname matching VM +- Last seen timestamp recent (< 5 min ago) +- System information populated (OS, IP, etc.) +- Status: Active + +If endpoint doesn't appear: +- Wait 2-3 minutes (collection interval) +- Check API logs for POST failures +- Verify agent service running +- Check firewall rules on host + +### 10. Test Tray Application (Optional) + +Test system tray app as standard user: + +```bash +# Login to VM as shopfloor user via RDP +# Start: C:\ProgramData\ShopDB\EndpointAgent\shopdb-endpoint-tray.exe +``` + +**Expected:** +- Tray icon appears in notification area +- Right-click shows menu +- Can open ShopDB link +- No errors in logs + +### 11. Summary Report + +**[X] Deployment successful:** +``` +Agent deployment complete: +[X] Built: agent (35MB), tray (69MB) +[X] Deployed to VM: C:\ProgramData\ShopDB\EndpointAgent\ +[X] Service status: Running +[X] Logs: No errors, connected to API +[X] Endpoint visible in UI: DESKTOP-VM01 (192.168.100.10) +[X] Last seen: 2 minutes ago + +Deployment successful! Agent is reporting normally. +``` + +**WARNING: Issues detected:** +``` +Agent deployment completed with warnings: +[X] Built and deployed +[X] Service started +[ ] Connection errors in logs (certificate validation) +[ ] Endpoint not appearing in UI + +Action needed: +1. Check agent logs: C:\ProgramData\ShopDB\EndpointAgent\logs\latest.log +2. Verify Flask API accessible from VM: curl -k https://192.168.100.1:5001/api/health +3. Check firewall rules allow VM -> host traffic on port 5001 + +Troubleshooting: See [[agent-deployment]] memory for detailed steps +``` + +## Troubleshooting + +**Service won't start:** +- Check Windows Event Viewer on VM +- Verify .NET runtime included (self-contained) +- Check for port conflicts (another instance running?) + +**Connection refused:** +- Verify Flask API running: docker compose ps +- Check host firewall: port 5001 open for VM +- Test from VM: `curl -k https://192.168.100.1:5001/api/health` + +**401 Unauthorized:** +- CollectorKey mismatch between registry and .env +- Check .env: `COLLECTOR_API_KEY=shopdb_pat_...` +- Verify registry: `reg query HKLM\SOFTWARE\GE\ShopDB /v CollectorKey` + +**Endpoint not appearing:** +- Wait 5 minutes (collection interval) +- Check API logs for POST /api/collectors/heartbeat +- Verify collector authentication enabled +- Check database: `SELECT * FROM endpoints;` + +## Rollback + +If deployment fails, rollback to previous version: + +```bash +# Stop service +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Stop-Service ShopDBEndpointAgent" + +# Restore previous build (if backed up) +# Or redeploy previous version from publish/agent/ + +# Restart service +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Start-Service ShopDBEndpointAgent" +``` + +## Related + +- `/build-agent` - Just build executables +- `/production-health` - Check overall system health +- [[agent-deployment]] - Detailed agent deployment guide +- [[vm-ssh-access]] - SSH key setup +- [[vm-network-access]] - Firewall configuration +- `.claude/setup/VM-USERS.md` - VM user accounts diff --git a/.claude/skills/docker-rebuild.md b/.claude/skills/docker-rebuild.md new file mode 100644 index 00000000..7d49d4ec --- /dev/null +++ b/.claude/skills/docker-rebuild.md @@ -0,0 +1,248 @@ +--- +name: docker-rebuild +description: Complete Docker rebuild workflow - build frontend, rebuild image, run migrations, verify health +--- + +# Docker Rebuild Workflow + +Rebuild Docker image after backend changes, dependency updates, or branch merges. Ensures clean state with latest code. + +## What This Skill Does + +Complete Docker rebuild process: +1. Build frontend (prebuilt approach for Zscaler proxy) +2. Rebuild Docker image with latest code +3. Restart containers cleanly +4. Run database migrations +5. Verify services healthy +6. Test both URLs work + +## When to Use This Skill + +**Required after:** +- Merging wilm branch (Python/dependency changes) +- Updating requirements.txt +- Changing Dockerfile.prebuilt +- System package changes (apt installs) + +**Optional after:** +- Backend code changes (Flask auto-reloads in dev) +- Frontend changes (just npm run build + restart api) + +## Workflow + +### 1. Check Current State + +```bash +docker compose ps +git status +``` + +Show: +- Which containers are running +- Whether there are uncommitted changes +- Current branch (should be wj or dev) + +### 2. Build Frontend First + +Using prebuilt approach (ADR: corporate proxy blocks npm in Docker): + +```bash +cd frontend +npm run build +cd .. +``` + +**Verify:** +- Build succeeds without errors +- `frontend/dist/` directory updated +- Bundle size reasonable (< 20MB) + +If build fails: +- Check for syntax errors +- Verify node_modules/ is up to date (npm ci if needed) +- Check for missing dependencies + +### 3. Stop Containers + +Clean shutdown: + +```bash +docker compose -f docker-compose.prebuilt.yml down +``` + +**What this does:** +- Stops all containers gracefully +- Removes containers (not volumes) +- Network cleaned up + +### 4. Rebuild Docker Image + +```bash +docker compose -f docker-compose.prebuilt.yml build api +``` + +**Monitor for:** +- Build completes without errors +- Python dependencies install successfully +- frontend/dist/ copied into image +- Image size reasonable (< 1GB) + +Common build issues: +- Dockerfile.prebuilt syntax error +- Missing files in build context +- Dependency conflicts in requirements.txt + +### 5. Start Containers + +```bash +docker compose -f docker-compose.prebuilt.yml up -d +``` + +**Verify startup:** +- Both containers start (db-1, api-1) +- Health checks pass +- Logs show no errors + +```bash +docker compose ps +docker compose logs api --tail 30 +``` + +Look for: +- "Listening at" from gunicorn (Flask ready) +- No Python exceptions +- Plugins loaded successfully + +### 6. Run Migrations + +Apply any new database migrations: + +```bash +# Core migrations +docker compose exec api flask db upgrade + +# Plugin migrations +docker compose exec api flask plugin upgrade-all +``` + +**Verify:** +- Migrations apply cleanly +- No "head not found" errors +- No circular dependency issues + +If migrations fail: +- Check logs for specific error +- Verify migration chains are linear (ADR-008) +- May need to manually fix migration conflicts + +### 7. Verify Services Healthy + +**Database connectivity:** +```bash +docker compose exec api flask shell -c "from shopdb.api import db; print(db.session.execute('SELECT 1').scalar())" +``` + +**Flask responding:** +```bash +curl -k https://localhost:5001/api/health +``` + +**Expected:** `{"status": "healthy"}` + +### 8. Test Both URLs + +Verify both development and production builds work: + +**Vite dev server** (if running): +- http://192.168.100.1:5173 +- Should show latest code +- Hot reload working + +**Flask production build**: +- https://192.168.100.1:5001 +- Should show updated frontend +- Login page loads +- No console errors + +### 9. Check Plugin Loading + +```bash +docker compose exec api flask plugin list +``` + +**Verify:** +- Expected plugins loaded +- No import errors +- Enabled plugins show as active + +If plugins fail to load: +- Check logs for import errors +- Verify plugin dependencies installed +- Check for circular imports + +### 10. Summary Report + +Provide clear status: + +**[X] Success:** +``` +Docker rebuild complete: +[X] Frontend built (3.2MB) +[X] Docker image rebuilt (742MB) +[X] Containers started and healthy +[X] Migrations applied (3 core, 5 plugin) +[X] All plugins loaded (12 active) +[X] URLs accessible + +Services ready: +- Flask API: https://localhost:5001 +- Vite dev: http://localhost:5173 (if running) + +You can now test changes. +``` + +**WARNING: Issues detected:** +``` +Docker rebuild completed with warnings: +[X] Frontend built +[X] Docker image rebuilt +[X] Containers started +[ ] Migration failed: plugin 'endpoints' has circular dependency +WARNING: Plugin 'printers' failed to load (import error) + +Action needed: +1. Check endpoints migration: plugins/endpoints/migrations/versions/ +2. Check printers import error in logs + +Logs: docker compose logs api --tail 50 +``` + +## Troubleshooting + +**Build takes too long (>5 min):** +- Docker cache may be invalid +- Try: `docker compose build --no-cache api` + +**Container won't start:** +- Check port conflicts (5001 already in use) +- Check Docker disk space: `docker system df` +- View logs: `docker compose logs api` + +**Migrations fail:** +- Verify migration chain linearity +- Check PLUGIN_TABLE_OWNERS coordination +- May need to drop test database and rebuild + +**Frontend not updating:** +- Verify `npm run build` ran successfully +- Check frontend/dist/ has latest files +- May need to clear browser cache + +## Related + +- `/pre-commit` - Pre-commit validation +- `/production-health` - Deep health check +- `docker-compose.prebuilt.yml` - Prebuilt Docker compose file +- [[docker-prebuilt-approach]] - Why we use prebuilt frontend +- ADR-008: Plugin migration ownership diff --git a/.claude/skills/full-deploy.md b/.claude/skills/full-deploy.md new file mode 100644 index 00000000..b8af61ec --- /dev/null +++ b/.claude/skills/full-deploy.md @@ -0,0 +1,577 @@ +--- +name: full-deploy +description: Complete deployment workflow for air-gapped production - validation, bundle creation, deployment, verification (saves 50-100 hrs/year from 312 hrs burden) +--- + +# Full Production Deployment Workflow + +Complete end-to-end deployment for air-gapped ShopDB production environment. Addresses 312 hrs/year coordination burden. + +## What This Skill Does + +Orchestrates complete deployment: +1. Pre-deploy validation (tests, health checks, security) +2. Create deployment bundle +3. Deploy to dev PC first (staging test) +4. Generate production deployment runbook +5. Transfer bundle across air gap +6. Deploy to production +7. Post-deploy validation +8. Rollback plan if issues detected + +## Air-Gap Environment + +``` ++--------------+ +---------------+ +------------+ +| Dev PC | Bundle | Work PC | Bundle | GitHub | +| (gitea) | ------> | (GitHub) | ------> | (public) | +| | | YOU ARE HERE | | | ++--------------+ +---------------+ +------------+ + | | + Dev Server Prod Server + (ops branch) (wj branch) +``` + +**Constraints:** +- No direct GitHub access from dev PC +- Manual bundle transfer between environments +- Production changes must be tested on dev first +- Work PC is coordination point + +## Prerequisites + +**Before starting deployment:** +- All changes committed to wj branch +- Tests passing (`pytest -v`) +- Style check passing (`scripts/check-naming-and-style.sh`) +- Frontend built (`npm run build`) +- No uncommitted changes (`git status` clean) + +## Workflow + +### Phase 1: Pre-Deploy Validation (Work PC) + +**1.1 Run Complete Validation Suite:** + +```bash +# Health check current state +/production-health + +# Security scan +/security-scan + +# Migration validation +/migration-check + +# Pre-commit checks +/pre-commit +``` + +**Expected:** All checks pass with no critical issues. + +**If validation fails:** +- Document failures +- Fix issues before proceeding +- Re-run validation +- Do NOT proceed with deployment if critical issues remain + +**1.2 Create Deployment Tag:** + +```bash +# Tag format: vYYYY.MM.DD-N (N = deployment number that day) +TAG="v$(date +%Y.%m.%d)-1" +git tag -a $TAG -m "Production deployment $(date +%Y-%m-%d)" +git push origin $TAG +``` + +**1.3 Document Changes Since Last Deploy:** + +```bash +# Get last deployment tag +LAST_TAG=$(git describe --tags --abbrev=0 HEAD^) + +# Generate changelog +git log $LAST_TAG..HEAD --oneline --no-merges > deployment-changelog.txt + +# Show commit summary +echo "Changes since last deploy ($LAST_TAG):" +git log $LAST_TAG..HEAD --oneline --no-merges | wc -l +echo "commits" +``` + +Review changelog for: +- Breaking changes +- Database migrations +- New dependencies +- Configuration changes + +### Phase 2: Create Deployment Bundle (Work PC) + +**2.1 Build Frontend:** + +```bash +cd frontend +npm run build +cd .. +``` + +Verify `frontend/dist/` updated with production build. + +**2.2 Create Bundle Archive:** + +```bash +BUNDLE_DATE=$(date +%Y%m%d_%H%M) +BUNDLE_NAME="shopdb-deploy-$BUNDLE_DATE.tar.gz" + +# Include everything except .git, node_modules, venv +tar -czf ~/_transfer/$BUNDLE_NAME \ + --exclude='.git' \ + --exclude='node_modules' \ + --exclude='venv' \ + --exclude='*.pyc' \ + --exclude='__pycache__' \ + --exclude='frontend/dist' \ + . + +# Separately bundle frontend (large) +tar -czf ~/_transfer/shopdb-frontend-$BUNDLE_DATE.tar.gz \ + frontend/dist/ + +echo "Bundles created:" +ls -lh ~/_transfer/shopdb-*$BUNDLE_DATE.tar.gz +``` + +**2.3 Generate Deployment Runbook:** + +Create `deployment-runbook-$BUNDLE_DATE.md`: + +```markdown +# ShopDB Deployment Runbook + +**Date:** $(date +%Y-%m-%d %H:%M) +**Bundle:** shopdb-deploy-$BUNDLE_DATE.tar.gz +**Tag:** $TAG +**Target:** Production (wj branch) + +## Pre-Deployment + +- [ ] Backup production database +- [ ] Create VM snapshot: /vm-snapshot create "before-deploy-$BUNDLE_DATE" +- [ ] Verify production health: /production-health +- [ ] Notify users of maintenance window + +## Deployment Steps + +### 1. Extract Bundle + +```bash +cd /path/to/shopdb-production +tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz +tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz +``` + +### 2. Install Dependencies (if changed) + +Check requirements.txt and package.json for changes: +```bash +# Python dependencies +pip install -r requirements.txt + +# Node dependencies (if needed) +cd frontend && npm ci && cd .. +``` + +### 3. Run Database Migrations + +```bash +docker compose exec api flask db upgrade +docker compose exec api flask plugin upgrade-all +``` + +**Watch for:** +- Migration errors +- Foreign key violations +- Timeout on large migrations + +### 4. Restart Services + +```bash +docker compose restart api +``` + +Verify restart: +```bash +docker compose ps +docker compose logs api --tail 50 +``` + +### 5. Verify Deployment + +- [ ] All containers running +- [ ] API responding: curl -k https://localhost:5001/api/health +- [ ] Frontend loads: https://localhost:5001 +- [ ] Login works +- [ ] Recent endpoints reporting + +### 6. Monitor for Issues + +```bash +# Watch logs for 5 minutes +docker compose logs -f api +``` + +## Post-Deployment + +- [ ] Verify endpoints reporting (check last_seen timestamps) +- [ ] Test critical workflows (asset creation, search) +- [ ] Check plugin loading: docker compose exec api flask plugin list +- [ ] Notify users deployment complete + +## Rollback Plan + +If issues detected: + +1. Restore VM snapshot: + ```bash + /vm-snapshot restore "before-deploy-$BUNDLE_DATE" + ``` + +2. Or rollback git: + ```bash + git checkout $LAST_TAG + docker compose down + docker compose up -d + ``` + +3. Restore database backup (if schema changed) + +## Changes in This Deployment + +$(git log $LAST_TAG..HEAD --oneline --no-merges) + +## Database Migrations + +$(docker compose exec api flask db history | grep -A 5 "current") + +## Testing Results + +- pytest: PASS (42 tests, 0 failures) +- Style check: PASS +- Security scan: PASS (0 critical, 0 high) +- Migration check: PASS (linear chains, no conflicts) +``` + +**2.4 Bundle Verification:** + +```bash +# Verify bundle integrity +tar -tzf ~/_transfer/$BUNDLE_NAME | head -20 + +# Check bundle size +du -h ~/_transfer/$BUNDLE_NAME +``` + +Expected: < 50MB (excluding frontend) +Expected frontend: < 20MB + +### Phase 3: Stage Deploy on Dev PC (Optional but Recommended) + +**3.1 Transfer Bundle to Dev PC:** + +(Manual: Copy bundle via USB, shared drive, or approved transfer method) + +**3.2 Deploy to Dev Server (ops branch):** + +```bash +# On dev PC +cd /path/to/shopdb-dev +tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz +tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz + +# Run deployment steps +docker compose exec api flask db upgrade +docker compose exec api flask plugin upgrade-all +docker compose restart api + +# Verify +/production-health +``` + +**3.3 Smoke Test on Dev:** + +- Test all plugin pages load +- Create test asset +- Run full test suite +- Check for errors in logs + +**If dev deploy fails:** +- Fix issues +- Create new bundle +- Re-test on dev +- Do NOT proceed to production + +### Phase 4: Production Deployment + +**4.1 Create Production Backup:** + +```bash +# Database backup +docker compose exec db mysqldump -u root -p shopdb > ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql + +# VM snapshot +/vm-snapshot create "before-deploy-$BUNDLE_DATE" +``` + +**4.2 Notify Users:** + +``` +Maintenance Window: [start time] - [end time] +ShopDB will be unavailable during deployment. +Expected downtime: 15-30 minutes +``` + +**4.3 Execute Deployment Runbook:** + +Follow `deployment-runbook-$BUNDLE_DATE.md` steps exactly. + +**4.4 Monitor Deployment:** + +Watch logs during restart: +```bash +docker compose logs -f api +``` + +Look for: +- [X] "Starting gunicorn" +- [X] "Listening at https://0.0.0.0:5001" +- [X] "Loaded plugins: 12" +- [ ] Exceptions or errors + +**4.5 Initial Verification:** + +```bash +# Quick checks (< 2 min) +docker compose ps # All running? +curl -k https://localhost:5001/api/health # API responds? +curl -k https://localhost:5001/ # Frontend loads? +``` + +### Phase 5: Post-Deploy Validation + +**5.1 Run Health Checks:** + +```bash +/production-health +``` + +**Expected:** All checks pass, no critical errors. + +**5.2 Verify Critical Workflows:** + +Test each critical workflow: + +1. **Login:** + - Test admin login + - Test standard user login + - Verify sessions work + +2. **Asset Management:** + - Create test asset + - Search for asset + - Update asset + - Delete test asset + +3. **Endpoint Reporting:** + - Check endpoint list loads + - Verify recent heartbeats (< 10 min) + - Check endpoint details page + +4. **Plugin Loading:** + ```bash + docker compose exec api flask plugin list + ``` + Expected: All plugins loaded, no import errors + +**5.3 Monitor for 1 Hour:** + +Watch logs for anomalies: +```bash +# Check every 10 minutes for first hour +docker compose logs api --since 10m | grep -i "error\|exception\|critical" +``` + +**5.4 Verify Endpoint Agents Still Reporting:** + +```bash +# Check last_seen timestamps +docker compose exec api flask shell -c " +from plugins.endpoints.models import Endpoint +from datetime import datetime, timedelta +recent = Endpoint.query.filter( + Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10) +).count() +print(f'Endpoints reporting in last 10 min: {recent}') +" +``` + +Expected: At least 1 (VM should report) + +### Phase 6: Rollback (If Issues Detected) + +**Rollback Decision Criteria:** + +**MUST rollback if:** +- Database migration failed +- Critical plugin won't load +- API completely non-functional +- Data integrity issues + +**SHOULD rollback if:** +- Multiple endpoints not reporting +- Critical workflow broken +- Performance severely degraded +- Security vulnerability introduced + +**Can monitor and fix if:** +- Single plugin has minor issue +- Non-critical feature broken +- Performance slightly slower +- Cosmetic UI issues + +**Rollback Procedure:** + +**Option A: VM Snapshot Restore (Fastest)** +```bash +/vm-snapshot restore "before-deploy-$BUNDLE_DATE" +``` + +Time: ~5 minutes +Scope: Complete system rollback + +**Option B: Git Rollback** +```bash +git checkout $LAST_TAG +docker compose down +docker compose -f docker-compose.prebuilt.yml build api +docker compose up -d +``` + +Time: ~10 minutes +Scope: Code only (database may need restore) + +**Option C: Database Restore (If Schema Changed)** +```bash +# Restore database backup +docker compose exec db mysql -u root -p shopdb < ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql + +# Or point-in-time recovery if binlog enabled +``` + +Time: ~15-30 minutes depending on database size + +**After Rollback:** +1. Verify system functional +2. Document what went wrong +3. Fix issues before retry +4. Notify users system restored + +### Phase 7: Deployment Complete + +**7.1 Update Documentation:** + +- Document any issues encountered +- Update deployment notes +- Record deployment time and downtime +- Note any manual steps required + +**7.2 Notify Stakeholders:** + +``` +Deployment complete: ShopDB v$TAG +- Deployment time: [duration] +- Downtime: [actual downtime] +- Status: Success / Issues noted +- Changes: [link to changelog] +``` + +**7.3 Monitor for 24 Hours:** + +Set reminder to check: +- Logs for errors +- Endpoint reporting +- User feedback +- Performance metrics + +**7.4 Cleanup:** + +```bash +# Remove old bundles (> 30 days) +find ~/_transfer/ -name "shopdb-deploy-*.tar.gz" -mtime +30 -delete + +# Remove old backups (> 30 days) +find ~/backups/ -name "shopdb-pre-deploy-*.sql" -mtime +30 -delete + +# Clean up old VM snapshots +/vm-snapshot cleanup +``` + +## Time Estimates + +| Phase | Time | Can Fail? | +|-------|------|-----------| +| Pre-deploy validation | 10-15 min | Yes (fix issues) | +| Bundle creation | 5 min | No | +| Stage deploy (dev) | 15-20 min | Yes (fix before prod) | +| Production backup | 5 min | No | +| Production deploy | 15-20 min | Yes (rollback) | +| Post-deploy validation | 15-30 min | Yes (rollback) | +| **Total** | **65-100 min** | | + +**Downtime:** 5-15 minutes (during production deploy phase) + +## Automation Opportunities + +**Current:** 312 hrs/year coordination burden + +**With this skill:** ~50-100 hrs saved/year + +**Further automation:** +- Automated bundle transfer (if air-gap can be bridged) +- Automated staging tests +- Automated rollback triggers +- Continuous deployment (requires air-gap solution) + +## Troubleshooting + +**Bundle extraction fails:** +- Check disk space: `df -h` +- Verify bundle integrity: `tar -tzf bundle.tar.gz` +- Try smaller extraction: `tar -xzf bundle.tar.gz --one-top-level` + +**Migration fails:** +- Check database connectivity +- Verify migration order (ADR-008) +- Check for data that violates new constraints +- See `/migration-check` for detailed validation + +**Services won't start:** +- Check logs: `docker compose logs api` +- Verify port availability: `netstat -tulpn | grep 5001` +- Check Docker status: `docker ps -a` +- Try rebuild: `/docker-rebuild` + +**Rollback fails:** +- VM snapshot may be corrupted +- Try alternative rollback method +- Contact system admin if all methods fail + +## Related + +- `/pre-commit` - Pre-deployment validation +- `/production-health` - Health monitoring +- `/migration-check` - Database migration validation +- `/docker-rebuild` - Container rebuild +- `/vm-snapshot` - VM snapshot management +- [[workflow-patterns]] - Common workflows +- [[maintenance-footprint-baseline]] - 312 hrs/year coordination burden +- ADR-004: Deployment topology diff --git a/.claude/skills/merge-wilm.md b/.claude/skills/merge-wilm.md index 28dc57f9..74755fbd 100644 --- a/.claude/skills/merge-wilm.md +++ b/.claude/skills/merge-wilm.md @@ -31,7 +31,7 @@ Merge the latest commits from wilm branch into wj branch, resolving common confl 5. Resolve csproj conflicts: - Keep ALL version history comments from both branches - Merge version numbers by taking higher version and bumping patch - - Example: wj has 1.13.5, wilm has 1.14.0 → result is 1.14.1 or 1.14.2 + - Example: wj has 1.13.5, wilm has 1.14.0 -> result is 1.14.1 or 1.14.2 - Add merge comment explaining what was combined 6. Resolve routes.py conflicts: @@ -72,7 +72,7 @@ Merge the latest commits from wilm branch into wj branch, resolving common confl - Agent and tray MUST be bumped together - Format: X.Y.Z - Merge scenarios: - - wj 1.13.5 + wilm 1.14.0 → 1.14.1 (take wilm minor, bump patch) + - wj 1.13.5 + wilm 1.14.0 -> 1.14.1 (take wilm minor, bump patch) - Add comment: "Merge wilm X.Y.Z changes with wj coordination releases" ## Notes diff --git a/.claude/skills/migration-check.md b/.claude/skills/migration-check.md new file mode 100644 index 00000000..06a7e29f --- /dev/null +++ b/.claude/skills/migration-check.md @@ -0,0 +1,328 @@ +--- +name: migration-check +description: Deep validation of plugin migrations for ADR-008 compliance - check linearity, dependencies, foreign keys +--- + +# Migration Safety Check + +Comprehensive validation of database migrations to prevent ADR-008 violations and production failures. + +## What This Skill Does + +Deep migration analysis: +1. Verify each plugin migration chain is linear (no branches) +2. Check for circular dependencies between plugins +3. Validate foreign key references exist +4. Test migrations up AND down +5. Verify PLUGIN_TABLE_OWNERS coordination +6. Check for duplicate version numbers +7. Identify orphaned migrations + +## Why This Matters + +**ADR-008: Plugin Migration Ownership** +- Each plugin owns its migration chain +- Chains must be linear (no branches) +- Migrations run in dependency order (PLUGIN_TABLE_OWNERS) +- Violations cause production deployment failures + +**Historical issues:** +- Plugin migration branches blocked deployments +- Circular dependencies caused deadlocks +- Missing foreign keys led to data integrity issues + +## Workflow + +### 1. List All Migrations + +Scan all plugin migration directories: + +```bash +find plugins/*/migrations/versions/ -name "*.py" -type f | sort +``` + +**For each plugin:** +- Count migration files +- Check naming convention (revision_description.py) +- Verify chronological order by timestamp + +### 2. Check Chain Linearity + +For each plugin, verify migration chain has no branches: + +```python +# Pseudo-code logic: +for plugin in plugins: + migrations = get_migrations(plugin) + + # Build dependency graph + graph = {} + for migration in migrations: + parent = migration.down_revision + graph[parent] = graph.get(parent, []) + [migration.revision] + + # Check for branches (parent with multiple children) + for parent, children in graph.items(): + if len(children) > 1: + ERROR: Branch detected at {parent} + Children: {children} +``` + +**What to flag:** +- Multiple migrations with same down_revision (branch) +- Migrations with no parent (orphaned) +- Migrations with missing parent (broken chain) + +**Example violation:** +``` +ERROR: Plugin 'endpoints' has branched migration chain: + Parent: a1b2c3d4 + Branch 1: e5f6g7h8 (Add certificate_thumbprint column) + Branch 2: i9j0k1l2 (Add last_login_at column) + + ADR-008 violation: Merge these branches or drop one +``` + +### 3. Check Plugin Dependencies + +Verify PLUGIN_TABLE_OWNERS coordination: + +```bash +grep -A 50 "PLUGIN_TABLE_OWNERS" shopdb/config.py +``` + +**What to check:** +- Plugins listed in dependency order +- If Plugin A references Plugin B's table, B must come first +- No circular dependencies (A depends on B, B depends on A) + +**Build dependency graph:** +```python +# For each plugin migration: +for migration in all_migrations: + # Check for foreign keys to other plugins + if references_other_plugin_table(migration): + add_dependency(current_plugin, other_plugin) + +# Check for cycles +if has_circular_dependency(dependencies): + ERROR: Circular dependency detected +``` + +**Example violation:** +``` +ERROR: Circular plugin dependencies: + endpoints -> computers (fk: computer_id) + computers -> endpoints (fk: endpoint_id) + + ADR-008 violation: Break circular reference + Recommendation: Create junction table or denormalize +``` + +### 4. Validate Foreign Key References + +For each foreign key in migrations, verify target table exists: + +```python +# Check op.create_foreign_key() calls +for migration in all_migrations: + for fk in get_foreign_keys(migration): + target_table = fk.target_table + target_plugin = get_owning_plugin(target_table) + + # Verify target plugin comes before current in PLUGIN_TABLE_OWNERS + if not comes_before(target_plugin, current_plugin): + ERROR: Invalid foreign key order +``` + +**What to flag:** +- Foreign key to table owned by later plugin +- Foreign key to non-existent table +- Foreign key without index on source column + +**Example violation:** +``` +WARNING: Plugin 'printers' references 'computers' table + Migration: 2024_03_15_add_computer_id.py + Foreign key: printer.computer_id -> computer.id + + PLUGIN_TABLE_OWNERS order: + printers (index 5) + computers (index 8) + + ERROR: printers loads before computers (table won't exist) + Fix: Move printers after computers in PLUGIN_TABLE_OWNERS +``` + +### 5. Test Migrations Up + +Apply all migrations in test database: + +```bash +# Create test database +docker compose exec api flask db upgrade + +# Upgrade each plugin +docker compose exec api flask plugin upgrade-all +``` + +**Monitor for:** +- Alembic errors +- Foreign key constraint violations +- Circular dependency deadlocks +- Duplicate table/column errors + +If migrations fail: +- Show exact error +- Identify which migration failed +- Suggest fix based on error type + +### 6. Test Migrations Down + +Verify migrations can roll back cleanly: + +```bash +# Downgrade plugins in reverse order +for plugin in reverse(PLUGIN_TABLE_OWNERS): + docker compose exec api flask plugin downgrade $plugin base +done + +# Downgrade core +docker compose exec api flask db downgrade base +``` + +**What to check:** +- All downgrades succeed +- No orphaned tables/columns +- Foreign keys dropped before tables + +**Common issues:** +- Downgrade drops table but doesn't drop foreign keys first +- Downgrade missing (NotImplementedError) +- Downgrade order violation + +### 7. Check for Duplicate Versions + +Ensure no two migrations share same revision ID: + +```bash +# Extract all revision IDs +grep -r "revision = " plugins/*/migrations/versions/*.py | cut -d"'" -f2 | sort | uniq -d +``` + +**If duplicates found:** +``` +ERROR: Duplicate revision IDs detected: + a1b2c3d4 in plugins/endpoints/migrations/versions/001_initial.py + a1b2c3d4 in plugins/computers/migrations/versions/003_add_field.py + + Fix: Regenerate one migration with new revision ID +``` + +### 8. Identify Orphaned Migrations + +Find migrations with parents that don't exist: + +```python +# Build set of all revision IDs +all_revisions = set(migration.revision for migration in all_migrations) + +# Check each migration's parent exists +for migration in all_migrations: + if migration.down_revision and migration.down_revision not in all_revisions: + ERROR: Orphaned migration {migration.revision} + Parent {migration.down_revision} not found +``` + +**Causes:** +- Parent migration deleted +- Migration from different branch merged incorrectly +- Manual migration editing gone wrong + +### 9. Summary Report + +**[X] All checks passed:** +``` +Migration safety check complete: + +Plugin Migration Chains: +[X] endpoints: 12 migrations, linear +[X] computers: 8 migrations, linear +[X] printers: 5 migrations, linear +[X] (... all plugins ...) + +Dependency Analysis: +[X] No circular dependencies +[X] All foreign keys valid +[X] PLUGIN_TABLE_OWNERS order correct + +Migration Tests: +[X] All migrations applied successfully +[X] All downgrades succeeded +[X] No duplicate revision IDs + +Migrations are safe to deploy! +``` + +**WARNING: Issues detected:** +``` +Migration safety check FAILED: + +CRITICAL ISSUES (block deployment): +1. Plugin 'endpoints' has branched migration chain + Parent a1b2c3d4 has 2 children + Action: Merge branches or drop conflicting migration + +2. Circular dependency: endpoints -> computers + Action: Break circular reference with junction table + +3. Foreign key order violation: printers -> computers + printers loads before computers in PLUGIN_TABLE_OWNERS + Action: Reorder plugins or remove foreign key + +HIGH PRIORITY (should fix before deploy): +4. Migration 'endpoints/003_add_field' has no downgrade + Action: Implement down() method + +5. Orphaned migration: computers/005_update + Parent revision not found + Action: Fix down_revision or remove migration + +WARNINGS (review but not blocking): +6. Large migration: endpoints/012_backfill (may be slow) + Action: Consider data migration strategy + +DO NOT DEPLOY until critical issues resolved! + +Detailed logs: /tmp/migration-check-{timestamp}.log +``` + +## Troubleshooting + +**Branch detected:** +- Option 1: Squash branch into single migration +- Option 2: Drop conflicting branch +- Option 3: Reorder to make linear + +**Circular dependency:** +- Create junction table (owned by one plugin) +- Denormalize (copy data instead of FK) +- Move table ownership to core + +**Foreign key order violation:** +- Reorder PLUGIN_TABLE_OWNERS +- Remove foreign key (use app-level relationship) +- Move both tables to same plugin + +**Migration fails to apply:** +- Check for missing dependencies +- Verify table/column doesn't already exist +- Check for data that violates constraints + +## Related + +- `/pre-commit` - Includes basic migration linearity check +- `/production-health` - Post-deployment validation +- ADR-008: Plugin migration ownership +- `PLUGIN_TABLE_OWNERS` in shopdb/config.py +- Alembic documentation: https://alembic.sqlalchemy.org/ diff --git a/.claude/skills/plugin-new.md b/.claude/skills/plugin-new.md new file mode 100644 index 00000000..53b054f7 --- /dev/null +++ b/.claude/skills/plugin-new.md @@ -0,0 +1,707 @@ +--- +name: plugin-new +description: Scaffold new ShopDB plugin following ADRs - directory structure, BasePlugin boilerplate, migrations, Vue component +--- + +# New Plugin Scaffolding + +Generate complete plugin structure following ShopDB architecture (ADR-001, ADR-008, ADR-010). + +## What This Skill Does + +Creates production-ready plugin scaffold: +1. Plugin directory structure +2. BasePlugin subclass with manifest +3. SQLAlchemy models directory +4. Flask API blueprint +5. Alembic migrations directory +6. Vue frontend component +7. README with checklist +8. Adds plugin to PLUGIN_TABLE_OWNERS + +Saves 30-60 minutes of setup per new plugin. + +## Usage + +`/plugin-new "plugin-name" "description"` + +Example: `/plugin-new "tooling" "Track hand tools and tooling inventory"` + +## Workflow + +### 1. Validate Plugin Name + +**Check naming conventions:** +- Lowercase, no underscores +- Singular form ("tool" not "tools", exception: existing plurals like "computers") +- No spaces or special characters +- Not a Python reserved word + +**Check plugin doesn't exist:** +```bash +if [ -d "plugins/$plugin_name" ]; then + echo "ERROR: Plugin '$plugin_name' already exists" + exit 1 +fi +``` + +**Suggest corrections:** +- "tool_tracking" -> "tooling" (remove underscore) +- "Tools" -> "tools" (lowercase) +- "hand-tools" -> "handtools" (remove hyphen) + +### 2. Create Directory Structure + +```bash +mkdir -p plugins/$plugin_name/{api,models,frontend/views,migrations/versions,tests} +``` + +**Expected structure:** +``` +plugins/tooling/ + __init__.py + plugin.py # BasePlugin subclass + api/ + __init__.py + routes.py # Flask blueprint + models/ + __init__.py + tool.py # SQLAlchemy model + frontend/ + views/ + ToolList.vue # Main Vue component + migrations/ + env.py # Alembic environment + script.py.mako # Migration template + versions/ # Migration files + tests/ + __init__.py + test_tooling.py # Plugin tests + README.md # Plugin documentation +``` + +### 3. Generate BasePlugin Subclass + +**File:** `plugins/$plugin_name/plugin.py` + +```python +""" +$Plugin_Name Plugin + +$Description +""" +from shopdb.api import BasePlugin + + +class ${Plugin_Name}Plugin(BasePlugin): + """$Description""" + + name = "$plugin_name" + display_name = "$Display Name" + description = "$description" + version = "0.1.0" + author = "ShopDB Team" + + # Dependencies (other plugins this depends on) + dependencies = [] + + # Frontend routes + frontend_routes = [ + { + "path": "/$plugin_name", + "name": "$Plugin_Name", + "component": "$Plugin_NameList", + "meta": {"requiresAuth": True, "title": "$Display Name"} + } + ] + + # Sidebar menu entry + sidebar_items = [ + { + "title": "$Display Name", + "icon": "mdi-tools", # Material Design Icon + "path": "/$plugin_name", + "permission": "$plugin_name.view" + } + ] + + # Permissions this plugin defines + permissions = [ + ("$plugin_name.view", "View $display_name"), + ("$plugin_name.create", "Create $display_name"), + ("$plugin_name.edit", "Edit $display_name"), + ("$plugin_name.delete", "Delete $display_name"), + ] + + def register_blueprints(self, app): + """Register Flask blueprints""" + from .api.routes import bp + app.register_blueprint(bp, url_prefix=f"/api/$plugin_name") + + def init_app(self, app): + """Initialize plugin with Flask app""" + self.register_blueprints(app) +``` + +**Variables to substitute:** +- `$plugin_name`: "tooling" +- `$Plugin_Name`: "Tooling" +- `$Display Name`: "Tool Tracking" +- `$description`: User-provided description + +### 4. Create SQLAlchemy Model + +**File:** `plugins/$plugin_name/models/$model_name.py` + +```python +""" +$Model_Name model for $plugin_name plugin +""" +from shopdb.api import db, Asset + + +class $Model_Name(Asset): + """$Description""" + + __tablename__ = "$tablename" + __mapper_args__ = {"polymorphic_identity": "$plugin_name"} + + # Polymorphic inheritance from Asset (ADR-001) + id = db.Column(db.Integer, db.ForeignKey('asset.id'), primary_key=True) + + # Plugin-specific fields + tool_type = db.Column(db.String(100), nullable=False) + manufacturer = db.Column(db.String(100)) + model_number = db.Column(db.String(100)) + serial_number = db.Column(db.String(100), unique=True) + purchase_date = db.Column(db.Date) + status = db.Column(db.String(50), default='available') + + # Relationships (example: assigned to employee) + assigned_to_id = db.Column(db.Integer, db.ForeignKey('employee.id')) + assigned_to = db.relationship('Employee', backref='tools') + + def __repr__(self): + return f"<$Model_Name {self.serial_number}>" + + def to_dict(self): + """Serialize model to dictionary""" + data = super().to_dict() # Get Asset base fields + data.update({ + 'tool_type': self.tool_type, + 'manufacturer': self.manufacturer, + 'model_number': self.model_number, + 'serial_number': self.serial_number, + 'purchase_date': self.purchase_date.isoformat() if self.purchase_date else None, + 'status': self.status, + 'assigned_to': self.assigned_to.to_dict() if self.assigned_to else None + }) + return data +``` + +**Models/__init__.py:** +```python +from .tool import Tool + +__all__ = ['Tool'] +``` + +### 5. Create Flask API Blueprint + +**File:** `plugins/$plugin_name/api/routes.py` + +```python +""" +API routes for $plugin_name plugin +""" +from flask import Blueprint, request, jsonify +from shopdb.api import db, requires_auth +from ..models import $Model_Name + +bp = Blueprint('$plugin_name', __name__) + + +@bp.route('/', methods=['GET']) +@requires_auth +def list_$plural(): + """List all $plural""" + items = $Model_Name.query.all() + return jsonify([item.to_dict() for item in items]) + + +@bp.route('/', methods=['GET']) +@requires_auth +def get_$singular(id): + """Get specific $singular by ID""" + item = $Model_Name.query.get_or_404(id) + return jsonify(item.to_dict()) + + +@bp.route('/', methods=['POST']) +@requires_auth +def create_$singular(): + """Create new $singular""" + data = request.json + + # Validate required fields + required = ['tool_type', 'serial_number'] + if not all(field in data for field in required): + return jsonify({'error': 'Missing required fields'}), 400 + + # Create item + item = $Model_Name(**data) + db.session.add(item) + db.session.commit() + + return jsonify(item.to_dict()), 201 + + +@bp.route('/', methods=['PUT']) +@requires_auth +def update_$singular(id): + """Update existing $singular""" + item = $Model_Name.query.get_or_404(id) + data = request.json + + # Update fields + for key, value in data.items(): + if hasattr(item, key): + setattr(item, key, value) + + db.session.commit() + return jsonify(item.to_dict()) + + +@bp.route('/', methods=['DELETE']) +@requires_auth +def delete_$singular(id): + """Delete $singular""" + item = $Model_Name.query.get_or_404(id) + db.session.delete(item) + db.session.commit() + + return '', 204 +``` + +### 6. Initialize Alembic Migrations + +**File:** `plugins/$plugin_name/migrations/env.py` + +Copy from template (similar to other plugins) + +**File:** `plugins/$plugin_name/migrations/script.py.mako` + +Copy from template + +**Create initial migration:** +```bash +docker compose exec api flask plugin revision -p $plugin_name -m "Initial $plugin_name schema" +``` + +This generates: `plugins/$plugin_name/migrations/versions/001_initial_schema.py` + +**Verify migration:** +- Check revision ID generated +- Verify down_revision is None (first migration) +- Verify upgrade() creates table +- Verify downgrade() drops table + +### 7. Create Vue Frontend Component + +**File:** `plugins/$plugin_name/frontend/views/$Plugin_NameList.vue` + +```vue + + + +``` + +### 8. Add to PLUGIN_TABLE_OWNERS + +**File:** `shopdb/config.py` + +```python +PLUGIN_TABLE_OWNERS = [ + 'employees', # Core dependency (if using Employee FK) + 'assets', # Core dependency (Asset polymorphism) + # ... existing plugins ... + '$plugin_name', # ADD NEW PLUGIN HERE +] +``` + +**Placement considerations:** +- After plugins you reference (foreign keys) +- Before plugins that reference you +- Alphabetical within same dependency level + +### 9. Create Plugin Tests + +**File:** `plugins/$plugin_name/tests/test_tooling.py` + +```python +""" +Tests for $plugin_name plugin +""" +import pytest +from shopdb.api import db +from plugins.$plugin_name.models import $Model_Name + + +def test_create_$singular(client, auth_header): + """Test creating $singular via API""" + response = client.post('/api/$plugin_name/', json={ + 'tool_type': 'Wrench', + 'serial_number': 'WR-001', + 'manufacturer': 'Craftsman' + }, headers=auth_header) + + assert response.status_code == 201 + data = response.json + assert data['tool_type'] == 'Wrench' + assert data['serial_number'] == 'WR-001' + + +def test_list_$plural(client, auth_header): + """Test listing all $plural""" + response = client.get('/api/$plugin_name/', headers=auth_header) + + assert response.status_code == 200 + assert isinstance(response.json, list) + + +def test_get_$singular(client, auth_header, test_$singular): + """Test getting specific $singular""" + response = client.get(f'/api/$plugin_name/{test_$singular.id}', headers=auth_header) + + assert response.status_code == 200 + data = response.json + assert data['id'] == test_$singular.id + + +def test_update_$singular(client, auth_header, test_$singular): + """Test updating $singular""" + response = client.put(f'/api/$plugin_name/{test_$singular.id}', json={ + 'status': 'in_use' + }, headers=auth_header) + + assert response.status_code == 200 + data = response.json + assert data['status'] == 'in_use' + + +def test_delete_$singular(client, auth_header, test_$singular): + """Test deleting $singular""" + response = client.delete(f'/api/$plugin_name/{test_$singular.id}', headers=auth_header) + + assert response.status_code == 204 + + # Verify deleted + response = client.get(f'/api/$plugin_name/{test_$singular.id}', headers=auth_header) + assert response.status_code == 404 + + +@pytest.fixture +def test_$singular(db_session): + """Create test $singular""" + $singular = $Model_Name( + tool_type='Hammer', + serial_number='TEST-001', + manufacturer='TestCo' + ) + db_session.add($singular) + db_session.commit() + return $singular +``` + +### 10. Create Plugin README + +**File:** `plugins/$plugin_name/README.md` + +```markdown +# $Display Name Plugin + +$description + +## Features + +- Track $plural with serial numbers +- Manage tool assignments +- Monitor tool status (available, in_use, maintenance, retired) + +## Models + +### $Model_Name + +Inherits from `Asset` (ADR-001: Asset as platform contract) + +**Fields:** +- `tool_type` (str): Type of tool +- `manufacturer` (str): Manufacturer name +- `model_number` (str): Model number +- `serial_number` (str, unique): Serial number +- `status` (str): Tool status + +## API Endpoints + +- `GET /api/$plugin_name/` - List all $plural +- `GET /api/$plugin_name/` - Get specific $singular +- `POST /api/$plugin_name/` - Create new $singular +- `PUT /api/$plugin_name/` - Update $singular +- `DELETE /api/$plugin_name/` - Delete $singular + +## Frontend + +Main component: `$Plugin_NameList.vue` +Route: `/$plugin_name` + +## Development Checklist + +- [ ] Models defined with proper relationships +- [ ] Initial migration created and tested +- [ ] API endpoints implemented with authentication +- [ ] Frontend component renders list and detail views +- [ ] Tests written and passing +- [ ] Added to PLUGIN_TABLE_OWNERS in correct order +- [ ] Documentation updated +- [ ] Permissions defined +- [ ] Sidebar menu item configured + +## Testing + +```bash +pytest plugins/$plugin_name/tests/ -v +``` + +## Migration + +```bash +# Create new migration +docker compose exec api flask plugin revision -p $plugin_name -m "description" + +# Apply migration +docker compose exec api flask plugin upgrade $plugin_name +``` +``` + +### 11. Summary Report + +``` +Plugin scaffolding complete: $plugin_name + +Created files: + [X] plugins/$plugin_name/plugin.py (BasePlugin subclass) + [X] plugins/$plugin_name/models/$model_name.py (SQLAlchemy model) + [X] plugins/$plugin_name/api/routes.py (Flask blueprint, 5 endpoints) + [X] plugins/$plugin_name/frontend/views/$Plugin_NameList.vue (Vue component) + [X] plugins/$plugin_name/migrations/ (Alembic initialized) + [X] plugins/$plugin_name/tests/test_$plugin_name.py (pytest tests) + [X] plugins/$plugin_name/README.md (documentation) + +Updated files: + [X] shopdb/config.py (added to PLUGIN_TABLE_OWNERS) + +Next steps: +1. Review generated code and customize as needed +2. Create initial migration: docker compose exec api flask plugin revision -p $plugin_name -m "Initial schema" +3. Apply migration: docker compose exec api flask plugin upgrade $plugin_name +4. Run tests: pytest plugins/$plugin_name/tests/ -v +5. Start dev server and test frontend: npm run dev +6. Commit changes: git add plugins/$plugin_name shopdb/config.py + +Time saved: ~45 minutes of manual scaffolding +``` + +## Customization + +After scaffolding, customize: + +**Models:** +- Add plugin-specific fields +- Define relationships to other plugins +- Add custom methods + +**API:** +- Add custom endpoints (search, filters, bulk operations) +- Implement business logic +- Add validation schemas + +**Frontend:** +- Customize table columns +- Add filters and search +- Implement detail views +- Add charts/visualizations + +**Tests:** +- Add tests for custom logic +- Test edge cases +- Add integration tests + +## Related + +- ADR-001: Asset as platform contract +- ADR-008: Plugin migration ownership +- ADR-010: Frontend plugin hooks +- `CLAUDE.md` - Plugin development guide +- `/pre-commit` - Test before committing diff --git a/.claude/skills/pre-commit.md b/.claude/skills/pre-commit.md new file mode 100644 index 00000000..c6c10bbf --- /dev/null +++ b/.claude/skills/pre-commit.md @@ -0,0 +1,186 @@ +--- +name: pre-commit +description: Complete pre-commit validation workflow - tests, style checks, build verification, ADR compliance +--- + +# Pre-Commit Validation + +Run complete validation workflow before committing changes. Catches issues early and ensures code quality. + +## What This Skill Does + +Comprehensive pre-commit checks: +1. Run full test suite (pytest -v) +2. Check naming and style conventions +3. Build frontend to catch errors +4. Verify both URLs work (Vite dev and Flask production) +5. Check for hardcoded site data (ADR-015 violations) +6. Verify plugin migrations are linear (ADR-008) +7. Summary of what's ready to commit + +## Workflow + +### 1. Run Backend Tests + +```bash +pytest -v +``` + +**What to check:** +- All tests pass +- No new test failures +- Migration tests pass (critical for ADR-008) + +If tests fail: +- Show which tests failed +- Provide targeted rerun command +- Ask if user wants to proceed anyway + +### 2. Check Naming and Style + +```bash +scripts/check-naming-and-style.sh +``` + +**What to check:** +- No unicode/emojis (plain ASCII only) +- Database table names lowercase concatenated +- Column names snake_case +- No LLM attribution comments +- No banned shorthand (e.g., "config" not "cfg") + +If style check fails: +- Show violations +- Offer to fix automatically (if simple) +- Explain ADR context for violations + +### 3. Build Frontend + +```bash +cd frontend +npm run build +cd .. +``` + +**What to check:** +- Build completes without errors +- No TypeScript/ESLint errors +- Bundle size reasonable (< 20MB) +- frontend/dist/ updated + +If build fails: +- Show build errors +- Check for common issues (missing imports, syntax errors) +- Suggest fixes + +### 4. Verify Both URLs Work + +Check that both development and production builds show same UI: +- Vite dev: http://192.168.100.1:5173 +- Flask prod: https://192.168.100.1:5001 + +**Quick verification:** +- Both URLs accessible +- Login page loads +- Sidebar shows same plugins +- No console errors in browser + +If URLs don't match: +- Flask may be serving old dist/ (need to restart) +- Vite proxy may be misconfigured + +### 5. Check for ADR-015 Violations + +Scan staged changes for hardcoded site-specific data: + +```bash +git diff --cached | grep -E "(WJF|WILM|Building [0-9]|Printer Model)" +``` + +**What to flag:** +- Building names ("Building 23", "WJF", "WILM") +- Printer model names +- Machine type names +- Site-specific configuration + +**Acceptable locations:** +- `shopdb/core/settings/` - Site settings +- `shopdb/core/seed/` - Seed data SQL +- `scripts/site_imports/` - Import tools + +If violations found: +- Show the lines +- Explain ADR-015: site data belongs in settings/seed/site_imports +- Suggest moving to appropriate location + +### 6. Check Migration Linearity (ADR-008) + +For each plugin with migration changes: + +```bash +# List migration files by date +ls -lt plugins/*/migrations/versions/*.py | head -20 +``` + +**What to verify:** +- Each plugin has linear chain (no branches) +- New migrations reference correct parent (down_revision) +- No duplicate version numbers +- PLUGIN_TABLE_OWNERS coordination if multiple plugins changed + +If issues found: +- Show the migration chain +- Identify branches or conflicts +- Explain ADR-008: plugin migration chains must be linear + +### 7. Summary and Recommendation + +Provide clear summary: + +**[X] All checks passed:** +``` +Pre-commit validation complete: +[X] Tests: 42 passed +[X] Style: No violations +[X] Frontend: Built successfully (3.2MB) +[X] URLs: Both working +[X] ADR-015: No hardcoded site data +[X] ADR-008: Migrations linear + +Ready to commit! +``` + +**WARNING: Some checks failed:** +``` +Pre-commit validation results: +[ ] Tests: 2 failed (test_endpoints_auth, test_computers_search) +[X] Style: No violations +[X] Frontend: Built successfully +[ ] ADR-015: Hardcoded building name in plugins/endpoints/api/routes.py:45 + +Recommendations: +1. Fix failing tests before committing +2. Move building name to shopdb/core/settings/buildings.py + +Proceed anyway? (Not recommended for production branch) +``` + +## When to Use This Skill + +**Always before:** +- Committing to wj (production branch) +- Creating a pull request +- Merging branches +- Creating a bundle for deployment + +**Optional for:** +- WIP commits on feature branches +- Quick fixes that will be squashed + +## Related + +- `/test-changes` - Lighter version (just tests + build) +- `/migration-check` - Deep migration validation +- `scripts/check-naming-and-style.sh` - Style checker script +- ADR-008: Plugin migration ownership +- ADR-015: Site-specific configuration diff --git a/.claude/skills/production-health.md b/.claude/skills/production-health.md new file mode 100644 index 00000000..bea93b14 --- /dev/null +++ b/.claude/skills/production-health.md @@ -0,0 +1,402 @@ +--- +name: production-health +description: Comprehensive health check of running system - services, plugins, database, API response times, logs +--- + +# Production Health Check + +Complete system health assessment for site lead confidence before/after deployments. + +## What This Skill Does + +Comprehensive health validation: +1. Check all services running (Docker containers) +2. Verify all plugins loaded and active +3. Check recent logs for errors +4. Test login flow works +5. Verify endpoints reporting +6. Check database connectivity and performance +7. Test API response times +8. Validate critical workflows + +## When to Use This Skill + +**Before deployment:** +- Baseline current health +- Ensure system stable before changes + +**After deployment:** +- Verify deployment succeeded +- Catch issues before users report them + +**Regular monitoring:** +- Daily health check +- After maintenance windows +- When investigating user reports + +## Workflow + +### 1. Check Docker Services + +```bash +docker compose ps +``` + +**Expected containers:** +- flask-shopdb-api-1: Up (healthy) +- flask-shopdb-db-1: Up (healthy) + +**What to flag:** +- Container stopped +- Container restarting +- Health check failing +- High restart count + +```bash +# Check uptime +docker compose ps --format "table {{.Service}}\t{{.Status}}" + +# Check restart count +docker inspect flask-shopdb-api-1 --format='{{.RestartCount}}' +``` + +**Warning threshold:** +- Restart count > 0 (investigate why it restarted) +- Uptime < 5 minutes (recently restarted) + +### 2. Verify Plugin Loading + +```bash +docker compose exec api flask plugin list +``` + +**Expected output:** +- All plugins listed +- Enabled plugins show as active +- No import errors + +**For each plugin, check:** +- Status: enabled/disabled +- Import succeeded (no errors) +- Version number +- Last updated timestamp + +**What to flag:** +``` +WARNING: Plugin 'endpoints' failed to load + Error: ImportError: cannot import name 'Asset' + Impact: Endpoint functionality unavailable + Action: Check plugin code, verify dependencies +``` + +### 3. Check Recent Logs for Errors + +Scan last 100 lines of API logs: + +```bash +docker compose logs api --tail 100 +``` + +**Pattern match for issues:** +- `ERROR` - Application errors +- `CRITICAL` - Critical failures +- `Exception` - Unhandled exceptions +- `Traceback` - Python stack traces +- `500 Internal Server Error` - API failures +- `Connection refused` - Database connectivity issues + +**Categorize errors:** +- **Critical**: Service crashes, database down, security breaches +- **High**: Plugin failures, API 500s, data corruption +- **Medium**: Performance warnings, deprecation notices +- **Low**: Info messages with ERROR level (false positives) + +**Report top errors:** +``` +Recent errors in logs (last 100 lines): +1. ERROR [endpoints] Failed to update endpoint status (count: 5) + Last seen: 2 minutes ago + Impact: Endpoint status may be stale + +2. WARNING [sqlalchemy] Connection pool exhausted (count: 2) + Last seen: 15 minutes ago + Impact: Slow API responses + +No critical errors detected. +``` + +### 4. Test Database Connectivity + +**Connection test:** +```bash +docker compose exec api flask shell -c "from shopdb.api import db; print(db.session.execute('SELECT 1').scalar())" +``` + +Expected: `1` + +**Connection pool status:** +```bash +docker compose exec api flask shell -c "from shopdb.api import db; print(db.engine.pool.status())" +``` + +**What to check:** +- Pool size vs. active connections +- Connection timeout rate +- Average query time + +**Query performance test:** +```bash +# Test simple query speed +time docker compose exec api flask shell -c "from shopdb.api import db, Asset; Asset.query.count()" +``` + +**Thresholds:** +- < 100ms: Good +- 100-500ms: Acceptable +- > 500ms: Investigate (table scans, missing indexes) + +### 5. Test API Endpoints + +**Health check:** +```bash +curl -k https://localhost:5001/api/health +``` + +Expected: `{"status": "healthy"}` in < 100ms + +**Authentication endpoint:** +```bash +curl -k https://localhost:5001/api/auth/check +``` + +Expected: 401 or 200 (depending on session) + +**Core API endpoints:** +```bash +# Test key endpoints (with timing) +curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/assets +curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/employees +curl -k -w "\nTime: %{time_total}s\n" https://localhost:5001/api/plugins +``` + +**Response time thresholds:** +- < 200ms: Good +- 200-500ms: Acceptable +- 500ms-1s: Slow (investigate) +- > 1s: Performance issue + +### 6. Test Login Flow + +Automated login test: + +```bash +# POST login request +curl -k -X POST https://localhost:5001/api/auth/login \ + -H "Content-Type: application/json" \ + -d '{"username":"testadmin","password":"admin123"}' \ + -c /tmp/cookies.txt + +# Verify session cookie received +grep session /tmp/cookies.txt +``` + +**Expected:** +- 200 OK response +- Session cookie set +- User data returned (id, username, roles) + +**What to flag:** +- 500 error (authentication broken) +- 401 error with correct credentials (database issue) +- No session cookie (session middleware broken) +- Slow response > 1s (database slow) + +### 7. Verify Endpoints Reporting + +Check that endpoint agent is reporting: + +```bash +# Count active endpoints +docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; print(f'Active endpoints: {Endpoint.query.filter_by(status=\"active\").count()}')" + +# Check recent heartbeats (last 10 min) +docker compose exec api flask shell -c "from shopdb.api import db; from plugins.endpoints.models import Endpoint; from datetime import datetime, timedelta; recent = Endpoint.query.filter(Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)).count(); print(f'Recent heartbeats: {recent}')" +``` + +**What to check:** +- At least 1 endpoint active (VM should be reporting) +- Recent heartbeats > 0 (agent connected in last 10 min) +- No endpoints with status 'error' + +**If no endpoints reporting:** +- Check agent service on VM +- Verify network connectivity (firewall) +- Check API collector authentication + +### 8. Check Frontend Accessibility + +**Production build (Flask serves):** +```bash +curl -k -I https://localhost:5001/ | head -5 +``` + +Expected: 200 OK, Content-Type: text/html + +**Vite dev server (if running):** +```bash +curl -I http://localhost:5173/ | head -5 +``` + +Expected: 200 OK + +**Browser tests:** +- Load https://localhost:5001 +- Check for JavaScript errors in console (F12) +- Verify sidebar loads +- Check plugin panels accessible + +### 9. Validate Critical Workflows + +Test end-to-end workflows: + +**Asset creation workflow:** +```bash +# Create test asset via API +curl -k -X POST https://localhost:5001/api/assets \ + -H "Content-Type: application/json" \ + -b /tmp/cookies.txt \ + -d '{"name":"test-asset-health-check","asset_type":"test"}' +``` + +Expected: 201 Created + +**Asset retrieval:** +```bash +curl -k https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt +``` + +Expected: 200 OK, asset data returned + +**Cleanup test asset:** +```bash +curl -k -X DELETE https://localhost:5001/api/assets/test-asset-health-check -b /tmp/cookies.txt +``` + +### 10. Check System Resources + +**Docker resource usage:** +```bash +docker stats --no-stream +``` + +**What to check:** +- CPU usage < 80% (unless under load) +- Memory usage < 80% of limit +- No memory leaks (usage trending up over time) + +**Disk space:** +```bash +docker system df +``` + +**Thresholds:** +- Images: < 10GB acceptable +- Containers: < 1GB acceptable +- Volumes: Depends on data size, monitor growth + +### 11. Summary Report + +**[X] System healthy:** +``` +Production health check: HEALTHY + +Services: +[X] Docker containers: 2/2 running (uptime: 3 days) +[X] Flask API: Responding in 45ms avg +[X] Database: Connected, pool healthy (12/20 connections) + +Plugins: +[X] All 12 plugins loaded successfully +[X] No import errors + +Endpoints: +[X] 1 active endpoint (VM) +[X] Last heartbeat: 2 minutes ago + +Performance: +[X] API response times: 50-200ms (good) +[X] Database queries: 20-80ms (good) +[X] CPU usage: 12% (low) +[X] Memory usage: 45% (normal) + +Logs: +[X] No critical errors in last 100 lines +WARNING: 2 warnings (connection pool exhausted) + +System is healthy and ready for production use. + +Minor action items: +- Investigate connection pool exhaustion (may need to increase pool size) +``` + +**WARNING: Issues detected:** +``` +Production health check: DEGRADED + +CRITICAL ISSUES: +1. Database connection failing + Error: could not connect to server + Impact: All API requests failing with 500 errors + Action: Check database service, verify credentials + +2. Plugin 'endpoints' failed to load + Error: ImportError: cannot import name 'Asset' + Impact: Endpoint functionality unavailable + Action: Check recent code changes, verify shopdb.api imports + +HIGH PRIORITY: +3. API response times degraded (800ms avg, was 100ms) + Impact: Slow user experience + Action: Check database indexes, query performance + +4. 15 endpoint agents not reporting (last seen > 1 hour) + Impact: Stale endpoint data + Action: Check agent service on VMs, network connectivity + +WARNINGS: +5. Container restarted 3 times in last hour + Action: Check logs for crash cause + +System is DEGRADED - address critical issues before deployment! + +Detailed logs: /tmp/health-check-{timestamp}.log +``` + +## Troubleshooting + +**API not responding:** +- Check Docker: `docker compose ps` +- Check logs: `docker compose logs api --tail 50` +- Restart if needed: `docker compose restart api` + +**Database connection issues:** +- Check database running: `docker compose ps db` +- Check credentials: .env file +- Test connection: `docker compose exec db mysql -u root -p` + +**Slow API performance:** +- Check database query times +- Look for missing indexes: `EXPLAIN SELECT ...` +- Check for table scans in logs + +**Plugins not loading:** +- Check import errors in logs +- Verify dependencies installed +- Check for circular imports + +## Related + +- `/docker-rebuild` - Rebuild if services unhealthy +- `/pre-commit` - Pre-deployment validation +- `/migration-check` - Database migration validation +- [[maintenance-footprint-baseline]] - Baseline metrics for comparison diff --git a/.claude/skills/security-scan.md b/.claude/skills/security-scan.md new file mode 100644 index 00000000..a72d2760 --- /dev/null +++ b/.claude/skills/security-scan.md @@ -0,0 +1,445 @@ +--- +name: security-scan +description: Run security checks from Sept 2026 audit - scan for hardcoded credentials, exposed keys, missing authentication, certificate issues +--- + +# Security Scan + +Monitor for security regressions based on September 2026 security audit findings (4 critical + 6 high). + +## What This Skill Does + +Automated security checks: +1. Scan for hardcoded credentials and API keys +2. Check for collector keys in committed files +3. Verify certificate validation enabled +4. Check authentication on sensitive endpoints +5. Scan for SQL injection vulnerabilities +6. Verify HTTPS enforcement +7. Check for exposed sensitive data in logs +8. Validate input sanitization + +## Background: Sept 2026 Audit + +**Critical findings addressed:** +1. Unauthenticated inventory updates (FIXED) +2. Deployment credential exposure in registry (MITIGATED) +3. No certificate validation in agent (PLANNED) +4. Hardcoded admin credentials (REMOVED) + +**This skill prevents regressions** - ensures fixes stay fixed and new code doesn't reintroduce vulnerabilities. + +## Workflow + +### 1. Scan for Hardcoded Credentials + +Search for credential patterns in code: + +```bash +# Passwords +grep -rn "password.*=.*['\"]" --include="*.py" --include="*.js" --include="*.vue" \ + shopdb/ plugins/ frontend/src/ \ + --exclude-dir=node_modules --exclude-dir=dist \ + | grep -v "password.*input\|password.*field\|password.*type\|password.*param" + +# API keys +grep -rn "api_key.*=.*['\"]" --include="*.py" shopdb/ plugins/ \ + | grep -v "api_key.*None\|api_key.*config\|api_key.*os.environ" + +# Tokens +grep -rn "token.*=.*['\"][a-zA-Z0-9]{20,}" --include="*.py" shopdb/ plugins/ + +# Database credentials +grep -rn "mysql://.*:.*@" --include="*.py" --include="*.yml" . +``` + +**Acceptable patterns:** +- `password = None` (default/placeholder) +- `password = os.environ.get(...)` (from environment) +- `password = config.get(...)` (from config) +- `password_field = "..."` (field name, not value) + +**Flag as violations:** +- `password = "admin123"` (hardcoded) +- `api_key = "sk_live_..."` (real key) +- `token = "eyJhbGciOi..."` (JWT token) +- `mysql://root:password@localhost` (database URL with creds) + +### 2. Check for Exposed Collector Keys + +Verify collector API keys not committed: + +```bash +# Check for shopdb_pat_ prefix (our collector keys) +grep -rn "shopdb_pat_" --include="*.py" --include="*.js" --include="*.md" . \ + --exclude-dir=.git + +# Check .env files not committed +git ls-files | grep -E "\.env$|\.env\." + +# Check for common secret patterns +grep -rn -E "(SECRET_KEY|COLLECTOR_KEY|API_KEY|PRIVATE_KEY).*=.*['\"][a-zA-Z0-9]{32,}" \ + --include="*.py" shopdb/ plugins/ +``` + +**Acceptable:** +- `.env.example` with placeholder values +- Comments referencing environment variables +- Memory files with redacted keys + +**Violations:** +- `.env` file committed +- Real collector key in code +- API key in configuration file + +### 3. Verify Certificate Validation + +Check that agent validates HTTPS certificates: + +```bash +# C# agent code +grep -n "ServerCertificateValidationCallback" agent/ShopDbEndpointAgent/ + +# Should NOT find: +# ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, sslPolicyErrors) => true; +# This disables all certificate validation! + +# Python collector code +grep -n "verify=False" shopdb/ plugins/ --include="*.py" +grep -n "ssl._create_unverified_context" shopdb/ plugins/ --include="*.py" +``` + +**Acceptable:** +- Development-only code with explicit comment +- Test code with mock certificates + +**Violations:** +- Production code with `verify=False` +- Agent accepts any certificate without validation +- No certificate pinning for critical endpoints + +### 4. Check Authentication on Sensitive Endpoints + +Verify authentication required for sensitive operations: + +```bash +# Find Flask routes +grep -rn "@.*route\|@bp.route" shopdb/ plugins/ --include="*.py" -A 3 + +# For each route, check if it has authentication: +# - @requires_auth decorator +# - @login_required decorator +# - Manual current_user check +``` + +**Sensitive operations that MUST have auth:** +- `/api/setup/*` (after initial setup) +- `/api/collectors/*` (collector authentication) +- `/api/admin/*` (admin only) +- `/api/*/create` (asset creation) +- `/api/*/update` (data modification) +- `/api/*/delete` (data deletion) + +**Flag missing authentication:** +```python +@bp.route('/api/endpoints/update', methods=['POST']) +def update_endpoint(): + # NO authentication check! + data = request.json + # ... update logic +``` + +**Correct patterns:** +```python +@bp.route('/api/endpoints/update', methods=['POST']) +@requires_auth # [X] Has authentication +def update_endpoint(): + ... + +@bp.route('/api/collectors/heartbeat', methods=['POST']) +@requires_device_token # [X] Collector authentication +def collector_heartbeat(): + ... +``` + +### 5. Scan for SQL Injection Vulnerabilities + +Check for raw SQL without parameterization: + +```bash +# Find string concatenation in SQL +grep -rn "execute.*+\|execute.*format\|execute.*%" \ + --include="*.py" shopdb/ plugins/ \ + | grep -v "# safe:\|# parameterized" + +# Find f-strings in SQL +grep -rn 'execute.*f".*{' --include="*.py" shopdb/ plugins/ + +# Find .format() in SQL +grep -rn "execute.*\.format\(" --include="*.py" shopdb/ plugins/ +``` + +**Vulnerable patterns:** +```python +# BAD: String concatenation +query = "SELECT * FROM assets WHERE name = '" + user_input + "'" +db.session.execute(query) + +# BAD: f-string +query = f"SELECT * FROM users WHERE username = '{username}'" +db.session.execute(query) + +# BAD: .format() +query = "DELETE FROM assets WHERE id = {}".format(asset_id) +``` + +**Safe patterns:** +```python +# GOOD: Parameterized query +query = "SELECT * FROM assets WHERE name = :name" +db.session.execute(query, {"name": user_input}) + +# GOOD: SQLAlchemy ORM +Asset.query.filter_by(name=user_input).first() + +# GOOD: SQLAlchemy Core +db.session.execute(select(Asset).where(Asset.name == user_input)) +``` + +### 6. Verify HTTPS Enforcement + +Check that HTTP is not allowed in production: + +```bash +# Flask configuration +grep -rn "PREFERRED_URL_SCHEME" shopdb/config.py + +# Should be: PREFERRED_URL_SCHEME = 'https' + +# Check for HTTP URLs in code +grep -rn "http://.*5001\|http://.*api" --include="*.py" --include="*.js" . \ + --exclude-dir=node_modules --exclude-dir=dist + +# Check Vite proxy config +grep -n "target.*http:" frontend/vite.config.js +``` + +**Acceptable:** +- Development Vite server (HTTP on 5173) +- HTTP in proxy target (SSL termination) + +**Violations:** +- Flask serving HTTP in production +- API calls without HTTPS +- Cookies without Secure flag + +### 7. Check for Sensitive Data in Logs + +Scan for patterns that expose sensitive data: + +```bash +# Check for password logging +grep -rn "logging.*password\|print.*password\|log.info.*password" \ + --include="*.py" shopdb/ plugins/ \ + | grep -v "password.*\*\*\*\|password.*redacted" + +# Check for API key logging +grep -rn "logging.*api_key\|print.*api_key" --include="*.py" shopdb/ plugins/ + +# Check for full request logging (may contain sensitive headers) +grep -rn "log.*request.headers\|print.*request.headers" \ + --include="*.py" shopdb/ plugins/ +``` + +**Violations:** +```python +# BAD: Password in logs +logging.info(f"Login attempt: {username} / {password}") + +# BAD: API key exposed +print(f"Using API key: {api_key}") + +# BAD: Full headers (may contain auth tokens) +logger.debug(f"Request headers: {request.headers}") +``` + +**Safe patterns:** +```python +# GOOD: Redacted password +logging.info(f"Login attempt: {username} / ***") + +# GOOD: Partial key +logger.debug(f"API key: {api_key[:8]}...") + +# GOOD: Specific header +logger.debug(f"Content-Type: {request.headers.get('Content-Type')}") +``` + +### 8. Validate Input Sanitization + +Check that user input is validated: + +```bash +# Find request.json usage without validation +grep -rn "request.json\|request.form\|request.args" \ + --include="*.py" shopdb/ plugins/ -A 5 \ + | grep -v "validator\|schema\|validate\|is_valid" +``` + +**Look for:** +- Direct use of `request.json['field']` without validation +- No length limits on string inputs +- No type checking on numeric inputs +- No sanitization of file paths + +**Vulnerable:** +```python +# BAD: No validation +@bp.route('/api/assets', methods=['POST']) +def create_asset(): + data = request.json # No validation! + asset = Asset(**data) # Directly creates from user input + db.session.add(asset) +``` + +**Safe:** +```python +# GOOD: Schema validation +from marshmallow import Schema, fields, validate + +class AssetSchema(Schema): + name = fields.Str(required=True, validate=validate.Length(max=255)) + asset_type = fields.Str(required=True, validate=validate.OneOf(['computer', 'printer'])) + +@bp.route('/api/assets', methods=['POST']) +def create_asset(): + schema = AssetSchema() + data = schema.load(request.json) # Validated! + asset = Asset(**data) +``` + +### 9. Summary Report + +**[X] No vulnerabilities found:** +``` +Security scan complete: PASS + +Credential Scan: +[X] No hardcoded passwords found +[X] No API keys in code +[X] No tokens exposed +[X] Collector keys not committed + +Certificate Validation: +[X] Agent validates certificates +[X] No verify=False in production code + +Authentication: +[X] All sensitive endpoints protected +[X] Collector endpoints use device tokens +[X] Admin endpoints require authentication + +SQL Injection: +[X] All queries parameterized +[X] No string concatenation in SQL +[X] ORM used consistently + +HTTPS Enforcement: +[X] Production uses HTTPS +[X] Cookies have Secure flag +[X] No HTTP URLs in production config + +Logging: +[X] No passwords in logs +[X] API keys redacted +[X] Sensitive data masked + +Input Validation: +[X] Schema validation on all POST/PUT endpoints +[X] Length limits enforced +[X] Type checking present + +No security issues detected! +``` + +**WARNING: Vulnerabilities found:** +``` +Security scan: FAILED + +CRITICAL (block deployment): +1. Hardcoded admin password in tests/fixtures/users.py + Line 15: password = "admin123" + Impact: Test credential could be used in production + Action: Use environment variable or generate random password + +2. Collector API key committed in .env file + File: .env (should be .gitignore'd) + Impact: API key exposed in version control + Action: Remove .env, regenerate key, add to .gitignore + +3. Certificate validation disabled in agent + File: agent/ShopDbEndpointAgent/ApiClient.cs:42 + Code: ServerCertificateValidationCallback = (_, _, _, _) => true + Impact: Agent accepts any certificate (MITM attack possible) + Action: Implement proper certificate validation + +HIGH PRIORITY: +4. Missing authentication on /api/endpoints/update + File: plugins/endpoints/api/routes.py:87 + Impact: Anyone can update endpoint data + Action: Add @requires_device_token decorator + +5. SQL injection vulnerability in asset search + File: shopdb/core/api/assets.py:45 + Code: f"SELECT * FROM assets WHERE name LIKE '%{search}%'" + Impact: SQL injection attack possible + Action: Use parameterized query or ORM + +MEDIUM: +6. Password logged in authentication.py:23 + Code: logger.info(f"Login: {username}/{password}") + Impact: Passwords exposed in log files + Action: Redact password: logger.info(f"Login: {username}/***") + +DO NOT DEPLOY until critical and high priority issues are fixed! + +Remediation guide: [[security-fix-implementation]] +``` + +## Automated Remediation + +Some issues can be auto-fixed: + +**Remove hardcoded credentials:** +```bash +# Replace hardcoded passwords with environment variables +sed -i 's/password = ".*"/password = os.environ.get("TEST_PASSWORD", "changeme")/g' tests/fixtures/*.py +``` + +**Add .env to .gitignore:** +```bash +echo ".env" >> .gitignore +git rm --cached .env +``` + +**Add authentication decorator:** +```python +# Before: +@bp.route('/api/endpoints/update', methods=['POST']) +def update_endpoint(): + ... + +# After: +@bp.route('/api/endpoints/update', methods=['POST']) +@requires_device_token +def update_endpoint(): + ... +``` + +## Related + +- [[security-audit-2026-09-11]] - Original audit findings +- [[security-fix-implementation]] - Detailed remediation plans +- `/pre-commit` - Include basic security checks in pre-commit +- `/production-health` - Runtime security monitoring +- ADR-016: Credential delivery (device tokens) diff --git a/.claude/skills/vm-snapshot.md b/.claude/skills/vm-snapshot.md new file mode 100644 index 00000000..e15bd4a7 --- /dev/null +++ b/.claude/skills/vm-snapshot.md @@ -0,0 +1,395 @@ +--- +name: vm-snapshot +description: Create, restore, list, and manage Hyper-V VM snapshots - 78 hrs/year time savings +--- + +# VM Snapshot Management + +Comprehensive Hyper-V snapshot management for test VMs. Saves ~78 hrs/year by automating manual snapshot workflow. + +## What This Skill Does + +Complete snapshot lifecycle: +1. Create snapshot with descriptive name +2. List all snapshots with dates and descriptions +3. Restore to specific snapshot +4. Delete old snapshots (cleanup) +5. Verify VM state after restore +6. Automated pre/post-snapshot health checks + +## Prerequisites + +**Hyper-V permissions required:** +- Hyper-V Administrator group membership +- Or: Run PowerShell as Administrator + +**VM details:** +- Name: ShopDB-Test-VM (or user-specified) +- Location: Hyper-V on local workstation +- Must be accessible via Get-VM cmdlet + +## Workflow + +### Common Operation: Create Snapshot + +Most frequent use case - create snapshot before risky changes. + +**Usage:** `/vm-snapshot create "description of what's about to change"` + +**Steps:** + +1. **Verify VM state:** +```powershell +Get-VM -Name "ShopDB-Test-VM" | Select-Object Name, State, Uptime +``` + +Expected: Running (preferred) or Stopped (acceptable) + +2. **Generate snapshot name:** +``` +Format: YYYY-MM-DD_HHmm_description +Example: 2026-09-11_1430_before-agent-deploy +``` + +Short descriptions preferred (< 50 chars). + +3. **Create snapshot:** +```powershell +Checkpoint-VM -Name "ShopDB-Test-VM" -SnapshotName "2026-09-11_1430_before-agent-deploy" +``` + +**Monitor for:** +- Snapshot creation progress +- Disk space sufficient (snapshots can be large) +- No errors during checkpoint + +4. **Verify snapshot created:** +```powershell +Get-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-09-11_1430_before-agent-deploy" +``` + +Expected: Snapshot exists with correct name and timestamp. + +5. **Report:** +``` +Snapshot created successfully: + Name: 2026-09-11_1430_before-agent-deploy + VM: ShopDB-Test-VM + Created: 2026-09-11 14:30:15 + Size: 2.3 GB + Parent: 2026-09-10_0900_clean-state + +You can now safely proceed with risky changes. +To restore: /vm-snapshot restore "2026-09-11_1430_before-agent-deploy" +``` + +### List All Snapshots + +**Usage:** `/vm-snapshot list` + +**Show snapshot tree:** +```powershell +Get-VMSnapshot -VMName "ShopDB-Test-VM" | + Select-Object Name, CreationTime, ParentSnapshotName, SizeMB | + Sort-Object CreationTime -Descending +``` + +**Format output:** +``` +Snapshots for ShopDB-Test-VM: + +Recent (last 7 days): + 2026-09-11_1430_before-agent-deploy (2.3 GB) - 2 hours ago + 2026-09-10_0900_clean-state (1.8 GB) - 1 day ago + 2026-09-09_1600_test-endpoints (2.1 GB) - 2 days ago + +Older (7-30 days): + 2026-09-01_1200_baseline (1.5 GB) - 10 days ago + 2026-08-28_0800_production-mirror (3.2 GB) - 14 days ago + +Old (>30 days, consider cleanup): + 2026-08-15_1000_initial-setup (1.2 GB) - 27 days ago + 2026-08-01_1400_before-wilm-merge (2.8 GB) - 41 days ago [CLEANUP CANDIDATE] + +Total snapshots: 7 +Total disk space: 14.9 GB + +Cleanup recommendations: +- Keep "baseline" and "production-mirror" (important) +- Remove "2026-08-01_1400_before-wilm-merge" (41 days old, > 30 day threshold) +``` + +### Restore to Snapshot + +**Usage:** `/vm-snapshot restore "snapshot-name"` + +**WARNING: WARNING: This will discard ALL changes since snapshot was created!** + +**Steps:** + +1. **Confirm restoration:** +Ask user to confirm: +``` +WARNING: RESTORE WARNING: + +You are about to restore VM to snapshot: + 2026-09-10_0900_clean-state (created 1 day ago) + +This will DISCARD all changes made since: +- Any agent deployments after Sept 10, 9:00 AM +- Any data added to ShopDB since Sept 10 +- Any configuration changes on VM + +Are you sure you want to proceed? + Type 'yes' to restore, anything else to cancel: +``` + +2. **Check VM state:** +```powershell +$vm = Get-VM -Name "ShopDB-Test-VM" +if ($vm.State -eq "Running") { + Write-Host "VM is running, will be stopped for restore" +} +``` + +3. **Perform restore:** +```powershell +# Stop VM if running +if ((Get-VM -Name "ShopDB-Test-VM").State -eq "Running") { + Stop-VM -Name "ShopDB-Test-VM" -Force +} + +# Restore snapshot +Restore-VMSnapshot -Name "2026-09-10_0900_clean-state" -VMName "ShopDB-Test-VM" -Confirm:$false + +# Start VM +Start-VM -Name "ShopDB-Test-VM" +``` + +4. **Wait for VM to boot:** +```powershell +Write-Host "Waiting for VM to boot..." +Start-Sleep -Seconds 30 + +# Wait for network +while (-not (Test-Connection -ComputerName 192.168.100.10 -Count 1 -Quiet)) { + Write-Host "Waiting for VM network..." + Start-Sleep -Seconds 5 +} +``` + +5. **Verify VM state:** +```bash +# SSH to VM +ssh -i ~/.ssh/vm_key SupportUser@192.168.100.10 "Get-Service ShopDBEndpointAgent" +``` + +Expected: Service exists (may be stopped, depends on snapshot state) + +6. **Report:** +``` +VM restored successfully: + Snapshot: 2026-09-10_0900_clean-state + VM State: Running + Network: Accessible at 192.168.100.10 + SSH: Connected + +Post-restore checklist: + 1. Verify agent service state + 2. Check endpoint reporting in ShopDB UI + 3. Test any workflows that depend on VM state + +VM is ready for use. +``` + +### Delete Snapshot(s) + +**Usage:** `/vm-snapshot delete "snapshot-name"` + +or + +**Usage:** `/vm-snapshot cleanup` (interactive cleanup of old snapshots) + +**Single snapshot deletion:** + +1. **Verify snapshot exists:** +```powershell +Get-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-08-01_1400_before-wilm-merge" +``` + +2. **Check if it's the current state:** +```powershell +$current = (Get-VM -Name "ShopDB-Test-VM").ParentCheckpointId +$snapshot = (Get-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-08-01_1400_before-wilm-merge").Id + +if ($current -eq $snapshot) { + Write-Error "Cannot delete snapshot: VM is currently at this state" +} +``` + +3. **Delete snapshot:** +```powershell +Remove-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-08-01_1400_before-wilm-merge" +``` + +**Note:** Hyper-V will merge snapshot disk into parent (can take time for large snapshots) + +4. **Monitor merge progress:** +```powershell +while ((Get-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-08-01_1400_before-wilm-merge" -ErrorAction SilentlyContinue)) { + Write-Host "Merging snapshot..." + Start-Sleep -Seconds 5 +} +``` + +5. **Report:** +``` +Snapshot deleted: + Name: 2026-08-01_1400_before-wilm-merge + Disk space freed: ~2.8 GB + Merge time: 45 seconds +``` + +**Automated cleanup:** + +For `/vm-snapshot cleanup`: + +1. List snapshots older than 30 days +2. Exclude important snapshots: + - Contains "baseline" in name + - Contains "production" in name + - Marked as "Production" in description +3. Show cleanup candidates +4. Ask user to confirm each deletion +5. Delete confirmed snapshots +6. Report total space freed + +### Export Snapshot (Advanced) + +**Usage:** `/vm-snapshot export "snapshot-name" "C:\Exports\path"` + +Exports snapshot to separate location for long-term archival: + +```powershell +$snapshot = Get-VMSnapshot -VMName "ShopDB-Test-VM" -Name "2026-09-10_0900_clean-state" +Export-VMSnapshot -VMSnapshot $snapshot -Path "C:\Exports\ShopDB-Snapshots\" +``` + +**Use cases:** +- Archive known-good states +- Transfer snapshot to another machine +- Backup before major infrastructure changes + +## Common Workflows + +### Before Risky Change + +```bash +# 1. Create snapshot +/vm-snapshot create "before-database-migration" + +# 2. Make risky change +# ... deploy code, run migration, etc. + +# 3. If failed, restore +/vm-snapshot restore "2026-09-11_1430_before-database-migration" + +# 4. If succeeded, keep snapshot for a while +# (cleanup later with /vm-snapshot cleanup) +``` + +### Weekly Cleanup + +```bash +# 1. List snapshots +/vm-snapshot list + +# 2. Review old snapshots (> 30 days) +# 3. Run automated cleanup +/vm-snapshot cleanup + +# 4. Verify important snapshots remain +/vm-snapshot list +``` + +### Baseline Management + +```bash +# Create monthly baseline +/vm-snapshot create "baseline-$(date +%Y-%m)" + +# Mark as important (manual: add "KEEP" to description) +# This prevents automated cleanup from removing it +``` + +## Troubleshooting + +**"Insufficient disk space":** +- Check Hyper-V storage location free space +- Run `/vm-snapshot cleanup` to remove old snapshots +- Consider moving VM to larger disk + +**"Access denied":** +- Run PowerShell as Administrator +- Verify Hyper-V Administrator group membership +- Check Hyper-V service is running + +**"VM is in wrong state":** +- Some operations require VM stopped +- Stop VM: `Stop-VM -Name "ShopDB-Test-VM" -Force` +- Retry operation + +**Snapshot merge takes too long:** +- Normal for large snapshots (can take 10+ minutes) +- Do not interrupt merge process +- Avoid creating too many snapshots (disk I/O intensive) + +**Restored VM not accessible:** +- Wait 60+ seconds for VM to fully boot +- Check VM console in Hyper-V Manager +- Verify network adapter enabled in VM settings + +## Best Practices + +**Naming convention:** +- Use `YYYY-MM-DD_HHmm_description` format +- Keep description concise (< 50 chars) +- Include "before-" prefix for pre-change snapshots + +**Snapshot frequency:** +- Before risky changes: Always +- Daily development: Optional (creates disk bloat) +- Weekly baseline: Recommended + +**Cleanup schedule:** +- Review snapshots weekly +- Remove snapshots > 30 days (unless important) +- Keep at least one "baseline" snapshot per month + +**Important snapshots:** +- Tag with "KEEP", "baseline", or "production" +- Document purpose in description +- Export to separate location for long-term storage + +## Time Savings + +**Manual snapshot workflow:** ~15 minutes +- Open Hyper-V Manager +- Navigate to VM +- Create checkpoint +- Name snapshot +- Wait for confirmation +- Verify in list + +**With this skill:** ~30 seconds +- Type `/vm-snapshot create "description"` +- Skill handles rest automatically + +**Annual savings:** 78 hours (based on 312 snapshots/year) + +## Related + +- [[vm-ssh-access]] - SSH access to VM +- [[agent-deployment]] - Agent deployment workflow +- `.claude/setup/VM-SNAPSHOT-GUIDE.md` - Detailed Hyper-V snapshot guide +- `.claude/setup/VM-USERS.md` - VM user accounts