shopdb-flask/.claude/skills/full-deploy.md

name, description

name	description
full-deploy	Complete deployment workflow for air-gapped production - validation, bundle creation, deployment, verification (saves 50-100 hrs/year from 312 hrs burden)

Full Production Deployment Workflow

Complete end-to-end deployment for air-gapped ShopDB production environment. Addresses 312 hrs/year coordination burden.

What This Skill Does

Orchestrates complete deployment:

1. Pre-deploy validation (tests, health checks, security)
2. Create deployment bundle
3. Deploy to dev PC first (staging test)
4. Generate production deployment runbook
5. Transfer bundle across air gap
6. Deploy to production
7. Post-deploy validation
8. Rollback plan if issues detected

Air-Gap Environment

+--------------+         +---------------+         +------------+
|   Dev PC     | Bundle  |   Work PC     | Bundle  |   GitHub   |
|   (gitea)    | ------> |  (GitHub)     | ------> |  (public)  |
|              |         | YOU ARE HERE  |         |            |
+--------------+         +---------------+         +------------+
       |                         |
  Dev Server               Prod Server
 (ops branch)              (wj branch)

Constraints:

• No direct GitHub access from dev PC
• Manual bundle transfer between environments
• Production changes must be tested on dev first
• Work PC is coordination point

Prerequisites

Before starting deployment:

• All changes committed to wj branch
• Tests passing (pytest -v)
• Style check passing (scripts/check-naming-and-style.sh)
• Frontend built (npm run build)
• No uncommitted changes (git status clean)

Workflow

Phase 1: Pre-Deploy Validation (Work PC)

1.1 Run Complete Validation Suite:

# Health check current state
/production-health

# Security scan
/security-scan

# Migration validation
/migration-check

# Pre-commit checks
/pre-commit

Expected: All checks pass with no critical issues.

If validation fails:

• Document failures
• Fix issues before proceeding
• Re-run validation
• Do NOT proceed with deployment if critical issues remain

1.2 Create Deployment Tag:

# Tag format: vYYYY.MM.DD-N (N = deployment number that day)
TAG="v$(date +%Y.%m.%d)-1"
git tag -a $TAG -m "Production deployment $(date +%Y-%m-%d)"
git push origin $TAG

1.3 Document Changes Since Last Deploy:

# Get last deployment tag
LAST_TAG=$(git describe --tags --abbrev=0 HEAD^)

# Generate changelog
git log $LAST_TAG..HEAD --oneline --no-merges > deployment-changelog.txt

# Show commit summary
echo "Changes since last deploy ($LAST_TAG):"
git log $LAST_TAG..HEAD --oneline --no-merges | wc -l
echo "commits"

Review changelog for:

• Breaking changes
• Database migrations
• New dependencies
• Configuration changes

Phase 2: Create Deployment Bundle (Work PC)

2.1 Build Frontend:

cd frontend
npm run build
cd ..

Verify frontend/dist/ updated with production build.

2.2 Create Bundle Archive:

BUNDLE_DATE=$(date +%Y%m%d_%H%M)
BUNDLE_NAME="shopdb-deploy-$BUNDLE_DATE.tar.gz"

# Include everything except .git, node_modules, venv
tar -czf ~/_transfer/$BUNDLE_NAME \
  --exclude='.git' \
  --exclude='node_modules' \
  --exclude='venv' \
  --exclude='*.pyc' \
  --exclude='__pycache__' \
  --exclude='frontend/dist' \
  .

# Separately bundle frontend (large)
tar -czf ~/_transfer/shopdb-frontend-$BUNDLE_DATE.tar.gz \
  frontend/dist/

echo "Bundles created:"
ls -lh ~/_transfer/shopdb-*$BUNDLE_DATE.tar.gz

2.3 Generate Deployment Runbook:

Create deployment-runbook-$BUNDLE_DATE.md:

# ShopDB Deployment Runbook

**Date:** $(date +%Y-%m-%d %H:%M)
**Bundle:** shopdb-deploy-$BUNDLE_DATE.tar.gz
**Tag:** $TAG
**Target:** Production (wj branch)

## Pre-Deployment

- [ ] Backup production database
- [ ] Create VM snapshot: /vm-snapshot create "before-deploy-$BUNDLE_DATE"
- [ ] Verify production health: /production-health
- [ ] Notify users of maintenance window

## Deployment Steps

### 1. Extract Bundle

```bash
cd /path/to/shopdb-production
tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz
tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz

2. Install Dependencies (if changed)

Check requirements.txt and package.json for changes:

# Python dependencies
pip install -r requirements.txt

# Node dependencies (if needed)
cd frontend && npm ci && cd ..

3. Run Database Migrations

docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all

Watch for:

• Migration errors
• Foreign key violations
• Timeout on large migrations

4. Restart Services

docker compose restart api

Verify restart:

docker compose ps
docker compose logs api --tail 50

5. Verify Deployment

• All containers running
• API responding: curl -k https://localhost:5001/api/health
• Frontend loads: https://localhost:5001
• Login works
• Recent endpoints reporting

6. Monitor for Issues

# Watch logs for 5 minutes
docker compose logs -f api

Post-Deployment

• Verify endpoints reporting (check last_seen timestamps)
• Test critical workflows (asset creation, search)
• Check plugin loading: docker compose exec api flask plugin list
• Notify users deployment complete

Rollback Plan

If issues detected:

1. Restore VM snapshot:

   /vm-snapshot restore "before-deploy-$BUNDLE_DATE"
   

2. Or rollback git:

   git checkout $LAST_TAG
   docker compose down
   docker compose up -d
   

3. Restore database backup (if schema changed)

Changes in This Deployment

$(git log $LAST_TAG..HEAD --oneline --no-merges)

Database Migrations

$(docker compose exec api flask db history | grep -A 5 "current")

Testing Results

• pytest: PASS (42 tests, 0 failures)
• Style check: PASS
• Security scan: PASS (0 critical, 0 high)
• Migration check: PASS (linear chains, no conflicts)

**2.4 Bundle Verification:**

```bash
# Verify bundle integrity
tar -tzf ~/_transfer/$BUNDLE_NAME | head -20

# Check bundle size
du -h ~/_transfer/$BUNDLE_NAME

Expected: < 50MB (excluding frontend) Expected frontend: < 20MB

Phase 3: Stage Deploy on Dev PC (Optional but Recommended)

3.1 Transfer Bundle to Dev PC:

(Manual: Copy bundle via USB, shared drive, or approved transfer method)

3.2 Deploy to Dev Server (ops branch):

# On dev PC
cd /path/to/shopdb-dev
tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz
tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz

# Run deployment steps
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
docker compose restart api

# Verify
/production-health

3.3 Smoke Test on Dev:

• Test all plugin pages load
• Create test asset
• Run full test suite
• Check for errors in logs

If dev deploy fails:

• Fix issues
• Create new bundle
• Re-test on dev
• Do NOT proceed to production

Phase 4: Production Deployment

4.1 Create Production Backup:

# Database backup
docker compose exec db mysqldump -u root -p shopdb > ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql

# VM snapshot
/vm-snapshot create "before-deploy-$BUNDLE_DATE"

4.2 Notify Users:

Maintenance Window: [start time] - [end time]
ShopDB will be unavailable during deployment.
Expected downtime: 15-30 minutes

4.3 Execute Deployment Runbook:

Follow deployment-runbook-$BUNDLE_DATE.md steps exactly.

4.4 Monitor Deployment:

Watch logs during restart:

docker compose logs -f api

Look for:

• "Starting gunicorn"
• "Listening at https://0.0.0.0:5001"
• "Loaded plugins: 12"
• Exceptions or errors

4.5 Initial Verification:

# Quick checks (< 2 min)
docker compose ps                          # All running?
curl -k https://localhost:5001/api/health  # API responds?
curl -k https://localhost:5001/           # Frontend loads?

Phase 5: Post-Deploy Validation

5.1 Run Health Checks:

/production-health

Expected: All checks pass, no critical errors.

5.2 Verify Critical Workflows:

Test each critical workflow:

1. Login:

   • Test admin login
   • Test standard user login
   • Verify sessions work

2. Asset Management:

   • Create test asset
   • Search for asset
   • Update asset
   • Delete test asset

3. Endpoint Reporting:

   • Check endpoint list loads
   • Verify recent heartbeats (< 10 min)
   • Check endpoint details page

4. Plugin Loading:

   docker compose exec api flask plugin list
   

   Expected: All plugins loaded, no import errors

5.3 Monitor for 1 Hour:

Watch logs for anomalies:

# Check every 10 minutes for first hour
docker compose logs api --since 10m | grep -i "error\|exception\|critical"

5.4 Verify Endpoint Agents Still Reporting:

# Check last_seen timestamps
docker compose exec api flask shell -c "
from plugins.endpoints.models import Endpoint
from datetime import datetime, timedelta
recent = Endpoint.query.filter(
    Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)
).count()
print(f'Endpoints reporting in last 10 min: {recent}')
"

Expected: At least 1 (VM should report)

Phase 6: Rollback (If Issues Detected)

Rollback Decision Criteria:

MUST rollback if:

• Database migration failed
• Critical plugin won't load
• API completely non-functional
• Data integrity issues

SHOULD rollback if:

• Multiple endpoints not reporting
• Critical workflow broken
• Performance severely degraded
• Security vulnerability introduced

Can monitor and fix if:

• Single plugin has minor issue
• Non-critical feature broken
• Performance slightly slower
• Cosmetic UI issues

Rollback Procedure:

Option A: VM Snapshot Restore (Fastest)

/vm-snapshot restore "before-deploy-$BUNDLE_DATE"

Time: ~5 minutes Scope: Complete system rollback

Option B: Git Rollback

git checkout $LAST_TAG
docker compose down
docker compose -f docker-compose.prebuilt.yml build api
docker compose up -d

Time: ~10 minutes Scope: Code only (database may need restore)

Option C: Database Restore (If Schema Changed)

# Restore database backup
docker compose exec db mysql -u root -p shopdb < ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql

# Or point-in-time recovery if binlog enabled

Time: ~15-30 minutes depending on database size

After Rollback:

1. Verify system functional
2. Document what went wrong
3. Fix issues before retry
4. Notify users system restored

Phase 7: Deployment Complete

7.1 Update Documentation:

• Document any issues encountered
• Update deployment notes
• Record deployment time and downtime
• Note any manual steps required

7.2 Notify Stakeholders:

Deployment complete: ShopDB v$TAG
- Deployment time: [duration]
- Downtime: [actual downtime]
- Status: Success / Issues noted
- Changes: [link to changelog]

7.3 Monitor for 24 Hours:

Set reminder to check:

• Logs for errors
• Endpoint reporting
• User feedback
• Performance metrics

7.4 Cleanup:

# Remove old bundles (> 30 days)
find ~/_transfer/ -name "shopdb-deploy-*.tar.gz" -mtime +30 -delete

# Remove old backups (> 30 days)
find ~/backups/ -name "shopdb-pre-deploy-*.sql" -mtime +30 -delete

# Clean up old VM snapshots
/vm-snapshot cleanup

Time Estimates

Phase	Time	Can Fail?
Pre-deploy validation	10-15 min	Yes (fix issues)
Bundle creation	5 min	No
Stage deploy (dev)	15-20 min	Yes (fix before prod)
Production backup	5 min	No
Production deploy	15-20 min	Yes (rollback)
Post-deploy validation	15-30 min	Yes (rollback)
Total	65-100 min

Downtime: 5-15 minutes (during production deploy phase)

Automation Opportunities

Current: 312 hrs/year coordination burden

With this skill: ~50-100 hrs saved/year

Further automation:

• Automated bundle transfer (if air-gap can be bridged)
• Automated staging tests
• Automated rollback triggers
• Continuous deployment (requires air-gap solution)

Troubleshooting

Bundle extraction fails:

• Check disk space: df -h
• Verify bundle integrity: tar -tzf bundle.tar.gz
• Try smaller extraction: tar -xzf bundle.tar.gz --one-top-level

Migration fails:

• Check database connectivity
• Verify migration order (ADR-008)
• Check for data that violates new constraints
• See /migration-check for detailed validation

Services won't start:

• Check logs: docker compose logs api
• Verify port availability: netstat -tulpn | grep 5001
• Check Docker status: docker ps -a
• Try rebuild: /docker-rebuild

Rollback fails:

• VM snapshot may be corrupted
• Try alternative rollback method
• Contact system admin if all methods fail

Related

• /pre-commit - Pre-deployment validation
• /production-health - Health monitoring
• /migration-check - Database migration validation
• /docker-rebuild - Container rebuild
• /vm-snapshot - VM snapshot management
• workflow-patterns - Common workflows
• maintenance-footprint-baseline - 312 hrs/year coordination burden
• ADR-004: Deployment topology