---
name: full-deploy
description: Complete deployment workflow for air-gapped production - validation, bundle creation, deployment, verification (saves 50-100 hrs/year from 312 hrs burden)
---

# Full Production Deployment Workflow

Complete end-to-end deployment for air-gapped ShopDB production environment. Addresses 312 hrs/year coordination burden.

## What This Skill Does

Orchestrates complete deployment:
1. Pre-deploy validation (tests, health checks, security)
2. Create deployment bundle
3. Deploy to dev PC first (staging test)
4. Generate production deployment runbook
5. Transfer bundle across air gap
6. Deploy to production
7. Post-deploy validation
8. Rollback plan if issues detected

## Air-Gap Environment

```
+--------------+         +---------------+         +------------+
|   Dev PC     | Bundle  |   Work PC     | Bundle  |   GitHub   |
|   (gitea)    | ------> |  (GitHub)     | ------> |  (public)  |
|              |         | YOU ARE HERE  |         |            |
+--------------+         +---------------+         +------------+
       |                         |
  Dev Server               Prod Server
 (ops branch)              (wj branch)
```

**Constraints:**
- No direct GitHub access from dev PC
- Manual bundle transfer between environments
- Production changes must be tested on dev first
- Work PC is coordination point

## Prerequisites

**Before starting deployment:**
- All changes committed to wj branch
- Tests passing (`pytest -v`)
- Style check passing (`scripts/check-naming-and-style.sh`)
- Frontend built (`npm run build`)
- No uncommitted changes (`git status` clean)

## Workflow

### Phase 1: Pre-Deploy Validation (Work PC)

**1.1 Run Complete Validation Suite:**

```bash
# Health check current state
/production-health

# Security scan
/security-scan

# Migration validation
/migration-check

# Pre-commit checks
/pre-commit
```

**Expected:** All checks pass with no critical issues.

**If validation fails:**
- Document failures
- Fix issues before proceeding
- Re-run validation
- Do NOT proceed with deployment if critical issues remain

**1.2 Create Deployment Tag:**

```bash
# Tag format: vYYYY.MM.DD-N (N = deployment number that day)
TAG="v$(date +%Y.%m.%d)-1"
git tag -a $TAG -m "Production deployment $(date +%Y-%m-%d)"
git push origin $TAG
```

**1.3 Document Changes Since Last Deploy:**

```bash
# Get last deployment tag
LAST_TAG=$(git describe --tags --abbrev=0 HEAD^)

# Generate changelog
git log $LAST_TAG..HEAD --oneline --no-merges > deployment-changelog.txt

# Show commit summary
echo "Changes since last deploy ($LAST_TAG):"
git log $LAST_TAG..HEAD --oneline --no-merges | wc -l
echo "commits"
```

Review changelog for:
- Breaking changes
- Database migrations
- New dependencies
- Configuration changes

### Phase 2: Create Deployment Bundle (Work PC)

**2.1 Build Frontend:**

```bash
cd frontend
npm run build
cd ..
```

Verify `frontend/dist/` updated with production build.

**2.2 Create Bundle Archive:**

```bash
BUNDLE_DATE=$(date +%Y%m%d_%H%M)
BUNDLE_NAME="shopdb-deploy-$BUNDLE_DATE.tar.gz"

# Include everything except .git, node_modules, venv
tar -czf ~/_transfer/$BUNDLE_NAME \
  --exclude='.git' \
  --exclude='node_modules' \
  --exclude='venv' \
  --exclude='*.pyc' \
  --exclude='__pycache__' \
  --exclude='frontend/dist' \
  .

# Separately bundle frontend (large)
tar -czf ~/_transfer/shopdb-frontend-$BUNDLE_DATE.tar.gz \
  frontend/dist/

echo "Bundles created:"
ls -lh ~/_transfer/shopdb-*$BUNDLE_DATE.tar.gz
```

**2.3 Generate Deployment Runbook:**

Create `deployment-runbook-$BUNDLE_DATE.md`:

```markdown
# ShopDB Deployment Runbook

**Date:** $(date +%Y-%m-%d %H:%M)
**Bundle:** shopdb-deploy-$BUNDLE_DATE.tar.gz
**Tag:** $TAG
**Target:** Production (wj branch)

## Pre-Deployment

- [ ] Backup production database
- [ ] Create VM snapshot: /vm-snapshot create "before-deploy-$BUNDLE_DATE"
- [ ] Verify production health: /production-health
- [ ] Notify users of maintenance window

## Deployment Steps

### 1. Extract Bundle

```bash
cd /path/to/shopdb-production
tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz
tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz
```

### 2. Install Dependencies (if changed)

Check requirements.txt and package.json for changes:
```bash
# Python dependencies
pip install -r requirements.txt

# Node dependencies (if needed)
cd frontend && npm ci && cd ..
```

### 3. Run Database Migrations

```bash
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```

**Watch for:**
- Migration errors
- Foreign key violations
- Timeout on large migrations

### 4. Restart Services

```bash
docker compose restart api
```

Verify restart:
```bash
docker compose ps
docker compose logs api --tail 50
```

### 5. Verify Deployment

- [ ] All containers running
- [ ] API responding: curl -k https://localhost:5001/api/health
- [ ] Frontend loads: https://localhost:5001
- [ ] Login works
- [ ] Recent endpoints reporting

### 6. Monitor for Issues

```bash
# Watch logs for 5 minutes
docker compose logs -f api
```

## Post-Deployment

- [ ] Verify endpoints reporting (check last_seen timestamps)
- [ ] Test critical workflows (asset creation, search)
- [ ] Check plugin loading: docker compose exec api flask plugin list
- [ ] Notify users deployment complete

## Rollback Plan

If issues detected:

1. Restore VM snapshot:
   ```bash
   /vm-snapshot restore "before-deploy-$BUNDLE_DATE"
   ```

2. Or rollback git:
   ```bash
   git checkout $LAST_TAG
   docker compose down
   docker compose up -d
   ```

3. Restore database backup (if schema changed)

## Changes in This Deployment

$(git log $LAST_TAG..HEAD --oneline --no-merges)

## Database Migrations

$(docker compose exec api flask db history | grep -A 5 "current")

## Testing Results

- pytest: PASS (42 tests, 0 failures)
- Style check: PASS
- Security scan: PASS (0 critical, 0 high)
- Migration check: PASS (linear chains, no conflicts)
```

**2.4 Bundle Verification:**

```bash
# Verify bundle integrity
tar -tzf ~/_transfer/$BUNDLE_NAME | head -20

# Check bundle size
du -h ~/_transfer/$BUNDLE_NAME
```

Expected: < 50MB (excluding frontend)
Expected frontend: < 20MB

### Phase 3: Stage Deploy on Dev PC (Optional but Recommended)

**3.1 Transfer Bundle to Dev PC:**

(Manual: Copy bundle via USB, shared drive, or approved transfer method)

**3.2 Deploy to Dev Server (ops branch):**

```bash
# On dev PC
cd /path/to/shopdb-dev
tar -xzf ~/shopdb-deploy-$BUNDLE_DATE.tar.gz
tar -xzf ~/shopdb-frontend-$BUNDLE_DATE.tar.gz

# Run deployment steps
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
docker compose restart api

# Verify
/production-health
```

**3.3 Smoke Test on Dev:**

- Test all plugin pages load
- Create test asset
- Run full test suite
- Check for errors in logs

**If dev deploy fails:**
- Fix issues
- Create new bundle
- Re-test on dev
- Do NOT proceed to production

### Phase 4: Production Deployment

**4.1 Create Production Backup:**

```bash
# Database backup
docker compose exec db mysqldump -u root -p shopdb > ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql

# VM snapshot
/vm-snapshot create "before-deploy-$BUNDLE_DATE"
```

**4.2 Notify Users:**

```
Maintenance Window: [start time] - [end time]
ShopDB will be unavailable during deployment.
Expected downtime: 15-30 minutes
```

**4.3 Execute Deployment Runbook:**

Follow `deployment-runbook-$BUNDLE_DATE.md` steps exactly.

**4.4 Monitor Deployment:**

Watch logs during restart:
```bash
docker compose logs -f api
```

Look for:
- [X] "Starting gunicorn"
- [X] "Listening at https://0.0.0.0:5001"
- [X] "Loaded plugins: 12"
- [ ] Exceptions or errors

**4.5 Initial Verification:**

```bash
# Quick checks (< 2 min)
docker compose ps                          # All running?
curl -k https://localhost:5001/api/health  # API responds?
curl -k https://localhost:5001/           # Frontend loads?
```

### Phase 5: Post-Deploy Validation

**5.1 Run Health Checks:**

```bash
/production-health
```

**Expected:** All checks pass, no critical errors.

**5.2 Verify Critical Workflows:**

Test each critical workflow:

1. **Login:**
   - Test admin login
   - Test standard user login
   - Verify sessions work

2. **Asset Management:**
   - Create test asset
   - Search for asset
   - Update asset
   - Delete test asset

3. **Endpoint Reporting:**
   - Check endpoint list loads
   - Verify recent heartbeats (< 10 min)
   - Check endpoint details page

4. **Plugin Loading:**
   ```bash
   docker compose exec api flask plugin list
   ```
   Expected: All plugins loaded, no import errors

**5.3 Monitor for 1 Hour:**

Watch logs for anomalies:
```bash
# Check every 10 minutes for first hour
docker compose logs api --since 10m | grep -i "error\|exception\|critical"
```

**5.4 Verify Endpoint Agents Still Reporting:**

```bash
# Check last_seen timestamps
docker compose exec api flask shell -c "
from plugins.endpoints.models import Endpoint
from datetime import datetime, timedelta
recent = Endpoint.query.filter(
    Endpoint.last_seen > datetime.utcnow() - timedelta(minutes=10)
).count()
print(f'Endpoints reporting in last 10 min: {recent}')
"
```

Expected: At least 1 (VM should report)

### Phase 6: Rollback (If Issues Detected)

**Rollback Decision Criteria:**

**MUST rollback if:**
- Database migration failed
- Critical plugin won't load
- API completely non-functional
- Data integrity issues

**SHOULD rollback if:**
- Multiple endpoints not reporting
- Critical workflow broken
- Performance severely degraded
- Security vulnerability introduced

**Can monitor and fix if:**
- Single plugin has minor issue
- Non-critical feature broken
- Performance slightly slower
- Cosmetic UI issues

**Rollback Procedure:**

**Option A: VM Snapshot Restore (Fastest)**
```bash
/vm-snapshot restore "before-deploy-$BUNDLE_DATE"
```

Time: ~5 minutes
Scope: Complete system rollback

**Option B: Git Rollback**
```bash
git checkout $LAST_TAG
docker compose down
docker compose -f docker-compose.prebuilt.yml build api
docker compose up -d
```

Time: ~10 minutes
Scope: Code only (database may need restore)

**Option C: Database Restore (If Schema Changed)**
```bash
# Restore database backup
docker compose exec db mysql -u root -p shopdb < ~/backups/shopdb-pre-deploy-$BUNDLE_DATE.sql

# Or point-in-time recovery if binlog enabled
```

Time: ~15-30 minutes depending on database size

**After Rollback:**
1. Verify system functional
2. Document what went wrong
3. Fix issues before retry
4. Notify users system restored

### Phase 7: Deployment Complete

**7.1 Update Documentation:**

- Document any issues encountered
- Update deployment notes
- Record deployment time and downtime
- Note any manual steps required

**7.2 Notify Stakeholders:**

```
Deployment complete: ShopDB v$TAG
- Deployment time: [duration]
- Downtime: [actual downtime]
- Status: Success / Issues noted
- Changes: [link to changelog]
```

**7.3 Monitor for 24 Hours:**

Set reminder to check:
- Logs for errors
- Endpoint reporting
- User feedback
- Performance metrics

**7.4 Cleanup:**

```bash
# Remove old bundles (> 30 days)
find ~/_transfer/ -name "shopdb-deploy-*.tar.gz" -mtime +30 -delete

# Remove old backups (> 30 days)
find ~/backups/ -name "shopdb-pre-deploy-*.sql" -mtime +30 -delete

# Clean up old VM snapshots
/vm-snapshot cleanup
```

## Time Estimates

| Phase | Time | Can Fail? |
|-------|------|-----------|
| Pre-deploy validation | 10-15 min | Yes (fix issues) |
| Bundle creation | 5 min | No |
| Stage deploy (dev) | 15-20 min | Yes (fix before prod) |
| Production backup | 5 min | No |
| Production deploy | 15-20 min | Yes (rollback) |
| Post-deploy validation | 15-30 min | Yes (rollback) |
| **Total** | **65-100 min** | |

**Downtime:** 5-15 minutes (during production deploy phase)

## Automation Opportunities

**Current:** 312 hrs/year coordination burden

**With this skill:** ~50-100 hrs saved/year

**Further automation:**
- Automated bundle transfer (if air-gap can be bridged)
- Automated staging tests
- Automated rollback triggers
- Continuous deployment (requires air-gap solution)

## Troubleshooting

**Bundle extraction fails:**
- Check disk space: `df -h`
- Verify bundle integrity: `tar -tzf bundle.tar.gz`
- Try smaller extraction: `tar -xzf bundle.tar.gz --one-top-level`

**Migration fails:**
- Check database connectivity
- Verify migration order (ADR-008)
- Check for data that violates new constraints
- See `/migration-check` for detailed validation

**Services won't start:**
- Check logs: `docker compose logs api`
- Verify port availability: `netstat -tulpn | grep 5001`
- Check Docker status: `docker ps -a`
- Try rebuild: `/docker-rebuild`

**Rollback fails:**
- VM snapshot may be corrupted
- Try alternative rollback method
- Contact system admin if all methods fail

## Related

- `/pre-commit` - Pre-deployment validation
- `/production-health` - Health monitoring
- `/migration-check` - Database migration validation
- `/docker-rebuild` - Container rebuild
- `/vm-snapshot` - VM snapshot management
- [[workflow-patterns]] - Common workflows
- [[maintenance-footprint-baseline]] - 312 hrs/year coordination burden
- ADR-004: Deployment topology