shopdb/sql/PROD_IMPORT_README.md

# Production Data Import - Process Documentation

**Date Created:** 2025-11-13
**Purpose:** Import notifications, notificationtypes, printers, and knowledgebase from production backup into dev database

## Overview

This process imports the latest production data while avoiding duplicates and preserving any dev-specific test data.

## Files Created

### Extraction Scripts
- `extract_prod_data.sh` - Extracts complete table dumps (structure + data) from production backup
- `extract_prod_data_inserts_only.sh` - Extracts only INSERT statements (no table structure)

### Extracted Data Files
- `prod_notifications.sql` - Full notifications table dump from production
- `prod_notificationtypes.sql` - Full notificationtypes table dump from production
- `prod_printers.sql` - Full printers table dump from production
- `prod_knowledgebase.sql` - Full knowledgebase table dump from production

### INSERT-Only Files (for clean import)
- `prod_notifications_inserts.sql` - Only INSERT statements for notifications (56 rows)
- `prod_notificationtypes_inserts.sql` - Only INSERT statements for notification types (4 rows)
- `prod_printers_inserts.sql` - Only INSERT statements for printers (45 rows)
- `prod_knowledgebase_inserts.sql` - Only INSERT statements for knowledgebase (214 rows)

### Import Scripts
- `import_prod_data.sql` - Original import script (uses SOURCE command, doesn't work with Docker)
- `import_prod_data_execute.sql` - Attempted direct execution (failed due to Docker file access)
- `import_prod_data_direct.sh` - First working version (cleared all data before import)
- `import_prod_data_clean.sh` - Improved version (INSERT-only approach, but syntax errors)
- **`import_prod_data_smart.sh`** - FINAL VERSION (uses REPLACE to handle duplicates intelligently)

### Rollback Script
- `rollback_prod_import.sh` - Restores data from backup tables if import fails or needs to be undone

## Import Strategy

### Approach: Smart Merge with REPLACE

Instead of DELETE + INSERT (which loses everything), we use **REPLACE INTO** which:
1. Inserts new records that don't exist
2. Updates existing records with matching primary keys
3. Preserves the integrity of the data

### Tables Imported

1. **notificationtypes** (lookup table) - 4 types
   - Info
   - Warning
   - Danger
   - Success

2. **notifications** - 56 production notifications
   - Active notifications for current awareness
   - Historical notifications for reference
   - Linked to notificationtypes via foreign key

3. **printers** - 45 production printers
   - Active printer configurations
   - IP addresses, models, locations
   - Printer install paths for self-service

4. **knowledgebase** - 214 knowledge base articles
   - How-to guides
   - Documentation links
   - Troubleshooting articles
   - Click tracking for popular articles

## Schema Differences Handled

### Issue 1: printers table - `fqdn` column
- **Production:** Has `fqdn` column
- **Dev:** Already has `fqdn` column
- **Resolution:** No change needed, column exists

### Issue 2: Extracted SQL includes table structure
- **Problem:** Full dumps include CREATE TABLE which conflicts with existing tables
- **Solution:** Extract only INSERT statements using grep

### Issue 3: INSERT statement formatting
- **Problem:** Original extraction had syntax errors with multiple INSERT statements
- **Solution:** Ensure proper SQL formatting and statement separation

## Bugs Fixed

### machine_edit.asp (Line 157) - Column Name Mismatch
**Error:** `Item cannot be found in the collection corresponding to the requested name or ordinal`

**Root Cause:** Code was using underscore-separated column names, but database has camelCase names

**Fixed Columns:**
- `is_third_party_managed` → `isthirdpartymanaged`
- `third_party_manager` → `thirdpartymanager`
- `ot_asset_system` → (column doesn't exist, removed reference)
- `ot_asset_device_type` → `dodassettype`

**File:** `/home/camp/projects/windows/shopdb/machine_edit.asp:157-160`

## Usage Instructions

### To Import Production Data

```bash
cd /home/camp/projects/windows/shopdb/sql
bash import_prod_data_smart.sh
```

The script will:
1. Create backup tables with `_backup_20251113` suffix
2. Show current counts (before import)
3. Use REPLACE INTO to merge production data
4. Show new counts (after import)
5. Display sample data for verification
6. Show comparison of before/after counts

### To Rollback Import

If something goes wrong or you need to restore the previous state:

```bash
cd /home/camp/projects/windows/shopdb/sql
bash rollback_prod_import.sh
```

Type `yes` when prompted to confirm rollback.

## Current Dev Database Status

**Before Any Import (baseline):**
- Notifications: 34
- Notification Types: 4
- Printers: 41
- Knowledgebase: 204

**After Failed Import Attempts:**
- Rolled back successfully to baseline counts

**Expected After Successful Smart Import:**
- Notifications: ~56 (replaced with production data)
- Notification Types: 4 (unchanged, same in prod)
- Printers: ~45 (replaced with production data)
- Knowledgebase: ~214 (merged with production data)

## Source Data

**Production Backup File:** `/home/camp/projects/windows/database-backup-11-13-25-eod.sql`
- **Date:** November 13, 2025 (End of Day)
- **Size:** 1.4 MB
- **Database:** shopdb

## Verification Queries

After import, verify data with these queries:

```sql
-- Check counts
SELECT
    (SELECT COUNT(*) FROM notifications) AS notifications,
    (SELECT COUNT(*) FROM notificationtypes) AS notificationtypes,
    (SELECT COUNT(*) FROM printers) AS printers,
    (SELECT COUNT(*) FROM knowledgebase) AS knowledgebase;

-- Check active notifications
SELECT notificationid, notification, starttime, endtime, isactive
FROM notifications
WHERE isactive = 1
ORDER BY starttime DESC
LIMIT 10;

-- Check active printers
SELECT printerid, printerwindowsname, ipaddress, isactive
FROM printers
WHERE isactive = 1
LIMIT 10;

-- Check popular knowledge base articles
SELECT linkid, shortdescription, clicks, isactive
FROM knowledgebase
WHERE isactive = 1
ORDER BY clicks DESC
LIMIT 10;
```

## Lessons Learned

1. **Docker MySQL Source Command Limitation:** The SOURCE command in MySQL doesn't work well when MySQL is in a Docker container and files are on the host. Solution: Use cat to pipe SQL into docker exec.

2. **Schema Drift:** Production and dev databases had schema differences. Always check column names match before importing.

3. **Full Dumps vs INSERT-Only:** Extracting full table dumps causes conflicts. Extract only INSERT statements for cleaner imports.

4. **REPLACE vs DELETE+INSERT:** Using REPLACE INTO is safer as it handles duplicates intelligently without losing all data first.

5. **Always Backup First:** Creating backup tables before import is essential for quick rollback capability.

## TODO for Future Improvements

- [ ] Create automated daily/weekly sync script
- [ ] Add validation checks before import (schema comparison)
- [ ] Create migration script to align dev schema with production
- [ ] Add logging of import results to a database table
- [ ] Create web UI for managing imports

## Support

For issues or questions, check:
- IIS Logs: `/home/camp/projects/windows/logs/shopdb/ex*.log`
- This documentation: `/home/camp/projects/windows/shopdb/sql/PROD_IMPORT_README.md`
- Session notes: `/home/camp/projects/windows/shopdb/SESSION_SUMMARY_2025-11-13.md`