Clean up sql directory after production sync
- Remove 27 completed one-off migration scripts - Remove old backup file (dev-backup-20251120) - Remove completed shell scripts for prod import/export - Archive migration_phase1-4 directories and documentation - Keep only view_consolidation.sql as active script All migrations have been applied to production. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
cproudlock
211
sql/archive/PROD_IMPORT_README.md
Normal file
211
sql/archive/PROD_IMPORT_README.md
Normal file
@@ -0,0 +1,211 @@
|
||||
# Production Data Import - Process Documentation
|
||||
|
||||
**Date Created:** 2025-11-13
|
||||
**Purpose:** Import notifications, notificationtypes, printers, and knowledgebase from production backup into dev database
|
||||
|
||||
## Overview
|
||||
|
||||
This process imports the latest production data while avoiding duplicates and preserving any dev-specific test data.
|
||||
|
||||
## Files Created
|
||||
|
||||
### Extraction Scripts
|
||||
- `extract_prod_data.sh` - Extracts complete table dumps (structure + data) from production backup
|
||||
- `extract_prod_data_inserts_only.sh` - Extracts only INSERT statements (no table structure)
|
||||
|
||||
### Extracted Data Files
|
||||
- `prod_notifications.sql` - Full notifications table dump from production
|
||||
- `prod_notificationtypes.sql` - Full notificationtypes table dump from production
|
||||
- `prod_printers.sql` - Full printers table dump from production
|
||||
- `prod_knowledgebase.sql` - Full knowledgebase table dump from production
|
||||
|
||||
### INSERT-Only Files (for clean import)
|
||||
- `prod_notifications_inserts.sql` - Only INSERT statements for notifications (56 rows)
|
||||
- `prod_notificationtypes_inserts.sql` - Only INSERT statements for notification types (4 rows)
|
||||
- `prod_printers_inserts.sql` - Only INSERT statements for printers (45 rows)
|
||||
- `prod_knowledgebase_inserts.sql` - Only INSERT statements for knowledgebase (214 rows)
|
||||
|
||||
### Import Scripts
|
||||
- `import_prod_data.sql` - Original import script (uses SOURCE command, doesn't work with Docker)
|
||||
- `import_prod_data_execute.sql` - Attempted direct execution (failed due to Docker file access)
|
||||
- `import_prod_data_direct.sh` - First working version (cleared all data before import)
|
||||
- `import_prod_data_clean.sh` - Improved version (INSERT-only approach, but syntax errors)
|
||||
- **`import_prod_data_smart.sh`** - FINAL VERSION (uses REPLACE to handle duplicates intelligently)
|
||||
|
||||
### Rollback Script
|
||||
- `rollback_prod_import.sh` - Restores data from backup tables if import fails or needs to be undone
|
||||
|
||||
## Import Strategy
|
||||
|
||||
### Approach: Smart Merge with REPLACE
|
||||
|
||||
Instead of DELETE + INSERT (which loses everything), we use **REPLACE INTO** which:
|
||||
1. Inserts new records that don't exist
|
||||
2. Updates existing records with matching primary keys
|
||||
3. Preserves the integrity of the data
|
||||
|
||||
### Tables Imported
|
||||
|
||||
1. **notificationtypes** (lookup table) - 4 types
|
||||
- Info
|
||||
- Warning
|
||||
- Danger
|
||||
- Success
|
||||
|
||||
2. **notifications** - 56 production notifications
|
||||
- Active notifications for current awareness
|
||||
- Historical notifications for reference
|
||||
- Linked to notificationtypes via foreign key
|
||||
|
||||
3. **printers** - 45 production printers
|
||||
- Active printer configurations
|
||||
- IP addresses, models, locations
|
||||
- Printer install paths for self-service
|
||||
|
||||
4. **knowledgebase** - 214 knowledge base articles
|
||||
- How-to guides
|
||||
- Documentation links
|
||||
- Troubleshooting articles
|
||||
- Click tracking for popular articles
|
||||
|
||||
## Schema Differences Handled
|
||||
|
||||
### Issue 1: printers table - `fqdn` column
|
||||
- **Production:** Has `fqdn` column
|
||||
- **Dev:** Already has `fqdn` column
|
||||
- **Resolution:** No change needed, column exists
|
||||
|
||||
### Issue 2: Extracted SQL includes table structure
|
||||
- **Problem:** Full dumps include CREATE TABLE which conflicts with existing tables
|
||||
- **Solution:** Extract only INSERT statements using grep
|
||||
|
||||
### Issue 3: INSERT statement formatting
|
||||
- **Problem:** Original extraction had syntax errors with multiple INSERT statements
|
||||
- **Solution:** Ensure proper SQL formatting and statement separation
|
||||
|
||||
## Bugs Fixed
|
||||
|
||||
### machine_edit.asp (Line 157) - Column Name Mismatch
|
||||
**Error:** `Item cannot be found in the collection corresponding to the requested name or ordinal`
|
||||
|
||||
**Root Cause:** Code was using underscore-separated column names, but database has camelCase names
|
||||
|
||||
**Fixed Columns:**
|
||||
- `is_third_party_managed` → `isthirdpartymanaged`
|
||||
- `third_party_manager` → `thirdpartymanager`
|
||||
- `ot_asset_system` → (column doesn't exist, removed reference)
|
||||
- `ot_asset_device_type` → `dodassettype`
|
||||
|
||||
**File:** `/home/camp/projects/windows/shopdb/machine_edit.asp:157-160`
|
||||
|
||||
## Usage Instructions
|
||||
|
||||
### To Import Production Data
|
||||
|
||||
```bash
|
||||
cd /home/camp/projects/windows/shopdb/sql
|
||||
bash import_prod_data_smart.sh
|
||||
```
|
||||
|
||||
The script will:
|
||||
1. Create backup tables with `_backup_20251113` suffix
|
||||
2. Show current counts (before import)
|
||||
3. Use REPLACE INTO to merge production data
|
||||
4. Show new counts (after import)
|
||||
5. Display sample data for verification
|
||||
6. Show comparison of before/after counts
|
||||
|
||||
### To Rollback Import
|
||||
|
||||
If something goes wrong or you need to restore the previous state:
|
||||
|
||||
```bash
|
||||
cd /home/camp/projects/windows/shopdb/sql
|
||||
bash rollback_prod_import.sh
|
||||
```
|
||||
|
||||
Type `yes` when prompted to confirm rollback.
|
||||
|
||||
## Current Dev Database Status
|
||||
|
||||
**Before Any Import (baseline):**
|
||||
- Notifications: 34
|
||||
- Notification Types: 4
|
||||
- Printers: 41
|
||||
- Knowledgebase: 204
|
||||
|
||||
**After Failed Import Attempts:**
|
||||
- Rolled back successfully to baseline counts
|
||||
|
||||
**Expected After Successful Smart Import:**
|
||||
- Notifications: ~56 (replaced with production data)
|
||||
- Notification Types: 4 (unchanged, same in prod)
|
||||
- Printers: ~45 (replaced with production data)
|
||||
- Knowledgebase: ~214 (merged with production data)
|
||||
|
||||
## Source Data
|
||||
|
||||
**Production Backup File:** `/home/camp/projects/windows/database-backup-11-13-25-eod.sql`
|
||||
- **Date:** November 13, 2025 (End of Day)
|
||||
- **Size:** 1.4 MB
|
||||
- **Database:** shopdb
|
||||
|
||||
## Verification Queries
|
||||
|
||||
After import, verify data with these queries:
|
||||
|
||||
```sql
|
||||
-- Check counts
|
||||
SELECT
|
||||
(SELECT COUNT(*) FROM notifications) AS notifications,
|
||||
(SELECT COUNT(*) FROM notificationtypes) AS notificationtypes,
|
||||
(SELECT COUNT(*) FROM printers) AS printers,
|
||||
(SELECT COUNT(*) FROM knowledgebase) AS knowledgebase;
|
||||
|
||||
-- Check active notifications
|
||||
SELECT notificationid, notification, starttime, endtime, isactive
|
||||
FROM notifications
|
||||
WHERE isactive = 1
|
||||
ORDER BY starttime DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- Check active printers
|
||||
SELECT printerid, printerwindowsname, ipaddress, isactive
|
||||
FROM printers
|
||||
WHERE isactive = 1
|
||||
LIMIT 10;
|
||||
|
||||
-- Check popular knowledge base articles
|
||||
SELECT linkid, shortdescription, clicks, isactive
|
||||
FROM knowledgebase
|
||||
WHERE isactive = 1
|
||||
ORDER BY clicks DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Docker MySQL Source Command Limitation:** The SOURCE command in MySQL doesn't work well when MySQL is in a Docker container and files are on the host. Solution: Use cat to pipe SQL into docker exec.
|
||||
|
||||
2. **Schema Drift:** Production and dev databases had schema differences. Always check column names match before importing.
|
||||
|
||||
3. **Full Dumps vs INSERT-Only:** Extracting full table dumps causes conflicts. Extract only INSERT statements for cleaner imports.
|
||||
|
||||
4. **REPLACE vs DELETE+INSERT:** Using REPLACE INTO is safer as it handles duplicates intelligently without losing all data first.
|
||||
|
||||
5. **Always Backup First:** Creating backup tables before import is essential for quick rollback capability.
|
||||
|
||||
## TODO for Future Improvements
|
||||
|
||||
- [ ] Create automated daily/weekly sync script
|
||||
- [ ] Add validation checks before import (schema comparison)
|
||||
- [ ] Create migration script to align dev schema with production
|
||||
- [ ] Add logging of import results to a database table
|
||||
- [ ] Create web UI for managing imports
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions, check:
|
||||
- IIS Logs: `/home/camp/projects/windows/logs/shopdb/ex*.log`
|
||||
- This documentation: `/home/camp/projects/windows/shopdb/sql/PROD_IMPORT_README.md`
|
||||
- Session notes: `/home/camp/projects/windows/shopdb/SESSION_SUMMARY_2025-11-13.md`
|
||||
Reference in New Issue
Block a user