shopdb/sql/PROD_IMPORT_README.md

Production Data Import - Process Documentation

Date Created: 2025-11-13 Purpose: Import notifications, notificationtypes, printers, and knowledgebase from production backup into dev database

Overview

This process imports the latest production data while avoiding duplicates and preserving any dev-specific test data.

Files Created

Extraction Scripts

• extract_prod_data.sh - Extracts complete table dumps (structure + data) from production backup
• extract_prod_data_inserts_only.sh - Extracts only INSERT statements (no table structure)

Extracted Data Files

• prod_notifications.sql - Full notifications table dump from production
• prod_notificationtypes.sql - Full notificationtypes table dump from production
• prod_printers.sql - Full printers table dump from production
• prod_knowledgebase.sql - Full knowledgebase table dump from production

INSERT-Only Files (for clean import)

• prod_notifications_inserts.sql - Only INSERT statements for notifications (56 rows)
• prod_notificationtypes_inserts.sql - Only INSERT statements for notification types (4 rows)
• prod_printers_inserts.sql - Only INSERT statements for printers (45 rows)
• prod_knowledgebase_inserts.sql - Only INSERT statements for knowledgebase (214 rows)

Import Scripts

• import_prod_data.sql - Original import script (uses SOURCE command, doesn't work with Docker)
• import_prod_data_execute.sql - Attempted direct execution (failed due to Docker file access)
• import_prod_data_direct.sh - First working version (cleared all data before import)
• import_prod_data_clean.sh - Improved version (INSERT-only approach, but syntax errors)
• import_prod_data_smart.sh - FINAL VERSION (uses REPLACE to handle duplicates intelligently)

Rollback Script

• rollback_prod_import.sh - Restores data from backup tables if import fails or needs to be undone

Import Strategy

Approach: Smart Merge with REPLACE

Instead of DELETE + INSERT (which loses everything), we use REPLACE INTO which:

1. Inserts new records that don't exist
2. Updates existing records with matching primary keys
3. Preserves the integrity of the data

Tables Imported

1. notificationtypes (lookup table) - 4 types

   • Info
   • Warning
   • Danger
   • Success

2. notifications - 56 production notifications

   • Active notifications for current awareness
   • Historical notifications for reference
   • Linked to notificationtypes via foreign key

3. printers - 45 production printers

   • Active printer configurations
   • IP addresses, models, locations
   • Printer install paths for self-service

4. knowledgebase - 214 knowledge base articles

   • How-to guides
   • Documentation links
   • Troubleshooting articles
   • Click tracking for popular articles

Schema Differences Handled

Issue 1: printers table - fqdn column

• Production: Has fqdn column
• Dev: Already has fqdn column
• Resolution: No change needed, column exists

Issue 2: Extracted SQL includes table structure

• Problem: Full dumps include CREATE TABLE which conflicts with existing tables
• Solution: Extract only INSERT statements using grep

Issue 3: INSERT statement formatting

• Problem: Original extraction had syntax errors with multiple INSERT statements
• Solution: Ensure proper SQL formatting and statement separation

Bugs Fixed

machine_edit.asp (Line 157) - Column Name Mismatch

Error: Item cannot be found in the collection corresponding to the requested name or ordinal

Root Cause: Code was using underscore-separated column names, but database has camelCase names

Fixed Columns:

• is_third_party_managed → isthirdpartymanaged
• third_party_manager → thirdpartymanager
• ot_asset_system → (column doesn't exist, removed reference)
• ot_asset_device_type → dodassettype

File: /home/camp/projects/windows/shopdb/machine_edit.asp:157-160

Usage Instructions

To Import Production Data

cd /home/camp/projects/windows/shopdb/sql
bash import_prod_data_smart.sh

The script will:

1. Create backup tables with _backup_20251113 suffix
2. Show current counts (before import)
3. Use REPLACE INTO to merge production data
4. Show new counts (after import)
5. Display sample data for verification
6. Show comparison of before/after counts

To Rollback Import

If something goes wrong or you need to restore the previous state:

cd /home/camp/projects/windows/shopdb/sql
bash rollback_prod_import.sh

Type yes when prompted to confirm rollback.

Current Dev Database Status

Before Any Import (baseline):

• Notifications: 34
• Notification Types: 4
• Printers: 41
• Knowledgebase: 204

After Failed Import Attempts:

• Rolled back successfully to baseline counts

Expected After Successful Smart Import:

• Notifications: ~56 (replaced with production data)
• Notification Types: 4 (unchanged, same in prod)
• Printers: ~45 (replaced with production data)
• Knowledgebase: ~214 (merged with production data)

Source Data

Production Backup File: /home/camp/projects/windows/database-backup-11-13-25-eod.sql

• Date: November 13, 2025 (End of Day)
• Size: 1.4 MB
• Database: shopdb

Verification Queries

After import, verify data with these queries:

-- Check counts
SELECT
    (SELECT COUNT(*) FROM notifications) AS notifications,
    (SELECT COUNT(*) FROM notificationtypes) AS notificationtypes,
    (SELECT COUNT(*) FROM printers) AS printers,
    (SELECT COUNT(*) FROM knowledgebase) AS knowledgebase;

-- Check active notifications
SELECT notificationid, notification, starttime, endtime, isactive
FROM notifications
WHERE isactive = 1
ORDER BY starttime DESC
LIMIT 10;

-- Check active printers
SELECT printerid, printerwindowsname, ipaddress, isactive
FROM printers
WHERE isactive = 1
LIMIT 10;

-- Check popular knowledge base articles
SELECT linkid, shortdescription, clicks, isactive
FROM knowledgebase
WHERE isactive = 1
ORDER BY clicks DESC
LIMIT 10;

Lessons Learned

1. Docker MySQL Source Command Limitation: The SOURCE command in MySQL doesn't work well when MySQL is in a Docker container and files are on the host. Solution: Use cat to pipe SQL into docker exec.

2. Schema Drift: Production and dev databases had schema differences. Always check column names match before importing.

3. Full Dumps vs INSERT-Only: Extracting full table dumps causes conflicts. Extract only INSERT statements for cleaner imports.

4. REPLACE vs DELETE+INSERT: Using REPLACE INTO is safer as it handles duplicates intelligently without losing all data first.

5. Always Backup First: Creating backup tables before import is essential for quick rollback capability.

TODO for Future Improvements

• Create automated daily/weekly sync script
• Add validation checks before import (schema comparison)
• Create migration script to align dev schema with production
• Add logging of import results to a database table
• Create web UI for managing imports

Support

For issues or questions, check:

• IIS Logs: /home/camp/projects/windows/logs/shopdb/ex*.log
• This documentation: /home/camp/projects/windows/shopdb/sql/PROD_IMPORT_README.md
• Session notes: /home/camp/projects/windows/shopdb/SESSION_SUMMARY_2025-11-13.md