shopdb/sql/DATA_MIGRATION_EXPLAINED.md

# Data Migration Scripts - What They Actually Do

**Date:** 2025-11-20
**Purpose:** Explain exactly what each data migration script does to production data
**Critical:** This preserves ALL production data while restructuring for Phase 2

---

## Overview: What We're Migrating

Your production database has data split across OLD tables that needs to be consolidated into NEW Phase 2 tables:

### Source Data (OLD Tables - Production)
- **pc table:** 286 PCs with hostnames, serial numbers, OS, etc.
- **machines table:** 275 equipment/machines (CNCs, lathes, mills, etc.)
- **pc_network_interfaces:** 705 network interfaces (IP addresses, MAC addresses)
- **pc_dualpath_assignments:** PC relationships for redundancy
- **machine_pc_relationships:** Which PCs control which equipment
- **machines.ipaddress1/ipaddress2:** IP addresses stored in machine records

### Destination (NEW Phase 2 Structure)
- **machines table:** Will contain BOTH 275 equipment + 286 PCs = 561 total records
- **communications table:** Will contain ALL network interfaces (from machines + PCs)
- **machinerelationships table:** Will contain ALL relationships (dualpath + control)
- **machinestatus table:** Replaces pcstatus (renamed for broader scope)

---

## Migration Script Breakdown

### Script 1: Backup Machine IP Addresses
**File:** `01_backup_machine_ips.sql`

**What It Does:**
```sql
-- Create temporary backup table
CREATE TABLE _backup_machine_ips AS
SELECT
  machineid,
  machinenumber,
  ipaddress1,
  ipaddress2
FROM machines
WHERE ipaddress1 IS NOT NULL OR ipaddress2 IS NOT NULL;
```

**Why:**
- Machines table currently has `ipaddress1` and `ipaddress2` columns
- We're about to DROP these columns (ALTER TABLE)
- But we need this data to move to the `communications` table
- This backup preserves ~275 machine IP addresses before columns are dropped

**Data Example:**
```
machineid | machinenumber | ipaddress1      | ipaddress2
----------|---------------|-----------------|----------------
1         | 0600          | 10.80.11.100    | NULL
2         | 0612          | 10.80.11.101    | 10.80.11.102
3         | 3001          | 10.80.12.50     | NULL
```

**Result:** Temporary backup table with all machine IPs preserved

---

### Script 2: ALTER Machines Table (Add Phase 2 Columns)
**File:** `02_alter_machines.sql`

**What It Does:**
```sql
ALTER TABLE machines
  -- Add new Phase 2 columns for PCs
  ADD COLUMN hostname varchar(100) DEFAULT NULL,
  ADD COLUMN serialnumber varchar(50) DEFAULT NULL,
  ADD COLUMN loggedinuser varchar(100) DEFAULT NULL,
  ADD COLUMN pctypeid int(11) DEFAULT NULL,
  ADD COLUMN osid int(11) DEFAULT NULL,
  ADD COLUMN controllertypeid int(11) DEFAULT NULL,
  ADD COLUMN controllerosid int(11) DEFAULT NULL,
  ADD COLUMN controllermodelid int(11) DEFAULT NULL,
  ADD COLUMN machinestatusid int(11) DEFAULT NULL,
  ADD COLUMN lastupdated datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  ADD COLUMN requires_manual_machine_config tinyint(1) DEFAULT '0',

  -- Remove old IP columns (data already backed up)
  DROP COLUMN ipaddress1,
  DROP COLUMN ipaddress2;
```

**Why:**
- Makes machines table able to store BOTH equipment AND PCs
- PCs identified by `pctypeid IS NOT NULL`
- Equipment identified by `pctypeid IS NULL`
- Removes IP columns because they'll go in communications table

**Impact on Existing Data:**
- ✅ All 275 existing machines preserved (no data loss)
- ✅ New columns added with NULL defaults (safe)
- ✅ ipaddress1/ipaddress2 dropped (but backed up in Script 1)

**After This Script:**
```
machines table now has:
- 275 existing equipment records (pctypeid = NULL)
- Ready to receive 286 PC records (pctypeid = NOT NULL)
```

---

### Script 3: Create New Phase 2 Tables
**File:** `05_create_phase2_tables.sql`

**What It Does:**
```sql
-- Create communications table
CREATE TABLE communications (
  comid int(11) NOT NULL AUTO_INCREMENT,
  machineid int(11) NOT NULL,
  comstypeid int(11) NOT NULL DEFAULT 1,
  address varchar(100) DEFAULT NULL,  -- IP address or COM port
  macaddress varchar(17) DEFAULT NULL,
  interfacename varchar(50) DEFAULT NULL,
  isprimary tinyint(1) DEFAULT 0,
  isactive tinyint(1) DEFAULT 1,
  settings text,
  PRIMARY KEY (comid)
);

-- Create machinerelationships table
CREATE TABLE machinerelationships (
  relationshipid int(11) NOT NULL AUTO_INCREMENT,
  machineid int(11) NOT NULL,
  related_machineid int(11) NOT NULL,
  relationshiptypeid int(11) NOT NULL,
  isactive tinyint(1) DEFAULT 1,
  notes text,
  PRIMARY KEY (relationshipid)
);

-- Create machinestatus table (copy from pcstatus)
CREATE TABLE machinestatus (
  machinestatusid int(11) NOT NULL AUTO_INCREMENT,
  machinestatus varchar(50) NOT NULL,
  PRIMARY KEY (machinestatusid)
);

INSERT INTO machinestatus (machinestatusid, machinestatus)
SELECT pcstatusid, pcstatus FROM pcstatus;

-- Create other Phase 2 tables (compliance, warranties, etc.)
```

**Why:**
- Creates infrastructure for unified data storage
- No impact on existing data (these are new, empty tables)

**After This Script:**
```
New empty tables created:
- communications (ready for network interface data)
- machinerelationships (ready for PC relationships)
- machinestatus (populated with pcstatus data)
- compliance, warranties, etc. (empty, ready for future data)
```

---

### Script 4: Migrate PCs from pc → machines
**File:** `06_migrate_pcs.sql`

**What It Does:**
```sql
INSERT INTO machines (
  hostname,           -- PC hostname
  serialnumber,       -- PC serial number
  loggedinuser,       -- Currently logged in user
  pctypeid,           -- PC type (Engineer, Shopfloor, Standard, etc.)
  osid,               -- Operating system
  modelnumberid,      -- PC model (Dell, HP, etc.)
  businessunitid,     -- Business unit assignment
  machinestatusid,    -- Status (In Use, Spare, etc.)
  machinenumber,      -- Machine this PC is assigned to
  alias,              -- PC alias/friendly name
  machinenotes,       -- Notes (was 'notes' in pc table)
  printerid,          -- Assigned printer
  isactive,           -- Active flag
  islocationonly      -- Always 0 for PCs
)
SELECT
  p.hostname,
  p.serialnumber,
  p.loggedinuser,
  p.pctypeid,         -- This makes it a PC (NOT NULL = PC)
  p.osid,
  p.modelnumberid,
  p.businessunitid,
  p.pcstatusid AS machinestatusid,  -- Map pcstatusid to machinestatusid
  p.machinenumber,
  p.alias,
  p.notes AS machinenotes,           -- Column rename
  p.printerid,
  p.isactive,
  0 AS islocationonly
FROM pc p
WHERE p.isactive = 1;
```

**Why:**
- Copies all 286 PCs from separate `pc` table into `machines` table
- Sets `pctypeid` to NOT NULL (this identifies them as PCs, not equipment)
- Renames `notes` → `machinenotes` to match Phase 2 schema
- Maps `pcstatusid` → `machinestatusid`

**Before:**
```
machines table: 275 equipment records
pc table: 286 PC records
```

**After:**
```
machines table: 561 total records
  - 275 equipment (pctypeid = NULL)
  - 286 PCs (pctypeid = 1, 2, 3, 4, 5, etc.)
pc table: 286 records (unchanged - kept as backup)
```

**Data Example:**
```
OLD (pc table):
pcid | hostname      | serialnumber | pctypeid | machinenumber
-----|---------------|--------------|----------|---------------
1    | SHOP-PC-01    | ABC123       | 3        | 0600
2    | ENG-WS-05     | XYZ789       | 2        | 3001

NEW (machines table after migration):
machineid | hostname      | serialnumber | pctypeid | machinenumber
----------|---------------|--------------|----------|---------------
276       | SHOP-PC-01    | ABC123       | 3        | 0600
277       | ENG-WS-05     | XYZ789       | 2        | 3001
```

**Important:**
- PC table is NOT dropped (kept as backup for 30 days)
- Duplicate check prevents re-running script

---

### Script 5: Migrate Machine IPs → communications
**File:** `07_migrate_machine_ips.sql`

**What It Does:**
```sql
-- Migrate ipaddress1 (primary interface)
INSERT INTO communications (
  machineid,
  comstypeid,     -- 1 = Network/TCP-IP
  address,        -- The IP address
  isprimary,      -- 1 = primary interface
  interfacename,  -- "Interface 1"
  isactive
)
SELECT
  machineid,
  1 AS comstypeid,
  ipaddress1,
  1 AS isprimary,
  'Interface 1' AS interfacename,
  1 AS isactive
FROM _backup_machine_ips
WHERE ipaddress1 IS NOT NULL AND ipaddress1 != '';

-- Migrate ipaddress2 (secondary interface)
INSERT INTO communications (
  machineid,
  comstypeid,
  address,
  isprimary,      -- 0 = secondary interface
  interfacename,  -- "Interface 2"
  isactive
)
SELECT
  machineid,
  1 AS comstypeid,
  ipaddress2,
  0 AS isprimary,
  'Interface 2' AS interfacename,
  1 AS isactive
FROM _backup_machine_ips
WHERE ipaddress2 IS NOT NULL AND ipaddress2 != '';
```

**Why:**
- Recovers IP addresses that were in machines.ipaddress1 and ipaddress2
- Moves them to unified communications table
- Marks ipaddress1 as primary, ipaddress2 as secondary

**Data Example:**
```
OLD (machines table - columns dropped):
machineid | ipaddress1    | ipaddress2
----------|---------------|-------------
1         | 10.80.11.100  | NULL
2         | 10.80.11.101  | 10.80.11.102

NEW (communications table):
comid | machineid | address       | isprimary | interfacename
------|-----------|---------------|-----------|---------------
1     | 1         | 10.80.11.100  | 1         | Interface 1
2     | 2         | 10.80.11.101  | 1         | Interface 1
3     | 2         | 10.80.11.102  | 0         | Interface 2
```

**Result:** ~275 machine IP addresses preserved in communications table

---

### Script 6: Migrate PC Network Interfaces → communications
**File:** `08_migrate_pc_interfaces.sql`

**What It Does:**
```sql
INSERT INTO communications (
  machineid,      -- PC's new machineid (from machines table)
  comstypeid,     -- 1 = Network
  address,        -- IP address
  macaddress,     -- MAC address
  isprimary,      -- Primary interface flag
  interfacename,  -- "Interface 1", "Interface 2", etc.
  isactive
)
SELECT
  m.machineid,    -- Find PC's new machineid by matching hostname
  1 AS comstypeid,
  pni.ipaddress,
  pni.macaddress,
  pni.isprimary,
  CONCAT('Interface ',
    ROW_NUMBER() OVER (PARTITION BY m.machineid ORDER BY pni.isprimary DESC)
  ) AS interfacename,
  pni.isactive
FROM pc_network_interfaces pni
JOIN pc p ON pni.pcid = p.pcid
JOIN machines m ON m.hostname = p.hostname AND m.pctypeid IS NOT NULL
WHERE pni.isactive = 1;
```

**Why:**
- Copies all 705 network interfaces from `pc_network_interfaces` table
- Joins to find PC's NEW machineid in machines table (by hostname match)
- Preserves IP addresses, MAC addresses, primary flag
- Generates interface names (Interface 1, Interface 2, Interface 3)

**Data Example:**
```
OLD (pc_network_interfaces):
interfaceid | pcid | ipaddress     | macaddress        | isprimary
------------|------|---------------|-------------------|----------
1           | 1    | 10.80.50.10   | 00:11:22:33:44:55 | 1
2           | 1    | 10.80.50.11   | 00:11:22:33:44:56 | 0
3           | 2    | 10.80.50.20   | 00:11:22:33:44:57 | 1

PC table:
pcid | hostname
-----|------------
1    | SHOP-PC-01
2    | ENG-WS-05

Machines table (after PC migration):
machineid | hostname      | pctypeid
----------|---------------|----------
276       | SHOP-PC-01    | 3
277       | ENG-WS-05     | 2

NEW (communications table):
comid | machineid | address       | macaddress        | isprimary | interfacename
------|-----------|---------------|-------------------|-----------|---------------
4     | 276       | 10.80.50.10   | 00:11:22:33:44:55 | 1         | Interface 1
5     | 276       | 10.80.50.11   | 00:11:22:33:44:56 | 0         | Interface 2
6     | 277       | 10.80.50.20   | 00:11:22:33:44:57 | 1         | Interface 1
```

**Result:** All 705 PC network interfaces migrated to communications table

---

### Script 7: Migrate PC Dualpath Relationships → machinerelationships
**File:** `09_migrate_relationships.sql`

**What It Does:**
```sql
-- Part 1: Migrate Dualpath (PC ↔ PC redundancy)
INSERT INTO machinerelationships (
  machineid,           -- Primary PC
  related_machineid,   -- Dualpath PC
  relationshiptypeid,  -- 2 = Dualpath
  isactive
)
SELECT
  m1.machineid,              -- Primary PC's new machineid
  m2.machineid,              -- Dualpath PC's new machineid
  2 AS relationshiptypeid,   -- Dualpath relationship type
  1 AS isactive
FROM pc_dualpath_assignments pda
JOIN pc p1 ON pda.pcid = p1.pcid
JOIN pc p2 ON pda.dualpath_pcid = p2.pcid
JOIN machines m1 ON m1.hostname = p1.hostname AND m1.pctypeid IS NOT NULL
JOIN machines m2 ON m2.hostname = p2.hostname AND m2.pctypeid IS NOT NULL;

-- Part 2: Migrate Controls (PC → Equipment)
INSERT INTO machinerelationships (
  machineid,           -- PC that controls equipment
  related_machineid,   -- Equipment being controlled
  relationshiptypeid,  -- 1 = Controls
  isactive
)
SELECT
  mpc.pcid AS machineid,        -- PC's machineid (from machines)
  mpc.machineid AS related_machineid,  -- Equipment's machineid
  1 AS relationshiptypeid,      -- Controls relationship
  mpc.isactive
FROM machine_pc_relationships mpc;
```

**Why:**
- Preserves PC redundancy relationships (dualpath)
- Preserves which PCs control which equipment
- Uses new unified machinerelationships table

**Data Example:**

**Dualpath (PC redundancy):**
```
OLD (pc_dualpath_assignments):
assignmentid | pcid | dualpath_pcid
-------------|------|---------------
1            | 5    | 12

PC table:
pcid | hostname
-----|-------------
5    | CNC-PC-01
12   | CNC-PC-02

Machines (after migration):
machineid | hostname    | pctypeid
----------|-------------|----------
280       | CNC-PC-01   | 3
291       | CNC-PC-02   | 3

NEW (machinerelationships):
relationshipid | machineid | related_machineid | relationshiptypeid
---------------|-----------|-------------------|-------------------
1              | 280       | 291               | 2 (Dualpath)
```

**Controls (PC → Equipment):**
```
OLD (machine_pc_relationships):
relationshipid | pcid | machineid
---------------|------|----------
1              | 280  | 50

NEW (machinerelationships):
relationshipid | machineid | related_machineid | relationshiptypeid
---------------|-----------|-------------------|-------------------
2              | 280       | 50                | 1 (Controls)
```

**Meaning:**
- PC with machineid 280 controls equipment with machineid 50
- PC with machineid 280 has dualpath redundancy with PC machineid 291

**Result:** All PC relationships preserved in unified table

---

## Summary: What Happens to Your Data

### Before Migration:
```
pc table:                    286 PCs
machines table:              275 equipment
pc_network_interfaces:       705 network interfaces
machine IPs (in machines):   ~275 IP addresses
pc_dualpath_assignments:     ~50 relationships
machine_pc_relationships:    ~100 relationships
```

### After Migration:
```
machines table:              561 records (275 equipment + 286 PCs)
communications table:        ~980 records (275 machine IPs + 705 PC interfaces)
machinerelationships table:  ~150 records (dualpath + controls)
machinestatus table:         Copied from pcstatus

OLD TABLES KEPT AS BACKUP (for 30 days):
pc table:                    286 records (unchanged)
pc_network_interfaces:       705 records (unchanged)
pc_dualpath_assignments:     ~50 records (unchanged)
machine_pc_relationships:    ~100 records (unchanged)
```

---

## Critical Safety Features

### 1. No Data Loss
- ✅ Old tables are NOT dropped (kept for 30 days)
- ✅ Can roll back if issues found
- ✅ Data copied, not moved

### 2. Duplicate Prevention
```sql
-- Example: Only insert PCs that don't already exist
INSERT INTO machines (...)
SELECT ...
FROM pc p
WHERE NOT EXISTS (
  SELECT 1 FROM machines m
  WHERE m.hostname = p.hostname AND m.pctypeid IS NOT NULL
);
```

### 3. Data Validation Queries
After each script runs, verification queries check:
- Row counts match (286 PCs in old table = 286 PCs in new table)
- No duplicate hostnames
- All relationships have valid machine IDs
- No orphaned records

---

## What Gets Modified vs Copied

### MODIFIED (Structure Only):
- **machines table** - Columns added/removed, but existing 275 equipment preserved
- **businessunits table** - Columns added, existing data unchanged
- **controllertypes table** - Columns added, existing data unchanged

### COPIED (Data Duplicated):
- **pc → machines** - 286 PCs copied, pc table kept as backup
- **pc_network_interfaces → communications** - 705 interfaces copied, old table kept
- **machine IPs → communications** - IPs extracted before columns dropped
- **Relationships → machinerelationships** - Copied, old tables kept

### CREATED (New):
- **communications table** - Brand new, receives copied data
- **machinerelationships table** - Brand new, receives copied data
- **machinestatus table** - Created, populated from pcstatus
- **compliance, warranties, etc.** - Created empty

---

## Rollback Strategy

If something goes wrong:

1. **Before ASP deployment:**
   - Drop new tables
   - Restore machines columns (ADD ipaddress1, ipaddress2)
   - Restore machine IPs from backup table
   - Delete migrated PCs from machines table

2. **After ASP deployment:**
   - Revert ASP code to Phase 1
   - Keep data in both old and new tables
   - Fix issues and retry

---

## Timeline

**Estimated execution time:**
- Script 1 (Backup): 1 second
- Script 2 (ALTER machines): 10 seconds
- Script 3 (CREATE tables): 30 seconds
- Script 4 (Migrate PCs): 30 seconds (286 inserts)
- Script 5 (Migrate machine IPs): 10 seconds (~275 records)
- Script 6 (Migrate PC interfaces): 1 minute (705 inserts)
- Script 7 (Migrate relationships): 30 seconds (~150 inserts)

**Total: ~3-4 minutes for data migration**

---

## Questions These Scripts Answer

**Q: Will I lose production data?**
A: No. Old tables are kept as backup. Data is copied, not moved.

**Q: What if the same PC exists in both tables?**
A: Duplicate check prevents re-inserting. Script can be re-run safely.

**Q: Can I roll back?**
A: Yes. Rollback scripts reverse all changes.

**Q: What happens to machine IPs when columns are dropped?**
A: Backed up to temp table first, then migrated to communications.

**Q: Will production be down during migration?**
A: Tables will be locked during ALTER and INSERT operations. Estimated 5-10 minutes.

**Q: What about data added after the backup was taken?**
A: Scripts work on live production database, not the backup file. All current data migrated.

---

**Status:** Explanation Complete
**Next Step:** Create actual SQL scripts based on this plan