README: refresh to current product surface; retire direct-DB migration guide
Overview and plugin list cover all twelve plugins (geenforce, network subnets, measuring tools, warranty, USB); naming examples use living tables/columns instead of retired pctypes/isvnc; API params match the implementation (perpage, dir, assettype); import section points at the IMPORT-API surface and the wjf reference loader. DATA_MIGRATION_GUIDE is now a pointer stub (its direct-DB approach is superseded).
This commit is contained in:
cproudlock
55
README.md
55
README.md
@@ -7,14 +7,20 @@ A modern rewrite of the classic ASP/VBScript ShopDB application using Flask (Pyt
|
||||
ShopDB tracks and manages:
|
||||
- **Machines** - CNC equipment, CMMs, inspection systems, etc.
|
||||
- **PCs** - Shopfloor computers, engineering workstations
|
||||
- **Printers** - Network printers with Zabbix integration
|
||||
- **Applications** - Software deployed across the shop floor
|
||||
- **Printers** - Network printers with Zabbix supply integration
|
||||
- **Network devices** - Switches, routers, and the subnet browser
|
||||
- **Measuring tools** - Gage-lab instruments with calibration tracking
|
||||
- **Applications** - Software deployed across the shop floor, with per-PC install tracking
|
||||
- **Employees** - Directory, recognition and training notifications
|
||||
- **Warranties** - Coverage records with Dell warranty lookups
|
||||
- **USB devices** - CMMC check-in/out tracking
|
||||
- **Knowledge Base** - Documentation and troubleshooting guides
|
||||
- **GE-Enforce manifests** - Imaging/software manifest editing and fleet compliance
|
||||
|
||||
## Tech Stack
|
||||
|
||||
**Backend:**
|
||||
- Python 3.x with Flask
|
||||
- Python 3.12 with Flask
|
||||
- SQLAlchemy ORM
|
||||
- MySQL 5.7+ database (5.6 works with extra utf8mb4 config; see docs/DEPLOY.md)
|
||||
- JWT authentication
|
||||
@@ -59,13 +65,13 @@ To maintain consistency with the legacy ShopDB database and codebase, the follow
|
||||
### Database
|
||||
|
||||
- **Table names:** Lowercase, single word, no underscores or dashes
|
||||
- Examples: `machines`, `pctypes`, `machinetypes`, `businessunits`
|
||||
- Examples: `assets`, `computers`, `printers`, `businessunits`
|
||||
- **Column names:** Lowercase, single word, no underscores or dashes
|
||||
- Examples: `machineid`, `machinenumber`, `pctypeid`, `isactive`, `createddate`
|
||||
- Examples: `assetid`, `assetnumber`, `hostname`, `isactive`, `createddate`
|
||||
- **Foreign keys:** Referenced table name + `id`
|
||||
- Examples: `locationid`, `vendorid`, `modelnumberid`, `pctypeid`
|
||||
- Examples: `locationid`, `vendorid`, `modelnumberid`, `computertypeid`
|
||||
- **Boolean columns:** Prefixed with `is` or `has`
|
||||
- Examples: `isactive`, `isshopfloor`, `isvnc`, `iswinrm`, `islicenced`
|
||||
- Examples: `isactive`, `isshopfloor`, `iscolor`, `isdhcp`, `islicenced`
|
||||
|
||||
### Code
|
||||
|
||||
@@ -77,9 +83,9 @@ To maintain consistency with the legacy ShopDB database and codebase, the follow
|
||||
### API
|
||||
|
||||
- **Endpoints:** Lowercase, plural nouns
|
||||
- Examples: `/api/machines`, `/api/pctypes`, `/api/locations`
|
||||
- Examples: `/api/machines`, `/api/computers`, `/api/locations`
|
||||
- **Query parameters:** Lowercase, single word
|
||||
- Examples: `?type=pc`, `?locationid=5`, `?isactive=true`
|
||||
- Examples: `?locationid=5`, `?isactive=true`, `?assettype=computer`
|
||||
|
||||
## Style Guidelines
|
||||
|
||||
@@ -92,7 +98,7 @@ To maintain consistency with the legacy ShopDB database and codebase, the follow
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- Python 3.12
|
||||
- Node.js 18+
|
||||
- MySQL 5.7+ (5.6 works with extra utf8mb4 config; see docs/DEPLOY.md)
|
||||
|
||||
@@ -158,12 +164,10 @@ npm run build # production build into frontend/dist (served by Flask)
|
||||
Complete first-run setup at `/setup`, or run `flask seed admin` for a headless
|
||||
admin account.
|
||||
|
||||
To import data from the legacy ShopDB MySQL database (one-time, see
|
||||
`migrations/DATA_MIGRATION_GUIDE.md`):
|
||||
|
||||
```bash
|
||||
python scripts/import_from_mysql.py
|
||||
```
|
||||
To import a site's legacy data, use the HTTP import surface: an admin API
|
||||
token plus [docs/IMPORT-API.md](docs/IMPORT-API.md) drive the whole migration
|
||||
through documented endpoints (`X-Import-Mode` preserves original timestamps).
|
||||
`scripts/site_imports/wjf/` is the West Jefferson reference loader.
|
||||
|
||||
For the full per-site deployment runbook see [docs/DEPLOY.md](docs/DEPLOY.md);
|
||||
for every environment variable and Setting key see [docs/CONFIG.md](docs/CONFIG.md).
|
||||
@@ -192,31 +196,36 @@ The REST API follows standard conventions:
|
||||
| PUT | `/api/machines/:id` | Update machine |
|
||||
| DELETE | `/api/machines/:id` | Soft delete machine |
|
||||
|
||||
Each asset plugin exposes the same CRUD pattern on its own prefix
|
||||
(`/api/computers`, `/api/printers`, `/api/network`, `/api/measuringtools`),
|
||||
and cross-cutting asset endpoints live under `/api/assets`.
|
||||
|
||||
Query parameters for list endpoints:
|
||||
- `page` - Page number (default: 1)
|
||||
- `per_page` - Items per page (default: 25)
|
||||
- `perpage` - Items per page
|
||||
- `sort` - Sort field
|
||||
- `order` - Sort direction (asc/desc)
|
||||
- `dir` - Sort direction (asc/desc)
|
||||
- `search` - Search term
|
||||
- `type` - Filter by asset type (computer, printer, machine, network_device)
|
||||
- `assettype` - Filter by asset type (computer, printer, machine, networkdevice, measuringtool)
|
||||
|
||||
## Plugin System
|
||||
|
||||
ShopDB supports plugins for extending functionality. See `CONTRIBUTING.md` for plugin development guidelines.
|
||||
|
||||
The image bundles eleven plugins; only the ones a site installs are loaded:
|
||||
The image bundles twelve plugins; only the ones a site installs are loaded:
|
||||
|
||||
- **computers** - Shopfloor PCs and workstations
|
||||
- **computers** - Shopfloor PCs and workstations, collector fleet ingest
|
||||
- **employees** - Employee directory
|
||||
- **geenforce** - GE-Enforce imaging/software manifests and fleet compliance
|
||||
- **machines** - CNC, CMM, and other shop-floor machines
|
||||
- **measuringtools** - Gage-lab instruments with calibration tracking
|
||||
- **knowledgebase** - Documentation and troubleshooting guides
|
||||
- **network** - Network devices
|
||||
- **network** - Network devices and subnets
|
||||
- **notifications** - Shopfloor notifications and recognition feed
|
||||
- **printers** - Extended printer management with Zabbix integration
|
||||
- **slides** - TV/kiosk slideshows
|
||||
- **usb** - CMMC USB check-in/out tracking
|
||||
- **warranty** - Dell warranty lookups
|
||||
- **warranty** - Warranty records with Dell lookups
|
||||
|
||||
## Legacy Migration
|
||||
|
||||
|
||||
@@ -1,321 +1,13 @@
|
||||
# ShopDB Flask Data Migration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes how to migrate data from the legacy `shopdb` database to the new `shopdb_flask` database schema.
|
||||
|
||||
## Database Configuration
|
||||
|
||||
**Development:**
|
||||
- Legacy database: `shopdb` (Classic ASP/VBScript schema)
|
||||
- New database: `shopdb_flask` (Flask/SQLAlchemy schema)
|
||||
- Connection: `mysql+pymysql://root:rootpassword@127.0.0.1:3306/shopdb_flask`
|
||||
|
||||
**Production:**
|
||||
- Follow the same migration steps on production MySQL server
|
||||
- Update connection string in `.env` accordingly
|
||||
|
||||
## Schema Differences
|
||||
|
||||
### Legacy Schema (shopdb)
|
||||
- `machines` table holds ALL assets (equipment, PCs, network devices)
|
||||
- `printers` table is separate
|
||||
- No unified asset abstraction
|
||||
|
||||
### New Schema (shopdb_flask)
|
||||
- `assets` table: Core asset data (shared fields)
|
||||
- `assettypes` table: Asset category registry
|
||||
- Plugin extension tables:
|
||||
- `equipment` - Manufacturing equipment details
|
||||
- `computers` - PC-specific fields
|
||||
- `networkdevices` - Network device details
|
||||
- `printers` - Printer-specific fields
|
||||
- Each extension links to `assets` via `assetid`
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Seed Reference Data
|
||||
|
||||
```bash
|
||||
cd /home/camp/projects/shopdb-flask
|
||||
source venv/bin/activate
|
||||
flask seed reference-data
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Asset types (equipment, computer, network_device, printer)
|
||||
- Asset statuses (In Use, Spare, Retired, etc.)
|
||||
- Machine types, operating systems, relationship types
|
||||
|
||||
### Step 2: Migrate Asset Types
|
||||
|
||||
```sql
|
||||
-- Insert asset types if not exists
|
||||
INSERT INTO assettypes (assettype, pluginname, tablename, description) VALUES
|
||||
('equipment', 'equipment', 'equipment', 'Manufacturing equipment'),
|
||||
('computer', 'computers', 'computers', 'PCs and workstations'),
|
||||
('network_device', 'network', 'networkdevices', 'Network infrastructure'),
|
||||
('printer', 'printers', 'printers', 'Printers and MFPs')
|
||||
ON DUPLICATE KEY UPDATE assettype=assettype;
|
||||
```
|
||||
|
||||
### Step 3: Migrate Equipment
|
||||
|
||||
```sql
|
||||
-- Migrate equipment from legacy machines table
|
||||
INSERT INTO assets (assetid, assetnumber, name, serialnumber, assettypeid, statusid,
|
||||
locationid, businessunitid, mapleft, maptop, notes,
|
||||
createddate, modifieddate, isactive)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.machinenumber,
|
||||
m.alias,
|
||||
m.serialnumber,
|
||||
(SELECT assettypeid FROM assettypes WHERE assettype = 'equipment'),
|
||||
m.statusid,
|
||||
m.locationid,
|
||||
m.businessunitid,
|
||||
m.mapleft,
|
||||
m.maptop,
|
||||
m.notes,
|
||||
m.createddate,
|
||||
m.modifieddate,
|
||||
m.isactive
|
||||
FROM shopdb.machines m
|
||||
JOIN shopdb.machinetypes mt ON m.machinetypeid = mt.machinetypeid
|
||||
WHERE mt.category = 'Equipment'
|
||||
AND m.pctypeid IS NULL;
|
||||
|
||||
-- Insert equipment extension data
|
||||
INSERT INTO equipment (assetid, equipmenttypeid, vendorid, modelnumberid,
|
||||
controllertypeid, controllervendorid, controllermodelid)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.machinetypeid,
|
||||
m.vendorid,
|
||||
m.modelnumberid,
|
||||
m.controllertypeid,
|
||||
m.controllervendorid,
|
||||
m.controllermodelid
|
||||
FROM shopdb.machines m
|
||||
JOIN shopdb.machinetypes mt ON m.machinetypeid = mt.machinetypeid
|
||||
WHERE mt.category = 'Equipment'
|
||||
AND m.pctypeid IS NULL;
|
||||
```
|
||||
|
||||
### Step 4: Migrate PCs/Computers
|
||||
|
||||
```sql
|
||||
-- Migrate PCs to assets table
|
||||
INSERT INTO assets (assetid, assetnumber, name, serialnumber, assettypeid, statusid,
|
||||
locationid, businessunitid, mapleft, maptop, notes,
|
||||
createddate, modifieddate, isactive)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.machinenumber,
|
||||
m.alias,
|
||||
m.serialnumber,
|
||||
(SELECT assettypeid FROM assettypes WHERE assettype = 'computer'),
|
||||
m.statusid,
|
||||
m.locationid,
|
||||
m.businessunitid,
|
||||
m.mapleft,
|
||||
m.maptop,
|
||||
m.notes,
|
||||
m.createddate,
|
||||
m.modifieddate,
|
||||
m.isactive
|
||||
FROM shopdb.machines m
|
||||
WHERE m.pctypeid IS NOT NULL;
|
||||
|
||||
-- Insert computer extension data
|
||||
INSERT INTO computers (assetid, computertypeid, hostname, osid,
|
||||
loggedinuser, lastreporteddate, lastboottime,
|
||||
isvnc, iswinrm, isshopfloor)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.pctypeid,
|
||||
m.hostname,
|
||||
m.osid,
|
||||
m.loggedinuser,
|
||||
m.lastreporteddate,
|
||||
m.lastboottime,
|
||||
m.isvnc,
|
||||
m.iswinrm,
|
||||
m.isshopfloor
|
||||
FROM shopdb.machines m
|
||||
WHERE m.pctypeid IS NOT NULL;
|
||||
```
|
||||
|
||||
### Step 5: Migrate Network Devices
|
||||
|
||||
```sql
|
||||
-- Migrate network devices to assets
|
||||
INSERT INTO assets (assetid, assetnumber, name, serialnumber, assettypeid, statusid,
|
||||
locationid, businessunitid, mapleft, maptop, notes,
|
||||
createddate, modifieddate, isactive)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.machinenumber,
|
||||
m.alias,
|
||||
m.serialnumber,
|
||||
(SELECT assettypeid FROM assettypes WHERE assettype = 'network_device'),
|
||||
m.statusid,
|
||||
m.locationid,
|
||||
m.businessunitid,
|
||||
m.mapleft,
|
||||
m.maptop,
|
||||
m.notes,
|
||||
m.createddate,
|
||||
m.modifieddate,
|
||||
m.isactive
|
||||
FROM shopdb.machines m
|
||||
JOIN shopdb.machinetypes mt ON m.machinetypeid = mt.machinetypeid
|
||||
WHERE mt.category = 'Network';
|
||||
|
||||
-- Insert network device extension data
|
||||
INSERT INTO networkdevices (assetid, networkdevicetypeid, hostname, vendorid, modelnumberid)
|
||||
SELECT
|
||||
m.machineid,
|
||||
m.machinetypeid,
|
||||
m.hostname,
|
||||
m.vendorid,
|
||||
m.modelnumberid
|
||||
FROM shopdb.machines m
|
||||
JOIN shopdb.machinetypes mt ON m.machinetypeid = mt.machinetypeid
|
||||
WHERE mt.category = 'Network';
|
||||
```
|
||||
|
||||
### Step 6: Migrate Printers
|
||||
|
||||
```sql
|
||||
-- Migrate printers to assets (printers are in separate table in legacy)
|
||||
INSERT INTO assets (assetnumber, name, serialnumber, assettypeid, statusid,
|
||||
locationid, businessunitid, notes, createddate, modifieddate, isactive)
|
||||
SELECT
|
||||
p.hostname,
|
||||
p.windowsname,
|
||||
NULL,
|
||||
(SELECT assettypeid FROM assettypes WHERE assettype = 'printer'),
|
||||
1, -- Default status
|
||||
p.locationid,
|
||||
p.businessunitid,
|
||||
NULL,
|
||||
p.createddate,
|
||||
p.modifieddate,
|
||||
p.isactive
|
||||
FROM shopdb.printers p;
|
||||
|
||||
-- Insert printer extension data (need to get the new assetid)
|
||||
INSERT INTO printers (assetid, printertypeid, vendorid, modelnumberid, hostname,
|
||||
windowsname, sharename, iscsf, installpath, pin,
|
||||
iscolor, isduplex, isnetwork)
|
||||
SELECT
|
||||
a.assetid,
|
||||
p.printertypeid,
|
||||
p.vendorid,
|
||||
p.modelnumberid,
|
||||
p.hostname,
|
||||
p.windowsname,
|
||||
p.sharename,
|
||||
p.iscsf,
|
||||
p.installpath,
|
||||
p.pin,
|
||||
p.iscolor,
|
||||
p.isduplex,
|
||||
p.isnetwork
|
||||
FROM shopdb.printers p
|
||||
JOIN assets a ON a.assetnumber = p.hostname
|
||||
WHERE a.assettypeid = (SELECT assettypeid FROM assettypes WHERE assettype = 'printer');
|
||||
```
|
||||
|
||||
### Step 7: Migrate Communications (IP Addresses)
|
||||
|
||||
```sql
|
||||
-- Migrate communications/IP addresses
|
||||
INSERT INTO communications (machineid, assetid, comtypeid, address,
|
||||
subnetid, isprimary, createddate, modifieddate, isactive)
|
||||
SELECT
|
||||
c.machineid,
|
||||
c.machineid, -- assetid = machineid for migrated assets
|
||||
c.comtypeid,
|
||||
c.address,
|
||||
c.subnetid,
|
||||
c.isprimary,
|
||||
c.createddate,
|
||||
c.modifieddate,
|
||||
c.isactive
|
||||
FROM shopdb.communications c;
|
||||
```
|
||||
|
||||
### Step 8: Migrate Notifications
|
||||
|
||||
```sql
|
||||
INSERT INTO notifications (notificationid, notificationtypeid, businessunitid, appid,
|
||||
notification, starttime, endtime, ticketnumber, link,
|
||||
isactive, isshopfloor, employeesso, employeename)
|
||||
SELECT * FROM shopdb.notifications;
|
||||
```
|
||||
|
||||
### Step 9: Migrate Supporting Tables
|
||||
|
||||
```sql
|
||||
-- Vendors
|
||||
INSERT INTO vendors SELECT * FROM shopdb.vendors
|
||||
ON DUPLICATE KEY UPDATE vendor=VALUES(vendor);
|
||||
|
||||
-- Models
|
||||
INSERT INTO models SELECT * FROM shopdb.models
|
||||
ON DUPLICATE KEY UPDATE modelnumber=VALUES(modelnumber);
|
||||
|
||||
-- Locations
|
||||
INSERT INTO locations SELECT * FROM shopdb.locations
|
||||
ON DUPLICATE KEY UPDATE locationname=VALUES(locationname);
|
||||
|
||||
-- Business Units
|
||||
INSERT INTO businessunits SELECT * FROM shopdb.businessunits
|
||||
ON DUPLICATE KEY UPDATE businessunit=VALUES(businessunit);
|
||||
|
||||
-- Subnets
|
||||
INSERT INTO subnets SELECT * FROM shopdb.subnets
|
||||
ON DUPLICATE KEY UPDATE subnet=VALUES(subnet);
|
||||
```
|
||||
|
||||
## Verification Queries
|
||||
|
||||
```sql
|
||||
-- Check migration counts
|
||||
SELECT 'Legacy machines' as source, COUNT(*) as cnt FROM shopdb.machines
|
||||
UNION ALL SELECT 'New assets', COUNT(*) FROM shopdb_flask.assets
|
||||
UNION ALL SELECT 'Equipment', COUNT(*) FROM shopdb_flask.equipment
|
||||
UNION ALL SELECT 'Computers', COUNT(*) FROM shopdb_flask.computers
|
||||
UNION ALL SELECT 'Network devices', COUNT(*) FROM shopdb_flask.networkdevices
|
||||
UNION ALL SELECT 'Printers', COUNT(*) FROM shopdb_flask.printers;
|
||||
|
||||
-- Verify asset type distribution
|
||||
SELECT at.assettype, COUNT(a.assetid) as count
|
||||
FROM shopdb_flask.assets a
|
||||
JOIN shopdb_flask.assettypes at ON a.assettypeid = at.assettypeid
|
||||
GROUP BY at.assettype;
|
||||
```
|
||||
|
||||
## Production Deployment Notes
|
||||
|
||||
1. **Backup production database first**
|
||||
2. Create `shopdb_flask` database on production
|
||||
3. Run `flask db-utils create-all` to create schema
|
||||
4. Execute migration SQL scripts in order
|
||||
5. Verify data counts match
|
||||
6. Update Flask `.env` to point to production database
|
||||
7. Restart Flask services
|
||||
|
||||
## Rollback
|
||||
|
||||
If migration fails:
|
||||
1. Drop all tables in `shopdb_flask`: `flask db-utils drop-all`
|
||||
2. Recreate schema: `flask db-utils create-all`
|
||||
3. Investigate and fix migration scripts
|
||||
4. Re-run migration
|
||||
|
||||
---
|
||||
Last updated: 2026-01-28
|
||||
# Legacy data migration (superseded)
|
||||
|
||||
The direct-database migration this guide used to describe is retired. Legacy
|
||||
data now enters through the HTTP import surface, which runs authorization,
|
||||
validation, auditing, and plugin hooks on every row:
|
||||
|
||||
- Contract: `docs/IMPORT-API.md` (admin API token + `X-Import-Mode: true`,
|
||||
which preserves original created/modified timestamps).
|
||||
- Adoption playbook for a new site: `docs/IMPORT-ADOPTION.md`.
|
||||
- Worked reference loader (West Jefferson classic-ASP schema, 15 stages,
|
||||
idempotent and resumable): `scripts/site_imports/wjf/`.
|
||||
- Full pilot sequence (stand up + import + verify + cutover):
|
||||
`docs/PILOT-DEPLOY.md`.
|
||||
|
||||
Reference in New Issue
Block a user