96a4a38c8c
Three defects broke the loader's own documented resume guarantee. All three were silent: nothing raised, the run finished cheerfully, and fewer rows arrived than anyone counted. THE PAGE SIZE WAS CLAMPED, SO THE 409-RESOLVE SEARCHED THE FIRST 100 ROWS _upsert asked for ?per_page=10000 and read one response. The server clamps any page size to MAX_PAGE_SIZE, which is 100 - not an error, not a warning, just 100 rows. Anything past that resolved to None, so the id never reached the crosswalk and the run quietly dropped a model's link on every machine that used it, or an application's installs. The target already holds more than 100 models and more than 100 applications, so this fired on any re-run or resume - exactly the case the loader documents as safe. _resolve_existing now tries the exact natural-key filter the API has exposed all along (?modelnumber= on models, ?appname= on applications, ?vendor= on vendors) and then pages properly, bounded. An endpoint without such a filter ignores the unknown parameter and the paged walk finds the row. stage_metrology repeated the same mistake in its own way: it skipped 409s entirely, so a resumed run left the tool out of the crosswalk and the Controls edge unmade - metrology PCs driving nothing. stage_printers already had the resolve branch; that stage was written later and missed it. THE CROSSWALK WAS NOT BOUND TO THE DATABASE IT DESCRIBED idmap.json maps this site's legacy ids to assetids in ONE database, and those assetids mean nothing anywhere else. Keeping it across a drop-and-rebuild left warranties and printer assignments attaching to whatever asset now holds that id: wrong data, no error. The runbook's two "drop and rebuild" steps never mentioned the file. The loader now writes a uuid marker into the TARGET's own settings table, so it dies with the data, and stamps it into the crosswalk. A crosswalk built against a different database is refused before a row is written. The database name alone would not do - a rebuild keeps the name. A file written before this guard is adopted rather than refused, so no existing crosswalk is stranded. Also adds --idmap, so one checkout can drive more than one target. THE SOURCE DATABASES ARE NAMED, NOT HARDCODED 'shopdb_src' appeared at 29 call sites, so aiming the loader at the prodscratch restore meant editing run.py. Those names are now LOGICAL and resolved in Source.db() from SOURCE_DB_MAIN / SOURCE_DB_USB / SOURCE_DB_EMPLOYEES, with host, port, user and password overridable too. The call sites are untouched - resolving centrally avoids missing one, and the literals now read as documentation of which source a stage draws from. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database". This is what stood between the loader and the prodscratch run. TWO SMALLER ONES _source_password() raised IndexError rather than its intended message whenever the DATABASE_URL user is not root, which it will not be on a real server. main() returned 1 on failure into a caller that discarded it. A run that logged 200 endpoint errors exited 0 and read as a success in a deploy log; it now exits non-zero. TESTS AND DOCS The loader had no tests at all. Sixteen now cover the resolve, the crosswalk binding and the source aliasing - the logic that can run without a MySQL source, which is the part that was wrong. Verified they are not vacuous by running the old algorithm against the same fixture: it returns None where the new one returns the row. The README claimed 15 stages "complete" with counts from 2026-07-13 that two later routing changes invalidated, and the runbook told operators to verify a fresh import against them. Both corrected, with the three gaps NOT fixed here recorded plainly: warranties, notifications and knowledgebase POST without a duplicate check so resuming across them doubles rows; there is no --dry-run or target guard; and the verify stage counts target assets against source machines, which is apples to oranges. Full suite 2178 passed.
ShopDB Flask
A modern rewrite of the classic ASP/VBScript ShopDB application using Flask (Python) and Vue 3. This application manages shop floor machines, PCs, printers, applications, and related infrastructure for manufacturing environments.
Overview
ShopDB tracks and manages:
- Machines - CNC equipment, CMMs, inspection systems, etc.
- PCs - Shopfloor computers, engineering workstations
- Printers - Network printers with Zabbix supply integration
- Network devices - Switches, routers, and the subnet browser
- Measuring tools - Gage-lab instruments with calibration tracking
- Applications - Software deployed across the shop floor, with per-PC install tracking
- Employees - Directory, recognition and training notifications
- Warranties - Coverage records with Dell warranty lookups
- USB devices - CMMC check-in/out tracking
- Knowledge Base - Documentation and troubleshooting guides
- GE-Enforce manifests - Imaging/software manifest editing and fleet compliance
Tech Stack
Backend:
- Python 3.14 with Flask
- SQLAlchemy ORM
- MySQL 5.7+ database (5.6 works with extra utf8mb4 config; see the DEPLOY wiki page)
- JWT authentication
- Plugin architecture for extensibility
Frontend:
- Vue 3 with Composition API
- Vue Router for navigation
- Pinia for state management
- Vite build system
Project Structure
shopdb-flask/
+-- shopdb/ # Flask application
| +-- core/
| | +-- api/ # REST API endpoints
| | +-- models/ # SQLAlchemy models
| | +-- schemas/ # Validation schemas
| | `-- services/ # Business logic
| +-- plugins/ # Plugin system
| `-- utils/ # Shared utilities
+-- frontend/ # Vue 3 application
| +-- src/
| | +-- api/ # API client
| | +-- components/ # Reusable components
| | +-- views/ # Page components
| | +-- router/ # Route definitions
| | `-- stores/ # Pinia stores
| `-- public/ # Static assets
+-- plugins/ # Bundled and external plugins
+-- migrations/ # Alembic migration chain (flask db upgrade)
+-- scripts/ # Import and utility scripts
`-- tests/ # Test suite
Naming Conventions
To maintain consistency with the legacy ShopDB database and codebase, the following naming standards apply:
Database
- Table names: Lowercase, single word, no underscores or dashes
- Examples:
assets,computers,printers,businessunits
- Examples:
- Column names: Lowercase, single word, no underscores or dashes
- Examples:
assetid,assetnumber,hostname,isactive,createddate
- Examples:
- Foreign keys: Referenced table name +
id- Examples:
locationid,vendorid,modelnumberid,computertypeid
- Examples:
- Boolean columns: Prefixed with
isorhas- Examples:
isactive,isshopfloor,iscolor,isdhcp,islicenced
- Examples:
Code
- Python variables: Follow database naming where applicable (lowercase, no underscores for model fields)
- JavaScript variables: camelCase for local variables, but match API field names from backend
- Vue components: PascalCase for component names
- CSS classes: Lowercase with dashes for multi-word classes
API
- Endpoints: Lowercase, plural nouns
- Examples:
/api/machines,/api/computers,/api/locations
- Examples:
- Query parameters: Lowercase, single word
- Examples:
?locationid=5,?isactive=true,?assettype=computer
- Examples:
Style Guidelines
- No emojis in code, comments, documentation, or UI
- Keep UI functional and professional
- Dark theme is the default
- Consistent table layouts across all list views
Setup
Prerequisites
- Python 3.14
- Node.js 18+
- MySQL 5.7+ (5.6 works with extra utf8mb4 config; see the DEPLOY wiki page)
ShopDB Flask uses MySQL as the canonical database. SQLite is used only for the
test suite (TestingConfig in shopdb/config.py points at an in-memory
SQLite). Do not run dev or production against SQLite.
Distribution
Source lives here. The Windows installer ships as a release asset rather than a
file in the repository - it is around 240 MB, well past what a repository takes
and well inside what a release asset does. There is no package or image
registry; a site installs from that .exe or builds from source.
Fast path (Docker)
The Docker image builds the Vue frontend and serves it from the API container, so a container deploy needs no separate Node build step.
cp .env.example .env
# Edit .env: set SECRET_KEY, JWT_SECRET_KEY, DATABASE_URL, CORS_ORIGINS,
# and the MYSQL_* passwords (see the CONFIG wiki page for every variable).
docker compose up -d --build
# Create the schema and seed the platform data (idempotent, safe to re-run):
docker compose exec api flask db upgrade
docker compose exec api flask seed permissions
docker compose exec api flask seed settings
docker compose exec api flask seed reference-data
Then browse to the site and complete the first-run setup wizard at /setup
(it creates the first admin account and captures site identity). To create the
admin headlessly instead of using the wizard:
docker compose exec api flask seed admin --username admin --email admin@facility.example.com
Manual path (venv + Node)
# Backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your database credentials and secrets.
export FLASK_APP=shopdb
flask db upgrade
flask plugin upgrade-all # per-plugin schema (ADR-008)
flask seed permissions
flask seed settings
flask seed reference-data
flask run --port 5001 # MUST be 5001 - the frontend dev server proxies here
# Frontend (separate terminal)
cd frontend
npm install
npm run dev # dev server on :5173
npm run build # production build into frontend/dist (served by Flask)
Complete first-run setup at /setup, or run flask seed admin for a headless
admin account. The repo ships VS Code config in .vscode/ (F5 debugs the
backend; a "Dev site" task runs both servers). A fuller day-one walkthrough,
including VS Code and troubleshooting, is the DEVELOPMENT-SETUP page in the
project wiki.
To import a site's legacy data, use the HTTP import surface: an admin API
token plus the IMPORT-API page in the project wiki drive the whole migration
through documented endpoints (X-Import-Mode preserves original timestamps).
scripts/site_imports/wjf/ is the reference site's loader, kept as a worked
example of the whole sequence.
Which deployment route
| Target | Use |
|---|---|
| Windows Server + IIS (how sister sites run) | the INSTALL-WINDOWS page in the project wiki - one installer .exe, offline, no manual IIS work. Day 2: the OPERATE-WINDOWS page in the project wiki |
| Linux / Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki |
| Linux / Docker, connected | the DEPLOY page in the project wiki |
For every environment variable and Setting key see the CONFIG page in the project wiki.
What the Windows installer requires
The installer checks all of this before it changes anything and refuses rather than half-installing.
| Minimum | Notes | |
|---|---|---|
| Windows Server | 2019 (build 10.0.17763) | Ships IIS 10.0, which is therefore the earliest IIS supported. Server 2016 is also IIS 10.0 but falls below the build floor and is refused. |
| Windows client (test boxes) | 10 22H2 (build 10.0.19045) or 11 | Pro or higher - Home has no IIS at all. |
| Architecture | 64-bit | The payload is cp314 win_amd64. |
| IIS Web Server role | installed beforehand | Install-WindowsFeature -Name Web-Server -IncludeManagementTools. Needs no internet. |
| Free disk | 5 GB | Refused below this. 40 GB is comfortable once uploads and config backups accumulate. |
| Database | bundled MySQL 8.4 LTS, or your own | Bring credentials if you use an existing server. |
The binding constraint is the operating system rather than IIS: the payload is Python 3.14 plus HttpPlatformHandler and URL Rewrite, and the handler itself runs on older IIS. An older host is untested rather than known-broken.
If you must deploy onto something older, the INSTALL-WINDOWS-IIS page in the project wiki is the manual procedure and has no OS gate - it even documents MySQL 5.6, which is that era of machine. The trade is stated there: it produces a server the installer will not subsequently upgrade.
Configuration
Environment variables (.env):
| Variable | Description |
|---|---|
DATABASE_URL |
MySQL connection string |
SECRET_KEY |
Flask secret key |
JWT_SECRET_KEY |
JWT signing key |
JWT_ACCESS_TOKEN_EXPIRES |
Access token TTL (seconds) |
LOG_LEVEL |
Logging verbosity |
API Documentation
The REST API follows standard conventions:
| Method | Endpoint | Description |
|---|---|---|
| GET | /api/machines |
List machines (filterable by type) |
| GET | /api/machines/:id |
Get machine details |
| POST | /api/machines |
Create machine |
| PUT | /api/machines/:id |
Update machine |
| DELETE | /api/machines/:id |
Soft delete machine |
Each asset plugin exposes the same CRUD pattern on its own prefix
(/api/computers, /api/printers, /api/network, /api/measuringtools),
and cross-cutting asset endpoints live under /api/assets.
Query parameters for list endpoints:
page- Page number (default: 1)perpage- Items per pagesort- Sort fielddir- Sort direction (asc/desc)search- Search termassettype- Filter by asset type (computer, printer, machine, networkdevice, measuringtool)
Where to start
the START-HERE page in the project wiki routes by what you are here to do - standing up a site, deploying the shop-floor tools, writing a plugin, integrating with the API, or diagnosing something. On the published mirror the same page is the wiki's front door.
Plugin System
ShopDB supports plugins for extending functionality. See CONTRIBUTING.md for plugin development guidelines.
The image bundles these plugins; only the ones a site installs are loaded. The current count and each plugin's version and migration head are in the PROJECT-MAP page in the project wiki, which is generated:
- backups - Machine configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
- computers - Shopfloor PCs and workstations, collector fleet ingest
- employees - Employee directory
- geenforce - GE-Enforce imaging/software manifests and fleet compliance
- machines - CNC, CMM, and other shop-floor machines
- measuringtools - Gage-lab instruments with calibration tracking
- knowledgebase - Documentation and troubleshooting guides
- network - Network devices and subnets
- notifications - Shopfloor notifications and recognition feed
- printedparts - 3D-printed part catalogue, kiosk issue tracking and stock alerts
- printers - Extended printer management with Zabbix integration
- slides - TV/kiosk slideshows
- tools - Tech Tools: barcode and QR generation laid out for real label stock
- usb - CMMC USB check-in/out tracking
- warranty - Warranty records with Dell lookups
Legacy Migration
This project replicates functionality from the classic ASP/VBScript ShopDB site. Key mappings:
| Legacy | Modern |
|---|---|
| ASP/VBScript | Flask/Python |
| Classic ADO | SQLAlchemy |
| Server-side HTML | Vue 3 SPA |
| Session auth | JWT tokens |
License
Internal use only.