Files

cproudlock 784964c907 Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import

Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.

What arrives:

  endpoints plugin  - a ShopDB agent reporting endpoint state, telemetry,
                      services and interfaces, with a command channel the agent
                      long-polls. Ingest stays on the generic collector route
                      (ADR-006); the agent key is a UI-managed single-scope
                      token whose revocation is an instant kill switch.
  computers policies - Configuration and Business Policies (RMM-style), scope
                      trees validated at write time, actions checked against
                      the endpoints command allowlist. Feature-gated: every
                      route 404s while the setting is off, so enabling it can
                      never half-expose the surface.
  asset CSV import  - templates, a collector script carrying the site path, and
                      a validate-then-commit import. A plugin can own a virtual
                      column plus a post-write row hook, which is how a PC's
                      machine number becomes a controls link without a
                      hand-built relationships sheet.
  agent/            - the Windows agent and its installer.

Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.

MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).

Review fixes carried on top:

  - six policy GETs were jwt_required(optional=True), which authenticates a
    caller without checking anything; they require computers.view now, and six
    redundant jwt_required decorators are gone
  - the agent installer took a default BaseUrl and CollectorKey, so a mistyped
    run silently pointed an endpoint at the wrong instance; both are mandatory
    and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
  - a committing CSV import wrote no audit row

Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.

Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.

2026-08-28 16:17:17 -04:00

11 KiB

Raw Blame History

ShopDB Flask

A modern rewrite of the classic ASP/VBScript ShopDB application using Flask (Python) and Vue 3. This application manages shop floor machines, PCs, printers, applications, and related infrastructure for manufacturing environments.

Overview

ShopDB tracks and manages:

Machines - CNC equipment, CMMs, inspection systems, etc.
PCs - Shopfloor computers, engineering workstations
Printers - Network printers with Zabbix supply integration
Network devices - Switches, routers, and the subnet browser
Measuring tools - Gage-lab instruments with calibration tracking
Applications - Software deployed across the shop floor, with per-PC install tracking
Employees - Directory, recognition and training notifications
Warranties - Coverage records with Dell warranty lookups
USB devices - CMMC check-in/out tracking
Knowledge Base - Documentation and troubleshooting guides
GE-Enforce manifests - Imaging/software manifest editing and fleet compliance

Tech Stack

Backend:

Python 3.14 with Flask
SQLAlchemy ORM
MySQL 5.7+ database (5.6 works with extra utf8mb4 config; see the DEPLOY wiki page)
JWT authentication
Plugin architecture for extensibility

Frontend:

Vue 3 with Composition API
Vue Router for navigation
Pinia for state management
Vite build system

Project Structure

shopdb-flask/
+-- shopdb/                 # Flask application
|   +-- core/
|   |   +-- api/            # REST API endpoints
|   |   +-- models/         # SQLAlchemy models
|   |   +-- schemas/        # Validation schemas
|   |   `-- services/       # Business logic
|   +-- plugins/            # Plugin system
|   `-- utils/              # Shared utilities
+-- frontend/               # Vue 3 application
|   +-- src/
|   |   +-- api/            # API client
|   |   +-- components/     # Reusable components
|   |   +-- views/          # Page components
|   |   +-- router/         # Route definitions
|   |   `-- stores/         # Pinia stores
|   `-- public/             # Static assets
+-- plugins/                # Bundled and external plugins
+-- migrations/             # Alembic migration chain (flask db upgrade)
+-- scripts/                # Import and utility scripts
`-- tests/                  # Test suite

Naming Conventions

To maintain consistency with the legacy ShopDB database and codebase, the following naming standards apply:

Database

Table names: Lowercase, single word, no underscores or dashes
- Examples: assets, computers, printers, businessunits
Column names: Lowercase, single word, no underscores or dashes
- Examples: assetid, assetnumber, hostname, isactive, createddate
Foreign keys: Referenced table name + id
- Examples: locationid, vendorid, modelnumberid, computertypeid
Boolean columns: Prefixed with is or has
- Examples: isactive, isshopfloor, iscolor, isdhcp, islicenced

Code

Python variables: Follow database naming where applicable (lowercase, no underscores for model fields)
JavaScript variables: camelCase for local variables, but match API field names from backend
Vue components: PascalCase for component names
CSS classes: Lowercase with dashes for multi-word classes

API

Endpoints: Lowercase, plural nouns
- Examples: /api/machines, /api/computers, /api/locations
Query parameters: Lowercase, single word
- Examples: ?locationid=5, ?isactive=true, ?assettype=computer

Style Guidelines

No emojis in code, comments, documentation, or UI
Keep UI functional and professional
Dark theme is the default
Consistent table layouts across all list views

Setup

Prerequisites

Python 3.14
Node.js 18+
MySQL 5.7+ (5.6 works with extra utf8mb4 config; see the DEPLOY wiki page)

ShopDB Flask uses MySQL as the canonical database. SQLite is used only for the test suite (TestingConfig in shopdb/config.py points at an in-memory SQLite). Do not run dev or production against SQLite.

Distribution

Source lives here. The Windows installer ships as a release asset rather than a file in the repository - it is around 240 MB, well past what a repository takes and well inside what a release asset does. There is no package or image registry; a site installs from that .exe or builds from source.

Fast path (Docker)

The Docker image builds the Vue frontend and serves it from the API container, so a container deploy needs no separate Node build step.

cp .env.example .env
# Edit .env: set SECRET_KEY, JWT_SECRET_KEY, DATABASE_URL, CORS_ORIGINS,
# and the MYSQL_* passwords (see the CONFIG wiki page for every variable).

docker compose up -d --build

# Create the schema and seed the platform data (idempotent, safe to re-run):
docker compose exec api flask db upgrade
docker compose exec api flask seed permissions
docker compose exec api flask seed settings
docker compose exec api flask seed reference-data

Then browse to the site and complete the first-run setup wizard at /setup (it creates the first admin account and captures site identity). To create the admin headlessly instead of using the wizard:

docker compose exec api flask seed admin --username admin --email admin@facility.example.com

Manual path (venv + Node)

# Backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your database credentials and secrets.

export FLASK_APP=shopdb
flask db upgrade
flask plugin upgrade-all      # per-plugin schema (ADR-008)
flask seed permissions
flask seed settings
flask seed reference-data
flask run --port 5001         # MUST be 5001 - the frontend dev server proxies here

# Frontend (separate terminal)
cd frontend
npm install
npm run dev          # dev server on :5173
npm run build        # production build into frontend/dist (served by Flask)

Complete first-run setup at /setup, or run flask seed admin for a headless admin account. The repo ships VS Code config in .vscode/ (F5 debugs the backend; a "Dev site" task runs both servers). A fuller day-one walkthrough, including VS Code and troubleshooting, is the DEVELOPMENT-SETUP page in the project wiki.

To import a site's legacy data, use the HTTP import surface: an admin API token plus the IMPORT-API page in the project wiki drive the whole migration through documented endpoints (X-Import-Mode preserves original timestamps). scripts/site_imports/wjf/ is the reference site's loader, kept as a worked example of the whole sequence.

Which deployment route

Target	Use
Windows Server + IIS (how sister sites run)	the INSTALL-WINDOWS page in the project wiki - one installer `.exe`, offline, no manual IIS work. Day 2: the OPERATE-WINDOWS page in the project wiki
Linux / Docker, air-gapped	the DEPLOY-AIRGAP page in the project wiki
Linux / Docker, connected	the DEPLOY page in the project wiki

For every environment variable and Setting key see the CONFIG page in the project wiki.

What the Windows installer requires

The installer checks all of this before it changes anything and refuses rather than half-installing.

	Minimum	Notes
Windows Server	2019 (build 10.0.17763)	Ships IIS 10.0, which is therefore the earliest IIS supported. Server 2016 is also IIS 10.0 but falls below the build floor and is refused.
Windows client (test boxes)	10 22H2 (build 10.0.19045) or 11	Pro or higher - Home has no IIS at all.
Architecture	64-bit	The payload is cp314 win_amd64.
IIS Web Server role	installed beforehand	`Install-WindowsFeature -Name Web-Server -IncludeManagementTools`. Needs no internet.
Free disk	5 GB	Refused below this. 40 GB is comfortable once uploads and config backups accumulate.
Database	bundled MySQL 8.4 LTS, or your own	Bring credentials if you use an existing server.

The binding constraint is the operating system rather than IIS: the payload is Python 3.14 plus HttpPlatformHandler and URL Rewrite, and the handler itself runs on older IIS. An older host is untested rather than known-broken.

If you must deploy onto something older, the INSTALL-WINDOWS-IIS page in the project wiki is the manual procedure and has no OS gate - it even documents MySQL 5.6, which is that era of machine. The trade is stated there: it produces a server the installer will not subsequently upgrade.

Configuration

Environment variables (.env):

Variable	Description
`DATABASE_URL`	MySQL connection string
`SECRET_KEY`	Flask secret key
`JWT_SECRET_KEY`	JWT signing key
`JWT_ACCESS_TOKEN_EXPIRES`	Access token TTL (seconds)
`LOG_LEVEL`	Logging verbosity

API Documentation

The REST API follows standard conventions:

Method	Endpoint	Description
GET	`/api/machines`	List machines (filterable by type)
GET	`/api/machines/:id`	Get machine details
POST	`/api/machines`	Create machine
PUT	`/api/machines/:id`	Update machine
DELETE	`/api/machines/:id`	Soft delete machine

Each asset plugin exposes the same CRUD pattern on its own prefix (/api/computers, /api/printers, /api/network, /api/measuringtools), and cross-cutting asset endpoints live under /api/assets.

Query parameters for list endpoints:

page - Page number (default: 1)
perpage - Items per page
sort - Sort field
dir - Sort direction (asc/desc)
search - Search term
assettype - Filter by asset type (computer, printer, machine, networkdevice, measuringtool)

Where to start

the START-HERE page in the project wiki routes by what you are here to do - standing up a site, deploying the shop-floor tools, writing a plugin, integrating with the API, or diagnosing something. On the published mirror the same page is the wiki's front door.

Plugin System

ShopDB supports plugins for extending functionality. See CONTRIBUTING.md for plugin development guidelines.

The image bundles these plugins; only the ones a site installs are loaded. The current count and each plugin's version and migration head are in the PROJECT-MAP page in the project wiki, which is generated:

backups - Machine configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
computers - Shopfloor PCs and workstations, collector fleet ingest
employees - Employee directory
geenforce - GE-Enforce imaging/software manifests and fleet compliance
machines - CNC, CMM, and other shop-floor machines
measuringtools - Gage-lab instruments with calibration tracking
knowledgebase - Documentation and troubleshooting guides
network - Network devices and subnets
notifications - Shopfloor notifications and recognition feed
printedparts - 3D-printed part catalogue, kiosk issue tracking and stock alerts
printers - Extended printer management with Zabbix integration
slides - TV/kiosk slideshows
tools - Tech Tools: barcode and QR generation laid out for real label stock
usb - CMMC USB check-in/out tracking
warranty - Warranty records with Dell lookups

Legacy Migration

This project replicates functionality from the classic ASP/VBScript ShopDB site. Key mappings:

Legacy	Modern
ASP/VBScript	Flask/Python
Classic ADO	SQLAlchemy
Server-side HTML	Vue 3 SPA
Session auth	JWT tokens

License

Internal use only.

11 KiB Raw Blame History