ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/wj' into wilm

Browse Source
This commit is contained in:
benjamin.baker
2026-09-07 05:17:24 -04:00
parent 04616fce8b d40f1a6a92
commit 879451b689
52 changed files with 4937 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files

116
.claude/INDEX.md Normal file
View File

@@ -0,0 +1,116 @@
# ShopDB Index - Load Only What You Need
Quick pointers to avoid reading entire wiki or codebase.
## When Working On...
**Plugin development**:
- Read: `CLAUDE.md` (overview)
- Grep: `class BasePlugin` in `shopdb/api/__init__.py`
- Reference: `../shopdb-flask.wiki/ADR-001-asset-as-platform-contract.md`
**Database migrations**:
- Read: `../shopdb-flask.wiki/ADR-008-plugin-migration-ownership.md`
- Grep: `PLUGIN_TABLE_OWNERS` for dependency order
- Pattern: `plugins/{name}/migrations/versions/*.py`
**Site-specific config**:
- Read: `../shopdb-flask.wiki/ADR-015-site-specific-configuration.md`
- Locations: `shopdb/core/settings/`, `shopdb/core/seed/`, `scripts/site_imports/`
**Kiosk/device auth**:
- Read: `../shopdb-flask.wiki/ADR-016-credential-delivery.md`
- Grep: `requires_device_token`
**Deployment/branches**:
- Read: `../shopdb-flask.wiki/ADR-004-deployment-topology.md`
- Scripts: `~/Documents/github/*.cmd`
**Frontend plugin panels**:
- Read: `../shopdb-flask.wiki/ADR-010-frontend-plugin-hooks.md`
- Pattern: `plugins/{name}/frontend/views/*.vue`
**Testing**:
- Read: `tests/conftest.py` for fixtures
- Pattern: `tests/test_plugins/test_{plugin}_{feature}.py`
## Plugin Map (20 plugins)
Core asset types:
- `computers`, `endpoints`, `printers`, `machines`, `measuringtools`, `network`, `tools`
Workflows:
- `printedparts`, `usb`, `slides`, `notifications`
Integrations:
- `warranty`, `geenforce`, `guides`, `knowledgebase`
Admin:
- `employees`, `backups`
## Key Files
- `shopdb/api/__init__.py` - Stable import surface (75 lines)
- `shopdb/config.py` - App configuration
- `shopdb/core/models/asset.py` - Base asset model
- `docker-compose.yml` - Dev stack definition
- `scripts/check-naming-and-style.sh` - Pre-commit hook
## Git Branches
**Current repo state**:
- `wj` (checked out) - Prod branch for WJ shop floor
- `main` - Public GitHub mirror, deploys nowhere
- `origin/wilm` - Upstream changes from benjamin.baker (86 commits ahead)
- `origin/dev-be` - Historical dev branch
**Branch workflow** (ADR-004):
- `wj` -> prod deployment (WJ site)
- `dev` -> ops deployment (monitoring/pilot sites, on gitea only)
- `main` -> GitHub public mirror only (no deployments)
- `wilm` -> fetch from GitHub, review, merge to wj or dev
**On this work PC**:
- Has GitHub credentials (ge-aero/shopdb-flask)
- Fetch wilm: `~/Documents/github/fetch-wilm-branch.cmd`
- Bundle to dev PC: `~/Documents/github/pull-shopdb-bundle.cmd`
## Common Paths
- Plugin code: `plugins/{name}/`
- Core models: `shopdb/core/models/`
- Core API: `shopdb/core/api/`
- Frontend core: `frontend/src/`
- Tests: `tests/test_plugins/`
- Migrations: `migrations/` (core) + `plugins/{name}/migrations/`
- Scripts: `scripts/`
- Docs: `../shopdb-flask.wiki/`
## Avoid Reading These Unless Needed
- `docs/openapi.json` (489 lines, generated)
- `frontend/dist/` (build output)
- `node_modules/`, `venv/`, `.pytest_cache/`
- `agent/` (C# endpoint agent, 10k+ lines)
- Full wiki (68 files) - use index above
## Token-Efficient Patterns
1. **Grep before Read**: Find the exact file/function first
2. **Read with limit**: Use offset/limit for large files
3. **Reference by path**: Cite `file:line` instead of quoting code
4. **Use memory**: Store repeated patterns in memory files
5. **Wiki on-demand**: Only read specific ADRs when relevant
## One-Line ADR Summaries
- ADR-001: Asset is the platform primitive for all equipment
- ADR-004: Three air-gapped environments, bundle workflow
- ADR-006: Endpoint agent sends telemetry, receives commands
- ADR-007: Semantic versioning, GitHub releases
- ADR-008: Each plugin owns its migration chain, no conflicts
- ADR-009: Frontend plugins gate on backend features
- ADR-010: Plugins register Vue components via manifest
- ADR-013: Lean builds include only needed plugins
- ADR-015: Site literals go in settings/seed/site_imports only
- ADR-016: Kiosks use device tokens, not user sessions

152
.claude/QUICK-REFERENCE.md Normal file
View File

@@ -0,0 +1,152 @@
# ShopDB Quick Reference
Fast lookup for architectural decisions and contracts.
## Critical ADRs (in wiki)
- **ADR-001**: Asset as platform contract - Core asset model, BaseAsset, composition
- **ADR-004**: Deployment topology - Air-gapped sites, bundle workflow
- **ADR-006**: Collector contract - Endpoint agent protocol
- **ADR-007**: Product versioning - Semantic versioning, release process
- **ADR-008**: Plugin migration ownership - PLUGIN_TABLE_OWNERS, migration chains
- **ADR-013**: Plugin catalog - Lean builds, plugin manifest
- **ADR-015**: Site-specific configuration - Where site literals belong
- **ADR-016**: Credential delivery - Device tokens, kiosk auth
## Import Surface (stable API)
```python
from shopdb.api import (
# Database
db,
# Core Models
Asset, Employee, User, Building, MapLevel,
# Plugin Base
BasePlugin,
# Auth/Authz
requires_role, requires_device_token,
# Utilities
badges, datevalues, queryargs
)
```
## Plugin Structure
```
plugins/{name}/
|-- plugin.py # BasePlugin subclass, manifest
|-- models/ # SQLAlchemy models
| +-- {name}.py
|-- api/ # Flask routes
| +-- routes.py
|-- frontend/ # Vue components
| +-- views/
| |-- {Name}List.vue
| |-- {Name}Detail.vue
| +-- {Name}Form.vue
|-- migrations/ # Alembic migrations
| +-- versions/
+-- services/ # Business logic (optional)
```
## Database Naming
- **Tables**: `printedpart`, `mappeddrives` (lowercase concatenated)
- **Columns**: `serial_number`, `last_seen` (snake_case)
- **Enforced**: `scripts/check-naming-and-style.sh`
## Branch Model
- `wj` -> prod (WJ shop floor)
- `dev` -> ops (monitoring/pilot)
- `main` -> public mirror (GitHub only)
- `wilm` -> upstream (benjamin.baker's changes)
## Common Commands
```bash
# Run tests
pytest -v
# Check style
scripts/check-naming-and-style.sh
# Create migration
cd plugins/{name}
alembic revision -m "description"
# Dev server
docker compose up
# Bundle for dev PC
# (on work PC) ~/Documents/github/pull-shopdb-bundle.cmd
# Fetch wilm changes
# (on work PC) ~/Documents/github/fetch-wilm-branch.cmd
```
## Test Patterns
```python
def test_feature(client, db_session):
# Setup
item = Model(name="test")
db_session.add(item)
db_session.commit()
# Execute
resp = client.get('/api/plugin/endpoint')
# Assert
assert resp.status_code == 200
assert resp.json['name'] == 'test'
```
## Site Data Rules (ADR-015)
[OK] **Allowed**:
- `shopdb/core/settings/buildings.py`
- `shopdb/core/seed/*.sql`
- `scripts/site_imports/*.py`
[FAIL] **Forbidden**:
- Hardcoded in plugin code
- Embedded in migrations
- Configuration files
## Migration Ownership (ADR-008)
Each plugin owns its chain. Coordinate cross-plugin FK changes via PLUGIN_TABLE_OWNERS:
```python
PLUGIN_TABLE_OWNERS = {
'asset': 'core',
'employee': 'employees',
'pc': 'computers',
'endpoint': 'endpoints',
# ...
}
```
## Device Tokens (ADR-016)
Kiosks require device authentication:
```python
from shopdb.api import requires_device_token
@bp.route('/kiosk/endpoint')
@requires_device_token(['usb_kiosk', 'parts_kiosk'])
def kiosk_view():
return render_template('kiosk.html')
```
## Wiki Location
Full docs: `~/Documents/github/shopdb-flask.wiki/`
All ADRs, deployment guides, plugin development docs live there.

27
.claude/settings.json Normal file
View File

@@ -0,0 +1,27 @@
{
"permissions": {
"allow": [
"bash:pytest*",
"bash:git status",
"bash:git log*",
"bash:git diff*",
"bash:git branch*",
"bash:docker compose*",
"bash:docker ps",
"bash:alembic*",
"bash:python scripts/*",
"grep:**",
"glob:**"
]
},
"hooks": {
"beforeCommit": [
"scripts/check-naming-and-style.sh"
]
},
"context": {
"alwaysInclude": [
".claude/INDEX.md"
]
}
}

13
.dockerignore
View File

@@ -1,23 +1,22 @@
# Keep the build context lean and prevent host artifacts from clobbering # Dockerignore for pre-built frontend approach
# in-image installs. node_modules is platform-specific: copying the host's # frontend/dist is INCLUDED (not ignored) for this build
# (Windows) tree over the container's `npm ci` breaks rollup's native binary.
**/node_modules **/node_modules
frontend/dist
frontend/src/.plugins-staged frontend/src/.plugins-staged
frontend/src/router/routes.gen.js frontend/src/router/routes.gen.js
# Python caches and virtualenvs # Python caches
**/__pycache__ **/__pycache__
**/*.pyc **/*.pyc
.venv .venv
venv venv
# Local dev and VCS noise # Local dev and VCS
.git .git
.gitignore .gitignore
*.log *.log
.env.local .env.local
# Installer build artifacts (not part of the app image) # Installer artifacts
installer-arch-dev/build installer-arch-dev/build
installer-arch-dev/dist installer-arch-dev/dist

22
.dockerignore.prebuilt Normal file
View File

@@ -0,0 +1,22 @@
# Dockerignore for pre-built frontend approach
# frontend/dist is INCLUDED (not ignored) for this build
**/node_modules
frontend/src/.plugins-staged
frontend/src/router/routes.gen.js
# Python caches
**/__pycache__
**/*.pyc
.venv
venv
# Local dev and VCS
.git
.gitignore
*.log
.env.local
# Installer artifacts
installer-arch-dev/build
installer-arch-dev/dist

2
.github/copilot-instructions.md vendored
View File

@@ -11,7 +11,7 @@ by CI (`.github/workflows/ci.yml`) - suggestions that break them fail the build.
- Avoid using `powershell.exe -File` for launching scripts, as it fails if the script file is missing or quarantined. - Avoid using `powershell.exe -File` for launching scripts, as it fails if the script file is missing or quarantined.
## Installer Logic ## Installer Logic
- The endpoint agent installer shipped to endpoints is generated by `_install_script()` in `plugins/endpoints/api/siteagent.py`, NOT `agent/installer/install-shopdbagent.ps1`. Any installer logic change must be made in `siteagent.py`, followed by rolling the containers. - The endpoint agent installer shipped to endpoints is generated by `_install_script()` in `plugins/endpoints/api/siteagent.py`, NOT `agent/installer/install-shopdbagent.ps1`. Any installer logic change must be made in `siteagent.py`, followed by rolling the containers.
- The user's workflow: the agent hands over only the built EXEs (`shopdbagent.exe`, `shopdb-endpoint-tray.exe`); the user uploads them via the "Upload New/Legacy Agents" field in the Build Site Agent UI and builds the package with the newest version selected at the top. - The user's workflow: the agent hands over only the built EXEs (`shopdbagent.exe`, `shopdb-endpoint-tray.exe`); the user uploads them via the "Upload New/Legacy Agents" field in the Build Site Agent UI and builds the package with the newest version selected at the top.
## Naming (LOCKED - the hook rejects violations) ## Naming (LOCKED - the hook rejects violations)

129
CLAUDE.md Normal file
View File

@@ -0,0 +1,129 @@
# ShopDB Flask - Claude Code Context
ShopDB is a manufacturing floor asset management system built with Flask, Vue 3, and PostgreSQL, deployed across air-gapped GE Aerospace sites.
## Architecture
- **Backend**: Flask 3.x with SQLAlchemy 2.x, plugin-based architecture
- **Frontend**: Vue 3 with Vite, component-based UI with plugin panels
- **Database**: PostgreSQL 15+ with Alembic migrations per plugin
- **Deployment**: Docker Compose with gunicorn, air-gapped sites via bundle workflow
- **Plugins**: 20+ plugins (computers, endpoints, printers, machines, etc.)
## Critical Constraints
### ADR-015: Site-Specific Configuration
Site literals (building names, printer models, machine types) MUST live in:
- `shopdb/core/settings/` - Site settings (buildings, roles, etc.)
- `shopdb/core/seed/` - Initial data SQL
- `scripts/site_imports/` - Import tools for Maximo/WINC data
NEVER hardcode site-specific data in plugin code or migrations.
### ADR-008: Plugin Migration Ownership
Each plugin owns its migration chain in `plugins/{name}/migrations/versions/`.
- Migrations run in dependency order defined in PLUGIN_TABLE_OWNERS
- Each plugin's chain must be linear (no branches)
- Migrations run automatically on deploy
### Database Naming Convention
- **Tables**: lowercase concatenated, no underscores (e.g., `printedpart`, `mappeddrives`)
- **Columns**: snake_case (e.g., `serial_number`, `last_seen`)
- **Indexes/constraints**: follow PostgreSQL conventions
- Enforced by `scripts/check-naming-and-style.sh` pre-commit hook
### Branch and Deployment Model
- `wj` -> prod (shop floor at WJ site)
- `dev` -> ops (monitoring/pilot sites)
- `main` -> nothing (GitHub public mirror only)
- Three air gaps: dev PC <-> work PC <-> GitHub
- Bundles carry code across gaps (pull-shopdb-bundle.cmd, fetch-wilm-branch.cmd)
### Code Style
- Plain ASCII only - no unicode, emojis, or em-dashes
- No LLM attribution comments (no "Co-Authored-By: Claude")
- Minimal comments - code should be self-documenting
- No banned shorthand (e.g., "config" not "cfg", "initialize" not "init")
## Plugin Contract
Plugins extend ShopDB via:
- `plugin.py` - BasePlugin subclass with manifest, routes, hooks
- `models/` - SQLAlchemy models
- `api/` - Flask blueprints and routes
- `frontend/views/` - Vue components auto-discovered by router
- `migrations/versions/` - Alembic migrations
Import from `shopdb.api` for stable surfaces:
```python
from shopdb.api import db, Asset, Employee, BasePlugin
```
## Documentation
Full documentation in the GitHub wiki at `../shopdb-flask.wiki/`:
**Critical ADRs**:
- ADR-001: Asset as platform contract
- ADR-004: Deployment topology (air-gapped sites)
- ADR-006: Collector contract (endpoint agent)
- ADR-007: Product versioning and releases
- ADR-013: Plugin catalog and lean builds
- ADR-016: Credential delivery
**References**:
- `API-REFERENCE.md` - Core API surface
- `PLUGIN-DEVELOPMENT.md` - Plugin authoring guide
- `DEPLOYMENT.md` - Deploy runbooks per site
- `TESTING.md` - Test patterns and CI
## Development Workflow
1. Feature branches off `dev` or `wj`
2. **Run tests**: `pytest -v` (REQUIRED before claiming completion)
3. **Check naming/style**: `scripts/check-naming-and-style.sh` (REQUIRED before commit)
4. Bundle for other PC or push to GitHub
5. PR to `wj` (for prod) or `dev` (for ops)
## Boundaries & Constraints
**DO NOT**:
- Edit generated files (frontend/dist/, docs/openapi.json)
- Change dependencies (requirements.txt, package.json) without approval
- Commit code with unicode/emojis (plain ASCII only)
- Hardcode site-specific data (buildings, models, etc.) - use settings/seed
- Skip tests or pre-commit hooks
- Break plugin migration chains (keep linear, coordinate via PLUGIN_TABLE_OWNERS)
**ALWAYS**:
- Run `pytest -v` before claiming work complete
- Run `scripts/check-naming-and-style.sh` before committing
- Use `shopdb.api` imports, not internal paths
- Keep database table names lowercase concatenated
- Reference ADRs by number when making architectural decisions
- Verify bundle/deploy scripts work before pushing to branches
## Testing Patterns
- Feature tests in `tests/test_plugins/test_{plugin}_{feature}.py`
- Use fixtures from `conftest.py` (app, client, db_session)
- Test migrations with `test_plugin_migrations.py`
- Frontend specs alongside components: `*.spec.js`
## Common Pitfalls
- **Site data in code**: Use settings/seed, not hardcoded values
- **Migration conflicts**: Keep plugin chains linear, coordinate with PLUGIN_TABLE_OWNERS
- **Unicode**: ASCII only - check logs, comments, commit messages
- **Import surface**: Use `shopdb.api`, not internal paths
- **Device tokens**: Kiosks require device_token authentication (ADR-016)
## Work PC Role
This machine (work PC):
- Has GitHub credentials (ge-aero/shopdb-flask)
- Carries bundles between dev PC and GitHub
- Deploys to prod and ops servers
- Mirrors wilm branch from GitHub to gitea
For active development, use the dev PC with gitea.

24
DEPLOY-NOW.bat Normal file
View File

@@ -0,0 +1,24 @@
@echo off
REM ShopDB Quick Deploy - Double-click this file on the VM
REM This will deploy ShopDB Docker in one click
echo ========================================
echo ShopDB Docker Quick Deploy
echo ========================================
echo.
REM Check if we're in the right directory
if not exist "docker-compose.prebuilt.yml" (
echo ERROR: Not in shopdb-flask directory
echo Please run this from the repository root
pause
exit /b 1
)
echo Starting deployment...
echo.
REM Run PowerShell deployment script
powershell.exe -ExecutionPolicy Bypass -File "%~dp0deploy-to-vm.ps1" -SiteName "shopdb-vm"
pause

177
DEPLOY-TO-VM.md Normal file
View File

@@ -0,0 +1,177 @@
# Deploy ShopDB to Hyper-V VM
Quick guide to deploy the Docker stack to a Windows 11 Hyper-V VM.
## Prerequisites on VM
1. **Docker Desktop** installed and running
2. **Git Bash** or **WSL** installed (for bash commands)
3. **PowerShell** (already on Windows 11)
## Step 1: Copy Files to VM
**Method A: Shared Folder**
```powershell
# On host PC, share the repo
# Then on VM, copy from network share
Copy-Item -Recurse \\host-pc\shopdb-flask C:\shopdb
```
**Method B: Git Clone**
```bash
# On VM
cd C:\
git clone git@github.com:ge-aero/shopdb-flask.git
cd shopdb-flask
git checkout wj
```
**Method C: Export/Import**
```powershell
# On host PC
Compress-Archive -Path C:\Users\270015376\Documents\github\flask-shopdb -DestinationPath C:\shopdb-deploy.zip
# Copy shopdb-deploy.zip to VM, then on VM:
Expand-Archive -Path C:\shopdb-deploy.zip -DestinationPath C:\shopdb
```
## Step 2: Run Deployment Script
```powershell
# On VM, open PowerShell as Administrator
cd C:\shopdb
# Option 1: Quick deploy with defaults
.\deploy-to-vm.ps1
# Option 2: Specify site name and password
.\deploy-to-vm.ps1 -SiteName "west-jefferson" -AdminPassword "YourSecurePassword123!"
# Option 3: See all options
.\deploy-to-vm.ps1 -Help
```
## What the Script Does
1. [OK] Checks Docker Desktop is running
2. [OK] Creates .env from template
3. [OK] Builds frontend (if needed)
4. [OK] Builds Docker images (2-3 minutes, offline packages)
5. [OK] Starts all services (db + api + backup)
6. [OK] Runs database migrations
7. [OK] Sets admin password
## Manual Deployment (Alternative)
If you prefer to run commands manually:
```bash
# 1. Create environment
cp .env.docker-dev .env
# 2. Build frontend (if not already built)
cd frontend
npm ci
npm run build
cd ..
# 3. Build and start Docker stack
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
# 4. Initialize database
docker exec flask-shopdb-api-1 flask db upgrade
# 5. Reset admin password (optional)
docker exec flask-shopdb-db-1 mysql -uroot -proot-dev-password-not-for-production shopdb_flask -e \
"UPDATE users SET passwordhash='scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547', failedlogins=0, lockeduntil=NULL WHERE username='admin';"
# (Password is: admin)
```
## Access
**URL:** https://localhost:5001 (or https://vm-ip-address:5001 from host)
**Login:**
- Username: `admin`
- Password: Whatever you set in deploy script (default: `ChangeMe123!`)
## Verify Deployment
```powershell
# Check all services are running
docker ps
# Should see:
# - flask-shopdb-db-1
# - flask-shopdb-api-1
# - flask-shopdb-backup-store-1
# - flask-shopdb-backup-watcher-1
# View logs
docker logs -f flask-shopdb-api-1
# Test backup
# Go to https://localhost:5001/settings/shopdbbackups
# Click "Back Up Now"
# Wait 30 seconds, should show as "Checked"
```
## Troubleshooting
**Docker not running:**
```powershell
# Start Docker Desktop manually
& "C:\Program Files\Docker\Docker\Docker Desktop.exe"
Start-Sleep -Seconds 30
```
**Port 5001 already in use:**
```powershell
# Edit .env and change API_PORT
# API_PORT=5002
```
**Can't access from host PC:**
```powershell
# On VM, check Windows Firewall
New-NetFirewallRule -DisplayName "ShopDB" -Direction Inbound -LocalPort 5001 -Protocol TCP -Action Allow
```
**SSL certificate warnings:**
- Normal for self-signed certs
- Click "Advanced" -> "Proceed to localhost (unsafe)"
## Stop/Start
```bash
# Stop
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml down
# Start (after first deployment)
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
```
## Update After Code Changes
```bash
# 1. Pull latest code
git pull origin wj
# 2. Rebuild frontend (if changed)
cd frontend && npm run build && cd ..
# 3. Rebuild and restart
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
```
## Offline Deployment Notes
**This deployment works completely offline** because:
- [OK] Frontend pre-built and checked into repo
- [OK] Restic binary pre-downloaded (8.6MB in repo)
- [OK] No npm/pip downloads during Docker build
- [OK] All dependencies baked into images
Perfect for air-gapped or restricted network environments!

235
DOCKER-DEPLOYMENT-COMPLETE.md Normal file
View File

@@ -0,0 +1,235 @@
# Docker Deployment - Complete and Working
**Date:** 2026-09-07
**Branch:** wj (post-wilm merge)
**Status:** [OK] Core system operational, backup watcher pending
## What's Running
```
CONTAINER STATUS PORT
flask-shopdb-db-1 Up (healthy) 3306/tcp (internal)
flask-shopdb-api-1 Up https://localhost:5001
```
**Gunicorn Workers:** 4 workers with gevent (PIDs 74-77)
**Plugins Enabled:** 12 of 17 installed
**Database:** MySQL 8.4 with all migrations applied
## Access
**URL:** https://localhost:5001
**Expected:** Browser warning for self-signed certificate (click through)
**Login:** admin / (needs password reset - see below)
## How to Use
### Start Stack
```bash
docker compose -f docker-compose.prebuilt.yml up -d
```
### Stop Stack
```bash
docker compose -f docker-compose.prebuilt.yml down
```
### View Logs
```bash
docker logs -f flask-shopdb-api-1
docker logs flask-shopdb-db-1
```
### Reset Admin Password
```bash
# Generate hash
docker exec flask-shopdb-api-1 python -c \
"from werkzeug.security import generate_password_hash; print(generate_password_hash('your-password'))"
# Update database (replace HASH with output above)
docker exec flask-shopdb-db-1 mysql -uroot -p$MYSQL_ROOT_PASSWORD shopdb_flask -e \
"UPDATE users SET passwordhash='HASH', failedlogins=0, lockeduntil=NULL WHERE username='admin';"
```
Quick reset to password "admin":
```bash
docker exec flask-shopdb-db-1 mysql -uroot -proot-dev-password-not-for-production shopdb_flask -e \
"UPDATE users SET
passwordhash='scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins=0, lockeduntil=NULL
WHERE username='admin';"
```
### Rebuild After Code Changes
```bash
# Rebuild frontend (if frontend code changed)
cd frontend
npm run build
cd ..
# Rebuild Docker image
docker compose -f docker-compose.prebuilt.yml build api
# Restart
docker compose -f docker-compose.prebuilt.yml up -d
```
### Run Database Migrations
```bash
docker exec flask-shopdb-api-1 flask db upgrade
```
### Plugin Management
```bash
# List plugins
docker exec flask-shopdb-api-1 flask plugin list
# Enable plugin
docker exec flask-shopdb-api-1 flask plugin enable PLUGIN_NAME
# Disable plugin
docker exec flask-shopdb-api-1 flask plugin disable PLUGIN_NAME
```
## Build Process Used
### Why Pre-Built Frontend?
**Problem:** Zscaler corporate proxy causes npm to fail inside Docker builds
**Solution:** Build frontend on host, copy into Docker image
### Build Steps Taken
1. **Build Frontend on Host**
```bash
cd frontend
npm ci
npm run build
# Creates frontend/dist/
```
2. **Build Docker Image (Simple)**
```bash
# Uses Dockerfile.prebuilt (Python only, no npm)
docker compose -f docker-compose.prebuilt.yml build
```
3. **Start Services**
```bash
docker compose -f docker-compose.prebuilt.yml up -d
```
4. **Auto-Bootstrap**
- Container entrypoint runs migrations
- Detects first deployment (no instance/plugins.json)
- Auto-installs all 17 plugins (disabled by default)
- Starts gunicorn
## Architecture
### Volumes
- `flask-shopdb_db_data` - MySQL database (persistent)
- `flask-shopdb_instance_data` - Plugin state, uploads, branding (persistent)
### Networks
- `flask-shopdb_default` - Internal bridge network
### Image
- **Built:** 2026-09-07 00:26:27 EDT
- **Size:** 914MB
- **Base:** python:3.14-slim
- **Frontend:** Pre-built Vue 3 SPA (from host)
## What's Not Working Yet
### Backup Watcher Service
**Status:** [FAIL] Build failed
**Reason:** Oracle Linux base image can't download packages due to SSL cert issues with Zscaler
**Impact:** Can't auto-execute backups on schedule (manual backups still work via one-shot container)
**Error:**
```
Cannot download repomd.xml: Curl error (60): SSL peer certificate or SSH remote key was not OK
```
**Workaround Attempted:**
- Copied Zscaler cert to scripts/backup/zscaler-root-ca.pem
- Dockerfile.backup copies cert and runs update-ca-trust
- Still fails because microdnf runs BEFORE cert is trusted
**Fix Needed:**
- Pre-download restic binary and bake into image OR
- Use different base image (Alpine/Debian instead of Oracle Linux) OR
- Skip backup watcher entirely, use cron on host to trigger one-shot backup container
## Production Readiness
**Core Application:** [OK] Production ready
**Deployment Process:** [WARN] Requires npm on host + Docker build
**Backup System:** [FAIL] Not complete (watcher service build fails)
### Recommended Next Steps
1. **CI/CD Pipeline** - Build images automatically, push to registry
2. **Pre-Built Images** - Sites pull instead of building (2 min deploy)
3. **Backup Watcher Fix** - Switch to Alpine base or pre-download restic
4. **TLS Certificates** - Use proper certs from CA (not self-signed)
5. **Secrets Management** - Externalize passwords (.env with strong values)
## Files Created
- `Dockerfile.prebuilt` - Simplified Docker build (no npm)
- `docker-compose.prebuilt.yml` - Docker Compose for pre-built approach
- `.dockerignore.prebuilt` - Allows frontend/dist to be copied
- `start-docker-quick.sh` - Quick restart script
- `DOCKER-PREBUILT-APPROACH.md` - Documents workaround
- `PRODUCTION-READINESS.md` - Honest assessment
## Comparison to Previous Attempts
| Approach | Result | Time | Issue |
|----------|--------|------|-------|
| Multi-stage Dockerfile | [FAIL] Failed | 15+ min | npm fails with Zscaler |
| Pre-built frontend | [OK] Success | 3 min | Works! |
| Backup containers | [FAIL] Failed | N/A | Oracle Linux SSL issues |
## Verification
```bash
# Test frontend
curl -k https://localhost:5001/
# Should return: <!DOCTYPE html>
# Test API
curl -k https://localhost:5001/api/settings
# Should return: {"data":[...]}
# Test database
docker exec flask-shopdb-db-1 mysql -uroot -p$MYSQL_ROOT_PASSWORD \
-e "SELECT COUNT(*) FROM shopdb_flask.users;"
# Should return: count of users
# Check workers
docker logs flask-shopdb-api-1 2>&1 | grep "Booting worker"
# Should show: 4 workers with PIDs
# Check plugins
docker exec flask-shopdb-api-1 flask plugin list | grep Enabled | wc -l
# Should return: 12
```
## Summary
ShopDB Docker deployment is **operational** for core functionality:
- [OK] Database running
- [OK] API serving requests
- [OK] Frontend built and accessible
- [OK] Plugins installed and working
- [OK] HTTPS configured
- [WARN] Backup watcher needs work (but manual backups functional)
The pre-built frontend approach successfully bypasses Zscaler corporate proxy issues. Build time is 2-3 minutes (vs 15+ for failed multi-stage approach).
For production deployment, recommend setting up CI/CD to build images automatically and push to registry so sites can pull pre-built images instead of building locally.

96
DOCKER-PREBUILT-APPROACH.md Normal file
View File

@@ -0,0 +1,96 @@
# Docker Pre-Built Frontend Approach
**Problem:** Zscaler corporate proxy causes npm to fail inside Docker builds with "Exit handler never called" error.
**Solution:** Build frontend on host (where npm works), copy pre-built dist/ into Docker image.
## Quick Start
```bash
# 1. Build frontend on host
cd frontend
npm ci
npm run build
# 2. Build Docker image (uses pre-built dist/)
cd ..
docker compose -f docker-compose.prebuilt.yml build
# 3. Start services
docker compose -f docker-compose.prebuilt.yml up -d
# 4. Initialize (first time only)
docker compose -f docker-compose.prebuilt.yml exec api flask db upgrade
docker compose -f docker-compose.prebuilt.yml exec api bash /app/scripts/bootstrap-plugins.sh
# 5. Access
https://localhost:5001
```
## Files
- `Dockerfile.prebuilt` - Simplified Dockerfile without npm build
- `docker-compose.prebuilt.yml` - Uses Dockerfile.prebuilt
- `.dockerignore.prebuilt` - Allows frontend/dist to be copied
## Why This Works
**Normal Dockerfile:**
- Multi-stage build
- Stage 1: Node 20 runs `npm ci` and `npm run build`
- Stage 2: Copies built frontend from stage 1
- **PROBLEM:** npm fails inside Docker with Zscaler proxy
**Pre-Built Approach:**
- Build frontend on Windows host (npm works here)
- Single-stage Dockerfile copies pre-built `frontend/dist/`
- Only installs Python dependencies
- **FASTER:** 2-3 minutes vs 15+ minutes
## Trade-offs
**Pros:**
- Avoids corporate proxy issues
- Much faster builds (no npm in container)
- Frontend build errors visible immediately
- Can iterate on frontend without rebuilding container
**Cons:**
- Requires npm on host machine
- Two-step build process
- frontend/dist must be in version control OR built before deployment
## For Production (CI/CD)
```yaml
# GitHub Actions / Jenkins example
- name: Build frontend
run: |
cd frontend
npm ci
npm run build
- name: Build Docker image
run: |
docker build -f Dockerfile.prebuilt -t shopdb:${{ github.sha }} .
- name: Push to registry
run: |
docker push shopdb:${{ github.sha }}
```
## Comparison
| Approach | Build Time | Corporate Proxy | Complexity | Production Ready |
|----------|-----------|----------------|------------|------------------|
| Multi-stage Dockerfile | 15+ min | [FAIL] Fails | High | [FAIL] No |
| Pre-built frontend | 2-3 min | [OK] Works | Medium | [WARN] Yes (with CI/CD) |
| Pre-built images from registry | 30 sec | [OK] Works | Low | [OK] Yes |
## Next Steps
For true production deployment:
1. Set up CI/CD to build frontend + Docker image
2. Push to container registry
3. Sites pull pre-built images
4. No build step at deployment site

398
DOCKER-SETUP.md Normal file
View File

@@ -0,0 +1,398 @@
# ShopDB Docker Setup
Complete Docker deployment for development and testing.
## Prerequisites
- Docker Desktop installed and running
- At least 8GB RAM allocated to Docker
- Zscaler corporate certificate extracted (if behind corporate proxy)
## Quick Start
```bash
# 1. Clone repository
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Extract Zscaler certificate (if needed)
powershell.exe -File C:/temp/extract-ca.ps1
# This creates corp-root-ca.pem in repo root
# 3. Build images
docker compose build
# 4. Start stack
docker compose up -d
# 5. Initialize database (first time only)
docker compose exec api flask db upgrade
# 6. Install plugins (first time only)
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
# 7. Access ShopDB
https://localhost:5001
```
Login: `admin` / (reset password - see below)
## Architecture
The Docker stack includes:
1. **MySQL Database** (`db` service)
- Port: 3306 (internal), mapped to host based on `.env`
- Volume: `db_data` for persistence
- Health checks enabled
2. **Flask API** (`api` service)
- Port: 5001 (HTTPS with self-signed cert)
- Built frontend served from `/app/frontend/dist`
- Gunicorn with gevent workers (4 workers, 2000 connections each)
- Volumes:
- `instance_data` - Plugin state, uploads, branding
- `./plugins` - Plugin code (read-only)
- `./scripts` - Deployment scripts (read-only)
- `./certs` - TLS certificates (read-only)
3. **Backup Service** (optional, via `docker-compose.backup.yml`)
- One-shot container for backups
- Triggered manually or via scheduler
## First-Time Setup
### 1. Extract Zscaler Certificate (Corporate Proxy)
Save this as `C:/temp/extract-ca.ps1`:
```powershell
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
$certPath = 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem'
Set-Content -Path $certPath -Value $builder.ToString() -Encoding ASCII
Write-Host "Staged corp-root-ca.pem"
```
Run: `powershell.exe -ExecutionPolicy Bypass -File C:/temp/extract-ca.ps1`
### 2. Generate TLS Certificates
```bash
cd certs
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout server.key -out server.crt -days 365 \
-subj "//CN=localhost"
```
Or use the provided script:
```bash
./scripts/generate-dev-certs.sh
```
### 3. Configure Environment
Copy and edit `.env`:
```bash
cp .env.docker-dev .env
# Edit if needed - defaults work for local dev
```
### 4. Build Images
```bash
docker compose build
```
**Note:** First build takes 10-15 minutes due to:
- npm install (~5 min with Zscaler proxy)
- Frontend Vite build (~2 min)
- Python dependencies (~3 min)
Subsequent builds use Docker layer cache and are much faster.
### 5. Start Services
```bash
docker compose up -d
```
Check status:
```bash
docker compose ps
docker compose logs -f api
```
### 6. Initialize Database
```bash
# Run migrations
docker compose exec api flask db upgrade
# Install and enable all plugins
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
```
The `apply-profile` command:
- Installs all 17 plugins
- Registers them in `instance/plugins.json`
- Leaves them disabled by default
- Enables them via Settings UI as needed
### 7. Reset Admin Password
```bash
# Generate hash for "admin" password
docker compose exec api python -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Update database
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
Or use quick method (sets password to `admin`):
```bash
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users SET
passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
## Import Production Database
```bash
# Copy SQL dump to container
docker cp latest.sql shopdb-api-1:/tmp/
# Import
docker compose exec db mysql -u root -p shopdb_flask < /tmp/latest.sql
# Run migrations to bring schema up to date
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```
## Common Commands
### View Logs
```bash
docker compose logs -f api
docker compose logs -f db
```
### Restart Services
```bash
docker compose restart api
docker compose down && docker compose up -d
```
### Rebuild After Code Changes
```bash
docker compose build api
docker compose up -d api
```
### Access Container Shell
```bash
docker compose exec api bash
docker compose exec db bash
```
### Run Flask Commands
```bash
docker compose exec api flask plugin list
docker compose exec api flask db current
docker compose exec api flask routes
```
### Backup & Restore
```bash
# Trigger backup
docker compose -f docker-compose.backup.yml up backup
# View backups
docker compose exec api flask plugin list
# (Accessible via Settings > Backups in UI)
# Restore (via UI at Settings > Backups > Select backup > Restore)
```
## Troubleshooting
### "npm ci failed" during build
**Cause:** Zscaler proxy blocking npm registry.
**Fix:**
1. Extract Zscaler certificate (see above)
2. Ensure `corp-root-ca.pem` exists in repo root
3. Rebuild: `docker compose build --no-cache`
### "database is unhealthy"
**Cause:** MySQL container not ready or port conflict.
**Fix:**
```bash
# Check logs
docker compose logs db
# Check if port 3306 is already in use
netstat -an | grep 3306
# If conflict, edit .env and set MYSQL_PORT=3307
```
### "Access denied for user 'shopdb'"
**Cause:** Database credentials mismatch.
**Fix:** Check `.env` file - `MYSQL_PASSWORD` must match `DATABASE_URL`.
### "Could not connect to database"
**Cause:** Database service not started or wrong hostname.
**Fix:**
```bash
# Check db is running
docker compose ps
# API must use hostname 'db' not 'localhost'
# In docker-compose.yml: DATABASE_URL=...@db:3306/...
```
### "ERR_CERT_AUTHORITY_INVALID" in browser
**Expected:** Self-signed certificate warning.
**Fix:** Click "Advanced" > "Proceed to localhost (unsafe)" - this is normal for dev.
Or generate proper cert:
```bash
# Add your machine's hostname to cert
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout certs/server.key -out certs/server.crt -days 365 \
-subj "//CN=$(hostname)" \
-addext "subjectAltName=DNS:localhost,DNS:$(hostname),IP:127.0.0.1"
```
### "Plugin X is already installed" but not visible
**Cause:** Plugin installed but not enabled.
**Fix:**
```bash
docker compose exec api flask plugin enable X
docker compose restart api
```
### Build is very slow
**Expected:** First build with Zscaler proxy takes 10-15 minutes.
**Speed up:**
- Subsequent builds use cache (much faster)
- If rebuilding often, comment out `--no-cache` flag
- Build frontend separately on host, copy dist/ into image
## Production Deployment Notes
For production:
1. **Change all secrets** in `.env`:
- `SECRET_KEY`
- `JWT_SECRET_KEY`
- `MYSQL_ROOT_PASSWORD`
- `MYSQL_PASSWORD`
2. **Use proper TLS certificates** (not self-signed):
- Place in `./certs/server.crt` and `./certs/server.key`
- Or use reverse proxy (nginx/IIS) for TLS termination
3. **Set `FLASK_DEBUG=0`** in production
4. **Configure backups**:
- Set up cron job: `0 2 * * * cd /path/to/shopdb && docker compose -f docker-compose.backup.yml up backup`
- Configure SMB or S3 destination in Settings > Backups
5. **Set `SHOPDB_ENV_LABEL`** to site name (e.g., "WEST-JEFFERSON")
6. **Review `docker-compose.yml`**:
- Adjust worker count based on load
- Configure proper `restart` policy
- Set resource limits if needed
## Architecture Notes
### Why HTTPS in Docker?
Secure context APIs (clipboard, geolocation) require HTTPS. Running over HTTP disables these browser features.
### Why Gevent Workers?
Endpoint agents long-poll `/api/endpoints/commands/next`. With threading, 200 agents = 200 threads = resource exhaustion. Gevent uses greenlets (cheap coroutines) so 2000+ connections per worker is feasible.
### Plugin Bootstrap
The entrypoint (`scripts/docker-entrypoint.sh`) automatically:
1. Runs database migrations
2. Checks if `instance/plugins.json` exists
3. If not, installs all available plugins (disabled)
4. Starts gunicorn
This ensures imported databases with plugin tables get matching plugin registry.
### Instance Data Persistence
The `instance_data` volume persists:
- `plugins.json` - Which plugins are enabled
- Uploaded files (floor plans, branding, photos, warranties, slides)
- Dell OAuth tokens
- Any plugin-specific state
**Important:** `docker compose down -v` deletes volumes - use plain `docker compose down` to preserve data.
## File Structure
```
flask-shopdb/
|-- docker-compose.yml # Main stack definition
|-- docker-compose.backup.yml # Backup service
|-- Dockerfile # Multi-stage build
|-- .env # Environment config (gitignored)
|-- corp-root-ca.pem # Zscaler cert (gitignored)
|-- certs/ # TLS certificates (gitignored)
| |-- server.crt
| +-- server.key
|-- scripts/
| |-- docker-entrypoint.sh # Container startup
| +-- bootstrap-plugins.sh # Plugin auto-install
+-- deploy/
+-- site-profile-universal.json # Full plugin list
```
## Next Steps
After setup:
1. Browse to https://localhost:5001
2. Login as `admin`
3. Go to Settings > Plugins
4. Enable plugins your site needs
5. Configure settings (CORS, backup destination, etc.)
6. Import data or start fresh
For local development with hot-reload, see `LOCAL-DEV-SETUP.md`.

6
Dockerfile
View File

@@ -123,6 +123,9 @@ COPY migrations/ ./migrations/
COPY scripts/ ./scripts/ COPY scripts/ ./scripts/
COPY wsgi.py ./ COPY wsgi.py ./
# Make entrypoint and bootstrap scripts executable
RUN chmod +x /app/scripts/docker-entrypoint.sh /app/scripts/bootstrap-plugins.sh
# Built SPA from stage 1. Flask serves it via register_frontend_routes. # Built SPA from stage 1. Flask serves it via register_frontend_routes.
COPY --from=frontendbuild /build/frontend/dist ./frontend/dist COPY --from=frontendbuild /build/frontend/dist ./frontend/dist
@@ -139,6 +142,9 @@ USER shopdb
EXPOSE 5001 EXPOSE 5001
# Entrypoint runs migrations and bootstraps plugins on first deployment
ENTRYPOINT ["/app/scripts/docker-entrypoint.sh"]
# Probe the public health route, not the login endpoint. Probing POST # Probe the public health route, not the login endpoint. Probing POST
# /api/auth/login logged a 400 every 30s and ran the credential path from a # /api/auth/login logged a 400 every 30s and ran the credential path from a
# health check. The route answers 200 even when the database is degraded, so the # health check. The route answers 200 even when the database is degraded, so the

47
Dockerfile.prebuilt Normal file
View File

@@ -0,0 +1,47 @@
# Dockerfile for pre-built frontend
# Use this when frontend/dist is already built on the host
# Avoids npm issues with corporate proxy
FROM python:3.14-slim AS base
WORKDIR /app
# Install system dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
pkg-config \
curl \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir gunicorn gevent
# Copy application code
COPY shopdb/ ./shopdb/
COPY plugins/ ./plugins/
COPY migrations/ ./migrations/
COPY scripts/ ./scripts/
COPY wsgi.py ./
# Copy pre-built frontend
COPY frontend/dist ./frontend/dist
# Make scripts executable
RUN chmod +x /app/scripts/docker-entrypoint.sh /app/scripts/bootstrap-plugins.sh
# Create instance directory
RUN mkdir -p /app/instance
# Expose port
EXPOSE 5001
# Set environment
ENV PYTHONUNBUFFERED=1
ENV FLASK_APP=shopdb
# Entrypoint runs migrations and starts gunicorn
ENTRYPOINT ["/app/scripts/docker-entrypoint.sh"]

362
LOCAL-DEV-SETUP.md Normal file
View File

@@ -0,0 +1,362 @@
# ShopDB Local Development Setup (Windows)
Complete guide for setting up ShopDB Flask development environment on Windows.
## Prerequisites
- Python 3.13+
- Docker Desktop (for MySQL and testing)
- Git
- Node.js 20+ (for frontend development)
## Quick Start
```bash
# 1. Clone and enter repo
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Create Python virtual environment
python -m venv venv
./venv/Scripts/activate
pip install -r requirements.txt
# 3. Start MySQL in Docker
docker compose up -d db
# 4. Configure environment
cp .env.docker .env
# Edit .env and set DATABASE_URL (see below)
# 5. Install and enable plugins
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe plugin install backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty
./venv/Scripts/flask.exe plugin enable <each-plugin-name>
# 6. Start Flask
./venv/Scripts/python.exe run_dev.py
```
Visit http://localhost:5001 and login with `admin` / `admin`.
## Detailed Setup
### 1. Database Setup
MySQL runs in Docker on **port 3307** (not 3306) to avoid conflicts with any local MySQL instance.
```bash
# Start MySQL container
docker compose up -d db
# Verify it's running
docker ps --filter name=shopdb-dev-db
# Should show: 127.0.0.1:3307->3306/tcp
# Test connection
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' -e "SELECT 1;"
```
### 2. Environment Configuration
Create `.env` file with:
```bash
# Database (port 3307 because Docker maps it there)
DATABASE_URL=mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev
# Flask
FLASK_ENV=development
FLASK_DEBUG=1
# Secrets (dev only)
SECRET_KEY=docker-dev-secret-key-change-for-production
JWT_SECRET_KEY=docker-dev-jwt-secret-change-for-production
# CORS
CORS_ORIGINS=http://localhost:5173,http://localhost:5001
# API Port
API_PORT=5001
# ShopDB Label
SHOPDB_ENV_LABEL=LOCAL-DEV
# Log Level
LOG_LEVEL=DEBUG
# Backup Storage (for local dev)
SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups
```
**Important:** Change `YOUR_USERNAME` to your actual Windows username.
### 3. Import Database (Optional)
If you have a production database dump:
```bash
# Import latest.sql
docker exec -i shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev < latest.sql
# Run migrations
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### 4. Plugin Installation
All 17 plugins must be installed before they can be enabled:
```bash
# Set DATABASE_URL for CLI commands
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
# Install all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin install $plugin
done
# Enable all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin enable $plugin
done
```
**Why this is necessary:**
- Plugin state is tracked in `instance/plugins.json`, not the database
- When you import a prod database, it has plugin tables but no plugin registry
- Installing registers plugins in the JSON file
- Enabling makes them active (but requires app restart)
### 5. Reset Admin Password
If you imported a prod database and don't know the password:
```bash
# Generate new password hash
./venv/Scripts/python.exe -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Reset via MySQL
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Or use the simpler method:
```bash
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Password is now: `admin` / `admin`
### 6. Starting Flask
**Always use `run_dev.py`** (not `flask run` directly):
```bash
./venv/Scripts/python.exe run_dev.py
```
This script:
- Loads `.env` file with DATABASE_URL
- Sets all environment variables
- Starts Flask without auto-reloader (prevents env var loss)
- Binds to 0.0.0.0:5001
**Do NOT use:**
- `flask run` - Won't load `.env` properly
- `flask run --reload` - Loses environment variables
### 7. Frontend Development (Optional)
To develop the Vue frontend:
```bash
cd frontend
npm ci
npm run dev
```
Frontend dev server runs on http://localhost:5173 and proxies API requests to Flask on port 5001.
For production build (served by Flask):
```bash
cd frontend
npm run build
# Outputs to frontend/dist/
```
## Troubleshooting
### "Can't connect to MySQL on 127.0.0.1:3306"
**Cause:** Flask is trying port 3306 instead of 3307.
**Fix:**
1. Make sure `.env` has `DATABASE_URL=...@localhost:3307/...`
2. Use `run_dev.py` to start Flask (NOT `flask run`)
3. If problem persists, restart Flask completely
### "Cannot import name '_backups_configured'"
**Cause:** Merge conflict between wj and wilm backup implementations.
**Fix:** This was fixed in commit `8a1fe0ada8` - pull latest wj branch.
### "Plugin X is not installed"
**Cause:** Plugin registry (`instance/plugins.json`) doesn't exist or is incomplete.
**Fix:** Run plugin installation commands (section 4 above).
### "Access is denied: '/backups'"
**Cause:** Backup system trying to use Linux path on Windows.
**Fix:**
1. Add to `.env`: `SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups`
2. Create directory: `mkdir backups`
3. Restart Flask with `run_dev.py`
### Docker npm install fails (Zscaler proxy)
**Cause:** Corporate proxy intercepts HTTPS, npm doesn't trust certificate.
**Fix:** Extract Zscaler root CA certificate:
```powershell
# Run this PowerShell script
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
Set-Content -Path 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem' -Value $builder.ToString() -Encoding ASCII
```
Then rebuild Docker: `docker compose build --no-cache`
## Common Tasks
### Run Tests
```bash
pytest
pytest tests/test_specific.py
pytest -v -s # Verbose with print output
```
### Run Migrations
```bash
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### Create Migration
```bash
./venv/Scripts/flask.exe db migrate -m "Description of changes"
```
### Check Plugin Status
```bash
./venv/Scripts/flask.exe plugin list
```
### Enable/Disable Plugin
```bash
./venv/Scripts/flask.exe plugin enable slides
./venv/Scripts/flask.exe plugin disable slides
```
### Trigger Backup
```bash
# Via API (requires admin JWT token)
curl -X POST http://localhost:5001/api/shopdbbackups \
-H "Authorization: Bearer YOUR_TOKEN"
# Or via Docker
docker compose -f docker-compose.backup.yml up backup
```
## Architecture Notes
### Why Port 3307?
Docker MySQL maps internal port 3306 to host port 3307 to avoid conflicts with any local MySQL installation. This is configured in `docker-compose.yml`:
```yaml
ports:
- "127.0.0.1:3307:3306"
```
### Why run_dev.py?
Flask's `flask run` command uses an auto-reloader that spawns a subprocess. This subprocess loses environment variables loaded from `.env`. The `run_dev.py` wrapper:
1. Loads `.env` via python-dotenv BEFORE importing Flask
2. Uses `--no-reload` flag to prevent subprocess spawning
3. Ensures DATABASE_URL and other vars persist
### Plugin Architecture
- Plugins are Python packages in `plugins/` directory
- Plugin state tracked in `instance/plugins.json` (not database)
- Plugin tables tracked via Alembic migration chains
- Each plugin has own migrations in `plugins/<name>/migrations/`
### Backup System
- Designed for Docker compose deployment
- API queues backups via sentinel file (`.backup-requested`)
- Actual backup execution requires Docker sidecar container
- On Windows dev, API can queue but won't execute without Docker
## File Structure
```
flask-shopdb/
|-- .env # Local environment config (gitignored)
|-- run_dev.py # Local development launcher
|-- shopdb/ # Flask application
| |-- __init__.py # App factory
| |-- config.py # Configuration classes
| |-- core/ # Core models and APIs
| +-- plugins/ # Plugin manager
|-- plugins/ # Plugin packages
| |-- backups/
| |-- computers/
| +-- ...
|-- frontend/ # Vue 3 SPA
| |-- src/
| +-- dist/ # Production build (gitignored)
|-- instance/ # Runtime state (gitignored)
| |-- plugins.json # Plugin registry
| +-- ...
|-- migrations/ # Core Alembic migrations
|-- scripts/ # Deployment and backup scripts
|-- backups/ # Local backup storage (gitignored)
+-- tests/ # Test suite
```
## Git Workflow
- `main` branch - Published/public version (GitHub)
- `wj` branch - Production deployment (West Jefferson site)
- `wilm` branch - Development branch (upstream from benjamin.baker)
- `dev` branch - Ops deployment branch
**Never commit to `main` directly** - it's published to public wiki.
## Additional Resources
- [Backup Setup](scripts/backup/README.md)
- [Plugin Development](docs/PLUGIN-HOOKS.md)
- [ADRs](wiki/ADRs/)
- [API Docs](http://localhost:5001/api/docs) (when Flask is running)

345
PRODUCTION-READINESS.md Normal file
View File

@@ -0,0 +1,345 @@
# ShopDB Production Readiness Assessment
**Last Updated:** 2026-09-07
**Branch:** wj (post-wilm merge)
## Executive Summary
[CRITICAL] **NOT PRODUCTION READY** in current state without significant operational investment.
The Docker deployment requires extensive manual setup, multiple services, and assumes expertise with Docker Compose, restic backup systems, and Flask plugin architecture. This assessment documents current gaps and provides recommendations.
## Current State: Deployment Complexity
### What Works
[OK] **Flask application code** - Stable, tested, 17 plugins functional
[OK] **Database schema** - Migrations work, data model is sound
[OK] **Frontend build** - Vue 3 SPA compiles and serves correctly
[OK] **Docker images** - Build successfully (with caveats below)
[OK] **Plugin architecture** - Clean, documented, extensible
### Critical Gaps
[FAIL] **One-command deployment** - Requires 7-10 manual steps
[FAIL] **Corporate proxy handling** - Zscaler cert extraction needed for npm
[FAIL] **Backup system complexity** - 3 containers + restic + scheduler
[FAIL] **TLS certificate generation** - Manual OpenSSL commands required
[FAIL] **Plugin initialization** - Must run `apply-profile` after first start
[FAIL] **Password reset** - No default admin password, requires manual SQL
[FAIL] **Build time** - 15+ minutes with corporate proxy
[FAIL] **Documentation scattered** - 3 different setup guides (Docker, Local, Backup)
## Why Is This So Complex?
### 1. Corporate Network Reality
**Problem:** Behind Zscaler proxy, npm can't trust npm registry
**Impact:** Docker build fails without extracted corporate root CA
**Workaround:** PowerShell script to extract cert before building
**Production Fix:** Pre-build images in CI/CD with cert, publish to registry
### 2. Plugin Architecture Design
**Problem:** Plugin state in `instance/plugins.json`, not database
**Impact:** Importing prod database doesn't register plugins
**Workaround:** Manual `flask plugin apply-profile` after first start
**Production Fix:** Auto-detect installed plugins from schema on boot
### 3. Backup System Architecture
**Problem:** Designed for restic with 3-container orchestration
**Impact:** Requires backup-store + backup-watcher + backup containers
**Workaround:** All documented, but complex to deploy
**Production Fix:** Simpler mysqldump-only mode with direct file storage
### 4. Multi-Stage Docker Build
**Problem:** Frontend build needs Node 20, backend needs Python 3.14
**Impact:** Multi-stage build is slow, complex, hard to debug
**Workaround:** Build works, but takes 15 minutes
**Production Fix:** Pre-build frontend on CI, copy dist/ into Python-only image
## Deployment Models (Ordered by Simplicity)
### Option 1: IIS + Windows Services (SIMPLEST)
**Best for:** Single Windows server deployment
**Complexity:** Low
**Maintenance:** Medium
**Setup:**
1. Install Python 3.13+
2. Install MySQL (or use existing instance)
3. Run installer (creates Windows service for Flask)
4. Configure IIS reverse proxy
5. Install plugins via UI
**Pros:**
- Native Windows deployment
- Familiar IIS management
- No Docker complexity
- Fast startup
**Cons:**
- Windows-only
- Manual updates
- No containerization benefits
**Production Ready:** [WARN] **YES** (with installer improvements)
### Option 2: Docker Pre-Built Images (RECOMMENDED)
**Best for:** Sites with Docker but no build capability
**Complexity:** Low-Medium
**Maintenance:** Low
**Setup:**
1. Pull pre-built image from registry
2. Run docker-compose with .env file
3. Access UI, enable plugins
**Pros:**
- No build step
- Fast deployment (2 minutes)
- Consistent environment
- Easy updates (pull new image)
**Cons:**
- Requires Docker
- Need image registry
- Initial registry setup
**Production Ready:** [OK] **YES** (requires CI/CD setup)
### Option 3: Docker Source Build (CURRENT)
**Best for:** Development, customization
**Complexity:** High
**Maintenance:** High
**Setup:**
1. Extract Zscaler cert
2. Generate TLS certs
3. Build Docker images (15+ min)
4. Configure .env with 15+ variables
5. Start docker-compose
6. Run migrations
7. Install plugins
8. Reset admin password
9. Configure backup system
10. Test everything
**Pros:**
- Full source control
- Customizable
- No external dependencies
**Cons:**
- Extremely complex
- Long build times
- Many failure points
- Requires Docker expertise
**Production Ready:** [CRITICAL] **NO** (too complex for operations)
### Option 4: Kubernetes (FUTURE)
**Best for:** Multi-site deployment, high availability
**Complexity:** Very High
**Maintenance:** Medium (with proper GitOps)
**Status:** Not implemented
**Effort:** 2-3 weeks of work
## What Would Make This Production Ready?
### Critical Path (Must Have)
1. **Pre-Built Images in Registry**
- Build images in CI/CD
- Push to container registry
- Sites just `docker pull`
- **Effort:** 1 week
2. **One-Command Setup**
- `./deploy.sh` does everything
- No manual cert extraction
- No manual plugin installation
- **Effort:** 3 days
3. **Automated Plugin Detection**
- Scan database schema on boot
- Auto-register plugins in JSON
- No manual `apply-profile`
- **Effort:** 2 days
4. **Simplified Backup (Option)**
- Simple mysqldump + tar mode
- Single container, no restic
- Good enough for 80% of sites
- **Effort:** 1 week
5. **Windows Installer Improvements**
- One-click install
- Includes Python runtime
- Creates service automatically
- **Effort:** 1 week
### Nice to Have (Future)
- Helm chart for Kubernetes
- Monitoring/metrics built-in
- Health check endpoints
- Automated testing in CI
- Rolling update strategy
## Honest Recommendations
### For West Jefferson (Your Site)
**Recommended:** Option 1 (IIS + Windows Services)
**Why:**
- You already have Windows infrastructure
- Simpler to operate
- Faster to deploy
- Your team knows IIS
**Steps:**
1. Build Windows installer
2. Deploy to Windows Server
3. Configure IIS reverse proxy
4. Import prod database
5. Done
**Timeline:** 2-3 days including testing
### For New Sites
**Recommended:** Option 2 (Docker Pre-Built Images)
**Prerequisites:**
- Set up CI/CD to build images
- Set up container registry (Docker Hub, Harbor, or AWS ECR)
- Create `deploy.sh` script
**Then sites can:**
```bash
# 1. Copy .env.example to .env and edit
cp .env.example .env
# 2. Run deployment
./deploy.sh
# 3. Access UI
https://your-server:5001
```
**Timeline for CI/CD setup:** 1-2 weeks
### For Current Docker Source Build
**Status:** Development/customization only, not for production
**Use when:**
- Testing new features
- Developing plugins
- Debugging issues
- Learning the system
**Don't use when:**
- Deploying to operations
- Training new ops staff
- Time-sensitive deployments
## Current Documentation Issues
### Too Many Guides
- `LOCAL-DEV-SETUP.md` - Windows development (port 3307, run_dev.py)
- `DOCKER-SETUP.md` - Docker source build (complex, 15+ steps)
- `PRODUCTION-READINESS.md` - This file (meta/assessment)
- `scripts/backup/README.md` - Backup system (3 containers, restic)
**Problem:** Which one do I follow?
**Solution:** Create decision tree:
- Developer on Windows -> LOCAL-DEV-SETUP.md
- Ops deploying to prod -> (DOESN'T EXIST YET - need simplified guide)
- Custom development -> DOCKER-SETUP.md
- Backup questions -> scripts/backup/README.md
### Missing Guides
- **PRODUCTION-DEPLOY.md** - Step-by-step for operations (IIS or Docker pull)
- **QUICK-START.md** - 5-minute version for evaluation
- **TROUBLESHOOTING.md** - Common issues, not scattered across files
- **UPGRADE-GUIDE.md** - How to update between versions
## What We Did Today (Honest Assessment)
[OK] **Merged wilm->wj** - 98 commits, wilm features now in wj
[OK] **Fixed local dev** - `run_dev.py` works for Windows development
[OK] **Documented Docker** - Comprehensive but too complex
[OK] **Added backup docs** - System works but needs expertise
[WARN] **Docker "production ready"** - Technically works, operationally complex
[FAIL] **Simple deployment** - Still requires too many manual steps
## Next Steps (Priority Order)
### Immediate (This Week)
1. **Create QUICK-START.md**
- 5-minute eval version
- Uses pre-built image (if available) OR local dev
- Gets people to working system fast
2. **Test Windows Installer**
- Build installer for wj branch
- Deploy to test Windows Server
- Document any issues
3. **Create deployment decision tree**
- Flowchart: what's my situation -> which method to use
- Don't make users read 4 docs to figure out where to start
### Short Term (Next Sprint)
4. **Set up CI/CD for image builds**
- GitHub Actions or Jenkins
- Build on every push to wj
- Push to internal registry
5. **Create `deploy.sh` script**
- One command: `./deploy.sh --env prod`
- Handles certs, plugins, migrations automatically
6. **Simplify backup system**
- Add "simple mode" - just mysqldump + tar
- Keep restic for advanced sites
- Document tradeoffs
### Long Term (Next Month)
7. **Auto-plugin detection**
- Boot-time schema scan
- No manual `apply-profile`
8. **Consolidate documentation**
- One entry point
- Clear paths for each audience
- Tested by someone who hasn't seen the system
## Conclusion
**Is ShopDB production-ready?**
- **The code:** Yes, it's solid
- **Docker source build deployment:** No, too complex
- **Windows installer deployment:** Yes, with improvements
- **Docker pre-built images deployment:** Yes, but needs CI/CD setup
**Bottom line:** The application is mature. The deployment story needs work. Focus should shift from "making Docker work" to "making deployment simple."
**Recommendation:** Invest 2-3 weeks in deployment automation (CI/CD, pre-built images, deploy.sh) rather than continuing to document complex manual processes.

199
agent/INSTALL-ON-VM.md Normal file
View File

@@ -0,0 +1,199 @@
# Install ShopDB Agent on Hyper-V VM
This guide shows how to install the fleet agent on your Hyper-V VM to report back to ShopDB running on the host PC.
## Configuration for Your Setup
**ShopDB Server (Host PC):**
- URL: `https://172.31.96.1:5001`
- Collector API Key: `shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43`
**VM Network:**
- VM can reach host at: `172.31.96.1`
- Host's Hyper-V virtual adapter IP
---
## Step 1: Build the Agent (On Host PC)
**Install .NET SDK first:**
```powershell
# On Host PC (this machine):
winget install Microsoft.DotNet.SDK.8
```
**Build agent executables:**
```powershell
cd C:\Users\270015376\Documents\github\flask-shopdb\agent
# Build service
dotnet publish ShopDbEndpointAgent -c Release -r win-x64 --self-contained -o C:\temp\shopdb-agent
# Build tray app (optional)
dotnet publish ShopDbEndpointAgent.Tray -c Release -r win-x64 --self-contained -o C:\temp\shopdb-agent-tray
# Package for VM
Compress-Archive -Path C:\temp\shopdb-agent -DestinationPath C:\temp\shopdb-agent.zip
```
---
## Step 2: Copy to VM
**Copy the ZIP to your VM:**
- Via RDP shared folder
- Via network share
- Via USB
**Extract on VM:**
```powershell
# On VM:
Expand-Archive C:\path\to\shopdb-agent.zip -DestinationPath C:\ProgramData\ShopDB\EndpointAgent
```
---
## Step 3: Configure Agent (On VM)
**Set registry configuration:**
```powershell
# Run as Administrator on VM:
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
# Point to HOST PC's IP (not localhost!)
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'BaseUrl' -Value 'https://172.31.96.1:5001'
# Use the collector key
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'CollectorKey' -Value 'shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43'
```
---
## Step 4: Test Agent (On VM)
**Test one report:**
```powershell
cd C:\ProgramData\ShopDB\EndpointAgent
.\shopdb-endpoint-agent.exe --once
```
**Expected output:**
```
[INFO] Agent starting...
[INFO] Collecting endpoint state...
[INFO] Reporting to https://172.31.96.1:5001/api/collector/endpoints
[INFO] Report successful
```
**If you see SSL certificate errors:**
The self-signed cert will cause warnings. You can either:
- Ignore them (agent still works)
- Or use HTTP (not recommended): `http://172.31.96.1:5001`
---
## Step 5: Install as Service (On VM)
**Create Windows service:**
```powershell
# Run as Administrator on VM:
sc.exe create "ShopDBEndpointAgent" `
binPath= "C:\ProgramData\ShopDB\EndpointAgent\shopdb-endpoint-agent.exe" `
start= auto `
DisplayName= "ShopDB Endpoint Agent"
sc.exe description "ShopDBEndpointAgent" "Reports endpoint telemetry to ShopDB server."
# Start the service
sc.exe start "ShopDBEndpointAgent"
```
**Check service status:**
```powershell
Get-Service ShopDBEndpointAgent
```
---
## Step 6: Verify on Host PC
**Check ShopDB received the report:**
1. Open: `https://localhost:5001/computers`
2. Look for your VM's hostname
3. Click on it to see endpoint data
**Or check logs:**
```bash
# On host PC:
docker logs flask-shopdb-api-1 | grep -i "collector/endpoints"
```
---
## Troubleshooting
### Agent can't reach server
**Test connectivity from VM:**
```powershell
# On VM:
Test-NetConnection -ComputerName 172.31.96.1 -Port 5001
```
**If blocked by firewall:**
```powershell
# On Host PC, allow inbound:
New-NetFirewallRule -DisplayName "ShopDB API" -Direction Inbound -LocalPort 5001 -Protocol TCP -Action Allow
```
### SSL certificate errors
The agent works even with self-signed cert warnings. To eliminate them:
**Option 1: Use HTTP (dev only)**
```powershell
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'BaseUrl' -Value 'http://172.31.96.1:5001'
```
**Option 2: Trust the cert**
- Export cert from host
- Import to VM's Trusted Root
### Agent reports but PC doesn't appear
**Check collector key matches:**
```powershell
# On VM:
Get-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'CollectorKey'
# Should match what's in ShopDB .env:
# shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43
```
---
## Uninstall
```powershell
# Stop and remove service
sc.exe stop "ShopDBEndpointAgent"
sc.exe delete "ShopDBEndpointAgent"
# Remove files
Remove-Item -Recurse C:\ProgramData\ShopDB\EndpointAgent
# Remove registry
Remove-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Recurse
```
---
## Quick Reference
**Host IP:** `172.31.96.1`
**ShopDB URL:** `https://172.31.96.1:5001`
**Collector Key:** `shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43`
**Agent Location:** `C:\ProgramData\ShopDB\EndpointAgent\`
**Service Name:** `ShopDBEndpointAgent`
**Report Interval:** 15 minutes (configurable in appsettings.json)

51
agent/QUICK-FIX-VM.md Normal file
View File

@@ -0,0 +1,51 @@
# Quick Fix: Agent Can't Connect from VM
## Problem
SSL certificate errors when connecting from VM to host.
## Solution: Use HTTP on Local Network
Since the VM and host are on a trusted local network (Hyper-V), use HTTP instead of HTTPS.
### On VM - Set HTTP in Registry:
```powershell
# On VM (as Administrator):
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'BaseUrl' -Value 'http://172.31.96.1:5001'
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name 'CollectorKey' -Value 'shopdb_pat_b4a3fe2a01abc6bc1cb685280e65a8c35d52ec43'
```
### Test it works:
```powershell
# On VM:
curl.exe http://172.31.96.1:5001
# Should return HTML
```
### Then run the agent:
```powershell
cd C:\ProgramData\ShopDB\EndpointAgent
.\shopdb-endpoint-agent.exe --once
```
## Why HTTP is OK Here
- [X] VM and host on same physical machine (Hyper-V)
- [X] Traffic never leaves the host
- [X] Hyper-V internal network is isolated
- [X] No external network exposure
For production multi-machine deployment, use HTTPS with proper certificates.
## Alternative: Fix HTTPS (More Complex)
If you need HTTPS:
1. Export the self-signed cert from host
2. Import to VM's Trusted Root Certification Authorities
3. Use HTTPS URL
See main INSTALL-ON-VM.md for details.

136
deploy-to-vm.ps1 Normal file
View File

@@ -0,0 +1,136 @@
# ShopDB Docker Deployment Script for Hyper-V VM
# Run this ON THE VM after copying the repo
param(
[string]$SiteName = "shopdb-vm",
[string]$AdminPassword = "ChangeMe123!",
[switch]$Help
)
if ($Help) {
Write-Host @"
ShopDB Docker Deployment Script
Usage:
.\deploy-to-vm.ps1 [-SiteName <name>] [-AdminPassword <password>]
Parameters:
-SiteName Site identifier (default: shopdb-vm)
-AdminPassword Initial admin password (default: ChangeMe123!)
-Help Show this help
Example:
.\deploy-to-vm.ps1 -SiteName "west-jefferson" -AdminPassword "SecurePass123!"
Prerequisites:
- Docker Desktop installed and running
- Git Bash or WSL available
"@
exit 0
}
$ErrorActionPreference = "Stop"
Write-Host "========================================" -ForegroundColor Cyan
Write-Host "ShopDB Docker Deployment" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
# Check Docker is running
Write-Host "[1/8] Checking Docker Desktop..." -ForegroundColor Yellow
try {
docker ps | Out-Null
Write-Host " OK: Docker is running" -ForegroundColor Green
} catch {
Write-Host " ERROR: Docker Desktop is not running" -ForegroundColor Red
Write-Host " Please start Docker Desktop and try again" -ForegroundColor Red
exit 1
}
# Check if we're in the repo
Write-Host "[2/8] Checking repository..." -ForegroundColor Yellow
if (-not (Test-Path "docker-compose.prebuilt.yml")) {
Write-Host " ERROR: Not in shopdb-flask directory" -ForegroundColor Red
Write-Host " Please cd to the repository root" -ForegroundColor Red
exit 1
}
Write-Host " OK: In repository root" -ForegroundColor Green
# Create .env if it doesn't exist
Write-Host "[3/8] Configuring environment..." -ForegroundColor Yellow
if (-not (Test-Path ".env")) {
Copy-Item ".env.docker-dev" ".env"
Write-Host " Created .env from template" -ForegroundColor Green
# Update site name
(Get-Content ".env") -replace 'SHOPDB_SITE=shopdb-dev', "SHOPDB_SITE=$SiteName" | Set-Content ".env"
(Get-Content ".env") -replace 'SHOPDB_ENV_LABEL=DOCKER-DEV', "SHOPDB_ENV_LABEL=$SiteName" | Set-Content ".env"
Write-Host " Configured for site: $SiteName" -ForegroundColor Green
} else {
Write-Host " OK: .env already exists" -ForegroundColor Green
}
# Build frontend if needed
Write-Host "[4/8] Checking frontend build..." -ForegroundColor Yellow
if (-not (Test-Path "frontend/dist/index.html")) {
Write-Host " Building frontend (this may take 2-3 minutes)..." -ForegroundColor Yellow
Set-Location frontend
npm ci
npm run build
Set-Location ..
Write-Host " OK: Frontend built" -ForegroundColor Green
} else {
Write-Host " OK: Frontend already built" -ForegroundColor Green
}
# Build Docker images
Write-Host "[5/8] Building Docker images..." -ForegroundColor Yellow
Write-Host " This will take 2-3 minutes (using offline packages)..." -ForegroundColor Yellow
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml build
Write-Host " OK: Images built" -ForegroundColor Green
# Start services
Write-Host "[6/8] Starting services..." -ForegroundColor Yellow
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
Start-Sleep -Seconds 10
Write-Host " OK: Services started" -ForegroundColor Green
# Run migrations
Write-Host "[7/8] Initializing database..." -ForegroundColor Yellow
docker exec flask-shopdb-api-1 flask db upgrade
Write-Host " OK: Database initialized" -ForegroundColor Green
# Set admin password
Write-Host "[8/8] Setting admin password..." -ForegroundColor Yellow
$passwordHash = docker exec flask-shopdb-api-1 python -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('$AdminPassword'))"
$dbPassword = (Get-Content ".env" | Select-String "MYSQL_ROOT_PASSWORD=(.+)" | ForEach-Object { $_.Matches.Groups[1].Value })
docker exec flask-shopdb-db-1 mysql -uroot -p$dbPassword shopdb_flask -e @"
UPDATE users SET
passwordhash = '$passwordHash',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"@
Write-Host " OK: Admin password set" -ForegroundColor Green
Write-Host ""
Write-Host "========================================" -ForegroundColor Green
Write-Host "Deployment Complete!" -ForegroundColor Green
Write-Host "========================================" -ForegroundColor Green
Write-Host ""
Write-Host "Access ShopDB at: https://localhost:5001" -ForegroundColor Cyan
Write-Host ""
Write-Host "Login credentials:" -ForegroundColor Yellow
Write-Host " Username: admin" -ForegroundColor White
Write-Host " Password: $AdminPassword" -ForegroundColor White
Write-Host ""
Write-Host "Services running:" -ForegroundColor Yellow
docker ps --format "table {{.Names}}\t{{.Status}}"
Write-Host ""
Write-Host "To view logs:" -ForegroundColor Yellow
Write-Host " docker logs -f flask-shopdb-api-1" -ForegroundColor White
Write-Host ""
Write-Host "To stop:" -ForegroundColor Yellow
Write-Host " docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml down" -ForegroundColor White
Write-Host ""

11
deploy/windows/web.config
View File

@@ -101,6 +101,17 @@
</rewrite> </rewrite>
SHOPDB-CLIENTIP-END --> SHOPDB-CLIENTIP-END -->
<security>
<requestFiltering>
<!-- IIS request filtering caps a request body at 30,000,000 bytes by
default and answers 413 before waitress sees it. The agent package
upload accepts 128 MB (MAX_CONTENT_LENGTH is 256 MB), and the tray
binary alone is ~69 MB, so the IIS cap has to be raised to match or
every agent upload fails on Windows while working under Docker. -->
<requestLimits maxAllowedContentLength="268435456" />
</requestFiltering>
</security>
</system.webServer> </system.webServer>
<!-- <!--

76
docker-compose.prebuilt.yml Normal file
View File

@@ -0,0 +1,76 @@
# Docker Compose using pre-built frontend
# Build frontend on host first: cd frontend && npm run build
# Then: docker compose -f docker-compose.prebuilt.yml build
services:
db:
image: mysql:8.4
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: shopdb_flask
MYSQL_USER: shopdb
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
volumes:
- db_data:/var/lib/mysql
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
api:
build:
context: .
dockerfile: Dockerfile.prebuilt
ports:
- "${API_PORT:-5001}:5001"
environment:
- DATABASE_URL=mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask
- SECRET_KEY=${SECRET_KEY}
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
- FLASK_ENV=${FLASK_ENV:-production}
- FLASK_DEBUG=${FLASK_DEBUG:-0}
- CORS_ORIGINS=${CORS_ORIGINS}
- SHOPDB_ENV_LABEL=${SHOPDB_ENV_LABEL}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- ZABBIX_URL=${ZABBIX_URL:-}
- ZABBIX_TOKEN=${ZABBIX_TOKEN:-}
- COLLECTOR_API_KEY=${COLLECTOR_API_KEY:-}
- COLLECTOR_API_KEY_ENDPOINTS=${COLLECTOR_API_KEY_ENDPOINTS:-}
volumes:
- instance_data:/app/instance
- ./plugins:/app/plugins:ro
- ./scripts:/app/scripts:ro
- ./certs:/app/certs:ro
depends_on:
db:
condition: service_healthy
restart: unless-stopped
command:
- gunicorn
- --bind
- 0.0.0.0:5001
- --worker-class
- gevent
- --workers
- "4"
- --worker-connections
- "2000"
- --timeout
- "60"
- --access-logfile
- "-"
- --access-logformat
- '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
- --error-logfile
- "-"
- --certfile
- /app/certs/server.crt
- --keyfile
- /app/certs/server.key
- wsgi:app
volumes:
db_data:
instance_data:

832
docs/openapi.json
View File

@@ -194,6 +194,9 @@
}, },
{ {
"name": "endpoints-siteagent" "name": "endpoints-siteagent"
},
{
"name": "core-shopdbbackups"
} }
], ],
"paths": { "paths": {
@@ -18765,9 +18768,8 @@
"plugin-printers" "plugin-printers"
], ],
"summary": "Find a printer by IP/FQDN (parity with classic printerlookup.asp; used by Zabbix to deep-link to a printer record).", "summary": "Find a printer by IP/FQDN (parity with classic printerlookup.asp; used by Zabbix to deep-link to a printer record).",
"description": "Find a printer by IP/FQDN (parity with classic printerlookup.asp; used by Zabbix to deep-link to a printer record).\n\n**Auth:** jwt-optional\n\n**Params:** ip or fqdn (one required; value matched against communication ipaddress)\n\n**Example:**\n```\ncurl 'http://localhost:5001/api/printers/lookup?ip=10.1.2.42'\n```", "description": "Find a printer by IP/FQDN (parity with classic printerlookup.asp; used by Zabbix to deep-link to a printer record).\n\n**Auth:** printers.view\n\n**Params:** ip or fqdn (one required; ip matches a communication address, fqdn matches the printer's hostname)\n\n**Example:**\n```\ncurl 'http://localhost:5001/api/printers/lookup?ip=10.1.2.42'\n```",
"security": [ "security": [
{},
{ {
"bearerAuth": [] "bearerAuth": []
} }
@@ -18788,6 +18790,12 @@
"$ref": "#/components/schemas/ErrorEnvelope" "$ref": "#/components/schemas/ErrorEnvelope"
} }
} }
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
} }
} }
} }
@@ -23332,6 +23340,414 @@
] ]
} }
}, },
"/api/endpoints/asset/{asset_id}/hardware": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: processor, memory and chassis inventory. Agent 1.7.0 and later only; an older agent leaves the...",
"description": "Asset-detail panel: processor, memory and chassis inventory. Agent 1.7.0 and later only; an older agent leaves the columns null and the panel hides itself rather than rendering blanks\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/hardware\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/osinfo": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: operating system, firmware and domain membership as one card (both answer \"what is this box...",
"description": "Asset-detail panel: operating system, firmware and domain membership as one card (both answer \"what is this box running\")\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/osinfo\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/storage": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: volume capacity and physical drive health as two sections. They share no key (volumes vs physical...",
"description": "Asset-detail panel: volume capacity and physical drive health as two sections. They share no key (volumes vs physical devices) and are never reconciled across sections\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/storage\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/diskhealth": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Physical drive health rows, anything predicting failure sorted first. No longer backs a panel of its own (Storage...",
"description": "Physical drive health rows, anything predicting failure sorted first. No longer backs a panel of its own (Storage renders it as a section) but kept as a stable read for callers outside the panel renderer\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/diskhealth\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/listeners": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: TCP sockets in LISTEN with the owning process, sorted by port. The full connection table is an...",
"description": "Asset-detail panel: TCP sockets in LISTEN with the owning process, sorted by port. The full connection table is an on-demand command, not standard telemetry\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/listeners\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/mappeddrives": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: mapped network drives with a health badge each. Session-scoped, so an empty list when nobody is...",
"description": "Asset-detail panel: mapped network drives with a health badge each. Session-scoped, so an empty list when nobody is signed in is a valid state, not a failure\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/mappeddrives\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/usercontext": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: session-scoped data from the interactive user probe. Mapped drives arrive from two sources that can...",
"description": "Asset-detail panel: session-scoped data from the interactive user probe. Mapped drives arrive from two sources that can disagree and the worse status wins, so a half-broken share is never shown healthy\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/usercontext\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/endpoints/asset/{asset_id}/iphistory": {
"get": {
"tags": [
"plugin-endpoints"
],
"summary": "Asset-detail panel: every address this asset has held, newest change first. \"Last changed\" is when a field on the row...",
"description": "Asset-detail panel: every address this asset has held, newest change first. \"Last changed\" is when a field on the row moved, NOT when the address was last seen\n\n**Auth:** endpoints.view\n\n**Params:** path: asset_id\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/endpoints/asset/1/iphistory\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "asset_id",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/scripts": { "/api/scripts": {
"get": { "get": {
"tags": [ "tags": [
@@ -24394,6 +24810,418 @@
} }
} }
} }
},
"/api/shopdbbackups": {
"get": {
"tags": [
"core-shopdbbackups"
],
"summary": "List every retained backup set, newest first, each with a compatibility flag saying whether it restores onto this build",
"description": "List every retained backup set, newest first, each with a compatibility flag saying whether it restores onto this build\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
}
},
"post": {
"tags": [
"core-shopdbbackups"
],
"summary": "Kick off an on-demand backup run. Requires the backup overlay: without a configured backups root there is nothing...",
"description": "Kick off an on-demand backup run. Requires the backup overlay: without a configured backups root there is nothing behind it\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -X POST -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
},
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"additionalProperties": true,
"description": "none"
}
}
}
}
}
},
"/api/shopdbbackups/{snapshotid}": {
"get": {
"tags": [
"core-shopdbbackups"
],
"summary": "Return one set's full manifest plus a resolved compatibility verdict",
"description": "Return one set's full manifest plus a resolved compatibility verdict\n\n**Auth:** admin\n\n**Params:** path: snapshotid\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/2026-09-06T120000Z\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "snapshotid",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
]
}
},
"/api/shopdbbackups/{snapshotid}/restore": {
"post": {
"tags": [
"core-shopdbbackups"
],
"summary": "Queue a restore of a set, gated on schema compatibility. DESTRUCTIVE: the load drops and replaces every table",
"description": "Queue a restore of a set, gated on schema compatibility. DESTRUCTIVE: the load drops and replaces every table\n\n**Auth:** admin\n\n**Params:** path: snapshotid\n\n**Example:**\n```\ncurl -X POST -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/2026-09-06T120000Z/restore\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
},
"404": {
"description": "No such record."
}
},
"parameters": [
{
"name": "snapshotid",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"additionalProperties": true,
"description": "path: snapshotid"
}
}
}
}
}
},
"/api/shopdbbackups/finalize-restore": {
"post": {
"tags": [
"core-shopdbbackups"
],
"summary": "Queue the post-restore schema upgrade for the in-container finalizer, and retry it if a previous attempt failed. Does...",
"description": "Queue the post-restore schema upgrade for the in-container finalizer, and retry it if a previous attempt failed. Does NOT run the upgrade in the request: that watcher runs it out of band, so a restore completes whether or not the browser that started it is still open, and Alembic cannot be killed by the worker timeout mid-chain\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -X POST -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/finalize-restore\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
},
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"additionalProperties": true,
"description": "none"
}
}
}
}
}
},
"/api/shopdbbackups/status": {
"get": {
"tags": [
"core-shopdbbackups"
],
"summary": "Live status of the most recent backup and restore runs, or idle when neither is moving",
"description": "Live status of the most recent backup and restore runs, or idle when neither is moving\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/status\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
}
}
},
"/api/shopdbbackups/history": {
"get": {
"tags": [
"core-shopdbbackups"
],
"summary": "Recorded backup and restore run history, newest first",
"description": "Recorded backup and restore run history, newest first\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/history\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
}
}
},
"/api/shopdbbackups/settings": {
"get": {
"tags": [
"core-shopdbbackups"
],
"summary": "Read the backup settings, with secret values masked",
"description": "Read the backup settings, with secret values masked\n\n**Auth:** admin\n\n**Params:** none\n\n**Example:**\n```\ncurl -H \"Authorization: Bearer $TOKEN\" http://localhost:5001/api/shopdbbackups/settings\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
}
},
"put": {
"tags": [
"core-shopdbbackups"
],
"summary": "Update the backup settings. A masked secret submitted unchanged leaves the stored value alone",
"description": "Update the backup settings. A masked secret submitted unchanged leaves the stored value alone\n\n**Auth:** admin\n\n**Params:** body: backup settings keys\n\n**Example:**\n```\ncurl -X PUT -H \"Authorization: Bearer $TOKEN\" -H 'Content-Type: application/json' -d '{\"backup_schedule\":\"0 2 * * *\"}' http://localhost:5001/api/shopdbbackups/settings\n```",
"security": [
{
"bearerAuth": []
}
],
"responses": {
"200": {
"description": "Success. Body is the success_response envelope: {status, data, meta}.",
"content": {
"application/json": {
"$ref": "#/components/schemas/SuccessEnvelope"
}
}
},
"default": {
"description": "Error. Body is the error envelope; the code and message are nested under data.error.",
"content": {
"application/json": {
"$ref": "#/components/schemas/ErrorEnvelope"
}
}
},
"401": {
"description": "Missing or invalid credentials."
},
"403": {
"description": "Authenticated, but not permitted."
}
},
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"additionalProperties": true,
"description": "body: backup settings keys"
}
}
}
}
}
} }
} }
} }

38
frontend/src/views/AppLayout.vue
View File

@@ -40,9 +40,9 @@
<div v-if="showDisplays" class="nav-section">Displays</div> <div v-if="showDisplays" class="nav-section">Displays</div>
<a v-if="showShopfloor" :href="withBase('/shopfloor')" target="_blank" class="external-link">Shopfloor Dashboard</a> <a v-if="showShopfloor" :href="withBase('/shopfloor')" target="_blank" class="external-link">Shopfloor Dashboard</a>
<a v-if="hasRoute('/tv')" :href="withBase('/tv')" target="_blank" class="external-link">TV Slideshow</a> <a v-if="hasRoute('/tv')" :href="withBase('/tv')" target="_blank" class="external-link">TV Slideshow</a>
<a v-if="hasRoute('/parts-kiosk')" :href="withBase('/parts-kiosk')" <a v-if="showPartsKiosk" :href="withBase('/parts-kiosk')"
target="_blank" class="external-link">Parts Kiosk</a> target="_blank" class="external-link">Parts Kiosk</a>
<a v-if="hasRoute('/usb-kiosk')" :href="withBase('/usb-kiosk')" <a v-if="showUsbKiosk" :href="withBase('/usb-kiosk')"
target="_blank" class="external-link">USB Kiosk</a> target="_blank" class="external-link">USB Kiosk</a>
<router-link v-if="authStore.isAdmin" to="/settings">Settings</router-link> <router-link v-if="authStore.isAdmin" to="/settings">Settings</router-link>
@@ -127,6 +127,7 @@ import { dashboardApi, notificationsApi } from '../api'
import { fiscalWeek } from '@/utils/fiscalWeek' import { fiscalWeek } from '@/utils/fiscalWeek'
import { getFacilityName, getSiteLogo, getServicenowUrls, getEnvLabel } from '../utils/siteSettings' import { getFacilityName, getSiteLogo, getServicenowUrls, getEnvLabel } from '../utils/siteSettings'
import { withBase } from '../utils/basePath' import { withBase } from '../utils/basePath'
import { isPluginEnabled, loadEnabledPlugins } from '../composables/enabledPlugins'
const router = useRouter() const router = useRouter()
const route = useRoute() const route = useRoute()
@@ -138,15 +139,36 @@ const authStore = useAuthStore()
// dead-ends on a blank page. Route existence is the true "is it in this build" // dead-ends on a blank page. Route existence is the true "is it in this build"
// test - more reliable than plugin-enabled state, which a copied registry can // test - more reliable than plugin-enabled state, which a copied registry can
// report for a plugin that was never staged. // report for a plugin that was never staged.
//
// The kiosks need BOTH tests. Staged-but-disabled is a real state: a full build
// stages every route, so a site that turns the plugin OFF still had the link,
// pointing at a kiosk its backend refuses. Enabled-but-unstaged is the state the
// paragraph above is about, and isPluginEnabled fails OPEN when it cannot reach
// the plugin list, so it cannot carry the test on its own.
const stagedRoutePaths = new Set(router.getRoutes().map(r => r.path)) const stagedRoutePaths = new Set(router.getRoutes().map(r => r.path))
const hasRoute = (path) => stagedRoutePaths.has(path) const hasRoute = (path) => stagedRoutePaths.has(path)
// Make kiosk visibility reactive - updates when plugin list loads
const pluginsLoaded = ref(false)
// Shopfloor Dashboard is a core view but runs entirely on the notifications // Shopfloor Dashboard is a core view but runs entirely on the notifications
// plugin (notificationsApi.getShopfloor + the calendar); without it the display // plugin (notificationsApi.getShopfloor + the calendar); without it the display
// is empty, so gate the link on the notifications plugin being in this build. // is empty, so gate the link on the notifications plugin being in this build.
const showShopfloor = hasRoute('/shopfloor') const showShopfloor = computed(() =>
&& (hasRoute('/notifications') || hasRoute('/calendar')) hasRoute('/shopfloor') && (hasRoute('/notifications') || hasRoute('/calendar'))
const showDisplays = showShopfloor || hasRoute('/tv') || hasRoute('/parts-kiosk') )
|| hasRoute('/usb-kiosk')
const showPartsKiosk = computed(() =>
pluginsLoaded.value && hasRoute('/parts-kiosk') && isPluginEnabled('printedparts')
)
const showUsbKiosk = computed(() =>
pluginsLoaded.value && hasRoute('/usb-kiosk') && isPluginEnabled('usb')
)
const showDisplays = computed(() =>
showShopfloor.value || hasRoute('/tv') || showPartsKiosk.value || showUsbKiosk.value
)
const routeViewKey = computed(() => const routeViewKey = computed(() =>
route.path.startsWith('/settings') ? '/settings' : route.path route.path.startsWith('/settings') ? '/settings' : route.path
) )
@@ -285,6 +307,10 @@ onMounted(async () => {
// granted mid-session stays invisible until the person logs out and back in. // granted mid-session stays invisible until the person logs out and back in.
if (authStore.isAuthenticated) authStore.fetchUser() if (authStore.isAuthenticated) authStore.fetchUser()
// Load enabled plugins list early so kiosk links render correctly
await loadEnabledPlugins()
pluginsLoaded.value = true
// Half-hourly: cheap, and it crosses midnight into the new week without a // Half-hourly: cheap, and it crosses midnight into the new week without a
// reload on a screen nobody touches. // reload on a screen nobody touches.
fiscalWeekTimer = setInterval(() => { currentFiscalWeek.value = fiscalWeek() }, 30 * 60 * 1000) fiscalWeekTimer = setInterval(() => { currentFiscalWeek.value = fiscalWeek() }, 30 * 60 * 1000)

34
frontend/src/views/SetupWizard.vue
View File

@@ -28,7 +28,8 @@
<div class="form-group"> <div class="form-group">
<label>Site logo <span class="hint">(optional)</span></label> <label>Site logo <span class="hint">(optional)</span></label>
<input type="file" accept="image/*" @change="uploadSiteLogo($event)" :disabled="logoUploading" /> <input type="file" accept="image/*" @change="uploadSiteLogo($event)" :disabled="logoUploading" />
<img v-if="siteLogo" :src="siteLogo" class="wizard-map-thumb" alt="site logo" /> <input v-model="logoFilename" type="text" class="form-control" placeholder="No file selected" readonly />
<img v-if="siteLogo" :src="withBase(siteLogo)" class="wizard-map-thumb" alt="site logo" />
<small class="hint">Skip to keep the shipped GE branding. You can change this later under Settings &gt; Branding.</small> <small class="hint">Skip to keep the shipped GE branding. You can change this later under Settings &gt; Branding.</small>
</div> </div>
</div> </div>
@@ -92,15 +93,17 @@
<!-- Floor map --> <!-- Floor map -->
<div v-else-if="current.key === 'map'"> <div v-else-if="current.key === 'map'">
<h2>Floor map</h2> <h2>Floor map</h2>
<p class="hint">Upload your facility blueprint (light and dark), and set its native pixel size.</p> <p class="hint">Upload your facility blueprint (light and dark) for the default level, and set its native pixel size. Additional levels can be added later in Settings.</p>
<div class="form-group"> <div class="form-group">
<label>Blueprint image (light theme)</label> <label>Blueprint image (light theme)</label>
<input type="file" accept="image/*" @change="uploadBlueprint('light', $event)" :disabled="mapUploading" /> <input type="file" accept="image/*" @change="uploadBlueprint('light', $event)" :disabled="mapUploading" />
<input v-model="blueprintLightFilename" type="text" class="form-control" placeholder="No file selected" readonly />
<img v-if="blueprintLight" :src="withBase(blueprintLight)" class="wizard-map-thumb" alt="light blueprint" /> <img v-if="blueprintLight" :src="withBase(blueprintLight)" class="wizard-map-thumb" alt="light blueprint" />
</div> </div>
<div class="form-group"> <div class="form-group">
<label>Blueprint image (dark theme)</label> <label>Blueprint image (dark theme)</label>
<input type="file" accept="image/*" @change="uploadBlueprint('dark', $event)" :disabled="mapUploading" /> <input type="file" accept="image/*" @change="uploadBlueprint('dark', $event)" :disabled="mapUploading" />
<input v-model="blueprintDarkFilename" type="text" class="form-control" placeholder="No file selected" readonly />
<img v-if="blueprintDark" :src="withBase(blueprintDark)" class="wizard-map-thumb dark" alt="dark blueprint" /> <img v-if="blueprintDark" :src="withBase(blueprintDark)" class="wizard-map-thumb dark" alt="dark blueprint" />
</div> </div>
<div class="form-row"> <div class="form-row">
@@ -203,14 +206,24 @@ const blueprintLight = ref('')
const blueprintDark = ref('') const blueprintDark = ref('')
const logoUploading = ref(false) const logoUploading = ref(false)
const siteLogo = ref('') const siteLogo = ref('')
// A file input shows the chosen name in the browser chrome, which the kiosk-
// style full-screen wizard hides, so the operator had no confirmation that the
// file they picked was the one being uploaded.
const logoFilename = ref('')
const blueprintLightFilename = ref('')
const blueprintDarkFilename = ref('')
async function uploadSiteLogo(event) { async function uploadSiteLogo(event) {
const file = event.target.files[0] const file = event.target.files[0]
if (!file) return if (!file) return
logoFilename.value = file.name
logoUploading.value = true logoUploading.value = true
try { try {
const { data } = await settingsApi.uploadBrandingLogo('site', file) const { data } = await settingsApi.uploadBrandingLogo('site', file)
siteLogo.value = data?.data?.value || '' // Same path every time, so without a cache-buster the preview shows the
// PREVIOUS logo and the upload reads as having silently failed. Display
// only - nothing persists this value.
siteLogo.value = data?.data?.value ? `${data.data.value}?t=${Date.now()}` : ''
toast.success('Logo uploaded') toast.success('Logo uploaded')
} catch (err) { } catch (err) {
toast.error(apiError(err, 'Upload failed')) toast.error(apiError(err, 'Upload failed'))
@@ -223,10 +236,12 @@ async function uploadSiteLogo(event) {
async function uploadBlueprint(theme, event) { async function uploadBlueprint(theme, event) {
const file = event.target.files[0] const file = event.target.files[0]
if (!file) return if (!file) return
if (theme === 'light') blueprintLightFilename.value = file.name
else blueprintDarkFilename.value = file.name
mapUploading.value = true mapUploading.value = true
try { try {
const { data } = await settingsApi.uploadMapBlueprint(theme, file) const { data } = await settingsApi.uploadMapBlueprint(theme, file)
const url = data?.data?.value const url = data?.data?.value ? `${data.data.value}?t=${Date.now()}` : ''
if (theme === 'light') blueprintLight.value = url if (theme === 'light') blueprintLight.value = url
else blueprintDark.value = url else blueprintDark.value = url
toast.success('Blueprint uploaded') toast.success('Blueprint uploaded')
@@ -307,6 +322,17 @@ onMounted(async () => {
} catch (err) { /* not set yet */ } } catch (err) { /* not set yet */ }
} }
} }
// Show whatever logo the site is already on, so re-running the wizard does
// not present an empty slot as if branding had never been set. Stored as a
// root-absolute path; the template applies withBase, so do NOT prefix here or
// a subpath mount gets it twice (/shopdb/shopdb/...).
const SHIPPED_LOGO = '/ge-aerospace-logo.svg'
try {
const response = await settingsApi.get('site_logo')
siteLogo.value = response.data?.data?.value || SHIPPED_LOGO
} catch (err) {
siteLogo.value = SHIPPED_LOGO
}
}) })
async function saveStep() { async function saveStep() {

31
open-firewall-for-vm.ps1 Normal file
View File

@@ -0,0 +1,31 @@
# Open Windows Firewall for ShopDB VM Access
# Run this as Administrator on the Host PC
Write-Host "Adding Windows Firewall rule for ShopDB API..." -ForegroundColor Cyan
try {
New-NetFirewallRule `
-DisplayName "ShopDB API - Port 5001" `
-Direction Inbound `
-LocalPort 5001 `
-Protocol TCP `
-Action Allow `
-Profile Any `
-ErrorAction Stop
Write-Host "SUCCESS: Firewall rule added!" -ForegroundColor Green
Write-Host ""
Write-Host "ShopDB is now accessible from VMs at:" -ForegroundColor Yellow
Write-Host " https://172.31.96.1:5001" -ForegroundColor White
Write-Host " https://172.30.0.1:5001" -ForegroundColor White
Write-Host ""
Write-Host "Test from VM with:" -ForegroundColor Yellow
Write-Host " Test-NetConnection -ComputerName 172.31.96.1 -Port 5001" -ForegroundColor White
Write-Host ""
} catch {
Write-Host "ERROR: Failed to add firewall rule" -ForegroundColor Red
Write-Host $_.Exception.Message -ForegroundColor Red
Write-Host ""
Write-Host "Make sure you're running PowerShell as Administrator!" -ForegroundColor Yellow
exit 1
}

264
plugins/computers/client/Set-ShopDBDeviceId.ps1 Normal file
View File

@@ -0,0 +1,264 @@
#requires -Version 5.1
<#
.SYNOPSIS
Seed HKLM:\SOFTWARE\GE\ShopDB\DeviceId from whatever identity a bay already
carries, so device identity can move off C:\Enrollment and into the
registry.
.DESCRIPTION
Phase 3 of docs/proposals/device-identity.md. The registry is the
destination because it survives the profile cleanup and disk hygiene that
eats C:\Enrollment, is readable as SYSTEM with no file ACL to get wrong, and
is already where the collector looks for BaseUrl and ApiToken.
This script does the one-time backfill: it reads the identity a bay holds
today and writes it to DeviceId. It is idempotent and safe to run every
GE-Enforce cycle.
THE SOURCE DEPENDS ON THE PC TYPE, AND THAT IS THE WHOLE POINT.
DeviceId names the DEVICE hanging off this PC. A machine number is a
different fact - "which bay is this" rather than "which device is this" -
and the two coincide for exactly one family:
CMM bays the machine number IS the instrument id (CMM4), which is
why cmmid.txt has always doubled as identity. Copying it
is correct.
part markers the machine number is the OPERATION (0613), and several
markers serve one operation. Copying it would name the
operation as the device. The collector refuses that
("asset X is not a Part Marker; not linked") because a
machine filed under a device label reads as a marker
everywhere downstream. So machine number is NEVER a
source here.
Keyence, no per-bay id exists at all. asset-id.txt is the only
Genspect, identity these have ever had.
wax-trace
everything not a device bay. DeviceId is not consumed for these
else pc-types, so writing one is a landmine for the day that
pc-type gains a device spec. Writes nothing.
9999 is the unassigned placeholder and is skipped everywhere, matching
Report-AssetToShopDB.ps1 and the server.
.PARAMETER Force
Overwrite a DeviceId that is already set to a DIFFERENT value.
Off by default on purpose. asset-id.txt is written ONCE at imaging and
nothing updates it when a device is swapped, so a stale file overwriting a
corrected registry value would re-introduce the exact defect this move is
meant to fix. A disagreement is reported and left alone for a human.
.PARAMETER PCType
Override the pc-type instead of reading C:\Enrollment\pc-type.txt. For
testing and for a bay whose file is missing.
.PARAMETER DeviceId
Write this value directly, skipping resolution. Still honours -Force and
the 9999 rule.
.PARAMETER LogPath
Append a transcript here as well as writing to the pipeline.
.EXAMPLE
.\Set-ShopDBDeviceId.ps1 -WhatIf
Show what would be written on this bay without touching the registry.
.EXAMPLE
.\Set-ShopDBDeviceId.ps1
Seed DeviceId. Safe to run repeatedly and from a GE-Enforce PS1 entry.
.NOTES
Exit codes:
0 wrote a value, it was already correct, or this is not a device bay
1 an error stopped the write
2 an existing DeviceId disagrees with the resolved value; needs a human
or -Force
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[switch]$Force,
[string]$PCType,
[string]$DeviceId,
[string]$LogPath
)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$RegPath = 'HKLM:\SOFTWARE\GE\ShopDB'
$RegValueName = 'DeviceId'
$Placeholder = '9999'
function Log {
param([string]$Message)
$line = "{0} [Set-ShopDBDeviceId] {1}" -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message
Write-Output $line
if ($LogPath) {
try { Add-Content -LiteralPath $LogPath -Value $line -ErrorAction Stop } catch { }
}
}
function Read-FirstLine {
# One line, trimmed, empty when absent or unreadable. A bay missing a file
# is the normal case, not an error.
param([string]$Path)
if (-not (Test-Path -LiteralPath $Path)) { return '' }
try {
$v = [string](Get-Content -LiteralPath $Path -First 1 -ErrorAction Stop)
return $v.Trim()
} catch {
Log "WARN could not read ${Path}: $($_.Exception.Message)"
return ''
}
}
function Get-MachineNoFromRegistry {
# eDNC/NTLARS, WOW6432Node first, matching Report-AssetToShopDB.ps1.
foreach ($rp in @(
'HKLM:\SOFTWARE\WOW6432Node\GE Aircraft Engines\DNC\General',
'HKLM:\SOFTWARE\GE Aircraft Engines\DNC\General'
)) {
try {
if (Test-Path $rp) {
$v = [string](Get-ItemProperty -Path $rp -Name MachineNo -ErrorAction Stop).MachineNo
if ($v) {
$v = $v.Trim()
if ($v -and $v -ne $Placeholder) { return $v }
}
}
} catch {
Log "WARN could not read MachineNo from ${rp}: $($_.Exception.Message)"
}
}
return ''
}
# Which pc-types drive a device, and whether the machine number is a legitimate
# identity for that device. Mirrors SUBORDINATE_DEVICE_MAP in
# plugins/computers/pctypemap.py. A site that added its own device pc-type
# through a subordinatedevice_<pctype> setting adds it here too.
$DeviceFamilies = @{
'gea-shopfloor-cmm' = @{ Label = 'CMM'; MachineNoIsDevice = $true }
'gea-shopfloor-keyence' = @{ Label = 'Vision System'; MachineNoIsDevice = $false }
'gea-shopfloor-genspect' = @{ Label = 'Genspect'; MachineNoIsDevice = $false }
'gea-shopfloor-waxtrace' = @{ Label = 'Form Tracer'; MachineNoIsDevice = $false }
'gea-shopfloor-partmarker'= @{ Label = 'Part Marker'; MachineNoIsDevice = $false }
}
# --- 1. what kind of bay is this -------------------------------------------
if (-not $PCType) { $PCType = Read-FirstLine 'C:\Enrollment\pc-type.txt' }
if (-not $PCType) {
Log 'no pc-type on this PC (C:\Enrollment\pc-type.txt missing or empty); nothing to do'
exit 0
}
if (-not $DeviceFamilies.ContainsKey($PCType)) {
Log "pc-type '$PCType' does not drive a device; DeviceId not written"
exit 0
}
$family = $DeviceFamilies[$PCType]
Log "pc-type '$PCType' drives a $($family.Label)"
# --- 2. resolve the identity ------------------------------------------------
$resolved = ''
$source = ''
if ($DeviceId) {
$resolved = $DeviceId.Trim()
$source = '-DeviceId parameter'
} else {
# Most stable identity first, and the machine number only where it really
# names the device.
if ($family.MachineNoIsDevice) {
$v = Read-FirstLine 'C:\Enrollment\cmm\cmmid.txt'
if ($v -and $v -ne $Placeholder) { $resolved = $v; $source = 'C:\Enrollment\cmm\cmmid.txt' }
if (-not $resolved) {
$v = Get-MachineNoFromRegistry
if ($v) { $resolved = $v; $source = 'eDNC MachineNo registry' }
}
}
if (-not $resolved) {
$v = Read-FirstLine 'C:\Enrollment\asset-id.txt'
if ($v -and $v -ne $Placeholder) { $resolved = $v; $source = 'C:\Enrollment\asset-id.txt' }
}
if (-not $resolved) {
# The 0.12.0 name, before the file was generalised.
$v = Read-FirstLine 'C:\Enrollment\measuringtool-id.txt'
if ($v -and $v -ne $Placeholder) { $resolved = $v; $source = 'C:\Enrollment\measuringtool-id.txt' }
}
}
if (-not $resolved) {
# Nothing to copy. Silence is the right answer: the collector still adopts
# or mints as it does today, and inventing a value here would name a device
# that nobody can account for.
Log "no identity found for this $($family.Label) bay; DeviceId not written"
exit 0
}
if ($resolved -eq $Placeholder) {
Log "resolved value is the $Placeholder placeholder; DeviceId not written"
exit 0
}
Log "resolved DeviceId '$resolved' from $source"
# --- 3. write it ------------------------------------------------------------
$existing = ''
try {
if (Test-Path $RegPath) {
$prop = Get-ItemProperty -Path $RegPath -Name $RegValueName -ErrorAction SilentlyContinue
if ($prop) { $existing = [string]$prop.$RegValueName; $existing = $existing.Trim() }
}
} catch {
Log "WARN could not read existing ${RegValueName}: $($_.Exception.Message)"
}
if ($existing -eq $resolved) {
Log "DeviceId already '$resolved'; nothing to do"
exit 0
}
if ($existing -and -not $Force) {
# The registry value wins by default. C:\Enrollment is written once at
# imaging and never updated on a device swap, so the file is the more
# likely one to be wrong.
Log "REFUSING to overwrite: DeviceId is '$existing', $source says '$resolved'"
Log 'Re-run with -Force if the file is right, or fix the file if the registry is.'
exit 2
}
try {
if (-not (Test-Path $RegPath)) {
if ($PSCmdlet.ShouldProcess($RegPath, 'Create registry key')) {
New-Item -Path $RegPath -Force | Out-Null
Log "created $RegPath"
}
}
$action = if ($existing) { "Set $RegValueName '$existing' -> '$resolved'" }
else { "Set $RegValueName '$resolved'" }
if ($PSCmdlet.ShouldProcess($RegPath, $action)) {
New-ItemProperty -Path $RegPath -Name $RegValueName -Value $resolved `
-PropertyType String -Force | Out-Null
Log "wrote $RegPath\$RegValueName = '$resolved'"
} else {
Log "WHATIF would write $RegPath\$RegValueName = '$resolved'"
}
} catch {
Log "ERROR could not write ${RegValueName}: $($_.Exception.Message)"
exit 1
}
exit 0

4
plugins/computers/frontend/views/ConfigurationPoliciesList.vue
View File

@@ -174,7 +174,9 @@ async function loadData() {
} }
if (types.status === 'fulfilled') { if (types.status === 'fulfilled') {
pctypes.value = types.value.data.data || [] // Plain array, same as applications below. Reading .data.data threw
// inside this try, so apps, actions and monitors never got assigned.
pctypes.value = types.value || []
} else { } else {
failures.push('PC types') failures.push('PC types')
console.error('Error loading PC types:', types.reason) console.error('Error loading PC types:', types.reason)

5
plugins/endpoints/plugin.py
View File

@@ -1121,7 +1121,10 @@ class EndpointsPlugin(BasePlugin):
if 'osinfo' in payload: if 'osinfo' in payload:
state.osinfojson = _as_json(payload.get('osinfo')) state.osinfojson = _as_json(payload.get('osinfo'))
if 'domain' in payload: if 'domain' in payload:
state.domain = payload.get('domain') # Clamped like every other String column here: an over-long value
# aborts the whole collector transaction under STRICT_TRANS_TABLES,
# so one bad agent report would drop a machine's entire telemetry.
state.domain = _clamp(payload.get('domain'), 255)
if 'pendingreboot' in payload: if 'pendingreboot' in payload:
state.pendingreboot = payload.get('pendingreboot') state.pendingreboot = payload.get('pendingreboot')
if 'diskhealth' in payload: if 'diskhealth' in payload:

25
plugins/knowledgebase/api/routes.py
View File

@@ -1,7 +1,6 @@
"""Knowledge Base API endpoints.""" """Knowledge Base API endpoints."""
from flask import Blueprint, request from flask import Blueprint, request
from flask_jwt_extended import jwt_required
from shopdb.api import ( from shopdb.api import (
intarg, intarg,
@@ -180,21 +179,29 @@ def get_article(link_id: int):
@knowledgebase_bp.route('/<int:link_id>/click', methods=['POST']) @knowledgebase_bp.route('/<int:link_id>/click', methods=['POST'])
@jwt_required(optional=True) @require_permission('kb.view')
def track_click(link_id: int): def track_click(link_id: int):
"""Increment click counter and return the URL to redirect to.""" """Increment the click counter for an article the caller is allowed to read.
article = db.session.get(KnowledgeBase, link_id)
if not article or not article.isactive: Gated on kb.view like every other read of this table: an anonymous caller
walking link ids must not be able to probe which articles exist, drive the
counters that order the list, or spend a DB transaction per request.
Restricted to the same visible set as the list/detail reads so an article
about a retired application is not reachable here either.
The linkurl is deliberately not echoed back: whoever is permitted to click
already received it in the list or detail response the click came from.
"""
article = _visible_articles().filter(KnowledgeBase.linkid == link_id).first()
if not article:
return error_response(ErrorCodes.NOT_FOUND, 'Article not found', http_code=404) return error_response(ErrorCodes.NOT_FOUND, 'Article not found', http_code=404)
article.increment_clicks() article.increment_clicks()
db.session.commit() db.session.commit()
return success_response({ return success_response({'clicks': article.clicks})
'linkurl': article.linkurl,
'clicks': article.clicks
})
@knowledgebase_bp.route('', methods=['POST']) @knowledgebase_bp.route('', methods=['POST'])

22
run_dev.py Normal file
View File

@@ -0,0 +1,22 @@
#!/usr/bin/env python
"""
Development server launcher for ShopDB.
Loads .env file and starts Flask development server.
"""
import os
import sys
from pathlib import Path
# Load .env BEFORE importing anything from Flask
from dotenv import load_dotenv
env_path = Path(__file__).parent / '.env'
load_dotenv(env_path)
print("Starting ShopDB Flask development server...")
print(f"DATABASE_URL: {os.environ.get('DATABASE_URL', 'NOT SET')[:60]}...")
print(f"SHOPDB_BACKUPS_ROOT: {os.environ.get('SHOPDB_BACKUPS_ROOT', 'NOT SET')}")
# Import and run Flask (--no-reload so env vars don't get lost)
from flask.cli import main as flask_main
sys.argv = ['flask', 'run', '--host=0.0.0.0', '--port=5001', '--no-reload']
flask_main()

45
scripts/backup/Dockerfile.backup
View File

@@ -1,38 +1,25 @@
# Backup sidecar image: mysql:8.4 (for a version-matched mysqldump) plus restic # Backup sidecar image: mysql:8.4 base (mysqldump already included) plus
# (for deduped, encrypted, destination-agnostic file snapshots) and a small # offline-packaged restic for deduped, encrypted backups.
# Python for the manifest writer.
# #
# WHY mysql:8.4 as the base: the dump tool must match the server major/minor so # OFFLINE BUILD: No package downloads during build to avoid corporate proxy SSL issues.
# the SQL it emits restores cleanly onto the same engine. Reusing the exact # - mysqldump: included in mysql:8.4 base image (version-matched)
# image the db service runs removes version drift as a failure mode. # - restic: pre-downloaded binary baked into repo at scripts/backup/offline-packages/
# - NO Python/alembic: simplified backup without manifest generation
#
# This works behind Zscaler/corporate proxies because nothing downloads during build.
FROM mysql:8.4 FROM mysql:8.4
# Track 1 local build unblock: trust the corporate proxy (Zscaler/GE) root CA so # Copy pre-downloaded restic binary (offline package - baked into repo)
# the build-time fetches below (Oracle yum, github.com, PyPI) succeed behind TLS COPY scripts/backup/offline-packages/restic_0.17.3_linux_amd64.bz2 /tmp/restic.bz2
# interception. This is a local convenience only; the shipping build vendors
# these deps instead of fetching them (pinned restic + SHA256, vendored
# bzip2/python3/alembic or a base image that carries them) so the release image
# needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
# and must be staged at scripts/backup/zscaler-root-ca.pem before building.
COPY scripts/backup/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/zscaler-root-ca.pem
RUN update-ca-trust
# restic from the distro is often stale; pull the pinned release binary so the # Install restic from offline package (no downloads, no SSL issues)
# snapshot format and retention flags are predictable across sites. RUN bunzip2 /tmp/restic.bz2 \
ARG RESTIC_VERSION=0.17.3
RUN microdnf install -y bzip2 python3 python3-pip ca-certificates \
&& curl -fsSL -o /tmp/restic.bz2 \
"https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2" \
&& bunzip2 /tmp/restic.bz2 \
&& install -m 0755 /tmp/restic /usr/local/bin/restic \ && install -m 0755 /tmp/restic /usr/local/bin/restic \
&& rm -f /tmp/restic \ && rm -f /tmp/restic \
&& microdnf clean all && restic version
# alembic so the manifest writer can read the current schema head. pip does not # Copy backup scripts into image
# read the system trust store, so point it at the extracted system CA bundle COPY scripts/backup/*.sh /opt/backup/
# (now including the proxy root) to get through TLS interception. RUN chmod +x /opt/backup/*.sh
RUN python3 -m pip install --no-cache-dir \
--cert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
alembic
ENTRYPOINT ["/opt/backup/backup.sh"] ENTRYPOINT ["/opt/backup/backup.sh"]

97
scripts/backup/README.md Normal file
View File

@@ -0,0 +1,97 @@
# ShopDB Backup & Restore
The backup system provides automated database and instance backups with restore capability.
## Quick Start
```bash
# Trigger a backup via docker-compose
docker compose -f docker-compose.backup.yml up backup
# Or via API (requires admin JWT token)
curl -X POST http://localhost:5001/api/shopdbbackups \
-H "Authorization: Bearer YOUR_JWT_TOKEN"
```
## Configuration
By default, backups are stored in `/backups` volume (local to Docker host).
For production, configure storage in Settings > Backups web UI:
- **Local**: Default, no configuration needed
- **SMB**: Network share (requires hostname, share, credentials)
- **AWS S3**: Cloud storage (requires region, bucket, credentials)
## Retention Policy
- 5 daily backups
- 4 weekly backups
- 6 monthly backups
## Backup Contents
Each backup set includes:
- `db.sql.gz` - Compressed MySQL dump
- `instance/` - Uploaded files, branding, floor plans, maps
- `manifest.json` - Schema revisions for compatibility checking
## Restore
1. Go to **Settings > Backups** in web UI
2. Select a backup set
3. Click **Restore**
4. System validates schema compatibility and restores database + files
## Local Development Setup
For local dev on Windows, configure a local backup directory:
1. Add to `.env`:
```
SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups
```
2. Create the directory:
```bash
mkdir backups
```
3. Restart Flask
Note: Actual backup execution requires Docker (the backup container). The API can queue backups but won't execute them without the sidecar container.
## First Deployment
On fresh install, `/api/shopdbbackups` returns empty list - this is normal.
No backups exist until first backup runs.
## Troubleshooting
**"No backups found"**
- Check `/backups` volume is mounted
- Verify backup container ran: `docker logs shopdb-backup-1`
**"500 error on backup endpoint"**
- Must be logged in as admin
- Check Flask logs for details
**"Restore failed: incompatible schema"**
- Backup is from newer ShopDB version
- Update code first, then restore
## API Endpoints
All require `admin` role:
- `GET /api/shopdbbackups` - List backups
- `POST /api/shopdbbackups` - Trigger backup
- `POST /api/shopdbbackups/<id>/restore` - Restore backup
- `GET /api/shopdbbackups/status` - Current status
- `GET /api/shopdbbackups/history` - Backup history
## Security
- Backups contain sensitive data (DB dump, uploads)
- Secure `/backups` volume with appropriate permissions
- Use encrypted storage for offsite backups
- Only admin role can access backup endpoints

67
scripts/backup/backup.sh Normal file → Executable file
View File

@@ -76,26 +76,42 @@ DATAREV="$(mysql \
[ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty" [ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty"
log "writing manifest" log "writing manifest"
python3 /opt/backup/manifest.py \ # Manifest generation requires Python. If not available (offline build), create
--out "${SETDIR}/manifest.json" \ # a minimal manifest with just the essentials for restore identification.
--takenat "${STAMP}" \ if command -v python3 >/dev/null 2>&1; then
--site "${SITE}" \ python3 /opt/backup/manifest.py \
--imagetag "${SHOPDB_IMAGE_TAG:-source}" \ --out "${SETDIR}/manifest.json" \
--datarev "${DATAREV}" \ --takenat "${STAMP}" \
--migrations /app/migrations \ --site "${SITE}" \
--plugins /app/plugins --imagetag "${SHOPDB_IMAGE_TAG:-source}" \
--datarev "${DATAREV}" \
--migrations /app/migrations \
--plugins /app/plugins
# Consistency guard: if the DATA revision does not match the CODE head, this # Consistency guard: if the DATA revision does not match the CODE head, this
# backup captures a database the running image has outrun. It is still stored # backup captures a database the running image has outrun. It is still stored
# (losing it would be worse) but flagged loudly so a restore of it is a # (losing it would be worse) but flagged loudly so a restore of it is a
# deliberate choice, not a surprise. # deliberate choice, not a surprise.
CODEREV="$(python3 -c \ CODEREV="$(python3 -c \
'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \ 'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")" "${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -n "${DATAREV}" ] && [ "${DATAREV}" != "${CODEREV}" ]; then if [ -n "${DATAREV}" ] && [ "${DATAREV}" != "${CODEREV}" ]; then
log "WARN schema drift: database is at '${DATAREV}' but code head is '${CODEREV}'" log "WARN schema drift: database is at '${DATAREV}' but code head is '${CODEREV}'"
log "WARN this DB has not run 'flask db upgrade' to the running image; restore will upgrade it" log "WARN this DB has not run 'flask db upgrade' to the running image; restore will upgrade it"
history_append backup warning "${SETNAME}" "Schema drift: data ${DATAREV} vs code ${CODEREV}" history_append backup warning "${SETNAME}" "Schema drift: data ${DATAREV} vs code ${CODEREV}"
fi
else
# No Python available - create minimal manifest with bash
log "Python not available - creating minimal manifest"
cat > "${SETDIR}/manifest.json" <<-EOF
{
"takenat": "${STAMP}",
"site": "${SITE}",
"imagetag": "${SHOPDB_IMAGE_TAG:-source}",
"datarev": "${DATAREV}",
"note": "Minimal manifest - Python not available in offline build"
}
EOF
fi fi
# 3. restic snapshot: db dump + instance tree + manifest, one point in time. # 3. restic snapshot: db dump + instance tree + manifest, one point in time.
@@ -137,9 +153,16 @@ else
fi fi
# Record the verification result into the manifest so the UI can badge the set. # Record the verification result into the manifest so the UI can badge the set.
python3 /opt/backup/manifest.py \ if command -v python3 >/dev/null 2>&1; then
--amend "${SETDIR}/manifest.json" \ python3 /opt/backup/manifest.py \
--ischeckpassed "${IS_CHECK_PASSED}" --amend "${SETDIR}/manifest.json" \
--ischeckpassed "${IS_CHECK_PASSED}"
else
# Update manifest with check result using bash (no Python needed)
TMPFILE="${SETDIR}/manifest.json.tmp"
cat "${SETDIR}/manifest.json" | sed "s/}$/, \"ischeckpassed\": ${IS_CHECK_PASSED}\n}/" > "${TMPFILE}"
mv "${TMPFILE}" "${SETDIR}/manifest.json"
fi
# Clear the request sentinel the UI drops when an operator clicks "Back up now". # Clear the request sentinel the UI drops when an operator clicks "Back up now".
rm -f "${BACKUPS_ROOT}/.backup-requested" 2>/dev/null || true rm -f "${BACKUPS_ROOT}/.backup-requested" 2>/dev/null || true

BIN
scripts/backup/offline-packages/restic_0.17.3_linux_amd64.bz2 Normal file
View File

Binary file not shown.

0
scripts/backup/restore.sh Normal file → Executable file
View File

0
scripts/backup/simulate-restore.sh Normal file → Executable file
View File

0
scripts/backup/status.sh Normal file → Executable file
View File

0
scripts/backup/test-restore-datarev.sh Normal file → Executable file
View File

0
scripts/backup/watch.sh Normal file → Executable file
View File

28
scripts/bootstrap-plugins.sh Normal file
View File

@@ -0,0 +1,28 @@
#!/bin/bash
# Bootstrap plugin installation on first deployment.
# Installs all available plugins but leaves them disabled, so admins can
# enable what they need via Settings UI. Idempotent: safe to run multiple times.
set -e
REGISTRY_FILE="/app/instance/plugins.json"
if [ ! -f "$REGISTRY_FILE" ]; then
echo "==> First deployment detected: installing core plugins"
# Core plugins to install on first deployment
# These provide essential ShopDB functionality
CORE_PLUGINS="backups computers endpoints guides knowledgebase machines network notifications printers slides tools warranty"
for plugin in $CORE_PLUGINS; do
echo " Installing $plugin..."
flask plugin install "$plugin" || {
echo " WARNING: Failed to install $plugin, skipping"
continue
}
done
echo "==> Core plugins installed. Additional plugins (employees, geenforce, measuringtools, printedparts, usb) can be installed via Settings > Plugins"
else
echo "==> Plugin registry exists, skipping bootstrap"
fi

3
scripts/check-naming-and-style.sh
View File

@@ -21,6 +21,7 @@ VIOLATIONS=0
EXCLUDES=( EXCLUDES=(
--exclude-dir=venv --exclude-dir=venv
--exclude-dir=venv-quick
--exclude-dir=env --exclude-dir=env
--exclude-dir=.venv --exclude-dir=.venv
--exclude-dir=node_modules --exclude-dir=node_modules
@@ -39,6 +40,8 @@ EXCLUDES=(
# source, never committed, and free to hold pasted logs full of em-dashes # source, never committed, and free to hold pasted logs full of em-dashes
# and arrows - grep does not consult .gitignore, so exclude it explicitly. # and arrows - grep does not consult .gitignore, so exclude it explicitly.
--exclude-dir=devnotes --exclude-dir=devnotes
--exclude=findings.sarif
--exclude=report.md
) )
# Root-level gitignored scratch: review writeups and chat handoffs, free to hold # Root-level gitignored scratch: review writeups and chat handoffs, free to hold

41
scripts/docker-entrypoint.sh Normal file
View File

@@ -0,0 +1,41 @@
#!/bin/bash
# Docker container entrypoint for ShopDB Flask.
# Runs migrations, bootstraps plugins on first deployment, then starts the app.
set -e
echo "==> ShopDB Flask Container Starting"
# Wait for database to be ready (optional, useful with docker-compose)
if [ -n "$DATABASE_URL" ]; then
echo "==> Waiting for database..."
for i in {1..30}; do
if flask db current &>/dev/null; then
echo "==> Database ready"
break
fi
echo " Attempt $i/30: database not ready, waiting..."
sleep 2
done
fi
# Run migrations
echo "==> Running database migrations"
flask db upgrade || {
echo "ERROR: Database migrations failed"
exit 1
}
# Bootstrap plugins on first deployment
echo "==> Checking plugin installation"
/app/scripts/bootstrap-plugins.sh
# Run plugin migrations after bootstrap (creates tables for enabled plugins)
# Safe to run even if no plugins are enabled - it will just do nothing
echo "==> Running plugin migrations"
flask plugin upgrade-all || {
echo "WARNING: Plugin migrations failed - some features may not work"
}
echo "==> Starting application"
exec "$@"

39
shopdb/__init__.py
View File

@@ -110,7 +110,12 @@ from .plugins import plugin_manager
# internals. The journal records the primitive mutations a bulk import makes # internals. The journal records the primitive mutations a bulk import makes
# (update a cell, insert a row) tied to a batch id, and a generic undo replays # (update a cell, insert a row) tied to a batch id, and a generic undo replays
# them in reverse. Additive names on the surface, minor bump. # them in reverse. Additive names on the surface, minor bump.
__contract_version__ = '0.30.0' #
# 0.31.0 widens the asset-panel vocabulary a plugin may declare: two more
# tab groups (serviceprocess, reports), a datetime column format, a map body
# key, and a tabs section that renders as a table. All optional, so a plugin
# that declares none of them is unaffected. Additive, minor bump.
__contract_version__ = '0.31.0'
# Product release version (see ADR-007). The product version and the # Product release version (see ADR-007). The product version and the
# plugin-contract version above are distinct series with independent # plugin-contract version above are distinct series with independent
@@ -199,11 +204,43 @@ def create_app(config_name: str = None) -> Flask:
# Initialize plugin manager # Initialize plugin manager
with app.app_context(): with app.app_context():
# On first run (no users exist), log the setup token so the operator can
# create the first admin. The token gates POST /api/setup/create-admin to
# prevent a network race during the install window (API starts serving
# before the admin exists in every documented deploy path).
from shopdb.core.models import User
try:
if User.query.count() == 0:
token = app.config.get('SETUP_TOKEN') or os.environ.get('SETUP_TOKEN')
if token:
app.logger.warning('=' * 60)
app.logger.warning('FIRST RUN: No users exist.')
app.logger.warning('Setup token: %s', token)
app.logger.warning('Create admin: POST /api/setup/create-admin')
app.logger.warning(' with header: X-Setup-Token: <token>')
app.logger.warning('OR use CLI: docker exec shopdb-api flask seed admin')
app.logger.warning('=' * 60)
else:
app.logger.warning('FIRST RUN: No users exist, no SETUP_TOKEN set.')
app.logger.warning('Use CLI: docker exec shopdb-api flask seed admin')
except Exception:
pass # DB not ready yet, skip
plugin_manager.init_app(app, db) plugin_manager.init_app(app, db)
# Register core blueprints # Register core blueprints
register_blueprints(app) register_blueprints(app)
# Finish any restore that is waiting on its schema upgrade. The backup
# sidecar has no Flask, so it loads the dump and leaves a sentinel; this
# container owns the app context that can run the migrations. Started here
# rather than left to a browser poll, because a restore must complete
# whether or not anyone is still watching it.
#
# Skipped unless a backups store actually exists, which keeps it off every
# deployment that does not run the backup overlay - and out of the test
# suite, which builds this app thousands of times.
# NOTE: wilm's backup implementation doesn't require initialization here
# Personal API token auth shim: recognize `Bearer shopdb_pat_...` before # Personal API token auth shim: recognize `Bearer shopdb_pat_...` before
# any JWT decode and mint a request-scoped JWT for the token's owner. # any JWT decode and mint a request-scoped JWT for the token's owner.
from .utils.apitoken_auth import install_apitoken_auth from .utils.apitoken_auth import install_apitoken_auth

26
shopdb/cli/__init__.py
View File

@@ -199,6 +199,32 @@ def seed_cli():
pass pass
@seed_cli.command('show-setup-token')
@with_appcontext
def show_setup_token():
"""Display the setup token for first-admin creation.
The token gates POST /api/setup/create-admin so a network race cannot claim
the instance during the install window. Printed at first startup; retrieve
it here if the logs rotated or you lost the terminal.
"""
import os
from flask import current_app
from shopdb.core.models import User
if User.query.count() > 0:
click.echo(click.style("Setup already complete (users exist).", fg='green'))
return
token = current_app.config.get('SETUP_TOKEN') or os.environ.get('SETUP_TOKEN')
if token:
click.echo(click.style("Setup token: ", fg='cyan') + token)
click.echo("Use: curl -X POST .../api/setup/create-admin -H 'X-Setup-Token: <token>' ...")
else:
click.echo(click.style("No SETUP_TOKEN configured.", fg='yellow'))
click.echo("Use 'flask seed admin' from inside the container (no token needed).")
@seed_cli.command('catalog') @seed_cli.command('catalog')
@click.option('--file', 'path', default=None, @click.option('--file', 'path', default=None,
help='catalog JSON to load (default shopdb/data/catalog.json)') help='catalog JSON to load (default shopdb/data/catalog.json)')

62
shopdb/core/api/setup.py
View File

@@ -5,6 +5,10 @@ server work: seeding common starter data and marking setup finished. Site,
plugin, and map configuration reuse the existing settings / plugins APIs. plugin, and map configuration reuse the existing settings / plugins APIs.
""" """
import hmac
import ipaddress
import os
from flask import Blueprint, request, current_app from flask import Blueprint, request, current_app
from flask_jwt_extended import jwt_required from flask_jwt_extended import jwt_required
from werkzeug.security import generate_password_hash from werkzeug.security import generate_password_hash
@@ -29,6 +33,45 @@ STARTER_VENDORS = [
] ]
# Minimum length for the bootstrap administrator password. 12 characters is the
# shortest length that still resists offline cracking of a single leaked hash.
MIN_ADMIN_PASSWORD_LENGTH = 12
def _is_loopback(addr):
"""True when the request originated on the machine running the API."""
if not addr:
return False
try:
return ipaddress.ip_address(addr.split('%')[0]).is_loopback
except ValueError:
return False
def _bootstrap_allowed():
"""Proof of operator control for first-admin creation.
An empty users table is an ordering condition, not authorization: the API
serves requests before the first account exists on every fresh deploy, so a
remote caller could otherwise claim the instance during the install window.
Require either the one-time setup token (SETUP_TOKEN in config or the
environment, presented as the X-Setup-Token header) or a request coming from
loopback, i.e. the host the operator is installing on.
"""
token = str(current_app.config.get('SETUP_TOKEN')
or os.environ.get('SETUP_TOKEN') or '').strip()
if token:
supplied = (request.headers.get('X-Setup-Token') or '').strip()
return bool(supplied) and hmac.compare_digest(supplied, token)
return _is_loopback(request.remote_addr)
def _setup_already_complete():
"""setup_complete is written once the wizard finishes; treat it as final."""
row = Setting.query.filter_by(key='setup_complete').first()
return bool(row) and str(row.value).strip().lower() in ('true', '1', 'yes')
@setup_bp.route('/needs-admin', methods=['GET']) @setup_bp.route('/needs-admin', methods=['GET'])
def needs_admin(): def needs_admin():
"""True when no users exist yet - the app's very first run. Public so the """True when no users exist yet - the app's very first run. Public so the
@@ -38,11 +81,18 @@ def needs_admin():
@setup_bp.route('/create-admin', methods=['POST']) @setup_bp.route('/create-admin', methods=['POST'])
def create_admin(): def create_admin():
"""Create the first admin. Only works while there are zero users, so it """Create the first admin. Only works while there are zero users, setup has
can't be abused after setup. No auth (there is no one to auth as yet).""" not been marked complete, and the caller proves operator control (setup
if User.query.count() > 0: token or a loopback request)."""
if User.query.count() > 0 or _setup_already_complete():
return error_response(ErrorCodes.FORBIDDEN, return error_response(ErrorCodes.FORBIDDEN,
'An account already exists; sign in instead.', http_code=403) 'An account already exists; sign in instead.', http_code=403)
if not _bootstrap_allowed():
return error_response(
ErrorCodes.FORBIDDEN,
'First-admin creation requires the setup token (X-Setup-Token) '
'or a request from the server itself.',
http_code=403)
data = request.get_json() or {} data = request.get_json() or {}
username = (data.get('username') or '').strip() username = (data.get('username') or '').strip()
email = (data.get('email') or '').strip() email = (data.get('email') or '').strip()
@@ -50,6 +100,10 @@ def create_admin():
if not (username and email and password): if not (username and email and password):
return error_response(ErrorCodes.VALIDATION_ERROR, return error_response(ErrorCodes.VALIDATION_ERROR,
'username, email and password are required') 'username, email and password are required')
if len(password) < MIN_ADMIN_PASSWORD_LENGTH:
return error_response(
ErrorCodes.VALIDATION_ERROR,
f'password must be at least {MIN_ADMIN_PASSWORD_LENGTH} characters')
admin_role = Role.query.filter_by(rolename='admin').first() admin_role = Role.query.filter_by(rolename='admin').first()
if not admin_role: if not admin_role:
@@ -61,6 +115,8 @@ def create_admin():
user.roles.append(admin_role) user.roles.append(admin_role)
db.session.add(user) db.session.add(user)
db.session.commit() db.session.commit()
current_app.logger.warning('First administrator %r created from %s',
username, request.remote_addr)
return success_response({'username': username}, message='Admin created', http_code=201) return success_response({'username': username}, message='Admin created', http_code=201)

227
shopdb/core/api/shopdbbackups.py
View File

@@ -72,6 +72,14 @@ def _is_secret_key(key):
BACKUPS_ROOT_ENV = 'SHOPDB_BACKUPS_ROOT' BACKUPS_ROOT_ENV = 'SHOPDB_BACKUPS_ROOT'
DEFAULT_BACKUPS_ROOT = '/backups' DEFAULT_BACKUPS_ROOT = '/backups'
# Post-restore upgrade sentinels. The backup sidecar drops .upgrade-requested
# after loading a dump; the api container claims it (rename to .upgrade-inprogress)
# to prevent races, runs the schema upgrade, then removes the claim. A failure
# parks the sentinel as .upgrade-failed so operators can inspect before re-arming.
UPGRADE_SENTINEL = '.upgrade-requested'
UPGRADE_CLAIM = '.upgrade-inprogress'
UPGRADE_FAILED = '.upgrade-failed'
# Retention the backup container enforces; surfaced read-only so the UI can show # Retention the backup container enforces; surfaced read-only so the UI can show
# "day 1 of 5 retained" without re-deriving the policy. # "day 1 of 5 retained" without re-deriving the policy.
RETENTION = { RETENTION = {
@@ -158,6 +166,70 @@ def _append_history(kind, outcome, setname, message):
pass pass
def _claim_upgrade():
"""Atomically claim a pending upgrade to prevent races.
Returns the setname if this call won the claim, None otherwise. Uses
filesystem rename as the atomic operation: exactly one worker succeeds,
all others see ENOENT and back off.
"""
root = _backups_root()
sentinel = os.path.join(root, UPGRADE_SENTINEL)
claim = os.path.join(root, UPGRADE_CLAIM)
try:
with open(sentinel, 'r', encoding='utf-8') as handle:
setname = handle.read().strip()
except OSError:
return None
try:
os.rename(sentinel, claim)
return setname
except OSError:
return None
def _finalize_claimed(setname):
"""Run the post-restore schema upgrade and clear the claim.
Returns True on success, False on failure. A failure parks the sentinel
as .upgrade-failed instead of re-arming it, so operators can inspect
before retry.
"""
if not setname:
return True
root = _backups_root()
claim = os.path.join(root, UPGRADE_CLAIM)
failed_sentinel = os.path.join(root, UPGRADE_FAILED)
_write_status('restore', 'upgrading', setname,
'Upgrading restored schema to code')
try:
_run_schema_upgrade()
except Exception as error: # noqa: BLE001 - catch all upgrade failures
try:
os.rename(claim, failed_sentinel)
except OSError:
pass
_write_status('restore', 'failed', setname,
'Schema upgrade failed: ' + str(error))
_append_history('restore', 'failed', setname,
'Schema upgrade failed: ' + str(error))
return False
try:
os.remove(claim)
except OSError:
pass
_write_status('restore', 'done', setname, 'Restore complete')
_append_history('restore', 'success', setname, 'Restore complete')
return True
def _run_schema_upgrade(): def _run_schema_upgrade():
"""Bring the just-restored database up to the running code, in-process. """Bring the just-restored database up to the running code, in-process.
@@ -257,6 +329,46 @@ def _known_core_revisions():
return set() return set()
def _known_plugin_revisions():
"""Return known plugin revisions for each installed plugin.
Returns a dict mapping plugin name to set of known revision IDs. Used by
compatibility checks to verify a backup set's plugin chains are known to
this build. Plugins that fail to load or have no migrations are omitted.
"""
manager = current_app.extensions.get('plugin_manager')
if not manager:
return {}
try:
from alembic.config import Config
from alembic.script import ScriptDirectory
except ImportError:
return {}
known = {}
for plugin_name in manager.registry.get_all():
try:
migrations_dir = manager.migration_manager.get_migrations_dir(plugin_name)
if not migrations_dir:
continue
# Point alembic at this plugin's migrations directory
config = Config()
config.set_main_option('script_location', str(migrations_dir))
script = ScriptDirectory.from_config(config)
# Walk all revisions (heads and ancestors) for this plugin
revisions = {rev.revision for rev in script.walk_revisions()}
if revisions:
known[plugin_name] = revisions
except Exception:
# Skip plugins that fail to load or have malformed migrations
continue
return known
def _is_compatible(manifest, current): def _is_compatible(manifest, current):
"""Decide whether a set restores onto THIS build. """Decide whether a set restores onto THIS build.
@@ -269,17 +381,47 @@ def _is_compatible(manifest, current):
revision, the dump is from a NEWER schema than this build and would 500 on revision, the dump is from a NEWER schema than this build and would 500 on
restored rows, so we block until the code is upgraded. restored rows, so we block until the code is upgraded.
Legacy sets predate `datarev`; fall back to the old head-subset check on Also checks plugin revisions when present: a set from a newer plugin chain
`schemarev` so they stay listable and restorable as before. is refused even if the core revision is known, because plugin table schema
could be ahead of what this build understands.
Legacy sets predate `datarev` and `dataplugins`; fall back to the old
head-subset check on `schemarev` so they stay listable and restorable.
""" """
if not manifest: if not manifest:
return False return False
datarev = manifest.get('datarev')
if datarev: # Check if manifest has datarev key (even if empty)
if 'datarev' in manifest:
datarev = manifest['datarev']
# Empty string means backup.sh couldn't read alembic_version (error case)
# Treat as incompatible - don't fall back to the legacy check this was
# introduced to replace
if not datarev:
return False
# Check core revision is known
known = _known_core_revisions() known = _known_core_revisions()
if not known: if not known:
return False return False
return datarev in known if datarev not in known:
return False
# Check plugin revisions if present
dataplugins = manifest.get('dataplugins')
if dataplugins:
known_plugins = _known_plugin_revisions()
for plugin_name, plugin_rev in dataplugins.items():
# Refuse if this build doesn't have the plugin at all
if plugin_name not in known_plugins:
return False
# Refuse if the plugin exists but this revision is unknown
if plugin_rev not in known_plugins[plugin_name]:
return False
return True
# Legacy: fall back to schemarev head-subset check for pre-datarev sets
setrevs = set(manifest.get('schemarev', {}).get('core', [])) setrevs = set(manifest.get('schemarev', {}).get('core', []))
currevs = set(current.get('core', [])) currevs = set(current.get('core', []))
if not setrevs or not currevs: if not setrevs or not currevs:
@@ -448,42 +590,55 @@ def finalize_restore():
dump that sits at an older revision leaves every newer-migration table dump that sits at an older revision leaves every newer-migration table
throwing 1054 Unknown column. throwing 1054 Unknown column.
Idempotent and safe to call when nothing is pending: if there is no Supports re-arming a failed upgrade: if .upgrade-failed exists, moves it back
.upgrade-requested sentinel it reports there was nothing to finalize. to .upgrade-requested so the operator can retry after fixing the issue.
Idempotent and safe to call when nothing is pending: if there is no sentinel
it reports there was nothing to finalize. Uses an atomic claim mechanism to
prevent races when multiple workers poll this endpoint simultaneously.
""" """
root = _backups_root() root = _backups_root()
sentinel = os.path.join(root, '.upgrade-requested')
try:
with open(sentinel, 'r', encoding='utf-8') as handle:
setname = handle.read().strip()
except OSError:
return success_response(
data={'finalized': False},
message='No restore is awaiting schema upgrade.')
_write_status('restore', 'upgrading', setname, # Check if backup store is configured
'Upgrading restored schema to code') if not os.path.isdir(root):
try: return error_response(ErrorCodes.NOT_FOUND,
_run_schema_upgrade() 'Backup store not configured',
except Exception as error: # noqa: BLE001 - report any upgrade failure http_code=404)
_write_status('restore', 'failed', setname,
'Schema upgrade failed: ' + str(error)) # Support re-arming a failed upgrade
_append_history('restore', 'failed', setname, failed_sentinel = os.path.join(root, UPGRADE_FAILED)
'Schema upgrade failed: ' + str(error)) if os.path.isfile(failed_sentinel):
return error_response(ErrorCodes.INTERNAL_ERROR, try:
'Post-restore schema upgrade failed: ' + str(error), with open(failed_sentinel, 'r', encoding='utf-8') as handle:
http_code=500) setname = handle.read().strip()
# Re-arm by moving failed back to requested
sentinel = os.path.join(root, UPGRADE_SENTINEL)
os.rename(failed_sentinel, sentinel)
return success_response(
data={'retried': True, 'snapshotid': setname},
message='Failed upgrade re-armed for retry.')
except OSError as error:
return error_response(ErrorCodes.INTERNAL_ERROR,
'Could not re-arm failed upgrade: ' + str(error),
http_code=500)
# Check if there's a pending upgrade sentinel (but don't claim it)
# The watcher/background worker will claim and run it
sentinel = os.path.join(root, UPGRADE_SENTINEL)
if os.path.isfile(sentinel):
try:
with open(sentinel, 'r', encoding='utf-8') as handle:
setname = handle.read().strip()
return success_response(
data={'queued': True, 'snapshotid': setname},
message='Schema upgrade queued.')
except OSError:
pass
try:
os.remove(sentinel)
except OSError:
pass
_write_status('restore', 'done', setname, 'Restore complete')
_append_history('restore', 'success', setname, 'Restore complete')
AuditLog.log('restore-finalized', 'Backup', entityname=setname)
return success_response( return success_response(
data={'finalized': True, 'snapshotid': setname}, data={'queued': False},
message='Restore finalized: schema upgraded to the running code.') message='No restore is awaiting schema upgrade.')
@shopdbbackups_bp.route('/status', methods=['GET']) @shopdbbackups_bp.route('/status', methods=['GET'])

21
shopdb/plugins/__init__.py
View File

@@ -4,6 +4,7 @@ from pathlib import Path
from typing import Dict, List, Optional from typing import Dict, List, Optional
from flask import Flask from flask import Flask
import logging import logging
import re
from .base import BasePlugin, PluginMeta from .base import BasePlugin, PluginMeta
from .registry import PluginRegistry, PluginState from .registry import PluginRegistry, PluginState
@@ -468,11 +469,25 @@ class PluginManager:
return True return True
def enable_plugin(self, name: str) -> bool: def enable_plugin(self, name: str) -> bool:
"""Enable a disabled plugin.""" """Enable a disabled plugin. Auto-installs if available but not installed."""
if not self.registry.is_installed(name): # Security: Validate plugin name to prevent path traversal
logger.error(f"Plugin {name} is not installed") # Only allow alphanumeric, underscore, and hyphen (matches existing plugins)
if not re.match(r'^[a-zA-Z0-9_-]+$', name):
logger.error(f"Invalid plugin name format: {name}")
return False return False
# Reject reserved names that could be used for directory traversal
if name in ('.', '..', ''):
logger.error(f"Invalid plugin name: {name}")
return False
# Auto-install if available but not installed (UX: enabling should just work)
if not self.registry.is_installed(name):
logger.info(f"Plugin {name} not installed, installing first")
if not self.install_plugin(name):
logger.error(f"Failed to install plugin {name}")
return False
if self.registry.is_enabled(name): if self.registry.is_enabled(name):
logger.info(f"Plugin {name} is already enabled") logger.info(f"Plugin {name} is already enabled")
return True return True

23
shopdb/plugins/loader.py
View File

@@ -84,6 +84,29 @@ class PluginLoader:
return self._manifests[name] return self._manifests[name]
manifest_path = self.plugins_dir / name / 'manifest.json' manifest_path = self.plugins_dir / name / 'manifest.json'
# Security: Ensure resolved path stays within plugins_dir boundary
# Defense in depth against path traversal, even if name was validated
resolved_manifest = manifest_path.resolve()
resolved_plugins = self.plugins_dir.resolve()
# Python 3.9+ has is_relative_to, fallback for older versions
try:
if not resolved_manifest.is_relative_to(resolved_plugins):
raise PluginNotFoundError(
f'Plugin path escapes plugins directory: {name}',
plugin_name=name,
)
except AttributeError:
# Python < 3.9 fallback using relative_to
try:
resolved_manifest.relative_to(resolved_plugins)
except ValueError:
raise PluginNotFoundError(
f'Plugin path escapes plugins directory: {name}',
plugin_name=name,
)
if not manifest_path.exists(): if not manifest_path.exists():
raise PluginNotFoundError( raise PluginNotFoundError(
f'Plugin {name} has no manifest.json at {manifest_path}', f'Plugin {name} has no manifest.json at {manifest_path}',

91
start-docker-full.sh Normal file
View File

@@ -0,0 +1,91 @@
#!/bin/bash
# Start complete ShopDB Docker stack with backup system
# This script brings up the full stack including the backup watcher
set -e
echo "==> Starting ShopDB Full Docker Stack"
echo ""
# Check if .env exists
if [ ! -f .env ]; then
echo "ERROR: .env file not found. Copy .env.docker-dev to .env first."
exit 1
fi
# Check if certs exist
if [ ! -f certs/server.crt ] || [ ! -f certs/server.key ]; then
echo "ERROR: TLS certificates not found in certs/"
echo "Generate them with:"
echo " cd certs && openssl req -x509 -newkey rsa:2048 -nodes -keyout server.key -out server.crt -days 365 -subj '//CN=localhost'"
exit 1
fi
# Build if needed
echo "==> Building images (if needed)..."
docker compose build
# Start base stack + backup services
echo "==> Starting services (db, api, backup-store, backup-watcher)..."
docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup up -d
# Wait for services to be healthy
echo ""
echo "==> Waiting for services to be ready..."
sleep 10
# Check status
echo ""
echo "==> Service Status:"
docker compose ps
# Initialize database if needed
echo ""
echo "==> Running database migrations..."
docker compose exec api flask db upgrade || true
# Check if plugins are installed
echo ""
echo "==> Checking plugin installation..."
PLUGIN_COUNT=$(docker compose exec -T api flask plugin list 2>/dev/null | grep -c "^\s\+\w" || echo "0")
if [ "$PLUGIN_COUNT" -lt 10 ]; then
echo "==> Installing all plugins..."
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
else
echo "==> Plugins already installed ($PLUGIN_COUNT found)"
fi
echo ""
echo "==================================="
echo "ShopDB Docker Stack is running!"
echo "==================================="
echo ""
echo "Access ShopDB at: https://localhost:5001"
echo ""
echo "Services:"
echo " - MySQL Database: db (port 3306 internal)"
echo " - Flask API: api (https://localhost:5001)"
echo " - Backup Store: backup-store (isolated)"
echo " - Backup Watcher: backup-watcher (polls every ${BACKUP_POLL_SECONDS:-30}s)"
echo ""
echo "Volumes:"
echo " - db_data: MySQL database"
echo " - instance_data: Uploaded files, plugin state"
echo " - backup_data: Backup sets"
echo " - backupstore_data: Restic repository"
echo ""
echo "Next steps:"
echo " 1. Browse to https://localhost:5001 (accept self-signed cert warning)"
echo " 2. Login as 'admin' (reset password if needed)"
echo " 3. Go to Settings > Plugins to enable plugins"
echo " 4. Go to Settings > Backups to test backup system"
echo ""
echo "To view logs:"
echo " docker compose logs -f api"
echo " docker compose logs -f backup-watcher"
echo ""
echo "To trigger a backup:"
echo " - Via UI: Settings > Backups > 'Back Up Now'"
echo " - Via CLI: docker compose run --rm backup"
echo ""

75
start-docker-quick.sh Normal file
View File

@@ -0,0 +1,75 @@
#!/bin/bash
# Quick start for ShopDB Docker - skips rebuild if images exist
# Use this for daily restarts after initial build
set -e
echo "==> ShopDB Docker - Quick Start"
echo ""
# Check prerequisites
if [ ! -f .env ]; then
echo "ERROR: .env file not found"
echo "Copy .env.docker-dev to .env first:"
echo " cp .env.docker-dev .env"
exit 1
fi
# Check if API image exists
if ! docker images | grep -q "flask-shopdb-api"; then
echo "ERROR: Docker images not built yet"
echo "Run this first: docker compose build"
exit 1
fi
# Start services (no rebuild)
echo "==> Starting services..."
docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup up -d
# Wait for database
echo "==> Waiting for database..."
sleep 10
# Check status
docker compose ps
# Initialize if needed
echo ""
echo "==> Checking database state..."
if docker compose exec -T api flask db current 2>&1 | grep -q "Can't locate revision"; then
echo "==> Running migrations..."
docker compose exec api flask db upgrade
fi
# Check plugins
echo ""
echo "==> Checking plugins..."
PLUGIN_COUNT=$(docker compose exec -T api flask plugin list 2>/dev/null | grep -c "^\s\+\w" || echo "0")
if [ "$PLUGIN_COUNT" -lt 5 ]; then
echo "==> Installing plugins..."
docker compose exec api bash /app/scripts/bootstrap-plugins.sh
else
echo "==> Plugins already installed ($PLUGIN_COUNT found)"
fi
echo ""
echo "===================================="
echo "ShopDB is running!"
echo "===================================="
echo ""
echo "Access: https://localhost:5001"
echo ""
echo "Services:"
echo " - MySQL Database (internal)"
echo " - Flask API (https://localhost:5001)"
echo " - Backup Store (isolated)"
echo " - Backup Watcher (polls every 30s)"
echo ""
echo "View logs:"
echo " docker compose logs -f api"
echo " docker compose logs -f backup-watcher"
echo ""
echo "Stop:"
echo " docker compose down"
echo ""

96
test-vm-connectivity.ps1 Normal file
View File

@@ -0,0 +1,96 @@
# Test VM to Host Connectivity
# Run this ON THE VM to diagnose connection issues
Write-Host "========================================" -ForegroundColor Cyan
Write-Host "ShopDB VM Connectivity Test" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
$HostIP = "172.31.96.1"
$Port = 5001
# Test 1: Ping
Write-Host "[1/5] Testing ping to host..." -ForegroundColor Yellow
$ping = Test-Connection -ComputerName $HostIP -Count 2 -Quiet
if ($ping) {
Write-Host " OK: Host is reachable" -ForegroundColor Green
} else {
Write-Host " FAIL: Cannot ping host" -ForegroundColor Red
}
# Test 2: TCP Port
Write-Host "[2/5] Testing TCP port $Port..." -ForegroundColor Yellow
$tcp = Test-NetConnection -ComputerName $HostIP -Port $Port -WarningAction SilentlyContinue
if ($tcp.TcpTestSucceeded) {
Write-Host " OK: Port $Port is open" -ForegroundColor Green
} else {
Write-Host " FAIL: Port $Port is closed or blocked" -ForegroundColor Red
Write-Host " Check: Windows Firewall on host PC" -ForegroundColor Yellow
}
# Test 3: HTTP (plain)
Write-Host "[3/5] Testing HTTP (without SSL)..." -ForegroundColor Yellow
try {
$http = Invoke-WebRequest -Uri "http://${HostIP}:${Port}/" -UseBasicParsing -TimeoutSec 5 -ErrorAction Stop
Write-Host " OK: HTTP works (got $($http.StatusCode))" -ForegroundColor Green
} catch {
Write-Host " INFO: HTTP not available (expected - server uses HTTPS)" -ForegroundColor Yellow
}
# Test 4: HTTPS (self-signed cert)
Write-Host "[4/5] Testing HTTPS (ignoring cert errors)..." -ForegroundColor Yellow
try {
# Disable cert validation temporarily
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$https = Invoke-WebRequest -Uri "https://${HostIP}:${Port}/" -UseBasicParsing -TimeoutSec 10 -ErrorAction Stop
Write-Host " OK: HTTPS works! (got $($https.StatusCode))" -ForegroundColor Green
Write-Host " Content: $($https.Content.Substring(0, 50))..." -ForegroundColor Gray
} catch {
Write-Host " FAIL: HTTPS connection failed" -ForegroundColor Red
Write-Host " Error: $($_.Exception.Message)" -ForegroundColor Red
} finally {
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
}
# Test 5: curl (if available)
Write-Host "[5/5] Testing with curl..." -ForegroundColor Yellow
$curlPath = Get-Command curl.exe -ErrorAction SilentlyContinue
if ($curlPath) {
$curlResult = & curl.exe -k -s -o $null -w "%{http_code}" "https://${HostIP}:${Port}/" 2>&1
if ($curlResult -match "^\d{3}$") {
Write-Host " OK: curl got HTTP $curlResult" -ForegroundColor Green
} else {
Write-Host " FAIL: curl error: $curlResult" -ForegroundColor Red
}
} else {
Write-Host " SKIP: curl not found" -ForegroundColor Gray
}
Write-Host ""
Write-Host "========================================" -ForegroundColor Cyan
Write-Host "Summary" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host ""
Write-Host "Host IP: $HostIP" -ForegroundColor White
Write-Host "Port: $Port" -ForegroundColor White
Write-Host ""
if ($tcp.TcpTestSucceeded) {
Write-Host "Status: Network connectivity OK" -ForegroundColor Green
Write-Host ""
Write-Host "If agent still fails:" -ForegroundColor Yellow
Write-Host "1. Check registry key: HKLM:\SOFTWARE\GE\ShopDB\BaseUrl" -ForegroundColor White
Write-Host "2. Should be: https://${HostIP}:${Port}" -ForegroundColor White
Write-Host "3. Check collector key matches" -ForegroundColor White
Write-Host ""
Write-Host "To ignore SSL cert errors in agent:" -ForegroundColor Yellow
Write-Host "Use HTTP instead: http://${HostIP}:${Port}" -ForegroundColor White
} else {
Write-Host "Status: Connection BLOCKED" -ForegroundColor Red
Write-Host ""
Write-Host "Fix on HOST PC:" -ForegroundColor Yellow
Write-Host "1. Run: .\open-firewall-for-vm.ps1 (as Administrator)" -ForegroundColor White
Write-Host "2. Or add firewall rule manually" -ForegroundColor White
Write-Host "3. Check Docker is running: docker ps" -ForegroundColor White
}
Write-Host ""

242
tests/test_core/test_shopdbbackups_finalize.py Normal file
View File

@@ -0,0 +1,242 @@
"""The post-restore schema upgrade: who runs it, and what happens when it fails.
A restore loads the dump in the Flask-less backup sidecar and leaves a sentinel,
because only the api container can run the migrations. These tests pin the two
properties that failure taught us to care about: the upgrade must not depend on
a browser staying open, and it must never run twice at once.
"""
import os
import pytest
from shopdb.core.api import shopdbbackups
@pytest.fixture
def backups_root(tmp_path, monkeypatch):
"""Point the backup routes at a real, empty directory."""
root = tmp_path / 'backups'
root.mkdir()
monkeypatch.setenv(shopdbbackups.BACKUPS_ROOT_ENV, str(root))
return root
def _arm(root, setname='wj-2026-09-06T120000Z'):
(root / shopdbbackups.UPGRADE_SENTINEL).write_text(setname, encoding='utf-8')
return setname
def test_only_one_claimant_wins_a_pending_upgrade(app, backups_root):
"""Two workers racing the same sentinel must not both run Alembic."""
setname = _arm(backups_root)
with app.app_context():
first = shopdbbackups._claim_upgrade()
second = shopdbbackups._claim_upgrade()
assert first == setname
assert second is None, 'a second worker claimed an upgrade already in flight'
assert not (backups_root / shopdbbackups.UPGRADE_SENTINEL).exists()
assert (backups_root / shopdbbackups.UPGRADE_CLAIM).exists()
def test_claiming_nothing_is_not_an_error(app, backups_root):
with app.app_context():
assert shopdbbackups._claim_upgrade() is None
def test_a_successful_finalize_clears_the_claim_and_says_done(app, backups_root,
monkeypatch):
setname = _arm(backups_root)
monkeypatch.setattr(shopdbbackups, '_run_schema_upgrade', lambda: None)
with app.app_context():
claimed = shopdbbackups._claim_upgrade()
assert shopdbbackups._finalize_claimed(claimed) is True
assert not (backups_root / shopdbbackups.UPGRADE_CLAIM).exists()
with app.app_context():
status = shopdbbackups._read_status('restore')
assert status and status['phase'] == 'done'
assert status['setname'] == setname
def test_a_failed_upgrade_parks_itself_instead_of_retrying_forever(app,
backups_root,
monkeypatch):
"""A failing migration must not be re-armed automatically.
Putting the sentinel back would run the same broken chain on every poll for
as long as the container lives.
"""
_arm(backups_root)
def _boom():
raise RuntimeError('Can\'t locate revision identified by abc123')
monkeypatch.setattr(shopdbbackups, '_run_schema_upgrade', _boom)
with app.app_context():
claimed = shopdbbackups._claim_upgrade()
assert shopdbbackups._finalize_claimed(claimed) is False
assert (backups_root / shopdbbackups.UPGRADE_FAILED).exists()
assert not (backups_root / shopdbbackups.UPGRADE_SENTINEL).exists(), \
'a failed upgrade re-armed itself and will retry forever'
assert not (backups_root / shopdbbackups.UPGRADE_CLAIM).exists()
with app.app_context():
status = shopdbbackups._read_status('restore')
assert status and status['phase'] == 'failed'
def test_the_endpoint_does_not_run_the_upgrade_itself(app, client, auth_headers,
backups_root, monkeypatch):
"""Alembic in a request can outrun the worker timeout and be killed mid-chain."""
_arm(backups_root)
ran = []
monkeypatch.setattr(shopdbbackups, '_run_schema_upgrade',
lambda: ran.append(True))
response = client.post('/api/shopdbbackups/finalize-restore',
headers=auth_headers)
assert response.status_code == 200
assert ran == [], 'the request ran the schema upgrade inline'
assert response.get_json()['data']['queued'] is True
assert (backups_root / shopdbbackups.UPGRADE_SENTINEL).exists(), \
'the endpoint consumed the sentinel the watcher needs to claim'
def test_the_endpoint_re_arms_a_failed_upgrade(app, client, auth_headers,
backups_root):
"""Retrying a failed upgrade is what this endpoint is for now."""
(backups_root / shopdbbackups.UPGRADE_FAILED).write_text('wj-set',
encoding='utf-8')
response = client.post('/api/shopdbbackups/finalize-restore',
headers=auth_headers)
assert response.status_code == 200
assert response.get_json()['data']['retried'] is True
assert (backups_root / shopdbbackups.UPGRADE_SENTINEL).read_text() == 'wj-set'
assert not (backups_root / shopdbbackups.UPGRADE_FAILED).exists()
def test_nothing_pending_reports_nothing_pending(app, client, auth_headers,
backups_root):
response = client.post('/api/shopdbbackups/finalize-restore',
headers=auth_headers)
assert response.status_code == 200
assert response.get_json()['data']['queued'] is False
def test_an_unconfigured_deployment_refuses(app, client, auth_headers,
tmp_path, monkeypatch):
"""No store behind the feature means say so, not 500 on a missing path."""
monkeypatch.setenv(shopdbbackups.BACKUPS_ROOT_ENV,
str(tmp_path / 'nothing-here'))
response = client.post('/api/shopdbbackups/finalize-restore',
headers=auth_headers)
assert response.status_code == 404
class TestPluginChainCompatibility:
"""The restore gate has to see plugin chains, not just the core one.
The failure it exists for: roll code back to an older plugin revision and
the CORE head can be identical, so a core-only gate passes a dump whose
plugin tables the code has never heard of. It loads - destroying what was
there - and only then dies in 'flask plugin upgrade-all'.
"""
def _manifest(self, core_rev, plugins=None):
manifest = {'datarev': core_rev, 'schemarev': {'core': [core_rev]}}
if plugins is not None:
manifest['dataplugins'] = plugins
return manifest
def _real_core_rev(self, app):
with app.app_context():
known = shopdbbackups._known_core_revisions()
assert known, 'expected this build to know its own core revisions'
return sorted(known)[0]
def _real_plugin_rev(self, app):
with app.app_context():
known = shopdbbackups._known_plugin_revisions()
assert known, 'expected this build to have plugin migration chains'
name = sorted(known)[0]
return name, sorted(known[name])[0]
def test_a_set_from_a_newer_plugin_chain_is_refused(self, app):
core = self._real_core_rev(app)
name, _ = self._real_plugin_rev(app)
manifest = self._manifest(core, {name: 'thisrevisiondoesnotexist'})
with app.app_context():
assert shopdbbackups._is_compatible(manifest, {}) is False, \
'a dump carrying unknown plugin data was called restorable'
def test_a_set_whose_plugin_chains_are_known_is_allowed(self, app):
core = self._real_core_rev(app)
name, rev = self._real_plugin_rev(app)
manifest = self._manifest(core, {name: rev})
with app.app_context():
assert shopdbbackups._is_compatible(manifest, {}) is True
def test_a_plugin_this_build_does_not_have_is_refused(self, app):
core = self._real_core_rev(app)
manifest = self._manifest(core, {'nosuchplugin': 'somerevision'})
with app.app_context():
assert shopdbbackups._is_compatible(manifest, {}) is False
def test_a_legacy_set_without_the_stamp_keeps_its_old_verdict(self, app):
"""Sets taken before dataplugins existed must not become unrestorable."""
core = self._real_core_rev(app)
with app.app_context():
assert shopdbbackups._is_compatible(self._manifest(core), {}) is True
assert shopdbbackups._is_compatible(
self._manifest(core, {}), {}) is True
def test_the_core_gate_still_refuses_an_unknown_core_revision(self, app):
name, rev = self._real_plugin_rev(app)
manifest = self._manifest('unknowncorerevision', {name: rev})
with app.app_context():
assert shopdbbackups._is_compatible(manifest, {}) is False
class TestDatarevEvidence:
"""An unreadable data revision must not be badged restorable."""
def _real_core_rev(self, app):
with app.app_context():
known = shopdbbackups._known_core_revisions()
return sorted(known)[0]
def test_an_empty_datarev_is_no_evidence_not_a_legacy_set(self, app):
"""backup.sh discards the mysql error, so datarev can be ''.
Falling back to the code-to-code head check there would badge the set
compatible using the very comparison datarev was introduced to replace.
"""
core = self._real_core_rev(app)
manifest = {'datarev': '', 'schemarev': {'core': [core]}}
with app.app_context():
assert shopdbbackups._is_compatible(
manifest, {'core': [core]}) is False
def test_a_manifest_with_no_datarev_key_still_falls_back(self, app):
"""A genuine legacy set predates the field entirely and keeps working."""
core = self._real_core_rev(app)
manifest = {'schemarev': {'core': [core]}}
with app.app_context():
assert shopdbbackups._is_compatible(
manifest, {'core': [core]}) is True