ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move the documentation to the line that ships, and correct it against the code

Browse Source 
The docs lived on one lineage and the fleet code on another. That is why six
pages described a shared fleet key as how an agent authenticates, why
COLLECTOR-AUTH had to warn that the per-PC credential code was not in the
repository it sat in, and why ADR-019 recorded a decision whose implementation
was somewhere else. Documentation that cannot see the code it documents goes
stale by construction, and no amount of editing fixes that.

So the corpus moves here: 84 files, 56 pages, 20 ADRs, the proposals, INDEX.md,
llms.txt and both gates. Filenames are unchanged, because a wiki page name is
derived from the filename and a rename breaks every link anybody saved.

THE POINT OF THE MOVE IS THE CORRECTION IT ENABLES. Every caveat that said "not
in this tree" was false the moment the files landed beside the code, so each was
rewritten from what is actually here:

- COLLECTOR-AUTH gained the real enroll request body, the two bounds on the
  unauthenticated route (per-IP rate limit, pending cap with stale rows dropped
  before a real PC is refused), and the rejected pending reason that binds to
  the endpoint identity rather than the nonce.
- ADR-019 now describes an implementation that is present, including its tests,
  and says plainly that what remains is the FLEET migration, not the code.
- proposals/device-identity claimed phase 1 was unbuilt. plugins/computers
  declares deviceid on the collector schema and prefers it over the older
  fallbacks. Corrected in the page, its summary and the index.

WINDOWS/IIS IS NOW DOCUMENTED AS THE LEGACY PATH, per the product decision, and
carefully: it is what production runs today. Every Windows page opens by saying
it is supported, not for a new site, and where a new site should go instead. No
procedure was cut - an operator on an IIS box at 2am needs these pages to work
exactly as well tomorrow. MIGRATE-TOPOLOGY stops being a footnote and becomes
the off-ramp, which is why it nearly doubled: what was there did not match how
the Docker stack really works here.

The Docker material that already lived on this line was kept and given
ownership rather than buried under the ported Windows-centric prose: DEPLOY now
describes the four services, the migrate service that runs the upgrade chain
before api starts, TLS terminated by nginx, and the plugin count corrected from
thirteen to seventeen - a number both lineages had wrong.

ROADMAP is rewritten from evidence rather than memory: what has shipped, read
out of the changelog, the ADRs and the manifests, and ahead of it exactly two
items - finalize a distributed release and polish the setup wizard.

PROJECT-MAP is generated and was stale; the GENERATOR was what was wrong.
--check passes now.

API-REFERENCE is retired to a pointer. /api/docs serves 573 operations with
derived auth, query parameters and harvested examples, and a hand-written page
competing with that is a liability. The file stays so its wiki page and its
inbound links survive.

AND THE MOVE TURNED ON A SCRUB THAT HAD NEVER RUN HERE. test_docs_publishable
keys on docs/*.md, which this line did not have, so it skipped - its own comment
predicted what would surface. Porting the corpus activated it and it found 48
lines that must not reach a published wiki: branch names abbreviating the site,
assistant-tooling paths, the site name in two deployment pages, and an employee
ID in a Windows path that the scrub does not even look for.
This commit is contained in:
cproudlock
2026-09-19 09:10:15 -04:00
parent 68567d9a27
commit f9d6b57f0f
43 changed files with 5747 additions and 1066 deletions
Download Patch File Download Diff File Expand all files Collapse all files

221
docs/deployment/DEPLOY-TO-VM.md
View File

@@ -1,177 +1,136 @@
# Deploy ShopDB to Hyper-V VM
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Put the Docker stack on a throwaway Windows VM to try it or rehearse a deploy, and what the helper script gets wrong.
loadwith: none
-->
# Deploying the stack to a Windows VM
Quick guide to deploy the Docker stack to a Windows 11 Hyper-V VM.
> **Where this sits:** a rehearsal, not a site. For a real site follow
> [DEPLOY](../DEPLOY.md). To only LOOK at a build with no source at all,
> [DEV-BUILD-TEST](../DEV-BUILD-TEST.md) pulls a published image and is one
> script.
## Prerequisites on VM
A Windows VM with Docker Desktop is the cheapest way to prove a deploy before
doing it for real: the stack is identical to a site's, the VM can be reverted,
and nothing on it matters. This page is the VM-specific part - getting the code
there, the firewall, reaching it from the host. Everything else is
[DEPLOY](../DEPLOY.md).
1. **Docker Desktop** installed and running
2. **Git Bash** or **WSL** installed (for bash commands)
3. **PowerShell** (already on Windows 11)
## On the VM
## Step 1: Copy Files to VM
1. **Docker Desktop**, installed and running.
2. **Git Bash or WSL**, for the bash commands.
3. Enough RAM given to Docker. The stack is MySQL plus a Python container plus
nginx; 8GB to Docker is comfortable.
## Step 1: Get the code onto the VM
**Method A: Shared Folder**
```powershell
# On host PC, share the repo
# Then on VM, copy from network share
Copy-Item -Recurse \\host-pc\shopdb-flask C:\shopdb
```
# From a network share
Copy-Item -Recurse \\<host>\shopdb-flask C:\shopdb
**Method B: Git Clone**
```bash
# On VM
cd C:\
git clone git@github.com:ge-aero/shopdb-flask.git
cd shopdb-flask
git checkout wj
```
# Or clone it
git clone <your-remote>/shopdb-flask.git C:\shopdb
**Method C: Export/Import**
```powershell
# On host PC
Compress-Archive -Path C:\Users\270015376\Documents\github\flask-shopdb -DestinationPath C:\shopdb-deploy.zip
# Copy shopdb-deploy.zip to VM, then on VM:
# Or hand it over as a zip, which is what an isolated VM usually wants
Expand-Archive -Path C:\shopdb-deploy.zip -DestinationPath C:\shopdb
```
## Step 2: Run Deployment Script
## Step 2: Bring it up
The pre-built path is the one to use on a VM: npm inside the image is slow and
is the step most likely to fail behind a proxy. Build the frontend once on the
host or on the VM, then build the image from it -
[DOCKER-PREBUILT-APPROACH](DOCKER-PREBUILT-APPROACH.md).
```powershell
# On VM, open PowerShell as Administrator
cd C:\shopdb
copy .env.example .env
# Edit .env: SECRET_KEY, JWT_SECRET_KEY, MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD,
# CORS_ORIGINS. Generate a secret with:
# python -c "import secrets; print(secrets.token_urlsafe(64))"
# Option 1: Quick deploy with defaults
.\deploy-to-vm.ps1
cd frontend; npm ci; npm run build; cd ..
# Option 2: Specify site name and password
.\deploy-to-vm.ps1 -SiteName "west-jefferson" -AdminPassword "YourSecurePassword123!"
# Option 3: See all options
.\deploy-to-vm.ps1 -Help
```
## What the Script Does
1. [OK] Checks Docker Desktop is running
2. [OK] Creates .env from template
3. [OK] Builds frontend (if needed)
4. [OK] Builds Docker images (2-3 minutes, offline packages)
5. [OK] Starts all services (db + api + backup)
6. [OK] Runs database migrations
7. [OK] Sets admin password
## Manual Deployment (Alternative)
If you prefer to run commands manually:
```bash
# 1. Create environment
cp .env.docker-dev .env
# 2. Build frontend (if not already built)
cd frontend
npm ci
npm run build
cd ..
# 3. Build and start Docker stack
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
# 4. Initialize database
docker exec flask-shopdb-api-1 flask db upgrade
# 5. Reset admin password (optional)
docker exec flask-shopdb-db-1 mysql -uroot -proot-dev-password-not-for-production shopdb_flask -e \
"UPDATE users SET passwordhash='scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547', failedlogins=0, lockeduntil=NULL WHERE username='admin';"
# (Password is: admin)
```
## Access
The `migrate` service runs the schema, the plugin bootstrap and the seeders
before `api` starts, so there is no separate migration step. Watch it with
`docker compose logs migrate`.
**URL:** https://localhost:5001 (or https://vm-ip-address:5001 from host)
**Login:**
- Username: `admin`
- Password: Whatever you set in deploy script (default: `ChangeMe123!`)
## Verify Deployment
Create the first admin at `https://localhost:5001/setup`, or headlessly:
```powershell
# Check all services are running
docker ps
# Should see:
# - flask-shopdb-db-1
# - flask-shopdb-api-1
# - flask-shopdb-backup-store-1
# - flask-shopdb-backup-watcher-1
# View logs
docker logs -f flask-shopdb-api-1
# Test backup
# Go to https://localhost:5001/settings/shopdbbackups
# Click "Back Up Now"
# Wait 30 seconds, should show as "Checked"
docker compose exec api flask seed admin --username admin --email admin@example.com
```
## Troubleshooting
The password is generated and printed once.
**Docker not running:**
```powershell
# Start Docker Desktop manually
& "C:\Program Files\Docker\Docker\Docker Desktop.exe"
Start-Sleep -Seconds 30
```
## Step 3: Reach it from the host
**Port 5001 already in use:**
```powershell
# Edit .env and change API_PORT
# API_PORT=5002
```
**Can't access from host PC:**
```powershell
# On VM, check Windows Firewall
New-NetFirewallRule -DisplayName "ShopDB" -Direction Inbound -LocalPort 5001 -Protocol TCP -Action Allow
```
**SSL certificate warnings:**
- Normal for self-signed certs
- Click "Advanced" -> "Proceed to localhost (unsafe)"
Then `https://<vm-ip>:5001` from the host. The certificate is self-signed until
you replace the pair in `certs/`, so the browser warns; that is expected on a
VM.
## Stop/Start
`CORS_ORIGINS` is an exact origin match, so it has to name the address the
browser really uses. Reaching the VM by IP while `CORS_ORIGINS` names
`localhost` is the usual cause of "the page loads and every request fails".
## About `deploy-to-vm.ps1`
The repository root carries `deploy-to-vm.ps1`, which wraps the above into eight
steps with `-SiteName` and `-AdminPassword` parameters. **Read it before you run
it.** As it stands it does not work on a clean clone, and neither problem is
subtle:
- It copies `.env.docker-dev` to create `.env`. That file is not in the
repository, so step 3 fails on a fresh checkout. Create `.env` from
`.env.example` yourself first and the script skips that branch.
- It addresses containers by the fixed names `flask-shopdb-api-1` and
`flask-shopdb-db-1`. Compose names containers after the project, which is the
directory name, so on a clone called `shopdb` those `docker exec` calls
miss. Use `docker compose exec api` instead, or `docker compose ps` to see
the real names.
It also runs only `flask db upgrade` for the database, which predates the
`migrate` service and is now redundant rather than wrong.
Its default `-AdminPassword` is a literal in the file and therefore public.
Pass your own, and never leave the default on anything reachable.
## Stop, start, update
```bash
# Stop
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml down
# Start (after first deployment)
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
```
## Update After Code Changes
`down` keeps the volumes. `down -v` deletes them, which is the database and
`instance/` together, with no prompt.
To take new code:
```bash
# 1. Pull latest code
git pull origin wj
# 2. Rebuild frontend (if changed)
cd frontend && npm run build && cd ..
# 3. Rebuild and restart
git pull
cd frontend; npm run build; cd ..
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
```
## Offline Deployment Notes
Rebuilding the frontend is not optional on the pre-built path: the image copies
whatever is in `frontend/dist/`, so a skipped `npm run build` ships the old UI
with no warning.
**This deployment works completely offline** because:
- [OK] Frontend pre-built and checked into repo
- [OK] Restic binary pre-downloaded (8.6MB in repo)
- [OK] No npm/pip downloads during Docker build
- [OK] All dependencies baked into images
## Why this works with no network
Perfect for air-gapped or restricted network environments!
With `frontend/dist/` already built and the Python wheels vendored, the image
build downloads nothing, which is what makes the same procedure the rehearsal
for [DEPLOY-AIRGAP](../DEPLOY-AIRGAP.md).

137
docs/deployment/DOCKER-PREBUILT-APPROACH.md
View File

@@ -1,96 +1,87 @@
# Docker Pre-Built Frontend Approach
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build the Docker image where npm cannot reach the registry, by compiling the frontend on the host first.
loadwith: none
-->
# Building the image with a pre-built frontend
**Problem:** Zscaler corporate proxy causes npm to fail inside Docker builds with "Exit handler never called" error.
> **Where this sits:** a variant of Step 2 of [DEPLOY](../DEPLOY.md), not a
> separate deployment. Everything else on that page still applies.
**Solution:** Build frontend on host (where npm works), copy pre-built dist/ into Docker image.
The shipped `Dockerfile` is a multi-stage build: stage 1 runs `npm ci` and
`npm run build` under Node, stage 2 copies the compiled SPA into the Python
image. Behind a proxy that re-signs HTTPS, stage 1 is where it dies - npm inside
the container does not trust the corporate root CA, and the failure reads
`npm ci failed` or the unhelpful `Exit handler never called`.
## Quick Start
The fix is to build the frontend where npm already works - the host, which has
the corporate CA in its trust store - and give the image the finished
`frontend/dist/`.
It is also much faster: two to three minutes instead of fifteen. A site with no
proxy problem at all may want it for that reason alone.
## Do it
```bash
# 1. Build frontend on host
# 1. Build the frontend on the host
cd frontend
npm ci
npm run build
# 2. Build Docker image (uses pre-built dist/)
npm run build # writes frontend/dist/
cd ..
# 2. Build and start the stack from the pre-built dist
docker compose -f docker-compose.prebuilt.yml build
# 3. Start services
docker compose -f docker-compose.prebuilt.yml up -d
# 4. Initialize (first time only)
docker compose -f docker-compose.prebuilt.yml exec api flask db upgrade
docker compose -f docker-compose.prebuilt.yml exec api bash /app/scripts/bootstrap-plugins.sh
# 5. Access
https://localhost:5001
```
## Files
The rest is [DEPLOY](../DEPLOY.md) unchanged: the `migrate` service runs the
schema, the plugin bootstrap and the seeders before `api` starts, nginx
terminates TLS on `API_PORT`, and the first admin comes from `/setup` or
`flask seed admin`.
- `Dockerfile.prebuilt` - Simplified Dockerfile without npm build
- `docker-compose.prebuilt.yml` - Uses Dockerfile.prebuilt
- `.dockerignore.prebuilt` - Allows frontend/dist to be copied
## The files this uses
## Why This Works
| File | What it is |
|---|---|
| `Dockerfile.prebuilt` | single-stage: Python dependencies plus a copy of `frontend/dist/`. No Node, no npm |
| `docker-compose.prebuilt.yml` | the same stack, building from `Dockerfile.prebuilt` |
| `.dockerignore.prebuilt` | permits `frontend/dist/` to be copied, which the normal `.dockerignore` excludes |
**Normal Dockerfile:**
- Multi-stage build
- Stage 1: Node 20 runs `npm ci` and `npm run build`
- Stage 2: Copies built frontend from stage 1
- **PROBLEM:** npm fails inside Docker with Zscaler proxy
## What it costs
**Pre-Built Approach:**
- Build frontend on Windows host (npm works here)
- Single-stage Dockerfile copies pre-built `frontend/dist/`
- Only installs Python dependencies
- **FASTER:** 2-3 minutes vs 15+ minutes
- **npm has to exist on the build host.** That is the whole trade: the trust
problem moves from the container to a machine that has already solved it.
- **It is two steps, and forgetting the first one ships a stale UI.** The image
copies whatever is in `frontend/dist/` at build time. A frontend change that
was not rebuilt is simply not in the image, and nothing warns you. If the UI
does not show your change, rebuild `dist/` before debugging anything else.
- **`frontend/dist/` has to reach the build.** Either it is committed, or the
pipeline builds it immediately before the image.
## Trade-offs
## The version of this that scales
**Pros:**
- Avoids corporate proxy issues
- Much faster builds (no npm in container)
- Frontend build errors visible immediately
- Can iterate on frontend without rebuilding container
**Cons:**
- Requires npm on host machine
- Two-step build process
- frontend/dist must be in version control OR built before deployment
## For Production (CI/CD)
The pre-built approach is a workaround for building at the deployment site. The
end state is not building there at all:
```yaml
# GitHub Actions / Jenkins example
- name: Build frontend
run: |
cd frontend
npm ci
npm run build
- name: Build Docker image
run: |
docker build -f Dockerfile.prebuilt -t shopdb:${{ github.sha }} .
- name: Push to registry
run: |
docker push shopdb:${{ github.sha }}
# CI: build the frontend, build the image, push it
- run: cd frontend && npm ci && npm run build
- run: docker build -f Dockerfile.prebuilt -t shopdb:$GIT_SHA .
- run: docker push <registry>/shopdb:$GIT_SHA
```
## Comparison
A site then pulls a tagged image and runs it - seconds, no build step, no npm
anywhere near the facility. `docker-compose.pull.yml` is the compose file for
that shape today, and `docker-compose.airgap.yml` is the same idea for a site
that loads an image tarball off removable media
([DEPLOY-AIRGAP](../DEPLOY-AIRGAP.md)). Finishing that path is the distributed
release item on [ROADMAP](../ROADMAP.md).
| Approach | Build Time | Corporate Proxy | Complexity | Production Ready |
|----------|-----------|----------------|------------|------------------|
| Multi-stage Dockerfile | 15+ min | [FAIL] Fails | High | [FAIL] No |
| Pre-built frontend | 2-3 min | [OK] Works | Medium | [WARN] Yes (with CI/CD) |
| Pre-built images from registry | 30 sec | [OK] Works | Low | [OK] Yes |
## Next Steps
For true production deployment:
1. Set up CI/CD to build frontend + Docker image
2. Push to container registry
3. Sites pull pre-built images
4. No build step at deployment site
| Approach | Build time | Behind a re-signing proxy | Where it belongs |
|---|---|---|---|
| Multi-stage `Dockerfile` | 15+ min | fails unless the CA is staged | a developer box with a clean network |
| Pre-built frontend (this page) | 2-3 min | works | a site that must build locally |
| Pull a published image | seconds | works | where every site should end up |

398
docs/deployment/DOCKER-SETUP.md
View File

@@ -1,398 +0,0 @@
# ShopDB Docker Setup
Complete Docker deployment for development and testing.
## Prerequisites
- Docker Desktop installed and running
- At least 8GB RAM allocated to Docker
- Zscaler corporate certificate extracted (if behind corporate proxy)
## Quick Start
```bash
# 1. Clone repository
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Extract Zscaler certificate (if needed)
powershell.exe -File C:/temp/extract-ca.ps1
# This creates corp-root-ca.pem in repo root
# 3. Build images
docker compose build
# 4. Start stack
docker compose up -d
# 5. Initialize database (first time only)
docker compose exec api flask db upgrade
# 6. Install plugins (first time only)
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
# 7. Access ShopDB
https://localhost:5001
```
Login: `admin` / (reset password - see below)
## Architecture
The Docker stack includes:
1. **MySQL Database** (`db` service)
- Port: 3306 (internal), mapped to host based on `.env`
- Volume: `db_data` for persistence
- Health checks enabled
2. **Flask API** (`api` service)
- Port: 5001 (HTTPS with self-signed cert)
- Built frontend served from `/app/frontend/dist`
- Gunicorn with gevent workers (4 workers, 2000 connections each)
- Volumes:
- `instance_data` - Plugin state, uploads, branding
- `./plugins` - Plugin code (read-only)
- `./scripts` - Deployment scripts (read-only)
- `./certs` - TLS certificates (read-only)
3. **Backup Service** (optional, via `docker-compose.backup.yml`)
- One-shot container for backups
- Triggered manually or via scheduler
## First-Time Setup
### 1. Extract Zscaler Certificate (Corporate Proxy)
Save this as `C:/temp/extract-ca.ps1`:
```powershell
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
$certPath = 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem'
Set-Content -Path $certPath -Value $builder.ToString() -Encoding ASCII
Write-Host "Staged corp-root-ca.pem"
```
Run: `powershell.exe -ExecutionPolicy Bypass -File C:/temp/extract-ca.ps1`
### 2. Generate TLS Certificates
```bash
cd certs
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout server.key -out server.crt -days 365 \
-subj "//CN=localhost"
```
Or use the provided script:
```bash
./scripts/generate-dev-certs.sh
```
### 3. Configure Environment
Copy and edit `.env`:
```bash
cp .env.docker-dev .env
# Edit if needed - defaults work for local dev
```
### 4. Build Images
```bash
docker compose build
```
**Note:** First build takes 10-15 minutes due to:
- npm install (~5 min with Zscaler proxy)
- Frontend Vite build (~2 min)
- Python dependencies (~3 min)
Subsequent builds use Docker layer cache and are much faster.
### 5. Start Services
```bash
docker compose up -d
```
Check status:
```bash
docker compose ps
docker compose logs -f api
```
### 6. Initialize Database
```bash
# Run migrations
docker compose exec api flask db upgrade
# Install and enable all plugins
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
```
The `apply-profile` command:
- Installs all 17 plugins
- Registers them in `instance/plugins.json`
- Leaves them disabled by default
- Enables them via Settings UI as needed
### 7. Reset Admin Password
```bash
# Generate hash for "admin" password
docker compose exec api python -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Update database
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
Or use quick method (sets password to `admin`):
```bash
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users SET
passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
## Import Production Database
```bash
# Copy SQL dump to container
docker cp latest.sql shopdb-api-1:/tmp/
# Import
docker compose exec db mysql -u root -p shopdb_flask < /tmp/latest.sql
# Run migrations to bring schema up to date
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```
## Common Commands
### View Logs
```bash
docker compose logs -f api
docker compose logs -f db
```
### Restart Services
```bash
docker compose restart api
docker compose down && docker compose up -d
```
### Rebuild After Code Changes
```bash
docker compose build api
docker compose up -d api
```
### Access Container Shell
```bash
docker compose exec api bash
docker compose exec db bash
```
### Run Flask Commands
```bash
docker compose exec api flask plugin list
docker compose exec api flask db current
docker compose exec api flask routes
```
### Backup & Restore
```bash
# Trigger backup
docker compose -f docker-compose.backup.yml up backup
# View backups
docker compose exec api flask plugin list
# (Accessible via Settings > Backups in UI)
# Restore (via UI at Settings > Backups > Select backup > Restore)
```
## Troubleshooting
### "npm ci failed" during build
**Cause:** Zscaler proxy blocking npm registry.
**Fix:**
1. Extract Zscaler certificate (see above)
2. Ensure `corp-root-ca.pem` exists in repo root
3. Rebuild: `docker compose build --no-cache`
### "database is unhealthy"
**Cause:** MySQL container not ready or port conflict.
**Fix:**
```bash
# Check logs
docker compose logs db
# Check if port 3306 is already in use
netstat -an | grep 3306
# If conflict, edit .env and set MYSQL_PORT=3307
```
### "Access denied for user 'shopdb'"
**Cause:** Database credentials mismatch.
**Fix:** Check `.env` file - `MYSQL_PASSWORD` must match `DATABASE_URL`.
### "Could not connect to database"
**Cause:** Database service not started or wrong hostname.
**Fix:**
```bash
# Check db is running
docker compose ps
# API must use hostname 'db' not 'localhost'
# In docker-compose.yml: DATABASE_URL=...@db:3306/...
```
### "ERR_CERT_AUTHORITY_INVALID" in browser
**Expected:** Self-signed certificate warning.
**Fix:** Click "Advanced" > "Proceed to localhost (unsafe)" - this is normal for dev.
Or generate proper cert:
```bash
# Add your machine's hostname to cert
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout certs/server.key -out certs/server.crt -days 365 \
-subj "//CN=$(hostname)" \
-addext "subjectAltName=DNS:localhost,DNS:$(hostname),IP:127.0.0.1"
```
### "Plugin X is already installed" but not visible
**Cause:** Plugin installed but not enabled.
**Fix:**
```bash
docker compose exec api flask plugin enable X
docker compose restart api
```
### Build is very slow
**Expected:** First build with Zscaler proxy takes 10-15 minutes.
**Speed up:**
- Subsequent builds use cache (much faster)
- If rebuilding often, comment out `--no-cache` flag
- Build frontend separately on host, copy dist/ into image
## Production Deployment Notes
For production:
1. **Change all secrets** in `.env`:
- `SECRET_KEY`
- `JWT_SECRET_KEY`
- `MYSQL_ROOT_PASSWORD`
- `MYSQL_PASSWORD`
2. **Use proper TLS certificates** (not self-signed):
- Place in `./certs/server.crt` and `./certs/server.key`
- Or use reverse proxy (nginx/IIS) for TLS termination
3. **Set `FLASK_DEBUG=0`** in production
4. **Configure backups**:
- Set up cron job: `0 2 * * * cd /path/to/shopdb && docker compose -f docker-compose.backup.yml up backup`
- Configure SMB or S3 destination in Settings > Backups
5. **Set `SHOPDB_ENV_LABEL`** to site name (e.g., "WEST-JEFFERSON")
6. **Review `docker-compose.yml`**:
- Adjust worker count based on load
- Configure proper `restart` policy
- Set resource limits if needed
## Architecture Notes
### Why HTTPS in Docker?
Secure context APIs (clipboard, geolocation) require HTTPS. Running over HTTP disables these browser features.
### Why Gevent Workers?
Endpoint agents long-poll `/api/endpoints/commands/next`. With threading, 200 agents = 200 threads = resource exhaustion. Gevent uses greenlets (cheap coroutines) so 2000+ connections per worker is feasible.
### Plugin Bootstrap
The entrypoint (`scripts/docker-entrypoint.sh`) automatically:
1. Runs database migrations
2. Checks if `instance/plugins.json` exists
3. If not, installs all available plugins (disabled)
4. Starts gunicorn
This ensures imported databases with plugin tables get matching plugin registry.
### Instance Data Persistence
The `instance_data` volume persists:
- `plugins.json` - Which plugins are enabled
- Uploaded files (floor plans, branding, photos, warranties, slides)
- Dell OAuth tokens
- Any plugin-specific state
**Important:** `docker compose down -v` deletes volumes - use plain `docker compose down` to preserve data.
## File Structure
```
flask-shopdb/
|-- docker-compose.yml # Main stack definition
|-- docker-compose.backup.yml # Backup service
|-- Dockerfile # Multi-stage build
|-- .env # Environment config (gitignored)
|-- corp-root-ca.pem # Zscaler cert (gitignored)
|-- certs/ # TLS certificates (gitignored)
| |-- server.crt
| +-- server.key
|-- scripts/
| |-- docker-entrypoint.sh # Container startup
| +-- bootstrap-plugins.sh # Plugin auto-install
+-- deploy/
+-- site-profile-universal.json # Full plugin list
```
## Next Steps
After setup:
1. Browse to https://localhost:5001
2. Login as `admin`
3. Go to Settings > Plugins
4. Enable plugins your site needs
5. Configure settings (CORS, backup destination, etc.)
6. Import data or start fresh
For local development with hot-reload, see `LOCAL-DEV-SETUP.md`.

372
docs/deployment/LOCAL-DEV-SETUP.md
View File

@@ -1,372 +0,0 @@
# ShopDB Local Development Setup (Windows)
Complete guide for setting up ShopDB Flask development environment on Windows.
## Prerequisites
- Python 3.13+
- Docker Desktop (for MySQL and testing)
- Git
- Node.js 20+ (for frontend development)
## Quick Start
```bash
# 1. Clone and enter repo
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Create Python virtual environment
python -m venv venv
./venv/Scripts/activate
pip install -r requirements.txt
# 3. Start MySQL in Docker
docker compose up -d db
# 4. Configure environment
cp .env.docker .env
# Edit .env and set DATABASE_URL (see below)
# 5. Install and enable plugins
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe plugin install backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty
./venv/Scripts/flask.exe plugin enable <each-plugin-name>
# 6. Start Flask
./venv/Scripts/python.exe run_dev.py
```
Visit http://localhost:5001 and login with `admin` / `admin`.
## Detailed Setup
### 1. Database Setup
MySQL runs in Docker on **port 3307** (not 3306) to avoid conflicts with any local MySQL instance.
```bash
# Start MySQL container
docker compose up -d db
# Verify it's running
docker ps --filter name=shopdb-dev-db
# Should show: 127.0.0.1:3307->3306/tcp
# Test connection
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' -e "SELECT 1;"
```
### 2. Environment Configuration
Create `.env` file with:
```bash
# Database (port 3307 because Docker maps it there)
DATABASE_URL=mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev
# Flask
FLASK_ENV=development
FLASK_DEBUG=1
# Secrets (dev only)
SECRET_KEY=docker-dev-secret-key-change-for-production
JWT_SECRET_KEY=docker-dev-jwt-secret-change-for-production
# CORS
CORS_ORIGINS=http://localhost:5173,http://localhost:5001
# API Port
API_PORT=5001
# ShopDB Label
SHOPDB_ENV_LABEL=LOCAL-DEV
# Log Level
LOG_LEVEL=DEBUG
# Backup Storage (for local dev)
SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups
```
**Important:** Change `YOUR_USERNAME` to your actual Windows username.
### 3. Import Database (Optional)
If you have a production database dump:
```bash
# Import latest.sql
docker exec -i shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev < latest.sql
# Run migrations
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### 4. Plugin Installation
All 17 plugins must be installed before they can be enabled:
```bash
# Set DATABASE_URL for CLI commands
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
# Install all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin install $plugin
done
# Enable all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin enable $plugin
done
```
**Why this is necessary:**
- Plugin state is tracked in `instance/plugins.json`, not the database
- When you import a prod database, it has plugin tables but no plugin registry
- Installing registers plugins in the JSON file
- Enabling makes them active (but requires app restart)
### 5. Reset Admin Password
If you imported a prod database and don't know the password:
```bash
# Generate new password hash
./venv/Scripts/python.exe -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Reset via MySQL
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Or use the simpler method:
```bash
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Password is now: `admin` / `admin`
### 6. Starting Flask
**Always use `run_dev.py`** (not `flask run` directly):
```bash
./venv/Scripts/python.exe run_dev.py
```
This script:
- Loads `.env` file with DATABASE_URL
- Sets all environment variables
- Starts Flask without auto-reloader (prevents env var loss)
- Binds to 0.0.0.0:5001
**Do NOT use:**
- `flask run` - Won't load `.env` properly
- `flask run --reload` - Loses environment variables
### 7. Frontend Development (Optional)
To develop the Vue frontend:
```bash
cd frontend
npm ci
npm run dev
```
Frontend dev server runs on http://localhost:5173 and proxies API requests to Flask on port 5001.
To point the proxy at a backend on another machine, set `VITE_DEV_API_TARGET`
in your shell instead of editing `vite.config.js`:
```bash
VITE_DEV_API_TARGET=https://10.0.0.5:5001 npm run dev
```
Certificate verification is off for an https target because those boxes
normally use a self-signed cert; set `VITE_DEV_SECURE=true` to enforce it.
For production build (served by Flask):
```bash
cd frontend
npm run build
# Outputs to frontend/dist/
```
## Troubleshooting
### "Can't connect to MySQL on 127.0.0.1:3306"
**Cause:** Flask is trying port 3306 instead of 3307.
**Fix:**
1. Make sure `.env` has `DATABASE_URL=...@localhost:3307/...`
2. Use `run_dev.py` to start Flask (NOT `flask run`)
3. If problem persists, restart Flask completely
### "Cannot import name '_backups_configured'"
**Cause:** Merge conflict between wj and wilm backup implementations.
**Fix:** This was fixed in commit `8a1fe0ada8` - pull latest wj branch.
### "Plugin X is not installed"
**Cause:** Plugin registry (`instance/plugins.json`) doesn't exist or is incomplete.
**Fix:** Run plugin installation commands (section 4 above).
### "Access is denied: '/backups'"
**Cause:** Backup system trying to use Linux path on Windows.
**Fix:**
1. Add to `.env`: `SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups`
2. Create directory: `mkdir backups`
3. Restart Flask with `run_dev.py`
### Docker npm install fails (Zscaler proxy)
**Cause:** Corporate proxy intercepts HTTPS, npm doesn't trust certificate.
**Fix:** Extract Zscaler root CA certificate:
```powershell
# Run this PowerShell script
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
Set-Content -Path 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem' -Value $builder.ToString() -Encoding ASCII
```
Then rebuild Docker: `docker compose build --no-cache`
## Common Tasks
### Run Tests
```bash
pytest
pytest tests/test_specific.py
pytest -v -s # Verbose with print output
```
### Run Migrations
```bash
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### Create Migration
```bash
./venv/Scripts/flask.exe db migrate -m "Description of changes"
```
### Check Plugin Status
```bash
./venv/Scripts/flask.exe plugin list
```
### Enable/Disable Plugin
```bash
./venv/Scripts/flask.exe plugin enable slides
./venv/Scripts/flask.exe plugin disable slides
```
### Trigger Backup
```bash
# Via API (requires admin JWT token)
curl -X POST http://localhost:5001/api/shopdbbackups \
-H "Authorization: Bearer YOUR_TOKEN"
# Or via Docker
docker compose -f docker-compose.backup.yml up backup
```
## Architecture Notes
### Why Port 3307?
Docker MySQL maps internal port 3306 to host port 3307 to avoid conflicts with any local MySQL installation. This is configured in `docker-compose.yml`:
```yaml
ports:
- "127.0.0.1:3307:3306"
```
### Why run_dev.py?
Flask's `flask run` command uses an auto-reloader that spawns a subprocess. This subprocess loses environment variables loaded from `.env`. The `run_dev.py` wrapper:
1. Loads `.env` via python-dotenv BEFORE importing Flask
2. Uses `--no-reload` flag to prevent subprocess spawning
3. Ensures DATABASE_URL and other vars persist
### Plugin Architecture
- Plugins are Python packages in `plugins/` directory
- Plugin state tracked in `instance/plugins.json` (not database)
- Plugin tables tracked via Alembic migration chains
- Each plugin has own migrations in `plugins/<name>/migrations/`
### Backup System
- Designed for Docker compose deployment
- API queues backups via sentinel file (`.backup-requested`)
- Actual backup execution requires Docker sidecar container
- On Windows dev, API can queue but won't execute without Docker
## File Structure
```
flask-shopdb/
|-- .env # Local environment config (gitignored)
|-- run_dev.py # Local development launcher
|-- shopdb/ # Flask application
| |-- __init__.py # App factory
| |-- config.py # Configuration classes
| |-- core/ # Core models and APIs
| +-- plugins/ # Plugin manager
|-- plugins/ # Plugin packages
| |-- backups/
| |-- computers/
| +-- ...
|-- frontend/ # Vue 3 SPA
| |-- src/
| +-- dist/ # Production build (gitignored)
|-- instance/ # Runtime state (gitignored)
| |-- plugins.json # Plugin registry
| +-- ...
|-- migrations/ # Core Alembic migrations
|-- scripts/ # Deployment and backup scripts
|-- backups/ # Local backup storage (gitignored)
+-- tests/ # Test suite
```
## Git Workflow
- `main` branch - Published/public version (GitHub)
- `wj` branch - Production deployment (West Jefferson site)
- `wilm` branch - Development branch (upstream from benjamin.baker)
- `dev` branch - Ops deployment branch
**Never commit to `main` directly** - it's published to public wiki.
## Additional Resources
- [Backup Setup](scripts/backup/README.md)
- [Plugin Development](docs/PLUGIN-HOOKS.md)
- [ADRs](wiki/ADRs/)
- [API Docs](http://localhost:5001/api/docs) (when Flask is running)

47
docs/deployment/PRODUCTION-READINESS.md
View File

@@ -1,7 +1,29 @@
# ShopDB Production Readiness Assessment
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: historical
summary: The 2026-09-07 assessment of why deployment was hard. Its gaps became the roadmap; its recommendation was reversed.
loadwith: none
-->
# Production readiness assessment, 2026-09-07
**Last Updated:** 2026-09-07
**Branch:** wj (post-wilm merge)
> **This is a record, not guidance.** Read it for WHY the deployment story
> needed work, not for what to do. Two things have moved since:
>
> - **Its recommendation was reversed.** It recommends Windows/IIS for the
> reference site and Docker only for new ones. The decision since is the
> other way round: Docker is the path for every site and Windows/IIS is the
> legacy topology that will be deprecated.
> - **Its "critical path" became the roadmap.** Pre-built images in a registry,
> one-command setup, automatic plugin registration and a simpler backup mode
> are now the distributed-release item on [ROADMAP](../ROADMAP.md). Several
> of its gaps are already closed: the `migrate` service runs the schema, the
> plugin bootstrap and the seeders with no manual step, and
> `docker-compose.prebuilt.yml` removes npm from the image build.
>
> Current pages: [DEPLOY](../DEPLOY.md) to stand a site up,
> [DOCKER-PREBUILT-APPROACH](DOCKER-PREBUILT-APPROACH.md) to build it behind a
> proxy, [ROADMAP](../ROADMAP.md) for what is left.
## Executive Summary
@@ -196,15 +218,17 @@ The Docker deployment requires extensive manual setup, multiple services, and as
## Honest Recommendations
### For West Jefferson (Your Site)
### For the reference site
**Recommended:** Option 1 (IIS + Windows Services)
**Why:**
- You already have Windows infrastructure
*Reversed since. See the note at the top of this page.*
**Why (as argued at the time):**
- The site already had Windows infrastructure
- Simpler to operate
- Faster to deploy
- Your team knows IIS
- The team knew IIS
**Steps:**
1. Build Windows installer
@@ -262,6 +286,9 @@ https://your-server:5001
- `PRODUCTION-READINESS.md` - This file (meta/assessment)
- `scripts/backup/README.md` - Backup system (3 containers, restic)
*Closed since: the first two were folded into `DEVELOPMENT-SETUP` and `DEPLOY`,
and `INDEX.md` is now the single entry point this section asks for.*
**Problem:** Which one do I follow?
**Solution:** Create decision tree:
@@ -279,7 +306,7 @@ https://your-server:5001
## What We Did Today (Honest Assessment)
[OK] **Merged wilm->wj** - 98 commits, wilm features now in wj
[OK] **Merged the upstream development branch down** - 98 commits
[OK] **Fixed local dev** - `run_dev.py` works for Windows development
[OK] **Documented Docker** - Comprehensive but too complex
[OK] **Added backup docs** - System works but needs expertise
@@ -296,7 +323,7 @@ https://your-server:5001
- Gets people to working system fast
2. **Test Windows Installer**
- Build installer for wj branch
- Build the installer for the deployment branch
- Deploy to test Windows Server
- Document any issues
@@ -308,7 +335,7 @@ https://your-server:5001
4. **Set up CI/CD for image builds**
- GitHub Actions or Jenkins
- Build on every push to wj
- Build on every push to the deployment branch
- Push to internal registry
5. **Create `deploy.sh` script**

29
docs/deployment/README.md
View File

@@ -1,29 +0,0 @@
# ShopDB Deployment Documentation
Complete deployment guides for development, Docker, and production environments.
## Quick Start
**Local development**: `LOCAL-DEV-SETUP.md`
**Docker deployment**: `DOCKER-SETUP.md`
**VM deployment**: `DEPLOY-TO-VM.md`
## Docker Deployment
- `DOCKER-SETUP.md` - Complete Docker setup and configuration
- `DOCKER-PREBUILT-APPROACH.md` - Workaround for corporate proxy blocking npm in Docker
- `PRODUCTION-READINESS.md` - Production deployment checklist
## Development
- `LOCAL-DEV-SETUP.md` - Setting up local development environment (Flask + Vite)
## VM Deployment
- `DEPLOY-TO-VM.md` - Deploying the C# endpoint agent to test VMs
## Related Documentation
- `.claude/setup/` - MCP servers, VM configuration, and development tools
- `.claude/maintenance/` - Operational checklists and maintenance procedures
- `../shopdb-flask.wiki/ADR-004-deployment-topology.md` - Architecture decision record for deployment topology