shopdb-flask/docs/DEPLOY.md

Deploy a site with Docker (connected facility)

Topology: Docker, at a site with network access. This is the path a new site takes, and the only one this line ships. No internet at all -> DEPLOY-AIRGAP. Where this sits: install -> operate -> update -> upgrade. Forward: UPGRADE for later versions, BACKUP-RESTORE for day-to-day custody of the data.

shopdb-flask is single-tenant per ADR-004. Each adopting facility runs its own stack: own database, own users, own enabled plugins, own secrets. This page is the runbook for a fresh site deploy that has network access, and it is also the reference for the environment variables and for how the application is published and learns who is calling it - other pages link here rather than restate it.

Which compose file

File	For
docker-compose.yml	this page: a site that can build the image from source
docker-compose.prebuilt.yml	the same stack built from an already-compiled frontend/dist/, for a box where npm cannot reach the registry. See DOCKER-PREBUILT-APPROACH
docker-compose.airgap.yml	DEPLOY-AIRGAP: runs a loaded image, never builds, and migrates itself with a one-shot migrate service
docker-compose.pull.yml	dev and pilot workstations that pull a published channel image instead of building
docker-compose.channel.yml	overlay adding SHOPDB_ENV_LABEL so a stable and an edge stack can run side by side
docker-compose.backup.yml	opt-in backup overlay (restic repository, watcher, one-shot). See BACKUP-RESTORE

Prerequisites

• Docker 24+ and Docker Compose v2 (or equivalent container runtime)
• A reverse proxy with TLS termination (nginx, traefik, Caddy, a corporate load balancer), or the certificate pair described in Step 2 - the application itself is not a TLS terminator by design
• A MySQL backup destination (offsite recommended)
• A clone of the repository (the product is distributed as source, not as a public image)

Step 1: Clone and configure

git clone <your-remote>/shopdb-flask.git
cd shopdb-flask
cp .env.example .env

Edit .env:

Variable	Required	Notes
FLASK_ENV	Yes	production for live sites
SECRET_KEY	Yes	python -c "import secrets; print(secrets.token_urlsafe(64))"
JWT_SECRET_KEY	Yes	Same generation, different value
DATABASE_URL	venv only	mysql+pymysql://shopdb:PASSWORD@db:3306/shopdb_flask?charset=utf8mb4. Under this compose file the services build their own from MYSQL_PASSWORD and never read this line - see the note below
CORS_ORIGINS	Yes	Comma-separated explicit origins. Wildcard rejected.
MYSQL_ROOT_PASSWORD	Yes	Container only
MYSQL_PASSWORD	Yes	Container only, must match DATABASE_URL password
MYSQL_PORT	No	Default 3306
API_PORT	No	Default 5001
LOG_LEVEL	No	Default INFO
ZABBIX_URL, ZABBIX_TOKEN	No	Only if printers plugin uses Zabbix
COLLECTOR_API_KEY	No	Shared key for /api/collector/* ingest. Required only if unattended collectors push data. Endpoint fails closed (denies) when unset.
COLLECTOR_API_KEY_<PLUGIN>	No	Per-plugin override (e.g. COLLECTOR_API_KEY_COMPUTERS), checked before the shared key (ADR-006)
EMPLOYEE_DB_HOST/USER/PASSWORD/NAME	No	Read-only HR directory for notifications + kiosks. No safe default for the password.
TRUSTED_PROXY_COUNT	No	venv/IIS only. Under compose it is pinned to 1 for the bundled nginx. See Step 7

CONFIG and shopdb/config.py are authoritative for every other variable and its default.

.env is not handed to the container wholesale. None of the compose files use env_file:; compose substitutes .env values into the environment: block it declares, and only the variables named there reach the application. As shipped, the api service forwards SECRET_KEY, JWT_SECRET_KEY, CORS_ORIGINS, LOG_LEVEL, ZABBIX_URL, ZABBIX_TOKEN, COLLECTOR_API_KEY, COLLECTOR_API_KEY_ENDPOINTS and SHOPDB_ENV_LABEL. Anything else in this table - a per-plugin collector key for some other plugin, the EMPLOYEE_DB_* set, the CMMC_USB_DB_* set, PLUGIN_TRUSTED_KEYS - needs a line adding to that block as well, or it is set in a file the application never reads.

Two entries are set by the compose file itself and ignored from .env:

• DATABASE_URL is composed in-file as mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask?charset=utf8mb4, on both api and migrate. MYSQL_PASSWORD is therefore the value that matters; editing DATABASE_URL in .env changes nothing under compose.
• TRUSTED_PROXY_COUNT is pinned to 1 because the stack contains its own nginx. See Step 7.

SHOPDB_ENV_LABEL defaults to DEV here on purpose: this file is the source-build template, and a raw source build is by definition not a release. Set it to the site's label for a production stack.

Step 2: Bring up the stack

docker compose build
docker compose up -d

What comes up

Four services, and knowing which is which saves most of the debugging:

Service	Does	Listens
db	MySQL 8.4, utf8mb4 server-wide, volume db_data	127.0.0.1:${MYSQL_PORT:-3306} - loopback only, for mysqldump and a local client. api reaches it over the compose network regardless
migrate	schema, plugin bootstrap and seeds, then exits. restart: "no"	nothing
api	gunicorn, gevent workers, serves the compiled SPA	127.0.0.1:5002 - debugging bypass only
nginx	terminates TLS and proxies to api	${API_PORT:-5001} -> 443

So the address a browser uses is https://<host>:${API_PORT}, served by nginx, and 5002 is a back door for curl on the host.

TLS lives in nginx, not in gunicorn. nginx.conf reads /etc/nginx/certs/server.crt and server.key, and ./certs is mounted there read-only. The repository ships a self-signed pair and certs/openssl.cnf; replace both files with the site's own certificate and docker compose restart nginx. The gunicorn command carries no --certfile, by design - one TLS terminator, one place to rotate a key.

The reason there is TLS at all rather than plain http: browser APIs that require a secure context, navigator.clipboard among them, refuse to run over plain http on a non-localhost origin, so a stack reached by hostname needs real TLS somewhere.

Why gevent and not threads. The endpoint agent long-polls /api/endpoints/commands/next and re-polls immediately on a 204, so every agent holds a connection essentially continuously and fleet size IS the concurrency. Under threads that was 4 x 8 = 32 slots for the whole site, and a worker's threads filled with sleeping agents while the CPU sat idle. A held connection under gevent is a greenlet, so idle is nearly free. Before changing --workers, read the connection arithmetic in the comment beside it: workers x (pool size + overflow) has to stay under the MySQL max_connections, and at the shipped 4 workers it is already about 90 of 151.

Building where npm cannot reach the registry

The image builds the Vue frontend in a first stage and copies the compiled SPA in, so docker compose build needs no separate Node step. Behind a proxy that re-signs HTTPS (Zscaler and the like), that first stage is where the build dies: npm inside the container does not trust the corporate root CA, and the failure reads npm ci failed or Exit handler never called.

Two ways out, in order of preference:

1. Give the build the corporate root CA. Export it to corp-root-ca.pem in the repository root - DEVELOPMENT-SETUP has the one-line PowerShell that bundles every trusted root - then docker compose build --no-cache.
2. Build the frontend on the host and skip npm in the image entirely. That is docker-compose.prebuilt.yml plus Dockerfile.prebuilt, and it is also two to three minutes instead of fifteen. See DOCKER-PREBUILT-APPROACH.

(For a bare-metal venv install instead: cd frontend && npm ci && npm run build, which writes frontend/dist/ for Flask to serve.)

The MySQL container initializes its volume on first run; its healthcheck probes over TCP rather than the socket, because the socket answers during init while the server is still about to restart. Watch the bring-up with docker compose logs -f migrate api: if ProductionConfig.validate() raises, the container exits with the offending env-var named in the log. Fix .env and docker compose up -d again.

Step 3: The schema, and what the migrate service already did

On this stack you do not run these by hand. The migrate service runs them on every up, in one place and one order, and api will not start until it exits 0:

flask db upgrade && /app/scripts/bootstrap-plugins.sh && flask plugin upgrade-all
  && flask seed permissions && flask seed settings && flask seed reference-data

up -d api still migrates first: compose honours service_completed_successfully even when a single service is named. A failing migration exits once and leaves the error as the last line of docker compose logs migrate, rather than crash-looping under a restart policy and burying it.

The rest of this section says what each command covers, because you will need that when one of them fails, when you are on a venv install with no migrate service, or when you are pruning a lean site (below). To run one by hand: docker compose exec api flask <command>.

flask db upgrade applies the core Alembic chain: the baseline migration plus every later migration, which together create all core AND bundled-plugin tables through the chain head. flask plugin upgrade-all then stamps each bundled plugin's own migration chain (the alembic_version_<plugin> tables) and applies any plugin-specific migrations added after the ownership cutover. Both commands are idempotent, so re-running them is safe. See ADR-008 for why plugin schema splits into per-plugin chains from the cutover forward.

Lean sites (ADR-014): the core chain creates every bundled plugin's tables, so a site that ships only some plugins still has the others' (empty) tables. To carry only core + chosen-plugin tables, prune the rest once, at initial provisioning, after the two commands above:

docker compose exec api flask plugin prune-schema            # dry-run, review
docker compose exec api flask plugin prune-schema --yes --force

It drops the tables of every plugin not installed on this site. --force is needed because the core chain seeds a few plugin reference tables (default access protocols, etc.); at first provisioning those hold only seeded defaults, before any site data. It refuses to drop a table that holds rows without --force, so it is safe to leave out of routine upgrades - run it only when provisioning a lean site or after deliberately removing a plugin. Installing a pruned plugin later recreates its tables automatically.

Charset: the schema is utf8mb4 (utf8mb4_unicode_ci). The docker-compose db service sets --character-set-server=utf8mb4, so the auto-created shopdb_flask database is utf8mb4. If you point at an external MySQL instead of the bundled container, create the database as utf8mb4 first, or it inherits the server default (often latin1) and the schema silently drifts:

CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

DATABASE_URL must keep ?charset=utf8mb4 so the connection matches. On MySQL older than 5.7 also enable innodb_large_prefix=ON + innodb_file_format=Barracuda, or the utf8mb4 indexes exceed the 767-byte prefix limit (error 1071). MySQL 5.7+ and 8.0 need no extra config.

Step 4: Seed permissions, settings, and reference data

The migrate service runs all three on every up, and they are idempotent, so this is a no-op once a site is current and is how a later release's new permission or new default setting reaches an existing site. They are listed here because you need to know what they cover, and because a venv install runs them itself:

docker compose exec api flask seed permissions
docker compose exec api flask seed settings
docker compose exec api flask seed reference-data

• seed permissions - creates the RBAC permission rows and the default roles the app checks with @require_permission. Run this before anyone logs in or permission checks have nothing to match.
• seed settings - writes the default Setting rows (branding, ServiceNow integration, floor-map placeholders, search toggles, site identity). A site overrides these later in Settings or the setup wizard.
• seed reference-data - creates default ModelType, AssetStatus, LocationType, CommunicationType, OperatingSystem, RelationshipType rows seeded with the platform contract values (partof, controls, connectedto). (Vendor, Location, and BusinessUnit are not seeded here; they come from seed demo.)

Step 5: Pick plugins to enable

The image bundles every plugin in the tree. PROJECT-MAP lists them with their versions, generated from the code; PLUGINS is the page for deciding which ones a site should carry. Only enabled plugins are loaded.

scripts/bootstrap-plugins.sh, inside the migrate service, has already installed a starting set on a site with no instance/plugins.json: backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty. It installs them disabled, and it skips itself entirely once the registry file exists, so it never fights a site's own choices.

That is every bundled plugin, so on this line the starting set and the shelf are the same list - a site turns one on rather than installing it:

docker compose exec api flask plugin list
docker compose exec api flask plugin enable warranty

Or put the whole list in a profile and keep that in version control, which is the thing to do when more than one site is involved:

docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json

deploy/site-profile.example.json is the template to copy for a site of your own. Enabling a plugin needs docker compose restart api to take effect.

To install a sister-site or third-party plugin (per ADR-003), drop its directory into <repo>/plugins/<name>/ (the docker-compose mounts this read-only into the container) and run flask plugin install <name>.

Step 6: Create the first admin (setup wizard)

The primary path is the first-run setup wizard. Once the stack is up and the DB is seeded, browse to the site (through the reverse proxy configured in Step 7, or directly at the API port during bring-up) and go to /setup. The wizard creates the first admin account and captures site identity (facility name, optional logo). It runs only while setup_complete is false; after it finishes the route redirects to the app.

Headless alternative (no browser, e.g. automated provisioning):

docker compose exec api flask seed admin --username admin --email admin@facility.example.com
# Password is generated and printed once. Store in your password manager.

Subsequent users are managed through the UI.

Step 7: TLS, and the proxy that is already there

The stack ships its own reverse proxy. The nginx service terminates TLS on ${API_PORT:-5001} and proxies to api over the compose network, forwarding X-Real-IP, X-Forwarded-For, X-Forwarded-Proto and Host, with 90-second proxy timeouts so an agent's long poll is not cut mid-hold. For most sites Step 7 is therefore: drop your certificate into ./certs, set API_PORT, and stop.

Put a corporate load balancer or an existing nginx in FRONT of that only if the site already has one. The shape then is proxy -> nginx -> api, which is two proxies, and TRUSTED_PROXY_COUNT has to say so (below). A site that would rather terminate TLS in its own proxy can instead remove the nginx service and publish api directly - a configuration like this one:

server {
  listen 443 ssl;
  server_name shopdb.facility-a.example.com;

  ssl_certificate     /etc/ssl/certs/shopdb.crt;
  ssl_certificate_key /etc/ssl/private/shopdb.key;

  location / {
    proxy_pass http://localhost:5001;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

(That block proxies to plain http, so it belongs on the same host as the container or on a trusted network segment.)

Real client addresses: TRUSTED_PROXY_COUNT

Behind a proxy the application sees the proxy's connection, not the browser's. TRUSTED_PROXY_COUNT is the number of proxies actually in front of this instance, and the application then enables ProxyFix, which resolves the client address, scheme, host and prefix from the X-Forwarded-* headers.

docker-compose.yml already sets it to 1 on the api service, because the bundled nginx is that one proxy. It is not read from .env there - the value belongs to the stack's shape, not to the site. Change it in the compose file, not in .env, and only when you change the shape: 2 if you put a corporate load balancer in front of nginx, 0 if you delete the nginx service and publish api directly with nothing ahead of it.

• It is 0 by default, deliberately. ProxyFix makes the application trust those headers, and any caller can send them. Enabled with nothing in front, a caller can claim a different source address or scheme - and the login rate limit and the account lockout are keyed on the client address.
• Leave it at 0 behind a proxy and the application is not wrong about the client so much as blind to it: every request looks like it came from the proxy. The whole site then shares one rate-limit bucket, and the audit log records the proxy for every action.
• Getting the scheme wrong has its own symptom: an absolute redirect built from the proxy's view sends an https page to an http URL the browser refuses. That is what made a redirect on the endpoints list unfollowable behind IIS.
• .env.example also carries TRUSTED_PROXY_COUNT, defaulting to 0. That line is for a venv or IIS install, where nothing sets it for you; under this compose file the compose value wins and the .env line is inert.

CORS_ORIGINS is an exact origin match (scheme + host + port, no path), so it has to name the address the browser actually uses, not the container's.

Step 8: Backups

Per-site backups are the site's responsibility: nightly dump to offsite storage, 14-day retention, a restore rehearsed quarterly. The database is not the whole backup - instance/ holds uploaded floor plans, branding, plugins.json and tokens that are not in MySQL, and under compose it is the instance_data named volume. BACKUP-RESTORE is the procedure for both, including the compose backup overlay.

docker compose exec -T db mysqldump -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask | gzip > backup-$(date +%F).sql.gz
docker compose run --rm -v "$PWD:/backup" api tar czf /backup/instance-$(date +%F).tar.gz -C /app/instance .

Step 9: Updates

Moving this site to a newer version is UPGRADE - back up, pull, rebuild, migrate, re-seed, restart, plus the version-specific notes. One thing belongs here rather than there, because it is about how this stack is built:

The instance_data volume, and stacks that predate it

up -d api REPLACES the container. Everything the site has written lives in the instance_data volume for exactly this reason: the enabled-plugin list (plugins.json), uploaded floor plans and branding, model and application images, employee photos, warranty proofs, slides, printed-part files, and the Dell OAuth token.

docker compose down is safe; docker compose down -v is not. The -v deletes the named volumes, which is both instance_data and db_data: the site's files and its database in one command. There is no confirmation prompt.

If your stack predates that volume, those files are in the old container's writable layer and an update discards them. Move them across ONCE, before the next rebuild:

docker compose cp api:/app/instance ./instance-rescued   # BEFORE pulling new code
docker compose up -d api                                 # creates the volume
docker compose cp ./instance-rescued/. api:/app/instance
docker compose exec -u root api chown -R shopdb:shopdb /app/instance
docker compose restart api

docker compose exec api flask plugin list   # the site's plugins, enabled again

Two things decide whether this works.

• The chown is not optional. docker compose cp writes the files with the copying user's numeric uid, which is only shopdb by coincidence if your host account happens to be uid 1000. Get it wrong and the site reads its restored files fine and cannot write new ones. Every copy into a container in these pages ends with this same chown, for this reason.
• Restore instance/ BEFORE plugin upgrade-all. That command works from plugins.json, so against an empty instance directory it upgrades nothing and reports success.

The symptom of having missed it is a site that comes back with its plugins disabled and image URLs that 404: the MySQL rows survived, the files did not. Re-enabling by hand works, but flask plugin apply-profile <profile.json> puts the same list back in one command and is the thing to keep in version control.

Importing an existing database

A site coming from an earlier ShopDB, or a test stack loading a copy of production, restores the dump and then brings the schema forward:

docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask < dump.sql
docker compose up -d migrate      # re-runs upgrade, bootstrap and the seeds
docker compose restart api

The order matters and so does the second line. A restored dump carries plugin TABLES but not instance/plugins.json, which is where the enabled-plugin list lives (ADR-008 and the note in Step 9). Without a bootstrap pass the site comes up with a full schema and no plugins registered, which reads as "all my data is gone". flask plugin apply-profile puts a known list back in one command.

Restore instance/ at the same time if you have it - see Step 9.

Common issues

Symptom	Cause	Fix
ConfigError: SECRET_KEY is required in production	.env missing or blank	Set SECRET_KEY in .env, re-up
ConfigError: CORS_ORIGINS must be a comma-separated allowlist	.env has *	Set explicit origins
PluginVersionError: requires core_version X but framework is Y	Plugin pinned a too-narrow range	Update manifest.json core_version or pin framework version
500s after flask db upgrade	Migration ran but app cached old schema	docker compose restart api
Cannot reach API after restart	Reverse proxy not pointing at the container's exposed port	Confirm API_PORT and proxy config
npm ci failed / Exit handler never called during build	A proxy re-signs HTTPS and npm in the build stage does not trust the corporate root CA	Stage corp-root-ca.pem in the repo root and docker compose build --no-cache, or build the frontend on the host (DOCKER-PREBUILT-APPROACH)
database is unhealthy, or db never goes green	MySQL still initializing, or the published port is taken on the host	docker compose logs db; if the port clashes, set MYSQL_PORT=3307 in .env. The mapping is loopback-only and does not affect api
Access denied for user 'shopdb'	MYSQL_PASSWORD changed after the db_data volume was initialized	MySQL sets the app user's password only on first init. Change it in MySQL, or drop the volume on a site with nothing in it yet
Could not connect to database from api or migrate	Something is pointing at localhost instead of the db service	Inside the stack the host is db. localhost in a container is that container
ERR_CERT_AUTHORITY_INVALID in the browser	The shipped certs/server.crt is self-signed	Expected on a first bring-up. Replace the pair in certs/ with the site's own certificate and docker compose restart nginx
A plugin is installed but nothing appears in the nav	Installed is not enabled, and enabling needs a restart	flask plugin enable <name> then docker compose restart api
The first build takes 15 minutes	npm install and the Vite build inside the image	Expected once; later builds hit the layer cache. To avoid npm in the image entirely see DOCKER-PREBUILT-APPROACH

Health check

# Through nginx, on API_PORT. -k because the shipped certificate is self-signed
# until you replace it.
curl -sk -X POST -H "Content-Type: application/json" \
  -d '{}' https://localhost:5001/api/auth/login | jq .
# Expect: {"status": "error", "data": {"error": {"code": "VALIDATION_ERROR", ...}}}

curl -sk https://localhost:5001/api/dashboard/health    # no authentication needed

api has its own healthcheck over plain http inside the stack, and the debugging bypass on the host is curl -s http://127.0.0.1:5002/api/dashboard/health. Comparing the two separates "the app is broken" from "nginx or the certificate is broken": bypass green and :5001 red means the problem is TLS or the proxy. If both are red or return no JSON, docker compose logs api.

References

• UPGRADE - moving this site to a newer version
• DOCKER-PREBUILT-APPROACH - building the image where npm cannot reach the registry
• DEV-BUILD-TEST - pulling a prebuilt stack onto a Windows box to look at it
• DEPLOY-AIRGAP - the same stack where nothing can be pulled
• BACKUP-RESTORE - backup and restore procedure
• CONFIG - every environment variable and every Setting key
• PLUGIN-QUICKSTART - building a custom plugin for your site
• ADR-004 - per-site instances rationale; ADR-003 bundled vs external plugins; ADR-006 per-plugin collector endpoints
• shopdb/config.py - all the env vars in one place