The claim poll holds a connection open server-side. Under gthread that pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a worker with 8 of 9 threads parked in timed sleep and UI requests hashed to that worker waiting 12-24s. Hold was cut to 5s as an interim fix. gevent removes the constraint rather than working around it: a held connection is a greenlet and the per-worker ceiling is worker-connections (2000), not 8 threads, so an idle hold no longer costs a UI request anything. Measured at 700 simulated agents plus UI, integration and kiosk traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding resource, not memory or DB connections. With that gone the 5s hold is a liability. Hold length sets the fleet's request rate, not its concurrency, because the agent re-polls immediately on a 204 - so hold=5 is 5x the requests of hold=25, against a burst bottleneck that measured as TLS handshake CPU. 25 also restores the margin against the agent's MinIdlePollDuration=5s floor used to tell a normal hold-window close apart from a transient failure; at 5 it was ~20ms. Timeout raised to 60 so gunicorn cannot kill a healthy long-poll mid-hold, and the access log now carries request duration - without it a slow request is indistinguishable from a fast one and latency has to be reconstructed from outside the container. Healthcheck moved off POST /api/auth/login, which logged a 400 every 30s and exercised the credential path from a probe.
157 lines
7.2 KiB
YAML
157 lines
7.2 KiB
YAML
# shopdb-flask single-site docker-compose template.
|
|
#
|
|
# Per ADR-004, each adopting facility runs its own stack. This template
|
|
# brings up MySQL + the API container and exposes the API on port 5001.
|
|
# The Vue frontend is served separately by the API in production builds
|
|
# (see register_frontend_routes in shopdb/__init__.py); for dev, run
|
|
# `npm run dev` in frontend/ on a separate port.
|
|
#
|
|
# Usage:
|
|
# cp .env.example .env
|
|
# # edit .env with site-specific secrets and origins
|
|
# docker compose up -d
|
|
#
|
|
# Refresh after pulling new code:
|
|
# docker compose build api
|
|
# docker compose up -d api
|
|
|
|
services:
|
|
db:
|
|
image: mysql:8.4
|
|
# utf8mb4 server-wide so the auto-created MYSQL_DATABASE is utf8mb4, not the
|
|
# image default. Keeps every site's schema on the same charset/collation.
|
|
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
|
|
restart: unless-stopped
|
|
environment:
|
|
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:?MYSQL_ROOT_PASSWORD must be set}
|
|
MYSQL_DATABASE: shopdb_flask
|
|
MYSQL_USER: shopdb
|
|
MYSQL_PASSWORD: ${MYSQL_PASSWORD:?MYSQL_PASSWORD must be set}
|
|
volumes:
|
|
- db_data:/var/lib/mysql
|
|
# Bind MySQL to loopback only. The api container reaches db over the
|
|
# compose network regardless of this mapping; the published port is just
|
|
# for local admin tools (mysqldump, a client on the host). Exposing 3306
|
|
# on all interfaces would put the database on the facility network.
|
|
ports:
|
|
- "127.0.0.1:${MYSQL_PORT:-3306}:3306"
|
|
healthcheck:
|
|
# 127.0.0.1, not localhost: localhost means the unix socket, and the
|
|
# entrypoint's init pass answers on the socket while running the server
|
|
# with --skip-networking. A socket ping therefore reports healthy DURING
|
|
# init, and the api/migrate services start against a server that is about
|
|
# to restart. Over TCP the probe stays red until the real server listens.
|
|
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
|
|
api:
|
|
build: .
|
|
restart: unless-stopped
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
environment:
|
|
FLASK_ENV: production
|
|
DATABASE_URL: mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask?charset=utf8mb4
|
|
SECRET_KEY: ${SECRET_KEY:?SECRET_KEY must be set}
|
|
JWT_SECRET_KEY: ${JWT_SECRET_KEY:?JWT_SECRET_KEY must be set}
|
|
CORS_ORIGINS: ${CORS_ORIGINS:?CORS_ORIGINS must be set}
|
|
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
|
ZABBIX_URL: ${ZABBIX_URL:-}
|
|
ZABBIX_TOKEN: ${ZABBIX_TOKEN:-}
|
|
COLLECTOR_API_KEY: ${COLLECTOR_API_KEY:-}
|
|
COLLECTOR_API_KEY_ENDPOINTS: ${COLLECTOR_API_KEY_ENDPOINTS:-}
|
|
# This file is the source-build dev stack (wilm/dev-be); stable and edge
|
|
# ship through the pull/airgap compose files, not this one. So default the
|
|
# badge label to DEV here - a raw source build is by definition not a
|
|
# release - while still allowing an override from .env.
|
|
SHOPDB_ENV_LABEL: ${SHOPDB_ENV_LABEL:-DEV}
|
|
ports:
|
|
- "${API_PORT:-5001}:5001"
|
|
# Terminate TLS in gunicorn with the mounted self-signed cert so the browser
|
|
# sees a secure context. navigator.clipboard and other secure-context-only
|
|
# web APIs refuse to run over plain http on a non-localhost origin, so the
|
|
# test box (accessed by FQDN) needs real https. A site swaps in its own cert
|
|
# under ./certs or fronts this with its IIS/nginx proxy.
|
|
command:
|
|
- gunicorn
|
|
- --bind
|
|
- 0.0.0.0:5001
|
|
# Async workers, not threads. The endpoint agent long-polls
|
|
# /api/endpoints/commands/next and re-polls immediately on a 204, so every
|
|
# agent holds a connection essentially continuously and fleet size IS the
|
|
# concurrency. Under gthread that meant 4 x 8 = 32 slots total, and a
|
|
# single worker's 8 threads could fill with sleeping agents while the CPU
|
|
# sat idle - measured: 8 of 9 threads in timed sleep, UI p95 24s.
|
|
#
|
|
# A held connection under gevent is a greenlet, not an OS thread, so idle
|
|
# is nearly free. Measured on this stack (devnotes/gevent-sizing-simulation.md):
|
|
# 700 simulated agents plus UI, integration and kiosk traffic served with
|
|
# zero errors at UI median 7.5ms, peak RSS 479MB. CPU, not memory or DB
|
|
# connections, is the binding resource.
|
|
- --worker-class
|
|
- gevent
|
|
- --workers
|
|
- "4"
|
|
# Per-worker greenlet ceiling. 4 x 2000 is far above any realistic fleet;
|
|
# it exists so a connection spike queues rather than being refused.
|
|
- --worker-connections
|
|
- "2000"
|
|
# 60, not 30: the timeout must exceed _CLAIM_HOLD_SECONDS in
|
|
# plugins/endpoints/api/routes.py or gunicorn kills a healthy long-poll
|
|
# mid-hold. Cheap here because a stuck connection costs a greenlet.
|
|
- --timeout
|
|
- "60"
|
|
- --access-logfile
|
|
- "-"
|
|
# %(D)s is request duration in microseconds. Without it a slow request is
|
|
# indistinguishable from a fast one in the log, and latency has to be
|
|
# reconstructed from outside the container. Keep in sync with the
|
|
# Dockerfile CMD.
|
|
- --access-logformat
|
|
- '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
|
|
- --error-logfile
|
|
- "-"
|
|
- --certfile
|
|
- /app/certs/server.crt
|
|
- --keyfile
|
|
- /app/certs/server.key
|
|
- wsgi:app
|
|
healthcheck:
|
|
# Match the TLS listener: probe over https and accept the self-signed cert
|
|
# (-k). The Dockerfile default probes http and would stay red once gunicorn
|
|
# only speaks TLS.
|
|
# /api/dashboard/health is the public probe route and answers 200 even when
|
|
# the database is down, so the body's status field is what decides health.
|
|
# Probing POST /api/auth/login instead logged a 400 every 30s and exercised
|
|
# the credential path from a health check.
|
|
test: ["CMD-SHELL", "curl --silent --insecure https://localhost:5001/api/dashboard/health | grep -q '\"status\":\"ok\"'"]
|
|
interval: 30s
|
|
timeout: 5s
|
|
start_period: 10s
|
|
retries: 3
|
|
volumes:
|
|
- ./plugins:/app/plugins:ro
|
|
# Standalone operator scripts shipped inside generated bundles (the
|
|
# endpoint-agent uninstaller). Read-only, same pattern as plugins: the
|
|
# site-agent packaging route reads uninstall-shopdbagent.ps1 from here at
|
|
# download time, so it must be visible in the container.
|
|
- ./scripts:/app/scripts:ro
|
|
# Self-signed TLS material for the dev/test https listener. Read-only;
|
|
# generated on the host under ./certs and never baked into the image.
|
|
- ./certs:/app/certs:ro
|
|
# /app/instance is WRITTEN state, not code: plugins.json (which plugins
|
|
# this site has enabled), uploaded floor plans, branding, model and
|
|
# application images, employee photos, warranty proofs, slides,
|
|
# printed-part files, and the Dell OAuth token. Without this volume a
|
|
# `docker compose build api && up -d api` recreates the container and
|
|
# takes all of it with it, so the site comes back with its plugins
|
|
# disabled and MySQL rows pointing at files that no longer exist.
|
|
- instance_data:/app/instance
|
|
|
|
volumes:
|
|
db_data:
|
|
instance_data:
|