Move API to gevent and restore the claim hold to 25s
The claim poll holds a connection open server-side. Under gthread that pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a worker with 8 of 9 threads parked in timed sleep and UI requests hashed to that worker waiting 12-24s. Hold was cut to 5s as an interim fix. gevent removes the constraint rather than working around it: a held connection is a greenlet and the per-worker ceiling is worker-connections (2000), not 8 threads, so an idle hold no longer costs a UI request anything. Measured at 700 simulated agents plus UI, integration and kiosk traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding resource, not memory or DB connections. With that gone the 5s hold is a liability. Hold length sets the fleet's request rate, not its concurrency, because the agent re-polls immediately on a 204 - so hold=5 is 5x the requests of hold=25, against a burst bottleneck that measured as TLS handshake CPU. 25 also restores the margin against the agent's MinIdlePollDuration=5s floor used to tell a normal hold-window close apart from a transient failure; at 5 it was ~20ms. Timeout raised to 60 so gunicorn cannot kill a healthy long-poll mid-hold, and the access log now carries request duration - without it a slow request is indistinguishable from a fast one and latency has to be reconstructed from outside the container. Healthcheck moved off POST /api/auth/login, which logged a 400 every 30s and exercised the credential path from a probe.
This commit is contained in:
27
Dockerfile
27
Dockerfile
@@ -109,8 +109,13 @@ RUN apt-get update \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt ./
|
||||
# gunicorn and gevent are container-only: requirements.txt is a universal,
|
||||
# hash-checked lockfile that also serves the Windows/IIS install, where waitress
|
||||
# is the server and neither of these is used. gevent is the worker class (see
|
||||
# the CMD below) because the endpoint long-poll holds a connection per agent,
|
||||
# which a thread pool cannot absorb at fleet scale.
|
||||
RUN pip install --no-cache-dir -r requirements.txt \
|
||||
&& pip install --no-cache-dir gunicorn
|
||||
&& pip install --no-cache-dir gunicorn gevent
|
||||
|
||||
COPY shopdb/ ./shopdb/
|
||||
COPY plugins/ ./plugins/
|
||||
@@ -134,21 +139,27 @@ USER shopdb
|
||||
|
||||
EXPOSE 5001
|
||||
|
||||
# Probe the public health route, not the login endpoint. Probing POST
|
||||
# /api/auth/login logged a 400 every 30s and ran the credential path from a
|
||||
# health check. The route answers 200 even when the database is degraded, so the
|
||||
# body's status field is what decides health. Compose files that terminate TLS
|
||||
# override this with an https probe; plain http is correct for the pull and
|
||||
# airgap stacks, which inherit this unchanged.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
CMD curl --silent http://localhost:5001/api/auth/login -X POST \
|
||||
-H "Content-Type: application/json" -d '{}' \
|
||||
| grep -q "VALIDATION_ERROR" || exit 1
|
||||
CMD curl --silent http://localhost:5001/api/dashboard/health \
|
||||
| grep -q '"status":"ok"' || exit 1
|
||||
|
||||
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
|
||||
# Async workers, matching docker-compose.yml - see the reasoning there. A
|
||||
# compose file overrides this CMD, but an image run without one (a plain
|
||||
# `docker run`, a site's own orchestrator) would otherwise still get the
|
||||
# blocking sync default and the same starvation.
|
||||
CMD ["gunicorn", \
|
||||
"--bind", "0.0.0.0:5001", \
|
||||
"--worker-class", "gthread", \
|
||||
"--worker-class", "gevent", \
|
||||
"--workers", "4", \
|
||||
"--threads", "8", \
|
||||
"--timeout", "30", \
|
||||
"--worker-connections", "2000", \
|
||||
"--timeout", "60", \
|
||||
"--access-logfile", "-", \
|
||||
"--access-logformat", "%(h)s %(l)s %(u)s %(t)s \"%(r)s\" %(s)s %(b)s %(D)sus \"%(f)s\" \"%(a)s\"", \
|
||||
"--error-logfile", "-", \
|
||||
"wsgi:app"]
|
||||
|
||||
Reference in New Issue
Block a user