Move API to gevent and restore the claim hold to 25s

The claim poll holds a connection open server-side. Under gthread that
pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a
worker with 8 of 9 threads parked in timed sleep and UI requests hashed
to that worker waiting 12-24s. Hold was cut to 5s as an interim fix.

gevent removes the constraint rather than working around it: a held
connection is a greenlet and the per-worker ceiling is worker-connections
(2000), not 8 threads, so an idle hold no longer costs a UI request
anything. Measured at 700 simulated agents plus UI, integration and kiosk
traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding
resource, not memory or DB connections.

With that gone the 5s hold is a liability. Hold length sets the fleet's
request rate, not its concurrency, because the agent re-polls immediately
on a 204 - so hold=5 is 5x the requests of hold=25, against a burst
bottleneck that measured as TLS handshake CPU. 25 also restores the
margin against the agent's MinIdlePollDuration=5s floor used to tell a
normal hold-window close apart from a transient failure; at 5 it was
~20ms.

Timeout raised to 60 so gunicorn cannot kill a healthy long-poll
mid-hold, and the access log now carries request duration - without it a
slow request is indistinguishable from a fast one and latency has to be
reconstructed from outside the container. Healthcheck moved off
POST /api/auth/login, which logged a 400 every 30s and exercised the
credential path from a probe.
This commit is contained in:
benjamin.baker
2026-09-06 18:08:29 -04:00
parent 7905917cd4
commit 6ae251d2ca
3 changed files with 69 additions and 25 deletions

View File

@@ -109,8 +109,13 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt ./
# gunicorn and gevent are container-only: requirements.txt is a universal,
# hash-checked lockfile that also serves the Windows/IIS install, where waitress
# is the server and neither of these is used. gevent is the worker class (see
# the CMD below) because the endpoint long-poll holds a connection per agent,
# which a thread pool cannot absorb at fleet scale.
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir gunicorn
&& pip install --no-cache-dir gunicorn gevent
COPY shopdb/ ./shopdb/
COPY plugins/ ./plugins/
@@ -134,21 +139,27 @@ USER shopdb
EXPOSE 5001
# Probe the public health route, not the login endpoint. Probing POST
# /api/auth/login logged a 400 every 30s and ran the credential path from a
# health check. The route answers 200 even when the database is degraded, so the
# body's status field is what decides health. Compose files that terminate TLS
# override this with an https probe; plain http is correct for the pull and
# airgap stacks, which inherit this unchanged.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl --silent http://localhost:5001/api/auth/login -X POST \
-H "Content-Type: application/json" -d '{}' \
| grep -q "VALIDATION_ERROR" || exit 1
CMD curl --silent http://localhost:5001/api/dashboard/health \
| grep -q '"status":"ok"' || exit 1
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
# Async workers, matching docker-compose.yml - see the reasoning there. A
# compose file overrides this CMD, but an image run without one (a plain
# `docker run`, a site's own orchestrator) would otherwise still get the
# blocking sync default and the same starvation.
CMD ["gunicorn", \
"--bind", "0.0.0.0:5001", \
"--worker-class", "gthread", \
"--worker-class", "gevent", \
"--workers", "4", \
"--threads", "8", \
"--timeout", "30", \
"--worker-connections", "2000", \
"--timeout", "60", \
"--access-logfile", "-", \
"--access-logformat", "%(h)s %(l)s %(u)s %(t)s \"%(r)s\" %(s)s %(b)s %(D)sus \"%(f)s\" \"%(a)s\"", \
"--error-logfile", "-", \
"wsgi:app"]