Move API to gevent and restore the claim hold to 25s

The claim poll holds a connection open server-side. Under gthread that
pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a
worker with 8 of 9 threads parked in timed sleep and UI requests hashed
to that worker waiting 12-24s. Hold was cut to 5s as an interim fix.

gevent removes the constraint rather than working around it: a held
connection is a greenlet and the per-worker ceiling is worker-connections
(2000), not 8 threads, so an idle hold no longer costs a UI request
anything. Measured at 700 simulated agents plus UI, integration and kiosk
traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding
resource, not memory or DB connections.

With that gone the 5s hold is a liability. Hold length sets the fleet's
request rate, not its concurrency, because the agent re-polls immediately
on a 204 - so hold=5 is 5x the requests of hold=25, against a burst
bottleneck that measured as TLS handshake CPU. 25 also restores the
margin against the agent's MinIdlePollDuration=5s floor used to tell a
normal hold-window close apart from a transient failure; at 5 it was
~20ms.

Timeout raised to 60 so gunicorn cannot kill a healthy long-poll
mid-hold, and the access log now carries request duration - without it a
slow request is indistinguishable from a fast one and latency has to be
reconstructed from outside the container. Healthcheck moved off
POST /api/auth/login, which logged a 400 every 30s and exercised the
credential path from a probe.

docker-compose.yml

@@ -79,28 +79,39 @@ services:
       - gunicorn
       - --bind
       - 0.0.0.0:5001
-      # Threaded workers, not the blocking sync default. An agent that fails
-      # the TLS handshake retries in a loop, and with 4 sync workers a handful
-      # of those pinned every worker: real UI requests queued 20+ seconds while
-      # the CPU sat idle. A stuck connection now holds one THREAD, and 32
-      # (4 x 8) have to be exhausted before a person waits.
+      # Async workers, not threads. The endpoint agent long-polls
+      # /api/endpoints/commands/next and re-polls immediately on a 204, so every
+      # agent holds a connection essentially continuously and fleet size IS the
+      # concurrency. Under gthread that meant 4 x 8 = 32 slots total, and a
+      # single worker's 8 threads could fill with sleeping agents while the CPU
+      # sat idle - measured: 8 of 9 threads in timed sleep, UI p95 24s.
       #
-      # Found in production at the other site (their commit db75ca7). Their
-      # server, our agent - so the same client behaviour will reach any site
-      # running the endpoint agent, and the fix belongs in the product rather
-      # than on one branch.
+      # A held connection under gevent is a greenlet, not an OS thread, so idle
+      # is nearly free. Measured on this stack (devnotes/gevent-sizing-simulation.md):
+      # 700 simulated agents plus UI, integration and kiosk traffic served with
+      # zero errors at UI median 7.5ms, peak RSS 479MB. CPU, not memory or DB
+      # connections, is the binding resource.
       - --worker-class
-      - gthread
+      - gevent
       - --workers
       - "4"
-      - --threads
-      - "8"
-      # 30, not 60: a hung connection frees its thread in half the time, which
-      # is what decides whether a retry storm is absorbed or queues.
+      # Per-worker greenlet ceiling. 4 x 2000 is far above any realistic fleet;
+      # it exists so a connection spike queues rather than being refused.
+      - --worker-connections
+      - "2000"
+      # 60, not 30: the timeout must exceed _CLAIM_HOLD_SECONDS in
+      # plugins/endpoints/api/routes.py or gunicorn kills a healthy long-poll
+      # mid-hold. Cheap here because a stuck connection costs a greenlet.
       - --timeout
-      - "30"
+      - "60"
       - --access-logfile
       - "-"
+      # %(D)s is request duration in microseconds. Without it a slow request is
+      # indistinguishable from a fast one in the log, and latency has to be
+      # reconstructed from outside the container. Keep in sync with the
+      # Dockerfile CMD.
+      - --access-logformat
+      - '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
       - --error-logfile
       - "-"
       - --certfile
@@ -112,8 +123,11 @@ services:
       # Match the TLS listener: probe over https and accept the self-signed cert
       # (-k). The Dockerfile default probes http and would stay red once gunicorn
       # only speaks TLS.
-      test: ["CMD", "curl", "--silent", "--insecure", "https://localhost:5001/api/auth/login",
-             "-X", "POST", "-H", "Content-Type: application/json", "-d", "{}"]
+      # /api/dashboard/health is the public probe route and answers 200 even when
+      # the database is down, so the body's status field is what decides health.
+      # Probing POST /api/auth/login instead logged a 400 every 30s and exercised
+      # the credential path from a health check.
+      test: ["CMD-SHELL", "curl --silent --insecure https://localhost:5001/api/dashboard/health | grep -q '\"status\":\"ok\"'"]
       interval: 30s
       timeout: 5s
       start_period: 10s