Move API to gevent and restore the claim hold to 25s

The claim poll holds a connection open server-side. Under gthread that
pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a
worker with 8 of 9 threads parked in timed sleep and UI requests hashed
to that worker waiting 12-24s. Hold was cut to 5s as an interim fix.

gevent removes the constraint rather than working around it: a held
connection is a greenlet and the per-worker ceiling is worker-connections
(2000), not 8 threads, so an idle hold no longer costs a UI request
anything. Measured at 700 simulated agents plus UI, integration and kiosk
traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding
resource, not memory or DB connections.

With that gone the 5s hold is a liability. Hold length sets the fleet's
request rate, not its concurrency, because the agent re-polls immediately
on a 204 - so hold=5 is 5x the requests of hold=25, against a burst
bottleneck that measured as TLS handshake CPU. 25 also restores the
margin against the agent's MinIdlePollDuration=5s floor used to tell a
normal hold-window close apart from a transient failure; at 5 it was
~20ms.

Timeout raised to 60 so gunicorn cannot kill a healthy long-poll
mid-hold, and the access log now carries request duration - without it a
slow request is indistinguishable from a fast one and latency has to be
reconstructed from outside the container. Healthcheck moved off
POST /api/auth/login, which logged a 400 every 30s and exercised the
credential path from a probe.
This commit is contained in:
benjamin.baker
2026-09-06 18:08:29 -04:00
parent 7905917cd4
commit 6ae251d2ca
3 changed files with 69 additions and 25 deletions

View File

@@ -1828,6 +1828,25 @@ def revoke_agentkey():
# How long the agent claim poll is held open server-side before returning empty.
# Kept below the agent HttpClient timeout and typical proxy idle timeouts so a
# quiet fleet re-polls cleanly instead of tripping a read timeout.
#
# 25s is the value this endpoint is designed around; it was temporarily dropped
# to 5s under the blocking gthread worker pool, where a held connection pinned
# one of a worker's 8 OS threads. Measured then: 14 agents at hold=25 left a
# single worker with 8 of 9 threads parked in timed sleep and UI requests hashed
# to that worker waited 12-24s. Under gevent a held connection is a greenlet and
# the per-worker ceiling is --worker-connections (2000), not 8 threads, so an
# idle hold no longer costs anything a UI request needs.
#
# Restored to 25 because hold length sets the fleet's REQUEST RATE, not its
# concurrency: the agent re-polls immediately on a 204, so hold=5 is 5x the
# requests of hold=25, and TLS handshake CPU was the measured burst bottleneck
# (devnotes/gevent-sizing-simulation.md). 25 also restores the margin against
# the agent's MinIdlePollDuration=5s floor, which CommandWorker.IsIdlePollClose
# uses to tell a normal hold-window close apart from a transient failure; at
# hold=5 that margin was ~20ms.
#
# Constraint: gunicorn --timeout must stay above this or a healthy long-poll is
# killed mid-hold. docker-compose.yml sets 60.
_CLAIM_HOLD_SECONDS = 25
_CLAIM_POLL_INTERVAL = 1.0