Move API to gevent and restore the claim hold to 25s
The claim poll holds a connection open server-side. Under gthread that pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a worker with 8 of 9 threads parked in timed sleep and UI requests hashed to that worker waiting 12-24s. Hold was cut to 5s as an interim fix. gevent removes the constraint rather than working around it: a held connection is a greenlet and the per-worker ceiling is worker-connections (2000), not 8 threads, so an idle hold no longer costs a UI request anything. Measured at 700 simulated agents plus UI, integration and kiosk traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding resource, not memory or DB connections. With that gone the 5s hold is a liability. Hold length sets the fleet's request rate, not its concurrency, because the agent re-polls immediately on a 204 - so hold=5 is 5x the requests of hold=25, against a burst bottleneck that measured as TLS handshake CPU. 25 also restores the margin against the agent's MinIdlePollDuration=5s floor used to tell a normal hold-window close apart from a transient failure; at 5 it was ~20ms. Timeout raised to 60 so gunicorn cannot kill a healthy long-poll mid-hold, and the access log now carries request duration - without it a slow request is indistinguishable from a fast one and latency has to be reconstructed from outside the container. Healthcheck moved off POST /api/auth/login, which logged a 400 every 30s and exercised the credential path from a probe.
This commit is contained in:
@@ -1828,6 +1828,25 @@ def revoke_agentkey():
|
||||
# How long the agent claim poll is held open server-side before returning empty.
|
||||
# Kept below the agent HttpClient timeout and typical proxy idle timeouts so a
|
||||
# quiet fleet re-polls cleanly instead of tripping a read timeout.
|
||||
#
|
||||
# 25s is the value this endpoint is designed around; it was temporarily dropped
|
||||
# to 5s under the blocking gthread worker pool, where a held connection pinned
|
||||
# one of a worker's 8 OS threads. Measured then: 14 agents at hold=25 left a
|
||||
# single worker with 8 of 9 threads parked in timed sleep and UI requests hashed
|
||||
# to that worker waited 12-24s. Under gevent a held connection is a greenlet and
|
||||
# the per-worker ceiling is --worker-connections (2000), not 8 threads, so an
|
||||
# idle hold no longer costs anything a UI request needs.
|
||||
#
|
||||
# Restored to 25 because hold length sets the fleet's REQUEST RATE, not its
|
||||
# concurrency: the agent re-polls immediately on a 204, so hold=5 is 5x the
|
||||
# requests of hold=25, and TLS handshake CPU was the measured burst bottleneck
|
||||
# (devnotes/gevent-sizing-simulation.md). 25 also restores the margin against
|
||||
# the agent's MinIdlePollDuration=5s floor, which CommandWorker.IsIdlePollClose
|
||||
# uses to tell a normal hold-window close apart from a transient failure; at
|
||||
# hold=5 that margin was ~20ms.
|
||||
#
|
||||
# Constraint: gunicorn --timeout must stay above this or a healthy long-poll is
|
||||
# killed mid-hold. docker-compose.yml sets 60.
|
||||
_CLAIM_HOLD_SECONDS = 25
|
||||
_CLAIM_POLL_INTERVAL = 1.0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user