Threaded gunicorn workers so a retry storm cannot starve the UI

This commit is contained in:
2026-09-03 17:38:42 -04:00
parent fd513c08eb
commit cb3975049c
2 changed files with 24 additions and 2 deletions

View File

@@ -139,10 +139,16 @@ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
-H "Content-Type: application/json" -d '{}' \
| grep -q "VALIDATION_ERROR" || exit 1
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
# compose file overrides this CMD, but an image run without one (a plain
# `docker run`, a site's own orchestrator) would otherwise still get the
# blocking sync default and the same starvation.
CMD ["gunicorn", \
"--bind", "0.0.0.0:5001", \
"--worker-class", "gthread", \
"--workers", "4", \
"--timeout", "60", \
"--threads", "8", \
"--timeout", "30", \
"--access-logfile", "-", \
"--error-logfile", "-", \
"wsgi:app"]

View File

@@ -79,10 +79,26 @@ services:
- gunicorn
- --bind
- 0.0.0.0:5001
# Threaded workers, not the blocking sync default. An agent that fails
# the TLS handshake retries in a loop, and with 4 sync workers a handful
# of those pinned every worker: real UI requests queued 20+ seconds while
# the CPU sat idle. A stuck connection now holds one THREAD, and 32
# (4 x 8) have to be exhausted before a person waits.
#
# Found in production at the other site (their commit db75ca7). Their
# server, our agent - so the same client behaviour will reach any site
# running the endpoint agent, and the fix belongs in the product rather
# than on one branch.
- --worker-class
- gthread
- --workers
- "4"
- --threads
- "8"
# 30, not 60: a hung connection frees its thread in half the time, which
# is what decides whether a retry storm is absorbed or queues.
- --timeout
- "60"
- "30"
- --access-logfile
- "-"
- --error-logfile