Merge origin/wj into wilm
wilm was one commit behind wj. The only conflict was in the gunicorn command, where both branches had already made the identical change (gthread, 4 workers, 8 threads, timeout 30) and only the explanatory comments differed. Kept the wj wording, which records the production diagnosis behind the change.
This commit is contained in:
@@ -139,10 +139,16 @@ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
|||||||
-H "Content-Type: application/json" -d '{}' \
|
-H "Content-Type: application/json" -d '{}' \
|
||||||
| grep -q "VALIDATION_ERROR" || exit 1
|
| grep -q "VALIDATION_ERROR" || exit 1
|
||||||
|
|
||||||
|
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
|
||||||
|
# compose file overrides this CMD, but an image run without one (a plain
|
||||||
|
# `docker run`, a site's own orchestrator) would otherwise still get the
|
||||||
|
# blocking sync default and the same starvation.
|
||||||
CMD ["gunicorn", \
|
CMD ["gunicorn", \
|
||||||
"--bind", "0.0.0.0:5001", \
|
"--bind", "0.0.0.0:5001", \
|
||||||
|
"--worker-class", "gthread", \
|
||||||
"--workers", "4", \
|
"--workers", "4", \
|
||||||
"--timeout", "60", \
|
"--threads", "8", \
|
||||||
|
"--timeout", "30", \
|
||||||
"--access-logfile", "-", \
|
"--access-logfile", "-", \
|
||||||
"--error-logfile", "-", \
|
"--error-logfile", "-", \
|
||||||
"wsgi:app"]
|
"wsgi:app"]
|
||||||
|
|||||||
@@ -79,18 +79,25 @@ services:
|
|||||||
- gunicorn
|
- gunicorn
|
||||||
- --bind
|
- --bind
|
||||||
- 0.0.0.0:5001
|
- 0.0.0.0:5001
|
||||||
# Threaded workers so one blocked request (e.g. an agent that fails the TLS
|
# Threaded workers, not the blocking sync default. An agent that fails
|
||||||
# handshake and retries in a loop) cannot pin a whole worker and starve the
|
# the TLS handshake retries in a loop, and with 4 sync workers a handful
|
||||||
# UI. With sync workers a handful of stuck clients exhausted all capacity.
|
# of those pinned every worker: real UI requests queued 20+ seconds while
|
||||||
- --worker-class
|
# the CPU sat idle. A stuck connection now holds one THREAD, and 32
|
||||||
- gthread
|
# (4 x 8) have to be exhausted before a person waits.
|
||||||
- --workers
|
#
|
||||||
- "4"
|
# Found in production at the other site (their commit db75ca7). Their
|
||||||
- --threads
|
# server, our agent - so the same client behaviour will reach any site
|
||||||
- "8"
|
# running the endpoint agent, and the fix belongs in the product rather
|
||||||
# Shorter timeout so a hung connection frees its thread quickly instead of
|
# than on one branch.
|
||||||
# holding it for a full minute.
|
- --worker-class
|
||||||
- --timeout
|
- gthread
|
||||||
|
- --workers
|
||||||
|
- "4"
|
||||||
|
- --threads
|
||||||
|
- "8"
|
||||||
|
# 30, not 60: a hung connection frees its thread in half the time, which
|
||||||
|
# is what decides whether a retry storm is absorbed or queues.
|
||||||
|
- --timeout
|
||||||
- "30"
|
- "30"
|
||||||
- --access-logfile
|
- --access-logfile
|
||||||
- "-"
|
- "-"
|
||||||
|
|||||||
Reference in New Issue
Block a user