b5737f47c1
Taken from the other site, who hit it in production today (their commit
db75ca7): agents that fail the TLS handshake retry in a loop, and with four
blocking sync workers a handful of them pinned every worker. Real UI requests
queued for 20+ seconds while the CPU sat idle - the box was not busy, it was
blocked.
gthread, 4 workers x 8 threads, and a 30-second timeout rather than 60. A stuck
connection now holds one thread out of 32 instead of one worker out of four, and
it lets go in half the time, which is the difference between absorbing a retry
storm and queueing behind it.
Their server, our agent. The client behaviour that caused it is the endpoint
agent, so it will reach any site running one - which makes this a product fix
rather than something that belongs on one site's branch.
Applied to the Dockerfile as well as docker-compose.yml. A compose file
overrides the image CMD, but docker-compose.airgap.yml, .channel.yml and
.pull.yml do not specify a command at all, so they take the image default - and
so does anybody running the image by hand. Fixing only compose would have left
every air-gapped install on sync workers.
WJ production is unaffected either way: it runs waitress under IIS, already with
--threads=8, so it was never exposed to this. That is worth knowing before
anyone treats it as a prod hardening item that was missing.
Scope: product
6.2 KiB
6.2 KiB