I'm hosting a Django-App which serves as an API-Endpoint. Unfortunately the App which uses the API does a lot of concurrent requests on page-load (in the realm of 80-90 requests).
Nginx is running as the reverse-proxy in front of gunicorn and reports the following problem:
kevent() reported that connect() failed (54: Connection reset by peer) while connecting to upstream
That lead me to scaling gunicorn. I tried -k gevent
and increasing --worker-connections
, but it didn't help much. I had about the same success with --threads
and sync-workers.
I noticed a few occurences of Listen queue overflow: 16 already in queue awaiting acceptance (55 occurrences)
in dmesg
, which lead me to increasing kern.ipc.somaxconn
to as much as 4096, but again: Without success.
As per the gunicorn-documentation, I use hey
to see if the proxy is doing the right thing and load-testing the Endpoint:
hey -c 100 -n 200 -H "Authorization: token ${token}" 'https://example.com/api/'
Which returns something along those lines (sometimes a bit better, sometimes a bit worse):
Status code distribution:
[200] 126 responses
[502] 74 responses
Relevant nginx-config:
worker_processes auto;
events {
worker_connections 1024;
accept_mutex on;
use kqueue;
}
upstream backend-api {
server 127.0.0.1:8000 fail_timeout=0;
}
server {
listen 443 ssl http2 accept_filter=httpready;
server_name example.com;
ssl_certificate /usr/local/etc/nginx/sites/example.com.crt;
ssl_certificate_key /usr/local/etc/nginx/sites/example.com.key;
location = /favicon.ico { access_log off; log_not_found off; }
root /usr/local/www/example.com/web;
index index.html ;
keepalive_timeout 5;
location ~ ^/(static|assets|index.html|$) {
root /usr/local/www/example.com/web;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_intercept_errors on;
proxy_pass http://backend-api;
}
}
Running gunicorn with:
gunicorn --workers=9 --bind 0.0.0.0:8000 --forwarded-allow-ips='*' -k gevent --worker-connections=1000 api.wsgi'
I can't seem to find a solution to this. No matter what I try, I can only get "less" 502's on a page-load, but never reliably none. What am I missing?
Gunicorn does not report any problems - the requests never seem to make it that far.