Score:0

100 concurrent requests to a Django-App with Gunicorn lead to "54: Connection reset by peer" and 502's

ru flag

I'm hosting a Django-App which serves as an API-Endpoint. Unfortunately the App which uses the API does a lot of concurrent requests on page-load (in the realm of 80-90 requests).

Nginx is running as the reverse-proxy in front of gunicorn and reports the following problem: kevent() reported that connect() failed (54: Connection reset by peer) while connecting to upstream

That lead me to scaling gunicorn. I tried -k gevent and increasing --worker-connections, but it didn't help much. I had about the same success with --threads and sync-workers.

I noticed a few occurences of Listen queue overflow: 16 already in queue awaiting acceptance (55 occurrences) in dmesg, which lead me to increasing kern.ipc.somaxconn to as much as 4096, but again: Without success.

As per the gunicorn-documentation, I use hey to see if the proxy is doing the right thing and load-testing the Endpoint: hey -c 100 -n 200 -H "Authorization: token ${token}" 'https://example.com/api/'

Which returns something along those lines (sometimes a bit better, sometimes a bit worse):

Status code distribution:
  [200] 126 responses
  [502] 74 responses

Relevant nginx-config:

worker_processes  auto;

events {
    worker_connections  1024;
    accept_mutex on;
    use kqueue;
}

upstream backend-api {
  server 127.0.0.1:8000 fail_timeout=0;
}

server {
    listen 443 ssl http2 accept_filter=httpready;
    server_name example.com;

    ssl_certificate /usr/local/etc/nginx/sites/example.com.crt;
    ssl_certificate_key /usr/local/etc/nginx/sites/example.com.key;

    location = /favicon.ico { access_log off; log_not_found off; }
    root /usr/local/www/example.com/web;
    index index.html ;

    keepalive_timeout 5;

    location ~  ^/(static|assets|index.html|$) {
        root /usr/local/www/example.com/web;
    }

    location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Real-IP $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_intercept_errors on;
      proxy_pass http://backend-api;
    }
}

Running gunicorn with: gunicorn --workers=9 --bind 0.0.0.0:8000 --forwarded-allow-ips='*' -k gevent --worker-connections=1000 api.wsgi'

I can't seem to find a solution to this. No matter what I try, I can only get "less" 502's on a page-load, but never reliably none. What am I missing?

Gunicorn does not report any problems - the requests never seem to make it that far.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.