Nginx error - 110: Connection timed out while connecting to upstream(django-python)

The error is

2021/08/18 8:57:28 [error] 19915#19915: *36133 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.11.12.1(proxy-ip), server: example.com, request: "GET /static/css/bootstrap.min.css HTTP/1.1", upstream: "http://127.0.0.1:8000/static/css/bootstrap.min.css", host: "www.example.com"

error brief description

The website is running fine always, but some times there is this above error which i think is due to the high traffic to the website, when the website goes down, the site will not come up immediately after restarting nginx and supervisor also

Some times it will take 5 to 6 hours for the site to come up

Servers - Ubuntu-18.04 LTS

webserver configs as follow

i have a proxy server and application server

1. proxy server - nginx

2. application server - running nginx & (supervisor - django application)

3. proxy server -nginx-config

    server {
   
        listen 443 ssl http2;
        ssl_certificate /etc/nginx/ssl/bundle.crt;
        ssl_certificate_key /etc/nginx/ssl/start.example.com.key;
        server_name example.com www.example.com ;
        location = /basic_status {
        stub_status;
        access_log   off;
        allow 1.2.3.4;
        deny all;
        }
        location /{
   
        proxy_connect_timeout       300;
        proxy_send_timeout          300;
        proxy_read_timeout          300;
        send_timeout                300;
        proxy_pass http://10.11.12.2;  #proxy to application server
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        add_header Access-Control-Allow-Origin .example.com;
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
        add_header X-XSS-Protection "1; mode=block";
        add_header "Pragma" "no-cache"; 
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Scheme $scheme;
   
        #add_header Content-Security-Policy "default-src 'unsafe-inline' 'self'; script-src 'unsafe-inline' 'self' ";
        error_page 404 /404.html;
        location = /404.html {
        root /var/www/error;
        internal;
        }
        error_page 500 502 503 504 /500.html;
        location = /500.html {
        root /usr/share/nginx/html;
        internal;
        }
   
    }
   

4. application server - nginx

   server {
            listen 80 default_server;
            listen [::]:80 default_server;
            server_name example.com www.example.com;
   
            location /{
   
                proxy_pass http://127.0.0.1:8000;
                proxy_read_timeout 180;
   
            }                                                                                                                    
     }
   

5. application-server - supervisor

    [program:portal]
    command =/root/portal_env/bin/gunicorn  portal.wsgi:application -b 0.0.0.0:8000 --timeout 180 --workers=3 ;
    user = root                                                ; User to run as
    directory = /root/portal_env/portal
    stdout_logfile = /root/portal_env/logs/portal.log ; Where to write log messages
    redirect_stderr = true                                       ; Save stderr in the same log
    autostart = true
    autorestart = true
    environment = LANG = en_US.UTF-8,LC_ALL = en_US.UTF-8              ; Set UTF-8 as default encoding
   

First I will come with some feedback for your setup.

When you are building Python application its not recommended to use gunicorn as server host, its build for developing, you shut use uWSGI for your Python application.

Next when you are using proxy in NGINX I will recommend you to disable buffering for your proxy

proxy_buffering                 off;

When you are disable buffering you do the site a little bit slower in load time for the client becures its need to sending the full package from the server to the client. my expirers tell me its not so much when we are speaking web developing and its protect you in the long term if you want to run your application on multi server behind a load balancer like NGINX Proxy, Docker Cluster or something like that.

After my feedback, its sound like you have a something in your code there just go into a loop and never stop or you have a lonnnnnnnng SQL connection/query there take more then 3min to execute.

What I will recommend here its enable slow-log on your database so you can get a log for all querys there take more then 0.5sec its can be a missing index in your database there out perfomes the site.

Normal Python did not take 3min to execute a script/code area so its sound like a external services eg. database, api or something in this case.

When you have enabled slow log for your database, a secound case will be create a simple log for your script with "start time" + "end time" and dump it with the method/page/url there are hitting this area and dump it to a file so you easy can open it, late it run in the 5-6 hours as you speak about or longer to catch your error.

I think its the best way, first problem will be to find why you code take 3min to execute.