Score:0

100% CPU usage from iowait caused by Nginx, not solved by turning sendfile on and directio off

us flag

I run a webservice that performs some basic image processing operations.

The webserver first accepts image uploads from users and temporarily stores them. A backend server then downloads the image through a HTTP get request and performs the actual processing. It is then sent back to the webserver. The user then downloads the images. The processed image is typically significantly larger than the original image.

The webservice occasionally encounters iowait spikes with high CPU usage and the server gives a request unavailable timeout error. The high iowait is from nginx.

The solution given for similar problems recommends turning sendfile on and directio off. I've also turned buffer and request_buffers off since I've read that that could also be a source of problem. Although this seems to reduce the occurrence of the problem somewhat, it still sporadically happens and I have no idea why.

I've copied my config files below. Does anyone have any recommendations on what else needs to be changed? The problem is really driving me nuts.

server {
     if ($http_user_agent ~ ^$){
        return 503;
    }

    if ($http_user_agent ~* "Mozilla/4.0\ \(compatible;\ MSIE\ 6.0;\ Windows\ NT\ 5.1;\ SV1;\ .NET\ CLR\ 1.1.4322;\ .NET\ CLR\ 2.0.50727\)") {
        return 503;
    }


    server_name [redacted: site];

    access_log /var/log/[redacted: logfile];
    error_log /var/log/[redacted: logfile];

    listen 0.0.0.0:443 ssl; 
    ssl_certificate [redacted: cert address]
    ssl_certificate_key [redacted: cert key]
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    error_page 502 /custom_502.html;
    location = /custom_502.html {
        root /usr/share/nginx/html;
        internal;
        }
    location = /error_502.png {
        root /usr/share/nginx/html;
    }
    client_max_body_size 300M;
    proxy_request_buffering off;


    location / {
                # forward application requests to the gunicorn server
                proxy_pass http://localhost:8001;
                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
        location /static {
                
                proxy_max_temp_file_size 0;
                directio        off;
                sendfile        on;
                alias [redacted: site location]/static;
                expires 5d;
        }

}


server {
      if ($http_user_agent ~ ^$){
        return 503;
    }

    if ($http_user_agent ~* "Mozilla/4.0\ \(compatible;\ MSIE\ 6.0;\ Windows\ NT\ 5.1;\ SV1;\ .NET\ CLR\ 1.1.4322;\ .NET\ CLR\ 2.0.50727\)") {
        return 503;
    }

  if ($host = [redacted: address]) {
        return 301 https://$host$request_uri;


    listen 0.0.0.0:80;
    server_name [redacted: site]; 
    return 404;     
    client_max_body_size 300M;

}

                                                       
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.