I run a webservice that performs some basic image processing operations.
The webserver first accepts image uploads from users and temporarily stores them. A backend server then downloads the image through a HTTP get request and performs the actual processing. It is then sent back to the webserver. The user then downloads the images. The processed image is typically significantly larger than the original image.
The webservice occasionally encounters iowait spikes with high CPU usage and the server gives a request unavailable timeout error. The high iowait is from nginx.
The solution given for similar problems recommends turning sendfile on and directio off. I've also turned buffer and request_buffers off since I've read that that could also be a source of problem. Although this seems to reduce the occurrence of the problem somewhat, it still sporadically happens and I have no idea why.
I've copied my config files below. Does anyone have any recommendations on what else needs to be changed? The problem is really driving me nuts.
server {
if ($http_user_agent ~ ^$){
return 503;
}
if ($http_user_agent ~* "Mozilla/4.0\ \(compatible;\ MSIE\ 6.0;\ Windows\ NT\ 5.1;\ SV1;\ .NET\ CLR\ 1.1.4322;\ .NET\ CLR\ 2.0.50727\)") {
return 503;
}
server_name [redacted: site];
access_log /var/log/[redacted: logfile];
error_log /var/log/[redacted: logfile];
listen 0.0.0.0:443 ssl;
ssl_certificate [redacted: cert address]
ssl_certificate_key [redacted: cert key]
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
error_page 502 /custom_502.html;
location = /custom_502.html {
root /usr/share/nginx/html;
internal;
}
location = /error_502.png {
root /usr/share/nginx/html;
}
client_max_body_size 300M;
proxy_request_buffering off;
location / {
# forward application requests to the gunicorn server
proxy_pass http://localhost:8001;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /static {
proxy_max_temp_file_size 0;
directio off;
sendfile on;
alias [redacted: site location]/static;
expires 5d;
}
}
server {
if ($http_user_agent ~ ^$){
return 503;
}
if ($http_user_agent ~* "Mozilla/4.0\ \(compatible;\ MSIE\ 6.0;\ Windows\ NT\ 5.1;\ SV1;\ .NET\ CLR\ 1.1.4322;\ .NET\ CLR\ 2.0.50727\)") {
return 503;
}
if ($host = [redacted: address]) {
return 301 https://$host$request_uri;
listen 0.0.0.0:80;
server_name [redacted: site];
return 404;
client_max_body_size 300M;
}