I have a live Ubuntu 22.04 server serving websites, that are mostly static but using PHP (Laravel) to display pages (there are some dynamic elements).
I'm using Php8.1-fpm and Nginx to serve the pages. I'm receiving about 40-50k traffic on a daily basis. The avg. server load for 1 minute is usually about 0.03. CPU usage is 12-15%, Memory usage is 10-12%.
Something happened today. The site was unavailable for hours. I only found out about it after like an hour. I checked everything and none of the above metrics have increased - actually they decreased (I think because there was no load on the server). I check the Nginx error logs and this is what I've found (just a few of them, because there were thousands of these lines):
2022/12/16 11:48:15 [error] 834#834: *2272 connect() to unix:/var/run/php/php8.1-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com", referrer: "example.com"
2022/12/16 11:48:15 [error] 834#834: *2274 connect() to unix:/var/run/php/php8.1-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com", referrer: "example.com"
2022/12/16 11:48:16 [error] 834#834: *2277 connect() to unix:/var/run/php/php8.1-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com"
2022/12/16 11:48:17 [error] 834#834: *2279 connect() to unix:/var/run/php/php8.1-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com", referrer: "example.com"
2022/12/16 11:48:25 [error] 834#834: *2200 upstream timed out (110: Unknown error) while reading response header from upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock", host: "example.com", referrer: "example.com"
After that I restarted the server and after a while everything started to work normally again. However I noticed that I'm getting these warnings:
2022/12/16 11:51:41 [warn] 834#834: *2692 an upstream response is buffered to a temporary file /var/lib/nginx/fastcgi/1/00/0000000001 while reading upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com", referrer: "example.com"
2022/12/16 12:07:46 [warn] 834#834: *4048 an upstream response is buffered to a temporary file /var/lib/nginx/fastcgi/2/00/0000000002 while reading upstream, client: *.*.*.*, server: example.com, request: "GET /blog HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php8.1-fpm.sock:", host: "example.com", referrer: "example.com"
I checked previous logs and it turns out that I get about 20-50 of these buffer warnings every day.
Here is my global Nginx config:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
My website nginx config:
server {
server_name example.com;
root /var/www/example.com/public;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options "nosniff";
index index.php;
charset utf-8;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
error_log /var/log/nginx/example.com-error_log warn;
error_page 404 /index.php;
location ~ \.php$ {
fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ /\.(?!well-known).* {
deny all;
}
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = example.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name example.com;
return 404; # managed by Certbot
}
The server I'm using has:
- 4GB RAM
- 2 vCPUs
- OS: Ubuntu 22.04 x64
I did some research but couldn't identify this exact issue I have. It seems like the warnings and errors are in connection. I read something about using Keep-Alives on Nginx, however, I checked my popular websites and they don't have it enabled, so I'm hesitant to add it.
Is there something unusual about my setup or where do you think the culprit is?
