Score:0

Slow server performance [nginx + php+fpm + mariadb]

co flag
Rob

I'm having problems finding a bottleneck of my slow server performance. Single wordpress site (10-40 online users) with fastcgi caching and full HTML cloudflare caching. Site performance is fine for visitors because of caching, logged in users are facing 1-10s page load times. Query monitor is used for debugging. Site was running way better on 1 core, 2gb ram machine. No idea what is causing this. Any help is much appreciated. Thank you.

Example of a slow single post load:Query monitor slow load

Example of good single post load: Query monitor fast load

I'm running KVM 2 Cores @ 3.50+ GHz 8 GB Memory 160 GB SSD.
    nginx version: nginx/1.14.1
    PHP 7.4.22 (cli) + memcached 
    10.6.3-MariaDB

root@localhost:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           7987        1546         195         121        6245        6020
Swap:          8191         268        7923

PHP-FPM settings: (Tried to increase them, no changes)

pm = dynamic
pm.max_children = 100
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 8
pm.max_requests = 200

Database

Server Version  5.5.5
Extension   mysqli
Client Version  70422 (7.4.22)
innodb_buffer_pool_size 2147483648 (~2 GB)
key_buffer_size 134217728 (~128 MB)
max_allowed_packet  16777216 (~16 MB)
max_connections 151
query_cache_limit   1048576 (~1 MB)
query_cache_size    67108864 (~64 MB)
query_cache_type    ON

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 2048;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    types_hash_max_size 2048;
    # server_tokens off;
    proxy_buffering on;
    access_log off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    # access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##
    
    ################## caches ###############
    proxy_cache_path  /etc/nginx/static-cache levels=1:2 keys_zone=s3_cache:200m max_size=1800G inactive=10y use_temp_path=off;

    fastcgi_cache_path /etc/nginx/fastcgi-cache levels=1:2 keys_zone=phpcache:50m max_size=1g inactive=7d use_temp_path=off;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";

    #############################
    client_max_body_size 2000M;



    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

nginx server conf

# pass the PHP scripts to FastCGI server
location ~ \.php$ {
        try_files $uri =404;
       fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
        fastcgi_index index.php;
            fastcgi_read_timeout 150;
        fastcgi_buffers 16 16k; 
        fastcgi_buffer_size 32k;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;

fastcgi_cache_bypass $skip_cache;
fastcgi_no_cache $skip_cache;
fastcgi_cache phpcache;
fastcgi_cache_valid 200 301 302 60m;
fastcgi_cache_use_stale error timeout updating invalid_header http_500 http_503;
fastcgi_cache_min_uses 1;
fastcgi_cache_lock on;
add_header X-FastCGI-Cache $upstream_cache_status;

}

EDIT: I did some testing outside Wordpress. Generating 100000 of random strings in for cycle.

    <?php $start_time = microtime(true); ?>

<?php        ini_set('display_errors', 1);
        ini_set('display_startup_errors', 1);
        error_reporting(E_ALL);
        
    # define("WP_USE_THEMES", false);
    #require_once("/var/www/server/wp-blog-header.php");


        for($i = 0; $i < 100000; $i++){
            generateRandomString(2);
        }


function generateRandomString($length = 10) {
    $characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
    $charactersLength = strlen($characters);
    $randomString = '';
    for ($i = 0; $i < $length; $i++) {
        $randomString .= $characters[rand(0, $charactersLength - 1)];
    }
    return $randomString;
}

        ?>

This page was generated in <?php echo(number_format(microtime(true) - $start_time, 2)); ?>

And results are very the same. Some requests finish in 0.1 second some takes up to 5 seconds. Looks like PHP is doing it's work just fine. Maybe it's php-fpm that slows things down then connecting with nginx?

Michael Hampton avatar
cz flag
Enable the [slow query log](https://mariadb.com/kb/en/slow-query-log/) (with a much more reasonable long_query_time of, say, 1s) then check it later to see if anything interesting has been logged.
Rob avatar
co flag
Rob
I did. Here is the last recorded slow query: https://pastebin.com/spyvBGSR EDIT: this is PHP slow log. Mysql doesn't seem to be a problem.
cn flag
If you routinely get `memcached-class-object-cache.php` referenced on top of the PHP-FPM slow log, it does indicate a problem with the Memcached instance/server. Why not Redis instead?
Rob avatar
co flag
Rob
Replaced Memcached with Redis. No improvements.
Rob avatar
co flag
Rob
Here is a screenshot from xdebug profile: https://prnt.sc/1tuike7 I don't have experience with this so this doesn't answer any question for me.
ua flag
Get rid of caching. You don't have a busy enough system for it to be worth using.
Rob avatar
co flag
Rob
Why would I want to get rid of caching and serve slow requests for my users?
Score:0
co flag
Rob

Looks like the problem is solved. Contacted my VPS provider, they confirmed that my server is thermal throttling and they will solve this problem. This answers why some requests are slow.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.