PHP scripts suddenly load very slow on Apache

I am testing why sometimes my PHP scripts takes long to load over network (>30sec) on my Apache 2.4 Ubuntu server with PHP-FPM 7.4 using mpm_event. The server was working normally in the past few months, this started happening a few days ago and I didn't change anything. I did a reboot, it didn't help.

I've made a simple test.php. Sometimes it loads normally (<100ms), but sometimes it takes 1 minute to load:

<?php echo "test\n"; ?>

• Server's CPU, RAM and IO are all normal (checked with htop).
• Static HTML files are loaded without any delays.
• Running script locally via SSH console is very fast.
• Apache error logs doesn't show anything unusual.
• I checked if there is some DDOS attack by checking the number of connected IPs from same /16 subnet and didn't find anything weird (e.g. >100 connections).

How can I debug this further to see why this is happening?

Some debug output which may help:

sudo service php7.4-fpm status

There may be several reasons for this behavior:

1. If this web server processes requests from the external network, then with the increase in the amount of traffic, the load could increase, which led to an increase in the server response time.
2. If your scripts use calls to external resources, then in this case the response time of your server may increase due to the low response speed of the external resource.

Log message:

[30-Sep-2021 03:36:46] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

is just proof of the increased load.

In both cases, you should determine the reasons for the load by analyzing the number of requests to the scripts, and if there are calls to external resources, make sure that they work correctly.

As you can see in your status output you have a task waiting to be started (5 active, 0 idle, 6 tasks). As you posted in your own answer (and I'm glad it worked) increasing the number of allowed children can be a good solution - but there's a lot that goes into optimizing php-fpm, and certainly more thought should be given to the entire system before making these configuration changes.

A solid guide is here.

But no matter what you should know when using static values:
if (process memory usage * max_children > RAM)
{ [crash apache] }

if (processing requirements * start_servers > CPU)
{ [crash apache] }

And always know your hardware before tweaking these settings, especially in dynamic/on demand (imo, easier to make mistakes).

If you're doing this for any type of mission critical business web server I'd aim to round up then double all estimates. i.e. largest process that can be called uses 178mb, so 200mb, and your current VM on [insert hosting provider/self] only has 1gb of RAM - I'd set max_children to 2 -- then when you upgrade your VM (what're you doing with 1gb in 2021??) and you have 8gb of RAM on your server, you can use max_children = 18 Notice in both examples the rounding is in favor of extra resources, and after doubling for fpm's purpose, leaves behind a chunk of memory for the OS, and other background processes to use.

Tweaking these settings can be immensely helpful, and anyone using apache should know how - just please make sure your hardware can handle the software configuration you set up.

We had almost this exact same issue last year.

Raising the max children will only offset the problem to a later date.

It turned out to be a slow MySQL database hosted on a dedicated server on our network for a blog.

Our PHP was configured to try connecting for 30 seconds and whenever this database decided to act up then it would chew up 100 PHP children.

We lowered it to 1 second and the problem went away. I don't recall if the database issue was network related or if we had to optimize the database itself.

You should check your Apache access log for that 2:30-3:30 timeframe and figure out if they are pages which connect to a database. Check the log for 500 errors leading up to the collapse of the server.