Score:0

Slow response time when server is idle, fast response time when server is loaded?

dk flag

I'm running a 40 thread / 125GB RAM server.

The server is based on CentOS 7.

I have noticed that the server response time is higher when the server is idle:

enter image description here

You can see on the screenshot here that the server response between 4pm and 10PM was lower than during other times.

I looked at the logs and GoogleBot was hitting us at 4 requests / second at that moment which loaded up the server. Most of the requests from GoogleBot were 302 redirects (large ecommerce catalog with daily changes in live products).

enter image description here

Here you can see the throughput of the server - during the times when the server was busy, that's when the response times were low.

How can I debug this?

What's causing this?

Could 302 redirects be cheaper than the 200 responses which skewed the data?

Could cache (Redis / Opcache / APCu) be evacuating too early which causes recreation of cache during idle time?

We're currently running: Apache 2.4 Nginx Proxy MySQL Redis Opcache APCu Elasticsearch

UPDATE:

Looking at the separated processes PHP is taking up the most time:

enter image description here

MySQL somewhat is correlating to PHP, but not fully:

enter image description here

djdomi avatar
za flag
Does this answer your question? [Can you help me with my capacity planning?](https://serverfault.com/questions/384686/can-you-help-me-with-my-capacity-planning)
dk flag
@djdomi this is not really a question about capacity - its a question about why when the server is idle, the response time is higher than when the server is loaded. Could there be any specific settings I should use to reduce the response time when the server is idle? I talked to my server guy and he said it could be because 302 requests are cheaper than 200, so it can be misinterpreting data because of the cheap 302 requests.
Hagen von Eitzen avatar
cn flag
The cause is probably something like this: In a busy state, most important parts are already cached in RAM from previous request (directories, static files, php skripts, probably even PHP bytecode, and even SQL queries), whereas in idle times every request may have to load almost everything from disk. In case your whole *server* is a VM, it may even be the case that this takes a little bit of time to obtain resources as well
dk flag
@HagenvonEitzen hmmmm, but you can see from the graphs, its not like there's a huge amount of time that passes between the weird states of low vs high response time. I'd think the graphs would be a lot smoother in between the transitions. The Server is outfitted with NVME drives and is not really inhibited by those(3500MB/s). Most of the data is in RAM already and we're only using about 70GB out of 125GB.
dk flag
FYI this is a baremetal system with NVME drives, 125 GB RAM, 2xCPUs / 40 threads.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.