Score:0

High CPU usage by Apache/MySQL

cn flag
a b

I have a problem with CPU usage on the website that uses WordPress, Apache, and MySQL. During the day, from time to time, CPU usage by MySQL and Apache goes up to 2400% (I have 24 cores in total), the server freezes, the average load goes up to 24.

Recently, there was a little more traffic than usual, but this thing shouldn't be permanent, right? I've updated the kernel, the database, libraries, restarted many times. And still, it freezes. I've looked at the process list of the DB, but there is nothing extraordinary. In the database, there are pretty large amounts of data. Just a couple of weeks ago it worked fine, and now it doesn't. So, it shouldn't be unoptimized queries.

What can be the causes of such behavior?

Update:

the result of A) SHOW GLOBAL STATUS LIKE 'com_%r%_table';

+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| Com_alter_table       | 5     |
| Com_create_table      | 34    |
| Com_drop_table        | 0     |
| Com_rename_table      | 0     |
| Com_show_create_table | 0     |
+-----------------------+-------+
5 rows in set (3.04 sec)

B) SHOW GLOBAL STATUS LIKE 'uptime%';

+---------------------------+--------+
| Variable_name             | Value  |
+---------------------------+--------+
| Uptime                    | 455524 |
| Uptime_since_flush_status | 455524 |
+---------------------------+--------+
2 rows in set (0.01 sec)

C) SHOW GLOBAL STATUS LIKE '%dirty%';

+--------------------------------+-------+
| Variable_name                  | Value |
+--------------------------------+-------+
| Innodb_buffer_pool_pages_dirty | 0     |
| Innodb_buffer_pool_bytes_dirty | 0     |
+--------------------------------+-------+
2 rows in set (0.00 sec)

p.s. I still have problems with the server. I needed to change the character set on one of the databases, and it took a little more than a day to finish, with just 400 000 rows. Before, it used to take some time, but not that much. I was wondering, could it be, that after the DDOS attack, there can be some changes to the database, so that it performs worse?

Update 2:

mysqltuner results:

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 8.0.26-0ubuntu0.20.04.2
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ---------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(0B)
[--] Log file /var/log/mysql/error.log is empty. Assuming log-rotation. Use --server-log={file} for explicit file
 
-------- Storage Engine Statistics ---------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in InnoDB tables: 262.4G (Tables: 179)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics ---------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Security Recommendations ---------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8
 
-------- CVE Security Recommendations ---------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
 
-------- Performance Metrics ---------------------------------------------
[--] Up for: 5d 11h 4m 6s (15M q [31.945 qps], 191K conn, TX: 80G, RX: 2G)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 31.4G
[--] Max MySQL memory    : 9.8G
[--] Other process memory: 0B
[--] Total buffers: 176.0M global + 65.1M per thread (151 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 9.8G (31.14% of installed RAM)
[OK] Maximum possible memory usage: 9.8G (31.14% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/15M)
[!!] Highest connection usage: 100%  (151/151)
[OK] Aborted connections: 0.09%  (174/191712)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[--] Query cache have been removed in MySQL 8
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 5M sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 0% (0 on disk / 2M total)
[OK] Thread cache hit rate: 92% (15K created / 191K connections)
[OK] Table cache hit rate: 99% (21M hits / 21M requests)
[OK] table_definition_cache(2000) is upper than number of tables(506)
[OK] Open file limit used: 0% (6/10K)
[OK] Table locks acquired immediately: 100% (9 immediate / 9 locks)
[OK] Binlog cache memory access: 99.57% (25538 Memory / 25647 Total)
 
-------- Performance schema ---------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
 
-------- ThreadPool Metrics ---------------------------------------------
[--] ThreadPool stat is disabled.
 
-------- MyISAM Metrics ---------------------------------------------
[--] MyISAM Metrics are disabled on last MySQL versions.
 
-------- InnoDB Metrics ---------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 128.0M/262.4G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 1 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 98.29% (925392031 hits/ 941450541 total)
[!!] InnoDB Write Log efficiency: 309.39% (25100125 hits/ 8112662 total)
[!!] InnoDB log waits: 0.00% (65 waits / 33212787 writes)
 
-------- Aria Metrics ---------------------------------------------
[--] Aria Storage Engine not available.
 
-------- TokuDB Metrics ---------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ---------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ---------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics ---------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------
General recommendations:
    Reduce or eliminate persistent connections to reduce connection usage
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: *link*
Variables to adjust:
    max_connections (> 151)
    wait_timeout (< 28800)
    interactive_timeout (< 28800)
    innodb_buffer_pool_size (>= 262.4G) if possible.
    innodb_log_file_size should be (=16M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_log_buffer_size (>= 16M)

Update 3:

Today my server froze again. The process that overloaded the CPU was the apache2. I've managed to stop the service. And suddenly everything started working smoothly. I've tried to do the backup of the database, the one with a lot of rows, and it worked just fine. But, after some time it all froze again: CPU usage by some processes was at 2400%, the load average exceeded 24. So, my suggestion, that it's not the apache, that loads the CPU, not the MySQL either. Some of the processes, like htop or gzip, which I'm using for the backup, also have high CPU usage, from time to time. What could it be then? Could this be the result of a DDOS attack, and if so, how can I fix it?

Gerrit avatar
cn flag
Running `perf top` during these instances of high CPU could be a starter. Readability would be helped hugely by installing the debug symbol packages of Apache, PHP and Mysql for your platform. If the problem is in the database, then `mysqladmin extended-status` may be useful. Particularly see if created_tmp_disk_tables or sort_merge_passes goes up a lot.
Wilson Hauck avatar
jp flag
Could you post TEXT results of A) SHOW GLOBAL STATUS LIKE 'com_%r%_table'; and B) SHOW GLOBAL STATUS LIKE 'uptime%'; and C) SHOW GLOBAL STATUS LIKE '%dirty%'; for analysis? There may be clues in your results. Welcome to serverfault.
a b avatar
cn flag
a b
@WilsonHauck I've added the requested results
Michael Hampton avatar
cz flag
Please post the complete output of mysqltuner.pl
a b avatar
cn flag
a b
@MichaelHampton added the mysqltuner output
Wilson Hauck avatar
jp flag
@ab Your results from the selective SHOW GLOBAL STATUS information eliminates DROP/CREATE table thrashing which can cause momentary freezes. There are no DIRTY pages causing trouble or delays. Will ask for additional information to continue looking at data from your instance.
Wilson Hauck avatar
jp flag
Additional information request. Any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; AND Optional very helpful information, if available includes - htop OR top for most active apps, ulimit -a for a Linux/Unix list of limits, iostat -xm 5 3 for IOPS by device and core/cpu count, for server workload tuning analysis to provide suggestions.
a b avatar
cn flag
a b
@WilsonHauck A) Sorry, I don't really know about that, and I'm not sure how to check it. B) SHOW GLOBAL STATUS; https://pastebin.com/hafPj9f3. C) SHOW GLOBAL VARIABLES; https://pastebin.com/WSEYa9EL. D) SHOW FULL PROCESSLIST; https://pastebin.com/LYhVkDR6. The audio_records query is the backup of the database. E) STATUS; https://pastebin.com/ziTNcE0Z. htop - https://ibb.co/0sTy30r. ulimit -a - https://pastebin.com/F9XnG7hZ. iostat - https://ibb.co/5YrzvbJ
a b avatar
cn flag
a b
Also, I've just noticed, even though when the load average was less than 5, and overall CPU usage was pretty low, the website still took a lot more time to answer, than usual. Some of the requests were canceled due to timeout
Wilson Hauck avatar
jp flag
@ab Your data analysis is in process. Hoping to post Suggestions in next 12 hours. Thank You.
Score:1
jp flag

Rate Per Second = RPS

Suggestions to consider for your my.cnf [mysqld] section

innodb_buffer_pool_size=22G  # from 128M to better accomodate your 262G of data
max_connections=256  # from 151 since you have had all connections used
thread_cache_size=150  # from 9 to reduce threads_created RPhr of 111
innodb_log_file_size=4G  # from 50M to support more than and hour before rotation
innodb_log_buffer_size=1G  # from 16M to support ~ 30 min before write to media

Performance should be significantly improved. There are many more opportunities to improve performance. View profile for contact info and free downloadable Utility Scripts to improve performance.

a b avatar
cn flag
a b
Thank you for the suggestions. I've added them, and for the time performance improved. But the memory usage spiked. I have only 32Gb of memory, and MySQL's maximum usage was 32Gb. When I've tried to optimize the table, it overloaded memory. Now virtual memory of mysqld is on 31.4Gb. But the problem stays the same. I've found a way to replicate it. When I try to open 6-10 pages at the same time, CPU usage increases. In the processlist there are a lot of sleeping processes, which request data from the website database. They don't stay there for too long, the most time I've seen is ~500sec.
Wilson Hauck avatar
jp flag
Your posted htop of a few days ago show many processes with time of more than 1h, when you want to discuss them, get in touch. If any suggestions were positive, consider an upvote or accept, please. Please post the query that opens 6-10 pages.
Wilson Hauck avatar
jp flag
@ab Could you post TEXT results to pastebin.com the results of show engine performance_schema status; Thank you.
a b avatar
cn flag
a b
Here is the "show engine performance_schema status;" output: https://pastebin.com/yKjjqZPt. After that, I've disabled the MySQL and the problem was still there, I'm not sure that this is the DB problem. Got to be something on a more deeper level.
Wilson Hauck avatar
jp flag
The last line of the posting indicates 228M of RAM used to store the performance_schema information. With 32G, not a problem. When will you have time to discuss your htop with the processes that have been running for hours. Are you hosting the zabbix software on this same server? The zabbix software should NEVER be hosted on a production server.
Score:0
us flag

Its very hard to say, but you are saying you are running WordPress and its peak your 24 cores out to 100% just remember a single query can't only use 1 thread at the time.

So something sound as very bad query performances and not direct into your Apache webserver, have you tried the "WP Redis Cache" plugin to cache your query's into Redis to save query lookup?

The next plugin you can try to install is "Query Monitor" it will show you the SQL queries you are calling on the fly, it is a very good debugging tool for WordPress.

Remember if you are developing your own plugins in WordPress you should take care of Redis your self by using the built-in functions to cache your query result.

And in the end of this way to debug, I will recommend you to enable slow-log query for MySQL for everything over 1 sec, you may find query's where index for the column(s) is/are missing.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.