Score:0

Sudden spike in PHP-FPM process CPU usage. How to track down?

cg flag

I have a magento2 store(with 10609 products) running on an ec2 instance(ubuntu 20.04, 16core and 32GB ram). It is a shared server with staging and live on the same instance.

A few days ago, I got an update from amazon saying high cpu usage. when i tracked down Php-fpm logs, slow logs and access logs i observed that there were sudden purging events in the access logs. when i saw PHP-fpm logs it normal.

access logs

. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
      [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:22 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"
. - - [01/Jan/2023:07:32:23 +0300] "PURGE / HTTP/1.1" 301 178 "-" "-"

The configuration file of php is as following:

pm = dynamic
pm.max_children = 70
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 7
php_value[upload_tmp_dir] = 
php_value[session.save_path] = 
php_admin_value[error_log] = /home/logs/
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 2048M
pm.max_requests = 500
request_slowlog_timeout = 10
slowlog = /home/logs/php_slow.log

The db conf is as following

* Fine Tuning
max_connections         = 2000
connect_timeout         = 50
wait_timeout            = 600
max_allowed_packet      = 2048M
thread_cache_size       = 128
sort_buffer_size        = 4M
bulk_insert_buffer_size = 16M
tmp_table_size          = 32M
max_heap_table_size     = 32M

default_storage_engine  = InnoDB
innodb_buffer_pool_size = 10G
innodb_log_buffer_size  = 16M
innodb_file_per_table   = 1
innodb_open_files       = 400
innodb_io_capacity      = 400
innodb_flush_method     = O_DIRECT


  [1]: https://i.stack.imgur.com/0A4w1.jpg
  [2]: https://i.stack.imgur.com/l8Lk9.jpg
Score:0
us flag

Any entity in the internet may have sent those PURGE / requests to your webserver.

It might be an attempt of Denial of Service attack.

There are several options to try to mitigate the issue:

  • If Magento does not use PURGE requests for anything useful, you can deny them in nginx configuration
  • If Magento uses those for some purpose, then you can consider implementing rate-limiting to this endpoint, for example 5 requests / minute. The actual number needs to be determined by Magento usage patterns.
  • Rate limiting can be implemented locally with your web server or via a service like Cloudflare.

There is no single setting that can solve this issue magically.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.