Score:0

CPU spikes on Ubuntu using RunCloud / AWS Lightsail

se flag

I am regularly getting alerts from my AWS Lightspeed server for high CPU usage. The configuration is 2vpu, 4gb ram, 80gb ssd. I use RunCloud for server management and have around 10 very low traffic WordPress sites.

I am running WordFence on all sites with the same settings, which I have dialed in as much as possible to lower resource use and harden security. Things like locking out after 3 attempts for 2 months, setting rate limiting for crawlers and humans, low resource scanning, traffic logging in security only mode.

I am also using RunCloud's plugin which provides NGINX cache, as well as Redis object cache, and CloudFlare enabled for all sites.

Occasionally MariaDB will crash, taking down all my sites. It hasn't done this since the latest tweaks to WordFence, but I can't be sure it won't happen.

When I run 'top' I can see occasional processes running anything up to 99% from php-fpm. Some of the usage is from WordFence, but I also see a lot of the following:

Most of the time when I run (within a couple of seconds) s-trace p xxxxx I will get strace: attach: ptrace(PTRACE_SEIZE, xxxxx): No such process

When I do get a result I am seeing a lot of things like: gettimeofday({tv_sec=1691575885, tv_usec=447551}, NULL) = 0 (many results)

accept(25, 0x7ffe63704c60, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=502, si_uid=0} --- close(25) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 4 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call) munmap(0x7f0b4d8ca000, 65536) = 0 ....

I also see a lot of these type of messages: newfstatat(AT_FDCWD, "/home/runcloud/webapps/xxxx/wp-content/plugins/wordfence/vendor/composer/jetpack_autoload_classmap.php", 0x7ffe636fdc20, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory) readlink("/home/runcloud/webapps/xxxxx/wp-content/plugins/wordfence/vendor/composer/jetpack_autoload_classmap.php", 0x7ffe63702de0, 4095) = -1 ENOENT (No such file or directory)

Jetpack has been deactivated and deleted on this and all other apps, but WordFence seems to be constantly searching for it.

These processes are consuming a lot of resource and sending the server out of burstable zone very often.

Can anyone shed any light on what these mean and how to configure the server?

Since I am not a sysadmin at all, and merely trying to host some WordPress sites, I do ot know what any of this means, or what to look for. If someone can suggest whether I need to post the full output I can do so.

Romeo Ninov avatar
in flag
Do you mean **LiteSpeed** Web Server?
WP Shala avatar
se flag
No, an instance of AWS Lightsail
jp flag
You seem to have a dangling symlink `/home/runcloud/webapps/xxxxx/wp-content/plugins/wordfence/vendor/composer/jetpack_autoload_classmap.php`. You can remove this file but it has nothing to do with your performance issues. As well `strace` won't give you any insight in this case. You need to show your instances resources and your applications resource allocation.
WP Shala avatar
se flag
this seems to have been cleaned up following the stopping and starting the server
WP Shala avatar
se flag
Actually being able to show resources used by each application would be really helpful, but I dont know how to do that. Both AWS and RunCloud only show the server load as cpu use and burst capacity. Anything in particular I should be looking for in the web app config?
Score:1
ws flag

When I do get a result I am seeing a lot of things like: gettimeofday({tv_sec=1691575885, tv_usec=447551}, NULL) = 0 (many results)

Most likely this is being caused by an error detected in PHP. The default behaviour in Wordpress is to suppress reporting of errors - but on a typical Wordpress installation with plugins, each request can generate a LOT of errors. Note that PHP still handles the error - it just does not appear in any logs.

Wordpress describes logging as "DEBUG mode".

Jetpack has been deactivated and deleted on this and all other apps, but WordFence seems to be constantly searching for it.

Are you sure it has been removed? Via the API? Including the directory? If you don't have wpcli - get it and check.

Occasionally MariaDB will crash

hmmmm. I've run several Wordpress installations (20-40) with moderate traffic volumes and never had mysql nor mariadb crash.

If you can't get an explanation from your logs after enabling them, you might want to take a look at xhprof or excimer.

WP Shala avatar
se flag
This morning 5am the server went into sustained 60% cpu usage and became unresponsive. It wouldnt reboot from AWS Lightsail panel, and had to be stopped and restarted (I now have a static ip set up) After stop/starting it stayed all day around 3% and then jumped to around 35% for 10 minutes. I am currently trying to research debug logs on Local server, with all sites running the same php version, synced as closely as possible. I have opened a ticket with the Woopayments plugin, which spawns jetpack-related cron jobs every 5m, and respawns after they are shut down in crontrol
WP Shala avatar
se flag
If I check /var/log/syslog it will tell me such and such db table crashed, but I think this could just be the table that happened to be open when the server crashed
WP Shala avatar
se flag
I also checked Monster insights (google analytics) traffic, and WordFence attacks. Nothing excessive except one site with around 750 blocked attacks in 24h, in line with weekly and monthly averages
WP Shala avatar
se flag
to clarify, jetpack has been completely deleted, no folder present
WP Shala avatar
se flag
no errors with debug enabled on localhost, only a couple of warnings
WP Shala avatar
se flag
Actually I get processor and memory usage per service in RunCloud, but only as a live view. I am attempting to install XHProf for PHP7 without breaking anything!
WP Shala avatar
se flag
"hmmmm. I've run several Wordpress installations (20-40) with moderate traffic volumes and never had mysql nor mariadb crash." - Did they all have WordFence running.. the free version without control over scheduling?
ws flag
Mostly the paid version, but a few free ones
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.