How to track specific logs in Ubuntu? (CPU)

We have this EC2 instance: T2.medium, running apache, with 4 virtual hosts (4 sites). Sometimes, out of nowhere, the CPU reaches very high levels, maybe an attack.

I've seen some of our wordpress files have been modified.

How could i check who has been writing in those files? How could i check the logs of the CPU to see what process has been affecting it? Are there any cloudwatch metrics i could use?

We have been doing some hardening to the server: updates, running AWS Inspector, lynis, modifying the ssh config file.

Is there any way to see who and how did they managed to enter and modified those wordpress files?

And what other practices of hardening do you recommend?

There are several questions here.

Who has been writing to the files

The operating system does not log this information, but there are some clues:

• The modification date
• The file permissions

Use the files' modification date to narrow down your search of your Apache access logs. Check at the very least for any POST requests and logins from around that time. For example this would show all login attempts:

zgrep 'POST /wp-login.php' /var/log/apache2/*access*

You can then filter the output by the time range you got from the files' modification time.

If the files that have been modified are only writable by certain system users, then you can be reasonably certain that they were modified by those system users.

What process(es) are pegging the CPU

This information is not logged by default. If it impractical to try to monitor the server "live" -- for example with top -- then there are various logging tools you can use. Here is a serverfault question where various tools are recommended for this purpose.

Determining if you have been hacked

This a larger topic, but the place I would start, since you mentioned modifications to the WordPress files, is to determine if these modifications are malicious. Run a WordPress malware scanner, and/or look for malicious patterns such as eval(base64_decode(, php web shells, and so on. If you're unsure, be persistent, be thorough, post more questions if you need to.

Determining how an attacker gained access

If you are reasonably certain that the site or sites were hacked, you can try to determine how the attacker gained access. The two most likely ways this could have happened are via login to an admin user account, or via a vulnerability. In most cases, it is difficult to determine with a high degree of certainty. But if you have been running software with a known vulnerability, especially one with a public exploit and that allows remote code execution, then this is a very likely possibility. And if a WordPress admin user has weak credentials, or their credentials were leaked, then this is a very likely possibility.

Further hardening

If you believe that the server has been compromised, then you should refer to the canonical answer on the subject.

This is not intended as a full answer, it's supplementary to the answer by sceox.

You should look at hardening Wordpress, and Wordpress file permissions.

I have things set up like this:

• One user / group owns the files
• PHP is part of a group that can read the Wordpress files inc plugins / themes / etc, but cannot write to them. It can write to the uploads folder so images can be uploaded using the Wordpress GUI. This makes it very difficult for anything on the internet to compromise the Wordpress files
• I have a script that uses the Wordpress CLI to do updates of Wordpress and the plugins at 2am.
• Any new plugins must be installed with Wordpress CLI. It's not as convenient, but it's MUCH more secure.

Here's the script I use, which runs on a cron job

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

echo
echo Wordpress Update and Permissions Script Starting
echo "$(date) Wordpress update and backup started"   >> /var/log/me/my-wordpress-upgrades 2>&1

# Function to upgrade wordpress
function upgrade_wordpress() {
    # set up folders in the formats needed
    dir=$1
    uploads=$1/wp-content/uploads

    echo Upgrading Wordpress core, plugins, themes in ${dir}
    sudo -H -u www-user bash -c "wp core update --path=$dir"
    sudo -H -u www-user bash -c "wp plugin update --all --path=$dir"
    sudo -H -u www-user bash -c "wp theme update --all --path=$dir"

    echo Setting wordpress permissions to 755 files and 644 folders
    find ${dir} -type d -exec chmod 755 {} \;
    find ${dir} -type f -exec chmod 644 {} \;
    chmod 440 ${dir}/wp-config.php

    echo Making uploads folder ${uploads} writable by the web server
    chown -R www-data:www-data ${uploads}

    echo Wordpress upgrade for $1 complete
    echo
    echo
}


echo Setting /var/www permissions to www-user:www-data
chown -R www-user:www-data /var/www/

# Run Wordpress update for each wordpress install
upgrade_wordpress /var/www/blog1
upgrade_wordpress /var/www/blog2