Centos 7 server performance problems. High load

I have a Centos 7 VPS at HostGator that has been very slow lately. HostGator have not been overly helpful in diagnosing the problem, insisting that everything is ok.

The server is a 4 core VPS with 8GB of RAM.

I figured out my cPanel backups are at least partially to blame, but I'm certain something else is amiss. I found they were basically running all day, because they were constantly pausing while server load was high (which is a good thing I guess). But this was compounding the issue further due to the backups consequently running all day long, further loading the server.

The sites on the server are loading slower than normal, and it seems intermittent, sometimes they load quickly, others there is a noticable (~3 second) delay before the pages load.

I host a few sites on the server (3) and although they get a bit of traffic we're not talking anything over 1200 visits a day between all three sites.

Currently my load averages are sitting at 4.33 3.59 3.38 (no backups running):

results of top command sorted by CPU

I'm kind of at a loss from here. I've spent hours looking at logs etc, trying to figure out what exactly is going on with no luck. I'm not getting DDOSed or anything like that (from what I can tell from my HTTP logs.

Can anyone suggest any steps I might take next to determine what is going on?

Thanks for any help you can provide.

For a server with 4 CPUs a load avg at around 4 is considered harmless, as this means at any time there are about 4 tasks being processed, which with 4 CPUs means each task is being actually processed.

I would suggest you run a benchmark against your sites, and figure out where the loading time is actually spent, in many cases this could be things like database connects that take time to establish, or external resources that prevent the site from being rendered.

Looking at your screenshot I see a four practically bored CPUs, loads of RAM to spare, which is mostly used for the disk cache. You didn't include the processes in the list (so the sorting is irrelevant), which also doesn't provide any pointers to what is at fault.

Lastly it could still be, that your provider runs older hardware or is overbooking their hypervisor machines to stay competitive.