Score:0

Mysql database often stops and won't restart

in flag

Server: Linode Dedicated 4G Ubuntu 18.04, Webmin + Virtualmin

Every now and then, mysql server stops and fails on restarting. Here is the logs I was able to find:

Apr 11 20:54:57 bamboo kernel: [ 2082.915857] [ 7664]   112  7664   406072    46295   745472        3             0 mysqld
Apr 11 20:54:57 bamboo kernel: [ 2082.915859] [ 7701]     0  7701   129687      307   679936     1435             0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915860] [ 7702]     0  7702   129683      253   667648     1487             0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915861] Out of memory: Kill process 7664 (mysqld) score 40 or sacrifice child
Apr 11 20:54:57 bamboo kernel: [ 2082.919284] Killed process 7664 (mysqld) total-vm:1624288kB, anon-rss:185180kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo kernel: [ 2082.941447] oom_reaper: reaped process 7664 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Failed with result 'signal'.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 20:54:57 bamboo systemd[1]: Stopped MySQL Community Server.

..........

Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 14:02:53 bamboo systemd[1]: Stopped MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: Starting MySQL Community Server...
Apr 11 14:02:53 bamboo mysqld[4345]: Initialization of mysqld failed: 0
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Control process exited, code=exited status=1
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 11 14:02:53 bamboo systemd[1]: Failed to start MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

Looks like it is a memory issue. But I am hosting some very small wordpress sites on this server, and it is impossible for them to use up the memory.

The server graphs: enter image description here

You can see there is a CPU usage spike, and it is when the mysql databases got stopped.

I was not doing anything special at that time so it is quite hard for me to reproduce the problem and observer the possible cause. I guess to investigate this I need to look deeper into the logs, but this is totally beyond my knowledge.

Can someone shed some light regarding how to find the possible cause?

Score:2
in flag

Out of memory: Kill process and oom_reaper... It's clearly a memory problem.

With MySQL stopped, what is the result of : free -h ?

Try to restart MySQL and check htop in an other console, to have a real time monitoring for processes, RAM, CPU...

Check the INNODB buffer size in a MySQL console, maybe it's more than your physical available memory :

show global variables like 'innodb_buffer_pool_size';

Do you have some SWAP partition ?

For the Wordpress side, there are Wordpress website, and Wordpress website... Do you have some plugins on it ? Some can be database intensive (or just not optimized).

ua flag
And try to figure out what query was running when it croaked.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.