Score:0

MySQL accessing a lot of deleted temp files

ca flag

MySQL version: mysql Ver 8.0.25-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))

I have a server that hosts a MySQL database and Apache Airflow. It was working fine for a couple of months. For a couple of days, my services are constantly crashing.

I see that the error is due to insufficient Disk Space: These are the errors that I see when I try to run my Airflow Service.

enter image description here

enter image description here

This is happening even though I see there is disk space left: enter image description here

I figured out that this could be due to a MySQL bug, https://community.oracle.com/tech/developers/discussion/4267115/mysql-daemon-not-releasing-deleted-temp-file

But I have a newer version of MySQL installed so this should not happen.

Tried restarting the server a couple of times.

Tried killing the MySQL services manually, but it always goes back to the same state.

So I am not even able to get it fixed temporarily.

enter image description here

Not able to understand how to fix the error permanently, any help will be appreciated.

Michael Hampton avatar
cz flag
Does this answer your question? [Disk full, du tells different. How to further investigate?](https://serverfault.com/questions/275206/disk-full-du-tells-different-how-to-further-investigate)
ua flag
Lower `max_connections` in my.cnf to only 30. Is `tmpdir` being set in the config file?
ca flag
@MichaelHampton no I don't think that is realted.
ca flag
@RickJames Lowered max connections, still the same.
Michael Hampton avatar
cz flag
Did you even look? What were the inode counts?
ca flag
Yes, @MichaelHampton, Inodes were 100% used, I increased the disk space. So seems it got solved temporarily. Mysql is still accessing those deleted files. So I meant, MySQL accessing deleted temp files is unrelated.
Michael Hampton avatar
cz flag
I think you'll just have to restart MySQL.
ca flag
I did that a couple of times, the PID changes, but it keeps accessing those files.
ua flag
100% inodes used -- sounds like lots of tiny files on the file system. In my several decades of using *nix, I don't remember running out of inodes before running out of space. I wouldn't even know what OS parameter to change to increase the ratio between inode and data blocks.
ca flag
@RickJames 100% inodes were because of Airflow, Each dag execution in Airflow creates a log directory(multiple jobs are running every couple of minutes), we have logrorate to delete the log files, but directories remain here. I suppose that is what caused it.
ca flag
I will write a cron to clean up empty directories and then try that.
ua flag
@ofnowhere - Suggest filing a bug report with Airflow,
ca flag
Not sure if it is a bad, but a bad implementation of a feature.
Score:1
ua flag

Out of inodes? Check with df --inodes

A quick, but temporary, fix would be to toss files in .Trash and Download

Maybe there are lots of unnecessary empty files: https://www.interserver.net/tips/kb/solution-running-inodes/ https://askubuntu.com/questions/231585/running-out-of-inodes

Maybe "compressing" a few little-used disk trees would help. (You will need a few free inodes before trying this.) Or tar up some trees that you don't need.

(etc)

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.