Twice I've noticed out of space errors in my php
app and both times I've received the error "No space left on device while writing config" when attempting to login via ssh
to troubleshoot the problem.
I have plenty of free disk space and both times my app worked again after restarting my server manually through my hosting company's website. Obviously, this is an inconvenience but when I have customers, it'll be completely unacceptable.
I've checked /var/log/messages
for anything that may help diagnose the problem. All I could find that seems relevant is:
rsyslogd[1032]: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': No space left on device [v8.2102.0-5.el8 try https://www.rsyslog.com/e/2013 ]
I do have this as a cron-job
running twice a day: find /tmp -atime +1 -delete
. I don't think this is causing the issue but I am not certain. Is this even a good way to clear /tmp
?
I guess as a quick fix, I could tell php
to call a bash script to restart the server every time it encounters an out of space error. This doesn't feel like a good idea though without understanding exactly what's malfunctioned and why.
I am using AlmaLinux 8.5
(very similar to Centos) with Nginx
and php-fpm
and I only have a VPS
. I'll edit my question if you think there's any relevant information I should include.
Edit
There's no point showing the results of any commands, until I encounter the error again. I've created a webpage to execute the commands using shell_exec
and to display the results on screen. At the time of the error, I should be able to run the commands because nothing is written to disk:

NB
A client SSL certificate and login details that only I have, is required to run php as root and to access this page so I'm not worried about the security implications of running php as root or calling shell_exec
with user data.
@NikitaKipriyanov suggested trying to keep my SSH
connection open. If I didn't already have things setup this way (php as root for admin stuff), then of course it'd make more sense to try disabling SSH
from timing out instead.
I will provide an update when I encounter this error again and have some results from my tests. Feel free to put the commands you think I should be executing into an answer, as I may upvote, and I'll accept the answer if it leads to me fixing the problem.
Edit - potential progress
Considering, I am sure that my system does not actually run out of disk space, I was expecting the problem to be a process still using a deleted file, or an error relating to a process that has crashed. There are plenty of articles stating this can cause out of disk space errors, but nothing about diagnosing this specific cause.
However, I've noticed my inode count has increased from 3% to 7% overnight. I do intentionally store data in lots of small files, however that should only account for my inode count increasing by a handful of inodes. I have crontab
automatically create and store backups. I do monitor this so I would notice any anomalies.
I think the problem is my php
$_SESSION
array is creating way too many temporary files. The size of the $_SESSION
array can increase linearly, so at present the amount of data stored is always insignificant in size. I don't create any backups of the $_SESSION
array, so currently, I won't notice this increasing. This is very easy for me to test and observe this so that will be the next step. I don't want to make any assumptions so I'm going to wait and see if the inode count approaches 100% causing a crash before I attempt to fix the problem.
NB
As soon as I've confirmed the problem, I will move this extra information into an answer.