Score:0

Cronjob "No space left on device" when logging error but there is space

sr flag

I am trying to troubleshoot some cronjob problems I am having so I wanted to start logging them.

* * * * * wget https://www.example.com/dosomething.php >> /var/log/myjob.log 2>&1

When I view my log file I am getting some No space left of device errors yet the cronjob runs successfully.

I checked my df and df -i to check my storage and inodes and I have plenty of storage.

What is causing this particular No space left on device error when it's logged?

--2022-12-09 19:32:08--  https://www.example.com/dosomething.php
Resolving www.example.com (www.example.com)... x.x.x.x
Connecting to www.example.com (www.example.com)|x.x.x.x|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
dosomething.php.962181: No space left on device

Cannot write to 'dosomething.php.962181' (Success).

enter image description here

Score:2
fr flag
anx

I suspect you saved a new file every time that script ran and hit some resource limit other than total file content size.

Try df -i to look for inode usage. Many filesystems impose a limit not only on the total volume of file contents, but additionally on file metadata entries required to store their names, owner, time, permission, .. If you see a million inodes, that would be well above common operating system usage.

Such command might hang for an unspecified time, but you could try to directly check likely directories to determine if you have 962181 empty files in one directory. Some filesystems impose additional limits on the number of individual entries below a single directory level. This is unlikely something you want, any application where many files are desirable would also be better served with fast (multi-level indexed) access.


How to deal with excessive number of files found?

If you want to keep potential output, but do not need to store the times at which your cronjob had previously run, you might want to selectively delete empty files and only carry on investigating what the non-empty ones say, e.g.:

# find /root -xdev -maxdepth 1 -type f -size 0 -name "dosomething.php.*" -delete

How to stop the problem from reoccurring in the future?

If you did not actually need to preserve the result of all previous downloads, consider specifying an output name (possibly even /dev/null to not store even the latest download) to be overridden on each call, as opposed to the default where curl just appends a number to output filename when it already exist in the directory.

* * * * * wget -O /root/latest-something.html https://example.com/something.php >> /var/log/myjob.log 2>&1
Cesar Bielich avatar
sr flag
Are these files it creates stored in the same directory as the script? I looked and I just have a handful of files in there. I also have plenty of inodes. Already checked that.
Cesar Bielich avatar
sr flag
Ok I see what you are saying about the files. I found millions of files in /root with copies of these cronjob scripts. Can I just delete these?
anx avatar
fr flag
anx
@CesarBielich depends. do you not need their contents? do you not need to know the times your cronjob was run any longer?
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.