Issue with 'mkdir(): No space left on device" in Docker Container. Might be some other issue though?

SScotti

11/22/23, 2:44 PM

I have some docker containers (NGINX / PHP, MySQL, etc.) that have been running fine for years. However, recently I started seeing this error when running a PHP script in Laravel in the NGINX / PHP container.

mkdir(): No space left on device

From what I can tell, it looks like it is being thrown / generated when the web app tries to send an e-mail during the Login OTP / 2FA step, using Illuminate\Support\Facades\Mail in Laravel. I am presuming that there is an attempt to write to some sort of cache folder / file or other directory within the container that must somehow be full or otherwise does not have permission to write to the location.

It looks like most of the nginx cache files are in: /var/cache/nginx in the container.

Although, nothing has changed in the code recently.

Is there a way to check the available space in the container, or otherwise set the size allocated to a Docker Container ?

588

1 + 0

php

nginx

cache

docker

Score:1

Server

mello

1/2/24, 4:28 PM

There is no limitation to the amount of disk a container can access by default. There is an option available but restricted to some storage drivers: storage driver options

More likely there is no more space left on the host device where the docker containers are running. You can verify it running:

# df -h

Another reason may be that there are no more inodes available (which can occur even without filling the filesystem). You can verify it running:

df -i
# or
tune2fs -l /dev/<your disk device>

+ 4

Nikita Kipriyanov

1/2/24, 5:23 PM

Also, not the disk space but *inode space* could have been exhausted. How to determine that depends on the file system; for ext2/3/4 it is possible with `tune2fs -l /dev/...`

mello

1/3/24, 9:36 AM

right, answer updated, thank u!

djdomi

1/3/24, 9:56 AM

@mello remind that you have to accept the answer if the question is solved. Else we will reminded until the end of the universe to get an acceptance ;)

SScotti

1/3/24, 12:15 PM

I think that was the issue. The inode space was exhausted, and then the syslog and auth.log got stuffed full of error messages. I am still not quite sure why the inodes got exhausted, except that the server had been running for months without a restart. I could possibly check and change the limits since I think they are set at the default. open files (-n) 1024

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Issue with 'mkdir(): No space left on device" in Docker Container. Might be some other issue though?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.