Score:0

rsync : write failed - No space left on device (28) despite using --inplace

kr flag

I am using a script to make a backup of a running Debian Stretch based Linux Distro (NextCloudPi)

The method in the script that does the backup uses rsync.

backup()
{
  mntimg
  sync
  rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/lost+found' --exclude '/media' --exclude '/mnt' \
--exclude '/proc' --exclude '/run' --exclude '/sys' --exclude '/tmp' --exclude '/var/swap' --exclude '/etc/udev/rules.d/70-persistent-net.rules' \
--exclude '/var/lib/asterisk/astdb.sqlite3-journal' "${OPTIONS[@]}" / "${MNTPATH}/"
..
..
}

When I run the script I point it to save the backup .img file to an externally attached USB-HDD drive.

This drive is EXT4 formatted and is mounted.. I can browse it from Manjaro file explorer. This is writable and has 2.3TB free space.

The backup file is going to be about 7.8GB and I have got 22GB free space on the rootfs (/) on the SD Card that I am backing up.

Each time I run the script I get an error rsync: write failed on.. no space left on device:

root@NEXTCLOUDPI:~# image-backup

Image file to create? /media/4TB2/nextcloudpi18Nov2021v3.img

Initial image file ROOT filesystem size (MB) [7526]? 7800

Added space for incremental updates after shrinking (MB) [0]? 

Create /media/4TB2/nextcloudpi18Nov2021v3.img (y/n)? y

Starting full backup (for incremental backups, run: /usr/local/bin/image-backup /media/4TB2/nextcloudpi18Nov2021v3.img)
rsync: write failed on "/tmp/img-backup-mnt/usr/src/linux-headers-4.14.93-Re4son-v7+/include/linux/suspend.h": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2]

Unable to create backup

root@NEXTCLOUDPI:~#

I still get the problem even if I add in the rsync option --inplace so that did not solve my problem.

I did a sudo du -sh /usr/src and the size is 150MB.
I've got 37,000 files and 12,000 sub-folders in /usr/src so I was thinking maybe I am running out of inodes but... I did a df -i and my inode usage is 14% in the root dir (/).

The problem seems to happen close to the end.. in this case a 7.9GiB file is created. I tried flashing that to an sd card with etcher but it didn't boot.

Any ideas what is going wrong here? I have enough space on the rootfs for rsync to save things to /tmp if it needs to. But even when I use the --inplace option it still says: rsync: write failed on "/tmp/... blah blah... No space left on device (28)

np flag
You've checked inodes count on live system or image? Because seems like this script creates an image which you've specified to be 7800 MB probably it was created with less total inodes than you need because your live rootfs are bigger. Try to mount created image & check it's size & inodes count with `mount -o loop ./img.raw /mnt/tmp && df -i /mnt/tmp && df /mnt/tmp`. What does it outputs (please update your answer)? Another thing to try is to create a bigger image at first (just to check at least) when your script prompts for it.
FlexMcMurphy avatar
kr flag
Thanks @NStorm. The script is working for me again. I had to allocate it about 4.5GB more space than it estimated it would need. I asked the scripts creator who thinks there may be something pathological in the filesystem of the OS I am backing up and that I may be using several (or large) sparse-allocated files somewhere. I'll probably do a fresh re-install of that server soon. Hopefully that will fix what's going wrong!
Score:1
np flag

Now based on your recent comment I can say that most likely it just run our of inodes. When you create ext4 fs by default mkfs allocates inodes based on partition/image size. So the less is the image size, the less is the inode count. You edit this script to allocate more inodes, just find a line where it does mkfs.ext4 and change the number of inode it allocates with -i or -N option.

Find this line in image-backup:

    mkfs.ext4 -q -b 4096 "${LOOP}p2" > /dev/null

and change it to:

    mkfs.ext4 -q -b 4096 -i 4096 "${LOOP}p2" > /dev/null

This will make 4x more inodes than the default settings.

I don't think something in your server can produce such effect. While sparse files might be the reason too, you can mitigate it with -S rsync option which will handle sparse files correctly. But it conflicts with --inplace option.

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.