Score:1

rsync refuses to exclude directory

mx flag

I'm trying to make a OS backup with rsync and have put this into crontab. which by itself works great, but for some reason it refuses to take /export into account while looking at the excludes...

0 2 * * 0 rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/export/*"} /media/backup-disk/server-backup/ >/dev/null 2>&1

It does exclude all other folders from the backup though.

The general reason I also want to exclude this /export/* is due to the fact that this backup disk is also mounted as bind to /export due to a nfs share of the disk to a remote server.

Due to the fact that is does include /export/* i keep on ending up with my disk being full due to a loop in my rsync as it starts to backup the disk itself to itself (/export/backup-disk/ -> /media/backup-disk/server-backup)

the mount of my /media/backup-disk to export is as followed in /etc/fstab:

/media/backup-disk       /export/backup-disk       none       bind  0  0 

So my general question is, why is it including /export/* while it is inside the exclude option of rsync, and how do I fix this?

Edit:

the result of a dry-run with --stats --progress (without redirecting stout en stin to /dev/null):

sudo rsync -aAXv --stats --progress / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/export/*"} /media/backup-disk/test/ > /media/backup-disk/test.txt 

cat /media/backup-disk/test.txt | grep -A1 export/        
export/
home/
--
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__init__.py
          9,060 100%   38.98kB/s    0:00:00 (xfr#17141, ir-chk=1174/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/day.py
         12,609 100%   54.24kB/s    0:00:00 (xfr#17142, ir-chk=1173/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/export_custom_job.py
         12,034 100%   51.77kB/s    0:00:00 (xfr#17143, ir-chk=1172/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/job.py
          8,970 100%   38.59kB/s    0:00:00 (xfr#17144, ir-chk=1171/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__pycache__/
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__pycache__/__init__.cpython-38.pyc
          8,809 100%   37.90kB/s    0:00:00 (xfr#17145, ir-chk=1169/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__pycache__/day.cpython-38.pyc
         12,206 100%   52.28kB/s    0:00:00 (xfr#17146, ir-chk=1168/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__pycache__/export_custom_job.cpython-38.pyc
         11,859 100%   50.79kB/s    0:00:00 (xfr#17147, ir-chk=1167/22834)
home/user/.local/lib/python3.8/site-packages/twilio/rest/bulkexports/v1/export/__pycache__/job.cpython-38.pyc
          9,302 100%   39.84kB/s    0:00:00 (xfr#17148, ir-chk=1166/22834)
...

This indicates that it does exclude export, but still somehow I keep finding my disk being full every time the cron runs due to it still copying stuff inside /export to my server-backup directory...

another thing I forgot to mention here is that I initially thought it could be an issue of forgetting /export/* at first, added it later on to the cron, but forgot to reboot (so no cron schedular reboot), but even after a reboot, it still manages to copy /export/* with the cron. Also seems a bit stupid that it would be needed to reboot as it should be able to handle edits to the cron every time it needs to run the command from the cron, but wanted to be sure that it didn't get solved after a reboot...

bac0n avatar
cn flag
What happens if you switch places with lost+found
terdon avatar
cn flag
Does it work as expected if you invert the logic and only include the things you want instead of excluding the others? `rsync -aAXv /{boot,bin,etc.home,net,opt,root,srv,var}`? Also, can you please [edit] your question and include the output of rsync showing the paths in `/export` it picks up? Depending on how they appear, that could give us some info. Finally, have you tried `rsync -aAXv / "/export/*"`? Does it work as expected if you only exclude `export` and nothing else?
ar flag
Your are also redirecting both stdout and stderr to /dev/null. Try first without that redirection and see what rsync has to say?
ar flag
Also, try removing the commas.
FGOD avatar
mx flag
great suggestions, let em try some of these things out with dry-run
FGOD avatar
mx flag
I'm retrying now with --stats --progress to have a better output of what rsync is seeing and will edit the post with it
FGOD avatar
mx flag
could this be the /dev/null thing?
hr flag
AFAIK cron jobs are run in `/bin/sh` by default - which does not support brace expansion. I'm surprised *any* of the excludes works - unless you have added an explicit `SHELL=/bin/bash` or similar prior to the job spec?
FGOD avatar
mx flag
@steeldriver so it would be better to make a file with a list of the exclusions and use the file in the exclusions?
FGOD avatar
mx flag
@sudodus well, disabling the bind mount will make it work as normally because there will be nothing in /export/ to copy anyway... It does work like it should doing it manually in terminal as you can see in the edit I did. I can try making a list of what does need to be copied instead but wouldn't that also make a big list as I wouldn't be able to do / as source and would need to give a list of all the for directories instead that needs to be copied?
hr flag
@FGOD imho it's OK to use the brace expansion - provided you ensure the command is run in a shell that supports it (ksh,zsh,bash,...) either by setting `SHELL` for the cronjob, or by moving the command to a separate shell script with the appropriate shebang and running that from cron instead
FGOD avatar
mx flag
@steeldriver i have tried running the rsync in sh and it indeed includes export. Still think it is weird that it does exclude the rest though. But will test later on today with setting the shell in the cron and running it.
FGOD avatar
mx flag
@steeldriver even when I add SHELL=/bin/zsh it is still picking up on /export. I made a script out of it now with #!/bin/zsh at the beginning and put that script inside the cron, and it does seem to work fine now. Thx for the tip on the way cron runs!
Score:2
mx flag

Seems the problem was indeed like @steeldriver mentioned, the way that crons are run (in sh instead of bash or zsh or any other shell). That made it incompatible with the brace expansion of the exclude list.

The tip on adding the shell to the cron didn't fix this issue, but putting the command inside a script and running the script did fix the issue.

So I now have a script which has the following:

#!/bin/zsh

rsync -aAXv --stats --progress / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/export/*"} /media/backup-disk/test  > /dev/null 2>&1

and the cron as following to make it run every sunday night at 2am:

0 2 * * 0 /opt/scripts/rsync-os-backup
sudodus avatar
jp flag
Thanks for sharing your solution :-)
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.