Score:1

rsync uncompressed to single compressed archive

ca flag

Question

How can I rsync uncompressed (but highly compressible) data to a single compressed archive (.tar.gz, .zip, etc) without transferring all the data each time?

Current

This is what I am using now:

rsync -a --delete-during "/home/" "/mnt/external-hdd/backups/"

On the local side (/home/), there will be thousands of small text files that are compressible. On the "remote" side (/mnt/external-hdd/backups/), there will be an exact copy of the local side.

Wanted

  • local side: same as before
  • "remote" side: a homes.tar.gz that contains all of the data from /home, that can be incrementally updated like how rsync works by default

I will not be independently accessing specific files from homes.tar.gz. If an issue occurs, I will restore it to whatever homes.tar.gz contains.

This will significantly reduce storage size since data is compressed.

Prefarably, it should not require tar czf homes.tar.gz backups/; mv homes.tar.gz /mnt/external-hdd/backups/homes.tar.gz. That is, tarring and transferring all the data each time.

vidarlo avatar
ar flag
Why rsync? Rsync is not made for working with archives.
fejyesynb avatar
ca flag
@vidarlo `rsync` is just an example tool. I have mentioned it so it is clear what I want (behavior like `rsync` that can incrementally update the remote, but where the remote stores it as an archive (`.tar.gz` or other compressed format)).
vidarlo avatar
ar flag
We don't do product recommendations here, sadly. There's a lot of backup solutions out there that supports both compression and incremental backups.
fejyesynb avatar
ca flag
I saw [`gzip`'s `--rsyncable` flag](https://linuxcommand.org/lc3_man_pages/gzip1.html) and thought `rsync` itself might be able to handle my use case.
Score:0
cn flag

I would use for gzip compression:

tar zcf /mnt/external-hdd/backups/home-$HOSTNAME-$(date +%F_%H%M%S).tar.gz /home

If you need better compression, but slower, use xz:

tar Jcf /mnt/external-hdd/backups/home-$HOSTNAME-$(date +%F_%H%M%S).tar.xz /home

You can speed up a bit xz if you have multi core CPU:

tar cf - /home |xz -T 0 > /mnt/external-hdd/backups/home-$HOSTNAME-$(date +%F_%H%M%S).tar.xz 
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.