Score:1

Compress input files from rsync before writing to disk

vn flag

I need to load big text files from another machine through a 10Gb link, files been creating by external/closed-source software that i don't have access to change (for example to make it to compress output files)

Currently network and disk IO usage is 100% on these machines (source and destination machines) so bottleneck of the system.

If the source machine could have compress text files in the first place there would be no problem at all, but i don't have access to source machine (files being rsync-ed to destination machine which i have access)

Is there anything at the destination machine that i can do to at least reduce the pressure on destination machine's disk.

I came up with a kind of ridiculous idea which is: to have a temp memory mapped drive with a few GBs, mount it as input directory of the rsync from source machine, then write a program to compress couple of text files each time, write the compressed output file (10X smaller size in compare to original files) to HDD and delete original files from memory mapped drive.

Is there a tool already doing this?! Any other recommendations?

I'm using ubuntu 18.04.

Best Regards

in flag
Product recommendations are off topic, but [fusecompress looks promising](https://www.systutorials.com/docs/linux/man/1-fusecompress/).
f.ald avatar
vn flag
fusecompress looke like a abandoned project (not updated for a long time) and i couldn't find a way to install it on linux
Score:0
cn flag
Bob

Is there anything at the destination machine that i can do to at least reduce the pressure on destination machine's disk.

When disk IO is the bottleneck the obvious solution is of course "get faster disks".

Alternatively when you have some CPU capacity to spare: several Linux filesystems support "transparent" compression where the file system driver performs on-the-fly compression while writing the data to the underlying block storage and decompression when the data is read again. Then you don't have to change anything at the application level.

f.ald avatar
vn flag
If we mount a ZFS drive over NFS and access files remotely, compressed/small files would be passed through network and decompressed on remote machine or vice versa?
djdomi avatar
za flag
you csn use tar and xz over nc or ssh
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.