Score:0

Can we use make tar multi-threaded without any sort of compression

us flag

I have to backup 500GB using the traditional tar -cvf command (without any compressions) However tar is adding each file one by one and this way it's taking a very long time

Is there any way to make tar multi threaded (like it can add multiple files to the tar archive at once)

shodanshok avatar
ca flag
You can't make `tar` itself multithreaded, but please add some more details to understand if/how we can make it faster. What specific command are you using? Are you tarring from/to the same disk? To another disk? To a NFS mount?
Score:2
gu flag

No, and you're unlikely to be CPU bound in this case anyway.

Improve your backup strategy if you can, or live with slow backups but the only way to speed things up with the mechanism you have now is faster disks.

Andrew Henle avatar
ph flag
*the only way to speed things up with the mechanism you have now is faster disks* Maybe not. If the backup process is, ummm, not well-thought-out and is doing things like wasting disk IO cycles with a process like "tar the file in place, then copy it to the backup media", it can easily be made 3-4 times faster by simply doing "tar the file directly to the backup media". The data being backed up should come off disk and go directly to the backup media - anything else is, ummm, poorly-designed.
Ginnungagap avatar
gu flag
Sure if the tar operation itself is slow, it will certainly help to send the data to another disk (or network or pigeon or whatever) so the disk can focus on reading, but it'll still be a largely IO bound process. I take your point about poorly designed backups though
Score:0
cn flag

CPU matters, you can try parallel tar with pigz.

As mentionned by /u/Andrew Henle

compressing on the source then transport could be more efficient than compressing on the fly to the destination.

For example :

time tar -I pigz -cf sourcebackup_$(date +"%Y-%m-%d-%H-%M").gz sourcebackup/

can be faster than :

time tar -I pigz -cf - /sourcebackup | ssh -t user@backupserver "sudo cat > /backup/sourcebackup_$(date +"%Y-%m-%d-%H-%M").tar.gz"
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.