Score:0

Fastest way to copy files locally in linux

in flag

So for performance reasons I decided to move 2TB of small files, randomly written during some time - from a 14TB SATA HDD to a 4TB m.2 NVME SSD, both locally attached.

I've been struggling for 2 days straight to get reasonable copy performance.

cp gets ~15MB/s, wich gives me an estimate of about two days of nonstop copying

rsync is even worse at ~5MB/s

I guess the poor performance is due to a randomness of physical location on the platter which brings me to a 'random read' performance marks.

Still, I'm pretty sure these 2TB of files are a semi-contiguous region of the 14TB drive, so simply dd'ing that region even with a 20% 'miss' overhead would give me a close to 200MB/s rated sequential read. I'll just 'throw away' the files I didn't need - but I can't find any software/method that would do it that way.

My question is, what is the fastest way to copy files locally in 2021?

Score:1
in flag

No its not due to fragmentation, small files on mechanical disk gets copied very slow. It is better to wait. No magic bullet this time.

in flag
From what I can tell, there are several solutions that are much faster than `cp` and `rsync`: [gcp](http://manpages.ubuntu.com/manpages/bionic/man1/gcp.1.html), [parcp](https://blogs.oracle.com/cloud-infrastructure/announcing-parallel-file-tools-for-file-storage), [parallel](https://www.gnu.org/software/parallel/), [fpsync](https://github.com/martymac/fpart), [Mutil](https://github.com/pkolano/mutil) and copying files is a [known problem in linux](https://lwn.net/Articles/789623/)
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.