what is taking so long when using rsync to copy from one portable hard drive to another

mark leeds

5/20/23, 4:09 AM

Hi All: I am using ubuntu 21.10 and trying to copy 500 Gig from one portable hard drive ( using WD Passport but I doubt that matters ) to another portable hard drive ( also WD Passport ). It's been going on for four days and I can't wake the ubuntu machine up in order to see what's happening. The screen is just black. But the lights on the portable drive that is being copied to are blinking so something seems to be going on.

When I used rsync, I didn't use any parameters which based on what I've read was probably a mistake. Should I stop the process and start over and use some parameters ? Even if I should, I don't know to wake my ubuntu up so I can type commands. I think that the copying process has just taken over the machine's memory so I can't do anything.

I don't mind starting over but I'm scared of losing the 500 Gig on the portable that's being copied from. Is there a way to wake up ubuntu and stop the whole process and re-start it but make it faster ? I think, after 4 days, something is not right ? Thanks.

0 + 0

rsync

matigo

5/20/23, 4:25 AM

Copying large amounts of data between external devices will saturate the USB channels, resulting in a heck of a slowdown. What is the brand/model of the computer itself? Are you copying over USB3? USB2? Are the external disks USB3? What is the file system you are copying to? With this information, it will be possible to sit down with a calculator to figure out how much time a machine might need to copy 500GB of data

mark leeds

5/20/23, 4:37 AM

@matigo: thanks for your help. the computer is a dell but I don't know the answers to the other questions. I always just plugged the hard drive into those holes in the front of the machine ? But I never did two simulataneously before. Also, don't know file system. Do you mean NTFS ?

mark leeds

5/20/23, 4:39 AM

Note that I just did alt + left arrow key and alt + right arrow key and the pink screen with the time came back for a while and said : "press key to unlock". The mouse would work intermittently but I never got prompted for the password so that I could unlock and eventually it went back to black screen. Got my hopes up for about 5 minutes. Thanks.

ubfan1

5/20/23, 4:41 AM

Search this site for slow copy. See things like https://askubuntu.com/questions/995946/decrease-of-transfer-rate-when-copying-large-amount-of-data and https://askubuntu.com/questions/1113235/very-slow-copying-files-from-exfat-to-ntfs Ensure your write drive is not using Shingled Magnetic Recording (SMR) -- honest vendors will say drive not suitable for backup (because of the slow copy). Most likely you system buffers fill up, and things drag to a crawl. Anytime your write is slower than the read, this will then fill up memory.

Jos

5/20/23, 7:43 AM

The good thing about rsync is that you can stop it anytime, and restart it by using the same command. So stop the process and reissue the command, will the `-v --progress` flags this time. This will give you an idea what it is doing.

mark leeds

5/20/23, 7:51 AM

Thanks ubfan1 for the links and Jos for the commands. Somehow the pink screen came back and the machine seems to have re-booted because the copying has stopped. Some copying was done so depending on how much, hopefully I can just continue from there. Thanks to all for your help.

mark leeds

5/22/23, 12:12 AM

Just to close this out and make it possibly useful for future readers. I had 100's of thousands of files and, probably explained by what matigo said, I could not get rsync to work nicely because things would just become too slow. So, then I googled and found out about the dd command. I'm not 100 percent sure but, so far, using the dd approach seems to be working much better. Nothing seems to be getting slowed down like it was when I used rsync. Thanks to all and I hope that this latest info is useful for someone else. There are tons of videos on the internet that explain the dd commands.

mark leeds

5/23/23, 10:57 PM

After dd is finished, if I tried do an ls from the command line, I got a read error. So, eventually I unmounted the hard drive and pulled it out of the USB port . When I put it back in, I was able to read the data that was copied to the hard drive and everything was fine. So, for things "to catch", it seems that you have to unmount the portable hard drive and then re-mount it. In summaery, dd is highly recommended for extremely large number of file transfers from one portable hard drive to another. My machine didn't slow down and I was able to do other things while the copy was being done.

mark leeds

5/23/23, 10:58 PM

Note that it took 3 days for the above copy to complete so it wasn't a small number of files nor a small amount of data.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: what is taking so long when using rsync to copy from one portable hard drive to another

TH: สิ่งที่ใช้เวลานานมากเมื่อใช้ rsync เพื่อคัดลอกจากฮาร์ดไดรฟ์พกพาเครื่องหนึ่งไปยังอีกเครื่องหนึ่ง

RO: ce durează atât de mult când folosești rsync pentru a copia de pe un hard disk portabil pe altul

RU: что занимает так много времени при использовании rsync для копирования с одного портативного жесткого диска на другой

VI: mất quá nhiều thời gian khi sử dụng rsync để sao chép từ ổ cứng di động này sang ổ cứng di động khác

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.