Score:3

Ubuntu

How can I tell if Duplicity has frozen?

user1149499

4/5/23, 9:12 PM

I am backing up an 8TB drive to a 16TB drive using Duplicity, and am wondering if there's any way to tell if it's getting stuck somewhere. This backup takes days, even with rsync, but I feel like it shouldn't take weeks, and at a certain point the "Properties" of the disk size stop increasing, but Duplicity (which I run from the command line) stays there, never returning an error. Is there a way to know what's going on? Or to force Duplicity to show what files it's backing up and to thus then see when they stop scrolling?

Some specifics that may be of use: both are ext4 file systems, both are internal hard drives that are being used externally via a universal USB-to-hard-drive connector, 3.5" drives. I am copying the entire file system in human-readable format as a way to have an encrypted backup of my backup (unencrypted) 8TB drive. I've been using this command:

sudo duplicity -v 2 /media/my/path/8TB/ file:///media/my/path/18TB

Based on a below comment, I realize using "-v 9" would give me more verbosity, but I'm currently waiting to see if this time it completes. One of the possible causes I'm thinking might be a simple loose connection, though the drives aren't really moved much, one of the connectors is old and seems a bit loose. Knowing more about whether I should think of other aspects would be very helpful.

Thank you in advance.

0 + 0

backup

duplicity

user1149499

4/6/23, 1:41 AM

@heynnema Maybe you intend this to be useless, and not to pick a fight but that's a worthless comment unless you also offer the reasons, put forth another different solution, or explain why Duplicity isn't up to the task due to X or Y aspect. I'm more than happy to consider alternatives (and you could be entirely right that it's not the right tool, not saying you're not right on the money), but just saying "that's the wrong tools" and leaving it at that is pretty lame. If you're unable or unwilling to actually help, why comment at all? (That's rhetorical; don't need an answer.)

user1149499

4/6/23, 1:43 AM

@heynnema And if I somehow misinterpreted or you got halfway through a comment and had to drop and come back, my apologies in advance. Just frustrated, so solutions/help is really what I'm here on the forum for. Again, sorry if I've misread your intentions.

heynnema

4/6/23, 3:40 AM

What format partition(s)... ext4, NTFS, etc? Are you copying some/all of the data? Is this a bootable drive? Is the target output an image file, or a human-readable directory structure? Is there anything else on the target drive? How are the drives connected... SATA, USB2, USB3, etc?

user1149499

4/6/23, 3:50 AM

@heynnema, these are all good questions. I'll edit the original to include this info. Hadn't realized this would help but it makes sense. Thank you.

heynnema

4/6/23, 4:05 AM

Are both drives using a universal USB-to-hard-drive connector(s)? USB2 or USB3? If they're internal drives, don't they have a SATA/whatever port that they connect to? Any reason to have encrypted backups, when the source is not encrypted? Have you ever needed to restore files/directories from the backup? Is there anything else on the target drive?

heynnema

4/6/23, 4:12 AM

Edit your question and show me screenshots of the `Disks` app SMART data for both drives.

heynnema

4/6/23, 4:12 AM

Is this a one-time backup, or something that happens on a regular basis?

user1149499

4/6/23, 4:14 AM

I'm currently running it manually. If I can verify it's been successful, I'd like to add it as a cron job.

heynnema

4/6/23, 4:17 AM

Please answer my other questions also. Such as... is the source disk bootable? Is there anything else on the target disk? USB->SATA connector cable on USB2 or USB3? SMART Data? etc.

user1149499

4/6/23, 4:19 AM

Sorry, been trying to see what the SMART app is, I'm running Lubuntu, actually which may mean I have to install that app first. But to answer the other questions, USB-SATA on USB3, and no, it's a fresh disk, just partitioned and formatted as ext4 via Gparted. Not bootable.

heynnema

4/6/23, 4:20 AM

The "SMART" app is the `Disks` program. Fresh disk? What are you backing up then? Why not use the native disk connections? USB->SATA is USB3 and connected to a USB3 port?

heynnema

4/6/23, 4:26 AM

Also see https://clonezilla.org/. Clonezilla is also in the Ubuntu repos.

user1149499

4/6/23, 2:19 PM

Yes, I know Clonezilla, have used it, but wasn't aware that it will clone and encrypt the way duplicity does. I've been very happy with Duplicity in the past, but this time it seems to be more wonky and I'm trying to figure out why. Anyway, thanks for the suggestions.

heynnema

4/6/23, 3:49 PM

Show me the SMART data for both drives. Also show me `sudo fdisk -l`.

user1149499

4/7/23, 4:08 PM

@heynnema Thanks for the suggestions, but I feel it's barking up the wrong tree without more of the "why" behind the requests. The disks are healthy and are functioning fine independently, and it seems only an issue during these long Duplicity runs. I suspect it may be a hardware issue of the USB adapters I'm using, maybe they heat up or something over extended use.

heynnema

4/7/23, 4:54 PM

The "why" for asking for SMART data is because if you have a drive with (pending) bad blocks, that might explain the freezing. You also haven't answered all of my requests for more info regarding exact current configuration. Remember, you came here for help. If you don't want to play, then I'm sorry, I can't help.

user1149499

4/7/23, 8:59 PM

I appreciate where you're coming from, @heynnema. Thank you for sharing the logic behind it. I've had bad blocks on drives before but there's usually an error in that case -- I/O or such, and duplicity stops. But you could be right. I'll see what I can dig up and come back if I have more useful info to offer. At the moment I'm tied up with some other things and need to wait a bit before I can get to this. Thanks again.

Score:3

Ubuntu

matigo

4/5/23, 10:23 PM

From the documentation, there are a couple of options you may be interested in:

--progress

    When selected, duplicity will output the current upload progress and estimated
    upload time. To annotate changes, it will perform a first dry-run before a full or
    incremental, and then runs the real operation estimating the real upload progress.

And:

--verbosity level, -vlevel

    Specify output verbosity level (log level).  Named levels and corresponding values
    are 0 Error, 2 Warning, 4 Notice (default), 8 Info, 9 Debug (noisiest).
    level may also be
    a character: e, w, n, i, d
    a word: error, warning, notice, info, debug

    The options -v4, -vn and -vnotice are functionally equivalent, as are the
    mixed/upper-case versions -vN, -vNotice and -vNOTICE.

This should give you quite the scrolling list on the monitor

0 + 0

user1149499

4/6/23, 1:46 AM

Thank you very much for this. I did have the "debug" info but wasn't sure if it was going to log into a log file or be visible on the terminal as it worked. More useful is the first point you bring up, the "--progress" option, which I had totally overlooked. I'm currently running it (so I'll wait until it completes/fails) but if I need to do this again as I suspect I will, I'll try the "progress" and see if that helps. I suspect it will give me a sense of whether it's working, but may not help me with the bigger question of why it's failing. Either way, thank you for your time!