Join Log in
Score:0
Hugh Ranalli avatar Server

rsync failure not detected by duply/duplicity; corrupted backup "successful"

de flag
Hugh Ranalli

First of all, I recognise these are not the most recent versions of duply and duplicity. The client runs servers on various OS versions, and we currently install duply and duplicity from the system repositories. I've searched and can find no indication that this is an issue fixed in a more recent version. If someone can show me that it is, we'll update the systems to install updated versions of these packages. But I'd rather know for sure, before investing that effort.

We backup files using duplicity, managed by duply, and run via cron. While a full backup was running, a maintenance activity at our hosting facility took the receiving server offline. Later, when attempting to restore from that backup to prepare a replacement for that server, I discovered the backup was corrupted, with several 0 byte files during the maintenance period. I didn't receive an email from cron about an error, and the logged output showed that while rsync reported an error, duplicity did not identify one ("Errors 0"):

Start duply v1.11, time is 2022-12-30 02:00:01.
Using profile '/etc/duply/sites'.
Using installed duplicity version 0.7.06, python 2.7.12, gpg 1.4.20 (Home: ~/.gnupg), awk '
GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.4, GNU MP 6.1.0)', grep 'grep (GNU grep) 2.25', bash 
'4.3.48(1)-release (x86_64-pc-linux-gnu)'.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - En/Decryption skipped. (Testing disabled)

--- Start running command PRE at 02:00:02.363 ---
Skipping n/a script '/etc/duply/sites/pre'.
--- Finished state OK at 02:00:02.415 - Runtime 00:00:00.051 ---

--- Start running command BKP at 02:00:02.433 ---
Attempt 1 failed. BackendException: Error running 'rsync -e 'ssh -oBatchMode=yes' /tmp/duplicity-upULdT-tempdir/mktemp-i_a13g-605 USER@SERVER:PATH/sites/duplicity-full.20221230T070002Z.vol604.difftar.gpg': returned 255, with output:

packet_write_wait: Connection to 12.34.567.890 port 22: Broken pipe
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1]

--------------[ Backup Statistics ]--------------
StartTime 1672383612.11 (Fri Dec 30 02:00:12 2022)
EndTime 1672388712.14 (Fri Dec 30 03:25:12 2022)
ElapsedTime 5100.03 (1 hour 25 minutes)
SourceFiles 21623
SourceFileSize 18523309075 (17.3 GB)
NewFiles 21623
NewFileSize 18523309075 (17.3 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 21623
RawDeltaSize 18514613173 (17.2 GB)
TotalDestinationSizeChange 17904721458 (16.7 GB)
Errors 0
---------------------------------------------

--- Finished state OK at 03:25:32.689 - Runtime 01:25:30.256 ---

--- Start running command POST at 03:25:32.700 ---
Skipping n/a script '/etc/duply/sites/post'.
--- Finished state OK at 03:25:32.715 - Runtime 00:00:00.014 ---

The corrupted files:

-rw------- 1 backup adm  26268756 Dec 30 03:08 duplicity-full.20221230T070002Z.vol597.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol598.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol599.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol600.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol601.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol602.difftar.gpg
-rw------- 1 backup adm         0 Dec 30 03:08 duplicity-full.20221230T070002Z.vol603.difftar.gpg
-rw------- 1 backup adm  26281466 Dec 30 03:22 duplicity-full.20221230T070002Z.vol604.difftar.gpg

What can be done to prevent this in the future? I don't mind a backup failing because of a server issue. I do want to know that it failed, so I can deal with it. I've looked at the duply, duplicity, and rsync pages, and can't see any additional options to specify that would affect this.

61
1 + 0
Score:0
avatar Server
us flag
ede

these versions are overall ancient and not maintained anymore. please upgrade to the latest duply 2.4.2/duplicity 1.2.1 and check if you still can reproduce the error. if so, please file a bug ticket on https://gitlab.com/duplicity/duplicity/-/issues .

as your versions are ancient there's a good chance that a bug as obvious as this one was fixed already.

if upgrading versions presents too big a challenge you might want to consider using sftp:// or another backend if rsync is unreliable in your case.

also, you may consider verifying your backups regularly so you will be spared surprises like these in the future.

+ 5
Hugh Ranalli avatar
de flag
Hugh Ranalli
I do understand that. In fact, I even pointed it out in my very first paragraph, and explained why I wasn't immediately going this route. This is a non-trivial exercise for us, so I am hoping to either confirm this is the solution (I searched the duplicity issues looking for something similar, with no success), or determine if there is something else (e.g. a command line switch) I should be using. I've attempted to research this, and don't think this was an unreasonable question before proceeding further.
0
Reply
us flag
ede
sorry. there is no command line switch to remedy rsync upload fails. btw. the log above actually does only mention "Attempt 1 failed." so it is likely that the upload succeeded on the next try. duplicity by default tries several times before failing eventually.
0
Reply
us flag
ede
added a workaround option to the answer in case version upgrades are no option
0
Reply
Hugh Ranalli avatar
de flag
Hugh Ranalli
Thank you for the additional information. The interesting thing in the logs is that while the rsync failure is logged (we direct output from the cron job to a log file), the duplicity status shows "Errors 0," which makes me think the error status didn't get transmitted up the chain. I have to doubt the transfer succeeded on the next attempt, as I have a corrupted backup and several 0 byte files. That shouldn't be the case if the transfer eventually succeeded, correct?
0
Reply
Hugh Ranalli avatar
de flag
Hugh Ranalli
I can't imagine another backend being more reliable than rsync, especially in this instance (the server being rebooted in the middle of the backup). I don't expect anything to be **that** bulletproof, just to be informed when it does happen. I've considered verifying backups, but we have a lot of backups, and verifying them isn't simple, because, as I understand it, duplicity has to decrypt and verify each file. If I automate the process, that means putting my highly protected encryption key on a server, which makes me nervous.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.