Score:0

Execute following steps after rsync is finished

id flag

First I thought that rsync isn't syncing everything described here Limitations of rsync? Not syncing everything

Now after a day I found out that it was actually still working/ running and now all the files are there. The problem is: My zip is instead of 1gb only 200mb.

Because the zip was created I thought rsync is "finished" and just doesn't download everything. My logic was: tar only gets executed when rsync is finished. But as it seems that is not the case.

How can I make tar wait until rsync is completely finished? So I don't end up with archive that's not complete?

This is my code:

USER="myusername"
SERVER="myserverip"
PORT="22"
SOURCE="shop"
TARGET="/volume1/BACKUPS/MyFolder/MySubfolder/data/"
LOG="/volume1/BACKUPS/MyFolder/MySubfolder/backup_data.log"

mkdir /volume1/BACKUPS/MyFolder/MySubfolder/

rsync -avz --progress -e "ssh -p $PORT" $USER@$SERVER:$SOURCE $TARGET --delete >> $LOG 2>&1
tar -zcvf /volume1/BACKUPS/MyFolder/archivename-$(date +%Y-%m-%d_%H_%M_%S).tar.gz -C /volume1/BACKUPS/MyFolder/MySubfolder/ .

This is maybe a second question, but somehow related to "only do something when the first thing is finished". How can I detect that the archive is also "finished"? So I don't end up opening/ unpacking an archive that still getting processed? Something like rename the file after the archiving is actually finished. How can I make sure that this renaming doesn't just start at some time and wait until the line before is actually ready, like with rsync?

Score:0
br flag

I don't see why this should run async, as long as there is no ampersand & in your original script at the end of line. Are you running this as a cron job? If yes, is the execution frequency maybe too high?

Logs from rsync should actually help here, at least to see if it is really finishing. Did you check them?

Apart from that, there are several ways to make these executions more controlled, as in 'do this if last was successfull'. The most simple one would be

rsync -abc --exwhyzed xxx && tar -abc xxx

note the double ampersand above. The latter command is then only executed if the first one has completed successful.

A more descriptive way would be if statements. Like

if rsync -abc --exwhyzed xxx ; then
    tar -abc xxx
fi

Also, why do you use the --progress option in combination with the verbose flag? When running nun-interactively and logging to a file, this doesn't seem very helpful to me. This is just a curious note, however.

jona avatar
id flag
Thanks. I will look into this. Yes, it is a cronjob. But this happened on initial first run and since then I didn't start it again (or the cronjob). The log is pretty spammed because of all the status percentages. But there is nothing in the log that says it's finished. Only for the single files. And there is nothing about any error. What does `xxx` stand for? Or is this a placeholder?
loopend avatar
br flag
Yes, these are just placeholders, just like the dummy options. It's just about the structure.
jona avatar
id flag
These are the last lines of the log `0 100% 0.00kB/s 0:00:00 (total: 58%) (xfer#47349, to-check=1004/85942) sent 1165678 bytes received 200646451 bytes 2028262.60 bytes/sec total size is 792975206 speedup is 3.93` So total was 58% and today (with no change in the log or me starting it again) it is finished (loaded the full thing via ftp and compares the files). And yes, `--progress` is pretty stupid, I will remove that to make the log more clear and run a new initial run
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.