Score:0

Server

rsync a directory with its own descendant

TravisCarden

6/25/23, 7:07 PM

tl;dr: I want to rsync a directory to its own descendant, and then rsync the said descendant back to the original directory--including deletions and exclusions in both directions.

Before you ask the obvious question, "Why would you want to do that?" or point out how much better another approach would be, this is a business requirement. It's not my choice, and I'm aware of the risks, so just indulge me. I do not intend to justify the approach any further.

Details:

I want to rsync a directory to its own descendant--i.e., a directory "underneath" or "inside" it, like parent to parent/child, for example--and then sync changes to the descendant back to the original directory, e.g., parent/child to parent, including deletions and exclusions in both directions. Visually, I need to do this:

parent -> parent/child
parent <- parent/child

The difficulties are to...

Prevent infinite recursion when going from the ancestor to the descendant
Not delete the source files mid-operation when syncing the descendant back to its ancestor (manifested as "file has vanished" errors)
Respect the exclusions all the while

0 + 0

rsync

Score:0

Server

TravisCarden

6/25/23, 7:07 PM

There are three parts to the solution. In order...

Exclude the descendant directory path relative to the ancestor, e.g., child when syncing from parent/child
Use the --delete-after flag instead of --delete
Make all exclusion paths relative to the source, e.g., exclude.txt as opposed to /var/www/parent/exclude.txt

Here's a test script (with some special formatting for nicer output):

#!/usr/bin/env bash

clear

printf "\n\e[33mrsync version:\n"
printf "%s--------------\e[39m\n"

rsync --version | head -n 1
# rsync  version 2.6.9  protocol version 29

rm -r parent

mkdir -p parent

touch parent/original.txt \
      parent/delete-from-child.txt \
      parent/exclude-from-child.txt

printf "\n\e[33mOriginal state:\n"
printf "%s---------------\e[39m\n"

tree -a --noreport -- parent

printf "\n\e[33mrsync to child:\n"
printf "%s---------------\e[39m\n"

printf "rsync --archive \\n      --delete-after \\n      --exclude=exclude-from-child.txt \\n      "$PWD"/parent/ \\n      "$PWD"/parent/child\n\n"
rsync --archive \
      --delete-after \
      --exclude=exclude-from-child.txt \
      "$PWD"/parent/ \
      "$PWD"/parent/child

tree -a --noreport -- parent; echo

printf "\e[33mMake changes:\n"
printf "%s---------------\e[39m\n"

rm parent/child/delete-from-child.txt
printf "rm parent/child/delete-from-child.txt\n"
touch parent/child/create-in-child.txt
printf "touch parent/child/create-in-child.txt\n"
touch parent/child/exclude-from-parent.txt
printf "touch parent/child/exclude-from-parent.txt\n\n"

printf "\e[33mrsync back to parent:\n"
printf "%s---------------------\e[39m\n"

printf "rsync --archive \\n      --delete-after \\n      --exclude=child \\n      --exclude=exclude-from-child.txt \\n      --exclude=exclude-from-parent.txt \\n      "$PWD"/parent/child/ \\n      "$PWD"/parent\n\n"
rsync --archive \
      --delete-after \
      --exclude=child \
      --exclude=exclude-from-child.txt \
      --exclude=exclude-from-parent.txt \
      "$PWD"/parent/child/ \
      "$PWD"/parent

tree -a --noreport -- parent; echo
# Or `find parent -type f; echo`, if you don't have `tree`

printf "%s\n\e[33mFinal comparison:\n"
printf "%s-----------------\e[39m\n"

diff --exclude=child parent parent/child; echo

Output:

rsync version:
--------------
rsync  version 2.6.9  protocol version 29

Original state:
---------------
parent
├── delete-from-child.txt
├── exclude-from-child.txt
└── original.txt

rsync to child:
---------------
rsync --archive
      --delete-after
      --exclude=exclude-from-child.txt
      /var/www/parent/
      /var/www/parent/child

parent
├── child
│   ├── delete-from-child.txt
│   └── original.txt
├── delete-from-child.txt
├── exclude-from-child.txt
└── original.txt

Make changes:
---------------
rm parent/child/delete-from-child.txt
touch parent/child/create-in-child.txt
touch parent/child/exclude-from-parent.txt

rsync back to parent:
---------------------
rsync --archive
      --delete-after
      --exclude=child
      --exclude=exclude-from-child.txt
      --exclude=exclude-from-parent.txt
      /var/www/parent/child/
      /var/www/parent

parent
├── child
│   ├── create-in-child.txt
│   ├── exclude-from-parent.txt
│   └── original.txt
├── create-in-child.txt
├── exclude-from-child.txt
└── original.txt


Final comparison:
-----------------
Only in parent: exclude-from-child.txt
Only in parent/child: exclude-from-parent.txt

0 + 0

Score:0

Server

András Korn

6/25/23, 8:14 PM

Another approach would be to set up bind mounts and thus make the two trees appear independent, then rsync as normal:

mkdir /mnt/parent
mkdir /mnt/child
mkdir /mnt/empty
mount --bind /path/to/parent/child /mnt/child
mount --bind /path/to/parent /mnt/parent
mount --bind /mnt/empty /mnt/parent/child

Now, when you do

rsync [args] /mnt/parent/. /mnt/child/.

the empty directory /mnt/empty will mask the real /path/to/parent/child directory; /mnt/parent/child will appear empty and will avoid infinite recursion.

When you do

rsync [args] /mnt/child/. /mnt/parent/.

the situation will be similar; /mnt/child/child/ will be empty (since you don't copy anything there in the first place), and /mnt/parent/child/ will also appear empty, so there is no risk of deleting anything from there.

This is perhaps less elegant than @TravisCarden's solution, but possibly more robust.

0 + 0

TravisCarden

6/28/23, 4:01 PM

How interesting! Unfortunately, I can't count on my particular target audience having access to mount or I would try this, but thanks for the idea!

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: rsync a directory with its own descendant

TH: rsync ไดเร็กทอรีที่มีลูกหลานของตัวเอง

RO: rsync un director cu propriul său descendent

RU: rsync каталог со своим потомком

VI: rsync một thư mục với hậu duệ của chính nó

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.