Score:1

Curly braces in a bash variable expands correctly, but the rsync isn't properly executing

cn flag

Why won't these braces work in a variable?

I have bash script with rsync with a --exclude option that calls a variable named EXCLUDE. That variable include literal curly braces.

#!/bin/bash

set -eB

EXCLUDE="{'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'}"

rsync -av --dry-run --exclude=$EXCLUDE web/ ${DEPLOY_USER}@${DEPLOY_HOSTNAME}:/sites/${DEPLOY_DOMAIN}/files/web

If I echo $EXCLUDE I get the correct string:

echo $EXCLUDE
{'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'}

If I run the script without the EXCLUDE variable I get the expected results:

rsync -av --dry-run --exclude={'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'} web/ username@123.456.789.0:/sites/xyz.org/files/web

building file list ... done
./
test.txt

However if I run the command with the EXCLUDE variable present the results are not correct:

rsync -av --dry-run --exclude=${EXCLUDE} web/ ${DEPLOY_USER}@${DEPLOY_HOSTNAME}:/sites/${DEPLOY_DOMAIN}/files/web

building file list ... done
./
node_modules
test.log
test.txt

Note that if I use echo !! to echo the last command that was run, the results are literally identical to the version without the $EXCLUDE variable:

echo !!
echo rsync -av --dry-run --exclude=$EXCLUDE web/ ${DEPLOY_USER}@${DEPLOY_HOSTNAME}:/sites/${DEPLOY_DOMAIN}/files/web
rsync -av --dry-run --exclude={'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'} web/ username@123.456.789.0:/sites/xyz.org/files/web

I can even take the command that was returned by echo !!, run it, and get the expected results!

I've tried enclosing $EXCLUDE in curly braces, I've tried escaping the braces in the variable, I've tried set -B and with and without -e. Even though bash seems to be sending literally curly braces to rsync rsync is not evaluating the contents to the braces.

Why won't these braces work in a variable?

tm flag
Brace expansion in bash happens before variable expansion.
Score:2
tm flag

The command with curly braces works because the shell runs the brace expansion on it.

--exclude={'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'}

becomes

--exclude=.env --exclude=.git* --exclude=*.log --exclude=config/ssh --exclude=/dev --exclude=node_modules --exclude=/web/app/uploads

Variable expansion happens later than brace expansion, so storing the braces in a variable doesn't work.

You can use an array instead:

excludes=(.env '.git*' '*.log' config/ssh /dev node_modules /web/app/uploads)
rsync -av --dry-run "${excludes[@]/#/--exclude=}" ...

The ${array[@]/#/PREPEND} syntax is described in man bash under Parameter Expansion, it prepends PREPEND to the beginning of each member of the array.

Slam avatar
cn flag
Fascinating! I had no idea there were different kinds of brace expansion. I thought all braces were essentially arrays. Now I see they aren't really arrays at all. Bash continues to impress me.
Score:1
hr flag

AFAIK rsync itself only understands simple globs in the --exclude argument. So you're relying on the interactive shell to process the brace expansion.

That means {'.env','.git*','*.log','config/ssh','/dev','node_modules','/web/app/uploads'} isn't "the correct string" - you want the rsync command line to be passed with --exclude='.env' --exclude='.git*' ... already expanded.

You could use brace expansion within an array, and then expand the array within the rsync command:

EXCLUDE=(--exclude={'.env','.git*','*.log','config/ssh','/dev','node_modules',/web/app/uploads'})

rsync -av --dry-run "${EXCLUDE[@]}" web/ username@123.456.789.0:/sites/xyz.org/files/web

See also How can we run a command stored in a variable?

Slam avatar
cn flag
Wild! I'm presuming Bash is smart enough to not expand the variable assignment of `EXCLUDE={--exclude={'.env','.git*',...}` into `EXCLUDE='.env' EXCLUDE='.git*' EXCLUDE=...`?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.