Ignoring the extraneous comma (,) characters, your issue here is with quoting.
Backticks are a deprecated syntax for command substitution, and retain that special meaning inside double quotes. So whereas
printf '%c%c\n' '`' '`'
works,
printf '%c%c\n' "`" "`"
is equivalent to printf '%c%c\n' "$(" ")", which attempts to execute a literal space as a command and pass the result as an argument to printf:
$ printf '%c%c\n' "`" "`"
 : command not found
So, why can't you simply define your alias using the single quoted form? Well:
- single quotes don't nest, so - alias key='printf '%c%c\n' '`' '`' ' 
 - concatenates - 'printf ',- %c%c\n, and- ' 'into a single format string, leaving (unquoted)
 - `' '`' '
 - which your interactive shell expands equivalent to - $(' ')' '- again trying to execute a literal space.
 
- on the other hand, if you try to use outer double quotes - alias key="printf '%c%c\n' '`' '`' "
 - then according to the quoting rules linked above, the single quotes become literal, and your shell again expands - ' 'as a command.
 
What does work is to escape the single quote:
$ unalias key
$ alias key='printf "%c%c\n" '\''`'\'' '\''`'\'' '
$ 
$ key
``
For your full alias that would be
alias key='printf "\n\033[33m= [ %c%c%c~+ |\ ]\n\n" '\''`'\'' '\''`'\'' "%" '
(although the quoting of "%" isn't strictly necessary).
If the "leaning toothpicks" become too much to handle, you could consider using a shell function instead of an alias, eliminating one level of quoting1:
$ key () { printf '\n\033[33m= [ %c%c%c~+ |\ ]\n\n' '`' '`' '%' ; }
$ 
$ key
= [ ``%~+ |\ ]
See also In Bash, when to alias, when to script and when to write a function?
- in fact, the bash manual's own section on aliases says "For almost every purpose, shell functions are preferred over aliases."