How to read with timeout without making bash consider it an error?

I have a read_user_input.sh script:

#!/bin/bash

# set -e

prompt="bla? [Y/n] "

while true; do
    read -p "$prompt" -n 1 -s -t 3 reply
    case $reply in
        ""|Y|y) echo "bla!!!"; break;;
        N|n) echo "no bla :( you suck"; break;;
        *) ;;
    esac
done

Works as I expected - i.e.

• User inputs "Y" -> bla
• User inputs "y" -> bla
• User hits Enter -> bla
• Timeout -> bla

However, when adding set -e - the read ends with error > 128.

From read --help:

Exit Status:

The return code is zero, unless end-of-file is encountered, read times out (in which case it's greater than 128), a variable assignment error occurs, or an invalid file descriptor is supplied as the argument to -u.

What would be the best way to overcome this?

1. adding || true seems not right, as it would hide any real errors.
2. I also don't want to remove the set -e.
3. One other thing I thought about was handling it in trap but that seems like an overkill

Here is a hack where is used command substitution which forks a subshell where -e is not enabled.

#!/bin/bash

set -e

prompt="bla? [Y/n] "

while true; do
    case $(read -p "$prompt" -n 1 -s -t 3 reply; echo $reply) in
        ""|Y|y) echo "bla!!!"; break;;
        N|n) echo "no bla :( you suck"; break;;
        *) ;;
    esac
done

Using @choroba answer, I changed my switch case into:

#!/bin/bash

set -e

prompt="bla? [Y/n] "

while true; do
    read -p "$prompt" -n 1 -s -t 3 reply || (( $? > 128 ))
    case $reply in
        ""|Y|y) echo "bla!!!"; break;;
        N|n) echo "no bla :( you suck"; break;;
        *) ;;
    esac
done