keydb active-replica multi-master php session storage

I'm looking to use keydb/redis for php session storage in a HA setup. Currently, I have two keydb instances running with the following config:

Instance 1 (192.168.2.10)

requirepass pass123
masterauth pass123
# you will need to configure the following
multi-master yes
active-replica yes
replicaof 192.168.2.9 6379

Instance 2 (192.168.2.9)

requirepass pass123
masterauth pass123
# you will need to configure the following
multi-master yes
active-replica yes
replicaof 192.168.2.10 6379

I have the following then configured within the php settings

session.save_handler = redis
session.save_path = "tcp://192.168.2.10:6379?timeout=1.5&auth=pass123, tcp://192.168.2.9:6379?timeout=1.5&auth=pass123"

When both instances are online, everything works as expected. When I take one of the keydb servers offline, php will error out if it selects that instance - instead of using one that is active.

Is there a way to this without running it through haproxy or sentinel style configuration?

I had a similar problem and did not want to go through the hassle up setting up haproxy. Instead, I put my Keydb hostnames in my /etc/hosts file and then created a check_keydb script that monitors each server and then modifies the /etc/hosts file if one of the Keydb servers goes done. This script runs once a minute from cron. It will also send me an email alert if it detects a problem.

To check the status of the Keydb servers, I use nc to send a PING command and then wait for a +PONG response.

In my case, I'm just using one primary Keydb server and the other is a hot backup. However, you could easily modify the script to handle multiple active servers. If a server goes down, just make the script remove its hostname from the /etc/hosts file.

Below is a copy of my script. Of course, you'll need to adjust the IP addresses and make any changes necessary for your particular configuration.

#
# Checks both connections to our mirrored Keydb/Redis servers.
# If one of them is down, adjust the keydb hostname in /etc/hosts
# and make it point to the IP address of the active server.
#

FROM_EMAIL="[email protected]"
NOTIFY_EMAIL="[email protected]"

PORT=6379

TARGET1=keydb1
TARGET2=keydb2

TARGET1_STATUS=up
TARGET2_STATUS=up

RESPONSE=$(echo PING | nc -w 2 $TARGET1 $PORT | tr -d '\r\n')
if [ "$RESPONSE" != "+PONG" ]
then
    TARGET1_STATUS=down
fi

RESPONSE=$(echo PING | nc -w 2 $TARGET2 $PORT | tr -d '\r\n')
if [ "$RESPONSE" != "+PONG" ]
then
    TARGET2_STATUS=down
fi

HOSTS_FILE=/etc/hosts

if [[ "$TARGET1_STATUS" != "up" || "$TARGET2_STATUS" != "up" ]]
then
    grep -v -E '^\s*10\.15\.124\.3[12]\s+keydb\s+' $HOSTS_FILE > /tmp/hosts.tmp

    cat <<__EOT__ | mail -r $FROM_EMAIL -s 'keydb server has problems' $NOTIFY_EMAIL
$TARGET1:$PORT $TARGET1_STATUS
$TARGET2:$PORT $TARGET2_STATUS
__EOT__
fi

if [ "$TARGET1_STATUS" != "up" ]
then
    cp /tmp/hosts.tmp $HOSTS_FILE
    echo "10.15.124.32  keydb   # main connection" >> $HOSTS_FILE

    echo $TARGET1:$PORT $TARGET1_STATUS
    exit 1
fi

if [ "$TARGET2_STATUS" != "up" ]
then
    cp /tmp/hosts.tmp $HOSTS_FILE
    echo "10.15.124.31  keydb   # main connection" >> $HOSTS_FILE

    echo $TARGET2:$PORT $TARGET2_STATUS
    exit 1
fi

echo "OK"
exit 0

WARNING: before you make your script active on a production server, make sure you save a copy of your original /etc/hosts file. I should know better after 30 years, but I forgot to do this and accidentally clobbered my original and caused several important services to go offline for a few minutes until I could restore the file from a backup. Just do a cp /etc/hosts /etc/hosts.bak and you could save yourself a lot of trouble if something goes wrong.

Also, if you have multiple webservers accessing your Keydb servers, you'll need to run this script on each server to make it adjust their local /etc/hosts file. However, if that's the case, it would probably make more sense to just set up haproxy and do real load-balancing/failover.

As an alternative to modifying the /etc/hosts file, you could create a similar script to modify your php.ini file and set your session.save_path = variable to only include the currently active Keydb servers (just remember to do a "reload" command for whatever webserver you're running).