Score:1

Redis Sentinel with TLS - how to get node FQDN rather than IP?

br flag

This may be a Serverfault or a Stack Overflow question, I'm not sure yet:

I've set up a simple three node Redis system with a master and two replica nodes, managing failover with Redis Sentinel. Redis and Sentinel network traffic is secured using Redis built-in TLS support and regular CA issued certificates.

Each Sentinel instance is configured to announce its hostname and to resolve DNS:

sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel announce-ip "redistest2.mydomain.com"

We have a web service using Servicestack to connect to the Sentinel instances. As long as we don't validate TLS certificates and hostnames everything works well: The web service can see the Redis Sentinel listeners, and when we terminate the current Master, the Redis cluster votes on a new one, and the web service switches to the new writable Redis node.

However, while the original Master node reports in with its FQDN, the two backup nodes only ever seem to report back to ServiceStack with their IP addresses.

A Sentinel log excerpt indicates that the backup nodes seem to use their host names:

28011:X 15 Feb 2023 15:23:10.817 * +sentinel sentinel <hex-string> redistest2.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379
28011:X 15 Feb 2023 15:23:10.821 * Sentinel new configuration saved on disk
28011:X 15 Feb 2023 15:23:10.897 * +sentinel sentinel <other-hex-string> redistest3.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379
28011:X 15 Feb 2023 15:23:10.901 * Sentinel new configuration saved on disk

Yet ServiceStack insists that it only receives the server IP addresses back from the server group:

Starting with sentinel.
Sentinel hosts: redistest1.mydomain.com:26379?ssl=true, redistest2.mydomain.com:26379?ssl=true, redistest3.mydomain.com:26379?ssl=true
Sentinel created
Host filter set.
Hostfilter: redistest1.mydomain.com:6379
Hostfilter: 10.100.60.72:6379
Hostfilter: 10.100.60.73:6379
RedisManager started.
Redis sentinel info: redistest primary: redistest1.mydomain.com:6379, replicas: 10.100.60.72:6379, 10.100.60.73:6379
Hostfilter: 10.100.60.72:6379
Hostfilter: 10.100.60.73:6379
Ping error with read only client: ServiceStack.Redis.RedisException: [14:23:47.626] Unable to Connect: sPort: 0, Error: One or more errors occurred.
(...)
---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

Is there anything more I can do on the Redis and/or Sentinel configuration side to ensure ServiceStack receives the actual host names of the Redis nodes in order for us to correctly validate the certificates used?

Score:0
br flag

The ServiceStack log contains the necessary clue to what went wrong: Sentinel did exactly what it should, but the Redis backup nodes were being referred to by IP address.

Similar to how Sentinel can return a string containing the FQDN in the sentinel announce-ip configuration statement, Redis can do the same with the replica-announce-ip configuration statement.

The solution was to add the following line to the Redis configuration file on all Backup hosts:

replica-announce-ip servername.mydomain.com

Servername in this case of course is the hostname of the machine running this specific backup node.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.