2

This may be a Serverfault or a Stack Overflow question, I'm not sure yet:

I've set up a simple three node Redis system with a master and two replica nodes, managing failover with Redis Sentinel. Redis and Sentinel network traffic is secured using Redis built-in TLS support and regular CA issued certificates.

Each Sentinel instance is configured to announce its hostname and to resolve DNS:

sentinel resolve-hostnames yes sentinel announce-hostnames yes sentinel announce-ip "redistest2.mydomain.com"

We have a web service using Servicestack to connect to the Sentinel instances. As long as we don't validate TLS certificates and hostnames everything works well: The web service can see the Redis Sentinel listeners, and when we terminate the current Master, the Redis cluster votes on a new one, and the web service switches to the new writable Redis node.

However, while the original Master node reports in with its FQDN, the two backup nodes only ever seem to report back to ServiceStack with their IP addresses.

A Sentinel log excerpt indicates that the backup nodes seem to use their host names:

28011:X 15 Feb 2023 15:23:10.817 * +sentinel sentinel <hex-string> redistest2.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379 28011:X 15 Feb 2023 15:23:10.821 * Sentinel new configuration saved on disk 28011:X 15 Feb 2023 15:23:10.897 * +sentinel sentinel <other-hex-string> redistest3.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379 28011:X 15 Feb 2023 15:23:10.901 * Sentinel new configuration saved on disk

Yet ServiceStack insists that it only receives the server IP addresses back from the server group:

Starting with sentinel. Sentinel hosts: redistest1.mydomain.com:26379?ssl=true, redistest2.mydomain.com:26379?ssl=true, redistest3.mydomain.com:26379?ssl=true Sentinel created Host filter set. Hostfilter: redistest1.mydomain.com:6379 Hostfilter: 10.100.60.72:6379 Hostfilter: 10.100.60.73:6379 RedisManager started. Redis sentinel info: redistest primary: redistest1.mydomain.com:6379, replicas: 10.100.60.72:6379, 10.100.60.73:6379 Hostfilter: 10.100.60.72:6379 Hostfilter: 10.100.60.73:6379 Ping error with read only client: ServiceStack.Redis.RedisException: [14:23:47.626] Unable to Connect: sPort: 0, Error: One or more errors occurred. (...) ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

Is there anything more I can do on the Redis and/or Sentinel configuration side to ensure ServiceStack receives the actual host names of the Redis nodes in order for us to correctly validate the certificates used?

Improve this question
0

1 Answer 1

Reset to default
1

The ServiceStack log contains the necessary clue to what went wrong: Sentinel did exactly what it should, but the Redis backup nodes were being referred to by IP address.

Similar to how Sentinel can return a string containing the FQDN in the sentinel announce-ip configuration statement, Redis can do the same with the replica-announce-ip configuration statement.

The solution was to add the following line to the Redis configuration file on all Backup hosts:

replica-announce-ip servername.mydomain.com

Servername in this case of course is the hostname of the machine running this specific backup node.

Improve this answer
1
  • The redis docs (github.com/redis/redis/blob/unstable/redis.conf) do not mention that a fqdn is valid for this option, but I take it it's still the right thing to do to get redis replicating via FQDNs. Commented May 6, 2024 at 9:36

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.