The task is to spin up a Redis instance in a Kubernetes cluster.
Will use the Master-Slave replication setup with Sentinels for monitoring and failover operations.
Check the Redis: replication, part 2 — Master-Slave replication, and Redis Sentinel post for more details.
Redis cluster vs Redis replication
See Redis: replication, part 1 — an overview. Replication vs Sharding. Sentinel vs Cluster. Redis topology. and Choose between Redis Helm Chart and Redis Cluster Helm Chart.
In short:
- Replica — includes a Redis Master instance that performs read-write operations and copies data to its Redis Slaves instance(s) which serves Read-only operations. During this, such a Salve can be promoted to the Master’s role if its Master fails.
- Cluster — have a sense when your Redis have more data than your server’s RAM. The Cluster can use Sharding and a client requesting a piece of data will be redirected to a node that keeps that data.
Ways to run Redis in Kubernetes
Let’s see how we can perform the task — to run a Redis with replication in a Kubernetes cluster.
- manual setup — see the How to Create a Primary-Replica Redis Cluster in Kubernetes
- Redis Operator — redis-operator
- Helm chart which the Redis Cluster — https://bitnami.com/stack/redis-cluster
- Helm with Redis Master-Slave replication — https://bitnami.com/stack/redis (our choice in this post)
In our current case, we don’t need to worry about data’s persistence as our Redis will be used as a cache service only so we don’t need for a Kubernetes PersistentVolume.
Helm chart deploy
At first, we will run Redis services from the chart, will take a short look at them, and then will proceed to the available parameters.
Add the Bitnami repository to your Helm:
$ helm repo add bitnami https://charts.bitnami.com/bitnami “bitnami” has been added to your repositories Deploy the Redis chart:
$ helm install backend-redis bitnami/redis NAME: backend-redis LAST DEPLOYED: Tue Sep 22 14:48:02 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: \*\* Please be patient while the chart is being deployed \*\* Redis can be accessed via port 6379 on the following DNS names from within your cluster: backend-redis-master.default.svc.cluster.local for read/write operations backend-redis-slave.default.svc.cluster.local for read-only operations To get your password run: export REDIS\_PASSWORD=$(kubectl get secret — namespace default backend-redis -o jsonpath=”{.data.redis-password}” | base64 — decode) To connect to your Redis server: 1. Run a Redis pod that you can use as a client: kubectl run — namespace default backend-redis-client — rm — tty -i — restart=’Never’ \ — env REDIS\_PASSWORD=$REDIS\_PASSWORD \ — image docker.io/bitnami/redis:6.0.8-debian-10-r0 — bash 2. Connect using the Redis CLI: redis-cli -h backend-redis-master -a $REDIS\_PASSWORD redis-cli -h backend-redis-slave -a $REDIS\_PASSWORD To connect to your database from outside the cluster execute the following commands: kubectl port-forward — namespace default svc/backend-redis-master 6379:6379 & redis-cli -h 127.0.0.1 -p 6379 -a $REDIS\_PASSWORD Get the password which was generated during the chart’s deployment:
$ kubectl get secret — namespace default backend-redis -o jsonpath=”{.data.redis-password}” | base64 — decode TySS43UhAW Run theport-forward to connect to the Redis Master instance:
$ kubectl port-forward — namespace default svc/backend-redis-master 6379:6379 Forwarding from [::1]:6379 -> 6379 Forwarding from 127.0.0.1:6379 -> 6379 Connect:
$ redis-cli -h 127.0.0.1 -p 6379 -a TySS43UhAW Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe. 127.0.0.1:6379> “It works!” (с)
Check which Services do we have here:
$ kk get svc redis-service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEredis-service NodePort 172.20.119.242 <none> 6379:32445/TCP 117d Its type is the NodePort while we need for a LoadBalancer.
Also, we need for Redis Sentinel, which is off by default:
sentinel.enabled Enable sentinel containers
false
Okay, let’s go to the options to enable Sentinel and configure a LoadBalancer.
Redis Options
Create a list of options that can be useful for you.
In my case this will be:
- global.redis.password
- metrics:
- metrics.enabled
- metrics.serviceMonitor.enabled
- metrics.serviceMonitor.namespace
- Persistent:
- master.persistence.enabled
- slave.persistence.enabled
- Service
- master.service.type
- master.service.annotations
- slave.service.type
- slave.service.annotations
- Sentinel
- sentinel.enabled
- sentinel.service.type — need for LBso clients can ask for Master/Slaves
- sentinel.service.annotations
Create a new file ~/Temp/redis-opts.yaml to keep our desired parameters:
global: redis: password: "blablacar" metrics: enabled: true serviceMonitor: enabled: true namespace: "monitoring" master: persistence: enabled: false service: type: LoadBalancer annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internal slave: persistence: enabled: false service: type: LoadBalancer annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internal sentinel: enabled: true service: type: LoadBalancer annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internal Update the deployment with -f to specify our parameters file:
$ helm upgrade — install backend-redis bitnami/redis -f ~/Temp/redis-opts.yaml Check the password:
$ kubectl get secret — namespace default backend-redis -o jsonpath=”{.data.redis-password}” | base64 — decode blablacar And load-balancer:
$ kk get svc -l app=redis NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backend-redis LoadBalancer 172.20.204.235 af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com 6379:30647/TCP,26379:32523/TCP 80s backend-redis-headless ClusterIP None <none> 6379/TCP,26379/TCP 80s backend-redis-metrics ClusterIP 172.20.66.2 <none> 9121/TCP 80s Connect:
$ redis-cli -h af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com -a blablacar Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.\ af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com:6379> Works, okay.
But why the LoadBalancer is Public when it was set to be internal?
This because the alb.ingress.kubernetes.io/scheme: internal is used for the ALB Ingress Controller while the chart creates a simple Kubernetes Service with the LoadBalancer type which will create an AWS Classic Load Balancer.
Read the documentation — https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer and update our annotations: instead of the “alb.ingress.kubernetes.io/scheme: internal" - specify the "service.beta.kubernetes.io/aws-load-balancer-internal: "true"":
... master: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" slave: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" sentinel: enabled: true service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" Update the deployment and check again:
$ kk get svc -l app=redis NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backend-redis LoadBalancer 172.20.178.72 internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com 6379:31192/TCP,26379:32239/TCP 17s\ backend-redis-headless ClusterIP None <none> 6379/TCP,26379/TCP 17s backend-redis-metrics ClusterIP 172.20.35.163 <none> 9121/TCP 17s Now, make a Write operation — must be served by the Redis Master instance:
$ admin@bttrm-dev-app-1:~$ redis-cli -h internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar SET testkey testvalue OK And a Read operation:
admin@bttrm-dev-app-1:~$ redis-cli -h internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar GET testkey “testvalue” Replication status:
admin@bttrm-dev-app-1:~$ redis-cli -h internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar info replication Replication role:master connected\_slaves:1 slave0:ip=10.3.50.119,port=6379,state=online,offset=144165,lag=1 … And Sentinel status:
admin@bttrm-dev-app-1:~$ redis-cli -h internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com -p 26379 -a blablacar info sentinel Sentinel sentinel\_masters:1 sentinel\_tilt:0 sentinel\_running\_scripts:0 sentinel\_scripts\_queue\_length:0 sentinel\_simulate\_failure\_flags:0 master0:name=mymaster,status=ok,address=10.3.33.107:6379,slaves=1,sentinels=2 Although write operations need to be performed after getting the Master’s address via Sentinels:
admin@bttrm-dev-app-1:~$ redis-cli -h internal-\*\*\*-1786016015.us-east-2.elb.amazonaws.com -p 26379 -a blablacar sentinel get-master-addr-by-name mymaster 1) “10.3.33.107” r2) “6379” Check the documentation.
Done.
Originally published at RTFM: Linux, DevOps, and system administration.
Top comments (1)
@setevoy Is it possible to setup Redis Sentinel cluster with slaves across mulitple k8s clusters?