Redis & Advanced Caching

Redis Sentinel and High Availability

20 min Lesson 21 of 30

Redis Sentinel and High Availability

Redis Sentinel provides high availability for Redis through automatic failover, monitoring, and configuration management. It ensures your Redis deployment remains operational even when master nodes fail.

What is Redis Sentinel?

Redis Sentinel is Redis's built-in high availability solution that monitors your Redis instances, detects failures, and automatically promotes replicas to masters when needed.

Key Features: Sentinel provides monitoring, notification, automatic failover, and acts as a configuration provider for clients to discover the current master address.

How Sentinel Works

Sentinel uses a quorum-based system where multiple Sentinel processes monitor Redis instances and vote to determine if a master is down and which replica should be promoted.

Sentinel Architecture:
┌─────────────┐
│  Client     │
└──────┬──────┘
       │ Ask for master address
┌──────▼──────┐
│  Sentinel 1 │
└─────────────┘
       │ Monitor
┌──────▼──────┐     ┌─────────────┐
│   Master    │────>│  Replica 1  │
└─────────────┘     └─────────────┘
       │
┌──────▼──────┐
│  Replica 2  │
└─────────────┘

Configuring Redis Sentinel

Create a sentinel.conf configuration file to set up monitoring and failover behavior:

sentinel.conf:
# Monitor a master named "mymaster" at 127.0.0.1:6379
# Quorum of 2 means 2 Sentinels must agree for failover
sentinel monitor mymaster 127.0.0.1 6379 2

# Master is considered down after 5000ms without response
sentinel down-after-milliseconds mymaster 5000

# Timeout for failover operation
sentinel failover-timeout mymaster 10000

# How many replicas can be reconfigured simultaneously
sentinel parallel-syncs mymaster 1

# Optional authentication
sentinel auth-pass mymaster yourpassword

Tip: Always run at least 3 Sentinel instances in production to ensure proper quorum and avoid split-brain scenarios.

Starting Sentinel

Launch Sentinel using the redis-sentinel command or redis-server in sentinel mode:

# Method 1: Using redis-sentinel
redis-sentinel /path/to/sentinel.conf

# Method 2: Using redis-server
redis-server /path/to/sentinel.conf --sentinel

Sentinel Commands

Monitor and manage your Sentinel deployment using these commands:

# Connect to Sentinel
redis-cli -p 26379

# Get master address
SENTINEL get-master-addr-by-name mymaster

# List all monitored masters
SENTINEL masters

# Get replicas of a master
SENTINEL replicas mymaster

# Get Sentinel instances monitoring a master
SENTINEL sentinels mymaster

# Force failover
SENTINEL failover mymaster

# Check if master is down
SENTINEL ckquorum mymaster

# Reset Sentinel state
SENTINEL reset mymaster

Automatic Failover Process

When Sentinel detects a master failure, it follows this process:

Failover Steps:

Detection: Multiple Sentinels agree master is down (reaches quorum)
Leader Election: Sentinels elect a leader to perform failover
Replica Selection: Leader chooses best replica based on replication offset and priority
Promotion: Selected replica is promoted to master using SLAVEOF NO ONE
Reconfiguration: Other replicas are reconfigured to replicate from new master
Notification: Clients are notified of new master address

Connecting Clients to Sentinel

Clients should connect to Sentinel to discover the current master, not hardcode the master address:

PHP Example with Predis:
$sentinels = [
    'tcp://127.0.0.1:26379',
    'tcp://127.0.0.1:26380',
    'tcp://127.0.0.1:26381'
];

$client = new Predis\Client($sentinels, [
    'replication' => 'sentinel',
    'service' => 'mymaster'
]);

# Client automatically discovers master
$client->set('key', 'value');

Node.js Example with ioredis:
const Redis = require('ioredis');

const redis = new Redis({
    sentinels: [
        { host: '127.0.0.1', port: 26379 },
        { host: '127.0.0.1', port: 26380 },
        { host: '127.0.0.1', port: 26381 }
    ],
    name: 'mymaster'
});

redis.set('key', 'value');

Health Checks and Monitoring

Sentinel continuously monitors Redis instances and reports their status:

# Check Sentinel health
redis-cli -p 26379 PING
# Output: PONG

# Get detailed master info
SENTINEL master mymaster
# Shows: ip, port, flags, num-slaves, quorum, etc.

# Monitor Sentinel events in real-time
redis-cli -p 26379 --csv psubscribe '*'

Monitoring Tip: Subscribe to Sentinel's pub/sub channels to receive real-time notifications about failovers, reconfigurations, and state changes.

Sentinel Notifications

Configure Sentinel to send notifications when important events occur:

Email Notification Script:
# sentinel.conf
sentinel notification-script mymaster /path/to/notify.sh

# notify.sh
#!/bin/bash
EVENT=$1
MASTER=$2
echo "Sentinel event: $EVENT for $MASTER" | mail -s "Redis Alert" admin@example.com

Replica Priority

Control which replicas are preferred during failover using replica-priority:

# In redis.conf for each replica
# Lower numbers = higher priority (0 = never promote)
replica-priority 100

# Check priorities
SENTINEL replicas mymaster

Warning: Never run an even number of Sentinels (2, 4, 6). Always use odd numbers (3, 5, 7) to avoid split-brain scenarios where quorum cannot be reached.

Best Practices

Production Setup:

Run at least 3 Sentinel instances across different servers/availability zones
Use the same quorum value across all Sentinel configurations
Set down-after-milliseconds appropriately (5000-30000ms typical)
Monitor Sentinel logs for failover events and anomalies
Test failover procedures regularly in staging environments
Never hardcode master addresses in application code
Use authentication (sentinel auth-pass) in production

Exercise: Set up a Redis Sentinel cluster with 1 master, 2 replicas, and 3 Sentinels. Simulate a master failure by stopping the master process and observe automatic failover. Verify clients can still connect after failover.