Redis & Advanced Caching

Redis Transactions and Pipelining

20 min Lesson 13 of 30

Redis Transactions and Pipelining

Redis provides two powerful features for executing multiple commands efficiently: transactions for atomic operations and pipelining for batch execution. Understanding these features is crucial for building high-performance applications.

Redis Transactions (MULTI/EXEC)

Redis transactions allow you to execute a group of commands atomically - all commands succeed or all fail:

Basic Transaction:

MULTI           # Start transaction
SET user:1 "John"
SET user:2 "Jane"
INCR user:count
EXEC            # Execute all commands

# Returns array of results:
1) OK
2) OK
3) (integer) 3

Key Characteristics:

All commands are queued and executed together
Atomic execution - no other client can interrupt
All-or-nothing: if one command has syntax error, none execute
If command fails at runtime (e.g., wrong type), other commands still execute

DISCARD - Cancel Transaction

Cancel a queued transaction before execution:

Example:

MULTI
SET user:1 "John"
SET user:2 "Jane"
DISCARD         # Cancel transaction - nothing executed

# Returns: OK

Laravel Redis Transactions

Laravel provides a clean API for Redis transactions:

Using transaction() Method:

use Illuminate\Support\Facades\Redis;

// Simple transaction
Redis::transaction(function ($redis) {
    $redis->set('user:1', 'John');
    $redis->set('user:2', 'Jane');
    $redis->incr('user:count');
});

// Transaction with return value
$results = Redis::transaction(function ($redis) {
    $redis->set('balance:1', 1000);
    $redis->decrby('balance:1', 100);
    return $redis->get('balance:1');
});

// $results contains array of command results

Manual Transaction Control:

Redis::multi();
Redis::set('key1', 'value1');
Redis::set('key2', 'value2');
Redis::incr('counter');
$results = Redis::exec();  // Execute and return results

// Or cancel
Redis::multi();
Redis::set('key1', 'value1');
Redis::discard();  // Cancel transaction

WATCH - Optimistic Locking

WATCH monitors keys for changes and aborts the transaction if any watched key is modified:

Optimistic Locking Pattern:

// Redis CLI example
WATCH balance:1        # Monitor this key
GET balance:1          # Read current value: 1000

# If another client modifies balance:1 here, transaction will fail

MULTI
DECRBY balance:1 100   # Deduct 100
EXEC                   # Execute only if balance:1 unchanged

# Returns: (nil) if watched key was modified
# Returns: [OK, 900] if successful

Laravel Implementation - Bank Transfer:

public function transfer(int $fromAccountId, int $toAccountId, float $amount)
{
    $maxRetries = 5;
    $attempt = 0;

    while ($attempt < $maxRetries) {
        try {
            // Watch both account balances
            Redis::watch("balance:{$fromAccountId}", "balance:{$toAccountId}");

            // Read current balances
            $fromBalance = (float) Redis::get("balance:{$fromAccountId}");
            $toBalance = (float) Redis::get("balance:{$toAccountId}");

            // Validate
            if ($fromBalance < $amount) {
                Redis::unwatch();
                throw new \Exception('Insufficient balance');
            }

            // Execute transaction
            $results = Redis::transaction(function ($redis) use (
                $fromAccountId, $toAccountId, $amount, $fromBalance, $toBalance
            ) {
                $redis->set("balance:{$fromAccountId}", $fromBalance - $amount);
                $redis->set("balance:{$toAccountId}", $toBalance + $amount);
                $redis->lpush('transactions', json_encode([
                    'from' => $fromAccountId,
                    'to' => $toAccountId,
                    'amount' => $amount,
                    'timestamp' => time()
                ]));
            });

            if ($results !== null) {
                return true;  // Success
            }

            // Transaction failed due to watched key change - retry
            $attempt++;
            usleep(50000);  // Wait 50ms before retry

        } catch (\Exception $e) {
            Redis::unwatch();
            throw $e;
        }
    }

    throw new \Exception('Transaction failed after maximum retries');
}

Important: Always call UNWATCH or execute/discard the transaction to release watched keys. Failing to do so can cause memory leaks.

Pipelining - Batch Command Execution

Pipelining sends multiple commands to Redis without waiting for individual responses, reducing network round-trips:

Without Pipelining (Slow):

// 1000 commands = 1000 network round-trips
for ($i = 0; $i < 1000; $i++) {
    Redis::set("key:{$i}", "value:{$i}");  // Wait for response each time
}
// Time: ~2 seconds (with 2ms latency per request)

With Pipelining (Fast):

// Laravel pipelining
Redis::pipeline(function ($pipe) {
    for ($i = 0; $i < 1000; $i++) {
        $pipe->set("key:{$i}", "value:{$i}");
    }
});
// Time: ~10ms (single network round-trip)

// Returns array of all results

Performance Boost: Pipelining can improve performance by 10-100x when executing many commands. It's especially effective for high-latency connections.

Pipelining vs Transactions

Key Differences:

Transactions (MULTI/EXEC):
✓ Atomic execution - all or nothing
✓ Commands execute in isolation
✓ Guarantees consistency
✗ Slower than pipelining
✗ Limited throughput

Pipelining:
✓ Maximum throughput
✓ Minimal network overhead
✓ Can include any commands
✗ Not atomic - commands can be interleaved
✗ No consistency guarantees
✗ Partial failures possible

Combining Pipelining and Transactions

Use both for maximum performance with atomicity:

Pipelined Transactions:

// Execute multiple independent transactions in one batch
Redis::pipeline(function ($pipe) {
    // Transaction 1: Update user 1
    $pipe->multi();
    $pipe->set('user:1:name', 'John');
    $pipe->set('user:1:email', 'john@example.com');
    $pipe->exec();

    // Transaction 2: Update user 2
    $pipe->multi();
    $pipe->set('user:2:name', 'Jane');
    $pipe->set('user:2:email', 'jane@example.com');
    $pipe->exec();

    // Transaction 3: Update counters
    $pipe->multi();
    $pipe->incr('users:count');
    $pipe->incr('updates:count');
    $pipe->exec();
});

Lua Scripting Basics

Lua scripts provide true atomic operations with complex logic:

Simple Lua Script:

// Redis CLI
EVAL "return redis.call('SET', KEYS[1], ARGV[1])" 1 mykey myvalue

// Laravel - Lua script for atomic increment with limit
$script = <<<LUA
local current = redis.call('GET', KEYS[1])
if not current then
    current = 0
end
if tonumber(current) < tonumber(ARGV[1]) then
    return redis.call('INCR', KEYS[1])
else
    return -1
end
LUA;

$result = Redis::eval($script, 1, 'counter', 100);
// Returns new counter value or -1 if limit reached

Laravel - Rate Limiter with Lua:

public function checkRateLimit(string $key, int $limit, int $window): bool
{
    $script = <<<LUA
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call('GET', key)

if not current then
    redis.call('SETEX', key, window, 1)
    return 1
elseif tonumber(current) < limit then
    redis.call('INCR', key)
    return tonumber(current) + 1
else
    return -1
end
LUA;

    $result = Redis::eval($script, 1, $key, $limit, $window);
    return $result !== -1;
}

Lua Advantages:

Guaranteed atomic execution
Complex logic on Redis server (reduced network traffic)
Better performance than WATCH-based optimistic locking
Scripts are cached by Redis (use SCRIPT LOAD for repeated execution)

Best Practices

When to Use Each Approach:

1. Use Transactions (MULTI/EXEC):
   - Simple atomic operations
   - 2-10 related commands
   - Clear success/failure semantics

2. Use WATCH + Transactions:
   - Conditional updates based on current values
   - Optimistic locking scenarios
   - Low contention keys

3. Use Pipelining:
   - Bulk operations (100+ commands)
   - Read-only operations
   - High-latency connections
   - When atomicity not required

4. Use Lua Scripts:
   - Complex conditional logic
   - High contention scenarios
   - Need guaranteed atomicity with logic
   - Repeated operations (cache script)

Practice Exercise:

Implement a shopping cart checkout using WATCH and transactions (check inventory, deduct stock, create order)
Create a bulk import function using pipelining to insert 10,000 records
Build a Lua script for atomic "pop and push" between two lists with conditional logic
Implement a distributed counter with Lua that resets automatically after reaching a threshold
Compare performance: run 1000 SET commands with and without pipelining, measure time difference