Patterns

35 items

System Design Pattern

Coordinationdistributed-lockmutexredlockcoordinationcritical-sectionadvanced

Distributed Locking Pattern

Mutual exclusion across distributed nodes

Used in: Redis, ZooKeeper, DynamoDB|25 min read

Summary

Distributed locking provides mutual exclusion across multiple processes or servers, ensuring only one client can access a resource at a time. Unlike local locks, distributed locks must handle network partitions, process crashes, and clock skew. Implementations include Redis (Redlock), ZooKeeper, etcd, and database-based locks. Key challenges are: ensuring lock release on failure (TTL/leases), preventing stale lock holders from operating (fencing tokens), and handling split-brain. Essential for coordinating access to shared resources in distributed systems.

Key Takeaways

Locks Must Auto-Expire

If lock holder crashes, lock must eventually release. Use TTL or lease-based expiration. But TTL creates timing issues - holder might still be working when lock expires.

Fencing Tokens Prevent Stale Operations

After lock expires, old holder might still send operations. Include monotonically increasing token with all operations. Storage rejects operations with old tokens.

Redlock is Controversial

Redis Redlock attempts fault-tolerant locking across N Redis instances. Requires majority. Criticized for clock dependency. Use with caution for critical sections.

Scenario: Two processes try to withdraw $100 from account with $150 balance.

Without locking: 1. Process A reads balance: $150 2. Process B reads balance: $150 3. Process A: $150 - $100 = $50, writes $50 4. Process B: $150 - $100 = $50, writes $50 5. Result: $50 balance, but $200 withdrawn!

With distributed lock: 1. Process A acquires lock 2. Process A reads $150, withdraws $100, writes $50 3. Process A releases lock 4. Process B acquires lock 5. Process B reads $50, insufficient funds!

Summary

Key Takeaways

Locks Must Auto-Expire

If lock holder crashes, lock must eventually release. Use TTL or lease-based expiration. But TTL creates timing issues - holder might still be working when lock expires.

Fencing Tokens Prevent Stale Operations

After lock expires, old holder might still send operations. Include monotonically increasing token with all operations. Storage rejects operations with old tokens.

Redlock is Controversial

Redis Redlock attempts fault-tolerant locking across N Redis instances. Requires majority. Criticized for clock dependency. Use with caution for critical sections.

ZooKeeper/etcd Are Safer

Consensus-based systems (ZK, etcd) provide linearizable operations. More reliable than Redis for critical locks. Higher latency but stronger guarantees.

Lock Granularity Matters

Coarse locks (one lock for all resources) are simple but serialize everything. Fine-grained locks (per-resource) allow parallelism but increase complexity.

Deadlock Detection/Prevention

Multiple locks can deadlock. Use lock ordering, timeouts, or deadlock detection. In distributed systems, timeouts are most practical.

Pattern Details

Scenario: Two processes try to withdraw $100 from account with $150 balance.

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Replication Pattern

Caching Strategies Pattern

Persistent Connections Pattern

Load Balancing Pattern

Fan-out Pattern

Fan-in Pattern

Circuit Breaker Pattern

Eventual Consistency Pattern

Queue-based Load Leveling Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern

Write-Ahead Log Pattern

API Gateway Pattern

Backend for Frontend Pattern

Sidecar Pattern

Idempotency Pattern

Rate Limiting Pattern

Backpressure Pattern

Pub/Sub Pattern

Strong Consistency Pattern

Conflict Resolution Pattern

Leader Election Pattern

Consensus Protocols Pattern

CQRS Pattern

LSM Trees Pattern

Sharding Pattern

Event Sourcing Pattern

Stream Processing Pattern

Change Data Capture Pattern