Patterns

35 items

System Design Pattern

Data Distributionfan-outwrite-amplificationtimelinefeeddistributionintermediate

Fan-out Pattern

Distributing writes to many readers efficiently

Used in: Twitter Timeline, News Feed, Notification Service|20 min read

Summary

Fan-out is a messaging pattern where a single message is distributed to multiple consumers simultaneously. When a producer publishes a message, it's delivered to all subscribed consumers in parallel, enabling broadcast communication. This pattern is fundamental to notification systems (one post notifies all followers), event-driven architectures (one order event triggers inventory, shipping, and billing services), and cache invalidation (one update invalidates caches across all servers). Fan-out enables loose coupling since producers don't need to know who or how many consumers exist.

Key Takeaways

One-to-Many Distribution

A single message from one producer reaches many consumers simultaneously. This is the opposite of load balancing where messages go to one of many workers. Every subscriber receives every message.

Decoupled Producers and Consumers

Producers don't know how many consumers exist or who they are. Consumers can be added or removed without changing producer code. This loose coupling enables independent scaling and deployment.

Parallel Processing

All consumers receive and process the message concurrently. This maximizes throughput when multiple systems need to react to the same event. Total processing time is the slowest consumer, not the sum.

Modern applications need to notify multiple systems when something happens:

E-commerce Order Example: When a customer places an order, you need to: - Update inventory (reserve items) - Process payment (charge credit card) - Send confirmation email - Update analytics (track conversion) - Notify warehouse (prepare shipment) - Update recommendation engine (purchase history)

Without fan-out, you would need synchronous calls to each system, creating tight coupling and slow response times.

Synchronous vs Fan-Out Architecture

Key Benefit

With fan-out, the Order Service publishes one message and returns immediately. All downstream processing happens asynchronously and in parallel. Response time drops from sum of all calls to just the publish time.

Summary

Key Takeaways

One-to-Many Distribution

A single message from one producer reaches many consumers simultaneously. This is the opposite of load balancing where messages go to one of many workers. Every subscriber receives every message.

Decoupled Producers and Consumers

Producers don't know how many consumers exist or who they are. Consumers can be added or removed without changing producer code. This loose coupling enables independent scaling and deployment.

Parallel Processing

Message Ordering Challenges

Different consumers may process messages at different speeds, leading to out-of-order processing. If consumer A processes message 2 before message 1, consistency issues can arise. Design for eventual consistency.

Delivery Guarantees Vary

Fan-out systems offer different guarantees: at-most-once (fast but may lose messages), at-least-once (reliable but may duplicate), exactly-once (slow but precise). Choose based on use case requirements.

Scaling Considerations

Fan-out factor (number of consumers) directly impacts system load. 1 message to 1M followers means 1M deliveries. Social networks use techniques like pull-on-demand for high-follower accounts.

Pattern Details

Modern applications need to notify multiple systems when something happens:

Without fan-out, you would need synchronous calls to each system, creating tight coupling and slow response times.

Synchronous vs Fan-Out Architecture

Key Benefit

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Queue-based Load Leveling Pattern

Replication Pattern

Caching Strategies Pattern

Fan-out Pattern

Fan-in Pattern

Persistent Connections Pattern

Load Balancing Pattern

Circuit Breaker Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern

Write-Ahead Log Pattern

API Gateway Pattern

Backend for Frontend Pattern

Sidecar Pattern

Idempotency Pattern

Rate Limiting Pattern

Backpressure Pattern

Pub/Sub Pattern

Eventual Consistency Pattern

Sharding Pattern

Conflict Resolution Pattern

Strong Consistency Pattern

Leader Election Pattern

Consensus Protocols Pattern

Stream Processing Pattern

Change Data Capture Pattern

Distributed Locking Pattern

Two-Phase Commit Pattern

LSM Trees Pattern

Event Sourcing Pattern

CQRS Pattern

Fan-out Pattern

Summary

Key Takeaways

One-to-Many Distribution

Decoupled Producers and Consumers

Parallel Processing

The Problem: Coordinating Multiple Systems

Synchronous vs Fan-Out Architecture

Key Benefit

Premium Content

Fan-out Pattern

Summary

Key Takeaways

One-to-Many Distribution

Decoupled Producers and Consumers

Parallel Processing

The Problem: Coordinating Multiple Systems

Synchronous vs Fan-Out Architecture

Key Benefit

Premium Content