Patterns

35 items

System Design Pattern

Data Distributionfan-inaggregationmetricscollectionconsolidationintermediate

Fan-in Pattern

Aggregating data from many sources into one destination

Used in: Metrics Collection, Log Aggregation, Analytics Pipeline|20 min read

Summary

Fan-in is a messaging pattern where multiple producers send messages that are aggregated by a single consumer or processing point. It's the opposite of fan-out: many sources converge to one destination. This pattern is essential for aggregating data from distributed sources (collecting logs from thousands of servers), implementing scatter-gather (query multiple databases and combine results), and coordinating parallel work (waiting for all workers to complete). Fan-in reduces complexity by providing a single point for aggregation, correlation, and final processing.

Key Takeaways

Many-to-One Aggregation

Multiple producers send messages to a single consumer or aggregation point. This centralizes processing and simplifies downstream logic. Common in logging, metrics collection, and data pipelines.

Correlation and Ordering

Messages from different sources may arrive out of order. Fan-in often requires correlation IDs to group related messages and ordering logic to process them correctly.

Backpressure Management

When many producers send to one consumer, the consumer can become overwhelmed. Implement backpressure mechanisms: rate limiting, buffering, or flow control to prevent overload.

Distributed Search Example:

When a user searches on an e-commerce site: 1. Query goes to multiple index shards (fan-out) 2. Each shard returns partial results 3. Results must be merged, ranked, and paginated (fan-in)

Log Aggregation Example:

1000 servers each produce logs - Each server: 1000 log lines/second - Total: 1M log lines/second - Need central storage for querying and alerting

Without fan-in, you'd need to query each server individually or duplicate logs everywhere.

Fan-In Pattern

Summary

Key Takeaways

Many-to-One Aggregation

Multiple producers send messages to a single consumer or aggregation point. This centralizes processing and simplifies downstream logic. Common in logging, metrics collection, and data pipelines.

Correlation and Ordering

Messages from different sources may arrive out of order. Fan-in often requires correlation IDs to group related messages and ordering logic to process them correctly.

Backpressure Management

When many producers send to one consumer, the consumer can become overwhelmed. Implement backpressure mechanisms: rate limiting, buffering, or flow control to prevent overload.

Scatter-Gather Pattern

Fan-out a request to multiple services, then fan-in the responses. Used for parallel queries, distributed search, and aggregating data from multiple sources.

Completion Detection

How do you know when all expected messages have arrived? Use timeouts, expected counts, or completion markers. Handle partial results gracefully.

Single Point Bottleneck

The aggregation point can become a bottleneck. Scale by partitioning (multiple aggregators by key), or use hierarchical fan-in (aggregate locally, then globally).

Pattern Details

Distributed Search Example:

When a user searches on an e-commerce site: 1. Query goes to multiple index shards (fan-out) 2. Each shard returns partial results 3. Results must be merged, ranked, and paginated (fan-in)

Log Aggregation Example:

1000 servers each produce logs - Each server: 1000 log lines/second - Total: 1M log lines/second - Need central storage for querying and alerting

Without fan-in, you'd need to query each server individually or duplicate logs everywhere.

Fan-In Pattern

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Replication Pattern

Caching Strategies Pattern

Persistent Connections Pattern

Load Balancing Pattern

Fan-out Pattern

Fan-in Pattern

Circuit Breaker Pattern

Eventual Consistency Pattern

Queue-based Load Leveling Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern

Write-Ahead Log Pattern

API Gateway Pattern

Backend for Frontend Pattern

Sidecar Pattern

Idempotency Pattern

Rate Limiting Pattern

Backpressure Pattern

Pub/Sub Pattern

Strong Consistency Pattern

Conflict Resolution Pattern

Leader Election Pattern

Consensus Protocols Pattern

CQRS Pattern

LSM Trees Pattern

Sharding Pattern

Event Sourcing Pattern

Stream Processing Pattern

Change Data Capture Pattern

Distributed Locking Pattern

Two-Phase Commit Pattern

Fan-in Pattern

Summary

Key Takeaways

Many-to-One Aggregation

Correlation and Ordering

Backpressure Management

The Problem: Aggregating Distributed Data

Fan-In Pattern

Premium Content

Fan-in Pattern

Summary

Key Takeaways

Many-to-One Aggregation

Correlation and Ordering

Backpressure Management

The Problem: Aggregating Distributed Data

Fan-In Pattern

Premium Content