Patterns

35 items

System Design Pattern

Data Processingstream-processingreal-timekafka-streamsflinkwindowingadvanced

Stream Processing Pattern

Real-time processing of continuous data streams

Used in: Kafka Streams, Flink, Spark Streaming|25 min read

Summary

Stream processing handles continuous, unbounded data flows in real-time, processing events as they arrive rather than in batches. Systems like Apache Kafka Streams, Apache Flink, and Spark Streaming enable sub-second processing of millions of events. Key concepts include windowing (grouping events by time), watermarks (handling late data), and exactly-once semantics. This pattern powers real-time analytics, fraud detection, live dashboards, and event-driven architectures where immediate processing is essential.

Key Takeaways

Unbounded, Continuous Data

Unlike batch, stream data has no end. Events arrive continuously. System must process indefinitely without accumulating unbounded state.

Event-Time vs Processing-Time

Events may arrive out of order or late. Event-time (when event occurred) differs from processing-time (when system sees it). Correct handling requires watermarks.

Windowing Groups Events

Windows group unbounded streams into finite chunks. Tumbling (fixed, non-overlapping), sliding (overlapping), session (activity-based).

Windowing Types

Summary

Key Takeaways

Unbounded, Continuous Data

Unlike batch, stream data has no end. Events arrive continuously. System must process indefinitely without accumulating unbounded state.

Event-Time vs Processing-Time

Events may arrive out of order or late. Event-time (when event occurred) differs from processing-time (when system sees it). Correct handling requires watermarks.

Windowing Groups Events

Windows group unbounded streams into finite chunks. Tumbling (fixed, non-overlapping), sliding (overlapping), session (activity-based).

Exactly-Once is Hard

Failures happen. Achieving exactly-once processing requires idempotent operations, transactional writes, and careful checkpoint management.

Backpressure Prevents Overload

When processing can't keep up with input rate, backpressure signals upstream to slow down. Prevents memory exhaustion and cascading failures.

State Management

Aggregations, joins, and windows require state. State must be checkpointed for fault tolerance. State size limits scalability.

Pattern Details

Windowing Types

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Replication Pattern

Caching Strategies Pattern

Persistent Connections Pattern

Load Balancing Pattern

Fan-out Pattern

Fan-in Pattern

Circuit Breaker Pattern

Eventual Consistency Pattern

Queue-based Load Leveling Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern

Write-Ahead Log Pattern

API Gateway Pattern

Backend for Frontend Pattern

Sidecar Pattern

Idempotency Pattern

Rate Limiting Pattern

Backpressure Pattern

Pub/Sub Pattern

Strong Consistency Pattern

Conflict Resolution Pattern

Leader Election Pattern

Consensus Protocols Pattern

CQRS Pattern

LSM Trees Pattern

Sharding Pattern

Event Sourcing Pattern