Patterns

35 items

System Design Pattern

Data Processingbatchmapreduceetlbulk-processinghadoopintermediate

Batch Processing Pattern

Processing large volumes of data in batches

Used in: Hadoop, Spark, ETL Pipelines|20 min read

Summary

Batch processing executes operations on large datasets in scheduled, bounded jobs rather than real-time. It's optimized for throughput over latency - process millions of records efficiently by grouping operations, optimizing I/O, and parallelizing work. MapReduce pioneered distributed batch processing; modern systems like Apache Spark, Hadoop, and AWS Batch handle petabyte-scale workloads. This pattern is essential for ETL pipelines, data analytics, report generation, and any workload where processing can be deferred.

Key Takeaways

Throughput Over Latency

Batch processing optimizes for total records processed, not individual response time. Processing 1B records in 1 hour is acceptable; 1ms per record is not required.

Bounded and Scheduled

Jobs have defined start and end. Run on schedule (nightly, hourly) or trigger. Unlike streaming, input is known and finite.

Parallelization is Key

Split data into partitions, process in parallel, merge results. Scales horizontally. MapReduce paradigm: map (transform) → shuffle (group) → reduce (aggregate).

MapReduce Pattern

Summary

Key Takeaways

Throughput Over Latency

Batch processing optimizes for total records processed, not individual response time. Processing 1B records in 1 hour is acceptable; 1ms per record is not required.

Bounded and Scheduled

Jobs have defined start and end. Run on schedule (nightly, hourly) or trigger. Unlike streaming, input is known and finite.

Parallelization is Key

Split data into partitions, process in parallel, merge results. Scales horizontally. MapReduce paradigm: map (transform) → shuffle (group) → reduce (aggregate).

Checkpointing Enables Recovery

Save progress periodically. On failure, restart from last checkpoint, not beginning. Critical for long-running jobs.

Idempotency Required

Jobs may be retried on failure. Same input must produce same output. Use write-audit-publish pattern for exactly-once semantics.

Resource Efficiency

Run during off-peak hours. Use spot/preemptible instances for cost savings. Scale up for job, scale down after.

Pattern Details

MapReduce Pattern

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Replication Pattern

Caching Strategies Pattern

Persistent Connections Pattern

Load Balancing Pattern

Fan-out Pattern

Fan-in Pattern

Circuit Breaker Pattern

Eventual Consistency Pattern

Queue-based Load Leveling Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern