Patterns

35 items

System Design Pattern

Storagewalwrite-ahead-logdurabilityrecoveryjournalingintermediate

Write-Ahead Log Pattern

Durability through logging before committing

Used in: PostgreSQL, Kafka, LevelDB|20 min read

Summary

Write-Ahead Log (WAL) is a durability technique where all modifications are written to a sequential log before being applied to the main data structure. If the system crashes, the log is replayed to recover to a consistent state. WAL enables atomic commits (either fully logged or not at all), point-in-time recovery, and replication. Every major database uses WAL: PostgreSQL (pg_wal), MySQL (redo log), SQLite, and distributed systems like Kafka and etcd. Understanding WAL is fundamental to understanding database internals.

Key Takeaways

Log Before Apply

Write changes to sequential log first. Only after log is durable, apply to main storage. On crash, replay log to recover. This ensures no committed data is lost.

Sequential Writes Are Fast

Log is append-only, sequential writes. Much faster than random writes to B-tree pages. Batch multiple changes in single log write for efficiency.

Enables Atomic Commits

Transaction is committed when log record is durable. Either entire transaction is in log (committed) or none of it (aborted). No partial transactions.

Write-Ahead Log Flow

Log Record Contents: - Transaction ID - Operation type (INSERT, UPDATE, DELETE) - Before image (old value, for UNDO) - After image (new value, for REDO) - Table/page identifiers - Timestamp/LSN (Log Sequence Number)

Summary

Key Takeaways

Log Before Apply

Write changes to sequential log first. Only after log is durable, apply to main storage. On crash, replay log to recover. This ensures no committed data is lost.

Sequential Writes Are Fast

Log is append-only, sequential writes. Much faster than random writes to B-tree pages. Batch multiple changes in single log write for efficiency.

Enables Atomic Commits

Transaction is committed when log record is durable. Either entire transaction is in log (committed) or none of it (aborted). No partial transactions.

Foundation for Replication

Ship WAL to replicas. Replicas replay log to match primary. This is how PostgreSQL streaming replication works. Single source of truth.

Checkpointing Limits Replay

Periodically flush dirty pages and record checkpoint in log. On recovery, only replay from last checkpoint. Prevents unbounded replay time.

Group Commit for Performance

Batch multiple transaction commits into single fsync. Amortizes expensive disk flush. Trade-off between latency and throughput.

Pattern Details

Write-Ahead Log Flow

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Queue-based Load Leveling Pattern

Replication Pattern

Caching Strategies Pattern

Fan-out Pattern

Fan-in Pattern

Persistent Connections Pattern

Load Balancing Pattern

Circuit Breaker Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern