Patterns

35 items

System Design Pattern

Data Processingcdcchange-data-capturedebeziumlog-basedevent-streamingadvanced

Change Data Capture Pattern

Capturing database changes as events

Used in: Debezium, Database Replication, Event Streaming|25 min read

Summary

Change Data Capture (CDC) tracks and captures changes made to database records, publishing them as events for downstream consumption. Instead of polling databases or dual-writing, CDC reads the database's transaction log to capture every INSERT, UPDATE, and DELETE. This enables real-time data synchronization, event-driven architectures, cache invalidation, and building materialized views. Tools like Debezium, AWS DMS, and database-native features make CDC accessible. CDC is essential for keeping data consistent across systems without tight coupling.

Key Takeaways

Log-Based CDC is Authoritative

Reading transaction logs (WAL, binlog) captures all changes exactly as they happened. No missed updates. No race conditions. Single source of truth.

Eliminates Dual-Write Problem

Writing to database AND cache/queue creates inconsistency on failures. CDC writes to database only; changes flow automatically to other systems.

Enables Event-Driven Architecture

Database changes become events. Other services react to changes without polling. Loose coupling between services.

Change Data Capture Flow

Why not dual-write?

python
# DANGEROUS: Dual-write pattern
def update_user(user_id, data):
    database.update(user_id, data)  # Step 1
    cache.invalidate(user_id)       # Step 2 - fails?
    kafka.publish(user_changed)     # Step 3 - fails?
    # If step 2 or 3 fails, systems are inconsistent!

CDC solves this: Write to database only. CDC captures change and publishes to all consumers.

Summary

Key Takeaways

Log-Based CDC is Authoritative

Reading transaction logs (WAL, binlog) captures all changes exactly as they happened. No missed updates. No race conditions. Single source of truth.

Eliminates Dual-Write Problem

Writing to database AND cache/queue creates inconsistency on failures. CDC writes to database only; changes flow automatically to other systems.

Enables Event-Driven Architecture

Database changes become events. Other services react to changes without polling. Loose coupling between services.

Real-time Data Synchronization

Keep data warehouse, search index, cache in sync with source database. Changes propagate in seconds, not hours.

Preserves Change History

CDC captures before and after values, timestamps, and operation type. Enables audit trails, temporal queries, and debugging.

Handles Deletes Properly

Polling misses deletes (deleted rows disappear). CDC captures DELETE events explicitly. Complete change stream.

Pattern Details

Change Data Capture Flow

Why not dual-write?

python
# DANGEROUS: Dual-write pattern
def update_user(user_id, data):
    database.update(user_id, data)  # Step 1
    cache.invalidate(user_id)       # Step 2 - fails?
    kafka.publish(user_changed)     # Step 3 - fails?
    # If step 2 or 3 fails, systems are inconsistent!

CDC solves this: Write to database only. CDC captures change and publishes to all consumers.

Trade-offs

Aspect	Advantage	Disadvantage

Patterns

Horizontal Scaling Pattern

Retry with Backoff Pattern

Queue-based Load Leveling Pattern

Replication Pattern

Caching Strategies Pattern

Fan-out Pattern

Fan-in Pattern

Persistent Connections Pattern

Load Balancing Pattern

Circuit Breaker Pattern

Bloom Filters Pattern

Time-Series Storage Pattern

Bulkhead Pattern

Batch Processing Pattern

Write-Ahead Log Pattern

API Gateway Pattern

Backend for Frontend Pattern

Sidecar Pattern

Idempotency Pattern

Rate Limiting Pattern

Backpressure Pattern

Pub/Sub Pattern

Eventual Consistency Pattern

Sharding Pattern

Conflict Resolution Pattern

Strong Consistency Pattern

Leader Election Pattern

Consensus Protocols Pattern

Stream Processing Pattern