System Design Fundamentals

11 items

Fundamentalscap-theoremconsistencyavailabilitypartition-tolerancedistributed-systemsfundamentalssystem-designintermediate

CAP Theorem & Consistency Models

Understanding the fundamental trade-offs in distributed data systems

Foundation knowledge|40 min read

Summary

The CAP theorem states that a distributed system can only guarantee two of three properties: Consistency (all nodes see the same data), Availability (every request gets a response), and Partition tolerance (system works despite network failures). Since network partitions are inevitable, the real choice is between CP (consistent but may be unavailable) and AP (available but may be inconsistent). Beyond CAP, understanding consistency models—strong, eventual, causal—helps you choose the right guarantees for each use case.

Key Takeaways

CAP is About Partitions, Not Normal Operation

CAP only forces a choice during network partitions. In normal operation, you can have all three. The question is: when a partition occurs, do you sacrifice consistency (serve potentially stale data) or availability (reject requests)?

Partition Tolerance is Not Optional

Network partitions will happen—switches fail, cables get cut, cloud regions disconnect. A system that doesn't tolerate partitions isn't distributed. So the real choice is CP or AP, not "pick any two."

Consistency is a Spectrum, Not Binary

Between "immediately consistent everywhere" and "eventually consistent someday" lies a spectrum: linearizable, sequential, causal, read-your-writes, eventual. Most applications need something in the middle, not the extremes.

The CAP theorem (Brewer's theorem) states that a distributed data store can only provide two of these three guarantees:

Consistency (C): Every read receives the most recent write or an error. All nodes see the same data at the same time.

Availability (A): Every request receives a non-error response, without guarantee that it contains the most recent write.

Partition Tolerance (P): The system continues to operate despite network partitions (communication failures between nodes).

CAP Theorem Visualization

The key insight: Network partitions are not optional in distributed systems. Networks fail, packets get lost, data centers disconnect. Any real distributed system must be partition tolerant.

Therefore, the real choice is: When a partition occurs, do you choose: - Consistency (CP): Reject requests to maintain data correctness - Availability (AP): Serve requests but possibly with stale data

CA systems (Consistent + Available, not Partition tolerant) can only exist as single-node systems—traditional RDBMS like PostgreSQL or MySQL before replication.

Summary

Key Takeaways

CAP is About Partitions, Not Normal Operation

Partition Tolerance is Not Optional

Consistency is a Spectrum, Not Binary

PACELC Extends CAP to Normal Operation

PACELC says: if Partitioned, choose A or C; Else (normal operation), choose Latency or Consistency. This captures the latency-consistency trade-off that CAP ignores during non-partition periods.

Strong Consistency Has a Latency Cost

To guarantee all nodes agree before responding, you must wait for network round trips. This adds latency proportional to network distance. Cross-region strong consistency means 100ms+ added to every write.

Eventual Consistency Requires Application Logic

"Eventually consistent" means conflicts can occur. Your application must handle reading stale data, resolving conflicts, and presenting sensible results during inconsistency windows. It's not free.

Deep Dive

The CAP theorem (Brewer's theorem) states that a distributed data store can only provide two of these three guarantees:

Consistency (C): Every read receives the most recent write or an error. All nodes see the same data at the same time.

Availability (A): Every request receives a non-error response, without guarantee that it contains the most recent write.

Partition Tolerance (P): The system continues to operate despite network partitions (communication failures between nodes).

CAP Theorem Visualization

The key insight: Network partitions are not optional in distributed systems. Networks fail, packets get lost, data centers disconnect. Any real distributed system must be partition tolerant.

CA systems (Consistent + Available, not Partition tolerant) can only exist as single-node systems—traditional RDBMS like PostgreSQL or MySQL before replication.

Trade-offs

Aspect	Advantage	Disadvantage
Strong Consistency (CP)	No stale reads, no conflicts, simple application logic, behaves like single node	Unavailable during partitions, higher latency (wait for quorum), lower throughput
Eventual Consistency (AP)	Always available, low latency (local reads/writes), high throughput, geographic distribution	Stale reads, conflicts require resolution, complex application logic, harder to reason about
Causal Consistency	Preserves intuitive ordering, no global coordination, can be highly available	More complex than eventual, requires dependency tracking, not as strong as linearizable
Tunable Consistency	Flexibility per operation, optimize for specific use cases, best of both worlds possible	Complexity in choosing right level, easy to misconfigure, harder to reason about overall behavior

System Design Fundamentals

Scalability Fundamentals

Latency, Throughput & Performance

Back-of-Envelope Calculations

Availability & Reliability Fundamentals