ddia-systems

---
name: ddia-systems
description: Use when choosing a database, weighing SQL vs NoSQL, diagnosing replication lag or a partitioning strategy, reasoning about consistency vs availability, ACID transactions, or stream processing.
category: Engineering Knowledge
version: 1.0.0
tools: []
---

# Designing Data-Intensive Applications Framework

A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.

## Core Principle

**Data outlives code.** Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.

## Scoring

**Goal: 10/10.** Score a data architecture by the Quick Diagnostic checklist below: award points per row answered "yes" with evidence (deliberate, documented trade-off), 0 where the answer is "no" or unknown.

- **9-10:** every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read/write/consistency requirements; failover tested.
- **5-6:** core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.
- **<=3:** choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.

Report the current score, which diagnostic rows failed, and the improvements needed to reach 10/10.

## The DDIA Framework

Seven domains for reasoning about data-intensive systems:

### 1. Data Models and Query Languages

**Core concept:** The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.

**Why it works:** Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.

**Key insights:**
- Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
- Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
- Polyglot persistence -- different stores for different access patterns -- is often the right answer

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **User profiles with nested data** | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |
| **Social network connections** | Graph model for relationship traversal | Neo4j Cypher: `MATCH (a)-[:FOLLOWS*2]->(b)` for friend-of-friend |

### 2. Storage Engines

**Core concept:** Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.

**Key insights:**
- LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
- B-trees: in-place updates, predictable read latency, write amplification from page splits
- Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **High write throughput** | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes/sec |
| **Mixed read/write OLTP** | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |

### 3. Replication

**Core concept:** Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.

**Why it works:** Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.

**Key insights:**
- Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure
- Multi-leader: better write availability across data centers, but complex conflict resolution
- Leaderless: highest availability via quorum reads/writes, but needs careful conflict handling

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Read-heavy web app** | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer |
| **Multi-region writes** | Multi-leader replication | CockroachDB or Spanner with bounded staleness |

### 4. Partitioning

**Core concept:** Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.

**Key insights:**
- Key-range partitioning supports efficient range scans but risks hotspots on sequential keys
- Hash partitioning distributes load evenly but destroys sort order, making range queries expensive
- Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Time-series data** | Key-range partitioning by time + source | Partition by `(sensor_id, date)` to avoid current-day write hotspot |
| **User data at scale** | Hash partitioning on user ID | Cassandra consistent hashing on `user_id` for even distribution |

### 5. Transactions and Consistency

**Core concept:** Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.

**Why it works:** Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.

**Key insights:**
- Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable
- Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits
- Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Account balance transfer** | Serializable transaction | `BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT;` |
| **Inventory reservation** | SELECT FOR UPDATE to prevent write skew | `SELECT stock FROM items WHERE id = X FOR UPDATE` before decrementing |

### 6. Batch and Stream Processing

**Core concept:** Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.

**Why it works:** Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.

**Key insights:**
- MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs
- Change data capture (CDC) turns database writes into a stream downstream systems can consume
- Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Daily analytics pipeline** | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse |
| **Real-time fraud detection** | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows |

### 7. Reliability and Fault Tolerance

**Core concept:** Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.

**Key insights:**
- A fault is one component deviating from spec; a failure is the whole system stopping -- fault tolerance prevents the former becoming the latter
- Hardware faults are random and independent; software faults are correlated and systematic (more dangerous)
- Human error is the leading cause of outages -- minimize opportunity for mistakes, maximize ability to recover

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Service communication** | Timeouts + retries with backoff | `retry(max=3, backoff=exponential(base=1s, max=30s))` with jitter |
| **Leader election** | Consensus algorithm (Raft/Paxos) | etcd or ZooKeeper for distributed locks and leader election |

## Common Mistakes

| Mistake | Why It Fails | Fix |
|---------|-------------|------|
| **Choosing a database by popularity** | Engines have fundamentally different trade-offs | Match storage engine to actual read/write patterns |
| **Ignoring replication lag** | Stale reads, phantom reads, lost updates | Implement read-your-writes and monotonic-read guarantees |
| **Distributed transactions everywhere** | 2PC is slow, fragile; coordinator is a SPOF | Design single-partition operations; use sagas across services |
| **Hash partitioning everything** | Destroys range query ability | Key-range partitioning for time-series; composite keys for locality |