Data-Driven Architecture
The store you pick decides the questions you can ask
Data shapes the architecture: choose the right store (relational vs NoSQL vs analytics) by access patterns, design read/query scaling with materialized views + caching, and keep the data tier honest with backups, replication, and lifecycle. Data topology = architecture majority.
What you will learn
- Match store type to access pattern (SQL/NoSQL/analytics)
- Design query scalability: caches, materialized views, partitioning
- Keep the data tier reliable (backup/replication/lifecycle)
Choose stores by access pattern: relational (joins/transactions/relations), NoSQL (schemaless fast key lookups, scale-out writes), analytics/OLAP (columnar, big scans). Scale reads via caching + materialized views + read replicas; scale writes via partitioning + sharding. Backups/replication decide RPO — the data line decides the nines.
What is this?
Stores: relational (Postgres/MySQL/Aurora-managed; ACID, joins, migrations), NoSQL (document: DynamoDB/CosmosDB/MongoDB — key-access, flexible schema, scale-out; wide-column/vector; etc.), analytics (Redshift/BigQuery/Synapse — columnar, huge scans, BI). Query-pattern mapping: if you need flexible queries over relationsuseful → SQL; predictable single-key/aggregation lookups at scale → NoSQL; heavy analytics scans → warehouse. Read scaling: caching (Redis/DAX) for hot keys, read replicas for latency/fresh reads, materialized views/rollups for dashboards, partitioning (by time/tenant) for hot-spot isolation. Write scaling: partitioning (table/db by key range), sharding (by tenant/hash), event-sourcing for high-write replay. Data hygiene: backups with PITR + cross-region, replication lag dashboarded, lifecycle (hot→cold→archive), schema migration discipline (backward compatible). Data topology drives DR (see that module) and cost (see cost module) — 'the data line is the architecture' is the credo.
Why do you need it?
- Wrong store = query pain that grows: joins on NoSQL, hotspots on single-table SQL at scale.
- Read-heavy architectures get 90% of their win from a right-sized cache + mviews, not fancy compute.
- Write scaling (partitioning/sharding) done up-front beats the 'one hot table' incident at 9x growth.
- RPO/lifecycle/backup discipline is where 'reliable' data actually lives.
Mental model
Choosing furniture by what rooms you host: a library (relations) vs. huge closet (fast retrieval) vs. warehouse (analytics)
Store choice
You wouldn't store the library in a closet nor the sweaters in the reading room. Match the store (furniture) to the access patterns (rooms) — the building (schema) follows.
How it works
- Enumerate access patterns: read-heavy? joins? predictable key lookups? reports?
- Map store per pattern: relational / NoSQL / warehouse (or a small, principled mix).
- Design read scaling: cache hot keys + stock mviews + replicas per SLA.
- Design write scaling: partitioning/sharding on the growth axis (time/tenant).
- Harden the data tier: PITR + replication + lifecycle + migration discipline.
Data-driven shape.
Examples
Store-by-pattern
# order service:# - relational (Postgres): orders/lineitems/transactions (joins, ACID)# - cache (Redis): hot order status reads (reads scale, TTL'd)# - analytics (warehouse): sales dashboards queried on columnar store# Read path: API → cache → (miss) DB; mview for weekly rollup.# Write path: partition by order-date range; shard by tenant if single-table hot.Three stores, three pattern matches, one healthy boundary (cache early, DB for truth, warehouse for analytics).
In production
- Read-heavy dashboards over materialized weekly rollups instead of live scans.
- NoSQL for key-access at scale (session, event, lookup) + relational for relations.
- Partition by month + shard by tenant with a documented growth trigger for the hot table.
Best practices
- Start from access patterns, not store brands; a store is a tool for a query shape.
- Cache with TTLs + version keys; never let the cache become a second DB that must be invalidated.
- Partition/shard on the actual growth axis (time/tenant); keep the trigger documented.
- Automate migrations (backward-compatible schema) into CI/deploy order.
- Backups/PITR + lag dashboards are non-negotiable for any store above 'toy'.
Common mistakes
- NoSQL for a join-y access pattern → developer 'joins in app' and the cost multiplies.
- Caching without invalidation → stale data that 'mostly works', then bites the read-after-write.
- Analytics queries hammering the OLTP store → dashboard latency + prod slowdown.
- Skipping migrations tooling → schema drift then a painful live alter.
Debugging tips
- Slow dashboard → reroute to mviews/warehouse; if still slow, mview refresh cadence is the fix, not the query.
- Cache drifts from the DB → the invalidate-on-write path is broken; version keys + TTL pass-through.
- One hot tenant kills the write path → tenant partition/shard; the growth trigger fired and wasn't acted on.
- RPO surprise at restore → the store's PITR was never verified; test restores per quarter (DR drills).
Interview checkpoint
Hands-on lab
- Map patterns to store shapes
- Create the cache + mview pass-through locally
- Screen the growth triggers (partition/shard)
Model a Data Tier by Access Pattern
35 min · advanced
Quick quiz
The fastest way to read-hot scale without new infra:
A hallmark that your cache has become a second DB: