Data Engineering Foundations
Beginner 14 min read Data Engineering Foundations

SQL vs NoSQL

Relational integrity vs operational scale and flexibility

SQL databases give you relational power, integrity and analysis; NoSQL databases give you flexible schemas, scale and specific access patterns. The right choice is workload, not fashion.

SQLNoSQLdocument storeskey-valuewide-columngraphCAP
Before we start

What you will learn

  • Map the four NoSQL families to their canonical workloads
  • Explain when rigid schemas protect versus hurt
  • Choose SQL vs NoSQL by query patterns and consistency needs
Introduction

What is this?

Relational (SQL) databases store data in normalized tables with enforced schema and ACID transactions, and query with a rich SQL language. NoSQL databases trade some of that structure for flexibility and scale, split into families: document, key-value, wide-column and graph. Data engineers meet both constantly — often in the same pipeline.

Motivation

Why do you need it?

  • SQL's strong integrity prevents corrupt data at the source; some systems must have it.
  • NoSQL's flexible schemas absorb real-world shape shifts (event payloads, user-generated data).
  • Scale story differs: NoSQL horizontal partitioning was designed for global scale; many SQL systems now scale well too.
  • Knowing both lets you pick the storage that matches the access pattern instead of forcing one engine on everything.
Think about it

Mental model

Imagine

A spreadsheet with strict columns vs index cards

Think of

SQL vs document store

A spreadsheet forces every row into the same columns and protects against typos — great for accounting. Index cards let each card say whatever it needs — flexible, but nothing stops a misspelled field finding its way in.

Mechanics

How it works

  • SQL: schema in DDL; tables + relations normalized; joins at query time; ACID via row locks/WAL.
  • Document stores (Mongo, DynamoDB-as-doc, Couch): one JSON document per record; embed related data; query by field, secondary indexes, aggregates.
  • Key-value (Redis, DynamoDB pure KV, Memcache): get/set by key with blazing latency; no queries by arbitrary fields.
  • Wide-column (Cassandra, HBase, Bigtable): sparse tables by (partition key, sort key); optimized for time-series and high write throughput.
  • Graph (Neo4j, Neptune): nodes + edges; path and relationship queries ('who friends whom').

Choosing a storage family by access pattern.

Query by many keys / joins / integrity
Whole records, flexible shape
Lookup by one key, ultra-low latency
Time-series / heavy writes by key
Relationship interrogation
Real world

In production

  • Transactional source-of-record systems (ledgers, orders, users) typically live in Postgres/MySQL (SQL), and analytics reads from a warehouse.
  • A session/token cache, leaderboard or queue uses Redis (key-value) for microsecond latency.
  • Clickstream and IoT time-series land in Cassandra (wide-column) for massive write throughput.
  • Fraud/relationship detection queries a graph store over connect-the-dots networks (Neo4j).
Guidelines

Best practices

  • Start relational unless you have a concrete reason (scale, flex-schema, latency) not to.
  • Model NoSQL documents around the read path: embed what you read together, reference what changes independently.
  • Design primary keys for your hot queries; NoSQL punishes scanning.
  • Confirm the consistency semantics you need — NoSQL is often eventually consistent.
Watch out

Common mistakes

  • Hash-parking everything into MongoDB and then writing cross-document joins the engine can't do.
  • Choosing 'new tech' instead of the right fit and paying the schema-less tax (duplicated, drift-prone data).
  • Ignoring that your NoSQL 'JSON column' still needs indexing for query speed.
  • Storing denormalized data everywhere then discovering you need transactional updates across copies.
Root cause analysis

Common issues

Error
Every dashboard query on a NoSQL dataset scans the whole collection
Root cause

Secondary index missing or queries don't filter on partition key

Fix

Reshape the key (or add indexes) so hot queries hit partitions by filter first.

Error
Same customer shows up as two documents
Root cause

No enforced unique natural key; app race or case-mismatch

Fix

Dedupe by a stable customer key on write and backfill the mismatch set.

Error
SQL database locks under analytical load
Root cause

Analytics ran against the OLTP store, blocking writes

Fix

Add a read replica or move analytical queries to the warehouse.

When things break

Debugging tips

  • Slow NoSQL query? Check the query pattern against the primary key/partition design first.
  • Duplicate records across documents usually mean missing natural-key design or a race in client writes.
  • If a 'schema-less' store keeps producing type errors, enforce the schema in the application/dto layer.
  • Count 'full collection scans' — an index or a better key fixes the pain.
Interviews

Interview checkpoint

Practice

Hands-on lab

Round-trip SQL into NoSQL Docs
35 min · intermediate
In progress
Objectives
  • Model a relational order dataset as documents for an order-lookup service
  • See the trade-off: simpler reads, denormalized redundancy
  • Reconcile the no-SQL projection back to SQL counts

Round-trip SQL into NoSQL Docs

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A service needs sub-millisecond get-by-ordernumber lookups of order objects. Best fit?

2

NoSQL databases always sacrifice querying flexibility for operational scale.

Go deeper

Official documentation