Data Lakes
Advanced 15 min read Data Lakes

The Lakehouse

A warehouse on cheap files, transactions and all

A lakehouse is a lake with warehouse features: transactional ACID on open files via a table format (Delta Lake, Iceberg), so you get cheap storage, BI-grade queries, SQL, and governance without maintaining two systems.

lakehouseACIDDelta LakeIcebergopen formatstransaction log
Prerequisites: Open Formats Catalog Layers
Before we start

What you will learn

  • Explain the table-format trick that gives lakes ACID
  • Describe snapshot isolation readers get
  • Compare lakehouse vs warehouse trade-offs
Introduction

What is this?

A lakehouse layers a transactional table format (Delta Lake, Apache Iceberg, or Hudi) over object storage and open columnar files. The format keeps a log of committed versions so writers commit atomically, readers see stable snapshots, and engines like Spark/Trino/DuckDB give warehouses-grade SQL — without a separate warehouse copy.

Motivation

Why do you need it?

  • REQUIRED slug: de-foundations links here as the canonical lakehouse reference.
  • It removes the warehouse-vs-lake fork: one copy, transactional, open.
  • The transaction log is the single idea that makes 'cheap files + ACID' possible.
Think about it

Mental model

Imagine

A bookshelf with a librarian's changelog

Think of

A lakehouse table format

Books (files) sit on cheap shelves, but a changelog records who moved/added/removed what and when. Readers take a frozen snapshot of the changelog, so everyone reads a consistent shelf while writers safely shuffle books.

Mechanics

How it works

  • A table format stores old/new data as Parquet files plus a transaction log (Delta log / Iceberg metadata) in the same location.
  • A commit appends a new log entry pointing to the new version (atomic file operations on object storage).
  • Readers pin a snapshot (version) and read only files that version points to — snapshot isolation.
  • Optimistic concurrency: writers retry on conflict; like a versioned git merge.
  • Built-in helpers: time travel (read a historical version), vacuum of old files, Z-order/sort compaction.

One commit in the transaction log.

Writer
Commit
Version v2
Reader at v1
Time travel
Hands-on

Examples

Lakehouse DML looks exactly like warehouse SQL

Lakehouse DML looks exactly like warehouse SQL
-- Delta Lake (SQL renders the warehouse experience on files):
MERGE INTO analytics.orders AS t
USING stage.new_orders AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET amount = s.amount
WHEN NOT MATCHED THEN INSERT ROW;
 
-- Snapshot reads via time travel:
SELECT * FROM analytics.orders VERSION AS OF 12 WHERE day = '2026-03-14';

MERGE, snapshots, and time travel are provided by the table format's log — the warehouse layer for free.

Real world

In production

  • A team replaces a nightly copy-into-warehouse with a single Delta table: one copy, SQL on top.
  • An ML group shares the same tables as BI; model features and dashboards read one snapshot.
  • A backfill is applied as a transaction: readers stay on the old version until the new commit is verified.
Guidelines

Best practices

  • Choose a table format once and standardize; mixing Delta/Iceberg per table multiplies ops.
  • Run compaction (file count down) and vacuum only after retention windows.
  • Tune optimistic-concurrency retries for heavier writers.
  • Match BI tools to the format: Trino/DuckDB read Iceberg and Delta natively.
Watch out

Common mistakes

  • Using multiple formats per table and breaking cross-engine consistency.
  • Vacuuming too early and destroying time travel or in-flight readers.
  • Forgetting writer coordination: grabbing full-table locks where incremental commit would be safer.
When things break

Debugging tips

  • A reader seeing stale data usually means the snapshot pointer has not advanced; check the commit log tail.
  • A 'file not found' during time travel points at vacuum garbage collection removing files still referenced.
  • MERGE suddenly slow = files fragmented; check file count per partition and run compaction.
Interviews

Interview checkpoint

Practice

Hands-on lab

ACID on Files With a Table Format (DuckDB + Delta/Iceberg-like)
35 min · advanced
In progress
Objectives
  • Show the transaction-log pattern: commit = atomic pointer flip
  • Demonstrate snapshot-stability while a writer commits
  • Do a MERGE-style upsert and read VERSION-style pinnable state

ACID on Files With a Table Format (DuckDB + Delta/Iceberg-like)

35 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A lakehouse table format gives snapshot isolation so readers see a consistent version while writers commit.

2

Delta Lake, Apache Iceberg, and Apache Hudi are best described as…

Go deeper

Official documentation

Databricks lakehouse
docs
Apache Iceberg
docs
Delta Lake docs
docs