Data Lakes
Advanced 14 min read Data Lakes

Delta Lake in Depth

The transaction log underneath every Spark table

Delta Lake is the table format that put transactional semantics on Spark workloads: JSON transaction log, optimistic concurrency, MERGE/UPDATE/DELETE, time travel, and file compaction — at the cost of log growth and GC discipline.

deltatransaction logcompactvacuummergeDDL DML
Prerequisites: Lakehouse
Before we start

What you will learn

  • Describe Delta's transaction log and optimistic concurrency
  • Use MERGE/UPDATE/DELETE semantics and time travel
  • Manage log growth, compaction, and vacuum carefully
Introduction

What is this?

Delta Lake is a table format (now part of the open delta.io spec) layered on Parquet: a directory plus a JSON transaction log recording every commit. Writers add immutable actions (files, deletes, partitions); readers replay the log for a consistent snapshot. MERGE/DELETE/UPDATE and time travel drop straight onto SQL APIs.

Motivation

Why do you need it?

  • It's the de-facto default for Spark-based lakes — the format most 'lakehouse' references mean.
  • The log gives ACID-like behavior for BI/ML without a warehouse system.
  • Compaction/vacuum discipline decides whether the log stays a friend or becomes a hazard.
Think about it

Mental model

Imagine

A git repo with commits (log) and checkouts

Think of

Delta Lake

Every delta commit is like a git commit; readers 'checkout' a version. Old commits are archived by compaction (git gc) and vacuum removes unreferenced files — done wrong, log bloat or lost snapshots follow.

Mechanics

How it works

  • A _delta_log directory stores JSON commits; each commit lists added/removed Parquet files and metadata.
  • Optimistic concurrency: on each commit, writers check the log's last version (table version tip) and retry on conflict.
  • MERGE/UPDATE/DELETE rewrite files touched by the change and log new file actions atomically.
  • Time travel reveals prior versions until vacuum deletes old files.
  • OPTIMIZE (compaction) merges many small files into large ones; VACUUM removes files older than a retention (default 7 days).

Delta commit flow and log growth.

Writer
Commit
Log versions
OPTIMIZE
VACUUM
Readers
Reference

Key commands

OPTIMIZE shop.orders;

Compact small files into larger ones for faster scans.

VACUUM shop.orders RETAIN 168 HOURS;

Physically delete old files older than the retention.

DESCRIBE HISTORY shop.orders;

Inspect the transaction log — who did what, when.

Hands-on

Examples

The flagship MERGE, then history

The flagship MERGE, then history
MERGE INTO shop.orders USING shop.stage_orders s
ON shop.orders.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET amount = s.amount, updated_ts = s.updated_ts
WHEN NOT MATCHED THEN INSERT (order_id, amount, updated_ts) VALUES (s.order_id, s.amount, s.updated_ts);
 
-- Time travel:
SELECT * FROM shop.orders VERSION AS OF 12 WHERE day = '2026-03-01';
SELECT * FROM shop.orders TIMESTAMP AS OF '2026-03-14 03:59:55' WHERE day = '2026-03-01';

The transaction log makes UPSERT and versioned reads warehouse-native on plain files.

Real world

In production

  • A nightly upsert job merges supplier changes into a single Delta orders table, replacing delete+reload.
  • A ML pipeline reads a pinned historical version so training and eval read the same data.
  • A BI layer queries Delta via Trino/DuckDB; OPTIMIZE keeps scans fast after heavy merges.
Guidelines

Best practices

  • Run OPTIMIZE after heavy MERGE/upsert bursts; small files accumulate fast.
  • Vacuum only after your longest reader and time-travel window passes.
  • Set spark.databricks.delta properties (autocompact, retention) deliberately, not by default.
  • Watch the log directory size: thousand-file commits slow list/read paths.
Watch out

Common mistakes

  • Vacuuming immediately after writing, destroying time travel and breaking in-flight readers.
  • Never optimizing, so a table with 200k tiny files scans like a dir listing.
  • Depending on default retention (7 days) for audit history longer than that.
When things break

Debugging tips

  • Slow reads usually mean tiny files; describe the table's numFiles and run OPTIMIZE.
  • A reader failing on 'file ... not in the latest version' is usually vacuum or path-permission drift.
  • Commit failures during concurrency are optimistic-lock retries; log them and re-run rather than guessing.
Interviews

Interview checkpoint

Practice

Hands-on lab

Delta-Style Log, Merge, and Version Reads (Local)
40 min · advanced
In progress
Objectives
  • Emulate a delta-like transaction log on top of Parquet files
  • Perform an upsert merge that logs a new version
  • Read 'version as of' and clean up old files with a vacuum rule

Delta-Style Log, Merge, and Version Reads (Local)

40 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Delta's VACUUM physically deletes old data files after a retention window.

2

OPTIMIZE on a Delta table mainly…

Go deeper

Official documentation

Delta Lake docs
docs
delta.io protocol spec
docs
Vacuum & optimize guide
docs