Data Lakes
Advanced 14 min read Data Lakes

Apache Iceberg in Depth

The table format engineered for lazy engines

Iceberg's metadata tree — manifest list, manifests, data files with column-level metadata — delivers file-level ACID, partition evolution, hidden partitioning, and painless RENAME/drop with table snapshots that any engine can read.

Icebergmanifestsnapshotpartition evolutionhidden partitioningmetadata tree
Prerequisites: Lakehouse
Before we start

What you will learn

  • Explain Iceberg's three-level metadata structure
  • Describe hidden partitioning and its benefit
  • Understand snapshot/ACID and why it's engine-agnostic
Introduction

What is this?

Apache Iceberg organizes a table into snapshots described by a metadata file, a manifest list, and manifests pointing to data files with per-column stats. This tree lets any engine read consistent snapshots, evolve partitions without rewrites, and let queries prune with stats — opening boatloads of SQL workbench portability.

Motivation

Why do you need it?

  • Its clean separation of metadata from data made 'many engines read one table' practical.
  • Hidden partitioning means writers don't maintain partition folders by hand; the table does.
  • Partition evolution is the feature that makes Iceberg the default for teams tired of re-sorting tables.
Think about it

Mental model

Imagine

A recipe card folder with a master index per month

Think of

Iceberg metadata tree

The master index (manifest list) lists monthly card packs (manifests); each card names the actual recipe file (data file) with useful margins (stats). Reading 'January chicken recipes' only touches matching packs.

Mechanics

How it works

  • A table's current metadata file (JSON) points at a manifest list for its latest snapshot.
  • A manifest list references manifests, one per file group; each manifest lists data files plus their stats.
  • Stats (value counts, lower/upper bounds, nulls) let engines prune files at scan time.
  • Snapshot = consistent set of files; commits append new metadata atomically, enabling RENAME/drop 'tables' as metadata moves.
  • Partitioning is tracked in metadata (hidden), so changing the partition spec rewrites only metadata, not data.

Iceberg's metadata tree for one snapshot.

Metadata file
Manifest list
Manifests
Data files
Readers
Reference

Key commands

SELECT * FROM orders$snapshots ORDER BY committed_at DESC;

See the table's snapshots (audit trail).

SELECT * FROM orders FOR TIMESTAMP AS OF '2026-03-14 03:00';

Read a historical snapshot via time travel.

ALTER TABLE orders ALTER COLUMN amount TYPE DECIMAL(14,2);

No-rewrite schema evolution, courtesy of metadata-driven reads.

Hands-on

Examples

Hidden partitioning makes evolution invisible

Hidden partitioning makes evolution invisible
-- Old style: partition folders manual, changing = rewriting data
s3://lake/orders/dt=2026-03-14/...
 
-- Iceberg style (hidden):
CREATE TABLE lake.orders (id BIGINT, ts TIMESTAMP, amount DECIMAL(12,2))
USING iceberg PARTITIONED BY (days(ts));
 
-- Change partition layout later WITHOUT rewriting data:
ALTER TABLE lake.orders PARTITION BY (months(ts));
-- Readers keep working; only metadata changed.

The partition spec lives in the table, not in the path — engines prune by table metadata, not by looking at folder names.

Real world

In production

  • A multi-engine shop publishes Iceberg tables and lets Spark, Trino, and DuckDB all read the same snapshots.
  • A team re-partitions by month after growing a year; no rewrite, no downtime.
  • A CDC pipeline lands hourly commits; readers pin the last-committed snapshot for consistent extracts.
Guidelines

Best practices

  • Let the statistics be your filter: write sorted by predicate columns so lower/upper bounds prune.
  • Set snapshot expiration and orphan-file cleanup in a job, not by guesswork.
  • Keep partitioning coarse (month) and rely on stats for finer pruning.
  • Use a catalog that enforces the atomic metadata updates required for ACID.
Watch out

Common mistakes

  • Manually naming partition folders under the table, fighting hidden partitioning.
  • Not expiring snapshots: one 1M-file table grows 30 manifest lists a day.
  • Assuming 'table dropped' deletes data files — delete/expire explicitly for real GC.
When things break

Debugging tips

  • A slow Iceberg read is usually stats that can't prune (unsorted/or-and), not the format.
  • Missing files after rollback mean orphan-cleanup ran while an older snapshot was referenced.
  • Two catalogs disagreeing often trace to cached metadata files; refresh the table's metadata entry.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Iceberg keeps partitioning metadata in the table definition so writers don't hand-maintain partition folders.

2

Which artifact points directly to the data files of a snapshot in Iceberg?

Go deeper

Official documentation

Apache Iceberg spec
docs
Iceberg partitioning
docs
Iceberg + Trino quickstart
docs