Data Lakes
Advanced 15 min read Data Lakes

Parquet Deep Dive

The bytes behind a five-star format

Under Parquet's hood: Dremel's definition/repetition levels for nested data, dictionary + RLE + bit-packing encodings, row-group/column-chunk/page layering, and the metadata footer that makes splits possible without reading the file body.

row groupcolumn chunkpagedictionary encodingRLErepetition levels
Prerequisites: Open Formats
Before we start

What you will learn

  • Trace the physical hierarchy row group → column chunk → page
  • Explain dictionary encoding and when it dominates
  • Describe how nested data is flattened with definition/repetition levels
Introduction

What is this?

Parquet is a Thrift-defined physical format: data splits into row groups; each column of a row group is a column chunk; each chunk is a sequence of pages. Pages hold data encoded with dictionary/RLE/bit-packing/delta encodings, plus statistics. At the tail, a footer records schema, row-group pointers, and metadata enabling split reads.

Motivation

Why do you need it?

  • Understanding page encodings explains why some columns compress 100x and others can't.
  • Row-group chunking is the unit that lets engines read different row groups in parallel.
  • The footer is why 'reading one column' costs a tiny header seek, not a full file scan.
Think about it

Mental model

Imagine

A layered magazine: issues (issues), sections per issue, paragraphs with margins

Think of

Parquet physical layout

To read faster, you open the magazine's table of contents (footer), then just the paragraphs you need; pages are compressed so the same text fits. The structure decides how much you must open.

Mechanics

How it works

  • Row groups: a row-group is a horizontal slice of rows (typically 128MB); one unit of split/parallelism.
  • Column chunk: the same column's values within a row group, contiguous on disk.
  • Pages: units of compression (often ~1MB); each page carries its own stats, compressed independently.
  • Dictionary encoding: maps distinct value → short code; for low-cardinality columns it's nearly lossless compression.
  • Nested data: definition levels (how deep a value is) and repetition levels (where lists repeat) flatten trees into columns.
  • The ending footer holds Thrift metadata: schema tree, row-group offsets, encodings, compressors, and page stats.

Parquet physical layering from file to page.

Parquet file
Row groups
Column chunks
Pages
Footer metadata
Reference

Key commands

python3 -c "import pyarrow.parquet as pq, sys; cols=pq.ParquetFile('f.parquet').metadata.row_group(0).column(0).statistics; print(cols)"

Read one column chunk's min/max/null stats via pyarrow.

duckdb -c "SELECT size FROM parquet_metadata('f.parquet')"

List column-chunk byte sizes per row group.

python3 -c "import pyarrow.parquet as pq; print(pq.read_metadata('f.parquet').serialized_size)"

Footer size — a hint of how rich the metadata is.

Hands-on

Examples

Dictionary encoding explained by size

Dictionary encoding explained by size
-- Cardinality is destiny:
-- column 'region' with 5 distinct values: dictionary codes ~3 bits each -> tiny
-- column 'uuid'   with ~all distinct:      dictionary no help; delta/plain used
 
# Generic win: after dictionary encoding, RLE compresses long runs of the SAME code
# (e.g. country in ts-sorted data) to near 1 bit/value.

Encoding choice is data-driven at write time; engines emit either a dictionary or plain/delta page per compression pass.

Real world

In production

  • An optimizer profiles pages: a huge dictionary-sized chunk tells the writer to disable dictionary or compress more.
  • A split reader assigns each row group to a Spark task, achieving parallelism without reading the whole file.
  • A 'kill wide columns' cleanup finds giant column chunks nobody selects and removes them from the schema.
Guidelines

Best practices

  • Keep row groups large enough (>=64MB-128MB) for parallel, smaller-than-table reads.
  • Prefer zstd; disable dictionary encoding for genuinely high-cardinality columns (it costs a pass).
  • Match page-size to the engine's IO: 1MB pages balance compression and random access.
  • Let statistics stay on; they are the cheapest pruning superpower in the file.
Watch out

Common mistakes

  • Setting an enormous/fine page size that makes every tiny read decompress a huge page.
  • Forgetting nested encoding: deeply nested JSON writes blow up pages unless structs/lists are modeled.
  • Reading the whole file to get a few columns — the footer+projection was designed exactly for that.
When things break

Debugging tips

  • When a column reads slowly, check if it's dictionary/RLE (fast) vs plain (slow) per page.
  • A file whose footer is enormous is crammed with per-page stats; budget it or disable statistics on noisy columns.
  • If 'small' parquet scans are slow, suspect micro-row-groups (fragmentation) instead of encoding.
Interviews

Interview checkpoint

Practice

Hands-on lab

Layering, Stats, and Encoding Under a Microscope
35 min · advanced
In progress
Objectives
  • Write low- vs high-cardinality columns and compare file sizes
  • Inspect row-group and column-chunk metadata with DuckDB
  • Demonstrate that dictionary-free high-cardinality data swells files

Layering, Stats, and Encoding Under a Microscope

35 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The unit of table-level parallelism when reading one Parquet file is the…

2

Parquet stores the file's schema and row-group offsets in a footer at the end of the file.

Go deeper

Official documentation

Apache Parquet format
docs
Parquet encodings deep dive
docs
Parquet page index spec
docs