Data Lakes
Intermediate 12 min read Data Lakes

Catalog Layers and Table Management

The metadata layer that makes files discoverable

A catalog maps tables to files: storage catalogs, Hive Metastore, AWS Glue, and open platforms like Unity Catalog. It decides who can discover, who can write, and how concurrent writers coordinate.

catalogHive MetastoreGlueUnity Catalogtable metadatadiscovery
Prerequisites: Lake Fundamentals
Before we start

What you will learn

  • Distinguish storage from catalog metadata
  • Describe catalog patterns (Hive Metastore, Glue, Unity Catalog)
  • Explain why catalog consistency matters for concurrent writers
Introduction

What is this?

A catalog keeps table metadata — location, schema, partitions, owners — apart from the data files themselves. Query engines consult it to turn 'SELECT FROM orders' into file paths. Hive Metastore, AWS Glue, and products like Unity Catalog are the common incarnations; their locking/consistency decides whether two writers to the same table can race.

Motivation

Why do you need it?

  • Nobody queries S3 paths by hand; the catalog is the glue between SQL and objects.
  • Catalog consistency (locking, transactions) is what makes 'table' operations safe on a lake.
  • Table format (Delta/Iceberg) + catalog = where the lakehouse gets its ACID.
Think about it

Mental model

Imagine

The library's card catalog

Think of

A data catalog

Cards (metadata) point to shelves (files) and record sign-out history (ownership). If the card says the book is on shelf A but it moved to B, searchers and shelvers lose trust — which is exactly catalog-vs-storage drift.

Mechanics

How it works

  • Entities: databases (schemas), tables (pointers to a location), partitions (path patterns), columns/statistics.
  • Hive Metastore: a relational DB (MySQL/Postgres) holding metadata, with table locks for coordination.
  • Glue Catalog: managed AWS metadata service backing Athena/EMR/Glue.
  • Unity Catalog: open metadata server offering governance, lineage, and managed tables beyond just metadata.
  • When writing, engines update catalog entries within the table format's transaction protocol to stay consistent.

The catalog decouples table name from file layout.

SQL: orders
Catalog row
File prefixes
Engines
Consumers
Reference

Key commands

aws glue create-database --database-input '{"Name":"analytics"}'

Create a Glue database (metadata namespace).

aws glue create-table --database-name analytics --table-input '{...}'

Register a table over S3 files.

SHOW TABLES; SHOW PARTITIONS sales;

Prove catalog and partition metadata are queryable by engines.

Hands-on

Examples

Catalog entry ~ table-pointer

Catalog entry ~ table-pointer
-- (Hive/Glue-style) register a table over files:
CREATE TABLE IF NOT EXISTS shop.orders (
  id BIGINT, customer_id BIGINT, amount DECIMAL(12,2), day DATE
)
USING PARQUET PARTITIONED BY (day)
LOCATION 's3://company-lake/analytics/orders/';
 
-- Now engines can:
SELECT sum(amount) FROM shop.orders WHERE day = '2026-03-14';

Engines read the catalog's location/partition mapping to prune to day=slices without you touching paths.

Real world

In production

  • A BI layer queries tables in Glue/Unity while the alpha writes are still files on S3.
  • Two teams share one catalog across engines: Spark writes, Trino reads, DuckDB ad-hocs the same table.
  • A governance team reviews catalog lineage and owners without owning any pipeline.
Guidelines

Best practices

  • Standardize one catalog as the source of truth, and sync mirrors rather than forking.
  • Keep catalog metadata and storage consistent: encode partitions/columns in the catalog at write time.
  • Configure catalog-level locking so concurrent writers can't clobber each other.
  • Register owners, tags, and access policies in the catalog, not in slack threads.
Watch out

Common mistakes

  • Running two catalogs for the same tables and letting their metadata drift apart.
  • Writing files without registering schema/partitions, so engines can't discover them.
  • Ignoring catalog lock timeouts when many writers converge on one table.
When things break

Debugging tips

  • A 'table not found despite files existing' problem = registration gap; register the table (or partition) explicitly.
  • Two engines seeing different schemas usually means one uses a stale catalog copy.
  • Lock-tenancy errors during parallel writes point at the catalog's concurrency configuration.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A catalog lives separate from the data files and can be queried by many engines at once.

2

Hive Metastore, AWS Glue, and Unity Catalog are examples of…

Go deeper

Official documentation

Hive metastore docs
docs
AWS Glue Data Catalog
docs
Unity Catalog (open)
docs