Data Lake Fundamentals
Keep everything, decide meaning later
A data lake stores raw, schema-light files in cheap object storage, letting you keep everything and decide meaning later. Its leverage is storage price and flexibility; its cost is governance discipline.
What you will learn
- Contrast lake vs warehouse storage semantics
- Describe lake zones (raw, curated, analytics)
- Name the governance obligations a lake inherits
What is this?
A data lake is a repository of raw files — JSON, CSV, Parquet, events — stored in cheap object storage, without a pre-stated schema. Data lands as-is; schemas are applied when read. This keeps everything at low cost and defers modeling, but demands governance (access, quality, cataloging) to avoid becoming an un-managed dump.
Why do you need it?
- Storage is orders cheaper than a warehouse's managed copy, so 'keep all raw data' becomes viable.
- Machine-learning and discovery workflows need untouched source fidelity that a modeled warehouse lacks.
- It's the origin story for the lakehouse and open formats in this module.
Mental model
A warehouse back-lot holding every shipment, unopened
A data lake
Flights are stored the moment they arrive; nobody's decided how to stack them, so you can find anything later (raw fidelity). But if nothing's labeled (governance), no one can find the shipment they need.
How it works
- Object storage (S3/GCS/blob) provides immutable files with prefix-based addressing; buckets act like folders.
- Files land in logical zones — raw/landing, curated, analytics — encoded in the prefix path.
- Schema-on-read: a query engine (Spark/DuckDB/Trino) interprets the file format at query time.
- Catalog services (Glue, Hive Metastore, Unity Catalog) register tables on top of the paths.
- Governance concerns: access at the file/prefix level, lineage, retentions, and 'stale file' handling.
The three-zone lake.
Examples
Prefixes ARE the zoning
s3://company-lake/ named/ # user/bucket/permission grain events/billing/ # raw clickstream/API payloads (schema-on-read) users/ # entity captures curated/ # cleaned, typed Parquet written by pipelines analytics/ # modeled star schemas read by BIThe naming convention encodes zones and access scope; tools map these prefixes into tables via a catalog.
In production
- An events pipeline drops raw JSON events into S3 'raw' the moment they arrive, before anyone models them.
- A governance dashboard lists 'curated' tables versus 'raw dumps' so access and retention differ per zone.
- A discovery engineer can point DuckDB directly at fresh raw files for quick exploration without copying to the warehouse.
Best practices
- Adopt a strict zone-and-prefix convention from day one; retrofitting costs more than naming.
- Write immutable files (no in-place editing) and version via paths/timestamps.
- Register everything in a catalog with ownership and quality tags.
- Enforce access control at the storage level as well as the table level.
Common mistakes
- Treating the lake as a garbage pit: anything lands with no prefix convention and no catalog.
- Copying every event into the warehouse 'just in case', paying twice.
- Believing schema-on-read removes the need for schema validation; bad files decay every consumer.
Debugging tips
- A 'missing file' in BI usually points at a catalog metadata mismatch, not a deleted file.
- When a lake query reads 'nothing', check the prefix path and file naming pattern vs the catalog's table definition.
- Slow cold queries are often many small files; that's the optimization topic later in this module.
Interview checkpoint
Hands-on lab
- Create raw/curated/analytics folders in a local 'object store'
- Land a raw JSON file, then curate it to Parquet
- Query the curated Parquet with DuckDB
Zone Your First Mini-Lake
30 min · beginner
Quick quiz
A data lake's chief advantage is…
Schema-on-read means the same raw file can flow into different shaped queries without rewriting the file.