Data Engineering Foundations
Intermediate 16 min read Data Engineering Foundations

Data Architecture

Layering and reference architectures

Data architecture is the intentional design of how data is ingested, stored, transformed, governed and consumed — and the reference architectures that encode those decisions.

data architecturemedallionlakehousestreaming architecturemodern data stack
Prerequisites: ETL vs ELT The Data Lifecycle
Before we start

What you will learn

  • Describe the standard layers of a data platform
  • Compare medallion, lakehouse and modern-data-stack architectures
  • Make trade-offs that keep an architecture simple enough to run
Introduction

What is this?

Data architecture is the blueprint of a data platform: which systems ingest, store, transform, serve and govern data, and how they are connected. Good architecture separates concerns into layers so each stage can scale, fail and evolve independently, and it encodes decisions teams won't relitigate every quarter.

Motivation

Why do you need it?

  • Without architecture, teams build ad-hoc pipes that collide, duplicate and contradict each other.
  • Layering lets you upgrade storage without rewriting transforms, and vice versa.
  • Reference architectures give you a starting point and a vocabulary to argue trade-offs.
  • Architecture is how you make cost, freshness and quality promises attainable and auditable.
Think about it

Mental model

Imagine

A building's floors

Think of

A data platform's layers

You don't wire the plumbing through the roof or hang a door in the ceiling — each floor has a purpose and connections follow the structure. Data platforms need the same discipline.

Mechanics

How it works

  • Ingest layer pulls from sources into raw storage; it only understands 'get it in'.
  • Raw storage (lake/staging) holds immutable copies of source truth.
  • Curated/transformed layer holds clean, deduplicated, modeled datasets.
  • Semantic/serving layer exposes metrics and governed views to consumers.
  • Governance and observability cut across every layer: lineage, quality, ownership, SLAs.

Layered data platform.

Sources
Ingest
Raw store
Curated layer
Serving
Real world

In production

  • Medallion (bronze/silver/gold) architecture: bronze holds raw arrays/events, silver cleans and joins to entity-level, gold models business analytics — a single pipeline stages increments through each.
  • Lakehouse: a single open-format store (Parquet/Delta, Iceberg) serves both data-engineering and ML workloads, avoiding two ecosystems.
  • Modern data stack: managed ingestion (Fivetran) → warehouse (Snowflake/BigQuery) → transform (dbt) → BI/catalog, chosen for ops velocity.
Guidelines

Best practices

  • Keep raw data immutable and versioned; treat it as evidence, not a scratch layer.
  • Separate ingest from transform owners so a modeling bug can't corrupt ingestion.
  • Write a short architecture decision record when choosing storage or serving tech.
  • Design for the organization's real scale, not the vendor's brochure.
Watch out

Common mistakes

  • More layers than the team can operate — a 'sophisticated' platform nobody can run on-call.
  • Copying data between layers with no lineage, so no one knows which table is authoritative.
  • Choosing a lakehouse/warehouse on trend without considering compliance or query patterns.
  • Letting every team build its own ingestion pipe to the same source with different results.
Root cause analysis

Common issues

Error
Two dashboards disagree on 'revenue'
Root cause

Serving from different layers/models with different definitions

Fix

Consolidate into a single semantic layer where revenue is defined once and both dashboards read it.

Error
Raw copy grows unchecked, cost exploded
Root cause

No retention/archival policy on the raw layer

Fix

Tier and purge raw per lifecycle policy; keep audit-controlled deep history only where required.

Error
Every team re-ingests Salesforce separately
Root cause

No shared ingest contract and no central catalog

Fix

Rename shared ingestion to a single owner and register it in the catalog with clear SLA.

When things break

Debugging tips

  • When a number disagrees across two dashboards, trace both to their raw-store roots and compare the 'what' at the source.
  • Check lineage to see which layer mutated the value last.
  • If ingestion is duplicated, pick one extractor as the owner and deprecate the rest.
  • For architecture drift, grep for tables nobody references anymore (dead data).
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which layer should store immutable copies of source data?

2

Adding more architectural layers always reduces operational risk.

Go deeper

Official documentation

Databricks — medallion architecture
docs