Data Engineering Foundations
Intermediate 15 min read Data Engineering Foundations

Data Governance Basics

Trust, ownership and audit, engineered not negotiated

Governance is the set of policies, catalogs, lineage, ownership and access controls that make data trustworthy, auditable and safe to use — implemented as tooling and automation.

governancecataloglineageownershipaccess controlpolicy
Prerequisites: Data Architecture
Before we start

What you will learn

  • Name the core governance artifacts: catalog, lineage, ownership, policy
  • Explain why catalogues and lineage are engineering deliverables, not just paperwork
  • Implement basic data-access and retention governance in pipelines
Introduction

What is this?

Data governance defines who may use which data, who owns it, where it came from and how trustworthy it is. In practice for engineers it means: a data catalogue that declares every dataset, column-level lineage from source to dashboard, explicit ownership and SLAs, and access control implemented in the platform — not in emails.

Motivation

Why do you need it?

  • Regulatory requirements (GDPR, SOC2, HIPAA) make discovery, deletion and access control mandatory.
  • Catalogs and lineage turn 'which table is right?' into a query, not a mystery.
  • Ownership gives every dataset a pager; without it, issues rot.
  • Access governance prevents the most common breach path: over-privileged internal access.
Think about it

Mental model

Imagine

A library with a card catalogue

Think of

A governed data platform

A library only works because every shelf, call number and borrower is catalogued and checked. Data without a catalog is a book warehouse with no call numbers: even the librarian can't find the right copy.

Mechanics

How it works

  • A catalogue registers datasets: name, schema, owner, freshness, quality tests, retention, tier.
  • Lineage records each dataset's parents (source → transform → dashboard), connecting upstream risk to downstream consumers.
  • Ownership assigns a human + escalation path per dataset; SLAs are encoded as monitored checks.
  • Access control is applied at compute (row/column-level if sensitive, role-based on tables).
  • Retention and tidy-up policies run automatically: archive, anonymize, purge per lifecycle.

Governance artifacts around the data platform.

Sources
Catalog
Lineage
Access control
Audit & policy
Real world

In production

  • A catalogue owns 400 dashboards' lineage: when a source column changes, the platform flags every downstream report before it breaks.
  • Row-level filters apply 'segment' rules so a support rep querying a shared table sees only their region's rows.
  • Retention automation purges raw clickstream after 90 days and archives financial data to cold tier for 7 years, per policy.
Guidelines

Best practices

  • Register every dataset in the catalog as the first step of pipeline creation.
  • Make ownership explicit and automatic: on pipeline creation, require owner + on-call.
  • Instrument column-level lineage for anything sensitive or high-frequency consumed.
  • Apply least-privilege as the default and review data access quarterly like a firewall rule review.
Watch out

Common mistakes

  • Treating governance as documentation nobody updates — it rots within weeks.
  • No column-level lineage, so a leaked PII field takes weeks to trace.
  • Granting 'admin' broad access because the model is 'internal only'.
  • Storing secrets or PII in open-format lake files without encryption or masking.
Root cause analysis

Common issues

Error
A forgotten wide-open role exposed a sensitive table
Root cause

Broad group grant with no least-privilege review

Fix

Switch to explicit grants, add a quarterly access review, and enable audit logging on sensitive reads.

Error
Data was deleted but no one knows whom to blame
Root cause

No owner and no retention review on the dataset

Fix

Require owner at creation and version/data-policy checkpoints before destructive ops.

Error
GDPR deletion request can't be fulfilled
Root cause

No delete path + data mirrored in raw copies

Fix

Design purge jobs that propagate through lineage to every derived copy, and test them regularly.

When things break

Debugging tips

  • Access denied suddenly? Check role/resource changes and the latest policy update first.
  • Can't find a table? Search the catalog before the storage layer - unregistered tables are governance failures.
  • Lineage missing an edge? Re-run lineage extraction and confirm the transform registered its inputs/outputs.
  • Audit a 'mystery' dataset: catalog entry + owner + lineage will tell you who to page.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which artifact directly connects a dashboard metric back to its source tables?

2

Least-privilege access should be the default for data consumers.

Go deeper

Official documentation

OWASP data protection
docs