Production Data Engineering
Intermediate 16 min read Production Data Engineering

Governance in Production

Control as an engineering practice

Governance is the operational side of control: a catalog that is the source of truth, classification, access, retention, lineage and compliance — working as engineering practice, not committee paperwork.

governancecatalogpolicyretentioncompliancelineage
Before we start

What you will learn

  • Run a governed catalog: owners, definitions, classification, retention as code
  • Operationalize access, audit and lifecycle policy
  • Map compliance obligations (GDPR, SOC2) to concrete catalog controls
Introduction

What is this?

Data governance in production is the everyday operation of control: a catalog that is the single source of truth for definitions, owners, classification, lineage and policy; access grants driven by role; retention and erasure executed by lifecycle jobs; and compliance mapped to concrete catalog fields and checks. It fails when it becomes a committee; it works when every change to data (create, grant, delete, move) passes through governed, versioned, auditable steps.

Motivation

Why do you need it?

  • Regulated (GDPR/HIPAA/SOC2) obligations become concrete, testable controls instead of promises.
  • A trustworthy catalog stops the 'define it yourself' chaos that produces duplicate, contradictory tables.
  • Retention and erasure automated by policy keep legal risk and cost in check.
  • Lineage through governance means impact analysis and audits use the same map.
Think about it

Mental model

Imagine

Building codes plus an on-site inspector

Think of

Data governance

Codes (policy) are dead text without the inspector checking every build step (catalog-driven controls, CI gates, audit). Governance is the inspector who checks doors, exits and wiring as work happens — not a binder of rules on a shelf.

Mechanics

How it works

  • Catalog as registry: every dataset declares owner, steward, class, PII flags, retention window at creation.
  • Policy as code: access rules, masking and lifecycle rules versioned in the repo and applied at deploy.
  • Access management: role-based grants via groups; requests flow through the catalog, not ad-hoc SQL.
  • Retention/lifecycle: policy-driven deletion/archival jobs with approval and audit for exceptions.
  • Erasure: GDPR right-to-erasure workflows query the classification inventory and execute per-subject deletes.
  • Audit + attestation: periodic access reviews and compliance checks run from the catalog, not from memory.

Governance operating loop through the catalog.

Define
Apply
Operate
Attest
Audit
Real world

In production

  • A bank's new-request catalog requires class and retention for every table; a CI check refuses deploy until they exist.
  • A retailer runs GDPR erasure as a lifecycle job: subjects in a queue are deleted from every table whose policy says PII, with confirmation logs.
  • A cloud company does quarterly access reviews generated from the catalog: stale grants are auto-revoked and exceptions are escalated to the steward.
Guidelines

Best practices

  • Make the catalog authoritative and code-checked at creation — fixing governance later is ten times the cost.
  • Automate retention/erasure; manual deletion is where legal risk hides.
  • Review access on a cadence and auto-revoke stale grants with audit.
  • Map compliance controls to exact catalog checks so 'are we compliant' is queryable.
Watch out

Common mistakes

  • Creating a governance committee that owns policy docs nobody executes.
  • Retention policies that exist but kill jobs that read beyond the window (policy vs reality divergence).
  • Audit as an afterthought — no immutable log of who granted what.
  • Governance applied only in production, so dev copies grow unclassified PII.
When things break

Debugging tips

  • 'Denied by policy' → check class + masking + retention together; contradictory layered policies are the top cause.
  • Erasure gaps → the classification scan missed a derived table; rerun discovery over lineage expansion.
  • Retention job deletes something needed → sign-off/approval workflow was bypassed; add a two-eye check to exceptions.
  • Reviews drift → schedule attestation from the catalog query so people review a real grant list.
Interviews

Interview checkpoint

Practice

Hands-on lab

Enforce a Governed Catalog with CI Checks
30 min · intermediate
In progress
Objectives
  • Declare classification, owner and retention at dataset creation
  • Run a CI gate that rejects undeclared datasets
  • Execute retention policy with approval and audit

Enforce a Governed Catalog with CI Checks

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The single source of truth for ownership, class and lineage is:

2

Retention policies should be executed automatically, with audit, not manually.

Go deeper

Official documentation