Governance in Production
Control as an engineering practice
Governance is the operational side of control: a catalog that is the source of truth, classification, access, retention, lineage and compliance — working as engineering practice, not committee paperwork.
What you will learn
- Run a governed catalog: owners, definitions, classification, retention as code
- Operationalize access, audit and lifecycle policy
- Map compliance obligations (GDPR, SOC2) to concrete catalog controls
What is this?
Data governance in production is the everyday operation of control: a catalog that is the single source of truth for definitions, owners, classification, lineage and policy; access grants driven by role; retention and erasure executed by lifecycle jobs; and compliance mapped to concrete catalog fields and checks. It fails when it becomes a committee; it works when every change to data (create, grant, delete, move) passes through governed, versioned, auditable steps.
Why do you need it?
- Regulated (GDPR/HIPAA/SOC2) obligations become concrete, testable controls instead of promises.
- A trustworthy catalog stops the 'define it yourself' chaos that produces duplicate, contradictory tables.
- Retention and erasure automated by policy keep legal risk and cost in check.
- Lineage through governance means impact analysis and audits use the same map.
Mental model
Building codes plus an on-site inspector
Data governance
Codes (policy) are dead text without the inspector checking every build step (catalog-driven controls, CI gates, audit). Governance is the inspector who checks doors, exits and wiring as work happens — not a binder of rules on a shelf.
How it works
- Catalog as registry: every dataset declares owner, steward, class, PII flags, retention window at creation.
- Policy as code: access rules, masking and lifecycle rules versioned in the repo and applied at deploy.
- Access management: role-based grants via groups; requests flow through the catalog, not ad-hoc SQL.
- Retention/lifecycle: policy-driven deletion/archival jobs with approval and audit for exceptions.
- Erasure: GDPR right-to-erasure workflows query the classification inventory and execute per-subject deletes.
- Audit + attestation: periodic access reviews and compliance checks run from the catalog, not from memory.
Governance operating loop through the catalog.
In production
- A bank's new-request catalog requires class and retention for every table; a CI check refuses deploy until they exist.
- A retailer runs GDPR erasure as a lifecycle job: subjects in a queue are deleted from every table whose policy says PII, with confirmation logs.
- A cloud company does quarterly access reviews generated from the catalog: stale grants are auto-revoked and exceptions are escalated to the steward.
Best practices
- Make the catalog authoritative and code-checked at creation — fixing governance later is ten times the cost.
- Automate retention/erasure; manual deletion is where legal risk hides.
- Review access on a cadence and auto-revoke stale grants with audit.
- Map compliance controls to exact catalog checks so 'are we compliant' is queryable.
Common mistakes
- Creating a governance committee that owns policy docs nobody executes.
- Retention policies that exist but kill jobs that read beyond the window (policy vs reality divergence).
- Audit as an afterthought — no immutable log of who granted what.
- Governance applied only in production, so dev copies grow unclassified PII.
Debugging tips
- 'Denied by policy' → check class + masking + retention together; contradictory layered policies are the top cause.
- Erasure gaps → the classification scan missed a derived table; rerun discovery over lineage expansion.
- Retention job deletes something needed → sign-off/approval workflow was bypassed; add a two-eye check to exceptions.
- Reviews drift → schedule attestation from the catalog query so people review a real grant list.
Interview checkpoint
Hands-on lab
- Declare classification, owner and retention at dataset creation
- Run a CI gate that rejects undeclared datasets
- Execute retention policy with approval and audit
Enforce a Governed Catalog with CI Checks
30 min · intermediate
Quick quiz
The single source of truth for ownership, class and lineage is:
Retention policies should be executed automatically, with audit, not manually.