Data Lakes
Intermediate 12 min read Data Lakes

Lake Security and Governance

Files are governed differently than tables

Lakes offer file-level access, encryption, and lineage — but the models, PII classification, and retention rules still live with the humans. Govern storage, catalog, and pipelines as one surface.

encryptionaccess controlPIIretentionlineagepolicy
Prerequisites: Lake Fundamentals Catalog Layers
Before we start

What you will learn

  • Set storage + catalog access control in layers
  • Classify PII and enforce redaction/masking
  • Apply retention and lineage as pipeline-native policy
Introduction

What is this?

Security for a lake means protecting objects, catalog, and pipelines as one surface: bucket/prefix-level IAM, region/encryption, data classification with masking, lineage so 'who reads what' is auditable, and lifecycle retention so data dies per policy. Every open file multiplies the audit surface.

Motivation

Why do you need it?

  • A lake stores everything — including PII nobody remembers it has.
  • File-level access is coarse; catalog-level governance is where the real control lands.
  • Compliance is a design property, not a feature bolt-on: build it into prefixes, roles, and pipelines.
Think about it

Mental model

Imagine

A facility with badge zones, windows, and an incinerator log

Think of

Lake security

Badges (IAM) control who enters zones; some rooms have frosted glass (masking of PII); sensitive forms have a printed deadline (retention) and every viewing is logged (lineage/audit).

Mechanics

How it works

  • Storage security: bucket/prefix policies, server-side encryption (SSE-KMS), and access-point baselines.
  • Catalog security: table/schema grants supersede path ACLs; e.g. column masking at the catalog.
  • Classification: tag objects/tables as PII; masking applied at the query layer, not in files.
  • Lineage: track table/file → pipeline → dashboard so access reviews and incidents route correctly.
  • Retention: lifecycle rules expire raw dumps; expiration is logged and tested in drills.

Layered lake security controls.

Object storage
Catalog
Classify
Lineage + audit
Lifecycle
Hands-on

Examples

Governance encoded in prefix and role

Governance encoded in prefix and role
# IAM-style: least-privilege bucket policy
{
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::company-lake/analytics/*",
  "Principal": { "AWS": "arn:aws:iam::...:role/analyst-reader" }
}
 
# catalog: mask PII in SQL
CREATE VIEW shop.customers_safe AS
SELECT id, name, substr(email,1,2)||'***' AS email_mask ... FROM raw.customers;

Storage gates the bytes; the catalog/view gates what anyone may actually read.

Real world

In production

  • An analyst role reads only /analytics prefixes; HR dumps live in a PII-flagged zone with separate roles.
  • A masking view extends customer emails to support while raw PII stays locked.
  • A data retention scanner expires 400-day-old raw event files and logs the deletion for compliance.
Guidelines

Best practices

  • Default-deny on storage; grant least privilege per zone prefix.
  • Classify tables at ingestion (PII, sensitive, public) so masking/retention follow automatically.
  • Route lineage through catalog/table level so access reviews aren't file archaeology.
  • Test retention with an expiration drill: unpublish a bucket, verify no reads, then delete.
Watch out

Common mistakes

  • Granting bucket-wide List/Get 'so BI works', turning every file into a trust boundary.
  • Masking only in dashboards: raw files on S3 still leak PII to anyone with the keys.
  • Deleting files with no audit trail — compliance asked for a log, not a guess.
When things break

Debugging tips

  • A 'secretly readable' table usually means a too-broad storage grant bypassing catalog controls.
  • When masking breaks, check the masking VIEW's grant vs the raw table path.
  • A retention job that deletes 'too much' often lacks a dry-run gate in front of the lifecycle rule.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Masking PII in dashboards is sufficient when raw files remain open on object storage.

2

Retention policies in a lake are best implemented as…

Go deeper

Official documentation

S3 security best practices
docs
GCS access control
docs
Unity Catalog access control
docs