Security in Production Data
Assume leak, enforce least privilege
Data security is policy first, tech second: classification and least privilege at row/column level, encryption in transit/at rest, secret handling, PII protection and audit — baked into pipelines, not bolted on.
What you will learn
- Classify data (public/internal/confidential/PII) and map controls to class
- Apply row/column-level policy, encryption and secret hygiene in pipelines
- Design audit trails and a leak-response path
What is this?
Production data security controls who can see what, protects it at rest and in transit, and keeps a record. The practice starts from classification (public vs internal vs confidential vs regulated/PII) and maps controls to class: IAM least privilege, row/column policies, masking and redaction, encryption (TLS, KMS), restricted buckets, secret management for credentials, and audit logs that cannot be edited by the subjects they cover. Pipelines must inherit these controls, not bypass them via 'service accounts with everything'.
Why do you need it?
- Breaches are usually insider/misconfiguration; blast radius = whatever everyone's role-inheritance can touch.
- PII is regulated (GDPR, HIPAA, PCI): both access control and kill-switch (masking/erasure) are legal duties.
- Secrets in code are the top cause of credential leaks — a leaked key is a key that 'everyone' could read.
- Without audit, you can't prove you did the control you claimed, and auditors notice.
Mental model
A hospital ward's medication cabinet
Data security controls
Nurses can access the cabinet (least privilege), it records who took what (audit), and certain meds require double-sign-off (PII clearance). The cabinet doesn't rely on the door being locked once — it assumes, and logs, every touch.
How it works
- Classify: tag datasets (public/internal/confidential/regulated) and columns (PII flags) in the catalog.
- Least privilege: roles with exactly the tables/columns they need; deny-by-default at bucket and table.
- Column/row policy: masking PII for general users, full access for cleared roles; row scoping by tenant.
- Encryption: TLS for transit, SSE-KMS at rest per dataset; customer-managed keys where regulated.
- Secrets: vault/secret manager for credentials, injected at runtime; never in code or images.
- Audit: log every access/query/export by principal+dataset; alerts on anomalous big-export or cross-role reads.
Controls stacked from classification out to audit.
In production
- A healthcare org tags patient names/email as PII; BI roles see masked values, compliance roles see raw, and every read is logged.
- A fintech runs its warehouse with roles scoped per tribe — the ML team can read features but not card numbers without elevated clearance.
- After a secret leaked in a GitHub repo, a platform rotates the key in under an hour because credentials live in a secret manager and were never in the image.
Best practices
- Automate classification: default deny; new tables get a class at creation (CI gate).
- Use workload identity/roles instead of long-lived keys; rotate the few that exist.
- Encrypt with per-dataset KMS keys so a compromise doesn't punch a hole in everything.
- Alert on anomaly: midnight bulk exports, cross-region copies, or a 'delete bucket' action.
Common mistakes
- A 'service account with full access' that everything (and anyone with the code) runs under — the ultimate least-privilege violation.
- Storing secrets in notebooks, DAG definitions or Docker layers that get committed.
- Masking columns in views but forgetting the raw table grants underneath.
- Audit logs anyone with an admin role can delete — the control becomes decoration.
Debugging tips
- Access denial → check policy inheritance across bucket, role and masking layers; deny-by-default makes the first 'it worked yesterday' usually a config drift.
- Leak suspected → check secret age + last rotation, audit the secret ref in pipelines, and invalidate first.
- Audit gap → alerts should trigger on the ACTION, not on 'a log was written'.
- PII slip → run classification scan over object sample; if an untagged column holds PII, the masking policy missed it.
Interview checkpoint
Hands-on lab
- Tag datasets/columns with class and PII flags
- Enforce column masking and row scoping
- Log an audit trail with a revocable role
Apply Row and Column Policy on a Dataset
30 min · intermediate
Quick quiz
Which is the biggest least-privilege violation commonly seen?
Secrets belong in version-controlled configuration, injected at deploy time.