Data Lakes on the Cloud (S3 + Glue)

• Explain how a metastore (Glue/Hive) turns object prefixes into queryable tables
• Set up a schema-on-read lake with raw, trusted and curated zones
• Secure the lake with Lake Formation-style policies and audit logs

• You keep raw data forever at object-storage cost and define structure at read time.
• glue/CSV/Parquet/JSON can all coexist — schema evolves without rewriting history.
• One metastore means Athena, Spark and warehouses share the same tables/views.
• Zone separation keeps 'original evidence' from 'analytics opinions' with clear policy boundaries.

Books (objects) in the stacks (bucket); the catalogue (metastore) says which shelf holds '2030 tax returns' and what format they're in. You can re-catalogue without moving a book.

• Write zone: files land in s3://lake/raw/{source}/dt=<day>/ in open formats.
• Register tables: Glue crawlers or deploy-time SQL create catalog entries pointing at prefixes.
• Schema-on-read: the catalog defines the view; files can be re-interpreted without transformation.
• Evolve: add columns/partitions; old objects stay untouched and readable under the new catalog definition.
• Govern: Lake Formation/IAM grant table-and-column access; audit logs record every read.
• Curate: Spark/Glue jobs write trusted/curated zones (dedupe, validate, model) as new catalog tables.

• A streaming pipelander writes click events to lake/raw/clickstream and a nightly Spark job dedupes them into trusted/clickstream_dedup, registered as a Glue table.
• A pharma firm keeps every clinical raw file forever for auditors, while curated aggregate tables are column-restricted so researchers can't touch PII.
• A retailer uses Glue crawlers to auto-register new SaaS export folders so the catalog is fresh without manual SQL.

• Standardize zones (raw/trusted/curated) and a partition convention as code, not habit.
• Register tables at deploy time, not with a dreaded 'cluster of crawlers doing chaos' at midnight.
• Put partitioning, compression and schema evolution conventions into the pipeline template.
• Enable access/audit logs on the lake — governance questions arrive faster than you think.

• Treating the metastore as optional — no catalog means every consumer re-invents the schema.
• Crawling-and-never-updating, so catalog partitions point at deleted files and queries break.
• Raw zone written directly by ad-hoc scripts nobody owns, breaking reproducibility.
• Column-level security ignored, so 'single team owns the lake' fails the first compliance review.

• table exists but shows zero rows → partitions registered but files live under an unregistered prefix.
• 'Schema mismatch' between engine and files → catalog definition vs Parquet schema drifted (re-run the register job).
• Query cost spike → a partition column moved out of the path so pruning gagged.
• Access denied but file is public → the metastore/Lake Formation policy differs from bucket policy.