Object Storage
The cheap, infinite shelf your lake lives on
Object storage (S3, GCS, Azure Blob) is the substrate of every modern lake: massively scalable, cheap, and eventually consistent in interesting ways. Buckets, prefixes, versioning, and lifecycle rules are its API.
What you will learn
- Explain what an object is (data + key + metadata)
- Use prefixes, versioning, and lifecycle rules
- Describe consistency semantics of object storage
What is this?
Object storage stores objects — a blob of bytes with a key and metadata — in a flat namespace addressed by bucket + key. It is massively parallel, durable, and cheap; 'folders' are just prefix conventions. Versioning, encryption, and lifecycle expiry make it the durable substrate of lakes and lakehouses.
Why do you need it?
- Every modern lake reads and writes objects; CPI costs nothing locally but drives everything remotely.
- Lifecycle rules are your cost control: hot → cold → expired is a config, not a job.
- Consistency (`read-after-write` vs `eventually consistent` list semantics) affects pipeline correctness.
Mental model
A postal system with an infinite number of labeled P.O. boxes
Object storage
Your address is bucket+key; the post office (storage) is durable and cheap, but asking it 'which letters exist' (list) is slower than asking for a specific letter (get). That's the ephemeral 'folder' behavior.
How it works
- Objects live in buckets; keys with '/' render as folders in tools but are really flat prefixes.
- Range reads and multipart uploads let big files stream efficiently.
- Versioning keeps deleted/overwritten objects recoverable inside the bucket.
- Lifecycle policies transition storage classes (standard→infrequent→glacier) and expire objects.
- Consistency: strong read-after-write for get, eventually-consistent for list in legacy modes; newer regions tightened it.
Bucket → prefix → object path.
Key commands
aws s3 cp raw/file.json s3://lake/raw/events/file.jsonUpload one object (multipart under the hood for big files).
aws s3api put-bucket-versioning --bucket lake --versioning-configuration Status=EnabledEnable recovery on overwrite/delete.
aws s3api get-object-tagging --bucket lake --key ...Read object metadata used to make shelf-life decisions.
Examples
Hive-style partitioning in the key
s3://company-lake/curated/orders/day=2026-03-14/part-0000.parquets3://company-lake/curated/orders/day=2026-03-14/part-0001.parquet # Engines prune by the = partitions encoded in the KEY, not by inspecting rows:SELECT * FROM read_parquet('s3://company-lake/curated/orders/day=*/part-*.parquet')WHERE day = '2026-03-14';Partition keys live in the path; predicate pushdown prunes by prefix without scanning.
In production
- A lake writes one Parquet file per partition per run; idempotent overwrite makes replays safe.
- Lifecycle tiering moves 90-day-old raw event files to colder storage automatically.
- Versioning lets a pipeline roll back a bad 'delete + rewrite' of a table's partition.
Best practices
- Design keys for listing and pruning: high-cardinality first (day), low-cardinality last.
- Use versioning on failsafe zones and expect the storage cost of old versions.
- Batch small files: thousands of tiny objects wreck read parallelism and listing costs.
- Enforce encryption-in-transit and server-side encryption defaults.
Common mistakes
- Treating prefix paths as a real directory tree with rename semantics; the API is put/get/delete/list.
- Backing an OLTP workload on object storage pretending it's a database.
- Ignoring list/consistency behavior when a listing drives a 'is it all there?' check.
Debugging tips
- A missing file in a 'ranged' read usually means the connect name/prefix case drifted, not data loss.
- Slow list iterations scream prefix-per-fine-grained-structure; widen the prefix or batch listing.
- When overwrites 'disappear', check versioning status and the exact object key.
Interview checkpoint
Quick quiz
Folders in S3-style object storage are actually…
Lifecycle policies can transition objects to colder, cheaper storage classes automatically.