Storage
Beginner 13 min read Storage

Object Storage

Unlimited, durable, HTTP-accessible data

Object storage (S3, Blob Storage, GCS) stores files as objects in flat buckets with rich metadata and HTTP access. It's the backbone of data lakes, static hosting and backups.

S3object storagebucketGCSBlob Storagekeydurability
Prerequisites: Cloud Service Models
Before we start

What you will learn

  • Explain how objects, keys and buckets differ from a filesystem
  • Compare S3, Azure Blob and GCS semantics
  • Back up and serve data with object storage
In 60 seconds

Object storage addresses data by a key inside a bucket over HTTP, scales to exabytes, and replicates copies across availability zones/regions for 11-9s durability. No directories, no in-place edits — PUT/GET/version.

Introduction

What is this?

S3 = Simple Storage Service (objects + keys, buckets as namespace, optional versioning, lifecycle, replication); Azure Blob Storage = containers/block blobs with tiers (hot/cool/cold/archive); GCS = buckets with classes (standard, nearline, coldline, archive). Objects hold data + metadata + tags; access is via HTTP(S), SDKs, or the console. Multi-part upload handles big objects; the object store is the universal data lake store.

Motivation

Why do you need it?

  • Durability and scale with zero server management.
  • Pay per GB stored and per request — no idle capacity.
  • Single HTTP interface serves web, ML, backup and analytics alike.
  • Native features (versioning, lifecycle, replication, events) replace DIY solutions.
Think about it

Mental model

Imagine

A warehouse where every crate has a barcode and shelf label

Think of

Object storage

You can't open the crate in place (no in-place edits) — you check it out (GET), change it, and check in a new version (PUT). The warehouse (bucket) and barcode (key) make any crate findable from anywhere with HTTP.

Mechanics

How it works

  • Client authenticates (IAM/keys/signed URLs) and addresses a bucket key over HTTPS.
  • Objects have content, content-type, size, metadata and version id if enabled.
  • Durability: erasure coding/replication spreads copies across multiple AZs or zones.
  • HTTP range reads and multipart upload handle huge objects efficiently.
  • Lifecycle rules transition objects to cheaper tiers and expire/delete at set ages.

Object lifecycle.

PUT object
Versioned?
Lifecycle
Replication
Read/expire
Hands-on

Examples

Object storage basics

Object storage basics
# AWS S3
aws s3 cp report.csv s3://my-bucket/ingest/2026-09/report.csv
aws s3 ls s3://my-bucket --recursive
# Azure Blob
az storage blob upload --container-name logs --name app.log --file app.log
az storage blob list --container-name logs
# GCS
gsutil cp report.csv gs://my-bucket/ingest/
gsutil ls gs://my-bucket/ingest/

All three speak HTTP/S3-style semantics: copy in, list, copy out — no mounts, no servers.

Real world

In production

  • A data platform lands Kafka topics into an S3 bucket organized as s3://lake/<env>/<topic>/<date>/
  • Blob Storage serves immutable audit logs with legal-hold and lifecycle to archive.
  • GCS buckets front a static site served via a CDN with signed URLs for downloads.
Guidelines

Best practices

  • Design a key layout (partition + date) that maps to queries and lifecycle.
  • Enable versioning on buckets that hold recoverable data.
  • Use lifecycle rules for tiering and expiry instead of manual garbage collection.
  • Enable bucket-level encryption and deny public writes by default.
Watch out

Common mistakes

  • Treating buckets like folders and doing thousands of small PUTs into one prefix.
  • Storing credentials/secrets in object files that are world-readable.
  • Forgetting lifecycle = costs grow as data ages into the hot tier.
  • Using a filesystem mental model: no rename/append; updates are new versions.
Root cause analysis

Common issues

Error
Access denied reading a 'public' object
Root cause

Bucket policy blocks or ACL/block-public-access is on

Fix

Verify IAM + bucket policy + block-public-access setting; use presigned URLs for external reads.

Error
Unexpected huge bill from a 'small' bucket
Root cause

Lifecycle not set; old versions/tiers piling up

Fix

Add lifecycle rules, prune old versions, and check the storage-class breakdown of objects.

When things break

Debugging tips

  • 403s: check bucket policy, object ACL, IAM role and signed URL expiry in that order.
  • Slow first GET: cold reads/tier transition; verify the object's storage class.
  • Version surprises: listing shows multiple versions when versioning on — filter max-keys.
  • Multipart uploads that never complete leave visible parts; abort them (list multipart).
Interviews

Interview checkpoint

Practice

Hands-on lab

Land, Version and Ingest Files into Object Storage
25 min · beginner
In progress
Objectives
  • Create a bucket/container and upload objects with CLI
  • Enable versioning and observe object versions
  • Generate presigned URLs for time-limited access

Land, Version and Ingest Files into Object Storage

25 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/1
1

What gives object storage its durability story?

Go deeper

Official documentation

AWS — S3
docs
Azure — Blob Storage
docs
GCP — Cloud Storage
docs