Big Data
Beginner 12 min read Big Data

The Big Data Problem

When one machine stops being enough

Big data is not about size alone — it's about data that no single machine can store, index or query within the SLA. The challenge is engineering your way out of the single-node bottleneck.

big datascalehorizontal scalingdistributed systemsvolume3 V's
Prerequisites: What is Data Engineering?
Before we start

What you will learn

  • Define big data in terms of the workload, not a byte count
  • Explain why scaling up a single machine stops working
  • Recognize the shape of a problem that needs a distributed approach
Introduction

What is this?

Big data refers to datasets so large or so fast-moving that a single machine cannot store, process or query them within acceptable time and cost. The modern definition is workload-driven: if adding vertical power (bigger RAM, faster disk) stops buying you linear gains, you have a big-data problem that horizontal scale — many machines working together — must solve.

Motivation

Why do you need it?

  • Single-node databases run out of I/O and memory long before they run out of use cases.
  • Cost scales superlinearly for vertical upgrades once you pass commodity hardware.
  • Distributing storage and compute across many cheap nodes is the only economical way to grow.
  • Distributed processing is a core data-engineering skill: most warehouses are built on it under the hood.
Think about it

Mental model

Imagine

One super-chef vs a brigade

Think of

Single node vs a distributed cluster

One chef can cook one dish fast, but a banquet for 10,000 needs a brigade of cooks, each owning a station, coordinated by a pass. Big data is running a kitchen brigade where no single cook can hold all the ingredients.

Mechanics

How it works

  • Data is split into chunks (partitions, blocks) distributed across many machines.
  • Compute moves to the data: each machine processes the chunk it holds, then results are merged.
  • A coordinator (master) tracks which machines hold what and schedules work across them.
  • Fault tolerance handles machines dying mid-job by re-running the lost work on survivors.
  • You trade coordination overhead (network, scheduling) for scale that grows by adding machines.

The single-node wall vs a distributed cluster.

1 node
Vertical scale-up
Distributed
Coordinator
Cluster scales out
Real world

In production

  • A search company indexes 100TB+ of web pages by sharding them across a cluster rather than buying a monster machine.
  • A telco stores a day of call records (billions of rows) in a distributed file system and queries it with a distributed engine.
  • An e-commerce platform tweets out 'scale to 10k nodes on Black Friday and back overnight' — impossible on one box.
Guidelines

Best practices

  • Estimate your real SLA before choosing architecture: many 'big data' problems are actually daily-batch problems.
  • Partition data by the natural query dimension (time, geo, tenant) so most queries touch few machines.
  • Store raw data once in a distributed layer and derive everything from it.
  • Design for node failure: assume any machine can die mid-job and that the job must still finish.
Watch out

Common mistakes

  • Building a Spark cluster for a 50GB CSV that a laptop could handle.
  • Buying one gigantic machine when two modest ones would be cheaper, faster and safer.
  • Forgetting that network bandwidth, not disk, is often the real bottleneck in distributed systems.
  • Choosing tooling for its hype rather than the workload's storage, latency and cost profile.
Root cause analysis

Common issues

Error
Cluster is big but queries are slow
Root cause

Poor partitioning or small files causing many tiny tasks and network overhead

Fix

Coalesce small input files and partition on the query's filter dimension.

Error
Adding nodes didn't speed things up
Root cause

The job is bound by a single stage (skew) or the driver, not parallel work

Fix

Profile stage durations; fix the straggler stage or driver bottleneck first.

Error
Job dies whenever a node goes down
Root cause

No fault tolerance configured; work isn't re-runnable from checkpoint

Fix

Enable task retries and checkpoint intermediate state so lost work restarts from the last commit.

When things break

Debugging tips

  • Start on a sample: if the logic is wrong you'll debug it 1000x easier on 10MB than on 10TB.
  • Check data distribution — a hotspot is a distributed-systems bug, not a 'too slow' bug.
  • Time each stage separately: shuffle, read and compute each have different fixes.
  • Have a benchmark baseline so you can tell 'regression' from 'inherently slow'.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which is the clearest signal that you have a big-data problem?

2

Big data is strictly a question of storage size.

Go deeper

Official documentation

Hadoop — HDFS architecture
docs