Big Data
Advanced 17 min read Big Data

Benchmarking and Tuning

Measure first, tune second, verify always

Tuning requires measurement: define SLAs, build benchmarks with realistic data volume and skew, record baselines, and only then change settings one at a time. Metrics (throughput, latency, shuffle ratio, cost) decide what's fixed.

benchmarktuningbaselineSLOthroughputlatencycost per query
Prerequisites: Spark Optimization
Before we start

What you will learn

  • Design a benchmark representative of production data
  • Pick the metrics that expose the bottleneck
  • Run a safe tuning sweep that preserves correctness
Introduction

What is this?

Benchmarking and tuning is the discipline of measuring a workload against an explicit SLA, forming a baseline, then adjusting the engine's knobs and re-measuring — changing one variable at a time. Benchmarks must mirror production shape (volume, cardinality, skew, concurrency) or they tune for a dataset that doesn't exist. Metrics like throughput, latency percentiles, shuffle ratio and cost-per-query tell you where to tune.

Motivation

Why do you need it?

  • Without a baseline you can't tell a regression from a re-run or a config change from luck.
  • Most 'tuning' is guesswork; a benchmark loop converts it into an experiment.
  • SLA-driven tuning avoids the trap of making a non-critical job faster.
  • Quantified cost-per-query becomes the negotiation currency for infrastructure spend.
Think about it

Mental model

Imagine

Training for a race

Think of

Benchmarking

You time a lap (baseline), improve one thing, and time again — not retrofitting the whole routine and guessing which change mattered. A stopwatch (metrics) is non-negotiable.

Mechanics

How it works

  • Define the SLA: target latency/throughput on the workload people actually care about.
  • Build data that mimics production (size, cardinality, skew); avoid 'perfect' synthetic data.
  • Record the baseline: run, collect metrics (task time, shuffle, spill, cost), archive the run.
  • Change ONE configuration (or code pattern) and re-measure against the same inputs.
  • Promote only changes that meet the SLA and survive a re-run and a backfill test.

The benchmark loop.

SLA
Realistic data
Baseline
Change one knob
Verify + promote
Real world

In production

  • A team benchmarks nightly ETL against an SLA, halting a refactor that made dev fast but production slow.
  • A platform measures cost-per-query on big scans and moves the top-50 to a cheaper engine.
  • An A/B of shuffle configuration on the same parquet set decides the fleet-wide default.
Guidelines

Best practices

  • Mirror production data shape exactly: skewed keys, nulls, real file counts and sizes.
  • Measure percentiles (p95/p99) for interactive queries, not just the mean.
  • Archive baselines with run-id and env so comparisons are legitimate.
  • Tune one variable at a time and always re-run the correctness assertions.
Watch out

Common mistakes

  • Optimizing a small synthetic dataset then shipping differently at scale.
  • Comparing runs on different node counts and calling it a 'config win'.
  • Measuring only wall time — shuffle, spill and cost tell you *why* and *how much*.
  • Tuning until correctness assertions break and slipping it anyway.
Root cause analysis

Common issues

Error
Benchmark passed but the real job is slow
Root cause

Synthetic data lacked production skew, cardinality or concurrency

Fix

Replay a sample of real production data/off-topic load and re-baseline.

Error
Different runs of the same job vary wildly
Root cause

Concurrency, cache state, or swarm noise not pinned

Fix

Run on a controlled cluster or reserve, repeat 3x, and use medians/percentiles.

Error
Config change sped up one job, broke several
Root cause

Overfitting a knob (e.g. forcing broadcast) for a single query

Fix

Sweep across a representative query set and pick configs that hold overall, not per-query.

When things break

Debugging tips

  • Re-run the baseline after any env/engine upgrade; 'tuning' vs 'platform' separated.
  • Percentile spreads (p50 vs p99) reveal stragglers/concurrency, not average speed.
  • Cost metrics (scan bytes, shuffle bytes, egress) explain fixes wall-time can't.
  • Keep the query identical when changing config; a rewritten query confuses the cause.
Interviews

Interview checkpoint

Practice

Hands-on lab

Benchmark a Query and Measure One Variable Change
35 min · advanced
In progress
Objectives
  • Build a benchmark dataset with production-like volume
  • Take a repeatable baseline (median of 3 runs)
  • Change one config and prove whether it helped with the same input

Benchmark a Query and Measure One Variable Change

35 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A config change made one job 3x faster but you didn't re-measure other jobs. The danger is...

2

A benchmark dataset that lacks production skew is still a faithful predictor.

Go deeper

Official documentation

TPC-H benchmark spec
docs
Spark performance resources
docs