Automation Testing
Intermediate 14 min read Automation Testing

Data-Driven Testing

Data-driven testing separates test logic from test data: one test body, many datasets. Parametrize inputs from arrays, CSV, JSON or Excel, and you get broad coverage, readable reports, and data that business users can own without touching code.

data-driven testingparametrizeparameterized testscsvjsontest datajunit parameterized
Prerequisites: Test Frameworks
Introduction

What is this?

Data-driven testing (DDT) keeps the test procedure fixed and varies only the data it runs against. The same test body executes once per dataset row - each row being an input set plus expected outcome. pytest's @pytest.mark.parametrize, JUnit's @ParameterizedTest and Playwright's fixtures for() all express this pattern, which turns one test into a matrix of cases with full reporting per row.

Motivation

Why do you need it?

  • One code path validates dozens of inputs, so coverage grows without a proportional maintenance burden.
  • A new edge case is a data row, not new code - non-developers can add cases from a CSV or JSON file.
  • Each row runs independently and reports separately, so a failing case does not hide the others.
  • Boundary conditions (empty strings, nulls, unicode, huge numbers) are easy to encode explicitly.
  • Data can be generated or sourced from production captures, giving realistic, high-value coverage.
Think about it

Mental model

Imagine

A recipe card

Think of

A data row

A single recipe (test body) describes how to cook any dish; the ingredients list (data row) is what changes. Cook the recipe ten times with ten different ingredient lists and you get ten meals from one card. Data-driven testing cooks the same recipe once per row.

Mechanics

How it works

  • Define a test body that takes inputs and an expected result as parameters.
  • Declare a data source: inline tuples, a CSV/JSON file, or a factory that yields rows.
  • The runner expands one case per dataset row and executes the body once for each.
  • Each case gets a unique test ID (for example test_add[3-6-9]) so reports and failures are traceable.
  • One row failing fails just that case; the runner continues the rest and reports the matrix.

One test body expanding across a data matrix.

Test body
Data source
Case expansion
Execute per row
Fail row only
Matrix report
Reference

Key commands

pytest test_login.py -k 'mobile or desktop'

Run only the parametrized cases whose test IDs match mobile or desktop.

mvn test -Dtest=PasswordValidatorTest

Run the parameterized JUnit 5 test class with Maven.

npx playwright test --grep 'checkout'

Filter test titles; parametrized Playwright fixtures produce per-case titles.

python -m pytest --collect-only -q

Preview the exact list of expanded cases and their IDs before running.

python -m pytest --durations=5

Show the slowest cases, useful when one data row dominates runtime.

npx jest -t 'password'

Run test cases whose names contain 'password' when tests are named from data.

Hands-on

Examples

JUnit 5 parameterized test with a CSV source

JUnit 5 parameterized test with a CSV source
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvFileSource;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.*;
 
class PasswordValidatorTest {
 
  @ParameterizedTest
  @ValueSource(strings = { "longenough", "1ng3$Passw0rd!x" })
  void acceptsValidPasswords(String password) {
    assertTrue(PasswordValidator.isValid(password));
  }
 
  @ParameterizedTest
  @CsvFileSource(resources = "/invalid-passwords.csv", numLinesToSkip = 1)
  void rejectsInvalidPasswords(String password, String reason) {
    assertFalse(PasswordValidator.isValid(password), reason);
  }
}

The first test feeds inline values with @ValueSource; the second reads rows from invalid-passwords.csv, one argument per column, skipping the header. Both execute once per value. The second argument (reason) doubles as a JUnit assertion message, so a failure says exactly why the case should be invalid.

Real world

In production

  • Checkout and payment suites run one flow against dozens of price, currency and flag combinations.
  • Validation tests read from CSV owned by QA or product, adding edge cases without code changes.
  • API contract tests replay captured production request/response pairs as a regression matrix.
  • Cross-browser matrixes combine data-driven inputs with browser parameterization in one report.
Guidelines

Best practices

  • Keep the data file close to the test and document columns with a header row and a README note.
  • Name cases meaningfully: pytest.param(...) with ids, or JUnit @DisplayName templates per row.
  • Assert both happy paths and rejection paths - including an explicit reason string for each negative case.
  • Validate the data file early (column count, encoding) so a bad CSV fails fast, not mid-matrix.
  • Avoid thousands of near-identical rows; keep high-signal cases that target boundaries and edge states.
Watch out

Common mistakes

  • Putting the expected value inside the test body instead of the data, silently duplicating logic per row.
  • Using Excel or CSV with fragile formatting (trailing commas, quotes, encoding) that breaks row parsing.
  • Mixing cases that need different setups in one parametrized test, forcing awkward branching in the body.
  • Copying data files into multiple suites so a fix in one place does not propagate to the others.
Root cause analysis

Common issues

Error
Premature end of CSV data
Root cause

CSV file truncated or an encoding issue (BOM, non-UTF8).

Fix

Re-export the CSV as UTF-8 without BOM and verify the trailing newline.

Error
Parameterized argument [0] did not match count of 2 parameters
Root cause

Column count in the file does not match the method's parameter count.

Fix

Align CSV columns with the method signature, or use numLinesToSkip and check the header.

Error
Collected 0 items
Root cause

Data file path or glob is wrong, or the file filter excludes the rows.

Fix

Verify the resources path and file naming, and run --collect-only to see what matched.

When things break

Debugging tips

  • Run --collect-only to confirm every row expanded into the case you expect, including IDs.
  • Use -k or --grep to run a single row's case instead of the whole matrix.
  • Add the row's data to the failure message so CI logs identify the exact input that failed.
  • When a row fails, diff it against a passing row to isolate which input changed the outcome.
Interviews

Interview checkpoint

Practice

Hands-on lab

Validate Checkout Pricing from a CSV
40 min · intermediate
In progress
Objectives
  • Create a CSV data source with pricing rows
  • Write a parametrized test that reads the CSV
  • Add a reason column for failing-case clarity
  • Run the matrix and interpret per-case results

Validate Checkout Pricing from a CSV

40 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which pytest feature lets one test run against a matrix of inputs?

2

If one data row fails in a parametrized test, the rest of the rows still run and are reported.

3

You added 50 rows to a CSV, but pytest reports 'Collected 0 items'. What is the most likely cause?

Quick reference

Cheat sheet

Data-driven idioms
@pytest.mark.parametrize('a,b,expected', [(1,2,3)])pytest inline rows
ids=['case-a', 'case-b']Meaningful case names
@CsvFileSource(resources='/data.csv')JUnit CSV source
@ValueSource(strings={...})JUnit single-arg source
testData.forEach((row) => test(...))JS data-driven loop
python -m pytest --collect-onlyPreview expanded cases
Go deeper

Official documentation

pytest - Parametrization
docs
JUnit 5 - Parameterized Tests
docs
Playwright - Parametrize & fixtures
docs
TestNG - Data Providers
docs