CI/CD
Intermediate 16 min read CI/CD

Testing in the Pipeline

Pipelines turn testing into a gate. Learn the test pyramid — unit, integration, end-to-end — how to enforce coverage, how to handle flaky tests without training teams to ignore red, and when contract testing beats brittle e2e suites.

unit testintegration teste2e testtest pyramidcoverageflaky testcontract testingtest gate
Prerequisites: Designing Pipelines
Introduction

What is this?

Automated testing in pipelines runs suites at every commit and fails the build on any regression. The test pyramid puts many fast unit tests at the base, fewer integration tests in the middle, and a small number of end-to-end tests at the top. Coverage gates enforce a minimum line coverage, and contract testing verifies API compatibility between services cheaply.

Motivation

Why do you need it?

  • A failing test in the pipeline is the earliest, cheapest place to catch a regression.
  • Fast tests give feedback within minutes, keeping developers productive.
  • The test pyramid balances speed against confidence — too many e2e tests make the suite slow and brittle.
  • Coverage gates prevent silent regressions where new code ships entirely untested.
  • Contract tests catch breaking API changes between services without running full environments.
Think about it

Mental model

Imagine

A quality inspector in a factory

Think of

The test pyramid

The inspector checks every bolt quickly (unit tests), samples how parts fit together (integration tests), and only at the end tests the finished product once (e2e). If you only tested finished products you would find defects far too late and at far too high a cost.

Mechanics

How it works

  • Unit tests exercise a single function or class in isolation, running in milliseconds across thousands of cases.
  • Integration tests verify components cooperate — database, cache, message queue — using real or lightweight substitutes.
  • End-to-end tests drive the whole system through its real interfaces, proving the user-facing flow works.
  • A coverage tool reports what fraction of lines are executed and fails the build below a configured threshold.
  • Contract tests pin the shape of an API so producer and consumer teams detect breaking changes without a full environment.

The test pyramid: many fast unit tests, fewer integration tests, a few e2e tests.

Unit tests
Integration tests
End-to-end tests
Coverage gate
Pipeline result
Reference

Key commands

npm test

Run the unit test suite defined in package.json.

pytest --cov=src --cov-fail-under=90

Run Python tests and fail if line coverage drops below 90%.

go test ./... -race

Run all Go tests with the race detector enabled.

npx playwright test

Run the end-to-end browser test suite.

npm run test:contract

Verify service-to-service API contracts with the contract test suite.

Hands-on

Examples

A coverage-gated test run in CI

A coverage-gated test run in CI
$ npm test -- --coverage
PASS src/calc.test.js
  ✓ add returns the sum (2 ms)
  ✓ subtract returns the difference (1 ms)
 
Coverage: 92.4% statements | 90% branches | 95% functions | 91% lines
ERROR: Coverage for lines (91%) does not meet global threshold (95%)
npm ERR! Test run failed — build stopped

The suite passes, but the coverage gate fails because line coverage is 91% against a 95% threshold. The pipeline stops and the commit is blocked until the missing paths are covered.

Real world

In production

  • Pull requests run unit and integration suites; merges to main run the slower e2e suite.
  • Coverage gates in CI prevent untested new code from shipping, often with patch-level thresholds.
  • Contract tests run in each service's pipeline so producers and consumers find breaking API changes before deploy.
  • Flaky tests are quarantined and auto-retried once; persistent flakes block the suite until fixed.
  • Slow test tiers move to nightly runs so the commit path stays fast.
Guidelines

Best practices

  • Keep unit tests hermetic — no network, no shared state, no sleeping.
  • Run the fast suite on every commit and reserve e2e tests for merges and releases.
  • Enforce coverage as a gate but set realistic thresholds that reflect the codebase.
  • Treat flaky tests as bugs: quarantine them immediately and fix or delete them.
  • Give tests deterministic fixtures and random seed handling so results never depend on ordering.
Watch out

Common mistakes

  • Building a huge e2e suite that takes an hour and fails randomly — teams stop watching it.
  • Gaming coverage with tests that assert nothing, inflating the percentage without protecting behavior.
  • Ignoring flaky tests until everyone habitually clicks rerun.
  • Letting tests touch production services or real user data.
Root cause analysis

Common issues

Error
Test passes locally but fails only in CI
Root cause

Environment drift — different Node/Python version, missing fixture, or network access in CI.

Fix

Match the CI runtime locally, pin versions in lockfiles, and make tests hermetic so they need no network.

Error
Coverage gate fails despite tests passing
Root cause

The coverage threshold is set above the suite's actual reach, or new code is untested.

Fix

Inspect the coverage report, add tests for the uncovered lines, then confirm the percentage passes.

Error
A test is intermittently red (flaky)
Root cause

Race conditions, timeouts, shared state or test ordering.

Fix

Quarantine the test, reproduce with repeated runs and a fixed seed, and fix the root cause rather than rerunning blindly.

When things break

Debugging tips

  • Rerun the failing test locally in isolation — if it passes, suspect ordering or shared state.
  • Use test filters to run only the failing module and check the traceback before blaming the suite.
  • Inspect the coverage report's uncovered lines to see exactly which branch the gate flagged.
  • Add a marker to intentionally flaky tests so they are excluded from blocking runs while being fixed.
Interviews

Interview checkpoint

Practice

Hands-on lab

Gate a Build on Tests and Coverage
30 min · intermediate
In progress
Objectives
  • Write source code and matching assertions by hand
  • Run test gates using shell exit codes
  • Enforce a coverage-style minimum with a count check
  • Simulate a flaky test and handle it deterministically

Gate a Build on Tests and Coverage

30 min · intermediate

0/6 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which tests belong at the base of the test pyramid?

2

A test that passes locally but fails in CI indicates a difference between the environments.

3

A suite is 90% covered by assertions but the build just failed on a regression in an untested edge case. What is the best next step?

Quick reference

Cheat sheet

Test tiers
unitFast, isolated, run on every commit
integrationComponents against real dependencies
e2eFull system, reserved for merges/releases
contractAPI compatibility between services
coverage gateFail below a line-coverage threshold
Common commands
npm testJS unit suite
pytest --cov=src --cov-fail-under=90Python + coverage gate
go test ./... -raceGo with race detection
npx playwright testBrowser e2e
jest --runInBandSerialize tests to debug ordering
Go deeper

Official documentation

Istanbul/nyc coverage tool
docs
Jest — testing framework
docs
Playwright end-to-end testing
docs
Practical Test Pyramid
docs