CI/CD
Intermediate 15 min read CI/CD

Designing Pipelines

Good pipelines are staged, cached, gated and secure. Learn how to split work into parallel and sequential stages, model dev/staging/prod environments, cache dependencies, inject secrets safely and handle failures with retries and approvals.

stagesparallelsequentialenvironmentdevstagingprodcachingsecretsfailure handlingapprovals
Prerequisites: CI/CD Fundamentals
Introduction

What is this?

Pipeline design is the discipline of structuring automation so it is fast, reliable and safe. Stages run sequentially when they depend on each other and in parallel when they do not. Environments model the journey of an artifact from dev through staging to production, each with its own secrets, approvals and risk profile.

Motivation

Why do you need it?

  • A well-staged pipeline gives feedback in seconds and keeps expensive steps for the right moments.
  • Parallel stages cut wall-clock time dramatically, which keeps developers waiting on green.
  • Separate environments catch integration issues before production and hold back risky changes.
  • Caching turns a ten-minute dependency install into a ten-second one.
  • Explicit failure handling and approvals make pipelines predictable instead of surprising.
Think about it

Mental model

Imagine

A house build schedule

Think of

A designed pipeline

The foundation must finish before framing starts (sequential), but plumbing and electrical can proceed in parallel once walls exist. You inspect work before handing over keys (approval). If anything fails inspection you fix it before moving to the next trade. Pipelines encode exactly this dependency graph.

Mechanics

How it works

  • Stages with dependencies run sequentially; independent stages run in parallel on separate workers.
  • Each stage runs in a clean environment with the dependencies it needs, often pulled from a shared cache.
  • Artifacts and caches are keyed by inputs (branch, hash, dependency lockfile) so identical runs reuse work.
  • Secrets are injected as environment variables from the platform's secret store, never written into the repo.
  • Failure handling is explicit: retries for transient errors, approvals and manual gates for promotions, and automatic rollbacks on deploy failure.

A three-environment pipeline with parallel test jobs and a gated promotion.

Commit
Unit tests
Integration + build
Deploy to dev
Approve promotion
Deploy to prod
Reference

Key commands

npm ci

Install dependencies exactly from the lockfile — reproducible and cacheable.

docker build --cache-from myapp:cache -t myapp:$CI_COMMIT_SHA .

Reuse cached layers from a previous build to speed up image builds.

curl -s -X POST '$DEPLOY_WEBHOOK'

Trigger an external deployment service from a stage.

env | grep -i secret

Confirm the injected secret is available as an environment variable.

git diff --name-only HEAD~1 HEAD

Compute which files changed to decide if a stage should run at all.

Hands-on

Examples

A GitLab CI pipeline with stages and a manual gate

A GitLab CI pipeline with stages and a manual gate
stages:
  - test
  - build
  - deploy
 
unit-tests:
  stage: test
  script:
    - npm ci
    - npm test
 
e2e-tests:
  stage: test
  script:
    - npm ci
    - npm run test:e2e
 
build-image:
  stage: build
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
  artifacts:
    paths: [image.txt]
 
deploy-prod:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  when: manual
  environment: production
  script:
    - ./deploy.sh

The two test jobs run in parallel on different runners. The build waits for tests. Production deployment only happens on main, and only when a human clicks the manual button — an approval gate before the riskiest step.

Real world

In production

  • Merge pipelines run lint and fast tests on every PR; main pipelines add full test suites and build.
  • Promotions are gated: automatic to dev, approval-gated to staging, change-controlled to production.
  • Dependency caches keyed by lockfile hash keep installs fast without serving stale packages.
  • Failures are retried for transient causes and alerted on otherwise, so flaky infrastructure does not block developers.
  • Environment-scoped secrets mean staging credentials never reach production jobs.
Guidelines

Best practices

  • Design stages so failures fail fast: cheap checks first, expensive checks later.
  • Run independent jobs in parallel and keep jobs small enough to isolate failures.
  • Key caches precisely (lockfile hash, OS, toolchain) so invalidation is automatic.
  • Never write secrets into code or logs — use the platform's secret store and masked variables.
  • Model environments explicitly with clear approval and rollback rules for each promotion.
Watch out

Common mistakes

  • Chaining every step into one giant job, so a lint failure reruns a full deploy sequence.
  • Cache poisoning: a stale, over-broad cache key silently serves outdated dependencies.
  • Committing secrets to the repo 'just for the pipeline to work'.
  • Repeating the same slow stage in every environment instead of promoting one artifact.
Root cause analysis

Common issues

Error
A parallel job fails only when run together with others
Root cause

Shared resources or global state: port conflicts, race conditions, or a shared cache.

Fix

Isolate jobs by runner or use unique ports/branches; serialize the racy job or make it re-run independently.

Error
Retry succeeded after the pipeline was already marked failed
Root cause

Retries were not configured, so a transient network or agent failure failed the job permanently.

Fix

Enable automatic retry for transient failure classes (runner timeouts, network) and keep permanent failures manual.

Error
Manual approval button never appears in the UI
Root cause

The rule or environment configuration prevents the job from being offered for manual run.

Fix

Verify the job's rules and conditions match the branch and that the environment exists in project settings.

When things break

Debugging tips

  • Inspect the pipeline DAG view to see exactly which job blocked which.
  • Rerun a single failed job instead of the whole pipeline to iterate faster.
  • Add a debug stage that prints environment and dependency versions when output is ambiguous.
  • Compare cache keys and artifact hashes between a passing and failing run to spot staleness.
Interviews

Interview checkpoint

Practice

Hands-on lab

Design a Three-Environment Promotion Path
30 min · intermediate
In progress
Objectives
  • Create per-environment configuration directories
  • Promote a config from dev to staging with a gate
  • Verify configuration integrity before promotion
  • Scan for accidentally committed secrets

Design a Three-Environment Promotion Path

30 min · intermediate

0/6 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which two stages could safely run in parallel?

2

Production credentials can be shared with the development environment to simplify pipeline configuration.

3

A pipeline's cache serves dependencies from a week ago and builds keep failing on a bug fixed yesterday. What is the most likely cause?

Quick reference

Cheat sheet

Design decisions
sequentialDependent stages, one after the other
parallelIndependent jobs on separate runners
cache keyLockfile hash + OS + toolchain
approval gateManual confirmation before promotion
retryRe-run transient failures automatically
Environment roles
devAutomatic deploys, relaxed rules
stagingMirrors prod, approval-gated
prodChange-controlled, monitored, reversible
secretsScoped per environment, masked
rollbackRevert to the previous artifact
Go deeper

Official documentation

GitLab CI/CD pipeline configuration
docs
GitHub Actions — using jobs
docs
The Twelve-Factor App — config
docs
OWASP Secrets Management Cheat Sheet
docs