Lesson 1 of 8 in CI/CD pipelines
The anatomy of a pipeline
A pipeline is a set of automated steps that runs whenever something happens to your code, such as a push. Continuous integration (CI) means building and testing every change. Continuous delivery or deployment (CD) means carrying the tested result toward users.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Git and collaboration, Python and scripting for ops
The problem this solves
Without a pipeline, the checks live in people's heads. One developer remembers to run the tests, another forgets, and a third builds the release on a laptop that has a slightly different tool version. A change that was never tested ends up in front of users.
A pipeline writes those steps down as code and runs them on a clean machine every time. The result is the same for everyone, it is recorded, and a failure stops the change before it spreads.
GitHub Actions is used for the examples here. GitLab CI and Jenkins use the same ideas under different names, so what you learn transfers.
How it works
- People and clients
- Data
- Control
- Runs code
A developer pushes a commit, the repository sends an event, the pipeline hands a job to a runner, the runner stores its output and reports pass or fail back to the repository.
- Developer to Repository: 1 push
- Repository to Pipeline: 2 trigger event
- Pipeline to Runner: 3 assign job
- Runner to Artifacts: 4 upload output
- Runner to Repository: 5 report status
- Control
- Runs code
- Data
The workflow starts the build job, the test job waits for the build job, the deploy job waits for the test job, and files travel between jobs through stored artifacts.
- Workflow to Build job: 1 starts
- Build job to Test job: 2 needs: build
- Test job to Deploy job: 3 needs: test
- Build job to Artifact: upload
- Artifact to Test job: download
A trigger is the event that starts a run, for example a push, a pull request, a schedule or a button press. The pipeline reads its definition from a file in your repository, so the process is versioned along with the code.
A pipeline is split into jobs. Each job is a group of steps that runs on one runner, which is a machine (usually a virtual machine or container) that starts clean and is thrown away afterwards. A step is one command or one reusable action.
Because each job starts on a clean runner, nothing carries over by accident. Files that a later job needs must be saved as artifacts and downloaded again. Jobs run in parallel unless you say one must wait for another.
Every step returns an exit code. The first non-zero code fails the step, the job stops, and the failure is reported to the repository. That is the same exit-code contract you use in shell scripts.
Try it yourself
Before using any CI service, write the pipeline as a shell script on your laptop. This is a Linux or macOS shell and only touches a throwaway folder.
- 1Write a three-step pipeline as a script
mkdir -p ~/pipeline-demo && cd ~/pipeline-demo cat > ci.sh <<'EOF' set -euo pipefail echo "step 1: build" mkdir -p dist echo "hello" > dist/app.txt echo "step 2: test" [ "${BREAK:-0}" = 0 ] || { echo "test failed"; exit 1; } test -s dist/app.txt echo "step 3: package" tar -czf app.tgz dist echo "pipeline passed" EOF bash ci.sh; echo "exit code: $?"You should see
step 1: build step 2: test step 3: package pipeline passed exit code: 0
set -estops the script at the first failing command. TheBREAKvariable lets you make the test step fail on purpose in the next step. - 2Make a step fail and watch the pipeline stop
rm -rf dist app.tgz BREAK=1 bash ci.sh; echo "exit code: $?" ls
You should see
step 1: build step 2: test test failed exit code: 1
Step 3 never ran and no
app.tgzwas produced. A CI service does exactly this and adds a clean machine, a trigger, stored logs and stored artifacts around it. - 3Map the script to pipeline words
Running
bash ci.shwas the trigger. The whole script was one job. Eachechosection was a step. Your laptop was the runner, andapp.tgzwas the artifact. The next concepts replace each of those with a hosted equivalent.
What breaks, and where to look
The pipeline passes on your laptop but fails on the runner with
command not foundorNo such file or directory.- WHY IT HAPPENS
- The runner is a fresh machine. It has none of your installed tools, none of your local files and, until a step fetches it, not even your code.
- CHECK FIRST
- Read the first red step in the run log. Ask what that step assumed was already there (a tool version, a file, the checked-out repository) and add a step that provides it.
The run is green, but a bug still reached production.
- WHY IT HAPPENS
- A step returned exit code 0 without checking anything. Common causes are a test command followed by
|| true, a test runner that found zero tests, or a script withoutset -e. - CHECK FIRST
- Search the workflow for
|| trueandcontinue-on-error, then open the test step log and confirm it reports a non-zero number of tests that ran.
When not to use this
- A solo script you run once a month does not need a pipeline. The value appears when several people push changes and you need the same checks every time.
- A pipeline only automates the checks you write. It does not replace code review or good tests.
Check yourself
Why must files be saved as artifacts to move from one job to the next?
Each job runs on its own clean runner, so nothing from an earlier job's disk is still there.
What decides whether a step passed or failed?
Its exit code. Zero passes, any other number fails the step and stops the job.
What is a trigger?
The event that starts a run, such as a push, a pull request, a schedule or a manual button.
Keep going
Lesson 47 of 92 in the DevOps path