Lesson 8 of 8 in CI/CD pipelines
Debugging failed pipelines
Debugging a pipeline means finding which step first returned a non-zero exit code and why, or finding why the run was never created at all. The log, the trigger filters and the workflow file location are the three places to look.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Writing a GitHub Actions workflow
The problem this solves
A red pipeline is a wall of log text, and the last line is usually just Process completed with exit code 1. The real cause sits higher up. Beginners often scroll the wrong way or re-run hoping it passes.
The second trap is the opposite: the workflow is valid YAML, you pushed it, and nothing happens. There is no red mark and no log, so it is easy to think GitHub is broken when a filter simply did not match.
How it works
- People and clients
- Control
- Runs code
A pushed commit must pass four checks in order: the file is in the right folder, its event matches, its branch filter matches and its path filter matches, before a run is created.
- Push to File location: 1 commit pushed
- File location to Branch filter: 2 file is read
- Branch filter to Path filter: 3 branch matches?
- Path filter to Run created: 4 paths match?
- People and clients
- Control
- Data
- Runs code
Open the first red step, read the lines above the final error, re-run with debug logging if that is not enough, and reproduce the same command locally.
- Failed run to First red step: 1 open first red step
- First red step to Lines above: 2 read lines above
- Lines above to Debug re-run: 3 need more detail
- Debug re-run to Run it locally: 4 try same command
For a failed run, open the run, find the first step marked red and expand it. Read upward from the end of that step for the actual error message. Later steps often fail only because the first one did.
If the log does not explain it, re-run the failed jobs with debug logging enabled. The re-run dialog has an option for it. Alternatively, set a secret or variable named ACTIONS_STEP_DEBUG to true, which adds more detail to the step logs, and ACTIONS_RUNNER_DEBUG to true, which adds runner diagnostic files to the downloadable log archive. Remove them when finished.
For a workflow that never starts, check in order: the file must be in .github/workflows/ and end in .yml or .yaml; the event must match; if you set both branches and paths, both must match, since the workflow runs only when both filters match. A skipped run also leaves required checks pending, which can block a pull request from merging.
Some triggers read the file from the default branch only. Scheduled runs use the default branch, and the Run workflow button for workflow_dispatch appears once the file exists on the default branch.
Try it yourself
Use the ci-demo repository. First make a workflow that is valid but never runs, then one that fails, then read both.
- 1Push a workflow that never runs
cat > .github/workflows/silent.yml <<'EOF' name: silent on: push: branches: [release] permissions: contents: read jobs: hi: runs-on: ubuntu-latest timeout-minutes: 5 steps: - run: echo "I ran" EOF git add . git commit -m "Add a silent workflow" git pushThe Actions tab shows runs for your other workflows, but none for
silent, because you pushed tomainand the filter saysrelease. Change[release]to[main], commit and push, and it runs. - 2Push a workflow that fails with a clear cause
cat > .github/workflows/fails.yml <<'EOF' name: fails on: workflow_dispatch: permissions: contents: read jobs: broken: runs-on: ubuntu-latest timeout-minutes: 5 steps: - run: echo "this step is fine" - name: Run a script that is missing run: ./deploy.sh - run: echo "this step is skipped" EOF git add . git commit -m "Add a failing workflow" git push gh workflow run fails.ymlIn the run, the second step is red and the third is skipped. Read the second step's log for the line explaining that
./deploy.shcould not be found. If you usegh,gh run view --log-failedprints only the failing steps. - 3Re-run with debug logging
On the failed run, choose Re-run failed jobs and tick the debug logging option in the dialog. Open the new log and compare how much more each step prints. Debug output can include more detail, so review a log before sharing it.
- 4Reproduce the failure locally
cd ~/ci-demo ./deploy.sh; echo "exit code: $?"
On your laptop you see the same message. The fastest fix loop is almost always to run the exact failing command on your own machine, in a folder with only the files git tracks.
What breaks, and where to look
You pushed a workflow and no run appears in the Actions tab.
- WHY IT HAPPENS
- A filter did not match. The usual suspects are
branchesnot listing the branch you pushed to,pathsnot containing any file in the commit, the file placed outside.github/workflows/, or a manual trigger that is missing because the file is not on the default branch yet. - CHECK FIRST
- Compare the
on:block with what you pushed (branch name and changed files withgit show --stat HEAD), then confirm the file path withgit ls-files .github/workflows.
A step fails with
./deploy.sh: Permission deniedand exit code 126, but the script runs on your laptop.- WHY IT HAPPENS
- The file was committed without the executable bit, so the runner's checkout is not allowed to run it.
- CHECK FIRST
- Run
git ls-files -s deploy.sh. A mode of100644is not executable. Fix it withgit update-index --chmod=+x deploy.sh, commit and push.
Only the last line
Process completed with exit code 1is visible, and the cause is unclear.- WHY IT HAPPENS
- That line only reports that a command returned non-zero. The reason was printed earlier in the same step, or in an earlier step that failed quietly.
- CHECK FIRST
- Scroll up inside the first red step. If there is still nothing, re-run with debug logging and add
set -xat the top of therun:block to see each command as it executes.
When not to use this
- Debug logging can print more of your environment into the log. Switch it off again, and never share a debug log publicly without reading it first.
- Re-running a failed pipeline without reading the log is a way to hide a flaky test, not to fix it. If a re-run passes, find out what was different.
Check yourself
Where is the real error in a failed run?
In the first red step, usually above its last line, not in the final exit code 1 message.
A workflow has branches: [main] and paths: ['src/**']. When does a push run it?
Only when the push is to main and at least one changed file matches src/**. Both filters must match.
What are two ways to get more detail from a failed run?
Re-run with debug logging enabled, or set ACTIONS_STEP_DEBUG to true (and ACTIONS_RUNNER_DEBUG for runner diagnostics).
Keep going
Lesson 54 of 92 in the DevOps path