Lesson 8 of 8 in CI/CD pipelines

Debugging failed pipelines

Core10 min

Debugging a pipeline means finding which step first returned a non-zero exit code and why, or finding why the run was never created at all. The log, the trigger filters and the workflow file location are the three places to look.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Writing a GitHub Actions workflow

The problem this solves

A red pipeline is a wall of log text, and the last line is usually just Process completed with exit code 1. The real cause sits higher up. Beginners often scroll the wrong way or re-run hoping it passes.

The second trap is the opposite: the workflow is valid YAML, you pushed it, and nothing happens. There is no red mark and no log, so it is easy to think GitHub is broken when a filter simply did not match.

How it works

Why a workflow does not start
Why a workflow does not startA pushed commit must pass four checks in order: the file is in the right folder, its event matches, its branch filter matches and its path filter matches, before a run is created.1 commitpushed2 file isread3 branchmatches?4 paths match?Pusha commit arrivesFile location.github/workflows/Branch filterbranches: [...]Path filterpaths: [...]Run createdvisible in Actions
  • People and clients
  • Control
  • Runs code

A pushed commit must pass four checks in order: the file is in the right folder, its event matches, its branch filter matches and its path filter matches, before a run is created.

  1. Push to File location: 1 commit pushed
  2. File location to Branch filter: 2 file is read
  3. Branch filter to Path filter: 3 branch matches?
  4. Path filter to Run created: 4 paths match?
How to investigate a red run
How to investigate a red runOpen the first red step, read the lines above the final error, re-run with debug logging if that is not enough, and reproduce the same command locally.1 open firstred step2 read linesabove3 need more detail4 try samecommandFailed runActions tabFirst red stepnot the last lineLines abovethe real errorDebug re-runmore detailRun it locallysame command
  • People and clients
  • Control
  • Data
  • Runs code

Open the first red step, read the lines above the final error, re-run with debug logging if that is not enough, and reproduce the same command locally.

  1. Failed run to First red step: 1 open first red step
  2. First red step to Lines above: 2 read lines above
  3. Lines above to Debug re-run: 3 need more detail
  4. Debug re-run to Run it locally: 4 try same command

For a failed run, open the run, find the first step marked red and expand it. Read upward from the end of that step for the actual error message. Later steps often fail only because the first one did.

If the log does not explain it, re-run the failed jobs with debug logging enabled. The re-run dialog has an option for it. Alternatively, set a secret or variable named ACTIONS_STEP_DEBUG to true, which adds more detail to the step logs, and ACTIONS_RUNNER_DEBUG to true, which adds runner diagnostic files to the downloadable log archive. Remove them when finished.

For a workflow that never starts, check in order: the file must be in .github/workflows/ and end in .yml or .yaml; the event must match; if you set both branches and paths, both must match, since the workflow runs only when both filters match. A skipped run also leaves required checks pending, which can block a pull request from merging.

Some triggers read the file from the default branch only. Scheduled runs use the default branch, and the Run workflow button for workflow_dispatch appears once the file exists on the default branch.

Try it yourself

Use the ci-demo repository. First make a workflow that is valid but never runs, then one that fails, then read both.

  1. 1
    Push a workflow that never runs
    cat > .github/workflows/silent.yml <<'EOF'
    name: silent
    on:
      push:
        branches: [release]
    permissions:
      contents: read
    jobs:
      hi:
        runs-on: ubuntu-latest
        timeout-minutes: 5
        steps:
          - run: echo "I ran"
    EOF
    git add .
    git commit -m "Add a silent workflow"
    git push

    The Actions tab shows runs for your other workflows, but none for silent, because you pushed to main and the filter says release. Change [release] to [main], commit and push, and it runs.

  2. 2
    Push a workflow that fails with a clear cause
    cat > .github/workflows/fails.yml <<'EOF'
    name: fails
    on:
      workflow_dispatch:
    permissions:
      contents: read
    jobs:
      broken:
        runs-on: ubuntu-latest
        timeout-minutes: 5
        steps:
          - run: echo "this step is fine"
          - name: Run a script that is missing
            run: ./deploy.sh
          - run: echo "this step is skipped"
    EOF
    git add .
    git commit -m "Add a failing workflow"
    git push
    gh workflow run fails.yml

    In the run, the second step is red and the third is skipped. Read the second step's log for the line explaining that ./deploy.sh could not be found. If you use gh, gh run view --log-failed prints only the failing steps.

  3. 3
    Re-run with debug logging

    On the failed run, choose Re-run failed jobs and tick the debug logging option in the dialog. Open the new log and compare how much more each step prints. Debug output can include more detail, so review a log before sharing it.

  4. 4
    Reproduce the failure locally
    cd ~/ci-demo
    ./deploy.sh; echo "exit code: $?"

    On your laptop you see the same message. The fastest fix loop is almost always to run the exact failing command on your own machine, in a folder with only the files git tracks.

What breaks, and where to look

  • You pushed a workflow and no run appears in the Actions tab.

    WHY IT HAPPENS
    A filter did not match. The usual suspects are branches not listing the branch you pushed to, paths not containing any file in the commit, the file placed outside .github/workflows/, or a manual trigger that is missing because the file is not on the default branch yet.
    CHECK FIRST
    Compare the on: block with what you pushed (branch name and changed files with git show --stat HEAD), then confirm the file path with git ls-files .github/workflows.
  • A step fails with ./deploy.sh: Permission denied and exit code 126, but the script runs on your laptop.

    WHY IT HAPPENS
    The file was committed without the executable bit, so the runner's checkout is not allowed to run it.
    CHECK FIRST
    Run git ls-files -s deploy.sh. A mode of 100644 is not executable. Fix it with git update-index --chmod=+x deploy.sh, commit and push.
  • Only the last line Process completed with exit code 1 is visible, and the cause is unclear.

    WHY IT HAPPENS
    That line only reports that a command returned non-zero. The reason was printed earlier in the same step, or in an earlier step that failed quietly.
    CHECK FIRST
    Scroll up inside the first red step. If there is still nothing, re-run with debug logging and add set -x at the top of the run: block to see each command as it executes.

When not to use this

  • Debug logging can print more of your environment into the log. Switch it off again, and never share a debug log publicly without reading it first.
  • Re-running a failed pipeline without reading the log is a way to hide a flaky test, not to fix it. If a re-run passes, find out what was different.

Check yourself

Where is the real error in a failed run?

In the first red step, usually above its last line, not in the final exit code 1 message.

A workflow has branches: [main] and paths: ['src/**']. When does a push run it?

Only when the push is to main and at least one changed file matches src/**. Both filters must match.

What are two ways to get more detail from a failed run?

Re-run with debug logging enabled, or set ACTIONS_STEP_DEBUG to true (and ACTIONS_RUNNER_DEBUG for runner diagnostics).

Lesson 54 of 92 in the DevOps path

Search

Search cheat sheets and projects