Lesson 2 of 7 in Security basics for DevOps

Keeping secrets out of git and images

Core12 minWritten for gitleaks 8.19 or newer (uses the `git` command; older versions use `detect`)

A secret is any value that proves identity or grants access, such as a password, API key, token or private key. Secrets management means keeping those values out of code, git history and images, delivering them to a program only when it runs, and replacing them on a schedule.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Shift left and think like a defender

The problem this solves

A secret committed to git is copied to every clone, every fork and every backup, and git keeps it in history after you delete the file. A secret baked into a container image sits in a layer that anyone who can pull the image can read. Public repositories are scanned by automated bots continuously, so an exposed key can be used within minutes.

Deleting the line is not a fix. Once a secret has been in a place others can read, treat it as known to others. The only reliable recovery is to revoke it and issue a new one, which is why you also need a way to rotate quickly.

How it works

The value lives in one place and is fetched at runtime
The value lives in one place and is fetched at runtimeA developer stores the secret value once in a secret store and commits only the name to git, the app fetches the value when it starts, and a rotation job replaces it on a schedule.1 storesvalue oncecommits names only2 fetches atstart3 replaces valueDeveloperowns the valueGit reponames only, no valuesSecret storevault or cloud serviceApp or CI jobruns with an identityRotationreplaces on a schedule
  • People and clients
  • Data
  • Security
  • Runs code
  • Control

A developer stores the secret value once in a secret store and commits only the name to git, the app fetches the value when it starts, and a rotation job replaces it on a schedule.

  1. Developer to Secret store: 1 stores value once
  2. Developer to Git repo: commits names only
  3. App or CI job to Secret store: 2 fetches at start
  4. Rotation to Secret store: 3 replaces value
What to do the moment a secret leaks
What to do the moment a secret leaksYou revoke the leaked secret first, issue a new one, check whether the old one was used, and only later clean the history.1 revokefirst2 replace3 tidy up4 look for misuseLeak foundscanner or reportRevoke keyturn the old one offIssue new keystore it properlyClean historyoptional, laterCheck logswas it used?
  • Other
  • Security
  • Control
  • Data
  • Runs code

You revoke the leaked secret first, issue a new one, check whether the old one was used, and only later clean the history.

  1. Leak found to Revoke key: 1 revoke first
  2. Revoke key to Issue new key: 2 replace
  3. Issue new key to Clean history: 3 tidy up
  4. Revoke key to Check logs: 4 look for misuse

Keep three things apart: the name of a secret, where its value is stored, and who may read it. Git and your Dockerfile hold only the name, such as DATABASE_PASSWORD. A secret store, which can be a cloud service or a tool such as HashiCorp Vault, holds the value. The running app proves who it is and receives the value at start time.

An environment variable is a fine delivery method for the value, but it is not storage. Where the variable gets its value from is the question that matters. A .env file on a laptop is acceptable if it is listed in .gitignore. A .env file copied into an image is not.

The second diagram gives the order of work after a leak. Revoke first, because every minute the old key works is a minute someone else can use it. Rewriting git history comes last, since people may already have copied the old commits.

Try it yourself

All values below are made up and refer to no real service. Everything happens in a throwaway folder, and the image step needs Docker. Install gitleaks from its official README if you want the scanning step.

  1. 1
    Create a repo and check that `.env` is ignored
    mkdir -p ~/secrets-demo && cd ~/secrets-demo && git init
    git config user.name demo && git config user.email demo@example.invalid
    echo 'API_KEY="Xk9fQ2vLm8Zp4RtYw7NcB3aHd6JsE1uG"' > .env
    git check-ignore -v .env; echo "exit code: $?"

    git check-ignore prints the matching rule when a path is ignored. No output and exit code 1 means git would happily commit this file. Fix it with echo .env >> .gitignore and run the check again.

  2. 2
    See that deleting a secret does not remove it from history
    echo 'API_KEY="Xk9fQ2vLm8Zp4RtYw7NcB3aHd6JsE1uG"' > config.txt
    git add config.txt && git commit -m 'add config'
    git rm config.txt && git commit -m 'remove config'
    git log --all --oneline -p -S'Xk9fQ2vL'

    The file is gone from the latest commit, yet -S finds the commit that added the value because git still stores it. This is why a leaked secret has to be revoked, not just removed.

  3. 3
    Scan the repo history with gitleaks
    gitleaks git -v .
    echo "exit code: $?"

    Exit code 1 means a leak was reported and 0 means none was found. The value above is shaped like a generic API key, so you should see a finding that names the commit. If nothing is reported, the detection rules did not match this made-up value, which is a reminder that scanners reduce risk and do not remove it.

  4. 4
    Watch a secret survive inside an image layer
    docker build -t leaky-demo - <<'EOF'
    FROM alpine:3.22
    RUN echo 'DB_PASSWORD=fake-not-real' > /app.env
    RUN rm /app.env
    EOF
    docker run --rm leaky-demo ls /app.env
    docker history leaky-demo

    The ls fails because the file is deleted in the final filesystem. The history output still lists the RUN echo ... command text, and the file sits in the earlier layer. Clean up with docker rmi leaky-demo, then rm -r ~/secrets-demo.

What breaks, and where to look

  • You deleted the file and pushed, but a scanner or a colleague still finds the key in the repo.

    WHY IT HAPPENS
    Git stores every commit. The key is in the history, in other clones and possibly in forks, so deleting it from the latest version changes nothing about exposure.
    CHECK FIRST
    Revoke the key at its provider first. Then find what contains it with git log --all -p -S'<part-of-key>' and decide whether rewriting history is worth the disruption.
  • .env shows up under Untracked files in git status, and you almost committed it.

    WHY IT HAPPENS
    The file is not listed in .gitignore, so git treats it like any other file.
    CHECK FIRST
    Run git check-ignore -v .env. If it prints nothing, add .env to .gitignore. If the file was already committed earlier, adding it to .gitignore does not untrack it, so also run git rm --cached .env.
  • A secret appears in CI job output, even though you stored it as a masked secret.

    WHY IT HAPPENS
    Masking matches the exact stored value. A transformed copy, such as a base64 or URL encoded version, or one printed by a debug flag, is not recognised.
    CHECK FIRST
    Search the job log for part of the value, and also its encoded forms. Delete the log, rotate the secret, and stop echoing variables in scripts.

When not to use this

  • A shared password manager is enough for a small team's human credentials. A dedicated secret store pays off when programs, many environments or audit trails are involved.
  • A scanner finds secrets that look like secrets. It cannot catch every custom token format, so combine scanning with the habit of never pasting credentials into files.

Check yourself

Why is deleting a committed secret not enough?

Git keeps it in history, and anyone who cloned or forked the repo already has a copy. The secret must be revoked and replaced.

What should you do first when a key leaks?

Revoke it so it stops working, then issue and store a new one. Cleaning history and writing the report come after.

Why can a secret still be read from an image after a later RUN rm?

Each Dockerfile instruction creates a layer, and an earlier layer still contains the file even when a later layer hides it.

Lesson 87 of 92 in the DevOps path

Search

Search cheat sheets and projects