criticalAdvancedSymptom-first#security#secrets#layers#dockerfile#leak

Secret baked into Docker image — API key, password, or cert in layer

docker history shows a COPY or ARG containing a secret in an early layer

Likely source:security

Focused incident mode — strips away everything but this runbook.

Symptoms

  • docker history shows a COPY or ARG containing a secret in an early layer
  • docker inspect reveals environment variables with embedded API keys or passwords
  • Secret visible in docker image history even after subsequent RUN that tries to delete it

A secret in any layer persists forever in the image. docker history is public to anyone with pull access. Rotate any secret that was ever baked into a published image.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Image layer history with secrets

    docker history <IMAGE> --no-trunc | grep -i 'key\|secret\|password\|token'

  • Environment variables in image

    docker inspect <IMAGE> --format='{{json .Config.Env}}'

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • docker history shows ARG API_KEY=<secret> or ENV PASSWORD=<secret>
  • docker inspect shows secret in .Config.Env
  • ARG values are visible in docker history but ENV persists in inspect

Confirm before fixing

docker history <IMAGE> --no-trunc | grep -iE 'ARG|ENV'
READ

Find ARG/ENV lines with secrets

Fix

docker build --build-arg API_KEY=$API_KEY --secret id=api_key,src=$API_KEY_FILE .
CHANGE

Use BuildKit --secret mount instead of ARG for runtime secrets

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
docker history <IMAGE> --no-trunc | grep -iE 'ARG|ENV|COPY|SECRET|KEY|PASSWORD'
READ

Scan layers for secrets

2
docker inspect <IMAGE> --format='{{json .Config.Env}}' | python3 -m json.tool
READ

Check image environment variables

3
docker run --rm --entrypoint /bin/sh <IMAGE> -c 'find / -maxdepth 3 -name "*.pem" -o -name "*.key" -o -name ".env" 2>/dev/null'
READ

Search for secret files in the filesystem

4
docker build --secret id=api_key,src=$API_KEY_FILE .
CHANGE

Rebuild with BuildKit secrets mount to avoid layer leaks

Prevention

  • Use BuildKit --mount=type=secret for build-time secrets instead of ARG or ENV.
  • Pass runtime secrets via --env-file, docker secrets, or a mounted secret volume.
  • Never COPY .env, credentials, or key files into an image — use .dockerignore to exclude them.
  • Scan images with docker history, trivy, or hadolint to catch secrets before push.

Related runbooks

Same stack, same failure family — worth a skim.

Docker build sends massive context — slow builds and out of memorymedium

Sending build context to Docker daemon X.XXMB/GB takes minutes

DockerFoundation3 causes
Open runbook
apt-get install fails during docker build — network or DNS issue in buildermedium

docker build fails at RUN apt-get install with 'Could not resolve archive.ubuntu.com'

DockerIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.