Lesson 8 of 8 in Containers with Docker

Small and safe images

Advanced12 min

A small, safe image contains only what the app needs at run time, runs as a non-root user, and keeps secrets out. Multi-stage builds compile in one stage and copy only the result into a clean final stage.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Writing a Dockerfile

The problem this solves

The tools you need to build software, such as compilers and package managers, are not needed to run it. If they ship in the final image, the image is larger to pull, slower to start and has more software that can contain vulnerabilities.

Two more defaults work against you. Containers run as root unless told otherwise, so a flaw in your app starts with elevated rights inside the container. And COPY . . copies everything in the folder, including things like a .env file or .git history.

How it works

Build in one stage, ship from another
Build in one stage, ship from anotherThe build stage compiles the program with all its tools, only the finished binary is copied into the small final stage, and the build tools are left behind.1 produces2 COPY --from3 discardedBuild stagecompiler and sourceBuilt programone small fileFinal stagesmall base, non-rootTools left behindnot in final image
  • Runs code
  • Data
  • Other

The build stage compiles the program with all its tools, only the finished binary is copied into the small final stage, and the build tools are left behind.

  1. Build stage to Built program: 1 produces
  2. Built program to Final stage: 2 COPY --from
  3. Build stage to Tools left behind: 3 discarded

In a multi-stage Dockerfile each FROM starts a new stage. You name one with FROM golang:1 AS build, do the heavy work in it, and later write COPY --from=build /out/hello /hello in the final stage to take just the output. Only the last stage becomes the image you run.

USER sets which user the later steps and the running container use. Create an unprivileged user in the image and switch to it, so the app does not run as root.

A .dockerignore file in the build context lists paths that are not sent to the builder, such as .git, node_modules and .env. It keeps the context small, speeds up builds and stops secrets from reaching a COPY.

Try it yourself

This builds a tiny web server written in Go. You do not need Go installed, because the build stage runs inside Docker. Use a throwaway folder.

  1. 1
    Create the program
    mkdir -p ~/learn-small && cd ~/learn-small
    cat > go.mod <<'EOF'
    module hello
    
    go 1.22
    EOF
    cat > main.go <<'EOF'
    package main
    
    import (
    	"fmt"
    	"net/http"
    )
    
    func main() {
    	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    		fmt.Fprintln(w, "hello from a small image")
    	})
    	http.ListenAndServe(":8080", nil)
    }
    EOF
  2. 2
    Write a multi-stage Dockerfile and a .dockerignore
    cat > Dockerfile <<'EOF'
    FROM golang:1 AS build
    WORKDIR /src
    COPY go.mod main.go ./
    RUN CGO_ENABLED=0 go build -o /out/hello .
    
    FROM alpine:3
    RUN adduser -D -u 10001 app
    USER app
    COPY --from=build /out/hello /hello
    EXPOSE 8080
    CMD ["/hello"]
    EOF
    printf '.git\nnode_modules\n.env\n' > .dockerignore

    CGO_ENABLED=0 makes a self-contained binary that does not depend on the build image's C library. adduser -D creates a user with no password.

  3. 3
    Build, run and compare sizes
    docker build -t hello-small .
    docker run -d --name hello-small -p 8080:8080 hello-small
    curl http://localhost:8080
    docker image ls

    Compare the SIZE of hello-small with the golang image used to build it. The compiler never reached the final image.

  4. 4
    Confirm it is not root
    docker run --rm --entrypoint whoami hello-small

    You should see

    app
  5. 5
    Clean up
    docker rm -f hello-small
    docker rmi hello-small
    cd ~ && rm -rf ~/learn-small

    The pulled golang:1 and alpine:3 base images remain. Remove them with docker rmi if you want the space back.

What breaks, and where to look

  • exec /hello: no such file or directory (older versions: standard_init_linux.go ... exec user process caused: no such file or directory) although the file is there.

    WHY IT HAPPENS
    The binary was built against a different C library than the final image has, so its loader is missing. A binary built on a glibc image often will not run on Alpine, which uses musl.
    CHECK FIRST
    Build a static binary, as with CGO_ENABLED=0 for Go, or use a final base that matches the build image. Run docker run --rm --entrypoint sh <image> -c 'ls -l /hello' to confirm the file exists.
  • After adding USER app, the app fails with permission denied when writing a log file or cache folder.

    WHY IT HAPPENS
    The folder was created by root during the build, so the unprivileged user cannot write to it.
    CHECK FIRST
    Run docker run --rm --entrypoint ls <image> -ld /path to see the owner. Create the folder and set ownership before switching users, or use COPY --chown=app:app.
  • A secret such as a .env file or private key is found inside a built image.

    WHY IT HAPPENS
    COPY . . sent everything in the build context, because no .dockerignore excluded it. Deleting the file in a later layer does not remove it from the earlier one.
    CHECK FIRST
    Run docker run --rm --entrypoint ls <image> -a /app to look. Add the file to .dockerignore, rebuild, and treat any secret that already got into a pushed image as leaked and rotate it.

When not to use this

  • Do not chase the smallest possible image at the cost of debuggability. A very minimal image may have no shell, which makes docker exec useless. Start with a reasonable small base.
  • A small image is not automatically secure. Scan images for known vulnerabilities and keep base images updated.

Check yourself

What does a multi-stage build let you leave out of the final image?

The build tools, such as compilers and package managers, and the source files that were only needed to produce the output.

Why run the app as a non-root user?

If the app is compromised, the attacker has the rights of an unprivileged user instead of root inside the container.

Why is deleting a secret file in a later Dockerfile step not enough?

The earlier layer that copied it still contains the file, and anyone with the image can extract it.

Lesson 46 of 92 in the DevOps path

Search

Search cheat sheets and projects