Lesson 5 of 8 in Kubernetes

ConfigMaps and Secrets: settings outside the image

Core11 min

A ConfigMap holds non-secret settings and a Secret holds sensitive values, both as objects in the cluster. Pods read them as environment variables or as files, so the same image can run with different settings.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Pods: the smallest thing Kubernetes runs

The problem this solves

If you bake a database address or password into an image, you need a different image for every environment, and the password sits in the image forever. A container image should be the same in development and production.

Kubernetes keeps settings in separate objects and injects them when the Pod starts. Changing a setting then means changing an object, not rebuilding an image.

How it works

How settings reach a container
How settings reach a containerThe same image runs in a container, and a ConfigMap and a Secret are attached to it as environment variables or files.1 same image2 settings3 credentialsImagesame everywhereConfigMapplain settingsSecretsensitive valuesContainerreads env or files
  • Data
  • Control
  • Security
  • Runs code

The same image runs in a container, and a ConfigMap and a Secret are attached to it as environment variables or files.

  1. Image to Container: 1 same image
  2. ConfigMap to Container: 2 settings
  3. Secret to Container: 3 credentials
Who can read a Secret
Who can read a SecretA request for a Secret passes an RBAC permission check at the API server, and the value is read from etcd, where it is stored unencrypted unless you turn on encryption at rest.1 ask for it2 allowed?3 read thevalueYoukubectl get secretAPI serveretcdbase64 text by defaultRBAC checkwho may read it
  • People and clients
  • Control
  • Data
  • Security

A request for a Secret passes an RBAC permission check at the API server, and the value is read from etcd, where it is stored unencrypted unless you turn on encryption at rest.

  1. You to API server: 1 ask for it
  2. API server to RBAC check: 2 allowed?
  3. API server to etcd: 3 read the value

Both objects hold key-value pairs. You can inject a key as an environment variable, or mount the whole object as a directory where each key becomes a file. Environment variables are fixed when the container starts. Mounted files are refreshed after a delay when the object changes, but your app has to re-read them.

Be clear about what a Secret is. Its values are stored base64-encoded, which is a reversible text encoding and not encryption. By default they are saved unencrypted in etcd, and anyone who can read the Secret, or can create a Pod in that namespace, can read the value. Treat it as a way to keep credentials out of images and manifests, and add protection yourself.

Real protection means restricting who can read Secrets with RBAC, turning on encryption at rest for etcd, and for high-value credentials using an external secret manager. Never commit a Secret manifest to Git.

Try it yourself

Create a ConfigMap and a Secret, use both in a Pod, then read the Secret back to see how little protection base64 gives.

  1. 1
    Create the objects
    kubectl create configmap app-config --from-literal=GREETING=hello
    kubectl create secret generic app-secret --from-literal=DB_PASSWORD=s3cret
  2. 2
    Use them in a Pod
    cat > cfg.yaml <<'EOF'
    apiVersion: v1
    kind: Pod
    metadata:
      name: cfg-demo
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "echo greeting=$GREETING; cat /etc/creds/DB_PASSWORD; echo; sleep 3600"]
          env:
            - name: GREETING
              valueFrom:
                configMapKeyRef:
                  name: app-config
                  key: GREETING
          volumeMounts:
            - name: creds
              mountPath: /etc/creds
              readOnly: true
      volumes:
        - name: creds
          secret:
            secretName: app-secret
    EOF
    kubectl apply -f cfg.yaml
    kubectl wait --for=condition=Ready pod/cfg-demo --timeout=60s
    kubectl logs cfg-demo

    The setting arrives as an environment variable and the secret arrives as a file named after its key.

  3. 3
    Decode the Secret yourself
    kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}'
    kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

    You should see

    czNjcmV0
    s3cret

    The first line is how the value is stored. The second decodes it with one command. On older macOS use base64 -D.

  4. 4
    Clean up
    kubectl delete pod cfg-demo
    kubectl delete configmap app-config
    kubectl delete secret app-secret
    rm cfg.yaml

What breaks, and where to look

  • Pod STATUS is CreateContainerConfigError.

    WHY IT HAPPENS
    The Pod references a ConfigMap, Secret or key that does not exist, or a typo in the name. The container is never created.
    CHECK FIRST
    Run kubectl describe pod <pod-name> and read the Events, which name the missing object. Then kubectl get configmap,secret in the same namespace.
  • You edited the ConfigMap, but the running app still shows the old value.

    WHY IT HAPPENS
    Environment variables are copied when the container starts and never change afterwards. Files mounted with subPath also do not update.
    CHECK FIRST
    Restart the Pods with kubectl rollout restart deployment/<name>, or mount the ConfigMap as a directory and make the app re-read the file.

When not to use this

  • Do not rely on base64 as protection. If a value would hurt you in a leaked manifest, use encryption at rest, tight RBAC or an external secret manager, and keep it out of Git.
  • A ConfigMap is limited in size. Large files and binary data do not belong there. Put them in a volume or an object store.

Check yourself

Is a Kubernetes Secret encrypted by default?

No. Values are base64-encoded, which anyone can reverse, and they are stored unencrypted in etcd unless you enable encryption at rest.

Why did the app not see your new ConfigMap value?

Environment variables are set once at container start. Restart the Pod, or read mounted files and re-read them in the app.

What error shows when a Pod points to a ConfigMap that does not exist?

CreateContainerConfigError. The Events from kubectl describe pod name the missing object.

Lesson 74 of 92 in the DevOps path

Search

Search cheat sheets and projects