Lesson 5 of 8 in Kubernetes
ConfigMaps and Secrets: settings outside the image
A ConfigMap holds non-secret settings and a Secret holds sensitive values, both as objects in the cluster. Pods read them as environment variables or as files, so the same image can run with different settings.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Pods: the smallest thing Kubernetes runs
The problem this solves
If you bake a database address or password into an image, you need a different image for every environment, and the password sits in the image forever. A container image should be the same in development and production.
Kubernetes keeps settings in separate objects and injects them when the Pod starts. Changing a setting then means changing an object, not rebuilding an image.
How it works
- Data
- Control
- Security
- Runs code
The same image runs in a container, and a ConfigMap and a Secret are attached to it as environment variables or files.
- Image to Container: 1 same image
- ConfigMap to Container: 2 settings
- Secret to Container: 3 credentials
- People and clients
- Control
- Data
- Security
A request for a Secret passes an RBAC permission check at the API server, and the value is read from etcd, where it is stored unencrypted unless you turn on encryption at rest.
- You to API server: 1 ask for it
- API server to RBAC check: 2 allowed?
- API server to etcd: 3 read the value
Both objects hold key-value pairs. You can inject a key as an environment variable, or mount the whole object as a directory where each key becomes a file. Environment variables are fixed when the container starts. Mounted files are refreshed after a delay when the object changes, but your app has to re-read them.
Be clear about what a Secret is. Its values are stored base64-encoded, which is a reversible text encoding and not encryption. By default they are saved unencrypted in etcd, and anyone who can read the Secret, or can create a Pod in that namespace, can read the value. Treat it as a way to keep credentials out of images and manifests, and add protection yourself.
Real protection means restricting who can read Secrets with RBAC, turning on encryption at rest for etcd, and for high-value credentials using an external secret manager. Never commit a Secret manifest to Git.
Try it yourself
Create a ConfigMap and a Secret, use both in a Pod, then read the Secret back to see how little protection base64 gives.
- 1Create the objects
kubectl create configmap app-config --from-literal=GREETING=hello kubectl create secret generic app-secret --from-literal=DB_PASSWORD=s3cret
- 2Use them in a Pod
cat > cfg.yaml <<'EOF' apiVersion: v1 kind: Pod metadata: name: cfg-demo spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo greeting=$GREETING; cat /etc/creds/DB_PASSWORD; echo; sleep 3600"] env: - name: GREETING valueFrom: configMapKeyRef: name: app-config key: GREETING volumeMounts: - name: creds mountPath: /etc/creds readOnly: true volumes: - name: creds secret: secretName: app-secret EOF kubectl apply -f cfg.yaml kubectl wait --for=condition=Ready pod/cfg-demo --timeout=60s kubectl logs cfg-demoThe setting arrives as an environment variable and the secret arrives as a file named after its key.
- 3Decode the Secret yourself
kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -dYou should see
czNjcmV0 s3cret
The first line is how the value is stored. The second decodes it with one command. On older macOS use
base64 -D. - 4Clean up
kubectl delete pod cfg-demo kubectl delete configmap app-config kubectl delete secret app-secret rm cfg.yaml
What breaks, and where to look
Pod STATUS is CreateContainerConfigError.
- WHY IT HAPPENS
- The Pod references a ConfigMap, Secret or key that does not exist, or a typo in the name. The container is never created.
- CHECK FIRST
- Run
kubectl describe pod <pod-name>and read the Events, which name the missing object. Thenkubectl get configmap,secretin the same namespace.
You edited the ConfigMap, but the running app still shows the old value.
- WHY IT HAPPENS
- Environment variables are copied when the container starts and never change afterwards. Files mounted with
subPathalso do not update. - CHECK FIRST
- Restart the Pods with
kubectl rollout restart deployment/<name>, or mount the ConfigMap as a directory and make the app re-read the file.
When not to use this
- Do not rely on base64 as protection. If a value would hurt you in a leaked manifest, use encryption at rest, tight RBAC or an external secret manager, and keep it out of Git.
- A ConfigMap is limited in size. Large files and binary data do not belong there. Put them in a volume or an object store.
Check yourself
Is a Kubernetes Secret encrypted by default?
No. Values are base64-encoded, which anyone can reverse, and they are stored unencrypted in etcd unless you enable encryption at rest.
Why did the app not see your new ConfigMap value?
Environment variables are set once at container start. Restart the Pod, or read mounted files and re-read them in the app.
What error shows when a Pod points to a ConfigMap that does not exist?
CreateContainerConfigError. The Events from kubectl describe pod name the missing object.
Keep going
Lesson 74 of 92 in the DevOps path