ConfigMaps & Secrets

Config lives outside the image: ConfigMaps and Secrets

Section 1 of 5 · ~3 min

Yesterday your Deployment ran a fixed image and reached other pods through a Service. But a real app needs settings — a log level, a feature flag, a database URL, an API key — and those change from one environment to the next. Bake them into the image and you rebuild for every change and end up shipping passwords in your registry. Kubernetes gives you two objects that keep config outside the image: the ConfigMap for non-sensitive settings and the Secret for sensitive ones.

A ConfigMap is a named bag of key/value pairs. You create it once, then a pod reads it one of two ways. As environment variables — pull one key with valueFrom.configMapKeyRef, or inject every key at once with envFrom.configMapRef, where each key becomes an env var. Or as files — mount the ConfigMap as a volume and every key appears as a file whose contents are the value, which suits a whole config file like nginx.conf.

A Secret works the exact same two ways — env var or mounted file — but is meant for passwords, tokens and connection strings. Here is the warning that catches everyone: a Secret is not encrypted. Its values are stored base64-encoded, which is reversible transport packaging, not security. Anyone who can run kubectl get secret -o yaml can pipe the value through base64 -d and read it in one line. A Secret earns its name only when you add encryption at rest on etcd and lock down RBAC so most people can’t read it at all. Base64 exists purely so binary values survive being carried inside YAML.

Real world: Think of a touring stage play. The script is the image — fixed, identical in every city. The set dressing and posters are the ConfigMap — swapped per venue, and nobody minds who reads them. The cash box combination is the Secret — the crew still needs it, but it lives in a locked safe backstage, not stapled to the call sheet. Same show everywhere; only the settings and the safe change.

This pattern is older than Kubernetes. The Twelve-Factor App methodology, written by the team behind Heroku, made “store config in the environment” a rule years ago: keep code identical across deploys and let each environment supply its own settings through the environment, never through edits to the source. ConfigMaps and Secrets are Kubernetes’ first-class way to honour that rule — the ConfigMap is your environment config, the Secret is the same idea with a lock on it.

For our running app that means the linkstash container never hard-codes where its database lives. You put DATABASE_URL in a Secret, reference it from the Deployment with secretKeyRef, and the container starts with that variable in its environment — exactly as it would read it on your laptop. Change databases and you edit one Secret, not the image. That single wire — Secret to Deployment — is the skill the Kubernetes Chaos mission leans on when half a stack is misconfigured and won’t start.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 25 minutes. This runs entirely on your local kind cluster from Day 67 — no cloud, no bill. Confirm you’re pointed at it first, then build a ConfigMap and a Secret, wire both into a Deployment, and prove the values land inside the pod. Pod name suffixes, IPs, ages and UIDs are unique to your run — yours will differ from the samples.

# 1. Confirm kubectl is talking to your local kind cluster, not something remote.
kubectl config current-context
# Output:
# kind-kind

# 2. linkstash-config.yaml — non-sensitive settings as a ConfigMap.
apiVersion: v1
kind: ConfigMap
metadata:
  name: linkstash-config
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"

# 3. Apply the ConfigMap and read it back.
kubectl apply -f linkstash-config.yaml
kubectl get configmap linkstash-config
# Output:
# configmap/linkstash-config created
# NAME               DATA   AGE
# linkstash-config   2      5s

# 4. Create the Secret imperatively — kubectl base64-encodes the value for you.
#    --from-literal avoids putting the raw string in a file you might commit.
kubectl create secret generic linkstash-db \
  --from-literal=DATABASE_URL='postgres://app:s3cret@db:5432/linkstash'
# Output:
# secret/linkstash-db created

# 5. Prove base64 is NOT encryption: fetch the stored value and decode it in one line.
kubectl get secret linkstash-db -o jsonpath='{.data.DATABASE_URL}' | base64 -d
# Output (readable plaintext — anyone with get access can do this):
# postgres://app:s3cret@db:5432/linkstash

# 6. linkstash-deploy.yaml — envFrom pulls every ConfigMap key; secretKeyRef pulls one Secret key.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: linkstash
spec:
  replicas: 1
  selector:
    matchLabels: { app: linkstash }
  template:
    metadata:
      labels: { app: linkstash }
    spec:
      containers:
        - name: linkstash
          image: nginx:1.27-alpine   # public stand-in for your Project-1 linkstash image
          envFrom:
            - configMapRef:
                name: linkstash-config
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: linkstash-db
                  key: DATABASE_URL

# 7. Apply the Deployment and wait for the pod to come up.
kubectl apply -f linkstash-deploy.yaml
kubectl get pods -l app=linkstash
# Output (pod name suffix and age are yours):
# deployment.apps/linkstash created
# NAME                        READY   STATUS    RESTARTS   AGE
# linkstash-6c9f8b7d4-x2k9p   1/1     Running   0          12s

# 8. Verify the config landed INSIDE the pod — env vars from both sources.
kubectl exec deploy/linkstash -- env | grep -E 'APP_ENV|LOG_LEVEL|DATABASE_URL'
# Output:
# APP_ENV=production
# LOG_LEVEL=info
# DATABASE_URL=postgres://app:s3cret@db:5432/linkstash

# 9. Add a volume mount so the whole ConfigMap appears as files under /etc/linkstash.
#    (Patch snippet — add this alongside the container in linkstash-deploy.yaml.)
          volumeMounts:
            - name: config
              mountPath: /etc/linkstash
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: linkstash-config

# 10. Re-apply, then read a key as a file — the filename is the key, contents are the value.
kubectl apply -f linkstash-deploy.yaml
kubectl exec deploy/linkstash -- cat /etc/linkstash/LOG_LEVEL
# Output (no trailing newline — the file holds exactly the value):
# info

# 11. Clean up so the next day starts fresh.
kubectl delete deploy/linkstash configmap/linkstash-config secret/linkstash-db
# Output:
# deployment.apps "linkstash" deleted
# configmap "linkstash-config" deleted
# secret "linkstash-db" deleted

Read step 8 back: APP_ENV and LOG_LEVEL arrived through envFrom off the ConfigMap, and DATABASE_URL through secretKeyRef off the Secret — the container never knew the difference. That is config externalized: one image, its settings supplied by the cluster at start.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three catch people the first time they wire a Secret into a Deployment on kind. Read the pod’s STATUS and its events slowly — that is where Kubernetes tells you exactly which key it couldn’t find.

Common error: The Deployment references a Secret or ConfigMap that doesn’t exist yet — a typo in the name, or applying the Deployment before the Secret:

NAME                        READY   STATUS                       RESTARTS   AGE
linkstash-6c9f8b7d4-x2k9p   0/1     CreateContainerConfigError   0          20s

Why: The kubelet can’t build the container’s environment because a referenced object is missing — kubectl describe pod shows Error: secret "linkstash-db" not found. The pod isn’t crash-looping; it never started, because Kubernetes won’t launch a container whose declared config it can’t resolve.

Fix: Check the reference name matches the object exactly (kubectl get secret,configmap), then create the missing one and let the Deployment retry — it recovers on its own once the object exists.

How you’d spot it in prod: A pod stuck at CreateContainerConfigError right after a deploy, with describe naming a “not found” secret or configmap, almost always means the manifest applied before its config — order the apply, or bundle both in the same manifest.

Common error: A secretKeyRef (or configMapKeyRef) names a key that isn’t in the object — here asking for DB_URL when the Secret’s key is DATABASE_URL:

Warning  Failed  ...  Error: couldn't find key DB_URL in Secret default/linkstash-db

Why: The object exists, but the key: under secretKeyRef must match one of its data keys exactly, and key names are case-sensitive. A mismatch leaves the container’s env incomplete, so the kubelet reports CreateContainerConfigError and names the missing key in the pod events.

Fix: List the real keys with kubectl describe secret linkstash-db (or -o jsonpath='{.data}') and align the key: in the manifest to one of them, then re-apply.

How you’d spot it in prod: CreateContainerConfigError whose event says “couldn’t find key” points at the manifest, not the cluster — the referenced object is fine; the key name in the workload spec is wrong.

Common error: Building a Secret’s value by hand with echo and getting a stray newline baked into it:

# echo appends a newline, so the decoded value ends in \n and the DB connection fails:
$ echo 'postgres://app:s3cret@db:5432/linkstash' | base64
cG9zdGdyZXM6Ly9hcHA6czNjcmV0QGRiOjU0MzIvbGlua3N0YXNoCg==

Why: echo without -n adds a trailing newline before base64 encoding it, so the app receives ...linkstash\n and its database driver fails to parse the host — a maddening bug because the value looks correct in kubectl get.

Fix: Let kubectl encode for you with --from-literal (it never adds a newline), or if you must encode by hand use printf %s or echo -n. Decode and eyeball the result — a value that ends in a visible line break is the tell.

How you’d spot it in prod: An app that rejects a connection string which reads as perfectly valid usually has an invisible trailing newline from a hand-rolled base64 — decode the Secret and check its exact length.

ConfigMaps & Secrets Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — especially the base64 question, because “is a Secret encrypted?” is a near-guaranteed screen for any Kubernetes role, and the honest answer (“no, it’s encoded”) is what interviewers want to hear. Recalling before revealing is what makes these stick when you’re asked cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Run kubectl create configmap demo --from-literal=k=v -o yaml --dry-run=client and read the manifest it prints; --dry-run=client is how you generate correct YAML without touching the cluster.
• 10 min — Recreate the Secret with --from-file= pointing at a real file (e.g. a .env) instead of --from-literal, then kubectl describe it and see how each filename becomes a key — the same trick works for whole certificate or config files.
• 15 min — Skim the Kubernetes docs on encrypting Secret data at rest to see what turns base64 encoding into real protection, and read why RBAC on the secrets resource matters as much as the encryption itself.