Week 11 review + Kubernetes Chaos

The Week-11 muscle: run it, reach it, configure it, keep it alive

Section 1 of 5 · ~3 min

Phase 3 left your app on AWS — deployed, load-balanced, monitored, but every scaling and healing decision still made by you. Week 11 handed that judgement to a cluster: you declare the state you want, and Kubernetes works to keep reality matching it. Five days built that skill, and today they collapse into one reflex — when a cluster goes sideways, you read what it’s telling you before you touch a single manifest.

The Kubernetes chain has three links, and you now own each one:

Run it. Day 68 drew the line between a Pod — the smallest deployable unit, one or more containers sharing a network and lifecycle — and a Deployment, the controller that keeps a set number of identical Pods running. Kill a Pod and the Deployment notices the gap and starts a replacement; that reconciliation loop is the whole point. You never nurse individual Pods — you declare how many you want and let the controller hold the line.

Reach it. Day 69 gave those ephemeral Pods a stable front door. A Service has a fixed name and cluster IP and load-balances across whichever Pods match its label selector, so the rest of the cluster reaches db by name no matter how often the Pods behind it are recreated — the same name-based discovery Docker’s network gave you, now cluster-wide.

Configure it and keep it alive. Day 70 pulled config out of the image: ConfigMaps for plain settings, Secrets for credentials, injected as environment variables or mounted files so one image runs in any environment. Day 71 taught the cluster to judge health — liveness and readiness probes decide when to restart a Pod or route traffic to it — and resource requests and limits tell the scheduler how much CPU and memory a Pod needs to be placed at all. Day 72 tied it together: when a workload misbehaves you kubectl describe to read the Events, kubectl logs (with --previous for a crashed container) to hear the app, and kubectl get events for the timeline — status, events, logs, in that order.

Real world: A thermostat doesn’t wait for you to flip the furnace on and off — you set the temperature you want and it works, minute by minute, to close the gap. Kubernetes runs the same loop for your app: you declare “three replicas of this Pod, with this config and these limits,” and a controller keeps nudging reality toward that number. When a Pod dies, no one gets paged — the loop just starts another. Today’s mission breaks the declared state on purpose so you feel the loop fight to restore it.

A named example makes it concrete. Kubernetes grew out of Borg, the cluster manager Google had run internally for years before open-sourcing the ideas in 2014. Borg’s core lesson — tell the system the state you want and let it reconcile, rather than scripting each start and stop by hand — is exactly the reconciliation loop you rehearse today. It’s why a deleted Pod, or a deleted Secret, doesn’t force a manual redeploy: the controller is already watching, ready to heal the moment you fix the cause.

That chain is why today is a mission, not a lecture. Reading about a cluster in chaos doesn’t build the reflex — rescuing one does. Kubernetes Chaos hands you a cluster where half the Pods are Pending and the rest are CrashLoopBackOff, and the week-11 tools to read the events, find what’s missing, and let the Deployment heal itself — without editing a single line of the app or its manifests.

Hands-On Lab

Section 2 of 5 · ~2 min

Today the lab is the mission. No kind cluster to spin up, no manifests to paste — Kubernetes Chaos runs entirely in your browser.

This is your Week-11 boss fight, and it closes the Kubernetes core of Phase 4. An on-call alert fires: half the Deployment’s Pods are stuck Pending, the rest are looping through CrashLoopBackOff, and the app is down. Nothing was redeployed — someone deleted the db-credentials Secret during a cleanup, and the cluster has been fighting to recover ever since. Everything you need is something you already learned in days 68–72, so play it as the walk from the diagram:

• See the chaos — cat ~/k8s/pods.txt (the saved snapshot; on a live cluster it’s kubectl get pods) shows the split: some Pods stuck at ContainerCreating, the rest CrashLoopBackOff with a climbing restart count. That status is the shape of the problem, not the cause (day 72).
• Read the events — cat ~/k8s/events.txt puts the reason on the table: FailedMount MountVolume.SetUp failed for volume "db-creds" : secret "db-credentials" not found for the mounting Pods, and back-off/crash lines for the others (days 71–72).
• Find the cause — cat ~/k8s/deploy.yaml: the Deployment both mounts (a secret volume) and injects (secretRef) a Secret that no longer exists. One deleted object broke both the mounting Pods and the crashing ones; nothing about the image or the Deployment spec is wrong (days 68, 70).
• Restore and let it heal — the fix isn’t a redeploy. Recreate the Secret with the same name and keys — kubectl create secret generic db-credentials … — and the Deployment’s controller restarts the Pods against it and reconciles back to healthy on its own (day 68’s self-healing loop).

Type help in the terminal to see the supported commands, and hint if you stall — it nudges without solving. There’s no penalty for poking around; the whole point is to run the get → describe → find → restore walk with your hands until it’s reflex, and to feel why you fix the cause and let the controller heal rather than restarting Pods by hand. Get the Deployment healthy — then, if you want the bragging rights, replay it and try to read the cause from the events in the fewest commands.

When every Pod finally reports Running, come back and note below which signal — the status, the events, or the logs — told you what was really wrong.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the mistakes that trip people up when they run the week-11 workflow for real on a kind cluster on WSL2 — the same signals the mission rehearses. Read the error text slowly; parsing it is the skill. Pod name suffixes, restart counts and ages are unique to each run — yours will differ from the samples below.

Common error: A Pod references a Secret that doesn’t exist, so the kubelet can’t build the container’s config:

NAME                        READY   STATUS                       RESTARTS   AGE
linkstash-7d9c8b6f4-2xk9p   0/1     CreateContainerConfigError   0          40s

kubectl describe pod names it exactly in the Events:

  Warning  Failed  12s (x4 over 40s)  kubelet  Error: secret "db-credentials" not found

Why: The Deployment injects a credential with envFrom or a secretKeyRef pointing at a Secret named db-credentials. With that Secret missing, Kubernetes can’t populate the environment, so it never starts the container — this is a config failure, before the app runs, which is why the status is CreateContainerConfigError and not CrashLoopBackOff.

Fix: Recreate the Secret with the name and keys the Deployment expects: kubectl create secret generic db-credentials --from-literal=username=app --from-literal=password=…. The kubelet retries automatically and the Pods start; no redeploy needed.

How you’d spot it in prod: A workload stuck at CreateContainerConfigError right after a config or cleanup change points at a missing ConfigMap or Secret, not a bad image. kubectl describe pod names the exact object — recreate it with the same name and keys.

Common error: A Pod mounts a Secret as a volume that doesn’t exist, so the kubelet can’t set up the volume and never creates the container:

NAME                    READY   STATUS              RESTARTS   AGE
web-6d8f7c9b5d-4xk2p    0/1     ContainerCreating   0          2m

kubectl describe pod shows it in the Events:

  Warning  FailedMount  ...  MountVolume.SetUp failed for volume "db-creds" : secret "db-credentials" not found

Why: A Secret used as a volume can’t be mounted if it’s missing, so the Pod stays ContainerCreating — distinct from CreateContainerConfigError, which is the env-var / secretKeyRef path. The kubelet won’t start the container until every volume is set up, so the container is never created.

Fix: Recreate the Secret with the same name and keys — kubectl create secret generic db-credentials --from-literal=username=app --from-literal=password=…. The kubelet retries the mount automatically and the Pod proceeds; no redeploy needed.

How you’d spot it in prod: A Pod stuck at ContainerCreating right after a config or cleanup change points at a volume that can’t be set up — a missing Secret or ConfigMap mount, not a bad image. kubectl describe pod names the exact volume and object — recreate it with the same name and keys.

Common error: A container starts, exits, and Kubernetes keeps restarting it with growing delays:

NAME                        READY   STATUS             RESTARTS      AGE
linkstash-7d9c8b6f4-p8w2v   0/1     CrashLoopBackOff   5 (60s ago)   4m

Why: CrashLoopBackOff isn’t the error — it’s Kubernetes backing off between restarts of a container that won’t stay up. The app is exiting on startup: often a credential or config value it reads at boot is missing or wrong, a dependency it needs isn’t reachable, or a liveness probe is killing a slow starter. The rising RESTARTS count is the tell.

Fix: Read the app’s own words with kubectl logs <pod>, and kubectl logs <pod> --previous to see the crashed instance’s output rather than the one starting now. Fix what it prints — supply the missing value, reach the dependency, or relax the probe — and the loop stops.

How you’d spot it in prod: A Pod whose RESTARTS climbs while READY stays 0/1 is crash-looping. kubectl logs --previous almost always shows the stack trace or the “missing X” line right before the exit — read it before you touch the manifest.

Kubernetes Interview Questions

Section 4 of 5 · ~1 min

The Pending-versus-CrashLoopBackOff split and the describe → logs → events reflex below are among the most common Phase-4 Kubernetes screening questions — a calm, layered answer beats a clever one every time. The answer bank renders right after this note. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~40 more minutes:

• 5 min — Replay Kubernetes Chaos and try to read the cause from the Events in the fewest commands — speed here is just knowing the get → describe → find → restore walk cold.
• 10 min — On your kind cluster, kubectl delete secret db-credentials on a running Deployment, watch the Pods break with kubectl get pods -w, then recreate it and watch the controller heal them — the exact loop the mission exploits.
• 10 min — Run kubectl get events --sort-by=.lastTimestamp next to kubectl describe pod and kubectl logs --previous on the same broken Pod, and feel which signal answers which question.
• 15 min — Skim the official Debug Running Pods guide and map each tool — describe, logs, events, exec — back to the day-72 debugging flow.