Health probes & resource requests/limits

Probes and limits: is it alive, is it ready, and how much can it take

Section 1 of 5 · ~3 min

Days 68–70 got your app running on the cluster — a Deployment schedules Pods, a Service load-balances them, ConfigMaps and Secrets feed them config. But “running” isn’t “healthy,” and Kubernetes has no idea what healthy means for your app until you tell it. Two mechanisms do that: probes describe how to check a container’s health, and requests and limits describe how much CPU and memory it may use. Get both right and the cluster heals and schedules your app for you; get them wrong and you meet today’s error list first-hand.

A probe is a check the kubelet runs against a container on a schedule — an HTTP GET, a TCP connect, or a command that must exit 0. Three kinds answer three different questions. A liveness probe asks is it alive? — fail it enough times and the kubelet restarts the container, the cure for a process wedged in a deadlock. A readiness probe asks is it ready for traffic? — fail it and the Pod is pulled from its Service’s endpoints, so no request is routed to it, but it is not restarted. A startup probe asks has it finished booting? — until it passes, liveness and readiness are held off, which protects a slow-starting app from being killed before it ever comes up.

              kubelet probes each container on a schedule
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
   startup probe         liveness probe        readiness probe
   "booted yet?"         "still alive?"        "ready for traffic?"
        │                     │                     │
   holds liveness &      fail x3 ─▶ RESTART    fail ─▶ remove Pod from
   readiness until           the container     the Service's endpoints
   it passes                                   (no traffic; NOT restarted)

Requests, limits, and the three QoS classes

A request is what the scheduler reserves: a Pod asking for 250m CPU and 128Mi memory only lands on a node with that much free, and it’s guaranteed at least that much. A limit is the hard ceiling the runtime enforces. The two resources hit their ceilings very differently. Exceed a memory limit and the kernel’s OOM killer terminates the container — you’ll see OOMKilled and exit code 137; the Pod usually restarts and, if it keeps overshooting, lands in CrashLoopBackOff. Exceed a CPU limit and nothing is killed — the container is simply throttled, given fewer CPU cycles, so it just runs slower. CPU is compressible; memory is not.

Those two numbers also decide a Pod’s Quality of Service class, which is the order the kubelet evicts Pods under node pressure. Set requests equal to limits on every resource and the Pod is Guaranteed (evicted last). Set some but not matching, and it’s Burstable. Set none at all and it’s BestEffort — first to be killed when the node runs low.

Real world: Probes are the triage nurse at a hospital; requests and limits are the bed budget. The nurse keeps checking each patient — one who’s crashed gets rushed back to a bed (liveness restart), one who’s contagious is kept out of the ward until cleared (readiness removes them from rotation), and a patient still in admissions isn’t disturbed (startup probe). Meanwhile the ward only admits as many patients as it has beds and oxygen for (requests), and no one is allowed to hog the whole oxygen supply (limits).

Kubernetes runs this on itself: the control plane’s own components expose /livez, /readyz and /healthz endpoints, and the kubelet probes the API server on exactly the same machinery you’re about to point at your app — the platform trusts its own probes to decide when the API is safe to send traffic to.

So the job today is to state, in YAML, what your container needs and what healthy looks like — and then watch Kubernetes act on it.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 30 minutes. This runs entirely locally on your kind cluster from Day 67 — no cloud, no bill. Type each command yourself and read the output. Pod name suffixes, IPs, node names, ages and UIDs are unique to each run — yours will differ from the samples below.

# 1. Confirm your kind cluster from Day 67 is up and the node is Ready.
kubectl get nodes
# Output (node name and AGE are yours — kind names the node after the cluster):
# NAME                 STATUS   ROLES           AGE    VERSION
# kind-control-plane   Ready    control-plane   4d     v1.31.0

# 2. probes.yaml — a Deployment with all three probes plus requests and limits.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: linkstash
spec:
  replicas: 1
  selector:
    matchLabels: { app: linkstash }
  template:
    metadata:
      labels: { app: linkstash }
    spec:
      containers:
        - name: web
          image: gcr.io/google-samples/hello-app:1.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /, port: 8080 }
            periodSeconds: 10
          startupProbe:
            httpGet: { path: /, port: 8080 }
            failureThreshold: 30
            periodSeconds: 2
          resources:
            requests: { cpu: 100m, memory: 64Mi }
            limits:   { cpu: 250m, memory: 128Mi }

# 3. Apply it, then watch the Pod come up. --watch streams status changes.
kubectl apply -f probes.yaml
kubectl get pods -l app=linkstash --watch
# Output (pod suffix is yours; note READY stays 0/1 until the readiness probe passes):
# NAME                         READY   STATUS              RESTARTS   AGE
# linkstash-6c9f8d7b4c-2xkzp   0/1     Pending             0          0s
# linkstash-6c9f8d7b4c-2xkzp   0/1     ContainerCreating   0          1s
# linkstash-6c9f8d7b4c-2xkzp   0/1     Running             0          3s
# linkstash-6c9f8d7b4c-2xkzp   1/1     Running             0          8s   <- readiness passed

# 4. Read how the kubelet registered the three probes (Ctrl-C the watch first).
kubectl describe pod -l app=linkstash | grep -iE 'liveness|readiness|startup'
# Output (delay/period/threshold echo your YAML):
# Liveness:   http-get http://:8080/ delay=0s timeout=1s period=10s #success=1 #failure=3
# Readiness:  http-get http://:8080/ delay=3s timeout=1s period=5s #success=1 #failure=3
# Startup:    http-get http://:8080/ delay=0s timeout=1s period=2s #success=1 #failure=30

# 5. Put a Service in front and confirm the Pod is in its endpoints ONLY when Ready.
kubectl expose deployment linkstash --port=80 --target-port=8080
kubectl get endpoints linkstash
# Output (the Pod IP appears because READY is 1/1 — a not-ready Pod would be absent here):
# NAME        ENDPOINTS          AGE
# linkstash   10.244.0.14:8080   5s

# 6. oom.yaml — a container told to grab 250M but limited to 100Mi. It will be OOMKilled.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oomtest
spec:
  replicas: 1
  selector:
    matchLabels: { app: oomtest }
  template:
    metadata:
      labels: { app: oomtest }
    spec:
      containers:
        - name: hog
          image: polinux/stress
          command: ["stress"]
          args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "0"]
          resources:
            requests: { memory: 64Mi }
            limits:   { memory: 100Mi }

# 7. Apply it and watch the container get killed and back off into a restart loop.
kubectl apply -f oom.yaml
kubectl get pods -l app=oomtest --watch
# Output (RESTARTS climbs; STATUS flips OOMKilled -> CrashLoopBackOff):
# NAME                       READY   STATUS             RESTARTS   AGE
# oomtest-7d4b9c6f5-abcde    0/1     Running            0          2s
# oomtest-7d4b9c6f5-abcde    0/1     OOMKilled          0          4s
# oomtest-7d4b9c6f5-abcde    0/1     CrashLoopBackOff   2          40s

# 8. Confirm the cause: Last State Terminated, Reason OOMKilled, exit code 137.
kubectl describe pod -l app=oomtest | grep -A3 'Last State'
# Output (128 + signal 9 (SIGKILL) = exit code 137 — the OOM killer's signature):
#     Last State:     Terminated
#       Reason:       OOMKilled
#       Exit Code:    137
#       Started:  <timestamp> (yours will differ)

# 9. Check each Pod's QoS class — it's derived from requests vs limits.
kubectl get pod -l app=linkstash -o jsonpath='{.items[0].status.qosClass}{"\n"}'
kubectl get pod -l app=oomtest   -o jsonpath='{.items[0].status.qosClass}{"\n"}'
# Output (both set some-but-not-matching requests/limits, so both are Burstable):
# Burstable
# Burstable

# 10. Clean up both Deployments and the Service so the next day starts fresh.
kubectl delete -f oom.yaml -f probes.yaml
kubectl delete service linkstash
# Output:
# deployment.apps "oomtest" deleted
# deployment.apps "linkstash" deleted
# service "linkstash" deleted

Read the run back: the healthy app stayed 0/1 until its readiness probe passed and only then joined the Service’s endpoints, while the memory hog was OOMKilled with exit code 137 and fell into CrashLoopBackOff. That is probes and limits doing their job — and the failure modes you’ll recognise on sight for the rest of Phase 4.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the mistakes that show up the first time people add probes and limits to a real workload. Read the error text slowly — parsing it is the skill.

Common error: A liveness probe with initialDelaySeconds too low (or no startup probe) on a slow-booting app, so the kubelet kills the container mid-startup:

Liveness probe failed: Get "http://10.244.0.14:8080/healthz": dial tcp 10.244.0.14:8080: connect: connection refused
Container web failed liveness probe, will be restarted

Why: The liveness probe starts firing before the app has finished booting and bound its port. Each failed probe counts toward failureThreshold; once it’s hit, the kubelet restarts the container — which starts the slow boot over, fails again, and drops the Pod into CrashLoopBackOff. The app was never broken; the probe just never gave it time to come up.

Fix: Add a startup probe so liveness is held off until boot completes, or raise the liveness initialDelaySeconds/failureThreshold. Prefer the startup probe — it stays patient during boot and strict afterwards.

How you’d spot it in prod: A Pod in CrashLoopBackOff whose logs show a clean startup right up to the moment it’s killed, with Liveness probe failed events in kubectl describe. The tell is that the restart interval matches your boot time, not a real crash.

Common error: A container terminated with OOMKilled and exit code 137:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Why: The container tried to use more memory than its limits.memory, and because memory can’t be reclaimed the way CPU can, the kernel’s OOM killer sent SIGKILL (128 + 9 = exit code 137). Kubernetes restarts it, and if it keeps overshooting the limit you get a CrashLoopBackOff of repeated kills.

Fix: If the app legitimately needs more, raise limits.memory (and the request with it). If it doesn’t, you have a leak or an unbounded buffer to fix in the app — don’t just keep raising the ceiling. Size the limit from real usage, not a guess.

How you’d spot it in prod: kubectl describe pod shows Reason: OOMKilled and Exit Code: 137 under Last State, and memory metrics sit pinned at the limit right before each restart. Exit 137 without OOMKilled usually means an external SIGKILL instead.

Common error: A Pod stuck Pending because its requests don’t fit any node:

0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Why: A request is a hard scheduling reservation. If no node has that much allocatable memory (or CPU) still free, the scheduler can’t place the Pod, so it never starts — it just sits Pending. This is not a crash or a failed container; the container was never created because the Pod was never scheduled.

Fix: Lower requests to what the app actually uses, or add node capacity (on kind, a bigger single node; in a real cluster, more nodes). Compare kubectl describe node Allocatable against Allocated resources to see the headroom you’re asking past.

How you’d spot it in prod: New Pods stuck Pending after a deploy while the old ones keep running, with a FailedScheduling event reading Insufficient cpu/Insufficient memory. It means the requests are too big for the cluster’s spare capacity, not that the image or app is wrong.

Kubernetes Probes & Limits Interview Questions

Section 4 of 5 · ~1 min

Health checks and resource sizing come up in almost every Kubernetes screen — an interviewer wants to hear that you know liveness restarts while readiness only drains traffic, and that a memory limit kills while a CPU limit throttles. Cover each answer below, say your own version out loud first, then compare — recalling before revealing is what makes it stick. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~35 more minutes:

• 10 min — Read the official Configure Liveness, Readiness and Startup Probes task; it covers exec and tcpSocket probes, timeoutSeconds and failureThreshold, the knobs you’ll tune constantly in real clusters.
• 10 min — On your kind cluster, give a busy container a CPU limit of 100m and watch kubectl top pod (needs metrics-server) sit pinned at the ceiling — CPU throttling in action, no restart, just slower.
• 15 min — Read Resource Management for Pods and Containers and its QoS classes section, then map each of your Pods’ qosClass back to the requests and limits that produced it.