Incident management — severity, runbooks, postmortems, error budgets

How real teams handle outages: severity, runbooks, postmortems, error budgets

Section 1 of 5 · ~3 min

Ninety days in, you can build and ship linkstash three ways. The last operational skill is what happens when it breaks in production — and how a team responds separates a junior from an engineer you’d trust on call.

Severity first. The moment something breaks, you classify it by impact: a SEV-1 is a full outage or data loss hitting most users — wake people up; SEV-2 is major but partial; SEV-3 and SEV-4 are minor or cosmetic, fixed in business hours. Severity sets urgency and who gets paged.

Someone runs the response. For anything serious you name an incident commander (IC) — one person who coordinates, decides, and communicates. The IC usually isn’t typing fixes; they keep responders from talking over each other and post status updates on a cadence. Without an IC, five people chase five theories and never sync.

Runbooks turn panic into procedure. A runbook is a pre-written, copy-pasteable sequence for a known failure mode — “disk at 95%, here’s how to reclaim it,” “cert expired, here’s the renew-and-reload.” It captures what an expert would do so a tired on-call engineer doesn’t reinvent it under pressure. Mitigate first — roll back, failover, scale — to stop the bleeding before chasing root cause.

Then you learn from it — blamelessly. The postmortem is a timeline, the root cause, contributing factors, and owned action items. Blameless means you fix the system, not the person. If one command took down a service, the question isn’t “who was careless” but “why did the system let one command do that.” Blame makes people hide mistakes; you lose the data you need to improve.

Real world: An incident is a kitchen fire during dinner service. Severity says whether it’s a flare-up or the whole line ablaze. The head chef becomes incident commander — not grabbing an extinguisher personally, but calling who does what so cooks don’t collide. The extinguisher-by-the-fryer is the runbook: rehearsed, in reach, no thinking required. Afterward the team asks why the oil got that hot and fixes the process — not who to fire.

Error budgets make reliability a number. An SLO is a target — say 99.9% availability a month. The error budget is the slack: 100% minus the SLO. A 30-day month is ~43,200 minutes, so 99.9% permits about 43 minutes of downtime; 99.99% permits about 4.3. Treat the budget as a currency: plenty left, ship risky features fast; burned through, you freeze risky launches and spend the next cycle on stability. That kills both extremes — chasing an impossible 100% and shipping with no guardrail.

A named example. On 28 February 2017, an AWS engineer debugging the S3 billing system in us-east-1 ran an established runbook command to remove a few servers — but a typo removed far more than intended, knocking out two S3 subsystems. S3 is so widely depended on that a big slice of the web went dark for hours. AWS’s public postmortem was textbook blameless: it never named the engineer, and its action items hardened the tooling — the removal command now refuses to take capacity below a safe threshold — instead of blaming a person for a typo the system should never have allowed.

Everything you shipped hangs off this — and tomorrow you defend it in an interview.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 20 minutes — and for once the artifact isn’t a running service, it’s a document. You’ll write a blameless postmortem for a real incident you already lived through, then compute an error budget for linkstash. In the Kubernetes Chaos mission (Day 73) a deleted db-credentials Secret sent your linkstash pods into a start failure — the perfect small outage to write up. First, reconstruct the two facts a timeline needs: what broke and how you knew.

# 1. The symptom: pods can't start because the container config can't be built.
kubectl get pods
# Output (pods never reach Running — no restarts because the container never started):
# NAME                         READY   STATUS                       RESTARTS   AGE
# linkstash-7c9d4f8b6d-4rk2p   0/1     CreateContainerConfigError   0          2m14s
# linkstash-7c9d4f8b6d-8xj9w   0/1     CreateContainerConfigError   0          2m14s

# 2. The proof: the pod's events name the missing dependency exactly.
kubectl describe pod linkstash-7c9d4f8b6d-4rk2p
# Output (tail — the Events block is where the root cause is spelled out):
# Events:
#   Type     Reason     Age                  From      Message
#   ----     ------     ----                 ----      -------
#   Normal   Scheduled  2m30s                default-scheduler  Successfully assigned default/linkstash-... to k3s
#   Warning  Failed     2m (x8 over 2m20s)   kubelet   Error: secret "db-credentials" not found

Now write the artifact. Create POSTMORTEM.md and fill each section — the shape below is the industry-standard skeleton.

# Postmortem: linkstash outage — db-credentials Secret deleted

- **Date:** 2026-05-14
- **Severity:** SEV-1 (all linkstash pods down; service fully unavailable)
- **Duration:** 14 minutes (11:02–11:16 UTC)
- **Author:** on-call engineer
- **Status:** resolved — action items open

**Summary:** A `kubectl delete secret db-credentials` intended for a scratch
namespace was run against the production context, removing the Secret linkstash
reads its `DATABASE_URL` from. Every pod failed to start with
`CreateContainerConfigError` until the Secret was recreated.

## Timeline (UTC)

- **11:02** — `db-credentials` Secret deleted (meant for a scratch namespace, run against the default context instead).
- **11:03** — A routine rollout recreates pods; they land in `CreateContainerConfigError`. No healthy pod remains.
- **11:05** — Alert fires: linkstash returns 502 at the ingress. Incident opened, SEV-1.
- **11:09** — `kubectl describe pod` shows `Error: secret "db-credentials" not found`. Root cause identified.
- **11:14** — Secret recreated from the sealed backup; pods roll to `Running`.
- **11:16** — End-to-end smoke test passes; incident resolved.

## Root cause & contributing factors

**Root cause:** the Secret `db-credentials`, a hard dependency of every linkstash
pod, was deleted, so the kubelet could not build the container's environment.

**Contributing factors:**
- No confirmation guard on destructive `kubectl` commands.
- The `kubectl` context defaulted to the production cluster.
- The Secret existed only in the cluster — no version-controlled, sealed backup.

## 5 whys

1. Why was linkstash down? Its pods couldn't start.
2. Why couldn't they start? The `db-credentials` Secret was missing.
3. Why was it missing? A command meant for another namespace deleted it.
4. Why did that command hit production? The default kube-context pointed there.
5. Why was there no safety net? The Secret had no backup and no delete guard.
→ Fix the system at whys 4 and 5, not the person at why 3.

## Action items

| # | Action | Owner | Due |
|---|--------|-------|-----|
| 1 | Store Secrets as SealedSecrets committed to the Git repo | A. Rao | 2026-05-21 |
| 2 | Add a shell prompt showing the active kube-context | A. Rao | 2026-05-18 |
| 3 | Add a readiness gate so a missing Secret pages before rollout | M. Diaz | 2026-05-28 |

linkstash’s SLO is 99.9% monthly availability. Compute the downtime that buys you — the error budget — and see how fast a tighter target burns it.

# 3. Downtime budget for common SLOs over a 30-day month (43,200 minutes).
python3 -c "m=30*24*60; [print(f'{s}%: {m*(1-s/100):.1f} min/month') for s in (99.9,99.99,99.999)]"
# Output:
# 99.9%: 43.2 min/month
# 99.99%: 4.3 min/month
# 99.999%: 0.4 min/month

The artifact belongs in the repo it documents — commit it beside your Project 3 capstone:

# 4. Version the postmortem with the code it explains.
git add POSTMORTEM.md
git commit -m "docs: blameless postmortem for the db-credentials Secret incident"
# Output:
# [main 5f2a1c9] docs: blameless postmortem for the db-credentials Secret incident
#  1 file changed, 41 insertions(+)
#  create mode 100644 POSTMORTEM.md

You now have a portfolio-grade postmortem — a document that proves you can not only build linkstash three ways but also reason calmly about the day it breaks. That is exactly what tomorrow’s interview drill will probe.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the incident and postmortem mistakes that quietly sink teams — and that a sharp interviewer listens for when you describe how you handle outages.

Common error: Writing a postmortem that blames the person who ran the command — “Alice deleted the Secret; Alice will be more careful” — and closing it there.

Why: Blame ends the investigation exactly where it should start. It answers who instead of why the system allowed it, so the real weaknesses — no delete guard, a context defaulting to prod, no backup — go unfixed. Worse, it teaches everyone to hide mistakes next time, and you lose the honest timelines that make postmortems useful at all.

Fix: Make it blameless. State what happened in system terms, run the 5 whys until you reach a process or tooling gap, and write action items that harden the system so the next tired engineer physically can’t cause the same outage. Names appear only as roles (“the on-call engineer”), never as culprits.

How you’d spot it in prod: A postmortem whose action items are all “be more careful” or “add more review” with no tooling change, and a team that has stopped volunteering what really happened during incidents — the two symptoms travel together.

Common error: Running an on-call rotation with no runbooks, so every incident is diagnosed and fixed from scratch by whoever happens to be paged.

Why: Known failure modes recur — disks fill, certs expire, a dependency goes down — but without a written procedure each recurrence is solved live, under stress, at whatever hour it strikes. Recovery time depends entirely on whether the one person who remembers the fix is awake and reachable. That’s slow, inconsistent, and it doesn’t scale past that one person.

Fix: After each incident, if the failure mode could recur, write a runbook: exact, copy-pasteable steps, not “investigate the issue.” Link it from the alert that fires. A postmortem’s action items are where runbooks are born — turning a painful one-off into a five-minute procedure next time.

How you’d spot it in prod: Mean-time-to-recovery that swings wildly by who’s on call, the same alert paging a senior engineer at 3am every few weeks, and a wiki with no runbooks — or runbooks so vague they say “restart the service and escalate.”

Common error: Ending a postmortem with action items that have no owner and no due date — “we should add backups,” “someone should improve alerting.”

Why: An action item with no name and no date is a wish, not a commitment. Everyone assumes someone else owns it, the incident fades from memory, and the same outage recurs three months later — now with a postmortem that proves you already knew how to prevent it. That’s postmortem theater: the ritual without the follow-through.

Fix: Every action item gets a single named owner and a real date, tracked in the same system as normal work (an issue, a ticket) so it can’t quietly vanish. Fewer, owned, dated items beat a long unowned list, and you review the open ones at the next incident review so they actually close.

How you’d spot it in prod: A folder of thorough postmortems for incidents that keep repeating, and action-item lists written in the passive voice (“backups should be added”) with empty owner columns — the tell that nobody actually holds them.

Incident Management Interview Questions

Section 4 of 5 · ~1 min

These four are the reliability and on-call questions that separate someone who has only built things from someone who has run them — say each answer out loud before you reveal it. The four questions and answers render right below.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~40 more minutes:

• 5 min — Read the actual AWS “Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region” postmortem and notice how it never names the engineer.
• 10 min — Turn today’s fix into a real runbook: the exact kubectl / SealedSecret steps to recreate db-credentials, copy-pasteable, linked from the alert.
• 10 min — Read the “Embracing Risk” / error-budget chapter of Google’s free online SRE book to see error budgets used at scale.
• 15 min — Draft a one-page incident-response checklist (a severity table, who’s IC, where the runbooks live) and drop it in your capstone repo.