Week 1 review + Server Down!

The Week-1 muscle: orient → investigate → fix → verify

Section 1 of 5 · ~2 min

Six days in, you have a pile of commands: whoami, pwd, ls, cd, cat, chmod, ps, kill. Today is not a seventh pile. It’s the day those commands stop being trivia and become a reflex — the exact reflex an on-call engineer runs when a server falls over.

Every incident, from a hobby VPS to a bank’s payment tier, follows the same four beats:

Orient. Where am I, who am I, what’s here? Day 1’s whoami and pwd, Day 2’s map of the filesystem, and Day 3’s ls/cd answer this in seconds. You never act on a box you can’t describe.

Investigate. What is actually wrong? Day 4 taught you to read — cat a log, then grep it for the one line that matters. Day 5’s permissions explain a whole class of failures (“it can’t write because it doesn’t own the file”). Day 6’s ps and top show what is really running versus what you assume is running.

Fix. Change exactly one thing. Day 6’s kill stops a runaway process; Day 5’s chmod/chown repairs access; an edit fixes a config. One change, chosen because the investigation pointed at it — not because you are guessing.

Verify. Did it actually recover? Re-run the same ps, re-read the same log, hit the health check. An incident isn’t over when you did something; it’s over when the evidence says the service is healthy again.

Real world: It’s 2 a.m. and your phone is screaming that checkout is down. You will not be calm, and you will not remember anything clever. What saves you is boring muscle memory: get your bearings, read the log, list the processes, stop the one that’s on fire, and watch the graph come back. The engineers who look unflappable in a crisis aren’t smarter — they just run this loop without thinking.

A named example makes it concrete. Google’s Site Reliability Engineering teams turned this into a formal discipline. Their incident-response practice has an on-call engineer assess the situation (orient), diagnose the cause (investigate), and mitigate it (fix) — and only close the incident after confirming the service recovered, with a blameless postmortem afterward so the loop teaches the whole team, not just whoever was awake. The commands you learned this week are the ground floor of exactly that job.

That is why Day 7 is a mission, not a lecture. Reading about incidents doesn’t build the reflex — running one does. The Server Down! lab below drops you onto a (simulated) 2 a.m. page and makes you walk all four beats with the commands from days 1–6. No new syntax. Just proof that the muscle works under pressure.

Hands-On Lab

Section 2 of 5 · ~1 min

Today the lab is the mission. No WSL2, no setup, nothing to paste — Server Down! runs entirely in your browser.

This is your Week-1 boss fight. It’s 2 a.m., PagerDuty just went off, and checkout is down on prod-web-01. You are the only one awake, and every minute costs money. Everything you need is something you already learned in days 1–6, so play it as the four beats:

• Orient with pwd, ls, and cd — get your bearings and notice who owns what on an unfamiliar box (days 1–3, and the permissions instinct from day 5).
• Investigate by cat-ing and grep-ing the server log for the line that names the offending process (day 4).
• Confirm the culprit with ps — cross-check what the log claims against what’s actually running and pegging the CPU (day 6).
• Fix and verify by sending the runaway process a kill signal, then re-checking ps and the log until checkout reads healthy again (day 6).

Type help in the terminal to see the supported commands, and hint if you stall — it nudges without solving. There is no penalty for poking around; the whole point is to run the orient → investigate → fix → verify loop with your hands, under a little (fake) pressure, until it’s reflex. Bring checkout back — then, if you want the bragging rights, replay it and try to finish in fewer commands.

When the recovery rolls in, come back and note below what tripped you up — that reflection is where the lesson sticks.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the mistakes that trip people up when they run the incident loop for real in WSL2 Ubuntu 24.04 — the same beats the mission rehearses (orient, fix, verify). Read the error text slowly; parsing it is the skill.

Common error: Fixing before orienting — you go to read the log, but you point grep at a path that doesn’t exist on this box:

grep: /var/log/app.log: No such file or directory

Why: The log lives somewhere else — a different app, a different distro layout, or a service-specific directory like /var/log/nginx/. grep cannot search a file that is not there, so it reports the path it could not open and exits non-zero.

Fix: List the directory first: ls /var/log (then ls /var/log/<service>/) to find the real filename, and grep that. This is the “orient” beat — confirm what is actually on the box before you investigate it.

How you’d spot it in prod: A CI step or a health script that greps a hard-coded log path fails the moment a service relocates its logs. The failing line names the missing path — fix the path in the script, don’t assume the log vanished.

Common error: Killing a process you don’t own — you spot the offender in ps, but it runs as another user (or as root), so your kill bounces:

bash: kill: (1) - Operation not permitted

Why: Linux only lets you signal processes you own. A normal user cannot kill root’s processes (or another user’s) without elevated privileges — the same ownership rule from day 5, now applied to processes instead of files.

Fix: Read the owner in the first column of ps aux, decide the stop is safe, then escalate deliberately with sudo kill <pid>. In a real incident you confirm you’re allowed to stop that service before you do.

How you’d spot it in prod: A restart script that runs as an unprivileged user silently fails to stop a root-owned daemon. Operation not permitted in its logs means the service account lacks rights — fix the ownership or the sudo rule, don’t loosen everything.

Common error: Skipping verify — you send kill 4821 to stop the runaway process, then reflexively fire it again “to be sure,” but it’s already gone:

bash: kill: (4821) - No such process

Why: The first kill worked; there is no longer a process with that PID, so the second signal has nothing to hit. The shell’s kill builtin reports the PID it could not find.

Fix: Don’t re-send the signal — verify instead. Re-run ps -p 4821 (it prints just the header row with no process line, and exits non-zero) or ps aux | grep <name> (no matching line) — either is your proof the process is gone. That’s the “verify” beat: you confirm recovery with fresh evidence, not by firing the fix twice.

How you’d spot it in prod: No such process in a shutdown script usually means the process already exited (or you captured a stale PID from a pidfile). Re-check that the service — not just one PID — is actually down before you treat it as an error.

Incident Response Interview Questions

Section 4 of 5 · ~1 min

These four are pure interview ammo — an incident walk-through is the most common Phase-1 screening question, and “tell me about a time a server was on fire” rewards a calm, specific answer over a clever one. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes:

• 5 min — Replay Server Down! and try to finish in the optimal 7 commands for a cleaner rank — speed here is just knowing the loop cold.
• 10 min — Run man ps and skim the aux and --sort flags; being able to rank processes by CPU or memory on demand is a daily reflex.
• 15 min — Reread the Linux for DevOps guide end to end now that the commands mean something — the second pass is where it consolidates.