FINAL BOSS: The Midnight Outage + what's next

Ninety days, converging on one pager at 00:14

Section 1 of 5 · ~3 min

Ninety days ago you opened a terminal for the first time and did not know what a shell was. Tonight you are on call for a full-stack outage, and every layer on fire is a layer you now own. This is not a new lesson — it is all of them at once.

Walk the arc. Phase 1 gave you the ground floor: the Linux filesystem, permissions, processes, pipes, ssh, and reading a log until it talks. Phase 2 packaged your app — Docker images and containers, a Dockerfile, networks and volumes, docker compose, and a CI pipeline in GitHub Actions that built and tested on every push. Phase 3 put it in the cloud on AWS: EC2, security groups, IAM, S3, and DNS through Route 53. Phase 4 made it resilient — Kubernetes deployments, Secrets, health checks, and Terraform to declare the whole thing as code. Then three projects turned that into a portfolio: you shipped linkstash on Docker (Project 1), on AWS (Project 2), and on Kubernetes/k3s (Project 3) — one app, carried up the entire stack. Phase 5 turned the work into a resume, a polished GitHub, and interview answers.

Tonight it converges. At 00:14 a single security-group change revoked tcp/443 ingress on sg-web — a fat-finger meant for sg-bastion. The load-balancer health checks to the web tier went red, so Route 53 failed traffic over to the passive region. And in that passive region the web pods are stuck in ContainerCreating (RESTARTS 0) because the db-credentials Secret they mount was deleted — the kubelet can’t complete volume setup, so the containers never start. One change, three symptoms, all four layers you studied — DNS, AWS networking, Kubernetes, and containers — alight at the same moment.

The whole capstone is one word: order. You fix upstream first. Re-open the network path, then the thing behind it can recover, then you confirm the top of the stack healed. Recreate the Secret before the security group is open and it does nothing — the cluster API is unreachable until the network is back. So the chain is aws → kubectl → dig, never the reverse.

Real world: A power cut hits a hospital and every floor goes dark. You don’t run floor to floor flipping switches — you go to the main breaker first, throw it, and the whole building comes back at once. Flip a third-floor switch while the main is off and nothing happens: you’ve operated on a symptom upstream of nothing. Tonight the revoked security group is the main breaker — everything downstream stays dark until you throw it, in that order.

A named example makes the stakes real. On 28 February 2017 an Amazon engineer debugging the S3 billing system in AWS us-east-1 ran a routine command to remove a few servers; a typo removed far more than intended, two core S3 subsystems had to fully restart, and a long list of services that leaned on S3 went down for hours. The trigger was a fat-finger exactly like tonight’s, and the recovery was the same shape: find the upstream cause, restore it, and let the cascade heal in order. Big outages are rarely exotic — they’re a small change meeting a chain of dependencies. Tonight you run that chain backwards to put it out.

Hands-On Lab

Section 2 of 5 · ~2 min

Today the lab is the mission. No WSL2, no setup, nothing to paste — The Midnight Outage runs entirely in your browser.

This is the final boss, and it closes the whole program. It’s 00:14, a SEV-1 pager just went off, and every region is returning 5xx. One security-group change knocked the stack over region by region — and it’s your call. Everything you need is something you already learned; play it as the upstream-first walk from the diagram, and let each layer prove itself before you move up to the next:

• Triage the pager — cat ~/incident.log. All regions are 5xx and DNS has already failed traffic over to the passive region (the failover A record 203.0.113.99). Read it top to bottom: the earliest line is nearest the cause (Phase 1’s habit of reading a log until it talks).
• Find the network block — cat ~/aws/sg-audit.txt. A 00:14 change REVOKEd tcp/443 ingress on sg-web — a fat-finger meant for sg-bastion. That is why the load-balancer health checks went red and tripped the failover. This is the root cause (Phase 3’s security groups).
• Re-open the security group — aws ec2 authorize-security-group-ingress --group-id sg-web --protocol tcp --port 443 --cidr 0.0.0.0/0. This re-opens the network path to the web tier. Nothing downstream can recover until this lands.
• Read the failover-cluster events — cat ~/k8s/events.txt. The passive-region pods are stuck in ContainerCreating (RESTARTS 0) on a FailedMount ... secret "db-credentials" not found — the mounted Secret was deleted, so the container never starts. That’s why the failover region is degraded too (Phase 4’s Secrets).
• Restore the missing Secret — kubectl create secret generic db-credentials --from-literal=password=<pw>. This only takes after the security group is open — run it early and the cluster API is unreachable, so it completes nothing. Order is the whole lesson.
• Confirm traffic is back on the primary — dig shop.opscanopy.io. Until the chain is done it still returns the failover 203.0.113.99; once the Secret is back, DNS heals and it resolves to the healthy primary 203.0.113.10. That — not “pods Running” — is how you know the outage is over.

Type help to see the terminal’s commands and hint if you stall — it nudges without solving. There’s no penalty for poking around; the point is to run the aws → kubectl → dig chain with your own hands until the order is reflex, and to feel why fixing downstream first fixes nothing. Bring it home in the optimal command count if you want the bragging rights — then come back and read the send-off in Go Deeper. You earned it.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the incident-response mistakes the mission is built to rehearse — the ones that turn a ten-minute recovery into an hour. Read each slowly; the skill is catching yourself before you make them at 00:14.

Common error: Fixing the downstream symptom before the upstream cause — recreating the db-credentials Secret while the security group is still revoked:

kubectl: the cluster API is unreachable — the network path is still blocked upstream.
Re-open the security group first (tcp/443 on sg-web), then recreate the Secret.

Why: The Secret and the crash-looping pods are downstream of the revoked security group. With tcp/443 still blocked, the cluster API isn’t even reachable, so kubectl create secret completes nothing — no Secret, no rollout. The operator burns time on the visible symptom while the actual cause sits one layer up, untouched.

Fix: Work strictly upstream-first. Re-open the network path with aws ec2 authorize-security-group-ingress ... --port 443 first; only then does recreating the Secret take and the deployment roll out 3/3. Read the incident top-down to find the earliest event, and fix in that order.

How you’d spot it in prod: You keep re-applying a downstream fix and it keeps “not sticking” — the change reports success but the component stays broken, or the API call times out. That’s the tell that something upstream is still blocking the path; stop, walk up the dependency chain, and fix the cause.

Common error: Treating the DNS failover as the bug instead of a symptom — trying to force traffic back to the primary while it’s still broken:

shop.opscanopy.io.  60  IN  A  203.0.113.99   (failover, unhealthy)
;; Still failed over — the primary health check has not recovered yet.

Why: Route 53 failed over because the primary’s health checks went red — the failover is the system protecting users, working as designed. Rolling DNS back by hand just points traffic at a region that is still down, and the health check would flip it away again anyway. The failover is a signal pointing at the real fault, not the fault itself.

Fix: Read the failover as a diagnosis: the primary is unhealthy — go find why (here, the revoked security group). Fix the primary, let its health checks recover, and DNS fails back on its own. Confirm with dig, don’t force it.

How you’d spot it in prod: Someone proposes “just flip DNS back” as the fix. That treats the safety mechanism as the problem. The right move is to make the primary healthy and let automated failback do its job — the failover was buying you time, not causing the outage.

Common error: Declaring victory on “pods are Running” without confirming DNS has healed back to the primary:

deployment/web rolled out (3/3 ready).
# ...but:
shop.opscanopy.io.  60  IN  A  203.0.113.99   (failover, unhealthy)

Why: Green pods are one component looking healthy, not the customer-facing symptom being gone. The pods can be 3/3 in the failover region while DNS still points users at the failover address — the top of the stack hasn’t recovered, so the incident is not over even though a dashboard is green.

Fix: Verify at the top of the stack, not the middle. The incident closes only when dig shop.opscanopy.io returns the healthy primary 203.0.113.10. Confirm the user-visible path end to end before you stand down and start the postmortem.

How you’d spot it in prod: An incident gets called “resolved” on a component metric (pods ready, CPU normal) while users still report errors. Always close on the customer-facing signal — the thing the pager fired on — not the first green light on the way there.

Incident Response Interview Questions

Section 4 of 5 · ~1 min

These are the calm-under-fire questions a hiring manager uses to tell a memorizer from someone who has actually run an incident — cover each answer, say your own version aloud first, then compare.

Go Deeper

Section 5 of 5 · ~1 min

What’s next. That’s it. Ninety days, one terminal session at a time, and you closed it by putting out a full-stack fire. This isn’t a to-do list for tonight — it’s how you keep the momentum after Day 90:

• 10 min — Replay The Midnight Outage and beat the optimal command count. Solving it upstream-first fast is the difference between reading incident response and owning it.
• 30 min — Keep building: pick one linkstash project and add the thing you cut for time — monitoring, an autoscaler, a second environment, a proper CI gate — then write up what changed. A portfolio that keeps moving beats one that froze on Day 85.
• This week — Contribute to open source: find a “good first issue” on a tool you already use (Docker, kubectl, Terraform, Astro), fix a doc or a small bug, and open your first external pull request. It’s the fastest credibility you can earn.
• This month — Aim at one certification only if a job you want asks for it: the CKA for Kubernetes or the AWS Solutions Architect Associate. Book a date so it’s real, and let the exam objectives drive your revision — don’t collect badges for their own sake.
• Ongoing — Run the job hunt like a pipeline: apply in small daily batches, tailor each resume to the posting using days 86–89, track every application, and treat each interview as a free rehearsal. You are genuinely ready — now go get paid for it.