Week 8 review + Database Recovery

The Week-8 muscle: build the cloud around your app, then recover it when it breaks

Section 1 of 5 · ~3 min

Week 7 gave you a safe account and a running box. Week 8 built the rest of the cloud around that box — the network, the storage, the database, and the traffic front door — and today you prove you can bring one of those pieces back from the dead.

Six days of architecture, in one breath:

The network (days 50–51). A VPC is your own private slice of AWS, carved into public subnets (routed to an internet gateway) and private subnets (no inbound), with a NAT gateway so private instances call out without being reachable — the bastion pattern that keeps a database off the public internet.

The storage (day 52). S3 keeps objects in buckets. You locked buckets private by default, wrote a bucket policy, served a static site, and set a lifecycle rule to expire or tier old objects so storage cost never creeps.

The database (day 53). RDS runs managed Postgres or MySQL for you — AWS handles patching, failover, and backups. Automated snapshots plus point-in-time recovery are the safety net today’s mission leans on.

The traffic (days 54–55). A load balancer spreads requests across instances while an auto scaling group adds and removes them with demand; Route 53 is DNS, and ACM issues the TLS certificate that puts the padlock on your domain.

Week 8’s real lesson: the cloud is a system, not a server — and the test of any system is not how it runs on a good day but whether you can recover it on a bad one.

Real world: An RDS snapshot restore is a spare-tyre change on the motorway, not a roadside repair. You don’t patch the blown tyre where it burst — you bolt on the intact spare (a fresh instance from a pre-drop snapshot) and drive to safety. The shredded tyre (the dropped table) is beyond saving; the whole skill is choosing a good spare and checking the bolts are tight before you pull back into traffic.

GitLab’s 2017 production outage is the case every ops engineer learns from: during an incident an engineer deleted the wrong database directory, and the team then discovered several of their backup methods hadn’t actually been working — the restore that saved them came from a snapshot taken by luck hours earlier. The lesson wasn’t “don’t make mistakes”; it was that a backup you have never restored is only a rumour. Today you make the restore itself the muscle.

That’s why today closes week 8 as a mission, not a lecture: reading about snapshots doesn’t build the reflex — restoring one under a SEV-1 does.

What this costs: ₹0 — Database Recovery runs entirely in your browser and touches no AWS account. The real money is in what you built days 50–55, so delete anything still running: a load balancer (EC2 → Load Balancers) is not free (~₹1,500+/mo), a NAT gateway (VPC → NAT Gateways) bills ~$40+/mo and can only be deleted (not stopped), a spare RDS instance outside the free tier bills silently, and a Route 53 hosted zone is ~$0.50/mo per zone.

Hands-On Lab

Section 2 of 5 · ~2 min

Today the lab is the mission. No AWS account to touch, no instance to restore for real, nothing that can bill you — Database Recovery runs entirely in your browser.

This is your Week-8 boss fight, and it closes the architecture half of Phase 3. It’s 14:02, a SEV-1 is open, and every /checkout call is throwing 500s because the orders table is simply gone — an intern ran DROP TABLE orders in prod. There is no undo for a DROP. What there is: RDS automated snapshots, and the newest one from before 14:02 still holds the table whole. Everything you need lives in ~/db, and every move maps to something you already learned in days 50–55, so play it as the walk from the diagram:

• Confirm what was lost — cd ~/db and read incident.log. The audit line is unambiguous: DROP TABLE orders at 14:02:07, then the app 500-ing on relation "orders" does not exist (day 53’s RDS mental model of where the data lives).
• Pick a snapshot from before the drop — read snapshots.txt. The newest pre-14:02 snapshot is rds:prod-db-2026-07-10-06-00. The tempting prod-db-manual-2026-07-10-1410 is a trap: it was grabbed at 14:10, after the drop, so the table is already gone inside it. (Manual snapshots carry no rds: prefix — that prefix marks AWS’s automated backups, so the name itself signals which is which.)
• Restore to a fresh instance — RDS never restores in place; it builds a new instance from the snapshot (day 53). Grab the right snapshot id and run the restore.
• Verify with psql — before you tell anyone it’s fixed, prove it: select count(*) from orders; must return the rows, not relation "orders" does not exist.

# The move that ends the outage: restore the NEWEST pre-drop snapshot into a
# fresh instance (RDS never restores in place — you always get a new box).
# (snapshot id yours will differ — pick the newest one from BEFORE the drop)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier prod-db-restored \
  --db-snapshot-identifier rds:prod-db-2026-07-10-06-00
# Then VERIFY before you call it fixed:
psql -c "select count(*) from orders;"
#   count
# ---------
#  1240132
# (1 row)

Verify, don’t assume: a restore that reports available only means the instance booted — the row count is what proves the table is actually back. Type help in the terminal to see the supported commands, and hint if you stall — it nudges without solving. There’s no penalty for poking around; the whole point is to run the confirm → pick → restore → verify walk with your hands until it’s reflex, and to feel why the verify step is the one you can never skip under pressure.

When the row count comes back and the crisis clears, come back and note below which step you’d have rushed under a real SEV-1 — that reflection is where the lesson sticks.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These are the mistakes that trip people up when they run a restore for real on RDS with the AWS CLI v2 — the same moves the mission rehearses. Read the error text slowly; parsing it is the skill.

Common error: Restoring a snapshot that was taken after the drop, then finding the table still missing on verify:

ERROR:  relation "orders" does not exist

Why: The restore itself succeeds — the CLI is happy — but the snapshot was captured after the DROP TABLE, so the table was already gone at the moment the snapshot froze. You faithfully restored a database that never had the table. This is exactly why the verify step exists.

Fix: Restore the newest snapshot from before the incident — here rds:prod-db-2026-07-10-06-00, not the manual one grabbed at 14:10. Check each snapshot’s SnapshotCreateTime against the drop timestamp before you restore, and always confirm with a query afterward.

How you’d spot it in prod: A restore that “worked” but leaves the app still erroring on a missing object almost always means the recovery point was on the wrong side of the incident. Line the backup’s timestamp up against the incident timeline before you trust it.

Common error: Passing a snapshot id that doesn’t exist in the region — a typo, or a snapshot that lives in a different region:

An error occurred (DBSnapshotNotFound) when calling the RestoreDBInstanceFromDBSnapshot operation: DBSnapshot not found: rds:prod-db-2026-07-10-0600

Why: Snapshots are regional and RDS matches the identifier exactly. A dropped digit, the wrong separator, or a CLI pointed at the wrong default region all mean the id genuinely isn’t there — so RDS refuses before it starts building anything.

Fix: List first with aws rds describe-db-snapshots and copy the DBSnapshotIdentifier verbatim, and pass --region if the snapshot lives somewhere other than your default. Then re-run the restore with the exact id.

How you’d spot it in prod: DBSnapshotNotFound for a snapshot you can see in the console is almost always a region mismatch or a copy-paste slip — check the region the CLI is targeting before assuming the snapshot was deleted.

Common error: The restore lands, but psql can’t reach the new instance to verify it:

psql: error: connection to server at "prod-db-restored.xxxx.us-east-1.rds.amazonaws.com" (10.0.3.14), port 5432 failed: Connection timed out

Why: A restored instance comes up with its own endpoint and, depending on how you launched it, its own security group and subnet — it does not inherit an open path to your client. If the DB security group doesn’t allow port 5432 from your source, the connection just hangs (week 8’s VPC and security-group rules, days 50–51).

Fix: Allow inbound TCP 5432 to the instance’s security group from your address (never 0.0.0.0/0), and confirm the instance sits in a subnet your client can actually reach. A timeout points at the network path, not at credentials.

How you’d spot it in prod: A verify query that times out (rather than being refused or rejected on auth) is a network problem — a security group, route, or subnet — not a bad password. “Timed out” means nothing answered; check the path before the credentials.

Database Recovery Interview Questions

Section 4 of 5 · ~1 min

The “someone dropped a prod table — recover it” scenario and the snapshot-versus-PITR basics below are among the most common Phase-3 cloud screening questions — a calm answer that restores to a fresh instance and verifies before cutting over beats a fast one every time. The answer bank renders right after this note. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~40 more minutes:

• 5 min — Replay Database Recovery and restore the right snapshot in as few steps as you can — confirm → pick → restore → verify, cold. Speed here is just knowing the recovery walk by heart.
• 10 min — In your own account (free tier), take a manual snapshot of a db.t3.micro and read its automated backup window and retention period settings. Taking a snapshot is free, and it’s the exact move the mission drills.
• 10 min — Read the RDS point-in-time recovery docs and note how PITR restores to any second in the retention window — the technique that shrinks your RPO from hours to seconds versus a nightly snapshot.
• 15 min — Skim the Reliability pillar of the AWS Well-Architected Framework on backup and restore, and note its core rule: a backup is only proven when you have actually restored from it — the discipline today’s mission builds.