Week 12 review + Terraform Trouble

The week-12 muscle: expose it, template it, manage it as code

Section 1 of 5 · ~3 min

Week 11 got your app running on Kubernetes. Week 12 made it shippable and repeatable — you stopped clicking things into existence and started declaring them. Six days built that, and today they collapse into one reflex: when infra breaks, read the plan, find what’s blocking the change, clear the blocker, then let the tool reconcile reality to your config.

The week had three movements, and you own each one:

Expose it. Day 74’s Ingress put a real front door on your cluster — one controller routing by hostname and path instead of a NodePort per service. Day 75’s Helm turned a pile of raw manifests into a templated chart with values.yaml, so one release can be installed, upgraded, and rolled back as a unit.

Codify it. Days 76–79 were Terraform. Day 76 drew the core loop — terraform init pulls providers, plan shows the diff between config and reality, apply makes it so, and all of it is tracked in state. Day 77 made configs reusable with variables and outputs, and taught you to respect that state file. Day 78 packaged infra into modules and split environments with workspaces. Day 79 pointed the AWS provider at real cloud and provisioned the project’s infra. (The same commands work in OpenTofu, the open-source fork.)

Two Terraform ideas carry today’s mission, so pin them down now. State is Terraform’s map from your config to the real resources it created; it is the single source of truth, and it is shared — the whole team writes to the same state. To stop two runs corrupting it, Terraform takes a state lock before any write and releases it when the run ends. Drift is the other half: when someone changes a live resource by hand, the world no longer matches the config, and the next plan shows the gap.

Real world: A state lock is the single bathroom key chained to a big wooden block at an old petrol station — only one person can hold it, so two people can’t use the room at once and make a mess. The abandoned lock in today’s mission is that key walking off in someone’s pocket: the room is empty, but nobody else can get in until the key comes back. terraform force-unlock is the manager cutting a spare — you only do it once you are sure the last person really left.

A named example makes the stakes concrete. HashiCorp’s own recommended S3 backend stores the state file in an S3 bucket and, since Terraform 1.10, locks it natively: set use_lockfile = true and the backend writes a small .tflock object into that same bucket with a conditional write, so the object’s own existence is the lock. (Older backends took the lock as a row in a separate DynamoDB table — the dynamodb_table argument, deprecated in Terraform 1.11 and still seen on setups that predate the native lockfile.) When a run dies before releasing that lock — a CI job killed mid-apply, or a laptop that lost Wi-Fi — the lock is stranded: every later apply is refused with Error acquiring the state lock even though nothing is really running. The cure is exactly today’s mission: read the lock info, confirm it is stale, terraform force-unlock <id>, then apply.

That is why today is a mission, not a lecture. Reading about a stranded lock and a drifted instance doesn’t build the reflex — clearing one does. Terraform Trouble hands you a red pipeline: an abandoned lock blocking every apply, and a live instance someone hand-shrank in the console. The order is the whole lesson — unlock first, then reconcile.

Hands-On Lab

Section 2 of 5 · ~2 min

Today the lab is the mission. No WSL2, no terraform init, nothing to paste — Terraform Trouble runs entirely in your browser.

This is your Week-12 boss fight, and it closes the IaC half of Phase 4. It’s 10:20 AM, the pipeline on tf-runner is red, and the change it was meant to ship never landed. Last night a teammate got impatient and ran terraform apply straight from their laptop. That one run left two messes, and everything you need to clean them up is in ~/infra: the plan, the error log, and the config. Play it as the walk from the diagram:

• See the drift — run terraform plan (or cat ~/infra/plan.txt). It shows one in-place change: ~ instance_type = "t3.micro" -> "m5.large". Someone hand-shrank aws_instance.web in the AWS console from m5.large down to t3.micro, so the live world drifted away from the checked-in config, which still says m5.large. Plan: 0 to add, 1 to change, 0 to destroy.
• Find the blocker — reading the plan isn’t enough, because apply won’t even start. cat ~/infra/errors.log and you find why: Error acquiring the state lock. The laptop’s apply died mid-run and never released the lock (ID: 7f3a9b2c, Who: teammate@laptop), so every apply since is refused before it does anything.
• Release the lock — this is the first fix, and it comes first for a reason. terraform force-unlock 7f3a9b2c releases the abandoned lock: “Terraform state lock released (ID 7f3a9b2c).” Try terraform apply before this and the mission refuses it — “Cannot apply: the state is still locked. Release it first with terraform force-unlock 7f3a9b2c.” Apply-before-unlock is the classic mistake, and the terminal blocks it.
• Reconcile — now that the lock is gone, terraform apply can run. It reconciles the drift and puts the instance back the way the config says: “Apply complete! Resources: 0 added, 1 changed, 0 destroyed. aws_instance.web restored to m5.large.” Run terraform plan again and it’s honest — “No changes. Your infrastructure matches the configuration.”

Type help in the terminal to see the supported commands, and hint if you stall — it nudges without solving. The point is to feel the order in your hands: you cannot reconcile through a held lock, so you release the stale lock first and reconcile second. Get the state matching reality — then, if you want the bragging rights, replay it and try to finish in the optimal command count.

When the pipeline finally reports reconciled, come back and note below which step you reached for out of order — that reflection is where the lesson sticks.

Common Errors & Fixes

Section 3 of 5 · ~4 min

These are the mistakes that trip people up running the week-12 Terraform loop for real — the same ones the mission rehearses. The code blocks below mirror the mission’s simulated log rather than verbatim backend output — the wording and IDs are simplified for teaching (the note under the first box shows what a real backend prints). Read the error text slowly; parsing it is the skill.

Common error: Running terraform apply while an abandoned state lock — one whose owning run already died — is still held:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: the lock is already held
Lock Info:
  ID:        7f3a9b2c
  Who:       teammate@laptop
  Created:   2026-07-10 11:24:16 UTC
  Info:      apply run from a laptop — the process died and never released it

Terraform acquires a state lock to protect the state from concurrent writes.

Why: Terraform takes a lock before every write and releases it when the run finishes. A run that dies mid-flight — a laptop that lost Wi-Fi, a killed CI job — never releases it, so the lock is stranded and every later apply is refused before it does anything, even though nothing is actually running.

Fix: Confirm nobody is really running Terraform, then release the stale lock with the exact ID from the error — terraform force-unlock 7f3a9b2c — and re-run terraform apply. The order is fixed: you cannot apply through a held lock, so force-unlock comes first, apply second.

How you’d spot it in prod: Every apply in the pipeline suddenly refused with this exact message, with no pipeline actually running and a Created timestamp hours old, means an earlier run died without releasing the lock. Read the lock info before you clear it.

Real vs. simulated: On a real DynamoDB-backed lock the message reads ConditionalCheckFailedException: The conditional request failed, not the friendlier “the lock is already held” shown above, and the ID is a UUID like a3f8c1e0-9b2d-4f6a-8c7e-1d2b3a4c5d6e. The short 7f3a9b2c is the mission’s simplified stand-in, kept here so the day matches what you type in Terraform Trouble.

Common error: Hand-editing a live resource in the cloud console instead of changing the config, then being surprised when the next plan wants to “undo” it:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # aws_instance.web has changed
  ~ resource "aws_instance" "web" {
      ~ instance_type = "m5.large" -> "t3.micro"
    }

Why: This is drift. Someone changed instance_type directly in the AWS console, so the live resource no longer matches the checked-in config. Terraform refreshes state at plan time, notices the out-of-band change, and — because the config is the source of truth — plans an in-place update to put it back to m5.large.

Fix: Do not hand-edit live infra to “fix” a problem; the next apply will only revert it. Change the resource in the .tf config and run terraform apply so config, state, and reality all agree — exactly what the mission’s main.tf comment insists on.

How you’d spot it in prod: A plan that proposes to change a resource nobody touched in code is out-of-band drift — someone clicked in the console. terraform plan -refresh-only shows the drift on its own so you can decide whether to reconcile it or update the config to match.

Common error: Running terraform apply from a laptop against shared state while a legitimate run is already in progress:

Error: Error acquiring the state lock

Lock Info:
  ID:        3d5e8a11
  Who:       ci-runner-7
  Created:   2026-07-12 10:02:44 UTC
  Info:      apply in progress (pipeline run #588)

Why: Unlike the stale lock above, this one is live — a real apply is running right now (here, CI run #588). Running Terraform from a laptop at the same time contends for the same shared state. Force-unlocking this would let two writers into the state at once and corrupt it.

Fix: Do NOT force-unlock a live lock. Wait for the active run to finish and release the lock on its own, then apply. The deeper fix is process: run apply from CI only, never a laptop, so there is one serialized writer and every run is reviewed and logged.

How you’d spot it in prod: A lock whose Who/Created shows a run that started seconds ago and is still going is a legitimate lock, not an abandoned one. Force-unlocking a live lock because you were impatient is how state-corruption incidents start.

Terraform State & Locking Interview Questions

Section 4 of 5 · ~1 min

State, locking, and drift are the Terraform questions a Phase-4 screen keeps returning to — a calm, ordered answer (unlock only when stale, then apply) beats a clever one. The answer bank renders right after this note; cover each, say your own version out loud first, then compare.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes:

• 5 min — Replay Terraform Trouble and try to clear it in the optimal command count — speed here is just knowing the read → find → unlock → apply order cold.
• 10 min — On a throwaway dir at ₹0, terraform init with the default local backend, apply a single null_resource, then open terraform.tfstate and read it. That JSON is the map Terraform locks before every write.
• 10 min — Read HashiCorp’s State Locking and force-unlock docs — note the warning that force-unlock is only for locks no run still holds.
• 5 min — Skim the terraform plan -refresh-only docs to see how teams detect drift on a schedule, before it turns into a surprise at deploy time.