Deploy strategies — rolling, blue-green, canary

Rolling, blue-green and canary: three ways to ship without downtime

Section 1 of 5 · ~3 min

You can build and push an image on every commit (Day 34). The last question is how to swap the running version for the new one without dropping traffic — and how to get back fast when the new one is bad. Three strategies cover almost every deploy, and they trade the same three things differently: risk, cost, and rollback speed.

Rolling replaces instances a few at a time. If you run four copies, the orchestrator kills one old copy, starts one new one, waits for it to pass a health check, then moves to the next. Traffic never stops because most copies are always up, and it needs no extra machines — you reuse the same pool — so it’s the cheapest. The catch: for a while old and new run side by side serving real users, and rollback means rolling backward through the same slow cycle. This is the default — a Kubernetes Deployment uses RollingUpdate out of the box.

Blue-green runs two full environments. Blue is live; you deploy the new version to an idle green, health-check it in private, then flip a router so 100% of traffic moves to green in one step. Rollback is the fastest possible — flip back to blue, which is still running, in seconds. The price is literal: you pay for two full environments during the switch.

Canary sends a small slice of live traffic — 1%, then 10%, then 50% — to the new version while everyone else stays on the old. You watch error rate and latency on that slice; if it stays healthy you widen it, if it degrades you cut it to zero. It catches a bad release with the least blast radius, but it’s the most machinery to run: traffic splitting, plus metrics good enough to judge a small sample.

Real world: Think of reopening a bridge after repairs. Rolling is opening it one lane at a time while cars keep crossing. Blue-green is building a second bridge beside it and switching every car over at once — instant to switch, instant to switch back, but you paid for two bridges. Canary is letting ten cars cross first and watching for cracks before you wave the rest through.

Strategy     How traffic reaches the new version (v2)            Rollback
──────────   ─────────────────────────────────────────────────  ──────────────────
Rolling      v1 v1 v1 v1 → v1 v1 v1 v2 → v2 v2 v2 v2             roll backward through
             replace one at a time, health-check each            the same slow cycle
Blue-green   BLUE(v1) live ┊ green(v2) idle → flip → green live   flip to BLUE (seconds)
Canary       90% → v1  ┊  10% → v2   widen if healthy             cut the slice to 0%

Health checks are what make rollback automatic

None of this is safe without a health check — an endpoint like /healthz the deploy polls before sending real traffic to a new instance. A rolling deploy won’t advance to the next instance until the new one is healthy; a blue-green switch is gated on green passing; a canary is judged on the slice’s health. Wire a threshold to that signal and rollback stops being a 2 a.m. human decision: if error rate crosses a line, the system reverts on its own. Netflix’s Spinnaker does exactly this with automated canary analysis — it scores the canary against a baseline and rolls back without waking anyone. The rule underneath every strategy: never shift traffic to a version that hasn’t proven itself healthy.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 25 minutes. You need Docker and docker compose in your WSL2 Ubuntu 24.04 terminal (Days 22–27). You’ll stand up a tiny two-version service behind an nginx router, then run a health-gated blue-green switch, an instant rollback, and a canary split — the whole traffic story without a cloud account. Image digests and container IDs are unique to each pull — yours will differ from the samples.

# 1. Make a lab dir and write the nginx router config. The upstream lives in its
#    own file so switching traffic is a one-line rewrite + reload.
mkdir -p ~/deploy-lab && cd ~/deploy-lab
cat > nginx.conf <<'EOF'
events {}
http {
  include /etc/nginx/upstream.conf;
  server {
    listen 80;
    location / { proxy_pass http://app; }
  }
}
EOF
echo 'upstream app { server blue:5678; }' > upstream.conf   # live = blue (v1)
ls
# Output:
# nginx.conf  upstream.conf

Save this as compose.yaml. blue and green are the same tiny echo image printing different versions; proxy is the router that decides who gets traffic:

# compose.yaml
services:
  proxy:
    image: nginx:alpine
    ports: ["8080:80"]
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./upstream.conf:/etc/nginx/upstream.conf:ro
  blue:
    image: hashicorp/http-echo
    command: ["-text=v1", "-listen=:5678"]
  green:
    image: hashicorp/http-echo
    command: ["-text=v2", "-listen=:5678"]

# 2. Bring up the router and the current live version (blue), then hit it.
docker compose up -d proxy blue
curl -s localhost:8080
# Output (image pull lines omitted; digests are yours-will-differ):
# v1

# 3. Deploy green (v2) alongside blue — it gets NO live traffic yet — then
#    health-check it privately from the router. THIS is the gate.
docker compose up -d green
docker compose exec proxy wget -qO- http://green:5678
# Output:
# v2

# 4. Green passed, so promote it: point the upstream at green and reload nginx.
#    This is the blue-green switch — 100% of traffic moves in one step.
echo 'upstream app { server green:5678; }' > upstream.conf
docker compose exec proxy nginx -s reload
curl -s localhost:8080
# Output:
# v2

# 5. Roll back INSTANTLY — blue is still running, so it's just a flip back.
echo 'upstream app { server blue:5678; }' > upstream.conf
docker compose exec proxy nginx -s reload
curl -s localhost:8080
# Output:
# v1

# 6. Now a canary: send ~10% of traffic to green, the rest stays on blue.
#    nginx splits by weight across the two upstream servers.
echo 'upstream app { server blue:5678 weight=9; server green:5678 weight=1; }' > upstream.conf
docker compose exec proxy nginx -s reload
# Output (no output on a clean reload):

# 7. Fire 20 requests and count the split — roughly 9:1, the canary weight.
for i in $(seq 20); do curl -s localhost:8080; done | sort | uniq -c
# Output (the split is random around the weight — yours will differ):
#      18 v1
#       2 v2

# 8. Tear the whole lab down.
docker compose down
# Output (container names/IDs are yours-will-differ):
# [+] Running 4/4
#  ✔ Container deploy-lab-proxy-1  Removed
#  ✔ Container deploy-lab-green-1  Removed
#  ✔ Container deploy-lab-blue-1   Removed
#  ✔ Network deploy-lab_default    Removed

Read it back: you ran a new version beside the old one, proved it healthy before a single user saw it, switched 100% of traffic with one reload, rolled back in seconds because the old version was still up, then dripped 10% of traffic to the new version and watched the split. Blue-green, rollback, and canary — the same three moves every real deploy pipeline automates, done by hand so you know what the automation is doing.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three are the traps that turn a “safe” deploy strategy into an outage. Read the error text slowly — parsing it is the skill.

Common error: Pointing the router at a candidate that isn’t running yet, so the reload is rejected:

nginx: [emerg] host not found in upstream "green:5678" in /etc/nginx/upstream.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Why: nginx resolves every upstream server name when it loads the config. If the green container was never started (or crashed), the name doesn’t resolve, the config fails its test, and nginx keeps serving the old config rather than applying a broken one. The switch silently didn’t happen.

Fix: Deploy and health-check the candidate before you point traffic at it — docker compose up -d green, confirm it answers 200, then rewrite the upstream and reload. That ordering (stand up, verify, switch) is the whole point of blue-green.

How you’d spot it in prod: A “completed” deploy where the site is still serving the old version, and the router logs an [emerg] host not found at reload time. The new environment was never actually up when you flipped.

Common error: Switching traffic before the new version is ready, so real users hit a broken backend:

$ curl -s -o /dev/null -w "%{http_code}\n" localhost:8080
502

Why: The upstream name resolved, so nginx accepted the switch — but the backend itself isn’t serving yet (still booting, failing to start, or crashing), so the proxy returns 502 Bad Gateway. Promoting without a passing health check sends live traffic straight into a version that can’t answer.

Fix: Gate every switch on a health probe and only reload once the candidate returns 200; keep the old version running so you can flip back instantly. This is exactly the check in lab step 3 — skip it and step 4 serves 502s.

How you’d spot it in prod: Error rate jumps toward 100% at the exact deploy timestamp, then drops the moment someone rolls back. A spike welded to the deploy marker is a switch that outran readiness.

Common error: A rollback that makes things worse because a database migration wasn’t backward-compatible:

ERROR:  column "full_name" does not exist
LINE 1: SELECT id, full_name FROM users;
                   ^

Why: Rolling and canary run two code versions against one database at the same time, and blue-green rollback assumes the old code still works against current data. If the new release renamed or dropped a column the old code reads, the version that doesn’t match breaks — and flipping back to the old version after a destructive migration can’t fix a schema that no longer has the column.

Fix: Make migrations backward-compatible with expand-then-contract: add the new column, deploy code that writes both old and new, backfill, and only drop the old column a later release once nothing reads it. Never couple a destructive schema change to the same deploy that needs the rollback.

How you’d spot it in prod: Errors that hit only some requests during a rollout (the ones served by the other version), or a rollback that increases errors instead of clearing them. The tell is a schema change shipped in the same release.

Deploy Strategy Interview Questions

Section 4 of 5 · ~1 min

Comparing the three strategies and reasoning about rollback safety are among the most common Phase-2 CD screening questions — a calm answer that names the risk-cost-rollback tradeoff beats reciting definitions. Cover each answer, say your own version out loud first, then compare — recalling before revealing is what makes it stick for interview day. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — In your deploy-lab, stop green (docker compose stop green) and re-run the health-gate wget from step 3 — watch it fail with a non-zero exit. That refusal to promote an unhealthy version is automated rollback in one command.
• 10 min — Read the Kubernetes Deployment rolling-update docs on maxSurge and maxUnavailable, and note how those two knobs trade deploy speed against spare capacity — the same tradeoff you tuned by hand today.
• 15 min — Skim the Argo Rollouts canary overview and map each piece — traffic weight, analysis/metric thresholds, automatic rollback — back to the blue-green and canary steps you ran in the lab.