Project 2, Day 3: deploy containers behind a load balancer

Wiring the tiers: ALB in front, Fargate in the middle, RDS behind

Section 1 of 5 · ~3 min

Days 1–2 you drew the architecture and laid the plumbing: a VPC with two public and two private subnets across two Availability Zones, a NAT gateway, IAM roles, and three security groups. Today all of it earns its keep — you deploy linkstash, the URL shortener from Project 1, as a real three-tier service on AWS.

The shape is the canonical AWS web-app pattern, and every piece has one job:

• ECR holds the image. You move linkstash:v1.0.0 — the one you shipped to GHCR on Day 45 — into a private repository in your account so Fargate pulls it fast in-region without depending on GHCR on every deploy. (A truly egress-free pull needs an S3 gateway endpoint plus ECR interface VPC endpoints; without them the pull traverses the Day-2 NAT gateway at ~$0.045/GB.)
• An Application Load Balancer lives in the public subnets. It is the only thing with a public face; users hit its DNS name.
• An ECS Fargate service runs the container in the private subnets, with no public IP — the only way in is through the ALB.
• RDS Postgres (db.t4g.micro) sits in the private subnets too, reachable only by the app.

Two connectors make it work: the target group and DATABASE_URL. The ALB doesn’t know about tasks directly — it forwards to a target group, and the ECS service registers each task’s IP into that group as it starts and deregisters it as it stops. The ALB health-checks /healthz on every target and routes only to those returning 200 — the health route you added on Project 1, Day 1, finally doing its job. Because Fargate uses awsvpc networking (each task gets its own ENI and IP), the target group’s type is ip, not instance.

The app finds its database through an environment variable: you read the RDS endpoint after the instance is created and set DATABASE_URL=postgresql://…@<endpoint>:5432/linkstash in the task definition, next to an awslogs log configuration so the container’s stdout streams to a CloudWatch log group.

Security groups keep the tiers honest by chaining: the ALB SG allows 80/443 from the internet; the app SG allows port 8000 only from the ALB SG; the RDS SG allows 5432 only from the app SG. Nothing is reachable except from the tier directly in front of it.

Real world: think of a restaurant. The ALB is the host at the door who greets every guest and walks them to a free table; the Fargate tasks are tables in a private room the public never wanders into; RDS is the kitchen, which only the waiters (the app) may enter. The health check is the host refusing to seat anyone at a table that isn’t ready.

Vanguard, the investment firm, is a widely-cited example: it moved its containerized services onto ECS Fargate behind Application Load Balancers so its teams could ship without owning the servers underneath, letting ECS register and health-check tasks automatically. The pattern you assemble by hand today is the same one it runs at scale.

Today you push the image, stand up RDS, register the task definition, create the ALB and service, and curl the ALB to shorten a real URL — the first time linkstash answers from the cloud.

Hands-On Lab

Section 2 of 5 · ~5 min

Budget about 30 minutes, most of it waiting on RDS. You’re driving the AWS CLI v2 as your IAM user (not root) in us-east-1, with Docker running locally, and the Day-2 VPC, subnets, security groups, NAT gateway and ecsTaskExecutionRole already in place. Account IDs, ARNs, subnet/SG IDs, the RDS endpoint, the ALB DNS name and image digests are unique to your account — yours will differ.

What this costs: This is the spend day — unlike earlier AWS labs you do not tear down at the end, because Day 4 builds on this stack, so it bills continuously until tomorrow’s teardown. Rough us-east-1 rates: the ALB ~$0.0225/hr plus a little per-LCU, the 0.5 vCPU / 1 GB Fargate task ~$0.025/hr, db.t4g.micro RDS ~$0.016/hr (RDS has a 12-month free-tier allowance on eligible legacy accounts; accounts created since mid-2025 get credits instead), the Day-2 NAT gateway ~$0.045/hr, and public IPv4 at $0.005/hr each. All in, roughly 8–12 cents an hour — a couple of dollars if you leave it up overnight. That’s expected: Day 4 ends by deleting every piece. If you must pause, aws ecs update-service --desired-count 0 and stop the RDS instance, but the stack is meant to stay up until tomorrow.

# 1. Confirm identity + region, then load the Day-2 VPC, subnets, and security groups by tag.
aws sts get-caller-identity --query Arn --output text
export AWS_DEFAULT_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
VPC=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=m90-vpc --query 'Vpcs[0].VpcId' --output text)
PRIV=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=tag:Name,Values='m90-private-*' --query 'Subnets[].SubnetId' --output text | tr '\t' ',')
PUB=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC Name=tag:Name,Values='m90-public-*' --query 'Subnets[].SubnetId' --output text | tr '\t' ',')
ALB_SG=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values=$VPC Name=group-name,Values=m90-alb-sg --query 'SecurityGroups[0].GroupId' --output text)
APP_SG=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values=$VPC Name=group-name,Values=m90-app-sg --query 'SecurityGroups[0].GroupId' --output text)
RDS_SG=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values=$VPC Name=group-name,Values=m90-rds-sg --query 'SecurityGroups[0].GroupId' --output text)
echo "priv=$PRIV pub=$PUB alb=$ALB_SG app=$APP_SG rds=$RDS_SG"
# Output (every ID is yours):
# arn:aws:iam::123456789012:user/devops-you
# priv=subnet-0p1,subnet-0p2 pub=subnet-0u1,subnet-0u2 alb=sg-0alb app=sg-0app rds=sg-0rds

# 2. Create a private ECR repo, then move the Day-45 image from GHCR into it.
REPO=$(aws ecr create-repository --repository-name linkstash \
  --query 'repository.repositoryUri' --output text)
docker pull ghcr.io/pushkar/linkstash:v1.0.0
aws ecr get-login-password | docker login --username AWS \
  --password-stdin "$ACCOUNT.dkr.ecr.us-east-1.amazonaws.com"
docker tag ghcr.io/pushkar/linkstash:v1.0.0 "$REPO:v1.0.0"
docker push "$REPO:v1.0.0"
echo "$REPO"
# Output (Login Succeeded, then layers pushed; digest and account are yours):
# Login Succeeded
# v1.0.0: digest: sha256:9f2c1a...e7 size: 1996
# 123456789012.dkr.ecr.us-east-1.amazonaws.com/linkstash

# 3. Put RDS in the private subnets: a DB subnet group, then a db.t4g.micro Postgres.
# (No --engine-version, so RDS provisions its current default Postgres 16.x minor — pinned minors get deprecated.)
aws rds create-db-subnet-group --db-subnet-group-name m90-db-subnets \
  --db-subnet-group-description "linkstash private subnets" \
  --subnet-ids $(echo $PRIV | tr ',' ' ') >/dev/null
aws rds create-db-instance --db-instance-identifier m90-linkstash \
  --engine postgres --db-instance-class db.t4g.micro \
  --allocated-storage 20 --db-name linkstash \
  --master-username linkstash --master-user-password 'ChangeMe_M90!' \
  --db-subnet-group-name m90-db-subnets --vpc-security-group-ids "$RDS_SG" \
  --no-publicly-accessible --query 'DBInstance.DBInstanceStatus' --output text
# Output:
# creating

# 4. Wait until the database is available (a few minutes), then read its endpoint into DATABASE_URL.
aws rds wait db-instance-available --db-instance-identifier m90-linkstash
ENDPOINT=$(aws rds describe-db-instances --db-instance-identifier m90-linkstash \
  --query 'DBInstances[0].Endpoint.Address' --output text)
DB_URL="postgresql://linkstash:ChangeMe_M90!@$ENDPOINT:5432/linkstash"
echo "$ENDPOINT"
# Output (your RDS endpoint hostname will differ):
# m90-linkstash.abcdefg1234.us-east-1.rds.amazonaws.com

# 5. Create a log group, then register the Fargate task definition (image, DATABASE_URL, awslogs).
aws logs create-log-group --log-group-name /ecs/linkstash 2>/dev/null || true
EXEC_ROLE=$(aws iam get-role --role-name ecsTaskExecutionRole --query 'Role.Arn' --output text)
cat > taskdef.json <<EOF
{ "family": "linkstash",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512", "memory": "1024",
  "executionRoleArn": "$EXEC_ROLE",
  "containerDefinitions": [
    { "name": "web", "image": "$REPO:v1.0.0", "essential": true,
      "portMappings": [ { "containerPort": 8000 } ],
      "environment": [ { "name": "DATABASE_URL", "value": "$DB_URL" } ],
      "logConfiguration": { "logDriver": "awslogs",
        "options": { "awslogs-group": "/ecs/linkstash",
          "awslogs-region": "us-east-1", "awslogs-stream-prefix": "web" } } } ] }
EOF
aws ecs register-task-definition --cli-input-json file://taskdef.json \
  --query 'taskDefinition.taskDefinitionArn' --output text
# Output (the ":1" is the revision — yours will differ):
# arn:aws:ecs:us-east-1:123456789012:task-definition/linkstash:1

# 6. Create the public ALB, a target group (type ip for Fargate), and an HTTP listener.
ALB_ARN=$(aws elbv2 create-load-balancer --name m90-alb --type application \
  --subnets $(echo $PUB | tr ',' ' ') --security-groups "$ALB_SG" \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)
TG_ARN=$(aws elbv2 create-target-group --name m90-linkstash-tg \
  --protocol HTTP --port 8000 --vpc-id "$VPC" --target-type ip \
  --health-check-path /healthz \
  --query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" \
  --protocol HTTP --port 80 \
  --default-actions Type=forward,TargetGroupArn="$TG_ARN" \
  --query 'Listeners[0].ListenerArn' --output text
# Output (ARNs are yours):
# arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/m90-alb/50dc.../8f2...

# 7. Create the cluster, then a Fargate service in the PRIVATE subnets wired to the target group.
aws ecs create-cluster --cluster-name m90 --query 'cluster.clusterName' --output text
aws ecs create-service --cluster m90 --service-name linkstash \
  --task-definition linkstash --desired-count 1 --launch-type FARGATE \
  --load-balancers "targetGroupArn=$TG_ARN,containerName=web,containerPort=8000" \
  --network-configuration "awsvpcConfiguration={subnets=[$PRIV],securityGroups=[$APP_SG],assignPublicIp=DISABLED}" \
  --query 'service.serviceName' --output text
# Output:
# m90
# linkstash

# 8. Wait for the service to stabilise (task pulled, registered, health-check green), then curl /healthz.
aws ecs wait services-stable --cluster m90 --services linkstash
DNS=$(aws elbv2 describe-load-balancers --load-balancer-arns "$ALB_ARN" \
  --query 'LoadBalancers[0].DNSName' --output text)
curl -s "http://$DNS/healthz"
# Output (the ALB DNS name is yours):
# {"status":"ok"}

# 9. Shorten a real URL through the ALB, then follow the code to prove the redirect works.
CODE=$(curl -s -X POST "http://$DNS/shorten" -H 'Content-Type: application/json' \
  -d '{"url":"https://opscanopy.com/mission-90/"}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["code"])')
echo "code=$CODE"
curl -si "http://$DNS/$CODE" | grep -iE 'HTTP/|^location:'
# Output (your code differs; the 307 points back at the original URL):
# code=ab12cd
# HTTP/1.1 307 Temporary Redirect
# location: https://opscanopy.com/mission-90/

# 10. Confirm the container's stdout is reaching CloudWatch (the awslogs driver at work).
aws logs tail /ecs/linkstash --since 5m | tail -n 3
# Output (your task ID, IP and timestamps differ):
# ... web/linkstash/1a2b  INFO: 10.0.12.7:0 - "GET /healthz HTTP/1.1" 200 OK
# ... web/linkstash/1a2b  INFO: 10.0.12.7:0 - "POST /shorten HTTP/1.1" 200 OK
# ... web/linkstash/1a2b  INFO: 10.0.12.7:0 - "GET /ab12cd HTTP/1.1" 307 Temporary Redirect

Read that back: {"status":"ok"} came from a container with no public IP, running in a private subnet, pulled from your own registry, reached only through the ALB — and the 307 proves it is talking to RDS, not a stub. linkstash is live on AWS. Leave everything running: tomorrow you give it a real domain and TLS, wire CloudWatch alarms, and only then tear the whole stack down.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three catch almost everyone the first time they wire a Fargate service to an ALB and a database. Read the error text slowly — parsing it is the actual skill.

Common error: The service never reaches a steady state; describe-services shows tasks starting and stopping, and the events name the target group:

(service linkstash) (port 8000) is unhealthy in (target-group m90-linkstash-tg) due to (reason Health checks failed)

Why: The ALB’s health check reaches a task over the network, but the app security group allows nothing inbound on 8000 from the ALB security group. The check times out, the target is marked unhealthy, ECS kills and replaces the task, and the loop repeats. The container is fine — the packet never arrives.

Fix: Authorize the app SG to accept 8000 from the ALB SG by reference: aws ec2 authorize-security-group-ingress --group-id "$APP_SG" --protocol tcp --port 8000 --source-group "$ALB_SG". This is the ALB→app link of the Day-2 chain; confirm it exists before blaming the health-check path.

How you’d spot it in prod: A service stuck cycling tasks with “Health checks failed” and no application error in the logs is almost always a network-path problem — security group, subnet route, or a health-check path that returns a non-2xx — not a crashing container.

Common error: The task runs and passes health checks, but the CloudWatch logs show the app failing to reach Postgres on startup:

psycopg.OperationalError: connection to server at "m90-linkstash...rds.amazonaws.com"
  (10.0.20.4), port 5432 failed: Connection timed out

Why: The RDS security group doesn’t allow 5432 from the app security group, so the database silently drops the connection. A timeout (not “connection refused” or “password authentication failed”) is the tell-tale of a security-group or routing block — the packets never reach the database engine.

Fix: Add the app→RDS rule: aws ec2 authorize-security-group-ingress --group-id "$RDS_SG" --protocol tcp --port 5432 --source-group "$APP_SG". Keep RDS --no-publicly-accessible; the only source that should ever reach 5432 is the app SG.

How you’d spot it in prod: A connection timeout to a database points at the network (security group / NACL / route), while “password authentication failed” points at credentials and “connection refused” points at the DB being down or the wrong port. Read which of the three you got before you start debugging.

Common error: create-service is rejected outright because the target group was created with the default target type:

An error occurred (InvalidParameterException) when calling the CreateService operation:
The provided target group has target type instance, which is incompatible with the awsvpc network mode associated with the task definition.

Why: A Fargate task uses awsvpc networking — each task gets its own ENI and IP — so the ALB must target IPs, not EC2 instances. create-target-group defaults to --target-type instance; leaving that default off makes the group incompatible with the task definition.

Fix: Recreate the target group with --target-type ip (lab step 6), point the listener at the new group, and create the service again. You can’t switch an existing target group’s type in place.

How you’d spot it in prod: Any “incompatible with the awsvpc network mode” error means an instance-type target group is being pointed at Fargate (or an EC2-launch task in awsvpc mode). The fix is always a new ip target group, never a task-definition change.

Load Balancer & ECS Service Interview Questions

Section 4 of 5 · ~1 min

These four are what a screening round asks once “can you run a container?” becomes “can you deploy a real service on AWS?” — cover each answer, say your own version out loud first, then compare, because recalling before revealing is what makes it stick. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 10 min — Move the database password out of the task definition: create a Secrets Manager secret and reference it through the task definition’s secrets block instead of environment. Read the ECS sensitive-data / secrets guide — it’s the production fix for interview question 3.
• 10 min — Scale the service: aws ecs update-service --cluster m90 --service linkstash --desired-count 2 and watch ECS register a second task in the other AZ. Read target group health, then refresh the ALB DNS a few times to see both tasks answer.
• 10 min — Preview tomorrow: read the ALB CloudWatch metrics list, then eyeball RequestCount, TargetResponseTime and UnHealthyHostCount for your target group in the console — the signals you’ll alarm on Day 4.