Project 3, Day 2: write the Terraform foundation

Provisioning a single-node k3s cluster on AWS with Terraform

Section 1 of 5 · ~3 min

Project 3 is the Phase 4 capstone. Over five days you take linkstash — the FastAPI URL shortener you built in Project 1 (image ghcr.io/pushkar/linkstash:v1.0.0, Uvicorn on :8000) — and stand it up on a real cloud Kubernetes cluster using everything Phase 4 taught: Terraform (days 76–80), Kubernetes (66–73), and Helm + Ingress (74–75). Yesterday you designed the architecture and priced it. Today you write the Terraform that turns that design into a running cluster.

One file tree, one apply. Under ~/linkstash/deploy/capstone/terraform/ you describe a small, deliberate AWS footprint: a VPC (10.0.0.0/16), a single public subnet (10.0.0.0/24) in us-east-1a, an internet gateway and a public route table, a security group (k3s-sg), one EC2 instance (t3.small, Ubuntu 24.04) whose user_data installs k3s, and an Elastic IP. The init → plan → apply loop from Day 79 turns those blocks into ten real resources, and the instance boots straight into a one-node cluster.

Why k3s on a single instance, not EKS. A managed EKS control plane costs about $0.10/hour — roughly $73/month — before a single worker node runs. For a five-day capstone you’d rather not pay that, so you run k3s: a fully CNCF-certified Kubernetes distribution packed into one ~60 MB binary, created by Rancher (now part of SUSE) for exactly this kind of lightweight, edge and single-node use. Because it’s certified, kubectl, your YAML manifests and Helm all behave identically to any other cluster — you learn the real API, not a toy. The honest tradeoff: this is one node in one AZ. There’s no control-plane HA and no second zone to fail over to — the opposite of Project 2’s two-AZ design — so it’s perfect for learning and demos, and something you’d never run production on. EKS is the answer when uptime justifies the bill.

Real world: The EKS control plane is a building’s 24/7 staffed front desk — always on, always paid for (~$73/month), whether one visitor arrives or none. Single-node k3s is being your own receptionist in a one-room office: a few dollars a month and perfectly fine for a demo — but if that one room floods, the whole operation stops. You pick the desk when downtime costs more than the salary.

How the cluster installs itself. The instance’s user_data runs one line — curl -sfL https://get.k3s.io | sh - — and k3s bundles the API server, scheduler, containerd, a Traefik ingress controller and a local-path storage class into that single binary. Within about 60–90 seconds of boot you have a working control-plane-and-worker node ready for tomorrow’s manifests.

Getting a kubeconfig you can use. k3s writes a kubeconfig to /etc/rancher/k3s/k3s.yaml, but it points at https://127.0.0.1:6443 — correct on the node, useless from your laptop. So you SSH in, copy it down, and rewrite 127.0.0.1 to the Elastic IP. The subtlety is TLS: the API server certificate must list that IP as a Subject Alternative Name, or kubectl rejects it — which is why the install passes --tls-san <Elastic IP>. Then export KUBECONFIG and kubectl get nodes reaches the cluster over the internet.

By the end of today terraform apply has drawn the whole foundation and one node reports Ready. Tomorrow you put workloads on it.

Hands-On Lab

Section 2 of 5 · ~6 min

What this costs: roughly ₹0 on a free-tier account if you tear it down the same day — but this is the day the meter starts. terraform apply launches a t3.small (~$0.0208/hr, about $15/month if left running 24/7) and allocates an Elastic IP (a public IPv4, billed $0.005/hr ≈ $3.65/month since Feb 2024 whether or not it’s attached). The VPC, subnet, internet gateway, route table and security group are all free to exist, and the 8 GB gp3 root disk is under a dollar a month. Call it ~$19/month all-in if you forget it’s running — the figure from yesterday’s cost table. On a brand-new free-tier account, swapping t3.small for t2.micro keeps this near ₹0. Leave it up only while you work through days 82–85; day 85 ends with terraform destroy.

Budget about 30 minutes. Drive this from your WSL2 Ubuntu 24.04 terminal with Terraform 1.10+, AWS CLI v2 and kubectl installed, authenticated as your IAM user (not root). Terraform IDs, the Elastic IP and the k3s version are unique to your run — yours will differ from the samples.

# 1. Confirm your tools and grab the public IP you'll scope SSH to.
terraform version
aws sts get-caller-identity --query Arn --output text
curl -s https://checkip.amazonaws.com
# Output (your versions, account and IP will differ):
# Terraform v1.15.2
# on linux_amd64
# arn:aws:iam::123456789012:user/devops-you
# 198.51.100.23

# 2. Lay out the capstone repo and pin the provider. Credentials are NOT in these files.
mkdir -p ~/linkstash/deploy/capstone/terraform && cd ~/linkstash/deploy/capstone/terraform
cat > versions.tf <<'EOF'
terraform {
  required_version = ">= 1.10"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}

provider "aws" {
  region = "us-east-1" # creds come from aws configure / env vars, never a key in this file
}

variable "ssh_cidr" {
  description = "Your public IP as a /32 — the only source allowed to SSH"
  type        = string
}
EOF
echo "versions.tf written"
# Output:
# versions.tf written

# 3. The network: a VPC, one public subnet, an internet gateway and a public route table.
cat > network.tf <<'EOF'
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  tags                 = { Name = "linkstash-vpc" }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.0.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
  tags                    = { Name = "linkstash-public-a" }
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "linkstash-igw" }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
  tags = { Name = "linkstash-public-rt" }
}

resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}
EOF
echo "network.tf written"
# Output:
# network.tf written

# 4. The security group: SSH from you only, HTTP/HTTPS from anywhere, the K8s API left closed.
cat > security.tf <<'EOF'
resource "aws_security_group" "k3s" {
  name        = "k3s-sg"
  description = "linkstash k3s node"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "SSH (you only)"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.ssh_cidr]
  }
  ingress {
    description = "HTTP (Traefik)"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description = "HTTPS (Traefik)"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  # 6443 (Kubernetes API) is deliberately NOT opened — reach the cluster over SSH instead.

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  tags = { Name = "k3s-sg" }
}
EOF
echo "security.tf written"
# Output:
# security.tf written

# 5. The instance: Ubuntu 24.04, k3s via user_data, an Elastic IP allocated up front.
cat > compute.tf <<'EOF'
data "aws_ssm_parameter" "ubuntu" {
  name = "/aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id"
}

resource "aws_key_pair" "k3s" {
  key_name   = "linkstash-k3s"
  public_key = file(pathexpand("~/.ssh/id_ed25519.pub"))
}

resource "aws_eip" "k3s" {
  domain = "vpc"
  tags   = { Name = "linkstash-eip" }
}

resource "aws_instance" "k3s" {
  ami                    = data.aws_ssm_parameter.ubuntu.value
  instance_type          = "t3.small"
  subnet_id              = aws_subnet.public.id
  vpc_security_group_ids = [aws_security_group.k3s.id]
  key_name               = aws_key_pair.k3s.key_name

  root_block_device {
    volume_size = 8
    volume_type = "gp3"
  }

  # The EIP is interpolated here, so k3s's API cert is valid for the public address.
  user_data = <<-CLOUD
    #!/bin/bash
    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san ${aws_eip.k3s.public_ip} --node-name linkstash-k3s" sh -
  CLOUD

  tags = { Name = "linkstash-k3s" }
}

resource "aws_eip_association" "k3s" {
  instance_id   = aws_instance.k3s.id
  allocation_id = aws_eip.k3s.id
}

output "k3s_public_ip" { value = aws_eip.k3s.public_ip }
output "ssh_command"   { value = "ssh ubuntu@${aws_eip.k3s.public_ip}" }
EOF
echo "compute.tf written"
# Output:
# compute.tf written

# 6. Download the AWS provider and write the lock file.
terraform init
# Output (your resolved 6.x version will differ — anything matching ~> 6.0 works):
# Initializing the backend...
# Initializing provider plugins...
# - Finding hashicorp/aws versions matching "~> 6.0"...
# - Installing hashicorp/aws v6.9.0...
# - Installed hashicorp/aws v6.9.0 (signed by HashiCorp)
# Terraform has created a lock file .terraform.lock.hcl to record the provider selections...
# Terraform has been successfully initialized!

# 7. Preview. Pass your IP as ssh_cidr so only you can SSH — nothing is created yet.
terraform plan -var="ssh_cidr=$(curl -s https://checkip.amazonaws.com)/32"
# Output (trimmed; the last line is the one that matters):
# data.aws_ssm_parameter.ubuntu: Reading...
# data.aws_ssm_parameter.ubuntu: Read complete after 0s
# Terraform will perform the following actions:
#   # aws_eip.k3s will be created
#   # aws_eip_association.k3s will be created
#   # aws_instance.k3s will be created
#   # aws_internet_gateway.igw will be created
#   # aws_key_pair.k3s will be created
#   # aws_route_table.public will be created
#   # aws_route_table_association.public will be created
#   # aws_security_group.k3s will be created
#   # aws_subnet.public will be created
#   # aws_vpc.main will be created
#
# Plan: 10 to add, 0 to change, 0 to destroy.

# 8. Create it. Type `yes` at the prompt — Terraform now makes the real AWS calls.
terraform apply -var="ssh_cidr=$(curl -s https://checkip.amazonaws.com)/32"
# Output (IDs and the Elastic IP are unique to your run — yours will differ):
#   Enter a value: yes
# aws_vpc.main: Creating...
# aws_eip.k3s: Creating...
# aws_key_pair.k3s: Creating...
# aws_eip.k3s: Creation complete after 1s [id=eipalloc-0abc123def4567890]
# aws_key_pair.k3s: Creation complete after 1s [id=linkstash-k3s]
# aws_vpc.main: Creation complete after 2s [id=vpc-0a1b2c3d4e5f67890]
# aws_internet_gateway.igw: Creation complete after 1s [id=igw-0abc1234]
# aws_subnet.public: Creation complete after 1s [id=subnet-0abc1234]
# aws_security_group.k3s: Creation complete after 2s [id=sg-0abc123def4567890]
# aws_route_table.public: Creation complete after 1s [id=rtb-0abc1234]
# aws_route_table_association.public: Creation complete after 1s [id=rtbassoc-0abc1234]
# aws_instance.k3s: Creating...
# aws_instance.k3s: Creation complete after 13s [id=i-0abc123def4567890]
# aws_eip_association.k3s: Creation complete after 1s [id=eipassoc-0abc1234]
#
# Apply complete! Resources: 10 added, 0 changed, 0 destroyed.
#
# Outputs:
# k3s_public_ip = "203.0.113.24"
# ssh_command = "ssh ubuntu@203.0.113.24"

# 9. Wait ~90s for k3s to boot, then pull its kubeconfig (root-only, so sudo cat) and
#    rewrite the loopback address to the Elastic IP so kubectl connects over the internet.
EIP=$(terraform output -raw k3s_public_ip)
ssh -o StrictHostKeyChecking=accept-new ubuntu@"$EIP" "sudo cat /etc/rancher/k3s/k3s.yaml" > ../k3s.yaml
sed -i "s/127.0.0.1/$EIP/" ../k3s.yaml
grep server: ../k3s.yaml
# Output (the loopback is now your Elastic IP):
# Warning: Permanently added '203.0.113.24' (ED25519) to the list of known hosts.
#     server: https://203.0.113.24:6443

# 10. Point kubectl at that kubeconfig and confirm the single node is Ready.
export KUBECONFIG=~/linkstash/deploy/capstone/k3s.yaml
kubectl get nodes
# Output (AGE and the exact k3s version will differ — this is the K8s 1.31 line):
# NAME            STATUS   ROLES                  AGE   VERSION
# linkstash-k3s   Ready    control-plane,master   92s   v1.31.5+k3s1

Read the last three outputs back: Apply complete! Resources: 10 added, server: https://203.0.113.24:6443, and linkstash-k3s Ready. Those three are the whole point of today — the foundation is real, the kubeconfig reaches it, and the node is up. Keep this terminal (or re-export KUBECONFIG) for the next three days; every manifest, chart and ingress lands on this exact node. And remember the meter is now running — this is not a teardown day, but day 85 is.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three trip up almost everyone the first time Terraform stands up a k3s box. Read the error text slowly — parsing it is the actual skill.

Common error: SSH to the new instance hangs and eventually times out:

ssh: connect to host 203.0.113.24 port 22: Connection timed out

Why: The security group allows port 22 only from the /32 you passed as ssh_cidr. If the IP you actually connect from is different — a corporate VPN, a residential IP that rotated since you ran curl checkip, or an IPv6 egress — the packets are dropped silently at the SG. A timeout (not “connection refused”) is the signature: refused means you reached a closed port; timeout means you never arrived.

Fix: Re-check your real IP with curl -s https://checkip.amazonaws.com, then re-run terraform apply -var="ssh_cidr=<that-ip>/32" — Terraform updates the SG rule in place, no rebuild. Turn off the VPN, or scope to its egress IP instead.

How you’d spot it in prod: A connection timing out on a locked-down port is almost always a security-group, NACL or source-IP mismatch, not a dead service. Check what CIDR the rule allows and what IP you’re really leaving from before you touch the host.

Common error: kubectl connects but rejects the certificate after you rewrite the kubeconfig:

E0712 10:41:07.882123       1 memcache.go:265] couldn't get current server API group list: Get "https://203.0.113.24:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 10.0.0.37, 10.43.0.1, not 203.0.113.24

Why: k3s’s API-server certificate only covers the SANs it knew at install time — loopback, the cluster IP, and the node’s internal address. Rewriting the kubeconfig’s server: line to the Elastic IP doesn’t add that IP to the cert, and kubectl validates the cert against the address it dialled, so it refuses.

Fix: Install k3s with --tls-san <Elastic IP>, exactly as compute.tf does. If a node is already up without it, add tls-san: [ "203.0.113.24" ] to /etc/rancher/k3s/config.yaml and sudo systemctl restart k3s — the cert is regenerated to include the new SAN.

How you’d spot it in prod: x509: certificate is valid for X, not Y always means you connected by a name or IP the server’s cert doesn’t cover. Fix the cert’s SANs (or connect by a covered name) — never paper over it with --insecure-skip-tls-verify on anything real.

Common error: terraform apply fails before any AWS call, on the key-pair resource:

Error: Invalid function argument

  on compute.tf line 7, in resource "aws_key_pair" "k3s":
   7:   public_key = file(pathexpand("~/.ssh/id_ed25519.pub"))

Invalid value for "path" parameter: no file exists at
/home/you/.ssh/id_ed25519.pub; this function works only with files that are
distributed as part of the configuration source code.

Why: The config reads your local public key to register it as the instance’s login key. If you’ve never generated an ed25519 keypair, that file isn’t there, and file() errors during evaluation — before the provider is even invoked, so nothing was created.

Fix: Generate one with ssh-keygen -t ed25519 -C "you@example.com" (accept the default path, set a passphrase), then re-run apply. If your existing key is RSA, point the path at ~/.ssh/id_rsa.pub instead.

How you’d spot it in prod: A Terraform file() / “no file exists” error is a missing local input, not an AWS problem. Because it fails at evaluation time, no resources were touched — fix the path and re-run with a clear conscience.

Terraform & k3s Interview Questions

Section 4 of 5 · ~1 min

These four come up whenever a role mixes infrastructure as code with running Kubernetes — cover the answers, say your own version out loud first, then compare. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — SSH in and run sudo k3s kubectl get pods -A on the node itself; watch Traefik, CoreDNS, local-path-provisioner and metrics-server already Running — the batteries k3s ships that you’ll lean on in days 83–84.
• 10 min — Run terraform state list and terraform show, and find where user_data (with your Elastic IP baked in) is recorded in state — a reminder that *.tfstate can hold sensitive values and must never be committed.
• 5 min — Read the k3s architecture docs and note how one binary embeds the API server, scheduler, containerd and a datastore, versus the separate managed control plane EKS bills for.
• 5 min — Read k3s’s --tls-san and certificate notes, then contrast with how EKS hands you a managed API endpoint and cert you never rotate yourself.
• 10 min — Add a .gitignore for k3s.yaml, *.tfstate* and .terraform/: the kubeconfig carries cluster-admin credentials, so committing it is as bad as committing an AWS key.