Project 3, Day 5: end-to-end run & writeup

Proving it works end to end — then destroying it

Section 1 of 5 · ~2 min

Over four days you designed the capstone (Day 81), let Terraform provision a VPC and a single t3.small running k3s (Day 82), wrote the Kubernetes manifests for an in-cluster Postgres and the linkstash app (Day 83), and packaged linkstash as a Helm chart behind a Traefik ingress with cert-manager TLS (Day 84). Today has three jobs — prove it works, write it up, tear it down — and the last one is the one people skip.

A deploy isn’t done because pods say Running. That only means the container started. It’s done when a real request completes. The end-to-end smoke test walks the whole path: a curl POST to https://links.example.com/shorten creates a short link (a row in the in-cluster Postgres), and a GET of that short URL returns a 307 redirect to the original. Every layer is exercised — DNS resolving your domain to the Elastic IP, Traefik terminating TLS, the linkstash Service routing to the pod on :8000, and the app reading and writing Postgres. The padlock is real too: the certificate is issued by Let’s Encrypt, the nonprofit CA behind the majority of the web’s TLS. If the handshake succeeds, the code comes back, and the redirect lands, the capstone works.

The README is the artifact that outlives the infrastructure. You’ll destroy the cluster within the hour, but the repo — Terraform, K8s manifests, the Helm chart, and a README that explains the architecture — is the portfolio piece you link from your resume. Write it for a reader who has your repo but not your memory.

Then the discipline Project 2 drilled: terraform destroy. This stack is one EC2 instance plus an Elastic IP, and both bill by the hour whether or not anyone visits — an Elastic IP keeps charging even after you stop the instance. Stopping is not free; only destroying is. terraform destroy deletes every resource in dependency order, and because it’s all in code you can bring the identical stack back tomorrow with terraform apply. That’s the payoff of IaC: teardown is safe because rebuild is one command.

Real world: think of a film production. You build the set, shoot the scene and keep the footage (the smoke test plus the git history are what you show), then strike the set the same night — because the soundstage bills by the day whether or not you’re filming. The movie survives; the set doesn’t.

Today you close the loop that started on Day 81 — and end it at ₹0.

Hands-On Lab

Section 2 of 5 · ~6 min

Budget about 30 minutes. Drive this from your workstation with the Day-84 stack still up: the k3s node reachable via its Elastic IP, the linkstash Helm release deployed, the in-cluster Postgres Running, and an Ingress on a host you own. Substitute the real domain you pointed at the Elastic IP for links.example.com; if you used the Let’s Encrypt staging issuer or a self-signed cert on Day 84, add -k to the curl calls (staging roots aren’t in your trust store). Elastic IP, hostnames, image digests, and resource IDs are unique to you — yours will differ. Your k3s/kubectl minor and Helm patch will differ too.

What this costs: the single t3.small (~$0.0208/hr ≈ $15/mo) and its Elastic IP ($0.005/hr while allocated) have been billing since Day 82 — roughly ~$19/month if you leave the stack up 24/7. Today’s terraform destroy (step 7) deletes the instance and releases the Elastic IP, and step 8 proves zero instances and zero addresses remain — so spend returns to ₹0. An Elastic IP that outlives its instance keeps billing, which is exactly why destroy must release it, not just detach. Do not stop after reading this box with the stack still up: on AWS, “stopped” is not “free” — only destroyed is.

# 1. Re-attach to the Day-84 cluster and confirm everything is still Running.
export KUBECONFIG=~/linkstash/deploy/capstone/k3s.yaml
kubectl get nodes
helm list
kubectl get pods
# Output — single node Ready, the release deployed, both pods up (AGE/RESTARTS differ):
# NAME            STATUS   ROLES                  AGE   VERSION
# linkstash-k3s   Ready    control-plane,master   3h    v1.31.5+k3s1
# NAME       NAMESPACE  REVISION  STATUS    CHART            APP VERSION
# linkstash  default    2         deployed  linkstash-0.1.0  1.0.0
# NAME                         READY   STATUS    RESTARTS   AGE
# linkstash-6b7d9c8f5-x9q2v    1/1     Running   0          24m
# postgres-6f8c9d5b7c-k4mtn    1/1     Running   0          58m

# 2. Confirm DNS resolves to the Elastic IP and the TLS cert is issued (Ready=True).
dig +short links.example.com
kubectl get ingress linkstash
kubectl get certificate
# Output (your Elastic IP + host differ; READY True = Let's Encrypt issued the cert):
# 203.0.113.24
# NAME        CLASS     HOSTS               ADDRESS       PORTS     AGE
# linkstash   traefik   links.example.com   203.0.113.24   80, 443   40m
# NAME            READY   SECRET          AGE
# linkstash-tls   True    linkstash-tls   38m

# 3. SMOKE TEST, create: POST a long URL over HTTPS and get a short code back.
curl -s -X POST https://links.example.com/shorten \
  -H 'content-type: application/json' \
  -d '{"url":"https://opscanopy.com/mission-90/"}'
# Output — the app wrote a row to in-cluster Postgres and returned the code (yours differs):
# {"code":"aB3xZ"}

# 4. SMOKE TEST, redirect: GET the short code — a 307 back to the original, then follow it.
curl -si https://links.example.com/aB3xZ | head -n 3
curl -s -o /dev/null -w 'followed → %{http_code} %{url_effective}\n' -L https://links.example.com/aB3xZ
# Output (HTTP/2 over TLS via Traefik; the location header is the original URL):
# HTTP/2 307
# location: https://opscanopy.com/mission-90/
# content-length: 0
# followed → 200 https://opscanopy.com/mission-90/

That is the capstone proven: TLS handshake, a write to Postgres, and a read that redirects. Now write the portfolio piece. Save the following into ~/linkstash/deploy/capstone/README.md — write it for someone who has the repo but not your memory:

# linkstash on Kubernetes — Phase 4 capstone

Deploys **linkstash** (a FastAPI URL shortener, `ghcr.io/pushkar/linkstash:v1.0.0`)
to a real cloud Kubernetes cluster, provisioned and destroyed entirely from code.

## Architecture

    curl ──DNS→EIP, TLS:443──▶ k3s node (Traefik) ──svc:80──▶ linkstash pod :8000 ──▶ Postgres 16

- **Cloud:** AWS `us-east-1`, one `t3.small` EC2 instance running **k3s**
  (single-node, single-AZ — chosen for cost, ~$19/mo vs ~$73/mo for an EKS
  control plane alone). Not HA; see *Limitations*.
- **Ingress + TLS:** k3s's built-in **Traefik** plus a cert-manager
  **Let's Encrypt** ClusterIssuer terminate HTTPS at `links.example.com`.
- **Data:** an in-cluster **Postgres 16** (Deployment + PVC on the `local-path`
  StorageClass); `DATABASE_URL` lives in the `linkstash-db` Secret. RDS is the
  production alternative.

## Layers

| Layer      | Path               | What it provisions                                              |
|------------|--------------------|-----------------------------------------------------------------|
| Terraform  | `terraform/`       | VPC, public subnet, IGW, route table, `k3s-sg`, EC2 + user_data installing k3s, Elastic IP |
| Kubernetes | `k8s/`             | Postgres Deployment/PVC/Service + the `linkstash-db` Secret       |
| Helm       | `chart/linkstash/` | templated linkstash Deployment/Service/Ingress/ConfigMap; `values.yaml` sets `image.tag`, `replicaCount`, `ingress.host` |

## Run it

    cd terraform && terraform apply -var="ssh_cidr=$(curl -s https://checkip.amazonaws.com)/32"
    export KUBECONFIG=../k3s.yaml
    kubectl apply -f ../k8s/               # Postgres + the linkstash-db Secret
    helm install linkstash ../chart/linkstash   # the app tier: Deployment, Service, ConfigMap, Ingress

## Limitations (honest)

- **Single node, single AZ:** if the instance or `us-east-1a` fails, linkstash is
  down. Project 2's AWS build used two AZs for exactly this reason.
- **In-cluster DB on a node-local PVC:** data lives on one disk, no managed
  backups or failover. Production would use RDS.

## Lessons learned

- Pods `Running` is not "it works" — the end-to-end smoke test (create → 307) is
  the real proof of done.
- `terraform destroy` returning the account to ₹0 is a feature, not an
  afterthought: IaC makes teardown safe because rebuild is one command.
- k3s collapses a managed control plane into one binary — ideal for learning and
  cost, wrong for production HA.

## Teardown

    cd terraform && terraform destroy -var="ssh_cidr=$(curl -s https://checkip.amazonaws.com)/32"   # releases every resource → ₹0

# 5. Tidy, then commit the capstone and push it as your portfolio piece. The Helm
#    chart (Day 84) now owns the app tier, so remove the raw manifests it replaced —
#    otherwise the finished repo double-defines linkstash and a fresh `helm install`
#    trips the ownership error from Day 84.
#    NOTE: k3s.yaml and terraform.tfstate hold live credentials — .gitignore them, never push them.
cd ~/linkstash/deploy/capstone
rm k8s/linkstash.yaml                     # Deployment + Service are templated in chart/linkstash now
# then trim k8s/config.yaml down to just the linkstash-db Secret (the chart owns the ConfigMap)
git add terraform/ k8s/ chart/ README.md
git commit -m "capstone: linkstash on k3s (Terraform + K8s + Helm)"
git push origin main
# Output (hashes/counts will differ):
# [main 7f3a9c2] capstone: linkstash on k3s (Terraform + K8s + Helm)
#  14 files changed, 486 insertions(+)
# To https://github.com/pushkar/linkstash.git
#    a1b2c3d..7f3a9c2  main -> main

# 6. TEARDOWN: destroy every AWS resource Terraform created. This is mandatory, not optional.
cd ~/linkstash/deploy/capstone/terraform
terraform destroy -auto-approve -var="ssh_cidr=$(curl -s https://checkip.amazonaws.com)/32"
# (ssh_cidr has no default, so destroy needs it too — the value is irrelevant while tearing down)
# Output (tail — Terraform destroys in dependency order; count matches your Day-82 apply):
# aws_instance.k3s: Destroying... [id=i-0abc123def4567890]
# aws_instance.k3s: Destruction complete after 41s
# aws_eip.k3s: Destroying... [id=eipalloc-0abc1234def567890]
# aws_security_group.k3s: Destroying... [id=sg-0abc1234]
# aws_subnet.public: Destroying... [id=subnet-0abc1234]
# aws_internet_gateway.igw: Destroying... [id=igw-0abc1234]
# aws_vpc.main: Destroying... [id=vpc-0abc1234]
# aws_vpc.main: Destruction complete after 1s
#
# Destroy complete! Resources: 10 destroyed.

# 7. VERIFY ₹0: no instance running (any state) and no Elastic IP allocated. Never trust destroy alone.
aws ec2 describe-instances --region us-east-1 \
  --filters Name=tag:Name,Values=linkstash-k3s Name=instance-state-name,Values=pending,running,stopping,stopped \
  --query 'Reservations[].Instances[].InstanceId' --output text
aws ec2 describe-addresses --region us-east-1 --query 'Addresses[].PublicIp' --output text
# Output — both empty. Zero instances, zero Elastic IPs: spend is back to ₹0.
#
#

Read the finish back: the smoke test proved a real request travels DNS → Elastic IP → Traefik/TLS → Service → pod → Postgres and comes home as a 307; the README and the pushed repo are the parts that outlive the cluster; and terraform destroy plus the step-7 verification returned the account to ₹0. A capstone you can rebuild from code tomorrow and that costs nothing while it’s down is exactly what a finished cloud project should look like.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three catch people on the last day — when the cert is staging, the destroy looks clean but isn’t, or the ingress host doesn’t match. Read the error text slowly; parsing it is the actual skill.

Common error: The HTTPS smoke test fails with a certificate error because the cluster is on the Let’s Encrypt staging issuer (or a self-signed fallback):

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

Why: the Day-84 staging/self-signed fallback issues a certificate whose root isn’t in your system trust store, so curl (and browsers) refuse it. Nothing is broken — the TLS is just untrusted, which is the whole point of staging (it dodges Let’s Encrypt’s tight production rate limits while you iterate).

Fix: for the smoke test, add -k (curl -k https://links.example.com/...) to skip verification. To get a trusted padlock, point a real domain’s A record at the Elastic IP and switch the ClusterIssuer to Let’s Encrypt production, then delete the cert Secret so cert-manager re-issues.

How you’d spot it in prod: a browser showing “Not secure” on a page that clearly has HTTPS, with the certificate issuer reading “(STAGING) Let’s Encrypt” — that’s the staging environment shipped by mistake, not a real trust failure.

Common error: terraform destroy reports success but AWS is still billing, because it ran where Terraform can’t see its state:

No changes. No objects need to be destroyed.

Either you have not created any objects yet or the existing objects were
already deleted outside of Terraform.

Why: Terraform only destroys what’s recorded in its state file. Run destroy from the wrong directory, or after terraform.tfstate was moved or deleted, and it sees an empty state — so it “succeeds” while the real EC2 instance and Elastic IP keep running and charging.

Fix: run destroy from ~/linkstash/deploy/capstone/terraform where the state lives, and confirm first with terraform state list (it should list the instance, EIP, VPC, etc.). If the state is genuinely lost, the step-7 aws ec2 describe-instances/describe-addresses checks find the orphans so you can delete and release them by hand.

How you’d spot it in prod: a bill that keeps climbing after a “clean” teardown. Never trust “Destroy complete” alone — always cross-check with an out-of-band aws query, which is exactly why step 7 exists.

Common error: The redirect test returns 404 page not found from Traefik instead of a 307:

404 page not found

Why: Traefik routes by the Host header. If you curl the raw Elastic IP, or a hostname that doesn’t match the Ingress rule, Traefik has no matching route and returns its own 404 (a 502/503 would instead mean the Service has no ready pod endpoints). This is a routing miss, not an app error.

Fix: curl the exact host in the Ingress (links.example.com), and confirm kubectl get ingress linkstash shows that host with an ADDRESS, plus kubectl get endpoints linkstash lists the pod IP. Fix the DNS name or the ingress.host value in values.yaml, then helm upgrade.

How you’d spot it in prod: a 404 whose body is Traefik’s plain-text page (not your app’s JSON) means the request never reached a backend — check host routing first; a 502/503 means it reached Traefik but the backend is down, so check pod readiness and Service endpoints.

Capstone Interview Questions

Section 4 of 5 · ~1 min

These four are what an interviewer asks when a project is on your resume: they probe whether you verified it, understood its costs, and know its limits. Cover each answer and say your own version out loud first — the four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 10 min — Rebuild the whole stack from ₹0: terraform apply, wait for the node, helm install, and time how long “nothing” → “live HTTPS” takes. That stopwatch is the real proof of Infrastructure as Code.
• 10 min — Read the k3s networking docs on the bundled Traefik and the Klipper service load balancer, and note how you’d disable Traefik to run ingress-nginx instead.
• 5 min — Skim HashiCorp’s terraform destroy docs and the note on state — understand why destroy is entirely state-driven, which is what the second Common Error above turns on.
• 5 min — Price this capstone as a production deployment on the Amazon EKS pricing page (control plane ~$0.10/hr ≈ $73/mo, before nodes) and compare it to your single-node k3s bill — the tradeoff you chose on Day 81.