Project 3, Day 3: build the Kubernetes manifests

From compose.yaml to Kubernetes objects: the linkstash manifest set

Section 1 of 5 · ~2 min

Yesterday you used Terraform to stand up a one-node k3s cluster on EC2 and copied its kubeconfig to your laptop. The cluster is empty. Today you fill it — not with docker compose up, but with a set of declarative manifests that describe the same two-service stack you built in Project 1, now as native Kubernetes objects.

In Project 1, compose.yaml held the whole stack in one file: a web service built from your Dockerfile and a db service running postgres:16. Kubernetes wants each concern as its own object, so the one file becomes a small set:

• The web service → a Deployment linkstash (one replica of ghcr.io/pushkar/linkstash:v1.0.0) fronted by a Service linkstash (ClusterIP, :80 → the pod’s :8000).
• The db service → a Deployment postgres, a Service postgres, and a PersistentVolumeClaim for its data.
• Compose’s inline environment: → a ConfigMap linkstash-config for the non-secret settings and a Secret linkstash-db for the DATABASE_URL.
• The named volume pgdata → the PVC, backed by k3s’s storage.

The one line that changes from Project 1 is the database host. Compose resolved the DB at the service name db; on Kubernetes the Service is named postgres, so the connection string becomes postgresql://postgres:postgres@postgres:5432/linkstash — same user, password and database, new host. In-cluster DNS resolves postgres to the Service’s ClusterIP, which forwards to the pod. Point it at db or localhost and the app can’t find its database.

Storage is the interesting part. k3s ships Rancher’s local-path-provisioner as its default StorageClass, local-path. When your PVC is first consumed by a scheduled pod, the provisioner carves a directory on that node’s disk and binds the claim to it. That binding mode — WaitForFirstConsumer — is why a fresh PVC sits at Pending until the Postgres pod is scheduled; it is normal, not a fault. Because the data lives on this single node’s disk, it survives a pod restart but not the node dying — the honest limit of a one-node cluster versus Project 2’s two-AZ RDS. In production you’d run a StatefulSet, or better, managed Postgres like Amazon RDS.

Real world: Moving house. Compose was one van — everything loaded together, driven in one trip. Kubernetes labels each item into its own crate with a manifest, so the movers can place, replace and reconnect each independently. The PVC is the storage locker you keep rented across the move: swap in a new pod and the data is still in the locker, waiting.

By the end of the lab you’ll have written four YAML files, applied them with one kubectl apply -f k8s/, watched both pods reach Running and the PVC Bound, and smoke-tested linkstash answering 200 on /healthz — the same app, the same image, now scheduled and wired by Kubernetes instead of Compose.

Hands-On Lab

Section 2 of 5 · ~5 min

Budget about 25 minutes. kubectl is already pointed at the Day-82 k3s cluster (context default). You’ll write four manifests under k8s/, apply them, watch both pods come up, read their logs, and smoke-test linkstash on :8000. Pod-name suffixes, cluster IPs, ages and versions are unique to your run — yours will differ from the samples.

What this costs: The k3s cluster from Day 82 is still billing. The t3.small instance (~$0.0208/hr ≈ $15/mo) and its attached Elastic IP ($0.005/hr while attached) keep running whether or not anything is deployed on them. Today adds no new AWS charge: the Postgres PVC uses k3s’s local-path StorageClass, which writes to a directory on the instance’s existing ~8 GB gp3 root disk — not a new EBS volume — so there is nothing extra to pay for. The total is unchanged at ~$19/month if left up 24/7 (≈ ₹0 if you’re on a Free-Tier-eligible t2.micro and tear down on Day 85). Leave the instance running to finish the project — Day 85 ends with terraform destroy.

# 1. Make the k8s/ folder, then confirm kubectl still points at the Day-82 k3s cluster
#    (one node Ready) and that k3s ships a default StorageClass for the PVC.
mkdir -p ~/linkstash/deploy/capstone/k8s && cd ~/linkstash/deploy/capstone
kubectl config current-context
kubectl get nodes
kubectl get storageclass
# Output (node name, k3s version and ages differ — your minor/patch too):
# default
# NAME            STATUS   ROLES                  AGE   VERSION
# linkstash-k3s   Ready    control-plane,master   18m   v1.31.5+k3s1
# NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
# local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  18m

# 2. k8s/config.yaml — non-secret settings in a ConfigMap; the DB URL + password in a Secret.
#    stringData keeps the Secret readable here; in prod you would NOT commit the real value.
apiVersion: v1
kind: ConfigMap
metadata:
  name: linkstash-config
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: linkstash-db
type: Opaque
stringData:
  # host is the Postgres Service name "postgres" (was "db" in Project 1's compose.yaml)
  DATABASE_URL: "postgresql://postgres:postgres@postgres:5432/linkstash"
  POSTGRES_PASSWORD: "postgres"

# 3. k8s/postgres.yaml — in-cluster Postgres 16: a PVC on local-path, a Deployment, a ClusterIP Service.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path      # k3s's built-in default StorageClass
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          env:
            - name: POSTGRES_USER
              value: "postgres"
            - name: POSTGRES_DB
              value: "linkstash"
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: linkstash-db
                  key: POSTGRES_PASSWORD
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  selector: { app: postgres }
  ports:
    - port: 5432
      targetPort: 5432

# 4. k8s/linkstash.yaml — the app: Deployment (config from the ConfigMap, DB URL from the Secret) + Service.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: linkstash
spec:
  replicas: 1
  selector:
    matchLabels: { app: linkstash }
  template:
    metadata:
      labels: { app: linkstash }
    spec:
      containers:
        - name: linkstash
          image: ghcr.io/pushkar/linkstash:v1.0.0
          envFrom:
            - configMapRef:
                name: linkstash-config
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: linkstash-db
                  key: DATABASE_URL
          ports:
            - containerPort: 8000
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8000
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: linkstash
spec:
  selector: { app: linkstash }
  ports:
    - port: 80
      targetPort: 8000

# 5. Apply every manifest in k8s/ at once — kubectl reads the files in name order.
kubectl apply -f k8s/
# Output:
# configmap/linkstash-config created
# secret/linkstash-db created
# deployment.apps/linkstash created
# service/linkstash created
# persistentvolumeclaim/postgres-data created
# deployment.apps/postgres created
# service/postgres created

# 6. Watch both pods reach Running and the PVC Bind. linkstash may show 1-2 restarts if it
#    started before Postgres was ready — it self-heals once the DB accepts connections.
kubectl get pods,svc,pvc
# Output (pod suffixes, cluster IPs and ages are yours):
# NAME                            READY   STATUS    RESTARTS      AGE
# pod/postgres-7b9c4d8f6-4mzjq    1/1     Running   0             55s
# pod/linkstash-6d5f7c9b8-q2xkt   1/1     Running   1 (30s ago)   55s
#
# NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
# service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP    19m
# service/postgres     ClusterIP   10.43.201.14   <none>        5432/TCP   55s
# service/linkstash    ClusterIP   10.43.88.203   <none>        80/TCP     55s
#
# NAME                                  STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS   AGE
# persistentvolumeclaim/postgres-data   Bound    pvc-9f0c1e2a-...      2Gi        RWO            local-path     55s

# 7. Confirm Postgres initialised and is accepting connections.
kubectl logs deploy/postgres | tail -n 3
# Output (your patch version will differ):
# PostgreSQL init process complete; ready for start up.
# 2026-07-12 09:14:02.551 UTC [1] LOG:  starting PostgreSQL 16.9 (Debian 16.9-1.pgdg120+1) on x86_64-pc-linux-gnu
# 2026-07-12 09:14:02.560 UTC [1] LOG:  database system is ready to accept connections

# 8. linkstash's logs prove it connected to Postgres — init_db() ran and Uvicorn is serving :8000.
kubectl logs deploy/linkstash | tail -n 4
# Output:
# INFO:     Started server process [1]
# INFO:     Waiting for application startup.
# INFO:     Application startup complete.
# INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

# 9. In one terminal, forward the linkstash Service (:80 -> the pod's :8000) to your laptop.
kubectl port-forward svc/linkstash 8080:80
# Output (leave this running):
# Forwarding from 127.0.0.1:8080 -> 8000
# Forwarding from [::1]:8080 -> 8000

# 10. In a second terminal, hit the liveness endpoint through the forward — a 200 with the JSON body.
curl -s http://localhost:8080/healthz
# Output:
# {"status":"ok"}

Read step 8 back: Application startup complete means init_db() reached Postgres over the postgres Service and created the links table — the app is wired to its database. The curl in step 10 proves it’s reachable and alive. You haven’t exposed it to the internet yet (that’s tomorrow’s Ingress) — but the workload itself is running on real cloud Kubernetes, config from a ConfigMap and its secret from a Secret, exactly as designed on Day 81.

Common Errors & Fixes

Section 3 of 5 · ~3 min

These three catch people the first time they deploy a private image and a database to k3s. Read the pod STATUS and its describe events slowly — Kubernetes names the exact object or host it couldn’t resolve.

Common error: The linkstash pod never pulls its image because the GHCR package is private and the node has no credentials:

NAME                        READY   STATUS             RESTARTS   AGE
linkstash-6d5f7c9b8-q2xkt   0/1     ImagePullBackOff   0          40s
#  describe events:
#  Failed to pull image "ghcr.io/pushkar/linkstash:v1.0.0": failed to resolve reference
#  "ghcr.io/pushkar/linkstash:v1.0.0": unexpected status from HEAD request: 403 Forbidden

Why: GitHub Container Registry packages are private by default. Your laptop pulled the image on Day 62 because you were logged in to GHCR; the k3s node’s containerd has no such login, so the registry refuses the pull with 403. This is an authorization problem, not a missing image — manifest unknown would mean the tag itself doesn’t exist.

Fix: Either mark the package public in its GitHub package settings, or create a pull Secret and reference it: kubectl create secret docker-registry ghcr --docker-server=ghcr.io --docker-username=pushkar --docker-password=<PAT>, then add imagePullSecrets: [{name: ghcr}] to the Deployment’s pod spec and re-apply.

How you’d spot it in prod: ImagePullBackOff/ErrImagePull with denied or 403 Forbidden in describe means credentials or visibility; manifest unknown or not found means the tag is wrong. Check which before touching the cluster.

Common error: linkstash restarts forever because DATABASE_URL still names the old Compose host db (or localhost) instead of the postgres Service:

NAME                        READY   STATUS             RESTARTS   AGE
linkstash-6d5f7c9b8-q2xkt   0/1     CrashLoopBackOff   4          2m
#  kubectl logs deploy/linkstash:
#  psycopg.OperationalError: could not translate host name "db" to address: Name or service not known

Why: init_db() runs during startup; if it can’t reach Postgres the app process exits, and Kubernetes restarts it — a crash loop. On this cluster the database’s DNS name is the Service name postgres; the Compose host db doesn’t resolve, and localhost points at the app’s own pod. (A transient one or two restarts while Postgres is still booting is different — that recovers on its own once the DB is ready.)

Fix: Set the host in the Secret’s DATABASE_URL to postgres, kubectl apply -f k8s/config.yaml, then kubectl rollout restart deploy/linkstash so the pod picks up the new value.

How you’d spot it in prod: CrashLoopBackOff whose logs show a name-resolution failure or connection refused on the DB host points at the connection string, not the app code — the workload can’t find its dependency.

Common error: The Postgres pod is stuck Pending because its PVC names a StorageClass that doesn’t exist on k3s (e.g. standard, copied from a Minikube tutorial):

NAME                       READY   STATUS    RESTARTS   AGE
postgres-7b9c4d8f6-4mzjq   0/1     Pending   0          2m
#  kubectl describe pvc postgres-data:
#  Warning  ProvisioningFailed   storageclass.storage.k8s.io "standard" not found

Why: A PVC binds only through a real StorageClass. Minikube’s default is named standard; k3s’s is local-path. Name a class the cluster doesn’t have and the PVC never provisions, so the pod that mounts it can never schedule — it waits at Pending forever, not CrashLoopBackOff.

Fix: Set storageClassName: local-path (as in step 3) — or delete the line entirely to fall back to the cluster’s default, which kubectl get storageclass shows is local-path (default) on k3s. Re-apply and the claim binds when the pod schedules.

How you’d spot it in prod: A pod stuck Pending with describe pod mentioning “unbound PersistentVolumeClaim”, and describe pvc naming a missing StorageClass, means the storage class is wrong for this cluster — portable manifests either omit the class or match the target platform’s.

Kubernetes Manifests Interview Questions

Section 4 of 5 · ~1 min

These four separate people who can kubectl apply a tutorial’s YAML from people who understand what each object is for — say your own answer out loud first, then compare, because recalling before revealing is what makes it stick. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — kubectl describe pvc postgres-data and kubectl get pv, then SSH to the node and find the actual directory the provisioner created under /var/lib/rancher/k3s/storage/ — that’s where your links table really lives.
• 10 min — Skim the Kubernetes StatefulSet docs and note why a database usually wants the stable identity and per-replica storage a Deployment doesn’t give it.
• 5 min — Generate a manifest instead of hand-writing it: kubectl create deployment demo --image=nginx --dry-run=client -o yaml and read what it prints — a fast way to get the boilerplate right.
• 10 min — Read about init containers and see how one that waits for Postgres would replace Compose’s depends_on: condition: service_healthy, so linkstash never crash-loops on a cold start.