Job-ready interview hub.

DevOps is a way of working: developers and operations share responsibility for shipping and running software, and automation shortens the loop between writing code and running it. SRE — Site Reliability Engineering — is a concrete implementation of similar ideas, popularized by Google: it treats operations as a software problem, with engineers who automate away toil and manage reliability against explicit targets called SLOs, backed by error budgets. DevOps names the culture and the pipeline; SRE names a specific role and toolkit for reliability. A tidy closing line: DevOps says 'you build it, you run it' — SRE adds 'and here is exactly how we measure whether it is running well.'

Because manual releases don't scale and don't repeat. A human deploying by hand depends on memory and attention at the worst possible moment — usually late at night. A script or pipeline runs the same way the hundredth time as the first, and every run is logged, reviewable, and easy to roll back. That changes the economics of shipping: when a release costs one click instead of one weekend, teams ship small changes often, and small changes are far easier to test and debug than a quarterly big bang. In a services company this compounds — a pipeline built for one client project becomes a template for the next, so quality goes up while delivery time goes down.

The terminal is only the window collecting keystrokes; it hands the line to the shell — bash on most servers. The shell parses the line, expands variables and wildcards, and decides whether it's a built-in like cd or an external program. For a program, it searches the directories in PATH, finds the executable — say /usr/bin/ls — and asks the kernel to start it as a new process. The process does its work, writes to standard output, which flows back to your screen, and exits with a status code: zero for success, non-zero for failure. That exit code matters — it's what scripts and CI pipelines check to decide whether to continue or fail the build.

The terminal is the program that draws the window: it takes keystrokes in and prints text out, nothing more. The shell is the interpreter running inside it — bash or zsh — that reads commands, expands variables, and starts processes. The distinction matters in DevOps because many shells you'll deal with have no terminal attached at all: a CI job, a cron task, or an SSH one-liner runs in a shell with no human window. That's why scripts declare their interpreter with a shebang like #!/usr/bin/env bash, and why 'it works in my terminal' doesn't guarantee it works in a pipeline — the environment, not the window, is what runs your code.

/etc holds system-wide configuration — plain text files that tell programs how to behave, like /etc/ssh/sshd_config or /etc/os-release. It is meant to be static and edited deliberately; nothing should write to it during normal running. /var is the opposite: data that varies while the system runs — logs in /var/log, caches, spool queues, databases. The rule I use is simple: config you set lives in /etc, state the machine generates lives in /var. That matters for backups and containers — you treat the two very differently, and you rarely bake /var into an image because it fills up at runtime.

An absolute path starts from the root of the filesystem — it begins with a slash, like /var/log/syslog, and means the same thing no matter where you are standing. A relative path is resolved from your current directory: log/syslog means 'the log folder inside wherever I am right now.' You check where 'here' is with pwd. In DevOps this bites people constantly — a script using relative paths works when you run it by hand from your home directory, then fails in a cron job or CI runner that starts somewhere else. The safe habit is absolute paths in scripts, relative paths only for quick interactive work.

On most Linux servers logs live under /var/log. System and boot messages sit in /var/log/syslog; authentication and sudo attempts in /var/log/auth.log; individual services often get their own file or folder, like /var/log/nginx/. My first move on an unfamiliar box is ls /var/log to see what is there, then tail -f a file to watch it live. On modern Ubuntu, systemd services also log to the journal, which you read with journalctl rather than a flat file — that is Day 10. Knowing logs live in /var also explains a classic outage: /var fills up, logs can no longer be written, and services start failing for no obvious reason.

/ is the root of the entire filesystem — the single top of the tree that everything else hangs off. ~ is a shortcut for your home directory, which for me is /home/pushkar; for the root user it is /root. So cd / takes you to the very top, while cd ~ (or just cd with no argument) takes you home. People mix these up early and it is a meaningful slip: a destructive command like rm -rf aimed at / is a catastrophe, aimed at ~ it merely wrecks your own files. Interviewers like this one because it quietly checks whether you understand the difference between the whole system and your little corner of it.

cp copies a file — you end up with two: the original stays put and a duplicate appears at the destination. mv moves it — the file leaves its old location and shows up at the new one, so there's only ever one copy. mv is also how you rename a file, because renaming is just moving it to a new name in the same directory: mv old.txt new.txt. One gotcha for both: if the destination already exists, they overwrite it silently, with no prompt. In scripts I add -i for an interactive confirm, or -n to never overwrite, so a deploy step can't clobber a file it shouldn't touch.

rm deletes immediately and permanently — there is no recycle bin on the command line, so the file's directory entry is gone the moment you press Enter. rm -rf compounds that: -r recurses into directories and -f forces past every 'are you sure?' prompt, so one wrong path can wipe a whole tree with no warning. The classic disaster is an empty variable — rm -rf $DIR/ becomes rm -rf / when $DIR is unset. That's why in production I never run it against a variable without checking the variable is set, I prefer explicit paths over wildcards, and I reach for -i or a trash tool on anything I'd hate to lose.

An absolute path starts from the filesystem root with a leading slash — /etc/nginx/nginx.conf points to the same file no matter where you are. A relative path starts from your current directory and has no leading slash — nginx.conf or ../logs/error.log — so it means different things depending on where you're standing. Rule of thumb: use absolute paths anywhere the working directory isn't guaranteed — cron jobs, systemd units, deploy scripts — because those run from a directory you didn't choose. Use relative paths for quick interactive work and inside a project, where they keep commands short and stay correct if the whole project moves.

Every directory contains two hidden entries: `.` is the directory itself, and `..` is its parent. So cd .. walks one level up, and cd . goes nowhere — it just refers to where you already are. They matter most when you run a program that lives in the current directory: the shell won't search the current directory for executables, so you write ./deploy.sh to say 'run the deploy.sh right here', not some deploy.sh on your PATH. You'll also see `..` in relative paths like ../config to reach a sibling directory, and in copies like cp -r ./src .. to write into the parent.

cat dumps a file's entire contents to your screen and drops you at the prompt — perfect for a short config you want to read in full, or for piping a file into another command. less opens the file in a pager: it loads only what fits on screen, so it stays instant even on a multi-gigabyte log, and lets you scroll, jump to the end with G, search with /pattern, and quit with q. Rule of thumb: cat when the file is small or you're piping it somewhere; less when the file is large or you want to move around inside it. Reaching for cat on a huge log is the classic way to freeze a terminal.

This is the famous one. Press Esc first to leave insert mode — that's the step people miss, because vim starts in normal mode and any text you typed put you in insert mode, where a bare :q just gets inserted as literal characters. Once you're back in normal mode, type :q and Enter if you changed nothing, :wq (write, then quit) or :x if you want to keep your edits, and :q! if you want to throw the changes away and get out. The exclamation mark means 'force it, I don't care about unsaved edits.' Esc, then the colon command, always works.

nano when you want a two-line change and zero ceremony — its shortcuts are printed along the bottom of the screen (Ctrl+O to save, Ctrl+X to exit), so there are no modes to remember. It's ideal for fixing one config value or leaving a quick comment. You reach for vim when nano isn't installed — and on hardened or minimal production images, often only vi is there — or when you're doing heavier editing and want its search, navigation, and repeatable commands. The honest interview answer: nano for speed and comfort, vim because it's the editor guaranteed to be on the box when you SSH into a stranger's server.

Don't cat it — that reads the whole file and can freeze your session on a multi-gigabyte log. Use less, which pages through the file and only holds what's on screen; inside it, G jumps to the end where the newest entries are, /pattern searches forward, n repeats the search, and q quits. If you only care about the most recent lines, tail does it directly: tail -n 100 file for the last hundred, or tail -f file to follow the file live as new lines are written — that's how you watch a service log during a deploy. head is the mirror image for the top of the file.

Read it left to right in four parts. The first character is the file type — a dash for a normal file, d for a directory. Then come three groups of three: the owner's permissions, the group's, and everyone else's. Within each group the letters are always in the same order — read, write, execute — and a dash means that permission is off. So -rwxr-xr-- is a file whose owner can read, write and run it, whose group can read and run it, and where everyone else can only read. I say it out loud in those chunks; once you can, ls -l stops looking like noise.

Each octal digit sets one class of user — owner, group, other — and the digit is the sum of read (4), write (2) and execute (1). So 7 is 4+2+1, all three; 5 is 4+0+1, read and execute but not write. chmod 755 therefore means the owner can read, write and execute, while the group and everyone else can read and execute but not modify it. It's the standard mode for a script or program you want anyone to run but only you to change, and for directories — because you need the execute bit on a directory just to enter it.

chmod changes the permission bits — what the owner, group and others are allowed to do with a file. chown changes who the owner and group actually are. They answer different questions: chmod is 'what can be done here', chown is 'whose file is this'. A quick example: if a log file is owned by root and your app can't write to it, chmod 666 would be the wrong, insecure fix; the right fix is usually chown to the user the app runs as. Changing ownership normally needs sudo, because Linux won't let you hand files around to dodge accountability.

Because the execute bit means different things for each. On a regular file, execute means 'run me as a program', and a text file, config or log should never be runnable — so 644 gives the owner read and write, everyone else read-only, and no execute anywhere. On a directory, though, the execute bit means 'you may enter this directory and reach the files inside'. Without it you can't cd in or open anything under it, even if you can see the name. So directories need 755 — read and execute for everyone, write only for the owner. Same numbers, but x flips meaning between the two.

Reach for sudo when you genuinely need to act as root for one command — installing a package, editing a system config, restarting a service. But if you find yourself prefixing every command with sudo just to work with your own project's files, that's a smell: the real problem is that the files are owned by the wrong user, and the fix is a one-time chown to set ownership correctly. Sudo everywhere is how permissions rot on a shared server — nobody can tell who's allowed to do what. In production I treat a pile of sudo as a sign the ownership model needs fixing, not more privilege.

A PID is a process ID — the unique number the Linux kernel assigns to every running process when it starts. It's how you refer to a specific process to inspect or control it: kill, renice, and top all take a PID. PID 1 is special — it's the first process the kernel starts (systemd on a modern Ubuntu box) and it's the ancestor of everything else, so you never kill it. PIDs aren't permanent: when a process exits its number is eventually reused. In practice you rarely memorise a PID — you find it with ps or top, then act on it. That find-then-act loop is the whole skill.

kill by default sends SIGTERM, signal 15 — a polite request to shut down. A well-behaved process catches it, finishes what it's doing, flushes buffers, closes files and sockets, and exits cleanly. kill -9 sends SIGKILL, signal 9, which the kernel enforces directly — the process can't catch, block, or ignore it, so it dies instantly with no cleanup. Always try SIGTERM first; reach for kill -9 only when a process is truly stuck and ignoring the polite ask. The tradeoff is that SIGKILL can leave corrupt files, orphaned locks, or half-written state behind. In an interview, the one-liner is: SIGTERM asks, SIGKILL forces.

I start with top — it shows a live, sorted view, and by default the busiest process floats to the top of the %CPU column. Press capital P to force a CPU sort, note the PID and command, and press q to quit. For a non-interactive snapshot, especially in a script or over SSH, I use ps aux --sort=-%cpu | head, which prints the top offenders once and exits. Once I have the PID I decide whether it's legitimate work or a runaway. If it's runaway I send SIGTERM first, then SIGKILL if it won't budge. On a modern box htop makes the same job friendlier, but top and ps are always installed.

A zombie — shown as defunct or state Z in ps — is a process that has already finished but whose entry lingers in the process table because its parent hasn't collected its exit status yet. That collection is called reaping, done via the wait system call. A zombie uses no CPU and almost no memory; it's just a bookkeeping stub holding a PID. You can't kill a zombie — it's already dead. The real fix is the parent: if it's buggy and never reaps, you restart or fix the parent, and when the parent dies, PID 1 adopts the orphaned zombies and reaps them. A handful is harmless; thousands means a broken parent.

They answer different questions. ps takes a one-time snapshot and exits — perfect for scripts, logs, and grepping for a specific process, as in ps aux | grep nginx. top is interactive and live: it refreshes every couple of seconds so you can watch CPU and memory move in real time, sort columns, and spot a spike as it happens. Rule of thumb: ps to capture a moment or feed another command, top to observe behaviour over time. In an incident I open top first to see what's happening now, then use ps to grab the exact line I want to paste into the ticket.

First I orient — I confirm which host and which service is actually affected before touching anything. Then I find the load: top, or ps aux --sort=-%cpu, ranks processes by CPU so the worst offender sits at the top, and I note its PID, user, and command. Before I kill anything I read that process's logs to understand what it's doing — a nightly backup starving the app is a very different story from a real traffic spike. Once I'm sure it's the culprit and safe to stop, I kill it and watch CPU drop and the app recover. The habit that matters: identify with evidence before you act, then verify recovery instead of assuming the kill worked.

ps aux --sort=-%cpu | head, or top, ranks processes by CPU so the worst offender is at the top — I read off its PID. To stop it I send a signal with kill <pid>, which defaults to SIGTERM: a polite 'please shut down' the process can catch to clean up first. If it ignores that and keeps burning CPU, kill -9 <pid> sends SIGKILL, which the kernel enforces with no chance to clean up. Then I re-run ps to confirm it's actually gone and check that the real service reclaimed the CPU. The point isn't just killing something — it's killing the right PID and confirming the box is healthy afterward.

SIGTERM — a plain kill <pid> — is the graceful signal: it asks the process to shut down, and a well-behaved process catches it to finish in-flight work, flush buffers, and close files cleanly. SIGKILL — kill -9 — can't be caught or ignored; the kernel terminates the process immediately, which risks corrupt state or half-written files. In an incident I reach for SIGTERM first so the process exits cleanly, and only escalate to SIGKILL if it's truly stuck and the bleeding is worse than the risk. Reflexively kill -9-ing everything is a rookie tell — it can leave a database or file half-written and turn one incident into two.

I don't trust the fix — I trust the evidence. After the change I re-run the exact check that showed the problem: ps to confirm the app process is back and stable, the log to confirm errors stopped and healthy lines resumed, and the health endpoint or a real request to confirm it's serving traffic. I watch for a minute rather than one green blink, because a crash-looping service can flash healthy between restarts. Only when the same signals that screamed 'down' now read 'up' do I call it resolved — and then I write down what happened while it's fresh, so the postmortem isn't fiction.

They divide up text processing. grep finds lines: you give it a pattern and it prints the lines that match — that is all it does, and it does it fast. sed edits a stream: its main job is substitution, s/old/new/, rewriting text as it flows past one line at a time, without loading the whole file. awk works in columns: it splits each line into fields you address as $1, $2, $NF and prints or computes on them. The rule I use is grep to narrow down to the lines I care about, sed to clean or rewrite those lines, and awk to pull specific fields out. Most real log work chains all three.

I use grep with -r, which walks a whole directory tree instead of a single file — grep -r timeout /etc searches every file under /etc. I almost always add -n so each match shows its line number, and often -i to ignore case, so grep -rni timeout /etc is my default. The output prefixes each hit with the file path and line, so I can jump straight to it. On a big tree I narrow it with --include to skip binaries and irrelevant files. And if I only want the filenames, not the matching lines, -l lists just the files that contain a match.

I run the file through sed with a substitution that matches the sensitive pattern and replaces it. For example sed 's/user=[a-z]*/user=***/g' blanks every username, and a similar rule masks tokens or card numbers. The key point I always mention: plain sed prints the redacted version to standard output and leaves the original file untouched, so I redirect the result to a new file and never overwrite the evidence. On a client engagement that matters, because the raw log stays intact for our own investigation while the shared copy is clean. If I genuinely want in-place editing I use sed -i, but deliberately — never by reflex.

I extract the column with awk, then count. awk '{print $7}' access.log pulls the seventh field — say the request path — one per line. I pipe that into sort so identical values sit together, then uniq -c to collapse and count them, then sort -rn to rank by frequency. The whole thing is awk '{print $7}' access.log then sort, uniq -c, sort -rn, and the top line is the most common value. It is the fastest way I know to answer questions like which endpoint is hit most or which IP is hammering us — no database, just a handful of small tools.

Every process gets two output streams: standard output, file descriptor 1, for its normal results, and standard error, fd 2, for diagnostics and warnings. They are separate so you can route them independently — pipe the real output into the next tool while still seeing errors on screen, or capture errors to their own file for alerting. If they were merged, a pipeline like ls | wc -l would count error messages as data. The practical rule: results go to stdout, anything a human needs to notice goes to stderr. That is also why the plain > never silences errors — it only redirects stdout, and stderr keeps flowing until you name fd 2 explicitly.

2>&1 means 'send standard error to wherever standard output is currently going.' You use it to fold both streams into one place — a log file or a pipe. Order matters because the shell applies redirections left to right, and 2>&1 copies the current destination of fd 1 at that moment. So > out.log 2>&1 first points stdout at the file, then aims stderr at the same file, capturing both. But 2>&1 > out.log copies stderr to the terminal, where stdout still is, then moves only stdout to the file — so errors still hit your screen. Always redirect stdout first, then merge.

I would build a pipeline: grep -c ERROR /var/log/app.log, or if I am chaining, cat app.log | grep ERROR | wc -l. grep filters the matching lines, wc -l counts them. The pipe is the point — each tool does one job and hands its stdout to the next through the kernel, no temp files. If I needed the count per hour I would extend it, say pipe into cut, then sort | uniq -c. On a live file I would use tail -f app.log | grep --line-buffered ERROR to watch matches stream in. For a client server this kind of one-liner is faster than opening any log viewer.

xargs reads items from standard input and turns them into command-line arguments for another program. You need it when a tool takes arguments rather than reading stdin — rm, mkdir, kill — so find . -name '*.tmp' | xargs rm deletes the matches. Piping straight into rm would not work, because rm ignores stdin. Two habits keep it safe: xargs -0 paired with find -print0, so filenames with spaces or newlines do not split into the wrong arguments, and xargs -r so the command does not run at all when the input is empty. Without those, xargs is a classic source of 'it deleted the wrong thing' incidents.

They are two separate decisions and people conflate them constantly. start launches the service right now, this boot — the moment you run it, the process comes up, and it stays up until you stop it or the machine reboots. enable is about the future: it creates the boot-time symlink so systemd starts the service automatically every time the machine boots. So a service you start but never enable runs now and is gone after a reboot; one you enable but never start won't come up until the next boot. In production you almost always want both, which is why systemctl enable --now does start and enable in a single command.

apt update refreshes the local index of what packages exist and at which version — it talks to the repositories and downloads the lists, but it installs and changes nothing on your system. apt upgrade is the one that actually acts: it downloads and installs newer versions of packages you already have, using the index that update just refreshed. The habit is to run update first, then install or upgrade, so you're working from a current list rather than a stale one. A common gotcha in Dockerfiles and provisioning scripts is skipping update, then apt install fails to find a package or pulls an outdated one.

Two commands, in order. First systemctl status <unit> — it shows whether the unit is active or failed, the exit code of the last start attempt, and the final few log lines, which is often enough to see the problem. If it isn't, journalctl -u <unit> shows that service's full log slice; I add -e to jump to the newest entries or -n 50 for the last fifty lines. For a config-driven service like nginx I'd also run its own validator — nginx -t — because the journal will usually point at a bad config line. The key idea is that systemd centralises logs, so you never have to guess which file in /var/log belongs to the service.

Because systemctl is a client that talks to systemd, and systemd has to be running as PID 1. A normal container doesn't boot an init system; it runs your one application process directly as PID 1, so there's no systemd for systemctl to connect to, and you get 'System has not been booted with systemd.' That's by design: containers are meant to run a single foreground process, and the container runtime handles restart and lifecycle instead of systemd. So inside a container you start your app directly in the Dockerfile's CMD, and you rely on the orchestrator — Docker or Kubernetes — for the enable-on-boot and restart behaviour systemd would give you on a VM.

First the name is resolved: my machine asks a DNS resolver, which returns an IP address for example.com. curl opens a TCP connection to that IP on port 443, the HTTPS port. Over that connection it does a TLS handshake so the traffic is encrypted, then sends an HTTP request — a GET for the path plus a Host header. The server replies with a status line like HTTP/2 200, response headers, and the body. curl prints the body and the process exits. If I add -v, curl narrates each of those steps, so when something breaks I can see whether it was DNS, the connection, TLS, or the HTTP status that failed.

CIDR — Classless Inter-Domain Routing — writes an IP range as an address plus a prefix length, like 10.0.0.0/24. The number after the slash is how many leading bits are fixed as the network part; the rest identify hosts. A /24 fixes 24 bits and leaves 8, so it covers 256 addresses, 10.0.0.0 to 10.0.0.255. A /16 fixes 16 bits — 65,536 addresses — and a /32 is a single host. The smaller the prefix number, the bigger the block. You read CIDR constantly in firewall rules, cloud subnets, and route tables, where 0.0.0.0/0 is shorthand for every address.

An IP address identifies a machine on the network — it says which host to reach. A port identifies which program on that host you want, because one server usually runs many services at once. The IP gets your packet to the right building; the port gets it to the right desk inside. Ports are 16-bit numbers, 0 to 65535, and the well-known ones are worth memorising: 80 for HTTP, 443 for HTTPS, 22 for SSH, 5432 for Postgres. So 93.184.216.34:443 means port 443 on that host. If the host is up but nothing is listening on the port, you get connection refused rather than a timeout.

I narrow down which link in the chain broke instead of guessing. First, does the name resolve — a quick lookup, or curl -v to see the resolved IP; a 'could not resolve host' points at DNS. If it resolves, can I connect to the port — curl -v shows the connect succeeding or 'connection refused', which usually means the service is not listening or a firewall blocks it. If the connection is fine, what HTTP status comes back — a 200, a 500 from the app, or a 502 from a proxy tell very different stories. Working DNS then connection then TLS then HTTP, in order, turns 'it is down' into one specific failing step.

I reach for ss — the modern replacement for netstat. My default is `ss -tlnp`: -t is TCP, -l is listening sockets only, -n keeps it numeric so it does not hang on reverse lookups, and -p names the process and PID owning each socket, which usually needs sudo. Add -u to include UDP. So to see why nginx is unreachable, `sudo ss -tlnp | grep :443` tells me instantly whether anything is bound to 443 and which process it is. If nothing is bound, the service never started or crashed. If it is bound to 127.0.0.1 instead of 0.0.0.0, it is listening on localhost only — the classic 'works on the box, not from outside' bug.

I work in layers, cheapest first, so I never guess. First, does the name resolve? `dig +short site.com` — no answer means DNS. Second, can I reach the host at all? `ping` the IP; if it fails, `traceroute` shows where it dies, pointing at routing or a firewall. Third, is anything listening? `ss -tlnp` on the box, or probe the port from outside. Fourth, does the app answer correctly? `curl -I` and read the status code. Each step rules out a whole layer, so within a minute I have narrowed 'down' to DNS, network, port, or app — and I am fixing the right thing instead of restarting servers at random.

That the network layer is fine — the host is up and routable — so the problem lives higher up. ping only proves ICMP reaches the machine; it says nothing about whether your application port is open or the app is healthy. Next I check the port with `ss -tlnp` on the server, or try connecting from outside. Common causes: the service crashed or never bound to the port, it bound to 127.0.0.1 instead of a public address, or a firewall or cloud security group blocks the port while still allowing ICMP. Then I `curl` the endpoint — 'connection refused' means nothing is listening, while a timeout usually means a firewall silently dropping packets.

I split them deliberately. To test the network without DNS, I ping an IP directly — like `ping 1.1.1.1`. If that works but `ping example.com` fails, the network is fine and name resolution is broken. To confirm, `dig example.com`: NXDOMAIN or no answer points at the record or the zone, while a timeout points at the resolver itself. I also query a known-good resolver directly with `dig @1.1.1.1 example.com`; if that answers but my local one does not, the fault is my configured resolver, not the domain. Separating 'is it reachable' from 'does the name resolve' stops me chasing the wrong layer for twenty minutes.

A password is a shared secret: the server stores a hash of it and you send it on every login, so it can be guessed, phished, reused across sites, or typed into the wrong window. Key authentication never sends a secret. Your private key stays on your laptop; the server only ever holds your public key. When you connect, the server sends a random challenge and your client signs it with the private key — the server verifies the signature against the public key. Nothing reusable crosses the wire, so an attacker sniffing the connection or breaching the server learns nothing that lets them log in as you later.

authorized_keys lives in the remote user's ~/.ssh directory and holds one public key per line — every key listed there is allowed to log in as that user. You install a key by appending its .pub line to the file. Permissions matter because SSH runs a check called StrictModes: if ~/.ssh is group- or world-writable, or authorized_keys is writable by anyone but the owner, sshd silently ignores the file and refuses the key. The safe values are 700 on ~/.ssh and 600 on authorized_keys. When key auth mysteriously fails and you fall back to a password prompt, wrong permissions on the server side are the first thing to check.

ssh -L is local port forwarding: it opens a port on your own machine and tunnels everything sent to it, through the encrypted SSH connection, to a destination reachable from the server. ssh -L 5432:localhost:5432 dev makes the server's PostgreSQL, which only listens on its own localhost, appear on your laptop's localhost:5432. You use it to reach internal services — databases, admin dashboards, a metrics endpoint — that are deliberately not exposed to the internet, without opening a firewall port. The traffic is encrypted end to end because it rides inside SSH, so it is far safer than exposing the service publicly just to reach it.

ed25519 is a modern elliptic-curve signature scheme. It gives strong security in a tiny, fixed-size key — the public key is one short line — where an equivalent RSA key must be 3072 or 4096 bits to be comparable, making it slower to generate and verify. ed25519 keys are fast, have no weak-parameter footguns, and are supported everywhere modern: OpenSSH, GitHub, GitLab, every cloud. RSA is still fine and you will meet it on older systems, so keep the ability to read an RSA key, but for anything you generate in 2026 the default answer is ssh-keygen -t ed25519. GitHub's own docs recommend it.

I follow the request down its layers and stop at the first one that fails. First name to IP: I dig the domain and read the answer — intermittent trouble often means two A records where one points at a dead host, so I look for multiple answers. Next reachability: I ping the IP to confirm the host is on the network at all. Then the port: ss -tlnp on the box, or curl from outside, to confirm something is actually listening on 443. Finally the response: curl -I to read the real status line. 'Sometimes' almost always means one member of a set — one DNS record, one load-balancer target — is broken while the rest are fine.

When you request shop.example.com, your machine asks a resolver — your ISP's, or a public one like 1.1.1.1 — to turn that name into an IP. If the answer isn't cached, the resolver walks the hierarchy: it asks a root server, which points to the .com nameservers, which point to example.com's authoritative nameservers, which return the actual A record. The resolver caches that answer for the record's TTL and hands it back. Two things bite you in practice: caching, so a changed record can take until the TTL expires to show up everywhere, and multiple records, where round-robin can send you to a different host on each lookup.

On the box itself I use ss — ss -tlnp lists TCP (t) listening (l) sockets with numeric ports (n) and the owning process (p), so I can see nginx is listening on 443 and not on 80. From outside I test the port with curl -I https://host or nc -vz host 443, because a port can be open locally but blocked by a firewall or security group in between. The old tool was netstat, but it isn't installed on Ubuntu 24.04 by default — ss is the modern replacement and faster on busy hosts. Confirming what's listening is the quickest way to split 'service down' from 'network blocked.'

Ping only proves ICMP round-trips to the host — that the box is powered on and on the network. It says nothing about whether the service is running or the port is reachable. Plenty of things leave ping green while the site is dead: the app crashed, nginx isn't listening on 443, a firewall or security group allows ICMP but blocks TCP 443, or DNS is handing back the wrong IP entirely. So after ping I always test the real path: ss to see what's listening, then curl to make the actual request and read the status code. Ping is a first sanity check, never proof that a service is up.

The shebang — the first line, like #!/usr/bin/env bash — tells the kernel which interpreter should run the rest of the file. Without it, whatever shell happens to invoke the script decides, so a script written for bash can break when a cron job or CI runner hands it to dash or sh instead. I always use #!/usr/bin/env bash rather than a hard-coded /bin/bash, because env finds bash wherever it is installed — that matters on Alpine or macOS where the path differs. The one-line rule I give in interviews: the shebang makes a script self-describing, so ./script.sh runs the same way no matter who launches it.

Because an unquoted variable gets word-split and glob-expanded by the shell before the command ever sees it. If file holds 'my report.txt' and I write rm $file, the shell splits it into two arguments and rm tries to delete 'my' and 'report.txt' — not the file I meant. Quoting it, rm "$file", keeps the value as one argument. The same bug bites empty variables: an unquoted empty $x inside [ ] collapses the test and throws 'too many arguments'. My default is to quote every variable expansion unless I have a specific reason not to. It is the single habit that prevents the most shell-script bugs in production.

A for loop is for a list you already know — files, service names, arguments: for f in *.log; do gzip "$f"; done runs once per item and stops. A while loop is for a condition that changes as you go — you do not know how many iterations it takes. I reach for while when I am waiting on something: while ! curl -sf http://localhost:8080/health; do sleep 2; done polls until a service answers. So the rule is: for when you can count the items up front, while when you are looping until a condition flips. Getting that wrong is how people write for loops that should have been retry loops.

With command substitution: $(command). today=$(date +%F) stores the date; count=$(ls /var/log | wc -l) stores a number. The output replaces the $( ) inline, so scripts react to the live system instead of hard-coded values. I always quote the result when I use it — echo "there are $count files" — for the same word-splitting reason as any variable. The older backtick form works too, but $( ) nests cleanly and is what I write. On client work this is what makes one script reusable across projects: it reads hostnames, dates and counts from the machine it runs on, so the same file behaves correctly on every environment instead of being edited per box.

It is the safety line I put at the top of every serious script. -e makes the script exit the moment any command fails, instead of blindly running the next one. -u turns using an unset variable into an error, so a typo'd variable name stops the script rather than silently expanding to nothing. -o pipefail makes a pipeline fail if any stage fails, not just the last — so mysqldump piped into gzip reports the dump breaking, not only gzip succeeding. Together they turn bash's forgiving defaults into fail-fast behaviour, which is exactly what you want for a backup or deploy script running unattended.

Every command returns an exit code when it finishes — a number the shell stores in the special variable $?. Zero means success; anything from 1 to 255 means failure, and some tools give specific codes, like grep returning 1 when it finds no match. I read it with echo $? right after the command, but in scripts I usually let control flow test it for me — 'if mycommand; then' runs the branch only when the command succeeded. My own functions set their code with return N and a whole script with exit N. That final code is what a CI job or systemd reads to decide whether the step passed.

return ends a function and hands an exit code back to whatever called it — the script keeps running. exit ends the whole script, and if the script is your current shell, it closes that shell. So inside a helper function I use return 1 to signal failure to the caller, which can then decide what to do next. I save exit for the top level, usually in a die() helper that logs a clear message and exits non-zero so the caller — a pipeline or a cron job — knows the run failed. On client work that distinction matters: a reusable function should not kill the whole automation just because one step it does can fail.

Two habits. First, set -euo pipefail at the top so any failing command, unset variable, or broken pipe stops the script immediately instead of continuing in a bad state. Second, a small die() helper — it logs a specific message and calls exit 1 — that I call after guard checks, like testing a directory exists before I try to back it up. So instead of a cryptic tar error halfway through, the user sees 'ERROR: source /data does not exist' and the script stops with a non-zero code. Failing fast with a readable message is what makes a script safe to run unattended and easy to debug from a log.

Both combine work from two branches, but they write different history. git merge takes the two branch tips and creates a new merge commit that ties them together — the original commits stay exactly where they were, so history honestly shows two lines of work that ran in parallel and joined. git rebase instead replays your commits one by one on top of the target branch, giving a straight, linear history as if you had branched from the latest code. The rule I follow: rebase your own local branch to tidy it before sharing, but never rebase commits you have already pushed and others have pulled, because rewriting shared history forces everyone else to untangle it.

A fast-forward merge happens when the branch you are merging into has not moved since you branched off it. Because there is no divergent work, Git does not need a merge commit — it simply slides the branch pointer forward to the tip of your branch, and the history stays perfectly linear, as if the commits were made directly on the target. If the target branch has moved on, a fast-forward is impossible and Git creates a real merge commit instead. In team workflows people often pass --no-ff to force a merge commit even when a fast-forward is possible, so the history records that a feature branch existed and when it landed.

A conflict happens when two branches change the same lines and Git can't decide which wins, so it stops and asks me. I run git status to see which files are conflicted, then open each one — Git marks the clash with <<<<<<<, ======= and >>>>>>> around the two versions. I edit the file to the correct final result, delete every marker, then git add the file to mark it resolved. When all files are staged I finish with git commit for a merge, or git rebase --continue if I was rebasing. The classic mistake is leaving a marker behind and shipping broken code, so I always read the file back before committing.

A .gitignore lists path patterns Git should never track — build output, dependency folders like node_modules, log files, and anything secret such as .env files or private keys. It matters for two reasons. First, noise: committing generated files bloats the repo and fills every diff with churn nobody reviews. Second, and far more serious, security — a committed .env or cloud key is leaked the moment it is pushed, and scrubbing it from history afterward is painful and unreliable, so the credential has to be rotated anyway. My habit is to add .gitignore before the first commit, so secrets and build junk never enter history in the first place.

Both talk to the remote, but fetch is read-only for your working tree. git fetch downloads new commits and updates your remote-tracking branches — origin/main and friends — without changing the branch you have checked out, so it is the safe way to see what landed upstream before you touch anything. git pull is git fetch followed by a merge (or a rebase, if configured) into your current branch, so it downloads and integrates in one step. I fetch when I want to look before I leap, and pull to catch my own branch up. The habit that saves you: pull before you push, so you integrate the team's work before adding yours.

A pull request is a proposal to merge one branch into another — usually your feature branch into main — hosted on a platform like GitHub. It creates a place for review: a teammate reads the diff, leaves comments, requests changes, and CI runs the tests automatically on every push to the branch. Nothing merges until review passes and CI is green. Pushing straight to main skips all of that — no second pair of eyes, no gate before untested code sits on the branch everyone deploys from. Most teams protect main so direct pushes are simply rejected. The PR is also a record: you can later see who approved a change and why.

Semantic versioning is a three-number scheme, MAJOR.MINOR.PATCH, like v2.4.1. You bump MAJOR for a breaking change, MINOR for a backward-compatible new feature, and PATCH for a bug fix. It lets anyone read a version and know how risky the upgrade is. To cut a release I create an annotated tag on the exact commit that shipped — git tag -a v2.4.1 -m 'message' — because an annotated tag stores the author, date and message, unlike a lightweight one. Then I push it explicitly with git push origin v2.4.1, since a normal push does not send tags. On GitHub that tag can become a Release. A tag never moves, so it permanently points to what went out.

That message means the remote has commits I do not have locally — a teammate pushed while I was working — so Git refuses to overwrite their history. The wrong move is force-pushing, which throws their work away. What I actually do is integrate first: git pull to fetch and merge (or rebase) their changes onto mine, resolve any conflicts, re-run the tests, then push again cleanly. On a busy shared branch this happens constantly, which is why teams use short-lived feature branches and merge through pull requests rather than all pushing one branch. Force-push, if ever needed, belongs only on your own unshared branch, with --force-with-lease.

Left to right they are minute, hour, day-of-month, month, and day-of-week, followed by the command. A star means 'every' for that field, so 0 2 * * * runs at 2 a.m. every day, and */15 * * * * runs every fifteen minutes. The field I always flag is the fifth: day-of-week counts from Sunday as 0. And there is a genuine trap — if you set both day-of-month and day-of-week to something other than star, cron treats them as OR, not AND, so 0 0 13 * 5 fires on the 13th and on every Friday, not only on Friday the 13th. Reading a crontab line correctly is half the skill.

Almost always the environment. cron runs your command in a bare shell: a minimal PATH, no .bashrc or .profile loaded, HOME set but little else. So a tool you call by its short name is not found, or a relative path resolves from the wrong directory, and the job dies. The tell is that it works when you run it by hand — because your interactive shell has a rich PATH the cron shell does not. I fix it by using absolute paths for every binary and file, setting any variables the script needs inside the script, and always redirecting output with >> /path/log 2>&1 so the failure gets written down instead of mailed into the void.

On any host running systemd — which is every current Ubuntu, Debian, or RHEL — I default to a timer. You write a .service for what to run and a .timer for when, and you get things cron cannot give you: output goes straight to the journal, so journalctl -u myjob.service shows every run with no manual redirect; Persistent=true catches up a job that was missed while the machine was off; and you get dependencies, resource limits, and proper failure tracking. In a services company that standardises on systemd, timers keep scheduling consistent with how every other service is run and observed. I still reach for cron inside containers or minimal images where systemd is not present.

It depends on the scheduler. With cron, the job's stdout and stderr are by default emailed to the user — and on a server with no mail set up that output just disappears, which is why people think cron 'ran nothing'. So I always redirect explicitly: append >> /var/log/myjob.log 2>&1 to the crontab line, capturing both streams. With a systemd timer there is nothing to wire up — the service's output is captured in the journal automatically, and I read it with journalctl -u myjob.service, narrowing by time with --since. Either way the rule is the same: a scheduled job you cannot see the output of is a job you cannot trust.

Every file has three permission groups — owner, group, others — each with a read, write and execute bit. Reading rwxr-xr-- left to right: the owner has rwx, full read, write and execute; the group has r-x, read and execute but no write; others have r--, read only. ls -l shows this string, and the character before it is the type — a dash for a file, d for a directory. I set it with chmod, either symbolic like chmod g+w file or numeric, where rwxr-xr-- is 754. The habit that matters in an incident: when something can't write, I check the owner and the bits before assuming the code is broken.

On any modern Ubuntu box services run under systemd, so I start with systemctl status nginx — it shows whether the unit is active, its main PID, and the last few log lines on one screen. For the full log I use journalctl -u nginx, and in an incident I add -e to jump to the end, -f to follow it live, or --since '10 min ago' to scope it. If the service is dead I read why in those logs before restarting, because a blind systemctl restart that works for thirty seconds and dies again just hides the real fault. Status first, logs second, restart only once I know the cause.

Both combine work from two branches, but they write history differently. git merge keeps both histories intact and adds a merge commit tying them together — the true record of what happened, but a busier graph. git rebase replays your commits on top of the target branch, producing a straight, linear history as if you'd branched from the latest code — cleaner to read, but it rewrites commit hashes. My rule: rebase your own local feature branch to tidy it before review, but never rebase a branch others have already pulled, because rewriting shared history breaks everyone's copy. On client repos I keep main merge-based so the shipped history stays honest.

Both run jobs on a schedule; the difference is what you get around the job. A cron line like 0 2 * * * is quick and universal, and it's still everywhere. But a systemd timer pairs a .timer with a .service unit, so the job's output lands in the journal — journalctl -u backup — and you get status, automatic logging, dependency ordering, and catch-up if the box was asleep at the scheduled time. On a modern Ubuntu server I default to a systemd timer for anything I need to observe or debug, and reach for cron only for a trivial one-off. The honest answer names cron but recommends the timer, and explains why.

On the box I use ss -tlnp — TCP, listening, numeric ports, plus the owning process — so I can see nginx is on 443 and nothing is on 80. To find what holds a specific port I pipe it: ss -tlnp | grep ':443'. From outside the host I test with curl -I https://host or nc -vz host 443, because a port open locally can still be blocked by a firewall in between. netstat did this job for years but isn't installed on Ubuntu 24.04 by default — ss is the modern replacement. Confirming what's actually listening is the fastest way to split 'service down' from 'network blocked.'

Almost always the environment, not the script. Cron runs jobs with a stripped-down environment: PATH is usually just /usr/bin:/bin, your login profile is never sourced, and the working directory is the user's home, not wherever you happened to be. So a tool in /usr/local/bin or under a version manager isn't found, a relative path resolves somewhere unexpected, and a variable you set in .bashrc is simply absent. My first move is to reproduce it the way cron sees it — run with a clean env — then fix the script to depend on nothing it isn't given: absolute paths, an explicit PATH at the top, and cd into the right directory. Modern systemd timers make this cleaner because you declare the environment.

chmod +x deploy.sh sets the execute bit, and after that ./deploy.sh runs the file as a program. The execute bit is one of the permission bits per class — owner, group, other — from Day 5: without it the kernel refuses to run the file even though you can still read it, which is why the error is Permission denied, not command not found. Two details I always mention: the shebang line, like #!/usr/bin/env bash, tells the kernel which interpreter to use, and bash deploy.sh runs the file regardless of the bit because you're handing it to bash explicitly. The execute bit only matters when you run the file directly.

Yes — git stores a file's mode, and the only variation it records is the execute bit: 100644 for a normal file, 100755 for an executable one. So if someone edits a script through a tool that drops the bit, git diff shows an old mode 100755 / new mode 100644 change with no content diff at all, and git blame points straight at the commit that did it. That's why a deploy script can 'break' with nothing in the code changing — the permission changed. To fix it in the repo you chmod +x and commit, or use git update-index --chmod=+x so the executable mode is what everyone checks out.

For anything on a modern Linux box I default to systemd timers. You still write the schedule in cron-style terms, but you get real logging through journalctl, the job runs as a proper unit you can start and inspect, you can express dependencies on other services, and OnCalendar plus Persistent will catch up a run the machine missed while it was off. Plain crontab is simpler and I'll still use it for a quick personal job, and you'll see it everywhere in older systems and in CI — GitHub Actions schedules workflows with the same five-field syntax. But when reliability and debuggability matter, a timer beats a crontab line whose only trace of failure is an email nobody reads.

A VM virtualizes hardware: a hypervisor gives each VM its own full operating system, including its own kernel, so a VM is heavy — gigabytes on disk and tens of seconds to boot. A container virtualizes the operating system instead: every container on a host shares that host's single Linux kernel and is really just an isolated process, walled off with namespaces and limited with cgroups. With no guest OS to boot, it starts in milliseconds and ships as tens of megabytes. The trade-off is real: VMs give stronger isolation and can run a different kernel; containers give speed, density and identical environments, which is why they won for shipping applications.

Two Linux kernel features do the real work, and Docker just orchestrates them. Namespaces give a container its own view of the system — its own process tree, network, mounts and hostname — so processes inside can't see the host or other containers. cgroups, or control groups, limit and account for resources: how much CPU, memory and I/O the container may use, so one can't starve the others. There's no lightweight VM hiding underneath on Linux — a container is a normal process on the host kernel with those two fences around it. That's also why they're cheap: starting one is basically starting a process, not booting a machine.

An image is the static, read-only template — a packaged filesystem plus metadata like the default command. A container is a running or stopped instance of an image, with a thin writable layer on top. It's exactly class versus object, or a program on disk versus a process: one image, many containers. That's why docker run ubuntu twice gives two independent containers from the same single ubuntu image, and why docker images still lists one image while docker ps -a lists both containers. In short, you build and pull images; you run, stop and remove containers. Keeping those verbs straight clears up most early Docker confusion.

It's a real trade-off, and naming it earns credit. Because every container shares the host's one kernel, isolation is weaker than a VM's: a kernel bug or a container escape can in principle reach the host or its neighbours, whereas each VM has its own kernel behind a hypervisor. In practice you manage it — don't run as root, drop Linux capabilities, keep the host kernel patched, and scan images. For genuinely untrusted or multi-tenant workloads you reach for stronger isolation like gVisor or Kata Containers, which put a real boundary back, or just use separate hosts. So containers trade some isolation for speed and density, and you harden rather than assume a VM-grade wall.

An image is a read-only template — your app plus its dependencies and a start command, frozen into layers. A container is a running instance of that image with a thin writable layer on top. The analogy I use is a class and its objects: one image, many containers. docker pull or docker build gives you images; docker run turns an image into a container. Deleting a container leaves the image untouched, so I can start a fresh one instantly. That split matters operationally — images are immutable and shippable, containers are disposable and often stateful, which is why you never keep important data in a container's writable layer without a volume.

latest is just the default tag Docker uses when you don't specify one — docker pull nginx really means nginx:latest. The trap is that latest isn't 'the newest version'; it's whatever the publisher last tagged as latest, so it moves under you. Two machines pulling nginx:latest a month apart can get different images, which quietly breaks reproducibility. In production I pin an explicit, immutable tag like nginx:1.27-alpine, or better a digest, so the image I tested is exactly the image that ships. latest is fine for a quick local experiment, but in a Dockerfile FROM or a deploy it's a classic source of 'works on my machine' bugs.

I use docker exec -it <container> sh, or bash if the image has it. exec runs a new process inside an already-running container; -i keeps stdin open and -t allocates a terminal, so together they give me an interactive shell. From there I can read config, look at files, or run the app's own CLI. The key distinction is exec versus run: run starts a brand-new container, exec attaches to one that's already up — reaching for run when I meant exec is a common slip. For a container that has already crashed I can't exec into it, so I read docker logs <container> instead, which shows what the main process wrote to stdout and stderr.

A registry is a server that stores and distributes images — you push images to it and pull them from it. Docker Hub is the default public registry, so docker pull nginx fetches from it automatically. Images are addressed as repository:tag, like nginx:1.27, and a registry can host public images anyone can pull or private ones that need docker login first. In real work you'll also meet private registries — cloud providers and code hosts each run one — but the commands don't change; only the address in front of the image name does. The mental model is simple: the registry is the shared library, docker pull borrows a copy, and docker run brings it to life locally.

RUN executes a command at build time and bakes the result into a new layer — that's how you install packages or compile code. CMD and ENTRYPOINT both run at container start, not build. The difference between those two: ENTRYPOINT sets the executable that always runs, and CMD supplies the default arguments, which anyone can override on docker run. A common pattern is ENTRYPOINT for the binary and CMD for its default flags. If you just need a simple default command, CMD alone is fine. The mistake I watch for is putting a startup command in RUN — it runs once during the build and is gone by the time the container actually starts.

Docker builds an image as a stack of layers, one per instruction, and caches each one. On a rebuild it reuses cached layers until it hits the first instruction whose inputs changed — from there down, everything is rebuilt. So order is a performance decision: put what rarely changes near the top and what changes constantly near the bottom. The classic example is copying your dependency manifest and installing packages before copying the rest of your source. That way editing one line of application code doesn't invalidate the expensive dependency-install layer, and your rebuilds drop from minutes to seconds. Get the order backwards and every edit reinstalls everything.

A .dockerignore file lists paths Docker should exclude from the build context — the set of files sent to the daemon when a build starts. Without it, docker build ships your entire directory, including node_modules, the .git folder, local env files and build artifacts, which is slow and can leak secrets into the image. With it, those never get sent or copied. It works like .gitignore: one glob pattern per line. The first entries I add on almost any project are node_modules and .git. It also makes COPY . . safe, because the junk you'd never want in an image is filtered out before COPY even runs.

A few habits do most of the work. Start from a slim base — python:3.12-slim or node:20-alpine instead of the full image — which alone can cut hundreds of megabytes. Combine related RUN commands and clean up in the same layer, because deleting files in a later layer doesn't shrink the earlier one that added them. Use .dockerignore so build junk never enters. And the biggest lever is a multi-stage build: you compile in a heavy builder stage and copy only the finished artifact into a tiny final image, leaving the compilers and dev dependencies behind. That's Day 29, but it's the technique that takes an image from gigabytes to tens of megabytes.

-p publishes a container's port to the host so traffic from outside can reach it. You read it left-to-right as host:container: -p 8080:80 means 'send anything arriving on the host's port 8080 into the container's port 80.' Without it, the container's port is only reachable from other containers on the same network, never from your laptop or the internet. The mistake I watch for is flipping the two numbers — -p 80:8080 forwards host port 80 to a container port nothing is listening on, and you get connection refused. So I always say it out loud when I type it: host first, container second.

EXPOSE is documentation. It's a line in the Dockerfile that records which port the app listens on — it opens nothing by itself and doesn't make the port reachable. -p, at run time, actually publishes the port to the host and sets up the NAT rules that forward traffic in. So EXPOSE 80 just tells the next person 'this app uses 80'; docker run -p 8080:80 is what lets you hit it. The one place EXPOSE has teeth is docker run with a capital -P, which publishes every EXPOSEd port to a random high host port. In interviews I sum it up as: EXPOSE declares, -p connects.

You put them on the same user-defined network and let them reach each other by container name. When I run docker network create appnet and start both containers with --network appnet, Docker runs an embedded DNS server that resolves each container's name to its current IP, so the app connects to 'db' or 'api' as a hostname — no IP addresses hard-coded. The gotcha I always flag: the default bridge network does not give name resolution, only user-defined ones do, which is one reason everyone uses compose. And you don't need -p for this — publishing is only for traffic from outside the host; container-to-container traffic stays on the internal network.

Every Docker install ships a default network called bridge, and any container you run without --network lands on it. It works, but it has two real limits: containers on it can only reach each other by IP address, not by name, and every container shares one flat network with no isolation. A user-defined bridge, created with docker network create, fixes both — it gives you automatic DNS so containers find each other by name, and it isolates that group of containers from everything else. So for anything beyond a one-off container I create a network per application, which is exactly what docker compose does for you automatically.

A container's filesystem is a thin writable layer stacked on the read-only image layers. Anything the process writes — logs, uploaded files, database rows — lands in that top layer, and that layer is created with the container and destroyed with it. So docker rm throws the data away along with the container; stopping and restarting the same container keeps it, but removing the container loses it. That is by design: containers are meant to be disposable and identical, so you can kill one and start a fresh copy without a second thought. Anything you need to outlive the container has to be written to a volume or a bind mount, which lives outside the writable layer. The rule I use is: image is the app, volume is the data, never store state in the container itself.

Both give a container storage that survives it, but they differ in who owns the path. A named volume is managed by Docker — you give it a name, Docker stores it under its own directory, and you never care exactly where. It is the right default for data like a database, because it is portable across hosts and Docker handles permissions. A bind mount maps a specific host directory into the container, so the container sees your actual files at a path you choose. That is ideal in development — mount your source code so edits show up live — but it ties the container to that host's layout and can bring host permission quirks. My rule: named volumes for data the app owns, bind mounts for files I own and edit.

I mount a named volume at the database's data directory. For Postgres that is /var/lib/postgresql/data, so docker run -v pgdata:/var/lib/postgresql/data postgres:17. The image writes all its state there, and because the volume outlives the container I can destroy and recreate the container — for an upgrade, say — and the data is still there when the new one mounts the same volume. Two things I am careful about: removing the container does not remove the volume, so cleanup has to be deliberate, and I never run two containers writing the same database volume at once. In production the data usually lives on managed storage or a real database service, but the volume pattern is exactly how you would run it locally or in a simple deployment.

They do the same job with different ergonomics. -v is the short, older syntax: -v pgdata:/var/lib/postgresql/data. It is compact, but it has a sharp edge — if the source has no leading slash Docker treats it as a named volume, so a mistyped bind path silently creates a surprise volume instead of erroring. --mount is the explicit key-value form: --mount type=volume,source=pgdata,target=/var/lib/postgresql/data, which is more verbose but self-documenting and fails loudly on a bad path. For quick interactive work I will use -v; in scripts and compose files, where clarity matters and a typo is expensive, I prefer --mount. Both end up in the same place — it is about how obvious the intent is.

Compose replaces a pile of docker run flags and docker network create commands with one declarative file. A real app is more than one container — a web service, a database, maybe a cache — each needing ports, volumes and a network to find the others. Doing that by hand is fragile and lives only in your shell history. A docker-compose.yml describes the whole stack: every container is a service, and one docker compose up -d creates the network, the volumes and every container in the right order. Anyone who has the file reproduces the identical environment, which is why Compose is the standard way to run multi-container apps in local development.

No, and this trips everyone up. depends_on only controls start order — it waits for the dependency's container to be created and started, not for the process inside to be ready to serve. So web can start the instant the db container exists while Postgres is still initialising, and web's first connection gets refused. To wait for readiness you add a healthcheck to the db — something like pg_isready — and gate the dependent with the long form depends_on with condition: service_healthy. The alternative, which I prefer for resilience, is to make the app retry its connection, because the database can also disappear at runtime, not just at startup.

Compose creates one network for the project automatically and attaches every service to it, so they reach each other by service name. If I have web and db services, web connects to the host db on its container port — no IP addresses, no links, no published ports needed for internal traffic. Docker's embedded DNS resolves db to the container's current IP, which matters because that IP changes on restart while the name never does. I only publish ports with the ports key when I want traffic from the host or the outside world; service-to-service traffic stays on the internal network. I declare custom networks only when I need isolation between groups of services.

stop halts the running containers but leaves them, the network and the volumes in place, so a later start brings the same containers back quickly. down is the teardown: it stops and removes the containers and the project network, giving you a clean slate. The key detail for anyone with data: down keeps named volumes by default, so your database survives — you only lose them if you add -v, which is a foot-gun in the wrong directory. In practice I use stop to pause work I'll resume and down to reset an environment, and I'm careful never to run down -v anywhere near real data.

Usually it isn't the image — the same image runs both places. It's what the container depends on and can't find. Started alone with docker run, an app has no database container, no shared network and no volume, so it dies the moment it tries to reach a db that was never started. Other classics: a bind-mount path that exists on your laptop but not the server, a missing environment variable or secret, or a host port already taken. My move is docker logs to read why it's dying, then check whether every dependency it assumes — another container, a network, a volume, an env var — actually exists on that host.

The moment the app is more than one container, which is almost always. A bare docker run starts a single container; a real app is a web process plus a database, maybe a cache and a queue, that all need to find each other on a network and keep their data in volumes. Compose lets me declare every service, network and volume in one compose.yaml and bring the whole stack up with docker compose up -d — reproducibly, in the right order. I'll still use docker run for a quick one-off, like a throwaway shell in an image. But anything with dependencies or state I want described in a file, not typed as an ever-growing run command.

I start with docker ps to confirm it's actually restarting rather than exited, then docker logs on it — the application almost always prints why it died right before it does. From there it's usually one of a few things: it can't reach a dependency like a database, a required environment variable is missing, the command in the image exits immediately, or it's out of memory. docker inspect shows the restart policy, the env, the mounts and the networks it's attached to. The key habit is reading the logs first instead of guessing — a restarting container is rarely a broken image, it's usually something in its environment or its dependencies that isn't there.

An image is the read-only template — the filesystem, dependencies and default command, built from a Dockerfile and stored in a registry like Docker Hub. A container is a running instance of that image, with a thin writable layer on top. The relationship is like a class and an object, or a program on disk versus a process: one image can spawn many containers, and each gets its own writable layer, so changes inside a running container don't alter the image. That's also why data written inside a container vanishes when it's removed unless you mount a volume — the writable layer is disposable by design, and the image it came from never changed.

A multi-stage build puts more than one FROM in a single Dockerfile. Each FROM starts a new stage; you name one — FROM golang:1.23 AS builder — do the heavy work there, compilers and go build, then start a fresh tiny final stage and COPY --from=builder only the finished binary into it. Everything in the builder — the toolchain, source and caches — is discarded. You use it because the shipped image ends up containing the artifact and nothing else: it pulls faster, costs less to store, and exposes a far smaller attack surface. A compiler or shell you left in by accident is something an attacker can use, so leaving them behind is a security win, not just a size one.

A few habits do most of the work. Start from a slim base — python:3.12-slim or node:20-alpine — which alone can cut hundreds of megabytes. Combine related RUN commands and clean up in the same layer, because deleting files in a later layer doesn't shrink the earlier one that added them. Use a .dockerignore so build junk never enters the context. But the biggest lever is a multi-stage build: compile in a heavy builder stage and copy only the finished artifact into a tiny final image, leaving the compilers and dev dependencies behind. For a compiled language that final stage can be distroless or scratch, taking an image from gigabytes down to tens of megabytes for the same app.

By default a container's process runs as root, and that root is the host's root — user namespaces aside, uid 0 inside is uid 0 outside. So if an attacker escapes the container through a kernel or runtime bug, they land as root on the node. Running as a non-root USER means a breakout lands as an unprivileged user instead, which is defence in depth. It also catches bad habits early: a process that assumes it can write anywhere fails loudly in testing rather than in production. I either add a USER line in the Dockerfile or start from a base that is already non-root, like distroless :nonroot, and I keep the filesystem read-only where the app allows it.

I run a scanner like docker scout or Trivy against the built image — docker scout cves myimage:1 or trivy image myimage:1. It reads the image's package inventory — the OS packages and the language dependencies baked into the layers — and matches their exact versions against public CVE databases. The output lists known vulnerabilities by severity so I can decide what to patch. The key is where it runs: I wire it into CI so a critical CVE fails the pull request, not production, and I rebuild on a fresh base to pick up upstream fixes. Scanning also explains why a slim or distroless base is safer — fewer packages means fewer things that can have a CVE in the first place.

First I run docker ps and read the status — 'Restarting (1)' tells me it's crash-looping and the number is the exit code. Since it isn't currently running I can't exec into it, so I read docker logs on it; the logs persist across restarts and almost always show the real cause — a missing env var, a config file that isn't there, a port it can't bind. If the logs are thin I run docker inspect and pull .State with jq to read the exit code and error string, and check whether a --restart policy is bouncing it. The loop is the same failure repeating, so I fix that one cause rather than restarting blindly.

docker logs shows what the main process already wrote to stdout and stderr — it's read-only history and my first move on any issue. docker exec starts a brand-new process inside a running container, like a shell, so I can poke around live without touching the main process. docker attach connects my terminal to the main process's own stdin and stdout — occasionally useful but risky, because Ctrl-C there can kill the container. In practice I live in logs and exec: logs to see what happened, exec -it sh to investigate from inside. attach I mostly avoid. Context matters too: exec and attach need the container running, while logs still works after it has exited.

docker inspect dumps the full JSON Docker holds about a container or image — its config, environment variables, mounts, networks, IP address, restart policy, and live state including the exit code. I reach for it when logs aren't enough and I need ground truth about how the container was actually configured, not how I assume it was. Because it's a huge blob I pipe it through jq to pull one branch, like docker inspect web | jq '.[0].NetworkSettings.IPAddress', remembering the .[0] because inspect returns an array. Classic uses: confirming which volume is really mounted, which network a container joined, or reading the exit code of one that died.

docker stats is the quickest look — it streams live CPU percent, memory used against the container's limit, network and block I/O, and PID count, refreshed each second. I add --no-stream for a single snapshot in a script. docker top lists the processes running inside the container, which tells me whether it's one runaway process or many. These read the same Linux cgroup counters that production monitoring tools graph, so they aren't toys — they're the same signal. If a container is being OOM-killed I'll see memory pinned at its limit in stats and a 137 exit code in docker inspect, which together point straight at 'raise the limit or fix the leak.'

Continuous integration means every push is automatically built and tested against the shared main branch, so breakage is caught in minutes instead of at a scary merge week. Continuous delivery adds to that: the tested build is packaged and kept always ready to release, and a human clicks or merges to actually ship it. Continuous deployment removes even that click — every green build goes straight to production on its own. The mental order I keep is CI proves the code is good, delivery makes it releasable, deployment releases it automatically. Most teams do CI plus continuous delivery and gate production behind an approval; full continuous deployment is a maturity and confidence decision, not a tooling one.

A pipeline is the ordered sequence of automated steps a code change runs through on its way to production, defined as code and triggered on every push. The stages are almost always the same three: build, test, then deploy. Build turns source into a runnable artifact — a Docker image, a jar, a bundle. Test runs unit and integration checks against that artifact. Deploy ships it to an environment. The key rule is that each stage only runs if the one before it passed, so the first failing stage stops the line and broken code never reaches the next step. That fail-fast ordering is the whole point: you find out something's wrong at build or test, not from users in production.

An artifact is the single built thing the pipeline produces once in the build stage — the image or package — and then carries forward unchanged. The discipline that matters is build once, promote the same bytes: you test that exact artifact, then move that same artifact to staging, then that same artifact to production. You never rebuild per environment. If you rebuilt for prod, you'd be shipping something you never actually tested — a slightly different dependency, a new base image — which quietly breaks the guarantee that 'it passed in staging.' So the artifact is the unit of promotion, and its identity staying constant is what makes each environment's green result trustworthy.

It buys speed and safety at the same time, which sounds contradictory until you automate. Manual deploys are slow, done rarely, and done by a stressed human at night, so each one is large and risky. A pipeline makes releasing boring: small changes ship often, every one is built and tested identically, and a rollback is just redeploying the last good artifact. That shrinks the blast radius — a bug in a tiny change is easy to find and undo. It also removes the bus factor and the 'works on my machine' gap, because the machine that tests is the machine that ships. In one line: CI/CD turns deployment from a scary event into a routine, repeatable, reversible action.

A workflow is an automated process defined in a YAML file you commit to your repo under .github/workflows/. GitHub watches that folder and runs the file when a matching event fires. The anatomy is small: name labels it, on lists the triggering events, and jobs holds one or more jobs. Each job picks a machine with runs-on and runs an ordered list of steps. A step either uses a prebuilt action or runs a shell command. Because the file lives in the repo, the pipeline is versioned with the code — a pull request can change the build itself, and reviewers see it. That's the whole model: push code, a file describes what to do, GitHub does it.

Both fire on code changes, but at different moments. push fires when commits land on a branch — it's your after-the-fact safety net, building whatever was just pushed. pull_request fires when a PR is opened or updated, and it runs against the merge result — your branch combined with the target — before anything lands. That's what produces the green check reviewers gate merges on: it answers 'will main still build if we merge this?' In practice I use both. push gives every commit a status; pull_request protects the branch everyone shares. Listing several events under on: means any one of them starts the run.

runs-on tells GitHub which machine the job needs. runs-on: ubuntu-latest requests a GitHub-hosted runner — a fresh, throwaway Ubuntu VM that GitHub provisions for this run and destroys afterward, so every run starts clean with no leftover state. A runner is simply the machine that executes your steps; hosted runners come with common tools preinstalled, and you can register your own self-hosted runners for special hardware or private networks. That throwaway nature is why nearly every workflow's first step is actions/checkout — the runner starts empty, without even your code, so you must clone it in before any step can see it. A clean box every time is the point.

A step is one item in a job, and it's one of two kinds. uses: pulls in a published, versioned action — reusable code someone packaged, like actions/checkout@v4, which clones your repo onto the runner. run: executes a shell command directly on the machine, exactly like typing it in a terminal — run: npm test, say. My rule of thumb: reach for an existing action when one fits the task (checking out code, setting up a language, logging into a registry) and drop to run for your own project commands. Pinning the action version with @v4 matters — it stops a surprise update from silently breaking your build.

By default jobs in a workflow run in parallel on separate runners, so to force an order I add needs: to the dependent job. needs: test on a deploy job holds it until test succeeds; needs: [lint, test] waits for both. Anything without a needs relationship keeps running concurrently, so I get parallelism where it's safe and ordering where it matters — lint and test together, build after both, deploy last. A failed dependency skips the jobs that need it, which stops a broken build from ever reaching deploy. It's the single mechanism for expressing 'this must happen before that' across jobs.

A matrix runs the same job many times with different inputs, defined under strategy.matrix. matrix.node: [20, 22, 24] expands into three parallel jobs, each with the matrix.node expression set to one version. I reach for it whenever I support more than one version or platform — testing across Node or Python versions, or Linux and Windows runners — because it replaces three near-identical copied jobs with one definition. Add a second key and it multiplies: node times os becomes six jobs. It keeps the workflow short and guarantees every combination is actually tested, and I can drop unwanted pairs with exclude or turn fail-fast off for full coverage.

Jobs run on separate machines and share no filesystem, so I use job outputs. In the producing job a step writes to the special file — echo tag=v1.2.3 >> $GITHUB_OUTPUT — gives itself an id, and the job maps that under outputs:. The consuming job declares needs: on the producer and reads it as needs.build.outputs.tag. That needs link is required — without it the output isn't visible. I use this for things like a computed version tag or an image digest that build produces and deploy consumes. For values inside a single job I'd use env or step outputs instead; job outputs are specifically the cross-job channel.

Secrets are stored in the repo's settings, not the code, and referenced as secrets.NAME; GitHub injects the value at run time and masks it in logs as three asterisks. You never hard-code a token in YAML because the file is readable by anyone with repo access and lives in git history forever — a leak you can't take back. Repository secrets are available to every workflow; environment secrets attach to an environment like production and can require an approval to unlock, which adds a gate before a deploy. One caveat I mention: secrets aren't passed to workflows triggered by pull requests from forks, so fork CI can't exfiltrate them.

Three official actions do the whole job. actions/checkout@v4 puts my Dockerfile on the runner, docker/login-action@v3 authenticates to the registry, and docker/build-push-action@v6 builds the image and, with push set to true, uploads it. For GitHub's own registry, ghcr.io, I log in with github.actor as the username and the built-in GITHUB_TOKEN as the password, and I add permissions: packages: write to the job so that token is allowed to publish. The build-push step gets a tags list — I tag with the commit SHA and often latest. That's a full continuous-delivery step: every push to main rebuilds and republishes the exact image that code produced.

GITHUB_TOKEN is a short-lived credential GitHub mints automatically for every workflow run and destroys when the job ends. Because it's scoped to just that run, there's no secret to create, rotate, or leak — a personal access token, by contrast, is a long-lived password you'd have to store and manage. By default the token can read the repo but can't publish packages, so I add permissions: packages: write to grant exactly that and nothing else. That's least privilege: the token can push images to GHCR for this repository and expires minutes later. Reaching for a PAT here is a common over-permissioned mistake — the built-in token is both safer and less work.

latest is a moving pointer — it's whatever the pipeline pushed last, so two pulls a day apart can be different images. In CI I tag every build with the commit SHA, like ghcr.io/owner/app:9f3c1a2, which is an immutable pointer to the exact code that produced it. If a release misbehaves I can read the running tag and know the precise commit, and I can pull that same image months later and get identical bytes. I usually push latest as well for convenience, but I deploy the SHA. That traceability — image to commit — is what makes rollbacks and incident forensics fast instead of guesswork.

It wraps Buildx, Docker's modern builder, so out of the box you get a clean build context, a proper tags list, and one step that builds and pushes together instead of two brittle shell commands. It hands you cache-from and cache-to so a CI build can reuse layers from a previous run and finish in seconds, plus multi-platform builds — amd64 and arm64 from one workflow — and build provenance and SBOM attestations for supply-chain security. You could script raw docker build and docker push, but you'd re-implement caching and multi-arch by hand. The action is the maintained, current way teams build images in GitHub Actions.

Pin to a full commit SHA — actions/checkout@11bd719… — instead of a moving tag like @v4. A tag is a pointer the action's maintainer can repoint at any time; a SHA names one immutable commit. That matters because a workflow runs third-party code with access to your repo and secrets: if a popular action is compromised and its tag rewritten, every repo on that tag pulls the malicious code automatically. That's exactly what happened with tj-actions in 2025. Pinning to a SHA means you run only the code you reviewed, and you update deliberately by bumping the SHA — ideally with Dependabot watching for new releases. Tags are convenient; SHAs are safe.

Every workflow gets an automatic GITHUB_TOKEN to talk to the GitHub API. By default it can be broad, so a compromised step could push code or packages you never intended. A top-level permissions block scopes it down — I set contents: read as the baseline and add a narrow grant like packages: write only on the job that pushes to GHCR. It's least privilege applied to CI: the token can do exactly what the pipeline needs and nothing more, so a hijacked action has a tiny blast radius. I always set it explicitly rather than trusting repo defaults, because those vary between repositories and an unset block is an easy thing to forget.

Two levers. First, fail-fast ordering: run the cheap checks — lint, unit tests — as a job the expensive build needs, so a broken commit dies in seconds instead of after a five-minute image build. Second, caching: cache dependencies and Docker layers with cache-from and cache-to: type=gha, so a rebuild reuses everything that didn't change. A fast pipeline is one people actually keep in the loop; a slow one gets bypassed with 'I'll just push straight to main.' I also keep independent jobs parallel and only serialize with needs where there's a real dependency. Speed and early failure aren't polish — they're what makes the pipeline trustworthy.

I scan the built image for known CVEs before it's pushed to the registry, and fail the job on a critical finding — a vulnerable image should never become the artifact a deploy pulls. The current tools are docker scout, built into Docker, and Trivy; both index the image's packages against advisory databases. Scanning after build but before push means a bad image stops in CI, not in production. I pair it with the image hygiene from earlier in the week — a slim, multi-stage image has fewer packages, so fewer things that can be vulnerable. Scanning isn't a one-off audit; it runs on every build, because new CVEs land against images that were clean yesterday.

Because local runs are optional, and optional checks get skipped under deadline pressure — someone forgets, or runs a stale version, or has a slightly different setup. CI removes the choice: it runs the exact same lint and tests, the same way, on a clean machine, on every push and pull request. The result is a shared, trustworthy signal — a green check means the code passed here, not 'passed on someone's laptop.' It also runs against the merge result of a PR, so it answers whether main will still be green after merging. Local runs are still useful for fast feedback; CI is what the team actually gates on.

A quality gate is a threshold that turns a measurement into a pass-or-fail rule. Coverage is the classic one: on its own it's just a number, but pytest --cov-fail-under=80 makes it a gate — if coverage lands below 80%, pytest exits non-zero. That's the mechanism behind every gate: the command returns a non-zero exit code, which fails the step, fails the job, and paints a red X. Lint works the same way — ruff check exits 1 on a finding. You don't write custom stop logic; the non-zero exit is the stop. Wire that red X to a required status check and the gate also blocks the merge.

Linting reads your code without running it and flags problems the parser can see — unused imports, undefined names, unreachable code, style violations. Tools like ruff for Python or eslint for JavaScript do this in milliseconds because they never execute anything. Unit tests do the opposite: they actually run small pieces of your code and assert it behaves — add(2, 2) returns 4. pytest and vitest run those. They're complementary, not alternatives: lint catches whole categories of mistakes cheaply before a single test runs, and tests catch logic errors lint can't see. A good pipeline runs lint first because it's fastest, then the tests.

A failing CI run only publishes a status; by itself it doesn't stop anyone from clicking merge. The missing piece is branch protection: on GitHub you add a rule to the default branch that marks the CI check as a required status check. Once it's required, GitHub disables the merge button until that check is green, so a red X genuinely blocks the merge. Without that rule the pipeline is advisory — it reports, but doesn't enforce. So the fix isn't in the workflow YAML at all; it's in the repository settings. That gap between 'we have tests' and 'broken code can't merge' is exactly what branch protection closes.

Semantic versioning is a three-part number, MAJOR.MINOR.PATCH — for example 2.4.1 — where each part signals what changed since the last release. You bump PATCH for backward-compatible bug fixes, so 2.4.1 to 2.4.2 is always safe to take. You bump MINOR for new features that don't break anything, and PATCH resets to zero. You bump MAJOR for a breaking change — something that used to work now won't — and both MINOR and PATCH reset. The value is the contract: a consumer reads the number and knows whether upgrading is safe or needs care. React uses it publicly — 18 to 19 is a major because APIs were removed.

A lightweight tag is just a name pointing at a commit — no extra data. An annotated tag, created with git tag -a, is a full git object that also stores who tagged it, when, and a message, and it can be signed. For releases you always want annotated: the tagger, date and message are part of the record, and git describe — which build scripts use to derive a version string — only considers annotated tags by default. A lightweight tag makes git describe fail or fall back to a bare SHA. Lightweight tags are fine for a quick private bookmark, but anything you release or ship should be annotated and pushed.

A commit SHA like 9f3c1a2 tells nobody what changed or whether upgrading is safe, and it's easy to lose. A release turns one exact commit into a named, meaningful point: a version number, an annotated tag pinning the commit, human-readable notes, and downloadable artifacts. That gives you three things deploying a raw commit doesn't — traceability, because the version maps to an exact commit and its changelog; communication, because release notes tell users what moved; and clean rollbacks, because 'roll back to v0.2.0' is unambiguous where 'roll back a few commits' isn't. It's the difference between shipping something you can talk about and reason about versus shipping an anonymous checkpoint.

Conventional commits put a type at the front of every commit message — fix:, feat:, or a feat!: / BREAKING CHANGE: marker — so the git log itself tells you the next version. The rule maps cleanly to semver: any commit since the last tag that's a fix: means at least a PATCH bump, any feat: means a MINOR bump, and any breaking-change marker means a MAJOR bump. You take the highest one present. That's what lets tools like semantic-release or release-please compute and tag the version automatically, and it's why a changelog can be generated from the log. Even by hand, it turns 'what's the next version?' from a judgment call into reading the commit types.

All three swap a running version for a new one without downtime; they differ in how traffic moves and what you pay. Rolling replaces instances a few at a time, health-checking each before the next — cheapest, no extra machines, but old and new run together and rollback is slow. Blue-green runs two full environments and flips all traffic from the live one to the idle one in a single step — rollback is instant because the old one is still up, but you pay for double capacity during the switch. Canary sends a small slice — 1, then 10, then 50 percent — watching errors and latency before widening; smallest blast radius, but the most tooling to run.

A health check is an endpoint the deploy polls — usually /healthz or /readyz — that returns 200 only when the instance is genuinely ready to serve. Every strategy gates on it: a rolling deploy won't move to the next instance until the new one is healthy, a blue-green switch waits for green to pass, and a canary is judged on the slice's health. Once a machine watches that signal against a threshold, rollback stops being a human decision — if the error rate or a failing probe crosses the line, the system stops the rollout and reverts on its own. The rule underneath is simple: never shift live traffic to a version that hasn't proven it's healthy.

Blue-green when a clean, instant cutover matters more than gradual exposure — a change you can't easily serve half-and-half, like a big framework upgrade — or when you want the simplest rollback: flip back to the environment that's still running. You pay for two full environments during the switch. Canary when you want to catch a bad release with the smallest blast radius and you have the traffic and metrics to judge a small sample — good for high-traffic user-facing services where 1 percent is still a real signal. It costs less capacity than blue-green but far more machinery: traffic splitting and reliable per-slice metrics. Low-traffic internal service? Rolling is usually enough.

State — usually the database. Blue-green makes the app rollback instant, but both environments share the same data, so a schema migration that isn't backward-compatible can strand you: if the new version dropped or renamed a column the old one needs, flipping back to blue hits a database it can no longer query. Same trap in rolling and canary, which run both versions at once by design. The fix is expand-then-contract: add the new column, deploy code that writes both, backfill, and only drop the old column a release later once nothing reads it. Decouple destructive schema changes from the deploy, and your fast rollback stays fast.

Because a workflow file is committed to the repo, so any key you paste in is now in git history forever — readable by anyone with repo access, and copied into every fork and clone. Deleting it later doesn't help; it stays in history. Instead, store the value as a GitHub Actions secret: it's encrypted at rest, injected at run time as ${{ secrets.NAME }}, and masked if it ever prints to a log. For cloud access I go one better and use OIDC, so there's no stored key at all — the runner gets a short-lived token per run. Rule of thumb: nothing sensitive in the file, and short-lived credentials over long-lived ones.

OIDC (OpenID Connect) lets a workflow prove its identity to a cloud provider and get a short-lived access token for that single run, instead of you storing a permanent access key as a secret. You configure a trust relationship — this repo, this branch, can assume this role — and add id-token: write to the job's permissions. The runner then exchanges a signed token for temporary credentials that expire in minutes. It's better because there's no long-lived key to leak, rotate, or find in an old secret store; access is scoped to the exact workflow and dies with the job. It's the same short-lived-beats-long-lived principle as GITHUB_TOKEN, extended to AWS, GCP or Azure.

Trivy is an open-source scanner from Aqua Security that finds known vulnerabilities (CVEs) in container images, filesystems and IaC. Point it at an image with trivy image myapp:tag and it lists every CVE in the OS packages and app libraries, each with a severity and the version that fixes it. In a pipeline I add it as a step after the build and turn it into a gate: --severity HIGH,CRITICAL --exit-code 1, so the job fails and the vulnerable image never ships. I usually generate an SBOM at the same time so we can answer 'are we affected?' fast when a new CVE lands. It's a security quality gate, exactly like the test gate earlier in the pipeline.

A tag like @v4 is a moving pointer — whoever controls the action's repo can re-point it, so the code running in your pipeline can change without you touching anything. That's a real supply-chain risk: a compromised or malicious action runs with your secrets and token. Pinning to a full commit SHA (@a1b2c3…) freezes exactly the code you reviewed; it can't change under you. The trade-off is you stop getting updates automatically, so I pair SHA pins with Dependabot, which opens PRs bumping the SHA to a new reviewed version. Same reasoning as pinning application dependencies to exact versions — reproducible, and no surprise code slipping in through a floating reference.

On push, GitHub matches the event to a workflow under .github/workflows and spins up a fresh runner. It checks out the code, runs the tests and any quality gates, and — if they pass — builds a Docker image from the Dockerfile. It tags that image with the commit SHA, logs in to a registry, and pushes it. A deploy step then pulls that exact tagged image and rolls it out, usually rolling or blue-green so there's no downtime. The thread running through all of it is traceability: the image in prod maps back to one commit, one test run and one scan, so I know exactly what shipped and can roll back to that tag.

First I read where it fails. Build and test are green and only the push to the registry dies, so it's a credentials problem, not a code one. 'Unauthorized: authentication required' means the login step handed the registry an empty or invalid token. My first check is the secret the workflow references — was it rotated, renamed, or never added to this repo? A registry token that was rotated and never re-added is the classic cause: the workflow still names the secret, but its value is gone, so login sends nothing and the registry rejects the push. The fix is to set the secret again — gh secret set REGISTRY_TOKEN — and re-run. I'd also confirm the new token still has push scope.

Secrets never go in the repo or the Dockerfile — anything committed is effectively public and lives forever in git history. In GitHub Actions I store them as encrypted repository or environment secrets and reference them as ${{ secrets.NAME }}; the runner injects them at run time and masks them in logs. For registry pushes I prefer the built-in GITHUB_TOKEN — short-lived and scoped to one run — over a long-lived personal token. Build-time secrets use --secret mounts, not build args, so they don't bake into a layer. And I grant least privilege: packages: write and nothing more, so a leaked token can do as little as possible.

All three replace an old version with a new one without a hard cutover. A rolling deploy swaps instances a few at a time, so old and new run side by side until every instance is updated — simple, but a bad version reaches everyone gradually. Blue-green keeps two full environments: you deploy to the idle one, test it, then flip all traffic at once, which makes rollback instant — you just flip back. Canary sends a small slice of traffic, say 5%, to the new version, watches the metrics, and ramps up only if it stays healthy, so a bad release hits few users. I choose based on the risk of the change and how fast I need to roll back.

It's a cheap, machine-readable way for other systems to ask 'are you alive?' without sending real traffic. A load balancer, an orchestrator like Kubernetes, or Docker Compose polls it on a schedule and only routes work to instances that answer 200. Keeping it separate from business routes matters: a basic liveness check shouldn't require auth or a database round-trip, so a momentary DB blip doesn't get the whole app killed — a deeper readiness probe that touches dependencies is a second endpoint. I add /healthz on the first commit because every later layer — Compose, CI smoke tests, the deploy target — wants it, and retrofitting it is pure friction.

The virtual environment (.venv/), byte-compiled caches (__pycache__/, *.pyc), local secrets (.env), and any local database or build artifacts. None belong in version control: the venv is machine-specific and huge, caches regenerate automatically, and .env holds credentials you must never commit. The rule of thumb is that anything reproducible from source or specific to one machine stays out; only source and pinned manifests like requirements.txt go in. Committing a venv or a .env is a classic first-project mistake — it bloats the repo and can leak secrets into git history, which is painful to purge. A good .gitignore on commit one prevents both.

Since PEP 668, distributions like Ubuntu 24.04 mark the system Python as externally managed — its packages are owned by apt, so pip installing into it can break OS tools that depend on specific versions. Rather than fight apt, you isolate: create a per-project virtual environment with python3 -m venv .venv, activate it, and pip install there. Inside the venv pip works normally and dependencies live beside your code, pinned in requirements.txt. The tempting shortcut, pip install --break-system-packages, does exactly what it says and is the wrong answer in an interview — it risks the base system. A venv (or pipx for CLI tools) is the correct, reproducible fix, and it's what your Dockerfile mirrors later.

Start by writing down the smallest thing that's still a real service — for linkstash, three endpoints and one table — and refuse features that don't serve the DevOps goal. A URL shortener is ideal: everyone understands it, it needs a database so Compose and volumes matter, and it has almost no business logic to distract you. I stub the endpoints first and wire the real logic last, so the pipeline exists before the code is 'done'. Scoping like this is a real skill: on the job you constantly trade feature scope against delivery, and a tight scope is what actually lets you ship.

A multi-stage build uses more than one FROM in the same Dockerfile. Each FROM starts a new stage; earlier stages are throwaway build environments, and the final stage is the image you ship. You compile or install in a heavy builder stage — with compilers, headers, dev tools — then COPY --from=builder just the finished artifact into a clean, minimal final stage. The compilers and caches never enter the shipped layers. The payoff is a smaller image (faster pulls, less storage) and a smaller attack surface, since a build toolchain an intruder could abuse simply isn't present. For linkstash it takes the image from roughly 480 MB down to about 215 MB.

By default a container's process runs as root inside the container, and that root can reach real privileges on the host through shared kernel features or mounted volumes. If an attacker exploits the app, running as root hands them far more to work with — installing packages, writing anywhere, escalating. Running as an unprivileged user contains the blast radius. In the Dockerfile you create a user (useradd --create-home --uid 1000 app) and add USER app before the CMD, so every process the container starts runs as that user. It's a one-line, near-zero-cost hardening step, and many security scanners flag any image that still runs as root.

psycopg2 is a C extension, so pip compiles it during install. That needs a C compiler (build-essential) and the Postgres client development headers (libpq-dev) — together hundreds of megabytes. Once compiled, the extension only links against the runtime shared library, libpq5, which is a few megabytes. The dev headers and compiler are build-time only. So the builder gets the full toolchain and the slim final stage installs just libpq5. Forget libpq5 in the final stage and the container crashes on boot with 'libpq.so.5: cannot open shared object file'. It is the split between what you need to build versus what you need to run.

Running pip install in the final stage would drag the whole build path back in — pip's download cache, and for anything with a C extension the compiler and dev headers again — defeating the point of multi-stage. By installing into a self-contained venv at /opt/venv in the builder and copying only that directory, the final stage gets the exact resolved dependencies with none of the machinery that produced them. A venv is relocatable as long as the Python version and base image match, which they do here (both python:3.12-slim). You put /opt/venv/bin on PATH and uvicorn and every dependency are there. It's the cleanest way to hand a finished dependency set between stages.

Plain depends_on only controls start order — it waits for the dependency's container to start, not for the service inside to be ready to serve. Postgres takes a moment to initialise, so a web service with only depends_on: [db] often races ahead and crashes with 'connection refused'. condition: service_healthy makes Compose hold the dependent service until the dependency's healthcheck reports healthy. You pair it with a healthcheck on the db — pg_isready on an interval. Now web isn't started until Postgres genuinely accepts connections. The distinction is readiness versus mere existence: the container being up is not the same as the database being ready, and only the healthcheck closes that gap.

Compose puts every service on a shared user-defined network and registers each one in an internal DNS under its service name. So from the web container, Postgres is reachable at the host db — the service's name — on its container port 5432. You do not use localhost: inside a container localhost is that container itself, so localhost:5432 would look for Postgres in the web container and fail. You also don't need to publish 5432 to the host for this; the ports key only maps host to container for outside access, whereas service-to-service traffic stays on the Compose network. That's why the app's DATABASE_URL points at db, not localhost or an IP.

A named volume is Docker-managed storage that lives independently of any container, so data outlives the container's lifecycle. I mount pgdata at Postgres's data directory (/var/lib/postgresql/data) so the database files persist across restarts and rebuilds — containers are disposable, the data isn't. docker compose down stops and removes the containers and the network but leaves named volumes intact, so my shortened links are still there next time. docker compose down -v additionally deletes those volumes, wiping the data — handy to reset a dev database, dangerous anywhere you care about the data. Bind mounts are the alternative when you want the files on the host filesystem.

A healthcheck is a command Compose runs inside the container on an interval; its exit code sets the container's health status — 0 is healthy, non-zero is unhealthy after the configured retries. For Postgres the right probe is pg_isready, which the official image ships: it checks that the server is accepting connections, not just that the process exists. I give it interval, timeout and retries so a slow first boot doesn't flap. The key is testing real readiness — 'can I actually connect?' — rather than something superficial like the port merely being open. Other services get their own probe: a web app is usually a curl to a /healthz endpoint that returns 200.

Because they run on different triggers and carry different permissions. My test job — ruff and pytest — runs on every push and pull request, because I want that feedback constantly and it needs nothing but read access. My build-and-push job runs only on a version tag and only after test passes, using needs: test. Publishing an image is a release action, not something I want for every work-in-progress commit. Splitting also lets me grant packages: write to just the build job, so the test job stays least-privilege. One workflow file, two jobs, a gate between them: fast feedback on everything, a published image only when I decide to release.

A service container is an extra container GitHub starts alongside your job for the life of that job — you declare it under services: with an image like postgres:16. The runner starts it, waits for its healthcheck, and exposes its ports on localhost. I use one so my tests hit a real PostgreSQL 16, the same engine and version my app runs in production, instead of a mock or SQLite that behaves differently. My tests connect with the same DATABASE_URL shape as compose; only the host changes from db to localhost. It's a few lines of YAML instead of a docker run script, and GitHub manages the container's lifecycle and health for me.

Two things gate it. First the trigger: I add tags: ['v*'] to the push event and put if: startsWith(github.ref, 'refs/tags/') on the build-and-push job, so it only runs when I push a version tag like v1.0.0. Second the dependency: needs: test means it won't even start unless the test job passed. So a normal branch push runs the tests and stops; a tag push runs the tests and, if they're green, builds and pushes the image. Releasing becomes a deliberate act — git tag, git push origin the tag — rather than a side effect of every commit to main.

Least privilege. GITHUB_TOKEN is minted per run, and by default it can read the repo but not publish packages. Only my build-and-push job actually pushes to GHCR, so I put the permissions block — contents: read, packages: write — on that job alone. The test job never gets package-write, so if a test step or a dependency it pulls in were compromised, it still couldn't publish an image. Granting the permission at the workflow level would hand write access to every job that doesn't need it. Scoping it to the one job that pushes keeps the blast radius as small as the task requires.

A lightweight tag is just a movable name pointing at a commit — no extra data. An annotated tag is a full git object: it stores who cut it, when, a message, and it can be GPG-signed. For releases you always use annotated (git tag -a), because GitHub Releases, gh release create, git describe, and tag-triggered CI all read that metadata. I cut v1.0.0 as annotated with a one-line message summarising the release. You can tell them apart with git cat-file -t v1.0.0: an annotated tag reports 'tag', a lightweight one reports 'commit'. Using a lightweight tag for a release loses the tagger, date and message you'd want for an audit trail.

Semantic versioning is MAJOR.MINOR.PATCH. 1.0.0 signals the first stable, public release — the API is now a promise, not a moving target. From there the rules are mechanical: bump PATCH (1.0.1) for backward-compatible bug fixes, MINOR (1.1.0) for backward-compatible new features, and MAJOR (2.0.0) only for a breaking change that forces consumers to update their code. So the number itself is a compatibility contract a reader can trust at a glance. Before 1.0.0 (the 0.x range) anything can change; cutting 1.0.0 is the moment you commit to that contract, which is exactly what a capstone release should do.

Trigger the workflow on tag pushes, not just branch pushes: on.push.tags with a glob like ['v*.*.*'] so only semver tags fire it. Then derive the image tag from the ref — docker/metadata-action with a semver pattern turns the git tag v1.0.0 into image tags v1.0.0, 1.0 and latest — and pass those to docker/build-push-action. The one gotcha is that git push does not send tags by default: you push the tag explicitly with git push origin v1.0.0 (or git push --follow-tags). That keeps everyday commits building the SHA image while a deliberate tag is what stamps a versioned, releasable image in the registry.

A CHANGELOG is the human-readable history of a project — what changed in each release and whether a consumer needs to care — which raw git log can't give you because commit messages are for authors, not users. Keep a Changelog is the widely-adopted convention: one section per version, newest on top, entries grouped under headings like Added, Changed, Fixed, Removed and Deprecated, with the version and date in the heading. A reader skims it in ten seconds to decide whether to upgrade. It also feeds release notes directly — I pass CHANGELOG.md to gh release create so the GitHub Release and the file never drift apart.

A Region is a geographic area — us-east-1 in Northern Virginia, ap-south-1 in Mumbai — and it is where you choose to run for latency, data-residency law, and price. An Availability Zone is one or more discrete data centres inside that region, with its own power, cooling and networking, sited far enough from its siblings that one fire or flood hits only that zone, but linked to them by fast private fibre. So the region is the country you pick; the AZs are separate buildings within it. The practical upshot I always add: I deploy across at least two AZs in a region, because a single AZ can fail and I don't want my app to fail with it.

It splits security between the provider and you. AWS is responsible for security OF the cloud — the physical data centres, the hardware, the hypervisor, the backbone network. I am responsible for security IN the cloud — patching my OS, my firewall and security-group rules, IAM users and permissions, and my data and its encryption. The line moves by service: on a raw EC2 instance I patch the OS; on a managed service like S3 or RDS AWS handles much more of the stack. The point I stress in an interview is that almost every headline breach is on the customer's side of that line — a public bucket or a leaked key — not the provider's buildings.

Because an AZ is a real physical failure boundary — one set of buildings with its own power and cooling that can go down on its own. If my whole app lives in a single AZ, a power cut or fire in that one location takes my service offline, however healthy my code is. Spreading instances across two or three AZs in the region means a zone can die and the load balancer just routes to the survivors, so users barely notice. It costs a little more and adds a load balancer, but multi-AZ is the baseline for anything I'd call production. Single-AZ is fine only for throwaway or dev workloads I can afford to lose.

Three factors. First, latency: I pick the region closest to my users, so a mostly-Indian audience goes in ap-south-1 Mumbai, not us-east-1. Second, data residency and compliance: if a law says customer data must stay in-country, the region choice is made for me regardless of latency. Third, cost: the same instance is priced differently by region, and not every service launches everywhere at once, so I check the service is actually available where I want it. For a global audience I'll run in several regions behind a global endpoint, but I still start by naming the one region that best fits latency, law and price.

They are three separate things people mix up. A user is a long-lived identity for a person or a script, with its own password and access keys. A role is an identity with permissions but no permanent credentials — it is assumed temporarily by a user, an EC2 instance or an AWS service, which hands back short-lived tokens that expire. Roles are how you avoid pasting long-lived keys onto servers. A policy is the JSON document that lists which actions on which resources are allowed or denied. Policies attach to users and roles; a policy on its own does nothing. Users and roles are who; a policy is what they may do.

The root user is tied to the account's sign-up email and can do literally anything — close the account, change billing, delete every resource — and you cannot scope it down. If those credentials leak, the whole account is gone. So the hygiene move is: give root a strong unique password, turn on MFA, and then stop using it. For everything day to day you sign in as an IAM user or an assumed role that only has the permissions that task needs. AWS's own guidance reserves root for the handful of tasks that genuinely require it, like changing the account's support plan. Everything else goes through IAM.

Least privilege means granting the smallest set of permissions a task actually needs, then widening only when something legitimately fails. In practice I start from a deny-by-default position — an IAM user or role with nothing — and attach a narrow policy, ideally an AWS-managed one like AmazonS3ReadOnlyAccess, or a custom policy scoped to specific actions and resource ARNs. I never hand out AdministratorAccess to a service or a script. For workloads I prefer roles over long-lived user keys, so credentials are short-lived and rotate automatically. The Capital One breach is the cautionary tale: an over-permissioned role let an attacker read data it never needed. Scope tight and a leaked credential leaks less.

Two free habits catch it early. First, AWS Budgets: I create a budget with a dollar limit and an email alert that fires the moment actual or forecast spend crosses a threshold — say 80 percent of $5 — so I hear about a runaway resource in hours, not on the invoice. Second, cost allocation tags: key/value labels like team=platform or env=dev on every billable resource, activated in the Billing console, so Cost Explorer breaks the bill down by team or project instead of one opaque number. Budgets are the smoke alarm; tags are the itemised receipt. Together they turn a scary end-of-month surprise into something you saw coming.

A security group is a stateful, allow-only virtual firewall attached to an instance's network interface. You write rules for traffic to permit — there are no deny rules — and because it's stateful, the return traffic for an allowed connection is let back out automatically. A network ACL is different: it sits at the subnet boundary, is stateless (you must allow both directions), and supports explicit allow and deny rules evaluated in order. The rule I use: security groups are the per-instance guest list I reach for first; NACLs are a coarse, subnet-wide backstop, often used to block a bad IP range across everything at once. Most day-to-day access control lives in security groups.

0.0.0.0/0 means the entire internet, so opening SSH that wide invites automated brute-force bots that scan public IPs constantly — a brand-new instance can be under attack within minutes. Instead I allow port 22 from my own IP as a /32, so only my address can reach it. Even better in real environments: don't expose 22 at all — use AWS Systems Manager Session Manager, which gives a shell through the AWS API with no inbound port open, or put instances in a private subnet behind a bastion or VPN. The principle is any firewall's: open the narrowest source range that still lets you do your job.

Stopping is like shutting the machine down: the instance halts, you stop paying for compute, but its EBS root volume persists, so the data survives and you can start it again later — though it usually gets a new public IP. Terminating deletes the instance for good; by default the root volume is deleted with it, the instance ID is gone, and you can't bring it back. I stop an instance I'll want tomorrow and terminate a throwaway I'm done with. The gotcha: a stopped instance still bills for its EBS storage and any Elastic IP, so 'stopped' isn't 'free' — only terminating and cleaning up the volumes truly stops the charges.

I SSH in with the key pair chosen at launch: ssh -i key.pem ubuntu@<public-ip> — the user is 'ubuntu' for Ubuntu AMIs, 'ec2-user' for Amazon Linux. If it times out rather than being refused, I treat it as a network problem, not authentication. First I check the security group actually allows port 22 from my current IP — home IPs change, so an old /32 rule may no longer match me. Then I confirm the instance has a public IP and sits in a public subnet with a route to an internet gateway. 'Permission denied (publickey)' is a different problem — wrong username or key. Long-term I'd prefer Session Manager over an open port 22.

I open Cost Explorer and group spend by service to see which line jumped — a tripled bill is almost always one service, usually EC2 compute. Then I group that service by usage type or by cost-allocation tag to pin the exact resource, and I check the EC2 console across regions, because the culprit is often a big instance running in a region I don't normally use. Once I've named it — frequently straight off its owner tag — I confirm it's safe to kill, terminate it, and watch daily spend flatten in Cost Explorer over the next day. Then I add a budget alert so the same thing can't happen silently again.

I don't rely on people remembering to check. I set AWS Budgets with alerts at 50%, 80% and 100% of the expected monthly spend, wired to email or Slack, so a runaway resource pages someone within a day instead of ambushing us at month end. I enforce cost-allocation tags — every resource carries an owner and a project — so Cost Explorer tells me not just what cost money but whose it was. And I keep least-privilege IAM so not everyone can launch a huge GPU box in the first place. Budgets catch it, tags attribute it, IAM limits the blast radius.

Stopping shuts down the OS but keeps the EBS volume, so you stop paying for compute while still paying a little for storage, and you can start it again later with its data intact. Terminating deletes the instance for good — and, by default, its root EBS volume — so it's gone and you stop paying for it entirely. The rule I use: stop when I'll want the box back soon, terminate when I'm done with it. For a forgotten resource burning money, terminate is the real fix — a stopped instance you keep 'just in case' is how the next surprise bill starts.

AWS is responsible for the reliability and pricing of the service; you're responsible for what you turn on and forget. That's the cost side of the shared responsibility model. On a client engagement I make it explicit: I set the account up with budgets, mandatory tags and least-privilege IAM before anyone launches anything, so cost ownership is designed in, not bolted on after the first shock. When a bill spikes, the answer is never 'AWS overcharged us' — it's a resource someone left running. My job is to make the running resources visible and attributable, so the person who launched it owns it.

A VPC, a Virtual Private Cloud, is your own isolated slice of the AWS network — a software-defined network where your instances, databases and load balancers live, walled off from every other customer. Everything you launch runs inside a VPC; nothing sits directly on the internet. The point is control and isolation: you decide which parts of the network can reach out, be reached, or only talk to each other. A database in a private subnet with no route out simply can't be reached from the internet, no matter who scans it. Running everything on public IPs would mean every resource is exposed by default — the VPC lets you expose only what you choose.

Its route table — nothing else. There's no 'public' checkbox on a subnet. A subnet is public when its associated route table has a route sending 0.0.0.0/0 to an internet gateway; that's the door out to the internet. A private subnet's route table has no such route, so its traffic never leaves the VPC. The exact same subnet becomes public or private purely by which route table you attach and what's in it. In practice you put web servers and load balancers in public subnets, and databases and internal services in private ones — then control the reachable path with route tables and security groups.

A CIDR block is an IP range written as an address plus a prefix length — 10.0.0.0/16 means the first 16 bits are fixed, leaving 65,536 addresses to use. For a VPC you choose a private range from RFC 1918 — 10.x, 172.16–31.x, or 192.168.x — so it never clashes with public internet addresses. AWS allows VPC blocks between /16 and /28. I pick something big enough to subdivide into subnets across availability zones with room to grow, and I make sure it doesn't overlap with other VPCs or on-prem networks I might peer or VPN to later — overlapping CIDRs are the classic thing that blocks a peering connection.

Three things all have to be true, so I check each. First, the route table on that subnet needs a 0.0.0.0/0 route pointing at an internet gateway that's actually attached to the VPC — no route, no exit. Second, the instance needs a public IP; a box in a public subnet without one still can't reach out or be reached. Third, its security group and the subnet's network ACL must allow the traffic. I work outward: local routing, then the gateway route, then the public IP, then the security group. If it's a timeout rather than a refusal, it's almost always the route table or a missing public IP, not the app.

The only technical difference is the route table. A public subnet has a route sending 0.0.0.0/0 to an internet gateway, so an instance there with a public IP can both reach the internet and be reached from it. A private subnet has no route to the internet gateway, so nothing on the internet can start a connection to it, and it can't reach out directly either. You put anything that shouldn't be internet-facing — databases, app servers, internal services — in private subnets, and only load balancers, bastions and NAT gateways in the public ones. Same subnet mechanics; the route table is what makes one public and one private.

A NAT gateway lets instances in a private subnet make outbound connections to the internet — apt updates, pulling images, calling APIs — without letting the internet start a connection back. It sits in a public subnet, and the private subnet's route table sends 0.0.0.0/0 to it; it translates outbound traffic to its own public IP and lets the replies back, but it's strictly one-way. It costs money because, unlike a security group or a route table, it's a managed, always-on appliance: AWS bills it per hour it exists plus per gigabyte processed, with no free tier — roughly $32 a month even idle. A forgotten NAT gateway is a classic silent VPC charge.

A bastion, or jump host, is a small hardened instance in a public subnet that you SSH into first, then hop from it to instances in private subnets over the VPC's internal network. The point is to shrink the attack surface: instead of exposing every instance to the internet, only the bastion is reachable, so it's the one box you harden and monitor. You lock its security group to your own IP on port 22, and configure the private instances to accept SSH only from the bastion's security group. Many teams now replace the bastion with AWS Systems Manager Session Manager, which gives an API-driven shell with no open inbound port at all.

You can't reach it directly — there's no internet route into a private subnet — so you go through the bastion. I SSH to the bastion's public IP, then on to the private instance's private IP. In practice I don't copy my key onto the bastion; I use ProxyJump: ssh -J ubuntu@bastion ubuntu@10.0.2.10, which tunnels through the bastion in one command and keeps the key on my laptop (agent forwarding with -A also works). The private instance's security group must allow port 22 from the bastion's security group. If the hop fails with 'Permission denied (publickey)' the key isn't being forwarded; if it times out, the security group isn't allowing the bastion.

S3 is Amazon's object storage: you store whole files as objects under a key inside a bucket, and read or write them over HTTP. It differs from a file system in that the namespace is flat — the slashes in a key are just naming, there are no real folders — and you can't edit part of an object in place; you replace the whole object. It differs from block storage like EBS, which hands one instance a raw disk to format and mount. S3 instead serves any number of clients over the network, scales effectively without limit, and is built for eleven nines of durability. I use it for backups, static assets, logs and data lakes.

Block Public Access is a master switch — at both the account and bucket level — that overrides any ACL or policy trying to make objects public. It's on by default on every new bucket, because misconfigured public buckets leaked data for years. A bucket policy is separate: a JSON document, same grammar as IAM, saying who may perform which s3: actions on which objects. The two interact strictly: even a policy granting s3:GetObject to Principal '*' does nothing while Block Public Access is on. To serve something public you deliberately turn the relevant switch off and attach the granting policy — two steps, so nothing becomes public by accident. For most workloads I leave it fully on.

I enable static website hosting on the bucket, set an index document like index.html and an error document, turn Block Public Access off, and attach a policy granting s3:GetObject to everyone — then the bucket serves my HTML, CSS and JS over a regional website endpoint with no server. The limits matter: the raw S3 website endpoint is HTTP only, has no custom-domain TLS, and no edge caching. So in production I put CloudFront in front for HTTPS, a custom domain via ACM, and caching, and I can then keep the bucket private and let CloudFront read it through an origin access control. S3 hosts the files; CloudFront makes it a real website.

Storage classes are S3's price-versus-access tiers. Standard is the default for hot data; Standard-IA and One Zone-IA are cheaper for infrequently accessed data but charge a retrieval fee; Glacier Instant, Flexible and Deep Archive are cheapest for cold archives, trading retrieval time for price. A lifecycle rule automates moving objects between them: I write a rule that, say, transitions objects to Standard-IA after 30 days, to Glacier after 90, and expires (deletes) them after a year — no cron job, S3 ages the data itself. It's the main lever for controlling storage cost on data whose access pattern cools over time, like logs or old backups. There's also Intelligent-Tiering, which moves objects automatically based on observed access.

RDS is a managed database service — AWS runs the engine (PostgreSQL, MySQL and others) and handles the undifferentiated heavy lifting: OS patching, engine upgrades, automated backups, failover and monitoring. On EC2 you install and run the database yourself, and every one of those tasks is on you. The trade is control versus effort: RDS gives up shell access to the host and some deep engine tuning in exchange for taking backups, patching and Multi-AZ failover off your plate. I reach for RDS by default for a normal relational workload, and only self-host on EC2 when I need an unsupported engine, a custom extension, or OS-level access RDS won't grant.

No — people conflate them constantly. Multi-AZ keeps a synchronous standby copy of your database in a second availability zone. You never read or write to the standby; it exists only for failover. If the primary's hardware or AZ fails, RDS flips the DNS endpoint to the standby, usually within a minute or two, with no data loss. A read replica is different: it's an asynchronous copy you can actually send read queries to, so it scales read traffic — but it can lag, and promoting it on failure is manual. Rule of thumb: Multi-AZ is for availability, read replicas are for read scaling. You can run both together.

Both are storage-level snapshots, but their lifecycle differs. Automated backups run daily in a window you set and, thanks to captured transaction logs, let you restore to any point in time within your retention period (1 to 35 days). Crucially, they're deleted when you delete the database instance. A manual snapshot is one you take yourself; it captures the DB at that moment and lives until you explicitly delete it — it survives the instance. So for anything I need to keep beyond the instance's life — before a risky migration, or as a long-term archive — I take a manual snapshot. That's also why deleting a test DB with 'skip final snapshot' avoids leaving snapshot storage behind.

Two layers. For network, I put RDS in private subnets with no public accessibility, and scope its security group to allow the database port (5432 for PostgreSQL) only from the application's security group or my own IP — never 0.0.0.0/0. A subnet group tells RDS which subnets it may live in, which is how I keep it off the public internet. For credentials, I never hard-code the password; I store it in AWS Secrets Manager (which can rotate it automatically) or SSM Parameter Store, and let the app fetch it at runtime via an IAM role. RDS also supports IAM database authentication, so short-lived tokens replace passwords entirely. Encrypt at rest with KMS, and enforce TLS in transit.

They're both AWS Elastic Load Balancers, but at different layers. An ALB works at layer 7 — it understands HTTP, so it can route by hostname, path or header, terminate TLS, and it's my default for web apps and microservices. An NLB works at layer 4 — raw TCP/UDP. It's faster, gives static IPs and an Elastic IP per AZ, and handles millions of connections at very low latency, so I use it for non-HTTP protocols, extreme throughput, or when a client needs a fixed IP to allowlist. Rule of thumb: HTTP app, reach for the ALB; raw TCP or a static IP requirement, reach for the NLB.

A health check is the load balancer repeatedly probing each target — usually an HTTP request to a path like /health — and only routing traffic to targets that respond correctly, say a 200 within the timeout. The target group defines the path, interval, timeout, and how many consecutive passes or fails flip the state. When a target starts failing, the load balancer marks it unhealthy and stops sending it requests, so users don't hit a broken box; when it recovers and passes again, it's put back into rotation. That automatic in-and-out is what lets a fleet survive one instance crashing without anyone getting paged.

A launch template is just the recipe for one instance — AMI, instance type, key pair, security group, user-data. On its own it launches nothing. An Auto Scaling Group uses that template to actually run and maintain a fleet: it keeps a desired count of instances alive between a min and a max, spreads them across Availability Zones, and replaces any that die or fail their health check. It also registers new instances into a target group so the load balancer sees them, and it runs scaling policies. So the template says what an instance looks like; the ASG decides how many exist, where, and reacts to failure and load.

Target tracking is a scaling policy where you name a metric and a target value — most often 'keep average CPU across the group at 50%' — and the ASG does the math to hold it there, adding instances when the metric runs hot and removing them when it cools. I prefer it to a fixed count because it reacts to real load automatically: I state the goal, not the instance number, so I don't over-provision for a peak that rarely comes or get caught short when traffic spikes. It's like a thermostat — I set 50%, and it adds or removes capacity to stay there. Cooldowns stop it flapping.

Both point one name at another, but an alias record is Route 53-specific and a CNAME is standard DNS. The practical difference: a CNAME cannot sit at the zone apex — the bare `example.com` — because DNS won't let a CNAME coexist with the mandatory SOA and NS records there. An alias record can, so it's how you point the naked domain at an ALB, CloudFront or S3. Alias records also resolve to the target's current IPs automatically, cost nothing to query, and Route 53 answers them internally. A CNAME works only on subdomains like `www`, points at any hostname anywhere, and is billed per query. Rule of thumb: alias for AWS targets and the apex, CNAME for external subdomains.

When you request an ACM certificate with DNS validation, ACM gives you a CNAME record to publish in your domain's DNS. Once ACM can resolve that record, it knows you control the domain and issues the cert. The reason it beats email validation is renewal: as long as that CNAME stays in place, ACM revalidates and renews the certificate automatically every year with zero human action — no expired-cert outages. Email validation, by contrast, sends an approval link to the domain's registered contacts and needs someone to click it, both at issue time and on renewal, which is fragile and easy to miss. For anything automated or long-lived, DNS validation into a Route 53 hosted zone is the set-and-forget choice.

ACM certificates are regional resources — a cert issued in `ap-south-1` can only be attached to load balancers and other resources in `ap-south-1`. So for an ALB you request the cert in the same region as the ALB. CloudFront is the exception everyone trips on: it's a global edge service, but it only reads certificates from `us-east-1`, so a cert for a CloudFront distribution must be requested there regardless of where your users or origin live. If you attach the wrong region's cert, the console simply won't list it. The habit: ALB cert in the ALB's region, CloudFront cert always in `us-east-1`.

The everyday records are A (name to IPv4), AAAA (to IPv6), CNAME (one name to another, subdomains only), MX (mail servers), TXT (SPF, DKIM and domain-verification strings), NS (delegation), and Route 53's own alias records for AWS targets. On top of record types, Route 53 has routing policies: simple for one target, weighted for splitting traffic by percentage (handy for canary releases), latency-based to send users to the closest region, failover paired with health checks for active-passive DR, and geolocation to route by country. Most zones start with plain A or alias and simple routing; you reach for weighted or latency policies once you run in more than one region.

There is no undo for a DROP, so I recover from a backup taken before it. RDS gives me two paths: restore from the newest automated or manual snapshot from before the incident, or use point-in-time recovery to a timestamp a second before the drop, which loses the least data. Both create a brand-new instance — RDS never restores in place, so the damaged database stays untouched while I work. I pick the recovery point closest to but before the drop, restore it, then verify the table is actually back before repointing the app. The discipline under pressure is: restore to a fresh instance, verify, then cut over — never restore in place and never skip the check.

Automated backups run daily inside a backup window and, combined with transaction logs, give you point-in-time recovery to any second within the retention period — up to 35 days. They're tied to the instance, so deleting it deletes them unless you choose to retain a final snapshot. Manual snapshots are ones you trigger yourself; they live until you explicitly delete them and survive instance deletion, which makes them ideal as a checkpoint before a risky change or for long-term keeping. The mental split: automated backups are the continuous safety net that powers PITR, and manual snapshots are the deliberate checkpoints you keep on purpose.

RPO, the recovery point objective, is how much data you can afford to lose — the gap between your last good backup and the incident. Daily 06:00 snapshots put your RPO in the hours; point-in-time recovery shrinks it to seconds. RTO, the recovery time objective, is how long recovery may take — and an RDS snapshot restore spins up a whole new instance, which can run from several minutes to an hour depending on size. So snapshots set your RPO floor and restore speed sets your RTO. When the business needs near-zero data loss and fast failover, that's when you reach for PITR plus Multi-AZ or a read replica, not just a nightly snapshot.

I never call it fixed just because the restore API returned available — that only means the instance booted, not that the data is right. I connect with a client and run a read-only query against the exact thing that was lost, here select count(*) from orders, and confirm the rows are really back. Then I check the app can actually reach the new endpoint — the restored instance has its own DNS name and security group — before I repoint production at it. The rule I hold under pressure: the fix isn't done until a query proves it. A green status from the restore is a promise; the row count is the proof.

A namespace is a container that groups related metrics so their names don't collide — AWS services use the AWS/… prefix, like AWS/EC2 or AWS/Lambda, and your own custom metrics go in a namespace you name. A dimension is a name/value pair that identifies which resource a metric belongs to: CPUUtilization in AWS/EC2 isn't one number, it's one per instance, and the dimension InstanceId=i-0abc… picks the instance you mean. To read or alarm on a metric you need all three coordinates — namespace, metric name, and the exact dimensions — because a different dimension set is a different metric entirely, which is the usual reason a new alarm sits in INSUFFICIENT_DATA.

A metric is a time series of numbers — one value per timestamp, cheap to store and fast to graph or alarm on, like CPU percent or request count. A log is text: the actual lines your app or system writes, which you search for the detail behind a metric spike. In CloudWatch Logs a log group is the named bucket for one app or resource — say /aws/ec2/my-app — and it holds retention and access settings. Inside a group, each source writes to its own log stream, usually one per instance or container. So the group is the logical application; the streams are the individual writers. Metrics tell you that something broke; logs tell you why.

An alarm watches one metric against a threshold over a number of periods and sits in one of three states: OK (inside the threshold), ALARM (breached for the configured periods), and INSUFFICIENT_DATA (not enough data points yet — common right after you create it, or when the dimensions don't match a live metric). The alarm itself only changes state; it doesn't email anyone. You attach an action, almost always an SNS topic ARN, to a state transition; when the alarm flips to ALARM, CloudWatch publishes to SNS, and SNS fans that out to every subscriber — your email, a Lambda, PagerDuty, or auto scaling. Metric to alarm to SNS to subscribers is the whole alerting chain.

The default AWS/EC2 metrics come from the hypervisor, which sees the instance from the outside — CPU, network, and volume-level disk I/O. It can't see inside the guest OS, so RAM used, swap, and disk-space-used simply aren't there. To get them you install the CloudWatch agent on the instance; it runs in the OS, reads memory and disk from the kernel, and pushes them as custom metrics (in a namespace like CWAgent) plus ships log files to a log group. That's also how you centralise /var/log/syslog and app logs. So 'no memory metric' isn't a bug — it's the boundary between what the hypervisor can measure and what needs an agent inside.

--query filters and reshapes a command's JSON response using JMESPath, a query language built into the CLI. So aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' returns just the instance IDs from a deeply nested response, no extra tool needed. The difference from jq is where the work happens and what you depend on: --query is native to the CLI, so it's always available on any box with the CLI installed and it runs before the data ever hits your shell. jq is a separate binary you pipe JSON into afterwards, and it's more powerful for complex transforms. My rule: --query for the everyday 'pull these fields' job, jq when I need real programming over the JSON.

A profile is a named set of credentials and settings in ~/.aws/credentials and ~/.aws/config. Instead of one identity, I keep dev, staging and prod as separate profiles and pick one per command with --profile prod, or for a whole shell with export AWS_PROFILE=prod. Each can point at a different account, region, or an assumed role. Why it matters: it's the guardrail that stops me running a destructive command against production when I meant staging — the classic career-ending mistake. In real teams profiles usually wrap short-lived credentials via IAM Identity Center or assume-role rather than long-lived keys, so switching accounts is a profile switch, not a re-login.

--output text prints bare values with no JSON quotes, brackets or commas, which is exactly what a shell wants. So IID=$(aws ec2 describe-instances --query 'Reservations[0].Instances[0].InstanceId' --output text) drops a clean i-0abc… straight into a variable, ready to pass to the next command. If I left the default json output, the variable would hold "i-0abc…" with the quotes baked in, and the next call would choke on them. json is for machines and jq, table is for reading with human eyes, and text is the scripting workhorse. Pairing --query to pick one field with --output text to strip the packaging is the core CLI-scripting move.

An idempotent script produces the same end state whether you run it once or ten times — no duplicates, no errors on a re-run. It's what lets you trust a script in automation. Two habits get me there. First, describe before you create: query for the resource and only create it if the query comes back empty, so a re-run doesn't make a second one or fail. Second, prefer operations that are naturally idempotent — tagging is the classic case, because create-tags with the same key just overwrites, so re-tagging is safe to repeat. Read-only describe calls are trivially idempotent, which is why they're the safe backbone of any reporting script.

ECR (Elastic Container Registry) is a private Docker registry that lives inside your AWS account. It does the same job as Docker Hub — you docker push images and docker pull them — but the repository is yours, in your region, and access is governed by IAM rather than a Docker Hub account. You authenticate with a short-lived token from aws ecr get-login-password piped into docker login. The reasons to use it over Docker Hub: images stay private by default, pulls from ECS or EKS in the same region are fast and free of egress, and you avoid Docker Hub's anonymous pull-rate limits. The trade-off is it's AWS-only, so a multi-cloud team might keep images somewhere neutral instead.

Both run your containers; the difference is who owns the servers. With the EC2 launch type you run a fleet of EC2 instances yourself — you size them, patch them, scale them, and bin-pack tasks onto them, paying for the instances whether or not tasks fill them. With Fargate there are no instances: you declare CPU and memory in the task definition, AWS finds capacity, and you pay per second only while the task runs. Fargate is simpler and has no idle servers to manage, so it's my default for spiky or low-volume work. EC2 can be cheaper at steady high scale, or when you need GPUs, custom kernels, or daemon-style access to the host.

A task definition is the recipe: which image, how much CPU and memory, which ports, environment variables, and IAM roles. It's versioned — every edit creates a new revision. A task is one running instance of that recipe: the actual container (or containers) up and running. A service keeps a desired number of tasks running: if a task dies, ECS launches a replacement to hold the count, and a service can register its tasks behind a load balancer so traffic spreads across them. The mental model I use: the task definition is the class, a task is an object, and the service is the supervisor that guarantees N objects always exist and are reachable.

I reach for Lambda when the work is short, event-driven, and spiky: responding to an S3 upload, an API Gateway request, a queue message. It scales to zero — you pay nothing when idle — and runs per-invocation up to 15 minutes, so bursty or infrequent workloads cost almost nothing. I choose Fargate when the work is long-running or steady: a web service that must stay up, anything over the 15-minute cap, or a process needing more memory, a full container image, and OS control. Amazon's Prime Video team moved a monitoring service off Step Functions and Lambda onto ECS for that reason — constant traffic made per-invocation overhead the wrong fit.

Prometheus pulls. On a fixed interval it scrapes an HTTP endpoint — usually /metrics — on each target and stores what it reads as time series. That's the opposite of push systems like StatsD, where the app sends metrics out. Pull has real advantages: Prometheus decides who to scrape from its own config or service discovery, so it always knows what should be up — a target that stops answering is itself a signal, the up metric goes to 0. There's no agent on every box preconfigured with a server address. The exception is short-lived batch jobs that die before a scrape; for those you push to a Pushgateway, which Prometheus then scrapes. But the default and the norm is pull.

rate() calculates the per-second average rate of increase of a counter over a time window — rate(http_requests_total[5m]) is requests per second, averaged across the last five minutes. You use it because the raw counter only ever climbs and resets to zero on restart; the number itself is meaningless, but its rate of change is the signal. rate() also corrects for those resets. The key rule: rate() is only for counters, never gauges. A gauge already goes up and down — memory in use, temperature — so its current value is what you graph directly, and running rate() on it gives nonsense. If in doubt, ask: does this metric only ever increase?

An exporter is a small process that translates some system's metrics into the Prometheus text format and serves them on a /metrics endpoint for Prometheus to scrape. It exists because most software doesn't speak Prometheus natively — the exporter bridges that gap. node_exporter is the canonical one: it runs on a Linux host and exposes machine-level metrics — CPU time per mode, memory, disk space and I/O, filesystem usage, network bytes, load average — hundreds of series describing the box itself. There are exporters for almost everything: the blackbox exporter probes endpoints from outside, and databases like Postgres and MySQL have their own. The rule of thumb: if a thing has an exporter, Prometheus can monitor it.

A counter only ever goes up — it counts occurrences of something, like total requests served or errors seen, and resets to zero only when the process restarts. You never read its raw value; you wrap it in rate() to see how fast it's climbing. A gauge goes both up and down — it's a snapshot of a value right now, like memory in use, queue depth, or temperature. For a gauge you graph the value directly, or take avg/max over time. Picking the right type matters because it decides how you query: rate() on counters, the value itself on gauges. Prometheus also has histograms and summaries for distributions like request latency, both built on counters.

A Prometheus alerting rule decides when something is wrong: it's a PromQL expression plus a for: duration, evaluated every scrape, and when it stays true for that window the alert fires. But Prometheus doesn't notify anyone — it pushes firing alerts to Alertmanager. Alertmanager decides who hears about it and how: it groups related alerts into one message so a rack failure isn't fifty pages, routes each alert to a receiver by matching labels, deduplicates alerts sent by replicated Prometheus servers, and lets you silence or inhibit during known work. So rules are detection; Alertmanager is notification and noise control. I keep the detection logic in the rule and the human policy — who, when, how loud — in Alertmanager.

An SLO is a reliability target — say 99.9% of requests succeed over 30 days. The error budget is the allowed failure, that 0.1%, roughly 43 minutes a month. It changes alerting in two ways. First, it gives a shared number product and engineering both trust: budget healthy, we ship features; budget nearly spent, we freeze and fix reliability. Second, it moves you from alerting on every blip to burn-rate alerting — you page only when you're consuming the budget fast enough to exhaust it soon. That kills the 2am page for one slow request while still catching real degradation early. Fewer alerts, and each one actually means something.

Elasticsearch indexes the full text of every log line, which makes arbitrary searches fast but is expensive in storage and compute. Loki takes the opposite bet: it indexes only a small set of labels — like container, level, namespace — and keeps the raw log compressed and unindexed. So it's much cheaper to run, and it reuses the exact label model Prometheus already uses, which is why it drops into Grafana right beside your metrics. The trade-off: you filter fast by label, then scan for text within that slice, rather than searching everything instantly. For cloud-native workloads where you already think in labels, that trade is usually worth it.

The alert fires in Prometheus when its rule expression stays true for the for: window. Prometheus sends it to Alertmanager, which runs it down the routing tree — label matchers read top to bottom. A severity="critical" alert might match a branch routing to PagerDuty; a warning falls through to a Slack receiver. Along the way Alertmanager groups alerts that share labels into one notification, applies any active silences, and dedupes if several Prometheus replicas sent the same alert. The receiver then does the delivery — PagerDuty, Slack, email, a webhook. So the labels you put on an alert are what actually decide who gets woken up, which is why label hygiene matters as much as the expression.

The load balancer is the only thing that should accept connections from the internet, so it goes in public subnets with a route to an internet gateway and a public IP. The app and database hold your logic and data — nothing on the internet should be able to open a socket to them, so they sit in private subnets with no inbound internet route. Traffic reaches the app only by passing through the ALB, where I can put TLS and a WAF. It's defence in depth: even if the app has a bug, the blast radius is one tier, because the database is unreachable except from the app tier.

An Availability Zone is an isolated datacentre with its own power and network. If I put everything in one AZ and that AZ has an outage — which does happen — linkstash goes down completely. Spreading the subnets across two AZs means the ALB can route to a healthy Fargate task in the surviving zone, and RDS can fail over to a standby in the other AZ. Two AZs is also the minimum an Application Load Balancer requires. It removes a whole class of single-datacentre failure while adding almost nothing to the bill — subnets and route tables are free. It's the cheapest meaningful step up in reliability.

Through a NAT gateway. A private subnet has no inbound internet route, but the Fargate task still needs outbound access — to pull its image and reach AWS APIs and CloudWatch. I put a NAT gateway in a public subnet and point the private route table at it for 0.0.0.0/0, so outbound connections work while nothing can initiate a connection inward. The cheaper alternative for AWS-only traffic is VPC interface endpoints for ECR and CloudWatch, which keep that traffic on the AWS network and let me drop the NAT gateway entirely — a real saving, since the NAT gateway is one of this design's biggest always-on charges.

The always-on ones bill by the hour regardless of traffic: the Application Load Balancer (roughly $0.0225/hour), the NAT gateway (about $0.045/hour plus per-GB), the RDS instance while it runs, and each public IPv4 address at $0.005/hour. Fargate bills per second only while a task runs, so scaling to zero tasks stops that charge — but the ALB and NAT gateway keep ticking. That's why an idle demo stack still costs real money, and why Day 4 of this project tears everything down — the ALB, the Fargate service, RDS, the NAT gateway and the Route 53 zone — rather than just stopping it. On AWS, 'stopped' rarely means 'free'.

The difference is one route. A public subnet's route table has a 0.0.0.0/0 route to an internet gateway, so resources in it can hold a public IP and be reached from the internet — that's where a load balancer goes. A private subnet has no such route, so nothing in it is directly reachable from outside; that's where I put app servers and the database. A private subnet can still reach out — to pull an image or hit an API — through a NAT gateway sitting in a public subnet, which allows outbound only. So 'public' and 'private' aren't a checkbox on the subnet; they're a property of the route table attached to it.

Both are IAM roles a Fargate task uses, but for different actors. The task execution role is assumed by the ECS agent — the platform — to pull the container image from ECR, fetch secrets, and write logs to CloudWatch; it carries AmazonECSTaskExecutionRolePolicy. The task role is assumed by my application code inside the container, for the AWS APIs the app itself calls — reading an S3 bucket, say, or Secrets Manager. Keeping them separate is least privilege: the plumbing that starts the container and my app's own permissions never share a credential. A common mistake is putting an app's S3 permission on the execution role — it works, but now the platform role is over-privileged.

A private subnet has no route to the internet gateway, so a task in it can't pull an image from ECR or reach a public API on its own. A NAT gateway, placed in a public subnet, gives it outbound-only internet: the task starts connections out, replies come back, but nothing outside can start a connection in. The catch is cost — a NAT gateway bills around $32–45/month plus per-GB data, running whether or not traffic flows. Cheaper alternatives: VPC endpoints (a gateway endpoint for S3, interface endpoints for ECR and CloudWatch) let tasks reach those AWS services privately with no NAT at all; or share a single NAT across AZs, trading some resilience for cost.

I chain them by referencing security groups, not IP ranges. The ALB's security group allows 80 and 443 from 0.0.0.0/0 — it's meant to be public. The app's security group allows the container port only from the ALB's security group, so nothing but the load balancer can reach the tasks. The database's security group allows 5432 only from the app's security group. Referencing a group instead of a CIDR means the rule follows the resource: as tasks scale and get new IPs, the rule still holds, because it names the group they belong to. The result is a one-way chain — internet → ALB → app → database — with no step opened wider than it needs.

Because only the load balancer needs a public face. The ALB sits in the public subnets and terminates internet traffic; the tasks sit in private subnets with no public IP, so nothing on the internet can reach a container directly — the only path in is through the ALB. That shrinks the attack surface to one well-understood front door I can protect with its security group, TLS, and a WAF. The tasks still reach out — to pull the image from ECR and talk to RDS — through the NAT gateway, so egress works without inbound exposure. It's defence in depth: a compromised dependency in the container still can't be hit from outside, only via the balancer.

A target group is the pool of backends an ALB forwards to, plus the health check that decides which are eligible. The listener says 'HTTP :80 → forward to this target group'; the group holds the targets and polls each one — for linkstash, GET /healthz — routing only to those returning 200. For Fargate in awsvpc mode every task gets its own elastic network interface and private IP, so the target type must be ip: the ECS service registers each task's IP as it starts and deregisters it as it stops. target-type instance is for EC2-backed targets and is rejected for awsvpc tasks. The health check is why the /healthz route from Project 1 finally matters.

I set it as an environment variable in the task definition — DATABASE_URL=postgresql://user:pass@<rds-endpoint>:5432/linkstash — reading the endpoint from RDS after it's created. That's fine for a lab, but a password sitting in the task definition is visible to anyone who can describe-task-definition and shows up in the console. In production I'd store the credential in AWS Secrets Manager (or SSM Parameter Store) and reference it through the task definition's secrets block, which injects it at runtime; the execution role gets read access to just that secret. The app code is identical — it still reads DATABASE_URL — but the value never lives in plaintext in the definition or logs.

Each tier should be reachable only from the tier in front of it, so I reference security groups as sources rather than IP ranges. The ALB SG allows 80/443 from the internet. The app SG allows the container port — 8000 — only from the ALB SG, so a task accepts traffic solely from the load balancer, never directly. The RDS SG allows 5432 only from the app SG, so the database accepts connections solely from the app. Referencing SGs, not CIDRs, means it keeps working as tasks come and go with new IPs. The result is least privilege at the network layer: skip the app→RDS rule and the container just times out connecting to Postgres.

An ALB has no fixed IP — AWS changes its addresses behind the DNS name — so you always point at the name, never an address. A plain CNAME would work, but Route 53 alias records are better in two concrete ways: they can sit at a zone apex (example.com, where CNAMEs are illegal), and alias queries are free rather than billed per lookup. An alias is a Route 53-only A record that targets an AWS resource by its hosted-zone ID and DNS name and follows it automatically. So for anything AWS-native — ALB, CloudFront, S3 website — I reach for an alias; a CNAME is what I'd use to point at a non-AWS host.

When you request an ACM certificate with DNS validation, ACM gives you a CNAME record to add to the domain's hosted zone. Once ACM sees that record resolve, it issues the cert and then auto-renews it as long as the record stays — no manual renewal, no expiry pages. You attach the cert to an HTTPS listener on port 443 that terminates TLS at the ALB. The port-80 listener I don't just leave forwarding plaintext: I change it to a 301 redirect to HTTPS, so a user typing http:// is bounced to the encrypted URL and no request ever completes in the clear. Terminate TLS at the edge, redirect everything else up to it.

An alarm watches one metric against a threshold over a period, for a number of evaluation periods, and flips between OK, ALARM, and INSUFFICIENT_DATA. For linkstash I set two. The first watches the ALB's HTTPCode_ELB_5XX_Count summed over five minutes — server errors the load balancer itself returns — because a spike there means the app is failing users. The second watches the ECS service's CPUUtilization averaged over five minutes, alarming above 80% for two periods, so I know when the container is running hot and needs more tasks. I also set treatMissingData to notBreaching on the 5xx alarm so quiet traffic doesn't false-alarm. Each alarm can notify an SNS topic that pages me.

Reverse of how you built it: edge first, then compute, then data, then network. So delete the Route 53 records, the HTTPS/HTTP listeners and the ACM cert; then the ECS service and cluster; then the RDS instance; then the ALB and target group; then the NAT gateway (and release its Elastic IP), the VPC endpoints, subnets, route tables, internet gateway and finally the VPC. The silent billers are the ones with no per-request cost so they're easy to forget: an idle ALB (~$16/mo), a NAT gateway (~$32/mo plus data), an RDS instance running 24/7, and any unreleased Elastic IP. Left together that is roughly ₹5,000+ a month for nothing.

Compose brings a stack up on one host and stops there — if that host reboots or a container dies, nothing restarts it, and you can't spread load across machines. Kubernetes adds four things compose can't: self-healing (it restarts pods and reschedules them off dead nodes automatically), scaling (change the replica count and it places the new copies for you), rolling updates with automatic rollback, and scheduling across many nodes. Underneath all of it is declarative desired-state: you say what you want running, and a control loop keeps reality matching it continuously. Compose describes a stack; Kubernetes keeps that stack alive across failures and traffic changes.

A Kubernetes cluster has two planes. The control plane is the brain: the API server you send every request to, etcd storing the cluster's desired state, the scheduler that decides which node each pod runs on, and controllers running the reconcile loops that keep actual state matching desired. The worker nodes are the muscle — each runs a kubelet that starts containers and reports health, plus your application pods. You talk only to the control plane, declaring what you want; the workers make it real. On kind both planes live in one Docker container by default, but the split matches a real multi-node cluster exactly.

Instead of running commands step by step — start this container, now this one — you hand Kubernetes a description of the end state you want: three replicas of this image, this much memory, this service exposed. Kubernetes stores that desired state and runs control loops that constantly compare it to what's actually running, then act to close any gap. If a pod dies and actual drops to two, the loop notices and starts a third. That's reconciliation, and it's why Kubernetes self-heals: nobody re-issues a command, the loop just keeps driving reality toward the declared state. It's the same declarative idea a compose file hints at, but enforced continuously rather than once at up time.

Often, honestly. Kubernetes earns its complexity when you have several services, need zero-downtime deploys, or must survive a node failing. For a single app on one VM, a side project, or a low-traffic internal tool it's overkill — docker compose or even a single container is simpler to run, cheaper, and far easier to debug at 3am. The cost of Kubernetes is real: more moving parts, a steeper learning curve, and more ways to break. So I reach for it when the workload genuinely needs self-healing or multi-node scale, not because it looks good on a résumé. Starting simpler and migrating later is usually the cheaper mistake.

Four pieces. The kube-apiserver is the front door — every request, from me, a controller or a node, is an HTTP call to it, and nothing talks to anything else directly. etcd is the cluster's memory: a consistent key-value store holding the whole desired and observed state, which is why losing it loses the cluster. The kube-scheduler decides which node a new Pod lands on, weighing free CPU, memory and constraints. The kube-controller-manager runs the reconcile loops — it compares what you asked for against what exists and closes the gap, like creating a replacement when a Pod dies. Nodes then run kubelet and kube-proxy, but those four are the brain.

kubectl sends the manifest as an HTTP request to the kube-apiserver, which authenticates me, validates the object, and writes the desired state into etcd — that's the whole of my part. From there controllers take over: the Deployment controller sees it wants three replicas and creates the Pods, the scheduler assigns each Pod to a node with room, and that node's kubelet tells the container runtime to pull the image and start the containers. Nothing ran the app because I said 'run' — I declared the state I wanted and the control plane converged on it. That declarative apply-and-reconcile loop is the core difference from typing docker run by hand.

Both run on every node but do different jobs. The kubelet is the workload agent: it takes the Pod specs the API server assigns to its node, tells the container runtime to start those containers, and continuously reports their health back up. If a container dies, the kubelet is what notices and restarts it per the spec. kube-proxy is the networking agent: it programs the node's iptables or IPVS rules so a Service's stable virtual IP load-balances to the right backend Pods, wherever they live. Rough split — kubelet makes containers run, kube-proxy makes Service traffic reach them. Neither makes scheduling decisions; that's the control plane's job.

A namespace is a virtual partition of one physical cluster — a scope for names and a boundary for quotas and access control. Two teams can each have a Pod called web in separate namespaces without colliding, and I can grant RBAC or set resource quotas per namespace. Kubernetes ships a few by default: default is where your objects go when you don't specify one, and kube-system holds the cluster's own machinery — CoreDNS, kube-proxy, the CNI, and on a real cluster the control-plane Pods. Keeping system workloads there separates them from application workloads, so kubectl get pods in default looks empty on a fresh cluster until you deploy something. I scope commands with -n.

A Pod is the smallest unit Kubernetes runs — one or more containers that share a network address and storage, scheduled together onto a node. It's disposable: if the node dies the Pod dies with it and nothing brings it back. A Deployment is a controller that manages Pods for you. You declare how many replicas you want and which image, and it creates a ReplicaSet that keeps exactly that many Pods running, replacing any that crash or vanish. So you almost never create a bare Pod in production — you create a Deployment and let it own the Pods. The Pod is the thing that runs; the Deployment is the thing that keeps it running.

A ReplicaSet's one job is to keep a set number of identical Pods running — it watches the cluster, counts Pods matching its label selector, and creates or deletes Pods until the count matches the desired replicas. You rarely create one directly, though. A Deployment sits on top and owns the ReplicaSet, and that layer is what gives you rolling updates and rollback: change the image and the Deployment spins up a new ReplicaSet, scaling the old one down gradually while keeping the revision history. Create a ReplicaSet alone and you lose all of that — you'd be editing Pods by hand. So the rule is simple: manage ReplicaSets through Deployments, never directly.

A rolling update replaces Pods gradually instead of all at once, so the app never goes fully down. When you change a Deployment's image it creates a new ReplicaSet and shifts Pods over a few at a time — bounded by maxSurge (how many extra it can add) and maxUnavailable (how many it can take down) — until the new version fully replaces the old. Each change is saved as a numbered revision. If the new image is broken, kubectl rollout undo flips back to the previous ReplicaSet, which is still there scaled to zero. I watch kubectl rollout status during a deploy and undo the moment it stalls, rather than waiting for it to time out.

Labels are key-value tags you attach to objects; a selector is a query that matches them. A Deployment uses them to know which Pods are 'its' Pods. Its spec.selector.matchLabels says, for example, app: web, and its Pod template stamps that same label onto every Pod it creates. The ReplicaSet then owns exactly the Pods whose labels match. The same mechanism is how a Service later finds Pods to send traffic to — it selects on labels too. One catch: the selector is immutable once set, and it must match the template's labels or the Deployment is rejected. So labels are the loose coupling that lets controllers and Services find Pods without hard-coding names.

Pods are disposable — Kubernetes recreates them constantly and each new pod gets a new IP, so you can't hard-code a pod's address. A Service is a stable virtual IP and DNS name that sits in front of a set of pods and load-balances across them. It uses a label selector to track which pods are alive right now, keeping a live list of endpoints. Your app talks to the Service name and never needs to know which pod answered, or that pods came and went. That's the whole point: a stable front door for an ever-changing set of backends. Without it, service-to-service calls would break every time a pod restarted.

They're three Service types, each building on the last. ClusterIP is the default: a virtual IP reachable only inside the cluster, ideal for pod-to-pod traffic like a web tier calling an API. NodePort does everything ClusterIP does and also opens the same high port on every node, so external traffic hitting a node's IP on that port is forwarded in. LoadBalancer does everything NodePort does and additionally provisions a real external load balancer from the cloud provider with a public IP. On a local cluster like kind there's no cloud, so a LoadBalancer service just sits pending. In production you rarely expose raw NodePort — you front services with an Ingress or a LoadBalancer.

Through cluster DNS. Every cluster runs CoreDNS, and every Service gets a DNS record of the form service.namespace.svc.cluster.local. A pod in the same namespace can use the short name; across namespaces you use the fully-qualified name. That name resolves to the Service's ClusterIP. From there kube-proxy — which programs iptables or IPVS rules on every node — rewrites the packet's destination to one of the healthy endpoint pods, load-balancing in the kernel. So the app just calls http://linkstash and never learns which pod answered. If name resolution fails, I check I'm using the right namespace or the FQDN, and that CoreDNS is healthy.

First I run kubectl get endpoints on the Service. If it shows none, the Service's selector doesn't match any pod's labels — the most common cause — so I compare it against kubectl get pods --show-labels. If endpoints exist but connections still fail, I check the Service's targetPort actually matches the port the container listens on. I confirm the pods are Ready, since only ready pods become endpoints. Then I test from inside the cluster with a throwaway pod, resolving the DNS name and curling the ClusterIP, to separate a DNS problem from a routing one. kube-proxy issues are rare, so I suspect labels and ports first.

Both are named key/value objects a pod reads as environment variables or mounted files, and both keep config out of the image. The difference is intent and handling. A ConfigMap holds non-sensitive settings — a log level, a feature flag, a public URL. A Secret holds sensitive values like passwords, tokens and connection strings; kubectl hides its values by default, it can be encrypted at rest on etcd, and RBAC is usually tighter on it. Functionally the wiring is almost identical — configMapKeyRef versus secretKeyRef — so the split is really about who may read it and how the platform protects it, not about a different mechanism.

Not by default — that's the single most common misconception. A Secret's values are base64-encoded, which is reversible packaging, not security: anyone who can run kubectl get secret -o yaml can pipe the value through base64 -d and read it in one line. To make Secrets actually secret you do three things: enable encryption at rest so etcd stores them encrypted, lock down RBAC so only the workloads and people who need them can get them, and keep them out of git — or use an external store like a cloud secrets manager or Sealed Secrets. Base64 only exists so binary values survive being carried inside YAML.

Two ways. As environment variables — pull a single key with valueFrom.configMapKeyRef, or inject every key at once with envFrom.configMapRef so each becomes an env var. Or as files — mount the ConfigMap as a volume and each key shows up as a file whose contents are the value. I reach for env vars for a handful of simple settings the app reads from the environment. I mount as a volume when the value is a whole config file — an nginx.conf, a application.yaml — that the app expects to read from a path. A bonus of the volume form is that updates to the ConfigMap propagate to the mounted files without a redeploy, which env vars don't do.

I'd put it in a Secret, not a ConfigMap, because a connection string carries a password. I create the Secret with the DATABASE_URL key, then reference it from the Deployment with env.valueFrom.secretKeyRef so the container gets DATABASE_URL in its environment at start — the app reads it exactly as it would locally. I never hard-code it in the image or the manifest checked into git. In production I'd back that Secret with encryption at rest and RBAC, or sync it from a real secrets manager, so the manifest references a secret by name without the value ever living in the repo.

A liveness probe answers 'is this container still working?' If it fails repeatedly, the kubelet restarts the container — it's the fix for a process that has hung or deadlocked. A readiness probe answers 'can this Pod take traffic right now?' If it fails, Kubernetes removes the Pod from its Service's endpoints so no requests are routed to it, but it is not restarted — it stays up, just out of rotation until it recovers. The classic mistake is using a liveness probe where you meant readiness: a slow dependency makes liveness fail, Kubernetes restarts a perfectly healthy container, and you get a restart loop instead of gracefully draining traffic.

A startup probe protects a container that takes a while to boot — a JVM app or one that runs migrations on start. Until the startup probe passes, the kubelet holds off both the liveness and readiness probes, so a slow boot can't be mistaken for a hang and get the container killed. You could instead give the liveness probe a large initialDelaySeconds, but that's a blunt trade-off: the delay applies for the whole life of the container, so after startup a real hang takes that same long delay to be caught. A startup probe lets you be patient during boot and aggressive afterwards — slow to give up at first, quick to restart once running.

A request is what the scheduler uses to place the Pod: it reserves that much CPU and memory on a node, and the Pod won't be scheduled unless a node has it free. A limit is the hard ceiling enforced at runtime. They behave differently per resource. If a container exceeds its memory limit it's OOMKilled — memory can't be reclaimed, so the kernel terminates it, and you see exit code 137. If it exceeds its CPU limit it isn't killed, just throttled to fewer cycles, so it runs slower. In short: requests are about scheduling and guarantees, limits are about capping, and memory limits bite by killing while CPU limits bite by slowing.

Kubernetes assigns every Pod one of three Quality of Service classes from its requests and limits. Guaranteed means every container sets requests equal to limits for both CPU and memory — these are evicted last. Burstable means at least one request or limit is set but they don't all match — the common real-world case. BestEffort means no requests or limits at all — first to be evicted. When a node runs low on memory the kubelet reclaims it by evicting Pods, worst QoS class first, so BestEffort Pods die before Burstable, and Guaranteed Pods are the last to go. That's why production workloads you care about should at least set requests, and critical ones set requests equal to limits.

CrashLoopBackOff means the container starts, exits, and restarts on a growing back-off — the image is fine, the process dies. I run the loop: kubectl get pods to confirm the status and rising RESTARTS, then kubectl describe pod to read the Events and the last exit code, then kubectl logs -p to see the dead container's final output. That's usually where the app prints why it died: a missing env var, a bad config path, or a dependency it can't reach. If the logs are empty I check the container's command and args, and exec into a debug container. The habit is reading the previous logs before guessing.

Two different voices. kubectl logs shows your application's own stdout and stderr — the app talking. kubectl describe shows Kubernetes' view of the object: its state, restart count, the image it's pulling, mounted config, and crucially the Events section — the control plane talking about your Pod. Rule of thumb: for anything that never started (ImagePullBackOff, Pending, CreateContainerConfigError) read describe's Events first, because the app never ran and has no logs. For anything that started and then died, read logs — with -p for the crashed instance — first. Events tell you the platform's problem; logs tell you the app's problem.

Pending means the scheduler hasn't placed it on any node, so nothing has started. I go straight to kubectl describe pod and read the Events — the scheduler writes a FailedScheduling event with the reason: Insufficient cpu or memory, an unbound PersistentVolumeClaim, or a taint the Pod doesn't tolerate. Then kubectl get nodes and kubectl describe node to compare the Pod's requests against the node's Allocatable capacity. Fixes depend on the reason: lower the requests, add capacity, fix a nodeSelector, or add a toleration. The key is that Pending is a scheduling problem, not an application one — logs won't help because the container never ran.

The -p / --previous flag reads the logs of the container's previous, terminated instance instead of the current one. It matters for CrashLoopBackOff: the running container may be empty, mid-boot, or already dead again by the time you look, but the previous instance printed exactly why it exited right before it died. So on a crash-looping pod I reach for logs -p to catch those last words. The catch is that it only works once the container has actually restarted — if RESTARTS is 0 there's no previous instance and kubectl returns a BadRequest. So I check RESTARTS first, then add -p.

Pending means the API server accepted the Pod but the scheduler hasn't placed it on a node yet. The classic cause is that the Pod's resource requests exceed what any node has free — the scheduler won't overcommit, so the Pod waits rather than starting where it can't fit. Other reasons: a node selector, taint or affinity rule that no node satisfies, or a PersistentVolumeClaim that hasn't bound. I run kubectl describe pod and read the Events at the bottom, where the scheduler writes the exact reason, like "0/3 nodes are available: Insufficient memory." Then I fix what it names — shrink the request, add capacity, or fix the claim — and it schedules.

It means the container starts, exits or crashes, and Kubernetes keeps restarting it — backing off longer between each attempt so it isn't hammering a broken app. The status isn't the bug; it's the symptom of a process that won't stay up. The cause is almost always in the app: a missing environment variable or Secret, a config file it can't find, a dependency it can't reach, or a liveness probe killing a healthy-but-slow start. I read kubectl logs, and kubectl logs --previous to see the crashed instance rather than the one starting now, then kubectl describe for the events and last exit code. The RESTARTS count climbing is the tell.

Deleting a Secret doesn't kill running Pods immediately, but the moment they restart or roll, the containers can't find it. If it's an env-var or secretKeyRef reference, new Pods hit CreateContainerConfigError before the app even runs; if the app reads the value at startup and it's gone, it crashes into CrashLoopBackOff. I confirm with kubectl get pods, then kubectl describe pod to see the event naming the missing object, e.g. secret "db-credentials" not found. The fix is to recreate the Secret with the same name and keys — kubectl create secret generic db-credentials — and the Deployment self-heals as its controller restarts the Pods against it. No app change, no redeploy.

I work top-down. kubectl get pods shows status and restart counts — the shape of the problem. kubectl describe pod adds the Events at the bottom, which name scheduling failures, image-pull errors and missing config in plain English. kubectl logs, with --previous for a crashed container, shows what the app itself printed before dying. kubectl get events --sort-by=.lastTimestamp gives the cluster-wide timeline when the problem spans objects. The discipline is the same as Docker debugging: read status, read events, read logs, in that order, instead of guessing. Most of the time the answer is already written in the describe output — you just have to read it.

port-forward is a debug tunnel — it serves one person from one terminal and dies when I close it. A NodePort opens one high port, 30000 and up, per Service on every node, so ten apps means ten odd ports, none on 80 or 443 and none with a hostname. Neither scales to real traffic. An Ingress gives me one entry point on 80/443 that routes by host and path to many Services: shop.example.com to one, /api to another, all behind a single address. I add apps without opening new public ports or handing users port numbers. It's the difference between a dozen side doors and one front desk.

The Ingress is just data — a Kubernetes object holding routing rules: this host and path go to that Service. On its own it does absolutely nothing; it's a routing table with nobody reading it. The Ingress controller is the Pod that makes it real — a reverse proxy like ingress-nginx that watches every Ingress object and reprograms its own config to actually forward traffic. So you need both: the resource declares intent, the controller enforces it. The classic beginner bug is creating an Ingress on a cluster with no controller installed — the ADDRESS stays empty and curl gets connection refused, because there's a rulebook but no one enforcing it.

pathType tells the controller how to match the request path against the rule. Prefix matches the path and everything beneath it — /app1 also matches /app1/health — which is what you want for a route fronting a whole app. Exact matches only that exact string, so /app1 matches but /app1/ and /app1/health don't. ImplementationSpecific hands the decision to the controller's own logic. I reach for Prefix almost always; Exact is for one specific endpoint. The bug I watch for is Exact on an app route — the root loads but every sub-page 404s, so it looks like the app is broken when it's really the path match being too strict.

One Ingress with two rules, behind one controller. If the apps share a hostname I split by path — /shop to the shop Service, /api to the api Service — each with pathType Prefix and the right backend service name and port. If they have their own names I split by host instead: shop.example.com and api.example.com as separate rules pointing at their Services. Either way it's a single entry point on 80/443, so I'm not handing users port numbers or opening a NodePort per app. In production I'd also terminate TLS at the Ingress with a tls block referencing a Secret, so both apps get HTTPS from that one place.

A chart is a versioned package of templated Kubernetes manifests plus a values.yaml of defaults. Instead of hand-maintaining a dozen YAML files across dev, staging and prod, you template the parts that change and pass a values file per environment. helm install renders the templates against your values and applies the result. The problem it solves is repetition and drift: one parameterised package replaces copy-pasted manifests, and the same chart at the same version deploys identically everywhere — only the values differ. It's the apt or npm of Kubernetes: someone publishes a chart, you install it by name and tune it with values rather than editing raw manifests.

helm install NAME CHART renders the chart and creates a release — a named, tracked instance recorded as revision 1. helm upgrade changes it: pass a newer chart or override values with --set or -f, and Helm renders again and applies the diff, bumping to revision 2. helm history shows every revision, and helm rollback NAME 1 returns to an earlier one — importantly it doesn't erase history, it creates a new revision that matches the old state. helm uninstall removes the release and its resources. Helm keeps this history in the cluster, as Secrets by default, which is how it knows what to diff and what to roll back to.

Templates in templates/ are Go text/template files with placeholders like {{ .Values.replicaCount }}. values.yaml supplies the defaults; --set and -f override them at install or upgrade time. Helm merges the values, renders every template into plain Kubernetes YAML, then sends that to the API server. You can see the rendered output without touching the cluster using helm template — it stops at the render step, which is ideal for reviewing a change or diffing it in CI. So the flow is always the same: templates plus merged values become rendered manifests, which get applied. Nothing magic reaches the cluster; it's ordinary YAML that Helm generated for you.

For anything with more than a couple of manifests that ships to multiple environments, Helm earns its keep: one chart, per-environment values, and a release history you can roll back. kubectl apply -f is fine for a single manifest or a quick experiment, but it has no notion of a release, no rollback, and no templating — you end up copy-pasting near-identical YAML. That said, Helm's templating can get gnarly, so some teams prefer Kustomize (overlays, no templating language) or a GitOps tool like Argo CD that syncs manifests from git. My rule: reach for Helm when packaging or consuming a reusable app; plain manifests or Kustomize when the config is small and static.

Imperative means I write the exact steps — install this, edit that, restart — and run them in order; the only record of the result is the box itself. Declarative means I describe the end state I want and let a tool work out the steps. Terraform is declarative: my .tf files say what should exist, and terraform plan computes the diff between that and reality. The big win is idempotence — run it twice and the second run is a no-op because state already matches. A shell script has no idea what it did last time, so re-running it can double-apply or fail halfway. Declarative code is also reviewable and diffable like any other code in the repo.

Four commands. terraform init prepares the directory — it reads which providers my config needs, downloads them, and writes a lock file so every machine uses the same versions. terraform plan is a dry run: it shows what it would create, change or destroy, marked with plus, tilde and minus, and touches nothing. terraform apply executes that plan after I confirm with 'yes' and makes reality match the config. terraform destroy tears the managed resources back down. In between, Terraform keeps state — a file mapping each resource in my config to the real object it made — so plan can tell 'create this' from 'already exists'. Init, plan, apply is the loop I run all day.

State is Terraform's record of what it manages — a JSON file, terraform.tfstate, mapping every resource in my config to the actual object it created, plus that object's current attributes. It's how plan knows the difference between something it must create and something that already exists, so it can compute a minimal diff. It matters because it's the tool's source of truth: lose it and Terraform forgets it ever built anything and tries to recreate everything; let it drift from reality and plans act on the wrong picture. On a team you never keep it on one laptop — it goes in a shared remote backend with locking so two applies can't corrupt it. That's tomorrow's topic.

They're the same tool, forked. Terraform is HashiCorp's, and in 2023 HashiCorp relicensed it from the open-source MPL to the Business Source License, which restricts some competing commercial uses. In response the community forked the last MPL version and created OpenTofu, now under the Linux Foundation and still MPL-2.0. For everything in this course the two are interchangeable — same HCL, same init-plan-apply loop, same providers from a compatible registry; you'd just type 'tofu' instead of 'terraform'. Which one a job uses is mostly a licensing and governance choice, not a day-to-day workflow difference. Knowing both exist, and why, is enough for an interview.

You declare inputs with variable blocks — each has a type and an optional default — and read them as var.name. You supply values three ways: a default in the block, a terraform.tfvars file Terraform auto-loads, or a -var (or -var-file) flag on the command line. When the same variable is set in more than one place the most specific wins: the default is lowest, then TF_VAR_ environment variables, then terraform.tfvars, and a CLI -var overrides them all. That precedence is the whole point — one config produces a dev file and a prod file by swapping tfvars or passing -var, with no change to the code.

State is Terraform's memory — a JSON file that maps each resource in your config to the real object it created, plus that object's current attributes. On the next apply Terraform reads state, compares it to your config and to reality, and plans only the difference. That's why it's the source of truth: delete it and Terraform forgets it owns anything and tries to recreate duplicates. You don't hand-edit it because one wrong field silently desyncs Terraform's picture from reality, and the next apply then acts on that bad picture — deleting or recreating live resources. If you must change state, use the terraform state subcommands like mv, rm and show, which edit it safely and keep it internally consistent.

An output block surfaces a value after apply — usually a computed attribute of a resource, like its id, an IP address, or a URL. Outputs serve three audiences: a human reading the apply summary, another module that consumes the value, and a script. terraform output prints them all; terraform output -raw NAME prints one value with no quotes or JSON wrapping, so you can pipe it straight into another command. Without outputs you'd dig through state to find what Terraform built. They're the clean, declared interface to the results — and because they can expose sensitive data, you mark those outputs sensitive = true so Terraform redacts them from the console.

A local terraform.tfstate is fine for one person, but it breaks the moment two engineers share infrastructure: each has a separate copy, both think they own the resources, and their applies silently overwrite each other. A remote backend fixes that — one shared copy of state everyone reads and writes, with state locking so only one apply runs at a time, plus version history you can roll back. HCP Terraform offers this as a managed service; a cloud object-store backend does it self-hosted. It also keeps the secrets baked into state off individual laptops. My rule of thumb: the moment a second person or a CI pipeline runs apply, state moves remote.

A module is a folder of Terraform files you call from another config with a module block. The folder doing the calling is the root module; the one it points at with source is a child. I pass inputs through its variable blocks and read results back from its output blocks — it's basically a function for infrastructure. I reach for one for reuse: instead of copy-pasting the same twenty lines to build a network three times, I write it once and call it three times with different inputs. That keeps things DRY, gives me one place to fix bugs, and lets me pull battle-tested modules off the Terraform Registry instead of writing them myself.

A workspace is a named instance of state for one configuration. Every config starts in the default workspace; terraform workspace new dev gives me a second, empty state, and terraform.workspace lets the config read the current name so I can stamp it into resource names. The catch — and interviewers love this — is that a workspace only forks the state file. It does not fork the backend or the provider credentials, so every workspace of a config still points at the same backend and the same cloud account. That makes workspaces great for cheap parallel copies like a per-branch test stack, but a poor tool for real staging-versus-prod isolation, where I'd use separate directories and backends instead.

Inputs are the module's variable blocks — the caller sets them in the module block, like name = "Ada". Outputs are the module's output blocks — the values it hands back, which the caller reads as module.<name>.<output>. So it's a clean contract: the caller only sees the inputs it's allowed to set and the outputs the module chose to expose; everything inside stays private. In my greeting module the input is the name to greet and the output is the file path it wrote, so the root config can print that path without knowing how the module built it. That encapsulation is what lets me swap a module's internals without touching any config that calls it.

Short answer: not on their own. Workspaces isolate state, not the things that actually keep prod safe — they share the same backend and the same provider credentials, so a careless apply in the wrong workspace can still hit the wrong account. They're perfect for ephemeral or parallel copies of the same thing: a demo, a per-pull-request stack, a quick experiment. For staging and production, which must never touch each other, I keep separate configurations with separate backends and separate credentials, usually in separate directories or repos. That way switching environments is a deliberate act, not a one-word terraform workspace select that's easy to forget. HashiCorp's own docs say the same thing.

Terraform authenticates through the AWS provider, which uses the same credential chain as the AWS CLI. In practice I run aws configure once so my key and secret sit in ~/.aws/credentials, and the provider picks them up automatically — I only set region in the provider block. It also reads the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION environment variables, which is how a CI job passes short-lived credentials. On an EC2 instance or in a pipeline I prefer assuming an IAM role over static keys, so nothing long-lived is stored at all. The one thing I never do is put the key in the .tf file — that is source code, and it gets committed to git.

Three commands, one loop. terraform init downloads the providers my config needs — here hashicorp/aws — and writes a lock file pinning their versions. terraform plan compares my HCL against the recorded state and the real AWS account, then prints exactly what it will add, change or destroy without touching anything. terraform apply executes that plan, makes the real AWS API calls, and records each resource's ID in state. The habit that keeps me safe is always reading the plan before I approve the apply: the 'Plan: 4 to add, 0 to change, 0 to destroy' line is the last chance to catch a mistake before it hits the account.

Because a .tf file is source code — it gets committed and pushed, so a hardcoded access key is now in a repo's history, often forever, even if you delete the line later. Leaked AWS keys get scraped from public GitHub within minutes and used to spin up crypto miners on your bill. So I keep credentials out of the code entirely: aws configure for local work, environment variables or an IAM role in CI. Secrets that genuinely must be in config, like a database password, go through variables marked sensitive and a real secret store, never a literal. And if a key ever does leak, I rotate it in IAM immediately rather than hoping nobody noticed.

terraform destroy reads the state file and deletes every resource Terraform created, in dependency order, after showing a plan you approve. It matters most on AWS because 'stopped' rarely means 'free' — a load balancer, a NAT gateway or a public IPv4 keeps billing by the hour whether or not anyone uses it. When a lab or a demo environment is defined in code, destroy is one command that guarantees I've left nothing quietly charging me, which is far safer than hunting resources in the console. In production I never destroy shared infra casually, but for ephemeral environments — a review app per pull request — build-then-destroy is the whole point of infrastructure as code.

Terraform keeps a state file mapping your config to the real resources it manages, and the whole team writes to that same shared state. Before any write — apply or destroy — it takes a lock so two runs can't change state at once; without it, one run could overwrite another's changes and state would stop matching reality. On a remote backend the lock is a small object or row the backend writes and clears when the run ends — the modern S3 backend uses a native lockfile, older setups a DynamoDB table. The danger is a run that dies mid-flight and never releases it, so every later apply is refused with 'Error acquiring the state lock' until someone clears it.

First confirm nobody is actually running Terraform — the lock might be legitimate. Read the lock info Terraform prints: the ID, who holds it, and when it was created. If it's stale — the holder's run died and won't return — release it with 'terraform force-unlock <id>' using the exact ID from the error, then re-run apply. Order matters: you can't apply through a held lock, so force-unlock comes first and apply second. Never force-unlock a lock a live run still holds; you'd let two writers corrupt the state. In production the real fix for repeat offences is running Terraform only from CI, never from a laptop.

Drift is when live infrastructure no longer matches your checked-in config — usually because someone changed a resource by hand in the cloud console. Terraform catches it at plan time: it refreshes state against the real world, compares that to your config, and shows the diff. If someone shrank an instance from m5.large to t3.micro in the console while the config still says m5.large, plan proposes an in-place update to put it back, and apply reconciles it — the config is the source of truth. The lesson is to never hand-edit live infra to fix something; change the config and apply, so config, state, and reality stay in agreement.

Shared state is the team's real infrastructure, and a laptop is the worst place to run apply against it. Your Wi-Fi can drop mid-run and leave a state lock abandoned, blocking everyone; you might be on a stale branch and apply changes that were never reviewed; and there's no audit trail of what ran. The fix is making a CI pipeline the only thing that runs apply — stable networking, the reviewed commit, a logged run, and the lock held for a predictable window. Laptops are for 'terraform plan' and reading state, never for writing it. That discipline is exactly what today's mission rehearses.

Cost and learning speed. A managed EKS control plane bills about $0.10/hour — roughly $73/month — before one worker node runs; a t3.small running k3s is a few dollars a month, near-free on the free tier. k3s is a CNCF-certified Kubernetes distribution, so kubectl, my manifests and Helm behave exactly as on any cluster — I learn the real API, not a toy. The honest tradeoff is that this is one node in one Availability Zone: no control-plane HA and no failover, the opposite of Project 2's two-AZ design. So k3s is right for learning, demos and edge; EKS is the answer once uptime justifies the bill.

Because the expensive mistakes on a cloud project are architectural, not typos, and they're cheapest to fix on paper. A planning day forces three decisions up front: what each resource is and why it exists, which ones bill by the hour, and the exact order to build them. I leave today with a resource inventory, a monthly cost estimate, and a build sequence — so the next four days are execution, not improvisation. It's the same discipline as Project 2: the day drawing the topology and pricing it is what separates a deploy that works from one that surprises you on the invoice or leaves resources billing after you think you're done.

The two that bill by the hour regardless of traffic are the EC2 instance — a t3.small at about $0.0208/hour, ~$15/month if left up — and the Elastic IP, a public IPv4 at $0.005/hour (~$3.65/month) that keeps charging even while the instance is stopped. The 8 GB gp3 root volume is under a dollar a month. Everything else in the design — the VPC, the subnet, the internet gateway, the route table and the security group — is free to exist. So an idle demo still costs the instance plus the IP, which is exactly why Day 85 ends with terraform destroy rather than just stopping the box.

Each owns a layer. Terraform (days 76-80) provisions the AWS foundation — the VPC, the EC2 instance and its Elastic IP — and installs k3s through the instance's user_data, so the whole environment is code I can rebuild or destroy with one command. Kubernetes (days 66-73) runs the workload: Deployments, a Service, a ConfigMap and a Secret describe linkstash and its in-cluster Postgres declaratively. Helm (days 74-75) packages those manifests into a versioned chart with values, and a Traefik Ingress plus cert-manager expose the app over HTTPS. Terraform builds the cluster, Kubernetes runs the app, Helm ships and exposes it — the three tools stacked exactly as a real team layers them.

Cost and simplicity. A managed EKS control plane is about $0.10/hour — roughly $73/month — before a single worker node runs; a t3.small running k3s is a few dollars a month, near-free on the free tier. k3s is a fully CNCF-certified Kubernetes distribution, so kubectl, my manifests and Helm all behave exactly as they would anywhere — I learn the real API, not a toy. The honest tradeoff is that this is one node in one AZ: no control-plane HA, no failover, the opposite of Project 2's two-AZ design. So single-node k3s is right for learning, demos and edge, and EKS is the answer once uptime justifies the bill.

k3s writes one to /etc/rancher/k3s/k3s.yaml on the server, readable only by root, with the API server set to https://127.0.0.1:6443. That's perfect on the node and useless from my laptop. So I SSH in, sudo cat the file, copy it down, and sed the 127.0.0.1 to the node's public address — here the Elastic IP. The catch is TLS: the API server certificate has to list that address as a Subject Alternative Name, or kubectl fails x509 validation. So I install k3s with --tls-san <Elastic IP> and the cert covers it. Then I export KUBECONFIG and kubectl get nodes talks to the cluster over the internet.

Different exposure needs per port. Port 22 is administrative — only I should reach it — so I scope inbound 22 to my /32, shrinking the brute-force and zero-day surface to almost nothing. Ports 80 and 443 are the app's front door: the whole point is that anyone can load linkstash, and Traefik on the node serves HTTP and HTTPS, so those stay open to 0.0.0.0/0. Port 6443, the Kubernetes API, I leave closed and reach the cluster over SSH instead, because exposing an API server to the internet is a real risk. It's least privilege applied per port, not one blanket rule for the whole box.

To break a dependency cycle. The instance's user_data installs k3s with --tls-san <Elastic IP>, so it needs the IP value at boot. But an EIP declared with an instance attribute would depend on the instance, and the instance depends on the EIP — Terraform can't resolve that loop. Allocating the EIP as a standalone resource gives me a stable public IP up front, which I interpolate into user_data, and a separate aws_eip_association attaches it once the instance exists. An Elastic IP also survives stop/start, so the address in my kubeconfig stays constant — which matters once day 84 points a domain at it.

For a single-node demo cluster, a Deployment with one replica plus a PersistentVolumeClaim is the simplest thing that keeps data across pod restarts — the pod is disposable, the PVC is not. In production I'd reach for a StatefulSet, which gives the database a stable network identity and stable per-replica storage and orders scaling safely. Better still, I'd take the database off the cluster entirely and use a managed service like Amazon RDS, so backups, failover and patching aren't my problem. Running stateful workloads on Kubernetes is doable but it's the hard mode; for a URL shortener's links table, managed Postgres is the boring, correct choice.

Both platforms give you service discovery by name — the name is just whatever you called the service. In compose.yaml the Postgres service was named db, so Compose's DNS resolved db to it. On Kubernetes I created a Service object named postgres, so in-cluster DNS resolves postgres.default.svc.cluster.local — usually just postgres — to that Service's ClusterIP, which forwards to the pod. So the only real change to the connection string from Project 1 is the host: db becomes postgres. Point it at localhost and the app talks to its own pod, not the database; point it at the old db and DNS returns nothing. The host must match the Service name.

No — that's expected on k3s. The built-in local-path StorageClass uses volumeBindingMode WaitForFirstConsumer, which deliberately delays binding until a pod that mounts the claim is scheduled. Only then does the provisioner know which node to carve the directory on, so it waits. The PVC flips from Pending to Bound the moment the Postgres pod is scheduled. It's only a real problem if it stays Pending after the pod is running — then I'd kubectl describe pvc and kubectl describe pod to see whether the pod can't schedule at all, or whether no default StorageClass is set. On a fresh k3s node, Pending-until-scheduled is the normal, healthy sequence.

The URL lives in a Secret named linkstash-db, and the Deployment injects it with env.valueFrom.secretKeyRef, so the container starts with DATABASE_URL in its environment — exactly how it read it locally. The image stays generic and the same v1.0.0 tag runs in any environment; only the Secret changes. I keep it out of the ConfigMap because a ConfigMap is for non-sensitive settings and its values print in plain sight. Honest caveat: a Secret is base64-encoded, not encrypted, so in production I'd add encryption at rest on etcd, tighten RBAC on the secrets resource, and sync from an external store rather than commit a manifest with the value in it.

Raw YAML works for one environment, but it has no version, no release history, and every environment-specific value — the image tag, the hostname, the replica count — is hard-coded, so you copy-paste and drift. A chart pulls those into values.yaml, renders the templates against them, and installs the result as a named release Helm tracks. That buys me three things: one command deploys the whole app tier, helm upgrade with a new tag is a clean rollout, and helm rollback restores the previous revision if it breaks. For a service I redeploy across dev and prod, that repeatability and rollback is worth the templating cost.

You install cert-manager and create a ClusterIssuer pointing at Let's Encrypt's ACME endpoint. Then you annotate the Ingress with cert-manager.io/cluster-issuer and add a tls block naming a secret. cert-manager's ingress-shim sees that, creates a Certificate object, and starts the ACME flow: it asks Let's Encrypt for the cert, gets an HTTP-01 challenge, serves a token under /.well-known/acme-challenge/ through a temporary route, and Let's Encrypt fetches it to prove I control the domain. On success cert-manager writes the signed cert into the Secret, Traefik serves it, and cert-manager renews it automatically before the 90-day expiry. I never touch a certificate file by hand.

Production Let's Encrypt has strict rate limits — famously about five duplicate certificates per domain per week — and while I'm debugging DNS, firewall rules and annotations I can easily burn through those and get locked out for days. The staging environment has far higher limits and behaves identically, so I point the ClusterIssuer at the staging ACME endpoint first and iterate freely. The only difference is that staging certs are signed by an untrusted root, so browsers warn and curl needs -k — but the whole plumbing is proven. Once a staging cert issues cleanly, I switch the issuer to production and get a real, trusted certificate on the first try.

On a plain cluster you install an ingress controller yourself — on Day 74 that was ingress-nginx via a manifest, plus wiring the node's ports. k3s bundles Traefik out of the box: it's deployed as a managed HelmChart in kube-system and exposed on the node's 80 and 443 by k3s's klipper service load balancer, so an Ingress with ingressClassName: traefik just works with nothing to install. The tradeoff is control — the annotations and config differ from nginx, and a managed platform like EKS gives you neither by default, so there you'd install a controller yourself. For a single-node k3s box, built-in Traefik is the least-effort correct choice.

Running only means the container started — it doesn't prove the app serves traffic. I ran an end-to-end smoke test against the public HTTPS endpoint: a curl POST to /shorten returned a JSON short code, which proves the request reached Traefik, got routed to the linkstash Service and pod, and the app wrote a row to the in-cluster Postgres. Then I curled that short URL and got a 307 redirect back to the original — proving the read path too. If the TLS handshake succeeds, the code comes back, and the redirect lands, every layer works: DNS, the Elastic IP, ingress, Service, pod, and database. That completed request is the definition of done.

Cost and reproducibility. A stopped instance still bills for its EBS volume, and its Elastic IP keeps charging while it's allocated — 'stopped' is not 'free' on AWS. terraform destroy deletes every resource and releases the Elastic IP, so spend actually returns to zero. The second reason is confidence: because the whole stack is in Terraform, I can destroy it now and recreate the identical environment tomorrow with one terraform apply. Teardown is only safe when rebuild is cheap, and IaC makes rebuild one command. I verify the destroy with aws ec2 describe-instances and describe-addresses — I never trust 'Destroy complete' alone; I confirm nothing is left billing.

It's a single EC2 instance in a single Availability Zone, so it has no high availability — if that node or that AZ fails, linkstash is down, unlike Project 2's two-AZ AWS design. Postgres runs in-cluster on a node-local PVC, so the data lives on one disk with no managed backups or failover; production would use RDS. And k3s bundles the whole control plane into one binary on the same node as the workload — perfect for cost and learning, but a managed control plane like EKS gives you a replicated, patched API server across AZs for about $73 a month. I chose k3s deliberately: near-₹0 to run and tear down, with the tradeoffs stated up front.

The browser resolves links.example.com, whose A record points at the instance's Elastic IP. The request hits port 443 on the EC2 node, where k3s's built-in Traefik ingress controller terminates TLS using the cert-manager-issued Let's Encrypt certificate. Traefik matches the Host header against the Ingress rule and forwards to the linkstash Service, a ClusterIP on port 80. The Service load-balances to a linkstash pod on port 8000, where Uvicorn runs the FastAPI app. To create or resolve a link the app connects to the in-cluster Postgres Service using the DATABASE_URL from the linkstash-db Secret. So: DNS → Elastic IP → Traefik/TLS → Service → pod → Postgres.

I'd take my linkstash app, because it shows a progression, not a one-off. It's a self-hostable link-saving service. I first containerized it with a multi-stage Dockerfile, which cut the image from about 900 MB to 360 MB. Then I deployed that image to AWS behind a load balancer, and finally ran it on a k3s Kubernetes cluster with a Deployment and Service, so it self-heals and scales. I lead with the outcome — smaller image, reproducible deploy, self-healing — then go as deep as they want into the how. The repo's in my resume, tagged at a release, so they can read the Dockerfile and manifests themselves.

On my third linkstash build. I only list a tool if I can point to where I ran it, so Kubernetes maps straight to that project: I wrote the Deployment, Service and ConfigMap, ran it on k3s, and watched it reschedule a pod after I killed one. I'm honest about depth — I've run a single-node cluster and read logs and events to debug it; I haven't operated a fifty-node production fleet. Interviewers respect 'here's exactly what I did and didn't do' far more than a skills list I can't defend. Everything on my resume, I can whiteboard.

I measured it directly. Before the multi-stage build, docker images showed the app at around 900 MB because it shipped the whole build toolchain. I split the Dockerfile into a build stage and a slim runtime stage that copies only the compiled artifact and its dependencies, and the final image dropped to about 360 MB — that's the 60%. I keep the before-and-after numbers because a claim without a measurement is just a feeling, and any interviewer can ask how I got it. If I can't measure a result, I don't put a number on it; I describe what I changed instead.

Because the work matters more than the title. Over 90 days I took one app from a container to AWS to Kubernetes, hitting the real problems — a crash-looping container, a security group that blocked traffic, a pod that wouldn't schedule — and I debugged each by reading logs and events, not guessing. That's the day-to-day of the role. I'm not claiming senior experience; I'm showing I can package, deploy, and operate a service and reason about it when it breaks. My resume links the repos, so you don't have to take my word for it — the Dockerfile, the Terraform, and the manifests are all there to read.

My profile leads with three pinned repos — linkstash on Docker, on AWS, and on Kubernetes — so the projects that back my résumé are the first thing they see, not tutorial forks. Each opens on a README that says in one line what it is and why I built it, with an architecture diagram right below and a demo they can watch without cloning. So in 30 seconds they know what I built, that it runs across three environments, and roughly how it's wired — before reading any code. That's deliberate: I treat the repo as the proof for every bullet on the résumé, and the top of the README as the part that has to earn the click.

Order matters more than length. It leads with what the project is in one sentence, then why it exists — the problem it solves — before a single install step. Right after that, an architecture diagram, because a picture answers 'how is this built' faster than paragraphs. Then a copy-paste quickstart, and a teardown so they can clean up. I follow the Readme-Driven-Development idea: write that top section as if before the code, so it describes the project in plain language, not the framework's default scaffold. A demo badge — an asciinema cast — near the top lets them watch it work. Framework boilerplate, if any, goes far below the fold.

A .gitignore first, before the first commit — tfstate, kubeconfig, .env, .pem and key files never get tracked. If something did get committed, deleting it in a new commit is not enough; it's in the history and public repos get scraped in minutes, so the credential is compromised. The real fix is to rotate the secret immediately, then purge it from history with git filter-repo or BFG and force-push. I also lean on GitHub's secret scanning, which alerts on known token formats. The habit is to treat any committed secret as a live incident — rotate first, clean up second — not a tidy-up I'll get to later.

linkstash is a self-hosted FastAPI URL shortener — POST a URL, get a short code that redirects. I built it three times on purpose, and that's the story. Project 1 containerized it with Docker and compose. Project 2 deployed that image on AWS — ECS Fargate behind an ALB with RDS across two AZs, provisioned with the AWS CLI. Project 3 moved it onto Kubernetes with k3s, so it self-heals and scales. Same app, three environments — exactly the Docker to AWS to Kubernetes arc of the whole program. The README leads with that arc and a diagram, plus an asciinema demo. If they want depth, each layer's repo has its own README and history.

It's the written record after an incident — a timeline, the root cause, contributing factors, and action items to stop it recurring. 'Blameless' means we treat failure as a system problem, not a person problem. If an engineer ran a command that took down a service, the question isn't 'why were they careless' but 'why did the system let one command do that, and why didn't a guardrail catch it.' Blame makes people hide mistakes, so you lose the very information you need to fix things. Google popularized this: you get honest timelines and fixes that harden the system instead of scapegoating whoever was on call.

An SLO is a reliability target — say 99.9% availability over a month. The error budget is the allowed unreliability: 100% minus the SLO. Do the math: a 30-day month is about 43,200 minutes, so 99.9% allows 0.1% down — roughly 43 minutes a month. Tighten to 99.99% and the budget drops to about 4.3 minutes. The point is it turns reliability into a currency: if you've spent little of the budget, you can ship risky features fast; if you've burned it, you freeze risky launches and spend the next sprint on stability. It stops both failure modes — chasing an impossible 100%, and shipping recklessly with no guardrail at all.

First you triage severity. A SEV-1 is a full outage or data loss affecting most users; SEV-2 is major but partial; SEV-3 and SEV-4 are minor or cosmetic. Severity sets urgency and who gets paged. For anything serious you name an incident commander — one person who coordinates, decides, and communicates, so responders aren't all talking over each other. The commander doesn't necessarily fix it; they run the response. You pull up the runbook if one exists, mitigate first — roll back, failover, scale up — to stop the bleeding before you chase root cause, and post status updates on a set cadence. Once it's resolved, you schedule a blameless postmortem. Mitigate, communicate, then learn.

A runbook is a pre-written, step-by-step procedure for a known failure mode — 'the database is at 95% disk, here's how to add volume and reclaim space,' or 'the certificate expired, here's the renew-and-reload sequence.' It captures what an expert would do so a tired engineer at 3am doesn't have to reinvent it under pressure. Good runbooks are specific and copy-pasteable: exact commands, not 'investigate the issue.' The value is speed and consistency — mean-time-to-recovery drops because the response is rehearsed, and the fix doesn't depend on which person happens to be on call. Without them, every incident is improvised from scratch, which is slow, error-prone, and unfair to whoever picks up the page.

Crisp answer first, then one concrete detail, then stop. Asked 'what's a readiness probe?', I say: 'It tells Kubernetes a pod isn't ready to serve yet, so the Service stops routing traffic to it until it passes — I used one on linkstash so it didn't take requests before Postgres was reachable.' That's the whole answer in two sentences. I don't launch into liveness and startup probes unless they pull for more. Rapid-fire tests breadth and composure, not depth — burying the right answer under thirty seconds of hedging reads as uncertainty even when you're correct. Give them the hook and let them decide how deep to go.

I don't guess the cause; I narrate a structure, because on the real job the cause is unknown. Four beats: check the layer, read the signal, form a hypothesis, verify. First I reproduce it and locate the symptom — is it DNS, the edge, the app, or the database? Then I read what the system is telling me: the HTTP status, kubectl get pods, logs, metrics. From that signal I form one testable hypothesis and say it out loud. Then I run the single command that confirms or kills it. If it's killed, I drop to the next layer and loop. Calmly walking that loop is worth more than blurting the right answer.

I use the STAR shape — Situation, Task, Action, Result — with a real story. Situation: deploying a new linkstash image to the k3s cluster. Task: ship it without downtime. Action: I changed DATABASE_URL in a hurry and typo'd the password, so every request started returning 500 while the pods still showed Running. Result: I read the logs, saw a Postgres auth failure, checked kubectl rollout history, and ran kubectl rollout undo — back to 200 in about two minutes. What I changed after: the connection string moved into a reviewed Secret instead of a hand-typed env var. Owning the mistake and the fix matters more than pretending you've never broken anything.

It's the triage loop run forward, layer by layer. The browser checks its cache, then DNS resolves the domain to an IP — a recursive resolver walking root, TLD and authoritative servers. A TCP connection opens to that IP on 443, and a TLS handshake negotiates the certificate and keys. The browser sends an HTTP request; a load balancer or ingress routes it to a server, which may hit an app and a database, and returns a response. The browser parses the HTML and fetches CSS, JS and images, then renders. I stop at whatever layer they probe — the point is I can traverse the whole path in order without losing the thread.

First I confirm scope and declare — a SEV-1 with every region 5xx is total and customer-facing, so I make sure on-call is paged, open an incident channel, and take the commander role if no one has. Then I triage from the top: read the alerts and the incident log to see what fired first, because the earliest event usually sits nearest the cause. Every region failing at once points at something shared — DNS, a global config, a security-group or IAM change — not one bad host. I resist touching anything until I understand the chain. If customers are down I'll mitigate before I fully diagnose, but I mitigate the cause, not a symptom.

Always the upstream cause. The symptom can't recover until the thing it depends on is healthy. In tonight's capstone a revoked security group blocked the network path, which reddened the load balancer's health checks, which tripped DNS failover, which surfaced a missing Secret downstream. If I recreate the Secret first, nothing happens — the cluster API is unreachable while the network is still blocked, so I've done work that completes nothing and I might fool myself into thinking I'm progressing. Re-open the security group, and only then does restoring the Secret take. Fixing downstream first at best wastes minutes and at worst hides the real cause.

No — the failover is the system working as designed. Route 53 health-checked the primary, saw it fail, and moved traffic to the passive region to keep the site reachable. Treating the failover as the bug sends you rolling DNS back, which just points traffic at a region that's still broken. The failover is a symptom and a signal: it tells me the primary's health checks went red, so I go find why. Tonight that was the revoked security group. I fix the primary, watch its health checks recover, and confirm DNS fails back on its own. The failover bought me time — I don't fight it, I use it.

When the customer-facing symptom is gone and I've confirmed it at the top of the stack — not when one component looks healthy. Tonight, pods going Running is not 'over': they can be 3/3 while DNS still points at the failover address, so users remain on a degraded path. I only call it once dig shows traffic back on the healthy primary. Then comes the real close: write the postmortem while it's fresh, capture the timeline and the fix order, and file the action items — like guarding that security-group change — so the same fat-finger can't page someone at midnight again.