A comprehensive guide to Linux fundamentals for DevOps: filesystem, permissions, process management, systemd, networking, SSH, bash scripting, and more.
Linux is the operating system under every Docker container, every Kubernetes node, and the majority of cloud VMs. Before you can deploy, debug, or automate anything in a modern infrastructure stack, you need to be comfortable at the shell. This guide takes you from the absolute fundamentals — how Linux thinks about files and processes — through the tools DevOps engineers reach for daily: permissions, grep/awk/sed, systemd, SSH, package managers, network commands, and production-grade bash scripting. Follow the Linux roadmap to see how these topics connect into a full learning path.
Introduction to Linux
Section 1 of 39 · ~6 min
Imagine you bought a brand new car. You see the dashboard, the steering wheel, the leather seats, the touchscreen infotainment system. But what actually moves the car? The engine. You rarely open the hood, you never see it during your daily drive, yet without it, nothing works. Linux is exactly that engine for the entire modern internet. Every time you scroll Instagram, order from Uber Eats, watch Netflix, or tap your phone to pay — somewhere in a datacenter, a Linux server is doing the heavy lifting. You never see it. You never think about it. But it’s powering 96% of the world’s digital infrastructure in 2026.
As an experienced frontend developer, you’ve been writing code that runs on top of this engine. You’ve been the person designing the dashboard. Now you’re about to learn what’s under the hood — and that’s exactly what the highest-paid DevOps engineers know cold. The transition from frontend to DevOps is one of the most valuable career moves in tech today, because you already understand how applications behave; now you’ll learn how they live, breathe, scale, and recover at the infrastructure level.
Real world: Think of Linux like the global air-traffic network. You don’t see the radar, the control towers, or the routing systems when you fly — but they’re the reason tens of thousands of flights move millions of passengers safely every single day. Linux is the air-traffic network of the internet. AWS, Azure, GCP, Netflix, Google, Facebook, WhatsApp, Spotify, Amazon — they all run on Linux.
Why Linux Dominates DevOps in 2026
If you walked into any tech company in a major tech hub today and asked their SRE team what operating system runs production, the answer is almost universally Linux. Here’s why this domination is now absolute:
~96% of public cloud servers run Linux — AWS EC2, Azure VMs, Google Compute Engine. Even Microsoft, which makes Windows, runs the majority of Azure on Linux.
100% of containers use the Linux kernel — Docker, Podman, containerd all rely on Linux namespaces and cgroups. Even “Windows containers” running on Docker Desktop spin up a hidden Linux VM underneath.
Every single Kubernetes node is Linux — kubelet, kube-proxy, CNI plugins, the entire control plane is designed for Linux first.
All major CI/CD runners are Linux — GitHub Actions, GitLab Runners, Jenkins agents, ArgoCD, CircleCI default to Linux.
Every observability stack runs on Linux — Prometheus, Grafana, Loki, Elastic, Datadog agents, OpenTelemetry collectors.
Edge computing and IoT — Raspberry Pi, AWS Greengrass, Azure IoT Edge, all Linux.
For a React developer transitioning to DevOps, this means one undeniable truth: you cannot avoid Linux. It’s not optional anymore. The good news? You already have Ubuntu 24.04 installed, which is the gold standard for DevOps learning in 2026.
Linux vs Windows vs macOS: The Honest Comparison
Feature
Linux
Windows
macOS
Cost
Free (most distros)
~$140 per license
Bundled with Apple hardware (premium)
Customization
Unlimited — change anything
Limited to UI/registry
Restricted — Apple’s way only
CLI Power
Native, world-class (bash, zsh)
Catching up (PowerShell, WSL2)
Strong (Unix-based, zsh default)
Server Use
~96% of internet servers
~3% (legacy enterprise AD)
Negligible (not a server OS)
Security
Excellent (permissions, SELinux)
Improving but still targeted
Strong (sandboxed by default)
Package Mgmt
apt, dnf, pacman, snap, flatpak
winget, choco, MSI installers
brew (community), App Store
Source Code
100% open source (GPL)
Proprietary (closed)
XNU kernel partially open, OS closed
DevOps Fit
The de-facto standard
Niche (Windows containers)
Good for local dev only
The Linux Distro Universe
One of the most confusing things for beginners is the sheer number of Linux distributions (“distros”). They’re all Linux at the core (same kernel), but they package the system differently — different installers, different package managers, different defaults, different philosophies. Here’s the map you need:
Distro
Base
Package Mgr
Best For
DevOps Use
Ubuntu
Debian
apt
Development, learning, desktops
Dev laptops, CI runners, EC2 default
RHEL
Fedora
dnf/yum
Enterprise production
Banks, telco, govt, regulated industries
CentOS Stream
RHEL upstream
dnf
Testing RHEL features early
Rolling preview of RHEL
Debian
Independent
apt
Stable servers, rock-solid uptime
Long-running production servers
Alpine
Independent (musl)
apk
Containers — only 5MB base!
99% of Docker base images
Rocky/AlmaLinux
RHEL clones
dnf
Free RHEL alternative
Post-CentOS production replacement
Arch Linux
Independent
pacman
Advanced users, customization
Personal workstations (rarely prod)
Amazon Linux 2023
Fedora-based
dnf
AWS-optimized workloads
Default EC2/ECS/Lambda runtime
Note: Which distro do real DevOps engineers actually use?
Dev laptop: Ubuntu 24.04 LTS or macOS
Production servers: RHEL, Amazon Linux 2023, Rocky Linux, or Ubuntu Server LTS
Docker base images: Alpine (tiny, fast), or distroless/Debian-slim
Kubernetes nodes: Ubuntu, Bottlerocket (AWS), Flatcar, or Talos Linux
Master Ubuntu first because apt-based skills transfer 80% to Debian and Kali. Then learn dnf for RHEL/Rocky/Amazon Linux. That covers ~95% of real-world DevOps scenarios.
Kernel vs Operating System
This is the single most common interview blunder. Junior candidates confuse “kernel” and “OS” all the time.
The Linux Kernel is the core program — roughly 30+ million lines of C code — written originally by Linus Torvalds in 1991 when he was a 21-year-old student in Helsinki, Finland. The kernel is just the engine — it talks to the CPU, RAM, disks, network cards, and so on. By itself, the kernel cannot give you a usable computer.
An Operating System is the kernel plus all the surrounding software needed to make it usable: the GNU utilities (ls, cat, grep, sed, awk), the shell (bash, zsh), system libraries (glibc), package managers (apt, dnf), and applications. So when you say “Linux,” what you usually mean is “GNU/Linux.”
Interview tip: If an interviewer asks “What is Linux?” — never just say “an operating system.” Say: “Linux is technically the kernel — a monolithic Unix-like kernel created by Linus Torvalds in 1991. When people say ‘Linux,’ they usually mean GNU/Linux, which is the kernel plus the GNU userland utilities, a shell, system libraries like glibc, and applications, all packaged by a distribution like Ubuntu or RHEL.”
Open Source and the GPL
Linux is released under the GNU General Public License version 2 (GPLv2). The GPL guarantees four fundamental freedoms (numbered 0 through 3, because programmers count from zero):
Freedom 0: Run the program for any purpose, without restriction.
Freedom 1: Study how the program works and modify it. Source code access is required.
Freedom 2: Redistribute copies to help others.
Freedom 3: Distribute your modified versions so the community benefits.
Why do massive corporations like Google, Meta, Amazon, and Microsoft pour billions into a “free” project? Because open-source Linux gives them zero licensing cost at scale, total customizability, security through transparency, vendor independence, and faster innovation.
Figures are rough 2026 market signals, not guarantees. They swing widely with company size, city, cost of living, and equity — treat them as the relative gaps between levels, not absolute targets.
Now that you know what Linux is, let’s open the hood and look at how it actually works. Linux follows a layered architecture — each layer has a specific job, and they communicate through well-defined interfaces. If you understand this architecture clearly, you’ll debug production issues 10x faster than engineers who only know commands by rote.
The Kernel: The Brain of the System
The Linux kernel is the lowest software layer above the hardware. It has six core responsibilities that you must internalize:
Process Management: Creating, scheduling, pausing, killing processes. The kernel decides which process gets CPU time. Tools like ps, top, htop, kill all interact with this subsystem.
Memory Management: Allocating RAM to processes, managing virtual memory, paging, swapping. The OOM (Out of Memory) killer lives here.
Device Drivers: Translating generic OS calls into hardware-specific signals — GPU, NIC, SSD, USB, Bluetooth all speak through drivers loaded as kernel modules.
Filesystem Management: Reading, writing, organizing data on disk through filesystems like ext4, xfs, btrfs, zfs, overlayfs (used by Docker).
Networking: The entire TCP/IP stack, sockets, packet routing, iptables/nftables — all kernel code.
Security & Permissions: User IDs, group IDs, file permissions, capabilities, SELinux/AppArmor, seccomp filters (used by Docker for sandboxing).
Shells: Your Window to the Kernel
You don’t talk to the kernel directly — you talk to a shell, and the shell talks to the kernel. A shell is just a program that reads commands, interprets them, and asks the kernel to do work via system calls.
Shell
Default On
Strengths
Weaknesses
DevOps Verdict
Bash
Ubuntu, RHEL, Debian, Alpine (ash)
Universal, scripting, POSIX compliance
Plain UX, no autosuggestions
Must master — every server has it
Zsh
macOS Catalina+
Themes (oh-my-zsh), autosuggestions, plugins
Slightly heavier, not on servers
Great for daily dev laptop use
Fish
None default
Friendly UX, autosuggest out of the box
Non-POSIX — scripts don’t transfer
Personal use only, never scripts
Dash
Ubuntu’s /bin/sh
Tiny, fast, POSIX-strict
Minimal features
Used by Ubuntu boot scripts
PowerShell
Windows
Object pipeline, cross-platform now
Heavy, alien to Unix folk
Skip unless Windows-shop
Tip: Use Zsh + oh-my-zsh on your Ubuntu dev laptop for daily comfort, but always write your shell scripts in Bash (with #!/usr/bin/env bash) so they run on every server you’ll ever touch. Production servers don’t have your fancy zsh setup.
System Libraries and Utilities
Between user programs and the kernel sit two crucial layers. System Libraries — the most important is glibc (GNU C Library). When your Node.js process needs to open a file, it calls a C function like fopen() in glibc, which translates to the kernel’s open() system call. Alpine Linux uses musl libc instead, which is why Alpine images are smaller but sometimes have subtle compatibility quirks.
System Utilities — the GNU coreutils package gives you ls, cp, mv, rm, cat, echo, mkdir, chmod, chown, and ~100 more. Then there’s util-linux, findutils, grep, sed, awk, tar, gzip — collectively known as “the Unix toolbox.”
User Space vs Kernel Space
Modern CPUs have two privilege modes: kernel mode (ring 0 on x86) where code can do anything, and user mode (ring 3) where code is sandboxed and must ask the kernel (via system calls) to do anything dangerous. Your React app, Node.js, Python scripts, even databases like PostgreSQL — all run in user space. Only the kernel and its drivers run in kernel space. You can literally see system calls with strace:
When you press the power button on a Linux machine (or click “Start instance” on an EC2 console), a tightly choreographed sequence brings the system to life. Understanding this is critical because most production outages happen during boot.
BIOS / UEFI: The first code that runs lives in firmware on the motherboard. It initializes the CPU and looks for a bootable device. On EC2, this is the Nitro hypervisor’s firmware.
POST (Power-On Self-Test): Hardware checks — RAM, keyboard, basic devices.
MBR / GPT Boot Sector: Firmware reads the first sector of the boot disk to find the bootloader.
GRUB (GRand Unified Bootloader): Displays the boot menu, lets you pick a kernel, and loads the kernel image and initramfs into memory. Config lives in /boot/grub/grub.cfg.
Kernel + initramfs: The kernel unpacks itself, the initramfs provides essential drivers to mount the real root filesystem. Kernel mounts / and starts PID 1.
systemd (PID 1): Modern init system. Reads /etc/systemd/system/, starts services in dependency order, mounts filesystems from /etc/fstab, brings up the network.
multi-user.target / graphical.target: systemd’s “runlevel” — on servers it stops at multi-user.target (text login).
Login prompt: getty on TTYs, sshd for remote, GDM for graphical.
Interview tip: “Walk me through what happens when you press the power button on a Linux server until you get a login prompt.” This is asked in literally every senior DevOps interview. Memorize the 8-step sequence above. Bonus points if you mention systemd targets, initramfs, and the role of /etc/fstab.
DevOps Tie-Ins: Why Boot Knowledge Pays
EC2 boot failures: When an EC2 instance won’t start, AWS shows you the serial console output — you see exactly which boot step failed.
GRUB recovery: Kernel update broke boot? You boot from a rescue ISO, chroot into the broken system, run grub-install and update-grub.
systemd unit debugging: A service won’t start at boot? systemctl status, journalctl -xeu service-name, check dependencies with systemctl list-dependencies.
Custom AMI building: Packer images, custom EKS node AMIs, Bottlerocket — all require understanding what boots and when.
Container startup: Containers skip BIOS/GRUB/kernel — they start at “PID 1 in a namespace.” tini exists exactly because containers need a proper PID 1.
Linux organises everything into a single tree rooted at /. Unlike Windows with drive letters, there is no C:\ — everything is a file or a directory beneath /. Understanding where things live stops you from wasting time searching for config files.
The Linux filesystem is a single tree rooted at /; each top-level directory has a defined purpose per the FHS standard.
Key Directories
Path
What lives here
/bin, /usr/bin
Essential user binaries (ls, cp, bash)
/sbin, /usr/sbin
System administration binaries (fdisk, iptables)
/etc
System-wide configuration files — everything in plain text
Virtual filesystem exposing kernel and process state
/sys
Virtual filesystem for kernel subsystems (devices, drivers)
/dev
Device files (/dev/sda, /dev/null, /dev/random)
/boot
Kernel image, initrd, GRUB config
/opt
Optional third-party software
/usr/local
Locally compiled software (takes precedence over /usr)
/run
Runtime data (PIDs, sockets) — tmpfs, cleared on boot
/mnt, /media
Mount points for temporary / removable filesystems
Navigating the Tree
bash
pwd # print working directoryls -la /etc # long listing including hidden filesls -lh /var/log # human-readable sizescd /var/log # change directorycd - # go back to previous directorycd ~ # go hometree -L 2 /etc # visual tree, depth 2 (install with apt)
Inodes and Hard Links
Every file is represented by an inode — a data structure storing metadata (owner, permissions, timestamps, block pointers) but not the filename. Directory entries are simply (name → inode number) mappings. This is why:
Hard links are just extra directory entries pointing to the same inode. Delete one, the data survives until the last link is removed.
Symbolic links (ln -s) store a path string. They can cross filesystem boundaries; hard links cannot.
df -i shows inode usage — a filesystem can run out of inodes before running out of blocks if you create millions of tiny files.
bash
ls -li /etc/hostname # shows inode number in first columnstat /etc/hostname # full inode metadata dumpln /etc/hostname /tmp/hn # hard link (same inode)ln -s /etc/hostname /tmp/hn_sym # symbolic link
Tip:/proc and /sys contain no data on disk — they are windows into kernel memory. cat /proc/cpuinfo reads CPU registers; cat /proc/$(pgrep nginx)/status reads a live process’s memory map. Learn to explore them freely.
As a DevOps engineer, 80% of your day will be spent reading logs, editing config files, and piecing together what went wrong on a server at 2 AM. Coming from React, you’ve used VS Code — but on a remote production server with no GUI, you have only the terminal. This section makes you fast at viewing files, slicing log data, and editing config files directly on Linux boxes.
Real world: On a production Ubuntu server, when nginx is throwing 502s, you SSH in and immediately run tail -f /var/log/nginx/error.log. Then you edit /etc/nginx/nginx.conf with vim, run nginx -t to validate, and reload. No VS Code, no GUI — just your terminal skills.
cat — Concatenate and View
cat dumps file contents to stdout. Great for small files; avoid on huge logs (use less instead).
bash
$ cat /etc/hostname$ cat -n /etc/nginx/nginx.conf # -n prefixes every line with its line number$ cat -A script.sh # -A shows hidden chars: $ for newline, ^I for tab, ^M for CRLF$ cat file1.txt file2.txt > combined.txt$ cat ~/.ssh/id_rsa.pub | ssh user@server 'cat >> ~/.ssh/authorized_keys'
Tip: If a bash script fails with weird “command not found” errors and you wrote it on Windows, run cat -A script.sh. If you see ^M$ at end of lines, that’s CRLF. Fix with tr -d '\r' < script.sh > clean.sh or dos2unix script.sh.
tac — Reverse cat
tac prints lines in reverse order (last line first). Useful for logs when you want the newest entries on top.
bash
$ tac /var/log/syslog | head -20 # 20 most recent syslog entries, newest first$ tac access.log | grep "POST /api/login" | head -5
less and more — The Pagers
less is the modern, more powerful pager. Rule: always use less.
bash
$ less /var/log/nginx/access.log # file is NOT loaded fully into memory — works on huge logs$ less +F /var/log/syslog # open directly in follow mode$ journalctl -u nginx | less
Key
Action
q
Quit
/pattern
Search forward
?pattern
Search backward
n / N
Next / previous match
g / G
First / last line
Space / b
Next / previous page
F
Follow mode (like tail -f) — Ctrl+C to exit
&pattern
Show ONLY lines matching pattern (filter)
head — First N Lines
bash
$ head /etc/passwd # first 10 lines (default)$ head -n 20 access.log$ head -c 100 /dev/urandom | base64 # grab first 100 BYTES of random data$ head -n -5 file.txt # everything EXCEPT last 5 lines
tail — Last N Lines and Follow Mode
If there’s ONE command you must master for DevOps, it’s tail -f.
Caution: tail -f vs tail -F: Production logs are rotated daily by logrotate. If you use tail -f and the file gets rotated, you’ll silently stop seeing new logs. Always use tail -F for long-running tails on production. Capital F = forever.
Critical:uniq only removes adjacent duplicates. You MUST sort first.
bash
$ sort ips.txt | uniq -c # -c prefixes each unique line with its count$ sort ips.txt | uniq -d # -d only shows lines appearing MORE than once$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head# THE classic DevOps one-liner: top 10 IPs hitting your server
Real world: Spot a DDoS in 5 seconds: awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20. If one IP has 50,000 requests and the next has 200, you’ve got a bot. Block it with iptables or ufw.
$ ls -la | tee listing.txt # see output AND save it$ echo "new line" | tee -a log.txt # -a append$ echo "127.0.0.1 myapp.local" | sudo tee -a /etc/hosts$ make 2>&1 | tee build.log
Tip: sudo tee: Bash redirection (>) happens in the calling shell BEFORE sudo runs. So sudo echo "x" > /etc/protected fails with “Permission denied.” Use echo "x" | sudo tee /etc/protected — tee runs under sudo and CAN write.
Text Editor: nano (beginner-friendly)
All shortcuts are visible at the bottom of the screen. Good for quick edits to /etc/hosts or a config file.
Shortcut
Action
Ctrl+O
Save (Write Out)
Ctrl+X
Exit
Ctrl+W
Where-is — search forward
Ctrl+\
Search & replace
Ctrl+K / Ctrl+U
Cut line / paste
Alt+/ or Ctrl+_
Go to line number
Text Editor: vim — Deep Dive
Vim is on every Linux server by default. Mastering it makes you 10x faster than someone fumbling with nano.
The Four Modes: Normal (navigate, delete, copy — keys are COMMANDS), Insert (type text — enter via i/a/o, exit via Esc), Visual (select text via v/V/Ctrl+V), Command (run commands via :).
Interview tip: Most Asked Vim Question: “What are the vim modes and how do you exit vim?” Answer: 4 modes — Normal, Insert, Visual, Command. To exit: press Esc to ensure you’re in Normal mode, then type :q (quit), :wq (write and quit), or :q! (force quit without saving).
Command
Action
:w
Save (write)
:q
Quit (errors if unsaved)
:q!
Force quit, discard changes
:wq or ZZ
Save and quit
i / a
Insert before / append after cursor
o / O
Open new line below / above
h j k l
Left, Down, Up, Right
w / b
Next / previous word
0 / $
Start / end of line
gg / G
Top / bottom of file
dd / yy / p
Delete line / yank line / paste
u / Ctrl+r
Undo / redo
/pattern / n
Search / next match
:%s/old/new/g
Replace ALL “old” with “new” in file
:%s/old/new/gc
Same, but confirm each
A solid starter ~/.vimrc for DevOps work:
bash
syntax onset numberset relativenumberset expandtab " convert tabs to spaces (critical for YAML/Python)set tabstop=4set shiftwidth=4set hlsearchset incsearchset ignorecaseset smartcasefiletype plugin indent on" YAML-specific (2-space indent — important for k8s/Ansible)autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab
Tip: Run vimtutor in your terminal right now. It’s a 30-minute interactive tutorial that ships with vim. By the end, you’ll know enough to edit any file confidently.
Linux permissions control who can read, write, or execute every file. Getting this wrong is a security vulnerability; getting it right is foundational operations knowledge.
# Symbolic modechmod u+x script.sh # add execute for ownerchmod go-w sensitive.conf # remove write for group and otherschmod a+r public.html # add read for everyonechmod u=rwx,g=rx,o= priv/ # set exactly# Octal mode (faster once memorised)chmod 755 script.sh # rwxr-xr-xchmod 644 config.yml # rw-r--r--chmod 600 ~/.ssh/id_rsa # rw------- (private key requirement)chmod 700 ~/.ssh/ # rwx------ (SSH dir requirement)chmod 777 /tmp/shared # rwxrwxrwx — almost always wrong# Ownershipchown pushkar:devops file.txt # change owner and groupchown -R www-data:www-data /var/www/ # recursivechgrp docker /var/run/docker.sock # change group only
Special Bits
Bit
On files
On directories
setuid (4xxx)
Run as owner’s UID (-rwsr-xr-x)
No effect
setgid (2xxx)
Run as owner’s GID
New files inherit directory’s group
sticky (1xxx)
No effect
Only owner/root can delete files (/tmp)
bash
chmod u+s /usr/bin/passwd # setuid — passwd needs root to modify /etc/shadowchmod g+s /var/www/uploads # setgid — all new files owned by www-data groupchmod +t /tmp # sticky — standard on /tmpls -la /usr/bin/passwd # -rwsr-xr-x (note the 's')
umask
The umask is subtracted from the default permissions when new files are created (default 666 for files, 777 for dirs).
bash
umask # 0022 → files get 644, dirs get 755umask 027 # more restrictive: files 640, dirs 750
Tip: ACLs (setfacl/getfacl) extend beyond the three-owner model. Use them when you need fine-grained per-user access to a shared directory without changing ownership.
grep filters lines matching a regular expression. It’s your first tool for searching logs, configs, and code at the command line. See also Piping Redirection for how grep composes with other tools. Build and test patterns first with the Regex Log Tester.
Basic Usage
bash
grep "error" /var/log/syslog # case-sensitive matchgrep -i "error" /var/log/syslog # case-insensitivegrep -n "timeout" app.log # show line numbersgrep -c "WARN" app.log # count matching linesgrep -v "DEBUG" app.log # invert — lines NOT matchinggrep -r "SECRET_KEY" /etc/ # recursive directory searchgrep -l "nginx" /etc/logrotate.d/* # only print filenamesgrep -w "root" /etc/passwd # whole-word match only
awk is a field-oriented text processor. Think of it as a per-line program: for each line, split into fields, run your script. It handles 80% of the log-parsing and report-generation tasks you’d otherwise reach for Python for.
Basic Syntax
bash
awk '{print $1}' file.txt # print first field (whitespace-delimited)awk '{print $1, $3}' file.txt # fields 1 and 3awk '{print NR, $0}' file.txt # NR = record (line) number, $0 = whole lineawk 'NR==5' file.txt # print only line 5awk 'NF > 3' file.txt # lines with more than 3 fields
sed is a non-interactive text editor that reads input line-by-line and applies editing commands. While vim/nano are interactive, sed runs in a CI/CD pipeline, in a deployment script, in a Dockerfile — anywhere you need to programmatically modify files without a human pressing keys.
Note: Why DevOps engineers can’t live without sed:
CI/CD config injection: swap localhost for prod URL during deploy.
Templating: render config files from templates with environment-specific values.
Batch find-and-replace: across hundreds of files in a single command.
Log scrubbing: remove PII, mask passwords before sharing logs.
Basic Syntax
bash
$ sed 's/old/new/' file # substitute (first occurrence per line)$ sed 's/old/new/g' file # substitute ALL occurrences (global)$ echo "hello world" | sed 's/world/devops/'hello devops
By default sed prints to stdout and does NOT modify the file. Use -i to edit in-place.
Key Operations
Command
What it does
s/old/new/
Substitute first match per line
s/old/new/g
Substitute all matches (global)
s/old/new/gi
Global, case-insensitive
/pattern/d
Delete lines matching pattern
3d
Delete line 3
1,5d
Delete lines 1 through 5
/pattern/p
Print matching line (usually with -n)
/pattern/i\text
Insert text BEFORE matching line
/pattern/a\text
Append text AFTER matching line
y/abc/xyz/
Transliterate: a→x, b→y, c→z
/start/,/end/d
Delete from /start/ through /end/
In-Place Editing
bash
$ sed -i 's/DEBUG=true/DEBUG=false/' .env # DANGEROUS — no undo!$ sed -i.bak 's/DEBUG=true/DEBUG=false/' .env # creates .env.bak first — RECOMMENDED
Caution: Always use -i.bak in production. A bad sed command can corrupt thousands of files instantly. The .bak safety net costs nothing.
Multiple Commands and Regex
bash
$ sed -e 's/foo/bar/g' -e 's/baz/qux/g' file # chain with -e$ sed 's/foo/bar/g; s/baz/qux/g' file # or semicolons$ sed -E 's/([0-9]+)/NUMBER/' file # -E for Extended Regex$ sed -E 's/(.+)@(.+)/User: \1, Domain: \2/' email.txt # backreferences
Real DevOps Examples
bash
$ sed 's/localhost/prod-server.com/g' config.yml # swap hostname (preview)$ sed -i 's/DEBUG=true/DEBUG=false/' .env # disable debug in prod$ sed -n '5,10p' logfile # print lines 5-10$ sed '/^#/d' config.conf # remove comment lines$ sed '/^$/d' file # strip blank lines$ sed -i '1i\#!/bin/bash' script.sh # add shebang at top$ sed 's/^[[:space:]]*//' file # trim leading whitespace$ sed -i 's|/old/path|/new/path|g' config # use | delimiter for paths$ sed -i '/^#PermitRootLogin/s/^#//' /etc/ssh/sshd_config # uncomment a line$ sed -i 's/\r$//' file.txt # convert CRLF to LF$ sed -E 's/(password=)[^&]+/\1******/g' app.log # mask passwords
sed vs awk — When to Use Which
Task
Use
Why
Find and replace text
sed
Built for substitution
Delete lines by pattern
sed
One-liner with /pattern/d
Column-based processing
awk
Field-aware out of the box
Arithmetic / aggregation
awk
Full programming language
Edit config files in CI/CD
sed
Quick, simple substitutions
Parse log columns and count
awk
Native associative arrays
Real world: Kubernetes ConfigMap templating with sed:
The Unix philosophy: small tools that do one thing well, composed via pipes. Mastering redirection turns individual commands into data pipelines.
Data flows left-to-right through pipes; > redirects final stdout to a file, while 2>&1 merges stderr into the same stream.
Redirection Operators
bash
# stdout redirectioncommand > file.txt # redirect stdout (overwrite)command >> file.txt # append stdoutcommand 2> errors.txt # redirect stderrcommand 2>&1 # merge stderr into stdoutcommand > file.txt 2>&1 # both stdout and stderr to filecommand &> file.txt # bash shorthand for above# stdin redirectioncommand < input.txt # feed file as stdincommand <<< "string" # here-string (bash)command << EOF # heredocline1line2EOF# /dev/null — discardcommand > /dev/null 2>&1 # suppress all outputcommand 2>/dev/null # suppress errors only
Pipes
bash
# Basic pipe: stdout of left → stdin of rightls -la | grep ".log"cat /etc/passwd | awk -F: '{print $1}' # inefficient, use awk directlyps aux | grep nginx | grep -v grep# xargs — build commands from stdinfind /var/log -name "*.log" | xargs wc -l # count lines in all logsfind /tmp -mtime +7 | xargs rm -f # delete old filescat servers.txt | xargs -I{} ssh {} "uptime" # run on each host# tee — split: stdout goes to both pipe and filecommand | tee output.txt | next_command # log and continue
Process Substitution
bash
# <(cmd) — treat command's output as a file (bash)diff <(sort file1.txt) <(sort file2.txt)comm -23 <(sort list1.txt) <(sort list2.txt) # lines only in list1# >(cmd) — feed output to command as if it were a filetee >(wc -l > line_count.txt) < big_file.txt
Practical Pipelines
bash
# Top 10 largest files in /varfind /var -type f -printf '%s %p\n' | sort -rn | head -10# Count unique IPs in nginx log for todaygrep "$(date +%d/%b/%Y)" /var/log/nginx/access.log | awk '{print $1}' | sort -u | wc -l# Show only failed systemd unitssystemctl list-units --state=failed --no-legend | awk '{print $1}'# Monitor a log in real time, highlight errorstail -f /var/log/syslog | grep --line-buffered -E "error|warn|crit"
Tip:grep --line-buffered forces grep to flush after every matched line when reading from a pipe — essential for real-time monitoring. Without it, grep buffers output and you get it in chunks.
In Linux, everything is a process. When you run ls, that’s a process. When a Docker container starts, that’s an (isolated) process. When nginx serves a request, that’s a process too. To become a DevOps engineer you must understand processes deeply.
What is a Process?
A process is a running instance of a program. The program (executable file like /usr/bin/node) sits on disk — when you execute it, the kernel loads it into memory, gives it a unique PID (Process ID), assigns CPU time, memory, and file descriptors.
PID — Process ID. Unique integer. PID 1 is always init / systemd.
PPID — Parent Process ID. Every process (except PID 1) has a parent.
UID / GID — User and Group ID the process runs as. Determines permissions.
TTY — The terminal it’s attached to (or ? for daemons).
NI — Nice value (priority, -20 highest to 19 lowest).
Foreground vs Background
A foreground process owns the terminal and blocks your shell until it exits. A background process detaches (append &) so the prompt returns immediately while it keeps running.
Linux creates new processes through two system calls — the famous fork() + exec() pair.
fork() — The parent process duplicates itself. The child gets a new PID; parent gets the child’s PID, child gets 0.
exec() — The child replaces its memory image with a new program. Same PID, different code.
wait() — Parent waits for child to finish and reaps its exit status.
When you type ls in bash: bash forks itself, the child execs /usr/bin/ls, bash waits for it to exit.
Process States
State
Code
Meaning
When you see it
Running
R
Currently executing or runnable
Active workloads
Sleeping (Interruptible)
S
Waiting for an event (I/O, signal, timer)
Most idle daemons
Disk Sleep (Uninterruptible)
D
Waiting for I/O — cannot be killed even with SIGKILL
NFS hang, broken disk
Stopped
T
Suspended by signal (SIGSTOP / Ctrl+Z)
After you hit Ctrl+Z
Zombie
Z
Dead but parent hasn’t reaped exit status
Bad parent process
Idle
I
Idle kernel thread
Kernel worker threads
Caution: D state is dangerous: Disk Sleep processes ignore all signals including SIGKILL. If you see lots of D state processes, your disk / NFS mount is likely broken. kill -9 won’t help.
Exploring /proc/[pid]/
The kernel exposes every process as a directory under /proc/.
bash
$ pgrep -f nginx | head -11842$ cat /proc/1842/cmdline | tr '\0' ' 'nginx: master process /usr/sbin/nginx -g daemon on; master_process on;$ cat /proc/1842/status | head -6Name: nginxState: S (sleeping)Pid: 1842PPid: 1$ readlink /proc/1842/exe # path to binary/usr/sbin/nginx
Why Processes Matter in DevOps
Containers ARE processes. A Docker container is just a regular Linux process with extra isolation provided by the kernel: namespaces (separate views of PIDs, network, mounts), cgroups (CPU, memory, I/O limits), chroot/pivot_root (restricted filesystem), and seccomp/capabilities (restricted syscalls).
Interview tip: “What’s the difference between a VM and a container?” — A VM virtualizes hardware and runs a full guest kernel. A container is a regular process on the host kernel with namespaces and cgroups for isolation. Container = process. VM = virtual machine.
Understanding how to inspect and manage processes is essential for debugging hung services, investigating CPU/memory spikes, and safely stopping background work.
ps aux # all processes, BSD syntax (a=all, u=user, x=no-tty)ps -ef # all processes, POSIX syntax (-e=all, -f=full)ps aux | grep nginx # filter by nameps -p 1234 -o pid,ppid,comm,etime,rss # specific PID, custom columns# pgrep / pkill — search/signal by namepgrep -l nginx # list PIDs matching "nginx"pgrep -u www-data # PIDs owned by www-datapkill -HUP nginx # send SIGHUP to all nginx processes (graceful reload)
top and htop
bash
top # interactive process viewer # Keys: k=kill, r=renice, u=filter by user, 1=per-CPU, q=quit # Press 'c' to show full command linehtop # enhanced viewer (apt install htop) # Mouse clicks work; F5=tree view, F6=sort column
Signals
bash
kill -l # list all signalskill PID # send SIGTERM (15) — polite shutdown requestkill -9 PID # SIGKILL — immediate termination (no cleanup)kill -HUP PID # SIGHUP (1) — reload config (nginx, rsyslog)kill -STOP PID # pause a processkill -CONT PID # resume a paused process# Sending to all instanceskillall nginx # by namepkill -f "python worker" # by full command line regex
Signal
Number
Meaning
SIGHUP
1
Hangup / reload config
SIGINT
2
Keyboard interrupt (Ctrl-C)
SIGQUIT
3
Quit with core dump
SIGKILL
9
Immediate kill (uncatchable)
SIGTERM
15
Graceful shutdown (default for kill)
SIGUSR1/2
10/12
User-defined (app-specific)
SIGSTOP
19
Pause (uncatchable)
SIGCONT
18
Resume
Background Jobs and nice
bash
# Background/foregroundcommand & # start in backgroundjobs # list background jobs in current shellfg %1 # bring job 1 to foregroundbg %2 # resume stopped job 2 in backgroundnohup ./script.sh & # immune to SIGHUP (survives shell logout)disown %1 # detach job from shell (no SIGHUP on exit)# nice / renice — CPU scheduling priority (-20=highest, 19=lowest)nice -n 10 ./heavy_script.sh # start with lower priorityrenice 5 -p 1234 # change running process
lsof and fuser
bash
lsof -i :80 # what's listening on port 80lsof -p 1234 # all files opened by PIDlsof -u www-data # files opened by a userlsof +D /var/log # all processes using files under /var/logfuser 80/tcp # PID using port 80fuser -k 80/tcp # kill it
Tip: When a deleted file is still consuming disk space, lsof | grep deleted shows which process still holds the file descriptor open. Restart that process to reclaim the space.
systemd is the init system on every modern Linux distro. It manages services, mounts, timers, sockets, and devices as a dependency graph. Everything that was once a shell script in /etc/init.d is now a unit file.
A systemd service cycles through states driven by systemctl start/stop/restart; enable/disable controls boot-time auto-start independently.
Essential systemctl Commands
bash
# Service lifecyclesystemctl start nginxsystemctl stop nginxsystemctl restart nginxsystemctl reload nginx # graceful config reload (if supported)systemctl status nginx # status + last 10 log lines# Enable/disable at bootsystemctl enable nginx # create symlink in wants/systemctl disable nginxsystemctl enable --now nginx # enable + start in one commandsystemctl is-enabled nginx # returns enabled/disabled/static# Inspectionsystemctl list-units --type=service --state=runningsystemctl list-units --state=failedsystemctl cat nginx # print unit filesystemctl show nginx # all propertiessystemctl list-dependencies nginx # tree of dependencies
Unit File Structure
Unit files live in /etc/systemd/system/ (your files) and /lib/systemd/system/ (package defaults). Your files take precedence.
# After creating/editing a unit filesystemctl daemon-reload # reload all unit filessystemctl enable --now myapp
journalctl — Reading Logs
bash
journalctl -u nginx # all logs for nginx unitjournalctl -u nginx -f # follow in real timejournalctl -u nginx --since "1 hour ago"journalctl -u nginx --since "2024-01-01" --until "2024-01-02"journalctl -u nginx -n 50 # last 50 linesjournalctl -p err -u nginx # only priority error and abovejournalctl -b # since last bootjournalctl -b -1 # previous bootjournalctl --disk-usage # how much disk the journal usesjournalctl --vacuum-size=200M # trim journal to 200 MB
systemd Timers
Timers replace cron with dependency tracking, logging via journal, and precise time expressions.
ini
# /etc/systemd/system/backup.timer[Unit]Description=Daily backup timer[Timer]OnCalendar=*-*-* 02:00:00 # every day at 02:00Persistent=true # run if missed while system was offRandomizedDelaySec=10min # spread load across fleet[Install]WantedBy=timers.target
bash
systemctl enable --now backup.timersystemctl list-timers # all timers + next trigger time
Tip: Always run systemctl daemon-reload after editing any unit file. Without it, systemd uses the cached version from memory and your changes are silently ignored.
User management is the foundation of Linux security. As a DevOps engineer, you’ll create service accounts for CI/CD pipelines, manage developer access, lock down compromised users, and configure sudo policies. Think of this like managing IAM roles in AWS — except it’s text files all the way down, and a single typo in /etc/sudoers can lock you out of your own server.
/etc/passwd and /etc/shadow
Every user account is defined in /etc/passwd. Despite the name, passwords are NOT stored here — they moved to /etc/shadow because /etc/passwd must be world-readable.
/etc/passwd has 7 colon-separated fields: username:x:UID:GID:GECOS:home_dir:login_shell
Caution: The -a in -aG is non-negotiable. Running sudo usermod -G docker alice (without -a) REPLACES all of Alice’s supplementary groups with just docker. She loses sudo, adm, dialout — everything. Always: usermod -aG. The -a stands for “append”.
passwd and chage — Password and Aging
bash
$ passwd # change YOUR own password$ sudo passwd alice # change alice's password as root$ sudo passwd -l olduser # lock account$ sudo passwd -e alice # force password change on next login$ sudo chage -l deploy # list aging policy$ sudo chage -M 90 -W 7 deploy # expire in 90 days, warn 7 days before$ sudo chage -d 0 alice # force password change at next login
su vs su -
su username — switches user, keeps your current environment.
su - username — switches AND runs a login shell, loading ~/.bashrc and resetting environment as if you logged in fresh.
Interview tip: “Difference between sudo su and su -?” su - requires the target user’s password and gives a full login shell. sudo su requires YOUR password (sudo authenticates the invoker), then runs su as root without a password prompt — but env vars may leak. Best practice: use sudo -i for a clean interactive root login shell.
sudo and /etc/sudoers
When you type sudo apt update: the setuid-root sudo binary reads /etc/sudoers, verifies your user is allowed, prompts for YOUR password (cached ~15 min), logs the action to /var/log/auth.log, then executes as the target user.
root ALL=(ALL:ALL) ALL # root can do anything%sudo ALL=(ALL:ALL) ALL # sudo group (Ubuntu default)%wheel ALL=(ALL) ALL # wheel group (RHEL)deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart nginx
Caution: NEVER edit /etc/sudoers with vim or nano directly. visudo validates syntax BEFORE saving. A single typo renders the file unparseable and breaks sudo for everyone — including root. Recovering requires single-user mode. Use sudo visudo or drop-in files in /etc/sudoers.d/ (chmod 0440).
Real DevOps Examples
bash
# Create a developer account with sudo + docker$ sudo useradd -m -s /bin/bash -G sudo,docker deploy && sudo passwd deploy# Lock a former employee's account immediately$ sudo passwd -l olduser && sudo usermod -L olduser && sudo usermod -s /usr/sbin/nologin olduser# Service account for CI/CD — no shell, no home login$ sudo useradd --system --shell /usr/sbin/nologin --home-dir /var/lib/jenkins jenkins# Audit who has sudo right now$ getent group sudo# Run a single command as another user$ sudo -u postgres psql
Real world: CI/CD Service Account — you NEVER want interactive login:
--system assigns a UID below 1000. nologin means even if SSH keys are stolen, no shell is granted. The locked password prevents su attacks. Defense-in-depth.
Groups are how Linux scales permissions beyond individual users. Instead of granting access to 20 developers one-by-one, you create a developers group, grant the group access, and add people to the group.
/etc/group Structure
Four colon-separated fields: groupname:x:GID:member1,member2,member3
System groups: 0–999. User groups: 1000+. The member list shows users whose supplementary group this is — a user’s primary group is set by the GID in /etc/passwd.
Primary vs Supplementary Groups
Primary group — exactly ONE per user, defined by GID in /etc/passwd. Files the user creates are owned by this group by default.
Supplementary groups — zero or many, listed in /etc/group. Grant additional permissions.
bash
$ id pushkaruid=1000(pushkar) gid=1000(pushkar) groups=1000(pushkar),27(sudo),998(docker),1100(developers)
Managing Groups
bash
$ sudo groupadd developers$ sudo groupadd -g 1500 deploy # specific GID$ sudo groupmod -n devops developers # rename$ sudo groupdel old-team # delete (refuses if it's anyone's primary group)$ sudo gpasswd -a alice developers # add member$ sudo gpasswd -d alice developers # remove member$ sudo gpasswd -A bob developers # make bob a group admin$ sudo usermod -aG docker pushkar # the standard "add to group" command
Caution:usermod -G without -a is destructive — it REPLACES all supplementary groups. Always usermod -aG. Mnemonic: “a is for ADD, without a it ANNIHILATES.”
newgrp — Switch Active Primary Group
If you were just added to a group, your current shell doesn’t know yet. newgrp spawns a subshell with the named group as primary:
bash
$ sudo usermod -aG docker pushkar$ docker ps # permission denied — current session predates the change$ newgrp docker # spawn subshell with docker group active$ docker ps # works!
Common System Groups
Group
Purpose
sudo
Debian/Ubuntu: members can use sudo
wheel
RHEL/CentOS equivalent of sudo
docker
Run docker without sudo (effectively root — be careful)
www-data
Apache/Nginx run as this user
adm
Read access to log files in /var/log/
systemd-journal
Read journalctl logs without sudo
disk
Raw disk access — DANGEROUS
Real world: Shared Deployment Directory with setgid — the setgid bit makes new files inherit the directory’s group:
Now any file created inside /opt/app is owned by the deploy group, not the creator’s primary group — so all team members can read/write each other’s files. This is how /srv/web/, /opt/app/, and shared CI artifact directories should be configured.
Tip: In Dockerfiles always apt-get update && apt-get install -y pkg in a single RUN layer. Splitting them means the cached update layer can have stale package lists when your install layer is rebuilt days later — causing “package not found” errors.
Disk management is one of the most frequent on-call problems in DevOps. “Server is down” usually means CPU pegged, OOM killer killing processes, or — most often — disk full. You must be fluent with df, du, partitioning tools, and /etc/fstab.
$ du -sh /var/log # total size of one directory$ du -h --max-depth=1 /var | sort -h # top-level breakdown, sorted$ du -hac --max-depth=2 . | sort -rh | head -20 # "what's eating my disk"$ sudo du -ah / 2>/dev/null | sort -rh | head -20 # whole system scan (slow)
gdisk is the GPT-only cousin of fdisk. Use it when fdisk complains about a GPT disk.
lsblk and blkid — Inspection
bash
$ lsblk # tree of block devices and mountpoints$ lsblk -f # with FS type and UUID — best "what's on this disk?" command$ sudo blkid # UUIDs for fstab entries
mount / umount
bash
$ sudo mount /dev/sdb1 /mnt/data$ sudo mount -t ext4 /dev/sdb1 /mnt/data # specify FS type$ sudo mount -o ro /dev/sdb1 /mnt/data # read-only$ sudo mount -o remount,rw / # remount root rw (rescue trick)$ sudo umount /mnt/data$ sudo umount -l /mnt/data # lazy unmount if "device is busy"$ findmnt # pretty tree view of all mounts
$ cat /etc/fstabUUID=a3f2e1b4-... / ext4 defaults,noatime 0 1UUID=b1c2d3e4-... /data xfs defaults,nofail 0 2tmpfs /tmp tmpfs defaults,size=2G,noexec 0 0/swapfile none swap sw 0 0nfs.internal:/exports/shared /mnt/nfs nfs defaults,_netdev,soft,timeo=30 0 0$ sudo mount -a # mount everything in fstab — TEST before reboot!$ sudo findmnt --verify # sanity-check fstab without mounting
The pass column controls fsck order at boot: 0=skip, 1=root only, 2=other filesystems.
Caution: A typo in /etc/fstab (wrong UUID, missing mountpoint, bad option) drops the system into emergency mode on next reboot. Always test with sudo mount -a after editing. For non-critical mounts add nofail; for network filesystems add _netdev. To recover from a boot hang: single-user mode, mount -o remount,rw /, fix fstab, reboot.
Mounting by UUID (Best Practice)
Device names like /dev/sdb can change across reboots. UUIDs are baked into the filesystem and never change. Always use UUIDs in fstab.
bash
$ sudo blkid /dev/sdb1/dev/sdb1: UUID="b1c2d3e4-..." TYPE="xfs"$ echo 'UUID=b1c2d3e4-... /data xfs defaults,nofail 0 2' | sudo tee -a /etc/fstab
LVM (Logical Volume Manager) adds a flexible layer between partitions and filesystems: PV (Physical Volume), VG (Volume Group), LV (Logical Volume). You can resize LVs without unmounting, take snapshots, and span multiple disks. Ubuntu Server defaults to LVM.
Real world: “No space left on device” but df shows free space? You’re out of inodes, not bytes. Every file/dir consumes one inode regardless of size. Diagnose with df -i — if any IUse% is 100%, that’s it. Common culprits: session files, mail queues, cache dirs with millions of tiny files. Fix: delete the small files; xfs allocates inodes dynamically and avoids this entirely.
Interview tip: “Disk is full, what do you do?” (1) df -h — which FS, how full. (2) df -i — rule out inode exhaustion. (3) du -h --max-depth=1 / | sort -h — find the heavy directory. (4) Check usual suspects: /var/log, /var/lib/docker (docker system prune), journal (journalctl --vacuum-size=500M). (5) Check deleted-but-open files: sudo lsof | grep deleted. (6) Extend the volume (cloud: resize EBS → growpart → resize2fs/xfs_growfs).
A filesystem is the contract between raw blocks on a disk and the files you see. Choosing the right one matters: ext4 for general Linux, xfs for big files and parallel I/O, btrfs/zfs for snapshots, tmpfs for ephemeral RAM-backed storage, overlayfs for Docker layers.
The Main Linux Filesystems
ext4 — Default on Ubuntu/Debian. Mature, journaled, max file 16 TB. Resize online with resize2fs.
xfs — Default on RHEL/Rocky/Amazon Linux. Excellent for large files and parallel I/O. Allocates inodes dynamically. Can grow online (xfs_growfs) but cannot shrink.
$ sudo mkfs.ext4 /dev/sdb1$ sudo mkfs.ext4 -L data -m 1 /dev/sdb1 # label "data", 1% reserved for root$ sudo mkfs.xfs -L data -f /dev/sdb1$ sudo mkfs.vfat -F 32 -n USB /dev/sdb1# ext4 grow (online or offline)$ sudo resize2fs /dev/sdb1 # grow to fill partition# ext4 shrink (MUST unmount + fsck first)$ sudo umount /data && sudo e2fsck -f /dev/sdb1 && sudo resize2fs /dev/sdb1 20G# xfs grow ONLY (online, pass MOUNTPOINT not device)$ sudo xfs_growfs /data
fsck — Filesystem Integrity
Caution: NEVER run fsck on a mounted filesystem — it corrupts it. Always unmount first, or boot from rescue media for the root filesystem.
bash
$ sudo umount /dev/sdb1$ sudo fsck.ext4 -f /dev/sdb1 # -f forces check even if "clean"$ sudo fsck -y /dev/sdb1 # auto-answer "yes" to repairs$ sudo xfs_repair /dev/sdb1 # xfs uses xfs_repair, not fsck$ sudo xfs_repair -n /dev/sdb1 # dry run — report only
Inodes — Deep Dive
An inode is a fixed-size record storing all metadata for a file except its name: type, permissions, owner UID/GID, size, timestamps (atime/mtime/ctime), link count, and pointers to data blocks. The filename lives in the directory entry — a (name → inode number) mapping. This single design choice explains hard links, symlinks, rename, and “deleted but still open” files.
bash
$ stat /etc/passwd # full inode metadata$ ls -i /etc/passwd # show inode number$ df -i / # inode usage for a filesystem
File operations in terms of inodes:
Rename / mv (same FS): directory entry name changes; inode untouched — instant even for huge files.
Hard link (ln): a second directory entry pointing to the same inode. Link count increments. Cannot cross filesystems. Cannot link directories.
Symbolic link (ln -s): a tiny file with its own inode whose content is a path string. Can cross filesystems, can point to directories, can dangle.
Delete (rm): removes the directory entry and decrements link count. Inode and data freed only when link count reaches 0 and no process has the file open.
Interview tip: Hard link vs Symlink (inode perspective). Hard link: a second directory entry pointing to the same inode — both names are first-class, same FS only, can’t link directories, survives deletion of one name while link count > 0. Symlink: a separate file whose data is a textual path, resolved at access time, can cross filesystems, can link directories, dangles if target removed. In ls -l: a hard link looks like a normal file with link count > 1; a symlink shows l and -> target.
Real world: Resizing an EBS volume on EC2 — three steps, miss one and df shows no change. (1) Modify the EBS volume size in the AWS console/CLI. (2) Grow the partition: sudo growpart /dev/nvme0n1 1. (3) Grow the filesystem: ext4 sudo resize2fs /dev/nvme0n1p1, xfs sudo xfs_growfs -d /. Entire process is online — no reboot, no unmount.
Networking knowledge separates engineers who can debug production incidents from those who can’t. These tools let you inspect interfaces, test connectivity, trace traffic, and diagnose service failures.
ip — Modern Network Configuration
bash
# Addressesip addr show # all interfaces and IPsip addr show eth0 # specific interfaceip addr add 192.168.1.10/24 dev eth0 # add IP (temporary)ip addr del 192.168.1.10/24 dev eth0# Routesip route show # routing tableip route add default via 192.168.1.1 # add default gatewayip route get 8.8.8.8 # which route is used to reach an IP# Linksip link showip link set eth0 upip link set eth0 down
ss — Socket Statistics (replaces netstat)
bash
ss -tlnp # TCP (-t) listening (-l) numeric (-n) with process (-p)ss -ulnp # UDP listeningss -tnp # all established TCP connectionsss -s # summary statistics# Common patternsss -tlnp | grep :80 # what's on port 80ss -tnp state ESTABLISHED # only established connectionsss -tnp dst 10.0.0.5 # connections to a specific host
dig — DNS Diagnostics
bash
dig example.com # A record (default)dig example.com MX # mail exchange recordsdig example.com NS # nameserversdig example.com TXT # TXT records (SPF, DKIM, etc.)dig @8.8.8.8 example.com # query specific DNS serverdig +short example.com # just the IPsdig -x 93.184.216.34 # reverse DNS lookupdig +trace example.com # trace full resolution chain
curl — HTTP Testing
bash
curl https://api.example.com/health # GET requestcurl -I https://example.com # headers only (HEAD)curl -X POST -H "Content-Type: application/json" \ -d '{"key":"value"}' https://api.example.com/datacurl -o /tmp/file.tar.gz https://example.com/file.tar.gz # downloadcurl -L https://short.url/abc # follow redirectscurl -w "%{http_code} %{time_total}s\n" -o /dev/null -s https://example.com # timingcurl --resolve example.com:443:10.0.0.5 https://example.com # test before DNS propagation
Other Essential Network Tools
bash
# ping / traceroute / mtrping -c 4 8.8.8.8traceroute 8.8.8.8mtr 8.8.8.8 # real-time traceroute (apt install mtr)# nc (netcat) — the network Swiss Army knifenc -zv host 22 # port scan / test connectivitync -l 9000 # listen on port 9000echo "PING" | nc -u host 514 # UDP test# tcpdump — packet capturesudo tcpdump -i eth0 port 80 -w capture.pcap # capture to filesudo tcpdump -i any host 10.0.0.5 and tcp # filter by host+protocolsudo tcpdump -r capture.pcap -A # read and print ASCII# watch — repeat a commandwatch -n 2 ss -tlnp # refresh every 2 seconds
Tip:ss -tlnp is your first command when a service “can’t start” — it shows exactly which ports are already in use and by which PID. Then systemctl status that PID to find the conflict.
Commands you can run, but understanding the actual configuration files is the next level. This section covers Ubuntu 24.04’s real config files — production servers, EC2 instances, on-prem bare metal all apply.
/etc/hosts — Local Hostname Resolution
Consulted before DNS (per nsswitch.conf order). The lifeline for local development and quick overrides.
Caution: On Ubuntu 24.04, /etc/resolv.conf is usually a symlink to /run/systemd/resolve/stub-resolv.conf managed by systemd-resolved. Direct edits get overwritten. Use Netplan or resolvectl instead:
bash
$ resolvectl status$ sudo resolvectl dns eth0 1.1.1.1 8.8.8.8
$ sudo netplan try # test for 120s, auto-rollback if SSH dies$ sudo netplan apply # commit changes
Caution: YAML is strict — 2-space indentation, NO TABS. Wrong indent = network down. Always use netplan try first on remote boxes.
NetworkManager (nmcli) — Desktop/Laptop
bash
$ nmcli device status$ nmcli con show # all profiles$ nmcli dev wifi connect "MySSID" password "secret"$ nmcli con mod eth0 ipv4.addresses 10.0.1.50/24 ipv4.gateway 10.0.1.1 ipv4.method manual
systemd-networkd and Hostname
Cloud servers usually use systemd-networkd (lightweight). Netplan generates its config under /run/systemd/network/.
Common bonding modes: active-backup (failover), balance-rr (round robin), 802.3ad (LACP, needs switch support). VLANs tag one physical NIC into multiple logical networks (802.1Q).
Note: Cloud reality check: 95% of production cloud instances (AWS EC2, GCP, Azure VM) take their IP via DHCP / cloud-init — static IPs are rarely set at the OS level because VPC subnets and ENIs handle assignment, and auto-scaling groups use dynamic IPs. But for on-prem, hybrid, edge devices, bastion hosts, and NFS servers, static IP is a must. Know both perspectives for interviews.
Interview tip: “Netplan vs NetworkManager?” — Netplan is a YAML frontend. The real backend is NetworkManager (desktop) or systemd-networkd (server). Ubuntu cloud images default to renderer: networkd.
SSH is how you access every cloud VM, container host, and remote server. Beyond basic login, it gives you encrypted tunnels, file transfer, and agent forwarding. Misconfiguring it creates security holes; understanding it properly lets you build zero-trust access patterns.
SSH key auth: the client signs a server-issued challenge with its private key; the server verifies using the stored public key — no password ever crosses the network.
Key-Based Authentication
bash
# Generate a key pair (Ed25519 is faster and more secure than RSA)ssh-keygen -t ed25519 -C "pushkar@work" -f ~/.ssh/id_ed25519# Copy public key to serverssh-copy-id -i ~/.ssh/id_ed25519.pub user@server# or manually:cat ~/.ssh/id_ed25519.pub | ssh user@server 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys'# Permission requirements (SSH will refuse with wrong perms)chmod 700 ~/.sshchmod 600 ~/.ssh/id_ed25519 # private keychmod 644 ~/.ssh/id_ed25519.pub # public keychmod 600 ~/.ssh/authorized_keyschmod 644 ~/.ssh/known_hosts# Start ssh-agent and load keyeval "$(ssh-agent -s)"ssh-add ~/.ssh/id_ed25519ssh-add -l # list loaded keys
Note: The public key body (the long string after ssh-ed25519) is base64-encoded binary. If you ever need to inspect or transform an encoded blob like this — or a base64 secret in a Kubernetes manifest — the Base64 Encoder/Decoder is handy.
~/.ssh/config
The client config file eliminates typing long hostnames and options every time:
sshconfig
# ~/.ssh/configHost bastion HostName 203.0.113.10 User ec2-user IdentityFile ~/.ssh/aws-key.pem ServerAliveInterval 60Host prod-db HostName 10.0.1.50 # private IP User ubuntu ProxyJump bastion # tunnel through bastion IdentityFile ~/.ssh/id_ed25519Host dev-* User developer IdentityFile ~/.ssh/dev-key StrictHostKeyChecking no # only for throwaway dev VMs UserKnownHostsFile /dev/null
# Local port forwarding: access remote service on your local portssh -L 5432:db-host:5432 bastion # connect to postgresql via bastion# Now: psql -h localhost -p 5432# Remote port forwarding: expose local port on the remote serverssh -R 8080:localhost:3000 server # server:8080 → your localhost:3000# Dynamic (SOCKS proxy): route browser traffic through the tunnelssh -D 1080 bastion # configure browser to use SOCKS5 localhost:1080# Background / non-interactivessh -fNL 5432:db:5432 bastion # -f = background, -N = no command
# /etc/ssh/sshd_config (key hardening settings)Port 2222 # non-standard port (minor obscurity)PermitRootLogin no # never allow root loginPasswordAuthentication no # key-only (most important setting)PubkeyAuthentication yesAuthorizedKeysFile .ssh/authorized_keysAllowUsers pushkar deployMaxAuthTries 3LoginGraceTime 20ClientAliveInterval 300ClientAliveCountMax 2X11Forwarding noAllowTcpForwarding yes # needed for tunnels
bash
sudo sshd -t # test config syntaxsudo systemctl reload sshd # apply without dropping connections
Tip: Always test your sshd config with sshd -t and keep an existing SSH session open while applying changes. If you lock yourself out of a cloud VM, you’ll need console access or a rescue snapshot.
Bash (Bourne Again SHell) is the default interactive shell on Ubuntu, Debian, RHEL derivatives, and most Linux distros. As a DevOps engineer, bash is your daily driver — every CI pipeline, every Dockerfile RUN, every cron job, every systemdExecStart is some flavour of shell glue.
Note: Ubuntu’s /bin/sh is actually dash (Debian Almquist Shell) — a POSIX-compliant shell with no bash extensions. Scripts written assuming bash but invoked as sh script.sh will fail on [[ ]], arrays, or $(()) arithmetic. Always be explicit about your shebang.
The Shebang and Running Scripts
bash
#!/bin/bash # Absolute path — works on virtually every Linux box#!/usr/bin/env bash # Looks up bash via PATH — portable to macOS/BSD
Command
Shell
Needs +x?
Variables persist in parent?
./script.sh
Subshell (from shebang)
Yes
No
bash script.sh
New bash subshell
No
No
source script.sh or . script.sh
Current shell
No
Yes
Variables
The cardinal rule: no spaces around the = sign. Always quote when expanding to handle spaces and globs safely.
bash
NAME="Pushkar"GREETING="Hello, $NAME"echo "${GREETING}!" # braces disambiguate variable name# Parameter expansionecho "${UNSET_VAR:-default}" # use default if unset/emptyecho "${MAYBE:=set_now}" # assign default if unset (sticky)echo "${REQUIRED:?must be set}" # error and exit if unsetecho "${NAME:+exists}" # expand to 'exists' only if setFILE="backup.tar.gz"echo "${#FILE}" # 13 (length)echo "${FILE%.gz}" # backup.tar (strip shortest suffix)echo "${FILE%%.*}" # backup (strip longest suffix)echo "${FILE/tar/zip}" # backup.zip.gz (replace first match)
Positional Parameters
Variable
Meaning
$0
Script name
$1 … $9
First nine positional arguments
$#
Number of arguments
"$@"
All args, each preserved as one quoted word
$$
PID of current shell
$?
Exit code of last command (0 = success)
$!
PID of last background process
Bash Strict Mode
Put this at the top of every serious script:
bash
set -euo pipefailIFS=$'\n\t'
Flag
What it does
-e (errexit)
Exit immediately when any command returns non-zero
-u (nounset)
Treat references to unset variables as errors
-o pipefail
Pipeline exit code is the rightmost non-zero status
IFS=$'\n\t'
Restrict word-splitting to newlines/tabs
Arithmetic and Input
bash
echo $((2 + 3 * 4)) # 14 — recommended formx=5((x++)) # x is now 6echo "scale=2; 7/3" | bc # floating point: 2.33read -p "Your name: " nameread -s -p "Password: " pass # silent inputread -t 10 -p "Quick: " ans # 10-second timeout
Tip: Use source only for shell configuration files (.bashrc, env loaders). Use ./script.sh for everything else — sourcing arbitrary scripts can pollute your interactive shell if they call exit or set -e.
Control flow in bash looks unfamiliar at first — then, fi, do, done, double brackets. It’s because the shell parses commands by whitespace, so keywords need clear separators. Once you internalize the patterns, complex automation becomes muscle memory.
if / elif / else
bash
if [[ "$USER" == "root" ]]; then echo "Running as root"elif [[ "$USER" == "pushkar" ]]; then echo "Hi Pushkar"else echo "Unknown user: $USER"fi# if on any command's exit codeif grep -q "ERROR" /var/log/syslog; then echo "Errors found in syslog"fiif systemctl is-active --quiet nginx; then echo "nginx is running"fi
Tip: Use [[ ]] in bash scripts always. Use [ ] only when writing a portable /bin/sh script. Inside [[ ]], do not quote the right side of =~ — quoting turns the regex into a literal string.
Comparison Operators
Integer ([[ ]] or (( )))
String ([[ ]])
Meaning
-eq
==
Equal
-ne
!=
Not equal
-lt
<
Less than
-gt
>
Greater than
—
-z STR
String is empty
—
-n STR
String is non-empty
—
STR =~ REGEX
String matches ERE regex
File Test Operators
Operator
True if…
-e FILE
File exists (any type)
-f FILE
Regular file
-d DIR
Directory
-r FILE
Readable
-w FILE
Writable
-x FILE
Executable
-s FILE
Exists and non-empty
case Statement
bash
case "$action" in start) echo "Starting..." ;; stop|halt) echo "Stopping..." ;; *.log) echo "That looks like a log file" ;; *) echo "Unknown action: $action" exit 1 ;;esac
Loops
bash
# for-in with a listfor env in dev staging prod; do echo "Deploying to $env"done# Brace expansion rangefor i in {1..10}; do echo "loop $i"; done# Glob iteration — always quote the variablefor f in /var/log/*.log; do [[ -f "$f" ]] || continue echo "Processing $f"done# C-style forfor ((i=0; i<5; i++)); do echo "index $i"done# while — read line by line (safest pattern)while IFS= read -r line; do echo "> $line"done < /etc/hosts# until with retryattempts=0until curl -sf https://api.example.com/health > /dev/null; do ((attempts++)) [[ $attempts -ge 5 ]] && { echo "giving up"; exit 1; } sleep 2done
Warning:for f in $(ls *.log) is a classic anti-pattern. If filenames have spaces, they get split. Use for f in *.log (glob directly) instead.
Ten Practical Examples
1. Retry with exponential backoff:
bash
URL="$1"MAX_TRIES=5DELAY=1for ((try=1; try<=MAX_TRIES; try++)); do if curl -sfL "$URL" -o /tmp/out; then echo "Success on attempt $try" exit 0 fi echo "Attempt $try failed, sleeping ${DELAY}s..." sleep "$DELAY" DELAY=$((DELAY * 2))doneecho "All $MAX_TRIES attempts failed" >&2exit 1
2. Wait for a port to be open (Docker Compose pattern):
bash
HOST=dbPORT=5432until (echo > /dev/tcp/"$HOST"/"$PORT") 2>/dev/null; do echo "Waiting for $HOST:$PORT..." sleep 1doneecho "$HOST:$PORT is up"
3. Read a CSV line-by-line:
bash
while IFS=, read -r name email role; do [[ "$name" == "name" ]] && continue # skip header echo "User: $name <$email> ($role)"done < users.csv
4. Interactive menu:
bash
while true; do cat <<MENU1) Show uptime2) Show diskq) QuitMENU read -p "Choice: " ch case "$ch" in 1) uptime ;; 2) df -h ;; q|Q) break ;; *) echo "Invalid choice" ;; esacdone
Interview tip: A common screening question is “write a script to find the 5 largest files in a directory tree.” Answer: find /path -type f -printf '%s %p\n' | sort -rn | head -5.
Once your scripts grow past 50 lines, you need functions, arrays, and disciplined error handling. This section covers the constructs that separate “shell script someone hacked together” from “production automation.”
Inside a function, $1, $2, $@, $# refer to the function’s arguments — not the script’s. Bash return N only sets the exit status (0-255); to return data, echo it and capture with $():
bash
get_kernel() { uname -r; } # write to stdoutis_root() { [[ $EUID -eq 0 ]]; } # last command's exit status IS the returnKERNEL=$(get_kernel)if is_root; then echo "Running as root"; fi
Local Variables
By default all bash variables are global. local (only valid inside functions) keeps a variable scoped:
bash
counter=0bump() { local counter=100 # shadows the global counter=$((counter + 1)) echo "inside: $counter"}bump # inside: 101echo "outside: $counter" # outside: 0
Caution:local var=$(some_cmd)masks the exit code of some_cmd because local itself succeeds. Under set -e, failures silently disappear. Safe idiom: declare first, assign second.
bash
local varvar=$(some_cmd) # now set -e catches a failure here
Arrays
bash
servers=(web01 web02 db01 cache01)echo "${servers[0]}" # web01 (zero-indexed)echo "${servers[@]}" # all elements as separate wordsecho "${#servers[@]}" # 4 (length)echo "${!servers[@]}" # 0 1 2 3 (indices)servers+=(db02 db03) # appendfor s in "${servers[@]}"; do echo "Server: $s"; doneecho "${servers[@]:1:2}" # slice — from index 1, take 2mapfile -t users < <(cut -d: -f1 /etc/passwd) # read command output into array
Associative Arrays (Hash Maps)
Bash 4+ via declare -A. Ubuntu 24.04 ships bash 5.2:
bash
declare -A env_urlenv_url[dev]="https://dev.example.com"env_url[prod]="https://example.com"echo "${env_url[prod]}"for key in "${!env_url[@]}"; do printf '%-10s -> %s\n' "$key" "${env_url[$key]}"doneif [[ -v env_url[qa] ]]; then echo "qa defined"; else echo "qa missing"; fi
Exit Codes and Convention
Code
Convention
0
Success
1
General error (catch-all)
2
Misuse / invalid arguments
126
Command found but not executable
127
Command not found
128 + N
Killed by signal N (130 = Ctrl-C, 137 = SIGKILL)
255
Exit code out of range
Note: Follow the exit-code convention. CI systems (Jenkins, GitLab, GitHub Actions) only know “did the job succeed or fail” by exit code. If your script exit 0s on error, the pipeline shows green while the deploy actually failed. Always non-zero exit on any failure path.
trap — Cleanup on Exit/Signal
bash
#!/usr/bin/env bashset -euo pipefailTMPDIR=$(mktemp -d)cleanup() { local rc=$? echo "Cleaning up (exit=$rc)..." rm -rf "$TMPDIR" return "$rc"}trap cleanup EXIT # EXIT = whenever the script endstrap 'echo "Interrupted"; exit 130' INT TERM # INT = Ctrl-C, TERM = killecho "working in $TMPDIR"
getopts — Argument Parsing
bash
while getopts "e:v:dh" opt; do case "$opt" in e) ENV="$OPTARG" ;; v) VERSION="$OPTARG" ;; d) DRY_RUN=1 ;; h) usage ;; *) usage ;; esacdoneshift $((OPTIND - 1))[[ -z "$ENV" || -z "$VERSION" ]] && { echo "-e and -v required" >&2; exit 2; }
Heredocs and Here-Strings
bash
NAME=Pushkarcat <<EOFHello, $NAMEToday is $(date +%F)EOFcat <<'EOF'Literal $NAME and $(date) — nothing expands hereEOFgrep -E '^[A-Z]' <<< "$LINE" # here-string — feed a single string as stdin
Tip: Install shellcheck (sudo apt install shellcheck) and run it on every script before committing. It catches ~80% of bash bugs — quoting issues, unset variables, deprecated syntax, common SC2086 word-splitting traps. Hook it into your CI.
Below are complete, working scripts you would actually run in production. Each uses strict mode, functions, error handling, and proper logging. They cover the automation patterns you will encounter repeatedly as a DevOps engineer.
Real world: Every script here is based on patterns used in production: locking, retries, Slack alerts, S3 upload, rollback on health-check fail. Tweak the paths and service names for your environment.
Script 1: server_health_check.sh
One-shot dashboard of CPU, memory, disk, and top processes. Exits non-zero if any threshold is breached so it can drive alerting from cron.
bash
#!/usr/bin/env bash# server_health_check.sh — quick host health snapshotset -euo pipefailIFS=$'\n\t'CPU_WARN=80MEM_WARN=85DISK_WARN=80EXIT_CODE=0hr() { printf '%s\n' "------------------------------------------------------------"; }header() { hr printf '%s @ %s\n' "$(hostname -f)" "$(date +'%F %T %Z')" printf 'kernel: %s uptime:%s\n' "$(uname -r)" "$(uptime -p)" hr}cpu_section() { local idle1 total1 idle2 total2 usage read -r _ a b c idle1 _ < /proc/stat total1=$((a + b + c + idle1)) sleep 1 read -r _ a b c idle2 _ < /proc/stat total2=$((a + b + c + idle2)) usage=$(( 100 * ((total2 - total1) - (idle2 - idle1)) / (total2 - total1) )) printf 'CPU usage: %d%% load: %s\n' "$usage" "$(cut -d' ' -f1-3 /proc/loadavg)" (( usage > CPU_WARN )) && { echo " WARN: cpu > ${CPU_WARN}%"; EXIT_CODE=1; }}mem_section() { local total used pct read -r _ total used _ < <(free -m | awk '/^Mem:/') pct=$(( 100 * used / total )) printf 'Memory: %d / %d MiB (%d%%)\n' "$used" "$total" "$pct" (( pct > MEM_WARN )) && { echo " WARN: memory > ${MEM_WARN}%"; EXIT_CODE=1; }}disk_section() { printf 'Disks:\n' df -hP -x tmpfs -x devtmpfs | awk 'NR>1 { printf " %-25s %5s used of %-5s on %s\n", $1, $5, $2, $6 }' while read -r usage mount; do (( usage > DISK_WARN )) && { echo " WARN: $mount > ${DISK_WARN}% (${usage}%)"; EXIT_CODE=1; } done < <(df -P -x tmpfs -x devtmpfs | awk 'NR>1 { gsub("%","",$5); print $5, $6 }')}main() { header cpu_section mem_section disk_section hr exit "$EXIT_CODE"}main "$@"
Script 2: deploy_app.sh
Git pull, npm build, systemd restart, health check, and automatic rollback on failure. Uses a lockfile to prevent concurrent deploys.
bash
#!/usr/bin/env bash# deploy_app.sh — pull, build, restart, verify, rollback on failset -euo pipefailAPP_DIR="/opt/myapp"SERVICE="myapp"HEALTH_URL="http://127.0.0.1:3000/health"LOCK="/var/run/deploy_${SERVICE}.lock"HEALTH_TRIES=20log() { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }die() { log "FATAL: $*" >&2; exit 1; }# Single-instance lockexec 9> "$LOCK"flock -n 9 || die "another deploy is in progress"cd "$APP_DIR" || die "$APP_DIR not found"OLD_SHA=$(git rev-parse HEAD)log "current SHA: $OLD_SHA"git fetch --quiet origingit reset --hard origin/mainNEW_SHA=$(git rev-parse HEAD)[[ "$OLD_SHA" == "$NEW_SHA" ]] && { log "already at latest, no-op"; exit 0; }log "new SHA: $NEW_SHA"rollback() { log "ROLLING BACK to $OLD_SHA" git reset --hard "$OLD_SHA" npm ci --omit=dev && npm run build || true systemctl restart "$SERVICE" die "deploy failed, rolled back"}npm ci --omit=dev || rollbacknpm run build || rollbacksystemctl restart "$SERVICE" || rollbackfor ((i=1; i<=HEALTH_TRIES; i++)); do if curl -fsS -m 3 "$HEALTH_URL" > /dev/null; then log "healthy after $i probe(s)" log "deploy OK: $OLD_SHA -> $NEW_SHA" exit 0 fi sleep 2donerollback
Script 3: service_monitor.sh
Watchdog for systemd units. Restarts down services and posts to Slack on incident. Designed for a cron-every-minute or systemd timer.
bash
#!/usr/bin/env bash# service_monitor.sh — auto-restart services and Slack on failureset -euo pipefailSERVICES=(nginx postgresql redis-server docker)MAX_RESTART=3SLACK_URL="${SLACK_WEBHOOK_URL:-}"HOST=$(hostname -f)log() { printf '[%s] %s\n' "$(date +%FT%T)" "$*"; }slack() { [[ -z "$SLACK_URL" ]] && { log "SLACK_WEBHOOK_URL not set, skipping notify"; return; } local msg="$1" curl -fsS -m 5 -X POST -H 'Content-Type: application/json' \ --data "{\"text\":\":rotating_light: [$HOST] $msg\"}" \ "$SLACK_URL" > /dev/null || log "WARN: slack post failed"}check_service() { local svc="$1" if systemctl is-active --quiet "$svc"; then log "$svc: OK"; return 0 fi log "$svc: DOWN — attempting restart" local attempt=1 while ((attempt <= MAX_RESTART)); do systemctl restart "$svc" || true sleep $((attempt * 2)) if systemctl is-active --quiet "$svc"; then log "$svc: recovered on attempt $attempt" slack "$svc was DOWN, restarted successfully on attempt $attempt" return 0 fi ((attempt++)) done log "$svc: FAILED after $MAX_RESTART attempts" slack "$svc is DOWN and failed to restart after $MAX_RESTART attempts" return 1}failures=0for s in "${SERVICES[@]}"; do check_service "$s" || ((failures++))doneexit "$failures"
Script 4: log_rotate.sh
Minimal logrotate alternative for ad-hoc log directories. Gzips logs older than N days, deletes anything older than M days. Safe for cron.
Parse nginx access logs for top IPs, top URLs, and 4xx/5xx counts. Drop-in incident-response tool.
bash
#!/usr/bin/env bash# log_analysis.sh — top IPs, URLs, status code breakdown from nginx logsset -euo pipefailLOG="${1:-/var/log/nginx/access.log}"TOP_N="${2:-10}"[[ -r "$LOG" ]] || { echo "cannot read $LOG" >&2; exit 1; }if [[ "$LOG" == *.gz ]]; then READ="zcat"; else READ="cat"; fiecho "=== Summary of $LOG ==="TOTAL=$($READ "$LOG" | wc -l)echo "Total requests: $TOTAL"echoecho "--- Top $TOP_N IPs ---"$READ "$LOG" | awk '{ print $1 }' | sort | uniq -c | sort -rn | head -n "$TOP_N"echoecho "--- Top $TOP_N URLs ---"$READ "$LOG" | awk '{ print $7 }' | sort | uniq -c | sort -rn | head -n "$TOP_N"echoecho "--- Status code breakdown ---"$READ "$LOG" | awk '{ code=$9 counts[code]++ if (code+0 >= 400 && code+0 < 500) c4xx++ else if (code+0 >= 500) c5xx++}END { for (c in counts) printf "%5d %s\n", counts[c], c | "sort -rn" close("sort -rn") printf "\nTotal 4xx: %d\nTotal 5xx: %d\n", c4xx+0, c5xx+0}'
Key Patterns to Internalize
Functions and libraries: Split helpers into lib/common.sh and source it. Define log(), warn(), and die() functions in every serious script.
trap for cleanup:
bash
TMPDIR=$(mktemp -d)cleanup() { rm -rf "$TMPDIR"; }trap cleanup EXITtrap 'echo "Interrupted"; exit 130' INT TERM
getopts for argument parsing:
bash
while getopts "e:v:dh" opt; do case "$opt" in e) ENV="$OPTARG" ;; v) VERSION="$OPTARG" ;; d) DRY_RUN=1 ;; h) usage ;; *) usage ;; esacdone
Interview tip: When asked “design a deploy script,” walk through: (1) idempotency, (2) lockfile to prevent concurrent runs, (3) capture old state for rollback, (4) health check after restart, (5) automatic rollback path, (6) logging and notifications. The deploy_app.sh above hits all six — internalize the pattern.
Warning: Never write rm -rf "$VAR/" without set -u and an explicit non-empty check. Empty variable + slash = root-level delete. Always: [[ -n "${DIR:-}" && -d "$DIR" ]] || die "bad DIR" before any destructive operation.
For a DevOps engineer, logs are the single source of truth. When a production app misbehaves at 3 AM, you don’t guess — you grep. On a Linux server you’ll be living inside /var/log and journalctl. Master these and you can debug anything.
Note: Mental model — Traditional Linux services write plain-text files to /var/log. Modern systemd services write to the binary journal, queried with journalctl. On Ubuntu 24.04 you have both.
Key Log Files — Cheat Sheet
Path
Distro
What’s in it
/var/log/syslog
Debian/Ubuntu
General system events — almost everything ends up here
/var/log/messages
RHEL/Rocky
RHEL equivalent of syslog
/var/log/auth.log
Debian/Ubuntu
SSH logins, sudo, PAM, su — every auth event
/var/log/secure
RHEL
RHEL equivalent of auth.log
/var/log/kern.log
Both
Kernel ring buffer messages (drivers, OOM kills)
/var/log/apt/history.log
Debian/Ubuntu
Every apt install/remove/upgrade
/var/log/dpkg.log
Debian/Ubuntu
Low-level package install/remove
/var/log/nginx/access.log
All
Nginx HTTP requests
/var/log/nginx/error.log
All
Nginx errors, upstream failures
/var/log/postgresql/postgresql-*.log
Ubuntu
PostgreSQL queries, errors, checkpoints
/var/log/cloud-init.log
Cloud VMs
EC2/GCE first-boot user-data output
journalctl — The systemd Journal
bash
$ journalctl -e # jump to end (most recent)$ journalctl -r # reverse — newest first$ journalctl -n 50 # last 50 lines$ journalctl -f # follow (like tail -f)$ journalctl -u nginx # only nginx.service$ journalctl -u nginx -f # follow nginx live$ journalctl --since "1 hour ago"$ journalctl --since today$ journalctl -p err # priority err or worse$ journalctl -b # this boot only$ journalctl -b -1 # previous boot$ journalctl -k # kernel only (dmesg-style)$ journalctl --disk-usage$ sudo journalctl --vacuum-time=7d # keep only last 7 days$ sudo journalctl --vacuum-size=500M # cap at 500 MB
Priority levels (numeric and named both work with -p): 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug.
logrotate — Stops Disks From Filling Up
Logs grow forever unless rotated. logrotate runs daily via /etc/cron.daily/logrotate and reads /etc/logrotate.conf plus per-service files in /etc/logrotate.d/.
bash
# /etc/logrotate.d/myapp/var/log/myapp/*.log { daily rotate 14 # keep 14 old copies size 100M # OR rotate when file > 100MB compress delaycompress missingok notifempty create 0640 myapp adm sharedscripts postrotate systemctl reload myapp.service > /dev/null 2>&1 || true endscript}
bash
$ sudo logrotate -d /etc/logrotate.d/myapp # dry run$ sudo logrotate -f /etc/logrotate.d/myapp # force run now
Tip: Without copytruncate or a postrotate reload, long-running daemons keep writing to the old (renamed) file descriptor — and you wonder why rotation did nothing. Always reload the service or use copytruncate.
rsyslog and Centralized Logging
rsyslog reads kernel and app messages and writes them to /var/log/* (or ships them to a remote server). Syntax: facility.priority destination.
bash
auth,authpriv.* /var/log/auth.log*.* @@logserver.internal:514 # TCP to a central log server$ logger -p local0.notice "Deploy v2.4.1 started by $USER" # write to syslog
One server: grep is fine. Twenty servers: you need an aggregator.
Stack
Components
Best for
ELK / Elastic
Filebeat → Logstash → Elasticsearch → Kibana
Full-text search, rich dashboards
Grafana Loki
Promtail → Loki → Grafana
Cheap, label-based
AWS CloudWatch Logs
CW agent → CW Logs → Logs Insights
AWS-native, zero infra
Real DevOps Log Examples
bash
$ awk '$9 ~ /^5/ {print $9}' /var/log/nginx/access.log | sort | uniq -c # 5xx breakdown$ awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head # top IPs$ sudo dmesg -T | grep -i "killed process" # was the system OOM-killed?$ systemctl --failed # failed services$ sudo du -sh /var/log/* | sort -h | tail -20 # disk usage per log dir
Real world: Tracing an SSH brute-force on a fresh VPS — auth.log balloons. Count failures and group by attacker IP:
Mitigation: install fail2ban, set PasswordAuthentication no in sshd_config, move SSH off port 22, or restrict via ufw allow from <your-IP> to any port 22.
Monitoring is the second pillar after logs. When a customer says “the site is slow,” you need to know within 30 seconds whether it’s CPU, memory, disk I/O, or network.
The three numbers are the average run-queue length over 1, 5, and 15 minutes. Load 1.42 on a 4-core box ≈ 35% utilized. Load 8.0 on 4 cores = processes queueing. 1-min > 15-min means load is increasing.
free — Memory
bash
$ free -h total used free shared buff/cache availableMem: 7.7Gi 2.1Gi 312Mi 85Mi 5.3Gi 5.3GiSwap: 2.0Gi 0B 2.0Gi
Caution: Trap for beginners: “free” looks scary low — that’s not the number you want. Linux aggressively uses RAM for buff/cache (disk caches), reclaimable instantly. The number that matters is available.
vmstat and iostat
bash
$ vmstat 2 5 # sample every 2s, 5 times# watch: r (runnable, should be <= CPU count), si/so (swap, should be 0), wa (CPU% waiting on I/O)$ sudo apt install -y sysstat$ iostat -xz 2 # extended, skip idle devices# %util near 100% = saturated. await > 20ms on SSD = problem.
sar — Historical Stats
sar (from sysstat) records metrics every 10 minutes, so you can look at yesterday’s CPU.
bash
$ sar -u 1 5 # live CPU$ sar -r 1 5 # live memory$ sar -n DEV 1 5 # live network per interface$ sar -u -f /var/log/sysstat/sa25 # CPU on the 25th
$ top # press 1=per-core, M=sort by memory, P=sort by CPU, c=full command, k=kill$ sudo apt install -y htop glances nload iftop iotop ncdu$ htop # color, scrollable, tree view (F5)$ glances # CPU+MEM+NET+DISK+containers in one screen$ sudo iotop -o # disk I/O per process; -o = only active$ ncdu /var # interactive du — navigate and delete junk
In top: RES (resident set size — actual physical RAM) is the number that matters; VIRT is mostly meaningless.
Real Monitoring Scenarios
bash
$ watch -n 5 'ps -o pid,rss,vsz,cmd -p $(pgrep -f myapp)' # watch for a memory leak$ ps aux --sort=-%mem | head -11 # top 10 by memory$ ps aux --sort=-%cpu | head -11 # top 10 by CPU$ iostat -xz 2 5 # find I/O bottleneck$ ss -tn state established | awk 'NR>1{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn$ ps aux | awk '$8 ~ /Z/ {print}' # find zombie processes
Tip: Interview answer for “server slow, what do you do?” — in this exact order: uptime (load), top/htop (process), free -h (memory), iostat -xz 2 (disk), ss -s (network), dmesg -T | tail (kernel). This is Brendan Gregg’s USE Method: Utilization, Saturation, Errors.
Cron is the Unix scheduler — a daemon that wakes up every minute and runs jobs whose time has come. You’ll use it for backups, certificate renewals, log shipping, health checks, and cleanup scripts. It’s old (1975!) but still everywhere.
bash
$ systemctl status cron # Ubuntu/Debian$ systemctl status crond # RHEL
Crontab Syntax
text
# ┌───────────── minute (0 - 59)# │ ┌─────────── hour (0 - 23)# │ │ ┌───────── day of month (1 - 31)# │ │ │ ┌─────── month (1 - 12)# │ │ │ │ ┌───── day of week (0 - 7, where 0 and 7 both = Sunday)# │ │ │ │ │# * * * * * command-to-run
Symbol
Meaning
Example
*
Any value
* * * * * — every minute
*/N
Every N units
*/15 * * * * — every 15 minutes
N-M
Range
0 9-17 * * * — every hour, 9 AM to 5 PM
N,M
List
0 8,12,18 * * * — 8 AM, noon, 6 PM
Named shortcuts replace the whole 5-field expression: @reboot, @hourly, @daily, @weekly, @monthly, @yearly.
Tip: Cron expressions are easy to get subtly wrong. Validate them against a human-readable description and the next few run times with the Cron Expression Tester before committing a crontab.
Managing Crontabs
bash
$ crontab -e # edit YOUR crontab$ crontab -l # list YOUR crontab$ crontab -ri # delete with confirmation$ sudo crontab -u deploy -e # edit deploy user's crontab
User crontabs live in /var/spool/cron/crontabs/<user> — always use crontab -e, never edit directly.
System Cron
System crontab files have an extra “user” field between the schedule and the command:
bash
$ sudo cat /etc/crontabSHELL=/bin/shPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/binMAILTO=root# m h dom mon dow user command17 * * * * root cd / && run-parts --report /etc/cron.hourly25 6 * * * root test -x /usr/sbin/anacron || (cd / && run-parts --report /etc/cron.daily)
Path
Purpose
/etc/crontab
Main system schedule (with user field)
/etc/cron.d/
Drop-in files (packages install here)
/etc/cron.hourly/.daily/.weekly/.monthly/
Drop a script in, runs on schedule via run-parts
run-parts executes every executable file in a directory. Two gotchas: scripts must be executable (chmod +x), and filenames must not contain dots.
PATH is minimal. Cron’s default PATH is just /usr/bin:/bin. Your docker, aws, pg_dump, node may not be found. Set PATH= at the top of the crontab or use full paths.
No shell aliases / .bashrc. Cron runs /bin/sh non-interactively — no rc files are sourced.
Working directory is $HOME. Use absolute paths or cd /opt/app && first.
Always redirect output. Unredirected stdout/stderr becomes mail to the user. Use >> /var/log/myjob.log 2>&1.
Use scripts, not inline one-liners — testable and version-controllable.
Idempotency / overlap. Use a lockfile: flock -n /tmp/myjob.lock -c '/path/to/job.sh'.
% is special — a literal % must be escaped as \%.
Caution: The #1 cron PATH gotcha: your script works when you run ./backup.sh by hand, but cron says docker: command not found. That’s because your shell’s PATH includes /usr/local/bin but cron’s doesn’t. Always set PATH= at the top of the crontab OR call binaries by absolute path.
Where cron is for recurring jobs, at is for one-shot jobs. “Restart this service at 3 AM tonight.” “Run this cleanup once, ninety minutes from now.” Then forget it.
$ at now + 5 minutesat> echo "Hello from the future" > /tmp/hello.txtat> systemctl restart nginxat> ^Djob 3 at Tue May 26 11:47:00 2026
Time formats at understands: now + 2 hours, now + 1 day, 14:30, 14:30 tomorrow, midnight, noon, teatime (16:00), 3pm Friday, 02:00 next month.
Piping Commands In
bash
$ echo "/opt/scripts/restart-app.sh" | at now + 1 hour$ echo "shutdown -h now" | sudo at 23:00$ at -f /opt/scripts/cleanup.sh now + 30 minutes
Managing Jobs
bash
$ atq # list pending jobs$ at -c 7 # show contents of job 7$ atrm 7 # cancel job 7
batch is at’s sibling: it runs queued commands only when the system load average drops below 1.5 — great for heavy one-off jobs.
Need
Use
Run every day / hour / week
cron (or systemd timer)
Run exactly once, in the future
at
“Restart server in 5 min if I don’t cancel”
at (great safety net during config changes)
Run a heavy job when the system is idle
batch
Retries, dependencies, complex orchestration
systemd timer, Airflow/Argo
Tip: The “dead man’s switch” trick: before editing risky SSH/firewall config, schedule a rollback: echo "cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config && systemctl restart ssh" | sudo at now + 5 minutes. Make your changes and test. If you lock yourself out, the job revokes them in 5 minutes. If everything works: sudo atrm <jobnum>.
systemd timers are the modern replacement for cron. More verbose to set up, but they integrate with the rest of systemd: structured logging in the journal, dependency management, resource limits, retries, and the ability to catch up after downtime.
Note: Key idea — a timer is a unit that triggers another unit. You always create TWO files: foo.timer (when to run) and foo.service (what to run).
A Complete Example: Daily DB Backup
ini
# /etc/systemd/system/myapp-backup.service[Unit]Description=Daily backup of myapp Postgres databaseWants=network-online.targetAfter=network-online.target postgresql.service[Service]Type=oneshotUser=postgresExecStart=/usr/local/bin/backup-myapp.shStandardOutput=journalStandardError=journal
ini
# /etc/systemd/system/myapp-backup.timer[Unit]Description=Run myapp DB backup daily at 02:00[Timer]OnCalendar=*-*-* 02:00:00Persistent=trueRandomizedDelaySec=5minUnit=myapp-backup.service[Install]WantedBy=timers.target
bash
$ sudo systemctl daemon-reload$ sudo systemctl enable --now myapp-backup.timer # enable the TIMER, not the service$ systemctl list-timers myapp-backup.timer$ sudo systemctl start myapp-backup.service # manually trigger once to test$ journalctl -u myapp-backup.service -n 50 # view output
If a run was missed (server off), run on next boot
RandomizedDelaySec=5min
Spread a fleet of identical jobs — prevents thundering herd
bash
$ systemctl list-timers # next/last run for every timer$ systemctl list-timers --all$ systemctl cat myapp-backup.timer
Cron vs systemd Timer
Feature
cron
systemd timer
Setup effort
One line
Two unit files
Logs
Cron log + your redirects
Centralized journal — journalctl -u name
Dependencies
None
After=, Wants=, Requires=
Retries on failure
Write your own
Restart=on-failure
Catch up missed runs
anacron only, blunt
Persistent=true
Resource limits
None
MemoryMax=, CPUQuota=, etc.
Test/dry-run
None
systemd-analyze calendar "..." + systemctl start
Familiarity
Universal
Linux-only
Tip: Migration tip: if you already have a cron job, you can convert it to a timer in 10 minutes. Bonus: journalctl -u myjob gives you every run, exit code, and stdout/stderr without setting up your own log redirection. That alone justifies the two extra files.
Security in Linux is not a feature you turn on, it’s a posture you maintain. As a DevOps engineer you’re responsible for hardening servers, protecting credentials, and reducing attack surface.
Note: Security is a habit, not an event. Check daily: who is logging in, what ports are open, are the latest patches applied. Hardening once and forgetting is the biggest mistake — attackers only need one small gap.
Strong Passwords (PAM Password Quality)
PAM is the framework Linux uses for authentication. The pam_pwquality module enforces strong passwords system-wide via /etc/security/pwquality.conf:
ini
minlen = 14minclass = 4 # uppercase + lowercase + digit + symboldcredit = -1 # require at least 1 digitucredit = -1 # require at least 1 uppercaseocredit = -1 # require at least 1 symbolretry = 3enforce_for_root = 1
Account lockout after failed attempts via pam_faillock in /etc/pam.d/common-auth (deny=5, unlock_time=900).
Caution: Test before you log out! A bad PAM config can lock you out of your own server. Always keep a second SSH session open while editing PAM files.
SSH Hardening Checklist (Recap)
Edit /etc/ssh/sshd_config:
ini
PermitRootLogin no # disable root loginPasswordAuthentication no # keys only — most important settingPort 2222 # reduce noiseAllowUsers deploy ops-aliceMaxAuthTries 3LoginGraceTime 30ClientAliveInterval 300X11Forwarding noPermitEmptyPasswords no
bash
$ ssh-keygen -t ed25519 -a 100 -C "pushkar@workstation"$ sudo sshd -t # test config syntax$ sudo systemctl reload ssh
Firewall
bash
# ufw — Ubuntu's friendly front-end$ sudo ufw default deny incoming$ sudo ufw default allow outgoing$ sudo ufw allow 2222/tcp comment 'SSH'$ sudo ufw allow 80,443/tcp$ sudo ufw allow from 192.168.1.0/24 to any port 5432 # Postgres LAN only$ sudo ufw limit 2222/tcp # rate-limit SSH$ sudo ufw enable$ sudo ufw status verbose
$ sudo systemctl enable --now fail2ban$ sudo fail2ban-client status sshd$ sudo fail2ban-client set sshd unbanip 203.0.113.45
Real world: On any internet-facing VM you’ll see SSH brute-force attempts within minutes of booting. fail2ban + non-standard port + key-only auth reduces auth.log from thousands of failure lines per day to near-zero. It’s the single highest-leverage hardening step.
SELinux vs AppArmor — Mandatory Access Control
Standard Unix permissions are discretionary (the owner decides access). MAC adds a second layer: even root cannot violate kernel-enforced policies.
Caution: Don’t disable MAC because “the app won’t work.” Set it to Permissive/Complain mode, reproduce the issue, read the audit log, then write a targeted policy exception. Disabling SELinux/AppArmor in production is a major red flag in security audits.
File Integrity — Hashes
bash
$ md5sum ubuntu-24.04.iso # FAST but cryptographically broken — checksums only$ sha256sum ubuntu-24.04.iso # the standard for downloads$ sha512sum kernel.tar.xz # extra paranoid$ sha256sum -c SHA256SUMS # verify against a published checksum file
Tip: Need to hash a string or small file quickly, or compare a download against its published digest without dropping to a shell? The Hash Generator produces MD5/SHA-1/SHA-256/SHA-512 in the browser.
GPG Basics
GPG provides encryption, signing, and key management — used everywhere from APT repo signing to package release verification.
Backups + tested restore — an untested backup is a hope, not a backup.
Least privilege — sudo whitelists, no shared accounts, separate prod/staging credentials.
Container security — non-root USER, drop capabilities, read-only rootfs, scan images (Trivy/Grype), use distroless/minimal base images.
Secrets management — Vault, AWS Secrets Manager, sealed-secrets; never commit secrets to git.
Monitor auth events — alert on failed sudo, new SSH keys, sudoers changes.
Real world: When you join a new DevOps team, run Lynis on a staging box in week one. It generates a 100+ point hardening report. Even mature shops typically score 60-70/100 — the gap is your low-hanging-fruit roadmap and an instant credibility builder.
Troubleshooting is the skill that separates a junior DevOps from a senior. Tools you can google. Methodology you have to internalise.
The Systematic Approach
Define the problem clearly. “The website is slow” is not a definition. “p95 latency on /api/checkout went from 200ms to 3s at 14:02 UTC for all users” is.
Gather data. Logs, metrics, kernel messages (dmesg), recent changes (deploys, config), system state (top, df, ss).
Reproduce it. Either in staging, or get exact steps.
Form a hypothesis. One thing at a time.
Test with the smallest change. Don’t change five things at once.
Document. Symptom, root cause, fix, prevention.
Note: The biggest mistake is skipping root cause in the rush to fix. A server restart gives 5 minutes of relief, but if you didn’t catch the root cause the issue returns in 3 days — and by then the logs have rotated. Slow down to go fast.
$ uptime; top -o %CPU; top -o %MEM; iostat -xz 1 5; vmstat 1 5; ss -s
Identify the bottleneck (CPU / IO / memory / network), then kill/throttle the offender, scale resources, or add caching. 80% of “server slow” tickets are: a runaway log filling disk, a query without an index, or memory pressure causing swap thrashing — check disk and swap first.
Service won’t start:
bash
$ sudo systemctl status nginx$ sudo journalctl -u nginx -n 100 --no-pager$ sudo nginx -t # config syntax check$ sudo ss -tulpn | grep :80 # port already in use?
Common causes: config syntax error, port already bound, missing dependency in After=, wrong User=, SELinux/AppArmor denial.
SSH “Permission denied (publickey)”:
bash
$ ssh -vvv -i ~/.ssh/id_ed25519 deploy@server # which key was tried?$ stat -c '%a %U %G %n' ~/.ssh ~/.ssh/authorized_keys# Required: ~ = 755/750, .ssh = 700, authorized_keys = 600, owned by the user
SSH “Connection refused” vs “timed out”: Refused = something at the destination actively rejected (sshd down, or firewall REJECT). Timed out = packets never arrived (routing, cloud security group, or a DROP rule). AWS/GCP/Azure security groups are the most common culprit.
Fix: add swap (short term), increase memory, fix the leaking app, or set MemoryMax= in the systemd unit. Java/Node in containers often get OOM-killed because they read the host’s memory, not the cgroup limit — set -Xmx or NODE_OPTIONS=--max-old-space-size to match the container limit.
Container not starting:
bash
$ docker ps -a # find the dead container's exit code$ docker logs --tail=200 myapp$ docker run --rm -it --entrypoint sh myapp:latest # poke inside
Common exit codes: 0 = clean exit, 1 = app error, 125 = docker daemon error, 126 = not executable, 127 = command not found, 137 = OOM-killed, 139 = segfault, 143 = SIGTERM. Volume permission mismatch (UID inside container vs host) is the #1 silent failure in dev environments.
Power Tools
Tool
Use For
Example
strace
Trace system calls
strace -f -e openat,connect -p 1234
lsof
List open files, sockets, FDs
lsof -i :8080, lsof +L1
perf
CPU profiling, flame graphs
sudo perf top
tcpdump
Packet capture
sudo tcpdump -i any -w cap.pcap port 443
bpftrace
eBPF tracing
sudo execsnoop-bpfcc
The USE Method
Brendan Gregg’s USE method: for every resource (CPU, memory, disk IO, disk capacity, network IO, file descriptors), check Utilization, Saturation, and Errors.
Real world: In every senior DevOps interview you’ll get an open-ended scenario: “Production API is returning 500s, walk me through what you do.” They’re testing whether you have a method: define → gather → hypothesise → test → document. Mention rollback as the first option before deep debugging in prod. That single framing puts you ahead of most candidates.
These are the highest-frequency Linux questions asked in DevOps and SRE interviews, condensed to the answer that signals real understanding. Drill them out loud — if you can teach the answer in your own words, you own it.
1. Difference between Linux and Unix? Unix is a proprietary OS family from the 1970s (AT&T Bell Labs); Linux is a free, open-source, Unix-like kernel created by Linus Torvalds in 1991. Linux follows Unix design principles but shares no code with original Unix. macOS is certified Unix (BSD lineage); Linux is not.
2. Difference between the kernel and the operating system? The kernel is the core program that talks to hardware and manages CPU, memory, devices, and processes. The OS is the kernel plus userspace — shell, libraries, utilities — that makes it usable. Calling the kernel “the OS” is the classic blunder.
3. Difference between bash and sh?sh is the POSIX-standard minimal shell; bash is a superset with arrays, [[ ]], brace expansion, and the function keyword. On Debian/Ubuntu, /bin/sh is dash, not bash — so a #!/bin/sh script using bash-only features works on RHEL but breaks on Ubuntu. Match your shebang to the features you use.
4. How do Linux file permissions work? Three triplets (owner/group/other), each with read (4), write (2), execute (1). Set symbolically (chmod u+x) or octal (chmod 755). Special bits: setuid (run as owner), setgid (inherit group / run as group), sticky (only owner can delete, as on /tmp).
5. Hard link vs symbolic link? A hard link is a second directory entry pointing to the same inode — same FS only, can’t link directories, survives deletion of any one name while link count > 0. A symlink is a separate file whose content is a path string, resolved at access time — can cross filesystems, can point to directories, dangles if the target is removed.
6. What happens from power-on to login prompt? BIOS/UEFI → POST → MBR/GPT boot sector → GRUB loads kernel + initramfs → kernel mounts / and starts PID 1 → systemd starts services in dependency order and mounts /etc/fstab → reaches multi-user.target → login (getty/sshd). Most production outages happen here.
7. What’s the difference between a VM and a container? A VM virtualizes hardware and runs a full guest kernel. A container is a regular process on the host kernel isolated with namespaces (PID, net, mount, user) and cgroups (CPU, memory, I/O limits). Container = process; VM = virtual machine.
8. Difference between kill -9 and kill -15?-15 (SIGTERM) is a polite shutdown request the process can catch and clean up after. -9 (SIGKILL) is uncatchable and immediate — no cleanup, risk of corrupt state. Always try SIGTERM first.
9. Explain fork() and exec().fork() duplicates the current process (child gets a new PID). exec() replaces the child’s memory image with a new program (same PID). The shell uses fork+exec+wait to run every command.
10. Disk shows full but df reports free space — what’s wrong? Either inode exhaustion (df -i shows 100% IUse) or a process holding a deleted-but-open file (lsof | grep deleted). For inodes, delete the millions of tiny files; for the open file, restart the holding process.
11. How do you debug a service that won’t start?systemctl status <svc>, then journalctl -u <svc> -n 100, then validate config (nginx -t, sshd -t), then check the port isn’t already bound (ss -tulpn). Common causes: config syntax, port conflict, wrong User=, missing dependency, MAC denial.
12. What’s the difference between cron and a systemd timer? Both schedule recurring work. systemd timers add journal logging, dependency ordering (After=/Wants=), retries (Restart=), catch-up of missed runs (Persistent=true), and resource limits. Cron is simpler and universal. Use timers when you need observability and dependencies.
13. How do you harden SSH on a production server?PasswordAuthentication no (key-only), PermitRootLogin no, restrict with AllowUsers, lower MaxAuthTries, non-standard port, add fail2ban, and use ed25519 keys. Always sshd -t and keep a second session open before reloading.
14. A server is slow — walk me through it. Triage in order: uptime (load vs core count), top/htop (CPU/mem hogs), free -h (look at available, not free), iostat -xz 2 (disk %util/await), ss -s (connections), dmesg -T | tail (kernel/OOM). This is the USE method: Utilization, Saturation, Errors.
15. How do you find what’s listening on a port / using a port?ss -tlnp | grep :80 or lsof -i :80 shows the listener and PID. To free it, identify the PID then stop that service (fuser -k 80/tcp as a blunt instrument).
Tip: In interviews, never just say “I don’t know” — add “but here’s how I’d find out.” Confidence with humility beats fake expertise. For an experienced developer pivoting into DevOps, that maturity is a genuine differentiator.
Bookmark this section — these are the reference sheets you will hit Ctrl+F on during your first 6 months on the job. Print them, paste them on your wall, whatever works. Real production-grade reference, no fluff.
1. Top 100 Linux Commands (Grouped)
Files & Directories
Command
Purpose
Example
ls
List directory
ls -lah --color
cd
Change directory
cd - (previous)
pwd
Print working dir
pwd -P (resolve symlinks)
mkdir
Make directory
mkdir -p a/b/c
rmdir
Remove empty dir
rmdir empty_folder
rm
Remove files
rm -rf node_modules
cp
Copy
cp -av src/ dst/
mv
Move/rename
mv old.txt new.txt
ln
Link (hard/soft)
ln -s /opt/app /usr/local/app
touch
Create empty / update mtime
touch app.log
stat
File metadata
stat /etc/passwd
file
Detect file type
file binary.bin
find
Find files
find . -name "*.log" -mtime +7
locate
Fast filename search
locate sshd_config
tree
Directory tree
tree -L 2
basename
Strip path
basename /a/b/c.txt
dirname
Strip filename
dirname /a/b/c.txt
realpath
Resolve absolute path
realpath ./app
Text Processing
Command
Purpose
Example
cat
Print file
cat /etc/os-release
tac
Reverse cat
tac access.log
less
Paged viewer
less +F app.log (follow)
more
Old pager
more big.txt
head
First N lines
head -n 20 file
tail
Last N lines
tail -f -n 100 app.log
grep
Pattern search
`grep -rEn “TODO
egrep / fgrep
ERE / fixed
`grep -E “a
sed
Stream editor
sed -i 's/dev/prod/g' app.conf
awk
Field processor
awk '{print $1,$9}' access.log
cut
Extract columns
cut -d: -f1 /etc/passwd
sort
Sort lines
sort -u -k2,2n
uniq
Dedupe (sorted)
`sort file
wc
Count
wc -l *.py
tr
Translate chars
tr 'a-z' 'A-Z'
tee
Write & pass through
`cmd
diff
Compare
diff -u a b
patch
Apply diff
patch -p1 < fix.patch
jq
JSON processor
jq '.items[].name' data.json
yq
YAML processor
yq '.services' docker-compose.yml
xargs
Build commands
`find . -name “*.bak”
Process
Command
Purpose
Example
ps
Process snapshot
ps -ef or ps aux
top
Live processes
top -o %CPU
htop
Better top
htop (F6 sort, F9 signal)
kill
Send signal
kill -9 1234
pkill
Kill by name
pkill -f gunicorn
killall
Kill all by name
killall nginx
pgrep
Find PIDs
pgrep -af java
nice
Run with priority
nice -n 10 ./batch.sh
renice
Change priority
renice 5 -p 1234
nohup
Survive logout
nohup ./run.sh &
jobs
Shell jobs
jobs -l
bg / fg
Background/foreground
bg %1 / fg %1
strace
Trace syscalls
strace -p 1234 -f -e network
lsof
Open files
lsof -i :8080
Network
Command
Purpose
Example
ip
Modern net tool
ip -br a / ip r
ifconfig
Legacy
ifconfig eth0
ping
ICMP test
ping -c 4 google.com
traceroute
Hop trace
traceroute -n 8.8.8.8
mtr
Live traceroute
mtr 1.1.1.1
dig
DNS lookup
dig +short A example.com
nslookup
DNS query
nslookup example.com 8.8.8.8
host
Simple DNS
host example.com
curl
HTTP client
curl -fsSL https://x.com
wget
Download
wget -c https://x.com/file
ss
Sockets
ss -tulnp
netstat
Legacy ss
netstat -tulnp
nc
Netcat
nc -zv host 443
nmap
Port scanner
nmap -sV -p 1-1000 host
tcpdump
Packet capture
tcpdump -i eth0 port 80
iptables
Firewall (legacy)
iptables -L -n -v
nft
nftables
nft list ruleset
System
Command
Purpose
Example
uname
Kernel info
uname -a
hostnamectl
Hostname/OS info
hostnamectl
uptime
Load avg
uptime
w / who
Logged users
w
last
Login history
last -n 20
date
Date/time
date -u +%Y-%m-%dT%H:%M:%SZ
timedatectl
Timezone/NTP
timedatectl set-timezone Asia/Kolkata
systemctl
Service control
systemctl status nginx
journalctl
Logs
journalctl -u nginx -f
dmesg
Kernel ring buffer
`dmesg -T
free
Memory
free -h
vmstat
Virtual memory stats
vmstat 1 5
iostat
IO stats
iostat -xz 1
sar
Historical perf
sar -u 1 5
env
Env vars
`env
which / type
Locate binary
type -a ls
Security & Users
Command
Purpose
Example
sudo
Privilege escalation
sudo -i
su
Switch user
su - deploy
passwd
Change password
sudo passwd alice
useradd
Add user
useradd -m -s /bin/bash alice
usermod
Modify user
usermod -aG docker alice
userdel
Delete user
userdel -r alice
groupadd
Add group
groupadd devops
id
Show IDs
id alice
chage
Pass aging
chage -l alice
getfacl/setfacl
ACLs
setfacl -m u:alice:rw file
fail2ban-client
Ban abusers
fail2ban-client status sshd
ufw / firewalld
Firewall frontends
ufw allow 22/tcp
openssl
Crypto Swiss-army
openssl s_client -connect host:443
Compression
Command
Purpose
Example
tar
Archive
tar -czvf a.tgz dir/
tar -x
Extract
tar -xzvf a.tgz
gzip / gunzip
gzip files
gzip -9 big.log
bzip2 / bunzip2
bzip2
bzip2 file
xz / unxz
xz (best ratio)
xz -T0 huge.tar
zip / unzip
Zip format
unzip -d out/ a.zip
zcat / zless
View gzipped
`zcat app.log.gz
Permissions
Command
Purpose
Example
chmod
Change mode
chmod 750 deploy.sh
chown
Change owner
chown -R deploy:deploy /opt/app
chgrp
Change group
chgrp www-data /var/www
umask
Default mask
umask 027
Disk
Command
Purpose
Example
df
Filesystem usage
df -hT
du
Directory usage
`du -sh *
lsblk
Block devices
lsblk -f
fdisk
Partition tool
sudo fdisk -l
parted
GPT partitioning
parted /dev/sdb print
mkfs
Make filesystem
mkfs.ext4 /dev/sdb1
mount / umount
Mount FS
mount /dev/sdb1 /mnt
blkid
UUIDs/labels
blkid /dev/sda1
ncdu
TUI disk usage
ncdu /var
fsck
Filesystem check
fsck -y /dev/sda1
lvm tools
LVM
pvs / vgs / lvs
Miscellaneous
Command
Purpose
Example
man / info
Manuals
man 5 crontab
tldr
Practical examples
tldr tar
history
Shell history
`history
alias
Define alias
alias ll='ls -lah'
watch
Repeat command
watch -n 2 'df -h'
time
Time a command
time ./build.sh
timeout
Limit runtime
timeout 30s ./probe
yes
Repeat string
`yes
screen / tmux
Terminal multiplexer
tmux new -s work
echo / printf
Print
printf "%s\n" "$VAR"
true / false
Exit code helpers
while true; do ...; done
2. Vim Cheat Sheet
Modes
Mode
Enter
Purpose
Normal
Esc
Default; movement & commands
Insert
i I a A o O
Type text
Visual
v
Character selection
Visual line
V
Line selection
Visual block
Ctrl+v
Column / rectangular edit
Command
:
Ex commands (:w, :q)
Replace
R
Overwrite mode
Terminal
:terminal
Embedded shell
Navigation
Key
Action
h j k l
Left, down, up, right
w / b
Next / previous word
e / ge
End of word / previous word end
0 / ^ / $
Line start / first non-blank / line end
gg / G
File top / file bottom
:42 or 42G
Jump to line 42
Ctrl+d / Ctrl+u
Half page down / up
Ctrl+f / Ctrl+b
Full page down / up
{ / }
Previous / next blank line
%
Match bracket
f x / F x
Find char forward / backward
* / #
Search word under cursor forward / backward
Editing
Key
Action
i / a
Insert before / after cursor
I / A
Insert at line start / end
o / O
New line below / above
x / X
Delete char forward / backward
dd / D
Delete line / to end of line
dw / d$ / dG
Delete word / to EOL / to EOF
yy / Y
Yank line
p / P
Paste after / before
cc / C
Change line / to EOL
cw / ciw / ci"
Change word / inner word / inside quotes
r / R
Replace one char / replace mode
u / Ctrl+r
Undo / redo
.
Repeat last change
>> / <<
Indent / dedent line
J
Join line below
Search & Replace
bash
/pattern " Search forward?pattern " Search backwardn / N " Next / previous match:noh " Clear highlight:s/old/new/ " Replace first on line:s/old/new/g " Replace all on line:%s/old/new/g " Replace all in file:%s/old/new/gc " Confirm each:%s/old/new/gI " Case-sensitive (capital I):5,15s/old/new/g " Replace in line range:'<,'>s/old/new/g " Replace in visual selection:g/pattern/d " Delete all lines matching:v/pattern/d " Delete lines NOT matching
Visual Mode
Key
Action
v / V / Ctrl+v
Char / line / block selection
d / y / c
Delete / yank / change selection
> / <
Indent / dedent block
~
Toggle case
u / U
Lower / upper case selection
:'<,'>sort
Sort selected lines
Ctrl+v then I text Esc
Insert column prefix on all lines
Advanced — Registers, Marks, Macros, Splits, Tabs
bash
" Registers (named clipboards)"ayy " Yank line to register a"ap " Paste register a:reg " Show all registers"+y " Yank to system clipboard (needs +clipboard)"+p " Paste from system clipboard" Marksma " Set mark a at cursor'a " Jump to line of mark a`a " Jump to exact position of mark a:marks " List all marks" Macrosqa " Start recording macro into a...do stuff...q " Stop recording@a " Play macro a@@ " Replay last macro10@a " Play macro 10 times" Splits:split file " Horizontal split (or :sp):vsplit file " Vertical split (or :vsp)Ctrl+w h/j/k/l " Move between splitsCtrl+w = " Equalize sizesCtrl+w _ " Maximize heightCtrl+w | " Maximize widthCtrl+w c " Close split" Tabs:tabnew file " Open in new tabgt / gT " Next / previous tab:tabclose " Close current tab:tabs " List tabs
3. File Permissions Reference
Octal Table (000–777)
Octal
Binary
rwx
Meaning
0
000
---
No access
1
001
--x
Execute only
2
010
-w-
Write only
3
011
-wx
Write + execute
4
100
r--
Read only
5
101
r-x
Read + execute
6
110
rw-
Read + write
7
111
rwx
Full access
Three digits = owner, group, others. So 750 = rwxr-x--- (owner full, group r-x, others nothing).
chmod u+x deploy.sh # add execute for ownerchmod go-rwx secret.key # remove all from group + otherschmod a=r README.md # read-only for allchmod -R u=rwX,g=rX,o= /app # safe recursive (capital X)chmod u+s /usr/bin/myapp # setuid
chown / chgrp
bash
chown alice file # change ownerchown alice:devs file # owner + groupchown :devs file # only groupchown -R deploy:deploy /opt/app # recursivechown --reference=other.txt file # copy ownership from other.txtchgrp www-data /var/www # change group only
4. Bash Scripting Cheat Sheet
Variables & Parameters
bash
name="alice" # no spaces around =echo "$name" # always quote variable expansionsecho "${name}_log" # disambiguate with bracesreadonly PI=3.14 # constantunset name # delete variableexport PATH="$PATH:/opt/bin" # export to children# Positional params inside a script$0 # script name$1..$9 # args 1..9${10} # 10th arg (need braces)$# # number of args$@ # all args (each quoted separately when "$@")$* # all args (one string when "$*")$? # exit code of last command$$ # current PID$! # PID of last backgrounded command# Parameter expansion${var:-default} # use default if var unset/empty${var:=default} # also assign default${var:?error} # error if unset${var:+alt} # use alt if var IS set${#var} # string length${var:2:4} # substring from pos 2, length 4${var/foo/bar} # replace first foo with bar${var//foo/bar} # replace ALL foo${var#prefix} # strip shortest prefix${var##prefix} # strip longest prefix${var%.txt} # strip shortest suffix${var%%.*} # strip longest suffix
Test Operators
Category
Operator
Meaning
Integer
-eq
equal
Integer
-ne
not equal
Integer
-lt
less than
Integer
-le
less or equal
Integer
-gt
greater than
Integer
-ge
greater or equal
String
= / ==
equal
String
!=
not equal
String
-z
zero-length (empty)
String
-n
non-empty
String
< / >
lexicographic (inside [[ ]])
File
-e
exists
File
-f
is regular file
File
-d
is directory
File
-L
is symlink
File
-r / -w / -x
readable / writable / executable
File
-s
non-empty (size > 0)
File
-O
owned by you
File
-G
owned by your group
File
a -nt b
a newer than b
File
a -ot b
a older than b
Control Structures
bash
# if / elif / elseif [[ $count -gt 10 ]]; then echo "many"elif [[ -z "$name" ]]; then echo "no name"else echo "ok"fi# casecase "$env" in prod|production) echo "prod" ;; dev|development) echo "dev" ;; *) echo "unknown"; exit 1 ;;esac# forfor f in *.log; do gzip "$f"donefor i in {1..5}; do echo "$i"; donefor ((i=0; i<10; i++)); do echo "$i"; done# while / untilwhile read -r line; do echo "got: $line"done < file.txtcount=0until [[ $count -ge 5 ]]; do ((count++))done# functionsdeploy() { local env="$1" # 'local' avoids polluting global scope echo "deploying to $env" return 0}deploy prod
Common Idioms
bash
# Strict mode (put at top of every serious script)set -euo pipefailIFS=$'\n\t'# Check command existsif ! command -v jq >/dev/null 2>&1; then echo "jq required" >&2; exit 1fi# Retry loop with backofffor attempt in 1 2 3 4 5; do curl -fsSL "$URL" && break sleep $((2 ** attempt))done# Parse args (getopts)while getopts "e:vh" opt; do case "$opt" in e) ENV="$OPTARG" ;; v) VERBOSE=1 ;; h) usage; exit 0 ;; *) usage; exit 1 ;; esacdoneshift $((OPTIND - 1))# Trap cleanuptmp=$(mktemp -d)trap 'rm -rf "$tmp"' EXIT INT TERM# Read file line by line (safe)while IFS= read -r line || [[ -n "$line" ]]; do process "$line"done < input.txt# Heredoccat <<EOF > /etc/myapp/confighost=$(hostname)env=$ENVEOF
Arrays
bash
# Indexed arrayservers=(web1 web2 web3)echo "${servers[0]}" # web1echo "${servers[@]}" # all elementsecho "${#servers[@]}" # length = 3servers+=(web4) # appendunset 'servers[1]' # delete elementfor s in "${servers[@]}"; do echo "$s"; done# Associative array (bash 4+)declare -A user_roleuser_role[alice]=adminuser_role[bob]=devecho "${user_role[alice]}"for k in "${!user_role[@]}"; do echo "$k => ${user_role[$k]}"done
Brace Expansion
bash
echo file{1,2,3}.txt # file1.txt file2.txt file3.txtecho {a..e} # a b c d eecho {01..05} # 01 02 03 04 05echo {1..10..2} # 1 3 5 7 9mkdir -p project/{src,test,docs}/{api,web}cp app.conf{,.bak} # quick backup -> app.conf, app.conf.bak
5. Regular Expressions
BRE vs ERE Side-by-Side
Feature
BRE (grep, sed)
ERE (egrep, grep -E, sed -E, awk)
Alternation
|
`
Grouping
\( \)
( )
One or more
\+
+
Zero or one
\?
?
Repetition
\{m,n\}
{m,n}
Zero or more
*
*
Backrefs
\1 \2
\1 \2
Rule of thumb: use grep -E (ERE) — fewer backslashes, fewer tears.
top # launch# Interactive keys:P sort by CPUM sort by memoryT sort by timeN sort by PIDk kill process (asks PID + signal)r renicec toggle full command line1 toggle per-CPU viewH toggle threadsW save config to ~/.toprcq quittop -b -n 1 # batch mode (for scripts)top -p 1234,5678 # only specific PIDstop -u deploy # only user deploy
htop
bash
htop # nicer top# F1 help F2 setup F3 search F4 filter F5 tree# F6 sort F7/F8 nice +/- F9 kill F10 quit# Space = tag a process; u = filter by user
ps
bash
ps aux # BSD style, all processesps -ef # System V styleps -eLf # threads toops -eo pid,user,%cpu,%mem,cmd --sort=-%cpu | headps --ppid 1234 # children of PID 1234ps -C nginx # find by command namepstree -p # process tree with PIDs
Memory & CPU
bash
free -h # memory in human unitsfree -m -s 2 # refresh every 2scat /proc/meminfocat /proc/cpuinfovmstat 1 5 # 5 samples, 1 sec apart# columns: r b | swpd free buff cache | si so | bi bo | in cs | us sy id wa stiostat -xz 1 # extended IO stats, skip idleiostat -mx 2 5 # MB/s, 5 samplesmpstat -P ALL 1 3 # per-CPU stats
sar (historical perf)
bash
sar -u 1 5 # CPUsar -r 1 5 # memorysar -b 1 5 # IOsar -n DEV 1 5 # networksar -q 1 5 # load avg + run queuesar -f /var/log/sysstat/sa15 # read past data (day 15)
df -hT # disk usage by FSdf -i # inode usagedu -sh /var/log/* # size per itemdu -sh * 2>/dev/null | sort -h # sortedncdu / # interactive TUIss -tulnp # listening TCP/UDP sockets + processesss -s # socket summaryss -tan state established # established TCP conns
8. Network Commands
ip (the modern tool)
bash
ip a # show all addressesip -br a # brief viewip -4 a show eth0 # only IPv4 on eth0ip r # routing tableip -br link # link state of interfacesip neigh # ARP table (neighbors)# Modifysudo ip addr add 10.0.0.10/24 dev eth1sudo ip addr del 10.0.0.10/24 dev eth1sudo ip link set eth1 upsudo ip link set eth1 downsudo ip route add default via 10.0.0.1
ping & traceroute
bash
ping -c 4 google.com # 4 packets then stopping -i 0.2 -c 20 host # 20 packets, 0.2s apartping -s 1472 -M do host # MTU test (don't fragment)traceroute -n 8.8.8.8 # numeric, no DNStraceroute -T -p 443 host # TCP traceroute on port 443mtr -rwn 1.1.1.1 # report mode, wide, numeric
kill 1234 # SIGTERM (default)kill -9 1234 # SIGKILL by numberkill -SIGTERM 1234 # by namekill -HUP $(pidof nginx) # reload nginx configpgrep -f gunicorn # find PIDs by full cmdlinepgrep -af gunicorn # also show cmdlinepkill -f gunicorn # kill all matchingpkill -HUP -f nginx # signal all matchingkillall nginx # kill by exact process namekillall -u alice # kill all of user alice's processes
nice / renice
bash
# Nice value range: -20 (highest priority) to 19 (lowest)nice -n 10 ./batch.sh # start with niceness 10renice 5 -p 1234 # change running processrenice -n 15 -u alice # all of alice's processesionice -c 3 -p 1234 # IO scheduling class (3=idle)
nohup & Job Control
bash
nohup ./long_job.sh > out.log 2>&1 & # survives logoutdisown %1 # detach job from shell./job.sh & # backgroundjobs # list shell jobsjobs -l # with PIDsfg %1 # foreground job 1bg %1 # resume in backgroundCtrl+Z # suspend foreground job
systemctl
bash
systemctl status nginxsystemctl start nginxsystemctl stop nginxsystemctl restart nginxsystemctl reload nginx # signal reload, no restartsystemctl enable --now nginx # enable + startsystemctl disable --now nginxsystemctl is-active nginxsystemctl is-enabled nginxsystemctl list-units --type=servicesystemctl list-units --failedsystemctl daemon-reload # after editing unit filessystemctl cat nginx # show unit filesystemctl edit nginx # overridesystemctl mask nginx # prevent any start (strong disable)
journalctl
bash
journalctl -u nginx # logs for nginxjournalctl -u nginx -f # follow (like tail -f)journalctl -u nginx --since "10 min ago"journalctl -u nginx --since "2026-05-26 09:00" --until "2026-05-26 10:00"journalctl -u nginx -p err # priority err and abovejournalctl -k # kernel only (= dmesg)journalctl -b # current bootjournalctl -b -1 # previous bootjournalctl --disk-usagejournalctl --vacuum-time=7d # keep only last 7 days
10. Cron Expressions
Field Syntax
bash
┌───────────── minute (0 - 59)│ ┌─────────── hour (0 - 23)│ │ ┌───────── day of month (1 - 31)│ │ │ ┌─────── month (1 - 12) or JAN-DEC│ │ │ │ ┌───── day of week (0 - 6) or SUN-SAT (0 and 7 = Sun)│ │ │ │ │* * * * * command_to_run
Special Characters
Char
Meaning
Example
*
Every value
* * * * * = every minute
,
List
0 9,12,18 * * * = 9am, 12pm, 6pm
-
Range
0 9-17 * * 1-5 = 9–5 Mon–Fri
/
Step
*/15 * * * * = every 15 minutes
L (extended)
Last
0 0 L * * = last day of month (not standard cron)
? (extended)
No value
Used in Quartz/Spring
20+ Real Examples
Expression
Runs
* * * * *
Every minute
*/5 * * * *
Every 5 minutes
*/15 * * * *
Every 15 minutes
0 * * * *
Every hour on the hour
0 */2 * * *
Every 2 hours
30 3 * * *
3:30 AM every day
0 0 * * *
Midnight every day
0 9 * * 1-5
9 AM weekdays
0 9 * * 1
9 AM every Monday
0 9 * * 0
9 AM every Sunday
0 9-17 * * 1-5
Every hour, 9–5, weekdays
0 0 1 * *
Midnight on 1st of month
0 0 1 1 *
Midnight on Jan 1 (yearly)
0 0 1,15 * *
1st and 15th at midnight
0 2 * * 6
2 AM every Saturday (weekly backup)
15 14 1 * *
2:15 PM on the 1st of each month
0 22 * * 1-5
10 PM weekdays
5 4 * * SUN
4:05 AM every Sunday
0 4 1-7 * 1
4 AM on first Monday of month (kind of)
*/10 9-17 * * 1-5
Every 10 min, business hours, weekdays
0 0 */3 * *
Midnight every 3 days
0 12 * * 1#1
Noon on first Monday (extended/Quartz)
@ Shortcuts
Shortcut
Equivalent
Meaning
@reboot
—
Once at startup
@yearly / @annually
0 0 1 1 *
Once a year
@monthly
0 0 1 * *
Once a month
@weekly
0 0 * * 0
Once a week (Sunday)
@daily / @midnight
0 0 * * *
Once a day
@hourly
0 * * * *
Once an hour
Managing Crontabs
bash
crontab -l # list current user's crontabcrontab -e # edit (uses $EDITOR)crontab -r # remove (be careful!)crontab -u alice -l # list alice's crontab (need root)# System-wide files:/etc/crontab # has a 'user' column/etc/cron.d/* # drop-in files (also need user column)/etc/cron.{hourly,daily,weekly,monthly}/ # scripts dir (no schedule)# Always set env + redirect output in cron jobs:SHELL=/bin/bashPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binMAILTO=ops@1buy.ai0 3 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1# Useful for verifying: test online at crontab.guru
Speed comes from muscle memory, not from typing faster. Until these shortcuts live in your fingers, you won’t feel the productivity boost. Pick 10 from this list, drill them this week, then come back for the next 10.
Terminal Keyboard Shortcuts (Readline / Bash)
Cursor & Editing
Shortcut
Action
Ctrl+A
Move to beginning of line
Ctrl+E
Move to end of line
Ctrl+B / Ctrl+F
Back / forward one character
Alt+B / Alt+F
Back / forward one word
Ctrl+U
Cut from cursor to beginning of line
Ctrl+K
Cut from cursor to end of line
Ctrl+W
Cut previous word
Alt+D
Cut next word
Ctrl+Y
Paste (yank) last cut
Ctrl+T
Swap current and previous character
Alt+T
Swap current and previous word
Ctrl+_
Undo last edit
Ctrl+L
Clear screen (keeps current line)
Process & Session
Shortcut
Action
Ctrl+C
Send SIGINT (interrupt) to foreground process
Ctrl+Z
Send SIGTSTP (suspend) — resume with fg or bg
Ctrl+D
EOF — exits shell if line is empty
Ctrl+S
Freeze terminal output (XOFF) — try not to hit this
Ctrl+Q
Resume frozen terminal (XON)
Ctrl+\
SIGQUIT — like Ctrl+C but with core dump
History & Search
Shortcut
Action
Ctrl+R
Reverse-i-search through history (press again for next match)
Ctrl+G
Cancel reverse search
Ctrl+P / Ctrl+N
Previous / next history entry
Alt+.
Insert last argument of previous command (press repeatedly)
History Expansion (the bang stuff)
Expansion
Meaning
!!
Previous command in full (e.g. sudo !!)
!$
Last argument of previous command
!*
All arguments of previous command
!N
Command number N from history
!-N
N commands back
!abc
Most recent command starting with abc
!?abc?
Most recent command containing abc
^old^new
Re-run previous command, replacing first old with new
!!:gs/old/new/
Re-run previous command, global substitution
bash
# Forgot sudo?$ apt updateE: Permission denied$ sudo !! # expands to: sudo apt update# Typo in long command?$ systemctl restrat nginx$ ^restrat^restart# Reuse last argument$ mkdir /opt/myapp$ cd !$ # cd /opt/myapp
Bash History Management
bash
# View / searchhistory # show full historyhistory 20 # last 20history | grep ssh # filter# Re-run by number!457 # run history entry 457!-2 # run 2 commands ago# Delete entrieshistory -d 457 # delete entry 457history -c # clear ALL history (session)history -w # write current session to file# Tune behavior in ~/.bashrcHISTSIZE=10000 # in-memory entriesHISTFILESIZE=20000 # on-disk entriesHISTCONTROL=ignoreboth:erasedups # skip dupes & lines starting with spaceHISTTIMEFORMAT="%F %T " # add timestampsHISTIGNORE="ls:ll:pwd:exit:clear:history" # never record theseshopt -s histappend # append, don't overwriteshopt -s cmdhist # multi-line commands as one entry# Share history across sessions in real time:PROMPT_COMMAND='history -a; history -n'
Tip: Prepending a command with a space (when HISTCONTROL includes ignorespace or ignoreboth) keeps it out of history — handy when typing one-off commands with secrets.
Tab Completion
Trigger
What it completes
Tab (once)
Complete if unambiguous
Tab Tab (double)
Show all possible completions
After cd
Directories only
After ssh / scp
Hostnames from ~/.ssh/config and ~/.ssh/known_hosts
# Enable programmable completion (most distros do this by default)# /etc/bash_completion or in ~/.bashrc:[ -r /usr/share/bash-completion/bash_completion ] && \ . /usr/share/bash-completion/bash_completion# Add tool-specific completionsource <(kubectl completion bash)source <(helm completion bash)complete -F __start_kubectl k # so 'k <tab>' also works# Show completions for a commandcomplete -p git
20+ Useful Aliases for ~/.bashrc
bash
# ---------- Navigation & Listing ----------alias ll='ls -lah --color=auto' # long, human, hidden, coloredalias la='ls -A --color=auto' # show dotfiles (no . and ..)alias l='ls -CF --color=auto' # quick column listingalias ..='cd ..' # up onealias ...='cd ../..' # up twoalias ....='cd ../../..' # up threealias -- -='cd -' # toggle to previous dir# ---------- Safety ----------alias rm='rm -i' # ask before removealias cp='cp -i' # ask before overwritealias mv='mv -i' # ask before overwritealias mkdir='mkdir -pv' # parents + verbose# ---------- Search / Filters ----------alias grep='grep --color=auto' # colored matchesalias egrep='egrep --color=auto'alias fgrep='fgrep --color=auto'# ---------- Disk / System ----------alias df='df -hT' # human + filesystem typealias du='du -h --max-depth=1' # human, one levelalias free='free -h' # human memoryalias ports='ss -tulnp' # what's listening?alias psg='ps aux | grep -v grep | grep -iE' # process search# ---------- Network ----------alias myip='curl -s https://ifconfig.me && echo' # public IPalias localip="ip -4 -br a | grep -v '^lo'" # local IPsalias weather='curl -s wttr.in/?format=3' # quick weatheralias headers='curl -sI' # response headers only# ---------- Git ----------alias g='git'alias gs='git status -sb' # short branch + statusalias gco='git checkout'alias gcm='git commit -m'alias gp='git pull --rebase'alias gpush='git push'alias gl='git log --oneline --graph --decorate -20'alias gd='git diff'# ---------- DevOps Tools ----------alias k='kubectl' # the universal short formalias kgp='kubectl get pods'alias kgs='kubectl get svc'alias kdp='kubectl describe pod'alias kx='kubectl config use-context'alias kns='kubectl config set-context --current --namespace'alias d='docker'alias dc='docker compose'alias dps='docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'alias tf='terraform'alias jc='journalctl -u' # usage: jc nginx -falias sc='systemctl' # usage: sc status nginx# ---------- Misc Quality of Life ----------alias path='echo -e ${PATH//:/\\n}' # print PATH one entry per linealias now='date "+%Y-%m-%d %H:%M:%S"' # quick timestampalias reload='source ~/.bashrc' # re-source after editsalias ports-listening='sudo lsof -i -P -n | grep LISTEN'alias ..git='cd $(git rev-parse --show-toplevel)' # jump to repo root
Rationale. Aliases like ll, .., k, d, gs shave 5–10 keystrokes off the dozens of times you type them daily. Safety aliases (rm -i, cp -i) save you from career-ending mistakes — yes, even seniors have done rm -rf / at 2 AM. The myip/localip/ports ones replace “wait, what’s that command again?” with one word.
To make aliases work in non-interactive contexts (rare but useful for scripts), put them in ~/.bash_aliases and source that file from ~/.bashrc.
Useful Bash Functions
bash
# --- mkcd: make a directory and cd into it in one step ---mkcd() { if [[ -z "$1" ]]; then echo "Usage: mkcd <dir>" >&2 return 1 fi mkdir -p -- "$1" && cd -P -- "$1"}# Usage: mkcd /opt/projects/new-app# --- extract: smart extractor for any common archive ---extract() { if [[ -z "$1" ]]; then echo "Usage: extract <archive>" >&2 return 1 fi if [[ ! -f "$1" ]]; then echo "extract: '$1' is not a regular file" >&2 return 1 fi case "$1" in *.tar.bz2|*.tbz2) tar -xjvf "$1" ;; *.tar.gz|*.tgz) tar -xzvf "$1" ;; *.tar.xz|*.txz) tar -xJvf "$1" ;; *.tar.zst) tar --zstd -xvf "$1" ;; *.tar) tar -xvf "$1" ;; *.bz2) bunzip2 "$1" ;; *.gz) gunzip "$1" ;; *.xz) unxz "$1" ;; *.zip) unzip "$1" ;; *.rar) unrar x "$1" ;; *.7z) 7z x "$1" ;; *.Z) uncompress "$1" ;; *) echo "extract: don't know how to handle '$1'" >&2; return 1 ;; esac}# Usage: extract release-v2.4.1.tar.gz# --- bak: quick timestamped backup of a file ---bak() { if [[ -z "$1" ]]; then echo "Usage: bak <file>" >&2 return 1 fi if [[ ! -e "$1" ]]; then echo "bak: '$1' does not exist" >&2 return 1 fi local ts ts=$(date +%Y%m%d-%H%M%S) cp -a -- "$1" "$1.bak.$ts" && \ echo "Backed up to $1.bak.$ts"}# Usage: bak /etc/nginx/nginx.conf# -> /etc/nginx/nginx.conf.bak.20260526-101530# --- Bonus: cdf -- cd into the directory of a file ---cdf() { if [[ -z "$1" ]]; then echo "Usage: cdf <file>" >&2 return 1 fi cd -- "$(dirname -- "$1")"}# --- Bonus: fkill -- fuzzy kill (needs fzf) ---fkill() { local pid pid=$(ps -ef | sed 1d | fzf -m | awk '{print $2}') [[ -n "$pid" ]] && echo "$pid" | xargs kill -"${1:-15}"}
Drop these into ~/.bashrc (or a sourced ~/.bash_functions file), run source ~/.bashrc, and they become first-class commands. The extract function alone will save you “wait, was it xjvf or xzvf?” lookups for the rest of your career. Set it up once, and it pays off for life.
This is where the real DevOps game begins. Containers, kernel tuning, boot internals, LVM, RAID — this is the layer that completes the transition from frontend developer to six-figure (USD) DevOps engineer. An interviewer will ask you: “How does a Docker container work under the hood?” — if you can explain namespaces + cgroups + overlayfs + capabilities, you’re already in the top 20% of candidates.
Containers Are Not Magic — They Are Linux Features
A container = namespaces (isolation) + cgroups (resource limits) + overlayfs (layered FS) + capabilities (privilege control). Docker, containerd, Podman — they all sit on this foundation. You don’t need a full hypervisor like a VM; the kernel is shared with the host, but each process sees an isolated world as if it had its own OS.
Linux Namespaces — A Process Gets Its Own Universe
Namespaces show a process group only a subset of the system. Inside a container, a process thinks it’s PID 1, with its own network stack and its own filesystem root.
Namespace
Flag
What It Isolates
Practical Effect
pid
CLONE_NEWPID
Process IDs
The container’s PID 1 might be PID 14523 on the host
mnt
CLONE_NEWNS
Mount points / filesystem view
The container sees its own /, not the host’s /
net
CLONE_NEWNET
Network interfaces, routes, iptables, sockets
Its own eth0, its own loopback, its own iptables rules
ipc
CLONE_NEWIPC
SysV IPC, POSIX message queues
Shared memory segments isolated
uts
CLONE_NEWUTS
Hostname, domain name
Each container gets its own hostname (hostname command)
user
CLONE_NEWUSER
UID/GID mappings
The container’s root (UID 0) maps to unprivileged user 100000 on the host
cgroup
CLONE_NEWCGROUP
Cgroup root view
The container sees only a subset of the cgroup hierarchy
Manual Namespace Creation — Build a Container Without Docker
bash
# Create a new PID + network + mount namespacesudo unshare --pid --net --mount --uts --ipc --fork --mount-proc bash# Go inside and checkecho $$ # Shows PID 1 (it's something else on the host)hostname container-demo # Changes only inside this namespaceps aux # Only the processes in this namespaceip addr # Only loopback (network is isolated)# Everything is cleaned up on exitexit
This is exactly what Docker does — it just adds an image, overlayfs, and orchestration on top.
Inspect Namespaces — lsns and /proc
bash
# List all namespaceslsns# Specific typelsns -t netlsns -t pid# Look at your own shell's namespacesls -l /proc/$$/ns/# Output:# lrwxrwxrwx 1 user user 0 May 26 10:23 cgroup -> 'cgroup:[4026531835]'# lrwxrwxrwx 1 user user 0 May 26 10:23 ipc -> 'ipc:[4026531839]'# lrwxrwxrwx 1 user user 0 May 26 10:23 mnt -> 'mnt:[4026531840]'# lrwxrwxrwx 1 user user 0 May 26 10:23 net -> 'net:[4026531992]'# lrwxrwxrwx 1 user user 0 May 26 10:23 pid -> 'pid:[4026531836]'# lrwxrwxrwx 1 user user 0 May 26 10:23 user -> 'user:[4026531837]'# lrwxrwxrwx 1 user user 0 May 26 10:23 uts -> 'uts:[4026531838]'# Enter a container's namespace (once you have the Docker PID)sudo nsenter --target 14523 --pid --net --mount
cgroups v2 (Ubuntu 24.04 Default)
cgroups (control groups) limit and account for the resource usage of processes. Ubuntu 24.04 uses the unified cgroup v2 — a single hierarchy at /sys/fs/cgroup.
bash
# Verify cgroup v2mount | grep cgroup# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)# Available controllerscat /sys/fs/cgroup/cgroup.controllers# cpu io memory pids hugetlb cpuset rdma misc# Create a new cgroup and set limitssudo mkdir /sys/fs/cgroup/myappecho "+cpu +memory +io +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control# CPU: 50% of one core (50000 us out of 100000 us period)echo "50000 100000" | sudo tee /sys/fs/cgroup/myapp/cpu.max# Memory: 512 MB hard limitecho "536870912" | sudo tee /sys/fs/cgroup/myapp/memory.max# PIDs: max 100 processesecho "100" | sudo tee /sys/fs/cgroup/myapp/pids.max# IO: weight (10-1000, default 100), per-device limits also possibleecho "default 50" | sudo tee /sys/fs/cgroup/myapp/io.weight# Put a process into the cgroupecho $$ | sudo tee /sys/fs/cgroup/myapp/cgroup.procs# Check current usagecat /sys/fs/cgroup/myapp/memory.currentcat /sys/fs/cgroup/myapp/cpu.stat
Tip: Docker’s --memory 512m --cpus="0.5" writes to exactly these cgroup files internally. A Kubernetes pod’s resources.limits also end up here via kubelet → containerd → runc.
Linux Capabilities — The Modern Replacement for Setuid
In old Unix, a binary was either root (UID 0, all-powerful) or a normal user. Linux capabilities break root’s powers into 40+ smaller chunks. ping needs a raw socket — it used to be setuid-root, now it’s given only CAP_NET_RAW.
bash
# Look at your shell's capabilitiescapsh --print# Look at a file's capabilitiesgetcap /usr/bin/ping# /usr/bin/ping cap_net_raw=ep# Give Nginx the capability to bind port 80 (a privileged port) without running as rootsudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/nginxgetcap /usr/local/bin/nginx# /usr/local/bin/nginx cap_net_bind_service=ep# Common capabilities# CAP_NET_BIND_SERVICE - bind ports < 1024# CAP_NET_ADMIN - network config (ip, iptables)# CAP_SYS_ADMIN - "almost root" — very powerful, be careful# CAP_CHOWN - change file ownership# CAP_DAC_OVERRIDE - bypass permissions# CAP_KILL - send a signal to any process# CAP_SETUID/SETGID - change UID/GID# Remove a capabilitysudo setcap -r /usr/local/bin/nginx# Docker drops a lot of caps by defaultdocker run --rm alpine sh -c 'apk add -q libcap; capsh --print' | grep Current# Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,...# (NOT full root capabilities)
# Check the current valuesysctl vm.swappinesssysctl net.core.somaxconn# Runtime change (gone after reboot)sudo sysctl -w vm.swappiness=10sudo sysctl -w net.core.somaxconn=4096# Persist it — create a file in /etc/sysctl.d/sudo tee /etc/sysctl.d/99-devops-tuning.conf <<'EOF'# Memory: prefer dropping cache over swapping (DB/app servers)vm.swappiness = 10vm.dirty_ratio = 15vm.dirty_background_ratio = 5vm.overcommit_memory = 1# Network: high-connection web servernet.core.somaxconn = 65535net.core.netdev_max_backlog = 5000net.ipv4.tcp_max_syn_backlog = 8192net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_fin_timeout = 15net.ipv4.ip_local_port_range = 1024 65535# File descriptorsfs.file-max = 2097152fs.inotify.max_user_watches = 524288# Connection tracking (for nat/firewalls)net.netfilter.nf_conntrack_max = 1048576EOF# Apply itsudo sysctl --system# All current valuessysctl -a | less
ulimit — Per-Process Resource Limits
bash
# Current limitsulimit -a# Open files limit (the most common bottleneck)ulimit -n # softulimit -Hn # hard# Raise it temporarily (current shell only)ulimit -n 65535# Permanent — /etc/security/limits.conf or /etc/security/limits.d/sudo tee /etc/security/limits.d/99-app.conf <<'EOF'# domain type item value* soft nofile 65535* hard nofile 1048576* soft nproc 32768* hard nproc 65535www-data soft nofile 100000www-data hard nofile 200000EOF# For systemd services — in the service unit file# [Service]# LimitNOFILE=1048576# LimitNPROC=65535
I/O Schedulers
bash
# Look at the current schedulercat /sys/block/sda/queue/scheduler# [mq-deadline] kyber bfq none# 'none' is best for NVMe — NVMe has its own queue managementecho none | sudo tee /sys/block/nvme0n1/queue/scheduler# For SATA SSD / HDD use 'mq-deadline' or 'bfq'echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler# Persist via udev rulesudo tee /etc/udev/rules.d/60-ioschedulers.rules <<'EOF'# NVMeACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"# SSDACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"# HDDACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"EOF
Caution: The most common cause of production outages — ulimit -n set too low. Under load, Node.js / Java apps crash at 1024 file descriptors with “EMFILE: too many open files”. Always set it to 65535+.
Boot Process Deep Dive
Press the power button → kernel running → systemd → login prompt. There are many layers in between:
Stage
UEFI Boot
Legacy BIOS
1. Firmware
UEFI firmware (modern)
BIOS (legacy)
2. Boot loader location
EFI System Partition (ESP), FAT32, mounted at /boot/efi
# Check whether you booted in UEFI models /sys/firmware/efi && echo "UEFI" || echo "BIOS"# Look at the ESP partitiondf -h /boot/efimount | grep /boot/efi# /dev/nvme0n1p1 on /boot/efi type vfat ...# EFI boot entriessudo efibootmgr -v# BootCurrent: 0001# Boot0001* ubuntu HD(1,GPT,...)/File(\EFI\ubuntu\shimx64.efi)
GRUB2 Configuration
bash
# Edit defaultssudo nano /etc/default/grub# Common settings:GRUB_DEFAULT=0GRUB_TIMEOUT=5GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"GRUB_CMDLINE_LINUX=""# Add a kernel parameter (example: disable IPv6, enable cgroup v2 — already default in 24.04)GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ipv6.disable=1"# Regenerate grub.cfgsudo update-grub# OR: sudo grub-mkconfig -o /boot/grub/grub.cfg# Rebuild initramfs (after a driver change or crypto change)sudo update-initramfs -u -k all# Kernel command line currently runningcat /proc/cmdline
initramfs (initial RAM filesystem)
Right after the kernel boots, it needs drivers to mount the real root filesystem (NVMe, RAID, LVM, encrypted disk). All of this happens in a temporary RAM-based filesystem — the initramfs. /boot/initrd.img-* is a compressed cpio archive.
bash
# See what's insidelsinitramfs /boot/initrd.img-$(uname -r) | head -50# Rebuildsudo update-initramfs -u -k all
systemd Boot Targets & Analysis
bash
# Current targetsystemctl get-default # graphical.target or multi-user.target# Set itsudo systemctl set-default multi-user.target# Total boot time for a serversystemd-analyze# Startup finished in 3.124s (kernel) + 8.456s (userspace) = 11.580s# Which service is slow?systemd-analyze blame# 4.123s snapd.service# 2.456s cloud-init.service# 1.234s apt-daily.service# Critical path (the serial dependency chain of boot)systemd-analyze critical-chainsystemd-analyze critical-chain nginx.service# Visual SVG plotsystemd-analyze plot > /tmp/boot-plot.svg# Open it in a browser — you get a beautiful timeline
LVM abstracts your disks. You pool multiple physical disks (PVs) into a group (VG), and carve flexible chunks (LVs) out of it — runtime resize, snapshots, and online migration all come for free.
The LVM stack, top to bottom: Filesystems (/ ext4, /var/lib/docker, /data xfs) sit on LVs (lv-root 20G, lv-docker 50G, lv-data 200G), which are carved from the VG (vg_main — Volume Group, 500 GB total, 230 GB free), which pools the PVs (/dev/nvme0n1p3 200G, /dev/nvme1n1 200G, /dev/nvme2n1 100G — physical disks / partitions / raw block devices).
Full End-to-End LVM Example
bash
# 1. Create Physical Volumes (turn raw disks into LVM-managed devices)sudo pvcreate /dev/nvme1n1 /dev/nvme2n1sudo pvs# 2. Create a Volume Group (a pool of PVs)sudo vgcreate vg_main /dev/nvme1n1 /dev/nvme2n1sudo vgssudo vgdisplay vg_main# 3. Create Logical Volumessudo lvcreate -L 50G -n lv-docker vg_mainsudo lvcreate -L 200G -n lv-data vg_main# Or all the free space:# sudo lvcreate -l 100%FREE -n lv-data vg_mainsudo lvs# 4. Create filesystemssudo mkfs.ext4 /dev/vg_main/lv-dockersudo mkfs.xfs /dev/vg_main/lv-data# 5. Mount themsudo mkdir -p /var/lib/docker /datasudo mount /dev/vg_main/lv-docker /var/lib/dockersudo mount /dev/vg_main/lv-data /data# 6. Add to /etc/fstab (UUID is best practice)echo "/dev/vg_main/lv-docker /var/lib/docker ext4 defaults,noatime 0 2" | sudo tee -a /etc/fstabecho "/dev/vg_main/lv-data /data xfs defaults,noatime 0 2" | sudo tee -a /etc/fstab# ===== NOW THE LIVE EXTEND — this is the real power of LVM =====# The disk is filling up. Grow lv-data by another 100Gsudo lvextend -L +100G /dev/vg_main/lv-datasudo xfs_growfs /data # XFS online grow# For ext4:# sudo lvextend -L +50G /dev/vg_main/lv-docker# sudo resize2fs /dev/vg_main/lv-docker# Combined (lvextend + filesystem resize in a single command):sudo lvextend -r -L +50G /dev/vg_main/lv-docker# Out of space in the VG? Add a new disksudo pvcreate /dev/nvme3n1sudo vgextend vg_main /dev/nvme3n1sudo lvextend -r -L +500G /dev/vg_main/lv-data# ===== SNAPSHOTS — for backups =====sudo lvcreate -L 10G -s -n lv-data-snap /dev/vg_main/lv-datasudo mount -o ro /dev/vg_main/lv-data-snap /mnt/snapshottar czf /backup/data-$(date +%F).tar.gz -C /mnt/snapshot .sudo umount /mnt/snapshotsudo lvremove -f /dev/vg_main/lv-data-snap
Tip: Cloud relevance — why LVM on AWS? To stripe or pool multiple EBS volumes. Take 4×500GB gp3 volumes on an instance and make a single 2TB volume with LVM — combined IOPS goes up and growing it is easy.
RAID — Redundancy and Performance
RAID Level
Min Disks
Redundancy
Capacity
Read/Write
Use Case
RAID 0 (stripe)
2
None — 1 fail = data gone
100%
Fastest R + W
Scratch / cache (cloud: avoid)
RAID 1 (mirror)
2
1 disk fail OK
50%
Fast R, normal W
OS disk, boot
RAID 5 (parity)
3
1 disk fail OK
(N-1)/N
Good R, slow W (parity calc)
Bulk storage (legacy)
RAID 6 (double parity)
4
2 disks fail OK
(N-2)/N
Good R, slower W
Large arrays, long rebuilds
RAID 10 (1+0)
4
1 per mirror pair OK
50%
Fastest R + W with redundancy
Databases, production
bash
# Software RAID with mdadmsudo apt install mdadm# RAID 10 from 4 diskssudo mdadm --create /dev/md0 --level=10 --raid-devices=4 \ /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1# Statuscat /proc/mdstatsudo mdadm --detail /dev/md0# Save config (otherwise it disappears on boot)sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.confsudo update-initramfs -u# Use as an LVM PV or a direct filesystemsudo mkfs.xfs /dev/md0
Caution: Cloud reality — AWS EBS / GCP PD / Azure Managed Disks are already replicated (3 copies cross-AZ). You rarely run RAID in the cloud — the provider already does it. Just attach multiple EBS volumes + LVM stripe for IOPS — that’s the cloud pattern. mdadm is only critical on bare-metal / on-prem.
Filesystem Tuning
bash
# ext4: lower reserved blocks (root-only) from 5% to 1% — save space on big data diskssudo tune2fs -m 1 /dev/vg_main/lv-data# Turn off forced fsck interval / mount count (production servers — avoid fsck on boot)sudo tune2fs -c 0 -i 0 /dev/vg_main/lv-data# Show all optionssudo tune2fs -l /dev/vg_main/lv-data# Mount options that matter in production# noatime — don't update atime on every read (huge perf gain, esp. for DBs, web)# nodiratime — skip directory atime# discard — SSD TRIM on delete (alternative: fstrim weekly via timer)# nofail — don't fail boot if the disk is missing (cloud detached EBS scenario)# nosuid,nodev,noexec — security for /tmp, /var# /etc/fstab example for production# UUID=... /data ext4 defaults,noatime,nodiratime,nofail 0 2# UUID=... /tmp tmpfs defaults,nosuid,nodev,noexec,size=2G 0 0# tmpfs (RAM-backed) in production — fast scratch / build directoriessudo mount -t tmpfs -o size=4G,mode=1777 tmpfs /var/cache/build
Note: Quick recap — A Docker container = unshare + cgroups + overlayfs + capability drop. Boot = UEFI → GRUB → kernel → initramfs → systemd. LVM = flexible disks, runtime grow. RAID = redundancy. You rarely need RAID in the cloud, but LVM and tuning are useful everywhere. All of these show up in interviews as the answer to “how does X work internally?”
So far we’ve looked at Linux on a single isolated machine. In the cloud the same Linux is running, but with different patterns — ephemeral instances, the metadata service, cloud-init bootstrapping, EBS attach/grow, replacing SSH with SSM. A developer-turned-DevOps engineer spends roughly 70% of each day on AWS Linux instances + Kubernetes nodes. This section covers that 70%.
Tip: Practical pick — Production Kubernetes nodes → Amazon Linux 2023 (also consider Bottlerocket). General app servers → Ubuntu 24.04. Enterprise compliance → RHEL/Rocky.
Instance Metadata Service (IMDS) — The Cloud’s /proc
Every EC2 instance gets a special endpoint: http://169.254.169.254. From here the instance can learn everything about itself — region, AZ, instance type, IAM role credentials, user-data, public IP. SDKs (boto3, aws-cli) also pull their IAM creds from here.
Caution: IMDSv1 is vulnerable to SSRF attacks — the Capital One breach happened through it. Always enforce IMDSv2 (HttpTokens=required) in the instance metadata options.
EBS Volumes — Attach, Format, Mount, Grow
bash
# After attaching a new volume, look at the device listlsblk# NAME SIZE TYPE MOUNTPOINTS# nvme0n1 30G disk# `-nvme0n1p1 30G part /# nvme1n1 100G disk <-- new EBS volume# No filesystem on it yetsudo file -s /dev/nvme1n1# /dev/nvme1n1: data (i.e. empty)# Create a filesystemsudo mkfs.ext4 /dev/nvme1n1# Modern alternative: sudo mkfs.xfs /dev/nvme1n1# Create a mountpoint and mount itsudo mkdir -p /datasudo mount /dev/nvme1n1 /data# /etc/fstab — UUID best practice; nofail critical for cloud# (if the instance reboots with the volume detached, boot won't fail)UUID=$(sudo blkid -s UUID -o value /dev/nvme1n1)echo "UUID=$UUID /data ext4 defaults,nofail,noatime 0 2" | sudo tee -a /etc/fstab# Verifysudo mount -adf -h /data
Grow EBS Volume In-Place (Zero Downtime)
bash
# Step 1: Increase the volume size from the AWS console or CLIaws ec2 modify-volume --volume-id vol-0abc123 --size 200# Wait for "optimizing" stateaws ec2 describe-volumes-modifications --volume-id vol-0abc123# Step 2: Inside the instance, the kernel sees the new sizelsblk# nvme1n1 200G disk# `-nvme1n1p1 100G part /data <-- partition is still 100G# Step 3: Grow the partition (if you're using a partition)sudo growpart /dev/nvme1n1 1# Filesystem directly on the raw device (no partition) — skip this step# Step 4: Grow the filesystem# ext4:sudo resize2fs /dev/nvme1n1p1# OR direct:sudo resize2fs /dev/nvme1n1# xfs (must be mounted):sudo xfs_growfs /data# Verifydf -h /data# /dev/nvme1n1 200G ...
Instance Store vs EBS
EBS
Instance Store
Persistence
Persistent — survives stop/start
Ephemeral — gone on stop/terminate
Replication
3× within AZ
None
Speed
Network-attached (NVMe over fabric)
Physically attached NVMe SSD — fastest
Snapshots
Yes
No
Detach & reattach
Yes
No
Use case
OS root, databases, app data
Cache, scratch, temp big-data shuffle
SSH to Cloud Instances
bash
# Classic: keypair-based SSHchmod 400 ~/.ssh/my-key.pemssh -i ~/.ssh/my-key.pem ec2-user@54.221.10.45 # Amazon Linuxssh -i ~/.ssh/my-key.pem ubuntu@54.221.10.45 # Ubuntu# A convenient alias in ~/.ssh/configcat >> ~/.ssh/config <<'EOF'Host prod-web HostName 54.221.10.45 User ubuntu IdentityFile ~/.ssh/my-key.pem ServerAliveInterval 60EOFssh prod-web# Bastion host pattern (private instance via public jump box)ssh -i key.pem -J bastion.example.com ubuntu@10.0.5.20# Or ProxyJump in your configHost prod-db HostName 10.0.5.20 User ubuntu IdentityFile ~/.ssh/my-key.pem ProxyJump bastion.example.com# ===== MODERN: SSM Session Manager (NO PORT 22, NO KEYS) =====# Pre-requisites: SSM agent on instance, IAM role with AmazonSSMManagedInstanceCoreaws ssm start-session --target i-0abc123def456# Port forwarding via SSM (RDS access through private instance)aws ssm start-session \ --target i-0abc123def456 \ --document-name AWS-StartPortForwardingSessionToRemoteHost \ --parameters '{"host":["mydb.cluster-xyz.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["15432"]}'# SSH over SSM (best of both)# ~/.ssh/config:Host i-* mi-* ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"ssh ec2-user@i-0abc123def456
Tip: Production gold-standard — never open port 22 in your security groups. Use SSM Session Manager — no public IP, no SSH key management, full audit log in CloudTrail, IAM-based access control. This is the modern DevOps practice.
cloud-init — First Boot Bootstrap
cloud-init is Linux’s standard first-boot configuration tool. It runs on AWS, GCP, Azure, OpenStack — everywhere. The user-data field you provide at launch time is parsed and executed by cloud-init.
bash
# Status checkcloud-init status# status: donecloud-init status --longcloud-init analyze show # boot time breakdown# Config filesls /etc/cloud/# cloud.cfg main config# cloud.cfg.d/ drop-ins (vendor + user)# templates/ hostname etc.# Logs (THE MOST IMPORTANT thing for debugging)sudo tail -f /var/log/cloud-init.log # detailedsudo tail -f /var/log/cloud-init-output.log # stdout/stderr of user-data# Re-run cloud-init (for testing)sudo cloud-init clean --logssudo cloud-init initsudo cloud-init modules --mode=configsudo cloud-init modules --mode=final
Format 1: Bash Script User-Data
bash
#!/bin/bash# user-data.sh — simple bash, runs as root on first bootset -euxo pipefailexec > >(tee /var/log/user-data.log | logger -t user-data -s 2>/dev/console) 2>&1echo "=== Starting bootstrap at $(date) ==="# Idempotency guardif [ -f /var/lib/bootstrap-done ]; then echo "Already bootstrapped, exiting" exit 0fi# Update + installapt-get update -yapt-get upgrade -yapt-get install -y nginx awscli docker.io curl jq# Enable servicessystemctl enable --now nginx dockerusermod -aG docker ubuntu# Pull config from S3 (IAM role attached)aws s3 cp s3://my-config-bucket/nginx.conf /etc/nginx/nginx.confsystemctl reload nginx# Mark donetouch /var/lib/bootstrap-doneecho "=== Bootstrap complete at $(date) ==="
In production, instead of doing a full user-data setup on every deploy, you use pre-baked AMIs — boot in 30 seconds instead of 5 minutes. But before creating an AMI, a “sysprep” (clean) is essential.
bash
# Pre-AMI cleanup script (run before creating image)#!/bin/bashset -e# Stop services that hold statesudo systemctl stop docker nginx# Remove SSH host keys (they'll regenerate on the instance's first boot)sudo rm -f /etc/ssh/ssh_host_*# Clear machine-id (cloud-init will regenerate)sudo truncate -s 0 /etc/machine-idsudo rm -f /var/lib/dbus/machine-idsudo ln -s /etc/machine-id /var/lib/dbus/machine-id# Clear cloud-init statesudo cloud-init clean --logs --seed# Clear logssudo find /var/log -type f -exec truncate -s 0 {} \;sudo rm -rf /var/log/journal/*# Clear bash historycat /dev/null > ~/.bash_historyhistory -csudo rm -f /root/.bash_history# Clear authorized_keys for default user (cloud-init re-injects)sudo rm -f /home/ubuntu/.ssh/authorized_keys# Trim filesystem (smaller snapshot)sudo fstrim -av# Now from your laptop/CI:aws ec2 create-image \ --instance-id i-0abc123 \ --name "web-base-$(date +%Y%m%d-%H%M)" \ --description "Nginx + Docker baseline" \ --no-reboot=false# Packer (recommended for repeatable builds) — basic config# packer build webserver.pkr.hcl# Packer launches temp instance, runs provisioners, creates AMI, terminates.
Don’t patch the server — launch a new instance from a new AMI, shift traffic, terminate the old one. Configuration drift = zero. Rollback = re-launch the previous AMI. This is the modern way.
2. cloud-init + Configuration Management Combo
bash
#cloud-config# cloud-init only "bootstraps Ansible/Chef" — that tool does the restpackage_update: truepackages: [ansible, git]runcmd: - git clone https://github.com/myorg/ansible-playbooks.git /opt/ansible - ansible-pull -U https://github.com/myorg/ansible-playbooks.git -i localhost, site.yml
3. Centralized Logging
Local logs on an instance (/var/log/*) are ephemeral — when the instance is terminated, the logs are gone. In production, always ship your logs:
# Security group inbound rules: (none — yes, ZERO)# Instances are in private subnet# Access only via:aws ssm start-session --target i-0abc123# And if you need the SSH protocol (scp, rsync, git):ssh ec2-user@i-0abc123 # ProxyCommand via SSM (config shown earlier)scp -r ./code/ ec2-user@i-0abc123:/home/ec2-user/
6. Putting It All Together — A Realistic Stack
bash
# Terraform spins up:# - VPC with private subnets# - Auto Scaling Group with Launch Template# - Launch Template references golden AMI (built by Packer)# - user-data does minimal "ansible-pull" for last-mile config# - IAM role attached with SSM + CloudWatch + S3 read# - Application Load Balancer in public subnet# - Security groups: ALB → instances on app port only, port 22 NEVER open# - EBS volumes with daily DLM snapshots# - CloudWatch agent ships /var/log + custom metrics# - Logs to CloudWatch Logs → subscription filter → Datadog/Splunk# Deploys happen by:# 1. Packer builds new AMI on git push# 2. Terraform updates Launch Template's AMI ID# 3. ASG instance refresh rolls instances one-by-one# 4. Old AMI kept for 7 days (instant rollback)# Day-2 ops:# - SSH via `aws ssm start-session`# - Logs in CloudWatch / Datadog# - Metrics in CloudWatch / Grafana# - Disk grow: modify-volume + growpart + resize2fs (zero downtime)# - Bigger instance: ASG rolling update with new instance type
Note: Wrap-up — Five things you’ll use daily with cloud Linux: (1) fetching instance data via IMDSv2, (2) EBS volume attach + format + mount + fstab nofail, (3) bootstrapping via cloud-init user-data, (4) keyless SSH access via SSM Session Manager, (5) the golden AMI + immutable infra pattern. You’ll repeat these so often they become muscle memory.
Tip: The interview question that lands you a senior-level (USD six-figure) offer — “Describe what happens from aws ec2 run-instances until an application is serving traffic.” Answer: an EBS root volume is created from the AMI snapshot, an ENI is attached, the instance boots (UEFI → GRUB → kernel → initramfs → systemd) → cloud-init pulls user-data from IMDS → packages install, config files are written, services start → the application binds to its port → the health check passes → the ALB target group marks it healthy → traffic routes. If you can explain that whole chain in 60 seconds, the job is yours.
You’ve built the foundation. Now you need to keep the muscle warm. This section is a curated map of where to go next — channels, platforms, books, and certifications that actually move the needle in the global DevOps market. Spend 30 minutes a day here for the next three months and you will be unrecognizable.
Tip: Study advice #1 — Pick ONE channel and finish it. Don’t channel-hop. Bookmark 50 videos and watch zero. Pick Abhishek Veeramalla’s Linux + DevOps playlist OR NetworkChuck’s Linux for hackers, and finish it end-to-end before you touch anything else.
YouTube channels worth your subscription
Concept-first creators (great for building intuition):
Abhishek Veeramalla — a gold standard for DevOps aspirants. His “DevOps Zero to Hero” playlist is one of the most recommended free resources in DevOps communities. Real interview questions, real scenarios.
Telusko (Navin Reddy) — clean Linux fundamentals, scripting basics, Docker primers. Great when you want a relaxed explanation.
Cyber Platter — strong on shell scripting, system administration, and Red Hat workflows. Useful for RHCSA prep.
Production-polished creators (depth + polish):
NetworkChuck — high-energy, hands-on. His “Linux for Hackers” series is the most fun way to fall in love with the terminal.
Learn Linux TV (Jay LaCroix) — calm, thorough, no-fluff. The Ubuntu Server and Bash scripting series are excellent.
TechWorld with Nana — best DevOps overview channel on YouTube. Her Docker, Kubernetes, and CI/CD crash courses are essentially free certification prep.
DistroTube — pure Linux culture, window managers, dotfiles, tooling. Watch when you want to fall deeper down the rabbit hole.
Free practice platforms (this is where skill actually grows)
OverTheWire — Bandit (overthewire.org/wargames/bandit) — 34 levels of SSH-driven Linux puzzles. If you can clear Bandit 0–20, you are operationally literate. This is non-negotiable.
SadServers (sadservers.com) — “broken Linux server” scenarios. You SSH in, diagnose, and fix. This is the closest thing to a real on-call shift you can practice for free.
Linux Journey (linuxjourney.com) — beautifully structured beginner-to-intermediate text path. Perfect for revision.
HackTheBox — privilege escalation and CTF-style Linux. Sharpens your understanding of permissions, processes, and networking.
TryHackMe — gentler than HTB. Their “Linux Fundamentals” and “Linux PrivEsc” rooms are interview-grade prep.
LeetCode Shell — small but mighty. Bash one-liners that show up in screening rounds (word frequency, transpose file, valid phone numbers).
Caution: Study advice #2 — Tutorial hell is real. For every 1 hour of video you watch, spend 2 hours in a terminal breaking things. Recruiters do not care that you watched 200 hours of content. They care that you can fix a full disk in 90 seconds on a shared screen.
Documentation — the most underrated tier
man — the original. man bash, man 5 crontab, man 7 signal. Learn to navigate with / and n.
tldr — community-maintained “just the examples” pages. Install with npm i -g tldr or apt install tldr. tldr tar beats reading the man page when you just need the syntax.
info — GNU’s hyperlinked docs. Heavier than man, but the coreutils info pages are gold.
DigitalOcean tutorials (digitalocean.com/community/tutorials) — the cleanest English Linux/server tutorials on the internet. Tagged by Ubuntu version. Bookmark this.
ArchWiki (wiki.archlinux.org) — even if you never touch Arch, the ArchWiki is the most accurate, distro-agnostic Linux reference that exists.
tldp.org — The Linux Documentation Project. Older but the “Advanced Bash-Scripting Guide” is a classic.
Books that earn shelf space
“How Linux Works” by Brian Ward (No Starch Press) — the single best book to understand why Linux behaves the way it does. Bootloaders, init, devices, networking. Read it twice.
“The Linux Command Line” by William Shotts — legally free PDF at linuxcommand.org/tlcl.php. The reference for shell mastery.
“Linux Bible” by Christopher Negus — fat, slightly dry, but covers RHEL/Fedora workflows thoroughly. Perfect companion for RHCSA.
“Site Reliability Engineering” (Google SRE Book) — free at sre.google/books. Not Linux-specific but it teaches you how to think like the engineers earning top-of-market (USD six-figure) salaries.
Bonus: “UNIX and Linux System Administration Handbook” by Nemeth et al. — the “purple book.” If you go senior sysadmin, this is the bible.
Certifications — what actually matters in the DevOps market
Ranked by ROI for someone at your stage (an experienced developer pivoting to DevOps, targeting six-figure USD roles):
RHCSA (Red Hat Certified System Administrator) — EX200. The most respected Linux cert in the industry. Hands-on exam, no MCQs. Recruiters at enterprises, Red Hat partners, and most product companies recognize it instantly. It isn’t cheap, but it’s worth it if your goal is Linux-heavy DevOps roles.
LFCS (Linux Foundation Certified Sysadmin). Vendor-neutral, performance-based, cheaper (~$300, often 50% off). Less brand recognition than RHCSA but globally respected.
CompTIA Linux+. Decent foundational cert. MCQ-based. Good for resumes scanned by ATS bots, weaker signal to senior engineers.
LPIC-1. Vendor-neutral, two exams. Solid content but lower brand pull than RHCSA.
Honest take:AWS Solutions Architect Associate (SAA-C03) + provable Linux skills (Bandit + a GitHub of scripts) beats any pure-Linux cert for six-figure (USD) DevOps roles. Hiring managers want cloud + Linux, not Linux alone. If you can only afford one cert this year, do SAA-C03 first, then RHCSA.
Online sandboxes (practice without burning your laptop)
Killercoda — instant Ubuntu/CentOS/K8s playgrounds in the browser. No signup needed for basic labs.
JSLinux (bellard.org/jslinux) — a full Linux kernel running in your browser via JavaScript. Useful for quick “what does this command do” checks on a locked-down machine.
DigitalOcean — $200 free credit for new accounts, droplets from ~$6/month after. Cleanest UX for spinning up real VMs.
Linode (Akamai) — similar to DO, $100 credit, slightly cheaper at the low end.
AWS EC2 free tier — t2.micro / t3.micro free for 12 months. This should be your daily-driver sandbox because it also teaches you AWS.
GCP free tier — e2-micro is free forever in select regions. Great for a persistent personal Linux server.
Note: Study advice #3 — Build in public. Push every script, every dotfile, every broken-then-fixed config to a GitHub repo called linux-journey-2026. Tweet/LinkedIn one thing you learned each week. In six months that repo + timeline is worth more than any certificate, because it’s proof of consistent practice — which is exactly what DevOps hiring managers screen for.
Linux is the floor, not the ceiling. Every DevOps tool you’ll touch — Docker, Kubernetes, Ansible, Terraform, AWS, GCP, CI/CD runners — is just Linux wearing a costume. Here’s how each next step builds directly on what you already know.
Linux → Bash (automate everything you just learned)
You already know cp, grep, find, awk, sed. Bash scripting is just gluing those into reusable files. The leap is small but career-changing: a sysadmin who writes scripts becomes a DevOps engineer.
Goal: be able to write a 50-line script with functions, argument parsing, error handling, and logging without Googling syntax.
Resource: “Bash Scripting Cheatsheet” (devhints.io/bash) + ShellCheck (shellcheck.net) for every script you write.
Project: write a backup script that tars a directory, uploads to S3, rotates old backups, and emails on failure. That one script demonstrates 80% of what mid-level DevOps does daily.
Linux → Docker (namespaces + cgroups + overlayfs, nothing magical)
Docker is not a separate technology. It’s three Linux kernel features in a trench coat: namespaces (process/network/mount isolation), cgroups (CPU/memory limits), and overlayfs (layered filesystems). You already learned about processes, mounts, and users — Docker is just those concepts with sharper edges.
7-day Docker plan:
Day 1: Install Docker. Run docker run -it ubuntu bash. Explore. Notice it feels exactly like a Linux box because it is one.
Day 2: Images vs containers. docker pull, docker ps -a, docker exec, docker logs. Build mental model.
Day 3: Write your first Dockerfile. Containerize a simple Node/React app (use your existing skills).
Day 4: Volumes and bind mounts. Networks (bridge, host, none). Port publishing.
Day 5: docker-compose. Spin up a 3-service stack (React + Node API + Postgres).
Day 6: Image optimization — multi-stage builds, alpine base images, .dockerignore. Cut a 1.2GB image to 80MB.
Day 7: Push to Docker Hub. Pull on a cloud VM. Run in production-like mode with restart policies and healthchecks.
Linux → Kubernetes (nodes are Linux, pods are processes)
A Kubernetes cluster is a fleet of Linux machines (nodes) running containers (pods, which are groups of Linux processes) coordinated by an API server. Everything you debug in K8s ultimately resolves to kubectl exec -it pod -- sh followed by the same Linux commands you already know.
Path:
Install minikube or kind locally. One-node cluster on your laptop.
Learn pods → deployments → services → ingress in that order. Don’t skip ahead.
Do the free Kubernetes Basics tutorial on kubernetes.io.
KillerCoda K8s scenarios — 30 minutes/day for 3 weeks.
Aim for CKAD (Certified Kubernetes Application Developer). It’s hands-on, performance-based, and can add a meaningful bump to your offer. CKA comes later when you’re ops-focused.
Linux → AWS (EC2 is Linux, Lambda is Linux, Fargate is Linux)
You will be shocked how much of “learning AWS” is actually “applying Linux on rented hardware.” An EC2 instance is a Linux VM. A Lambda function runs on Amazon Linux 2. ECS Fargate runs containers on Linux. RDS is Postgres/MySQL on Linux. The cloud is just someone else’s Linux box with an API in front.
Path:
AWS Solutions Architect Associate (SAA-C03) — your gateway cert. ~$150, three months of evening study. Stephane Maarek’s Udemy course is the standard.
Hands-on: build a 3-tier app (VPC + EC2 + RDS + S3 + CloudFront) and tear it down with Terraform.
AWS DevOps Engineer Professional — once you have 1+ year of real AWS experience. This is the cert that anchors senior-level (USD six-figure) conversations.
30-day Linux-to-DevOps ramp (one focused task per day)
Ready to apply what you’ve learned? Work through the Hands-on DevOps Projects to build real systems with Linux, Docker, and CI/CD.
Install Ubuntu 24.04 in a VM. Customize .bashrc, install zsh + oh-my-zsh.
Complete OverTheWire Bandit levels 0–10.
Complete Bandit levels 11–20.
Write a shell script that audits your home directory: file count, largest files, oldest files.
Master grep, awk, sed with the GNU “Mastering Text Processing” cheatsheet.
Set up SSH key auth between your laptop and a free-tier EC2 instance.
Write a systemd service unit for a Node.js app. Enable, start, check status, view logs with journalctl.
Configure UFW or iptables on your VM. Allow only SSH + HTTP/HTTPS.
Install nginx, serve a static site, configure a reverse proxy to a Node app on :3000.
Add Let’s Encrypt (certbot) for HTTPS on a free domain (DuckDNS or Freenom).
Write a cron job that backs up /etc nightly to a tarball.
Learn rsync. Sync the backups to an S3 bucket using the AWS CLI.
Read /proc/cpuinfo, /proc/meminfo. Write a script that summarizes system health.
Use strace on a misbehaving process. Identify which syscall is failing.
Install Docker. Run hello-world, then Ubuntu, then nginx.
Containerize your favorite React side project. Write the Dockerfile from scratch.
Multi-stage build: cut your image size by 80%.
Write a docker-compose.yml for React + Express + Postgres + Redis.
Push your image to Docker Hub. Pull and run it on the EC2 instance.
Install minikube locally. Deploy your app as a Kubernetes Deployment + Service.
Add a Kubernetes ConfigMap and Secret. Wire them to your pod.
Add Horizontal Pod Autoscaler. Generate load with hey or ab and watch it scale.
Write a Bash script that deploys your app: build image → push → kubectl apply → check rollout.
Set up GitHub Actions: on push to main, build Docker image and push to Docker Hub.
Extend GitHub Actions: SSH to EC2 and pull the new image (a tiny CD pipeline).
Sign up for AWS free tier. Launch an EC2 via the console, then via AWS CLI.
Create an S3 bucket. Upload, download, set lifecycle policy to delete after 30 days.
Write a Terraform file that creates a VPC + 1 EC2 + 1 S3 bucket. Apply, destroy, repeat.
Install Prometheus + Grafana on minikube. Scrape node metrics. Build one dashboard.
Final boss: deploy your React + Express + Postgres app to EC2, behind nginx with HTTPS, with a GitHub Actions pipeline that auto-deploys on push, and a cron-based DB backup to S3. Take a screenshot. Post it on LinkedIn with the GitHub repo. You are now a junior DevOps engineer.
30-60-90 day vision (and the 6-month payoff)
Day 30 — Linux comfort. You can SSH anywhere, debug a stuck process, write a 100-line bash script, and edit any system config without panic. You speak fluent terminal.
Day 60 — Docker + AWS basics. You’ve containerized 3+ apps, you understand VPC/EC2/S3/IAM, and you’ve deployed something real to production with HTTPS. You can answer “explain Docker to a junior” in an interview.
Day 90 — K8s + CI/CD interview-ready. You’ve passed (or can pass) CKAD-style scenarios, you have a working GitHub Actions pipeline in a public repo, and you’ve cleared 3-5 mock interviews. Target: junior/mid DevOps roles at strong product companies. With your background, recruiters will fast-track you because you bring product sense most freshers lack.
Month 6 — Mid-level inflection. One real production system on your resume, SAA-C03 passed, CKAD passed, contributing to an open-source DevOps tool. Target: mid-level DevOps / SRE / Platform Engineer roles. The frontend-to-DevOps story becomes your superpower — you understand what developers actually need from a platform, which most pure-ops engineers don’t.
Note: Prefer a day-by-day path? This is covered in Mission 90 Days 1–20 — a free 90-day guided DevOps program with browser terminal missions.
Real world: The pep talk you came here for. You are not late. The global DevOps market in 2026 is desperate for engineers who can talk to developers and ops — because most candidates can only do one. Your years of shipping real features is not a liability; it is a wedge. Recruiters will pay a premium for someone who has actually pushed code to production and now understands the pipeline that delivers it.
Six months from now, a hiring manager is going to look at your GitHub, see 180 days of consistent commits, see one solid end-to-end project, see SAA-C03 and maybe CKAD on the resume, and make you a strong offer without blinking. The only thing standing between you and that offer is showing up to the terminal every day between now and then.
Yes — the overwhelming majority of servers, containers, and CI runners are Linux. Comfort with the shell, the filesystem, permissions, processes, and systemd is foundational for any DevOps role.
What Linux skills do DevOps engineers need most?
Navigating the filesystem, file permissions and ownership, text processing (grep/awk/sed), managing services with systemd, package management, networking and SSH, and bash scripting for automation.
How do Linux file permissions work?
Each file has read/write/execute bits for owner, group, and others. They can be set symbolically (chmod u+x) or with octal digits (chmod 755), where 4=read, 2=write, 1=execute summed per class.
How do I use grep, awk, and sed for log analysis?
grep filters lines by pattern, awk extracts and computes over columns, and sed edits streams. Chaining them through pipes is the core log-wrangling workflow on Linux.
How do I manage services with systemd?
Use systemctl to start, stop, enable, and check the status of units, and journalctl to read their logs. Unit files in /etc/systemd/system define how services run.
How do I SSH into a remote Linux server securely?
Use key-based authentication instead of passwords, disable root login, change defaults in sshd_config, and use an SSH agent. ssh user@host connects; scp/rsync transfer files.