Text power tools — grep, sed & awk one-liners

grep, sed & awk: the log-wrangling trio

Section 1 of 5 · ~3 min

Almost every operations task eventually becomes the same problem: a file — or a live stream of log lines — is far too big to read by eye, and you need the three lines that matter, or you need to reshape every line into something cleaner. Three tools, present on every Linux box for decades, handle nearly all of it. Learn where each one fits and you stop scrolling through logs and start interrogating them.

grep finds lines. You give it a pattern and a file (or a pipe) and it prints only the matching lines — that is the whole job, and the one you run most. Four flags carry you a long way: -i matches case-insensitively, so Error and ERROR both hit; -n prefixes each match with its line number; -v inverts the match to show every line that does not contain the pattern (ideal for filtering out noise); and -r recurses a whole directory tree instead of one file. grep -rn error /var/log reads as “show me, with line numbers, every error under the log directory.”

sed edits a stream. Its workhorse is substitution: sed 's/old/new/' replaces the first old on each line, and adding a g — s/old/new/g — replaces every occurrence on the line. Because sed reads one line at a time and never loads the whole file, it rewrites a ten-gigabyte log as happily as a ten-line one. You reach for it to redact a secret, fix a typo across a config, or normalise text before the next tool sees it. One catch bites everyone: by default sed prints the result and leaves the file on disk untouched.

awk works in columns. It splits each line into fields on whitespace and numbers them $1, $2, $3… with $0 meaning the whole line and $NF the last field. awk '{print $1}' prints just the first column; awk '{print $3, $NF}' prints two. The moment your data has columns — an access log, ps output, a CSV — awk pulls out exactly the fields you want without a brittle pattern.

The real power is the division of labour: grep narrows, sed cleans, awk extracts. Tomorrow you pipe them into one line; today you learn the shape of each so you know which to reach for.

Real world: Picture a warehouse of paper delivery receipts. grep is the clerk who pulls only the receipts stamped “FAILED”. sed is the next clerk, who blacks out the card number on each one. awk is the last clerk, who copies just the order-ID column onto a fresh list. Three specialists on one line — nobody ever reads the whole warehouse.

A named example makes it concrete. When Netflix engineers chase a spike in HTTP 500s, they do not open a log viewer first — they grep " 500 " access.log to isolate the failed requests, then awk '{print $7}' to see which URLs are failing, then count them. That pipeline turns a million-line log into a ranked list of broken endpoints in seconds — exactly what an on-call engineer reaches for before the dashboards have even finished loading.

You will not memorise every flag today — you will learn find, edit, extract, so that when a log is on fire you already know which tool answers the question.

Hands-On Lab

Section 2 of 5 · ~4 min

Budget about 25 minutes. Open your WSL2 Ubuntu 24.04 terminal (if you have not set up WSL2 yet, do Day 0 first — macOS and Linux users can follow along in their built-in terminal). You will build a tiny log file, then interrogate it with each tool in turn — type every command yourself and read every line of output.

# 1. Make a sample log to practise on (6 lines, like a trimmed web-app log).
cat > ~/app.log <<'EOF'
2026-07-09 10:01 INFO user=alice GET /home 200
2026-07-09 10:02 ERROR user=bob POST /login 500
2026-07-09 10:03 WARN user=alice GET /cart 200
2026-07-09 10:04 error user=carol GET /pay 503
2026-07-09 10:05 INFO user=dave GET /home 200
2026-07-09 10:06 ERROR user=eve POST /login 500
EOF
cat ~/app.log
# Output:
# 2026-07-09 10:01 INFO user=alice GET /home 200
# 2026-07-09 10:02 ERROR user=bob POST /login 500
# 2026-07-09 10:03 WARN user=alice GET /cart 200
# 2026-07-09 10:04 error user=carol GET /pay 503
# 2026-07-09 10:05 INFO user=dave GET /home 200
# 2026-07-09 10:06 ERROR user=eve POST /login 500

# 2. grep finds lines. Print only the lines containing the exact pattern ERROR.
grep ERROR ~/app.log
# Output (note: lowercase "error" on line 4 does NOT match — grep is case-sensitive):
# 2026-07-09 10:02 ERROR user=bob POST /login 500
# 2026-07-09 10:06 ERROR user=eve POST /login 500

# 3. -i ignores case, so ERROR and error both match.
grep -i error ~/app.log
# Output:
# 2026-07-09 10:02 ERROR user=bob POST /login 500
# 2026-07-09 10:04 error user=carol GET /pay 503
# 2026-07-09 10:06 ERROR user=eve POST /login 500

# 4. -n prefixes each match with its line number — so you can jump straight to it.
grep -n ERROR ~/app.log
# Output:
# 2:2026-07-09 10:02 ERROR user=bob POST /login 500
# 6:2026-07-09 10:06 ERROR user=eve POST /login 500

# 5. -v INVERTS the match: every line that does NOT contain INFO (filtering out noise).
grep -v INFO ~/app.log
# Output:
# 2026-07-09 10:02 ERROR user=bob POST /login 500
# 2026-07-09 10:03 WARN user=alice GET /cart 200
# 2026-07-09 10:04 error user=carol GET /pay 503
# 2026-07-09 10:06 ERROR user=eve POST /login 500

# 6. -r recurses a whole directory tree. Make two log files, then search both at once.
mkdir -p ~/logs && cp ~/app.log ~/logs/api.log && cp ~/app.log ~/logs/web.log
grep -rn ERROR ~/logs
# Output (each hit shows file path : line number : line):
# /home/pushkar/logs/api.log:2:2026-07-09 10:02 ERROR user=bob POST /login 500
# /home/pushkar/logs/api.log:6:2026-07-09 10:06 ERROR user=eve POST /login 500
# /home/pushkar/logs/web.log:2:2026-07-09 10:02 ERROR user=bob POST /login 500
# /home/pushkar/logs/web.log:6:2026-07-09 10:06 ERROR user=eve POST /login 500

# 7. sed substitutes. s/old/new/ rewrites the first match on each line as text flows past.
sed 's/ERROR/CRITICAL/' ~/app.log
# Output (the two ERROR lines are rewritten; the stream is printed, not saved):
# 2026-07-09 10:01 INFO user=alice GET /home 200
# 2026-07-09 10:02 CRITICAL user=bob POST /login 500
# 2026-07-09 10:03 WARN user=alice GET /cart 200
# 2026-07-09 10:04 error user=carol GET /pay 503
# 2026-07-09 10:05 INFO user=dave GET /home 200
# 2026-07-09 10:06 CRITICAL user=eve POST /login 500

# 8. Prove it: the file on disk is UNCHANGED. sed printed to the screen, not to app.log.
cat ~/app.log
# Output (still ERROR — to save, redirect to a new file: sed '...' ~/app.log > clean.log):
# 2026-07-09 10:01 INFO user=alice GET /home 200
# 2026-07-09 10:02 ERROR user=bob POST /login 500
# ...

# 9. The g flag replaces EVERY match on a line, not just the first. (<<< feeds sed a string.)
sed 's/0/#/g' <<< "code 200 200 500"
# Output — all six zeros replaced:
# code 2## 2## 5##

# 10. awk splits each line into fields on whitespace. $1 is the first column (the date).
awk '{print $1}' ~/app.log
# Output:
# 2026-07-09
# 2026-07-09
# 2026-07-09
# 2026-07-09
# 2026-07-09
# 2026-07-09

# 11. Print two fields: $3 (the log level) and $NF (the LAST field — the status code).
awk '{print $3, $NF}' ~/app.log
# Output:
# INFO 200
# ERROR 500
# WARN 200
# error 503
# INFO 200
# ERROR 500

Read the last outputs back: grep found the lines you cared about, sed rewrote them (without touching the file), and awk pulled out just the columns — find, edit, extract, the three questions every log answers.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three trip up almost everyone in their first week with the trio. Read the error text slowly — learning to parse it is the actual skill.

Common error: Running grep on a directory but forgetting the -r flag — grep ERROR /var/log — prints:

grep: /var/log: Is a directory

Why: Without -r, grep expects a file, not a directory. Handed a directory, it refuses to read it as if it were a stream of lines and reports the type mismatch instead of silently doing nothing.

Fix: Add -r to recurse the tree: grep -r ERROR /var/log. Pair it with -n for line numbers (grep -rn ERROR /var/log), and --include='*.log' if you want to skip binaries and rotated archives.

How you’d spot it in prod: Is a directory in a script or CI log means a command was pointed at a folder where it wanted a file. It is almost always a missing -r/-R flag or a glob that expanded to a directory — check the argument, not the tool.

Common error: Writing a sed substitution but forgetting the closing slash — sed 's/ERROR/CRITICAL' ~/app.log — prints:

sed: -e expression #1, char 16: unterminated `s' command

Why: The s command needs three delimiters: s/pattern/replacement/. With only two slashes, sed reaches the end of the expression still waiting for the closing one and cannot tell where the replacement ends, so it aborts.

Fix: Add the trailing slash: sed 's/ERROR/CRITICAL/' ~/app.log. The char 16 in the message is the position in the expression where sed gave up — a useful pointer when the substitution is long.

How you’d spot it in prod: unterminated 's' command in a deploy or config-rewrite step is a malformed sed expression — usually a missing delimiter or an unescaped / inside a path you are substituting. Switch the delimiter (sed 's|/old|/new|') when the text itself contains slashes.

Common error: Expecting sed 's/…/…/' to edit the file — running it, seeing the changed output, then finding the file untouched:

$ sed 's/ERROR/CRIT/' ~/app.log   # looks fixed on screen
$ cat ~/app.log
2026-07-09 10:02 ERROR user=bob POST /login 500   # still ERROR

Why: By default sed writes the transformed stream to standard output and never modifies the input file. What you saw on screen was a throwaway copy; the original bytes on disk were never touched.

Fix: Redirect to a new file — sed 's/ERROR/CRIT/' ~/app.log > ~/app.clean.log — or, to edit in place, use sed -i (and -i.bak to keep a backup). Reach for -i deliberately, never on a file you cannot afford to rewrite.

How you’d spot it in prod: A “fix” script that runs cleanly yet the service keeps using the old value is the classic tell — the sed ran, printed, and discarded its output while the config on disk never changed. Diff the file before and after, or check for the missing -i.

Text-Processing Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — name what grep, sed and awk each do before you reveal each answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~25 more minutes today:

• 5 min — Run man grep and read the OPTIONS section; skim what -E, -o and -c add to the four flags you learned today.
• 10 min — Rebuild today’s Netflix-style pipeline on your own file: awk '{print $NF}' ~/app.log | sort | uniq -c to count how many times each status code appears. (Pipes are tomorrow — this is a preview.)
• 10 min — Read the “Introduction to Linux” section of the Linux for DevOps guide for how text tools fit the wider server-side picture.