Pipes, redirection & command chaining

The three streams — and how to redirect them

Section 1 of 5 · ~3 min

Every program Linux starts is born wired to three channels called streams. Standard input (stdin, file descriptor 0) is where it reads; standard output (stdout, fd 1) is where its normal results go; standard error (stderr, fd 2) is where its complaints go. By default stdin is your keyboard, and both stdout and stderr paint onto your terminal — which is exactly why a command’s results and its error messages look jumbled together on screen. They are two separate streams sharing one window. The shell’s superpower is that you can unplug any channel and point it somewhere else.

Redirection rewires a stream to a file. > sends stdout to a file, truncating (overwriting) whatever was there; >> appends instead. The catch that trips up everyone: > redirects only stdout — errors still hit your screen, because stderr is a different stream. To capture errors you name fd 2 yourself: command 2> errors.log. To fold errors into the same destination as output, you write 2>&1, read as “make fd 2 go wherever fd 1 is now going.” Order matters: > file 2>&1 captures both, but 2>&1 > file does not — the shell applies redirections left to right, so it copies stderr to the terminal before stdout is moved to the file.

        stdin (fd 0)                        stdout (fd 1) ──┐
  keyboard ──────────▶ [ command ] ──────────▶             ├──▶ your screen
                            │                 stderr (fd 2) ─┘
                            └──────────────────────┘

  ls /etc | grep conf | wc -l
     └fd1─▶ stdin┘  └fd1─▶ stdin┘      each | wires one command's
                                       stdout into the next one's stdin

Pipes, tee, xargs and chaining

A pipe (|) connects the stdout of one command directly to the stdin of the next — no temp file, no disk; the kernel hands the bytes across in memory. This is the beating heart of the Unix philosophy: small tools that each do one job, snapped together into a line that does something none of them could alone. ls /etc | wc -l counts entries; ps aux | grep nginx finds a process. A pipe carries stdout only — stderr keeps going to your screen unless you add 2>&1 before the |.

Real world: A pipeline is a factory conveyor belt. Raw parts enter at one end; each station does exactly one operation — stamp, weld, paint — and slides the piece to the next station. No station hoards the whole product, and you can re-order or swap stations to build something new. grep, sort and wc are the stations; the | is the belt between them.

tee splits the belt: it writes the stream to a file and passes the same bytes onward, so you save a result and keep processing it. xargs solves a mismatch — some tools (rm, mkdir, kill) read arguments, not stdin, so find . -name '*.tmp' | xargs rm feeds each name in as an argument. Finally, chaining operators sequence whole commands by exit code: A && B runs B only if A succeeded, A || B runs B only if A failed, and A ; B runs B no matter what. This is how scripts stay honest — make build && make deploy never ships a broken build.

The pipe itself was invented in 1973 at Bell Labs by Doug McIlroy, who pushed the Unix team to make programs “work together” as the default. Fifty years on, that single | character is still how you assemble a one-line log analysis on a production server without writing a script at all.

Hands-On Lab

Section 2 of 5 · ~3 min

Budget about 20 minutes. Open your WSL2 Ubuntu 24.04 terminal (if you have not set up WSL2 yet, do Day 0 first — macOS and Linux users can follow along in their built-in terminal). Type each command yourself and watch which stream each line comes from.

# 1. Send stdout to a file with > (this OVERWRITES the file every time).
echo "first line" > notes.txt
cat notes.txt
# Output:
# first line

# 2. >> APPENDS instead of overwriting — the earlier line survives.
echo "second line" >> notes.txt
cat notes.txt
# Output:
# first line
# second line

# 3. One command, two streams: the good result is stdout, the complaint is stderr.
ls /etc/hostname /nope
# Output (the error is stderr, the path is stdout — both hit your screen):
# ls: cannot access '/nope': No such file or directory
# /etc/hostname

# 4. > redirects ONLY stdout — the error still lands on your screen.
ls /etc/hostname /nope > out.txt
# Output (this line is stderr, so > did not capture it):
# ls: cannot access '/nope': No such file or directory
cat out.txt
# Output (only stdout went to the file):
# /etc/hostname

# 5. Name fd 2 to capture the error; 2>&1 folds errors into wherever stdout is going.
ls /etc/hostname /nope > all.log 2>&1
cat all.log
# Output (both streams now in one file):
# ls: cannot access '/nope': No such file or directory
# /etc/hostname

# 6. A pipe (|) feeds one command's stdout into the next command's stdin — no temp file.
ls /etc | wc -l
# Output (count of entries in /etc; your number will differ):
# 158

# 7. Chain filters: read a file, keep only the matching line.
cat /etc/os-release | grep VERSION_CODENAME
# Output:
# VERSION_CODENAME=noble

# 8. tee splits the stream: write to a file AND pass it on down the pipe.
ls /etc | tee etc-list.txt | wc -l
# Output (wc still gets the count; etc-list.txt also holds the full listing):
# 158

# 9. xargs turns stdin into ARGUMENTS for a tool that ignores stdin (mkdir).
echo "alpha beta gamma" | xargs mkdir
ls -d alpha beta gamma
# Output:
# alpha  beta  gamma

# 10. && runs the next command only on success (exit 0); || runs it only on failure.
mkdir demo && echo "made it"
# Output:
# made it
rmdir /nope || echo "that failed, as expected"
# Output:
# rmdir: failed to remove '/nope': No such file or directory
# that failed, as expected

# 11. ; runs commands in sequence no matter what; 2>/dev/null throws errors away.
false ; echo "runs anyway"
ls /nope 2>/dev/null ; echo "error was discarded"
# Output:
# runs anyway
# error was discarded

Before you close the terminal, say the flow back out loud: > overwrites and >> appends, > catches only stdout while 2> and 2>&1 handle errors, | wires stdout into the next stdin, tee saves a copy mid-pipe, xargs turns lines into arguments, and &&/||/; sequence whole commands by their exit code.

Common Errors & Fixes

Section 3 of 5 · ~2 min

These three catch almost everyone in their first week of piping and redirecting. Read the error text slowly — learning to parse it is the actual skill.

Common error: Redirecting a command’s output back into its own input file — running sort data.txt > data.txt to sort a file in place.

(no error is printed — but data.txt is now 0 bytes, and the data is gone)

Why: The shell sets up redirections before the command runs. > data.txt opens the file for writing and truncates it to empty immediately, so by the time sort opens data.txt to read, there is nothing left. Silent data loss, no error.

Fix: Never redirect into a file you are also reading. Use the tool’s own in-place flag — sort data.txt -o data.txt — or write to a temp file and move it: sort data.txt > tmp && mv tmp data.txt.

How you’d spot it in prod: A log-processing cron or CI step that writes back to its own input silently zeroes the file. You notice when a downstream job reports empty input, not from any error in the failing step itself.

Common error: Trying to capture errors into a log but writing the redirections in the wrong order — mycommand 2>&1 > build.log.

error: something failed
(the error prints on the terminal; build.log holds only stdout)

Why: Redirections are applied left to right. 2>&1 copies fd 2 to wherever fd 1 points right now — still the terminal — and only then does > build.log move fd 1 to the file. So stderr was aimed at the terminal before stdout ever moved.

Fix: Redirect stdout first, then merge: mycommand > build.log 2>&1. In bash you can also use the shorthand mycommand &> build.log.

How you’d spot it in prod: A CI artifact or log file that looks complete but is mysteriously missing the very error that failed the step — the failure is on the console output only, not in the saved log.

Common error: Piping into xargs when the upstream command matched nothing — for example find . -name '*.log' | xargs rm on a directory with no logs.

rm: missing operand
Try 'rm --help' for more information.

Why: GNU xargs runs the target command at least once even when stdin is empty, so rm is invoked with no arguments and errors. Worse, without -0 a filename containing a space or newline is split into several arguments, so rm can act on the wrong paths.

Fix: Add -r (--no-run-if-empty) so nothing runs on empty input, and pair find -print0 with xargs -0 for names with spaces: find . -name '*.log' -print0 | xargs -0 -r rm.

How you’d spot it in prod: A cleanup job that errors on quiet days when nothing matched, or — the dangerous case — deletes unexpected files whenever a path contains a space, showing up as data missing after a routine batch run.

Pipes & Redirection Interview Questions

Section 4 of 5 · ~1 min

Cover the answers below and say your own version out loud first — trace where each stream goes before you reveal the answer. Recalling before revealing is what makes these stick when an interviewer asks them cold. The four questions and answers render right after this note.

Go Deeper

Section 5 of 5 · ~1 min

Optional extras if you have ~30 more minutes today:

• 5 min — Run man xargs and read the -0, -r and -n flags; then try printf 'a\nb\nc\n' | xargs -n1 echo item: to see how -n1 runs the command once per item.
• 10 min — Experiment with /dev/null as a black hole: command > /dev/null 2>&1 runs something and discards all output. This is the single most common redirect you will see in cron jobs and scripts.
• 15 min — Read the “Working with files and text” section of the Linux for DevOps guide to see how pipes tie together the grep/sed/awk tools from Day 8.