Pipes and Text Processing¶

The three standard streams¶

Every process starts with three open file descriptors: 0 (stdin), 1 (stdout), 2 (stderr) — this is the concrete, practical surface of the "everything is a file descriptor" idea that page covers.

graph LR a["cmd1"] -->|"stdout (fd 1) piped to"| b["cmd2's stdin (fd 0)"] b --> c["cmd2's stdout (fd 1)"] a -.->|"stderr (fd 2) NOT piped —\ngoes straight to the terminal"| term["Terminal"]

cmd1 | cmd2 connects cmd1's stdout only to cmd2's stdin — cmd1's stderr is untouched by the pipe and still goes directly to the terminal (or wherever fd 2 was already pointed). This is why a command's error messages still show up on screen even while its normal output is being piped elsewhere — the pipe only ever redirects fd 1 unless told otherwise.

Redirection order matters — the classic gotcha¶

command > output.log 2>&1    # (A): stdout AND stderr both go to output.log
command 2>&1 > output.log    # (B): stderr goes to the TERMINAL, only stdout goes to output.log

These look almost identical and produce genuinely different results, because redirections are applied left to right, and 2>&1 means "point fd 2 at wherever fd 1 currently points" — not "wherever fd 1 will point in the future."

graph TD subgraph A["(A): > output.log 2>&1"] a1["1. fd 1 -> output.log"] --> a2["2. fd 2 -> wherever fd 1 NOW points (output.log)"] a2 --> ares["Both fd 1 and fd 2 -> output.log"] end subgraph B["(B): 2>&1 > output.log"] b1["1. fd 2 -> wherever fd 1 NOW points (terminal)"] --> b2["2. fd 1 -> output.log"] b2 --> bres["fd 1 -> output.log, but fd 2 still -> terminal"] end

In (A), by the time 2>&1 executes, fd 1 has already been redirected to the log file, so fd 2 correctly copies that same destination. In (B), 2>&1 executes first, while fd 1 still points at the terminal — fd 2 copies that, and the subsequent > output.log only changes fd 1 afterward, leaving fd 2 pointed at the terminal it captured a moment earlier. Getting this order backward is the single most common reason "I redirected both stdout and stderr to a file" doesn't actually capture both.

grep, sed, awk — the short version¶

grep -n "ERROR" app.log          # find matching lines, with line numbers
grep -c "ERROR" app.log          # count matching lines
sed 's/foo/bar/g' file.txt       # substitute foo -> bar, all occurrences per line
awk -F',' '{print $2}' data.csv  # print the 2nd comma-separated field
awk '$3 > 100 {print $1}' data.txt  # print field 1 where field 3 exceeds 100

grep finds lines matching a pattern. sed transforms text stream-by-stream (the s/old/new/g substitution is by far its most common use — g means replace every occurrence per line, not just the first). awk treats input as structured fields (split by whitespace by default, or -F for a custom delimiter) and can filter and compute per-field — the second example only prints a field when a numeric condition on another field holds, which is the kind of lightweight filtering awk is reached for instead of writing a small script in a general-purpose language.

`xargs`: turning a list of lines into arguments¶

find . -name "*.tmp" | xargs rm
find . -name "*.log" -print0 | xargs -0 rm

Many commands (rm, grep, cp) take arguments on the command line, not from stdin — find ... | xargs rm bridges that gap, converting each line of piped input into an argument to rm. The -print0/-0 pair (null-byte separated instead of newline-separated) exists specifically to handle filenames that contain spaces or even literal newlines correctly — plain newline-separated xargs breaks on a filename with a space in it, splitting one filename into two arguments the same way unquoted word splitting does.

Common pitfall¶

Writing command > output.log 2>&1 & (backgrounding a command with combined output redirection) and expecting Ctrl+C to stop it is a common confusion — once backgrounded, the shell's Ctrl+C (SIGINT) goes to the foreground process group, not the backgrounded one; stopping it needs kill %1 (by job number) or kill <pid> directly, tying into signals and job control covered in Process and Job Management.