Pipes, Redirection and xargs in Bash: A Practical Guide

The real power of the command line comes from connecting small tools. A pipe sends one command's output into another command; redirection sends output to files or reads input from them; xargs turns a list into command arguments. Together they let you build in one line what would otherwise take a script.

Three streams: stdin, stdout, stderr

Every process starts with three open streams, each with a number called a file descriptor:

Normal results go to stdout and error messages go to stderr. Both appear on your screen, which is why it is easy to forget they are separate, until you redirect one and the other still shows up.

Pipes

The pipe | connects the stdout of the command on the left to the stdin of the command on the right:

$ ls /etc | wc -l
$ ps aux | grep '[n]ginx'
$ cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

All the commands in a pipeline run at the same time, with data flowing between them, so a pipeline works on huge files without storing intermediate results. Only stdout goes through the pipe; errors still go to the terminal. To pipe both, use 2>&1 | or the Bash shorthand |&:

$ make 2>&1 | less
$ make |& less

A pipeline's exit status is the status of its last command. set -o pipefail makes it fail if any command fails, and the array ${PIPESTATUS[@]} holds each command's status.

Redirecting output

$ ls > files.txt          # stdout to a file (overwrite)
$ ls >> files.txt         # stdout to a file (append)
$ ls /nope 2> errors.txt  # stderr to a file
$ ls /nope 2>/dev/null    # throw errors away
$ cmd > all.txt 2>&1      # stdout and stderr to the same file
$ cmd &> all.txt          # Bash shorthand for the same
$ cmd > out.txt 2> err.txt   # each to its own file

/dev/null is a special file that discards everything written to it.

Why the order of 2>&1 matters

2>&1 means "make stderr point to wherever stdout points right now". Redirections are processed left to right, so:

$ cmd > file 2>&1     # stdout to file, then stderr to the same place: both in file
$ cmd 2>&1 > file     # stderr to the terminal (where stdout was), then stdout to file

The second form looks wrong but is occasionally useful: it pipes only the errors onward while the normal output goes to a file.

Redirecting input

< feeds a file into a command's stdin:

$ wc -l < access.log
$ mysql mydb < schema.sql

A here-string, <<<, feeds a single string:

$ tr a-z A-Z <<< "shout"
SHOUT
$ read -r first rest <<< "hello big world"

A here-document feeds several lines, up to a marker word of your choice:

$ cat <<EOF > config.ini
[app]
user = $USER
created = $(date +%F)
EOF

Variables and $(...) are expanded inside. Quote the marker, as in <<'EOF', to keep the text exactly as written, which is what you want when generating scripts. <<-EOF strips leading tab characters (not spaces), so you can indent here-docs inside functions.

tee: write to a file and keep going

tee copies its input to a file and to stdout, so you can save output and still see it or pipe it further:

$ ./build.sh 2>&1 | tee build.log
$ ./build.sh 2>&1 | tee -a build.log      # append instead of overwrite

tee is also the standard way to write to a root-owned file. sudo echo "x" > /etc/file fails, because the redirection is done by your own unprivileged shell, not by sudo:

$ echo "127.0.0.1 myapp.test" | sudo tee -a /etc/hosts > /dev/null

Process substitution

<(command) runs a command and presents its output as a file name. That lets you use command output where a program expects a file. The classic use is comparing two outputs:

$ diff <(sort list1.txt) <(sort list2.txt)
$ comm -12 <(sort list1.txt) <(sort list2.txt)    # lines in both
$ diff <(ssh web01 cat /etc/nginx/nginx.conf) /etc/nginx/nginx.conf

It is also the right way to feed a while read loop without losing variables, as shown in the loops guide. The reverse form, >(command), is a file name that writes into a command, so tee >(gzip > out.gz) > out.txt saves a plain and a compressed copy at once. Process substitution is a Bash feature; it does not work in plain sh.

xargs: turn input into arguments

Some commands read stdin, but many only take arguments: rm, mkdir, chmod, git add. xargs reads items from stdin and runs a command with those items as arguments:

$ cat dirs.txt | xargs mkdir -p
$ echo "a b c" | xargs -n 1 echo
a
b
c

By default xargs packs as many items as fit onto each command line, which is fast. Its main options:

$ cat urls.txt | xargs -n 1 -P 4 curl -sO          # download 4 at a time
$ cat hosts.txt | xargs -I {} ssh -n {} uptime      # one ssh per host
$ printf '%s\n' *.txt | xargs -I {} cp -- {} {}.bak   # copy each file to NAME.bak

Spaces in file names

xargs splits its input on spaces and newlines, and treats quotes and backslashes specially. A file called holiday photo.jpg becomes two arguments. That makes ls | xargs and find | xargs unsafe for arbitrary file names. The fix is null separators from end to end:

$ find . -name "*.log" -print0 | xargs -0 rm --
$ find . -name "*.jpg" -print0 | xargs -0 -P 4 -n 20 jpegoptim

With -I {}, xargs splits only on newlines, which handles spaces, but quotes in names (like it's.txt) still cause an "unmatched single quote" error, and names containing newlines still break. When the list comes from find, -exec ... {} + is often simpler; see the find guide.

Common patterns

$ command 2>&1 | tee -a run.log            # watch and log
$ command > /dev/null 2>&1                 # silence everything (cron jobs)
$ output=$(command 2>&1)                   # capture output and errors in a variable
$ grep -rl "oldname" src/ | xargs sed -i 's/oldname/newname/g'   # careful: edits files
$ seq 1 10 | paste -sd+ | bc               # add up numbers: 55
$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head   # most used commands

More one-liners live in the cheatsheet, and the Redirections section of the Bash manual covers every operator.

Quick reference