Pipes: connecting commands with |

A redirection connects a command to a file. A pipe connects a command to another command: a | b runs both at once, with everything a writes to its stdout arriving at b's stdin. No temporary file, no waiting for a to finish first. The vertical bar is the single most important character in the Unix toolbox, because it lets small programs that each do one thing be combined into something none of them does alone.

Reading a pipeline

Read left to right and ask at each bar "what text is flowing through here?". In ls /bin | head -5, a list of names flows out of ls and head keeps the first five. In cat words.txt | sort | uniq -c | sort -rn, the lines of the file are sorted, collapsed into counted groups, then sorted again by that count, biggest first. Each stage is a plain command you could run alone; the pipeline is just the glue.

Commands that read stdin when given no file name are called filters: sort, uniq, wc, head, tail, grep, tr, cut, sed, awk and many more. Commands that produce lists, like ls, ps, find or cat, usually sit at the left end. Only stdout travels through a pipe; stderr still goes to the terminal, unless you add 2>&1 before the bar.

~% ls /bin | wc -l
73
~% ls /bin | head -3
arch
ash
busybox
~% cat /etc/passwd | cut -d: -f1 | sort | head -4
bin
daemon
mail
nobody
~% ls /root /nope 2>&1 | wc -l
2
~% ls /bin | grep grep
egrep
fgrep
grep

Pipe or redirection?

Pipelines and exit status

A pipeline's exit status is that of its last command. grep nothing file | wc -l succeeds even when grep found nothing, because wc ran fine and printed 0. Keep that in mind later when you use && or if around pipelines: test the stage that matters, or restructure so it comes last.

Long pipelines can be split across lines: a line that ends in | continues on the next one, which is how you will see them written in scripts. Build them incrementally: run the first command, check the output looks right, add a stage, check again.