Pipes and IO redirection in Linux and Unix

In this tutorial we will talk about redirection of standard streams, stdin, stdout, stderr in Unix-like operating systems. The Unix philosophy is to write programs that do one thing, but do it well and write them to work together, so the output of a program can be used as input to another, this is great because we can transfer data continuously from a program to another without manually feeding data into each one.

Unix-like standard streams
Diagram 1.0

A real life example of the diagram 1.0 using the ps and grep commands.

The ps aux will show every process running on your system, while the grep bash will print the lines that matches the “bash” string. The pipe operator “|” will feed the output of the ps command (stdout) into the input of the grep command (stdin), so the result will be every process that includes the “bash” string.

ps aux | grep bash
tester     949  0.0  0.1  20920  4720 pts/0    S    12:00   0:00 bash
tester     956  0.0  0.0  12784   956 pts/0    S+   12:00   0:00 grep bash

But wait.. what about stderr?

Run the following find command as normal user, we will search for the ip_forward file in the /proc directory and match the “ipv4” string with grep.

find /proc -name ip_forward | grep ipv4
...
find: `/proc/3/fdinfo': Permission denied
find: `/proc/3/ns': Permission denied
/proc/sys/net/ipv4/ip_forward
find: `/proc/94/task/94/fd': Permission denied
find: `/proc/94/task/94/fdinfo': Permission denied
...

The result is not what we might expect, because we used grep to match only the lines with the “ipv4” string. The explanation is that the lines containing the “Permission denied” string don’t reach the grep command, they are errors which by default goes to display, through the stderr stream (see diagram 1.0). This example can be fixed by redirecting the stderr to stdout using the redirection operator “>“.

find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

Back to the basics.

Standard streams can be found in many environments, shortly they are input and output connections between a process and a device or pseudo-device. Each stream is mapped to a file descriptor (FD), a non-negative integer assigned by the operating system. Linux and Unix handles the IO using three streams, check out the table from below.

Standard Streams
FD Name Short name Description
0 Standard Input stdin The program reads data from a physical device, usually a keyboard or redirection.
1 Standard Output stdout Where a program writes its output data.
2 Standard Error stderr Where a program writes its error messages.

Redirect to another program via pipeline, run the commands as normal user.

| Redirect stdout to grep's stdin, stderr goes to display.

find /proc -name ip_forward | grep ipv4
...
/proc/sys/net/ipv4/ip_forward
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/fdinfo': Permission denied
...

|& Redirect stdout and stderr to grep, this is a shorthand for "2>&1 |".

find /proc -name ip_forward |& grep ipv4
or
find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

2>&1 1>/dev/null Redirect stderr to stdout and send find's stdout to /dev/null, literally send stderr to grep and throw find's stdout away.

find /proc -name ip_forward 2>&1 1>/dev/null | grep map_files
...
find: `/proc/29353/map_files': Permission denied
find: `/proc/33761/map_files': Permission denied
find: `/proc/33898/map_files': Permission denied
find: `/proc/33902/map_files': Permission denied
find: `/proc/33971/map_files': Permission denied
find: `/proc/37080/map_files': Permission denied

Read data from file using the “<” operator.

cat < /etc/resolv.conf

Redirect to file.

> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward > testfile.log

>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward >> testfile.log

&> Redirect stdout and stderr to a file, creates the file or overwrites it.

find /proc -name ip_forward &> testfile.log

&>> Redirect stdout and stderr to a file, creates the file or appends to it.

find /proc -name ip_forward &>> testfile.log

Redirect to file, explicitly.

1> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward 1> testfile.log

1>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward 1>> testfile.log

2> Redirect stderr to a file, creates the file or overwrites it, stdout goes to display.

find /proc -name ip_forward 2> testfile.log

2>> Redirect stderr to a file, creates the file or appends to it, stdout goes to display.

find /proc -name ip_forward 2>> testfile.log

Redirect to file, explicitly, combined.

Redirect stdout and stderr to different files, creates the files or overwrites it.

find /proc -name ip_forward 1> output.log 2> error.log

Redirect stdout and stderr to different files, creates the files or appends to it.

find /proc -name ip_forward 1>> output.log 2>> error.log

Leave a Reply

Your email address will not be published. Required fields are marked *