Pipes and IO redirection in Linux and Unix

In this tutorial we will talk about redirection of standard streams, stdin, stdout, stderr in Unix-like operating systems. The Unix philosophy is to write programs that do one thing, but do it well and write them to work together, so the output of a program can be used as input to another, this is great because we can transfer data continuously from a program to another without manually feeding data into each one.

Unix-like standard streams
Diagram 1.0

A real life example of the diagram 1.0 using the ps and grep commands.

The ps aux will show every process running on your system, while the grep bash will print the lines that matches the “bash” string. The pipe operator “|” will feed the output of the ps command (stdout) into the input of the grep command (stdin), so the result will be every process that includes the “bash” string.

ps aux | grep bash
tester     949  0.0  0.1  20920  4720 pts/0    S    12:00   0:00 bash
tester     956  0.0  0.0  12784   956 pts/0    S+   12:00   0:00 grep bash

But wait.. what about stderr?

Run the following find command as normal user, we will search for the ip_forward file in the /proc directory and match the “ipv4” string with grep.

find /proc -name ip_forward | grep ipv4
...
find: `/proc/3/fdinfo': Permission denied
find: `/proc/3/ns': Permission denied
/proc/sys/net/ipv4/ip_forward
find: `/proc/94/task/94/fd': Permission denied
find: `/proc/94/task/94/fdinfo': Permission denied
...

The result is not what we might expect, because we used grep to match only the lines with the “ipv4” string. The explanation is that the lines containing the “Permission denied” string don’t reach the grep command, they are errors which by default goes to display, through the stderr stream (see diagram 1.0). This example can be fixed by redirecting the stderr to stdout using the redirection operator “>“.

find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

Back to the basics.

Standard streams can be found in many environments, shortly they are input and output connections between a process and a device or pseudo-device. Each stream is mapped to a file descriptor (FD), a non-negative integer assigned by the operating system. Linux and Unix handles the IO using three streams, check out the table from below.

Standard Streams
FD Name Short name Description
0 Standard Input stdin The program reads data from a physical device, usually a keyboard or redirection.
1 Standard Output stdout Where a program writes its output data.
2 Standard Error stderr Where a program writes its error messages.

Redirect to another program via pipeline, run the commands as normal user.

| Redirect stdout to grep's stdin, stderr goes to display.

find /proc -name ip_forward | grep ipv4
...
/proc/sys/net/ipv4/ip_forward
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/fdinfo': Permission denied
...

|& Redirect stdout and stderr to grep, this is a shorthand for "2>&1 |".

find /proc -name ip_forward |& grep ipv4
or
find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

2>&1 1>/dev/null Redirect stderr to stdout and send find's stdout to /dev/null, literally send stderr to grep and throw find's stdout away.

find /proc -name ip_forward 2>&1 1>/dev/null | grep map_files
...
find: `/proc/29353/map_files': Permission denied
find: `/proc/33761/map_files': Permission denied
find: `/proc/33898/map_files': Permission denied
find: `/proc/33902/map_files': Permission denied
find: `/proc/33971/map_files': Permission denied
find: `/proc/37080/map_files': Permission denied

Read data from file using the “<” operator.

cat < /etc/resolv.conf

Redirect to file.

> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward > testfile.log

>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward >> testfile.log

&> Redirect stdout and stderr to a file, creates the file or overwrites it.

find /proc -name ip_forward &> testfile.log

&>> Redirect stdout and stderr to a file, creates the file or appends to it.

find /proc -name ip_forward &>> testfile.log

Redirect to file, explicitly.

1> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward 1> testfile.log

1>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward 1>> testfile.log

2> Redirect stderr to a file, creates the file or overwrites it, stdout goes to display.

find /proc -name ip_forward 2> testfile.log

2>> Redirect stderr to a file, creates the file or appends to it, stdout goes to display.

find /proc -name ip_forward 2>> testfile.log

Redirect to file, explicitly, combined.

Redirect stdout and stderr to different files, creates the files or overwrites it.

find /proc -name ip_forward 1> output.log 2> error.log

Redirect stdout and stderr to different files, creates the files or appends to it.

find /proc -name ip_forward 1>> output.log 2>> error.log

View network interfaces and change (spoof) MAC address in Linux

This is a basic Linux networking task and can be done via command line and graphical interface. The NetworkManager software is used to create and view network connections with graphical interface, which may look different depending on the desktop environment. Because the NetworkManager is easy to use and has a lot of front ends, this tutorial will cover only the command line method, we will use the ip and ifconfig commands, it’s recommended to use the ip command because ifconfig is obsolete, so open your favorite terminal emulator and let’s start!

Display network devices, state, MAC address, IP address, network class and statistics

As you see I have three interfaces, “lo” is the loopback interface, “eth0” the wired network card and “wlan0” the wireless card, I am using the classic naming scheme.

ip -s addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast 
    2640       44       0       0       0       0        
    TX: bytes  packets  errors  dropped carrier collsns 
    2640       44       0       0       0       0            
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.2/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5ef3:fcff:feec:f454/64 scope link 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast 
    149454253  137261   0       0       0       40      
    TX: bytes  packets  errors  dropped carrier collsns 
    16640485   92276    0       0       0       0       
12: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:02 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast 
    0          0        0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    0          0        0       0       0       0       

Legend: Interface, State, MAC Address, IP Address, Network Class, Statistics

Or with the obsolete ifconfig

ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.2  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::5ef3:fcff:feec:f454  prefixlen 64  scopeid 0x20
        ether 00:00:00:00:00:01  txqueuelen 1000  (Ethernet)
        RX packets 59133  bytes 35479715 (33.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 46498  bytes 9805836 (9.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xf7ca0000-f7cc0000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1  (Local Loopback)
        RX packets 124  bytes 7440 (7.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 124  bytes 7440 (7.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 00:00:00:00:00:02  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Legend: Interface, State, MAC Address, IP Address, Network Class, Statistics

Displaying the routing table, 192.168.0.1 is my default gateway

ip route
default via 192.168.0.1 dev eth0

Or with the obsolete route and netstat commands

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 eth0

Displaying DNS servers

cat /etc/resolv.conf
nameserver 192.168.0.1

Change (spoof) the MAC address

ip link set dev eth0 down
ip link set dev eth0 address 11:11:11:11:11:11
ip link set dev eth0 up

Or with the obsolete ifconfig

ifconfig eth0 down
ip link set dev eth0 address 11:11:11:11:11:11
ifconfig eth0 up