Pipes and IO redirection in Linux and Unix

In this tutorial we will talk about redirection of standard streams, stdin, stdout, stderr in Unix-like operating systems. The Unix philosophy is to write programs that do one thing, but do it well and write them to work together, so the output of a program can be used as input to another, this is great because we can transfer data continuously from a program to another without manually feeding data into each one.

Unix-like standard streams
Diagram 1.0

A real life example of the diagram 1.0 using the ps and grep commands.

The ps aux will show every process running on your system, while the grep bash will print the lines that matches the “bash” string. The pipe operator “|” will feed the output of the ps command (stdout) into the input of the grep command (stdin), so the result will be every process that includes the “bash” string.

ps aux | grep bash
tester     949  0.0  0.1  20920  4720 pts/0    S    12:00   0:00 bash
tester     956  0.0  0.0  12784   956 pts/0    S+   12:00   0:00 grep bash

But wait.. what about stderr?

Run the following find command as normal user, we will search for the ip_forward file in the /proc directory and match the “ipv4” string with grep.

find /proc -name ip_forward | grep ipv4
...
find: `/proc/3/fdinfo': Permission denied
find: `/proc/3/ns': Permission denied
/proc/sys/net/ipv4/ip_forward
find: `/proc/94/task/94/fd': Permission denied
find: `/proc/94/task/94/fdinfo': Permission denied
...

The result is not what we might expect, because we used grep to match only the lines with the “ipv4” string. The explanation is that the lines containing the “Permission denied” string don’t reach the grep command, they are errors which by default goes to display, through the stderr stream (see diagram 1.0). This example can be fixed by redirecting the stderr to stdout using the redirection operator “>“.

find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

Back to the basics.

Standard streams can be found in many environments, shortly they are input and output connections between a process and a device or pseudo-device. Each stream is mapped to a file descriptor (FD), a non-negative integer assigned by the operating system. Linux and Unix handles the IO using three streams, check out the table from below.

Standard Streams
FD Name Short name Description
0 Standard Input stdin The program reads data from a physical device, usually a keyboard or redirection.
1 Standard Output stdout Where a program writes its output data.
2 Standard Error stderr Where a program writes its error messages.

Redirect to another program via pipeline, run the commands as normal user.

| Redirect stdout to grep's stdin, stderr goes to display.

find /proc -name ip_forward | grep ipv4
...
/proc/sys/net/ipv4/ip_forward
find: `/proc/1/task/1/ns': Permission denied
find: `/proc/1/fd': Permission denied
find: `/proc/1/fdinfo': Permission denied
...

|& Redirect stdout and stderr to grep, this is a shorthand for "2>&1 |".

find /proc -name ip_forward |& grep ipv4
or
find /proc -name ip_forward 2>&1 | grep ipv4
/proc/sys/net/ipv4/ip_forward

2>&1 1>/dev/null Redirect stderr to stdout and send find's stdout to /dev/null, literally send stderr to grep and throw find's stdout away.

find /proc -name ip_forward 2>&1 1>/dev/null | grep map_files
...
find: `/proc/29353/map_files': Permission denied
find: `/proc/33761/map_files': Permission denied
find: `/proc/33898/map_files': Permission denied
find: `/proc/33902/map_files': Permission denied
find: `/proc/33971/map_files': Permission denied
find: `/proc/37080/map_files': Permission denied

Read data from file using the “<” operator.

cat < /etc/resolv.conf

Redirect to file.

> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward > testfile.log

>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward >> testfile.log

&> Redirect stdout and stderr to a file, creates the file or overwrites it.

find /proc -name ip_forward &> testfile.log

&>> Redirect stdout and stderr to a file, creates the file or appends to it.

find /proc -name ip_forward &>> testfile.log

Redirect to file, explicitly.

1> Redirect stdout to a file, creates the file or overwrites it, stderr goes to display.

find /proc -name ip_forward 1> testfile.log

1>> Redirect stdout to a file, creates the file or appends to it, stderr goes to display.

find /proc -name ip_forward 1>> testfile.log

2> Redirect stderr to a file, creates the file or overwrites it, stdout goes to display.

find /proc -name ip_forward 2> testfile.log

2>> Redirect stderr to a file, creates the file or appends to it, stdout goes to display.

find /proc -name ip_forward 2>> testfile.log

Redirect to file, explicitly, combined.

Redirect stdout and stderr to different files, creates the files or overwrites it.

find /proc -name ip_forward 1> output.log 2> error.log

Redirect stdout and stderr to different files, creates the files or appends to it.

find /proc -name ip_forward 1>> output.log 2>> error.log

Run Ubuntu on MacBook from USB flash drive

MacBook Ubuntu Live

Running a Linux distro from USB drive can be very useful, you can do almost anything just like from an installed operating system. It’s an easy way to run something on Linux or just see how Ubuntu works, the biggest disadvantage is that the USB flash drives are much slower than SSD drives. In this tutorial we will create a bootable USB drive in macOS and test it on MacBook Air.

Test environment: macOS 10.13.1
Download ISO: Ubuntu 16.04

Create the bootable USB drive (MBR/UEFI scheme)

a.) Identify the USB drive, in my example the device is /dev/disk2.

diskutil list
/dev/disk0 (internal, physical):
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         121.1 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +121.1 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume SSD 128GB               34.8 GB    disk1s1
   2:                APFS Volume Preboot                 18.5 MB    disk1s2
   3:                APFS Volume Recovery                520.8 MB   disk1s3
   4:                APFS Volume VM                      1.1 GB     disk1s4

/dev/disk2 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: FDisk_partition_scheme *15.5 GB disk2 1: DOS_FAT_32 UBUNTU 15.5 GB disk2s1

b.) Erase the /dev/disk2, create one big partition /dev/disk2s1 and format as FAT32 with the label “UBUNTU”. All data on /dev/disk2 will be destroyed!

diskutil partitionDisk /dev/disk2 1 MBR FAT32 UBUNTU R
Started partitioning on disk2
Unmounting disk
Creating the partition map
Waiting for partitions to activate
Formatting disk2s1 as MS-DOS (FAT32) with name UBUNTU
512 bytes per physical sector
/dev/rdisk2s1: 30248048 sectors in 1890503 FAT32 clusters (8192 bytes/cluster)
bps=512 spc=16 res=32 nft=2 mid=0xf8 spt=32 hds=255 hid=2 drv=0x80 bsec=30277630 bspf=14770 rdcl=2 infs=1 bkbs=6
Mounting disk
Finished partitioning on disk2
/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *15.5 GB    disk2
   1:                 DOS_FAT_32 UBUNTU                  15.5 GB    disk2s1

c.) Mount the Ubuntu ISO image.

mkdir /Volumes/UBUNTUISO/
hdiutil attach -nomount ubuntu-16.04.3-desktop-amd64.iso
/dev/disk3              Apple_partition_scheme
/dev/disk3s1            Apple_partition_map
/dev/disk3s2            Apple_HFS
mount -t cd9660 /dev/disk3 /Volumes/UBUNTUISO

d.) Copy the Ubuntu installation files to the USB drive, it might take a few minutes.

cp -Rv /Volumes/UBUNTUISO/ /Volumes/UBUNTU/

e.) Unmount the USB drive.

diskutil unmountDisk /dev/disk2

* Note The bootable USB drive will work only on systems with UEFI firmware.

Run Ubuntu

a.) Turn on your MacBook and press the Option (⌥) key, you should see the Startup Manager, which allows you to boot from other disks.
b.) Select the USB drive and press Enter.

MacBook select USB drive

c.) When Grub loads, select “Try Ubuntu without installing” and press Enter.

MacBook Ubuntu Loading