Pipes are extremely important, and even critical to our ability to do amazing things on the command line.
First, it's important to recognize that they were invented by Douglas McIlroy during the early days of Unix. Thanks, Doug! Next, what is a pipe, in this context? Notice the use of pipes in this simple command-line program that lists each logged-in user a single time no matter how many logins they have active .
who | awk '{print $1}' | sort | uniq
Pipes are the glue that holds these command-line programs together. Pipes allow the standard output from one command to be "piped" to the standard input of the next command.
Think about how this program would have to work if we could not pipe the data stream from one command to the next. The first command would perform its task on the data and then the output from that command would have to be saved in a file. The next command would have to read the stream of data from the intermediate file and perform its modification of the data stream, sending its own output to a new, temporary data file. The third command would have to take its data from the second temporary data file and perform its own manipulation of the data stream, and then store the resulting data stream in yet another temporary file. At each step the data file names would have to be transferred from one command to the next in some way.
I cannot even stand to think about that, it is so complex.
Linux philosophy
It is only with the use of pipes that several tenets of the Linux philosophy make sense.
Each of these programs is small [is beautiful] and each does one thing well. They are also filters; they take standard input, process it in some way, and then send the output to standard output.
The use of standard I/O for input and output is one of the secondary tenets of the Linux philosophy. It is only through the use of pipes that implementation of these programs as filters to send processed data streams from their own standard output to the standard input of the program above is even possible.
The Great Command-Line Challenge
Please note: This contest is now closed to further entries. DPB, 08/01/2015
As I promised in a previous article, I have a challenge for you. It's one that involves pipes as part of the solution—it's a simple problem with a solution that I use nearly every day.
I have all of my own computers plus a few that I manage for other people configured to send emails to my own email account. I have configured my email client to move most of these administrative emails into a single folder to make it easy to find them all. Over the past year, I have collected over 50,000 emails in this folder.
These emails consist of output from rkhunter (Rootkit Hunter), logwatch, cron jobs, and Fail2ban... among others.
Fail2ban is open source software that dynamically bans IP addresses of hosts that attempt to maliciously access my own hosts on the Internet. Fail2ban does this by adding rules to iptables. Each time an IP address is banned, Fail2ban sends an email.
In order to perform a quick analysis of the Fail2Ban messages, I use my email client export into a single CSV file the subject lines of these emails, which contain the IP addresses of the hosts that have been banned. This export contains all of the subjects for all of the emails, not just those from Fail2Ban.
Your mission, should you accept it, is to create a single command-line program to count the number of emails from each IP Address that has attempted to access my hosts using SSH. Download the file to try your hand at that. Be aware that this file contains more than 50,000 lines and is over 6MB in size. And, this is not a task that you would want to tackle by hand. I have changed the domains of the hosts that sent the emails to me, but the IP addresses of the attacking hosts have not been modified.
Just to be clear, the command-line program you create to do this should be only one line long and must use pipes to channel the flow of data from one command to the next. Wrapping is OK. And, your "one line" can wrap so that it fits your terminal width.
To enter the challenge: Send an email to me with "Great Command-Line Challenge" in the subject line, then your answer (one line of code) in the body of the email. One winner will be selected for each of the following criteria: first correct answer, shortest correct answer, and most creative answer. Each winner will get an Opensource.com T-shirt. The challenge is open to submissions through July 31, 2015 at 11:59 pm EDT. Read more about the official rules.
For extra credit: Include the name of the country of each IP address.
Good luck!
11 Comments