Best Couple of 2015: tar and ssh

No readers like this yet.
Best couple of cats

Internet Archive. Modified by Opensource.com. CC BY-SA 4.0

The best couples complement each other, and each member of the couple contributes unique and irreplaceable parts to the whole. But some couples are very odd. Such is the case with our best couple this year: the tar and ssh commands.

Wait—what?!

Yup, that's right, the tar and ssh commands work together in interesting ways, especially when used with full consideration of the capabilities of Standard I/O (STDIO), which is also known as standard streams.

ssh

The ssh command is a secure and sophisticated form of terminal emulator that allows one to log in to a remote computer to access a shell session and run commands. So I could log in to a remote computer and run the ls command on the remote computer. The results are displayed in the ssh terminal emulator window on my local host. The Standard Output (STDOUT) of the command is displayed on my terminal window, but it remains on the remote host and cannot be used by the local host.

That is trivial and everyone does that. But the next step is a bit more interesting. Rather than maintain a terminal session on the remote computer and issuing multiple commands, I can simply use a command like the following to run a single command on the remote computer with the results being displayed on the local host. This assumes that SSH public/private keypairs (PPKP) are in use and I do not have to enter a password each time I issue a command to the remote host:

ssh remotehost ls

So now I can use the results of that command on my local host because the standard output data stream is sent through the SSH tunnel to the local host. OK, that is good, but what does it mean?

Let's look at the tar command before answering that question.

tar

The tar command is used to make backups. The name tar stands for Tape ARchive, but the command can be used with any type of recording media such as tape, hard drives, thumb drives and more. A command like the following can be used to create a backup of a home directory on the local host:

tar -cvf /tmp/home.tar /home

This command created a tar file—also called a tarball—named home.tar in the /tmp directory. That file is a backup of everything in the home directory. Well, that's nice, but also not very interesting because it is very common.

But what can be interesting is, although many people do not realize it, if the target output file is not specified using the -f option, the output of the tar command is sent directly to STDOUT:

tar -cv /home

That means that the complete output of the tar command—the files being backed up —is sent to the terminal, which opens up some interesting possibilities, such as redirecting the STDOUT data stream to a backup file. That looks like the following command:

tar -cv /home > /tmp/home.tar

This command performs the same function as the first tar command in this section, but in a somewhat different and more interesting manner.

The Odd Couple

We can use a command similar to the following to back up the home directory of the remote host to the /tmp directory of that remote host:

ssh remotehost "tar -cvf /tmp/home.tar /home"

Note that the command to be executed on the remote host is enclosed in quotes to ensure that the correct command is executed remotely; this is a bit of clarification for both the shell and for us humans. A slight change to this command gives us one in which we simply redirect the output of the tar command to the /tmp directory on the remote host:

ssh remotehost "tar -cv /home > /tmp/home.tar"

This command produces exactly the same result as the previous one. In this case, the STDOUT data stream of the tar command is maintained entirely on the remote host and is redirected to the backup file. The next command, however, is the one that opens up many new possibilities. Can you see what it does?

ssh remotehost "tar -cv /home" > /tmp/home.tar

In this case, the STDOUT data stream from the tar command is sent through the SSH connection to the local host. This stream of data is then redirected to the backup file /tmp/home.tar on the local host. By simply moving the trailing quote to the left, the command is changed so that we now have a command that can do backups of remote hosts to a local host.

I use our couple of the year every day to perform backups. I have a script that uses ssh and tar, along with SSH public key encryption, to perform backups of several remote hosts to an external USB hard drive on a local host. These two commands simplify a necessary task, and the best part is that they are free and open source software—free as in beer, as well as free as in speech.

So let's hear it for this year's Opensource.com Best Couple: tar and ssh.

David Both
David Both is an Open Source Software and GNU/Linux advocate, trainer, writer, and speaker. He has been working with Linux and Open Source Software since 1996 and with computers since 1969. He is a strong proponent of and evangelist for the "Linux Philosophy for System Administrators."

12 Comments

A very well-written article, David! I think of classic books written by people like Rob Pike or Brian Kernighan about "doing stuff with Unix commands" and it makes me a bit sad that doing stuff in a terminal window is considered so arcane. Articles like this show us all that it isn't arcane at all. Thank you!

Thank you for your kind comment. I am not sure you are comparing me to Pike and Kernighan, but I am still honored to be mentioned in the same paragraph as them.

I agree that it is sad that so many people shy away from the CLI because they think it is so arcane, when it is really not. It is just different.

It is my hope that some of the articles I write will inspire people to tyr the CLI.

Thanks again!

In reply to by clhermansen

David, I agree with Chris. I rarely work in a terminal window, but I always enjoy reading your articles. Thanks for picking our Best Couple of the Year!

I think Neal Stephenson expressed it very well in "In the Beginning was the Command Line" when he said that driving a car equipped with a GUI would be a lot easier, but a lot less safe.

I like to imagine the driver, seeing the car in front slow down: OK, File->Brakes, now up comes the slider for brake press...CRASH.

BTW some of us are old enough to remember when the command line was, if not actually non-existent, fare from common.

I've been using this "couple" for years. This is my favourite way to do P2V, V2V migrations for a long time : root@live_on_new_formated_disk# ssh remotehost "tar czpf /" tar xzpf -

That is a very elegant way of performing the migration without using any intervening temporary files. I like it!

Thanks for sharing it.

In reply to by SysOp (not verified)

Trying googling "ssh tar pipe tar". There are about a thousand articles showing you how to do the ssh-tar-pipe-tar trick which you _failed to include in your article_, but the rest of us have been using for 20+ years.

At least someone put it here in the comments.

In reply to by dboth

Great article David! I used tar and ssh a lot. I ran Lotus Notes on a Red Hat box and had to backup the mail file every night. I wrote a script that include tar, ssh and rsync and backed up the mail every night. Linux is so powerful on the command line.

Thanks David, great article. I've used tar & ssh a lot but never really thought to couple them.

You are welcome! I am happy my article provided you with some new insights.

To me, it really all goes back to the Linux Philosophy. Each program does one thing and does it well. ssh is really good at encrypted communications and tar is really good at creating and managing streams of archival data. Combining these powerful programs allows us to do things that the creators of each probably never even thought of. And there are many hundreds of such programs available to those of us who use the CLI.

Thanks for your comment.

In reply to by MicheleMcN

I nominate this article for least researched and most incomplete article of 2015.

It also qualifies for a 1995 flashback, but even then we know the "ssh-tar-pipe-tar" trick (which isn't even in this article).

Just google for "ssh tar pipe tar" and you'll find a thousand articles more informative than this one.

At first I thought this would be the "ssh-tar-pipe-tar" trick, and I was all set to recommend using rsync with ssh, to copy files to/from a remote systems. However David did come up with a use case I hadn't thought of, namely creating a local tarball from files on a remote system.

I'll also give him bonus points for explaining the use of redirection in an ssh command. That's something that has confused several beginners, and he covered it well.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.