Howdy!
Okay, enough horsing around. UNIX command line paradigms are powerful ideas and methods of accomplishing things in a short time. This is my, far from complete, list of tricks I’ve picked up over the years using Linux & friends.
Speeding up tarball download/extract
$ wget -O- http://example.org/tarball.tar.gz | tar -xvzf-
This little one liner has saved time and disk space, basically it forces wget to dump the file you want to download into tar which directly starts extracting it as it comes in. This has the advantage of being one command you can easily type, not having the need to temporarily store the tarball while you download it and then extract the contents (making the extract a in-place operation) and speeding up the total time wasted, either your internet bandwidth or harddrive writing speed will be the bottleneck (most often the latter), but at least you won’t have to wait for one to finish to start on the other.
Bulk copy progress indicator
$ tar -cf- /path/to/dir | pv | \ ssh user@box 'cat > /path/to/archive.tar'
This on-liner makes a tarball from a source directory, pipes that through the pv (pipe view) utility, which shows throughput speed, progress and ETA if it can determine it, and finally pumps it over the network to a remote box which writes it to a file. Instead of waiting around and wondering when the job will be done, you now at least have some idea how long it’s going to take.
$ tar -cf- /path/to/dir | pv -c -N tar | gzip -c | \ pv -c -N gzip | ssh user@box 'cat > /path/to/archive.tar.gz'
This one is even nicer, compressing the output as it goes. pv has a very nice feature, activated by the switch -c which uses cursor positioning escape characters to draw the progress indicator. When you have several pv processes doing this in a pipe-chain (such as the above command) all output is written neatly on screen, without interfering with each others’ updates. Using -N name will cause pv to write that name out before the progress indicator. This is a good thing, because when using the -c switch, there is no telling which pv process will end up on top of the other when drawing the progress bars.
netcat, swish army knife of TCP/IP
user@box1 $ pv /dev/sda | lzop -c | nc -l 9876 user@box2 $ nc box1 9876 > box1-sda.lzo
This simple set of commands has saved my ass quite a couple of times. It works across networked computers, on one box it let’s pv take in all the data on a partition (like dd or cat would), prints a progress indicator, compress the who thing with a minimal CPU overhead using lzop and then pipe it to netcat, which is listening for incoming TCP connections on port 9876. After that’s set up netcat will block until something connects to port 9876. On the second box we do just this and write that to a file. A remote backup which, although not pretty, does the job. Fast. There’s almost no overhead when using netcat for file transfers like this.
user@box1 $ nc -e '/bin/bash' -l -p 9876 user@box2 $ nc box1 9876
Perhaps the nastiest way to get a remote shell on a machine. box1 has netcat listening on port 9876, and when something connects to it netcat simply connects stdin and stdout of whatever process it starts (in this case bash) to the in- and output streams of the connection. On box2, you’ll be greeted not by your usual command prompt though, a lot of basic stuff gets left behind, but if you ever need a remote shell very very badly, this is one way to do it.
SSH reverse tunneling
user@firewalledbox $ ssh -R 1337:localhost:22 user@remotebox user@remotebox $ ssh localhost -p 1337
This one can be a bit tricky to grasp, it certainly took me some time before I got the hang of it. Basically this trick allows you to connect to a firewalled machine without fancy-pants hole-punching or other invasive tactics. The machine behind a firewall logs into a remote PC, and sets up a reverse tunnel back to itself. It binds to remotebox:1337, which in turn can connect to itself on that port to log into the firewalled machine. You can change the string 1337:localhost:22 to anything you want, as long as you get that the first port number is the one which SSH binds to on the remote box, and the hostname:port combo is the machine you want to connect to, this doesn’t have to be localhost, it can be any machine accessible to the firewalled box (like a server on the local network that’s not available from the internet).
I guess that’s it for now, I’ll probably come back later for another round with new tricks.