May « 2010 « hatsuseno's blog

Archive for May, 2010

Secure and safe computing on the go

Tuesday, May 11th, 2010

I’m a bit paranoid when it comes to the “evil corporations and governments trying to spy on us all”, as such I like doing encryption tests on old an esoteric storage devices, like JAZ-drives and LS120 ‘SuperDisk’ technology. Recently though, especially because of all the news of laptops being checked on airports and such, I figured it was about time I got serious and see how (semi-) full-disk encryption works for GNU/Linux, specifically Gentoo.

This walkthrough assumes you are setting up a new Gentoo system and are not trying to migrate an existing system to a encrypted setup.

There are a couple of options mostly noted on blogs and in manuals;

Home-partition encryption: Perhaps the easiest to set up, this just encrypts the partition where all user data ought to go. In gentoo this is pretty easy, if I recall correctly there are init.d scripts in place that take care of mounting this for you and prompting for a passphrase at an appropriate time.
Full disk encryption: Also relatively easy, but this requires you to have another volume (usb key, cd-rom, floppy?) with you at all times to boot the system and decrypt the drive in order for the system to start.
Unencrypted boot partition: This devides the partitions on your drive between one small boot partition and a encrypted blob that is further broken up into smaller pieces using LVM or some similar volume management software.

To each his own, and I personally like the unencrypted boot, because it encrypts more than just my home folder, and when in place is easy to extend upon or change. The rest of this blog post is dedicated to the setup of this method of encryption, although the other two options could easily be derived from the steps taken.

First up, the partition layout. As noted in the description, the unencrypted boot-partition take on data encryption divides the drive up into two chunks, in my case like this:

Blockdevice	Size
sda1	64 MiB
sda2	the rest

sda2 will first be encrypted and the encrypted contents will contain a LVM2 VG. This means that all LVs created from this VG will be fully encrypted under a shared passphrase. Adding and removing LVs (analogous with partitions) becomes very easy under LVM, and also adds some extra features like online resizing of the LV (if the underlying filesystem supports online resizing as well).

Phase 1 - Preparations

First things first, create the partitions as described above with your favourite partitioning tool (mine's cfdisk, but anything that partitions will do fine). When you've done this it's time to completely overwrite the volume-to-be-encrypted with random data, making it very difficult to estimate the amount of data hidden in the encrypted volume.

You can use a tool like shred for this task, but by default it overwrites the volume at thrice with pseudo-random data. This satisfies some paranoid people who believe magnetic drives retain some part of their previous polarization, perhaps making it possible to read data already overwritten. I don't have any proof in favor or against this theory, and if you have the time it can't hurt to have the driver fully overwritten three times. For those who want to get this done with as quickly as possible a simple dd will suffice.

# A simple direct copy, use cat or pv for enhanced throughput
$ dd if=/dev/urandom of=/dev/sda2

-or-

# use -v for increased verbosity, i.e. a progress bar
$ shred -v /dev/sda2

Phase 2 - Encryption and LVM

This is where the magic happens, but it turns out to be really easy to do. First we encrypt sda2 using cryptsetup, next we create the VG and LVs so we can start using our new volumes.

# encrypt using the Rijndael cipher, use 256 bit key size
$ cryptsetup luksFormat -c aes -s 256 /dev/sda2
 
# open the newly created volume
$ cryptsetup luksOpen /dev/sda2 crypt

Alrighty, if all went as it should you should now have a decrypted volume located at /dev/mapper/crypt. Next up, LVM.

# I name my volume group after the computer's hostname
$ vgcreate cluebrick /dev/mapper/crypt
 
# create rootfs lv, 20 GiB big in the vg cluebrick
$ lvcreate -L 20G -n root cluebrick
 
# create swap lv, 2 GiB in the vg cluebrick
$ lvcreate -L 2G -n swap cluebrick
 
# create the home volume... etc
$ lvcreate -L 32G -n home cluebrick

So far so good, these LVs should be accessible through /dev/cluebrick/{root,swap,home}. If that's not the case, try this set of commands, outdated systems might pick up on the LVs after these.

# if at first you don't succeed...
$ vgchange -a n
$ vgexport -a 
$ vgimport -a
$ vgchange -a y

Phase 3 - System installation

I'm going to leave this one up to you, if you are a Gentoo buff you should be able to pick up from here, starting with formatting your newly created volumes. There are plenty tutorials out there that do a better job then I would at covering the installation. If you're going to migrate an existing system, this would be the time to start copying files back and forth.

Phase 4 - Kernel configuration

Rolling your own kernel isn't very difficult, nor is it a required part of the process, but for completeness' sake I'll note some things you need to take care of in order for this system to work. I do not recommend doing this if you don't have prior experience with compiling kernels, there's a lot more settings, flipswitches and knobby thingies that can royally bite you in the ass.

In the kernel menuconfig, be sure to check the following settings and adjust if necessary

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
 
Device Drivers  --->
  [*] Block devices  --->
    <*>   RAM block device support

If you like, you can let the kernel take care of the creation of a initramfs by setting the following variable:

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
    (/usr/src/initramfs)    Initramfs source file(s)

Depending on whether you chose to let the kernel roll your initramfs instead of you doing it yourself, a few steps will be different, I'll highlight the differences as they come. Personally I let the kernel roll the initramfs for me, that way I don't have to fiddle with my bootloader settings.

Phase 5 - Building the initramfs

In order for the system to boot you'll need what's called a early rootfs. Basically this is a small filesystem contained in your boot partition or inside the kernel image itself which as the tools and scripts on board to deal with the decryption of your rootfs and setting it up so Linux can continue booting as usual. It's important to note that this is a very small, purpose built rootfs, it won't contain a lot of tools you're used to when working on the command line. A short list of items that do go into this image, and the reason why. Also, because we have to be lean on functionality all programs included will have to be statically compiled, or you have to be willing to manually copy the required libraries, something that's hard to maintain and very cumbersome to debug when it hits the fan.

busybox: busybox contains most of the system tools you'll need in bootstrapping your system. Things like mount, sleep and a simple shell are all provided by one big binary which installs symlinks for the tools it provides when instructed to do so. Gentoo provides the USE-flag "static" in combination with it's package for this program, which causes busybox to also be built as a static binary that we can use.
LVM: This one is a no-brainer, we need LVM to access our root filesystem. Gentoo's Portage also provides the "static" USE-flag for this package, making it easy for us to use it for this purpose.
cryptsetup: Also a no-brainer. Here the USE-flag however is not "static", but "dynamic", and it needs to be explicitly negated. (So, add "-dynamic" the USE variable in make.conf)

Alright, let's get crackin'.

# make the fs structure
$ mkdir /usr/src/initramfs
$ cd /usr/src/initramfs
$ mkdir -p bin dev etc sbin usr var root mnt/root proc sys tmp
 
# copy essential binaries
$ cp `which busybox.static` bin/bb
$ cp `which cryptsetup` sbin
$ cp `which lvm.static` sbin/lvm
 
# housekeeping
$ touch init
$ chmod +x init
$ cd bin
$ ln -s bb busybox
$ cd ../sbin
 
# create symlinks for all LVM supported operations.
$ ./lvm help 2>&1 | grep -e '^ *[plv][vg]' | \
  awk '{print $1}' | xargs -n1 ln -s lvm
 
# on to /dev
$ cd ../dev
$ cp -a /dev/console /dev/tty /dev/zero /dev/null .

Next up, the init script. We're using busybox's shell, so don't expect anything too fancy scripting wise, we just need to get our system up, and that's it. The more you code, the more you can break. Copy the following listing into your script and customize where necessary.

#!/bin/busybox sh
 
busybox --install -s
 
mount -t proc none /proc
mount -t sysfs none /sys
 
# populate /dev
echo "/sbin/mdev" > /proc/sys/kernel/hotplug
mdev -s
 
# sleep 1 second so we don't get overrun by kprintf's
sleep 1
 
# safeguard against failed unlocking
cryptsetup luksOpen /dev/sda3 crypt || exec /bin/sh
 
# try and find all LVM volumes
vgscan
vgchange -a y
 
# safeguard against failed mount
mount /dev/cluebrick/root /mnt/root || exec /bin/sh
 
umount /proc
umount /sys
 
exec switch_root /mnt/root /sbin/init

Of course, this is a very crude init script, it'll get the job done, and even provide you with a rescue shell should the two major operations fail (unlocking the encrypted volume and mounting your decrypted root filesystem), but a lot of stuff could (and probably should) be added to make it more robust. There's plenty to read on the internet about init scripts in general, and expanding on this script should be fairly easy. One thing that you should consider is that when mdev populates /dev and switch_root does it's job, the created files will exists even in the new root filesystem, and might cause problems with device nodes not being where they should. If your rootfs' init process starts complaining about devices not being available, try manually populating /dev with the devices you need, and leave mdev out of it.

Phase 6 - Wrapping it up

For those who chose to let the kernel build the actual initramfs image from the structure in /usr/src/initramfs, you only have to do the usual bootloader antics to get it deployed, the image is contained in bzImage, as such, you should be set to give your setup a whirl. For the other group, we still have to build the image and make the bootloader pass it to the kernel. Here goes.

$ cd /usr/src/initramfs
$ find . -print0 | cpio -ov -0 --format=newc | \
  gzip -9 > /boot/initramfs.cpio.gz

This creates the initramfs image right in /boot, so make sure it's mounted before you do this. Next up is the bootloader. Most tutorials on the web use grub for this purpose, so I'll do it with lilo. Edit lilo.conf to suit your needs, making it look something like this:

image = /boot/bzImage
        label = gentoo
        root = /dev/cluebrick/root
        initrd = /boot/initramfs.cpio.gz
        read-only

Conclusion

If everything went as it should've, you now have a fully working, secure machine. There's a lot of stuff you could still add to the init script to make it more robust, like a telnet server when something fails, or some fancy echos that give a more verbose and colorful output when booting, but the core should be easy enough to extend upon. Any comments or criticism is welcome, but save me the grammar crud .

Tags:cryptsetup, dmcrypt, encryption, initramfs, linux, luks, lvm, lvm2
Posted in Tips and Tricks | No Comments »

Tips and Tricks for console cowboys

Tuesday, May 11th, 2010

Howdy!

Okay, enough horsing around. UNIX command line paradigms are powerful ideas and methods of accomplishing things in a short time. This is my, far from complete, list of tricks I’ve picked up over the years using Linux & friends.

Speeding up tarball download/extract

$ wget -O- http://example.org/tarball.tar.gz | tar -xvzf-

This little one liner has saved time and disk space, basically it forces wget to dump the file you want to download into tar which directly starts extracting it as it comes in. This has the advantage of being one command you can easily type, not having the need to temporarily store the tarball while you download it and then extract the contents (making the extract a in-place operation) and speeding up the total time wasted, either your internet bandwidth or harddrive writing speed will be the bottleneck (most often the latter), but at least you won’t have to wait for one to finish to start on the other.

Bulk copy progress indicator

$ tar -cf- /path/to/dir | pv | \
  ssh user@box 'cat > /path/to/archive.tar'

This on-liner makes a tarball from a source directory, pipes that through the pv (pipe view) utility, which shows throughput speed, progress and ETA if it can determine it, and finally pumps it over the network to a remote box which writes it to a file. Instead of waiting around and wondering when the job will be done, you now at least have some idea how long it’s going to take.

$ tar -cf- /path/to/dir | pv -c -N tar | gzip -c | \
  pv -c -N gzip | ssh user@box 'cat > /path/to/archive.tar.gz'

This one is even nicer, compressing the output as it goes. pv has a very nice feature, activated by the switch -c which uses cursor positioning escape characters to draw the progress indicator. When you have several pv processes doing this in a pipe-chain (such as the above command) all output is written neatly on screen, without interfering with each others’ updates. Using -N name will cause pv to write that name out before the progress indicator. This is a good thing, because when using the -c switch, there is no telling which pv process will end up on top of the other when drawing the progress bars.

netcat, swish army knife of TCP/IP

user@box1 $ pv /dev/sda | lzop -c | nc -l 9876
user@box2 $ nc box1 9876 > box1-sda.lzo

This simple set of commands has saved my ass quite a couple of times. It works across networked computers, on one box it let’s pv take in all the data on a partition (like dd or cat would), prints a progress indicator, compress the who thing with a minimal CPU overhead using lzop and then pipe it to netcat, which is listening for incoming TCP connections on port 9876. After that’s set up netcat will block until something connects to port 9876. On the second box we do just this and write that to a file. A remote backup which, although not pretty, does the job. Fast. There’s almost no overhead when using netcat for file transfers like this.

user@box1 $ nc -e '/bin/bash' -l -p 9876
user@box2 $ nc box1 9876

Perhaps the nastiest way to get a remote shell on a machine. box1 has netcat listening on port 9876, and when something connects to it netcat simply connects stdin and stdout of whatever process it starts (in this case bash) to the in- and output streams of the connection. On box2, you’ll be greeted not by your usual command prompt though, a lot of basic stuff gets left behind, but if you ever need a remote shell very very badly, this is one way to do it.

SSH reverse tunneling

user@firewalledbox $ ssh -R 1337:localhost:22 user@remotebox
user@remotebox $ ssh localhost -p 1337

This one can be a bit tricky to grasp, it certainly took me some time before I got the hang of it. Basically this trick allows you to connect to a firewalled machine without fancy-pants hole-punching or other invasive tactics. The machine behind a firewall logs into a remote PC, and sets up a reverse tunnel back to itself. It binds to remotebox:1337, which in turn can connect to itself on that port to log into the firewalled machine. You can change the string 1337:localhost:22 to anything you want, as long as you get that the first port number is the one which SSH binds to on the remote box, and the hostname:port combo is the machine you want to connect to, this doesn’t have to be localhost, it can be any machine accessible to the firewalled box (like a server on the local network that’s not available from the internet).

I guess that’s it for now, I’ll probably come back later for another round with new tricks.

Posted in Tips and Tricks | No Comments »

Custom order sorting in PHP

Sunday, May 2nd, 2010

Every once in a while you find your self in need of a generic sorting function that orders in a very specific, and non-machine understandable way. Like sorting items from ‘one’ to ‘four’. Instead of building a small subset of those kind of sort operations into the language PHP has usort family (short for user-defined sort). In the background this sort still works as a quicksort, but the developer is tasked with making a function that does the comparison between different elements of the array. It’s up to him to define whether a value is smaller, equal or bigger than another.

So, keeping the ‘one’ to ‘four’ example, I present you with a very simple custom sort function.

function customCompare($a, $b) {
  // We flip the array to get the positional indices 
  $order = array_flip(array('one', 'two', 'three', 'four'));
 
  // Sanity check, $a and $b must exist in the order array
  foreach(array($a, $b) as $arg) {
    if(!array_key_exists($arg, $order))
      trigger_error("`$arg' is not a sortable key");
  }
 
  // The positions in the order array dictate if $a is bigger
  // than $b, as such we can simply subtract the integers
  // from the flipper order array to get an integer value
  // which is smaller, equal or bigger than 0 depending on
  // $a and $b's relative position in the array.
  return $order[$a] - $order[$b];
}
 
// An example array to be sorted
$example = array(
  'four' => 4,
  'one' => 1,
  'three' => 3,
  'two' => 2);
 
// Do the actual sort
uksort($example, 'customCompare');
 
// Dump results
print_r($example);

The dumped array will look like this

Array
(
    [one] => 1
    [two] => 2
    [three] => 3
    [four] => 4
)

If you, like me, like closures and are running PHP 5.3, we can make the compare function a lot nicer and easier to maintain.

function createSorter($order) {
  $order = array_flip($order);
 
  return function($a, $b) use ($order) {
    // Sanity check, again
    foreach(array($a, $b) as $arg) {
      if(!array_key_exists($arg, $order))
        trigger_error("`$arg' is not a sortable key");
    }
 
    return $order[$a] - $order[$b];
  }
}
 
// Sort array $array in the 'one' to 'four' order.
uksort($array, createSorter(
  array('one', 'two', 'three', 'four')));

Tags:algorithms, code, quicksort
Posted in PHP | No Comments »

hatsuseno's blog