Archive for the ‘Tips and Tricks’ Category

Seventeen or Bust; a gentoo daemon script

Tuesday, July 12th, 2011

Seventeen or Bust

It’s been a while since I’ve just my available CPU cycles for vanity, but I’m a sucker for stats. Normally, you’d just start the binary mprime in some terminal and let it be, I didn’t like that for my servers (basically, I’d have to run SoB under some user, in some screen, and whenever something happens I’d have to manually interfere to kill it or restart it). So I wrote a very simple rc script to handle it for me.

The script

/etc/init.d/seventeenorbust

#!/sbin/runscript
 
depend() {
        use net
}
 
start() {
        PWHOME="$(getent passwd $USER | awk -F: '{ print $6 }')"
 
        ebegin "Starting SeventeenOrBust"
        env TERM="xterm" \
                start-stop-daemon \
                        --start \
                        --user $USER \
                        --env HOME="${PWHOME:-/home/$USER}" \
                        --name sob \
                        --exec /usr/bin/screen -- -dmS sob /home/sob/mprime -d
        eend $?
}
 
stop() {
        ebegin "Stopping SeventeenOrBust"
        start-stop-daemon --stop --signal 2 --name sob
        eend $?
}

/etc/conf.d/seventeenorbust

USER="sob"

Setup

  1. Copy the above files to your filesystem
  2. $ useradd -m -p "sob" sob
  3. Add a DenyUsers entry in your sshd_config for the sob account, don’t want people getting access to your box with a default user
  4. Login as the sob user, download and install the SoB binaries in your homedir and configure SoB accordingly (usually, this is just setting up your username in prime.txt
  5. (Optional); $ rc-update add seventeenorbust default

Like to view progress?

If you like to keep tabs on your instance, completion percentage and such, you can log in as the sob-user and run $ screen -dr sob to attach to te daemon’s screen session.

Posted in gentoo, Tips and Tricks | No Comments »

Secure and safe computing on the go

Tuesday, May 11th, 2010

I’m a bit paranoid when it comes to the “evil corporations and governments trying to spy on us all”, as such I like doing encryption tests on old an esoteric storage devices, like JAZ-drives and LS120 ‘SuperDisk’ technology. Recently though, especially because of all the news of laptops being checked on airports and such, I figured it was about time I got serious and see how (semi-) full-disk encryption works for GNU/Linux, specifically Gentoo.

This walkthrough assumes you are setting up a new Gentoo system and are not trying to migrate an existing system to a encrypted setup.

There are a couple of options mostly noted on blogs and in manuals;

Home-partition encryption
Perhaps the easiest to set up, this just encrypts the partition where all user data ought to go. In gentoo this is pretty easy, if I recall correctly there are init.d scripts in place that take care of mounting this for you and prompting for a passphrase at an appropriate time.
Full disk encryption
Also relatively easy, but this requires you to have another volume (usb key, cd-rom, floppy?) with you at all times to boot the system and decrypt the drive in order for the system to start.
Unencrypted boot partition
This devides the partitions on your drive between one small boot partition and a encrypted blob that is further broken up into smaller pieces using LVM or some similar volume management software.

To each his own, and I personally like the unencrypted boot, because it encrypts more than just my home folder, and when in place is easy to extend upon or change. The rest of this blog post is dedicated to the setup of this method of encryption, although the other two options could easily be derived from the steps taken.

First up, the partition layout. As noted in the description, the unencrypted boot-partition take on data encryption divides the drive up into two chunks, in my case like this:

Blockdevice Size
sda1 64 MiB
sda2 the rest

sda2 will first be encrypted and the encrypted contents will contain a LVM2 VG. This means that all LVs created from this VG will be fully encrypted under a shared passphrase. Adding and removing LVs (analogous with partitions) becomes very easy under LVM, and also adds some extra features like online resizing of the LV (if the underlying filesystem supports online resizing as well).

Phase 1 - Preparations

First things first, create the partitions as described above with your favourite partitioning tool (mine's cfdisk, but anything that partitions will do fine). When you've done this it's time to completely overwrite the volume-to-be-encrypted with random data, making it very difficult to estimate the amount of data hidden in the encrypted volume.

You can use a tool like shred for this task, but by default it overwrites the volume at thrice with pseudo-random data. This satisfies some paranoid people who believe magnetic drives retain some part of their previous polarization, perhaps making it possible to read data already overwritten. I don't have any proof in favor or against this theory, and if you have the time it can't hurt to have the driver fully overwritten three times. For those who want to get this done with as quickly as possible a simple dd will suffice. 

# A simple direct copy, use cat or pv for enhanced throughput
$ dd if=/dev/urandom of=/dev/sda2

-or-

# use -v for increased verbosity, i.e. a progress bar
$ shred -v /dev/sda2

Phase 2 - Encryption and LVM

This is where the magic happens, but it turns out to be really easy to do. First we encrypt sda2 using cryptsetup, next we create the VG and LVs so we can start using our new volumes.

# encrypt using the Rijndael cipher, use 256 bit key size
$ cryptsetup luksFormat -c aes -s 256 /dev/sda2
 
# open the newly created volume
$ cryptsetup luksOpen /dev/sda2 crypt

Alrighty, if all went as it should you should now have a decrypted volume located at /dev/mapper/crypt. Next up, LVM.

# I name my volume group after the computer's hostname
$ vgcreate cluebrick /dev/mapper/crypt
 
# create rootfs lv, 20 GiB big in the vg cluebrick
$ lvcreate -L 20G -n root cluebrick
 
# create swap lv, 2 GiB in the vg cluebrick
$ lvcreate -L 2G -n swap cluebrick
 
# create the home volume... etc
$ lvcreate -L 32G -n home cluebrick

So far so good, these LVs should be accessible through /dev/cluebrick/{root,swap,home}. If that's not the case, try this set of commands, outdated systems might pick up on the LVs after these.

# if at first you don't succeed...
$ vgchange -a n
$ vgexport -a 
$ vgimport -a
$ vgchange -a y

Phase 3 - System installation

I'm going to leave this one up to you, if you are a Gentoo buff you should be able to pick up from here, starting with formatting your newly created volumes. There are plenty tutorials out there that do a better job then I would at covering the installation. If you're going to migrate an existing system, this would be the time to start copying files back and forth.

Phase 4 - Kernel configuration

Rolling your own kernel isn't very difficult, nor is it a required part of the process, but for completeness' sake I'll note some things you need to take care of in order for this system to work. I do not recommend doing this if you don't have prior experience with compiling kernels, there's a lot more settings, flipswitches and knobby thingies that can royally bite you in the ass.

In the kernel menuconfig, be sure to check the following settings and adjust if necessary

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
 
Device Drivers  --->
  [*] Block devices  --->
    <*>   RAM block device support

If you like, you can let the kernel take care of the creation of a initramfs by setting the following variable:

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
    (/usr/src/initramfs)    Initramfs source file(s)

Depending on whether you chose to let the kernel roll your initramfs instead of you doing it yourself, a few steps will be different, I'll highlight the differences as they come. Personally I let the kernel roll the initramfs for me, that way I don't have to fiddle with my bootloader settings.

Phase 5 - Building the initramfs

In order for the system to boot you'll need what's called a early rootfs. Basically this is a small filesystem contained in your boot partition or inside the kernel image itself which as the tools and scripts on board to deal with the decryption of your rootfs and setting it up so Linux can continue booting as usual. It's important to note that this is a very small, purpose built rootfs, it won't contain a lot of tools you're used to when working on the command line. A short list of items that do go into this image, and the reason why. Also, because we have to be lean on functionality all programs included will have to be statically compiled, or you have to be willing to manually copy the required libraries, something that's hard to maintain and very cumbersome to debug when it hits the fan.

busybox
busybox contains most of the system tools you'll need in bootstrapping your system. Things like mount, sleep and a simple shell are all provided by one big binary which installs symlinks for the tools it provides when instructed to do so. Gentoo provides the USE-flag "static" in combination with it's package for this program, which causes busybox to also be built as a static binary that we can use.

LVM
This one is a no-brainer, we need LVM to access our root filesystem. Gentoo's Portage also provides the "static" USE-flag for this package, making it easy for us to use it for this purpose.
cryptsetup
Also a no-brainer. Here the USE-flag however is not "static", but "dynamic", and it needs to be explicitly negated. (So, add "-dynamic" the USE variable in make.conf)

Alright, let's get crackin'.

# make the fs structure
$ mkdir /usr/src/initramfs
$ cd /usr/src/initramfs
$ mkdir -p bin dev etc sbin usr var root mnt/root proc sys tmp
 
# copy essential binaries
$ cp `which busybox.static` bin/bb
$ cp `which cryptsetup` sbin
$ cp `which lvm.static` sbin/lvm
 
# housekeeping
$ touch init
$ chmod +x init
$ cd bin
$ ln -s bb busybox
$ cd ../sbin
 
# create symlinks for all LVM supported operations.
$ ./lvm help 2>&1 | grep -e '^ *[plv][vg]' | \
  awk '{print $1}' | xargs -n1 ln -s lvm
 
# on to /dev
$ cd ../dev
$ cp -a /dev/console /dev/tty /dev/zero /dev/null .

Next up, the init script. We're using busybox's shell, so don't expect anything too fancy scripting wise, we just need to get our system up, and that's it. The more you code, the more you can break. Copy the following listing into your script and customize where necessary.

#!/bin/busybox sh
 
busybox --install -s
 
mount -t proc none /proc
mount -t sysfs none /sys
 
# populate /dev
echo "/sbin/mdev" > /proc/sys/kernel/hotplug
mdev -s
 
# sleep 1 second so we don't get overrun by kprintf's
sleep 1
 
# safeguard against failed unlocking
cryptsetup luksOpen /dev/sda3 crypt || exec /bin/sh
 
# try and find all LVM volumes
vgscan
vgchange -a y
 
# safeguard against failed mount
mount /dev/cluebrick/root /mnt/root || exec /bin/sh
 
umount /proc
umount /sys
 
exec switch_root /mnt/root /sbin/init

Of course, this is a very crude init script, it'll get the job done, and even provide you with a rescue shell should the two major operations fail (unlocking the encrypted volume and mounting your decrypted root filesystem), but a lot of stuff could (and probably should) be added to make it more robust. There's plenty to read on the internet about init scripts in general, and expanding on this script should be fairly easy. One thing that you should consider is that when mdev populates /dev and switch_root does it's job, the created files will exists even in the new root filesystem, and might cause problems with device nodes not being where they should. If your rootfs' init process starts complaining about devices not being available, try manually populating /dev with the devices you need, and leave mdev out of it.

Phase 6 - Wrapping it up

For those who chose to let the kernel build the actual initramfs image from the structure in /usr/src/initramfs, you only have to do the usual bootloader antics to get it deployed, the image is contained in bzImage, as such, you should be set to give your setup a whirl. For the other group, we still have to build the image and make the bootloader pass it to the kernel. Here goes.

$ cd /usr/src/initramfs
$ find . -print0 | cpio -ov -0 --format=newc | \
  gzip -9 > /boot/initramfs.cpio.gz

This creates the initramfs image right in /boot, so make sure it's mounted before you do this. Next up is the bootloader. Most tutorials on the web use grub for this purpose, so I'll do it with lilo. Edit lilo.conf to suit your needs, making it look something like this:

image = /boot/bzImage
        label = gentoo
        root = /dev/cluebrick/root
        initrd = /boot/initramfs.cpio.gz
        read-only

Conclusion

If everything went as it should've, you now have a fully working, secure machine. There's a lot of stuff you could still add to the init script to make it more robust, like a telnet server when something fails, or some fancy echos that give a more verbose and colorful output when booting, but the core should be easy enough to extend upon. Any comments or criticism is welcome, but save me the grammar crud :) .

Tags:, , , , , , ,
Posted in Tips and Tricks | No Comments »

Tips and Tricks for console cowboys

Tuesday, May 11th, 2010

Howdy!

Okay, enough horsing around. UNIX command line paradigms are powerful ideas and methods of accomplishing things in a short time. This is my, far from complete, list of tricks I’ve picked up over the years using Linux & friends.

Speeding up tarball download/extract

$ wget -O- http://example.org/tarball.tar.gz | tar -xvzf-

This little one liner has saved time and disk space, basically it forces wget to dump the file you want to download into tar which directly starts extracting it as it comes in. This has the advantage of being one command you can easily type, not having the need to temporarily store the tarball while you download it and then extract the contents (making the extract a in-place operation) and speeding up the total time wasted, either your internet bandwidth or harddrive writing speed will be the bottleneck (most often the latter), but at least you won’t have to wait for one to finish to start on the other.

Bulk copy progress indicator

$ tar -cf- /path/to/dir | pv | \
  ssh user@box 'cat > /path/to/archive.tar'

This on-liner makes a tarball from a source directory, pipes that through the pv (pipe view) utility, which shows throughput speed, progress and ETA if it can determine it, and finally pumps it over the network to a remote box which writes it to a file. Instead of waiting around and wondering when the job will be done, you now at least have some idea how long it’s going to take.

$ tar -cf- /path/to/dir | pv -c -N tar | gzip -c | \
  pv -c -N gzip | ssh user@box 'cat > /path/to/archive.tar.gz'

This one is even nicer, compressing the output as it goes. pv has a very nice feature, activated by the switch -c which uses cursor positioning escape characters to draw the progress indicator. When you have several pv processes doing this in a pipe-chain (such as the above command) all output is written neatly on screen, without interfering with each others’ updates. Using -N name will cause pv to write that name out before the progress indicator. This is a good thing, because when using the -c switch, there is no telling which pv process will end up on top of the other when drawing the progress bars.

netcat, swish army knife of TCP/IP

user@box1 $ pv /dev/sda | lzop -c | nc -l 9876
user@box2 $ nc box1 9876 > box1-sda.lzo

This simple set of commands has saved my ass quite a couple of times. It works across networked computers, on one box it let’s pv take in all the data on a partition (like dd or cat would), prints a progress indicator, compress the who thing with a minimal CPU overhead using lzop and then pipe it to netcat, which is listening for incoming TCP connections on port 9876. After that’s set up netcat will block until something connects to port 9876. On the second box we do just this and write that to a file. A remote backup which, although not pretty, does the job. Fast. There’s almost no overhead when using netcat for file transfers like this.

user@box1 $ nc -e '/bin/bash' -l -p 9876
user@box2 $ nc box1 9876

Perhaps the nastiest way to get a remote shell on a machine. box1 has netcat listening on port 9876, and when something connects to it netcat simply connects stdin and stdout of whatever process it starts (in this case bash) to the in- and output streams of the connection. On box2, you’ll be greeted not by your usual command prompt though, a lot of basic stuff gets left behind, but if you ever need a remote shell very very badly, this is one way to do it.

SSH reverse tunneling

user@firewalledbox $ ssh -R 1337:localhost:22 user@remotebox
user@remotebox $ ssh localhost -p 1337

This one can be a bit tricky to grasp, it certainly took me some time before I got the hang of it. Basically this trick allows you to connect to a firewalled machine without fancy-pants hole-punching or other invasive tactics. The machine behind a firewall logs into a remote PC, and sets up a reverse tunnel back to itself. It binds to remotebox:1337, which in turn can connect to itself on that port to log into the firewalled machine. You can change the string 1337:localhost:22 to anything you want, as long as you get that the first port number is the one which SSH binds to on the remote box, and the hostname:port combo is the machine you want to connect to, this doesn’t have to be localhost, it can be any machine accessible to the firewalled box (like a server on the local network that’s not available from the internet).

I guess that’s it for now, I’ll probably come back later for another round with new tricks.

Posted in Tips and Tricks | No Comments »