It’s been a while since I’ve just my available CPU cycles for vanity, but I’m a sucker for stats. Normally, you’d just start the binary mprime in some terminal and let it be, I didn’t like that for my servers (basically, I’d have to run SoB under some user, in some screen, and whenever something happens I’d have to manually interfere to kill it or restart it). So I wrote a very simple rc script to handle it for me.

The script

/etc/init.d/seventeenorbust

#!/sbin/runscript
 
depend() {
        use net
}
 
start() {
        PWHOME="$(getent passwd $USER | awk -F: '{ print $6 }')"
 
        ebegin "Starting SeventeenOrBust"
        env TERM="xterm" \
                start-stop-daemon \
                        --start \
                        --user $USER \
                        --env HOME="${PWHOME:-/home/$USER}" \
                        --name sob \
                        --exec /usr/bin/screen -- -dmS sob /home/sob/mprime -d
        eend $?
}
 
stop() {
        ebegin "Stopping SeventeenOrBust"
        start-stop-daemon --stop --signal 2 --name sob
        eend $?
}

/etc/conf.d/seventeenorbust

USER="sob"

Setup

1. Copy the above files to your filesystem
2. $ useradd -m -p "sob" sob
3. Add a DenyUsers entry in your sshd_config for the sob account, don’t want people getting access to your box with a default user
4. Login as the sob user, download and install the SoB binaries in your homedir and configure SoB accordingly (usually, this is just setting up your username in prime.txt
5. (Optional); $ rc-update add seventeenorbust default

Like to view progress?

If you like to keep tabs on your instance, completion percentage and such, you can log in as the sob-user and run $ screen -dr sob to attach to te daemon’s screen session.

Recently HTC rolled out the Android 2.2 upgrade for my Desire, finally enabling USB Tethering out of the box. Often enough I used the WiFi hotspot feature on my previous phone to enable my laptop to have internet on the go. Android’s got this feature as well nowadays, but I don’t really like it because it really, and I mean *really*, puts the drain on your phone’s battery. USB Tethering seems like a much better solution.

Setup for Android

This is by far the easiest part, just hook up your phone, drag down the status bar and instead of charge only, or whatever mode it is in at that time, set it to USB Tethering.

Setup for Gentoo

This part is a bit trickier, especially when you roll your own kernel. Open up menuconfig for your kernel and enable these options

Device Drivers --->
  [*] Network device support --->
    USB Network Adapters --->
      [*] Multi-purpose USB Networking Framework
        <*> CDC Ethernet support
        <*> CDC EEM support
        <*> Simple USB Network Links (CDC Ethernet subset)
          [*] Embedded ARM Linux links
  [*] USB Support --->
    <*> USB Modem (CDC ACM) support
    <*> USB Wireless Device Management support

Now build your kernel as usual and reboot. When you now plug in your Android phone when it’s in USB Tethering mode, if all is well, you should be able to see a new network interface, usbX, X usually being 0. The link might be down, so check it with the command

$ ifconfig -a

Which will output something like this

usb0      Link encap:Ethernet  HWaddr 26:10:72:ab:38:0e  
            BROADCAST MULTICAST  MTU:1500  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Which means it all worked. Now all you have to do is run a DHCP request on that interface, and Android will handle the rest. You can set up a init.d script for this interface much the same way you would for a normal interface. Just make a symlink from /etc/init.d/net.lo to /etc/init.d/net.usb0 and run that script whenever you want.

Beautifying the whole thing

Running around on the console starting bootscripts like that doesn’t suit everyone, including me when I’m just on the road and stuff needs to work. I recommend using something like wicd or NetworkManager to handle it for you. Using wicd it’s pretty easy to make it automagically use the usb0 interface when it appears, so you’ll only have to plug it on for it to work.

grabw is a bash script that allows you to

1. Take a screenshot of a window by X window ID
2. Take a screenshot of a window the user first has to click, using xwininfo
3. The above two with or without asking for a filename using Zenity’s file selection dialogue window

Combine this with awesome’s ability to spawn processes on the fly and pass parameters (such as window IDs, *hint hint*) this is one feature I’m glad to have back

grabw 0.2 has been released into public domain, you can find it on my projects page.

This walkthrough assumes you are setting up a new Gentoo system and are not trying to migrate an existing system to a encrypted setup.

There are a couple of options mostly noted on blogs and in manuals;

Home-partition encryption

Perhaps the easiest to set up, this just encrypts the partition where all user data ought to go. In gentoo this is pretty easy, if I recall correctly there are init.d scripts in place that take care of mounting this for you and prompting for a passphrase at an appropriate time.

Full disk encryption

Also relatively easy, but this requires you to have another volume (usb key, cd-rom, floppy?) with you at all times to boot the system and decrypt the drive in order for the system to start.

Unencrypted boot partition

This devides the partitions on your drive between one small boot partition and a encrypted blob that is further broken up into smaller pieces using LVM or some similar volume management software.

To each his own, and I personally like the unencrypted boot, because it encrypts more than just my home folder, and when in place is easy to extend upon or change. The rest of this blog post is dedicated to the setup of this method of encryption, although the other two options could easily be derived from the steps taken.

First up, the partition layout. As noted in the description, the unencrypted boot-partition take on data encryption divides the drive up into two chunks, in my case like this:

sda2 will first be encrypted and the encrypted contents will contain a LVM2 VG. This means that all LVs created from this VG will be fully encrypted under a shared passphrase. Adding and removing LVs (analogous with partitions) becomes very easy under LVM, and also adds some extra features like online resizing of the LV (if the underlying filesystem supports online resizing as well).

Phase 1 - Preparations

First things first, create the partitions as described above with your favourite partitioning tool (mine's cfdisk, but anything that partitions will do fine). When you've done this it's time to completely overwrite the volume-to-be-encrypted with random data, making it very difficult to estimate the amount of data hidden in the encrypted volume.

You can use a tool like shred for this task, but by default it overwrites the volume at thrice with pseudo-random data. This satisfies some paranoid people who believe magnetic drives retain some part of their previous polarization, perhaps making it possible to read data already overwritten. I don't have any proof in favor or against this theory, and if you have the time it can't hurt to have the driver fully overwritten three times. For those who want to get this done with as quickly as possible a simple dd will suffice.

# A simple direct copy, use cat or pv for enhanced throughput
$ dd if=/dev/urandom of=/dev/sda2

-or-

# use -v for increased verbosity, i.e. a progress bar
$ shred -v /dev/sda2

Phase 2 - Encryption and LVM

This is where the magic happens, but it turns out to be really easy to do. First we encrypt sda2 using cryptsetup, next we create the VG and LVs so we can start using our new volumes.

# encrypt using the Rijndael cipher, use 256 bit key size
$ cryptsetup luksFormat -c aes -s 256 /dev/sda2
 
# open the newly created volume
$ cryptsetup luksOpen /dev/sda2 crypt

Alrighty, if all went as it should you should now have a decrypted volume located at /dev/mapper/crypt. Next up, LVM.

# I name my volume group after the computer's hostname
$ vgcreate cluebrick /dev/mapper/crypt
 
# create rootfs lv, 20 GiB big in the vg cluebrick
$ lvcreate -L 20G -n root cluebrick
 
# create swap lv, 2 GiB in the vg cluebrick
$ lvcreate -L 2G -n swap cluebrick
 
# create the home volume... etc
$ lvcreate -L 32G -n home cluebrick

So far so good, these LVs should be accessible through /dev/cluebrick/{root,swap,home}. If that's not the case, try this set of commands, outdated systems might pick up on the LVs after these.

# if at first you don't succeed...
$ vgchange -a n
$ vgexport -a 
$ vgimport -a
$ vgchange -a y

Phase 3 - System installation

I'm going to leave this one up to you, if you are a Gentoo buff you should be able to pick up from here, starting with formatting your newly created volumes. There are plenty tutorials out there that do a better job then I would at covering the installation. If you're going to migrate an existing system, this would be the time to start copying files back and forth.

Phase 4 - Kernel configuration

Rolling your own kernel isn't very difficult, nor is it a required part of the process, but for completeness' sake I'll note some things you need to take care of in order for this system to work. I do not recommend doing this if you don't have prior experience with compiling kernels, there's a lot more settings, flipswitches and knobby thingies that can royally bite you in the ass.

In the kernel menuconfig, be sure to check the following settings and adjust if necessary

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
 
Device Drivers  --->
  [*] Block devices  --->
    <*>   RAM block device support

If you like, you can let the kernel take care of the creation of a initramfs by setting the following variable:

General setup  --->
  [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
    (/usr/src/initramfs)    Initramfs source file(s)

Depending on whether you chose to let the kernel roll your initramfs instead of you doing it yourself, a few steps will be different, I'll highlight the differences as they come. Personally I let the kernel roll the initramfs for me, that way I don't have to fiddle with my bootloader settings.

Phase 5 - Building the initramfs

In order for the system to boot you'll need what's called a early rootfs. Basically this is a small filesystem contained in your boot partition or inside the kernel image itself which as the tools and scripts on board to deal with the decryption of your rootfs and setting it up so Linux can continue booting as usual. It's important to note that this is a very small, purpose built rootfs, it won't contain a lot of tools you're used to when working on the command line. A short list of items that do go into this image, and the reason why. Also, because we have to be lean on functionality all programs included will have to be statically compiled, or you have to be willing to manually copy the required libraries, something that's hard to maintain and very cumbersome to debug when it hits the fan.

busybox

busybox contains most of the system tools you'll need in bootstrapping your system. Things like mount, sleep and a simple shell are all provided by one big binary which installs symlinks for the tools it provides when instructed to do so. Gentoo provides the USE-flag "static" in combination with it's package for this program, which causes busybox to also be built as a static binary that we can use.

LVM

This one is a no-brainer, we need LVM to access our root filesystem. Gentoo's Portage also provides the "static" USE-flag for this package, making it easy for us to use it for this purpose.

cryptsetup

Also a no-brainer. Here the USE-flag however is not "static", but "dynamic", and it needs to be explicitly negated. (So, add "-dynamic" the USE variable in make.conf)

Alright, let's get crackin'.

# make the fs structure
$ mkdir /usr/src/initramfs
$ cd /usr/src/initramfs
$ mkdir -p bin dev etc sbin usr var root mnt/root proc sys tmp
 
# copy essential binaries
$ cp `which busybox.static` bin/bb
$ cp `which cryptsetup` sbin
$ cp `which lvm.static` sbin/lvm
 
# housekeeping
$ touch init
$ chmod +x init
$ cd bin
$ ln -s bb busybox
$ cd ../sbin
 
# create symlinks for all LVM supported operations.
$ ./lvm help 2>&1 | grep -e '^ *[plv][vg]' | \
  awk '{print $1}' | xargs -n1 ln -s lvm
 
# on to /dev
$ cd ../dev
$ cp -a /dev/console /dev/tty /dev/zero /dev/null .

Next up, the init script. We're using busybox's shell, so don't expect anything too fancy scripting wise, we just need to get our system up, and that's it. The more you code, the more you can break. Copy the following listing into your script and customize where necessary.

#!/bin/busybox sh
 
busybox --install -s
 
mount -t proc none /proc
mount -t sysfs none /sys
 
# populate /dev
echo "/sbin/mdev" > /proc/sys/kernel/hotplug
mdev -s
 
# sleep 1 second so we don't get overrun by kprintf's
sleep 1
 
# safeguard against failed unlocking
cryptsetup luksOpen /dev/sda3 crypt || exec /bin/sh
 
# try and find all LVM volumes
vgscan
vgchange -a y
 
# safeguard against failed mount
mount /dev/cluebrick/root /mnt/root || exec /bin/sh
 
umount /proc
umount /sys
 
exec switch_root /mnt/root /sbin/init

Of course, this is a very crude init script, it'll get the job done, and even provide you with a rescue shell should the two major operations fail (unlocking the encrypted volume and mounting your decrypted root filesystem), but a lot of stuff could (and probably should) be added to make it more robust. There's plenty to read on the internet about init scripts in general, and expanding on this script should be fairly easy. One thing that you should consider is that when mdev populates /dev and switch_root does it's job, the created files will exists even in the new root filesystem, and might cause problems with device nodes not being where they should. If your rootfs' init process starts complaining about devices not being available, try manually populating /dev with the devices you need, and leave mdev out of it.

Phase 6 - Wrapping it up

For those who chose to let the kernel build the actual initramfs image from the structure in /usr/src/initramfs, you only have to do the usual bootloader antics to get it deployed, the image is contained in bzImage, as such, you should be set to give your setup a whirl. For the other group, we still have to build the image and make the bootloader pass it to the kernel. Here goes.

$ cd /usr/src/initramfs
$ find . -print0 | cpio -ov -0 --format=newc | \
  gzip -9 > /boot/initramfs.cpio.gz

This creates the initramfs image right in /boot, so make sure it's mounted before you do this. Next up is the bootloader. Most tutorials on the web use grub for this purpose, so I'll do it with lilo. Edit lilo.conf to suit your needs, making it look something like this:

image = /boot/bzImage
        label = gentoo
        root = /dev/cluebrick/root
        initrd = /boot/initramfs.cpio.gz
        read-only

Conclusion

If everything went as it should've, you now have a fully working, secure machine. There's a lot of stuff you could still add to the init script to make it more robust, like a telnet server when something fails, or some fancy echos that give a more verbose and colorful output when booting, but the core should be easy enough to extend upon. Any comments or criticism is welcome, but save me the grammar crud .

Okay, enough horsing around. UNIX command line paradigms are powerful ideas and methods of accomplishing things in a short time. This is my, far from complete, list of tricks I’ve picked up over the years using Linux & friends.

Speeding up tarball download/extract

$ wget -O- http://example.org/tarball.tar.gz | tar -xvzf-

This little one liner has saved time and disk space, basically it forces wget to dump the file you want to download into tar which directly starts extracting it as it comes in. This has the advantage of being one command you can easily type, not having the need to temporarily store the tarball while you download it and then extract the contents (making the extract a in-place operation) and speeding up the total time wasted, either your internet bandwidth or harddrive writing speed will be the bottleneck (most often the latter), but at least you won’t have to wait for one to finish to start on the other.

Bulk copy progress indicator

$ tar -cf- /path/to/dir | pv | \
  ssh user@box 'cat > /path/to/archive.tar'

This on-liner makes a tarball from a source directory, pipes that through the pv (pipe view) utility, which shows throughput speed, progress and ETA if it can determine it, and finally pumps it over the network to a remote box which writes it to a file. Instead of waiting around and wondering when the job will be done, you now at least have some idea how long it’s going to take.

$ tar -cf- /path/to/dir | pv -c -N tar | gzip -c | \
  pv -c -N gzip | ssh user@box 'cat > /path/to/archive.tar.gz'

This one is even nicer, compressing the output as it goes. pv has a very nice feature, activated by the switch -c which uses cursor positioning escape characters to draw the progress indicator. When you have several pv processes doing this in a pipe-chain (such as the above command) all output is written neatly on screen, without interfering with each others’ updates. Using -N name will cause pv to write that name out before the progress indicator. This is a good thing, because when using the -c switch, there is no telling which pv process will end up on top of the other when drawing the progress bars.

netcat, swish army knife of TCP/IP

user@box1 $ pv /dev/sda | lzop -c | nc -l 9876
user@box2 $ nc box1 9876 > box1-sda.lzo

This simple set of commands has saved my ass quite a couple of times. It works across networked computers, on one box it let’s pv take in all the data on a partition (like dd or cat would), prints a progress indicator, compress the who thing with a minimal CPU overhead using lzop and then pipe it to netcat, which is listening for incoming TCP connections on port 9876. After that’s set up netcat will block until something connects to port 9876. On the second box we do just this and write that to a file. A remote backup which, although not pretty, does the job. Fast. There’s almost no overhead when using netcat for file transfers like this.

user@box1 $ nc -e '/bin/bash' -l -p 9876
user@box2 $ nc box1 9876

Perhaps the nastiest way to get a remote shell on a machine. box1 has netcat listening on port 9876, and when something connects to it netcat simply connects stdin and stdout of whatever process it starts (in this case bash) to the in- and output streams of the connection. On box2, you’ll be greeted not by your usual command prompt though, a lot of basic stuff gets left behind, but if you ever need a remote shell very very badly, this is one way to do it.

SSH reverse tunneling

user@firewalledbox $ ssh -R 1337:localhost:22 user@remotebox
user@remotebox $ ssh localhost -p 1337

This one can be a bit tricky to grasp, it certainly took me some time before I got the hang of it. Basically this trick allows you to connect to a firewalled machine without fancy-pants hole-punching or other invasive tactics. The machine behind a firewall logs into a remote PC, and sets up a reverse tunnel back to itself. It binds to remotebox:1337, which in turn can connect to itself on that port to log into the firewalled machine. You can change the string 1337:localhost:22 to anything you want, as long as you get that the first port number is the one which SSH binds to on the remote box, and the hostname:port combo is the machine you want to connect to, this doesn’t have to be localhost, it can be any machine accessible to the firewalled box (like a server on the local network that’s not available from the internet).

I guess that’s it for now, I’ll probably come back later for another round with new tricks.

So, keeping the ‘one’ to ‘four’ example, I present you with a very simple custom sort function.

function customCompare($a, $b) {
  // We flip the array to get the positional indices 
  $order = array_flip(array('one', 'two', 'three', 'four'));
 
  // Sanity check, $a and $b must exist in the order array
  foreach(array($a, $b) as $arg) {
    if(!array_key_exists($arg, $order))
      trigger_error("`$arg' is not a sortable key");
  }
 
  // The positions in the order array dictate if $a is bigger
  // than $b, as such we can simply subtract the integers
  // from the flipper order array to get an integer value
  // which is smaller, equal or bigger than 0 depending on
  // $a and $b's relative position in the array.
  return $order[$a] - $order[$b];
}
 
// An example array to be sorted
$example = array(
  'four' => 4,
  'one' => 1,
  'three' => 3,
  'two' => 2);
 
// Do the actual sort
uksort($example, 'customCompare');
 
// Dump results
print_r($example);

The dumped array will look like this

Array
(
    [one] => 1
    [two] => 2
    [three] => 3
    [four] => 4
)

If you, like me, like closures and are running PHP 5.3, we can make the compare function a lot nicer and easier to maintain.

function createSorter($order) {
  $order = array_flip($order);
 
  return function($a, $b) use ($order) {
    // Sanity check, again
    foreach(array($a, $b) as $arg) {
      if(!array_key_exists($arg, $order))
        trigger_error("`$arg' is not a sortable key");
    }
 
    return $order[$a] - $order[$b];
  }
}
 
// Sort array $array in the 'one' to 'four' order.
uksort($array, createSorter(
  array('one', 'two', 'three', 'four')));

I’ve been a Gentoo user since forever, and for some time I’ve been using awesome for a window manager instead of the desktop environments Gnome or KDE. One thing I never got around to change was the wallpaper. My terminals are pseudo-transparent, and I never disliked the default ‘awesome’ wallpaper. @ZackLeonhart‘s bitching and moaning changed that.

Turns out that aside from editing the system-wide theme script, by default located in /usr/share/awesome/themes/* (which I consider to be a BadThink™), there is no straight forward way of doing it nicely in the Gentoo provided default setup. So here’s what you do.

1. Create yourself a lua.rc in ${HOME}/.config/awesome
$ cp /etc/xdg/awesome/rc.lua ~/.config/awesome
2. Recursively copy all themes from the system-wide dirs to your local configdir.
   $ cp -rvf /usr/share/awesome/themes ~/.config/awesome
3. In your copy of lua.rc you will find a variable called theme_path, change it to theme_path = os.getenv("HOME") .. "/.config/awesome/themes/path/to/theme.lua"
4. Edit the variable theme.wallpaper_cmd to contain the string "eval `cat ~/.fehbg`"
5. Choose a wallpaper you like, execute $ feh --bg-scale /path/to/wallpaper.jpg

And you’re done! If you want to change your wallpaper, simply use feh again with a random other file, and make sure the file is readable when awesome is starting up, and it’ll set it up automagically. Basically, awesome’s ‘beautiful’ library takes your typed out command and executes that in order to set the wallpaper, feh has a convenient way of storing the default wallpaper by simply dumping the command that was used to set it up in the first place in a configuration file in your home directory, hence the eval `cat whatever` trick.

Screenshot of my new wallpaper in Awesome

P.S.: Don’t think you can leave out the call to os.getenv() in step 3, awesome doesn’t expand the tilde (‘~’), so the path it’ll try to access will be invalid.

P.P.S.: This is more of a note for me, in case I ever lose my epic new wallpaper. It can be found here.

The spirit remains, but there have been a few changes in the language as a whole. The entire codeblock-as-type feature has been dropped until it can be further perfected syntactically, and I’ve taken a more pragmatic approach in general, things need to get done after all.

Built-in types

boolean

A boolean type, true or false.

integer

Pretty obvious, integer numbers, 64bit signed.

float

Less obvious, implemented as a C-style double.

string

A string type, most types can be cast to this.

list

Simple, LISP-style, linked-list.

array

Variable-length, numerically indexed arrays.

hash

Variable-length hashmap, /[[:alnum:]_]+?/ keys.

function

Standard first-class citizen functions.

class

First-class citizen classes.

dynamic

Special untyped variable, can contain *any* type value at any time, default type

Examples of primitive types

boolean b = true;
integer x = 5;
float j = 5.4E-4;
string s = 'test string';
list l = list(1, 2, 3, 4, 5);
array m = array('a', 'b', 'c', 'd', 'e');
hash z = hash(a: 1, b: 2, c: 3, d: 4, test_key: 666);
 
function f = function(a, b, c, d) uses (m) {
  sum(a, b, c, d, m[1]);
};
 
class c = class(
  private integer attr_name: 5,
  public function constructor myConstructor: function() {
    this.attr_name *= 10;
  }
);
dynamic y = 20;
p = 20;

A few sidenotes;

• Keys in hash-maps are unquoted, because they are properly constrained.
• Functions are always closures, but require explicit declaration of the enclosed variables.
• Classes syntactically resemble hash-maps, but have modifier keywords, like private, protected, constructor and destructor.
• The variables y and p are for all intents and purposes identical in content and type, untyped variables *are* dynamic variables.
• All composite types (like lists, arrays and classes) have a constructor-function, which is an actual function that can be called with eval()-esque intent.

In-depth look into the composite types

Lists

Lists are singly-linked lists, and the functions list(), first() and rest() are the primitive operators on lists.

x = list(1, 2, 3, 4, 5);
first(x);
  => 1
rest(x);
  => list(3, 4, 5)

first() and rest() are modifying functions, they pop from the list whatever they return, so the list x is now a list with the value (2). Trying to pop from an empty array results in a ‘null’ value being returned. While-loops can be constructed around this, but lists are iteratable, so the built-in foreach() will do just fine.

foreach(list(1, 2, 3, 4, 5) as li) { ... }

Arrays

Arrays are numerically indexed, sparse datastructures. So you can have a value at index 1 and at 5, nothing stops you from breaking sequence. By default an array can contain values of any type (so it’s implicitly an array of dynamic values) but this behaviour can be changed easily.

integer array a = array(1, 2, 3, 4, 5, 6, 7, 8, 9);
a[9] = 10;
a[1] = null;
a;
  => array(1, 3, 4, 5, 6, 7, 8, 9, 10);
 
a[1];
  => null;

Array a is explicitly an array of integer values, and so cannot be assigned anything else. Doing so will result in a run-time error. (Type-checking is always done in run-time).
Unsetting array indecis is done by setting the value to null, as you can see when evaluating the array, index 1 does not exist. Evaluating a non-existant index of an array produces null as well. There is no difference between non-existing indecis and values set to null.

Hash-maps

Similar to arrays in syntax to arrays, maps are variable in length, and implicitly of type ‘dynamic’. This can be changed in the same way it can for arrays.

integer hash m = hash(a: 1, b: 2, c: 3, d: 4);
x = map(function (val, key) { ... }, m);

Introducing the function map, although it could be easily constructed using lower-level operations, a high-order function provided by the language runtime.

Functions

Being first-class citizens, functions get assigned to variables, and nothing else. All functions are closures, but require a list of variables to enclose. This is to keep GC simpler and it requires the developer think about what he or she wants to enclose.

square = function(x) { return x^2; } // f(x) = x^2;
 
deriv = function(f, dx) {
  return function(x) uses (f, dx) {
    return (f(x + dx) - f(x)) / dx
  };
};
 
d_square = deriv(square, 0.001); // Approximates f'(x) = 2x;

Provided is an example demonstrating the use of this language in creating a function that produces a function approximating the derivative of another function.

Classes

Classes are a bit different from the usual OOP approach, and are halfway between normal OOP classes and prototype-based languages. Classes are first-class citizens, and can be modified in run-time. Immediately the usual implied trust that a class will remain the same during it’s entire lifetime no longer exists, although there are methods of regaining that trust.

p = class(
  private integer x,
  private integer y,
 
  public constructor function myConstructor: function() {
    this.x = 5; this.y = 5;
  },
 
  public function toString() {
    return "x: " ~ x ~ ", y: " ~ y;
  }
);
 
object = new p;
p.toString();
  => string "x: 5, y: 5"

Looks reasonable enough, create a class, make an object, execute method, the usual. Inheritance is also mostly the same as PHP and Java classes.

k = class() extends p; // Empty class extending class p.
 
z = interface(public function toList);
t = class(
  public function toList: function() {
    return list(this.x, this.y);
  }
) extends p implements z;

t is a class implementing interface z and extending class p. It’s a bit ugly with the modifiers at the bottom, but it works fine. But because classes are also first-class citizens, and not the global structures we know from Java and C++ for example, you can hide classes in context, effectively using the code-scope mechanism to implement a system similar to namespaces in that it provides some classes can be directly accessed from the entire project, and some are internal to a function, a class or any other scope-seperating block.

The role/mixin/trait feature described in the first post about this programming language has been dropped until I get my head around it well enough to see the implications of adding it.

That about sums it up for the built-in types for version 1.0 of the language specification, though I’m sure there are flaws in here that need resolving before it can be implemented properly. Critisism, as always, is appreciated, save for spelling errors and such, I am already aware my English skill needs upgrading For now, I can start working on the grammar/syntacical parser of the interpreter. After that comes a symbol table and after that I’ll be sure to post an update here.

]]> http://blog.hatsuseno.org/index.php/building-a-programming-language-part-2/feed/ 0 http://blog.hatsuseno.org/index.php/building-a-programming-language-part-1/ http://blog.hatsuseno.org/index.php/building-a-programming-language-part-1/#comments Mon, 29 Jun 2009 11:27:32 +0000 hatsuseno http://blog.hatsuseno.org/?p=48 As a follow-up from a school assignment (where I had to build a full parser for a very simple language) I decided to construct a language of my own, just for fun.

Orignally, I was a C programmer, my part-time job entails PHP, and I have a fondness for languages like Scheme, Lua and Javascript. Drawing inspiration from all these language, I have (for now) decided on a syntax like this;

Variables

int x = 42;
dynamic j = 'x';
y = 'x';

j and y are both of the same type, ‘dynamic‘, in the case of y, this is implied. Dynamic variables can contain any type of content, others are type strict.

Built-in types

int x = 42;
float j = 4.5;
string s = "hello world";
array l = (1, 2, 3, 4, 'a', 'b', 'c', 'd');
map m = [a: 1, b: 2, c: 3, d: 4];
code c = { a = 5; a++; doSomething(a); };
function f = function() {};
class c = class [];

Statements are first class citizens of the language, albeit not for the right reasons yet.

Functions

string s = "Amount ";
 
f = function(int x, function y, z) uses (s) {
  x += y(z);
  return s ~ x;
}

This here example shows literal strings as a language construct, and first class citizenship of functions. All functions are closures, but require (like in PHP) the desired variables in the closure to be explicitly summed up. String concatenation is done through the ‘~‘ operator.

Classes

surface = class [
  private x : 0,
  private int y,
 
  public constructor makeNew : function {
    this->x = 5;
    this->y = 7;
  },
]

Classes, in my opinion, are more like hash-tables than functions, as such, the syntax reflects this. ‘constructor‘ is a special keyword that indicates that the method defined is the constructor of said object. As shown, the name of this method can be anything. I haven’t quite decided on overloading and it’s implications, so I’ve left it at that. As with functions, classes are first class citizens of the language. That means they can be assigned to variables, copied and have operations performed on them. An example:

// Class with two attributes, implied public, of undefined type
a = class [ x, y ];
 
// 'b' is an empty class that extends 'a'
b = class [] extends a; 
 
c = interface [ public int x, public function getName ];
d = class []; // 'd' is an empty class
d implements c; // which now implements 'c'

Operators like ‘extends‘ and ‘implements‘ can be applied to a class during it’s entire lifetime. Instances of those classes made before the operations do not change with them.

/* Classes can be extended with anonymous classes,
 * same for interfaces.
 * Not really useful, but syntactically it's valid
 * sidenote: newlines are regarded as whitespace */
a = class [ x, y ] extends class [ 
     function getName() { return 'name'; }, int c ]
     implements interface [ function getType ];
 
x = class [ function getThing(a, b) { return a + b; } ];
 
a has x;     // 'a' now has a method 'getThing'
b extends a; // but 'b' does not.

A trick I picked up from Perl 6, class-traits. I believe this will be very beneficial in bridging traditional classes and prototype-based languages by allowing class extending without adding to the inheritance chain.

Operators

/* All common symbolic operators (+, -, &&, extends) have
 * function-like counterparts. Internally, the symbolic ops
 * are just syntactic sugar for the function-like ops */
print(2 + 2);
print(add(2, 2));
 
print(true && false);
print(and(true, false));
 
// Variable amount of parameters when it makes sense
print(and(true, true, true, true, true, true, false));
print(subtract(40, 3, 4, 6, 2, 13));

But also

/* Symbol quoting stolen from Lisp,
 * symbols a, b and c are not evaluated */
function a = _function(('a, 'b, 'c), (x, y, z), {
  return add(a, b, c) * subtract(x, y, z); });

Which, as you can see is exceedingly ugly. Neither concise nor elegant, the syntax has grown to represent what I want to avoid in general. This either means I will need to get a clue as to resolving this syntactically, or I will have to remove the operators-as-functions feature from the language, but I haven’t decided. The entire concept of code as a first class citizen of the language (like in Smalltalk) was created out of necessity to make this work.

Conclusion (for now, anyway)

Although I have started with a parser for the basic language, stuff like the operators-as-functions and first class citizenship of codeblocks need to be resolved to add to the whole of the language instead of looking like I just clobbered it all together. Also, I still need to name it… ‘project cwx’ sounds increasingly cheezy.

]]> http://blog.hatsuseno.org/index.php/building-a-programming-language-part-1/feed/ 0 http://blog.hatsuseno.org/index.php/php-and-xml-manipulation/ http://blog.hatsuseno.org/index.php/php-and-xml-manipulation/#comments Thu, 28 May 2009 23:19:30 +0000 hatsuseno http://blog.hatsuseno.org/?p=32 Recently I’ve been fiddeling around with an old project of mine, and decided to extend it into a full-blown framework, just for kicks. Born from GDO, a ORM layer for PHP, it’s supposed to serve no real purpose but personal gratification.

Anyway, parallel with the framework I am building a website to showcase it. I also decided that in this specific case all the HTML documents should be built from either PHPs DOM or SimpleXML libraries, I personally dislike having HTML strewn across the project, so I’ve seperated structures like forms and tables into classes and the idea is that I can simply append them to the current page at a specific node and it’ll render the HTML when everything is added.

The problem

Here in lies the problem, PHPs XML libraries (at least the ones that I tried) require you to drag the top node to which you want to append whatever around in all helper classes and such, allow me to demonstrate.

<?php
$document = new DOMDocument();
 
$table = new DOMElement('table');
 
$row = new DOMElement('tr');
$row->appendChild(new DOMElement('td', 'test element'));
 
$table->appendChild($row);
$document->appendChild($table);

One would expect this all to work out fine and produce the following XML document;

<?xml version='1.0' encoding='utf-8' ?>
<table><tr><td>test element</td></tr></table>

In fact, it won’t. It won’t even run. PHP bails out during run time with the error “DOMException: No Modification Allowed Error“. This is because every DOMElement object not created with DOMDocument::createElement() is by default read-only, as you can read in the description of DOMDocument::__construct().

Besides begging the PHP developers to make every DOMElement not read-only on construction, the obvious solution would be to add the DOMElement in question to the DOMDocument and continue building from there, but that’s the problem. It seems counter-intuitive to pass the DOMElement or DOMDocument that you want to append your seperate structure to to every constructor of every XML producing class (or via normal methods of course).

I’ve taken the literal example of PHPs DOM classes, but I’ve tested SimpleXML as well, which provides even less flexibilty regarding node reuse and other fancy stuff one could want to do with XML documents.

Workarounds

There are several solutions to this problem, to name a few:

• Create a singleton class that holds one instance of DOMDocument, and use a wrapper method somewhere to either get a valid DOMElement
• Have all classes that produce XML nodes use a fake DOMDocument, and then use DOMDocument::importNode() at the return-point to import the nodes (very inefficient, especially when using a deep-copy).
• Have the classes that produce XML nodes return nested arrays containing all name -> content relations of the nodes and run through them when it’s sensible to do so.

I’ll leave it as an exercise to the reader to pick out the best method. I haven’t found a solution yet, but I hope to contact the DOM XML module maintainer and discuss this with him/her.

]]> http://blog.hatsuseno.org/index.php/php-and-xml-manipulation/feed/ 1