d(a)emonkeeper's purgatory

Recently Debian 6.0 “Squeeze” has been released. Among a lot of improvements and enhancements, the community introduced a completely new distribution as technology preview namely GNU/kFreeBSD. There you get pretty much what the name indicates: a BSD kernel (from the FreeBSD project) altogether with a GNU user land as known from Debian.

About kFreeBSD

Within the Linux ecosystem Debian is best known for its package system, a large set of supported software and the community’s strict understanding how a stable distribution should behave. This is, why I prefer Debian over other Linux distributions. Now, as Squeeze came out, things are a bit different again. In fact, kFreeBSD is not even Linux, but a BSD kernel which is a modern heritage of a long history of UNIX (-like) operating systems. Among those BSD based operating systems the best known flavors are NetBSD, OpenBSD and FreeBSD. Debian people took the core of the FreeBSD operating system – the kernel – (that is where the name kFreeBSD is derived from) and plugged it into the existing Debian architecture. This includes most notably the GNU user land (in contrast to original FreeBSD which uses BSD user land) but also all this fancy stuff like Debian packages, improvements and behavior (think of /etc/network/interfaces for example).

People having experience with FreeBSD might wonder whether Debian kFreeBSD supports BSD ports or native Linux binaries. Upstream FreeBSD kernel has a compatibility layer to support Linux syscalls, yet the default mode for kFreeBSD is the native BSD interface. One can run native (Free-) BSD binaries from the ports package system, as well as some Linux binaries. Most likely there is no reason to do so though, since apt-get is available of course.

To date kFreeBSD is still a technology preview as mentioned above. That means it runs reasonably stable and most packages have been ported, but some things are broken or not yet usable. However there are things available, that aren’t for Linux at all. This is, why kFreeBSD is a great enhancement. This includes, but is not limited to:

• ZFS – Yes really! kFreeBSD brings native ZFS support to people used to Debian’s native look and feel
• Jails – BSD jails are enhanced chroot environments, bringing additional security and support for enslaved IP addresses (a similar approach exists for Linux as out of tree patch, called Linux-Vserver)
• pf – The OpenBSD packet filter, a very good firewall (most people would agree more intuitive and usable compared to the Linux netfilter/iptables)

Installation

Unfortunately the term “technology preview” also implicates some bugs, the most annoying affects the Debian Installer (“d-i”) which effectively prevents a reasonable partitioning for experiments with ZFS (see partitioning problems in the d-i errata, even though some people claim one could.) I will present a workaround to the installation issues. People having installed Debian previously won’t see any obvious changes. Indeed the Debian Installer for kFreeBSD is exactly the same as for Linux.

Preparation

For my experiments I used VirtualBox OSE. If one is interested to install kFreeBSD on dedicated hardware those people may want to skip this step, otherwise feel free to do it likewise. First I downloaded the kFreeBSD netinst CD image:

[arno@snowball:~/fw]$ wget http://cdimage.debian.org/debian-cd/6.0.0/kfreebsd-amd64/iso-cd/debian-6.0.0-kfreebsd-amd64-netinst.iso
--2011-02-15 16:58:58--  http://cdimage.debian.org/debian-cd/6.0.0/kfreebsd-amd64/iso-cd/debian-6.0.0-kfreebsd-amd64-netinst.iso
...
2011-02-15 16:59:56 (1.72 MB/s) - “debian-6.0.0-kfreebsd-amd64-netinst.iso” saved [102787072/102787072]

Next I created two logical volumes on my LVM volume group. Please note this is intentional to get a working ZFS setup. I failed to get a Debian kFreeBSD including ZFS straight out of the installation. To workaround this issue, I installed a minimum Debian system using a UFS file system and added ZFS manually afterwards. To achieve this I created to volumes. When using LVM this is as easy as “lvcreate -L 2G -n debian-kfreebsd-disk1 lvm-vg” or “lvcreate -L 10G -n debian-kfreebsd-disk2 lvm-vg” respectively. Moreover VirtualBox lacks GUI support for raw disk images, so I need to create them manually:

[arno@snowball:~/VirtualBox]$ VBoxManage internalcommands createrawvmdk -filename "/home/arno/VirtualBox/HardDisks/Debian kFreeBSD Disk1.vmdk" -rawdisk /dev/lvm-vg/debian-kfreebsd-disk1  
RAW host disk access VMDK file /home/arno/VirtualBox/HardDisks/Debian kFreeBSD Disk1.vmdk created successfully.
[arno@snowball:~/VirtualBox]$ VBoxManage internalcommands createrawvmdk -filename "/home/arno/VirtualBox/HardDisks/Debian kFreeBSD Disk2.vmdk" -rawdisk /dev/lvm-vg/debian-kfreebsd-disk2 
RAW host disk access VMDK file /home/arno/VirtualBox/HardDisks/Debian kFreeBSD Disk2.vmdk created successfully.

Finally I created a new virtual machine. During the initial virtual machine configuration I chose to use an existing disk and picked debian-kfreebsd-disk1 for this purpose. Now I was finally ready to install kFreeBSD. The VirtualBox configuration itself is straightforward, although kFreeBSD does not support every hardware configuration. The following hardware emulation is working fine:

• Chipset: PIIX3 with IO APIC
• IDE Controller: PIIX4 with IO Cache

The installation itself is not different than every other Debian installation. There seem to exist a lot of problems with the partitioner though, so make sure you choose “guided – use entire disk” when being prompted about disk configuration and go on with “Separate /home partition” (all due to bug #593733 and some more). Yes, this is odd, but it seems the only way to get both working, disk partitioning and grub-pc (bug #612838).

The remaining parts of the installation are not notable and seem to work without problems. You should now be able to boot into your kFreeBSD system. Moreover you can attach the second disk now which is going to be our ZFS volume.

First steps

Make sure you stopped VirtualBox and installed the additional LVM volume. Afterwards boot and your new system and log in. It should feel very similar to what you might know from GNU/Linux:

root@kfreebsd:~# uname -a
GNU/kFreeBSD kfreebsd 8.1-1-amd64 #0 Tue Jan  4 15:07:39 CET 2011 x86_64 amd64 Intel(R) Core(TM)2 Duo CPU     T8300  @ 2.40GHz GNU/kFreeBSD
root@kfreebsd:~# dpkg -l | grep "kernel"
ii  kfreebsd-image-8-amd64        8.1+dfsg-7.1                kernel of FreeBSD 8 image
ii  kfreebsd-image-8.1-1-amd64    8.1+dfsg-7.1                kernel of FreeBSD 8.1 image

Some other things might look a bit different to what you are used to:

root@kfreebsd:~# cat /etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
# The loopback network interface
auto lo0
iface lo0 inet loopback
 
# The primary network interface
auto em0
iface em0 inet dhcp
root@kfreebsd:~# ifconfig em0
em0: flags=8843<up ,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<rxcsum ,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 8:0:27:9f:1:c7
        inet6 fe80::a00:27ff:fe9f:1c7%em0 prefixlen 64 scopeid 0x1 
        inet 192.168.2.21 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=3<performnud ,ACCEPT_RTADV>
        media: Ethernet autoselect (1000baseT <full -duplex>)
        status: active

Generally speaking you should now have too much problems to feel comfortable if you used Debian GNU/Linux before.

Setup ZFS

Even though the installer fails to install directly to ZFS (at least for me!), you can easily experiment with it after booting into the system. Remember, I attached an additional hard disk before, I connected before I booted into the system. I’m now going to creat a ZFS volume on it and use it immediately. Besides the device naming, which is a bit different, the task is not really different compared to Linux:

root@kfreebsd:~# fdisk /dev/ad1 
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x622ee77b.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
 
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
Command (m for help): p
 
Disk /dev/ad1: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x622ee77b
 
    Device Boot      Start         End      Blocks   Id  System
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305): 
Using default value 1305
 
Command (m for help): t  
Selected partition 1
Hex code (type L to list codes): a5
Changed system type of partition 1 to a5 (FreeBSD)
 
Command (m for help): wq
The partition table has been altered!
 
Syncing disks.

Now one should have the newly created partition available on /dev/ads1. I can continue to setup ZFS now. First of all I create a new ZFS pool called zfs_pool on this partition.

root@kfreebsd:~# apt-get install zfsutils
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  zfsutils
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 114 kB of archives.
After this operation, 279 kB of additional disk space will be used.
Get:1 http://ftp.de.debian.org/debian/ squeeze/main zfsutils kfreebsd-amd64 8.1-4 [114 kB]
Fetched 114 kB in 0s (371 kB/s) 
Selecting previously deselected package zfsutils.
(Reading database ... 10108 files and directories currently installed.)
Unpacking zfsutils (from .../zfsutils_8.1-4_kfreebsd-amd64.deb) ...
Processing triggers for man-db ...
Setting up zfsutils (8.1-4) ...
Starting ZFS subsystem... zvol swap filesystems.
root@kfreebsd:~# zpool create zfs_pool /dev/ad1s1 
root@kfreebsd:~# zpool list
NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
zfs_pool  9.94G    72K  9.94G     0%  ONLINE  -

That was it. I can access the ZFS pool through its mount point /zfs_pool now. Let’s have some fun with ZFS: First let’s try the snapshot ability:

root@kfreebsd:/zfs_pool# echo "let's have fun with snapshots" > /zfs_pool/fun
root@kfreebsd:/zfs_pool# cat /zfs_pool/fun
let's have fun with snapshots
root@kfreebsd:/zfs_pool# zfs snapshot zfs_pool@snapshot
root@kfreebsd:/zfs_pool# zfs list -t all
NAME                USED  AVAIL  REFER  MOUNTPOINT
zfs_pool            183K  9.78G  19.5K  /zfs_pool
zfs_pool@snapshot      0      -  19.5K  -

A snapshot is a copy-on-write volume of an existing volume. That is, it will consume no space until changes on the original volume are being made. Snapshots are atomic, making it ideal to backup data or to roll back a system to a given pre-defined state. Let’s see:

root@kfreebsd:/zfs_pool# echo "snapshots ain't fun" > /zfs_pool/fun 
root@kfreebsd:/zfs_pool# cat /zfs_pool/fun
snapshots ain't fun

After having modified the original volume one can still access previous snapshots easily (please note, access to snapshot volumes is read only):

root@kfreebsd:/zfs_pool# zfs set snapdir=visible zfs_pool
root@kfreebsd:/zfs_pool# cat /zfs_pool/.zfs/snapshot/snapshot/fun 
let's have fun with snapshots

If I don’t like the changes I did on the volume, I can also roll back to any given snapshot immediately:

root@kfreebsd:/zfs_pool# cat /zfs_pool/fun 
snapshots ain't fun
root@kfreebsd:/zfs_pool# zfs rollback zfs_pool@snapshot
root@kfreebsd:/zfs_pool# cat /zfs_pool/fun 
let's have fun with snapshots

One of the greatest features of ZFS is the ability to make reliable copies on the fly of existing file systems. This can also be used to backup data:

root@kfreebsd:/zfs_pool# zfs send zfs_pool@snapshot | zfs receive zfs_pool/testbackup
root@kfreebsd:/zfs_pool# zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zfs_pool              242K  9.78G  19.5K  /zfs_pool
zfs_pool/testbackup  19.5K  9.78G  19.5K  /zfs_pool/testbackup
root@kfreebsd:/zfs_pool# ls /zfs_pool/testbackup
fun
root@kfreebsd:/zfs_pool# cat /zfs_pool/testbackup
root@kfreebsd:/zfs_pool# cat /zfs_pool/testbackup/fun 
let's have fun with snapshots

More to come.

pf

Another great thing for kFreeBSD is the ability to run OpenBSD’s packet filter pf. Unlike Linux’ netfilter the OpenBSD pf also includes support for QoS, CARP and the ability to be monitored by tcpdump. Debian kFreeBSD includes pf support out of the box. As simple example I will show a simple stateful rule allowing my local machine only to access the box.

text

root@kfreebsd:~# cat fw 
pass in quick on em0 proto tcp from 192.168.2.9 to em0 port ssh
block return all
root@kfreebsd:~# modprobe pf
root@kfreebsd:~# pfctl -f fw
root@kfreebsd:~# pfctl -sr
pass in quick on em0 inet proto tcp from 192.168.2.9 to 192.168.2.21 port = ssh flags S/SA keep state
block return all

Summary

I really like to combine the advantages of both worlds into a single system. With Debian I can now run the powerful BSD kernel as well as the traditional Linux kernel. Both have their advantages, but I’m not forced to make a decision anymore as I don’t have to learn BSD and its user land since I can use the environment I’m used to altogether with fancy stuff like pf and ZFS.

Without doubt kFreeBSD needs some more bug hunting and work until being ready for productive use. I’m pleasantly surprised from kFreeBSD so far, really.