d(a)emonkeeper's purgatory

Until recently Debian kFreeBSD, the FreeBSD powered Debian branch, had some serious issues when dealing with ZFS. The situation improved dramatically since then and ZFS is quite usable in recent snapshots. That’s enough reason to work on an iSCSI target for kFreeBSD which combines the powers of ZFS with an industry standard for block level data access. Read on, to learn how to get a Debian kFreeBSD iSCSI target share for ZFS volumes.

Why iSCSI rocks

Before I start, a few words about terminology which is quite unique: Despite its name, iSCSI is not (only) a host bus protocol but a specification to carry (rather) regular SCSI frames through a TCP/IP network. This allows peers to share disks through an IP network, or access them respectively. iSCSI is an industry standard for storage area networks or any other application where you would like to get raw disk disk access, usually to a central storage server. In iSCSI terminology the target is the server side component which provides actual data store, whereas the initiatior is the client component accessing volumes.

There are good reasons to go with iSCSI instead of network shares. Basically there are two ways of sharing data through a network: block-wise access or logically on file level. If you used to know Samba (CIFS) or NFS you know about file level access already. Both of those alternatives are (sort of) network file systems. A high level protocol implementing either of those alternatives acts as proxy then and redirects the usual kernel syscalls through the network. If, for example, one wants to access a file, the network proxy translates this to a remote procedure call like this: “open file /a/b/c and return it to me“. Obviously that’s inefficient, since most of the time clients using files don’t actually need the entire file, but they only operate on certain fragments of it at time. This is even worse if you need to deal with large binary files like data bases or disk images.This is a problem which can’t be addressed on such a high level where you deal with files.

This is where iSCSI does things different. First, iSCSI is a low level technology: its not a file system, but a tool which operates on blocks. It does not address files (or data in general) by their logical name, but by blocks. iSCSI has no higher understanding of file systems, files or logical addressing. The corresponding data retrieval from my example above would be something like “return my 42 bytes starting with offset 0xDEADBEEF“. This only returns raw data as required by higher level tools running on the client side. That’s great for speed and reduces load from both, the remote physical disk storing the data and the network. In fact, you can come close to wire capacity when using iSCSI which is pretty hard to reach with remote file level access.

Its the client’s responsibility to abstract the raw block access to something actually useful on higher levels. Therefore, most iSCSI shares are just formatted with a regular file system like ext3 on it. From that on, the volume can be treatened like every other local block device. However please note, sometimes that is not not what you actually want. Local file systems mostly don’t do any decentralized locking or allowing concurrent access. This has to be realized by other means, iSCSI is not designed to address this. If one wants to access files from more than one node concurrently a cluster file system is needed.

Prework and Purpose

A recent Wheezy snapshot is needed to have ZFS working on installation time or to boot from a ZFS root file system. However ZFS is not a requirement to run an iSCSI target after all, any decent kfreeBSD works. An iSCSI volume can be shared from any disk, volume or disk image which can be accessed on file or block level. If a fresh install is needed, it is suggested to use a recent weekly snapshot (or here for old x86 systems). This is to get a decent debian-installer which works better with ZFS file systems, as the Squeeze installer has some serious bugs with ZFS.

As explained, iSCSI is not limited to actual disks. One can actually export almost anything as iSCSI data store. This includes pure virtual disk volumes which can be managed through a volume manager. On Linux one would perhaps use the Logical Volume Manager (“LVM”) for this, on kfreeBSD however, the power of ZFS comes handy. On Solaris iSCSI support is even directly supported on the file system layer for ZFS. Its enough to mark a ZFS volume as shared to get iSCSI access to it, when the required service was started before.  This nifty way is not (yet) possible on kfreeBSD (or upstream on pure FreeBSD).

Finally please realize Debian kFreeBSD has no support for iSCSI targets for the time being. Therefore I ported istgt from FreeBSD to Debian kfreeBSD. That is a user space daemon with great support for most iSCSI features including multipath, failover and of course CHAP authentication. As of today, FreeBSD supports two modern iSCSI targets: The other alternative is iet which is a kernel space driver for Linux, later ported to FreeBSD. The kernel driver performs better, but I wasn’t particularly happy to port such an invasive out-of-tree patch back to Debian. Moreover the patch requires much more effort to get it working on kFreeBSD. Download links for the istgt package can be found below. I prepared binary packages for kfreebsd-amd64, but source packages are available as well.

To get things started a working Debian kfreeBSD setup is needed as I explained. I was suggesting to use ZFS as volume storage as explained, but that’s definitively no requirement. One can even use plan disk images (“dd if=/dev/zero of=/images/volume bs=1M size=$yoursize_in_mb“), but that way one wouldn’t be stuck on kfreeBSD, since ZFS is where actual fun comes from.

First, one needs to install the istgt package. As I didn’t setup a full APT repository (let me know if there are requests for this), packages have to be installed with dpkg directly. If required, call “apt-get -f install” afterwards to satisfy dependencies. After installing istgt, a configuration is needed. That’s not trivial task, but nothing to be worried about either. To get started, the shipped example configuration is fine and comes with sensible defaults to start with.

root@kfreebsd:~# dpkg -i /home/arno/istgt/istgt_0.4~20110529-1_kfreebsd-amd64.deb
Selecting previously deselected package istgt.
(Reading database ... 32551 files and directories currently installed.)
Unpacking istgt (from .../istgt_0.4~20110529-1_kfreebsd-amd64.deb) ...
Setting up istgt (0.4~20110529-1) ...
Processing triggers for man-db ...
root@kfreebsd:~# cp /etc/istgt/istgt.conf.sample /etc/istgt/istgt.conf
root@kfreebsd:~# cp /etc/istgt/auth.conf.sample /etc/istgt/auth.conf

IQN, Portals, Initiators, Units, … Oh My!

As a first step, authentication can be ignored. This eases life a bit if one used to deploy such a setup in a secure local network, keeping in mind anyone can access shared volumes then. Striaght below is a rather minimalistic, but full working configuration. This is /etc/istgt/istgt.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

[Global]
Comment "Global section"
# node name (not include optional part)
NodeBase "iqn.2011-07.org.debian.kfreebsd"
 
# files
PidFile /var/run/istgt.pid
AuthFile /etc/istgt/auth.conf
LogFacility "daemon"
 
# socket I/O timeout sec. (polling is infinity)
Timeout 30
NopInInterval 20
 
# Let everyone discover
DiscoveryAuthMethod None
 
MaxSessions 16
MaxConnections 4
 
[UnitControl]
  Comment "Internal Logical Unit Controller"
  #AuthMethod CHAP
  #AuthGroup AuthGroupControl
  Portal UC1 127.0.0.1:3261
  Netmask 127.0.0.1
 
[PortalGroup1]
  Comment "Portal 1"
  Portal DA1 192.168.2.21:3260
 
[InitiatorGroup1]
  Comment "Initiator Group1"
  InitiatorName "ALL"
  Netmask 192.168.2.0/24
 
# TargetName, Mapping, UnitType, LUN0 are minimum required
[LogicalUnit1]
  Comment "Hard Disk"
  TargetName disk1
 
  Mapping PortalGroup1 InitiatorGroup1
  AuthMethod None
  #AuthMethod CHAP
  #AuthGroup AuthGroupDisk1
  UnitType Disk
  LUN0 Storage /dev/zvol/kfreebsd-ad0s1/target Auto

First thing which should be changed is the node name on line four. In iSCSI, targets are identified by a qualified node name. There can pretty much be chosen whatever one wants here. However the naming scheme is standardized as iSCSI Qualified Name (IQN) one should follow. This naming scheme is documented in RFC 3721. In short, a valid name starts with “iqn” followed by entities separated by dots.  The first entity is a date code, specifying the registration date of the domain following. Next entity is a naming authority consisting of a reversed full qualified domain name. This results in a string like “iqn.YYYY-MM.com.example“.

Remaining parts of the “Global” section are pretty obvious so I won’t go too much into detail here. Moreover refer to the sample configuration for all available parameters. Most are related to network tuning and not relevant to get the big picture.

The configuration file consists of INI style sections and they deal with parts of the setup. The remaining sections configure available portals, initators, controller and actual volumes. This sounds more confusing than it should, as that’s only somewhat of an odd naming for obvious components:

• A portal is a TCP/IP tuple which a target makes available. Large scale setups may have more than one of such portals. For most setups it ought to be enough to configure a single IP address altogether with a corresponding TCP port, where the iSCSI target is listening on. I did so in line 30.
• Usually iSCSI volumes are critical resources. Most initiators (remember, those are the clients) would fail badly, when a volume isn’t accessible anymore. This implies one can’t just restart a iSCSI target daemon every now and then just because one wants to change some volumes. To come over this problem, istgt provides a control channel which allows to change volumes at run time. This channel is configured in lines 21 to 26. Note, any authentication to access the control channel was disabled (see line 23 and 24). This means, anyone connecting from localhost is able to manage volumes.
• Finally an initiator group is needed. This is basically a list of clients, which are allowed to connect to a given volume. For most setups that is perhaps just a subnet specification, where clients are expected to connect from.

In line 37 and following lines, everything is put together for a single volume share. Such volume declarations can be repeated to provide several volumes through the same portal. First, a volume is identified by an arbitrary (per-portal unique) string. iSCSI will use this name to address a particular volume shared by a portal by that name. This means, the node name specified in line 4 and and the disk name from line 40 result in a volume being shared as “iqn.2011-07.org.debian.kfreebsd:disk1” in my example. In line 42 the portal and initiator groups previously defined are being linked to the volume share. Once more I disabled any authentication, refer to line 43 and successive regarding this.

Finally I configured the physical disk backend for the share. This is the last line of the shown configuration above. That is where data is stored. I used to use a ZFS volume here as said, therefore the only thing needed to specify is the device path. This used to be “/dev/zvol/kfreebsd-ad0s1/target” for my example, where this paths refers to a volume called “target” in a ZFS pool called “kfreebsd-ad0s1“.

You  can create such a target like this:

root@kfreebsd:~# zpool list
NAME             SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
kfreebsd-ad0s1  11.1G  1004M  10.1G     8%  ONLINE  -
root@kfreebsd:~# zfs create -V 5gb kfreebsd-ad0s1/target
root@kfreebsd:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
kfreebsd-ad0s1         5.98G  4.97G  1003M  /
kfreebsd-ad0s1/target     5G  9.97G    16K  -
root@kfreebsd:~# ls -l /dev/zvol/kfreebsd-ad0s1/target
crw-r----- 1 root root 0, 82  6. Jul 19:19 /dev/zvol/kfreebsd-ad0s1/target

So far about configuration, istgt can now be started. I suggest to start istgt with the “-D” for the first time, to prevent it from detaching from the terminal. This helps to catch debugging output (even more can be retrieved with “-t all“). Once the setup seems to work, it is suggested to start istgt from the init.d script, the package has installed (“/etc/istgt/istgt.conf“).

root@kfreebsd:~# istgt -D
istgt version 0.4 (20110529)
traditional mode
LU1 HDD UNIT
LU1: LUN0 file=/dev/zvol/kfreebsd-ad0s1/target, size=5368709120
LU1: LUN0 10485760 blocks, 512 bytes/block
LU1: LUN0 5.0GB storage for iqn.2011-07.org.debian.kfreebsd:disk1
LU1: LUN0 command queuing disabled
sock=9, addr=192.168.2.21, peer=192.168.2.31
Login(discovery) from iqn.1993-08.org.debian:01:9d3f85687a4 (192.168.2.31) on (192.168.2.21:3260,1), ISID=23d000000, TSIH=1, CID=0, HeaderDigest=off, DataDigest=off
istgt_iscsi.c:5165:worker: ***ERROR*** conn->state = 2
Connections(tsih 1): 0
sock=9, addr=192.168.2.21, peer=192.168.2.31
drop old connections iqn.2011-07.org.debian.kfreebsd:disk1 by iqn.1993-08.org.debian:01:9d3f85687a4,i,0x00023d010000
Login from iqn.1993-08.org.debian:01:9d3f85687a4 (192.168.2.31) on iqn.2011-07.org.debian.kfreebsd:disk1 LU1 (192.168.2.21:3260,1), ISID=23d010000, TSIH=1, CID=0, HeaderDigest=off, DataDigest=off

iSCSI initiator setup

The output above already shows a successful connection attempt from an initiatior. In this section I will show how to connect to the configured target by using a Linux initiator. Debian comes with everything which is needed to setup an iSCSI initiator. You will need two packages, “iscsitarget-dkms” for the kernel module, and “open-iscsi” for the user space controlling the module.  It does not need any configuration as it comes with sensible defaults, but this can certainly be tweaked. Please note, iSCSI connections are meant to be static; this means, as soon as a session is established, open-iscsi tries to remember it and to re-establish this connection for consecutive boots and daemon (re-)starts. This session configuration is being stored in /etc/iscsi/nodes/. Once set up, open-iscsi needs to scan for portals and their offered volumes. There is a “discover” call, which can be issued on a portal to retrieve offered disks. The iscsiadm command comes handy here and fills /etc/iscsi/nodes/ with handy defaults for found portals:

root@test:/home/arno# /etc/init.d/open-iscsi start
Starting iSCSI initiator service: iscsid.
Setting up iSCSI targets:
iscsiadm: No records found!
.
Mounting network filesystems:.
root@test:/home/arno# iscsiadm  -m discovery -t st -p 192.168.2.21
192.168.2.21:3260,1 iqn.2011-07.org.debian.kfreebsd:disk1

Since I didn’t configure any authentication, a simple discovery call ought to be enough to get access to all disks, a portal provides (or the availability of the disk listing at all). Therefore everything which needs to be done, is to let iscsiadm scan for available portals and the images they provide by using a single command line argument. To identify a portal, only the IP address is needed. The port can be omitted, unless a non standard listening port was configured on the target side before. By default, an iSCSI target is listening on port 3260.

The next and final step is to “login” to the discovered portal, which actually starts accessing a shared volume. Immediately after doing so, the volume becomes available, as dmesg should indicate.

root@test:/home/arno# iscsiadm  -m node  --targetname "iqn.2011-07.org.debian.kfreebsd:disk1" --portal "192.168.2.21" --login
Logging in to [iface: default, target: iqn.2011-07.org.debian.kfreebsd:disk1, portal: 192.168.2.21,3260]
Login to [iface: default, target: iqn.2011-07.org.debian.kfreebsd:disk1, portal: 192.168.2.21,3260]: successful
root@test:/home/arno# dmesg | tail
[10035.625340] iscsi: registered transport (iser)
[10066.142376] scsi3 : iSCSI Initiator over TCP/IP
[10067.474954] scsi 3:0:0:0: Direct-Access     FreeBSD  iSCSI DISK       0001 PQ: 0 ANSI: 5
[10067.478021] sd 3:0:0:0: Attached scsi generic sg2 type 0
[10067.485879] sd 3:0:0:0: [sdb] 10485760 512-byte logical blocks: (5.36 GB/5.00 GiB)
[10067.486992] sd 3:0:0:0: [sdb] Write Protect is off
[10067.486997] sd 3:0:0:0: [sdb] Mode Sense: 83 00 00 08
[10067.488339] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[10067.504840]  sdb: unknown partition table
[10067.522947] sd 3:0:0:0: [sdb] Attached SCSI disk

Once one went beyond this point, the access to a remote iSCSI share is absolutely transparent and can be threatened like every other block device. Everything which is possible with a local block device, can be realized with a iSCSI volume as well. For example it is straightforward to create a partition table and partitions on it which can be formatted and mounted like a local file system.

root@test:/home/arno# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xbf785d7d.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
 
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
 
Command (m for help): p
 
Disk /dev/sdb: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 10292 * 512 = 5269504 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbf785d7d
 
   Device Boot      Start         End      Blocks   Id  System
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1018, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1018, default 1018):
Using default value 1018
 
Command (m for help): wq
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.
root@test:/home/arno# mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1309649 blocks
65482 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736
 
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@test:/home/arno# mount /dev/sdb1 /mnt/
root@test:/home/arno# touch /mnt/hello_iscsi_world

That’s it. This shiny new disk device can be used as if it where local as said. Beware: iSCSI is a low level protocol as discussed before. It does not care at all what is being installed on the disk or how the stuff on it is being accessed. This means iSCSI certainly supports concurrent parallel access to the same volume, but this does not imply the file system installed on it supports that as well. Such a local file system cannot be used on more than one initiator (client) at the same time! This is because the ext file system is not prepared for this. A cluster file system like OCFS2 or GFS is required to achieve that, otherwise an immediate risk of data loss or damaging the file system will occur.

If a volume is not needed anymore or an initiator should disconnect for other reasons, the “logout” command can be used similarly:

root@test:/home/arno# iscsiadm  -m node  --targetname "iqn.2011-07.org.debian.kfreebsd:disk1" --portal "192.168.2.21" --logout

Authentication

So far istgt was not configured to secure data access at all. This perhaps a bit of an artificial setup, as authentication is definitively something wanted in real world setups. iSCSI allows mutual authentication, this means either node can authenticate the other side. I won’t cover target authentication here, but I will explain how to authenticate the initiator on the target side.

The target can ask the initiator to authenticate for the discovery phase already, but this makes life more complicated than necessary for little benefit unless you consider exported names confidental. Hence, unless running the iSCSI target on a public facing interface, there are few reasons to do so, except for paranoid reasons. There is nothing wrong to be paranoid though. Generally it is enough to authenticate login requests to specific disks. To enable authentication, the following change in istgt.conf within the “LogicalUnit1” is required: :

AuthMethod CHAP
AuthGroup AuthGroupDisk1

This references an authentication group named “AuthGroupDisk1” in /etc/istgt/auth.conf which should look like this:

[AuthGroupDisk1]
Comment "Auth Group Disks"
Auth "test" "tset"

When trying to access such a protected volume, this is supposed to fail with a login failure upon login in iscsiadm. Credentials can be passed to the target by iscsiadm by configuring them. This can be done globally, by specifying credentials in /etc/iscsi/iscsid.conf, or per session. The easiest way to achieve this, is to use iscsiadm again (or, settings can alternatively be changed manually in /etc/iscsi/nodes/):

iscsiadm  -m node  --targetname "iqn.2011-07.org.debian.kfreebsd:disk1" --portal "192.168.2.21" -o update -n node.session.auth.authmethod -v CHAP
iscsiadm  -m node  --targetname "iqn.2011-07.org.debian.kfreebsd:disk1" --portal "192.168.2.21" -o update -n node.session.auth.username -v test
iscsiadm  -m node  --targetname "iqn.2011-07.org.debian.kfreebsd:disk1" --portal "192.168.2.21" -o update -n node.session.auth.password -v tset

By the way, volumes can be restored upon startup automatically by changing node.startup to “automatic” similarly to passing login credentials.

Controlling the iSCSI target instance

As I explained before there are good reasons not to restart the entire target daemon just because a volume should be added or removed respectively. This is what istgtcontrol is for. That’s an iSCSI target control client which accesses the command channel interface which was configuredd before (in the “UnitControl” section of the istgt configuration file).  Now, only istgtcontrol needs to learn about this configuration there. The easiest way to achieve this is to configure it similarly to the istgt configuration file. That said, it is enough to put the following configuration to /etc/istgt/istgtcontrol.conf:

[Global]
  Comment "sample configuration"
  Timeout 60
 
  # authentication information
  AuthMethod Auto
  #AuthMethod CHAP
  #Auth "testuser" "secret"
 
  Host localhost
  Port 3261
 
  Flags "rw"
  Size "auto"

The file is rather self-explanatory. Note I left out authentication once again. This can be applied in a way very similar to the volume authentication shown before. Once the file was changed, volumes can be managed from the command line:

root@kfreebsd:/home/arno/istgt# istgtcontrol list
iqn.2011-07.org.debian.kfreebsd:disk1
DONE LIST command

Unfortunately the interface is still a bit limited for the time being.

Downloads

  istgt_0.4~20110529-1_kfreebsd-amd64.deb (155.1 KiB, 406 hits)

  istgt_0.4~20110529.orig.tar.gz (219.3 KiB, 418 hits)

  istgt_0.4~20110529-1.debian.tar.gz (4.6 KiB, 374 hits)

  istgt_0.4~20110529-1.dsc (829 bytes, 355 hits)

If you are interested in the source package, you can also do:

dget http://daemonkeeper.net/repository/istgt_0.4~20110529-1.dsc