d(a)emonkeeper's purgatory

People interested to learn how to use enterprise hardware routers usually won’t have the possibility to access those devices in an experimental lab environment because its price. Hence, to get pratice you need some workarounds: This is where “Olive” enters the game. Olive the pure software skeleton, forming the essential software component of a JUNOS router. You might know, the JUNOS platform is based on ordinary PC hardware (e.g. in contrast to IOS which runs on a MIPS like architecture). It is not entirely straightforward to deploy such a machine, this is why I am going to present you a step by step tutorial how to install and configure a virtual machine image based on the VirtualBox platform. However, chances are, other hypervisors will work as well.

Prerequisites

In case you were not aware: JUNOS runs on top of FreeBSD. The big picture to get a running JUNOS instance on a PC platform is to prepare an “Olive“, which is basically a JUNOS release, being bootstrapped over an unmodified, ordinary FreeBSD. This is no special hack, or a virtual machine concept for JUNOS; it is just a JUNOS running on pure PC hardware with no forwarding hardware.

Please note, hardware support for JUNOS is limited, as official “Routing Engines” (which is in Juniper’s terminology the x86 base components, a router consists of, e.g. CPU, chip set, memory) are rather standardized. That means, you need to match as close as possible to the usual RE hardware. I think it is not necessary to say, you won’t get any support to deploy an Olive on generic hardware from Juniper. Indeed “Olive” does not even exist, as of Juniper’s point of view. So you won’t find any official documentation related to it, and don’t expect the JTAC support to answer your questions related to an Olive. Obviously it is silently supported by Juniper though, since it could be much harder to get an Olive up and running. So the official creed is “don’t ask, don’t tell“.

There are some prerequisites you need to fulfill some requirements to get a running JUNOS image. In particular you need at least:

• A physical x86 PC or a virtual machine monitor capable to run x86 hardware. JUNOS is known to work on Qemu (Tutorial), VMWare (Tutorial) and VirtualBox (see below). Xen might work in HVM mode as well, although I have no evidence someone ever tried.
• A supported network interface card. Note, JUNOS only supports a limited number of actual network interface cards. The most comprehensive list of supported devices I know is this list. A popular device type, supported by both JUNOS and VirtualBox is the Intel PRO/1000 MT Desktop adapter (82540EM). NICs from Intel seem to be generally supported, although some adapters found on Intel’s Atom platforms are known not to work.
• FreeBSD. The particular version depends on the JUNOS version you want to bootstrap. The “Release Notes” of your JUNOS version might indicate the underlying FreeBSD version. For 9.X series releases, FreeBSD 4.11 seems to work just fine, I had no success with a more recent FreeBSD. Get it from the FreeBSD archive here. The “Mini-Inst” ISO works just fine.
• Well, you need JUNOS. JUNOS is copyrighted software, so be aware you can’t download it easily from somewhere. I used JUNOS 9.3R1 (the worldwide “export” version) for J-series router. If you own a valid Juniper subscription, you can get access to the “J-series Install Bundle” here. Besides, this disclaimer does apply: Don’t ask me where you can obtain your copy – I don’t know. Don’t ask me to provide download links – I can’t tell, don’t post download links – I won’t be amused, don’t ask anyone for assistance to get your copy – they won’t be amused.
• On your host system you need some utilities: screen (the program GNU screen I mean) and socat to access the serial terminal. Note, the socat program is purely optional, but works around the necessity to run VirtualBox as root. For Debian, both tools are packaged under the corresponding name and are easy to be installed as they do not require any configuration. I will come back later to this.
• If you plan to use VDE (“Virtual Distributed Ethernet“) on your host network, optionally install this as well. The Debian package is called vde2.
• An OpenSSH server on the virtual machine host. This is not strictly necessary, but the simplest way to exchange files with the host later.

Prepare a Virtual Machine

I won’t go too much into detail here. VirtualBox is rather straightforward in its use. I presume VirtualBox is up and running. Follow instructions for your favorite Linux distributions if you don’t know how to. For the Debian distribution you can simply do a apt-get install virtualbox-ose virtualbox-ose-dkms virtualbox-ose-qt and you are pretty much done. Once you are done, go and step ahead. First, create a new virtual virtual machine. When asked, choose BSD as OS type and FreeBSD as version. If you are on AMD64, make sure you setup a pure x86 environment (i386).

The VirtualBox setup wizard

As storage type, you are free to choose whatever you prefer. I would suggest to use a plain, fixed-size image stored in a file based storage. Following the JUNOS release notes you will need at least 1 GiB of disk size, but I failed with such small images previously. Note, you need some free space left to store the actual JUNOS image and the FreeBSD installation as well. Therefore I used a 5 GiB disk, just to make sure I won’t run into troubles.

Same applies for memory: The release notes imply you are going to need at least 256MiB of memory, but I failed to install JUNOS with less than 512MiB previously (with a very cryptic error message upon bootstrapping). Since I can afford it, I chose 1 GiB memory for my virtual machine therefore. Finish the setup wizard now, but before you are actually trying to start the virtual machine, make sure the assistant attached the network adapter to connect the guest operating system. It should be configured to make use of NAT for now (which should be default in VirtualBox). This is not strictly necessary, and we are going to change this later anyway, but to ease the installation, start with NAT here.

Installation of FreeBSD

Next, go and install FreeBSD. This should not be too complicated and rather seamless. If you are familiar with FreeBSD, you can skip this section. Just make sure you follow my guide for the partitioning, everything besides is not really important and will be overruled by JUNOS later anyway.

The 'first run wizard', choose the CD-ROM image.

When starting the virtual machine for the first time, VirtualBox will present you an installation wizard to aid you to start installation of a guest operating system. You can use that assistant to load the FreeBSD installer image you downloaded hopefully before, when I was referring to it. Again, I used 4.11-RELEASE-i386-miniinst.iso to setup BSD. Yes, it is intentional to use that old and obsolete version. Quickly after starting the virtual machine with the FreeBSD image being mounted as installation media, you should be asked whether you want to configure the kernel. You can choose “skip kernel configuration and continue with installation” here. Moreover select a “standard” installation, when asked.

FreeBSD partition setup (1)

Beware, here be dragons: To make the JUNOS installation seamless afterwards, pay attention when setting up the partitions. When fdisk is started by the installer, hit the “A” key to make fdisk use the entire disk, and type “Q” afterwards to go on.

FreeBSD partition setup (2)

Next, the “disklabel editor prompts up. Apply the following partitioning (see screen shot above as well; UFS is fine, don’t use ZFS if you are on a later version of FreeBSD):

• ad0s1a; mount point: /; size 1024M
• ad0s1b; use as: swap; size 1024M
• ad0s1c; mount point: /config; size 12M
• ad0s1d; mount point: /var; use remaining space which is ~ 3059M if you created a 5GiB disk before

FreeBSD Installation

Next, when asked, install the FreeBSD boot manager (“BootMgr“). The remaining parts of the installer aren’t that important. To safe time you can safely choose “minimal” (“smallest configuration possible”) when being asked about the distribution you want to install. Afterwards, the actual installation starts – be patient and wait until it completes and set a root password when asked to do.

As soon as FreeBSD finished to copy the data to your disk, you will be prompted to answer some post-installation questions; for example it might want to know from you whether you need SLIP, the Internet superserver (“inetd”), FTP, Linux compatibility and so on. You can safely answer “no” to all those questions. That brings you back to the distribution selection menu, but this time choose “exit install” and shut down the virtual machine when you are told that would be safe. Remember to remove the installation media (i.e. the CD-ROM image) now. You should be able to boot into your freshly installed FreeBSD:

FreeBSD up and running

Serial Terminal

Now I instruct you to configure a serial terminal. Note, JUNOS will redirect all output to a serial terminal, this means you won’t be able to see any output until JUNOS is up and running on your monitor output. Hence you need to enable a serial port on your virtual machine, you can use to access the router firmware afterwards. Note, I will instruct you to configure a serial terminal for FreeBSD here as well before. Strictly speaking, that is not required, as JUNOS will take care on its own. However I feel more comfortable to use my native Linux terminal, rather than the VirtualBox GUI display – if you feel the same, enable the serial terminal as well. You are very likely to spend some more time on that FreeBSD shell right now, so make it as comfortable as possible to you.

Enable serial console

Since you should have a freshly booted FreeBSD virtual machine running upon this point, enable the serial terminal on the guest operating system first (don’t worry, we will configure the actual virtual device later). To do so, open a root shell on your FreeBSD and edit /boot/loader.conf You can use vi (sorry, no vim in base installation) or edit which is easier to use (similar to nano on Linux). You need to append

console=”comconsole”

to that file. Alternatively, do do:

echo 'console="comconsole"' >> /boot/loader.conf
cat /boot/loader.conf 
# -- sysinstall generated deltas -- #
userconfig_script_load="YES"
console="comconsole"

Next, you need to spawn a the serial terminal on the serial port. Open the file /etc/ttys and change the entry starting with ttyd0 as follows:

ttyd0 “/usr/libexec/getty Pc” vt100 on secure

Serial terminal setup

You can use your favorite editor once again to do so. Power off the machine now by typing halt. Once the machine is stopped, go and enable the serial port on your VirtualBox virtual machine settings dialog. To do so, switch to “serial port” enable “port 1″ (COM 1) and switch the port type to be “Host Device” if you do run VirtualBox as root (you better don’t!). As non-root user, the required ioctl syscall will fail, so better use “Host Pipe” in that case. Make sure, you checked “Create Pipe” and choose an arbitrary file name to connect that pipe. I chose /home/arno/VirtualBox/JUNOS/ttyS0.raw, you are free to use whatever you prefer – just ensure the path is writable for the user you run VirtualBox as. Now you are able to start the virtual machine once again. It is a bit more complicated to access the serial terminal when provided as pipe on Linux – this is why I told you to install socat before. If you preferred to let VirtualBox create a host device for the virtual machine, you can connect it directly. Since I selected the pipe mode before, I use the socat utility to create a pseudo terminal out of that pipe as follows (note my file name outlined above):

cd ~/VirtualBox/JUNOS
socat UNIX-CONNECT:ttyS0.raw PTY,link=ttyS0
# this call should block. Open another terminal an execute
screen ~/VirtualBox/JUNOS/ttyS0

Serial console is up and running

This should drop you into a serial console, providing access to your FreeBSD guest instance on the host console, as soon as the kernel boots up.

Prepare for JUNOS Bootstrap

Now we’re actually going to boostrap JUNOS over the FreeBSD installation. Make sure the serial console is up and running at this point, as you won’t see any output after the next reboot. Moreover, once more make sure, your host network connectivity is set to NAT to ease your life. First, configure your guest network. since you set NAT, VirtualBox provides you an embedded DHCP server for host network access. All you need is to request an IP now, to ensure host connectivity:

Setup networking on FreeBSD

dhclient em0
ping -c3 10.0.2.2

Don’t worry if your guest gets another IP, however chances are you will get “10.0.2.15” as guest IP, whereas the host system should be accessible on “10.0.2.2“. This step is necessary to transfer the JUNOS installation package to your guest operating system. Therefore I am running an OpenSSH server on my host system, which I do use to transfer my JUNOS copy. Do something like

cd /root
scp arno@10.0.2.2:/home/arno/fw/jinstall-9.3R1.7-export-signed.tgz .

on your FreeBSD guest, where /home/arno/fw/jinstall-9.3R1.7-export-signed.tgz points to my copy of the JUNOS image in my case.

Copy JUNOS to the image

Next step is a bit annoying if you try to install a copy of JUNOS being more recent or equal to JUNOS 7.4. Since that version, the JUNOS installer verifies the hardware platform first, to ensure the PIC is supported (or not existent in our case). Obviously we don’t, so installing JUNOS will fail with a cryptic error message like

ELF binary type “0″ not known.
Abort trap

Who says Matryoshka dolls are for wimps only?

However we can bypass this check by replacing the program, responsible to perform that PIC check by another program – “/usr/bin/true” is just fine. Its only requirement is, to make sure the “checkpick program terminates with an exit code of 0 (which, you might now, indicates success on Unix). When extracting the FreeBSD package that is providing JUNOS, you will notice, there are several tarballs stacked within each other, where the actual tarball finally containing the JUNOS package, is being verified against a checksum coming along that data first.

Therefore, basically you need to extract the JUNOS “jinstall-signed” tarball as provided by Juniper, replace the checkpic program by /usr/bin/true and package all together again. The jinstall-signed tarball will contain a jinstall archive, along with checksums over that tarball and some other files we are not interested in. The unsigned jinstall tarball will contain again another archive called pkgtools.tgz. You need to extract that archive as well, which will finally contain the checkpic program you need to replace. As soon as you did, track back and package all those archives again until you reach the penultimate hierarchy: You will see files ending with .md5 and .sha1 there. Those need to be updated, to match your forged tarball you fabricated before. This is required to make pkg_add accept the package; you can use the md5 program and the openssl respectively to do so.

Once again, step per step, this time with concrete commands:

cd /root
mkdir jinstall
cd jinstall
tar zxfv ../jinstall-9.3R1.7-export-signed.tgz
# will give you something like
# +CONTENTS
# +COMMENT
# +DESC
# +INSTALL
# jinstall-9.3R1.7-export.tgz
# jinstall-9.3R1.7-export.tgz.md5
# jinstall-9.3R1.7-export.tgz.sha1
# jinstall-9.3R1.7-export.tgz.sig
# certs.pem
# issu-indb.tgz
mkdir jinstall-9.3R1.7-export
cd jinstall-9.3R1.7-export
tar zxfv ../jinstall-9.3R1.7-export.tgz 
# will give you something like
# +CONTENTS
# +COMMENT
# +DESC
# +INSTALL
# +EINSTALL
# +REQUIRE
# bootstrap-install-9.3R1.7.tar
# jbundle-9.3R1.7-export.tgz
# pkgtools.tgz
mkdir pkgtools
cd pkgtools
tar zxfv ../pkgtools.tgz 
# will give you something like
# pkg/manifest
# pkg/manifest.certs
# pkg/manifest.sha1
# pkg/manifest.sig
# bin/checkpic
# ^^^^^^^^^^^^^^^
# this is the bad guy. Let's replace it
cp /usr/bin/true bin/checkpic 
./bin/checkpic ; echo $?
# should return "0"

Now back track again. Be careful not to forget to update the checksums:

tar zcfv ../pkgtools.tgz *
cd ..
rm -rf pkgtools
tar zcfv ../jinstall-9.3R1.7-export.tgz *
tar zcfv ../jinstall-9.3R1.7-export.tgz *
cd ..
rm -rf jinstall
md5 -q jinstall-9.3R1.7-export.tgz >  jinstall-9.3R1.7-export.tgz.md5
openssl sha1 jinstall-9.3R1.7-export.tgz > jinstall-9.3R1.7-export.tgz.sha1

At this point you should have replaced the checksums matching your updated jinstall package. Unfortunately openssl will be a bit more verbose than necessary, the resulting .sha1 file will look like this:

SHA1(jinstall-9.3R1.7-export.tgz)= f197509c768183017d3445226fae0280110da43

Unfortunately FreeBSD’s package system won’t understand that format, so remove everything but the checksum itself. You can use the edit editor once again. The result should be a checksum only, looking like this:

f197509c768183017d3445226fae0280110da43

The remaining steps just stack packages once again:

# see notes above
edit jinstall-9.3R1.7-export.tgz.sha1
cat jinstall-9.3R1.7-export.tgz.sha1 
f197509c768183017d3445226fae0280110da43
tar zcfv ../jinstall-9.3R1.7-export-signed.tgz * 
 
cd ..
rm -rf jinstall
# Now install the package
pkg_add -f jinstall-9.3R1.7-export-signed.tgz

Start JUNOS installation

You are almost done upon that point. You need to reboot once again and be patient. This will drop you into a freshly installed JUNOS CLI:

# reboot
Mar  4 18:39:59  reboot: rebooted by root
login: root
 
--- JUNOS 9.3R1. built 2008-11-12 23:40:11 UTC
root@% cli
root> show version 
Model: olive
JUNOS Base OS boot [9.3R1.7]
JUNOS Base OS Software Suite [9.3R1.7]
JUNOS Kernel Software Suite [9.3R1.7]
JUNOS Packet Forwarding Engine Support (M/T Common) [9.3R1.7]
JUNOS Packet Forwarding Engine Support (M20/M40) [9.3R1.7]
JUNOS Online Documentation [9.3R1.7]
JUNOS Routing Software Suite [9.3R1.7]

Get JUNOS Up and Running

Wohoo! JUNOS boots!

You did it! You should have your own “Olive” up and running. Shutdown the image one last time. This time we will finalize the network configuration, since NAT is probably not what you want to experiment with a router. Feel free to use what suites best to you here. I’d suggest you to use VDE which stands for “Virtual Distributed Ethernet“. VirtualBox bundles support for that into recent versions. This allows you to configure a virtual switch capable to configure VLANs and stuff like that on your virtual host network.

VDE configuration

You might also want to use more than one interface for your router. Verify once again you configured the right network interface chip set – as said not everyone is supported. Everything else can be left as is. If you chose to use VDE, make sure you started the switch pipe interface. You may want to enable debug mode to see ports to be connected to that switch:

[arno@snowball:~]$ vde_switch
vde$ debug/add port/+
1000 Success
vde$ 
3021 port/+ 01
3021 port/+ 02
3021 port/+ 03
3021 port/+ 04
vde$

If everything went right, you can access the virtual interfaces straight out of the box on your Olive. Note the rather unusual device naming though (at least for JUNOS):

root> show interfaces terse em*    
Interface               Admin Link Proto    Local                 Remote
em0                     up    up  
em1                     up    up  
em2                     up    up  
em3                     up    up