d(a)emonkeeper's purgatory

Sometimes you feel the requirement to replace an installed operating system in-band, which means while the old system is running and providing you access to the hardware. Say you’ve got a “less desired” Linux distribution on a server and you want to replace it by Debian, but you have no physical access to it and neither the possibility to install a bootable installation media (CD, USB, …) through a KVM switch or similar.

In this tutorial I’m going to replace a running Fedora Linux installed on a single hard disk by Debian Squeeze, only by using a remote SSH connection and its swap partition.

Prerequisites

The starting point, a pure standard Fedora installation

To apply my hints to your particular use case, it is not required to have a special hard disk layout or a special Linux distribution installed. Every recent Linux distributions might suit the purpose of being replaced by Debian, if it works at least partially as expected by several Debian tools. Note it must be some flavor of Linux though: You can’t bootstrap Debian over an existing Windows for instance, neither Solaris will. FreeBSD with the Linux compatibility layer enabled could eventually work, but I didn’t tried. I would happily like to hear about your failure oder success with FreeBSD. Problem is, you need the ability to execute Linux binaries and to install a boot loader which requires /dev access.

For my setup I used Fedora 14, which was installed into a single partition altogether with 2 GiB swap space. Note, it is important to understand the order of partitions do make a difference! If you installed your swap partition in the beginning of the hard disk, this eases your life a lot. This is, however, a very unlikely setting, therefore I didn’t assume you need. On the other hand I won’t cover a Linux installation spanning over several partitions (although this is does not fundamentally change the procedure). My partitioning looks like this:

• /dev/sda1 (ext4, used as root file system “/”)
• /dev/sda2 (swap, used as swap space)

You will need some spare disk space which can be overwritten: Swap is fine, unpartitioned or unallocated space is fine, a mounted partition which is relinquishable is fine (think of /home among others may work, /var probably not. Be careful you will lose data in this case!). In total you won’t need much space, something around 512 MiB is ok (but you might need some cleanups in between then), a bit more won’t hurt. I used a 2 GiB swap partition, but I never required more than 523 MiB during the migration process.

Outline

This is a brief outline how I’m going to replace the existing Fedora installation by Debian. My assumption is, as said, you don’t have BIOS access and your only possibility to access the installation is remote through SSH. Don’t worry if you need more explanation, I will step through in detail afterwards:

1. Claim some unused space, something around 512+ MiB is enough
2. Format unused space
3. Bootstrap Debian into the unused space
4. Do the post installation setup (setup /etc/network/interfaces, /etc/fstab, …)
5. Install the Debian grub boot loader into the master boot record (replace the boot loader from the original distribution)
6. Boot into the new Debian installation
7. Format the former root partition where the original Linux distribution is installed
8. Shift the Debian installation to the fresh formatted root partition, (which was previously containing Fedora in my case)
9. Reinstall boot loader again
10. Boot into the new, shifted system
11. Format swap partition, use it as swap partition again

Some random thoughts you may want to consider:

• This procedure is risky. It may result in an un-bootable system. If something gos wrong there is no recovery procedure you could use from remote.
• If you use LVM things are slightly more complicated, but the big picture is still the same.
• My goal is to replace a distribution in place without changing actual partition layout. An alternative approach would replace partitions by growing the Debian partition after the first boot. Note this is not possible if there is no unclaimed or unused space after the root partition. You can not easily expand a partition with new space available in front. This means for a common partitioning like mine, where I install Debian into /dev/sda2, no free space is available after that partition, only before. This won’t allow you to grow such a partition.

Prepare installation disk

I already explained my partitioning before. Besides my starting position is a standard Fedora 14 installation, not having anything special in it. My Fedora installation has its root partition in and swap in /dev/sda2. Obviously I also enabled the SSH server.

[root@bootstrap arno]# fdisk -l
Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ae12b
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    36866047    18432000   83  Linux
/dev/sda2        36866048    41943039     2538496   82  Linux swap / Solaris
[root@bootstrap arno]# cat /etc/fedora-release 
Fedora release 14 (Laughlin)

Swap space is not essential for Linux and can be detached from a running system, so I’m going to use that partition to install Debian in it. This is a temporary solution, since swap space usually tends to be small, when compared to data disks. I will show how to shift data from this swap partition to the actual root file system later. First use swapoff /dev/sda2 to disable the swap partition and do a mkfs.ext4 /dev/sda2 to create a ext4 file system on it.

[root@bootstrap arno]# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/sda2                               partition       2538492 0       -1
[root@bootstrap arno]# swapo
swapoff  swapon   
[root@bootstrap arno]# swapoff /dev/sda2 
[root@bootstrap arno]# mkfs.ext4 /dev/sda2 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
158720 inodes, 634624 blocks
31731 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=650117120
20 block groups
32768 blocks per group, 32768 fragments per group
7936 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912
 
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Bootstrap Debian

One of Debian’s greatest features is its bootstrap ability. You need very little to install a minimal but (almost) complete Debian installation into a single directory on an existing file system without much fuss. All you need is the debootstrap utility, which is a set of helper scripts, doing a Debian base install into a directory of your choice.

It requires a network connection and a Perl interpreter to run and a POSIX compliant shell. If you are already on a Debian system, you can simply install deboostrap through apt, package name is deboostrap (in reality you probably want to use apt-get dist-upgrade on such a system instead). If your distribution does not offer bootstrap, you can get the source package. You don’t need to compile anything, just run it straight out of the extracted tar ball. Chances are, your distribution offers debootstrap as package, Fedora for example does. Unfortunately Fedora won’t include perl in the default installation, so chances are you need to install both:

[root@bootstrap arno]# yum install debootstrap.noarch perl
...
Complete!

Now you need to mount the ext4 file system you created previously. Use the mount point to bootstrap Debian into it. Don’t create any subdirectories! Fortunately this isn’t really complicated:

[root@bootstrap arno]# mount /dev/sda2 /mnt 
[root@bootstrap arno]# debootstrap --arch amd64 squeeze /mnt http://ftp.debian.org/debian
I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: insserv libbz2-1.0 libdb4.8 libslang2 
I: Found additional base dependencies: libnfnetlink0 libsqlite3-0 
I: Checking component main on http://ftp.debian.org/debian...
...
I: Base system installed successfully.

You will see quite a lot of output about packages being downloaded and installed. The procedure should result in a directory structure representing a Debian root file system. The “debootstrap output ends with I: Base system installed successfully., look for it to find out whether the procedure was successful.

As next step you have to mount /dev, /proc and /sys into the file Debian system. Strictly speaking /sys is not required, but it won’t hurt. Use a bind mount to clone the /dev tree. You need those pseudo file systems and the device drivers to complete the installation within the Debian chroot. You can run apt-get within the chroot, therefore you can install whatever you want, to get your system up and running in the next reboot. Note deboostrap won’t install a complete system, most notably boot loader and kernel is missing. Also some essential configuration is not realized by it. Moreover, if you need some special configuration, think of firmware packages for example, a WiFi network, or a VLAN setup, you are supposed to install all required packages by hand. By default, only simple wired network connections are supported out of the box.

Within the upcoming command sequence the chroot call is most essential. It runs an interactive shell within a special root directory. The resulting shell looks and feels pretty much like every other Debian system. Note there are some limitations and most notably you still run the Fedora kernel and its network configuration implicitly and /dev is from the host system’s kernel.

[root@bootstrap arno]# mount proc /mnt/proc -t proc
[root@bootstrap arno]# mount sys /mnt/sys -t sysfs
[root@bootstrap arno]# mount --bind /dev /mnt/dev/
[root@bootstrap arno]# chroot /mnt/
root@bootstrap:/# apt-get update 
Get:1 http://ftp.debian.org squeeze Release.gpg [1671 B]
Ign http://ftp.debian.org/debian/ squeeze/main Translation-en                                                                                               
Hit http://ftp.debian.org squeeze Release                                                                                                                   
Hit http://ftp.debian.org squeeze/main amd64 Packages                                                                                                       
Fetched 1671 B in 10s (159 B/s)                                                                                                                             
Reading package lists... Done

Make sure you still are in the chroot as explained above. You should install a kernel and the grub2 boot loader now. The editor vim in my command line below is purely optional, feel free to choose your personal preference instead. During installation of the kernel you might see some warnings: It is safe to ignore “Can not write log, openpty() failed (/dev/pts not mounted?)” and “df: Warning: cannot read table of mounted file systems: No such file or directory” there.

Install kernel and remaining bits

Afterwards build a minimal /etc/fstab file, suitable to boot your system. To avoid any confusion I prefer to use disk UUIDs instead of device names, as does the Debian installer for new installed systems as well. Use the blkid command to determine the disk UUID of your former swap partition, and add it as root file system to your /etc/fstab file:

root@bootstrap:/# apt-get install linux-image-2.6-amd64 grub2 vim
root@bootstrap:/# echo "proc            /proc           proc    defaults        0       0" >> /etc/fstab 
root@bootstrap:/# blkid /dev/sda2
/dev/sda2: UUID="8e5a4870-6f4f-481c-b36e-3b5735c19b18" TYPE="ext4" 
root@bootstrap:/# echo "UUID=8e5a4870-6f4f-481c-b36e-3b5735c19b18 /           ext4    defaults        0       2" >> /etc/fstab

Next configure your network. If you connect from remote this is crucial! You need to know yourself how to configure your network and adapt network configuration to match your requirements. Open /etc/network/interfaces in your favorite editor. Mine looks like this for a DHCP auto configuration. Note again, this might not suite your needs.

# Used by ifup(8) and ifdown(8). See the interfaces(5) manpage or
# /usr/share/doc/ifupdown/examples for more information.
 
# The loopback network interface
auto lo
iface lo inet loopback
 
allow-hotplug eth0
iface eth0 inet dhcp

See the referenced man pages for more examples. Now set up a system host name and verify if your name server configuration looks suitable. debootstrap copies your existing /etc/resolv.conf file from the host system if found.


root@bootstrap:/# echo "bootstrap" > /etc/hostname
root@bootstrap:/# vi /etc/resolv.conf # copied over


The installation of the grub boot loader into the master boot record

root@bootstrap:/# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-2.6.32-5-amd64
df: Warning: cannot read table of mounted file systems: No such file or directory
 
root@bootstrap:/# passwd 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

A random side note: If you switch from Fedora, CentOS or RHEL the passwd command might fail like this:

root@bootstrap:/# passwd 
passwd: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 is not authorized to change the password of root

This is because those distributions enable SELinux by default, which prevents to set a root password. However /etc/shadow within the chroot does not carry a SELinux context, as Debian does not enable or enforce SELinux policies unless explicitly installed. To solve this issue leave the chroot for a moment (just type exit) and mount

/selinux

into it, since passwd will read in

/selinux

whether the caller is authorized to change the password file on a SELinux enabled system. Afterwards switch again into the Debian chroot and try again:

[root@bootstrap arno]# mount --bind /selinux /mnt/selinux/
[root@bootstrap arno]# chroot  /mnt
root@bootstrap:/# passwd 
...

Once you did, you are almost done. Maybe you want to install the SSH server as well:

root@bootstrap:/# apt-get install ssh
Setting up openssh-server (1:5.5p1-6) ...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Restarting OpenBSD Secure Shell server: sshd.
Setting up xauth (1:1.0.4-1) ...
Setting up ssh (1:5.5p1-6) ...
...

As last step you need to install the boot loader on your disk. Be careful, upon completion of this step you won’t be able to boot into your original system anymore. You need to replace the boot loader from the original operating system since that one would boot you into Fedora. Therefore, for my setup I need to replace Fedora’s bootloader by Debian’s grub.

If you want to be more careful, you might want to consider to install Debian’s grub into the partition boot record /dev/sda2 instead of the master boot record, and chain load it from the original (i.e. Fedora’s) boot loader instead. This needs additional tuning though, I won’t cover this here.

root@bootstrap:/# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-amd64
done
root@bootstrap:/# grub-install /dev/sda
Installation finished. No error reported.

The Debian boot loader shows up

Reboot into the temporary system

That was it. Leave the chroot and reboot your machine. If everything went good you are booted into Debian which hopefully brings up your network interface. Note the RSA host key changed, therefore you will see a nasty SSH warning. In this particular case you are safe to ignore it.

[arno@snowball:~]$ ssh 192.168.2.22
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
[arno@snowball:~]$ ssh 192.168.2.22 -lroot
root@192.168.2.22's password: 
Linux bootstrap 2.6.32-5-amd64 #1 SMP Mon Mar 7 21:35:22 UTC 2011 x86_64
 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
...

Log in and verify whether you actually booted into your new system. You should have booted into your former swap partition, which is /dev/sda2 in my case. Until now the actual data partition, which contains your former operating system, is kept as is. Now, as the data partition is not being actively used anymore you can start working with it. Therefore you can format it now and prepare it to be used as root file system for Debian afterwards:

root@bootstrap:~# mkfs.ext4 /dev/sda1 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1152816 inodes, 4608000 blocks
230400 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
141 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000
 
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 23 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Prepare data partition

You can simply copy everything from your existing and previously installed Debian installation and purge the old copy afterwards. However make sure you keep file permissions, owener and perhaps time stamps. I suggest you to use find in combination with cpio. See the full command below. The -xdev argument makes sure, find won’t cross file system boundaries which is required to avoid copy loops, since /mnt must be skipped while copying data (moreover /proc, /dev will cause problems as well, if you don’t skip them). The cpio command looks rather cryptic but basically I enable ‘copy-pass mode‘ (read more about cpio in the corresponding man page) and make sure you preserve modification times. The -V will print a dot for every file being copied over.

root@bootstrap:~# mount /dev/sda1 /mnt/
root@bootstrap:~# cd /
root@bootstrap:/# find . -xdev -print0 | cpio -pamd0V /mnt 
........................................................................................................................
854050 blocks

Make sure you didn’t forget anything. Don’t worry about temporary files like /etc/mtab, files in /var/run and so on. Fortunately Debian won’t care upon next boot for most of them. Nonetheless you need to change some things before you can actually boot this copied data. Therefore mount pseudo file systems (/dev, /proc, /sys) again and chroot into the copied data partition. Change /etc/fstab once again, but this time make sure you choose the data partition as future root file system.

root@bootstrap:~# mount /dev/sda1 /mnt 
root@bootstrap:~# mount --bind /sys /mnt/sys
root@bootstrap:~# mount --bind /proc /mnt/proc/
root@bootstrap:~# mount --bind /dev /mnt/dev
root@bootstrap:~# chroot /mnt/
root@bootstrap:/# blkid /dev/sda1
/dev/sda1: UUID="44b02850-daf9-4401-985a-a1b2211bd3fb" TYPE="ext4" 
root@bootstrap:/# vi /etc/fstab 
root@bootstrap:/# cat /etc/fstab 
# UNCONFIGURED FSTAB FOR BASE SYSTEM
proc            /proc           proc    defaults        0       0
UUID=44b02850-daf9-4401-985a-a1b2211bd3fb /           ext4    defaults        0       2

Once again you need to rebuild the grub configuration, this time you will update the file system you want to boot from, as /boot/grub, being copied, still points to the swap partition in /dev/sda2. Fortunately Debian will take care about those things if you call update-grub within the chroot. Again: Call it from the chroot with /etc/fstab already being updated. Same holds for grub-install which will update the master boot record to point to /dev/sda1 to lookup for /boot.

root@bootstrap:/# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-amd64
done
root@bootstrap:/# grub-install /dev/sda
Installation finished. No error reported.

As soon as you did, leave the chroot and reboot. You should be boot into the same system as before, but this time booted from /dev/sda1 where, in my case, Fedora was originally installed. This leaves /dev/sda2 available for swap usage again.

root@bootstrap:~# mount 
/dev/sda1 on / type ext4 (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
root@bootstrap:~# mkswap /dev/sda2 
Setting up swapspace version 1, size = 2538492 KiB
no label, UUID=1858a22d-e622-4744-8930-4214b0907f4b
root@bootstrap:~# swapon /dev/sda2 
 
You can verify, everything looks good by using <code>free</code> or querying <code>/proc/swaps</code>:

root@bootstrap:~# free 
             total       used       free     shared    buffers     cached
Mem:       1027264      38812     988452          0       2972      14356
-/+ buffers/cache:      21484    1005780
Swap:            0          0          0
root@bootstrap:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              18G  627M   16G   4% /
tmpfs                 502M     0  502M   0% /lib/init/rw
udev                  497M  108K  497M   1% /dev
tmpfs                 502M  4.0K  502M   1% /dev/shm

Remaining Bits

That was it. You successfully replaced your former Linux distribution by Debian if you reached this step. Your Debian installation might behave like every other upon this point. You might tweak your /etc/fstab again to take care about pseudo file system you need and aren’t mounted automatically.