Xen
From EECS
Contents |
Installing Xen
Debian packages
This section covers installing Xen on Debian testing.
Install Xen utilities and scripts from Debian packages
aptitude install xen-utils-3.0
Install the Xen version of libc, so that TLS (/lib/tls) does not have to be mv-ed away
aptitude install libc-xen
Install the Xen hypervisor (not needed if the hypervisor will be installed by compiling from sources)
aptitiude install xen-hypervisor-3.0-i386
Networking
By default the /etc/xen/scripts/network-dummy script is used to set up networking. Change it to create two bridges, one for eth0 and one for eth1.
#!/bin/sh dir=$(dirname "$0") script=/etc/xen/scripts/network-bridge # eth0 $script "$@" netdev=eth0 vifnum=0 bridge=xenbr0 # eth1 $script "$@" netdev=eth1 vifnum=1 bridge=xenbr1 exit 0
Compile and install the linux kernel as shown later.
Kernel
Repository update
UPDATE: changed to xen-3.0.3 !!!
initialize the repository -- only the first time
cd /usr/src/xen mkdir xen-3.0.3-testing.hg hg init xen-3.0.3-testing.hg cd xen-3.0.3-testing.hg
update the repository -- you may also want to keep the old version rather than update
hg pull -u http://xenbits.xensource.com/xen-3.0.3-testing.hg
At the moment we are using changeset 11775:328b62294512 (date: Tue Oct 17 22:09:52 2006 +0100) You can change the working version to a particular changeset with
hg co [changeset]
OLD VERSION was:
initialize the repository -- only the first time
cd /usr/src/xen mkdir xen-3.0-testing.hg hg init xen-3.0-testing.hg cd xen-3.0-testing.hg
update the repository -- you may also want to keep the old version rather than update
hg pull -u http://tx.downloads.xensource.com/xen-3.0-testing.hg
At the moment we are using changeset 9697:18e8e613deb9 (date: Tue May 30 18:14:05 2006 +0100) You can change the working version to a particular changeset with
hg co [changeset]
Kernel compile
The currently used Xen kernel is kept on meat in /usr/src/xen/xen-3.0-tesing.hg/.
Please see the end of /usr/src/xen/xen-3.0-testing.hg/README. It contains very useful information.
With Xen 3.0 the recommended way is to have just one kernel used for both domU and dom0 (unless short kernel compile times are needed, e.g. for kernel development, where a stripped down domU kernel compiles much faster). For netlab deployment, linux kernel compiled from the Xen repository is used.
If you simply want to add some options and recompile the kernel:
cd /usr/src/xen/xen-3.0-testing.hg make linux-2.6-xen-config CONFIGMODE=menuconfig (or xconfig) make linux-2.6-xen-build make linux-2.6-xen-install
The compiled kernels can be found in /usr/src/xen/xen-3.0-testing.hg/dist/install/boot. You may install them in the following way:
cd /usr/src/xen/xen-3.0-testing.hg/dist/install/boot cp -d * /boot/
Please note that the kernel can also be compiled in the traditional way by going into the kernel directory and using make menuconfig, make, ... This implies, that make-kpkg can also be used. It may be neccessary to supply --arch=i386 when compiling the Debian way.
Hypervisor compile
If the repository has been updated, the new kernel may no longer be compatible with the old Xen hypervisor. To build a new hypervisor, you can simply do
make xen
Please note that this information was taken from the README and has not yet been verified. The hypervisor should then be in dist/install/boot.
Complete Xen dist compile
To compile the rest of the Xen tools, Xen hypervisor and linux kenrel(s), use
make dist
For compiling the utilities, python-dev package is needed.
Grub configuration
To boot Xen, an entry in grub config file is needed
title Xen kernel /boot/xen-3.0.2-3.gz module /boot/vmlinuz-2.6.16.13-xen root=/dev/md0 ro
Dom0 memory
Dom0 memory can be limited by appending dom0_mem=<number in KB> to the Xen hypervisor command line in grub configuration.
Guest configuration
Config files for domU's (guests) are kept in /etc/xen_guests/<guest name>/<guest name>.cfg
A sample domU config file looks as follows
kernel = "/boot/vmlinuz-2.6-xen" memory = 32 name = "matus" nics = 1 vif = [ 'bridge=xenbr0', 'bridge=xenbr1' ] disk = ['phy:/dev/vg0/matus-root,sda1,w', 'phy:/dev/vg0/matus-swap,sda2,w'] root = "/dev/sda1 ro"
Guests need a root partition and optionally can use also a swap partition. LVM is being used to allocate partitions (logical volumes) for domU's from volume group //vg0//. A partition (logical volume) can be created with
lvcreate -L <size in MB> -n <logical_volume_name> /vg0
The new logical volume can be accessed via /dev/vg0/<lvname>. The naming scheme for logical volumes is <domu_name>-root and <domu_name>-swap for root and swap partitions, respectively.
A filesystem can be created in the new logical volume with
mke2fs /dev/vg0/<lvname> tune2fs -i 0 -c 0 /dev/vg0/<lvname>
And for swap
mkswap /dev/data_vg/<domU_name>-swap
Afterwards, the skeleton domU root can be copied into the new domU (cp -a). Actually, dd could be used to save some tiping, but the copying of all blocks could take longer.
In the domU config file acces to the new logical volume would be configured like
disk = ['phy:/dev/datavg0/<domU_name>-root,sda1,w', 'phy:/dev/vg0/<domU_name>-swap,sda2,w']
Automatic domU start on dom0 boot
To have a domU start automatically when dom0 is booted up, a symlink to the guest's config file should be added into /etc/uml/auto/. All guests will automatically be shutdown during dom0 shutdown.
The script taking care of automatically started/stopped domains (/etc/init.d/xendomains) seems to be doing xm save/restore. This should break if the kernel is changed.
Clock
Clocks of the domU's are tied to the clock of the dom0. Hence, runnning ntpdate only in dom0 should be sufficient.
DomU's can be given independent clocks by doing
echo 1 > /proc/sys/xen/independent_wallclock
or specifying on the kernel command line
independent_wallclock=1
Automated creation of domUs on veggie (xen-tools)
xen-tools is configured on veggie. Discussion below is relevant only for the configuration on veggie.
Create a domU with
xen-create-image --hostname=test-bla-host --role=minimal --memory=64
A role is then chosen with the --role switch. The roles are defined in in /etc/xen-tools/role.d/. Useful roles are
- cl - CAOS lab domU
- an - Advanced Networking assignment domU
- minimal
A role script usually sets up accounts, adds sudo rights, installs and removes packages, ... The cl role script also sets up /etc/network/interfaces by looking up the hostname in DNS.
Various hook scripts executed for each domU are in /etc/xen-tools/hooks.d. Only executable scripts are run. Furthremore, scripts in /usr/lib/xen-tools/ are executed.
Files from /etc/xen-tools/skel are copied to the domUs.
Things to do manually after a domU is created
- dpkg-reconfigure passwd - enable shadow passwords and set root password
- edit /etc/network/interfaces - set IP addresses
Automated creation of domUs on fruit with Lenny (xen-tools)
Discussion below is relevant only for the configuration on fruit and Lenny distribution.
File modifications
In the first step, a few modifications to the module and configuration files needs to be applied.
/etc/modules contains kernel modules to load at boot time, and the loop field needs to be set to:
loop max_loop=64
/etc/xen/xend-config.sxp is a xend configuration file. Ensure that the following fields are set as:
#(network-script network-dummy) #(network-script 'network-bridge netdev=eth0') (network-script 'network-fruit') [...] (vif-script vif-bridge)
Creating Image-Based Virtual Machine
The modification of xen-tools configuration file should produce an error-less image. The /etc/xen-tools/xen-tools.conf file needs to be modified as follows:
[...] dist = lenny # Default distribution to install. [...] kernel = /boot/vmlinuz-`uname -r` initrd = /boot/initrd.img-`uname -r` [...] mirror = http://ftp.de.debian.org/debian/ # enables one to access the repository initially, since only ipv4 will be available at creation [...] serial_device = hvc0 [...] disk_device = xvda [...]
Assign IPv4,6 addresses in the bind files of kiwi. The new IP address will be used in the process of image creation in the next step.
Given that the newly-assigned IP address is 212.212.212.212 we create a guest domain, example-domain as follows:
xen-create-image --hostname=example-domain --size=4Gb --swap=256Mb --ip=212.212.212.212 --memory=128Mb --arch=amd64 --role=eecs
The roles are stored in /etc/xen-tools/role.d and in this case, eecs role has been manually defined and stored in the /role.d directory. The above command should create both the disk and the swap images along with the file system. The configuration file associated with the hostname is stored in: /etc/xen/example-domain.cfg. The "Netowrking" field in this file should be modified to:
vif = [ 'bridge=eth0', 'bridge=eth1' ]
Using the example-domain.cfg file we start the machine with console by xen manager:
xm create -c /etc/xen/example-domain.cfg
Log-in as root, no password required. At the moment, networking is down, ipv6 module is not present and the interfaces have to be configured in the /etc/network/interfaces file similarly to:
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # # The primary network interface auto eth0 iface eth0 inet static address 212.XXX.XXX.XXX gateway XXX.XXX.XXX.XXX netmask XXX.XXX.XXX.XXX # iface eth0 inet6 static address XXXX:XXX:XXX:XXXX::XX netmask 64 gateway XXXX:XXX:XXX:XXXX::X # # The secondary network interface auto eth1 iface eth1 inet static address 192.XXX.XXX.XXX netmask 255.255.255.0 # iface eth1 inet6 static address XXXX:XXX:XXX:XXXX::XX netmask 64 gateway XXXX:XXX:XXX:XXXX::X
Restart networking or reboot the image:
reboot
Only few modules are present at the image creation time. Others, in particular ipv6, need to be added manually. At the time of writing, the 2.6.26-1-xen-amd64 version was mounted on the image. Therefore, the linux-modules-2.6.26-1-xen-amd64_2.6.26-13lenny2_amd64.deb modules package needed to be downloaded and installed. This can be easily done by running an "apt-get dist-upgrade" command after adding the following mirror to the '/etc/apt/sources.list' file:
deb http://security.debian.org/debian-security lenny/updates main
Next, insert the ipv6 module with insmod or modprobe and reboot the image. Now, you can ping and ping6 machines on the network. IPv6 is up and running.
Possible problems: console hangs, no connection
If you try to create the machine (i.e., xm create -c <lv.conf>), and then to login at its terminal, or ssh to it from a remote box, but the terminal hangs, then there are 2 things that need to be taken into account.
1. Intallation of the udev. (take down the machine, mount it, create a script that installs udev at the start up, verify that the sources.list of the apt is suitable).
2. Need to make sure that the serial device of the created machine (i.e., serial_device = hvc0) and the one specified in the /etc/inittab of the vm are the same.
3. You might need to add the following line to /etc/fstab
echo "none /dev/pts devpts defaults 0 0" >> /etc/fstab
if the corresponding line for the pts is not present in the /etc/mtab.
4. After installation of the udev, there might be some follow up problems with the networking, i.e., cannot connect to the box. A solution is a removal of the misbehaving z25-persistent-net.rules in the /etc/udev/rules.d directory and rebooting the machine.
There might also be an error of the sort:
INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: cannot execute "/sbin/getty" INIT: Id "1" respawning too fast: disabled for 5 minutes
It is connected to the fact that the /sbin/getty has been removed (most probably due to the fact that the linux-modules-2.6.X-xen-X has been installed). You would either need to get getty back in place, or have an alternative (i.e., rungetty or fgetty) running instead. After installing a getty-like application, make sure that you indicate it in the /etc/inittab file, and some adjustment to the speed might be necessary (i.e., modifying or removing at all the 38400 bauds mentioned in the /etc/inittab).
Setting up a separate network for experiments
Several time it was needed to provide a network connecting several domUs within one dom0. This network would be used for experiments and users might have root access to domUs. Therefore, this network should be isolated as much as possible. Especially, the internal netlab network should be protected. The easy solution turned out to be creating just an empty bridge:
brctl addbr xenbr2 ifconfig xenbr2 up
Pleae note that without the ifconfig up command communication on xenbr2 did not work. The domU configuration would then contain
vif = [ 'bridge=xenbr0', 'bridge=xenbr2' ]
DomU management (/usr/sbin/xm)
xm is used to manipulate mainly the domU guests
start a guest
/usr/sbin/xm create -c <guest config file>
i.e., for carrot you would use /uml/xen/guests/carrot.cfg as the config file. The -c switch attaches you to the domU console and can be omitted if you do not wish to attach the the console.
shutdown a guest
/usr/sbin/xm shutdown -H <guest name>
i.e., for carrot you would use carrot as guest name
check which guests are running
/usr/sbin/xm list
use a guest's console
/usr/sbin/xm console <guestname>
exit the console with Ctrl-]
Fixing routing table on dom0
The routing table for IPv6 sometimes get broken. Maybe the reason is that routing advertisements go via several logical devices, which use them to autoconfigure themselves, but should not. This results in not IPv6 routing to be broken. The following seems to fix the problem.
echo 0 > /proc/sys/net/ipv6/conf/xenbr0/autoconf echo 0 > /proc/sys/net/ipv6/conf/xenbr0/accept_ra
route -A inet6 | grep 2001 ip -f inet6 route del 2001:638:709:4::/64 dev xenbr0 route -A inet6 del default route -A inet6 add default gw 2001:638:709:4::1 dev eth0
Useful links
- main xen webpage
- http://www.cl.cam.ac.uk/Research/SRG/netos/xen/
- user manual
- http://www.cl.cam.ac.uk/Research/SRG/netos/xen/readmes/user/user.html
- downloads
- similar as above
- compiling
- the README in unstable seems more complete than the other ones
- list of fruits and berries
- http://en.wikipedia.org/wiki/List_of_fruits
- list of vegetables
- http://en.wikipedia.org/wiki/List_of_vegetables
- list of spices
- http://en.wikipedia.org/wiki/List_of_spices
Xen 2.0
This howto is for Xen 2.0. Please note that the netlab installation has been migrated to Xen 3.0, making this part obsolete.
- at the moment most xen related stuff (sources, config files, ...) are on veggie in /uml/xen
- filesystem images, kernels and config fiels are in /uml/xen/guests
- oops, we're running out of loop devices...
- Can we simply create more devices or do we have to change something in the kernel?
Kernel
compiling xen kernels on veggie
configure the domU kernel
cd /uml/xen/xen-2.0/linux-2.6.11-xenU make ARCH=xen menuconfig
configure the dom0 kernel
cd /uml/xen/xen-2.0/linux-2.6.11-xen0 make ARCH=xen menuconfig
compile the kernels
cd /uml/xen/xen-2.0/ make kernels
install kernels from /uml/xen/xen-2.0/dist/install/boot
cd /uml/xen/xen-2.0/dist/install/boot cp -d * /boot/
There have been problems compiling a xen-enabled (patched kernel) ourselves. This was most probably caused by the gcc-4.0 compiler. Even gcc-3.4 and gcc-3.3 don't seem to help.
Patches can be generated by
make mkpatches
unstable
Seems like I have just complied a xen kernel. The unstable snapshot from 2005-09-27 compiles with gcc-4.0. Should reside in /uml/xen/2005-09-27/xen-unstable. I will try booting it and comment on that later.
It seems like we cannot boot a xen 3.0 guest kernel in a 2.0 (stable) dom0.
The machine seems significantly slower - starting services after entering runlevel 2 takes ages. I just gave up on NIS.
xend is not started. Most probably we need a different version of python.
stable/testing
Both stable and testing give compiler errrors with some assembler code. According to the xen-users lists, post from 09 August 2005 [[1]] we have to use gcc-3.2. This version seems not to be available in any of the current Debian distributions, but is still available on backports:
/etc/apt/source.list: deb http://snapshot.debian.net/archive pool gcc-3.2 deb-src http://snapshot.debian.net/archive pool gcc-3.2
Did not try to install yet.
According to README, also gcc-3.3 should work. Maybe... but we already have gcc-3.3 installed and it did not help.
snapshots.debian.net has had a disk failure and it seems like the gcc-3.2 packages are lost:
Err http://snapshot.debian.net pool/gcc-3.2 gcc-3.2-base 1:3.2.3-9 404 Not Found Err http://snapshot.debian.net pool/gcc-3.2 cpp-3.2 1:3.2.3-9 404 Not Found Err http://snapshot.debian.net pool/gcc-3.2 gcc-3.2 1:3.2.3-9 404 Not Found
The debian package for xen uses the 2.0.6 xen kernel. It patches the Makefile to use gcc-3.4. We can do that by hand. It also need packages
- kernel-source-2.6.11
- kernel-patch-debian-2.6.11
None of the packages can be found. We can download the vanilla 2.6.11, but I guess the debian patches make 2.6.11 compile with gcc-3.4. So we somehow need to get these patches.
Hmm..., it seems like the kernel-source-2.6.11 pacakge should already contain the debian patches.
stable
Finally got the debian package for 2.6.11 kernel and patches from packages.debianbase.de. there seem to be also deb packagages for xen 2.0.7 (current stable version of xen), while the official debian packages are only for 2.0.6.
Even te debian patches don't fix the compile error for xen 2.0.6, linux kernel 2.6.11.
testing
According to a message on the xen-users list, xen testing 2.0.8 should use the 2.6.12 linux kernel. vanilla 2.6.12 compiles on debian testing (on veggie). Will try out to get the xen version to compile...
compiling xen kernels on fruit
Same as on veggie, but xen sources reside in /usr/src/xen
configure the domU kernel
cd /usr/src/xen/xen-2.0/linux-2.6.11-xenU make ARCH=xen menuconfig
configure the dom0 kernel
cd /usr/src/xen/xen-2.0/linux-2.6.11-xen0 make ARCH=xen menuconfig
compile the kernels
cd /usr/src/xen/xen-2.0/ make kernels
install kernels from /usr/src/xen/xen-2.0/dist/install/boot
cd /usr/src/xen/xen-2.0/dist/install/boot cp -d * /boot/
