Xen 4 and Libvirt From Source on CentOS 6

Posted on by Randy

Install some prerequisites:

yum groupinstall "Development Libraries" "Development Tools"

yum install mercurial python-devel dev86 iasl ncurses-devel ncurses \
glib2-devel glib2 openssl-devel yajl-devel libuuid-devel libuuid \
pciutils-devel pciutils texinfo kernel-xen bridge-utils  gnutls gnutls-devel \
libxml2 libxml2-devel libnl libnl-devel libxslt libxslt-devel pygtk2 xorg-x11-xauth \
xorg-x11-fonts* device-mapper* gnome-python2-gconf pygtk2-libglade dbus-x11 \
gtk-vnc-python netcf netcf-devel netcf-libs vte vte-devel

Pull the source code and build Xen.

cd /usr/src
hg clone -r RELEASE-4.1.2 http://xenbits.xensource.com/xen-4.1-testing.hg
cd xen-4.1-testing.hg/
make dist -j4
make install

Build and install Libvirt management tools.

cd /usr/src
wget http://libvirt.org/sources/libvirt-0.9.12.tar.gz
tar -zxf libvirt-0.9.12.tar.gz
cd libvirt-0.9.12/
./configure --prefix=/usr
make -j4
make install
ldconfig

cd /usr/src
wget http://virt-manager.org/download/sources/virtinst/virtinst-0.600.1.tar.gz
tar -zxf virtinst-0.600.1.tar.gz
cd virtinst-0.600.1/
python setup.py install

cd /usr/src
wget http://virt-manager.org/download/sources/virt-manager/virt-manager-0.9.1.tar.gz
tar -zxf virt-manager-0.9.1.tar.gz
cd virt-manager-0.9.1/

Install the xen-enabled Dom0 kernel:

yum install http://au1.mirror.crc.id.au/repo/kernel-xen-release-6-3.noarch.rpm
yum install kernel-xen

Edit /etc/grub.conf, make changes to the first ‘xen.gz’ line and change the next two lines to start with ‘module’.

       kernel /xen.gz dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 dom0_vcpus_pin
        module /vmlinuz-2.6.32.57-2.el6xen.x86_64 ro root=UUID=efff8fe3-523b-4620-a01f-d948cd43c49a rd_MD_UUID=836f9712:2e50a8a6:b1eabaa6:19f7ff34 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto rd_MD_UUID=0698abc6:b72a8b69:f1e3b4e8:2a9fc55f rd_NO_LVM rd_NO_DM rhgb quiet
        module /initramfs-2.6.32.57-2.el6xen.x86_64.img

Setup the init scripts.

chkconfig --add xencommons
chkconfig --add xend
chkconfig --add xen-watchdog
chkconfig --add libvirtd
chkconfig --add libvirt-domains
chkconfig xencommons on
chkconfig xend on
chkconfig xen-watchdog on
chkconfig libvirtd on
chkconfig libvirt-domains on

Now it’s time to reboot, and manage your domains with virt-install and/or virt-manager.

Some issues that I haven’t been able to solve.

1. CentOS paravirtual domains hang during bootup with Kudzu due to the VNC framebuffer. They also fail to poweroff after shutdown and hang with 100% CPU usage. This doesn’t appear to be a libvirt specific issue; I could replicate it with pure Xen as well. To work around this, disable VNC framebuffer using virt-install with –nographics as follows. 

virt-install -n centos -r 2048 --file /dev/vg0/centos --os-variant=rhel6 -\
-nographics -p -l http://mirror.fastserv.com/centos/6/os/x86_64/ -b virbr0 -d

Unfortunately, now virt-manager doesn’t know how to access text console. You have to use virsh [domain] console from the command line from now on.

2. Libvirt Xen driver does not support managedsave and ends up terminating the DomU ungracefully when Dom0 reboots. If you happen to have DomU on a MD RAID-backed LVM, this will crash Dom0 with a kernel oops as MD attempts to go read-only with domains still attached. If anyone knows a workaround I am keen to this. Until then, I really can’t use this setup in any production environment.

3. Trying to work around this, if you change /etc/sysconfig/libvirt-domains to shutdown the DomU’s instead of trying to (unsuccessfully) save them, libvirt attempts to shutdown Domain-0 and hangs the shutdown process until the timeout (default 300 seconds) is reached.

4. CentOS 6.2 seems to have a buggy e1000e driver (at least when used on an X9SCL+-F motherboard) and at one point went completely offline requiring a hard power cycle. Research reveals I’m not the only one with issues with this combination.

My final thoughts are that Xen+Libvirt are certainly not a production ready combination. Every time I thought I solved a problem I uncovered several more and finally gave up after (3). Unfortunately, I don’t have enough time to work these bugs out and had to use Ubuntu+KVM in a crunch to get things done.

Comments are closed.