XenServer 6.2 with Software RAID

Historically XenServer does not support software RAID out-the-box, and this is unchanged in the latest 6.2 release. We can convert it to RAID after installation.

First we set up the 2nd disk by copying the partition tables whilst enabling a degraded RAID on it, then copy data over with custom initrd.

sgdisk -R/dev/sdb /dev/sda  # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb
# Sleep 5 seconds here if you script this...
yes|mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing # Create md0 (root)
yes|mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing # Create md0 (swap)
yes|mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb3 missing # Create md0 (storage)
# Sleep 5 seconds here if you script this...
mkfs.ext3 /dev/md0 # Create root FS
mount /dev/md0 /mnt # Mount root FS
cp -xR --preserve=all / /mnt # Replicate root files
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
zcat initrd-`uname -r`-raid.img | cpio -i
sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
rm -f initrd-`uname -r`-raid.img
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
rm -f /mnt/boot/initrd-2.6-xen.img
ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-2.6-xen.img
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
cd /mnt && extlinux  --raid -i boot/
sgdisk /dev/sda --attributes=1:set:2
sgdisk /dev/sdb --attributes=1:set:2
cd && umount /dev/md0
sync
reboot

Now, make sure BIOS forces booting from 2nd disk. This is VERY important! After reboot, finish off:

sgdisk -R/dev/sda /dev/sdb  # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3  # If this command gives error, you need to forget/destroy an active SR first
mdadm --detail --scan > /etc/mdadm.conf
xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=[host_uuid] name-label="Local Storage" shared=false type=lvm

Before going into production, try booting from each disk (with the other removed from the boot priorities), then restore the boot priority to normal.

Be careful with Xenserver patches, especially any patch which requires a reboot – if the initrd image is rewritten (for example, kernel updated), you need to carefully rebuild the initrd for RAID support again which is NOT covered in this article.

28 Responses to XenServer 6.2 with Software RAID

  1. Andre Zapala says:

    Hi Randy,

    what size hard drive that was made to raid, it is possible to do on a 3TB disk? the GPT partition may be?

    hug

    1. Randy says:

      Xenserver uses GPT by default, 3TB is fine.

  2. Andre Zapala says:

    Hello again Randy,

    when I give the reboot and start the 2nd hard disk crashes in the xenserver screen before finalizing the boot process and followed all the steps correctly.

    Any idea on how to solve this?

    sorry for the english, I’m using google translator: D

    hug

  3. Randy says:

    Pay careful attention after the ‘cp -vxpR / /mnt’ command. Everything after that involves making it bootable, I think you missed a step either with FSTAB or rebuilding INITRD with raid support.

  4. Andre Zapala says:

    I followed all the steps until the REBOOT command and all worked, but not start the 2nd disk

  5. Andre Zapala says:

    # sfdisk –id –force /dev/sdb 1 fd

    WARNING: GPT (GUID Partition Table) detected on ‘/dev/sdb’! The util sfdisk doesn’t support GPT. Use GNU Parted.

    WARNING: GPT (GUID Partition Table) detected on ‘/dev/sdb’! The util sfdisk doesn’t support GPT. Use GNU Parted.

    Done
    //

    Then:
    # sgdisk -p /dev/sdb
    Invalid partition data!

  6. alessandro abram says:

    sfdisk –id /dev/sdb 1 fd # don’t run . I have tried sgdisk –new=3:16779264:DISK_SIZE-34 /dev/sdb# that runs.
    At the end at reboot, starting with 2ND disk the system dont start. How can I test my installation?
    All commands , apart sfdisk, seem fie. Any suggestion?
    Thank you very

  7. alessandro abram says:

    Sorry I read my post, its wrong: I tried :
    sgdisk –typecode=1:fd00 /dev/sdb
    sgdisk –typecode=2:fd00 /dev/sdb
    sgdisk –typecode=3:fd00 /dev/sdb
    that’s fine.

    1. Randy says:

      Thanks for the catch, I’ve updated the tutorial.

  8. federico says:

    anyone know what should i do if i want to update with xen patches, one of the patches is kernel related.

    is it ok after the update to execute this part of the above code to regenerate initrd or should i do something else, if anyone could help me or give me some instructions


    sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
    mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
    mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
    zcat initrd-`uname -r`-raid.img | cpio -i
    sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
    rm initrd-`uname -r`-raid.img
    find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
    rm -f /mnt/boot/initrd-2.6-xen.img
    ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-2.6-xen.img
    sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
    cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
    cd /mnt && extlinux --raid -i boot/

    1. Randy says:

      You need to replace every instance of `uname -r` with the new kernel version – because the uname command outputs the current (old) version. Also you don’t need either of the following:

      sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/etc/fstab # Update fstab for new RAID device

      cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb

      If you find a working set of commands, please post them.

  9. vasil says:

    for me working great.
    thank you randy!
    Next, i try some testing (changing disks, test failures eg…)
    have a nice day

  10. Lefty says:

    I think there is -f required in ‘rm initrd-`uname -r`-raid.img’

  11. Steven Schoch says:

    Instead of “cp -vxpR “, I suggest “cp -vxR –preserve=all”.
    Otherwise, the hard links will not be preserved, and more disk space on the root will be used.

    1. Randy says:

      Thanks for the tip. I’ve updated the article.

  12. Alex says:

    Was going along pretty nice, then I got to

    cd /mnt && extlinux –raid -i boot/

    which threw “Segmentation Fault”.

    Any idea what to do?

  13. Pablo says:

    What are the commands to forget and destroy an active storage repository?

    You reference it here:

    “mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first”

    Thanks in advance.

  14. Pablo says:

    I get the answare in this link


    [root@xenserver-01 ~]# xe sr-list
    uuid ( RO) : 4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    name-label ( RW): Local storage
    name-description ( RW):
    host ( RO): xenserver-01
    type ( RO): lvm
    content-type ( RO): user

    [root@xenserver-01 ~]# xe pbd-list sr-uuid=4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    uuid ( RO) : 15bbb4cf-fe30-2226-8315-ea311f49834d
    host-uuid ( RO): 748267e1-ecb7-4657-b655-bc1c1beefa7f
    sr-uuid ( RO): 4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    device-config (MRO): device: /dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBfa2921f3-020c0477-part3
    currently-attached ( RO): true

    [root@xenserver-01 ~]# xe pbd-unplug uuid=15bbb4cf-fe30-2226-8315-ea311f49834d
    [root@xenserver-01 ~]# xe pbd-destroy uuid=15bbb4cf-fe30-2226-8315-ea311f49834d
    [root@xenserver-01 ~]# xe sr-forget uuid=4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    [root@xenserver-01 ~]# xe sr-list

    Very good post. Thanks for that.

  15. Muchtall says:

    Anyone tried this on XenServer Creedence (RC) yet?

    I get this on bootup:

    raidautorun: RAID_AUTORUN failed: No such device.

    It looks like mkinitrd is pulling in the raid1 module, so I’m not sure what could be wrong here.

  16. Muchtall says:

    ..and I forgot to mention, I had to “modprobe md” before the “mdadm –create …” commands. You’ll need to add that to the process/script on Creedence.

  17. Muchtall says:

    So I found a workaround: You have to add mdassemble and mdadm.conf to your initrd, and replace the raidautorun commands with mdassemble. Here’s the modified script I just used to complete this setup on Creedence:


    sgdisk -R/dev/sdb /dev/sda # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
    sgdisk --typecode=1:fd00 /dev/sdb
    sgdisk --typecode=2:fd00 /dev/sdb
    sgdisk --typecode=3:fd00 /dev/sdb
    sleep 5 # Sleep 5 seconds here if you script this...
    modprobe md_mod
    yes|mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing # Create md0 (root)
    yes|mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing # Create md0 (swap)
    yes|mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb3 missing # Create md0 (storage)
    sleep 5 # Sleep 5 seconds here if you script this...
    mkfs.ext3 /dev/md0 # Create root FS
    mount /dev/md0 /mnt # Mount root FS
    cp -xR --preserve=all / /mnt # Replicate root files
    sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
    mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
    mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
    zcat initrd-`uname -r`-raid.img | cpio -i
    # This doesn't work on Creedence...
    # sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
    # So doing this instead...
    mdadm --detail --scan > /etc/mdadm.conf
    cp -p /etc/mdadm.conf etc/mdadm.conf
    cp -p /sbin/mdassemble.static bin/mdassemble
    sed -i 's/raidautorun \/dev\/md0/mdassemble/' init
    rm -f initrd-`uname -r`-raid.img
    find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
    rm -f /mnt/boot/initrd-3.10-xen.img
    ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-3.10-xen.img
    sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
    cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
    cd /mnt && extlinux --raid -i boot/
    sgdisk /dev/sda --attributes=1:set:2
    sgdisk /dev/sdb --attributes=1:set:2
    cd && umount /dev/md0
    sync
    reboot
    # Tell BIOS to use disk B

    #After reboot to disk B...
    sgdisk -R/dev/sda /dev/sdb # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
    sleep 5 # Sleep 5 seconds here if you script this...
    mdadm -a /dev/md0 /dev/sda1
    mdadm -a /dev/md1 /dev/sda2
    mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first
    xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F": " '{print $2}'` name-label="Local Storage" shared=false type=lvm

    # Watch rebuild progress and wait until no arrays are rebuilding before proceeding with any reboot
    watch "mdadm --detail /dev/md* | grep rebuild"

    Hope this helps someone else out there!

  18. Muchtall says:

    Looks like XenServer “Creedence” just became XenServer 6.5! http://xenserver.org/open-source-virtualization-download.html

  19. Karsten Behrens says:

    On the final 6.5 release, you need to change some more parts to get it working:

    – use “–metadata=0.90” in all mdadm create commands, seems like extlinux doesn’t like the 1.2 metadata that’s default now

    – no need to change the “init” file anymore, also no need to copy mdadm.conf or mdassemble (which doesn’t even exist in /sbin anymore). Instead of raidautorun or mdassemble, the init file now contains this by default:

    echo Scanning and configuring dmraid supported devices
    mdadm -As --auto=yes

    So I guess we could skip the whole part with unpacking, altering and repacking the initrd. The mkinitrd does a good job.

  20. Olli says:

    @ Muchtall says: January 14, 2015 at 4:26 pm

    First thanks a lot where for your post relating Creedence.

    I set up XenServer 6.5, followed your script,
    but my system hasn’t got a “/sbin/mdassemble.static”
    so “cp -p /sbin/mdassemble.static bin/mdassemble” fails.

    I tried copying the mdadm.conf to /mnt/etc/. and found “mdadm.static”, used it, but while switching to disk B there is a Boot failure
    Sorry for this stupid question, but where you got the file from or do you know another workaround for XS 6.5?

  21. Olli says:

    @ Karsten Behrens says:
    January 22, 2015 at 8:25 pm

    Thanks a lot for your perfect hint. I saw the advise by the system while creating the raid devices and no md_mod was loaded, but …
    Anyway, the following steps are working (mashup of posts from Muchtall and Karsten Behrens) and I’am very lucky to cross it from the list …

    # Prepare /dev/sdb
    sgdisk –zap-all /dev/sdb
    sgdisk –mbrtogpt –clear /dev/sdb
    sgdisk -R/dev/sdb /dev/sda # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
    sleep 5 # Sleep 5 seconds here if you script this…
    sgdisk –typecode=1:fd00 /dev/sdb
    sgdisk –typecode=2:fd00 /dev/sdb
    sgdisk –typecode=3:fd00 /dev/sdb
    sleep 5 # Sleep 5 seconds here if you script this…
    modprobe md_mod # load raid, because it isn’t load by default (XS6.5 only)
    yes|mdadm –create /dev/md0 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb1 missing # Create md0 (root)
    yes|mdadm –create /dev/md1 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb2 missing # Create md0 (swap)
    yes|mdadm –create /dev/md2 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb3 missing # Create md0 (storage)
    sleep 5 # Sleep 5 seconds here if you script this…
    mkfs.ext3 /dev/md0 # Create root FS
    mount /dev/md0 /mnt # Mount root FS
    cp -xR –preserve=all / /mnt # Replicate root files
    sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/etc/fstab # Update fstab for new RAID device
    mount –bind /dev /mnt/dev
    mount -t sysfs none /mnt/sys
    mount -t proc none /mnt/proc
    chroot /mnt /sbin/extlinux –install /boot
    dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb
    chroot /mnt
    mkinitrd -v -f –theme=/usr/share/splash –without-multipath /boot/initrd-`uname -r`.img `uname -r`
    exit
    sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/boot/extlinux.conf # Update extlinux for new RAID device
    cd /mnt && extlinux –raid -i boot/
    sgdisk /dev/sdb –attributes=1:set:2
    cd && umount /dev/md0
    sync
    reboot
    # Tell BIOS to use disk B
    #After reboot to disk B…
    sgdisk -R/dev/sda /dev/sdb # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
    sgdisk /dev/sda –attributes=1:set:2
    sleep 5 # Sleep 5 seconds here if you script this…
    mdadm -a /dev/md0 /dev/sda1
    mdadm -a /dev/md1 /dev/sda2
    mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first
    xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F”: ” ‘{print $2}’` name-label=”Local Storage” shared=false type=lvm
    # Watch rebuild progress and wait until no arrays are rebuilding before proceeding with any reboot
    watch “mdadm –detail /dev/md* | grep rebuild”

    Have a nice try …

  22. Olli says:

    Of course there is missing the creating of mdadm.conf

    So add an
    mdadm –detail –scan > /mnt/etc/mdadm.conf

    Sorry, nobody is perfect …

  23. Ilya says:

    Olli says:
    January 27, 2015 at 9:21 am

    Hi!
    Thx for instruction! But something missing. Even adding “mdadm –detail –scan > /mnt/etc/mdadm.conf”, before rebooting to disc B I Can’t unmount.
    # cd && umount /dev/md0
    umount failed. Device /mnt is busy.

    Trying “umount -l” but startup fails on Disc B.

  24. Nick says:

    Thanks for this, it finally got my RAID setup working for xenserver 6.5. The last command, xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F”: ” ‘{print $2}’` name-label=”Local Storage” shared=false type=lvm
    didn’t work for me, it returned an error “Unable to contact server. Please check server and port settings.”
    I removed the backtick code and just manually placed the UUID of my xenserver host and it worked great. Thanks!