XenServer 6.2 with Software RAID

28 Responses to XenServer 6.2 with Software RAID

1. Andre Zapala says:

   March 28, 2014 at 12:18 pm

   Hi Randy,

   what size hard drive that was made to raid, it is possible to do on a 3TB disk? the GPT partition may be?

   hug
   1. Randy says:

      March 28, 2014 at 12:20 pm

      Xenserver uses GPT by default, 3TB is fine.
2. Andre Zapala says:

   March 28, 2014 at 1:27 pm

   Hello again Randy,

   when I give the reboot and start the 2nd hard disk crashes in the xenserver screen before finalizing the boot process and followed all the steps correctly.

   Any idea on how to solve this?

   sorry for the english, I’m using google translator: D

   hug
3. Randy says:

   March 28, 2014 at 1:48 pm

   Pay careful attention after the ‘cp -vxpR / /mnt’ command. Everything after that involves making it bootable, I think you missed a step either with FSTAB or rebuilding INITRD with raid support.
4. Andre Zapala says:

   March 28, 2014 at 1:52 pm

   I followed all the steps until the REBOOT command and all worked, but not start the 2nd disk
5. Andre Zapala says:

   March 28, 2014 at 2:12 pm

   # sfdisk –id –force /dev/sdb 1 fd

   WARNING: GPT (GUID Partition Table) detected on ‘/dev/sdb’! The util sfdisk doesn’t support GPT. Use GNU Parted.

   WARNING: GPT (GUID Partition Table) detected on ‘/dev/sdb’! The util sfdisk doesn’t support GPT. Use GNU Parted.

   Done
   //

   Then:
   # sgdisk -p /dev/sdb
   Invalid partition data!
6. alessandro abram says:

   April 26, 2014 at 2:32 am

   sfdisk –id /dev/sdb 1 fd # don’t run . I have tried sgdisk –new=3:16779264:DISK_SIZE-34 /dev/sdb# that runs.
   At the end at reboot, starting with 2ND disk the system dont start. How can I test my installation?
   All commands , apart sfdisk, seem fie. Any suggestion?
   Thank you very
7. alessandro abram says:

   April 26, 2014 at 2:36 am

   Sorry I read my post, its wrong: I tried :
   sgdisk –typecode=1:fd00 /dev/sdb
   sgdisk –typecode=2:fd00 /dev/sdb
   sgdisk –typecode=3:fd00 /dev/sdb
   that’s fine.
   1. Randy says:

      April 28, 2014 at 11:16 am

      Thanks for the catch, I’ve updated the tutorial.
8. federico says:

   June 6, 2014 at 8:11 am

   anyone know what should i do if i want to update with xen patches, one of the patches is kernel related.

   is it ok after the update to execute this part of the above code to regenerate initrd or should i do something else, if anyone could help me or give me some instructions

   
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
zcat initrd-`uname -r`-raid.img | cpio -i
sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
rm initrd-`uname -r`-raid.img
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
rm -f /mnt/boot/initrd-2.6-xen.img
ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-2.6-xen.img
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
cd /mnt && extlinux --raid -i boot/

   1. Randy says:

      June 6, 2014 at 8:23 am

      You need to replace every instance of `uname -r` with the new kernel version – because the uname command outputs the current (old) version. Also you don’t need either of the following:

      sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/etc/fstab # Update fstab for new RAID device

      cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb

      If you find a working set of commands, please post them.
9. vasil says:

   June 11, 2014 at 9:37 am

   for me working great.
   thank you randy!
   Next, i try some testing (changing disks, test failures eg…)
   have a nice day
10. Lefty says:

    July 6, 2014 at 9:10 am

    I think there is -f required in ‘rm initrd-`uname -r`-raid.img’
11. Steven Schoch says:

    September 3, 2014 at 7:02 pm

    Instead of “cp -vxpR “, I suggest “cp -vxR –preserve=all”.
    Otherwise, the hard links will not be preserved, and more disk space on the root will be used.
    1. Randy says:

       September 19, 2014 at 1:08 pm

       Thanks for the tip. I’ve updated the article.
12. Alex says:

    October 20, 2014 at 8:32 pm

    Was going along pretty nice, then I got to

    cd /mnt && extlinux –raid -i boot/

    which threw “Segmentation Fault”.

    Any idea what to do?
13. Pablo says:

    December 1, 2014 at 10:46 pm

    What are the commands to forget and destroy an active storage repository?

    You reference it here:

    “mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first”

    Thanks in advance.
14. Pablo says:

    December 1, 2014 at 10:57 pm

    I get the answare in this link

    
[root@xenserver-01 ~]# xe sr-list
uuid ( RO) : 4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
name-label ( RW): Local storage
name-description ( RW):
host ( RO): xenserver-01
type ( RO): lvm
content-type ( RO): user

    [root@xenserver-01 ~]# xe pbd-list sr-uuid=4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    uuid ( RO) : 15bbb4cf-fe30-2226-8315-ea311f49834d
    host-uuid ( RO): 748267e1-ecb7-4657-b655-bc1c1beefa7f
    sr-uuid ( RO): 4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    device-config (MRO): device: /dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VBfa2921f3-020c0477-part3
    currently-attached ( RO): true

    [root@xenserver-01 ~]# xe pbd-unplug uuid=15bbb4cf-fe30-2226-8315-ea311f49834d
    [root@xenserver-01 ~]# xe pbd-destroy uuid=15bbb4cf-fe30-2226-8315-ea311f49834d
    [root@xenserver-01 ~]# xe sr-forget uuid=4cf3cc3b-b8d4-db4a-1be1-7ff403811af7
    [root@xenserver-01 ~]# xe sr-list

    Very good post. Thanks for that.
15. Muchtall says:

    January 14, 2015 at 12:46 pm

    Anyone tried this on XenServer Creedence (RC) yet?

    I get this on bootup:

    raidautorun: RAID_AUTORUN failed: No such device.

    It looks like mkinitrd is pulling in the raid1 module, so I’m not sure what could be wrong here.
16. Muchtall says:

    January 14, 2015 at 12:49 pm

    ..and I forgot to mention, I had to “modprobe md” before the “mdadm –create …” commands. You’ll need to add that to the process/script on Creedence.
17. Muchtall says:

    January 14, 2015 at 4:26 pm

    So I found a workaround: You have to add mdassemble and mdadm.conf to your initrd, and replace the raidautorun commands with mdassemble. Here’s the modified script I just used to complete this setup on Creedence:

    
sgdisk -R/dev/sdb /dev/sda # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb
sleep 5 # Sleep 5 seconds here if you script this...
modprobe md_mod
yes|mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing # Create md0 (root)
yes|mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing # Create md0 (swap)
yes|mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb3 missing # Create md0 (storage)
sleep 5 # Sleep 5 seconds here if you script this...
mkfs.ext3 /dev/md0 # Create root FS
mount /dev/md0 /mnt # Mount root FS
cp -xR --preserve=all / /mnt # Replicate root files
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
zcat initrd-`uname -r`-raid.img | cpio -i
# This doesn't work on Creedence...
# sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
# So doing this instead...
mdadm --detail --scan > /etc/mdadm.conf
cp -p /etc/mdadm.conf etc/mdadm.conf
cp -p /sbin/mdassemble.static bin/mdassemble
sed -i 's/raidautorun \/dev\/md0/mdassemble/' init
rm -f initrd-`uname -r`-raid.img
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
rm -f /mnt/boot/initrd-3.10-xen.img
ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-3.10-xen.img
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
cd /mnt && extlinux --raid -i boot/
sgdisk /dev/sda --attributes=1:set:2
sgdisk /dev/sdb --attributes=1:set:2
cd && umount /dev/md0
sync
reboot
# Tell BIOS to use disk B

    #After reboot to disk B...
    sgdisk -R/dev/sda /dev/sdb # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
    sleep 5 # Sleep 5 seconds here if you script this...
    mdadm -a /dev/md0 /dev/sda1
    mdadm -a /dev/md1 /dev/sda2
    mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first
    xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F": " '{print $2}'` name-label="Local Storage" shared=false type=lvm

    # Watch rebuild progress and wait until no arrays are rebuilding before proceeding with any reboot
watch "mdadm --detail /dev/md* | grep rebuild"


    Hope this helps someone else out there!
18. Muchtall says:

    January 14, 2015 at 5:45 pm

    Looks like XenServer “Creedence” just became XenServer 6.5! http://xenserver.org/open-source-virtualization-download.html
19. Karsten Behrens says:

    January 22, 2015 at 8:25 pm

    On the final 6.5 release, you need to change some more parts to get it working:

    – use “–metadata=0.90” in all mdadm create commands, seems like extlinux doesn’t like the 1.2 metadata that’s default now

    – no need to change the “init” file anymore, also no need to copy mdadm.conf or mdassemble (which doesn’t even exist in /sbin anymore). Instead of raidautorun or mdassemble, the init file now contains this by default:

    echo Scanning and configuring dmraid supported devices
mdadm -As --auto=yes

    So I guess we could skip the whole part with unpacking, altering and repacking the initrd. The mkinitrd does a good job.
20. Olli says:

    January 25, 2015 at 11:42 am

    @ Muchtall says: January 14, 2015 at 4:26 pm

    First thanks a lot where for your post relating Creedence.

    I set up XenServer 6.5, followed your script,
    but my system hasn’t got a “/sbin/mdassemble.static”
    so “cp -p /sbin/mdassemble.static bin/mdassemble” fails.

    I tried copying the mdadm.conf to /mnt/etc/. and found “mdadm.static”, used it, but while switching to disk B there is a Boot failure
    Sorry for this stupid question, but where you got the file from or do you know another workaround for XS 6.5?
21. Olli says:

    January 27, 2015 at 9:21 am

    @ Karsten Behrens says:
    January 22, 2015 at 8:25 pm

    Thanks a lot for your perfect hint. I saw the advise by the system while creating the raid devices and no md_mod was loaded, but …
    Anyway, the following steps are working (mashup of posts from Muchtall and Karsten Behrens) and I’am very lucky to cross it from the list …

    # Prepare /dev/sdb
    sgdisk –zap-all /dev/sdb
    sgdisk –mbrtogpt –clear /dev/sdb
    sgdisk -R/dev/sdb /dev/sda # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
    sleep 5 # Sleep 5 seconds here if you script this…
    sgdisk –typecode=1:fd00 /dev/sdb
    sgdisk –typecode=2:fd00 /dev/sdb
    sgdisk –typecode=3:fd00 /dev/sdb
    sleep 5 # Sleep 5 seconds here if you script this…
    modprobe md_mod # load raid, because it isn’t load by default (XS6.5 only)
    yes|mdadm –create /dev/md0 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb1 missing # Create md0 (root)
    yes|mdadm –create /dev/md1 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb2 missing # Create md0 (swap)
    yes|mdadm –create /dev/md2 –level=1 –raid-devices=2 –metadata=0.90 /dev/sdb3 missing # Create md0 (storage)
    sleep 5 # Sleep 5 seconds here if you script this…
    mkfs.ext3 /dev/md0 # Create root FS
    mount /dev/md0 /mnt # Mount root FS
    cp -xR –preserve=all / /mnt # Replicate root files
    sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/etc/fstab # Update fstab for new RAID device
    mount –bind /dev /mnt/dev
    mount -t sysfs none /mnt/sys
    mount -t proc none /mnt/proc
    chroot /mnt /sbin/extlinux –install /boot
    dd if=/mnt/usr/share/syslinux/gptmbr.bin of=/dev/sdb
    chroot /mnt
    mkinitrd -v -f –theme=/usr/share/splash –without-multipath /boot/initrd-`uname -r`.img `uname -r`
    exit
    sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/boot/extlinux.conf # Update extlinux for new RAID device
    cd /mnt && extlinux –raid -i boot/
    sgdisk /dev/sdb –attributes=1:set:2
    cd && umount /dev/md0
    sync
    reboot
    # Tell BIOS to use disk B
    #After reboot to disk B…
    sgdisk -R/dev/sda /dev/sdb # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
    sgdisk /dev/sda –attributes=1:set:2
    sleep 5 # Sleep 5 seconds here if you script this…
    mdadm -a /dev/md0 /dev/sda1
    mdadm -a /dev/md1 /dev/sda2
    mdadm -a /dev/md2 /dev/sda3 # If this command gives error, you need to forget/destroy an active SR first
    xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F”: ” ‘{print $2}’` name-label=”Local Storage” shared=false type=lvm
    # Watch rebuild progress and wait until no arrays are rebuilding before proceeding with any reboot
    watch “mdadm –detail /dev/md* | grep rebuild”

    Have a nice try …
22. Olli says:

    January 27, 2015 at 9:37 am

    Of course there is missing the creating of mdadm.conf

    So add an
    mdadm –detail –scan > /mnt/etc/mdadm.conf

    Sorry, nobody is perfect …
23. Ilya says:

    February 25, 2015 at 8:31 am

    Olli says:
    January 27, 2015 at 9:21 am

    Hi!
    Thx for instruction! But something missing. Even adding “mdadm –detail –scan > /mnt/etc/mdadm.conf”, before rebooting to disc B I Can’t unmount.
    # cd && umount /dev/md0
    umount failed. Device /mnt is busy.

    Trying “umount -l” but startup fails on Disc B.
24. Nick says:

    February 28, 2015 at 12:34 am

    Thanks for this, it finally got my RAID setup working for xenserver 6.5. The last command, xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=`xe host-list name-label=`hostname -s` params=uuid | head -1 | awk -F”: ” ‘{print $2}’` name-label=”Local Storage” shared=false type=lvm
    didn’t work for me, it returned an error “Unable to contact server. Please check server and port settings.”
    I removed the backtick code and just manually placed the UUID of my xenserver host and it worked great. Thanks!