Debian PXE boot image from scratch

After facing issue after issue with Idera/R1soft’s PXE and CD-ROM boot media — specifically, ancient kernels which are unable to obtain network connectivity on modern hardware, I decided to roll my own PXE boot image including the R1soft agent and a few other tools with the latest Debian stable. It turned out to be easier than I expected — kudos to the debian-live project.

Boot into a debian Live-CD and set up the bootstrap environment. I’m using 7.6.0 (latest stable at this time).

apt-get install debootstrap squashfs-tools
mkdir -p work/chroot
cd work

If you’re creating the image from scratch, use the following commands.

debootstrap --arch=amd64 stable chroot
cp /etc/network/interfaces chroot/etc/network/interfaces # Needed for network connectivity

If you are re-working an existing image, you can import it instead.

wget http://webserver/debian-live/filesystem.squashfs
unsquashfs -f -d chroot filesystem.squashfs
rm -f filesystem.squashfs

Now we prep the image and chroot.

cp /etc/resolv.conf chroot/etc/resolv.conf # Needed for network connectivity
chroot chroot
mount none -t proc /proc
mount none -t sysfs /sys
mount none -t devpts /dev/pts
export HOME=/root
export LC_ALL=C

Now you can do whatever you want to the image by installing packages and modifying configurations. You need to install the kernel at a minimum if you want to load modules after boot such as raid. You can install kernel headers and compile 3rd party modules like the r1soft cdp agent. This is a minimal image, so don’t forget basic filesystem utilities like mdadm if you’re installing the r1soft agent; I learned the hard way and had to re-work the image several times. I enabled auto-login on tty1 through tty3 by installing mingetty and modifying inittab. I also like to redirect syslog to tty4 and disable any console logging on tty1 – tty3.

When you think you’re done, clean up and exit the chroot.

apt-get clean
rm -rf /tmp/*
rm /etc/resolv.conf
umount -lf /proc
umount -lf /sys
umount -lf /dev/pts
exit

Compress and package the new filesystem image.

mksquashfs chroot filesystem.squashfs -e boot

Send it off to your HTTP webserver:

scp filesystem.squashfs user@webserver:/var/www/html/debian-live/.

And a working pxelinux.cfg entry after we’ve got the kernel and initrd in the correct place on the tftp server:

LABEL deblive
        KERNEL /debian-live/debian-live-7.6.0-amd64-standard.vmlinuz
        APPEND initrd=/debian-live/debian-live-7.6.0-amd64-standard.initrd.img dhcp ethdevice=eth0,eth1 boot=live fetch=http://webserver/debian-live/filesystem.squashfs

Grub with Xen Guests

In order for grub to ‘see’ virtual disks, you have to tell it about them with the ‘device’ command:

# grub
grub> device (hd0) /dev/xvda
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

Keep in mind, installing grub on the MBR is not always necessary with Xen PV guests, but is highly useful for V2P conversions.

XenServer 6.2 with Software RAID

Historically XenServer does not support software RAID out-the-box, and this is unchanged in the latest 6.2 release. We can convert it to RAID after installation.

First we set up the 2nd disk by copying the partition tables whilst enabling a degraded RAID on it, then copy data over with custom initrd.

sgdisk -R/dev/sdb /dev/sda  # Replicate partion table from /dev/sda to /dev/sdb with unique identifier
sgdisk --typecode=1:fd00 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb
# Sleep 5 seconds here if you script this...
yes|mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing # Create md0 (root)
yes|mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing # Create md0 (swap)
yes|mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sdb3 missing # Create md0 (storage)
# Sleep 5 seconds here if you script this...
mkfs.ext3 /dev/md0 # Create root FS
mount /dev/md0 /mnt # Mount root FS
cp -xR --preserve=all / /mnt # Replicate root files
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab # Update fstab for new RAID device
mkdir /mnt/root/initrd-raid && cd /mnt/root/initrd-raid
mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
zcat initrd-`uname -r`-raid.img | cpio -i
sed -i 's/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/' init
rm -f initrd-`uname -r`-raid.img
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img
rm -f /mnt/boot/initrd-2.6-xen.img
ln -s initrd-`uname -r`-raid.img /mnt/boot/initrd-2.6-xen.img
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/boot/extlinux.conf
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
cd /mnt && extlinux  --raid -i boot/
sgdisk /dev/sda --attributes=1:set:2
sgdisk /dev/sdb --attributes=1:set:2
cd && umount /dev/md0
sync
reboot

Now, make sure BIOS forces booting from 2nd disk. This is VERY important! After reboot, finish off:

sgdisk -R/dev/sda /dev/sdb  # Replicate partion table from /dev/sdb to /dev/sda with unique identifier
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3  # If this command gives error, you need to forget/destroy an active SR first
mdadm --detail --scan > /etc/mdadm.conf
xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=[host_uuid] name-label="Local Storage" shared=false type=lvm

Before going into production, try booting from each disk (with the other removed from the boot priorities), then restore the boot priority to normal.

Be careful with Xenserver patches, especially any patch which requires a reboot – if the initrd image is rewritten (for example, kernel updated), you need to carefully rebuild the initrd for RAID support again which is NOT covered in this article.

EXT and LVM Local Storage in XenServer

On a new XenServer deployment EXT storage is default. Although EXT is nice (ability to manipulate raw VHD files and use sparse storage), LVM is faster and more stable. Here is how to switch between storage modes (be sure to back up all data first, as this will wipe everything in the repo):

Use ‘xe sr-list’ and ‘xe pbd-list’ to find your local storage SR and it’s PBD. Also note the physical device number / partition in the PBD, so you can re-create it on the same device later on.

Unplug the active SR’s PBD (physical block device):

xe pbd-unplug uuid=<pbd_uuid>

Destroy the SR:

xe sr-destroy uuid=<sr_uuid>

Create a new SR of the specified type and location:

xe sr-create content-type=user device-config:device=/dev/<device> host-uuid=<host_uuid> name-label="Local Storage" shared=false type=<lvm|ext>

You can verify the SR again with ‘xe sr-list’ and it should already be in XenCenter as the default.

Removing RAID metadata

Sometimes a hardware RAID controller or fakeraid (BIOS) can leave metadata that makes it impossible to install Windows or Linux, or it installs correctly, but causes a kernel panic or a 0xb7 blue screen error on the first boot. The only method I could find to delete the metadata *quickly* is to zero out the last 512KB of data on the disk using the following command:

dd if=/dev/zero of=$YOUR_DEV bs=512 seek=$(( $(blockdev --getsz $YOUR_DEV) - 1024 )) count=1024

Replace $YOUR_DEV with the physical device, such as /dev/sda

You could just zero the whole disk, but that could take hours. This command executes in less than a second.

Syncing User Filesystems Between Two Cpanel Servers

After doing a WHM multi-account copy, you may want to refresh the users home directories (including email and web files). This one liner uses rsync over SSH to do a quick sync of only changed files, emails, and website files. Run these commands on the NEW server.

First you need to create a list of domains in a text file ‘domains.txt’. Replace 1.2.3.4 with the remote hostname or IP.

while read i ; do rsync -aHxv root@1.2.3.4:/home/$(/scripts/whoowns $i)/ /home/$(/scripts/whoowns $i) ; done < domains.txt

Alternatively, you can use a list of users instead of domain names (use this method if there are multiple domains per user):

while read i ; do rsync -aHxv root@1.2.3.4:/home/$i/ /home/$i ; done < users.txt

Syncing All MySQL Databases Between two Cpanel Servers

After you do a WHM multi-site transfer, you may want to resync all the MySQL databases before doing a DNS change. Here’s a quick and easy way to sync each Cpanel user’s databases on the new Cpanel server. After manually setting up a remote mysql root user on the original server (and opening a firewall port if needed), run this command on the NEW server.

cat /var/cpanel/databases/dbindex.db | grep -E ': .+' | awk '{FS=": ";print $1}' |[ \t]+|:)/, "")};1' | xargs -L 1 /root/sync_db.sh <remote_host> <remote_root_pass>

If you only want to sync (or resync) one or two databases, you can copy /var/cpanel/databases/dbindex.db to a different location and edit it as needed.

Here’s the bash sync script required for this command:

http://djlab.com/stuff/sync_db.sh