XenServer recovery – XAPI won’t start

If running XenServer on a RAID array with writeback (controller cache) enabled and no battery backup, don’t lose power. If you do, you will discover the state.db is corrupt and XAPI fails to start. Everything is down, Dom0 lost it’s network config, you’re screwed. Maybe. This tutorial assumes you’ve got a metadata backup, all your VDI’s on LVM local storage, and the server is not part of a pool. If you meet this criteria, you can be back up in just a few minutes.

First, remove the corrupt state.db and restart XAPI:

[root@xen ~]# mv /var/xapi/state.db /var/xapi/state.db.bad
[root@xen ~]# xe-toolstack-restart

Reboot the server again for safe measure, then make sure you can see the XE Console. Reconfigure the network as needed.

Look for your orphaned SR and obtain it’s UUID (the UUID is after ‘VG_Xenstorage-‘)

[root@xen ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3   <--- Take note of this for later
  VG Name               VG_XenStorage-768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09   <--- Use this UUID

  PV Size               3.63 TB / not usable 9.14 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              951549
  Free PE               337665
  Allocated PE          613884
  PV UUID               QZ2vnb-1rDi-sFKD-6ofg-czgs-BYjn-u6dO6x

Using the UUID in the VG Name above, reintroduce your storage:

[root@xen ~]# xe sr-introduce uuid=768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09 type=lvm name-label="Primary Storage" content-type=user

Take note of the sr-uuid result of the previous command, you need to plug it in next. Now look for the disk ID so we can create the VBD. Pro-tip – press TAB after host-uuid= and it’ll auto complete.

[root@xen ~]# ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root  9 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f -> ../../sda
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part3 -> ../../sda3 <---- This one!

[root@xen ~]# xe pbd-create host-uuid=c7b099d9-1d50-402f-ae3b-1a52ca811a8f sr-uuid=768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09 device-config:device=/dev/disk/by-id/scsi-3600605b008f8a2501b80cee707491e0f-part3

Take note of the new PBD UUID and plug in the new PBD:

[root@xen ~]# xe pbd-plug uuid=9d657c71-7e85-ace8-719c-1d20890fce59

Now go into XE Console, search for and restore the metadata backup. You have automatic metadata backups enabled, right?

NOTE: If you are running tagged VLAN’s, before starting any VMs use XenCenter to straighten out the VLAN config on the host network before starting any VM’s. You will need to edit each one and assign to the correct interface.

Good luck.

Magento2 – Upgrade using Composer

This is the preferred, Composer method for upgrading Magento 2.X.X between minor versions. Replace X.X with the minor version numbers of the target version. The last line is optional, use it only if you’re in the middle of a migration and need to continue to be able to sync data with a 1.x site. If composer fails, this usually means you manually installed magento via GitHub or a tarball. You should probably go back and reinstall magento via Composer to avoid a world of hurt down the road.

php bin/magento maintenance:enable
composer require magento/product-community-edition 2.X.X --no-update
composer update
rm -rf var/di var/generation
php bin/magento cache:clean
php bin/magento cache:flush
php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento indexer:reindex
php bin/magento maintenance:disable
composer require magento/data-migration-tool:2.X.X

Windows Vista through Windows 10 Boot Repair

After cloning a larger HDD to a smaller SSD, you will most likely face a blinking cursor or other boot issues.   The following steps should get you back on track from a rescue command prompt:

bootrec.exe /fixmbr
bootsect.exe /nt60 all /force
bootrec.exe /rebuildbcd

If you are getting integrity check errors on winload.exe, you can try this:

bcdedit.exe /set {bootmgr} nointegritychecks ON
bcdedit.exe /set {default} nointegritychecks ON 

If you are having registry issues (complaining that it’s corrupt or missing), you can try the following (assuming your system drive is C):

bcdedit.exe /set {bootmgr} device partition=C:
bcdedit.exe /set {default} device partition=C:

GL-MT300A tricks – WIFI Client Bridge

In a search to replace several NetGear WNCE2001 devices which had over time, proven themselves to be so unreliable they would literally kill entire wireless AND wired networks by randomly hijacking DHCP. I tried many devices — some either couldn’t bridge at all, others ‘worked’ but had such terrible range or connection quality they were practically useless, others were simply too large, the list goes on.

Almost ready to give up on the project, I discovered the GL-MT300A – a tiny Linux / OpenWRT powered cube for around $35.   The list of built-in features is impressive.   But I need it for only one thing – a small, USB powered wifi client-mode bridge.

3154ku5uvqlI had high hopes – given the fact it ran OpenWRT along with the LUCI web interface – that I’d be able to do practically whatever I want through a nice GUI.

Unfortunately I was unable to create a working bridged repeater setup due to a limitation in this particular (MediaTek) hardware.   Apparently, only Broadcom hardware can natively bridge between wired and wireless networks.

With some google-fu, I discovered an open-source ‘software fix’ for this hardware limitation called ‘relayd’.   It’s available through the built-in package manager and can be installed with a few clicks.

Here are the resulting specifications:

  • WIFI bridged repeater to the physical WAN port (my only ‘must have’)
  • WIFI routed repeater to physical LAN port and WIFI hotspot (NAT, administration).  You could also put the LAN/hotspot in bridged mode but it requires several more steps and changes that I do not cover it in this tutorial.
  • Manage WIFI networks list and hotspot settings through the ‘basic’ (non-LUCI) GUI.

I attribute the following steps to more detailed information gathered here and here, adapted to the firmware and GUI on this particular hardware:

  1. Connect laptop or PC to the LAN port, and perform a factory reset (hold reset button for at least 10 seconds)
  2. Log into with a browser and do the initial setup.
  3. Standard GUI -> In WAN setup – Set to ‘repeater’ mode and join a network.  Wait at least a minute for the connection to come fully online before the next step.
  4. Standard GUI -> Make sure you are on the latest firmware (v2.22 as of this writing).   Do another factory reset after flashing it.
  5. Standard GUI -> Go into ‘app repo’ (wait for the packages to populate),  seach for “relayd” and install both packages which appear.   The GUI will show ‘error status 255’ each time but don’t worry, it worked.
  6. Advanced GUI -> Network -> Interfaces – Delete WAN6 (this interface is unneccesary/redundant after we repurpose the WAN port).
  7. Advanced GUI -> Network -> Interfaces – Edit WAN and change protocol to ‘Static’ and set the IP address to an IP on a random unused subnet – I used /   Also (VERY important) – go to physical settings and create a bridge between VLAN interface eth0.2 (WAN) and the Wireless Network Client.
  8. Advanced GUI -> Network -> Interfaces – Create an interface called ‘stabridge’ with type set to ‘Relay Bridge’.   On the next page, set the IP address to the IP you used in the previous step ( in this case), and select the WAN and WWAN interfaces for relaying.
  9. Advanced GUI -> System -> Startup -> Disable and stop ‘firewall’ and ‘firewall_gl’ services
  10. Advanced GUI -> System -> Startup -> Enable and ‘restart’ relayd

At this point you can plug into the WAN port and receive a DHCP IP from your primary router, as we expect in ‘bridged repeater’ mode.  Relayd handles the broadcast/DHCP traffic exchange between your wireless network and the WAN port.   If you plug into the LAN port or connect to the WIFI hotspot (routed repeater), you’ll receive an IP in the 192.168.8.x range from the GL-MT300A’s built-in DHCP server and your traffic will be routed using NAT.

I’ve found using the LAN port (or WIFI hotspot) is an easy way access the GUI to change WIFI networks or other settings since the management IP will always be  NOTE: You can access the GUI on the upstream network or WAN port by going to the WWAN’s IP address in your browser.   However keep in mind the WWAN IP is assigned by DHCP, so you should consider using a static DHCP reservation upstream.

To change the WIFI network just repeat step 3.  I’ve discovered that the device will remember all network connections and automatically connect to to the best available, and you can edit the list in the basic GUI as well.

To manage the hotspot, go into the standard GUI and toggle the ‘switch’ to enable/disable or click on the WIFI icon to change the settings.   NOTE: Disabling the hotspot causes latency and packet loss due to a bug in the custom firmware, which constantly tries to ‘bring up’ the disabled interface.  Killing off ‘gl_health’ fixes the latency and syslog errors, however, the wifi client will no longer auto-connect which basically renders the device useless.

One other thing I’ve noticed is the device regularly ‘phones home’ to gl-inet.com and several other sites for firmware checks and dynamic DNS service.   There is no way to disable this in either GUI nor in any configuration files.   To stop the unneccesary requests, you can chmod ‘000’ the following files using SSH to prevent their execution:

root@GL-MT300A:~# chmod 000 /usr/lib/ddns/glddnsupdater.sh
root@GL-MT300A:~# chmod 000 /usr/bin/glautoupdater

Lastly, it should go without saying: the firewall is completely disabled so this should ONLY be used in trusted, private network environments.   NEVER allow this configuration to be openly reachable from the internet (e/g using as a travel router) without re-enabling the firewall and carefully setting up the rules.   I use these only in private/trusted networks so I haven’t spent time on firewall configuration testing.

Enable megasas2.sys Critical Mass Storage Driver

This reg key will enable the ability to boot from MegaRAID storage devices using the megasys2.sys driver. Useful to run before hand when upgrading or otherwise changing hardware to something with a LSI 9260, 9271 or similar card. This reg key REQUIRES that you already have the correct megasas2.sys driver installed one way or another.


OpenVPN Slow Downloads on Windows clients

Recently I upgraded my remote office connection to 100Mbps (cable) and spent almost a month dealing with slow downloads over an openVPN client connection. Uploads were fine (maxing out the pipe) but downloads were 12-20 Mbps at best and fluctuating like crazy. TCP client mode actually performed better than UDP so I knew something was wrong. After making sure my MSS/MTU was fine and UDP iperf tests were able to max out my connection (without the VPN, proving my ISP wasn’t throttling UDP), the following config on the server side is what ended up fixing it:

sndbuf 0;
rcvbuf 0;
push "sndbuf 393216";
push "rcvbuf 393216";

I now get 105 Mbps downloads on my openVPN connection (115 without VPN). Turns out the default buffers are just too small for high speed tunnels. Who would have known.

Debian PXE boot image from scratch

After facing issue after issue with Idera/R1soft’s PXE and CD-ROM boot media — specifically, ancient kernels which are unable to obtain network connectivity on modern hardware, I decided to roll my own PXE boot image including the R1soft agent and a few other tools with the latest Debian stable. It turned out to be easier than I expected — kudos to the debian-live project.

Boot into a debian Live-CD and set up the bootstrap environment. I’m using 7.6.0 (latest stable at this time).

apt-get install debootstrap squashfs-tools
mkdir -p work/chroot
cd work

If you’re creating the image from scratch, use the following commands.

debootstrap --arch=amd64 stable chroot
cp /etc/network/interfaces chroot/etc/network/interfaces # Needed for network connectivity

If you are re-working an existing image, you can import it instead.

wget http://webserver/debian-live/filesystem.squashfs
unsquashfs -f -d chroot filesystem.squashfs
rm -f filesystem.squashfs

Now we prep the image and chroot.

cp /etc/resolv.conf chroot/etc/resolv.conf # Needed for network connectivity
chroot chroot
mount none -t proc /proc
mount none -t sysfs /sys
mount none -t devpts /dev/pts
export HOME=/root
export LC_ALL=C

Now you can do whatever you want to the image by installing packages and modifying configurations. You need to install the kernel at a minimum if you want to load modules after boot such as raid. You can install kernel headers and compile 3rd party modules like the r1soft cdp agent. This is a minimal image, so don’t forget basic filesystem utilities like mdadm if you’re installing the r1soft agent; I learned the hard way and had to re-work the image several times. I enabled auto-login on tty1 through tty3 by installing mingetty and modifying inittab. I also like to redirect syslog to tty4 and disable any console logging on tty1 – tty3.

When you think you’re done, clean up and exit the chroot.

apt-get clean
rm -rf /tmp/*
rm /etc/resolv.conf
umount -lf /proc
umount -lf /sys
umount -lf /dev/pts

Compress and package the new filesystem image.

mksquashfs chroot filesystem.squashfs -e boot

Send it off to your HTTP webserver:

scp filesystem.squashfs user@webserver:/var/www/html/debian-live/.

And a working pxelinux.cfg entry after we’ve got the kernel and initrd in the correct place on the tftp server:

LABEL deblive
        KERNEL /debian-live/debian-live-7.6.0-amd64-standard.vmlinuz
        APPEND initrd=/debian-live/debian-live-7.6.0-amd64-standard.initrd.img dhcp ethdevice=eth0,eth1 boot=live fetch=http://webserver/debian-live/filesystem.squashfs

Grub with Xen Guests

In order for grub to ‘see’ virtual disks, you have to tell it about them with the ‘device’ command:

# grub
grub> device (hd0) /dev/xvda
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

Keep in mind, installing grub on the MBR is not always necessary with Xen PV guests, but is highly useful for V2P conversions.