XenServer recovery – XAPI won’t start

Posted on by Randy

If running XenServer on a RAID array with writeback (controller cache) enabled and no battery backup, don’t lose power. If you do, you will discover the state.db is corrupt and XAPI fails to start. Everything is down, Dom0 lost it’s network config, you’re screwed. Maybe. This tutorial assumes you’ve got a metadata backup, all your VDI’s on LVM local storage, and the server is not part of a pool. If you meet this criteria, you can be back up in just a few minutes.

First, remove the corrupt state.db and restart XAPI:

[root@xen ~]# mv /var/xapi/state.db /var/xapi/state.db.bad
[root@xen ~]# xe-toolstack-restart

Reboot the server again for safe measure, then make sure you can see the XE Console. Reconfigure the network as needed.

Look for your orphaned SR and obtain it’s UUID (the UUID is after ‘VG_Xenstorage-‘)

[root@xen ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3   <--- Take note of this for later
  VG Name               VG_XenStorage-768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09   <--- Use this UUID

  PV Size               3.63 TB / not usable 9.14 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              951549
  Free PE               337665
  Allocated PE          613884
  PV UUID               QZ2vnb-1rDi-sFKD-6ofg-czgs-BYjn-u6dO6x

Using the UUID in the VG Name above, reintroduce your storage:

[root@xen ~]# xe sr-introduce uuid=768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09 type=lvm name-label="Primary Storage" content-type=user
768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09

Take note of the sr-uuid result of the previous command, you need to plug it in next. Now look for the disk ID so we can create the VBD. Pro-tip – press TAB after host-uuid= and it’ll auto complete.

[root@xen ~]# ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root  9 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f -> ../../sda
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 12 22:36 scsi-3600605b008f8a2501b80cee707491e0f-part3 -> ../../sda3 <---- This one!

[root@xen ~]# xe pbd-create host-uuid=c7b099d9-1d50-402f-ae3b-1a52ca811a8f sr-uuid=768e6b7c-e466-1ef8-eee1-8a7e0b2bdd09 device-config:device=/dev/disk/by-id/scsi-3600605b008f8a2501b80cee707491e0f-part3
9d657c71-7e85-ace8-719c-1d20890fce59

Take note of the new PBD UUID and plug in the new PBD:

[root@xen ~]# xe pbd-plug uuid=9d657c71-7e85-ace8-719c-1d20890fce59

Now go into XE Console, search for and restore the metadata backup. You have automatic metadata backups enabled, right?

NOTE: If you are running tagged VLAN’s, before starting any VMs use XenCenter to straighten out the VLAN config on the host network before starting any VM’s. You will need to edit each one and assign to the correct interface.

Good luck.

Comments are closed.