Clone a live linux system with Rsync over SSH

Update: 5/3/17 — Add additional notes for CentOS7 (dracut + grub2) and XenServer targets

All commands are run on new server.

1. Boot into rescue mode (iso, pxe, ect).

2. Create partitions with ‘fdisk /dev/sda’. Type 83 for non-RAID fs, 82 for swap, type fd in the case of RAID (all partitions). Flag boot partition as bootable.

In the case of SSD, add -S 32 -H 32 to the fdisk command and start the first partition on sector 2 for proper alignment.

If using RAID, duplicate the partition table after creating it on the first disk:

dd if=/dev/sda of=/dev/sdb bs=1 count=64 skip=446 seek=446

2. Create RAID array (if applicable).

# For SSD, add: --chunk=128
mdadm --create /dev/md0 -e 0.90 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1  ## /boot
mdadm --create /dev/md1 -e 0.90 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2  ## Swap
mdadm --create /dev/md2 -e 0.90 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3  ## /

3. Create filesystems

For spin disk:

mkfs.ext4 /dev/md0 # /dev/sda1 for non-RAID
mkfs.ext4 /dev/md2 # /dev/sda3 for non-RAID

For SSD (non RAID):

mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda1
mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda3


mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md0 ## stripe-width = stride x N disks
mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md2 ## stripe-width = stride x N disks

4. Mount filesystems

mkdir /mount
mount /dev/md2 /mount  ## /dev/sda3 for non-RAID
mkdir {/mount/boot,/mount/dev,/mount/sys,/mount/proc,/mount/tmp}  
mount /dev/md0 /mount/boot ## /dev/sda1 for non-RAID

5. Sync filesystems with Rsync over SSH (Ex: is source machine)

rsync -aHxv --numeric-ids --progress root@* /mount --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp
rsync -aHxv --numeric-ids --progress root@* /mount/boot --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp  ## Only if /boot is on separate partition in source machine

If applicable: stop mysql on the source machine and resync the databases to prevent corruption:

rsync -aHxv root@* /mount/var/lib/mysql

6. Update mdadm.conf

mdadm --examine --scan > /mount/etc/mdadm.conf

7. Update fstab (if needed)

ls -la /dev/disk/by-uuid # to get new UUID's
vi /mount/etc/fstab

8a. Install bootloader (CentOS5, CentOS6 / grub)

root (hd0,0)
setup (hd0)
root (hd1,0)  ## for RAID
setup (hd1)  ## for RAID

8b. Install bootloader (CentOS7 / grub2)

Do step 10 (chroot) first, then this

grub2-install /dev/sda # Use correct device(s), repeat as neccesary
# Good time to clean up /etc/default/grub, then:
grub2-mkconfig -o /etc/grub2.cfg

# If migrating to Xen/XenServer:
dracut --add-drivers "xen-blkfront xen-netfront xen:vbd" --regenerate-all --force

9. Optional: change IP address if both machines need to be online

vi /mount/etc/sysconfig/network-scripts/ifcfg-eth0

10. Hint: you can chroot into the cloned filesystem – for example – initramfs rebuilding:

cd /mount/
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
chroot .

11. Cross fingers, reboot


Find symlinks on Cpanel

Find all possible root symlinks (leftover from attempted exploits) and save them as a list ‘symlinks.txt’.

ls /var/cpanel/users | grep -v "\`\|\.\|cpanel\|root\|mysql\|nobody" | \
while read CPUSER; do find /home/$CPUSER -type l -not \( -lname "/home/$CPUSER/*" \
-o -lname "*rvsitebuilder*" -o -lname "[^/]*" -o -lname "/usr/local/apache/domlogs/*" \
-o -lname "/usr/local/urchin/*" \) ; done \
> symlinks.txt &

Simple Monitoring with Email Alerts that works on LSI MegaRAID and Adaptec

This tool will poll the output of command(s) or URL(s), and send email alerts if the output changes, contains (or not contains) certain text, or becomes unavailable. It’s a light weight, reliable monitoring replacement for the pile of garbage most RAID vendors include with their cards. Complex excludes and finds can be specified as Perl regex patterns — useful for suppression various noise (RAID verify, ect). The default configuration includes examples that will monitor local LSI or Adaptec cards.

Note – Tested on CentOS only. if someone wants to try Alien on Debian please let me know how it works.

rpm -ivh

After installation, edit /etc/cloud.ini then:

service cloud-mon restart

Make sure you have logrotate installed to prevent log growth. This package includes a simple logrotate script.

Winsplit Revolution (Like Windows Snap but for Multiple Displays)

I just added a 4th monitor to my office PC and finally admitted to needing help with window placement, far beyond Windows 7’s limited snap feature, this tool really gets the job done:

It’s a free, light-weight ‘snap’ on steroids designed specifically for multi-display setups.

Categories: Uncategorized. Comments Off on Winsplit Revolution (Like Windows Snap but for Multiple Displays)

Guest isolation in XenServer 6.1 / XCP 1.6

Since XenServer 6.1 (XCP 1.6) there is a new feature that allows you to lock VIF to specific MAC and IP addresses. This is nice (and also very buggy!), but it doesn’t provide any security other than keeping VMs from stealing each others IPs. A better solution should allow to (optionally) isolate traffic between groups of VMs as well. For example, prevent users from accessing other users VMs over a common private/backend network, but allow communication between VMs in the same group and with external networks.

Step 1: We must relocate the VIF locking data into the VM’s vm-data store, similar to how security groups are managed by Nova (openstack). This lets us use more fields and options.

Step 2: Patch /opt/xensource/libexec/setup-vif-rules to fix *several* bugs as well as take an extra locking mode (isolated). In isolate mode, VMs can only communicate to IPs in the allowed list. Use a common set of IPs on each set of VMs in the same ‘security group’ and your various groups of guests are isolated from each other.

Step 3: Patch another bug in /etc/xensource/scripts/vif to prevent orphaned rules from piling up in openvswitch when VMs are restarted.

Patches are here:

This will only work with openvswitch networking mode and has been tested on XS 6.1 and XCP 1.6. Do not try it in bridged mode (you shouldn’t be using bridged mode anyways).

XenServer multipath configuration for LIO targets

XenServer multipath.conf with special support for LIO-based iSCSI targets to maximize multipath performance and ensures 100% stability. path_grouping_policy setting doesn’t seem to matter (between group_by_prio and multibus) in most basic setups. Invalid lines (as reported by XenServer 6.1) have also been removed.


        device {
                vendor "LIO-ORG"
                product "*"
                path_grouping_policy group_by_prio
                path_checker tur
                getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
                prio_callout "/sbin/mpath_prio_alua /dev/%n"
                path_selector "round-robin 0"
                hardware_handler "1 alua"
                rr_weight uniform
                rr_min_io 2
                failback immediate

Migrate a XenServer VM without a Pool or Shared Storage

Source is now in github, please help me make it better! If you just want a 32-bit binary (to run on Dom0), download it from here instead.

With the help of Ben Booth’s Xen::API (Perl module) I put together a VM migration script to export a VM directly to another host with no intermediary file. The transfer occurs over XAPI with no temp files or local disk interaction. This script can run directly on the source or destination host, or any server in between. Bear in mind, you will have the best speeds and least network overhead running this directly on the destination host.

As of today, MigrateVM has been tested and works fine on XenServer 5.6 through 6.5.


-sh : source host
-su : source user (usually root)
-sp : source pass
-sv : source VM label or UUID
-dh : destination host
-du : destination user
-dp : destination pass
-ds : destination SR (optional)

If any of the options are omitted, you will be prompted for them.

Example output:

[root@cl-ash-h1 ~]# ./migratevm
Enter source host name/IP (blank = localhost):
Enter username for (blank = root):
Enter password for ************
Enter source vm name or uuid on my_vm
Enter destination host name/IP (blank = localhost):
Enter username for localhost (blank = root):
Enter password for localhost: ******
Destination SR on localhost (blank for default):
Starting transfer
...................    12.0%, 30618.43 (KB/sec)

Download the script like this:

tar zxf migratevm-1.0.2.tar.gz && cd migratevm-1.0.2

If you get ‘bad ELF’ or something like that on a 64 bit system, try to install 32-bit glibc, for example:

Older XenServers: yum install glibc.i686
XenServer 6.5:  yum install glibc.i686 --enablerepo=base --enablerepo=updates --disablerepo=citrix

Binary and source are both included in the tarball.

Version 1.0.2 has an updated binary build which should now run on XenServer 6.5. We needed to static link expat into the binary because it is no longer installed by default on XS 6.5.

X9SCM / X9SCL Network Timeout

Supermicro X9SCM and X9SCL main boards will lose network connection after some heavy traffic, especially on RHEL/CentOS 6. Updating BIOS and driver will not always fix this:

Oct 19 18:32:49 zeus kernel: ------------[ cut here ]------------
Oct 19 18:32:49 zeus kernel: WARNING: at net/sched/sch_generic.c:267 dev_watchdog+0x26d/0x280() (Not tainted)
Oct 19 18:32:49 zeus kernel: Hardware name: X9SCL/X9SCM
Oct 19 18:32:49 zeus kernel: NETDEV WATCHDOG: eth1 (e1000e): transmit queue 0 timed out
Oct 19 18:32:49 zeus kernel: Modules linked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc nf_conntrack vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length vhost_net xt_hl xt_tcpmss macvtap xt_TCPMSS macvlan iptable_mangle iptable_filter xt_multiport xt_limit tun xt_dscp ipt_REJECT ip_tables kvm_intel kvm vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse snd_pcsp snd_pcm snd_timer video i2c_i801 tpm_tis tpm tpm_bios serio_raw i2c_core snd shpchp output soundcore snd_page_alloc ext3 jbd mbcache ahci e1000e [last unloaded: scsi_wait_scan]
Oct 19 18:32:49 zeus kernel: Pid: 4, comm: ksoftirqd/0 veid: 0 Not tainted 2.6.32-15-pve #1
Oct 19 18:32:49 zeus kernel: Call Trace:
Oct 19 18:32:49 zeus kernel: <IRQ> [<ffffffff8106c608>] ? warn_slowpath_common+0x88/0xc0
Oct 19 18:32:49 zeus kernel: [<ffffffff8106c6f6>] ? warn_slowpath_fmt+0x46/0x50
Oct 19 18:32:49 zeus kernel: [<ffffffff8147c6fd>] ? dev_watchdog+0x26d/0x280
Oct 19 18:32:49 zeus kernel: [<ffffffff8107fcac>] ? run_timer_softirq+0x1bc/0x380
Oct 19 18:32:49 zeus kernel: [<ffffffff8147c490>] ? dev_watchdog+0x0/0x280
Oct 19 18:32:49 zeus kernel: [<ffffffff81075413>] ? __do_softirq+0x103/0x260
Oct 19 18:32:49 zeus kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30

So far, the only fix that has worked for me is doing two things.

1. Patch the NIC firmware (do this for both eth0 and eth1):

Download Here

2. Then add the following kernel parameter in your grub.conf (or menu.lst depending on OS flavor):


Now reboot. You shouldn’t lose your network again.