Pacemaker / Corosync / DRBD Cheatsheet

Posted on by Randy

Monitor the status:

crm_mon

Migrate all resources to another node:

crm resource migrate rg_main <fqdn_node_name>

Take node offline and online (be careful, this sets a ‘prefer’ to the other node to force a transition, which may or may not get removed afterwards):

crm node standby
crm node online

Start and stop all resources (warning, this will take them completely offline, NOT migrate):

crm resource stop rg_main
crm resource start rg_main

Show configuration:

crm configure show

If resources are stuck in ‘(unmanaged) FAILED’ state, e.g. due to a failed stop action, you can clear it out:

crm_resource -P
crm resource cleanup rg_main

Be careful — this could trigger a migration if the stuck resources were preventing one. Make sure you’re ready for one.

Monitor the cluster status along with fail counts:

crm_mon --failcount

One-shot status output:

crm status

Check DRBD status:

cat /proc/drbd

DRBD split-brain cleanup on secondary node:

drbdadm disconnect main
drbdadm -- --discard-my-data connect main

DRBD split-brain cleanup on primary node:

drbdadm disconnect main
drbdadm primary main
drbdadm connect main

Scheduler optimizing on large arrays (untested):

echo deadline > /sys/block/sdb/queue/scheduler
echo 0 >  /sys/block/sdb/queue/iosched/front_merges
echo 150 > /sys/block/sdb/queue/iosched/read_expire
echo 1500 > /sys/block/sdb/queue/iosched/write_expire

Exchange 2012 Allow Relay from IP

Posted on by Randy

Enable mail relaying from a specific IP or range of IPs.

1. Exchange Management Console -> Server Configuration -> Hub Transport

2. New Receive Connector

* Set remote network to the IP or range you want to allow relay from (default is any IP so watch out here)
* “Externally Secured” authentication
* “Exchange Servers” permission group
* All other options disabled

Increasing max files or folders per directory on Linux EXT filesystem

Posted on by Randy

If a (poorly coded) app reaches the maximum number of files or folders per directory in Linux, you may see errors like this:

Error happened when generating Download Link.
Please try again or Contact administrator.
(ERROR:mkdir)

A quick and dirty way to increase the limit (and overall performance of the system) is to add the dir_index flag to the ext filesystem, then reindex:

tune2fs -O dir_index /dev/sda3
updatedb &

AS Number Stats on Linux

Posted on by Randy

I’ve got a couple of Linux machines that are sitting outside of the Sflow ‘zone’ and AS traffic stats go unmeasured. I wanted to get a rough idea on the number of connections per AS number so here’s a little app that parses netstat and sorts AS numbers by number of connection.

NOTE1: This won’t work on Cpanel servers due to tmp restrictions.

NOTE2: Specify an alternative (e.g newer) GeoIPASNum.dat file with the –geo option.

wget http://djlab.com/stuff/asnum
chmod +x asnum
./asnum

Example output:

[root@mx1 ~]# ./asnum
(18) | RFC1918 or BOGON
(11) | AS29889 Fast Serv Networks, LLC
(4) | AS3320 Deutsche Telekom AG
(2) | AS7922 Comcast Cable Communications, Inc.

Quick and (very) dirty cron script

#!/bin.sh
#/root/doasnum.sh
#*/5 * * * * /root/./doasnum.sh >> /var/log/asnum.log

thedate=`date`
echo "***********************"
echo $thedate
/root/asnum
echo "***********************"

Don’t forget to rotate the logs.

#/etc/logrotate.d/asnum
/var/log/asnum.log
{
        rotate 7
        daily
        missingok
}

Clone a live linux system with Rsync over SSH

Posted on by Randy

Update: 5/3/17 — Add additional notes for CentOS7 (dracut + grub2) and XenServer targets

All commands are run on new server.

1. Boot into rescue mode (iso, pxe, ect).

2. Create partitions with ‘fdisk /dev/sda’. Type 83 for non-RAID fs, 82 for swap, type fd in the case of RAID (all partitions). Flag boot partition as bootable.

In the case of SSD, add -S 32 -H 32 to the fdisk command and start the first partition on sector 2 for proper alignment.

If using RAID, duplicate the partition table after creating it on the first disk:

dd if=/dev/sda of=/dev/sdb bs=1 count=64 skip=446 seek=446

2. Create RAID array (if applicable).

# For SSD, add: --chunk=128
mdadm --create /dev/md0 -e 0.90 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1  ## /boot
mdadm --create /dev/md1 -e 0.90 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2  ## Swap
mdadm --create /dev/md2 -e 0.90 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3  ## /

3. Create filesystems

For spin disk:

mkfs.ext4 /dev/md0 # /dev/sda1 for non-RAID
mkfs.ext4 /dev/md2 # /dev/sda3 for non-RAID

For SSD (non RAID):

mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda1
mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda3

For SSD (RAID):

mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md0 ## stripe-width = stride x N disks
mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md2 ## stripe-width = stride x N disks

4. Mount filesystems

mkdir /mount
mount /dev/md2 /mount  ## /dev/sda3 for non-RAID
mkdir {/mount/boot,/mount/dev,/mount/sys,/mount/proc,/mount/tmp}  
mount /dev/md0 /mount/boot ## /dev/sda1 for non-RAID

5. Sync filesystems with Rsync over SSH (Ex: 1.2.3.4 is source machine)

rsync -aHxv --numeric-ids --progress root@1.2.3.4:/* /mount --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp
rsync -aHxv --numeric-ids --progress root@1.2.3.4:/boot/* /mount/boot --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp  ## Only if /boot is on separate partition in source machine

If applicable: stop mysql on the source machine and resync the databases to prevent corruption:

rsync -aHxv root@1.2.3.4:/var/lib/mysql/* /mount/var/lib/mysql

6. Update mdadm.conf

mdadm --examine --scan &gt; /mount/etc/mdadm.conf

7. Update fstab (if needed)

ls -la /dev/disk/by-uuid # to get new UUID's
vi /mount/etc/fstab

8a. Install bootloader (CentOS5, CentOS6 / grub)

grub
root (hd0,0)
setup (hd0)
root (hd1,0)  ## for RAID
setup (hd1)  ## for RAID
exit

8b. Install bootloader (CentOS7 / grub2)

Do step 10 (chroot) first, then this

grub2-install /dev/sda # Use correct device(s), repeat as neccesary
# Good time to clean up /etc/default/grub, then:
grub2-mkconfig -o /etc/grub2.cfg

# If migrating to Xen/XenServer:
dracut --add-drivers "xen-blkfront xen-netfront xen:vbd" --regenerate-all --force

9. Optional: change IP address if both machines need to be online

vi /mount/etc/sysconfig/network-scripts/ifcfg-eth0

10. Hint: you can chroot into the cloned filesystem – for example – initramfs rebuilding:

cd /mount/
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
chroot .

11. Cross fingers, reboot

reboot

Find symlinks on Cpanel

Posted on by Randy

Find all possible root symlinks (leftover from attempted exploits) and save them as a list ‘symlinks.txt’.

ls /var/cpanel/users | grep -v "\`\|\.\|cpanel\|root\|mysql\|nobody" | \
while read CPUSER; do find /home/$CPUSER -type l -not \( -lname "/home/$CPUSER/*" \
-o -lname "*rvsitebuilder*" -o -lname "[^/]*" -o -lname "/usr/local/apache/domlogs/*" \
-o -lname "/usr/local/urchin/*" \) ; done \
> symlinks.txt &