Randy\'s Blog - Part 3

Pacemaker / Corosync / DRBD Cheatsheet

Posted on April 20, 2013 by Randy

Monitor the status:

crm_mon

Migrate all resources to another node:

crm resource migrate rg_main <fqdn_node_name>

Take node offline and online (be careful, this sets a ‘prefer’ to the other node to force a transition, which may or may not get removed afterwards):

crm node standby
crm node online

Start and stop all resources (warning, this will take them completely offline, NOT migrate):

crm resource stop rg_main
crm resource start rg_main

Show configuration:

crm configure show

If resources are stuck in ‘(unmanaged) FAILED’ state, e.g. due to a failed stop action, you can clear it out:

crm_resource -P
crm resource cleanup rg_main

Be careful — this could trigger a migration if the stuck resources were preventing one. Make sure you’re ready for one.

Monitor the cluster status along with fail counts:

crm_mon --failcount

One-shot status output:

crm status

Check DRBD status:

cat /proc/drbd

DRBD split-brain cleanup on secondary node:

drbdadm disconnect main
drbdadm -- --discard-my-data connect main

DRBD split-brain cleanup on primary node:

drbdadm disconnect main
drbdadm primary main
drbdadm connect main

Scheduler optimizing on large arrays (untested):

echo deadline > /sys/block/sdb/queue/scheduler
echo 0 >  /sys/block/sdb/queue/iosched/front_merges
echo 150 > /sys/block/sdb/queue/iosched/read_expire
echo 1500 > /sys/block/sdb/queue/iosched/write_expire

Exchange 2012 Allow Relay from IP

Posted on April 3, 2013 by Randy

Enable mail relaying from a specific IP or range of IPs.

1. Exchange Management Console -> Server Configuration -> Hub Transport

2. New Receive Connector

* Set remote network to the IP or range you want to allow relay from (default is any IP so watch out here)
* “Externally Secured” authentication
* “Exchange Servers” permission group
* All other options disabled

Increasing max files or folders per directory on Linux EXT filesystem

Posted on March 25, 2013 by Randy

If a (poorly coded) app reaches the maximum number of files or folders per directory in Linux, you may see errors like this:

Error happened when generating Download Link.
Please try again or Contact administrator.
(ERROR:mkdir)

A quick and dirty way to increase the limit (and overall performance of the system) is to add the dir_index flag to the ext filesystem, then reindex:

tune2fs -O dir_index /dev/sda3
updatedb &

Reset Supermicro IPMI password with ipmitool

Posted on March 24, 2013 by Randy

This will set the ‘ADMIN’ user password back to default of ‘ADMIN’. You could set it to anything you want, this is just an example.

yum -y install ipmitool
modprobe ipmi_devintf
ipmitool -I open user set password 2 ADMIN

Strong Password Encryption in PHP

Posted on March 6, 2013 by Randy

Don’t try to reinvent a complex and highly refined wheel, especially when customer data is involved. Please use phpass on any PHP project. (that’s PH-Pass)

http://www.openwall.com/phpass/

Good reading for password hashing in general.

http://www.openwall.com/articles/PHP-Users-Passwords

Cpanel Apache Symlink Protection

Posted on March 6, 2013 by Randy

wget http://layer1.rack911.com/before_apache_make -O /scripts/before_apache_make
chmod 700 /scripts/before_apache_make
## Rebuild apache (might want to do it thru WHM instead)
/scripts/easyapache --build

AS Number Stats on Linux

Posted on March 6, 2013 by Randy

I’ve got a couple of Linux machines that are sitting outside of the Sflow ‘zone’ and AS traffic stats go unmeasured. I wanted to get a rough idea on the number of connections per AS number so here’s a little app that parses netstat and sorts AS numbers by number of connection.

NOTE1: This won’t work on Cpanel servers due to tmp restrictions.

NOTE2: Specify an alternative (e.g newer) GeoIPASNum.dat file with the –geo option.

wget http://djlab.com/stuff/asnum
chmod +x asnum
./asnum

Example output:

[root@mx1 ~]# ./asnum
(18) | RFC1918 or BOGON
(11) | AS29889 Fast Serv Networks, LLC
(4) | AS3320 Deutsche Telekom AG
(2) | AS7922 Comcast Cable Communications, Inc.

Quick and (very) dirty cron script

#!/bin.sh
#/root/doasnum.sh
#*/5 * * * * /root/./doasnum.sh >> /var/log/asnum.log

thedate=`date`
echo "***********************"
echo $thedate
/root/asnum
echo "***********************"

Don’t forget to rotate the logs.

#/etc/logrotate.d/asnum
/var/log/asnum.log
{
        rotate 7
        daily
        missingok
}

Obfuscate or Encrypt Perl

Posted on March 1, 2013 by Randy

There are 3 primary options:

1. Use PAR with PAR::Filter::Obfuscate or PAR::Filter::Crypto

2. Use Filter::Crypto::CryptFile (requires certain modules installed on target system)

3. Encrypt modules with Module::Crypt.

Clone a live linux system with Rsync over SSH

Posted on February 19, 2013 by Randy

Update: 5/3/17 — Add additional notes for CentOS7 (dracut + grub2) and XenServer targets

All commands are run on new server.

1. Boot into rescue mode (iso, pxe, ect).

2. Create partitions with ‘fdisk /dev/sda’. Type 83 for non-RAID fs, 82 for swap, type fd in the case of RAID (all partitions). Flag boot partition as bootable.

In the case of SSD, add -S 32 -H 32 to the fdisk command and start the first partition on sector 2 for proper alignment.

If using RAID, duplicate the partition table after creating it on the first disk:

dd if=/dev/sda of=/dev/sdb bs=1 count=64 skip=446 seek=446

2. Create RAID array (if applicable).

# For SSD, add: --chunk=128
mdadm --create /dev/md0 -e 0.90 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1  ## /boot
mdadm --create /dev/md1 -e 0.90 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2  ## Swap
mdadm --create /dev/md2 -e 0.90 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3  ## /

3. Create filesystems

For spin disk:

mkfs.ext4 /dev/md0 # /dev/sda1 for non-RAID
mkfs.ext4 /dev/md2 # /dev/sda3 for non-RAID

For SSD (non RAID):

mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda1
mkfs.ext4 -b 1024 -E stride=128,stripe-width=128 -O ^has_journal /dev/sda3

For SSD (RAID):

mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md0 ## stripe-width = stride x N disks
mkfs.ext4 -b 1024 -E stride=128,stripe-width=256 -O ^has_journal /dev/md2 ## stripe-width = stride x N disks

4. Mount filesystems

mkdir /mount
mount /dev/md2 /mount  ## /dev/sda3 for non-RAID
mkdir {/mount/boot,/mount/dev,/mount/sys,/mount/proc,/mount/tmp}  
mount /dev/md0 /mount/boot ## /dev/sda1 for non-RAID

5. Sync filesystems with Rsync over SSH (Ex: 1.2.3.4 is source machine)

rsync -aHxv --numeric-ids --progress root@1.2.3.4:/* /mount --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp
rsync -aHxv --numeric-ids --progress root@1.2.3.4:/boot/* /mount/boot --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp  ## Only if /boot is on separate partition in source machine

If applicable: stop mysql on the source machine and resync the databases to prevent corruption:

rsync -aHxv root@1.2.3.4:/var/lib/mysql/* /mount/var/lib/mysql

6. Update mdadm.conf

mdadm --examine --scan &gt; /mount/etc/mdadm.conf

7. Update fstab (if needed)

ls -la /dev/disk/by-uuid # to get new UUID's
vi /mount/etc/fstab

8a. Install bootloader (CentOS5, CentOS6 / grub)

grub
root (hd0,0)
setup (hd0)
root (hd1,0)  ## for RAID
setup (hd1)  ## for RAID
exit

8b. Install bootloader (CentOS7 / grub2)

Do step 10 (chroot) first, then this

grub2-install /dev/sda # Use correct device(s), repeat as neccesary
# Good time to clean up /etc/default/grub, then:
grub2-mkconfig -o /etc/grub2.cfg

# If migrating to Xen/XenServer:
dracut --add-drivers "xen-blkfront xen-netfront xen:vbd" --regenerate-all --force

9. Optional: change IP address if both machines need to be online

vi /mount/etc/sysconfig/network-scripts/ifcfg-eth0

10. Hint: you can chroot into the cloned filesystem – for example – initramfs rebuilding:

cd /mount/
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
chroot .

11. Cross fingers, reboot

reboot

Find symlinks on Cpanel

Posted on February 7, 2013 by Randy

Find all possible root symlinks (leftover from attempted exploits) and save them as a list ‘symlinks.txt’.

ls /var/cpanel/users | grep -v "\`\|\.\|cpanel\|root\|mysql\|nobody" | \
while read CPUSER; do find /home/$CPUSER -type l -not \( -lname "/home/$CPUSER/*" \
-o -lname "*rvsitebuilder*" -o -lname "[^/]*" -o -lname "/usr/local/apache/domlogs/*" \
-o -lname "/usr/local/urchin/*" \) ; done \
> symlinks.txt &

Page 3 of 15«1 234 5 6...14 15 »