Pre-analysis of a DDOS attack on a Cpanel or Linux server

Determine the nature of the attack (SYN, GET, ect):

netstat -nat | awk '{print $6}' | sort | uniq -c

The following will list all the IPs connecting to the server in order of most connections.

netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

We can see which domains are most active (in the case of a GET style attack):

cd /usr/local/apache/domlogs/
ls -ltr |tail -50

Then, we can see which IPs are active on a particular domain and take appropriate action (drop in firewall, ect):

cd /usr/local/apache/domlogs/
tail -f <domainname> | awk {'print $1'}

Tracking SPAM on a Cpanel Server

If you are using a cpanel/WHM server then the MTA will be Exim by default.

Enable Mailheaders by running /scripts/easyapache script.  After that, check the mail header using:

exim -Mvh <message id>

From that you will able to find the source of spam.

Categories: Linux and Technical. Comments Off on Tracking SPAM on a Cpanel Server

Setting up Adaptec Storage Manager on a headless Ubuntu/Debian server

Adaptec RAID cards have huge performance gains over other cards, but the management features in Linux stink. Really bad. If you don’t have a GUI installed, you cannot set up monitoring or alerts, but thankfully you can use a Windows machine to set this up remotely. Additionally, there are no .deb packages so we have to convert the .rpm to a .deb package.

Download the latest installer in RPM format:

sudo bash ### Dang it Debian!  LOL
wget http://download.adaptec.com/raid/storage_manager/asm_linux_x64_v6_30_18507.rpm

Convert the .rpm to a .deb package and install it:

apt-get install alien
alien --scripts asm_linux_x64_v6_30_18507.rpm
dpkg -i storman_6.30-18508_amd64.deb

Start the ‘StorMan’ agent:

/usr/StorMan/StorMan.sh

Add the following to /etc/rc.local (before the ‘exit 0’ line) so it starts automatically as the included init script is broken miserably. Don’t forget the ampersand or else you’ll hang up the boot process:

/usr/StorMan/StorMan.sh &

Now you can remotely connect using a remote Windows or Linux GUI to check RAID status and set up email alerts. It’s painfully obvious Adaptec is catering to the Windows crowd on this one, but at least there’s a work around.

Setting up a large disk array in Linux through LVM

A very large disk array (hardware RAID) combined with LVM (Logical Volume Management) will give you a vast amount of flexibility. Without LVM, you’ll have to reboot the entire machine to realize the new space when adding RAID members. Using LVM, you can take advantage of OLE (On Line Expansion) and expand your existing volumes without rebooting. In this post, I discuss the creation of the initial VG (Volume Group), PV (Physical Volume) and LV (Logical Volume) which compose LVM. I’ll discuss OLE later in another post.

In this example, a large RAID array exists at /dev/sdb. There are no partitions on this disk; it is freshly created in the RAID card’s BIOS.

root@rj04:~# fdisk -l

...snip...

Disk /dev/sdb: 2977.4 GB, 2977474543616 bytes
255 heads, 63 sectors/track, 361990 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

1. Create a new PV (Physical Volume) on the new disk. This is the ‘top layer’ of LVM:

pvcreate /dev/hdb

2. Create a new VG (Volume Group) inside the new PV. This is the ‘middle layer’ of LVM:

vgcreate vg0 /dev/hdb

3. Create a new LV (full size). This is the ‘bottom layer’ of LVM where you can put your filesystems:

lvcreate -n lvol0 vg0 -L 2.70T

4. Create and format the filesystem. Remember to choose the correct block size (4k block size allows up to 8Tb size, which is enough for the max our array will expand to):

mkfs -t ext3 -m 0 -v -b 4096 /dev/vg0/lvol0

5. Add mount point, update fstab, and mount the new disk:

mkdir /data ## Create mount
echo "/dev/vg0/lvol0  /data   ext3    defaults        0       2" >> /etc/fstab ## Make filesystem aware of the mount
mount /data  ## Actually mount the disk

6. Verify your work:

root@rj04:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
...snip...
/dev/mapper/vg0-lvol0
                      2.7T  202M  2.7T   1% /data

Installing mod_limitipconn on a Cpanel server

Absolutely essential in a shared hosting environment. This example is relevant to Apache 2.2.x.

#############
## Compile mod_ipconnlimit
cd /root
wget http://dominia.org/djao/limit/mod_limitipconn-0.23.tar.bz2
tar xjf mod_limitipconn-0.23.tar.bz2
cd mod_limitipconn-0.23
/usr/local/apache/bin/apxs -cia mod_limitipconn.c  ### this needs to be redone after each apache recompile


## in WHM, add to apache Pre VirtualHost Include (all versions)
<IfModule mod_limitipconn.c>
    # Set a server-wide limit of 10 simultaneous downloads per IP,
    # except for image folders.
    MaxConnPerIP 10
    NoIPLimit image/*
    NoIPLimit images/*
</IfModule>
##############

Working with WinPE 3.0 images

Mount a WinPE image:

dism /Mount-Wim /WimFile:winpe.wim /index:1 /MountDir:mount

Inject PNP drivers into the WinPE image:

Dism /image:mount /Add-Driver /driver:C:\drivers\blah\mydriver.inf

Un-mount and commit the image:

Dism /Unmount-Wim /MountDir:mount /Commit

See what’s in the image:

imagex /info install.wim

Merge images:

imagex /export source.wim 1 destination.wim "Image Name" /compress maximum

Create a boot cd ISO:

oscdimg -n -bc:\pe\amd64\etfsboot.com c:\pe\amd64\ISO c:\pe\amd64\winpe_amd64.iso

More info in working with images:

http://technet.microsoft.com/en-us/library/dd799244%28WS.10%29.aspx

Info on working with unattend.xml

http://technet.microsoft.com/en-us/library/cc722132%28WS.10%29.aspx

Some notes for older WinPE images; where dism.exe doesn’t work, you need imagex and peimg:

Mount an older WinPE image:

imagex.exe /mountrw winpe.wim 1 mount

Inject PNP drivers into the WinPE image:

peimg.exe /inf=c:\drivers\blah\*.inf mount\Windows

Un-mount and commit the image:

imagex.exe /unmount mount /commit

Each time you ‘commit’ the image it grows in size. Let’s bring it back down to normal:

imagex /compress maximum /export winpe.wim 1 winpe.wim.optimized

Limiting Bandwidth in Linux

The tc command in Linux can be used for fine-grained control over bandwidth throughput. In this case I have limited public outbound traffic to 50Mbps, and internal network traffic to 450Mbps.

The syntax of tc is quite complex. So much so, I found a neat utility called ‘tcng’ (Traffic Control Next Generation) that interprits a much simpler C like syntax and converts it to a string of tc commands.

Here is the tcng script:

dev "eth0" {
  egress {
   class( <$fs> ) // Internal networks
    if ip_dst/22 == 192.168.0.0
    if ip_dst/21 == 192.168.208.0
    if ip_dst/24 == 192.168.240.0
   ;
    class( <$all> ) // All other IP addresses
        if 1
   ;
   htb() {
    class ( rate 500Mbps, ceil 500Mbps ) {
     $fs = class ( rate 450Mbps, ceil 450Mbps ) {sfq;} // Limit internal/trusted network to 450Mbps
     $all = class ( rate 50Mbps, ceil 50Mbps ) {sfq;} // Limit all other networks to 50Mbps
    }
   }
  }
 }

Here is the output it created, which I rolled into a bash script:

# ================================ Device eth0 ================================
tc qdisc add dev eth0 handle 1:0 root dsmark indices 4 default_index 0
tc qdisc add dev eth0 handle 2:0 parent 1:0 htb
tc class add dev eth0 parent 2:0 classid 2:1 htb rate 62500000bps ceil 62500000bps
tc class add dev eth0 parent 2:1 classid 2:2 htb rate 56250000bps ceil 56250000bps
tc qdisc add dev eth0 handle 3:0 parent 2:2 sfq
tc class add dev eth0 parent 2:1 classid 2:3 htb rate 6250000bps ceil 6250000bps
tc qdisc add dev eth0 handle 4:0 parent 2:3 sfq
tc filter add dev eth0 parent 2:0 protocol all prio 1 tcindex mask 0x3 shift 0
tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 2 tcindex classid 2:3
tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 1 tcindex classid 2:2
tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0xd055f000 0xfffffc00 at 16 classid 1:1
tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0x4a73d000 0xfffff800 at 16 classid 1:1
tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0xd109ee00 0xffffff00 at 16 classid 1:1
tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 at 0 classid 1:2

The following command clears out any previous tc settings. You need to run it each time you want to change your traffic limits or configuration:

tc qdisc del root dev eth0

Intel D945GCLF Ethernet timeouts; firmware bug

The Intel D945GCLF (Atom 230) main board has a serious bug in the Realtek gigabit NIC firmware. Any sustained transfer over 100Mbps will eventually cause the NIC to go offline until the network stack is reloaded or machine rebooted. I reproduced the issue in CentOS, Debian, and Windows.

No driver fixes the issue; you have to update the BIOS to the latest version from the Intel website to resolve this.

Lockfiles in Bash

Here’s a simple ‘skeleton’ script that will allow your Bash scripts to use a PID file, or ‘lockfile’. This ensures that only one instance can run at a time which is useful for daily Cron activities such as mirror updates.

#!/bin/bash
pidfile=/var/run/sync.pid
if [ -e $pidfile ]; then
pid=`cat $pidfile`
if kill -0 &>1 > /dev/null $pid; then
echo "Already running"
exit 1
else
rm $pidfile
fi
fi
echo $$ > $pidfile
#do your thing here
rm $pidfile

Categories: Linux. Comments Off on Lockfiles in Bash

Reclaim ‘missing’ space on a Linux partition with tune2fs

If you aren’t storing critical system files on a partition, you can free up a ton of space. This will reduce the ‘reserved’ space on a Linux partition to 0%. I freed up 45Gb of ‘lost’ space on a 900Gb partition using the following command:

tune2fs -m 0 /dev/sda4

Note: DO NOT set the reserved space to 0 on a system partition.

Categories: Linux. Comments Off on Reclaim ‘missing’ space on a Linux partition with tune2fs