CentOS/Redhat IPSEC and EC2

So it turns out my 5 minute vpn doesn’t work in EC2 because the ESP/AH protocols (50 and 51) are blocked on the AWS network.

This is no big deal tho, as NAT-T allows one to tunnel IPSEC over UDP… however getting it to work on CentOS required a bit of a hack.

If you have already tried setting up an IPSEC vpn, shut it down with ifdown ipsec1 and remove your /etc/racoon/192.168.56.101.conf (or whatever IP yours is).

To start the hack on BOTH boxes, you need to edit /etc/sysconfig/network-scripts/ifup-ipsec. Around line 215 you need to insert nat_traversal force;… like this….

BEFORE:

        case "$IKE_METHOD" in
           PSK)
              cat >> /etc/racoon/$DST.conf << EOF
        my_identifier address;
        proposal {
                encryption_algorithm $IKE_ENC;
                hash_algorithm $IKE_AUTH;
                authentication_method pre_shared_key;
                dh_group $IKE_DHGROUP;
        }
}

AFTER:

        case "$IKE_METHOD" in
           PSK)
              cat >> /etc/racoon/$DST.conf << EOF
        my_identifier address;
        nat_traversal force;
        proposal {
                encryption_algorithm $IKE_ENC;
                hash_algorithm $IKE_AUTH;
                authentication_method pre_shared_key;
                dh_group $IKE_DHGROUP;
        }
}

Again, on both boxes update your /etc/sysconfig/network-scripts/ifcfg-ipsec1 files so that AH is disabled… because AH doesn’t like NAT… like this….


[root@CentOS2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ipsec1 
DST=192.168.56.101
TYPE=IPSEC
ONBOOT=yes
IKE_METHOD=PSK
AH_PROTO=none
[root@CentOS2 ~]#

On your iptables policy make sure that UDP 500 and UDP 4500 are permitted and volia.

# tcpdump -n -i eth1 port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
20:26:49.257590 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xa), length 116
20:26:49.261076 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xa), length 116
20:26:50.260942 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xb), length 116
20:26:50.262939 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xb), length 116
20:26:51.261298 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xc), length 116
20:26:51.264974 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xc), length 116
20:26:52.262289 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xd), length 116
20:26:52.265488 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xd), length 116
20:26:53.264008 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xe), length 116
20:26:53.267003 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xe), length 116
20:26:54.265655 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xf), length 116
20:26:54.267264 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xf), length 116
20:26:55.267459 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0x10), length 116
20:26:55.269678 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0x10), length 116
14 packets captured
14 packets received by filter
0 packets dropped by kernel
#

IPSEC VPN Tunnelling over UDP…. done!

RedHat Cluster – How to Disable Fencing

I’ve spent far too long googling how to disable fencing…. I can only guess that because you shouldn’t really disable fencing no-one wants to post a how to… so for the hard of hearing.

Do NOT disable fencing on your RedHat Cluster unless you really know what you’re doing! Fencing is designed to protect your data from corruption, if you disable fencing your data is at RISK, you have been warned!

I however am working on building a GFS DRBD cluster, as far as I can gather DRBD doesn’t need fencing, and the bottom line is my data is personal data not mission critical and if my website goes down due to my disabling fencing then it’s no big deal.

Rant over, here we go….. To disable fencing, create a custom fence agent.

Fence agents are simply scripts in /sbin, I’ve created /sbin/myfence and here are the contents.

#!/bin/bash
echo "success: myfence $2"
exit 0

Next, change your cluster.conf…

<?xml version="1.0"?>
<cluster alias="linickx" config_version="41" name="linickx">
        <cman expected_votes="1" two_node="1" />

        <clusternodes>
                <clusternode name="CentOS1" nodeid="1" votes="1">
                         <fence>
                                <method name="1">
                                        <device nodename="CentOS1" name="myfence"/>
                                </method>
                        </fence>
                </clusternode>

                <clusternode name="CentOS2" nodeid="2" votes="1">
                        <fence>
                                <method name="2">
                                        <device nodename="CentOS2" name="myfence"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>

        <fencedevices>
                <fencedevice agent="myfence" name="myfence"/>
        </fencedevices>
        <rm/>
</cluster>

If you’re running SELINUX don’t forget to update that! … start with restorecon /sbin/myfence then update your policy.

This is the policy I’ve created…

module fenced 1.0;

require {
        type fenced_t;
        type shell_exec_t;
        class file { read execute };
}

#============= fenced_t ==============
allow fenced_t shell_exec_t:file { read execute };

If you save the above as fenced.te, then run this to install it..

checkmodule -M -m -o fenced.mod fenced.te
semodule_package -o fenced.pp -m fenced.mod
semodule -i fenced.pp

You should now be able to start cman, fencing will start but will return success for any fencing issues without actually doing anything!

Happy non-fencing!

GRE example for CentOS/RHEL

I’m not sure why GRE isn’t in RedHat’s Documentation, but setting up a GRE tunnel between two RedHat boxes is quite straight forward…

On Host1 (192.168.56.101)…

[root@CentOS1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-tun0 
DEVICE=tun0
BOOTPROTO=none
ONBOOT=no
TYPE=GRE
PEER_OUTER_IPADDR=192.168.56.102
PEER_INNER_IPADDR=192.168.168.2
MY_INNER_IPADDR=192.168.168.1
[root@CentOS1 ~]#

On host2 (192.168.56.102) ….

[root@CentOS2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-tun0 
DEVICE=tun0
BOOTPROTO=none
ONBOOT=no
TYPE=GRE
PEER_OUTER_IPADDR=192.168.56.101
PEER_INNER_IPADDR=192.168.168.1
MY_INNER_IPADDR=192.168.168.2
[root@CentOS1 ~]#

Bring the interfaces up….

[root@CentOS1 ~]# ifup tun0

.. on host2…

[root@CentOS2 ~]# ifup tun0

And we’re done! … see the proof in the pudding below….

[root@CentOS1 ~]# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-05-08-80-3C-00-00-00-00-00-00-00-00  
          inet addr:192.168.168.1  P-t-P:192.168.168.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP  MTU:1476  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:168 (168.0 b)  TX bytes:756 (756.0 b)

[root@CentOS1 ~]# ping 192.168.168.2
PING 192.168.168.2 (192.168.168.2) 56(84) bytes of data.
64 bytes from 192.168.168.2: icmp_seq=1 ttl=64 time=1.51 ms
64 bytes from 192.168.168.2: icmp_seq=2 ttl=64 time=2.13 ms
64 bytes from 192.168.168.2: icmp_seq=3 ttl=64 time=2.12 ms

--- 192.168.168.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.511/1.921/2.132/0.289 ms
[root@CentOS1 ~]#

The other end…

[root@CentOS2 ~]# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-05-08-80-4C-00-00-00-00-00-00-00-00  
          inet addr:192.168.168.2  P-t-P:192.168.168.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP  MTU:1476  Metric:1
          RX packets:42 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3528 (3.4 KiB)  TX bytes:4536 (4.4 KiB)

[root@CentOS2 ~]# ping 192.168.168.1
PING 192.168.168.1 (192.168.168.1) 56(84) bytes of data.
64 bytes from 192.168.168.1: icmp_seq=1 ttl=64 time=4.39 ms
64 bytes from 192.168.168.1: icmp_seq=2 ttl=64 time=1.41 ms
64 bytes from 192.168.168.1: icmp_seq=3 ttl=64 time=2.57 ms

--- 192.168.168.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2005ms
rtt min/avg/max/mdev = 1.419/2.795/4.393/1.224 ms
[root@CentOS2 ~]# 

Here we show the tunnelled packets…

[root@CentOS1 ~]# tcpdump -n -i eth1 proto 47
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
13:45:59.429315 IP 192.168.56.102 > 192.168.56.101: GREv0, length 88: IP 192.168.168.2 > 192.168.168.1: ICMP echo request, id 55053, seq 7, length 64
13:45:59.429315 IP 192.168.56.101 > 192.168.56.102: GREv0, length 88: IP 192.168.168.1 > 192.168.168.2: ICMP echo reply, id 55053, seq 7, length 64
13:46:00.530528 IP 192.168.56.102 > 192.168.56.101: GREv0, length 88: IP 192.168.168.2 > 192.168.168.1: ICMP echo request, id 55053, seq 8, length 64
13:46:00.530686 IP 192.168.56.101 > 192.168.56.102: GREv0, length 88: IP 192.168.168.1 > 192.168.168.2: ICMP echo reply, id 55053, seq 8, length 64
13:46:01.418447 IP 192.168.56.102 > 192.168.56.101: GREv0, length 88: IP 192.168.168.2 > 192.168.168.1: ICMP echo request, id 55053, seq 9, length 64
13:46:01.418526 IP 192.168.56.101 > 192.168.56.102: GREv0, length 88: IP 192.168.168.1 > 192.168.168.2: ICMP echo reply, id 55053, seq 9, length 64

6 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@CentOS1 ~]#

Since we can see the ICMP packets inside the GRE tunnel that show’s us that GRE is in clear text… to add some security setup a simple IPSEC VPN :)

Reference: http://juliano.info/en/Blog:Memory_Leak/Bridges_and_tunnels_in_Fedora

5 Minute CentOS/RHEL VPN

I’m looking at running two servers on EC2; as we all know the most important thing about running services in the cloud is encryption!

Whilst googling on how to setup a host-to-host IPSEC VPN I was surprised at how easy it is…

On Host1 (192.168.56.101)…

[root@CentOS1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ipsec1 
DST=192.168.56.102
TYPE=IPSEC
ONBOOT=no
IKE_METHOD=PSK
[root@CentOS1 ~]#
[root@CentOS1 ~]# cat /etc/sysconfig/network-scripts/keys-ipsec1 
IKE_PSK=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[root@CentOS1 ~]#
[root@CentOS1 ~]# ifup ipsec1

On host2 (192.168.56.102)…

[root@CentOS2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ipsec1 
DST=192.168.56.101
TYPE=IPSEC
ONBOOT=no
IKE_METHOD=PSK
[root@CentOS2 ~]#
[root@CentOS2 ~]# cat /etc/sysconfig/network-scripts/keys-ipsec1 
IKE_PSK=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[root@CentOS2 ~]#
[root@CentOS2 ~]#ifup ipsec1

… done!!!

[root@CentOS1 ~]# tcpdump -n -i eth1 host 192.168.56.102
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
09:46:37.306292 IP 192.168.56.101 > 192.168.56.102: AH(spi=0x0aff2b10,seq=0x203): ESP(spi=0x00a0a3cc,seq=0x203), length 84
09:46:37.310197 IP 192.168.56.102 > 192.168.56.101: AH(spi=0x09f82154,seq=0x203): ESP(spi=0x098f0ff9,seq=0x203), length 68
09:46:38.175048 IP 192.168.56.101 > 192.168.56.102: AH(spi=0x0aff2b10,seq=0x204): ESP(spi=0x00a0a3cc,seq=0x204), length 84
09:46:38.179017 IP 192.168.56.102 > 192.168.56.101: AH(spi=0x09f82154,seq=0x204): ESP(spi=0x098f0ff9,seq=0x204), length 68
09:46:39.313583 IP 192.168.56.101 > 192.168.56.102: AH(spi=0x0aff2b10,seq=0x205): ESP(spi=0x00a0a3cc,seq=0x205), length 84
09:46:39.316427 IP 192.168.56.102 > 192.168.56.101: AH(spi=0x09f82154,seq=0x205): ESP(spi=0x098f0ff9,seq=0x205), length 68

6 packets captured
6 packets received by filter
0 packets dropped by kernel
[root@CentOS1 ~]#

Now this is a simple IKE pre-shared key vpn, you might want to google for using certificates for stronger authentication, you can also edit /etc/racoon/racoon.conf to change your IPSEC parameters.

Reference: http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-vpn.html

UPDATE: To make this work in EC2, you need to enable NAT-T see my hack here!

Slammed by SPAM

Since moving to AWS linickx.com is suffering heavy pressure from blog-comment-spammers.

The above is a graphs of a days max CPU usage…. it ain’t pretty! Whilst I formulate a plan my apologies to my visitors.

Amazon CloudWatch CPU Gauge

The free monitoring you get with Amazon CloudWatch is pretty sweet, graphs are cool but gauges are cooler!

To get this working I’m having to run a bash script via cron every 5minutes which gets the max CPU usage from AWS and then downloads an image from the google charts API, I think the result is pretty snazzy!

Before trying to run this on your machine you will need the CloudWatch API Tools installed.

I’ll run though my bash script… to start with I need some date and time variables to work with:

# Setup some Time Variables
NowMins=`date "+%M"`
NowHrs=`date "+%H"`
Today=`date "+%Y-%m-%d"`

The free cloudwatch tier gives you five minute samples, I have seen that the output is updated every 10 minutes, so I’ll take 10 away from the current minutes to get the last “5″ minute result from cloudwatch…. it seems like odd logic but it does work… honest!

# The last five minute reading is avilable every ten mintues
FiveMinsAgo=$(($NowMins - 10))

We then need to fix the result of FiveMinsAgo since if the time now is 1minute past the current hour taking away ten will give a negative number rather than 51 minutes past the previous hour.

# If our calcualtion is a negative number correct hours and minutes
if [ $FiveMinsAgo -lt 0 ]
then
	FiveMinsAgo=$(($FiveMinsAgo + 60))
	NowHrs=$(($NowHrs - 1))
fi

# Re-Apply leading zero that gets dropped by calculations
if [ $FiveMinsAgo -lt 10 ]
then
        FiveMinsAgo="0"$FiveMinsAgo
fi

Once that’s all sorted, we can format a StartTime option for the API tools and make the request…

#Formate Start Time for API Tools
StartTime=$Today"T"$NowHrs":"$FiveMinsAgo":00.000Z"

# Get Results from API Tools
Result=`mon-get-stats CPUUtilization --namespace "AWS/EC2" --statistics "Maximum"  --dimensions "InstanceId=i-123456" --start-time $StartTime `

In the above you will need to change i-123456 with your Instance ID… as you can see I have stored the result as a variable which we cut it up below using awk to leave only the CPU % of the instance for use with google charts.

Percent=`echo $Result | awk '{print $3}'`

Chart="https://chart.googleapis.com/chart?chs=200x125&cht=gom&chd=t:"$Percent"&chl="$Percent"%&chco=00AA0066,FF804066,AA000066&chf=bg,s,FFFFFF00"

curl $Chart -o cpu.png

The final piece of the puzzle above is downloading the chart from google using curl, which results in cpu.png.

I then save this image on my webserver and create my own status dashboard! I’ve attached a copy of the script for your download if you wish to re-create the same thing.

Happy Gauging!