Tag Archives: Linux

apc.php for rhel / centos

Note to self: The apc.php (script for monitroing apc performance) is stored in – /usr/share/doc/php-pecl-apc-3.1.3p1

Speed, I am Speed

Recently I’ve been focusing on getting linickx.com not only available, but snappy too!

  1. HA Proxy distributes the load across two cloud servers
  2. Varnish Caches have a hit rate of between 30% & 49%
  3. WP Super Cache serves up static files to Varnish
  4. HypderDB distributes MySQL Requests across my two servers

My next step is to get some minification going, either with page-speed or a WP-Plugin.. they seem to be a bit tricky to troubleshoot tho!

If you are looking to measure your websites performance, I’ve been using pingdom for server response times and webpagetest.org for gathering client load speeds.

Hacking Cisco ISE UDI

ISE Virtual Machine that thinks it a CAM
The back story… you’ve deployed your ISE appliance and the world is great! Your management need you to make a change “right now” but that virtual machine in the lab you have been using for testing is 91 days old and the eval no longer works. You raise a case to get budget and a PO over to Cisco for a lab ISE appliance or license but this change is critical, if only there was a way to use your appliance license on your VM?

Perhaps you should log into your ISE appliance and make a note of the Product Identifier (PID), Version Identifier (VID) and the Serial Number (SN).

What you might want to do now is shutdown your ISE VM and mount the disk… I always have a CentOS server kicking around for this kind of thing, so if I was to do this, I would mount the ISE virtual disk as an extra disk that CentOS has access to.

From within CentOS you can use fdisk -l to view the hard drive partitions… When you’re hacking a VM you mount as many of the ISE partitions as you can (some will fail) to see what’s there. On my test machine /dev/sdb7 was the partition of interest as it had an /opt directory (cisco always install stuff in opt).

Inside mount-point/opt/system/bin/ you might find a file called cars_udi_util, that’s the puppy that the license is bound to.

What you might want to do is rename that file and replace it with something that always gives the “right” answer. Attached is cars_udi_util.txt, a shell script I have been testing, edit the top of the file and insert the PID/VID/SN you found earlier.

Now save the cars_udi_util.txt to mount-point/opt/system/bin/cars_udi_util, that’s right remember to remove the .txt!

Unmount the disk, shutdown Centos and boot up ISE.

Now I’ve been hacking my machine and after this change the services wouldn’t start (show application status ise) to fix that I ran application reset-config ise from the ISE CLI Shell, rebooted and Voila! …The machine booted up with a blank default config.

After changing the default admin password (from cisco) it would now be possible for you to use your proper appliance license on your VM… of course this is only a temporary thing and I fully expect & recommend you undo these changes as soon as your new license arrives from Cisco.

Happy Hacking!

OCFS2 issues

This morning I’ve had issues with my linickx.com cluster, the file system on both nodes went to read-only and I ended up in a world of pain.

[root@georgia ~]# sudo /etc/init.d/httpd start
Starting httpd:
[root@georgia ~]# tail -f /var/log/messages
Jan  9 09:48:35 georgia kernel: [  474.259265] (httpd,1712,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259271] (httpd,1712,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259276] (httpd,1712,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259282] (httpd,1712,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 09:49:31 georgia kernel: [  530.660071] o2net: no longer connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 09:49:31 georgia kernel: [  530.661856] ocfs2: Unmounting device (147,0) on (node 2)
Jan  9 09:59:46 georgia kernel: [ 1145.772174] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 2
Jan  9 10:01:05 georgia kernel: [ 1223.911192] o2net: connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 10:01:09 georgia kernel: [ 1227.933348] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 1 2
Jan  9 10:01:09 georgia kernel: [ 1227.938693] ocfs2: Mounting device (147,0) on (node 2, slot 1) with ordered data mode.
Jan  9 10:02:35 georgia kernel: [ 1314.467741] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
Jan  9 10:02:35 georgia kernel: [ 1314.467754] File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted.
Jan  9 10:02:35 georgia kernel: [ 1314.467764] (httpd,2389,0):ocfs2_search_chain:1729 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467771] (httpd,2389,0):ocfs2_claim_suballoc_bits:1902 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467778] (httpd,2389,0):__ocfs2_claim_clusters:2185 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467783] (httpd,2389,0):ocfs2_local_alloc_new_window:1204 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467790] (httpd,2389,0):ocfs2_local_alloc_slide_window:1306 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467798] (httpd,2389,0):ocfs2_reserve_local_alloc_bits:695 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467803] (httpd,2389,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467809] (httpd,2389,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467814] (httpd,2389,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467821] (httpd,2389,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 10:02:36 georgia kernel: [ 1315.046965] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
^C
[root@georgia ~]#

What made this odd is that running fsck.ocfs2 as suggested made no difference, as the output said that the disk was clean.

[root@georgia ~]# fsck.ocfs2 /dev/drbd0
fsck.ocfs2 1.4.4
Checking OCFS2 filesystem in /dev/drbd0:
  Label:              linickxcluster
  UUID:               E9447DBE28154DAEA1B988CEC573EB64
  Number of blocks:   1048535
  Block size:         4096
  Number of clusters: 1048535
  Cluster size:       4096
  Number of slots:    4

/dev/drbd0 is clean.  It will be checked after 20 additional mounts.
[root@georgia ~]#

I learn that in fact the above output was a lie! For any future googlers seeing the same issue, run:

fsck.ocfs2 -fy /dev/drbd0

The f & y force a check and fix any found issues, the force on my filesystem found the errors and we appear to be back online :)

PHP to solve problems

PHP make you think of web app’s right? … well, did you know you can run it from the CLI to?

Recently I’ve been doing a lot of spreadsheet and sub-netting type stuff, whilst doing this mundane work I’ve realised that I can get scripts to work for me. I’ve started to post a few PHP network functions to github which I’ve been using.

Here’s an example: I have a nokia firewall, the routing table in voyager is shown in the following format -

Network / CIDR Mask , Gateway
10.0.0.0 / 8 , 10.0.0.1

I need that same routing in a Cisco ASA format -

Network , Mask , Gateway
10.0.0.0, 255.0.0.0, 10.0.0.1

Now this is not a problem for a few routes but the firewall I’m looking at now has 177 static routes, which I don’t want to convert manually.

Roll in PHP!
Save the original routing table as a .csv file. ColA = ip/mask , ColB = gateway.

Save this a route_conv.php

<?php

	/**

		Change the below to your CSV File.

	**/

	$firewall_csv = "./routes_cdr.csv"; 

	/**

		Function to convert CIDRs such as "23" to decimall dotted like "255.255.254.0"
		I've got more of these: https://gist.github.com/1309388

	**/

	function cidr2mask($netmask) {

		$netmask_result="";
		for($i=1; $i <= $netmask; $i++) {
		  $netmask_result .= "1";
		}

		for($i=$netmask+1; $i <= 32; $i++) {
		    $netmask_result .= "0";
		}

		$netmask_ip_binary_array = str_split( $netmask_result, 8 );

		$netmask_ip_decimal_array = array();
		foreach( $netmask_ip_binary_array as $k => $v ){
		    $netmask_ip_decimal_array[$k] = bindec( $v ); // "100" => 4
		}

		$subnet = join( ".", $netmask_ip_decimal_array );

		return $subnet;

	}

	ini_set('auto_detect_line_endings', true); // detect CR

	if (file_exists($firewall_csv)) {

		$file = fopen($firewall_csv, 'r');

		$counter = 0; // array counter

		while (($data = fgetcsv($file)) !== FALSE) {

			list($ip, $netmask) = split( "/", $data[0] ); // SPLIT Col A into IP & Mask

			$netmask = cidr2mask($netmask); // Covert Mask

			$gateway = $data[1]; // Col B

			/**
				This echo is the CSV style output, but you could change this to echo "route add $ip $mask $ gateway \n" for unix style output.
			**/

			echo $ip . "," . $netmask . "," . $gateway . "\n";

		}

		fclose($file);

	} else {

		echo "404: $firewall_csv \n"; // FILE NOT FOUND.

	}
?>

from your CLI run “php route_conv.php” and enjoy the output!

Building a free Dynamic DNS client with rackspace Cloud


As a cloud server customer you get access to rackspace’s free DNS service.

When I fist saw this product I had an instance light-bulb moment, I could stop paying for a dynamic DNS service and build my own private one. As a broadband (DHCP) user I have a very basic requirement of needing to regularly update an A record so that I can find my pc :)

To bring my idea into fruition I began researching; I need a cli tool which I could run from cron on my linux box (to send the DNS update requests to rackspace). In my research I found rscurl, a cli tool to control cloud servers, as rackspace have a standard API for all their products I have been able to use rscurl to develop rsdns.

rsdns is a series of cli tools to adding/deleting/changing rackspace DNS records, as part of the tool development I have created a script called rsdns-dc.sh to run on my machine, below is a short how to:

How to get free dynamic dns from rackspace.

Continue reading →

F5 BigIP LTM VE works in Virtual Box

Something I discovered ages ago (so long ago that my trial license expired) but forgot to post is that you can get an LTM VE to work in Virtual Box.

To get started download the ESX image from the F5 VE Trial Page, when you get the download import the OVA into virtualbox.

The only thing I needed to tweak after the import was the interface settings, you need two intels and a PCNet, the PCNet is the management interface. Set the PCNet to host only networking, give your laptop/pc an ip address on the host only network a 192.168.1 address and you’re good to go!

You may experience HIGH CPU issues after boot, but since these boxes are based on linux, you can use the divider=10 centos trick.

Enjoy your virtual load balancing!

Lowing VirtualBox priorities

One of the things I’d really like is process priorities for virtual box. In the forum I posted a couple of shell commands that I regularly type… which gets a bit tedious, following a recent article on lifehacker reviewing mac text expanding I’ve been prompted to automate a few things… below is a little shell script to lower the priority (renice) of all running virtual machines.

The advantage of doing this is that your host machine stays snappy, responsive and won’t get too over-loaded by jobs on your VMs!

#!/bin/bash
ps -xo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

The above code works on a mac; although I haven’t tested it, I recon to get it running on Linux you need to update the PS command, by swapping the x for an e… like this…. 

#!/bin/bash
ps -eo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

Have fun, suggestions and improvements welcome.

CentOS/Redhat IPSEC and EC2

So it turns out my 5 minute vpn doesn’t work in EC2 because the ESP/AH protocols (50 and 51) are blocked on the AWS network.

This is no big deal tho, as NAT-T allows one to tunnel IPSEC over UDP… however getting it to work on CentOS required a bit of a hack.

If you have already tried setting up an IPSEC vpn, shut it down with ifdown ipsec1 and remove your /etc/racoon/192.168.56.101.conf (or whatever IP yours is).

To start the hack on BOTH boxes, you need to edit /etc/sysconfig/network-scripts/ifup-ipsec. Around line 215 you need to insert nat_traversal force;… like this….

BEFORE:

        case "$IKE_METHOD" in
           PSK)
              cat >> /etc/racoon/$DST.conf << EOF
        my_identifier address;
        proposal {
                encryption_algorithm $IKE_ENC;
                hash_algorithm $IKE_AUTH;
                authentication_method pre_shared_key;
                dh_group $IKE_DHGROUP;
        }
}

AFTER:

        case "$IKE_METHOD" in
           PSK)
              cat >> /etc/racoon/$DST.conf << EOF
        my_identifier address;
        nat_traversal force;
        proposal {
                encryption_algorithm $IKE_ENC;
                hash_algorithm $IKE_AUTH;
                authentication_method pre_shared_key;
                dh_group $IKE_DHGROUP;
        }
}

Again, on both boxes update your /etc/sysconfig/network-scripts/ifcfg-ipsec1 files so that AH is disabled… because AH doesn’t like NAT… like this….


[root@CentOS2 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ipsec1
DST=192.168.56.101
TYPE=IPSEC
ONBOOT=yes
IKE_METHOD=PSK
AH_PROTO=none
[root@CentOS2 ~]#

On your iptables policy make sure that UDP 500 and UDP 4500 are permitted and volia.

# tcpdump -n -i eth1 port not 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
20:26:49.257590 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xa), length 116
20:26:49.261076 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xa), length 116
20:26:50.260942 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xb), length 116
20:26:50.262939 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xb), length 116
20:26:51.261298 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xc), length 116
20:26:51.264974 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xc), length 116
20:26:52.262289 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xd), length 116
20:26:52.265488 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xd), length 116
20:26:53.264008 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xe), length 116
20:26:53.267003 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xe), length 116
20:26:54.265655 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0xf), length 116
20:26:54.267264 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0xf), length 116
20:26:55.267459 IP 192.168.56.101.ipsec-nat-t > 192.168.56.102.ipsec-nat-t: UDP-encap: ESP(spi=0x08de7c32,seq=0x10), length 116
20:26:55.269678 IP 192.168.56.102.ipsec-nat-t > 192.168.56.101.ipsec-nat-t: UDP-encap: ESP(spi=0x03787bd0,seq=0x10), length 116
14 packets captured
14 packets received by filter
0 packets dropped by kernel
#

IPSEC VPN Tunnelling over UDP…. done!

SELINUX and OSSEC IPTables error

OSSEC is my favourite linux HIDS however now that I’m running a SELINUX secured web server I noticed that my active responses were not working after a reboot.

After enabling SELINUX, I started getting alerts about the following problem in my messages file….

Nov 11 12:16:22 amy kernel: type=1400 audit(1289477782.569:8): avc:  denied  { read write } for  pid=2551 comm="iptables" path="socket:[5261]" dev=sockfs ino=5261 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=unix_dgram_socket

This appears to be ossec trying to update iptables, but failing as they’re in different contexts… now I’m no selinx expert but this CentOS Wiki Page helped… run the following command which will create osseciptables.pp in the current directtory…

root@amy# grep iptable /var/log/messages | audit2allow -M osseciptables

This creates a new binary module that can be installed with….

/usr/sbin/semodule -i osseciptables.pp

You can view current selinux modules with …

/usr/sbin/semodule -l

If you want to see what is being created by audit to allow, try the following…

root@amy# grep iptable /var/log/messages | audit2allow -m osseciptables

module osseciptables 1.0;

require {
        type iptables_t;
        type initrc_t;
        class unix_dgram_socket { read write };
}

#============= iptables_t ==============
allow iptables_t initrc_t:unix_dgram_socket { read write };
root@amy#

I hope this helps some future googler!