Category Archives: Blog

Blog posts by Nick

OCFS2 issues

This morning I’ve had issues with my linickx.com cluster, the file system on both nodes went to read-only and I ended up in a world of pain.

[root@georgia ~]# sudo /etc/init.d/httpd start
Starting httpd:
[root@georgia ~]# tail -f /var/log/messages
Jan  9 09:48:35 georgia kernel: [  474.259265] (httpd,1712,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259271] (httpd,1712,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259276] (httpd,1712,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259282] (httpd,1712,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 09:49:31 georgia kernel: [  530.660071] o2net: no longer connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 09:49:31 georgia kernel: [  530.661856] ocfs2: Unmounting device (147,0) on (node 2)
Jan  9 09:59:46 georgia kernel: [ 1145.772174] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 2
Jan  9 10:01:05 georgia kernel: [ 1223.911192] o2net: connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 10:01:09 georgia kernel: [ 1227.933348] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 1 2
Jan  9 10:01:09 georgia kernel: [ 1227.938693] ocfs2: Mounting device (147,0) on (node 2, slot 1) with ordered data mode.
Jan  9 10:02:35 georgia kernel: [ 1314.467741] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
Jan  9 10:02:35 georgia kernel: [ 1314.467754] File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted.
Jan  9 10:02:35 georgia kernel: [ 1314.467764] (httpd,2389,0):ocfs2_search_chain:1729 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467771] (httpd,2389,0):ocfs2_claim_suballoc_bits:1902 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467778] (httpd,2389,0):__ocfs2_claim_clusters:2185 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467783] (httpd,2389,0):ocfs2_local_alloc_new_window:1204 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467790] (httpd,2389,0):ocfs2_local_alloc_slide_window:1306 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467798] (httpd,2389,0):ocfs2_reserve_local_alloc_bits:695 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467803] (httpd,2389,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467809] (httpd,2389,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467814] (httpd,2389,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467821] (httpd,2389,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 10:02:36 georgia kernel: [ 1315.046965] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
^C
[root@georgia ~]#

What made this odd is that running fsck.ocfs2 as suggested made no difference, as the output said that the disk was clean.

[root@georgia ~]# fsck.ocfs2 /dev/drbd0
fsck.ocfs2 1.4.4
Checking OCFS2 filesystem in /dev/drbd0:
  Label:              linickxcluster
  UUID:               E9447DBE28154DAEA1B988CEC573EB64
  Number of blocks:   1048535
  Block size:         4096
  Number of clusters: 1048535
  Cluster size:       4096
  Number of slots:    4

/dev/drbd0 is clean.  It will be checked after 20 additional mounts.
[root@georgia ~]#

I learn that in fact the above output was a lie! For any future googlers seeing the same issue, run:

fsck.ocfs2 -fy /dev/drbd0

The f & y force a check and fix any found issues, the force on my filesystem found the errors and we appear to be back online :)

NYr2012

I’m not big on New Year’s Resolutions but this year I can feel a couple brewing.

Better Communication
I’m going to move to text only email, not because I’m an advocate… actually the opposite, I think RTF/HTML email can bring clarity to text… anyway … reason being I think that my communication could be improved. If I remove the luxury of bold/italic/etc then my text has to stand alone for what it is, hopefully this will lead to better text.

Social Networks
I fall in and out of love with my twitter & facebook account but FB’s move to disable the auto-magic import of linickx.com posts has forced me to think about this. My basic requirement is that, LINICKX.com is my source of noise and I need an elegant way to broadcast :)

Happy New yr to All!

Start with the ReadMe

I haven’t written much on here about RSDNS as I’m trying to keep everything in the github ecosystem.

Currently there is no web-page-documentation, forums or wiki’s. RSDNS usage is all documented in the README.md which should be the fist place to look if you want to get started. If you have a problem post an issue, if you can fix a problem raise a pull request.

All community contributions are greatly appreciated, happy RackSpace DNS everyone!

PHP to solve problems

PHP make you think of web app’s right? … well, did you know you can run it from the CLI to?

Recently I’ve been doing a lot of spreadsheet and sub-netting type stuff, whilst doing this mundane work I’ve realised that I can get scripts to work for me. I’ve started to post a few PHP network functions to github which I’ve been using.

Here’s an example: I have a nokia firewall, the routing table in voyager is shown in the following format -

Network / CIDR Mask , Gateway
10.0.0.0 / 8 , 10.0.0.1

I need that same routing in a Cisco ASA format -

Network , Mask , Gateway
10.0.0.0, 255.0.0.0, 10.0.0.1

Now this is not a problem for a few routes but the firewall I’m looking at now has 177 static routes, which I don’t want to convert manually.

Roll in PHP!
Save the original routing table as a .csv file. ColA = ip/mask , ColB = gateway.

Save this a route_conv.php

<?php

	/**

		Change the below to your CSV File.

	**/

	$firewall_csv = "./routes_cdr.csv"; 

	/**

		Function to convert CIDRs such as "23" to decimall dotted like "255.255.254.0"
		I've got more of these: https://gist.github.com/1309388

	**/

	function cidr2mask($netmask) {

		$netmask_result="";
		for($i=1; $i <= $netmask; $i++) {
		  $netmask_result .= "1";
		}

		for($i=$netmask+1; $i <= 32; $i++) {
		    $netmask_result .= "0";
		}

		$netmask_ip_binary_array = str_split( $netmask_result, 8 );

		$netmask_ip_decimal_array = array();
		foreach( $netmask_ip_binary_array as $k => $v ){
		    $netmask_ip_decimal_array[$k] = bindec( $v ); // "100" => 4
		}

		$subnet = join( ".", $netmask_ip_decimal_array );

		return $subnet;

	}

	ini_set('auto_detect_line_endings', true); // detect CR

	if (file_exists($firewall_csv)) {

		$file = fopen($firewall_csv, 'r');

		$counter = 0; // array counter

		while (($data = fgetcsv($file)) !== FALSE) {

			list($ip, $netmask) = split( "/", $data[0] ); // SPLIT Col A into IP & Mask

			$netmask = cidr2mask($netmask); // Covert Mask

			$gateway = $data[1]; // Col B

			/**
				This echo is the CSV style output, but you could change this to echo "route add $ip $mask $ gateway \n" for unix style output.
			**/

			echo $ip . "," . $netmask . "," . $gateway . "\n";

		}

		fclose($file);

	} else {

		echo "404: $firewall_csv \n"; // FILE NOT FOUND.

	}
?>

from your CLI run “php route_conv.php” and enjoy the output!

Unread Gmail on your OSX Desktop

1) Install GeekTool
2) Run this script….


USERNAME="me@gmail.com"
PASSWORD="password"

EMAIL=`curl -u $USERNAME:$PASSWORD --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"`

if [ -n "$EMAIL" ]
	then

	echo "INBOX:"
	echo "-----------------------------------------"

	IFS=$'\n'
	for i in $EMAIL
	do
		len=${#i}
		if [ "$len" -gt 40 ]
			then
			echo ${i:0:37} "..."
		else
			echo $i
		fi
	done
fi

3) smile