OCFS2 issues

This morning I’ve had issues with my linickx.com cluster, the file system on both nodes went to read-only and I ended up in a world of pain.

[root@georgia ~]# sudo /etc/init.d/httpd start
Starting httpd:
[root@georgia ~]# tail -f /var/log/messages
Jan  9 09:48:35 georgia kernel: [  474.259265] (httpd,1712,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259271] (httpd,1712,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259276] (httpd,1712,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 09:48:35 georgia kernel: [  474.259282] (httpd,1712,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 09:49:31 georgia kernel: [  530.660071] o2net: no longer connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 09:49:31 georgia kernel: [  530.661856] ocfs2: Unmounting device (147,0) on (node 2)
Jan  9 09:59:46 georgia kernel: [ 1145.772174] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 2
Jan  9 10:01:05 georgia kernel: [ 1223.911192] o2net: connected to node amy (num 1) at 10.176.128.7:7777
Jan  9 10:01:09 georgia kernel: [ 1227.933348] o2dlm: Nodes in domain E9447DBE28154DAEA1B988CEC573EB64: 1 2
Jan  9 10:01:09 georgia kernel: [ 1227.938693] ocfs2: Mounting device (147,0) on (node 2, slot 1) with ordered data mode.
Jan  9 10:02:35 georgia kernel: [ 1314.467741] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
Jan  9 10:02:35 georgia kernel: [ 1314.467754] File system is now read-only due to the potential of on-disk corruption. Please run fsck.ocfs2 once the file system is unmounted.
Jan  9 10:02:35 georgia kernel: [ 1314.467764] (httpd,2389,0):ocfs2_search_chain:1729 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467771] (httpd,2389,0):ocfs2_claim_suballoc_bits:1902 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467778] (httpd,2389,0):__ocfs2_claim_clusters:2185 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467783] (httpd,2389,0):ocfs2_local_alloc_new_window:1204 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467790] (httpd,2389,0):ocfs2_local_alloc_slide_window:1306 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467798] (httpd,2389,0):ocfs2_reserve_local_alloc_bits:695 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467803] (httpd,2389,0):ocfs2_reserve_clusters_with_limit:1190 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467809] (httpd,2389,0):ocfs2_lock_allocators:2546 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467814] (httpd,2389,0):ocfs2_write_begin_nolock:1732 ERROR: status = -22
Jan  9 10:02:35 georgia kernel: [ 1314.467821] (httpd,2389,0):ocfs2_write_begin:1856 ERROR: status = -22
Jan  9 10:02:36 georgia kernel: [ 1315.046965] OCFS2: ERROR (device drbd0): ocfs2_validate_gd_self: Group descriptor #419328 has bit count 32256 but claims that 45941 are free
^C
[root@georgia ~]#

What made this odd is that running fsck.ocfs2 as suggested made no difference, as the output said that the disk was clean.

[root@georgia ~]# fsck.ocfs2 /dev/drbd0
fsck.ocfs2 1.4.4
Checking OCFS2 filesystem in /dev/drbd0:
  Label:              linickxcluster
  UUID:               E9447DBE28154DAEA1B988CEC573EB64
  Number of blocks:   1048535
  Block size:         4096
  Number of clusters: 1048535
  Cluster size:       4096
  Number of slots:    4

/dev/drbd0 is clean.  It will be checked after 20 additional mounts.
[root@georgia ~]#

I learn that in fact the above output was a lie! For any future googlers seeing the same issue, run:

fsck.ocfs2 -fy /dev/drbd0

The f & y force a check and fix any found issues, the force on my filesystem found the errors and we appear to be back online :)

NYr2012

I’m not big on New Year’s Resolutions but this year I can feel a couple brewing.

Better Communication
I’m going to move to text only email, not because I’m an advocate… actually the opposite, I think RTF/HTML email can bring clarity to text… anyway … reason being I think that my communication could be improved. If I remove the luxury of bold/italic/etc then my text has to stand alone for what it is, hopefully this will lead to better text.

Social Networks
I fall in and out of love with my twitter & facebook account but FB’s move to disable the auto-magic import of linickx.com posts has forced me to think about this. My basic requirement is that, LINICKX.com is my source of noise and I need an elegant way to broadcast :)

Happy New yr to All!

Start with the ReadMe

I haven’t written much on here about RSDNS as I’m trying to keep everything in the github ecosystem.

Currently there is no web-page-documentation, forums or wiki’s. RSDNS usage is all documented in the README.md which should be the fist place to look if you want to get started. If you have a problem post an issue, if you can fix a problem raise a pull request.

All community contributions are greatly appreciated, happy RackSpace DNS everyone!

Unread Gmail on your OSX Desktop

1) Install GeekTool
2) Run this script….


USERNAME="me@gmail.com"
PASSWORD="password"

EMAIL=`curl -u $USERNAME:$PASSWORD --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"`

if [ -n "$EMAIL" ]
	then

	echo "INBOX:"
	echo "-----------------------------------------"

	IFS=$'\n'
	for i in $EMAIL
	do
		len=${#i}
		if [ "$len" -gt 40 ]
			then
			echo ${i:0:37} "..."
		else
			echo $i
		fi
	done
fi

3) smile

What am I talking about

I use a fair amount of “text speak” here is a quick translation:

AFAIK = As Far As I Know
AKA = Also Known As
ASAP = As Soon As Possible
BTW = By The Way
ETA = Estimated Time of Arrival
FAQ = Frequently Asked Question
FUD = Fear, Uncertainty and Doubt
FYI = For Your Information
WTF = What The Feck?
LOL = Laugh Out Loud
IMHO = In My Humble Opinion
PEBKAC = Problem Exists Between Keyboard And Chair
ROTFL = Rolling On The Floor Laughing
RTFM = Read The Fecking Manual or Read The Fine Manual
THX = Thanks
WRT = With Regard To

LINK: http://cl.ly/0j1d3p162S3q0k2C3h2F

HAT Tip -> http://luisbg.blogalia.com//historias/70143 <- I’ve actullay edited this list sligtly to refect my mannerisms.

Secret Keys for the Cloud

I’ve had an idea, whether it’s a good one or not is yet to be seen; one of the big issues to cloud application and servers is encryption key management, there is a simple chicken n egg issue, if the secret key is on the server/application then it’s a vector to be attacked if the key isn’t then usability issues exist.

My idea is a CA / DH kinda thing, what if the actual key used for encryption was derived from the cloud it’s self, the basic premise is adding an extra layer to be compromised in order for an attacker to decypt the data.

Using RedHat’s new OpenShift service I’ve knocked up a demo -> secretkey-linickx.rhcloud.com. The demo is over HTTP (not HTTPS) so You wouldn’t use the demo in production probably because you do not trust me but I’ve pushed the code to github -> github.com/linickx/secretkey for users/dev/people/someone to take a copy and have a play.

Comments welcome, Pull requests preferred!

2011-07-26 UPDATE: Openshift has SSL termination, HTTPS does work, however as seen in my commit log the PHP cannot detect it as the SSL is being handled by a proxy.

linickx on github


For your social coding pleasure, linickx code is now in github!

Yesterday I completed the subversion mirror of my WordPress projects – phpbb_recent_topics , root cookie and linickx lifestream – now this isn’t a migration, it is a mirror! For the time being I’m happy using the subversion tools provided by Automattic and the WordPress team but I understand that git is gaining momentum and many are switching, basically I’m hoping this makes it easier for the WP community to get in touch or make suggestions to the code.

I’m also working on uploading some of my old work, I rely heavily on google to broadcast my wares and perhaps there are some old dinosaurs that need resurrecting by a new community of devs? Well if you’re feeling nosey “A is for Abandonware

It’s likely that new code and projects will appear on git hub, I’ve been toying with running subversion on linickx.com but now that Xcode4 has git built in this cloud based social service might be a better option…. I guess only time will tell!

Enjoy!

Undelete

Link

I’ve just used this little gem to recover files off a memory card… awesome!

PhotoRec is file data recovery software designed to recover lost files including video, documents and archives from hard disks, CD-ROMs, and lost pictures (thus the Photo Recovery name) from digital camera memory. PhotoRec ignores the file system and goes after the underlying data, so it will still work even if your media’s file system has been severely damaged or reformatted.
PhotoRec is free – this open source multi-platform application is distributed under GNU General Public License.

Link: http://www.cgsecurity.org/wiki/PhotoRec

GTD in your Shell

For the last few months I’ve felt better at managing my actions and tasks thanks to Steve Losh’s T. T is a simple python script that allows you to manage a simple task list from your shell prompt.

I use my shell all the time to ping stuff, renice processes and you know, whatever (like in the image below)…..

Screenshot of T in iTerm

… having my todo list right there in front of me has been really helpful (The numbers in the square brackets!). Steve’s website suggests a quick update to your shell profile to add your task list to your prompt but I found running a python script each time I do anything quite slow, especially since I have two lists.

Below is my bash .profile file, there you can see the rather OTT change(s) I’ve made to integrate my todo lists into my shell prompt. I have two lists (work & personal) the number of things to do shown in my shell prompt gets updated each time the lists change (thanks to md5)…. and I hide my work list over the weekend ;)

alias p='python ~/bin/t/t.py --task-dir ~/Dropbox/Tasks --list personal.tasks --delete-if-empty'
alias w='python ~/bin/t/t.py --task-dir ~/Dropbox/Tasks --list work.tasks --delete-if-empty'

function nick_PS1 {

        if [ -e "/Users/nick/Dropbox/Tasks/Personal.tasks" ]
        then
                CurrentPersonalMD5=`md5 ~/Dropbox/Tasks/Personal.tasks | awk '{print $4}' | cat `

                if [ -e "~/.tasks.personal.md5" ]
                then
                        LastPersonalMD5=`cat ~/.tasks.personal.md5`
                fi

                if [ "$CurrentPersonalMD5" == "$LastPersonalMD5" ]
                then
                        if [ -e "~/.tasks.personal.no" ]
                        then
                                NumOfPersonal=`cat ~/.tasks.personal.no`
                        fi
                else
                        NumOfPersonal=`p | wc -l | sed -e's/ *//'`
                        echo $NumOfPersonal > ~/.tasks.personal.no
                        echo $CurrentPersonalMD5 > ~/.tasks.personal.md5
                fi

                if [ $NumOfPersonal -ne 0 ]
                then
                        LEFT='['
                        RIGHT=']'
                        Personal="p:$NumOfPersonal"
                fi

        else
                NumOfPersonal=0
        fi

DayOfWeek=`date "+%u"`
if [ $DayOfWeek -lt 6 ]
then
        if [ -e "/Users/nick/Dropbox/Tasks/Work.tasks" ]
        then
                CurrentWorkMD5=`md5 ~/Dropbox/Tasks/Work.tasks | awk '{print $4}'`

                if [ -e "~/.tasks.work.md5" ]
        then
                LastWorkMD5=`cat ~/.tasks.work.md5`
        fi

                if [ "$CurrentWorkMD5" == "$LastWorkMD5" ]
        then
                        if [ -e "~/.tasks.work.no" ]
                        then
                                NumOfWork=`cat ~/.tasks.work.no`
                        fi
                else
                                        NumOfWork=`w | wc -l | sed -e's/ *//'`
                                        echo $NumOfWork > ~/.tasks.work.no
                                        echo $CurrentWorkMD5 > ~/.tasks.work.md5
                fi

                if [ $NumOfWork -ne 0 ]
                then
                        LEFT='['
                        RIGHT=']'
                        Work="w:$NumOfWork"
                fi

        else
                NumOfWork=0
        fi
else
 NumOfWork=0
fi

        if [ $NumOfPersonal -ne 0 ]
        then
                if [ $NumOfWork -ne 0 ]
                then
                        SEP="|"
                fi
         fi

        RESULT="$LEFT$Personal$SEP$Work$RIGHT"

        if [ -z $RESULT ]
        then
                RESULT="linickx@macbookpro"
        fi

        echo $RESULT
}

export PS1="\$(nick_PS1) \W \$"

… I think my next stop will be using geektool show the list/reminders on my desktop.

Lowing VirtualBox priorities

One of the things I’d really like is process priorities for virtual box. In the forum I posted a couple of shell commands that I regularly type… which gets a bit tedious, following a recent article on lifehacker reviewing mac text expanding I’ve been prompted to automate a few things… below is a little shell script to lower the priority (renice) of all running virtual machines.

The advantage of doing this is that your host machine stays snappy, responsive and won’t get too over-loaded by jobs on your VMs!

#!/bin/bash
ps -xo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

The above code works on a mac; although I haven’t tested it, I recon to get it running on Linux you need to update the PS command, by swapping the x for an e… like this….

#!/bin/bash
ps -eo pid,command | grep -v grep | grep startvm | while read line ;
do
        procID=`echo $line | awk '{print $1}'`
        sudo renice +10 -p $procID
done

Have fun, suggestions and improvements welcome.

OS X: Change google Chrome to Search .co.uk instead of .com


I’ve just gotten around to solving this little niggle; having google search google.com by default instead of google.co.uk was one of those little annoyances that I was just living with.

Today I found a solution on Chrome Bug 1521 on Comment 39

- Quit Chrome
- Open ~/Library/Application Support/Google/Chrome/Local State in your favorite text editor.
- Search for the strings ‘last_known_google_url’ and ‘last_prompted_google_url’ adn replace their values to your preferred Google base URL (e.g. www.google.com)
- Save and start chrome back up.

Yep, that works!